Seminar: From Entity-Centric to Event Centric Multimodal Knowledge Acquisition
Manling Li
Ph.D. Candidate, University of Illinois
Urbana-Champaign
Wednesday, January 18, 2023
11:00 AM - 12:00 PM
via Zoom
Abstract
Events (what happened, who, when, where, why) describe fundamental human activities and are the core knowledge commicated through multiple forms of information, such as text, images, videos, or other data modtalities. Our minds represent events at various levels of granularity and abstraction, which allows us to quickly access historical scenarios and reason about the future. Traditionally, multimodal information consumption has been entity-centric with a focus on concrete concepts (such as objects, object types, physical relations), or oversimplifying event understanding to be single-modal (text-only or vision-only), local, sequential and flat. Real events are multimodal structured and probabilistic. Hence, I focus on Multimodal Information Extraction, and propose Event-Centric Multimodal Knowledge Aquisition to transform traditional entity-centric single-modal knowledge into event-centric multi-modal knowledge. Such a transformation poses two significant challenges: (1) understanding multimodal semantic structures that are abstract (such as events and semantic roles of objects): I will present a novel framework, CLIP-Event, to learn visual semantic structures via a zero-shot cross-modal transfer; (2) understanding temporal dynamics: I will introduce Event Graph Schema to capture complex timelines, intertwined participant relations and multiple possible outcomes. Such Event-Centric Multimodal Knowledge opens up the next generation of information access for deep semantic understandings behind the multimodal information. I will also introduce the strong power of event graphs on supporting long-standing open problems, such as timeline generation, meeting summarization, and question answering.
Biography
Manling Li is a Ph.D. candidate at the Computer Science Department of University of Illinois Urbana-Champaign. Her work on multimodal knowledge extraction won the ACL'20 Best Demo Paper Award, and the work on the scientific information extraction from COVID literature won NAACL'21 Best Demo Paper Award. She was a recipient of Microsoft Research PhD Fellowship in 2021. She was selected as a DARPA riser in 2022, and a EE CS Rising Star in 2022. she was awarded C.L. Dave and Jane W.S. Liu Award, and has been selected as a Mavis Future Faculty Fellow. She led 19 students to develop the UIUC information extraction system and ranked 1st in DARPA AIDA evaluation in 2019 and 2020. She has more than 30 publications on multimodal knowledge extraction and reasoning, and gave tutorials about event-centric multimodal knowledge at ACL'21, AAAI'21, NAACL'22, AAAI'23, etc. Additional information is available at https://limanling.github.io/