- admin
- 0 Comments
- 618 Views
Abstract:
Traditionally, multimodal factual knowledge extraction has been entity-centric with a focus on concrete concepts (such as objects, object types, physical relations, e.g., a person in a car), but lacks ability to understand abstract semantics (such as events and semantic roles of objects, e.g., driver, passenger). However, such event-centric semantics are the core knowledge communicated, regardless whether in the form of text, images, videos, or other data modalities.
At the core of my research in Multimodal Information Extraction (IE) is to bring such deep factual knowledge view to the multimodal world. My work opens up a new research direction Event-Centric Multimodal Knowledge Acquisition to transform traditional entity-centric single-modal knowledge into event-centric multi-modal knowledge. Such a transformation poses two significant challenges: (1) understanding multimodal semantic structures that are abstract (such as events and semantic roles of objects): I will present my solution of zero-shot cross-modal transfer (CLIP-Event), which is the first to model event semantic structures for vision-language pretraining, and supports zero-shot multimodal event extraction for the first time; (2) understanding long-horizon temporal dynamics: I will introduce Event Graph Model, which empowers machines to capture complex timelines with multiple alternative outcomes. I will also show its positive results on long-standing open problems, such as timeline generation, meeting summarization, and question answering. I will then lay out how I plan to promote factuality and truthfulness in multimodal information access, through a structured knowledge view that is easily explainable, highly compositional, and capable of long-horizon reasoning.
About our speaker:
Manling Li (University of Illinois Urbana-Champaign, USA) Manling Li is an incoming assistant professor at the Computer Science Department of Northwestern University. Before joining Northwestern in Fall 2024, she will be a postdoc at Stanford University. She obtained her PhD degree in computer science at University of Illinois Urbana-Champaign in 2023. Her work on multimodal knowledge extraction won the ACL’20 Best Demo Paper Award, and the work on scientific information extraction from COVID literature won NAACL’21 Best Demo Paper Award. She was a recipient of Microsoft Research PhD Fellowship in 2021. She was selected as a DARPA Riser in 2022, and an EE CS Rising Star in 2022. She was awarded C.L. Dave and Jane Liu Award, and has been selected as a Mavis Future Faculty Fellow. She has more than 30 publications on multimodal knowledge extraction and reasoning, and gave tutorials about event-centric multimodal knowledge at ACL’21, AAAI’21, NAACL’22, AAAI’23, etc. Additional information is available at https://limanling.github.io.
