Funding Agency: CIA. This project is funded through an IC Postdoctoral Fellowship program. See http://www.icpostdoc.org for more information.

This project is kicking off in January 2011.

Analysis of massive multimedia collections is at the core of the intelligence community today and will be increasingly so in the future. Video data is being collected at alarming rates and yet there exists no comprehensive forensic toolset that enables the intelligence analyst to quickly exploit and analyze video in the context of the massive collections. Sifting through hours of video to find a needle is laborious and error-prone. Video analysis needs to happen at the semantic level to facilitate efficient and effective exploitation. We pose and investigate the semantic video summarization problem, which requires a joint solution to semantic entity extraction, entity-entity relationship extraction, dynamic event recognition, and video categorization.

We investigate an approach for semantic summarization of video content across massive collections. Our approach is grounded in formal ontologyÑindeed the semantics we use to capture the domain entities and how they interrelateÑbut this ontology is jointly induced from the data and established by the human domain experts (i.e., interactive machine learning). The ontology is rigorously married to the underlying statistical mathematical representation (a multilevel Markov network) and inference is automatic on a given video.