People: Jason Corso (PI), Raymond Fu, Werner Ceusters, Venkat Krovi, and Michalis Petropoulos
Funding Agency: DARPA Mind's Eye in TCTO.

This project kicked off in June 2010.

Comprehensive visual scene understanding has long been the ultimate challenge in computer vision research. While images and videos of the natural world are highly structured and redundant (Kersten, 1987; Ruderman, 1994), they exhibit complex appearance and shape, complex hierarchical scale-varying nature, and occlusion. Early successes have focused on particular sub-problems, such as face detection (Viola and Jones, 2002, 2004). State of the art systems are capable of detecting instances of objects-the "nouns" of the scene-among few hundreds of object classes (Fei-Fei et al., 2004) and contests such as the PASCAL Challenge annually pit the world's best object detection methods on novel datasets. Although some may argue these object detection methods have not been thoroughly evaluated in the wild, a more elusive problem now presents itself: the "verbs" of the scene. As Biederman stated, nearly 30 years ago, specifying not only the elements in an image but also the manner in which they are interacting and relating to one another is integral to full image understanding (Biederman, 1981).

However, representing and recognizing actions (especially those of humans), with a view to understanding their underlying motivation, has proved to be an extremely challenging task because: (A) Motion is the projected output of a set of coordinated actions of often high-dimensional systems, an extremely high-dimensional neuro-musculo-skeletal one in the case of humans (and thus not particularly well-suited for any attempt at reconstruction solely on the basis of visual observation of coordinated actions); (B) Motion occurs and gets described semantically/linguistically at a wide variety of spatiotemporal scales (i.e. varying levels of abstraction serve to agglomerate or subdivide either spatial- and temporal-characteristics and careful attention paid to appropriate creation of "equivalence classes"); (C) Most importantly, the unambiguous extraction of intent from motion alone can never be achieved due to the significant dependence upon contextual knowledge (making the case for the development of a systematic ontology in which to ground the visual reasoning).

• May 11 Annotation of human keypoints, human segmentations, and related processing. This set includes all videos used in the ARL Feb '11 roundtable set (52 videos) fully annotated. [Download]
• August 11 Annotation of verb times on C-D1 (2740 of the 3492 remaining ones being added). This set includes time labels for each of the verb's identified in the C-D1 videos (using the DARPA HR data). [Download]

Please acknowledge any use of these releases

• The Summer of Code 2010 The Visual Noun is closely related to the ISTARE effort and will help define the foundation on which the new research will build.

[1]	J. J. Corso. Toward parts-based scene understanding with pixel-support parts-sparse pictorial structures. Pattern Recognition Letters: Special Issue on Scene Understanding and Behavior Analysis, 2013. To appear. Early version appears as arXiv.org tech report 1108.4079v1. [ bib | .pdf ]
[2]	R. Xu, P. Agarwal, S. Kumar, V. N. Krovi, and J. J. Corso. Combining skeletal pose with local motion for human activity recognition. In Proceedings of VII Conference on Articulated Motion and Deformable Objects, 2012. [ bib | .pdf ]
[3]	P. Agarwal, S. Kumar, J. Ryde, J. J. Corso, and V. N. Krovi. Estimating human dynamics on-the-fly using monocular video for pose estimation. In Proceedings of Robotics Science and Systems, 2012. [ bib | .pdf ]
[4]	S. Sadanand and J. J. Corso. Action bank: A high-level representation of activity in video. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2012. [ bib | code | project | .pdf ]
[5]	C. Xu and J. J. Corso. Evaluation of super-voxel methods for early video processing. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2012. [ bib | code | project | .pdf ]
[6]	Y. Miao and J. J. Corso. Hamiltonian streamline guided feature extraction with applications to face detection. Technical Report 1108.3525v1, arXiv, August 2011. [ bib ]
[7]	J. A. Delmerico, P. David, and J. J. Corso. Building facade detection, segmentation, and parameter estimation for mobile robot localization and guidance. In Proceedings of International Conference on Intelligent Robots and Systems, 2011. [ bib | project | data | .pdf ]
[8]	P. Agarwal, S. Kumar, J. J. Corso, and V. N. Krovi. Estimating dynamics on-the-fly using monocular video. In Proceedings of 4th Annual Dynamic Systems and Control Conference, 2011. [ bib | .pdf ]
[9]	A. Y. C. Chen and J. J. Corso. Propagating multi-class pixel labels throughout video frames. In Proceedings of Western New York Image Processing Workshop, 2010. [ bib | .pdf ]
[10]	W. Ceusters, J. J. Corso, Y. Fu, M. Petropoulos, and V. Krovi. Introducing ontological realism for semi-supervised detection and annotation of operationally significant activity in surveillance videos. In Proceedings of the 5th International Conference on Semantic Technologies for Intelligence, Defense and Security (STIDS), 2010. [ bib | .pdf ]