This research explores advanced execution environments for object-oriented languages. Traditional debugging is procedural in that a programmer must proceed step-by-step and object-by-object in order to uncover the cause of an error. In contrast, we explore a declarative approach to debugging consisting of a flexible set of queries over a program's execution history as well as over individual runtime states. This runtime information is depicted in a visual manner during program execution in order to aid the debugging process. A practical system called JIVE (Java Interactive Visualization Environment) has been developed embodying these principles and is available as a plug-in for Eclipse.
We depict the current state of execution UML-like object diagrams, and the history of execution by UML-like sequence diagrams. Although these diagrams were developed for representing designs, we show that they can be used at run-time as well, thereby providing a uniform notation for comparising design and execution. Our methodology makes use of these diagrams as a means of formulating queries and reporting results in a visual manner. Declarative queries and object/sequence diagrams work in a synergistic manner to achieve scalable visualizations: queries help the user to focus on specific regions of the diagrams, while the diagrams provide a framework for reporting the answers to queries. JIVE also supports revisiting past runtime states, either through reverse stepping of the program or through queries that report information from past states.
There are several research topics that emerge from this project: (i) temporal data models and query languages for expressing debugging queries; (ii) compact representations of diagrams to facilitate scalability; (iii) efficient algorithms for diagram layout; (iv) debugging with multiple execution runs; (v) debugging multi-threaded and concurrent programs; and (vi) diagrammatic approach to program verification.
D. Lessa, J. Chomicki, B. Jayaraman, A Temporal Model for Program
Debugging, In Proc. Symposium on Database Programming Languages, August 2011.
D. Lessa, J.K. Czyz, B. Jayaraman, JIVE: A Pedagogic Tool for Visualizing the Execution of Java Programs, CSE TR 2010-13, December 2010.
D. Lessa, B. Jayaraman J.K. Czyz, Query-based Debugging and Scalable Visualizations, CSE TR 2010-09, September 2010.
D. Lessa, B. Jayaraman, A Temporal Model for Debugging and Visualization, CSE TR 2010-10, September 2010.
D. Lessa, B. Jayaraman, and J.K. Czyz, Query-based Debugging and Scalable Visualizations, CSE TR 2010-, September 2010.
J.K. Czyz and B. Jayaraman, Declarative and Visual Execution Environments in Eclipse, Proceedings of OOPSLA Eclipse Technology Exchange, 2007.
H.Z. Girgis and B. Jayaraman, JavaDD: Logic Based Debugging of Java Programs, Proc. ICLP Workshop of Logic Programming Environments, 2006.
H. Girgis, B. Jayaraman, P.V. Gestwicki: Visualizing Errors in Object Oriented Programs. Proc. ACM Conference on Object Oriented Programming, Languages, Systems, and Applications, 2005. Expanded version available as CSE TR 2004-19.
P.V. Gestwicki: Interactive Visualization of Object Oriented Programs. Ph.D. dissertation, CSE Department, June 2005. Available as: TR 2005-15.
P.V. Gestwicki and B. Jayaraman: Methodology and architecture of JIVE. Proc ACM Symposium on Software Visualization, pp. 95-104, 2005.
P. Gestwicki and B. Jayaraman, Methodology and Architecture of JIVE, Proc. ACM Conference on Object Oriented Programming, Languages, Systems, and Applications, Vancouver, BC, 2004. Expanded version available as CSE TR 2004-20.
P. Gestwicki and B. Jayaraman, Interactive Visualization of Java Programs, Proc. IEEE Symposium on Human Centric Computing, Languages and Environments, Washington DC, Springer-Verlap, 2002.
P. Gestwicki, B. Jayaraman, A. Garg, From Class Diagrams to Object Diagrams: A Systematic Approach, CSE 2004-21, 2004.
B. Jayaraman, P. Gestwicki, and A. Garg, Interactive Program Visualization and Graph Drawing, Proc. International Conference on Discrete Mathematics and Applications, December 2004. Also available as: CSE 2004-21.
H. Girgis et al, Visual Queries in Interactive Execution of Java Programs, CSE TR 2004-19, 2004.
Our recent research has been concerned with developing 'smart indoor environments'. This is a physical environment that is embedded with sensors, actuators, computers, and communication devices that are interfaced gracefully with humans via natural interfaces such as speech, gestures, etc. Our current focus has been on identification and tracking of occupants using biometric capture devices such as cameras, microphones, etc., and also answering spatio-temporal queries about the whereabouts of occupants. Such systems can be used in homes for elderly or disabled, department stores, and eventually in larger arenas such as airports.
We have developed a state-transition framework in which events abstract different types of biometric recognition (e.g., face, voice, or height recognition), and transitions abstract different types of reasoning (e.g., spatial and temporal reasoning). The state-transition system is fundamentally probabilistic because biometric recognition is inexact in nature. We have shown that the use of declarative knowledge and reasoning in conjunction with recognition greatly enhances the overall performance of the smart environment. We have developed an approach to characterizing the performance of a smart environment in terms of information-theoretic concepts of precision and recall.
As part of our current and future work, we are examining smart indoor environments in which recognition and reasoning are augmented with user interaction via speech and gestures, and actions that effect changes in the environment. We are also interested in smart environments that improve their performance by learning from past behaviors. This is joint work with Venu Govindaraju and Vivek Menon (Amrita U).
Menon, V., Jayaraman, B., and Govindaraju, V. Three R's of Cyber Physical
Spaces, IEEE Computer, September 2011
Menon, V., Jayaraman, B., and Govindaraju, V. Multimodal Identification and Tracking in Smart Environments. Journal of Personal and Ubiquitous Computing, Special Issue. March 2010, pp 1-10, Springer Journal.
Menon, V., Jayaraman, B., and Govindaraju, V. Integrating Recognition and Reasoning in Smart Environments. Proceedings of the 2008 Intelligent Environments (IE 08) Conference, available via IEEE Xplore Digital Library.
Menon, V., Jayaraman, B. and Govindaraju, V. Biometrics Driven Smart Environments: Abstract Framework and Evaluation. Proceedings of Ubiquitous Intelligence and Computing (UIC 08) Conference, Springer-Verlag (2008), pp. 75-89.
The goal of our research is to model complex systems using a compositional approach. We are exploring a declarative variant of the traditional paradigm called 'constrained objects'. This paradigm is being developed at Buffalo, and is useful in high-level modeling of engineering and biological systems, as well as in other domains. A constrained object is one whose state is governed by laws of behavior, or constraints. When several such objects are combined together to form a complex system, the emergent behavior is determined by a process of constraint satisfaction. Constrained objects can model static as well as dynamic systems, and can also be combined with domain-specific visualizations that enable the end-user to interact with the underlying model in a natural medium. Research topics include: modeling language design, efficient execution, analysis of model inconsistency, domain-specific visualizations, modeling dynamic and hybrid systems, integrating powerful solvers, etc.
S. Diwakar and B. Jayaraman, Constrained Objects for Neuronal Simulation
and Modeling, International Symposium on Recent Trends in Neurosciences,
P. Tambay, Constrained Objects for Modeling Complex Systems, Ph.D. dissertation, University at Buffalo, 2004.
P. Tambay and B. Jayaraman, The Cob Programmer's Manual, CSE Dept TR 2003-1, University at Buffalo, 2003.
Y. Zhang, Case Studies in Constrained Objects, M.S. Thesis, University at Buffalo, 2004.
R.J. Raux and B. Jayaraman, Modeling Dynamic Systems with Constrained Objects, CSE Dept TR 2004-05, University at Buffalo, 2004.
B. Jayaraman and P. Tambay, Implementation Techniques for Constrained Objects. CSE Dept TR 2004-04, University at Buffalo, 2004.
B. Jayaraman and P. Tambay, Compositional Semantics for Diagrams using Constrained Objects, Proc. Second International Conference on Theory and Application of Diagrams, 2002.
B. Jayaraman and P. Tambay, Modeling Engineering Structures with Constrained Objects Proc. International Symposium on Practical Applications of Declarative Languages, Portland, OR, pp. 28-46, Springer-Verlag, 2002.
B. Jayaraman and P. Tambay, Constrained Objects for Modeling Complex Structures, Proc. ACM Conference on Object-Oriented Programming, Systems, and Applications, Minneapolis, 2000.
B. Jayaraman, T. McKernan, S. Raghavan, and S. Shanker, CobWeb: Constraint-Based XML for the Web, Technical Report, CSE Department, U at Buffalo Environments, 2001.
We are also developing a programming paradigm based upon both constraints and preferences. While constraints specify the mandatory requirements of a problem, preferences specify optional or desired requirements and serve as a basis for determining optimal solutions to the constraints. There are numerous applications of this approach, including combinatorial optimization, database querying, and natural language parsing. Essentially, in this framework we can: (a) designate certain predicates as optimization predicates, and state the criteria for determining their optimal solutions; and (b) designate which optimization predicates are to be relaxed and state the criteria for relaxation. These criteria are understood using the concept of preference and hence we call this framework preference logic programming. In joint work with S. Mantha and K. Govindarajan, we have developed the foundations of such a language, and we are presently interested in applications and implementation techniques.
K. Govindarajan, B. Jayaraman and S. Mantha,
Logic Programming with Constraints and Preferences, 1998.
Expanded version of a paper which appeared in: Proc. 23rd ACM Symposium on Principles of Programming Languages, pp. 91-105, St. Petersburg, FL.
K. Govindarajan, B. Jayaraman, and S. Mantha, Preference Queries in Deductive Databases, New Generation Computing, 19(1), 2000. B. Jayaraman, K. Govindarajan, and S. Mantha, Preference Logic Grammars, Computer Languages, 24:179--196, 1998.
K. Govindarajan, B. Jayaraman and S. Mantha, Preference Logic Programming, Proc. International Conference on Logic Programming, pp. 731-745, MIT Press, 1995.