Situvis: Visualising Multivariate Context Information to Evaluate Situation Specifications Adrian K. Clear, Ross Shannon, Thomas Holland, Simon Dobson, Aaron Quigley and Paddy Nixon Systems Research Group School of Computer Science & Informatics UCD Dublin Ireland adrian.clear@ucd.ie ABSTRACT a coffee break”, or being “in a research meeting”, without One of the many challenges faced when evaluating context- requiring the user to understand any of the dozens of distinct aware ubiquitous systems is to gain some understanding of sensor readings which may have gone into making up these the constant influx of context data into the system. Else- situations. Situations are thus a natural view of a context- where, context has been distilled into more natural abstrac- aware system, whereas the individual pieces of context are tions called situations with the aim of making these sys- each “a measurable component of a given situation” [10]. tems more understandable and intuitive to develop appli- cations for, though codifying and manipulating these situ- As the context information available to a context-aware sys- ations still presents problems. We introduce Situvis, a tool tem at any moment is so extensive, dynamic and highly di- we have developed based on the Parallel Coordinates Visual- mensional, it is a significant challenge for a system observer isation technique, which assists users by visually represent- to ascribe significance to changes in the data or identify emer- ing the conditions that need to be present for a situation to gent trends, much less capture the transient situations that be triggered in terms of the real-world context that is being are occurring amid the churn of the data. recorded in their system, and allows the user to visually in- spect these properties, evaluate their correctness, and change The visualisation of large and complex multivariate data them as required. We describe the use of our tool with a sets, such as those that context-aware system developers work small user study. with, is becoming increasingly crucial in helping those de- velopers to organise and distill data into usable informa- Author Keywords tion [2]. Interactive visualisation tools help the viewer per- form visual data analysis tasks: exploring patterns and high- Context, context-aware systems, situations, situation program- lighting and defining filters over interesting data. ming, visualisation, visual data mining Here, we present Situvis, a scalable visualisation tool for il- ACM Classification Keywords lustrating and evaluating the makeup of situations in a context- D.1.7 Visual programming, aware ubiquitous system. Our tool is based on well-founded H.1.2 Human information processing situation specification semantics. By incorporating real sit- uation traces and annotations, Situvis assists system devel- INTRODUCTION opers in constructing and evaluating sound and complete sit- In context-aware systems, context data is derived from mul- uation specifications, affording them a better understanding tiple heterogeneous sensors. These sensors may be networked of the situation space, and the reliability of modelling with physical instruments in the environment (measuring factors situations based on real, recorded sensor data. It is a frame- like temperature, noise volume or humidity) or software sen- work that allows developers to understand, at a high level, sors retrieving information from the web or various data feeds. how their system will behave given certain inputs. These context data are voluminous, highly multivariate, and constantly being updated as new readings are recorded. The following section provides a formal description of sit- uation specifications and a review of some challenges faced Situations have been proposed as a higher-level abstraction when working with context and situations. We then describe of context data [8], freeing the user from having to deal with the details of the Situvis tool, including a demonstration of raw context and allowing more expressive adaptations. Situ- its utility, followed by an informal evaluation and discussion ations are more natural for users to work with, as they define of its properties. commonly-experienced occurrences such as a user “taking CONTEXTS AND SITUATIONS Copyright is held by the authors. UbiComp ’08 Workshop W2 – Ubiquitous Systems Evaluation (USE ’08) Situation specifications September 21st, 2008 Based on our experience with modelling context for adap- This paper is not an official publication of UbiComp ’08. tive systems [4, 10], and from the extensive literature on the subject [5, 6, 8], we can make some observations: the in- Related to these properties is perceptual distinguishability. coming sources of context are viewed as a finite number of If a system state exists that involves multiple situations, the variables: either nominal or categorical values, e.g., activity observer should be able to distinguish between those situa- levels {idle, active, highly active . . . }; or quantitative ordi- tions. It may be the case that two situations’ specifications nal values which may be defined over some known interval, are satisfied by a given occurrence. In the case of compatible e.g., noise level in decibels {0, 140}. situations, this could be due to one situation’s specification subsuming another. If the specifications are incompatible, Location information will typically arrive as individual val- however, we must have a means to re-evaluate them. Situ- ues for an object’s x, y and z values, and may be recorded ations are generally specific to behaviours, and, as a result, by numerous disparate positioning systems, but is modeled their compatibility requirements are determined by the com- as a higher-level abstraction to make it easier to reason with. patibility requirements of behaviours. We have previously completed research that allows compo- nent x, y and z coordinates to be composed into a symbolic The adaptive systems that we are concerned with are user- representation, given some domain information [15], and so centric, and so user feedback is an important aspect of their can work with locations as readable as “Simon’s office” or evaluation. Situation annotation is a particularly useful mode “Coffee Area”. Our visualisation tool can accept either raw of feedback for us, because it allows us to contrast situation sensor data or these higher-order categorised data. specifications with actual traces of these situations. It also exposes the subjective nature of situations. However, to de- Situations are high-level abstractions that serve as a suitable velop sound and complete situation specifications, it is nec- model with which to develop context-aware systems. In or- essary to capture two facets of reality: those situations that der for a system to be able to recognise situations, they must our specification must successfully characterise; and those first be specified. The semantics of situation specification situations that it should not. Therefore, the annotated situa- can be seen in the work of Henricksen [8] and Loke [11]. tions are an important guideline, but the traces of undesired Based on this work, we make some assumptions about situ- situations are also important to avoid false positives. ation specification so that situations specified using declar- ative languages such as these could simply be “plugged-in” Situations can range from the very simple to the very com- to our tool. plex, depending on the number of contextual components they are defined over. The more complex a situation be- Situation specifications are boolean expressions (sometimes comes, the more difficult it is to pick out similarities or dif- called assertions in computer programming)—they are ei- ferences between multiple situations in aggregate, without ther true or false, denoting occurrence and non-occurrence, the support of a visual analytics tool. This ability is impor- respectively. Assertions may be composed using the logical tant because the similarity of one situation to another de- operators AND (∧), OR (∨), and NOT (¬), resulting in richer termines the possibility of them occurring together or in se- expressions. Domain-specific operators can be defined to quence within a small period of time. complement these operators. For example, for location we could define a “distance” operator. The distance operator may take two arguments and return a numerical value of the Visualisation of context data distance between them. Domain-specific operators are also There exist myriad visualisation techniques, from time-series required for situation specification. For example, for many to multi-dimensional scatter plot methods, which can be ad- context dimensions an essential operator is one that takes a apted to the exploration of multidimensional context data. value and a range and returns true if that value exists within Our focus here is not only on the exploration of such con- the range. text data, but also the scope of the higher order situations, their specification, and data cases which fall outside the set We can thus define a situation specification as a concatena- boundaries. The Table Lens, a focus+context visualisation, tion of one or more assertions about contexts, which leads supports the interactive exploration of many data values in a us to the following formal definition: semi-familiar spreadsheet format [14]. In practice, due to the distortion techniques employed, users can see 100 times as A situation specification consists of one or more assertions many data items within the same screen space as compared about context that are conjoined using the logical operators with a standard spreadsheet layout. Rather than showing the AND (∧), OR (∨), and NOT (¬). Assertions may comprise detailed numeric values in each cell, a single row of pixels, further domain-specific expressions on context, given that relating to the value in the cell, is shown instead. The Ta- the required semantics are available. ble Lens affords users the ability to easily study quantitative data sets, but categorical values are not well supported. An alternative to a single visualisation are co-ordinated, linked Properties of context-awareness visualisation techniques employing brushing and querying. Situation specifications are essential in achieving two very Three linked views of multi-dimensional data—using a Prin- important properties of context-aware systems: soundness cipal Components Analysis (PCA) view, interactive brush- and completeness [11]. A system is sound if it does not give ing, and dimensional querying with parallel bargrams—are false positives when determining a situation; it is complete employed in the Antenna Design Gallery [12]. Here the ac- if it contains specifications for all situations to be detected. tions or queries in any one window or view are reflected in all. A user selecting a range within a given dimension re- as the size of the data set grows larger, techniques have been duces the data cases in the PCA view, and in the element val- designed to cluster or elide sub-sets of the data to allow the ues highlighted in the other bargrams. This encourages ex- dominant patterns to be seen [1]. Direct interaction to filter ploration of a large, multivariate data set, as different facets and highlight sections of the data encourages experimenta- of the data can be seen in each view. tion to discover additional information. PARALLEL COORDINATES Hierarchical clustering [7] uses colour to visually distinguish Parallel Coordinate Visualisations (PCVs) are a standard two- cases that share a certain range of values into a number of dimensional technique ideally suited to large, multivariate sets, increasing the readability of the diagram. We use a data sets [9]. The technique excels at visually clustering similar technique to group case lines that are assigned to a cases that share similar attribute values across a number of certain situation, colour-coding these as a group. Different independent discrete or continuous dimensions, as they can situations can be colour-coded so that the interplay of the be visually identified through the distribution of case lines context traces that correspond to them can be easily seen. within the visualisation [3]. The user can see the full range of the data’s many dimensions, and the relative frequencies EVALUATING SITUATIONS WITH SITUVIS at which values on each axis are recorded. These features are visible in Figure 1, which shows context data from our Description & case-study user study, which we will describe in the next section. Situation-awareness is commonly applied to adaptive sys- tems as a means to introduce useful cues for automatic be- PCVs give users a global view of trends in the data while haviour adaptation. System developers are tasked with codi- allowing direct interaction to filter the data set as desired. A fying situations that their system should respond to by tying set of parallel horizontal axes are drawn, which correspond together loose configurations of sensor readings. Because of to attributes of the readings in the system. In our case, the the constant fluctuation in the values of these sensor readings readings are records of context data at a certain time, with (due to minute changes in the property being measured, or each axis representing a sensor in the system. Then, a set of due to the accuracy of the sensor), situation definitions are n-dimensional tuples are drawn as a set of purple polylines frequently composed of a set of ranged intervals that give the which intersect each axis at a certain point, corresponding to developer some more latitude to cover more of the sample the value recorded for that attribute. Discrete and quantita- space than if they had to define a situation for every possi- tive axes can be presented in the same view. ble combination of sensor readings. When these ranges are all logically conjoined and the value from each sensor falls within range concurrently, the behaviour attached to this sit- uation is invoked. Situvis is built using Processing [13], a Java-based visualisa- tion framework which supports rapid prototyping of visual- isation techniques.1 Each context dimension is represented in Situvis as a vertical axis. Each axis is divided equally based on the number of values that could be recorded for this dimension. For example, the axis for location contains six points representing the symbolic locations that we chose to include in our analysis. A situation trace is represented as a polyline—a line drawn starting at the leftmost axis and continuing rightwards to the next adjacent and so on, inter- secting each axis at the point that represents the value that the context has in that situation trace. For example if, in a given situation, a user’s computer activity level is “idle”, and their location is “canteen”, and these two axes are adjacent, then a line will be drawn between those two points. Each Figure 1. Our Parallel Coordinates Visualisation. This is a view of 96 overlaid context traces with 8 data dimensions gathered over 3 days. situation trace is plotted on the axes and the result is a view Strong correlations can be seen between the three days recorded: the of all of the situations, significant and insignificant, that oc- subject spent the majority of all three days at their desk (the first value curred in the system over a period of time. on the “Location” axis), with some deviations due to coffee breaks or visits to their supervisor’s office at irregular times. In order to carry out our case-study, we required real con- text data with which we could characterise situations. We As all the polylines are being drawn within the same area, the chose to gather context data and situation annotations man- technique scales well to large data sets with arbitrary num- ually over a three day period. While the capabilities exist bers of attributes, presenting a compact view of the entire to collect these context data automatically, we chose to col- data set. Axes can be easily appended or removed from the lect the data through manual journaling, so that we did not visualisation as required by the dimensions of the data. 1 You can read more about Situvis and interact with a demo of the As Parallel Coordinates have a tendency to become crowded software at http://situvis.com. Figure 2. A view of the Situvis tool showing 3 days of a user’s context, with traces annotated as being in a “meeting” situation highlighted (left). These situations occurred at many different times throughout the day in two different locations, with a range of values for the other contexts. The user can interactively expand or contract the situation definition along any of the axes. In this case, they have chosen to modify the situation specification to allow for more colleagues to be present, the noise level to be greater and the possibility of talking on the phone (right). need to factor in issues with the aggregation, uncertainty or text dimensions characterise them. We can see that “Time” provenance of the context data. and “Supervisor presence” are not useful due to the multi- ple split lines on their axes. Hence they are ineffective when Our trial subject recorded their context every fifteen minutes defining constraints. The specification is clear from the other (10am–6pm) for three consecutive weekdays. They captured dimensions, however, and could be expressed as: context in the form of time, location, noise-level, number of colleagues present, their supervisor’s presence (true or {1 ≥ No.colleagues ≤ 2} ∧ false), their phone use (either taking a call or not), calendar {Location = (meeting room data (being busy or having no appointments), and computer ∨ supervisor’s office)} ∧ activity. For simplicity, the noise-level was recorded on a {Phone use = none} ∧ 4-point scale of quiet, conversation, chatty, and noisy. Like- {Computer activity ≥ idle} ∧ wise, computer activity level was scaled as idle for an hour {Noise-level = conversation} ∧ or more, idle for less than an hour, active, and highly active. {Calendar = busy} We defined six symbolic locations: meeting room, canteen, sports center, supervisor’s office, subject’s desk, and a lec- None of these values alone can characterise “Meeting”, as ture theatre. Figure 1 shows a view of the Situvis tool with the trace data illustrates. Furthermore, each dimension may all of these traces plotted together in one view. not always be available. Situvis allows one to identify com- binations of dimensions which, when taken together can pro- The subject also annotated what, if any, situation they were vide a good estimation of the situation. For example, “Loca- in at the time of data capture. These annotations are used in tion” taken with “No. of colleagues” is a good indication of Situvis to identify situations that require specification in the “Meeting”, as the interval that they create does not contain system. polylines that characterise different situations. This can also give system developers an insight into which sensors in their system are the most useful, and which types of sensors they Specifying situations with context should invest in to gain the most added benefit in terms of Situation specifications are structured according to the defi- the expressiveness of their system. nition we discussed previously. Situvis enables a developer to select all occurrences of a given annotated situation, and add further cases to this definition using interactive brushing Situation evolution of polylines, or by dragging a range indicator on the left of When existing specifications are overlaid on the trace poly- the axis to expand or contract the range of values covered lines, the developer can see where specifications are too strong by this specification. The user can evaluate existing situa- or weak. Constraints that are too strong will cause the sys- tion specifications overlaid against actual trace data and see tem to sometimes fail in determining when that situation is where they need to be modified. occurring. Constraints that are too weak may be wrongly in- terpreted as an occurrence of the specified situation, when in An example of this process can be seen in Figure 2. The fact a different situation is occurring. When the overlaid sit- trial subject annotated multiple occurrences of a “Meeting” uation encompasses traces that are not relevant, the user can situation. By selecting these traces, it is evident what con- strengthen the constraints. Similarly, the user can weaken constraints to include traces that happen to fall outside the existing specification. We hypothesise that as more trace data is added and anno- tated, the constraints that we have defined for “Meeting” may be too strong. By overlaying our specification on top of the polylines, it will be obvious where constraints need to be strengthened, weakened or even excluded altogether. Situvis enables a developer to drag the boundaries of speci- fications to change the polylines that they cover, essentially changing the constraints of the situation. Situation evaluation Context-aware adaptive systems are very sensitive to incom- patible behaviours. These are behaviours that conflict, either due to device restrictions, such as access to a public display, or due to user experiences, such as activating music playback while a meeting is taking place. Situations are closely tied to Figure 3. A view of two distinct situations. The higher blue range is behaviours—they define envelopes in which behaviour oc- a meeting situation, whereas the lower green range is a seminar that curs. As a result, their specifications are directly responsible occurred after normal work hours. The dissimilarities between these situations are clear from the tool, and the specifications can be further for compatibility requirements. By harnessing this factor, teased apart if required. we can address another key aspect of situation evaluation. Conceptually relating situations to each other from a be- haviour compatibility standpoint is an overwhelming task re-evaluate them accordingly. Current approaches entail the for a developer. We recognise that there are two situation modification of constraints based on data logs or experience. relationships that may lead to incompatibility: Situvis allows developers to visually overlay specifications on traces, and tailor their constraints as a result. Unlike tra- subsumption if a subsumes b, and b occurs, then a will cer- ditional methods, Situvis clearly depicts cases where con- tainly occur. straints are too strong or too weak. overlap if a overlaps b, then a and b may co-occur. In addition, we have identified a scenario that a tool like Situ- vis could address in the future. It is concerned with the no- Our tool allows multiple situation specifications to each be tion of closeness of situations. Two situation specifications coloured distinctly. When two or more situations are shown are close if small changes in context can cause an evolution together, the overlap in their constituent contexts is clear, as from one to the other, a property easily identified from con- well as the scope of their dissimilarities. This view allows text constraints. Close situations may be significant as the the developer to alter constraints where necessary, while the transition step from one to the other is small in terms of prob- overlap and subsumption relationships are refreshed and dis- able context changes. Visualising these relationships in Situ- played on-the-fly. A screenshot of this scenario is seen in vis will allow a developer to identify the following: sugges- Figure 3. tions of areas where the situation associated with a behaviour may be incomplete; and points where the system behaviour DISCUSSION may be unstable. The former is used to increase developer We have shown, using a case-study, the utility of Situvis awareness of situation-behaviour associations that they may in the situation specification and evaluation processes. The have omitted. The latter is useful for highlighting obtrusive traditional approach to situation specification is subject to behaviours associated with close situations—points where a limitations: context constraints are based on static defini- see-saw-like cycle from one to another may occur in a short tions of these concepts in knowledge representation struc- time frame, resulting in an erratic user experience. One can tures (e.g., ontologies); they are derived from a developer’s thus introduce inertia by strengthening constraints, making conceptual understanding of certain situations; and do not the transition step between them larger. contain methodologies for ensuring soundness and complete- ness. To address these limitations, Situvis presents a devel- Some contexts may be relevant only when combined with oper with a reference point for situation specification and other dimensions. Ideally, we would display all of the con- evaluation through the display of actual trace data and situa- text information that is available in the system for a partic- tion annotations. The relevance of context to a specification ular situation annotation. A developer could then eliminate is made clear, and contrasting situation traces can be used as contexts that are not useful based on visual analysis. How- a guide for specification. ever, we have yet to evaluate the feasibility of this approach in large-scale systems. Context-aware systems are dynamic—sensors, users and habits are constantly changing. Hence, we cannot expect situa- Some other context dimensions are also not easily repre- tion specifications to remain static. It must be possible to sented on a line. In particular, Location, with its domain relations like subsumption, is difficult to represent in two 2. P. Ball. Data visualization: Picture this. Nature, dimensions. We are researching techniques to flatten hierar- 418(6893):11–13, 2002. chies for a more intuitive representation. 3. S. Card, J. Mackinlay, and B. Schneiderman. Readings in Information Visualization: Using Vision to Think. CONCLUSIONS AND FUTURE WORK Morgan Kaufmann, 1999. We have presented Situvis, a tool in development which uses 4. A. K. Clear, S. Knox, J. Ye, L. Coyle, S. Dobson, and a Parallel Coordinate Visualisation to illustrate situation traces P. Nixon. Integrating multiple contexts and ontologies and specifications. The tool assists developers in describ- in a pervasive computing framework. In Contexts and ing situations through direct interaction, providing a natu- Ontologies: Theory, Practice and Applications, pages ral interface for a developer of context-aware systems. By 20–25, Riva Del Garda, Italy, August 2006. stacking many instances of context together in one view, it becomes simple to inspect the correlation between situation 5. J. Coutaz and G. Rey. Foundations for a theory of specifications and the actual situations that occur during de- contextors. Computer Aided Design of User Interfaces, ployment. Situvis enables a developer to evaluate the sound- 2002. ness and completeness of situation specifications within the 6. A. K. Dey. Understanding and using context. Personal framework of real data. Ubiquitous Computing, 5(1):4–7, 2001. By visually analysing the overlap of situation specifications 7. Y.-H. Fua, M. O. Ward, and E. A. Rundensteiner. within their system, the developer can identify where multi- Hierarchical parallel coordinates for exploration of ple situations require similar context values to be activated. large datasets. In VIS ’99: Proceedings of the Such overlaps may imply problems in the situation specifica- conference on Visualization ’99, pages 43–50, Los tions, as conflicting behaviours may be triggered by concep- Alamitos, CA, USA, 1999. IEEE Computer Society tually similar situations. Thus, the developer can compare Press. situations against others, and change the situation’s specifi- cations to become stronger or weaker as necessary. 8. K. Henricksen. A Framework for Context-Aware Pervasive Computing Applications. PhD thesis, The We are developing a metric of the closeness of situations School of Information Technology and Electrical for use in evaluating soundness and completeness. Close Engineering, University of Queensland, September situations may frequently occur one after the other, which 2003. may lead to unpredictable system behaviour from a user’s 9. A. Inselberg and B. Dimsdale. Parallel coordinates: a perspective. We hope that the Situvis tool will prove useful tool for visualizing multi-dimensional geometry. In VIS in helping to avoid this oscillation. ’90: Proceedings of the 1st conference on Visualization ’90, pages 361–378, Los Alamitos, CA, USA, 1990. A weakness of the current version of the Situvis tool is that IEEE Computer Society Press. it does not explicitly support probabilities in situation spec- ifications. In many context-aware applications, robust prob- 10. S. Knox, A. K. Clear, R. Shannon, L. Coyle, S. Dobson, abilistic inference is a requirement to handle the naturally A. J. Quigley, and P. Nixon. Towards Scatterbox: a fuzzy data in the system. We are considering the addition of context-aware message forwarding platform. In Fourth an overlay which will allow users to set up a probability dis- International Workshop on Modeling and Reasoning in tribution, though this requires a more in-depth study of the Context in conjunction with Context ‘07, pages 13–24, treatment of uncertainty in situations. Roskilde, Denmark, August 2007. 11. S. W. Loke. Representing and reasoning with situations We are ongoing in our investigation of properties of situa- for context-aware pervasive computing: a logic tions that can be exploited for further evaluation. For exam- programming perspective. The Knowledge Engineering ple, Situvis could also be used by users of the context-aware Review, 19(3):213–233, 2005. system as a gateway to user programming: helping them to unroll the cause of a situation activation, so that they can 12. A. Quigley, D. Leigh, N. Lesh, J. Marks, K. Ryall, and gain insight into why the system began to behave as it did. K. Wittenburg. Semi-automatic antenna design via sampling and visualization. Antennas and Propagation Acknowledgements: This work is partially supported by an Society International Symposium, 2002. IEEE, 2, 2002. EMBARK Scholarship from the Irish Research Council for 13. C. Reas and B. Fry. Processing: a learning environment Science, Engineering and Technology and by Science Foun- for creating interactive web graphics. In SIGGRAPH dation Ireland under grant number 03/CE2/I303-1, “LERO: ’03: ACM SIGGRAPH 2003 Sketches & Applications, the Irish Software Engineering Research Centre.” pages 1–1, New York, NY, USA, 2003. ACM. 14. T. Tenev and R. Rao. Managing multiple focal levels in REFERENCES table lens. Infovis, 00:59, 1997. 1. A. Artero, M. de Oliveira, and H. Levkowitz. Uncovering clusters in crowded parallel coordinates 15. J. Ye, A. K. Clear, and S. Dobson. Towards a formal visualizations. Information Visualization, 2004. semantics for pervasive adaptive systems. Under INFOVIS 2004., pages 81–88, 2004. revision for The Computer Journal, 2008.