The potential of Recall and Precision as interface design parameters for information retrieval systems situated in everyday environments Ayman Moghnieh Josep Blat Universitat Pompeu Fabra Universitat Pompeu Fabra C/Tanger 122-140, E-08018 C/Tanger 122-140, E-08018 Barcelona, Spain Barcelona, Spain ayman.moghnie@upf.edu josep.blat@upf.edu ABSTRACT entrances, and public squares, represent new border zones that In this paper, we investigate ways for a tighter integration of IR maintain connectivity and mutual presence between the real and and HCI in new urban contexts, as HCI expands its reach outside the digital worlds, and actively sustain flows of useful or relevant the workplace towards environments where efficiency and information towards nearby people who in-turn search, discover, performance no longer constitute the backbone of interaction and interact with the displayed information. requirements. In particular, we propose to use Recall and The human interaction with information via situated interfaces Precision as design parameters to describe the information settings creates new challenges for conventional information retrieval (IR) and performance of situated interfaces acting as retrieval systems systems: first, the relationship between people and digital in these environments. To explore this notion, we follow an information spaces becomes more explicit and the technology that inductive design research process by which different prototypes supports it more ubiquitous. Second, the human interaction with are designed, developed, and evaluated. Our experience shows information spaces adopts a more direct approach supported by that Recall and Precision, as design parameters, help to reflect the the coming of age of new interaction paradigms (e.g. touch, information requirements onto the interface design, and contribute gesture, speech) that emulate the manipulation of objects. Third, to adapting IR to the contemporary challenges it faces, although the information space hosted by a situated interface tends to be more work is needed to consolidate its role vis-à-vis the growing specialized in subjects and themes befitting the environment ubiquity of computer technologies. where the interface is situated, and the goals and interests of the people present in it. Fourth, the interaction properties may vary Categories and Subject Descriptors considerably in terms of interaction duration and the amount of H.5.2 User Interfaces. user attention delegated to the situated interface [1]. These challenges, among others [2], justify the search for a tighter General Terms coupling of interface and interaction design, and IR systems, by Design, Experimentation, Human Factors, Theory. which IR as a supporting technology for interacting with information contributes to making the interface design more transparent and the human-information interaction more fluid and Keywords direct. Therefore, we reason that the performance of situated Information Retrieval, Human-Information Interaction, Situated interfaces as IR systems ought to be attuned according to the Interfaces, Interface and Interaction Design nature of each specific interaction scenario, given that a maximization of IR performance, may not be adequate for 1. INTRODUCTION answering the interaction design requirements in all kinds of user As computer technologies become more ubiquitous and versatile, experiences with situated interfaces [5, 10]. Consequently, IR and get further integrated in human environments, several genres performance tilts towards becoming a design issue that determines of situated information interfaces (e.g. interactive peripheral some of the characteristics of situated interfaces that mediate this displays, ambient displays, and interactive surfaces) are starting to interaction. assume a mediating role between people and digital information Currently, two metrics (Recall and Precision) are used to assess spaces in different environments. From an HCI perspective, these the performance of IR systems in response to user queries [3]. situated interfaces, primarily found in public and semi-public Recall is the fraction of retrieved information elements from the environments such as malls, public transportation, building entire existing set of elements that are relevant to the user query in the information space. Precision is the fraction of retrieved Copyright © 2011 for the individual papers by the papers’ elements found relevant with respect to the user query, over the authors. Copying permitted only for private and academic entire set of retrieved elements. However, the query as a middleman between humans and information spaces goes against purposes. This volume is published and copyrighted by the the transparent design of situated interfaces that support a direct editors of euroHCIR2011. interaction with information spaces. In addition, the information spaces hosted by situated interfaces are usually predetermined or pre-queried in accordance with the specific interests of potential users and the characteristics or nature of the environments where Miller’s Law argues that the total number of different objects that the hosting interfaces are situated. Instead of querying, the explicit humans can simultaneously hold in their working memory is momentarily needs of users are answered by direct interaction approximately seven [4]. This affects the manner by which with the visualized information. This superlatively converts the information is perceived when the cardinality of the visualized set relevance of the displayed information to the user interests from a of objects increases. In particular, there is a natural observable performance factor to a design issue. tendency to perceptually cluster or group these objects recursively whenever the perceivable number exceed Miller’s threshold. To Therefore, we argue that the definition of Recall and Precision can observe this phenomenon, eight 10 minutes long think-aloud be loosened or reinterpreted to respectively describe the quantity sessions were organized with eight different university students of retrieved information elements and their visual diversity as that watched InformationCasserole showing magazine ads displayed on the interface, since relevance is no longer a progressively being added to the water container, and commented performance factor from an HCI stance. These two metrics can on how the number of ads shown in the casserole affects the way consequently act as parameters that bind the design and they perceive the set of visualized ads. performance of situated interfaces as retrieval systems to the informational expectations of users, by controlling the amount and We observed that when one object is shown, it tends to engage the diversity of visualized information in order to maximize the subjects in a prolonged and detailed examination. This changes transparency of their designs to support a direct human- when two to seven objects are displayed since subjects become information interaction. more interested in identifying relations among the objects and comparing them. The interest in object relations abates with a In order to explore this idea further, we followed a line of higher object number, and instead the relations among clusters or inductive design research by conceptualizing, designing, and collections of objects start to proportionally grab attention. When evaluating experimental prototypes. We first introduce two sets of the number of visualized objects crosses a certain threshold, prototypes devised to understand how users perceive the quantity which we estimate at Miller’s number squared, the casserole and visible diversity of information objects. We then define becomes perceptually saturated and the subjects begin to treat the parameterization scales for Recall and Precision based on these set of ads as a space, reasoning about different regions in it. In experiments. In order to develop a thoughtful understanding of conclusion, we find that the quantity of visualized objects (R) is how Recall and Precision, which we will consecutively refer to as perceived in four different density thresholds, and to each we R and P, can act as design parameters for situated interfaces, we accord a parameter value: R=0 for visualizing no or a single use them in the analysis, design, and evaluation of five different object; R=1 for a single collection of seven or less objects; R=2 situated interfaces. Next, we investigate how these two parameters for seven or less collections; and R=3 for single information space can be dynamically controlled by users through the design of two or more than seven squared objects. This is reflected in figure 2. interactive interfaces for searching and browsing news articles. We conclude by assessing our experience and discuss the viability and implications of our approach. 2. RECALL AND PRECISION FROM A PERCEPTUAL STANCE Figure 2. R as a design parameter In order to study the effects that the visible diversity of information objects (P) has on the manner by which people perceive information, eight paper-based prototypes similar to the InformationCasserole were conceived. Each prototype shows a combination of twelve to fifteen information objects from different genres (e.g. classified ads, news headlines, blog posts, news pictures, movie posters, youtube videos, secondhand goods, and city events). The object genre was emphasized and differentiated by aesthetic design. The visible object diversity encourages people to search for relations among visualized objects [6]. Therefore, the combinations, ranging from one to eight genres, were designed to encourage subjects to search for Figure 1. An instance of the InformationCasserole prototypes patterns and relations among the objects. Six twenty minutes think-aloud sessions were organized with subjects whom were InformationCasserole is a series of video prototypes (figure 1) asked to search for and identify different genres of objects in each designed to study the effect that the number of visualized of the eight combinations presented in random order. elements (R) has on the way humans perceive the information revealed on the interface. They show classified ads from As expected, the subjects perceptually clustered the objects magazines and newspaper floating on different levels in a glass primarily in accordance to their genre. However, they sometimes container filled with slowly moving water. Therefore, their tended to search for inner-divisions in objects of the same genre settings emulate a transparent interface design and foster a direct (e.g. clustering movies according to their cinematic kind or news relationship between the human and digital information spaces. articles in familiar news categories), or to merge related genres as a single genre (e.g. news articles and blog posts, or movie posters and news pictures). In total, the subjects perceived the diversity of objects (P) in four different levels, and to each level we accord a · The amount of available user attention (e.g. MetroWindow corresponding parameter value inversely proportional to the disposes of little attention in contrast with DigiJuke). number of visible object genres: the first level is a single-genre · The duration of human interaction with information (e.g. diversity (P=3); the second level is a diversity of two to three NewsWall remains in contact for prolonged durations, while the genres (P=2); the third level refers to diversity of three to four interaction with YouServe is more momentarily). genres (P=1); the fourth level describes a diversity of five to seven genres of objects (P=0). Figure 3 shows the number of visible · The convergence or divergence of the information seeking genres of objects in each of the eight combinations as seen by the tasks (e.g. YouServe supports finding a specific library service, subjects, and the P value of each of the four identified diversity while Arts&Movies is designed to acquaint people with many levels. movies). Table 1. Values of R and P parameters for each interface Situated interface Recall Precision Arts&Movies 2 1 DigiJuke 3 3 YouServe 1 2 NewsWall 1 1 MetroWindow 0 3 The results of this R and P qualification are summarized in table Figure 3. P as a design parameter 1. They show how R and P can characterize, from a perceptual stance, the role of a situated interface as an information retrieval 3. SITUATED INTERFACES AS IR engine, and parameterize the design of its information settings SYSTEMS accordingly. For example, when the user objectives are to search In order to assess how R and P act as design parameters for the for specific objects (e.g. YouServe), R is minimized, while P can information settings of situated interfaces, the following five be maximized when the search converges on specific genres (e.g. interfaces that act as retrieval systems in real-world environments MetroWindow) or minimized when it diverges to cover many were analyzed, and for each a corresponding design was genres (e.g. NewsWall). A maximized R signals that the developed and evaluated in settings that resemble or emulate its interaction tackles a large number of objects. In this case, when P deployment environment. is maximized (e.g. DigiJuke), it determines that this large number is a single collection of similar objects, or, when it is minimized The Arts&Movies is a situated interface intended for movie (e.g. Arts&Movies), it signals that this large number of objects is a theatre lobbies to support the search and discovery of new visually diversified information space. interesting movies through an animated visualization that draws attention to relationships between movies and concepts. The The designers also developed the interfaces information DigiJuke is installed inside a bar to allow people to browse and architecture and aesthetic design, but these activities lies outside select music songs on the touch-screen, and play their video clips the scope of this paper. The final designs are shown in figure 4. accompanied by related images on the projection display. The YouServe prototype is collocated in a university library lobby to assist people in familiarizing themselves with the available library services, and finding a service relevant to specific needs. The NewsWall is a large display situated in the news production room of a broadcasting corporation. The prototype subtly visualizes the constantly evolving news information space on the web. The MetroWindow is designed for metro wagons and broadcasts summarized local news about cultural and civic events in the city of Barcelona. In related works [7, 8] we have argued how R and P, as design parameters, can be quantified during requirement analysis and used alongside other aspects to conceptualize the design of information interfaces. For each situated interface, a couple of Figure 4. The situated interfaces final designs designers analyzed the characteristics of three entities being: the deployment environment, the humans present in it, and the adequate information space, which was defined based on an 4. USER CONTROL OVER R AND P understanding of the needs and goals of the humans alongside the Based on the discerned ability of R and P to describe the nature of the environment and the information and activity flows information settings of situated interfaces and consequently their that it hosts. Based on this analysis, the designers qualified the performance as information retrieval systems, we explored the values of R and P for each situated interface, and consequently possibility of allowing users to control them dynamically in described its information settings, being the quantity of classic search and retrieval scenarios. Therefore, we designed two information to visualize and its visible diversity. This experimental prototypes (figure 5) for querying a large qualification of R and P was defined in accordance with several information space of news articles, by which users can set and non-disjoint or co-dependent situational aspects of human- control the values of both R and P. The prototypes were evaluated information interaction such as: to assess the feasibility of this approach and its utility. The NewSearch prototype collocates two slide-bars adjacently to re-querying, a more profound study should be conducted for the query textbox for setting R and P explicitly, and returns an further analysis. Such endeavor will constitute the essence of our equivalent clustered visualization of news articles. Users control future work. the number of clusters (discerned by color) by P and their average cardinality by R. The 3DQuery prototype uses a tag-map as a new 6. DISCUSSION concept for defining user queries, and shows a corresponding map The approach that we presented in this paper demonstrates that a of news articles. The tag-map is a rectangular box where users can tighter integration of HCI and IR is possible, by exploring the place different tags of distinct sizes. The position of each tag potential of R and P as design parameters for the information determines that of the corresponding cluster of news articles, and settings of situated interfaces. The use of these two performance the tag size the cluster cardinality. metrics as design parameters may be seen as controversial, however, it is justified given that efficiency and information relevance no longer constitute the backbone of user expectations in all cases of human-information interaction. Instead, new aspects of human-information interaction (e.g. emotional, cognitive, experiential, situational, and cultural) are affecting the manner by which we conceptualize information systems. Our approach does not comprehensively address all these aspects, and therefore can be complemented by introducing new parameters to reflect with a higher affinity the aspects of human-information interaction onto the system design. Figure 5. NewSearch (left) and 3DQuery (right) prototypes Each prototype was evaluated by a different group of ten subjects 7. ACKNOWLEDGEMENTS in the lab. The subjects were asked to browse and read the The authors would like to thank Oriol Galimany and other collection of news articles for fifteen minutes, and then answer a members of the Interactive Technology Group at Universitat set of open-ended questions concerning their utility and usability. Pompeu Fabra for their support. The user evaluations of both prototypes showed that their learning curve is not negligible. Subjects were not naturally inclined to use 8. REFERENCES the slide-bars of NewSearch to control the information settings. [1] Vogel, D. and Balakrishnan, R. 2004. Interactive public An explanation for this may well be that they are accustomed to a ambient displays: transitioning from implicit to explicit, given query paradigm and the difficulty lies in making the public to personal, interaction with multiple users. paradigm change [9]. However, this issue requires further Proceedings of UIST '04, pp. 137- 146. investigations. Subjects found it easy to use the tag-map paradigm in general, but it was deemed too complicated for simple queries [2] NJ Belkin. Some (what) grand challenges for information and more useful for prolonged search and exploration since it retrieval. ACM SIGIR Forum, 2008 allows users to dynamically adjust queries and therefore [3] R.A. Baeza-Yates and B. Ribeiro-Neto. 1999. Modern eliminates or reduces the need for re-querying. Information Retrieval. Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA. The experience and knowledge gathered with the design and evaluation of these two prototypes would be used for developing [4] Miller G. The Magical Number Seven, Plus or Minus Two: future prototypes that intent to delegate more intuitively a Some Limits on Our Capacity for Processing Information. dynamic control over the information settings of information The Psychological Review, 1956. retrieval interfaces to their users. [5] L. Hallnäs and J. Redström. 2001. Slow Technology, Designing for Reflection. Personal Ubiquitous Comput. 5, 3 5. CONCLUSIONS (January 2001), 201-212. During the course of this paper we have explored ways to tightly [6] Koffa, K. (1935): Principles of Gestalt Psychology. London, integrate IR and HCI in a variety of human-information Routledge & Kegan Paul Ltd. interaction scenarios where interfaces act as information retrieval [7] Moghnieh, A., & Blat, J. (2009). A basic framework for systems. In particular, we studied how R and P as design integrating social and collaborative applications into learning parameters can describe the information settings of these environments. Proceedings of m-ICTE’09 Vol. 2 (pp. 1057- interfaces. Both aspects were parameterized on a 0-3 scale on the 1061), 2009. basis of conducted experiments to analyze different possible information settings. Consequently, five situated interfaces were [8] Moghnieh, A., Sayago, S., Arroyo, E., Sopi, G., and Blat, J. designed and analyzed to discern how R and P are qualified Parameterized User-Centered Design for Interacting with during requirement analysis, and how together they describe the Multimedia Repositories. In Proc. MMEDIA '09, IEEE. information settings of situated interfaces, and therefore help [9] B. Buxton. 2007. Sketching User Experiences: Getting the reflect the interaction requirements onto the interface design. Design Right and the Right Design. Morgan Kaufmann Finally, we investigated the feasibility and utility of delegating Publishers Inc. CA, USA. control of R and P dynamically to users during classic search and [10] S. Bødker. 2006. When second wave HCI meets third wave retrieval scenarios, and concluded that while this approach is challenges. In Proceedings of NordiCHI '06. clearly advantageous for exploration tasks and tasks that require