=Paper=
{{Paper
|id=None
|storemode=property
|title=The Potential of Recall and Precision as Interface Design Parameters for Information Retrieval Systems Situated in Everyday Environments
|pdfUrl=https://ceur-ws.org/Vol-763/paper8.pdf
|volume=Vol-763
|dblpUrl=https://dblp.org/rec/conf/eurohcir/MoghniehB11
}}
==The Potential of Recall and Precision as Interface Design Parameters for Information Retrieval Systems Situated in Everyday Environments==
https://ceur-ws.org/Vol-763/paper8.pdf
The potential of Recall and Precision as interface design
parameters for information retrieval systems situated in
everyday environments
Ayman Moghnieh Josep Blat
Universitat Pompeu Fabra Universitat Pompeu Fabra
C/Tanger 122-140, E-08018 C/Tanger 122-140, E-08018
Barcelona, Spain Barcelona, Spain
ayman.moghnie@upf.edu josep.blat@upf.edu
ABSTRACT entrances, and public squares, represent new border zones that
In this paper, we investigate ways for a tighter integration of IR maintain connectivity and mutual presence between the real and
and HCI in new urban contexts, as HCI expands its reach outside the digital worlds, and actively sustain flows of useful or relevant
the workplace towards environments where efficiency and information towards nearby people who in-turn search, discover,
performance no longer constitute the backbone of interaction and interact with the displayed information.
requirements. In particular, we propose to use Recall and The human interaction with information via situated interfaces
Precision as design parameters to describe the information settings creates new challenges for conventional information retrieval (IR)
and performance of situated interfaces acting as retrieval systems systems: first, the relationship between people and digital
in these environments. To explore this notion, we follow an information spaces becomes more explicit and the technology that
inductive design research process by which different prototypes supports it more ubiquitous. Second, the human interaction with
are designed, developed, and evaluated. Our experience shows information spaces adopts a more direct approach supported by
that Recall and Precision, as design parameters, help to reflect the the coming of age of new interaction paradigms (e.g. touch,
information requirements onto the interface design, and contribute gesture, speech) that emulate the manipulation of objects. Third,
to adapting IR to the contemporary challenges it faces, although the information space hosted by a situated interface tends to be
more work is needed to consolidate its role vis-à-vis the growing specialized in subjects and themes befitting the environment
ubiquity of computer technologies. where the interface is situated, and the goals and interests of the
people present in it. Fourth, the interaction properties may vary
Categories and Subject Descriptors considerably in terms of interaction duration and the amount of
H.5.2 User Interfaces. user attention delegated to the situated interface [1].
These challenges, among others [2], justify the search for a tighter
General Terms coupling of interface and interaction design, and IR systems, by
Design, Experimentation, Human Factors, Theory. which IR as a supporting technology for interacting with
information contributes to making the interface design more
transparent and the human-information interaction more fluid and
Keywords direct. Therefore, we reason that the performance of situated
Information Retrieval, Human-Information Interaction, Situated interfaces as IR systems ought to be attuned according to the
Interfaces, Interface and Interaction Design nature of each specific interaction scenario, given that a
maximization of IR performance, may not be adequate for
1. INTRODUCTION answering the interaction design requirements in all kinds of user
As computer technologies become more ubiquitous and versatile, experiences with situated interfaces [5, 10]. Consequently, IR
and get further integrated in human environments, several genres performance tilts towards becoming a design issue that determines
of situated information interfaces (e.g. interactive peripheral some of the characteristics of situated interfaces that mediate this
displays, ambient displays, and interactive surfaces) are starting to interaction.
assume a mediating role between people and digital information
Currently, two metrics (Recall and Precision) are used to assess
spaces in different environments. From an HCI perspective, these
the performance of IR systems in response to user queries [3].
situated interfaces, primarily found in public and semi-public Recall is the fraction of retrieved information elements from the
environments such as malls, public transportation, building entire existing set of elements that are relevant to the user query in
the information space. Precision is the fraction of retrieved
Copyright © 2011 for the individual papers by the papers’ elements found relevant with respect to the user query, over the
authors. Copying permitted only for private and academic entire set of retrieved elements. However, the query as a
middleman between humans and information spaces goes against
purposes. This volume is published and copyrighted by the
the transparent design of situated interfaces that support a direct
editors of euroHCIR2011.
interaction with information spaces. In addition, the information
spaces hosted by situated interfaces are usually predetermined or
pre-queried in accordance with the specific interests of potential
users and the characteristics or nature of the environments where Miller’s Law argues that the total number of different objects that
the hosting interfaces are situated. Instead of querying, the explicit humans can simultaneously hold in their working memory is
momentarily needs of users are answered by direct interaction approximately seven [4]. This affects the manner by which
with the visualized information. This superlatively converts the information is perceived when the cardinality of the visualized set
relevance of the displayed information to the user interests from a of objects increases. In particular, there is a natural observable
performance factor to a design issue. tendency to perceptually cluster or group these objects recursively
whenever the perceivable number exceed Miller’s threshold. To
Therefore, we argue that the definition of Recall and Precision can observe this phenomenon, eight 10 minutes long think-aloud
be loosened or reinterpreted to respectively describe the quantity sessions were organized with eight different university students
of retrieved information elements and their visual diversity as that watched InformationCasserole showing magazine ads
displayed on the interface, since relevance is no longer a progressively being added to the water container, and commented
performance factor from an HCI stance. These two metrics can on how the number of ads shown in the casserole affects the way
consequently act as parameters that bind the design and they perceive the set of visualized ads.
performance of situated interfaces as retrieval systems to the
informational expectations of users, by controlling the amount and We observed that when one object is shown, it tends to engage the
diversity of visualized information in order to maximize the subjects in a prolonged and detailed examination. This changes
transparency of their designs to support a direct human- when two to seven objects are displayed since subjects become
information interaction. more interested in identifying relations among the objects and
comparing them. The interest in object relations abates with a
In order to explore this idea further, we followed a line of higher object number, and instead the relations among clusters or
inductive design research by conceptualizing, designing, and collections of objects start to proportionally grab attention. When
evaluating experimental prototypes. We first introduce two sets of the number of visualized objects crosses a certain threshold,
prototypes devised to understand how users perceive the quantity
which we estimate at Miller’s number squared, the casserole
and visible diversity of information objects. We then define becomes perceptually saturated and the subjects begin to treat the
parameterization scales for Recall and Precision based on these set of ads as a space, reasoning about different regions in it. In
experiments. In order to develop a thoughtful understanding of conclusion, we find that the quantity of visualized objects (R) is
how Recall and Precision, which we will consecutively refer to as perceived in four different density thresholds, and to each we
R and P, can act as design parameters for situated interfaces, we accord a parameter value: R=0 for visualizing no or a single
use them in the analysis, design, and evaluation of five different object; R=1 for a single collection of seven or less objects; R=2
situated interfaces. Next, we investigate how these two parameters for seven or less collections; and R=3 for single information space
can be dynamically controlled by users through the design of two
or more than seven squared objects. This is reflected in figure 2.
interactive interfaces for searching and browsing news articles.
We conclude by assessing our experience and discuss the viability
and implications of our approach.
2. RECALL AND PRECISION FROM A
PERCEPTUAL STANCE
Figure 2. R as a design parameter
In order to study the effects that the visible diversity of
information objects (P) has on the manner by which people
perceive information, eight paper-based prototypes similar to the
InformationCasserole were conceived. Each prototype shows a
combination of twelve to fifteen information objects from
different genres (e.g. classified ads, news headlines, blog posts,
news pictures, movie posters, youtube videos, secondhand goods,
and city events). The object genre was emphasized and
differentiated by aesthetic design. The visible object diversity
encourages people to search for relations among visualized
objects [6]. Therefore, the combinations, ranging from one to
eight genres, were designed to encourage subjects to search for
Figure 1. An instance of the InformationCasserole prototypes patterns and relations among the objects. Six twenty minutes
think-aloud sessions were organized with subjects whom were
InformationCasserole is a series of video prototypes (figure 1) asked to search for and identify different genres of objects in each
designed to study the effect that the number of visualized of the eight combinations presented in random order.
elements (R) has on the way humans perceive the information
revealed on the interface. They show classified ads from As expected, the subjects perceptually clustered the objects
magazines and newspaper floating on different levels in a glass primarily in accordance to their genre. However, they sometimes
container filled with slowly moving water. Therefore, their tended to search for inner-divisions in objects of the same genre
settings emulate a transparent interface design and foster a direct (e.g. clustering movies according to their cinematic kind or news
relationship between the human and digital information spaces. articles in familiar news categories), or to merge related genres as
a single genre (e.g. news articles and blog posts, or movie posters
and news pictures). In total, the subjects perceived the diversity of
objects (P) in four different levels, and to each level we accord a · The amount of available user attention (e.g. MetroWindow
corresponding parameter value inversely proportional to the disposes of little attention in contrast with DigiJuke).
number of visible object genres: the first level is a single-genre · The duration of human interaction with information (e.g.
diversity (P=3); the second level is a diversity of two to three NewsWall remains in contact for prolonged durations, while the
genres (P=2); the third level refers to diversity of three to four interaction with YouServe is more momentarily).
genres (P=1); the fourth level describes a diversity of five to seven
genres of objects (P=0). Figure 3 shows the number of visible · The convergence or divergence of the information seeking
genres of objects in each of the eight combinations as seen by the tasks (e.g. YouServe supports finding a specific library service,
subjects, and the P value of each of the four identified diversity while Arts&Movies is designed to acquaint people with many
levels. movies).
Table 1. Values of R and P parameters for each interface
Situated interface Recall Precision
Arts&Movies 2 1
DigiJuke 3 3
YouServe 1 2
NewsWall 1 1
MetroWindow 0 3
The results of this R and P qualification are summarized in table
Figure 3. P as a design parameter
1. They show how R and P can characterize, from a perceptual
stance, the role of a situated interface as an information retrieval
3. SITUATED INTERFACES AS IR engine, and parameterize the design of its information settings
SYSTEMS accordingly. For example, when the user objectives are to search
In order to assess how R and P act as design parameters for the for specific objects (e.g. YouServe), R is minimized, while P can
information settings of situated interfaces, the following five be maximized when the search converges on specific genres (e.g.
interfaces that act as retrieval systems in real-world environments MetroWindow) or minimized when it diverges to cover many
were analyzed, and for each a corresponding design was genres (e.g. NewsWall). A maximized R signals that the
developed and evaluated in settings that resemble or emulate its interaction tackles a large number of objects. In this case, when P
deployment environment. is maximized (e.g. DigiJuke), it determines that this large number
is a single collection of similar objects, or, when it is minimized
The Arts&Movies is a situated interface intended for movie (e.g. Arts&Movies), it signals that this large number of objects is a
theatre lobbies to support the search and discovery of new visually diversified information space.
interesting movies through an animated visualization that draws
attention to relationships between movies and concepts. The The designers also developed the interfaces information
DigiJuke is installed inside a bar to allow people to browse and architecture and aesthetic design, but these activities lies outside
select music songs on the touch-screen, and play their video clips the scope of this paper. The final designs are shown in figure 4.
accompanied by related images on the projection display. The
YouServe prototype is collocated in a university library lobby to
assist people in familiarizing themselves with the available library
services, and finding a service relevant to specific needs. The
NewsWall is a large display situated in the news production room
of a broadcasting corporation. The prototype subtly visualizes the
constantly evolving news information space on the web. The
MetroWindow is designed for metro wagons and broadcasts
summarized local news about cultural and civic events in the city
of Barcelona.
In related works [7, 8] we have argued how R and P, as design
parameters, can be quantified during requirement analysis and
used alongside other aspects to conceptualize the design of
information interfaces. For each situated interface, a couple of
Figure 4. The situated interfaces final designs
designers analyzed the characteristics of three entities being: the
deployment environment, the humans present in it, and the
adequate information space, which was defined based on an 4. USER CONTROL OVER R AND P
understanding of the needs and goals of the humans alongside the Based on the discerned ability of R and P to describe the
nature of the environment and the information and activity flows information settings of situated interfaces and consequently their
that it hosts. Based on this analysis, the designers qualified the performance as information retrieval systems, we explored the
values of R and P for each situated interface, and consequently possibility of allowing users to control them dynamically in
described its information settings, being the quantity of classic search and retrieval scenarios. Therefore, we designed two
information to visualize and its visible diversity. This experimental prototypes (figure 5) for querying a large
qualification of R and P was defined in accordance with several information space of news articles, by which users can set and
non-disjoint or co-dependent situational aspects of human- control the values of both R and P. The prototypes were evaluated
information interaction such as: to assess the feasibility of this approach and its utility.
The NewSearch prototype collocates two slide-bars adjacently to re-querying, a more profound study should be conducted for
the query textbox for setting R and P explicitly, and returns an further analysis. Such endeavor will constitute the essence of our
equivalent clustered visualization of news articles. Users control future work.
the number of clusters (discerned by color) by P and their average
cardinality by R. The 3DQuery prototype uses a tag-map as a new 6. DISCUSSION
concept for defining user queries, and shows a corresponding map
The approach that we presented in this paper demonstrates that a
of news articles. The tag-map is a rectangular box where users can
tighter integration of HCI and IR is possible, by exploring the
place different tags of distinct sizes. The position of each tag
potential of R and P as design parameters for the information
determines that of the corresponding cluster of news articles, and
settings of situated interfaces. The use of these two performance
the tag size the cluster cardinality.
metrics as design parameters may be seen as controversial,
however, it is justified given that efficiency and information
relevance no longer constitute the backbone of user expectations
in all cases of human-information interaction. Instead, new
aspects of human-information interaction (e.g. emotional,
cognitive, experiential, situational, and cultural) are affecting the
manner by which we conceptualize information systems. Our
approach does not comprehensively address all these aspects, and
therefore can be complemented by introducing new parameters to
reflect with a higher affinity the aspects of human-information
interaction onto the system design.
Figure 5. NewSearch (left) and 3DQuery (right) prototypes
Each prototype was evaluated by a different group of ten subjects 7. ACKNOWLEDGEMENTS
in the lab. The subjects were asked to browse and read the The authors would like to thank Oriol Galimany and other
collection of news articles for fifteen minutes, and then answer a members of the Interactive Technology Group at Universitat
set of open-ended questions concerning their utility and usability. Pompeu Fabra for their support.
The user evaluations of both prototypes showed that their learning
curve is not negligible. Subjects were not naturally inclined to use 8. REFERENCES
the slide-bars of NewSearch to control the information settings. [1] Vogel, D. and Balakrishnan, R. 2004. Interactive public
An explanation for this may well be that they are accustomed to a ambient displays: transitioning from implicit to explicit,
given query paradigm and the difficulty lies in making the public to personal, interaction with multiple users.
paradigm change [9]. However, this issue requires further Proceedings of UIST '04, pp. 137- 146.
investigations. Subjects found it easy to use the tag-map paradigm
in general, but it was deemed too complicated for simple queries [2] NJ Belkin. Some (what) grand challenges for information
and more useful for prolonged search and exploration since it retrieval. ACM SIGIR Forum, 2008
allows users to dynamically adjust queries and therefore [3] R.A. Baeza-Yates and B. Ribeiro-Neto. 1999. Modern
eliminates or reduces the need for re-querying. Information Retrieval. Addison-Wesley Longman Publishing
Co., Inc., Boston, MA, USA.
The experience and knowledge gathered with the design and
evaluation of these two prototypes would be used for developing [4] Miller G. The Magical Number Seven, Plus or Minus Two:
future prototypes that intent to delegate more intuitively a Some Limits on Our Capacity for Processing Information.
dynamic control over the information settings of information The Psychological Review, 1956.
retrieval interfaces to their users. [5] L. Hallnäs and J. Redström. 2001. Slow Technology,
Designing for Reflection. Personal Ubiquitous Comput. 5, 3
5. CONCLUSIONS (January 2001), 201-212.
During the course of this paper we have explored ways to tightly [6] Koffa, K. (1935): Principles of Gestalt Psychology. London,
integrate IR and HCI in a variety of human-information Routledge & Kegan Paul Ltd.
interaction scenarios where interfaces act as information retrieval
[7] Moghnieh, A., & Blat, J. (2009). A basic framework for
systems. In particular, we studied how R and P as design
integrating social and collaborative applications into learning
parameters can describe the information settings of these
environments. Proceedings of m-ICTE’09 Vol. 2 (pp. 1057-
interfaces. Both aspects were parameterized on a 0-3 scale on the
1061), 2009.
basis of conducted experiments to analyze different possible
information settings. Consequently, five situated interfaces were [8] Moghnieh, A., Sayago, S., Arroyo, E., Sopi, G., and Blat, J.
designed and analyzed to discern how R and P are qualified Parameterized User-Centered Design for Interacting with
during requirement analysis, and how together they describe the Multimedia Repositories. In Proc. MMEDIA '09, IEEE.
information settings of situated interfaces, and therefore help [9] B. Buxton. 2007. Sketching User Experiences: Getting the
reflect the interaction requirements onto the interface design. Design Right and the Right Design. Morgan Kaufmann
Finally, we investigated the feasibility and utility of delegating Publishers Inc. CA, USA.
control of R and P dynamically to users during classic search and [10] S. Bødker. 2006. When second wave HCI meets third wave
retrieval scenarios, and concluded that while this approach is challenges. In Proceedings of NordiCHI '06.
clearly advantageous for exploration tasks and tasks that require