-
The think aloud approach is one example method used for
eliciting qualitative details of user experience, but both the
experimenter effect and the weaknesses of introspection are
well known [
14
]. Some physiological logging approaches,
such as eye tracking, heart rate, body temperature, and
pupil-size monitoring can also be used if the participant is
in a lab environment. Studies even consider brain scanning
methods to estimate user cognitive load [
5
]. Can we elicit
cognitive aspects from logs of distant users? This position
paper explores the potential of eliciting cognitive actions
from usage logs, which we know are part of search.
COGNITIVE ACTIONS DURING SEARCH
Many models of information seeking assume that there are
cognitive stages in the search process. Marchionini [
10
],
Ellis [
4
], and Kuhlthau [
9
], all identify stages such as need
identification, examining results, and reflecting on whether
a task has been completed. Similarly relevance judgments
[
11
] are presumed to be a key part of searching as a user
chooses which search results to view.
Many analytical evaluation methods for interfaces define
cognitive actions. The Keystroke Level Model (KLM) [
3
]
was designed to estimate how long it would take to perform
a task with a user interface, by providing time estimates for
actions like clicking and typing. Further, KLM suggests that
the average time for a mental action is around 1.2 seconds
and may include actions such as: initiating a task, making a
strategy decision, retrieving a chunk from memory, visual
search on the screen, thinking of a task parameter (like a
keyword for a query), and verifying that something has
happened. The GOMS method (Goals, Operators, Methods,
and Selection rules) identified two types of non-interactive
actions: cognitive and perceptual. Cognitive actions include
initiating, choosing, planning. Perceptual actions include
reading and performing visual search. These were later
made more explicit in a variation called CPM-GOMS
(Cognitive-Perceptual-Motor GOMS – also Critical Path
Method GOMS), suggesting these cognitive, perceptual and
motor (interactive) actions may occur in parallel [
7
].
Bates discussed both mental and physical actions in an
analysis of different levels of search strategies [
1
]. Her
model, which was operationalised in a recent information
seeking evaluation framework [
16
], suggests that there are
four levels of strategy: Strategies, Stratagems, Tactics, and
Moves. She defines these moves as ‘An identifiable thought
or action that is a part of information searching’. Tactics,
such as comparing, narrowing results, expanding results,
varying queries, etc, are made up of moves. Stratagems,
such as checking journal issues or searching for citations,
are made up of a combination of tactics and joining moves.
Finally strategies, which are similar to realistic work tasks
like verifying a citation, or researching for a report, are
made up of a combination of stratagems, tactics, and
moves. Consequently, all four levels involve cognitive
actions. Bates’ definition of moves, and subsequently the
information seeking evaluation method by Wilson and
colleagues, takes a much less rigid view of mental actions
compared to timeframe analyses like KLM and GOMS.
INTERFACE ELEMENTS FOR FEEDBACK
Elements or features of user interfaces are often designed to
provide feedback to users or support orientation. Although
these often-passive elements, like breadcrumbs, can be used
to navigate around an interface, they may be often used
without any direct interaction. Anecdotally, Pickens has
blogged about the dependence on usage logs1 and the value
that can be gained from classifications without direct
interaction2. This topic was discussed in the CHI09
Sensemaking workshop. Further, at CHI09, an audience
question asked whether tag clouds are better for aiding
retrieval or providing contextual information about results.
Empirically, Wilson and colleagues have shown that users
can recall labels from faceted classifications that did not
receive direct interaction [
15
].
IDENTIFYING COGNITIVE ACTIONS IN USAGE LOGS
The solution for identifying cognitive actions from usage
logs is by no means obvious. Several existing studies,
however, can provide some insights into how we might
begin to do so. Multiple studies have, for example, noted
that users sometimes move their mouse to the most relevant
result seen so far while continuing to scan results [
2, 12
].
The combination of eye tracking and mouse tracking used
tells us more about both perceptual actions (scanning the
results) and cognitive actions (judging relevance), before
interaction occurs (clicking). Further this reinforces the
notion that we can use triangulation of, in this case, logging
methods to build richer pictures of search experiences.
Similarly, in a study performed by schraefel and colleagues
[
13
], audio previews were provided with labels in the facets
of a classical music dataset. The hypothesis was that
multiple previews would improve user choices while
browsing, and would ‘back out’ of their decisions less
often. This mental action of ‘backing out’ on a decision was
measured in logs by a pattern of interactions showing the
user clicking on higher levels of the classifications from
their previous position. In this case, therefore, certain
cognitive actions were modeled as a sequence of physical
interactions, in an environment where mouse and eye
tracking were not used. Although schraefel and colleagues
identified specific mental actions, it may be possible to
identify common interaction patterns that abstractly
represent known perceptual and cognitive search Moves.
CONCLUSIONS
Search is irrefutably made up of both mental and physical
actions: we cannot interact with a system without first
choosing how to interact with it. The challenge, therefore, is
to try to elicit common mental actions from logs of physical
interactions. There are two key avenues that we envisage
for beginning to do so. First, triangulation of multiple
measures is already known to provide a richer
understanding of user experiences and applies to logging
too. Second, modeling sequences of physical interactions
may allow us to estimate what has happened in the gaps.
Regardless of how it is eventually achieved, the key
position held here is that evaluating searcher experiences
with usage logs should focus on what happens between the
captured physical interactions.