Wikipedia is a resource used by many people for many different purposes. We posit that it might be bene cial to alter the content or the way content is presented depending on the task context. Here we describe a small pilot lab study to investigate features of interaction that might help to infer the contextual situation surrounding wikipedia search tasks. We describe our e ort to collect data and analyse relationships between the features and the assigned task context.
Information portals such as Wikipedia represent rich
sources of information covering an incredibly broad range of
topics. Many Wikipedia entries are also long and can cover
aspects ranging from overviews and introductions to more
detailed descriptions of advanced aspects that are perhaps only
suitable for topic experts. Single pages can also contain not
only text, but images, info-graphics, lists and navigational
information. Previous research suggests that these resources
will have several di erent contexts of use. For example,
Marchionini [
Wikipedia contributors are encouraged to create pages in a way that meets the needs of as many users as possible by including information on a topic with su cient quantity, quality and completeness and structuring the content in a way that makes sense generally. Nevertheless, one could imagine that di erent content or di erent presentations of the same content might be more suitable in speci c contexts. For example, lookup tasks may be best supported when facts in an article are presented as a list that can be scanned easily. In such scenarios, content such as images may be less helpful and perhaps even distracting. Contrastingly, in casual-leisure situations, users may want to focus on multimedia content or have information presented in a way that encourages browsing and information discovery.
We believe examples like this suggest there may be bene t in moving away from static pages, which try to cater for all usage situations, to dynamic pages that are generated appropriately based on the context of use. As a rst step towards exploring this hypothesis, in this paper, we investigate how the context of use { the task type being performed { might be detected automatically from user-interactions with the system. We want to establish if the way the user interacts with the system, e.g. his mouse and keyboard interactions, eye movements, and click behaviour can provide implicit feedback regarding the usage scenario and user goals.
With this aim in mind, we present a small pilot study that allows us to evaluate a methodology for detecting the features of interaction that might help us infer the contextual situation surrounding a user's search task. We collect interaction data in the context of a controlled laboratory study and analyse relationships between the features of interaction and the assigned task context. The data show that for the small number of users in our study, the behaviour exhibited when completing tasks of di erent types is very di erent; users interact with di erent types of content in di erent ways. Further, we provide evidence that it is possible, at least for some users, to predict these behaviours based purely on mouse and keyboard interactions. 2.
In the IR community a large amount of work has been performed to establish if interaction data can be used as a surrogate for explicit relevance judgements. This is known as implicit relevance feedback. Early research in this area demonstrated a correlation between the time spent reading a Examine EX Navigate NV label description RE User is reading text SC User scans content e.g. headlines, lists or whole page User examines element
User navigates element label Headline HD List LI Picture PI Charts, tables etc. IG Other navigation ON element label Text passage TX Introduction IN Info Box IB
Links in Wikipedia WI
document and explicit relevance judgements [
Interaction data can also be used as a means to predict
user emotions. For example, Fox et al., show that query log
features can be used to predict searcher satisfaction [
A third group of studies show correlations between di
erent styles of interactions e.g. for some users visual attention
on the screen can be predicted via mouse coordinates [
In this section we provide details of the data collected and explain the motivation behind recording the data. 3.1
Data was collected via a laboratory based user study with 4 users. The participants were information science students (1 male, 3 female) aged between 20 and 30. All of the participants were experienced wikipedia users and were comfortable using the wikipedia search facilities. Although this user population is not large or diverse enough to provide generalisable results, it is su cient for our aims, which were to evaluate and improve the methodology and get a sense for the feasibility of our ideas.
Each participant performed 6 Wikipedia search tasks (2 of each of the 3 types of interest - lookup, learn and casualleisure). The tasks were presented in the form of a simulated scenario and were ordered randomly to minimise learning e ects. Example tasks for each type are shown in Figure 2.
After initially greeting the participant, the experimental procedure was explained in person. Then, to prevent biases, the participant was led automatically through the experiment on screen, with task descriptions, questionnaires and a web-browser window appearing when appropriate. The experimenters observed the tasks remotely in an adjoining room, where the participant's screen was mirrored. 3.2
We collected a large amount of data from each participant before, during and after the study.
Questionnaires: A pre-study questionnaire collected demographics, search experience, and experience with wikipedia of the participants. Pre-and post-task questionnaires elicited perceptions of the task and domain knowledge, of success and the experience including emotional aspects, and nally a post-study questionnaire provided general impressions of the experiment.
Eyetracking Data: We recorded participant gaze patterns
using an SMI RED eye-tracker. The associated BeGaze
software recorded videos les of screen interactions with an
additional layer indicating the area of the screen where the user
is focusing his gaze. We manually annotated these complete
overlaid video sequences with two labels. The rst describes
what the user is doing ("action"). This is a simple coding
scheme but aligns with reading psychology research [
It was the annotator who decided which action to code at what moment by following the focus displayed in the layer on top of the recorded screen. The second label describes the content ("element") being focused on and is derived from the elements available in Wikipedia pages. The label was assigned when the focussed on an area on the screen so long that the annotator could assume the element in the area was perceived.The full set of labels for actions and elements is presented in Fig. 1. The intuition behind the labels was that the style of reading for di erent task types and the content elements used will be very di erent. By labelling videos in this way we could test this intuition empirically.
Browser Logs: We instrumented the refox web-browser to log all user interactions during the search process.
Timestamp information was used to align interaction data from di erent sensors.
Lookup: Last night you watched a documentary about the sinking of the Titanic. Suddenly you wonder how many passengers were on board when the catastrophe happened. Search in Wikipedia for this information.
Learn: Friends from abroad are visiting Germany and you plan to travel together to visit the small but beautiful city of Regensburg.
As preparation for the trip you want to know more about the city and its history. Use Wikipedia to do this. Casual-leisure: You have a few minutes before your class starts but you are already sitting in the lecture hall. Kill this time using wikipedia using the next six minutes to look at whatever topic(s) take your fancy.
We analyse the data in two stages. First, in Section 4.1, we examine the distribution of video labels for di erent types of task to determine if users behave di erently or focus their attention on di erent kinds of topics when completing di erent task types. Second, in Section 4.2, we show how these labels can, in turn, be predicted using interaction data from the eyetracker and browser. The rst stage provides evidence that the user's preferences for content elements depends on the search task, endorsing our suggestion to customise web pages at run time. The second stage provides some evidence for our hypothesis that the interactions a user performs in a browser may be used to predict which actions he trying to complete and which content elements he is preferring at that moment.
Technical di culties meant we were only able to work with data for 6 casual-leisure, 4 lookup tasks and 4 learn tasks. We rst divided the data into 500ms frames, allowing us to normalise the counts by task length, and counted relative frequencies of frames for which label combinations occur for each task type (see Table 1). Visually inspecting the distribution of content for actions, suggests the reading style and the elements of content interacted with were very di erent in di erent task contexts. This is con rmed by pair-wise comparisons using chi-squared tests for the distributions content elements for each possible pair of task types (see Table 2).
Examining the results in Table 2, we observe that all but one combination of action type shows highly signi cant differences in the distribution of content elements examined. The exception is the distribution of elements for lookup and casual-leisure tasks, which initially seems counterintuitive, as one would expect these two tasks to be very di erent. Below we summarise the main similarities and di erences between the task-types and attempt to explain what these mean in the context of our work.
When completing lookup tasks, the participants do not typically read content, the exception being page introductions. Instead they scan large portions of the page very quickly, looking for the snippets of information that will satisfy their speci c information need. They tend to scan a number of di erent kinds of content elements during tasks. This can be seen from Table 1 with counts being spread over text passages, introduction, info boxes, lists and headers. Images are noticeably missing from lookup tasks. It seems as if the participants have decided that for the tasks assigned, images will not useful and are able to avoid them.
Learn and casual-leisure di er from lookup tasks in that they both tend to be longer in time and have more interactions. They also both involve reading actions, which were rare for lookup. By this we mean that the user focuses attention on whole passages of text and attends the text from left to right and line by line. Another similarity between learn and casual-leisure tasks is the way that text passages are consumed, with the counts for these tasks being very similar. There are di erences between learn and casual-leisure tasks, particularly in terms of the elements used other than text passages. During learn tasks the focus tended to be on headers, while for casual leisure, the focus was on elements such as introductions and info boxes, which allow the user to gain an overview of what a page is about and allow them to judge whether it is interesting or not. We assume that headers are useful for learn tasks because here there is a concrete information need i.e. users do not just need to nd something that is interesting or not, but need speci c informational content. In this sense headers will help the user determine whether a paragraph is worth reading or not. 4.2
To determine if the manually assigned labels can be predicted from interaction data alone, we calculated statistics for counts of the synchronous occurrences of video labels and input events for the 500ms frames introduced above. As we were searching for the simplest features possible (so they could eventually be computed easily during a browser session at runtime) we used the frequencies of the most common mouse events and the average saccade distance (i.e. eye movement) per frame as features. More precisely, for each frame we descretised these features into two levels: low and high based on the mean value over all frames.
Table 3 (left) gives an example for the information we
computed from the raw log data. In order to understand
whether the knowledge of the mousemove frequency is
relevant for predicting user actions and content elements, we
performed a series of 2-squared tests for all six search tasks
for one of the test participants chosen at random (in total
about 30 minutes of interaction). The results are reported in
Table 3(right). With the exception of the rare click events,
all features are highly signi cant. We interpret this as a
positive indication that for individual users { depending on their
personal interaction style (see [
Task *** *** ing session. The results of the 2-squared tests indicate that knowing at run-time whether the observed input events occur below or above average at any point of time increases the accuracy of predicting the video labels as annotated for that moment as the distribution P (actionjevent = low) di ers signi cantly from the distribution P (actionjevent = high) for any annotated action and for any annotated element type. This oberservation opens the way for runtime prediction of the user action and preferred elements. From that information, the system can predict the current task type and use this information for generating content dynamically.
The preliminary data analysis we have presented provides clues that, rstly, reading behaviour and preferences for content elements depend on the surrounding task context and, secondly, both behaviour and preferences may be predicted for individual users based on their interaction style.
There are several limitations to this work. That we only
have data from four participants from a relatively
homogenous group means we cannot generalise. However, we claim
that the presented methodology is well suited to address
our long term research questions outlined in the
introduction and the pilot has provided us with insight into how to
improve a full study. In addition to resolving several
technical challenges, we have learned that the great care will
need to be taken when simulating tasks. For example, were
few images looked at in lookup tasks, simply because of the
tasks we chose? We also plan to look at more complicated
prediction features and account for the fact that individual
di erences in participants (cognitive, reading style [