Studies involving user interfaces typically involve the capturing and recording (logging) of key user interactions between the user and the system being examined. However, anecdotal evidence suggests that researchers often implement their own logging infrastructure-sometimes in a piecemeal fashion-which can lead to numerous implementation mistakes (due to misunderstanding or ignoring diferences between web browsers, for example). While eforts have been made to develop interaction logging solutions for experimentation and commercial use, many solutions either use obsolete technology, are prohibitively expensive, are complex to use (and require extensive programming knowledge), or have no source code available. To address these issues, we have developed LogUI, an easy-to-use yet powerful interaction logging framework that can capture virtually any user interaction within a web-based environment. LogUI has been successfully used in several user studies since its launch. This paper provides an in-depth discussion into how we have designed LogUI, and provides narrative on the key challenges that we are looking to address moving forward.
which is dificult to implement correctly without extensive knowledge of the many developmental nuances.
Web applications [17, 26] are commonplace in our daily Anecdotal evidence (at least within the IIR community)
lives. The interplay that takes place between a user and suggests that researchers often work on developing their
a web application are in essence a series of interactions own web-based interaction logging software. This is
ofwith the elements that make up the webpage. This web- ten achieved in tandem with the implementation of their
page is rendered by the user’s web browser. Within the main experimental apparatus [18, 46, 47, 61]. However,
web browser, the elements are represented internally as we argue that this is highly undesirable. Lessons we have
part of the Document Object Model (DOM) [12, 34]. A user learned from our own experiences—and observations
of a contemporary web application will typically make from others in the community—suggest that developing
hundreds or thousands of interactions (such as clicking logging infrastructure is non-trivial, with researchers
on an element) with the rendered webpage’s DOM during bypassing issues such as network latency (for sending
its lifespan. For example, a recent user study reported captured events to a server); misunderstanding specific
by Roy et al. [52] observed 799 recorded interactions implementation details between diferent browsers (such
(e.g. mouse clicks) on average, over 120 participants. as how they interpret events in the DOM) [53]; or
simThis average value was observed over a prescribed ten ply forgetting to capture key interactions. Ultimately,
minute period, where participants interacted with a sim- these pitfalls may lead to low quality interaction logs,
ple Search Engine Results Page (SERP). with missing and/or noisy (or “raw” [
In order for researchers to gather insights into how this, inevitable post-hoc frustrations will arise. While a
users make use of a given web application’s interface, we series of separate interaction logging solutions have been
need a means to capture and record (or log) the aforemen- developed, we argue that we lack existing, easy-to-use,
tioned interactions that take place [
MLogger [
ALF [
WHOSE [21] UXjs [58]
Big Brother [54]
YASBIL [6] Li et al. [33]
LogUI [36]
Observer XT [68]
2010
2020
lenges to extend its functionality. This paper discusses video recordings of participants interacting with the
systhese challenges—both past and future—along with an tem in question [19, 50], permitting researchers to replay
overview of existing interaction logging systems. LogUI interactions post-hoc [66]. Hoiem and Sullivan [
Interaction logging software has a long history in com- These concerns led to the first generation of interaction
puting science literature, with pioneering implementa- logging systems, as shown in Figure 1 around the early
tions appearing long before the establishment of the 1990s. These were automatic in nature [56], whereby a
World Wide Web (WWW) [5] as the de facto commu- system not only supported the running of some system,
nications medium. Early discussions on the develop- but could also log the raw user events taking place—such
ment of interaction logging software date back as far as as key presses on a keyboard, or movement/clicks of a
the early 1990s [
Increasing interface complexity means that capturing
what users do becomes more of a challenge. Before
interaction logging software were established, the only
realistic way to perform usability testing [32] was for a
researcher to record a participant’s interactions with a
pen and paper [
1Modeless interface designs contain a number of components (e.g. dropdown menus, multiple windows, dialog boxes) instead of more simplistic, procedural, state-based interfaces—which were typically text-based. The following decade saw a number of platform-specific implementations. These captured a user’s interactions with (sometimes specific) native applications running on a platform—with solutions developed for MS-DOS, Microsoft Windows, and Apple’s Mac OS X/macOS.
In addition to the work by Philips and
Dumas [45], Hoiem and Sullivan [
OK
crosoft.2 They discussed how their solution evolved over
a number of iterations. Separate work by Noldus [42]
reported on a MS-DOS solution that could capture and
save keyboard interactions to a plaintext file. Both
solutions were confusingly called Observer, with the latter
still in development today as Observer XT [68]. Observer
XT captures mouse and keyboard events, and has been
used by several researchers [14, 59, 63]. Other select
solutions include Datalogger [66], InputLogger [62],
Recording User Input (RUI) [29], and AppMonitor [
Towards the start of the noughties, researchers were
beginning to investigate how the Internet could be used
to send interaction logs from native applications to a
central server [22]. This client-server approach to logging
profoundly increased the volume of logs that could be
captured. Advancements were made possible thanks to
improving networking capabilities at the time [
The DOM, as outlined by Dogac et al. [12], is a
treelike, object-oriented Application Programming Interface
(API) that is is used by programs that interpret HTML
documents [34]. The tree-like structure is derived from
each document’s source, and DOM objects—called
elements—are linked together through inheritance. Each
element contains a number of properties, such as class
names. CSS selectors can be used to apply styling to
change the appearance of elements as rendered within
the browser’s viewport; JavaScript engines interpret code
that programmatically manipulates the DOM over the
lifespan of a webpage in the browser. Following the
event-driven programming paradigm, developers track
As WWW technologies began to mature, it became
feasible to run user experiments through a web browser. Early
web-based interaction logging solutions captured
interactions on a webpage through the approach outlined in §2.3,
but recording them was a challenge—asynchronous
approaches for connecting to a server (like AJAX ) were
not widely available until the mid-noughties. Instead,
an intermediary proxy server, sitting between the client
(hosting the user’s browser) and server (hosting the web
application) was used. The proxy server would inject
JavaScript into the webpage as it was served to the client,
with the first recorded example using this approach being
WebQuilt [24]. For many years, this was the preferable
solution—a user’s navigation history could be logged to
the same server (the proxy server), avoiding Cross-Site
Scripting (XSS) [49] issues. The seminal UsaProxy [3, 4]
then followed, this time using AJAX to record
interactions within a page. This approach was used in a number
of studies [
Proxy-based solutions were preferred over the use of
2The project outlined by Hoiem and Sullivan [
Client (Web Application) 4 5 3 browser extensions. Users would often be reluctant to use these thanks to their historical reputation of being vectors for malware delivery [60]. Despite this, notable examples of solutions using this second approach include Wrapper [25], Search Logger [57], and the Lemur Toolbar.
The most recent phase of web-based interaction
logging software follows UsaProxy [3, 4] in making use of
so-called Web 2.0 [43] technologies, such as AJAX and
advancements in client-side scripting. Feature-rich APIs
and improved client-server connectivity eliminated the
need for proxy- and extension-based solutions. Examples
of these solutions include MLogger [
Nonetheless, further solutions have recently been
developed such as UXJs [58] and Big Brother [54]. These
solutions work well, but capture all events occurring on
a page without filtering, which may result in interaction
logs that are too “raw” [
This concern is also present with commercially available tools, such as Google Analytics, Hotjar, or Matomo.
Indeed, Google Analytics is largely designed for the capture of coarse-grained interactions, such as page viewing history [48, 58] (as opposed to fine-grained interactions, such as mouse clicks on a specific page). In addition, Matomo requires elements one wishes to capture interactions for to be given a specific class [58]. In essence, one can argue that this increases coupling between the logging infrastructure and apparatus.
LogUI is designed to ofer a complete, end-to-end
solution for capturing and recording a user’s interactions
with a given web application. It abstracts many of the
complex design decisions to the point that researchers
and developers can simply embed LogUI into their
apparatus, and start using it. LogUI comes with both a client
and server, each with their own responsibilities. The
design decisions behind LogUI have been motivated from
both our experiences and the recommendations of other
researchers [
The fundamental principle of LogUI is that of less is more.
From experience, we have found that if too much data
is captured, it can be overwhelming. Our principle here
(as opposed to other solutions like Big Brother [54] or
UXJs [58]) is to enable researchers to capture only what
they need (at the expense of greater complexity)—and
provide event context (through event coding [45]). As
previously mentioned, Hoiem and Sullivan [
3The Binder binds events to a specific function within LogUI, and does not interfere with EventListeners from elsewhere.
Browser Events Contrasting to events which are capConfiguration Object Central to our attempts of tured from interactions with DOM elements, LogUI also tackling the above problem is the configuration object supports a number of so-called browser events. Not per4 . This provides custom event coding functionality [45], taining to a specific element, events include the resizing and is central to the LogUI client. The configuration ob- of the browser’s viewport, a gain/loss of focus to the viewject is comprised of a series of rules which state exactly port window, and a change in the URL in the address bar. what should be captured, and when. CSS selectors and More are detailed in the documentation. CSS specificity rules are used to determine what listeners are bound to what elements (e.g. 5 uses element 3.2. Architecture Overview #box and the click event). Names are used for bindings, so that eventual logs report these names for easy identification of a logged event A .
LogUI utilises the client-server architecture—with the client responsible for capturing interaction data, and the server currently responsible for receiving and storing it Metadata One of the limitations of decoupling inter- for later access by researchers. action logging software from the main experimental apparatus is that researchers lose the ability to record the 3.2.1. LogUI Client internal state of the said apparatus with captured events. The LogUI client can be used within any web-based appliLogUI provides a solution to this problem with so-called cation, hosted on some web application server 1 . Built metadata. Metadata is unique to the element being inter- in a highly modular way, the LogUI client uses the conacted with, and can be appended to the captured event temporary Node.js JavaScript ecosystem to create a with ease. As an example, C shows the value of an Browserified bundle. This bundle is then dropped into input box, which represents query terms entered by a the target application via a <script> tag 2 . user. We can derive data from a number of sources, such Source components of the LogUI client each have as the corresponding DOM element’s attributes and/or its their own distinct responsibilities; a noteworthy example properties, or even from the state and/or props of the un- is the Binder component, which automatically binds derlying React component 7 (if React is used—support EventListeners to the elements in the DOM, as sefor other frameworks can be added in the future). lected by researchers.3 The component has a bespoke algorithm which parses the configuration object to work Application-Specific Data Similar to metadata de- out what DOM elements should be targeted. In addiifned above, we also provide researchers with the ability tion, it also uses the contemporary MutationObserver to append application-specific data to all captured events. web API to listen for changes to the DOM, and automatiThese data are global to the context in which a web-based cally appends listeners to new elements as necessary. As application is run, and has been used for capturing data such, this one feature alone makes LogUI an ideal fit for such as the user ID, group ID, and interface variant, as contemporary web frameworks like React—and as it conshown by B . These data (as well as metadata) can then siders the DOM directly, it is also completely framework be easily filtered post-hoc to report statistics, for example, agnostic. The client uses the principle of unobtrusive capover a given experimental condition. ture [56]; LogUI functions silently, with users unaware that it is recording their interactions.
The LogUI client is controlled through a simple API 3 that provides functionality for researchers to start and stop interaction logging as and when required (amongst other functionality). The only requirement is for a configuration object to be present 4 . When active, interactions on DOM elements that were bound to 5 are interpreted by the browser 6 and passed to the LogUI client. Metadata and application-specific data are then appended to captured events 7 if they are to be stored; the final result is then sent to the LogUI server 8 .
Event Groupings While researchers are free to use whatever DOM events they wish (e.g. click), we also provide a number of so-called event groupings. These abstract from native, browser-level events—and provide additional logic to remove much of the complexity of reliably and accurately capturing certain events.
For example, we have implemented a mousehover grouping, which internally listens for mouseenter and mouseleave events, and logs them separately. Furthermore, a scrollable grouping listens for scroll events on a page and/or element, reporting only start and end 3.2.2. LogUI Server The LogUI server’s 8 primary purpose is to receive incoming captured events from the LogUI client—after a connection has been established (§3.2.3). LogUI server has been fully containerised, meaning that a new instance of the server can be easily started on any installation with the correct version of Docker installed. Full documentation is provided on how to do this.
The server endpoint 9 accepts connections from instances of the LogUI client, and stores incoming data in backend data stores 10 . At present, we use an instance of MongoDB to store data, where each log entry is stored as a JSON object. We also include a worker process 11 within the containerised environment. This exposes a web server hosting the LogUI control application (§3.4).
This web server is only intended to be accessed by researchers using the given instance of LogUI server. Both the endpoint and worker processes use the Django framework to provide the necessary infrastructure for fulfilling requests. Like the LogUI client, the design of the server is modular in nature—with specific responsibilities for each component. 3.2.3. Client-Server Protocol A C D
B Communications between the LogUI client and server LogUI produces interaction log files in JSON format; logs are achieved via WebSockets [39] 12 to provide for a can easily be parsed by any analysis library that supports full-duplex channel between them. This contemporary JSON, such as Pandas. Each recorded log entry corresolution is preferable to AJAX-based implementations, sponds to a unique JSON object; details of this output are as WebSockets are not subject to the Same Origin Pol- available as part of the LogUI documentation. Refer to icy, meaning the LogUI server can reside elsewhere from Figure 4 for an example of a logged entry. the page’s host. WebSockets also guarantee that data Of particular interest in the illustrated log entry examwill arrive at the recipient in the order in which it was ple are the values for the applicationID, flightID, sent—and with only one connection needing to be estab- and sessionID keys D . These denote to the concepts of lished when the LogUI client starts, bandwidth and other an application, flight, and session, respectively. In LogUI resource requirements are reduced [56]. parlance, an application is the representation of a single
Sitting on top of the WebSocket connection is the web application (e.g. experimental apparatus). A flight LogUI protocol; an application-specific series of rules to is a variant of an application; think of a user study with enable the client and server to be able to understand one three conditions. In this case, the given application would another. The protocol includes a handshaking routine, have three flights. Finally, a session is the grouping of whereby the LogUI client sends an encrypted authorisa- a single user’s interactions, over one or more webpages tion token which, when decrypted, provides information pages, examined in a single sitting. These key/value pairabout the web application that the LogUI client is watch- ings are appended to every log entry, such that filtering ing. This helps the LogUI server know who is connecting. by either of the three is trivial to achieve. Connections are terminated by the server if the web application is unrecognised. 3.4. LogUI Control Application
The protocol also considers what happens when the connection is lost. If this happens, the client will continue to capture all events (and cache them). While doing so, it will also attempt to reconnect to the server. When reconnection is successful, the client will immediately send its cache to the server (after the initial handshake).
This process ensures that no loss of captured interaction events result when temporary connectivity issues occur.
LogUI also comes with an elementary control application 13 . This React-based application allows researchers to login to an instance of LogUI server, and control various aspects of the server 14 —such as creating applications and flights, or viewing session details and authorisation tokens. Further functionality includes the ability for researchers to download complete logs for a given flight, which can then be used in subsequent data analysis 15 .
Fast Setup A total of three experimental systems have been adapted to work with LogUI so far. In addition, LogUI has also been integrated with Jupyter Notebooks to capture interactions within its interface (such as logging what cells are being interacted with). Researchers who have used LogUI have highlighted the ease of being able to get everything required up and running quickly with their experimental system. This success has been attributed to the the easy-to-follow documentation, which has a step-by-step guide for setting up LogUI.
Identified Dificulties Although researchers were ultimately successful with integrating LogUI into their apSince its initial release in March 2021, LogUI has been paratus, this was not however without its dificulties. successfully used (to the best of our knowledge) in five Researchers at times struggled to figure out where to add unique research projects, one of which is now pub- the necessary API 3 calls into their codebase to start lished [52]. Each of the researchers who used LogUI LogUI. With SearchX being built with React, for examrelayed feedback about their experiences in informal dis- ple, finding the correct points to start and stop LogUI cussions. We now outline some of issues raised by the was crucial, and did require some efort to successfully researchers, some of which are elaborated on in §5. achieve. This is a crucial point: as we advertise LogUI to be decoupled, more can be done to enable it to work without this (potential) requirement, which we discuss more in §5. However, other approaches we took to mitigate the efects of decoupling logging infrastructure from the core experimental system—such as metadata—were reported to be straightforward to use.
Despite the achievements in developing LogUI, a number of non-trivial challenges remain that would add value to the project as a whole. From researcher feedback or discussions in the literature, we have identified a series of challenges and potential ideas that we outline below.
4According to caniuse.com, support for the
MutationObserver is high, at ~97%—but not quite 100%!
How would we cater for the 3% without support?
Integrating Experimental Systems with LogUI
One of the experimental systems integrated with LogUI is
SearchX [46], a large-scale, modular, open-source search
framework designed as the apparatus to be used in
highquality crowdsourced IIR-based user studies. Indeed, the Testing Infrastructure While the basic interaction
study reported by Roy et al. [52] used SearchX to provide logging capabilities of LogUI are functional, and it has
a straightforward SERP (consisting of ten blue links plus been demonstrated to work over diferent web browsers,
a small interface widget) to crowdsourced workers—with are all interactions being captured in the same way across
LogUI being used to record interactions by the work- each browser? We know that browser vendors can
interers within the said SERPs. Five distinct interfaces were pret standards in diferent ways, leading to slight
varitrialled, with each interface logged as part of a unique ations in how web applications look on diferent
platLogUI flight. Additional application-specific data were forms [53]. To examine this issue in more detail, we plan
supplied to LogUI, allowing for the recording of interac- to develop an extensive compatibility testing [
Backwards Compatibility As we outlined above,
wide-ranging support a variety of web browsers and
platforms is cruicial for success of LogUI. While the
iniSuccess Stories Indeed, across all systems, re- tial focus of our eforts has been to ensure LogUI works
searchers have commented on the simplicity of defining with all major, contemporary web browsers, an instance of
what interactions to capture through the configuration LogUI attempting to run on an older browser (say,
withobject (§3.1). Of course, being well-versed in web tech- out support of the MutationObserver web API, §3.2.1)
nologies, we would imagine that this may be more of is inevitable as we gain traction.4 To address this
shorta hindrance for those less experienced. One researcher coming, we should invest time in developing LogUI to
has noted the ease with which their gathered logs could be cater for older browsers. Thanks to readily-available
be parsed—the lack of “fluf” (erroneous, or superfluous Polyfills in the JavaScript ecosystem, patching holes in
events) made the post-hoc script writing process for data what a browser supports should be straightforward.
analysis much quicker, and less error-prone. This is a
vindication of the less is more approach (§3.1), and also Improving Device Support Work also needs to be
of our event groupings which resulted in more compact undertaken to ensure that support is suficient for the
logs—especially with scrolling and mouse hover
interactions, two of the major obstacles we have experienced
when conducting user studies in the past [35].
increasing number of events associated with touchscreen nising video against captured events [
Reducing Integration Barriers As discovered
by Hoiem and Sullivan [
A major stumbling block at present is the need for the
configuration object (§3.1), a necessary trade-of to
uphold the less is more principle. Taking this further,
GUI-based tools to generate this object may be useful
Analytics Dashboard One of our major goals with (as demonstrated by uBlock Origin [
Improving the LogUI Stack As part of developing LogUI to provide a solution for providing analytics and the visualisation of captured data, we will need to modify the LogUI server-side stack to promote such analyses— 6. Conclusions especially at scale. At present, the stack simply acquires and stores interaction data for post-hoc download by re- In this paper, we have presented discussed LogUI, our searchers. For future development, we may follow the new web-based interaction logging infrastructure. Based path of the ELK stack, using Kibana, Logstash, and Elas- on the recommendations and observations of researchers ticsearch to provide the necessary groundwork and func- who have developed similar systems, our implementation tionality for data analysis. We must also consider the has been well-received by several researchers who have scalability of any developed stack, both in terms of the successfully used LogUI for their experimentation since number of users, plus the volume of data that must be its initial release in March 2021. Despite the positive examined for any analysis to take place. feedback we have received, several exciting ideas for future development have been discussed that will improve Recording Viewports To complement the proposed the ofering that LogUI provides to potential end users. changes above, recording the viewport of a user’s Our end goal for LogUI is to be a complete end-to-end browser would also be desirable. While there are un- solution for interaction logging, all without the need for doubtedly challenges to capturing, storing and synchro- researchers to write a line of (analysis) code.
Research and development of LogUI has been supported by NWO projects SearchX (639.022.722) and Aspasia (015.013.027). Thanks to , Johanne, and Nirmal for their much-appreciated proofreading eforts. We would also like to thank our two anonymous reviewers for their comments—especially those of reviewer two. Your pointers greatly improved the clarity of this work, and will hopefully lead to interesting discussions. 40 (4) (2004) 655–675. [62] S. Trewin, InputLogger: General-purpose logging of keyboard and mouse events on an Apple Macintosh, Behavior Research Methods, Instruments, & Computers 30 (2) (1998) 327–331. [63] A. van Drunen, E. L. van den Broek, A. J. Spink, T. Hefelaar, Exploring workload and attention measurements with uLog mouse data, Behavior research methods 41 (3) (2009) 868–875. [64] P. Voigt, A. Von dem Bussche, The EU general data protection regulation (GDPR), A Practical Guide, 1st Ed. 10. [65] X. Wei, Y. Zhang, J. Gwizdka, YASFIIRE: Yet Another System for IIR Evaluation, in: Proc. 5th IIiX, 316–319, 2014. [66] S. J. Westerman, S. Hambly, C. Alder, C. W. WyattMillington, N. M. Shryane, C. M. Crawshaw, G. R. J. Hockey, Investigating the human-computer interface using the Datalogger, Behavior Research Methods, Instruments, & Computers 28 (4) (1996) 603– 606. [67] G. Williams, Apple Macintosh computer., Byte 9 (2) (1984) 30–31. [68] P. H. Zimmerman, J. E. Bolhuis, A. Willemsen, E. S.
Meyer, L. P. J. J. Noldus, The Observer XT: A tool for the integration and synchronization of multimodal signals, Behavior research methods 41 (3) (2009) 731–735.