UTAssistant: a Web Platform Supporting Usability Testing in Italian Public Administrations

Abstract

Even if the benefits of the usability testing are remarkable, it is scarcely adopted in the software development process. To foster its adoption, this paper presents a Web platform, UTAssistant, that supports people, also without skills in Human-Computer Interaction (HCI), in evaluating Web site usability.

Introduction

HCI literature proposes different methods for evaluating usability of interactive systems. Among them, usability testing is generally considered the most complete form of evaluation, because it assesses usability through samples of real users and permits to gather a wide range of qualitative and quantitative data, whose analysis allow to detect possible issues about the system and the interface elements which cause them [ 7 ].

Although the benefits of this technique are unquestionable, it is hardly adopted due to the required effort and time needed to set up reliable user testing. A frequently used technique is the thinking aloud in which users are asked to verbalize their actions during the interaction with the system to complete a set of critical tasks [ 7 ]. During the task execution, the evaluators can collect quantitative data, e.g. task time/clicks, and qualitative data, e.g. facial expressions and externalized comments. In addition, questionnaires are typically administered to evaluate the users’ perceived usability. Then, all the collected data are analyzed to evaluate the software usability with respect to the administered tasks, thus producing statistics like the task completion time or number of clicks to accomplish tasks, task success rate, heat map of the user behavior, questionnaire results and graphs, and so on. To foster the adoption of usability testing technique, in the past few years, there has been massive growth of usability testing tools. For example, professional platforms like Morae, Ovo Logger and User Testing support HCI experts in most of the evaluation activities [ 2 ][ 9, 11 ]. Despite their powerful features and automatization of data gathering and analysis, their use is limited by their price and/or the need to install them on dedicate machines (e.g., Morae and Ovo Logger). Simpler and cheaper tools are also available but they typically cover part of the evaluation process, for example the user behavior analysis obtained gathering and elaborating the mouse movements/clicks (e.g., UsabilityTool) or the facial expressions and externalized comments recording (e.g. Camtasia).

This paper proposes a Web platform, called UTAssistant, which assists evaluators, even without skills in HCI, in performing usability testing.

UTAssistant automatizes and simplifies the usability testing process in all its phases, i.e. design, execution and data analysis.

UTAssistant

UTAssistant is a Web platform designed and developed within the PA++ Project, an initiative coordinated by the High Institute of Communication and Information Technology (ISCOM) scientific and technical body of the Italian Ministry of Economic Development (MISE), whose main goal is to promote the usability practices in the Italian Public Administration (PA). In particular, the PA++ (A Public Administration + mobile and + usable: design and evaluation of Public Administration Web sites) project involves a grouo of people that are members of the GLU (Gruppo di Lavoro per l’Usabilità, in English, Working Group on Usability). The GLU is related to the Italian Ministry of Public Administration and is working to support people of the PA website staff in performing usability evaluation activities of their websites and other e-government systems. Among the different activities of the GLU, in May 2015, a new version of the eGLU and eGLU-M Protocol 2.1 has been published [ 1 ]; its aim is to guide web masters and web editors, who do not have experience on usability and UX evaluation, in performing usability testing of the websites they work on, by committing limited resources in terms of time and people.

UTAssistant has been conceived to support remote usability testing executed by using desktop and mobile devices, from the test design to the analysis of the data collected following the guidelines provided by the eGLU 2.1 and eGLU-M evaluation protocol. UTAssistant has been entirely developed as Web platform. The idea is to provide Italian PA with a lightweight and simple service that does not require any installation on user devices or special requirements. In fact, thanks to the use of the recent evolutions of the HTML5 standard, the browser gathers data from desktop and mobile peripherals like webcam, microphone, mouse, keyboard, and so on. From a technological point of view this is an important step forward with respect to the state-of-the-art, which can foster a wide adoption of this tool and, consequently, of the usability testing technique. Indeed, the existing software for usability testing, require dedicated devices (e.g. Morae) or cover part of the evaluation process and often do not support the evaluation of web site used on mobile devices that today represent the most common interaction device. In the following sections, it is described how UTAssistant supports evaluators in designing a usability test and in analyzing the results, as well as how participants are driven by UTAssistant in completing usability tests.

Test Design

The first phase of a usability testing is its design. The most important issue of this phase is the study creation that mainly consists in defining a set of tasks users have to complete during the test execution. After the logging into the system, evaluators create a new study: a wizard procedure guides them in designing the test following three main steps. At the first step of the test design, the evaluators have to define: a) general information about the study (e.g., a title, a short introduction), b) data UTAssistant has to gather while participants carry out the tasks (e.g., mouse/keyboard data log, audio/video from participant webcam/microphone), and c) questionnaires participants have to complete during/at the end of the study. According to the eGLU protocol, UTAssistant actually implements post-questionnaires like SUS [ 5 ], NPS [ 8 ], UMUX-Lite [ 6 ], USE [ 10 ]. Another postquestionnaire, not mentioned by the eGLU protocols but also implemented in UTAssinstant are: AttrakDiff, [ 4 ] and the NASA-TLX questionnaire [ 3 ]. After defining the study general information, at the second step evaluators define the task list. For each task, they have to provide the title, the starting/ending URL, the goal, the duration. Finally, the third step requires evaluators to specify the study participants, by selecting them from a list of users already registered in the platform or by typing their email. Then the study has to be saved. The invited participants receive an email describing all the instructions to participate in the usability testing.

Test Execution

A usability testing execution is typically managed by one or more HCI experts, which assist participants in carrying out the study tasks, gather data (e.g., externalized comments, task times/click) and administer the questionnaires. UTAssistant aims to automatize all the execution steps guiding participants even without the physical presence of evaluators. Once the participant receives an email, by clicking the URL reported inside it, UTAssistant is open in a browser on the desktop or mobile device. First, UTAssistant shows general indications about the platform (e.g., a short description of the toolbar with all the useful commands) as well as the privacy policies indicating that data like mouse/keyboard log data and webcam/microphone stream could be captured. Then, UTAssistant administers the tasks one at a time. For each task, the platform shows the goal in a pop-up window. A minimal toolbar is positioned on top of the task web page reporting the current task goal and description, the duration time, the task number, and the buttons to go to the next task, or stop the study. If the user is executing the last task, the “Next task” button becomes “Fill in the questionnaire”. All the steps, are completely assisted by UTAssistant that simulates the presence of the HCI expert.

Test Data Analysis

One of the most time-consuming phase of a usability testing is the data analysis. HCI experts have to store, merge and analyze the mouse logs, the video/audio recordings, the questionnaire results, and so on. UTAssistant assists evaluators in performing these activities, thus making easier and lighter the entire analysis. UTAssistant provides different types of tools for the automatic data analysis and to produce information that facilitate the evaluators’ work in detecting usability issues.

One of the most important usability metric is the task success rate, i.e. the percentage of tasks that users complete correctly during the study [ 7 ]. UTAssistant creates a report showing the general success rate (“Tasso di successo medio complessivo” in Italian as shown at the bottom of Figure 1) and a table reporting the details for each task (column)/user (row). In addition, for each task UTAssistant shows: its title, the number of pages the user visited to carry out it, the number of notes evaluator made during the analysis, the date and time in which the task starts and finishes, the average time and the average number of clicks all the users needed to complete the task.

A different analysis of the mouse/keyboard logs is provided by the heatmaps, i.e., graphical visualizations overlapped to a web page that show where people click/touch, move the mouse, scroll. It uses hotter colors, e.g. orange or bright red, on areas that get the most clicks/movements and colder colors, e.g. blues and purples, on areas that get the less clicks/movements. For each web page, UTAssistant builds a heatmap that evaluators can analyze to understand if some area of the web page wrongly attracted the users’ attention due to some user interface elements. Figure 2 shows an example of mouse heatmap.

UTAssistant automatically analyzes questionnaire data and provides proper statistics and graphs to summarize the results and to help evaluators in understanding the perceived usability, cognitive load and so on.

Conclusion

This paper has presented a web-based platform, called UTAssistant, developed in the project PA++, to support people working in PA website staff in performing usability tests. A preliminary evaluation of the UTAssistant, involving 2 people acting as evaluators and 6 people as participants, showed that evaluators appreciated a lot the support to the evaluation activities, while participants enjoyed the possibility to participate in a user test, without being observed.

Acknowledgements

This work is supported by ISCOM under grant PA++ (A Public Administration + mobile and + usable: design and evaluation of Public Administration Web sites).

References

1. eGLU and eGLU-M Protocol 2 .1. Retrieved from http://www.funzionepubblica.gov.it/glu.aspx Last Access June 4th, 2017

2. Morae . Retrieved from https://www.techsmith.com/morae.html Last Access June 10th, 2017

3. Hart S. G. and Staveland L. E. ( 1988 ). Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research . Advances in psychology 52(1) , 139 - 83 .

4. Hassenzahl M. , Burmester M. , and Koller F. ( 2003 ). AttrakDiff: A questionnaire to measure perceived hedonic and pragmatic quality . In Proc. of the Mensch & Computer . 187- 96 .

5. Lewis J. and Sauro J. ( 2009 ). The Factor Structure of the System Usability Scale . In Human Centered Design - HCD 2009 , Kurosu M. Ed. Lecture Notes in Computer Science , Vol. 5619 , Springer Berlin Heidelberg, 94 - 103

6. Lewis J. R. , Utesch B. S. , and Maher D. E. ( 2013 ). UMUX-LITE: when there's no time for the SUS . In Proc. of the Conference on Human Factors in Computing Systems (CHI '13) . Paris, France, ACM, New York, NY, USA, 2099 - 102 .

7. Preece J. , Sharp H. , and Rogers Y. 2015 . Interaction Design-beyond human-computer interaction . John Wiley & Sons.

8. Reichheld F. F. ( 2003 ). The one number you need to grow . Harvard business review 81 ( 12 ), 46 - 55 .

9. Ovo Logger . Retrieved from http://www.ovostudios.com/ Last Access June 10th, 2017

10. Terenzi M. , Di Nocera F., and Ferlazzo F. ( 2006 ). Firmitas, Utilitas, Venustas: Assessing the Validity of the Usability Evaluation (Us . E.) Questionnaire . Developments in Human Factors in Transportation,

11. UserTesting. Retrieved from https://www.usertesting.com/ Last Access June 10th, 2017