Many improvements of the interplay between usability evaluation and software development rely either on better methods for conducting usability evaluations or on better formats for presenting evaluation results in ways that are useful for software designers and developers. Both approaches involve a complete division of work between developers and evaluators, which is an undesirable complexity for many software development projects. This paper takes a different approach by exploring to what extent software developers and designers can be trained to carry out their own usability evaluations. The paper is based on an empirical study where 36 teams with a total of 234 first-year university students on software development and design educations were trained in a simple approach for user-based website usability testing that was taught in a 40 hour course. This approach supported them in planning, conducting, and interpreting the results of a usability evaluation of an interactive website. They gained good competence in conducting the evaluation, defining task assignments and producing a usability report, while they were less successful in acquiring skills for identifying and describing usability problems.
H.5.2 [Information Interfaces and Presentation]: User
Interfaces – Graphical user interfaces (GUI), Theory and
methods.
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. To copy
otherwise, or republish, to post on servers or to redistribute to lists,
requires prior specific permission and/or a fee.
activities are mutually dependent, but in practice there is often too
little or no fruitful interplay between them [
Considerable efforts have been devoted to improve the interplay
between usability evaluation and software development. A
substantial part of these efforts reflect two typical approaches.
The first approach focuses on better methods. The aim is to
improve the products of usability evaluations through use of
methods that provide better support to evaluators that carry out
usability evaluations. During the last 20 years, a whole range of
methods have been developed within this approach. A prominent
and influential example is Rubin [
The second approach focuses on better feedback. The aim is to
improve the impact of usability evaluations on user interaction
design. This is achieved in a variety of ways, typically by
improving the format that is used to feed the results of usability
evaluations back into user interaction design. The classical format
for feedback is an extensive written report, but there have been
numerous experiments with alternatives to the report; see [
Compared to both of these approaches, website development is, however, particularly challenging. Websites exhibit a huge and unprecedented amount of information, services and purchasing possibilities, and the users of websites are a tremendously heterogeneous group that use websites for a multitude of purposes any time, any place. Due to this, website developers must accommodate a massive variety of user preferences and capabilities.
Many contemporary websites suffer from problems with low
usability, e.g. an early investigation of content accessibility found
that 29 of 50 popular websites were either inaccessible or only
partly accessible [
This paper presents results from an empirical study of a course where first-year students in software development and design educations were trained to conduct their own user-based usability evaluations. The aim of the approach behind this course is to facilitate direct integration of usability evaluation into software development by removing the division between evaluators and developers. In the study, we explored whether designers and software developers who had received a 40 hour training course could conduct a usability evaluation of a reasonable quality. In the following section 2, we present previous work related to our study. In section 3, we describe the study in detail. The results of the study are presented in section 4, and section 5 discusses additional aspects of the results. Finally, section 6 provides the conclusion.
The idea of reducing the gap between software development and
usability evaluation by broadening the skills of software
developers is not new. It has been suggested that education of
software developers in usability engineering could contribute to
reduce the problems with the usability that characterize many
software products. This suggestion focused on a general
awareness of usability issues and on the early activities in a
development project [
It has also been discussed on a more general level how
development teams can best be trained to use fundamental
techniques from the usability engineering discipline. This requires
systematic empirical studies of the true costs of learning and
applying usability engineering techniques [
We conducted a search on the web on training of software developers in usability engineering. We found a group of companies that offer training courses for software developers in various methods from the usability engineering discipline. The two most common methods were the so-called discount usability evaluation techniques (expert inspection and walkthrough) and user-based empirical testing based on a think-aloud protocol. There were much fewer and mostly shorter courses on general usability topics.
Such courses for practitioners respond to the request for training
of practitioners in usability topics [
A notable exception to this limited amount of research is an
empirical study of training of software engineering students in a
language for describing and analysing user interface designs [
We have conducted an empirical study of a training course that is intended to teach software developers and designers to conduct usability evaluations. The aim of the study was to provide the participants with skills in formative usability evaluation.
We studied the training course in a first year university curriculum. The course included ten class meetings, cf. Table 1, each lasting four hours that was divided evenly between two hours of lecture, and two hours of exercises in smaller teams. The course required no specific skills in information technology which is the reason why class meeting number one and five included introductions to technological issues. The purpose of the exercises was to practice selected techniques from the lectures. In the first four class meetings, the exercises made the students conduct small usability pilot tests in order to train and practice their practical skills with selected methods. The exercises in the last six class meetings were devoted to conducting a realistic usability evaluation of a specified website.
The course introduced a number of methods for usability testing.
The first was the conventional method for user-based testing with
the think-aloud protocol [
The students were required to document their work by handing in a usability report. The instructors suggested to the students that the usability report should consist of 1) executive summary (1 page), 2) description of the usability evaluation method applied (2 pages), 3) results of the evaluation, primarily a list and detailed description of the identified usability problems for the website that was evaluated (5-6 pages), and 4) discussion of the method that was applied (1 page). The report would typically amount to around 10 pages of text. It was further emphasized that the problems identified should be categorized, at least in terms of major and minor usability problems. In addition, the report should include appendices with all data material produced such as logfiles, tasks assignments for test subjects, questionnaires etc. A prototypical example of a usability report was given to the students.
We chose www.hotmail.com as the website for our study. This website provides advanced interactive features and functionalities appropriate for an extensive usability test. Furthermore, it facilitates evaluations with both novice and expert test subjects due to its vast popularity. Finally, it has been used in other usability evaluations that have been published, which enabled us to compare the results of the student teams in our study with other result (this is further explained below under Data Analysis).
The participants were first-year university students enrolled in
four different studies at a faculty for natural sciences and
engineering. The first of the four studies was informatics, which is
a user-oriented IT education with focus on software development
but also with elements of design in general. The other three
studies were architecture and design, planning and environment,
and chartered surveyor, which all shared a focus on design in
general but also had elements of software development. All four
groups of students participated together in the course described in
this paper. None of the participants had any experience with
usability evaluation prior to the study.
36 teams involving a total of 234 students (87 females, 37%)
participated in the course and our study. Each team was required
to distribute the roles of test subjects, loggers, and test monitor
among themselves. This was done before the second class
meeting, well before they started the evaluation of the Hotmail
website. 129 (55%) of the students acted as test subjects, 69
(30%) as loggers, and 36 (15%) as test monitors, cf. [
Due to the pedagogical approach of the university, each team had their own office equipped with a personal computer and Internet access. Most teams conducted the tests in their office, while the rest did it in one of their homes. After the tests, the entire team worked together on the analysis and identification of usability problems and produced the usability report.
The student teams were required to apply the techniques
presented in the course. After the second class meeting, the test
monitor and loggers of each team received a two-page scenario
specifying the web-based mail service www.hotmail.com that
they should focus on in the usability evaluation. The scenario also
specified a comprehensive list of features that emphasized the
specific parts of www.hotmail.com they were supposed to
evaluate. The test monitor and the loggers examined the system,
designed tasks, and prepared the evaluation, cf. [
The main data collected in the study was the usability reports that were handed in by the teams. The 36 reports had an average length of 11.4 pages (SD=2.76) excluding the appendices, which had an average length of 9.14 pages (SD=5.02). 30 (83%) of the 36 teams provided information on task completion times for 107 (83%) of the 129 subjects, and they had an average session time (with one user) of 38.10 minutes (SD=15.32 minutes). We did not collect any data on the way the students performed during the evaluation, and we did not monitor or record how they carried out the evaluations.
All reports were analyzed, evaluated, and marked by the two authors of this paper according to the following three steps. Step 1. We designed a scheme for the evaluation of the 36 reports by analyzing, evaluating and marking five randomly selected reports out of the total of 36 reports. Through discussions and negotiations we came up with an evaluation scheme with 17 variables as illustrated in Table 3. The 17 variables were divided into the following three overall categories: evaluation (the way the evaluation was conducted), report (the presentation of the evaluation and the results), and results (the outcome of the usability evaluation). Finally, we described, defined, and illustrated all 17 variables in a two-page marking guide. Step 2. We worked individually and marked each of the 36 reports in terms of the 17 variables by using the marking guide. The markings were made on the following scale of 1 to 5: 1= wrong answer or no answer at all, 2=poor or imprecise answer, 3=average answer, 4=good answer, and 5=outstanding answer. We also counted the number of identified usability problems in each of the 36 usability reports. We defined a usability problem as something in the user interaction that prevents or delays users in realizing their objectives. Each time a report would described such an obstacle or delay, we would count that as a usability problem. Finally, we specified intervals for grading of the identification of usability problems based on their distribution on the following scale: 1=0-3 problems, 2=4-7 problems, 3=8-12 problems, 4=12-17 problems, and 5>17 problems.
Step 3. All reports and grades were compared and a final assessment on each variable was negotiated. In case of disagreements on a grade, we employed the following procedure: 1) if the difference was one grade, we would renegotiate the grade based upon our separate notes; 2) if the difference was two grades, we would reread and reassess the report together focusing only on the variable in question. For our study, no disagreement exceeded two grades. For each report, we also went through the set of usability problems that each of us thought they had identified. We negotiated each team’s list of usability problems until we had consensus on that as well.
To examine the overall performance of the students, we included
two additional sets of data in the study. Firstly, we compared the
student reports to usability reports produced by teams from
professional laboratories. These reports were selected from a pool
of usability reports produced in another research study where nine
usability laboratories received the same scenario as we used and
conducted similar usability tests of www.hotmail.com, cf.
[
The overall results show that the student teams did quite well. It is not surprising that the professionals did better on most variables. It was, however, surprising to us that on some variables, the students had a comparable performance and on a few variables they even performed better than the professional teams.
These three variables relate to the way the usability evaluation was conducted, see Table 4. On variable 1, conducting the evaluation, the professional teams have an average of 4.38 (SD=0.74). This is almost one grade higher than the student teams and a Mann-Whitney U Test shows strong significant difference between the student teams and the professional teams (z=-2.68, p=0.0074). On variable 2, task quality and relevance, the students performed slightly better than the professionals, but this difference is not significant (z=0.02, p=.984). No significant difference was found on variable 3, questionnaire/interviews quality and relevance (z=-1.63, p=0.1031).
These six variables relate to the quality of the usability report that was the tangible result of the usability evaluations, see Table 5. Report
Test description 3.03 (0.94) The student teams did not perform as well as the professionals on the description of the test, and this difference is significant (z=2.15, p=0.0316). On the other hand, the student teams actually performed significantly better than the professional teams on the quality of the data material in the appendices (z=2.07, p=0.0385). On the clarity of the usability problem list, we found a strong significant difference in favour of the professional teams (z=2.98, p=0.0029). There is also a significant difference on the teams’ executive summary, where the professionals are better (z=2.27, p=0.0232), and a strong significant difference on the clarity of the entire report (z=-3.15, p=0.0016). Finally, no significant difference was found for the layout of the report (z=-1.02, p=0.3077) although the number for the professional teams is slightly higher.
The pivotal result of the usability reports was the usability problems that were identified and the descriptions of them. There are eight variables on this category, see Table 6.
On the number of problems identified, the student and professional teams performed rather differently. The student teams were on average able to identify 7.9 usability problems (in the marking scale: Mean 2.56, SD 0.84) whereas the professional teams on average identified 21.0 usability problems (in the marking scale: Mean 4.13, SD 1.13). A Mann-Whitney U Test confirms strong significant difference between the student and professional teams on this variable (z=-3.09, p=0.002). It is, however, interesting that the professional teams actually performed very dissimilar on this variable, as they identified from 7 to 44 usability problems. Thus the professional team that identified the lowest number of usability problems actually performed worse than the average student team.
The professional teams performed better than the student teams on categorization of the usability problems that were identified, but the difference is not significant (z=-1.84, p=0.0658). On the practical relevance of the identified usability problems, the professional teams performed better, and this difference is significant (z=-2.56, p=0.0105).
On the overview of the qualitative results, the professional teams did significantly better than the students (z=-1.99, p=0.0466). On the other hand, the student teams provided better overview of the quantitative results, but this difference is not significant (z=0.90, p=0.3681).
There is no significant difference on the use of literature (z=-0.05, p=0.9601). The conclusions are better in the usability reports from the professional teams, and this difference is strong significant (z=-3.13, p=0.0017). No significance was found for the teams’ own evaluations of the test procedure they employed (z=-1.00, p=0.3173).
The strong differences between the student teams and the professionals in the production of results, e.g. the usability problem identified, made us conduct a more detailed analysis of potential causes.
A Spearman Rank Correlation shows a weak positive correlation between the way the evaluation was conducted and the number of identified usability problems, but this correlation is not significant (marking (r2=0.061, p>0.718), actual (r2=0.089, p>0.599)). The same can be concluded for the correlation between the quality and relevance of the tasks and the number of identified usability problems (marking (r2=0.239, p>0.157), actual (r2=0.235, p>0.165)). Thus, our study indicates that the student’s competence in planning and conducting a usability test does not necessarily influence the outcome of the evaluation in terms of the number of usability problems identified.
When looking at the corresponding variables for the professional teams, we find that there is a high correlation between the quality and relevance of the tasks and the number of identified usability problems for the professional teams and this correlation is significant (r2=0.741, p<0.05). Furthermore, a weak correlation exists between the way the evaluation was conducted and the number of identified usability problems, but this correlation is not significant (r2=0.336, p>0.374).
Introducing more test subjects in usability evaluations will usually (at least in theory) generate a higher number of identified usability problems. In our study, the average number of test subject was 3.6 (SD=0.65), ranging from one team using only two test subjects to one team using five test subjects. However, we found only a negligible positive correlation between the number of test subjects and the number of identified usability problems, as this correlation was not significant (marking (r2=0.247, p>0.143), actual (r2=0.238, p>0.159)). The test subjects had a rather varied experience with www.hotmail.com, but there is no significant correlation between the number of novice subjects and the number of identified problems (marking (r2=0.119, p>0.482), actual (r2=0.119, p>0.481)).
Correlations between the length of the tests and the number of identified usability problems for the 36 teams (grading and actual numbers) are illustrated in Figure 1. Considering the total time spent on all tests in each team, we identify a great variation ranging from 56 minutes to 225 minutes (mean=113.26 minutes, 5 4 3 2 1 5 4 3 2 1 0 50 100 150 200
250
Total minutes
As a complementary perspective, we analyzed the basic skills of the students and their performances in other university activities in the same semester. We examined the correlation between the combined grade obtained by each of the 36 teams (based on the individual grades of team members) in other major coursework and the number of identified usability problems.
Practical relevance of problems No. of problems (actual) SD=65.59 minutes). A minor correlation exists between the total time spent on the test and the number of identified problems, but the correlation is not significant (r2=0.280, p>0.098). This is also the case when looking at the actual number of problems against time spent (r2=0.329, p>0.051). This correlation is, however, close to being significant. The grade is reported on a scale from zero (not satisfactory) to nine (outstanding). A Spearman Rank Correlation Test shows only a slight positive correlation between the grade of the students and the number of identified usability problems (marking (r2=0.103 p>0.542), actual (r2=0.130, p>0.441)). This correlation between grades and identified number of usability problems is illustrated in Figure 2.
As emphasized in the introduction, several studies have found that
many websites suffer from low usability [
One of our key findings concerns identification and categorization
of usability problems. The student teams identified significantly
fewer problems than the professional teams. On average, the
student teams found 7.9 usability problems, whereas the
professional teams on average found 21 usability problems. This
difference is important since uncovering of usability problems is a
key purpose of a formative usability evaluation. The student
teams did, however, perform rather differently on this variable.
One student team identified no problems at all. This team might
have misunderstood the assignment, but we cannot tell from their
usability report, which was the basis for our analysis. The best
performing students were two teams that identified 16 problems.
Most of the student teams identified no more than 10 problems.
The professional teams also performed rather differently. It has
been shown before that usability evaluators find different
problems; this has been denoted as the evaluator effect [
Another reason for the differences between student teams and professionals in identifying and describing usability problems may be the specific design of the training course. We might have focused too little on discussing the nature of a usability problem and provided too few examples. We could also have treated this in more detail by presenting specific examples of relevant and irrelevant problems. Our analysis of the reports from the student teams clearly suggests that this topic received too little attention.
This article has presented the results from a study of a course that was employed to train software developers and designers in conducting usability evaluations of a website. The idea behind this effort was that if developers can conduct their own usability evaluations, the gap between usability evaluation and software design will disappear.
The course was based on a simple approach to usability testing that quickly teaches fundamental usability skills. Whether this approach is effective has been explored through a large empirical study where 36 student teams from the first year of software development and design-oriented educations were trained in and applied the approach to evaluate the usability of the Hotmail website.
The overall conclusion is that the student teams were able to conduct usability evaluations and produce usability reports of a reasonable quality and with relevant results. However, when compared to professional evaluator teams, there were clear differences. The student teams performed well in defining good tasks for the test subjects, and the data material in their reports was significantly better than the professionals. They were less successful on several of the other variables, and they performed clearly worse when it came to the identification of problems, which is a main purpose of a usability test. It was also difficult for them to express the problems found in a manner that would be relevant to a software developer working in practice.
Time pressure is a key reason why established knowledge and
methodologies are ignored in many website development projects
[
Our study is limited in a number of ways. First, the environment in which the evaluations were conducted was in many ways not optimal for the usability test sessions. In some cases, the students were faced with slow Internet access that might have influenced the results. Second, motivation and stress factors could prove important in this study. None of the teams volunteered for the course and the study, and none of them received any payment or other kind of compensation. All teams participated in the course because it was a mandatory part of their curriculum, but they did not have to pass an exam in the course itself. This implies that students did not have the same kinds of incentives for conducting the usability test sessions as the evaluators from the professional usability laboratories. Thirdly, the demographics of the test subjects are not varied with respect to age and education. Most test subjects were approximately 21 years of age with approximately the same school background and recently started on an IT or design-oriented education.
The use of university students as a substitute for real software developers and designers working in practice has often, and rightly, been criticized. Yet in this case, it is less questionable. With a group of software developers from practice, it would be difficult to distinguish between their experience and the effect of the training course. With students who have basic knowledge about software development but no practical experience, that empirical problem vanishes.
Having said that, it could still be very interesting to conduct a similar study with real website developers and designers. This might be combined with a longitudinal study of the long-term effect on the quality of the websites developed. The main shortcoming that came up in our analysis was the students’ lack of skill in identifying and describing usability problems. A different study could be based on a training course that was changed to focus directly on identification of usability problems.
We would like to thank the students for their participation in the study.