We propose two metrics to demonstrate the impact integrating human-computer interaction (HCI) activities in software engineering (SE) processes. User experience metric (UXM) is a product metric that measures the subjective and ephemeral notion of the user's experience with a product. Index of integration (IoI) is a process metric that measures how integrated the HCI activities were with the SE process. Both metrics have an organizational perspective and can be applied to a wide range of products and projects. Attempt was made to keep the metrics light-weight. While the main motivation behind proposing the two metrics was to establish a correlation between them and thereby demonstrate the effectiveness of the process, several other applications are emerging. The two metrics were evaluated with three industry projects and reviewed by four faculty members from a university and modified based on the feedback.
Large contracted software development companies with tens of thousands of employees are often involved in a wide variety of software development projects, often in an off-shore mode. Managers of user experience (UX) groups in such companies need to track progress of each project and ensure the quality of deliverables. They are often required to juggle across projects a limited resource – the time of their best UX professionals. While Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
Another challenge faced by UX groups is integrating HCI in
established SE processes. The field of HCI has a large amount of
literature on user-centred design methods, techniques and
processes [
Contracted software companies often promise a level of process compliance to their clients. UX managers need summary measures of process compliance of their projects to ensure that the company lives up to its promise. One way would be to measure how integrated were the HCI activities with SE processes. Our second proposal is a process metric (IoI) that would be one such measure. If validated, IoI and UXM can also be used to demonstrate the return on investment on integration of HCI with SE – if higher IoI consistently leads to higher UXM, it makes sense to invest in better integration of HCI with SE.
The main objective of this paper is to share with other metrics researchers the lessons we have learned from attempting to incorporate UXM and IoI in live industrial projects.
We begin with an introduction to different attempts done in recent years on applying metrics in HCI. Next, our metrics proposals are described. Finally, the evaluation methodology used so far to analyse the results of study is described.
Metrics are thoroughly discussed in software engineering
literature. Fenton and Pfleeger [
Though the word ‘metric’ itself is seldom used in the practice of usability, several measures are often used. Seconds taken to withdraw money from the ATM, the number of keystrokes to enter a word in a complex script, the number of errors made to complete a banking transaction or the percent of users who abandon the shopping cart on checkout are all examples of quantitative measures of the user experience afforded by the product. However, none of these are summary measures that can be used for apple-to-apple comparison across projects varying in domains, platforms and contexts. While several research papers talk about metrics related to usability and HCI, this paper only focuses on those that give a summary measure.
Lewis [
Lin et al [
Sauro et al [
Measuring the wider notion of user experience (as opposed to
usability) is relatively new concept in HCI and is attracting
attention of the academic as well as the industrial world. Usability
parameters are typically related to the processing of information
or completion of tasks. However, affective reactions and
emotional consequences play important role in the overall user
experience [
None of the summary metrics mentioned above measure the experience of a product with reference to all user and business goals relevant to a product. Many are too complex to compute practically on an on-going basis in the industrial practice. They lack the flexibility required to serve the needs of a wide variety of projects or to mature with the UX group. And finally, there seems to be almost no work on measuring integration of HCI activities with SE processes.
Fenton and Pfleeger [
We propose user experience metric (UXM), a product metric that measures the quality of user experience. The motivations are: • to measure the user experience of a product in reference to its user experience goals • to develop a flexible metric that can be applied across a variety of projects, irrespective of domain, context, platform, process model or usability technique • to develop a flexible metric that that will mature with the organization • and to compute the metric with minimal additional costs and efforts.
UXM is product metric on a scale of 0-100, where 100 represents the best user experience possible and 0 represents the worst.
Goals: High-level user experience goals guide the design of interactive systems.
Parameters: Each high-level user experience goal is broken down into a set of parameters that help the designer to achieve and measure the higher-level goal in a direct manner. For example Learnability can have parameters like Conceptual model clarity, Language understandability, Minimal training time, Consistency with earlier version etc.
Weightage: Each goal has a weightage between 0-5 where 0 represents that the goal is not important, 2 represents the typical importance and 5 represents that it is very important. Further, each parameter under a goal also has a weightage attached. Score: Each parameter has a score between 0-100, where 0 represents the worst possible user experience on account of that parameter and 100 represents the best possible user experience. Guidelines: The purpose of the guidelines is to help evaluators assign a score to the parameters. Guidelines let the goal-setters express themselves better and interpret goals for the context of a project – e.g. “‘Consistency with earlier version’ means all frequent and critical tasks from earlier version are unchanged.” Further, guidelines tell the evaluators when to assign which score: “The interface clearly communicates the correct conceptual model. Strongly agree = 100, Weakly agree = 75, Neutral = 50, Weakly disagree = 25 and Strongly disagree = 0”.
Goals and parameters are a way to express the desired user
experience and performance in the design. Though expressing
user experience goals is a common activity in HCI design, there is
no standard way of doing so. There are many ways to describe the
high level user experience goals. For example, ISO 9126-1
describes usability in terms of understandability, learnability,
operability and attractiveness [
We adopt goals from Shneiderman as the high-level user experience goals for a product and express them as: learnability, ease of use, speed of use, error-free use, retention and subjective satisfaction. We added ‘ease of use’ to the list as we thought that it is an important user experience goal distinctly different from the other factors such as speed. We also generalized the expressions of some of the goals. For example we express ‘time to learn’ as ‘learnability’ since it allows for expression of other concerns such as understandability of language or clarity with which the interface communicates the conceptual model in addition to the time users take to learn the interface. We believe that this allows the designers to express a wider set of goals. Our proposal of an initial set of goals and parameters are listed in Table 1. However, we must highlight that this is not a prescribed, exclusive set. We give the evaluators and stakeholders freedom to derive additional, relevant parameters that express their goals. Goals and parameters could be added, removed or combined according to the context of the project, the needs of the users, the vision of the stakeholders and UX professionals and to fit the terminology that the product development team is familiar and comfortable with. The initial list is meant to give users a starting point, while the flexibility is meant to allow the metric to mature with the experience of the organization using UXM.
As Shneiderman [
The process for computing UXM for a product has these steps: Goal Setting: Early in the project, typically just after user studies but before design, an HCI professional and stakeholders identify goals and parameters for each goal, assign weightage to each goal and parameter and decide evaluation guidelines for the parameters.
Scoring: Immediately after a usability evaluation, one or more independent HCI professionals assign a score to each parameter of each goal. The usability evaluation could be either user-based (e.g. a usability test) or review-based (e.g. heuristic evaluation). UXM Calculation: UXM is the sum of the weighted average of the scores of all goals. UXM = ∑ ( Wg x Sg / ∑ Wg ), where Wg is the weightage of a goal and Sg = ∑ ( Wp x Sp / ∑ Wp ) where Wp is the weightage of a parameter and Sp is the score of that parameter. Scores of some of the parameters can be directly linked to the findings of the usability evaluations (for example, % of users who did not make errors while doing benchmark tasks, or % of users who thought the product was engaging). Other parameters may not be so easily linked numerically (e.g. conceptual model confusions discovered during a think aloud test or problems identified during heuristic evaluation). In such cases, evaluators consider the guidelines and their own experience to arrive at a score for each parameter. If there are multiple evaluators, a simple average across evaluators is deemed to be the score for a given parameter. Multiple evaluators assign scores independently to begin with. If there is a significant variation in their scores, the evaluators discuss the parameter and have the opportunity to converge their scores before the average is calculated. In case of applications with multiple user profiles, separate UXM should be calculated for each profile. Calculation of UXM could be a part of every usability evaluation of the project, but we recommend that it should certainly be a part of the final usability evaluation, beyond which no design changes are planned for. column of the upper part of Table 1). Next, they broke down goals into parameters and assigned them weightages (shown in the second column of the lower part of Table 1). The team’s experience from previous Indic text input projects and mobile phone projects helped them arrive at these weightages. The team then evaluated the interface and assigned scores to each parameter (shown in the third column of the lower part of Table 1). A weighted average of parameter scores gave the score for each goal (shown in the light grey cells of the third column of lower part of Table 1). A weighted average of the goal scores gave the UXM value (shown in the dark grey cells of the upper part of Table 1). Parameter evaluation guidelines have not been listed in this paper due to space constraints.
We conceive Index of Integration (IoI) as an empirical process metric, nominally on a scale of 0-100, where 100 represents the best possible integration of HCI activities in the software development activities and 0 represents the worst. The metric consists of these distinctions:
described in a software engineering process model.
HCI Activities: These are prescribed for each phase of the software engineering process model.
Weightage: Each HCI activity will have a given weightage on the scale of 0-5 where 0 represents that the activity is not important, 3 reflects the typical importance in most projects and 5 indicates that this activity is very important in the context of that project. Score: Each activity has a score associated with it. The score is given on a rating of 0-100, where 100 represents the best case situation where the activity was done in the best possible manner, in the most appropriate phase of software development and with the best possible deliverables. 0 represents the worst case situation where the activity was not done at all.
that help the evaluation of each activity.
Software engineering phases have been extensively described in
literature. For example, the phases of the waterfall process model
are Communication, Planning, Modelling, Construction and
Deployment [
On the other hand, no widely accepted industry-wide
specifications of HCI activities for given SE phases have emerged
so far. But there have been a few proposals. For example, [
Communication Project initiation
User studies
Ideation Product definition
Evaluation and refinement Requirements ifPilanining
Estimating Scheduling Tracking
Modelling Detailed UI prototyping Usability evaluation Requirements anCaolnysstirsuction
Code
Test Usability evaluation
Deployment
Delivery Support
Feedback
Weightage of some HCI activities could vary within a range in the context of a project. For example, if the domain or users are unknown to the UX team, it may be very important to do contextual user studies in the communication phase (weightage = 4). On the other hand, if the UX team is already very familiar with the context and the domain and if they have a lot of experience designing similar products, it may be less important (weightage = 2). Table 2 summarises loosely recommended weightages for HCI activities for the waterfall model. Guidelines may define the techniques used to carry out activities, the skills and experience levels of the people doing the activities, the deliverables and other parameters that affect the fidelity of the activity. For example, following are the guidelines for the activity of Contextual user studies and user modelling in Table 2:
Organizational data gathering and user studies were done before requirements were finalised User studies were done in the context of the users by the method of contextual inquiry User studies were done with at least 10 users in each profile User studies were done by people with experience in user studies in a similar domain in at least 2 projects The findings including user problems, goals, opportunities and constraints were analyzed, documented and presented in 6. 7.
an established user modelling methodology such as personas, work models, affinity diagram or similar Competitive products and earlier versions of the product were evaluated for potential usability problems by using discount usability evaluation methods such as Heuristic Evaluation or better User experience goals were explicitly agreed upon before finalizing requirements 100 = All the above are true; 75 = At least five of the above are true, including 7, 50 = At least three of the above are true, including 7; 25 = At least two of the above are true, 0 = None of the above are true
Refinement of the Prototype
2 3 1 5 4 3 1 39 6 75 63 53 44 29 46 45 The process for computing IoI for a project has these steps: Company HCI Process Prescription: The HCI group in the company prescribes what HCI activities should be done in which phase of SE process, expected deliverables from each activity, suggested weightages for each activity and suggested activity evaluation guidelines. As it often happens, a contract software development company may follow not one SE process, but several. In that case the HCI design process needs to be integrated with each SE process. The prescribed process also suggests a weightage for each HCI activity and guidelines to score each activity.
Project HCI Process Definition: After getting a project brief, the UX professional fine-tunes the weightages for the prescribed HCI activities after considering the domain, the users and project context. For example, if the HCI team has recently done contextual user studies in the same domain for a similar product and is already very knowledgeable about the context, then he may reduce the weightage of contextual user studies. On the other hand if the team is less knowledgeable, he may increase the weightage. He should consult colleagues in the development team and business stakeholders before finalizing the weightage. Process Evaluation: After the project is over, a group of independent UX professionals review the HCI activities and evaluate them for process compliance and give a score for each activity on a scale of 0 to 100. They may reduce the score if an activity was done with lower fidelity, resulted in poor quality deliverables or was done later than prescribed. In case of multiple evaluators, an average across evaluators is deemed to be the score.
IoI Calculation: The metric is found by computing the weighted average of the scores of all activities: IoI = ∑ ( Wa x Sa / ∑ Wa ), where Wa is the weightage for a particular HCI activity, Sa is the score (from 0-100) for that activity. In case there is a lot of divergence in scores of a particular HCI activity, the activity is discussed and reviewers are given a chance to change their score before an average is taken.
Table 2 shows calculation of IoI for an example project. First, senior UX professionals defined the HCI activities, activity weightages and evaluation guidelines that the company should be following. Then a project that had recently ended was selected for retrospective review. The project manager and the UX professionals working on the project fine-tuned the weightages for the project context. The second column of Table 2 contains these weightages. A group of reviewers comprising of some project insiders and outsiders reviewed and rated the HCI activities in the project and its IoI was calculated. The third column of Table 2 contains the average scores assigned to each activity by the reviewers.
Guidelines for evaluating all HCI activities listed in Table 2 have been created. More guidelines for evaluating HCI activities as part of extreme programming process have been created as well. Both have been omitted here due to space constraints.
The authors evaluated the metric in two ways. First, UXM and IoI metrics were computed for retrospectively projects in Tech Mahindra, a large contracted software development company. In each case, the metrics computation was done by HCI professionals from the project, independent HCI professionals and project stakeholders. At the end of metrics computation, feedback was taken from participants of each project about the metrics. Second, the metrics were presented to a group of faculty members from a reputed university and their comments were noted. Three of these were faculty members from the computer science discipline. One was a faculty member from a design school who is also an expert in cognitive psychology.
It typically took about 3 hours to compute both IoI and UXM for each project. The time included explaining the two metrics, weightage assignment and scoring. This seemed to be optimum time, longer meetings were difficult to schedule. The projects performed similarly in IoI and UXM scores – the one project that had a high UXM value also had a high IoI value. Participants, particularly project stakeholders, were at home with the activity of metric calculation. To them, the activity seemed to bring HCI closer to SE. It seemed to create lot of buy-in for HCI activities from the project stakeholders. One project stakeholder said “I never thought we could think so much [about user experience].” The activity seemed to be more successful in projects where several stakeholders from the project participated as it stimulated discussion among stakeholders. While the participants appreciated the organizational perspective, the metrics seemed of less use to the projects as the projects were already over. Participants suggested that metrics should be calculated mid-way through the project while course correction was still possible.
Specifically, UXM helped the HCI designers and project stakeholders to make goals explicit. One HCI designer remarked, “Had we done this earlier, I would have known where to focus.” The teams usually added a few goal parameters (typically 2-3 per project) and adjusted weightage to suit UXM to their project. They confirmed that this flexibility is indeed desirable. Though parameter evaluation guidelines for UXM helped, more details were desired. Participants did not make changes to the parameter evaluation guidelines except when new parameters were added. Giving examples of HCI goals (learnability, ease of use etc.) helped participants to set goal parameters and weightages. One stakeholder remarked: “without these inputs it would have been difficult to [assign weightage and scores].” In case of a few UXM parameters, divergent scores emerged for some parameters in each project. Usually variations happened in parameters where the evaluation guidelines were not understood well or were interpreted differently by evaluators. In such cases, it was felt, that it was better to let participants discuss the parameter and change ratings to converge scores if they so desire. Reducing the number of steps in scoring a parameter (e.g. 0-25-50-75-100) helped reduce variation among scores. More detailed UXM parameter evaluation guidelines with examples will further help in reducing divergence.
Computing IoI was useful for project stakeholders as they could see the importance of HCI activities in the SE context. The HCI activities integrated in SE process models were acceptable as suggested. Though they were explicitly prompted, none of the project stakeholders wanted changes to the prescribed HCI activities, their weightage or evaluation guidelines. An important feedback was need for process models specifically targeted to redesign projects. Process models typically discuss new product development. Given that many industry projects are “next version of X” type, process models must be specifically adapted for them. Walking through the activity evaluation guidelines helped in scoring as all stakeholders were not aware of all HCI activities. It was felt that IoI should be computed before computing UXM as this minimizes bias.
The metric descriptions presented in this paper are a result of iterative modifications that reflect the feedback and lessons learnt.
It is important to discuss the limitations and risks of the two metrics proposed. Both UXM and IoI are summary measures that leave out much information. They allow a drill-down to constituent components, but do not point to specific problems or give suggestions for improvement. But summary measures are useful in many contexts, particularly for comparison across projects. Such comparisons can help UX groups understand what works and what doesn’t and improve performance year-on-year. Perhaps most important limitation of UXM comes from the ephemeral nature of a ‘user experience’. Any attempt to numerically embody such an abstract phenomenon is bound to be subjective and measures could differ according to the interpretation of the evaluators. Further, large companies are involved in software development projects for many clients, across domains, platforms, users, use contexts and task complexity, frequency and criticality.
Finally, there is a risk that because UXM measures are low cost, organizations may be tempted to sacrifice all user-facing activities (such as usability tests or field studies) in its favour. We do not recommend this at all. The purpose of UXM is not to replace these established methods but to supplement them and to help them mature.
In spite of these limitations, we believe that UXM is useful. UXM shows the extent to which user experience goals were achieved in a particular project. We found that breaking up abstract notions of user experience into specific goals and parameters helped evaluators focus on one issue at a time and reduced the subjectivity in measurement. Making the evaluation criteria explicit and averaging across several evaluators further reduced the subjectivity in judgement. The risk of variety in products (the apples-and-oranges risk) was partly mitigated by selecting goalparameters relevant to each project and giving custom weightage to each parameter.
The main limitation of IoI is that it does not measure the absolute process quality of the project, rather how compliant was a project to the prescribed process. There are no widely-accepted integrated process models at this stage. Yet, IoI in conjunction with UXM may be used to verify the effectiveness of new process model proposals. If UXM and IoI are correlated, the new proposal seems acceptable. On the other hand if the UXM and IoI do not show a correlation, it questions the prescribed process model. UXM and IoI have an organizational perspective and make more sense while looking across hundreds of projects rather than within each project individually. They have very low additional overheads on the process and are easy to integrate in the process. Overall feedback indicates that UXM and IoI are useful and practical in evaluating products and processes. There was a lot of buy-in from project stakeholders calculating metrics as there was a lot of willingness to track, control the user experience of the product. The aspects that metric calculation was light-weight and independent of specific usability methods were particularly liked.
In future, we plan to use metrics prospectively throughout the duration of projects and demonstrate their usefulness during the project. We will be building more elaborate tools and guidelines to improve the consistency of weightages and scores. We also propose to do a rigorous validation of the two metrics in experimental and industrial situations.
In its current form, UXM goals, parameters and weightages have to be chosen on the basis of experience of individuals. However, it is possible to design tools in future that will collate experience of several practitioners to help in choices of future goals, parameters and weightages. A similar tool for IoI can also evolve the specification of processes.
We thank the anonymous reviewers for invaluable comments on improving our presentation of this material. In particular, we thank Ved Prakash Nirbhay and Deepak Korpal from Tech Mahindra for allowing us to interact with their team members while testing the metrics in different software projects. We also thank members of User Interaction Design Group at Tech Mahindra Ltd. for participating in User Experience metrics evaluation. We also thank Pramod Khambete for his continuous support and appreciation. We thank Prof. NL Sarda, Prof. UA Athavankar, Prof. Umesh Bellur and Prof. S Sudarshan for their continuing guidance and suggestions in developing the two metrics.