User Experience Metric and Index of Integration: Measuring Impact of HCI Activities on User Experience Anirudha Joshi Sanjay Tripathi Indian Institute of Technology Bombay Tech Mahindra Ltd. Mumbai 400076, India Pune 411004, India +91 9820345569 +91 9922963298 anirudha@iitb.ac.in stripathi@techmahindra.com ABSTRACT there are numerous usability metrics to evaluate specific projects, We propose two metrics to demonstrate the impact integrating there are few that allow organizations to easily track progress human-computer interaction (HCI) activities in software across projects. Our first proposal is product metric (UXM) that engineering (SE) processes. User experience metric (UXM) is a measures the user’s experience with a product. The objective is to product metric that measures the subjective and ephemeral notion provide a summary measure of the user experience of a product of the user’s experience with a product. Index of integration (IoI) that is independent of the domain, context of use, platform or the is a process metric that measures how integrated the HCI software development process, so that the manager is able to activities were with the SE process. Both metrics have an make judgments across projects. organizational perspective and can be applied to a wide range of Another challenge faced by UX groups is integrating HCI in products and projects. Attempt was made to keep the metrics established SE processes. The field of HCI has a large amount of light-weight. While the main motivation behind proposing the two literature on user-centred design methods, techniques and metrics was to establish a correlation between them and thereby processes [1], [3], [17], [23] etc. These proposals are excellent demonstrate the effectiveness of the process, several other demonstrations of how user centred design can result in improved applications are emerging. The two metrics were evaluated with user experience design. Unfortunately, there continue to exist three industry projects and reviewed by four faculty members major gaps between HCI and SE, in academics, literature and from a university and modified based on the feedback. industrial practice. The IFIP working group 2.7/13.4 on User Interface Engineering remarks that ‘there are major gaps of Categories and Subject Descriptors communication between the HCI and SE fields: the architectures, D.2.8 [Software Engineering]: Metrics – process metrics, processes, methods and vocabulary being used in each community product metrics. are often foreign to the other community’ [7]. For example, while SE literature admits that communication with the customer is an unsolved problem, even recent editions of standard text books on General Terms software engineering such as [13] and [20] do not suggest use of Measurement, Design, Human Factors established user study techniques like [1] during communication. Example projects shown in [13] and [20] seem to take HCI design Keywords lightly, prematurely and without following any process. A User experience metrics, HCI-SE integration. detailed critique of SE literature from an HCI perspective is presented in [11]. There have been several proposals to integrate HCI in SE process models (for example, [5], [12], [21]) but none have become popular in the industry. One reason could be 1. INTRODUCTION concerns about return on investments. Though there is plenty of Large contracted software development companies with tens of evidence of the a return on investment of usability activities in thousands of employees are often involved in a wide variety of general [2], there is no direct evidence that shows that better software development projects, often in an off-shore mode. integration of HCI activities in SE processes will lead to better Managers of user experience (UX) groups in such companies products at less cost. need to track progress of each project and ensure the quality of Contracted software companies often promise a level of process deliverables. They are often required to juggle across projects a compliance to their clients. UX managers need summary limited resource – the time of their best UX professionals. While measures of process compliance of their projects to ensure that the company lives up to its promise. One way would be to measure Permission to make digital or hard copies of all or part of this work for how integrated were the HCI activities with SE processes. Our personal or classroom use is granted without fee provided that copies are second proposal is a process metric (IoI) that would be one such not made or distributed for profit or commercial advantage and that measure. If validated, IoI and UXM can also be used to copies bear this notice and the full citation on the first page. To copy demonstrate the return on investment on integration of HCI with otherwise, or republish, to post on servers or to redistribute to lists, SE – if higher IoI consistently leads to higher UXM, it makes requires prior specific permission and/or a fee. sense to invest in better integration of HCI with SE. I-USED’08, September 24, 2008, Pisa, Italy The main objective of this paper is to share with other metrics and do not leave room for project-specific goals (e.g. “do it right researchers the lessons we have learned from attempting to the first time”). incorporate UXM and IoI in live industrial projects. Sauro et al [22] proposed a ‘single, standardized and summated’ We begin with an introduction to different attempts done in recent usability metric for each task by averaging together four years on applying metrics in HCI. Next, our metrics proposals are standardized values for task time, errors, completion and described. Finally, the evaluation methodology used so far to satisfaction. Their calculation however is based on the equal analyse the results of study is described. weightage. Tasks, domains, users, contexts and platforms vary a lot and it does not make sense to give equal weightage in all contexts. Moreover, the metric ignores some aspects such as 2. METRICS IN HCI learnability and ease of use, which might be important in some Metrics are thoroughly discussed in software engineering contexts. literature. Fenton and Pfleeger [4] describe measurement as “the process by which numbers or symbols are assigned to attributes of Measuring the wider notion of user experience (as opposed to entities in the real world in such a way as to describe them usability) is relatively new concept in HCI and is attracting according to clearly defined rules”. Pressman [20] highlights the attention of the academic as well as the industrial world. Usability subtle difference between measurement and metrics – parameters are typically related to the processing of information measurement occurs as the result of collection of one or more data or completion of tasks. However, affective reactions and points, while a software metric tries to relate the measures in emotional consequences play important role in the overall user some way. IEEE Standard Glossary [6] defines a metric as “a experience [16]. In some product contexts, we may need to quantitative measure of the degree to which a system, component consider visceral, behavioural and reflective elements [19], or process possesses a given attribute”. aesthetics [25], enjoyment [10] and creativity [24]. Though the word ‘metric’ itself is seldom used in the practice of None of the summary metrics mentioned above measure the usability, several measures are often used. Seconds taken to experience of a product with reference to all user and business withdraw money from the ATM, the number of keystrokes to goals relevant to a product. Many are too complex to compute enter a word in a complex script, the number of errors made to practically on an on-going basis in the industrial practice. They complete a banking transaction or the percent of users who lack the flexibility required to serve the needs of a wide variety of abandon the shopping cart on checkout are all examples of projects or to mature with the UX group. And finally, there seems quantitative measures of the user experience afforded by the to be almost no work on measuring integration of HCI activities product. However, none of these are summary measures that can with SE processes. be used for apple-to-apple comparison across projects varying in domains, platforms and contexts. While several research papers 3. USER EXPERIENCE METRIC talk about metrics related to usability and HCI, this paper only Fenton and Pfleeger [4] emphasise the importance of goals in focuses on those that give a summary measure. metrics: “a measurement program can be more successful if it is Lewis [14] used a rank based system of assessing competing designed with the goals of the project in mind”. User experience products based on user’s objective performance measure and goals are very important in driving the design of interactive subjective assessment. The metric is useful for a relative products. They help speed up the design process, make the design comparison between like products with similar tasks but it does activity more tangible and help evaluate the design. User not result in a measure that can be used across unlike products. experience goals can be understood easily, even by non-UX- professionals, and they have a significant overlap with business McGee [18] derives a single usability scale across users by goals. Stakeholders outline the user experience goals and UX including additional reference tasks. However McGee does not professionals fine-tune them on the basis of their knowledge and suggest how to derive a single measure for usability from findings from user studies. User experience goals are (and should measures for the different tasks. Further, this work is completely be) available early in a project – another plus when it comes to dependent on the technique of usability evaluation. This is not metric calculation in a practical situation. always practical in a global contracted software company striving to move up the HCI maturity ladder. The other limitation of this We propose user experience metric (UXM), a product metric that method is that it relies only on perception of users and ignores measures the quality of user experience. The motivations are: perspectives of other stakeholders, particularly the goals of • to measure the user experience of a product in reference business stakeholders. to its user experience goals Lin et al [15] propose the Purdue Usability Testing Questionnaire • to develop a flexible metric that can be applied across a based on eight HCI considerations to derive a single weighted variety of projects, irrespective of domain, context, average score for usability. While the approach does lead to a platform, process model or usability technique single usability score, the selected eight considerations • to develop a flexible metric that that will mature with the (compatibility, consistency, flexibility, learnability, minimal organization action, minimal memory load, perceptual limitation and user • and to compute the metric with minimal additional costs guidance) seem to be a mix of usability goals and heuristics that and efforts. achieve those goals. Secondly, the weightage for parameters is to UXM is product metric on a scale of 0-100, where 100 represents be assigned by the evaluator during the evaluation without the best user experience possible and 0 represents the worst. consulting stakeholders. Thirdly, the listed eight considerations UXM consists of these distinctions: and the questions listed under each of them seem to be limiting Goals: High-level user experience goals guide the design of terminology that the product development team is familiar and interactive systems. comfortable with. The initial list is meant to give users a starting Parameters: Each high-level user experience goal is broken point, while the flexibility is meant to allow the metric to mature down into a set of parameters that help the designer to achieve with the experience of the organization using UXM. and measure the higher-level goal in a direct manner. For example As Shneiderman [23] states, ‘a clever design for one community Learnability can have parameters like Conceptual model clarity, of users may be inappropriate for another community’ and also, Language understandability, Minimal training time, Consistency ‘an efficient design for one class of tasks may be inefficient for with earlier version etc. another class’. Weightages express the relative importance of goals and parameters in the context of a project. For example, a Weightage: Each goal has a weightage between 0-5 where 0 product meant to be used several times a day by a call-centre represents that the goal is not important, 2 represents the typical agent is likely to have higher weightage for ‘speed of use’. A one- importance and 5 represents that it is very important. Further, time use product like a web site for visa application for a tourist each parameter under a goal also has a weightage attached. might insist on learnability and error-free use. On the other hand, Score: Each parameter has a score between 0-100, where 0 a life-critical product to be used in a operation theatre is likely to represents the worst possible user experience on account of that rate highly error-free use and may sacrifice learnability. A game parameter and 100 represents the best possible user experience. would perhaps give highest weightage to ‘subjective satisfaction’. Guidelines: The purpose of the guidelines is to help evaluators The evaluators and stakeholders assign the weightage to set the assign a score to the parameters. Guidelines let the goal-setters context of the project. Goal-setters should be aware that while it express themselves better and interpret goals for the context of a may be tempting to set a high weightage to each goal, it may not project – e.g. “‘Consistency with earlier version’ means all be necessary, practical, or even possible to achieve such a design. frequent and critical tasks from earlier version are unchanged.” The weightages should reflect the priorities of the stakeholders Further, guidelines tell the evaluators when to assign which score: and users. The weightage would also help prioritize usability “The interface clearly communicates the correct conceptual evaluation activity – the highest rated goals and parameters must model. Strongly agree = 100, Weakly agree = 75, Neutral = 50, be evaluated more thoroughly, while the lower weighted goals Weakly disagree = 25 and Strongly disagree = 0”. could be perhaps evaluated by a discount method. The process for computing UXM for a product has these steps: Goals and parameters are a way to express the desired user Goal Setting: Early in the project, typically just after user studies experience and performance in the design. Though expressing but before design, an HCI professional and stakeholders identify user experience goals is a common activity in HCI design, there is goals and parameters for each goal, assign weightage to each goal no standard way of doing so. There are many ways to describe the and parameter and decide evaluation guidelines for the high level user experience goals. For example, ISO 9126-1 parameters. describes usability in terms of understandability, learnability, Scoring: Immediately after a usability evaluation, one or more operability and attractiveness [8]. ISO 9241 on the other hand independent HCI professionals assign a score to each parameter defines usability as the extent to which a product can be used by of each goal. The usability evaluation could be either user-based specified users to achieve specified goals with effectiveness, (e.g. a usability test) or review-based (e.g. heuristic evaluation). efficiency and satisfaction in a specified context of use [9]. Shneiderman [23] describes goals for user interface design in UXM Calculation: UXM is the sum of the weighted average of terms of five human factors central to evaluation: time to learn, the scores of all goals. UXM = ∑ ( Wg x Sg / ∑ Wg ), where Wg is speed of performance, rate of errors by users, retention over time the weightage of a goal and Sg = ∑ ( Wp x Sp / ∑ Wp ) where Wp is and subjective satisfaction. the weightage of a parameter and Sp is the score of that parameter. We adopt goals from Shneiderman as the high-level user Scores of some of the parameters can be directly linked to the experience goals for a product and express them as: learnability, findings of the usability evaluations (for example, % of users who ease of use, speed of use, error-free use, retention and subjective did not make errors while doing benchmark tasks, or % of users satisfaction. We added ‘ease of use’ to the list as we thought that who thought the product was engaging). Other parameters may it is an important user experience goal distinctly different from not be so easily linked numerically (e.g. conceptual model the other factors such as speed. We also generalized the confusions discovered during a think aloud test or problems expressions of some of the goals. For example we express ‘time to identified during heuristic evaluation). In such cases, evaluators learn’ as ‘learnability’ since it allows for expression of other consider the guidelines and their own experience to arrive at a concerns such as understandability of language or clarity with score for each parameter. If there are multiple evaluators, a simple which the interface communicates the conceptual model in average across evaluators is deemed to be the score for a given addition to the time users take to learn the interface. We believe parameter. Multiple evaluators assign scores independently to that this allows the designers to express a wider set of goals. begin with. If there is a significant variation in their scores, the Our proposal of an initial set of goals and parameters are listed in evaluators discuss the parameter and have the opportunity to Table 1. However, we must highlight that this is not a prescribed, converge their scores before the average is calculated. exclusive set. We give the evaluators and stakeholders freedom to In case of applications with multiple user profiles, separate UXM derive additional, relevant parameters that express their goals. should be calculated for each profile. Calculation of UXM could Goals and parameters could be added, removed or combined be a part of every usability evaluation of the project, but we according to the context of the project, the needs of the users, the recommend that it should certainly be a part of the final usability vision of the stakeholders and UX professionals and to fit the evaluation, beyond which no design changes are planned for. column of the upper part of Table 1). Next, they broke down Table 1. An example UXM calculation goals into parameters and assigned them weightages (shown in the second column of the lower part of Table 1). The team’s Goals Weightage Score experience from previous Indic text input projects and mobile Learnability 4 78.6 phone projects helped them arrive at these weightages. The team then evaluated the interface and assigned scores to each Speed of use 2 77.1 parameter (shown in the third column of the lower part of Table Ease of use 3 65.6 1). A weighted average of parameter scores gave the score for Error free use 3 67.5 each goal (shown in the light grey cells of the third column of lower part of Table 1). A weighted average of the goal scores Retention 1 75.0 gave the UXM value (shown in the dark grey cells of the upper Subjective satisfaction 2 78.6 part of Table 1). Parameter evaluation guidelines have not been listed in this paper due to space constraints. UXM Value 73.3 4. INDEX OF INTEGRATION Goal Parameters Weightage Score We conceive Index of Integration (IoI) as an empirical process metric, nominally on a scale of 0-100, where 100 represents the Learnability 78.6 best possible integration of HCI activities in the software Conceptual model clarity 3 75 development activities and 0 represents the worst. The metric Language understandability 0 0 consists of these distinctions: Minimal learning time 5 75 Software Engineering Phases: These are the broad phases as described in a software engineering process model. Consistency with earlier version 1 50 HCI Activities: These are prescribed for each phase of the Visibility of choices and data 4 100 software engineering process model. Consistency with other products 1 50 Weightage: Each HCI activity will have a given weightage on the Speed of use 77.1 scale of 0-5 where 0 represents that the activity is not important, 3 User control and freedom 3 75 reflects the typical importance in most projects and 5 indicates that this activity is very important in the context of that project. No memory and cognitive load 4 100 Internal consistency 4 75 Score: Each activity has a score associated with it. The score is given on a rating of 0-100, where 100 represents the best case Customization 0 0 situation where the activity was done in the best possible manner, Automation and shortcuts 1 0 in the most appropriate phase of software development and with the best possible deliverables. 0 represents the worst case Ease of use 65.6 situation where the activity was not done at all. Minimal user task load 5 75 Activity evaluation guidelines: These spell out considerations Automation of routine tasks 3 50 that help the evaluation of each activity. Error free use 67.5 Software engineering phases have been extensively described in Good feedback 4 75 literature. For example, the phases of the waterfall process model are Communication, Planning, Modelling, Construction and Error tolerance 3 50 Deployment [20]. Error recovery 3 75 On the other hand, no widely accepted industry-wide Retention 75.0 specifications of HCI activities for given SE phases have emerged so far. But there have been a few proposals. For example, [12] Retention 3 75 prescribes that the Communication phase of the waterfall model Subjective satisfaction 78.6 should have these HCI design activities: Contextual user studies Visceral appeal 2 75 and user modelling, Ideation, Product definition and Usability evaluation and refinement of product definition. Figure 1 Behavioural appeal 4 75 summarizes the HCI activities suggested for the waterfall model Reflective appeal 1 100 phases based on these recommendations. Table 1 shows an example UXM calculated by a team for an Indic text input interface for novice users on a mobile phone. The team was given a default set of higher level goals, parameters and example parameter evaluation guidelines. The team first assigned the weightages for higher level goals (shown in the second an established user modelling methodology such as Communication personas, work models, affinity diagram or similar Project initiation User studies 6. Competitive products and earlier versions of the product Ideation were evaluated for potential usability problems by using Product definition discount usability evaluation methods such as Heuristic Evaluation and Evaluation or better refinement Requirements 7. User experience goals were explicitly agreed upon before ifi i Planning finalizing requirements Estimating 100 = All the above are true; 75 = At least five of the above Scheduling are true, including 7, 50 = At least three of the above are Tracking true, including 7; 25 = At least two of the above are true, 0 = None of the above are true Modelling Detailed UI prototyping Usability Table 2. An example IoI calculation evaluation SE Phases and HCI Activities Weightage Score Requirements analysis Construction Communication Code Contextual user studies and user 4 39 Test modelling Usability evaluation Ideation 2 6 Deployment Product definition 3 75 Delivery Support Usability evaluation and refinement 1 63 Feedback of product definition Modelling Figure 1. Integration of HCI activities with the phases of the Detailed UI Prototyping 5 53 waterfall model [12]. The HCI activities corresponding to Usability Evaluation and 4 44 each phase have been underlined. Refinement of the Prototype Construction Weightage of some HCI activities could vary within a range in the Development support reviews by 3 29 context of a project. For example, if the domain or users are usability team unknown to the UX team, it may be very important to do Usability evaluation (summative) 1 46 contextual user studies in the communication phase (weightage = 4). On the other hand, if the UX team is already very familiar IoI Value 45 with the context and the domain and if they have a lot of experience designing similar products, it may be less important The process for computing IoI for a project has these steps: (weightage = 2). Table 2 summarises loosely recommended weightages for HCI activities for the waterfall model. Company HCI Process Prescription: The HCI group in the company prescribes what HCI activities should be done in which Guidelines may define the techniques used to carry out activities, phase of SE process, expected deliverables from each activity, the skills and experience levels of the people doing the activities, suggested weightages for each activity and suggested activity the deliverables and other parameters that affect the fidelity of the evaluation guidelines. As it often happens, a contract software activity. For example, following are the guidelines for the activity development company may follow not one SE process, but of Contextual user studies and user modelling in Table 2: several. In that case the HCI design process needs to be integrated 1. Organizational data gathering and user studies were done with each SE process. The prescribed process also suggests a before requirements were finalised weightage for each HCI activity and guidelines to score each activity. 2. User studies were done in the context of the users by the method of contextual inquiry Project HCI Process Definition: After getting a project brief, the UX professional fine-tunes the weightages for the prescribed HCI 3. User studies were done with at least 10 users in each profile activities after considering the domain, the users and project 4. User studies were done by people with experience in user context. For example, if the HCI team has recently done studies in a similar domain in at least 2 projects contextual user studies in the same domain for a similar product and is already very knowledgeable about the context, then he may 5. The findings including user problems, goals, opportunities reduce the weightage of contextual user studies. On the other and constraints were analyzed, documented and presented in hand if the team is less knowledgeable, he may increase the weightage. He should consult colleagues in the development team closer to SE. It seemed to create lot of buy-in for HCI activities and business stakeholders before finalizing the weightage. from the project stakeholders. One project stakeholder said “I never thought we could think so much [about user experience].” Process Evaluation: After the project is over, a group of The activity seemed to be more successful in projects where independent UX professionals review the HCI activities and several stakeholders from the project participated as it stimulated evaluate them for process compliance and give a score for each discussion among stakeholders. While the participants appreciated activity on a scale of 0 to 100. They may reduce the score if an the organizational perspective, the metrics seemed of less use to activity was done with lower fidelity, resulted in poor quality the projects as the projects were already over. Participants deliverables or was done later than prescribed. In case of multiple suggested that metrics should be calculated mid-way through the evaluators, an average across evaluators is deemed to be the project while course correction was still possible. score. Specifically, UXM helped the HCI designers and project IoI Calculation: The metric is found by computing the weighted stakeholders to make goals explicit. One HCI designer remarked, average of the scores of all activities: IoI = ∑ ( Wa x Sa / ∑ Wa ), “Had we done this earlier, I would have known where to focus.” where Wa is the weightage for a particular HCI activity, Sa is the The teams usually added a few goal parameters (typically 2-3 per score (from 0-100) for that activity. In case there is a lot of project) and adjusted weightage to suit UXM to their project. divergence in scores of a particular HCI activity, the activity is They confirmed that this flexibility is indeed desirable. Though discussed and reviewers are given a chance to change their score parameter evaluation guidelines for UXM helped, more details before an average is taken. were desired. Participants did not make changes to the parameter Table 2 shows calculation of IoI for an example project. First, evaluation guidelines except when new parameters were added. senior UX professionals defined the HCI activities, activity Giving examples of HCI goals (learnability, ease of use etc.) weightages and evaluation guidelines that the company should be helped participants to set goal parameters and weightages. One following. Then a project that had recently ended was selected for stakeholder remarked: “without these inputs it would have been retrospective review. The project manager and the UX difficult to [assign weightage and scores].” professionals working on the project fine-tuned the weightages for In case of a few UXM parameters, divergent scores emerged for the project context. The second column of Table 2 contains these some parameters in each project. Usually variations happened in weightages. A group of reviewers comprising of some project parameters where the evaluation guidelines were not understood insiders and outsiders reviewed and rated the HCI activities in the well or were interpreted differently by evaluators. In such cases, it project and its IoI was calculated. The third column of Table 2 was felt, that it was better to let participants discuss the parameter contains the average scores assigned to each activity by the and change ratings to converge scores if they so desire. Reducing reviewers. the number of steps in scoring a parameter (e.g. 0-25-50-75-100) Guidelines for evaluating all HCI activities listed in Table 2 have helped reduce variation among scores. More detailed UXM been created. More guidelines for evaluating HCI activities as parameter evaluation guidelines with examples will further help part of extreme programming process have been created as well. in reducing divergence. Both have been omitted here due to space constraints. Computing IoI was useful for project stakeholders as they could see the importance of HCI activities in the SE context. The HCI 5. METRICS EVALUATION activities integrated in SE process models were acceptable as The authors evaluated the metric in two ways. First, UXM and IoI suggested. Though they were explicitly prompted, none of the metrics were computed for retrospectively projects in Tech project stakeholders wanted changes to the prescribed HCI Mahindra, a large contracted software development company. In activities, their weightage or evaluation guidelines. An important each case, the metrics computation was done by HCI feedback was need for process models specifically targeted to professionals from the project, independent HCI professionals and redesign projects. Process models typically discuss new product project stakeholders. At the end of metrics computation, feedback development. Given that many industry projects are “next version was taken from participants of each project about the metrics. of X” type, process models must be specifically adapted for them. Second, the metrics were presented to a group of faculty members Walking through the activity evaluation guidelines helped in from a reputed university and their comments were noted. Three scoring as all stakeholders were not aware of all HCI activities. It of these were faculty members from the computer science was felt that IoI should be computed before computing UXM as discipline. One was a faculty member from a design school who is this minimizes bias. also an expert in cognitive psychology. The metric descriptions presented in this paper are a result of iterative modifications that reflect the feedback and lessons learnt. 5.1 Findings It typically took about 3 hours to compute both IoI and UXM for each project. The time included explaining the two metrics, 6. DISCUSSIONS It is important to discuss the limitations and risks of the two weightage assignment and scoring. This seemed to be optimum metrics proposed. Both UXM and IoI are summary measures that time, longer meetings were difficult to schedule. The projects leave out much information. They allow a drill-down to performed similarly in IoI and UXM scores – the one project that constituent components, but do not point to specific problems or had a high UXM value also had a high IoI value. Participants, give suggestions for improvement. But summary measures are particularly project stakeholders, were at home with the activity useful in many contexts, particularly for comparison across of metric calculation. To them, the activity seemed to bring HCI projects. Such comparisons can help UX groups understand what parameters and weightages. A similar tool for IoI can also evolve works and what doesn’t and improve performance year-on-year. the specification of processes. Perhaps most important limitation of UXM comes from the ephemeral nature of a ‘user experience’. Any attempt to numerically embody such an abstract phenomenon is bound to be 8. ACKNOWLEDGMENTS subjective and measures could differ according to the We thank the anonymous reviewers for invaluable comments on interpretation of the evaluators. Further, large companies are improving our presentation of this material. In particular, we involved in software development projects for many clients, thank Ved Prakash Nirbhay and Deepak Korpal from Tech across domains, platforms, users, use contexts and task Mahindra for allowing us to interact with their team members complexity, frequency and criticality. while testing the metrics in different software projects. We also Finally, there is a risk that because UXM measures are low cost, thank members of User Interaction Design Group at Tech organizations may be tempted to sacrifice all user-facing activities Mahindra Ltd. for participating in User Experience metrics (such as usability tests or field studies) in its favour. We do not evaluation. We also thank Pramod Khambete for his continuous recommend this at all. The purpose of UXM is not to replace support and appreciation. We thank Prof. NL Sarda, Prof. UA these established methods but to supplement them and to help Athavankar, Prof. Umesh Bellur and Prof. S Sudarshan for their them mature. continuing guidance and suggestions in developing the two metrics. In spite of these limitations, we believe that UXM is useful. UXM shows the extent to which user experience goals were achieved in a particular project. We found that breaking up abstract notions of 9. REFERENCES user experience into specific goals and parameters helped [1] Beyer, H., Holtzblatt, K., Contextual Design: Defining evaluators focus on one issue at a time and reduced the Customer Centered Systems, Morgan Kaufman (1998) subjectivity in measurement. Making the evaluation criteria [2] Bias, R., Mayhew, D. (Eds), Cost-Justifying Usability, explicit and averaging across several evaluators further reduced Second Edition: An Update for the Internet Age, Morgan the subjectivity in judgement. The risk of variety in products (the Kaufmann (2005) apples-and-oranges risk) was partly mitigated by selecting goal- parameters relevant to each project and giving custom weightage [3] Cooper, A., Riemann, R., About Face 2.0 the Essentials of to each parameter. Interaction Design, Wiley (2003) The main limitation of IoI is that it does not measure the absolute [4] Fenton, N.E., Pfleeger, S.L., Software Metrics – A Rigorous process quality of the project, rather how compliant was a project and Practical Approach, Thomsan Brooks/Cole (2002) to the prescribed process. There are no widely-accepted integrated [5] Göransson, B., Lif, M., Gulliksen, J., Usability Design – process models at this stage. Yet, IoI in conjunction with UXM Extending Rational Unified Process with a New Discipline. may be used to verify the effectiveness of new process model International Workshop on Interactive Systems Design, proposals. If UXM and IoI are correlated, the new proposal seems Specification, and Verification (2003) acceptable. On the other hand if the UXM and IoI do not show a [6] IEEE Standard Glossary of Software Engineering correlation, it questions the prescribed process model. Terminology, IEEE, 1993 UXM and IoI have an organizational perspective and make more [7] IFIP working group 2.7/13.4 on User Interface Engineering, sense while looking across hundreds of projects rather than within Bridging the SE & HCI Communities: http://www.se- each project individually. They have very low additional hci.org/bridging/index.html (2004), accessed August, 2008 overheads on the process and are easy to integrate in the process. [8] International Organization for Standardization, ISO/IEC Overall feedback indicates that UXM and IoI are useful and 9126-1:2001 Software Engineering - Product Quality (2001) practical in evaluating products and processes. There was a lot of buy-in from project stakeholders calculating metrics as there was [9] International Organization for Standardization, ISO 9241- a lot of willingness to track, control the user experience of the 1:1997 Ergonomic requirements for office work with visual product. The aspects that metric calculation was light-weight and display terminals (VDTs) (1997) independent of specific usability methods were particularly liked. [10] Jordan, P. W., Designing pleasurable products, Taylor & Francis (2000). 7. FUTURE WORK [11] Joshi, A: HCI in SE Process Literature, Indo-Dan HCI In future, we plan to use metrics prospectively throughout the Research Symposium, IIT Guwahati (2006) duration of projects and demonstrate their usefulness during the project. We will be building more elaborate tools and guidelines [12] Joshi, A., Sarda N.L.: HCI and SE: Towards a ‘Truly’ to improve the consistency of weightages and scores. We also Unified Waterfall Process. HCI International ‘07 (2007) propose to do a rigorous validation of the two metrics in [13] Kroll, P., Kruchten, P., The Rational Unified Process Made experimental and industrial situations. Easy, Pearson Education (2003) In its current form, UXM goals, parameters and weightages have [14] Lewis, J., A Rank-Based Method for the Usability to be chosen on the basis of experience of individuals. However, Comparison of Competing Products. Human Factors and it is possible to design tools in future that will collate experience Ergonomics Society 35th Annual Meeting 1312--1316 of several practitioners to help in choices of future goals, (1991) [15] Lin, H. Choong, Y. Salvendy, G. A proposed index of [21] Pyla, P. S., Pérez-Quiñones, M. A., Arthur, J. D., Hartson, H. usability: a method for comparing the relative usability of R.: Towards a model-based framework for integrating different software systems. Behaviour & Information usability and software engineering life cycles. Interact 2003 Technology (1997) Workshop on “Closing the Gaps: Software Engineering and [16] Mahlke, S., Understanding users’ experience of interaction, Human Computer Interaction” 67--74 (2003) in Marmaras, N., Kontogiannis, T., Nathanael, D. (Eds.), [22] Sauro, J., Kindlund, E., A method to standardize usability Proc. EACE '05 (2005). 243-246. metrics into a single score. CHI '05 401-409 (2005) [17] Mayhew, D., The Usability Engineering Lifecycle: A [23] Shneiderman, B.: Designing the User Interface, Strategies for Practitioner's Handbook for User Interface Design; Morgan Effective Human-Computer Interaction. 4th edition. Addison Kaufmann; 1998 Wesley (2004) [18] McGee, M., Master usability scaling: magnitude estimation [24] Swallow, D., Blyth, M., Peter, W., Grounding experience: and master scaling applied to usability measurement, Proc. relating theory and method to evaluate the user experience of CHI’00, ACM Press (2004) 335-342. smart-phones, Proc. 2005 Annual Conference on European [19] Norman, D.A., Emotional Design: Why We Love (or Hate) Association of Cognitive Ergonomics (2005) 91-98. Everyday Things, Basic Books (2004). [25] Tractinsky, N., Katz, A. S. & Ikar, D.. What is beautiful is [20] Pressman, R.: Software Engineering – a Practitioner’s usable, Interacting with Computers, 13 (2000) 127-145 Approach. 6th edition. McGraw Hill (2005)