This paper presents a case study of what was intended to be a qualitative usability evaluation of a CRM (Customer Relationship Management) system but finally ended as a cross-disciplinary service design innovation workshop. This text presents evaluation framework and main categories of obtained results, discussed from the viewpoint of redesigning the CRM system as an e-service for internal customers. Discussion of key success factors and lessons learned from this study conclude the paper.
INTRODUCTION
Usability of business IT systems has been a topic of
numerous studies since the beginnings of HCI [
Research perspectives concerning interactive systems in recent years evolved a lot: systems engineering perspective so dominant tree decades ago has been replaced UserCentred Design (UCD) perspective now. In recent years also User Experience (UX), Value-Based Design and Service Design perspectives brought research methodologies closer to a real social and economic context in which contemporary interactive systems have been actually used. Social interactions on-line and – in general – human behaviour on-line have become new, intriguing research issues, regarding both private and business life. PROBLEM DESCRIPTION Problem background A multi-modular CRM (Customer Relationship Management) system has been used by a large Polish financial company, but in the focus of this evaluation there was included only the CRM module used by call-centre operators for serving daily hundreds of customers by the phone.
This usability evaluation project was undertaken mainly due to systematic complaints arriving from the call-centre operators, who were claiming that poor system usability dramatically slows down the customer service. Moreover, recently there have been incoming signals that customers are getting increasingly irritated by time-taking call-centre procedures even in small matters. As a result, after reaching some critical mass, these operator complaints were seriously taken and finally the CRM usability improvement project has been launched.
Evaluation framework The company so far has not had their own usability staff, so an evaluation team has been formed of: two external usability consultants, four employees: the CRM system “owner” from the IT department and three senior call-centre operators (department leaders), very experienced in dealing with different types of financial products.
In order to streamline the teamwork, following evaluation procedure was accepted:
Crowdsourcing method will be used at first for gathering by e-mail all observed complaints from front-line operators in the call-centre.
Collaborative expert review of typical operator procedures will be performed for major operational paths. 1. 2.
Complaints collected from front-line operators will be aggregated with evaluator’s comments as to their relevance and feasibility for planned usability improvements. 4. Supplementary expert evaluation (checklist and heuristic) will be applied for assessing the user interface compliance with HCI guidelines. 5. Final report (PowerPoint presentation to be discussed with the IT department and the executives) will be prepared, showing prioritized recommendations and their projected impact on system usability.
Evaluation context The team worked over a week several hours a day, thoroughly analyzing a live demo of on-the-phone customer service and watching literally each step performed by senior operators. The system was operated from a laptop in a training room, with live CRM picture projected onto a big screen so as all team members could have a good visibility of the spots where the usability problems were identified. The demo was accompanied with narrative “user stories” by senior operators explaining the purpose and meaning of each action performed in a call-centre conversation context. During the demo presentation front-line operators’ remarks and suggestions from crowdsourcing have been reviewed and supplemented by senior operators’ comments on the possible impact a specific flaw might have on the customer service speed and quality.
It seemed noteworthy that senior operators often referred to the fact that the conversation flow with the customer onthe-phone was strictly regulated by the company procedures. However, because of different reasons on the side of the customer the default conversation flow often must be adapted on-the-fly to the context - and the CRM system should be flexible enough to let the operator work that way.
During the teamwork we could observe gradually changing focus of attention from usability of the CRM system to analyzing user experience of an operator. In the background, however, we have been also considering the user experience of the customer on-the-phone; it is indirectly affected by perceived service quality, resulting from the combination of the CRM system usability and the momentary UX of a call-centre operator.
EVALUATION RESULTS Usability and UX aspects Despite many usability flaws have been detected, in general in this CRM system using tab-based web interface with plenty of editable forms, operators basically met no problem in finding a suitable navigation path matching the actual needs of the customer on-the-phone.
However, it turned out that the most important operator UX discomforts with the CRM system were caused by some other factors, like: necessity to frequently quit the CRM system in order to find information available only in other modules (e.g. off-line contact history data), or necessity to verify currently displayed data in other sources.
The issues of sub-optimal visual design, demanding manual control or inconsistent data fields labelling have been also raised, and later confirmed in the expert evaluation review. While the team approached identifying dimensions of user experience, it also turned out that operators were very creative in finding various workarounds to overcome existing usability problems because their actual performance was very much affected by the bonus system, which was fed by the data from automatic monitoring of operator’s actions in the CRM system. These observations helped to understand actual operators’ work habits, motivations and attitudes, bringing important ethnographic insight to the scope of this evaluation study.
Organizational aspects During evaluation sessions the team discussions very often evolved from pure usability towards user experience (UX) issues, interpreted in twofold manner: (1) Operator experience, covering a set of emotions resulting from the CRM system behaviour and simultaneously, from the customer behaviour on the phone line; (2) Customer experience, covering the set of emotions resulting from the perceived quality of specific on-thephone service.
When discussing the screens and procedures, the team members realized that the CRM system usability problems must be seen as a part of overall service quality landscape, also relevant to the way how operators actually do their best with the existing CRM system (trying to earn their bonus, though).
As a result, a set of guidelines was proposed for the final evaluation report, covering issues such as: visual design and interaction flow improvements, software improvements (technical quality), better formatting of usability specifications for external software vendors.
More importantly, a set of classified recommendations was made, aimed at improving operators’ trust to the CRM system and operators’ relationship with the company brand, as the employer.
Other outcomes Apart from usability- and UX-relevant outcomes, other key findings of this study were important: negative operator’s UX resulting from suboptimal usability of the CRM system is likely to affect the quality of service offered to the on-the-phone customer; therefore improving usability of the backstage CRM is a good investment for enhancing the quality of serving the customer by the call-centre; in this project company managers experimentally decided to gather usability comments from CRM operators by open internal crowdsourcing, and also by encouraging other staff members to contribute to the project; it produced surprisingly fruitful outcomes and resulted in creating a unique crossdepartmental cooperation around this project; front-line operators turned out to be highly motivated to deliver their comments in crowdsourcing and to participate in further redesign process of the CRM system, which is the main tool in their work environment; this attitude may suggest the premise of positive relationship with the employer, reflected here in their commitment.
Finally, during subsequent evaluation sessions a crossdisciplinary perspective was developed in the project team, which seemed to contribute much to the project success. Otherwise it wouldn’t be possible to embrace the complexity of discovered problems: evaluation viewpoints that were very diverse at the start, have been gradually negotiated and aggregated during evaluation teamwork, at the end usually resulting in a set of balanced and feasible recommendations.
POST-EVALUATION REMARKS Key success factors At this point, after completing the evaluation part of this project, some key success factors could be identified:
The first success factor - already mentioned - was very productive crowdsourcing, which delivered dozens of valuable comments and suggestions from the front-line. Consequently, senior operators and the CRM owner (IT) used their expertise to frame collected suggestions into a specific task context and were very active in searching for feasible solutions.
In both cases it was visible the staff was aware how the usability flaws affect the service quality for external customer, despite natural motivation to improve operator’s experience and comfort as well. 1. 2. 4. 5. 6.
7.
In this project creating an ambient evaluation environment was also very important for facilitating effective teamwork: a round table configuration, circular information flow, ongoing visual contact, a wall-size projected CRM screen as a central focus of attention - all these elements all helped to stimulate group dynamics in this project.
The next important success factor was agile-like evaluation cycle which formed the canvas for the analytic part of the project. This cycle was repeated regularly for each discovered usability problem and consisted of following sequence: executing step-by-step specific task situation in the CRM system, accompanied by “user stories”, reviewing situation-relevant suggestions from crowdsourcing, comments and 3. locating and classifying user interface problems, brainstorming for possible solutions1, searching for the problem cause and origin, problem diagnosis and reference to the procedures or local organizational context, documenting proposed solution (or a set of).
This cycle was iterated for each detected problem and it allowed conducting unstructured analysis. Iterative conversational method, asking “naive” questions and refining answers through the unrestricted creation of ideas have finally led to developing solution proposals. In this cycle “the art of asking right questions” to the senior operators also played some role; it was essential for focusing attention on important UX aspects and for creative exploration of problem space.
Finally, the integrating role of senior operators was crucial during evaluation sessions: they enabled putting the operators’ complaints into the screen context and into the task/organizational context, both essential for external usability experts for proper interpreting high-level interaction design principles to a specific screen or conversation scene.
Novel evaluation elements Despite of direct outcomes aimed for the CRM system redesign, in this project some novel elements emerged: A. Usability evaluation converted into innovation workshop When developing proposals for improving the operator UX, both individual creativity and team-discussed refinements 1 brainstorming for possible solutions was intentionally located in this cycle before finding the problem cause were combined, using spontaneous brainstorming and also analytic conceptual refinements.
Starting from visions of specific screens with improved interaction elements, the amount of creativity input was growing so fast, that it gradually converted usability evaluation sessions into a sort of innovation workshop. The list of proposed improvements and innovations was long, and they could be sorted into two groups: ideas relevant to UX, user interface and the CRM system, aimed at improving operator UX with the CRM system; ideas relevant to various organizational improvements related to the back-stage activities.
Due to sensitivity of this project, invited external usability experts were able to operate the CRM systems only via an authorised senior operator.
Paradoxically, the apparent shortage of direct experience from “feel” of the system resulted in more extensive discussions, because domain experts (senior operators) had to explain in more detail the meaning/purpose sense of each click and each operation.
In seems that forced restrictions in access to the system apparently facilitated developing a multi-point, crossdisciplinary evaluation perspective for team members.
From the external customer viewpoint everything is a service, and from the operator viewpoint everything what is provided to facilitate his/her work can be also considered a service (on-line or off-line, respectively).
As such, the CRM system actually is an internal e-service aimed at operators who are internal customers. Analogically, the other part of the system (voice interface with an operator) is the front-stage e-service aimed at external customers.
Treating a CRM system holistically as kind of e-service (twofold: internal and external), helped to identify complementary values produced for internal and for external customers. In general, this perspective seems useful also for prospective evaluations of other IT systems in this company.
KNOWLEDGE MANAGEMENT ASPECTS The teamwork performed in this project can be divided into three parts: 1. 2. 3.
analytic - typical evaluation, based on general HCI
and usability evaluation methodologies [
In both analytic and creative parts knowledge-intensive
tasks have been performed, involving cross-disciplinary
knowledge diffusion among team members. Knowledge
transfers typical for usability consulting have been
described in [
Fig. 1. Value chain in service systems, adapted from [
Finally, during the final report presentation there was the knowledge transfer between the evaluation team and the project sponsors - company executives.
SERVICE DESIGN PERSPECTIVE Starting from a routine usability study, this project has eventually raised the significance of broader UX evaluation focus, namely treating the interactive system as a service system, which produces value for internal and for external customers.
This perspective is coherent with the concept of service
value chain proposed by Heskett [
Fig.1. (in the lower part) shows the parts of the service value chain included in this evaluation, but also organizational issues, which should be included as internal service quality factors.
Adopting service value chain perspective may result in remarkable redefining the role of HCI in current IT projects: while IT these days is often merely a vehicle for launching specific on-line services (internal or external), HCI and interaction design are often expected to build UX-competitive advantage and deliver value to users (customers); possibly better UX results may be achieved if an interactive system is designed as a service system (IT-based), aimed to offer value for specific group of customers.
Service design perspective involves the issue if value
coproduction:
in on-line service systems value for customer is
coproduced in part by quality of human-computer
interaction, but in the other part by quality of
human-socioeconomic relationships relevant to
actual system usage, like convenience, cost-saving,
community etc.
in on-line service design process value is also
coproduced by participating clients/users (Value
CoCreation), what extends the current scope of
UserCentred Design and UX design closer to
increasingly popular the Service Design approach
[
Developing profitable on-line relationships, involves mutual sharing of values produced by specific business model.
In case of on-line service systems this perspective places current HCI design practices much closer to economics, especially if the user is a conscious consumer (external, internal) willing to consume, abut also willing to coproduce value in a specific business context relationship. CONCLUSIONS This evaluation study produced several novel outcomes, unexpected at the beginning of this project: effective use of crowdsourcing, use of narrative “user stories” ethnographically presenting operators’ work habits, as well as using elements of Co-Design and Value Co-Creation, characteristic for the Service Design approach.
This project also led to a deeper understanding that: in e-business systems projects HCI has many touchpoints with service design, many interactive systems can be designed as ITbased service systems, producing value for both internal and external customers, in usability evaluation and UX design users/customers should be involved as value coproducers, what extends their role in the current UCD approach. Consequently, service value chain concept may be applied for many corporate IT systems, which should be treated as e-services designed jointly with User-Centred and Service Design approaches.
ACKNOWLEDGEMENTS This work was partly supported by the Polish National Science Centre under the contracts No. DEC2011/01/M/HS4/04995 and 4591/B/H03/2011/40. The author would like to thank the three anonymous reviewers for their valuable comments on first version of this paper.