=Paper=
{{Paper
|id=None
|storemode=property
|title=A Cross-Disciplinary UX Evaluation of a CRM System
|pdfUrl=https://ceur-ws.org/Vol-922/paper7.pdf
|volume=Vol-922
|dblpUrl=https://dblp.org/rec/conf/iused/Sikorski12
}}
==A Cross-Disciplinary UX Evaluation of a CRM System==
https://ceur-ws.org/Vol-922/paper7.pdf
A Cross-Disciplinary UX Evaluation of a CRM System
Marcin Sikorski
Gdansk University of Technology,
Faculty of Management and Economics
ul. Narutowicza 11/12
80-233 Gdansk, Poland
Marcin.Sikorski@zie.pg.gda.pl
ABSTRACT actually used. Social interactions on-line and – in general –
This paper presents a case study of what was intended to be human behaviour on-line have become new, intriguing
a qualitative usability evaluation of a CRM (Customer research issues, regarding both private and business life.
Relationship Management) system but finally ended as a
cross-disciplinary service design innovation workshop. This PROBLEM DESCRIPTION
text presents evaluation framework and main categories of
Problem background
obtained results, discussed from the viewpoint of
A multi-modular CRM (Customer Relationship
redesigning the CRM system as an e-service for internal
Management) system has been used by a large Polish
customers. Discussion of key success factors and lessons
financial company, but in the focus of this evaluation there
learned from this study conclude the paper.
was included only the CRM module used by call-centre
operators for serving daily hundreds of customers by the
Author Keywords
phone.
usability, User Experience, User-Centred Design, Service
Design, collaborative design, Intranet This usability evaluation project was undertaken mainly
ACM Classification Keywords
due to systematic complaints arriving from the call-centre
H.1.2. Human factors; H.5.2. User interfaces; H.5.3. Group operators, who were claiming that poor system usability
and Organization Interfaces; dramatically slows down the customer service. Moreover,
recently there have been incoming signals that customers
are getting increasingly irritated by time-taking call-centre
procedures even in small matters. As a result, after reaching
INTRODUCTION some critical mass, these operator complaints were
Usability of business IT systems has been a topic of seriously taken and finally the CRM usability improvement
numerous studies since the beginnings of HCI [5, 6, 7, 11]. project has been launched.
Usability of company Intranets and other back-stage IT
systems still has a big impact on work efficiency. These Evaluation framework
systems are today an essential part of each digital The company so far has not had their own usability staff, so
workplace [1], serving as corporate information repositories an evaluation team has been formed of:
and facilitating internal communication, teamwork and
workflows. two external usability consultants,
Research perspectives concerning interactive systems in four employees: the CRM system “owner” from
recent years evolved a lot: systems engineering perspective the IT department and three senior call-centre
so dominant tree decades ago has been replaced User- operators (department leaders), very experienced
Centred Design (UCD) perspective now. In recent years in dealing with different types of financial
also User Experience (UX), Value-Based Design and products.
Service Design perspectives brought research In order to streamline the teamwork, following evaluation
methodologies closer to a real social and economic context procedure was accepted:
in which contemporary interactive systems have been
1. Crowdsourcing method will be used at first for
gathering by e-mail all observed complaints from
front-line operators in the call-centre.
2. Collaborative expert review of typical operator
procedures will be performed for major
operational paths.
3. Complaints collected from front-line operators will However, it turned out that the most important operator UX
be aggregated with evaluator’s comments as to discomforts with the CRM system were caused by some
their relevance and feasibility for planned usability other factors, like:
improvements.
necessity to frequently quit the CRM system in
4. Supplementary expert evaluation (checklist and order to find information available only in other
heuristic) will be applied for assessing the user modules (e.g. off-line contact history data), or
interface compliance with HCI guidelines.
necessity to verify currently displayed data in
5. Final report (PowerPoint presentation to be other sources.
discussed with the IT department and the
executives) will be prepared, showing prioritized The issues of sub-optimal visual design, demanding manual
recommendations and their projected impact on control or inconsistent data fields labelling have been also
system usability. raised, and later confirmed in the expert evaluation review.
While the team approached identifying dimensions of user
Evaluation context experience, it also turned out that operators were very
The team worked over a week several hours a day, creative in finding various workarounds to overcome
thoroughly analyzing a live demo of on-the-phone customer existing usability problems because their actual
service and watching literally each step performed by senior performance was very much affected by the bonus system,
operators. The system was operated from a laptop in a which was fed by the data from automatic monitoring of
training room, with live CRM picture projected onto a big operator’s actions in the CRM system. These observations
screen so as all team members could have a good visibility helped to understand actual operators’ work habits,
of the spots where the usability problems were identified. motivations and attitudes, bringing important ethnographic
The demo was accompanied with narrative “user stories” by insight to the scope of this evaluation study.
senior operators explaining the purpose and meaning of
each action performed in a call-centre conversation context. Organizational aspects
During the demo presentation front-line operators’ remarks During evaluation sessions the team discussions very often
and suggestions from crowdsourcing have been reviewed evolved from pure usability towards user experience (UX)
and supplemented by senior operators’ comments on the issues, interpreted in twofold manner:
possible impact a specific flaw might have on the customer (1) Operator experience, covering a set of emotions
service speed and quality. resulting from the CRM system behaviour and
It seemed noteworthy that senior operators often referred to simultaneously, from the customer behaviour on the phone
the fact that the conversation flow with the customer on- line;
the-phone was strictly regulated by the company (2) Customer experience, covering the set of emotions
procedures. However, because of different reasons on the resulting from the perceived quality of specific on-the-
side of the customer the default conversation flow often phone service.
must be adapted on-the-fly to the context - and the CRM
system should be flexible enough to let the operator work When discussing the screens and procedures, the team
that way. members realized that the CRM system usability problems
must be seen as a part of overall service quality landscape,
During the teamwork we could observe gradually changing also relevant to the way how operators actually do their best
focus of attention from usability of the CRM system to with the existing CRM system (trying to earn their bonus,
analyzing user experience of an operator. In the though).
background, however, we have been also considering the
user experience of the customer on-the-phone; it is As a result, a set of guidelines was proposed for the final
indirectly affected by perceived service quality, resulting evaluation report, covering issues such as:
from the combination of the CRM system usability and the visual design and interaction flow improvements,
momentary UX of a call-centre operator.
software improvements (technical quality),
EVALUATION RESULTS
better formatting of usability specifications for
Usability and UX aspects external software vendors.
Despite many usability flaws have been detected, in general
in this CRM system using tab-based web interface with More importantly, a set of classified recommendations was
plenty of editable forms, operators basically met no made, aimed at improving operators’ trust to the CRM
problem in finding a suitable navigation path matching the system and operators’ relationship with the company brand,
actual needs of the customer on-the-phone. as the employer.
Other outcomes B. Flexible teamwork
Apart from usability- and UX-relevant outcomes, other key
findings of this study were important: In this project creating an ambient evaluation environment
was also very important for facilitating effective teamwork:
negative operator’s UX resulting from suboptimal a round table configuration, circular information flow,
usability of the CRM system is likely to affect the ongoing visual contact, a wall-size projected CRM screen
quality of service offered to the on-the-phone as a central focus of attention - all these elements all helped
customer; therefore improving usability of the to stimulate group dynamics in this project.
backstage CRM is a good investment for
The next important success factor was agile-like evaluation
enhancing the quality of serving the customer by
cycle which formed the canvas for the analytic part of the
the call-centre;
project. This cycle was repeated regularly for each
in this project company managers experimentally discovered usability problem and consisted of following
decided to gather usability comments from CRM sequence:
operators by open internal crowdsourcing, and also
1. executing step-by-step specific task situation in the
by encouraging other staff members to contribute
CRM system, accompanied by “user stories”,
to the project; it produced surprisingly fruitful
outcomes and resulted in creating a unique cross- 2. reviewing situation-relevant comments and
departmental cooperation around this project; suggestions from crowdsourcing,
front-line operators turned out to be highly 3. locating and classifying user interface problems,
motivated to deliver their comments in
4. brainstorming for possible solutions1,
crowdsourcing and to participate in further
redesign process of the CRM system, which is the 5. searching for the problem cause and origin,
main tool in their work environment; this attitude
6. problem diagnosis and reference to the procedures
may suggest the premise of positive relationship
or local organizational context,
with the employer, reflected here in their
commitment. 7. documenting proposed solution (or a set of).
Finally, during subsequent evaluation sessions a cross- This cycle was iterated for each detected problem and it
disciplinary perspective was developed in the project team, allowed conducting unstructured analysis. Iterative
which seemed to contribute much to the project success. conversational method, asking “naive” questions and
Otherwise it wouldn’t be possible to embrace the refining answers through the unrestricted creation of ideas
complexity of discovered problems: evaluation viewpoints have finally led to developing solution proposals.
that were very diverse at the start, have been gradually
In this cycle “the art of asking right questions” to the senior
negotiated and aggregated during evaluation teamwork, at
operators also played some role; it was essential for
the end usually resulting in a set of balanced and feasible
focusing attention on important UX aspects and for creative
recommendations.
exploration of problem space.
POST-EVALUATION REMARKS Finally, the integrating role of senior operators was crucial
Key success factors during evaluation sessions: they enabled putting the
At this point, after completing the evaluation part of this operators’ complaints into the screen context and into the
project, some key success factors could be identified: task/organizational context, both essential for external
usability experts for proper interpreting high-level
A. Staff commitment interaction design principles to a specific screen or
The first success factor - already mentioned - was very conversation scene.
productive crowdsourcing, which delivered dozens of
valuable comments and suggestions from the front-line. Novel evaluation elements
Despite of direct outcomes aimed for the CRM system
Consequently, senior operators and the CRM owner (IT) - redesign, in this project some novel elements emerged:
used their expertise to frame collected suggestions into a
specific task context and were very active in searching for A. Usability evaluation converted into innovation workshop
feasible solutions. When developing proposals for improving the operator UX,
In both cases it was visible the staff was aware how the both individual creativity and team-discussed refinements
usability flaws affect the service quality for external
customer, despite natural motivation to improve operator’s
experience and comfort as well. 1
brainstorming for possible solutions was intentionally
located in this cycle before finding the problem cause
were combined, using spontaneous brainstorming and also From the external customer viewpoint everything is a
analytic conceptual refinements. service, and from the operator viewpoint everything what is
provided to facilitate his/her work can be also considered a
Starting from visions of specific screens with improved
service (on-line or off-line, respectively).
interaction elements, the amount of creativity input was
growing so fast, that it gradually converted usability As such, the CRM system actually is an internal e-service
evaluation sessions into a sort of innovation workshop. The aimed at operators who are internal customers.
list of proposed improvements and innovations was long, Analogically, the other part of the system (voice interface
and they could be sorted into two groups: with an operator) is the front-stage e-service aimed at
external customers.
ideas relevant to UX, user interface and the CRM
system, aimed at improving operator UX with the Treating a CRM system holistically as kind of e-service
CRM system; (twofold: internal and external), helped to identify
complementary values produced for internal and for
ideas relevant to various organizational improvements external customers. In general, this perspective seems
related to the back-stage activities. useful also for prospective evaluations of other IT systems
B. Forced multipoint analysis in this company.
Due to sensitivity of this project, invited external usability KNOWLEDGE MANAGEMENT ASPECTS
experts were able to operate the CRM systems only via an The teamwork performed in this project can be divided into
authorised senior operator. three parts:
Paradoxically, the apparent shortage of direct experience 1. analytic - typical evaluation, based on general HCI
from “feel” of the system resulted in more extensive and usability evaluation methodologies [6, 7, 11],
discussions, because domain experts (senior operators) had
to explain in more detail the meaning/purpose sense of each 2. creative - brainstorming and evaluating solutions,
click and each operation. based on Double Diamond model [2],
In seems that forced restrictions in access to the system 3. constructive - documenting redesign
apparently facilitated developing a multi-point, cross- recommendations, to be implemented later in
disciplinary evaluation perspective for team members. another project.
C. CRM system as an internal e-service In both analytic and creative parts knowledge-intensive
tasks have been performed, involving cross-disciplinary
A cross-disciplinary evaluation perspective has finally led knowledge diffusion among team members. Knowledge
to putting the CRM system in the wider context of the call- transfers typical for usability consulting have been
centre services offered to customers. described in [10], and they again appeared in this CRM
back-stage systems front-stage systems
web-based services, e-services
Fig. 1. Value chain in service systems, adapted from [4]
system case. In this project cross-disciplinary knowledge actual system usage, like convenience, cost-saving,
transfer resulted in: community etc.
novel understanding of the CRM system as e- in on-line service design process value is also co-
service (with a direct impact on UX of internal produced by participating clients/users (Value Co-
customers, indirect on UX of external ones); Creation), what extends the current scope of User-
Centred Design and UX design closer to
converting usability evaluation framework into a increasingly popular the Service Design approach
sort of innovation workshop, aimed on developing [12].
creative solutions for improving customer service
Developing profitable on-line relationships, involves
converting HCI design focus into service design mutual sharing of values produced by specific business
perspective, adopted for further developments in model.
CRM redesign project.
In case of on-line service systems this perspective places
Finally, during the final report presentation there was the current HCI design practices much closer to economics,
knowledge transfer between the evaluation team and the especially if the user is a conscious consumer (external,
project sponsors - company executives. internal) willing to consume, abut also willing to co-
produce value in a specific business context relationship.
SERVICE DESIGN PERSPECTIVE
Starting from a routine usability study, this project has CONCLUSIONS
eventually raised the significance of broader UX evaluation This evaluation study produced several novel outcomes,
focus, namely treating the interactive system as a service unexpected at the beginning of this project: effective use of
system, which produces value for internal and for external crowdsourcing, use of narrative “user stories”
customers. ethnographically presenting operators’ work habits, as well
This perspective is coherent with the concept of service as using elements of Co-Design and Value Co-Creation,
value chain proposed by Heskett [4], which argues that characteristic for the Service Design approach.
internal service quality (incl. tools for serving customers) This project also led to a deeper understanding that:
affects employee satisfaction and job commitment.
Consequently, in this case of CRM system the operator UX in e-business systems projects HCI has many
has an indirect impact on customer UX and on future touchpoints with service design,
relationships with the work environment as a part of the
many interactive systems can be designed as IT-
internal branding.
based service systems, producing value for both
Fig.1. (in the lower part) shows the parts of the service internal and external customers,
value chain included in this evaluation, but also
organizational issues, which should be included as internal in usability evaluation and UX design
service quality factors. users/customers should be involved as value co-
producers, what extends their role in the current
Adopting service value chain perspective may result in UCD approach.
remarkable redefining the role of HCI in current IT
projects: Consequently, service value chain concept may be applied
for many corporate IT systems, which should be treated as
while IT these days is often merely a vehicle for e-services designed jointly with User-Centred and Service
launching specific on-line services (internal or Design approaches.
external), HCI and interaction design are often
expected to build UX-competitive advantage and
deliver value to users (customers); ACKNOWLEDGEMENTS
possibly better UX results may be achieved if an This work was partly supported by the Polish National
interactive system is designed as a service system Science Centre under the contracts No. DEC-
(IT-based), aimed to offer value for specific group 2011/01/M/HS4/04995 and 4591/B/H03/2011/40. The
of customers. author would like to thank the three anonymous reviewers
for their valuable comments on first version of this paper.
Service design perspective involves the issue if value co-
production: REFERENCES
in on-line service systems value for customer is co- 1.Bernard R. (1998). The Corporate Intranet. Wiley and
produced in part by quality of human-computer Sons, New York.
interaction, but in the other part by quality of 2.British Design Council. URL:
human-socioeconomic relationships relevant to http://www.designcouncil.org.uk/designprocess
3.Dourish P. (2006). Implications for Design. CHI 2006 9.Probst G., Raub S., Romhardt K. (2000). Managing
Proceedings, 22-27 April, 2006. ACM. 541-550. Knowledge: Building Blocks for Success. Wiley.
4.Heskett J.L., Jones T.O., Loveman G.W. (1994). Putting 10.Sikorski M., Garnik I., Ludwiszewski B., Wyrwiński J.
the Service-Profit Chain to Work. Harvard Business (2011). Knowledge Management Challenges in
Review, March-April 1994. 164-174. Collaborative Design of a Virtual Call Centre. In: Koenig
5.ISO 9241: Ergonomic requirements for office work with A., Dengel A., Hinkelman K., Howlett R., Lakhani C.J.
visual display terminals. Parts 10-17 (eds): Knowledge-Based and Intelligent Information and
Engineering Systems. KES 2011, Part II, Springer LNAI
6.Landauer, T. K. (1988). Research methods in human- 6882, 2011. 657–666.
computer interaction. In M.Helander (Ed.), Handbook of
Human-Computer Interaction (pp. 905-928). Elsevier, 11.Shneiderman B., Plaisant C. (1994). Designing the User
New York. Interface: Strategies for Effective Human-Computer
Interaction. Addison-Wesley, Reading, MA.
7.Hix D., Hartson R. (1993). Developing User Interfaces:
Ensuring Usability Through Product and Process. John 12.Stickdorn M., Schneider J. (2010). This is Service
Wiley & Sons, New York. Design Thinking. BIS Publishers, Amsterdam.
8.Nonaka, I. (1994). A Dynamic Theory of Organizational
Knowledge Creation. Organization Science 5 (1). 14-37.