In order to study how the notion of User Experience (UX) evolved over the last few years, an international survey originally conducted in 2008 by Law et al. [1] has been replicated. Its main goal was to get some insights on the points of view from practitioners on the notion of UX. After having slightly adapted the initial (English) survey and having translated it into French and German, more than 758 valid answers have been collected from all over the world. This experience report aims at illustrating some of the challenges involved in the replication of such a study as well as successes and limitations.
Presented at RepliCHI2013. Copyright © 2013 for the individual papers by the papers’ authors. Copying permitted only for private and academic purposes. This volume is published and copyrighted by its editors.
Some concepts in the field of HCI are widely spread and
used by practitioners even if a lack of empirical
research prevents the true understanding of their
meaning and impacts [
The original study has been first spread during the main conference CHI’08 before being broadcast through several communication channels. Results have been published the following year in the proceedings of CHI’09, as a 10-pages long paper. 275 answers had been collected at that time from 25 countries. In order to adapt to our project’s multicultural context and to reach a wider audience within the Frenchspeaking community of UX practitioners, all questionnaire items have been translated, from the English master version to French and German (both languages being commonly used in Luxembourg). A back translation process has been applied to ensure the quality and validity of the process.
Several reasons may explain the choice to replicate this UX survey. First of all, as User Experience is still a concept in maturation, it was worth taking stock of the situation four years after the initial study in order to see a possible evolution in the representations, points of view and practices associated to UX. Replication acts here as a way to check whether the results still apply in a different context to the original study, especially in a different temporality.
Moreover, the translation into two others languages allowed us to reach a wider audience, especially in the multicultural context in which the present work was involved. As this study constituted an exploratory step within a wider Luxemburgish project focused on UX Design, gathering additional knowledge about the French- and German-speaking practitioners’ community (not well represented in the initial study) seemed crucial to us. By trying to draw an accurate picture of the current situation of UX and building on that basis, we aim at achieving the best solutions possible to design for UX.
This study may be considered as a direct replication, since differences between both studies are limited to: A minor extension through the translation in French and German languages. The original English version was kept as default language and still represented 58.4 % of the completed surveys. Additional sociodemographics items aimed at better categorizing participants and acting as control variables to analyze the data.
The UX questionnaire encompasses 3 sections: Background: respondents were asked to first answer 13 questions about their job and educational background, their level of familiarity with UX or the importance of UX in their actual work. Sociodemographic information (age, gender, country of residence) was also collected.
UX Statements: respondents were asked to assess their agreement level with 23 UX statements on a 5-point Likert scale.
UX Definitions: Five UX definitions were presented. For each of them, participants were asked to answer the following open-ended question “What do you think of this definition?”. Finally, participants were asked to choose which definition suits them best and to freely comment on the reasoning for their choice.
The main differences between the initial study and the replication lay in additional sociodemographics to better categorize respondents. The following questions have therefore been added to the initial survey: current job position, level of familiarity with the concept of UX and collaboration with people working in the field of UX.
The survey was broadcast online from February to April 2012, on multiple communication channels. As for the original study, practitioners’ forums, social networks and mailing lists were the main vector of dissemination. From a total of 898 returned questionnaires, 758 valid questionnaires have been retained to compute the data.
Our results mainly confirmed the original findings on the understanding of UX. Our classification of UX statements sorted by mean-agreement is very similar to the original one. Uniqueness of an experience, importance of social and cultural context, and finally temporal dynamics remained highlighted as crucial by the respondents. Interestingly, our larger sample size allowed us to identify some patterns describing how the differences in UX perception and choices of a UX definition significantly vary with background variables. Analyses of qualitative data (open-ended questions) are still ongoing and may show differences between the replication and the original study. These questions will indeed probably allow us to identify a range of issues that may be underlined by the respondents in 2012 but were not previously conceptualized through the UX statements defined in 2008.
Repeating a conceptual survey presents inherent
challenges due to the relative volatility of some
concepts and notions developed in HCI, but also due to
the volatility of the main object of HCI. Driven towards
novelty and innovation some terms used in this
research field tend to emerge as popular trends and
fade away quickly without having been really analyzed
through the lens of empirical research. Some authors in
HCI suspect that it could have been the case for UX,
which is often used as an umbrella term to designate a
wide range of fuzzy and dynamic concepts such as
affects, hedonism or aesthetics [
When working in a non-English speaking country, replication (or even partial use of existing tools only) generally involves the translation of those tools into the native language of the users composing the target population and sample. The administration of a questionnaire in the native language of respondents allows to give them a better understanding of the items and to decrease the rate of people being excluded or who abort due to language difficulties. However, translating a survey may become very complex when dealing with conceptual topics (as it is the case here), which already involve several ambiguous items (whether intended or not by their authors) in their original version. The present study was translated into German and French. Even if a back translation process has been used to verify the reliability of the translation, it is not yet sure that concepts were understood in the same way across different languages (and maybe even across different respondents for the same language). To overcome this difficulty when computing the data, note that we also compared the level of nonunderstandability of the items (respondents had the option to check “I don’t understand”). Being almost similar for each language and similar to the level found in the original study, the translation was considered fairly reliable.
SAMPLING AND DIFFUSION OF THE SURVEY Replicating a research work dealing with the definition of a concept implies reaching a comparable sample both in terms of sample size and characteristics. However, how should we deal with this kind of exploratory survey that did not involve a random and representative sample? As the whole population of practitioners working in a field related to UX is not clearly defined, it was decided to simply broadcast the survey on the web. We were aware that several biases may have impacted previous results (and may also impact ours), especially the fact that only selfmotivated and careful respondents would answer the questionnaire. Moreover, it was impossible to know with accuracy neither the number of people touched by the survey (probably thousands of them), nor the coverage of the target population. However, every research design choice has strengths and weaknesses. The diffusion method chosen for the original study has clearly advantages in terms of reaching a wide audience, which fulfilled the primary exploratory goal of the study and provided us with information on what kind of practitioners declare working directly or indirectly on topics related to User Experience. We succeeded in reaching an international sample larger than the original one (n= 758 in 2012 vs. n=275 in 2008) but still almost equivalent in characteristics. The larger sample size had two main advantages: first it allowed detecting more subtle differences in the understanding and perceptions of the notion of UX according to background variables; second it allowed detecting societal evolution related to the field of HCI (e.g. an increase in the number of UX practitioners coming from Asia, Middle-East or Africa).
LIMITATIONS HIGHLIGHTED IN THE ORIGINAL SURVEY DESIGN
Replicating research implies repeating a study exactly
the way it has been conducted the first time.
Unfortunately, it is close to impossible to design studies
without any limitation and thus most studies present
some limitations, highlighted by the authors or not,
that need to be copied for the sake of replication. While
this is not intended to depreciate previous work at all, it
should highlight that repeating mistakes or inaccuracies
may be hard to accept as researches always strive for
progress. In the case of the UX Survey, we noticed
some possibilities for improvement regarding the
survey design (e.g. reduction of the number of items,
rephrasing of ambiguous UX statements,
rotation/counterbalancing of items or reflection on
open-ended questions). These improvements could
have been done quite easily with a new pre-testing
phase involving a few users. Although we were aware
of those limitations, replication forbids any major
changes in the survey design (since it may bias the
results) and we had to accept this as a matter of fact.
The solution we found to overcome this issue was to
extend our data collection. As some data cannot be
easily quantified, and as this is especially the case here
when dealing with a conceptual representation of User
Experience, additional in-depth interviews with
practitioners were conducted in order to better
understand their representations of the concept and the
way they made use of it. Concomitant with the diffusion
of the UX Survey, 25 interviews were conducted during
the first semester 2012. A semi-directive interview
guide has been created, mainly based on the principal
questions included in the UX Survey [
By replicating a previous UX survey, we intended to gain further insight into the maturational process the concept of UX undergoes. Further, we aimed at validating previous findings almost taken for granted by the HCI community (e.g. uniqueness of an experience, influence of the context, or temporal dynamics of UX). Despite some challenges and difficulties to overcome, replication of such a survey appeared valuable and highly interesting for the community. Every research design has strengths and weaknesses, requiring choices to be made with regard to the research objective. Replicating a research work therefore implies both benefits from the strengths and applying the limitations of the original study.
The present project is supported by the National
Research Fund, Luxembourg. The authors would like to
thank the authors of the initial survey conducted in
2008 [