In gamification research, most game design elements are currently being tested simultaneously. To gain more detailed knowledge about the effect of game design elements in non-game contexts, the impact of individual elements should be considered more closely. Also, the interactive experience of using augmented reality in real-world environment is being explored increasingly in a range of commercial environments. This article is about an experimental study (N=80) which investigates the effects of two game design elements (badges and leaderboards) in an augmented reality supported warehouse setting, on user's motivation and task completion time. The implementation of the leaderboard significantly increased performance, but both game design elements generate mixed results about the influence on motivation. The results show that gamification can help to make working in warehouses more interesting and possibly more efficient. Nevertheless, it should be taken into account that the use of game design elements can have negative side effects.
Motivating individuals is a challenging task, especially when it comes to activities that are little or no fun. Especially work in a warehouse, in particular order picking, can be tiresome and boring. But this is a very important part because the most labor and costs in a warehouse emerge in the area of order picking [1]. Goetschalckx and Ashayeri [2] define order picking as “the activity by which a small number of goods is extracted from a warehousing system to satisfy a number of independent customer orders”. Costs for order picking arise not only from wages but also from errors in the processing of orders, such as picking the wrong articles or the wrong number of articles. Activities such as order picking are very monotonous and repetitive and thus very demotivating in the long run. This leads to very high staff turnover rates and thus to rising costs for hiring new employees [3].
One way to create motivation in such situations is gamification. Deterding et al. [4]
define gamification as “the use of game design elements in non-game contexts”.
Using gamification to increase performance in production and logistics, e.g. a warehouse
setting, has recently become a rising topic in research as well as in practice [5].
However, work can also motivate on its own, especially if a flow condition is achieved [6].
One way to create or promote a flow condition is gamification [
Also supporting workers in order picking by using augmented reality is another
way to fight the decrease of motivation. A very common definition for augmented
reality defines it as a “form of virtual reality where the participant's head-mounted
display is transparent, allowing a clear view of the real world” [
The aim of this study is to test two game design elements, leaderboards and badges, individually using augmented reality to determine possible differences in their effect on motivation and performance of warehouse workers and eventually getting positive results.
The rest of our article is structured as follows: In section two we give a brief overview of the related work and we conclude this section with our hypotheses. The next section describes the study in detail, giving information about the procedure of our experiment, measurements and the subjects which participated. The results follow in the fourth section. In the last section, the results are discussed and pointed to limitations and possible future research. 2
Previous studies have already shown that badges and leaderboards have a motivating
effect [
In order to understand the effect of individual game design elements correctly, they should also be tested in different contexts. Also none of the previous studies deals with the individual application of badges or leaderboards in logistics. The present 1 https://en.wikipedia.org/wiki/Microsoft_HoloLens - retrieved 20 March 2019 work is intended to provide new insights into this research field combining it with augmented reality.
A study investigating the use of gamification in order picking was conducted by
Sailer et al. [
In a second study Sailer et al. [
Passalacqua et al. [
However, it is not possible to derive the effect of the individual elements in the context of warehouse work from one of these studies. The present article aims to address the aforementioned research gap by investigating leaderboards and badges individually in the context of order picking.
In line with previous findings [
To test our hypotheses, we conducted a between-subject experiment. The independent variable is the use of the different game design elements: badges and leaderboard (badges vs. control condition and leaderboard vs. control condition). The dependent variables were user motivation (measured through a basic psychological needs questionnaire) and performance (time needed to complete the task). 3.1
We performed the experiment with an order picking application for Microsoft HoloLens, which we developed for performing this research, followed by a questionnaire asking different questions about the satisfaction of basic psychological needs, experimental manipulation check and demographic data like age and sex. HoloLens are augmented reality glasses that allow users to see and interact with virtual information directly in their field of view. This will enable e.g. workers in a warehouse to pick orders without holding a device or piece of paper in their hands, which makes the work much easier. We created three different versions of the order picking application. Two versions display game design elements, one badges and another one a leaderboard. In the control condition, no game design elements were presented.
In each version of our experiment users have to complete a short tutorial explaining the task and teaching how to use the HoloLens. Upon clicking on a button to start the experiment, subjects are randomly assigned to one of the three experimental conditions. In addition, the subjects are asked to enter their name or a pseudonym. The name or pseudonym is required in the condition with the leaderboard, where it can be displayed. The subjects are explained the name/pseudonym would be essential for linking the data from the experiment with those of the subsequent survey. During the experiment, all subjects have to pick ten fixed orders. These are identical for all subjects. The goal is to complete all orders as fast as possible without neglecting accuracy. Each order contains three to five predefined products, which are located in shelfs on a test course. The course consists of several shelves in which different products are placed. In addition, there is a fixed starting point at which the subjects get new orders and a new box for each order in which the picked products are stored. Finished orders have to be delivered back to starting point. When a new order is executed, the subjects takes a box and follow a green path (see Figure 1 left), which is displayed via the HoloLens, guiding to the first product in the order. The shelf, in which the next product is located, is marked with a red frame (see Figure 1 right). If the subject reaches this shelf and clicks on the red frame, a text will appear to inform the subject to activate the QR code scanner in the HoloLens. After removing the product from the shelf, the subject scans the product, confirming the correctness of the removal. After the product has been scanned successfully and put to the carrying box, the subject is shown the path to the next shelf. Once the subject has collected all relevant products, the green path will guide the subject back to the starting point. There the subject confirms the delivery of the order and receives a new order and a new empty box. If the subject has fulfilled all ten orders, the message “Thank you for your participation!” appears and the subject can take off the HoloLens. In the leaderboard condition, subjects can see a big blue virtual screen next to the starting point (see Figure 2 right). On the screen they can compare their current score to four fictitious subjects in a leaderboard. The ranking is sorted by the time (in seconds) taken to complete the last task. The leaderboard is updated after each completed order. If the subject reaches a place on the leaderboard, the name/pseudonym entered during the tutorial will be displayed in green color surrounded by two arrows on the leaderboard. By using self-chosen names or pseudonyms, we expect the subjects to be more emotionally involved. To avoid a confounding effect of the positioning on the leaderboard on subjects’ motivation, all subjects are shown the same ranking. Only their position on the leaderboard can change according to their performance. The remaining slots always remain identical. The scores, e.g. time, on the leaderboard were determined on the basis of a brief pre-study.
Similar to the leaderboard condition, subjects in the badge group will also see a blue board at the starting point (see Figure 2 left). The board shows the badges collected during the experiment. In addition to the visual representation on the board, the badges also appear for three seconds in the middle of the field of view of the HoloLens as soon as they are unlocked and an acoustic signal in the form of a short jingle can be heard. The badges were placed directly in the center of the field of vision, as it was determined during a test phase that they were otherwise undetectable. This is partly due to the very small field of view of the HoloLens.
A total of 10 different badges were designed for the experiment. Four of the badges are awarded for picking certain products. Two other badges are tied to a time condition. Three of the badges were built on top of each other and are awarded for walking a predefined distance. The last badge is awarded when a subject has processed all orders correctly. The badges were designed in such a way that they are distributed as evenly as possible over the entire course of the experiment. In order not to neglect the actual task in favor to obtain as many badges quickly as possible, the subjects are not informed in advance how the badges are awarded. Only when a badge has been unlocked, then it can be clicked on the board and a text with the information how the badge was awarded will be displayed.
After the HoloLens experiment the subjects are asked to complete a survey. In order to minimize any influence on the subjects by the experimenter, the questionnaire is provided on a separate computer and answered by the subjects alone without the presence of the experimenter. The purpose of the survey is to determine the motivation of the subjects. It also asks about demographic data of the subjects such as age, height, employment and gender. At the end of the questionnaire, the subjects can finally provide general feedback and comments on the experiment in a free-text field. 3.2
Performance was measured by tracking the time each subject needed to complete the
orders. This was done automatically by the HoloLens application. Additionally a
survey was used to measure the motivation of the users during the task. We adapted the
survey items from the study by Sailer et al. [
A total of 80 subjects took part in the experiment and completed the survey. As a reward for their participation, the subjects received sweets. However, this remuneration was not explicitly promoted and not stated in the beginning of the experiment. Seven subjects had to be excluded from the analysis because the data was not completely stored in the log file due to technical issues and therefore not available.
Since the motivation of the subjects could only be measured if they were not seriously distracted from their experimental condition, we conducted a manipulation check for each game design element. After that ten subjects had to be excluded because they did not meet this criterion, e.g. they could not name the correct experimental condition, after asking them if they observed a leaderboard, badges or nothing. Four subjects were removed from the leaderboard condition, five from the badge condition and one from the control group.
Thus, the sample for statistical analysis consists of N = 63 subjects. Of these, 35 are students (55.5 %), 27 employees (42.9 %) and 1 self-employed individual (1.6 %). 32 of the subjects were men (50.8 %) and 31 women (49.2 %). The mean age is 30.92 years (SD 10.84). The youngest individual is 19 years old and the oldest is 60 years old. In total, 9 of the subjects (14.3 %) already had experience in order picking. 4
The distribution of subjects to the three experimental conditions as well as the mean,
median and standard deviation of the time in seconds that subjects needed to complete
the experiment are given in Table 1. The fastest subject only needed 8.35 minutes to
finish the experiment, the slowest subject needed twice as long with 19.47 minutes.
According to the Shapiro-Wilk test [
The comparison between the control group (CG) (Mdn = 2.00) and the group with the leaderboard (Mdn = 2.17) shows for experience of decision freedom a statistically non-significant result: U = 129.00, z = -1.19, p = .117 (one sided), r = -.19. Also the comparison between the CG and the group with badges (Mdn = 2.00) does not become statistically significant for this variable: U = 250.00, z = -1.02, p = .155 (one sided), r = -.14. Regarding the experience of task meaningfulness, there is no statistically significant result for comparing the means between CG (Mdn = 6.50) and the group with the leaderboard (Mdn = 5.83): U = 115.00, z = -1.62, p = .053 (one sided), r = -.26. The comparison between the CG and the group with badges (Mdn = 6.00) is also not statistically significant: U = 290.00, z = -.20, p = .419 (one sided), r = -.03. In summary, it can be stated that no influence of the two examined game design elements on the feeling of autonomy of the subjects could be determined.
The comparison of the mean values for the experience of competence becomes statistically significant between the CG (Mdn = 5.00) and the group with the leaderboard (Mdn = 3.88): U = 103.00, z = -1.97 p = .025 (one sided), r = -.32. However, no statistically significant result is obtained for the comparison between CG and the group with badges (Mdn = 5.50): U = 271.00, z = -.58, p = .281 (one sided), r = -.08. Interestingly, the results are contrary to the established hypothesis. As it can be seen in Figure 3, the mean of the CG is higher than the value of the leaderboard group. The leaderboard thus had a negative impact on the experience of competence.
Regarding the feeling of relatedness, the means between the CG (Mdn = 2.17) and the group with the leaderboard (Mdn = 1.33) differ statistically significant: U = 106.50, z = -1.72, p = .043 (one sided), r = -.28. The comparison of the means between the CG and the group with badges (Mdn = 1.00) also becomes statistically significant for this variable: U = 170.50, z = -2.29, p = .011 (one sided), r = -.33. Nevertheless, one can also see here, considering the descriptive results, that the mean of the CG is highest. Subjects who have received the leaderboard or badges, felt less socially related than subjects who could see no game design elements.
Since an overarching increase in the four motivation variables was not found in either the leaderboard or badge group compared to the CG, both hypotheses H1a and H1b are discarded. 4.2
With regard to the task completion time, the Shapiro-Wilk test yields a statistically significant result with p = .007. Thus, there is again no normal distribution of the data and the Mann-Whitney U test is used again as method for statistical analysis.
The comparison of the means between the CG and the group with the leaderboard yields one-sidedly tested a statistically significant result U = 113.00, z = -1.66, p = .048, r = -.27. It can be assumed that subjects who receive a leaderboard accomplished the experiment faster than subjects who haven’t seen a game design element. This supports hypothesis H2b.
A comparison of the means of the CG with those of the group with badges does not yield a statistically significant result U = 278.00, z = -.44, p = .330 (one-sided), r = -.06. Consequently, H2a cannot be supported because it is assumed that subjects of the group with badges did not accomplish the experiment faster than subjects who did not get a game design element displayed.
Recent studies show that the effect of gamification can be influenced by both age
and gender [
In the leaderboard group a strong positive correlation was found between task completion time and age, r = .698, p = .006. The results indicate that younger subjects accomplished the tasks faster than older subjects. For the group with badges no correlation between the age of the subjects and the task completion time could be determined, r = .257, p = .215. In the control group also no correlation between age and task completion time could be observed, r = .129, p = .547. Since a correlation only occurs in the group with the leaderboard, the assumption can be made that leaderboards may have a stronger effect on younger individuals.
For both groups a strong correlation could be determined between the gender of the subjects and the task completion time: leaderboard r = .573, p = .032; badges r = .527, p = .007. Since men were coded with zero and women with one, it could be deduced from the results that men accomplished the tasks faster than women. However, the influence of gender can be traced back to the height of the subjects. A partial correlation between sex and task completion time, taking into account the height of subjects, revealed no correlation between the two factors: leaderboard r = .227, p = .360; badges r = .179, p = .404. The mean value for the height is 183 cm (SD 6 cm) for men and 167 cm (SD 8 cm) for women. Since men are taller on average, they have a bigger step length and can therefore accomplish the tasks faster. 5
Our motivation for the present study was to assess how leaderboards and badges
affect subjects’ performance (completion time) and motivation in an augmented reality
order picking setting. Contrary to the results of previous studies [
The findings of the present study suggest that leaderboards are an effective way to influence user behavior. However, the positive effect on the task completion time achieved by the leaderboard could not be replicated for the badge element. One reason that the badges did not work as expected could be that they distracted the subjects. This is supported by observing the subjects, who stopped between the orders to have a look at the board with the badges they had achieved. Possible positive effects on the task completion time of the subjects could thus have been lost.
It was not possible to determine the expected positive effect on the motivation of the subjects either in the group with the leaderboard or in the group with badges. Contrary to expectations, even negative effects on psychological needs could be recorded. Both the leaderboard and the badges had a negative effect on the feeling of relatedness. The leaderboard also had a negative influence on the subjects' perceived experience of competence. We interpret these results as such that the design of the leaderboard as a competitive leaderboard (in contrast to e.g. a cooperative team leaderboard) may have led to these effects. 5.1
The present study has limitations, which offer starting points for future research. First, we only examined the short-term effects of leaderboards and badges. Since the augmented reality technology used is still very novel, most of the subjects were very fascinated by this. This could blur the positive effect of gamification. Therefore, it is necessary to repeat the experiment with a significantly longer duration.
Also the difference between the two conditions should be noted. It is possible to use goal-setting behavior in the leaderboards condition but not in the badges condition. The badges were implemented in such a way that subjects could not see the criteria for unlocking in advance. In other words, each awarded badge would be a surprise to the subject. This means that the badge system, unlike the leaderboard system which displayed task completion times, did not allow subjects to set goals. We do not see this issue as a thread to the validity of our results but we acknowledge that the comparison between the two conditions is not straightforward. For future studies we suggest to provide the badges not as a surprise. But this might lead to even longer task completion times, because the subjects could spend at the beginning too much time to find out, how to unlock the different badges. Future studies must clarify these issues.
Another possibility would be to carry out a similar study under real conditions in practice. On the one hand, it could also be determined under real conditions whether an influence on the error rate is detected, which is also an important cost factor in order picking. On the other hand, it would be possible to study individuals who are familiar with the job and are therefore likely to be less motivated by the novelty of the task, as may have been the case with the subjects in this study.
Since the results concerning the influence of age on the effect of gamification was only a byproduct of this study but offered interesting and surprising results, the effect of this factor should be considered in a future study. Additionally the attitude towards new technologies might be an interesting confounding variable, worth to conduct analysis using technology acceptance questionnaires.
Finally what should be kept in mind is that subjects have repeatedly stated that HoloLens is very uncomfortable to wear. This circumstance could have had a negative effect on the motivation and satisfaction of the subjects. For subsequent studies, the use of more comfortable data glasses of the upcoming generation is recommended. 5.2
The results show that the use of game design elements has to be done with caution in order to avoid unwanted or negative effects. Leaderboards showed significant effects for performance but negative for motivation. Badges instead were superior for the feeling of competence but fell short in terms of performance. These results were only possible through the experimental design and comparison of individual game design elements as well as taking into account social and context-related influences. This shows the importance of comparing individual game design elements.
Though leaderboards showed positive effects on performance, motivation might be endangered and could have in the long run undesirable effects, e.g. leaderboards can harm the feeling of relatedness if competition is the main focus of the game design element. But also applying a very common game design element like badges can be no panacea, if context is not considered completely, e.g. distraction by novelty. Gamification can therefore be no solution without considering context.