Gamification of an open access quiz with badges and progress bars: An experimental study with scientists

Abstract

Gamification is the use of game-design elements in non-game contexts. Gamification has been used in many different contexts to motivate different audiences, but the effect on scientists has not yet been sufficiently investigated. In a field study with 28 scientists the effects of the game design elements badge and progress bar are examined for their potential to raise motivation for scientists to answer questions in the context of an open access quiz. Previous findings show that engaging with the topic open access can be perceived by scientists as “boring”. By gamifying a quiz about open access with badges or a progress bar we aim to create incentives for scientists to engage with the topic. In our study both game design elements provide a statistically significant increase in the number of questions answered compared to a control group.

Keywords

• Gamification
• Open Access
• Motivation
• Badge
• Progress Bar
• Scientists

Introduction

Gamification refers to the use of game design elements in a context other than that of a game [ 6 ]. The concept of gamification has meanwhile established itself in the field of human-computer interaction [ 31 ]. In various application cases from fitness and health [ 19, 28 ] to working environments [ 18, 32, 36 ] to education and training [ 4, 17 ] it could be shown that gamification has a positive influence on increasing motivation and performance. But how gamification can be used to motivate scientists, who mostly work under special conditions and have special criteria for the evaluation of their performance, has not yet been sufficiently researched [ 10 ].

Usually work can have the potential to motivate on its own, especially if a flow condition is achieved [ 5 ]. Gamification is a method to create or promote a flow condition [ 15 ]. But in contrast to flow theory, which requires full concentration to get into a flow condition [ 5 ], gamification works with both conscious and unconscious perceived game design elements.

Gamification studies with scientists are rare [ 10 ] and usually not in an experimental setting [ 11 ]. This is also the case for gamification studies with the context of open access. In addition scientists want to focus on other things and not on open access, because they consider this topic as complex and boring [ 20 ]. However this topic is of highest importance for all scientists who publish their publicly funded results [ 8 ].

The aim of this article is to examine the impact of the game design elements badge and progress bar on scientists in the context of an open access quiz. In addition this is a first contribution to research the effect of gamification on scientists with an experimental approach.

The rest of our article is structured as follows: An overview of the current state of research is given in the following section. Section three describes the study in detail, giving information about the procedure of our experiment, measurements and the subjects which participated. The results follow in the fourth section. In the last section, the results are discussed and pointed to limitations and possible future research. 2

Related work

Gamification is used in science to promote participation in citizen science projects such as Galaxy Zoo or Foldit [ 2, 9, 30 ]. Through the use of game design elements, incentives are created so that users generate new solutions or classify images and not only once but preferably on a long-term basis.

Kidwell and colleagues provide empirical evidence how badges can motivate scientists to publish scientific results according to open science principles. The authors showed that the award of open science badges motivates scientists to make their research data freely available [ 21 ].

Scientific platforms such as ImpactStory and ResearchGate also use game design elements such as badges or progress indicators to motivate scientists. Hammarfelt and colleagues [ 16 ] look at the impact of gamification on these platforms. The authors suspect that the use of game design elements creates a motivating feeling when publications and online interactions with other users are transformed into points. They also assume that their own position within the academic community can be more easily defined through gamification. Scientific progress and comparison with other users is thus facilitated by converting publications into points [ 16 ].

Feger and colleagues address the question of how gamification can be used in science workplaces [ 10 ]. They present problems and challenges, such as the aspect that progress in scientific work is difficult to quantify. In a follow-up study, based on an extensive survey of high energy physicists, they developed various gamified approaches to motivate scientists to provide reproducible data [ 11 ].

Mekler [ 27 ] has already been able to show in a study on points, levels and leaderboards that individually applied game design elements can have an effect on behaviour. For the present study, the two game design elements badge and progress bar were selected in order to examine their effects individually and thus assess their potential to encourage scientists to answer questions in a quiz in the context of open access. Our study is a preliminary study of a larger study with the aim of improving the impact of gamification as a tool for disseminating and improving knowledge about open access. Both game design elements are used to build nonfiction gamified experiences [37] and are easier to reuse in other contexts than narratives, for example.

Badges in the context of games and gamification are digital artefacts which are given to the user for the fulfilment of certain tasks and are therefore a visual representation [ 1 ]. They can be used to create a comparison with others or to challenge oneself [ 13 ]. The game design element badge has already proven itself in other studies as an effective instrument for increasing user activity [ 14, 26 ]. Morris et al. [ 29 ] could already show with an experimental study that the use of badges and goals has no influence on the acquired knowledge. Building on this, the present study will not examine any learning effect, but only the motivation of the subjects to engage with a specific topic for longer period of time. In our context this topic is open access.

In contrast to badges, the progress bar element is a less frequently studied game design element in gamification [ 22, 34 ]. It is a simple visual way of informing the users about their progress. Initial studies have shown that even this simple form of feedback can motivate users [ 26 ]. Besides games, this easily implemented element is also used in many other areas. These include, for example, software to show a progress in an installation process or in surveys to give the subjects an overview of how many questions he or she still has to answer. Despite the general assumption that progress bars should increase the completion rate of surveys, studies have not been able to prove this effect [ 7 ] or even produced opposite results [ 23 ]. However, we have shown in a previous study that progress indicators can have a positive motivational effect when answering a general quiz by a general audience [ 26 ]. Therefore we assume that this positive effect is expectable as well with scientists and that they can be motivated by the game design element progress bar to answer more questions in an open access quiz. 3

Method

The aim of our study is to investigate whether scientists can be motivated by a gamified quiz to answer more questions on the topic of open access. To this end, a field study in the form of an online quiz on the topic of open access was developed. A between-subjects design was chosen to determine the effect of the individual game design elements. The following three experimental conditions were used for the experiment: a control group (CG) without gamification, an experimental group with progress bar (PB) and an experimental group with badges (BA). 3.1

Hypotheses

In line with previous findings [ 14, 25, 26 ] the following two hypotheses were formulated:

H1: The subjects in the group with progress bar (PB) answer on average more open access questions than the control group (CG).

H2: The subjects in the group with badges (BA) answer on average more open access questions than the control group (CG). 3.2

Procedure

To answer our hypotheses we developed an experimental setting with a quiz. The quiz is based on the idea that questions are only answered as long as the subjects enjoy it. This has been explicitly stated to the subjects in the invitation e-mail and at the first page of the experiment. There is no minimum number of questions to answer. Instead the quiz can be finished at any time. Subjects are randomly and permanently assigned to one of the three test conditions (CG, BA or PB). All subjects reached the online quiz through the same internet URL. Due to the random division into groups, the subjects were not aware that there is more than one test condition. At the end of the experiment, the mean number of questions answered is compared between the three groups.

The quiz developed for the field study consists in total 29 multiple-choice questions on the topic of open access. Each question is provided with four possible answers of which always only one is correct. In all three test conditions, the subjects are immediately informed of the correct or incorrect answer to a question. The correct answer is highlighted in green colour and wrong answers are displayed in red colour (see Fig. 1). All questions are always asked in the same order, so that we can compare the results between the different groups.

All questions address different aspects of open access. In a pre-test before conducting the experimental field study, five subjects were asked to give feedback about our online quiz. All questions were described as "interesting" and "important". At this point, the questions were tested exclusively with the non-gamified version of the quiz. This form of presentation of the questions at the pre-test was found to be "monotonous" by the subjects. This supports the findings of Kelty [ 20 ] and provides the prerequisites for using gamification to increase motivation to engage users for a significantly longer period of time with the topic.

The badges are displayed to the user above the questions during the quiz (see Fig. 1 right). Before a badge is unlocked, only a grey circle is visible in the overview. So the subjects can estimate that further badges can be achieved. This can appeal to the collective instinct [37] and thus motivate to continue with an activity. The badges were designed in such a way that they can be unlocked as consistently as possible in the course of the experiment. The subjects are not informed in advance about the criteria for the award of the badges. This was done on the purpose not to encourage the subjects to just unlock the badges instead of engaging extensively with the questions. In addition, unexpected unlocking can have a motivating effect on people who react positively to surprise effects [37]. Just after a badge is unlocked the condition for its assignment is disclosed and the associated visual badge is displayed.

Eight different badges were designed for the quiz. One of the badges is bound to a time condition and is activated for answering a question within five seconds. Three of the badges are awarded for a certain number of questions answered correctly one after the other. Three more badges are given for correctly answering certain individual questions and the last badge is assigned for answering all questions in the quiz.

A simple horizontal progress bar was used for the other game design element (see Fig. 1 left). However, the progress bar is not filled equally, but in uneven proportions. This procedure is based on the assumption that the maximum number of questions cannot be predicted easily and that this vague approach might motivate the subjects [ 24 ]. If the maximum number of questions is known, the subjects might be additionally motivated to answer more questions, which would lead to a confusion of the results, as this would set an implicit goal. This would make it impossible to attribute our results only to the game design elements because the ceteris paribus assumption would not be applicable due to the intervening factor of the additional variable, more specifically the implicit goal. It is also known from research on the influence of progress indicators on the completion rate of surveys that they can also be demotivating if they give the participant the feeling he or she is not progressing fast enough [ 4 ]. The variation in progress should add a random factor that is more motivating than a monotonous increase in the display and still reflect progress.

At the beginning of the quiz the subjects were also asked to indicate their age and gender and after completing the quiz they were given the opportunity to leave a comment in a text field. The subjects did not receive any compensation for their participation in this experiment. 4

Results

Over a period of one week, 28 subjects were acquired at three German institutes who are also project partners of the forthcoming larger study. Only employees of the institutes were contacted and no students. They received an e-mail with the request to participate in a study and could access the quiz anonymously by clicking on a general link to the page where the quiz was provided. Both in the e-mail and on the website the subjects were informed that they only had to answer questions for as long as they enjoyed it. A total of three volunteers did not report their gender. Of the other subjects, 12 were male and 13 female. The mean age was 38.57 years (standard deviation 13.90). The experiment presented here is a preliminary study of a larger study. This explains the short runtime and the relatively small number of subjects.

On average, 23.32 of 29 maximum possible questions were answered (standard deviation 9.24). 18 subjects (64 %) answered all 29 questions, of which three were in the control group (corresponding to 27 % in this group), six in the group with badges (100 %) and nine in the group with progress bar (81 %). The evaluation thus shows a ceiling effect that is particularly pronounced in the two experimental conditions. The discussion can already be anticipated and additional questions should be developed for further use of the open access quiz. This can also be seen from the text field comments of the subjects, which at the end of the experiment could be submitted on a voluntary basis. Several respondents from all three groups noted that they would have liked to have answered further questions.

A list of the number of subjects per test condition, as well as the number of those who answered all questions, the mean value of answered questions and the associated standard deviation are given in Table 1.

An analysis of variance is performed for statistical evaluation. The test to check the homogeneity of the variances (Levene test) for the number of answered questions yields a statistically significant result with p = .001, the Levene statistics is 8.78. All following results are therefore based on an unequal variance and the corrected results are reported accordingly.

The analysis of variance yields a statistically significant difference between the individual test conditions, F(2, 25) = 6.47, p = .005. Since the homogeneity of the variances is not given, the Welch test must correct accordingly. The comparison of the mean values of the control group with the group with progress bar provides a statistically significant result, t(18.19) = 2.67, p = .008. Consequently, the hypothesis H1 can be supported and it can be assumed that the progress bar motivated the subjects to answer more questions.

The comparison of the mean values between the control group and the experimental group with badges also shows a statistically significant result, t(10.00) = 4.04, p = .001. It can thus also be supported hypothesis H2 that the game design element badge also motivated the subjects to answer further questions.

The number of subjects in the individual test conditions differs significantly and, in addition, the number of subjects in the test conditions is very low. This can result in conservative effects in their statistical significance due to a low test power [ 12 ]. Therefore, it is all the more remarkable that statistically significant results have been obtained. This is also shown by the effect size of Δ = .99 for the comparison between the control group and the progress bar. The effect size for the comparison between the control group and the group with badges is Δ = 1.22. Thus a strong effect can be demonstrated for both comparisons. For such a small sample size and the problems normally associated with this size [ 12 ], the results are particularly remarkable. 5

Discussion and future work

The present study was able to show that the game design elements progress bar and badge had a motivating effect in the context of an open access quiz. Based on the results of Kidwell et al. [ 21 ], the assumption can thus be supported that the game design elements can also be used in a different context in order to generate or increase motivation for a gamified quiz in the context of open access.

Following on the results of Mekler et al. [ 27 ], we were also able to observe an effect of individual game design elements on the behaviour of the subjects in our study. We could show for the two game design elements badges and progress bar that they can individually influence behaviour in comparison to a control group. These results are also consistent with the results of Hamari [ 14 ] who could show also positive effects for badges in the context of a sharing community.

Our result that a positive effect on the number of questions answered was achieved in contrast to the studies addressing with the use of progress bars in surveys [ 7, 23 ] could be attributed to various factors. For example this could be due to the relatively small number of questions. Despite the manipulation of the progress bar, the subjects could anticipate that they were progressing relatively quickly. Another point could be the quiz setting, which differs from a classical survey by stating the correct or wrong answers on a question. It is possible that this additional element could positively influence the effect of the progress bar, as it is assumed that game elements influence each other [ 32 ].

The assumption of Feger et al. [ 10, 11 ] that gamification is also suitable to address the very special target group of scientists, can also be supported on the basis of the results of our preliminary study. Nevertheless, the practical feasibility of this idea should be investigated in further experiments.

Despite the successful implementation of the field study, there are some limitations. First of all the small number of subjects is basically a problem for every field study. In the present context this can be explained because the study was scheduled to take one week to complete and that it is a preliminary study of a larger study.

The restriction that the data were collected exclusively in Germany could also have had an influence on the results. Maybe the level of knowledge and implementation of open access makes a distinction in some cases between Germany and other countries. Also cultural factors may have an influence on the impact of gamification [ 35 ].

In addition, for further investigations of different game design elements with the same research design, more questions need to be developed to avoid the ceiling effect present in this study, which occurred especially in the experimental conditions. This is also in line with the comments from our subjects.

Another point that should be further examined is whether a cooperative design of the game design elements would be preferable in the scientific context. According to Feger et al. [ 11 ], the design of gamified scientific work environments should rather be based on a cooperative approach, as the scientists surveyed seemed to prefer such a design. Although the present study did not include a cooperative element we achieved very positive results. In order to be able to demonstrate possible advantages of cooperative design methods, an experimental investigation of cooperative gamification approaches would have to be carried out in the scientific context.

Also it should be examined whether an online quiz with additional questions is too artificial to approach the question of how open access can be made more motivating. Instead, another way could be to let subjects make fictitious selection decisions in an experiment (conjoint analysis), similar to a procedure described by Schöbel and colleagues [ 33 ]. In such an experiment, for example, scientific articles could be sorted in an order in which they would be prioritized for download. Different aspects of open access could be visualised and highlighted by gamification, which is not possible with a quiz or a survey. A further possibility for experimental research could be to offer different publication options for one's own publication and to evaluate these according to the probability of submission by the subjects. The variants could differ with regard to the number, design and quality of game design elements and bibliometric data.

Acknowledgments References

This research was funded by the Federal Ministry of Education and Research (BMBF) in Germany as part of the research project OA-FWM (16OA044C).

References

1. Antin , J. , Churchill , E.F. : Badges in Social Media: A Social Psychological Perspective . In: CHI 2011 Gamification Workshop Proceedings . pp. 1 - 4 , Vancouver, British Columbia, Canada ( 2011 ).

2. Bowser , A. et al.: Using Gamification to Inspire New Citizen Science Volunteers . In: Proceedings of the First International Conference on Gameful Design , Research, and Applications - Gamification '13. pp. 18 - 25 ACM Press, Toronto, Ontario, Canada ( 2013 ).

3. Buckley , P. , Doyle , E.: Individualising Gamification: An Investigation of the Impact of Learning Styles and Personality Traits on the Efficacy of Gamification Using a Prediction Market . Computers & Education . 106 , 43 - 55 ( 2017 ). https://doi.org/10.1016/j.compedu. 2016 . 11 .009.

4. Conrad , F.G. et al.: The Impact of Progress Indicators on Task Completion. Interacting with Computers . 22 , 5 , 417 - 427 ( 2010 ). https://doi.org/10.1016/j.intcom. 2010 . 03 .001.

5. Csikszentmihalyi , M. : Flow: The Psychology of Optimal Experience . Harper Collins, New York, NY, USA ( 2009 ).

6. Deterding , S. et al.: From Game Design Elements to Gamefulness: Defining “Gamification.” In: Proceedings of the 15th international academic MindTrek conference: Envisioning future media environments . pp. 9 - 15 ACM Press ( 2011 ). https://doi.org/10.1145/2181037.2181040.

7. Dirk Heerwegh , Geert Loosveldt: An Experimental Study on the Effects of Personalization,Survey Length Statements, Progress Indicators, and SurveySponsor Logos in Web Surveys . Journal of Official Statistics . 22 , 2 , 191 - 210 ( 2006 ).

8. European Commission : Open access - H2020 Online Manual , https://ec.europa.eu/research/participants/docs/h2020-funding -guide/cross-cuttingissues/open-access-data-management/open-access_en .htm, last accessed 2019 /10/09.

9. Eveleigh , A. et al.: “I Want to Be a Captain! I Want to Be a Captain!”: Gamification in the old Weather Citizen Science Project . In: Proceedings of the First International Conference on Gameful Design , Research, and Applications - Gamification '13. pp. 79 - 82 ACM Press, Toronto, Ontario, Canada ( 2013 ). https://doi.org/10.1145/2583008.2583019.

10. Feger , S. et al.: Just Not The Usual Workplace: Meaningful Gamification in Science . Gesellschaft für Informatik e.V. 113 - 118 ( 2018 ). https://doi.org/10.18420/muc2018-ws03- 0366.

11. Feger , S.S. et al.: Gamification in Science: A Study of Requirements in the Context of Reproducible Research . In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems - CHI '19 . pp. 1 - 14 ACM Press, Glasgow, Scotland UK ( 2019 ). https://doi.org/10.1145/3290605.3300690.

12. Field , A.P. : Discovering Statistics Using Ibm Spss Statistics: And Sex and Drugs and Rock “N” Roll . SAGE, London, UK ( 2013 ).

13. Gibson , D. et al.: Digital Badges in Education . Educ Inf Technol . 20 , 2 , 403 - 410 ( 2015 ). https://doi.org/10.1007/s10639-013-9291-7.

14. Hamari , J.: Do Badges Increase User Activity? A Field Experiment on the Effects of Gamification. Computers in Human Behavior . 71 , 469 - 478 ( 2015 ). https://doi.org/10.1016/j.chb. 2015 . 03 .036.

15. Hamari , J. , Koivisto , J.: Measuring Flow in Gamification: Dispositional Flow Scale-2 . Computers in Human Behavior. 40 , 133 - 143 ( 2014 ). https://doi.org/10.1016/j.chb. 2014 . 07 .048.

16. Hammarfelt , B.M.S. et al.: Quantified Academic Selves: The Gamification of Science Through Social Networking Services. Information Research . 21 , 2 , ( 2016 ).

17. Hanus , M.D. , Fox , J. : Assessing the Effects of Gamification in the Classroom: A Longitudinal Study on Intrinsic Motivation, Social Comparison , Satisfaction, Effort, and Academic Performance. Computers & Education . 80 , 152 - 161 ( 2015 ). https://doi.org/10.1016/j.compedu. 2014 . 08 .019.

18. Huschens , M. et al.: On the Role of Social Comparison Processes in Gamified Work Situations . In: Proceedings of the 52nd Hawaii International Conference on System Sciences - HICSS 52 . pp. 1446 - 1455 ( 2019 ).

19. Johnson , D. et al.: Gamification for Health and Wellbeing: A Systematic Review of the Literature . Internet Interventions . 6 , 89 - 106 ( 2016 ). https://doi.org/10.1016/j.invent. 2016 . 10 .002.

20. Kelty , C. : BEYOND COPYRIGHT AND TECHNOLOGY: What Open Access Can Tell Us about Precarity , Authority, Innovation, and Automation in the University Today. Cultural Anthropology. 29 , 2 , 203 - 215 ( 2014 ). https://doi.org/10.14506/ca29.2.02.

21. Kidwell , M.C. et al.: Badges to Acknowledge Open Practices: A Simple, Low-Cost, Effective Method for Increasing Transparency . PLoS Biol . 14 , 5 , e1002456 ( 2016 ). https://doi.org/10.1371/journal.pbio. 1002456 .

22. Koivisto , J. , Hamari , J. : The Rise of Motivational Information Systems: A Review of Gamification Research . International Journal of Information Management . 45 , 191 - 210 ( 2019 ). https://doi.org/10.1016/j.ijinfomgt. 2018 . 10 .013.

23. Liu , M. , Wronski , L. : Examining Completion Rates in Web Surveys Via Over 25 ,000 Real-World Surveys. Social Science Computer Review. 36 , 1 , 116 - 124 ( 2018 ). https://doi.org/10.1177/0894439317695581.

24. Marczewski , A. : Gamification : A Simple Introduction. Lulu . Com ( 2013 ).

25. Mazarakis , A. , Bräuer , P. : Badges or a Leaderboard? How to Gamify an Augmented Reality Warehouse Setting . In: Proceedings of the 3rd International GamiFIN Conference - GamiFIN 2019 . pp. 229 - 240 ( 2019 ).

26. Mazarakis , A. , Bräuer , P. : Gamification is Working, but Which One Exactly? : Results from an Experiment with Four Game Design Elements . In: Proceedings of the Technology, Mind, and Society . p. 22 : 1 ACM , New York, NY, USA ( 2018 ). https://doi.org/10.1145/3183654.3183667.

27. Mekler , E.D. et al.: Towards Understanding the Effects of Individual Gamification Elements on Intrinsic Motivation and Performance . Computers in Human Behavior . 71 , 525 - 534 ( 2015 ). https://doi.org/10.1016/j.chb. 2015 . 08 .048.

28. Miller , A.S. et al.: A Game Plan: Gamification Design Principles in Mhealth Applications for Chronic Disease Management . Health Informatics Journal. 22 , 2 , 184 - 193 ( 2016 ). https://doi.org/10.1177/1460458214537511.

29. Morris , B.J. et al.: Comparing Badges and Learning Goals in Low- and High-Stakes Learning Contexts . J Comput High Educ . 31 , 3 , 573 - 603 ( 2019 ). https://doi.org/10.1007/s12528-019-09228-9.

30. Ponti , M. et al.: Science and Gamification: The Odd Couple? In: Proceedings of the 2015 Annual Symposium on Computer-Human Interaction in Play - CHI PLAY '15 . pp. 679 - 684 ACM Press, London, United Kingdom ( 2015 ). https://doi.org/10.1145/2793107.2810293.

31. Rapp , A. et al.: Strengthening Gamification Studies: Current Trends and Future Opportunities of Gamification Research . International Journal of Human-Computer Studies. 127 , 1 - 6 ( 2018 ). https://doi.org/10.1016/j.ijhcs. 2018 . 11 .007.

32. Sailer , M. et al.: Fostering Development of Work Competencies and Motivation via Gamification . In: Mulder, M. (ed.) Competence-based Vocational and Professional Education . pp. 795 -818 Springer International Publishing, Cham ( 2017 ). https://doi.org/10.1007/978- 3- 319 -41713-4_ 37 .

33. Schöbel , S. et al.: More Than the Sum of Its Parts - Towards Identifying Preferred Game Design Element Combinations . In: Learning Management Systems . pp. 1 - 12 , Seoul, South Korea ( 2017 ).

34. Sümer , M. , Aydın , C.H. : Gamification in Open and Distance Learning: A Systematic Review . In: Spector, M.J. et al. (eds.) Learning, Design, and Technology. pp. 1 -16 Springer International Publishing, Cham ( 2018 ). https://doi.org/10.1007/978-3- 319 -17727-4_ 115 - 1 .

35. Thom , J. et al.: Removing Gamification from an Enterprise SNS . In: Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work - CSCW '12 . pp. 1067 - 1070 ACM Press, Seattle, Washington, USA ( 2012 ). https://doi.org/10.1145/2145204.2145362.

36. Warmelink , H. et al.: Gamification of the Work Floor: A Literature Review of Gamifying Production and Logistics Operations . In: Proceedings of the 51st Hawaii International Conference on System Sciences . pp. 1108 - 1117 ( 2018 ). Zichermann, G. , Cunningham , C. : Gamification by Design: Implementing Game Mechanics in Web and Mobile Apps. O'Reilly Media , Sebastopol, CA, USA ( 2011 ).