=Paper=
{{Paper
|id=Vol-1358/paper4
|storemode=property
|title=How to Design Experimental Research Studies around Digital Badges
|pdfUrl=https://ceur-ws.org/Vol-1358/paper4.pdf
|volume=Vol-1358
|dblpUrl=https://dblp.org/rec/conf/lak/McDanielF15
}}
==How to Design Experimental Research Studies around Digital Badges==
https://ceur-ws.org/Vol-1358/paper4.pdf
How to Design Experimental Research Studies around
Digital Badges
Rudy McDaniel Joseph R. Fanfarelli
University of Central Florida University of Central Florida
P.O. Box 161990, Orlando, FL 32816 P.O. Box 161990, Orlando, FL 32816
rudy@ucf.edu joseph.fanfarelli@ucf.edu
ABSTRACT Currently, there is little support for new badging
Digital badging is becoming more popular in an assortment of researchers looking to move into this area of scholarship.
disciplines, both academic and professional. Along with the This paper aims to fill that gap by providing
success of practical badging initiatives, badging research is also recommendations for planning and executing a badging
moving at a rapid pace, a rate of change that may be intimidating study.
to the uninitiated wishing to study them. However, there is a great
need for additional research in light of the complexity of badging Badging research often spans multiple disciplines including
and the many contexts in which badging occurs. This paper psychology, computer science, educational technology, and
outlines an approach to designing research studies around digital the visual arts. As a result, some researchers from
badges to assist researchers who are new to the field and looking disciplines not trained in research methods may wish to
to contribute. It begins by discussing how to form relevant learn basic strategies for conducting quantitative research
research questions and how to approach the literature review, so that we can more fully understand badge design and
providing useful references as starting points. It then continues on
to experimental design recommendations, discusses useful
function, in an empirical sense, across academic
practices during experimentation, and concludes with boundaries. While researchers working in disciplines such
recommendations for data analysis. Additionally, this paper as psychology and educational technology may already be
describes challenges that are specific to badging and places them well versed in experimental research design, those working
in context of the research design process. Multiple examples are in other fields may not. To address this issue and to provide
provided to clarify these concepts. context for the unique aspects of badging research, this
paper situates badge-specific recommendations within the
Categories and Subject Descriptors context of general good research practice. Overall, the
A.0 General – Conference proceedings. purpose of this paper is to outline an approach for
General Terms designing and developing badge-based research protocols
Measurement, Performance, Design, Experimentation based on our prior experiences developing and
administering several such studies [3], [4], [5]. This is
Keywords accomplished by discussing the research process as it
Digital Badging, Education, Learning, Motivation, Goal relates to studies specifically designed around digital
Setting, Credentials, Assessment, Experimental Design, badges. Topics discussed include developing appropriate
Research Design. research questions for badges, considering how to develop
dependent variables in these contexts, and recognizing the
1. INTRODUCTION unique characteristics of badge-based data analysis.
Digital badging initiatives have gained traction in multiple
professional domains and for a number of purposes. As 2. DEVELOPING THE RESEARCH
evidenced by their use as digital credentialing technologies QUESTIONS
[1], reward systems [2], and as tools in other scenarios, Badges are used for a variety of purposes, from
badging is an emerging practice containing several incentivizing actions and behaviors to tracking performance
dimensions ripe for study. However, good badging studies outside normal channels of assessment and evaluation.
require researchers to make a number of careful decisions They are also used in a variety of settings and for many
that can seem overwhelming for those new to the subject. purposes, from informal science learning in museums or
wilderness scouting programs to formal coursework in the
public school system and higher education.
Copyright © by the paper’s authors. Copying permitted for
private and academic purposes. Thus, when designing research questions for a badging
In: D. Hickey, J. Jovanovic, S. Lonn, J.E. Willis, III (eds.): study, it is useful to first consider the context and
Proceedings of the Open Badges in Education (OBIE 2015) mechanics of the badging system. Before honing in on
Workshop, Poughkeepsie, New York, USA, 16-Mar-2015, particular research questions, for example, several more
published at http://ceur-ws.org. general areas should first be considered. The following
procedural prompts are useful for thinking about how a which the research is occurring. They should additionally
research study might be generally framed, and then later consider the audiences the investigators will have access to.
operationalized, through more specific questions:
For example, in addition to the unique decisions made
What is the context of the badging system? In other words, regarding the badging process, a study might be designed
how is the badge system being used? Is it deployed in a around the following research questions:
children’s museum for the purposes of informal science
learning? Is the system integrated into a learning • How do badges affect motivation toward group
management system for the purpose of motivating projects in an online course focused on introductory
undergraduate college students? Are badges being used in psychology?
conjunction with a mobile fitness app to encourage healthy • Do students who earn more badges perform better in
lifestyle choices? The overall context for the badging the course, as evidenced by earning higher grades?
system is important to consider up front as a necessary The questions should be specific and measurable and may
precursor to the formation of specific research questions. be explored through different types of research designs. In
Who will design the badges? Will the experimenter design general, although the research questions may be refined
them? As an expert in the literature, the experimenter may after conducting the literature review, the overall purpose
be able to develop the most scientifically sound system. for the research should be decided upon at the beginning of
However, the experimenter is likely not as well-suited for the research process. This purpose will set the stage for the
design tasks such as graphic design or textual layout. In remainder of the protocol design.
addition, design heuristics for badging research are still In addition to the impact and influence of badges, the
lacking. While past studies have developed some researcher might also identify the type of badging system
promising general guidelines, comprehensive blueprints for they wish to study in the formulation of his or her research
successful badge design for specific circumstances do not questions. For instance, if the aim is to compare open badge
yet exist. systems such as Mozilla’s Open Badges to a proprietary
Is it necessary for the designer to have a relationship with badging system developed by a commercial vendor, the
the end users? While a researcher may understand the following research questions might be more appropriate:
current state of badging knowledge, an educator may have • Are students more likely to consider badges useful
a better understanding of the curriculum, the tasks with when they can be permanently displayed outside of the
which her students are having difficulty, and knowledge course, even after completing their coursework?
about what motivates those students. • Are there differences in motivation or engagement
Who is awarding the badges? In other words, what is the toward course modules in students who use badge
central badging authority within a system? Badges can be system A as compared to badge system B?
provided for many reasons. They can be objective (e.g., a Another method of studying badges is in regards to their
learner received a score of 90% or better) or subjective operational components. As Hamari and Eranti [6] explain,
(e.g., a learner demonstrated exceptional effort on an badges can be broken down into three primary components:
assignment). The question of badge authority is especially a signifier, a completion logic, and a reward. These parts
important for subjective badges. Subjectivity, by definition, can each be considered separately within the design of a
is ambiguous and open to interpretation. Thus, the study to investigate badges using a finer degree of
perception of exceptional effort may differ by instructor. granularity. For example, here are two potential research
There are a number of other guiding questions that are also questions focusing on separate components of badging
useful for research design, but these are some of the more according to Hamari and Eranti’s framework:
common prompts likely to be helpful across a variety of • What visual signifiers are most effective for capturing
study types. The ways in which these questions are the attention of players in the badges used within a
answered will influence the way research hypotheses are racing-themed video game?
formed. As always, it is important to be clear and detailed • How does the perceived degree of difficulty within the
in the formation of these experimental research questions. completion logic of puzzle-based games impact player
It is also important to be precise during the pre- enjoyment of those types of games?
experimental phase of the study in regards to the unique
aspects of the badging system under investigation. There is a theoretically infinite number of potential
research questions to be explored by badging studies. This
By formulating clear research questions at the onset, one flexible, purpose-driven research question generation
can better determine the particular approach and process presents opportunities for badging studies to
instruments with which to design the study protocol. The connect with or augment other research in areas such as
research questions should take into account both the sociology, psychology, or digital media and industrial
objectives of the research and the environmental factors in design. For example, while badges are frequently thought
of as rewards or credentialing systems, they can also
perform a number of other roles, serving as goal-setting 4.1 Experimental Manipulations
mechanisms, social status indicators, and group identifiers Since the specific badging functionality has already been
[7]. Design cues can be tweaked as necessary to serve each selected by this point in the research design, identifying the
of these purposes more readily and empirical data can experimental manipulations should not be overly taxing.
suggest particular areas of the user interface in which to Just as with any other experimental study, a control and
concentrate. experimental group will be necessary.
3. Literature Review There are some unique concerns which should be thought
After the research questions have been formulated, it is through, however. For instance, if individual badges are
time to plan out the specifics of the study. This cannot be beings studied, the experimenter should record, without the
effectively executed without a good working knowledge of participants’ knowledge, who would have earned the
the literature. As with any study, a thorough literature badges had the badging system been implemented in the
review is necessary to identify the conclusions of previous control. This will enable a comparison between those who
research and to discover what is not yet understood. New completed the requirements for earning the badge of
badging articles are being published frequently, so it is interest and received it against those who completed the
important to stay current with the latest developments in requirements and did not receive it. Otherwise, it will not
the field. In particular, annotated bibliographies focused on be possible to know if differences in the dependent
gamification (e.g., [8]) and digital badges (e.g., [9]) are variables between the control and experimental groups
valuable, as are studies outlining prior implementations of were due to the presence or absence of the badge itself, or
badges and the lessons learned through those experiences due to some other factor associated with completing the
(e.g., [10]). Articles focused on the required and optional requirements for earning the badge.
components and qualities of badges [6] can provide a
deeper understanding of badges while also providing ideas 4.2 Demographics
for elements that can be manipulated during Individual differences appear to be present in badging [7],
experimentation. [5], [11], but the results are not yet conclusive. It is
important for future studies to continue to collect
The diverse nature of badging, in combination with the demographic data in order to understand how badges are
frequency of new publications, can often make finding perceived of and received by learners of varying genders
relevantly focused research difficult. Fortunately, the and ethnicities.
desired functionality of the system (e.g., reward or
credentialing) has already been decided upon when Additionally, badging does include some novel
formulating research questions. This can serve as a guide demographic concerns that have implications for
for appropriately narrowing the literature review from a demographic survey design. For example, in response to
large but broad set of results to a more narrow and precise the proliferation of badging in video games [12], [13], [14],
body of work. The literature review might begin with video game players may have much more experience with
research related to the broader focus of the research badging than other participants. In games, badges may also
questions, such as badge motivation, reward, or be known as achievements or trophies. Since the effects of
credentialing. One needs to branch out to other relevant prior interaction with badging are not yet known, it can be
fields after this literature has been exhausted. For example, helpful to include a question or instrument to assess the
if badges are being studied as rewards with a specific extent of prior interactions with games to aid the analysis of
emphasis on their potential to improve motivation, one may any unexpected results at the end of experimentation.
consider browsing the psychology-based literature to learn Having this information will allow researchers to control
more about theories of motivation. Or, if credentialing is for prior gaming experience within the sample during the
the focus, research on professional certifications may yield data analysis phase.
relevant knowledge. As digital badging is a fairly new field 4.3 Dependent Variables
of study, it is possible that the specific research on digital
A variety of dependent variables are relevant to badging
badging related to the chosen topic may be sparse or even
studies, but some have received more interest than others.
nonexistent. Considering broader areas or related
When badges are studied as rewards, intrinsic motivation
disciplines of study will enable the discovery of knowledge
[15] is commonly measured as an indicator of the
that is potentially transferrable and may yield insights that
participants’ desire to complete the task simply for the
prove valuable during hypothesis formation and
reward of having completed the task. This is in contrast to
experimental design.
extrinsic motivation which refers to the participants’ desire
4. EXPERIMENTAL DESIGN to complete the task in order to gain some external reward
After assimilating relevant background knowledge, such as money or a trophy. Intrinsic motivation is typically
experimental design can begin. measured through the interest/enjoyment subscale of the
intrinsic motivation inventory [16].
Learner engagement is another dependent variable that structuring verbal and visual information within the context
serves as an indicator of the participants’ willingness to of badges.
take an active role in the experimental task. This construct
When in-house developing prowess is insufficient, an
can be measured using questionnaires like the Classroom
external developer is needed to develop the badging
Survey of Student Engagement [17], the Student Course
system. In this scenario, it is obviously better to hire
Engagement Questionnaire [18], or, for game and
experienced developers if at all possible. With these
simulation-based studies, an engagement questionnaire
professionals, finishing the system on time is more likely
[19]. Engagement can also be measured in terms of
than it is in developing one’s own system. Moreover, the
participants’ level or frequency of activity on the
same flexibility can be included in the developed system
experimental task. This might be indicated by the number
since it is still being developed from scratch to meet the
of answers submitted, the number of times logged into the
specifications of the customer. If, however, development
system, or the number of minutes spent on task.
costs are high and budgets are small, hiring an experienced
Badges are frequently implemented to improve developer may not be an option. Or, financing the
performance, so performance measures can also serve as development of a very limited system may be the only
dependent variables. The specific metrics used will depend possibility.
heavily on the experimental task. In education studies,
When working with a badge system developer,
participants’ final grades are typically appropriate data
communication becomes very important. If the system’s
points to collect. Accuracy, as a ratio of correct to incorrect
specifications are not well understood by the developer,
answers or classifications, is another popular metric that
errors may be made which could delay development or
extends beyond academic environments. The researcher
result in a system which does not meet the researcher’s
should consider which metrics best indicate success or
standards. Furthermore, bugs and specification
failure in the environment being studied.
discrepancies may not be discovered until experimentation
In some instances, badges themselves can be used as has already begun, introducing variations that introduce
dependent variables. For instance, the number of badges minor confounds or even completely invalidate the data.
earned may be a useful metric in correlational studies. This was the case with one of the authors’ prior studies.
Perhaps, as learners earn more badges, their perceived self- They found out after the study had concluded that the
efficacy increases. Or, participants who earn more badges participants enrolled in the non-badging section of a course
may exhibit more goal-directed behavior. If these questions were all emailed the same badge award notifications that
are of interest to a researcher, they can be formulated as the participants in the badging section received. This was
guiding research questions at the beginning of a research problematic because the badges did not actually exist for
study’s design (see Section 2). those users enrolled in the non-badging section. A glitch in
the badging system therefore led to problems with the
5. SELECTING A BADGING SYSTEM separation between badging and non-badging sections of
Selecting a badging system for an experiment is one of the the course, leading to unreliable data.
most important tasks in designing a badging study. The
chosen system will influence major aspects of the protocol It should be noted that either of these methods for
including the risks of the study and the capabilities in developing a customized system comes at a cost. These two
badging experimentation. There are two primary routes: 1) models provide the utmost flexibility for researchers, but
Develop a system or hire a developer to develop the they both require overhead that often makes this method of
system; or 2) Use a commercial system that has already design impractical. It may take too long to develop a
been developed. system that meets the needs of the experimenter. Or, it may
be too expensive. In these instances, the enhanced
5.1 Developing a System flexibility may not be justified by the necessary resources
Developing a badging system provides the greatest range of to enable that flexibility. Further, if strict deadlines exist,
capabilities. When a system is developed from scratch, it attempting to develop a system in time to meet the study’s
can be built to exact specifications with the exact badging goals will introduce a moderate level of risk into the study.
features one needs for a particular group of users. This is
important because there is evidence to suggest that badging 5.2 Using a Commercial System
systems must be well-designed in order to fulfill their
When working with a tight budget or a strict timeline, a
intended function [2, 4].
commercial system may be the only viable option. Here,
It is certainly possible to design and implement one’s own
the researcher will choose one of several commercially
system, but this may require skills outside of one’s
available badging solutions. These systems typically offer
capabilities. Designing a badging system on one’s own
limited customization and may or may not require a fee,
requires a number of diverse skills including programming,
though this fee is typically very small in comparison to the
database design, graphic design, and instructional design. A
cost of developing a new system. Perhaps the largest
background in writing or technical communication is also
benefit of commercial systems is that they are already
useful to determine the most effective methods for
developed and the majority of their bugs have already been really hire a graphic designer”). This provides a better
corrected. These systems also offer good transparency in understanding of whether the results were due to the
terms of features. The limitations and affordances are inclusion/exclusion of badges, or due to some unconsidered
immediately apparent. extraneous variable such as visual design.
6. DURING EXPERIMENTATION 6.3 Temporal complexity
Experimentation in badging studies is very similar to other The experimenter may also wish to examine badge access
human-in-the-loop studies, but requires some special patterns. This creates challenges in regards to time. What
considerations. It is important for the experimenter to be is the appropriate duration in which to consider the earning
aware of the special considerations of badge research and of badges, and what sort of longitudinal impact will the
be prepared to record any unplanned events or unexpected acquisition of these badges have for participants? Further,
observations regarding the badge system. Researchers how will badges be made available to earners on a long
administering badge-specific studies may want to pay term basis? When badges can be viewed on demand,
particular attention to these three dimensions of complexity differences in the number of times the badges were
during the study. accessed in a particular experimental session may be
indicative of engagement or interest. A user who exerts
6.1 Technical and operational complexity additional effort to frequently view badges or a list of
As with any digital system, badging systems have the possible badges that can be earned is probably more
capacity to malfunction. This may happen due to problems interested in the system than a user who never or rarely
within the software or usability issues with the participants. accesses the badges.
While the best course of action is to thoroughly test the
system to avoid such issues before experimentation begins, In sum, researchers are still trying to understand how to
bugs may still end up in the experimental system, build better badges. An understanding of what factors are
potentially compromising the data. Once experimentation important to their effectiveness and which areas of
begins, the system should be regularly monitored and complexity are most relevant to digital badging studies will
evaluated to identify any errors that arise. For instance, a help to further this effort. Factors related to these areas of
participant may mention receiving a badge that they did not complexity may arise unexpectedly during experimentation
earn, or not receiving a badge that they did earn. If the and should be recorded.
experimenter decides that the participant is correct, the
malfunction is obvious and may be quickly remedied (after
7. DATA ANALYSIS
During data analysis, it is important to consider who
a note is made to evaluate the impact of this confound).
actually earned and did not earn each badge. If badges
However, participants may not be familiar with the
could be earned multiple times, it is also important to
criterion for earning a badge, or the criterion may be too
consider the number of times each badge was earned. By
subjective for the participant to make a confident decision.
doing this, the researcher can test for effects of the
As a result, the experimenter should routinely check to
inclusion, earning, or award of badges on an individual
ensure that the badges are being awarded in the way that
level, identifying the impact of badges independently
was designed. This can ensure that errors are caught early
instead of just observing the system which contains them.
and can be accounted for during data analysis.
These analyses could yield results that help badging system
6.2 Behavioral complexity designers understand which criteria are the most useful for
Participant behaviors in these types of studies can be very badges in their system.
informative. Especially if the experiment only lasts for a Also, simply having a badge available in a course is
short duration and the experimenter and participants are co- unlikely to be sufficient as a manipulation if the badge is
located, it is helpful to record behavioral observations. never earned or seen. For example, consider the case of
Noticing behaviors that are indicative of changes in hidden badges, or badges users do not know about until
dependent variables can provide useful information for data they are earned. If these hidden badges are never earned,
interpretation or ideas for future experimentation. For they serve the same role as if they never existed; they are
example, participants shouting in frustration or satisfaction invisible to the user. The fact that the badges technically
may be showing indicators of engagement or immersion. could have been earned are unlikely to have affected the
Or, if the badging system uses notifications that interrupt dependent variables. In other words, the relationships
the task whenever a badge is awarded, and a student groans between badges can be more complex than they seem at
and tries to quickly click out of the notification every time first glance. Badge visibility should be considered
it appears, this could be an indicator that the notification carefully during data analysis. At minimum, be sure to
system is invasive. A future study may wish to see if these make the distinction between:
notifications are detrimental to the badging system’s
effectiveness. Participants may mention how much they • Badges that were available, but never seen
enjoyed or disliked a particular aspect of the badging • Badges that were available, and were seen, but not
experience (e.g., “The badges were hideous. You should earned
• Badges that were available, seen, and earned once [8] Hamari, J., Koivisto, J., and Sarsa, H. 2014. Does
• Badges that were available, seen, and earned multiple gamification work? A literature review of empirical
times studies on gamification. In Proceedings of the 47th
Hawaii International Conference on System Sciences
Breaking down the results in this manner will enable (Hawaii, USA, January 6 – 9, 2014).
formation of more specific conclusions regarding the data.
[9] Grant, S. and Shawgo, K.E. 2013. Digital badges: An
8. CONCLUSION annotated research bibliography. Retrieved from
While general rules and strategies that pertain to http://hastac.org/digital-badges-bibliography.
experimental design are often applicable to badging
experiments, badging presents some novel challenges that [10] Grant, S. 2014. What counts as learning: Open badges
require careful consideration. It is important to understand for new opportunities. Digital Media and Learning
these challenges because more research is needed in this Research Hub. Irvine, CA. Retrieved from
area. Badging research is still in its infancy, despite the http://dmlhub.net/sites/default/files/WhatcountsAsLear
rapid growth it has recently experienced. Although recent ning_Grant.pdf
years have generated exciting insights and ideas about [11] Montola, M., Nummenmaa, T., Lucero, A., Boberg,
digital badging, future research will continue to illustrate M., and Korhonen, H. 2009. Applying game
the precise conditions in which digital badges thrive. achievement systems to enhance user experience in a
Sound research design will help us to design the photo sharing service. In Proceedings of Mindtrek:
experiments that collect the empirical data that help us to Everyday Life in the Ubiquitous Era (Tampere,
outline these conditions. Meticulous experimentation will Finland, September 30 – October 02, 2009).
yield data that will inform practitioners and future
researchers alike, enabling higher quality research and [12] Microsoft. 2015. Xbox One achievements and
more effective badging implementations. challenges. Xbox One Support. Retrieved from
http://support.xbox.com/en-US/xbox-
9. REFERENCES one/games/achievements-and-challenges
[1] Carey, K. 2012. Show me your badge. The New York
[13] Sony. 2015. Get trophies. Get recognition. Playstation
Times. Retrieved from
Community. Retrieved from
http://blogs.maryville.edu/kfletcher/files/2012/11/Sho
http://us.playstation.com/community/mytrophies
w-Me-Your-Badge-NYTimes.pdf
[14] Steam. 2015. Game stats and achievements in steam.
[2] Blair, L. 2012. The use of video game achievements to
Steam Support. Retrieved from
enhance player performance, self-efficacy, and
http://support.steampowered.com/kb_article.php?ref=4
motivation. Doctoral Dissertation, University of
462-UIOB-0425&l=English
Central Florida.
[15] Ryan, R.M. and Deci, E.L. 2000. Self-determination
[3] McDaniel, R. and Fanfarelli, J.R. in press. A digital
theory and the facilitation of intrinsic motivation,
badging dataset focused on performance, engagement,
social development, and well-being. American
and behavior-related variables from observations in
Psychologist. 55(1) (2000). 68-78.
web-based university courses. British Journal of
Educational Technology. [16] SelfDeterminationTheory.org. n.d. Intrinsic motivation
inventory (IMI). Self-Determination Theory: An
[4] Fanfarelli, J. 2014. The effects of narrative and
Approach to Human Motivation and Personality.
achievements on learning in a 2D platformer video
Retrieved from
game. Doctoral Dissertation, University of Central
http://www.selfdeterminationtheory.org/questionnaires
Florida.
/10-questionnaires/50
[5] McDaniel, R., Lindgren, R., and Friskics, J. 2012.
[17] Ouimet, J.A. and Smallwood, R.A. 2005. Assessment
Using badges for shaping interactions in online
measures: CLASSE–The Class Level Survey of Student
learning environments. In Proceedings of IPCC
Engagement. Assessment Update, 17(6), 13-15.
(Orlando, FL, USA, October 8 – 10, 2012).
[18] Handelsman, M.M., Briggs, W.L., Sullivan, N., and
[6] Hamari, J. and Eranti, V. 2011. Framework for
Towler, A. 2005. A measure of college student course
designing and evaluating game achievements. In
engagement. J. Educational Research, 93(3), 184-191.
Proceedings of DiGRA (Hilversum, Netherlands,
September 14 - 17, 2011). [19] Charlton, J. and Danforth, I. 2005. Distinguishing
addiction and high engagement in the context of online
[7] Antin, J. and Churchill, E.F. 2011. Badges in social
game playing. Computers in Human Behavior, 23(3),
media: A social psychological perspective. In
1531-154.
Proceedings of CHI (Vancouver, BC, Canada, 2011).