Web-enabled digital badges are quickly transforming the way that individual and collaborative learning is supported, recognized, and assessed in digital learning contexts. Badges contain specific claims and evidence supporting those claims and they have the potential to also transform the way that researchers study learning. Because digital badges are so new, there are few examples or models for studying them or using them to study learning. This paper introduces six research designs for studying learning with digital badges that emerged in a study of thirty projects funded to develop digital badges in a 2012 competition. These principles distinguish between summative, formative, and “transformative” research, and between using conventional forms of evidence and using the evidence contained in digital badges.
Digital badges are web-enabled tokens of learning and accomplishment. They are different than grades, transcripts, and certificates because they contain specific claims and detailed evidence supporting those claims, and this information can be readily accumulated and shared in digital networks. The Design Principles Documentation (DPD) project was carried out to capture the design principles for using digital badges that emerged across thirty projects funded to develop badges in the 2012 Digital Media and Learning (DML) competition funded by the MacArthur and Gates Foundations. This project uncovered the principles for using badges to recognize learning, assess learning, motivate learning, and study learning. This paper summarizes the principles uncovered for studying learning. Most of the thirty project studied in the recent project used badges to recognize some form of collaborative learning, and all of this learning was computer supported. Research and evaluation are contentious topics in education. This is because people disagree on what counts as “evidence” and what methods count as “scientific.” A 2002 report by the US National Research Council argued that the “gold standard” of scientific educational research is randomized experimental trials [1]. But the NRC also recognized that many of the most important ideas that might be tested in experimental research are unlikely to be discovered in experimental studies. This seems certain to be the case with digital badges in education.
Thanks to the DML competition and extensive media coverage, many schools and programs are considering using digital badges. This means that many are also beginning to ask about the research evidence concerning the effectiveness of digital badges. Digital badges are so new that just a handful of studies have made it through the peer review process. Grant and Shawgo’s annotated bibliography provides a nice summary of recent badges research and provides additional relevant resources from other contexts [2]. After the initial badges competition, HASTAC announced a separate research competition to study digital badges and made awards to five badges research projects. Some of these will be discussed below. Few of the 2012 awardees included any formal research or evaluation studies in their original proposals. Notably, the DML 2012 competition did not require that proposals include detailed evaluation plans. This seems like a wise decision. This is because requiring detailed evaluation plans may have led projects to prematurely search for “summative” evidence that badges “worked” before they had a chance to maximize the formative potential of digital badges to support learning. However, interviews with project leaders whose badge systems are now in place revealed that many were starting to think quite seriously about the sorts of studies they might conduct. This paper aims to help move these efforts forward by providing a framework for organizing these efforts.
Attempting to makes sense of the possible kinds of studies that might be carried out with digital badges revealed three dimensions for thinking about research: systematicity, purpose, and evidence. Arguably, the distinguishing feature of “research” is that it is systematic. Research involves systematically gathering some sort of evidence and attempting to document things in a way that could inform others. The design principles that the DPD project is identifying for recognizing, assessing, and motivating learning are mostly not coming out of systematic studies. In other words, the thirty projects are systematically developing badging practices, rather than more general principles that might apply across multiple badging projects. In response our project overlaid a more systematic framework that is expected to eventually result in more systematic knowledge about research designs for study studying badges.
Building on the existing literature on assessment purposes, a second dimension follows from the distinction between summative studies “of badges” and formative studies “for badges.” The purpose of summative studies are more naturalistic examination of the way the world is, while the purpose of formative studies are more interventionist efforts to change things. While most summative studies are intended to eventually formative, the impact is much less direct and specific. Finally, there is transformative research that examines how entire learning ecosystems are changed or created around badges.
A third dimension is between studies that do not use the evidence of learning contained in digital badges and studies that do use this evidence. Badges contain the actual evidence (or links to evidence such as artifacts produced by learners) to support particular claims of proficiency or accomplishment. There is usually a lot of negotiation involved in deciding what learning should be recognized with badges and how that learning will be assessed. As such, the evidence contained in badges will embody the values of the program or organization that issued them. As the DPD project learned, a number of the projects ended without a functional badging system because projects simply could not manage to negotiate the claims, evidence, and assessments to associate with their badges. This seems to bolster the credibility of the information of the information in the other projects that were able to negotiate these issues.
Focusing on systematic studies and crossing research purposes and types of evidence yields the six research designs shown in Table 1. The following descriptions of each research design draw on selected examples from the DML competition as well as the studies being conducted by the awardees in the 2013 HASTAC Badges Research Competition.
Summative studies of digital badges are the largest category of badges research. Some relied more on interpretive methods and qualitative evidence. For example, HASTAC Badges Research awardee Katie Davis (University of Washington) studied how students and teachers in the Providence After School Alliance experience the badges used to give high school credit for expanded learning opportunities. Davis and her team used interviews, questionnaires, and observations to explore (a) how badges fit in the academic and peer culture, (b) the role that badges play in motivation and achievement, and (c) whether badges connect in-school and after-school experience. Likewise, studies by HASTAC Badges Research awardee Jan Plass (New York University) and colleagues video recorded game play in publicly available games with and without digital badges. They analyzed those recordings for trends and insights into participants’ perceptions and valuations of badges, and for changes in gameplay patterns due to badges. Other summative studies of badges might rely more on correlational methods and focus on individual differences and variables. In one of the first published peerreviewed studies of digital badges, Abramovich, Schunn, and Higashi explored mastery-based and participationbased badges in an intelligent tutoring system for teaching proportional reasoning in mathematics [3]. They measured self-reported motivation toward mathematics before and after the game, pre-post gains in proportional reasoning, and opinion toward badges. Correlational analyses revealed both positive and negative effects of badges on learner motivation, and that these finding interacted in turn with student ability and types of badges.
Other studies of the impact of digital badges are using experimental methods, such as creating different versions of the same types of badges issued. For example, the final study that Plass conducted modified a geometry game to examine the impact of two different types of badges. They compared mastery badges (based on players’ own progress mastering learning goals) and performance badges (based on players’ performance relative to others). They showed impact of the different badges on a range of individual outcomes, including motivation and learning, results that provide generalizable principles about the impact of these two common types of badges.
Other studies will formatively intervene more directly in
badge system design. One distinctly formative effort is the
study by HASTAC Badges Research Awardee Jim
Diamond of the Educational Development Center.
Diamond has already been working intensively with the
DML/Gates 2012 Awardees Who Built America? (WBA)
teacher mastery badge system. Diamond’s study asked
some of the same questions as Davis’ study of PASA. For
example Diamond asked about the role that WBA badges
play in teacher professional development, and examined the
ways that badge-related activities influence the
development of an online teacher professional development
community. What pushes this research into the formative
category is that Diamond is asking these questions while
directly participating in efforts to build the badging system
and the online professional development network.
Studying things as they are changing quickly becomes
complicated. And studying one’s own practice requires
extra attention to ensure generalizability. Diamond
certainly recognized this in his proposal. This is why he is
using design-based research (DBR) methods. As articulated
by Paul Cobb and colleagues in 2003, DBR builds “local”
theories in the context of iterative refinements of practice
[
Many projects are using digital badges to create new
learning ecosystems or transform existing ones. Some of
the projects are beginning to study this process
systematically. Consider the pilot study carried out
by Global Kids of a new badging system for their youth
programs. A DML award paired them with DML Badge
System awardee Learning Times to
implement BadgeStack in Global Kids’ Race to the White
House and Virtual Video Project programs. The report of
the pilot study describes how badges impacted the
educational programs that Global Kids had already
developed and provides some examples of what this might
look like. For example, they found that their youth leaders
received 48 confirmations that submitted work met the
requirements of their program for badges, as well as 10
indications that the evidence did not meet their
requirements. They pointed out that confirming both “took
extra time—for the youth to submit the evidence and the
GK staff to review and evaluate—but the goal of providing
formative assessment was significantly advanced” [
Other systematic studies of the transformational effects of
badges on ecosystems are likely to emerge in the Summer
of Learning and various Hive projects. Another example is
the dissertation study being conducted by Rafi Santo. A
grant from the New York Community Trust [6] is
supporting his extended study of the diffusion of
innovations [
Using the evidence contained in badges offers new
opportunities for summative research of badges. This
includes studies of the credibility of claims made in badges.
This question naturally has come up a lot around digital
badges. Jacobs, in a 2012 article in US News & World
Report suggested badges might someday overturn the
monopoly that colleges currently hold on formal
credentials—but only if badges are proven credible [
The evidence contained in digital badges has many other potential uses. The aforementioned pilot study of badges at Global Kids provides initial examples of the how programs can use the evidence to study how learning occurs in their programs. Before Global Kids introduced badges, their primary evidence of learning in program evaluations were summaries of blog entries that students were asked (but not required) to make. With digital badges it was simple to link to a detailed description of the badges that were offered to program participants. Additionally, the details of who earned what badges provide a surprisingly comprehensive picture of the learning that was supported by the program. Examining the order in which badges were earned also allowed Global Kids to begin studying the paths that learners took through their programs. Given the challenges that many schools and programs face in evaluating and studying learning, the introduction of digital badges seems poised to unlock enormous potential in this regard.
The evidence contained in digital badges also has the potential for systemic efforts to formatively improve badge systems. Consider, for example, the work of Stacy Kruse, Creative Director of DML 2012 awardee Pragmatic Solutions. Kruse is collaborating with the Digital OnRamps project in Philadelphia and several educational initiatives at the Corporation for Public Broadcasting. As Kruse put it in response to an interview question about badges research, “Before I started working with digital badges, I was working on learning analytics.” This kind of experience has left Kruse and colleagues quite enthusiastic about building learning analytics directly into the badging systems they are building, and using those results to dynamically refine what badges are available, how they are displayed, etc.
Interviews with other DML awardees uncovered some other promising efforts to use the evidence in badges to transform badging systems. GoGoLabs CEO Lisa Dawley and the Planet Stewards project used badges to connect educational content from the National Oceanic and Atmospheric Administration to the Next Generation Science Standards. One of their challenges is mapping the game-like curricular “quests” to the standards. Such mapping is notoriously difficult and a major obstacle to standards-based reform. Curricular activities naturally touch on multiple standards, and systems need redundancy so that students and teachers can select from multiple activities. Because badges can be more specific and because they contain actual evidence of learning, they open up entirely new formative possibilities for mapping. This same evidence can then be used summatively to examine the learning trajectories that students take.
Eventually researchers are likely to begin using the evidence in digital badges to systematically study and improve entire learning ecosystems. In this way it seems possible that digital badges might ultimately transform the entire learning analytics movement. But this seems unlikely to even get started until clear research design principles for summative and formative studies using the evidence in badges emerges.
[1] Towne, L. and Shavelson, R.J., Eds. 2002. Scientific Research in Education. National Academy Press, Washington, D.C. [2] Grant, S. and Shawgo, K.E. 2013. Digital badges: An annotated bibliography. http://hastac.org/digitalbadges-bibliography [3] Abramovich, S., Schunn, C.D., and Higashi, R. 2013.
Are badges useful in education?: It depends upon the type of badge and expertise of learner. Educational Technology Research & Development. 61, 2 (April 2013), 217-232. DOI=10.1007/s11423-013-9289-2 [6] http://www.nycommunitytrust.org/AboutTheTrust/Coll aborativeFunds/HiveDigitalMediaLearningFund/tabid/ 620/Default.aspx