Research Designs for Studying Individual and Collaborative Learning with Digital Badges Daniel T. Hickey James E. Willis, III Indiana University Indiana University 1900 East Tenth Street, Room 506 1900 East Tenth Street, Room 504 Bloomington, Indiana 47406 Bloomington, Indiana 47406 001-812-856-2344 001-812-856-1483 dthickey@indiana.edu jaedwill@indiana.edu ABSTRACT (DPD) project was carried out to capture the design Web-enabled digital badges are quickly transforming the principles for using digital badges that emerged across way that individual and collaborative learning is supported, thirty projects funded to develop badges in the 2012 Digital recognized, and assessed in digital learning contexts. Media and Learning (DML) competition funded by the Badges contain specific claims and evidence supporting MacArthur and Gates Foundations. This project uncovered those claims and they have the potential to also transform the principles for using badges to recognize learning, assess the way that researchers study learning. Because digital learning, motivate learning, and study learning. This paper badges are so new, there are few examples or models for summarizes the principles uncovered for studying learning. studying them or using them to study learning. This paper Most of the thirty project studied in the recent project used introduces six research designs for studying learning with badges to recognize some form of collaborative learning, digital badges that emerged in a study of thirty projects and all of this learning was computer supported. funded to develop digital badges in a 2012 competition. Research and evaluation are contentious topics in These principles distinguish between summative, education. This is because people disagree on what counts formative, and “transformative” research, and between as “evidence” and what methods count as “scientific.” A using conventional forms of evidence and using the 2002 report by the US National Research Council argued evidence contained in digital badges. that the “gold standard” of scientific educational research is randomized experimental trials [1]. But the NRC also Categories and Subject Descriptors recognized that many of the most important ideas that K.3.1 [Computer Uses in Education]: Collaborative might be tested in experimental research are unlikely to learning. be discovered in experimental studies. This seems certain to be the case with digital badges in education. General Terms Measurement, Design, Human Factors, Theory. 2. RESEARCH AND EVALUATION OF Keywords DIGITAL BADGES Research Methodologies, Lifelong Learning, Learning Thanks to the DML competition and extensive media Analytics/Educational Data Mining, Open Digital Badges. coverage, many schools and programs are considering using digital badges. This means that many are also 1. INTRODUCTION beginning to ask about the research evidence concerning the effectiveness of digital badges. Digital badges are so Digital badges are web-enabled tokens of learning and new that just a handful of studies have made it through the accomplishment. They are different than grades, peer review process. Grant and Shawgo’s annotated transcripts, and certificates because they contain specific bibliography provides a nice summary of recent badges claims and detailed evidence supporting those claims, and research and provides additional relevant resources from this information can be readily accumulated and shared in other contexts [2]. After the initial badges competition, digital networks. The Design Principles Documentation HASTAC announced a separate research competition to study digital badges and made awards to five badges Copyright © by the paper’s authors. Copying permitted for research projects. Some of these will be discussed below. private and academic purposes. In: D. Hickey, J. Jovanovic, S. Lonn, J.E. Willis, III (eds.): Few of the 2012 awardees included any formal research or Proceedings of the Open Badges in Education (OBIE 2015) evaluation studies in their original proposals. Notably, the Workshop, Poughkeepsie, New York, USA, 16-Mar-2015, DML 2012 competition did not require that proposals published at http://ceur-ws.org. include detailed evaluation plans. This seems like a wise decision. This is because requiring detailed evaluation plans may have led projects to prematurely search for information in the other projects that were able to negotiate “summative” evidence that badges “worked” before they these issues. had a chance to maximize the formative potential of digital badges to support learning. However, interviews with 4. SIX RESEARCH DESIGN PRINCIPLES project leaders whose badge systems are now in place STUDYING DIGITAL BADGES revealed that many were starting to think quite seriously Focusing on systematic studies and crossing research about the sorts of studies they might conduct. This paper purposes and types of evidence yields the six research aims to help move these efforts forward by providing a designs shown in Table 1. The following descriptions of framework for organizing these efforts. each research design draw on selected examples from the DML competition as well as the studies being conducted by 3. IMPORTANT DISTINCTIONS FOR the awardees in the 2013 HASTAC Badges Research STUDYING DIGITAL BADGES Competition. Attempting to makes sense of the possible kinds of studies Table 1. Six research designs that might be carried out with digital badges revealed three dimensions for thinking about research: systematicity, Purpose Evidence purpose, and evidence. Arguably, the distinguishing feature of “research” is that it is systematic. Research involves Using Using Evidence in systematically gathering some sort of evidence and Conventional Badges attempting to document things in a way that could inform Evidence others. The design principles that the DPD project is Summative 1. Research OF 4. Research WITH identifying for recognizing, assessing, and motivating Badges Badges and OF learning are mostly not coming out of systematic studies. In Badges other words, the thirty projects are systematically developing badging practices, rather than more general Formative 2. Research FOR 5. Research WITH principles that might apply across multiple badging Badges Badges and FOR projects. In response our project overlaid a more systematic Badges framework that is expected to eventually result in more Transformative 3. Research FOR 6. Research WITH systematic knowledge about research designs for study Ecosystems Badges and FOR studying badges. Ecosystems Building on the existing literature on assessment purposes, a second dimension follows from the distinction between summative studies “of badges” and formative studies “for 4.1 Research OF Badges badges.” The purpose of summative studies are more Summative studies of digital badges are the largest naturalistic examination of the way the world is, while the category of badges research. Some relied more on purpose of formative studies are more interventionist interpretive methods and qualitative evidence. For example, efforts to change things. While most summative studies are HASTAC Badges Research awardee Katie Davis intended to eventually formative, the impact is much less (University of Washington) studied how students and direct and specific. Finally, there is transformative research teachers in the Providence After School Alliance experience that examines how entire learning ecosystems are changed the badges used to give high school credit for expanded or created around badges. learning opportunities. Davis and her team used interviews, A third dimension is between studies that do not use the questionnaires, and observations to explore (a) how badges evidence of learning contained in digital badges and studies fit in the academic and peer culture, (b) the role that badges that do use this evidence. Badges contain the actual play in motivation and achievement, and (c) whether evidence (or links to evidence such as artifacts produced by badges connect in-school and after-school experience. learners) to support particular claims of proficiency or Likewise, studies by HASTAC Badges Research awardee accomplishment. There is usually a lot of negotiation Jan Plass (New York University) and colleagues video involved in deciding what learning should be recognized recorded game play in publicly available games with and with badges and how that learning will be assessed. As without digital badges. They analyzed those recordings for such, the evidence contained in badges will embody the trends and insights into participants’ perceptions and values of the program or organization that issued them. As valuations of badges, and for changes in gameplay patterns the DPD project learned, a number of the projects ended due to badges. Other summative studies of badges might without a functional badging system because projects rely more on correlational methods and focus on individual simply could not manage to negotiate the claims, evidence, differences and variables. In one of the first published peer- and assessments to associate with their badges. This seems reviewed studies of digital badges, Abramovich, Schunn, to bolster the credibility of the information of the and Higashi explored mastery-based and participation- based badges in an intelligent tutoring system for teaching proportional reasoning in mathematics [3]. They measured by Global Kids of a new badging system for their youth self-reported motivation toward mathematics before and programs. A DML award paired them with DML Badge after the game, pre-post gains in proportional reasoning, System awardee Learning Times to and opinion toward badges. Correlational analyses revealed implement BadgeStack in Global Kids’ Race to the White both positive and negative effects of badges on learner House and Virtual Video Project programs. The report of motivation, and that these finding interacted in turn with the pilot study describes how badges impacted the student ability and types of badges. educational programs that Global Kids had already developed and provides some examples of what this might Other studies of the impact of digital badges are using look like. For example, they found that their youth leaders experimental methods, such as creating different versions received 48 confirmations that submitted work met the of the same types of badges issued. For example, the final requirements of their program for badges, as well as 10 study that Plass conducted modified a geometry game to indications that the evidence did not meet their examine the impact of two different types of badges. They requirements. They pointed out that confirming both “took compared mastery badges (based on players’ own progress extra time—for the youth to submit the evidence and the mastering learning goals) and performance badges (based GK staff to review and evaluate—but the goal of providing on players’ performance relative to others). They showed formative assessment was significantly advanced” [5]. The impact of the different badges on a range of individual report explained that this sort of assessment had never been outcomes, including motivation and learning, results that carried out in the educational programs that Global Kids provide generalizable principles about the impact of these offer. two common types of badges. Other systematic studies of the transformational effects of 4.2 Research FOR Badges badges on ecosystems are likely to emerge in the Summer Other studies will formatively intervene more directly in of Learning and various Hive projects. Another example is badge system design. One distinctly formative effort is the the dissertation study being conducted by Rafi Santo. A study by HASTAC Badges Research Awardee Jim grant from the New York Community Trust [6] is Diamond of the Educational Development Center. supporting his extended study of the diffusion of Diamond has already been working intensively with the innovations [7] in the Hive NYC. This and other such DML/Gates 2012 Awardees Who Built America? (WBA) efforts promise to provide more specific research design teacher mastery badge system. Diamond’s study asked principles for studying the creation and transformation of some of the same questions as Davis’ study of PASA. For learning ecosystems via badges and other specific example Diamond asked about the role that WBA badges innovations. Formative studies of entire learning play in teacher professional development, and examined the ecosystems are incredibly complex. There are many ways that badge-related activities influence the variables to consider, numerous principles and features to development of an online teacher professional development be refined, and many methods that might be used. There are community. What pushes this research into the formative also complex issues that arise when attempting to link the category is that Diamond is asking these questions while learning of students/mentees with the learning of directly participating in efforts to build the badging system teachers/mentors. and the online professional development network. 4.4 Research WITH Badges and OF Badges Studying things as they are changing quickly becomes Using the evidence contained in badges offers new complicated. And studying one’s own practice requires opportunities for summative research of badges. This extra attention to ensure generalizability. Diamond includes studies of the credibility of claims made in badges. certainly recognized this in his proposal. This is why he is This question naturally has come up a lot around digital using design-based research (DBR) methods. As articulated badges. Jacobs, in a 2012 article in US News & World by Paul Cobb and colleagues in 2003, DBR builds “local” Report suggested badges might someday overturn the theories in the context of iterative refinements of practice monopoly that colleges currently hold on formal [4]. Generally speaking, DBR studies start with some credentials—but only if badges are proven credible [8]. As relatively general design principles for getting from the badges begin to function as more formal credentials, current state of affairs to the desired state of affairs. The employers and college admissions officers are wondering back and forth process of translating the general principles about the reliability of the assessments behind the badges into specific features yields specific design principles. and validity of the claims made in badges. Some have Importantly, this process also reveals the key aspects of the noted that the credibility of conventional credentials learning context that support the specific design principles. (grades and transcripts) is seldom systematically 4.3 Research FOR Ecosystems scrutinized. Nonetheless, more formal badges are likely to Many projects are using digital badges to create new trigger studies using conventional criteria from educational learning ecosystems or transform existing ones. Some of and psychological testing (e.g., internal reliability, the projects are beginning to study this process construct validity, generalizability, etc.). Casilli argued that systematically. Consider the pilot study carried out being web-enabled means that the validity of the claims made in any badges will ultimately be crowdsourced [9]. touch on multiple standards, and systems need redundancy This means that evidence from formal reliability and so that students and teachers can select from multiple validity studies might be meaningless if relevant personal activities. Because badges can be more specific and or professional networks collectively ignore or dismiss that because they contain actual evidence of learning, they open evidence. She points out that if this turns out to be true up entirely new formative possibilities for mapping. This efforts to understand the credibility of badges will have to same evidence can then be used summatively to examine look beyond the validity literature to consider research the learning trajectories that students take. about the credibility of information on the Internet. One promising example is Fogg’s taxonomy of credibility, 4.6 Research WITH Badges and FOR which includes presumed, surface, reputed, and earned Ecosystems credibility [10]. Eventually researchers are likely to begin using the The evidence contained in digital badges has many other evidence in digital badges to systematically study and potential uses. The aforementioned pilot study of badges at improve entire learning ecosystems. In this way it seems possible that digital badges might ultimately transform the Global Kids provides initial examples of the how programs entire learning analytics movement. But this seems unlikely can use the evidence to study how learning occurs in their to even get started until clear research design principles for programs. Before Global Kids introduced badges, their summative and formative studies using the evidence in primary evidence of learning in program evaluations were badges emerges. summaries of blog entries that students were asked (but not required) to make. With digital badges it was simple to link to a detailed description of the badges that were offered to 5. REFERENCES program participants. Additionally, the details of who earned what badges provide a surprisingly comprehensive [1] Towne, L. and Shavelson, R.J., Eds. 2002. Scientific picture of the learning that was supported by the program. Research in Education. National Academy Press, Examining the order in which badges were earned also Washington, D.C. allowed Global Kids to begin studying the paths that [2] Grant, S. and Shawgo, K.E. 2013. Digital badges: An learners took through their programs. Given the challenges annotated bibliography. http://hastac.org/digital- that many schools and programs face in evaluating and badges-bibliography studying learning, the introduction of digital badges seems poised to unlock enormous potential in this regard. [3] Abramovich, S., Schunn, C.D., and Higashi, R. 2013. Are badges useful in education?: It depends upon the 4.5 Research WITH Badges and FOR Badges type of badge and expertise of learner. Educational The evidence contained in digital badges also has the Technology Research & Development. 61, 2 (April potential for systemic efforts to formatively improve badge 2013), 217-232. DOI=10.1007/s11423-013-9289-2 systems. Consider, for example, the work of Stacy Kruse, [4] Cobb, P., Confrey, J., Lehrer, R., and Schauble, L. Creative Director of DML 2012 awardee Pragmatic 2003. Design experiments in educational research. Solutions. Kruse is collaborating with the Digital On- Educational Researcher. 32, 1 (Jan./Feb. 2003), 9-13. Ramps project in Philadelphia and several educational initiatives at the Corporation for Public Broadcasting. As [5] Global Kids. 2012. Beta testing the Global Kids Kruse put it in response to an interview question about badging system: A Global Kids, Inc. report. badges research, “Before I started working with digital http://olpglobalkids.org/sites/default/files/story/2012/0 badges, I was working on learning analytics.” This kind of 9/GKSummerBadgeBetaReport.pdf experience has left Kruse and colleagues quite enthusiastic [6] http://www.nycommunitytrust.org/AboutTheTrust/Coll about building learning analytics directly into the badging aborativeFunds/HiveDigitalMediaLearningFund/tabid/ systems they are building, and using those results to 620/Default.aspx dynamically refine what badges are available, how they are displayed, etc. [7] Santo, R. 2013. Networked innovation and Hive NYC: Pop-ups as particle accelerators. Interviews with other DML awardees uncovered some http://empathetics.org/2013/01/28/networked- other promising efforts to use the evidence in badges to innovation-and-hive-nyc-popups-as-particle- transform badging systems. GoGoLabs CEO Lisa Dawley accelerators/ and the Planet Stewards project used badges to connect educational content from the National Oceanic and [8] Jacobs, J. 2012. Digital badges threaten college’s Atmospheric Administration to the Next Generation monopoly on credentials: Badges for lifelong learners’ Science Standards. One of their challenges is mapping the knowledge and skills are considered a ‘game-changing game-like curricular “quests” to the standards. Such strategy.’ http://www.usnews.com/education/best- mapping is notoriously difficult and a major obstacle to colleges/articles/2012/01/20/digital-badges-threaten- standards-based reform. Curricular activities naturally colleges-monopoly-on-credentials [9] Casilli, C. 2012. Badge system design: What we talk [10] Fogg, B.J. 2002. Persuasive technology: Using about when we want to talk about validity. Persona. computers to change what we think and do. Ubiquity https://carlacasilli.wordpress.com/2012/05/21/badge- (December 2002), 89-120. DOI= system-design-what-we-talk-about-when-we-talk- 10.1145/764008.763957 about-validity/