An Online Badging System Supporting Educators’ STEM Learning Chris Gamrat Heather Toomey Zimmerman Penn State University Penn State University Learning, Design, and Technology Learning, Design, and Technology University Park, PA 16802 University Park, PA 16802 gamrat@psu.edu heather@psu.edu ABSTRACT badges can be housed as online representations within In this paper, we investigate how a digital badging system was private badging systems, they can also be shown on other used as part of an informal, not-for-credit professional websites to serve as visual representation of learners’ development project. Teacher Learning Journeys was designed experiences to outside audiences through Open Badges for personalized science learning for educators in K-12 standards, an initiative of the Mozilla Foundation. When schools, museums, universities, and teaching colleges through badges are considered as a communicative tool, researchers employing two levels of micro-credentials: lower achievement have been renaming digital badges as Open Badges [13]. digital stamps and higher achievement digital badges. When shown to external audiences, digital badges can Researchers conducted a qualitative collective case study contain metadata visible by others (i.e., the issuer, centered on 36 teachers; the primary data were records from certifying agency, activity description, expertise the badge learners’ interactions within the digital badge system; signifies, and evaluation criteria). Thus, badges offer secondary data came from a survey at the end of the transparency to the assessment of individuals’ experience and two interviews with 11 focal teachers. Findings achievements and are available for scrutiny [10]. In this suggest the following design principles: (a) two levels of way, digital (or open) badges with their associated assessment can support personalized learning, (b) mastery of metadata are part of people’s professional portfolios that learning can be demonstrated and assessed through reflective logs, (c) collaboration during and after badging activities can document learning, skills, and experiences in an easy-to- provide value to the learners, and (d) establishment of share format. Open Badges and digital badges are terms relevance of badging experiences can support the application used throughout the technology-enhanced learning field. of content outside the badging system. Our system (described below) was designed to be Open- Badge compatible, but this functionality was not available Categories and Subject Descriptors for learners during the study period, so we use the term K.3.1 [Computers and Education]: Computer uses in digital badges in this paper. Education – collaborative learning Digital badges have been used in social media and online games in five key areas identified by Antin and Churchill General Terms [3]: (a) setting goals and providing feedback on goal Design, Human Factors achievement, (b) providing instruction about what activities are possible, (c) building a user’s reputation based on Keywords interests, (d) serving as a status symbol and documenting Digital badges, Open Badges, Science Learning, STEM achievements, and (e) showing affiliation with a Education, Professional Development, Lifelong Learning, community. As digital badges have been used increasingly Learning Across Settings in education, three of these five gaming categorizations 1. DIGITAL BADGES IN EDUCATION from above have been conceptualized as important for use Digital badges have been proposed as tools that learners in learning environments: (a) rewarding and motivating can use to support short-term through lifelong learning achievements, (b) credentialing or recognizing learning within and across various educational institutions [10], (with badges serving as micro-assessments), and (c) acting [11]. We define digital badges as online representations of as markers that learning has occurred for learners learning experiences, which serve as micro-credentials that themselves, the learners’ teachers or mentors, and the document learners’ expertise and skills. While digital learners’ peers [2], [13]. 1.1 Research Findings on Badges within Copyright © by the paper’s authors. Copying permitted for private and academic purposes. Educational Systems In: D. Hickey, J. Jovanovic, S. Lonn, J.E. Willis, III (eds.): While many types of educational institutions are adopting Proceedings of the Open Badges in Education (OBIE 2015) digital badges, empirical research studies (qualitative, Workshop, Poughkeepsie, New York, USA, 16-Mar-2015, quantitative, or design-based investigations) into how published at http://ceur-ws.org digital badges (and now Open Badges) support learning have only recently been published. Because the research is badges to support self-regulation in an online STEM new, themes on the role of digital and Open Badges in course, Cucchiara and colleagues [8] developed interrelated education are emerging. badges, in what they referred to as a “badge ecosystem,” to incorporate technical and interpersonal (i.e., soft) skills. Through an analysis of 30 badging projects funded by the These nontechnical badges were important aspects of the Badges for Badges for Lifelong Learning Competition holistic badge offering for this project. (funded by the MacArthur Foundation) Hickey et al. [11] offer numerous design principles for recognizing, 1.2 Digital Badges as Pedagogical Tools motivating, assessing, and studying learning with digital Within the broader landscape of technology-enhanced badges and within badging systems. Six of these design learning research, our research interests are aligned to principles are related to our inquiry: (1) support learners to badges as pedagogical tools. Our perspective is that badges set goals related to the badging activities, (2) map learning can support learning and act as markers that learning has trajectory through badge display pages and badging happened. The recognition that learning has happened is activity logs, (3) promote collaboration through the important not only for learners, but also for their teachers discussion of personal backgrounds and experience in the and mentors, and their peers in the same learning subject matter, (4) use hierarchy of badges or leveled badge environment [2], [13]. In this way, we consider badges as systems, which suggests learners pick interlocked content pedagogical tools used by learners and their teachers and and decide their level of assessment, (5) use mastery mentors to support informal learning. learning to focus on awarding badges to learners who reflected on their activity and described how they might Hickey et al. [11] have categorized the extant badging implement it in the future—outside the badging system, research, into various types. Given the Hickey et al. and (6) enhance the badges’ validity with expert judgment, typology of badging research, our analysis falls into which asserts that master practitioners or other experts can research with badges and research of badges [11]. This be used to review the work of learners and to act as means that we are considering how to improve the impact mentors. of badges in the learners’ lives (i.e., research of badges), while we are also attending to the metadata within the In research with 51 middle school pupils (seventh and badging system to improve the badging experiences eighth grade) who were learning science, technology, (research with badges) [11]. Our research group has created engineering, and math (STEM), Abramovich, Schunn, and a series of projects to investigate digital badges as a means Higashi [1] investigated the interaction of badges with for increasing learners’ choices and decision making within learners’ prior knowledge and motivation. To accomplish and across informal, formal, and professional development their study, the team designed two types of digital badges: activities. Research with badges and for badges refers to one was awarded for participation (without regard for our intent to use badges to offer improvements for systemic learning outcomes) and the other was awarded for impact such as offering a mechanism to support demonstrated mastery of STEM skills (as an assessment). professional development needs. Abramovich et al. found that learners were motivated Our project’s overarching research goal is to examine differently by the badges. Participation-only badges were learners’ badge-earning pathways in order to refine earned eventually by every student and had little to no learning theory about how the things that learners learn in impact on learning and motivation. In the case of the skill- one setting can be applied to another consequential setting oriented badges, the more skill badges earned, the more via computer tools. For example, our team’s prior work has those learners expected to be successful in mathematics. examined the role of: digital photography in supporting The authors concluded that learners who earned the least learning across settings [20], mobile computers in number of skill-oriented badges also had less expectation supporting meaningful conversations about science where for STEM content learning success than those who earned families’ prior experiences are made relevant to new more skill-oriented badges. experiences in informal institutions, [21] and From their study, Abramovich et al. make two technologically-enhanced web 2.0 tools in supporting recommendations to the design of badging systems. First, connections from community to school [22]. Our project they suggest that to maintain high levels of learner here builds on our team’s prior work of designing for motivation badging system designers include few learning across settings. In addition, within this badging participation-only badges and have many skill-oriented project, we seek to develop design principles applicable to digital badges available. Second, based on an analysis of improving the educational programming related to badges survey data, Abramovich et al. suggest the provision of and the badging system designs of badge issuers. instructions on the criteria needed to earn a badge will also In this analysis, we focus on investigating the following assist to keep motivation high. research question: How do learners interact with elements While much of the research has occurred within STEM of a digital badging system including the goal statements, learning environments, not all badges are focused on logs, materials submitted to earn a stamp or badge, and STEM content. For instance, from their case study of their mentor? We answer this question through a tiered case study analysis of 36 schoolteachers and with the in-depth A teacher started their learning journey by creating a TLJ analysis of 11 of the 36 teachers. Of these 11 focal account and writing a goal statement. While optional, the teachers, we present a thick description of two teachers’ goal statement was used to meet goal-setting needs by experiences—one teacher who represented a typical badge helping the teacher to begin to articulate a learning pathway system user and one teacher who represented an intense [11]. Next, the teacher could review all 54 PD badging user of badging. activities or search through them by topic or grade range and decide which to add initially to their itinerary. Only the 2. TEACHER LEARNING JOURNEYS teacher and the research team could access each itinerary. (TLJ) BADGING SYSTEM In this section, the teacher could review the badging TLJ was created to support teacher professional activities they wanted to accomplish or remove badging development (PD) in STEM subjects for K-12 education in activities they no longer wanted to complete. the United States. Teachers accessed the TLJ system to engage in STEM learning at their own pace and to meet The teacher then could initiate any PD badging activity their own identified needs and a level of depth that from their itinerary by attending a webinar or engaging in reflected the needs of the pupils in their classrooms. other educational activities. In TLJ, no credit was available just for participation; the ideas had to be applied to the 2.1 TLJ’s Goal to Support STEM Learning teacher’s educational setting. The teacher then chose to In the first year of the TLJ system’s operation that is submit their materials to earn a stamp or a badge, as presented here, educational badging options included 54 described in Section 2.3. A National Aeronautics and Space activities related to three areas of science and engineering: Administration (NASA) education expert was assigned as a solar system (19 badging activities), weather and climate mentor to the teacher based on the content area of the (20 badging activities), and engineering (16 badging badge. The mentor provided written feedback to the activities). The teachers earned badges by reflecting on teacher, which resulted in the (a) earning of a badge or a activities that they participated in; these badging activities stamp or (b) additional activities to be completed. The covered a wide-range of PD activities. PD activities used in feedback was stored in the system as part of the metadata our badging program included synchronous webinars, associated with their engagement with a particular activity asynchronous discussion boards, educational media and and could be accessed by the teacher, the mentor, and the texts, archived online presentations by scientists, engineers, researchers. Then, the teacher’s itinerary would change to and education experts. Figure 1 shows one of the 20 show that the badge was earned. badging activities in the Weather & Climate library. After earning a badge or stamp, the teacher would have the option to anonymously fill out a survey on the activity, which gave the research and development team valuable feedback. The teacher would also be able to view the status of their badges and stamps in portfolio pages for the earned achievements. Additionally, any badges or stamps that were being reviewed by mentors were also displayed with the text pending superimposed on the badge image. The Figure 1. The Remote Sensing badging activity within teacher could also review their work in the log section. This the weather and climate topic area. feature of TLJ acted as a blog-style page that captured each entry the teacher made for the activities completed as well 2.2 Features of the Badging System as any additional evidence they uploaded for review by a Teachers were introduced to the system through an mentor or that they wanted to be included in their badging instructional video created by the principal investigator of portfolio. the project. This video elaborated on badges and each of the functions available. Also, teachers were invited to use 2.3 Features of the TLJ Stamps and Badges as many of the available badging PD resources as they An important feature of the TLJ badges was that they were wanted. leveled [11]. Once teachers selected the PD badging activities from TLJ, they controlled their level of The TLJ badging system was developed utilizing a travel engagement with the topic by adjusting the type of micro- metaphor. TLJ helped teachers to set their unique learning credential sought for learning —  a TLJ stamp or TLJ badge. goals for their badging journey, find relevant badging A TLJ stamp reflects the teacher’s need for a more cursory activities to support their goals, compose a log of their engagement with and consequent assessment of an activity. experience for the activity that could be submitted to earn The stamp is the lowest level badge. A TLJ badge is earned badges, receive feedback from an expert as they continued for a higher level of mastery and shows a more in-depth their journey, and store all of their badges engagement with not only the STEM concept but also with (accomplishments) in display pages styled to be similar to applying the STEM concepts to the teachers’ professional that of the stamp inside of a passport. settings. The TLJ badge is the highest-level badge, and for the sake of clarity in the paper, we will only refer to the highest level of badges as badges; the lower level will be cultural tools to mediate STEM meaning-making for referred to as stamps. teachers along the dimensions of collaboration, authenticity, and personalization, as described below. To illustrate the difference between badges and stamps, the PD badging activity, Scale Models: The Earth-Moon 3.2 Dimensions of Mobile Learning System in the Solar System is a good example. This Scale Environments Models activity has three options for teachers: (1) the Our TLJ research and development was guided by a entry-level activity is earning a TLJ stamp (Figure 2 left); framework for mobile learning [8] because TLJ was teachers attend the webinar and write one brief reflective intended for use by learners as they moved across social post, (2) a higher level of mastery means earning a TLJ and technological settings—in their schools, homes, or badge (Figure 2 right), teachers also attend the webinar but communities accessed by desktops, smartphones, tablets, they write a full lesson plan on incorporating the content and other computers. While our prior work focused only on into classroom activity, or (3), teachers can earn both the the personalization dimension of mobile learning [9], this TLJ stamp and badge for this Scale Models activity by analysis focuses equally on authenticity (also referred to as meeting both the stamp and badge criteria. To make the relevance), collaboration, and personalization because stamp and badges visually distinct, the stamp was given a when related to teacher PD, these three learning dimensions design akin to a passport office rubber stamp and the badge address the prior critiques of ineffective PD [5] and show an embroidered patch as shown in Figure 2. what is possible through digital badging systems. 3.2.1 Authenticity In the Kearney et al. model [14], authenticity is a mobile learning dimension comprised of two subcomponents: contextualization and situatedness. Contextualization is high when a learner is exposed to learning situations that are perceived to be realistic or relevant [4] to their personal or professional life. By making use of TLJ, teachers focus on the specific PD badging activities that are relevant to the topics and grade levels that they are teaching. This helps teachers create an authentic context for the badges available through the TLJ system. Situatedness is high when a Figure 2. A TLJ stamp (left) and a TLJ badge (right). badging learning experience is related to or within a 3. THEORETICAL FRAMEWORK community of practice [19]. This can be a full participation, To support our analytical work, our theoretical framework such as engaging in conversations with the fellow STEM considers bringing together the concept of cultural tools teachers, or the engagement can be a peripheral activity, from sociocultural theory [18] with three dimensions of such as observing discussions of the high-engagement mobile learning [14] (i.e., authenticity, collaboration, and webinar participants. This can also relate to the value of personalization). As such, we examine how badges and the badge as status to the community of practitioners within badging system acted as cultural tools to mediate the which the teachers are engaged [2], [3], [13]. teachers’ learning in regard to authenticity to relevant 3.2.2 Collaboration STEM teaching and learning in K-12 schools, collaboration With the emphasis of socio-cultural considerations, with other teachers and mentors, and personalization to cooperative, communicative, and collaborative efforts to their educational setting. support meaning-making are included as a dimension in the 3.1 Cultural Tools to Mediate Learning Kearney et al. [14] framework for mobile learning. Vygotsky’s [18] work in sociocultural theory maintains that Collaboration is included in this framework to permit learning requires people to use cultural tools with the help learners multiple opportunities to articulate their of experienced others. Cultural tools are artifacts such as understandings as they engage with others in shared physical objects, conceptual symbols (including language), meaning-making experiences. Conversation, both verbal and representations that people use within goal-directed and written, provides for a valuable ongoing exchange of actions [13]. For example, within studies of technology- ideas that support the development of STEM teaching enhanced learning, Cole [7] describes three kinds of practices. In addition to shared meaning-making, artifacts that serve as cultural tools: (a) physical objects collaboration is meant to include the exchange of (e.g., a computer, calculator, or hammer), (b) information between learners and between learners and representations and ways of acting, and (c) imagined mentors. Through providing collaboration, cooperation, and artifacts (e.g., games and art). In this way, cultural tools are communication opportunities, badge system designers can the set of resources available to members of a cultural encourage the learners to become more connected with group to mediate meaning-making. In our work, we each other and share experiences. In the case of TLJ, this consider how the badges and the badging system served as sharing goes beyond STEM content to also include information about student learning, needed resources (e.g., in each category of 0–5 years, 6–15 years, and more than favorite web links), and even lesson plans that work within 15 years of work experience. The research participants local contexts. came from the United States but with the majority in the eastern and central US regions. 3.2.3 Personalization Personalization is a dimension of mobile learning that 4.2 Data Collection allows for learner-centered education because learners can Four different types of data were collected during year one: customize many factors related to their learning. Kearney et (a) the 36 participants’ stamp and badge logs and goal al. [8] refer to personalization as a concept that supports the setting statements (n=183 user-generated documents, “just enough, just-in-time, just-for-me” aspect of mobile including stamp and badge logs and goal statements), (b) learning which can allow learners to create their own online year one evaluation survey (n=29, 80.5% of total), pathway of learning. Agency [15], [17], a subcomponent of (c) online activity evaluation surveys for each complete personalization, is high when a learner is able to negotiate badge or stamp activity (n=76), and (d) pre- and post- and make decisions about their own education, such as interviews with 11 preselected teachers. These data types what subject matter to take and what goals to set. The are described in-depth below. dimension of customization furthers personalization by allowing the learner to select what they need to meet their 4.2.1 System Logs and User Generated Documents personal and professional goals. TLJ offers teachers the The system logs captured all user-generated documents. ability to both set their learning goals and to meet these The system logs also captured when a teacher added a goals by navigating through and choosing from available badging activity to their itinerary and the recognition PD activities. Each activity is a smaller part of what might (stamp, badge, or both) that they received for it. normally make up a larger workshop or course. This allows 4.2.2 Summative Survey and Post-badging Surveys teachers to customize what they spend time on to meet both A 13 question electronic survey was sent out as a their personal needs and the needs of their workplace (and summative evaluation at the end of year one to the 36 pupils). While the field of educational technology might participants. The response rate was 80.5%. The survey also use the term personalization to refer to personalization captured opinions on the overall experience with TLJ as offered through data mining patterns generated by an well as possible future improvements. The activity intelligent computer system, here, we use the term evaluation surveys were made available upon completion personalization to refer to providing teachers with the of any badging activities. The activity surveys were agency and the ability to customize. intended to understand the perceptions of value and quality 4. METHODOLOGY of the individual activities and to determine if any activities The purpose of this paper is to understand the learners’ needed significant revision for year two of TLJ. experiences with and through TLJ. We seek to use learner 4.2.3 Pre- and Post-Interviews with Focal Teachers badging artifacts and interviews to understand how the TLJ For the interviews, 11 teachers were strategically selected teachers interacted with elements of a digital badging by the categories shown in Table 1. The pre- and post- system including the goal statements, logs, materials interviews were conducted over the phone. Each interview submitted to earn a stamp or badge, and their mentor. lasted 45 minutes to one hour. The questions were focused Hickey and team [11] have categorized the funded research on perceptions of TLJ and other experiences with features (both completed and in progress) into various types; our of the badging environment and included: “Did you find research falls into the category of ‘research of badges’ value in setting a purpose statement for your learning where researchers “study the badge system’s impact and journey?” “Did you find the itinerary an easy way to keep integration into learners’ lives” (p. 57). Often, research of track of your professional development activities? and “Did badges use includes interviews, surveys, and data mining you find value in the feedback you received from the [11], and as such, our research follows those norms. education specialists that reviewed your badging logs?” 4.1 Research Participants 4.3 Case Study Structure Teachers were recruited from partnering organizations, The research team designed a strategic case study [16] at with the assistance of the National Science Teachers three levels. At the first level of the case, the badging Association and NASA, via listservs and emails. Teachers records of all 36 teachers from year one were considered. at all levels in all kinds of educational institutions were This level of analysis was investigated through the data invited to participate in this professional development collected by the activity surveys, the post-TLJ survey, and experience. Teachers were not compensated for the system logs. Second, a more fine-grained analysis participating. All teachers who expressed interest were examined 11 teachers who completed pre- and post- accepted in the study, but only the 36 teachers who earned interviews for year one of TLJ (shown in Table 1). These at least one stamp or badge in year one were included in interviews with teachers were used to ensure that out-of- this analysis. The majority of research participants were at badging data were used to enrich the analysis. To the elementary or middle school levels—with nearly a third understand learner intention and learner background, these teachers serve to help contextualize the TLJ badging badging system—in their classrooms, in their schools, and system in greater depth. in their lives. Table 1. Matrix sampling for 11 pre- and post-TLJ The analytical process began by iteratively reading the interviews records of 11 teachers who participated in the interviews. We made this choice because we had data from outside the Work Elementary Middle School TLJ system about their interests in badging, the STEM experience Teacher Teachers content and their workplace conditions. The interviews 2 teachers 2 teachers focused on the teachers’ experiences earning digital badges 0–5 Years (Lucy, (Deborah, and stamps, using the online badging system to support Annmarie) Cindy) learning, and how the overall TLJ experience connected to their educational institution. 1 teacher 2 teachers 6–15 Years As we read the 11 teachers’ goal setting statements, badge (Nancy) (Lily, Zoe) activity logs, and interview transcripts, we applied codes for personalization, collaboration, and authenticity. First, 2 teachers 2 teachers the coded excerpts were examined within each case (e.g., 15+ Years (Erin, Hazel) (Barbara, Sally) one teacher’s experiences). Second, the coded segments were compared and contrasted within an Excel spreadsheet to understand patterns within the badging experience of the Third, at the most fine-grained level, we sought to elucidate 11 focal teachers. We used the analysis of the 11 teachers learners’ complex interactions with the badges, stamps, and to strengthen our coding book and to develop preliminary the badging system within TLJ. To conduct this in-depth analytical insights before moving to the third step of the analysis, two teachers were selected — one as an example analysis. In the third analysis step, we examined the logs of average engagement in TLJ and the other as an example from earning badges and stamps and goal statements from of high-level engagement in TLJ. These two teachers were all participating 36 teachers. By looking at the full dataset, strategically sampled as the typical user and the extreme we were then able to find examples and counter examples user. The first participant, Barbara, received a total number of our themes. We then turned to the activity surveys and of stamps and badges similar to the year one average. The summative surveys to compare the trends from the second participant, Sally, completed more than double the teachers’ logs and reflections to survey responses. average teacher. In this way, our examination includes both Throughout the analysis process, we relied most heavily on a typical learner (Barbara) and a heavily involved learner the badging activity logs and the goal setting reflections. (Sally, see Figure 3). Both Barbara and Sally agreed to However, the interview data and survey responses allowed participate in year one as part of their summer professional us to triangulate our findings and to ensure we understood learning and expressed enthusiasm when getting started. the teachers’ experiences from their perspectives. They were selected in order to focus on two people with similar backgrounds, yet they experienced different 5. FINDINGS learning journeys given their interests, expertise, and In this section, we answer our research question about the personal goals as described in Section 4.4. kinds of interactions learners had with the elements of the TLJ badging system. Overall, the learners, who in this study are teachers, reported an estimated average of 13.4 hours (range 2 to 40 hours) of engagement with TLJ activities during the summer. We first describe the way that the full group of 36 teachers used TLJ’s various components, with explanatory details from the 11 interview participants, especially the focal case study educators: Barbara and Sally. Figure 3. Case study participant Sally’s stamps and 5.1 Learning Organized by the Itinerary badges. Each participant’s journey began with writing a personal learning goal statement, which allowed support for teachers in their personalized learning experience. In particular, it 4.4 Data Analysis helped in identifying the badged activities they would want The team conducted an analysis of the TLJ experiences of to complete over the summer and add them to a personal the 36 teachers that participated within the year one itinerary. As Table 2 shows, learners identified over half of implementation, which was run from June to August 2012. the activities that they wanted to complete at the start of the Our analysis focuses on understanding learners’ experience, yet completed very few. Across the 36 experiences within the TLJ badging system and how these teachers, 221 activities were picked during the first few experiences could be useful to interactions outside the weeks of the program, but only 11 were completed. Time at the beginning was spent exploring and planning for the PD they needed in order to meet their personal goals. By options within the badging system, rather than earning examining Barbara and Sally’s goals, selections, and stamps or badges. This supports the motivating learning completed PD, we found that their focus was narrow when principle to set goals described by Hickey and colleagues compared to other teachers that completed at least one [11]. During year one, teachers were able to consider their stamp. From these findings, we can tell that support may be goals through drafting a focused statement and then needed to help teachers in identifying focused and creating a learning trajectory [11] for themselves by adding manageable learning goals. activities to their itinerary. Table 2 also demonstrates fluctuations in the engagement in PD over the time of the 5.2 Choosing a Stamp or a Badge first year of TLJ. Personalization was also clear when analyzing the topics that learners selected. Our earlier work found that teachers Table 2. Stamps and badges completed versus the most often sought the entry-level stamp as micro- potential activities the teachers selected credential, rather than a badge [9]: the 36 participating teachers earned 133 stamps (86.4%) and 21 badges (13.6%) June July August Total across the three STEM content areas. We posit that by Selected 221 176 29 426 providing teachers the ability to make decisions about their All 36 assessment (high or low levels of mastery) provided them Teachers Completed 11 109 34 154 the flexibility to personalize their PD to align with their existing and desired expertise teachers needed for Selected 4 3 0 7 successful teaching. The total in Table 2 is compared to Sally Barbara and Sally’s experiences with TLJ and demonstrates Completed 1 8 0 9 a wide discrepancy in selected activities to do and those completed. While Barbara and Sally completed nearly as many activities as they selected for their itinerary, this is Selected 5 2 0 7 Barbara not the case with the overall 36 teachers. The full group Completed 2 3 0 5 demonstrated that the teachers selected many more than they were able to accomplish in the summer of year one. This may provide additional reasoning for creating leveled As evidenced by Sally and Barbara, setting more specific [11] badges. By offering lower-stakes or smaller and attainable goals may support teacher focus while granularity badges, learners may be able to select and exploring, selecting, and completing PD activities. Barbara complete more of the badges that they’ve identified as focused her goals on engaging her specific class and grade, relevant to their learning. stating that she wanted “to learn new and innovative approaches to teaching the solar system to my sixth grade Personalization was not just in the topic area but in the students which will allow them to become more interested approach to PD. Barbara and Sally each worked and excited.” The PD that Barbara completed for stamp- exclusively on one topic. Barbara worked on solar system level recognition was mostly centered on using technology activities and began by exploring PD related to a general in the classroom. In her initial interview, Barbara’s understanding of planets, next focused specifically on emphasis on finding innovative teaching tools was Mercury for three activities, and then returned to more synonymous with using technology in the classroom. In the general solar system PD again. Sally chose only weather end, Barbara completed five Solar System stamps. and climate activities. Her selection was similar to Barbara’s where she would vary her content focus. Sally Sally wanted to focus on increasing her personal chose PD activities with a fairly consistent alternating knowledge to “assist my students in understanding of how pattern between weather and climate content. Sally reported the cycling of matter in and out of the atmosphere relates.” she was initially not enthusiastic about teaching weather Many other educators, like Sally, used their goal statements and climate science in class, but she felt strongly about to demonstrate an interest in helping their pupils to finding activities within the TLJ that would energize her improve. However, Sally’s goal was more specific with and support her to feel more prepared to work with her regard to content and improving her knowledge. It is clear middle school classes. Barbara also expressed interest in to see that Sally’s learning goal was exactly what she finding solar system PD activities in order to help her pursued in TLJ content. All of the PD that she received prepare for her lessons in the upcoming school year. recognition for was in the category of weather and climate, which aligned with her goals. At the end of year one, Sally 5.3 Activity Logs completed six Weather & Climate stamps and three To earn a badge or a stamp, teachers had to write and Weather & Climate badges. submit an activity log. The teachers’ activity logs consisted of their explanations of how, when, and where the TLJ During their post-interview, both Sally and Barbara badging STEM materials will be used to enhance pupils’ commented that they saw value in setting purpose learning in their university, school, or museum. For these statements because it helped them think critically about the teachers, the TLJ content not only filled a gap in STEM provide opportunities for her pupils to discuss in class other resources as discussed above, but TLJ activities were used topics, such as climate change and land formations. to transform current teaching practices. Teachers discussed how the TLJ material could be used to meet specific needs 5.4 Mentor Feedback on Activity Logs of their pupils. Nancy earned six Solar System stamps and In order for a teacher to receive recognition for completing one Weather & Climate stamp. In her activity logs, she a PD activity, a reflective log was submitted for review. wrote about the unique learning needs of individual pupils These required logs for completed PD also provided an and used TLJ resources to help those diverse pupils enjoy opportunity to make professional learning an authentic interactive activities. For example, Nancy commented, learning experience. Sally made use of her reflections as “[TLJ] helped me absorb the information….If it was useful write up for other colleagues in order to share the resources for me, then it would be useful for learners in classroom she found. Barbara reiterated the value of the reflections as environments.” Nancy, like many other teachers in our a reminder for what worked and what did not for her class. dataset, was not only looking for quality content in the Both Sally and Barbara made use of the logs to make direct badging activities for her own learning, but in her activity applications to their own teaching. log, she carefully considered how the delivery and The NASA education experts within TLJ that acted as activities of the STEM content would work in the diversity mentors frequently provided feedback to learners related to of her class. the submitted logs for the badging activities. This feedback Erin was one of the most robust TLJ users; she completed often included suggestions related to the topic, common eight Engineering stamps; four Engineering badges, two STEM misconceptions they or their pupils may face, and Weather & Climate stamps, one Weather & Climate badge, pedagogical approaches appropriate to the STEM content and one Solar System stamp. In her activity logs, Erin presented. Barbara described her experience with the described how she could supplement the TLJ materials to feedback as offering “really great suggestions.” This bridge the gap for pupils who had little exposure to the dedicated support from these expert educators supports content or to challenge more advanced pupils. To earn her TLJ’s effort to enhance the credibility of the badging seven stamps, Nancy wrote in her activity logs how she activities with experts employed [11] with respect to the would adapt the TLJ content for the younger elementary evaluation of logs and the awarding of badges and stamps. pupils in her classroom. Specifically, Nancy developed 5.5 Collaboration during PD Activities and lesson plans that incorporated the detailed NASA images and simulations in her teaching. Nancy also reflected on after PD Activities were Completed how she would use the materials from the badging system Discussions were supported through the use of TLJ to engage her pupils more fully in the STEM content. She webinars, which allowed the teachers to communicate with wrote, “students who are not highly motivated in science or other teachers and with teacher educators. This mathematics and students who are not challenged enough communication was short-duration (1–1.5 hours) and done through the use of engineering processes would enjoy the over an internal chat program that was available to all interactive activities if they are implemented and facilitated participants. However, it still provided benefits for teachers effectively.” that did not know each other prior to the webinar. For example, in her post-interview Barbara describes her Through the TLJ activity logs, the teachers often wrote interest in having the opportunity to discuss what might about the topic of inclusion and of a student-centered work in their classrooms as evidenced by forming teacher approach. Zoe described herself as a learner who enjoys cohorts in order to provide an on-going channel for the hands-on and visually rich instruction in her logs. Zoe exchange of dialogue and artifacts such as lesson plans, completed two Solar System stamps. In her log, she wrote student collected data, and feedback. Cohorts within TLJ that she liked “how all the activities presented are hands-on could provide teachers with long-term opportunities for and visual to reach a variety of learners. If the students are discussion and collaboration. Long-term benefits for able to see it and manipulate it they are more likely to learn teachers could include a chance for sharing successes, it.” Barbara wrote that an activity was just “okay” for her, identifying ways to improve, and developing shared lesson but then after trying to “put myself in my 6th grade plans. Short discussions provided benefits for teachers as students’ shoes and see how they would respond,” she well, such as learning how each teacher approached a concluded that with the help of some supplemental the lesson differently, but could also lead into a longer term activity would be “a great listening activity.” engagement. For example, in one of her logs Erin writes Lily used her logs to reflect on how she would use projects about attending an engineering webinar, describing how and classroom discussions in her upcoming school year. each teacher took a turn in discussing the activity, and then Lily wrote logs to earn one Solar System stamp, two how she later connected with some of these teachers Engineering stamps, three Weather & Climate stamps, and through another online resource for STEM educators. one Weather & Climate badge. She intended to use the While the full TLJ analysis of the 36 teachers showed information that she learned within the badging system to evidence that all the teachers valued the collaboration. When surveyed about future social and collaborative functionality for TLJ, 67% of the survey participants —the 5.6 Relevance of Badging Activities to Life second most popular response—agreed with the statement: Outside the Badging System “I would like to be part of a cohort of teachers similar to The PD that TLJ provided allowed for personalization that me that I could work within online workshops and other resulted in reports of relevance and authentic practice. activities.” More specifically, the cases of Barbara and Teachers were given the opportunity to submit feedback for Sally illustrate the nuance in the existing opportunities to each completed activity and among the 76 responses, 96% collaborate via synchronous or asynchronous conversations either agreed or strongly agreed to the statement “this with other teachers. For example, Sally stated that while activity is relevant to my teaching.” Across all the teachers’ she did participate in discussions, she was equally happy responses to the post-survey, 93% agreed that the TLJ being an observer of the other teachers’ exchanges. When experience influenced their teaching for the upcoming asked about future collaborative opportunities, both school year. In addition to the PD content, the activities teachers wanted more opportunities to engage in were also structured to require a reflection where teachers discussions like those in the webinars. Barbara further in some way consider how they might apply what they have elaborated that she would like to continue the PD activity learned back to their classroom. discussion later with others that completed the activity in order to share what worked and did not work when trying Within the case study analysis, Sally explained she was to implement the lesson. While the analysis of data only able to find relevance with TLJ by being able to “hunt and demonstrates relatively short duration collaboration, future choose” among all the PD activities to find what best meets iterations of TLJ would benefit from badges designed to her needs as a middle school teacher. While Barbara shared support opportunities for peer modeling similar to that the resources she found with her partner teacher to make made available in the work of Ching and Hursh [6]. use of them in their lesson plans. Since starting to share these resources, Barbara moved closer to making her TLJ The collaborative aspects of TLJ reached into the teachers’ experiences situated within her district’s teaching work with their pupils. The teachers reported incorporating, community. In addition to engaging smaller professional or the intention to incorporate, materials from the TLJ communities, such as the school and district level, the badging activities in their classroom to support the teachers in our study contributed to an educator online development of collaborative skills in their pupils or with learning community that was supported by TLJ and can their peers. These collaborative aspects were reported to be help to promote exchange between teachers from all across included within pupils’ group projects, classroom the country. Through itineraries and logs, teachers are able discussions, and on the teachers’ websites for use with to map their own learning trajectories [11] of past and pupils and parents. Erin, as a leader in her own STEM future learning which may help learners to visualize their school, needed enhanced PD that covered content beyond badge experiences and learning over time. the standard science curriculum, given the special emphasis on STEM in her workplace. Erin wrote in her logs how the For the teachers in our study, the earning of the actual TLJ materials were already successfully incorporated into stamps and badges was secondary to the learning of the training that she facilitated for other teachers during the STEM content area. For example, Barbara and Sally summer that emphasized collaborative skills. In addition to described the stamps and badges as “nice” but “not Erin, other teachers wrote in the activity logs that they necessary” respectively. These two teachers were interested would share the resources from the TLJ activities with their in the content for their own personal goals, as well as colleagues. professional goals. Barbara wrote: “the big value is in the activities themselves.” This finding demonstrates that the While designed to support both asynchronous, independent value of these awards of completion is currently not evident learners, as well as groups of learners, these findings to the general educational community. Both collegial and further support Hickey and colleagues’ recommendation to competitive benefits can be gained from improving a promote collaboration [11]. Collaboration was an important teacher’s ability to showcase their accomplishments and in theme among the teachers’ logs. Teachers wrote about the future, researchers and educators working on badging collaborating with local universities (Lily and Erin), a need to make the value of badging clearer to school principal (Erin), and other teachers (Anne Marie). Teachers districts, employers, policymakers and other opinion appreciated the feedback provided in TLJ by their mentor, leaders. However, the overall opinion of badges and stamps and with this support, teachers felt confident enough to take was favorable as evidenced in the post-survey (Figure 4). on larger projects like science fairs (Lily), science When asked to select as many of the following forms or academies, and leadership roles relating to their own recognition you would like to receive for recognition of schools PD (Erin). As teachers engaged in a community of your work in TLJ, teachers ranked badges and stamps as learners, it was most often through professional learning highly as continuing education credits (CEUs) and below opportunities arranged with peers in their district or school printable portfolios and letters of recognition. In this building, as exemplified by Erin’s experience. regard, future redesigns of TLJ may include a portfolio with stamps, badges, and details describing the learning goals of the PD. Portfolios like this can then be shared with colleagues, administrators, parents of the children in their learning. Many wrote or spoke in their interviews that they classroom, or even friends and family who are interested in found the direct application of TLJ materials to their STEM topics. classrooms and customization of the activities through TLJ logs meaningful parts of the badging experience. The importance of making connections from the badging materials to their own learning and that of their pupils was especially compelling in the excerpts from Sally’s and Barbara’s interviews and logs. Collaboration was a key aspect of the TLJ badging system because the teachers shared experiences and education resources with other TLJ teachers, teachers in their educational institutions, and even the pupils and parents that they served. The logs and interview data showed that TLJ provided opportunities for teacher dialog that was relevant to their work environments. Importantly, our data showed that the Figure 4. The types of recognition preferred by the 36 badging activities allowed the TLJ teachers to share lesson TLJ research participants. plans in relation to how the engineering, solar system, and weather and climate content could be made relevant to the 6. DISCUSSION lives of youth. This analysis focused on the experiences of 36 teachers in TLJ in regard to how TLJ badges and badging system 6.2 Methodological Implication for Studying served as a cultural tool to support STEM learning in Badging Systems with Metadata relation to dimensions of personalization, collaboration, As part of our research with badges [10], we developed and authenticity (i.e., relevance). Our project was oriented techniques for studying the impact of the badging through to understanding learners’ badge-earning trajectories as a the data and metadata contained in the badging system lens into (a) the refinement of mobile learning theory that itself. Given that digital badging is an emerging technology aims to support learners’ reflective activities and self- without many empirical guides to studying badging expressed goals and (b) the development of technology- impacts, an important outcome of our work for the field of enhanced learning practice through the distillation of technology-enhanced learning is to consider which forms design principles applicable to badge issuers in improving of data were useful to our analysis. Most importantly, we their educational programming and instructional design found that no one data source served our research needs. decisions. To reflect our goal, this final section is divided The ability to triangulate the badging experience of learners into three parts. The first subsection reflects on how the in TLJ with the multiple data sources (i.e., pre- and post- activities supported or did not support learners’ reflection interviews, activity surveys, summative survey, goal-setting and personal or professional goals obtainment. The second statements, and badging logs) helped to support how subsection considers methodological implications for teachers used badges for STEM learning and PD. We also studying learning with and of badges. The final section found that collecting data at the start, during, and at the end returns to our orienting design principles [2], [3], [11], [13] of year one helped elucidate the learning journeys as to advance the field of technology-enhanced learning teachers’ experiences with badging grew over time. understandings of badges as cultural tools to mediate Collecting activity logs and activity surveys immediately STEM learning. after each activity was completed was especially important 6.1 TLJ Activities in Support of Learners’ to attending to any redesign work that was needed for year two and beyond. Personal and Professional Goals From our analysis of interviews, summative interviews, and We also found the following data was missing that would badging activity logs, TLJ teachers found themselves in have helped our work. During year one of TLJ, we did not decision-maker roles to personalize their STEM learning. collect the feedback from the mentors. This could have TLJ teachers first selected their learning goals and then helped the research team to better understand how and if they identified personally applicable badging activities that teachers were taking this feedback into account in badges they believed best met their own goals. By providing completed afterward. Moreover, this study would also have options to the teachers, the TLJ educators explored STEM benefited from the collection of data related to content at depths and breadths that matched their collaborative engagement between the learners and the individualized expertise, teaching experience, and experts and among learners while engaging in more classroom needs. For our interviews and activity logs, we synchronous webinar badge activities, including capturing know that this is important to not only their professional chat logs and other forms of communication between learning but to their personal satisfaction with the TLJ teachers. This would have helped the research team to badging program. All the teachers wrote that they found explore how learners supported the learning of their peers benefits in making choices to personalize their STEM through a more in-depth focus on collaboration and the role in the PD activities. Given the success and the popularity of of conversations to support learning. these prompts, this structure could be repeated by TLJ in other aspects of the design. For instance, specific questions 6.3 Implications for Future Design Work could be asked as part of the reflection logs that were While we had a relatively small dataset of 36 teachers, our required to be submitted to earn a stamp. More clearly work is suggestive of several design principles for digital incorporating specific prompts and supporting framework badges that can be refined through future data collection into the structure of each badge or stamp activity, we posit, and analysis. First, the teachers in our full sample (and 11 could help to support teachers to reflect on certain aspects focal teachers) personalized their badging experience in of their learning or future use of what they have learned two key ways: selecting specific badging activities and then thus making their experience more authentic. Being able to choosing an appropriate level of assessment (stamps, focus on a few parts of their learning may help to provide badges, or both) for their own expertise, goals, and an even more favorable experience with required classroom needs. This aspect of the design was highly reflections by all teachers. valued by our research participants across multiple data sources. In order to support additional personalization in 7. CONCLUSION future iterations, we will continue to use goal-setting In this paper, we explored the utility of combining purpose statements in the initial goal-setting section of our perspectives on technology-enhanced learning, cultural work as well as at least two levels of assessment. Our tools, and mobile learning frameworks to design and study findings from some teachers suggest that goal setting could digital badges with K-12 teachers. From our analysis of the be improved by providing examples to help guide the initial experiences of the teachers that participated in the project’s creation of goals that are attainable, specific, and focused first year, we suggest that future badging systems collect and possibly by encouraging teachers to rework their goal multiple forms of feedback (or data) from participants to statement after a certain number of badges were earned. As support meaningful understanding of the badging system. shown in Table 2, the teachers selected many more We also found that including choices in the types of activities than they could have completed in the time that content, the types of assessment, and the pathway sought they allotted for themselves over the summer. Also, the was important to our participants in terms of teachers selected many more activities than they could have acknowledging their expertise, interests, and classroom completed in the time that they allotted for themselves over needs. These finding support the following design the summer. To address this finding, rather than asking principles: (a) two levels of assessment (i.e., stamps and learners to rework their goal-setting statements after a badges) can support personalized learning, (b) mastery of certain number of badges are obtained, the system could be learning can be demonstrated and assessed through designed to automatically send out reminders asking reflective logs completed to earn a badge or stamp, (c) learners to revisit their set goals at specific periods. collaboration during and after badging activities (with colleagues within the badging system and outside of it) can Given that many teachers, including Sally and Barbara, provide value to the learners, and (d) establishment of expressed interest in collaborating with others as they relevance of badging experiences to professional practice earned badges, more long-term collaborative opportunities can support the application of content outside the badging can be designed into the content in our next design system after the recognitions are earned. iteration. For instance, creating cohorts or other groupings could provide people with the ability to discuss their 8. ACKNOWLEDGMENTS badging learning experiences. This cohort model would The development of the TLJ system and this research allow learners to exchange information such as data project were partially supported by the NASA Aerospace collected by their classroom and pedagogical approaches Education Services Project and Penn State University’s that have worked in a particular lesson. These cohorts could Center for Online Innovation in Learning. The authors be set at the start of the experience or after so many badges express appreciation to the participating teachers for are earned to create groupings of people with similar participating in our research project. We acknowledge team STEM content experiences. These collaborations might members Kyle Peck and Jaclyn Dudek for their also be supported by using the TLJ system to allow contribution to the development of the TLJ system and for teachers to create their own groupings or to match their scholarly conversations about badges as pedagogical geographically located teachers together. tools. Finally, the logs were identified by most teachers as helpful 9. REFERENCES to connect the STEM learning within the badging system to [1] Abramovich, S., Schunn, C. and Higashi, R. M. 2013. their educational institution. This suggests there may be Are badges useful in education?: It depends upon the benefits from incorporating even more supportive type of badge and expertise of learner. Educational structures into the badging system related to the activity Technology Research and Development, (Mar. 2013) logging required to earn a stamp or badge. For example, 61, 2, 217–232. within the collaborative PD webinars, reflective questions were posed to all the teachers to support their engagement [2] Ahn, J., Pellicone, A., and Butler, B. S. 2014. Open [12] Jakobsson, A., & Davidsson, E. 2012. Using badges for education: what are the implications at the sociocultural frameworks to understand the intersection of open systems and badging?. (2014) significance of interactions at science and technology Research in Learning Technology, 22, 23563 - centers and museums. In (E. Davidsson & A. http://dx.doi.org/10.3402/rlt.v22.23563 Jakobsson, Eds.) Understanding Interactions at Science Centers and Museums (pp. 3-21). Rotterdam, [3] Antin, J., and Churchill, E. F. 2011. Badges in social NL: SensePublishers. media: A social psychological perspective. CHI 2011 Gamification Workshop Proceedings. [13] Jovanovic, J. and Devedzic, V. 2014. Open badges: http://uxscientist.com/?sort=post_date&page=7 Challenges and opportunities. Advances in Web-Based Learning–ICWL 2014. Springer International [4] Barron, L. et al. 1991. Technology and the design of Publishing, 56-65. generative learning environments. Educational Technology. (1991) 31, 5, 34–40. [14] Kearney, M., Schuck, S., Burden, K., and Aubusson, P. 2012. Viewing mobile learning from a pedagogical [5] Borko, H. 2004. Professional development and teacher perspective. Research in Learning Technology. (Feb. learning: Mapping the terrain. Educational Researcher. 2012), 20, 1063519, 1–17. 33, 8, 3–15. [15] Pachler, N., Bachmair, B., and Cook, J. 2010. Mobile [6] Ching, C. C. and Hursh, A. W. 2014. Peer modeling devices as resources for learning: Adoption trends, and innovation adoption among teachers in online characteristics, constraints and challenges. Mobile professional development. Computers and Education. Learning. N. Pachler et al., eds. Springer US. 73–93. 73, 72–82. [16] Stake, R.E. 1995. The Art of Case Study Research. [7] Cole, M. 1998. Cultural Psychology: A Once and Sage Publications. Future Discipline. Harvard University Press. [17] Traxler, J. 2009. Learning in a mobile age. [8] Cucchiara, S., Giglio, A., Persico, D., and Raffaghelli, International Journal. 1, (March 2009), 1–12. J. E. 2014. Supporting self-regulated learning through digital badges: A case study. New Horizons in Web [18] Vygotsky, L.S. 1978. Mind in Society: The Based Learning (pp. 133-142). Springer International Development of Higher Psychological Processes. Publishing. Harvard University Press. [9] Gamrat, C., Zimmerman, H. T., Dudek, J., and Peck, [19] Wenger, E. 1998. Communities of Practice: Learning, K. 2014. Personalized workplace learning: An Meaning and Identity. Cambridge University Press. exploratory study on digital badging within a teacher [20] Zimmerman, H. T., Gamrat, C., and Hooper, S. 2014. professional development program. British Journal of Digital postcards from summer camp. TechTrends. 58, Educational Technology. (Nov. 2014), 45, 6, 1136– 2 (March/Apr. 2014), 87–92. 1148. [21] Zimmerman, H.T. Land, S. M., McClain, L. R., [10] Halavais, A. M. C. 2012. A geneaology of badges. Mohney, M. R., Choi, G. W., and Salman, F. H. 2013. Information, Communication & Society. (Apr. 2012), Tree Investigators  : Supporting families’ scientific talk 15, 3, 354–373. in an arboretum with mobile computers. International [11] Hickey, D. T., Otto, N., Itow, R., Schenke, K., Tran, C. Journal of Science Education, Part B. 5, 1 (Oct. 2013), and Chow, C. 2014. Research design principles for 44–67. studying learning with digital badges. (2014). Badges [22] Zimmerman, H. T. and Weible, J. L. In press. Learning Design Principles Documentation Project. Indiana in and about rural places: Connections and tensions University Center for Research on Learning and between students’ everyday experiences and Technology. environmental quality issues in their community. Cultural Studies of Science Education.