=Paper=
{{Paper
|id=Vol-1850/TEFA2016_paper_4-2
|storemode=property
|title=Distributed Assessment with Open Badges of 21st Century Skills
|pdfUrl=https://ceur-ws.org/Vol-1850/TEFA2016_paper_4-2.pdf
|volume=Vol-1850
|authors=Ilona Buchem
}}
==Distributed Assessment with Open Badges of 21st Century Skills==
https://ceur-ws.org/Vol-1850/TEFA2016_paper_4-2.pdf
Distributed Assessment with Open Badges
of 21st Century Skills
Ilona Buchem1
1
Beuth Hochschule für Technik Berlin, Luxemburgerstr. 10,
13351 Berlin, Germany
buchem@beuth-hochschule.de
Abstract. This paper introduces Open Badges as instruments supporting
multiple forms of assessment and recognition of competencies and literacies
including what has been discussed as “21st century skills”, such as information
and communication literacy, leadership and innovation skills. Key features of
Open Badges, i.e. interoperability, metadata and user hub, are outlined to show
how Open Badges support alternative forms of assessment including
distributed, evidence-based and peer-driven assessment. A selected case study
of the BeuthBonus qualification program for migrant academics at Beuth
University of Applied Sciences Berlin illustrates how Open Badges may
enhance distributed assessment based on learner-generated evidence and
affirmation of learner’s claims by significant others. Finally, the transformative
potential of distributed assessment with Open Badges is discussed in view of
both psychological and social effects of distributes assessment.
Keywords: Open badges, micro-credentials, 21st century skills, assessment
1 Introduction
The Internet and digital media have created unprecedented opportunities to connect,
communicate and learn beyond traditional learning environments. In the digital age
learners wanting to use these new opportunities to reach personal, academic and
career goals, need new ways of assessment and recognition of skills acquired in more
and more diverse learning environments. Open Badges are a concept and a
technology enabling new forms of federated, digital credentialing and providing new
ways of recognising skills, achievements and associated evidence from multiple
sources in a portable, interoperable and verifiable way [1]. Open Badges as support
diverse forms of assessment through enhanced representation, verification and
communication of 21st century skills acquired in a classroom, on the job, in a MOOC,
in an online community, or in any other digital and non-digital learning environment.
This paper discusses the potential of distributed assessment with Open Badges based
on a case study from the BeuthBonus qualification program for migrant academics in
view of both psychological and social effects, especially related to the emerging
identity and employability.
In view of the strong need for a clear picture about how citizens are equipping
themselves with the skills demanded in the 21st century [2], Open Badges offer new
opportunities for (a) goal-setting (e. g. by visualising which 21st century skills matter,
what criteria need to be fulfilled to gain a certain level of proficiency of a particular
21st century skill), (b) identity-formation (e. g. by warranting claims about own skills
as resources and affirming or disaffirming these claims by others), (c) extending the
reach of assessment (e. g. by applying criteria from the world of work and providing
evidence in support of claims of skills), and (d) enhancing data transparency (e. g.
by making relevant data, such data about organisations issuing badges, skills
represented by badges, earners or holders of badges and thus bearers of certain skills,
evidence provided to demonstrate skills, which may be made available for learners,
educators, policy-makers, employers).
Transforming lives, promoting social inclusion and career development requires
good information about the skills that are needed and available [2]. Representations of
skills displayed on the web with help of Open Badges may be used to assess
personal/individual skills but also to assess organisational/system performance, e.g.
education systems, workplace practices and policies in developing 21st century skills.
2 Open Badges
This section describes three key features of Open Badges as instruments for
assessment. These include (1) interoperability, enabling issuing, collecting and
displaying micro-credentials on websites compatible with the Open Badge standard,
(2) metadata, allowing issuers of Open Badges to annotate each single badge with
rich information about the issuer, earner and the criterial and skills represented by a
badge, and (3) user hub allowing to accumulate badges earned from different issuers
or systems. These features enhance new forms of skill assessment.
2.1 Interoperability
The Open Badges Infrastructure (OBI) is an Open Source solution and is published
under the Mozilla Public Open Licence. The three-tier OBI architecture is composed
of (a) issuing systems (e.g. LMS Moodle), (b) a user hub (e.g. Badge Backpack), and
(c) displayer systems (e.g. CMS Wordpress) (cf. Fig. 1). The Open Badges
Infrastructure (OBI) is an underlying technology of Open Badges that supports
issuing, collecting and displaying portable digital badges across the web. In oder to be
interoperable, badges must contain code that is compatible or aligned with the OBI
technical specifications. Only interoperable badges, aligned to the technical
specifications of the OBI, may be displayed outside a proprietary or closed badge
platforms [3].
Fig. 1. Open Badges System Infrastructure.
2.2 Metadata
Unlike traditional credentials, Open Badges include specific claims about experience,
skills, or competencies of a learner. These can be associated with detailed evidence to
supports those claims [3]. Different layers of metadata embedded in Open Badges
make each badge a verifiable representation of a competency, skill, literacy etc. Each
badge contains metadata about the issuer, the earner, the criteria, the description of a
badge, the issuance date, the web address to “evidence,” and other information that
make each Open Badges to an information-rich digital artefact. The metadata is a set
of standards that make it possible for other systems to process and recognise the
badge. The metadata “travels” with a badge outside the platform in which it was
issued [3]. Once an Open Badge is earned by the learner, the assertion metadata is
created (cf. Fig. 2). Assertions includes unique information about the earned badge.
Viewers (e.g. educators, peers, employers) can verify the badge based on the metadata
contained in the assertion and displayed from earner's public collections in user hub.
Fig. 2. Open Badges Metadata.
2.3 User hub
The user hub is a user-owned repository in which badges earned from diverse issuers
or systems can be collected and managed into collections. A central user hub provided
by Mozilla is the Badge Backpack. The backpack is an authorised data repository and
a service which allows badge earners to collect and manage their badges in one place.
The mechanism for using the Badge Backpack is fully user-owned. The earner has to
accept the badge that was issued to him or her, i.e. the badge has to be “claimed” first.
The earner may or may not claim (accept) the badge and push it to the backpack.
Once a badge is stored in the backpack, the user can manage it, e.g. adjust privacy
settings, group badges and display badges in an OBI-compatible system (e.g. Moodle,
Wordpress, Drupal) (cf. Fig. 3).
Fig. 3. Open Badges Backpack.
3 Assessment of 21st century skills with Open Badges
As the demand for skills continues to shift towards more sophisticated tasks requiring
creativity, innovation, communication, collaboration and problem-solving skills [2],
and as traditional forms of assessment and credentialing still focus on subject-specific
knowledge, alternative ways of assessment are needed to make 21st century skills and
the criteria necessary to demonstrate these skills transparent and accessible to learners
themselves, but also to educators, employers, policy-makers and other stakeholders.
Despite the fact that the Internet has opened new opportunities for learning, the still
prevailing knowledge-based, summative forms of assessment make use of 20th-
century approaches which do not allow to capture a full picture of individual
competencies and achievements needed for the demands of the 21st century [4], [5].
Especially, in view of competency-based education, new forms of assessment, such
as multiple measures, measures of opportunity to learn, performance assessment,
peer-driven assessment, self-assessment, are needed to capture and demonstrate
competencies developed inside and outside classroom environments [4]. Furthermore,
21st century skills encompass a complex range of diverse elements beyond
knowledge, establishing close links to the identity of the learner, which is derived
from multiple views about oneself, as well as interactions, relationships and
memberships [6]. Taking this perspective, it is important for the assessment of 21st
century skills to focus on identity as a key outcome of skill development [9].
Open Badges may be used to support the assessment of 21st century skills marking
the pathways of competency and simultaneously identity development over time. For
example, an Open Badge may be issued to recognise a set of characteristics making
up for what is considered as “multicultural literacy” and a 21st century skill. An Open
Badge representing multicultural literacy would specify (observable) characteristics
of a multiculturally literate person in form of criteria for earning the badge. A learner
could then “claim the badge” by providing the evidence with the aim of
demonstrating how she/he fulfils the requirements defined as necessary to consider a
person multiculturally literate. The process of claiming a badge at the same time
constitutes warranting claims about own identity (such as: “I consider myself
multiculturally literate”).
Open Badges may be used to support both formative and summative assessment.
As an instrument of summative assessment or “assessment of learning”, Open Badges
are used to mark the achievement of a certain level of proficiency at the end of a
learning event, e.g. upon completion of an entire MOOC. As an instrument of
formative assessment or “assessment for learning”, Open Badges are used to provide
feedback and outline possible paths of progress in further course of learning [5], [7].
Assessment for learning with Open Badges may exploit diverse assessment methods
such as observation, reflection, demonstration and interpretation of evidence in ways
that enhance ongoing learning [8]. As instruments of formative assessment, Open
Badges may be used as springboards for discussions and reflections about learning.
The possibility to include evidence in Open Badges enables the learner to attach
selected evidence to support the claim about a given skill, in this way demonstrating
what the learner thinks counts as relevant evidence. This again can be included in
formative assessment, with the type of evidence provided by the learner being used as
an object of discussion and self-/reflection. The evidence contained in the badge
allows to assess this evidence against the criteria defined for earning a badge but also
to compare the evidence of one learner with the evidence of other learners, in this way
enhancing social awareness. This type of evidence-based assessment helps to
overcome a the weaknesses of current assessment practices, enhancing multi-
perspective, negotiable and distributed forms of assessment, in which a person being
assessed takes an active role [9].
Given the evolving nature of the open standard of Open Badges (the standard
keeps being extended by the Open Badges developer community), new features, such
as the endorsement extension, additionally enhance such alternative forms of
assessment as distributed assessment. The endorsement feature enables to aggregate
assessment from many sources in one single digital artefact. In this way other
stakeholders, including peers or employers, may indicate approval and acknowledge
the result of assessment by endorsing an assertion included in the Open Badge.
4 Case study “BeuthBonus Badges for Migrant Academics”
The following case study from Beuth University of Applied Sciences in Berlin,
Germany, illustrates the transformative potential of distributed assessment with Open
Badges based on learner-generated evidence and affirmation of learner’s claims by
significant others. The case study outlines some of the psychological and social
effects of distributed assessment with Open Badges, especially in view of identity
formation and employability. The context of the case study is a qualification program
BeuthBonus for migrant graduates with higher education degrees acquired outside
Germany: http://beuthbonus.beuth-hochschule.de. The target group are migrant
academics who, despite their degrees, cannot find employment adequate to their level
of education. This especially includes “at-risk academics”, such as unemployed
graduates, graduates working under precarious conditions, refugees and asylum-
seeking academics. The objective of the qualification program BeuthBonus is to
enhance employment opportunities in IT-related fields by updating existing skills and/
or developing additional skills (upskilling). This includes competencies or literacies
considered as “21st century skills”, especially the ones sought after on the German
labor market and in the information technology sector.
The BeuthBonus program at Beuth University is part of the federal network
“Integration through Qualification” (Network IQ): http://www.netzwerk-iq.de/
network-iq-start-page.html, which objective has been to improve employment
opportunities for migrants in Germany. The BeuthBonus program as part of the larger
Network IQ is committed to the objectives set out by the Network IQ and contributes
by enhancing employability and work opportunities of migrant graduates in IT fields.
BeuthBonus issues Open Badges as digital (micro) credentials to participants who can
successfully demonstrate 21st century skills in areas such as language, intercultural,
team, management, leadership or innovation.
The BeuthBonus program is also one of the pilot programs in the European
Erasmus+ strategic partnership on Open Badges called “Open Badge Network,
OBN”: http://openbadgenetwork.com. The Open Badge Network (OBN) brings
together organisations from across Europe to support the development of an Open
Badge ecosystem, promoting the use of Open Badges to recognise non-formal and
informal learning. This project aims to provide a trusted source of independent
information, tools and informed practice to support people who are interested in
creating, issuing and earning badges across Europe. One of the outputs of the OBN
Erasmus+ project are guidelines for establishing Open Badge practices within
territories, based on existing models of practice. The guidelines are tested in two city-
based projects in Berlin (DE) and Groningen (NL) and will be later refined based on
the feedback from both pilots. BeuthBonus as an OBN pilot is committed to the
overall objective of the Open Badge Network (OBN) and contributes by providing an
example of establishing Open Badges at a territorial level.
Open Badges issued in the BeuthBonus program have been designed in
cooperation with local stakeholders as a set of seven 21st century skill sets which are
crucial for successful employment in the IT sector and beyond. To identify and define
the seven skill areas, the insights from the predecessor project CreditPoints: http://
creditpoints.beuth-hochschule.de/ (which was one of the first pilot projects in
Germany aiming at enhancing employment opportunities of highly qualified migrants
as opposed to a large number of programs for low-skilled migrants), and the
recommendations from BeuthBonus project advisory board with representatives from
regional organisations from the IT sector have been taken into consideration. The
seven skill areas encompass: (1) team player skills, (2) leadership skills, (3) language
skills, (4) management skills, (5) social media skills, (6) intercultural skills and (7)
innovation skills. Each skill area has been defined at three levels of proficiency (A*,
B** and C***) following the indicators recommended by the European Qualification
Framework (EQF). Figure 4 illustrates the design on Open Badges at three levels in
the BeuthBonus program for the team player skill set.
Furthermore, each skill area has been defined in view of employment, i.e. using
criterial and language relevant to the world of work rather than world of academia.
Based on the conceptualisation of the key skill areas, Open Badges have been
designed using a number of tools and methods, such as (a) ProfilPASS: http://
www.profilpass-online.de (a tool for biographical skill assessment), (b) the Badge
Canvas: http://www.digitalme.co.uk/assets/pdf/digitalme-badge-design-canvas.pdf (a
tool for concept design of Open Badges), and (c) EQF competency level indicators.
Open Badges in the BeuthBonus program, called BeuthBonus Badges, were
developed in a series of workshops, in which both project staff and program coaches
jointly and iteratively developed the concepts of BeuthBonus Badges. The program
coaches working closely with BeuthBonus participants throughout their individual
qualification path, help the participants develop or advance competencies in the seven
areas of the 21st century skills. In later stages of development, the members of the
advisory board were consulted on the different elements of design and the feedback
from the advisory board was included to improve the overall design (technical,
textual, visual) of BeuthBonus Badges.
Fig. 4. BeuthBonus Badges at three levels of proficiency (A*, B** and C***).
All BeuthBonus Badges, i. e. 21 badges in total (7 skills areas x 3 levels each),
have been designed in two versions - national and international - in order to enhance
the employment opportunities in the highly diversified and globalised IT sector:
• The national version in German language may be found in the Learning
Management System of Beuth University (Moodle) and on the project website:
http://beuthbonus.beuth-hochschule.de/qualifizierung/badges-digitale-
kompetenzabzeichen/
• The international version in English language may be found on UK-based
Open Badging platform Open Badge Academy provided by a non-profit
organisation DigitalMe: https://www.openbadgeacademy.com/beuthbonusbadges
BeuthBonus Badges in both national and international versions are available to all
participants in the BeuthBonus program. Participants learn about BeuthBonus Badges
at the beginning of their qualification program, so that they can understand early on
which skills beyond their degrees matter for employment success and what criteria
needs to be fulfilled to achieve a specific badge. In an introductory workshop, the
participants choose the badges they would like to earn from the BeuthBonus program
and are encouraged to reflect which criteria are likely for them to achieve. This
preliminary activity aims at enhancing reflection about own identity (Who am I? What
are my current skills/resources?) and setting of own goals (What do I want to
achieve? Why is it important?). Later in the program, participants can go back to their
original choices and review them on their own or in consultation with coaches, e. g.
taking a joint decision that a skill at highest level cannot be reached yet or may not be
relevant for the employment type strived for by the participant.
Since Open Badges are designed as claims about being literate in a specific area,
the processes of (internal) self-assessment and (external) assessment by coaches,
project staff and peers, which are an integral as part of the Open Badges concept in
the BeuthBonus program, contribute to identity formation of migrant academics. The
emerging identity is constituted by a reciprocal process of (a) warranting claims or
disclaims related to particular skill areas by the participant himself/herself and (b)
affirming or disaffirming claims/disclaims by significant others such as coaches and
peers [9], [10]. In this way, internal and external assessment embedded in the Open
Badge practice in the BeuthBonus program may be viewed as distributed assessment.
This type of assessment draws upon “the active engagement of the persons being
assessed in the joint production (with the assessors) of what are taken to be facts
about those persons” [8]. In this sense, distributed assessment with Open Badges
entails learners as producers of claims through evidence and assessors as interactors
who support learners in reviewing claims in the process of “synthesis of (internal)
self-definition and the (external) definitions of oneself offered by others” [10].
BeuthBonus Badges are issued manually by assessors, i. e. the process of issuing
badges includes the human judgment as opposed to an automatic issuing, in which
case the quality of the evidence cannot be properly assessed. The manual issuing
practice is crucial to the distributed assessment approach, as it enhances the
negotiation and renegotiation of what counts as relevant evidence.
Assessors in the BeuthBonus program apply diverse assessment methods
depending on the skill area being assessed. For example, leadership skills are assessed
using the biographical methods using the assessment tool ProfilPass and focusing on
the assessment of personal reports about past experiences in which leadership skills
were demonstrated according to the learner, while social media skills are assessed
based on current and directly observable activities of a learner on social media.
In general, distributed assessment may be viewed a process of exploration and
interpretation of facts, and in consequence affirmation or disaffirmation of claims
about skills within a complex education-employment nexus [9]. This distributed
process has a potential to transform the identities of learners as it is the case in the
BeuthBonus program. By being an active party in the assessment process, migrant
academics can engage more effectively in warranting their claims about identity.
5 Summary
Open Badges have been discussed and used as a game changing technology and
approach with a potential to transform assessment practices [7]. Since assessment
with Open Badges supports generating claims about individual skills and evidence to
support those claims, assessment produces rich information such as metadata and
learner-generated evidence rather than simple assessment scores. This rich
information has both psychological and social implications. From a psychological
perspective, it contributes to identity formation of a learner who has to engage
intensely with a given skill in order to understand what is required to earn a badge.
This process can be enhanced by using distributed assessment, in which learners and
assessors negotiate the meaning of criteria and evidence. From social perspective, rich
information produced and documented by assessment with Open Badges is available
to others, including educators, peers and employers and open new opportunities for
other to engage with the earner of the badge. Displaying own badges with individual
evidence can encourage others to get in touch with the holder of the badge, for
example in view of joint projects, employment or other opportunities. In this way,
assessment supported by Open Badges extends the traditional view of what counts as
assessment and what the role of assessment may be [7].
The specification of criteria necessary to earn a badge contributes to an increased
transparency of assessment practices which in turn may lead to improved quality of
assessment altogether. Alternative forms of assessment, such as distributed
assessment, enhance meaningful assessment practice as part of identity formation and
extends the reach of assessment from education to self-/employment. New types of
information linked to assessment, such as learner-generated evidence, social
interactions, endorsements, have the potential to transform assessment practices and
improve the visibility of 21st century skills and their distribution among citizens.
Acknowledgments. This contribution is part of the effort to establish Open Badges in
Europe as part of the strategic partnership (KA2) called “Open Badge Network”
founded by the European Erasmus+ program: http://www.openbadgenetwork.com
References
1. Casilli, C., Hickey, D.: Transcending conventional credentialing and assessment paradigms
with information-rich digital badges. The Information Society, 32:2, 117-129 (2016).
2. OECD: Skilled for Life? KEY FINDINGS FROM THE SURVEY OF ADULT SKILLS
(2013). Retrieved from http://www.oecd.org/skills/piaac/SkillsOutlook_2013_ebook.pdf
3. Badge Alliance Endorsement Working Group. Badge Endorsement: Getting Started (2014).
Retrieved from: https://goo.gl/c7uRSo
4. OECD: Formative Assessment – Improving Learning in Secondary Classrooms, (What
Works series), Paris (2005).
5. Grant, S.: What counts as learning. Open Digital Badges for New Opportunities. The DML
Research Hub Report Series on Connected Learning (2014). Retrieved from http://
dmlhub.net/wp-content/uploads/files/WhatCountsAsLearning_Grant.pdf
6. Johnson, D.W., Johnson, R.T. (2012). Coooperative Learning and Conflict Resolution:
Essential 21st Century Skills. In J. Bellanca, R. Brandt (Eds.). 21st Century Skills:
Rethinking How Students Learn. Solution Tree, Leading Edge Series, pp. 201-220.
7. Hickey, D., Willis, J.E.: Research Designs for Studying Individual and Collaborative
Learning with Digital Badges. In Proceedings of the Open Badges in Education (OBIE
2015) Workshop, Poughkeepsie, New York, USA, (2015).
8. Torrance, H.: Formative assessment at the crossroads: conformative, deformative and
transformative assessment, Oxford Review of Education, 38:3, 323-342 (2012).
9 Holmes, L. (2002). Emergent identity, education and distributed assessment: an
ethnomethodological exploration. Ethnomethodology: A Critical Celebration conference,
University of Essex, March 2002. URL: http://www.re-skill.org.uk/papers/assessment.html
10. Jenkins, R. (1996). Social Identity. London, Routledge.