=Paper=
{{Paper
|id=None
|storemode=property
|title=Repository Services for Outcome-based Learning
|pdfUrl=https://ceur-ws.org/Vol-681/paper01.pdf
|volume=Vol-681
}}
==Repository Services for Outcome-based Learning==
https://ceur-ws.org/Vol-681/paper01.pdf
Repository Services for Outcome-based Learning
Michael Totschnig1, Michael Derntl2, Israel Gutiérrez3, Jad Najjar1,
Roland Klemke4, Joris Klerkx5, Erik Duval5, Franz Müller1
1
Vienna University of Economics and Business
2
University of Vienna, Faculty of Computer Science
3
Universidad Carlos III de Madrid
4
Open University of the Netherlands
5
Katholieke Universiteit Leuven
michael.totschnig@wu.ac.at
Abstract. Despite the existence of numerous standards and specifications in
technology-enhanced learning, there is a lack of interoperability of artifacts and
services throughout the whole lifecycle of outcome-based education. In this pa-
per we present the concept and prototypical implementation of an open archi-
tecture that aims to remedy these issues by providing a unified metadata and
service layer for making key educational resources sharable, storable, findable,
and interoperable. The reference model and its supporting technology architec-
ture are tested by a family of prototypes implemented as extensions to or adap-
tations of existing mainstream systems like Moodle, .LRN, Elgg and Facebook.
Keywords: outcome-oriented education, learning object repository, middle
layer
1 Introduction
Conceiving services that make learning resources usable for design and delivery of
outcome-based learning is a challenge that current learning object repositories are not
yet able to meet. On the one hand, learning designs are dissociated from the learning
opportunities (course offerings) where they have been or will be put into context, and
thus it is difficult for the users (both the learner and the teacher) of those repositories
to understand how they can benefit from these resources in the most fruitful way. On
the other hand, learning outcomes are not yet defined and linked in a systematic way
to learning designs, and even if they were, information about the learning or teaching
history of the user would be needed in a standardized format, if the system was
supposed to make meaningful suggestions.
The ICOPER Reference Model (IRM) aims at providing a framework through
which innovative learning processes that exploit rich linkages between teaching me-
thods, learning designs, learner assessments, learning resources, learning outcome
definitions, user profiles for achieved learning outcomes and for learning needs, and
learning opportunities can be stored, shared and delivered through standardized ser-
vices and data formats. The Open ICOPER Content Space (OICS) is conceived as the
testbed for implementing the IRM. Figure 1 displays the key processes in designing
3
and delivering outcome-based education. From a pedagogical perspective, the align-
ment of learning outcomes with assessment methods and teaching methods is one of
the core foundations of the Bologna Process [6]. Implementing this foundation, the
IRM as a reference model and the OICS as its reference implementation represent a
visionary yet realistic approach to fully (i.e. conceptually and technically) supporting
key pedagogical processes. These include the definition of intended learning out-
comes; the reuse and creation of teaching methods and learning designs; the selection
of content; and the offering and delivery of concrete learning opportunities. Going
beyond current pedagogical and institutional practice, this approach even enables
management of learner assessments, including verification and certification of
learners’ achievement of intended learning outcomes; publishing of learning outcome
profiles; and institutional exchange of achievement data (between and among
universities and companies).
Fig. 1. Key processes in outcome-based education.
In this paper, we present the architecture of the services that the OICS offers to
client applications. These services draw upon existing specifications for content
aggregation, federated search and publication that have been validated in numerous
projects and organizations such as MACE, MELT, GLOBE, EducaNext and
ARIADNE. All of those have been integrated into a middle layer API that provides
easy-to-use services for the support of outcome based learning and teaching scenarios.
After describing the conceptual and data models defined in the IRM, we explain
how these models are made available through the OICS services, and how these ser-
vices are consumed by the prototypical tools developed in the ICOPER project.
2 OICS Architecture
The OICS infrastructure is built as a composition of services developed and main-
tained at Vienna University of Economics and Business (WU) and K.U. Leuven
(KUL). At both sides a learning object repository is used for storing metadata har-
4
vested from the ICOPER content providers. Metadata is synchronized between them
through the Simple Publishing Interface (SPI) protocol
Content is fed into the OICS either through the OAI-PMH protocol or a publication
service based on the SPI protocol1. The ARIADNE harvester [2] enables the
management of OAI-PMH targets and is integrated with a set of services improving
the management of the repositories and the quality of the metadata:
The registry service provides a catalogue of up-to-date information about learning
object repositories (LORs) and allows the harvester instance used for the OICS to
retrieve information about the OAI-PMH endpoint.
The validation service ensures that only metadata records are stored which comply
with the ICOPER LOM Application Profile (see Section 3.1), which is based on
IEEE Learning Object Metadata (LOM).
The transformation service allows applying mappings between foreign vocabula-
ries to ICOPER specific ones.
The identifier service generates unique and persistent identifiers that are added to
metadata records upon ingestion into the OICS. Resolution to multiple views of the
resource is provided through a simple web service.
These services are explained in more detail in [2].
2.1 OICS Middle Layer
The OICS middle layer tries to bundle the requirements of the IRM together in a co-
herent API that is accessible from a variety of systems and tools (see Figure 2). Its
key focus is the integration of concepts and data related to the key processes in out-
come-based education. The OICS middle layer provides services for search and re-
trieval of learning resources, for publication, for the management of users and groups
and for the management of learning outcome profiles within these key processes.
Search and Retrieval. The search and retrieval service gives access to the OICS
resources by providing specific access methods for the different types of objects
(learning outcome definitions, teaching methods, learning designs and learner assess-
ments). Three example bindings for this service have been implemented:
The ATOM binding exposes all resources as ATOM feeds that can be filtered
based on values in the LOM metadata.
The JSON binding uses a REST interface and provides the results to the client
tools in the JavaScript Object Notation data format. JSON is a lightweight data
format heavily used by web developers due to its simplicity (e.g. native evaluation
of results in JavaScript) compared to the traditional XML data format approaches,
which often require cumbersome DOM-based processing.
A PHP search script forwards PLQL [3] expressions to the SQI SOAP end point
provided by the KUL repository.
Publication. The OICS implements the Sword/AtomPub binding of the SPI -
specification, learning objects and metadata records can be published to collections.
Extending the SPI specification, The OICS implementation allows updating and re-
1 For metadata editing and demonstration purposes the OICS also provides a web UI.
5
trieving of parts of the metadata record thus making it very easy to query and mani-
pulate individual metadata fields relevant in specific use cases.
Open ICOPER Content Space
Learning Outcome search / index Desktop
applications
Definitions
search / index
Learning Outcome / export
Profiles Learning
Management
Service Layer
Systems
search / index
Teaching Methods
search / index
Learning designs/ / recommend
Units of Learning Widgets, 3rd
party tools, …
Assessment search / index
Resources
…
Content provider
Fig. 2. OICS middle layer
Learning Outcome Profiles. Users of the OICS can manage their learning out-
come profile according to the Personal Achieved Learning Outcomes (PALO) data
model (see Section 3.3) both through a web UI and the same publication service as
used for publishing learning resources.
3 Conceptual and Data Models of the ICOPER Reference Model
The OICS as a repository managing shareable educational resources implements the
following data models: (1) the ICOPER LOM Application Profile (AP) as a unified
metadata layer above learning designs, teaching methods, assessment methods,
learner assessments and learning content, (2) an IEEE RCD-based specification for
representing learning outcome definitions (LOD) and (3) the Personal Achieved
Learning Outcomes (PALO) specification [1] for learning outcome profiles.
6
3.1 ICOPER LOM Application Profile
A metadata schema is needed to describe and store resources in a referatory. As one
of the most widely used, supported, and implemented standards, IEEE LOM was
profiled to enable the description of learning design resources (teaching methods,
learning designs, assessment methods, and learner assessments), but care was taken to
make the same profile also applicable to other types of learning resources stored in
the OICS. The resulting ICOPER LOM AP ensures that resources described using
different standards and specifications like IMS Learning Design (LD), IMS Question
and Test Interoperability (QTI), and so forth, become semantically interoperable.
As one central feature, the AP allows defining intended learning outcomes for all
resources in the OICS; to enable this, the Educational category of LOM was extended
with an element containing a link to a learning outcome definition. This simple exten-
sion enables several added-value use cases and scenarios, as indicated and prototypi-
cally demonstrated in Section 6.
3.2 Learning Outcome Definitions (LOD)
The Learning Outcome Definitions (LOD) data model defines a conceptual base
schema for describing and sharing learning outcome definitions in the context of
online and technology enhanced learning. The data model provides a way to capture
the key characteristics of a learning outcome, independently of its use in any particu-
lar context or target group. This model should enable the storage and retrieval of
learning outcomes across learning systems that deal with learning outcomes data.
This specification is based on, and is an application profile of, the IEEE Reusable
Competency Definitions (RCD) standard. EEE RCD is the only widely accepted
standard for describing competencies. It is a continuation (and replacement) of the
early efforts on the development of IMS Reusable Definition of Competency or Edu-
cational Objective (RDCEO). This specification profiles IEEE RCD with one
metadata element defining the type of the learning outcome and its associated value
domain to capture whether a learning outcome refers to knowledge, skill or
competence following the definitions of the European Qualification Framework [4].
Instances that conform to the LOD specification also conform to the IEEE RCD stan-
dard, which ensures interoperability between the OICS and other systems.
3.3 Personal Achieved Learning Outcomes (PALO)
The Personal Achieved Learning Outcomes (PALO) data model [1] is a simple
schema proposed to capture information on knowledge, skills and competences
achieved by a learner and relations between those outcomes. Information on the
context where the learning outcomes are obtained or applied, evidence records and
levels (e.g. proficiency level) associated to the outcomes are also part of this schema.
One of the main challenges of communities and systems that deal with learning
outcome information is interoperability. Different communities and systems may use
different data models to represent information on skills, knowledge or competence
obtained by a person that is required for a job or a task. The PALO specification is a
step towards a common model supporting the exchange of such data, to enhance inte-
7
roperability of personal learning outcome information between, for example, learning
management systems, e-portfolios, social applications and recruitment systems.
This data model enables describing relations between learning outcomes of
learners, in addition to contextual and evidence related information. The PALO
schema should enable capturing the following:
Relations between achieved learning outcomes, regardless of the taxonomies or
ontologies they belong to;
Contextual information on where the achieved learning outcome is obtained or
applied;
Information about all types of evidence and assessment that prove the achievement
of a learning outcome;
Information about levels and ranking of an achieved learning outcome, like
proficiency level.
The PALO model has been proposed as EU specification for capturing data of
personal achieved learning outcomes at CEN Workshop on Learning Technologies2.
4 Linking Learning Outcomes, Teaching Methods, Learning
Opportunities and Learning Designs
The following hypothetical scenario illustrates how university administration, faculty
and learners can benefit from making the systems that manage teaching and learning
processes interoperable through the adoption of the services described in the IRM and
provided by the OICS.
1. At University X, a new curriculum for a course program is developed. Each course
description is linked to learning outcome definitions (LOD) and suggested teaching
methods (TM), both stored in the OICS.
2. Best practices for TMs have been elaborated by a consortium of universities, and
for each of them a template has been elaborated that can be used by instructional
designers.
3. An instructional designer creates a new learning design (LD) for the program. He
retrieves the LODs mentioned in the curriculum from the LOD repository and
searches the TM repository for a suitable template including assessment methods.
He imports it into his authoring environment, adds resources and learner assess-
ments retrieved from the OICS. Since learner assessments are linked to learning
outcomes and assessment methods, he is able to retrieve the most relevant re-
sources. The LD is made available for feedback in a restricted collection.
4. Additional links to LODs and TMs can be added by program management.
5. Once the LD has been approved by the program management, the LD is published.
6. It is imported into the institutional LMS, and automatically a learning opportunity
is pushed to a registry service for learning opportunities.
7. Upon each completion of the learning opportunity, feedback from teachers and
learners is collected and the metadata is enriched.
2 See http://www.cen.eu/cen/Sectors/Sectors/ISSS/Activity/Pages/WSLT.aspx
8
8. Learners completing the learning opportunity, including the successful finishing of
learner assessment (LA), have their learning outcome profile augmented with
entries for each achieved learning outcome. The achieved learning outcomes are
evidenced by assessment records (AR), results of the assessment process.
9. Prospective learners that use the OICS LOD repository for identifying learning
outcomes that correspond to their learning needs, will be able to retrieve other
learners that already have achieved these outcomes, and since their profile also
links to units of learning and learning opportunities, relevant recommendations can
be presented to the learner.
The different parts of this high level scenario have been implemented by partners
of the ICOPER consortium in various prototypes that make use of the OICS reposi-
tory services.. In the following, we describe for each of the processes in Figure 1 how
they are realized in some of the prototypes.
4.1 Learning Outcomes - Definitions
Learning outcomes can be defined for a single course, taught by several teachers, or
be standardized across universities or whole domains. Instead of describing learning
outcomes from scratch every time a new course is created, instructional designers
should be provided with a list of relevant learning outcome definitions that they can
link to their courses. For example, IEEE and ACM Computer Science task force has
defined the list of learning outcomes for all computer science courses. These learning
outcomes have been imported into the OICS and can be reused across different course
and universities. In this way, different universities use common learning outcome
terms for describing what students would achieve by finishing a specific course.
The OICS also provides a simple-to-use web interface for creating and editing
them. A browsing interface is currently implemented for the ICOPER public website.
New definitions can also be created from within several authoring environments and
LMS and are submitted to the OICS through the publication service.
4.2 Teaching Methods and Learning Design: OpenGLM
OpenGLM (short for Open Graphical Learning Modeler) is an open-source learning
design editor, developed at Universität Wien. It is able to manipulate learning designs
compliant with the IMS Learning Design specification. It was built to visually support
the creation and reuse of teaching methods and learning designs. OpenGLM uses the
search and retrieval services and the publishing service of the OICS Middle Layer; it
uses those parts of the services that are related to retrieving, searching, editing,
enriching and publishing learning designs (packaged as IMS LD units of learning),
teaching methods and learning outcomes. OpenGLM is the key prototype supporting
use cases in the “Teaching Methods and Learning Design” process in Figure 1.
4.3 Learning Content: OICS Roundtrip Authoring Re-use
Open University of the Netherlands and Humance AG have implemented an OICS
roundtrip authoring re-use prototype, that brings together an asset management soft-
9
ware (MediaLibrary), an authoring environment (author42), and an LMS (OLAT).
This prototype addresses the authoring and creation of units of learning with strong
support for re-use of existing materials. It focuses on three main processes:
1. The collaborative collection and organization of media assets.
2. The collaborative creation of learning units based on these media assets.
3. The preparation for re-use of media assets and learning units.
The prototype has several connections to the OICS: through OAI-PMH targets in-
dividual media assets (MediaLibrary) as well as units of learning (author42, OLAT)
are made available to the OICS infrastructure together with their LOM-based meta-
data. The OICS search services are integrated in author42 to enable seamless integra-
tion of existing media in the authoring process. More details on this prototype can be
found in an accompanying paper [5].
4.4 Learning Opportunities
Knowledge Markets Consulting Ges.m.b.H. is implementing a prototype on top of its
2know2 platform, that allows to announce learning opportunities for courses that are
linked to learning outcomes and teaching methods. The learning outcomes and
teaching methods are directly stored in and retrieved from the OICS via the search,
retrieval and publishing services of the OICS Middle Layer. New learning opportuni-
ties can be announced at the 2know2 platform with a news article and an RSS feed
and can also be published in the repository for learning opportunities at the OICS.
4.5 Learner Assessment
University Carlos III of Madrid (UC3M) has developed a prototype integrated into the
.LRN platform that demonstrates various use cases related to the learner assessment
process:
(1) Outcome-based search of assessment resources. Once defined assessment
method and learning outcomes of a course, the instructional designer should find
appropriate assessment resources aligned to them, and this process is enabled by the
.LRN module. The designer is able to search for assessment resources in the OICS by
keyword, but he is also able to filter the results by the intended learning outcomes of
the course (available in the learning outcome definitions repository) and by the
assessment method he decided to use to orchestrate the resources.
The described use case makes use of the search service of the middle layer API of
the OICS in order to search assessment resources from several repositories. This ser-
vice also provides means for the filtering process by learning outcomes and/or as-
sessment method.
(2) Sharing annotations about assessment resources. Once an activity involving
a published assessment resource has finished, teachers can make use of the annotation
system in order to provide information about the students’ performance. Teachers
could also collect students’ feedback to annotate the assessment resource with. This
information enriches the published resource for potential instructional designers will-
ing to re-use it.
10
This sharing annotation system uses some services of the OICS via the middle
layer API. Firstly, the publication service is used to publish the resources and their
LOM metadata in the OICS. Secondly, the service that allows updating the metadata
record of a resource is used to update it with annotation information.
4.6 Learning Outcomes – Achievements
UC3M’s prototype also takes care of the publication of the achieved learning out-
comes into a learner learning outcome profile on the OICS.
After the completion of the course, and therefore the assessment activities, it is
time for teachers to officially close the course. The implemented application provides
the teacher with an interface to facilitate this task. It shows a list of the students of the
course and the assessment result (grade) of each assessment activity carried out in the
course. There is also a final grade automatically calculated as the arithmetic mean of
all the results, but the teacher can modify it taking into account other factors. The
teacher can also provide some textual feedback for any student’s final grade.
Once the teacher has finished this task, he can officially close the course. The re-
sults of this action is that the profiles of all the students passing the course (in this
case it means a final grade of 5 or more because the prototype uses a grading scale 0
to 10, proper to the Spanish system) will be updated with the achievements of the
course, that is, the learning outcomes achieved. These achievements are also evi-
denced by an assessment record, which is an official record corresponding to the final
grade of the course and has the University as the assessing body.
This process uses the service of the OICS middle layer that allows updating the
learner’s profile with assessment records and achievements.
Other tools and environments have been integrated with OICS services, and are
able to publish achievements into a learner’s profile or display them:
Umeå University, Sweden, has developed a Moodle block that imports learning
designs into a course and exports learning outcomes for students enrolled in it.
IMC, Germany, has extended its LMS Clix with functionality for the execution of
outcome-based learning designs as well as the management of learners' PALO.
eXact learning solutions (formerly Giunti Labs), Italy, has implemented access to
OICS’ learning outcome related features into its learn eXact enterprise LCMS.
AGH – University of Science and Technology, Poland, has integrated the display
of PALO profiles into Facebook.
Tallinn University, Estonia, has integrated the e-portfolio environment, Elgg, with
OICS services for searching learning resources and for publication of PALO data.
5 Conclusions and Future Directions
With the services we have designed and implemented for the OICS, we want to make
learning resources available in the contexts where outcome related education takes
place: LMS, personal learning environments, social networks. By providing richer
linkages between learning needs, learning designs, teaching methods, learning
outcomes and learning opportunities, teachers and learners will be able to make more
11
innovative use of available learning objects. The OICS is also designed to store
information about user’s experiences as annotations, and thus to create still more
opportunities to discover relevant resources.
In order to provide a good practice of enabling the design and delivery of outcome
based learning, prototypes implemented in ICOPER project extended the functionality
of existing systems that are already used by teachers and learners.
Currently, the implemented prototypes are being evaluated by target end users like
learners, teachers and instructional designers. The goal of the evaluation is to deter-
mine:
Ease of use, usefulness and completeness of implemented outcomes based
functionalities;
Added value and innovation of introduced functionalities to target users;
The extent that such applications are being adopted by universities
Interoperability level of data exchanged using the specifications adopted and
implemented in this project.
Acknowledgements. The work presented in this paper was supported by the
European Commission in the eContentplus project ICOPER (ECP-2007-EDU-
417007).
References
1. Najjar, J., Derntl, M., Klobučar, T., Simon, B., Totschnig, M., Grant, S., Pawlowski, J.: A
Data Model for Describing and Exchanging Personal Achieved Learning Outcomes
(PALO), International Journal of IT Standards and Standardization Research (IJITSR), vol.
8, issue 2 (2010)
2. Klerkx, J., Vandeputte, B., Parra, G., Santos, J.L., Van Assche, F., Duval, E.: How to
Share and Reuse Learning Resources: the ARIADNE Experience, European Conference
on Technology Enhanced Learning (ECTEL) (2010)
3. Ternier, S., Massart, D., Campi, A., Guinea, S., Ceri, S., Duval, E.: Interoperability for
Searching Learning Object Repositories. The ProLearn Query Language. In: D-Lib
Magazine. vol. 14, nr. 1/2 (2008). http://www.dlib.org/dlib/january08/ceri/01ceri.html
4. EQF. The EQF for lifelong learning, Office for the publication of the EC, ISBN 978-92-
79-0847-4 (2000)
5. Klemke, R., Schmitz, B.: Authoring for re-use in outcome-oriented learning scenarios.
Se@m 2010, the 4th International Workshop on Search and Exchange of e-le@rning
Materials, Barcelona, Spain (2010)
6. European Commission: The Bologna Declaration (2000).
http://ec.europa.eu/education/policies/educ/bologna/bologna.pdf
12