=Paper=
{{Paper
|id=Vol-1599/5CHANGEE_2015
|storemode=property
|title=An Infrastructure for Cross-platform Competence-based Assessment
|pdfUrl=https://ceur-ws.org/Vol-1599/5CHANGEE_2015.pdf
|volume=Vol-1599
|authors=Julian Dehne,Ulrike Lucke
|dblpUrl=https://dblp.org/rec/conf/ectel/DehneL15
}}
==An Infrastructure for Cross-platform Competence-based Assessment==
https://ceur-ws.org/Vol-1599/5CHANGEE_2015.pdf
Trends in Digital Education:
Selected papers from EC-TEL 2015 Workshops CHANGEE, WAPLA, and HybridEd
An infrastructure for cross-platform competence-based
assessment
Julian Dehne and Ulrike Lucke
University of Potsdam,
August Bebel Strae, 64625 Bensheim, Germany
{dehne@uni-potsdam.de, ulucke@uni-potsdam.de,
http://apache.cs.uni-potsdam.de/de/profs/ifi/mm
Abstract. Competency based learning and e-portfolios are integral parts of
modern teaching repertoires. Media and computer technology play an important
role in supporting such scenarios. This paper presents an infrastructure con-
sistent of a generic competency database that can be connected to various learn-
ing management systems or e-portfolio systems to implement competence-
based assessment. The concepts shown in this paper are an important step to-
wards supplementing traditional ways to mark students. Students’ achievements
are modelled as the acquisition of competencies and represented as such.
Keywords: e-portfolio, competency assessment, grading, infrastructure.
1 Introduction
Schools have relied on grading their students using ordinal scales or rational scales.
Depending on the educational system they might use percentage grading (Ontario) or
plain numbers from one to six (Germany). There are some concerns about the educa-
tional value of marks in principle, because the coefficient of predicting adult success
is very low [3]. In Germany schools like the Montessorischule or the Waldorfschule
do not rely on grades at all. Assuming we need grades to improve or assess learning
outcomes the question remains if the scales currently used are reliable in describing
the progress students make.
Grading like other descriptive actions depends on the technology available at the
time. Historically, distances have been measured in feet and sticks because lasers,
GPS or triangulation were not invented yet. Similarly, educational progress has been
described differently throughout history. Grading students in a modern sense using
ordinal scales has only been around since the 20th century [7]. Giving marks on a
numerical scale introduced the chance to compare one student to his fellow on a range
of subjects and curriculae. Before that teachers could only compare students in their
domain which was fully sufficient as education was limited to one profession only.
Another historic development is the decrease of canonized course contents. Following
the constructivist movement in education students should find a personalized access
29
Trends in Digital Education:
Selected papers from EC-TEL 2015 Workshops CHANGEE, WAPLA, and HybridEd
to knowledge. Furthermore, the sheer volume of knowledge has increased, so that it is
not possible anymore for any teacher to know everything in his or her subject.
If we agree with the critics of numerical grading, supplementary assessment con-
cepts are needed. This paper tries to answer the question as to whether grading by
using numerical scales could be attributed to lacking technologies. It also deals with
the question of how alternatives could be achieved. Assuming it is possible to store
students’ competencies electronically on a detailed level, a set of competencies vali-
dated by the organization and presented in e-portfolios may be a way of supplement-
ing or even replacing grades. This paper tackles the research question by looking at
ways to store competencies consistently for a larger number of people (or organiza-
tions) and in a generic fashion. This is a prerequisite for the bigger task of creating a
competency based assessment and thus sets the scope of this paper.
The paper is structured as follows. First we look into the current state of research
with a focus on (technical) competency models that help to store competencies in a
generic fashion. Next we present the competency model developed. Then we present
the implementation of the competency database and explain the intended usage in-
cluding a process model.
2 Current state of research
The Oxford Dictionary of Education defines competencies as “The ability to per-
form to a specified standard” [13]. In spite of its brevity this definition emphasizes
the two aspects this paper relies on: The possibility to link a competency to a certain
action being performed and the existence of standardized curricula that can be lever-
aged as a framework. Action competency has been described in more detail elsewhere
[14]. In the context of e-learning the term ‘competency’ has several meanings which
result in different modelling and implementation of competency aware systems.
For instance, the industrial approach sees effective management of competencies
as a way to foster human resources development [4]. The main reason to deal with
competencies or knowledge this way lies in the advantage of identifying knowledge
gaps or bottlenecks. Here the economic benefit is most visible. Competencies are then
described according to the need to rate a person’s ability to do a certain job.
Another approach focuses on the idea that learning objects such as assignments or
documents contain useful metadata [5]. This knowledge is used to create a model of
the user that has been in touch with the learning objects. Recommender systems [1]
are a logical consequence of this approach. More sophisticated teaching approaches
based on the constructivist paradigm harness the additional information available [8].
Here the competencies are modelled in a more complex fashion incorporating the
metadata of the documents, the activities of the user that can be monitored in e-
learning systems and the output generated by the learner.
The COMBA model [12] argues that competencies should not be modelled in nu-
merical fashion (such as marks) but as a nesting competency tree whose edges stand
for prerequisites. This way the learner can be assisted with summative assessment and
improved feedback on deficiencies. There have been some attempts to model compe-
30
Trends in Digital Education:
Selected papers from EC-TEL 2015 Workshops CHANGEE, WAPLA, and HybridEd
tencies this way with ontologies [2] [6]. These approaches are also compatible with
the IEEE reusable competency map [9]. The TELOS-project [10] is the biggest at-
tempt known to the author to create a generic comprehensive software framework for
competencies based on ontologies. However, it lacks some of the rationales of the
COMBA model and looks discontinued.
That is why we decided to conceptualize an improved competency model based on
existing standards that is generic enough to be useful for any domain and which can
be persisted in an competency database in order to allow for reasoning and inter-
institutional usage
3 Towards a generic competency model
One possibility to model competencies is to integrate them into the learner model.
This way competencies are described as aspects of the situation, the intended learning
outcomes and the dimensions of the indicators [15, p. 248]. Linking the evidences to
the activity model like the LMS Learning Object Metadata standard (LOM) [11] re-
duces the generality of the model if the activity model is specific to the subject.
Moreover, modelling the assessment and the indicators according to the competencies
is very difficult and time consuming if this has to be repeated every time a lesson is
planned. A more generic model is needed. Separating the competency model from the
activity model has some distinct advantages: existing metadata models can be reused
where tools are established already. The links between the competencies and their
evidences in form of activities can be pushed into the competency model or the activi-
ty model depending on the focus of the application.
The model is generic in terms of looking at aspects and relations of competencies
(structural perspective) instead of taking the perspective of the curriculum, faculty,
subject or school. Facing diverging pedagogical cultures in different disciplines this is
the only way to generalize the concept. The only restriction is based on the earlier
definition that a competency must be visible in terms of assessment and that can be
formulated in a sentence. It must relate to a performance in the real world. Teacher
education classes at our institution have created a de facto standard for formulating
competencies this way to simplify the analytical process of extracting metainfor-
mation out of the formulated competency:
Learner : operator : [action domain | knowledge domain]
List([(subcompetence)])
For instance:
– Computer science students : implement : service oriented architectures
(by programming : webservices)
(by designing : interfaces)
31
Trends in Digital Education:
Selected papers from EC-TEL 2015 Workshops CHANGEE, WAPLA, and HybridEd
– Computer science students : create : java programs
(by using : dependency injection)
This approach does not impose any requirements regarding the actual domain. It is
used in subjects as different as political science and computer science. It also means
that competencies are not computed as a result of activities. They are inserted inde-
pendently. They derive from the curricula or the teacher’s internal course structure.
By clearly separating the activity model and the competency model this way it be-
comes possible to create a big pool of competencies that can be used in various appli-
cations.
4 Implementation
Primarily, we present the features developed. After that we explain the software
prototype based on an activity model. The development of a new grading scale pre-
sented here is only one application of the database created. As a software framework
it stands for itself and can be used in all kinds of human resources oriented applica-
tions. Nevertheless, we focus on this use case because it shows the full innovative
potential.
The following list highlights some of the features we have developed:
– The competency database
– Excel import of competencies
– Web services (SOAP, SPARQL and REST)
– Plugins for learning management systems
Our goal was to implement a prototype that can be used as a read-only competency
database. The excel import allows us to role it out at the institutional level. Web ser-
vices are required to connect the prototype to other software systems like the campus
management system or the library. Independency was also reached with the user inter-
face by abstracting from the context with a mixture of webservices and Javascript
injection. Most importantly, the persistence layer was implemented using an rdf/owl
format stored in a high performance triple store.
Linking competencies to actual performances has been implemented in a very gen-
eral way. Evidence links may point to websites of the e-portfolio or tasks achieved in
a learning management system. This leaves it to the teacher to decide what counts as
an evidence. This could even be a link to a website showing exam results which
means coming full circle concerning the grading concept. All the systems providing
evidences for a competence must implement a webservice that requires a link and a
readable name. Showcase implementations include Moodle (LMS system) activities
and Liferay (portlet container) group activities. The figure 1 on page 5 shows the
implemented system from a component perspective with the example of a university
as a learning organization (instead of school or company).
32
Trends in Digital Education:
Selected papers from EC-TEL 2015 Workshops CHANGEE, WAPLA, and HybridEd
Fig. 1. Component view of the implemented competency database
5 Process model
The first step is to agree on a set of competencies and their structure for a given or-
ganization. Most of the time a curricula is given which can be used as a starting point.
However, in order to arrive at a useful level of detail the teachers in the organization
have to enter their course structure in terms of competency graphs into the database.
Both specifying the learning goals and ordering them in in terms of learning trails
formally are time consuming. Here it is up to the organization to enforce compliance.
One could argue that good classes should always be planned properly but reality may
be different. One of the advantages of the approach presented in this paper is that
once this has been done for an organization this can be used by similar organizations.
Existing competency schemes can be used and pooled here. For instance, the Europe-
an Union has created a competency matrix for language education. Schools have in-
ternal curricula in form of pdf documents and universities, too. Using a database ca-
pable of full text search is an advantage here even if the process is halted at this point.
33
Trends in Digital Education:
Selected papers from EC-TEL 2015 Workshops CHANGEE, WAPLA, and HybridEd
If competencies are linked to the teacher’s knowledge and perspective the con-
sistency is improved at the institutional level because the teachers’ implicit assump-
tions are made transparent to their colleagues. Instead of saying that course A requires
course B, course A now requires a set of competencies which are linked to other
courses giving the student the chance to cherry-pick the courses they like. This reduc-
es the number of students complaining that they are learning certain things twice or
that courses do not match their needs. Another advantage is the possibility to tag edu-
cational contents to make them more accessible in different scenarios. Having a
standard set of activities or lessons ready in order to teach a certain competency may
improve the level of teaching and the redundancy of teachers’ preparation. This kind
of high level semantics cannot be reached with other approaches like machine learn-
ing or rule based systems.
One of the logical consequences of improving transparency is the visibility of con-
flicts in teaching methodology and basic assumptions. This may cause problems. If
two teachers have radically different views on a subject, they will enter conflicting
facts into the competency database which need to be moderated. Since this is an or-
ganizational problem, it can only be solved by stimulating dialogue between the col-
leagues in question. Depending on the flexibility of the organization this poses some
challenges.
The second step is the actual usage of the competency database for assessment.
There are two roles that can be differentiated: the teacher who evaluates the portfolio
and the student producing the digital artifacts. We look at the teacher’s perspective
first. The process is visualized in figure 2 on page 8.
Select competencies: The first step is for the teacher to select a set of competen-
cies relevant for the course. Only the selected competencies are visible in the course
context. Assuming the number of competencies in the repository grows it becomes
necessary to filter them intelligently. Moreover, the teacher has the option to mark
certain competencies mandatory for the students to pass the course. In the example of
an university this done most of the time within the campus management system. Us-
ing the competency database, it simplifies the process of creating module descriptions
as older competency templates can be reused. Also big changes compared to earlier
courses can be visualized as students and pupils might base their decisions to take
classes on experiences of older students.
Define order of acquisition: In a subsequent step the order is defined in which the
competencies are supposed to be acquired and linked. This way the student is present-
ed with a smaller set of competencies he or she can adapt his or her studies to. This
information can be used to structure classes and improve the match of the course level
compared to the students’ knowledge and skill set. This is usually done within the
campus management system or offline. However, current systems, especially in big
organizations, struggle when taking into account cross-disciplinary redundancy in
classes.
Link evidences to activities: This is the default state of the system. Students pro-
duce electronic artifacts during their studies. These are linked to the competencies
with the effect that follow up competencies are presented. This takes place within the
34
Trends in Digital Education:
Selected papers from EC-TEL 2015 Workshops CHANGEE, WAPLA, and HybridEd
e-portfolio system or even learning apps designed to allow the student to reflect his or
her progress. We currently work on apps tailored to this purpose, too.
View progress of users: As soon as an activity is linked to a competency it is
marked as evidence for this particular competency. The sum of the competencies
linked compared to the number of competencies selected earlier shows the overall
progress the student has made. Filtering the competencies will result in partial pro-
gress view. This way students are not graded in comparison to their peers but in view
of the possible degree of competence they can reach in the organization overall.
View evidence links of user: Finally, evidences can be validated, invalidated,
commented on or deleted by the teacher. This is the point where traditional evaluation
methods are needed to qualify the teacher’s decision concerning the evidences.
From a student’s point of view the system behaves very similarly. Students are not
allowed to influence the selection of competencies or their ordering. Students are,
however, allowed to link the competencies themselves or for their fellow students to
make the process more engaging and to lift some of work from the teacher. The
teacher does have the option to counteract problematic behavior by invalidating or
deleting evidence links. Furthermore, students are allowed to view the progress of
their peers and comment on the decisions the teacher has made concerning the valida-
tion.
The process passes through several systems that all deal with the same competen-
cies. This improves the knowledge management within the organization if the as-
sumptions regarding the learning goals are externalized and made accessible at every
state of the process. This addresses existing needs within most bigger learning organi-
zations not mentioning the pedagogical value expressed earlier.
6 Conclusions and further work
We argue that when leaving school a set of formulated competencies may be more
useful in describing a student’s skills then grades. An interesting argument worthy of
a conclusion is that grades are negative (’you only have reached this percentage com-
pared to a perfect student’) whereas competencies much like levels in computer
games are positive (’you have reached this level of competency in these subjects’) and
more constructive (’you are missing these sub-competencies to be as good in pro-
gramming as Student X’). However, if the competencies formulated are meant be
equally useful in comparing students as the numerical scale of grades there is a need
for a standardized assessment and description framework based on competencies.
First complete descriptions are available for selected disciplines.
From a pedagogical point of view we analyzed competency based learning, portfo-
lio learning and merged the concepts. Competence based learning offers the chance to
grade students with a badge like tag system instead of marks. This way transparency
and consistency of the curriculum can be improved. Portfolio learning offers the tools
to use an equally generic assessment scheme to go along with the generic competency
model. This way activities can be linked as evidences for competencies independent
of the subject being taught.
35
Trends in Digital Education:
Selected papers from EC-TEL 2015 Workshops CHANGEE, WAPLA, and HybridEd
The main challenge was the creation of a competency model that conceptually as
well as technically could be used in all subjects and organizations (generality). Instead
of empirically determining categories we look at competencies (or learning goals) as a
plain text formulated concept of what a student should learn. A grammar for formulat-
ing standardized competencies was proposed that supplements the enhanced compe-
tency model developed. The presented competency model includes inheritance and
recursive competencies which makes the competency model more generic then those
currently available. Whereas some of these ideas have been present in technological
standards for some time we implemented a competency database as a reference in
order to make the concept more useful in practice. Here we have achieved what we
think is a generic competency database.
Fig. 2. Activity model of the developed prototype
36
Trends in Digital Education:
Selected papers from EC-TEL 2015 Workshops CHANGEE, WAPLA, and HybridEd
The second challenge was ensuring consistency and re-usability. Here a lot of
technological questions arose concerning the computerlinguistic comparison of plain
text competencies, using reasoning to validate the internal logic of learning graphs
and developing competency based recommender systems. We have made some pro-
gress but it needs a lot of work still.
Next we will facilitate inserting new competency schemes and try to tackle the
pending research questions discussed. Furthermore, we are currently developing
frontends and plugins for popular learning management systems in order to make the
system easier accessible for other organizations. If this is successful, the research in
semantic technologies will lead to a supplementary method of grading academic
achievement on a nominal scale. This way we strive for a meaningful way of grading
students’ progress.
References
1. Adomavicius, G., Tuzhilin, A.: Toward the next generation of recommender systems: a
survey of the state-of-the-art and possible extensions. Knowledge and Data Engineering,
IEEE Transactions on 17(6), 734–749 (June 2005)
2. Brut, M., S`edes, F.: Ontology-Based Solution for Personalized Recommendations in E-
Learning Systems. Methodological Aspects and Evaluation Criterias. In: Advanced Learn-
ing Technologies (ICALT). pp. 469–471. IEEE Computer Society, Venedig (Juli 2010)
3. Cohen, P.: College grades and adult achievement: A research synthesis. Research in High-
er Education 20(3), 281–293 (1984), http://dx.doi.org/10.1007/ BF00983503
4. Hirata, K. et al.: Competency proficiency ontology. In: Hirashima, T., Mohd, A., Kwok,
L.F., Wong, S.L., Kong, S., Yu, F.Y. (eds.) Workshop proceedings of the 18th internation-
al conference on computers in education. pp. 292–299. Faculty of Educational Studies,
Universiti Putra Malaysia, Serdang (December 2010)
5. Lucke, U. et al.: The role of metadata for the automated generation of educational
courseware. In: Roy, W. (ed.) Proceedings of the 2nd European Conference on e-Learning
(ECEL). pp. 93–104. Academic Conferences (October 2003)
6. M. Brut et al.: A rule-based approach for developing a competency-oriented user model for
e-learning systems. In: Fourth International Conference on Internet and Web Applications
and Services. pp. 555–560. IEEE Computer Society, Venedig (May 2009)
7. Moll, M.: A brief history of grading. Teacher Newsmagazine 11(3), 1–2 (November 1998)
8. Noelting, K. et al.: Learner Centrism and Constructivism: New Paradigms for E-Learning?
In: Cantoni, L., McLoughlin, C. (eds.) World conference on Educational Multimedia, Hy-
permedia and Telecommunications. pp. 2434–2441. Association for the Advancement of
Computing in Education (AACE), Chesepeake (Juli 2004)
9. Ostyn, C.: W20 working group of the IEEE Learning Technology Standards Committee
(IEEE Standard P1484201, 2006)
10. Paquette, G.: An ontology and a software framework for competency modeling and man-
agement. Journal of Educational Technology & Society 10(3), 1–21 (April 2007)
11. Richards, T.: IEEE Learning Technology Standards Committee (LTSC) (IEEE Standard
P1484-12:2002, 2002)
12. Sitthisak, O. et al.: Adaptive learning using an integration of competence model with
knowledge space theory. In: Advanced Applied Informatics (IIAIAAI), 2013 IIAI Interna-
tional Conference on. pp. 199–202 (August 2013)
37
Trends in Digital Education:
Selected papers from EC-TEL 2015 Workshops CHANGEE, WAPLA, and HybridEd
13. Wallace, S.: A dictionary of education. Oxford paperback reference, Oxford University
Press, Oxford und New York (2009)
14. Weinert, F.: Concept of competence: A conceptual clarification. In: Rychen, D., Salganik,
L. (eds.) Defining and selecting key competencies, pp. 45–65. Hogrefe & Huber Publish-
ers, Seattle (2001)
15. Wieken, J.H.: Towards a meta-model to utilize the measuring of competence. In: Proceed-
ings of the 29th Annual European Conference on Cognitive Ergonomics. pp. 245–248.
ACM, New York (August 2011)
38