=Paper=
{{Paper
|id=Vol-1738/IWTA_2016_paper7
|storemode=property
|title=Support Teachers' Predictions of Learning Success by Structural Competence Modelling
|pdfUrl=https://ceur-ws.org/Vol-1738/IWTA_2016_paper7.pdf
|volume=Vol-1738
|authors=Michael D. Kickmeier-Rust,Dietrich Albert
|dblpUrl=https://dblp.org/rec/conf/ectel/Kickmeier-RustA16
}}
==Support Teachers' Predictions of Learning Success by Structural Competence Modelling==
https://ceur-ws.org/Vol-1738/IWTA_2016_paper7.pdf
Support Teachers’ Predictions of Learning Success
by Structural Competence Modelling
Michael D. Kickmeier-Rust Dietrich Albert
Graz University of Technology Graz University of Technology
Plüddemanngasse 104 Plüddemanngasse 104
8010 Graz, Austria 8010 Graz, Austria
+43 316-873-30636 +43 316-873-30630
michael.kickmeier-rust@tugraz.at dietrich.albert@tugraz.at
ABSTRACT learner in a holistic and fair manner, and provide appropriate
feedback, teachers need to be equipped with the appropriate
Learning Analytics is one of the most promising major trends in technology. With that regard, a substantial body of research work
educational technology. However, Learning Analytics is very and tools already exist.
often a rather statistical approach to the understanding of
educationally relevant data. Theory-driven approaches are much LEA’s BOX (www.leas-box.eu) is a project, funded under
sparser. In the context of the European Lea’s Box project the EU’s Seventh Framework Programme and stands for a
(www.leas-box.eu), we aim at developing methods for analysing practical LEarning Analytics tool Box, that provides
data coming from multiple sources on the basis of psychological a competence-centred, multi-source formative
theories from the area of Intelligent Tutorial Systems, namely assessment methodology,
Competence-based Knowledge Space Theory (CbKST) and
Formal Concept Analysis (FCA). These well-elaborated based on sound psycho-pedagogical models (i.e.,
approaches allow us to identify competencies on an atomic level, Competence-based Knowledge Space Theory and
to establish structural, multi-dimensional knowledge spaces, and Formal Concept Analysis),
to identify individual learning paths and knowledge gaps. In this
intelligent model-based reasoning services,
paper we introduce an approach to utilize the mentioned theories
to predict learning paths, the Learning Performance Vector, and innovative visualization techniques,
individual limits, the so-called individual learning Horizon.
and features to open and negotiate learner models;
Keywords
LEA’s BOX is dedicated to develop a learning analytics
Learning Analytics, Competence-based Knowledge Space Theory,
toolbox that is intended to enable educators to perform
Formal Concept Analysis, Learning Performance Vector,
competence-centered, multi-source learning analytics, considering
Learning Horizon
their real practical needs. Thus, the project spends significant
1. INTRODUCTION efforts on a close and intensive interaction with educators in form
Using Learning analytics and educational data mining are more of design focus groups and piloting studies.
than recent buzz words in educational research: they signify one The tangible result of LEA’s BOX manifest in form of a
of the most promising developments in improving teaching and Web platform for teachers and learners that provides links to the
learning. While many attempts to enhance learning with mere existing components and interfaces to a broad range of
technology failed in the past, making sense of a large amount of educational data sources. Teachers will be able to link the various
data collected over a long period of time and conveying it to tools and methods that they are already using in their daily
teachers in a suitable form is indeed the area where computers and practice and that provide software APIs (e.g., Moodle courses,
technology can add value for future classrooms. However, electronic tests, Google Docs, etc.) in one central location. More
reasoning about data, and in particular learning-related data, is not importantly, the platform hosts the newly developed LA/EDM
trivial and requires a robust foundation of well-elaborated psycho- services, empowering educators to conduct competence-based
pedagogical theories. analysis of rich data sets. A key focus of the platform will enable
The fundamental idea of learning analytics is not new. In teachers not only to combine existing bits of data but to allow
essence, the aim is using as much information about learners as them to “generate” and collect data in very simple forms, not
possible to understand the meaning of the data in terms of the requiring sophisticated hard- or software solutions. Finally, we
learners’ strengths, abilities, knowledge, weakness, learning want to open new ways to display the results of learning analytics
progress, attitudes, and social networks with the final goal of - leaving the rather statistical dashboard approach, moving
providing the best and most appropriate personalized support. towards structural visualizations and towards opening the internal
Thus, the concept of learning analytics is quite similar to the idea learner models.
of formative assessment. “Good” teachers of all time have strived 2. THE LEARNING HORIZON
to achieve exactly this goal. However, collecting, storing,
In the centre of conceptual research in the field of CbKST and
interpreting, and aggregating information about learners that
FCA was the so called Learning Performance Vector (LPV) and
originates from a school year, or even in a lifelong learning sense)
requires smart technology. To analyse this vast amount of data, the Learning Horizon. The principle idea of this constructs is to
use CbKST and FCA as means of predictive analytics. The
give it educational meaning, visualize the results, represent the
fundamental idea, thereby, is to consider the past learning
performance in terms of CbKST-like learning paths , the current
progress of an individual learner as well as a summary of peer
performance (if available) and to match learning time and
remaining time with the learning goals. In such a way we aim at
deriving estimations of an individual’s learning success and the
degree to which a desired learning goal can be achieved. The
foundations of this approach are not only competence structures
and formal concepts (e.g., competencies over learners) but also
temporal information, weighting information of activities and
achievements, and difficulty aspects of future learning tasks. In
the end, we try to establish an algorithm that is capable of melding
those information into robust predictions of learning success – in
other terms of the likelihood that a particular student can reach the
learning goals in a given amount of time – the Learning Horizon.
Of course, the predictions are unstable and blurred in the
beginning and certainly the predications are more valid, the more
time has passed and the more information the system has. Still,
the approach is capable, so we hope, to give early indications of
performance problems, so that it is still possible for educators to
intervene appropriately. In addition, a particular strength is that
the CbKST/FAC approach allows for finding concrete directions Figure 1. Illustration of a Competence structure; to
where a learner needs support and guidance. bottom most node indicate the empty set of competencies, the
lines are the possible learning paths, and the top most node
3. ELEMENTS OF THE LEARNING indicates the possession of all the competencies of a domain.
HORIZON AND THE LPV
3.1 Competence Structures and Performance 3.2 Formal Contexts
FCA, the analysis of formal context, is a related formal
The first element we consider is clearly a competence structure
psychological approach. The idea is to identify patterns in a
(Figure 1). Very briefly, we decompose a learning domain (e.g.,
universe of two dimensions. Imagine there is a set of
2nd grade maths) into atomic chunks of knowledge or aptitude. In
competencies and a set of students. There is a multitude of
a second step we try to find a natural course of learning or, in
clusters, some students hold the one some the other competencies.
other terms, we try to find the prerequisite structure: which
FCA allows to quickly analyse the patterns and identify relevant
elements need to be learned before another piece can be acquired.
clusters, even more, hierarchies. If FCA is applied on the
This gives us a combinatorics model of a learning domain and a
competency models of CbKST, we have the opportunity to meld
certain understanding of how learning and development occurs.
pedagogically inspired domain models with pattern identification
Now, it must be highlighted that competencies and learning,
mechanisms. By this means we can identify clusters of good and
abilities and aptitudes are latent constructs. One cannot directly
not so good learners, we can establish a hierarchy of performance,
observe the real “knowledge” of another person. It takes
and, at each step, we can determine which competencies are
indicators and evidences, in its simplest form a school test. We
lacking, and therefore which educational measures would be
know, very well, that tests are not necessarily objective. Students
necessary. In general, there is a broad variety of educationally
can be inattentive and fail although they have the knowledge or
relevant questions that can be addressed using the paired CbKST /
competence, some may guess the right answer incidentally. So in
FCA approach (cf. Bedek, Kickmeier-Rust & Albert, 2015).
the end, there is a good portion of uncertainty in assessment.
When talking about the underlying competencies, we need to 3.3 Likelihoods, Weights, and their
account for this fact. And we need to account for that in a careful
and conservative way. The CbKST approach does that by Extensions
establishing stochastic relationships. Each indicator, each piece of In recent works we demonstrated that the traditional approaches
evidence, each test result is only one indicator that contributes to of using Hasse diagrams for visualizing competence structures
the whole picture, but it contributes only with a certain and lattice graphs for displaying formal contexts can be extended
probability. The more evidence we can aggregate, mirroring the in meaningful ways. One idea suggested by (Kickmeier-Rust &
same competencies and competence structures, the clearer and Albert, 2015) was to extend Hasse diagram visualizations by
more robust our picture (our model of the learner) gets. Of course, adding a difficulty (a weight) dimension to the diagram by
we have to consider that different evidences have different illustrating the length of edges in correspondence to their weight
weights, a different impact, on the learner model. A simple (difficulty). There are two important aspects to this idea. On the
multiple choice test weighs less than an oral exam within which a one hand, it introduces weights as levels of difficulties and the
teacher can explore the real knowledge of a student, exhibiting necessary efforts to make the step from one to another
abilities in real live weighs more than filling in the right answers. competence state, on the other hand, it provides valuable
information to inspire the LPV and the estimation of a Learning
Horizon. In addition to that, a simple yet important fact is that
subject matter is increasing in difficulty over time. This definitely
must be another variable in our model of learning.
individual learning path). In a next step, given the concrete
competence state of the learner, we have to identify the possible
3.4 What Peers are Doing paths towards to defined learning goal, which is a (rather small)
Now, when it’s about to estimate a student’s potential progress subset of all possible paths. Equally to the computation of the
and chances to accomplish a course on time, e central element is a difficulty to reach the current state, we can compute the potential
comparison to other learners. [It shall be highlighted that this is difficulty of all possible paths to the goal, whereas we have to
optional, since the LPV can be computed without peer compute the average difficulty of all possible paths. This now is
information!] If a particular student appears being clearly ahead of an indicator for the efforts that are necessary for an individual
the majority or, in a worse case, behind the majority, a teacher can learner to reach the learning goal.
receive corresponding and actionable information from analytics.
When link the progress of a student within a given span of
Here also a meta-perspective comes into play, namely the time, we can make a prediction about how far a student can come
degree to which a teacher is capable of setting the right learning within the remaining time (of a course, for example). So, as a final
goals for a particular group of students and the ability to reach the step, we can identify exactly those states (and therefore the
goals. This is a non-trivial aspect to Learning Analytics tools. competencies) a particular will be able to reach within the time
Oftentimes, a teacher is seen as the ultimate key luminary in a limits. The set of those states is, now finally, the student’s
certain domain. This, however, is not necessarily true. Teacher Learning Horizon.
may completely misjudge the abilities and potentials of a group of
students (and there is a variety of reasons why this may happen).
So, a dimension of a group comparison can add substantial
information about individual progress as well as a teacher’s plans.
In the end, this analysis offers a fountain of deeper insights.
Finally, it’s worth mentioning that a theoretically sound peer
comparison offers the option for a motivation boost of individual
efforts, almost like the principle of badging or gamification.
Position and achievements in peer groups have tremendous
motivational powers, however, the must be utilized very carefully
and thoughtfully!
4. PREDICATION ALGORITHM
So what do we have: A competence structure (or competence
space). This structure gives us a model of the learning domain,
starting from point 0 (in this particular domain) leading to the
Figure 2. Conceptual illustration of the approach.
complete mastery. In other terms, a competence structures is the
manifestation of all possible and reasonable states a person can be
in. This allows us to identify the progress of a particular learner
given the timeline of a course. Mathematically speaking we have
the sum of all possible learning paths. This indicates the average
learning efforts, given that transitions have specific difficulties or
5. CURRENT STATUS AND OUTLOOK
weights (cf. Figure 2). The LPV and LH approach appear being an interesting method for
educationally relevant predictions that compliments the existing
We have a set of competencies Q = {a, b, c, …. } with a rather statistical methods. While these methods usually make
predictions on the basis of a comparison of an individual learner
competence structure. The sum of the resulting competence states with a possibly large set of other students and their achievements,
tence state to the introduced approach is primarily based on information about
another has a difficulty parameter, which in turn is the average of the learning domain, the competencies, their characteristics, and
the difficulty parameters of the competencies being a part of the their relationships. The advantages are, on the one hand, that the
state, we have a set of tuples of the start competence state, the end noise of statistical comparisons is reduced; on the other hand,
, w]. This results in a set of analyses and predictions can be made without referring to a large
basis of existing student data. The latter point is of particular
we have a set of indicators providing evidences for competencies: interest when focussing on school education: usually schooling is
I = { ei, {c} * w}, with a given weight w. Based on the evidences a diverse analogues setting where not much data is generated
we can estimate the likelihood of each competency. The where data that is available are not aggregated and where the
probability of a competence state is the average of its nature of data is extremely diverse (Kickmeier-Rust, Bull, &
Albert, 2016).
To identify the learning path of a person, we identify the The introduced approach is implemented in the Lea’s
state with the highest probability in certain time steps. Depending Box Learning Analytics Toolbox (www.leas-box.eu). As
on the nature of the concrete use case this may rely on the events emphasized in the introductory section, this online platform is
when evidences are put into the system or, alternatively on a tailored to the concrete demands of teachers and provides a set of
timely basis (e.g., weekly or monthly). internal Learning Analytics tools and, which is a key focus of the
project, APIs to link a large number of external tools (such as
Now for each step we compute the difficulty (as a value from
learning apps, e-learning systems, cloud tools) to the system. The
0 to 1). The sum of the values gives us an indicator for how many vision is to allow an easy aggregation of all the data that are
efforts a student has to spend on her learning history (the available, even if there is not much data and not coherent data,
and make the most of it in terms of a formative evaluation and 7. References
feedback and a more evidence based individualisation of teaching. [1] Bedek, M. A., Kickmeier-Rust, M. D., & Albert, D. (2015).
Presently we are evaluating the validity of the approach Formal concept analysis for modelling students in a
on the basis of large data sets from professional learning solutions technology-enhanced learning setting. In M. Kravcik, A.
in Turkey and the US. In Turkey we have access to the Vitamin Mikroyannidis, V. Pammer, M. Prilla, T. D. Ullman (Eds.),
learning platform (https://www.vitaminegitim.com/vittrin/) which Proceedings of the 5th Workshop on Awareness and
offers a broad offer of courses and tests. In addition we include Reflection in Technology Enhanced Learning at EC-TEL
data from the US product Adaptive Curriculum 2015, Toledo, Spain, September 15, 2015.
(https://www.adaptivecurriculum.com/us/), which offers courses [2] Falmagne, J.-C., Albert D., Doble, D., Eppstein D., Hu X.
for middle and high school levels. The first experiences are quite (2013). Knowledge Spaces: Applications in Education.
promising; the predications of the system yield a substantial fit to Berlin: Springer.
the patterns we find in the large data sets. We will investigate the
[3] Kickmeier-Rust, M. D., & Albert, D. (Eds.) (2012). An
predictive power further and will specifically address the question
Alien's guide to multi-adaptive educational games. Santa
whether the analyses and predictions are also valid for the data
Rosa, CA: Informing Science Press.
lean school scenarios in comparison to the data rich evolution
scenarios. The recent developments as well as the continuous [4] Kickmeier-Rust, M. D., Bull, S., & Albert, D. (2016). LEA’s
study results are frequently posted on the Lea’s Box website BOX: Practical Competence-oriented Learning Analytics
(www.leas-box.eu) as well on Lea’s Facebook account and Open Learner Modeling. In Proceedings of the
(www.facebook.com/LeasLearning). workshop Learning Analytics for Learners at LAK'16
conference, 26-29 April, 2016, Edinburgh, UK.
[5] Kickmeier-Rust, M. D., Steiner, C. M., & Albert, A. (2015).
6. Acknowledgements Uncovering Learning Processes Using Competence-based
This work is based on the LEA’s BOX project, contracted under Knowledge Structuring and Hasse Diagrams. In Proceedings
number 619762, which is supported by the European Commission of LAK15, Workshop Visual Approaches to Learning
(EC) under the Information Society Technology priority of the 7th Analytics. March 16-20, 2015, Poughkeepsie, NY.
Framework Programme for research and development. This [6] Reimann, P., Bull, S., Kickmeier-Rust, M. D., Vatrapu, R.,
document does not represent the opinion of the EC and the EC is Wasson, B. (Eds.) (2015). Measuring and Visualizing
not responsible for any use that might be made of its content. Learning in the Information-Rich Classroom. New York, NY:
Routledge.