Learning analytics means gathering a broad range of data, bringing the various sources together, and analyzing them. However, to draw educational insights from the results of the analyses, these results must be visualized and presented to the educators and learners. This task is often accomplished by using dashboards equipped with conventional and often simple visualizations such as bar charts or traffic lights. In this paper we want to introduce a method for utilizing the strengths of directed graphs, namely Hasse diagrams, and a competence-oriented approach of structuring knowledge and learning domains. After a brief theoretical introduction, this paper highlights and discusses potential advantages and gives an outlook to recent challenges for research.
Using methods and tools from Learning Analytics (LA) can be
considered best practice and is a key factor for making education
more personalized, adaptive, and effective. Analyzing a variety of
available data to uncover learning processes, strengths and
weaknesses, competence gaps undoubtedly is a prerequisite for a
formatively-inspired guidance, for changing and adjusting
educational measures and teaching, and not least for disclosing
and negotiating learner models [
However, LA operates in a delicate and complex area. On the one hand, facing today’s classroom realities, we often find technology-lean environments, which do not easily allow or support recording the necessary data. Also, from a sociopedagogical perspective, learning must be seen as a process of social interaction that not always occurs in front of some electronic. Thus, LA must be based on fewer data. On the other hand, it is rather easy to visualize learning on a superficial level using perhaps the aforementioned traffic lights or bar charts. The added value to the teachers is likely of limited utility to them. To provide a deeper and more formative insight into the learning history and the current state of a learner (beyond the degree to which a teacher might know it intuitively) requires finding and presenting complex data aggregations. This, most often, bears the significant downside that it is hard to understand. Challenges for LA and its visualizations, for example, are to illustrate learning progress (including learning paths) and - beyond the retrospective view - to display the next meaningful learning steps/topics. In this paper we introduce the method of directed graphs, the socalled Hasse diagrams, for structuring learning domains and for visualizing the progress of a learner through this domain.
A Hasse diagram is a strict mathematical representation of a
socalled semi-order in form of a directed graph that reads from
bottom to top. A semi-order is a type of mathematical ordering of
a set of items with numerical values by identifying two items as
equal or comparable if the values are within a given interval of
error or noise. Semi-orders were introduced in mathematical
psychology by Duncan Luce in 1956 [
In an educational context, a Hasse diagram can display the nonlinear path through a learning domain starting from an origin at the beginning of an educational episode (which may be a single school lesson but could also be the entire semester). Moreover, the elements in the diagram may refer to (latent) competencies, to learning objects or test items. Figure 1 illustrates the simple example of typical learning objects in a certain domain. The beginning of a learning episode is usually shown as { } (the empty set) at the bottom of the diagram. Now a learner might attend three learning objects (K, P, H), which is indicated by the edges; this, in essence, establishes three possible learning paths. After H, as an example, this learner might attend K, or H but not T yet, which in turn opens further three branches for the learning path until reaching the final state, within which all learning objects have been attended.
As claimed initially, in the context of formative LA, a
competence-oriented approach is necessary. Thus, a Hasse
diagram can be used to identify and display the latent
competencies of a learner in the form of so-called competence
states. An elaborated theoretical approach to do so is
Competence-based Knowledge Space Theory (CbKST). The
approach originates from Jean-Paul Doignon and Jean-Claude
Falmagne [
We interpret the performance of a learner (e.g., mastering an addition task) in terms of holding or not holding the respective competency. In addition, recent developments of the approach are based on a probabilistic view of having or lacking certain competencies. In our example, mastering one specific addition task allows the conclusion that the person is able to add two numbers (to hold this competency) only to a certain degree or probability. When thinking of a multiple-choice item with two alternatives, as another example, mastering this item allows only to 50 percent that the person has the required competencies/ knowledge.
On the basis of these fundamental views, CbKST is looking for the involved entities of aptitude (the competencies) and a natural structure, a natural course of learning in a given domain. For example, it is reasonable to start with the basics (e.g., the competency to add numbers) and increasingly advance in the learning domain (to subtraction, multiplication, division, etc.). As indicated above, this natural course is not necessary linear, which bears significant advantages over other learning and test theories. As a result we have a set of competencies in a domain and potential relationships between them. In terms of learning, the relationships define the course of learning and thus which competencies are learned before others. In CbKST such relationships are called prerequisite relations or precedence relations. On the basis of competencies and relationships, in a next step, we can obtain a so-called competence space, the ordered set of all meaningful competence states a learner can be in. As an example, a learner might have none of the competencies, or might be able to add and subtract numbers; other states, in turn, are not included in this space, for example it is not reasonable to assume that a learner holds the competency to multiply numbers but not to add them. By the logic of CbKST, each learner is, with certain likelihood, in one of the competence states.
As claimed, Hasse diagrams are capable of holding a number of important information for an educator to evaluate the learning progress and also to make recommendations. In this paper we want to highlight such advantages.
As outlined, a competency space is the collection of meaningful states a learner can be in. Depending on the domain, the amount of possible states might be huge. The big advantage, however, is that depending on the degree of structure in the domain, by far not all possible combinations of competencies are reasonable and thus part of the space. When zooming into the diagram, a teacher can exactly identify the set of competencies that is most likely for the learner, by zooming out color-coding can illustrate the most likely locations of a learner within the space. When looking at the entire space, it is obvious at first site at which completion level a learner is approximately (rather at the beginning or almost finished). These zoom levels are shown in Figure 2. Technically, there is a variety of options to achieve the coding, for example, bolding, greying, or color coding, whereas likely states are displayed more distinctly than such with low probability.
Equal to individual states, Hasse diagrams can represent group
distributions. Defined by a certain confidence interval of
probabilities those states and areas can be made more salient that
hold the highest percentage of learners of a group. By this means,
specific areas in the competency space become apparent within
which the most learners are and, in contrast also positive or
negative outliners pop out the diagram. A different method was
suggested by [
In addition to having insight into groups’ and individuals’ current
states of learning, the learning history, the so-called learning
paths, are of interested for educators; on the one hand for
planning future activities, on the other hand, for negotiation and
documenting the achievements of a learning episode (e.g., a
semester). Learning paths can be simply displayed by highlighting
the edges between the most likely state(s) over time. As for the
states, various probable paths can be realized by making more
likely paths more intensive (by color coding or line thickness).
Figure 3 shows a simple example. A key strength of presenting
learning paths, as indicated, is opening up the learner model to the
learners (perhaps parents) themselves [
Hasse diagram offers information about two very distinct concepts, the inner and outer fringes. The inner fringe indicates what a learner can do / knows at the moment. Mathematically it refers to all sets of competencies, which hold all competencies of the current state but one. This inner fringe is a clear hypothesis of which test/assessment items this learner can master within the margins of a certain probability. Such information may be used to generate effective and individualized tests. The test generation can be complemented with group information. If an educator has very clear information in which competency areas of the space most of the learners are, she can generate or select test item covering exactly those competencies. The big advantage of such approach is the effectiveness of a test for identifying competency states or for ranking the learners can be maximized while the efforts for this evaluation (e.g., the number of test items) can be minimized. And of course the test can be optimized to differentiate different learners and the individual capabilities.
On the other hand, the outer fringes determine which competencies should be addressed in a next educational step. Mathematically is refers to all states which include all the competencies of the current state plus one. These fringes provide a clear set of recommendations about the most effective learning activities for a specific individual or a specific group of learners. Moreover, outer fringes, together with learning paths, allow specifically planning the most effective ways of reaching a specific learning goal (which not necessarily is the final stage of the competence space, the full set, and which is not necessarily the same goal for all individual learners).
When supporting teachers with information about learning
processes, the concept of costs or learning pace (sometimes
referred to as learning trajectories) is of distinct importance. Cost
and pace can be considered as the time or any other measure of
effort it takes to proceed from one competence state to another. In
a Hasse diagram this information can be displayed by varying the
length of the edges accordingly. If an educational leap requires a
lot of efforts or time the edges are displayed proportionally longer
than such that happens rather quickly. This method was
introduced initially by [
A further important aspect in the context of LA is aligning the
rather fine grained and low level approach to view competencies
on a deeper level of granularity to more general concepts or rather
superordinate notions of achievement. A general concept can be
considered a higher level cluster of competencies; for example,
sub-dividing mathematics into clusters like linear equations,
nonlinear equations, and vector arithmetic. Lower level competencies
can be linked to one or more of those ‘chapters’. Equally, one
might view learning processes in a domain in terms of maturity.
For example, writing skills can be on a low level of maturity,
involving certain competencies and abilities, and on a higher one.
Such approach is given, for example, in the CEFR language skills
(cf. http://en.wikipedia.org/wiki/Common_European_Framework
_of_Reference_for_Languages). Finally, teaching might involve
the achievement of certain milestones, which should be reached
step by step. Hasse diagrams allow identifying such milestones
even if they were unclear or unknown initially. Considering that
milestones as bottlenecks, i.e. unique competence states, each
learning must pass, such bottlenecks immediately pop out in of
the diagram. In a formative sense, it is easy for an educator to
located their learners in their approach to or exceeding of such
milestones (cf. Figure 2). A slightly different variant was
introduced by [
The features of Hasse diagrams and the arising advantages for LA
appear all well and good. However, the key question is, where do
they data for computing the probabilities of competence states
come from. And everything stands or falls with this question. As
for all techniques of LA, it depends on a data rich approach to
education, the more and the better data exist, the better is the
quality of LA conclusions. CbKST and Hasse diagrams are no
exception to that. However, the approach of separating latent
competencies, which more or less develop and exist in the black
box ‘human brain’, and the performance they determine, bears
particular advantages. On the one hand, performance, e.g. test
scores, classroom participation, homework, etc., is not only
determined by competencies or aptitude; there is a variety of
aspects contributing to a certain performance, e.g., motivation,
daily constitution, tiredness, external distractors, nutrition, health
status, etc. On the other hand, CbKST-ish competence spaces are
rather stable, once set up and validated properly. The advantage
lays in the fact that performance such as test results, behaviors,
achievements, etc. is considered as probability-based indicators
for certain competencies. Mathematically this relationship is
established in form of interpretation and representation functions
[
There is little doubt that frameworks, techniques, and tools for LA will increasingly be part of a teacher’s professional life in the near future. The benefits are convincing – using the (partly massive) amount of available data from the students in a smart, automated, and effective way, supported by intelligent systems in order to have all the relevant information available just in time and at first sight. The ultimate goal is to formatively evaluate individual achievements and competencies and provide the learners with the best possible individual support and teaching. Great. The idea of formative assessment and educational data mining is not new but the hype over recent years resulted in scientific sound and robust approaches becoming available, and usable software products appeared. However, when surveying the educational landscape, at least that of the EU, the educational daily routines are different. We face technology-lean classrooms and schools, we face a lack of proper teacher education in using ICT in schools – not mentioning of using techniques of LA in schools. We face a certain aloofness to use breaking educational technologies and a well-founded pedagogical view that learning ideally is analogous and socially embedded and doesn’t occur in front of some kind of electronic device. These are all experiences and results of a large scale European research project named Next-Tell (www.nexttell.eu) that was looking into educationally practices across Europe and that intended to support teachers where exactly they are today with suitable ICT as effective and as appropriately as possible.
The framework of CbKST offers a rigorously competence-based, probabilistic, and multi-source approach that accounts for the latent and holistic abilities of learners and therefore accounts for the recent conceptual change in Europe’s educational systems towards a more competence-oriented education including multisubject competencies and superordinate 21st century (soft) skills. No matter if data are rich or lean, a teacher is supported to the best possible degree and with a variety of important information about individual and group-based learning processes and performances and not least about the performance of learners and about the educator’s own performance. The probabilistic dimension allows teachers to have a more cautious view of individual achievements – it might well be that a learner has a competency but fails in a test; vice versa, a student might luckily guess an answer.
From an application perspective, in the context of European projects we developed and evaluated tools that cover the techniques and approaches described in this paper. In the NextTell project, for example, we developed a software tool named ProNIFA, which allowed linking multiple sources of evidence of learning and building CbKST-based learner models. We piloted various school studies and gathered feedback from teachers. In the end, and this can be considered an outlook for future developments, we had to find out that the ‘massive’ Hasse diagrams are overburdening teachers’ understanding and mental models about individual and class-based learning. Moreover, in order to understand the classical Hasse diagrams, it required (too) massive efforts in training teachers to fully utilize the potentials of those diagrams. Large scale surveys yielded that most educators still prefer simple but information-wise shallow visualizations such as traffic lights or bar charts significantly over more information-rich approaches such as Hasse diagrams or, just to mention another interesting approach, parallel coordinates . Therefore, recent efforts, e.g., in the LEA’s BOX (www.leasbox.eu) project, seek to adjust and advance the classical Hasse diagrams to such visualizations that are intuitively understood by educators and, at the same time, hold the same density of information. In particular, focus of research is on an advancement of Hasse diagrams towards specific mental models teachers may hold, such as a starry night sky or organic, biological structures such as cells of a living being. Also, abstraction and simplification techniques are investigated, e.g., fisheye lenses or streamgraphs. In conclusion, the utility of CbKST-ish approaches to LA, involving a separation of latent competencies and observable behaviors/performance, as well as having a conservative, probabilistic, multi-source approach appears to be a striking classroom-oriented, next-level contribution to LA, learner modelling, and model negotiations.
This work is based on the finalized project Next-Tell, which was supported by the European Commission (EC) under the Information Society Technology priority of the 7th Framework Programme for research and development as well as the running LEA’s BOX project, contracted under number 619762, of the 7th Framework Programme. This document does not represent the opinion of the EC and the EC is not responsible for any use that might be made of its content.