Learning analytics means gathering a broad range of data, bringing the various sources together, and analyzing them. However, to draw educational insights from the results of the analyses, these results must be visualized and presented to the educators and learners. This task is often accomplished by using dashboards equipped with conventional and often simple visualizations such as bar charts or traffic lights. In this paper we want to introduce a method for utilizing the strengths of directed graphs, namely Hasse diagrams, and a competence-oriented approach of structuring knowledge and learning domains. After a brief theoretical introduction, this paper highlights and discusses potential advantages and gives an outlook to recent challenges for research.
Using methods and tools from Learning Analytics (LA) can be considered best
practice and is a key factor for making education more personalized, adaptive, and
effective. Analyzing a variety of available data to uncover learning processes,
strengths and weaknesses, competence gaps undoubtedly is a prerequisite for a
formatively-inspired guidance, for changing and adjusting educational measures and
teaching, and not least for disclosing and negotiating learner models [
However, LA operates in a delicate and complex area. On the one hand, facing today’s classroom realities, we often find technology-lean environments, which do not easily allow or support recording the necessary data. Also, from a socio-pedagogical perspective, learning must be seen as a process of social interaction that not always occurs in front of some electronic. Thus, LA must be based on fewer data. On the other hand, it is rather easy to visualize learning on a superficial level using perhaps the aforementioned traffic lights or bar charts. The added value to the teachers is likely of limited utility to them. To provide a deeper and more formative insight into the learning history and the current state of a learner (beyond the degree to which a teacher might know it intuitively) requires finding and presenting complex data aggregations. This, most often, bears the significant downside that it is hard to understand. Challenges for LA and its visualizations, for example, are to illustrate learning progress (including learning paths) and – beyond the retrospective view – to display the next meaningful learning steps/topics.
In this paper we introduce the method of directed graphs, the so-called Hasse diagrams, for structuring learning domains and for visualizing the progress of a learner through this domain. 2
A Hasse diagram is a strict mathematical representation of a so-called semi-order
in form of a directed graph that reads from bottom to top. A semi-order is a type of
mathematical ordering of a set of items with numerical values by identifying two
items as equal or comparable if the values are within a given interval of error or noise.
Semi-orders were introduced in mathematical psychology by Duncan Luce in 1956
[
The mathematical properties of a semi-order and the Hasse diagrams are (i) reflexivity, (ii) anti-symmetry, and (iii) transitivity. Reflexivity refers to the view that an item, perhaps a competency, references itself in a cause/effect sense. Anti-symmetry demands that if one entity is a prerequisite of another, this relationship is not invertible; as an example, if competency x is a prerequisite to develop competency y, y cannot be the perquisite of competency x. Finally, transitivity means that whenever an element x is related to an element y, and y is in turn related to an element z, then x is also related to z. In principle, the direction of a graph is given by arrows of the edges; by convention however, the representation is simplified by avoiding the arrow heads, whereby the direction reads from bottom to top. In addition, the arrows from one element to itself (reflexivity property), as well as all arrows indicating transitivity are not shown in Hasse diagrams. The following image (Figure 1) illustrates such a diagram. Hasse diagrams enable a complete view to (often huge) structures. Insofar, they appear to be ideal for capturing the large competence or learning spaces occurring in the context of assessment and learning recommendations (for example, all the competencies involved in the math curriculum for a specific age).
In an educational context, a Hasse diagram can display the non-linear path through a learning domain starting from an origin at the beginning of an educational episode (which may be a single school lesson but could also be the entire semester). Moreover, the elements in the diagram may refer to (latent) competencies, to learning objects or test items. Figure 1 illustrates the simple example of typical learning objects in a certain domain. The beginning of a learning episode is usually shown as { } (the empty set) at the bottom of the diagram. Now a learner might attend three learning objects (K, P, H), which is indicated by the edges; this, in essence, establishes three possible learning paths. After H, as an example, this learner might attend K, or H but not T yet, which in turn opens further three branches for the learning path until reaching the final state, within which all learning objects have been attended.
As claimed initially, in the context of formative LA, a competence-oriented
approach is necessary. Thus, a Hasse diagram can be used to identify and
display the latent competencies of a learner in the form of so-called competence states.
An elaborated theoretical approach to do so is Competence-based Knowledge Space
Theory (CbKST). The approach originates from Jean-Paul Doignon and Jean-Claude
Falmagne [
On the basis of these fundamental views, CbKST is looking for the involved entities of aptitude (the competencies) and a natural structure, a natural course of learning in a given domain. For example, it is reasonable to start with the basics (e.g., the competency to add numbers) and increasingly advance in the learning domain (to subtraction, multiplication, division, etc.). As indicated above, this natural course is not necessary linear, which bears significant advantages over other learning and test theories.
As a result we have a set of competencies in a domain and potential relationships between them. In terms of learning, the relationships define the course of learning and thus which competencies are learned before others. In CbKST such relationships are called prerequisite relations or precedence relations. On the basis of competencies and relationships, in a next step, we can obtain a so-called competence space, the ordered set of all meaningful competence states a learner can be in. As an example, a learner might have none of the competencies, or might be able to add and subtract numbers; other states, in turn, are not included in this space, for example it is not reasonable to assume that a learner holds the competency to multiply numbers but not to add them. By the logic of CbKST, each learner is, with certain likelihood, in one of the competence states. 3
As claimed, Hasse diagrams are capable of holding a number of important information for an educator to evaluate the learning progress and also to make recommendations. In this paper we want to highlight such advantages 3.1
As outlined, a competency space is the collection of meaningful states a learner can be in. Depending on the domain, the amount of possible states might be huge. The big advantage, however, is that depending on the degree of structure in the domain, by far not all possible combinations of competencies are reasonable and thus part of the space. When zooming into the diagram, a teacher can exactly identify the set of competencies that is most likely for the learner, by zooming out color-coding can illustrate the most likely locations of a learner within the space. When looking at the entire space, it is obvious at first site at which completion level a learner is approximately (rather at the beginning or almost finished). These zoom levels are shown in Figure 2. Technically, there is a variety of options to achieve the coding, for example, bolding, greying, or color coding, whereas likely states are displayed more distinctly than such with low probability.
Equal to individual states, Hasse diagrams can represent group distributions.
Defined by a certain confidence interval of probabilities those states and areas can be
made more salient that hold the highest percentage of learners of a group. By this
means, specific areas in the competency space become apparent within which the
most learners are and, in contrast also positive or negative outliners pop out the
diagram. A different method was suggested by [
In addition to having insight into groups’ and individuals’ current states of
learning, the learning history, the so-called learning paths, are of interested for educators;
on the one hand for planning future activities, on the other hand, for negotiation and
documenting the achievements of a learning episode (e.g., a semester). Learning paths
can be simply displayed by highlighting the edges between the most likely state(s)
over time. As for the states, various probable paths can be realized by making more
likely paths more intensive (by color coding or line thickness). Figure 3 shows a
simple example. A key strength of presenting learning paths, as indicated, is opening up
the learner model to the learners (perhaps parents) themselves [
Hasse diagram offers information about two very distinct concepts, the inner and outer fringes. The inner fringe indicates what a learner can do / knows at the moment. Mathematically it refers to all sets of competencies, which hold all competencies of the current state but one. This inner fringe is a clear hypothesis of which test/assessment items this learner can master within the margins of a certain probability. Such information may be used to generate effective and individualized tests. The test generation can be complemented with group information. If an educator has very clear information in which competency areas of the space most of the learners are, she can generate or select test item covering exactly those competencies. The big advantage of such approach is the effectiveness of a test for identifying competency states or for ranking the learners can be maximized while the efforts for this evaluation (e.g., the number of test items) can be minimized. And of course the test can be optimized to differentiate different learners and the individual capabilities.
On the other hand, the outer fringes determine which competencies should be addressed in a next educational step. Mathematically is refers to all states which include all the competencies of the current state plus one. These fringes provide a clear set of recommendations about the most effective learning activities for a specific individual or a specific group of learners. Moreover, outer fringes, together with learning paths, allow specifically planning the most effective ways of reaching a specific learning goal (which not necessarily is the final stage of the competence space, the full set, and which is not necessarily the same goal for all individual learners). 3.4
When supporting teachers with information about learning processes, the concept
of costs or learning pace (sometimes referred to as learning trajectories) is of distinct
importance. Cost and pace can be considered as the time or any other measure of
effort it takes to proceed from one competence state to another. In a Hasse diagram
this information can be displayed by varying the length of the edges accordingly. If an
educational leap requires a lot of efforts or time the edges are displayed proportionally
longer than such that happens rather quickly. This method was introduced
initially by [
A further important aspect in the context of LA is aligning the rather fine grained
and low level approach to view competencies on a deeper level of granularity to more
general concepts or rather superordinate notions of achievement. A general concept
can be considered a higher level cluster of competencies; for example, sub-dividing
mathematics into clusters like linear equations, non-linear equations, and vector
arithmetic. Lower level competencies can be linked to one or more of those
‘chapters’. Equally, one might view learning processes in a domain in terms of maturity.
For example, writing skills can be on a low level of maturity, involving certain
competencies and abilities, and on a higher one. Such approach is given, for example, in
the CEFR language skills1. Finally, teaching might involve the achievement of certain
milestones, which should be reached step by step. Hasse diagrams allow identifying
such milestones even if they were unclear or unknown initially. Considering that
milestones as bottlenecks, i.e. unique competence states, each learning must pass,
such bottlenecks immediately pop out in of the diagram. In a formative sense, it is
easy for an educator to located their learners in their approach to or exceeding of such
milestones (cf. Figure 2). A slightly different variant was introduced by [
The features of Hasse diagrams and the arising advantages for LA appear all well and
good. However, the key question is, where do they data for computing the
probabilities of competence states come from. And everything stands or falls with this
question. As for all techniques of LA, it depends on a data rich approach to education, the
1 http://en.wikipedia.org/wiki/Common_European_Framework_of_Reference_for_Languages
2 CMMI refers to the so-called Capability Maturity Model Integration approach which
models development processes (e.g., in production) on different predefined levels [
Fig. 4. Illustrating learning efforts (as costs or pace). The longer the more efforts/time it took to acquire a further competency.
There is little doubt that frameworks, techniques, and tools for LA will increasingly be part of a teacher’s professional life in the near future. The benefits are convincing – using the (partly massive) amount of available data from the students in a smart, automated, and effective way, supported by intelligent systems in order to have all the relevant information available just in time and at first sight. The ultimate goal is to formatively evaluate individual achievements and competencies and provide the learners with the best possible individual support and teaching. Great. The idea of formative assessment and educational data mining is not new but the hype over recent years resulted in scientific sound and robust approaches becoming available, and usable software products appeared. However, when surveying the educational landscape, at least that of the EU, the educational daily routines are different. We face technology-lean classrooms and schools, we face a lack of proper teacher education in using ICT in schools – not mentioning of using techniques of LA in schools. We face a certain aloofness to use breaking educational technologies and a well-founded pedagogical view that learning ideally is analogous and socially embedded and doesn’t occur in front of some kind of electronic device. These are all experiences and results of a large scale European research project named Next-Tell3 that was looking into educationally practices across Europe and that intended to support teachers where exactly they are today with suitable ICT as effective and as appropriately as possible.
The framework of CbKST offers a rigorously competence-based, probabilistic, and multi-source approach that accounts for the latent and holistic abilities of learners and therefore accounts for the recent conceptual change in Europe’s educational systems towards a more competence-oriented education including multi-subject competencies and superordinate 21st century (soft) skills.
No matter if data are rich or lean, a teacher is supported to the best possible degree and with a variety of important information about individual and group-based learning processes and performances and not least about the performance of learners and about
the educator’s own performance. The probabilistic dimension allows teachers to have a more cautious view of individual achievements – it might well be that a learner has a competency but fails in a test; vice versa, a student might luckily guess an answer.
From an application perspective, in the context of European projects we developed and evaluated tools that cover the techniques and approaches described in this paper. In the Next-Tell project, for example, we developed a software tool named ProNIFA, which allowed linking multiple sources of evidence of learning and building CbKSTbased learner models. We piloted various school studies and gathered feedback from teachers. In the end, and this can be considered an outlook for future developments, we had to find out that the ‘massive’ Hasse diagrams are overburdening teachers’ understanding and mental models about individual and class-based learning. Moreover, in order to understand the classical Hasse diagrams, it required (too) massive efforts in training teachers to fully utilize the potentials of those diagrams. Large scale surveys yielded that most educators still prefer simple but information-wise shallow visualizations such as traffic lights or bar charts significantly over more informationrich approaches such as Hasse diagrams or, just to mention another interesting approach, parallel coordinates.
Therefore, recent efforts, e.g., in the LEA’s BOX4 project, seek to adjust and advance the classical Hasse diagrams to such visualizations that are intuitively understood by educators and, at the same time, hold the same density of information. In particular, focus of research is on an advancement of Hasse diagrams towards specific mental models teachers may hold, such as a starry night sky or organic, biological structures such as cells of a living being. Also, abstraction and simplification techniques are investigated, e.g., fisheye lenses or streamgraphs. An impression of the learning analytics portal for teachers is given in Figure 6. The upper image shows a menu screen where teachers have access to their personal tools and widgets, e.g., external learning apps. The lower image shows the first release of the Hasse diagram visualization, including color coding of competencies’ probabilities according the analyses.
In conclusion, the utility of CbKST-ish approaches to LA, involving a separation of latent competencies and observable behaviors/performance, as well as having a conservative, probabilistic, multi-source approach appears to be a striking classroomoriented, next-level contribution to LA, learner modelling, and model negotiations.
This work is based on the finalized project Next-Tell, which was supported by the European Commission (EC) under the Information Society Technology priority of the 7th Framework Programme for research and development as well as the running LEA’s BOX project, contracted under number 619762, of the 7th Framework Programme. This document does not represent the opinion of the EC and the EC is not responsible for any use that might be made of its content.