More and more learning activities take place online in a selfdirected manner. Therefore, just as the idea of self-tracking activities for tness purposes has gained momentum in the past few years, tools and methods for awareness and self-re ection on one's own online learning behavior appear as an emerging need for both formal and informal learners. Addressing this need is one of the key objectives of the AFEL (Analytics for Everyday Learning) project. In this paper, we discuss the di erent aspects of what needs to be put in place in order to enable awareness and self-re ection in online learning. We start by describing a scenario that guides the work done. We then investigate the theoretical, technical and support aspects that are required to enable this scenario, as well as the current state of the research in each aspect within the AFEL project. We conclude with a discussion of the ongoing plans from the project to develop learner-facing tools that enable awareness and selfre ection for online, self-directed learners. We also elucidate the need to establish further research programs on facets of self-tracking for learning that are necessarily going to emerge in the near future, especially regarding privacy and ethics.
a university program) and where online activities happen through a provided eLearning system. However, whether or not they are formally engaged in learning, more and more learners are now using a large variety of online platforms and resources which are not necessarily connected with their learning environment or with each other. Such use of online resources tends to be self-directed in the sense that learners make their own choices as to which resource to employ and which activity to realize amongst the wide choice o ered to them (MOOCs, tutorials, open educational resources, etc). With such practices becoming more common, there is therefore value in researching the way in which to support such choices.
In several other areas than learning where self-directed activities are
prominent (e.g. tness), there has been a trend in recent years following the
technological development of tools for self-tracking [
AFEL7 (Analytics for Everyday Learning) is an European Horizon 2020
project which aim is to address both the theoretical and technological
challenges arising from applying learning analytics [
In this paper, we discuss the research and development challenges associated with achieving those goals and describe initial results obtained by the project in three key areas: theory (through cognitive models of learning), technology (through data capture, processing and enrichment systems) and support (through the features provided to users for visualizing, exploring and drawing conclusions from their learning activities). We start by describing a motivating scenario of an online, self-directed learner to clarify our objective. 2
Below is a speci c scenario considering a learner not formally engaged in a speci c study program, but who is, in a self-directed and explicit way, engaged in online learning. The objective is to describe in a simple way how the envisioned AFEL tools could be used for self-awareness and re ection, but also to explore what the expected bene ts of enabling this for users/learners are:
Jane is 37 and works as an administrative assistant in a local mediumsized company. As hobbies, she enjoys sewing and cycling in the local forests. She is also interested in business management, and is considering either developing in her current job to a more senior level or making a career change. Jane spends a lot of time online at home and at her job. She has friends on Facebook with whom she shares and discusses local places to go cycling, and others with whom she discusses sewing techniques and possible projects, often through sharing YouTube videos. Jane also follows MOOCs and forums related to business management, on di erent topics. She often uses online resources such as Wikipedia and online magazines. At school, she was not very interested in maths, which is needed if she wants to progress in her job. She is therefore registered on Didactalia8, connecting to resources and communities on maths, especially statistics.
Jane has decided to take her learning seriously: She has registered to use the AFEL dashboard through the Didactalia interface. She has also installed the AFEL browser extension to include her browsing history, as well as the Facebook app. She has not included in her dashboard her emails, as they are mostly related to her current job, or Twitter, since she rarely uses it.
Jane looks at the dashboard more or less once a day, as she is prompted by a noti cation from the AFEL smart phone application or from the Facebook app, to see how she has been doing the previous day in her online social learning. It might for example say \It looks like you progressed well with sewing yesterday! See how you are doing on other topics..." Jane, as she looks at the dashboard, realizes that she has been focusing a lot on her hobbies and procrastinated on the topics she enjoys less, especially statistics. Looking speci cally at statistics, she realizes that she almost only works on it on Friday evenings, because she feels guilty of not having done much during the week. She also sees that she is not putting as much e ort into her learning of statistics as other learners, and not making as much progress. She therefore makes a conscious decision to put more focus on it. She adds new goals on the dashboard of the form \Work on statistics during my lunch break every week day" or \Have achieved a 10% progress compared to now by the same time next week". The dashboard will remind her of how she is doing against those goals as she goes about her usual online social learning activities. She also gets recommendations of things to do on Didactalia and Facebook based on the indicators shown on the dashboard and her stated goals.
While this is obviously a ctitious scenario, which is very much simpli ed, it shows the way tools for awareness and self-re ection can support online selfdirected learning, and it provides a basis to investigate the challenges to address in order to enable the development of tools of the kind that are described, as discussed in the rest of this paper.
One result of the advent of the Internet as a mass phenomenon was a slight change in our understanding of constructs such as \knowledge" and \learning". In such contexts as described above, it is by no means a trivial task to identify and to assess learning. Indeed, in order to understand how learning emerges from a collection of disparate online activities, we need to get back to fundamental, cognitive models of learning, as we cannot make the assumption that usual ways to test the results of learning are available.
Traditionally, the acquisition metaphor was frequently used to describe
learning processes [
When talking about learning in digital environments, we can consequently
de ne learning as the activity of learners encountering at least partly new
information in form of digital artifacts. In principle, every single interaction between
a learner and an artifact can entail learning processes. Learning can either
happen occasionally and accidentally or in the course of planned and at least partly
structured learning activities [
Based on these theoretical considerations, the challenge in building tools for self-tracking of online, self-directed learning is to recognize to what extent encountering and processing a certain artifact (a resource) induced learning. In the co-evolution model, we assume that what we can measure is the friction (or irritation) which triggers internalization processes, i.e. what does the artifact bring to the cognitive system that leads to its evolution. At the moment, we distinguish three forms of \frictions", leading to three categories of indicators of learning: { New concepts and topics: The simplest way in which we can think about how an artifact could lead to learning is through its introduction of new knowledge unknown to the learner. This is consistent with the traditional acquisition metaphor. In our scenario, this kind of friction happens for example when Jane watches a video about a sewing technique previously unknown to her. { Increased complexity: While not necessarily introducing new concepts, an artifact might relate to known concepts in a more complex way, where complexity might relate to the granularity, speci city or interrelatedness with which those concepts are treated in the artifact. In a social system, the assumption of the co-evolution model is that the interaction between individuals might enable such increases in understanding of the concepts being considered through iteratively re ning them. In our scenario, this kind of friction happens for example when Jane follows a statistics course which is more advanced than the ones she had encountered before. { New views and opinions: Similarly, known concepts might be introduced \in a di erent light", through varying points of views and opinions enabling a re nement of the understanding of the concepts treated. This is consistent with the co-evolution model in the sense that it can be seen either as a widening of the social system in which the learner is involved, or as the integration into di erent social systems. In our scenario, this kind of friction happens for example when Jane reads a critical review of a business management methodology she has been studying.
What appears evident from confronting the co-evolution model and the types of indicators described above with the scenario of the previous section is that such indicators and models should be considered within distinct \domains" of learning. Indeed, Jane in the scenario would relate to di erent social systems for example for her interest in sewing, cycling, business management and statistics. The concepts that are relevant, the levels of complexity to consider and the views that can be expressed are also di erent from each other in those domains.
We call those domains of learning learning scopes. In the remainder of this paper, we will therefore consider a learning scope to be an area or theme of interest to a learner (sewing, business, etc.) to which are attached (consciously or not) speci c learning goals, as well as a speci c set of concepts, topics and activities. 4
Considering the conclusions from the previous section, the key challenge at the intersection of theory and technology for self-tracking of online, self-directed learning is to devise ways to compute the kind of indicators that are useful to identify and approximate some quanti cation of the three types of frictions within (implicit/emerging) learning scopes. Before that, however, we have to face more basic technical challenges to set in place the mechanisms to collect, integrate, enrich and process the data necessary to compute those indicators. 4.1
The AFEL project aims at identifying the features that characterize learning
activities within online contexts across multiple platforms. With that, we
contribute to the eld of Social Learning Analytics that is based on the idea that
new ideas and skills are not only individual achievements, but also the results of
interaction and collaboration [
Therefore, AFEL adopted an approach to identify reliable data sources and to structure their capture process, which is based on an e ort to classify data sources, rather than the data themselves. Such an exercise in classi cation is important as it is the result of an e ort to understand what dimensions of the activities through the Web should be captured, before setting out to detect speci c learning activity factors. The resulting taxonomy revolves around a core of seven types of entities that a candidate data source has a potential for describing; these are further speci ed into sub-categories that capture a speci c set of dimensions, some of which are common to users and communities (e.g. learning statements), or to users (e.g. indicators of expertise) and learning resources (e.g. indicators of popularity). Those categories are at the core of the proposed AFEL Core Data Model9, an RDF vocabulary largely based on schema.org and which is, amongst other things, used to aggregate the datasets that AFEL makes publicly available10.
The following challenge for AFEL is to integrate data from a large number
of sources into a shared platform, using the core data model to integrate and
make them processable. The approach taken is to create a \data space", which
keeps most of the data sources intact at the time of on-boarding and being
integrated at query time through a smart API, following the principles set out in [
Beyond data storage and integration, the key to enable extracting the
features necessary to compute the kind of indicators mentioned in the previous
section is to connect those datasets at a semantic level, i.e. to enrich the raw data
into a more complete \Knowledge Graph". In other words, connecting the di
erent entities with each other and extracting from unstructured or semi-structured
sources entities of interest that can connect the data from a wide range of places.
In AFEL, we use entity linking approaches [
Additionally, AFEL provides a methodology to determine the characteristic features, which allow learning activities to be detected and described, and consequently the attributes that instantiate them, in di erent data sources identi ed within the project. This methodology facilitates an initial speci cation of the features relevant to learning activities by presenting an instantiation of them on some of key data sources. Furthermore, with our methodology, we also outline a top-down perspective of feature engineering indicating that features identi ed in AFEL are applicable in di erent use cases, in general online contexts and that they can be extracted from our data basis. 4.2
In this section, we present a short pilot experiment in which we implemented an initial version of showing indicators based on topics included in the learning activities of a user (consistently with what described in Section 3). This relies on some of the technical aspects described above, including data capture and enrichment.
The data: We use approximately 6 weeks of browsing history data for a user, obtained through the AFEL browser extension12, which pushes this information as the user is browsing the web. Each activity is described as an instance of the concept BrowsingActivity in the AFEL Core Data Model, with as properties the URL of the page accessed and the time at which it was accessed. In our illustrative example, this corresponds to 42 707 activities, making reference to 12 738 URLs of webpages.
Topic Extraction: The rst step to extracting the learning scopes from the activity data is to extract the topics of each resource (webpage). For this, we rst use DBpedia Spotlight13 to extract the entities referred to in the text in the form of Linked Data entities in the DBpedia dataset14. DBpedia is a Linked Data version of Wikipedia, where each entity is described according to various properties, including the categories in which the entity has been classi ed in Wikipedia. We therefore query DBpedia to obtain up to 20 categories from the ones directly connected to the entities, or their broader categories in DBpedia's category taxonomy. 12 https://github.com/afel-project/browsing-history-webext 13 http://spotlight.dbpedia.org 14 http://dbpedia.org
For example, assume the learner views a YouTube video titled LMMS Tutorial | Getting VST Instruments.15 When mining the extracted text (stripped of HTML markup), DBpedia Spotlight detects that the description of this video mentions entities such as <http://dbpedia.org/resource/LMMS> (dbp:LMMS for short - a digital audio software suite) or dbp:Virtual Studio Technology. Querying DBpedia reveals subject categories for dbp:LMMS, such as <http://dbpedia.org/resource/Category:Free audio editors> (or short, dbc:Free audio editors) or dbc:Software drum machines. The detected category dbc:Free audio editors is in turn declared in DBpedia to have broader categories such as dbc:Audio editors or dbc:Free audio software. All of these elements are included in the description of the activity that corresponds to watching the above video, to be used in the next step of clustering activities.
On our browsing history data, running the resources through DBpedia Spotlight extracted 20 876 distinct entities, each being added 20 categories on average. To give an idea of the scale, the nal description of the 6 weeks of activities of this one learner takes approximately 1.1GB of space and took between 1 and 15 seconds to compute for each activity (depending on the size of the original text, using a modern laptop with a good internet connection).
Clustering activities: In the next step, we use the description of the activities as produced through the process described above in order to detect candidate learning scopes, i.e. groups of topics and activities that seem to relate to the same broader theme. To do this, we consider the set of entities and categories obtained before similarly to the text of documents and apply a common document clustering process on them (i.e. TFIDF vectorization and k-Means clustering). We obtain from this a set of k clusters (with k being a given) that group activities based on the overlap they have in the topics (entities and categories) they cover. We label each cluster based on the entity or category that best characterizes it in terms of F-Measure (i.e. that covers the maximum number of activities in the cluster, and the minimum number of activities outside the cluster), representing the target of the topic scope.
The clustering technique we applied (k-Means) requires to x the number of clusters to be obtained in advance. We experimented with numbers between 6 and 100, to see which could best represent the width and breadth of interests of this particular learner. Here, we used 50 as it appeared to lead to good results (as future work, we will integrate ways to automatically discover the ideal number of clusters for a learner). Figure 2 shows the clusters obtained and their size. The gray line describes all activities in the topic scope, i.e. all activities that have been included in the cluster. As can be seen, the clusters are unbalanced between the ones with a large number of activities (Google, Web Programming) with thousands of activities, and the ones representing only a few hundreds of activities.
Topic-based indicator: In the initial scenario we are considering here, we focus on a topic-based indicator which consist in checking whether an activity introduces 15 https://www.youtube.com/watch?v=aZKra7rNspU new topics (entities or categories) into the learning scope (cluster) in which it is included. We therefore \play back" the sequence of browsing activities from the learner's history, checking at each time how many new topics are being introduced that were not present in the previous activities of the learner in this scope.
Looking again at Figure 2, it is interesting to look at the di erence between the gray line (number of activities in the topic scope) and the black line, representing the number of activities that have integrated new topics in the scope and can therefore be considered learning activities. For example, since the user uses many Google services for basic tasks (such as Gmail for emails), it is not surprising that the Google scope, while being the largest in activities, does not actually include much detected learning activities. What is obvious however is that the balance is much di erent for other clusters that can be clearly identi ed for including large amounts of learning activities.
Indeed, we can see the value of the process here by comparing the learning trajectories of the learner according to the de nition of contributions to di erent learning scopes considered. For example, the scope on Digital Technology, representing the largest number of learning activities, can be seen in Figure 3 (top) as a broad topic on which the learner is constantly (almost everyday) learning new things. In contrast, the learning scope on Web Programming, although very related, is one where we can assume the learner already has some familiarity and only makes a signi cant increment in their learning punctually, as can be seen by the jump around 08 September in Figure 3 (bottom). The current state in the implementation of the aforementioned aspects takes the form of a prototype learner dashboard, available from the Didactalia platform. The dashboard illustrated in Figure 4 includes initial placeholder indicators for the kind of frictions identi ed in Section 3 and is implemented on the technologies described above. It is however a preliminary result, showing the ability to technically integrate the di erent AFEL components into an rst product. It will be further evolved in order to truly address the scenario of Section 2, including user feedback and more accurate indicators.
A key aspect to achieve the goal in our everyday learning scenario is that
the user should have control over what is being monitored. Indeed, the learner
should be able to decide what area of the data should be displayed, according
to which indicator and which dimension of the data (e.g. speci c topics, times,
resources or platforms). Our approach here is to rely on a framework for exible
dashboards based on visualization recommendation, implemented through the
VizRec tool [
One way to move the learner to action is via recommending learning resources
that appear to be relevant considering the current state of the learner [
However, an important aspect, which is still missing, is how such measures of similarity can be based on metrics that are relevant to learning rather than on basic content or pro le similarity. Indeed, the objective here would be to recommend learning resources (or even learning resource paths) that have already been helpful for other users with a similar learning goal and a similar learning state (in terms of the concepts, complexities and views already encountered). In other words, the recommendations can be based on a meaningful view of what the suggested resources might contribute to learning. 6
In the previous section, we discussed how to theoretically and technically implement tools for self-awareness targeted at self-directed online learning. Those tools are currently at early stages of development. Beyond those aspects however, other challenges will be faced by the AFEL consortium. One of them includes facilitating the adoption of these tools by a wide variety of users. Indeed, the actual usefulness and value of such personal analytics dashboards and learning assistant technologies have not been formally assessed and the participation of the learner community in their development is necessary in order to ensure that they reach their potential. The approach taken by AFEL here is to start with the community of learners in the Didactalia platform, enabling the dashboard for them and through that, supporting them in integrating data from other platforms. With a large number of users, we will be able to collect enough data to understand how such monitoring can truly support users in reaching awareness of their learning behavior, and how this can help them take decisions with respect to their own learning.
Another aspect which is not discussed in this paper is the ethical implications of realizing such tools and reaching a wide-adoption. As mentioned above, each of the learners is assigned their own data space on the AFEL platform, which is only accessible by them. However, as mentioned in the scenario of Section 2, support to the learner might be better achieved by enabling them to compare their own behavior with others, and we aim to make some aggregated data available to others for research purposes. Proper anonymisation techniques need to be applied in order to ensure that external parties cannot infer information about speci c learners from having access to those tools and data.
Beyond privacy however, it is also important to ensure that the e ect of the
tool might not turn out to be negative. Existing work have shown a number
of ethical harms that might come out of enabling self-governance in a number
of domains, despite the obvious positive e ects [
This work has received funding from the European Union's Horizon 2020 research and innovation programme as part of the AFEL (Analytics for Everyday Learning) project under grant agreement No 687916.