In this paper, we present preliminary results of AFEL-REC, a recommender system for social learning environments. AFEL-REC is build upon a scalable software architecture to provide recommendations of learning resources in near real-time. Furthermore, AFEL-REC can cope with any kind of data that is present in social learning environments such as resource metadata, user interactions or social tags. We provide a preliminary evaluation of three recommendation use cases implemented in AFEL-REC and we find that utilizing social data in form of tags is helpful for not only improving recommendation accuracy but also coverage. This paper should be valuable for both researchers and practitioners interested in providing resource recommendations in social learning environments.
Recommender systems aim to predict if a specific user will like
a specific resource. To do so, recommender systems analyze past
usage behavior (e.g., clicks or likes) with the goal to generate a
personalized list of potentially relevant resources [
Especially in the field of TEL, recommender systems have
become an important research area over the past decade [
We believe that our work contributes to the under-researched portfolio of recommender systems for social learning environments. Furthermore, we present an overview of use cases and preliminary evaluation results, which should be valuable for both researchers and practitioners interested in providing resource recommendations in social learning environments. 2
In this section, we present our AFEL-REC system by providing a detailed description of its software architecture as well as potential use cases that can be realized for recommending resources in social learning environments. 2.1
The software architecture of AFEL-REC is based on the scalable
recommendation framework ScaR3 [
Data Modification Layer DML Service Provider
SP Recommender Engine RE Recommender Evaluator REV get recommendations add (social) data /
interactions s
evaluate algorithms update profiles Recommender Customizer RC
Data Modification Layer (DML) & Solr. The DML encapsulates all CRUD operations (i.e., create, retrieve, update, delete) in one module and therefore, enables easy access to the underlying data backend. As depicted in Figure 1, AFEL-REC uses the high-performance search platform Apache Solr4. This data backend solution not only guarantees scalability and near real-time recommendations but also the support of multiple data sources. While most recommender systems rely on rating-based data, AFEL-REC is also capable of processing relevant social information such as tags.
Recommender Engine (RE). The RE is the heart of AFEL-REC and is responsible for calculating recommendations. As we are using Apache Solr, we can benefit from its build-in data structures for eficiently calculating user and resource similarities, and ranking resources based on their relevance for a specific recommendation context. In Section 2.2, we discuss possible use cases and algorithms that can be realized using AFEL-REC’s recommender engine. Recommender Customizer (RC). The RC is used to change the parameters (e.g., the neighborhood size n) of the recommendation approaches on the fly. Thus, it holds a so-called recommendation profile for each approach, which can be accessed and changed by the system administrator. These changes are then broadcast to the RE to be aware of how a specific approach should be executed. Recommender Evaluator (REV). The REV is responsible for evaluating the recommendation algorithms implemented in the RE. Thus, it can be executed to perform an ofline evaluation with training/test set splits (see Section 3) or an online evaluation with A/B-tests.
ZooKeeper. We are using Apache ZooKeeper5 for handling the communication between the modules and for ofering horizontal scaling. Thus, in cases in which we observe a high request load, we can start multiple instances of the same module (indicated by the arrows in the DML and RE modules). 2.2
Using this software architecture, AFEL-REC is capable of supporting
seven use cases for providing recommendations in social learning
environments:
UC1: Recommendation of Popular Resources in the Social
Learning Environment. The first use case is a non-personalized
one and is especially useful for new users of a social learning
environment without any user interactions so far (i.e., cold-start users
[
UC2: Recommendation of Resources That Like-Minded Users
Have Interacted With. The second use case is a personalized
one and thus, analyzes past user interactions to specifically tailor
recommendations towards learners. Collaborative Filtering (CF)
algorithms are typically chosen to realize such a use case [
UC3: Recommendation of Resources Based on Social Information. This use case is similar to UC2 but this time two users are treated as similar if they have shared some social information in the past. This social information can be friendship connections, group memberships or social tags. We hypothesize that social information is capable of providing a richer semantic representation of a user’s interests than pure interaction data. Therefore, we also think that this should positively influence the coverage of the recommendations as validated in Section 3 of this paper.
UC4: Recommendation of Resources That are Similar to the
Resources the User Has Interacted With. One disadvantage of
UC2 and UC3 is that it can only be applied to resources, which
already have user interactions or social information attached to them.
This means that cold-start resources without any user interactions
or social information cannot be recommended. To overcome this,
UC3 aims at utilizing resource similarities for personalized
recommendations by using a Content-based Filtering (CBF) approach [
UC5: Recommendation of (Alternative) Resources for a Specific Resource. This use case is related to UC4 but provides contextualized recommendations instead of personalized ones. This means that recommendations are not based on the learner but based on a specific resource by finding alternative ones. Thus, similar to UC4, the most similar resources for the given resource are recommended using CBF.
UC6: Recommendation of (Alternative) Resources for a Specific User and a Specific Resource. The next use case also focuses on recommending alternative resources for a specific resource but this time in a personalized manner. Such use case can be implemented using a contextualized CF approach. This means that we search for similar users of the target user, who have also interacted with the target resource. Thus, this use case is similar to Amazon’s “Users who bought this, also bought that” recommender. UC7: Recommendation of Resources for a Specific User and a Specific Learning Goal. Finally, the last use case tackles recommendations in a personalized and adaptive manner. While UC1 to UC6 have focused on providing relevant recommendations, they have neglected the learning goal of the user. Such a learning goal could be the aim to focus on a specific topic or to receive more dificult learning resources. To realize adaptive recommendations, the suggested resources could be re-ranked using a feature boosting technique. For example, if the learning goal is to study more dificult resources, then resources with a higher complexity (e.g., measured via a readability score) should be boosted and easy ones should be down-graded in the recommendation list. 3
In this section, we present preliminary evaluation results for AFELREC. The aim of this evaluation is twofold: (i) we want to show that AFEL-REC is capable of providing recommendations using data gathered from a real-world social learning environment, and (ii), Number of interactions (i.e., clicks) Number of users Number of learning resources Number of social tags Average number of interactions per user Average number of interactions per learning resource Average number of tags per learning resource 1,879,761 1,274,858 35,346 485,295 1.47 53.18 13.73 we want show that social information can enhance the prediction accuracy and coverage of recommendations. To do so, we focus on UC1 to UC3 presented in the previous section. 3.1
We collected our data from the social learning environment
Didactalia between the 26th of February 2017 until the 28th of May 2018.
This included 1,879,761 user interactions (i.e., clicks on learning
resources) by 1,274,858 users on 35,346 learning resources. This
resulted in 1.47 interactions per user and 53.18 interactions per
learning resource on average. Additionally, 485,295 social tags were
applied to these learning resources, which resulted in 13.73 tags per
resource on average. The full statistics of our dataset are shown in
Table 1. To date, the only social information we are using in our data
are tags but we are planning to extend this by also incorporating
connections between users or group memberships [
For evaluating AFEL-REC, we split our dataset into training and test sets. Therefore, we followed common practice in the research area of recommender systems and information retrieval by using the most recent 20% of interactions of each user for testing and the remaining 80% for training. This dataset splitting technique ensures that the chronological order of the data is preserved and thus, that the future is predicted based on past user interactions.
For measuring the accuracy of the recommendations, we use a
rich set of metrics, namely Recall (R@20, measured for k = 20
recommended resources), Precision (P@1, for k = 1), F1-score (F1@10
for k = 10), Mean Reciprocal Rank (MRR@20, for k = 20), Mean
Average Precision (MAP@20, for k = 20) and normalized Discounted
Cumulative Gain (nDCG@20, for k = 20) [
We evaluated UC1 - UC3 presented in Section 2.2 to show (i) the general usefulness of AFEL-REC for providing recommendations in social learning environments, and (ii) that social information in the form of tags is helpful for improving the recommendation accuracy and coverage. In the future, we will also evaluate UC4 - UC7.
The MP (MostPopular) approach refers to UC1 and is a
nonpersonalized algorithm, which recommends the most frequently
used learning resources in the system. This algorithm also works
for cold-start users [
The CFi algorithm refers to UC2 and calculates the neighborhood
of a user on the basis of interaction data (i.e., clicks). Thus, two users
are treated as similar if they have interacted with the same learning
resources in the past [
Similar to CFi , CFt is a Collaborative Filtering-based approach
but, as discussed in UC3, this one calculates the user neighborhood
based on social tags. Thus, two users are treated as similar if they
have used the same social tags in the past [
The preliminary results of our evaluation are shown in Table 2. While the lowest accuracy estimates are provided by the unpersonalized MP approach (UC1), this approach also provides the highest UC with 100%. This means that MP is capable of providing recommendations for all users in the social learning environment.
When looking at the next algorithm, CFi (i.e., CF on the basis of user interactions, UC2), we notice that this approach provides approx. 10 times higher accuracy values than MP. We contribute this to the personalization factor of the algorithm. However, one drawback of CFi is the rather small UC of 40%, which means that it cannot generate any recommendations for 60% of the users.
Finally, CFt (i.e., CF on the basis of social tags, UC3), provides not only the best results with respect to recommendation accuracy but also a larger coverage (C) than CFi . This result shows that social information can indeed help to improve recommendations in social learning environments. 4
In this paper, we presented AFEL-REC, a recommender system for social learning environments. AFEL-REC is build upon a scalable software architecture based on powerful open-source frameworks such as Apache Solr or Apache ZooKeeper and thus, is capable of providing near real-time recommendations of learning resources. We have demonstrated the usefulness of AFEL-REC by discussing use cases that can be realized with it and by providing preliminary evaluation results using data gathered from the Spanish social learning environment Didactalia. Our evaluation results show that social information in the form of tags can be used to enhance the accuracy and coverage of learning resource recommendations. These ifndings should be of interest for both researchers and practitioners interested in providing resource recommendations in social learning environments.
Future Work. One limitation of this work is that we have only
investigated tags as potential social information for learning
resource recommendations. Thus, for future work, we plan to extend
AFEL-REC by also considering additional types of social
information such as friendship connections or group memberships. Also,
we plan to include our cognitive-inspired tag recommender
algorithms, which have been specifically useful in the context of TEL
[
Acknowledgments. The authors would like to thank Didactalia and the AFEL consortium. This work was supported by the KnowCenter GmbH Graz (Austrian FFG COMET Program) and the EUfunded H2020 project AFEL (GA: 687916).