FullBrain: a Social E-learning Platform Mirko Biasini1 , Vittorio Carmignani1 , Nicola Ferro2 , Panagiotis Filianos1 , Maria Maistro3 , and Giorgio Maria di Nunzio2 1 FullBrain, https://fullbrain.org/ {mirko, vitto, panos}@fullbrain.org 2 University of Padua, Padova, Italy {ferro,dinunzio}@dei.unipd.it 3 University of Copenhagen, Copenhagen, Denmark mm@di.ku.dk Abstract. We present FullBrain, a social e-learning platform where stu- dents share and track their knowledge. FullBrain users can post notes, ask questions and share learning resources in dedicated course and con- cept spaces. We detail two components of FullBrain: a Social Information Retrieval (SIR) system equipped with query autocomplete and query au- tosuggestion, and a Leaderboard module to improve user experience. We analyzed the day-to-day users’ usage of the SIR system, measuring a time-to-complete a request below 0.11s, matching or exceeding our UX targets. Moreover, we performed stress tests which lead the way for more detailed analysis. Through a preliminary user study and log data anal- ysis, we observe that 97% of the users’ activity is directed to the top 4 positions in the leaderboard. Keywords: e-learning · social networks · social information retrieval · gamification · leaderboards 1 Introduction E-learning platforms have been improving the quality of education in the digital sphere, with an increasing popularity over the past two decades. Nevertheless, the main approach has been to port the offline experience to its virtual analogue, as in the case of Massive Open Online Course (MOOC) platforms, which offer video lectures, note taking, quizzes, forums, etc. Well known examples of MOOC platforms are edX [2], Udemy [3] and Coursera [1]. However, MOOCs struggle to keep users engaged: approximately 88% of learners do not return after a first year of usage and on average only 8% of students per cohort are retained [67]. Moreover, we believe MOOC platforms to be structurally lacking. By mimick- ing the physical education experience, they miss the possibility of providing the novel experience that the digital medium can offer. A multitude of digital learn- ing features has been proposed, such as personalization, gamification and social, Copyright c 2021 for this paper by its authors. Use permitted under Creative Com- mons License Attribution 4.0 International (CC BY 4.0). This volume is published and copyrighted by its editors. IRCDL 2021, February 18-19, 2021, Padua, Italy. connectivist learning, however the attempts to align these elements have been sparse. In this context, we identify two main challenges. First, the social component is a fundamental part of the learning process [7, 25, 48, 69], and when this is not provided by e-learning platforms, it is usually supplemented, in official university courses, by different services, as Slack [71], Teams [53], Piazza [63], Moodle [55], etc. However, many students consider these solutions inadequate and, when possible, they prefer to directly contact a peer they personally know and trust. Between each other, they use makeshift solu- tions like Facebook Groups or Messenger [20] and WhatsApp [21] group chats. Students there share the same context and can immediately assist on queries with understanding of the course requirements. The above, however, force users to combine a multitude of services to achieve all their goals. Second, a single cen- tralized and reliable source of learning material is not available for online learn- ers. Students often search and access to different sources through Web search engines, however they struggle in determining the quality of each resource, es- pecially when they do not have enough knowledge to critically review them. The results are often ordered following an opaque process. The above require extended time to review learning resources before usage. The necessity of social e-learning platforms able to address the above challenges has became evident during the COVID-19 pandemic, when there has been a large scale transfer of learning activities to online spaces. According to Unesco [77] over 90% of the world students are confined at home. Providing alternative learn- ing solutions has become the top priority. We present FullBrain a social e-learning platform which addresses these chal- lenges. FullBrain is framed in the context of connectivism theory [26, 70], which represents learning as a network phenomenon influenced by technology and so- cialization. FullBrain offers open, dedicated spaces corresponding to university courses and concepts. It allows students to interact with these spaces, to post thoughts, ask questions and share, rate and comment online learning resources, tagging them with instructional metadata. These learning resources are orga- nized in a crowdsourced ontological structure forming a reviewed digital library. In this paper, we first present FullBrain’s features and then focus on two funda- mental components of its architecture: Social Information Retrieval (SIR) Sys- tem, that powers our search algorithms, and a user leaderboard, that encourages social connectivity in concept and course spaces. This paper is organised as fol- lows: Section 2 details the related work; Section 3 introduces FullBrain’s design, features and system’s architecture; Section 4 and Section 5 present the SIR and the leaderboard components respectively; Section 6 showcases the prototyped so- lutions, which are evaluated in Section 7. Finally, Section 8 includes conclusions and future work. 2 Related Work FullBrain belongs to the category of social e-learning platforms. Such efforts rarely have been part of literature and FullBrain stemmed from a few research papers: Gil and Martin-Bautista [25] propose a connectivist, ontological knowl- edge support system with a focus on collectivist knowledge building; Liu et al. [48, 49] propose systems for collaborative relevance assessment with tasks as context; and Chen and Su [13] propose SAP, an ontology-based social learning platform with learning paths. However none of these approaches have been adopted in a larger scale. edX [2], Coursera [1] and Udemy [3] are MOOC platforms that provide con- tent, but no immediate social networking tools. In contrast, learning platforms that offer social learning, as Golden [19] and Expii [18], have a wiki approach to knowledge. Differently, FullBrain fully integrates the learning and social compo- nents in a single platform. In addition to the social networking features, FullBrain offers a SIR Sys- tem [41], which ranks results using both topical and user similarity. The user similarity is calculated based on the distance in the social graph between the searcher and the results [40,42,78]. We propose an architecture of an IRS specif- ically designed in the context of Online Social Networks [6] able to handle dif- ferent entities, rather than only users (k-n partite relationships [5]). To the best of our knowledge such architecture is novel. Finally, FullBrain exploits gamification approaches through a leaderborad. Gamification has proven to be beneficial to increase user engagement in numer- ous contexts [16, 43, 44, 62]. Leaderboards are a workhorse of gamification [45] and are considered one of the key elements in game design [34]. According to Zichermann [23], leaderboards can follow two approaches: absolute and relative. Upon these, leaderboards emphasize continuous performance, status reporting, comparison, socialization, and competition [8, 14]. 3 FullBrain System In this section we present the elements that comprise FullBrain’s user interface and system. We start with the description of the main features in Section 3.1, the underlying social graph in Section 3.2 and the overall architecture in Section 3.3. 3.1 Features Six entities represent the core of FullBrain’s offering: Posts, Streams, Sources, Courses, Concepts and User Profiles. We detail them next. As in any regular Social Network, the users are able to create posts to share ideas. Also, users can post anonymously and specify if it is a question or a note. In the foreseeable future, they will be able to mark a question as ‘resolved’ if the issue was addressed. Posts are feature-heavy: users can like, comment and share any post they have access to. Streams refer to collections of consecutive posts in a specific space: in the context of a course, concept or user profile page. The user home page is an example of stream, where users receive posts aggregated from streams of the entities they have included (followed). An operational definition of source is: a resource to learn one or multiple subjects which can be freely accessed online. Simply, sources are URL links point- ing to a destination website which provide learning content. Sources are shared by members of the FullBrain community. These resources combined with the user who shared them have a set of metadata that form the source card: – Name, affiliation and level of the user who shared the resource. The level is an indicator of the user’s reputation in the platform. It depends on, for example, user’s interactions or obtained reviews on shared resources. This will be further addressed in Section 5; – Learning styles, which can be: watching, reading, practise/exercise, listening, group activity or tutorial; – Title which highlights the content of the source; – Instructions on how to traverse the source. For instance, if the source is a video, the instructions may be: “watch from minute 2:03 to 5:40 to learn concept X”; – Prerequisites: concepts that users have to know in advance to understand the learning source. Clicking into a prerequisite results in opening the related concept page; – Social stats: star reviews and comments allow to express an opinion on the source relevance for future viewers. We refer to course as an academic course associated with its respective In- stitution. Examples can be ‘Machine Learning’ and ‘Database Systems’ at the University of Padua (Unipd) or ‘Business Management’ at the Copenhagen Busi- ness School (CBS). Each course has a dedicated space with group features: users can join and post thoughts and questions. Moreover, students can share learning resources helpful to pass the course/exam. The course’s sources slightly differ from the ones in concepts: they do not have prerequisites, but tags, which allow to label the material based on the purpose: #lecture1, #exams or #exercises. An operational definition of concepts can be as follows: the smallest unit of knowledge, idea or process, which can not be divided further without becoming an other concept or losing utility for the task at hand. It is important to note that this is a soft definition, a guideline which is open for interpretation by FullBrain’s members. Every FullBrain member can create a concept space, which allows post and resource sharing. In that respect inter- actions are very similar to Course spaces. Concepts are linked through learning resource prerequisites, forming crowdsourced ontologies. Moderating the validity of these ontologies is expected to be a social endeavor as well. A user experience workflow is not proposed currently and is matter of future research. Each user has a user profile page, which includes the user name, institu- tional affiliation, description and picture. In addition it shows the list of includers (followers) and included users (followed users), the lists of the courses and con- cepts the user has included and a stream of their posts. contains Playlist has_source Source views has_prerequisite rates comments creates Concept knows Origin follows has_post in_progress is_from follows mentions views has_course views User comments follows follows views mentions Course views creates likes Tag reshares Post Tags mentions has_post Fig. 1. FullBrain Social Graph with entities as nodes and related relationships as edges. 3.2 Social Graph A social network can be seen as a graph [17, 56, 57]. Indeed, Elveny et al. [17] present the users as social actors which interact with each other creating rela- tionships. In FullBrain, the relationships between users are: asymmetric, explicit, non-confirmed, regular and unsigned [29, 30]. The FullBrain social graph, shown in fig. 1, can be seen as an extended social graph with the k-partite definition [5], where nodes can be of different types. We recognize the main entities in Section 3.1 as nodes. In addition, we find Origins (institutions), Tags (labels for learning material and posts) and Playlists (public or private collections of favorite sources created by users). As edges, we can see all the possible relationships between the various entities (nodes). 3.3 Architecture Figure 2 displays FullBrain’s architecture. All virtual machines and databases are hosted in Azure [52]. Cloud services ensure scalability and continuity. We split FullBrain’s architecture into three main parts: front-end, back-end and databases. The front-end is the layer in direct contact with the user. When a user nav- igates through the FullBrain platform, it creates a series of requests. Those are forwarded from the front-end browser client to the back-end server. The back-end is in charge of handling those requests, and return back the expected results. Multiple microservices compose the back-end [51]. To each Microservice (MS) we assign specific functionalities. MSs can communicate using gRPC [27] (a lan- guage independent protocol) using Protobuf serialization. Such architecture gives the possibility of developing each of the MSs in a different programming language to maximize the performance based on peculiarities and available libraries. Finally, as we can see in fig. 2, each MS is connected to different databases, depending on their operations/calls. PostgreSQL [64] maintains most of Full- Brain’s entities’ fields data, and it ensures integrity between them. Neo4j [59], Fig. 2. FullBrain Architecture. In orange the Microservices (MS)s that contain the functions we describe in Section 4 and Section 5. a graph database, manages the different entity relationships seen in Figure 1. MongoDB [54] is mostly utilised for logging of user’s activities and requests. 4 Social Information Retrieval (SIR) System Carmignani [11] describes FullBrain’s SIR system, which resides in the Search MS (fig. 2). It offers three functionalities: Search, Query Autocomplete (QAC) and Query Suggestion (QS). Search and QAC estimate relevance by computing the similarity between the query, the entities and the user, and then order the results accordingly to such value [46]. The overall similarity S [40] for a query q with respect to an entity e and a user u (searcher) is calculated as follow: αST (q, e) + βSU (u, e) S(u, q, e) = ∈ [0, 1] α, β ∈ R+ (1) α+β where ST and SU are, respectively, topical and user similarity. We choose α = β = 1 to give the same importance to the topical and user components. The user similarity SU (u, e) in Equation (1) is computed as follows: d(u, e) SU (u, e) = 1 − ∈ [0, 1] (2) max d where d is the length of the shortest path between u and e in the social graph (Section 3.2). For efficiency reasons, we calculate an approximation of d using a modified version of the landmark embedding method [6,35,50,65,73,75,78]. According to Lampe et al. [42] “people are more likely to search for other users who are part of their real life friends-network, rather than other members”. Thus, we modify the standard algorithm by not saving distances greater than 3 (assuming them as infinite) as suggested by Vieira et. al. [78], since entities at such distance would not be relevant for the user. This assumption has saved, in our case, more than 80% of landmarks-nodes stored distances. The computation of the topical similarity ST (q, e) in Equation (1) differs for search and QAC. We denote it by SSearch and SQAC respectively. The search function results include: concepts, users, courses, sources and posts. SSearch , is computed as a combination of partial [12, 37] and exact match as follows: SP artial (q, e) + SExact (q, e) SSearch (q, e) = ∈ [0, 1] (3) 2 For partial matching SP artial (q, e), we employ q-grams [28,32,39,47,58,76]. Given the query q and an entity name ne (e.g. user or concept name), we apply the same text operations on q and ne , retrieving their respective set of q-grams, Gq and Gne : |Gq ∩ Gne | SP artial (q, e) = ∈ [0, 1] (4) |Gq ∪ Gne | For exact matching SExact (q, e), we use the Vector Space Model [46] and calculate SExact (q, e) as the cosine similarity of the vector representations q and e. Specifically for the latter, we use the union of e’s fields, like name and description (giving weights according to the fields’ importance): SExact (q, e) = cos(q, eunion ) ∈ [0, 1] (5) FullBrain’s QAC makes possible to retrieve a Qlist [72] of entities (users, concepts and courses) that are relevant for the user while typing the query itself [72]. As demonstrated in a variety of studies [9, 74, 80, 81], such features improve user satisfaction. The elements that populate the Qlist are ranked with the overall similarity in Equation (1), where the user similarity is computed as in Equation (2) and the topical similarity is SQAC (q, e) = SP artial (q, e). We use only partial matching to not compromise the speed of retrieval required by QAC. Finally, we describe the Query Suggestion (QS) component. We refer to QSs as the elements which populate the QList before typing any character into the query input form. QSs are either past queries (input for search function) or actual entities (direct link to the related web page). Such elements are retrieved based on the user’s search history (more recent would result in a higher rank) without any similarity computation. We also suggest trending queries and entities. Those are retrieved using a modified version of the Most Popular Completion (MPC) model [10, 36, 68]; where a time filter is applied following the finding of [74, 79], which have shown that queries’ popularity is heavily affected by the time. 5 Leaderboard As previously mentioned, gamification can positively impact users’ interactions and contributions. Thus, we designed and developed a leaderboard aiming to increase learners’ engagement in FullBrain. Our platform offers a multitude of Up Vote Up Vote Action Add Share Rate Comment Add Share Rate Comment Include Comment Comment Entity Source Post User Cou. Con . Avg Effort 4.4 3.3 1.6 2.5 1.1 2.5 1.4 1.3 2.1 1.7 1.6 Avg Value 4.7 4.5 4.7 3.9 4 4.5 4.4 4.4 4.1 4 3.1 Sum 9.1 7.8 6.3 6.4 5.1 7.0 5.8 5.7 6.2 5.7 4.7 Table 1. Leaderboard points system actions’ scores actions, which can be performed in different contexts, such as courses and con- cepts. Combining different sets of actions and goals, a variety of leaderboards can be generated to address different motivational areas. Upon these, leader- boards provide learners with elements for reflection, comparison, motivation, socialization, and competition [8, 14]. Points system and mechanics To be created, leaderboards require a point system. The point system is a mechanism defining a set of criteria, which are used to assign points to the users. Through these points, a score is computed. Finally, users are ranked based on their final score. As suggested by Jia et. al [38], leaderboards need to provide a sense of fairness to the users. To generate a points system as fair as possible, we categorized all available actions based on two parameters: effort and value. The former assesses the effort required to perform the action, inside and outside the platform. The latter is an evaluation of the value the performed action provides to the platform. Among all the possible actions available in FullBrain, 11 were selected to build the points system. Precisely, add, share, rate, comment, upvote comment a source, add, share, like, comment, upvote comment a post, and include user/ course/concept. Several users were interviewed to assign an effort and a value score to each action. Scores were given between 1 and 5. The higher the score, the higher the effort/value. Next, these were averaged out to generate the final points. As a result, the final points system was built as shown in table 1. Every time a user performs one of these actions, the corresponding points are assigned to the user. As described above, different types of leaderboards can be generated based on actions and contexts. To do so, several activity metadata are stored every time a user performs an action: the user performing the action, the performed action, the location where the action is performed (e.g. concept, course, user), the object undergoing the action (e.g. source, post, concept, etc), the date and time the action is performed, and the corresponding points. Finally, these activities are filtered based on one or more parameters, to generate the desired leaderboard. To handle delete actions, such as deleting a post, analogous data are stored but with a negative score instead. Deleting actions affect only the user who performs the deleting action. Users who earned points by interacting with the deleted element remained unaltered. This is done to ensure fairness. Leaderboard design To be effective, leaderboards need to be designed based on the audience and the context [15, 33, 38, 45]. In FullBrain, concepts have a larger and diverse audience compared to courses. Indeed, as courses refer to a specific university, they are generally restricted to those students who are currently taking that course. Conversely, concepts are followed by learners all around the world. To determine the most appropriate design for each scenario, interviews, A/B tests [61] and thinking-aloud usability evaluations [31] were conducted aiming to determine four main points. – Leaderboard design: four different designs were tested: an absolute, a relative, and two hybrids. The absolute solution focuses on the top players, whereas the relative is centered on the active user [23,38]. On top of these, we proposed two hybrid designs, which combine both approaches to exploit their advantages. Each design affects different motivational areas like reflection, competition, or socialization. – Leaderboard items: which and how information should be displayed in the leaderboards. Here, focus was directed towards users’ scores, to determine whether showing points is more effective or not. – Time: Leaderboard points can be calculated based on user activities in- cluded in a time window of the current week, month, semester etc. We should determine which time windows provide valuable information to users. – Leaderboard type: identify which type of leaderboards are more inter- esting for users. For example, we can have Top Contributor, Top Reviewer or Top Influencer leaderboards. Each ranks users based on points collected from a different set of activities. The results of these experiments led to the final designs described in the following section. Finally, a specific leaderboard is generated for each concept and course. Note, the way we store data allows creating general leaderboards, comprising groups of concepts and/or courses. For example, the top contributors of a specific university or of a big subject like Machine Learning. More research has been conducted on that subject by Biasini [4]. 6 Prototype Social Information Retrieval System - Figure 3 displays the results of us- ing FullBrain’s SIR in the FullBrain main search bar. Figure 3A shows the QS, where five past queries and entities are suggested together with as many popu- lar ones. Figure 3B instead, shows a QAC output and how the use of trigrams permits to retrieve “Vittorio Carmignani” despite the input typos. Lastly, Fig- ure 3C,D,E show three different example of the search function. In C, q=“pca”, we can recognize the concept, together with posts and sources. In D, q=“dtu” (Technical University of Denmark), the results are retrieved even if not present in the entities’names. This is because we are using additional fields in the search process. Ultimately, in E, q=“Vittorio Karmignani”, despite the typo we can still retrieve the results thanks to the use of partial matching in the search function. A working example of the SIR system is available online4 . 4 https://youtu.be/6YOm-xobCzA Fig. 3. Example of QS (A), QAC (B) and Search Results (C,D,E) for the user “Vittorio Carmignani”. Moreover, the legend underneath highlights the different entities. Leaderboard - We propose two different hybrid solutions as a result of user interviews, A/B tests and usability evaluations: Hybrid-Absolute Design and Hy- brid 50-50 Design (fig. 4). We use the hybrid-Absolute design for concepts: an absolute leaderboard shows the top 10 users and the active user is always shown at the bottom of the leaderboard, regardless of his/her ranking. For courses we use the hybrid 50-50 design, a novel approach combining an absolute leader- board, on the top, with a relative leaderboard centered on the active user, on the bottom. In both solutions, users can access top users’ profiles and universities, as well as include them in their home stream. A time filter allows users to change the time window used to build the leaderboard. In addition, users can change the type of the leaderboard choosing between: Top Contributor, which ranks users accounting for all types of actions, and Top Responder, which ranks users accounting just for the frequency of their answers and the quality of their com- ments. Both solutions are partially implemented in the platform and we are currently working to complete them. 7 Evaluation and Results Social Information Retrieval System - Carmignani [11] focused on the ef- ficiency of FullBrain’s IRS. Based on J. Nielsen’s studies [60], he set that the time to complete a QS, QAC and Search request should be, respectively, 0.1s (instantly), in [0.1-1]s (small delay without interrupting user’s flow of thought) and [1-1.5]s (it is not an interactive function). Hence, we have logged the time to complete a QS, QAC and Search requests on FullBrain from all user clients using the platform for 3 weeks. The results are shown in the “Day-to-day” us- age columns in table 2. We can see that all three functions were within the bounds set. Therefore, we performed stress tests to test the limits of the architecture. Particularly we made 1000/n requests from n different users in A B Time filter Time filter Top contributors Last month Top contributors Last month Top contributors Last month Top contributors Last month Last week Last week Mark Lennon Mark Lennon Mark Lennon Mark Lennon 1 Technical University of Denmark INCLUDE 1 Technical University of Denmark INCLUDE Last month 1 Technical University of Denmark INCLUDE Last month 1 Technical University of Denmark INCLUDE Last 6 months Current semester Emma Shiffrin Emma Shiffrin Emma Shiffrin Emma Shiffrin 2 INCLUDE 2 INCLUDE Last year 2 INCLUDE All time 2 INCLUDE University of Padua University of Padua University of Padua University of Padua All time Select semester Carlos Rodriguez Carlos Rodriguez Carlos Rodriguez Spring 2020 Carlos Rodriguez 3 Copenhagen Business School INCLUDE 3 Copenhagen Business School INCLUDE 3 Fall 2020 Copenhagen Business School INCLUDE 3 Copenhagen Business School INCLUDE Spring 2019 Helen Larsen Helen Larsen John Ligety John Ligety INCLUDE INCLUDE Fall 2019 INCLUDE INCLUDE Technical University of Denmark Technical University of Denmark Harvard University Harvard University Elena Bianchi University of Verona INCLUDE Leaderboard type Elena Bianchi University of Verona INCLUDE Leaderboard Ole Riis type Copenhagen Business School INCLUDE Ole Riis Copenhagen Business School INCLUDE John Ligety Top contributors John Ligety Last month ... Top contributors Last month ... INCLUDE INCLUDE Harvard University Top contributors Harvard University Top contributors Irene Diaz Irene Diaz Mark Lennon 91 Mark Lennon INCLUDE 91 INCLUDE 1 responders Top INCLUDE 1 responders Top University of Barcelona INCLUDE University of Barcelona Ole Riis Technical Ole RiisUniversity of Denmark Technical University of Denmark INCLUDE INCLUDE Copenhagen Business School Copenhagen Business School Helen Larsen Helen Larsen Emma Shiffrin 92 Emma Shiffrin INCLUDE 92 INCLUDE 2 INCLUDE 2 Technical University of Denmark INCLUDE Technical University of Denmark Irene Diaz University Irene Diazof Padua University of Padua { INCLUDE INCLUDE University of Barcelona University of Barcelona 3 Carlos Rodriguez Copenhagen Business School INCLUDE 93 3 John Butler Carlos Rodriguez Active user Technical University of Denmark Copenhagen Business School INCLUDE INCLUDE 93 John Butler Technical University of Denmark INCLUDE Frederik Bruun Frederik Bruun INCLUDE INCLUDE Technical University of Denmark Technical University of Denmark Elena Bianchi Elena Bianchi Helen Larsen 94 John Ligety INCLUDE 94 INCLUDE INCLUDE University of Verona INCLUDE University of Verona Alice Rossi Technical University of Denmark Alice Rossi Harvard University INCLUDE INCLUDE University of Padua University of Padua Frederik Bruun Frederik Bruun Elena Bianchi 95 Ole Riis INCLUDE 95 INCLUDE INCLUDE Technical University of Denmark INCLUDE Technical University of Denmark } Active user University of Verona Copenhagen Business School 93 John Butler Technical University of Denmark 93 John Butler Technical John University of Denmark Ligety ... See less See less INCLUDE Harvard University Irene Diaz 91 INCLUDE See less See less University of Barcelona Ole Riis INCLUDE Copenhagen Business School H l L Fig. 4. Leaderboard designs for Concept and Course spaces. (A) Hybrid-Absolute de- sign for concepts. (B) Hybrid 50-50 design for courses. Fig. 5. Setup for Stress Tests and components specs. parallel (with n ∈ {1, 2, 4, 8, 16, 32, 64}) using ghz [24]. The test setup is shown in fig. 5. The functions were applied on a dataset having a total of 5724 entities and 21512 relationships. The results are shown in the stress test columns in ta- ble 2. All functions show a similar performance patterns: they all break the set bounds after n = 4. In contrast, it is important to remark here that, by using the website, it is not possible to reproduce this kind of loaded environment (with only these number of users): there would be a delay between two users’ requests due to the UI and the physical action of submitting them. Although this delay can be minimal, it can be still valuable since in normal conditions a request is answered in 0.1s. In contrast, we know that this delay is zero in case of stress test (since the requests are programmatically made). Thus, we believe that the system can support far more than 64 parallel users using the platform (admitting that the tests machine utilized are not suitable for a production environment). Indeed, between user request submission and its answer computation, there is a whole structure of microservices and communications that takes place. If the microservices are the problem, then we could create clones and balance the load of requests. If the problem is the computation, we need to know if it is due to physical limits (machine not performant enough) or the actual implementation. Thus, further analysis would be required to understand exactly in which step the bottleneck is of the overall process. Day-to-Day Stress Tests Avg Time (s) for a Request with Avg Count Count n concurrent users Time 1 2 4 8 16 32 64 QS 8743 0.06s 1000 0.039 0.078 0.159 0.3444 0.614 1.191 2.390 QAC 7599 0.11s 1000 0.506 1.323 0.920 1.912 4.769 7.611 15.029 Search 512 0.10s 1000 0.234 0.555 0.594 1.751 2.135 1.052 7.754 Table 2. Efficiency Study on FullBrain’s SIR based on day-to-day usage on the plat- form and conducted stress tests. Leaderboard - During the experiments conducted to determine the final leaderboard designs, 73% of 15 testers preferred the hybrid-absolute design for concepts. When asked to justify such a choice, participants claimed that in broader contexts like concepts, they primarily care about top users. They only want to see how top users learn and get inspired from them. Conversely, opposite results were found for courses. Indeed, 75% chose the hybrid 50-50 solution. Here, users explained that since they have real relationships with other students, they also want to see similar classmates. This was justified for two reasons. Firstly, users want to assess themselves compared to the rest of the class. Secondly, users feel more comfortable to approach similar students rather than top ones [4]. To validate leaderboards’ effect on users’ engagement, we performed a longitudinal validation. We collected users activities made through the leaderboards, for a total of 30 days. Even though the project constraints did not allow us to pro- duce evidence with statistical significance, preliminary results suggest that users’ activities are mainly focused on top positions. In fact, 55% of the activities ob- served in the leaderboards occurred in the best users (1st position in the table). Moreover, more than 97% happened within the first 4 positions. 8 Conclusions and Future work FullBrain has shown a lot of promise to provide a digital home for learners. From the release of the social features on FullBrain (October 2020) until the current publication we have experienced an increase of 146% in new registrations, totalling at approximately 500 users. In addition, we have described FullBrain’s Social Information Retrieval System (SIR) and leaderboards. The SIR provides the following functions: Search, Query Autocomplete and Query Suggestions (no query input). These permit to explore platform’s content, search through multiple entities while using the social graph to rank the results themselves. Moreover, the architecture is able to build the related user’s responses in less than 0.11s. For the Leaderboards, we propose a hybrid-absolute design in concepts and a hybrid 50-50 design for courses, based on user interviews. Through preliminary analysis we recognise that Leaderboards produce user activity focused on the top ranking members with 97% of activity directed to the top 4 positions. Larger user numbers allow us to experiment with Machine Learning and Deep Learning systems. Previous work by Filianos [22] has attempted to use Multigated Mixture of Expert (MMoE) [82] and Mixture of Sequential Experts (MoSE) [66] models to rank posts in our home page stream. As our user base grows, we consider FullBrain to be very fertile ground for learning analytics and user activity dataset research. References 1. Coursera: Build Skills with Online Courses from Top Institutions. https://www. coursera.org/, Accessed: 2020-12-12 2. edX: Free Online Courses by Harvard, MIT, & more. https://www.edx.org/, Accessed: 2020-12-12 3. Udemy: Online Courses - Learn Anything, On Your Schedule. https://www.udem y.com/, Accessed: 2020-12-12 4. Biasini, M.: Designing analytics tools for decision-making in a crowd-sourced ed- ucational platform for STEM students. Master’s thesis, University of Padua (12 2020) 5. Bouadjenek, M.R., Hacid, H., Bouzeghoub, M.: Social networks and information retrieval, how are they converging? A survey, a taxonomy and an analysis of so- cial information retrieval approaches and platforms. Information Systems 56, 1–18 (2016). https://doi.org/10.1016/j.is.2015.07.008, http://dx.doi.org/10.1016/j .is.2015.07.008 6. Boyd, D.M., Ellison, N.B.: Social network sites: Definition, history, and schol- arship. Journal of Computer-Mediated Communication 13(1), 210–230 (2007). https://doi.org/10.1111/j.1083-6101.2007.00393.x, https://academic.oup.com /jcmc/article/13/1/210/4583062 7. Burke, M., Marlow, C., Lento, T.: Feed me: Motivating newcomer contribution in social network sites. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. p. 945–954. CHI ’09, Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1518701.1518847, https: //doi.org/10.1145/1518701.1518847 8. Butler, C.: The effect of leaderboard ranking on players’ perception of gaming fun. Online Communities and Social Computing. 5th International Conference, Ocsc 2013 Held As Part of Hci International 2013. Proceedings. Lncs 8029 pp. 129–36, 129–136 (2013). https://doi.org/10.1007/978-3-642-39371-6 15 9. Cai, F., De Rijke, M.: A survey of query auto completion in information re- trieval. Foundations and Trends in Information Retrieval 10(4), 273–363 (2016). https://doi.org/10.1561/1500000055 10. Cai, F., Liang, S., De Rijke, M.: Time-sensitive personalized query auto- completion. In: CIKM 2014 - Proceedings of the 2014 ACM International Con- ference on Information and Knowledge Management. pp. 1599–1608 (2014). https://doi.org/10.1145/2661829.2661921, http://dx.doi.org/10.1145/26618 29.2661921. 11. Carmignani, V.: Design and Development of an Information Retrieval System on an Educational Social Network for STEM students. Master’s thesis, University of Padua (12 2020) 12. Chaudhuri, S., Kaushik, R.: Extending autocompletion to tolerate errors (2009). https://doi.org/10.1145/1559845.1559919, http://www.amazon.com 13. Chen, Y., Su, C.: Ontology and learning path driven social e-learning platform. In: 2016 International Symposium on Computer, Consumer and Control (IS3C). pp. 358–361 (2016). https://doi.org/10.1109/IS3C.2016.99 14. Christy, K.R., Fox, J.: Leaderboards in a virtual classroom: A test of stereotype threat and social comparison explanations for women’s math performance. Computers and Education 78, 66–77 (2014). https://doi.org/10.1016/j.compedu.2014.05.005 15. Codish, D., Ravid, G.: Personality based gamification: How different personali- ties percive gamification. Ecis 2014 Proceedings - 22nd European Conference on Information Systems (2014) 16. Costa, J.P., Wehbe, R.R., Robb, J., Nacke, L.E.: Time’s up: Studying leaderboards for engaging punctual behaviour. Acm International Conference Proceeding Series pp. 26–33 (2013). https://doi.org/10.1145/2583008.2583012 17. Elveny, M., Syah, R., Elfida, M., Nasution, M.K.: Information Retrieval on so- cial network: An Adaptive Proof. IOP Conference Series: Materials Science and Engineering 300(1) (2018). https://doi.org/10.1088/1757-899X/300/1/012055 18. Expii: Expii — Learn Creatively (2020), https://www.expii.com/ 19. Expii: Golden — The intelligent knowledge base (2020), https://golden.com/ 20. Facebook: Messenger, https://www.messenger.com/ 21. Facebook: WhatsApp, https://www.whatsapp.com/?lang=it 22. Filianos, P.: A novel Deep Learning ranking model to order posts on a social e- learning platform. Master’s thesis, Technical University of Denmark (12 2020) 23. Gabe Zichermann, C.C.: Gamification by design: Implementing game mechanics in web and mobile apps. O’Reilly Media (2011) 24. Ghz: ghz · Simple gRPC benchmarking and load testing tool, https://ghz.sh/ 25. Gil, R., Martin-Bautista, M.: A novel integrated knowledge sup- port system based on ontology learning: Model specification and a case study. Knowledge-Based Systems 36, 340 – 352 (2012). https://doi.org/https://doi.org/10.1016/j.knosys.2012.07.007, h t tp : / / w w w . sciencedirect.com/science/article/pii/S0950705112001955 26. Goldie, J.: Connectivism: A knowledge learning theory for the digital age? Medical Teacher 38, 1064 – 1069 (2016) 27. Google: gRPC – A high-performance, open source universal RPC framework, ht tps://grpc.io/ 28. Gravano, L., Ipeirotis, P.G., Jagadish, H.V., Koudas, N., Muthukrishnan, S., Pietarinen, L., Srivastava, D.: Using q-grams in a DBMS for Approximate String Processing. Tech. Rep. 4 (2001), http://dblp.uni-trier.de/db/journals/debu /debu24.html\#GravanoIJKMPS01 29. Guy, I.: People recommendation on social media. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 10100 LNCS, pp. 570–623. Springer Verlag (2018). https://doi.org/10.1007/978-3-319-90092-6 15 30. Guy, I., Zwerdling, N., Ronen, I., Carmel, D., Uziel, E.: Social media recommenda- tion based on people and tags (2010). https://doi.org/10.1145/1835449.1835484, www.stumbleupon.com 31. H. Birns, J., A. Joffre, K., F. Leclerc, J., Andrews Paulsen, C.: Getting the whole picture: Collecting usability data using two methods – concurrent think aloud and retrospective probing (2016). https://doi.org/10.1.1.111.5466 32. Hall, P.A., Dowling, G.R.: Approximate String Matching. ACM Computing Sur- veys (CSUR) 12(4), 381–402 (12 1980). https://doi.org/10.1145/356827.356830, https://dl.acm.org/doi/10.1145/356827.356830 33. Hamari, J., Koivisto, J., Sarsa, H.: Does gamification work? - a literature re- view of empirical studies on gamification. Proceedings of the Annual Hawaii International Conference on System Sciences pp. 6758978, 3025–3034 (2014). https://doi.org/10.1109/HICSS.2014.377 34. Hickman, T.: Book review : Reeves, b., & read, j.l. (2009). total engagement: Using games and virtual worlds to change the way people work and businesses. harvard business school press. Computers in Human Behavior 26(6), 1276–1277 (2010). https://doi.org/10.1016/j.chb.2010.03.035 35. Hotz, M., Chondrogiannis, T., Wörteler, L., Grossniklaus, M.: Experiences with implementing landmark embedding in Neo4j. Proceedings of the ACM SIGACT-SIGMOD-SIGART Symposium on Principles of Database Systems (2019). https://doi.org/10.1145/3327964.3328496, https://doi.org/10.1145/ 3327964.3328496 36. Hu, Y., Xiao, C., Ishikawa, Y.: Context-Sensitive Query Auto-Completion with Knowledge Base. Tech. rep. (2018) 37. Ji, S., Li, G., Li, C., Feng, J.: Efficient interactive fuzzy keyword search (2009). https://doi.org/10.1145/1526709.1526760 38. Jia, Y., Liu, Y., Yu, X., Voida, S.: Designing leaderboards for gamification: Per- ceived differences based on user ranking, application domain, and personality traits. Conference on Human Factors in Computing Systems - Proceedings 2017-, 1949– 1960 (2017). https://doi.org/10.1145/3025453.3025826 39. Kim, M.S., Whang, K.Y., Lee, J.G.: n-Gram/2L-approximation: A two-level n- gram inverted index structure for approximate string matching. Tech. Rep. 6 (2007) 40. Kirchhoff, L., Stanoevska-Slabeva, K., Nicolai, T., Fleck, M.: Using Social Network Analysis To Enhance Information Retrieval Systems. In: Proceedings of the 5th Conference on Applications of Social Network Analysis (sep 2008) 41. Kirsch, S.M., Gnasa, M., Cremers, A.B.: Beyond the Web: Retrieval in social in- formation spaces. Tech. rep. (2006). https://doi.org/10.1007/11735106 9 42. Lampe, C., Ellison, N., Steinfield, C.: A face(book) in the crowd: Social Searching vs. social browsing. Tech. rep. (2006). https://doi.org/10.1145/1180875.1180901 43. Landers, R.N., Bauer, K.N., Callan, R.C.: Gamification of task performance with leaderboards: A goal setting experiment. Computers in Human Behavior 71, 508– 515 (2017). https://doi.org/10.1016/j.chb.2015.08.008 44. Landers, R.N., Landers, A.K.: An empirical test of the theory of gamified learning: The effect of leaderboards on time-on-task and academic performance. Simulation and Gaming 45(6), 769–785 (2014). https://doi.org/10.1177/1046878114563662 45. Leung, W.: How do one’s peers on a leaderboard affect oneself? Conference on Human Factors in Computing Systems - Proceedings pp. 1–11 (2019). https://doi.org/10.1145/3290605.3300397 46. Levene, M.: Search Engines: Information Retrieval in Practice. Tech. Rep. 5 (2011). https://doi.org/10.1093/comjnl/bxq039 47. Li, G., Feng, J., Li, C.: Supporting search-as-you-type using SQL in databases. IEEE Transactions on Knowledge and Data Engineering 25(2), 461–475 (2013). https://doi.org/10.1109/TKDE.2011.148, http://www.ijcttjournal.org 48. Liu, D.R., Wu, I.C.: Collaborative relevance assessment for task-based knowledge support. Decision Support Systems 44(2), 524 – 543 (2008). https://doi.org/https://doi.org/10.1016/j.dss.2007.06.015, http://www.scienced irect.com/science/article/pii/S0167923607000991 49. Liu, D.R., Wu, I.C., Yang, K.S.: Task-based k-support system: disseminating and sharing task-relevant knowledge. Expert Systems with Applications 29(2), 408 – 423 (2005). https://doi.org/https://doi.org/10.1016/j.eswa.2005.04.036, http: //www.sciencedirect.com/science/article/pii/S0957417405000898 50. Madkour, A., Aref, W.G., Rehman, F.U., Rahman, M.A., Basalamah, S.: A Survey of Shortest-Path Algorithms. Tech. rep. (2017), http://arxiv.org/abs/1705.020 44 51. Mazzara, M., Meyer, B.: Present and ulterior software engineering (2017). https://doi.org/10.1007/978-3-319-67425-4, https://doi.org/10.1007/978-3- 319-67425-4 52. Microsoft: What is Azure – Microsoft Cloud Services — Microsoft Azure (2019), https://azure.microsoft.com/en-us/overview/what-is-azure/ 53. Microsoft: Microsoft Teams — Group Chat, Team Chat & Collaboration (2020), https://www.microsoft.com/en-gb/microsoft-365/microsoft-teams/group- chat-software 54. MongoDB: The most popular database for modern apps — MongoDB (2020), https://www.mongodb.com/ 55. Moodle: Moodle - Open-source learning platform — Moodle.org (2018), https: //moodle.org/ 56. Nasution, M.K., Hardi, M., Syah, R.: Mining of the social network extraction. Journal of Physics: Conference Series 801(1) (2017). https://doi.org/10.1088/1742- 6596/801/1/012020 57. Nasution, M.K.: Social network mining (SNM): A definition of rela- tion between the resources and SNA. International Journal on Advanced Science, Engineering and Information Technology 6(6), 975–981 (2016). https://doi.org/10.18517/ijaseit.6.6.1390 58. Navarro, G., Sutinen, E., Tarhio, J.: Indexing text with approximate q-grams. In: Journal of Discrete Algorithms. vol. 3, pp. 157–175. Elsevier (jun 2005). https://doi.org/10.1016/j.jda.2004.08.003 59. Neo4j: Neo4j Graph Platform – The Leader in Graph Databases (2020), https: //neo4j.com/ 60. Nielsen, J.: Response Time Limits: Article by Jakob Nielsen (1993), https://ww w.nngroup.com/articles/response-times-3-important-limits/ 61. Nielsen, J.: Putting a/b testing in its place (2019), https://www.nngroup.com/ar ticles/putting-ab-testing-in-its-place/ 62. Ortiz-Rojas, M., Chiluiza, K., Valcke, M.: Gamification through leaderboards: An empirical study in engineering education. Computer Applications in Engineering Education 27(4), 777–788 (2019). https://doi.org/10.1002/cae.12116 63. Piazza: Piazza . Ask . Answer . Explore . Whenever. (2017), https://piazza.com/ 64. PostgreSQL: PostgreSQL: The world’s most advanced open source database (2013), https://www.postgresql.org/http://www.postgresql.org/ 65. Potamias, M., Bonchi, F., Castillo, C., Gionis, A.: Fast shortest path distance estimation in large networks (2009). https://doi.org/10.1145/1645953.1646063 66. Qin, Z., Cheng, Y., Zhao, Z., Chen, Z., Metzler, D., Qin, J.: Multitask Mixture of Sequential Experts for User Activity Streams, p. 3083–3091. Association for Computing Machinery, New York, NY, USA (2020), https://doi.org/10.1145/ 3394486.3403359 67. Reich, J., Ruipérez-Valiente, J.A.: The mooc pivot. Science 363(6423), 130–131 (2019). https://doi.org/10.1126/science.aav7958, https://science.sciencemag .org/content/363/6423/130 68. Shokouhi, M.: Learning to personalize query auto-completion (2013). https://doi.org/10.1145/2484028.2484076 69. Shum, S.B., Ferguson, R.: Social learning analytics. Journal of Educational Tech- nology & Society 15(3), 3–26 (2012), http://www.jstor.org/stable/jeductech soci.15.3.3 70. Siemens, G.: Connectivism: A learning theory for the digital age. International Journal of Instructional Technology and Distance Learning 2 (2004) 71. Slack: Where work happens — Slack (2019), https://slack.com/ 72. Smith, C.L., Gwizdka, J., Feild, H.: The use of query auto-completion over the course of search sessions with multifaceted information needs. Information Processing and Management 53(5), 1139–1155 (2017). https://doi.org/10.1016/j.ipm.2017.05.001, h t tp : / / d x . d o i . o r g / 1 0 . 1 0 1 6 / j .ipm.2017.05.001 73. Sommer, C.: Shortest-path queries in static networks (2014). https://doi.org/10.1145/2530531, http://dx.doi.org/10.1145/2530531 74. Tahery, S., Farzi, S.: Customized query auto-completion and suggestion — A review (2020). https://doi.org/10.1016/j.is.2019.101415, https://doi.org/10.1016/j. is.2019.101415 75. Tretyakov, K., Armas-Cervantes, A., Garcı́a-Bãuelos, L., Vilo, J., Dumas, M.: Fast fully dynamic landmark-based estimation of shortest path distances in very large graphs (2011). https://doi.org/10.1145/2063576.2063834 76. Ukkonen, E.: Approximate string-matching with q-grams and maxi- mal matches. Theoretical Computer Science 92(1), 191–211 (1 1992). https://doi.org/10.1016/0304-3975(92)90143-4 77. UNESCO: COVID-19: Learning Cities on the Front Line (2020), https://en.une sco.org/news/covid-19-learning-cities-front-line 78. Vieira, M.V., Golgher, P.B., Fonseca, B.M., De Reis, D.C., Damazio, R., Ribeiro-Neto, B.: Efficient search ranking in social networks (2007). https://doi.org/10.1145/1321440.1321520 79. Wang, Y., Ouyang, H., Deng, H., Chang, Y.: Learning Online Trends for Interactive Query Auto-Completion. IEEE Transactions on Knowledge and Data Engineering 29(11), 2442–2454 (11 2017). https://doi.org/10.1109/TKDE.2017.2738639 80. White, R.W., Marchionini, G.: Examining the effectiveness of real-time query ex- pansion. Information Processing and Management 43(3), 685–704 (may 2007). https://doi.org/10.1016/j.ipm.2006.06.005 81. Zhang, A., Goyal, A., Kong, W., Deng, H., Dong, A., Chang, Y., Gunter, C.A., Han, J.: AdaQAC: Adaptive query auto-completion via implicit negative feed- back. SIGIR 2015 - Proceedings of the 38th International ACM SIGIR Confer- ence on Research and Development in Information Retrieval pp. 143–152 (2015). https://doi.org/10.1145/2766462.2767697, http://dx.doi.org/10.1145/2766462 .2767697 82. Zhao, Z., Hong, L., Wei, L., Chen, J., Nath, A., Andrews, S., Kumthekar, A., Sathiamoorthy, M., Yi, X., Chi, E.: Recommending what video to watch next: A multitask ranking system. In: Proceedings of the 13th ACM Conference on Recommender Systems. p. 43–51. RecSys ’19, Association for Computing Ma- chinery, New York, NY, USA (2019). https://doi.org/10.1145/3298689.3346997, https://doi.org/10.1145/3298689.3346997