Explanation: An Interactive Course Recom mendation System for University Environments

Abstract

The abundance of courses available in university and the highly personalized curriculum is often overwhelming for students who must select courses relevant to their academic interests. A large body of research in course recommendation systems focuses on optimizing prediction and improving accuracy. However, those systems usually aford little or no user interaction, and little is known about the influence of user-perceived aspects for course recommendations, such as transparency, controllability, and user satisfaction. In this paper, we argue that involving students in the course recommendation process is important, and we present an interactive course recommendation system that provides explanations and allows students to explore courses in a personalized way. A within-subject user study was conducted to evaluate our system and the results show a significant improvement in many user-centric metrics.

1. Introduction

A course recommendation system suggests a student de

cide what they should study as per their requirements, which can solve the increasingly severe problem of information overload of course selection. Diferent from the ommendation domain, the interaction factor is essential for course recommendations in universities.

Course recommendations in universities particularly sufer from the cold start problem. Every year, there are freshmen enroll in, who have dificulty navigating their new academic and environment. It is dificult for a traditional course recommendation system to make successful suggestions for those new students without enough available information. Moreover, the necessary information is often too small to generate precise recommendations even for senior students. One common practice is using popular courses regardless of students’ interests when the system is short of students’ information and behavior. However, a promising alternative is to capture their preferences interactively. That is, if we could involve students in the recommendation process, we may get better results. parency [ 2 ]. Also, our proposed approach could increase

Many researchers have focused on recommending coursescontrol how recommendations are produced. To address the usability of the course recommendation system com- Second, those systems do not support exploration, which pared to previous works that only focus on improving is particularly important in the context where students accuracy. go through a broad exploratory phase before specializing. Finally, those systems often behave like a “black box” and do not give explanations that would allow students to 2. Related work reflect on their course selection.

In contrast to the approaches that consider more about 2.1. Interactive recommendation system the accuracy of predicted results, in this work, we build an interactive course recommendation system, which allows students to interactively improve the recommendations and bring their own preferences to the system. Also, it has the benefit of allowing better exploration, as well as the increased explanatory value of the recommendation algorithm.

3. The CourseQ system

In this section, we present CourseQ, a web-based inter

active course recommendation system to help students with diferent information needs to find suitable courses. We first propose a visualization based on a topic model in Section 3.1. Then we describe how we incorporate it as an interactive course recommendation interface in Section 3.2.

3.1. Visualization

Current recommendation systems often produce recommendations that fit well the user’s requirements automatically, trying to reduce the user’s interaction efort and cognitive load [ 3 ]. However, such recommendation systems generally do not allow the user to influence or control the recommendation process, which may lead to filter bubble efects. Also, users may feel too much dominated by the system because it dificult for them to give feedback [ 4 ]. More recently, the potential of interactive recommendation approaches has been highlighted to solve these problems.

Several researchers have proposed interactive visualizations to support interaction with recommendation systems [ 5, 6, 7, 8 ]. Visual representation of information can strongly influence users’ understanding of complex data and help reduce cognitive eforts. Several interactive recommendation systems focus on allowing users to control the recommendation process [ 9, 10 ]. Those applications let users have a more active role to iteratively refine the result set towards their requirements. Their results show that the recommendations are more likely to be accepted by users if the system ofers a higher level of user control. It has also been shown that interactive recommendation systems have the potential to support better exploration [ 11, 12 ], and increase the diversity of content [ 13 ].

Findings from previous works suggest a great benefit of interactive recommendation. However, those works are limited to traditional recommendations such as movie or music, which may significantly difer from the course recommendation in education field.

To understand the relationships of each course and dis

play them in the latent space, we collected data for 380 courses from the syllabus of our university. First, we extracted the text content of collected course data after filtering irrelevant content such as instructor’s name.

Then we used the Latent Dirichlet Allocation (LDA) generative probabilistic model [20] to fit a topic model to the course data collected to give a latent representation for each course. After employing the topic model, we got a k-dimensional vector representation for each course where k is the topic number. The latent representation of course content provides us a convenient way to show the relationships among courses which is an important mea2.2. Course recommendation system surement in our recommendation system. Finally, Linear Discriminant Analysis (LDA) and T-Distributed StochasCourse selection is a critical activity for students in higher tic Neighbor Embedding (T-SNE) were used to reduce education contexts. Various methods have been used the dimensionality of these vectors to the 2D layout. The in applications for course recommendation systems by visualization afects the way the system processes the learning from historical enrollment data. information by displaying the course relevance in two

A related body of work focus on recommending courses dimensions layout. It helps the student to understand the to students that will match their interests [ 14, 15 ]. An- course content according to its topic distribution. Based other set of recommendation method involves mining on the topic model, the interface presents each item (a relationships and discovering sequences from historical course) as a circle node on the canvas in a 2D layout data [ 16, 17 ]. Recently, representation learning uses neu- (Figure 1f). We colored each course node according to ral network architecture has been used in this domain the topics for a visual explanation. Our interface support [18, 19]. However, those systems sufer from several dis- zooming and panning the visualization, the layout also advantages: First, those systems ofer little user interac- could be shifted by the slider (Figure 1g). Recommendation and do not permit students to change their interests. tions are displayed as the corresponding course nodes courses within this department will be shown. Students and their labels are highlighted within the visualization. can explore popular courses for convenience’s sake and We hope that the ability of interactive visualization could it is helpful to figure out the similarity or diferences explain the recommendation results and help students among departments, comprehend the course selection to explore more within the latent space. In terms of pattern, and build their learning path. topic number, we find that too many topics may hard Based on the student’s interest topics associated with to visualize and colorize while too few may cause poor selected keywords, the system recommends courses for performance, as a result, we set 6 as a practical number students and shows them in the latent topic space. Recin our follow-up experiments. It means that a course will ommendations are displayed as the corresponding course be represented by a vector with 6 dimensions. nodes and their labels are highlighted within the visualization. Upon clicking on the node of the recommended 3.2. Interface design course, various information about this course are shown in the right-sidebar (Figure1i), students can explore ofiFigure 1 illustrates the design of the interface. Diferent cial information provided by the university such as course functions that help students interact with the system to period, instructor, date, time, location, and course descripifnd suitable courses are demonstrated at the top of the in- tions. To explain why a course is recommended, we used terface. Students can see all topics and related keywords a grouped bar chart, as seen in Figure 1j, which shows the determined by the topic model respectively in Figure topic distribution of the selected course. The colors of the 1a, and construct their interest by selecting keywords bars match those of the circle nodes from the visualization via a drop-down list (Figure 1b). The keywords that the to show their relations. With the bar charts, students can student selects will be used as a seed for recommenda- compare the topic distributions among diferent courses tions. Besides, they could filter the results based on their to help their decision-making process. Finally, the stuown needs (e.g., the requirement of the degree program, dent can click on the button to like a course or cancel course period, time slot, unit) as shown in Figure 1c. For it as seen in Figure 1k. On the bottom of the interface, example, a student dislike waking up in the early morning Figure 1h, students can see the list of courses they liked. so he/she would like to filter morning classes out when In this part of the interface, they can also click the course exploring the system. On the upper right side, Figure 1d, to check the detailed information or edit their list to genstudents can use a search box with auto-completion to erate personalized results. Every time the student liked ifnd courses. This is suitable for situations when a clear a course while browsing the recommendation result or search goal has been formed. Moreover, we have the de- exploring with the visualization, it will be added into the partment information extracted from historic enrollment like list automatically to calculate the student interest todata in the system (Figure 1e). Upon clicking on one of gether with the selected keywords. Also, students could the buttons that represent diferent departments, popular edit their like list conveniently, which allows them to provide immediate feedback and control the system to baseline interface. Considering the fairness of the comgenerate a more personalized result. parison, we implemented all features as same as CourseQ. Students can search for courses of interest, get recom3.3. Generating recommendations mendations by select keywords, click the recommended course to check details with the information sidebar, and Our system recommends courses based on the student click the button to like and save a course he/she is interinterest corresponding to topic distribution. The stu- ested in. However, to have a better understanding of userdent interest is extracted from the keywords that he/she perceived transparency and experience of exploration, selected and courses he/she liked while exploring the the visualization and filter component are removed from system. To calculate convenience, the vector of courses the baseline interface. The topic distribution component and keywords, based on course content and topic dis- which acts as an explanation for users is not provided in tribution, is stored in two separate data structures. The this interface either. Instead, a ranked list was selected as recommended courses are ranked based on their simi- a traditional way of presenting recommendation results. larity to the student interest. To this end, we computed the Euclidean distance between the vector of student in- 4.1. Participants terest and the vector of each course. The student’s ’like’ list is also important information for the system to give We recruited 32 participants (22 male, 10 female) for the more personalized results. Every time the student ’liked’ user study. The participants are all students who came a course while browsing the recommendation result or from diferent departments of our university, their ages exploring the visualization, it will be added to the ‘like’ ranged from 19 to 28 (M=25.5, SE=0.39). The study was list automatically to calculate the student’s interest to- conducted fully online because of the Covid-19 situation gether with the selected keywords. Also, students could of this year. edit their ’like’ lists conveniently, which allows them to provide immediate feedback and control the system to generate a more personalized result.

4.2. Experimental setup and data collection

4. Evaluation

To evaluate the system in terms of subjective efectiveness and quality, we developed a baseline system as a comparison that uses the same algorithm, values, and dataset as CourseQ. Figure 2 illustrates the design of the

We used online meeting software (Zoom) to communicate with our participants and asked them to access our interfaces by a web browser. The two diferent interfaces were tested in a within-subject design to avoid the influence in the first trial for the second. The first half of the participants will use the CourseQ interface and then use the baseline interface. The other half uses the baseline interface first, and then CourseQ. We asked participants presented a significant diference between the two interto fill in a questionnaire to collect their demographic and faces. The participants tended to interact more with the personal characteristics data. Then we show the intro- visualization in CourseQ (M=65.34) than the ranked list duction of the experiment and the video tutorial of two in the baseline interface (M=12.13). This finding is not interfaces. After that, they were asked to freely interact surprising because the baseline interface lacks the visuwith the interface to find relevant courses (at least five) alization information that pushes the participant to click matching their interests. They could use all features of more to explore within the item space. Moreover, the the respective interface and were not restricted in time. participants tended to interact more with the Information After performing the tasks, participants filled in a ques- sidebar in CourseQ (M=23.2) than the baseline interface tionnaire (5-point Likert scale, 1-completely disagree, 5- (M=7.8). Also, there is a significant diference in the time completely agree), that measured diferent aspects of the spent on the task between CourseQ (M=542.28) and the recommendation system using the ResQue framework baseline interface (M=290.31). This hints that CouresQ [21]. We also collected and analyzed logging data to cap- could serve as an interactive exploration interface that ture user interactions with the various elements of the delivered more interesting information to engage. interface during the experiment. Finally, we conducted a qualitative interview to ask their opinions about two diferent systems. 6. CONCLUSION

5. Preliminary results

5.1. User Feedback

To compare user feedback, we analyzed the results of post-stage questions using paired sample t-tests. Figure 3 presents the diferent aspects of subjective feedback from the participants. CourseQ received a significantly higher rating for four aspects: Perceived Accuracy(Q1), Information Suficiency(Q4), Explanation & Transparency(Q8), and Confidence & Trust(Q11). The baseline scored higher than CourseQ in Perceived Ease of Use(Q6), which is not strange because the richer functionality in CourseQ might cost more efort for participants to use. In other questions, although not significantly, CourseQ scored higher than the baseline.

5.2. Interaction patterns

To better understand the use of the system, we logged the clicks of participants as well as the time they consumed through the task. Table 1 shows the user interaction statistics for two interfaces. Overall, the click frequency

In this paper, we presented CourseQ, a course recommen

dation system by combining visualization technique with recommendation technique to help the exploration and explanation of the recommendation process through an interactive interface.

An online within-subject user study (N=32) was presented to evaluate the interaction and recommendation concept of CourseQ, compared with a baseline system. Our preliminary results show that CourseQ is potentially useful to the students. Also, most participants indicated that they feel confident and trust using CourseQ and will use it again.

There are some limitations to this work that needs to be articulated. The scale of reported user studies is relatively small, and the current gender distribution of participants (more males) may have a gender bias.

For future work, we will analyze user behaviors and feedback for a comprehensive understanding. Moreover, we aim to investigate more sophisticated visualizations to show structure-related topics, for example, show the prerequisite courses.

Acknowledgments

This work was supported by JSPS KAKENHI Grant Numbers JP16H06304, JP20H00622, JP20K19939.

Scholars walk: A markov chain framework for course recommendation., International Educational Data Mining Society (2019). [18] Z. A. Pardos, W. Jiang, Combating the filter bubble: Designing for serendipity in a university course recommendation system, arXiv preprint arXiv:1907.01591 (2019). [19] Z. A. Pardos, Z. Fan, W. Jiang, Connectionist recommendation in the wild: on the utility and scrutability of neural networks for personalized course guidance, User Modeling and User-Adapted Interaction 29 (2019) 487–525. [20] D. M. Blei, A. Y. Ng, M. I. Jordan, Latent dirichlet allocation, the Journal of machine Learning research 3 (2003) 993–1022. [21] P. Pu, L. Chen, R. Hu, A user-centric evaluation framework for recommender systems, in: Proceedings of the fith ACM conference on Recommender systems, 2011, pp. 157–164.

References

[1] B. Ma , Y. Taniguchi , S. Konomi , Course recommendation for university environments ., International Educational Data Mining Society ( 2020 ).

[2] C. He , D. Parra , K. Verbert , Interactive recommender systems: A survey of the state of the art and future research challenges and opportunities , Expert Systems with Applications 56 ( 2016 ) 9 - 27 .

[3] P. Pu , L. Chen , R. Hu , Evaluating recommender systems from the user's perspective: survey of the state of the art, User Modeling and User-Adapted Interaction 22 ( 2012 ) 317 - 355 .

[4] E. Zudilova-Seinstra , T. Adriaansen , R. Van Liere , Overview of interactive visualisation , in: Trends in Interactive Visualization , Springer, London, 2009 , pp. 3 - 15 .

[5] B. Gretarsson , J. O'Donovan , S. Bostandjiev , C. Hall , T. Höllerer, Smallworlds: visualizing social recommendations , in: Computer graphics forum, volume 29 , Wiley Online Library, 2010 , pp. 833 - 842 .

[6] J. O'Donovan , B. Smyth , B. Gretarsson , S. Bostandjiev , T. Höllerer, Peerchooser: visual interactive recommendation , in: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems , 2008 , pp. 1085 - 1088 .

[7] F. Du , C. Plaisant , N. Spring , B. Shneiderman , Finding similar people to guide life choices: Challenge, design, and evaluation , in: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems , 2017 , pp. 5498 - 5544 .

[8] F. Du , C. Plaisant , N. Spring , K. Crowley , B. Shneiderman , Eventaction: A visual analytics approach to explainable recommendation for event sequences , ACM Transactions on Interactive Intelligent Systems (TiiS) 9 ( 2019 ) 1 - 31 .

[9] S. Bostandjiev , J. O'Donovan , T. Höllerer , Tasteweights: a visual interactive hybrid recommender system , in: Proceedings of the sixth ACM conference on Recommender systems , 2012 , pp. 35 - 42 .

[10] Y. Jin , B. Cardoso , K. Verbert , How do diferent levels of user control afect cognitive load and acceptance of recommendations? , in: CEUR Workshop Proceedings , volume 1884 , CEUR Workshop Proceedings , 2017 , pp. 35 - 42 .

[11] I. Andjelkovic , D. Parra , J. O'Donovan , Moodplay: Interactive mood-based music discovery and recommendation , in: Proceedings of the 2016 Conference on User Modeling Adaptation and Personalization , 2016 , pp. 275 - 279 .

[12] F. Gutiérrez , S. Charleer , R. De Croon, N. N. Htun , G. Goetschalckx, K. Verbert , Explaining and exploring job recommendations: a user-driven approach for interacting with knowledge-based job recommender systems , in: Proceedings of the 13th ACM Conference on Recommender Systems , 2019 , pp. 60 - 68 .

[13] C.-H. Tsai , P. Brusilovsky , Beyond the ranked list: User-driven exploration and diversification of social recommendation , in: 23rd international conference on intelligent user interfaces , 2018 , pp. 239 - 250 .

[14] R. Morsomme , S. V. Alferez , Content-based course recommender system for liberal arts education ., International Educational Data Mining Society ( 2019 ).

[15] X. Jing , J. Tang , Guess you like: course recommendation in moocs , in: Proceedings of the International Conference on Web Intelligence , 2017 , pp. 783 - 789 .

[16] S. B. Aher , L. Lobo, Combination of machine learning algorithms for recommendation of courses in e-learning system based on historical data, Knowledge-Based Systems 51 ( 2013 ) 1 - 14 .

[17] A. Polyzou , A. N. Nikolakopoulos , G. Karypis,