=Paper=
{{Paper
|id=Vol-2903/IUI21WS-ESIDA-3
|storemode=property
|title=Exploration and Explanation: An Interactive Course Recommendation System for University Environments
|pdfUrl=https://ceur-ws.org/Vol-2903/IUI21WS-ESIDA-3.pdf
|volume=Vol-2903
|authors=Boxuan Ma,Min Lu,Yuta Taniguchi,Shin’ichi Konomi
|dblpUrl=https://dblp.org/rec/conf/iui/MaLTK21
}}
==Exploration and Explanation: An Interactive Course Recommendation System for University Environments==
https://ceur-ws.org/Vol-2903/IUI21WS-ESIDA-3.pdf
Exploration and Explanation: An Interactive Course
Recommendation System for University Environments
Boxuan Maa , Min Lub , Yuta Taniguchia and Shin’ichi Konomib
a
Kyushu University, Graduate School of Information Science and Electrical Engineering, Fukuoka, Japan
b
Kyushu University, Faculty of Arts and Science, Fukuoka, Japan
Abstract
The abundance of courses available in university and the highly personalized curriculum is often overwhelming for students
who must select courses relevant to their academic interests. A large body of research in course recommendation systems
focuses on optimizing prediction and improving accuracy. However, those systems usually afford little or no user interaction,
and little is known about the influence of user-perceived aspects for course recommendations, such as transparency, con-
trollability, and user satisfaction. In this paper, we argue that involving students in the course recommendation process is
important, and we present an interactive course recommendation system that provides explanations and allows students to
explore courses in a personalized way. A within-subject user study was conducted to evaluate our system and the results
show a significant improvement in many user-centric metrics.
Keywords
Course Recommendation, Visualization, Exploration, Explanation
1. Introduction that align with students’ interests extracted from their his-
torical data, but students may not choose courses based
A course recommendation system suggests a student de- purely on their interests. For instance, many students
cide what they should study as per their requirements, have no idea what they want to study, and their choice
which can solve the increasingly severe problem of infor- of courses is aimless [1]. Besides, student interests and
mation overload of course selection. Different from the goals can change as they explore and learn new things,
traditional movie recommendation domain or music rec- their preferences extracted from historical data may dif-
ommendation domain, the interaction factor is essential fer from their current interests. So, involving the student
for course recommendations in universities. in the recommendation process becomes more significant
Course recommendations in universities particularly than in other domains.
suffer from the cold start problem. Every year, there are Also, the cost to students of making an inappropriate
freshmen enroll in, who have difficulty navigating their decision is much higher than investing two hours watch-
new academic and environment. It is difficult for a tradi- ing a movie they don’t like or listening to a song they are
tional course recommendation system to make successful not interested. In a domain such as a course recommen-
suggestions for those new students without enough avail- dation and learning goal discovery in universities, course
able information. Moreover, the necessary information selection is a low-frequency behavior. Students only need
is often too small to generate precise recommendations to make decisions every new semester for four academic
even for senior students. One common practice is using years. However, it can have a long-lasting effect on the
popular courses regardless of students’ interests when student as improperly selecting courses would seriously
the system is short of students’ information and behav- affect their course achievements, even leads students to
ior. However, a promising alternative is to capture their drop out.
preferences interactively. That is, if we could involve Recently, a large body of research focuses on devel-
students in the recommendation process, we may get oping course recommendation systems. However, those
better results. systems afford little user interaction and lack options to
Many researchers have focused on recommending coursescontrol how recommendations are produced. To address
these challenges which have not been well explored in
Proceedings of the ACM IUI 2021 Workshops, April 13–17, 2021,
College, USA the research community, this work presents an interac-
Envelope-Open ma.boxuan.611@s.kyushu-u.ac.jp (B. Ma); tive course recommendation system by combining visu-
lu@artsci.kyushu-u.ac.jp (M. Lu); yuta.taniguchi.y.t@gmail.com alization techniques with recommendation techniques to
(Y. Taniguchi); konomi@artsci.kyushu-u.ac.jp (S. Konomi) support the diverse information needs of students. The
Orcid 0000-0002-1566-880X (B. Ma); 0000-0001-7503-1301 (M. Lu); interactive feature stresses user involvement with the
0000-0003-3298-8124 (Y. Taniguchi); 0000-0001-5831-2152
(S. Konomi) system, allows users to flexibly explore large-item spaces
© 2021 Copyright for this paper by its authors. Use permitted under Creative
Commons License Attribution 4.0 International (CC BY 4.0).
while providing a high level of user control and trans-
CEUR
Workshop
Proceedings CEUR Workshop Proceedings (CEUR-WS.org)
http://ceur-ws.org
ISSN 1613-0073
parency [2]. Also, our proposed approach could increase
the usability of the course recommendation system com- Second, those systems do not support exploration, which
pared to previous works that only focus on improving is particularly important in the context where students
accuracy. go through a broad exploratory phase before specializing.
Finally, those systems often behave like a “black box” and
do not give explanations that would allow students to
2. Related work reflect on their course selection.
In contrast to the approaches that consider more about
2.1. Interactive recommendation system the accuracy of predicted results, in this work, we build an
Current recommendation systems often produce recom- interactive course recommendation system, which allows
mendations that fit well the user’s requirements auto- students to interactively improve the recommendations
matically, trying to reduce the user’s interaction effort and bring their own preferences to the system. Also, it
and cognitive load [3]. However, such recommendation has the benefit of allowing better exploration, as well as
systems generally do not allow the user to influence or the increased explanatory value of the recommendation
control the recommendation process, which may lead algorithm.
to filter bubble effects. Also, users may feel too much
dominated by the system because it difficult for them to
give feedback [4]. More recently, the potential of interac-
3. The CourseQ system
tive recommendation approaches has been highlighted In this section, we present CourseQ, a web-based inter-
to solve these problems. active course recommendation system to help students
Several researchers have proposed interactive visu- with different information needs to find suitable courses.
alizations to support interaction with recommendation We first propose a visualization based on a topic model
systems [5, 6, 7, 8]. Visual representation of information in Section 3.1. Then we describe how we incorporate
can strongly influence users’ understanding of complex it as an interactive course recommendation interface in
data and help reduce cognitive efforts. Several interac- Section 3.2.
tive recommendation systems focus on allowing users to
control the recommendation process [9, 10]. Those ap-
plications let users have a more active role to iteratively 3.1. Visualization
refine the result set towards their requirements. Their To understand the relationships of each course and dis-
results show that the recommendations are more likely play them in the latent space, we collected data for 380
to be accepted by users if the system offers a higher level courses from the syllabus of our university. First, we
of user control. It has also been shown that interactive extracted the text content of collected course data af-
recommendation systems have the potential to support ter filtering irrelevant content such as instructor’s name.
better exploration [11, 12], and increase the diversity of Then we used the Latent Dirichlet Allocation (LDA) gen-
content [13]. erative probabilistic model [20] to fit a topic model to the
Findings from previous works suggest a great benefit course data collected to give a latent representation for
of interactive recommendation. However, those works each course. After employing the topic model, we got
are limited to traditional recommendations such as movie a k-dimensional vector representation for each course
or music, which may significantly differ from the course where k is the topic number. The latent representation of
recommendation in education field. course content provides us a convenient way to show the
relationships among courses which is an important mea-
2.2. Course recommendation system surement in our recommendation system. Finally, Linear
Discriminant Analysis (LDA) and T-Distributed Stochas-
Course selection is a critical activity for students in higher tic Neighbor Embedding (T-SNE) were used to reduce
education contexts. Various methods have been used the dimensionality of these vectors to the 2D layout. The
in applications for course recommendation systems by visualization affects the way the system processes the
learning from historical enrollment data. information by displaying the course relevance in two
A related body of work focus on recommending courses dimensions layout. It helps the student to understand the
to students that will match their interests [14, 15]. An- course content according to its topic distribution. Based
other set of recommendation method involves mining on the topic model, the interface presents each item (a
relationships and discovering sequences from historical course) as a circle node on the canvas in a 2D layout
data [16, 17]. Recently, representation learning uses neu- (Figure 1f). We colored each course node according to
ral network architecture has been used in this domain the topics for a visual explanation. Our interface support
[18, 19]. However, those systems suffer from several dis- zooming and panning the visualization, the layout also
advantages: First, those systems offer little user interac- could be shifted by the slider (Figure 1g). Recommenda-
tion and do not permit students to change their interests.
Figure 1: The screenshot of the CourseQ. The interface supports the exploration of recommended courses in left and detail
inspection in right. (Some text is in Japanese, and the instructor’s name has been pixelated for privacy protection).
tions are displayed as the corresponding course nodes courses within this department will be shown. Students
and their labels are highlighted within the visualization. can explore popular courses for convenience’s sake and
We hope that the ability of interactive visualization could it is helpful to figure out the similarity or differences
explain the recommendation results and help students among departments, comprehend the course selection
to explore more within the latent space. In terms of pattern, and build their learning path.
topic number, we find that too many topics may hard Based on the student’s interest topics associated with
to visualize and colorize while too few may cause poor selected keywords, the system recommends courses for
performance, as a result, we set 6 as a practical number students and shows them in the latent topic space. Rec-
in our follow-up experiments. It means that a course will ommendations are displayed as the corresponding course
be represented by a vector with 6 dimensions. nodes and their labels are highlighted within the visual-
ization. Upon clicking on the node of the recommended
3.2. Interface design course, various information about this course are shown
in the right-sidebar (Figure1i), students can explore offi-
Figure 1 illustrates the design of the interface. Different cial information provided by the university such as course
functions that help students interact with the system to period, instructor, date, time, location, and course descrip-
find suitable courses are demonstrated at the top of the in- tions. To explain why a course is recommended, we used
terface. Students can see all topics and related keywords a grouped bar chart, as seen in Figure 1j, which shows the
determined by the topic model respectively in Figure topic distribution of the selected course. The colors of the
1a, and construct their interest by selecting keywords bars match those of the circle nodes from the visualization
via a drop-down list (Figure 1b). The keywords that the to show their relations. With the bar charts, students can
student selects will be used as a seed for recommenda- compare the topic distributions among different courses
tions. Besides, they could filter the results based on their to help their decision-making process. Finally, the stu-
own needs (e.g., the requirement of the degree program, dent can click on the button to like a course or cancel
course period, time slot, unit) as shown in Figure 1c. For it as seen in Figure 1k. On the bottom of the interface,
example, a student dislike waking up in the early morning Figure 1h, students can see the list of courses they liked.
so he/she would like to filter morning classes out when In this part of the interface, they can also click the course
exploring the system. On the upper right side, Figure 1d, to check the detailed information or edit their list to gen-
students can use a search box with auto-completion to erate personalized results. Every time the student liked
find courses. This is suitable for situations when a clear a course while browsing the recommendation result or
search goal has been formed. Moreover, we have the de- exploring with the visualization, it will be added into the
partment information extracted from historic enrollment like list automatically to calculate the student interest to-
data in the system (Figure 1e). Upon clicking on one of gether with the selected keywords. Also, students could
the buttons that represent different departments, popular edit their like list conveniently, which allows them to
Figure 2: The screenshot of the baseline application. The interface shows recommended courses in left and detail inspection
in right. a) Keyword input, b) Search bar, c) Ranked list, d) Information sidebar, e) Like button, f) Like list.
provide immediate feedback and control the system to baseline interface. Considering the fairness of the com-
generate a more personalized result. parison, we implemented all features as same as CourseQ.
Students can search for courses of interest, get recom-
3.3. Generating recommendations mendations by select keywords, click the recommended
course to check details with the information sidebar, and
Our system recommends courses based on the student click the button to like and save a course he/she is inter-
interest corresponding to topic distribution. The stu- ested in. However, to have a better understanding of user-
dent interest is extracted from the keywords that he/she perceived transparency and experience of exploration,
selected and courses he/she liked while exploring the the visualization and filter component are removed from
system. To calculate convenience, the vector of courses the baseline interface. The topic distribution component
and keywords, based on course content and topic dis- which acts as an explanation for users is not provided in
tribution, is stored in two separate data structures. The this interface either. Instead, a ranked list was selected as
recommended courses are ranked based on their simi- a traditional way of presenting recommendation results.
larity to the student interest. To this end, we computed
the Euclidean distance between the vector of student in-
4.1. Participants
terest and the vector of each course. The student’s ’like’
list is also important information for the system to give We recruited 32 participants (22 male, 10 female) for the
more personalized results. Every time the student ’liked’ user study. The participants are all students who came
a course while browsing the recommendation result or from different departments of our university, their ages
exploring the visualization, it will be added to the ‘like’ ranged from 19 to 28 (M=25.5, SE=0.39). The study was
list automatically to calculate the student’s interest to- conducted fully online because of the Covid-19 situation
gether with the selected keywords. Also, students could of this year.
edit their ’like’ lists conveniently, which allows them to
provide immediate feedback and control the system to 4.2. Experimental setup and data
generate a more personalized result.
collection
We used online meeting software (Zoom) to communi-
4. Evaluation cate with our participants and asked them to access our
To evaluate the system in terms of subjective effective- interfaces by a web browser. The two different interfaces
ness and quality, we developed a baseline system as a were tested in a within-subject design to avoid the influ-
comparison that uses the same algorithm, values, and ence in the first trial for the second. The first half of the
dataset as CourseQ. Figure 2 illustrates the design of the participants will use the CourseQ interface and then use
the baseline interface. The other half uses the baseline
Figure 3: User feedback analysis results. (Significance level: (*) p < 0.05).
interface first, and then CourseQ. We asked participants presented a significant difference between the two inter-
to fill in a questionnaire to collect their demographic and faces. The participants tended to interact more with the
personal characteristics data. Then we show the intro- visualization in CourseQ (M=65.34) than the ranked list
duction of the experiment and the video tutorial of two in the baseline interface (M=12.13). This finding is not
interfaces. After that, they were asked to freely interact surprising because the baseline interface lacks the visu-
with the interface to find relevant courses (at least five) alization information that pushes the participant to click
matching their interests. They could use all features of more to explore within the item space. Moreover, the
the respective interface and were not restricted in time. participants tended to interact more with the Information
After performing the tasks, participants filled in a ques- sidebar in CourseQ (M=23.2) than the baseline interface
tionnaire (5-point Likert scale, 1-completely disagree, 5- (M=7.8). Also, there is a significant difference in the time
completely agree), that measured different aspects of the spent on the task between CourseQ (M=542.28) and the
recommendation system using the ResQue framework baseline interface (M=290.31). This hints that CouresQ
[21]. We also collected and analyzed logging data to cap- could serve as an interactive exploration interface that
ture user interactions with the various elements of the delivered more interesting information to engage.
interface during the experiment. Finally, we conducted
a qualitative interview to ask their opinions about two
different systems. 6. CONCLUSION
In this paper, we presented CourseQ, a course recommen-
5. Preliminary results dation system by combining visualization technique with
recommendation technique to help the exploration and
5.1. User Feedback explanation of the recommendation process through an
interactive interface.
To compare user feedback, we analyzed the results of An online within-subject user study (N=32) was pre-
post-stage questions using paired sample t-tests. Figure 3 sented to evaluate the interaction and recommendation
presents the different aspects of subjective feedback from concept of CourseQ, compared with a baseline system.
the participants. CourseQ received a significantly higher Our preliminary results show that CourseQ is potentially
rating for four aspects: Perceived Accuracy(Q1), Informa- useful to the students. Also, most participants indicated
tion Sufficiency(Q4), Explanation & Transparency(Q8), that they feel confident and trust using CourseQ and will
and Confidence & Trust(Q11). The baseline scored higher use it again.
than CourseQ in Perceived Ease of Use(Q6), which is There are some limitations to this work that needs
not strange because the richer functionality in CourseQ to be articulated. The scale of reported user studies is
might cost more effort for participants to use. In other relatively small, and the current gender distribution of
questions, although not significantly, CourseQ scored participants (more males) may have a gender bias.
higher than the baseline. For future work, we will analyze user behaviors and
feedback for a comprehensive understanding. Moreover,
5.2. Interaction patterns we aim to investigate more sophisticated visualizations
to show structure-related topics, for example, show the
To better understand the use of the system, we logged the prerequisite courses.
clicks of participants as well as the time they consumed
through the task. Table 1 shows the user interaction
statistics for two interfaces. Overall, the click frequency
Table 1
User interaction statistics (Significance level: (*) p < 0.05)
CourseQ Baseline
Component - Behavior M(SE) M(SE) P-Value
Ranked List - Total Clicks - 12.13(6.76)
Scatter Plot - Total Clicks 65.34(13.55) -
Navigation and Keywords Input - Total Clicks 11.81(2.29) 12.29(3.42)
Search and Filter - Total Clicks 25.22(5.26) 2.9(3.72) *
Department Feature - Total Clicks 7.81(1.71) -
Information and Explanation - Total Clicks 23.22(5.59) 7.8(4.2) *
’Like’ list - Total Clicks 1.63(0.85) 3.2(1.1)
Time Spent - Second 542.28(105.38) 290.31(56.89) *
Acknowledgments sequences, ACM Transactions on Interactive Intel-
ligent Systems (TiiS) 9 (2019) 1–31.
This work was supported by JSPS KAKENHI Grant Num- [9] S. Bostandjiev, J. O’Donovan, T. Höllerer,
bers JP16H06304, JP20H00622, JP20K19939. Tasteweights: a visual interactive hybrid recom-
mender system, in: Proceedings of the sixth ACM
conference on Recommender systems, 2012, pp.
References 35–42.
[1] B. Ma, Y. Taniguchi, S. Konomi, Course recommen- [10] Y. Jin, B. Cardoso, K. Verbert, How do different
dation for university environments., International levels of user control affect cognitive load and ac-
Educational Data Mining Society (2020). ceptance of recommendations?, in: CEUR Work-
[2] C. He, D. Parra, K. Verbert, Interactive recom- shop Proceedings, volume 1884, CEUR Workshop
mender systems: A survey of the state of the art Proceedings, 2017, pp. 35–42.
and future research challenges and opportunities, [11] I. Andjelkovic, D. Parra, J. O’Donovan, Moodplay:
Expert Systems with Applications 56 (2016) 9–27. Interactive mood-based music discovery and recom-
[3] P. Pu, L. Chen, R. Hu, Evaluating recommender mendation, in: Proceedings of the 2016 Conference
systems from the user’s perspective: survey of the on User Modeling Adaptation and Personalization,
state of the art, User Modeling and User-Adapted 2016, pp. 275–279.
Interaction 22 (2012) 317–355. [12] F. Gutiérrez, S. Charleer, R. De Croon, N. N. Htun,
[4] E. Zudilova-Seinstra, T. Adriaansen, R. Van Liere, G. Goetschalckx, K. Verbert, Explaining and explor-
Overview of interactive visualisation, in: Trends in ing job recommendations: a user-driven approach
Interactive Visualization, Springer, London, 2009, for interacting with knowledge-based job recom-
pp. 3–15. mender systems, in: Proceedings of the 13th ACM
[5] B. Gretarsson, J. O’Donovan, S. Bostandjiev, C. Hall, Conference on Recommender Systems, 2019, pp.
T. Höllerer, Smallworlds: visualizing social recom- 60–68.
mendations, in: Computer graphics forum, vol- [13] C.-H. Tsai, P. Brusilovsky, Beyond the ranked list:
ume 29, Wiley Online Library, 2010, pp. 833–842. User-driven exploration and diversification of social
[6] J. O’Donovan, B. Smyth, B. Gretarsson, S. Bostand- recommendation, in: 23rd international conference
jiev, T. Höllerer, Peerchooser: visual interactive on intelligent user interfaces, 2018, pp. 239–250.
recommendation, in: Proceedings of the SIGCHI [14] R. Morsomme, S. V. Alferez, Content-based course
Conference on Human Factors in Computing Sys- recommender system for liberal arts education., In-
tems, 2008, pp. 1085–1088. ternational Educational Data Mining Society (2019).
[7] F. Du, C. Plaisant, N. Spring, B. Shneiderman, Find- [15] X. Jing, J. Tang, Guess you like: course recommen-
ing similar people to guide life choices: Challenge, dation in moocs, in: Proceedings of the Interna-
design, and evaluation, in: Proceedings of the 2017 tional Conference on Web Intelligence, 2017, pp.
CHI Conference on Human Factors in Computing 783–789.
Systems, 2017, pp. 5498–5544. [16] S. B. Aher, L. Lobo, Combination of machine
[8] F. Du, C. Plaisant, N. Spring, K. Crowley, B. Shnei- learning algorithms for recommendation of courses
derman, Eventaction: A visual analytics ap- in e-learning system based on historical data,
proach to explainable recommendation for event Knowledge-Based Systems 51 (2013) 1–14.
[17] A. Polyzou, A. N. Nikolakopoulos, G. Karypis,
Scholars walk: A markov chain framework for
course recommendation., International Educational
Data Mining Society (2019).
[18] Z. A. Pardos, W. Jiang, Combating the filter
bubble: Designing for serendipity in a university
course recommendation system, arXiv preprint
arXiv:1907.01591 (2019).
[19] Z. A. Pardos, Z. Fan, W. Jiang, Connectionist recom-
mendation in the wild: on the utility and scrutabil-
ity of neural networks for personalized course guid-
ance, User Modeling and User-Adapted Interaction
29 (2019) 487–525.
[20] D. M. Blei, A. Y. Ng, M. I. Jordan, Latent dirichlet al-
location, the Journal of machine Learning research
3 (2003) 993–1022.
[21] P. Pu, L. Chen, R. Hu, A user-centric evaluation
framework for recommender systems, in: Proceed-
ings of the fifth ACM conference on Recommender
systems, 2011, pp. 157–164.