=Paper=
{{Paper
|id=Vol-2704/paper4
|storemode=property
|title=A Student-oriented Tool to Support Course Selection in Academic Counseling Sessions
|pdfUrl=https://ceur-ws.org/Vol-2704/paper4.pdf
|volume=Vol-2704
|authors=Jaime Castells,Mohammad Poul Doust,Luis Galárraga,Gonzalo Gabriel Méndez,Margarita Ortiz-Rojas,Alberto Jiménez
|dblpUrl=https://dblp.org/rec/conf/ectel/CastellsDGMOJ20
}}
==A Student-oriented Tool to Support Course Selection in Academic Counseling Sessions==
https://ceur-ws.org/Vol-2704/paper4.pdf
A Student-oriented Tool to Support Course
Selection in Academic Counseling Sessions
Jaime Castells1 , Mohammad Poul Doust2 , Luis Galárraga3 ,
Gonzalo Gabriel Méndez1 , Margarita Ortiz-Rojas1 , and Alberto Jiménez1
1
Escuela Superior Politécnica del Litoral, ESPOL, Guayaquil, Ecuador
{jaime.castells, margarita.ortiz, alberto.jimenez}@cti.espol.edu.ec
gmendez@espol.edu.ec
2
Université Jean Monnet, St. Étienne, France
mohammad.poul.doust@etu.univ-st-etienne.fr
3
INRIA, Rennes, France
luis.galarraga@inria.fr
Abstract. Course selection and recommendation are important aspects
of any academic counseling system. The Learning Analytics community
has long supported these activities via automatic, data-based tools for
recommendation and prediction. We contribute to this body of research
with a tool that allows students to select multiple courses and predict
their academic performance based on historical academic data. The tool
is intended to be used prior to counseling sessions in which students
plan their upcoming semester at the Ecuadorian university ESPOL. This
paper presents the tool’s design and implementation and discusses its
potential to improve the student-counselor discourse.
Keywords: Learning Analytics · Academic Performance Prediction ·
Educational Data mining.
1 Introduction
Advising students throughout their educational career is the main duty of any
university counseling system [14]. An important aspect of academic counseling
is the recommendation of courses to students. This task is usually performed
by a designated counselor who assists students in selecting the most appropriate
courses at a given point of their degree. This is possible thanks to the counselor’s
knowledge of the academic program as well as her ability to make good use of
historical data. Such data can be used to, e.g., identify successful and problematic
academic paths or to characterize the difficulty of the available courses [11,12].
Despite the obvious benefits of academic advising in the mission of univer-
sities, course recommendations made by human counselors are not infallible.
Recommendations can be inconsistent across counselors (with different coun-
selors providing different advice to the same student), and across students (i.e.,
the same counselor can give different advice to students with a similar academic
Copyright c 2020 for this paper by its authors. Use permitted under Creative Com-
mons License Attribution 4.0 International (CC BY 4.0).
history). Recommendations can also be influenced by subjective factors. The per-
ceived difficulty of a given course, for example, may depend on the counselor’s
personal learning/teaching experience.
To cope with some of these issues, recent research by the Learning Ana-
lytics community has proposed solutions to assist counselors in making better
decisions [3,7]. Nevertheless, course recommendation remains a time consuming
activity that often needs to take place within a short period of time [6]. Each
student has a different history, a different set of strengths, and different prefer-
ences that make her unique. This implies that, for every new student, counselors
will likely be confronted to previously unseen scenarios that require a dedicated
analysis. Optimizing this task is of vital importance for universities as it incurs
time and financial resources.
This paper reports the efforts made in this direction at the Ecuadorian uni-
versity ESPOL4 . We address the aforementioned issues with a tool that supports
students in preparation for their academic counseling session and helps them de-
cide which courses to take. We present here a prototype of our tool, which allows
students to define arbitrary sets of available courses and predicts the student’s
performance. We also describe the data-oriented approach we took in designing
our tool. This includes a description of how our prediction models take into ac-
count a student’s own academic history, the performance achieved in the past by
other students, and several difficulty estimators of the courses under considera-
tion. We finally discuss the challenges and opportunities we have so far identified
and present future research directions.
2 Related Work
Learning Analytics and Educational Data Mining focus on taking advantage of
computational tools to collect and analyze educational data [9,15,16]. Learning
Analytics research efforts have sought to support academic programs stakehold-
ers through visual interfaces for several purposes [18]: to understand interactions
within learning environments, to support instructional design, to explore learn-
ing progress, to understand forum discussions, to promote student reflection,
and to identify students at risk of dropping out.
Other contributions from EDM and Machine Learning (ML) focus on the de-
velopment of algorithms that better predict and discover facts from educational
phenomena. Building upon these algorithms, a few tools support academic coun-
seling activities (e.g., [1,4,17]). Notable examples that take a visual approach in
this category include LISSA [3,13], which assists counselors in planning enroll-
ment of first-year students who have previously failed courses. LADA [7] is an-
other counselor-oriented visual dashboard with planning and prediction modules
that uses clustering to estimate a student’s risk of failing chosen courses.
Both LISSA [3,13] and LADA [7] pay a lot of attention to the needs of aca-
demic counselors and policymakers. Less support has been provided to students
within the context of academic counseling. KMCD [19] was a pioneer work in
4
Escuela Superior Politécnica del Litoral (http://www.espol.edu.ec)
this space that sought to help students choose their majors. Our tool shares with
the VLA tools and systems mentioned above, the goal of helping people gain in-
sight from educational data. However, it mainly aims at supporting students to
make more effective decisions. This objective situates our tool within the area of
decision support systems, which seek to “help people overcome their biases and
limitations, and make decisions more knowledgeably and effectively.” [5, p. 3324].
Our tool, on the other hand, takes a visual approach to expedite the course selec-
tion and recommendation aspects of the academic counseling system of an actual
higher education institution (HEI). To this end, it makes extensive use of EDM
and ML techniques to provide personalized academic performance estimations.
3 Case Study: Academic Counseling at ESPOL
The Escuela Superior Politécnica del Litoral (ESPOL) is an engineering-oriented
university in Guayaquil, Ecuador with over 10 000 students in its 32 undergradu-
ate programs. Academic counseling was first established in ESPOL in 1991 with
the purpose of helping students detect their strengths and needs, but it was not
until 2013 that a formal, systematic academic counseling system was deployed.
Counselors are lecturers chosen by workload availability who are assigned,
on average, twenty five students. Counseling sessions take place twice every
semester: right before the semester begins for course-recommendation purposes
and in the middle of the semester to monitor the student’s progress. Each coun-
seling period lasts about two weeks and students must meet with their counselors
for a 15-minute session.
A common issue that we have identified through in-house observations at
ESPOL is that counselors complain about the duration of the counseling session.
They claim sessions hardly take under 15 minutes as students often need more
time to decide on the courses they want to take. This is exacerbated when the
students face particular issues that need to be addressed during the meeting.
Counselors have also reported that students seldom prepare themselves for
the meetings in which they plan their upcoming semester. This lead to rushed
decisions made on the fly, while talking to the counselor. These observations sug-
gest that a system aimed to be used by students before they attend their coun-
seling session could alleviate the counselor’s workload. Ultimately, this would
place more agency on the students’ side regarding their academic choices.
4 Data-oriented Academic Advising
This work makes extensive use of Machine Learning (ML) techniques to generate
models that can accurately predict the performance of a student in a given
course. To this end, we consider the student’s own academic history, her progress
and followed academic path, and her target workload. We first describe the data
used for our analysis (Sec. 4.1). This is followed by a description of the features
that characterize students (Sec. 4.2) and the models we generated (Sec. 4.3).
4.1 Data
We analyzed a dataset containing the academic history of 2 543 Computer Sci-
ence (CS) undergraduate students from ESPOL. This data spans the period
2002–2012. We worked with a simplified version of the CS curriculum previously
studied in [11] and [12]. This academic program is composed of 26 core subjects:
seven basic science courses, sixteen of professional instruction, and three from
the humanities category.
4.2 Feature Selection
Feature selection is the task of selecting the “signals” that provide most infor-
mation about the variable we aim to predict. An example of a feature in our
context is the grade of a student in a given course taken in a previous semester.
Based on our knowledge of the study program and the way students plan
their semesters, we tested a set of features that provide information about the
performance of students in a course. The list shown below describes the final set
of features with which we characterize students:
Previous Grades: If course A is a pre-requisite of course B, it is natural to
assume that the topics covered in A are necessary to understand the topics
of B. This implies that if a student has performed well in A, she will likely
succeed in B. On these grounds, to predict a student’s grade in a given course,
we used as input features her grades in all the corresponding pre-requisites5
as well as in previous trials of the target course. Since students may have
multiple grades associated to a course (in case of failure), we always use the
latest obtained grade as features. The only exception to this rule is the target
course: if the student has grades from previous trials, we take the most recent
of those grades, otherwise we do imputation by considering the average grade
among all students that have passed the target course.
Course and Semester Difficulty Estimators: Some courses are more dif-
ficult or demanding than others, which means that for some of them, students
are more likely to obtain lower marks. Our model considers the difficulty of
a course j through three different features. First, it characterizes the course’s
grading standard (α) and its grading stringency (β) as proposed by Caulkins
et al. [2]. These values are computed according to Equations 1 and 2, where
GPAi represents the overall academic performance of student i; rij is the stu-
dent’s grade in the course; and Nsj is the total number of students that have
taken the course. In general, the bigger the αj (> 1), the more difficult the
course. On the other hand, β estimates how much on average the course j
shifts all students’ grades up or down. Therefore, the smaller the β is, the
easier the coursePcan be considered.
GP Ai 2 P
(GP Ai − rij )
i
αj = P (1) β = i
(2)
(rij ∗ GP Ai ) j
Nsj
i
5
Our results confirm this intuition: The best grade predictor of the performance of a
student in a given course is typically her grade in the course’s closest pre-requisite.
Our model resorts also to the skewness of the distribution (denoted by γ)
produced by the distances between the GPAs of all the students that took a
given course and their corresponding grade in that course. This estimator is
part of the definition of β and has been used in data-driven curricular analy-
ses [11,12]. All these scores are computed from the academic history.
Failing History: The failing history of a student may convey hints about her
skills or attitude towards the topic of a course. Hence, our model defines the
repeating frequency of a student in a given course as the number of times the
student has taken the course before passing it. This feature is equals to zero
when the student intends to take the lecture for the first time.
4.3 Generated Models
We built training sets using the aforementioned features for different courses in
the study program. For courses with few training points, we built training sets
at the level of semesters. We did so by merging the records from courses usually
taken together by students (i.e., they have common features) and adding an
additional feature that identifies the course the record belongs to.
After testing different models such as linear regression, random forests and
gradient boosting trees (GBT), we picked GBT for two reasons. First, it exhibits
the best accuracy. Second, it allows us to deliver lowers and upper bounds for
predictions by means of quantile regression [8]. On the downside GBT models
are not interpretable: one cannot know the exact effect of the input signals on
the model’s answer. This limitation supposes a problem in our scenario because a
mere prediction does not suffice for proper counseling. Counselors always provide
explanations for their predictions in order to support their arguments.
To overcome this shortcoming, we resorted to a post-hoc interpretability
module. SHAP [10] is a explanation layer for ML models that relies on game
theory to calculate the contribution of each feature (called the Shapley value
of the feature) to the answer of a black-box model. SHAP is based on linear
attribution, i.e., if we denote by δy the difference between the grade predicted
by a model and the average grade, the Shapley value of a feature x measures the
contribution of x to δy . Our prototype uses SHAP on top of our GBT prediction
model to show the features that contributed the most to a predicted grade.
5 Our Prototype
We integrated the outcomes of our data mining analyses within an interactive
interface that enables students to plan out their upcoming semesters. The tool
allows to compose sets of courses interactively and predicts for each the range
the student’s grade would fall in.
5.1 Program and Semesters Views
On opening the tool, the Program view shows the student’s career program with
courses organized into a grid and linked to their corresponding pre- and co-
requisites (Fig 1a). Courses appear color-coded according to their type (green
(b) Semesters view
(a) Program view
Fig. 1: Program and semesters view of a student’s academic history.
for basic science, yellow for humanities, and purple for professional training).
The grade of the courses that the student has taken appears in green if the
course has been passed and in red otherwise. This view also shows the number
of times the student took a given course (if greater than one). Courses that have
been repeated are depicted as a group of stacked rectangles.
The Semesters view reorganizes the elements of the Program view to show
individual enrollment instances. This view uncovers the elements of the stacks
that represent repeated courses and depicts when, within her own academic
history, the student took each course. Figure 1b shows the academic history of
the student of Figure 1a, which spans a period of six semesters.
5.2 Prediction Mode
The interface provides a button to enable the prediction mode. Upon activation,
courses in the Program view that the student has already passed are disabled.
This also happens to those courses that the student would not be able to take be-
cause of missing requisites (Fig. 2.A). In the prediction mode, students can select
(or deselect), via clicking, courses they may be interested in taking. Each new
course selection or deselection triggers the execution of our prediction algorithms.
In turn, this updates the content of the Prediction Results view, which shows the
performance prediction for each selected course as a range on a horizontal range
between 0 and 10 (Fig. 2.B). The range corresponds to the confidence interval of
our GBT prediction. We decided against showing here the exact value predicted
by our model as a single point. We wanted to convey the fact that the provided
prediction is an estimate and, as such, it carries some uncertainty. We also felt
that depicting a single value could mislead students and bias their expectations,
which would be counterproductive for the goals we pursue.
The tool shows the prediction results in a red-yellow-green divergent color
scale with a zero value of 6. This is the minimum grade required to pass a course
in ESPOL. Figure 2.B shows the predictions results for a set of four courses.
Fig. 2: Prediction mode. Selecting courses on the Program view (A) results in
the performance predictions shown in (B). The weekly workload is shown in (C).
These results are quite optimistic for two of these courses (shown in the green
part of the range), regular for one course (shown in between orange and green)
and pretty pessimist about another (shown in a redish tone).
An additional view shows a breakdown of the the weekly workload the student
would face due to their current course selection (Fig. 2.C). The load decompo-
sition is shown as a bar chart with three categories: theory, autonomous work,
and practicals. This visualization is also updated by the user’s interactions on
the Program view when in the prediction mode.
5.3 Supporting Understanding of Predicted Results
In addition to predicting the performance that a student could achieve in a given
set of courses, an important goal of our tool is to provide insights on the reasons
behind its predictions.
To this end, our current prototype provides access to the historic distribution
of grades obtained by all the students that have passed a given course (Fig. 3a).
This is possible by moving the mouse pointer onto a course of the Program view
when the prediction mode is active. This visualization allows users to get an
overall impression of a course’s difficulty as it shows how other students from
the same academic program usually perform.
The tool also provides an explanation of the factors that may have a critical
impact in the student’s performance. We provide a visualization of the Shapley
values output by our interpretability module discussed in section 4.3. These
visual explanations are available through the light blue icon shown to the left of
each course on the Prediction Results view (Fig. 2.B).
An example of such visualization for the Human-Computer Interaction (HCI)
course is shown in Figure 3b. Two horizontal bar charts show the ten features
that have the more positive (green) and negative (red) impact on the predicted
(a) Grades historic distribution
(b) Relative impact of features
Fig. 3: Additional views available in the Prediction mode
performance. The features are sorted by their contribution to the prediction
outcome and their name and relative contribution are available through a tooltip
that appears on hovering the bars of the visualization. In the depicted example,
the feature that most contributes positively to the performance in the HCI course
is the grade of Programming Fundamentals. That is, students with a good grade
on the latter course are likely to perform well in HCI. On the other hand, the
total weekly load of the semester assessed is the main factor that could negatively
impact the student performance.
6 Open Questions and Future Work
Designing and implementing our tool posed several questions. In this section, we
discuss some of them and outline directions for future research.
Dealing with curricular modifications: Academic programs are dynamic
by nature. Often, courses are split or merged, some are created by the needs of
the market and others disappear for similar reasons. Tracing these changes is
vital to generate models that are consistent for all the students of an academic
program. Some curricular modifications, however, can be particularly prob-
lematic. Consider the removal of an important predictor course. Estimating
the performance of students who do not have a grade for this course will re-
quire a different model. However, the number of students to whom this change
applies will be small right after the curricular modification is done. This, in
turn, can lead to over-fitting.
Improving explanations of predicted performance: Explainability is
crucial to produce predictions that can be understood by humans. While we
account for this by showing the relative contribution of the features used in
our models, our current implementation is far from being explainable. These
visualizations are not enough to fully open the AI black boxes we use.
One of our goals in this regard is to provide explanations of the predicted
results in natural language. Ideally, this should consider not only the impact
of the most prominent features, but also data on what makes a feature more
or less understandable. This is a promising venue for future work that we will
consider. However, the question of how to translate the models’ output into
fully human understandable language is out of the scope of this paper.
Beyond a simplified program: Our prototype is part of an ongoing effort
to improve and optimize the process of academic counseling at ESPOL. As
reported in this paper, our tool works with a simplified version of the CS aca-
demic program that was valid between 2000 and 2012. Before and after this
period, many curricular modifications took place at ESPOL. So, this limitation
is mainly due to the unavailability of consistent data. Our immediate plans
include to generate prediction models from more recent data. This, however,
is not a trivial task as mapping courses among different versions of the cur-
riculum can be challenging. We thus plan to ask for expertise on the transition
rules that were applied between the performed curricular modifications.
Need of empirical evaluation: Our research aims at supporting a human
decision process via automatic prediction. Thus, assessing the perceived ben-
efits of our prototype both on the students’ and the counselors’ side is of
paramount importance. We plan to run several observational studies to gather
users’ impressions on the system and its usefulness. Pilots in more realistic set-
tings will also be performed. These steps, however, require prediction models
that can be used by current students.
7 Conclusion
This paper presented the design and implementation of a tool that supports
students in preparation for the counseling sessions where they plan their up-
coming semester. Given a selection of courses, the tool estimates the student’s
performance on each course. We described the data-driven approach used to
generate the prediction models based on several factors such as the student’s
own academic history, the performance of other students, and several features
that characterize the difficulty of courses. This research aims at improving and
optimizing ESPOL’s academic counseling system.
8 Acknowledgments
Work funded by the LALA project (grant no. 586120-EPP-1-2017-1-ES-EPPKA2-
CBHE-JP). This project has been funded with support from the European
Commission. This publication reflects only the views of the authors, and the
Commission cannot be held responsible for any use which may be made of the
information contained therein.
References
1. Aguilar, S., Lonn, S., Teasley, S.D.: Perceptions and Use of an Early Warning
System During a Higher Education Transition Program. In: Proceedings of the
Fourth International Conference on Learning Analytics And Knowledge. pp. 113–
117 (2014). https://doi.org/10.1145/2567574.2567625
2. Caulkins, J.P., Larkey, P.D., Wei, J.: Adjusting GPA to Reflect Course Difficulty
(1 1996). https://doi.org/10.1184/R1/6470981.v1
3. Charleer, S., Moere, A.V., Klerkx, J., Verbert, K., Laet, T.D.: Learning Analytics
Dashboards to Support Adviser-Student Dialogue. IEEE Transactions on Learning
Technologies 11(3), 389–399 (2018). https://doi.org/10.1109/TLT.2017.2720670
4. Elcullada-Encarnacion, R.F.: Academic Advising System using Data Mining
Method for Decision Making Support. In: 2018 4th International Conference on
Computer and Technology Applications (ICCTA). pp. 29–34. IEEE (2018)
5. Fox, J.: Decision support systems. In: Smelser, N.J., Baltes, P.B. (eds.) Interna-
tional Encyclopedia of the Social & Behavioral Sciences, pp. 3323 – 3327. Perga-
mon, Oxford (2001). https://doi.org/10.1016/B0-08-043076-7/00554-4
6. Grupe, F.H.: An internet-based expert system for selecting an academic major:
www. mymajors. com. The Internet and higher education 5(4), 333–344 (2002)
7. Gutiérrez, F., Seipp, K., Ochoa, X., Chiluiza, K., Laet, T.D., Verbert, K.: Lada: A
learning analytics dashboard for academic advising. Computers in Human Behavior
(2018). https://doi.org/10.1016/j.chb.2018.12.004
8. Koenker, R.: Quantile Regression. Econometric Society Monographs, Cambridge
University Press (2005). https://doi.org/10.1017/CBO9780511754098
9. Long, P., Siemens, G.: Penetrating the fog: analytics in learning and education.
Italian Journal of Educational Technology 22(3), 132–137 (December 2014)
10. Lundberg, S.M., Lee, S.I.: A unified approach to interpreting model predictions.
In: Advances in Neural Information Processing Systems 30, pp. 4765–4774 (2017)
11. Méndez, G., Ochoa, X., Chiluiza, K.: Techniques for data-driven curriculum anal-
ysis. In: Proceedings of the Fourth International Conference on Learning Analytics
And Knowledge. pp. 148–157 (2014). https://doi.org/10.1145/2567574.2567591
12. Mendez, G., Ochoa, X., Chiluiza, K., De Wever, B.: Curricular design analysis:
A data-driven perspective. Journal of Learning Analytics 1(3), 84–119 (2014).
https://doi.org/10.18608/jla.2014.13.6
13. Millecamp, M., Gutiérrez, F., Charleer, S., Verbert, K., De Laet, T.: A qualitative
evaluation of a learning dashboard to support advisor-student dialogues. In: Pro-
ceedings of the 8th international conference on learning analytics and knowledge.
pp. 56–60. ACM (2018)
14. Mostafa, L., Oately, G., Khalifa, N., Rabie, W.: A case based reasoning system
for academic advising in egyptian educational institutions. In: 2nd International
Conference on Research in Science, Engineering and Technology (ICRSET’2014)
March. pp. 21–22 (2014)
15. Picciano, A.G.: The evolution of big data and learning analytics in american higher
education. Journal of Asynchronous Learning Networks 16(3), 9–20 (2012)
16. Qu, H., Chen, Q.: Visual analytics for mooc data. IEEE Computer Graphics and
Applications 35(6), 69–75 (Nov 2015). https://doi.org/10.1109/MCG.2015.137
17. Roushan, T., Chaki, D., Hasdak, O., Chowdhury, M.S., Rasel, A.A., Rahman,
M.A., Arif, H.: University course advising: Overcoming the challenges using deci-
sion support system. In: 16th Int’l Conf. Computer and Information Technology.
pp. 13–18. IEEE (2014)
18. Vieira, C., Parsons, P., Byrd, V.: Visual learning analytics of educational data: A
systematic literature review and research agenda. Computers & Education 122,
119–135 (2018)
19. Zhou, Q., Yu, F.: Knowledge-based major choosing decision making for remote
students. In: 2008 International Conference on Computer Science and Software En-
gineering. vol. 5, pp. 474–478 (Dec 2008). https://doi.org/10.1109/CSSE.2008.379