Course selection and recommendation are important aspects of any academic counseling system. The Learning Analytics community has long supported these activities via automatic, data-based tools for recommendation and prediction. We contribute to this body of research with a tool that allows students to select multiple courses and predict their academic performance based on historical academic data. The tool is intended to be used prior to counseling sessions in which students plan their upcoming semester at the Ecuadorian university ESPOL. This paper presents the tool's design and implementation and discusses its potential to improve the student-counselor discourse.
Advising students throughout their educational career is the main duty of any university counseling system [14]. An important aspect of academic counseling is the recommendation of courses to students. This task is usually performed by a designated counselor who assists students in selecting the most appropriate courses at a given point of their degree. This is possible thanks to the counselor's knowledge of the academic program as well as her ability to make good use of historical data. Such data can be used to, e.g., identify successful and problematic academic paths or to characterize the di culty of the available courses [11,12].
Despite the obvious bene ts of academic advising in the mission of universities, course recommendations made by human counselors are not infallible. Recommendations can be inconsistent across counselors (with di erent counselors providing di erent advice to the same student), and across students (i.e., the same counselor can give di erent advice to students with a similar academic Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). history). Recommendations can also be in uenced by subjective factors. The perceived di culty of a given course, for example, may depend on the counselor's personal learning/teaching experience.
To cope with some of these issues, recent research by the Learning Analytics community has proposed solutions to assist counselors in making better decisions [3,7]. Nevertheless, course recommendation remains a time consuming activity that often needs to take place within a short period of time [6]. Each student has a di erent history, a di erent set of strengths, and di erent preferences that make her unique. This implies that, for every new student, counselors will likely be confronted to previously unseen scenarios that require a dedicated analysis. Optimizing this task is of vital importance for universities as it incurs time and nancial resources.
This paper reports the e orts made in this direction at the Ecuadorian university ESPOL4. We address the aforementioned issues with a tool that supports students in preparation for their academic counseling session and helps them decide which courses to take. We present here a prototype of our tool, which allows students to de ne arbitrary sets of available courses and predicts the student's performance. We also describe the data-oriented approach we took in designing our tool. This includes a description of how our prediction models take into account a student's own academic history, the performance achieved in the past by other students, and several di culty estimators of the courses under consideration. We nally discuss the challenges and opportunities we have so far identi ed and present future research directions. 2
Learning Analytics and Educational Data Mining focus on taking advantage of computational tools to collect and analyze educational data [9,15,16]. Learning Analytics research e orts have sought to support academic programs stakeholders through visual interfaces for several purposes [18]: to understand interactions within learning environments, to support instructional design, to explore learning progress, to understand forum discussions, to promote student re ection, and to identify students at risk of dropping out.
Other contributions from EDM and Machine Learning (ML) focus on the
development of algorithms that better predict and discover facts from educational
phenomena. Building upon these algorithms, a few tools support academic
counseling activities (e.g., [
Both LISSA [3,13] and LADA [7] pay a lot of attention to the needs of academic counselors and policymakers. Less support has been provided to students within the context of academic counseling. KMCD [19] was a pioneer work in 4 Escuela Superior Politecnica del Litoral (http://www.espol.edu.ec) this space that sought to help students choose their majors. Our tool shares with the VLA tools and systems mentioned above, the goal of helping people gain insight from educational data. However, it mainly aims at supporting students to make more e ective decisions. This objective situates our tool within the area of decision support systems, which seek to \help people overcome their biases and limitations, and make decisions more knowledgeably and e ectively." [5, p. 3324]. Our tool, on the other hand, takes a visual approach to expedite the course selection and recommendation aspects of the academic counseling system of an actual higher education institution (HEI). To this end, it makes extensive use of EDM and ML techniques to provide personalized academic performance estimations. 3
The Escuela Superior Politecnica del Litoral (ESPOL) is an engineering-oriented university in Guayaquil, Ecuador with over 10 000 students in its 32 undergraduate programs. Academic counseling was rst established in ESPOL in 1991 with the purpose of helping students detect their strengths and needs, but it was not until 2013 that a formal, systematic academic counseling system was deployed.
Counselors are lecturers chosen by workload availability who are assigned, on average, twenty ve students. Counseling sessions take place twice every semester: right before the semester begins for course-recommendation purposes and in the middle of the semester to monitor the student's progress. Each counseling period lasts about two weeks and students must meet with their counselors for a 15-minute session.
A common issue that we have identi ed through in-house observations at ESPOL is that counselors complain about the duration of the counseling session. They claim sessions hardly take under 15 minutes as students often need more time to decide on the courses they want to take. This is exacerbated when the students face particular issues that need to be addressed during the meeting.
Counselors have also reported that students seldom prepare themselves for the meetings in which they plan their upcoming semester. This lead to rushed decisions made on the y, while talking to the counselor. These observations suggest that a system aimed to be used by students before they attend their counseling session could alleviate the counselor's workload. Ultimately, this would place more agency on the students' side regarding their academic choices. 4
This work makes extensive use of Machine Learning (ML) techniques to generate models that can accurately predict the performance of a student in a given course. To this end, we consider the student's own academic history, her progress and followed academic path, and her target workload. We rst describe the data used for our analysis (Sec. 4.1). This is followed by a description of the features that characterize students (Sec. 4.2) and the models we generated (Sec. 4.3). 4.1 We analyzed a dataset containing the academic history of 2 543 Computer Science (CS) undergraduate students from ESPOL. This data spans the period 2002{2012. We worked with a simpli ed version of the CS curriculum previously studied in [11] and [12]. This academic program is composed of 26 core subjects: seven basic science courses, sixteen of professional instruction, and three from the humanities category. 4.2
Feature selection is the task of selecting the \signals" that provide most information about the variable we aim to predict. An example of a feature in our context is the grade of a student in a given course taken in a previous semester.
Based on our knowledge of the study program and the way students plan their semesters, we tested a set of features that provide information about the performance of students in a course. The list shown below describes the nal set of features with which we characterize students:
Previous Grades: If course A is a pre-requisite of course B, it is natural to assume that the topics covered in A are necessary to understand the topics of B. This implies that if a student has performed well in A, she will likely succeed in B. On these grounds, to predict a student's grade in a given course, we used as input features her grades in all the corresponding pre-requisites5 as well as in previous trials of the target course. Since students may have multiple grades associated to a course (in case of failure), we always use the latest obtained grade as features. The only exception to this rule is the target course: if the student has grades from previous trials, we take the most recent of those grades, otherwise we do imputation by considering the average grade among all students that have passed the target course.
Course and Semester Di culty Estimators: Some courses are more difcult or demanding than others, which means that for some of them, students are more likely to obtain lower marks. Our model considers the di culty of a course j through three di erent features. First, it characterizes the course's grading standard ( ) and its grading stringency ( ) as proposed by Caulkins et al. [2]. These values are computed according to Equations 1 and 2, where GPAi represents the overall academic performance of student i; rij is the student's grade in the course; and Nsj is the total number of students that have taken the course. In general, the bigger the j (> 1), the more di cult the course. On the other hand, estimates how much on average the course j shifts all students' grades up or down. Therefore, the smaller the is, the easier the course can be considered.
j = Pi (PriijGPGAPi2Ai) (1) j = Pi (GPNAsji rij ) (2) 5 Our results con rm this intuition: The best grade predictor of the performance of a student in a given course is typically her grade in the course's closest pre-requisite. Our model resorts also to the skewness of the distribution (denoted by ) produced by the distances between the GPAs of all the students that took a given course and their corresponding grade in that course. This estimator is part of the de nition of and has been used in data-driven curricular analyses [11,12]. All these scores are computed from the academic history. Failing History: The failing history of a student may convey hints about her skills or attitude towards the topic of a course. Hence, our model de nes the repeating frequency of a student in a given course as the number of times the student has taken the course before passing it. This feature is equals to zero when the student intends to take the lecture for the rst time. 4.3
We built training sets using the aforementioned features for di erent courses in the study program. For courses with few training points, we built training sets at the level of semesters. We did so by merging the records from courses usually taken together by students (i.e., they have common features) and adding an additional feature that identi es the course the record belongs to.
After testing di erent models such as linear regression, random forests and gradient boosting trees (GBT), we picked GBT for two reasons. First, it exhibits the best accuracy. Second, it allows us to deliver lowers and upper bounds for predictions by means of quantile regression [8]. On the downside GBT models are not interpretable: one cannot know the exact e ect of the input signals on the model's answer. This limitation supposes a problem in our scenario because a mere prediction does not su ce for proper counseling. Counselors always provide explanations for their predictions in order to support their arguments.
To overcome this shortcoming, we resorted to a post-hoc interpretability module. SHAP [10] is a explanation layer for ML models that relies on game theory to calculate the contribution of each feature (called the Shapley value of the feature) to the answer of a black-box model. SHAP is based on linear attribution, i.e., if we denote by y the di erence between the grade predicted by a model and the average grade, the Shapley value of a feature x measures the contribution of x to y. Our prototype uses SHAP on top of our GBT prediction model to show the features that contributed the most to a predicted grade.
We integrated the outcomes of our data mining analyses within an interactive interface that enables students to plan out their upcoming semesters. The tool allows to compose sets of courses interactively and predicts for each the range the student's grade would fall in. 5.1
On opening the tool, the Program view shows the student's career program with courses organized into a grid and linked to their corresponding pre- and corequisites (Fig 1a). Courses appear color-coded according to their type (green (b) Semesters view (a) Program view for basic science, yellow for humanities, and purple for professional training). The grade of the courses that the student has taken appears in green if the course has been passed and in red otherwise. This view also shows the number of times the student took a given course (if greater than one). Courses that have been repeated are depicted as a group of stacked rectangles.
The Semesters view reorganizes the elements of the Program view to show individual enrollment instances. This view uncovers the elements of the stacks that represent repeated courses and depicts when, within her own academic history, the student took each course. Figure 1b shows the academic history of the student of Figure 1a, which spans a period of six semesters. 5.2
The interface provides a button to enable the prediction mode. Upon activation, courses in the Program view that the student has already passed are disabled. This also happens to those courses that the student would not be able to take because of missing requisites (Fig. 2.A). In the prediction mode, students can select (or deselect), via clicking, courses they may be interested in taking. Each new course selection or deselection triggers the execution of our prediction algorithms. In turn, this updates the content of the Prediction Results view, which shows the performance prediction for each selected course as a range on a horizontal range between 0 and 10 (Fig. 2.B). The range corresponds to the con dence interval of our GBT prediction. We decided against showing here the exact value predicted by our model as a single point. We wanted to convey the fact that the provided prediction is an estimate and, as such, it carries some uncertainty. We also felt that depicting a single value could mislead students and bias their expectations, which would be counterproductive for the goals we pursue.
The tool shows the prediction results in a red-yellow-green divergent color scale with a zero value of 6. This is the minimum grade required to pass a course in ESPOL. Figure 2.B shows the predictions results for a set of four courses. These results are quite optimistic for two of these courses (shown in the green part of the range), regular for one course (shown in between orange and green) and pretty pessimist about another (shown in a redish tone).
An additional view shows a breakdown of the the weekly workload the student would face due to their current course selection (Fig. 2.C). The load decomposition is shown as a bar chart with three categories: theory, autonomous work, and practicals. This visualization is also updated by the user's interactions on the Program view when in the prediction mode. 5.3
In addition to predicting the performance that a student could achieve in a given set of courses, an important goal of our tool is to provide insights on the reasons behind its predictions.
To this end, our current prototype provides access to the historic distribution of grades obtained by all the students that have passed a given course (Fig. 3a). This is possible by moving the mouse pointer onto a course of the Program view when the prediction mode is active. This visualization allows users to get an overall impression of a course's di culty as it shows how other students from the same academic program usually perform.
The tool also provides an explanation of the factors that may have a critical impact in the student's performance. We provide a visualization of the Shapley values output by our interpretability module discussed in section 4.3. These visual explanations are available through the light blue icon shown to the left of each course on the Prediction Results view (Fig. 2.B).
An example of such visualization for the Human-Computer Interaction (HCI) course is shown in Figure 3b. Two horizontal bar charts show the ten features that have the more positive (green) and negative (red) impact on the predicted (a) Grades historic distribution (b) Relative impact of features performance. The features are sorted by their contribution to the prediction outcome and their name and relative contribution are available through a tooltip that appears on hovering the bars of the visualization. In the depicted example, the feature that most contributes positively to the performance in the HCI course is the grade of Programming Fundamentals. That is, students with a good grade on the latter course are likely to perform well in HCI. On the other hand, the total weekly load of the semester assessed is the main factor that could negatively impact the student performance.
Designing and implementing our tool posed several questions. In this section, we discuss some of them and outline directions for future research.
Dealing with curricular modi cations: Academic programs are dynamic by nature. Often, courses are split or merged, some are created by the needs of the market and others disappear for similar reasons. Tracing these changes is vital to generate models that are consistent for all the students of an academic program. Some curricular modi cations, however, can be particularly problematic. Consider the removal of an important predictor course. Estimating the performance of students who do not have a grade for this course will require a di erent model. However, the number of students to whom this change applies will be small right after the curricular modi cation is done. This, in turn, can lead to over- tting.
Improving explanations of predicted performance: Explainability is crucial to produce predictions that can be understood by humans. While we account for this by showing the relative contribution of the features used in our models, our current implementation is far from being explainable. These visualizations are not enough to fully open the AI black boxes we use. One of our goals in this regard is to provide explanations of the predicted results in natural language. Ideally, this should consider not only the impact of the most prominent features, but also data on what makes a feature more or less understandable. This is a promising venue for future work that we will consider. However, the question of how to translate the models' output into fully human understandable language is out of the scope of this paper. Beyond a simpli ed program: Our prototype is part of an ongoing e ort to improve and optimize the process of academic counseling at ESPOL. As reported in this paper, our tool works with a simpli ed version of the CS academic program that was valid between 2000 and 2012. Before and after this period, many curricular modi cations took place at ESPOL. So, this limitation is mainly due to the unavailability of consistent data. Our immediate plans include to generate prediction models from more recent data. This, however, is not a trivial task as mapping courses among di erent versions of the curriculum can be challenging. We thus plan to ask for expertise on the transition rules that were applied between the performed curricular modi cations. Need of empirical evaluation: Our research aims at supporting a human decision process via automatic prediction. Thus, assessing the perceived bene ts of our prototype both on the students' and the counselors' side is of paramount importance. We plan to run several observational studies to gather users' impressions on the system and its usefulness. Pilots in more realistic settings will also be performed. These steps, however, require prediction models that can be used by current students.
This paper presented the design and implementation of a tool that supports students in preparation for the counseling sessions where they plan their upcoming semester. Given a selection of courses, the tool estimates the student's performance on each course. We described the data-driven approach used to generate the prediction models based on several factors such as the student's own academic history, the performance of other students, and several features that characterize the di culty of courses. This research aims at improving and optimizing ESPOL's academic counseling system.
Work funded by the LALA project (grant no. 586120-EPP-1-2017-1-ES-EPPKA2CBHE-JP). This project has been funded with support from the European Commission. This publication re ects only the views of the authors, and the Commission cannot be held responsible for any use which may be made of the information contained therein. 2. Caulkins, J.P., Larkey, P.D., Wei, J.: Adjusting GPA to Re ect Course Di culty (1 1996). https://doi.org/10.1184/R1/6470981.v1 3. Charleer, S., Moere, A.V., Klerkx, J., Verbert, K., Laet, T.D.: Learning Analytics Dashboards to Support Adviser-Student Dialogue. IEEE Transactions on Learning Technologies 11(3), 389{399 (2018). https://doi.org/10.1109/TLT.2017.2720670 4. Elcullada-Encarnacion, R.F.: Academic Advising System using Data Mining Method for Decision Making Support. In: 2018 4th International Conference on Computer and Technology Applications (ICCTA). pp. 29{34. IEEE (2018) 5. Fox, J.: Decision support systems. In: Smelser, N.J., Baltes, P.B. (eds.) International Encyclopedia of the Social & Behavioral Sciences, pp. 3323 { 3327. Pergamon, Oxford (2001). https://doi.org/10.1016/B0-08-043076-7/00554-4 6. Grupe, F.H.: An internet-based expert system for selecting an academic major: www. mymajors. com. The Internet and higher education 5(4), 333{344 (2002) 7. Gutierrez, F., Seipp, K., Ochoa, X., Chiluiza, K., Laet, T.D., Verbert, K.: Lada: A learning analytics dashboard for academic advising. Computers in Human Behavior (2018). https://doi.org/10.1016/j.chb.2018.12.004 8. Koenker, R.: Quantile Regression. Econometric Society Monographs, Cambridge
University Press (2005). https://doi.org/10.1017/CBO9780511754098 9. Long, P., Siemens, G.: Penetrating the fog: analytics in learning and education.
Italian Journal of Educational Technology 22(3), 132{137 (December 2014) 10. Lundberg, S.M., Lee, S.I.: A uni ed approach to interpreting model predictions.
In: Advances in Neural Information Processing Systems 30, pp. 4765{4774 (2017) 11. Mendez, G., Ochoa, X., Chiluiza, K.: Techniques for data-driven curriculum analysis. In: Proceedings of the Fourth International Conference on Learning Analytics And Knowledge. pp. 148{157 (2014). https://doi.org/10.1145/2567574.2567591 12. Mendez, G., Ochoa, X., Chiluiza, K., De Wever, B.: Curricular design analysis: A data-driven perspective. Journal of Learning Analytics 1(3), 84{119 (2014). https://doi.org/10.18608/jla.2014.13.6 13. Millecamp, M., Gutierrez, F., Charleer, S., Verbert, K., De Laet, T.: A qualitative evaluation of a learning dashboard to support advisor-student dialogues. In: Proceedings of the 8th international conference on learning analytics and knowledge. pp. 56{60. ACM (2018) 14. Mostafa, L., Oately, G., Khalifa, N., Rabie, W.: A case based reasoning system for academic advising in egyptian educational institutions. In: 2nd International Conference on Research in Science, Engineering and Technology (ICRSET'2014) March. pp. 21{22 (2014) 15. Picciano, A.G.: The evolution of big data and learning analytics in american higher education. Journal of Asynchronous Learning Networks 16(3), 9{20 (2012) 16. Qu, H., Chen, Q.: Visual analytics for mooc data. IEEE Computer Graphics and
Applications 35(6), 69{75 (Nov 2015). https://doi.org/10.1109/MCG.2015.137 17. Roushan, T., Chaki, D., Hasdak, O., Chowdhury, M.S., Rasel, A.A., Rahman, M.A., Arif, H.: University course advising: Overcoming the challenges using decision support system. In: 16th Int'l Conf. Computer and Information Technology. pp. 13{18. IEEE (2014) 18. Vieira, C., Parsons, P., Byrd, V.: Visual learning analytics of educational data: A systematic literature review and research agenda. Computers & Education 122, 119{135 (2018) 19. Zhou, Q., Yu, F.: Knowledge-based major choosing decision making for remote students. In: 2008 International Conference on Computer Science and Software Engineering. vol. 5, pp. 474{478 (Dec 2008). https://doi.org/10.1109/CSSE.2008.379