Choosing appropriate courses for a semester is a challenging task for undergraduate students. To facilitate the course selection process, diferent course recommendation systems have been proposed implementing diferent machine learning algorithms and techniques. At the same time, recently, there has been a rapid development of Large Language Models (LLMs) (e.g., GPT4, Llama3, and Gemini), which have been adopted in numerous applications and have influenced all walks of life. In this paper, we explore their potential to assist stakeholders in higher education with course recommendation. We explore two diferent ways to directly (and ofline, to ensure no sensitive data leakage) generate recommendations from pre-trained or fine-tuned LLM models. We also propose a novel ChatGPT-assisted course recommendation model (GPTaCR) which follows a diferent methodology. It utilizes the output of ChatGPT to form rules that capture relationships among courses. Based on these rules, we generate a set of courses for every student to take next, given their prior enrolment history. We use a real-world dataset to evaluate the performance of our proposed models compared to other relevant course recommendation approaches. The findings highlight that our proposed manages to best combine the rich knowledge base of ChatGPT with information about past students' enrollment history. We hope that this work can be a source of motivation for researchers to look into how LLMs might improve recommendation performance and other educational data mining tasks.
Use LLMs for EDM
In higher education, one of the dificulties faced by students
is to select a set of courses every semester and balance the
course workload within a semester. While selecting courses,
they need to consider degree requirements, prerequisites
of courses, their interests and career choices, and which
courses will build their base knowledge to take more
advanced courses in the future. So, choosing a good set of
courses is a non-trivial task for students. They may get
assistance from the departmental course catalog or other
senior students. However, their insights might not be
personalized to the preferences, experience, and background
of a particular student. Academic advisors can also help,
but in most institutions, the high ratio of students to
advisors limits the time and dedication an advisor can devote
to a single student, consequently limiting the usefulness of
their interactions [
The development of data mining and machine learning
models can assist student advising by generating
personalized course recommendations for each student based on
insightful analysis of historical data records of past
students. For example, researchers propose diferent course
recommendation systems building Markov chains [
The fast development of large language models (LLMs)
and their chatbot prototypes like ChatGPT3.5 and
ChatGPT4 has increased the potential to use NLP approaches for
∗Corresponding author.
ics assessment questions [
In particular, we explore diferent ways to use LLMs for
the task of course recommendation. We start by using
simple, ofline, pre-trained models directly for course
recommendation. Next, we consider fine-tuning strategies for
simpler LLMs to enrich the knowledge base with student
enrolment data. Next, inspired by the association rule mining
technique introduced in [
Researchers propose diferent course recommendation
systems (CRS) for university students by collecting historical
course enrollment data from warehouses [
For course recommendation in university environments
using enrollment data, an association rule mining algorithm
(Apriori) has been proposed to capture the association of
all courses taken by each student1[
Whereas, NLP approaches focus more on textual
information (e.g., course descriptions) to calculate similarities
between courses by analyzing the topics and knowledge
components each course covers. Term Frequency–Inverse
Document Frequency (TF–IDF) is the most common
technique to analyze course descriptions and find similar courses
by analyzing how many times a word appears in the
document of each course [
Language models have been applied and explored in
education settings for diferent topics such as providing
personalized feedback and assistance to students2[
First, we use an open-source pre-trained base LLM model to generate recommendations. In this approach, we use the Llama-3-8B-Instruct model1 provided by hugging face2 to get inference for our data. Here, we do not use any historical course enrollment data for re-training or fine-tuning the model. We just provide all the course codes with course names and prior course registration history of a student as one instance (input) and ask the Llama3 model (ofline prompting running codes) to recommend a set of courses (output) by analyzing the provided information. An example of the instruction and input format for a target student is depicted in Figure 1.
We cannot generate recommendations from the
opensource GPT2 [
Alternatively, we can fine-tune LLMs to enrich their
knowledge with historical data relevant to our specific task at
hand. A similar approach has also been proposed for item
recommendation [
Data preparation: Following the prior work for item
recommendation [
(last) semester. We also provide course descriptions as input to fine-tune the model to capture the semantic similarity of courses as described in 4[0] where there is no output (empty string) for those texts.
Environmental set-up: We use the transformer library
of hugging face4 to access the base pre-trained language
models (Llama-3-8B and GPT2-1.5B). We use Autotokenizer
to tokenize each word to fine-tune the Llama3 model and
GPT2tokenizer for the GPT2 model. We use BitsandBytes
and parameter-eficient fine-tuning (PEFT) libraries for
finetuning eficiently by reducing computing requirements.
PEFT library provides LoraConfig to use QLoRa [
Train model and inference We define SFTTrainer with training arguments that include which layers of the base model to apply the adapters (we choose the attention layer because we want to generate attention scores for each course to recommend and update the parameters accordingly) and diferent hyperparameters like number of epochs, learning rate, etc. Then we train the base model with our dataset and use the fine-tuned LLM model for inference. From validation and test sets, we provide a target student’s prior courses as input, keep the output empty, and then generate recommended course list as output from the fine-tuned model.
The next step we could explore is to use a more recent and powerful LLM, e.g., ChatGPT4. However, we cannot enter students’ registration history in ChatGPT4, as that would violate student privacy, so we introduce two alternative approaches. Inspired by the association rule mining (ARM) approaches for course recommendation, we want to generate rules in the form of → , where represents a prior course or frequent sets of courses (course sets) and represents the output collected by ChatGPT4 when asked to suggest future courses for a student that has taken the courses in . Based on the overlap of with the past courses that a student has taken, we generate the recommendations in a similar manner as in ARM. Our GPTaCR model and its steps are illustrated in Figure3.
In the GPTaCR-1C approach, we provide one course as a prior course of a student and ask ChatGPT4 to suggest a set of courses that are relevant to this course by analyzing provided contexts (course names, acronyms, and descriptions). In this way, we generate suggestions for all the available
courses which means we get618 rules for 618 courses in our dataset.
For the GPTaCR-FCS model, we use the students’ course registration history to generate frequent course sets. We consider courses to be the items, and each student’s course history over all the semesters is one transaction. Based on all transactions in the training data, we use the concept of ARM to generate frequent itemsets (FI), i.e., the course sets with support greater or equal to a minimum support threshold. Here, each FI consists of one or more courses frequently taken together. We also generate and tested the maximal frequent itemsets (MFI) where all the subsets of a frequent itemset are removed (i.e., if {c1, c2, c3} is a MFI, we do not consider its subsets, i.e., {c1, c2}, {c2, c3}).
While the number of FI (or even of MFI) that we get from the previous step can be quite high, we need to reduce them and make it more practical for us to test those course sets with ChatGPT. At the same time, it is desirable for the course sets to cover as many unique courses as possible. As a result, we cannot simply use a higher minimum support threshold to get fewer FI, as in this case, the coverage of courses will be very poor. What we do is randomly sample course sets from FI or MFI to use them as input in ChatGPT4 prompts. While sampling, we have almost as many unique courses as in the sampled frequent itemsets and the average length of itemsets remains similar to FI or MFI.
Prompt ChatGPT and form rules: For each possible
course set of the previous step, we use ChatGPT4 [
A rule gets activated if its antecedent part is matched (using a % of match threshold) by a student’s prior courses. From each activated rule, we examine two possibilities regarding courses that we could recommend: first, consider only the courses in consequent sides of the activated rules, or secondly, also consider the unmatched courses in antecedent part of the activated rules.
Scoring rules: We use three diferent ways to calculate the score of a course to recommend. (i) We use the confidence values estimated by ChatGPT4 for a rule to compute the recommendation score for each course of the activated rules. (ii) We count how many times an unmatched course appears on any side of the activated rules and the total count is the recommendation score for that course. We denote this scoring rule as the baseline-counting. (iii) We use the matching ratio and a parameter () to indicate from which side of the activated rule an unmatched course is recommended to compute the recommendation score of that course. The formula for calculating the score of a recommended course from any side of the rule is as follows:
+ match ratio∗ if ∈ RHS
Score ( ) = {
+ match ratio∗ (1 − ) if ∈ LHS
(1)
where is the unmatched course in an activated rule ,∈
[
Rule-based course recommendation: After using any of the scoring methods, we have a recommendation score for each course. We finally recommend the courses with the highest recommendation scores. If no rule gets activated for a student, we recommend popular courses which were not been taken by that student before.
In ChatGPT4, we create a customized GPT4 chat, write instructions, and examine two cases when providing different contextual information, as shown in4. In the first case,CNames, we provide only the available list of courses from our training data (i.e., course codes and their names) as context, and ask for suggestions. In the second case, CNames+Desc, we provide additional context by uploading separate files in the knowledge part: the available list of courses, definitions of diferent acronyms of courses, and available course descriptions. In both methods, we provide 15 students’ information at a time, generate suggestions for them and collect the rules. Under additional chat settings in ChatGPT4, there is an option to use conversational data to improve the model which we unselect, as recommendations for later students may get biased and ambiguous by previous course sets and suggested courses from ChatGPT4.
We collected a real-world dataset from Florida International University, a public university in the US, that includes the course enrollment history of undergraduate students of the Computer Science Department for nearly nine years. As our recommendation should represent good practices, we solely take into account the information of students who have earned a degree, and we eliminate instances where a student had a grade lower than a C in a course. We also eliminate any dropped courses from students’ histories. Courses that show up in our dataset fewer than three times are eliminated. Following preprocessing, we have3328 students’ course enrollment histories and647 distinct courses. Then, we split the dataset into 3 sets: training, validation, and test sets, with 2973, 1231, 657 students and 618, 540, 494 unique courses, respectively. For testing, we use the last three semesters and the preceding three semesters are used for validation and model selection. The remaining semesters are retained in the training set. In the validation and test sets, we eliminate the courses that are not available in the training set because some models can not recommend these new courses. Moreover, we eliminate any students who took courses in less than three semesters, as we require the records from at least the last two semesters in order to produce recommendations for a student. Finally, there are 618 unique courses retained in training, validation, and test sets. Each student may be associated with more than one instance, one for each potential target semester with a prior history length of at least 2 semesters. Additionally, we collect course names, descriptions, and definitions of acronyms for each course in our dataset from the university’s undergraduate catalogs.
For fine-tuned LLM models, we follow the experimental setup as described in the Subsection3.2. The CRwLLM and CRwFine methods did not have any external hyperparameters that we needed to tune.
For the GPTaCR-1C model, we explore diferent numbers
of suggested courses from ChatGPT4 = [
For the GPTaCR-FCS model, we repeat our experiments three times and report the average statistics and measures in this paper, as we use sampling to limit the number of course sets. We do so to capture any variability of the results caused by the sampling and to ensure that the results are accurate and representative.
For our first step, we apply the Apriori algorithm for
frequent itemset generation from mlxtend [
In the last step, using the rules, we generate
recommendations and we explore diferent parameters that afect the
process. We test the percentage of match threshold = [
Course recommendation: For all the models, we
recommend the courses with higher recommendation scores
for the target semester. Following prior work [
As in prior work [
We evaluate our models compared to other baselines and competing approaches that similarly use relevant data. For example, we do not consider methods that might use grading information. We particularly focus on comparing competing approaches that process the textual information in some form.
PopInTerm We implement a popularity-based approach
as the baseline for course recommendation 4[
Association Rule Mining (ARM) As described in prior
work [
TF-IDF We implement the TF-IDF model using the course
descriptions for all courses to recommend similar and
relevant courses [
PLANBERT We re-implement the PLANBERT model
which is also a language model proposed in 7[]. In this
model, we do not use any future reference courses like the
proposed model, only use the course enrollment data to
train the model to capture historical course enrollment
patterns. We make one sentence with all the courses taken by a
student to prepare each instance of the training data. Then,
we use the pre-trained DistilBERT model from the Hugging
Face5 and fine-tune it with our data. Using the data collator
for Language Modeling6, we tokenize the words including
some masked tokens where = [
Table 3 presents the performance comparison between CRwLLM, CRwFine, GPTaCR models, and other existing approaches. Our proposed GPTaCR-FCS model (which uses the CNames+Desc context) outperforms the other approaches in all the metrics. It utilizes the power of ChatGPT for textual analysis. GPTaCR-FCS also performs better than the ARM and PLANBERT models for the test data providing slightly better recall (0.298) and %1+ rel scores (58.14%). ARM is already a well-established algorithm that captures all the associations among items present in the historical data since it considers all the high-confidence rules. In our approach, we partially and indirectly consider student enrollment data (only through the frequent course sets used). The method takes into account enrollment patterns. We can recommend the unmatched courses from the left-hand side of the rule, which includes the frequent course sets.
We observe a fine-tuned LLM model, CRwFine(llama) provides a better validation recall score than the GPTaCR model when we use only course enrollment data to re-train the base model. However, at the same time, the test performance 5https://huggingface.co/learn/nlpcourse/chapter7/3?fw=tf 6https://huggingface.co/docs/transformers/main_classes /data_collator#transformers.default_data_collator is significantly lower, indicating that the fine-tuned recommendations do not generalize well for unseen instances. Interestingly, when we use both course enrollment and description data in the fine-tuning process, the performance of both fine-tuned models degrades. This happens because for each instance, we have much more text, and the model might have trouble focusing on the most relevant information.
Overall, while we partially use student enrolment data and provide limited course information, ChatGPT4 can build reasonable recommendations by taking advantage of its vast knowledge base. Fine-tuning an LLM is a potential solution to utilize all the course enrollment data by using them oflfine preserving the privacy of students but we might need more powerful LLMs to handle all the relevant information. The performance of our proposed models serves as a proof of concept and a first look into LLM’s capabilities in recommendation tasks in education.
Since the GPTaCR models perform the best, we further delve into how they are afected by diferent hyperparameters. We present the efect of diferent hyperparameters of GPTaCR1C model in Table 4. We observe best test Recall@ and %1+ rel scores by asking ChatGPT4 to suggest 4 courses, all using all the history of a student to match with the one prior course (LHS) of a rule and scoring based on course-taking semester. We observe the biggest variability for the size of the right-hand side, while the scoring method does not dramatically change the results.
We present the efects of diferent hyperparameters of the GPTaCR-FCS model in Table5. In the second row, we present the best set of hyperparameters for whom we get the best Recall@ and %1+ rel scores. The performance observed refers to the case when we use FI samples rather than MFI samples. In the latter case, we get test recall 0.277 and %1+ rel 54.83%. One reason is that students manage to activate more rules in FI samples rather than in MFI, resulting in more reliable recommendations in the end. Moving forward, we will only present results and discuss approaches that sample from FI.
In the other rows of Table5, we present the best scores for each parameter value where empty cells indicate the best possible hyperparameters for the parameter noted in that row. First, we can see that we get the best scores when we use more contexts (course names, acronyms, and descriptions) than only using course names. The improvement in performance though is quite limited, indicating that ChatGPT4 uses its existing knowledge to infer the topics covered by only using the course names. As a result, when we add the course descriptions, we do not get a significant improvement.
Second, we observe better performance when we use more samples (best validation recall using 700 samples and best test recall using 600 samples). It is understandable that if we use fewer samples (and consequently, fewer rules), we get more instances with few or no activated rules, leading to unreliable recommendations. Third, we get the best scores with 3 courses suggested by ChatGPT4, while the other values have similar performance. A total of 3 courses in a Valid 0.342 semester are also commonly taken by the students of our dataset. When we ask ChatGPT4 to suggest more than 3 courses, it may suggest some less relevant courses and overall recall scores become lower.
Fourth, we can see better performance when using the lower percentage of thresholds (20, 30, 40) of matching the antecedents of the rules with the students’ prior courses. One possible reason could be that the more we increase the match threshold, the fewer rules activated, resulting in diminished recommendation performance. Fifth, we also examine the best performance achieved when we consider all, the last 2, or only the last semester of a student’s registration history to match it to the rules. The results indicate that the more history we use, the better results we will get. When we use all history, we manage to match more rules, resulting in better performance and a lower number of instances for which we cannot ofer personalized recommendations. Sixth, we get much better results when we use both sides of the activated rules to recommend courses to a student, i.e., both the consequent courses and any unmatched courses in the antecedent that the student has not taken yet. This indicates that the frequent course sets capture important historical trends in the enrolment data about which courses are commonly taken together. This is an indirect way of taking into account student-course interactions.
We need to acknowledge some limitations existing when using ChatGPT to build a recommendation system for university students. Firstly, we need to be very careful when using sensitive student data. As mentioned before, we cannot directly put real-world student-course interactions to ChatGPT4 due to privacy concerns about students’ data. Secondly, it might be tricky to track what part of the provided information ChatGPT4 uses to generate the recommendation. These tools have a particular limit on the length of text that they can process, and it can be hard to track this when interacting with the tool. If someone is not careful, ChatGPT might return suggestions while forgetting information provided earlier, resulting in meaningless output. That is even more common when using ChatGPT3.5. Additionally, LLMs sufer from hallucinations sometimes, which in our case could mean suggesting invalid course codes. In our experiments, we noticed this phenomenon only when our prompts had too long input character length. Once we took care of that, we checked but did not notice any hallucinations again. In general, someone should validate the output to ensure the outcomes are meaningful.
Third, for the confidence scores provided by ChatGPT4, we do not have a clear idea about how they are calculated, so they are not very reliable, as shown by our results (since the other scoring methods perform better). Finally, LLMs do not ofer reliable recommendations when provided with just course content and one student’s registration history. This serves as a warning for the students who might try to directly use an LLM with their own data.
Our goal is to explore ways to utilize LLMs for the task of course recommendation. ChatGPT4 can understand conversational text data and analyze the provided context to suggest relevant courses. We evaluate direct methods for recommendation, where we directly ofer information to the model and request recommendations for the next courses a student should take. We also introduce a novel framework that uses ChatGPT4 to create rules between prior and future courses. We also explore providing diferent contexts (course names, acronyms, and course descriptions) to ChatGPT4 so that it can find relevant and related courses. Our results highlight that when using directly ChatGPT4, we might not have relevant recommendations, but our proposed approach manages to better capture semantic information on course information and the insights provided by historical enrollment data. Based on our work, we believe more research is needed to explore the LLM capabilities in course recommendation. We hope to inspire other EDM researchers to consider LLMs for diferent problems in educational settings. 1–9