GPT models' ability to automatically score students' writing makes them promising to assess students' interdisciplinary learning quality, a significant but unaddressed gap. While standard GPT models have challenges in understanding contextual knowledge, previous research suggests that knowledge-empowered fine-tuned (KEFT) GPT models can overcome the limitations. This study examined 1) whether KEFT GPT models can accurately label interdisciplinary learning quality based on learning process and outcome data, and 2) how to implement these models within a learning analytics (LA) platform, including three major steps. First, to establish a ground truth dataset, two pairs of researchers independently coded and discussed the interdisciplinary learning quality of 400 online posts and 190 sections from 16 essays based on an interdisciplinary learning quality codebook. Second, we employed KEFT GPT models to evaluate interdisciplinary learning quality. Results indicated that the models achieved accuracy comparable to human researchers. Third, the models were integrated into an LA platform, TopicWise, which automates evaluation and provides tailored feedback. This study showcased the feasibility of applying KEFT GPT models in LA to analyse student learning processes and outcomes. Next, we will conduct user studies to examine TopicWise's impact on students' interdisciplinary learning and identify areas for improvement.
Interdisciplinary learning combines perspectives, methods, and strategies from various disciplines to
address complex issues that cannot be fully understood within a single field [
ChatGPT, a chatbot powered by foundation large language models (LLMs) like GPT-3.5 and
GPT4o, developed by OpenAI [
GPT-4 to provide feedback to student essays and found the GPT models can provide more readable
and consistent feedback than human teachers in data science courses. These studies show promising
results in applying LLMs to automatically evaluate students' learning processes and outcomes and
provide feedback, an important research topic of LA [
LA is a research area that focuses on gathering, analysing, and reporting data about learners and
their environments to gain insights and improve both the learning experience and the conditions
that support it [
RQ1: Can knowledge-empowered approaches increase GPT models' accuracy in analysing interdisciplinary learning quality?
RQ2: How can a prototype LA platform be designed to leverage GPT models for providing automated feedback on students' interdisciplinary learning quality?
Interdisciplinary learning refers to the process of incorporating knowledge and perspectives from
multiple disciplines to solve problems or explain phenomena beyond the boundary of a singular
discipline [
LA can effectively analyse real-time learning processes by collecting process data from digital
tools like learning management systems (LMS), automatically analysing data with algorithms, and
providing visualised dashboards and personalised feedback [20]. Various applications of LA have
been utilised in interdisciplinary learning. For instance, Lee et al. [
However, GPT's expertise in specific domains may be limited, which can result in nonsensical or inappropriate responses to specific prompts [27]. Fine-tuning is a technique that can help mitigate this limitation and improve GPT performance on specific tasks. Fine-tuning refers to the additional training of pre-trained models to customise them for specific tasks or datasets [28]. One benefit of this approach is the ability to tailor models to enhance their performance in specific tasks, which requires only 50 to 100 examples for training [29]. The fine-tuned GPT models have been shown to be effective in several studies. Chae and Davidson [30] suggested that fine-tuning is an optimal solution for researchers due to its relatively high accuracy and low cost.
However, fine-tuning methods rely heavily on the pre-training data [29], which may limit their
ability to handle tasks requiring knowledge not included in their initial training set. The
knowledgeempowered method, which incorporates external knowledge into the model, may further improve
GPT performance on specific tasks by expanding the model's understanding beyond the pre-training
dataset [31]. The basic premise of this technique is that by integrating additional information, models
can enhance their comprehension of content and generate better output [32]. Hu et al. [33] combined
domain knowledge (geo-knowledge) with GPT and showed that external knowledge is indispensable
for guiding the behaviour of GPT models. Similarly, Yang et al. [34] conducted a study to use external
knowledge bases to enhance pre-trained language models for machine reading comprehension. They
found that incorporating structured knowledge from knowledge bases significantly improved
models' accuracy on benchmarks like ReCoRD and SQuAD1.1. Overall, these studies have shown
that by integrating external knowledge into models, the performance of models in specific tasks
significantly improved [
The promising results highlight the potential of using knowledge-empowered strategies to analyse the interdisciplinary learning quality of students' work. In a recent study, we employed knowledge-empowered approaches—such as dictionary-based knowledge to address terminology that GPT models struggle to understand, and rule-based knowledge to capture implicit mechanisms outlined in codebooks—to fine-tune GPT models. The findings revealed that these strategies significantly enhanced the performance of GPT-3.5 in evaluating interdisciplinary learning process data (e.g., online posts). However, this approach has yet to be applied to GPT-4 models or to learning outcome data such as final essays. Building on these strategies, this study seeks to develop an interdisciplinary LA platform capable of processing real-time data. The platform will provide students with timely feedback on the quality of their interdisciplinary learning and offer actionable suggestions to foster improvement.
The following sections will present a prototype interdisciplinary LA platform, TopicWise, by giving an overview of the platform, detailing how Knowledge-empowered fine-tuned GPT models have been trained and their performance, and showing the user interface design.
This platform is designed to evaluate students’ interdisciplinary learning and provide actionable
feedback for improvement. As Figure 1 shows, students can upload files (e.g., essays) or short texts
(e.g., posts, discussions) through the user interface. After that, the text will be delivered to
knowledge-empowered fine-tuned models for processing to generate relevant feedback, including
comments on the text and suggestions for improvement from four dimensions of interdisciplinary
learning quality: diversity, cognitive advancement, disciplinary grounding, and integration [
To prepare ground truth data for model training and testing, we manually analysed 400 posts collected from the Miro platform and 16 essays. The existing literature on manual content analysis of essays indicates that smaller units, rather than entire texts, are more appropriate for studying the general standard of essays [35]. Therefore, to retain the consistency and integrity of ideas, we analysed students' essays at their most granular level, focusing on the smallest sections, which typically consist of several paragraphs and represent the deepest layer of content organisation. This approach divided the 16 essays into 190 data points.
Thereafter, two human coders independently labelled students' posts. Another two coders labelled essays, all using the codebook of interdisciplinary learning quality, which consists of diversity (the number of disciplines represented in the text), cognitive advancement (the depth and clarity of the articulated viewpoints), disciplinary grounding (the extent to which the text applies disciplinary knowledge), and integration (the degree to which perspectives from multiple disciplines are synthesised). We used Cohen's Kappa score, as shown in Table 1, to evaluate the inter-rater reliability between human raters on each dimension of interdisciplinary learning quality. Human coders subsequently discussed and settled their differences, reaching an agreement on each item, which was regarded as the ground truth of interdisciplinary learning quality.
For both the post and the essay dataset, 80% of the data were randomly selected as training data,
which were applied to fine-tune the GPT-3.5 and GPT-4o-mini models, while the remaining 20% of
the data were tested, which is explained in detail in Section 3.2.2. The frequency of each code for
each dimension in the training dataset and the testing dataset is displayed in Table 2. The dimensions
refer to the elements of the interdisciplinary learning quality [
Table 1 Inter-rater reliability (Cohen’s Kappa) between human raters on notes and essays Data Diversity Cognitive Disciplinary Integration Overall advancement grounding Post 0.83 0.82 0.83 0.73 0.83 Essay 0.84 0.71 0.67 0.57 0.72
We systematically crafted the prompts for GPT models following the stages in Figure 2. We first adopted the interdisciplinary learning codebook, drawing upon educational theories. Following that, we created a template to translate the natural language of the codebook into a structure that GPT could process. For instance, we used a standardised format like a conditional statement (i.e., if-else) to represent the rules in the codebook. Ultimately, each tailored prompt was generated using the template and contained the following components (see Table 4): (1) A system message defining GPT’s persona; (2) A tailored task instruction outlining the task and its needs; and (3) A rule derived from the codebook, offering guidelines and examples relevant to different levels of a specific dimension.
We also utilised CoT to effectively instruct GPT models with step-by-step tasks. Guided by CoT, there are three main steps (see Table 3) in the prompts: Firstly, task clarification provides essential details such as the requirements and desired output. Secondly, in the task breakdown, the tasks are divided into smaller, manageable parts. Lastly, the logical sequence instruction guides GPT in understanding the relations and mechanisms among these breakdown tasks. Through these three steps, we created a structured framework designed to address complex tasks with CoT methods. Tailored Prompt System messages "You are an encyclopaedia that can precisely evaluate the disciplines reflected in the following notes." Rules sourced from the "Please see all the information as a single Codebook paragraph and evaluate the cognitive advancement level of students essays. Return only numerical values 0, 1, and 2." Tailored task instructions "Return 2 if the content provides detailed reasoning and specific examples to demonstrate a deep understanding of the topic." Chain-of- Task clarification "Please see all the information as a single Thought paragraph and answer the below two Prompting questions about the cognitive advancement of essays. Please return yes or no." Task breakdown "Question 1: Does the paragraph have basic explanations or causalities or examples or mechanisms or elaborations of phenomena."; "Question 2: Does the paragraph provide detailed reasoning and specific examples to demonstrate a deep understanding of the topic?" Logical sequence instructions "Question 2 is an extended one based on
Question 1."
In this study, two types of knowledge, dictionary-based and rule-based knowledge, were integrated into prompts to enhance the models' performance.
Dictionary-based knowledge includes specific words with predefined categories [35]. For example, in analysing the diversity dimension, a discipline dictionary was provided. The dictionary included eleven disciplines: 'Arts and humanities'; 'Business and economics'; 'Clinical, pre-clinical and health'; 'Computer science'; 'Education'; 'Engineering and technology'; 'Law'; 'Life sciences'; 'Physical sciences'; 'Psychology'; and 'Social sciences'. If a student mentions terms like "copyright," GPT might struggle to classify them correctly. To address this, prompts were structured as: "If students mention terms such as WORD (copyright), it reflects content related to the LABEL 'Law.'" This method was applied across other dimensions of interdisciplinary learning quality to improve accuracy. This study applied Dictionary-based knowledge on Diversity and Cognitive advancement dimensions because GPT models do not have context knowledge about these two dimensions and thereby need specific examples. The dictionary-based knowledge can also be applied to other circumstances when LLMs cannot understand specific instances.
Rule-based knowledge, on the other hand, uses task-specific logic derived from relationships outlined in codebooks. For instance, if no disciplines are mentioned in a student's text (Diversity = 0), disciplinary grounding should also be 0, as no disciplinary knowledge is present. Similarly, if fewer than two disciplines are mentioned (Diversity < 2), Integration is likely 0, as interdisciplinary synthesis is absent. These rules were embedded into prompts using structures like: "IF DIMENSION A (Diversity) is 0, THEN DIMENSION B (Disciplinary Grounding) is likely to be 0." By encoding such logic, rule-based knowledge ensures the model considers the interplay between dimensions, enhancing its ability to perform deductive coding effectively. This study applied rule-based knowledge on Disciplinary grounding and Integration dimensions because these two dimensions rely on the outcomes of Diversity. The rule-based knowledge can also be applied to other circumstances when LLMs cannot understand the implicit rules in the task, especially in tasks with interdependence.
Experiments were operated on GPT-3.5 and GPT-4o-mini models. To answer RQ1, we tested the models in four modes: prompts (directly use prompts), fine-tuning (apply fine-tuning), knowledgeempowered prompts (embed knowledge in prompts) and knowledge-empowered fine-tuning (apply knowledge-empowered prompts and fine-tuning). Cohen's Kappa scores, presented in Tables 4 and 5, were used to measure the agreement between GPT-generated labels and human-coded ground truth for both posts and essays, with human inter-rater reliability serving as the benchmark.
The results indicated that knowledge-empowered approaches enhance both prompt-based and fine-tuning methods. Knowledge-empowered methods demonstrated clear improvements for posts (learning process data), validating their effectiveness. For essays (learning outcome data), these methods enhanced performance on Diversity and Disciplinary grounding for knowledge-empowered prompts and augmented accuracy on most dimensions, except for Diversity (0.91 vs. 0.78) in finetuned models.
Overall, integrating knowledge-empowered strategies with fine-tuning significantly increased GPT models' agreement with human coders, achieving or surpassing expert-level proficiency in analysing interdisciplinary learning quality Table 4 Cohen’s Kappa scores in student posts
Diversity Integration 3.3. User interface After getting the models, we implemented them into an LA platform we are developing: TopicWise (https://a-ori-topic-wise.vercel.app/).
Figure 3 presents a screenshot of TopicWise for providing scores and feedback for essays. In the "Scores Comparison", students can view their scores across the four interdisciplinary learning dimensions—diversity, cognitive advancement, disciplinary grounding, and integration—and compare them with the average scores from the database. This comparative feature intends to help students understand their performance in the context of their peers and highlight areas for improvement. The "Paragraph Annotations" section provides a more granular analysis, offering scores for each specific paragraph in the essay. Additionally, the platform explains the reasoning behind each score and provides targeted feedback for improvement. This detailed breakdown identifies strengths and weaknesses in students’ writing, aiming to foster a deeper understanding of interdisciplinary learning principles and guiding their revisions. The platform also supports realtime feedback for online discussion posts, as shown in Figure 4. When students upload their posts to TopicWise, the system quickly analyses the content, assigns scores for interdisciplinary dimensions, and delivers immediate feedback. This instant evaluation enables students to refine their posts during discussions, promoting more effective engagement with interdisciplinary concepts and improving learning outcomes over time.
TopicWise’s ability to deliver timely feedback gives it the potential to support students in interdisciplinary learning through reflective essay writing and interactive online discussion. It has the potential to provide students with accessible, actionable insights into their performance, empowering them to make meaningful progress in their interdisciplinary learning.
The study found that knowledge-empowered approaches can enhance the performance of GPT-3.5 and GPT-4o-mini models, achieving accuracy comparable to human experts. The trained models were subsequently integrated into a prototype LA platform and were able to offer automated scoring and feedback on students’ interdisciplinary learning quality. By leveraging these advancements, students can gain insights into their performance in real-time. The use of knowledge-based finetuning highlights its potential as a robust method for enhancing LA, making it a promising approach in educational contexts.
Knowledge-empowered approaches amplified both prompt-based and fine-tuned model performance. By incorporating small-scale domain-specific dictionaries and rule-based logic, the study extended findings on external knowledge integration in prompt engineering in automated essay analysis [35]. This method also addresses the issue of relying on large knowledge graphs [36] or extensive knowledge bases [34], emphasising the efficacy of tailored knowledge enhancements during fine-tuning. Interestingly, this study found that knowledge-empowered fine-tuning was more effective for evaluating posts than essays. Posts typically require less complex reasoning than essays; reasoning demands deeper critical thinking and subject-specific expertise, making them more challenging for GPT models to analyse [37]. Although CoT prompting was used to assist with reasoning tasks, essays’ intricate structure and nuanced content posed greater difficulties for the models. This highlights a gap in the capabilities of GPT models when handling more cognitively demanding tasks, suggesting the need for further refinement to support evaluations requiring advanced reasoning skills.
The study also introduces TopicWise, a prototype interdisciplinary LA platform that automates the evaluation process and provides tailored feedback to students. This platform not only aims to provide automated scoring but also aims to deliver tailored feedback to students, helping them understand and improve their performance. By enabling dynamic feedback and continuous monitoring, the platform has the potential to enhance students’ interdisciplinary learning practices. User studies need to be conducted next to evaluate and refine the tool.
However, this study acknowledges several limitations. First, the models are trained based on a relatively small dataset of online posts and essays from a specific cohort of undergraduate students. Whether the methods can be generalised to other datasets needs further research, raising questions about generalizability. Further research is needed to determine whether these methods are applicable to broader and more diverse datasets. Second, the study only tested GPT-3.5 and GPT-4o-mini, leaving unexplored the potential of other language models, such as LLaMA and Gemini, which could offer different perspectives or improved capabilities. Third, the LA platform has not been tested by users like instructors and students. We plan to conduct user studies after further refining the tool.
Despite these limitations, the study highlights the potential of combining fine-tuning with knowledge-empowered strategies for evaluating both learning process data (e.g., online posts) and outcome data (e.g., essays). The integration of these trained models into an LA platform further enhances the approach by providing immediate, data-driven feedback. The platform has the potential to support educators in fostering interdisciplinary skills while optimising the assessment process. Future work will focus on expanding the dataset, testing additional models, and conducting user studies to ensure that the platform meets the needs of educators and students.
The research is conducted with the support of the Energy Research Institute @ NTU, Interdisciplinary Graduate Programme, Nanyang Technological University, Singapore.
The author(s) have not employed any Generative AI tools in essay writing.