Large language models (LLMs) are a potential solution for solving the significant assessment load in big courses with numerous assignments. However, the quality of the automated assessment may not match the evaluation by teachers or other experts. In this paper, we examine the automated assessment of programming-related tasks in a large-scale introductory programming course. The study is structured into two parts: first, we examine how reliably LLMs can assess the tasks compared to course teachers when provided the same rubric. Second, we try to find out if a simple autonomous agent pipeline, mimicking a review board, can improve the assessment outcome. The study was conducted on a university-level introductory Python course with more than 500 students. We chose a total of four programming-related assignments from the four final weeks of the course. First, we provided the selected LLM models with the student answers accompanied by an evaluation rubric and a simple prompt and recorded the resulting scores and feedback comments. Second, we built a pipeline of autonomous agents with diferent roles of a review board and used the student submissions as input for the pipeline, again recording the scores and comments. In the article, we discuss the feasibility and the performance of the given approaches. We also provide a detailed analysis of the comparison between the results of the two approaches and the teacher-assessed results, and discuss the diferences in the results and the likely reasons for them. Finally, we outline the potential for future work.
The performance of large language models (LLMs) has increased significantly in a short time. In addition
to this, the use of autonomous agents [
The need for automating assessment [
This paper reports a research setup where LLM in diferent settings was used for automatically assessing introductory programming course assignment submissions. The purpose is to find out what kind of diferences there are in the review results of diferent LLM reviewers and human reviewers. As a bigger goal, we are trying to find out whether it is possible to automate the review process, and if not, what are the biggest obstacles preventing us from doing so.
This paper is structured as follows: First, we present the background and some related work about using AI in evaluating student work. In the third section, we present the context, including the course and the assignments evaluated. in the fourth section, the research setup is presented, including the cost analysis of utilizing LLM tools and the preparation of data for analysis. The results are presented in
Section 5 and discussed in the sixth Section, where we also present the limitations of the current work. Finally, a conclusion and ideas for future work are presented.
Automatic assessment in programming education has great potential benefits. As Paiva et al. [
Estévez-Ayrez et al. [
One of the biggest limitations of using LLMs is a phenomenon called "hallucination. Xu et al.
[
Jiang et al. [
There are some general ways of trying to improve LLM performance [
It should be noted that the temperature setting of the LLM has a great efect on the produced output,
as it changes the probability distribution (randomness) of the next token. Quite often it is argued that the
lowest temperature is the best, as it gives the most consistent result. However, one could argue that this
might not always lead to the best possible results. For example, in coding generation utilizing adaptive
temperature seems to improve results when high temperature is used for challenging tokens and low
temperature for confident tokens [
The usage of autonomous agents as part of the review process or automatic assessment has been
explored in diferent settings. [
The course under review is called Fundamentals of Programming and is quite a typical 7-week
introductory programming course (often referred to as CS1 course). The programming language used is Python,
and the course contains the fundamental concepts of the imperative programming paradigm, such as
variables and expressions, conditional and repetitive statements, functions, I/O, and data structures.
Some additional Python features, such as the use of the Pygame [
The course consists of lectures, tutorial exercises, demonstration assignments, and a final exam.
The tutorial exercises are fully automatically assessed by an education tool called ViLLE [
Both programs contained syntax errors as well as several logical errors.
• Week 3: The third week was about planning and commenting programs. In the first task, the
students were required to plan a program for keeping track of expenses. The plan should include
function signatures and docstrings [
The first two weeks of assignments were presumably easier to assess as the answers could be mostly verified programmatically. The tasks in the third week were more open-ended with several acceptable solutions. The fourth week was also assumed to be more dificult to assess, as the solutions were at least partly visually verified.
The pipeline for research was constructed with Python programming language, utilizing LangChain
[
For the Commenter and Chairperson roles, a full chat history of the previous steps was provided as initial input. The prompts are displayed in the Appendix A. All submissions were submitted through all three diferent reviewer roles. The temperature of all of the autonomous agents was 0.2, and the used model was GPT-4o (gpt-4o-2024-08-06). In addition, we also submitted a small subset (N=100) of the ifrst-week exercises to the GPT o1-preview model (o1-preview-2024-09-12) to gain a reference and see the performance diference in state-of-the-art reasoning and a more traditional, foundational LLM that is generally used. With o1, only the Reviewer role was used.
The rubric used for each week was the same as the rubric used to instruct the human reviewers. This led to some unforeseen dificulties: for example, in weeks 1, 2, and 3, an abbreviation was used to denote the maximum score. In some cases, the model got the score scaling wrong. Hence, we needed to exclude some of the reviews from the final analysis due to incomparable grading.
With the main analysis, a total of 22,8 million input tokens and 3,5 million output tokens were used
with GPT-4o. At the time of the analysis, this led to a combined cost of ≈ 60€ (or $61.50). The subset
of the first-week tasks submitted to the o1 model cost 0.3 million input tokens and 1 million output
tokens, which led to a total cost of ≈ 126.5€ ($130). A more detailed breakdown of the token costs can
be seen in Table 1. The costs for pricing were calculated utilizing OpenAI’s pricing page [
It should be noted that the cost can greatly vary depending on the selected model. Additionally, the tokenizers used can and will greatly afect the cost-efectiveness as well. Furthermore, although submissions were code files, most of them, including rubrics and exercise tokens, were written in Finnish. This increases the counted token amount significantly as the tokenizer OpenAI uses is optimized mainly for English instead of Finnish.
Before the analysis started, we removed all submissions from students who had not explicitly given a research permit. After this, the remaining data was pseudonymized by replacing the usernames with randomly generated numbers throughout the data. These code numbers allowed us to connect the reviews from diferent reviewer roles without dealing with personal information.
Before the analysis, we also needed to remove some inconsistent data. As a general rule, for each submission, this meant the data rows where the analysis was incomplete were erased from the data. 160 140 120 tn100 u oC 80 60 40 20 0 140 120 100 tn 80 u o C 60 40 20 0
Original Reviewer Commenter Chairperson Original Reviewer Commenter Chairperson
Typically, these cases arose because the submission had not been reviewed by a human, either due to late submission or suspicions of academic dishonesty. Some submissions were not reviewed by any of the models or the results were in unusable form (see Section 4). The N between diferent tasks (assignments) variates also because the students did not complete as many tasks at the end of the course as they did in the beginning.
The total number of submissions reviewed for each week is displayed in Table 2. In addition to running all submissions through GPT-4o via diferent roles in the pipeline, we picked a hundred first submissions from week 1 and ran them through the newest (at the time of writing) Chat GTP model, o1. This was done to see if there are noticeable diferences between the models, providing guidelines for the future. Due to drastically higher costs, using the newest model for all submissions was not seen as a generally usable option at the time of writing the article.
Since the human instructors provided a combined score for each week (a single integer between 0 and 20 for both tasks), the comparison must be done week-by-week instead of comparing individual tasks. Still, the two tasks each week were quite similar which means that the comparison can still provide valid results.
gap in amounts. Partly the point diference between roles Reviewer and Commenter can be explained by the pre-processing of the answers, because of a need to add some additional curly brackets in the submissions before review (see the Appendix A for reference), which sometimes lead the model to reduce points even when ignoring this edge case was included in the prompt of the model. However, this only explains a small amount of the detected deviation in the point distribution.
Finally, one hundred submissions to week 1 exercises were ran through the newest Chat GPT model, o1. The comparison to human reviewer and LLM roles using the older model, GPT 4o, are displayed in Table 7.
Based on the data, it seems that the newest model outperforms the old model in similarity to human reviewers in most categories. However, it should be noted that only 100 submissions from one week were used for comparison.
The results show, that while in some conditions the LLM reviewers can get close to human reviewers, there is still a considerable gap between them. The average scores were quite close, but the U-test revealed statistically significant diferences between the human reviewer scores and the LLM scores. Within a one-point threshold, the best-performing LLM roles could get quite close to human reviewers, and with a four-point threshold even closer. Still, in some tasks, roughly ten percent of tasks were graded diferently even when considering the 4 point threshold. This indicates, that replacing human reviewers with LLM counterparts requires more work.
Even though the rubrics for the assignment were quite detailed, there is always a human factor involved in the review process. It should be noted that part of the diference in LLM vs. human review may be caused by the fact that, in the vast majority of cases, each submission was assessd by only one teacher. However, this efect may be in this particular case a little diluted, since there were a total of 15 reviewers. Still, we can say that this likely leads to a more spread in the grades given by a human reviewer.
Diferent methods to improve the automatic assessment utilizing LLMs should be considered as well. For example, instead of asking LLM to give a total score by decreasing points from the full score based on errors, we could try asking LLM to provide a checklist of errors. This could improve the performance of the LLM. It is also noteworthy that our experiment was conducted without providing the model with any examples of already graded exercises (zero-shot). As mentioned in Section 4, the model was given the exercise description, rubric and the submission to be evaluated. Providing previously-graded submissions and/or reference answers could potentially increase the quality of the grading.
One could argue that flexibility is the greatest asset that LLMs can ofer in automated assessment
in introductory programming education. Unit tests provide a way to measure the student’s progress,
but they also limit the possible exercise types and scope. If an LLM can reliably assess the work of a
student, it could broaden the variety of possible exercise types even on massive online courses. This
would also enable scoring of partially completed answers that may not even compile — traditional unit
testing is typically unable to score such cases. Moreover, concepts such as program design, code quality,
and commenting are usually dificult or impossible to assess with unit testing frameworks. Novice
students often struggle with programming language syntax [
The usage of the Open AI’s newest model, o1, seemed promising. With the subset of the first-week task, the model seemed to achieve results that were closer to human reviewers than the older, generally used model. However, there is a big diference in cost-efectiveness as well, and as such, using the newest model would probably not be possible in most of the courses at the moment due to higher costs. Still, this seems to indicate that while the models get better, it should be possible to assign more and more review duties to them.
When the individual submission reviews were inspected, we found that in some cases LLM noticed an error in the submission that a human reviewer had missed. On the other hand, in some cases, LLM reviewers assessed the exercises wrong.
When delving deeper into the exercises for a more detailed analysis, we noticed that a common example of LLM falsely reducting points was when the students utilized Python’s flexible nature. For example, LLM claimed that a string input of "+2" cast into an integer would cause an error, and the correct way to do it would be to parse the number without the plus sign. In reality, the parsing works without an issue. This kind of problem could probably be solved by giving the LLM access to a code compiler. In the string manipulation functions, LLM noticed that some of the exercises did not refer to a correct parameter variable given to a function, but instead directly modified the original global variable. In some cases, this was missed by the course personnel evaluating the submission.
In the third week, there are cases where the human reviewer has given full marks even though the specifications on the rubric are not fully fulfilled: for example, in the task requiring function signature only, some students had started implementing a given program as well. If partial implementation was done, human reviewers did overlook some missing specification present in the provided rubric, but the LLM reviewers considered it a mistake. Other challenges on the third-week assignments were the problems in defining the return types - for example, LLM reviewers considered defining None as a return type an error, as rubric mentioned defining None as a return type was not necessary to gain full marks, while human reviewers followed the rubric.
In the Pygame exercises, the visual elements provided some problems. For example, the appearance of the snowman was not explicitly stated in the rubric, and the LLM deducted points for example if the snowman was missing eyes (which were not explicitly stated to be included in the instructions). Furthermore, in the snowflake assignment, LLM made some incorrect point reductions due to a misunderstanding that the requirement of difering size images would mean scaling them programmatically, while instead, the goal was to utilize the images of diferent size snowflakes provided with assignment.
Many of the problems listed above could have been avoided by providing more detailed descriptions in the rubric. It should also be noted that there were cases where the review provided by LLM was more accurate than the human counterpart. As seen in the results, LLM can achieve quite high performance, with some common mistakes made by LLM only revealed after inspecting many diferent submission reviews. For this reason for any future endeavors, we recommend first running a large subset of the exercises through the pipeline, then inspecting more closely the reviews where the human review difers, paying close attention to the rubric, and trying to ensure the consistent quality of human reviews.
Due to reviewers giving a combined score for each week instead of scoring the individual exercises, we were not able to compare the scores exercise by exercise. It is likely, that if inspected separately, the diference in the marks given by LLM roles and the human reviewers would probably appear smaller. Still, even more rigorous testing is needed, as the achieved performance is quite reliant on the used model. Additionally, as the used language was Finnish, this will likely have had a negative impact on the performance, as the vast majority of the training data of the used model is written in English instead of Finnish.
Each submission was run through the evaluation pipeline only once using a low temperature (0.2) to
provide more consistent results. Utilizing diferent settings could and likely would afect the performance
of the used models. Furthermore, it should be noted that the submissions were given into the pipeline
in a zero-shot setting. In many diferent tasks giving the model examples, as well as reference answers
would likely improve the performance of the method. Giving examples in input often improves the
performance of the models across various tasks [
Even though the model did not have access to code execution or visual output of programs, the LLM did quite well in the automatic assessment. The results and faced challenges highlight the need for rigorous testing before a larger data set is run through a pipeline that utilizes LLMs. Still, there are potential benefits in automating the assessment, for example, the possibility to ofer students immediate feedback on their submission and the possibility to fix the mistakes before moving on to the next tasks. The largest challenges in giving students direct access to a similar system are the related costs, as well as the LLM’s inability to reliably identify a full mark answer.
Future work should include approaches to minimizing costs. In the case of code submissions, it could be beneficial to first translate the submissions and all related material into English. Naturally, the performance in such cases would then be also dependent on the translation process. Another option would be to train and/or use models that are already optimized for the given language. The better results of the limited testing of the newest Open AI model also seem promising, but the larger-scale testing and general usage would require lower costs.
Exploring more diverse strategies in how agents communicate, as well as experimenting with various temperature settings should be examined in future work. Additionally, expanding the roles of the agents to incorporate other roles could be beneficial. Similarly, producing justifications using higher temperature settings and then the final score using lower temperatures could work as well. Finally, expanding the current work by providing the LLM access to code execution, as well as visual outputs of the executed programs could be an interesting approach.
This work has been supported by FAST, the Finnish Software Engineering Doctoral Research Network, funded by the Ministry of Education and Culture, Finland.
During the preparation of this work, the author(s) used OpenAI’s diferent models, e.g. GPT-o1 as a helpful tool for creating the pipeline. After using these tool(s)/service(s), the author(s) reviewed and edited the content as needed and take(s) full responsibility for the publication’s content.
System
You are a computer science university teacher teaching a fundamentals of programming course. Below is an exercise description, a rubric, and a student submission. Your task:
2. Give final marks at the end.
Output the final marks inside <FINAL_MARKS> tags. Please remember that all of the curly braces have been replaced with double curly braces by your instruction and students should not receive any penalty from double curly braces in their submission.
Human
You are a computer science university teacher teaching a fundamentals of programming course. Below is an exercise description, a rubric, and a student submission. Your task:
2. Give final marks at the end.
System
You are a computer science university teacher teaching a fundamentals of programming course. You are part of a grade review board. Your task is to evaluate the validity and correctness of the previous teacher’s grading based on the exercise, rubric, and the teacher’s review. Then, provide your own final marks. Finally, output the final marks inside <FINAL_MARKS> tags. Please remember that all of the curly braces have been replaced with double curly braces by your instruction and students should not receive any penalty from double curly braces in their submission.
Human
Above are the original exercise description, rubric, and student submission (for your reference), plus the prior grading. Please provide your commentary and updated marks if needed.
System
You are the chairperson of the grade review board in a fundamentals of programming course. Your task:
Please provide the final marks as Chairperson. Remember to place them in <FINAL_MARKS> tags.