Argument mining algorithms analyze the argumentative structure of essays, making them a valuable tool for enhancing education by providing targeted feedback on the students' argumentation skills. While current methods often use encoder or encoder-decoder deep learning architectures, decoder-only models remain largely unexplored, ofering a promising research direction. This paper proposes leveraging open-source, small Large Language Models (LLMs) for argument mining through few-shot prompting and fine-tuning. The small size of these models and their open-source nature ensure accessibility, privacy, and computational eficiency, enabling schools and educators to adopt and deploy them locally. Specifically, we perform three tasks: segmentation of student essays into arguments, classification of the arguments by type, and assessment of their quality. We empirically evaluate the models on the “Feedback Prize - Predicting Efective Arguments” dataset of grade 6-12 students essays and demonstrate how fine-tuned small LLMs outperform baseline methods in segmenting the essays and determining the argument types while few-shot prompting yields comparable performance to that of the baselines in assessing quality. This work highlights the educational potential of small, open-source LLMs to provide real-time, personalized feedback, enhancing independent learning and writing skills while ensuring low computational cost and privacy.
Writing well-structured essays can be challenging for students, as they require not only quality argument
components but also cohesive connections between them [
Efective argument mining entails several subtasks, such as segmenting the essay into distinct
argument components, classifying their type, assessing their quality and establishing relationships
between them [
While high-performing argument mining methods have potential, they are often dificult to access and
scale, especially in educational settings [
We propose leveraging open-source2, small Large Language Models (LLMs) to perform a complete pipeline in argument mining (from segmenting the text in arguments to classifying their type and assessing their quality) in an educational context. By means of few-shot prompting and fine-tuning, these models can execute these tasks locally on the student’s laptop, ensuring accessibility and maintaining computational eficiency. Figure 1 depicts the proposed method’s pipeline, outlining each stage of the process.
Contributions In this paper, we make several contributions to this domain. First, we address the gap in leveraging small, open-source LLMs for argument mining, combining argument segmentation, type classification, and quality assessment. Second, we propose a computationally eficient and privacypreserving approach, enabling local analysis on standard devices through fine-tuning and few-shot prompting of the LLMs. Finally, by evaluating our approach on a benchmark dataset of student essays, we demonstrate its ability to deliver actionable feedback on a local computer, fostering improved writing skills for students grades 6-12 while preserving privacy. Our method advances argument mining in resource-constrained educational settings and highlights the transformative potential of LLMs in personalized education.
This paper is organized as follows: Section 2 reviews the most relevant literature, providing the background and context for our research. In Section 3, we describe our methodology. Section 4 presents and analyses our experimental results. Finally, Section 5 provides a discussion of the findings, followed by a conclusion and an outline of the limitations in Section 6.
Argument mining Argument mining is a complex field that aims to identify, classify, and analyze
argumentative structures within text [
State-of-the-art methods in argument mining typically rely on deep neural networks [
Providing meaningful feedback from such analyses is particularly impactful in educational contexts.
Actionable feedback enables students and educators to identify strengths and areas for improvement,
with standardized scoring systems serving as valuable tools to guide learning and enhance outcomes [
Given a human-written essay, the goal is to perform three tasks: (1) segmenting it into arguments; (2) classifying each argument into one of seven categories—Lead, Position, Claim, Counterclaim, Rebuttal, Evidence, Concluding Statement; and (3) assessing the quality of each argument using three levels —Inefective, Adequate, Efective. Segmentation is conducted first, followed by argument type classification and quality assessment. These latter tasks can be performed independently or jointly.
We investigate the efectiveness of three open-source, small LLMs, namely Qwen 2.5 7B [
The first approach applies few-shot prompt-tuning sequentially: first, the LLM is prompted to segment
the essay into arguments. Then another prompt is used for argument type classification and/or quality
assessment. The prompts are built by concatenating the following textual elements:
3https://huggingface.co/blog/jjokah/small-language-model
1. Few-shot examples are given in the format depicted by the examples included in Tables 7, 8 and
9 in Appendix A.3, depending on the task. The label of the type of argument or its quality is added
between each argument in the essay. The example essays are extracted from the training set split used
in Ding et al. [
Examples of the expected output format and the handling of incorrect outputs are detailed in Appendix A.4. After segmentation, classifying the entire text at once may cause label mismatches, misaligning them with arguments. To simplify this task for smaller LLMs, we classify one argument per prompt. However, the full essay is still provided as context, given that the type classification and quality assessment of each argument depend on the essay’s overall argumentation strategy.
Also note that the specific structure, content and phrasing of each prompt have been meticulously designed after an intensive efort of prompt engineering. Any changes to the sequence or omission of parts significantly afect both the accuracy and the format of the output. An alternative approach to prompt-tuning the LLM would involve asking it to replace each separator with its corresponding label (type, quality, or both), as demonstrated in the fine-tuning process described in the next section. With this method, the output should exactly replicate the original essay, with classified labels inserted at the appropriate points between arguments. However, this approach did not deliver good performance with the small LLMs as these models struggle to reproduce the input text word-for-word and often introduce errors such as adding extra words, omitting parts of the text, or inserting additional separators. For output examples, see Appendix A.3.
The second approach involves fine-tuning an LLM for the following tasks: (1) argument segmentation; (2) argument type classification; (3) argument quality assessment; and (4) argument type classification and quality assessment together. The three small, open-source models were fine-tuned on Google Colab using an A100 GPU on each of the tasks. To establish a comparative upper bound, we also fine-tuned GPT-4o mini using OpenAI’s API. Further details on the fine-tuning of this model can be found in Appendix A.2.3.
Data Fine-tuning requires the use of a dataset. In our study, we used a collection of argumentative
essays written by U.S. students in grades 6-12, annotated by expert raters. The dataset is derived from
the Kaggle competition “Feedback Prize - Predicting Efective Arguments,” 4 which constitutes a subset
of the PERSUADE 2.0 Corpus. This subset includes approximately 6,900 essays from a total of 26,000
argument components, representing just over a quarter of the corpus. The essays were selected to
achieve a balanced distribution of discourse elements across varying levels of efectiveness [
For fine-tuning, we used the split provided by Ding et al. [
Setting We performed the fine-tuning of the small, open-source LLMs using the SFTTrainer module from the TRL library5. We employed Unsloth6 to optimize performance and reduce memory usage.
Additionally, we incorporated Low-Rank Adaptation (LoRA) [
Input sequence and target Both the input sequence and the fine-tuning target were formatted in accordance with the previously described few-shot prompting methodology. For the segmentation task, the input was the original essay, while the target output was the corresponding essay segmented by a designated SEP label (See Table 7 in Appendix). In contrast, for the joint task of argument type classification and quality assessment, the input consisted of the essay already segmented with the SEP labels (See Table 7 in Appendix). The target, in this instance, was defined as the same essay further partitioned into discrete arguments, with each argument interleaved with labels specifying both the type of argument and its quality (See Table 10 in Appendix). This approach ensures methodological consistency across tasks and facilitates a systematic evaluation of model performance on both segmentation and combined argument type classification and quality assessment tasks. See Appendix A.2.3 for more details.
Inference The fine-tuned models were run on an Apple M1 Pro laptop with 32 GB RAM using
Ollama7, an open-source framework that enables users to run, create, and share LLMs locally on their
machines. We did not provide any few-shot examples demonstrating how to perform the task or how
to specify the output format. To evaluate performance, we used the same test set as Ding et al. [
We performed all our evaluations on the test set of the “Feedback Prize - Predicting Efective Arguments,”
8 datasets. We use the same test set employed by Ding et al. [
Segmentation F1 score We compute the F1 score for segmentation at the token level using the BIO framework, where each token at the beginning of an argument is tagged as B, tokens inside the argument are tagged as I and O denotes when a token is not part of an argument, which is not the case in our task as all the tokens are supposed to belong to an argument.
Type and quality F1 score We adopt the evaluation method used by Ding et al. [
|| Matched predicted arguments are considered true positives if they are of the same type (or quality) as the gold argument. Otherwise, they are classified as a false negative. Unmatched predicted arguments are considered false positives and labeled as Echec.
Spelling errors LLMs are required to reproduce each input essay—originally written by children—segmented with the SEP separator. Because these essays contain numerous spelling errors, smaller LLMs often attempt to correct them automatically, thereby altering the text and complicating direct comparisons with the ground truth. Since our goal is not to address spelling errors and the dataset labels do not account for them, we first correct the essays before providing them to the LLM to ensure accurate and consistent segmentation. To do so we use language-tool-python, a wrapper for LanguageTool9.
We compare our method with several encoder-based variants following the approach proposed by
Ding et al. [
Figure 2 compares the performance of the small, open-source models on the segmentation task, evaluated in their best few-shot settings (either zero or three) and fine-tuned configurations. Additionally, we include the Longformer as a state-of-the-art baseline, and GPT-4o mini (with three-shot learning and ifne-tuned) as a commercial upper bound. Error bars indicate the standard deviation across three runs. All LLM models surpass the Longformer’s performance, demonstrating the advantages of large-scale pretraining and transfer learning. Fine-tuned models exhibit substantial performance gains over their few-shot counterparts, underscoring the efectiveness of supervised adaptation. Among all small opensource models, the fine-tuned Llama 3.1 8B achieves the highest F1 score of 87.52, an increase of 18.05 points or 26.00% in segmentation performance over the baseline (Longformer) and 6.41 points but 3.7% below GPT-4o mini’s performance. Figure 4 in Appendix A.5.4 depicts the overlap between the inferred segmentation and the gold segmentation across models. Fine-tuned Llama 3.1 8B achieves the highest overlap with the ground truth, outperforming Longformer and aligning with the macro F1 results. Figure 5 in Appendix A.5.3 reports the average number of arguments per essay across models for further segmentation analysis.
Figure 3 shows the macro-averaged F1 scores for the argument type (left) and quality (right) tasks, respectively, across various models and experimental settings. The figure compares the performance of the three small open-source models in the best few-shot (either zero or three) and fine-tuned settings, along with the baseline and GPT 4o mini (three-shot and fine-tuned). The F1 scores are reported for two task setups: individual classification (indiv.) and joint classification of both argument type and quality together (joint). Results are shown for two segmentation conditions: gold (provided) and inferred. Transparent colors indicate the gold condition, while less transparent colors represent inferred segmentation. Circles and triangles correspond to the joint and individual setups, respectively. Error bars represent the standard deviation of the results.
Argument type classification Regarding the type classification task, models using the gold segmentation consistently outperform the models with inferred segmentation and the baseline. We also observe significant improvements when fine-tuning the small, open-source LLMs, both with the gold and inferred segmentation. The best-performing model with the gold segmentation is Gemma 2 9B ifne-tuned, with an F1 score of 79.74, which is 10.27 points or 14.78% larger than the baseline (BERT) but 5.1% below GPT-4o mini’s performance. The best-performing model with inferred segmentation is Llama 3.1 8B fine-tuned, with an F1 score of 51.45, which is 12.27 points or 31.32% larger than the baseline (Longformer + BERT) but 19.85% lower than GPT-4o mini fine-tuned. Generally, models yield better performance in the joint than in the individual setup.
Argument quality assessment Regarding the argument quality assessment task, models using the gold segmentation also consistently outperform the models with inferred segmentation. In this case, the best-performing small open-source model is Gemma 2 9B with three-shot learning, achieving an F1 score of 44.56, which is 19.96 points or 81.14% larger than the baseline’s performance but 18.98% lower than the GPT-4o mini fine-tuned performance. With inferred segmentation, the best performing small, open-source LLM (Gemma 2 9B with three-shot learning) achieves similar performance to the baseline, both of them below that of GPT-4o mini. Interestingly, fine-tuning, in this case, tends to worsen the performance across models, particularly when given the gold segmentation.
Note that the results for Llama 3.1 Qwen 2.5 and GPT-4o mini fine-tuned are absent in the individual setup due to repeated inference failures. Refer to Section 6 for more details.
To better understand the previously reported performance figures, we summarize the results at the label level for the best-performing model (Llama 3.1 8B fine-tuned in the argument type classification task and Gemma 2 9B three-shot learning in the argument quality assessment task) in the joint set up and the inferred segmentation configuration. Tables 1, 2, (and 12 in Appendix) depict the precision, recall and F1 score for the argument type classification, quality assesment and segmentation tasks, respectively. Furthermore, Tables 14, 15 and 13 in Appendix contain the corresponding confusion matrices.
Type Lead Position Claim C claim Rebuttal Evidence Concluding Argument type classification As seen in Table 1 the model is the most accurate in classifying Concluding statements, followed by Evidence. In contrast, the model exhibits the lowest performance when classifying Rebuttals and Counterclaims. Interestingly, the model has very high recall (96.97%) but low precision (48.48%) when classifying Lead arguments, suggesting that it often over-labels arguments as Lead. There is large misclassification rates of Claim, Position, and Evidence, into Echec entries, confirming that segmentation errors contribute to lowering the performance in the classification task, see Table 14 in the Appendix.
Quality assessment Regarding quality assessment, the best performance of the model is in the Adequate label, struggling with the Inefective (low recall) and especially the Efective (low precision) labels. In fact, Adequate tends to be over-predicted by the model. Part of the classification errors are also due to segmentation mistakes, as reflected by the Echec column in the confusion table in Table 15 in the Appendix. Quality Inefective Adequate Efective
In this paper, we have explored the potential of three small, open-source LLMs—namely, Qwen 2.5 7B, Llama 3.1 8B, Gemma 2 9B—to perform three argument mining tasks in an educational setting: argument segmentation, argument type classification, and argument quality assessment. We have experimented with both few-show prompting and fine-tuning, comparing small open-source LLMs to commercial LLMs (GPT-4o mini) and state-of-the-art encoders. From these extensive experiments, we draw several findings.
First, small and open-source LLMs are able to efectively perform argument mining tasks with significantly better performance than state-of-the-art baselines (Longformer and BERT). Commercial small LLMs, such as GPT4o-mini, yield the best performance.
Second, diferent approaches provide the best results depending on the task. Whereas finetuned models consistently outperform few-shot prompting in the argument segmentation and type classification tasks, we observe the opposite behavior in the argument quality assessment task.
Third, model performance in the tasks of interest does not necessarily increase with the number of
shots when performing few-shot prompting, which is consistent with what has been reported in the
literature [
Fourth, joint fine-tuning setups where both tasks were carried out at the same time tend to yield
better results than individual setups where the tasks were performed independently, showing a strong
link between classifying argument types and assessing their quality [
Fifth, the automatic segmentation and classification of the type of argument seems to be an easier task
than the assessment of the quality of the arguments, likely due to dificulties in creating high-quality
and consistent ground truth quality assessments across essays 10 [
Finally and most importantly, this study highlights the potential of open-source, small LLMs, running
locally on personal computers, to support students in the development of their essay-writing skills. Our
framework prioritizes privacy and accessibility, addressing the challenge of developing eficient models
for local use without consuming too many resources [
In this paper, we have presented a study of the potential of small, open-source LLMs for argument
mining, investigating their efectiveness in both few-shot prompting and fine-tuning setups.
Finetuning proved especially valuable for argument segmentation and type classification such that small,
open-source LLMs significantly outperform state-of-the-art approaches by 18.05 points in argument
segmentation and 10.27 points in argument type classification. Our experiments also illustrate the value
of joint setups for improved argument type classification. By focusing on models running locally on
students’ personal computers, our research promotes accessibility and resource eficiency, illustrating
the potential of open-source, small LLMs as a promising tool for educational applications.
10See Appendix A.6 for a discussion on the annotation quality
Our work is not exempt from limitations that we plan to address in future work. First, the performance
on the argument quality assessment task is low for all models, including state-of-the-art methods. We
hypothesize that the poor performance might be due to the quality of the annotated data, as suggested
by other authors [
Second, the fine-tuned Qwen 2.5 7B, Llama 3.1 8B, and GPT4o mini in the individual setup were unable to perform the argument type classification and quality assessment tasks due to repeated failures during inference. While the exact cause remains to be investigated, potential reasons include the hyperparameters used for the fine-tuning not being suitable for the individual setup or instability in the fine-tuned models.
Finally, we evaluated small LLMs using the PERSUADE 2.0 corpus dataset, which consists solely of English high-school writing. As a result, it remains uncertain whether findings can be generalized to other educational contexts and languages. Furthermore, a “clinical” evaluation in the classroom with students would be necessary to assess the real-world efectiveness of the proposed approach beyond what can be measured with benchmark datasets. We leave such an evaluation to future work.
The integration of AI-based argument mining for automatically evaluating student essays raises ethical
concerns related to data privacy, fairness, accountability and human autonomy [
L.F. and N.O. have been partially supported by a nominal grant received at the ELLIS Unit Alicante Foundation from the Regional Government of Valencia in Spain (Resolución de la Conselleria de Innovación, Industria, Comercio y Turismo, Dirección General de Innovación). L.F. has also been partially funded by a grant from the Banc Sabadell Foundation.
During the preparation of this work, the author(s) used Generative AI to check grammar, formal tone, and spelling. After using these tool(s)/service(s), the author(s) reviewed and edited the content as needed and take(s) full responsibility for the publication’s content.
The experimental setup consists of multiple configurations based on diferent task types, segmentation methods, adaptation strategies, and model choices. Table 3 provides an overview of the experiment variants. The tasks include argument segmentation, argument type classification, and argument quality assessment. Two segmentation approaches are considered: gold segmentation (ground truth) and inferred segmentation (automatically performed by the model). The setup can be either individual, where argument type and quality are classified separately, or joint, where both are assessed together. We explore two adaptation strategies: few-shot learning (ranging from zero to four-shot) and fine-tuning. The models used in the experiments fall into three categories: encoder-based models, small open-source LLMs, and a proprietary model, GPT-4o mini.
Category
Task Segmentation
Setup Adaptation strategy Models
Options - Argument segmentation - Argument type classification - Argument quality assessment - Gold - Inferred - Individual: type or quality - Joint: type and quality - Few-shot (zero to four-shot) - Fine-tuned - Encoder-based: - BERT - BERT with two heads - Longformer - Longformer with two heads - Small open source LLMs: - Llama 3.2 3B - OLMo 2 7B - Qwen 2.5 7B - DeepSeek R1 7B - Llama 3.1 8B - Gemma 2 9B - GPT 4-o mini
The following encoders are used for diferent task variants, following the approach proposed by Ding
et al. [
The training was conducted for 10 epochs and using the same specific setting used in Ding et al. [
Few-shot prompting queries Complementing the Section 3.2 in the main paper, Table 4 presents the expected output format for diferent tasks, while Table 5 lists the queries used to prompt the LLMs.
Tasks
Type
Quality Type and Quality
Few-shot output format
{"TYPE": ["Position"]} {"QUALITY": ["Adequate"]} {"TYPE AND QUALITY":
["Position", "Adequate"]} #TASK: Segment the following essay into distinct argument components. After each argument component, insert the marker <SEP>. Keep the original text in the same order without adding, removing, or altering any words (other than inserting the <SEP> markers). #GUIDELINES: Identify each coherent segment that forms a logical unit of the argument (e.g., claims, premises, evidence, or conclusions).
You are a strict AI evaluator specializing in detecting the type of argument components in essays. The argument types are as follows: - Lead: An introduction that begins with a statistic, quotation, description, or other device to grab the reader’s attention and point toward the thesis. - Position: An opinion or conclusion on the main question. - Claim: A statement that supports the position. - Counterclaim: A statement that opposes another claim or provides an opposing reason to the position. - Rebuttal: A statement that refutes a counterclaim. - Evidence: Ideas or examples that support claims, counterclaims, or rebuttals. - Concluding Statement: A statement that restates the claims and summarizes the argument.
You are a strict AI evaluator specializing in assessing the quality of argument components in essays. Each component should be rated as one of the following: - Inefective: The component is unclear, unconvincing, or poorly structured. - Adequate: The component is understandable and somewhat convincing but lacks strong support or clarity. - Efective: The component is well-structured, clear, and strongly supports the argument.
Following the dataset split used in [
Tasks Segmentation
Type (Indiv.) Quality (Indiv.) Type + Quality
Input Format
Essay Essay + SEP Essay + SEP Essay + SEP
Target Format
Essay + SEP separator
Essay + TYPE separators
Essay + QUALITY separators Essay + TYPE and QUALITY sep.
Fine-tuning small, open source LLMs Qwen 2.5 7B, Llama 3.1 8B, and Gemma 2 9B were finetuned on a single GPU. We used the AdamW optimizer ( 1 = 0.9, 2 = 0.999) in 8-bit precision. A weight decay of 0.01 was applied to all weights except biases and normalization layer parameters. The mini-batch size was 2, and we opted for 12 warmup steps. Regarding QLoRA, the rank of LoRA modules, is 16, the LoRA scaling factor, is 16 with 0 dropout and 4-bit quantization. Gradient accumulation is set to 4, with a learning rate of 1 − 4 or 5 − 4 , depending on the model and setup configuration, and a cosine learning rate schedule. Thanks to the early stopping method, training terminated after approximately 100 to 400 steps.
Fine-tuning GPT-4o mini GPT-4o mini-2024-07-18 was fine-tuned, using the OpenAI fine-tuning platform11. The hyperparameters, number of epochs, learning rate, and batch size were automatically determined, resulting in 3 epochs, a learning rate of 1.8, and a batch size of 6.
Hi, i’m Isaac, i’m going to be writing about how this face on Mars is a natural landform or if there is life on Mars that made it. The story is about how NASA took a picture of Mars and a face was seen on the planet. NASA doesn’t know if the landform was created by life on Mars, or if it is just a natural landform. <SEP>. On my perspective, I think that the face is a natural landform because I dont think that there is any life on Mars. In these next few paragraphs, I’ll be talking about how I think that is is a natural landform <SEP>I think that the face is a natural landform because there is no life on Mars that we have descovered yet <SEP> [...] Though people were not satified about how the landform was a natural landform, in all, we new that alieans did not form the face. I would like to know how the landform was formed. we know now that life on Mars doesn’t exist. <SEP>
Table 7 presents an example of essay segmentation format used as few-shot and fine-tuned output format for the segmentation task and input format for the argument type classification task. Table 8, Table 9 and Table 10 present examples of the few-shot and fine-tuned output format used for the argument type classification and quality assessment tasks in the individual and join setups, respectively. Note that the tables retain spelling errors present in the original essay, as they directly reflect the source text. Hi, i’m Isaac, i’m going to be writing about how this face on Mars is a natural landform or if there is life on Mars that made it. The story is about how NASA took a picture of Mars and a face was seen on the planet. NASA doesn’t know if the landform was created by life on Mars, or if it is just a natural landform. <Lead>. On my perspective, I think that the face is a natural landform because I dont think that there is any life on Mars. In these next few paragraphs, I’ll be talking about how I think that is is a natural landform <Position>I think that the face is a natural landform because there is no life on Mars that we have descovered yet <Claim> [...] Though people were not satified about how the landform was a natural landform, in all, we new that alieans did not form the face. I would like to know how the landform was formed. we know now that life on Mars doesn’t exist. <Concluding Statement> Hi, i’m Isaac, i’m going to be writing about how this face on Mars is a natural landform or if there is life on Mars that made it. The story is about how NASA took a picture of Mars and a face was seen on the planet. NASA doesn’t know if the landform was created by life on Mars, or if it is just a natural landform. <Adequate>. On my perspective, I think that the face is a natural landform because I dont think that there is any life on Mars. In these next few paragraphs, I’ll be talking about how I think that is is a natural landform <Adequate>I think that the face is a natural landform because there is no life on Mars that we have descovered yet <Adequate> [...] Though people were not satified about how the landform was a natural landform, in all, we new that alieans did not form the face. I would like to know how the landform was formed. we know now that life on Mars doesn’t exist. <Inefective>
Hi, i’m Isaac, i’m going to be writing about how this face on Mars is a natural landform or if there is life on Mars that made it. The story is about how NASA took a picture of Mars and a face was seen on the planet. NASA doesn’t know if the landform was created by life on Mars, or if it is just a natural landform. <Lead, Adequate >. On my perspective, I think that the face is a natural landform because I dont think that there is any life on Mars. In these next few paragraphs, I’ll be talking about how I think that is is a natural landform <Position, Adequate> I think that the face is a natural landform because there is no life on Mars that we have descovered yet <Claim, Adequate> [...] Though people were not satified about how the landform was a natural landform, in all, we new that alieans did not form the face. I would like to know how the landform was formed. we know now that life on Mars doesn’t exist. <Concluding Statement, Inefective>
Handling incorrect outputs During inference, the LLM’s responses did not always adhere to the specified output format required to perform properly the argument mining tasks. In such cases, the inference was repeated up to five times. If the issue persisted, the argument (or the essay) was discarded. Examples of incorrect outputs Table 11 presents two examples of incorrect outputs generated with Llama 3.1 8B for the argument type task, along with their corresponding prompt. Note how in the first output, Llama 3.1 8B argues that it is not able to perform the task because it is too complex for it. In the second output, a part of the essay is missing: “The Electoral College is also a modern sense", and Llama 3.1 8B has omitted the classifications of some arguments.
LLMs, fine-tuned models required more inference retries and experienced higher failure rates compared to their few-shot counterparts. Furthermore, increasing the number of shots in the few-shot setting led to a higher number of retries and failures. Notably, Qwen 2.5 7B in the few-shot setting exhibited the Prompt“#QUERY: You are a very strict AI evaluator specializing in detecting the type of argument in essays. The argument types are as follows: - <Lead >: An introduction that begins with a statistic, quotation, description, or other device to grab the reader’s attention and point toward the thesis. - <Position >: An opinion or conclusion on the main question. - <Claim >: A statement that supports the position. - <Counterclaim >: A statement that opposes another claim or provides an opposing reason to the position. - <Rebuttal >: A statement that refutes a counterclaim. - <Evidence >: Ideas or examples that support claims, counterclaims, or rebuttals. - <Concluding Statement >: A statement that restates the claims and summarizes the argument. #TASK: Analyze the following essay and replace only each instance of <SEP>with the argument type of the sentence immediately preceding it. Do not insert any additional markers or segmentations beyond replacing <SEP >. If a sentence does not have <SEP>, leave it unmodified. #ESSAY: Dear,Florida I think that we should keep the Electoral College becuase it keeps the number of votes and it produces a clear winner. <SEP>The Electoral College restores some of the weight in the political balance that large states (by population) lose by virtue of the senate decreed in the constitution. <SEP>The Electoral College is a process that consists of 538 electors. Each state equals the number of members in the house of representives. The Electoral also select what their responsibilities are. <SEP>The Electoral college is widely regarded as an anachronism a non-democratic method. <SEP>The Electoral College is also a mordern sense. Also no region (south,northeast,etc.) has enough electoral votes to elect a president. Residents also feel that their votes dont get counted that the new president will have no regard for thier interest, that he really isnt their president. <SEP>Electoral College alson avoids the problem of elections in which no candidate receives a majority of the vote cast. <SEP>" Output 1 “I can’t help you with this task. This task requires identifying and labeling specific argument types in an essay, which is a complex task that may require human judgment and expertise. If you’d like to discuss this further, I can try to provide more information on the topic of argument type identification." Output 2 “After analyzing the essay, I will replace each instance of <SEP>with the argument type of the sentence immediately preceding it. Here is the modified essay: Dear, Florida I think that we should keep the Electoral College because it keeps the number of votes and it produces a clear winner. <Claim >The Electoral College restores some of the weight in the political balance that large states (by population) lose by virtue of the senate decreed in the constitution. <Claim >The Electoral College is a process that consists of 538 electors. Each state equals the number of members in the house of representives. The Electoral also select what their responsibilities are. <Counterclaim >The Electoral college is widely regarded as an anachronism a non-democratic method. <Position >Also, no region (south,northeast,etc.) has enough electoral votes to elect a president. Residents also feel that their votes dont get counted that the new president will have no regard for thier interest, that he really isnt their president. <Claim >Electoral College alson avoids the problem of elections in which no candidate receives a majority of the vote cast. Note: Some sentences don’t have <SEP>as they are not part of an argument sequence or are standalone statements." highest failure rate among all models.
To assess the eficiency of the proposed methods, we report the following metrics that consider the imbalance in the distribution of labels.
Metrics per label, namely precision, recall, and F1 score for each individual label to have an in-depth look at the classifier’s performance on a per-label basis.
Multi-label confusion matrix to provide a detailed breakdown of model performance across argument categories, highlighting both accurately predicted cases (along the diagonal) and common misclassifications. The values in the matrix give insights into which argument types or quality assessment labels the proposed method distinguishes efectively and where it struggles.
The macro-averaged F1 score is the mean of the F1 scores for each label, treating all labels equally. Thereby it provides a measure of overall performance across all labels without considering label imbalance.
For classes, the macro-averaged F1 score is 1 = 1 ∑︀ =1 1,, where the F1 score of class is given by: 1, = 2 · Precision · Recall Precision + Recall , with Precision and Recall representing the precision and recall scores of class .
Table 12 depicts the precision, recall, and F1 score for the argument segmentation task, and Table 13 the corresponding confusion matrix.
Segm. Precision Recall F1-score B 66.01 88.32 75.55
I 99.79 99.18 99.48
Llama 3.1 8B fine-tuned shows very strong performance in predicting the I label (precision, recall, and F1-score are above 99%). However, for the B label (beginning of an argument), while recall is quite high (88.32%), precision is noticeably lower (66.01%), suggesting the model sometimes over-predicts beginnings. The confusion matrix confirms that the model occasionally misclassifies B tokens as I, but rarely the other way around. Overall, this points to strong segmentation performance, with the main challenge being the precise identification of the beginning of an argument (B).
Figure 4 shows the overlap with the gold segmentation and predicted segmentation across models.
As a complementary analysis of Table 1 in the main paper, Table 14 displays the confusion matrix using the fine-tuned Llama 3.1 8B model on the joint setup with the segmentation inferred by the model. Note that the entire Echec row consists of zeros as this label is not present in the ground truth dataset, and it is counted when the predicted argument does not match the ground truth argument.
Type Lead Lead 10.33 Pos 0.00 Clai 0.00 C Clai 0.00 Reb 0.00 Evid 0.33 Ccl 0.00 Ech 0.00
As a complementary analysis of Table 2 in the main paper, Table 15 displays the confusion matrix of ifne-tuned Gemma 2 9B on the joint setup with the segmentation inferred by the model.
Quality Inefective Adequate Efective Echec
Inefective 32.33 72.67 4.00 0.00
Adequate Efective 16.00 2.00 95.22 10.67 40.33 15.00 0.00 0.00
Echec 26.67 86.33 34.67 0.00
Data quality issues—such as inconsistent annotations, missing data, or biased labeling—can skew model performance and reduce the reliability of automatic essay assessments, leading to inflated or misleading metrics and limit the generalization of results. In our experiments, we identified noise in the annotations. Hence, future research should aim to address these limitations by improving annotation guidelines, enhancing inter-rater reliability and expanding the available dataset to include a more diverse range of essays.
We evaluated three recent small open-source LLMs on the argument type classification task, using
three-shot prompting. Table 16 contains the macro-averaged F1 score for the following models:
• Llama 3.2, 3B. Llama 3.2 a multilingual auto-regressive language model which uses an optimized
transformer architecture, released in September 2024 by Meta. See https://ollama.com/library/
llama3.2:3b.
• OLMo 2 7B. OLMo 2 is the latest iteration of the fully open language model, featuring dense
autoregressive models with enhanced architecture and training methodologies, released in
November 2024 by the Allen Institute for AI [
It supports eight languages and ofers a context window of up to 128,000 tokens, enabling it to
handle extensive conversational contexts. Released in July 2024 by Meta [
Additionally, we tested Mistral v 0.2 12 and Falcon 3 13. However, the majority of their outputs did not conform to the expected format, making it impossible to evaluate their performance.
Model Llama 3.2 3B OLMo 2 7B
Qwen 2.5 7B DeepSeek R1 7B
Llama 3.1 8B Gemma 2 9B