With the growing adoption of large language models (LLMs) in educational settings, there is an urgent need for systematic and scalable evaluation methods. Traditional natural language generation metrics such as BLEU, ROUGE and METEOR excel at measuring surface‐level linguistic quality but fall short in evaluating the interactive, adaptive nature of dialogue alignment of conversational agents, particularly in relation to their intended design. To address these gaps, we propose an evaluation strategy that extends beyond technical evaluation (linguistic coherence and semantic relevance). In this pilot study we compare human and LLM-based evaluation of a conversational agent, with a focus on Socratic dialogue as a specific instantiation. Early results indicate that our LLM-as-a-Judge aligns closely to human evaluators for clear, surface‐level qualities like encouragement and actionable guidance, but less on subtle pedagogical behaviours such as recognising errors and maintaining natural dialogue flow. These early results underscore the promise of LLM-based evaluators for scalable assessment of tutoring behaviours while highlighting the need for targeted fine-tuning and hybrid approaches to improve nuanced error detection and dialogue coherence.
With the rapid advancement of artificial intelligence (AI) in education, there has been an increasing interest in developing intelligent tutoring systems to enhance and improve students’ learning experiences. One such approach is an AI tutor, which employs the Socratic dialogue method, i.e., encouraging students to think critically by asking thought-provoking questions rather than providing direct answers [1, 2, 3]. Unlike conventional tutoring models that rely on direct explanations or answer provision, Socratic tutors employ dialogue to guide students through self-discovery, prompting reflection and exploration of underlying concepts.
Despite advances in natural language generation (NLG) and large language models (LLMs), existing evaluation metrics such as BLEU, ROUGE, METEOR, and BERTScore predominantly assess surfacelevel linguistic quality and semantic similarity. While these metrics efectively measure fluency and coherence, they do not capture the adaptive, interactive, and pedagogical nuances inherent in Socratic tutoring. To bridge this gap, we introduce a pedagogically grounded evaluation framework tailored to AI tutors deploying Socratic dialogue.
Our contribution is twofold: first, we propose a rubric that operationalises key dimensions of Socratic
CEUR
ceur-ws.org
The AI tutor is built on the principles of Socratic teaching — an approach that uses questioning and dialogue to guide students toward deeper understanding, commonly associated with the objective of developing critical thinking, rather than directly lecturing or providing answers [1, 2, 4]. This AI tutor acts as a teacher’s assistant, helping to extend the teacher’s ability to ask questions while the students are working. By having these conversations, the teacher is able to know the magnitude of a student’s understanding in a particular subject or topic and uncover what their misconceptions are. A key limitation of doing this manually is the enormous amount of time and efort required by teachers to participate in these conversations.
The evaluation of large language models (LLMs) has advanced considerably, particularly in their application to AI tutoring systems. Early approaches relied on traditional natural language generation (NLG) metrics, such as BLEU and ROUGE, which are efective in assessing fluency, but insuficient in capturing the pedagogical efectiveness of AI tutors [ 5]. To address these shortcomings, Maurya et al. [6] introduced the Unified Evaluation Taxonomy, a framework for systematically assessing AI tutors based on eight pedagogical dimensions: mistake identification, mistake location, revealing the answer, providing guidance, actionability, coherence, tutor tone, and human-likeness. This taxonomy provides a standardised approach to evaluating AI tutors and aligns with established educational best practices.
Automated metrics provide quantitative tools to assess the technical quality of dialogue systems, including AI tutors. In assessing the quality of Socratic dialogues, Favero et al. [7] used BERTScore [8], which employs contextual embeddings to evaluate semantic similarity, ofering a deeper evaluation of meaning compared to BLEU [9], ROUGE [10], and METEOR [11] Another recent metric used is DialogRPT, which is tailored for conversational AI. It evaluates response coherence and human-likeness, making it particularly relevant for dialogue-based tutoring systems [6]. This metric was also used with BERTScore in a recent shared task in 2023 to automatically evaluate teacher-student dialogue [12].
These metrics can be valuable for assessing the technical aspects of the responses of AI Socratic tutors. However, these automated NLP metrics have limitations when it comes to assessing the pedagogical aspects efectively and the interactive nature of AI Socratic tutors (this was done manually in the aforementioned shared task [12]). Moreover, many of these metrics are based on a comparison with predetermined reference answers, which may not be entirely suitable for Socratic dialogue, where the learning process and the individual exploration of the student are more important than arriving at a specific answer [ 13]. While technical metrics ofer a valuable assessment of the surface-level linguistic quality of the dialogue, they lack the capacity to evaluate the pedagogical impact of the Socratic interaction.
To address the limitations of technical metrics, pedagogical frameworks and taxonomies ofer structured ways to evaluate the educational efectiveness of AI tutors employing Socratic dialogue. One notable approach is the eight dimensions taxonomy for pedagogical abilities proposed in recent research [6]. Grounded in learning sciences principles, this taxonomy provides a standardised approach to evaluating key aspects of the tutoring interaction and the AI’s ability to efectively address student errors. Each dimension is evaluated using a three-tier labelling system (Yes, To some extent, No), allowing for nuanced assessment of the AI tutors’ pedagogical abilities.
Automated technical metrics ofer valuable quantitative data regarding the linguistic quality of the dialogue, including aspects such as fluency, coherence, and semantic similarity [ 14]. On the other hand, pedagogical frameworks provide qualitative insights into the learning process, focusing on the AI’s ability to guide student thinking, foster critical reasoning, and promote deeper understanding. Human feedback also plays a crucial role in evaluating aspects that automated systems may find challenging to assess, such as the depth of critical thinking stimulated by the interaction and the overall learning experience from the student’s perspective [15].
Adapted from Maurya et al. [6] and learning sciences literature (e.g., [16, 3]), we propose the following set of dimensions, metrics and questions that form our proposed tentative rubric framework (Table 1) for automated evaluation of AI tutors.
Our rubric operationalises Socratic tutoring principles by breaking them down into four overarching dimensions—Mistake Remediation, Scafolding, Guidance and Actionability, and Coherence and Tone—each with specific metrics and binary (Yes/No) judgments, with “NA” (Not Applicable) as an option for Mistake Remediation (when there is no mistake that needs to be remedied).
In our case study, we evaluated the AI tutor built on an EdTech platform supporting students in building concept maps. This tool facilitates the students to define nodes and relationships (concept mapping) based on a topic and/or subject. In this exercise, the students are asked to build and link nodes regarding the topic of “How do we move our school paper registers to an IT system?”.
We created the dialogue turns using OpenAI’s GPT-4o to reflect two contrasting tutor styles (Socratic vs. non-Socratic). For each stage of the tutoring process—Early, Mid, and End—we generated 15 turns per dialogue snippet in each style, yielding a total of 90 turns.
To ensure our evaluation is robust and generalises beyond a handful of canned examples, we deliberately built variability into both content and form. By sampling multiple dialogue variants, we can collect a distribution of rubric scores—both from human raters and from an LLM-as-judge—and examine not only mean performance but also the spread of scores. A narrow spread would imply consistent rubric compliance, whereas a wide spread might reveal varying rubric conformance or tutor strategies that only work in certain phrasings. This systematic injection of variability allows us to stress-test each rubric dimension and to ensure that our evaluation is assessing the functionality of the tutor’s moves—not just the superficial wording.
In the system prompt for both versions, the following criteria were explicitly mentioned: 1. Not all metrics presented in our proposed rubric should be satisfied. 2. Each scenario reflects diferent stages in the concept mapping exercise (i.e., Early, Mid, and End). 3. Add variability in some stages where the tutor fails on some metrics, particularly on the version where the tutor is not being Socratic.
The following shows an excerpt of the Early stage dialogue between a student and the AI tutor: [Start of dialogue] Tutor: What do you think is the main problem with using paper registers in your school? Student: They’re slow and sometimes teachers forget to fill them out.
Tutor: That’s a good observation. How might that affect the school as a whole? Student: Maybe it causes delays in knowing who’s present or absent.
Tutor: Great. Can you think of what kinds of data a digital register system would need to handle? ...
The conversation data used for this evaluation can be found at https://zenodo.org/records/15724072.
A total of twelve independent human evaluators scored the dialogue snippets. For highest level of
education, ten hold postgraduate degrees (Master’s or PhD) and two hold Bachelor’s degrees. Their
self-reported familiarity with the Socratic method ranged from “Not familiar” (5 raters) through “Heard
of it” (
We prompted a task-specific LLM—LearnLM 2.0 Flash Experimental 1—to act as an evaluator using the same rubric metrics. LearnLM is an experimental task-specific model that has been trained to align with learning science principles. The concept and implementation of LLM-as-a-Judge is not particularly new in the field of automated evaluation [ 17, 18]; however, in tasks that require specialised knowledge, the validity of using LLMs as judges remains uncertain, and human experts are still necessary to be in the loop [19]. For each dialogue scenario, we constructed a prompt of the form:
System: You are an educational evaluator of an AI tutor. The following is the rubric you will be using to evaluate the tutor's response. <rubric> The following is the current dialogue snippet from [Early/Mid/End] stage of the exercise. <dialogue> Task: Given the student’s and the tutor’s response, answer the following questions. The answer options are shown in square brackets for each question: <question> 1Technical report: https://goo.gle/LearnLM-dec24, API docs: https://ai.google.dev/gemini-api/docs/learnlm
We ran the prompt in a zero-shot setting and collected the model’s scoring. We then measured alignment with the human annotations via exact match rate and Cohen’s κ per metric, and analysed common error patterns to identify areas for prompt or model improvement. We chose a diferent LLM to the synthetic sample creation given the inherent bias that exists in LLMs evaluating own generation [20].
Figure 1 shows the box plot of Cohen’s κ value between the human evaluators and the LLM-as-a-Judge.
The box plot highlights notable diferences in how well our LLM evaluator aligns with human judgments across the six metrics. At one end of the spectrum, Encouraging tone shows near-perfect agreement (mean κ ≈ 0.97), with nearly all values clustered at or very close to 1.00, indicating that the model almost always matches human evaluators in detecting a supportive, positive tone. Similarly, Provision of guidance to move to the next step yields strong consensus (mean κ ≈ 0.83), with most scores between 0.67 and 1.00 and only a single outlier at 0.00.
In contrast, the model struggles with the more nuanced task of mistake remediation: both Identifying mistake and Acknowledging mistake hover around low medians (≈ 0.20 and 0.28, respectively), feature wide ranges extending slightly below zero, and even exhibit negative outliers—signalling occasional direct disagreement with human raters. Provision of appropriate support without giving answers directly (median κ ≈ 0.62) shows generally reliable but imperfect alignment, while Natural-ness of conversation flow proves the most variable metric (mean κ ≈ 0.39, full range 0.00–1.00), suggesting that judgments of conversational coherence are highly subjective and context‐dependent.
Overall, these results indicate that our LLM evaluator excels at scoring clear, surface‐level qualities like encouragement and actionable guidance, but faces challenges in replicating human sensitivity to subtle pedagogical behaviours such as recognising errors and maintaining natural dialogue flow.
Despite these insights, our study has several limitations. First, our evaluation relies on a small, synthetic corpus of concept-mapping dialogues within a single domain, which may not generalise to other subjects or authentic classroom settings. Second, the binary Yes/No rubric captures only the presence or absence of target behaviours and cannot distinguish varying degrees of instructional quality or nuance. Regardless, this was intentional to ensure that for the pilot study, the LLM responses can only be chosen from a set of options (for uniformity and constraint in LLM response). Third, while our LLM-as-a-Judge leverages an education-tuned model, we evaluated it under zero-shot prompting rather than few-shot or comprehensive fine-tuning on pedagogical annotations, which likely constrained its sensitivity to subtle teaching cues. Finally, variability in human annotator backgrounds and interpretations introduces additional noise, prompting the need for clearer guidelines and more diverse evaluator pools in future work.
In conclusion, early results from our pilot study suggests that LLM evaluators may efectively and eficiently assess clear, surface-level tutoring behaviours but currently fall short on nuanced error recognition and conversational coherence. Future work will focus on further refining the rubric framework, targeted fine-tuning, improved prompting strategies, and hybrid human–LLM workflows to enhance subtle pedagogical judgments. Additionally, extending evaluations to other subject domains and conducting longitudinal studies on student learning outcomes will be critical for building robust, generalisable assessment frameworks.
Part of this work was completed when one of the authors (MD) was on a “Faculty fellowship” with the EdTech startup, Grafinity ( https://graffinity.io/), as a supplement to the their “Contracts for Innovation: AI tools for education” catalyst project funded by the UK Department for Education. We would also like to acknowledge Matthew Pryor (Grafinity) and Harry Moss (UCL’s ARC - Advanced Research Computing centre) for feedback and support in related work during the same project. ZP’s contribution was co-funded through UCL’s Global Engagement Seed Funding and the TransEET project (https: //transeet.eu/) in the context of the AIED Community of Interest (COI). TransEET is HORIZON-WIDERA2021-ACCESS-03-01 project (Grant no. 101078875) funder by the EU and by the UK Research and Innovation (UKRI) under the UK government’s Horizon Europe Guarantee funding scheme. MM’s contribution was partly supported from the CHAILD project - Children’s Agency In the age of AI: Leveraging InterDisciplinarity (UKRI ref. MR/Z505882/1). The views expressed in this publication are those of the authors and not necessarily those of their institutions or the funders.
The authors have not employed any Generative AI tools in the writing of this paper.