Artificial intelligence (AI) in education has the potential to personalize learning, but trustworthiness is essential for its successful implementation. The present paper provides an overview of the challenges identified during the development and evaluation of the Trust-AI platform. A number of trustworthiness challenges have been identified related to the validity and reliability of AI assessments, explainability of AI models, fairness of AI algorithms that are prone to bias, and overall safety, privacy, and security of our Trust-AI platform. To address these challenges, this paper suggests strategies we have implemented during the design and development of the Trust-AI platform, as well as the mitigation actions we plan to implement during the deployment phase of our platform. The paper concludes that the implementation and deployment of trustworthy AI systems in education needs to be driven by an ongoing, comprehensive dedication to trustworthiness assessment based on ethical values and stakeholder involvement.
The potential of implementing AI in Science, Technology, Engineering and Mathematics (STEM)
education is tremendous; AI has the potential to bring revolutionary changes in the learning process.
AIenabled learning tools ofer new capabilities in STEM learning, such as personalized learning, immediate
feedback, and representation of concepts in exciting and interactive methods, which help to personalize
the learning process to individual students [
As AI tools are growing more complex, it is not only used in basic grade calculation but is now used
in smart tutoring [
This study aims to help advance the discussion of trustworthy AI in STEM education. The focus
of our research is to identify the AI trustworthiness challenges associated with the Trust-AI platform
[
This section presents the methodology (Figure 1) that we followed to identify the challenges related to trustworthiness of our Trust-AI platform. The approach follows a multi-phase process: first, core principles were defined based on a literature review; next, stakeholder workshops captured trust-related challenges; and finally, system demonstration and evaluation involved teacher interaction, feedback collection, and validation of AI features.
Our research started by conducting a systematic literature review to determine the current challenges
of trustworthy Artificial Intelligence in education [
In the third phase of our methodology, the Trust-AI platform was demonstrated and explained during three workshops, involving 60 teachers, allowing them to have a first-hand understanding and evaluation of the system and its contribution to typical STEM teaching activities. With the Trust-AI platform, each student follows a customized learning pathway suggested by the AI assistant, which also corrects misconceptions and ofers immediate feedback. This personalized interaction supports diferentiated learning and encourages student autonomy while maintaining teacher oversight. By facilitating one-on-one interactions between students and tutors, this setup aims to improve the usual student-teacher dynamic and foster deeper learning. Additionally, teachers can track students’ progress in real time, which enables them to ofer tailored advice and step in when needed.
Additionally, the Trust-AI platform includes an AI learning pathways recommender system that lfags underperforming activities using student performance data. Teachers can review these recommendations, so that they can either accept or reject them, assessing their alignment with educational standards and pedagogical goals. Their feedback is recorded and used to continuously refine the AI system through reinforcement learning, forming a human-in-the-loop architecture where educators remain central in decision-making.
Following the workshops and in order to uncover more information, we held interviews with teachers to elicit their opinion and evaluate trustworthiness characteristics of the Trust-AI platform. To quantify teachers’ feedback, we also asked them to complete a structured questionnaire to gauge the perceived trustworthiness of AI efectiveness as well as perceived risks, such as over reliance, academic integrity, and data privacy. The combination of literature-grounded design, stakeholder consultation, live system evaluation, and teacher-centered feedback mechanisms enabled a comprehensive exploration of the challenges and opportunities in building a trustworthy AI system for STEM education.
This section discusses the challenges related to AI trustworthiness as identified in the Trust AI platform.
We group the identified challenges under the seven characteristics of trustworthy AI systems defined
by the NIST Framework [
Validity and reliability were prioritized as the most important characteristics to ensure the trustworthiness of our platform. Failure in this area can lead to misleading guidance among students. Specifically, a major risk for the Trust-AI platform is its potential inability to accurately assess the level of a student’s problem-solving skills. In case the system provides assessments that are not correct, teachers will not trust it. Teachers will not use the system if they lack confidence in it. Furthermore, inaccurate evaluation of student performance may result in recommendations for personalized learning that are not reliable. These problems have the potential to interfere with the learning process and produce unreliable educational results.
This risk of inaccuracy extends to all functions of the Trust-AI platform. The ability to suggest personalized learning paths depends entirely on the accuracy of problem-solving skills assessment. An inaccurate assessment can lead to learning recommendations that are not relevant to the students’ needs. Similarly, feedback from the system should be consistently accurate and personalized for student learning needs. Unreliable or inaccurate feedback will result in students not trusting the system.
Equally important is the robustness of the Trust-AI platform. Robustness is the feature of the system to work with the same and reliable results in diferent classroom conditions, incomplete student inputs, and other unpredicted situations. Without a robust platform, there will be a possibility of having unstable or inconsistent outputs given unexpected inputs and variations in classroom conditions. These failures may disrupt the process of learning by providing incorrect suggestions, lowering the accuracy of feedback, and eventually making teachers and students unwilling to use the platform.
When an AI’s decision-making process is a ‘black box’, it creates a fundamental barrier to trust. Educators cannot be expected to implement recommendations without understanding their rationale. A significant challenge is the erosion of teacher trust. If a teacher cannot understand why the AI recommended a specific learning path, they cannot professionally endorse it or integrate it into their teaching. An AI learning pathway recommendation without justification is not useful. If the system flags an activity as ‘at-risk’ without explaining which concepts they are struggling with, the teacher has no basis for intervention. This lack of justifying a specific decision and clarity about the overall system design and data creates an ‘accountability vacuum’ where no one can be held responsible for the AI’s outcomes.
In education, safety is related to the well-being of students and teachers. One relevant identified challenge is the fear of educators that their role may be replaced by the system. The feeling of potential loss of jobs due to AI performing and taking control of the essential teaching aspects encourages professional insecurity, causing teachers to distrust and resist the emerging technology in the educational ifeld.
Another challenge related to safety is that over-reliance on AI for basic tasks such as lesson planning and assessment could cause reduced teacher involvement in the educational process. The blindly outsourcing of the basic activities to the AI system threatens the phenomenon of de-skilling. This may mean that pedagogical competence and creativity may be impaired in educators, and this compromises the teaching process and reduces the outcome of learning in the students.
One of the major challenges regarding the implementation of AI into education concerns transparency and accountability. One of the most important elements of the educational setting is transparency, which encourages trust needed among students, educators, and institutions. Yet, this contradicts this principle directly since the AI algorithm is usually opaque, as is the case with highly advanced AI algorithms. With our Trust-AI platform, it is important for teachers to understand the logic behind the algorithm. There is a strong need that teachers to understand how AI models are assessing the problem-solving abilities of students and what data have been used to train the AI model so as not to reproduce the biases built into the system. Moreover, at the central level, there must be transparency in order to allow a clear chain of accountability. When an algorithm fails or produces a biased result, the black box characteristic of a system is not known, and it is unclear who should be held liable between a teacher or the AI developer. This confusion is not only an undermining factor of trust, but it is also a violation of procedural fairness.
Privacy protection of students’ behavior data, skills, and performance was raised as a major concern
by teachers. The NIST Privacy Framework [
Security is one of the pillars of trust in any education technology infrastructure, and the challenges it poses must be dealt with in a multidimensional manner. Although our platform is specifically built not to collect any personally identifiable student data, and hence addresses some of the most serious challenges, there are still some security concerns, particularly as far as application of an efective access control model is concerned. Since our platform works with anonymized data of interaction and assessment, it is necessary to prioritize data integrity, as well as the right to access information.
In a multi-user environment with diferent roles of users, like in our case where we have students, teachers and administrators access rights, incorrect access rights may create vulnerabilities that are cause of great concern to security. The possibility that the assessment results will be seen by the students is a serious one. Even anonymization of data may result in performance anxiety when students are afraid to make errors because of this perception. Also, presentation of assessment data to the students may result in inadvertent peer comparison or the sense of exposure, which interferes with psychological safety among students. Thus, a well-defined access control mechanism cannot be seen as a system feature, but rather a basic requirement to address the confidentiality of data, maintain the integrity of a system, and achieve user confidence.
Moreover, most systems containing student data are vulnerable to attacks. These risks include overt threats such as data breaches, which can seriously damage user trust, as well as more subtle forms like data poisoning, where malicious inputs manipulate the behavior of AI systems. However, our platform is less likely to face such severe consequences. Since the Trust-AI platform is designed to support the educational process, assisting students and teachers without storing sensitive personal data, it presents a lower-value target for potential attackers.
In educational AI systems, fairness is crucial as any bias can reinforce or create inequalities, disproportionately afecting students based on diferent factors. Since our platform is not intended to hold demographic or any other sensitive personal data, direct discrimination by protected characteristics is automatically eliminated. Nonetheless, the issues of fairness remain on a more insidious, algorithmic level. One possible issue is bias in algorithms. This means that the AI models may be trained on biased data in the background. These may cause the system to be biased towards certain forms of expression or approaches to solving problems at the expense of the students who have alternative forms.
To build a trustworthy AI system in the education domain, a wide range of mitigation actions must be taken across the diferent phases of the AI lifecycle. In this section, we present the strategies we have implemented during the design and development of the Trust AI platform, as well as the mitigation actions we plan to implement during the deployment phase of our platform to address the challenges described in the previous section. assess- Accuracy of assessment Correlate outcomes with Unreliable educational reand benchmarking of PISA; continuously moni- sults; misleading feedback; results; Integration of tor student performance; loss of teacher trust; withHuman-In-The-Loop ap- teacher review of AI out- drawal of using the platproaches comes; use accurate inter- form
action data Promote transparency Display performance metand implement explain- rics and explanations linked ability features to proposals; allow human oversight
Lack of explainability and accountability; teachers may not trust system decisions Apply a privacy-by- GDPR-aligned data prac- Legal noncompliance; misdesign approach tices; secure anonymization use of data; trust erosion
of student data Ensure fairness across student performance categories
Manual oversight of Systematic exclusion of flagged issues; reinforce- learners; amplifying inment learning with human equities in outcomes feedback Security nesses weak- Design for minimal risk
of data exposure Teacher overreliance on AI
Maintain human oversight and pedagogical control
Clear access rules; data integrity enforcement; limited attack surface; adopt proactive security measures
Data breaches; unauthorized access; loss of stakeholder confidence Keep teacher in-the-loop; Reduced teacher agency promote critical assessment and resistance to system of AI; ofer AI training adoption; poor educational outcomes
Table 1 presents an overview of the main trustworthiness challenges identified, paired with a corresponding strategy, concrete mitigation measures implemented within the platform, and the potential risks if these issues are not addressed.
Validity in educational AI systems refers to the degree to which the system’s outputs, such as student assessments, feedback and content adaptation, accurately represent the intended learning outcomes. Ensuring validity is crucial when AI is used to estimate complex traits like problem-solving skills or conceptual understanding, as flawed evaluations might mislead both educators and students.
One prominent challenge related to the validity and reliability of our platform is whether the
classification of students who used the AI system into high, low, and moderate performers has been
done correctly. To confirm the validity of our student performance categorization, we will constantly
monitor and assess students’ progress in consecutive STEM lab courses. Additionally, the classification
of the students will be compared to world-renowned models. In particular, our assessments of
problemsolving abilities will be correlated with the performance indicators identified in the PISA framework of
the OECD Program for International Student Assessment (PISA)[
In addition, human-in-the-loop approaches for teacher oversight should be established. That way, the educators can revise and certify AI results and take over control of the process when they believe their professional discretion overrides the technology output in order to control quality and align the technology with their professional judgment. To address this, the platform allows AI suggestions to be monitored in real-time, and the educators are ofered the analytics dashboard that displays the visualization of the student’s performance and thus promotes the process of formative validation. Lastly, through providing educators with a choice to accept AI recommendations or reject them, the validity is once again strengthened so as to coordinate the automated products with the pedagogical intentions and the practical observations in a classroom environment.
The system should be highly explainable and transparent as a way of demystifying the black box and gaining confidence. The important step is to integrate explainable AI techniques, which give clear and human understanding explanations of the results of the system. This enables the teachers and students to know the reasons that a certain learning path was suggested, which is central to establishing trust. In our system, we have implemented explainability mechanisms that provide insights into the reasoning behind recommended learning pathway changes. By analyzing and displaying student performance data from prior implementations, our platform provides an explainability mechanism (Figure 2). The system presents aggregated performance metrics for each activity and highlights risk indicators when an activity either poses consistent challenges or lacks suficient challenge for learners. This makes the rationale behind each recommendation explicit and provides teachers with transparent evidence supporting AI-generated suggestions. Specifically, the system ofers teachers explanations as to why it identifies an activity as dificult for students and suggests that it needs to be modified. For every AI-generated activity suggestion, specific performance metrics are linked and made fully visible to the reviewing teacher. Educators can accept or reject these proposals, with each decision logged alongside the reviewer’s identity and a timestamp, supporting human oversight and auditability.
To make sure the AI system does not become a crutch instead of the helping tool, a culture of responsible use has to be established. Both students and teachers should have clear and explicit policies outlining the efective use of AI and establishing acceptable limits between the use of AI and another form of academic dishonesty. This needs to be followed by AI training for teachers, which goes beyond technical knowledge to encourage the ability to critically assess the AI results and the limitations of the technology. The system itself is created in such a manner that it enhances and does not substitute for a person and his abilities. The AI can take certain administrative duties conventionally assigned to educators so that these professionals can work on the duties that only humans can perform. The concern with the integrity of the educational process also lies in the fact that it is essential that a human be in a loop during critical decision-making.
The security of student data is not negotiable and needs privacy by design strategy. This implies incorporating strong privacy-enhancing technologies at the beginning such as end-to-end encryption and strict regimes of data minimization to collect only the data that is strictly necessary. The system should be fully compliant with data protection laws such as GDPR and should have clear policies per user that allow him or her full control over the data. The school districts are supposed to maintain ultimate ownership of all the data of the students, and the contracts may not allow the vendor to use the data for other unintended purposes. There is also the need to adopt proactive measures such as giving the system frequent and stringent security scrutinizes, penetration, and malicious resistance testing to guard against external risks to security, such as the ability to deport venerability and counter malicious attacks that may occur.
A privacy-by-design approach has been developed, minimizing data collection to only what is necessary for its operation. No student data are collected or stored, and student interactions are linked to system-generated identifiers. Teacher accounts require names and emails for authentication on the platform. This data is collected with explicit consent during account creation, under clearly defined terms of use and data handling policies. All teacher data are securely stored, access-controlled and never shared with third parties.
The solution was designed with security in mind and thus no identifiable student data is stored or processed. All data of the interaction is anonymized and can only be linked with machine-generated identifiers, keeping no sensitive information revealed and stored. With the implementation of this architecture, the threat of being victimized by the misuse of data or use of data to expose their identity as a result of breach of security is extremely minimized.
The willingness to implement fairness demands the initiative to investigate and detect bias in all phases of the AI lifecycle. It is important to integrate bias detection algorithms and conduct regular bias audits that can identify and correct discriminatory patterns. Using diverse, and representative data to train the models is necessary to avoid a possible introduction of bias into the system in the first place. Last but not least, a multi-stakeholder AI ethics review board can ofer essential guidance to keep the development of fair AI practices.
Our solution, without relying on demographic or personal identifiers, supports equitable learning experiences by dynamically analyzing student performance. Interactional data such as correctness, and timing are the only factors on which all AI recommendations and analytics are made, which lowers the probability of bias caused by background knowledge and knowledge. Moreover, the system combines category-sensitive performance analysis and identifies activities that discriminate against any performance groups, actively managing equity. Educators maintain absolute control over identified problems and suggested solutions and play the role of a human-in-the-loop to approve or ignore AI recommendations. Reinforcement learning in taking a decision to either approve a proposed change or even reject it encourages fairness as well as reduces algorithmic biases.
In this paper we presented a comprehensive analysis of the AI trustworthiness challenges associated with the Trust-AI platform identifying significant challenges across validity, reliability, explainability, fairness, safety, and privacy. To address these issues, a wide range of mitigation actions must be taken during the design and development of our platform. We also presented several strategies for addressing the identified challenges, as well as the mitigation actions we plan to implement during the deployment phase of our system to address the identified challenges.
Our study is limited in scope by focusing on one specific AI system; still, it helped us identify several empirical findings and draw the following conclusions that may help educational institutions to promote an environment of trust, transparency, and accountability in AI-driven education. Building and sustaining a trustworthy, educational AI system requires a framework that incorporates validity assessment, explainable AI methodologies, bias detection algorithms, human-in-the-loop approaches for teacher oversight, and teacher training. Moreover, the successful integration of AI in education is not merely a technical achievement but an ongoing commitment to ethical principles and stakeholder collaboration, ensuring that technology serves to empower, not undermine, the fundamental goals of learning.
The work presented here is funded by the European Union’s Horizon FAITH project (Fostering Artificial Intelligence Trust for Humans towards the optimization of trustworthiness through large-scale pilots in critical domains), Grant agreement No: 101135932. The work presented here reflects only the authors’ view and the European Commission is not responsible for any use that may be made of the information it contains. During the preparation of this work, the author(s) used ChatGPT in order to: Grammar and spelling check. After using these tool(s)/service(s), the author(s) reviewed and edited the content as needed and take(s) full responsibility for the publication’s content.