This study aims to explore the potential and limitations of large language models (LLMs) such as ChatGPT, Gemini, and Copilot, in the context of chemistry education. The primary objective of the study is to compare the efectiveness of LLMs in solving chemistry tasks and to identify the key challenges associated with their implementation in education. These LLMs were selected based on a survey of students which indicated their widespread use due to their free accessibility. To evaluate the potential of LLMs in chemistry education, we employed them to solve tasks corresponding to diferent levels of knowledge in diferent subfields of chemistry. A comparative evaluation of LLMs' performance against that of average Ukrainian students was conducted. The results indicate that while LLMs show promise mainly in tasks not demanding deep logical reasoning, they are generally inferior to students. Key challenges in using LLMs in chemistry education identified include understanding the nuances of chemistry as a complex and multifaceted science, abstract concepts used in chemistry, recognition of chemical compound formulas, chemical reaction equations, limitations in logical reasoning, language barriers, and the occurrence of AI hallucinations. Additionally, there is a need for students to develop skills in crafting efective queries and prompts to enhance the eficiency of working with LLM. While LLMs are promising, their implementation requires addressing the identified limitations.
A characteristic feature of modern times is the continuous dynamics of change in all spheres of human
life: economy, politics, science, and education. The dominant trend in the development of contemporary
civilization is its transition into an information society, wherein information and communication
technologies become the objects of human activity, providing all the necessary conditions for the
formation and development of the personality of the new formation. The rapid development of the
global Internet network has led to a computer revolution in the information world, where the computer
serves as the primary means of telecommunication. Considering that the current stage of development
of pedagogical science in the world is characterized by an intensive search for new ways to improve the
quality of education, information and communication technologies have become powerful tools in this
process [
Text-based generative artificial intelligence, utilizing a type of artificial neural network known as a general-purpose transformer, is particularly popular. This type of AI, often referred to as Large Language Models (LLMs), is commonly known as a generative pre-trained transformer, or GPT.
GPTs and their ability to automatically generate text became available to the global research
community in 2018. The launch of ChatGPT in 2022, which ofered free access and a user-friendly interface,
became a sensation and led to active searches and technological solutions for other companies to create
and launch new similar systems. By mid-2023, other alternatives to ChatGPT [
Many other tools based on the aforementioned LLMs are also emerging, such as ChatPDF, which can work with PDF documents and analyze them, or Perplexity, which serves as a knowledge hub and helps users find answers to queries based on their needs. Similarly, the process of integrating LLMs into other products, such as web browsers, is ongoing. In Ukraine, users gained access to artificial intelligence in 2023, and now the range of services based on generative artificial intelligence has significantly expanded.
The increasingly deeper and active penetration and integration of AI into human activity could not
fail to impact one of the most important components of human development, namely education.
Worldwide, the initial concerns about the use of AI in education were linked to fears that students
would use its capabilities primarily to cheat on academic tasks, thereby undermining the value of
educational assessment, certification, and qualifications [
As indicated by the analysis of recent research, significant regarding the use of AI in the field of
education, particularly in higher education, in educational and research activities, are occurring in the
following key areas [
Educational tools based on artificial intelligence can ofer personalized learning experiences, automate
routine tasks, and provide real-time feedback and assessment. As numerous studies by scientists around
the world have shown, AI has the potential to be a useful tool in teaching and learning Chemistry,
particularly for creating interactive simulations, answering questions, and providing feedback on
student work [
The results of studying the attitudes of natural science teachers toward the use of artificial intelligence
in teaching show that teachers are generally positive about integrating AI into the educational process.
Key factors influencing their willingness to use AI include self-confidence, expected benefits, ease of
use and general attitude towards AI technologies. Al Darayseh [
AI-based tools can be useful for teaching Chemistry by ofering interactive simulations, answering
questions, and providing feedback on student work. There are a number of studies regarding the use of
AI in chemistry education, but it should be noted that the overwhelming majority of them focus on
ChatGPT. For instance, dos Santos [
Another group of researchers explored the potential of using artificial intelligence to enhance
Chemistry teaching in high schools in Vietnam, using ChatGPT as an example. They identified the potential
of the ChatGPT text generation model from OpenAI. Their study showed that ChatGPT performed
well on intermediate-level Chemistry exams, but struggled with questions that required a high level of
knowledge application (e.g., analyzing and solving complex problems). Overall, ChatGPT’s responses
showed lower performance than most Vietnamese students in responses, indicating limitations in the
application of this tool. At the same time, a number of advantages of using ChatGPT were highlighted,
including increasing student engagement through interactive learning; providing immediate answers
and explanations to student questions; personalizing learning by adapting responses to individual
student needs; fostering critical thinking through open-ended questions and alternative points of
view; providing access to additional learning materials such as links and examples; and facilitating
self-directed learning [
In a similar study, Williams and Fadda [
Xuan-Quy et al. [
Humphry and Fuller [
Williams and Fadda [
Given the importance of hands-on experience in chemistry education, researchers have turned their
attention to integrating large language models (LLMs) into chemistry experimentation and laboratory
training, which opens up new possibilities for enhancing the eficiency and safety of the learning
process [
During the preparatory phase of laboratory work, LLMs can provide students with personalized
assistance by clarifying procedures described in lab manuals and explaining the specifics of equipment
setup. This allows students to better understand the objectives of the experiment and potential risks
without the need to process lengthy manuals. Furthermore, LLMs can reinforce safety measures,
ensuring students are fully aware of the importance of adhering to laboratory safety guidelines [
During experiments, the integration of LLMs with augmented reality (AR) technologies opens up
new possibilities for personalized guidance. These systems can collect real-time data on students’
actions and provide instant feedback, suggestions, and warnings. Such interaction not only enhances
the educational process but also significantly improves safety level in the laboratory by identifying and
correcting unsafe practices in real time [
During the analysis of experimental results, LLMs can serve as a powerful tool for bridging practical
observations with theoretical knowledge. They can assist students in analyzing collected data, facilitate
discussions about experimental outcomes, and link obtained data to relevant findings in the scientific
literature. This enables students to better comprehend their research results and their significance
within a broader scientific context [
Oh and Kang [
Given the diversity of prompting strategies, ranging from basic approaches like “input-output”
prompting [
In general, an analysis of existing scientific reports on the application of AI in education underscores the paramount importance and relevance of this topic. And the fact that the capabilities of AI have only begun to be utilized by both teachers and students attests to its pressing nature, indicating that it is currently at the forefront of attention. To properly establish the trajectory of the teacher-studentAI interaction, it is necessary to explore the functionality of AI that can be used in education for optimization purposes, as well as the limitations of AI, identifying its weaknesses and possible ways to correct them.
Analyzing the capabilities of AI will also help determine which specific needs of users (teachers and students) can be addressed using these technologies, as well as which features can best meet their needs.
Key challenges in using AI for chemical education can be identified. Firstly, this pertains to the
complexity of chemical science, which encompasses numerous nuances [
When considering the use of generative AI in the context of Ukrainian education, another important
aspect must be noted. It is crucial to remember that tools based on generative artificial intelligence
perform much better with the English language than with any other [
Firstly, the English language has a smaller vocabulary size compared to the Ukrainian language. This means that language processing algorithms can operate with fewer tokens (words), which simplifies their operation and improves speed and accuracy. Secondly, the English language has fewer word forms and word variations compared to the Ukrainian language. For example, many English words have only one form, while Ukrainian words can have multiple variations depending on context, gender, number, etc. This makes token analysis and recognition in Ukrainian more challenging. And thirdly, since English is used as a language of communication in many fields, including science, technology and business, there are far more sources of information available in English than in Ukrainian. Consequently, due to the smaller vocabulary and complexity of word-formation forms of the Ukrainian language, artificial intelligence algorithms can work more efectively with the English language in terms of token processing, and ChatGPT, like most other AI models, was trained on tokens tailored for the English language.
Our experience shows that Ukrainian students have recently started actively using AI in almost all types of their academic activities, believing that AI can provide the correct answer to any question. Often, they use AI as their primary source of information, placing complete trust in its outputs. As a result, students often fail to critically analyze the information and responses generated by AI, neglecting to verify them against reliable sources such as books, scientific journals, etc. Therefore, the aim of our research was to study the potential of AI to act as a tutor and assist students in the learning process, helping them find answers to questions of varying complexity, and to explore the potential of using some large language models in teaching Chemistry in the Ukrainian-language educational environment.
To begin with, we conducted research to identify the most popular AI-based services among pre-service teachers at Sumy State Pedagogical University named after A. S. Makarenko, whose education programs include the study of chemical disciplines. We also analyzed the services used by teachers in preparing for chemistry classes. For this purpose, we developed a questionnaire using Google Forms and employed interview methodology.
The questionnaire included the following questions: • Do you have experience using artificial intelligence? • Rate your level of familiarity with the types and principles of artificial intelligence and neural networks on a scale from 1 to 5 (1 – low, 5 – high). • What tools have you used? • If you have used artificial intelligence, what was the purpose? • Have you used AI-based services to solve tasks or find answers to questions in chemical disciplines? • Have you encountered cases where AI-based services provided incorrect answers or solved tasks incorrectly? • Have you experienced instances of AI hallucinations (when AI generated something that cannot be true)? Please specify the service where you noticed hallucinations.
• When did you notice AI hallucinations more frequently? Additionally, the method of interviewing students about their expectations from AI was used.
The next step was to assess the potential of LLMs and their applicability in addressing chemistryrelated questions and tasks. To achieve this, we identified specific chemical disciplines and types of questions that would help us analyze the limitations of LLM usage. Additionally, we conducted a comparative analysis of the capabilities of selected LLMs in solving diferent tasks related to various chemical disciplines studied by students at Sumy State Pedagogical University named after A. S. Makarenko, who are future chemistry teachers. We then compared the results of the LLMs to the average scores achieved by the students during testing. In total, the test results of 36 students were analyzed.
A review of studies on the topic has shown that large language models have limitations in
understanding Chemistry as a subject despite their seemingly well-reasoned generated responses, as they
lack the ability to think and reason logically or demonstrate understanding [
Disciplines such as Structure of Matter, Organic Chemistry, Environmental Chemistry and Laboratory Chemical Practice (LCP) are among the compulsory courses for students studying Chemistry. It should be noted that our many years of experience show that the first three of these disciplines are quite challenging for students to study, since they encompass all modern concepts not only in the field of Chemistry, but also Physics and Mathematics (Structure of Matter), Biology and Ecology (Environmental Chemistry). Organic Chemistry, in particular, is actually based on a huge amount of data on the structures of substances, their properties and relationships, operating with substance formulas. Laboratory Chemical Practice involves the formation of students’ knowledge about laboratory glassware, equipment, reagents, and basic operations with them.
The results of the first stage of our study showed that, during the educational process, our students use LLMs, and not specific neural networks intended for the study of chemistry, which is understandable given their pedagogical specialization. As the survey revealed, the most popular tools among both students and teachers are the freely accessible ChatGPT 3.5, Gemini (Google Bard), and Copilot (figure 1).
Therefore, we proceeded with further research using these specific large language models. These services were given identical questions related to the selected disciplines: Structure of Matter, Organic Chemistry, Environmental Chemistry and Laboratory Chemical Practice. Students provided answers to these same questions while studying these academic disciplines during the 2023-2024 academic year using the Moodle distance learning platform, which is used at our university. Examples of some test questions and the answers given by the LLMs are shown in table 2.
The test questions in Organic Chemistry were formulated to assess knowledge of nomenclature, the structure of various classes of organic compounds, methods of their synthesis and their chemical reactivity. We selected the topic “Alkenes and Alkadienes”. Let’s analyze the large language models responses to some of these questions. We chose several lower-order questions simply requiring demonstrating knowledge of certain facts or laws. Correct answers to such questions are well-known facts, presented in almost all textbooks on Organic Chemistry, even covered in high school Chemistry courses, and do not require additional logical reasoning. However, all LLMs provided incorrect answers to some of these questions. For example, in the question: “In the industry, buta-1,3-diene is obtained by the Lebedev method (dehydration). What substance is used for this as raw material?” all three AIs made a mistake. Moreover, both ChatGPT and Copilot provided their answers with explanatory comments. And for this question, the argumentation was based on the assertion that the Lebedev process involves the dehydration of acetylene. However, acetylene is an unsaturated hydrocarbon that does not contain a hydroxyl group and therefore cannot be dehydrated. Other questions required not just knowledge of a fact or law, but the ability to analyze and apply them in non-standard situations. Gemini failed to cope with the task because all its answers were incorrect and far from the truth. Particular attention is drawn to its response to the question about the reaction of propene with hydrochloric acid, with the formula HCI. Here, it was necessary to demonstrate knowledge of the reactivity of alkenes and take into account Markovnikov’s rule. The response provided by the LLM indicates that it considered this reaction as addition, which is correct. However, the product it indicated contains 2 chlorine atoms, which is nonsense, since the formula of hydrochloric acid contains only one chlorine atom. Therefore, it is evident that in this case, the AI hallucinated, as it replaced the reagent and performed the addition of a chlorine molecule instead of hydrochloric acid. ChatGPT provided an incorrect answer, while Copilot gave the correct answer with proper reasoning. Additionally, for some questions, ChatGPT ofered two answer options, one of which was correct, but the justification provided was irrelevant to the correct answer.
Gemini performed slightly better than ChatGPT and Copilot on test questions in the discipline Structure of Matter. All three large language models mostly provided correct answers to lower-order questions. However, for questions requiring data analysis, comparison, and logical inference, they either provided incomplete or incorrect responses. We aimed to investigate the factual basis utilized by the AI systems when answering more complex questions and why they formulated incorrect solutions. For example, in questions where it is necessary to select substances with a specific type of crystal lattice, and to provide the correct answer, it is necessary to analyze the structure of the substances ofered in answers and correlate the structure with lattice types. We received correct answers to some questions, but not to others. For instance, in the question “Choose substances with an atomic crystal lattice: a) H2O, CO2, b) NaNO3, Fe, c) KCl, NH3, d) SiO2, C,” all three AIs chose answer b) NaNO3, Fe, which is incorrect. The argument for the response to this question was as follows:
“Therefore, only NaNO3 and Fe have an atomic crystal lattice. Here are some additional explanations: An atomic crystal lattice is characterized by a high melting point, hardness, brittleness, and lack of electrical conductivity in the solid state.”
In response to our objection that “NaNO3 is an ionic compound, can it have an atomic lattice?” we received similar responses from LLMs:
“You are correct, NaNO3 is an ionic compound, not atomic. Ionic compounds have a crystal lattice consisting of cations and anions arranged in a specific order. This order is significantly diferent from the atomic crystal lattice, where atoms are bonded by covalent bonds. I made a mistake when I mentioned NaNO3 as an example of a substance with an atomic crystal lattice. The correct answer to the question: Choose substances with an atomic crystal lattice. a) H2O, CO2 b) NaNO3, Fe c) KCl, NH3 d) SiO2, C Answer: d) SiO2, C SiO2 has an atomic crystal lattice, where Si atoms are bonded to O atoms by covalent bonds. C also has an atomic crystal lattice. I apologize for the confusion.”
In order to check whether the LLMs learning process is occurring, we asked the same question again after a few days using the same account. This time, ChatGPT provided the correct answer. However, when we asked the same question from a diferent account, ChatGPT again gave the same incorrect answer. When the question was asked to Gemini from the same account as before, it once again provided an incorrect answer. To ensure whether the AI “understands” questions in Ukrainian, we asked the same question in English but still received the same incorrect answer. This indicates that the essence of the question was understood, but the AI’s analysis of the question was incorrect, resulting in an incorrect response.
The discipline Laboratory Chemical Practice plays a significant role in the training of chemistry specialists, providing them with the necessary theoretical knowledge, practical skills, and abilities for successful professional activity. We asked LLMs to provide answers to test questions for the Laboratory Chemical Practice exam.
ChatGPT showed a very low result, much lower than the average result of the students. Incorrect answers were related to understanding the properties of materials used in laboratory glassware, rules and principles of working with chemical reagents, principles and methods of substance purification operations, and the purpose and use of analytical balances, as well as the properties and structure of lfames. Correct answers were provided for questions concerning the purpose and use of laboratory glassware and equipment for specific operations, which represent commonly known information and do not require cognitive processing.
Gemini and Copilot also showed lower results compared to the average result of students on the same test questions, although they had more correct answers than ChatGPT. Gemini performed better on questions about purification operations of substances, properties, and the structure of flames. At the same time, it made mistakes on questions where ChatGPT provided correct answers. Copilot, on the other hand, performed somewhat better overall.
It is noteworthy that Gemini and Copilot, while providing explanations to the questions, often give correct explanations but choose the wrong option in the test. For example, in the question: “Analyze the statement about the use of beakers: 1) for measuring the volume of precipitates formed during the settling of suspensions; 2) for determining the volumes of two immiscible liquid phases,” both AIs indicate that beakers are used for measuring liquid volumes. However, they choose an answer that suggests beakers cannot measure the volume of two immiscible liquids or a liquid and a precipitate. A similar situation occurs with Gemini on the question: “Crucible tongs should be placed on the table so that their curved ends are directed a) downwards, b) upwards, c) they should not be placed on the table.” In the explanation, Gemini correctly notes that the curved ends of the crucible tongs should not touch the table surface, yet selects an entirely diferent answer that contradicts this explanation. This indicates that while the AI can find information on separate aspects of a problem and even explain them, it fails to construct elementary (from a human point of view) logical chains and draw correct conclusions. Based on the given example: if beakers can measure the volumes of liquids, and immiscible liquids are still liquids, then beakers can measure the volumes of immiscible liquids. And so on.
Such inconsistency also exists in the choice of answers by ChatGPT. When answering question “What is the name of the method for purifying substances, which involves obtaining a saturated solution of the substance at high temperature followed by cooling?”, the LLM selects the wrong answer, implying that extraction is a method of purifying substances involving obtaining a saturated solution at high temperature followed by cooling. After the chosen answer, we asked ChatGPT to justify its response. The AI justified its choice as follows:
“Extraction addresses the question of the method of purifying substances because extraction involves extracting a substance from a solution using another solvent.“
Perhaps the presence of the word “solution” in the definition of both operations “confused” the AI, preventing it from making the correct answer choice.
The opposite problem is also observed. Copilot chooses the correct answer in the test but explains it incorrectly. For the non-reproductive question requiring analysis of the given situation, “Analyze the statements: 1) burettes without stopcocks are NOT used when working with iodine solutions; 2) burettes without stopcocks are NOT used when working with alkaline solutions,” Copilot chose the correct answer: only the first statement is correct. However, it explained it as follows: iodine can evaporate and cause corrosion of the metal stopcock of the burette, while alkaline solutions do not have this problem, so burettes without stopcocks can be used with them. The following dialogue then took place: Me: “Can a burette have a metal stopcock? It would quickly corrode when interacting with various substances. A burette should be made of chemically inert materials in order to last a long time.” Copilot: “You are absolutely right, and I apologize for the misunderstanding. A metal stopcock on a burette is really not a good idea. It is important that the burette is made of chemically inert materials, such as glass or special plastics, to avoid corrosion and interaction with reagents.”
After this, the AI was asked to answer the same question again. This time, Copilot responded that neither solution could be used as a titrant in burettes without stopcocks, which is actually an incorrect statement. In its explanation, the AI stated that “alkaline solutions ... can cause corrosion of the metal parts of the burette,” thus not abandoning its initial misconception even after clarification of the question.
The discipline Environmental Chemistry encompasses the study of physicochemical processes in the atmosphere, hydrosphere, lithosphere, and biosphere, familiarizing students with the chemical composition and patterns of chemical element migration in soils, water, and the atmosphere, as well as forming an understanding of the geochemical role of living matter and the influence of chemical substances.
As our study showed, when generating answers to test questions for the course “Environmental Chemistry,” all three LLMs performed better compared to the average student results in this discipline. However, there were still errors in the generated answers, with the LLMs making mistakes on diferent questions. ChatGPT, which had the lowest percentage of correct answers, chose incorrect answers for questions regarding the prevalence of chemical elements in the Universe, the chemical composition of natural waters and the Earth, and the mechanism of the greenhouse efect. Gemini also gave incorrect answers about the prevalence of chemical elements in the Universe and the chemical composition of natural waters and the Earth. Unlike ChatGPT, Gemini correctly answered the question about the mechanism of the greenhouse efect but made a mistake on the question about the anomalous properties of water. Copilot provided the highest number of correct answers for both lower-level questions and more complex ones.
It is noteworthy that all three LLMs we analyzed gave incorrect answers to the test question “Among the chemical elements that make up the Earth, the most prevalent is...”. For example, ChatGPT responded, “Silicon is one of the most abundant elements in the Earth’s crust,” demonstrating a misunderstanding of the question’s essence. Gemini’s answer, “Oxygen is the most abundant element in the Earth’s crust,” showed that the AI failed to distinguish between the terms “Earth” and “Earth’s crust” due to the peculiarities of translation from English. This is probably why it was unable to select the correct answer “Iron”. Copilot performed even worse, providing the incorrect answer “Oxygen”, with the erroneous explanation that “Oxygen is the most abundant element forming our planet”.
We were also interested in the answers from the three LLMs to the question that required completing the phrase, “In river waters, the most abundant ions are...”. Only Copilot provided the correct answer and explanation: “...Calcium ions and carbonate ions. This is due to the dissolution of carbonate rocks, such as limestone.” ChatGPT failed to choose the correct answer, while Gemini hallucinated and selected an answer and explanation for a completely diferent question: “...Sodium ions and chloride ions – these are the two most common ions in seawater”.
Although generative artificial intelligence tools demonstrate significant success in processing and generating information, their accuracy in complex fields such as Chemistry remains limited. However, if, before posing questions, AI is first prompted to provide individual definitions and scientific facts related to the question content and then asked to provide the correct answer, the percentage of incorrect answers significantly decreases – to 10-15% of the total number of questions. These results indicate significant issues with the reliability of answers provided by modern AI services in response to chemical questions. This may be due to both the imperfections of the algorithms themselves and the complexity of the subject area.
Received results of answers to test questions on all four disciplines, obtained from ChatGPT 3.5, Gemini, Copilot and students of Sumy State Pedagogical University named after A. S. Makarenko, studying chemical disciplines, were summarized by calculating the percentage of correct answers. The results are presented in table 3.
As can be seen from the results obtained, Gemini showed the best average result in the disciplines of Structure of Matter and Laboratory Chemical Practice compared to ChatGPT and Copilot. However, it should be noted that Gemini immediately ofers three answer options, so-called Drafts, which difer from each other. In one of the options, the LLM hallucinated and “invented” answer choices that were not among the original options in the task, and moreover, they were incorrect. We attempted to determine how the LLMs arrived at incorrect answers to some questions. LLMs provided facts that they used for answers, but they couldn’t generalize them to reach a logical conclusion. After explanations, they agreed that they made mistakes. It is worth noting that Gemini attributed the reason for the error to incomplete mastery of the Ukrainian language and lack of access to some Ukrainian sources of information. However, Gemini showed the worst performance in Organic Chemistry. In Environmental Chemistry, Gemini and ChatGPT provided the same number of correct answers, although they made mistakes in diferent questions. Their performance in these tests was higher than that of the students. However, Copilot showed the best results. Providing answers to questions that comprehensively assess theoretical knowledge and practical skills in handling laboratory equipment, observing safety rules, and understanding the basic methodologies used in chemical practice (Laboratory Chemical Practice), Copilot achieved the best average result, while the other LLMs performed worse than the students. This fact overall correlates with our other results, confirming the inconsistency of both LLMs in solving more complex tasks.
As our interviews revealed, students often use AI expecting absolutely correct answers and solutions. More than half of the respondents indicated that they use LLM capabilities for studying, preparing homework, and finding materials for course works, presentations, and reports.
Our research showed that although generative AI-based tools demonstrate significant success in processing and generating information, their accuracy in complex fields such as chemistry remains limited and cannot consistently provide absolutely correct and accurate answers.
Overall, the results demonstrated by LLMs in tests across diferent chemical disciplines were predominantly lower than the results of the students. It is worth noting that the analysis of student responses showed that they often make mistakes in lower-level questions, which involve simply demonstrating the definition of a term or phenomenon. However, they perform better in solving questions that require cognitive activity and the ability to establish logical dependencies. A generalized analysis of LLMs responses to diferent types of questions showed that, on the contrary, it more often provides correct answers to simple questions that only require the reproduction of facts and information, but performs worse on tasks requiring productive thinking, data analysis, and the establishment of logical relationships between facts. Moreover, in the context of Chemical science, this is not always explained by linguistic factors. Nevertheless, there is a specificity of Chemical science related to the characteristics of chemical terminology and chemical laws and exceptions to these laws, regulating the relationships between terms and facts.
Considering the current insuficient “competence” (the ability to efectively utilize knowledge and demonstrate skills) of large language models such as ChatGPT, Gemini, or Copilot in handling chemical terminology, laws, and facts, they cannot yet be considered efective tools for tutoring. This raises the question of how AI capabilities can be utilized for chemical education.
The results of our survey of students at our faculty showed (figure 2) that 61.9% of respondents occasionally used AI services to solve problems and find answers to questions in chemical disciplines. The majority of respondents who used AI encountered incorrect answers or solutions provided by generative artificial intelligence. 42.9% of students indicated that this happened often, prompting them to check and correct the AI’s answers. Additionally, 47.6% of respondents encountered the phenomenon of “hallucination” in AI, where it invented nonexistent things. Moreover, 52.4% of those who noticed hallucination observed it specifically in responses to chemical questions and problems. In our opinion, the phenomenon of “hallucination” is particularly dangerous because uncritical acceptance of such answers or information presented can lead to the formation of false beliefs among students.
Our experience as users of AI fully aligns with the results we obtained. Unfortunately, it is necessary to approach almost all answers provided by LLMs to questions or tasks in the field of Chemistry critically and cautiously, verifying them. Additionally, the free version of ChatGPT 3.5 is currently unable to work with images, such as formulas, substance structures, or reaction equations. While Gemini can process graphical images, it often misinterprets them. However, there are some areas of pedagogical activity where these LLMs can be genuinely useful. This includes generating ideas of various kinds, from topics for reports or essays, creating plans for them, and even selecting theoretical material, to the topic of scientific research. For instance, we used LLMs to develop educational and methodological support for a practice-oriented course, “Equipment of the School Chemistry Classroom”. With the help of AI, a comprehensive set of instructional materials was created, including lecture and practical session plans, and instructional materials for classes, which were designed based on a template provided by the teacher. AI can also assist in generating assignments and problem-based questions. For the mentioned course, test questions for individual student assignments were generated. It is important to note that all AI-generated materials were thoroughly reviewed and revised by the teacher to address any inaccuracies. Particular attention was given to verifying and adapting the test questions to ensure their accuracy and alignment with the curriculum and pedagogical requirements.
LLMs can also assist in generating tasks and problem questions. It can even be used to find ideas for possible solutions to various problem questions. However, this information still needs to be critically evaluated, meaning that the integration of AI into education does not negate the necessity of learning independently.
The conducted research has shown that while ChatGPT and Gemini are popular due to their accessibility, they are prone to errors, especially in more complex, productive-level chemistry questions. Overall, large language models based on artificial intelligence, such as ChatGPT, Gemini, and Copilot, undoubtedly have a certain potential for application in education, including chemistry education. They can provide satisfactory answers to questions at the reproductive level related to various branches of Chemistry, utilizing a vast array of information they have been provided with. In some cases, they even demonstrate better results than student answers.
Despite significant progress in the development of AI technologies, these systems still face a number of challenges and limitations in the context of chemistry education. The complexity and multifaceted nature of Chemistry, the presence of abstract concepts, specific formulas, and equations, create dificulties for the accurate processing and interpretation of information by artificial intelligence algorithms. Additionally, imperfect proficiency in the Ukrainian language and limited access to relevant sources of information can lead to errors or incorrect answers. As our research results show, the use of LLMs for organizing chemistry education is only possible in combination with careful control. Teachers should verify and evaluate the work of LLMs to ensure its accuracy.
A promising direction for the application of LLMs, in our opinion, is the organization and stimulation of discussions and debates, as LLMs are capable of generating interesting and unusual ideas that can serve as the basis for collaborative work and research. The errors and “hallucinations” made by AI tools can also be utilized by the teacher, as they can be ofered to students for analysis. Through such generated errors, students can learn to critically evaluate information. When working with AI, students should understand and remember that LLMs are not infallible, and it is always necessary to verify information using other reliable sources. Another important aspect requiring the attention of researchers and educators is the improvement of LLM responses through prompt optimization.
Further research prospects in this field are as follows. Firstly, it is necessary to explore the opportunities ofered by AI for chemistry education and learning in general. This will help understand which specific functions can be useful in the educational context, which tasks or processes can be automated or facilitated. These aspects may include personalized learning support, automated assessment and reporting, personalized learning, and recommendations for educational materials and methods. It is also crucial to develop students’ understanding of prompt engineering strategies and to train them for more efective utilization of artificial intelligence.
Additionally, it is important to explore the impact of AI on students’ motivation and engagement in learning, which is particularly relevant for chemistry education. Understanding which AI tools can stimulate students’ interest and help them maintain motivation throughout the learning process is crucial for the successful integration of these technologies.
Furthermore, it is essential to explore the ethical aspects of using AI in education, including issues related to the privacy and security of student data, fairness of algorithms, and prevention of the emergence or exacerbation of inequalities in learning.
These aspects are crucial for the practical implementation of LLMs in the educational process and require in-depth study to ensure optimal performance and quality of educational outcomes. They necessitate detailed analysis, experimental research, and an interdisciplinary approach. Such a comprehensive approach to researching the efectiveness and potential applications of AI in education, including chemistry education, will enable the development of a balanced strategy for using these technologies to achieve the best outcomes for both educators and students.