A characteristic feature of modern times is the continuous dynamics of change in all spheres of human life: economy, politics, science, and education. The dominant trend in the development of contemporary civilization is its transition into an information society, wherein information and communication technologies become the objects of human activity, providing all the necessary conditions for the formation and development of the personality of the new formation. The rapid development of the global Internet network has led to a computer revolution in the information world, where the computer serves as the primary means of telecommunication. Considering that the current stage of development of pedagogical science in the world is characterized by an intensive search for new ways to improve the quality of education, information and communication technologies have become powerful tools in this process [1]. Various forms of information and communication technologies (ICTs) have found active application in education, ranging from electronic textbooks [2], online learning technologies [3,4,5], mobile applications [6,7] to augmented and virtual reality technologies [8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23]. The latest term in the digital revolution is artificial intelligence (AI) [24,25], including generative AI. Generative artificial intelligence (GenAI) is a technology that automatically generates content in response to queries. GenAI actually creates new content using existing content. Its output may encompass formats including all symbolic representations of human though: texts, images (including drawings, photographs, videos and animations), music and software code. GenAI learns from data it collects from web pages, conversations in social networks, and other online media. It generates its content by statistically analyzing the order of words, pixels, or other elements in the data it has learned, and identifying and replicating common patterns (such as which words typically follow other words, and in what order).
Text-based generative artificial intelligence, utilizing a type of artificial neural network known as a general-purpose transformer, is particularly popular. This type of AI, often referred to as Large Language Models (LLMs), is commonly known as a generative pre-trained transformer, or GPT.
GPTs and their ability to automatically generate text became available to the global research community in 2018. The launch of ChatGPT in 2022, which offered free access and a user-friendly interface, became a sensation and led to active searches and technological solutions for other companies to create and launch new similar systems. By mid-2023, other alternatives to ChatGPT [26] became available, most of which were free (within certain limits), as well as services for generating images, videos, and music, some of which are listed in table 1. Many other tools based on the aforementioned LLMs are also emerging, such as ChatPDF, which can work with PDF documents and analyze them, or Perplexity, which serves as a knowledge hub and helps users find answers to queries based on their needs. Similarly, the process of integrating LLMs into other products, such as web browsers, is ongoing. In Ukraine, users gained access to artificial intelligence in 2023, and now the range of services based on generative artificial intelligence has significantly expanded.
The increasingly deeper and active penetration and integration of AI into human activity could not fail to impact one of the most important components of human development, namely education. Worldwide, the initial concerns about the use of AI in education were linked to fears that students would use its capabilities primarily to cheat on academic tasks, thereby undermining the value of educational assessment, certification, and qualifications [27]. A clear consequence of such concerns was the prohibition by some educational institutions on students' use of AI [28]. However, in other institutions, a more optimistic approach was taken towards the use of AI [29], believing that it is more progressive not to prohibit its use, but to provide support for both teachers and students in utilizing tools based on generative AI [30]. Today, it should be noted that artificial intelligence is increasingly being integrated into education with the aim of enhancing student learning efficiency and improving teaching practices.
As indicated by the analysis of recent research, significant regarding the use of AI in the field of education, particularly in higher education, in educational and research activities, are occurring in the following key areas [31,32]:
• assessment (including automatic assessment and evaluation of educational progress and students' attitude to learning, individual and group assessment, etc.); • predicting learning status (predicting student withdrawals, at-risk groups, innovative abilities, career decisions), productivity or satisfaction, improving the learning experience; • assistance (providing support to students in their educational pursuits, for example, anthropomorphic presence, which includes virtual agents and intervention through digital programs); • tutoring (providing and supporting individual strategies and approaches to students, taking into account their characteristics and needs); • learning management (learning analytics, sequence of educational plans and programs, development of instructions, and student allocation).
Educational tools based on artificial intelligence can offer personalized learning experiences, automate routine tasks, and provide real-time feedback and assessment. As numerous studies by scientists around the world have shown, AI has the potential to be a useful tool in teaching and learning Chemistry, particularly for creating interactive simulations, answering questions, and providing feedback on student work [33]. It can be used to create personalized learning experiences for students [34,35].
The results of studying the attitudes of natural science teachers toward the use of artificial intelligence in teaching show that teachers are generally positive about integrating AI into the educational process. Key factors influencing their willingness to use AI include self-confidence, expected benefits, ease of use and general attitude towards AI technologies. Al Darayseh [36] found in his study that the easier teachers can integrate AI into teaching Natural Sciences, the more they believe in its benefits and are more willing to use it.
AI-based tools can be useful for teaching Chemistry by offering interactive simulations, answering questions, and providing feedback on student work. There are a number of studies regarding the use of AI in chemistry education, but it should be noted that the overwhelming majority of them focus on ChatGPT. For instance, dos Santos [37] explored the potential of generative chatbots based on artificial intelligence, including GPT-4 and BingChat, in chemical education. The study demonstrated that ChatGPT and BingChat act as "thinking agents" fostering critical thinking, problem solving, concept understanding, creativity, and personalized learning.
Another group of researchers explored the potential of using artificial intelligence to enhance Chemistry teaching in high schools in Vietnam, using ChatGPT as an example. They identified the potential of the ChatGPT text generation model from OpenAI. Their study showed that ChatGPT performed well on intermediate-level Chemistry exams, but struggled with questions that required a high level of knowledge application (e.g., analyzing and solving complex problems). Overall, ChatGPT's responses showed lower performance than most Vietnamese students in responses, indicating limitations in the application of this tool. At the same time, a number of advantages of using ChatGPT were highlighted, including increasing student engagement through interactive learning; providing immediate answers and explanations to student questions; personalizing learning by adapting responses to individual student needs; fostering critical thinking through open-ended questions and alternative points of view; providing access to additional learning materials such as links and examples; and facilitating self-directed learning [38].
In a similar study, Williams and Fadda [39] analyzed the responses of ChatGPT and ChatGPT Plus to questions related to carbohydrate chemistry, a topic frequently included in many chemistry curricula. The authors demonstrated that ChatGPT Plus performed significantly worse on test questions. Overall, both language models performed better on simple, common questions for which ample information is available.
Xuan-Quy et al. [40,41,42], Xuan-Quy and Le [43,44] obtained interesting results in a series of their studies, identifying the potential of ChatGPT for different science subjects at the high school graduate level. It was found that the AI performed better on social science questions, such as Literature, History, Geography, and Civic Education, and showed slightly worse results in answering questions on natural science: Mathematics, Physics, Chemistry, and Biology.
Humphry and Fuller [45] discuss the potential use of ChatGPT in chemistry laboratories at the bachelor's education level. The authors propose using ChatGPT as a teaching tool for students to demonstrate their understanding of specific topics by detecting and correcting chatbot's errors. We agree with the authors' observation that ChatGPT is prone to conceptual errors in many of its chemistryrelated responses and explanations and should not be used for general chemistry education. Overall, it is emphasized that ChatGPT can be useful for explaining basic concepts, but requires further development to effectively teach more complex topics.
Williams and Fadda [39] investigated the potential of integrating ChatGPT into a "flipped class" model for teaching chemistry. They demonstrated that this approach allowed students to take a more active role in their learning, while the teacher assumed a supervising or guiding role. Additionally, the authors tested ChatGPT's ability in writing annotations, abstracts, and essays on chemistry topicstasks that are common and significant in the scientific community. However, the results indicated that the bot often produced vague and repetitive text, frequently including inaccurate fabricated information and invented references.
Given the importance of hands-on experience in chemistry education, researchers have turned their attention to integrating large language models (LLMs) into chemistry experimentation and laboratory training, which opens up new possibilities for enhancing the efficiency and safety of the learning process [46]. LLMs can be applied throughout the entire laboratory workflow, from preparation to result analysis, enabling a more holistic and personalized learning experience [47].
During the preparatory phase of laboratory work, LLMs can provide students with personalized assistance by clarifying procedures described in lab manuals and explaining the specifics of equipment setup. This allows students to better understand the objectives of the experiment and potential risks without the need to process lengthy manuals. Furthermore, LLMs can reinforce safety measures, ensuring students are fully aware of the importance of adhering to laboratory safety guidelines [48,49].
During experiments, the integration of LLMs with augmented reality (AR) technologies opens up new possibilities for personalized guidance. These systems can collect real-time data on students' actions and provide instant feedback, suggestions, and warnings. Such interaction not only enhances the educational process but also significantly improves safety level in the laboratory by identifying and correcting unsafe practices in real time [50,51].
During the analysis of experimental results, LLMs can serve as a powerful tool for bridging practical observations with theoretical knowledge. They can assist students in analyzing collected data, facilitate discussions about experimental outcomes, and link obtained data to relevant findings in the scientific literature. This enables students to better comprehend their research results and their significance within a broader scientific context [45].
Oh and Kang [52] proposed integrating AI into a laboratory experiment involving a carbon dioxide fountain by using AI-based technology to regulate the laboratory setup. This approach not only enhanced students' understanding of the underlying chemical process and laboratory experiment but also provided them with insights into the potential of AI in scientific research. Healy and Blade [53] developed a project based on a state-of-the-art AI model for studying organic molecules of interest due to their pharmaceutical significance and for further research, as well as for planning their synthesis.
The authors also demonstrated the potential of this project for online education and enhancing student engagement in the learning process. In another study, Joss and Müller [54] assigned a task to students requiring them to create a correlation for predicting the boiling points of organic compounds. The processing of data for over 6,000 substances was conducted by the students using an artificial neural network. It should be noted that working with AI tools requires a distinct set of knowledge and skills compared to traditional search engines. One of the most crucial factors for effectively working with AI is crafting the correct prompt, i.e., formulating an appropriate input query [53,55]. Researchers are currently paying significant attention to this issue. For instance, Tassoti [56] studied how chemistry students work with AI and demonstrated that most students lack skills in effective prompting and tend to simply copy and paste questions, thereby limiting the effectiveness of their interactions with AI.
Given the diversity of prompting strategies, ranging from basic approaches like "input-output" prompting [57] to more advanced methods such as "Chain-of-Thought" [58] and "Tree-of-Thought" prompting [59], and considering that students outside of digital technology fields are generally unaware of these approaches, there is a need to train students in creating effective queries and prompts to enhance their interactions with AI. Although the issue of prompt formulation was not the focus of our study, as it is more relevant for open-ended queries, it remains a pressing issue in the field.
In general, an analysis of existing scientific reports on the application of AI in education underscores the paramount importance and relevance of this topic. And the fact that the capabilities of AI have only begun to be utilized by both teachers and students attests to its pressing nature, indicating that it is currently at the forefront of attention. To properly establish the trajectory of the teacher-student-AI interaction, it is necessary to explore the functionality of AI that can be used in education for optimization purposes, as well as the limitations of AI, identifying its weaknesses and possible ways to correct them.
Analyzing the capabilities of AI will also help determine which specific needs of users (teachers and students) can be addressed using these technologies, as well as which features can best meet their needs.
Key challenges in using AI for chemical education can be identified. Firstly, this pertains to the complexity of chemical science, which encompasses numerous nuances [60] and makes it difficult to capture in AI algorithms. Secondly, accurate interpretation of chemical terms and concepts is crucial, as they play a significant role in comprehending scientific material. Many chemical concepts are abstract and challenging to comprehend, with multiple meanings, leading to potential misinterpretation by artificial intelligence. Thirdly, chemical formulas, structures, and equations have a clear structure that AI may not always recognize accurately, affecting its ability to generate correct answers or solve problems. Therefore, to obtain correct answers to chemistry-related questions and issues, it is essential to consider the context of the question, understand the meanings of chemical terms and concepts, and apply logical thinking and analysis.
When considering the use of generative AI in the context of Ukrainian education, another important aspect must be noted. It is crucial to remember that tools based on generative artificial intelligence perform much better with the English language than with any other [61]. This is because AI perceives queries as tokens, which are fundamental building blocks upon which the model learns, understands, and processes language. This is where the most significant and crucial language differences arise.
Firstly, the English language has a smaller vocabulary size compared to the Ukrainian language. This means that language processing algorithms can operate with fewer tokens (words), which simplifies their operation and improves speed and accuracy. Secondly, the English language has fewer word forms and word variations compared to the Ukrainian language. For example, many English words have only one form, while Ukrainian words can have multiple variations depending on context, gender, number, etc. This makes token analysis and recognition in Ukrainian more challenging. And thirdly, since English is used as a language of communication in many fields, including science, technology and business, there are far more sources of information available in English than in Ukrainian. Consequently, due to the smaller vocabulary and complexity of word-formation forms of the Ukrainian language, artificial intelligence algorithms can work more effectively with the English language in terms of token processing, and ChatGPT, like most other AI models, was trained on tokens tailored for the English language.
Our experience shows that Ukrainian students have recently started actively using AI in almost all types of their academic activities, believing that AI can provide the correct answer to any question. Often, they use AI as their primary source of information, placing complete trust in its outputs. As a result, students often fail to critically analyze the information and responses generated by AI, neglecting to verify them against reliable sources such as books, scientific journals, etc. Therefore, the aim of our research was to study the potential of AI to act as a tutor and assist students in the learning process, helping them find answers to questions of varying complexity, and to explore the potential of using some large language models in teaching Chemistry in the Ukrainian-language educational environment.
To begin with, we conducted research to identify the most popular AI-based services among pre-service teachers at Sumy State Pedagogical University named after A. S. Makarenko, whose education programs include the study of chemical disciplines. We also analyzed the services used by teachers in preparing for chemistry classes. For this purpose, we developed a questionnaire using Google Forms and employed interview methodology.
The questionnaire included the following questions:
• Do you have experience using artificial intelligence?
• Rate your level of familiarity with the types and principles of artificial intelligence and neural networks on a scale from 1 to 5 (1 -low, 5 -high). Additionally, the method of interviewing students about their expectations from AI was used. The next step was to assess the potential of LLMs and their applicability in addressing chemistryrelated questions and tasks. To achieve this, we identified specific chemical disciplines and types of questions that would help us analyze the limitations of LLM usage. Additionally, we conducted a comparative analysis of the capabilities of selected LLMs in solving different tasks related to various chemical disciplines studied by students at Sumy State Pedagogical University named after A. S. Makarenko, who are future chemistry teachers. We then compared the results of the LLMs to the average scores achieved by the students during testing. In total, the test results of 36 students were analyzed.
A review of studies on the topic has shown that large language models have limitations in understanding Chemistry as a subject despite their seemingly well-reasoned generated responses, as they lack the ability to think and reason logically or demonstrate understanding [62]. Fergus et al. [63] showed that AI is quite good at answering questions related simply to knowledge demonstration, but has limitations in processing questions that require interpretation of non-textual information, such as analysis of structural formulas. Taking these facts into account, we selected multiple-choice questions of varying complexity. The lower-order questions required only the correct description of specific concept or phenomenon. The tests also included a series of higher-order questions that required the ability to analyze the provided information, consider the context, and think logically.
Disciplines such as Structure of Matter, Organic Chemistry, Environmental Chemistry and Laboratory Chemical Practice (LCP) are among the compulsory courses for students studying Chemistry. It should be noted that our many years of experience show that the first three of these disciplines are quite challenging for students to study, since they encompass all modern concepts not only in the field of Chemistry, but also Physics and Mathematics (Structure of Matter), Biology and Ecology (Environmental Chemistry). Organic Chemistry, in particular, is actually based on a huge amount of data on the structures of substances, their properties and relationships, operating with substance formulas. Laboratory Chemical Practice involves the formation of students' knowledge about laboratory glassware, equipment, reagents, and basic operations with them.
The results of the first stage of our study showed that, during the educational process, our students use LLMs, and not specific neural networks intended for the study of chemistry, which is understandable given their pedagogical specialization. As the survey revealed, the most popular tools among both students and teachers are the freely accessible ChatGPT 3.5, Gemini (Google Bard), and Copilot (figure 1). Therefore, we proceeded with further research using these specific large language models. These services were given identical questions related to the selected disciplines: Structure of Matter, Organic Chemistry, Environmental Chemistry and Laboratory Chemical Practice. Students provided answers to these same questions while studying these academic disciplines during the 2023-2024 academic year using the Moodle distance learning platform, which is used at our university. Examples of some test questions and the answers given by the LLMs are shown in table 2.
The test questions in Organic Chemistry were formulated to assess knowledge of nomenclature, the structure of various classes of organic compounds, methods of their synthesis and their chemical reactivity. We selected the topic "Alkenes and Alkadienes". Let's analyze the large language models responses to some of these questions. We chose several lower-order questions simply requiring demonstrating knowledge of certain facts or laws. Correct answers to such questions are well-known facts, presented in almost all textbooks on Organic Chemistry, even covered in high school Chemistry courses, and do not require additional logical reasoning. However, all LLMs provided incorrect answers to some of these questions. For example, in the question: "In the industry, buta-1,3-diene is obtained by the Lebedev method (dehydration). What substance is used for this as raw material?" all three AIs made a mistake. Moreover, both ChatGPT and Copilot provided their answers with explanatory comments. And for this question, the argumentation was based on the assertion that the Lebedev process involves the dehydration of acetylene. However, acetylene is an unsaturated hydrocarbon that does not contain a hydroxyl group and therefore cannot be dehydrated. Other questions required not just knowledge of a fact or law, but the ability to analyze and apply them in non-standard situations. Gemini failed to cope with the task because all its answers were incorrect and far from the truth. Particular attention is drawn to its response to the question about the reaction of propene with hydrochloric acid, with the formula HCI. Here, it was necessary to demonstrate knowledge of the reactivity of alkenes and take into account Markovnikov's rule. The response provided by the LLM indicates that it considered this reaction as addition, which is correct. However, the product it indicated contains 2 chlorine atoms, which is nonsense, since the formula of hydrochloric acid contains only one chlorine atom. Therefore, it is evident that in this case, the AI hallucinated, as it replaced the reagent and performed the addition of a chlorine molecule instead of hydrochloric acid. ChatGPT provided an incorrect answer, while Copilot gave the correct answer with proper reasoning. Additionally, for some questions, ChatGPT offered two answer options, one of which was correct, but the justification provided was irrelevant to the correct answer.
Gemini performed slightly better than ChatGPT and Copilot on test questions in the discipline Structure of Matter. All three large language models mostly provided correct answers to lower-order questions. However, for questions requiring data analysis, comparison, and logical inference, they either provided incomplete or incorrect responses. We aimed to investigate the factual basis utilized by the AI systems when answering more complex questions and why they formulated incorrect solutions. For example, in questions where it is necessary to select substances with a specific type of crystal lattice, and to provide the correct answer, it is necessary to analyze the structure of the substances offered in answers and correlate the structure with lattice types. We received correct answers to some questions, but not to others. For instance, in the question "Choose substances with an atomic crystal lattice: a) H 2 O, CO 2 , b) NaNO 3 , Fe, c) KCl, NH 3 , d) SiO 2 , C," all three AIs chose answer b) NaNO 3 , Fe, which is incorrect. The argument for the response to this question was as follows: "Therefore, only NaNO 3 and Fe have an atomic crystal lattice. Here are some additional explanations: An atomic crystal lattice is characterized by a high melting point, hardness, brittleness, and lack of electrical conductivity in the solid state. "
In response to our objection that "NaNO 3 is an ionic compound, can it have an atomic lattice?" we received similar responses from LLMs: "You are correct, NaNO 3 is an ionic compound, not atomic. Ionic compounds have a crystal lattice consisting of cations and anions arranged in a specific order. This order is significantly different from the atomic crystal lattice, where atoms are bonded by covalent bonds. I made a mistake when I mentioned NaNO In order to check whether the LLMs learning process is occurring, we asked the same question again after a few days using the same account. This time, ChatGPT provided the correct answer. However, when we asked the same question from a different account, ChatGPT again gave the same incorrect answer. When the question was asked to Gemini from the same account as before, it once again provided an incorrect answer. To ensure whether the AI "understands" questions in Ukrainian, we asked the same question in English but still received the same incorrect answer. This indicates that the essence of the question was understood, but the AI's analysis of the question was incorrect, resulting in an incorrect response.
The discipline Laboratory Chemical Practice plays a significant role in the training of chemistry specialists, providing them with the necessary theoretical knowledge, practical skills, and abilities for successful professional activity. We asked LLMs to provide answers to test questions for the Laboratory Chemical Practice exam.
ChatGPT showed a very low result, much lower than the average result of the students. Incorrect answers were related to understanding the properties of materials used in laboratory glassware, rules and principles of working with chemical reagents, principles and methods of substance purification operations, and the purpose and use of analytical balances, as well as the properties and structure of flames. Correct answers were provided for questions concerning the purpose and use of laboratory glassware and equipment for specific operations, which represent commonly known information and do not require cognitive processing.
Gemini and Copilot also showed lower results compared to the average result of students on the same test questions, although they had more correct answers than ChatGPT. Gemini performed better on questions about purification operations of substances, properties, and the structure of flames. At the same time, it made mistakes on questions where ChatGPT provided correct answers. Copilot, on the other hand, performed somewhat better overall.
It is noteworthy that Gemini and Copilot, while providing explanations to the questions, often give correct explanations but choose the wrong option in the test. For example, in the question: "Analyze the statement about the use of beakers: 1) for measuring the volume of precipitates formed during the settling of suspensions; 2) for determining the volumes of two immiscible liquid phases, " both AIs indicate that beakers are used for measuring liquid volumes. However, they choose an answer that suggests beakers cannot measure the volume of two immiscible liquids or a liquid and a precipitate. A similar situation occurs with Gemini on the question: "Crucible tongs should be placed on the table so that their curved ends are directed a) downwards, b) upwards, c) they should not be placed on the table. " In the explanation, Gemini correctly notes that the curved ends of the crucible tongs should not touch the table surface, yet selects an entirely different answer that contradicts this explanation. This indicates that while the AI can find information on separate aspects of a problem and even explain them, it fails to construct elementary (from a human point of view) logical chains and draw correct conclusions. Based on the given example: if beakers can measure the volumes of liquids, and immiscible liquids are still liquids, then beakers can measure the volumes of immiscible liquids. And so on.
Such inconsistency also exists in the choice of answers by ChatGPT. When answering question "What is the name of the method for purifying substances, which involves obtaining a saturated solution of the substance at high temperature followed by cooling?", the LLM selects the wrong answer, implying that extraction is a method of purifying substances involving obtaining a saturated solution at high temperature followed by cooling. After the chosen answer, we asked ChatGPT to justify its response. The AI justified its choice as follows:
"Extraction addresses the question of the method of purifying substances because extraction involves extracting a substance from a solution using another solvent. "
Perhaps the presence of the word "solution" in the definition of both operations "confused" the AI, preventing it from making the correct answer choice.
The opposite problem is also observed. Copilot chooses the correct answer in the test but explains it incorrectly. For the non-reproductive question requiring analysis of the given situation, "Analyze the statements: 1) burettes without stopcocks are NOT used when working with iodine solutions; 2) burettes without stopcocks are NOT used when working with alkaline solutions," Copilot chose the correct answer: only the first statement is correct. However, it explained it as follows: iodine can evaporate and cause corrosion of the metal stopcock of the burette, while alkaline solutions do not have this problem, so burettes without stopcocks can be used with them. The following dialogue then took place: Me: "Can a burette have a metal stopcock? It would quickly corrode when interacting with various substances. A burette should be made of chemically inert materials in order to last a long time. " Copilot: "You are absolutely right, and I apologize for the misunderstanding. A metal stopcock on a burette is really not a good idea. It is important that the burette is made of chemically inert materials, such as glass or special plastics, to avoid corrosion and interaction with reagents. "
After this, the AI was asked to answer the same question again. This time, Copilot responded that neither solution could be used as a titrant in burettes without stopcocks, which is actually an incorrect statement. In its explanation, the AI stated that "alkaline solutions ... can cause corrosion of the metal parts of the burette, " thus not abandoning its initial misconception even after clarification of the question.
The discipline Environmental Chemistry encompasses the study of physicochemical processes in the atmosphere, hydrosphere, lithosphere, and biosphere, familiarizing students with the chemical composition and patterns of chemical element migration in soils, water, and the atmosphere, as well as forming an understanding of the geochemical role of living matter and the influence of chemical substances.
As our study showed, when generating answers to test questions for the course "Environmental Chemistry, " all three LLMs performed better compared to the average student results in this discipline. However, there were still errors in the generated answers, with the LLMs making mistakes on different questions. ChatGPT, which had the lowest percentage of correct answers, chose incorrect answers for questions regarding the prevalence of chemical elements in the Universe, the chemical composition of natural waters and the Earth, and the mechanism of the greenhouse effect. Gemini also gave incorrect answers about the prevalence of chemical elements in the Universe and the chemical composition of natural waters and the Earth. Unlike ChatGPT, Gemini correctly answered the question about the mechanism of the greenhouse effect but made a mistake on the question about the anomalous properties of water. Copilot provided the highest number of correct answers for both lower-level questions and more complex ones.
It is noteworthy that all three LLMs we analyzed gave incorrect answers to the test question "Among the chemical elements that make up the Earth, the most prevalent is... ". For example, ChatGPT responded, "Silicon is one of the most abundant elements in the Earth's crust, " demonstrating a misunderstanding of the question's essence. Gemini's answer, "Oxygen is the most abundant element in the Earth's crust, " showed that the AI failed to distinguish between the terms "Earth" and "Earth's crust" due to the peculiarities of translation from English. This is probably why it was unable to select the correct answer "Iron". Copilot performed even worse, providing the incorrect answer "Oxygen", with the erroneous explanation that "Oxygen is the most abundant element forming our planet".
We were also interested in the answers from the three LLMs to the question that required completing the phrase, "In river waters, the most abundant ions are... ". Only Copilot provided the correct answer and explanation: "...Calcium ions and carbonate ions. This is due to the dissolution of carbonate rocks, such as limestone. " ChatGPT failed to choose the correct answer, while Gemini hallucinated and selected an answer and explanation for a completely different question: "...Sodium ions and chloride ions -these are the two most common ions in seawater".
Although generative artificial intelligence tools demonstrate significant success in processing and generating information, their accuracy in complex fields such as Chemistry remains limited. However, if, before posing questions, AI is first prompted to provide individual definitions and scientific facts related to the question content and then asked to provide the correct answer, the percentage of incorrect answers significantly decreases -to 10-15% of the total number of questions. These results indicate significant issues with the reliability of answers provided by modern AI services in response to chemical questions. This may be due to both the imperfections of the algorithms themselves and the complexity of the subject area.
Received results of answers to test questions on all four disciplines, obtained from ChatGPT 3.5, Gemini, Copilot and students of Sumy State Pedagogical University named after A. S. Makarenko, studying chemical disciplines, were summarized by calculating the percentage of correct answers. The results are presented in table 3. As can be seen from the results obtained, Gemini showed the best average result in the disciplines of Structure of Matter and Laboratory Chemical Practice compared to ChatGPT and Copilot. However, it should be noted that Gemini immediately offers three answer options, so-called Drafts, which differ from each other. In one of the options, the LLM hallucinated and "invented" answer choices that were not among the original options in the task, and moreover, they were incorrect. We attempted to determine how the LLMs arrived at incorrect answers to some questions. LLMs provided facts that they used for answers, but they couldn't generalize them to reach a logical conclusion. After explanations, they agreed that they made mistakes. It is worth noting that Gemini attributed the reason for the error to incomplete mastery of the Ukrainian language and lack of access to some Ukrainian sources of information. However, Gemini showed the worst performance in Organic Chemistry. In Environmental Chemistry, Gemini and ChatGPT provided the same number of correct answers, although they made mistakes in different questions. Their performance in these tests was higher than that of the students. However, Copilot showed the best results. Providing answers to questions that comprehensively assess theoretical knowledge and practical skills in handling laboratory equipment, observing safety rules, and understanding the basic methodologies used in chemical practice (Laboratory Chemical Practice), Copilot achieved the best average result, while the other LLMs performed worse than the students. This fact overall correlates with our other results, confirming the inconsistency of both LLMs in solving more complex tasks.
As our interviews revealed, students often use AI expecting absolutely correct answers and solutions. More than half of the respondents indicated that they use LLM capabilities for studying, preparing homework, and finding materials for course works, presentations, and reports.
Our research showed that although generative AI-based tools demonstrate significant success in processing and generating information, their accuracy in complex fields such as chemistry remains limited and cannot consistently provide absolutely correct and accurate answers.
Overall, the results demonstrated by LLMs in tests across different chemical disciplines were predominantly lower than the results of the students. It is worth noting that the analysis of student responses showed that they often make mistakes in lower-level questions, which involve simply demonstrating the definition of a term or phenomenon. However, they perform better in solving questions that require cognitive activity and the ability to establish logical dependencies. A generalized analysis of LLMs responses to different types of questions showed that, on the contrary, it more often provides correct answers to simple questions that only require the reproduction of facts and information, but performs worse on tasks requiring productive thinking, data analysis, and the establishment of logical relationships between facts. Moreover, in the context of Chemical science, this is not always explained by linguistic factors. Nevertheless, there is a specificity of Chemical science related to the characteristics of chemical terminology and chemical laws and exceptions to these laws, regulating the relationships between terms and facts.
Considering the current insufficient "competence" (the ability to effectively utilize knowledge and demonstrate skills) of large language models such as ChatGPT, Gemini, or Copilot in handling chemical terminology, laws, and facts, they cannot yet be considered effective tools for tutoring. This raises the question of how AI capabilities can be utilized for chemical education.
The results of our survey of students at our faculty showed (figure 2) that 61.9% of respondents occasionally used AI services to solve problems and find answers to questions in chemical disciplines. The majority of respondents who used AI encountered incorrect answers or solutions provided by generative artificial intelligence. 42.9% of students indicated that this happened often, prompting them to check and correct the AI's answers. Additionally, 47.6% of respondents encountered the phenomenon of "hallucination" in AI, where it invented nonexistent things. Moreover, 52.4% of those who noticed hallucination observed it specifically in responses to chemical questions and problems. In our opinion, the phenomenon of "hallucination" is particularly dangerous because uncritical acceptance of such answers or information presented can lead to the formation of false beliefs among students.
Our experience as users of AI fully aligns with the results we obtained. Unfortunately, it is necessary to approach almost all answers provided by LLMs to questions or tasks in the field of Chemistry critically and cautiously, verifying them. Additionally, the free version of ChatGPT 3.5 is currently unable to work with images, such as formulas, substance structures, or reaction equations. While Gemini can process graphical images, it often misinterprets them. However, there are some areas of pedagogical activity where these LLMs can be genuinely useful. This includes generating ideas of various kinds, from topics for reports or essays, creating plans for them, and even selecting theoretical material, to the topic of scientific research. For instance, we used LLMs to develop educational and methodological support for a practice-oriented course, "Equipment of the School Chemistry Classroom". With the help of AI, a comprehensive set of instructional materials was created, including lecture and practical session plans, and instructional materials for classes, which were designed based on a template provided by the teacher. AI can also assist in generating assignments and problem-based questions. For the mentioned course, test questions for individual student assignments were generated. It is important to note that all AI-generated materials were thoroughly reviewed and revised by the teacher to address any inaccuracies. Particular attention was given to verifying and adapting the test questions to ensure their accuracy and alignment with the curriculum and pedagogical requirements.
LLMs can also assist in generating tasks and problem questions. It can even be used to find ideas for possible solutions to various problem questions. However, this information still needs to be critically evaluated, meaning that the integration of AI into education does not negate the necessity of learning independently.
The conducted research has shown that while ChatGPT and Gemini are popular due to their accessibility, they are prone to errors, especially in more complex, productive-level chemistry questions. Overall, large language models based on artificial intelligence, such as ChatGPT, Gemini, and Copilot, undoubtedly have a certain potential for application in education, including chemistry education. They can provide satisfactory answers to questions at the reproductive level related to various branches of Chemistry, utilizing a vast array of information they have been provided with. In some cases, they even demonstrate better results than student answers.
Despite significant progress in the development of AI technologies, these systems still face a number of challenges and limitations in the context of chemistry education. The complexity and multifaceted nature of Chemistry, the presence of abstract concepts, specific formulas, and equations, create difficulties for the accurate processing and interpretation of information by artificial intelligence algorithms. Additionally, imperfect proficiency in the Ukrainian language and limited access to relevant sources of information can lead to errors or incorrect answers. As our research results show, the use of LLMs for organizing chemistry education is only possible in combination with careful control. Teachers should verify and evaluate the work of LLMs to ensure its accuracy.
A promising direction for the application of LLMs, in our opinion, is the organization and stimulation of discussions and debates, as LLMs are capable of generating interesting and unusual ideas that can serve as the basis for collaborative work and research. The errors and "hallucinations" made by AI tools can also be utilized by the teacher, as they can be offered to students for analysis. Through such generated errors, students can learn to critically evaluate information. When working with AI, students should understand and remember that LLMs are not infallible, and it is always necessary to verify information using other reliable sources. Another important aspect requiring the attention of researchers and educators is the improvement of LLM responses through prompt optimization.
Further research prospects in this field are as follows. Firstly, it is necessary to explore the opportunities offered by AI for chemistry education and learning in general. This will help understand which specific functions can be useful in the educational context, which tasks or processes can be automated or facilitated. These aspects may include personalized learning support, automated assessment and reporting, personalized learning, and recommendations for educational materials and methods. It is also crucial to develop students' understanding of prompt engineering strategies and to train them for more effective utilization of artificial intelligence.
Additionally, it is important to explore the impact of AI on students' motivation and engagement in learning, which is particularly relevant for chemistry education. Understanding which AI tools can stimulate students' interest and help them maintain motivation throughout the learning process is crucial for the successful integration of these technologies.
Furthermore, it is essential to explore the ethical aspects of using AI in education, including issues related to the privacy and security of student data, fairness of algorithms, and prevention of the emergence or exacerbation of inequalities in learning.
These aspects are crucial for the practical implementation of LLMs in the educational process and require in-depth study to ensure optimal performance and quality of educational outcomes. They necessitate detailed analysis, experimental research, and an interdisciplinary approach. Such a comprehensive approach to researching the effectiveness and potential applications of AI in education, including chemistry education, will enable the development of a balanced strategy for using these technologies to achieve the best outcomes for both educators and students.