AIs are everywhere in our daily lives and can now do anything. When it comes to scientific simplification, they can do it too, but are they as good as humans? In this paper, we will explain our work and results in the context of task 3 of the SimpleText track. We used the ChatGPT AI by asking it different requests about simple and complex scientific sentences. With new technologies and the development of increasingly powerful AIs, it's now possible to get robots to do almost anything. Numerous artificial intelligence has emerged in recent years, including ChatGPT. ChatGPT is an AI specialized in dialog and is accessible to everyone.
Thanks to AI, it is now possible to simplify scientific sentences very easily, but what about the quality of the simplification? How far can artificial intelligence go, and is it as good as humans?
ChatGPT is a very performant AI that can answer most of the questions we ask. After requesting a lot of simplification from the bot, we realized that the results were significantly different depending on the words we used in our question. Depending on the request, we got very different answers, even if the requests were quite similar. We tested the AI with simple and difficult scientific sentences.
We first tried to ask ChatGPT to “popularize” a simple sentence without difficult terms several times instead of “simplifying” it to see how he would simplify it. (ChatGPT's definition of popularize is “making something more widely known or accessible to the general public.”) First request: “Popularize this sentence: This technique was implemented inside a Personal Digital Assistant (PDA) portable device.”. (We asked every request three times to see if we would get different results.) 1st ChatGPT’s answer: Check out this catchy version: "We brought this incredible technique to life within a sleek and portable Personal Digital Assistant (PDA) device!” 2nd ChatGPT’s answer: Let's give this sentence a popular twist: “Witness the brilliance of this technique embedded within a portable powerhouse - the Personal Digital Assistant (PDA) device!” 3rd ChatGPT’s answer: Let's make this sentence more widely appealing: “Experience the power of this technique seamlessly integrated into a portable Personal Digital Assistant (PDA) device, making it accessible to all!”
The answer is simplified, and the tone has also changed. Words like “incredible” or “sleek” are not necessary for the simplification. We can also see the apparition of an exclamation mark at the end of every sentence. By asking ChatGPT to popularize the sentence, we obtain a simplified sentence with a less formal tone. Like said in the 3rd answer, ChatGPT popularized the sentence by making it "more widely appealing” and not only “more widely known or accessible”. The IA added terms or sentences that are not from the source text and that are not needed for the simplification.
To obtain a simplified sentence, we tried to formulate a different request with the same sentence.
Second request: “Simplify this sentence: This technique was implemented inside a Personal Digital Assistant (PDA) portable device.” 1st ChatGPT’s answer: This technique was used in a portable Personal Digital Assistant (PDA) device. 2nd ChatGPT’s answer: This technique was used in a portable PDA device. 3rd ChatGPT’s answer: This technique was used in a portable PDA device.
Compared to the first request, ChatGPT simplified the sentence by reformulating it. He replaced words (that are not difficult to understand) with even simpler words. (for example: “implemented” → “used in”). In the 2nd answer, it even removed the meaning of the abbreviation PDA.
We then tried to simplify with ChatGPT more sentences containing difficult scientific terms with two different requests.
First request: “Popularize this sentence: To solve this problem, we suggest a new interpolation method that utilizes an artificial neural network to find the best values for interpolated pixels.” 1st ChatGPT’s answer: Let's popularize this sentence: “We've got the ultimate solution to tackle this problem - introducing a groundbreaking interpolation method! Our secret weapon? An advanced artificial neural network that works its magic to discover the perfect values for those interpolated pixels.” 2nd ChatGPT’s answer: Check out this popularized version: "Say goodbye to this problem once and for all! We've got an innovative interpolation method up our sleeves. Brace yourself for the cutting-edge power of an artificial neural network that effortlessly uncovers the ideal values for those in-between pixels." 3rd ChatGPT’s answer: Introducing the solution you've been waiting for! Our revolutionary interpolation method, powered by an artificial neural network, is here to conquer this problem. Say farewell to pixel imperfections as our method expertly uncovers the optimal values, paving the way for flawless results.
In the first answer, all the difficult terms are not simplified or popularized like we asked, simplifying the sentence is only the addition of attention-seeking terms. We face the same problem in the 2nd except this time ChatGPT simplified “interpolated pixels” with “in-between pixels”
We then tried to simplify the complex sentence with another request to obtain the best sentence possible with all the difficult terms simplified.
Second request: Simplify this sentence with simple terms without losing its meaning: To solve this problem, we suggest a new interpolation method that utilizes an artificial neural network to find the best values for interpolated pixels. 1st ChatGPT’s answer: To fix this issue, we propose a new way of filling in missing pixels using a computer program that imitates the human brain and figures out the most accurate values for those pixels. 2nd ChatGPT’s answer: To solve this issue, we recommend using a new technique that uses a special computer program called an artificial neural network. This program helps us find the most suitable values for the missing pixels. 3rd ChatGPT’s answer: To solve this problem, we propose using a new method that uses a computer program called an artificial neural network. This program helps us find the most accurate values for the pixels that are missing or need to be estimated.
In the first answer, all the difficult terms are simplified with easier terms to understand. We asked the same request again to see which results we’ll obtain, but “artificial neural network” was not simplified anymore.
We can see here and with the previous requests the inconsistency of ChatGPT. Furthermore, we can also see that we need to ask very specific questions to obtain the perfect sentence, fully simplified.
Another problem we encountered with ChatGPT is the chat data history. We request the IA simplify multiple sentences about the same subject. We simplified sentences one by one without clearing the conversation in between.
Simplify this sentence: This paper presents M-CALL, a mobile computer-assisted language learning courseware for Korean language learners.
courseware that incorporates a cyber pet game to make it more engaging for learners than traditional computer-assisted language learning methods.
M-CALL is a language learning courseware for Korean language learners that can be accessed on a personal digital assistant with a public wireless LAN for mobile learning. the Korean language that uses a cyber pet game to make it more interesting. It runs on a personal digital assistant with a public wireless LAN, and learners can grow their cyber pet by solving problems related to Korean language learning.
In every answer we got, ChatGPT added information about the previous request. It also repeated the name of the application in every sentence, even though it was not in the source text. We encountered this problem a few times. If the chat is not often cleared, the IA remembers the answers and information and then proceeds to repeat that information in every answer about the subject. For some subjects, it can help understand the meaning of the text, but it often adds too much information in the sentence.
After trying a lot of different requests for the simplification of scientific sentences, the results we obtained are acceptable but not simplified enough. Most of the time, difficult words are not simplified, and some abbreviations are not explained. As we said, the AI is also very inconsistent in the simplifications. A term is being simplified in a first answer but not in a second one, for example.
In conclusion, from a personal point of view, it’s a very practical tool for scientific simplification, but from a professional point of view, it’s not enough. ChatGPT’s simplifications are not simplified enough, and his answers should be post-edited to obtain a better sentence, easier to understand for the public.
I would like to thank my professor and internship supervisor for her help and advice while I was using ChatGPT. This paper was created as part of the CLEF event of 2023.
This research was funded, in whole or in part, by the French National Research Agency (ANR) under the project ANR-22-CE23-0019-01.