This study analyzes the possibilities of using artificial intelligence tools to improve the quality of translations into English. It is shown that this process should necessarily involve English specialists who are competent in the subject area to which the text is related. A methodology is proposed for assessing the text quality and, if necessary, improving it for translations using the ChatGPT chatbot version of the GPT-3.5 model. We suggest creating a toolkit based on this methodology, which can be used to improve the quality of Englishlanguage publications. The toolkit will be useful for input control and editing, as well as improving the quality of expert formulations in decision support.
The availability of artificial intelligence (AI) tools [
This has become especially noticeable over the past year with the introduction of wide access to
the use of linguistic models, such as ChatGPT [
The use of these tools is particularly relevant given Ukraine's close integration into international
geopolitical structures and the importance of the English language use in Ukraine. Currently, work is
underway to develop a law on the use of English in Ukraine [
In addition, it should be noted that most existing text translation software tools do not provide translations of sufficient quality. Machine translation signs are typically detected during technical control of a future publication's text, prompting publishing editors to send the authors' works for revision before review. Sometimes it is believed that the presence of machine translation features in a text indicates a low level of elaboration of the work submitted for review by the international community, although this is not always the case. This usually only indicates flaws in the translation, not the work itself. The shortcomings of automatic translation tools make the proposed study all the more relevant. Based on the above, it is proposed to investigate and analyze the possibilities of using existing linguistic tools with AI elements and suggestions for using these tools to analyze the quality of translation of a text into English, preserve the semantic consistency of the translation, compliance with spelling and grammar standards of presentation of the material, improve the style of the text, as well as use this toolkit in learning English with the acquisition of translation skills.
The following studies on test translation are essentially related to the field of Natural Language Processing (NLP). These studies are based on rather global questions that are largely relevant to the area under consideration: Can AI tools replace expert knowledge? Is it possible to rely only on AI recommendations when making decisions in this area?
It seems indisputable that in the linguistic field, NLP, and other fields, decisions need to rely on all
available knowledge about the subject area. When translating a text, the required knowledge includes
the subject area, historical context, synonyms, phrases, and stylistics, as well as ensuring the text's
unambiguous meaning [
The survey results indicate that the majority of knowledge used (42%) is not formalized or registered on any sort of data carriers. This knowledge is possessed only by expert specialists, and AI tools are potentially unable to use this knowledge to provide recommendations. Therefore, AI can be a useful NLP tool, but it cannot entirely replace a subject matter expert. Although AI is known to be capable of analyzing large amounts of data and identifying semantic dependencies, it cannot always consider the unique context and features of each specific situation.
An expert, such as a linguist, translator, or editor, can consider the context and uniqueness of each case, resulting in a more accurate and reasonable translation. AI may be limited in its ability to adapt to innovations, neologisms in the language, whereas an expert is more adaptable. In addition, AI cannot replace human ethics and moral principles, which should be taken into account when presenting material. A linguistic expert can use his knowledge of the historical context, experience, and intuition, as well as AI as an auxiliary tool to gain knowledge and make correct conclusions. He can perform an in-depth analysis of what is described in the text, consider the context and peculiarities of a particular situation, and provide the most accurate and reasonable translation possible. Thus, combining the expert PPR and AI capabilities is the most effective approach to achieving the research goal of identifying opportunities and ways to use AI tools to improve the quality of English translation.
We propose the following course of research:
The input data for the study consists of English-language professional texts that are subject to quality control. The texts are created by specialists in a specific field who are not English translation professionals. The AI system provides recommendations on how to improve the quality of the submitted texts upon request. Certified linguists-translators of English, knowledgeable in the subject area, are involved as experts in the validation of recommendations provided by the AI system.
The study addresses the following questions: Are all AI recommendations acceptable? Can unacceptable ones be automatically identified? What percentage of recommendations do not improve translation quality (i.e. are unacceptable)?
An experimental study was conducted to determine the reliability of recommendations provided by
AI tools for improving the translation of professional texts into English. For the study, we selected
currently available AI tools, specifically ChatGPT [
Thus, the following properties of a publicly available AI system have been identified and studied: The volume of text submitted for analysis to the system at a time should not exceed 2000 words. Otherwise, in most cases, the system does not analyze the entire text and generates an unexpected and/or unacceptable result. The same applies to small amounts of text (1-2 sentences) when the content does not allow us to fully determine translation standards and writing style.
It is more efficient to formulate queries to the system in English, as we get more thorough results due to the peculiarities of AI model training.
It is advisable to go through the process of obtaining recommendations step by step, sentence by sentence, as this way we can get more detailed explanations of the recommendations provided.
Normally, the system generates 5 to 10 recommendations for improving translation for a given amount of text. Therefore, it is recommended to generate recommendations several times. The system has a peculiarity: if there are too many requests to continue generating results, it may occasionally include unacceptable recommendations in the resulting list. These may include repeating recommendations for certain sentences, or providing recommendations for sentences that do not exist in the text. In this regard, the study raises the issue of determining the optimal number of requests for generating recommendations to make the most of AI capabilities and avoid unacceptable recommendations. Obviously, this number depends on the quality of the source text, its volume, etc. 3.1.
The experimental study included 3 stages: 1. Each of the non-native English-speaking respondents provided English-language scholarly texts in the given subject area. Most of these texts were self-translated into English from Ukrainian or possibly other languages. The study did not analyze or control the process of creating or originating the texts. Therefore, it is possible that machine translation tools may have been used, at least, in part. It is important to note that all respondents in this group are competent in the field chosen for this pilot study and the English-language texts they provided also belong to the same field.
2. AI tools (based on the GPT-3.5 architecture) were used to generate recommendations for improving the quality of the resulting English translations. Particularly, we used certain prompts to the AI software system: "I can give you below a new fragment of scientific text. Can you show me a few sentences with poor quality of English (orthography and grammatical correctness) and give stepby-step tips to improve them?"
3. The second group of expert validators assessed the reliability and quality of the original English translated texts and the recommendations for translation improvement provided by the AI tool. This group consisted of experts who were sufficiently competent in the subject area under review. Each expert in the group is proficient in English. The expert group could consult with scholars and professionals in the subject area, if needed.
At the third stage, we obtained expert opinions on the recommendations provided by the AI system. Since each recommendation for translating a particular text fragment usually concerned only a single sentence, the recommendations had to be compared with sentences. That is, one of the assumptions made in the experimental study was that there was a mutually unambiguous correspondence between the sentences in the text under study and the recommendations provided by the AI system. With rare exceptions, the AI system's recommendations were to combine several sentences into one or to break a long sentence into simpler ones, but such rare cases were ignored in this experiment as insignificant. 3.2.
When assessing the reliability of recommendations for each individual text fragment, the group of expert validators used the following expert questionnaire (questionnaire) with multiple answers to one question: "How much better is the quality of the proposed AI recommendation (based on the GPT-3.5 architecture) than the original English-language text?"
The answer options included: 1. the AI recommendation is unacceptable; 2. the AI recommendation is acceptable, but equivalent in quality; 3. the AI recommendation is acceptable and requires minor adjustments; 4. the AI recommendation is acceptable and does not need to be adjusted.
Essentially, the experts were asked to evaluate [
Table 1 contains some examples of original text formulations from respondents, corresponding recommendations from the AI tool and estimations by expert validators. Green color of background marks minor adjustments of expert-validator in appropriate case.
The questionnaire information for each recommendation/sentence is summarized in a table, where each text fragment corresponds to a separate row. In addition to the data from the expert survey on AI recommendations for improving the translation of text fragments, the following information was collected for each fragment that may be useful in further research. The table, along with an example of how to fill in quantitative information about a particular text fragment, is presented below (see Table 2). Along with the data on the text fragment: number of sentences, number of words, number of characters with spaces, and number of characters without spaces, the table also contains information on the received recommendations for improving the text provided by AI. The total number of received recommendations for each text fragment includes all recommendations received during the entire session of the AI system, including requests to continue issuing recommendations.
Theoretically, there can be an infinite number of commands to the AI system to continue providing recommendations for the current text fragment. However, we propose to stop this process as soon as the sequence of recommendations provided consists only of inadequate ones. Such recommendations will be considered inapplicable to a given text fragment either because the fragment does not contain the sentence that is recommended to be changed or because a recommendation for a particular sentence has already been given during a session with this fragment. After a certain number of recommendations for improving the translation quality, this AI model begins to provide inadequate recommendations, and requests to the system are stopped. By utilizing a range of dynamic features, it is Dynamic features enable the creation of feasible to create various biometric vectors and various biometric vectors and decisiondecision-making algorithms, including statistical making algorithms, such as statistical methods and neural networks. methods and neural networks, for improved user authentication based on handwritten signatures.
It should be noted that the calculated estimates We should note that when users input obtained will be higher when in systems used their own signature into the system, the user`s own signature. calculated estimates obtained are
typically higher.
Processing of complex solutions with big Processing complex solutions with a number of connected sub-elements is large number of connected subcomplicated by the need to analyze a elements is complicated by the need to significant number of parameters of these analyze a significant number of objects contained into potential solutions, and parameters, as well as the necessity of the necessity of unification of their structure. unifying their structure.
In this work, we consider the last subtask In this work, we focus on the last where all compared CIOs are based on a single subtask, which involves comparing CIOs ontology and have a similar structure. that are based on a single ontology and
have a similar structure.
Currently, the process of making and terminating requests to the AI system is not automated, and this issue could be addressed in future research. When the process of obtaining recommendations is terminated by a human operator, two more quantitative characteristics become known: the number of inadequate recommendations that the AI system issued before stopping the current session and the number in order of the first in the sequence of inadequate recommendations. Table 2 lists these characteristics for further analysis. The remaining quantitative parameters, which are the main results of the experiment, are obtained based on the expert validators' assessment of only adequate recommendations. These data are obtained from the questionnaires filled out by the expert validators.
In order to ensure statistical credibility of the research, we calculated the necessary number of experiment instances. Evaluation of statistical credibility was conducted based on the central limit theorem. If we set the confidence probability value at P = 0.95 (i.e., the probability that the random variable value falls within confidence interval β), and confidence interval size for the given experimental study is β = 0.05, the minimum necessary number of experiment instances can be calculated based on the following inequality: n p (1 p) F 1P 2 ,
2 where F 1 is the inverse Laplace function; p is the frequency of repetition of value of the random characteristic under consideration. We select the value of p based on previously obtained experiment results as the “worst” probability/frequency (i.e. the one closest to 0.5). As a result of test experiment series, we gathered 358 assessments from a expert validators. The results of test experiment series are
Among the frequencies, defined based on the second column of the Table 1: {139/358≈0,388; 6/358≈0,017; 30/358≈0,084; 183/358≈0,511}, the worst one according to the specified criterion is frequency p = 0.511, which we will input into the formula for calculation.
After inputting all the respective values into the formula, we get: then: F 10.952 3.84 , n 0.511 (1 0.511) (0.05)2 and, finally, n ≥ 383.814. It means, that in order to draw credible conclusions based on the experiment results, it is sufficient to perform at least 384 repetitions of the experiment.
The final results of the experimental study are summarized in Table 4. total number of recommendations received unacceptable recommendations acceptable recommendations that are equivalent in quality to the original wording acceptable recommendations that require minor adjustments acceptable recommendations that do not require adjustment presented in Table 3. acceptable recommendations that do not require adjustment
The following parameters of the AI system used to improve the quality of translations into English are the most informative ones in terms of determining the reliability of the recommendations provided by the system: Percentage of successful recommendations approved by a group of expert validators. The sequence number of the first inadequate recommendation includes repetition or analysis of non-existent text. This number should be determined in relation to the length (number of words or number of sentences) of the text fragment for which the quality improvement recommendation is requested.
The total volume (number of texts, number of words, number of sentences) of the text corpus under study. 3.3.
In the pilot study, 58 English translations of text passages provided by respondents whose native language was not English were analyzed. It is important to note that all texts belong to a single, common subject area. The following percentage distribution was obtained among the total number of adequate recommendations based on the expert validators' assessment of the recommendations provided by the AI system to improve the quality of translations: 53.5% of the recommendations do not require adjustments; quantity 358 63% of recommendations that improve the quality of the source translation; 64.2% of recommendations that at least do not worsen the translation;
1.3% of recommendations are useless (the quality of the translation remains approximately the same);
35.8% of recommendations are harmful, as they will lead to a deterioration in the translation quality.
All of these relative values are derived from the distribution of the total number of adequate recommendations provided by the AI system. These values are determined with appropriate precision based on the statistical confidence that can be achieved with a given number of experiment replications. In this case, the experiment replications refer to the evaluation of each recommendation for improving the English translation provided by the AI system. 3.4.
The practical findings of the experimental study should serve as a methodology for using AI tools
to control and improve the quality of English translations. Such a methodology can be useful for
editorial boards of English-language publications, editors of electronic resources, and, to some extent,
can be used to improve the quality of expert formulations of objects in decision support systems.
Also, such tools may be in demand in connection with the implementation of legislative initiatives to
expand the use of the English language in Ukraine [
The texts in English that were received from the respondents in the pilot study can, in practice, be parts of up to 2000 words from articles by authors submitted to an English- language scientific publication or conference abstracts for publication. The texts listed above meet the criterion of belonging to a single subject area, because periodicals and collections of publications of reports of the scientific community at conferences (symposia, seminars) always have a clear thematic focus and belong to one or more related areas in a particular subject area. Since texts of publications typically exceed 2000 words, it is advisable to divide them into separate parts of an acceptable size for the study, up to the level of a complete sentence. It is convenient to divide them according to structural elements, if any, including the names of structural subdivisions. If the volume of unstructured text exceeds the acceptable threshold (2000 words), it is advisable to divide it into a minimum number of manageable parts of approximately equal size. The study analyzed and experimentally confirmed the inexpediency of splitting a single sentence, as well as the overlap of text parts when splitting a text.
The former confirms the importance of maintaining the semantic integrity of the text, which is a key factor taken into account by AI. As for the latter conclusion, the simultaneous inclusion of one or more consecutive sentences at the end of one part of the text and at the beginning of the next part of the text usually does not provide sensitively different recommendations for improving the text quality of these common sentences, so we consider it inappropriate to overlap texts. This can only complicate and confuse the process of obtaining recommendations for improving the quality of the translation.
It is worth noting, and this should be taken into account in future studies, that in addition to
content correspondence, the quality of translation is influenced by the parameter of text perception
unambiguity [
In the field of translation, as in many NLP-related areas, there is a place for expert opinion, and
this is common to the field of decision-making support. Therefore, let's consider the possibility of
applying the results of the study in decision support systems [
When improving the quality of English-language formulations using the proposed methodology, there are a number of differences from traditional text translations. It is worth noting that AI tools need a semantic context to evaluate a particular sentence. Since a phrase is usually just a short expression that is related to the wording of the object that was revealed during decomposition, it is necessary to find a way to transform a set of phrases that together represent the overall model of the subject area into a hierarchically structured text. The text here is represented as a set of meaningful sentences, each of which, by definition, is a set of words that usually express a complete thought.
Other approaches may be required to improve the quality of the formulations. It is worth considering providing the AI system with a list of all formulations, which will, to some extent, reflect the general context of the entire subject area. However, these issues certainly require further research in the future. Developing appropriate tools for decision support systems is still a pending task. 3.5.
It should be noted that since only the currently widely available AI tools were used in the study, namely ChatGPT with the current version of the GPT-3.5 model, the results and methods cannot be extended to other AI tools or model versions. Most of the experimentally obtained recommendations for using AI tools to improve the quality of text translations are specific to a particular implementation of these tools. In addition, the research only covers translations into English, and the methodological recommendations apply only to such translations. Although this is extremely relevant for Ukrainian realities, it significantly limits the scope of the research results. Recommendations for using AI tools for languages other than English may vary significantly.
It is also important that the subject area of the texts should be the same for all those analyzed during a particular session to receive recommendations for improving the quality of translations.
All these limitations are related to the existing features of the models, training corpora, and training methods used to create AI tools. Despite certain limitations, the research methodology proposed to improve the quality of English translations is expected to be useful and can be applied to developing NLP tools for various languages and using various AI tools and models.
As a result of the study, the originally developed methodology was used, which utilizes currently available AI tools to improve the quality of text translations into English. The experimental analysis showed that 53.5% of the AI tool recommendations for improving the translation did not require correction, 63% of them improved the translation, 64.2% did not worsen it, 1.3% of the recommendations were useless, and 35.8% were harmful.
It is shown that the methodology used in the experimental study can be applied in practice to improve the quality of text translations into English. Furthermore, it is concluded that in the case of practical implementation, this technology is suitable only for automated use with the involvement of a group of experts. It is not advisable to entirely rely on the recommendations of AI tools, specifically the ChatGPT chatbot of the GPT-3.5 language model. Nevertheless, AI systems are useful, powerful, and effective tools for enhancing the quality of English translations. In the future, there will be a need for development of appropriate software tools based on currently available and soon-to-be available advanced AI systems. These developments are necessary in the field of ESP and will help expand the use of English in Ukraine, which is a pressing issue.