The paper presents a classification of errors based on the comparative analysis of the errors in machine translation (MT) output and a human computer-assisted translation (CAT) on a cloud platform. The impact of the human factor is analyzed in the raw dataset of errors of a Hebrew-Russian CAT on a cloud platform. The CAT platform belongs to a set of tools for computer-mediated communication (CMC). CMC peculiarities affect the CAT output. The acceptability and usability of a target text (TT) do not directly depend on the number and gravity of errors in CMC. The classification of translation errors integrates approaches to error recognition that have been developed in the industry and academia. We distinguish the errors that automatic evaluation systems are able to recognize from those that human translators and post-editors have to handle. Based on the errors analysis, we offer three categories of errors in CAT: (1) fluency errors, which damage the TT readability; (2) accuracy errors, which misrepresent the content; and (3) functional errors, which distort the objective of the source text (ST) translation and obstruct the perception of the TT in the target culture. CAT tools affect the distribution of errors in the TT.
The translation industry transforms the translators’ environment into computer-mediated communication (CMC) with the help of cloud platforms for computer-assisted translation (CAT). CAT platforms provide customer-relationship management, as the platforms offer tools to search for a project, perform translation and localization, discuss an output with a customer, and transfer payments. In addition to machine translation (MT) systems and shared translation memory (TM), CAT platforms combine all translation tools in one place. Since the industry demands accurate and high-quality translations in practical domains that are delivered quickly, a CAT platform offers a solution to enhance operational activity. Nevertheless, MT and CAT outputs have to be post-edited. The insertion of segments from MT output into manually translated documents increases probability of errors because of differences in translation strategies. In the new technological environment, translators have become post-editors and revisers of MT output. Post-editing and revising all products before their delivery belong to translator’s professional competencies, along with ability to negotiate with customers [1].
CMC restricts a set of communicative tools (e.g., gestures and facial expression) in virtual team professional communication; the virtual environment requires adjustment of social norms and standards [2]. While working on a CAT platform, translators are exposed to the difficulties of collaborating with teammates, who often represent different cultures, on the one hand; on the other hand, translators lack feedback and an explanation of the rationale behind decisions [3]. Thus, the CAT platform facilitates translating but can provoke new errors due to multitasking without relevant feedback. CAT platforms can cause new errors due to their design and joint project management. The errors caused by design
2020 Copyright for this paper by its authors. and clumsy use of computer technologies were studied in 1994 [4]. Nowadays, technologies, which facilitate computer-human interaction, generate new grounds for a human translator to err in CMC.
To revise the CAT or MT output, the translator needs to verify information in the source text (ST). The translator mines the web to obtain appropriate data; the data mining presupposes the use of English since the English language prevails on the web: 54% of the top 10 million websites offer their content in English. Thus, English has become the lingua franca in CMC. Therefore, the new technologies compels translators to acquire at least three languages to become competitive in the industry and arrange business and corporate communication in multinational companies. Meanwhile, trilingualism provokes cross-linguistic influences due to parasitic connections between L2 and L3 mental representations [5]. Because of their use of CMC, translators are sensitive to inter-language interference and the effect of the lingua franca, which can cause particular errors even in the translator’s native tongue.
When revising the target text (TT), the translator has to understand the message in the ST, which needs transfer to the target culture. The transference of the message not only directly depends on the accuracy, fluency, and readability of the TT but also includes the cultural associations, customs and genre models of the target culture. Human translators are responsible for localization of final product since CAT and automatic error evaluation systems do not process intercultural differences. The new technological environment in CMC professional communication makes it necessary to describe errors on CAT platforms. 1.2.
CMC and new technologies put translators under pressure since they obtain new responsibilities while working on CAT platforms. Meanwhile, professional translators, who work in global corporation and receive their colleagues’ professional support, are able to use CAT, apply automatic evaluation systems and post-editing tools. Nevertheless, they mostly prefer to translate from the scratch and revise their manual translation than to use the tools [6]. The translators avoid technologies since they do not learn the current state-of-the art and advantages of TM, on the one hand, and, on the other hand, are aware of the inconsistencies in the TT caused by sharing TM [7].
Error recognition in MT output is sometimes performed by automatic errors evaluation systems. Automatic evaluation of MT output maps the content of the ST segment to the corresponding segment in the TT. Systems for the automatic error evaluation are still lack reliability in evaluation semantic coherence, anaphora resolution, stylistic and functional integrity at the document level. The set of the errors recognizable by the automatic evaluation systems are of importance for the translators and posteditors since the errors unrecognizable by the systems required manual detection and consideration. The translator/post-editor is a key specialist in transferring the communicative value of the ST to the target culture.
To prevent errors evoked by the CAT platform, we need to describe and analyze them based on the well-known error classifications. Error analysis allows us to adjust the classification of errors in translation to the new technological environment and to new standards in the industry.
The paper aims are to: • Distinguish the errors that automatic evaluation systems are able to recognize from those that human translators themselves have to handle • Show how the distribution of the errors are affected by the use of translation tools on the CAT platform • Clarify the impacts of the human factor and the technological environment (CMC and CAT tools) on the errors distribution.
In our research, we study the distribution of the errors in Hebrew-Russian CAT on a cloud platform. After describing approaches to the errors detection and classification in the translation industry and academia, we show the distribution of errors in a Hebrew-Russian translation project on the CAT platform. Since our research is based on the material of translation into Russian, we take into account errors classification for Russian as the TL. We compare distributions of the error in CAT and neural MT (NMT) output to distinguish peculiar errors in CAT, and to clarify factors that can cause a translator to err. In the discussion, we clarify the impact of the human factor and technological environment.
To revise CAT or MT output, the translator has to annotate errors and offer a correct version. An error annotation indicates its position in the classification. In the industry and academia, researchers use different grounds to annotate and classify the errors. Errors in translation belong to humans and MT systems; thus, classifications of errors in translation reflect the material researchers deal with. 2.1.
In academia, the differences in the theoretical approaches to training courses for professional translators lead to discrepancy in errors classifications. The classifications reflect three procedures in the process of translation and localization: analyzing ST, selecting strategies for transforming ST into TT, and revising TT [8]. The ST analysis reveals the ST message and stylistic devices. Failures in this procedure bring an inadequate TT and distortion of the source message. ‘Vertical’ approach to translation presupposes the ST comprehension before reformulating its message by target language (TL) means and choosing a degree of TT localization. The ‘vertical’ approach is associated with topdown and non-linear translation strategies. The ‘horizontal’ approach assumes that the source language (SL) units evoke the TL corresponding structures in the translator’s memory, and the process of message transcoding results in ST comprehension. The ‘horizontal’ approach corresponds to bottom-up and linear strategies of translation. Meanwhile, the ‘horizontal’ translation strategy corresponds to the basic MT algorithm, which is not able to process the ST on the document level before processing it from the first segment to the last one. Thus, human translators have advantage to use both approaches in translation.
The selection of the ST transformation strategies depends on the degree of the necessary localization of the ST. The selection of an irrelevant strategy damages the TT perception in the target culture, which leads to misunderstanding the source message [9]. The damage is revealed in the fluency, accuracy and adequacy of the TT. Subclasses of fluency errors are indicated according to cross-linguistic contrast of the source and target languages. The errors in ST analysis and erroneous translation strategy affect the TT accuracy and adequacy.
Error classifications are of importance in the translation quality assessment (TQA). The TQA presupposes the distinguishing of errors in accuracy, style, grammar, and formatting [10]. The classes are included in the typology of translation errors, but their subclasses vary for different language pairs. While evaluating mistakes in human translations compared to MT output, researchers distinguish errors that lead to unacceptable output and errors in adequacy [11]. Adequacy errors are identified through juxtaposing the ST and TT. Adequacy errors show the discrepancy between the source message and its transfer to the target culture, while acceptability errors show the irrelevance of the TT discourse features in the target culture. In [8], the adequacy errors show inconsistency in text-external, while the errors of style and content diminish adequacy in text-internal. Thus, the errors in the text-internal adequacy overlap the accuracy errors. The text-external adequacy errors correspond to the acceptability errors.
The classification in [12] was worked out for human translators. The authors offer four different categories of errors: (1) errors in content and semantics, (2) discourse errors, (3) errors in evaluation (strategy of translation), and (4) errors in usage. The first category corresponds to the accuracy errors including text-external fact errors and the text-internal adequacy errors. The category of the text-internal adequacy errors includes the distortion of genre features. The errors in evaluation concern the TT acceptability in the target culture. The errors in usage cover grammar and incorrect lexical choice, which belong to fluency and accuracy errors, respectively. However, all four categories include errors in accuracy manifested in semantic distortion. The accuracy errors include the omission and addition of information, semantic shifts due to errors in terminology, incorrect choices in naming entities, etc. The logical and fact errors of text-external are considered as errors in content transferring. Grammatical errors affect the style (e.g., a syntactic construction from colloquial speech is inappropriate in the official style). Nevertheless, this classification distinguishes discourse (or functional) errors, which require TT revision on the document level.
Detailed classifications of errors observed in translations into Russian were generated for corpuses of the Russian language for MT and human translation: Russian learner translator corpus [13] and Corpus for Russian data-to-text generation [14]. The classifications contain categories of linguistic, discourse and semantic (incorrect logical connections and presuppositions) errors. Thus, the classifications distinguish errors in fluency, accuracy and adequacy of the TT. The category of linguistic errors includes classes of errors in grammar, lexical choice, style, orthography and punctuation. Deletions and insertions (omissions and additions) belong to the discourse category [14]. The classes and subclasses show the results of top-down TT revising.
The chart in the Figure 1 represents a combination of the academic views on the classification of the errors in translation disregarding discrepancies in the researchers’ positions.
Error recognition needs to combine top-down and bottom-up strategies for TT monitoring; error evaluation must involve the discourse perspective to recognize semantic and logical connections among segments on the document level and the value of each segment in the TT [15]. Linguistic errors and errors in semantics might be recognized through comparing each segment of the TT to its origin. However, semantic shifts, omissions and additions of information become obvious through applying the top-down and non-linear revision strategies. Semantic shifts, omissions and additions as well as style shifting might represent discourse (or functional) errors since the TT does not fit the discourse features in the target culture due to these errors.
Thus, in academia, the researchers detect the categories of errors in the text-internal and textexternal. The text-internal errors diminish TT fluency and accuracy; the fluency errors vary according to the linguistic typology of the TL. The text-external errors distort TT acceptability and functioning in the target culture. 2.2.
In the industry, researchers consider the possibility of producing an automatic error evaluation, which could be applied to the MT workflow as a step to select a version of a sentence translation within a set of translation hypotheses. The error classification has to be clear and precise to avoid ambiguities. In the Multidimensional Quality Metrics guidelines, the error classification includes category for accuracy and fluency errors [16]. The category of the accuracy errors includes mistranslations, terminological errors, omissions, additions and untranslated segments [17]. The distinction between mistranslations and errors in terminology reflects importance of the terminology in the industry since the translation industry deals with spheres of communication where MT output is acceptable and widespread. In the MT post-editing guide [18], the authors recommend to examine inconsistencies in terminology and to offer its disambiguation. Untranslated segments occur in MT output since an algorithm skips the last positions in long dependency chains [19]. The classification of the fluency errors from the guidelines needs to be adjusted to the linguistic categories of the TL. Due to the adjustment, the morphologically rich languages receive additional subclasses in grammatical classes for different inflection errors (for Slavic languages: gender, person, number, case) [16]. The adjustment allows for the identification of the grammatical errors that are commonly shared by Slavic languages. However, the additional error subclasses do not cover the contrast between grammatical systems of the languages [20]. Nevertheless, the subclasses improve the ability of automatic algorithms to provide fluent MT output. Klubička et al. showed how the adjustment of the error classification to the language typology affects productivity of the automatic evaluation systems [16].
In the mapping of these categories of errors to the academic classifications, we consider fluency errors as linguistic ones, while accuracy errors are mostly associated with errors in semantics. The fluency errors category includes grammatical, stylistic, orthographic and punctuation errors. The grammatical errors class is the most typical among the classes of the category; more than 80% errors in MT output belong to the grammatical errors [17]. Morphological errors are regular in the MT into languages with rich morphology [17]. In human translations, grammatical errors also dominate in the category of fluency errors [21]. Automatic errors evaluation systems are unable to take into account discourse features. Discourse relations and discourse connectives establish a set of correlations among entity names and pronouns; they are helpful in polysemous words disambiguation, etc. [22]. Nevertheless, the discourse features lack formalization at the state-of-the art level for automatic evaluation systems. Therefore, the classifications in the industry do not include any particular category for discourse or functional errors. These classifications do not consider fact errors and logical errors since MT systems process a natural language but do not verify a correlation between the text content and the global image of the world. The text-external errors do not represent a particular category in these classifications.
Thus, the differences in the academic and industrial approaches to error evaluation and classification are revealed in avoiding consideration of the text-external and discourse errors by industrial professionals. In the industry, classifications seem to be more transparent and multipurpose because of the necessity to match the standards of MT systems and automatic evaluation systems technologies. 2.3.
Specific tools are developed to recognize and evaluate language deviations in the final product of the translation. The metrics allow for the movement from subjective estimations to objective issues; however, only humans are able to determine whether an issue is an error.
Automatic systems are mostly trained to recognize, evaluate and correct fluency errors in English that is of importance for the global communication and the translation industry. Bryant et al. show the progress in automatic grammatical errors correction systems trained on the new extended English corpus [23]. However, the efficacy of automatic recognition and correction in translation into morphologically rich target languages is usually lower [24]. The classes for fluency errors are relevant for all languages, while the subclasses might be different according to TL typology.
A promising approach to the automatic detection of fluency and accuracy errors is based on identifying linguistic check-points and generating a check-point database from the parallel bilingual corpus [20]. This approach, to some extent, corresponds to the academic theory of ‘critical points’ in the translator’s decision-making [25]. The ‘critical points’ theory takes into consideration potential cross-linguistic interference and the translator’s intervention, as well as contrastive descriptions of the SL and the TL. The check-points database includes a list of the linguistic units in the SL and their translations. To create the database, the developer needs to define the desired linguistic categories that describe the contrast between the SL and the TL in the most accurate way. The linguistic categories characterize the differences covering the lexicon, phraseology, morphology and syntax of the TL. The set of distinguished features indicates the potential errors in MT output. However, the linguistic checkpoint approach is still far from being able to provide relevant results in the automatic detection and evaluation of errors in MT output. The contrastive description of linguistic categories provides the grounds for the accurate classification of errors.
The distribution of the error categories and classes in the aspect of their recognition by automatic systems is shown in Figure 2.
Unrecognizable by the automatic systems errors correspond to errors in adequacy and acceptability within text-external errors in translation. A human translator on a CAT platform and a post-editor of an MT output are responsible for handling the errors.
The objective of the paper is to describe the impact of the human factor on the errors in CAT on a cloud platform and to classify the errors. The classification of errors on the CAT platform is explained through comparison with the errors observed in MT output taking into account the possibility to evaluate the errors by the automatic evaluation systems.
The research is conducted on the raw dataset of Hebrew-Russian translation projects on the CAT platform (approximately 43,000 word forms). The dataset of Hebrew-Russian translation projects represent drafts edited by translators, which require final evaluation and proofreading. The instances in the dataset contain references to a segment of their ST and an indication of the applied translation tool (for instances see Figure 3). The dataset includes a draft of a joint project1 that allows us to examine errors on the document level including wrong translation strategy and mistakes in transferring the author’s opinion. The ST in Hebrew includes approximately 35,000 words in 3118 segments on the CAT platform. In the Russian TT, segments with errors include 1066 word forms. Thus, this translation of a tourist guide from Hebrew into Russian was analysed as a case study in the aspect of the TT readability and adequacy. 1 https://www.visit.ashdod.muni.il/wp-content/uploads/travelers-guide/ru/
Smartcat has a friendly user interface providing access to various tools and communication with teammates (see Figure 4).
We apply the error analysis method to the material to obtain the distribution of errors and errors types in the CAT platform output where an MT system and TM are in use. A post-editor and an expert who are native Russian speakers manually annotated the errors in the draft. The errors were annotated according to the classification shown in Figure 2; syntax errors are included in the grammatical errors since they combine errors in a word order and morphology.
The workflow on the CAT cloud platform includes (see Figure 5): • Manual translation and implementing MT and TM into TT segments (when needed) • Comments to ambiguous segments and verified information • Transfer of translated segments to the individual TM to save them for further projects • Editing TT segments • Indicating edited segments for a project manager • Downloading the TT to edit and revise the CAT output.
We analyze the errors in the CAT output, which the translator already edited and revised.
We compare the set of annotated errors with a distribution of errors in NMT output described in [26]. Let us explain our choice of this description. Modern Hebrew is a language with limited resources. The Google NMT involves English as a pivot while producing translations from languages with limited resources [27]. The Google neural MT (NMT) offers a suitable solution for MT from/into Modern Hebrew that is available for all customers. Therefore, presently, Hebrew-Russian and Russian-Hebrew translation with the help of the Google NMT involves English because of the limited digital resources in Hebrew that are required for training NMT system. The implication of using a morphologically poor language as a pivot for translating between two morphologically rich languages is that much of the data are lost, and the output tends to be ungrammatical. In translation from Hebrew (Modern and Archaic) into English, omissions and additions occur due to the high degree of compression in Hebrew [28]. Therefore, the omissions and additions might penetrate into the translation from Hebrew into other languages due to the involvement of English as the pivot. Thus, we analyze the errors in EnglishRussian NMT outputs as well. The detailed analysis of the distribution of the errors in the EnglishRussian NMT output is shown in [26]; we use these results of the analysis as a baseline for our comparison.
We will describe the errors in translation starting from automatically recognizable errors; then, we will proceed to analyze the errors that automatic systems are not able to recognize. While describing the errors, we provide readers with detailed semantic and discourse analysis. Based on the analysis, we will discuss our classification and the influence of CAT platform features and the human factor on the error distribution.
The analysis of the error distribution was carried out on the translation project involving a document of the journalistic style since this project allows us to study discourse errors and errors in translation strategies on the document level. A team of professional translators performed a high-quality translation of the tourist guide; some errors were caused by applying different personal styles. The error distribution in the revised CAT output reflects the particular characteristics of the translation and revision on the CAT platform. The distribution of the errors in the draft is distinct from the chart (see Figure 6).
Unrecognizable by automatic evaluation systems errors cover 18%, accuracy errors cover 24%, while fluency errors cover 58% of the errors in the draft.
The percent of the orthographic and punctuation errors is surprisingly high, as the CAT platform gives access to automatic spelling and grammar checking of the translated segments. In the translation on the CAT platform, the errors are caused mostly by the inter-language interference due to the effect of English. Almost 60% of the spelling errors follow the norms for capitalization in English (see Ковчег Завета (Ark of the Covenant) in segment 100 in Figure 3). These norms in the English language are vastly different from those in Hebrew and Russian. The errors in punctuation reflect the influence of English as well [29]. In the Russian translation of the Hebrew ST, the class of grammatical errors includes the borrowing of English syntactic constructions that causes the influence of the rules of English punctuation on the TT in Russian. Particular peculiarities of the CAT platform design induce errors in the spelling and punctuation. The text formatting indicators in the working window on the CAT platform (see Figure 4) are able to cover punctuation marks.
The grammatical errors mostly appear in long compound sentences or complex sentences with long chains of syntactic dependencies (see segment 99 in Figure 3). These errors belong to the word order and morphological subclasses. The morphological errors occur in combinations with governing verbs (Verb+Noun) or prepositions (Prep+Noun) when the translator erred in Noun inflections. The errors in syntax, including an incorrect word order, are often caused by inter-language interference that brings in borrowings of syntactic constructions from English and Hebrew. In (1), the syntactic construction in the Russian translation is borrowed from the English language: (1) תינוריע הירפס תמקהל ןומימ תורוקמ יתשפיח ונמזבש ,ינאו and I that in time that I looked for sources funding for construction library municipal а я, находясь тогда в поиске средств для создания муниципальной библиотеки and I has being then in search of funds for constructing municipal library
Word order errors regularly occur on the CAT platform due to the influence of inter-language interference. In addition to inter-language interference, the list of syntax errors includes mischoices of prepositions, irrelevant gerund and participle constructions.
The stylistic errors are caused by the preference of the official style in the TT that is irrelevant to the style of Russian tourist guides. The style shifting in a sentence regularly occurs in the TT as in (2) where the literary style collocation всяческое добро occurs in the sentence of the official style: (2) םיגוסה לכמ סעצנומש דצל ,םיאלקחהמ תורישי תוקריו תוריפ ןאכ ואצמת you will find here fruits and vegetables directly from the farmers along with small worthless objects of all kinds Вы найдете здесь овощи и фрукты всех сортов, привезенные фермерами, а также всяческое добро
You will find here fruits and vegetables of all kinds brought by farmers as well as all kind of goods. 4.2.
The richness of the Russian vocabulary provides the translators from Hebrew with sets of semantically similar lexemes, which differ in their semantic valence and connotations (see (2)).
Omissions and additions are rare in the human translation. The errors occur on the CAT platform when the translator does not revise the MT output while incorporating it into the TT. Nevertheless, an omission appeared in manual translation when the translator simplified the meaning of the sentence. (3) .העבגב ואצמנ הידירשש הדוצמה יבשוי תוהזל רשאב תורבס יתש ולעה םירקוחה The researches raised two opinions of the identity inhabitants the fortress whose remains were found on hill. Исследователи приводят два возможных варианта жителей крепости, руины которой находятся на холме.
Researchers bring in two possible options of inhabitants of fortress remain which are located on hill.
In (3), the translator transformed two theories about the origin of the fortress inhabitants into two options or types of the fortress inhabitants.
In manual translation on the CAT platform, the translator was confused by the terminology: (4) םידדונ תולוחב הסוכמ ףוחה רושימ היה םירשעה האמה לש םיעבראה תונש Years forties century twenty was plain the coast covered in migrating sands До сороковых годов прошлого столетия прибрежная равнина была покрыта зыбучими песками Till forties years of previous century coastal plain was covered by quicksand.
A Russian reader would obtain a wrong information due to the terminology substitution in (4) since the sentence in the ST refers to another peculiarity of the Israeli coastal plain.
Nevertheless, accuracy errors are 24%, while fluency errors are 58% of the errors in the draft. 4.3.
A case of inconsistency is disclosed when a marine in an Israeli port is referred to by the word from international vocabulary along with the Russian word in the different chapters of the TT: (5) הלוחכה הנירמה
Blue marine Голубая марина / Синяя марина / Голубая пристань для яхт
Blue marine / Dark-blue marine / Blue dock for yacht The inconsistent reference to the marine represents the error in the translation strategy.
The discourse errors distort the discourse features of the tourist guide. In a description of an Israeli chess club, the translator unintentionally quoted an adventurer from the famous Russian satirical novels: (6) דחאכ עונהו םירגובמה תוירוגטקב םימישרמ םיגשיה םע ,תימואלניבו תימואל טמחש תמצעמ Superpower chess national and international with achievements impressive in categories adult and youth as one. Шахматная держава, национальная и международная, прославившаяся достижениями как в юношеской, так и во взрослой категориях.
Chess empire national and international famous by achievements both in youth and in adult categories.
The unintentional quote added an ironic estimation to the city described as ‘Chess Empire’ that contradicts the pragmatic purpose of the tourist guide translation. Besides that, the semantic discrepancy alongside style shifting distorted the discourse peculiarities of the TT.
The errors reveal the lack of the top-down and non-linear strategies and neglecting cultural associations while revising the project.
The error distribution for human translations on CAT platforms differs from the error distribution observed in NMT output2. According to [30], users of MT output expect to find the following errors: • Accuracy errors (64% of respondents) • Fluency errors (57%) • Stylistic errors (40%) The accuracy errors are evaluated as the most typical feature of MT output.
The occurrence of translation errors depends on the typology of the TT and the contrastive characteristics of the SL and the TL. Error analysis based on MT output and manual translation from/into Hebrew has attracted the interest of few researchers ([31 – 33], [28]). To the best of our knowledge, the results of errors analysis in Hebrew-Russian MT and CAT have not been discussed yet.
Since our material represents the translation into the morphologically rich language, we take into consideration the difference in the error distributions in NMT output and human translation into languages with rich morphology. Error analysis in translation from English into morphologically rich languages are included in our discussion since English is the pivot language in the Google NMT system. Our results correspond to the descriptions of translators’ errors in morphologically rich languages (Dutch, German, Arabic, Hebrew, and Slavic languages).
The distribution of the fluency and accuracy errors in the English-Russian NMT output (according to [14, 26]) and Hebrew-Russian CAT on the cloud platform is shown on the chart in Figure 7. The chart shows the data (percentage) for punctuation, grammatical, mistranslation, omission, addition error classes. 2 We avoid discussing the difference between statistical and neural MT, as it is irrelevant to our research.
In the MT output for different language pairs, fluency errors slightly dominate over accuracy errors: 51.7% vs 48.3% [17]. In the human translation, accuracy errors cover one-third, while fluency errors include two-third of the linguistic errors [21]. In general, the human translation appears to be more accurate than the MT output.
While discussing the contrast between errors distributions in NMT outputs for morphologically rich languages and those in the CAT on the cloud platform, we take into consideration the automatically unrecognizable errors as well.
Orthographic and punctuation errors. Orthographic and punctuation errors do not frequently occur in NMT outputs. In English-Russian NMT output, 4% of the errors were annotated as errors in punctuation [14]. In the English-Dutch manual translation of newspaper articles, punctuation errors covered 8% of the errors; in the English-Dutch post-edited NMT output of the same material, punctuation errors represented 5% of the errors, while typos covered 7% of the evaluated errors [11]. Capitalization errors are considered relatively harmless since they do not affect the TT content [11]. Thus, orthographic and punctuation errors cover less than 10% of the translators’ and post-editors’ errors; NMT output appears to be better than human translation. Punctuation errors in the Hebrew-Russian translation frequently occurred on the CAT platform probably due to indicators of text formatting in the working window that distract the translators’ attention. Inter-language interference also causes these errors.
Grammatical errors. Grammatical errors are the most typical class of fluency errors in NMT output into Russian. According to human evaluations, English-Russian NMT output received marks “Near native or Native” for 75% segments [26]. The most frequent errors were morphological (38%), while wrong word order occurred in 9% of the segments [26]. Differences in the morphology of the SL and the TL affect the probability of morphological and syntactic errors. Nevertheless, morphological errors are frequent in MT output even for typologically similar kindred languages as Hebrew and Arabic [16, 32]. The morphological information learned by NMT systems depends on the morphology and syntax of the TL [33]. The syntactic dependencies in a clause are often distorted in translation into a morphologically rich language due to insufficiency of morphological markers in the SL [33]. In a morphologically rich language, NMT systems sometimes generate incorrect syntactic dependencies and apply surplus markers of word connections in the TT. The errors in syntactic agreement and governing are able to penetrate post-edited MT output, such as in the English-Dutch translation of newspaper articles [15]. Meanwhile, errors in syntactic agreement could be foreseen due to the linguistic checkpoints as described in [20]. For Hebrew-Russian NMT, the combinations with governing verbs (Verb+Noun) or prepositions (Prep+Noun) are to be included in the set because they are extremely sensitive to inter-language interference that can cause errors.
In human translations, grammatical errors uncover inter-language interference and particular translators’ difficulties in selecting inflections [19]. Loan translation and irrelevant borrowings from English regularly occur in CMC [34]. Meanwhile, Hebrew-Russian and English-Russian inter-language interference is obvious in morphology and syntax (see (1)). Compared to the grammatical errors in NMT output, human translation errors on CAT platforms reveal fewer morphological and word order errors (see Fig. 5). Human translators are able to avoid errors in syntactic agreement, verbal tense and aspect but err in using prepositions and other indicators of syntactic governing [19].
Stylistic errors. The stylistic errors class includes style shifting and the use of an incorrect register. Automatic evaluation systems sometimes recognize these errors in MT output as mistranslation when the TL lexeme does not correspond to the style and register of the original word in the SL [35]. However, their damage to the product depends on the genre and discourse features of the ST. Appropriate register and style are of importance in healthcare texts [36]. Style shifting is unallowable in politics and business communication. Since official and legal documents are very sensitive to style shifting and to tone/register errors while being translated, the United Nations Parallel Corpus contains manually translated documents [37]. To the best of our knowledge, statistics regarding the errors in the MT of official documents have not been published. In the Hebrew-Russian translation on the CAT platform, the stylistic errors cover 13% of errors. The class of stylistic errors includes shifts to the official style and colloquial speech. Recipients’ sensitivity to style shifting varies in different cultures. Therefore, the automatic evaluation and detection of these errors tends to be more complicated.
The inaccuracy seems to be a typical feature of MT output [30]. Three different semantic procedures for ST content processing lead to accuracy errors in TT: semantic shifting (mistranslation and terminology errors), omission and addition.
Mistranslations. Mistranslations cover approximately 30% of the errors in English-Russian NMT output [26, 38]. Low frequency words and expressions processing by MT systems often results in lexical mischoice [11]. When the algorithm fails to process a combination of words, it may select an occasional lexeme of the TL [19]. This mistranslation subclass is unlikely to occur in human translations. In technical, legal, healthcare and academic translations, the selection of an incorrect lexeme brings in reputational and material damage [39]. On the CAT platform, mistranslations appear under the influence of inter-language interference or due to lack of the competence in the Russian culture (see (6)). Nevertheless, mistranslations in CAT or manual translation are rare in comparison with those in MT output. Mistranslations, including terminology errors, cover 30% of the translation errors in the English-Russian NMT output [26], while these errors cover 22% on the CAT platform. The terminology errors were as frequent on the CAT platform, as they are in MT output (see [25, 26, 35]). Thus, the Hebrew-Russian CAT is slightly more accurate than the NMT output.
Omissions. An omission causes a semantic gap in a TT segment compared to its origin. Omissions cover 12% of the errors in English-Russian NMT output. Erroneous omissions are frequent in translations from/into Hebrew both in MT output and in manual translation [28, 31, 32]. In the study of an English-Hebrew manual translation of a healthcare document, the list of inconsistencies due to omissions covers 39% of the errors [31]. In the draft on the CAT platform, the omission appears once.
Additions. In both MT output and manual translations, additions (insertions) appear to be infrequent. Additions cover 11% of the errors in English-Russian NMT output. In an English-Russian NMT of fiction, additions did not appear at all [38]. Meanwhile, with the intent of localizing the content to the target culture, translators manually added new words in the TT. Recipients prefer translations with optional additions and explanations in intercultural communication [40]. Additions are immensely helpful in focusing recipients’ imagination; however, erroneous additions fail to provide a focus. Not including the necessary (or obligatory) additions, we consider optional additions potentially erroneous.
Since the accuracy and fluency errors are recognizable by automatic systems, development of an automatic evaluation system, which is available at CAT platforms, might enhance the CAT quality.
Unrecognizable errors belong to the category of text-external adequacy errors according to [8]. Textinternal adequacy errors cover omissions and distortions in the text content, style and genre. The omissions are included in the accuracy errors category; the distortions are sometimes revealed in mistranslations, which automatic evaluation systems are able to recognize. Based on the classifications in [8, 12] and our analysis of the errors on the CAT platform, we distinguish four classes of the adequacy errors. We refer to these errors as functional errors category (see Figure 8). The category corresponds to the text-external adequacy errors and text-internal errors (as erroneous anaphora resolution in distant segments of the ST). The classes of the category cover the discourse, genre, cultural and semiotic aspects of the text.
The category contains these classes of errors since they obstruct the functioning of the TT in the target culture [12]. These crucial errors diminish the value of the TT and misrepresent the author’s point of view and the ST message.
Errors in transferring the author’s opinion. On the CAT platform, errors in the translation strategy sometimes occur due to the different approaches applied by the teammates of the joint project. The errors cause a misrepresentation of the conceptual content and its background expressed in the ST. Since the errors become obvious on the document level, the automatic evaluation systems fail to recognize them in the MT output. To evaluate these errors, the systems need to consider the ST content within a relevant professional field. With regard to translators, they do not consider TT as “their text” [41]; therefore, professional translators intend to transfer the author’s viewpoint into the TT. Simplifications of the viewpoint are often caused by an asymmetry between the source and target languages and cultures. The simplification of the narrative appears due to the selection of lexemes in the TL that lack the important semantic features [42]. In the CAT of the tourist guide, the author’s emotional viewpoint was sometimes substituted by an official evaluation due to selecting an official style.
Errors in the translation strategy. Errors in the translation strategy are associated with incorrect processing the ST message that is reflected in the TT and its targeting. Strategy errors include integrating the gloss into the final product. The gloss integrating is a side effect of the ‘horizontal’ translation by an MT system or human translator. Errors in the analysis of the ST domain by the human translator may also lead to choosing an incorrect strategy. An instance of an error in the human strategy of translation is presented in applying various models of transferring names and brands in the different parts of the document (see (5)).
Pragmatic and discourse errors. Pragmatic and discourse errors are mostly unrecognizable by automatic evaluation systems due to their connection to the social norms, genres and culture [22]. The class of discourse errors partly overlaps with the class of stylistic errors. A discourse error presupposes a mismatch of the language style and the objective of the communication in the social context, while a stylistic error appears in mismatches of the language unit and the context. Mistranslations are also included in the discourse errors class when the incorrect lexical choice causes a violation of cultural norms or distortion of quotations and cultural associations (see (6)). Errors in anaphora resolution in distant segments are considered discourse errors since establishing incorrect connections for pronouns distorts the semantic coherence of the text integrity. Pragmatic errors are revealed in the mismatch of the pragmatic peculiarities of speech acts. In CAT, discourse errors are manifested in stylistic inconsistency on the document level or an incorrect communicative register that does not match the objective and the value of the communicative act represented in the TT. The errors might be recognizable in MT output through a non-linear comparison with the ST. The industry has not implemented new systems and models of the non-linear comparison since the projects are still far from the state-of-the art level.
Text-external errors in information processing (bringing logical and fact errors in the TT) occur due to limited capacities of MT systems and a translator’s failure to process ST content against the backdrop of the general image of the world. Instances of the errors are discussed in [18, 31, 39] in different aspects. Authors of the MT post-editing guide [18] emphasize a connection of these errors to terminology in MT output. In medical discourse, the fact errors and wrong logical connections are associated with additions and simplification through a lexical mischoice [31]. Byrne explains legal consequences of the fact errors for the translator and translators’ responsibility for information verification [39]. The error are invisible for automatic evaluation systems. This class includes errors in the procedures of information verification in the CAT cloud platform workflow (see Figure 5); translators are responsible to handle information and data analysis. 5.2.
Human translators contribute to the CAT on cloud platforms providing the ST message transferring with a holistic view and understanding of the target culture. They are able to switch translation strategies when needed and evaluate discrepancies between source and target cultures. The translators are responsible to revise the CAT output and handle the automatically unrecognizable errors applying nonlinear strategy for TT revising.
Nevertheless, the errors in the edited CAT output might be caused by the human factor. The errors are associated with ST segmentation, the platform design and its perception by users. Segmentation of the ST on the CAT platform is able to provoke errors since the segmentation prompts the translator to perceive a segment as a separate utterance. Due to the segmentation, the translator may use various translation strategies while selecting a lexeme in the TL to refer to the same object in different segments due to manual bottom-up translation (see (5)). The opportunity to avoid this error presupposes implementing TM; however, the translator do not trust the shared TM or overestimate its reliability [6]. Experienced translators adopt a non-linear strategy, focusing on the global structure and content of the ST [15], but they are hardly able to apply this strategy while working under pressure.
Translators often prefer to translate directly from the scratch applying bottom-up approach. The errors in orthography and punctuation occurs in manual translation on the CAT platform due to the effect of the design on translator’s perception of the linguistic material. These errors penetrate into the draft when the translator and editor neglect the required tools or overestimate them.
When communicating with colleagues and verifying information in the ST, the translator uses the lingua franca. Three active languages in communication increase probability of the inter-language interference [5] thereby the grammatical errors, mistranslations and incorrect terminology caused by the interference penetrate into the draft.
Since the CAT output combines MT and manual translation, access to automatic error evaluation systems from СAT platforms might improve the fluency and accuracy of the CAT output. The linguistic check-points approach to enhance productivity of the automatic evaluation systems might be helpful for recognition of the errors including those that are caused by the inter-language interference. Some of the check-points are common for almost all language pairs (e.g., idioms and collocations, ambiguous words, modal verbs, etc.). For translation from Hebrew into Russian, as far as we consider, the list of critical linguistic categories includes word order, peculiar syntactic constructions (such as noun phrases and participle constructions), verbal aspect and grammatical voice.
To evaluate adequacy of the TT, the translator needs to apply the top-down and non-linear strategies to the CAT output revision. The top-down strategy of the TT monitoring reveals inconsistencies in transmitting the author’s opinion. The non-linear strategy allows evaluating the coherence and the adequacy of the distant segments of the TT.
The influence of the human factor is obvious in the CMC. In CMC, the acceptability and usability of a TT do not directly depend on the number and gravity of errors [34]. An MT output or a poor quality TT could be accepted. The norms of the communicative sphere and the objective of the message affect TT acceptability and usability.
The human factor may diminish the TT quality due to the following peculiarities of the CAT cloud platform workflow and design: • distractors in the working window design; • necessity of translanguaging (inter-language interference between English and the TL along with the influence of the SL on the TL); • multitasking (switching from the text generating to the text evaluating while combining manual translation with post-editing of MT or TM insertions); • accessibility of the computer tools that leads to overestimation of their reliability Thus, the technologies in the translation industry enhance the speed of translation delivery, but they are still required further development to improve the quality of translation. The workflow of the CAT cloud platforms forces translators to improve their professional competencies including linguistic, communication and technological skills.
Technologies facilitate translating while transforming the translation process into CMC with English as the lingua franca. CAT platforms and MT systems allow for translating and detecting errors as well as for the quick delivery of the final product; translators can concentrate their efforts on transferring the content of the ST and its message from the source to the target culture. Nevertheless, the new technological environment provokes new challenges and reveals the importance of a holistic view of text peculiarities for translation. Different types of the translation errors are recognizable due to combining bottom-up and top-down revising strategies. To detect functional errors, a translator needs to switch strategies while revising MT or CAT outputs, giving priority to the top-down strategy. The translator and post-editors are responsible for correcting the functional errors since the industry is unable to provide a state-of-the-art technology for the recognition and evaluation of functional errors for morphologically rich languages. In the industry, error classifications do not include functional errors because the developers of MT systems and CAT platforms have not been able to develop a reliable tool to prevent these errors. Nevertheless, the industry has developed automatic systems to detect the basic classes of errors and evaluate them for various language pairs. Meanwhile, the functional errors belong to the translators’ area of responsibility.
The details of CAT platform design lead to fluency errors, while the use of new computer aids increases the likelihood of inter-language interference. The errors in CAT reflect the insufficiency of the bottom-up strategy in revising the TT. The top-down strategy for ST and TT processing belongs to the human’s area of expertise. The translator still represents the key figure in the industry. 7. References [1] European Masters in Translation Competence Framework – 2017, European Commission, EU, 2017. URL: https://ec.europa.eu/info/sites/info/files/emt_competence_fwk_2017_en_web.pdf. [2] K. Cagiltay, B. Bichelmeyer, A. G. Kaplan, Working with multicultural virtual teams: critical factors for facilitation, satisfaction and success, Smart Learning Environments 2 (2015) 11. https://doi.org/10.1186/s40561-015-0018-7. [3] A. Sakamoto, M. Foedisch, No news is good news, Translation Spaces 6 (2) (2017) 333–352. doi: 10.1075/TS.6.2.08SAK. [4] D. D. Woods, L. J. Johannesen, R. I. Cook, N. B. Sarter, Behind human error: Cognitive systems, computers and hindsight, in A.W. Schopper (Ed.), State-of-the art report (SOAR) Number 94-01, Crew System Ergonomic Information Analysis Center, Ohio, Wright-Patterson Air Force Base, 1994. URL: https://apps.dtic.mil/dtic/tr/fulltext/u2/a492127.pdf. [5] P. Ecke, Parasitic vocabulary acquisition, cross-linguistic influence, and lexical retrieval in multilinguals, Bilingualism: Language and Cognition 18 (2) (2015) pp.145–162. doi: 10.1017/S1366728913000722. [6] O. Blagodarna, Insights into post-editors' profiles and post-editing practices, Revista Tradumàtica: traducció i tecnologies de la informació i la comunicació 16 (2018) 35–51. doi: 10.5565/rev/tradumatica.198. [7] U. Reinke, State of the art in translation memory technology, in: G. Rehm, F. Sasaki, D. Stein, A.
Witt (Eds.), Language technologies for a multilingual Europe: TC3 III (Translation and Multilingual Natural Language Processing 5). Berlin: Language Science Press, 2018, pp.55–85. doi: 10.5281/zenodo.1291947. [8] G. Hansen, A Classification of errors in translation and revision, in: M. Forstner, L.-J. Hannelore, P. A. Schmitt (Eds.) CIUTI-Forum 2008: Enhancing Translation Quality: Ways, Means, Methods, Bern, Peter Lang, 2009, pp.313–326. URL: https://gydehansen.dk/media/59/a-classification-oferrors-in-translation-and-revisionpeter-lang.pdf. [9] C. Nord, Translating as a purposeful activity: Functionalist approaches explained, 2nd. ed.,
London, Routledge, 2018. doi: https://doi.org/10.4324/9781351189354. [10] M. Thelen, Translation quality assessment or quality management and quality control of translation?, in: B. Lewandowska-Tomaszczek, M. Thelen (Eds), Translation and meaning, Part 8, International Maastricht-Lodz Duo Colloquium on “Translation and Meaning” 4th 23-25 September 2005 Lodz, Poland, Universitaire Pers Maastricht UPM, Maastricht, 2008, pp.411– 424. [11] J. Daems. A translation robot for each translator? A comparative study of manual translation and post-editing of machine translations: process, quality and translator attitude. PhD thesis, Ghent University, Belgium, 2016. Id 8058017. URL: https://biblio.ugent.be/publication/8058017. [12] D. M. Buzadzhi, V. V. Gusev, V. K. Lanchikov, D. V. Psurtsev, New Scope at the Classification of Translation Errors, Translation Centre of Russian Federation, Moscow, Russia, 2009 (In Russian). [13] M. Kunilovskaya, N. Morgoun, A. Pariy, Learner vs. professional translations into Russian: lexical profiles, The International Journal for Translation and Interpreting Research 10 1 (2018) 33–52. doi: 10.12807/ti.110201.218.a03. [14] A. Shimorina, E. Khasanova, C. Gardent, Creating a Corpus for Russian Data-to-Text Generation using neural machine translation and post-editing, in: T. Erjavec, M. Marcińczuk, P. Nakov, J. Piskorski, L. Pivovarova, J. Šnajder, J. Steinberger, R.Yangarber (Eds.) Proceedings of the 7th Workshop on Balto-Slavic Natural Language Processing, August, 2019, published by Association for Computational Linguistics, Florence, Italy, pp.44–49. doi: 10.18653/v1/W193706. [15] J. Daems, S. Vandepitte, R. J. Hartsuiker, L. Macken, Identifying the machine translation error types with the greatest impact on post-editing effort, Frontiers in psychology 8 (2017) pp.1282. doi.org/10.3389/fpsyg.2017.01282. [16] F. Klubička, A. Toral, V. M. Sánchez-Cartagena, Fine-grained human evaluation of neural versus phrase-based machine translation, The Prague Bulletin of Mathematical Linguistics 108 (1) (2017) 121–132. doi: 10.1515/pralin-2017-0014. [17] N. Aranberri, E. Avramidis, A. Burchardt, O. Klejch, M. Popel, M. Popovic, 2016. Tools and guidelines for principled machine translation development, in: N. Calzolari, K. Choukri, T. Declerck, S. Goggi, M. Grobelnik, B. Maegaard, J. Mariani, H. Mazo, A. Moreno, J. Odijk, S. Piperidis (Eds.), Proceedings of the 10th International Conference on LREC. Portorož, Slovenia, Published by the European Language Resources Association, 2016. pp.1877–1882. URL: https://www.aclweb.org/anthology/volumes/L16-1/ [18] I. Massardo, J. van der Meer, S. O’Brien, F. Hollowood, N. Aranberri, K. Drescher, MT postediting guidelines, The Netherlands, TAUS Signature Editions, 2016. [19] I. G. Ovchinnikova, (2019). Working on Computer-assisted Translation platforms: New advantages and new mistakes, Russian Journal of Linguistics, 23(2) (2019) 544-561. doi: 10.22363/2312-9182-2019-23-2. [20] M. Popović, Error classification and analysis for machine translation quality assessment, in: J.
Moorkens, S. Castilho, F. Gaspari, S. Doherty (Eds), Translation Quality Assessment, Volume 1 of Machine Translation: Technologies and Applications, Springer, Cham, 2018, pp.129–158. doi.org/10.1007/978-3-319-91241-7_7. [21] E. Barbu, C. P. Escartín, L. Bentivogli, M. Negri, M. Turchi, C. Orasan, M. Federico, The First Automatic Translation Memory Cleaning Shared Task, Machine Translation 30 (3-4) (2016) 145–166. doi.org/10.1007/s10590-016-9183-x. [22] K. S. Smith, On integrating discourse in machine translation, in: B. Webber, A. Popesku-Belis, J. Tiedemann (Eds.), Proceedings of the Third Workshop on Discourse in Machine Translation, Published by Association for Computational Linguistics, Copenhagen, Denmark, 2017, pp.110– 121. doi: 10.18653/v1/W17-48. [23] C. Bryant, M. Felice, Ø. E. Andersen, T. Briscoe, (2019, August). The BEA-2019 shared task on grammatical error correction, in: H. Yannakoudakis, E., Kochmar, C. Leackok, N. Madnani, I. Pilan, T. Zesch (Eds.), Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications, Published by Association for Computational Linguistics, Florence, Italy, 2019, pp. 52–75. URL: https://www.aclweb.org/anthology/W19-44. [24] L. Specia, K. Harris, A. Burchardt, M. Turchi, M. Negri, I. Skadina, Translation Quality and Productivity: A Study on Rich Morphology Languages, in: S. Kurohashi, P. Fung (Eds.), Proceedings of Machine Translation Summit XVI, Published by European Association for Machine Translation, Volume 1: Research Track, Nagoya, Aichi, Japan, 2017, pp. 55-71. URL: http://aamt.info/app-def/S-102/mtsummit/2017/wpcontent/uploads/sites/2/2017/09/MTSummitXVI_ResearchTrack.pdf. [25] J. Munday, Evaluation in translation: Critical points of translator decision-making. Routledge,
London, 2012. https://doi.org/10.4324/9780203117743. [26] S. Castilho, J. Moorkens, F. Gaspari, R. Sennrich, V. Sosoni, P. Georgakopoulou, P. Lohar, A.
Way, A. Barone, M. M. Gialama, A comparative quality evaluation of PBSMT and NMT using professional translators, in: S. Kurohashi, P. Fung (Eds.), Proceedings of Machine Translation Summit XVI, Published by European Association for Machine Translation, Volume 1: Research Track. Nagoya, Japan, 2017, pp.116–132. URL: http://aamt.info/app-def/S102/mtsummit/2017/conference-proceedings/. [27] M. Johnson, M. Schuster, Q. V. Le, M. Krikun, Y. Wu, Z. Chen, N. Thorat, F. Viégas, M.
Wattenberg, G. Corrado, M. Hughes, J. Dean, Google’s multilingual neural machine translation system: Enabling zero-shot translation, Transactions of the Association for Computational Linguistics 5 (2017) 339–351. doi: 10.1162/tacl_a_00065. [28] T. Cheesman, A. Roos, Version Variation Visualization (VVV): Case studies on the Hebrew Haggadah in English, in: Special Issue on Computer-Aided Processing of Intertextuality in Ancient Languages of Journal of Data Mining and Digital Humanities, July, 5 (2017) 1–12. HAL Id: halshs-01532877 URL: https://halshs.archives-ouvertes.fr/halshs-01532877v2. [29] I. Ovchinnikova, D. Morozova, Comparative analysis of errors in MT output and computerassisted translation: Effect of the human factor, in: M. Forcada, A. Way, J. Tinsley, D. Shterionov, C. Rico, F. Gaspari (Eds.), Proceedings of Machine Translation Summit XVII, Published by European Association for Machine Translation, Volume 2: Translator, Project and User Tracks, Dublin, Ireland, 2019, pp.88–94. URL: https://www.aclweb.org/anthology/W1967. [30] B. Heinisch, V. Lušicky, User expectations towards machine translation: A case study, in: M.
Forcada, A. Way, J. Tinsley, D. Shterionov, C. Rico, F. Gaspari (Eds.) Proceedings of Machine Translation Summit XVII, Published by European Association for Machine Translation, Volume 2: Translator, Project and User Tracks, Dublin, Ireland, 2019, pp.42–48. URL: https://www.aclweb.org/anthology/volumes/W19-67. [31] A. Neuman, D. F. Greenberg, D. R. Labovitz, L. A. Suzuki, Cross‐cultural adaptation of the Sensory Profile: establishing linguistic equivalency of the Hebrew version, Occupational Therapy International 11(2) (2004)112–130. https://doi.org/10.1002/oti.201.