The paper examines the main problems that arise when translating Ukrainian Sign Language and the differences between Ukrainian Sign Language and Ukrainian Written Language. The wellknown methods of machine translation of sign languages are described, in particular, machine translation based on rules, statistical machine translation, machine translation based on ontologies, and the relatively new neural machine translation. A study was conducted on the application of statistical machine translation, namely the application of the IBM #1 model and the EM algorithm for aligning words in the sentences of the corpus of parallel texts "Ukrainian Written Language - Ukrainian Sign Language". Examples of the application of the statistical method of translation are given and the main results are described. Alignment matrices for sentence structures of the same type are generalized.
More than 44,000 people with hearing impairment are registered with the All-Ukrainian public organization of the disabled "Ukrainian Society of the Deaf". The World Federation of the Deaf (WFD) unites more than 70 million deaf people around the world and 135 national associations of the deaf.
People who communicate in sign language should be provided with comfortable access
to modern information resources. To achieve this goal, it is necessary to solve the difficult
task of translating sign language into a corresponding text record. Sign language (SL) is a
natural language with a grammatical structure and vocabulary that differs from written
language [
As a result of the growing number of applied research, the latest technologies are being used to improve the situation of people with physical disabilities. Therefore, the development of methods and means of translating sign language into text is a promising direction of research. For free communication with the deaf, it is enough to develop a system of translating sign language into text and vice versa.
Ukrainian Sign Language is a natural way of communication for deaf Ukrainians and is an integral part of their personality. The sequence of words in a sign language sentence is different from written language. Despite the lack of prepositions and cases, Ukrainian Sign Language is a multi-level linguistic system that has a wide range of lexical and grammatical tools for expressing opinions and analyzing information. Just like spoken language, sign language changes over time: new gestures appear, foreign ones are borrowed.
Tracing Sign Language literally reproduces the sentences of the written language. Gestures are used to show the root of a word, and dactyl is used to show prepositions, prefixes, suffixes, and endings. Usually, tracing sign language is used for dictation. However, the Ukrainian Sign Language is the main language in the communication of people with hearing impairments, it is much easier to understand in contrast to the tracing sign language.
The task of translating Ukrainian Sign Language (USL) into Ukrainian Written Language
(UWL) belongs to the tasks of computer translation. The following well-known computer
translation methods are used for sign language translation: rule-based translation,
statistical-based translation, ontology-based machine translation, phrase-based machine
translation, neural machine translation method. Translation programs based on rules
analyze the text and build its translation based on built-in dictionaries and a set of rules for
a given language pair [
USL has its own language organization and is considered according to the main provisions of structural linguistics. National Sign Languages are independent of the corresponding sound languages and have their own history and structure. USL consists of three main functional-structural components: kinetics (the composition of kinem), vocabulary (a set of gestures), grammar (a set of rules and means for their implementation).
The problem of computer translation of Ukrainian Sign Language into Ukrainian Written Language is caused by the fact that Ukrainian Sign Language is a language without a written form. Therefore, in order to translate USL into the Ukrainian Written Language, it is necessary to create a certain written record for USL. Since Ukrainian Sign Language is a means of communication in which only visual-kinetic means are used to convey information (hand gestures, lip articulation, facial expressions and emotions), all these features must be taken into account for the translation of USL into written language.
The translation of Ukrainian Sign Language is a complex task, which includes the analysis of the grammar of USL, the construction of rules for the translation of Ukrainian Sign Language into text and vice versa.
The main problems of computer translation from Ukrainian Sign Language into written Ukrainian are: • • • •
Use of the dactyl alphabet, pointing gestures, transliteration of proper names and terms along with gestures.
To develop a system of statistical machine translation from Ukrainian Sign Language into Ukrainian Written Language and vice versa, the following steps should be taken into account:
2. Recording of recognized signs into gloss notation; 3. Statistical machine translation of glosses into text and vice versa; 4. Transforming glosses into a conversational model; 5. Reproduction of signs by the avatar.
In this article, we consider only a part of the translation system, namely statistical machine translation.
The main steps of such a system are schematically shown in Figure 1.
There are many scientists in whole world are engaged in researching the problems of
translation of foreign sign languages, for example, S. Morrissey and A. Way [
Advances in statistical machine translation allow it to be used for automatic sign language translation. In the work of scientists A. Otman and M. Jemni [11], [12] statistical machine translation for English Sign Language is considered. Examples of the use of IBM models #1-3 for translation from English Written Language to English Sign Language, as well as the EM algorithm for word alignment are given.
The work [
Despite the fact that linguistic studies of Ukrainian Sign Language have been carried out by scientists for a long time, there is still no complete description of the grammar of USL and the basic principles of USL translation.
Ukrainian scientists Yu. V. Krak, O.V. Barmak et al. from V.M. Glushkov Institute of Cybernetics of the National Academy of Sciences [13] proposed information technology for modeling the Ukrainian Sign Language. During the implementation of the technology, scientists faced problems related to the presence of two sign languages of communication tracing and Ukrainian Sign Languages, the fact that there is no unambiguous correspondence of words to signs, as well as the lack of programs for the grammatical analysis of Ukrainian language sentences. In paper [14] new tools of alternative communication for persons with verbal communication disorders are described.
In scientific work [
Rule-based machine translation into Ukrainian Sign Language using concept dictionary
are described in [
In paper [
The scientific work [15] are described the first evaluation of the quality of automatic translation between Myanmar sign language (MSL) and Myanmar written text, in both directions. The authors proposed three different statistical machine translation (SMT) approaches: phrase-based, hierarchical phrase-based, and the operation sequence model. They are used for this translation methods MSL-Myanmar parallel corpus. The scientists used three different segmentation schemes: syllable segmentation, word segmentation and sign unit-based word segmentation. The results show that the highest quality machine translation was attained with syllable segmentations for both MSL and Myanmar written text.
In article [11] the authors described statistical machine translation of written English text to sign language. The scientists proposed a novel approach to build artificial corpus using grammatical dependencies rules owing to the lack of resources for sign language. The parallel corpus was the input of the statistical machine translation, which was used for creating statistical memory translation based on IBM alignment algorithms. These algorithms were enhanced and optimized by integrating the Jaro–Winkler distances in order to decrease training process. Subsequently, based on the constructed translation memory, a decoder was implemented for translating English text to the ASL using a novel proposed transcription system based on gloss annotation. The results were evaluated using the BLEU evaluation metric.
An overview of known methods of Sign Language Translation (SLT) is described in the work [16]. Authors also describes a review about the possible tasks related to SLs, the metrics used for the generated glosses and spoken language text and a summary of all the available public datasets and whether they are suitable for the SLT task or not. Moreover, the survey lists the challenges that need to be tackled within the SLT research and also for the adoption of SLT technologies, and proposes future research lines.
In paper [17] authors discuss machine translation from sign to spoken languages. The neural machine translation for Sign Language Translation is investigated. Describes the main problem of neural machine translation, in particular, small data sets for translation. Many datasets consider limited domains of discourse and generally contain recordings of non-native signers. This has implications on the quality and accuracy of translations generated by models trained on these datasets, which must be taken into account when evaluating SLT models.
The authors [18] described several techniques, commonly used in low resource machine translation scenarios, for machine translation from spoken language text to sign language glosses. Data augmentation, semi-supervised Neural Machine Translation, transfer learning and multilingual NMT were used for the experiments. The results of experiments carried out on two natural datasets including gloss annotation (RWTHPHOENIX-Weather 2014T dataset and the Public DGS Corpus) indicate an increase BLEU metric to 6.18.
In paper [
The paper [19] describes translation between German Sign Language glosses and German written language. The authors focuses on the second-stage gloss translation component, which is challenging due to the scarcity of publicly available parallel data. Their approach is based on gloss translation as a low-resource machine translation task and contains hyperparameter search and back-translation. For experiments, the authors use the RWTH-PHOENIX-Weather 2014T dataset. The resulting gloss-text system improves over the baseline system by a margin of 2.44 BLEU. The biggest problem, the authors note, is limited parallel data.
The scientific work [20] formalize German Sign Language Translation in the framework of Neural Machine Translation for both end-to-end and pretrained settings (using expert knowledge). To achieve NMT from sign videos, the authors employed CNN based spatial embedding, various tokenization methods, to jointly learn to align, recognize and translate sign videos to spoken text. The sign language translation dataset corpus (PHOENIX14T) was assembled for conducting experiments. Using the end-to-end frame-level method and glosslevel tokenization networks, a BLEU-4 score were achieved 9.58 and 18.13 respectively.
In paper [21] authors described the research various deep learning–based methods for encoding sign language as inputs, and analyzed the several machine translation methods using three different sign language datasets. The authors use translation methods for several sign languages, such as German Sign Language (GSL), American SL (ASL) and Chinese SL (CSL). Developed by the authors transformer model outperformed all the other sequence-to-sequence models on the GSL and CSL datasets using OpenPose features.
Statistical sign language machine translation systems use bilingual corpora that contain complete sentences. Such corpora are used to train these statistical systems. But when it comes to sign language, two main problems arise. The first problem is the lack of large corpora. Existing corpora use gloss notation (one gloss = one sign), which is too complex to learn. In addition, the inconsistent use of system notation complicates the task [13]. For Ukrainian Sign language, we have created a corpus of more than 230 sentences, which contains sentences in the Ukrainian Written Language and their translation into Ukrainian Sign Language, taking into account the basic rules of translation. The second problem is the lack of a standard for notation of signs.
We will consider the model of statistical machine translation of sign language, which is
based on lexical translation of words, that is, word-to-word translation [
Due to the difference in the order of words in Ukrainian Written Language and Ukrainian Sign Language sentences, the words need to be redistributed (aligned) during the translation process. Here is an example of a sentence in which the words UWL and USL are in the same order:
All models of statistical machine translation are based on the principle of word alignment. To align the positions of the words in the sentence, the alignment function is used, which maps each UWL word in position i to the USL word in position : : → . For the example above, the alignment function is ∶ {1 → 1, 2 → 2, 3 → 4, 4 → 3} . пишу наукову статтю (I am writing a scientific article)
[Ya pyshu naukovu stattiu ] Я ПИСАТИ СТАТТЯ НАУКОВА (I WRITE ARTICLE SCIENTIFIC)
Also, when translating to align words, there are cases when one word in UWL can correspond to many words in USL and vice versa (alignment function) ∶ {1 → 1, 2 → 2, 3 → 2, 4 → 3}): пишу статтю (I am writing an article)
[Ya pyshu stattiu ] ПИСАТИ 2 СТАТТЯ (I WRITE ARTICLE)
4 3 статтю (I wrote an article)
[Ya pysala stattiu] UWL USL 1 Я Я 1 2
3 1 Я
2 писала Я ПИСАТИ БУВ СТАТТЯ 1 2 3 4 (I WRITE WAS AN ARTICLE) [YA PYSATY BUV STATTIA]
The following rule is used to translate interrogative sentences in Ukrainian sign language: question words (for example, "how", "where", "when", "why", "how much") are always placed at the end of the sentence. For example, an interrogative sentence in USL is rendered as follows (alignment function ∶ {1 → 2, 2 → 3, 3 → 1, 4 → 4}): 1 Де ти 2
3 4 працюєш ? (Where do you work?)
[De ty pratsiuiesh ?] ТИ ПРАЦЮВАТИ ДЕ ? (YOU DO WORK WHERE ?) 1 2 3 4 [TY PRATSUVATY DE ?] Another example an interrogative sentence with the same alignment: UWL Хто черговий сьогодні
? (Who is on duty today?) 2 3 ?
( 1,...,flf) of length into a sentence UWL = ( 1,...,ele) of length with alignment of each word of UWL into a word USL in accordance with the alignment function a: j → as follows [12]: (e,a| ) =
є ( + 1) =1 ∏ ( | ( )) (1)
If there is a manually aligned corpora of parallel texts, then it is possible to estimate the parameters of the IBM #1 model by maximum likelihood. Since this corpus for Ukrainian Sign Language does not exist, we use the EM algorithm [12]. Let's consider the application of the EM algorithm on a simple example (Figure 2).
Я (I [YA ПИСАТИ
PYSATY ] ПИСАТИ СТАТТЯ (WRITE ARTICLE) [PYSATY STATTIA]
НАУКОВА СТАТТЯ (SCIENTIFIC ARTICLE) [NAUKOVA STATTIA] 1 1 article) [Ya pyshu naukovu stattiu] the alignment matrix will look like this (Figure 3):
F Я ПИСАТИ СТАТТЯ
Я ПИСАТИ НАУКОВА ПИСАТИ НАУКОВА
Я СТАТТЯ e я я я пишу пишу пишу наукову наукову статтю статтю
Alignment matrices can be generalized for the following sentence types (noun(pronoun), verb, noun(pronoun)) and shown in Figure 4. Alignment matrix for the following sentence types: noun(pronoun), verb, adjective, noun(pronoun) shown in Figure 5.
Noun(Pronoun)
Verb
Adjective
Noun(pronoun) Figure 5: Word alignment matrix for types of sentences: noun(pronoun), verb, adjective, noun(pronoun)
The words of the USL sentence (rows) are aligned to the words of the UWL sentence (columns) as shown in the alignment matrix. Alignment may not always be one-to-one. In Figure 6 shows an example when one word in UWL “писала” (wrote) [pysala] corresponds to two words in USL “ПИСАТИ БУВ” (WRITE WAS) [PYSATY BUV].
NOUN(PRONOUN) VERB AUXILIARY VERB NOUN(PRONOUN) Noun(Pronoun)
Verb
Noun(pronoun)
Finally, generalize the alignment matrices for interrogative sentence types, where the question word is always placed at the end for Ukrainian Sign Language (see Figure 7). It should be noted that this generalization applies only to these types of interrogative sentences.
NOUN(PRONOUN) VERB PRONOUN(ADVERB, NUMERAL) ? Pronoun(adverb, numeral)
Noun(pronoun)
Verb
?
An overview of the known methods of machine translation was carried out. A relatively new method of neural machine translation was studied.
The analysis of the obtained results showed the expediency of using statistical machine translation for Ukrainian Sign Language. In particular, the IBM #1 model was used to translate USL words into Ukrainian Written Language and the EM algorithm to align words in sentences. Experiments were conducted on the corpora of parallel sentences "Ukrainian Written Language - Ukrainian Sign Language", which contains 230 sentences. In addition, alignment matrices for sentence structures of the same type are generalized.
The scientific novelty consists in the creation of a corpus of parallel texts "Ukrainian Written Language - Ukrainian Sign Language", the application of IBM model #1 for the translation of sentences from this corpus.
The practical value lies in the possibility of translation from Ukrainian Sign Language to Ukrainian Written Language and vice versa for people with physical disabilities.
The following studies will focus on a more detailed study of neural machine translation for Ukrainian Sign Language.
The research reported in this paper was supported by Information Systems and Networks Department of Lviv Polytechnic National University. We would like to thank Lviv special boarding school of Maria Pokrova for deaf children. [10] R. San-Segundo, A. Pérez, D. Ortiz, L. F. D’Haro, M. I. Torres, F. Casacuberta, Evaluation of Alternatives on Speech to Sign Language Translation, in: Proceedings of Interspeech 2007, Antwerp, Belgium, 2007, pp. 2529-2532. [11] A. Othman, M. Jemni, Statistical Sign Language Machine Translation: from English written text to American Sign Language Glos, IJCSI International Journal of Computer Science Issues Vol. 8, Issue 5 (2011) 65-73. [12] A. Othman, M. Jemni, Designing High Accuracy Statistical Machine Translation for Sign Language Using Parallel Corpus: Case Study English and American Sign Language, Journal of Information Technology Research, Vol. 12, Issue 2 (2019) 134–158. doi: 10.4018/JITR.2019040108 [13] Yu. G. Krivonos, Yu. V. Krak, A.V. Barmak, Information technology for modelling the
Ukrainian sign language, Journal Artificial Intelligence, Issue 3(2009) 186-197. [14] Yu. G. Kryvonos, I.V. Krak, O.V. Barmak, R.O. Bagriy, New Tools of Alternative Communication for Persons with Verbal Communication Disorders, Cybernetics and Systems Analysis, 52(5) (2016) 665–673. [15] S. Z. Moe, Y. K. Thu, H. W. Hlaing, H. M. Nwe, N. H. Aung, H. A. Thant, N. W. Min, Statistical Machine Translation between Myanmar Sign Language and Myanmar Written Text, 2018, MERAL Portal. URL: oai:meral.edu.mm:recid/00007665. [16] A. Núñez-Marcos, O. Perez-de-Viñaspre, G. Labaka, A survey on Sign Language machine translation, Expert Systems with Applications, Vol. 213, Part B (2023). doi: 10.1016/j.eswa.2022.118993 [17] M. De Coster, D. Shterionov, M. Van Herreweghe, J. Dambre, Machine translation from signed to spoken languages: state of the art and challenges, Universal Access in the Information Society (2024) 1-27. doi:10.1007/s10209-023-00992-1. [18] D. Zhu, V. Czehmann, E. Avramidis, Neural Machine Translation Methods for Translating Text to Sign Language Glosses, in: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, July 9-14, 2023, pp. 12523–12541. [19] X. Zhang, K. Duh, Approaching Sign Language Gloss Translation as a Low-Resource Machine Translation Task, in: Proceedings of the 18th Biennial Machine Translation Summit, Virtual USA, 1st International Workshop on Automatic Translation for Signed and Spoken Languages, 2021, pp. 60-70. [20] N. C. Camgoz, S. Hadfield, O. Koller, H. Ney, R. Bowden, Neural sign language translation, in: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, pp. 7784-7793. [21] T. Ananthanarayana, P. Srivastava, A. Chintha, A. Santha, B. Landy, J. Panaro, A. Webster, N. Kotecha, S. Sah, T. Sarchet, R. Ptucha, I. Nwogu, Deep Learning Methods for Sign Language Translation, ACM Transactions on Accessible Computing 14(4) 1-30. doi:10.1145/3477498.