The development of home video recording has had a big impact in the development of video documents containing Italian Sign Language (LIS) sentences. LIS2SPEECH is an ongoing project by Orbyta Tech s.r.l. to build a complete translation chain from LIS to speech. The idea is to build a free software framework to transform video containing LIS sentence into Italian vocal sentences. In this way, LIS signers can indirectly produce Italian vocal sentences. In this paper we describe two milestones for LIS2SPEECH, that are: i. the development of some deep neural models trained by using data augmentation technique, and ii. the construction of a new dataset for LIS to Italian. Referring to the first point, a number of deep learning models were developed and tested. Then data augmentation was performed by using some geometric transformations to the videos belonging to the original training set. With reference to the second point, we constructed the TGLIS-227 dataset by using video and audio segmentation techniques, starting from a corpus of RAI newscasts. This dataset is a novelty in the current research panorama as there are no public datasets for LIS with sentence-level granularity.
In this paper, we present the architecture of a real-time translation system from Italian Sign Language (henceforth LIS) to Italian speech. An ideal system platform for this task should be composed of three main modules: • a first module which, starting from an input video, returns the glosses1 contained in the video. So, this module performs a Sign Language Recognition
Task. • a second module that translates the glosses into the Italian language. So, this module performs a
Sign Language Translation task. • a third module for text-to-speech system, that is for pronouncing the sentences in Italian. So, this module performs a Text to Speech task.
The task of Sign Language Recognition (SLR) is a classification task that allows to automatically obtain the glosses corresponding to the signs performed by a signer in a video. In general, the SLR is approached as a multiclass classification problem, i.e. a type of supervised learning which, on the basis of a (statistical) model, can associate the correct gloss among the , where is the cardinality of the LIS dictionary considered. In the specific settings of the LIS2SPEECH 2 project, the input is a sequence of signs encoded in a video while the labels are the corresponding glosses. So, this is a case of supervised learning, where the models are trained on a dataset containing numerous examples (signs) labeled with the relative class (gloss).
Considering the module for translation from LIS to
Italian, this implement a task where is true the rule “more
data is better data”. In this paper we follow this
prescripby using a data augmentation technique. We enlarge the other research studies have also been based on
Transpossibility given by a relatively small initial dataset, by formers architectures [
For the LIS there are few datasets (see Section 3), and
as a consequence we used the only one suitable for SLR,
2. Deep Learning models for that is the A3LIS-147 dataset [12]. A3LIS-147 was built
by the A3LAB research group of the Università
politecItalian SLR nica delle Marche, in collaboration with the ENS (Ente
Nazionale Sordi, the Italian National Deaf institution) of
In all experiments concerning neural learning there are Ancona. The dataset is composed of 1480 video. The
two crucial ingredients, these are the dataset and the corpus contains 147 standard (natural) signs, plus one
neural architecture. In this Section, we describe a number special (artificial ) sign for representing the “silence” (sil
of work related to our project (Section 2.1), we describe sign). The latter is not a sign belonging to an LIS natural
the dataset employed in our experiments (Section 2.2), dictionary, but it encodes the common resting position
we describe the details of the preprocessing and training in corpus conversations.
applied (Section 2.3) and, finally we report results of our Crucially, and in contrast to most electronic LIS
dictioexperiments (Section 2.4). naries, all the signs of A3LIS-147 have been performed by
10 diferent signers. This peculiar property of the corpus
2.1. Related Works on SLR allows us to use it as a training set for the isolated sign
language recognition task (ISLR henceforth). For this
speThe major related works in sign language recognition cific task, each video represents a single sign preceded
consider 2 important features: the input and neural mod- and followed by the sil sign (or rest).
els used. First, the input could be static (an image for each
sign) or dynamic (a video for each sign). The diferent
granularity of the input allows the SLR to be divided into 2.3. Preprocessing and Training
two diferent tasks: isolated sign language recognition In this Section, we describe the development of a deep
(ISLR) and continuous sign language recognition (CSLR). neural system for ISLR trained on the A3LIS-147.
The former takes a single sign as input and outputs the First of all, we have a preprocessing phase for
concorresponding gloss. The latter instead takes as input a verting videos into numerical data suitable for learning.
sentence or a sequence of signs and returns the correct In preprocessing, we extracted a total of 543 keypoints
sequence of glosses. [
ber of signs increased, provoking an out of memory error. 2. The results had a very bad recognition accuracy when considering the entire 148 signs dataset. For these two reasons, we decided to perform two other preprocessing steps on the data for solving these issues.
These two steps work, in some senses, in two opposite directions. On the one hand, we performed data reduction (Section 2.3.1) for optimizing the number of features given in input to the neural ISLR classification model. On the other hand, we performed data augmentation (Section 2.3.2) on the number of videos for each sign. Indeed, we realized that 10 videos for each sign are too small number to allow the network to correctly classify. We discuss the impact of these two steps in Section 2.4 where we report the results of the experiments with various neural models.
We reduced the number of keypoints extracted for each frame with the Google model by considering: Translation The following transformation applied to each original keypoint (x,y,z):
was , , = ( + ∆ , + ∆ , ) ∆ = .. (− , ) ∆ = .. (− , ) where ∆ represents the displacement on the x axis, while ∆ represents the displacement on the y axis. Both these delta were randomly extracted from an uniform distribution by using, as the range, the values representing the maximum translation downwards, upwards, rightwards, leftwards. Using these limits, we guarantee that all the keypoint coordinates are still values between 0 and 1. The randomly extracted values are the same for all frames of a video. In Figure 2c, we report an example of applying this transformation.
Rotation The following transformation was applied to each original keypoint (x,y,z): = (− ) − ( − ) + = ( − ) + ( − ) + • the number of keypoints of the face is higher than other parts of the body and this could nega- = tively afect the training of the model giving too much importance to this part of the body. For this reason we developed a function which allows us to go from 468 to 128 representative keypoints The rotation was performed with respect to the on the contours of the face, eyes, eyebrows and center of all keypoints = (, ). is mouth. the angle of rotation: the value is randomly extracted • The Mediapipe documentation recommends dis- from the uniform distribution between (− 20, 20) and carding the dimension because the Google sys- is the same for all the keypoints of the frames of one tem still has low performances in predicting the video. In Figure 2a we can see an example of applying depth. For this reason, in some tests we discarded this transformation to the keypoints of a video frame. the of each keypoint. In other words, we converted the original 3D data produced by medi- Flip The following transformation was applied to each apipe into 2D by just removing the value5. original keypoint (x,y,z): • Finally, we applied the principal component analysis (PCA) to reduce the total number of keypoints to four-six components, which represents , , = (2 + , , ) the 95% of explained variance. 2.3.2. Data Augmentation It is an axial symmetry with respect to the straight line = parallel to the y axis. In our case k corresponds We applied a Data Augmentation technique by increasto the x coordinate of the center of all keypoints. This ing the number of videos for each sign by making some type of transformation is important because the same geometric transformations to the originals. In particular sign can also be performed symmetrically, because there we performed: translation, rotation, flip and smoothing. are right-handed and left-handed signers. Without this 5In a diferent trial, we have tried to set the coordinate to zero. transformation, symmetrical signs cannot managed (a) original and rotated (b) original and smooth (c) original and translated (d) original and flipped properly by the neural models, since it recognizes them as diferent stimulus. In Figure 2d, we can see an example of applying this transformation to the keypoints of a video frame.
Smooth This type of transformation was implemented
by considering the study presented in [
The first group, called the general group, contains all the possible combinations (480 tests) of the parameters shown in Table 2 (Appendix B). In this group all the values of the parameters were tested. The second group, called specific data augmentation group, is designed to test the impact of each data augmentation transformation on results. It contains some combinations of all the possible combinations in Table 3 (Appendix B)
The results of each test is reported with the results of accuracy, precision and recall curve, confusion matrix, F1-score.
We upload the results of the test on the online platform QlikSense6 in order to build a dashboard that allows us to visualize them. In this Section, we use the these graphical representations to comment results.
In Figure 3 we observed that if we consider the first group of tests, without data augmentation, the average accuracy decreases if the number of signs increases. The accuracy of a naive model that selects a class at random among the N possible ones using a uniform probability distribution are indicated in red. We use this baseline for accuracy for the models that we have trained. Moreover,
to correctly evaluate the generalization power, we consider for accuracy, the precision, the recall, the F1-score and the confusion matrix. In the Figure 3 and 4, we have that the baselines are 33% for 3 signs, 6% for 16 signs, 2% for 50 signs, 0.6% for 148 signs. In general, considering N signs, the baseline is equal to 1 since the classes are balanced in the training dataset.
In Figure 5, we consider average accuracy for diferent neural models, and we have a very wide range of values for accuracy (from 0 to 0.8). In contrast, in the data since the classification task is evaluated on a relatively small dataset of distinct signs (148 signs).
A small prototype system was developed to test the performances of the neural models implemented in real-time.
A number of issues arose for this specific context. As Figure 6: Box plot avg accuracy VS models - with data aug- we discussed, all the neural models were trained by apmentation plying PCA to the data, but this specific preprocessing step creates problems for a real time application. Indeed, for each prediction of the input, it is necessary to apply augmentation experimentation, there is no decrease in PCA to the data collected slowing the real-time perforaccuracy when the number of signs increases (Figure 4) mance. To solve this specific issue, we used a specific and there is a significant increase in the accuracy with parallel thread, and by running the PCA on another parrespect to the models (Figure 6). Furthermore from the allel thread: in this way we have been able to reduce the second group of tests we understood that the smooth is impact of this problem. By using the OpenCV library the best transformation. This demonstrates that increas- we have developed a function that allowed us to read ing the number of videos for each sign, that is simulating the video from the webcam frame by frame. So, for each the generation of new videos, had an important impact frame, the detection of the mediapipe keypoints was peron the results. Moreover, the results show that in Figure formed and saved them in an array. The first prediction 7 it seems that there is no real diference in using all took place when we have collected this information for at the coordinates for each keypoint (that are x,y,z) or only least frames, where is the number of frames of the two (that are x,y). From experimentations, the best neu- input shape of the model being tested. After that, each ral model seems to be CONVNET (configured with the extraction would corresponded to a prediction which is specific parameters in Table 4, Appendix B). Indeed, we still based on the last frames present in the array of achieved 100% accuracy (Figure 8) and 100% precision, extracted keypoints. Predictions that exceeded a certain recall, F1-score on each class. However, we are awzre threshold of probability fixed (in tests performed = that these impressive results can be due to overfitting, 0.7), are shown in a bar at the top of the window.
According to [
So, exploiting these features, we developed a system to automatically segment each newscasts video. By identifying the parts of the newscast video containing the silence-LIS in the images and the silence in the audio, we produced a number of videos containing a single news item for each one. The working hypothesis is that the the silences correspond to the transition from one news to an another.
To perform silence-LIS detection we use YOLOv7 (an object detector). To train a YOLO model we needed many images that represented the object to detect, that is the silence-LIS. Since no silence datasets exist, we built it by extracting frames from RAI newscasts. So, we obtained a silence dataset that contained 8000 silence LIS images with resolution 640x640 and with two annotations: only hands or hands+elbows. 3.2. Pipeline construction We downloaded 20 newscasts from Rai Play and we did a pre-processing step by removing the theme song and by cropping the video to focus on the LIS signer. Then we detected silence-LIS using the YOLO model trained on the silence dataset. Thereafter we detected silences audio for each video: first we extracted audio from video (using moviepy library functions) then we detected only silence that were at least 2 seconds long (using PyDub library function). All detections were recorded into tabular CSV format. Moreover, we built a filter algorithm that took in input detections and returned the ranges that corresponds to the transition from one news item to an other. By using this information, we splitted the newscast in the corresponding news by obtaining a new video file and an audio file for each news. Finally, we annotated each news item (in CSV) with two extra fields:
nition (pytesseract) to crop the title of the news items shown in the video 2. the transcript: we applied speech recognition (the Azure SDK speech to text) These annotations are important in a translation context because the topic represents the context of translation and the transcript represents the target of translation. 3.3. Dataset Final Structure The final dataset is called TGLIS-227 since it is composed of 227 distinct news items extracted from LIS newscasts editions. For each of them we have: 1. a video (mp4): containing the LIS news item. 2. an audio (wav): that is the audio of the news item in the Italian (vocal) language. 3. topic (in csv): containing the topic of the news item (Appendix E), 4. transcript (in csv): that is the automatic transcription of the news item(Appendix F).
In this work we have presented two main results obtained in the LIS2SPEECH project. First, we have described the application of a number of data augmentation techniques to some deep neural models in the task of ISLR. We proved with experiments that some of these transformations have a strong impact on the final performance of the classification task. Second, we built the TGLIS-227, a new sentence-level dataset for LIS, applying a new procedure for the automatic segmentation of the newscasts.
In future work we intend to develop the following two ideas:
they contain; 2. to develop a system like Common Voice [17] to collect more data to build an open source dataset for LIS; Note that we built an automatic procedure that could be applied several times in order to increase the size of the dataset. Moreover, a more challenging development could be to
A crucial weakness of the actual dataset is the lack of a encode additional two video features that are: the lips standard for LIS transcription in some written form. This and facial expressions. Finally, we noted that very often linguistics issue requires the collaboration with Deaf or- in the news the signers “read” the gloss by using their ganizations and could be performed by using annotation lips and, moreover, express an emotion related with the tools for videos such as ELAN [16]. gloss by using their facial expression.
Finally note that for copyright issues we cannot distribute the audio/video content of news items directly, but References only the annotations (Appendixes C, D, E, F)8. However, by using the timestamps of each news item (Appendix C and Appendix D), and requesting access to the “Teche Rai”, it is possible to extract the video and the audio from original newscasts9. 3.4. Testing deep learning for ISLR on
TGLIS-227 videos We tried to test the best model described in section 2 on the TGLIS-227 dataset (section 3). Since we do not have an LIS annotation (e.g in glosses) for each video, we did a very raw evaluation of the correctness of the ISLR predictions by using the lemma corresponding to the Italian news transcript. In particular, we counted the number of matches between predictions and lemmas, obtaining around 33% of correct matches. This low value is consequence of the the diferent size of the training dataset, containing only 147 signs, with respect to the size of the TGLIS-227 dataset, containing around 5000 lemma. 8data are available at this GitHub: https://github.com/BeanRepo/TG LIS-227 9Note that the timestamps are calculated on the videos without start-end theme songs sign language recognition, 2018 Conference on Signal Processing And Communication Engineering
Systems (SPACES) (2018) 194–197. [7] J. Huang, W. gang Zhou, H. Li, W. Li, Sign language recognition using 3d convolutional neural networks, 2015 IEEE International Conference on
Multimedia and Expo (ICME) (2015) 1–6. [8] R. Kumar, A. Bajpai, A. Sinha, Mediapipe and cnns for real-time asl gesture recognition, 2023.
arXiv:2305.05296. [9] M. Borg, K. P. Camilleri, Sign language detection “in the wild” with recurrent neural networks, in: ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2019, pp. 1637–1641. doi:10.1109/ICAS
SP.2019.8683257. [10] G. Samaan, A. Wadie, A. Attia, A. Asaad, A. Kamel,
S. Slim, M. Abdallah, Y.-I. Cho, Mediapipe’s landmarks with rnn for dynamic sign language recognition, Electronics 11 (2022) 3228. doi:10.3390/el ectronics11193228. [11] D. Kothadiya, C. Bhatt, K. Sapariya, K. Patel, A.-B.
Gil-González, J. M. Corchado, Deepsign: Sign language detection and recognition using deep learning, Electronics 11 (2022). URL: https://www.mdpi .com/2079-9292/11/11/1780. doi:10.3390/electr onics11111780. [12] M. Fagiani, S. Squartini, E. Principi, F. Piazza, A new italian sign language database, 2012. doi:10.1 007/978-3-642-31561-9_18. [13] V. Grishchenko, V. Bazarevsky, R. Engineers, G. Research, Mediapipe holistic — simultaneous face, hand and pose prediction, on device, 2020. URL: https://ai.googleblog.com/2020/12/mediapipe-hol istic-simultaneous-face.html. [14] A. Cardinaletti, Il progetto spread the sign, BLITYRI (2016). doi:https://hdl.handle.net/10278 /3691616. [15] Sign-Hub, Sign-hub: Wp 2.4, 2020. URL: https: //hdl.handle.net/11403/sign- hub- wp- 24/v1, ORTOLANG (Open Resources and TOols for LAN
Guage) –www.ortolang.fr. [16] T. L. A. R. f. h. Nijmegen: Max Planck Institute for
Psycholinguistics, Elan[computer software], 2023.
URL: https://archive.mpi.nl/tla/elan. [17] R. Ardila, M. Branson, K. Davis, M. Henretty,
M. Kohler, J. Meyer, R. Morais, L. Saunders, F. M.
Tyers, G. Weber, Common voice: A massivelymultilingual speech corpus, in: Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), 2020, pp. 4211–4215.
(a) BIGRU (b) BILSTM
Appendix C. Video Timestamps10 This appendix is a sample of video timestamps file. The full version is available on github: https://github.com/BeanRepo/TGLI S-227
Appendix D. Audio Timestamps11 This appendix is a sample of audio timestamps file. The full version is available on github: https://github.com/BeanRepo/TGLI S-227 10All data in this appendix is protected by Creative Commons Licence
CC BY-NC-SA 4.0. 11All data in this appendix is protected by Creative Commons Licence
CC BY-NC-SA 4.0.
Appendix E. Topic News12
This appendix is a sample of topic file. The full version is available on github: https://github.com/BeanRepo/TG LIS-227
I RISULTATI DELLE ELEZIONI USA: ITALIA PARTNER IMPORTANTE FUGA DALLA RUSSIA PER NON ARRUOLARSI IL PRIMO ESPERIMENTO DI DIFESA PLANETARIA L’ITALIA BATTE L’UNGHERIA ED È NELLE FINAL FOUR ELEZIONE PRESIDENTE, IERI FUMATA NERA NUOVO IMPULSO AL CONFRONTO TRA I PARTITI CRISI UCRAINA, ALTA TENSIONE RALLENTA LA CURVA DELL’EPIDEMIA OGGI BERRETTINI GIOCA I QUARTI DI FINALE L’ELEZIONE DEL PRESIDENTE, CONTATTI TRA I PARTITI COVID, ALLO STUDIO ESTENSIONE GREEN PASS CRISI UCRAINA, DIPLOMAZIA AL LAVORO STRAGE DI LICATA, UN PAESE IN LUTTO GIORNO DELLA MEMORIA, PAPA : MAI PIÙ QUESTI ORRORI QUIRINALE, ALLE 11 COMINCIA LA QUINTA VOTAZIONE UCRAINA, TELEFONATA BIDEN-ZELENSKY OK DELL’EMA ALLA PILLOLA ANTI-COVID TENNIS, SEMIFINALE BERRETTINI-NADAL 12All data in this appendix is protected by Creative Commons Licence
CC BY-NC-SA 4.0. ...
Ed è tutto grazie per averci seguito. Il tg uno torna alle 8, buona giornata.
Un giorno dal tg uno la corsa al Quirinale comincerà alle 11, il quinto giorno di votazioni. Il centrodestra sarebbe orientato a votare uno dei nomi proposti nei giorni scorsi. Contrario a questa scelta il centrosinistra, che per protesta potrebbe uscire dall’Aula al momento del voto.
Intanto il presidente della Camera Roberto Fico.
Ha convocato alle 10:15 la Conferenza congiunta dei capigruppo di Camera e Senato per decidere se procedere a una doppia votazione giornaliera.
Cresce la tensione tra Stati Uniti e Russia sulla questione Ucraina telefonata tra Zelensky e Biden.
Per il Presidente americano c’è la possibilità concreta che i russi invadano l’Ucraina nel mese di Febbraio.
La situazione Covid in Italia rallenta la curva dei contagi, calano recovery e terapie intensive e si discute della possibilità di cambiare il sistema delle fasce a colori delle regioni e anche le regole che riguardano la scuola.
Intanto è arrivato l’OK dell’EMA alla pillola anti COVID di freezer.
Tamponi sospetti e Green pass fasulli chiuso un centro analisi in provincia di Trento e Stop a un secondo punto prelievi nel capoluogo Trentino. 5 le persone indagate.
Il tennis nella semifinale degli Australian Open, in campo Matteo Berrettini e Rafa Nadal. Il punteggio al momento è di due set a uno per lo spagnolo.
Ed è tutto grazie per averci seguito. Il tg uno torna alle 8, buona giornata.
...