Given their status as unwritten visual-gestural languages, research on the automatic recognition of sign languages has increasingly implemented multisource capturing tools for data collection and processing. This paper explores advancements in Italian Sign Language (LIS) recognition using a multimodal dataset in the medical domain: the MultiMedaLIS Dataset. We investigate the integration of RGB frames, depth data, optical flow, and skeletal information to develop and evaluate two computational models: Skeleton-Based Graph Convolutional Network (SL-GCN) and Spatiotemporal Separable Convolutional Network (SSTCN). RADAR data was collected but not included in the testing phase. Our experiments validate the effectiveness of these models in enhancing the accuracy and robustness of isolated LIS signs recognition. Our findings highlight the potential of multisource approaches in computational linguistics to improve linguistic accessibility and inclusivity for members of the signing community.
Italian Sign Language (LIS- Lingua dei Segni Italiana) is
the primary means of communication within the Italian
signing community. Due to their visual-gestural
modality, sign languages (SLs) were initially not
considered fully-fledged linguistic systems. However,
since the 1960s, beginning with Stokoe’s pioneering
works [
In the Italian context, the study of signed
communication began in the early 1980s, involving both
hearing and deaf researchers. At that time, what we now
call LIS was still mostly unnamed and was often referred
to as ‘mime’ or ‘gesture’ by both signers and non-signers
alike [
In the late 1980s, Italy introduced subtitles in movies
on television, marking a step toward content
accessibility. The importance of media accessibility,
through subtitles or LIS interpreting, was accentuated
during the COVID-19 pandemic. The need for equitable
access to critical information for deaf individuals
became evident, with efforts born within the community
000-0002-7087-1819 (G. Caligiore), 0000-0002-1171-5672 (R.
Mineo); 0000-0001-6653-2577 (C. Spampinato); 0000-0001-6893-7076
(E. Ragonese); 0000-0002-2441-0982 (S. Palazzo);
0000-0003-30831676 (S. Fontana)
© 2024 Copyright for this paper by its authors. Use permitted under
Creative Commons License Attribution 4.0 International (CC BY 4.0).
stressing the central role of LIS in ensuring that the deaf
signers received accessible information during
challenging times [
Within this evolving societal and linguistic framework, the increased media visibility of LIS and the introduction of video capturing tools in daily lives, language collection emerges as a central issue. For SLs, the need for comprehensive collections is particularly significant. Unlike oral languages, which in some cases have developed standardized written systems, SLs must rely on video collections to capture signed communication accurately. These videos, whether raw or annotated, are essential for analyzing SLs with both qualitative and quantitative evidence.
The development and use of preferably annotated SL datasets or corpora are crucial for training and validating automatic recognition models, and access to high-quality data from diverse SLs and cultural contexts enhances the generalizability of these solutions. Comprehensive data collections of this kind ensures that models can effectively understand and process the wide range of linguistic and cultural nuances present in different SLs.
In the domain of automatic sign language
recognition (SLR) of LIS, the integration of visual and
spatial information presents a complex challenge. As
mentioned, LIS operates through the visual-gestural
channel. More precisely, it is characterized as
multimodal2 (signed discourse is comprised of manual
and body components) and multilinear (manual and
body components are performed simultaneously) [
Machine learning technologies, particularly deep
learning neural networks, have facilitated the
development of more precise and robust models for SL
interpretation. These models are able to refine their
performance through training on diverse and complex
2 Given our group’s interdisciplinarity, we found “multimodal” can
mean different things depending on one’s background: in linguistics,
it refers to the employment of manual and body components while
signing, while in computer vision, it means using multiple capturing
tools. To differentiate, we use “multisource” for capturing tools.
Thus, “multimodal” in this text follows SL linguistics terminology.
datasets. Additionally, computer vision plays a central
role in this field by enabling real-time analysis and
interpretation of body and manual components [
A significant challenge in applying deep learning and computer vision methods to SLR lies in ensuring the quality and adequacy of training data, which is essential for achieving optimal model performance.
Therefore, in this study, we focus on evaluating the efficacy of the MultiMedaLIS Dataset (Multimodal Medical LIS Dataset) and assessing various deep learning models for SLR which employ advanced deep learning techniques to interpret isolated signs by integrating diverse data types such as RGB video, depth information, optical flow, and skeletal data.
We benchmark our Dataset with two models: the Skeleton-Based Graph Convolutional Network (SLGCN) and the Spatiotemporal Separable Convolutional Network (SSTCN). These models are trained on the MultiMedaLIS Dataset, showcasing how the incorporation of multisource data can enhance the accuracy of sign recognition. This approach aims at testing the potential of integrating different data modalities to improve the robustness and performance of SLR systems.
In this section, we discuss the state of the art from two perspectives considered during our work on the Dataset: LIS data collection and SLR tools
SL researchers in Italy have been actively engaged in the creation of LIS corpora and datasets. This effort involves a complex process of video data collection and annotation, as SL datasets can vary significantly depending on their intended use. Within this context, SL data collections can be categorized into two main types. The first type includes datasets that feature videos depicting continuous signing, capturing the flow and context of natural SL usage. The second type comprises datasets that focus on isolated signs, which are individual signs presented separately from continuous discourse.
The scarcity of available LIS data collections has prompted researchers to develop their own resources. Several smaller-scale LIS corpora have been independently established, each serving distinct purposes based on the type of data collected.
The methodologies employed for collecting LIS data
encompass a diverse array of approaches, ranging from
naming tasks to semi-structured and spontaneous
interviews with deaf signers, to video recording sessions
involving hearing individuals learning LIS as a second
language (L2) or second modality (M2) [
Despite the predominant private nature of corpora
collections, an exception to the accessibility challenge is
found in the online dictionary SpreadTheSign, a project
originating in 2004. Initially conceived as a dictionary
for SLs, SpreadTheSign has evolved into a versatile
resource for language documentation [
Like SL data collections, SLR approaches can be broadly classified into two main categories: those that rely on specialized hardware and those that use visual information. The former employ specialized hardware, such as gloves able to capture precise hand movements. While these systems can provide detailed data, they are often considered intrusive and can compromise the natural flow of communication. Additionally, they are unable to capture the full spectrum of SLs, which includes manual and body components. In contrast, vision-based approaches use visual information captured by cameras, including RGB, depth, infrared, or a combination of these. These methods are less intrusive for users, as they do not require the use of special equipment.
In SLR, a challenge lies in effectively capturing both
body movements and specific motions of hands, arms,
and face. For instance, [
Another study proposes an iterative optimization
alignment network tailored for weakly supervised
continuous SLR [
[
However, as previously noted, RGB-based SLR
systems can raise privacy concerns, particularly when
processing visual data in cloud environments or for
machine learning training [
As part of the Dataset discussed in this work, we have also collected RADAR data and are actively analyzing it. However, preliminary results are not available at this time, so they are not included in this report. Currently, RADAR-based solutions have demonstrated robust performance across diverse environmental conditions, highlighting the productivity of incorporating this sensor technology in data collection efforts. Nevertheless, many existing RADAR solutions are tailored to recognizing a limited set of signs, highlighting the ongoing challenge of expanding vocabulary recognition capabilities in datasets like the one discussed in the following section.
The MultiMedaLIS [
For the data recording protocol, the DIEEI group developed a customized recording software to collect the LIS data, supplemented with a desktop computer and a modified keyboard transformed into a pedal board. This pedal board, equipped with two pedals, allowed handsfree navigation of the software, enabling users to move forward (by pushing on the right pedal) or backward (by pushing on the left pedal) while maintaining a neutral recording position3. During sessions, one of 126 Italian labels or alphabet letters was displayed on a screen, with adjustable display time for preparation and transition from one sign to the other. Each recording started from a neutral position, and the right pedal marked the completion of a sign. If errors occurred, the left pedal allowed re-recording. The software’s interface features a color-coded background: yellow for preparation and green for recording. Additionally, it supports flexible data expansion, accepting word lists from text files for easy customization in future collections.
After the recording process, Dataset included
synchronized data capturing facial expressions, hand
and body movements and comprises a total of 25,830
sign instances. This includes 205 repetitions of 100
different signs and the 26 signs of the LIS alphabet [
Regarding the specific form of signs, the
MultiMedaLIS Dataset includes a lexicon of standard,
isolated signs that are not combined within utterances.
3 The neutral recording position referenced is a seated position in
which the user has their arms extended along the sides of the torso,
elbows bent at 90°, and palms facing downward [
These signs reflect forms commonly found in online dictionaries and educational materials. To ensure the accuracy of the data, sign variants performed by a professional LIS interpreter during the collection of a test dataset were compared with the same variants found in the online dictionary SpreadTheSign. This comparison aimed to select documented versions of each sign for inclusion in the Dataset. By incorporating these documented variants, we aimed to enhance its precision, reliability, and real-world applicability. This approach contributed to ensuring that the Dataset aligns with established standards and supports effective research and application in the field of LIS.
When discussing recording tools for state-of-the-art
multimodal corpora in the Italian context, such as the
Corpus LIS [
Portable and non-invasive recording tools are chosen specifically for their ability to capture data in familiar, and sometimes domestic, settings without disrupting the signer’s surroundings, aiming to maintain the authenticity of the signed interactions and minimize any discomfort or distraction for the participants.
To capture LIS for recognition with minimal
invasiveness we integrated a combination of recording
tools. A 60GHz RADAR sensor, employed to capture
detailed manual motion data, provided Time- and
Frequency-Domain data and Range Doppler Maps for
distinguishing moving objects at 13 fps. For more
structured depth and facial recognition data, the
Realsense D455 depth camera and Kinect v1 were
incorporated. The Realsense D455, equipped with dual
infrared cameras and RGB mode, captured depth data at
848x480 pixels and RGB data at 1280x720 pixels, both at
30 fps, enabling the tracking of facial expressions
through 68 facial points. The Zed v1 and Zed v2 cameras
provided high-resolution stereoscopic data, recording at
1920x1080 pixels and 25 fps, with capabilities for
generating depth maps and 3D point clouds.
Additionally, the Zed v2 offered tracking for 18 body
points in both 2D and 3D [
By prioritizing portability and non-invasiveness, high-quality data can be still collected, while respecting the privacy and comfort of the individuals recorded. Anonymization is achieved through the use of the RADAR sensor, which we introduced specifically to address privacy concerns inherent in face-to-face signed communication.
The MultiMedaLIS Dataset was designed with the aim of supporting the development of SLR models by enabling the collection and integration of information through various data modalities: • RGB frames: images extracted from videos. • Depth data: three-dimensional information for each RGB frame • Optical flow: to emphasize movement • Skeletal data: face landmarks and body joints One of the main components of the Dataset are RGB frames, which are images extracted from videos. These frames provide a two-dimensional visual representation of the signs performed by the signer, capturing details such as hand positions and facial expressions. The Dataset includes depth data, providing a threedimensional aspect to the images. allowing for more detailed information on the distance and relative position of elements in the scene. This type of data is particularly useful for understanding the spatial dynamics of signs.
Alongside RGB and depth data, the MultiMedaLIS Dataset also contains optical flow information, which describes the movement between consecutive frames. Optical flow is essential for capturing the direction and speed of movements, providing a more detailed understanding of the transitions between various signs. Finally, the Dataset includes skeletal data, representing face landmarks and body joints, allowing for precise tracking of joint and body segment positions, facilitating the analysis of signs in terms of joint movements.
Managing this multimodal data is an emerging topic in computational linguistics. By combining different sources of information, it is possible to significantly improve the performance of SLR models. For example, integrating depth data with RGB frames can provide a more complete representation of signs, while adding optical flow and skeletal data can further enrich the analysis of movement’s temporal structure. In our view, the MultiMedaLIS Dataset provides a solid foundation
In the context of automatic SLR, various approaches and model architectures have been tested to leverage the characteristics of multimodal data in the MultiMedaLIS Dataset.
The SL-GCN (Skeleton-Based Graph Convolutional
Network) represents a significant innovation in this
field. This model generates skeletal data from videos and
creates temporal graphs that capture the spatiotemporal
relationships between joint movements. Through
finetuning and the combination of different data streams,
SL-GCN has demonstrated high accuracy in sign
recognition [
Another prominent architecture is the SSTCN
(Spatiotemporal Separable Convolutional Network) [
RGB frames are crucial for the visual representation
of signs. The process of splitting videos into frames,
cropping, and normalization optimally prepares the data
for analysis by deep learning models. The use of dense
optical flow presents significant challenges in sign
recognition. Optical flow extraction using the Farneback
algorithm [
For the training of the models, we employed a multistream approach that integrates skeletal, RGB, and depth data to improve sign recognition accuracy. The models were trained on a NVIDIA Tesla T4 16GB GPU using the Adam optimizer with an initial learning rate of 0.001 and a batch size of 8. We applied cross-validation to ensure the robustness of the results, splitting the Dataset into training (70%) and validation (15%) subsets and data augmentation techniques, such as color jittering, changing the brightness, contrast, saturation and hue, to increase the diversity of the training data and improve generalization.
The loss function adopted for training was categorical cross-entropy, appropriate for multi-class classification tasks. The models were trained for a maximum of 100 epochs, with an early stopping criterion set to terminate training if no improvement in validation loss was observed for 10 consecutive epochs. For evaluation, we used a test set comprising 15% of the Dataset, ensuring that the models were tested on unseen data.
The results demonstrate the model’s efficiency in leveraging multi-modal data for improved outcomes. As can be seen in Table 1, the SL-GCN multi-stream model achieved the best accuracy, with a Top-1 accuracy of 97.98% and a Top-5 accuracy of 99.94%, surpassing the performance of models using single data streams such as skeletal joints, bones, or motion alone. This demonstrates the advantage of combining multiple streams of information to capture both spatial and temporal dynamics of signs. optical flow data alone, reaching just 56.31% accuracy, suggesting that while the optical flow provides valuable information on motion, it lacks the richness of spatial features found in RGB and depth data. The HHAencoded depth data, when processed with the ResNet(2+1)D model, achieved an accuracy of 88.04%, confirming that depth information is complementary, but not as effective as RGB data in isolation.
In this study, our goal was to demonstrate our first steps into testing the efficacy of the MultiMedaLIS Dataset in contributing to the advancement of the field of SLR through multisource approaches. The integration of RGB frames, depth data, optical flow, and skeletal data has provided a comprehensive basis for developing and evaluating SLR models. Our experiments with the SLGCN and SSTCN architectures have highlighted advancements in recognizing isolated LIS signs in medical semantic contexts, given the domain of our Dataset.
The SL-GCN model, trained on skeletal data to construct temporal graphs, achieved accuracy in capturing spatiotemporal relationships critical to sign recognition. This approach not only enhances the precision of rendering LIS signs but is also reinforced by a Dataset able to support robust graph-based convolutional networks in multimodal SLR tasks. At the same time, our Dataset proved robust, precise and variable enough for SSTCN model testing, focusing on spatiotemporal separable convolutions, revealing robust performance in extracting spatial dynamics from RGB frames.
Having validated the visual modalities on the mentioned models, we have promising preliminary results on adapting these models to accept RADAR data. We plan to extract the pre-trained RADAR data processing module and use it independently during inference. This approach will eliminate the need for RGB visual data. Furthermore, we plan to expand the Dataset by applying the same protocol with 10 deaf signers. This will effectively increase the current Dataset, enhancing the generalizability across different signers. Our goal is to develop an autonomous, resource-constrained system (thanks to the exclusion of RGB data) that operates onedge or even offline. This cost-effective solution can be used in any emergency contexts where direct access to interpreting is not available.