=Paper=
{{Paper
|id=Vol-2968/paper6
|storemode=property
|title=SignON: Bridging the gap between Sign and Spoken Languages
|pdfUrl=https://ceur-ws.org/Vol-2968/paper6.pdf
|volume=Vol-2968
|authors=Horacio Saggion,Dimitar Shterionov,Gorka Labaka,Tim Van de Cruys,Vincent Vandeghinste,Josep Blat
|dblpUrl=https://dblp.org/rec/conf/sepln/SaggionSLCVB21
}}
==SignON: Bridging the gap between Sign and Spoken Languages==
https://ceur-ws.org/Vol-2968/paper6.pdf
SignON: Bridging the gap between Sign and Spoken
Languages
SignON: cerrando la brecha entre las lenguas de signos y las
lenguas orales
H. Saggion D. Shterionov
Universitat Pompeu Fabra Tilburg University
horacio.saggion@upf.edu D.Shterionov@tilburguniversity.edu
G. Labaka T. Van de Cruys
Universidad del Paı́s Vasco KU Leuven
gorka.labaka@ehu.eus tim.vandecruys@kuleuven.be
V. Vandeghinste J. Blat
Instituut voor de Nederlandse Taal Universitat Pompeu Fabra
and KU Leuven josep.blat@upf.edu
vincent@ccl.kuleuven.be
Abstract: This article presents an overview of the SignON European project which
aims to develop technology for automatic translation between sign and oral lan-
guages (and vice-versa). In order to achieve this objective, the project takes a
multi-disciplinary approach by involving the deaf community, sign language lin-
guistics, research in sign language recognition, speech recognition, natural language
processing and machine translation (MT), 3D animation and avatar technology, and
application development. The project follows a user-centered, community driven
approach to the development of technology.
Keywords: Neural Machine Translation; Sign Language; Automatic Sign Language
Recognition; Text Simplification; Avatar.
Resumen: Este artı́culo describe el proyecto europeo SignON, que tiene como
objetivo desarrollar tecnologı́a de traducción automática entre lenguas de signos y
lenguas orales. Para lograr este objetivo, el proyecto adopta un enfoque multidis-
ciplinario al involucrar signantes de lenguas de signos, lingüı́stas de las lenguas de
signos, tecnologı́a de reconocimiento automático de lengua de signos, reconocimiento
automático de voz, procesamiento del lenguaje natural y traducción automática, an-
imación 3D y la tecnologı́a de avatar y desarrollo de aplicaciones. El proyecto sigue
un enfoque centrado en el usuario e impulsado por la comunidad sorda para el de-
sarrollo de una tecnologı́a apropiada.
Palabras clave: Traducción automática neuronal; lenguas de signos; re-
conocimiento automático de lengua de signos; simplificación de textos; avatar.
1 Introduction hearing loss1 ; it is estimated that this num-
ber will double by 2050. According to the
World Federation of the Deaf (WFD), over
Access to information is a human right. In 70 million people are deaf and communicate
the modern, globalised world this implies ac- primarily via a sign language (SL).
cess to multilingual content and cross-lingual Machine translation (MT) (Koehn, 2009)
communication with others. Crossing lan- is a core technique for reducing language
guage barriers is essential for global infor- barriers that has advanced, and seen many
mation exchange and unobstructed, fair com- breakthroughs since it began in the 1950s
munication. The World Health Organisation (Johnson et al., 2017), to reach quality lev-
(WHO) reports that there are some 466 mil- 1
https://www.who.int/news-room/fact-
lion people in the world today with disabling sheets/detail/deafness-and-hearing-loss
Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
21
els comparable to humans (Hassan et al., language recognition (SLR), automatic
2018). Despite the significant advances of speech recognition (ASR), and natural
MT for spoken languages in the recent cou- language understanding (NLU).
ple of decades, MT is in its infancy when it 4. Research and development of a novel Lan-
comes to SLs. The complexity of the prob- guage Independent Meaning Repre-
lem, automatically translating between SLs sentation for interlingua MT. It will be
or SL and spoken languages, requires a multi- based on current vector representations
disciplinary approach (Bragg et al., 2019). (Lample et al., 2018), symbolic compo-
In this paper we present the SignON nents (Baker, 2014) or hybrid representa-
project which focuses on the research and tions of the input/output message.
development of a sign language translation 5. Sign, speech and text synthesis.
mobile application and an open communica- SignON will convert an SL specific
tions framework. SignON aims to rectify syntactic-semantic representation in the
the lack of technology and services for au- target SL via a customizable 3D virtual
tomatic translation between sign and spo- signer (i.e. avatar). It will also produce
ken languages, through an inclusive, human- text output in the different oral languages
centric digital transformation solution facili- adapted to the user by, for example, sim-
tating communication between deaf and hard plifying the text.
of hearing (DHH) and hearing individuals. 6. Wide-range of supported languages
and extensibility of the framework.
2 Project overview During the project we will provide sup-
SignON is a Horizon 2020 project which port for the following SLs: Irish SL (ISL),
aims to develop a communication service that British SL (BSL), Flemish SL (VGT),
translates between sign and spoken (in both Dutch SL (NGT) and Spanish SL (LSE) as
text and audio modalities) languages and well as English, Irish, Dutch and Spanish
caters for the communication needs between verbal languages. However, we design the
DHH and hearing individuals. Currently, hu- SignON application and framework to be
man interpreters are the main medium for extensible to new sign and spoken/written
sign-to-spoken, spoken-to-sign and sign-to- languages.
sign language translation. The availability
2.2 Challenges
and cost of these professionals is often a limit-
ing factor in communication between signers Achieving the aforementioned objectives, and
and non-signers. The SignON communica- thus the envisaged service is not a simple en-
tion service will translate between sign and deavour.
spoken languages, bridging language gaps First, the difference between sign and verbal
when professional interpretation is unavail- languages as well as between the different SLs
able. (Vermeerbergen and Van Herreweghe, 2010)
makes it difficult to adopt translation pro-
2.1 Objectives cesses that have been developed for verbal
A primary objective of the SignON project languages for the use-cases of sign-to-verbal
is to create a service that translates between (and vice-versa) or sign-to-sign translation.
sign and verbal languages. This high-level Second, there has been a lack of sufficiently
objective is broken down to the following 6 advanced technology. In order to recognise
lower-level objectives: and understand SL, tools need to be able to
1. Co-creation workflow and commu- process digital representations of signers (e.g.
nity. We aim to bring researchers and de- 2D or 3D video) and to compress the un-
velopers in a close collaboration with the derlying information into a meaningful (for
main stakeholder groups to drive the re- both humans and machines) representation
search and development in SignON. (De Coster, Van Herreweghe, and Dambre,
2. Development of the SignON Frame- 2020). Often, attempts at sign language
work and Mobile application which recognition (SLR) require cumbersome addi-
will deliver the SignON service to the user. tional hardware, such as gloves and bracelets,
3. Automated recognition and under- which incorrectly assume that an SL is sim-
standing of SL and verbal lan- ply articulated only on the hands.
guage input through advanced sign A third issue is the scarce amounts of anno-
22
tated SL materials or parallel data in which 5. The National Microelectronics Applications
signs are linked to text making it difficult to Centre Ltd (MAC), Ireland
train state-of-the-art models. 6. Pompeu Fabra University (UPF), Spain
Forth, the gap between DHH and hearing 7. Technological University Dublin (TUDublin),
Ireland
communities is big and it is often expressed
8. Trinity College Dublin (TCD), Ireland
as a lack of demand for and of willingness to 9. VRT, Belgium
adopt technological solutions. While technol- 10. Ghent University (UGent), Belgium
ogy could be of enormous benefit for each of 11. Vlaams GebarentaalCentrum (Flemish Sign
these communities, it has not yet reached the Language Centre – VGTC), Belgium
expectations of its potential users. 12. University College Dublin (UCD), Ireland
To address these challenges SignON is 13. Stichting Katholieke Universiteit (RU), The
exploring: (i) a multilingual representa- Netherlands
tion common for both sign and verbal lan- 14. Nederlandse TaalUnie (NTU), The Nether-
lands
guages (InterL); (ii) sophisticated deep learn- 15. Katholieke Universiteit Leuven (KULeuven),
ing methods for recognition; (iii) efficient on- Belgium
the-fly synthesis of detailed 3D avatars; (iv) 16. European Union of the Deaf (EUD), Belgium
an adaptive pipeline to allow the updating 17. Tilburg University (TiU) (scientific lead), The
of the models based on user input; and (v) Netherlands
a co-creation methodology bringing together
SignON researchers and the DHH commu- 4 A Co-creation Approach
nity.
SignON aims to reduce the gap be-
2.3 Current Developments tween the stakeholder communities through
MT for SLs has been addressed with different a user-centred and community-driven re-
approaches from rule-based methods (Porta search and development approach, involv-
et al., 2014), through statistical (Morris- ing stakeholder-led user profiles from its in-
sey, 2008) and to neural machine translation ception. Our co-creation strategy relies on
(NMT) (Yin and Read, 2020). Given the ob- a continuous communication and collabora-
jective of developing a multilingual, extensi- tion with DHH communities to iteratively
ble, language independent framework for rep- (re)define use-cases, co-design and co-develop
resenting and translating language, the work the SignON service and application, assess
we conducted in the first 6 months of this the quality and validate their acceptance.
project focused on (i) the adaptation of the Through co-creation we will ensure that
state of the art mBART model (Liu et al., the developed solution is (i) accepted by the
2020) to translating between English, Span- users; and (ii) that it will continue to evolve
ish and Dutch in both bilingual (the model beyond the lifetime of the SignON project.
is fine-tuned on two languages) and multilin-
gual (the model is fine-tuned on all languages 5 SignON App and framework
sequentially) settings as well as to experiment This project will develop a free, open-source
with text simplification (Saggion, 2017); (ii) service and framework for conversion be-
infrastructure and framework development to tween video (capturing and understanding
support all interleaved components; and (iii) sign language), audio (for speech, including
analysis of the current stakeholders’ attitude, atypical speech) and text, translating be-
perception and vision related to SL transla- tween sign and spoken languages, delivered
tion. to its users via an easy to use mobile ap-
plication. The operational workflow of the
3 Consortium
SignON application and framework is illus-
The following organizations participate in the trated in Figure 1.
SignON consortium: The SignON communication and trans-
1. Dublin City University (DCU) (coordinator), lation mobile application, each user’s in-
Ireland
2. Fincons Group (FINC), Switzerland
terface to the overall cloud platform and
3. Instituut voor de Nederlandse Taal (INT), The SignON framework, will run on standard
Netherlands modern smartphone and tablet devices with-
4. University of the Basque Country out the need for special equipment. Fur-
(UPV/EHU), Spain ther details on the early-stage development
23
De Coster, M., M. Van Herreweghe, and
J. Dambre. 2020. Sign language recog-
nition with transformer networks. In Pro-
ceedings of the 12th Language Resources
and Evaluation Conference, May.
Hassan, H., A. Aue, C. Chen, V. Chowd-
hary, J. Clark, C. Federmann, X. Huang,
M. Junczys-Dowmunt, W. Lewis, M. Li,
S. Liu, T. Liu, R. Luo, A. Menezes,
T. Qin, F. Seide, X. Tan, F. Tian, L. Wu,
S. Wu, Y. Xia, D. Zhang, Z. Zhang, and
Figure 1: The SignON application and M. Zhou. 2018. Achieving human par-
framework workflow. ity on automatic chinese to english news
translation. ArXiv, abs/1803.05567.
of the SignON application can be found Johnson, M., M. Schuster, Q. V. Le,
in (O’Flaherty et al., 2021). M. Krikun, Y. Wu, Z. Chen, N. Thorat,
F. Viégas, M. Wattenberg, G. Corrado,
6 Conclusions and Future work M. Hughes, and J. Dean. 2017. Google’s
This paper presents an overview of the multilingual neural machine translation
SignON project. This project interleaves system: Enabling zero-shot translation.
state-of-the-art research with continuous Transactions of the Association for Com-
communication and verification with the user putational Linguistics, 5:339–351.
communities, a process that we refer to as co- Koehn, P. 2009. Statistical Machine Trans-
creation. In its lifetime, SignON focuses on lation. Cambridge University Press.
English, Irish, Dutch and Spanish verbal lan-
guages and the following sign languages: ISL, Lample, G., A. Conneau, L. Denoyer, and
BSL, VGT, NGT, LSE. However, the design M. Ranzato. 2018. Unsupervised ma-
of the SignON framework allows for easy in- chine translation using monolingual cor-
tegration of new languages. pora only. In International Conference on
Learning Representations.
Acknowledgments Liu, Y., J. Gu, N. Goyal, X. Li,
This work is supported by the European S. Edunov, M. Ghazvininejad, M. Lewis,
Commission under the Horizon 2020 pro- and L. Zettlemoyer. 2020. Multilingual
gram ICT-57-2020 - “An empowering, inclu- denoising pre-training for neural machine
sive Next Generation Internet” with Grant translation.
Agreement number 101017255. We thank all Morrissey, S. 2008. Data-Driven Machine
members of the SignON consortium. Translation for Sign Languages. Ph.D.
thesis, Dublin City University.
References O’Flaherty, J., M. P. Scipioni, M. Villa,
Baker, C. F. 2014. FrameNet: A knowledge E. Keane, and M. Giovanelli. 2021. Early-
base for natural language processing. In stage development of the signon appli-
Proceedings of Frame Semantics in NLP: cation and open framework – challenges
A Workshop in Honor of Chuck Fillmore and opportunities. In Proceedings of Ma-
(1929-2014), pages 1–5, June. chine Translation Summit XVIII Volume
2: User Track, virtual, August. European
Bragg, D., O. Koller, M. Bellard, L. Berke, Association for Machine Translation.
P. Boudreault, A. Braffort, N. Caselli,
M. Huenerfauth, H. Kacorri, T. Ver- Porta, J., F. López-Colino, J. Tejedor, and
hoef, C. Vogler, and M. Ringel Morris. J. Colás. 2014. A rule-based translation
2019. Sign language recognition, genera- from written spanish to spanish sign lan-
tion, and translation: An interdisciplinary guage glosses. Computer Speech & Lan-
perspective. In The 21st International guage, 28:788–811, 05.
ACM SIGACCESS Conference on Com- Saggion, H. 2017. Automatic Text Simplifi-
puters and Accessibility, page 16–31. cation. Morgan & Claypool Publishers.
24
Vermeerbergen, M. and M. Van Herreweghe.
2010. Sign languages and sign language
research. In The handbook of psycholin-
guistic and cognitive processes: perspec-
tives in communication disorders. Psy-
chology Press, pages 709–729.
Yin, K. and J. Read. 2020. Better
sign language translation with STMC-
transformer. In Proceedings of COLING,
pages 5975–5989, December.
25