Tailoring Automatic Text Simplification Output for Deaf and Hard of Hearing Adults

Abstract

This paper presents a research proposal into tuning the outputs of an Automatic Text Simplification (ATS) system to the needs of the deaf and hard-of-hearing (DHH) community. There is an overview of simplification methods, and studies on ATS where it relates to the DHH community. It then presents an experimental design where research participants are presented with simplified text from an established method of ATS, and simplified text which has been post-processed to move the sentence topic to initial position - a common feature of sign languages. Each simplification method is also evaluated by automatic metrics.

Keywords

• eol>accessibility
• natural language processing
• sign language
• text simplification

1. Introduction

behind their hearing peers throughout childhood [ 4 ], which prevents access to further education and employment opportunities. In the United Kingdom, 17% of hearing children progress to top-level universities, compared to 9% of their deaf peers2. Across the European Union the unemployment rate is much higher among the DHH community, tracking with employment ifgures for anyone with a disability 34.

This reality is not inevitable. It is suggested that empowering people with disabilities including deafness will not only promote equality of opportunities, but also costs less in terms of providing unemployment benefits 5. With good quality ATS alongside the nascent area of sign language (machine) translation [ 5 ], providing a large amount of adequate materials for DHH people will become more practical and reliable.

The idea of providing ATS technology for reading assistance has been established [ 6, 7 ] as beneficial in improving access to information for DHH people. Recent work has also evaluated methods of ATS for reading assistance from a human-computer interaction standpoint [ 8 ], and the ability of evaluative metrics to measure text complexity for DHH people [ 3, 9 ]. This study aims to progress the current research by tuning methods of ATS to the needs of DHH people. It will adapt the output of an ATS system based on preferences of the DHH community from current literature, and propose some novel methods based on typologically common features of sign language.

In Section 2.1, an outline of ATS is drawn followed by a more in-depth discussion of research in the field related to DHH people in Section 2.2. Section 2.3 explores how knowledge of sign language grammar may interact with ATS research. The proposed methodology based on the research questions in Section 3 is described in Section 4. Finally, other considerations surrounding the study are located at the end of the paper in Section 5.

1.1. SignON

This work is being carried out in conjunction with the SignON6 European Union project [ 10 ]. Its goal is to create a translation service between signed and spoken languages in Europe. This initiative is community-led, with input and leadership from the DHH community at every level. It is possible that using simplified text during the training of its machine translation system would be beneficial as a pre-processing step [ 11 ].

2. Related Work

2.1. Text simplification strategies

There are three main strategies for ATS: Lexical, syntactic, and a hybrid approach encompassing both. Lexical simplification falls into two main tasks [ 1 ], finding synonyms as word-replacement candidates and word sense disambiguation (WSD). To find synonyms, it is possible to use a 2https://www.independent.co.uk/news/education/education-news/deaf-students-russell-group-universitieslevel-results-access-national-deaf-childrens-society-a8463301.html 3https://ec.europa.eu/social/main.jsp?catId=738&langId=en&pubId=7482 4Exact figures are not available, as ‘disability’ does not have an exact definition across EU states and institutions 5https://www.eud.eu/news/deaf-and-employment-crisis/ (European Union of the Deaf) 6https://signon-project.eu/ lexical database such as WordNet [ 12 ] or more recently word2vec [ 13 ] which also contains syntactic information. Context-sensitive lexical simplification [ 11 ] ensures that replacement synonyms remain grammatical, and do not alter the intended meaning. WSD has multiple strategies which interpret the idea of word complexity diferently. Some approaches rely solely on word frequency, while others incorporate word length [ 1 ].

Syntactic, or sentence-level, simplification relies on the transformation of linguistic structures which boost readability, i.e. require lower decoding and linguistic abilities [ 4 ] for the reader. For example, an utterance may be more readable if the argument structure and word order are more canonical for a given language, or through paraphrase - shortening the length and removing additional qualifiers and subordinate clauses from an utterance. Strategies for syntactic ATS have developed over the years from a hand-crafted rule-based approach [ 14, 15 ], into leveraging state-of-the-art neural machine translation models [ 16 ]. Table 2.1 shows an example of each text simplification strategy.

It has been alluded to that large databases and corpora are required to perform ATS. For English and Spanish, the proposed languages for this work, there are many resources specifically designed for this task. Simple English Wikipedia7 is a central resource spawning corpora such as the large Parallel Wikipedia Simplification (PWKP) dataset [ 17 ] and Wikilarge [ 18 ]. However, caution must be used as Simple English Wikipedia is not a parellel corpus - rather a spontaneously written new language for the encyclopaedia. In fact, up to half of the PWKP corpus are not simplifications [ 1, 19 ].

Without a simple Spanish Wikipedia-type resource, there are corpora for parallel Spanish/simplified Spanish text. As an example, Simplext [ 20 ] contains 200 utterance pairs. There are also tools to create novel simplifications for a monolingual corpus. Candidate ranking [ 1 ] and context-aware algorithms like CASSA [ 21 ] can do Ngram-based lexical simplification. Text alignment [ 22 ] as part of a pipeline, and recent deep learning methods [ 11, 16 ] may also be used to create sentence-level simplification.

2.2. Text simplification and DHH accessibility

For around 20 years [ 6 ], there has been work into both lexical and syntactic simplification for readability focused on DHH readers. Inui and colleagues also highlighted the need for multiple degrees of reading aptitude in the output, as literacy levels are highly variable across the community [ 3, 7, 9 ] and a ‘one size fits all’ approach is likely to be of-putting for DHH readers [ 8 ]. Following on from the goal of SignON [ 10 ], it is crucial to design outputs and systems based entirely on the needs and preferences of the target community for accessibility.

As a starting point, it would be useful to follow an established accessibility framework. The EASIER tool [ 23 ] was built around the Web Content Accessibility Guidelines 8, which are designed to make text content on the web available to everyone. In addition, Alonzo and colleagues studied preferences of the DHH community on the visualisation of a prototype lexical simplification system [ 8 ]. They state that it should be imperative to provide community tailored ATS output in their preferred display format, with a degree of autonomy in accessing the simplifications for themselves. It has been successful in other populations [ 23, 24 ], and ATS as reading assistance is argued to have an objective benefit to the DHH community [ 6 ].

In addition, evaluating ATS systems in the DHH communities has mostly been achieved through subjective, manual methods [ 9 ]. As automatic metrics for evaluating ATS exist (e.g. SARI [ 25 ]), it would be beneficial to incorporate these into a study alongside the opinions of target users. A logical next step is therefore to implement a syntactic or hybrid ATS system with post-processing and visualisation steps informed by the preferences of DHH readers.

2.3. Sign language interface

It is important to consider the language which most DHH people use in their daily lives, sign language. Unfortunately, datasets which could be used to create widely-available NLP-based technology are few and far between for sign languages [ 5, 26 ]. Widening accessibility may currently be achieved by simplifying text in the ambient language. By incorporating features of sign language grammar into the simplified ambient language text as far as remains grammatical, the information structure may be more accessible to DHH readers.

There are some typologically-common grammatical features between sign languages, including those which are not closely related. In the paradigm of word order classification for spoken languages, e.g. SOV (Subject-Verb-Object), sign languages are flexible on word order. Instead sign languages tend to topicalise information [ 27 ], move discourse-prominent information to the ifrst position in an utterance. An example of topicalisation is shown in Figure 1. In NLP, there exist methods to extract the topic of an utterance. For example, the Latent Dirichlet Allocation algorithm can be used to pick out topic words from word distributions in a probabilistic manner [ 28 ]. If the topic noun phrase (NP) of a sentence could be fronted, this could be a method of making simplified utterance more accessible to DHH readers.

Many more linguistic features are shared by typologically-distant Sign Languages. Sign Languages tent to be productive in terms of derivational morphology, so compounding to reflect functions like negation or locative constructions is common [ 27, 29 ]. Another syntactic feature is the compulsory subject [ 30 ], usually in first position [ 29 ] in contrast to pro-drop being possible and frequent among the world‘s spoken languages. Modelling these features is another interesting and feasible question with current NLP tools. However, the present investigation focuses entirely on topic modelling.

3. Research questions

The proposed study will focus on the following questions: • Which ATS system strategies are preferred by DHH readers? • Do topic-focused simplified sentences benefit readability for DHH people compared to a baseline of lexically simplified sentences? • Do automatic metrics for ATS correlate with the preferences of DHH readers? • Does ATS system design cater for DHH readers at varying levels of reading literacy?

4. Proposed methodology

This section describes some possible techniques and experimental settings which may be efective in answering the primary research questions outlines so far. It discusses some methods of evaluating the experimental results, and how conclusions may be drawn from them. Predictions about the results are also made. Section 5 tackles challenges in conducting this research and other contextual factors it may be necessary to consider.

The key idea in the experiment settings is to compare a baseline of simplified sentences with the same simplified sentences which have been post-processed to 1) Find the topic words in an utterance, 2) Move the topic NPs to the start of the utterance, and 3) Ensure the utterance remains grammatical. The original, ‘complex’ sentences will also be used in comparative evaluation by the participants. Figure 2 illustrates the outline of the experimental design, showing the three diferent output types which will be shown to the participants. It is envisaged that both extant corpora for the input sentences, and an extant ATS tool for generating baseline simplified sentences, will be used.

4.1. Experimental setup

This study will be participant-based, and recruitment could take place through the SignON project and within the associated participating DHH communities, pending ethical and feasibility approval. The target ambient language communities are English and Spanish, with signers coming from the British Sign Language (BSL), Irish Sign Language (ISL), and Lengua de Señas Española9 (LSE) communities. Research participants will be invited to self identify as ‘deaf’, ‘hard-of-hearing’, and other identities covering the great diversity found in the DHH community [ 26 ]. In order to best analyse how reading literacy compares to the evaluation of an ATS system, it may also be necessary to evaluate the reading literacy of the participants. This could either be done through a short assessment (c.f. [ 8 ]), or by additional demographic questions about for example - highest educational level achieved. In terms of avoiding bias in the results of the study, it would be beneficial to recruit at least twelve participants who have a range of reading literacy ability and vary in terms of their self-defined identity.

The research participants will be presented with three types of sentences: • Complex sentences from a corpus • Simplified sentences drawn from the complex sentences from an extant ATS tool • Simplified, topic-focused sentences after pre-processing the simplified sentences These sentences will be taken from a public domain parallel English-Spanish corpus, such as the OPUS10 repository of corpora. Larger corpora are preferred, because there will be a greater number of sentences where sentence transformation with topic-prominence is possible. It is therefore possible to control for domain and parallel content shown to the research participants across language communities. The simplification tool used should be near to state-of-the-art, but could be either lexical (e.g. [ 11 ]), or hybrid (e.g. [ 15 ]) to provide a baseline of simplified sentences.

It is crucial to present participants grammatical and realistic utterances which have undergone topic-fronting. The most efective method to avoid transformation errors and ungrammaticality is to manually identify, extract, and move topic-prominent noun phrases to sentence-initial position. However, manual processing may be unfeasible or very time consuming with a large corpus. One possible way of automating this process could be to identify an utterance‘s topic by means of Latent Dirichlet Allocation [ 28 ]. The noun phrase containing the identified topic may 9Spanish Sign Language 10https://opus.nlpl.eu/ then be extracted through its branches on a dependency parse tree, such as those automatically generated in the spaCy11 library. The resulting noun phrase can then be moved to the start of the sentence. This process automates the one depicted in Figure 1.

The displaying of example sentences to research participants will follow pre-established DHH community visual preferences [ 8 ] and more general accessibility guidelines for ATS [ 23 ]. Experimental instructions at every stage will be in plain language, in a large font, and without any distractions on-screen. The user interface will be interactive as possible to provide autonomy to participants.

4.2. Empirical evaluation

During the experiment, the participants will be invited to rate a given output sentence or text from one of the three output types defined in Figure 2. Their opinion-based scores could be presented on a Likert-type rating scale where 1 =‘This is impossible to understand’ and 5 =‘This is very easy to understand’. Alternatively, pairs of output type sentences could be presented as an ABX test where the participants choose which sentence is more readable out of the pair. This method is thorough and provides a direct comparison between each output type for every sentence, but will ultimately mean subjecting participants to duplicate sentences. It is also important to gather a qualitative analysis from the participants. Participant comments about system design, and their preferences on system process and output have been invaluable in previous studies [ 8, 24, 26 ].

More indirect participant evaluation can be undertaken. As described in [ 9 ], measurements of reading speed and possibly eye gaze can be taken to judge the complexity and readability of a given utterance. Finally, automatic metrics evaluating the two simplified outputs against the complex output will complement the participant-based study - and make the results of the planned study comparable to other works in the field. The SARI metric [ 25 ] is an industrystandard in ATS which compares input and output sentences, as well as a gold-standard reference simplification. SacreBLEU [ 31 ], taken from machine translation research, is another useful metric which allows cross-study evaluation.

4.3. Predictions

In terms of answering the proposed research questions, it is likely from previous studies that DHH readers at all literacy levels will find simplified sentences more readable. As for sentences with topic focus, it is unclear how these sentences will be evaluated by DHH people as they are often a non-canonical construction in written English and Spanish. Automatic metrics are likely to disfavour them, while it is hoped post-processing in this manner will accommodate the argument structure which is familiar to sign language users.

5. Further considerations

This work is in its beginning stages of the design process, and sets out to provide more clarity on text simplification’s role for the DHH community. It plans to objectively evaluate the outputs of ATS systems for DHH people specifically, and compare these with the subjective analyses - an open area of research [ 9 ]. In the absence of on-demand sign language translation in the near future [ 5, 8 ], improving the quality of ATS in what is commonly the L2 of DHH people is crucial for their access to information. It ofers a methodology which tailors the linguistic output of a baseline ATS system based on one syntactic feature typologically common in sign languages. In order to mitigate other syntactic efects, perhaps lexical simplification would be the best system to use to create the simplified outputs for the experiments.

Further experiments could explore incorporating more linguistic features which are prominent in sign languages, or even incorporate pictographic elements in the on-screen output as a step towards on-demand sign transformation. It is hoped that the participant-based elements of this research will take place as soon as possible, and that the work will be complete within the lifetime of the SignON project12.

Acknowledgments

I would like to thank my supervisor, Prof. Horacio Saggion, with his input and guidance in preparing this research proposal.

This work has been conducted within the SignON project. SignON is a Horizon 2020 project, funded under the Horizon 2020 program ICT-57-2020 - “An empowering, inclusive, Next Generation Internet” with Grant Agreement number 101017255.

References

[1] H. Saggion , Automatic text simplification, Synthesis Lectures on Human Language Technologies 10 ( 2017 ) 1 - 137 . doi: 10 .2200/S00700ED1V01Y201602HLT032.

[2] S. Stoll , N. C. Camgöz , S. Hadfield , R. Bowden , Text2sign: Towards sign language production using neural machine translation and generative adversarial networks , Int. J. Comput. Vis . 128 ( 2020 ) 891 - 908 . URL: https://doi.org/10.1007/s11263-019-01281-2.

[3] L. Berke , S. Kafle , M. Huenerfauth , Methods for evaluation of imperfect captioning tools by deaf or hard-of-hearing users at diferent reading literacy levels , in: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI '18 , Association for Computing Machinery, New York, NY, USA, 2018 , p. 1 - 12 . URL: https://doi.org/10.1145/ 3173574.3173665. doi: 10 .1145/3173574.3173665.

[4] L. Wauters , W. van Bon, A. Tellings , Reading comprehension of dutch deaf children, Reading and Writing 19 ( 2006 ) 49 - 76 . doi: 10 .1007/s11145-004-5894-0.

[5] T. Jantunen , R. Rousi , P. Raino , M. Turunen , M. Valipoor , N. García , Is There Any Hope for Developing Automated Translation Technology for Sign Languages? , 2021 , pp. 61 - 73 . doi: 10 .31885/9789515150257.7.

[6] K. Inui , A. Fujita , T. Takahashi , R. Iida , T. Iwakura, Text simplification for reading assistance: A project note , in: Proceedings of the Second International Workshop on Paraphrasing - 12The project runs from January 2021-December 2023 , with a timeline available at https://signon-project.eu/aboutsignon/workplan/ Volume 16, PARAPHRASE '03, Association for Computational Linguistics, USA, 2003 , p. 9 - 16 . URL: https://doi.org/10.3115/1118984.1118986.

[7] P. Kushalnagar , S. Smith , M. Hopper , C. Ryan , M. Rinkevich , R. Kushalnagar , Making cancer health text on the internet easier to read for deaf people who use american sign language , Journal of Cancer Education 33 ( 2016 ). doi:10.1007/s13187-016-1059-5.

[8] O. Alonzo , M. Seita , A. Glasser , M. Huenerfauth , Automatic Text Simplification Tools for Deaf and Hard of Hearing Adults: Benefits of Lexical Simplification and Providing Users with Autonomy, Association for Computing Machinery , New York, NY, USA, 2020 , p. 1 - 13 . URL: https://doi.org/10.1145/3313831.3376563.

[9] O. Alonzo , J. Trussell , B. Dingman , M. Huenerfauth , Comparison of Methods for Evaluating Complexity of Simplified Texts among Deaf and Hard-of-Hearing Adults at Diferent Literacy Levels, Association for Computing Machinery , New York, NY, USA, 2021 . URL: https://doi.org/10.1145/3411764.3445038.

[10] H. Saggion , D. Shterionov , T. Van de Cruys, V. Vandeghinste , J. Blat, SignON: Bridging the gap between sign and spoken languages , in: Proceedings of the 37th Conference of the Spanish Society for Natural Language Processing (SEPLN 2021 ), Málaga, Spain, 2021 , pp. 22 - 24 .

[11] K. C. Sheang , Context-aware automatic text simplification , in: Proceedings of the Doctoral Symposium of the XXXV International Conference of the Spanish Society for Natural Language Processing (SEPLN2019) , CEUR Workshop Proceedings , Bilbao, Spain, 2019 , pp. 56 - 62 .

[12] G. A. Miller , R. Beckwith , C. Fellbaum , D. Gross , K. J. Miller , Introduction to WordNet: An On-line Lexical Database* , International Journal of Lexicography 3 ( 1990 ) 235 - 244 . URL: https://doi.org/10.1093/ijl/3.4.235.

[13] T. Mikolov , K. Chen , G. Corrado, J. Dean , Eficient estimation of word representations in vector space , 2013 . arXiv: 1301 . 3781 .

[14] R. Chandrasekar , C. Doran , B. Srinivas , Motivations and methods for text simplification , in: COLING 1996 Volume 2: The 16th International Conference on Computational Linguistics , 1996 . URL: https://aclanthology.org/C96-2183.

[15] D. Ferrés , M. Marimon , H. Saggion , A web-based text simplification system for english , Procesamiento del Lenguaje Natural 55 ( 2015 ) 191 - 194 .

[16] K. Omelianchuk , V. Raheja , O. Skurzhanskyi , Text simplification by tagging, 2021 . arXiv: 2103 . 05070 .

[17] Z. Zhu , D. Bernhard , I. Gurevych , A monolingual tree-based translation model for sentence simplification , in: Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010 ), Coling 2010 Organizing Committee , Beijing, China, 2010 , pp. 1353 - 1361 . URL: https://aclanthology.org/C10-1152.

[18] X. Zhang , M. Lapata, Sentence simplification with deep reinforcement learning , in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing , Association for Computational Linguistics, Copenhagen, Denmark, 2017 , pp. 584 - 594 . URL: https://aclanthology.org/D17-1062. doi: 10 .18653/v1/ D17 -1062.

[19] W. Xu , C. Callison-Burch , C. Napoles, Problems in current text simplification research: New data can help, Transactions of the Association for Computational Linguistics 3 ( 2015 ) 283 - 297 . URL: https://aclanthology.org/Q15-1021. doi: 10 .1162/tacl_a_ 00139 .

[20] H. Saggion , E. Gómez-Martínez , E. Etayo , A. Anula , L. Bourg, Text simplification in simplext . making text more accessible 47 ( 2011 ).

[21] R. Baeza-Yates , L. Rello , J. Dembowski , Cassa: A context-aware synonym simplification algorithm , 2015 , pp. 1380 - 1385 . doi: 10 .3115/v1/ N15 -1156.

[22] S. Bott , H. Saggion , An unsupervised alignment algorithm for text simplification corpus construction , in: Proceedings of the Workshop on Monolingual Text-To-Text Generation, Association for Computational Linguistics, Portland, Oregon, 2011 , pp. 20 - 26 . URL: https: //aclanthology.org/W11-1603.

[23] L. Moreno , R. Alarcon , I. Segura-Bedmar , P. Martínez , Lexical simplification approach to support the accessibility guidelines , in: Proceedings of the XX International Conference on Human Computer Interaction , Interacción ' 19 , Association for Computing Machinery, New York, NY, USA, 2019 . URL: https://doi.org/10.1145/3335595.3335651.

[24] L. Rello , R. Baeza-Yates , S. Bott , H. Saggion , Simplify or Help? Text Simplification Strategies for People with Dyslexia, Association for Computing Machinery , New York, NY, USA, 2013 . URL: https://doi.org/10.1145/2461121.2461126.

[25] W. Xu , C. Napoles , E. Pavlick , Q. Chen , C. Callison-Burch , Optimizing statistical machine translation for text simplification , Transactions of the Association for Computational Linguistics 4 ( 2016 ) 401 - 415 . URL: https://aclanthology.org/Q16-1029.

[26] D. Bragg , O. Koller , M. Bellard , L. Berke , P. Boudreault , A. Brafort , N. Caselli , M. Huenerfauth , H. Kacorri , T. Verhoef , C. Vogler , M. Ringel Morris, Sign language recognition, generation, and translation: An interdisciplinary perspective , in: The 21st International ACM SIGACCESS Conference on Computers and Accessibility , ASSETS '19, Association for Computing Machinery, New York, NY, USA, 2019 , p. 16 - 31 . URL: https://doi.org/10.1145/3308561.3353774.

[27] J. Porta , F. López-Colino , J. Tejedor , J. Colás , A rule-based translation from written spanish to spanish sign language glosses , Computer Speech & Language 28 ( 2014 ) 788 - 811 . doi: 10 .1016/j.csl. 2013 . 10 .003.

[28] D. M. Blei , Probabilistic topic models , Commun. ACM 55 ( 2012 ) 77 - 84 . URL: https: //doi.org/10.1145/2133806.2133826. doi: 10 .1145/2133806.2133826.

[29] M. A. Rodríguez González , Lenguaje de signos, 2003 . URL: http://www.cervantesvirtual. com/nd/ark:/59851/bmc08633.

[30] R. San-Segundo, R. Barra , R. Córdoba , L. D'Haro , F. Fernández , J. Ferreiros , J. Lucas , J. Macías-Guarasa , J. Montero , J. Pardo , Speech to sign language translation system for spanish , Speech Communication 50 ( 2008 ) 1009 - 1020 . URL: https: //www.sciencedirect.com/science/article/pii/S0167639308000149. doi:https://doi.org/ 10.1016/j.specom. 2008 . 02 .001, iberian Languages.

[31] M. Post , A call for clarity in reporting BLEU scores , in: Proceedings of the Third Conference on Machine Translation: Research Papers , Association for Computational Linguistics, Belgium, Brussels, 2018 , pp. 186 - 191 . URL: https://www.aclweb.org/anthology/W18-6319.