English. Mathematical expressions are complex hierarchical structures of symbols that are usually accessed by visual inspection. These expressions are seldom rendered with natural language since users are not usually required to read them aloud. People with a visual impairment generally use LATEX with screen readers to acquire mathematical expressions. However, LATEX can be verbose, slow to listen, and dificult to learn. This work proposes a way to make mathematical expressions easier to be accessed by people with disabilities by exploiting their hierarchical structures. We describe and evaluate a dialogue system to vocally navigate mathematical expressions in English. In contrast with standard screen readers, the vocal interaction allows people to query the system about sub-parts of the expressions. Italiano. Le espressioni matematiche sono complesse strutture gerarchiche di simboli generalmente esplorate visivamente. Queste espressioni raramente vengono rappresentate tramite linguaggio naturale, perché agli utenti di solito non è richiesto di leggerle ad alta voce. Le persone con disabilità visiva di solito usano il LATEX con uno screen reader per ascoltare le espressioni matematiche. Tuttavia, il LATEX può essere verboso, lento da ascoltare e dificile da imparare. Questo lavoro propone una alternativa per rendere più accessibili le espressioni matematiche alle persone con disabilità sfruttando la loro struttura gerarchica. Descriviamo ed valutiamo quindi, un sistema di dialogo per navigare vocalmente le espressioni matematiche in inglese. A diferenza dei normali screen reader, l'interazione vocale consente alle persone anche di interrogare il sistema sugli elementi che compongono le espressioni.
architecture, i.e. a sentence planner and a realizer. 2.2. Speech-to-text solutions for entering
The interaction design of the SDS is quite straightfor- maths expressions ward: the SDS pronounces the mathematical sentences and the user can interrupt it to “navigate” the expres- This section introduces the solutions which have been sion (e.g. ask for repetition of its parts). So, the dialogue investigated to write mathematics through speech inmanager component coordinates the recognition of a rep- put. TalkMaths [18] [19] is a prototype application etition command with the generation of (a subpart of) which translates a limited set of arithmetic, algebraic the mathematical sentence. and trigonometric expressions from spoken English into
We performed a user-based evaluation on the efec- LaTeX or MathML. It adopts Dragon Naturally Speaktiveness and the usability of the SDS. The system imple- ing2 (DNS) as speech recognition system. The translation mentation and evaluation were conducted with the in- rules are defined only for English and the recognition volvement of visually impaired experts. The results show implements a dictation model based on pauses, which that the developed system has a good impact both on the slow down the dictation process [20]. comprehension of mathematical expressions and on the Mathifier [ 21] is an open source software module user experience, constituting a promising approach for which converts a subset of mathematical expressions helping people with visual impairments. from English into LaTeX. It combines a dictionary, a lan
The paper is structured as follows: In Section 2, we re- guage model and an acoustic model to recognize matheport related work. In Section 3, we describe the main com- matical English utterances. It is based on Sphinx-4 [22] to ponents of the SDS. In Section 4, we describe a human- recognize speech. This project has not been maintained based evaluation of the SDS and in Section 5, we closes and updated regularly. the paper with some consideration and future work. Fi- CamMath [20] is a proof of concept prototype applinally, Section 6 discusses some limitations of our SDS. cation designed to prove the advantages of continuous speech over discrete utterance of mathematical expressions in English. 2. Related Work Metroplex MathTalk3 is a commercial application that provides speech input of arithmetic, algebra, calculus and
visual impariments EquatIO4 enables dictation of simple maths expressions in English in MS Word and in GSuite applications.
Even though these applications have been designed to enable speech recognition of mathematical expressions, none of them has addressed the needs of people with visual impairments by combining speech input and speech output.
Many diferent solutions have been investigated to enable
people with visual impariments to access mathematical
expressions. They can be divided up into two main
categories: systems to read mathematics and systems to
type and simplify maths expressions. The former
category includes applications to read LaTeX in PDF files
[
The latter category includes specialized applications that are designed to facilitate students with visual impairments to simplify expressions in LaTeX format [14] or in a multimodal work environment specifically designed for inclusive classes [15, 16] and to work with bi-dimensional mathematical procedures as arithmetic operations [17].
In this section we describe the main components of the SDS. As most rule-based SDSs [23], the information flow follows a path initiated by the user, who starts the interaction with a request to read a specific mathematical expression (Fig. 1). The SDS pronounces the mathematical sentence. The user listens to the produced sentence, and possibly interrupts the SDS asking for clarification. At this point the SDS answers to the request and the dialogue goes on. In Fig. 1 we report the architecture of the SDS, which is based on three main components, that are the response generator (described in Section 3.1), the language understanding and the dialogue management (described both in Section 3.2).
is unambiguous: for each operator (empty tag–e.g.
3.1. Response Generator and <eq/>), there is an opening and closing tag (i.e.
<apText-to-Speech ply>...</apply>). Within these tags, we can find nested
parts of the mathematical sentence such as other
operThe generation side of the SDS follows the traditional ators and the variables that compose the arguments of
NLG architecture composed of document planning, sen- each operator.
tence planning and realizer [24]. In particular, for the For implementing the sentence planning, we model the
mathematical sentence generation, we follow the pipeline syntactic structure of the English mathematical
expresdescribed in Mazzei et al. [
The document planning consists of an encoding of the operators in Table 1 (e.g. +) are generally modeled with mathematical expression from LATEX into a semantically declarative sentence (e.g. ), while elementary unambiguous format, i.e. CMML, through the LatexML functions (e.g. ) are modeled with a noun phrase (e.g. tool [25], specifying some heuristics defined by the spe- ). cific mathematical sub-domain (e.g. algebra).
Let us consider the following formula as an example: Symbol Operator English Form “ × = {(, ) | ∈ , ∈ }” and its LATEX representation: “A \times B = { (x, y) \mid x \in A, y \in B}”. The + mathematical formula will be converted in the following − CMML format: / <apply> <eq/> <apply> <cartesianproduct/> <ci>A</ci> <ci>B</ci> </apply> <apply> <conditional-set/> <apply> <pair/> <ci>x</ci> <ci>y</ci> </apply> <apply> <and/> <apply> <in/> <ci>x</ci> <ci>A</ci>
ways to compose a sentence. Taking as an example the minus operator, it can appear as: (1) 5, (2) (a noun phrase and a declarative phrase, respectively).
The diference between these two sentences depends on how the operator is used. In (1) is used to define a negative value, in (2) is used as the diference between two variables. From a realisation point, we need to distinguish two diferent forms within the category of algebraic, arithmetic and set operators: (1) unary form (e.g. 5): a so-called adjective phrase must be defined the system to repeat a specific part of the mathematical where the operator (e.g. ) works as an adjective; sentence (e.g. “What is the limit of the second integer?”). (2) binary form (e.g. ): a declarative structure In this case, the specific operator in the user’s request must be defined where the operator (e.g. ) works will be searched. If matched, the system produces an as the parent node of the expression. answer with the requested part of the operator. Finally,
The realisation phase uses SimpleNLG [26], which is a the resuming intent lets the user resume the interaction Java library for morphological realization and lineariza- after an interruption (e.g. “Go on”). tion in English. Note that we added special symbols The main algorithm of this module is represented into SimpleNLG lexicon to produce both parentheses below: and pauses. Finally, we control the pronunciation, volume and pitch by using the Speech Synthesis Markup dialogue(): Language (SSML)5. As the last step, the mathematical 1. Say "I'm starting to say the sentence" sentence is pronounced through a Text-to-Speech tech- 2. Activate the Wake-up Word Detection nology. system
We experiment with two diferent vocal synthesizers: 3. For each word within the sentence: the commercial AWS Polly6 and the open source eSpeak7 4. Say the word (both support SSML). AWS Polly is based on an advanced 5. If the Wake-up Word Detection deep learning technology and has a human-kind voice, system has detected "Hey stop": which is clear and highly user-adjustable, whereas eSpeak 6. Say "Ok, I'm listening to you" is based on a so-called formant synthesis method and 7. Activate the Request produces a robotic voice. It is worth noting that eSpeak Recognition system is a very familiar voice for visually impaired people. The 98.. Say "I'vFeulffiinlilshtehderreeaqduiensgtthis users can choose which vocalizer to use. sentence, but I'm still here for you" 10. Activate the Request Recognition system
start and the end of the expression. After the activation of the wake-up word detection system, for each word the The speech recognition module in Fig. 1 is composed of system will pronounce it. If the user says the wake-up two diferent sub modules: Wake-up word detection and word, the system will stop and put itself into a listening Speech-to-Intent. A wake-up word [27] is a keyword that phase waiting for a request. Completed the expression, triggers the speech-to-intent module. The classification the System can continue to answer user’s requests. is binary and happens in real time. For this task we The “fulfill the request” method in line 8 works as follows: used Porcupine (v.2.1) [28] which has good results in comparison to other commercial systems. A Speech-toIntent system is able to recognize a user’s intent in a very specific context. The system works on a small vocabulary of terms and classifies each user’s request. For this task we used Rhino (v.2.1) [29] which allowed us to detect in real time the user’s vocal commands for our mathematical 4. context.
The intent is translated into a request which is in- 5. terpreted by the language understanding module with 6. regular expressions, which matches the intent produced 7. by the speech recognition with a domain specific speech act. These speech acts are: (1) repetition, (2) query and (3) resuming. The repetition intent lets the user ask the repetition (from a subpart) of the mathematical sentence (e.g. “Repeat from the first integer”). In this case the sys- 10. tem searches within the content response what the user 11. wants to be repeated. The query intent lets the user ask 12. lim Easy expressions - Few parenthesises and nodes - Experiment 10 10 13 14 15 Nodes 10 20 21 28 31
If the intent has been interpreted as repetition, the system will search within the content response what the user wants to be repeated. If the user’s intent has been interpreted as query, the specific operator in the user’s request will be searched. If matched, the system will answer the requested part of the operator. The dialogue management module is in charge of making decisions.
If the interpreted request does not correspond to any of the supported possibilities, it asks the user to repeat the request. Otherwise, the dialogue management searches the content of the response for the user and sends it to the response generator module.
To evaluate the SDS, we conducted two experiments involving visually impaired users with the approval of the
tion we tested the efectiveness and solidity of the English
generation system in a similar way as Mazzei et al. [
Exact Match (EM) and SPICE [31]. EM is 1 if the original CMML and the one obtained by the user are the same, and 0 otherwise. SPICE is obtained by calculating the F-score of the overlapping between the original CMML tree and the one obtained from the user. The overlapping is measured by decomposing the CMML trees in typed elementary substructures, which are operands, operators and their relations. Experiment 1 results (Table 4) show that the generation system seems to be efective since the
SXQaTJH2tAnHKg9PHSysurOSzf3xbSvtKG7Ig/formResponse? users obtained a good understanding of the expressions. pated freely and without compensation. In Experiment 2 In Experiment 2 we recruited five blind native Italian speakers that declared a good maths knowledge and proifciency in English; however, one of the users dropped out because of their low competency. The users particithe users connected via Google Meet to a client running the SDS. This modality has been decided on the basis of COVID-19 restrictions still in force at that time. A Dificult expressions - More parenthesises and nodes - Experi
User
Metrics
Tot. (25)
Easy (10)
Dificult (15) 1 2 avg
EM SPICE EM SPICE EM SPICE 0.92 0.98 1.00 1.00 0.96 0.99 facilitator established the Google Meet connection, presented the experimental protocol (see the instructions presented to the users in Fig. 2), and observed the user interactions while remaining neutral. After a short time (about 30 minutes) when they could practice with the SDS, the users were presented with 3 easy and 3 dificult expressions chosen among the 10 expressions of Experiment 1 (Table 5 and Table 6). The SDS used Polly for the speech synthesis. The users interacted autonomously with the SDS and they could interrupt the system and ask questions. Finally, the users were asked to write down the expressions questionnaire similarly to Experiment 1. ∫︁
1
Nodes 20 21 31 Dificult expressions - More parenthesises and nodes - Experi
ation measures. Moreover, the users also compiled the User Experience Questionnaire (UEQ) [32] to evaluate their experience in terms of attractiveness, perspicuity, eficiency, dependability, stimulation and novelty 9 .
of Experiment 1 both on EM and on SPICE (cf. Table 7). This could be explained by the complexity of the setting, because users had to learn how to use a new tool in a short time, whereas the users in Experiment 1 were familiar with the linear fruition of a Youtube audio. We observed that the performance of the users improved over the time of the experiment, which can be due to an acquired familiarity of the tool.
ferent attributes. For each attribute we report the score of the attribute on a scale between − 3 and +3 and compare it with a benchmark provided by UEQ [32] that includes a dataset with 468 products evaluated by 21, 175 users. It is worth noting that the SDS scored high over stimulation, novelty and attractiveness and fair over perspicuity, eficiency and dependability. The scores over eficiency User
Metrics
Tot. (6)
Easy (3)
Dificult (3) 1 2 3 4 avg and novelty are consistent with the hypothesis that users would benefit from a longer training time to become proifcient with this new tool, that they however deem stimulating and attractive. These preliminary experiments seem to be promising, nevertheless it would be beneficial to enlarge the pool of users. However, it is known in accessibility studies [33] that involving visually impaired people in experiments is significantly hard and several studies tend to engage only sighted people.
In this paper we described a SDS designed for allowing
visually impaired people to access mathematical
expressions. In Experiment 1 we focused on the understanding
of the mathematical sentences generator for English (i.e.
using EM and SPICE measures), replicating the good
results obtained for Italian in Mazzei et al. [
The SDS developed in this paper has two main limitations.
The design interaction is limited to repetition request concerning a subpart of the expression. A better interaction could consider the possibility to ask for mathematical clarification on the role of a subpart (e.g. “what is x?”).
The evaluation has two aspects that could be improved: 1. the limited number of testers, and 2. no native English speakers participated. of mathematical expressions for blind students: a 2004. comparative study between mathml and latex, in: [23] K. Jokinen, M. McTear, Spoken Dialogue Systems, 2019 IEEE International Conference on Engineering, Synthesis lectures on human language technologies, Technology and Education (TALE), IEEE, 2019, pp. Morgan & Claypool Publishers, 2010. URL: https: 1–5. //books.google.it/books?id=ualwulnD020C. [12] A. Bansal, M. Balakrishnan, V. Sorge, Com- [24] E. Reiter, R. Dale, Building Natural Lanprehensive accessibility of equations by visually guage Generation Systems, Natural Lanimpaired, ACM SIGACCESS Access. Comput. guage Processing, Cambridge University 126 (2020) 1. URL: https://doi.org/10.1145/3386280. Press, 2000. URL: http://prp.contentdirections. 3386281. doi:10.1145/3386280.3386281. com/mr/cupress.jsp/doi=10.2277/052102451X. [13] J. Seo, S. McCurry, A. Team, Latex is not easy: Cre- doi:DOI:10.2277/052102451X. ating accessible scientific documents with r mark- [25] B. Miller, Latexml: A LaTeX to XML converter, down, Journal on Technology and Persons with https://math.nist.gov/~BMiller/LaTeXML, 2007.
Disabilities 7 (2019) 157–171. [26] A. Gatt, E. Reiter, SimpleNLG: A realisation en[14] S. Arooj, S. Zulfiqar, M. Qasim Hunain, S. Shahid, gine for practical applications, in: Proceedings of A. Karim, Web-alap: A web-based latex editor for the 12th European Workshop on Natural Language blind individuals, in: The 22nd International ACM Generation (ENLG 2009), 2009, pp. 90–93. SIGACCESS Conference on Computers and Acces- [27] Y. Wang, Wake word detection and its applications, sibility, volume 28, 2020, pp. 1–6. Johns Hopkins University, 2021. [15] V. Sorge, Supporting visual impaired learners in [28] Picovoice, Benchmarking a Wake Word editing mathematics, in: Proceedings of the 18th In- Detection Engine, https://picovoice.ai/blog/ ternational ACM SIGACCESS Conference on Com- benchmarking-a-wake-word-detection-engine/, puters and Accessibility, 2016, pp. 323–324. 2018. [16] C. Bernareggi, Non-sequential mathematical nota- [29] Picovoice, Picovoice Console — Rhino Speechtions in the lambda system, in: ICCHP, Springer, to-Intent Engine, https://picovoice.ai/docs/ 2010, pp. 389–395. quick-start/console-rhino/, 2018. [17] A. Gerino, N. Alabastro, C. Bernareggi, D. Ahme- [30] L. Pandolfi, ANALISI MATEMATICA 1, Dipartitovic, S. Mascetti, Mathmelodies: inclusive design mento di Scienze Matematiche “Giuseppe Luigi Laof a didactic game to practice mathematics, in: grange”, Politecnico di Torino, 2013. International Conference on Computers for Handi- [31] P. Anderson, B. Fernando, M. Johnson, S. Gould, capped Persons, Springer, 2014, pp. 564–571. SPICE: semantic propositional image caption eval[18] A. Wigmore, G. Hunter, E. Pflügel, J. Denholm- uation, CoRR, abs/1607.08822, 2016.
Price, V. Binelli, Using automatic speech recogni- [32] M. Schrepp, User experience questionnaire handtion to dictate mathematical expressions: The devel- book. all you need to know to apply the ueq successopment of the “talkmaths” application at kingston fully in your project, https://www.ueq-online.org/, university., Journal of Computers in Mathematics 2015.
and Science Teaching 28 (2009) 177–189. [33] E. Brulé, B. J. Tomlinson, O. Metatla, C. Joufrais, [19] A. M. Wigmore, E. Pflugel, G. J. Hunter, J. Denholm- M. Serrano, Review of quantitative empirical evaluPrice, M. Colbert, Talkmaths better! evaluating ations of technology for people with visual impairand improving an intelligent interface for creat- ments, in: Proceedings of the 2020 CHI Conference ing and editing mathematical text, in: 6th Inter- on Human Factors in Computing Systems, 2020, pp. national Conference on Intelligent Environments, 1–14.
IEEE, 2010, pp. 307–310. [20] C. Elliott, J. Bilmes, Computer based mathematics using continuous speech recognition, Vocal Interaction in Assistive Technologies, Games and More (2007). [21] S. N. Batlouni, H. S. Karaki, F. A. Zaraket, F. N.
Karameh, Mathifier—speech recognition of math equations, in: 18th International Conference on Electronics, Circuits, and Systems, IEEE, 2011, pp.
301–304. [22] W. Walker, P. Lamere, P. Kwok, B. Raj, R. Singh,
E. Gouvea, P. Wolf, J. Woelfel, Sphinx-4: A flexible open source framework for speech recognition,