English. In this paper we present the main features of a rule-based architecture to transform a LATEX encoded mathematical expression into its equivalent mathematical sentence form, i.e. a natural language sentence expressing the semantics of the mathematical expression. Moreover, we describe the main results of a first human based evaluation of the system for Italian language focusing on speech synthesis engines.
Computational linguistics can help people in many ways, especially in the field of assistive technologies. In the case of mathematical domain, blind people can access to a mathematical expression by listening its LATEX source. However, this process has several drawbacks. First of all, it assumes the knowledge of the LATEX. Second, listening LATEX is slow and error-prone, since LATEX is a typographical language, that is a language designed
Copyright c 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). for specifying the details of typographical visualization rather than for efficiently communicate the semantics of a mathematical expression. For instance, the simple LATEX expression f (x) is a typographical description and so it represents both the function application of f to x, and the multiplication of the variable f for the variable x surrounded by parenthesis.
There are many lines of research to enable
people with sight impairments to access
mathematical contents. It is possible to embed
mathematical expressions in web pages not only as
images but through MathML or MathJax (Cervone,
2012) and in PDF documents produced from
LaTeX
In this paper we follow another direction: we
consider the possibility to produce a
mathematical sentence, i.e. a natural language sentence
expressing the semantics of a mathematical
expression. Indeed, the idea to use mathematical
sentences for improving the accessibility of
mathematical expressions has been previously
presented and experimented for Spanish in
We have two research goals in this paper. The
first goal is to describe a system for
transforming a mathematical expression natively encoded
in LATEX in its equivalent mathematical sentence
(cf. Figure 1). The processing flow follows a
wellknown approach, called interlingua in the field of
machine translation
w sentemncaethr.itten
audio math. sentence
LATEX mathematical expression
L
M CM
Indeed, the process of generating a mathematical expression from its LATEX source is a two-step algorithm. In the first step the LATEX is analyzed and its semantics is represented in Content MathML (CMML henceforth), a W3C standard for expressing the syntax and the semantics of mathematical expressions1. In the second step, the CMML representation is used as input of the S2S (Semantics to Speech) module, that is a NLG module generating the mathematical sentence. Note that the S2S module inserts in the sentence parenthesis and pauses too. The sentence will finally be transformed in audio format encoding by an external synthesis engine.
The second goal of this paper is to give a first evaluation of the performance of two distinct synthesis engines in the domain of mathematical sentence. With a pilot experimentation conducted with four blind people, we will compare the perception of the mathematical sentences of a neuralnetwork based speech engine and of a formantbased speech engine.
In Section 2 we will describe the main features of the developed system, in Section 3 we will describe the experimentation and finally in Section 4 we end the paper with some conclusions and introducing future work. 2
The first step of our algorithm is the generation
of CMML associated to a LATEX formula. We
based this step on an external tool named
LatexML
Mathematical notation has been conceived with
the aim of representing mathematical concepts
using a specific written symbolic language. As
working hypothesis, we decided to assume a
“specialized” syntactic analysis for a number of
mathematical objects. For instance, x plus three
indicates the action of adding one quantity to
another, so it can be represented as a declarative
structure. As a consequence, plus can be
analysed as verb and this assumption can be extended
to all the mathematical sentences. In this paper we
considered only the mathematical structures
belonging to the subfield of the mathematical
analysis. In particular, we considered all the
expressions in an Italian analysis book
In Table 1, we reported some examples of syntactic constructions for mathematical expressions. We decided to analyse and represent the mathematical sentences of relational operators as copula sentences (a e` maggiore di b, a is greater than b), algebraic operators as declarative sentences (a prodotto cartesiano b, a cartesian product b), logical operators as conjunctions (a o b, a or b), elementary operators (e.g. radice, radical), sequence (e.g. limite, limit), calculus (e.g. integrale, inte<apply> <implies/> <apply> <gt/> <ci>x</ci> <ci>b</ci> </apply> <apply> <lt/> <apply> <abs/> <apply> <ci>f</ci> <ci>x</ci> </apply> </apply> <ci>M</ci> </apply> </apply> gral) as noun phrases (La radice quadrata di x, the square root of x), pairs and conditional sets as reduced relatives (L’insieme delle x tali che x e` minore di 3, the set of x such that x is less than 3). Our syntactic representations for mathematical operators in the analysis domain could have alternative representations or could be specialized in a more refined classification (c.f. (Chang, 1983)), but we decided to use only eight category for sake of simplicity.
Traditional NLG architectures split the
generation process into three distinct phases, that are
document planning, sentence planning and realization
The input of the sentence planner is a mathematical expression in the form of enhanced CMML. In order to associate a sentence plan, that is a a sort of under-specified tree-based syntactic structure, we devised a recursive algorithm that traverses topdown the CMML structure.
By considering the eight categories used to classify all mathematical expressions, for each category we designed a prototypical sentence plan that will be used in the recursive process. Each prototype builds a specific linguistic construction (e.g. copula, reduced relative etc.), that is designed for giving syntactic roles to the arguments of the specific mathematical construction. For instance, on the left of the Figure 3, we reported the prototypical sentence plan for the conditional set mathematical structure and on the right of we reported an example of its instantiation. In the final produced structures we have that, (1) the leaves of the sentence plan are lemmas rather than words, (2) the syntactic relations among the nodes are expressed using both dependency relations (e.g. subj, complement) as well as constituency nodes (e.g. Prepositional Phrase, PP). Note that this is the input format for sentence plan required by the SimpleNLG realizer (see Section 2.2).
In order to build a complete sentence plan for a mathematical sentence by using the eight categories for mathematical expressions, there are two important issues.
The first issue concerns the perception of precedence of the arithmetic operator. Listening mathematics has some peculiarities with respect to reading it. For instance, division is granted a higher precedence than addition, and during the reading process the expression a + b=c is parsed as a + cb without ambiguities. A different result arises if one listens the equivalent mathematical sentence a plus b divided by c without reading the expression: we experimented that the most frequent perceived parse is a +cb . After a limited numinsieme op1
prep di il det compl compl op2
prep tali che
NP
complement il insieme PP complement
PP di NP tale che Clause il x
subj NP V
obj AdjP
complement
PP x essere minore di NP 0 ber of experiments in listening arithmetic expressions with distinct (blind and not blind) people, we decided to state as working hypothesis that the precedence of the arithmetic operators are perceived in the reverse order when one listens a mathematical expressions without reading it2.
A second issue is how to represent the correct structures of the operators. In other words, how we can build a mathematical sentence unambiguously equivalent to a +cb ? A trivial but effective solution is to use parenthesis, that is to produce the mathematical sentence open parenthesis a plus b close parenthesis divided by c. However, the drawback of this solution is the length of the sentence that, for very complex expressions, can augment substantially.
In order to account for both the issues, we
modified the sentence planner in two ways.
First, we decided to model parenthesis as
lexical items, that is we considered open-parenthesis
and closed-parenthesis as two new lexical items
of the SimpleNLG lexicon which can be used
as pre-modifier and post-modifier of a
mathematical sentence respectively. Second, similar
to
2We have not been able to find any scientific reference on this point.
Note that a parenthesis has to be considered necessary with respect to the inverted precedence order hypothesis stated above. In the pause strategy, all the necessary pauses are inserted in the sentence plan. In the smart strategy, all the necessary parentheses are inserted in the higher nodes of the sentence plan, and the necessary pauses are inserted close to the leaves of the sentence plan. This is a hybrid strategy that combines parentheses and pauses in order to have a less verbose mathematical sentence. 2.2
In order to produce a spoken mathematical
sentences in Italian with the SimpleNLG-it realizer
The actual version of the mathematical sentence generator has been interfaced with two speech synthesis engines, that are the web service provided by the IBM-Watson framework4 (W-engine henceforth), and the Espeak API5 (Eengine henceforth). W-engine is a commercial, closed software based on deep learning, while Eengine is a free, open-source software based on formant synthesis algorithms. Note that for not visual impaired people W-engine sounds more fluent but, in contrast, for visual impaired people Eengine sounds more familiar since it is used by a widespread free screen reader. 3
In order to have a first evaluation of the generation system, we built a web-based test explicitly designed for visually impaired people. We designed a questionnaire composed by a 6 multiple choices questions concerning personal data, a core of 25 open questions each one concerning the listening of a mathematical sentence and its comprehensibility, 1 Likert-scale question globally comparing LATEX and system comprehensibility, 1 open question for free comments.
The 25 core questions have a all the same schema: there is a audio file encoding a mathematical sentence and there is a open form for transcribing it. In the compilation instructions, we asked the users to fill this section by using “LATEX or with other non ambiguous formal representation”. The mathematical expressions obtained have been manually translated to CMML for evaluation. We implemented the questionnaire by using the Google Form framework, that was 3https://www.w3.org/TR/ speech-synthesis11/
4https://www.ibm.com/watson/services/ text-to-speech/ 5http://espeak.sourceforge.net ID E1 E4 E6 E8 E10 lim x!x0
Z
B = f(x; y) j x 2 A; y 2 Bg x > b =) jf (x)j < M f (x) f (x0)
f 0(x0) x x0 1 pm2 lim x2 1 + dx = arcsin 1 n
= e n x m preliminarily judged accessible by a blind person.
In this paper we discuss the results of 10 core questions of the questionnaire that have been created by using the 5 mathematical expressions belonging to the Table 2. We use the W-engine to build 5 mathematical sentences and the E-engine to build other 5 mathematical sentences. Note that we change the names of the variables in the two set of sentences.
In order to score the comprehension of the
user we decided to use the SPICE
In Table 3 we reported the averaged values of SPICE for W-engine and E-engine. A first view of data seems suggest a preference for the E-engine, but there is not a significant effect on the performance of the system: by applying the t-test we obtained for 0:08 (two-tailed p-value), indicating no statistical significance. So, new experiments with more trials and users are necessary to statistically confirms the preference of for the E-engine.
In Table 4, we report the The distribution of the answers in Likert scale for the question of the web form concerning comprehensibility, that is “Quanto sei d’accordo con la frase: - La frase pronunciata e` facile da capire -” (How much do 0.96 (0.06) 0.99 (0.03) 0.95 (0.12) 0.99 (0.03) 0.97 (0.06) 0.97 (0.04) 0.97 (0.06) 0.97 (0.04) you agree with the sentence: - The pronounced sentence is easy to understand -”). The value 1 corresponds to “per nulla” (nothing), the value 7 corresponds to “completamente” (completely). It seems from data that there is not notable difference between the perceived comprehensibility of the W-engine with respect to the E-engine and the t-test we obtained for the Likert score is 0:67 (twotailed p-value). 4
In this paper we have presented a study on the generation of mathematical sentences, i.e. natural language sentences encoding mathematical expressions6. In particular, we have described the main features of the system and the a first experimentation centred on the evaluation of two distinct speech engine. The results of the experimentation suggests a good performance of the formantbased synthesis engine with respect to the neuralnetwork base synthesis engine. However, more data is necessary to achieve statistical significance.
In future work we intend to repeat the evaluation of the system for Italian with a larger number of users and to repeat the experiment by using English lanaguage too. 6The described system can be freely downloaded at https://bitbucket. org/tesimagistralemonticone/ formula-to-speech/ Computers and Accessibility, ASSETS ’18, pages 352–354, New York, NY, USA. ACM. [Anderson et al.2016] Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016. SPICE: semantic propositional image caption evaluation. CoRR, abs/1607.08822. [Cervone2012] Davide Cervone. 2012. Mathjax: A platform for mathematics on the web. Notices of the American Mathematical Society, 59, 02. [Chang1983] Lawrence A. Chang. 1983. Handbook for spoken mathematics (larry’s speakeasy). Lawrence Livermore Laboratory, The Regents of the University of California., 1.