-
Mathematics is capable of explaining complex concepts and relations in a
compact, precise, and accurate way. The general applicability of mathematics
allows a certain level of ambiguity in its expressions. This ambiguity is regularly
mitigated by short explanations following or preceding these mathematical
expressions, that serve as context to the reader. Along with context dependency,
inherent issues of linguistics (e.g. non-formality) make it even more
challenging for computers to understand mathematical expressions. Said that, a system
capable of capturing the semantics of mathematical expressions automatically
would be suitable for several applications, from improving search engines to
recommender systems. Consider for example the the lower bound for Van der
Waerden numbers
W (2; k) > 2k=k":
(1)
Learning connections, such as between W and the entity ‘Van der Waerden’s
number ’ from above, requires a large specically labeled scientic database that
contains mathematical objects. Word embedding techniques has received
significant attention over the last years in the Natural Language Processing (NLP)
community, especially after the publication of word2vec [
13
]. Recently, more and
more projects try to adapt this knowledge for solving Mathematical Information
Retrieval (MIR) tasks [
3, 9, 26, 24
]. In this paper, we explore some of the main
aspects that we believe are necessary to leverage the learning of mathematics by
computer systems. We explain, with our evaluations of word embedding
techniques on the arXMLiv 2018 [
4
] dataset, why current ML approaches are not
applicable for MIR tasks, yet.
Current MIR approaches [
8, 22, 20
] try to extract textual descriptors of the
parts that compose mathematical equations. This leads to two main issues: (i)
how to determine the parts which have their own descriptors, and (ii) how to
identify correct descriptors over others. Answers to (i) are more concerned in
choosing the correct denitions for which parts of a mathematical expression
should be considered as one mathematical object [
7, 25, 21
]. Current denitions,
such as the content MathML 3.0 4 specication, are often imprecise. Consider
i, where is a vector and i its i-th element. In this case, i should be
considered as a composition of three content identiers, each one carrying its own
individualized semantic information, namely the vector , the element i of the
vector, and the index i. However, with the current specication, the denition
of these identiers would not be canonical. Because of these problems in current
standards, nowadays work focusing their eorts on (ii).
Schubotz et al. [
22
] presented an approach for scoring pairs of identiers and
descriptors by the number of words between them. They made the assumption
that correct deniens appear close to the identier and to the complex
mathematical expression that contains this same identier. Kristianto et al. [
8
] introduce
a ML approach, in which they train a Support Vector Machine (SVM) to
consider sentence patterns and other characteristics as features (e.g. part-of-speech
(POS) tags, parse trees). Later, [
20
] combine the aforementioned approaches and
use pattern recognition based on the POS tags of common identier-deniens
pairs, the distance measurements, and SVM, reporting results for precision and
recall of 48.60% and 28.06%, respectively. More recently, some projects try to use
embedding techniques to learn patterns of the correlations between context and
mathematics. In [
3
], they embed single mathematical symbols, while Krstovski
and Blei [
9
] represent complex mathematical expressions as single unit tokens for
IR. Recently, M. Yasunaga et al. [
24
] explore an embedding technique based on
recurrent neural networks to improve topic models by considering mathematical
expressions.
2
Machine Learning on Embeddings
The word2vec [
13
] technique computes real-valued vectors for words in a
document using two main approaches: skip-gram and continuous bag-of-words (CBOW).
Both produce a xed length n-dimensional vector representation for each word
in a corpus. In the skip-gram training model, one tries to predict the context of a
4 https://www.w3.org/TR/MathML3/
given word, while CBOW predicts a target word given its context. In word2vec,
context is dened as the adjacent neighboring words in a dened range, called a
sliding window. The main idea is that the numerical vectors representing similar
words should have close values if the words have similar context.
The lack of solid references and applications that provide the semantic
structure of natural language for mathematical identiers make their disambiguation
process challenging. In natural texts, one can try to infer the most suitable word
sense for a word based on the lemma 5 itself, the adjacent words, dictionaries,
thesaurus and so on. However, in the mathematical arena, the scarcity of
resources and the exibility of redening their identiers take this issue to a more
delicate scenario. The text preceding or following the mathematical equation is
essential for its understanding.
More recently, [
9
] propose a variation of word embeddings for mathematical
expressions. Their main idea relies on the construction of a distributed
representation of equations, considering the word context vector of an observed word
and its word-equation context window. They treat equations as single-unit words
(EqEmb), which eventually appears in the context of dierent words. They also
try to explore the eects of considering the elements of mathematical
expressions separately (EqEmb-U). While they present some interesting ndings for
retrieving entire equations, little is said about the vectors representing equation
units and how they are described in their model.
Nowadays mathematics in science is mostly either given in LATEX or MathML.
The former is used by humans for writing scientic documents. The latter, on
the other hand, is popular in web representations of mathematics due to its
machine readability and XML structure. There has been a major eort to
automatically convert LATEX expressions to MathML [
21
] ones. However, neither
LATEX nor MathML are practical formats for embeddings. Considering the
equation embedding techniques in [
9
], we devise three main types of mathematical
embeddings.
Mathematical Expressions as Single Tokens: EqEmb [
9
] uses entire
mathematical expressions as one token. In this type, the inner structure of the
mathematical expression is not taken into account. For example, Equation (1) is
represented as one single token t1. Any other expression, such as W (2; k), in
the surrounding text of (1), is an entirely independent token t2, i.e. no relation
between W (2; k) and (1) can be trained.
Stream of Tokens: This approach represents mathematical expressions as a
sequence of its inner tokens. This approach has the advantage of learning all
mathematical tokens. However, complex mathematical expressions may lead to
long chains of elements, which increases noise 6. There are several approaches
to reduce the noise, e.g. by only considering identiers and operands [
3
] or by
implementing a long short-term memory (LSTM) architecture that is capable of
handling longer chains [
24
]. Later in this paper, we present a model based on
5 canonical form, dictionary form, or citation form of a set of words
6 Noise means, the data consists of many uninteresting tokens that aect the trained
model negatively.
the stream of tokens approach. We will show that this approach is valuable to
capture relations between mathematical expressions but not between expressions
and their descriptors.
Semantic Groups of Tokens: The third approach of embedding
mathematics is only theoretical, and concerns the aforementioned problems related to
the vague denitions of identiers and functions in a standardized format (e.g.
MathML). As previously discussed, current MIR and ML approaches would
benet from a basic structural knowledge of mathematical expressions, such that
variations of function calls (e.g. W (r; k) and W (2; k)) can be recognized as the
same function. Instead of dening a unied standard, current techniques use
their own ad-hoc interpretations of structural connections, e.g., i is one
identier rather than three [
21, 20
]. We assume that an embedding technique would
benet from a system that is able to detect the parts of interest in
mathematical expressions prior any training processes. However, such system still does not
exist.
2.1
Performance of Math Embeddings
The examples illustrated in [
3, 9, 24
] seem to be feasible as a new approach for
distance calculations between complex mathematical expressions. While
comparing mathematical expressions is essentially practical for search engines or
automatic plagiarism detection systems, these approaches do not seem to
capture the components of complex structure separately, which are necessary for
other applications, such as automated reasoning. Another aspect to be
considered is that in [
9
] they do not train mathematical identiers, preventing their
system from learning connections between identiers and deniens.
Additionally, the connection between entire equations and deniens is, at some level,
questionable. Entire equations are rarely explicitly named. However, in the
extension EqEmb-U [
9
], they use a Syntax Layout Tree (SLT) [
27
] representation
to tokenize mathematical equations and to obtain specic unit-vectors, which is
similar to our identiers as tokens approach.
In order to investigate the discussed approaches, we apply variations of a
word2vec implementation to extract mathematical relations from the arXMLiv
2018 [
4
] dataset, an HTML collection of the arXiv.org preprint archive 7, which
is used as our training corpus. We used the no_problem and warning subsets for
training. There are other approaches that also produce word embeddings given
a training corpus as an input, such as GloVe [
15
], fastText [
1
], ELMo [
16
], and
USE [
2
]. The choice for word2vec is justied because of its general applicability
and robustness in several NLP tasks [
6, 5, 10, 11, 17, 19
].
We replace all mathematical expressions by the sequence of the identiers
it contains, i.e., W (2; k) is replaced by ‘W k’. Further, we remove all common
English stopwords from the training corpus. Finally, we train a word2vec model
(skip-gram) using the following hyperparameters conguration 8: vector size of
7 https://arxiv.org/
8 Non mentioned hyperparameters are used with their default values as described in
the Gensim API [
18
]
300 dimensions, a window size of 15, minimum word count of 10, and a
negative sampling of 1E 5. The trained model is able to partially incorporate
semantics of mathematical identiers. For instance, the closest 27 vectors,
considering cosine simularity, to the mathematical identier f are mathematical
identiers themselves and the fourth closest noun vector to f is vfunction .
Inspired by the classic king-queen example, we explore which tokens perform best
to model a known relation. Consider an approximation vvariable va v vf,
where vvariable represents the word variable, va the identier a, and vf represents
f . We are looking for v that ts best for the approximation. We call this measure
the semantic distance to f with respect to a given relation between two vectors.
We perform an extensive evaluation on the rst 100 entries of the
MathMLBen benchmark [
21
]. We evaluate the average of the semantic distances with
respect to the relations between vvariable and vx, vvariable and va, and vfunction
and vf. In addition, we consider only results with a cosine similarity of 0:7 or
above to maintain a minimum quality in our results. The overall results were
poor with a precision of p = :0023 and a recall of r = :052. For the identier W
(Equation (1)), the evaluation presents four semantically close results: functions,
variables, form, and the mathematical identier q. Even though expected, the
scale of the presented results are astonishing.
Based on the presented results, one can still argue that more settings should
be explored (e.g. dierent embedding techniques, parameters) and dierent
preprocessing steps adopted. Nevertheless, the overall results would not be improved
to a point of being comparable to [
20
] ndings, which report a precision of
p = 0:48. The main reason for this is that, mathematics as a language is highly
customizable. Many of the dened relations between mathematical concepts and
their descriptors are only valid in a local scope. Consider, for example, an author
that notes his algorithm by . This does not change the general meaning of ,
even though it eects the meaning in the scope of the article. Current ML
approaches only learn patterns of most frequently used combinations, e.g., between
f and function. Furthermore, we assume this is a general problem that dierent
embedding techniques and tweaks of settings.
3
Make Math Machine Learnable
A case study [
23
] has shown that only 70% of mathematical symbols are explicitly
declared in the context. Four reasons are causing an explicit declaration in the
context: (a) a new mathematical symbol is dened, (b) a known notation is
changed, (c) used symbols are present in other contexts and require specications
to be properly interpreted, or (d) authors declarations were redundant (e.g. for
improving readability). We assume (d) is a rare scenario compared to (a-c),
unless in educational literature. On the other hand, (d) is most valuable for
learning algorithms.
In cases (b-c), used notations are ambiguous. To overcome this issue, it
requires an extensive database that collects all semantic meanings for
mathematical expressions. In [
25
], they propose the use of tags, similarly to the POS tags in
linguistics, but for tagging mathematical TEX tokens. As a result, a lexicon
containing several meanings for a large set of mathematical symbols is developed.
Such lexicons might enable the disambiguation approaches in linguistics (e.g. via
WordNet [
14
]) to be used in mathematical embeddings in the near future.
Usually, research documents represent state-of-the-art ndings containing
new and unusual notations and lack of extensive explanations (e.g. due to page
limitations). In contrast, educational books carefully and extensively explain new
concepts, thus they are rich of cases (a) and (d). Matsuzaki et al. [
12
] presented
some promising results to automatically pass Japanese university entrance
exams. The system required several manual adjustments. It illustrates the potential
of a well-structured digital mathematical library that distinguishes the dierent
levels of progress in articles (e.g. introductions vs. state-of-the-art publications)
for ML algorithms.
Another problem in recent publications, is the lack of standards for properly
evaluating MIR algorithms, leading to several publications that present
promising results without an extensive evaluation [
3, 9, 24
]. While ML algorithms in
NLP benet from available extensive training and testing datasets, ongoing
discussions about interpretations of mathematical expressions [
21
], and imprecise
standards thwarts research progress in MIR. A common standard for
interpreting semantic structures of mathematics would help to overcome the issues of
dierent evaluation techniques. Therefore, we introduce Mathematical Objects
of Interest (MOI). The goal of MOIs is to combine the advantages of concepts
(1-3) and propose a unied solution for interpreting mathematical expressions.
We suggest MOIs as a recursive tree structure in which each node is an MOI
itself. The current workaround of the problematic example of i as an element of
the vector in content MathML is vague and inappropriate for content specic
tasks. As an MOI, this expression would contain three nodes, with i as the
parent node of two leaves and i. While it rst seems non-intuitive that , as
the vector, is a child node of its own element, this structure is able to incorporate
all three components of semantic information of the expression. Hence, an MOI
structure should not be misinterpreted as a logical network explaining
semantic connections between its elements, but as a highly exible and lightweight
structure for incorporating semantic information of mathematical expressions.
4