Documents from Science, Technology, Engineering, and Mathematics (STEM) disciplines usually contain a signi cant amount of mathematical formulae alongside text. Some Mathematical Information Retrieval (MathIR) systems, e.g., Mathematical Question Answering (MathQA), exploit knowledge from Wikidata. Therefore, the mathematical information needs to be stored in items. In the last years, there have been e orts to de ne several properties and seed formulae together with their constituting identi ers into Wikidata. This paper summarizes the current state, challenges, and discussions related to this endeavor. Furthermore, some data mining methods (supervised formula annotation and concept retrieval) and applications (question answering and classi cation explainability) of the mathematical information are outlined. Finally, we discuss community feedback and issues related to integrating Mathematical Entity Linking (MathEL) into Wikidata and Wikipedia, which was rejected in 33% and 12% of the test cases, for Wikidata and Wikipedia respectively. Our long-term goal is to populate Wikidata, such that it can serve a variety of automated math reasoning tasks and AI systems.
Mathematical Information Retrieval (MathIR) systems, such as Document Recommender (DocRec), Mathematical Question Answering (MathQA), and Automatic Document Classi cation (ADC), need to process and query mathematical formulae. Since Wikidata has been proven useful as a semantic grounding database for Natural Language Processing (NLP) approaches and applications, it was consequential to transfer and adapt classical IR and NLP methods to the special case of mathematical knowledge. In 2016, we implemented support for mathematical properties, such as `de ning formula' (P2534), which were pro? Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). This work was supported by DFG grant GI-1259-1. posed4 and used5. Later, additional properties to include the semantics of the formula identi ers were added 6.
Our long-term goal is to build math.wikipedia.org, a large collaborative, semiformal, machine-readable, language-independent mathematics encyclopedia. Its purpose will be to provide the backbone for automated reasoning tasks, concept entity linking, knowledge-graph population, question answering, and more. #R e t r i e v e a l l i t e m s w i t h ` d e f i n i n g f o r m u l a ' p r o p e r t y P2534 SELECT ? f o r m u l a WHERE f
? i t e m wdt : P2534 ? f o r m u l a . g g 4 https://www.wikidata.org/w/index.php?title=Property:P2534&oldid=303933381 5 https://www.wikidata.org/w/index.php?title=Q35875&oldid=303968820 6 https://www.wikidata.org/w/index.php?title=Property:P4934&oldid=646697942 7 https://query.wikidata.org
The remainder of this paper is structured as follows. In Section 2, we describe how the knowledge can be distilled by annotating mathematical documents (papers, articles, etc.). We show how this can be accelerated using an annotation recommender system. In Section 3, we present standards and systems for benchmarking the knowledge for Mathematical Information Retrieval (MathIR) experiments. Mathematical Entity Retrieval and Linking methods are introduced, and community feedback on incorporating MathEL data into Wikidata and Wikipedia is discussed. Section 4 outlines MathQA and DCE as two example applications of MathEL and concludes with an outlook to challenges and future work. 2
The process of Mathematical Entity Linking can be comprised of 1) Mathematical Entity Annotation and 2) Mathematical Entity Retrieval. In this chapter, we start with 1) by presenting approaches for document annotation and its acceleration by annotation recommendation. 2.1
Document annotations are generally employed to provide additional
information about a resource (e.g., comments) or to link resources (e.g., to URLs). The
Web Annotation Data Model8 speci es the annotation model structure (id, type,
property, relationship) in JSON format. Moreover, RDF classes and ontologies
should be de ned and serialized according to the Web Annotation Vocabulary9.
Several annotation tools and recommender systems for linked data have been
developed so far. Tietz et al. present a system for Wordpress [
To disambiguate and match mathematical expressions in Wikipedia articles to
Wikidata items [
We presented the system with its applications at the Wikiworkshop21
(WWW21 conference) [
If the user clicks on a formula or identi er in the loaded document , recommendations are displayed as shown in Figure 3 for the example formula F = m a, which is seeded into Wikidata as the item `Newton's second law of motion' (Q3268014).
Figure 4 shows an example where the formula name recommendation is very
speci c within a concept hierarchy.
The purpose of the rst testing phase of the system was to elaborate on how
the mathematics knowledge contained in Wikipedia articles can be transferred to
Wikidata statements. For the annotation, we developed the following annotation
rules or guidelines:
{ Annotate identi ers rst, such that the formula name recommendation
retrieval from Wikidata via the `has part' properties is enabled;
{ Do not annotate identi er describing objects, such as `gas', `solid', `line'
instead of quantities or constants;
{ Ignore derivative d characters, such as in d=dt and all indices (superscript or
subscript);
{ Locally di erent meanings of the same identi er within an article should be
avoided (appeal to editors);
{ Ignore block-level formulae that are not relations (equations, inequations,
etc.) or do not have a single identi er right-hand side, e.g., Pi Ii = Pi ri2mi,
0 = :::, dE = :::. Also, ignore formulae in tables, and derivations;
{ Proper names (e.g., `Planck constant') must be capitalized according to the
conventions from `Content dictionary description' (DRMF) [
During the annotation process, we discovered the following issues: { It is not possible to parse equations with no spaces between identi ers, e.g., in the right-hand side of the LATEX string `L = rmv'; { There are di erent common practices to denote vectors in LATEX, e.g., \vec vs. \mathbf; { There are di erent common practices for properties in Wikidata to include the semantics of the formula elements or identi er, e.g., `has part' (P527) `calculated from' (P4934) - see the discussion in Section 3.3; { Sometimes two names are both commonly used to denote the same Formula Concept, e.g., `M-sigma relation' (Q3424023) and `Faber{Jackson relation' (Q1390162); { In case the Wikidata item for a Formula Concept was missing, and we had to create it, we needed to reinsert the new QID into the annotations table manually.
In the future, the process of discovering new issues and requirements to improve the system and extend the annotation guidelines will be continued. Wikimedia users can collaboratively contribute to this joint endeavor. 3 3.1
The open-source and open access formula benchmark system MathMLben12 was
introduced to facilitate the conversion between di erent mathematical formats
such as LaTeX variations and Computer Algebra Systems (CAS) [
For each Gold ID entry or formula, there is an input eld for the Formula
Name, Formula Type (de nition, equation, relation or general formula),
Original Input TeX and manually Corrected TeX together with a Hyperlink to
the source. The Semantic LaTeX Input eld is used for the semantic
annotations, as a grounding for the generation of Content MathML with Wikidata
annotations by LaTeXML [
In 2018, we rst introduced linking mathematical formula content to Wikidata,
both in MathML and LATEX markup [
The following snipped exempli es how Einstein's famous formula E = mc2, the item `mass-energy equivalence' (Q35875) can be found via a SPARQL query. Based on the snippet, a formula search engine on Wikidata can be implemented. #R e t r i e v e a l l i t e m s with l a b e l , d e s c r i p t i o n , and formula , whose d e f i n i n g f o r m u l a p r o p e r t y ( P2534 ) c o n t a i n s t h e s t r i n g `E=mc^ 2 '
SELECT ? item ? i t e m L a b e l ? i t e m D e s c r i p t i o n ? d e f i n i n g F o r m u l a WHERE f ? item wdt : P2534 ? d e f i n i n g F o r m u l a ; FILTER( c o n t a i n s ( ? d e f i n i n g F o r m u l a , `E=mc^ 2 ' @en ) ) SERVICE w i k i b a s e : l a b e l g
g
Figure 6 illustrates how to make use of the `has part' (P527) property to get all items with formula whose identi ers are annotated as `energy' (Q11379) and `speed of light' (Q2111). Based on the snippet, a semantic formula search engine on Wikidata can be implemented. 3.3
In [
The formula linkings from the annotated Wikipedia articles were included in the Wikitext via qid attribute of the <math> tag. After uploading the edited articles to Wikipedia, the following issues were pointed out by the community: { One community member responded that Wikidata should be usable independently of Wikipedia, not having to comply with the technical requirements for the special page display. { It was pointed out that currently, for the Wikdata items that have a `de ning formula' (P2534), the use of the `calculated from' (P4934) property is much higher than `has part' (P527). The claim was that `has part' is only a relict from the past, which will not be used anymore for newly populated items. { Studying some sample equations with `calculated from' properties, another user found that for `Gauss's law for magnetism' (Q1195250) with the formula r B = 0 calculated from `magnetic eld' B and `divergence' r does not make sense. On the other hand, it was asked if `length' or `time' was indeed a `part of' `acceleration' ? In summary, concerns about the general validity of both properties were expressed. { Furthermore, the coexistence and di erent bene ts of the properties `quantity symbol (string)' (P416), `quantity symbol (LaTeX)' (P7973), and `de ning formula' (P2534) for the subexpression strings were discussed. { One user argued that using the latter, only two properties (P2234 and P527) would be needed to develop applications, such as the special page, which is planned to be displayed as popup in the future13. { It was argued that the property P2234 would be general enough to be suitable for longer expressions, such as f(x) or \exp, which are not just symbols. { Another user replied that one should distinguish between a de nition (typically using an `=' sign) and citation (possibly using di erent symbols, such as m or for `mass') of a quantity.
Less than half a day after the execution of the script, two Wikipedia editors started a discussion on our user's talk page14, pointing out the following issues: 13 https://phabricator.wikimedia.org/T208758 14 https://en.wikipedia.org/wiki/User talk:PhilMINT { The `de ning formula' property of the corresponding Wikidata item is edited and evolving independently from the formula strings linked in the Wikipedia articles; { The Wikidata items need to be very speci c to account for a particular formula. See for example `kinetic energy' (Q46276): T = 12 mv2 vs. `kinetic energy of rotating body' (Q104145205): Er = 12I!2; { If a Wikidata item has two or more `de ning formula' properties, only the rst is displayed in the `Special page' (even if the `has part' identi er annotations refer to another). This is problematic, e.g., with `Hooke's law' (Q170282), which currently has both F = kX and = E" as `de ning formula'; { Sometimes disambiguation is needed to distinguish the physics terms from other word meanings, e.g., `work' (Q42213) with description: `energy transferred to an object via the application of force on it through a displacement' vs. `work' (Q6958747): `particular form of activity, sold by many people to sustain themselves'; { In the `Equations of motion' article, a one-line formula includes three sub formulae with di erent meanings that would need three di erent QIDs; { It should be possible for a Wikipedia reader to edit the formula and identi ers directly on the special page, such that the changes get transferred to Wikidata; { The special pages and corresponding Wikidata items should have a `what links here' link, providing a list of every page with a Wikilink to this page. This way, dependencies can be analyzed, and editors warned.
In summary, issues with the formula data representation in both Wikipedia and Wikidata as well as their exchange communication were identi ed. A subsequent discussion on the issue to nd an appropriate property for the identi er semantics was started15. 4
In this nal section, we outline some applications of Mathematical Entity Linking (with Wikidata) and conclude with an outlook to future work potential. 4.1
Motivated by an increasing number of questions on Question Answering (QA)
platforms like Math Stack Exchange (MSE) [
Figure 7 shows the MathQA UI with an example relationship question. A
Question Parsing Module transforms natural language questions into a triple
representation. Subsequently, a Formula Retrieval Module queries the Wikidata
knowledge-base for the requested formula and presents the result to the user.
The user can subsequently choose values for the occurring variables and order
a calculation that is done by a Calculation Module. For the retrieved formula
match, the Wikidata source link is displayed and identi er names and constant
values (e.g., for the speed of light) are fetched from the respective items. The
system employs the formula properties that are discussed in Section 1.
Another application of MathEL can is document subject classi cation [
Mathematical Entity Linking can help to foster explainability for STEM
documents. We are currently working on rst approaches for STEM document
classi cation explainability using classical and mathematical Entity Linking [
In this resource paper, we showed how Wikidata can be employed for the semantic grounding of mathematical entities in documents. We introduced the data model of mathematical statements and discussed the use of di erent properties for the semantics of formula identi er parts. We presented some example SPARQL queries to access the mathematical knowledge in Wikidata. Next, we discussed how to obtain entity linking data via document annotation using systems such as our `AnnoMathTeX' formula and identi er annotation recommender system. We introduced guidelines for formula annotation and reported issues occurring in the annotation process. Moreover, we presented possibilities to benchmark Mathematical Entity Linking (MathEL) with Wikidata using our `MathMLben' gold-standard UI. Next, we discussed community feedback on Wikidata and Wikipedia on the usage of MathEL and the di erent properties involved. Finally, we introduced two applications of MathEL: Mathematical Question Answering (MathQA) and document classi cation explainability.
Since mathematical Formula Concepts [
Summarizing, our key ndings are 1) Wikidata is a valuable resource for storing and retrieving mathematical knowledge; 2) The data model and representation of mathematical statements still su ers from various issues, such as disagreement on the use of speci c properties; 3) By creating a formula semantics benchmark (MathMLben) and annotation recommender system (AnnoMathTeX), we are able to foster reproducibility of MathIR experiments and speed up the process of Wikidata knowledge-base and knowledge-graph population; 4) The machine-interpretable mathematical knowledge can be exploited by various MathIR AI systems, such as question answering or automated mathematical reasoning.
MathEL research is currently at an early stage, and there is still a lot of questions and potential to explore. In the future, we will continue our research on the discovery and recognition of Formula Concept as a basis for MathEL. Moreover, we will promote applications, such as MathQA and STEM document classi cation explainability, by extending our existing systems and approaches. Furthermore, we will introduce new applications, such as our currently developing `PhysWikiQuiz' physics question generation and interrogation system using Wikidata. Finally, we will create and publish large MathEL benchmark datasets and integrate annotation entity linking recommendations into the editing view of Wikipedia articles.