This work is dedicated to population of the OntoMathEdu ontology by definitions of mathematical concepts. OntoMathEdu is a new educational mathematical ontology, intended to be used as a Linked Open Data hub for mathematical education, a linguistic resource for intelligent mathematical language processing and an end-user reference educational database. We propose a template-based method for automatical extraction of definitions from educational mathematical texts in Russian. The method has been implemented on the base of the “OntoIntegrator” system and evaluated on a collection of educational texts from the yaklass.ru website. The obtained F-measure is 89.2%.
This paper is dedicated to population of the OntoMathEdu ontology by definitions of mathematical concepts.
OntoMathEdu is a new educational mathematical ontology [
The ontology underlines the eduation platform of OntoMath digital ecosystem [3], an ecosystem of ontologies, text analytics tools, and applications for mathematical knowledge management, including semantic search for mathematical formulas [4] and a recommender system for mathematical papers [5].
OntoMathEdu is organized in three layers: a foundational ontology layer, a domain ontology layer and a linguistic layer. The domain ontology layer contains language-independent concepts, covering secondary school mathematics curriculum. The linguistic layer provides linguistic grounding for these concepts, and the foundation ontology layer provides them with meta-ontological annotations.
The concepts are organized in two main hierarchies: the hierarchy of objects and the hierarchy of reified relationships. The description of concept contains its name in English, Russian and Tatar, axioms, and relations with other concepts. Figure 1 represents the Altitude of a triangle concept in the WebProtege ontology editor.
In order for using of the ontology for educational purposes, the description of the concepts has to be complemented by its definitions. The manual annotation of definition is timeconsuming task, so some automatic method is needed. In this paper, we propose a method for automatic extraction of concepts definitions from educational texts in Russian.
The rest of the paper is organized as a following. In Sect. 2 we survey the related works. In Sect. 3 we propose the method for definitions extraction. In Sect. 4 we describe our experiments of application of this method. And in Conclusion we propose the future work.
Definition extraction is a popular topic in NLP research. Relying to [6], we can determine three directions to definition extraction.
The first direction is the rule-based approach. The majority of these works use symbolic methods. These methods are based on lexico-syntactic patterns or features, which are manually crafted or semi-automatically learned [7]. Patterns are either very simple sequences of words (e.g. “refers to”, “is defined as”, “is a”) or more complex sequences of words, parts of speech, and chunks. A version of the language of lexical-syntactic patterns for Russian is proposed by E.I. Bolshakova et. al. [8, 9, 10, 11]. As well as they apply this method in various tasks, for example, for extracting the terms and their links for constructing a subject index for scientific text. The rule-based approach is intuitive and has high precision, and used for diferent languages [12, 13, 14]. Another related field is hypernym detection (see e.g., [15, 16] Because many hypernym definitions use the pattern “ is a (type of) ”, and this form is indeed a common structure of the sentence with a definition.
Another fully automated method is proposed by C. Borg and colleagues [17, 18]. They apply genetic programming and rules to difer between definitions and non-definitions. But rules are learned for only one category of patterns, namely “is” patterns. However, most methods have both low recall and precision, because definitional sentences occur in highly variable and potentially complex syntactic structures.
The second direction is the feature engineering approach. This approach uses the statistical machine learning models (i.e., SVM, support vector machine and CRF, conditional random ifelds) [19]. However, this approach not be adapted to new domains eficiently as the designed features might be unavailable or less efective in the new domains.
And the third direction is connected to the deep learning approach which has been recently shown its ability to efectively exploit the word embedding via multiple layers of neural networks [20, 6]
A mathematical text contains a set of structural elements, such as definition, theorem, lemma, proof, etc. Automatic methods for recognizing the structural elements of mathematical texts can help extract from the text relevant data that can be used to create specialized knowledge bases of mathematical objects, and, in particular, to replenish mathematical ontologies.
The OntoMathEdu ontology developed by the authors of this paper includes a hierarchy of mathematical objects whose elements should have corresponding definitions which are extracted, in a general case, from various sources of knowledge. Moreover, since the ontology is focused on educational applications and supports various educational levels, it becomes necessary to describe mathematical objects at diferent levels of abstraction depending on the educational level. Thus, we need to classify definitions by level of complexity. The developed method for annotating definitions of mathematical objects makes it possible to extract definitions of various structural complexities with graphic and text components from mathematical texts.
Let us describe the main ideas of the approach developed to implement the method of extracting definitions from mathematical texts. Each structural element of the text is associated with a set of initial and final lexical-syntactic patterns through which the corresponding segments are recognized in the text. Recognizing the exact boundaries of segments is generally a very complicated procedure for automatic methods, so in some cases, only approximate estimates of the boundaries of segments of structural elements can be obtained, especially for structural elements with complex semantics (definition, statement, etc.). One of the solutions is generation of signal lexical-syntactic patterns which indicate that within the boundaries of a fixed fragment (most often a sentence) a given structural element is present.
To highlight the definition, we also used sets of start and end tags of the constructed definition model.
The developed method for annotating the structural elements of a mathematical text is based on lexical-syntactic laws and rules that describe the lexical-syntactic features of using the elements in scientific, technical, and educational texts in Russian.
Thus, to build a model of a structural element of a text, it is necessary: • to highlight the set of lexical and syntactic patterns characterizing the structural elements of the text. For this purpose, we use collections of mathematical texts; • to describe the lexical composition and syntactic models of the selected patterns; • to develop methods for recognizing lexical-syntactic patterns in mathematical textbooks, taking into account potential ambiguity, which in some cases directly correlates with pattern type recognition.
To write lexical-syntactic patterns, we use a special language of syntactic patterns in which a specific pattern is represented as a finite sequence of tokens with specified grammatical characteristics and fixed semantics.
The semantic annotation of the structural elements of a text is represented in XML notation. The list of lexical and syntactic patterns in the current version is open and serves to refine the method.
Template syntax models include the following groups of elements: • introductory words (for example: therefore, thus, it means, and others); • conjunctions and particles (for example: if, since); • collocations; • syntactic models such as “ 1 – это 1” and “ 1 называется 5” ( 1 is 1).
These models include the morphological characteristics of each unit. In these examples, 1 is a nominal group in the Nominative case and 5 is a nominal group in the Instrumental case (specific word forms are highlighted in italics). For example: the syntactic model “ 5|<под>+ 5+ 3 1 1 2+ 1” (Preposition + Noun phrase in the Instrumental case + Verb, 3 person (p3), singular ( 1), Present tense ( 1), Passive voice ( 2)+ Noun phrase in the Nominative case) is implemented in a Russian context such as “под 5 понимается 1” ( 1 is understood as 5); • a cliche´ (for example, <admits a form>, <satisfies a condition>); • elements of paratext (heading units such as Definition, Theorem, Lemma, Proof; also special symbol “□” which is the sign of the end of a proof, etc.).
Consider the main problems of the method of extracting definitions from mathematical texts.
From the point of view of its syntactic structure, the definition in mathematical documents is defined by a standard set of structural schemes: 1. 2. 3. 4. 1 1, 5 3 1 1 2 1, 1 − 5, 1 2 1 3 + where 1 is a noun phrase in the Nominative case and NP5 is a noun phrase in the Instrumental case; Cop is a copula; 3 1 1 2 is a verb in the form of 3 person ( 3), singular ( 1), Present tense ( 1), Passive voice ( 2); − is a short form of passive participle; 2 1 3 + is an analytical verb form with features such as person ( 1), plural ( 2), Future tense ( 3), and Active voice.
The scientific style is characterized by the use of constructions with verbs in the form of 3 person, plural (“we give a definition”, “we will use the definition”). In Russian, the word form definition has two homonymous forms in the Nominative and Accusative cases. In contexts with verbs to give, to apply etc. it is used in the Accusative case whereas in headings it appears in the Nominative case.
Recognition of the boundaries of a definition fragment is carried out by using templates of the initial and final tags of this structural element. The list of initial tags contains a heading unit (a definition) and a set of verbal formulas (e.g. “we give a definition”, “give a definition”, “we will use a definition”). The text segment linked with the definition contains, as a rule, one sentence. Therefore, if the initial tag of the definition is set, then the final tag is set at the end of the corresponding sentence. Signal tags for the definition are represented by a set of verbal formulas that are built into certain syntactic models describing definition segments. Signal tags are contained within the definition segment and can probabilistically establish its boundaries (within the boundaries of the sentence containing the signal tag).
The syntactic models with a signal tag are (the signal tag is in italics): • 1 - <это/is> 1, • 1 <названо/ was denoted> 5, • 1 <называется/ is denoted> 5, • <аналогично определяется/ is similarly denoted> 1, • <назовем/ we denote>, • 1 <будем называть/ we will denote> 5.
Evaluation of the method of annotating the structural element “Definition ” of a mathematical document using signal tags is based on the data of the experiment which was carried out using NLP models and methods implemented in the onto-logical-linguistic system “OntoIntegrator” [21, 22].
For the experiment, a collection of educational texts was selected from the “I Class” website of a geometry course for grade 7 (https://www.yaklass.ru/p/geometria#program-7-klass). The educational texts relate to 4 studied sections: basic geometric knowledge, triangles, parallel lines, and relations between the sides and angles of a triangle. In each of the sections, there is a theory subsection containing basic definitions which are illustrated by drawings. Quite often, theoretical information at this level is given as a description of corresponding figures.
The purpose of the experiment was to extract all the definitions in the geometry course for grade 7 and to match each mathematical concept of a definition to the OntoMath Edu ontology concept and also to annotate the ontology concept by a definition extracted from the text.
The annotation model of the structural element “Definition ” was implemented using the “OntoIntegrator” ontological-linguistic system. Using the designed approach, we developed the conceptual model “Definition ” implemented in the model ontology, which is the basic component of the “OntoIntegrator” system.
The graphic structure of the conceptual model “Mathematical Definition ” is shown in Figure 2. The vertices of the graph are concept models from the model ontology, and the edges are relations defined in the model ontology. Here the “ Model Aggregation” relation is used.
The “Mathematical definition ” conceptual model allows us to detect any definitions in texts. The conceptual structure contains two main types of concept models: concept properties (light gray background) and concept models of the -implementation type (dark gray background). All the necessary parameters for a complete description of the mathematical definition model are transmitted by the parameters of the “model aggregation” relationship, to which all elements are linked. Among the concept models of the -implementation type, we use syntactic models that are connected with the search procedures for discontinuous structures and sentence segmentation and semantic models that are connected with identification of text boundaries of a definition, as well as with verification of various semantic search conditions for the required entities. The “Signal tag of the definition ”, “Initial tag of the definition ”, “Final tag of the definition ”, “Verisimilar markers of definition beginning ”, and “Verisimilar markers of definition ending ” conceptual models determine the conditions for detecting the beginning, the middle and the end of the mathematical definition by diferent procedures. Such detection is based both on the syntactic properties of the definition and on the ontological markup of the text by the OntoMathEdu ontology concepts.
We have analyzed the results of the experiment and distinguished several types of definitions. Further, in the examples, the ontological object will be highlighted in italic.
The first type includes classical definitions in which object (an ontology object) is defined as “is denoted the th/called the th” (e.g. A circle is the set of all points in a plane equidistant from a fixed point in the plane called the center).
The second type, in contrast to the first one, gives distant construction of the object name in Russian (e.g. “Если обе стороны угла лежат на одной прямой, угол называют развёрнутым / If both sides of the angle are on one straight line, the angle is called the straight [angle]”) or more complex example as “the triangle side ... opposite to the corner”. In this case we need to reformulate (to bundle) the parts of the object name (“in our example as cторона, противолежащая угл"(RU)/ “a side of a triangle that subtends the opposite angle"). We developed a special procedure for processing distant names to recognize the names of an ontology object.
The third type is also a classic type of definition in which the defined object (an ontology object) is specified in the syntactical construction “ is this ” (e.g. “A line segment, or a segment, is a set of points consisting of two points on a line, called endpoints, and all of the points on the line between the endpoints”). This model allowed us to identify some set of geometric objects that are missed in the current version of the ontology (e.g. “A dimension is a comparison of a measurement object with a selected unit of measure”).
The fourth type includes mathematical expressions in the definition body (e.g. “ is the side opposite ∠ and that ∠ is the angle opposite side ”).
The fith type includes a drawing in the definition body as in the example below.
The sixth type is a complex definition consisting of several sentences (e.g. “ A circle is the set of all points in a plane equidistant from a fixed point in the plane called the center. A radius of a circle is a line segment from the center of the circle to any point of the circle”). The main dificulty is that the boundaries of such a definition are not clear. The current example of automatic detection of such cases gives an error in the placement of the beginning tag, due to the fact that the first sentence does not include the appropriate signal tag. Automatic analysis of this type of definition requires additional semantic analysis, which is our future research.
In total, 56 definitions for various geometric objects and their properties are extracted from 4 studied sections from the collection of educational texts in our experiment. Table 1 at Appendix contains examples of definitions extracted automatically from school texts. Statistics on extracted definitions are given in Table 2 (if a definition was extracted only partially, we count it as 0.5 definition).
Thus, the method precision of extracting definitions in a text on geometry is 91% (5 erroneous definitions, including incomplete ones) and the method recall is 87.5% (7 definitions were not extracted from texts). F-measure is 89.2%.
The suficiently high precision of extracting definitions in texts allows us to automatically annotate definitions in a text body, and also to autocomplete annotations of ontology concepts.
In this paper we presened a template-based method for populating the OntoMathEdu ontology by definitions, automatically extracted from educational mathematical texts in Russian. The method has been implemented on the base of the “OntoIntegrator” system and evaluated on a small collection of educational texts from the yaklass.ru website. The obtained F-measure is 89.2%. As a future work we are going to: 1. apply the proposed method to other text collections; 2. complement a template-based approach by deep learning ones; 3. extend the developed method for extracting definitions from professional mathematical texts and apply it for population the ontology of professional mathematics OntoMathPRO, starting with the Computability theory domain.
The work was funded by Russian Foundation of Basic Research according to the research projects no. 19-29-14084 and 20-31-70012. [2] A. Kirillovich, O. Nevzorova, M. Falileeva, E. Lipachev, L. Shakirova, OntoMathEdu: a new linguistically grounded educational mathematical ontology, in: C. Benzmüller, B. Miller (Eds.), Proceedings of the 13th International Conference on Intelligent Computer Mathematics (CICM 2020), Lecture Notes in Artificial Intelligence, vol. 12236, Springer, 2020. https://doi.org/10.1007/978-3-030-53518-6_10. [3] A. Elizarov, A. Kirillovich, E. Lipachev, O. Nevzorova, Digital ecosystem OntoMath: Mathematical knowledge analytics and management, in: L. Kalinichenko, S. Kuznetsov, Y. Manolopoulos (Eds.), XVIII International Conference on Data Analytics and Management in Data Intensive Domains (DAMDID/RCDL 2016), Communications in Computer and Information Science, vol. 706, Springer, 2017, pp. 33–46. https://doi.org/10.1007/9783-319-57135-5_3. [4] A. Elizarov, A. Kirillovich, E. Lipachev, O. Nevzorova, Semantic formula search in digital mathematical libraries, in: Proceedings of the 2nd Russia and Pacific Conference on Computer Technology and Applications (RPC 2017), IEEE, 2017, pp. 39–43. https: //doi.org/10.1109/RPC.2017.8168063. [5] A. M. Elizarov, A. V. Kirillovich, E. K. Lipachev, A. B. Zhizhchenko, N. G. Zhil’tsov, Mathematical knowledge ontologies and recommender systems for collections of documents in physics and mathematics, Doklady Mathematics 93 (2016) 231–233. https: //doi.org/10.1134/S1064562416020174. [6] A. P. B. Veyseh, F. Dernoncourt, D. Dou, T. H. Nguyen, Definition extraction using linguistic and structural features, in: M. Walker, H. Ji, A. Stent (Eds.), Proceedings of the 34th AAAI Conference on Artificial Intelligence (AAAI 2020), vol. 34, no. 05: AAAI-20 Technical Tracks, AAAI, 2020, pp. 9098–9105. https://doi.org/10.1609/aaai.v34i05.6444. [7] J. L. Klavans, S. Muresan, Evaluation of the DEFINDER system for fully automatic glossary construction, Proceedings of the American Medical Informatics Association Symposium (2001) 324–328. https://www.ncbi.nlm.nih.gov/pubmed/11825204. [8] E. Bolshakova, N. Efremova, A heuristic strategy for extracting terms from scientific texts, in: M. Y. Khachay, et al. (Eds.), Revised Selected Papers of the 4th International Conference on Analysis of Images, Social Networks and Texts (AIST 2015), Communications in Computer and Information Science, vol. 542, Springer, 2015, pp. 297–307. https://doi.org/10.1007/978-3-319-26123-2_29. [9] E. Bolshakova, K. Ivanov, Term extraction for constructing subject index of educational scientific text, in: Computational Linguistics and Intellectual Technologies: Papers from the Annual International Conference “Dialogue”, 2018, pp. 143–149. http://www.dialog-2 1.ru/media/4291/bolshakovaei_ivanovkm.pdf. [10] E. Bolshakova, N. Efremova, K. Ivanov, Terminological information extraction from Russian scientific texts: Methods and applications, in: G. Wohlgenannt, et al. (Eds.), Proceedings of 3rd Workshop on Computational linguistics and language science (CLLS 2018), EPiC Series in Language and Linguistics, vol. 4, EasyChair, 2019, pp. 95–106. https://doi.org/10.29007/k93q. [11] Lexico-syntactic pattern language, ???? URL: http://lspl.ru, http://lspl.ru, last accessed 2020/05/08. [12] A. Przepiórkowski, Ł. Degórski, M. Spousta, K. Simov, P. Osenova, L. Lemnitzer, V. Kuboň, B. Wójtowicz, Towards the automatic extraction of definitions in Slavic, in: J. Piskorski, H. Tanev (Eds.), Proceedings of the Workshop on Balto-Slavonic Natural Language Processing, ACL, 2007, pp. 43–50. https://www.aclweb.org/anthology/W07-1706. [13] A. Storrer, S. Wellinghof, Automated detection and annotation of term definitions in german text corpora, in: N. Calzolari, et al. (Eds.), Proceedings of the 5th International Conference on Language Resources and Evaluation (LREC’06), ELRA, 2006, pp. 2373– 2376. https://www.aclweb.org/anthology/L06-1066/. [14] I. Fahmi, G. Bouma, Learning to identify definitions using syntactic features, in: R. Basili, A. Moschitti (Eds.), Proceedings of the Workshop on Learning Structured Information in Natural Language Applications, ACL, 2006, pp. 64–71. https://www.aclweb.org/antholo gy/W06-2609/. [15] R. Snow, D. Jurafsky, Y. N. Andrew, Learning syntactic patterns for automatic hypernym discovery, in: L. Saul, Y. Weiss, L. Bottou (Eds.), Proceedings of the 17th International Conference on Neural Information Processing Systems (NIPS 2004), MIT Press, 2005, pp. 1297–1304. https://papers.nips.cc/paper/2659-learning-syntactic-patterns-for-automati c-hypernym-discovery. [16] V. Shwartz, E. Santus, D. Schlechtweg, Hypernyms under siege: Linguistically-motivated artillery for hypernymy detection, in: M. Lapata, P. Blunsom, A. Koller (Eds.), Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2017). Volume 1: Long Papers, ACL, 2017, pp. 65–75. https://www.ac lweb.org/anthology/E17-1007/. [17] C. Borg, M. Rosner, G. Pace, Evolutionary algorithms for definition extraction, in: G. Sierra, M. Pozzi, J.-M. Torres (Eds.), Proceedings of the 1st Workshop on Definition Extraction (WDE ’09), ACL, 2009, p. 26–32. https://www.aclweb.org/anthology/W09-4405/. [18] C. Borg, Automatic definition extraction using evolutionary algorithms. Master’s thesis,
University of Malta, 2009. http://staff.um.edu.mt/cbor7/publications/2009Thesis.pdf. [19] E. Westerhout, Definition extraction using linguistic and structural features, in: G. Sierra, M. Pozzi, J.-M. Torres (Eds.), Proceedings of the 1st Workshop on Definition Extraction (WDE ’09), ACL, 2009, pp. 61–67. https://www.aclweb.org/anthology/W09-4410/. [20] L. Espinosa-Anke, S. Schockaert, Syntactically aware neural architectures for definition extraction, in: M. Walker, H. Ji, A. Stent (Eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL 2018).
Volume 2: Short Papers, ACL, 2018, pp. 378–385. https://doi.org/10.18653/v1/N18-2061. [21] O. Nevzorova, V. Nevzorov, Ontology-driven processing of unstructured text, in: S. Kuznetsov, A. Panov (Eds.), Proceedings of the 17th Russian Conference on Artificial Intelligence (RCAI 2019), Communications in Computer and Information Science, vol. 1093, Springer, 2019, pp. 129–142. https://doi.org/10.1007/978-3-030-30763-9_11. [22] O. Nevzorova, V. Nevzorov, A. Kirillovich, A syntactic method of extracting terms from special texts for replenishing domain ontologies, in: Proceedings of the 2nd Russia and Pacific Conference on Computer Technology and Applications (RPC 2017), IEEE, 2017, pp. 127–131. https://doi.org/10.1109/RPC.2017.8168083. Если обе стороны угла ле- If both sides of the angle are
Часть прямой, ограниченThe part of the line limited by ная двумя точками, называ- two points is called a segment ется отрезком жат на одной прямой, угол называют развернутым on one straight line, the angle is called the straight [angle] Сторону, которая
лежит напротив
угла, называют противолежащей углу, и
The side that lies opposite to the corner is called the opposite to the corner, and угол называют противоле- the corner is called the жащим стороне. opposite to the side. Треугольник – это геометрическая
фигура, образованная тремя
отрезками, которые соединяют три не лежащие на одной прямой точки.
На плоскости две прямые и , которые не пересекаются, называются параллельЕсли две прямые пересекает третья прямая, то углы называются так <Image>: накрест лежащие углы: ∠3 и ∠5; ∠2 и ∠8; соответственные углы: ∠1 и ∠5, ∠4 и ∠8, ∠2 и ∠6, ∠3 и ∠7; односторонние углы: ∠3 и ∠8, ∠2 и ∠5. ными и обозначаются ∥ . ∥ .
A triangle is a geometric figure formed by
three segments that connect three points
not lying on one straight line.
On the plane, two lines and that do not intersect are called parallel and denoted by If two lines intersect the third line, then the angles are called like this <Image>: angles lying crosswise: ∠3 and ∠5, ∠2 and ∠8, corresponding angles: ∠1 and ∠5, ∠4 and ∠8, ∠2 and ∠6, ∠3 and
∠7; one-sided angles: ∠3 and ∠8, ∠2 and ∠4. Отрезок, который соединяет две точки на окружности, A line that connects two points on a circle is called a chord. # 1 2 3 4 5 6 7 8 9 10 11 12 13 14
URL 13 1 1 3