EmEL++: Embeddings for ++ Description Logic Sutapa Mondala,c , Sumit Bhatiab and Raghava Mutharajua a Knowledgeable Computing and Reasoning (KRaCR) Lab, IIIT-Delhi, India b IBM Research AI, New Delhi, India c TCS Research & Innovation, India Abstract Knowledge graph (KG) embedding models have recently gained increased attention. However, most of the existing models for KG embeddings ignore the structure and characteristics of the underlying ontology. In this work, we present EmEL++ embeddings – an ontology-based embedding model for the ++ description logic. EmEL++ maps the classes and the relations in an ontology to an n-dimensional vector space such that the relations between classes and relations in the ontology are preserved in the vector space. We evaluate the proposed embeddings on four different datasets and show that the proposed embeddings outperform the traditional knowledge graph embeddings on the subsumption reasoning task. Keywords Ontology, ++ , Description Logic, Geometric Embeddings 1. Introduction Methods for learning embedding functions that map the underlying entities to a vector space have gained significant attention in recent times. Different methods for learning embedding functions try to preserve the critical properties of, and relations between, the underlying en- tities (such as words, concepts, documents, nodes and edges in a graph) in the 𝑛-dimensional vector space. A variety of embeddings for Knowledge Graphs [1, 2, 3, 4, 5, 6] have been pro- posed. These methods differ in terms of the underlying properties of the knowledge bases preserved in the vector space. For example, the TransE [1] model considers the relations in a KG as a translation operator over the entities, TransH [2] models the relations as a hyper- plane in the vector space to allow for reflexive, one-to-many, many-to-one, and many-to-many relations, DistMult [3] uses Matrix Factorization based approach to bring similar entities and relations together in the vector space. However, most of the KG embedding models focus on capturing the structural properties of the graph and the interaction between the entities and do not take into account the constraints and characteristics of the underlying ontology. Consequently, In A. Martin, K. Hinkelmann, H.-G. Fill, A. Gerber, D. Lenat, R. Stolle, F. van Harmelen (Eds.), Proceedings of the AAAI 2021 Spring Symposium on Combining Machine Learning and Knowledge Engineering (AAAI-MAKE 2021) - Stanford University, Palo Alto, California, USA, March 22-24, 2021. " sutapa.mondal@tcs.com (S. Mondal); sumitbhatia@in.ibm.com (S. Bhatia); raghava.mutharaju@iiitd.ac.in (R. Mutharaju)  0000-0003-0870-1015 (S. Mondal); 0000-0002-8146-4100 (S. Bhatia); 0000-0003-2421-3935 (R. Mutharaju) © 2021 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings http://ceur-ws.org ISSN 1613-0073 CEUR Workshop Proceedings (CEUR-WS.org) the embeddings produced by such methods are not suited for reasoning tasks such as classifi- cation, satisfiability and consistency checking. Kulmanov et al. [7] have recently proposed EL Embeddings (ElEm) and have described a geometric interpretation of embedding ++ ontolo- gies in an 𝑛-dimensional vector space (Section 3). However, the ElEm embeddings are limited in terms of the coverage of ++ constructs as they ignore the role oriented constructs in such ontologies. Further, the use case considered by them is predicting protein-protein interactions that is modeled as a traditional link prediction task in knowledge bases. In this work, we ad- dress the limitations of ElEm embeddings and propose EmEL++ embeddings. We build upon the framework introduced by Kulmanov et al. [7] and extend it to offer more complete cover- age of the ++ semantics (Section 3.1) for performing subsumption reasoning task. Baader et al. [8] have shown that all the standard reasoning tasks in ++ ontologies can be reduced to subsumption task. Thus, to the best of our knowledge, we present the first attempts at per- forming reasoning tasks by embedding ontologies in a vector space. We compare our proposed approach using four ontologies of different sizes and characteristics. Our evaluation shows that EmEL++ outperform the traditional KG embeddings at the reasoning task and are also able to preserve the characteristics of underlying ontologies better. We would also like to emphasize that performing reasoning in the vector space is critical as it has the potential to speed up the reasoning process significantly. As we describe in Section 4, the subsumption task in vector space involves computing distances between the source class and all the other classes in the ontology. In the worst case, this is an 𝑂(𝑛) operation where 𝑛 is the number of classes. Thus, irrespective of the complexity of the underlying ontology, the subsumption task could be per- formed in 𝑂(𝑛) time. Further, with uses of techniques such as semantic hashing or binarized embeddings [9], the similarity based search operations can be performed in 𝑂(1) time. There- fore, we believe that embedding based approaches, despite their lower accuracies than standard reasoners and no theoretical guarantees of performance, offer a promising direction of future research to develop more efficient reasoners, especially for more complex description logics such as  (OWL 2 DL). 2. Related Work A wide range of methods for computing KG embeddings have been proposed. Node2Vec [10] initiated the idea of learning features for networks addressing the scalability challenge. Al- though their results are decent for link prediction task but they make assumptions on con- ditional independence of the feature space which may not hold true in real world scenarios. Eventually, this concept got popularized with KGs wherein, a fact is represented as a triple of the form (h,r,t). Semantic matching KG models exploit similarity based scoring functions and match latent semantics of entities and relations based on vector representations. For exam- ple, [4] proposed RESCAL which uses multiple matrices to represent relations among entities but scalability remains an issue. [3] proposed DistMult to overcome challenges of RESCAL. Although, DistMult is similar to RESCAL but DistMult ensures low number of parameters for relations by restricting the matrices. Later, translational based models for KG embeddings used distance based scoring functions. This technique gained most attention due to its simplicity to measure the correctness of a fact. It measures the plausibility of a fact as distance between the entities after translation carried out by relation. TransE[1], TransH [2] and TransR[6] being different variants of the same. TransE being the most representative model, treats the relations as translations such that given a fact, relation vector r minimizes the distance between h and t in vector space. Whereas, TransH interprets relations as translating operator on a hyperplane and TransR models entities and relations in two distinct spaces, performing translation on cor- responding relation space. [11] worked on a novel approach inspired by the theory of quantum logic to embed a Knowledge Base (KB) in  description logic. Existing works on ontology embedding such as Onto2vec [12] focused on using word2vec as an underlying model. Most of this work focuses on encoding the entities and relations, but they lack in handling the com- plex relations in an ontology. Recently, [13] pointed out the lack of expressivity in the classical approaches to model relations for KG embeddings. Moreover, their work indicates that geo- metric models are a better way to learn embeddings for ontologies. Further, [14] present a new approach using deep learning with knowledge based systems to emulate reasoning structure. Although, they perform experiments on a synthetic and a non-synthetic dataset, but in our work we look at multiple datasets with different characteristics and reason about their varying performances. However, [7] tries to overcome the drawbacks of KG embeddings but it does not address all the ++ constructs that are relevant to capture the relations present in an ontology. Further, the evaluation is focused only on link prediction task. 3. Background 3.1. Embedding ++ Ontologies in a Vector Space A description logic signature is a tuple ⟨𝑁𝐶 , 𝑁𝑅 , 𝑁𝐼 ⟩, where 𝑁𝐶 , 𝑁𝑅 , and 𝑁𝐼 are countably infinite, mutually disjoint sets of concept names, role names, and individual names respectively. In the following discussion, {𝐶, 𝐶1 , 𝐶2 , 𝐷, 𝐸, ⊤, ⊥} ∈ 𝑁𝐶 , {𝑅, 𝑆, 𝑅1 , 𝑅2 } ∈ 𝑁𝑅 , and {𝑎, 𝑏} ∈ 𝑁𝐼 . All the axioms in the ++ description logic can be reduced to one of the normal forms [15] as follows: 1. All the TBox axioms are in one of the four normal forms (𝐶1 ⊑ 𝐷, 𝐶1 ⊓ 𝐶2 ⊑ 𝐷, ∃𝑅.𝐶1 ⊑ 𝐷, and 𝐶1 ⊑ ∃𝑅.𝐶2 ). 2. The bottom concept can only appear on the right side of the equations, and can only appear in the first three normal forms. 3. All role inclusions are of the form 𝑅 ⊑ 𝑆 or 𝑅1 ◦𝑅2 ⊑ 𝑆. Further, the instantiation and role assertion axioms in the ABox can be converted into TBox axioms as follows: 𝐶(𝑎) ⟶ {𝑎} ⊑ 𝐶 𝑅(𝑎, 𝑏) ⟶ {𝑎} ⊑ ∃𝑅.{𝑏} Thus, with the above transformations, all the axioms in an ++ ontology can be reduced to one of the normalized forms and the task of embedding ontologies in a vector space requires us to learn mapping functions for classes and relations that are part of the normal forms. Figure 1: Geometric Representation of classes and relation 3.1.1. Intuition. Typically, for mapping the entities of interest to a vector space, requires to learn a mapping function subject to certain constraints, encoded in the form of an objective function that is optimized during the training phase. These objective functions are designed such that spe- cific properties of the underlying entities are also retained in the vector space. For example, the word2vec [16] model for word embeddings minimizes the distance between contextually similar words, RDF2Vec [17] adapts language modeling approach to capture local information from the graph sub-structures, and TransE [1] model for KGs. Similarly, in this work we learn mapping functions that can embed ++ ontologies in a vector space. In order to do so, we build upon and extend the framework proposed by Kulmanov et al. [7] that interprets a class in the ontology as an 𝑛-ball (defined by its radius and center) in the vector space. Let us consider two classes 𝐶 and 𝐷 such that 𝐶 ⊑ 𝐷. Let these two classes be represented by their respective 𝑛-balls 𝑏𝑐 and 𝑏𝑑 in the vector space such that 𝑏𝑐 ∶ {𝑐⃗, 𝑟𝑐 } and 𝑏𝑑 ∶ {𝑑⃗ , 𝑟𝑑 }; where 𝑐⃗ and 𝑑⃗ are the centers and 𝑟𝑐 and 𝑟𝑑 are the radii of the respective 𝑛-balls. Geometrically, if 𝐶 ⊑ 𝐷, the mapping function should aim to ensure that the 𝑏𝑐 lies inside 𝑏𝑑 (Figure1 (a)). Similarly, if 𝐶 and 𝐷 are disjoint, the respective 𝑛-balls should not overlap with each other in the vector space (Figure1 (b)). Further, similar to the TransE model [1], the relations in the ontology are interpreted as translations operating on the classes. More specifically, if 𝐶 ⊑ ∃𝑅.𝐷, the center of 𝑛-ball representing 𝐶 can be moved to the center of the 𝑛-ball representing 𝐷 (Figure1 (c)). 3.2. Loss Functions With the intuitive framework described above, let us now describe the objective functions that should be optimized during the training phase to learn the mapping functions. Let 𝑒𝑣 ∶ 𝐂 ∪ 𝐑 ⟼ 𝑅 𝑛 be the mapping function that maps each class and relation to a unique vector in the 𝑛-dimensional embedding space. For 𝐶𝑖 ∈ 𝐂, the resulting vector corresponds to the centre of the 𝑛-ball representing the class. Further, let 𝑒𝑟 ∶ 𝐶 ⟼ 𝑅 + be the mapping function that maps each class into a non-negative real number, that represents the radius of the 𝑛- ball corresponding to class 𝐶. Thus, the pair (𝑒𝑣 , 𝑒𝑟 ) of functions represents the operations needed to embed an ++ ontology into an 𝑛-dimensional space. We now describe the various loss functions to represent the different constructs in ++ . The total loss that needs to be minimized during the learning process is the sum of the individual loss functions. 3.2.1. Loss Functions for the Four Normal Forms: As described before, the first normal form (𝐶 ⊑ 𝐷) when embedded in a vector space can be interpreted geometrically as two 𝑛-balls, such that the 𝑛-ball corresponding to class 𝐶 lies inside the 𝑛-ball corresponding to 𝐷. Hence, our mapping functions 𝑒𝑣 and 𝑒𝑟 should bring the centers of the two classes closer to each other, and give the sub-class a smaller radius than the super-class. The loss function presented in Equation 1 captures this intuition and penalizes the mappings that do not adhere to these constraints. Also note that in addition to the above constraints, we also add margin loss (𝛾 ) and a normalization loss that brings the centres of 𝑛-balls of all the classes on the unity sphere. ⎛ ⎞ ⎜ ⎟ ⎜ ⎟ | | | | ‖ ‖ 𝐶⊑𝐷 (𝑐, 𝑑) = max ⎜0, ‖𝑒𝑣 (𝑐) − 𝑒𝑣 (𝑑)‖ + 𝑒𝑟 (𝑐) − 𝑒𝑟 (𝑑) −𝛾 ⎟ + |‖‖𝑒𝑣 (𝑐)‖‖ − 1 | + |‖‖𝑒𝑣 (𝑑)‖‖ − 1 | (1) ( ⏟⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏟⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏟ ⏟⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏟⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏟ ) | | | | ⎜ ⎟ ⎜ penalize if the two penalize if sub-class ⎟ centers are far away has larger radius ⎝ ⎠ In the vector space, the second normal form, i.e., 𝐶 ⊓ 𝐷 ⊑ 𝐸, implies that the 𝑛-ball for class 𝐸 should completely engulf the area of intersection of 𝑛-balls for classes 𝐶 and 𝐷. The first term in the loss function (Equation 2) imposes a penalty if the classes 𝐶 and 𝐷 are disjoint. The second and third terms together enforce that the center of the 𝑛-ball for class 𝐸 lies in the area of intersection of 𝑛-balls for classes 𝐶 and 𝐷 such that it satisfies the normal form. Finally, the fourth term requires the radius of the 𝑛-ball of 𝐸 to be greater than the smallest radii among 𝑛-balls of 𝐶 and 𝐷. 𝐶⊓𝐷⊑𝐸 (𝑐, 𝑑, 𝑒) = max (0, (‖‖𝑒𝑣 (𝑐) − 𝑒𝑣 (𝑑)‖‖ − 𝑒𝑟 (𝑐) − 𝑒𝑟 (𝑑) − 𝛾 )) + max (0, (‖‖𝑒𝑣 (𝑐) − 𝑒𝑣 (𝑒)‖‖ − 𝑒𝑟 (𝑐) − 𝛾 )) + max (0, (||𝑒𝑣 (𝑑) − 𝑒𝑣 (𝑒)|| − 𝑒𝑟 (𝑑) − 𝛾 )) + max 0, (min (𝑒𝑟 (𝑐), 𝑒𝑟 (𝑑)) − 𝑒𝑟 (𝑒) − 𝛾 ) ( ) |‖ | | | | | + |‖𝑒𝑣 (𝑐) − 1 ‖‖| + |‖‖𝑒𝑣 (𝑑) − 1 ‖‖| + |‖‖ 𝑒𝑣 (𝑒) − 1 ‖‖| | | | | | | (2) The first two normal forms are concerned with the mappings of classes and properties of their respective 𝑛-balls in the vector space. The next two normal forms, i.e., NF3 and NF4, involve relations and how they are associated with the classes. Recall that similar to TransE [1], we consider the relations in ontology as translations that operate on class instances. Consider the normal form 𝐶 ⊑ ∃𝑅.𝐷. In the vector space, 𝐶 and 𝐷 are represented as two 𝑛-balls 𝑏𝑐 and 𝑏𝑑 , respectively. If 𝑒𝑣 (𝑅) is the vector for 𝑅 in vector space, then adding 𝑒𝑣 (𝑅) to a point in 𝑏𝑐 should move it to a point in 𝑏𝑑 (i.e., 𝑅 translates the points in 𝑏𝑐 to points in 𝑏𝑑 ). The following loss functions capture these semantics as expressed by the third and fourth normal forms. | | | | 𝐶⊑∃𝑅.𝐷 (𝑐, 𝑑, 𝑟) = max (0, (‖‖𝑒𝑣 (𝑐) + 𝑒𝑣 (𝑟) − 𝑒𝑣 (𝑑)‖‖ + 𝑒𝑟 (𝑐) − 𝑒𝑟 (𝑑) − 𝛾 )) + |‖‖𝑒𝑣 (𝑐)‖‖ − 1 | + |‖‖𝑒𝑣 (𝑑)‖‖ − 1 | | | | | (3) | | | | ∃𝑅.𝐶⊑𝐷 (𝑐, 𝑑, 𝑟) = max (0, (‖‖𝑒𝑣 (𝑐) − 𝑒𝑣 (𝑟) − 𝑒𝑣 (𝑑)‖‖ − 𝑒𝑟 (𝑐) − 𝑒𝑟 (𝑑) − 𝛾 )) + |‖‖𝑒𝑣 (𝑐)‖‖ − 1 | + |‖‖𝑒𝑣 (𝑑)‖‖ − 1 | | | | | (4) 3.2.2. Handling Bottom Concept (⊥): Recall from the discussion in Section 3 that the bottom concept can appear only on the right hand side of the first three normal forms [8]. We now present the loss functions for each of the three special cases. The resulting first normal form 𝐶 ⊑ ⊥ indicates that class 𝐶 is unsatisfiable. Thus, in the vector space, we represent this constraint by reducing the radius of class 𝐶 to zero. This is achieved by the following loss function. 𝐶⊑⊥ (𝑐) = 𝑒𝑟 (𝑐) (5) Next, the second normal form with the bottom concept is 𝐶 ⊓ 𝐷 ⊑ ⊥ indicating that 𝐶 and 𝐷 are disjoint. In the vector space, this indicates that the 𝑛-balls of classes 𝐶 and 𝐷 are non- overlapping. This is captured by the following loss function. | | | | 𝐶⊓𝐷⊑⊥ (𝑐, 𝑑) = max (0, (𝑒𝑟 (𝑐) + 𝑒𝑟 (𝑑) − ‖‖𝑒𝑣 (𝑐) − 𝑒𝑣 (𝑑)‖‖ + 𝛾 )) + |‖‖𝑒𝑣 (𝑐)‖‖ − 1 | + |‖‖𝑒𝑣 (𝑑)‖‖ − 1 | (6) | | | | Finally, the third normal form ∃𝑅.𝐶 ⊑ ⊥ indicates that in the vector space translating 𝐶 by 𝑅 results in an unsatisfiable class. We already require the radius of unsatisfiable classes to be zero (Equation 5) and since translation does not change the radius of the original class, we have the following loss function. ∃𝑅.𝐶⊑⊥ (𝑐, 𝑟) = 𝑒𝑟 (𝑐) (7) 3.2.3. Loss Functions for Role Inclusions and Role Chains: The role vectors in our proposed framework serve the purpose of translating one class to an- other class. The constraints considered until now have imposed restrictions on the role vectors based on their relations with the 𝑛-balls of the concerned classes. We now present two loss functions to capture the constraints imposed by role inclusions and role chains in the ontol- ogy. The role inclusion of 𝑅 ⊑ 𝑆 implies that the vectors 𝑒𝑣 (𝑅) and 𝑒𝑣 (𝑆) in the vector space should be nearby because any translation produced by 𝑅 should also be producible by 𝑆 plus both the vectors should be in the same direction. This intuition is captured by the following loss function represented by Equation 8. Herein, the first term is indicative of the distance that ensures the vectors 𝑒𝑣 (𝑅) and 𝑒𝑣 (𝑆) lie in near vicinity of each other. The second term captures the directional aspect of roles in vector space such that they tend to be in same direction. | 𝑒𝑣 (𝑟).𝑒𝑣 (𝑠) || | |‖ | | | 𝑅⊑𝑆 (𝑟, 𝑠) = max (0,‖‖𝑒𝑣 (𝑠) − 𝑒𝑣 (𝑟)‖‖ − 𝛾 ) + |1 −‖ ‖‖ ‖|+ |‖𝑒𝑣 (𝑟)‖‖ − 1 | + |‖‖𝑒𝑣 (𝑠)‖‖ − 1 | (8) | ‖𝑒𝑣 (𝑟)‖‖𝑒𝑣 (𝑠)‖ || | | | | | Next, we consider the hierarchy defined by the role chain 𝑅1 ◦𝑅2 ⊑ 𝑆. In the vector space, this implies that if class 𝐶 can be translated to class 𝐸 by successive application of 𝑅1 and 𝑅2 , it can also be translated to 𝐸 directly by the vector for role 𝑆 while preserving the direction of role vectors. The following loss function captures this behavior represented by Equation 9. | (𝑒𝑣 (𝑟1 ) + 𝑒𝑣 (𝑟2 )).𝑒𝑣 (𝑠) || | 𝑅1 ◦𝑅2 ⊑𝑆 (𝑟1 , 𝑟2 , 𝑠) = max (0,‖‖𝑒𝑣 (𝑠) − 𝑒𝑣 (𝑟1 ) − 𝑒𝑣 (𝑟2 )‖‖ − 𝛾 ) + |1 −‖ ‖‖ ‖| | ‖(𝑒𝑣 (𝑟1 ) + 𝑒𝑣 (𝑟2 ))‖‖𝑒𝑣 (𝑠)‖ || (9) | |‖ | | | | | + |‖𝑒𝑣 (𝑟1 )‖‖ − 1 | + |‖‖𝑒𝑣 (𝑟2 )‖‖ − 1 | + |‖‖𝑒𝑣 (𝑠)‖‖ − 1 | | | | | | | Often, negative sampling is employed during the training phase to learn better embeddings as negative samples can be easily generated to enhance the training data available. In order to incorporate negative samples in the training phase, the following loss function is employed. | | | | 𝑙𝑜𝑠𝑠𝐶⋢∃𝑅.𝐷 (𝑐, 𝑑, 𝑟) = max (0, 𝑒𝑟 (𝑐) + 𝑒𝑟 (𝑑) − ‖‖𝑒𝑣 (𝑐) + 𝑒𝑣 (𝑟) − 𝑒𝑣 (𝑑)‖‖ + 𝛾 ) + |‖‖𝑒𝑣 (𝑐)‖‖ − 1 | + |‖‖𝑒𝑣 (𝑑)‖‖ − 1 | | | | | (10) Thus, the total loss for learning the embedding function is the sum of all the loss functions given by Equations 1- 10. Further, we also add the constraint that radius of the satisfiable classes are non-negative and penalize the total loss for learning negative radius for classes. 3.3. Training and Implementation Given an ++ ontology, we first normalize the ontology to generate the normal forms. These normal forms then constitute as a set of TBox statements wherein each axiom is treated as a positive sample. This normalization is performed using the OWL APIs and the APIs provided by the jCel reasoner which implements the normalization rules [18]. We then introduce nega- tive samples using the third normal form. We randomly generate corrupted axioms following 𝐶 ⊑ ∃𝑅.𝐷, by replacing 𝐶 or 𝐷 with 𝐶 ′ or 𝐷 ′ such that neither 𝐶 ′ ⊑ ∃𝑅.𝐷 nor 𝐶 ⊑ ∃𝑅.𝐷 ′ are asserted axioms in the ontology. Therefore, based on the facts the training process learns ontology embedding such that the facts hold true. The code for training of embeddings and optimization is implemented using Python and Tensorflow library, and Adam optimizer [19] is used for updating the embeddings. We start the learning process by initializing the embedding vectors for classes and relations by random values. We process the training samples in mini-batches for each of the losses defined for the normal forms along with the losses for roles, and update the embeddings depending upon the total loss i.e. the sum of all the loss functions. The update process is carried till saturation or a fixed number of epochs. 4. Experiments and Results 4.1. Datasets We use following four commonly used and publicly available ontologies of varying size and different characteristics. 1. SNOMED CT [20] is one of the most comprehensive ontology of clinical terms with more than 989186 TBox statements involving 307712 classes and 60 relations. Table 1 Different ontologies used in this work and count of different types of axioms. NF𝑖 represents the 𝑖 𝑡ℎ normal form as described in Section 3. Ontology SNOMED ANATOMY GO GALLEN Disjoint 0 184 30 0 Role Inclusion 11 89 3 958 Role Chain 1 31 6 58 NF1 446628 122142 85480 28890 NF2 27779 2121 12131 13595 NF3 482330 152289 20324 28118 NF4 32449 2143 12129 13597 2. Anatomy [21] is an ontology captures linkages of different phenotypes to genes. It consists of 278883 TBox statements involving 106495 classes and 218 relations. 3. Gene Ontology(GO) [22] unifies the representation of gene across all species. It consists of 130094 TBox statements, 45907 classes and 16 relations. 4. GALEN [23] also represents clinical information. It consists of 84537 TBox statements with 24353 classes and 1010 relations. Table 1 presents the four ontologies in the increasing order of their size (number of axioms) and highlights the differences between them in terms of the types of axioms. For instance, we note that SNOMED has 0 disjoint axioms and only 1 role chain axiom, despite being the largest amongst the four ontologies. On the other hand, GALEN, being one of the smaller ontologies, has the highest number of role inclusion (958) and role chain (58) axioms. Also, observe that ANATOMY is the only ontology considered that has the representation of all the EL constructs considered in this work. 4.2. Baselines We consider the following commonly used knowledge graph embeddings for comparison with our proposed EmEL++ embeddings. 1. TransE [1], one of the most frequently used embedding model for knowledge graph, introduced the idea of translation based embeddings where the relations between entities is interpreted as a translation operation between the entities. 2. TransH [2], is an extension of TransE that better handles reflexive, one-to-many, many- to-one, and many-to-many relations. TransH considers relations as hyperplanes in the embedding space. The translation operation is then performed over the projections of entities on the hyperplane. 3. DistMult [3], a matrix factorization based embedding model, has been found empirically to perform well at compositional reasoning tasks. 4. EL Embeddings (ElEm) [7] is one of the first embedding models for the ++ descrip- tion logic based model. Our proposed model is also an extension of ElEm embeddings and enhances ElEm by introducing additional constraints for a more comprehensive cov- erage of ++ description logic. Table 2 Best performing Hyper-parameters for each model. 𝑛 indicates the dimension of embedding vectors and 𝛾 is the margin loss parameter. EmEL++ ElEm TransE TransH DistMult 𝑛 𝛾 𝑛 𝛾 𝑛 𝛾 𝑛 𝛾 𝑛 𝛾 GALEN 50 0.0 50 0.0 100 -0.1 100 -0.1 100 -0.1 GO 100 -0.1 100 -0.1 100 -0.1 100 -0.1 100 0.1 ANATOMY 200 -0.1 200 -0.1 100 -0.1 100 -0.1 100 -0.1 SNOMED CT 100 -0.1 100 -0.1 50 -0.1 50 -0.1 50 -0.1 We use the pykeen framework [24] for implementations of TransE, TransH, and DistMult embedding models. For ElEm embeddings, we used the source code provided by the authors1 . Our implementation of EmEL++ is publicly available at https://github.com/kracr/EmELpp. 4.3. Experimental Protocol For learning the embeddings by different models, we first normalize the ontologies as described in Section 3. Next, we remove 30% of the subclass relation pairs from the normalized ontology to be used for validation (20%) and testing (10%). The remaining ontology with 70% sub-class relation pairs is used as the training set for learning the embedding functions. Further, we take an inferences set that consists of the inferences drawn on the training set using a standard Elk reasoner to evaluate the performance of learned embeddings. We perform hyper-parameter tuning using the 20% validation set and report the performance of fine-tuned models on the test set. The hyper-parameters to tune for all the models are the dimensions of the embedding vectors, and the margin parameter 𝛾 . We consider 𝑛 = {50, 100, 200} and 𝛾 = {−0.1, 0, 0.1} yielding nine different settings. The best performing hyper-parameters for each of the models are reported in Table 2. 4.4. Reasoning Performance of EmEL++ We chose subsumption as the main task to evaluate the effectiveness of the proposed EmEL++ embeddings. Note that once we have embedded the ontologies in a vector space, we have to reduce all the tasks we want to accomplish to operations that can be performed in an 𝑛- dimensional space. We reduce the task of subsumption in the embedding vector space as a distance-based operation. Given a test instance of the form 𝐶 ⊑ 𝐷, we take 𝐷 as our source class and rank all the other classes in the ontology in increasing order of their distance from 𝐷 in the vector space. We then compare the effectiveness of different embedding models based on the rank at which 𝐶 is present in the ranked list. An embedding model that successfully captures the subclass relation between the two classes should be able to assign vector representations to the two classes that are very close to each other, hence, producing a lower rank for 𝐶. Table 3 summarizes the performance of embedding models for test and inferences set. We report and evaluate the performance using six metrics. Hits at ranks 1, 10 and 100 report the fraction of test cases for which the expected class was found within top 1, 10 and 100 ranks, respectively. A median rank of 𝑚 means that for 50% of the test cases, the correct answer was 1 https://github.com/bio-ontology-research-group/el-embeddings Table 3 Embedding Models Ranking based Performance on Test and Inferences Data Test Data Inferences Data Metric TransE TransH DistMult ElEm EmEL++ TransE TransH DistMult ElEm EmEL++ Hits@1 0.00 0.00 0.00 0.01 0.02 0.00 0.00 0.00 0.06 0.07 Hits@10 0.00 0.00 0.00 0.08 0.11 0.00 0.00 0.00 0.18 0.23 GALEN Hits@100 0.00 0.00 0.00 0.15 0.16 0.00 0.00 0.00 0.31 0.35 AUC 0.54 0.48 0.51 0.64 0.65 0.61 0.47 0.50 0.84 0.85 Median Rank 10748 11721 12600 6658 6623 8536 12773 12111 750 585 90th %ile Rank 21308 21825 21823 19681 20635 18871 21890 22029 10627 10339 Hits@1 0.00 0.00 0.00 0.01 0.01 0.00 0.00 0.00 0.06 0.07 Hits@10 0.00 0.00 0.00 0.08 0.09 0.00 0.00 0.00 0.30 0.32 Hits@100 0.00 0.00 0.00 0.23 0.24 0.00 0.00 0.00 0.49 0.49 GO AUC 0.53 0.44 0.50 0.73 0.78 0.83 0.44 0.50 0.86 0.90 Median Rank 20079 26280 22493 4302 2908 3718 26015 22977 113 122 90th %ile Rank 40177 41996 40425 41225 33547 20791 41938 40707 26090 17527 Hits@1 0.00 0.00 0.00 0.04 0.04 0.00 0.00 0.00 0.26 0.27 ANATOMY Hits@10 0.00 0.00 0.00 0.19 0.17 0.00 0.00 0.00 0.71 0.65 Hits@100 0.01 0.00 0.00 0.41 0.39 0.01 0.00 0.00 0.84 0.79 AUC 0.53 0.44 0.49 0.73 0.76 0.68 0.46 0.49 0.97 0.96 Median Rank 46679 61349 52486 640 381 29336 58987 53690 3 3 90th %ile Rank 91235 96192 93619 105111 105274 66940 93177 94823 1501 2557 Hits@1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.08 0.07 SNOMED CT Hits@10 0.00 0.00 0.00 0.03 0.03 0.00 0.00 0.00 0.30 0.28 Hits@100 0.00 0.00 0.00 0.08 0.06 0.00 0.00 0.00 0.42 0.37 AUC 0.50 0.48 0.50 0.63 0.64 0.56 0.48 0.50 0.81 0.83 Median Rank 150876 157186 151624 80289 87413 125431 157284 153020 1704 3321 90th %ile Rank 274465 278455 275982 277874 261359 253452 279666 275345 220771 192339 found below rank 𝑚. 90𝑡ℎ percentile rank denotes the rank value below which the correct class was found for 90% of the test cases. The first observation that we make from Table 3 is that ElEm and EmEL++ embeddings perform better than the three commonly used KG embeddings (TransE, TransH, and DistMult). This observation highlights the inadequacy of traditional KG embeddings that do not consider the ontological constructs and rely only on the structural properties of the underlying graph. Both ElEm and EmEL++ embeddings incorporate specific constraints and charactersitics of ++ description logic, and hence, the embeddings produced by these models are better at retaining the properties of the underlying ontologies in the vec- tor space. Next, we note from the two tables that there is no clear winner among ElEm and EmEL++ . While EmEL++ outperforms ElEm for the GALEN and GO ontologies, there is no clear winner for Anatomy and SNOMED ontologies as their performance varies. This observation is consistent with previous empirical studies comparing different link prediction methods that found that no single method outperforms across a variety of datasets [25, 26]. We speculate that this divergence in performance could be attributed to the different distributions of the types of axioms in the ontology (ref. Table 1). Over (or under) representation of certain types of axioms may lead to the optimization process giving more (or less) weight to the correspond- ing loss functions during the training phase. Understanding the exact mechanism behind the performance characteristics of different ontologies is a crucial and challenging area of future research. Table 4 Accuracies achieved by the ElEm and EmEL++ embeddings in terms of geomteric interpretation of the classes in different ontologies. Training Testing Inferences ElEm EmEL++ ElEm EmEL++ ElEm EmEL++ GALEN 0.27 0.64 0.20 0.53 0.27 0.64 GO 0.45 0.59 0.35 0.44 0.48 0.62 ANATOMY 0.09 0.48 0.07 0.22 0.10 0.49 SNOMED CT 0.24 0.55 0.18 0.34 0.22 0.48 4.5. Preserving Ontology Characteristics in Vector Space Next, we compare the ElEm and EmEL++ in terms of retaining the underlying characteristics of ontology in vector space. Recall that both the models map the classes in an ontology to 𝑛-balls in the vector space. Further, the mapping is such that the 𝑛-ball of a super-class subsumes the 𝑛-balls of its sub-classes. Thus, for a test instance 𝐶 ⊑ 𝐷, we check that the 𝑛-ball of class 𝐶 lies inside the 𝑛-ball of class 𝐷 in the vector space. Note that since we have the centers and radii of the corresponding 𝑛-balls, this can be checked easily. Table 4 presents the training, testing, and the inference accuracy obtained for the two embedding models for this task. We report accuracy values, i.e., the fraction of instances where the subsumption relation between the classes was maintained in the vector space. Note that this is a much stricter criterion for even if the subclass 𝑛-ball is slightly outside the 𝑛-ball of the superclass, it will be considered a failure. We observe from Table 4 that EmEL++ outperforms the ElEm embeddings for all the datasets and across all settings. This indicates that EmEL++ embeddings are better at preserving the class relationships in the mapped vector space than ElEm embeddings. 5. Conclusions We proposed EmEL++ , an embedding model for ++ ontologies. EmEL++ builds upon and extends the previously proposed ElEm embeddings by focusing on role inclusions and role chains and offers a more complete coverage of ++ . Experiments with four different ontolo- gies showed that EmEL++ outperforms traditional KB embeddings on the subsumption rea- soning task. Further, when compared with ElEm embeddings, it is able to better preserve the underlying semantics of the ontologies in the vector space. We have also shown how to per- form the subsumption reasoning task in a vector space, which is an 𝑂(𝑛) operation in the worst case. We believe this is an important capability and it offers exciting directions for future work. References [1] A. Bordes, N. Usunier, A. Garcia-Duran, J. Weston, O. Yakhnenko, Translating embeddings for modeling multi-relational data, in: Advances in neural information processing systems, 2013, pp. 2787–2795. [2] Z. Wang, J. Zhang, J. Feng, Z. Chen, Knowledge graph embedding by translating on hyperplanes, in: Twenty-Eighth AAAI conference on artificial intelligence, 2014. [3] B. Yang, W.-t. Yih, X. He, J. Gao, L. Deng, Embedding entities and relations for learning and inference in knowledge bases, arXiv preprint arXiv:1412.6575 (2014). [4] M. Nickel, V. Tresp, H.-P. Kriegel, A three-way model for collective learning on multi-relational data., in: Icml, volume 11, 2011, pp. 809–816. [5] T. Trouillon, J. Welbl, S. Riedel, É. Gaussier, G. Bouchard, Complex embeddings for simple link prediction, International Conference on Machine Learning (ICML), 2016. [6] Y. Lin, Z. Liu, M. Sun, Y. Liu, X. Zhu, Learning entity and relation embeddings for knowledge graph completion, in: Twenty-ninth AAAI conference on artificial intelligence, 2015. [7] M. Kulmanov, W. Liu-Wei, Y. Yan, R. Hoehndorf, EL embeddings: geometric construction of mod- els for the description logic el++, in: Proceedings of the 28th International Joint Conference on Artificial Intelligence, AAAI Press, 2019, pp. 6103–6109. [8] F. Baader, S. Brandt, C. Lutz, Pushing the EL Envelope, LTCS-Report LTCS-05-01, Chair for Au- tomata Theory, Institute for Theoretical Computer Science, Dresden University of Technology, Germany, 2005. See http://lat.inf.tu-dresden.de/research/reports.html. [9] V. Misra, S. Bhatia, Bernoulli embeddings for graphs, in: Thirty-Second AAAI Conference on Artificial Intelligence, 2018. [10] A. Grover, J. Leskovec, node2vec: Scalable feature learning for networks, in: Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining, 2016, pp. 855–864. [11] D. Garg, S. Ikbal, S. K. Srivastava, H. Vishwakarma, H. Karanam, L. V. Subramaniam, Quantum Embedding of Knowledge for Reasoning, in: Advances in Neural Information Processing Systems, 2019, pp. 5595–5605. [12] F. Z. Smaili, X. Gao, R. Hoehndorf, Onto2vec: Joint vector-based representation of biological entities and their ontology-based annotations, Bioinformatics 34 (2018) i52–i60. [13] Ö. Özçep, M. Leemhuis, D. Wolter, Cone semantics for logics with negation, in: Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI, 2020, pp. 1820–1826. [14] A. Eberhart, M. Ebrahimi, L. Zhou, C. Shimizu, P. Hitzler, Completion reasoning emulation for the description logic el+, arXiv preprint arXiv:1912.05063 (2019). [15] F. Baader, S. Brandt, C. Lutz, Pushing the EL envelope, in: IJCAI, volume 5, 2005, pp. 364–369. [16] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, J. Dean, Distributed representations of words and phrases and their compositionality, in: Advances in neural information processing systems, 2013, pp. 3111–3119. [17] P. Ristoski, H. Paulheim, Rdf2vec: Rdf graph embeddings for data mining, in: International Se- mantic Web Conference, Springer, 2016, pp. 498–514. [18] J. Mendez, jcel: A Modular Rule-based Reasoner., in: ORE, 2012. [19] D. P. Kingma, J. Ba, Adam: a method for stochastic optimization. CoRR abs/1412.6980 (2014), 2014. [20] K. Donnelly, SNOMED-CT: The advanced terminology and coding system for ehealth, Studies in health technology and informatics 121 (2006) 279. [21] C. J. Mungall, C. Torniai, G. V. Gkoutos, S. E. Lewis, M. A. Haendel, Uberon, an integrative multi- species anatomy ontology, Genome biology 13 (2012) R5. [22] G. O. Consortium, The Gene Ontology (GO) database and informatics resource, Nucleic acids research 32 (2004) D258–D261. [23] A. Rector, J. Rogers, P. Pole, The GALEN high level ontology (1996). [24] M. Ali, H. Jabeen, C. T. Hoyt, J. Lehmann, The KEEN Universe, in: International Semantic Web Conference, Springer, 2019, pp. 3–18. [25] D. Liben-Nowell, J. Kleinberg, The link-prediction problem for social networks, Journal of the American society for information science and technology 58 (2007) 1019–1031. [26] L. Lü, T. Zhou, Link prediction in complex networks: A survey, Physica A: statistical mechanics and its applications 390 (2011) 1150–1170.