Introduction

Ontologies, knowledge graphs in the majority of domains have a taxonomy as a backbone. Relations in taxonomies usually comprise class-subclass relations between concepts, or instance-class relations connecting a specific entity representation and a concept [3,13]: relations of both types can be called IS-A relations or hypernym relations [24]. Development of an ontology in a new domain usually begins from constructing its taxonomy, which determines the ontology scope.

It make it easier to build a taxonomy, various approaches were proposed for extracting hypernym relations for new terms from texts including specific patterns, word co-occurrences, distributional characteristics of words, and others [27]. Currently, the important component of extracting hypernym relations from texts are vector representations (embeddings) of words, which can provide an additional evidence of semantic similarity between words [12,18,28], which is important for identification of hypernym concept for a new term.

Word vectors can be calculated using various text collections and various methods, which means that different vector representations capture the context in different ways, resulting in a wide variety of vector representations for the same words. From here, we can be suppose that some combinations of vectors, so-called meta-embeddings [8], can improve vector representation of words, which allows achieving better prediction of semantic similarity between words or their hypernym concepts.

It was already shown that meta-embeddings improved performance in word analoqy and similarity tasks [41,8,7]. In recent work [38], it was shown that combinations of general word embeddings calculated on large Internet text collections have substantial impact on the performance in taxonomy enrichment of general lexical-semantic resources such as WordNet [24] and RuWordNet [20].

In this paper we show that in the task of extracting taxonomic relations in a specific domain, meta-embeddings combining general (out-of-domain) and indomain embeddings significantly improve the hypernym concept prediction from the given taxonomy for a new term. We experiment on an information-security text collection, which is used to enrich the Ontology on Natural Sciences and Technologies [11,36] in the information-security domain.

Related Work

Traditional methods for hypernym detection include pattern-based methods, searching for specific hypernym patterns in sentences [15,30,31], methods based on similarity of word vector representations [12,18], and also combined approaches integrating various context and similarity features of words [35,4,34].

In 2016, the taxonomy enrichment task was organized as a shared task at SemEval workshop (task 14) [16]. At this task, the participants should to attach words with definitions to correct hypernyms in WordNet [24] using their definitions. In 2020, a new open evaluation on taxonomy enrichment of the Russian wordnet RuWordNet [20] RUSSE'2020 was organized [27]. The task was to find correct hypernyms from an older RuWordNet version for words described in a newer RuWordNet version.

In the RUSSE-2020 evaluation of predicting RuWordNet hypernym synsets for new words [27], the participants used various word embeddings (static -fastText [5], word2vec [21], and contextualized -BERT [10]), the available Ru-WordNet taxonomy structure, hypernym and co-hyponym patterns, definitions of words from Wiktionary, and global search engines results [2,9,37,27].

Recent methods to hypernym extraction exploit graph-based representations of taxonomy structure. Liu et al. [19] use node2vec embeddings of graph structures [14] for taxonomy induction. Aly et al. [1] use hyperbolic Poincare embeddings [26] for automatic generation of taxonomies. Graph convolutional networks (GCNs) [17] are applied to the link prediction task on large knowledge bases. In [29], the authors study graph-based representation methods on the Diachronicwordnets dataset, which contains several English and Russian WordNet versions and correct answers of links of words from newer versions to concepts of older versions.

Most current approaches of taxonomy enrichment are based on vector representations of new words and existing concepts in a taxonomy [28]. To improve vector representations, the combined approaches of some source vector representations such as vector concatenation or averaging can be used [8]. In [41] it was shown that using singular value decomposition (SVD) over concatenation of several source vectors can improve the results in several tasks with the ability to control the final vector size.

Autoencoders [7], called Autoencoded Meta-Embeddings (AEME), became a further development of the idea of creating meta-embeddings. In [7], the authors proposed several algorithms (CAEME, AAEME and etc.) for combination various word vectors in one vector by encoding initial vectors in some metaembedding space and then decoding backward. The CAEME approach tries to reproduce source vectors from the concatenation of encoded representation of these vectors. In the AAEME approach each vector is mapped to a fixed-size vector and all encoded representations are averaged, but not concatenated, which restricts the vector dimension.

In [25] the authors investigated the performance of the autoencoders depending on the loss function (MSE loss, KL-divergence loss, cosine distance loss and also their combinations). They found that there is no evident winner across tasks and that different loss functions should be chosen for different applications.

In [38] the best results for enriching taxonomies of general lexical-semantic resources such as WordNet [23] and RuWordNet [20] were achieved using AAEME encoders with triplet loss, combining fastText, glove and word2vec embeddings in a single meta-representation. The meta-representation was further used for training of a supervised model, which also included features from Wiktionary.

In the current paper we study the performance of meta-embedding approaches in domain-specific taxonomy enrichment: we experiment with assigning new terms from the information-security domain to Ontology in natural sciences and technologies (OENT).

OENT Ontology

We study the task of domain-specific taxonomy enrichment using Ontology of Natural Sciences and Technologies (OENT) [11,36]. The OENT ontology [11] is presented as a semantic net of concepts and relations between them, each concept is connected with the set of words and phrases, which can express this concept in documents (text entries). All text entries of the same concepts can be called a synset similar to WordNet synsets [23]. For example, "Mathematical analysis" concept has Russian text entries (synset) as matematicheskii analis, matanalis, matan. Synsets in OENT can include different parts of speech: nouns, adjective, verbs, or adverbs.

The OENT ontology is used for automatic document analysis in informationanalytical systems, which includes providing of conceptual search, query expansion using the ontology, knowledge-based document categorization etc [36].

OENT comprises large volumes of concepts and terminology from several scientific disciplines and technological domains presented as a connected semantic network of concepts with corresponding text entries and relations between concepts [11]. The ontology was started from extracting terms from specialized text collections (web-sites, school and university text books) in mathematics, physics, geology, biology, and chemistry. Currently, this initial terminology is collected in the OENT subset called OENT-lite.

In further projects, the available conceptual structures were elaborated to more specific levels, also the terminologies of technological domains such as oil-and-gas industry, power energy, education policy and techniques, computer technologies, and information security were added to OENT. The full version of OENT consists of 106K concepts and 308K single and multi-word terms, while the OENT-lite consists of 37K concept and 133K terms.

In the current study we use the OENT-lite ontology and the terminology of the Information security domain to study the enrichment of the ontology with a domain-specific terminology via extracting hypernym concept relations from domain-specific text collections.

Taxonomy Enrichment Task

The task of taxonomy enrichment consists in finding an appropriate concept from a given taxonomy for a new word, which can be considered as a hypernym or class for this word. Similar to RUSSE-2020 evaluation, the task is to find a direct (the closest) hypernym concept from the taxonomy [28]. To make the extraction less restricted but to keep it quite precise, the second order hypernym concepts (hypernyms of hypernyms) are also considered as correct answers. In this, we try to simulate the work of knowledge engineers, which should find the most specific concept in a taxonomy to attach a new domain term.

The taxonomy enrichment is treated as a ranking task where the correct answers should be in the top of a candidate list [28]. In contrast to the classification task, the ranking is a more appropriate setting in conditions when the share of correct answers is much smaller that the overall number of candidates.

As a subject area for taxonomy enrichment, the information security domain was chosen, represented by a corpus of 500 thousand texts. For this corpus, a frequency list was built so that each word occurs at least 50 times.

The OENTCyber dataset for evaluation hypernym detection was constructed as follows:

1. All one-word text entries of concepts from the OENT ontology were selected so that they appear in the full version, but are absent in the OENT-lite version; 2. From this list, only words for which the hypernyms are present in the OENTlite were taken.

As a result, a dataset of 4372 words was obtained, and the task was to predict hypernyms (OENT concepts) for a given set of words using OENTlite and the available corpus of articles from information-security domain. This dataset contains specific names such as "chrome", "amazon", "cisco", etc and specific terms such as "css3", "dbscan", "dll", etc.

Method of Hypernym Concept Prediction

In our approach, we use word embeddings to generate a list of most similar taxonomy entries (words or phrases from the taxonomy) to the target word according to cosine similarity. For each target word, the top 20 taxonomy entries are considered. The number of elements for consideration was chosen experimentally [38]. For each entry in the similarity list, all corresponding concepts, their direct and second-order hypernyms are extracted from the taxonomy. They are considered as candidate concepts to be hypernyms of the target word.

For candidate hypernym concepts, several features are calculated. Logistic regression is used to predict the probability of a candidate to be a hypernym of the target word. The calculated features are as follows:

the minimum, average, and maximum similarities of the target word to all words of the concept synset; the features based on hyponyms of a candidate concept synset:

• we extract all hyponyms (lower classes) of the candidate concept;

• for each word/phrase in each hyponym synset we compute their similarity to the target word; • we compute the minimum, average, and maximum similarity for each hyponym synset; • we form three vectors: a vector of minimums of similarities, average similarities, and maximum similarities of hyponym synsets; • for each of these vectors we compute minimum, average, and maximum.

We use these resulting 9 numbers as features. the minimum, average, and maximum similarity level of the concept in the merged candidate list:

• the level is 0 if the concept was added based on similarity to the target word; • the level of 1 is for the immediate hypernyms of the word in the similarity list; • the level of 2 is for the hypernyms of the hypernyms of words in the similarity list. the number of occurrences (n) of the concept in the merged candidate list and the quantity log 2 (2 + n) which serves for smoothing.

In total, 17 features were calculated. Training data were generated randomly and automatically from the OENT-lite (thus, the data do not contain test data).

We use two models for calculating word embeddings: fastText [6] and word2vec [22]. In order to obtain vectors for words absent in embedding models, the following procedures were carried out:

-For FastText, embeddings if not in the vocabulary, are obtained in a natural way, by calculating vectors by the model itself; -For Word2Vec, embeddings are calculated by averaging the vectors of maximum prefixes for the constituent words of a multi-word expressions. There is a limitation on the minimum length of a prefix word, which is 4 characters; -For meta-embeddings, if there is no vector for a word in any model, the corresponding source vector is initialized with zeros.

6 Meta-Embeddings in Taxonomy Enrichment Task

In our work we compare simple meta-embeddings such as concatenation of source embeddings and SVD over the concatenation and two variants of autoencoders generating meta-embeddings: Concatenated Autoencoded Meta-Embeddings (CAEME) and Averaged Autoencoded Meta-Embeddings (AAEME), which have shown good results in previous works [7,38]. Suppose we have two source embeddings s 1 (w) and s 2 (w), their encoders E 1 (w) and E 2 (w) and their decoders D 1 (w) and D 2 (w). Meta-embedding m(w) in CAEME is constructed as the L 2 -normalised concatenation of two encoded source embeddings E 1 (s 1 (w)) and E 2 (s 2 (w)):

m(w) = E 1 (s 1 (w)) ⊕ E 2 (s 2 (w)) ||E 1 (s 1 (w)) ⊕ E 2 (s 2 (w)|| 2 , (1)

where ⊕ is the concatenation operation.

In CAEME, the dimensionality of the meta-embedding space is the sum of the dimensions of the source embeddings. The AAEME encoder can be seen as a special case of the CAEME encoder, where the meta-embedding is computed by averaging the two encoded sources in (1) instead of their concatenation. Averaging gives the possibility to avoid increasing the dimensionality of the meta-embedding.

The AAEME encoder computes the meta-embedding of a word w from its two source embeddings s 1 (w) and s 2 (w) as the L 2 -normalised sum of two encoded versions of the source embeddings E 1 (s 1 (w)) and E 2 (s 2 (w)):

m(w) = E 1 (s 1 (w)) + E 2 (s 2 (w)) ||E 1 (s 1 (w)) + E 2 (s 2 (w))|| 2 . (2)

The CAEME and AAEME decoders reconstruct the source embeddings from the same meta-embedding m(w), thereby implicitly using both common and complementary information in the source embeddings.

The overall objective of autoencoder training is given below. Function f can be any distance or similarity measure as MSE, KL-divergence, or cosine distance.The coefficients λ 1 and λ 2 can be used to give different emphasis to the reconstruction of the two sources.

Loss w (E 1 , E 2 , D 1 , D 2 ) = w (λ 1 f s1(w),ŝ1(w) + λ 2 f s2(w),ŝ2(w) ),(3)

where ŝi (w) -decoded embeddings corresponding to s i (w). Jointly learning of E 1 , E 2 , D 1 , D 2 minimises the total reconstruction error given by Equation 3. To obtain meta-embedding representations after training, only the encoders are applied, which convert the input source embeddings into a meta representation. Further, these meta-embedding vectors are used as vector representations of words.

The standard loss function for AEME approaches we used was cosine distance loss. We have tried variations and combinations between MSE loss, KL divergence loss and cosine distance loss, and last one works best in our case.

We can impose additional restrictions on AEME models during training. One of such restrictions is the use of triplet loss.

The triplet loss function is a loss function for machine learning algorithms where some basic example (anchor) is compared with positive and negative examples. The goal is to minimize the difference in distance between base and positive examples and base and negative examples. In this case, there is often some margin parameter that controls how much the distance to the negative example is greater than to the positive one. One of the first formulations of the triplet loss equivalent approach was introduced in [33] for the metric learning problem. The use of a similar loss function for modifying algorithms has also been used in the problems of image similarity [39], face recognition [32], text classification [40] and other tasks.

We restrict a word to be closer to the words that are semantically related to it according to the taxonomy than to a randomly chosen word with some margin:

L(w a , w p , w n ) = max(||m(w a ) − m(w p ))||− ||m(w a ) − m(w n ))|| + margin, 0), (4)

where ||.|| is a distance function, w a is the target word, w p and w n are positive and negative words, respectively. The algorithm of calculating triplet loss is as follows:

1. for each word presented in the taxonomy, we compile a list of semantically related words which includes synonyms, hyponyms and hypernyms; 2. at each epoch, we randomly select K positive words from this related words set and form a set of K negative words by selecting them randomly from the vocabulary; 3. if the word is not presented in the taxonomy, then we cannot form a list of related words for it. In this case, we generate positive vectors for it by adding random noise to its vector; 4. next, we calculate the triplet margin loss by combining the triplet loss with the original loss as α * loss + (1 − α) * triplet loss.

We use the following parameters for the triplet loss: K = 5, margin = 0.1, alpha = 0.005. These parameters were selected via grid search with AAEME algorithm.

Experiments

The quality of the approach was evaluated using two general (external) source vector representations: fastText 1 and word2vec 2 . Also different meta-embedding approaches were investigated: concatenation, SVD over concatenation, CAEME, AAEME (with and without triplet loss).

In addition to the two "external" vector models word2vec and fastText, two vector models (word2vec and fastText, respectively) were trained on the information security text corpus (hereinafter "internal"). The training parameters were as follows: window = 3, vector size = 300, epochs = 10, method = skip-gram. The performance of models trained on the domain corpus and also their combination with more "powerful" models was investigated, since 500 thousand texts is significantly less compared to text collections on which external word2vec and fastText models were trained.

For evaluation of hypernym prediction, a traditional measure for ranking tasks Mean Average Precision measure is used. This measure achieves the maximal value equal 1 when all correct answers are located in the beginning of a ranking list:

M AP = 1 N N i=1 AP i ; AP i = 1 M n i prec i × I[y i = 1].(5)

Another traditional metric for such tasks is Mean Reciprocal Rank (MRR), depending on positions of the first correct answers. This measure is equal to maximal value 1, when all first correct answers are located on 1st positions in ranking lists for all target words.

M RR = 1 N N i=1 1 rank i , (6)

where rank is the position of the first relevant item in the ranked list.

Where N and M are the number of predicted and ground truth values, respectively, prec i is the fraction of ground truth values in the predictions from 1 to i, y i is the label of the i-th answer in the ranked list of predictions, and I is the indicator function.

In order to evaluate the quality of the approach in such a setting, the described methods of constructing meta-embeddings were used, and each vector model was evaluated separately.

In case of using the AEME approaches for all vector models, it was necessary to determine the individual contributions of each vector model when calculating the loss function. The following weights were obtained experimentally: the weight of 1.0 for the internal models trained on the corpus, the weight of 5.0 for the external word2vec model, and the weight of 2.0 for the external fasttext model. The results can be seen in Table 1 (external models), Table 2 (internal models), and Table 3 (combination of external and internal models).

From Tables 1 and 2, we can see that powerful external models calculated on large text collections still predict hypernym concepts much better than internal, domain-specific models. In both cases all variants of meta-embeddings better predict hypernyms than source vectors. The best results are achieved by encoders with triplet loss. The combination of all models (Table 3) achieves the best results in the hypernym concept prediction. The best prediction results are much higher than the prediction on any of source models. The results achieved by encoders are much better than the results of simple approaches (concatenation and SVD).

Analysis of Results

We analysed hypernym predictions for new words for which correct predictions were not found in the Top-10 of correct answers and found the following cases:

-Predicted hypernyms correspond to senses missed in the taxonomy. For example, word "halo" is described in OENT-lite only in the sense of Russian helicopter, but also this word can mean a computer game. Predicted hypernyms include the concept "computer program" at the first position of the candidate list and the "game" concept at eighth position. -A predicted hypernym concept is quite valid and convey another aspect of a target word. For example, for verb "to cache", the correct answer in OENT is concept "data storing", but the first predicted hypernym concept "computer technology" seems also correct; -Predicted hypernyms may be too general in OENT, but more specific in predictions, for example for word "amazon" the correct answers are concepts: "American company", "company", "foreign company". The predicted concepts contain such concept as "American software company", "American tech company". These predicted concepts seem to be more correct; -In many cases predictions are very semantically close to correct answers but not correct. For example, for word "CSS3" the correct answers are concepts "document markup language", "formal language". The predicted hypernym concepts are "programming language", "scripting language", "object-oriented language", "computer technologies"; there are also numerous examples when too general hypernyms are predicted; in some cases predicted conceptr are very far from reasonable answers and are difficult for explanation.

To show that the higher confidence of the model correlates with better results of hypernym concept prediction, we calculated the plots of dependence of correct answers on the weight of the first prediction. Figure 1 shows the proportion of a correct hypernym concept among first 1, 3, 5, and 10 answers depending on the prediction weight. It can be clearly seen that the higher weight of a predicted hypernym leads to the higher proportion of correct answers. This means that model predictions with high predicted weights and absence of correct answers in the top can be considered as a source of improving hypernym class descriptions in the ontology.

Conclusion

In this paper we considered the problem of adapting the OENT ontology to a specific domain of information security: for new words from an information-security text collection, a hypernym concept from the OENT ontology has to be predicted. We investigated methods for combining different word embeddings in a single meta-embedding. The meta-embeddings methods included concatenation of initial embeddings, SVD over the concatenation, two variants of autoencoders aimed to learn better word embeddings from initial vectors. We showed that the use of meta-embeddings improves the performance of the system for the considered datasets. SVD always improves the results compared to concatenation. Autoencoder-based meta-embeddings achieve the best results in all cases. It can also be seen that adding the triplet loss improves the results significantly.

It has been also shown that the use of vector models trained on specific domain in combination with the meta-embedding approach can improve the quality of hypernym concept prediction. It can also be seen that the quality of the approach on the specific domain is worse then on general domain [38]. We plan to make OENT-lite and the related hypernym dataset publicly available.