This paper provides an insight into the possibility of how to find ontologies most relevant to scientific texts using artificial neural networks. The basic idea of the presented approach is to select a representative paragraph from a source text file, embed it to a vector space by a pre-trained fine-tuned transformer, and classify the embedded vector according to its relevance to a target ontology. We have considered diferent classifiers to categorize the output from the transformer, in particular random forest, support vector machine, multilayer perceptron, k-nearest neighbors, and Gaussian process classifiers. Their suitability has been evaluated in a use case with ontologies and scientific texts concerning catalysis research. From results we can say the worst results have random forest. The best results in this task brought support vector machine classifier.
can help in classifying the information presented within A domain ontology defines a set of representational prim- the text as well as to connect the input text with data. This itives with which to model a domain of knowledge or would allow for automated selection of ontologies and discourse. The representational primitives are typically respective classification of the text. Diferent text data classes, attributes, and relationships. The definitions of could thus be compared automatically in an understandthe representational primitives include information about able way and connected with corresponding research their meaning and constraints on their logically consis- data. Ontologies represent "a formal specification of a tent application. Classes can be defined in two ways: shared conceptualization" [7] and can thus be used to by annotating their definitions, or by connecting classes express knowledge and data in a formalized, standardwith each other and with properties. Each domain ontol- ized description language to specify terms and relations ogy typically uses domain-specific definitions of terms between those terms. denoting its primitives. Current ontology recommenders, such as the NCBO
The FAIR research data management (Findable, Access- ontology recommender [8], score annotations based on
able, Interoperable, and Reuseable) needs a consistent words similar to preferred and alternate labels of
ontoldata representation in ontologies, particularly for rep- ogy classes and term frequency. In contrast to this, this
resenting the data structure in the specific domain [
With regard to sources we have studied described in copying input words as extralogical symbols to the outpart 2 of this article, we are not aware that classifiers put, as well as the ability to enrich the training set with learned from the results of representational learning have new annotated examples. In [23], a mapping is estabever been used to determine the most relevant of a given lished between ontologies and a pair of interacting asset of ontologies. sociative memories. One of them stores assertions, and the other stores entailment rules. The most recent work [21] describes a method for the specific task of extracting 2. Applicability of Artificial Neural a taxonomy from an embedding of a knowledge graph.
Networks to Ontologies Over that embedding, which can be obtained for
example with RDF2Vec, hierarchical agglomerative clustering
In connection with learning and extending ontologies, ar- is performed, first without using type information, and
tificial neural networks (ANNs) have been primarily used then injecting types into the hierarchical clustering tree.
for identification of concepts, relations and attributes In addition, an axiom induction algorithm is applied to
[10, 15, 18]. With respect to relations, some ANN-based each cluster in the resulting tree, which allows to identify
methods have been developed specifically for subsump- new classes corresponding to those axioms that describe
tion relations needed for the construction of taxonomies their respective clusters accurately enough.
[11, 14, 21, 30]. In connection with integration of on- Neural networks are often used due to their strengths
tologies, they have been primarily used for ontologies in natural language processing task. Ontology
construcmatching aka ontologies alignment [
Closest to the proposed project is the way ANNs have printer, thus they contain meta-information about the been recently used in connection with translating into contained text related to the position on the page. ThereOWL [27, 28], with predicate chaining and restriction fore, it is not easy to address a single paragraph. If the [23], and with taxonomy extraction from knowledge ifle is read using the basic library for PDF files and the graphs [21]. In [27], ontology learning is tailored as a newline mark is used as the splitter, it returns only a transductive reasoning task that uses two recurrent neu- single row, not the whole paragraph. Another issue is ral networks to translate text in natural language into connected with multi-column documents. If the docuOWL specifications in description logic. That approach ment does not include information about where the text was further developed in [28], resulting in a system based continues, software libraries for text extraction from PDF on a single recurrent network of GRU type. It uses an usually continue with the next letter on the same row.
One solution to get text data from multi-column PDF is to use Microsoft Word engine. Its engine is able to solve both problems and parse text properly. It identifies structural information in text such as headings, paragraphs and sentences. Each document may contain texts irrelevant to the topic of interest, for example references, acknowledgement etc.
Specifications of the ontologies are most often stored in OWL files. OWL [ 2] is a specific kind of XML for ontologies. Text that describes classes and relations may be stored in diferent tags, depending on the decision of the ontology designer.
The basic schema of BERT is given in Figure 1. The tokenized input at first passes through the encoder, which embeds sentences to elements of an Euclidean space. These vectors are used as input to the BERT decoder. BERT returns one vector for each input. Each input sequence contains a special token at the beginning marked as CLS. Vectors embedding the tokens of an input sequence can be arranged into a matrix. The ifrst row of the matrix is the embedding of the whole input. Details of BERT are described in [3] and on the https://huggingface.co/docs/transformers/model_doc/bert. These embeddings of every input paragraph are taken into account for the final assignment of the most relevant ontology to the paragraph.
tering, it is suitable to represent words or other parts of The embeddings obtained in BERT are used as inputs for text by vectors in an Euclidean space. Such representa- classifiers classifying a given input part of text (e.g., a tion is mostly the result of representation learning by paragraph) with respect to its relevance to the considANNs. In the area of text analysis and processing, the ered ontologies. Those classifiers have been trained on probably most successful representation learning algo- the embeddings of the annotations from the considered rithm is BERT (Bidirectional Encoder Representations ontologies because for them, the ground truth (i.e., the from Transformers)[3]. ontology to which the annotation belongs) is known.
BERT needs to be trained using large amount of texts. We have decided to select five classifiers implemented That is why some pretrained version is typically used, in scikit-learn [4]. They are the following: and often subsequently fine-tuned using texts concerning the considered topic. Such fine-tuning is often performed even if the pretrained network was trained, apart from general texts, also with texts from some broader relevant domain (biology, medicine, chemistry, etc.). 1. Random forest (RF): An ensemble classifier that ifts a number of classification trees on various sub-samples of the training data and uses some aggregation function to improve the predictive accuracy and control over-fitting. Usually, each tree in the ensemble is built using a sample drawn with replacement (i.e., a bootstrap sample) from the training set. Furthermore, when splitting each node during the construction of a tree, the best split is found using either all input features or a random subset of a given size. The purpose is to decrease the variance of the forest estimator.
Indeed, individual decision trees typically exhibit
high variance and tend to overfit. The injected
randomness in forests yield decision trees with
somewhat decoupled prediction errors. By taking
an average of those predictions, some errors can
cancel out. RFs achieve a reduced variance by
combining diverse trees, sometimes at the cost of
a slight increase in bias. Typically, the variance
reduction yields an overall better model [
non-linear models and capability to learn models
in real-time (on-line learning). But the MLP with
hidden layers have a non-convex loss function
where there exists more than one local minimum.
Therefore, diferent random weight initializations
can lead to diferent validation accuracy. A MLP
requires tuning a number of hyperparameters
such as the number of hidden neurons, layers,
and iterations. Moreover, it is sensitive to feature
scaling [
process of a chemical reaction. Using a catalyst in a chemical reaction usually allows said reaction to take place faster and allows for more moderate reaction conditions. Catalysis-based chemical synthesis is applied at roughly 90% of chemical processes in chemical industry. The scientific domain of catalysis is highly interconnected to other sciences and thus spans over many topics from material sciences to process design [5, 6].
The texts that have been used for fine-tuning BERT, have been taken from scientific papers in catalysis. These articles have been by PowerShell script extracted to Microsoft Word documents. Thanks to its engine, paragraphs and titles are marked properly, so paragraphs with relevant texts have been extracted and with BERT embedding prepared for classification.
We conduct our experiments on a set of five ontologies
from the chemical domain (Table 1) gathered within the
NFDI4Cat project [
Hence, these ontologies are classes to which classifiers assign new parts of text. The data have been divided into training and testing datasets in stratified proportion 1:1. The testing dataset has been divided into 20 disjoint subsets, assuming that disjointness is a suficient condition for their independence. The training dataset have been under-sampled in order to mitigate overfitting during
fying all 20 testing datasets are in Table 3. The table is complemented with boxplots (Figure 2), where the following quality measures are presented for each classifier: accuracy, F1 score, precision and recall. The random forest classifier had the worst results of all experiments.
Other models had significantly better results. The best accuracy had the Gaussian process, its mean accuaracy was 97.5 % with very low standard deviation.
The diferences between the considered classifiers
were tested for significance by the Friedman test. The
basic null hypothesis that the mean accuracy for all 5
classifiers coincides was strongly rejected, with the achieved
significance = 3.02 × 10− 12. For the post-hoc
analysis, we employed the Wilcoxon signed rank test with
two-sided alternative for all 10 pairs of the compared
classifiers, because of the inconsistence of the more
common mean ranks post-hoc test, as pointed out in [
For correction to multiple hypotheses testing, we used the Holm method. The results are given in Table 4, good results has Support vector machine and Gaussian process classifier.
At first, the PDFs were transformed into Microsoft Word using PowerShell scripts. The output files have been processed by a python library for parsing docx files. As a 4.4. Classification of Scientific Texts with result relevant paragraphs have been extracted for clas- respect to Relevant Ontologies sification according to the most relevant ontology. The irrelevant paragraphs contained acknowledgement, ref- For this experiment, we had no ground truth as to which erences, titles and too short paragraphs (shorter than 100 of the available ontologies is the most relevant for each letters) have been skipped. considered paragraph of text. We employed two collec
The annotations in the specifications of given ontolo- tions of scientific papers from the area of catalysis. The gies have been extracted using XML parser for python small one are papers dealing with the topic of methananamed BeautifulSoup. Extracted paragraphs have also tion of CO2, it consists of 28 PDFs, from which we have been used for BERT fine-tuning. The chosen version of extracted 1 485 relevant paragraphs. The large one is the BERT was recobo/chemical-bert-uncased from the the digital archive of papers (co-)authored by scientists Huggingface portal [9]. Using the fine-tuned BERT, every from the Leibniz Institute of Catalysis (with the excepparagraph has been transformed into a 768-dimensional tion of very few papers with read protection), it consists numeric vector. of 3 450 PDFs, from which we have extracted 144 490
The extraction of annotations from OWL files has been relevant paragraphs. The BERT embeddings of those performed using a python XML parser. Individual annota- paragraphs were classified by the five trained classifiers. tions have been again embedded into the 768-dimensional The confidence is probability over all classes, that source vector space using the fine-tuned BERT. paragraphs fits into target class. Every paragraph can
For the employed classifiers, their implementations be classified to more than one target class with specific in ScikitLearn [4] has been used. The optimal values confidence. The sum of confidences of each paragraph of hyperparameters of each classifier were determined is one. In this experiment were used models trained in using a 5-fold cross-validation applied to a grid-search previous experiment. with the grid values listed in Table 2. In order to mitigate overfitting, training data have been undersampled. 4.4.1. Results for the small dataset Statistic computations have used the scipy, statsmodels and pingouin python libraries. tain among all classifiers, assigning most paragraphs to predicted ontology and the second highest class confithe CHEBI ontology, but some pragraphs also to each of dence is shown. Again, the highest values are achieved the remaining four. by the SVM and MLP, whereas the Gaussian process
Figure 4 uses instead of the count of class predictions and random forest have only small margin between the their confidences. The confidence of the SVM and MLP predicted and second most confident ontology, and the is very high, whereas that of the Gaussian process and k-nearest neighbor has quite a high margin, but not so random forest is substantially lower. The k-nearest neigh- high as SVM or MLP. bors classifier has rather high confidence also.
In Figure 5, the margin between the confidence of the 4.4.2. Results for the large dataset the k-nearest neighbors classifier.
In Figure 8, the margin between the confidence of the Figure 6 depicts the count of paragraphs from the large predicted ontology and the second highest class confidataset that each classifier assigned to each ontology. dence is shown. Again, the highest values are achieved The Gaussian process, k-nearest neighbor, MLP and SVM by the SVM, whereas the Gaussian process and random assigned almost all paragraphs to the NCIT ontology. forest have only small margin between the predicted The random forest is most uncertain among all classifiers, and second most confident ontology, and the MLP and assigning most paragraphs to the CHMO ontology, but k-nearest neighbor have quite a high margin, but not so some pragraphs also to each of the remaining four. high as SVM.
Figure 7 using confidences of class predictions shows, that the confidence of the SVM is very high, whereas that of the Gaussian process and random forest is substantially lower. A rather high confidence have also the MLP and 4.4.3. Summary results for both datasets texts. Successful processing input texts and ontologies often requires a quite hard and laborious job. Here have From results in the first experiment we can say SVM has been used classifiers in combination with the representagood results on testing data in many metrics. The results tion learning by BERT, that may help make this process for both datasets show that the SVM classifier has very faster. Our idea was to use embedding of each paragraph high confidences and very high margins between top from PDFs as input to classifiers. We used a pretrained two confidences. Hence, the results indicate that for a BERT that have been fine-tuned using chemical articles. large majority of the unknown scientific texts, the most The output embeddings from fine-tuned BERT were used relevant ontology is NCIT. as an input to the classifiers. We have experimented with ifve diferent classifiers, in particular random forest, sup5. Conclusion port vector machine, multilayer perceptron, k-nearest neighbors, and Gaussian process. The random forest was This paper provides an insight into the possibility to auto- not successful, its accuracy was the worst of all models. matically determine ontologies most relevant to scientific The best results had Gaussian process and support vector machine. to predict the next token based on all tokens before it.
In second experiment the considered classifiers have In future research, it is desirable to try diferent transbeen tested and compared on scientific papers from the formers. We would like to extract knowledge from ANNs domain of catalysis. The ground truth was not known in the context of learning. The main direction of our rethere. The k-nearest neighbor and Gaussian process had search is extending and integrating ontologies. We plan very low margin between first and second highest con- to use also graph neural networks to incorporate them ifdence. The highest confidence among all classifiers into representation learning. had the support vector machine. It had also the highest margin among them.
The biggest weakness of this article is the lack of Acknowledgments
ground truth for the classification of scientific articles,
which makes it impossible to evaluate this classification. The research reported in this paper has been supported by
Therefore, we plan to use methods for reducing the im- the German Research Foundation (DFG) funded projects
pact of unknown ground truth. Our idea is to use inter- NFDI2/12020 and 467401796, and by the Grant Agency
polation between annotations using GPT-2 and GPT-3 of the Czech Technical University in Prague, grant No.
networks. GPT (Generative Pre-trained Transformer) SGS20/208/OHK3/3T/18.
[
pp. 4171–4186 [19] Kolozali S., Fazekas G., Barthet M., Sandler M. B.: A [4] Pedregosa et al.: Scikit-learn: Machine Learning in framework for automatic ontology generation based Python, Journal of Machine Learning Research 12 on semantic audio analysis. Audio Engineering Soci(2011), pp. 2825–2830 ety International Conference, pp. 87-–96, 2014. [5] Benvenuto M. A., Plaumann H.: Industrial Cataly- [20] Li G.: CNN based ontology learning algorithm and sis (2021). Berlin, Boston: De Gruyter (De Gruyter applied in PE data IAENG International Journal of STEM). Computer Science, 48:1-–8, 2021. [6] American Chemical Society Report: Technology Vi- [21] Martel F., Zouaq A.: Taxonomy extraction using sion 2020 The Chemical Industry, December 1996 knowledge graph embeddings and hierarchical clus[7] Borst W. N. Construction of Engineering Ontologies tering SAC’21, pp. 836-–844, 2021. for Knowledge Sharing and Reuse Enschede Centre [22] Memariani A., Glauer M., Neuhaus F., Mossakowski for Telematics and Information Technology (CTIT), T., Hatings J.: Automated and explainable ontology 1997 extension based on deep learning: A case study in [8] Martínez-Romero M., Jonquet C., O’Connor M. J., the chemical domain. 3rd International Workshop Graybeal J., Pazos A., Musen M. A.: NCBO On- on Data Meets Applied Ontologies, pp. 1-–16, 2021. tology Recommender 2.0: An enhanced approach [23] Mercier C., Chateau-Laurent H., Alexandre F., for biomedical ontology recommendation Journal of Viéville T.: Ontology as neuronal-space manifold: biomedical semantics 8 (1), pp. 21 (2017) Towards symbolic and numerical artificial embed[9] Recobo AI Company: BERT for Chemical In- ding Workshop on Knowledge Representation for dustry https://huggingface.co/recobo/ Hybrid and Compositional AI, pp. 1—11, 2021. chemical-bert-uncased HuggingFace - Trans- [24] Mueller R. M., S. Abdullaev: Deep cause: Hypothformers - BERT (2022) esis extraction from information systems papers [10] Al-Aswadi F. N., Chan H. Y., K.H. Gan: Extracting with deep learning for theory ontology learning Ansemantic concepts and relations from scientific publi- nual Hawaii International Conference on System cations by using deep learning. Proceedings of IRICT Sciences, pp. 6250-–6259, 2019.
2020, pp. 374—383, 2021. [25] Teslya N., Savosin S.: Matching ontologies with [11] Althubaiti S., Kafkas S., Abdelhakim M., Hoehndorf Word2Vec-based neural network ICCSA, pp. 745— R.: Combining lexical and context features for au- 756, 2019. tomatic ontology extension. Journal of Biomedical [26] Oba A., Paik I., Kuwana A.: Automatic classificaSemantics, 11:article no. 1, 2020. tion for ontology generation by pretrained language [12] Bento A., Zouaq A., M. Gagnon: Ontology match- model International Conference on Industrial, Engiing using convolutional neural networks. LREC, pp. neering and Other Applications of Applied Intelli5648-–5653, 2020. gent Systems, pp. 210—221, 2021. [13] Chakraborty J., Yaman B., Virgili L., Konar K., [27] Petrucci G., Ghindini C., Rospocher M.: Ontology Bansal S. K.: OntoConnect: Results for OAEI 2020. learning in the deep EKAW, pp. 480-–495, 2016.
OM ISWC, pp. 204-–210, 2020. [28] Petrucci G., Rospocher M., Ghindini C.: Expres[14] Espinoza-Anke L., Ronzano F., Saggion H.: Hyper- sive ontology learning as neural machine translation nym extraction: Combining machine-learning and Journal of Web Semantics, pp. 52-–53:66-–82, 2018. dependency grammar. CICLing, pp. 372-–383, 2015. [29] Potoniec J.: Learning OWL 2 property character[15] Gupta N., Podder S., Annervaz K. M., Sengupta S.: istics as an explanation for an RNN Bulletin of the Domain ontology induction using word embeddings. Polish Academy of Sciences, Technical Sciences, pp.
ICMLA, pp. 115—119, 2016. 68:1481-–1490, 2020. [16] Hao L., Lei C., Efthymiou V., Quamar A., Özcan F., [30] Navarro-Almanza R., Juárez-Ramírez R., Castro J. et al.: MEDTO: Medical data to ontology matching R.: Automated ontology extraction from unstrucusing hybrid graph neural networks. KDD’21, pp. tured texts using deep learning Intuitionistic and 2946-–2954, 2021. Type-2 Fuzzy Logic Enhancements in Neural and [17] Hourali M., Montazer G. A.: Using ART2 neural Optimization Algorithms: Theory and Applications, network and bayesian network for automating the pp. 727—755. Springer, 2020. ontology constructing process. Procedia Engineer- [31] Ristoski P., Paulheim H.: Rdf2vec: Rdf graph eming, 29:3914—3923, 2012. beddings for data mining, International Semantic [18] Katyshev A., Anikin A., Denisov M., Petrova T.: In- Web Conference, pp. 498—514, 2016.