The reasoner is the mechanism for interpreting the semantics of web ontology language. This paper focuses on reasoner performance study and predicting it by use of machine learning. Reasoner evaluation is very challenging as reasoner's efficiency may vary on different ontologies with the same complexity level. Different reasoners give different inference for the same ontology. Thus, reasoner could be enhanced for some however not for all ontologies. Here, paper focus on reasoner performance variability of reasoner and how ontology features affect reasoner performance. The main goal is to provide simple, efficiently computable guidelines to users. For prediction, supervised machine learning is used as a machine learning technique which help us to capture these dependencies. First introduced a new collection of efficiently computable ontology features, that characterize the design quality of an OWL ontology. Second, modeling of two learning problems: first, predicting the overall empirical hardness of OWL ontologies regarding a group of reasoners; and then, anticipating single reasoner robustness when inferring ontologies under some online usage constraints. To fulfill this goal, a generic learning framework is used, which integrates the introduced ontology features. The framework employs a rich set of machine learning models and feature selection methods. Furthermore, we used multi-label learning by analyzing the learned models unveiled a set of crucial ontology features likely to alter the empirical reasoner robustness.
The problem that we want to focus is on
semantic web reasoner performance measure
and empirical assessment of multiple reasoners.
An application developer can find the best
suitable reasoner for given ontology. We
proposed here machine learning techniques
based on given ontology features to predict
correctness or relevance and time for reasoning
task by a set of reasoners. First, we have done
experimentation for the empirical study of
individual reasoner’s performance prediction
for its correctness and reasoning time using
various ML model. After that, we proposed a
multi-label classification technique to predict
reasoning time and relevance of reasoner.
Here, we have considered both profiles of
ontology OWL (DL and EL). We have checked
the performance parameter and compared it
with the benchmark. The Semantic Web
requires a standard, machine-processable
representation of ontologies. The W3C has
defined standard models and languages for this
purpose. There are standard languages used for
semantic web, Resource description framework
(RDF) [
Description logic-based Reasoners are crucial elements to work with OWL ontologies. They are sued to produce explicit knowledge from ontologies to check their consistency and many other things. People are building an ontology by putting on domain knowledge and trying to get more expressive and representative ontologies. But more the ontology is expressive; the more reasoning is complex. In the worst case, reasoning can be nondeterministic doubled exponential. Thankfully, in practice, the reasoning is feasible even with very expressive ontologies. However, in general there is the theoretical complexity does not meet the empirical complexity.
There is an ample number of reasoners
available for the semantic web application, and
it is difficult for the application developer to
choose right reasoner for an ontology for the
domain-specific application. For evaluating
ontology reasoners, OWL Reasoner Evaluation
workshop organized every year. In this
evaluation process, there are two significant
issues one that is a disparity of reasoner’s
computing time which causes efficiency
problem, i.e., for the same size and expressivity
classes we get different computational time.
The second problem is related to a disparity of
reasoner’s computed results, which produce
correctness problem, i.e., for the same size and
expressivity classes, we get different agreement
level. For resolving above two issues, there are
various explanations given by many researchers
[
Main research gap in this area is an
exponential growth in a number of the reasoner;
there is a variable empirical performance of
reasoner. There is a lack of prior knowledge and
expertise in this field. So main crux of this gap
is how to help an application developer to
choose the appropriate and suitable reasoner to
work with domain-specific-ontology for a
given application. To address these issues,
many researchers [
Machine learning can bring a solution to this problem since it can help us to anticipate future reasoner behaviors by analyzing past running. Predicting single reasoners performance includes predicting the ranking of reasoners given an input ontology. However, all these solutions have many drawbacks. So, we proposed a new approach, the automatically rank reasoner, to recommend the fastest reasoner.
The contribution of this work is to proposed reasoner performance parameter prediction method. Which is experimented and implemented using ORE framework tool using Python libraries.
Ontology features are qualitative and quantitative attributes covering structural and syntactic measures of a given ontology. Ontology metrics or functions are used for deciding reasoner performance prediction.
Use of ontology metrics can predict
classification time. Based on parameters
presented in [
This brief study is to know the types of reasoner
available with their characteristics and
descriptions. Attributes of ontology reasoners.
In paper [
Syntatic Description
Signature Size (SSIGN)
Axiom Size(OAS)
DL Family Hierarchy (HC,
HP) Cohesion Richness
Axioms Constructors
Classes Properties Individual
SC, SOP,SDP,
SI, SDT OWL PROFILE
MDepth
ADepth HC-Cohesion HP-Cohesion RRichness KBF (Set) CCF (set) CDP (set) OPCF (set) NFF (set)
Msibling ASibling
MTangledness
Tangledness OP Cohesion
omt Cohesion AttrRichness
ATF(Set) ACCM OCCD
CCYC HRF(set) ISF (set)
ADF(Set) CCP (Set)
CDISJ
reasoners. e.g., classification performance,
TBOX consistency, checking performance, etc.
There are various Reasoners available,
comparative survey presented based on
papers[
There is a requirement to measure, benchmark,
and characterize the performance of various
reasoner available. The main aim of the SEALS
project was to evaluate the DL-based reasoners.
The comparison of three reasoners was made
from standard inference servies. They have [
The classification is done on the ontologies using different reasoners. A comprehensive study is done regarding the variability and size of the dataset with more than three-hundred ontologies. They have also found some unique attributes with a thorough study. Such characteristics are used in reasoner’s comparison and selection for given set of performance criteria.
Paper [
Classification and Prediction are two main
techniques of Machine Learning, especially in
supervised learning, which required to apply
during performance prediction of reasoners.
Classification is ML technique which is used to
identify the class for a new object like ontology,
text or images, etc. from given set of classes.
Reasoner performance [
Semantic Web applications with ontologies, the behavior of reasoners used is very unpredictable. There are two main reasons for this; one reasoner would exhibit enormous scatter in computational runtime across the same ontologies and secondly, reasoners would derive different inferences for the same input ontology. These show the hardness of understanding reasoner’s empirical behaviors for good reasoner developers.
For selecting the best reasoner for semantic web application using evaluation of reasoners performance, our hypothesis is that based on ontology features and metrics. We can predict reasoner’s performance and can predict bestsuited reasoner using machine learning techniques, especially using multi-label learning algorithms and ranking techniques. Following are steps to follow for Reasoner Performance Prediction.
• Import Data contain ontology features and reasoner performance parameters.
Data set of standard OAEI. • Select standard Test data and Train data given in dataset. • Define various features using feature selection, i.e., ontology characteristics and metrics. Define Target, i.e., Reasoning time, and Reasoner status. • They fit multi-target classifiers for relevant and irrelevant reasoners for given ontology set. • Arrange Reasoners for, given all ontology using relevant first and then according to the order of time after relevant reasoner put irrelevant reasoners according to the order of time. • Give ranks according to the above arrangements. • Fit classifier / Regression to predict ranks.
The main aim is to do work on automatically predict a reasoner’s time efficiency and correctness. To achieve this goal, people have worked and suggested machine learning approaches, which includes the following steps. First, we required to work on the set of valuable ontology features, which will be used for learning ontology by machine learning model. ORE’2014 Framework is widely used to conduct experiments on various reasoners for their performance on given ontology corpus. At last, deployed supervised machine learning techniques to learn predictive models of reasoner performance based on previous execution. By interpreting these models, we can observer that few main features may change the performance of reasoner.
Feature selection is one of the prime steps in preprocessing dataset for training model in machine learning. The main purpose of feature selection is selecting the most relevant features by excluding non-useful features. Other researchers have used supervised discretization method (MDL), the Relief method (RLF); the CFS Subset method (CFS). We will have used Feature variance and Feature correlation with label data.
The supervised learning algorithms can be divided or grouped as logically based algorithms such as decision trees, Artificial Neural Network (ANN) based techniques such as multi-layered perceptron, Statistical learning algorithms such as Bayesian network and SVM. We can use some supervised machine learning algorithms like Random Forest, Simple Logistic Regression, Multilayer Perceptron, ANN-based learner, SMO SVM based learner and IBk K-Nearest-Neighbor based algorithm.
Limitation of Single Label based learning is that it may not give consistent output for the selection of reasoner based on multiple criteria. Multi-Label based learning with multi-criteria is useful because single criteria may not give a consistent result.
The reason to apply multi-label classification is,
for each ontology, there may be multiple
possible correct reasoners. This inspired us to
do multi-label classification for predicting
relevancy of given ontology. Here Ranking of
reasoners can be decided based on multiple
criteria, i.e., like correctness or relevance of
reasoner for given ontology and time taken for
doing reasoning of that ontology. So, to decide
out of all possible correct reasoners, we need to
decide and identify the first one to experiment
for given ontology. That is why we finalized
two criteria for ranking reasoners that is
correctness and time required for reasoning.
The solution of Reasoner selection
methodology recently discussed and suggested
by Alaya in [
In place of the above method to decide the better approach, we have done experiments with various ML model of MLC, where we found Ensemble Approach of MLC better compare to Binary relevance, especially we used Ensembled of Random Forest model. For MTR also, we have used Random forest for regression, which outperforms the Regression chain method.
If we compare the different problem transformation method for Multi-label learning, classifier chain is not advisable to exploit the correlation between targets. It gives a better chance of ranking higher to reasoner or label predicted last. Label power set method is not technically suitable for 1900 dataset in which a combination of 10 become more than 1000. In another word number of classes increase to more than 500, which is not good. Because of that, we have used Binary relevance as multilabel learning and Random forest as a base algorithm for multi-label learning because it performs better than KN, Logistic Regression, MLP, AdaBoost, Navi Bias, and QDA.
Evaluation and assessment measures are used to check the quality of ML model. For binary ML scenario, we could have TP, TN, FP, and FN value used for assessment. From this, we can calculate F1-measure, Precision, and Recall.
For assessment of ML with multi-class models, with an imbalanced dataset, we can use assessment measures like the F1-measure, Kappa coefficient, and Matthews the correlation coefficient. These measures we proposed to select the reasoner best predictive model. Assessment of relevance prediction model and compare with the existing system using Hamming loss and F1 measure.
Experiments to collect data for empirical
behaviors of reasoners for classification task of
a given set of OWL ontologies. For this we
work with the evaluation tools in ORE
(Ontology Reasoner Evaluation Workshop)
[
Ontologies data set is taken from ORE Corpora2, around 1900 ontologies collected from this source which used for reasoner performance prediction.
Reasoners3 set from popular categories are selected as candidates for performance evaluation and prediction process. Reasoner correctness/robustness and performance time is generated for 10 reasoners which have shown good efficiency in classification task of ontologies, during ORE competitions. The list of 10 reasoners includes ELK, Konclude, MORe, ELepHant, HermiT, TrOWL, Pellet, FaCT++, Racer, JFact.
Start by evaluating the reasoners; we have to find empirical data. They describe the performance on a large set of ontologies. So, select to use tools proposed in the ontology reasoner evaluation workshop. We tool their framework ORE. We set classification challenges (DL & EL) 1900 ontologies. All the DL ontologies are to be handled by 8 reasoners, and 2nd challenge #EL ontologies will be handled by ten reasoners 8 + ELK and Elephant. A time limit of 3 minutes.
Steps for an experiment using Machine learning applied to estimate the best reasoner for ontology: 1. Import Data 2. Feature selection 3. Select test and train data 4. Apply ML methods for predicting reasoner relevance (for 10 reasoners) 5. Apply ML methods for predicting reasoner time (for 10 reasoners)
For Reasoner’s performance prediction, we have applied various machine learning models like k-NN, Decision Tree, Random Forest, Neural Network, and AdaBoost. After applying this model on the dataset, we predicted Execution time as the target variable. We have measured and compare Error rate, i.e., Root Mean Square (RMS) Error given by each model for every 10 different reasoners. Figure 2 shows that Random Forest is performing best for all ten reasoners compare to all other models. A neural network is the worst model for the majority of reasoner performance prediction. 8.00E+09 6.00E+09 4.00E+09 2.00E+09 0.00E+00 1.5
1 0.5
0
1 0
Nearest Neighbors
Decision Tree
Random Forest
Neural Net
AdaBoost models. We have also checked the performance parameter of prediction using F1 measure as per Figure 4 graph. This graph exhibits that Radom Forest gives the best result in terms of F1 measure. By this graph, we can conclude the reasoner is the dominant reasoner in the DL ontologies. We can see that Hermit having a high rate of correctness is very slow, EL is dominant reasoner when in handling EL ontologies. All of this data will serve to create a learning data set. So, we try to divide the data in to Train and Test data to learn the mulitRAkSOR predictive models; then we assessed the relevance of the predictive quality of reasoner relevance. Our result shows our algorithm outperformed the existing solution.
We have used Multi-Label Classification using problem transformation methods and Adapted Algorithms like MLkNN, BPMLP, RAkEL, and Random forest. MLkNN It is a version of existing KNN for the multilabel learning task.
It does not divide the problem into subproblems. BPMLP, this is a multi-label version of Neural Network-based algorithm.
RAkEL is Random k Label set method.
Random Forest special version for multi-label classification we have used.
We can observe results about multi-label learning method for prediction of reasoner’s performance using the various parameters like Hamming-Loss, Accuracy, Jaccard-Similarity, Precision, Recall, and F1-measures.
Multi-Label Learning Model Performance Analysis 1 0
Hamming-Loss
Accuracy
Jaccard-similarity
Precision
Recall
F1-Measure MLkNN
BPMLP
RAkEL
RF
For Semantic web heterogeneous store data in a structured way using Ontology concept, to fetch answer of the query of user, we required reasoner and logical rules. For understanding and using the semantic data on the web, there is a requirement of the reasoner.
For selecting appropriate reasoner by Semantic Web application developer, we have proposed a machine learning-based models for relevance and reasoning time prediction for given ontology. We have applied the multi-label learning method predicting the rank of various reasoners. Using single label prediction methods for given data set of ontologies and reasoner, we have shown that Random Forest is giving the best performance in terms of performance parameters. Same way, for multilabel learning model also Random forest variance outperform MLkNN, BPMLP, RAKEL and recently proposed MultiRakSOR in terms of Hamming-Loss (0.015), Accuracy, Jaccard-Similarity, Precision, Recall and F1measures (0.97).
In future work, we could expand this approach for a greater number of ontologies and also on multiple domains. We could also extend our work in the future for SPARQL query performance measurement benchmark using a greater number of queries empirically by increasing number of experimentations.