The volume of healthcare data generated during the COVID-19 pandemic is having a significant impact on tabulating, summarizing, and indexing the facts that could help healthcare workers plan and prevent the spread, according to [1,2]. Due to the accessibility of existing biomedical knowledge repositories, the current coronavirus pandemic highlights the need for automatic relation extraction techniques. In recent years, there has been a lot of time supporting the use of patterns in prediction models [3].
Machine learning models are rapidly becoming a powerful resource in healthcare. However, the quality of these models depends on the availability of high-quality training data. In addition to large datasets being necessary, these training sets must be robust and accurate. However, obtaining comprehensive and accurate real-world data for machine learning in healthcare is challenging due to privacy and ethical issues associated with such data.
Rule mining automatically mines the logical rules from a given knowledge base (KB). For example, the interesting rule mining methods find that "If A is the husband of B, and A lives in the USA, then B also lives in the USA". This type of rule a mined based on certain confidence. These are necessary to have a complete KB. Rules are widely used in data and ontology for alignment and fact-checking purposes.
The Resource Description Framework (RDF) relies on graph-based structures. The description from a graph illustrates the relationships between the entities. Also, the information is decentralized, so connecting two graphs create a new graph. RDF follows an open-world assumption, facts that are stated are considered true, and the facts that are not stated are considered unknown Motivation With the abundance of healthcare data available, it is critical for decision-makers to use it for predictive and preventive measures. Semantic data mining using ontology and transformer-based methods can reveal hidden inferences from data. This encourages decisionmakers to keep track of data points that are relevant or interesting. The main focus of this paper is to infer interesting facts from two corpora of COVID-19 using the proposed interesting framework. More precisely, our contribution is as follows:
• We define a framework for data interestingness using domain ontology.
• We propose a novel technique to identify interesting rules using ontology and transformersbased methods. • We compare the performances of two BioBERT Models for interestingness in COVID-19 data Further, we demonstrated the usage of the proposed approach for paraphrasing the rules for decision-makers. The remainder of the paper is as follows: Section 2 discusses the data and methods. Section 3 proposes the Ontology-based Data Interestingness (ODBI) framework used in this study. Section 4 discusses the results from two COVID-19 corpora by comparing their semantic nature of it. Section 5 concludes the paper by outlining future research directions.
Information extraction aims at automatically extracting information from unstructured data sources. Applications include information retrieval, opinion mining, sentiment analysis, question answering, and machine translation.
In computer science, the domain-specific task requires ontology as data and semantic model [1]. An ontology generally consists of an agreed (i.e., semantics) understanding of a specific field, axiomatization, explicitly expressed in a computer resource as a logical theory [4].
Association Rule Mining (ARM) is the most important topic in data mining research. Its goal is to discover interesting correlations, patterns, and associations between groups of items in transaction databases. Telecommunication networks, market and risk management, and inventory control all use association principles. Finding interesting association rules is a popular and current topic in data mining techniques [5,6,7].
In the state-of-the-art, several measurements are proposed, with ontology being less explored. An ontology that uses the semantic web, where data is represented as Resource Description Framework (RDF) triples (subject, predicate, object) makes it machine understandable. This fortifies the system to infer knowledge using the underlying schema of ontology [8]. The publication "Attention is all you need" by [9] presented the Transformers architecture (2017). The architecture of transformers is encoder-decoder. The BERT model has recently produced cutting-edge results in a variety of NLP tasks in the same context. It's a different kind of transfer learning. BERT's primary operating mode is a transfer by fine-tuning similar to the one used by ULMFiT. Additionally, BERT can be used in the transfer mode by removing features like ELMo. Early detection model using Chat bot analytical language resources of descriptive questions to extract interesting facts. Three distinct models, CT-BERT, BERTweet, and Roberta are tuned on COVID-19-linked text data to distinguish between fake and real news [10].
These successful studies demonstrate that ontologies can be used to improve the performance and enhance the usability of complex data analytics systems. The transformer models used in the study were pre-trained on biological data, giving them a deeper understanding of the terminology used in biomedicine. We use these state-of-the-art transformer-based methods for generating rule embeddings and cluster them further to analyze them with semantic scores for interesting ones.
This section explains the preliminary definitions and dataset with the proposed OBDI methodology.
Ontology and ARM methods closely work towards the data interestingness [1]. In data mining literature, association rule mining is widely used for rule generation based on frequent patterns. This section aims to provide the readers with the necessary background knowledge.
we call X − → Y as association rule. To have the strong association rule, we need to compute the support and confidence as indicated in equations 1 and 2. Rules are defined considering our domain information.
(1)
Tbox: define the schema or an ontology. Abox refers to RDF triples at the instance level. G is a labeled graph structure produced by connecting the relations with concepts. Figure 1 illustrates the importance of ontology.
Definition 3. Data Interestingness: Our notion of data is derived by integrating domain ontology with data in RDF and user interest rules. Data Download Data Access may be requested to HFWS.
Data Instances 120000
In this work, we used two COVID-19 corpora from the Indian state of Karnataka. 12 The data statistics is illustrated in Table 1.
We generate the embedding using the BioClinicalBERT and CovidBERT models in this study.
BioClinicalBERT is a model trained with data corpora.
BioClinicalBERT is a model that is initialized on BioBERT (BioBERT-Base v1.0 + PubMed 200K + PMC 270K) and then it is trained on the MIMIC III [11] notes. These MIMIC notes consist of electronic health records from ICU patients of a hospital. For the pretraining of this model, the authors utilized a batch size of 32, a maximum sequence length of 128 with a learning rate of 5 * 10 −5 . The models were trained for 150,000 steps using all MIMIC notes.
CoviBERT is a model that Deepset trains on AllenAI's COR19 dataset which consists of various scientific articles about coronaviruses. The model is initialized on BERT word piece vocabulary. Then, using the sentence-transformers library, it is fine-tuned on the SNLI and MultiNLI datasets to construct universal sentence embeddings using the average pooling technique and a softmax loss [12].
The proposed OBDI framework, as in Figure 2, is an ontology-based mining framework that uses semantic similarity to determine the interestingness of rules. OBDI's goal is to automatically generate rules and knowledge from datasets to improve future decision-making process efficiency. OBDI's logic structure is as follows: RDF data instances are created from a dataset and a domain ontology. These data are backed by the domain experts' knowledge and also ontology concepts. Interesting rules are formulated as shown in Table 2,3 and 4 using the ontology and experts' knowledge. The OBDI methods include the IntApriori proposed [13]. It's significant that the generated rules are processed by BERT models for semantic scores to determine a rule's importance and degree of interest.
This section discusses the semantic association rules generated from COVID-19 data and how it is processed on BioClinicalBERT and CovidBERT for identifying the similarity-based interesting rules. With the detailed analysis from the current state-of-the-art, BioClinicalBERT and CovidBERT are used in this study.
In a given set of rule embeddings, we find clusters that are closely related to the embeddings. This mapping is facilitated by BioBERT embeddings [14] of the rules generated by ontologybased mining. This helps reduce the rules' search space to have the most interesting ones. The cluster centroid is considered as the interesting point indicated as I . The rules that match the cluster I value are termed as the most interesting ones. Focusing on the rules in the particular cluster, we use BioClinicalBERT and CovidBERT [15] embeddings and text summarization model to find the best-matched rules by generated the summarization of the cluster. Further, this summarization is treated as a paraphrase to decision-makers for future actions.
The goal of the OBDI framework is to generate interesting rules, given the data and the domain ontology. The COP and COKPME ontology is used for generating semantic Association rules [13]. Our previous studies illustrate the design and implementation of COP and COKPME 5 . Table 2 shows the semantic association rules of the KATrace COVID-19 Dataset. Further, these rules will be used by BioClinicalBERT and CovidBERT to identify the interesting rules.
With the object and data properties defined in the COP ontology, the relationships inferred by the reasoner are the initial path for interesting fact generation. We define a set of rules that are operated on COP ontology for interesting fact generation. A few of the rules are indicated in Figure 3.
The rules are generated using ontology-based methods. A few of the rules with higher confidence are listed in Table 2. The generated rules are semantically annotated so that decisionmakers can interpret them and take the appropriate actions. The results show the patient's age, status, the location from which he traveled, and the treatment provided.
Tables 3 and 4 describe the rules associated with its interesting index (I). The cluster centroid values are used as interesting data points, as are the embedding values that point to specific rules. Clustering using K-means [16] is applied to both the CovidBERT and BioClinicalBERT embedding sets. Both models generated five centroid points, five of which were interesting (I). Interesting rules are extracted from the rules pointing to the I value. The output of K-Means clustering on CovidBERT and BioClinicalBERT embeddings is shown in Figure 4.
Figure 5. depicts the distribution plot of the average of word embeddings obtained by the two models BioClinicalBERT and CovidBERT. The model's embedding distribution is also typical. The distributed rules demonstrate the model's understanding of the input rules. Table 5 also describes the ontology relationships distributed across the rule embeddings. It has been discovered thattreatmentProvided and sufferFrom are the two majorly identified ontology relationships. The box-whisker plot of the semantic scores obtained from the two models BioClinicalBERT and CovidBERT is shown in 6. When compared to the BioClinicalBERT model, the CovidBERT model has a lot of variation in the semantic score. This demonstrates the two models' different levels of comprehension. The BioClinicalBERT model calculates high cosine similarity values between these rules, implying that they are very similar. The value of the min, max and mean similarity scores are as depicted in Table 6 The results show that the methodology learns to generate interesting facts based on the simple linguistic feature (COVID-19 Corpora) which are embedded in textual data using the BioClinicalBERT and CovidBERT model. The paraphrased summary of the identified interesting rules is as follows:
• Patient with {ILI, Diabetic} are highly prone to COVID-19 Infection. • Below the age group of 35 is all suggested to have {MPHQ} advice. So healthcare facilities should be reserved for higher age groups. • Many health workers are infected and admitted to their own hospitals, creating a shortage of resources. • The most widely documented symptom in the COVID-19 dataset is the common flu.
The decision-makers understand these paraphrased rules for having preventive and predictive analysis.
This article proposes a novel methodology for mining ontology based on interesting facts from the COVID-19 data corpora. The Mined ontology-based rules are used as input to the transformer-based models like BioClinicalBERT and CovidBERT for interesting rules. The aggregate value of all rule embeddings is clustered. Next, using the cluster centroid, the Interestingness index (I) is derived and illustrated as the most interesting rule. Further, with the similarity scores from both models, the rules are compared for their similarity index. It observed that BioClinicalBERT outperformed CovidBERT with the similarity score by giving high relevance to the generated rules. As future research directions, this study is continued to compare the model-generated rules with domain expert rules to justify our claims. Another possible extension of this work could be to use it in the applications like the state-of-the-art COVID-19 Sentiment Analysis Toolkit.
This work was supported in part by the Department of Health and Family Welfare Services (HFWS), Government of Karnataka, India. We also extend our special thanks to the E-Health section of HFWS, Government of Karnataka, India, for providing all the necessary support and encouragement. Also, we would like to thank two anonymous reviewers for commenting on earlier versions of this paper.