We applied machine learning to predict which of COVID-19-related papers will be highly cited, yielding an extension for the Covid-on-the-Web knowledge graph. Symbolic and deep-learning (BERT) ML performed comparably. LIME-based explanation is also included as part of the produced graph. Among the current proliferation of knowledge graphs (KGs), research-oriented ones are a particular species. They can be understood as concise, structured representations of various kinds of scholarly knowledge, and have the potential to bridge between overwhelmingly large corpora of scientific texts and the potential recipients of scholarly knowledge who only have limited reading capacity. Numerous projects [1], [2] apply NLP techniques in order to extract key facts from research papers so that they can be exploited independently of their original contexts of publication, without the necessity to read the papers in extenso. The quality of the service provided by the KGs however depends on the quality of papers they represent: knowledge from papers making impact in the scientific community should thus be prioritized.
CEUR Workshop Proceedings 2https://github.com/Wimmics/CovidOnTheWeb © 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
category
high 2538 low 8063 high 8058
From our previous experiments we learned that in biomedical scientific document processing, the TF-IDF or bag of words (BOW) representation with random forest or neural network (BERT) learners achieve state-of-the-art results for diferent combinations of document representation. Also, in most cases, the abstract and title had more impact on classifying a research paper than the bibliometric data had. Therefore we only used the research paper titles and abstracts, for the predictive task. 3https://www.kaggle.com/datasets/allen-institute-for-ai/CORD-19-research-challenge 4https://opencitations.net/
NN NN RF RF RF RF RF RF
TF-IDF TF-IDF TF-IDF BOW BOW BOW
P low 0.71 high 0.77 low 0.69 high 0.71 low 0.77 high 0.73 low 0.68 high 0.68 low 0.71 high 0.76 low 0.72 high 0.77 low 0.71 high 0.76 low 0.71 high 0.77 low 0.71 high 0.76 Document representation The Term Frequency – Inverse Document Frequency (TF-IDF) weighting system is the most popular text representation utilized throughout various previous studies. Using the unigrams, bigrams and trigrams from the titles and abstracts we developed the TF-IDF input data table. A binary representation of the input data table (BOW ) was also included for comparison, for the same features.
Next, we employed an embedding-based representation approach that is viewed as a
cuttingedge within NLP-based language models, BERT [
Explanation algorithm The LIME (Local Interpretable Model-agnostic Explanations) [
We used 70% training data and 30% test data by random sampling. The overall accuracy was
used to evaluate the results, but we also computed the per-class accuracy. Table 2 shows the
Precision, Recall, F1 score and accuracy (per-class and average) of the neural network (BERT) and
random forest approach to testing data. As we see, a traditional multi-purpose machine learning
algorithms, random forest, performs well like a neural network (BERT). This is not so surprising
since also in some other reported cases the diference in performance between BERT, TF-IDF,
and BOW was relatively small [
As regards the RDF output, we store the predicted citation rate category (high or low) together with the citation count from OpenCitations and with the LIME-based interpretation, for every research paper from the Covid-on-the-Web KG. In the GitHub repository5, the classified data from the covid-on-the-web corpus is available. An example is as follows6; the LIME-based explanation (stored just a long string in lexinfo:explanation) is displayed as truncated: < h t t p s : / / c i m p l e . v s e . c z / c o v i d - o n - t h e - w e b / 1 0 . 1 0 1 6 / j . y m e t h . 2 0 0 5 . 0 5 . 0 0 8 > a f a b i o : R e s e a r c h P a p e r , b i b o : A c a d e m i c A r t i c l e , s c h e m a : S c h o l a r l y A r t i c l e ; b i b o : d o i ” 1 0 . 1 0 1 6 / j . y m e t h . 2 0 0 5 . 0 5 . 0 0 8 ” ; c i t o : C i t a t i o n 9 3 ; < h t t p s : / / c i m p l e . v s e . c z / c o v i d - o n - t h e - w e b / e x p C i t a t i o n R a t e > h i g h ; l e x i n f o : e x p l a n a t i o n ” ( ’ n o v e l ’ , - 0 . 0 3 0 8 6 7 ) , ( ’ s t r u c t u r e s ’ , - 0 . 0 2 5 7 8 9 ) , . . . ” ; s c h e m a : u r l < h t t p s : / / d o i . o r g / 1 0 . 1 0 1 6 / j . y m e t h . 2 0 0 5 . 0 5 . 0 0 8 > .
We have made an initial exploration on augmenting a research-oriented KG with the predicted impact of the underlying papers, obtained via machine learning.
Our next step will be to evaluate this simple approach in the context of a more comprehensive support for users, in particular, the fact checkers, in getting access to scientific literature and its authors. As regards the actual predictive ML technology, the BERT model, having been merely trained on general textual data (English Wikipedia and the BooksCorpus), did not outperform classical ML models in this first try. We however assume that it would work better if trained on domain-specific data such as bio-medical research papers. Also, we also considering to external KGs, such as encyclopaedic ones (DBpedia, Wikidata), into the learning process. 5https://github.com/corei5/Enhancement-of-the-Covid-on-the-Web 6Prefixes for common vocabularies omitted; they can be retrieved via https://prefix.cc.
This research is being supported by CIMPLE project (CHIST-ERA-19-XAI-003). The authors also would like to thank Sören Auer and Open Research Knowledge Graph (ORKG) group for providing valuable feedback and some more idea to enhance this with some other research in the ORKG environment.