=Paper=
{{Paper
|id=Vol-2871/paper7
|storemode=property
|title=Automatic Identification and Classification of Integrated Knowledge Content in an Interdisciplinary Field: A Case Study on eHealth
|pdfUrl=https://ceur-ws.org/Vol-2871/paper7.pdf
|volume=Vol-2871
|authors=Shiyun Wang,Jin Mao,Hao Xie
|dblpUrl=https://dblp.org/rec/conf/iconference/WangMX21
}}
==Automatic Identification and Classification of Integrated Knowledge Content in an Interdisciplinary Field: A Case Study on eHealth==
https://ceur-ws.org/Vol-2871/paper7.pdf
1st Workshop on AI + Informetrics - AII2021
Automatic identification and classification of integrated
knowledge content in an interdisciplinary field: A case
study on eHealth
Shiyun Wang1[0000-0002-4409-7751], Jin Mao1,2,* [0000-0001-9572-6709] and Hao Xie1[0000-0002-1788-
0468]
1
Center for Studies of Information Resources, Wuhan University, Wuhan 430072, China
2
School of Information Management, Wuhan University, Wuhan 430072, China
*Corresponding author. Email: danveno@163.com
Abstract. Investigation of knowledge integration characteristics, especially from
the content perspective, is essential for understanding the formation and evolu-
tion of interdisciplinary fields. However, in previous studies, it involves consid-
erable time and effort for researchers to recognize the integrated knowledge con-
tent in an interdisciplinary field and to analyze the content characteristics. There-
fore, we have studied the automatic methods to identify the explicit integrated
knowledge phrases from citation contexts in interdisciplinary field papers and
recognize the functions of integrated knowledge phrases by utilizing word em-
bedding techniques and deep learning models. To evaluate the performance of
our methodology, we constructed an experimental dataset by taking the eHealth
field as a case of interdisciplinary field. From the experimental results, we ob-
tained Recall, Precision and F1 scores of 0.838, 0.989 and 0.907 for the explicit
integrated knowledge identification process, and Recall, Precision and F1 scores
of 0.856, 0.863 and 0.842 in the unknown phrases test dataset in knowledge func-
tions classification.
Keywords: Knowledge Integration, Interdisciplinary Field, Semantic Function
Recognition, Deep Learning.
1 Introduction
Interdisciplinary research is often considered as an important driver in modern science
[1]. The essence of interdisciplinary research is successful recombination of existing
disconnected knowledge units from various disciplines [2], which may lead to novel
ideas and accelerate scientific breakthroughs. The increasing number of emerging in-
terdisciplinary fields demonstrate that interdisciplinary research has become an im-
portant mode in science.
Scientists and policy makers have attempted to explore the characteristics of interdis-
ciplinary research to promote the development of interdisciplinary research. Several
bibliometric indicators, e.g., Rao-Stirling [3], have been proposed to measure the inter-
disciplinarity of research domains or publications. Most studies used citation analysis
to investigate knowledge diffusion relations between an interdisciplinary field and its
Copyright 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0
International (CC BY 4.0).
2
source disciplines through references [4-6]. However, these studies only measure
knowledge dissemination at the paper and journal level, rather than from the perspec-
tive of knowledge units. A few recent studies have explored knowledge integration and
evolution of an interdisciplinary field from the content perspective to understand the
formation and development of an interdisciplinary field [1,7-9]. Nonetheless, the iden-
tification of integrated knowledge content involved considerable human efforts in these
studies. To foster the subsequent analysis and knowledge mining at a large scale, auto-
matically identifying integrated knowledge content is in great demand.
In this study, we investigate the integrated knowledge content by an interdisciplinary
field. We applied NLP techniques to automatically identify the integrated knowledge
units from citation sentences and the texts of reference publications. And, we classified
the functions of integrated knowledge through deep learning models. An experimental
dataset of the eHealth field was constructed to validate the effectiveness of our meth-
odology.
2 Methodology
2.1 Definitions and Problem Formulation
Scientific publications record various forms of knowledge integration, e.g., the cooper-
ation of researchers and the citations to the references of different disciplines. In this
article, we investigate the knowledge integrated in an interdisciplinary field, which can
be reflected through citation relations. To this end, we propose integrated knowledge
phrases and a classification scheme based on knowledge functions, which are defined
as follows.
Integrated Knowledge Phrases. Citation contexts often express relevant information
about cited articles [10]. To reflect the integrated knowledge units, we use the noun
phrases extracted from citation sentences that also appear in the text of the correspond-
ing cited publications. We contend that the shared phrases between the two counter-
parts explicitly signify the transferred knowledge.
Knowledge Classification Schema. Several classification frameworks have been pro-
posed to annotate semantic functions of concepts or terms in scientific papers [11-12].
However, the classes in these frameworks, e.g., problems, solutions, and goals, are too
general to analyze knowledge integration in a domain at a fine-grained level. In our
previous study, we proposed a knowledge classification schema for the annotation of
integrated knowledge content, which comprises seven categories, including Research
Subject, Theory, Research Methodology, Technology, Entity, Data, and Others [9]. Re-
search Subject is broader than other categories, which covers multiple kinds of domain-
specific research subjects, e.g., diseases, drugs, and research themes. The identification
of domain entities have become significant tasks in many recent studies, e.g., Biomed-
ical NER task [13], and some tools have been developed, e.g., PubTator Central [14].
Therefore, we do not involve the Research Subject category, as well as the Others cat-
egory. The final framework in this study is shown in Table 1.
3
Table 1. The classification framework of integrated knowledge phrases.
Category Description Exemplar phrases
Theory Theory related phrases e.g., TAM, social cognitive theory, trans-
theoretical model
Research method- Methodology used in e.g., systematic review, analysis, meta anal-
ology research ysis, randomized control trial
Technology Technique, device and e.g., mobile phone, web, smartphone, app
system that used in re-
search
Entity Human related research e.g., patient, woman, child, adolescent
object
Data Phrases related to da- e.g., twitter, qualitative datum, clinical da-
taset, data source and tum
data material
The research design of this study is summarized in Fig. 1. The task of integrated
knowledge phrases identification includes knowledge phrases extraction from citation
sentences and corresponding reference texts, and knowledge phrases matching between
the two sources. For the task of classifying the functions of integrated knowledge
phrases, several deep learning models are adopted.
Fig. 1. Research framework of this study
2.2 Dataset Construction
We collected full texts of papers in an interdisciplinary field. For each paper, citation
sentences and bibliography data (title, PMID, etc.) were extracted and linked via the in-
text citation tags in the text, e.g., “[1], [2-8]”. Next, we complemented the metadata of
the references, e.g., titles and abstracts, as the cited texts. Then, each in-text citation
generates a pair of citation-reference. An in-text citation of several references was split
4
into multiple citation-reference pairs. The section titles of citation sentences were also
fetched. The citation-reference pairs records are constituted as the initial dataset for the
following processes.
2.3 Integrated Knowledge Phrases Identification
Knowledge Phrases Extraction. For each citation-reference pair, we used spaCy, an
open-source Python natural language processing toolkit to extract noun phrases in the
citation sentences and reference texts. We only selected phrases with 2 to 4 words rather
than retained all the phrases with less than 7 words as in our previous study [9]. We
also removed the phrases started or ended with numbers and the phrases with single
characters. Moreover, scispaCy [15], a Python package for processing biomedical sci-
entific text was applied to expand abbreviations in the text.
Knowledge Phrases Matching. The extracted knowledge phrases from the two
sources were lemmatized and stemmed using the NLTK package before matching, so
that different variations of the same word could be matched correctly. Next, we used a
combination of three approaches to match the phrases from citation sentences with
those from the corresponding references for each citation-reference pair.
Direct Matching. This approach only counts the identical knowledge phrases from the
two sources as matched phrases.
Indirect Matching. In this process, the extracted knowledge phrases from the citation
sentences will be exactly matched with the sentences in corresponding reference text
using regular expressions, and vice versa. This approach could identify those phrases
with the same meaning but with different collocations. For example, “focus group” and
“focus group method” could be matched through this method.
The above two approaches were combined as the baseline method for the knowledge
matching process. We further applied a phrase similarity calculation approach based on
word embedding techniques to identify those phrases with similar meaning but are rep-
resented in different word collocations.
Word Embedding + Cosine Similarity. Word embedding technique was utilized to
transform phrases into high dimensional vectors, and then cosine similarity of phrase
vectors was calculated. We selected two word embedding models, GloVe and BERT
(Bidirectional Encoder Representations from Transformers). In short, GloVe and
BERT are both language representation models which can be used to vectorize the
words. Word vectors involve rich semantic and contextual information of the words in
the training corpus. However, compared to conventional word embedding models, such
as GloVe, BERT introduces position encoding to describe sequence position infor-
mation, and takes the jointly left and right contexts for each occurrence of a given word
into account, which could capture more contextual information. In this paper, we used
a 100 dimensions GloVe model [16] that was pre-trained on the dataset of Wikipedia
5
2014 and Gigaword 5, and a 12 layers, 768 hiddens BERT base model [17] pre-trained
on Wikipedia and BookCorpus, a corpus with 11,038 unpublished books.
Then, we integrated the matched noun phrases identified by the above three approaches,
and removed the deduplicated phrases. The retained phrases were denoted as the inte-
grated knowledge phrases.
2.4 Integrated Knowledge Phrases Classification
We applied several deep learning models to classify integrated knowledge phrases,
which consists of the following major modules:
Input Module. For the input module, we considered several contextual information of
the phrases, which could be divided into semantic information and syntactic infor-
mation.
Semantic Information. Both citation sentences and corresponding reference texts con-
tain the semantic contextual information of phrases. Therefore, we included both of
them as the input text features.
Syntactic Information. Different sections in the scientific text may have different func-
tions [18]. For example, the Introduction section describes more information about the
background of the study, while the Methods section depicts the methods applied. The
section title of the citation sentences where the integrated knowledge phrases occur may
cover some useful contextual information for the knowledge classification.
For each integrated knowledge phrase of each citation-reference pair, we combined the
three features, i.e., the citation sentence, the corresponding reference text, and citation
section title as the contextual information field of the integrated knowledge phrase, and
spliced it with the phrase as the input sequence of the deep learning models.
Classification Module. We applied five deep learning models, including LSTM,
TextCNN, BERT, BERT+SVM, and BERT+XGBoost for the phrase classification
task.
LSTM. LSTM (Long-Short Term Memory) [19] is a type of Recurrent Neural Network
(RNN), but designed to model chronological sequences and their long-range depend-
encies more precisely than conventional RNNs. Therefore, it could provide more long-
distance contextual information. The contextual information and integrated knowledge
phrases were embedded respectively in the embedding layers by using the GloVe model
in this paper.
TextCNN. TextCNN [20] is a text classification technique using the Convolutional Neu-
ral Network(CNN). CNN is a kind of artificial neural network, in which the output of
each layer is used as the input of the next layer of the neuron. Generally, it includes
6
four parts, including embedding layer, convolutional layer, pooling layer, and fully con-
nected layer. The input layer of the model is the same as the LSTM model.
BERT. BERT [17] is not only a pre-trained language representation model, but can also
act as a classifier, if we fine-tuned the model with just one additional output layer. For
each integrated knowledge phrase, the text sequence with contextual information and
integrated knowledge phrase was fed to the BERT model, and was embedded into vec-
tors in the embedding layers, including token embedding, segment embedding, and po-
sition embedding. Then, these embeddings were spliced in the fully connected layer,
and fed to the output layer, which acted as a classifier for predicting the function label
of integrated knowledge phrase.
BERT + SVM. SVM (Support Vector Machine) [21] is a kind of generalized linear
classifier, which determines the best decision boundary between vectors that belong to
a category and that do not belong to it. In this model, we used the BERT to extract the
pre-trained embeddings of our text features and fed them to the SVM classifier.
BERT + XGBoost. XGBoost [22] represents “Extreme Gradient Boosting”, which is an
implementation of gradient boosted decision trees. It is designed to push the limit of
computations resources for boosted tree algorithms and has achieved great performance
and speed in applied machine learning recently. We also used BERT to vectorize the
input text sequence in this model.
2.5 Evaluation
We chose precision, recall and F1 score to measure the performance of the integrated
knowledge phrases identification and classification methods.
For the identification task, precision is calculated as the number of correctly identified
phrases divided by the total number of phrases identified through the automatic meth-
ods. Recall value is calculated by the number of correctly identified phrases divided by
the number of phrases should be identified through the annotation.
In the classification task, we used weighted precision and recall scores of all the cate-
gories. For each category, precision is calculated as the number of correctly labelled
integrated knowledge phrases by the total number of integrated knowledge phrases of
this category the classified models recognized, and recall is calculated as the number
of correctly labelled integrated knowledge phrases of this category by the number of
phrases that should be labelled as this category. For the overall precision and recall
score of the test dataset, we assign weight to the precision and recall value of each
category by the proportion of phrases in each category, and then plus all the weighted
precision scores and weighted recall scores of all categories.
Finally, each F1 score is twice of the multiplication value of precision and recall score
divided by the sum of the two values.
7
3 Experiments and Results
3.1 Experimental Datasets
We constructed an eHealth dataset to test the performance of our methodology frame-
work. XML files of 3,221 eHealth papers published from 1999 to 2018 were down-
loaded from two high impact journals, Journal of Medical Internet Research and JMIR
mHealth and uHealth. The metadata of the references were complemented from Web
of Science (WoS) and PubMed. Overall, we obtained 199,461 citation-reference pairs.
Two datasets were constructed for the two tasks in our methodology. For the identifi-
cation procedure, we randomly selected 100 citation-reference pairs to manually anno-
tate the matched phrases and obtained 105 matched phrases in total. For the classifica-
tion task, we selected 45,166 matched phrases identified in our previous study [9],
which were labelled as “Research Methodology”, “Technology”, “Entity”, “Data”, and
“Theory”. It was randomly divided into ten folds, eight of them for training, which
contains 36,133 phrases; one for validation, containing 4,517 phrases; and the remain-
ing one, including 4,516 phrases, for test.
3.2 Results of Integrated Knowledge Phrases Identification
We chose the combination of direct and indirect matching approach, which was used
in the previous study [9], as the baseline method. In this paper, we applied a new ap-
proach, Word Embedding + Cosine Similarity, for improvement. Two word embedding
models, GloVe and BERT, were compared. The threshold of cosine similarity was tun-
ing among 0.7, 0.8, and 0.9. The evaluation results in Table 2 show that the baseline
approach of our method already has a good performance, with precision of 1.0 and F1
score of 0.900. However, the recall value is relatively low. Regarding the new ap-
proaches, all of them obtained a higher recall value than the baseline method. This
means that we could recognize more integrated knowledge phrases with the new ap-
proach, which is more effective for the knowledge integration analysis. We observe
that with the increase of the threshold of cosine similarity, the precision of the method
is rising, while the recall value is decreasing. To comprehensively measure the perfor-
mance of our new method, we further calculated the F1 score. It demonstrates that the
Baseline + BERT approach with the 0.9 cosine similarity threshold has the greatest
performance, with F1 value of 0.907.
Table 2. Identification results on the standard dataset.
Methods Recall Precision F1
Baseline 0.819 1.000 0.900
Baseline + GloVe (0.7) 0.857 0.823 0.840
Baseline + GloVe (0.8) 0.857 0.836 0.846
Baseline + GloVe (0.9) 0.848 0.948 0.895
Baseline + BERT (0.7) 0.867 0.715 0.784
Baseline + BERT (0.8) 0.848 0.919 0.882
Baseline + BERT (0.9) 0.838 0.989 0.907
8
3.3 Results of Phrases Classification
Five models were trained in the training dataset, and the hyperparameters of the trained
models were tuned in the validation dataset. Then, the test dataset was used for the
evaluation. We calculated the weighted indicators on the overall test dataset as well as
on the unknown phrases dataset. The unknown phrases dataset contains 291 phrases in
the test dataset but neither occur in the training dataset nor in the validation dataset. We
considered these unknown phrases would measure the generalization ability of the
models better than the whole test dataset since some phrases in the test dataset have
occurred in the training and validation dataset and the models may have remembered
the features of these phrases, although with different contextual information.
As shown in Table 3, the models with BERT embedding, i.e., BERT, BERT+SVM and
BERT+XGBoost, appeared more effective than other deep learning models, i.e., LSTM
and TextCNN. The BERT model itself already has a powerful semantic understanding
and syntax analysis capabilities, which achieved the greatest performance on the overall
test dataset, with a precision of 0.980, a recall of 0.980 and a F1 score of 0.980. As for
the unknown phrases dataset, BERT+XGBoost is the best model. This result reflects
the high efficiency, flexibility and portability of the XGBoost algorithm.
Table 3. Classification results on the test dataset.
Methods Test Dataset Recall Precision F1
(weighted) (weighted) (weighted)
LSTM Overall phrases 0.851 0.874 0.849
Unknown phrases 0.649 0.669 0.647
TextCNN Overall phrases 0.956 0.957 0.956
Unknown phrases 0.797 0.813 0.795
BERT Overall phrases 0.980 0.980 0.980
Unknown phrases 0.811 0.811 0.810
BERT+SVM Overall phrases 0.971 0.972 0.971
Unknown phrases 0.842 0.838 0.839
BERT+XGBoost Overall phrases 0.973 0.973 0.972
Unknown phrases 0.856 0.863 0.842
4 Conclusion
In this paper, we provided a new methodology to automatically identify the explicit
integrated knowledge phrases from citation contexts in interdisciplinary field papers
through word embedding techniques, and then utilize several deep learning models to
classify the functions of the integrated knowledge phrases. The eHealth field was taken
as a case of interdisciplinary field to evaluate the performance of our methodology. The
results show that BERT has a great performance, not only as a pre-trained language
representation model but also as a classifier. In the integrated knowledge phrases iden-
tification process, it obtained Recall, Precision and F1 scores of 0.838, 0.989 and 0.907
9
respectively when combined with the baseline string match method. Meanwhile, it
achieved the weighted Recall, Precision and F1 scores of 0.980, 0.980 and 0.980 on the
overall test dataset in the classification task. Moreover, when utilized the XGBoost al-
gorithm along with the BERT model, the model achieved the greatest performance on
the unknown phrases dataset, with weighted Recall, Precision and F1 scores of 0.856,
0.863 and 0.842.
In general, this paper is one of the primary works to apply the word embedding tech-
niques and deep learning models in the integrated knowledge phrases identification and
classification of an interdisciplinary field. This automatic methodology would contrib-
ute to the deep investigation of knowledge integration in an interdisciplinary field from
the content perspective. In addition, it could be applied to the knowledge interaction
exploration between any source and target publications.
However, there are also some limitations in this study. First, in the integrated
knowledge phrases identification step, we only considered the knowledge explicitly
integrated from the references to the citation sentences, but did not include other
knowledge integration forms to comprehensively investigate the knowledge integration
of an interdisciplinary field. Second, the cited reference texts in this article were repre-
sented by the metadata of the references rather than cited texts identified in the full text
of references. The integrated knowledge not contained in the metadata of references
may be lost. Finally, the dataset we used just covers two journals in the eHealth field,
more annotation datasets from various disciplines are further needed to test the perfor-
mance of our methodology.
References
1. Ba, Z., Cao, Y., Mao, J., & Li, G.: A hierarchical approach to analyzing knowledge
integration between two fields—a case study on medical informatics and computer sci-
ence. Scientometrics 119(3), 1455-1486 (2019).
2. Fortunato, S., Bergstrom, C. T., Börner, K., Evans, J. A., Helbing, D., Milojević, S., ...
& Barabási, A. L.: Science of science. Science 359(6379), eaao0185 (2018).
3. Leydesdorff, L., & Rafols, I.: Indicators of the interdisciplinarity of journals: Diversity,
centrality, and citations. Journal of Informetrics 5(1), 87–100 (2011).
4. Borgman, C.L., & Rice, R.E.: The convergence of information science and communi-
cation: A bibliometric analysis. Journal of the American Society for Information Sci-
ence 43(6), 397–411 (1992).
5. Chang, Y. W., & Huang, M. H.: A study of the evolution of interdisciplinarity in library
and information science: Using three bibliometric methods. Journal of the American
Society for Information Science and Technology, 63(1), 22-33 (2012).
6. Leydesdorff, L., & Probst, C.: The delineation of an interdisciplinary specialty in terms
of a journal set: The case of communication studies. Journal of the American Society
for Information Science and Technology 60(8), 1709–1718 (2009).
7. Xu, J., Bu, Y., Ding, Y., Yang, S., Zhang, H., Yu, C., & Sun, L: Understanding the
formation of interdisciplinary research from the perspective of keyword evolution: A
case study on joint attention. Scientometrics 117(2), 973-995 (2018).
10
8. Engerer, V.: Exploring interdisciplinary relationships between linguistics and infor-
mation retrieval from the 1960s to today. Journal of the Association for Information
Science and Technology 68(3), 660-680 (2017).
9. Mao, J., Wang, S., & Shang, X.: Investigating interdisciplinary knowledge flow from
the content perspective of citances. In: EEKE@JCDL 2020, pp. 40-44 (2020).
10. Elkiss, A., Shen, S., Fader, A., Erkan, G., States, D., & Radev, D.: Blind men and ele-
phants: What do citation summaries tell us about a research article? Journal of the Amer-
ican Society for Information Science and Technology 59(1), 51–62 (2008).
11. Kondo T, Nanba H, Takezawa T, et al.: Technical trend analysis by analyzing research
papers’ titles. In: Proceedings of the Language and Technology Conference on Human
Language Technology, Challenges for Computer Science and Linguistics, pp. 512-521.
Springer, Heidelberg (2009).
12. Heffernan K, Teufel S.: Identifying problems and solutions in scientific text. Scien-
tometrics 116(2), 1367-1382 (2018).
13. Zhao, S., Su, C., Lu, Z., & Wang, F.: Recent advances in biomedical literature mining.
Briefings in Bioinformatics, bbaa057 (2020).
14. Wei, C. H., Allot, A., Leaman, R., & Lu, Z.: PubTator central: automated concept an-
notation for biomedical full text articles. Nucleic acids research 47(W1), W587-W593
(2019).
15. Neumann, M., King, D., Beltagy, I., & Ammar, W.: Scispacy: Fast and robust models
for biomedical natural language processing. arXiv preprint arXiv: 1902.07669 (2019).
16. Pennington, J., Socher, R., & Manning, C. D.: Glove: Global vectors for word represen-
tation. In: Proceedings of the 2014 conference on empirical methods in natural language
processing (EMNLP), pp. 1532-1543(2014).
17. Devlin, J., Chang, M. W., Lee, K., & Toutanova, K.: Bert: Pre-training of deep bidirec-
tional transformers for language understanding. arXiv preprint arXiv:1810.04805
(2018).
18. Bertin, M., Atanassova, I., Gingras, Y., & Larivière, V.: The invariant distribution of
references in scientific articles. Journal of the Association for Information Science and
Technology 67(1), 164-177(2016).
19. Hochreiter, S., & Schmidhuber, J.: Long short-term memory. Neural computation 9(8),
1735-1780 (1997).
20. Kim, Y.: Convolutional neural networks for sentence classification. arXiv preprint
arXiv: 1408.5882 (2014).
21. Cortes, C., & Vapnik, V.: Support-vector networks. Machine learning 20(3), 273-297
(1995).
22. Chen, T., & Guestrin, C.: XGBoost: A Scalable Tree Boosting System. In: Proceedings
of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and
Data Mining, pp. 785–794. ACM, New York, USA (2016).