hTe extraction of sentences with specific meaning in academic literature is an important work in academic full-text bibliometrics. This research atempts to establish a practical model of extracting conclusion sentences from academic literature. In this research, SVM and SciBERT models were trained and tested using academic papers published in JASIST from 2017 to 2020. eTh experimental results show that SciBERT is more suitable for extracting thesis conclusion sentences and the optimal F1-value is 77.51%.
hTeory of computation~Theory and algorithms for application domains~Machine learning theory~Models of learning
Full-text data of academic literature mainly contains external
characteristics and content characteristics. Since the creation of
citation index by Garfield, citation analysis based on external
characteristics of literature has been widely applied in various
fields. However, due to the limitations of data and technology,
the previous bibliometric have many defects, including rough
statistical method and single indication ability [
The research conclusion sentence refers to the sentence that contains the research conclusion. In the academic full text, research conclusion sentences are divided into citation research conclusion sentences and thesis research conclusion sentences. Citation research conclusion sentences refer to experimental results and conclusions in quotation sentences, such as ‘Taylor’s work shows that the special purpose syntactic parsers perform well on morphological descriptions.’. Thesis research conclusion is the author's statement of his own research results, such as ‘In this way, we extended earlier work to the case that the impact factor can have a value lower than one.’.
Automatically extracting thesis research conclusion sentences can promote the development of automatic summarization and originality evaluation of academic papers. Therefore, this research attempts to construct an automatic recognition model of the thesis research conclusion sentence based on the deep learning techniques. 2 2.1
As for data annotation, first, we use Python's NLTK module to segment the full text of the paper in sentence units. Then, 7 postgraduates majoring in information science manually annotated the sentences. For sentences that are not sure how to label, the decision will be made after group discussion and the experimenter completes the final review. The discriminant criteria of the thesis research conclusion sentence are as follows: (1) Semantically speaking, the sentence content is a summary of the author's own work experience, observations or actual research results. (2) The content of the sentence can be a reasoning and qualitative interpretation of the experimental Copyright 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). results, but it cannot be a straightforward description of the data of the experimental results.
Data imbalance, that is, the gap between the number of positive
and negative samples used to train the model is too large, which
is one of the most widespread problems in contemporary
machine learning [
Before the start of the formal experiment, we tested different hyper parameters combinations on a small part of the experimental corpus to explore the optimal settings for SVM and SciBERT. At the same time, considering the performance of the computer hardware used in the experiment, the final hyper parameters are set as follows. SciBERT (scibert-scivocabuncased): 256 for Maximum sequence length, 64 for batch size, 2e-5 for learning rate, 3 for training epoch, case insensitive. The penalty function of SVM is set to 2, the kernel function is RBF, and TF-IDF is used to vectorize the text. The research uses a tenfold cross-validation strategy, and the operating effect of the model is measured by Precision, Recall and F1-Value. Table 2 shows the results of the experiment.
SVM
MIN AVG MAX MIN AVG
Compared to the sentences extracted by the SciBERT model with the manually annotated sentences, recognition errors of the SciBERT that have been discovered are as follows: (1) Recognizing the sentence describing the graph as the thesis research conclusion sentence. The possible reason for this problem is that the sentence describing the graph normally has phrases such as "as shown in" at the beginning, and these words are also important features of the thesis research conclusion sentence. (2) Recognizing research hypothesis sentences as thesis research conclusion sentences. According to observations, the thesis conclusion sentence is similar to the hypothesis sentence in terms of grammar and semantics. (3) Recognizing citation conclusion sentences without quotation mark as thesis research conclusion sentence. It indicates that some special words or symbols may afect the judgment of the model. 4
This research provides a practical method for extracting conclusion sentences of thesis research from academic literature. This research shows that SciBERT is relatively superior than SVM for automatically extracting thesis conclusion sentences. This research uses a negative sample strategy to alleviate the problem of data imbalance and to enable faster model optimization, which may reduce the complexity of negative samples. Therefore, data augmentation needs to be achieved by adding more positive samples in the future. In addition, the position of the sentence in the article also needs to be considered to optimize the performance of the model. Finally, some research conclusion sentences extracted contain pronouns and do not have perfect semantics when read alone. Therefore, research on Co-Reference Resolution should be carried out.
hTe authors acknowledge the National Natural Science Foundation of China (Grant Numbers:71974094) for financial support.