Entities and relations concisely reflect important information related to the subject matter of the literature, which is essential for understanding and analyzing it. In scientific research, methods are indispensable tools and important research objects for solving scientific problems (methods include tasks, discipline-specific methods, models, algorithms, and metrics, etc.). Therefore, 'method' is indispensable for the understanding and analysis of academic literature. This paper aims to extract method-like entities and relations from scientific abstracts using a semantically enhanced deep learning model. We explored the impact of linguistic information on the entity and relation extraction task, and to this end, we added additional POS tag information to the word vectors obtained through the pre-trained model to highlight POS tag information, which proved to be superior to the pre-trained word vectors alone. Individually, in the entity recognition part, the token sequence length of entities is considered as the feature, and in the relationship extraction part, performing max pooling over the context between entity candidates has been proven better than full context, additionally, the distance between entity candidates is embedded by us as an additional feature. Entity type is also entered as an additional feature. The sequences of rich token representations constitute a span, over which entities and relations are learned jointly. The results on several datasets show that the embedding of rich semantic information outperforms the original span-based model.
In the era when the data was dense, a large number of papers are published daily[1-3]. For most academic researchers, considering the diversity and explosive growth of research in the field, the speed of reading is much slower than the speed of publication, making it impossible to always access the latest methods from the most recent literature, which means that traditional manual searches to find scientific methods become challenging[4]. Therefore, in order to help scholars form a methodological system for their research directions and obtain the most cutting-edge methods related to the research content while greatly saving the labor and time of researchers, it is essential to study the method to extract methods from large-scale academic literature. As is well known, entity recognition (ER) and relationship extraction (RE) are essential and challenging tasks in natural language processing (NLP), which can be beneficial to information extraction from academic literature. Entity recognition and relationship extraction from academic texts refer to identifying academic entities, such as Task, Method, Metric, and so on, and extracting semantic relations among these entities, e.g Evaluate-For, Used-For, and so on. ER and RE from literature are used in a wide range of academic applications, including academic information retrieval, knowledge graph construction, question answering, article recommendation, etc. The joint model of ER and RE from scholarly texts aims to extract the entity-relationship-entity tuples. For example, the following sentence S1 shown in Figure 1 contains two entities; we delineate the mention with square brackets and the corresponding entity types with suffixes：
Previous research has shown that method entities identified by complex rules are more responsive to searchers' needs than those identified by matching terms in academic literature[5]. As a result, researchers have proposed more complex rules to accomplish the extraction task, including cue words, language patterns, lexicality, word position, etc[6-10]. duck et al.[11] created a named entity recognizer used in bioinformatics called bioNerDS to extract software entities and dataset entities from papers. Noun phrases from academic papers were extracted and scored based on different rules. In the first round, candidate entities are checked to see if they appear in the generated dictionary. In the second round, strong rules extracted from the article, such as version information, references and URLs, are classified into positive and negative rules and assigned different scores. In the last round, some clues, such as specific verbs, and indicative but ambiguous titles, are combined into weak clues and assigned scores. Candidate entities were scored according to their compliance with the rules and judged as method entities based on their final scores.
In this paper, we propose a semantically enhanced term entity relation extraction model to jointly extract method entities and relationships from the abstracts of scientific papers. We use a recent model called SPERT as the baseline, which uses a pre-trained transformer. A shallow entity classifier and a shallow relationship classifier are applied to extract entities and relations, respectively. The transformer generates embeddings of tokens in the abstract, and merges the embeddings of a span of tokens into one. Many natural language processing tasks benefit from the use of linguistic information, such as part-ofspeech tagging, but they are less explored in deep neural models of NER and RE. Petasis et al.[12] believed that named entities were proper nouns (PN), which served as the name of someone or something. From the perspective of ontology, Alfonseca and Manandhar[13] proposed that named entities were objects used to solve specific problems. Borrega et al.[14] defined named entities in detail from the perspective of linguistics, stipulating that only nouns and noun phrases can be used as named entities. Although these definitions are not uniform, it can be sure that named entities at least the vast majority are nouns or noun phrases. The relationships between entity pairs also seem to be related to entity types, for example, we often find "used-for" relations between "generic" entities and "method" entities. In relation extraction, contextual information between candidate entities has been proven to be superior to global contextual information. And in addition to the important information brought by the semantics itself, the distance information between candidate entities should also be an important feature. For this purpose, we counted the distance between relational entities, as shown in Table 1. Through the data, we can find that Conjunction and Feature-of method types basically appear in close entities while other relationship types are more average. By counting the percentage of different relationships appearing before and after the entity type, we found that the entity type characteristics also have an important influence on the relationship, here we take a table as an example, as shown in Table 2. Therefore, we propose a semantically enhanced model. The main contributions of our work are as follows: • We propose an improved joint model of NER and RE for academic texts, in which the type of entity and the distance between candidate entities that potentially constitute the relationship are considered in RE, and the token sequence length of an entity is considered in NER. • We enrich the initial embeddings, the initial embeddings are augmented by semantic information and syntactic information. • Experiments on real data sets validate the effectiveness of our proposed method.
With the development of technology, deep learning methods are becoming the focus of research in the field of machine learning. Deep learning models focus more on the capability of machines and abandon complex feature engineering, which greatly reduces the labor as well as time cost required for extracting feature engineering compared to statistical models of machine learning. Therefore, deep learning has become a very important research direction.
The supervised deep learning method used in relation extraction can solve the main problems of manual feature extraction and error propagation that exist in classical methods. The low-level features are combined to form more abstract high-level features. At present, supervised relation extraction methods mainly include pipeline approaches and joint approaches. 2.1
Pipeline approaches refer to the extraction of relations between entities directly based on the entity recognition already done. The early pipeline approaches mainly used two types of structures, convolutional neural networks (CNNs) and recurrent neural networks (RNNs). Among them, CNNs with diverse convolutional kernels are good for recognizing structural features of the target, while RNNs can fully consider the dependency between long-range words and their memory function is good for recognizing sequences. Zeng et al.[15] used CNN to extract word-level and sentence-level features for the first time, and improved the accuracy of the relationship extraction model by using the hidden layer and softmax layer for relationship classification. Socher et al.[16] first used an RNN approach for entity relation extraction, which uses recurrent neural networks to syntactically parse the sentences in the annotated text and then obtains a vector representation considering the syntactic structure of the sentences after continuous iterations.
As the research progresses, CNN and RNN methods are continuously improved and refined, and many variants are generated, such as long short-term memory (LSTM), and bidirectional long short-term memory (Bi- LSTM), which are improved to solve the gradient disappearance. Xu et al.[18] proposed an LSTM-based relation extraction method based on the shortest path of syntactic dependency analysis tree, incorporating features such as word vector, part of speech tags, WordNet, and syntax, using maximum pooling layer, softmax layer, etc. In addition, with the application of graph convolutional network (GCN) in the field of natural language processing, GCN has been increasingly used for mining and exploiting potential information between entities, providing new ideas for solving relation overlap and entity overlap, and thus further promoting the development of relation extraction. Schlichtkrull et al.[19] proposed the use of relational graph convolutional neural networks (R-GCNs) on two standard knowledge bases to accomplish link prediction and entity classification, respectively, where link prediction extracts missing relations and entity classification completes the missing attributes of entities; Zhang et al.[20] proposed an extended graph convolutional neural network that can effectively handle arbitrary dependency structures in parallel and facilitate the extraction of entity relationships to effectively utilize negative class data.
Although the Pipeline approach is easy to implement, the entity model and the relation model can use independent datasets and do not need to label both entity and relation datasets, there are several disadvantages:
a) error accumulation: errors in entity extraction will affect the performance of the next step of relation extraction.
Table 1: Relation entity distance of SciERC.
Ave[>11]*4 0.6% 0.3% 3.0% 1.1% 2.1% 4.5% 3.1% All b) entity redundancy: the redundant information brought by the candidate entity pairs without relationships will enhance the error rate and increase the computational complexity, since the extracted entities are firstly paired, and then extract relations between entity pairs.
c) lacking interaction: the intrinsic connection and dependency between the two tasks are ignored. 2.2
Joint approaches can further use the potential information between the two tasks to mitigate the disadvantage of error propagation. The difficulty of joint approaches is how to enhance the interaction between the entity model and the relation model. In early works, the connection method relied heavily on fine-grained feature engineering to establish the interaction between NER and RE[21-23]. Recently, end-to-end neural networks have proven successful in extracting relational triples[15,24-26], becoming the mainstream for joint entity and relation extraction.
Based on their differences in encoding task-specific features, most existing approaches can be divided into two categories: sequential encoding and parallel encoding. Sequential encoding generally encodes task features in the sequential order of NER and then RE, and this encoding approach can keep the later encoded features from affecting the first encoded features directly away, resulting in unbalanced intertask interactions. Zeng et al.[27] and Wei et al.[28] are typical examples of this category. They extracted features for different tasks in a predefined order. Parallel encoding uses two independent encoders to generate task features, which have no interaction other than shared input, leading to insufficient intertask interaction, and in contrast to sequential encoding, models built based on this scheme do not need to worry about the effect of encoding order. , and encoded the entity and relationship information separately, and finally completed the extraction of task-specific features in two separate submodels, respectively. Both encoding methods have their own drawbacks, the inter-task interaction, and in contrast to sequential encoding, models built based on this scheme do not need to worry about the effect of encoding order, and encoded the entity and relation information separately, and finally completed the extraction of task-specific features in two separate submodels, respectively. Both encoding methods have their own drawbacks, the inter-task interaction in sequential encoding is oneway with a specific order, while the problem with parallel encoding is that they only retain the shared features and actively ignore the features that are task-beneficial separately for each task.
In this section, we provide a detailed explanation of our model, the framework is shown in Figure 2. We develop a joint model including five components: 1) an embedding layer, which converts tokens into embedding vectors, 2) a POS encoder, which converts tokens into part-of-speech, 3)a fusion module, which fuses word embedding and part-of-speech embedding into one vector,4)a shallow entity classifier, which classifies any possible sequence of consecutive tokens and 5)a shallow relationship classifier, which classifies relation for any given set of entity pairs. 3.2
Given a sequence of sentences = { , , … , }, the embedding layer transforms the sentences into the vector matrix, in which each token in the sentence is represented by a pre-trained embedding. The embedding of each token consists of two parts: pre-trained transformer and POS embeddings.
a) Pretrained transformer: The recognition and relationship extraction of academic terminology entities are different from the conventional entity relationship extraction, which is more specific and the relation between terms is more abstract. Traditional methods are based on manually generated features, while pre-training techniques are now widely used in deep learning, which have achieved good performance in computer vision, natural language processing, and other fields. Usually, a model that has been trained on large-scale data can achieve satisfactory performance with simple training, i.e., finetuning. Obtaining a high-quality initial value of parameters with the help of pre-training techniques not only reduces the training burden but also helps to improve the model generalization ability.
However, as Bert received pre-training on general texts from Wikipedia and book corpora, its performance on domain-specific tasks proved to be suboptimal in several previous works [29-30]. These empirical findings have driven the development of domain-specific pre-trained language models. For example, SciBERT in the scientific domain and BioBERT and ALBERT in the biomedical domain. so the domain-specific pre-trained model SciBERT and BioBERT was used for domain academic entities, through which academic entity features were obtained to obtain better feature representations.
We split each sentence into a sequence of tokens = { , , , … , , }, where [CLS]
and [SEP] are special symbols. The [CLS] captures the contextual information of the text, while the [SEP]
acts as a separator to separate adjacent sentences between them. We use transformer to generate
pretrained embeddings as in (
b) POS Tag: Part of speech is the important information a word carries, which reflects the components that the word plays. Dependency syntactic parse in natural language processing is the conversion of sentences into trees based on POS tags. Xu et al.[18] proposed an LSTM-based relation extraction method based on the shortest path of the syntactic dependency analysis tree, incorporating features such as word vector, part-of-speech, WordNet, and syntax, and using maximum pooling layer, softmax layer, etc. for relation classification. The addition of POS ultimately achieves the goal of effect enhancement. So we take POS tags into consideration. We generate part of speech tags for the input sentences and assign the POS tag of the parent word to each child word tag it generates. We use a directed embedding matrix to generate a embedding sequence = { , , , … , , }for each positional tag of dimension d2. The BERT embedding of the token and the lexical POS embedding are then stitched together to obtain a new vector representation of dimension. We use a directed embedding matrix to generate embeddings for each positional tag of dimension d2.
=
, , ,⋅⋅⋅, , where the output embedding is a combination of the above two vectors. The dimension of the output embedding is + . 3.3
To detect entities, a representation ,⋅⋅⋅, entity.
= max ,⋅⋅⋅, where , and 1+ 2 is the embedding dimension.
In order to study the influence of the length of entity span, we counted the lengths of all entity spans,
as shown in Table 3. Overall, the percentage of entity span lengths on the interval (
where , and 1+ 2+ 3 is the embedding dimension.
Finally,[CLS] , which represents the sentence context, is concatenated with ′ to obtain the vector ″ , which is passed through a softmax classifier to predict the entity type.
vector is obtained by doing max-pooling of the embedding of a sequence of tokens of length k of each successive possible constituent ′
= ∘ ″ = ′ ∘
= ⋅ ″
+
(
Those spans that are classified as by the entity classifier are filtered out. For the remaining spans, the task is to identify the relation between every pair of them. Consider a pair of spans , occurs before 2 in the input sentence. We assume relations to be asymmetric, so the relation directed from 1 to 2 may be different from that directed from 2 to 1, and each of them must be separately where 1 classified. We take the representations, ,⋅⋅⋅,
, where c is the embedding of the first token following
1 and c is that of the last token preceding 2 in the sentence, and max-pooling:
2 and s2 to s1 are separately encoded as in (
Where , and 1+ 2 is the embedding dimension.The candidate relations from span 1 to , = max
,⋅⋅⋅, = ′ ∘ ,
∘ ′ = ′ ∘ ,
∘ ′
The results are passed through a simple classifier with a confidence interval (results beyond are considered to have this relationship) and a sigmoid activation function to predict the type of relation. We have tried to include logical information and distance between candidate entities to predict entity type. candidate entities.
We train a specific width embedding matrix
to obtain an embedding for the distance between
(
The loss function of the joint model is the sum of the cross-entropy losses of the entity classifier and the relational classifier. End-to-end training is performed by back-propagation, and the transformer is fine-tuned during the training process. To train the entity classifier, we used real entity spans as positive samples and added some non-entity spans as negative samples.
Our goal is to extract method entities and relations from the scientific literature, so we evaluated our model on SciERC, a dataset from the scientific literature that contains both entities
Length of tokenized entities 12.9% 12.7% 4.3% 9.4% 8.7% 1.1% 9.0% [>10] 8.3% 7.3% 3.0% 4.2% 2.8% 0.0% 4.5%
Ave[>10]*4 1.8% 1.4% 1.7% 1.9% 0.7% 0.0% 0.6% and relations.
The SciERC dataset is constructed from 500 abstracts of papers in the field of artificial intelligence, with a total of 2687 sentences. It contains six scientific entities as well as seven relations. The six scientific entities are Task, Method, Metric, Material, Other-Scientific-Term, and Generic, while the seven methods are Compare, Conjunction, Evaluate-For, Used-For, Feature-of, Part-of, and Hyponym-of. We follow the official cutoff methods: train (1861), dev (275), and test (551).
We use the standard Precision (P)(
stand for true positive, false positive, and false negative, respectively. where ,
and
Performance on SciERC. We report the performance on the SciERC dataset in Table 4. We compare the experimental results for six different models, and the F1 values for all three tasks improve compared to the baseline SPERT. Compared to the 0.36% F1 value improvement for the NER task, the performance improvement for the RE task is relatively significant, with Boundaries RE and Strict RE improving by 1.26% and 2.23%, respectively.
The ablation study in Table 5 shows the effect of removing entity span length, entity type, and relation distance on the final classification score. In the ablation experiments, we take the average of the three best results out of 20 experiments. Intending to compare the upper limits of the effect of feature. We observe that removing the entity span length decreases the F1 value of NER by 0.4% and has an impact on the subsequent RE tasks as well. Removing the entity type feature reduced the F1 scores of boundaries RE and strict RE by 1.3% and 1.8%, respectively, and removing the relation distance reduced the F1 scores of boundaries RE and strict RE by 1.16% and 1.0%, respectively, so we can conclude that entity = = 1 =
∗∗ type has a significant effect on relation extraction, especially on strict RE, while as the similar distance feature, the improvement of relationship distance for RE is significantly higher than that of entity span for NER.
We propose a semantically enhanced deep-learning model for extracting entities and relations from the scientific literature. We explored the impact of linguistic information on entity and relation extraction tasks, for which we added additional lexical information to the word vectors obtained through the pretrained model to highlight lexical information, which proved to be superior to the pre-trained word vectors alone. Individually, in the entity recognition part, the length of the entity's token sequence is considered as a feature, while in the relation extraction part, entity type and relation distance are added, which also improves the accuracy of the task. In the future, extending the in-sentence feature information to inter-sentence contextual information is a promising challenge.
This research was supported by the Young Scientists Fund of the National Natural Science Foundation of China under Grant[71904078], Natural Science Foundation of Jiangsu Province of China under Grant[BK20190793], the Postgraduate Research and Practice Innovation Program of Jiangsu Province of China under Grant[GJFXW21001].