The terminologies in different disciplines vary greatly, and the annotated corpora are scarce, which have limited the portability of information extraction models. The content of scientific articles is still underutilized. This paper constructs an intelligent platform for information extraction and knowledge mining, namely IEKMMD. Two innovative technologies are proposed: Firstly, a phraselevel scientific entity extraction model combining neural network and active learning is designed, which can reduce the model's dependence on large-scale corpus. Secondly, a translation-based relation prediction model is provided, which improves the relation embeddings by optimizing loss function. In addition, the platform integrates the advanced entity recognition model (spaCy.NER) and the keyword extraction model (RAKE). It provides abundant services for fine-grained and multi-dimensional knowledge, including problem discovery, method recognition, relation representation and hot spot detection. We carried out the experiments in three different domains: Artificial Intelligence, Nanotechnology and Genetic Engineering. The average accuracies of scientific entity extraction respectively are 0.91, 0.52 and 0.76.
• Computing methodologies • Artificial intelligence • Natural
language processing • Information extraction
With the progress of science and technology, there are more and
more fields and scientific articles. Information extraction and
knowledge mining in the specific field enable scholars to quickly
grasp the overall outline of information, and track the
development of fine-grained knowledge. There are many mature
models to extract information from texts, such as BiLSTM-CNN
[
How to extract information without massive annotated corpus
is a big challenge. Active Learning (AL) [
Relation predication is another basic technology for
knowledge organization. Translation models see relation as a
process of translating the head entity to the tail entity, which have
been widely used to predict relations. There are some classic
translation models proposed from different perspectives: TransE
[
This paper aims to construct an intelligent platform for information extraction and knowledge mining, which can be used in multiple domains without much human intervention. The main contributions are as follows: 1). with the limited annotated corpus, an effective method combining neural network with active learning recognizes scientific entities in multiple domains; 2). By optimizing the loss function, an improved translation model represents the semantic vectors more accurately and reaches the convergence state faster with a small loss score compared with the original model. 2
The technology framework of our platform is shown in Figure 1.
This platform includes two innovative technologies: 1) the model Copyright 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). combining neural network with active learning extracts "problem" and "method" entities, 2) the improved translation model predicts relations between "problem" and "method" entities. At the same time, the platform integrates two excellent tools (spaCy.NER1 and RAKE2 ) to recognize the named entities and keywords. Finally, this platform provides a variety of knowledge services for researchers, including problem discovery, method recognition, relation representation and hot spot detection. Besides, the analyzers can perform richer downstream tasks based on our platform, such as discipling analysis, trend explosion, new technology detection, and so on.
discovery Method recognition Relation represent Hotspots detection
prediction Named entity extraction Keyword extraction AI GIS Bio ……
Scientific entity recognition contributes to extract phrases from scientific articles. These phrases consist of several words which describe the focus of article or the method proposed by author. In order to reduce the dependence on annotated corpus, this paper provides a semi-supervised learning model combining neural network with active learning.
The framework of the information extraction model is shown in Figure 2. Firstly, the learning engine trains the parameters of neural network by using a small number of annotated samples (dozens of abstracts with semantic labels). Then, the trained neural network predicts the labels of unannotated samples and inputs the predicted scores to the selecting engine. Secondly, according to the active learning strategies, the selecting engine decides which samples are valuable and should be annotated manually. Only the top 10% most valuable samples are labelled by experts. Thirdly, the manually annotated samples are added into the training set to re-train the neural network, in order to improve the performance of label prediction. The whole process runs repeatedly until the performance of model has no significant optimization. Finally, the trained model predicts the “problems” 1 https://spacy.io/ 2 https://github.com/aneesha/RAKE and “methods” for all the unlabeled articles. More details about parameter setting will be discussed in Section 3.1.
feature jointing
CRF decoding unlabeled samples
LearningEngine reepcmWmnorfeoofwgaotoovavnchterrieieiododtldienon PrBLIIe---dattOOOOOOtaabaisscsekktkled
predicted score lossscore< threshold all samplesarelabelled
No labeled samples we provide noavel method for face emotion recognition we provide noavel method for face emotion recognition
CNNcharacter encoding Bi-LSTM word encoding
Yes methods problems SelectingEngine expert annotation selected samples Ssssa…eeemnnn…tttp---12nle Vassslcccu…oooerrr…eees’’’c---o21nre
Margin NSE MNLP LWP
Sample Predicted score sss…eeennn…ttt---12n sssccc…ooorrr…eee---12n
Here we choose CNN-BiLSTM-CRF [
In addition, we propose a hybrid approach for the selecting engine. Firstly, the value score of each unlabeled sample is respectively computed by four different types of active learning strategies, and the sum of them is set as the final value score. Secondly, the value scores are listed in descending order, only the top 10% most valuable samples are selected to be annotated manually in each iteration.
This paper picked out three classical strategies from the
uncertain sampling methods: margin [
Relation prediction decides whether a "problem" and a "method" is related or not. That means if a “problem” is related to a “method”, the method can be used to solve this problem.
Translation model sees the relation in the triple (head entity,
relation, tail entity) as a translational between two entities. There
is a series of translation models. TransE [
After comparing the performance of various translation models, we choose TransH to predict relations, which keeps balance between accuracy and efficiency. To solve the problems of oneto-many, many-to-one, many-to-many relations, TransH generates the relation-specific translation vector in the relation-specific hyperplane rather than in the same space of entity embeddings. h h⊥ t t⊥ dr wr
As shown in Figure 3, the relation in its hyperplane has a translation vector , the head embedding ℎ and the tail embedding in have their projection vectors ℎ⊥ and ⊥ . The defined score function is: ||ℎ⊥ + − ⊥||22.
However, the original TransH model does not match our goal exactly. We achieved three improvements. 1)
TransH constructs the negative samples by replacing the head or tail entity with others in the positive samples. However, the replaced one may also be correct because of synonyms, which introduced many false negative labels into training. Considering that there are only two types of relationships, we simply construct the negative samples by modifying the correct relationship into its antonym. By this change, it is more convenient to construct a balanced annotated corpus. Moreover, the score function (ℎ, ) is redefined as Equation (1), which aims to move the attention from entity to relation.
(ℎ, ) = ||(ℎ ⊥ − ⊥) − ||22 (1) 2)
Comparing with the original model that initializes the entities with the random vectors, we use the word2vec model to 3) 2.3 generate the semantic representation of all head and tail entities.
To improve the ability of feature learning for the unknown entities, we add one hidden layer of linear transformation respectively for the head entities and tail entities.
We use an enterprise open source toolkit spaCy.NER to recognize the named entities. spaCy.NER implements a very fast and efficient system based on the statistical machine learning algorithms, which can recognize 18 entity types, such as Person, Organization, Location, Geopolitics entity.
Furthermore, keyword extraction is achieved by the open
source toolkit RAKE (Rapid Automatic Keyword Extraction).
RAKE is an automatic keyword extraction technique. Based on
the statistical method, RAKE outperformed TextRank and other
supervised learning models, which obtained a high F value [
We evaluate the performance of information extraction of IEKMMD in the field of Artificial Intelligence (AI). There are two datasets be used.
1) The top 100 AI conferences were picked out by the domain experts, and their abstracts were acquired from NSTL database3, total in 9753 sentences. Next, we built the truth datasets. Each sentence is annotated synchronously by two students in the corresponding subjects (task, method or other). The annotation results are checked by one expert. The annotation format is shown as Figure 4. The AI annotated corpus contains 26,0000 tokens.
We use active learning
extract information | O | O |
| B-method I-method
| B-task
|
I-task to | O
2) FTD datasets4 shared by Stanford University in the field of Computational Linguistics. It comes from the Conference of the Association for Computational Linguistics and ranges from 1965 to 2009, which containing four types of labels: focus, technique, domain and other, in total 2628 sentences.
In addition, we show the effect of knowledge mining in three different kinds of domains. We choose three popular keywords (Neural Networks, Nano Structure and Genetic Engineering) that respectively respect the subjects of Computer Science, Material and Medicine to acquire abstracts from NSTL database. 200
4 https://nlp.stanford.edu/pubs/FTDDataset_v1.txt abstracts of each subject are randomly selected from SCI journals and are used to verify the practical application effect of IEKMMD.
We set the baselines only using the CNN-BiLSTM-CRF (CBC) model trained on all annotated samples. For each dataset (AI or FTD), the best performance is as the baseline, so as to detect whether active learning helps reduce the scale of annotated corpus for supervised learning models. The scale of training sets and the best F1 scores of CBC model are shown in Table 1.
In the model of IEKM-MD, initially only 0.01% annotated samples are used to carry out the cold starting process, then the highest valuable samples (10%) are added into the training sets in each iteration. Only if the F1 score of IEKM-MD reaches the baseline, can the learning process be stopped. The label scales and F1 scores of AI and FTD datasets in each iteration are show in Table 2. The results reflect that Neural Networks achieved the best performance with 0.93 accuracy of problem extraction and 0.89 accuracy of method extraction. The average accuracy of three fields reveals that problem extraction has a better score than method extraction. The first reason is that the total mentions of problem are smaller than methods, and they are usually described in the noun phrases, which contribute to an easier pattern to be caught by model. The second reason is that one article may contain multiple methods, which are modified by multiple attributives or adverbials, making it more challenging to recognize the complete methods.
However, our platform performed worst in the field of Nano Structure. This may because that the articles of Nano Structure include many complex and specialized terms in the subjects of biology, physics, chemistry, electronics, and metrology. Our platform still lacks the professional knowledge to learn the specific features.
The extracted top 10 problems of three fields are shown in Table 4, which reveal that Neural Networks focuses on the classification, prediction and recognition problems of data and images in the subject of Computer Science. Nano Structure covers a wide range, including physics, biology, chemistry, and so on, which focuses on the applications on the basic disciplines. Therefore, the extracted problems involve detection, analysis and prediction of energy, atom and medicine. The scope of Genetic Engineering is relatively narrow and is related to drug development, disease treatment, and biological manufacturing in the biomedical field.
Top
Neural Network
Nano Structure
Genetic Engineering
Classification
Prediction Pattern recognition
Optimization
By predicting the relations between problem and method, we construct the method-problem networks for different domains. As shown in Figure 5, the methods and problems which were separate in the articles of Neural Network are linked by relation prediction. The red dots refer to methods, and the blue dots refer to problems.
9 10
Specifically, we can get more details from the abovementioned network. By setting the method X-Ray Diffraction (XRD) as a center, Figure 6 reveals that what problems are solved by XRD. They are Assisted Synthesis, Biomedical Application, Biosynthesis of Silver Nanoparticles and so on. Hotspots are the most popular research topics. We use the extracted keywords to pick out the hotspots in multiple domains. As a hotspot, the total number occurring in articles should be increased year by year or keeps a steady top order in last three years. According by this rule, Figure 7 shows the hotspots in the field of Neural Networks. They are distinct from the scientific entities recognized in section 3.1, which have no semantic type but reflect the popularity degree of terms. 4
This paper introduced an innovative and intelligent platform IEKM-MD to extract information and mine knowledge from scientific articles in multiple domains. One contribution is providing a hybrid active learning strategy to solve the problem of annotated corpus scarcity in supervised learning model. Another contribution is designing an improved Translation embedding approach based on TransH model to optimize the performance of relation prediction. Three datasets in Neural Networks, Nano Structure and Genetic Engineering show that our platform is enable to achieve various knowledge services with a high accuracy in multiple domains.
This work is supported by the project “Annotation and evaluation of the semantic relationship between geographical entities in Chinese web texts” (Grant No. 41801320) from the National natural science foundation of China youth science foundation.