Automatically extracting fine-grained S&T problems from think-tank reports written by numerous experts, has become one of the efective ways to perceive the global trend of S&T development. We transform the automatic identification task for fine-grained S&T problems into a multi category S&T entity extraction task with contextual semantics. To address the shortage of high-quality data sets and fully exploit the potential of LLMs, we take LLMs as annotators and puts them into an active learning loop to determine which samples to annotate eficiently. During the cyclic data annotation process, we simultaneously trained the target's entity extraction model ”RoBERTa-BiLSTM-CRF”. Finally, the model achieved an F1 value of 86.02% in our task. The efectiveness and reliability of the model were verified by comparing it with the benchmark model through experiments. This study to some extent solves the problem of manually annotating dataset dependencies, while providing high-quality data support and efective model methods for mining and analyzing fine-grained S&T problems.
CEUR ceur-ws.org
The think tank is composed of multidisciplinary experts in a country and gathers national intellectual resources, which is an important force to influence government decision-making and promote social development. Usually, think tank reports tend to focus on major issues of great concern to the national government or the public, which represent indicators and weather vane of national policies and scientific research, and have high intelligence values. Therefore, the automatic extraction of scientific ports can further clarify policy and public concerns eficiently and objectively. This paper defines ”fine-grained
tics”.
Most of the S&T problem representations extracted
by researchers in the past have adopted several
methods such as manual annotation, rule-based matching,
machine learning-based, hybrid model-based, and deep
ual annotation to analyze the distribution of methods in
diferent academic journals. However, those expert
annotation methods are relatively highly accurate, but costly
Joint Workshop of the 5th Extraction and Evaluation of Knowledge
Entities from Scientific Documents and the 4th AI + Informetrics
(EEKE∗Corresponding author.
ods, Xuesi Li et al. [
Since 2020, large language models (LLMs) have
exhibited remarkable few-shot performance in information
extraction tasks, with only a few demonstrations and
well-designed prompts. Under the prevalent
“Languageand technological problems mentioned in think tank re- speech into the algorithm to enhance its performance.
logical solutions, and technological routes”, and further
limited conditions such as application scenarios, techno- Long Short-Term Memory networks to achieve
perforanalogizes them as ”S&T entities with contextual seman- ods (such as SVM), which also proved the usefulness of
learning-based methods. H. Chu and Q. Ke [
In this paper, we subdivide S&T entities into multiple grained categories. Depending on the type of scientific solution sought, they can be distinguished into: identification and judgment about the research object and the inherent mechanisms and laws of research. Correspondingly, the research objects include ”technological methods”, ”system devices”, ”scientific experiments”, ”scientific materials”, and ”databases name”.
Examples include ”cell-based cancer immunotherapy and gene therapy”, ”ferrosilicon alloy latent heat photovoltaic cells”, ”deep underground neutrino experiments” and ”two-dimensional materials for future heterogeneous electronic devices”. And the underlying mechanisms of things, such as “the principle of evolution controlled from top to bottom”.
2.1. Data
we mainly uses a rule-based extraction method as a cold start, combined with manual correction, to obtain a small amount of high-quality contextual S&T entities databases. As of now, there are a total of 162 lexical and syntactic rules. Then, combined with the dependency syntax analysis function of a pretrained HanLP model, candidate scientific entity phrases with contextual semantics are obtained.
briefing data monitored and compiled by various departments of the Chinese Academy of Sciences and the State Council, which is available on the agency’s website1 .
The data source includes: (1) the trends of top scientific journals, showcasing the latest scientific research achievements in disciplines such as physics, Earth, and biology; (2) the latest strategic deployments of various countries in the field of S&T, representing the direction of national S&T development.
These information contents can to some extent
represent the will of the country and scientists [
In the extraction part of based on LLM active annotation, the main goal is to gradually fine screen a small-scale annotated data from a large amount of unlabeled data, while using a large language model as the annotation model. At the same time, a S&T entity extraction model called ”Roberta-BiLSTM-CRF” is trained.
1) Optimizing LLM as better annotator.According
to literature research, it has been found that the
cur2.2. Main Framework rent GPT series models are highly sensitive to diferent
PROMPT expressions. When diferent annotators use
Based on the above data sets, the research work of this pa- diferent PROMPT expressions, there is a significant
difper mainly includes three parts: initial annotation based ference in the response results of GPT. The robustness of
on syntatic rules, active annotation based on LLM, and the model on NLP tasks is relativaly weak [
This paper adopts the Chain of thought (CoT) prompts strategy, which gradually generates label sequences that 1http://www.casisd.cn/zkcg/ydkb/kjqykb/ meet expectations by setting some conditions in each https://news.sciencenet.cn/AInews/newlist.aspx? model. Guided by the CoT approach, this article transhttp://www.globaltechmap.com/document/index forms this task into a multi round Q&A question, en
3.1. Analysis of data annotation results
abling the GPT model to gradually locate the fine-grained categories of S&T entities contained in the text through Initial supervised data based on the rule annotation conversation, and finally annotate them. Specifically, In the data annotation based on statistical rules, it was this chapter focuses on the construction process of PRO- found that the extraction efect of the model was: accuMOPT for diferent categories of S&T problems, as shown racy: 0.36; recall rate: 0.82; F1 value: 0.50. That is to say, in Figure 2. the majority of S&T phrases annotated by statistical rule2) Active data acquisition. Active learning (AL) based annotation methods are not within the category of seeks to reduce labeling eforts by strategically choos- S&T, and their level of S&T cannot be accurately judged. ing which examples to annotate. We consider the stan- Therefore, an AI model that can deeply understand and dard pool-based setting, assuming that a large pool of analyze semantics is particularly needed for annotation unlabeled data is available. AL loop starts with a and extraction. seed labeled set . At each iteration, we train a model on and then use acquisition func- 3.2. Analysis of LLM annotation results tion (·, ) to acquire a batch consisting of examples from . We then query the LLM annotator to label . The labeled batch is then removed from the pool and added to labeled set , and will serve as training data for the next iteration. The process is repeated for times.
Active acquisition strategies generally maximize either uncertainty or diversity. On one hand, uncertaintybased methods(such as Maximum Entropy, Least Confidence) leverage model predictions to select hard examples. On the other hand, diversity-based methods(such as K-Means) exploit the heterogeneity of sampled data.
Firstly, we randomly selected 20 texts from the annotated dataset as the test set to determine the number of examples in the Few-shot strategy.The results showed that 5-shot had the highest accuracy at 76.3%. The test shows that the more relevant and semantically similar the given examples are to the test text, the better the annotation effect of GPT3.5. In the 1-shot scenario where an example is given, the performance of the given example is sensitive and unstable to GPT3.5; Overall, 5-shot prompt performs better because combining multiple random examples can reduce the impact of noise.
After determining the number of given examples in the Few shot strategy, we conducted multiple tests to select the most efective example for each stage of the PROMPT.
The performance of each prompt stage is shown in Table 1.
obtained, and select the data to be annotated in the (1) In terms of category judgment, GPT’s performance
next iteration using the acquisition function mecha- is almost perfect. That is to say,for classification tasks
nism. Among them, the target model uses the Chinese with more popular query semantics and more obvious
RoBERTa-WWM[
Finally, after multiple rounds of annotation and manual proofreading, a total of 19745 sentences formed a supervised training dataset. 3.3. Analysis of Model extraction efect
as seed labeled set from initially annotated dataset, use the whole 19745 sentences as and randomly acquired 100 samples per batch for 10 iterations, which generate 9,921 annoted samples in total.
Baselines We compare RoBERTa-BiLSTM-CRF with the following baselines: (1) In-context learning (i.e. PROMPTING). The PROMPTING enables LLM to conduct few-shot inference without fine-tuning. (2) SUPERVISED(i.e. BERT-BiLSTM-CRF). The surpervised model is trained on whole clean-labeled data .
Accelerating with Active Learning The last layer in the above extraction model is the CRF model, whose output result is the probability score of the BIO label corresponding to each character. Here, we use this probability score as the confidence score and input it into two uncertainty based active learning strategies. The results show that maximal entropy active learning strategies enable extraction model to be more eficient and more capable. The results of the S&T entities extraction tasks are shown in Table 2.
with contextual semantics from think tank reports can more eficiently capture key research development directions. In this paper, GPT is used as the teacher model and Roberta-Bilstm-CRF is used as the student model. Through active learning method, the training data generated by GPT is fine-tuned to the local extraction model, forming a set of feasible fine-grained S&T entity recognition framework.
The biggest limitation of our study is that it mainly focuses on the discussion of the efectiveness of the method, and the standard of high accuracy has not been reached in practical engineering applications, and the model efect will continue to be optimized in the future.