Citation analysis methods mainly focus on the quantitative indicators, such as the cited number and the H-index, while ignoring the deeper information such as citation function and citation sentiment. Therefore, studying and analyzing the functions and sentiments of citations can more efectively evaluate an article and uncover its underlying information. As for data, this study investigated the existing dataset of citation sentiment classification (CSC), collected and organized a high-quality and available dataset. As for model, based on the pre-trained language model BERT and its variants, a model called DictSentiBERT is proposed to modify attention mechanism using sentiment dictionary, and a series of baseline models are designed for comparative experiments. The experimental results show that compared to the original BERT and baseline models such as RNN and TextCNN, the DictSentiBERT improves the accuracy of CSC and maintains the highest Macro-F1 score.
Authors of academic articles establish various relations between diferent papers by citing concepts, methods, conclusions, and experimental processes to support their work, or introducing their own work by pointing out shortcomings in previous works. Therefore, studying these relations is of great significance for exploring implicit information or evaluating the quality and influence of papers. The analysis and mining of citation behavior can help reveal knowledge structures, research hotspots, research trends, and academic exchange networks within the research field.
The demand and function of citation analysis in
academic community are gradually increasing, and citation
analysis is no longer just to evaluate the academic value
of research results. However, the traditional citation
analysis methods mainly focus on the quantitative indicators,
such as the cited number and the H-index, ignoring the
deeper information of citation function and citation
emotion. The work of Radicchi Filippo[
Sentiment dictionary, machine learning, and deep
learning are the three most common methods. Small[
sentiment detection features, and constructed a corpus pre-trained model methods guided by prior knowledge
containing 8736 instances. Poria[
The method of pre-trained models is gradually becom- tion performance without requiring a large amount of
ing popular. Beltagy[
This study focuses on integrating prior knowledge into shown in Figure 1. This study firstly investigated the
expre-trained models. Tingyu Xia[
In summary, these studies have made outstanding con- research collect supplement complete corpus
tributions in the field of CSC. The method of sentiment SciCite
dictionary is relatively simple, but it is limited by the 2. Preprocessing
cquulatlittoyaadnadptcotovecroangsetaonfttlhyechdaicntgioinngartyh, emmaeksi.nMgeitthdoifi-ds idnruespmtalioncvcaeetes cionrnesfmtalioncvctieensg inrepmaroevnethteexsets srsyepmmecboiovalels Rmveiamslsuoienvsge
of machine learning can achieve high accuracy, which, 3. Manual Screening
hmoawneuvaelrs,erleelcytihoenavainlyd othnefyeamtuarye feancgeincehearlilneng,greesqiunireinfi-g iinmrespmtaroonpvceeesr Ininrcesomtmaonpvcleeeste wrietpmhaiormtvitepiortoenpxetr relablealbwelrong
ciency and generalization when processing large-scale
data. Deep learning methods perform well, but their ap- corpus
plication may be limited for tasks that lack large-scale
annotated data. The pre-trained model does not require Figure 1: Flow Chart of Data Processing
large-scale data, but if there is a significant diference
between the trained corpus and the task corpus, its
effectiveness will also be greatly reduced. Integrating the
prior knowledge into the task of CSC can further improve 3.1. Source and Supplement
the efectiveness. Among them, integrating knowledge After conducting detailed research, it was found that
graphs or constructing domain ontologies with BERT although there are many studies on CSC, such as the
can achieve better results. However, there are also some dataset collected by Xu[
Based on the aforementioned research’s shortcomings ifeld. and gaps, this study aims to explore the application of
It’s natural to think about transfer learning because obtaining data of movie reviews, social media reviews, or e-commerce reviews is simple, direct and easy. But this is problematic beacuse there are significant diferences in language style, purpose, structure, etc. between the texts of scientific papers and those of film reviews.
1) Scientific papers usually adopt formal, professional and objective language style, and try to avoid subjective and emotional expression. Film reviews, on the other hand, place more emphasis on emotional expression and personal subjective opinions. 2) Scientific papers usually adopt standard structured forms,while film reviews are more liberal and typically include content such as movie introductions, personal impressions, and ratings, without a fixed structure and format. 3) The subject range of scientific papers and film reviews is also diferent. Scientific papers usually involve various professional fields in the academic ifeld, including biology, chemistry, physics, etc., while film reviews mainly involve film, television industry and related topics.
To sum up, the diferences between scientific papers
and film reviews are multifaceted, involving the purpose,
mode, intensity, object and audience of emotional
expression. So the use of Transfer learning is not efective. Due
to the lack of other solutions, this study still insists on
using the dataset2 proposed by Athar[
In order to further improve the accuracy of training
at the content level, after conducting comprehensive
research on multiple publicly available datasets, we
consider using the SCICite dataset proposed by Arman[
1) Remove missing values. 2) Remove instances with same text but diferent
labels.
3) Remove instances with duplicate text and labels.
4) Remove text within parentheses by regular
expressions because the content are unrelated to
sentiment analysis, such as "The two systems we
use are ENGCG (Karlsson et al., 1994)"
5) Remove various types of special symbols, only
retaining English text and numbers. Actually
symbols can also provide some emotional
information, such as question marks and exclamation
marks. In addition, some network symbols may
also reflect emotions. However, BERT seems not
sensitive to punctuation and other symbols
according to Adam’s[
Due to low quality of the dataset, some obvious problematic data were still discovered after preprocessing, which is as lised in Tabel 1.
ing models depends on the quality of training data.
Therefore, we attempts to maximize data quality through
manual review and screening. For the above questions (1), (2)
3.2. Preprocessing and (3), the original data will be directly deleted, and for
the question (4), those sentences will be re-labeled. To
The study by Mercier[
there are negative or positive intensity, they are added together, and 1 is added to obtain the final score. For The idea of DictSentiBERT is to integrate the prior in- example, if the word “book” does not have polarity, then formation of sentiment dictionary into the BERT, adjust the weight is assigned to 1. While the word “good” has attention mechanisms to better capture and understand a positive intensity of 0.5 and a negative intensity of 0, emotional information of scientific citations, and achieve resulting in a final weight of 1.5. higher accuracy in the classification. As shown in Figure 2, the model adopts the following architecture, including 4.2. BERT Layer input layer, BERT layer, modified attention layer, and output layer.
The BERT layer consists of two main structures: embedding and encoder. The input vector is composed of three diferent embedding, namely wordpiece embedding, position embedding, and segment embedding. The encoder of a transformer consists of a multi-head attention layer, a regularization layer and a forward propagation layer. In the standard BERT model, there are 12 layers of encoders and the word vector dimension is 768. In this study we use the vanilla base-bert.
Due to the varying importance of vocabulary and feature weights in the text, attention mechanism is introduced to learn the dependency relationships between vocabulary and pay special attention to the important vocabulary. Therefore, the accuracy of classification can be further improved by assigning diferent weights to focus on important parts of the context and the specific calculation formula is as follows:
In the input layer, coeficient for adjusting attention weights is calculated in advance by SentiWordNet and pos_tag. SentiWordNet is a dictionary used for sentiment analysis, which assigns an emotional intensity score to the three dimensions of positivity, negativity, and objectivity for each word in WordNet. However, the dictionary itself does not have the ability to handle polysemy, so we introduce NLTK’s tagging tool: pos_tag. Firstly, we use BERT’s tokenizer for word segmentation, converting the sentences into the standard form of BERT input. Next, we annotate each word and assign weight to it with SentiWordNet according to its part of speech. If there is only neutral intensity score, the weight is assigned to 1. If (, , ) = ( √ )
Where is the obtained attention weight matrix and the process of ⊙ is as follows: ⊙
⎡11 = ⎢ ⎣ . . . 1 ⎤ . . . ... ⎥⎦
⎡1 ⊙ ⎢⎣ . . . ⎤ . . .
... ⎥⎦ 1 × 1 × (5)
As for a sequence 1 . . . , the wights of them is 1 . . . calculated by SentiWordNet in the input layer. The other lines of wights matrix equal the first line. = + (, , ) = ( √ )
In this step the obtained weights matrix is applied to the original attention matrix. DictSentiBERT processes the input sentences and calculates attention scores as follows: = ⊙ + (1) (2) (3) (4)
The output layer is a fully connected layer that connects and transforms the outputs of the model, and uses the softmax function to calculate the probability score for each category. The final output is the label of input, including neutral, positive, and negative. Some examples are listed in the appendix.
Two basic pre-trained models: BERT and SCIBERT are used. On this basis, FeedForward NN (FNN), LSTM, TextCNN, Self-Attention and DictSensiBERT proposed in this paper are designed for experiments.
The code was written with PyTorch v1.10 in Python v3.7. And the model was trained on a 16GB RTX A4000 for 50 epochs each with 80% training set and 20% test set. The batch size was set to 32, the learning rate was 5e-6. The AdamW optimizer with a warm-up rate of 0.1 and the cross-entropy loss function were used for optimization.
BERT is 91.23%, with an average Macro-F1 score of 75%. SCIBERT performs better, with an average accuracy of 94.80% and an average Macro-F1 score of 85%. This indicates that SCIBERT trained in scientific texts is more suitable for CSC. On the other hand, it can also be observed that under the same basic pre-trained model, the performance of DictSentiBERT has also been improved to a certain extent, which proves that the pre-trained model incorporating sentiment dictionary is more conducive to extracting emotional information.
This study proposes DictSentiBERT, which adjusts attention mechanism based on sentiment dictionary, and applies it to sentiments classification of scientific citation. We conducted research and organized a high-quality CSC dataset, designed the DictSentiBERT model and a series of baseline models for comparative experiments. Results indicate that pre-trained models can efectively classify sentiments of scientific citations, and SCIBERT performs better than native BERT on this task. Furthermore, DictSentiBERT can improve classification accuracy while maintaining high Macor-F1 score. In summary, this study provides a high-quality CSC dataset and a new model for the sentiments classification of scientific citations. However, this study still sufers from quantity and quality of dataset and a larger corpus is needed to make further improvement and experiment. In the future, we can try to imitate the training process of SCIBERT, collect large-scale scientific citation texts, and adjust the MASK mechanism to focus on emotional words of MLM tasks. Then, we can use the oficial tool set provided by Google to train BERT from scratch. Alternatively, we can try relying on syntax trees and other methods to focus on the characteristics of sentiment analysis from the perspectives of syntax, grammar, and morphology. Finally, the latest GPT large model can also be combined to use AIGC to modify and guide pre-trained models.
This research is supported by High-performance Computing Platform of Peking University. The work is also supported by "National Social Science Foundation of China" Big Data-Driven Research on Semantic Evaluation System of Scientific and Technological Literature (Grant No.21&ZD329).