Research is a continuous phenomenon. It is recursive in nature. Every research is based on some earlier research outcome. A general approach in reviewing the literature for a problem is to categorize earlier work for the same problem as positive and negative citations. In this paper, we propose a novel automated technique, which classifies whether an earlier work is cited as sentiment positive or sentiment negative. Our approach first extracted the portion of the cited text from citing paper. Using a sentiment lexicon we classify the citation as positive or negative by picking a window of at most five (5) sentences around the cited place (corpus). We have used Naïve-Bayes Classifier for sentiment analysis. The algorithm is evaluated on a manually annotated and class labelled collection of 150 research papers from the domain of computer science. Our preliminary results show an accuracy of 80%. We assert that our approach can be generalized to classification of scientific research papers in different disciplines.
In this section we introduce some formal terms and preliminaries for better understanding of technicalities in paper.
In Citation Analysis, root paper or cited paper is the research paper that is being cited in another
research paper. Citation is a quotation that is used to refer a root paper. Citing Paper is the
research paper that consists of a reference to the root paper. Cited Area is the paragraph in
citing paper with citation of root paper. Terms are inspired from
In Text Categorization, Tokenization is the process that breaks down sentences of text into word(s) for further processing. It is done using Regular Expressions, which is a generic way of representing a string to be searched from a large corpus of text. Sentiment Analysis is done to classify the corpus into positive or negative categories based on the words extracted during tokenization. Lexicon is a dictionary that consists of key words which are divided into positive and negative categories. Positive Incrementer and Negative Incrementer are dictionaries, besides the Lexicon that were used in our approach, consisting of phrases of words that were observed to be inclining a particular sentence to have an overall positive or negative weight.
In our experimental study, Target Data is the data extracted from the cited paragraph. Citation
Types are the different categories into which target data are placed after sentiment analysis.
Naïve-Bayes Classifier is a popular machine learning algorithm for text categorization.
MonteCarlo Technique is a computational algorithm that rely on repeated random sampling.
Precision, Recall and F1-score are calculated, taken from
Precision is the fraction of retrieved instances that are relevant, calculated through the equation:
P = |{Relevant Documents} ∩ {Retrieved Documents}| / |{Retrieved Documents}| Recall is the fraction of relevant instances that are retrieved, calculated through the equation:
R = |{Relevant Documents} ∩ {Retrieved Documents}| / |{Relevant Documents}| F- Measure is a measure that combines precision and recall. It is the harmonic mean of precision and recall. The balanced F1-score, calculated through the equation:
F1 = 2*(Precision*Recall) / (Precision + Recall)
Authors have taken different strategies in approaching citation analysis,
Mostly techniques reviewed are focused on one discipline while we assert that our approach
can be generalized to classification of scientific research papers in different disciplines, since,
instead of cue words for a specific domain, we are using generalized sentiment lexica used in
13. Giving a new interpretation to the findings/statements in Cited Paper.
In our approach we have grouped these citation types into three generalized sentiment types: TYPE-I: Positive
Referring to Table 1, Type-I strictly refers to 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, Type-II strictly refers
to 10, 12, 13 and Type-III refers to 4. However 1, 6 may fall under all three Types and 2, 3 may
fall under Type-I and Type-II. Such overlapping nature of categories creates dis-agreement
between annotators
In this paper, we propose a novel automated technique, which classifies whether an earlier work is cited as sentiment positive or sentiment negative. Our approach first extracted the portion of the cited text from citing paper. Using a sentiment lexicon we classify the citation as positive or negative by picking a window of at most five (5) sentences around the cited place (corpus). The algorithm is evaluated on a manually annotated and class labelled collection of 150 research papers from the domain of computer science. Complete process is explained in Figure 2, and reference to the example papers are provided. Root paper followed by citing papers.
We collected a data set consisting of 150 research papers, manually downloaded in pdf format from Google Scholar. To further strengthen our basis for citation types we devised manual annotation of citation corpuses from all research paper in our data set and then manually classifying them into Type-I and Type-II. The results are shared in Table 2.
Figure 2. Overview of CRC All the annotation were done manually by the co-authors and the results were further crosschecked by other co-authors. Inter-annotator agreement was fair, however, to further strengthen our annotation results, we feel that annotation should be done by individuals who have at least a post graduate degree in their fields related to the corpus. In future, we plan to work with PhD Scholars in the field of computer science to provide annotated data in their field of research. For every root paper that we queried, we stored its top 10 citing papers. To make our approach more robust we included research papers of different writing styles such as Institute of Electrical and Electronics Engineers (IEEE) Standard Style, American Psychological Association (APA) Style and American Medical Association (AMA) Manual of Style. We summarize variations in referencing style in Table 3, along with percentage of sample papers in our dataset.
[1] Name Paper, Journal. Year [1], [1,2] 67%
APA
15% AMA
Journal. Year [1], [1,2] or 1-2 18%
Root paper and all its citing papers were placed in separate folders. The first step was to convert
all pdf files in dataset into text format. Python Library ‘PDF Miner’
To process data for analysis phase we extract useful information from all text files in the dataset.
From root paper we extract the title and the abstract section. Similarly, from citing paper we
extract the title of citing paper, reference (number) which points to in-text citations made to
root paper, reference to root paper, corpus, continuing reference i.e. reference to any research
paper other than root paper in corpus region, an annotated label assigned to each citing paper.
First the title of each root paper is extracted, but pdf-to-text conversion tend to modify
formatting style, making it difficult to construct a regular expression to identify title of each
root paper. Instead we programmed a Scrapping Module API in Python
XML Parser creates a separate file for each citing paper containing its Class (Label), Reference
(number) and Corpus. These files were then placed in respective subfolder (Type I, Type II).
Once the dataset is ready, we made features upon which citing papers will be automatically
classified. For this purpose we included following lexicon dictionaries:
Positive and Negative lexica: Derived from
Positive Incrementer and Negative Incrementer: Derived from
Feature Extractor iteratively reads each files. It uses Text Blob library
We have used Naïve-Bayes Classifier for sentiment analysis, using Scikit-learn library
In this section we discuss our findings for data extraction and classification. For data extraction
our results show 86% accuracy, which is not only higher than the estimated precision score of
80% by
Type I Type II Average/Total either due to pdf-to-text conversion limitations for images and remaining were because referencing style were not correctly followed by the author. Results are explained in Table 4. After extracting the data from the citing papers, the data was analysed with the help of NaïveBayes classifier. Our preliminary results show an accuracy of 80%, explained in Table 5. The reason for low recall for Type II was because we had only 21 papers that were manually annotated to be negative while the rest of the 109 papers were annotated to be positive. Due to lack of data our classifier couldn't accurately train itself on Type II papers. To further strengthen our result and remove low sampling bias of Type II papers we have used two approaches. In our first approach, the data set was adjusted to 42 research papers. For testing and training we used 50% data. To balance Type I and Type II papers, the technique was applied on different window sizes of Type I papers, while using all 21 papers of Type II. The windows were set to 0-21, 22-43, 44-65, 66-87, 88-109 and 110-125. Different accuracy were obtained at each window and the average accuracy was approximately 78%. Results are explained in Table 6. Secondly, the data was analysed using the Monte-Carlo Technique, to obtain stability in classification. Further changes were made to the Type I windows accordingly, explained in
The purpose of our paper was to help the research scholars by minimizing the time required to find the relevant research, for a topic of interest. In CRC, we proposed to classify the citations into three categories i.e. Type-I (Positive), Type-II (Negative) and Type-III (Neutral). A researcher now if desires to know any further advancements in paper under study can directly refer to Type-I papers, or if he wishes to know any research giving rebuttal of findings, in paper under study, can refer to Type-II paper. Our preliminary results show an accuracy of 80%. We assert that the technique can be generalized to classification of scientific research papers. Currently, support for Type-III papers in CRC is in progress, and we are working on a lexicon dictionary with neutral words. In future, we plan to provide a web portal to assist the research scholars in automatically searching and downloading citing papers, for a root paper, and classification of citing papers into sentiment categories.