Summary peer reviews and abstracts represent opinions of reviewers and authors on the same scientific paper. Their focus and statement may be different. We propose primary measurement to compare them from readability and semantic function types. The results show that summary peer reviews highlight some distinct function types, and the terminology in peer reviews is not as dense as in abstracts. That means summary peer reviews can be complement to abstracts in literature searches, and can help readers understanding papers more thoroughly.
Information systems → Information retrieval → Retrieval tasks and goals → Information extraction Summary peer review, readability, function type, term density, functional component Scientific papers are the records of research activities. Critical information is contained in functional components of discourse, such as the research purpose, the problem definition, methods, experiments and contributions. Highlights of a study can be discovered from these components. Peer reviews are undoubtedly a quality source to reveal the highlights since reviewers are selected authorities with broad scholarly vision and strict taste. The summary reviews reflect peers’ evaluation by comments on functional components of discourse. And thus the reviews are worthy of consideration in readers’ paper selection. In order to identify the value of summary peer reviews from abstracts, we make comparison study on two problems, i.e., (1) the difference on functional components and readability of the two texts; and (2) the focus aspects highlighted by summary peer reviews. In order to demonstrate a scientific study, authors usually organize the content of their papers in compliance with well-established norms. For example, one common rhetorical structure of abstracts is Introduction-Method-Results-Discussion (IMRD) [1]. With the trend of structuralizing knowledge in scientific papers, researchers try to extract information by dividing papers to semantic functional components [5]. Fine-grained functional components have also been defined. Wang et.al proposed Functional Units Ontology (FUO) based on functional unit theory [2]. The FUO includes 12 classes such as background, theme, method, experiment etc., and 28 subclasses. In fact, functional components can be obtained not only inside papers but also from their summary peer reviews. Reviewers judge innovation and contribution of a paper according to components such as the research purpose, the problems, the methods, etc.; and then summarize their positive or negative opinions in the peer reviews. Kang Dongyeop et al. created the first open dataset of review comments, PeerRead [3]. The characteristic of the accepted and the rejected papers have been compared from the vocabulary usage and psychology linguistics based on reviews [4]. The agreement between the sentimental polarity of the reviews and the acceptance has also been studied. But these studies did not compare the focus of reviews and abstracts. 3
An observation of this study is that, in summary peer reviews and abstracts, the focus and statement are different. The comparison is shown in Figure 1. Firstly, defining the function types of reviews and abstracts; Second, annotating sentences by types; Thirdly, analyzing how the summary peer reviews highlight a paper’s value from readability and the focus aspects. each type is listed in Table 1. Sentences in summary peer reviews and abstracts are annotated to these types.
Term density and function type ratio are defined to compare the difference between abstracts and reviews.
Term density is the average number of general terms or terminology appeared in sentences of each type of functional components. In contrast to terminology, general terms refer to widely-accepted concepts or named entities without distinct domain specificity. Term density and sentence length reflect the readability of a text since readers may feel tough to understand if the sentence is long and with lots of terms or terminology.
Type ratio counts percentage of sentences of each function type. It reflects the focused aspects by reviewers and authors. 4 4.1
1. Reviews and Abstracts. The dataset was collected from Acta
Psychologica Sinica (journal.psych.ac.cn/) , including a total 774
papers publis
Readability can be reflected from sentence length and term density. As shown in Table 2, the average sentences length of R is shorter than that of A. Table 3 lists the term density of each T. Focus Aspect is reflected by type ratio, as shown in column T% of Table 2. The percentage of type background, process and result in A is 12.1%, 23.0% and 39.2%, much higher than those in R, i.e. 4.5%, 18.7% and 12.7%. While on the other side, the percentage of type theme and strength is 23.0% and 28.6%, higher than 11.5% and 1.0% in abstracts. It means the reviewers emphasize how reasonable and innovative a study is. At the same time, the authors try to clearly demonstrate the research process and results. 5
Generally speaking, summary peer reviews focus on different aspects from abstracts, and express in an easy-to-read way. That would be helpful in readers’ understanding and selecting papers. In the further study, it is necessary to find out how summary peer reviews could improve literature search.