s, what may limit the capacity of nding the right paper for a speci c research purpose. Given the size limitations and the concise nature of abstracts, they usually omit explicit references to some contributions and impacts of the paper. Therefore for certain information retrieval tasks they cannot be considered as the most appropriate excerpt of the paper to base these operations on. In this paper we have studied other kinds of summaries, built upon textual fragments falling under certain categories of the scienti c discourse, such as outcome, background, approach, etc, in order to decide which one is more appropriate in order to substitute the original text. In particular, two novel measures are proposed: (1) internalrepresentativeness, which evaluates how well a summary describes what the full-text is about and (2) external-representativeness, which evaluates the potential of a summary to discover related texts. Results suggest that summaries explaining the method of a scienti c article express a more accurate description of the full-content than others. In addition, more relevant related articles are also discovered from summaries describing the method, together with those containing the background knowledge or the outcomes of the research paper.
In this paper we present our rst steps on the analysis of the quality of research
article summaries. Our goal is to nd the strengths and weaknesses of approaches
leveraging exclusively on abstracts against those based on scienti c discourse
categories such as approach, challenge, background, outcomes and future work. Since
the main contributions and impacts of a research article are not always included
explicitly in the abstract, as in the case of [
The paper is organized as follows: Section 2 highlights recent studies on text mining research articles and presents the steps followed to measure the representativeness of abstracts and research article summaries based on rhetorical categories. It describes both the classi er used to identify those categories in papers and the representational model and similarity metric used to compare textual units. Experimental results comparing the di erent kind of summaries are shown in Section 3. Finally, Section 4 presents conclusions from the experiments. 2
Recent studies [
In this paper, we show our initial analysis about the representativeness of
research article summaries, considering those based exclusively on abstracts and
those based on their discursive structure (approach, challenge, background,
outcomes and future work )[
First of all, we need to identify the rhetorical parts of a research paper. Some
approaches have been proposed to summarize scienti c articles [
A representational model is required not only to measure distances between text
fragments but, more importantly, to help to understand the di erences in their
content. Topic models are widely used to uncover the latent semantic structure
from text corpora. In particular, Probabilistic Topic Models represent documents
as a mixture of topics, where topics are probability distributions over words.
Latent Dirichlet Allocation (LDA)[
Thus, we have used a LDA model to describe the inherent topic distribution of
papers in the corpus. Some hyper-parameters need to be estimated: the number
of topics (k), the concentration parameter ( ) for the prior placed on documents'
distributions over topics and the concentration parameter ( ) for the prior placed
on topics distributions over terms. Since the target of this experiment is not to
evaluate the quality of the representational model, but to compare their topic
distributions, we accepted as valid values those widely used in the literature:
= 0:1, = 0:1 , and k = 2 pn=2 = 44 where n is the size of the corpus.
Similarity Measure Feature vectors in Topic Models are topic distributions
expressed as vectors of probabilities. Hence we opt for Jensen-Shannon
divergence (JSD)[
measures which in general are symmetric [
J SD(p; q) =
T X pi log 2 pi + i=1 pi + qi
T X qi log i=1 2 qi qi + pi where T is the number of topics and p; q are the topics distributions.
And the similarity measure used in our analysis is based on the JSD
transformed into a similarity measure as follows [
similarity(Di; Dj ) = 10 JSD(p;q) where Di; Dj are the documents and p; q the topics distributions of each of them. (1) (2) 3
The corpus used in the experiments was created by combining journals in di
erent scienti c domains such as Advances in Space Research, Procedia Chemistry,
Journal of Pharmaceutical Analysis and Journal of Web Semantics. In total
1,000 papers were added, 250 from each journal. Both the abstract and the
fullcontent of these documents were directly retrieved from the Elsevier API 5 by
using the librAIry [
Since the annotation process to automatically discovers the rhetorical parts of a research paper (Section 2.1) is sensitive to the structure of the phrases that are used when writing the text, only 20% of papers in the corpus could be fully annotated with all the fragments considered. In fact, these categories are not present in the same proportion in the corpus: approach (90%), background (78%), outcome (73%), challenge (57%) and future work (21%) 3.1
The internal-representativeness of a summary measures the similarity of this summary against the original full-text research paper. This similarity is based on the JSD between the topic distribution of each of them.
Since LDA considers documents as bag-of-words, the text length (e.g.
fullcontent or summaries) a ects the accuracy of the topic distributions inferred
by the topic model described in Section 2.2. The occurrences of words in short
texts are less discriminative than in long texts where the model has more word
counts to know how words are related [
Also, the relative presence of each of them in a paper ( gure 2) shows an
unexpected result when compared to the IMRaD format [
All pairwise similarities between full-papers, abstracts and rhetorical-based summaries are calculated to measure the internal-representativeness of a summary with respect to the original text, i.e. the topic-based similarity value (equation 2) between the probability distributions of the full-text and each of the summaries. Results (table 1) suggest than summaries created from the approach content are more representative than others, i.e. the distribution of topics describing the text created from the approach content is the most similar to the one corresponding to the full-content of the paper.
Min abstract 0.0489 approach 0.0499 background 0.0463 challenge 0.0426 futureWork 0.0000 outcome 0.0485 The external-representativeness metric tries to measure how di erent is the set of related documents obtained from summaries with respect to those derived from the original full-text. In terms of precision, recall and f-measure, a comparison has been performed to analyze the behavior of the summaries when trying to discover related content compared to use the full-text of the article.
By using the same topic model previously created, similarities among all pairs of documents were also calculated according to equation 2. Then, a minimum score or similarity threshold is required to de ne when a pair of papers are related. Each threshold is used to create a gold-standard which relates articles to others based on their similarity values. In order to discover that lower bound of similarity, a study about trends in the similarity scores ( g 3) as well as distributions of topics in the corpus ( g 4) was performed. We can see that topics are not equally balanced across papers. This fact generates separated groups of strongly related papers. We think this phenomena is due to our usage of a corpus created from journals where di erent domains are equally balanced. Then, we considered a similarity score equals to 0:99 ( g 3) as the threshold from which strong relations appear. However, to cover di erent interpretations of similarity, from those based on sharing general ideas or themes to those that imply to share a more speci c content, the following list of thresholds was considered in the experiments: 0.5, 0.6, 0.7, 0.8, 0.9, 0.95 and 0.99.
For each similarity threshold, a gold-standard was created based on considering as related those papers with a similarity value upper than the selected threshold. Results ( gure 5) comparing the related papers inferred from the fullcontent with those inferred from the partial-content representation (i.e. abstract or rhetorical parts) suggest that strongly related papers are mainly discovered by using the summary created from the approach section. The reason for this may be based on the average size of this type of summaries or the particular content included in this part of a paper. While other summaries include more general-domain words, the approach content includes more speci c words that describe the method or the nal objective of the paper. So, for higher similarity thresholds, i.e. for strongly related papers, the recommendations discovered by using the approach are more precise than those discovered by using the abstract.
In terms of recall ( gure 6), the upward trend followed by the approach, the outcome and the background content remarks the assumption of summaries containing key words allow to discover more similar papers than others. Moreover, since recall overlooks false-negatives classi cations, it suggests that these parts of a research paper share more words than others with strongly related papers but they may also present commonalities with highly related papers, except in case of approach which still exhibits higher precision.
As expected, only summaries created from the approach, the outcome and the background content maintain high accuracy values ( g 7) even for high similarity thresholds. Along with the results showed in gure 8, where the same three rhetorical classes present the lowest standard deviation over the f-measure, they can be considered as the most robust summaries containing the ideas that better characterize the paper compared to others. We have studied the Topic-based similarities among scienti c documents based on their abstract sections with respect to summaries corresponding to their scienti c discourse categories. For this purpose, two novel measures have been proposed: (1) internal-representativeness and (2) external-representativeness.
Results show that summaries created from the approach, outcome or background content of a paper describe more accurately its full-content in terms of overall ideas and related documents than abstracts. Although those summaries are more extensive in number of characters than other with similar precision such as the abstract content, they have proven to be particularly helpful discovering strongly related papers, i.e. papers with a similarity value close to 1.0.
In order to avoid an in uence of the size of the summaries on the accuracy of
the results, in future work we plan to use probabilistic topic model algorithms
oriented to handle short-texts such as BTM [