The rapidly growing amount of data that scientific content providers should deliver to a user makes them create effective recommendation tools. A title of an article is often the only shown element to attract people's attention. We offer an approach to automatic generating titles with various levels of informativeness to benefit from different categories of users. Statistics from ResearchGate used to bias train datasets and specially designed post-processing step applied to neural sequence-to-sequence models allow reaching the desired variety of simplified titles to gain a trade-off between the attractiveness and transparency of recommendation.
The amount of information scientific society produces on a daily basis results in the
necessity of researchers to have proper guidance in a digital space. The function of the
virtual assistance is performed by various scientometric systems, research paper
recommender systems
The recommendation usually has a form of imprint often limited only by a title (as in the case with email subscriptions associated with limited space and lack of time to attract people’s attention). Eventually, the success of recommendation depends on the informativeness of the title of an article subject to user’s intentions and acknowledgment with a certain scientific field. This denotes the necessity of finding a way of varying the title of the same paper for different categories of users.
The focus of this paper is in developing models for creating a variety of simplified
versions of the titles of scientific articles which would be condensed and informative
enough and at the same time would correspond to the original topic of a paper to
maintain users’ loyalty. We aim at supporting two scenarios of personalized
simplification: the first ensuring narrow focus on specific scientific concepts for goal-oriented
experts and the second providing a general overview for researchers working on the
edge of a topic willing to expand their horizons. The second case should not be
treated as a generation of clickbaits (catchy short misleading headings) that are to be
blocked with the use of efficient machine learning approaches
There is a variety of algorithms that could be used for title simplification which is a
rapidly growing research area
The remainder of the paper is structured as follows. In Section 2, we propose a method for scientific title diversification and simplification. Section 3 is devoted to describing the datasets used for training. Section 4 denotes the experiment setup. Section 5 provides results of numerical experiments. Section 6 is devoted to human evaluation. In Section 7, finally, we discuss results and outline future work. 2
Recent advances in natural machine translation (NMT) incite to solve the task in a supervised manner controlling the style of a title by conditioning training data. The method we propose comprises the following steps: a) selecting a subset from an abstract-to-title dataset to impose conditions that would force a model to generate hypotheses with desirable properties; b) training a sequence-to-sequence (seq2seq) model; c) applying a model to title-to-title generation; d) performing post-processing step to remove unnecessarily repeated tokens; e) filtering titles with improper structure. The remainder of this section describes each step in details.
To create titles of different styles for various categories of researchers several datasets should be used. The set of highly popular scientific titles may help to generate attractive headings for users with interests peripheral to the subject of a paper. The condition to have a multi-word noun phrase NPmw in a target text is to avoid producing overly shortened pointless titles. In case each training example contains a reference text Rt and a target text Tt that have similar NPmw (at least 2 common terms), a model might learn to preserve the most important concepts from original titles needed by experts. Figure 1 shows the training example with similar NPmw-s in Rt and Tt.
Input sequence (an abstract, lower case, tokenized, truncated) the main goal of this research is to study whether or not the order of presentation of the premises in a logical argument form , such as a conditional reasoning task , could affect
Target sequence
(a title, tokenized, lower case)
effects of order of presentation
on conditional reasoning
We chose a bidirectional LSTM
Since the task differs from general NMT task and summarization task by the
absence of need in tracking alignment, traditional coverage mechanism
The last step consists in filtering improper titles, i.e., generated sequences that have less than two NPmw-s similar to some NPmw-s of the source title. In those use cases when even potentially pointless output is required, this step should be skipped. 3
We chose ResearchGate1 platform as a source of data. It has a recommender system and therefore openly counts the number of times a page with a paper was visited to provide reasonable recommendations that motivates authors to be more visible.
We selected 150K imprints of articles on various topics using a wide list of general
scientific words
dation data consisted in leaving only examples with Nv > 1, at least one common term in Rt and Tt, and Lt > 20.
To detect noun phrases we used Spacy chunker
Selecting the first 75 characters of the reference text is generally used as a baseline in
summarization tasks; cf., e.g.,
Several seq2seq models (M1, M2, …) with the above-described architectures differed
by a number of layers were applied to various datasets to bias the style of output text.
They were then extended with post-processing PS (M1ps, M2ps, …) and filtering steps,
which are novel for the best knowledge of the author; cf. Table 1 for details.
For the final model assessment, we used measures BLEU
The intermediate models created at checkpoints during the training were assessed and the best by NPdiff-p were selected as resulting. 5
The most of the basic models performed reasonably: produced titles were in general shorter than original, multiple-word noun phrases from reference title covered a significant part of the generated title (NPdiff-p = 0.68 on average). However, some models, especially M5, introduced many repetitions (for all checkpoints): the BLEU value reflected it being equal to 0.18 for M5 while the average value for the rest of models
was equal to 0.35. Since BLEU depends on a number of same word occurrences, the increase of it by 24% on average due to PS attests usefulness of the step (cf. Table 2). Filtering step allowed dropping less informative titles so that one can take advantage even of poor models reducing a risk to present misleading picking-eye headings or generic topics to an end user (cf. Figure 3 for examples of generated texts). model MBase M1ps+F M2ps+F M3ps+F M4ps+F M5ps+F M6ps+F
picking-eye The Romans Know?
generic topics
Access to Specialist Medical Services: a Pilot Study s e l t i t l a n i F The extension of basic models led to an increase of NPdiff-p by 9% and rouge-L-f by 11% on average. Table 3 gives an idea of the variation of titles of different models in style and in compression rate. It is worth noting that 1-layer models M1 and M2 trained on conditioned datasets reached higher values for the majority of measures in comparison to models M3 and M4 fed with generic data. This highlights rationality in pre-directing the training. 6
For human evaluation, we selected five papers of the NLP research group (TALN UPF) with titles longer than 93 characters (10-18 words). Their authors who own Ph.D. degrees were asked to rank output titles for these papers including original title by preference on clicking if they saw a title briefly in a daily email digest. To face different decision criteria assessors worked with papers of their authorship (for simulating expert behavior) and with papers of their colleagues (expanding horizons use case). If some titles in a set were the same or assessors did not have any preference between two similar titles they were allowed to rank them equally. The top models sorted by the average rank and examples of titles from one set are listed in Table 4. Noted final increase of NPdiff-p and rouge-L-f indicates that common subsequences became longer in relation to the length of titles meaning that offered post-processing step with filtering plays an important role in forming a fluent text. At the same time, the output should not have been just one of the original subsequences, therefore, we did not aim at reaching too high precision values.
Pure state-of-the-art seq2seq models without post-processing step got low ranks on human evaluation. The models M1ps and M2ps have a higher average rank of 6. Their titles are well-formed and represent a combination of original multi-word expressions (cf. Table 3 for relatively high scores of rouge-2-r), however, less corresponding to the topic that is partly reflected by comparatively lower values of rouge-L-p. The outputs of the models M3ps and M5ps were often preferred to original titles. Having 1.3 times shorter titles than M5ps, conditionally trained M6ps achieved almost the same average score. The baseline has the highest rank since it often better preserves the meaning although does not always form a complete phrase. The main drawback is that it usually only generalize a title to some extent (in case of well-turned subsequence) and miss details experts might need.
The close average ranks of models and rouge-L-f on the same level for all models denote an opportunity to overcome the general problem of lacking the variability in neural seq2seq generation. Different title styles give a possibility to reach a preferable trade-off between the conciseness of the title and its transparency.
For future work, we plan to gain value from methods of paraphrasing
The presented work was supported by the European Commission under the contract numbers H2020-700024-RIA, H2020-700475-IA, H2020-779962-RIA, H2020786731-RIA, and H2020-825079-RIA and by the Russian Foundation for Basic Research under the contract number 18-37-00198. Many thanks to the four anonymous reviewers for their valuable comments, and to the five postdoctoral researchers for their high responsiveness in the evaluation and insightful feedback.