Which tasks in digital libraries might be interesting and addressable for us as a community now in the near future, given the recent developments in NLP? This paper attempts to make a guess. (Instead of actually even attempting to answer the question in its title.) What would we need to do today if we, at some point in the future, wanted to be able to answer this question objectively, quanti ably, and on a large text base? Many similar questions in the same vein exist, such as where the research of a eld as a whole is headed, where the currently most contested research issues in a eld are, which new ideas emerged in the past 5 years, and which ones of these are game-changers. I will argue that we might well be able to o er support for the answering of such questions sometime in the future, particularly if we are willing to turn our attention to entailment and inference in scienti c writing.
Something always irks me when I read \future work" sections; be it the \future work" sections of papers I review, those of papers I read for my research, or the ones I have to compose when writing my own papers. Questions such as { what exactly is the purpose of these sections, and what is the status of the ideas in these sections? How should I interpret these descriptions in other papers, and how should I write my own?
In terms of communicative function, what a \future work" section is supposed to accomplish is less clear-cut than, for instance, a \methods" section. Ideas we express there don't really \count" in terms of the contributions of the paper: the research ideas we present there are hypothetical (they are not yet done), so no real contribution can be claimed for them. Lacking a potential reward, why would we o er up our unprotected secret research plans to anybody who might want to snatch them? On the other hand, social convention requires that we write something; we can't just leave the space empty. True, occasionally one has some ongoing work that one urgently wants to tell the community about, or a particularly clever idea that ts neatly. But I suspect that many of the ideas I read about are mechanically written possible avenues for future research, which get forgotten almost as soon as the paper is published. Maybe they get taken up by somebody { more likely not. (I know this because I sometimes do this myself.)
But \future work" sections could also in principle be like a market for ideas, a notice board where we announce our true intentions, and where we compete with our readers { just who manages to beat the time and bring out the next paper on this hot future topic? Perhaps the main inspiration for a large proportion of papers ever written was indeed such an open research question previously posed in a paper the authors read?
I wished that an enterprising social scientist, bibliometrist or historian of science did this research, and I also wished he or she would (be able to) use the large data repositories we now have at our ngertips when doing bibliographic searches. I would personally like a quanti ably answer, which is precise and supported by much textual historical evidence. Could this possibly be an application of large-scale digital libraries in combination with natural language technology? Maybe not quite yet, but thinking about what we would need in order to answer this question could provide some near future goals for NLP.
The short answer to what we would need to do is to match up descriptions of planned research in an earlier set of papers, with research contributions of papers later in time (either by the same or by di erent authors). The detailed answer is more complicated and concerns technologies such as paraphrase detection, citation link processing, citation block detection, entailment detection and many more. 1.1
Suppose we nd the following sentence in the \future work" section of a paper. (1)
We intend to investigate more sophisticated ways of document representation and of extracting a citation's context.
We then start a search for this task, and given that the communicative purpose of a scienti c paper (its knowledge claim) is generally quite clearly marked, let's say we nd a paper (by the same authors) in the future of the rst paper (say, a year later), with the following title: (2)
Context Matters: Towards Extracting a Citations Context Using Linguistic Features We can't help concluding that the original intention of the authors, stated in sentence (1) above, must have been real, as it was obviously followed up by exactly the kind of research predicted. Manually nding examples where other researchers take up a suggestion from a previous \future work" section is a bit harder to do. Of course, in many cases we might never nd a good match.
It is clear that a search engine that could provide a social scientist with objective, hard data on how many times a plausible match occurs would have to be pretty intelligent. It is de nitely addressing a text understanding problem. (My de nition of \text understanding" is \any process that obtains new knowledge, i.e., something that is true and relevant but that isn't explicitly stated in the text). My guess is, however, that such a system wouldn't have to be impossibly intelligent, given today's NLP technology. 1.2
Paraphrases, pragmatic e ects, and concreteness
The rst component we would need is very robust paraphrase detection. Research
in paraphrase detection is long-standing, with many successful approaches [
The closest we have come as a eld to a shared inference task is the RTE [
This has more recently been complemented by visual entailment tasks, such
as the one based on image captions that underlies the SNLI, the Stanford
Natural Language Inference corpus [
Recognising and processing presuppositions, and implicatures in general, can also help immensely. Apart from knowing about what is logically entailed in a text, knowing what is implicated is another important mosaic stone in the inference puzzle. Implicatures are de ned as those statements about the world that are assumed to be true by the speaker and transmitted \along with" their literal message, without being explicitly stated. (3) a. b.
Miller et al. did not manage to verify whether saturation was reached.
Miller et al. did not verify whether saturation was reached.
In contrast to entailment, implicatures cannot be cancelled by negation. If I state sentence (3-a) above, we understand that Miller et al. attempted the veri cation, but that the test was inconclusive. If I state sentence (3-b) above, I trigger a very di erent meaning, namely that they didn't want to verify, and probably didn't even run the test. Please also note that the presupposition \they had the intention of verifying" survives even if we turn the negative sentence into a positive one.
This is a remarkable di erence, in that sentences (3-a) and (3-b) are truthconditionally equivalent; the only thing that distinguishes them is the verb manage to. Presuppositions are distinguished from general implicatures in that they are lexically trigged. (4)
Miller attempted to model how X works.
Miller modelled how X works.
Another example concerns conversational implicatures. Stating sentence (4-a)
above conversationally implicates that Miller et al. didn't manage to
satisfactorily model how X works. (If they had, in the author's opinion, then sentence (4-b)
would have been more appropriate. This follows from the Gricean maxims [
There are few works that model pragmatic reasoning e ects
computationally on a large scale, but some resources such as presupposition trigger lexicons
have been created, and some automatic research exists on detecting
presuppositions and implicatures beween pairs of verbs, e.g. [
Overall, I believe that such works can contribute towards the question in the title of this paper { nding links between people's proclaimed intentions of doing something (as stated in the \future work" sections), and the reports of actually having done that thing (in a research paper). They can potentially also help nd similarities between methods used in di erent papers, and determine what the exact di erence might be in cases where a contrast between some methods exists but where the nature of the di erence is not spelled out by the authors.
Scienti c writing, like all texts, contain many implicatures. According to the Gricean Maxims, we would be insulting the reader's intelligence (or sound pompous) if we tell them more than they need to know to make the inference themselves, and the search space for possible inferences is very, very large. In my opinion, any work on automatic inference should therefore start with the clearcut, small-step cases, such as presuppositions and implicatures, where there is normally very high human agreement that these statements have objectively been asserted into the discourse, and what the implicature is. This is opposed to other types of inference, where much more world knowledge is needed to make the inference.
Finally, I will propose that judging the \abstractness" of a \future work" item would also be useful. This could serve to distinguish the following pair of sentences: (5) a. In future work we intend to evaluate the algorithm as part of a dialogue understanding system on state of the art benchmarks. b. . . . it is not known exactly how well the model will perform in the real world. Future work will examine installing models in real world applications.
The rst avenue for future work (sentence (5-a)) is quite concrete, and the
chances are high that the researchers, having already implemented their
algorithm, might conceivably next perform the rather concrete action of running an
evaluation. In sentence (5-b), the future work suggested sounds abstract and
vague, and like work of a di erent kind altogether, for which the research team
probably are neither equipped, nor have any real intention of doing. Research in
metaphor classi cation has contributed some methods for addressing the task of
judging the abstractness of phrases [
Predicting which \future work" suggestion is taken up in later work is an exciting task. An additional question is that if some such suggestion does get taken up, whether it is by the authors of the original paper or by somebody else. Given an acceptably precision matching procedure, a natural gold standard for the prediction task o ers itself, because in citation-based research we can manipulate the time scale of papers we take into account, in order to simulate a \future" that is to be predicted.
There are various related scientometric questions we might ask about the
development of a eld, such as the task of describing the emergence and
development of a scienti c eld [