Starting from the experience of the TERENCE European project, the paper shows challenges that require a combined effort of natural language processing, automated temporal reasoning and, finally, human computer interaction. The paper starts introducing the problem of producing high quality temporal annotations for texts, and argues for a combined automated temporal reasoning and natural processing approach to tackle it. The paper then speculates that the approach would benefit from knowledge of the specific domain and of how humans interact with the annotation process, which triggers two further challenges explored in the remainder of the paper, at the intersection of natural language processing, automated reasoning and human computer interaction.
Reading is an important means for language acquisition, communication, sharing
information and ideas. Reading transforms print to speech and print to meaning through
a negotiation of meaning between the text and its reader, as a problem solving
activity. Developing the capabilities of children to comprehend written texts is key to
their development as young adults. More than 10% of 7–10 year old children are poor
comprehenders: they have difficulties in comprehending texts, e.g., making inferences
concerning the temporal flow of a story. TERENCE (10.2010-09.2013) was an FP7
European project that developed the first adaptive learning system with learning material
for primary-school poor comprehenders, made of stories and quiz-like games for
reasoning about stories, in English and in Italian. The material is immersed in a game world
and delivered in an adaptive fashion according to children’s learning needs, investigated
through contextual inquiries with text comprehension experts and activities with
children [
Training children to reason about the temporal flow of stories as in TERENCE required to have c. 12 games of different complexity per story, for a total of more than c. 200 games per language.
Such figures led the TERENCE researchers to tackle an ambitious goal: to design
a semi-automated process for generating inference-making games, of different levels,
starting from stories, with the aim of significantly reducing human interventions in the
generation process. In order to meet their goal, TERENCE researchers chose Artificial
Intelligence (AI) for automatically extracting from stories data for semi-automatically
generating inference-making games, progressively training children to text
comprehension. AI took the form of Natural Language Processing (NLP) for the
languagedependent aspects of games, and Automated Reasoning (AR) with temporal constraints,
for completing the NLP work [
How to use NLP and AR to extract temporal information from stories that is critical for their comprehension?
The TERENCE Consortium tackled that and related challenges by developing an
annotation schema based on the TimeML markup language [
The automatically generated textual games were evaluated by education experts
with a qualitative evaluation design, similar to Amazon Mechanical Turk, through an
interface designed for them [
In particular, of relevance for URANIA, the evaluation results suggested that errors
in the generation process were also due to the quality of the TimeML annotation of texts
for extracting temporal data, which is in turn affected by the quality of corpora over
which NLP systems are trained for recognising and annotating temporal information in
texts, e.g., see [
That led us to embark on a novel journey and tackle a new ambitious goal: to analyse errors that recur through TimeML corpora, setting “time out of joint”. 2
Analysis of Human Errors and Novel Challenges
TimeML is used in resources such as the TimeBank corpus [
In TimeML, time is assumed to be linearly ordered over the real line, and relations between events are interpreted by relying on the standard order between interval endpoints. TimeML defines a qualitative time entity (e.g., action verbs) called EVENT, and a quantitative time entity (e.g., dates) called TIMEX. In particular, in TimeML events can be expressed by tensed and untensed verbs, but also by nominalizations (e.g. invasion, discussion, speech), predicative clauses (e.g., to be the President of something), adjectives (e.g. dormant) or prepositional phrases (e.g., on board).
TIMEX temporal expressions include specific dates (e.g. June 11, 1989), times (twenty to ten), durations (three months) and sets (twice a week). TIMEXs are also assigned a value that makes explicit, in ISO 8301 format, to which specific time the expression is anchored.
Time entities (EVENT and TIMEX) are linked through a TLINK relations.
Several TLINK relations have been introduced in TimeML, intuitively interpreted as Allen
interval algebra basic relations [
When human annotators manually add TimeML annotations, they can introduce different mistakes, especially in connection with temporal relations; each such mistake has potentially an impact on the quality of NLP systems that recognise and annotate temporal data. Such mistakes range from simple ones to subtle ones.
Examples of simple errors occur when annotators add two relations, such as “before” and its inverse “after”, between the same two EVENT or TIMEX expressions. Such a situation is inconsistent with the assumption that time is linearly ordered—an event cannot be simultaneously before and after another. Preprocessing techniques can help in finding and fixing such errors.
Other mistakes creep into the manual annotation process, which are much harder to detect for humans: those are the cases of inconsistencies due to a chain of temporal relations between events, possibly distant in a text. An example of such error is in Table 1, found in the AQUAINT TimeML corpus. The specific error is the incorrect annotation of a “before” TLINK relation, which is inconsistent with other TLINK relations.
The way in which annotators work on corpora and the tools they have currently at
their disposal all have an impact on the quality of their annotation work; see [
The journey into the manual annotation work led us to face a novel general challenge that require again a combined effort, specifically, of NLP and AR, and related to the TimeML manual annotation process:
How can AR and NLP improve on the quality of TimeML annotation work?
The question has been recently tackled in a novel manner by [
TLINKs Link ID Source Target Relation l93 ei48 ei47 BEFORE l587 ei47 t0 BEFORE l588 ei48 t0 INCLUDES Other temporal entities t0 = 2000-04-03 Interpretation over the real-line Error If ei48 is before ei47, and includes t0, then ei47 cannot be before t0 as well. already existing in the document. In other words, the system aid annotators in avoiding future annotation errors due to inconsistent TLINKs. However, the system does not support annotators if annotation errors sneaked early into the manual annotation, and potentially affect future correct choices of annotators. For instance, reconsider the excerpt reported in Table 1. The wrong TLINK annotation, with identifier l587, is BEFORE between ei47 and t0. This is inconsistent with the other reported TLINK annotations. If annotators introduce l587 as first, then the annotation guidelines and tool may not detect it as error and may instead mark the other annotations as errors.
In such cases, it seems beneficial to consider a complementary approach and system,
which aids annotators in finding annotation errors introduced in annotated documents
and due to inconsistent TLINKs (once interpreted as Allen relations or as relations
of other temporal calculi). The interpretation of TLINK as Allen relations, or as
relations of a different qualitative calculus, should be flexible and domain-dependent, as
advanced in [
Therefore another specific challenge is as follows, which derives from the previous one.
How can knowledge of human processing of texts help in devising combined NLP and AR solutions for setting time in joint in document analysis?
For tackling the challenge, we are currently implementing and testing strategies for
rapidly identifying possible inconsistencies, as well as designing how to highlight them
in currently available tools for manual annotation and suggesting how to fix them (e.g.,
CAT [
Last but not least, no support system for humans annotating texts can avoid the human-computer interaction aspects of the work, which will again require to cope with how annotators work. That is the final challenge we believe relevant for URANIA.
How can knowledge of human processing of texts help in devising visual metaphors that help humans in their text annotation work?
As for the visual tool, a preliminary support only for consistency checking was
presented in [