=Paper=
{{Paper
|id=None
|storemode=property
|title=Exploiting Inference to Improve Temporal RDF Annotations and Queries for Machine Reading
|pdfUrl=https://ceur-ws.org/Vol-966/STIDS2012_T13_Schrag_InferenceInTemporalRDF.pdf
|volume=Vol-966
|dblpUrl=https://dblp.org/rec/conf/stids/Schrag12a
}}
==Exploiting Inference to Improve Temporal RDF Annotations and Queries for Machine Reading==
https://ceur-ws.org/Vol-966/STIDS2012_T13_Schrag_InferenceInTemporalRDF.pdf
Exploiting inference to improve temporal RDF
annotations and queries for machine reading
Robert C. Schrag
Digital Sandbox, Inc.
McLean, VA USA
bschrag@dsbox.com
Abstract—We describe existing and anticipated future benefits inferable bounds between any two points and will detect
of an end-to-end methodology for annotating formal RDF contradictory time point relation sets.
statements representing temporal knowledge to be extracted A fluent is an object-level, domain statement (e.g., FluentA:
from text, as well as for authoring and validating test and/or attendsSchool(Jansa LubljanaUniversity)) whose truth value is
application queries to exercise that knowledge. Extraction is a function of time. It is taken to be true at time points where it
driven by a target ontology of temporal and domain concepts
holds and not to be true at time points where it does not hold.
supporting an intelligence analyst’s timeline tool. Both the tool
We reify a fluent in an observation—a meta-level statement
and the methodology are supported at several points by an
implemented temporal reasoning engine, in a way that we argue whose object is a fluent, whose subject is a time interval, and
ultimately advances machine reading technology by increasing whose predicate is a holds property (e.g.,
both sophistication and quality expectations about temporal holdsThroughout(FluentA Interval1), when FluentA is observed
annotations and extraction. over Interval1, corresponding to, say, September, 1980).
The events of interest to us, which we call transition events,
Index Terms—temporal knowledge representation and occur at individual time points and may cause one or more
reasoning, extracting formal knowledge from text, machine fluents to change truth value. We represent events (like the
reading, annotation interfaces and validation birth of Jansa) as objects with attribute properties like agent,
patient, and location, and we relate events to time intervals with
I. INTRODUCTION an occurs property (e.g., occursAt(BirthOfJansa Point2), where
Machine reading—that is, automatic extraction of formal Point2 is associated with an interval corresponding to the date
knowledge from natural language text—has been a September 17, 1958). As usual with the event calculus, such
longstanding goal of artificial intelligence. Effective extraction events can initiate fluents (e.g., occursAt(BirthOfJansa Point2)
into RDF has the potential to make targeted knowledge initiates FluentB: alive(Jansa Interval3), where Interval3 is
accessible in the semantic web. We recently supported a large- begun by Point2) or terminate them (e.g., DeathOfJansa… ).
scale evaluation of temporal knowledge extraction from text by The temporal reasoning engine implements appropriate axioms
providing RDF/OWL ontology for target statements and a to perform fluent initiation and termination.
corresponding reasoning engine for query answering. Along Note that an observer may report information about the
the way, we discovered… temporal extent of a fluent without communicating anything
• How inference could improve annotation—the manual about initiation or termination. E.g., if text says Abdul and
extraction of formal temporal statements—and Hasan lived next door to each other in Beirut in 1999, we don’t
question authoring for evaluation or for applications. know when Abdul or Hasan may have moved to or from
• How, coupled with annotation and question authoring Beirut. When text says Abdul moved to Beirut in 1995 and
processes, inference could ultimately drive more emigrated in 2007, we use the properties clippedBackward and
sophisticated machine reading capabilities. clippedForward regarding the fluent residesInGPE-spec(Abdul
BeirutLebanon) to indicate initiation and termination by
II. TEMPORAL KNOWLEDGE REPRESENTATION AND anonymous (unrepresented) transition events, so that we can
REASONING FOR TIMELINE DEVELOPMENT also initiate or terminate temporally under-constrained like-
Our temporal logic is based loosely on the event calculus fluent observations (e.g. Abdul lived in Beirut during the
[10], as follows. 1990s).
A time interval is a convex collection of time points— The reasoning engine’s implementation, using
intuitively, an unbroken segment of a time line. Time intervals AllegroGraph, Allegro Prolog, and Allegro Common Lisp from
begin and end with time points, which may be constrained Franz, Inc., can answer any conjunctive query. While not yet
relative to each other or relative to a calendar. The ontology heavily optimized, it is at least fast enough to support machine
includes a rich set of relational properties over time points and reading system evaluation over newspaper articles where cross-
intervals, and the reasoning engine will calculate tightest document entity co-reference is not required.
The combined extraction and reasoning capability was IV. LESSONS LEARNED WITH IMPLEMENTATION PROPOSED
conceived to support an intelligence analyst’s timeline tool in Here, we propose some further approach elements that we
which a GUI would be populated with statements about entities expect to lead to high-quality temporal annotations,
(e.g., persons) of interest extracted from text. Our evaluation including…
of machine reading capabilities was based on queries similar to A. Interfaces and workflows deliberately designed to
those we would have expected from such a tool’s API. It also support capture of all statements specified as
supposed the analysts could formulate their own, arbitrary extraction targets (see section A)
questions, such as Query 1: Find all persons who were born in
B. Graphical time map display including fluents and
Ljubljana in the 1950s and attended Ljubljana University in the
events (see section B)
1980s, the titles that they held, the organizations in which they
held these titles, and the maximal known time periods over • On-line inference to elucidate inter-
which they attended and held these titles. relationships and potential contradictions
• Visual feedback to let users help assure quality
III. LESSONS LEARNED AND REALIZED IN IMPLEMENTATION themselves
This indirect, query answering style of machine reading • Time map-based widgets supporting user
evaluation makes it especially important that we perform knowledge entry
effective quality control of formal queries in the context of the C. Technology adaptable to test or application question
formal statements we expect to be extracted from answer- authoring (see section C)
bearing documents. We thus developed the test query D. Quantitative temporal relation annotation evaluation
validation approach illustrated in Figure 1. Considering Query (see section V.A).
1’s formalization (see Figure 10 in section IV.B), it’s worth A. Annotation workflows
noting that we used the methodology illustrated here to debug a
Fluents are simple statements that we can readily represent
number of subtle errors occurring in our earlier (manual)
in RDF. The example in Figure 2 focuses on the fluent about
formulations. When each such formulation did not result in the
one Janez Jansa attending Ljubljana University—only on the
answers expected, we traced inference to identify a point of
failure, corrected this, and then iterated until correct. fluent, not the full observation including temporal information
(i.e., only that Jansa attends the school, not when). The
Our machine reading technologists told us early on that
technology needed to annotate such information is well
they preferred macro-level relational interfaces that would
understood and (excepting perhaps the last bullet about
streamline away micro-level details of time points and
modality) has been well enough exercised that we may
intervals. We thus provide a language of flexible specification
routinely expect good results. This includes multi-frame GUIs
strings (spec strings) that expand to create time points,
where a user can produce stand-off annotations by highlighting
intervals, and relations inside our reasoning engine. We also
text and by clicking in drop-down boxes select relations,
provide ontology to associate the temporal aspects Completed,
classes, and instances. In part because these tools have
Ongoing, and Future with fluents (e.g., he used to live there vs. preceded reading for formal knowledge extraction, they may
he is living there vs. he is going to live there) and with the not use our intended representation internally—i.e., they may
reporting dates of given articles, to afford a relational interface for historical reasons internally use a representation (e.g., XML
reasonably close in form to natural language sources. For the that is not RDF) tailored to linguistic phenomena rather than
Completed or Future aspects, we also can capture quantitative associated with any formal ontology.
lag or lead information (e.g., he lived there five years ago or he
will move in five days from now).
Diagnose KR&R issues.
Compare answers.
Formalize.
Given NL… Document Query Answer(s)
Manually author selected Formalize Execute
Produce statements to support query. query to
KR… expected inference. get results.
Apply temporal reasoning engine.
Figure 1. We validate test queries by making sure that natural language (NL) and formal knowledge representation (KR)
versions of documents, queries, and answers agree, diagnosing and debugging as necessary.
1. Select relation.
…Jansa graduated from 2. Specify argument identifiers, respecting
Ljubljana University… co-reference.
Source text 3. Select / highlight / designate
corresponding text.
Formalization 4. Capture any counter-factual modality info.
attendsSchool(Janez_Jansa Ljubljana_University)
Figure 2. The workflow to annotate a fluent
1. Select one of time interval or point.
Reporting
date 2. Capture any beginning date and backward
Dec 28, 2007… clipping info.
3. Capture any ending date and forward
…Jansa graduated from clipping info.
Ljubljana University in 4. Capture any duration info.
1984… 5. If ending point is unconstrained w.r.t.
reporting date:
a. Capture reporting aspect.
b. Capture any reporting lag info.
6. Capture any other relative temporal info
available.
Fluent clipped forward
at ending point
attendsSchool(Janez_Jansa Ljubljana_University)
1984 Entered info (user writes)
[,1984-12-31] [1984-01-01,1984-12-31] Inferred info (user reads)
Figure 3. The workflow to annotate a holds statement (a fluent observation)
Capturing the temporal information associated with the • There is no duration information. (We don’t know
given observation of a fluent in a holds statement requires how long Jansa was at school.)
following a sequence of actions and decisions in a deliberately • The ending point is well before the reporting date, so
designed workflow, as outlined in Figure 3. We have we skip to the next step.
highlighted, by color- and typeface-coding, some temporal • There is no other relevant temporal information.
elements in the source text, along with corresponding steps in • To indicate clipping, the graphic fills the time point
the workflow and elements of the associated graphical symbol (making it solid).
representation. Our reasoning engine expands the entered coarse date 1984
Addressing the workflow step by step, we see that: into earliest and latest possible calendar dates bounding the
• There is no reason to believe Jansa attended school for observation’s ending point. It also infers an upper bound on its
only one day (the time unit associated with a time beginning time point.
“point” in our machine reading evaluation), so we As illustrated in Figure 4, we invoke a similar workflow for
choose a time interval (and the predicate event occurrence. Because our representation for events is
holdsThroughout) rather than a time point (and simpler than that for observations, this workflow has fewer
holdsAt). Schrag [8] argues that holdsAt almost never steps. Our ontology treats birth as a fluent transition event—it
is appropriate, and in future this step may be omitted. occurs at a given time point, and it causes a transition of the
• We find no beginning date information. (In the vital status of the person born (from FuturePerson to Alive).
absence of such information, there is no benefit to Our graphical representation here accordingly just depicts a
asserting backward clipping.) single time point (not an interval). We can use basically the
• We find (and have highlighted) a coarse-grained same workflow to capture a non-transition event (e.g., a legal
ending date (1984). We indicate that our fluent is trial) that occurs over more than one time point.
clipped forward, assuming that Jansa no longer attends
the school after graduation.
…Born on September 17, 1. Select event type.
1958 in Ljubljana, Jansa… 2. Specify argument identifiers,
respecting co-reference.
3. Select / highlight / designate
corresponding text.
4. Capture any hypothetical modality
info.
5. Capture any date info.
6. If an event’s date is otherwise
unconstrained w.r.t. reporting date:
BirthEvent(Janez_Jansa, Ljubljana) a. Capture reporting aspect.
b. Capture any reporting lag info.
1958-09-17 7. Capture any other relative temporal
info available.
Figure 4. Workflow to annotate a transition event
BirthEvent(Janez_Jansa, Ljubljana)
On demand:
1958-09-17 • Trace back from bounds to user-
attendsSchool(Janez_Jansa Ljubljana_University) entered information.
[0D,15Y3M12D] o Date of the birth event
[1958-09-18,1984-12-31] 1984
• Display / hide entered or inferred
bounds on…
Automatically: o Beginning points, ending points
• Display in order any time points that are o Durations
ordered unambiguously. • Focus on a particular time
• Display inferred bounds. window, location, person, …
o Rules: Can’t attend school before being • Highlight time points that are
alive; being born makes one alive. ordered / unordered w.r.t. to a
• Highlight bounds contradictions. selected, reference time point.
Figure 5. A time map with both a fluent and a transition event
judgments about the information they enter. If any inferred
B. Displaying integrated time maps bound seemed odd, a user could click on it to identify which of
Figure 5 illustrates a time map including both the birth his/her own entered information (then highlighted in the
event and the school attendance fluent from earlier figures. It display) might be responsible. The time map display tool
also suggests functional requirements to be satisfied would automatically launch such an interaction when it
automatically/by default and upon user demand. Note that we detected a contradiction among inputs.
now have automatically displayed—from on-line temporal The time map displayed in Figure 6 includes all the
inference—a lower bound on the fluent observation’s information from the source text that is necessary to answer
beginning date: Jansa could not have attended school until after Query1. The last fluent observation (at bottom right, where
he was born. (The “day” time point granularity used in our Jansa is prime minister) exercises workflow steps that earlier
machine reading evaluation leads to some non-intuitive effects, time map elements don’t. We have no ending date for this
like not being alive until the day after one is born. We can observation, but we do have present tense reference to Jansa as
easily correct this using an interval constraint propagation the prime minister, so we appeal to the reporting aspect
arithmetic including infinitesimals [6][7][8].) We’ve also Ongoing. From the source text he was elected prime minister
indicated bounds on the fluent observation’s duration on November 9, 2004, we can bound the observation’s
(calculated as ending date bounds interval minus starting date beginning point.
bounds interval). Propagating effects like this can maximize
visual feedback to users, expanding their basis for quality
BirthEvent(Janez_Jansa, Ljubljana)
1958-09-17
attendsSchool(Janez_Jansa Ljubljana_University)
[0D,9235D]
[1958-09-18,1984-12-31] 1984
personHasTitleInOrganization(Janez_Jansa Defence_Minister Slovenia)
1990 1994
personHasTitleInOrganization(Janez_Jansa Defence_Minister Slovenia) ElectionEvent(Slovenia,
Prime_Minister,
2000 2000 2004-11-09 Janez_Jansa)
personHasTitleInOrganization(Janez_Jansa Prime_Minister Slovenia)
[2004-11-09, 2007-12-28] 2007-12-28
Slovenia national election day 2004,
per user-established relative temporal
Reporting date, via reporting
reference
aspect Ongoing
Figure 6. A time map with more statements extracted from the same article
ElectionEvent(Slovenia,
Widget: User selects: Prime_Minister,
• Relation (from menu)
, Janez_Jansa)
• Subject , object (via mouse) 2004-11-09
‘
‘
‘
‘
‘
personHasTitleInOrganization(Janez_Jansa Prime_Minister Slovenia)
‘
‘
Figure 7. Using the GUI to establish relative temporal reference
Our user also has entered the election event. An election is access to these effectively, so that our user is empowered
not necessarily a fluent transition event, at least in that an without being overwhelmed.
elected candidate does not always take office immediately. So, Figure 8 shows formal statements that would be created
we rely on the user to establish relative temporal reference directly by the user’s actions (i.e., not also including those
between the election event and the fluent observation’s created indirectly by inference) in entering the information
beginning. See the depicted constraint, whose entry is reflected in our finished time map. We have highlighted
illustrated in Figure 7. Establishing relative temporal reference fluents and some other key statements, each of which appears
requires the selection of a pair of time points and/or intervals to near several related statements. Our time map represents
be related and of an appropriate temporal relation between Jansa’s birth event in a non-standard way, repeated here in
them. Here, we just need the time point at which the election different color type, beside the italicized, standard statements.
occurs to be less than or equal to the time point at which Jansa We have not similarly formalized the event of Jansa’s election
takes office. as PM, and Figure 8 includes just statements about that event’s
While a few common relations may be all that most users point of occurrence.
will ever need, we do have a lot of relations [8] that a user Clearly, we can do a lot of formal work for the user behind
could in principle choose from. We should be able to provide the scenes.
F_school: attendsSchool (Janez_Jansa Ljubljana_University)
holdsThroughout(F_school I_school)
clippedForward(F_school I_school)
hasTimeIntervalSpecString(I_school [,1984])
hasPersonBorn(birth Janez_Jansa)
occursAt(birth P_birth)
hasTimePointSpecString(P_birth 1958-09-17) BirthEvent(Janez_Jansa, Ljubljana)
hasPersonBorn(birth Janez_Jansa)
hasBirthEventGPE-spec(birth GPEspec)
hasCityTownOrVillage(GPEspec ljubljana_Ljubljana_Slovenia)
hasNationState(GPEspec Slovenia)
type(Defence_Minister MinisterTitle)
F_PTIO_DM_1: personHasTitleInOrganization(Janez_Jansa Defence_Minister Slovenia)
holdsThroughout(F_PTIO_DM_1 I_PTIO_DM_1)
clippedBackward(F_PTIO_DM_1 I_PTIO_DM_1)
clippedForward(F_PTIO_DM_1 I_PTIO_DM_1)
hasTimeIntervalSpecString(I_PTIO_DM_1 [1990,1994])
F_PTIO_DM_2: personHasTitleInOrganization(Janez_Jansa Defence_Minister Slovenia)
holdsThroughout(F_PTIO_DM_2 I_PTIO_DM_2)
clippedBackward(F_PTIO_DM_2 I_PTIO_DM_2)
clippedforward(F_PTIO_DM_2 I_PTIO_DM_2)
hasTimeIntervalSpecString(I_PTIO_DM_2 [2000,2000])
F_PTIO_PM: personHasTitleInOrganization(Janez_Jansa Prime_Minister Slovenia)
holdsThroughout(F_PTIO_PM I_PTIO_PM)
clippedBackward(F_PTIO_PM I_PTIO_PM)
hasBeginningTimePoint(I_PTIO_PM I_PTIO_PM_beginning)
hasTimePointSpecString(Slovenia_2004_Election_Day 2004-11-09)
timePointGreaterThanOrEqualTo(I_PTIO_PM_beginning Slovenia_2004_Election_Day)
hasReportingAspect(I_PTIO_PM Ongoing)
ref(annotation I_PTIO_PM)
annotation(document annotation)
hasReportingChronusSpecString(document 2007-12-28)
Figure 8. Formal statements associated with the time map in Figure 6
Query 1: Find all persons who were born in
BirthEvent(?person, Ljubljana)
Ljubljana in the 1950s and attended Ljubljana
University in the 1980s, the titles that they held,
the organizations in which they held these titles,
and the maximal known time periods over which
they attended and held these titles.
1950-01-01 1959-12-31
attendsSchool(?person Ljubljana_University)
?attendanceIntervalSpec
1980-01-01 1989-12-31
personHasTitleInOrganization(?person ?title ?org)
?titleIntervalSpec
Figure 9. The time map covering our Query1
• We are asking about only one answer (set of variable
C. Adaptation to test or application question authoring
values satisfying the query) at a time. The supporting
We might reuse much of the same machinery in a question statements in our earlier time map include three
authoring interface, in which a user can formalize a query, as separate sets of bindings for the variables ?title and
illustrated for Query1 in Figure 9. This time map display is ?org.
even less cluttered than the one for this query’s supporting We have introduced intervals to represent the 1950s and the
statements, for a couple of reasons. 1980s, and we have selected time point/interval relationships
• We are making general statements, rather than specific appropriate to the query’s conditions. These relationships are
ones, so don’t use as many dates or long identifiers. associated with particular idioms used in our formalization in
Rather, we use variables (here beginning with ?). Figure 10.
hasPersonBorn(?birth ?person)
hasBirthEventGPE-spec(?birth ?GPEspec)
hasCityTownOrVillage(?GPEspec ljubljana_Ljubljana_Slovenia)
hasTimeIntervalSpecString(?I_range_birth [1950-01-01,1959-12-31])
occursWithin(?birth ?I_range_birth)
hasTimeIntervalSpecString(?I_range_school [1980-01-01,1989-12-31])
holdsWithin(?F_school ?I_range_school)
maximallyHoldsThroughout(?F_school ?I_school)
hasTimeIntervalSpecString(?I_school ?attendanceIntervalSpec)
?F_title: personHasTitleInOrganization(?person ?title ?org)
maximallyHoldsThroughout(?F_title ?I_title)
hasTimeIntervalSpecString(?I_title ?titleIntervalSpec)
Figure 10. Formalization of the query in Figure 9’s time map, covering Query 1
Our query asks about the “maximal known time periods” Others have applied limited temporal reasoning in post-
over which the fluents hold, and we associate (via a query processing of temporal annotations, to…
authoring workflow step) an “interval spec” variable with each A. Compute the closure of qualitative pairwise time
fluent’s observation interval. Per our formalization, this will be interval relations, as one step in assessing a machine
bound, on successful query execution, to a string that describes reader’s precision and recall performance (see section
lower and upper bounds on the observation interval’s A)
beginning point, ending point, and duration. The formalization B. Ascertain the global coherence of captured qualitative
uses the properties occursWithin (for born in the 1950s) and relations (see section B).
holdsWithin (for attended school in the 1980s) to accommodate Our implementation can go further, as described below.
the temporal relations selected for the query authoring time
A. Quantitative temporal relation annotation evaluation
map. We know to use maximallyHoldsThroughout (vice the
less restrictive holdsThroughout) for the fluents’ observation Evaluating temporal annotations typically has been limited
intervals because the query’s author has included (via the to (Allen’s [1]) qualitative relations (e.g., before, overlaps,
invoked widget) associated spec string variables. contains), and quantitative information about dates and
Thus, it appears that we might enable non-specialists to durations typically has been evaluated only locally—at the
author effective test queries (or, in a transition/application level of temporal expressions (AKA “TIMEXs” [3]). The
setting, domain queries), without requiring the intervention of a reasoning applied has been strictly interval-based, neglecting
KR specialist. One angle on this proposed work might be to important quantitative information about dates and durations
determine the extent to which readers who are not (temporal) widely available in text. This approach is taken by Setzer et al.
knowledge representation specialists can perform such tasks [9], e.g.
consistently—alternatively, to determine the amount of training Our temporal reasoning engine, which is point-based,
(e.g., pages of documentation, number of successfully naturally accommodates arbitrary bounds on the metric
completed test exercises) required to qualify an otherwise-non- durations that separate time points and uses global constraint
specialist to perform the task well. That said, rather than propagation to calculate earliest and latest possible dates/times
“dumb down” the task, to accommodate non-expert readers, we for any point (including the beginning and ending points of all
propose to ratchet up annotator performance expectations—to temporal intervals), as well as tightest bounds on durations.
achieve the highest-quality results possible so that we can drive This approach also usually affords sufficient global
research regarding extraction of temporal knowledge by perspective for a robust recall statistic. Adapting the standard
machines from text to new levels of sophistication. The approach for evaluating interval relations [11], we can discard
machine reading researchers whose systems are under from our gold standard annotations any redundant relations
evaluation quite reasonably ask, before they embark on a until we determine a set spanning globally calculated bounds.
mission of technological advancement, “Is this task feasible for Then we can count members of this spanning set whose
humans, with acceptable consistency?” We’d like to answer addition to a user’s candidate set results in tightening of bounds
that question in the best way that we can. in the latter, to determine recall.
Only when every member of a set of points is unrelated to
V. RELATED WORK AND PROPOSED ADVANCES the calendar (i.e., we have only point ordering and interval
Beyond test questions and answers, the entire machine duration information) do we lack calendar bounds supporting
reading community would benefit from having a large volume meaningful recall assessment. Then, however, by choosing any
of good temporal logic annotations available. Time is a key point in a connected set to serve as a reference (in place of the
topic in language understanding, engendering much current calendar), we can apply the same approach as above.
community interest. TimeML [4], which emphasizes XML It may reasonably be argued that at some threshold of
annotation structures rather than RDF ontology and representational complexity the brute force transitive closure-
relationships, has been used in the TempEval temporal and-spanning tree approach to computing recall and precision
annotation activities (see, e.g., www.timeml.org/tempeval2/) of an extracted knowledge base (set of statements) must
and advanced as an international standard [5]. We are become impractical. Our quantitative temporal statements are
interested in exploring the synergy between this work and ours. certainly richer than the typical qualitative ones, and
(depending on knowledge base size) we may be pushing up
against this threshold with them. Our query answering
evaluation paradigm is more broadly applicable, presuming REFERENCES
inference over extracted statements remains tractable—for the [1] J. Allen, “Maintaining knowledge about temporal intervals,” in
queries of interest. Communications of the ACM. 26, pp. 832–843, November 1983.
B. Ascertaining global coherence [2] A. Bittar, P. Amsili, P. Denis, L. Danlos, “French TimeBank: An
ISO-TimeML annotated reference corpus,” in Proceedings of the
Waiting until annotation is done to infer bounds and detect 49th Annual Meeting of the Association for Computational
contradictions neglects opportunities to give annotator’s (e.g., Linguistics, pp. 130–134, 2011.
time map-based) feedback and receive their best-effort [3] L. Ferro, L. Gerber, I. Mani, B. Sundheim, and G. Wilson,
corrections. As Bittar et al. [2] comment, “Manually “TIDES 2005 standard for the annotation of temporal
eliminating incoherencies is an arduous task, and performing expressions,” MITRE Corporation, 2005.
an online coherence check during annotation of relations would [4] J. Pustejovsky et al., “TimeML: Robust specification of event and
be extremely useful in a manual annotation tool.” We propose temporal expressions in text,” AAAI Technical Report SS-03-07,
this. 2003.
[5] J. Pustejovsky, K. Lee, H. Bunt, and L. Romary, “ISO-TimeML:
VI. SUMMARY
an international standard for semantic annotation,” in
We have outlined existing and anticipated future benefits of Proceedings of the Seventh International Conference on
an end-to-end methodology for… Language Resources and Evaluation (LREC), Malta. May 18-21,
• Annotating formal RDF statements representing 2010.
temporal knowledge to be extracted from text [6] R. Schrag, J. Carciofini, and M. Boddy, “Beta-TMM Manual
• Authoring and validating test and/or application (version b19),” Technical Report CS-R92-012, Honeywell SRC,
queries to exercise that knowledge. 1992.
These capabilities are supported by an implemented temporal [7] R. Schrag, M. Boddy, and J. Carciofini. “Managing disjunction
reasoning engine. They and the engine are intended to support for practical temporal reasoning,” in Principles of Knowledge
a timeline tool conceived for use by intelligence analysts. We Representation and Reasoning: Proceedings of the Third
have explained how these benefits can advance machine International Conference (KR-92), pp 36–46, 1992.
reading technology by increasing both sophistication and [8] R. Schrag, “Best-practice time point ontology for event calculus-
quality expectations about temporal annotations and extraction. based temporal reasoning,” 7th International Conference on
Semantic Technologies for Intelligence, Defense, and Security
ACKNOWLEDGMENT (STIDS), 2012.
[9] A. Setzer, R. Gaizauskas, and M. Hepple, “The role of inference
Thanks to other participants in DARPA’s Machine Reading in the temporal annotation and analysis of text,” Language
research program—especially to other members of the SAIC Resources and Evaluation v. 39, pp. 243–265, 2005.
evaluation team, including Global InfoTek (the author’s former [10] M. Shanahan, “The event calculus explained,” in Artificial
employer). Intelligence Today, ed. M. Wooldridge and M. Veloso, Springer
Lecture Notes in Artificial Intelligence no. 1600, pp.409-430,
1999.
[11] X. Tannier and P Muller, “Evaluation metrics for automatic
temporal annotation of texts,” in Proceedings of the Sixth
International Conference on Language Resources and
Evaluation (LREC’08), 2008.