EXCITEMENT is a novel, open software platform for Textual Entailment (TE) which uses the UIMA framework. This paper discusses the design considerations regarding the roles of UIMA within EXCITEMENT Open Platform (EOP). We focus on two points: a) how to best design the representation of entailment problems within UIMA CAS and its type system. b) the integration and usage of UIMA components among non-UIMA components.
Textual Entailment (TE) captures a common sense notion of inference and
expresses it as a relation between two natural language texts. It is de ned as
follows: A Text (T) entails a Hypothesis (H), if a typical human reading of T
would infer that H is most likely true [
EXCITEMENT open platform (EOP) has been developed to address those
problems. It is a suite of textual inference components which can be combined
into complete textual inference systems. The platform aims to become a common
development platform for RTE researchers, and we hope that it can establish
itself in the RTE community in a similar way to MOSES [
Compared to Machine Translation, however, a major challenge is that semantic processing typically depends on linguistic analysis as well as large knowledge sources, which is a direct source of the reusability problems mentioned above. In this paper, we focus on the architectural side of the platform which was designed with the explicit goal of improving component re-usability. We have adopted UIMA (Unstructured Information Management applications) and UIMA CAS (Common Analysis Structure) as the central building blocks for data representation and preprocessing within EOP.
One interesting aspect is that our adoption of UIMA has been partial and parallel. By partial, we mean that there are two groups of sharable components within EOP: the \core" components and the \LAP" components (see Section 2). We have adopted UIMA only for LAPs; however, we use UIMA CAS as one of the standard data containers, even in non-UIMA components. Parallel refers to the fact that we allow non-UIMA components to be integrated into our LAPs transparently. 2
RTE systems traditionally rely on self-de ned input types, pre-processing (linguistic annotation) representations, and resources, tailored to a speci c approach to RTE. EXCITEMENT open platform (EOP) tries to alleviate this situation by providing a generic platform for sharable RTE components. The platform has the following requirements.
Reusing of existing software : The platform must permit easy integration and re-using of existing softwares, including language processing tools, RTE components, and knowledge resources.
Multilinguality : The platform is not tied to a speci c language. Adding suites for a new language in the future should not be restricted by the platform design.
Raw entailment problems
EXCITEMENT Platform
Linguistic Analysis
Pipeline (LAP)
Linguis7c Analysis Tools Component Independence : Components of EOP should be independent and complete as they are. So they can be used by di erent RTE approaches. This is also true for linguistic annotation pipelines and their components: An annotation pipeline as a whole, or an individual component of the pipeline, can be replaced with equivalent components. Google." as an entailment. The example system gets a dependency parse tree of the text, and starts the rewriting process. On each iteration, it generates possible entailed sentences by querying knowledge bases. In the example, lexical knowledge is used on the rst rewriting (buy entails acquire), and syntactic knowledge (change to passive voice) is used on the second derivation. The process will generate many derived candidates per iteration. The algorithm must employ a good search strategy to nd the best rewriting path from text (T) to hypothesis (H).
On this example, there are three major component types. One is the knowledge component type that supports knowledge look-up, another is generation of derived parse trees, and nally the decision algorithm itself drives the search process and makes the entailment decision. Expressing behaviors of such components in terms of annotations on the artifact, might be possible, but is very hard and counter-intuitive.
Following this line of reasoning, we decided that the EC components are better thought of as Java modules whose common behavior is de ned by a set of Java interfaces, data types, and contracts, and have de ned them accordingly in the EXCITEMENT open platform speci cation.1 More speci cally, we have de ned a typology of components. They include a type for the top-level EDA as well as (currently) ve major component types: (1) a feature extractor (get a T-H pair CAS, return a set of features for the T-H pair); (2) a semantic distance calculator (get a T-H pair CAS, return semantic similarity); (3) a lexical resource type (lexical relation database); (4) a syntactic resource type (phrasal relation database); (5) an annotation component (dynamic enrichment of entailment problems).
Although UIMA components are not suitable for conceptualizing inference components, we decided to keep CAS as the data container even in the EC components as far as possible to take advantage of the CAS objects created in the LAP. Thus, various components (including EDAs) gets CAS (as JCas) as an argument on their methods. Also note that LAP and EC boxes are independent: 1 Speci cation and architecture for EXCITEMENT open platform, http://excitementproject.eu/index.php/results
Entailment Metadata language, channel, docId, collectionId, ...
Entailment Pair
pairId, goldAnswer, text, hypothesis Text View
Subject of Analysis
That was ...
POS Annotations
Token Annotations Dependency Dep Annotations
Pos.
PR Token Token
Pos.
V
Gov dep.NSUBJ ... ... as long as the CAS holds correct data, the EC components does not care which pipeline has generated the data.
CAS for Entailment Problems The input to any RTE system is a set of entailment problems, typically TextHypothesis pairs, each of which is represented in one CAS. Figure 3 shows a pictorial example of the CAS data structure for the example pair (T, H1) from Section 1. It contains the two text fragments (in two views) and their annotations (here, POS tags and dependencies), as well as global data such as generic metadata (e.g., language) and entailment-speci c metadata (e.g., the gold-standard answer).
On the level of the CAS representation, we had to address two points: one is the representation of entailment problems in terms of CASes, the other one is the type de nitions.
Regarding the rst point, general practice in text analysis use cases is to
have one UIMA CAS corresponding to one document. This suggests representing
both text and hypothesis (including, if available, their document context) as
separate CASes. However, we decided to store complete entailment problems as
individual CASes, where each CAS has two named views (one view for text, the
other for hypothesis). This approach has two major advantages: rst of all, this
enables us to represent cross-annotations between texts and hypotheses, notably
alignments, which can be added by annotators. Second, this enables us to de ne a
straightforward extension from \simple" entailment problems (one text and one
hypothesis) to \complex" entailment problems (one text and multiple hypotheses
or vice versa, as in the \RTE search" task [
Regarding the second point, we adopted the DKPro type system [
Details about the newly de ned types can be found in the platform speci cation, and the type de nition les are part of the platform code distribution. 3.2
Wrapping the Linguistic Annotation Pipeline One decision that may be surprising at the rst glance is that we de ned our own top-level Java interface for users of the LAP that hides UIMA's own runtime access methods. This interface dictates the common capabilities that all pipelines of LAP should provide.
The reason for this decision is twofold and pragmatic in nature, making transitioning to and using the EOP as easy as possible for developers.
The rst aspect is the learning curve. We would like to avoid the need for Entailment Core developers to deeply understand UIMA AEs and Aggregated Analysis Engines (AAEs). We feel that a deep understanding of these points requires substantial e ort but is not really to the point, since many EC developers will only want to use pre-existing LAPs. By making the UIMA aspect of the LAP transparent to the Entailment Core, EC developers do not need to know how the LAP works internally beyond knowledge of the (fairly minimal) LAP interface. Of course, the EC developers still need to understand UIMA CAS very well.
The second aspect is migration cost. If the LAP pipelines were nothing but UIMA AEs, all analysis pipelines of existing RTE systems would have to be deeply refactored, which comes at a considerable cost. Our approach allows such analysis pipelines to be kept largely intact and merely surrounded by a wrapper that provides the requires functionality and converts their output into valid UIMA CASes according to the EOP's speci cation.
Nevertheless, there are good reasons to encourage the use of AE-based LAPs: AE-based components are generally much more exible, and they are very easy to assemble into AAE pipelines. Therefore, we encourage AE-based LAP development by providing ready-to-use code that implements our LAP interface, taking a list of AEs as input. Thus, if the individual components are already present as AEs, the implementation e ort to assemble them into a LAP is near zero. In this sense, we see our LAP interface as a thin wrapper above UIMA with the purpose of enabling peaceful co-existence between UIMA and non-UIMA pipelines. In the long run, we also hope to provide some new AEs back to the UIMA community. 4
In this section, we discuss two open questions that we are facing in future work. CAS in non-UIMA environments. There is considerable number of best-practice strategies for handling CAS objects (reset the data structure instead of creating a new one; use a CAS pool instead of generating multiple CASes, etc). When a CAS is used in an UIMA context (i.e., in the LAP), it is not hard to guide the developers to follow these rules. However, with CAS being used as a general data container throughout the EOP, developers also often encounter CAS (JCas) objects outside speci c UIMA contexts, and we have found it harder to guide the developers towards \proper usage".
For example, one part of the EXCITEMENT project is concerned with the
construction of Entailment Graphs [
Annotation Styles: Hidden dependencies. One of the EOP design requirement was the clear separation of LAP and EC. This has been fairly well achieved, at least on a technical level.
However, it is clear that there are still implicit dependencies between linguistic analysis tools and entailment core components. Consider the case of syntactic knowledge components such as DIRT-style paraphrase rules in the Entailment Core. Such components store entailment rules as pairs of partial dependency trees which have typically been extracted from large corpora. If the corpus used for rule induction was parsed with a di erent parser than the current entailment problem, then matching the sentence against the rule base will result in missing rules, due to di erences in the analysis style. Note that this implicit dependency does not break the UIMA pipeline, since it does not involve the use of a novel type system, but rather di erences in the interpretation of shared types. We are currently investigating what type of \style di erences" can be observed from actual annotators.
In this paper, we have provided an overview of the EXCITEMENT open platform architecture and its adoption of UIMA. We have adopted and adapted UIMA CAS and the DKPro type system as a exible, language-independent data container for Textual Entailment problems. UIMA also provides the backbone for platform's LAP components. There are several open issues that is to be resolved in the future, but the EXCITEMENT project has already pro ted substantially from the use of the abstractions that UIMA o ers as well as the integration of existing components from UIMA communities.
The rst version of EXCITEMENT open platform has been nished2 with three fully running RTE systems integrated with all core components and annotation pipelines. The platform currently supports three languages (German, Italian and English), and is also shipped with various tools and resources for TE researchers. We believe that the platform will become a valuable tool for researchers and users of Textual Entailment.
Acknowledgment. This work was supported by the EC-funded project EXCITEMENT (FP7 ICT-287923).