Document authors cannot routinely afford the overhead imposed by current semantic annotation tools. Some characteristics of their task can be exploited to provide them with a tool that will reduce the effort required to create both the document content and their accompanying semantic annotations. SemanticWord is such a semantic annotation tool. SemanticWord is an environment based in MS Word that integrates content and markup authoring, providing customizable tools that allow simultaneous generation of content and semantic annotations, an annotation scheme that allows annotations to be reused when content is reused, a customizable library of templates containing partially annotated text, and an automatic information extraction system with the tools for refining and augmenting its output.
Automatic Information Extraction systems have been sug• • • • •
SemanticWord offers an environment for authoring
annotated text documents based in MS Word. Its aim is to
reduce the burden involved in authoring semantic
annotagested as an alternative method for generating semantic
annotations. Unfortunately, this technology is only able to
extract sufficient information to fill in a flat template and
cannot capture the relationship graph that connects the
instances ([
Clearly, current markup authoring tools are inadequate for the task of routinely authoring content. Fortunately, some characteristics of this task, as it applies to some authors, can be exploited to reduce the cost of producing these annotations. Some of these characteristics are:
fined to a few topics. In this case it is worthwhile to spend some effort in setting up an environment tailored to these topics. The savings from producing multiple documents will more than recoup the tailoring cost.
ple, different documents include common actors and places, share the same context, or update on previous accounts. This characteristic can be exploited to reuse the annotations along with the content.
SemanticWord is a semantic annotation tool designed with this kind of task in mind. Some of the features included in SemanticWord are:
to be reused when content is reused.
the tools for refining and augmenting its output. tions. Authors are given a familiar and uniform environment where the creation of content and semantic descriptions can be freely interleaved. In many case both of them can be achieved in a single operation.
SemanticWord extends MS Word in several dimensions (see Figure 1). First, MS Word GUI is augmented with toolbars that support the creation of semantic descriptions (or annotations) that are attached to text regions. The GUI is also extended to show these annotations embedded within the text and to support their direct manipulation through mouse gestures. Second, SemanticWord extends Word’s reach by opening a channel to the Semantic Web. Content from the Semantic Web (both ontology definitions and factual descriptions) is brought into SemanticWord to compose annotations that are later dumped back into the Semantic Web. Third, SemanticWord extends Word services by integrating AeroDAML, an automated information extraction system. AeroDAML analyzes and annotates the text of the document as it is being typed, appearing to the author as a service analogous to Word’s spelling and grammar checking. Finally, SemanticWord supports the rapid composition of annotated text through template instantiation.
The above extensions were implemented using standard Microsoft extensibility technology. Annotations are rendered with ActiveX controls that can be placed in a document, implement their own behavior, control their GUI, and save their internal state. Automatic text analysis is driven by SmartTags technology that supports background parsing and tagging of the document text as it is being typed. The rest is supported by an Office COM Add-in that responds to MS Office/MS Word built-in events (e.g., DocumentOpen) and extend Word’s menus and toolbars. The document content is manipulated through Word’s COM API.
SemanticWord annotations are based in the DAML+OIL
language [
Annotated Templates
MS
Word Semantic
Word Ontology and KB Server
AeroDAML
(IE) language developed for the Semantic Web that supports the definition of machine-readable ontologies and the linking of terms in documents to ontologies.
SemanticWord annotations are attached to regions of text, not to the document as a whole. There are two types of annotations: instances references and triple bags. An instances reference associates a text region with a “referencable” instance of a class. Triple bags describe the content of a text region with a collection of triples that follow DAML+OIL’s subject-predicate-object model. The subject is an instance, the predicate is a property defined in an ontology, and the object can either be an instance or a value. SemanticWord Annotations are retained across text copy/cut and paste operations.
Figure 2 illustrates a fragment of an annotated document. An instance reference is rendered by enclosing the annotated text between square brackets and with an icon adjacent to the closing bracket. A triple bag is rendered by enclosing the annotated text between square brackets and displaying a checkbox and a triples table adjacent to the closing bracket. The checkbox allows the user to display or hide the table. To facilitate the handling of heavily annotated documents, the text associated to an individual annotation can be highlighted and all annotation marks can be made invisible.
An Instance reference icon can be dragged and dropped over a cell corresponding to the subject or object of a triple. Cells filled using this method do not store a direct reference to the dropped instance but rather establish a link with the dragged instance reference. Updating the linked instance reference to refer to a different instance will alter the triple too. This level of indirection improves maintainability. Triple cells can also be filled by picking instances and properties from special purpose browsers called choosers (See Figure 3). Choosers can use the values already stored in a triple to constrain the lists of choices offered to the user. For example, if the subject and object of a row are already filled in then the corresponding property chooser will only show the properties whose domain and range are consistent with those entries. Because SemanticWord does not enforce consistency, these constraints can be relaxed. The choosers also provide other filters for constraining the choices shown. For example, the instance chooser includes filters for listing only the instances that have already been referenced in the document. The instance choosers can selectively list instances corresponding to preexisting semantic web markup (provided by the Ontology and KB Server) or new instances defined in the current document. They also allow users to create new locally defined instances or provisional instances (described below), a function that a user would invoke if the listed choices do not include the desired instance. SemanticWord does not impose any order for filling in table cells, and can persist the state of tables containing rows with one or more empty cells.
Locally defined instances are instances that cannot be referenced from outside the document. Provisional instances are an artifact to postpone the identification of an instance that is being used to describe relationships. Ultimately, provisional instances must be replaced by references to external or locally defined instances. SemanticWord keeps track of the provisional instances and assists users in replacing them.
One obstacle that we noticed in other systems when composing a triple is that the role of the instances in the triple cannot be established before examining the definition of the predicate property. For example, determining who is the subject and who is the object in the relationship between an employee and her employer depends on how the property that relates both of them is (arbitrarily) defined. Assigning an instance to the subject or the object of a triple prematurely might preclude the possibility of establishing the relationship. To avoid this problem in SemanticWord, the property chooser can optionally list reversed properties. Reversed properties are ordinary properties that assume the subject and the object of a triple are switched. Reversed properties is only an artifact to add another degree of liberty in the order in which the triple arguments are filled -the generated DAML markup switches the subject and object of a triple when a reversed property was selected.
SemanticWord was conceived with the goal of minimizing the burden involved in authoring semantic annotations. This burden is reduced through several techniques.
SemanticWord provides an environment for authoring semantic annotations that is tightly integrated to MS Word.
Word is the most massively adopted product for authoring text documents. SemanticWord includes a set of tools that economize the production of semantic descriptions and exploit opportunities for the simultaneous generation of text and annotations. Two examples of these tools are personal class toolbars and the cascading class menus, both illustrated in Figure 4.
Personal Class Toolbars: Personal Class Toolbars constitutes a convenient tool for generating both content and annotations together with just one mouse click. Users can create any number of Personal Class Toolbars, each one of them tied to a single class. Each personalized class toolbar contains an instance selection combo box and buttons to create instance references corresponding to the selected instance or a new one. If at the time the user creates an instance reference the document contains a selected region of text, then the instance reference will be attached to that region. If no text is currently selected, then both the “label” of the instance reference will be inserted in the document at the current text insertion point, and the new reference will be associated with the inserted text.
Personal class toolbars save effort when a small percentage of classes or instances account for a substantially larger percentage of the instance references that an author will need.
Classes Cascading Menu: A cascading class menu includes an entry for every named class in the ontology attached to the document. This menu gives users access to most of the operations related to ontology classes, including defining new instances, creating personal class toolbars, and opening instance choosers. When a user executes
The circular icon containing an I Bar (like that adjacent to “BAGRAM”) references an external instance from the semantic web. A smiley face icon (like that adjacent to “weapons cache”) references a locally defined instance. The boxed legend below the “weapons cache” instance reference is its tool tip. If the instance icon in the subject or object column of a table is overlapped by a small arrow in its lower left corner (like the one in the object column of the first row) then the cell is linked to an instance reference annotation. Modifying or deleting the linked instance reference will affect the triple too. If the instance icon is not overlapped by a small arrow (like the one the subject column of the first row) then the cell contains a direct reference to an instance. any of these functions from this menu, the menu entry corresponding to the selected class is duplicated and placed at the top of the menu so the user can access it easily the next time that she needs it. The cascading hierarchy is determined by the subclass hierarchy of the ontology. Classes with multiple superclasses appear in the cascade under each superclass.
Direct Manipulation of Annotations: Direct manipulation of annotations is another method of simplifying the production of semantic annotations. In SemanticWord users can compose semantic annotations by manipulating other annotations that are placed within the document. For example, the subject and object of a triple can be filled by dragging instance references annotations over the triple. For some users this method is faster and more natural than searching for those same instances in instance browsers.
Authors should not be forced to follow a strict order in carrying out the many steps involved in authoring semantic descriptions. Many of the features that support this principle have been introduced before. These features are summarized in this section.
• • •
Even the determination of which instance is the subject and which is the object can be postponed by means of the reversed properties. New instances can be created from the instance choosers avoiding a disruption of the triple’s composition process. Unlike other annotation tools, triples are laid out in a tabular structure rather than in a tree or other structures that impose a topological dependency among its nodes.
pose a triple that violates ontology constraints. The user can make the changes that would fix this conflict at a time convenient to her. Consistency is taken into account when filtering suggested choices for composing a triple, but the user can deactivate these filters with a single button click.
instance can still be used to describe relationships. This is achieved through the use of provisionary instances, which can be used wherever definitive instance can but remind the user of the unconcluded task. SemanticWord will assist users in assigning identity to these instances.
Annotations are attached to text regions and are going to be reused when those regions are reused. In particular, annotations are carried over along text cut/copy and paste operations and when fragments of a document are reused elsewhere in the same document in other documents based on the same ontology.
SemanticWord integrates an information extraction system
(IES). Automatic information extraction technology
promises to significantly reduce the human overhead involved in
the semantic annotation task. Although this technology has
not reached a level of sophistication required to capture
deep relationships in text ([
SemanticWord uses AeroDAML, an IES developed at
Lockheed Martin [
Property Chooser
The choices correspond to the filling of the property and object columns of the second row of the triples table of Figure 2. The listed choices are constrained by the content of the other cells of the selected triple. These filters can be relaxed by toggling the buttons on the top toolbars. The Instance chooser also supports the definition of new instances. ing proper nouns but their classification tends to be overly general. It also fails to recognize most of the relations between instances. For example, AeroDAML succeeds in classifying Kabul as a Place but failed in finding the more specific class City, perhaps because there was nothing in the text that might clue AeroDAML about this fact. SemanticWord let AeroDAML to recognize and classify proper nouns but expects the user to refine the classification and to specify their relationships.
SemanticWord drives the information extraction process on the fly. As the user types the content of the document, a background thread feeds new or modified text to AeroDAML in paragraph units (roughly), obtains the extracted entities with their position in the text, and underlines those text regions with a blue wiggly line. This procedure is carried out in a way that resembles Word spelling and grammar checking and is implemented in terms of Microsoft SmartTags technology.
The user can examine the extracted entities and convert them into instance reference annotations. As part of this conversion the user has the option of refining the extracted type. Once an extracted entity has been transformed into an instance reference it behaves just like a natively created instance reference. In particular, it can be dragged and dropped onto cells of triple bags to describe the relationships that AreoDAML missed.
Annotated text templates reduce the amount of work involved in authoring both semantic annotations and document content. A template consists of a text fragment annotated with semantic and template related descriptions, and persists as a (typically quite small) word document.
A template may be inserted into a document just like any other document. Both the text and annotations of the template are copied into the target document. After insertion, the copy can still be subjected to further editing and annotating.
Templates are authored in SemanticWord in template design mode. All annotations tools described previously are also available for annotating templates in template design mode plus an additional toolbar that includes the template specific authoring tools described below. We expect that non-programmers would be able to author templates.
Instance Placeholder: An instance placeholder annotates a region of text that needs be replaced by an instance reference when the template is used in a document. It also serves as the surrogate for an instance reference, and as such, it can participate as the subject or the object of one or more triples in the template’s triple bags.
An instance placeholder is rendered like an instance reference annotation but with a different icon. In design mode this icon can be dragged over triple tables to compose the semantic annotations that describe the template. It can also be dragged over another instance placeholder to specify a co-reference requirement. In instantiation mode, this icon is a drop site for the concrete instance that is going to be bound to the instance placeholder.
When an instance placeholder is bound to an instance reference, the label of the instance reference replaces the template’s text and all co-referential instance placeholders are bound to that instance.
Optional group: An optional group delimits a region of
The last two toolbar rows belong to SemanticWord. The first row contains two juxtaposed personal class toolbars. The first one is tied to the class “Terrorist Organization” and has selected the instance “al Qaeda”. The second one is tied to “Country” and has selected “Afghanistan”. Clicking in the Check button will generate both the text and the annotation corresponding to the selected instance. The other buttons are for defining new instances before inserting their text and annotation. The last toolbar row has its classes cascading menu opened. This menu provides access to several class related functions. The most recently chosen classes get added to the top of the menu (like Weapon, Terrorist Organization, and Country) for easy access. text in the template that can be optionally included in the instantiation of the template. The text delimited by an optional group can contain annotations and other groups. In particular, it can contain instance placeholders. Opting to delete an optional group from an instantiated template will automatically remove any triples having a cell linked to an instance placeholders within the deleted group.
Repeated group: Like an optional group, repeated group annotation delimits a region of text and can also contain other groups and annotations. During instantiation the user can ask that a repeated group be replicated any number of times. Each replication of the group creates its own incarnation of the instance placeholders that it contains. When the group is replicated, all triples with cells linked to the instance placeholders contained in the group are replicated as well.
The utility of annotated templates is enhanced by the IES described above. The IES analyses the document and generates instance reference annotations corresponding to the concrete entities mentioned in the text. These instance references can be dragged over the template instance place holders to instantiate the template and generate instantiated triples describing their relationships.
In SemanticWord, semantic descriptions are distributed throughout the document and attached to text regions that “support” their content. This is not a requirement for the semantic web. Most of the semantic markup authoring tools reported in the literature do not adopt this practice. The descriptions they produce are associated only with a document, not with portions of that document.
We speculate that relating a semantic description to the text that supports it has advantages in terms of annotation authoring, reuse, maintenance, and validation. However, we also recognize that this practice might introduce unnecessary complications.
Some of the advantages of attaching semantic descriptions to text are: • • •
Annotations are carried over along text cut/copy and paste operations and when document fragments are reused in other documents.
the content of the document can more easily be validated and maintained.
Authors might find it natural to find annotations by finding, through familiar text search/scroll mechanisms, the text to which the annotations are attached. Contrast this with browsing the semantic markup directly. For example, in SemanticWord authors compose triples by dragging around instance references placed within the text. • • • •
the text fragment, or a region containing it, is deleted. Generally, this is desirable because the document’s content no longer supports the statement formalized by the deleted annotation.
region – particularly at its boundaries – heuristics must be used to adjust the boundaries. The use of paired brackets for rendering these regions keeps the user informed of the result of these heuristics.
Although SemanticWord is biased toward the attachment semantic annotations to text, it does not mandate it, opening a whole spectrum of hybrid compromises. For example, authors might choose to attach instance references to text but to describe their relationships in a single global triple bag. Moreover, not even the instance reference annotations are required because the triples can be filled directly from instance choosers. More serious use of SemanticWord will be required to weigh the pros and cons of this approach..
Research in semantic annotations is still in its infancy. A
number of systems have been developed to date that
demonstrate different capabilities. However, the approaches
adopted by these systems do not necessarily compete
against each other but rather address different issues.
Ont-O-Mat ([
S-CREAM [
SMORE [
Among the described annotation tools only SemanticWord provides an environment for document authoring and semantic annotation that extends a COTS product that authors have already adopted (MS Word). SemanticWord is also the only one that associates semantic annotations within text regions and consequently facilitates annotation reuse and maintainability.
SemanticWord integrates into a widely used COTS product an environment for authoring document content and annotations. It includes several features intended to minimize the cost involved in authoring semantic annotations: customizable tools for generating content and annotations simultaneously, direct manipulation of annotations embedded in the document, reusable annotations, annotated text templates, and an information extraction system including support for refining and augmenting its output.