The process of document annotation for the Semantic Web is complex and time consuming, as it requires a great deal of manual annotation. Information extraction from texts (IE) is a technology used by some of the most recent systems for actively supporting users in the process and reducing the burden of annotation. The integration of IE systems in annotation tools is quite a new development and in our opinion there is still the necessity of thinking the impact of the IE system in the process of annotation. In this paper we discuss two main requirements for active annotation: timeliness and tuning of intrusiveness. Then we present and discuss a model of interaction that addresses the two issues and Melita, an annotation framework that implements such methodology.
The effort behind the Semantic Web (SW) is to add content to
web documents in order to access knowledge instead of
unstructured material, allowing knowledge to be managed in an
automatic way. Much work is done on (1) the definition of
standards for representation of knowledge (e.g. XML, RDF, OIL),
(2) the definition of structures for knowledge organization (e.g.
ontologies) and (3) the population of such knowledge structures.
(1) and (2) actually provide the necessary infrastructure for the
Semantic Web. (3) actually requires methodologies for creating
semantically enriched documents. It is reasonable to expect users
to manually annotate new documents up to a certain degree, but
annotation is a slow time-consuming process that involves high
costs. Therefore it is vital for the Semantic Web to produce
automatic or semi-automatic methods for extracting information
from web-related documents, either for helping in annotating new
documents or to extract additional information from existing
unstructured or partially structured documents. In this context,
Information Extraction from texts (IE) is one of the most
promising areas of Human Language Technologies for the
Semantic Web. IE is an automatic method for locating important
facts in electronic documents for successive use, e.g. for
annotating documents or for information storing (such as
populating an ontology with instances). In this perspective IE is
the perfect support for knowledge identification and extraction
from Web documents as it can – for example - provide support in
documents annotation either in an automatic way (unsupervised
1 Department of Computer Science, University of Sheffield, Regent
Court, 211 Portobello Street, S1 4DP, Sheffield, UK, email
{fabio|alexiei|yorick}@dcs.shef.ac.uk
extraction of information) or semi-automatic way (e.g. as support
for human annotators in locating relevant facts in documents, via
information highlighting). In the last years a big effort has been
spent in the IE community on the use of Machine Learning for
helping in porting IE systems to new applications/domains
[
Timeliness: when the IE system (IES) is trained on blocks of texts, there is a time gap between the moment in which annotations are inserted by the user and the moment in which they are used by the system for learning. User and system work in strict sequence, one after the other. This sequential scheduling hampers true collaboration. If a batch of texts contains many similar documents, users may spend a lot of time in annotating similar documents without receiving feedback from the IES for the simple reason that no learning is scheduled for the moment. The IES is not supportive to the user neither it is efficient since similar cases are of very little use for the learner because they cannot offer the variety of phenomena that empower learning. The bigger the size of the batch of texts the worse the problem of lack of timeliness is. A true collaboration implies a (re)training of the system after every annotated text is released by the user. Training can take a considerable amount of CPU time, therefore stop the annotation session for a while. A positive collaboration requires not to constraint the user time to the IES training time (otherwise the intrusiveness of the IES increases). We believe that an intelligent scheduling is needed to keep timeliness in learning without increasing intrusiveness. It is also important to bear in mind that timeliness is a matter of perception from the user side, not an absolute feature, therefore what is important is that users do not perceive any delay or impediment. The focus is on effective collaboration not on timeliness at any cost.
Intrusiveness: in all the experiments with active learning done so far it turned out difficult to avoid bothering users with proposed annotations generated by unreliable rules (e.g. induced using an insufficient number of cases). This problem is mainly related to the tuning of the IES behaviour. Some IES provide internal tuning methods for balancing features such as precision and recall or the minimum number of cases to be covered in order to accepted a rule for annotation. Such tuning methodologies are designed for IE experts since they require a deep knowledge of the underline IE system. This is especially true because the user goal is tuning the level of intrusiveness in the annotation process and very often there is no obvious correspondent in the IES tuning methodology. For example Amilcare allows to modify error thresholds for rules, number of cases covered by rules for acceptance, balance of precision and recall in rule tuning: none of these correspond directly to tuning the level of intrusiveness (even if large part of it relies in the precision/recall balance). The acceptable level of intrusiveness is subjective: some users might like to receive suggestions largely regardless from their correctness, while others do not want to be bothered unless suggestions are absolutely reliable. We think that a user-friendly interaction methodology must be implemented to help in selecting the appropriate level of intrusiveness, without requiring users to cope with the complexity of tuning an adaptive IE system.
In this paper we present an IE-based annotation methodology for the Semantic Web that takes into account the problems of timeliness and intrusiveness mentioned above.
In our model the annotation process is split into two main phases from the system point of view: (1) training and (2) active annotation with revision. In user terms the first corresponds to unassisted annotation, while the latter just requires correction of annotation proposed by the IES. During training users annotate texts without any contribution from the IES. In this phase the IES uses the annotations inserted by the user to train its learner. During this phase the IES is constantly inducing rules. We can define two sub-phases: (a) bootstrapping and (b) training with verification. During bootstrapping the only IES task is to learn from the user annotations. This sub-phase can be of different length according to the specific IES requirements. It depends on the minimum number of examples needed for a minimum training. During the second sub-phases, the user continues with the unassisted annotation, but the behaviour of the IES changes. With some rules already available the IES silently competes with the user in annotating the document. When the annotation process is finished, the IES automatically compares its annotations with those inserted by the user and calculates its accuracy. Missing annotations or mistakes are used to retrain the learners. The training phase ends when the accuracy in annotating can provide the user preferred level of pro-activity and therefore it is possible to move to the next phase: active annotation. We will discuss in the following section how this condition is verified.
In this phase the annotation methodology is heavily based on the suggestions of the IES and the user main task is to correct and integrate the suggested annotations (i.e. remove or add annotations). Human corrections and integrations are inputted back to the IES for retraining. This is the phase where the real system-user cooperation takes place: the system helps the user in annotation; the user feeds back the mistakes to help the system perform better. In user terms this is where the added value of the IES becomes apparent, because it heavily reduces the amount of annotation the user has to insert. This supervision task is much more convenient from both cognition and actions. Correcting annotations is simpler than annotating bare texts, it is less time consuming and it is also likely to be less error prone. 3.
The proposed model of interaction is based on non-intrusive and timely active annotation. The first level of non-intrusiveness is that the IES does not require any specific interface for annotation or any specific adaptation by the user. It integrates in the usual user environment and provides suggestions for possible annotations in a way that is both familiar and intuitive for the user. To some extent users could even ignore that an IES is working for them. The interaction with the user is left to the annotation interface, a tool designed for specific user classes and therefore able to elicit the tuning requirements by using the correct terminology for the specific domain. Even the correct settings and requirements for the appropriate IES’s settings must be elicited through the interface (and then converted in the IES specific settings thorough an API).
Intrusiveness is the risk related to proactivity. As mentioned, there
are a number of ways in which the IES can be intrusive with
respect to the user task. On the one hand when the system
suggests annotation during phase 2 (active annotation with
revision), it can bother users with unreliable annotations. The
requirement here is to enable users to tune the IES behaviour so
that the level of suggestions is appropriate. The annotation
interface must bridge the qualitative vision of users (e.g. a request
to be more/less active or accurate) with the specific IES settings
(e.g. change error thresholds) [
Training requires time and for this reason most of the current systems use a batch mode of training so to limit the time in which the user has to wait while the system trains to specific moments (e.g. coffee time). As explained above, the batch approach presents timeliness problems: users may have to annotate a number of similar texts before the learner is activated and the IES is able to suggest annotations.
An appropriate scheduling of the learning phase can both improve timeliness between user’s annotation and system learning and limits the user idle time to the minimum. If we observe how time is spent in the annotation process (select a document, manually annotate the document, save the annotation), we notice that most of the user time is spent in the manual annotation process. For this reason we believe that this is the right moment to train the IES in the background without the user noticing it. In principle it would be possible to treat every annotation event in the interface as a request to train on a specific example, but this requires the ability to retreat annotations in case of user errors and this makes the interaction with the IES quite complex. In our method the IES works in the background with two parallel and asynchronous processes. On the one hand while the user annotates document n the system learns the annotations inserted in document n-1, i.e. the learner is always one document behind the user. At the same time (i.e. as a separate process) the IES applies the rules induced in the previous learning sessions (i.e. from document 1 to document n-2) in order to extract information (either for suggesting annotations during active annotation or in order to silently test its accuracy during unassisted learning). This means that the annotation capability is always two steps behind. The advantage is that there is no idle time for the user, as the annotation of a document generally requires a great deal more time than training on a single text.
As explained above timeliness is not fully obtained with the above interaction methodology: the IES annotation capability always refers to rules learned by using the entire annotated corpus but the last document. This means that the IES is not able to help when two similar documents are annotated in sequence. From the user point of view such a situation is equivalent to train on batches of two texts, with all the disadvantages of batch training mentioned above (even if a batch of size two is quite small). In this respect the collaboration between the system and the user fails in being effective. Timeliness is a matter of perception from the user side, not an absolute feature, therefore the only important matter – we believe – is that users perceive it. In this respect we start from the consideration that in many applications the order in which documents are annotated is random. Generally users adopt criteria such as date of creation or file name order in directories. In such cases it is possible to organize the annotation order so to avoid the possibility of presenting similar documents in sequence and therefore to hide the lack of timeliness. In order to implement such a feature we need a measure of similarity of texts from the annotation point of view. The IES can be used to work out such a measure. At the end of each learning session all the induced rules are applied to the whole unannotated corpus. As result two main subsets in the corpus are detected: texts were the available rules fire (i.e. annotations can be added: positive subset) and texts were they do not fire at all (uncovered texts: negative subset). Each text in the positive subset can be associated with a score given by the number of annotations that can be added. The score can be used as an approximation of similarity among texts: inserted annotations mean similarity with respect to the part of the corpus annotated so far, no inserted annotation means actual difference. Such information can be used to make the timeliness more effective: a completely uncovered document is always followed by a fairly covered document. In this way a difference between successive documents is very likely and therefore the probability that similar documents are presented in turn within the batch of two (i.e. the blindness window of the system) is very low. Incidentally this strategy also tackles another major problem in annotation, i.e. user boredom. This is the major reason why the level of user productivity and effectiveness falls proportional to time. Presenting users with radically different documents should avoid the boredom that comes from coping with very similar documents in sequence. In the next section a first implementation of the discussed interaction model is presented. We introduce both the IES used (Amilcare) and the annotation interface (Melita). Finally we discuss how the current implementation meets the requirements described.
Amilcare is a tool for adaptive Information Extraction from
text (IE) designed for supporting active annotation of
documents for the Semantic Web. It performs IE by
enriching texts with XML annotations, i.e. the system
marks the extracted information with XML annotations.
The only knowledge required for porting Amilcare to new
applications or domains is the ability of manually
annotating the information to be extracted in a training
corpus. No knowledge of Human Language Technology is
necessary. Adaptation starts with the definition of a tag-set
for annotation possibly organized as an ontology where
tags are associated to concepts and relations. Then users
have to manually annotate a corpus for training the learner.
An annotation interface is to be connected to Amilcare for
annotating texts using XML mark ups. As mentioned
Amilcare has been integrated with a number of annotation
tools so far, including MnM[
When operating in training mode, Amilcare induces rules for
information extraction. The learner is based on (LP)2, a covering
algorithm for supervised learning of IE rules based on Lazy-NLP
[
The learner induces two types of rules: tagging rules and
correction rules. A tagging rule is composed of a left hand side,
containing a pattern of conditions on a connected sequence of
words, and a right hand side that is an action inserting an XML tag
in the texts. Each rule inserts a single XML tag, e.g.
</speaker>. This makes the approach different from many
adaptive IE algorithms, whose rules recognize whole pieces of
information (i.e. they insert both <speaker> and
</speaker>[
When working in extraction mode, Amilcare receives as input a
(collection of) text(s) with the associated scenario (including the
rules induced during the training phase). It preprocesses the text(s)
by using Annie and then it applies its rules and returns the original
text with the added annotations. The Gate annotation schema is
used for annotation [
Melita is an ontology-based demonstrator for text annotation. The goal of Melita is not to produce a further annotation interface, but a demonstrator of how it is possible to actively interact with the IES in order to meet the requirements of timeliness and tunable pro-activity mentioned above. Melita’s main control panel is depicted in figure 3. It is composed of three main areas: 1. The ontology (left) representing the annotations that can be inserted; annotations are associated to concepts and relations. A specific colour is associated to each node in the ontology (e.g. “speaker is depicted in blue). 2. The document to be annotated (centre-right). Selecting the portion of text with the mouse and then clicking on the node in the ontology insert annotations. Inserted annotations are shown by turning the background of the annotated text portion to the colour associated to the node in the hierarchy (e.g. the background of the portion of text representing a speaker becomes blue). 3. The IES suggestion area (bottom) where some of the suggested annotations are presented.
Melita does not differ in appearance from other annotation interfaces such as the Gate annotation tool, or MnM or Ontomat. This is because – as mentioned – it is a demonstrator to show how a typical annotation interface could interact with the IES. The novelty of Melita is the possibility of (1) tuning the IES so to provide the desired level of proactivity and (2) scheduling texts so to provide timeliness in annotation learning. The typical annotation cycle in Melita follows the two-phase cycle based on training and active annotation described in the previous section. Users may not be aware of the difference between the two phases. They just will notice that at some point the annotation system will start suggesting annotations and that they have a way to influence when and with which modalities this will happen. Suggestions can be presented in the suggestion area or in the document area according to a number of criteria. When presented in the suggestion area an explicit selection (on the tick box) is required to the user to accept the suggestion, otherwise the suggestion is not inserted. When presented directly into the document under annotation suggestions are displayed using the same colour code (e.g. blue background for speaker), but they are made recognizable as suggestions because of a special coloured border. The assumption here is that annotations are considered correct unless the user removes them explicitly. The presentation strategy adopted displays unstable tags (i.e. tags not yet fully reliable) in the suggestion area, while tags considered reliable by the system are displayed directly in the document. Note that reliability is independent for each piece of information. For example a system can become quite reliable in a short time in recognizing some information (e.g. seminar start time) requiring more training examples for others (e.g. speaker). In this case there will be a moment in which the suggested annotations for the time will be inserted in the document pane while the annotations for the speaker will go into the suggestion panel.
Users can customize the behaviour of the IES tuning the level of IES proactivity thus changing the level of intrusiveness by using a special slidebar (fig.4). It allows to set two thresholds that divide the accuracy space in three areas: the first level decides which is the minimum accuracy the IES must be able to reach in order to start inserting annotation in the suggestion panel. The second threshold defines the minimum accuracy the system must reach before starting suggesting in the document panel. In the example in figure 4 the system will suggest in the suggestion panel when its accuracy is between 43 and 75% and in the document panel when greater than 75%. When accuracy is less than 43% the IES does not suggest (i.e. it is still in training mode). This general default holds for all the nodes in the ontology, but it can be overridden for specific tags by using the same kind of window. Changing the default for specific tags is useful because users can have different feelings about intrusiveness for different kinds of information depending on the effort required to identify and select that piece of information. It is worth noting that the same slidebar shows the accuracy currently reached by the IES for the specific information: it is the blue filler mark that grows from the bottom (around 10% in figure 4). It is a feedback on the current status of the IES, e.g. if it is in training mode, if it is suggesting in the suggestion panel, etc. Moreover such feedback should support an intuitive changing of the current IES behaviour, e.g. turn off the IES suggestions by lifting up the two arrows beyond the blue maximum level. Note that the same information is presented near each node in the ontology panel: a small square is divided in three parts (corresponding to the three areas above). The small square fills in the same way the slidebar fills. In this way the user has always a feedback on the current status for each piece of relevant information.
We performed a number of experiments for demonstrating how
fast the IES can converge to an active annotation status and to
quantify its contribution to the annotation task, i.e. its ability to
suggest correctly. We selected the CMU seminar announcements
corpus, where 483 emails are manually annotated with speaker,
starting time, ending time and location of seminars. Such corpus
was already used for evaluating a number of adaptive algorithms
[
Tag Prec Rec
In this paper we have presented a modality of interaction between an adaptive IES and a classical annotation interface for the Semantic Web. We have defined a modality in which the interface and the IES cooperate in order to obtain effective annotation in the way preferred by a specific user. We have also explained how to organize learning in order to reduce or avoid any idle time from the user point of view. Then we have discussed how it is possible to maintain a reasonable timeliness in learning from examples while hiding to users the delay necessary for training the underlying IES. Finally we have presented Melita, a demonstrator that implements such methodology and we have described how user configurations in Melita are turned into settings for Amilcare. We believe that this methodology of interaction between the IES and the annotation interface allows to fully exploiting the potentiality of adaptive IE for annotating texts because: 1.
It inserts in the usual user environment without imposing particular requirements on the annotation interface used to train the IES. (2) It maximizes the cooperation between user and IES: users insert annotations in texts as part of their normal work and at the same time they train the IES. The IES in turn simplifies the user work by inserting annotations similar to those inserted by the user in other documents; this collaboration is stime etime location speaker
Amount of Texts needed for training 30 20 30 100
made timely and effective by the fact that the IES is retrained after each document annotation.
The modality in which the IES system suggests new annotations is fully tunable and therefore easily adaptable to the specific user needs/preferences.
It allows to timely train the IES without disrupting the user pace with learning sessions consuming a large amount of CPU time (and therefore either stop or slow down the annotation process).
Future work will consider the better formalization of the way in which Melita’s settings are turned into IES settings. The currently adopted solution is still under evaluation and it needs further development and experiments, as currently it is completely arbitrary and the risk is to produce an opaque effect on the user with respect to the way in which the IES is influenced.
The current work has been carried on in the framework of the AKT project (Advanced Knowledge Technologies, http://www.aktors.org), an Interdisciplinary Research Collaboration (IRC) sponsored by the UK Engineering and Physical Sciences Research Council (grant GR/N15764/01). AKT involves the Universities of Aberdeen, Edinburgh, Sheffield, Southampton and the Open University (www.aktors.org). AKT is a multimillion pound six year research project that started in 2000. Its objectives are to develop technologies to cope with the six main challenges of knowledge management: acquisition, modelling, retrieval/extraction, reuse, publication and maintenance. The work on annotation interfaces described in this work would not have been possible without the discussions and interactions with Enrico Motta, Mattia Lanzoni and John Domingue (Open University), Steffen Staab and Siegfried Handschuh (University of Karlsruhe). Amilcare uses Annie for preprocessing (www.gate.ac.uk). Thanks to the Gate group for providing Annie and for help in integrating it into Amilcare. B i b l i o g r a p h y [14] F. Ciravegna: “Challenges in Information Extraction from Text for Knowledge Management”, IEEE Intelligent Systems and Their Applications, November 2001. [15] S. Soderland: `Learning information extraction rules for semistructured and free text', Machine Learning, (1), 1-44, 1999. [16] A. Douthat, “The message understanding conference scoring software user's manual”, in [17]