Faced with rapidly growing numbers of publicly available natural language documents, it is becoming increasingly difficult to extract the underlying knowledge in a structured manner. Thus, annotation of documents for the purpose of extracting this information is an important task in many research domains today. Creating these annotations is mostly done manually, even though it is a very time consuming work and requires deep understanding and domain knowledge (Hirschman et al., 2012).
TextAI1 is a tool developed to support annotators as a first step on the way to minimize the effort of extracting information from written texts. There exists already a wide range of annotation editors (Neves and Leser, 2012). Thus, we chose not to implement an additional standalone editor, but to build our system around the TextAE2 tool, which is an existing open source editor. We extended TextAE by providing additional features to manage documents of multiple users and predicting entities and relations, which can then be adopted by users into their set of annotations. The system learns from its users and improves prediction of relations over time based on previously annotated documents. We use the biomedical domain as use case for our system and as basis for evaluation. The developed approach, however, can be used across different domains that require manual annotation. With its automated annotation suggestions and its multi-user support, TextAI introduces capabilities that are conceptually different from other annotation tools.
Automatic Annotation Suggestions. Various studies on prediction of annotations confirm that automated recommendations increase the speed and improve the quality of annotations. Lingren et al. (Lingren et al., 2014) determined that automatic annotation suggestions for named entities result in 13.83% to 21.5% time saving without reducing the inter-annotator agreement (IAA) or qualitative annotator performance. Additionally, Fort and Sagot (Fort andSagot, 2010) andSouth et al. (South et al., 2014) showed a significant gain in quality. Hernandez et al. (Hernandez et al., 2014) reported an improvement of non-expert annotator performance using automated namedentity recognition. The WebAnno annotation tool (Yimam et al., 2014) includes automatic suggestions for three generic structures: spans, relations, and chains. It integrates an external machine learning tool, which requires users to configure the features themselves. While this allows for domain specific optimization, it excludes non-expert users from using this functionality. By allowing definition of custom annotation labels, WebAnno increases its flexibility, however eliminates labels as a suitable machine learning feature. Annotation Editors. TextAE is a browser-based annotation tool which comes without a server backend, but supports importing documents and annotations based on a simple JSON format. Being implemented using HTML and JavaScript, TextAEs functionality can be easily expanded. Furthermore, it supports both a wide range of keyboard shortcuts and usability improvements that aim to increase performance of its users. The brat rapid annotation tool (Stenetorp et al., 2012) features a client-server architecture, supports annotation of documents by multiple users simultaneously and allows comparison of different sets of annotations for one document. We chose to adapt several of these features and enhance TextAE to support them, while leaving it a simple and easy to use editor, as feature-rich tools, such as brat, quickly become complicated to use as they require complex configuration.
Annotating documents with entities and relations traditionally involves manually highlighting entities in the document and marking relations between them. Our system employs machine learning techniques to automate these steps by using the workflow depicted in Figure 1. The user interacts with a central front-end component, which is used to manage annotation tasks, documents and users, as well as importing documents from multiple sources. Users also interact with the annotation editor for editing and reviewing documents and their annotations. In the back end, we use an in-memory database (IMDB), which provides document storage, text analysis features and integration with machine learning algorithms. The middleware layer mediates between the different interfaces and the IMDB and deals with all logic concerning user and document management.
The user interacts with a central front-end component, which allows importing documents from multiple sources, such as the local file system, e.g., files in the BioC file format (Comeau et al., 2013) or by querying PubMed3 , based on plain text. After importing documents into TextAI, these can be loaded from the database (cf. Back end). Besides document storage, the IMDB further offers basic text mining and analysis features that we use and expand. Domain independence is achieved through the introduction of another level of hierarchy called "tasks", which can be used to organize documents under one semantic group. This has the technical implication that each task has its own machine learning models, hence annotations within one task have no impact on predictions in another one.
Users can review and edit documents and their annotations through the annotation editor. The later also allows to trigger NER, relationship prediction, and other methods aimed at improving annotation quality and speed. We extended TextAE, which is a powerful standalone annotation tool based on JSON-formatted input, that allows loading text and creating annotations and relations. It allows adding multiple annotations to each position in the document and displaying different layers of annotations through color coding. Instead of displaying different kinds of annotations, such as POS tags and domain-specific information, e.g., medical terms, we use different colors for annotations made by different users on the same document. Because of the HTML span-tag-based implementation for annotation rendering, TextAE displays overlapping annotations in a stacked way instead of inline. Users can create custom labels for their annotations in TextAI. However, we ask the users to map their custom labels to one of the UMLS semantic types4 , in order to improve the ability of our system to learn based on annotations made by users and to normalize the annotations made by different users and tasks. For instance, when annotating the DDI corpus (Herrero-Zazo et al., 2013), users could create four labels, one for each drug type included in this corpus, but they are asked map every type to a UMLS semantic type, for instance, "T2000 -Clinical Drug" or "T121 -Pharmacologic Substance". Currently, we only allow mapping a label to one single UMLS semantic type.
Annotating large corpora is a task that can involve multiple experts, which requires user management to be available in a collaborative annotation system. A problem that occurs when multiple people do the same work is the creation of annotations with similar, but not identical meaning. Therefore, our system nudges users towards better annotations by asking them to map their labels to the UMLS semantic types. Further, during annotation, users can choose to hide their annotations from both other users and the machine learning part of our system, while still being able to use all of its features.
Users can correct the predictions made by the system by selectively adding them to their own annotation sets and editing them if necessary. At any point in the process, users can also manually add, remove, and alter entities and relations to their own set.
We use Rserve5 , which provides an interface to the statistical computing language R with its extensive text mining and machine learning capabilities. Further, our system relies on the text analysis functionalities of an IMDB for entity and relation predictions. Besides the documents, taskspecific metadata, such as trained models, domain specific stopword lists and NER dictionaries are stored in the database, allowing fast access to im-portant domain-dependent information.
The automated suggestion of annotations involves two major steps, NER and relation prediction. For NER we apply a simple dictionary-based approach using the UMLS dictionary and part-of-speech (POS) tagging of the documents.
We train two support vector machines6 to detect relations between two entities: one to determine whether a relation exists between two entities, and in case of positive, one to classify the type of relation. The relations are represented by n-dimensional feature vectors. We get descriptive characteristics of the relation of two entities by combining lexical and syntactical features In the pre-processing step, the document text is split in sentences and every word is tokenized, lemmatized and POS-tagged. Similar to the approach described in (Bui et al., 2014), we divided the sentence into three groups by adding the prefix "b" to each token appearing before the first entity, the prefix "i" to the words in between the two entities and the prefix "a" to the words after the second entity. While we remove stop words and ignore entities in the context, we do not filter out punctuation, i.e. comma, colon and semicolon, since their appearance between two entities can be a strong indicator that there is no relation between these entities. We only consider three tokens on either side of each entity in order to emphasize the near context around the annotations. The annotated types of both entities are considered as a feature as well. We also include the distance, i.e. the number of words and the number of characters between two entities, as a feature. We avoid using trigger words as proposed by (Bui et al., 2014), since this would contradict with our domain-independence objective.
Every newly annotated document is used to retrain the models for relationship extraction, thus learning over time to improve its performance. Further, using the UMLS predefined set of types also improves the learning capabilities of our system, as entities can now be chosen from a finite set and the entity type becomes a stronger feature for relation prediction. Since training these models is only possible and reasonable given a certain amount of information, TextAI needs a set of preannotated documents, either by importing or manually annotating a few documents of the corpus in advance.
As both database and front-end components have highly independent schemata and interfaces, transforming data between them is a key role of the middleware. The middleware implements a RESTful interface representing users, tasks, documents, user-documents and their content to pass on information about data objects to the browserbased front end. Additionally, user management and access rights management are handled here. An integrated user model ensures that every action taken is properly authenticated and authorized. Our annotation editor can load and export data provided in JSON format, thus the middleware is also responsible for transforming information between the different representations in our system. In addition, the middleware provides an interface to import and export documents. Internally, we use the BioC format (Comeau et al., 2013), as it is general enough to be used across different annotation domains.
We focused on the medical domain and used the DDI corpus (Herrero-Zazo et al., 2013) to evaluate the performance of our NER and relation extraction procedures.
NER. In our evaluation, we consider not only exact matches of the gold-standard and the predicted entities, but also overlapping entity label markers are accepted. Our average F-1 score was 77.3% with a recall of 85.62% and a precision of Relation Extraction. In total, the DDI corpus names five different relation types: non-relation, mechanism, effect, advise and a general interaction. We achieved a precision of 72.95% on the test set of the DDI corpus, averaged over 10 iterations. Other researchers who performed relation prediction on the DDI corpus achieve comparable performance results of 60.9% and 62.99% macro averaged F-score (Thomas et al., 2013a).
Training Set Size. When users annotate new relations, the systems performance increases since the SVM models are retrained with new user input. We measure the impact of this input on the efficiency of our classifier with a setup as follows. Starting with only one annotated document, we train both SVM classifiers on the given data and evaluate on 20 randomly selected documents. We then select 10 more documents for training and again test on 20 test documents. This cycle repeats until 500 documents are included in the training set. Figure 3 illustrates the F-score for each of the DDI interaction types averaged over 10 runs with varying training documents.
In comparison to our performance for other relations, the "interaction" type obtained lower results and these have oscillated over our experiments. This relation type is under-represented in the DDI corpus as it constitute only 6% of the annotated DDI relations. As discussed in previous work (Thomas et al., 2013b), this resulted in lower performance of the systems for this relation type on the test set. We believe that the oscillation on the results occurs for those test sets which con-tained more or less instances of the "interaction" relation type that could not be correctly detected by our system.
We have presented a prototype that extends the annotation editor TextAE with multi-user functionality and annotation prediction. This was achieved by creating a concept of per-user annotation sets and tasks, as well as a NER framework and relation prediction algorithm. Our system provides users with functionality for annotations prediction without interfering with their day-to-day annotation work.
As future work, we plan on conducting a user study on annotation speed and quality. Further, we also want to explore NER algorithm based on machine learning and on the labels which are normalized to the UMLS semantic types and not only on the current dictionary-based approach. Finally, semi-supervised learning approaches, such as active learning to leverage user feedback, could improve NER and relation extraction even further.