NLP often relies on the development of a set of gold standard annotations, or ground truth, for the purpose of training, testing and evaluation. Distant supervision [17] is a helpful solution that has given linked data sets a lot of attention in NLP, however the data can be noisy. Human annotators can help to clean up this noise, however for Clinical NLP domain knowledge is usually believed to be required from annotators, making the process for acquiring ground truth more difficult. The lack of annotated datasets for training and benchmarking is considered one of the big challenges of Clinical NLP [6].
Furthermore, the assumption that the gold standard represents a universal and reliable model for language is flawed [4]. Disagreement between annotators is usually eliminated through overly prescriptive guidelines, resulting in data that is neither general nor reflects language's inherent ambiguity. The process of acquiring ground truth by working exclusively with domain experts is costly and non-scalable.
Crowdsourcing can be a much faster and cheaper procedure than expert annotation, and it allows for collecting enough annotations per task in order to represent the diversity inherent in language. Crowd workers, however, generally lack medical expertise, which might impact the quality and reliability of their work in more knowledge-intensive tasks.
Our approach can overcome the limitations of gathering expert ground truth, by using disagreement analysis on crowd annotations to model the ambiguity inherent in medical text. We have previously shown our approach can improve relation extraction classifier performance over annotated data provided by experts, can effectively identify low-quality workers, and identify issues with the annotation tasks themselves. In this paper we explore the hypothesis that our sentence-level metrics are providing useful information about sentence clarity, and present initial results on the value of different approaches to scoring that the traditional precision, recall, and accuracy.
Crowdsourcing ground truth has shown promising results in a variety of domains. [12] compared the crowd versus experts for the task of part-of-speech tagging. The authors also show that models trained based on crowdsourced annotation can perform just as well as expert-trained models. [14] studied crowdsourcing for relation extraction in the general domain, comparing its efficiency to that of fully automated information extraction approaches. Their results showed the crowd was especially suited to identifying subtle formulations of relations that do not appear frequently enough to be picked up by statistical methods.
Other research for crowdsourcing ground truth includes: entity clustering and disambiguation [15], Twitter entity extraction [11], multilingual entity extraction and paraphrasing [8], and taxonomy creation [9]. However, all of these approaches rely on the assumption that one black-and-white gold standard must exist for every task. Disagreement between annotators is discarded by picking one answer that reflects some consensus, usually through using majority vote. The number of annotators per task is also kept low, between two and five workers, also in the interest of eliminating disagreement. The novelty in our approach is to consider language ambiguity, and consequently inter-annotator disagreement, as an inherent feature of the language. The metrics we employ for determining the quality of crowd answers are specifically tailored to quantify disagreement between annotators, rather than eliminate it.
The role of inter-annotator disagreement when building a gold standard has previously been discussed by [19]. After empirically studying part-of-speech datasets, the authors found that inter-annotator disagreement is consistent across domains, even across languages. Furthermore, most disagreement is indicative of debatable cases in linguistic theory, rather than faulty annotation. We believe these findings manifest even more strongly for NLP tasks involving semantic ambiguity, such as relation extraction. In assessing the Ontology Alignment Evaluation Initiative (OAEI) benchmark, [7] found that disagreement between annotators (both crowd and expert) is an indicator for inherent uncertainty in the domain knowledge, and that current benchmarks in ontology alignment and evaluation are not designed to model this uncertainty.
Human annotation is a process of semantic interpretation. It can be described using the triangle of reference [13], that links together three aspects: sign (input text), interpreter (worker), referent (annotation). Ambiguity for one aspect of the triangle will propagate and affect the others -e.g. an unclear sentence will cause more disagreement between workers. Therefore, in our work, we use metrics to harness disagreement for each of the three aspects of the triangle, measuring the quality of the worker, as well as the ambiguity of the text and the task.
We set up an experiment to train and evaluate a relation extraction model for a sentence-level relation classifier. The classifier takes, as input, sentences and two terms from the sentence, and returns a score reflecting the likelihood that a specific relation, in our case the cause relation between disorders and symptoms, is expressed in the sentence between the terms. Starting from a set of 902 sentences that are likely to contain medical relations, we constructed a workflow for collecting annotations through crowdsourcing. This output was analyzed with our metrics for capturing disagreement, and then used to train a model for relation extraction. In parallel, we also constructed a model based on data from a traditional gold standard using domain experts, that we then compare to the crowd model.
The dataset used in our experiments contains 902 medical sentences extracted from PubMed article abstracts. The MetaMap parser [1] ran over the corpus to identify medical terms from the UMLS vocabulary [5]. Distant supervision [17] was used to select sentences with pairs of terms that are linked in UMLS by one of our chosen seed medical relations. The intuition of distant supervision is that since we know the terms are related, and they are in the same sentence, it is more likely that the sentence expresses a relation between them. The seed relations were restricted to a set of eleven UMLS relations important for clinical decision making [21] (see Tab.1). All of the data that we have used is available online at: http://data.crowdtruth.org/medical-relex. For collecting annotations from medical experts, we employed medical students, in their third year at American universities, that had just taken United States Medical Licensing Examination (USMLE) and were waiting for their results. Each sentence was annotated by exactly one person. The annotation task consisted of deciding whether or not the UMLS seed relation discovered by distant supervision is present in the sentence for the two selected terms.
The crowdsourced annotation is performed in a workflow of three tasks (Fig. 1). The sentences were pre-processed to determine whether the terms found with distant supervision are complete or not; identifying complete medical terms is difficult, and the automated method left a number of terms still incomplete, which was a significant source of error for the crowd in subsequent stages, so the incomplete terms were sent through a crowdsourcing task (FactSpan) in order to get the full word span of the medical terms. Next, the sentences with the corrected term spans were sent to a relation extraction task (RelEx), where the crowd was asked to decide which relation holds between the two extracted terms. We also added four new relations (e.g. associated with), to account for weaker, more general links between the terms (see Tab.1). The workers were able to read the definition of each relation, and could choose any number of relations per sentence. There were options for the cases when the terms were related, but not by those we provided (other), and for no relation between the terms (none). Finally, the results from RelEx were passed to another crowdsourcing task (RelDir) to determine the direction of the relation with regards to the two extracted terms. (FactSpan and RelDir) were added to the basic RelEx task to correct the most common sources of errors from the crowd.
All three crowdsourcing tasks were run on the CrowdFlower platform4 with 10-15 workers per sentence, to allow for a distribution of perspectives. Even with three tasks and 10-15 workers per sentence, compared to a single expert judgment per sentence, the total cost of the crowd amounted to 2/3 of the sum paid for the experts. In our case, cost was not the limiting factor for the experts, but their time and availability.
For each crowdsourcing task in the workflow, the crowd output was processed with our metrics -a set of general-purpose crowdsourcing metrics [3]. These metrics attempt to model the crowdsourcing process based on the triangle of reference [18], with the vertices being the input sentence, the worker, and the target relations. Our theory is that ambiguity and disagreement at any of the vertices (e.g. a sentence with unclear meaning, a poor quality worker, or an unclear relation) will propagate in the system, influencing the other components. For example, a worker who annotates an unclear sentence is more likely to disagree with the other workers, and this can impact that worker's quality. A low quality worker is more likely to disagree with the other workers, and this can impact the apparent quality of the sentence. If one of the target relations is itself ambiguous, it will be difficult to identify and will generate disagreement that may have nothing to do with the quality of sentences or workers. Our metrics account for this by isolating the signals from the workers, sentences, and the target relations, and more accurately evaluating each. In previous work we have validated this premise in several empirical studies [3].
In this paper we focus specifically on sentence quality, to evaluate our claim that low quality sentences are difficult to annotate, and likewise difficult for ma- Sent. sentence 0 0 1 10 1 2 0 0 1 0 0 0 0 0 Sent.1 vector 3 1 7 0 0 0 0 0 3 0 0 0 1 0 Sent.2 sentence -0 0 0.09 0.96 0.09 0.19 0 0 0.09 0 0 0 0 0 Sent.1 relation score 0.36 0.12 0.84 0 0 0 0 0 0.36 0 0 0 0.12 0 Sent.2 crowd model -1 -1 -0.97 0.99 -0.97 -0.94 -1 -1 -0.97 - chines to process. To measure this effect, we begin with a simple representation of the crowd output from the RelEx task:
annotation vector: the annotations of one worker for one sentence. For each worker i their solution to a task on a sentence s is the vector W s,i . If the worker selects a relation, its corresponding component would be marked with '1', and '0' otherwise. For instance, in the case of RelEx, the vector will have fourteen components, one for each relation, none and other. sentence vector: For every sentence s, we sum the annotation vectors for all workers on the given task:
The sentence vector is a simple representation of the annotations on a sentence, and leads to the sentence-relation score, which measures, for each relation, the degree to which a sentence vector diverges from perfect agreement on that relation. It is simply the cosine similarity between the sentence vector and the unit vector for the relation: srs(s, r) = cos(V s , r). The higher the value of this metric, the more clearly the relation is expressed in the sentence. The purpose of the experiments is to provide evidence that the srs is measuring the clarity, or inversely the ambiguity, of a sentence with respect to a particular relation, and that sentences with low scores present difficulty for the crowd, experts, and machines alike.
We use a two-step process to eliminate low-quality worker annotations. We run the sentence metrics and filter out sentences whose quality score is one standard deviation below the mean, then we run our worker metrics [2] on the remaining sentences and filter out all workers below a trained threshold. The purpose of the first step is to ensure the worker quality scores are not adversely impacted by confusing sentences. We remove all low quality worker annotations and re-evaluate the sentence metrics on all sentences.
At the highest level our research goal is to investigate crowdsourcing as a way to gather human annotated data for training and evaluating cognitive systems. In these experiments we were specifically gathering annotated data for a sentencelevel relation extraction classifier [21]. This classifier is trained per individual relation, by feeding it both positive and negative examples. It offers support for both discrete labels, and real values for weighting the confidence of the training data entries, with positive values in (0, 1], and negative values in [−1, 0).
To test our approach, we gathered four annotated data sets and trained classifier models for the cause relation using five-fold cross-validation over the 902 sentences: In order to directly compare the expert to the crowd annotations, it was necessary to annotate precisely the same sentences using each method, and train the classifier on each set. The limitation on batch size came from the availability of our experts, we were only able to use them for 902 sentences. In a batch this small, we found that the sentence-relation score, which ranged between [0,1] and rarely assigned a weight of 1, diluted the positive signal too much in comparison to the expert scores which were simply 0 or 1. We experimented, on a different data set, with rescaling the scores and selected the range that yielded the highest quality score, specified above.
In order for a meaningful comparison between the crowd and expert models, we verified the sentences to provide a ground truth -a discrete positive or negative label on each sentence used in evaluation (for training, only the scores from the respective data set were used). While the main purpose of this work is to move beyond discrete labels for truth, we needed a reference standard to establish that our approach is at least as good as the accepted practice. To produce this reference standard, we first selected the positive/negative threshold for sentence-relation score in the crowd dataset that yielded the highest agreement between the crowd and the experts, and then accepted all 755 sentences where the experts and crowd agreed as true positives. The remaining sentences were manually evaluated and assigned either a positive, negative, or ambiguous value. The ambiguous cases were subsequently removed resulting in 902 sentences. In this way we created reliable, unbiased test scores, to be used in the evaluation of the models. In some sense, removing the ambiguous cases penalizes our approach, which is designed specifically to help deal with them, but again we want to first establish our approach is at least as good as accepted practice.
As reported in [10] and summarized here, we compared each of the four datasets to our vetted reference standard, to determine the quality of the cause relation annotations, as shown in Fig. 2. As expected, the baseline data was the lowest quality, followed closely by the single crowd worker. The expert annotations achieved an F1 score of 0.844. Since the baseline, expert, and single sets are binary decisions, they appear as horizontal lines. For the crowd annotations, we plotted the F1 against different sentence-relation score thresholds for determining positive and negative sentences. Between the thresholds of 0.6 and 0.8, the crowd out-performs the expert, reaching a maximum of 0.907 F1 score at a threshold of 0.7. This difference is significant with p = 0.007, measured with McNemar's test [16].
We next wanted to verify that this improvement in annotation quality has a positive impact on the model that is trained with this data. In a cross-validation experiment, we trained the model with each of the four datasets for identifying the cause relation (discussed in more detail in [10]). The results of the evaluation (Fig. 3) show the best performance for the crowd model when the sentence-relation threshold for deciding between negative/positive equals 0.5. Trained with this data, the classifier model achieves an F1 score of 0.642, compared to the expert-trained model which reaches 0.638. McNemar's test shows statistical significance with p = 0.016. This result demonstrates that the crowd provides training data that is at least as good, if not better than experts.
We believe the discrete notion of truth is obsolete and should be replaced by something more flexible. For the purposes of semantic interpretation tasks for which crowdsourcing is appropriate, we propose our annotation-level metrics as a suitable replacement. In this case, the sentence − relationscore gives a realvalued score that measures the degree to which a particular sentence expresses a particular relation between two terms. We believe the preliminary experiments demonstrate the approach is sound. Our primary results evaluate the sentencerelation score as a measure of the clarity with which a sentence expresses the relation. To this end, we define the following metrics:
sentence weight: For a given positive/negative threshold τ , if srs(s) ≥ τ for sentence s then the sentence weight w s = srs(s), otherwise w s = 1 − srs(s). weighted precision: We collect true and false positives and negatives in the standard way based on the vetted reference standard, such that tp(s) = 1 iff s is a true positive, and 0 otherwise, similarly for f p, tn, f n. Where normally p = tp/(tp + f p), weighted precision p = s w s tp(s) s w s (tp(s) + f p(s))
.
weighted recall: Where normally r = tp/(tp + f n), weighted recall r = s w s tp(s) s w s (tp(s) + f n(s))
.
weighted f-measure: Is the harmonic mean of weighted precision and recall:
If the srs metric is a true measure of clarity, then we would expect it to be more likely for low clarity sentences to be wrong, and less likely for high clarity sentences, and this should be revealed in an overall increase of the weighted scores over the unweighted. In Tab. 3, we show a comparison of five data sets. In the first two columns, the annotation quality of each data set is shown, comparing the F1 to the weighted F1'. The F1' scores are higher in all cases, revealing that human annotators are indeed having trouble correctly annotating these sentences. The baseline scores are the least affected by the weighting, which also fits with our intuition since the baseline does not use human judgment at all. The next six columns in each row show classifier performance when trained by that dataset. The first pair of columns compare the F1 to F1', and for interest the final four columns show the precision and recall. In all cases the classifier F1' is greater than F1, indicating that, as with humans, machines have trouble correctly interpreting sentences with a low srs. The only weighted metric that does not increase is the baseline recall, again this is justified as the baseline does not actually require any interpretation.
In Fig. 4 we show how the classifier performs throughout the possible thresholds, the weighted scores are consistently higher.
We also analyzed the data to understand the overlap between the crowd scores and the experts. In Fig. 5 we compared the frequency of sentences with cause annotations at different sentence-relation scores (measured with kernel density estimation [20]) to the expert annotations of the same sentences. The result shows high agreement between the crowd and the expert -a low sentencerelation score is highly correlated with a negative expert decision, and a high score is highly correlated with a positive expert decision. In Fig. 6 we show the number of sentences in which the crowd agrees with the expert (on both positive and negative decisions), plotted against different positive/negative thresholds for the sentence-relation score of cause. The maximum agreement with the expert set is at the 0.7 threshold, the same as for the annotation quality F1 score (Fig. 2), with 755 sentences in common between crowd and expert. The remaining 147 sentences were manually evaluated to build the test partition.
A widespread use of linked data for information extraction is distant supervision, in which relation tuples from a data source are found in sentences in a text corpus, and those sentences are treated as training data for relation extraction systems. Distant supervision is a cheap way to acquire training data, but that data can be quite noisy, which limits the performance of a system trained with it. Human annotators can be used to clean the data, but in some domains, such as medical NLP, it is widely believed that only medical experts can do this reliably. Current methods for collecting this human annotation attempt to minimize disagreement between annotators, but end up failing to capture the ambiguity inherent in language. We believe this is a vestige of an antiquated notion of truth being a discrete property, and have developed a powerful new method for representing truth.
In this paper we have presented results that show that using a larger number of workers per example -up to 15 -can form a more accurate model of truth at the sentence level, and significantly improve the quality of the annotations. It also benefits systems that use this annotated data, such as machine learning systems, significantly improving their performance with higher quality data. Our primary result is to show that our scoring metric for sentence quality in relation extraction supports our hypothesis that higher quality sentences are easier to classify -for crowd workers, experts, and machines, and our model of truth allows us to more faithfully capture the ambiguity that is inherent in language and human interpretation.
The authors would like to thank Chang Wang for support with using the medical relation extraction classifier, and Anthony Levas for help with collecting the expert annotations.