Human assessments by either experts or crowdworkers are used extensively for the evaluation of systems employed on a variety of text generative tasks. In this paper, we focus on the human evaluation of textual summaries from knowledge base triple-facts. More speci cally, we investigate possible similarities between the evaluation that is performed by experts and crowdworkers. We generate a set of summaries from DBpedia triples using a state-of-the-art neural network architecture. These summaries are evaluated against a set of criteria by both experts and crowdworkers. Our results highlight signi cant di erences between the scores that are provided by the two groups.
In the last decade, crowdsourcing has gained increased interest since it o er the
methods to reach large amounts of online contributors capable of performing
in a small time large amounts of short human intelligence tasks. In particular,
it has served the evaluation purposes in di erent areas of computer science,
such as information retrieval [
Human judgements are used for the evaluation of many systems employed
on a variety of text generative tasks ranging from Machine Translation [
yThe authors contributed equally to this work.
In this paper, we focus on the human evaluation of textual summaries from
knowledge base triple-facts [
Our experiments have showed that there are signi cant di erences between the scores that are provided by experts and the crowdworkers. Our future work will focus on the methods with which the crowdworkers should be trained in order to perform more accurately on similar tasks. 2
We run a crowdsourcing task according to which we evaluate 20 summaries that have been generated with the Triples2GRU system that has been proposed by Vougiouklis et al.. We regard each summary as a a concise representation in natural language of an input set of triple-facts. Each summary is generated by Triples2GRU given a set of 8 to 18 triples1, and is evaluated by 10 workers.
Before starting the task, the workers are presented with general instructions. They are also informed with respect to the ethics approval that we have received for the carrying out of this experiment. The task consists of three phases through which workers were required to evaluate a given summary: (i) text uency (with an integer number between 1 and 6), (ii) information coverage, by classifying as \Present" or \Absent" each triple-fact from a given list, and (iii) contradictions, by classifying each one of the aforementioned facts as \Direct Contraction" or \Not a Contradiction". At the beginning of each phase, the workers are presented with de nitions, suggestions, examples and counter-examples. Each worker was rewarded with 0:20$. After the carrying out of the experiment, the same 20 summaries are also evaluated under the same setup by two experts. 3
Fluency. For each summary, (i) we computed the average of the uency scores that have been assigned by the 10 workers. Then, (ii) we computed the average of all the values obtained in (i) resulting in an average of 4:8 out of 6. The average uency with which the experts evaluated the 20 summaries was 5:28. The ANOVA test computed on the two uency score series produced p < 0:05. Consequently, we can claim that compared to the experts, crowdworkers tend to systematically underestimate the summaries' uency by 0:5 out of 6.
1The pre-trained version of Triples2GRU that we used (i.e. https://github.com/pvougiou/Neural-Wikipedian) accepts up to 22 triples as input.
Coverage. Workers evaluated the coverage of each summary with respect to a set of triple-facts that generated it. Each summary is aligned with 8 18 facts. The assessments were made by choosing between two labels: (i) \Present" for facts that are either implicitly or explicitly mentioned in the summary, and (ii) \Absent" for the rest. We compute the percentage of the \Present" facts for each summary. Then, similarly to uency, for each summary, we rst compute the average of coverage across the workers, and then the average across all the summaries. The average coverage for all the 20 summaries was 26:85%. In our second experiment, two experts repeated together the same evaluation resulting in an average of 39:71% of facts covered by the summaries. As a result, workers tend to undercount the presence of facts in the generated summaries (con rmed by ANOVA test p < 0:05). Finally, a positive signi cant correlation (Pearson = 0:64) pointed out that workers evaluate coverage in a consistent manner with the experts.
Contradictions. Workers were required to evaluate possible contradictions between the information in a given summary and the respective facts that generated it. Workers were required to mark as \Direct Contradiction" facts that contradict the summary, and as \Not a Contradiction" the rest. For each summary, we compute the percentage of facts that are labelled as contradictions by each single workers. Similarly to coverage, (i) for each summary, we computed the average of the percentages of contradictions of all the workers, and (ii) we averaged the contradiction scores across all the summaries. In a preliminary version of our experiments, each fact was to be marked as either \Contradiction" or \Not Contradiction". However, this proved inadequate since workers were marking facts that were not covered in the summary as contradicting, resulting in an average of 50% of facts whose information is contradicted in the summaries. In order to minimize the e ect of contradictions, besides changing the available labels for each triple-fact, in the contradiction instructions (shown before the third phase of the task), we explicitly noted that contradictions should be rare and that we expected many summaries without any of them. As shown in Fig. 1, we advise workers to identify as contradictions only \Direct contradictions" whose information is explicitly negated in the corresponding summary. Our nal result of 30% represents the average of contradicting facts per summary. The same evaluation was made by the two expert, and the average percentage of triple-facts that are contradicted in the summaries was 0:7%. Consequently, workers tend (ANOVA test, p < 0:05) to signi cantly overestimate the presence of facts that are contradicted in the generated summaries. 4
In this paper, we presented preliminary results of a work aimed to explore the use of crowdsourcing for the evaluation of NLG systems. In particular, we focused on the evaluation of textual summaries that are generated from triple-facts. We compared the results of two studies, one that has been performed by experts and one by crowdworkers. The evaluations were conducted in three phases: (i) the uency of the summary, (ii) the coverage, and (iii) the contradictions of a summary; the latter two are assessed with respect to the given triple-facts. Our preliminary analysis shows that crowdworkers tend to underestimate the uency of the summaries by 0:5 out of 6. While coverage is judged consistently across both experts and crowdworkers, it is signi cantly underestimated by the latter. Lastly, despite the fact that we emphasised on the low number of expected contradicting facts, workers strongly overestimated their presence.
A natural extension of this work is to identify the type of facts (i.e. predicates) that in uence negatively the workers' judgement. Further studies will focus on minimising this bias by both training workers on how to identify only direct contradictions, and increasing the quality control of the experiment.
This research is partially supported by the Answering Questions using Web Data (WDAqua) and QROWD projects, both of which are part of the Horizon 2020 programme under respective grant agreement Nos 642795 and 723088.