Introduction

Text simplification is the manual or automatic process of generating a simpler version of a complex text or sentence by preserving its meaning. Simplified texts are easier to understand, for example, for non-native speaker or people with lower literacy. Besides simplicity, meaning preservation and grammaticality are important criteria for a good simplification of a text. Thus, these criteria are also used to evaluate automatic text simplification systems [1]. Therefore, the original text and its generated simplified version are aligned to a simplification pair. This pair can be evaluated manually or automatically [1].

So far, manual evaluation of text simplification is the most reliable evaluation method to judge text simplification [1], as for example the existing automatic evaluation metrics either focus only on lexical changes, e.g., SARI [2], or the meaning preservation, e.g., BLEU [3]. Nevertheless, human evaluation also has its weakness because no best practice for text simplification evaluation exists. Currently, three dimensions are most often used in research, i.e., meaning preservation, simplicity and fluency [1]. Even if there is agreement on these dimensions, there is no agreement on the question and scale used for evaluation (see [4,5,6,7]). Even if Likert scales [8] are often used in text simplification evaluation and other evaluation tasks, many options exist for using and interpreting a Likert scale [9].

In this paper, we analyse the interpretation of different existing scales, including Likert scales, of human evaluation in 6 text simplification datasets. We investigate whether different scale interpretations exist by looking at human ratings of simplification pairs for which the original and the simplified sentences are identical. In detail, we answer the following research questions: I) Do human annotators agree on one label in the judgment of simplicity of identical sentence pairs, e.g., the middle or the lowest score value? II) Do human annotators agree on one label in the judgment of meaning preservation of identical sentence pairs, i.e., the highest score value? III) Do human annotators stick to their interpretation of a rating scale in all of their ratings?

In the following, we will first summarise the state of the art in current manual evaluation of text simplification. Then, we describe our methods and data and build our hypotheses. Afterwards, we present our results, conclude with some final interpretation and discussion of the results and mention possible future works.

Related Work

The human evaluation of natural language processing tasks is very costly and time-consuming, hence, automatic metrics are developed and optimised. For text simplification evaluation also some evaluation directions exist, e.g., evaluation on multiple references [2], evaluation without any reference [10] or evaluation of structural simplifications [11]. But all of these metrics still have some limitations, hence, they should be only used for quickly comparing and assessing different text simplification systems [1].

For a more detailed evaluation, human evaluation is required. In human text simplification evaluation, common evaluation dimensions exist, .i.e., meaning preservation, simplicity, and grammatically [1], but there is no agreement on the questions asked per evaluation dimension or the scale used for evaluation.

For the same dimensions, e.g., fluency (also called grammaticality), several definitions and questions exist: in [11], they ask the raters if the output sentence is grammatical, [12] ask if the simplified sentence is grammatical and fluent, and [13] state that "fluency indicates if the output is syntactically correct. ". Even if the statements sound similar, they emphasise different points and, hence, the raters may focus on different points during the rating. Especially if a rater is not an expert in text simplification, minor differences may lead to incomparable results. There is also a discussion of whether sentence pairs should be rated by experts or crowd workers of the target group [1].

Furthermore, there is no agreement on a rating scale, most approaches prefer Likert scales (see [6,5]) but others prefer continuous scales (see [7]). However, Likert scales are also differently used, e.g., a scale ranging from 1 to 5 (see [5]) or -2 to +2 (see [6]). Following [9], Likert scales can also be differently used regarding other aspects, e.g., single-item vs. multi-item, same distance between consecutive points (ordinal vs. interval), odd or even number of points, each point labeled vs. only end points labeled, descending vs. ascending order, negatively or positively stated items.

In text simplification evaluation, the most common rating scales are 5 point Likert-scales, e.g., [5], a scale from -2 to +2, e.g., [6], and a continuous scale from 0 to 100, e.g., [14,7]. On the one hand, [7] argue that a continuous scale leads to more consistency in inter-annotator agreement in text simplification evaluation as already proofed for machine translation. On the other hand, [6] prefer a Likert scale with negative to positive scale points including a neutral middle point because they are helpful to rate sentence pairs in which the simplified sentence is more or equally complex as the original sentence. However, both scales include a middle point element. Following [9,15], annotators interpret the middle point as, e.g., "undecided", "neutral", or "no opinion", which might be not always the interpretation the scale developers have intended. Overall, the different scales and their interpretations make it difficult to compare the ratings of different system outputs and, therefore, distort text simplification evaluation.

Method

Data

As we want to analyse the interpretation of rating scales by different annotators, a dataset with ratings of at least 2 annotators is required. Therefore, in our analysis, we focus on QATS [16] 1 , HSplit [6] 2 , PWKP test [11] 3 , ASSET [7] 4 , human-likert and system-likert [14] 5 , and Fusion [18,19] 6 . An overview of their relevant evaluation dimensions, scales and number of raters per dataset is given in Table 1.

Additionally, in all datasets, grammaticality is also rated. However, it is rated only absolutely on the simplified sentence and not in relation to the original sentence, so we do not consider it in the analysis. The simplicity rating of QATS is also not considered for the same reason.

Hypotheses

The dataset selection already showed the differences between the human evaluation in text simplification. Even if the name and the idea behind the evaluation dimensions are very similar, the judgements are collected I) on scales with different sizes, i.e., 3, 5 and 100, II) on scales with different point names, i.e., "good", to "bad" or "strongly disagree" to "strongly agree", III) by crowd workers or experts, an IV) on different item types, i.e., questions or statements, V) on different types of simplification pairs, i.e., manually or automatically simplified sentences, VI) on sources which are reused for text simplification, e.g., English Wikipedia and Simple English Wikipedia (in HSplit) or which are directly designed for text simplification (in ASSET, human-likert), VII) on sentence pairs with different aspirations in the simplicity level, e.g., the simplified sentence must be simpler or the simplified sentence can also be more complex. Hence, the following points make it difficult to compare judgements of text simplification systems reported in system papers. In the following, we will analyse if more problems in human evaluation exist. Therefore, we analyse if the annotators consistently understand the scales in each of the datasets.

To analyse the interpretation of the scale, we compare the ratings of simplification pairs in which no change was made from the original to the simplified sentence. These sentence pairs are further called no-change pairs. As complexity assessment is a subjective task, different ratings of the simplifications are expected. But if the simplified sentence is identical to the original sentence, the rating can be expected to be the same because not the absolute simplicity of the sentence is measured but the change/simplification which does not exist in this case. Hence, we use the no-change pairs of the datasets to check whether different interpretations of the rating scales exist. An overview of the proportion of no-change pairs per dataset and their size themselves are given in Table 2.

We will focus on the analysis of the evaluation dimensions of simplicity and meaning preservation. The interpretation of the grammaticality dimensions couldn't be analysed as in all datasets grammaticality was only rated for the simplified sentence but not for the original sentence. In the analysis, we will verify the following hypotheses, which are based on the dataset and scale descriptions in the previous section.

Hypothesis 1: In HSplit and Fusion, the simplicity rating of no-change pairs are equal to the neutral element, i.e., 0. The simplicity ratings in HSplit are judged on a scale ranging form -2 to +2 including the neutral element 0. Following the scale definition in [6], the neutral element of the scale indicates that the simplicity of the original and the simplified sentence of a pair are the same. Hence, we hypothesise that the simplicity ratings of no-change pairs in HSplit and Fusion are equal to 0. A score of -2 indicates a more complex simplified sentence and +2 a more easy simplified sentence compared to the original sentence.

Hypothesis 2: In ASSET, human-likert, and system-likert, the simplicity ratings of no-change pairs are equal to the lowest element of the scale, i.e., 0, as it indicates the worst simplification. In ASSET, human-likert, and system-likert, the annotators rate the relative simplicity of simplification pair based on their level of agreement to a given statement. The scale ranges from 0 (strongly disagree) to 100 (strongly agree). Hence, the lowest value indicates a rejection of the statement, which is interpretable as the worst simplification. In the rating instruction [20], the question raises how to annotate the sentence pair if the original and the simplified sentence are exactly the same. The answer refers to the formulation of the dimension that some change should have been made. However, it does not indicate an expected behaviour of the annotators, e.g., do not judge an identical pair or judge with a specific value. Hence, we can only assume that the lowest score, i.e., 0, indicates that the simplified sentence is more complex than the original sentence as well as that the simplified sentence is as simple/complex as the original sentence. Following this interpretation, a score of 50 would indicate that the simplified sentence is more, roughly 50% more, simple than the original sentence.

Hypothesis 3: The meaning preservation rating is equal to the maximum element in QATS, HSplit, PWKP test, ASSET, human-likert, system-likert, and Fusion. In no-change pairs, the meaning of the original sentence is exactly the same as in the simplified sentence. As meaning preservation measures in all corpora the extent to which the meaning is preserved in the simplified compared to the original sentence, we hypothesise the highest possible value for no-change pairs in the evaluation dimension of meaning preservation. The highest possible value for QATS and PWKP test7 is 3, for HSplit 5, and for ASSET, human-likert and system-likert 100, respectively. In contrast, the lowest possible values would indicate that the simplified sentence has a completely different meaning than the original sentence. Even if the scale has a middle element, this element does not have to indicate a neutral element as for the simplicity scale in HSplit. Following [9], it is also possible to express the indecision of the rater, which is more likely in this case.

Hypothesis 4: If different interpretations of the scales exist, the rater groups' ratings significantly differ for sentence pairs in which the original and the simplified sentences are not identical.

If at least one of the previous hypotheses can be disproved, the rating behaviour of the annotators will be analysed in more detail. The deviation in the scores of the hypothesis lead to the assumptions that the raters differently understood the rating scales. To evaluate the extent of the misunderstanding, we compare the ratings per sentence pair, including sentence pairs with changes, of different rating groups, e.g., preferring the highest and middle value of the scale. For example, if a rater group rated the simplicity of no-change pairs of ASSET with 50 and not with the assumed 0 score, we have a closer look at their simplicity ratings on pairs with a change. If a rater group prefers 50 for no-change pairs, they most likely annotate the pairs with a change differently than the rater group preferring 0. Hence, it is hypothesised that the ratings of such rater groups significantly differ from each other.

Results and Discussion

Each of the selected datasets contains some no-change pairs which are rated by a different number of annotators. An overview of the number of no-change pairs and annotators of nochange pairs per dataset is provided in Table 2. For the ratings of ASSET, human-likert, and system-likert, we normalised the human judgements by their individual mean and standard deviation, following the description in [7,14]. In the following, we will analyse the raters' interpretation of the dimensions simplicity and meaning preservation to disprove or corroborate the hypotheses.

Table 2

Overview of the size of the datasets. Sentences correspond to the number of different original sentences in the datasets, whereas sentence pairs correspond to the number of different simplification pairs, e.g., produced by different systems or humans. No-change pairs are sentences in which the original and the simplified sentence are exactly the same. An annotation record is a rated score on one of the evaluation dimensions of one of the sentence pairs by one rater.

QATS HSplit PWKP test ASSET system-likert human-likert Fusion

Simplicity Rating

In HSplit, the ratings of the experts are consistent and corroborate Hypothesis 1. All ratings of the 346 identical sentence pairs agree on the assumed neutral value of 0, except one annotator for three of overall 7840 annotation records (0.03%).

In Fusion, on average, only 6 of 338 no-change sentence pairs (1.88%) are not scored with the neutral value as assumed in Hypothesis 1. The overall average score of all no-change pairs' simplicity judgements of all three annotators are equal to -0.0026±0.05. Interestingly deviations in both directions exist, i.e., closer to more simple and closer to more difficult.

In ASSET, the annotators do not agree with their ratings for the no-change pair on the dimension of simplicity. For each pair, roughly half of the annotators decides on the minimum value, which is hypothesised, and roughly the other half on the middle value. One annotator per no-change pair rates simplicity with the highest possible score. In contrast to Hypothesis 2, the simplicity ratings in ASSET are not always equal to the lowest element.

Table 3

Overview of the simplicity ratings (normalised with their mean and standard deviation) of all annotators which rated more than one no-change pair of human-likert or system-likert. The first column contains an anonymised version of the worker ids (each worker is assigned to an id following the occurrence of its name in the dataset). The last two columns in both tables highlight if the annotators rate a similar score for the pairs (same) or not (differ).

worker_id Similar to the annotators' behaviour in ASSET, in human-likert and system-likert, the annotators can be split into three rating groups: preferring 0 for no-change pairs, preferring 50 or 100. Again, in contrast to Hypothesis 2, the simplicity ratings in human-likert and system-likert are not always equal to the lowest element. Further analysis is required to check whether these ratings are done by mistake or due to different scale interpretations (see subsection 4.3).

The results of these datasets show that some crowd-workers and experts interpret the simplicity scale as hypothesised. In contrast, the number of points and points ranging from negative to positive or only positive, seem to influence the interpretation of the scale: Both datasets with a scale ranging between -2 and +2 achieved a higher consistency than the scale ranging between 0 and 100. The different interpretations might be due to different understandings of the middle point of the scale [9].

Meaning Preservation Rating

For the dimension of meaning preservation, the human ratings in human-likert, system-likert and ASSET rather meet the values assumed in the hypotheses, i.e., close to 100. For all 5 identical pairs, more than 80% rate a score of the maximum category (80 to 100). But for some of the sentence pairs, one of the annotators rate the meaning preservation also either with a value between 0 and 19 or 40 and 59. Hence, a small proportion also interpreted the scale differently than hypothesised in Hypothesis 3. In comparison to the simplicity rating scale, the meaning preservation scale seems clearer to understand, which might be due to a clearer formulation of the scale item.

The annotators of HSplit again agree all in the same rating, here the maximum value, except for 8 out of 346 identical pairs (2.31%). Furthermore, in QATS all no-change pairs are rated with the highest is "good". In Fusion, 15 of the 338 no-change pairs (4.43%) were rated with a different value than the highest value. The overall average score of all no-change pairs' meaning preservation judgements of all three annotators is equal to 4.98±0.12. Hence, Hypothesis 3 can also be approved for HSplit, QATS and Fusion.

In contrast, in PWKP test, the ratings are below the values hypothesised. Each of the annotators rated the no-change pairs with a score ranging on average from 2.275 to 2.525. 3 of the 5 annotators annotated half of the no-change pairs with the highest value, but another rater only selected it for 30% of the pairs. Hence, for PWKP test Hypothesis 3 is disproved. However, it must be considered that the alignment of PWKP test was reproduced. Hence, the results of PWKP must be interpreted with caution because the found effects might be due to a misalignment.

Consistent Interpretations

Roughly half of the annotators in ASSET, human-likert and system-likert either annotated simplicity with the lowest value and the other half with the middle value. As stated in Hypothesis 4, we analyse whether the annotators stick to their scale interpretation or not.

In system-likert and human-likert, 16 of 34 annotators rated more than one no-change pair on the simplicity dimension. 10 of the 16 annotators are consistent in their ratings (see Table 3), they rated the no-change pairs all either with a score between 0 and 19 or 40 and 59. Looking closer at the ratings, 5 of the 10 raters, decided on a score between 0 and 19 on all of their no-change pairs, as hypothesised in Hypothesis 2, and the other half on a score between 80 and 100. However, also 6 of 16 raters alternate between the lowest, middle or highest value, hence, they seem to have no clear scale interpretation.

In ASSET, 20 crowd workers annotated more than one no-change pair. 13 of them always annotated the same value for all simplicity ratings of their no-change pairs (see Table 4). Similar to system-likert and human-likert, the annotators are split into nearly equally sized groups preferring either the lowest or the highest value for simplicity. Overall, we, can confirm, that different simplicity scale interpretations occur in system-likert, human-likert and ASSET. The different understandings of the lowest value might be due to a not-intended misinterpretation of the middle value [9].

To further investigate the different scale interpretations also on simplification pairs with a change, we divided the raters into groups based on their preferred score on the no-change pairs, i.e., preference-1 and preference-50. The groups are compared sentence-wise on the evaluation dimension of simplicity.

In the averages of the simplicity rating of both groups, the different interpretations are also Comparing all sentence pairs with changes rated by both rater groups using a Mann-Whitney-U-test, the simplicity ratings are significantly differing between both groups in system-likert and human-likert (U=252213.0, p≤0.01) and ASSET (U=64127.0, p≤0.01). Hence, it seems that both groups interpret the simplicity scale differently, but apply their different interpretations to all rated pairs. Hypothesis 4 can be corroborated.

Conclusion and Future Work

Concerning the research questions asked, in the dataset analysed, human annotators (experts and crowd workers) mostly agree on one label, i.e., the highest value of the scale, in the judgements of meaning preservation. In contrast, the analysis has also shown that different scale interpretations exist for the evaluation dimension of simplicity in the dataset with crowdsourced human ratings on a continuous scale. Some raters prefer the lowest value and some the middle value of the scale to indicate the same level of simplicity in no-change pairs. However, the values are not randomly seeded, a clear distinction between raters who annotate the lowest or neutral element on several no-change pairs is possible. This leads to the assumption that they did not rate the lowest or the middle element by mistake but understood the scale differently.

Following the analysis results, on the one hand, the interpretation of the simplicity scale is consistent when rated by experts or using a neutral element for simplicity. On the other hand, crowd-workers had different interpretations of the simplicity scale, i.e., either the lowest or the middle element of the scale indicate no change in simplicity. The scale and the annotations also could e.g., by reformulating the definition or scale ending, the crowd-workers could get more certain by seeing more examples before the annotation, or one could rely only on (trained) experts.

In contrast, the expert ratings in HSplit, PWKP test and the crowd worker ratings in Fusion regarding all evaluation dimensions and the ratings in ASSET regarding meaning preservation are congruent with the values hypothesised. Overall, a deeper analysis of the interpretation of human rating scales in text simplification is required. Therefore, a user study could be conducted in which several sentence pairs with and without changes would be rated on different scales or with different instructions by crowd workers and experts.

Not only the different scale interpretations by human raters but also the different implementations of the scales of human evaluation limit the comparison of human evaluation of text simplification. Hence, best practices as e.g. published for natural language generation [21] are in high demand for text simplification. We hope that this paper increases the awareness of problems in text simplification evaluation and kicks off a discussion regarding these challenges, e.g., training of human annotators or showing examples to them, developing clear and precise statements or questions for evaluation dimensions, best number of points of a scale, e.g., 0 to 100 or -2 to +2, or rating of experts or crowd workers.