Labels representing value judgements are commonly elicited using an interval scale of absolute values. Data collected in such a manner is not always reliable. Psychologists have long recognized a number of biases to which many human raters are prone, and which result in disagreement among raters as to the true gold standard rating of any particular object. We hypothesize that the issues arising from rater bias may be mitigated by treating the data received as an ordered set of preferences rather than a collection of absolute values. We experiment on real-world and articially generated data, nding that treating label ratings as ordinal, rather than interval data results in an increased inter-rater reliability. is nding has the potential to improve the eciency of data collection for applications such as Top-N recommender systems; where we are primarily interested in the ranked order of items, rather than the absolute scores which they have been assigned.
Value-judgements, personal preferences and aitudes — oen
supplied in the form of a numerical rating — are an important source of
data for training machine learning models, ranging from emotion
recognition applications [
Psychologists have long known of a number of bias errors to
which these scales are prone [
Label rankings represent an alternative method of judgement elicitation to absolute ratings. When we elicit ranking data from an oracle, we present a set of objects to be ranked, and the oracle orders this set of objects in terms of how strongly a particular aribute is judged to be present. Values provided in this manner are known as relative measurements, as each value is dependent on the other items present in the ranking set. e data collected from such rankings may only be treated as ordinal data; it allows us to determine a natural ordering of the data, but gives us no information as to how inherent a particular aribute may be on any absolute, common scale.
Inter-rater reliability (IRR) measures the overall level of agreement between raters as to the values they provide for identical data. Although some disagreement among raters is inevitable in situations involving judgements with some degree of subjectivity; we assume that, given enough ratings for a particular item, the average rating will eventually converge on its true, gold standard rating (i.e. the average rating which would be obtained were we to sample the entire propulation). A high IRR for a label-set indicates that there is lile disagreement among raters as to the true gold standard for that item; leaving us with more reliable data. Disagreement between raters stems from genuine dierences in opinions on the one hand, and factors such as noise and rater bias on the other. An increase in IRR is only desirable if the increase is due to reducing the laer.
By proposing a general framework for recasting queries requiring absolutely valued rating labels as a task needing only a series of relative ratings, we hope to improve the reliability and validity of scale-based data collection. is proposition relies on the intuition that corpora of relatively-valued labels will produce higher levels of inter-rater reliability (IRR) than those consisting of absolute values. e intuition, in turn, is based on the assumption that although rater biases aect the numeric label which raters will assign to an item, as bias is constant for each individual, it should not aect the order of preference of items for any given rater.
D
In our current work, we investigate this intuition empirically, hypothesizing that ordinal-valued datasets systematically result in a higher level of inter-rater reliability than their absolute-valued equivalents. Section 2 reviews related research on absolute and relative rating methods motivating the current study. Section 3 describes the methodology behind our experiment to test this hypothesis both on articially generated and real-world datasets. We relay our ndings in Section 4 and discuss the implications for future work in Section 5 2
RELATED WORK
e psychologist, Arthur Blumenthal argues that the human mind
is not capable of providing truly absolute ratings; that when we
are asked to provide labels on absolute scale, we compare each
item to similar items we have seen before. He argues that absolute
judgement involves the ”relation between a single stimulus and some
information held in short term memory about some former
comparison stimuli or about some previously experienced measurement scale
using which the observer rates the single stimulus” [
To take a simple example, consider two raters, rater S, a severe
rater, and rater L, a lenient rater. Both raters are asked to provide
labels for a set of items. Rater S tends to give lower than average
ratings for items in general, whereas rater L tends to give higher
than average ratings. Table 1 shows the interval ratings provided
by both raters; while Table 2 shows these ratings as ordinal (ranked)
data. Although there is very lile agreement between raters on
individual items, there is evident agreement as to the ranking of
items.
for example in multi-criteria decision making [
When gathering absolutely-valued labels, there is no guarantee
that all raters share the same understanding of the absolute,
idealised scale on which the system is based, and this can result in
signicantly dierent behaviours, leading to ultimately unreliable
data. On top of this, it has been shown that results obtained from
rating exercises can be signicantly inuenced by the choice of
scale itself (5-point scale vs 7-point scale, for example) [
is possibility is further evidenced by the experience of
Devillers et al. in collecting emotion-annotation data as part of the
HUMAINE project [
We hypothesize that label sets collected on an interval scale (i.e. absolute numbers, for example, on a scale of 1 - 9) will exhibit less IRR than labels collected on an ordinal scale (i.e., labels provided in terms of ranks within the dataset, from best to worst). Although absolute ratings contain more information than relative ratings (a relative ordering can be deduced from absolute ratings, but not vice versa) —and as such, may naturally increase the IRR of the resulting labels —we believe that some of the improvement is not accounted for by this factor alone. In order to investigate our hypothesis we examine 4 datasets introduced in previously published literature. All of the datasets under consideration were rated using interval labels. We then convert these labels into ordinal data using simple intra-rater ranking; and use Krippendor’s α , which adjusts for the inherent diculty of the problem, to compare the inter-rater reliability of both datasets. is section describes the datasets used in the experiment and discusses the suitability of Krippendor’s α as a comparison metric. 3.1 In order to explore the impact of rater bias on IRR we generated articial datasets of ratings for items provided by raters exhibiting one of the four rater biases, discussed in Section 1; lenient, severe, restricted and extreme. Our rst step is to model a set of items to be rated. ese items could represent, for example, movies, where the goal is to assign a score to each movie representing how good it is; or a joke, where the goal is to rate the joke based on how funny it is. In any case, we are not interested so much in the particular item being rated so much as the gold standard towards which the average rating from a large number of raters would converge. Each item is represented as a randomly selected real number between 1 and 9 representing this gold standard rating. We generated 50 such items to be rated.
We next generate a base rating for each rater. is base rating is determined by adding a modier randomly selected from a normal distribution with μ 0 and σ 1. is value is rounded to the nearest whole number and represents the rating this rater would assign in the absence of rater bias. Each rater’s base rating diers slightly, representing the inherent subjectivity of labels based on valuejudgements.
Next, we generate a bias modier, representing the extent to which a rater’s inherent bias will aect the label provided. e bias modier is a strictly positive value drawn from a normal distribution with μ 0 and σ 1. We ensure the modier is positive by taking the absolute value of the number drawn and disregarding the sign. We split the raters into 4 groups of 20; with each group exhibiting one of the rater biases discussed in 1. e manner in which this bias modier is applied to each rater’s base rating is dependent on the type of bias this rater exhibits. ese are as follows:
Lenient ese raters tend to give ratings above the average for all items. e bias modier is added to the base rating for raters falling into this category Severe Raters falling into this category tend to give lowerthan-average ratings for all items. e bias modier is subtracted form the base rating for raters falling into this category Restricted ese raters favour ratings around the mid-point of the scale. If the base rating is below the mid-point, the bias modier is added to the base rating. If the base rating is above the mid-point, the bias modier is subtracted from the base rating.
Extreme is nal group of raters favours ratings at either end of the scale. If the base rating is below the mid-point of the scale, the bias modier is subtracted from the base rating. If the base rating is above the mid-point, the bias modier is added to the base rating.
Figure 1 shows the impact of each of these rating biases on the labels provided. e histograms in grey show the distribution of base ratings for a particular item before bias has been applied. e coloured histograms show the distribution of labels for each of the 4 groups aer bias has been applied. 3.2
Empirical Datasets
e Jester Dataset, rst introduced by Goldberg et al. [
e BoredomVideos dataset is adapted from the corpus
introduced by Soleymani et al. [
e MovieLens dataset in its original format consists of 10 million ratings provided by 72,000 users across 10,000 dierent movies. We extracted a subset of 5,720 ratings, provided by 20 users across 286 movies. Ratings were provided on a scale of 0:5 - 5, in steps of 0:5. is subset contained no missing values.
e Vera am Mittag German Audio-Visual Emotional Speech
Database (VAM), created by Grimm et al. [
All of the datasets described in Section 3.2 consist of ratings provided on an absolute scale. In order to convert these ratings to rankings we use a simple intra-rater ranking function with average rank used in the case of ties. Given a set of raters R, and a set of items to rate, X, with ri j representing the rating provided for the ith item by the jth rater and Ri representing the set of all ratings provided by rater i, the interval and ordinal values for these ratings are described in Equation 1.
LI nt erval = ri j LOr dinal = rankRi ¹ri j º xi 2 X , ri 2 R (1) 3.4
Inter-rater reliability can be computed using a wide variety of
measurements, the choice of which is oen dependent on the properties
of the data being investigated [
Krippendor’s α can be applied to multiple raters simultaneously, and allows comparisons to be made between values obtained on data using diering scales of measurement. e α value ranges from -1 (perfect disagreement) to 1 (perfect agreement). Having calculated the overall α for each dataset, we decompose the results by calculating the α for each pair of raters individually. e α value obtained for each pair of raters shows the level of agreement for that pair of raters independent of all others. We plot the individual α values on a heatmap, demonstrating that the improved accuracy results from a consistent increase in the pair-wise accuracy across all raters. 4 4.1
We compared the IRR of the simulated datasets using Krippendor’s α , treating the data rst as interval and then as ordinal data. Aer 30 repetitions, the α for ordinal rankings was consistently higher than that of the interval data, and a paired Wilcoxon Signed-Rank test rejected the alternative hypothesis, that the shi in mean α was 0, with a p value of ¡ 0.001. Figure 2 depicts the Krippendor’s α for both interval and ordinal data over 30 iterations as a box plot. is experiment demonstrates that ordinal data is more reliable than interval data, under the assumptions we made when modelling our articial labels. However, it remains to be seen whether this improvement persists when working with real-world datasets. 4.2
Table 3 summarises the α values for each dataset when treated both as interval and as ordinal data. Although the underlying agreement in each of the datasets was low —as can be expected with datasets containing fundamentally subjective labels —we obtained consistently higher IRR by treating the ratings as ordinal data. e MovieLens dataset, in particular, showed a considerable improvement when treated as ordinal data. Although the improvement in the Jester dataset and the BoredomVideos dataset was smaller, the improvement relative to the original inter-rater agreement was quite high. e relative improvement of the VAM dataset was lower than the others. We noted in Section 4.1 that rater bias is a particular feature of subjective ratings. e lower performance of the VAM dataset may be due in part to the fact that identifying emotions in others is less subjective than making personal value judgements and so the inherent rater bias in this dataset is less pronounced than the genuine dierences in opinions between raters.
Figure 3 visualises the IRR improvement of ordinal treatment over interval on the BoredomVideo dataset. Each rater is shown on both the X axis and the Y axis, with each cell representing the Krippendor’s α of IRR between the raters. Krippendor’s α scores above 0, indicating higher agreement than that expected by chance are shown as blue, while Krippendor’s α scores below 0, indicating lower agreement than that expected by chance are represented as red. is visualisation shows that the improvement in IRR was consistent across all pairs of raters reinforcing the ndings in Section 4.1 that ranking data yields signcantly higher IRR than rating data. 5
Our experiments suggest that treating label sets as ordinal, rather than interval in scale tends to suppress the eects of label bias in generating inter-rater disagreement, and consequently leads to more reliable datasets. As this increased reliability results from a reduction of noise (i.e. rater bias), we believe this nding could be utilised to generate reliable labels with fewer oracle queries, reducing the overall cost of data collection. is nding, however, comes with a number of caveats.
Firstly, ordinal data cannot be directly transposed back to an interval scale. is approach will work best when we are more interested in the relative ordering of items rather than their absolute values, for example, Top-N recommendation. If absolute label values are required, a further step, (and further information) will be needed to infer absolute values from our ranked data. We believe that investigating possible approaches to making such an inference may expand the applicability of our proposed method of data collection.
Secondly, our experiments were conducted on real-world data
gathered by researchers using an absolutely-valued measurement
scale. One of the reasons absolute values are more commonly used
in label collection is that it they are easier to elicit from oracles.
Researchers have long been aware of the diculty in geing raters
to rank large collections of items [