We introduce Sentimentator; a publicly available gamified web-based annotation platform for fine-grained sentiment annotation at the sentence-level. Sentimentator is unique in that it moves beyond binary classification. We use a tendimensional model which allows for the annotation of 51 unique sentiments and emotions. The platform is gamified with a complex scoring system designed to reward users for high quality annotations. Sentimentator introduces several unique features that have previously not been available, or at best very limited, for sentiment annotation. In particular, it provides streamlined multi-dimensional annotation optimized for sentence-level annotation of movie subtitles. Because the platform is publicly available it will benefit anyone and everyone interested in finegrained sentiment analysis and emotion detection, as well as annotation of other datasets.
The main problem with, even conventional, sentiment analysis methods tends to boil
down to a lack of tagged corpora. Proper annotation is costly and can be unfeasible in
some cases [
A crowd-sourced gamified annotation scheme based on Plutchik’s eight emotions
[
The reason we have chosen to gamify the annotation process is the increased
accuracy [
The output of sentiment analysis is often expressed as a numeric value on a sliding
scale of negative, neutral, and positive sentiments or simply a ternary score of one of
the aforementioned values. This approach is limited [
For this to be feasible, a new approach beyond positive and negative is necessary.
We propose to use Plutchik’s eight core emotions [
With the use of an intensity measure, Sentimentator effectively allows for sentiment annotations on the entire wheel. Furthermore, because intensity adjustment and combination of emotions is made possible, the difficulty of the annotation task does not increase linearly with the number of dimensions in our scheme. A further 24 combinations of emotions are possible through combinations of the eight core emotions such that, for example, ’awe’ can be expressed through annotating for ’fear’ and ’surprise’. Therefore 51 unique emotions and sentiments are described by the Sentimentator annotation scheme.
1Source of figure and table: https://en.wikipedia.org/wiki/Contrasting_and_ categorization_of_emotions
Serenity Acceptance Apprehension Distraction
Grief Loathing
Rage Vigilance
Although this dataset is only being developed at the moment, when it has been
completed and tested, it will be made publicly available. Once we have some
annotated data, it can be used as training and testing data. Previous work suggests that
our approach, if implemented correctly, should be on par with or better than some of
the best methods available at the moment [
There are a number of areas where sentiment analysis could become an invaluable
tool for digital humanities scholars. Some examples of these areas are history,
literature, translation studies, language, and social sciences. Possible approaches for
historians and social scientists could be to study how the attitude towards a specific topic has
changed through time [
In section 2 we present an overview of relevant related work and current approaches. In section 3 we discuss gamification from a theoretical perspective and in section 3.1 we shine the light on our framework and platform. In this section we also consider the practical applications of the ideas discussed in section 3 in greater detail. The last two sections are reserved for future work and a concluding discussion. 2
There are many approaches to mining data for sentiments. They range from purely
lexical to fully unsupervised [
There are a few applications that offer similar solutions to ours on some level (see
for example [
Most current approaches still focus on the positive-negative axle of polarity. This
binary, or at best ternary with ’neutral’, approach is far too restricted for many
applications [
Although sentence or phrase level sentiment analysis is important for many
applications [
There has long been a discussion on how classifiers trained on data from one
domain might not work as well when applied on data from a different domain [
Gamification happens when game elements are used in a non-game context to ”improve
user experience and engagement” [
Sentimentator players (annotators) select emotions and sentiments for random sentences. Other common gamification elements included in Sentimentator are badges, leaderboards, levels/rank, as well as avatars, and feedback. Variation in the gaming content is key to minimize the repetitiveness of tasks. We offer annotators simple annotation tasks, correction of automatically annotated data tasks, and ranking of sentence tasks.
Groh discusses some pitfalls of gamification stating that ”pleasure is not additive
and rewards can backfire” [
A simple way to increase the relatedness of our platform is to allow players to see
their own and their peers’ progress as well as in real-time see how their work impacts
their grade (if annotation is part of coursework) or some other real-world benefit. This
can be done partly by leaderboards, but also by showing the student a progress bar
that shows how close they are to the next goal/rank/level. As with Zooniverse3 [
For Sentimentator to allow annotators to feel competent and that they are improving they need feedback on their progress in relation to others. It is not desirable for the annotators to see how other annotators have annotated the same data, but annotations can still be compared and scored. When comparing annotations with those made by other annotators, the reliability/accuracy score is dependent on the reliability rating of the other annotator. If annotations correlate better with annotators with a higher reliability rating then the score given is also higher and vice versa. This means that the rank of a player also affects how other players score. Additionally, a score is affected by how well the annotation correlates with validated test sentences. See sections 3.1 and 3.2 for more details about gameplay and scoring.
The validated sentences are sentences that have been annotated by expert annotators who have received thorough instructions on how to annotate with the aim of consistency across annotators. The results of these expert annotators will be reviewed before they are used as seed sentences. The ”gamer” annotators will receive a similar tutorial via Sentimentator, but their annotations will generally only be compared against the validated seed sentences and the annotations of their peers.
The first players of Sentimentator are students of language technology. It is difficult to not offer these students extrinsic rewards (in the form of extra credit and such), especially in the initial stages of gathering testing and training data. Some of this loss of autonomy is combated by emphasizing the scientific contribution that they make and keeping them posted about e.g. articles published using datasets they helped create. Once the platform is open to all, however, there is significant autonomy. 3.1
The annotators are greeted by an info screen where they are presented with Plutchik’s
[
The first type of gameplay consists of annotating unvalidated pre-annotated sentences. The sentences have been tagged by using simple lexical comparison. The 3https://www.zooniverse.org/ 4https://www.planethunters.org/ annotator/player needs to judge whether the analysis is correct or needs adjustment. The scoring is a simple fraction of full scores until the annotations can be compared to peer annotations.
Both validated and unvalidated sentences are presented to the annotator who does not know which type is in question. The annotator/player needs to recognize the emotions and sentiments present in the sentence without context. The scoring is a simple fraction of full scores until the annotations can be compared to peer annotations. In the case of validated sentences, the scores received follow the formula in 3.2 and significantly impact rank.
All of the gametypes have intensity of sentiments/emotions built into the annotation
through the use of a slider that is pre-set to 50%. The slider can be adjusted higher or
lower to signify the intensity the annotator judges the sentence to possess. For the
ranking task sentences, whether the intensity has been adjusted or not, are shown and
by dragging and dropping the sentences in order from most intense to least intense, we
are able to get more accurate intensity scores through this best-worst scaling approach
[
All annotations are done without giving any context as suggested by the results
achieved by Boland et al. [
As discussed in section 3 about Gamification, it is important for players to feel competent and like they are mastering a skill. Therefore scoring is one of the most important aspects of gamification. Players need to feel that they are being compensated appropriately for the work they are doing even if it is a game and the compensation is in the form of points.
Players accumulate both rank (R) and level where rank is a prestige or reliability score based on how well the player’s annotations correlate with validated test sentences. where Tsv stands for total score from validated sentences and Vmax for maximum possible score for the player in question from validated sentences
Level is a straight-forward measure of the number of annotated sentences.
R =
Tsv
Vmax Level =
Tsv Vmax
Ap Ta 100 where Ap stands for total sentences annotated by player and Ta stands for the total number of sentences in the dataset and 0 R 1, and 0 Level 100 There are two main types of scores; those based on rank (i.e. prestige or reliability) and those based on validated sentences. All tasks yield a pre-adjustment score (S). This score is the score that is based on simply doing the task without any regard to how well the task has been completed or how it correlates to other players annotations.
The calculations for the score received from annotating validated sentences (Sv) is fairly straight-forward.
Sv =
S Vs where Vs stands for the max score possible for that task as per the score for the validated sentence
As for the score based on peer annotation (Sp), this score accumulates rank only after a certain number of annotations have been made for the same sentence. The rank (or reliability/prestige) rating of the annotator (Roa) who has annotated the same sentence before influences the score for the annotation as per the following: Ps
Soa Sp =
Roa where Ps stands for the pre-adjustment annotation score of the peer and Soa stands for the score of the other annotator. In practice this will work much like a weighted average across all peers. The number of annotators per sentence is also limited. The rank influences the score as it is at the time of the annotation, i.e. the rank that was valid at the time of the annotation is considered. If an annotators rank improves or declines, it is a reflection of their annotation skill in real-time, not when they did the original annotation. Therefore dynamic scoring would not accurately reflect the reliability of an annotation. 3.3
There is a lot to consider when choosing the right annotation. It is virtually impossible
for all annotators to annotate every sentence exactly the same. This results in noisy
annotations. Hsueh et al. [
As our scoring already relies on validated sentences even when annotating
unvalidated sentences, we are unlikely to need much screening for noisy annotations. It is,
however, important to keep the possibility of excluding noisy annotators from the final
annotation output. It is also important to be able to exclude ambiguous examples from
the annotations in order to maximize the quality of the labels [
All annotations will have been annotated by at least three annotators before they are made final. Naturally, these tags will not always be identical. The way Sentimentator is constructed allows for easy checking of differing annotations. The first step is the automatic comparison against validated sentences. The second is to defer to the annotation made by the highest ranked annotator. However, where discrepancies are deemed considerable, annotations can be flagged to be reviewed by experts. 3.4
We use the publicly available dataset OPUS.6 Our initial focus is the English and
Finnish parallel corpus of movie subtitles, but the number of possible languages to
annotate is only limited by the data itself. The current version has been tested on eight
languages. We chose movie subtitles [
6http://opus.lingfil.uu.se - We use the newest, 2018, version which has at the time of writing not yet been made publicly available.
The evaluation of this framework can only begin once a certain amount of lines have been annotated and cross-checked. For a demonstration, some results can be achieved with approximately 1000 lines annotated, but for proper sentiment analysis at least four times that is required. This means that at least three people will need to annotate 4000 lines, preferably many more people annotating tens of thousands of lines/sentences.
One simple way of spreading out this task, and to be able to utilize expert annotators for a low cost, is to outsource it as extra-credit coursework in computational linguistics, or corpus linguistics courses and similar. Once enough data has been annotated for training and testing data, we can evaluate our framework and compare it against the current gold-standard.
We plan on evaluating the final dataset by taking into account both the distribution of the data and classification performance using a set of different classifier types. We intend to evaluate the distributional balance of the data in regard to the amount and quality of lines/sentences of each label or label combination. This way we reveal patterns in the dataset which may affect classification results. For example, sentences of a given label may be considerably longer or shorter than sentences of another label, or contain rare words. Similarly, the sentences may originate from a movie of a specific genre or time period and thus contain a particular type of language use, such as jargon or archaic words. This allows us to evaluate the sparsity of the data in both the dataset as a whole as well as across different labels. We can then assess whether some parts of the dataset are more sparse and thus less likely to allow classifiers to detect meaningful patterns.
Using a set of different classifiers also allows us to evaluate the quality of the dataset. By building confusion matrices for each classifier, we can observe the classification accuracy, precision, recall, and F-measure for each class in the dataset as well as the overall performance of the classifier.
Other future work includes testing the finalized semi-supervised algorithm on actual datasets. In addition to the suggestions in the Introduction, some possible explorations could be newspaper or online discussion forum data dumps with the search keys for migration and other current issues.
A comprehensive set of high-quality annotations also allows for comparison between intra-lingual annotations of the same sentences by different users as well as identifying possible patterns in cross-lingual annotations of parallel sentences. Another interesting question to investigate is whether showing users sentences which have already been annotated influences their choices when choosing the most suitable tags for those sentences. In this research setting, users would choose the gameplay option where they evaluate annotated sentences with the task of either accepting or editing those annotations. This data would then be compared to parallel annotations of sentences which users have annotated from scratch.
We also hope that other researchers in various fields including computational linguistics as well as humanities etc. will find both the annotation platform and the dataset useful and publish their own research based on our work.
We have introduced Sentimentator, a publicly available, gamified, web-based annotation tool specifically for fine-grained sentiment analysis. Not only do we go beyond binary sentiment classification, but our annotation scheme allows us even more detailed fine-grained annotation by adjusting the intensity of Plutchik’s eight core emotions. The expansion gives us a possible eight core emotions with three intensities each, and 24 combinations of the core emotions with a total of 48 separate emotions and an additional two sentiments plus neutral, i.e. 51 total sentiments and emotions available for annotation (See figure 1 and table 1 for specifics).
The gamification of annotation decreases the cost of annotation and increases the
size of the final dataset. It has also been shown to give more accurate annotations than
traditional crowd-sourcing methods [
This type of data is rare to come by, and we have high hopes for the applications of the dataset and the platform itself.
Why games will lead the way,