We use a gold standard under construction for sentiment analysis in Swedish to explore how attitudes towards immigration change across time and media. We track the evolution of attitude starting from the year 2000 for three different Swedish media: the national newspapers Aftonbladet and Svenska Dagbladet, representing different halves of the left-right political spectrum, and the online forum Flashback. - Aftonbladet (http://www.aftonbladet.se/), a daily newspaper with a daily circulation of 154,900 (2014). It describes itself as “independent social-democrat”.1 - Svenska Dagbladet (SvD) (http://www.svd.se/), a daily newspaper with a daily circulation of 143,400 (2013). It describes itself as “independent conservative”.2 - The Flashback Forum (https://www.flashback.org/), a Swedish internet forum, with an Alexa ranking of 9,978, the 42nd in Sweden (2018).3 The paper is organized as follows. In Section 2 we describe the data annotated from the three different sources, in Section 3 we present and discuss the results of our analysis, and in Section 4 we conclude and look ahead.
The topic of immigration has been at political center stage for the last two decades, across
Europe and many Western countries in general and certainly in Sweden. During the last decade,
political parties whose defining issue is reducing or opposing immigration have evolved from
extra-parliamentary movements to having significant representation in their national
parliaments
In this paper, we analyze the sentiment towards immigration in three Swedish media across different times from the year 2000. The media analyzed are the following:
We have obtained annotations of attitudes towards immigration as part of our ongoing effort
to build a gold standard for sentiment analysis in Swedish
Table 1: Immigration-related terms used to retrieve documents from sources
As part of the annotation task, annotators were instructed to annotate each paragraph in each document with an attitudinal value towards the topic of immigration. The sentiment value could be one among 5 possible options: (‘very negative’, ‘negative’, ‘neutral’, ’positive’, ‘very positive’). The following explanation was used in the annotation manual to define the meaning of positive and negative in the context of this topic: “We define a positive sentiment towards the topic of immigration as being in favour of more ‘open borders’ policies, negative sentiment as being in favour of ‘closed borders’ policies, and neutral sentiment as being in favour of keeping the immigration-related policies at the moment of the writing. ‘Very positive’ and ‘Very negative’ describe a stronger sentiment in this regard.”
For this annotation task, the annotators were further instructed to treat titles as paragraphs. The annotators were also instructed to provide a attitudinal value to documents as a whole.
Table 2 shows the number of documents and paragraphs annotated for each source. source documents paragraphs sentences words Aftonbladet 1,184 18,979 42,408 617,442 SvD 852 19,360 44,884 677,385 Flashback 1,834 12,021 20,840 279,399 Total 3,870 50,360 108,132 1,574,226
Table 2: Number of documents and paragraphs in the annotated corpus 3
A total of 40 documents were annotated by all annotators, so inter-annotator agreements could be calculated. Krippendorff’s alpha using an interval metric was 0.34 for documentlevel annotations and 0.44 for paragraph-level annotations.
Figure 1a shows the average sentiment obtained for each source and each time period. This average is obtained using the following quantification of the sentiment values: (‘very negative’: 2, ‘negative’: 1, ‘neutral’:0, ’positive’:+1, ‘very positive’:+2). For the period 2000–2010 the number of data points was lower, so it was divided into two 5-year periods to achieve sufficient statistical significance. (i.e. otherwise, the confidence intervals in Figures 1a and 2a before 2010 would have been so wide that they would have overlapped with each other horizontally and vertically).
Figure 1b shows the distributions for the same sources and periods. Darker shades correspond to more frequent values. The neutral (0) category is omitted because for any time period most of the elements are annotated as neutral (it includes cases where the topic is not mentioned) and its inclusion would require a range for the shades that would not allow us to see the differences among other categories. However, for each time period the neutral value is equal to the difference between 1 and the sum of the other 4 categories.
Figures 2a and 2b show the same statistics calculated over the sentiment associated to paragraphs independently (titles are included as paragraphs). The results are similar, which is an expected result since the emotion towards a topic expressed in a document should be positively correlated with the average emotion in its paragraphs. However, because most documents contain more than one paragraph, the paragraph-level sentiment values are statistically more significant (the confidence intervals are narrower), and we consider them our primary source for extracting conclusions.
It is clear from the data that the sources can be ordered by their general sentiment towards immigration – from more positive to less positive – as Aftonbladet>SvD>Flashback, with Aftonbladet and SvD being relatively close and Flashback taking a more distinctive negative attitude.
However, Aftonbladet and SvD held indistinguishable positions in the period 2005–2010 (There is no data for SvD for the first period 2000-2005). After that period Aftonbladet took a slightly more positive stance. In the year 2013, there is a shift in Flashback towards less negative stance. However, in the year 2014, a shift towards a more negative stance is (a) Average document sentiment towards immigration for our three sources. 95% confidence intervals are shown for each time period.
(b) Distribution of document sentiment towards immigration for our three sources. (a) Average paragraph sentiment towards immigration for our three sources. 95% confidence intervals are shown for each time period.
(b) Distribution of paragraph sentiment towards immigration for our three sources. noticeable in all three sources. By the following year, the stance of both newspapers becomes more positive until it reaches the previous levels, while the negative shift becomes more pronounced in Flashback and stabilizes at 2011–2012 levels. 4
In this paper, we have described some early findings from a preliminary quantitative analysis of a corpus consisting of newspaper and online forum texts about immigration and immigration policies in Sweden, manually annotated for sentiment on multiple levels. In this small study, we have looked only at document- and paragraph-level sentiment annotations, and not at all at the fine-grained phrase- and sentence-level aspect-based sentiment annotation that we also asked our annotators to add to the texts, and which make up the bulk of the annotations.
We are not political scientists, contemporary historians or sociologists, but it seems to us that the results of our study indicate that this kind of sentiment annotation could provide a valuable addition to the methodological toolbox of those disciplines, allowing large-scale analysis of opinion trends over time and opinion differences among media.
Our ultimate purpose in creating this corpus is to provide training and evaluation data –
a gold standard – to be used in the development of automatic aspect-based sentiment
analysis systems for Swedish text
This work has been supported by a framework grant (Towards a knowledge-based culturomics; contract 2012-5738) as well as funding to Swedish CLARIN (Swe-Clarin; contract 2013-2003), both awarded by the Swedish Research Council, and by infrastructure funding granted to Språkbanken by the Swedish Research Council and the University of Gothenburg.