Currently the demand for text analytics grows due to the fact that textual data is being created online in large amounts. A number of tools are available for various tasks related to natural language processing such as sentiment analysis and text pre-processing. The majority of these tools are trained using unrepresentative data, which may lead to unintended biases altering the results. Previous research indicates that sentiment analysis tools show gender and race biases, and word embeddings discriminate against women. This research investigates previously unde ned non-native speaker bias in sentiment analysis, i.e. unintended discrimination against English texts written by non-native speakers of English. Non-native speakers of English tend to use cognates, English words that have origin in the speaker's language. To measure the nonnative speaker bias in 4 lexicon-based sentiment analysis systems, a new Cognate Equity Evaluation Corpus was created, based on previous work in the literature for measuring racial and gender biases. The tools gave signi cantly di erent scores to English texts with features of non-native speakers. The bias discovered in lexicon-based tools was mitigated by updating 4 lexicons for English cognates in 3 languages. This paper proposes a generalisable framework for measuring and mitigating non-native speaker bias.
The number of domains depending on data analysis and machine learning has
been growing with a previously unseen pace. As it happens, the fairness of
machine learning models and their application is becoming increasingly more
important. There is, however, no single established de nition of unintended bias [
Natural language processing (NLP) is an area of machine learning which
includes various types of analysis of textual data, e.g. text classi cation, sentiment
analysis, text generation and so on. Just like in machine learning in general,
unintended biases have been noticed in NLP - e.g. word embeddings which are used
for various tasks ranging from Google news sorting to abusive content ltering
have been found to be biased against women [
It has been shown that non-native speakers of English tend to use cognates,
i.e. English words that look similar to the words of their native language in
their English texts more often (e.g. French speakers are more likely to use
`fatigue' (French/Latin origin, cognate of French `fatigue'), and German speakers
are more likely to use `weary' (Proto-Germanic origin, cognate of Frisian
`wuurig') in the same context because of the etymology of those words [
To address the issue of unintended biases the following research question was speci ed and tackled: `To what extent can the expansion of the Equity Evaluation Corpus (EEC) with cognates enable measuring and, where necessary, mitigating bias in lexicon-based sentiment analysis systems against English texts written by non-native speakers (speakers of French, Italian and Spanish)?'
To address the research question several sentiment analysis (VADER, SentimentR, TextBlob, A nn) systems were measured and debiased. The speci ed research question aims at solving the issues of unintended biases in natural language processing, and bringing value to NLP researchers, linguists, and data analysts who deal with textual data. The results of this research will help practitioners reduce a method bias in the application of NLP methods. This will especially be the case in the use of online data, for example from Social Media, online reviews, blogs etc. The sentiment analysis systems speci ed were chosen based on their popularity and functionality. The languages are chosen based on comparatively large number of English cognates they have.
The rest of the paper is structured as follows: section 2 presents relevant related work and its in uence on this work; section 3 describes the methodology applied; section 4 summarises the implementation, evaluation and results of our study; nally, section 5 concludes the paper.
Fairness in machine learning is a growing concern both in industry and academia. The scope of the reviewed literature is from 1996 to 2019. 2.1
As the reliance on machine learning for decision making grows in various spheres
of business and life, the fairness of those decisions becomes increasingly
important. To the best of the authors' knowledge there is no standardised de nition
of the concept of fair machine learning, and depending on the domain
algorithmic fairness can be understood in a number of ways. It is suggested, that
machine learning algorithms or models have to be biased towards some
particular decision, however, when this bias is unintended, a model or algorithm can
be considered unfair [
The source of bias can be di erent on a case by case basis, but generally it
is observed that bias comes from the training data [
Generally, there are 2 types of fairness that provide guidelines for determining
a case-speci c approach - individual fairness and counterfactual one. Individual
fairness comes from the classi cation tasks - it is suggested that two similar
observations or data points should be classi ed into the same category. Originally
it was achieved using a distance metric that would calculate how similar the
observations are [
In this paper, individual fairness is assumed to be the guiding type of fairness
- i.e. it is suggested that similar sentences, one written by native speakers and
another written by non-native speakers should get similar results from NLP
systems. However, unlike in [
When it comes to NLP, word replacement as a method to determine if bias is
prevalent. To measure the bias against identity terms such as `muslim', `gay', and
others in toxicity classi cation, [
A similar approach was used for measuring bias against age in sentiment
analysis. Sentences containing explicit (`old', `young') or implicit (words that
are likely to be used by old or young people) age encodings were used to test
15 sentiment analysis tools. The sentiment was measured for sentences with the
term originally contained in the sentence, and then with the opposite one [
[
The three examples described above are very useful for our approach, as they focus on the in uence of a particular word like an identity term or an age related adjective on the outcome of the sentiment analysis results. The focus of this paper is also word-based - it relies on synsets, or pairs of words, that include a cognate and its non-cognate synonym.
It is found, that generally mitigation of biases in sentiment analysis and related tasks can be performed either through changing an algorithm (in-processing) or through changing the training data (pre-processing).
Changing the training data was observed in several papers related to
sentiment analysis. To mitigate the bias related to age, 2 custom sentiment analysis
tools based on the bag-of-words text classi cation were built with a logistic
regression. Training data used for Sentiment140 4 was modi ed to suit the needs
of the research - i.e. measure bias against age originating from the training
data. Training data for the dataset was ltered to identify records containing
words related to age like `young', `old' and some others resulting in about 14
000 tweets. As a result it was con rmed that the age bias in sentiment analysis
at least partially derives from the labels in the training data, and as long as the
examples including words related to age are excluded, bias can be signi cantly
mitigated [
Similarly to the previous example, mitigation of biases can also be done
through an unsupervised balancing of the training dataset [
It is also important to point out that in some cases mitigation of biases is
not attempted due to the complexity of the task. [
Since the main focus of this paper is on English texts produced by non-native speakers, it is important to understand how di erent this type of English texts can be and how strongly it in uences the outcomes of various NLP tasks.
The term non-standard English can be applied to all the non-standard
dialects and variants of English spoken by non-native speakers, and also to the
non-standard Englishes spoken by native speakers such as Cockney accent or
Scottish accent. Each non-native speaker might use a slightly di erent version
of English depending on their native language and level of pro ciency in
English [
One of the features of non-native Englishes includes using cognates - English
words that have a similarly looking or sounding word in the speaker's native
language. It was observed that even speakers with high level of English pro
ciency tend to prefer cognates over other words if a cognate exists. The presence
of a cognate word in a piece of text can be enough to identify a speaker's native
language. Examples of such words include a pair `weariness' (German origin)
- `fatigue' (French origin) - a German speaker is more likely to use the rst
word, whereas a French speaker is likely to choose the second one because of
the etymology of the words ( [
Non-standard English includes not only di erent vocabulary choices, but also
di erent pronunciation. Various accents of English introduce a great variety of
sounds for the same letter combinations. This can lead to issues in automated
speech recognition (ASR) [
Consideration of these and other speci cs of non-standard English is essential for NLP tasks, due to the fact that English is the primary communication language in many online communities and geographical locations regardless of the native language of the speakers. 3
Both components of the adopted scienti c methodology - measurement and
mitigation of biases - are implemented following a modi ed KDD approach [
For the implementation of this work, 5 data sources were used, 4 of which are
open source, and one created by the authors. To process these data, 4 sentiment
analysis lexicons (VADER, SentimentR, TextBlob, A nn) were used.
Etymological WordNet: Etymological WordNet [
Reddit L2 Dataset: Reddit L2 Dataset [
Equity Evaluation Corpus: Equity Evaluation Corpus is a dataset created
by [
uation Corpus (CEEC) is based on the Equity Evaluation Corpus (EEC) [
To provide a reliable corpus for non-native speaker bias measurement, a sentence template, EEC emotion words and cognate pair words were used.
The template used is `Talking about OBJECT made me feel EMOTION WORD.'. This template is based on the template from EEC `PERSON SUBJECT made me feel EMOTION WORD.' There are several reasons behind not introducing more sentence templates. First is that sentence templates require sufcient time to be veri ed, as there are multiple methodologies behind creating them, and thus left for future work. Second is that the purpose of the corpus is to provide NLP researchers with an e cient way of detecting non-native speaker biases. Assigning sentiment takes di erent amount of time depending on the sentiment tool and the amount of text. Since the number of word pairs is much higher than in EEC, it was decided that one template with 4 emotions and 40 emotion words would be su cient at this stage.
In every sentence the OBJECT was replaced with one of the words from each cognate pair. To get a cognate pair, all the English words with origin in the target language were selected from the Etymological Wordnet. Then each word was assigned a synset (synonym set) using the WordNet package in Python. Every word had a di erent number of synonyms, ordered by how often they are used. Only the most commonly used synset was kept for each word to scope the study and ensure balance in the dataset. Only noun pairs were kept for 2 reasons: 1) every part of speech would require a di erent template, so it was decided to keep only one due to the time limits; 2) nouns were the most represented part of speech. Di erent numbers of pairs for each origin language correspond to the nature of etymology of English words.
The emotion words for the template were retrieved from the EEC. In total 4 emotions were present in EEC - anger, fear, joy, sadness. Each emotion was represented by 10 words, e.g. `angry', `frustrated', `serious', `enraged', `excited', `relieved'.
In total 3 versions of CEEC were created, as can be seen in Table 1. The features required for this work are the \level" of the sentiment analysis and the text that will be used. When it comes to the level, sentence is chosen as a lot of textual data is being created online in microblog environments such as Twitter or Facebook, and the texts of this kind are usually short and do not always include more than 2-3 sentences. Another reason is the eld standard as was observed in the literature, most papers focus on sentences.
Sentiment analysis results may di er depending on capitalisation,
punctuation marks, word choice, contrasting conjunctions (e.g. `but'), negators (e.g.
`not'), degree modi ers (e.g. `very', `slightly', `a lot') [
Languages of word origin chosen for this paper are French, Spanish and Italian. The choice is motivated by a high number of cognates in these languages due to etymological links between English and these languages. The work described in this paper relies on freely available sentiment analysis tools that can be seen in Table 2. To be chosen for the work, the tool had to satisfy the following requirements: it has to be completely free or have a free trial version; it has to be available for English language texts; it has to be available whether in R or Python, the languages commonly used by data scientists and NLP researchers; the sentiment value assigned to sentences has to be numeric and not categorical (e.g. `good', `bad', `neutral'). While the selected tools are not the only ones that satisfy these requirements, they are all used in a variety of settings (e.g. teaching and research) and diverse enough to a ord some degree of variability in the results. To measure bias the CEEC dataset created for this work was used. Sentiment score was assigned to all the sentences using the 4 sentiment analysis tools VADER, A nn, and TextBlob in Python, and SentimentR in R. The sentiment scores from all tools were merged into one dataset, and the di erences were calculated for a more detailed overview.
Assignment of the sentiment was performed using special commands available in the libraries. The standard settings were used for all the tools to achieve the score that is embedded in the lexicons of each tool. 4 4.1
To evaluate the results, a Wilcoxon signed rank test was performed for each emotion. For each origin language and for each emotion the group of sentences containing the cognate words was compared to the group of sentences with no cognate word. Where the di erence was signi cant at p-value less than 0.05, the presence of bias was con rmed. For each sentiment system the results were di erent. The scores were recorded in tables for each of the cognate origin language - French (Table 3), Spanish (Table 4), and Italian. Only the rst two are presented as there was no signi cant bias observed against Italian cognates. SentimentR shows bias against the biggest number of French cognates - 133 out of 292. TextBlob shows bias against the least number of cognates - 19 for polarity and 20 for subjectivity. Essentially this means, that when a French person uses a cognate, e.g. `de cit' which has a French etymology instead of `shortage', the resulting sentence is likely to receive a more extreme sentiment score. It can be seen that the bias against cognates is present in sentences with various emotions in almost all sentiment tools. When it comes to Spanish cognates, SentimentR also shows bias against the biggest number of cognates - 6 out of 32. V=0 indicates that the scores for sentences with cognates and sentences without cognates were identical and as a consequence no bias was observed, as in the case of TextBlob. The scores were checked to ensure this. Measurement of bias against Italian cognates showed no signi cant bias due to the fact that none of the words in the pairs is present in any of the default lexicons.
Mitigating the Bias: To mitigate the bias against texts written by nonnative speakers of English, the steps described below were taken. Due to comparatively smaller sample of sentences representing positive emotions (negative to positive sentences ratio is 3:1) only debiasing tools for negative emotions was added as objective of the research, to ensure fair and generalisable results.
First, the results of the bias measurement were observed for every sentiment system, and the word pairs with di erence in scores were organised into lists. The lists were updated in the following way - cognates that had di erent score in the lexicon were changed to have the same score as the non-cognate word, and the cognates that were not in the lexicons were added with the score equal to the non-cognate word from the pair. This was done by assigning the scores to the words, and then adding the resulting cognate lists.
For Vader there is a special command that was used to update the lexicon by adding the words with their scores.
For A nn no such command was found. To update the lexicon, the TXT le was found in the Python directory, and amended according to the bias. This le then can be used instead of the original lexicon, by either using a special set of commands, or by replacing the original le with the updated lexicon.
TextBlob debiasing is similar to A nn in terms of manually updating the lexicon le in the Python directory. The le has to have the same name as the original lexicon. In the case of di erent lists for di erent languages it can create some inconvenience, however, renaming les is not time consuming and guarantees that the approach will work.
SentimentR allows adding any lexicon to the system. To debias the tool, the original lexicon was amended to re ect the correct scores for the cognates and saved as a new lexicon key. Unlike all the other tools, SentimentR allows to have various lexicons at the same time - i.e. lexicons debiased for French, and Italian cognates can exist as separate les simultaneously. 4.2
To determine the e ectiveness of the debiasing process, the second round of signi cance testing was performed. The results show that after debiasing, the sentences with Spanish cognates and sentences without them receive identical scores in all 4 sentiment analysis systems for not only 3 negative emotions, but also the positive one (joy).
As for the French cognates, it was possible to debias all sentiment tools for all negative emotions - sadness, anger, and fear.
The test results show that SentimentR and TextBlob still show bias against cognates in sentences with positive emotion words related to joy. Debiasing tools for positive emotions is not in the objectives of this research, however, some insights given below were discovered.
After analysing the pairs with di erences in scores it was concluded that in case of SentimentR the bias remained where synonyms consisted of 2 words, e.g. for French cognate `boutique' the paired synonym is `dress shop'. SentimentR is structured in such a way that phrases like this get broken down and receive di erent scores. In the case of TextBlob, the reason behind the remaining bias is that the lexicon has di erent scores for various meanings of the a ected words. To debias further, additional ltering and more data are necessary. It is also identi ed, that due to the fact that emotion words related to negative emotions such as anger, fear, and sadness are more extreme in scores, the bias for them was mitigated without additional ltering or data.
and used to debias sentiment analysis systems for cognates not only in English but other languages as well, as can be seen in Figure 2. It was successfully performed for 4 emotions in 4 tools for Spanish cognates, and for 3 emotions in 4 tools for French cognates. In this paper, we have illustrated that texts with non-native speakers characteristics are likely to be handled quite di erently by sentiment analysis tools. To show this, we expanded the Equity Evaluation Corpus (EEC) to enable the measurement of bias against English texts written by speakers of French, Italian, and Spanish in lexicon-based analysis systems (VADER, SentimentR, TextBlob, A nn) for 4 emotions. We then demonstrated how to mitigate this bias against speakers of Spanish and French in 4 sentiment systems for 3 emotions. No bias was detected against speakers of Italian, so no mitigation was required.
This work can be extended in a number of ways. Increasing the number of languages observed and the number of sentence patterns used would permit a more expansive discussion of non-native speaker bias. It would be useful to also observe the bias against more emotions, especially positive ones. We have studied dictionary-based tools, as such corpus based merit study too, but would require extensions to be methodology developed here as it would involve building machine learning models to facilitate sentiment analysis of texts.
The results of this research contribute to fairness in machine learning literature, ensuring that non-native speakers of English face less discrimination in research designs. The ndings of this work can enable data analysts and researchers improve the way textual data is analysed to avoid unintended bias against English texts produced by non-native speakers at a bigger scale.