VADER is a rule-based sentiment analysis tool for English texts with a social media focus. GerVADER is a German adaptation of VADER, which was developed following the steps of VADER's development process. VADER showed high F1 scores especially for the social media domain, whereas the German adaptation achieved much lower results within the same domain, although on other test data. In this work we examine the question of whether these di erences are language-speci c. Therefore we apply an improved version of GerVADER to German texts and compare the results with the application of VADER to the same texts that are automatically translated into English. The benchmarking showed, that the translation combined with VADER achieves up to 5% higher F1 scores in all test cases, which can be explained by the translation tools automatic xing of awed sentences. However, native language tools can still be viable, since it saves time and costs and does not need another dependency to a third party service.
Sentiment analysis describes the process of automatically rating texts or sentences with a sentiment value. The sentiment value ranges from negative, to neutral to positive and can be expressed as a numeric value or a classi cation in one of the three sentiment categories. Compared to machine learning based approaches classi cation can also be done by rule-based algorithms which have the advantage that they do not require any training data. However, developing a rule-based tool requires linguistic knowledge and is signi cantly more dependent on the target language. Hence one can not simply transfer one language's features, e.g. German, to another language, e.g. English. The languages can differ in their grammar and overall sentence structure, making it very error prone to simply transfer the algorithm to another language. How can negation be detected, or what is the meaning of a speci c punctuation? Are there words that can have di erent meanings in di erent contexts and how can you derive those? Machine learning approaches simply train on lots of annotated data and extract the features with techniques like word embedding, but for a rule-based approach the developer has to design the process of detecting word patterns themselves. While a rule-based tool does not need bootstrap data, it will still need some form of lexicon with individual words or phrases to derive sentiments from. These lexicons are usually created by humans and are often rated by di erent people to o er an average sentiment value.
In this work, which was part of a student project at Bielefeld University of
Applied Sciences, we will rst discuss some improvements to our rule-based
sentiment analysis tool GerVADER [
2.1
VADER is an abbreviation for \Valence Aware Dictionary and sEntiment
Reasoner" and is free to use. It is a rule based tool for analyzing sentiments for
English sentences. VADER managed to outperform other lexicon-based approaches
as well as machine learning models. Especially in the domain of social media
VADER achieved high F1 scores of up to 96%. [
GerVADER is an adaptation of the VADER tool for the German language.
The process of VADER has been replicated in some steps, such as the crowd
rating for the lexicon which is based on the SentiWS lexicon [
SCARE is a corpus consisting of Google Play Store reviews. The reviews are
categorized by their star ratings (1 to 5) and are split into 11 app categories. In
total there are over 800.000 user reviews. [
The SB10k corpus consists of German tweets that are humanly labeled into
the three sentiment categories: positive, neutral, negative. It consists of almost
10.000 tweets. Both corpora will be used for benchmarking purposes. [
For translating our test data, we have mostly relied on Googles translation service. One of the datasets has been additionally translated with MyMemory.
Googles translation service is based on the Google Neural Machine
Translation (GNMT) system. Its hybrid model consists of a Transformer [
MyMemory1 is a large collection of Translation Memories that are collected and provided by humans and organizations. The translations are saved as words or sequences in databases which can then be matched by the users input. As of now there are over 4 billion human contributions. 3
The process section is divided into two subsections. In the rst we analyze the aws of GerVADER and how we improved the algorithm. In the second subsection we will give insight on the benchmarking itself. 3.1
The analysis of the initial version of GerVADER [
Both VADER tools cover their own languages. A question however arises whether it even makes sense to translate a sentiment analysis tool to one's native language. To investigate on this we translated our test corpora with Google and MyMemory.
For MyMemory only the SB10k corpus was used, whereas for Google Translate we tested multiple corpora (SB10k, SCARE). Additionally we constructed a SCARE Balanced corpus, consisting of 400 positive, 400 negative and 400 neutral reviews of each SCARE corpus le. Thus a balanced le of all 3 sentiments is built, consisting of 13.200 entries. 1 MyMemory by translated LABS https://mymemory.translated.net/ 2 GitHub - GerVADER https://github.com/KarstenAMF/GerVADER GerVADER improved in all three areas for our test corpora by adjusting the original algorithm rules as well as adding new features such as fuzzy-matching. Thus the overall classi cation score of German texts has increased.
When comparing VADER with GerVADER Table 1 shows that VADER outperforms GerVADER of up to 5%. Even with the improvements in GerVADER, it is still outmatched. This is due to the fact, that translation tools do not just translate sentences word by word, but also consist of features such as fuzzymatching and entity or POS (part-of-speech) tagging. Those are features, that we have partly integrated into GerVADER as well. Googles API is pre-trained with methods of Deep Learning on millions of data and adjusted to word and phrase sequences and not a simple word-to-word translation. Therefore it goes beyond a simple translation mechanism. As a result, spelling errors are corrected and the overall structure of the sentences is adapted to the desired output language. Thus it is not just a simple translation but also a text correction, which may explain the better results for VADER in the benchmark. 5
While GerVADER has been improved, looking at the translation comparison the question arises whether GerVADER serves any purpose. Developing a native language adopted tool is challenging and has lots of potential for creating new aws. It requires linguistic knowledge in the target language, but this allows one to address language speci c characteristics more appropriately than with a translation. Also, the translation service is an additional dependency, which can be problematic in factors of costs, time and complexity. With the current version it might not be worth it to trade GerVADER for VADER for a maximum of 5% F1 score improvement. However, translation tools handle the linguistic features for the developer and are therefore an interesting research topic for VADER and similar tools that are available in several languages.