This paper presents the system we submitted to the Swiss German language detection shared task, part of the GermEval 2020 Campaign, held at the SwissText & KONVENS 2020 conference. The goal of the task is to identify if a given text snippet is written in Swiss German. Our approach includes a reformulation of a binary to a multi-way classification problem, a character filter, a neural RNNbased classifier, and the addition of synthetic noise to the training set. The official evaluation of our submitted system results in an F1 score of 96.8%, achieving the second place in this shared task.
In this paper we describe our approach and results for the Swiss German language detection shared task (GSWID 2020) at the SwissText & KONVENS conference. The objective of the shared task is to construct a system to automatically identify Swiss German (GSW) text snippets.
Generally, language identification has been
viewed as a solved problem “suitable for
undergraduate instruction”, as McNamee (2005)
deprecatingly remarks in the title of his paper. However,
it is not clear if this view holds true for text
snippets that are (1) short, (2) noisy, (3) from
multiple domains, (4) written in a scarce resource
language, or (5) which consist of non-standardized
dialects
Previous approaches based on classical machine
learning typically utilize character level features
like single characters, character combinations
(ngrams) and capitalization together with models
such as naive bayes classifiers, support vector
machines, and decision trees
In this paper we describe our system, comprising: (1) a reformulation of a binary to a multi-way classification problem, (2) a BiGRU-based neural architecture, (3) a character-based filter, and (4) a noisifier module. 2
Provided Data The shared task organizers provide a list of approximately 2,000 GSW tweets to be used as positive training examples.1 The use of further training material is explicitly allowed and encouraged. In the following paragraphs we give a review of additionally collected data.
Swiss German We collect GSW data from the
following sources: the NOAH corpus
Note that the following set memberships hold: B C D and A \ B = ;.
We only collect languages from B, with special focus on C and D, since text snippets written in a language from A can be filtered out in a rule-based manner.
We collect data for all languages from the
aforementioned corpus collection of the University of
Leipzig. For Standard German, we additionally
gather texts from the Hamburg Dependency
TreeBank
2We define the GSW character set as the set of characters found on a GSW keyboard. This differs slightly from e.g. a Standard German Keyboard, which lacks characters like “e`”, “a`” and “e´”.
Language Swiss German (GSW) Standard German English Italian Dutch Swedish Luxembourgish Norwegian French Low German West Frisian Portuguese Romanian Tagalog Bavarian Lombard Yiddish Croatian Northern Frisian Other Total
Noise Through manual inspection of the tweets that were provided as training data we observed that they are significantly noisier than the rest of our training data.
We identify two kinds of noise in this data: token-level noise and character level noise. Both can be produced on purpose or by accident. Token-level noise consists of words, phrases or citations in other languages, mainly English or Standard German, in otherwise GSW tweets. Character-level noise consists of omissions, insertions or repetitions of single characters. Examples can be found in Appendix B. 3
Task Formalization We formalize the task as follows: Assign a label y 2 f0; 1g to an input sequence of characters x = fx0; x1; x2; :::; xng, where 0 corresponds to the class swiss german and 1 corresponds to the class not-swiss german.
However, the not-swiss german class is a very broad category since it not only contains all other languages, some of which are similar to GSW, but also all possible string sequences that do not appear in GSW. Thus, we hypothesize that more finegrained labels will lead to more homogeneous and better separable classes.
Following this line of reasoning, we define three different granularity levels: binary, ternary and fine-grained. The binary setting corresponds to the task formalization described above. In the ternary setting we split the class not-swiss german into the classes standard german and other. And in the fine-grained setting, each language present in table 1 corresponds to one class, with an additional class other. For our collected data set, this leads to a total of 23 classes.
Pipeline We construct a pipeline where an incoming text snippet is first cleaned of hashtags, mentions and URLs. Then a rule-based character filter decides if the text snippet is a member of A and if so, immediately classifies the text snippet as not-swiss german. If the text snippet is not part of A, it might be an instance of swiss german and hence is clipped to a prespecified length, which we treat as a hyper parameter, and fed into a neural classifier.
During training time, we make two modifications to the pipeline: (1) The rule-based character filter is left out because our data only consists of text snippets from languages in B. (2) We make use of an additional noisifier, which adds noise specifically modeled after the noise that is actually encountered in GSW text snippets on the web. In the rest of this section we describe the main parts of the pipeline in detail.
of characters x, the character-based filter computes the relative frequency of characters in x that do not appear in the GSW character set. If this frequency surpasses a given threshold, x is labeled as not-swiss german.
Neural Model Our neural model comprises
character embeddings, a BiGRU
We ignore all BiGRU outputs and only use the last hidden states h!n and h0.3
Each hidden state is fed into a block of two dense layers with dropout before both layers and the rectified unit linear function in between. The outputs of the two dense blocks z1 and z2 are concatenated and fed into a final dense layer with the number of classes as the output dimension. We apply a log-softmax function to the output to turn the neural activations into a probability distribution over the target classes. Note that the number of target classes depends on the chosen level of granularity.
For optimization we use the negative log
likelihood loss combined with the Adam optimizer
Noisifier Based on the assumption that the test data has a similar amount of noise as the tweets provided for training, we introduce a noisifier with the goal of injecting this type of noise into the entire training data, which contains large amounts of text snippets from “clean” resources like news texts. We refer to this difference in noise between corpora as noisiness gap. Recall that we observed token-level and character-level noise in the training data in section 2. In what follows, we will address both types of noise separately.
For the token level noise we created a handcrafted list L consisting of English and Standard German words often found in GSW tweets, comments and messages. Additionally, we add mentions of Swiss locations to L.4
3In earlier experiments we also used the BiGRU outputs by concatenating them with the last hidden states and then fed this entire feature vector into dense layers. However, we found that using these outputs decreased performance.
4We try to avoid that the model learns to associate Swiss location names with GSW text which presumably would lead to false positives.
The token-level noisifier receives as input a clean training example x consisting of k tokens and the two thresholds p1 2 [0; 1] and p2 2 [0; 1) with p1 > p2. For each token in x, a noise token l 2 L is inserted with a probability of 1 p1. We hypothesize that the presence of one noise token increases the probability of additional noise tokens. To model this, we use a higher second probability 1 p2 for repeatedly adding an additional noise token. We define an upper bound of k=2 for the number of inserted noise tokens c under the assumption that a text snippet with c k=2 does not resemble the original language of x anymore. See algorithm 1 for more details.
The algorithm inserting character level noise receives as input a token-level noisified training example xT noise and analogous to token-level noise injection, the two thresholds p3 2 [0; 1] and p4 2 [0; 1) with p3 > p4. Additionally, the algorithm receives a character set C, consisting of alphanumeric and punctuation characters from the Latin 1 character set. At each character in xT noise character-level noise is injected with a probability of 1 p3. The noise consists of either character insertion, omission, or repetition. All three types of noise are equally likely to happen. We hypothesize that the presence of character-level noise makes more such noise likelier. Thus, in case of insertion or repetition, we repeatedly add additional noise characters with a probability of 1 p4. See algorithm 2 for more details.
Our implementation of the approach described in this section using PyTorch will be published at https://github.com/JonathanSchaber/shared task. 4
Our submitted model achieves an F1 score of 96.8% in the official evaluation on the test set, resulting in a second place, 1.4% behind the best model.
Table 2 gives an overview of different hyperparameter settings with the corresponding results on the development and test set.5 We report the following observations: (1) More fine-grained classes generally lead to better results. (2) There is a strong performance drop from development to test set supporting our noisiness gap assumption. (3) Injecting noise alleviates this drop and, compared to the same configurations without noise, leads to relative performance increases ranging from 1.2% to 2.7% F1 score on the test set.
(4) Increasing embedding dimensionality leads to more stable results over different granularities. (5) Clipping after 100 characters leads to a bisection of training time while on average upholding performance.
Since the test set does not contain languages from A the character-based filter is rarely triggered and its impact on performance is negligible. However, the filter might be important when detecting GSW text in settings where languages in A occur more frequently. More information about hyperparameters and hardware is given in Appendix C. 5
This paper described our submission to the GSWID 2020 shared task. We introduced a BiGRU-based architecture, a character-based filter and a noisifier module. Our evaluation results 5The test set evaluation relies on gold labels that were made available after the submission deadline. show that more fine-grained classes and adding noise to the training data leads to performance increases. Further investigations will concern pretraining, transformer-based architectures, and a more sophisticated noisifier.
Acknowledgments Above all, we would like to thank Simon Clematide who supervised this project and suggested more fine-grained classes. Further, we thank the shared task organizers, especially Pius von Da¨niken who clarified our questions, and also our proofreaders and reviewers. We formally define our architecture as follows: Let E(xi) denote a function that returns the embedding for a given character xi 2 x. The last hidden states h!n, h0 are given by h!n, h0, !o0, ..., o!n, on, ... o0 = BiGRU(E(x0); :::; E(xn)). (1) We feed each last hidden state into a block of dense layers defined as f block(v) = W2T dr(ReLU(W1T dr(v) + b1)) + b2 (2) where W1 2 R300 150 and W2 2 R150 50 denote the weight matrices of the block’s first and second layer, dr denotes a dropout function, b1 and b2 denote learnable biases, and ReLU denotes the rectified linear unit activation function.
z1 is computed as z1 = f block(h0) and z2 respectively as z2 = f block(h!n). Note that the weights of the two dense blocks are not shared, but initialized and trained independently. We concatenate z1 and z2 to z, which is then fed through a final layer formalized as follows: f f inal(v) = log-softmax(W T v + b) (3) with W 2 R100 Q where Q is the # target classes.
B
As examples for data containing token- and character-level noise, consider the following two made up text snippets.6 Character sequences we regard as noise are boldfaced.
Da¨ bus isch stablibe, mis ticket nu¨me gu¨ltig trying to stay chill ooohhhh neiiiii mir ha¨nds no¨d gschafft
In the following algorithms, r() denotes a function which returns a random value 2 [0; 1].
In algorithm 2 parameter A contains f’omission’, ’insertion’, ’repetition’g.
For a given example input our noisifier with parameters settings as shown in the hyper parameter table in Appendix C introduces noise structures into non-noisy texts, like the following:
6For copyright reasons we do not cite or display real tweets in the publication.
Algorithm 1 Token-level noise injection Input: x; p1; p2; L Output: xT noise t split x into tokens initialize array u for each tj 2 t do if r1 r() > p1 then add randomly chosen token l 2 L to u while r2 r() > p2 ^ c < k=2 do
add randomly chosen token l 2 L to u end while end if append tj to u end for return concatenate u to string xT noise Algorithm 2 Character-level noise injection Input: xT noise; p3; p4; C; A Output: xCnoise initialize empty string xCnoise for each xi 2 xT noise do if r3 r() > p3 then a choose random action 2 A if a = ’omission’ then
continue else if a = ’insertion’ then
b choose random character 2 C else if a = ’repetition’ then
b xi end if add b to xCnoise while r4 r() > p4 do
add b to xCnoise end while end if add xi to xCnoise end for return xCnoise clean: Viele Personen sind nicht der U¨ berzeugung. noisy: Viele Personen sind nicht der U¨ erzeugunnng. clean: Hast du schon die neue xbox 3 gesehen? noisy: Hast du music schon die neue xbox 3 geesehen? clean: You’ll never guess what happened this morning. noisy: You’ll never guess Jwhat happened this morninnng. clean: Le tigre est un grand chat de proie originaire d’Asie. noisy: Le tigre estt un grand chatde proie originaire d’Asie. clean: C’e` ancora una mancanza di chiarezza, non possiamo farci nulla. noisy: C’e` ancor una mancanza di chiarezza, non possiamo St. Moritz Friso¨r farci nulla.
As is obvious from these examples, the noise injected by the noisifier still looks quite different from human created noise, thus a more sophisticated noisifier is desirable. during ablation testing. In the table below, we report the parameters we left unchanged during ablation testing. Value 300 300 150 50 100 # target classes
0.1 0.001
15 0.99 0.6 0.97 0.5 0.8 all experiments.
After two epochs the learning rate is decreased from 0.001 to 0.0001 and after six epochs the learning rate is further decreased to 0.00003.
We ran our models on a NVIDIA GeForce GTX TITAN X graphics processing unit.