This paper presents our approach for SwissText & KONVENS 2020 shared task 2, which is a multi-stage neural model for Swiss German (GSW) identification on Twitter. Our model outputs either GSW or non-GSW and is not meant to be used as a generic language identifier. Our architecture consists of two independent filters where the first one favors recall, and the second one filter favors precision (both towards GSW). Moreover, we do not use binary models (GSW vs. not-GSW) in our filters but rather a multi-class classifier with GSW being one of the possible labels. Our model reaches F1-score of 0.982 on the test set of the shared task.
Out of over 8000 languages in the world
However, Twitter data is linguistically diverse
and especially includes tweets in many low-resource
languages/dialects. Having a better performing Twitter
LID can help us to gather large amounts of (unlabeled)
text in these low-resource languages that can be used to
enrich models in many down-stream NLP tasks, such
as sentiment analysis
However, the generalization of state-of-the-art NLP models to low-resource languages is generally hard due to the lack of corpora with good coverage in these languages. The extreme case is the spoken dialects, where there might be no standard spelling at all. In this paper, we especially focus on Swiss German as our low-resource dialect. As Swiss German is a spoken dialect, people might spell a certain word differently, and even a single author might use different spelling for a word between two sentences. There also exists a dialect continuum across the German-speaking part of Switzerland, which makes NLP for Swiss German even more challenging. Swiss German has its own pronunciation, grammar and also lots of its words are different from German.
There exists some previous efforts for discriminating
similar languages with the help of tweets metadata
such as geo-location
LIDs that support GSW like fastText
In this paper, we use two independently trained filters to remove non-GSW tweets. The first filter is a classifier that favors recall (towards GSW), and the second one favors precision. The exact same idea can be extended to N consecutive filters (with N 2), with the first N 1 favoring recall and the last filter favoring precision. In this way, we make sure that GSW samples are not filtered out (with high probability) in the first N 1 iterations, and the whole pipeline GSW precision can be improved by having a filter that favors precision at the end (N -th filter). The reason that we use only two filters is that adding more filters improved the performance (measured by GSW F1-score) negligibly on our validation set.
We demonstrate that by using this architecture, we can achieve F1-score of 0.982 on the test set, even with a small amount of available data in the target domain (Twitter data). Section 2 presents the architecture of each of our filters and the rationale behind the chosen training data for each of them. In section 3, we discuss our LID implementation details and also discuss the detailed description of used datasets. Section 4 presents the performance of our filters on the held-out test dataset. Moreover, we demonstrate the contribution of each of the filters on removing non-GSW filters to see their individual importance in the whole pipeline (for this specific test dataset). 2
In this paper, we follow the combination of N 1 filters favoring recall, followed by a final filter that favors more precision. We choose N = 2 in this paper to demonstrate the effectiveness of the approach. As discussed before, adding more filters improved the performance of the pipeline negligibly for this specific dataset. However, for more challenging datasets, it might be needed to have N > 2 to improve the LID precision.
Both of our filters are multi-class classifiers with GSW being one of the possible labels. We found it empirically better to use roughly balanced classes for training the multi-class classifier, rather than making the same training data a highly imbalanced GSW vs. non-GSW training data for a binary classifier, especially for the first filter (section 2.1) which has much more parameters compared to the second filter (section 2.2). 2.1
The first filter should be designed in a way to favor GSW recall, either by tuning inference thresholds or by using training data that implicitly enforces this bias towards GSW. Here we follow the second approach for this filter by using different domains for training different labels, which is further discussed below. Moreover, we use a more complex (in terms of the number of parameters) model for the first filter, so that it does the main job of removing non-GSW inputs while having reasonable GSW precision (further detail in section 4). The second filter will be later used to improve the pipeline precision by removing a relatively smaller number of non-GSW tweets.
Our first filter is a fine-tuned BERT
2Training details available at https://huggingface.
co/bert-base-german-cased
For this filter, we choose the same eight classes for
training LID as Linder et al. (2019) (the dataset classes
and their respective sizes can be found in section 3.1).
These languages are similar in structure to GSW (such
as German, Dutch, etc.), and we try to train a model
that can distinguish GSW from similar languages
to decrease GSW false positives. For all classes
except GSW, we use sentences (mostly Wikipedia
and Newscrawl) from Leipzig text corpora
Most GSW training samples (SwissCrawl data) come from forums and social media, which are less formal (in structure and also used phrases) than other (non-GSW) classes samples (mostly from Wikipedia and NewsCrawl). Moreover, as our target dataset consist of tweets (mostly informal sentences), this could make this filter having high GSW recall during the inference phase. Additionally, our main reason for using a cased tokenizer for this filter is to let the model also use irregularities in writing, such as improper capitalization. As these irregularities mostly occur in informal writing, it will again bias the model towards GSW (improving GSW recall) when tweets are passed to it, as most of the GSW training samples are informal. 2.2
For this filter, we also train a multiclass classifier with
GSW being one of the labels. The other classes are
again close languages (in structure) to GSW such
as German, Dutch and Spanish (further detail in
section 3.1). Additionally, as mentioned before, our
second filter should have a reasonably high precision
to enhance the full pipeline precision. Hence, unlike
the first filter, we choose the whole training data
to be sampled from a similar domain to the target
test set. non-GSW samples are tweets from SEPLN
2014
As the described training data is rather small
compared to the first filter training, we should also train a
simpler architecture with significantly fewer parameters.
We take advantage of fastText
In this section, we describe the datasets and the hyperparameters for both filters in the pipeline. We also describe our preprocessing method that is specifically designed to handle inputs from social media. 3.1
For both filters, we use 80% of data for training, 5% for validation set and 15% for the test set. 3.1.1
The sentences are from Leipzig corpora
The sentences are mostly from Twitter (except for
some GSW samples from Swiss SMS corpus
In this section, we evaluate our two filters performance (either in isolation or when present in the full pipeline) on the held-out test dataset of the shared task. We also evaluate the BERT filter on its test data (Leipzig and SwissCrawl samples). 4.1
We first evaluate our BERT filter on the test set of the first filter (Leipzig corpora + SwissCrawl). In Table 3 we demonstrate the filter performance on different labels. The filter has an F1-score of 99.8% on the GSW test set. However, when this model is applied to Twitter data, we expect a decrease in performance due to having short and also informal messages. In Table 4, we can see both filters performance either in isolation or when they are used together. As shown in this table, the model improvement by adding the second filter is rather small. The main reason can be seen in Table 5 as the majority of non-GSW filtering is done by the first filter for the shared-task test set (Table 6).
Precision 0.9742 0.9076 0.9811 0.9915
Recall 0.9896 0.9892 0.9834 0.3619
F1-score 0.9817 0.9466 0.9823 0.5303 Our designed LID outperforms the baseline significantly (Table 4) which underlines the importance of having a domain-specific LID. Additionally, although the positive effect of the second filter is quite small on the test set, when we applied the same architecture on randomly sampled tweets (German tweets according to Twitter API), we observed that having the second filter could reduce the number of GSW false positives significantly. Hence, the number of used filters is indeed totally dependent on the complexity of the target dataset. 5
In this work, we propose an architecture for spoken dialect (Swiss German) identification by introducing a multi-filter architecture that is able to filter out non-GSW tweets during the inference phase effectively. We evaluated our model on the GSW LID shared task test-set, and we reached an F1-score of 0.982.
However, there are other useful features that can be
used during training, such as orthographic conventions
in GSW writing, as observed by Honnet et al. (2017),
which their presence might not be easily captured even
by a complex model like BERT. Moreover, in this paper,
we did not use tweets metadata as a feature and only
focused on tweet content, although they can improve
LID classification for dialects considerably