This paper presents early results of a weakly supervised language model induction approach for language segmentation of multilingual texts with a special focus on short texts.
Twitter tweets often contain non-standard
language and they are limited to 140
characters. While not a problem in itself, these
restrictions can pose difficulties for
natural language processing systems
Supervised methods perform well on the
task of language identification in general
Language segmentation consists in
identifying the language borders within a
multilingual text
Generation starts at the beginning of the text, takes the first word and decomposes it into uni-, bi- and trigrams. These n-grams are then added to the initial language model, which is empty at the start. For each following word, the existing language models evaluate the word in question. The highest ranking model is updated with the word. If no model scores higher than the threshold value for model creation, a new model is created.
Backwards generation works exactly the same, but starts at the end of the text and moves towards the beginning of the text.
Finally, the two models that have the most similar unigram distribution are merged. This way, the language models iteratively amass information about different languages.
The induction step is repeated at least twice. At the end of the induction, while there are two models that have a similarity greater than a certain threshold value, these models are merged.
Language segmentation is then performed by assigning each word in the text to the model that yields the highest probability for the word in question. 3
Table 1 shows the results for a set of example tweets manually collected from Twitter. For all tweets, a gold standard has been manually created and evaluated against. The evaluation is that of a clustering task; the words of a text are clustered around different induced language models. Whenever the language model induction outperformed the supervised trained language models, the score is indicated in bold.
Besides the F score (F1), the F5 score
is also indicated. This score sets to 5,
weighting recall higher than precision. This
means that throwing together pairs that are
separate in the gold standard is penalized
more strongly than splitting pairs that
occur together in the gold standard
For comparison purposes, a supervised
approach as described in
For this toy example, the results show that the language model induction seems to work reasonably well with scores comparable to the supervised approach, sometimes even performing better than the supervised approach.
Closer inspection of the results reveals that the language model induction tends to generate too many clusters for a single language, resulting in a degradation of the accuracy, while on the other hand also being able to separate the different languages surprisingly well.
For example, the first tweet “Μόλις ψήφισα αυτή τη λύση Internet of Things, στο διαγωνισμό BUSINESS IT EXCELLENCE.” is decomposed into two English clusters and two Greek clusters, with one erroneous inclusion of ‘EXCELLENCE.’ in the Greek cluster. • Things, • Μόλις λύση διαγωνισμό EXCELLENCE. • Internet of BUSINESS IT • ψήφισα αυτή τη στο Tweet 1 Tweet 2 Tweet 3 Tweet 4 Tweet 5
The second tweet “Demain #dhiha6 Keynote 18h @dhiparis “The collective dynamics of science-publish or perish; is it all that counts?” par David” and its decomposition. It is clear that we have one English cluster and one French cluster, and two other clusters, one of which could be labeled ‘Named Entity’ cluster and the other possibly ‘English with erroneous inclusion of @dhiparis’. Interestingly, the French way of notating time ‘18h’ is also included in the French cluster.
• Keynote “The collective of sciencepublish or perish; it all that counts?” • Demain 18h par • #dhiha6 David
The third tweet “Food and breuvages in Edmonton are ready to go, just waiting for the fans #FWWC2015 #bilingualism” is split into one acronym group, three English clusters and one French cluster with the erroneous inclusion of ‘go’.
• #FWWC2015 • breuvages, go • Food, Edmonton, to, for, the • in, waiting, #bilingualism • and, are, ready, just, fans
The fourth tweet “my dad comes back from poland with two crates of strawberries, żubrówka and adidas jackets omg” again is split into two English clusters and one Polish cluster with the erroneous inclusion of ‘back’. • comes, from, with, two, crates, of, strawberries, jackets, omg • my, dad, poland, and, adidas • back, żubrówka
Finally, the last tweet ”Buna dabo naw (coffee is our bread).” is decomposed as follows. The English words are split across four clusters while the transliterated Amharic text is clustered together. The splitting is due to the structure of the tweet; there is not enough overlapping information to build an English cluster.
• (coffee • bread). • is • our • Buna dabo naw 4
The paper has presented the early findings of a weakly supervised approach for language segmentation that works on short texts. By taking the text itself as basis for the induced language models, there is no need for training data. As the approach does not rely on external language knowledge, the approach is language independent.
The results seem promising, but the approach has to be tested on more data. Still, being able to achieve results comparable to supervised approaches with a weakly supervised method is encouraging. 5
Future work should concern the reduction of the number of generated clusters, ideally arriving at one cluster per language. Alternatively, it would be possible to smooth the frequent switching of language models by taking context into account.
Also, since the structure of the text strongly influences the presented approach, some form of text normalization could be used to increase the robustness of the system. GaloTyri. “Μόλις ψήφισα αυτή τη λύση Internet of Things, στο διαγωνισμό BUSINESS IT EXCELLENCE.”. 19 June 2015, 12:06. Tweet.
HBS (HBS_Tweets). ”Food and breuvages in Edmonton are ready to go, just waiting for the fans #FWWC2015 #bilingualism”. 6 June 2015, 23:29. Tweet.
katarzyne (wifeyriddim). ”my dad comes back from poland with two crates of strawberries, żubrówka and adidas jackets omg”. 8 June 2015, 08:49. Tweet.
TheCodeswitcher. ”Buna dabo naw (coffee is our bread).”. 9 June 2015, 02:12. Tweet.