English. The POS tagger developed by Mivoq to tag tweets according to PosTwITA task guidelines as defined at Evalita 2016 is presented. The system obtained third position with 92.7% of accuracy.
Twitter messages (Tweets) are challenging for
Natural Language Processing (NLP) due to the
conversational nature of their content, the
unconventional orthography and the 140 character limit
of each tweet
Text-To-Speech (TTS) systems make large use of NLP technologies as part of input preprocessing, in order to cope with: homographs disambiguation: TTS systems may use POS tagging as a preliminary step to identify the correct pronunciation for those words that shares the same written form, but are pronounced differently. Many Italian homographs can be disambiguated using the POS tag (e.g., the string “ancora” has two possible pronunciations according to the fact that we are referring to the noun “anchor” or to the adverb “still”), although in some cases more information is needed; words expansion: as not all the text correspond to pronounceable words, TTS systems need to convert some strings into pronounceable words (e.g., numbers, units, acronyms, URL, . . . ). POS tags are useful to identify the function of a string and perform correct expansion (e.g., the string “1” can be expanded into “uno”, “un” and “una”, according to the POS tags of the surrounding strings); prosody prediction: prosody includes several aspects of speech, such as intonation, stress, tempo, rhythm and pauses, that are often not perfectly encoded by grammar or by choice of vocabulary, but still are important for the communication and should be correctly rendered by TTS systems. POS tags correlate with several prosodic aspects (e.g., content words are generally produced with more prominence than function words) and thus are useful for prosody prediction.
This work is the first attempt of the author to develop a POS tagger suitable for usage in a TTS system dealing with tweets. 2
The proposed system is the combination of several taggers and resources:
Hunpos
Six different taggers, corresponding to different combinations of these resources, have been tested in a 10-fold cross-validation scheme. Three taggers have been trained on the PosTwITA training data and thus can be used independently to solve the Evalita 2016 PosTwITA task. Two of them have been trained on Evalita 2009 Pos Tagging training data and can be used to solve that task instead. The sixth tagger combines the above taggers and is the system that has been proposed. 2.1
Hunpos has been used as a black box, without the use of an external morphological lexicon: an attempt have been made to use the ISTC pronunciation dictionary, but performance degraded. Hunpos has been trained on PosTwITA training data, where it obtained an average accuracy of 92.51%, and on Evalita 2009 Pos Tagging training data, where it obtained and average accuracy of 95.72%. 2.2
Yamcha allows the usage of arbitrary features and
can be used to implement a wide range of
taggers. Features combinations are implicitly
expanded using a polynomial kernels and exploited
by SVM
Several feature sets have been tested, using the default parameters for Yamcha (i.e., only pair wise multi class method and second degree polynomial kernel have been used). Yamcha has been trained on PosTwITA training data and obtained an average accuracy of 95.41%.
The baseline experiment with Yamcha consists in
using features proposed by its author for English
POS-tagging
The default feature window has been used in this experiment (i.e., for each word form, features for previous two word forms and next two word forms are used, as long as the annotation results of the previous two word forms). Average accuracy is reported in table 1.
A rule-based annotator for typical twitter
elements
Although the accuracy improvement due to this feature was marginal (see table 1), it was present in all the tests and allowed almost perfect match of all Twitter specific elements, which is very important for words expansion. 2.2.3
Phonetic normalization has been proposed to deal
with the many alternate spelling of words in
English tweets
Twitter elements rule-based annotation.
In order to reduce the number of features, prefixes, suffixes and twitter annotations of the surrounding words has not been considered. The system achieved an average accuracy of 94.61%. 2.2.4
Finally 12 Boolean flags has been added, by performing a dictionary lookup using the normalized strings. Each flag corresponds to a PosTwITA tag (VERB CLIT, VERB, INTJ, PROPN, NOUN, ADJ, ADP, ADP A, SYM, ADV, DET, NUM) and is set to true if the ISTC dictionary contains a Tanl POS tag that can be mapped into it. By adding this feature the system achieved and average accuracy of 95.41%. 2.3 The same feature sets used with Yamcha have been tested with CRFSuite, leading to very similar results, as shown in table 1. CRFSuite has been trained on both PosTwITA and on Evalita 2009 Pos Tagging training data sets, obtaining similar accuracy for both. 2.4
The final system is a combination of five taggers based on Yamcha, by adding their output to the feature set. Tags associated to the surrounding tokens (3 previous and 3 next) are considered: using a larger window helped reducing errors with AUX and VERB tags. Results for individual taggers and the final system are shown in table 1. The system achieved an average accuracy of 95.97%. Implementing the same system using only the three taggers trained on PosTwITA data, lead to a very similar average accuracy of 95.74%, however the proposed system achieved better results in all the tests. 3
YB YB+T YN+T YN+T+D MT
Evalita 2009 POS Tagging 95.72%
95.41%
Evalita 2016 PosTwITA 92.51% 93.17% 93.30% 94.61% 95.41% 95.97% 93.02% 94.17% 95.31%
Table 1 reports average accuracy obtained in 10fold cross-validation experiments on Evalita 2016 PosTwITA and Evalita 2009 POS Tagging data sets. Each column corresponds to a different tagger and each row corresponds to a different feature set, as described in section 2. YB is the baseline feature set described in section 2.2.1, YB+T is the baseline feature set with rule-based Twitter elements’ annotation described in section 2.2.2, YN+T is the feature set described in section 2.2.3 88.95% (89.75%) 88.91% (89.72%) 90.10% (91.01%) 91.36% (92.27%) 92.71% (93.74%) 88.86% (89.58%) and YN+T+D is the YN+T feature set with the addition of dictionary usage as described in section 2.2.4. MT is the final system described in section 2.4. Table 2 reports accuracy results for the same configurations on the PosTwITA test set. In this case results after manual correction of the test set are reported below the official results. 4
Results in table 1 and table 2 shows that Yamcha and CRFSuite behave very similarly. By using YN+T+D feature set, CRFSuite achieves accuracy similar to that of Hunpos on the Evalita 2009 POS Tagging training set. With that feature set, performance of CRFSuite on both Evalita 2009 POS Tagging and Evalita 2016 PosTwITA training sets are very similar, suggesting the idea that YN+T+D feature set is quite stable and can be used successfully for both tasks. It would be interesting to include similar features in Hunpos in order to confirm the hypothesis.
Results on the Evalita 2016 PosTwITA test set shows a big accuracy loss, suggesting a mismatch between the training and the test sets. Manual correction of the test set, performed by the author, alleviated the differences, but still results are not comparable. Table 3 reports the 10 most frequent tokens in the PosTwITA training and test sets. The test set includes only function words and punctuation, but the most frequent word in the training set is the proper noun “Monti” and the word “governo” (government) is also among the most frequent tokens. Including at-mentions, hashtags and without considering the case, the word “monti” appears in 3460 tokens, making it the most frequent token in the data set and suggesting a very narrow topic. On the other hand the test set topic seems more general: the most frequent tokens are either words or punctuation marks and the first proper noun, “Italia” (Italy), appears at position 43. Given the topic mismatch, the tagger combination seems more stable than individual taggers.
The author goal was to investigate the possibility to implement a POS tagger suitable for reading tweets within a TTS system. Confusing NOUN and PROPN tags, and confusing ADJ, NOUN, AUX and VERB tags (in particular with nouns derived from adjectives or with adjectives derived from verbs) are among the most frequent errors. These errors do not typically affect the pronunciations. Hashtags, at-mentions and URL are correctly recognized with just one error, so that correct expansion of these elements can be performed. Several emoticons were wrongly annotated as punctuation, due to the limited set of Unicode emoji recognized by the rule-based annotation system and can be easily fixed by extend the match to the whole Unicode emoji set.
The difference in terms of accuracy between CRFSuite with YN+T+D feature set and the tagger combination, does not seem to justify the overhead of running multiple taggers; it would be interesting to train the taggers on a more general data set, eventually using the proposed tagger to bootstrap its annotation. Assuming that the pronunciation dictionary is already available in the TTS, the YN+T+D feature set described in section 2 seems appropriate for the POS tagging task for both tweets and more conventional text.