English. This article describes the system that participated in the POS tagging for Italian Social Media Texts (PoSTWITA) task of the 5th periodic evaluation campaign of Natural Language Processing (NLP) and speech tools for the Italian language EVALITA 2016. The work is a continuation of Stemle (2016) with minor modifications to the system and different data sets. It combines a small assertion of trending techniques, which implement matured methods, from NLP and ML to achieve competitive results on PoS tagging of Italian Twitter texts; in particular, the system uses word embeddings and character-level representations of word beginnings and endings in a LSTM RNN architecture. Labelled data (Italian UD corpus, DiDi and PoSTWITA) and unlabbelled data (Italian C4Corpus and PAIS A`) were used for training. The system is available under the APLv2 open-source license.
tema che ha partecipato al task POS tagging for Italian Social Media Texts (PoSTWita) nell’ambito di EVALITA 2016, la 5° campagna di valutazione periodica del Natural Language Processing (NLP) e delle tecnologie del linguaggio.
Il lavoro e` un proseguimento di quanto descritto in Stemle (2016), con modifiche minime al sistema e insiemi di dati differenti. Il lavoro combina alcune tecniche correnti che implementano metodi comprovati dell’NLP e del Machine Learning, per raggiungere risultati competitivi nel PoS tagging dei testi italiani di Twitter. In particolare il sistema utilizza strategie di word embedding e di rappresentazione character-level di inizio e fine parola, in un’architettura LSTM RNN. Dati etichettati (Italian UD corpus, DiDi e PoSTWITA) e dati non etichettati (Italian C4Corpus e PAISA`) sono stati utilizzati in fase di training.
Part-of-speech (PoS) tagging is an essential processing stage for virtually all NLP applications. Subsequent tasks, like parsing, namedentity recognition, event detection, and machine translation, often utilise PoS tags, and benefit (directly or indirectly) from accurate tag sequences.
Actual work on PoS tagging, meanwhile,
mainly concentrated on standardized texts for
many years, and frequent phenomena in
computermediated communication (CMC) and Web
corpora such as emoticons, acronyms, interaction
words, iteration of letters, graphostylistics,
shortenings, addressing terms, spelling variations, and
boilerplate
On the other hand, the interest in automatic evaluation of social media texts, in particular for microblogging texts such as tweets, has been growing considerably, and specialised tools for Twitter data have become available for different languages. But Italian completely lacks such resources, both regarding annotated corpora and specific PoS-tagging tools.1 To this end, the POS tagging for Italian Social Media Texts (PoSTWITA) task was proposed for EVALITA 2016 concerning the domain adaptation of PoS-taggers to Twitter texts.
Our system combined word2vec (w2v) word embeddings (WEs) with a single-layer Long Short Term Memory (LSTM) recurrent neural network (RNN) architecture. The sequence of unlabelled w2v representations of words is accompanied by the sequence of n-grams of the word beginnings and endings, and is fed into the RNN which in turn predicts PoS labels.
The paper is organised as follows: We present our system design in Section 2, the implementation in Section 3, and its evaluation in Section 4. Section 5 concludes with an outlook on possible implementation improvements. 2
Overall, our design takes inspiration from as far back as Benello et al. (1989) who used four preceding words and one following word in a feedforward neural network with backpropagation for PoS tagging, builds upon the strong foundation laid down by Collobert et al. (2011) for a neural network (NN) architecture and learning algorithm that can be applied to various natural language processing tasks, and ultimately is a variation of Nogueira dos Santos and Zadrozny (2014) who trained a NN for PoS tagging, with characterlevel and WE representations of words.
Also note that an earlier version of the system
was used in Stemle (2016) to participate in the
EmpiriST 2015 shared task on automatic linguistic
annotation of computer-mediated communication
/ social media
Recently, state-of-the-art results on various
linguistic tasks were accomplished by architectures
using neural-network based WEs. Baroni et al.
(2014) conducted a set of experiments comparing
the popular w2v
Word embeddings provide high-quality low
dimensional vector representations of words from
large corpora of unlabelled data, and the
representations, typically computed using NNs, encode
many linguistic regularities and patterns
The morphology of a word is opaque to WEs, and
the relatedness of the meaning of a lemma’s
different word forms, i.e. its different string
representations, is not systematically encoded. This means
that in morphologically rich languages with
longtailed frequency distributions, even some WE
representations for word forms of common lemmata
may become very poor
We agree with Nogueira dos Santos and Zadrozny (2014) and Kim et al. (2015) that subword information is very important for PoS tagging, and therefore we augment the WE representations with character-level representations of the word beginnings and endings; thereby, we also stay language agnostic—at least, as much as possible—by avoiding the need for, often language specific, morphological pre-processing. 2.3
Language Models are a central part of NLP. They are used to place distributions over word sequences that encode systematic structural properties of the sample of linguistic content they are built from, and can then be used on novel content, e.g. to rank it or predict some feature on it. For a detailed overview on language modelling research see Mikolov (2012).
A straight-forward approach to incorporate WEs into feature-based language models is to use the embeddings’ vector representations as features.2 Having said that, WEs are also used in NN 1http://www.evalita.it/2016/tasks/postwita 2For an overview see, e.g. Turian et al. (2010). architectures, where they constitute (part of) the input to the network.
Neural networks consist of a large number of
simple, highly interconnected processing nodes in
an architecture loosely inspired by the structure of
the cerebral cortex of the brain
RNNs are NNs where the connections between
the elements are directed cycles, i.e. the networks
have loops, and this enables them to model
sequential dependencies of the input. However,
regular RNNs have fundamental difficulties
learning long-term dependencies, and special kinds of
RNNs need to be used
Overall, with this design we not only benefit from available labelled data but also from available general or domain-specific unlabelled data. 3
We maintain the implementation in a source code repository at https://github.com/ bot-zen/. The version tagged as 1.1 comprises the version that was used to generate the results submitted to the shared task (ST).
Our system feeds WEs and character-level subword information into a single-layer RNN with a LSTM architecture. 3.1
When computing WEs we take into consideration Levy et al. (2015): they observed that one specific configuration of w2v, namely the skip-gram model with negative sampling (SGNS) ”is a robust baseline. While it might not be the best method for every task, it does not significantly underperform in any scenario. Moreover, SGNS is the fastest method to train, and cheapest (by far) in terms of disk space and memory consumption”. Coincidentally, Mikolov et al. (2013b) also suggest to use SGNS. We incorporate w2v’s original C implementation for learning WEs3 in an independent pre-processing step, i.e. we pre-compute the WEs. Then, we use gensim4, a Python tool for unsupervised semantic modelling from plain text, to load the pre-computed data, and to compute the vector representations of input words for our NN. 3.2
Our implementation uses a one-hot encoding with a few additional features for representing subword information. The one-hot encoding transforms a categorical feature into a vector where the categories are represented by equally many dimensions with binary values. We convert a letter to lower-case and use the sets of ASCII characters, digits, and punctuation marks as categories for the encoding. Then, we add dimensions to represent more binary features like ’uppercase’ (was uppercase prior to conversion), ’digit’ (is digit), ’punctuation’ (is punctuation mark), whitespace (is white space, except the new line character; note that this category is usually empty, because we expect our tokens to not include white space characters), and unknown (other characters, e.g. diacritics). This results in vectors with more than a single one-hot dimension. 3.3
Our implementation uses Keras, a high-level NNs
library, written in Python and capable of running
on top of either TensorFlow or Theano
The input to our network are sequences of the same length as the sentences we process. During training, we group sentences of the same length into batches and process the batches according to sentence length in increasing order. Each single word in the sequence is represented by its subword information and two WEs that come from two sources (see Section 4). For unknown words, 3https://code.google.com/archive/p/ word2vec/ 4https://radimrehurek.com/gensim/ i.e. words without a pre-computed WE, we first try to find the most similar WE considering 10 surrounding words. If this fails, the unknown word is mapped to a randomly generated vector representation. In Total, each word is represented by 2; 280 features: two times 500 (WEs), and sixteen times 80 for two 8-grams (word beginning and ending). If words are shorter than 8 characters their 8-grams are zero-padded.
This sequential input is fed into a LSTM layer
that, in turn, projects to a fully connected output
layer with softmax activation function. During
training we use dropout for the projection into the
output layer, i.e. we set a fraction (0:5) of the input
units to 0 at each update, which helps prevent
overfitting
We used our slightly modified implementation to participate in the POS tagging for Italian Social Media Texts (PoSTWITA) shared task (ST) of the 5th periodic evaluation campaign of Natural Language Processing (NLP) and speech tools for the Italian language EVALITA 2016. First, we describe the corpora used for training, and then the specific system configuration(s) for the ST. 4.1 4.1.1
didi-it
The Italian part consists of around 100,000 tokens collected from 20 profiles of Facebook users residing in South Tyrol. This version has about 20,000 PoS tags semi-automatically corrected by a single annotator.
The anonymised corpus is freely available for research purposes.
Universal Dependencies (UD) is a project that is developing cross-linguistically consistent treebank annotation for many languages.5
italian-UD6 (version from January 2015)
corpus was originally obtained by conversion from
ISDT (Italian Stanford Dependency Treebank)
and released for the dependen
The corpus contains around 317,000 tokens in around 13,000 sentences from different sources and genres. It is available under the CC BY-NCSA 3.07 license. 4.1.3 PoSTWITA (PoS and w2v) postwita is the Twitter data made available by the organizers of the ST. It contains Twitter tweets from the EVALITA2014 SENTIPLOC corpus: the development and test set and additional tweets from the same period of time were manually annotated for a global amount of 6438 tweets (114,967 tokens) and were distributed as the development set. The data is PoS tagged according to UD but with the additional insertion of seven Twitterspecific tags. All the annotations were carried out by three different annotators. The data was only distributed to the task participants. 4.1.4 C4Corpus (w2v) c4corpus8 is a full documents Italian Web corpus that has been extracted from CommonCrawl, the largest publicly available general Web crawl to date. See Habernal (2016) for details about the corpus construction pipeline, and other information about the corpus.
The corpus contains about 670m tokens in 22m
sentences. The data is available under the
CreativeCommons license family.
4.1.5 PAISA` (w2v)
paisa
7Creative Commons Attribution-NonCommercialShareAlike 3.0 Unported, i.e. the data can be copied and redistributed, and adapted for purposes other than commercial ones. See https://creativecommons.org/ licenses/by-nc-sa/3.0/ for more details.
8https://github.com/dkpro/ dkpro-c4corpus in the context of the project PAISA` (P´ıattaforma per l’Apprendimento dell’Italiano Su corpora Annotati) with the aim to provide a large resource of freely available Italian texts for language learning by studying authentic text materials.
The corpus contains about 270m tokens in about 8m sentences. The data is available under the CC BY-NC-SA 3.09 license. 4.2
For the ST we used one overall configuration for the system but three different corpus configurations for training. However, only one corpus configuration was entered into the ST: we used PoS tags from didi-it + postwita (run 1), from italianUD (run 2), and from both (run 3). For w2v we trained a 500-dimensional skip-gram model on didi-it + italian-UD + postwita that ignored all words with less than 2 occurrences within a window size of 10; it was trained with negative sampling (value 15). We also trained a 500-dimensional skip-gram model on c4corpus + paisa that ignored all words with less than 33 occurrences within a window size of 10; it was trained with negative sampling (value 15).
The other w2v parameters were left at their default settings10.
The evaluation of the systems was done by the organisers on unlabelled but pre-tokenised data (4759 tokens in 301 tweets), and was based on a token-by-token comparison. The considered metric was accuracy, i.e. the number of correctly assigned PoS tags divided by the total number of tokens.
(1) didi-it + postwita (2) italian-UD (3) didi-it + postwita + italian-UD Winning Team
We believe, the unexpectedly little performance gain from utilizing the much larger italian-UD data over the rather small didi-it + postwita data may be rooted in the insertion of Twitter-specific tags into the data (see 4.1.3), something we did not account for, i.e. 18; 213 of 289; 416 and more 9https://creativecommons.org/licenses/ by-nc-sa/3.0/ 10-sample 1e-3 -iter 5 -alpha 0.025 importantly 7; 778 of 12; 677 sentences had imperfect information during training. 5
We presented our submission to the PoSTWITA task of EVALITA 2016, where we participated with moderate results. In the future, we will try to rerun the experiment with training data that takes into consideration the Twitter-specific tags of the task.
The computational results presented have been achieved in part using the Vienna Scientific Cluster (VSC).