English. Recent scientific studies on natural language processing (NLP) report the outstanding effectiveness observed in the use of context-dependent and task-free language understanding models such as ELMo, GPT, and BERT. Specifically, they have proved to achieve state of the art performance in numerous complex NLP tasks such as question answering and sentiment analysis in the English language. Following the great popularity and effectiveness that these models are gaining in the scientific community, we trained a BERT language understanding model for the Italian language (AlBERTo). In particular, AlBERTo is focused on the language used in social networks, specifically on Twitter. To demonstrate its robustness, we evaluated AlBERTo on the EVALITA 2016 task SENTIPOLC (SENTIment POLarity Classification) obtaining state of the art results in subjectivity, polarity and irony detection on Italian tweets. The pre-trained AlBERTo model will be publicly distributed through the GitHub platform at the following web address: https://github.com/ marcopoli/AlBERTo-it in order to facilitate future research.
The recent spread of pre-trained text representation models has enabled important progress in
Copyright c 2019 for this paper by its authors. Use
permitted under Creative Commons License Attribution 4.0
International (CC BY 4.0).
Natural Language Processing. In particular,
numerous tasks such as part of speech tagging,
question answering, machine translation, and text
classification have obtained significant contributions
in terms of performance through the use of
distributional semantics techniques such as word
embedding. Mikolov et al. (2013) notably
contributed to the genesis of numerous strategies for
representing terms based on the idea that
semantically related terms have a similar vector
representations. Such technologies as Word2Vec
A Task-Independent Sentence Understanding
Model is based on the idea of creating a deep
learning architecture, particularly an encoder and
a decoder, so that the encoding level can be used
in more than one NLP task. In this way, it is
possible to obtain a decoding level with weights
optimized for the specific task (fine-tuning). A
general-purpose encoder should, therefore, be able
to provide an efficient representation of the terms,
their position in the sentence, context,
grammatical structure of the sentence, semantics of the
terms. One of the first systems able to satisfy
these requirements was ELMo
As pointed out in the previous sections, the aim
of this work is to create a linguistic resource for
Italian that would follow the most recent
strategies used to address NLP problems in English. It
is well known that the language used on social
networks is different from the formal one, also as a
consequence of the presence of mentions,
uncommon terms, links, and hashtags that are not present
elsewhere. Moreover multiple language models in
their multilingual version, are not performing well
in every specific language, especially with a
writing style different from that of books and
encyclopedic descriptions
Figure 1 shows the BERT and AlBERTo
strategy of learning. The “masked learning” is
applied on a 12x Transformer Encoder, where, for
each input, a percentage of terms is hidden and
then predicted for optimizing network weights in
back-propagation. In AlBERTo, we implement
only the “masked learning” strategy, excluding the
step based on “next following sentence”. This is a
crucial aspect to be aware of because, in the case
of tweets, we do not have cognition of a flow of
tweets as it happens in a dialog. For this reason,
we are aware that AlBERTo is not suitable for the
task of question answering, where this property is
essential. On the contrary, the model is well suited
for classification and prediction tasks. The
decision to train AlBERTo, excluding the ”next
following sentence” strategy, makes the model similar
in purposes to ELMo. Differently from it, BERT
and AlBERTo use transformer architecture instead
on biLSTM which have been demonstrated to
perform better in natural language processing tasks.
In any case, we are considering the possibility to
learn an Italian ELMo model and to compare it
with the here proposed model.
In order to tailor the tweet text to BERT’s
input structure, it is necessary to carry out
preprocessing operations. More specifically, using
Python as the programming language, two
libraries were mainly adopted: Ekphrasis
Normalizing URL, emails, mentions, percents, money, time, date, phone numbers, numbers, emoticons;
Tagging and unpacking hashtags.
The normalization phase consists in replacing each term with a fixed tuple < [entity type] >. The tagging phase consists of enclosing hashtags with two tags < hashtag > ::: < =hashtag > representing their beginning and end in the sentence. Whenever possible, the hashtag has been unpacked into known words. The text is cleaned and made easily readable by the network by converting it to its lowercase form and all characters except emojis, !, ? and accented characters have been deleted. An example of preprocessed tweet is shown in Figure 2.
SentencePiece is a segmentation algorithm used for learning the best strategy for splitting text into terms in an unsupervised and languageindependent way. It can process up to 50k sentences per seconds and generate an extensive vocabulary. It includes the most common terms in the training set and the subwords which occur in the middle of words, annotating them with ’##’ in order to be able to encode also slang, incomplete or uncommon words. An example of a piece of the vocabulary generated for AlBERTo is shown in Figure 3. SentencePiece also produced a tokenizer, used to generate a list of tokens for each tweet further processed by BERT’s create pretraining data.py module.
1https://github.com/google-research/ bert/
sentencepiece
The dataset used for the learning phase of
AlBERTo is TWITA
The AlBERTo model has been trained using the following configuration: b e r t b a s e c o n f i g = f ” a t t e n t i o n p r o b s d r o p o u t p r o b ” : 0 . 1 , ” d i r e c t i o n a l i t y ” : ” b i d i ” , ” h i d d e n a c t ” : ” g e l u ” , ” h i d d e n d r o p o u t p r o b ” : 0 . 1 , ” h i d d e n s i z e ” : 7 6 8 , ” i n i t i a l i z e r r a n g e ” : 0 . 0 2 , ” i n t e r m e d i a t e s i z e ” : 3 0 7 2 , ” m a x p o s i t i o n e m b e d d i n g s ” : 5 1 2 , ” n u m a t t e n t i o n h e a d s ” : 1 2 , ” n u m h i d d e n l a y e r s ” : 1 2 , ” p o o l e r f c s i z e ” : 7 6 8 , ” p o o l e r n u m a t t e n t i o n h e a d s ” : 1 2 , ” p o o l e r n u m f c l a y e r s ” : 3 , ” p o o l e r s i z e p e r h e a d ” : 1 2 8 , ” p o o l e r t y p e ” : ” f i r s t t o k e n t r a n s f o r m ” , ” t y p e v o c a b s i z e ” : 2 , ” v o c a b s i z e ” : 128000 g # Input data pipeline config TRAIN BATCH SIZE = 128 MAX PREDICTIONS = 20 MAX SEQ LENGTH = 128 MASKED LM PROB = 0 . 1 5 # Training procedure config EVAL BATCH SIZE = 64 LEARNING RATE = 2e 5 TRAIN STEPS = 1000000 SAVE CHECKPOINTS STEPS = 2500 NUM TPU CORES = 8
The training has been performed over the Google Collaborative Environment (Colab)3, Using a 8 core Google TPU-V24 and a Google Cloud Storage Bucket5. In total, it took 50 hours to create a complete AlBERTo model. More technical details are available in the Notebook ”Italian Pre-training BERT from scratch with cloud TPU” into the project repository. 4
We evaluate AlBERTo on a task of sentiment
analysis for the Italian language. In
particular, we decided to use the data released for
the SENTIPOLC (SENTIment Polarity
Classification) shared task
decide whether a given message is subjective or objective”;
cide whether a given message is of positive, negative, neutral or mixed sentiment”; Irony Detection: “a system must decide whether a given message is ironic or not”.
Data provided for training and test are tagged with six fields containing values related to manual annotation: subj, opos, oneg, iro, lpos, lneg. These labels describe consequently if the sentence is subjective, positive, negative, ironical, literal positive, literal negative. For each of these classes, there is a 1 where the sentence satisfy the label, a 0 instead.
The last two labels “lpos” and “lneg” that describe the literal polarity of the tweet have not been considered in the current evaluation (nor in the official shared task evaluation). In total, 7410 tweets have been released for training and 2000 for testing. We do not used any validation set because we do not performed any phase of model selection during the fine-tuning of AlBERTo. The evaluation was performed considering precision (p), recall (r) and F1-score (F1) for each class and for each classification task.
Subjectivity Polarity Pos.
Polarity Neg.
Irony Subjectivity Polarity Pos.
Polarity Neg.
Irony
System AlBERTo Unitor.1.u Unitor.2.u samskara.1.c ItaliaNLP.2.c AlBERTo fine-tuning. We fine-tuned AlBERTo four different times, in order to obtain one classifier for each task except for the polarity where we have two of them. In particular, we created one classifier for the Subjectivity Classification, one for Polarity Positive, one for Polarity Negative and one for the Irony Detection. Each time we have re-trained the model for three epochs, using a learning rate of 2e-5 with 1000 steps per loops on batches of 512 example from the training set of the specific task. For the fine-tuning of the Irony Detection classifier, we increased the number of epochs of training to ten observing low performances using only three epochs as for the other classification tasks. The fine-tuning process lasted 4 minutes every time.
Table 1 show the output obtained from the official evaluation script of SENTIPOLC 2016. It is important to note that the values on the individual classes of precision, recall and, F1 are not compared with them of the systems that participated in the competition because they are not reported in the overview paper of the task. Nevertheless, some considerations can be drawn. The classifier based on AlBERTo achieves, on average, high recall on class 0 and low values on class 1. The opposite situation is instead observed on the precision, where for the class 1 it is on average superior to the recall values. This note suggests that the system is very good at classifying a phenomenon and when it does, it is sure of the prediction made even at the cost of generating false negatives.
On each of the sub-tasks of SENTIPOLC, it can be observed that AlBERTo has obtained state of the art results without any heuristic tuning of learning parameters (model as it is after finetuning training) except in the case of irony detection where it was necessary to increase the number of epochs of the learning phase of fine-tuning. Comparing AlBERTo with the best system of each subtask, we observe an increase in results between 7% and 11%. The results obtained are exciting, from our point of view, for further future work. 5
In this work, we described AlBERTo, the first Italian language understanding model based on social media writing style. The model has been trained using the official BERT source code on a Google TPU-V2 on 200M tweets in the Italian language. The pre-trained model has been finetuned on the data available for the classification task SENTIPOLC 2016, showing SOTA results. The results allow us to promote AlBERTo as the starting point for future research in this direction. Model repository: https://github.com/ marcopoli/AlBERTo-it
The work of Marco Polignano is funded by project ”DECiSION” codice raggruppamento: BQS5153, under the Apulian INNONETWORK programme, Italy. The work of Valerio Basile is partially funded by Progetto di Ateneo/CSP 2016 (Immigrants, Hate and Prejudice in Social Media, S1618 L2 BOSC 01.