English. This paper presents some experiments for the construction of an highperformance PoS-tagger for Italian using deep neural networks techniques (DNN) integrated with an Italian powerful morphological analyser that has been applied to tag Italian tweets. The proposed system ranked third at the EVALITA2016PoSTWITA campaign.
In recent years there were a large number of works
trying to push the accuracy of the PoS-tagging
task forward using new techniques, mainly from
the deep learning domain
In this study, still work-in-progress, we set-up
a PoS-tagger for Italian able to gather the highest
classification performances by using any available
language resource and the most up-to-date DNN.
We used AnIta
The general PoS-tagger has been described in
The set of input features for each token is basically formed by two different components: the word embedding and some morphological information. 2.1
All the embeddings used in our experiments were
extracted from a twitter corpus composed by
200 millions of tokens, belonging to 11
millions of tweets downloaded at the beginning of
2012 (February and March), by using the tool
word2vec1
As described in
With regard to unknown words handling and sentence padding we followed the same procedure for the general tagger described in the cited paper, managing each sentence as one single sequence padded at the borders.
All the experiments presented in this paper has
been performed using Keras2. Keras provides
some basic neural network blocks as well as
different learning procedures for the desired network
1https://code.google.com/archive/p/word2vec/
2https://github.com/fchollet/keras/tree/master/keras
configuration and simple tools for writing new
blocks. In our experiments we used Bidirectional
Long Short-Term Memory - LSTM
Figure 1 shows the DNN structure used in our
experiments.
As for the general tagger
The DNN hidden layers were composed by 256 neurons. 4.2
The usual way to set up an experiment following this suggestions involves splitting the gold standard into three different instance sets: the training set, for training, the validation set, to determine the stopping point, and the test set to evaluate the system. However, we are testing our systems on real evaluation data that has been already word2vec Embed.
Hyperpar. Value type SkipGr. size 100 (1/2) win. 5 neg. sampl. 25 sample 1e-4 iter 15
window 5
batch (win) 1/4*NU batch (seq) 1 Opt. Alg. Adam Loss Func. Categ.CE split by the organisers into development and test set. Thus, we can divide the development set into training/validation set for optimising the hyperparameters and define the stopping epoch, but, for the final evaluation, we would like to train the final system on the complete development set to adhere to the evaluation constraints and to benefit from using more training data.
Having two different training procedures for the optimisation and evaluation phases leads to a more complex procedure for determining the stopping epoch. Moreover, the typical accuracy profile for DNN systems is not smooth and oscillate heavily during training. To avoid any problem in determining the stopping point we smoothed all the profiles using a bezier spline. The procedure we adopted to determine the stopping epoch is (please look at Fig. 2): (1) find the first maximum in the validation smoothed profile - A; (2) find the corresponding value of accuracy on the smoothed training profile - B; (3) find the point in the smoothed development set profile having the same accuracy as in B - C; (4) select the epoch corresponding at point C as the stopping epoch - D. First of all we split the Development Set into a proper training set (109,273 tokens) and a validation set (12,132 tokens) for setting up the entire system, to verify the correctness of the whole tagging process and to derive a first estimate of the tagger performances. We ran some experiments with three different seeds and, after having applied the early stop procedure described above, we derived the optimal stopping epoch to be used for the final testing and the tagging performances on the training/validation pair. Table 2 outlines these results.
(A) (B) (C) Avg.
95.56 95.49 95.53 95.53
12 13 10
We presented two kinds of results for the final evaluation: (1) the first official run was derived by applying the same random seed as the configuration (A), and (2) we submitted also, as an unofficial run, a tagged version obtained by combining all the three configurations using a voting scheme.
In Table 3 we can see our system performances, namely AnIta-BiLSTM-CRF (ABC), compared with all the systems that participated at the PoSTWITA 2016 task. 5
The proposed system for PoS-tagging, integrating DNNs and a powerful morphological analyser, exhibited very good accuracy results when applied to the PoSTWITA task of the EVALITA 2016 campaign.
Looking at the official results, and comparing them with the experiments we devised to set up our system, it is easy to note the large difference in performances. During the setup phase we obtained coherent results well above 95% of accuracy, while the best performing system in the official evaluation exhibit performances slightly above 93%. It is a huge difference for this kind of task, rarely observed in real experiments.
In my opinion there is only one reason that explains this difference in performances: the documents in the test set are not drawn from the same kind of corpus as the development set and this is not a desirable condition unless you explicitly organise a domain adaptation task. The TS, as well as the DS, have been inherited from the SENTIPOLC task of the same evaluation campaign, thus the problem could be the same also for other tasks of the same evaluation campaign.