English. This paper reports on a set of experiments with different word embeddings to initialize a state-of-the-art BiLSTM-CRF network for event detection and classification in Italian, following the EVENTI evaluation exercise. The network obtains a new state-of-the-art result by improving the F1 score for detection of 1.3 points, and of 6.5 points for classification, by using a single step approach. The results also provide further evidence that embeddings have a major impact on the performance of such architectures.
Italiano. Questo contributo descrive una serie di esperimenti con diverse rappresentazioni distribuzionali di parole (word embeddings) per inizializzare una rete neurale stato dell’arte di tipo Bi-LSTMCRF per il riconoscimento e la classificazione di eventi in italiano, in base all’esercizio di valutazione EVENTI. La rete migliora lo stato dell’arte di 1.3 punti di F1 per il riconoscimento, e di 6.5 punti per la classificazione, affrontando il compito in un unico sistema. L’analisi dei risultati fornisce ulteriore supporto al fatto che le rappresentazioni distribuzionali di parole hanno un impatto molto alto nei risultati di queste architetture.
Current societies are exposed to a continuous flow
of information that results in a large production of
data (e.g. news articles, micro-blogs, social
media posts, among others), at different moments in
time. In addition to this, the consumption of
information has dramatically changed: more and more
people directly access information through social
media platforms (e.g. Facebook and Twitter), and
are less and less exposed to a diversity of
perspectives and opinions. The combination of these
factors may easily result in information overload and
impenetrable “filter bubbles”. Events, i.e. things
that happen or hold as true in the world, are the
basic components of such data stream. Being able to
correctly identify and classify them plays a major
role to develop robust solutions to deal with the
current stream of data (e.g. the storyline
framework
Event detection and classification has seen a
growing interest in the NLP community thanks to
the availability of annotated corpora
Since the EVENTI campaign, there has been a lack of further research, especially in the application of deep learning models to this task in Italian. The contributions of this paper are the followings: i.) the adaptation of a state-of-the-art sequence to sequence (seq2seq) neural system to event detection and classification for Italian in a single step approach; ii.) an investigation on the quality of existing Italian word embeddings for this task; iii.) a comparison against a state-of-the-art discrete classifier. The pre-trained models and scripts running
eventievalita2014/ the system (or re-train it) are publicly available. 2. 2
We follow the formulation of the task as specified in the EVENTI exercise: determine the extent and the class of event mentions in a text, according to the It-TimeML <EVENT> tag definition (Subtask B in EVENTI).
In EVENTI, the tag <EVENT> is applied to every linguistic expression denoting a situation that happens or occurs, or a state in which something obtains or holds true, regardless of the specific parts-of-speech that may realize it. EVENTI distinguishes between single token and multitokens events, where the latter are restricted to specific cases of eventive multi-word expressions in lexicographic dictionaries (e.g. “fare le valigie” [to pack]), verbal periphrases (e.g. “(essere) in grado di” [(to be) able to]; “c’e`” [there is]), and named events (e.g. “la strage di Beslan” [Beslan school siege]).
Each event is further assigned to one
of 7 possible classes, namely:
OCCURRENCE, ASPECTUAL, PERCEPTION,
REPORTING, I(NTESIONAL) STATE,
I(NTENSIONAL) ACTION, and STATE.
These classes are derived from the English
TimeML Annotation Guidelines
The EVENTI corpus consists of three datasets: the Main Task training data, the Main task test data, and the Pilot task test data. The Main Task data are on contemporary news articles, while the Pilot Task on historical news articles. For our experiments, we focused only on the Main Task. In 2https://github.com/tommasoc80/Event_ detection_CLiC-it2018
3https://sites.google.com/site/
eventievalita2014/file-cabinet
addition to the training and test data, we have
created also a Main Task development set by
excluding from the training data all the articles that
composed the test data of the Italian dataset at the
SemEval 2010 TempEval-2 campaign
Tables 1 and 2 report, respectively, the distribution of the events per token part-of speech (POS) and per event class. Not surprisingly, verbs are the largest annotated category, followed by nouns, adjectives, and prepositional phrases. Such a distribution reflects both a kind of “natural” distribution of the realization of events in an Indo-european language, and, at the same time, specific annotation choices. For instance, adjectives have been annotated only when in a predicative position and when introduced by a copula or a copular construction. As for the classes, OCCURRENCE and STATE represent the large majority of all events, followed by the intensional ones (I STATE and I ACTION), expressing some factual relationship between the target events and their arguments, and finally the others (REPORTING, ASPECTUAL, and PERCEPTION). 3
We adapted a publicly available Bi-LSTM
network with a CRF classifier as last layer
emnlp2017-bilstm-cnn-crf
Wikipedia vs. crawled web document vs. large textual corpora or archives):
Berardi2015 w2v
As for the other parameters, the network maintains the optimized configurations used for the 5Parameters: negative sampling 10, context window 10 6Berardi2015 w2v and Berardi2015 glove uses a 2015 dump of the Italian Wikipedia 7Wikipedia dump not specified.
8https://github.com/facebookresearch/
fastText/blob/master/pretrained-vectors.
md
9Parameters: context window 5.
10http://www.corpusitaliano.it/
Class
OCCURRENCE
ASPECTUAL
I STATE
I ACTION
PERCEPTION
REPORTING
STATE
Overall Events
event detection task for English
Evaluation is conducted using the EVENTI
evaluation framework. Standard Precision, Recall,
and F1 apply for the event detection. Given that
the extent of an event tag may be composed by
more than one tokens, systems are evaluated both
for strict match, i.e. one point only if all tokens
which compose an <EVENT> tag are correctly
identified, and relaxed match, i.e. one point for
any correct overlap between the system output and
the reference gold data. The classification aspect
is evaluated using the F1-attribute score
We approached the task in a single-step by detecting and classifying event mentions at once rather than in the standard two step approach, i.e. detection first and classification on top of the detected elements. The task is formulated as a seq2seq problem, by converting the original annotation format into an BIO scheme (Beginning, Inside, Outside), with the resulting alphabet being B-class label, I-class label and O. Example 1 below illustrates a simplified version of the problem for a short sentence: (1) input problem
Marco (B-STATE j I-STATE j . . . j O) pensa (B-STATE j I-STATE j . . . j O) di (B-STATE j I-STATE j . . . j O) andare (B-STATE j I-STATE j . . . j O) a (B-STATE j I-STATE j . . . j O) casa (B-STATE j I-STATE j . . . j O)
Relaxed Evaluation
F1 F1-class 0.892 0.725 0.882 0.714 0.903 0.756 0.886 0.725 0.883 0.705 0.884 0.685 .
(B-STATE j I-STATE j . . . j O) O 3.1
Results for the experiments are illustrated in
Table 3. We also report the results of the best
system that participated at EVENTI Subtask B,
FBKHLT
The results of the Bi-LSTM-CRF network are
varied in both evaluation configurations. The
differences are mainly due to the embeddings used to
initialize the network. The best embedding
configuration is Fastext-It that differentiate from all
the others for the approach used for generating
the embeddings. Embedding’s dimensionality
impacts on the performances supporting the findings
in
The network obtains the best F1 score, both for
detection (F1 of 0.880 for strict evaluation and
0.903 for relaxed evaluation with Fastext-It
embeddings) and for classification (F1-class of 0.756
for strict evaluation, and 0.751 for relaxed
evaluation with Fastext-It embeddings). Although
FBKHLT suffers in the classification subtask, it
qualifies as a highly competitive system for the
detection subtask. By observing the strict F1 scores,
FBK-HLT beats three configurations
12p-value > 0.005 with McNemar’s test. 13p-value < 0.005 with McNemar’s test. supported by the fact that these results are obtained using a single step approach, where the network has to deal with a total of 15 possible different labels.
We further compared the outputs of the best model, i.e. Fastext-It, against FBK-HLT. As for the event detection subtask, we have adopted an event-based analysis rather than a token based one, as this will provide better insights on errors concerning multi-token events and event parts-ofspeech (see Table 1 for reference). 14 By analyzing the True Positives, we observe that the FastextIt model has better performances than FBK-HLT with nouns (77.78% vs. 65.64%, respectively) and prepositional phrases (28.00% vs. 16.00%, respectively). Performances are very close for verbs (88.04% vs. 88.49%, respectively) and adjectives (80.50% vs. 79.66%, respectively). These results, especially those for prepositional phrases, indicates that the Bi-LSTM-CRF network structure and embeddings are also much more robust at detecting multi-tokens instances of events, and difficult realizations of events, such as nouns.
Concerning the classification, we focused on the mismatches between correctly identified events (extent layer) and class assignment. The Fastext-It model wrongly assigns the class to only 557 event tokens compared to the 729 cases for FBK-HLT. The distribution of the class errors, in terms of absolute numbers, is the same between the two systems, with the top three wrong classes being, in both cases, OCCURRENCE, I ACTION and STATE. OCCURRENCE, not surprisingly, is the class that tends to be assigned more often by both systems, being also the most frequent. However, if FBK-HLT largely overgeneralizes OCCURRENCE (59.53% of all class errors), this corresponds to only one third of the errors (37.70%) in the Bi-LSTM-CRF network. Other notable differences concern I ACTION (27.82% of errors for the Bi-LSTM-CRF vs. 17.28% for FBK-HLT), STATE (8.79% for the Bi-LSTM-CRF vs. 15.22% for FBK-HLT) and REPORTING (7.89% for the Bi-LSTM-CRF vs. 2.33% for FBK-HLT) classes. 4
This paper has investigated the application of different word embeddings for the initialization of a state-of-the-art Bi-LSTM-CRF network to solve the event detection and classification task in Italian, according to the EVENTI exercise. We obtained new state-of-the-art results using the Fastext-It embeddings, and improved the F1-class score of 6.5 points in strict evaluation mode. As for the event detection subtask, we observe a limited improvement (+1.3 points in strict F1), mainly due to gains in Recall. Such results are extremely positive as the task has been modeled in a single step approach, i.e. detection and classification at once, for the first time in Italian. Further support that embeddings have a major impact in the performance of neural architectures is provided, as the variations in performance of the Bi-LSMTCRF models show. This is due to a combination of factors such as dimensionality, (raw) data, and the method used for generating the embeddings.
Future work should focus on the development of
embeddings that move away from the basic word
level, integrating extra layers of linguistic
analysis (e.g. syntactic dependencies)
The author wants to thank all researchers and research groups who made available their word embeddings and their code. Sharing is caring. arXiv preprint