English. In the context of semantic interpretation of live soccer commentaries in Italian, we propose an annotation schema for relevant events and their argument structure, on whose basis we annotated a reference evaluation corpus. We investigated automatic event classification and used Active Learning to reduce the cost of acquiring domain-specific training data.
This work focuses on understanding the content
of live commentaries of sport games. This form
of written reporting has become very popular in
recent years, and almost every national Italian
online newspaper has a section dedicated to live sport
commentaries. Live commentaries have several
interesting properties: (i) they are short
descriptions of an event written by professionals while
the event is happening; their form is much
simpler than a full spoken running commentary; (ii)
they have a clear and simple structure, typically
based on the timing of the sport event; (iii) they
are often associated with metadata (e.g. La Roma
passa in vantaggio [Roma takes the lead] is
associated with the metadata GOAL); (iv) finally, they
describe visual scenes, which is relevant to
automatic alignment of multimedia content (e.g. align
a sequence of frames in a video with the
corresponding commentary), a topic of emerging
interest in Computational Linguistics (see, for
instance,
In this paper we first define an annotation framework for the semantic interpretation of online soccer commentaries in Italian (Section 3), which includes the detection and classification of relevant events, as well as the identification of their argument structure. Based on this annotation schema, which could also be used for the annotation of tweets or other short online comments, we manually annotated a collection of commentaries in Italian to be used as a gold standard (Section 4). As a first step towards a comprehensive system for automatic interpretation of soccer events we focused on event detection and classification (i.e. event extraction), and used Active Learning to build a training corpus (Section 5). We show that this procedure is very effective, allowing our system to reach an F1 of 77.25, with considerable savings of annotation time (Section 6). 2
Most of the work on event detection and
classification focuses either on the news
Event detection and classification on news has
been of interest for English, Italian and Spanish
in the TempEval evaluation campaigns
The surging interest of the NLP community for event detection and classification in the sport domain, on the other hand, is shown by the hackathon recently organized on extraction of soccer events from Tweets in French, English and Arabic (http://hackatal.github. io/2016/).
Fort and Claveau (2012) present a corpus of match commentaries and transcripts of video commentaries of soccer games in French, which has been annotated with entities (e.g. players, referees), events (e.g. corner, penalty) and some relations (e.g. pass, replace player) and van Oorschot et al. (2012) propose a method to extract relevant events of games in Dutch using the quantity of tweets posted per minute.
Event extraction in the sport domain is even
more important as far as analysis of video
In the domain of automatic alignment of
multimedia content, the analysis of both texts, videos
and audio is necessary, and the research focuses
on the alignment of the events detected in the
three media
In our annotation framework, semantic interpretation of soccer events consists of the following steps: (i) soccer event recognition and classification, (ii) recognition and classification of the entities involved in the soccer event, and (iii) identification of the argument relations between the soccer event and the participant entities. 3.1
Soccer event annotation is inspired by the ItTimeML definition of event and follows its minimal chunk rule, according to which only the head of the event phrase is included in the annotated text span (Caselli et al., 2011a). The main difference with the It-TimeML framework is that we restrict it to verbal and nominal events and to a semantically defined set of relevant events.
In particular, we identified six semantic categories of events relevant to the soccer domain (and a number of sub-categories):
Referee decision includes events that are characterized as such due to a referee’s intervention; examples of subcategories are Yellow card and Offside;
Kick includes events in which the ball is kicked by a player; examples of subcategories are Penalty, Corner, Pass (e.g. apre in (1)), Shot on goal, and Free kick;
Interruption includes events in which a player interrupts the action of the opposing team examples of subcategories are Clearance and Intercept (e.g. anticipato in (1));
Possession includes events where the ball, although moving, does not go from one player to another; as subcategories we find, for example, Dribbling and Holding possession;
Goal includes events where a team scores (we did not devise subcategories for Goal);
No ball includes (i) events where a player doesn’t have the ball (e.g. inserimento in (1)), and (ii) events not involving the ball, such as pushing or knocking to the ground (no subcategorization). (1) 71: Griezmann passa a Pogba che apre per Matuidi, inserimento in area del centrocampista del Psg, che viene anticipato. [Griezman for Pogba who in turn passes to Matuidi, the Psg midfield player makes a forward run for the ball but gets beaten to it] 3.2
In order to annotate entities relevant to the soccer domain, we identified four categories, i.e. Player, Team, Referee, and Coach. Entities include both named entities (e.g. Griezmann and Psg in (1)) and nominal entities (e.g. centrocampista [middle field player] in (1)) and textual span is identified according to the minimal chunk rule (as was done for events). The annotation of the argument structure of an event is performed through the creation of links called ARG rel between each event and its arguments (which can be either entities or events). Inspired by PropBank (Bonial et al., 2010), we also defined four numbered arguments to be assigned to each ARG rel in the form of an attribute: ARG 0 and ARG 1 correspond to the required arguments of a predicate, e.g. agent and patient respectively, while ARG 2 and ARG 3 correspond to arguments that occur with high-frequency for a certain predicate.
In (1), for instance, we have an ARG rel between passa and Griezmann (ARG 0) and an ARG rel between passa and Pogba (ARG 2). 4
Based on the annotation schema described in
Section 3, we manually annotated a corpus of nine
soccer games
We computed the inter-annotator agreement
(IAA) over 46 commentaries annotated by two
annotators (two halves from two different games). In
terms of Dice’s coefficient
In order to extract and classify soccer events in online commentaries, we used a supervised machine learning approach. We had a system for event detection (trained on news articles annotated following It-TimeML) available, which did not perform well on the soccer domain (it obtained an F1 of 40.8 and recall of 50.1 on our reference corpus).
Games Commentaries Commentaries/game Tokens Tokens/com.
Goal Kick Interruption Possession Referee decision No Ball Event Player Referee Coach Team ARG rel
As a consequence, a training corpus specifically developed for this task was needed.
We therefore exploited the TEXTPRO-AL
Active Learning platform
As illustrated in Figure 1, an AL cycle consists of the following steps:4 1. Train a model using the annotated commentaries5 (step 3); 2. Repeat the following cycle until the batch6 is full: (a) Select, from an unlabeled database of commentaries (see Section 5.1), a commentary that matches the first event string in the error queue7 (i.e. the event with the lowest confidence) (step 4); (b) Pre-annotate the example (step 5); (c) Correct the annotation (done manually by an annotator) (step 1); (d) Add the annotated example to the batch (step 2a); 4The AL cycle is repeated until a stopping criteria is verified; for instance, until the system reaches a pre-defined performance.
5At the beginning the training corpus is empty, so the first commentary is randomly selected and added to the batch.
6The batch size was set to 2 for the first 24 examples and then to 10. These values were chosen to enable frequent retraining of the model and an update of the confidence scores and system errors.
7The error queue (or system global memory) contains the history of the system errors corrected by the annotator.
(e) Save in the error queue the annotated events with their model confidence score (step 2b);
Our system is highly customizable: the event detection classification system can easily be substituted by a different system for different classification tasks, like NER and PoS tagging. 5.1
The unlabeled database used in the AL procedure is composed of commentaries of 101 soccer games from DirettaGoal,8 La Repubblica,9 Tuttosport,10 and Eurosport.11 We extracted the online commentaries of all games of the Euro 2016 Cup and of the final 6 rounds of Campionato di Serie A 2015-2016. In total 6,573 commentaries were collected, with 155,005 tokens. 5.2
The error-based selection process exploits the idea that the corrections done by the annotator can be used to select new examples more efficiently. The system has a memory in which the events contained in the checked commentaries are stored, together with the system’s confidence score and the indication of whether the system was right or wrong.
The system for event detection and classification
is based on machine learning, using the SVM
algorithm implemented in TinySVM and included in
Yamcha
The AL system described in the previous section was used by a non-expert annotator who annotated events in soccer commentaries for seven working days. This resulted in a training corpus of 1,377 commentaries, that is, around 200 commentaries per day (see Table 1).
The evaluation of our system was performed by comparing it to the reference annotated corpus described in Section 4. The learning curve in Figure 2 represents the results obtained by the system in terms of precision, recall and F1-measure as the training set was progressively extended. At the beginning the training set was empty, so the performance of the system was null. After the annotation of 200 commentaries, the system reached 53.27 F1, and after 800 commentaries it obtained 70.94 F1. At the end of our experiment, almost 1,400 commentaries had been annotated and the system’s performance was 76.65 F1 (73.42 of recall and 80.16 of precision). The peak performance is 77.25 F1 and was reached with 1,347 commentaries (i.e. almost 32,000 tokens). 7
We presented a new annotation framework for the interpretation of online soccer commentaries, as 12Referee decision, Kick, Interruption, Possession, Goal, No Ball Event and O for tokens that are not part of an event. 13https://github.com/hltfbk/MT-EQuAl well as the reference annotated corpus we created.14 We also described our system for event extraction from live soccer commentaries in Italian. It exploits the TEXTPRO-AL Active Learning platform, which allowed us to reach a significant F1 (77.25) in seven working days of a non-expert annotator. The annotation was performed for Italian but the method and the annotation schema we devised can be applied to other languages. The only language dependent component is the feature extractor used by the event detection module.
As for ongoing work, we are working at parameter optimization on the Active Learning framework (particularly, we are interested in the relations between the size of the unlabeled dataset, the frequency of the re-training, and the confidence score used by the selection procedure). We also plan to extend the current system by adding the detection of the argument structure of events.
We thank Valentino Frasnelli for his contribution, which consisted of manually annotating the data. This work has been partially funded by the Euclip project, in collaboration with Euregio.15 14Currently the annotated data are not be distributed due to copyright issues.
15http://www.euregio.it notation Tool. In Proceedings of the 8th International Conference on Language Resources and Evaluation (LREC’12), pages 333–338, Istanbul, Turkey, May. European Language Resources Association (ELRA).
Steven Bethard. 2013. Cleartk-timeml: A minimalist approach to tempeval 2013. In Proceedings of the Seventh International Workshop on Semantic Evaluation, SemEval ’13, Atlanta, Georgia, USA. Claire Bonial, Olga Babko-Malaya, Jinho D. Choi, Jena Hwang, and Martha Palmer. 2010. Propbank annotation guidelines, version 3.0. Technical report, Center for Computational Language and Education Research, Institute of Cognitive Science, University of Colorado at Boulder. http://clear.colorado.edu/compsem/ documents/propbank_guidelines.pdf. Romain Cabasson and Ajay Divakaran. 2003. Automatic extraction of soccer video highlights using a combination of motion and audio features. In Storage and Retrieval for Media Databases 2003, Santa Clara, CA, USA, January 22, 2003, pages 272–276. Tommaso Caselli, Valentina Bartalesi Lenzi, Rachele Sprugnoli, Emanuele Pianta, and Irina Prodanof. 2011a. Annotating Events, Temporal Expressions and Relations in Italian: the It-TimeML Experience for the Ita-TimeBank. In Linguistic Annotation Workshop, pages 143–151.