This is an overview of the Workshop on Semantic Analysis at the SEPLN congress held in Sevilla, Spain, in September 2018. This forum proposes to participants four di erent semantic tasks on texts written in Spanish. Task 1 focuses on polarity classi cation; Task 2 encourages the development of aspect-based polarity classi cation systems; Task 3 provides a scenario for discovering knowledge from eHealth documents; nally, Task 4 is about automatic classi cation of news articles according to safety. The former two tasks are novel in this TASS's edition. We detail the approaches and the results of the submitted systems of the di erent groups in each task.
The Workshop on Semantic Analysis at the SEPLN1 (in Spanish Taller de Analisis Semantico en la SEPLN, TASS) is the evolution of the Workshop on Sentiment Analysis at the SEPLN which is being held since 2012.
1http://www.sepln.org/workshops/tass The aim of the workshop is the furtherance of the research in semantic tasks on texts written in Spanish, roughly speaking in Spanish data. The edition 2018 has proposed two new challenges (Tasks 3 and 4), and provided several linguistic resources.
The processing of health data is attracting the attention of the Natural Language Processing (NLP) research community (Denecke,
Copyright © 2018 by the paper's authors. Copying permitted for private and academic purposes. 2015). In this line, Task 3 proposes modelling the human language in a scenario in which Spanish electronic health documents could be machine readable from a semantic point of view. This Task 3 consists of detecting and classifying concepts for semantic relating them. Task 4 is related to the brand safety concept, which is crucial for the reputation of a brand or the company of the brand. Task 4 proposes the classi cation of the level of safety of a news for the publication of a ads spot of a brand according to the headline of that news.
Tasks 3 and 4 provided speci c datasets
for accomplishing the proposed challenge,
and are described in Sections 2.3.1 and 2.4.1
respectively. Task 1 provided an extension of
the InterTASS corpus, that was presented in
the edition of 2017
The paper is organised as follows: Section 2 describes all the tasks proposed in the edition of year 2018. The speci c details of each Subtask are in Section 2.1, 2.2, 2.3 and 2.4 respectively. Section 3 exposes the conclusions of the paper. 2
Spanish Semantic Analysis
Tasks As mentioned before, TASS is a relevant workshop for semantic analysis tasks, particularly for Spanish. In 2018, new resources and challenges were introduced to evolve Sentiment Analysis systems to a semantic level. In the last editions, several research groups from di erent countries, like Uruguay or Costa Rica, presented their systems, and it was mandatory to make an e ort to build adequate resources for their languages.
In addition, society and companies are interested in new speci c challenges, and for this reason new tasks arise, while maintaining the main task (global polarity).
In this Section, we describe the four tasks of the edition of 2018, namely Section 2.1 expose the details of Task 1; Section 2.2 describes the corpus and the systems submitted to Task 2; Section 2.3 is focused on the Task 3; and Section 2.4 describes all details of Task 4. 2.1 Task 1 This task focused on the evaluation of polarity classi cation systems at tweet level of tweets written in Spanish.
The submitted systems had to face, as usual, the lack of context due to length of tweets written in an informal language with misspelling or emojis, even onomatopeias.
But this edition brought new challenges to this task:
Multilinguality: training, tests and development corpus contain tweets written in Spanish from Spain, Peru and Costa Rica.
Generalization: Several corpora have been used. One of them is the development set, so it follows a similar distribution. The second corpus is the test set of the General Corpus of TASS, which was compiled some years ago, so it may be lexically and semantically di erent from the training and development data.
Furthermore, the system will be evaluated with test sets of tweets written in the Spanish language spoken in di erent
American countries.
The General Corpus of TASS has been
provided in the same way as previous
editions. Further details in
However, International TASS Corpus (InterTASS) is a corpus released in 2017 that has been updated for this edition with new texts. It is composed of tweets written in di erent varieties of Spanish (for Spain, Peru and Costa Rica), so it exhibits a large amount of lexical and even structural di erences in each variant. The main purpose of compiling and using an inter-varietal corpus of Spanish for the evaluation tasks is to challenge participating systems to cope with the many faces of this language worldwide.
Datasets were annotated with 4 di erent polarity labels positive, negative, neutral and none), and systems had to identify the orientation of the opinion expressed in each tweet in any of those 4 polarity levels.
The Spanish variety part was released in
2017 and its description can be found in
The Peru and Costa Rica varieties have been released for this edition. The tweets distributions are shown in Tables 2 and 3 respectively for both variants.
Four sub-tasks were proposed, working with the datasets of the di erent countries:
Subtask-1: Monolingual ES. training and test were the InterTASS ES datasets.
Subtask-2: Monolingual PE. training and test were the InterTASS PE datasets.
Subtask-3: Monolingual CR. training and test were the InterTASS CR datasets.
Subtask-4: Cross-lingual. The training could be done with any dataset, but using a di erent one for the evaluation, in order to test the dependency of systems on a language.
Results were submitted in a plain text le with the following format: t w e e t i d n t p o l a r i t y
Accuracy and the macro-averaged versions of Precision, Recall and F1 were used as evaluation measures. Systems were ranked by the Macro-F1 and Accuracy measures.
For task 1 ve system were presented. Most of them make use of deep learning algorithms, combining di erent ways of obtaining the word embeddings.
INGEOTEC.
RETUYT-InCo.
ITAINNOVA. Montanes, Aznar, and del Hoyo (2018) analyse the use of convolutional network models (CNN), LSTM, Bidirectional LSTM (BiLSTM) and a hybrid approach between CNN and LSTM. The combination CNN-LSTM has been chosen as it integrates the bene ts of both models. They choose the CNN-LSTM combination because it integrates the bene ts provided from both models. ELiRF-UPV. Gonzalez, Hurtado, and Pla (2018b) explore di erent approaches based on Deep Learning. Speci cally, they study the behaviour of the CNN, Attention Bidirectional Long Short Term Memory (Att-BLSTM) and Deep Averaging Networks (DAN). In order to study the behaviour of the di erent models, they carry out an adjustment process. They get the best results in InterTASS-ES. However, linguistic variability a ects the choice of architecture and its hyperparameters, so the application of the same system to InterTASS-CR and InterTASS-PE tasks, without making any adjustment, has not allowed to obtain results as competitive as in InterTASS-ES.
ATALAYA. Luque and Perez (2018) presented a system that uses a weighted scheme to average the subword-aware embeddings obtained from preprocessed tweets that have been enriched with data obtained from machine translation. This novel solution involves translating tweets into another language and back into the source language, to lexically and grammatically increase them.
Tables 4, 5 and 6 show the results obtained in the monolingual subtasks (Spain, Costa Rica and Peru variants).
For the cross-lingual runs, the participants selected an InterTASS dataset to train their systems and a di erent one to test, in order to test the dependency of systems on a language. Tables 7, 9 and 8 show the results obtained in these cross-lingual subtasks.
The overall results, in terms of F1, obtained with the monolingual and multilingual systems for the Spanish and Costa Rica collections are quite comparable, but the one with the Peru collection fall by around 10%. 2.2
Task 2
Task 2, Aspect-based Sentiment Analysis,
proposes the development of aspect-based
polarity classi cation systems. Similar to
previous editions
The Social-TV corpus was collected during
the 2014 Final of \Copa del Rey"
championship in Spain. After ltering out
useless information, a subset of 2,773 tweets
was obtained. The details of the corpus
are described in
STOMPOL (corpus of Spanish Tweets for
Opinion Mining at aspect level about
POLitics) is a corpus for the task of Aspect Based
Sentiment Analysis. The corpus is composed
of 1,284 tweets manually annotated by two
annotators, and a third one in case of
disagreement. The details of the corpus are
described in
Only the research group ELiRF
Participants were expected to submit up to 3 experiments for each provided collection, each in a plain text le with tweet identi cation, aspect and polarity.
For evaluation, exact match with a single
label combining \aspect-polarity" was used.
Similarly to Task 1, the macro-averaged
ver2.3 Task 3
NLP methods are increasingly being used to
mine knowledge from unstructured content of
health
In order to capture the semantics of a broad range of health related text, eHealthKD proposes the identi cation of two types of elements: Concepts and Actions. Concepts are key phrases that represent actors or entities which are relevant in a domain, while Actions represent how these Concepts interact with each other. Actions and Concepts can be linked by two types of relations: subject and target, which describe the main roles that a Concept can perform. Also, four speci c semantic relations between Concepts are de ned: is-a, part-of, property-of and same-as. Figure 1 provides an example.
To simplify and normalise the extraction process, the overall task is divided into three subtasks:
Subtask A is concerned with the extraction of the relevant key phrases.
Subtask B is concerned with the classication of the key phrases identi ed in 2International Workshop on Semantic Evaluation 3Conference and Labs of the Evaluation Forum
Subtask A as either Concept or Action.
Subtask C is concerned with the discovery of the semantic relations between pairs of entities.
To compute the evaluation metrics for each subtask, we de ne the following sets for comparing the annotations between both the expected output (gold standard) and the actual output in each subtask:
when one gold and one given annotation exactly match.
two key phrases have a non-empty intersection.
C, when an annotation in the gold output is not provided by the system.
C, when an annotation given by the system does not appear in the gold output.
when one assigned label is incorrect. B,
To measure the individual subtasks results as well as overall results, the eHealth-KD challenge proposes three evaluation scenarios.
Scenario 1. The rst scenario requires all subtasks (i.e. A, B and C) to be performed sequentially. The input in this scenario consists of plain text (100 sentences), and participants must submit the three output les corresponding to subtasks A, B and C. In this scenario the overall quality of the participant systems is evaluated. So, a combined micro F1 metric was de ned, taking into account results of the three tasks:
F1ABC
= = = =
PABC + RABC
TABC + 12 PA TABC + PA + MA + IB + MC
TABC + 12 PA TABC + PA + SA + IB + SC CA + CB + CC (1) (2) (3) (4) Scenario 2. In the second scenario only subtasks B and C are performed. Hence, participants receive plain text inputs and the corresponding outputs for subtask A (a different subset of 100 sentences). This scenario allows participants to focus on the key phrases classi cation, without being a ected by errors related to the extraction of key phrases. Like Scenario 1, a combined micro F1 is de ned which takes into account the results for subtasks B and C:
F1BC PBC RBC TBC = = = = 2 PBC RBC PBC + RBC
TBC TBC + IB + MC
TBC TBC + IB + SC
CB + CC Scenario 3. Finally, the third scenario evaluates only subtask C. Participants are provided with plain text inputs and the corresponding output of subtasks A and B (a nal subset of another 100 sentences). In this scenario, the following metric is de ned for evaluation:
F1C PC RC = 2 = =
PC RecC PC + RC
CC CC + SC
CC
CC + MC
For competition purposes, the best system is de ned as the submission that maximises the macro-average F1 across all three scenarios:
F1 =
F1ABC + F1BC + F1C 3
For evaluation purposes, a corpus of healthrelated sentences in Spanish was manually built and tagged. The corpus consists of a selection of articles collected from the MedlinePlus4 website. These les contain several entries related to health and medicine topics, and environmental topics strongly related to health care. Spanish language items were converted to a plain text document, processed, and manually tagged using the Brat Files Sentences Annotations
- Concepts - Actions
- subject - target
- is-a - part-of - property-of - same-as
annotation tool5 by 15 human annotators divided into seven groups. The nal 1; 173 tagged sentences were organised in three collections: training, development and test. Table 12 summarises the main statistics of the corpus. 2.3.2 Analysis of the Results eHealth-KD challenge attracted the attention of a total 31 registered teams of which six of then successfully concluded their participation. Their results are summarised in Table 13. The following tag labels are designed to provide an overview of the main characteristics of each participant system: S: Uses shallow supervised models such as CRF, logistic regression, SVM, decision trees, etc.
D: Uses deep learning models, such as LSTM or convolutional networks.
E: Uses word embeddings or other embedding models trained with external corpora.
K: Uses external knowledge bases, either explicitly or implicitly (i.e, through thirdparty tools).
R: Uses hand crafted rules based on domain expertise.
N: Uses natural language processing techniques or features, i.e., POS-tagging, dependency parsing, etc.
trained on the training corpus, was de ned. This strategy consists of a dummy approach based solely on the text of key phrases. This technique collects all training data and stores three maps: (1) key phrases associated with their most common class (either Concept or Action); (2) pairs of concepts associated with their most common relation; and (3) tuples of <Action,Concept> associated with their most common role. At prediction time, these maps are used to select a key phrase, decide its class, and predict relations and roles.
Once the shared subtask ended, the o cial results were published. However, some participants noticed that their systems provided duplicated outputs on some occasions. These duplicated outputs, even if correct, were being counted as spurious after the rst match. To account for this duplication, the evaluation script was modi ed to remove duplicated outputs from the participants submissions prior to calculating the evaluation metrics. Table 14 shows this second version of the metrics, where variations in scores are highlighted in bold text. This proved not to be a signi cant problem, since only two participants were a ected, and even though their metrics improved marginally, the overall results or the main conclusions of the shared subtask did not change.
The results of this task, eHealth-KD, show that a variety of approaches, on the whole, deal e ectively with the health knowledge discovery problem. However, issues still need to be resolved to obtain highly competitive systems. The best performing submissions include classic supervised learning, deep learning and knowledge-based techniques. In subtask A, the best approach (UC3M) is based on a CRF model with pre-trained embeddings as features. This approach obtains similar scores in subtask B. In general, subtask B appears to be easier than the rest, which is understandable given that there are only two classes and there is a large correlation between word lemmas and their classes (as shown by the relatively high performance of the baseline).
Subtask C, in concordance with Scenario 3, does not exceed 45% in F-score. This reinforces the belief that this task is di cult 7This extracts lexical and syntactic features for each token. Afterwards, it applies a set of handcrafted heuristics for each subtask. to deal with, even after having applied novel approaches (i.e. TALP and LaBDA) based on convolutional neural networks.
The best-performing systems in each scenario highly coincide with all three task results. For Scenario 1, the top performing strategy belongs to UC3M, which achieves the best scores in subtasks A and B, and the overall best result in the shared subtask (averaged across all three scenarios), pretty close to SINAI. Likewise, the best strategy in Scenario 3 corresponds to TALP, which achieves the best score for subtask C. However, for the overall results, other participants such as SINAI and UPF-UPC achieve higher average scores, even though their performance in subtask C and Scenario 3 is practically negligible. In contrast, these teams obtain relatively high scores in subtasks A and B.
The diverse nature and complexity of the three subtasks make it di cult to design a single fair evaluation metric. For this reason, we consider that each system submission gets more accurate results related to the speci c sub-problems that it tackles. Although generalisation across the three subtasks is a desirable characteristic, advances in any particular subtask are also very valuable.
In general, the most competitive approaches in individual subtasks are dominated by state-of-the-art machine learning. In the particular case of subtask C, where modern deep learning approaches seem to outperform classic techniques. In addition, incorporating domain-speci c knowledge provides a signi cant boost to the performance. Most participants use NLP features, either explicitly, or implicitly captured in word embeddings and other representations. An interesting phenomenon is that the best systems in subtask A do not correlate with the best systems in subtask C. This suggests that the optimal approach for either subtask is di erent, giving rise to an interesting research line that would explore integrated approaches to simultaneously solving these three subtasks. The overall results show that general purpose knowledge discovery in domain-speci c documents is potentially a proli c research area, particularly for the Spanish language. We expect similar future initiatives to provide fruitful evaluation scenarios where researchers can deploy techniques from several domains, and compete in friendly contests to improve the state-of-the
UC3My SDEN When news are about natural disasters, readers usually feel negative emotions (sadness, for instance), whereas when those news are about the last championship won by your favourite team, readers feel positive emotions like happiness. Moreover, it is commonly assumed in marketing that emotions aroused in the reader by news articles have an impact in the perception of the advertisements displayed along with those articles. Thus, from that marketing perspective, if a company wants to promote their brand, the ads should better be associated to (i.e., shown with) news that arouse positive emotions.
The objective of Task-4 is to encourage the development of systems that can classify a news article into safe (positive emotions, so safe for ads) or unsafe (negative emotions, so better avoid ads). This task could be considered as a kind of stance classi cation, on the positioning of readers of news contents. The task is a strong challenge because it has to deal with the polarity of feeling (safe vs. unsafe) and to work in combination with a (pseudo) thematic classi cation to be able to determine the meaning of the news. For example, the reduction of tra c accidents has a negative feeling because of the accidents, but the context of reducing the numbers of accidents makes those bad news good news, hence safe news.
The Spanish brANd Safe Emotion corpus (SANSE) corpus was speci cally built for this task. RSS feeds of di erent online newspapers written in di erent varieties of Spanish (Argentina, Chile, Colombia, Cuba, Spain, USA, Mexico, Peru and Venezuela) were collected for over a month. Finally 15,152 articles were captured, containing the URL, the publication date and the headline. News summaries were also collected for several sources, but nally they were discarded to make the dataset consistent and homogeneous.
Then 2,000 articles (L1 subset) were randomly selected and were manually annotated into an emotional categorisation of SAFE or UNSAFE, from the point of view of the general public of each corresponding country. The other 13,152 articles (L2 subset) were not annotated.
Dev. (Spain) Test (Mexico) Test (Cuba) Test (Chile) Test (Colombia) Test (Argentina) Test (Venezuela) Test (Peru) Test (USA)
As the datasets were annotated with two levels of safety: SAFE and UNSAFE, the task can be considered as a binary classi cation task.
The annotation was carried out by two human annotators (the two organisers of the task), and, for those cases with no agreement between the two annotators, a third annotator undid the tie. A safe headline of a news was de ned as an utterance that arises a positive or neutral emotion in the reader and is not related to a controversial topics: religion, extreme wing political topics, or controversial topics (those that arise strong positive emotions to some readers but strong negative emotions to other ones). An unsafe headline was de ned as an utterance that arises negative emotions on the reader.
Some examples in Spanish: As sera el nuevo pan integral en Espan~a, segun una nueva ley en marcha. ! SAFE This will be the new integral bread in Spain, according to a new law underway. Casi 300 municipios de Colombia en riesgo electoral. ! UNSAFE Almost 300 municipalities in Colombia at electoral risk.
The agreement of the annotation was 0.58
according to
The L1 subset was then again divided in three subsets, speci cally: training, development and test. The statistics of the three subsets are in Table 15. 2.4.2
Two subtasks were proposed. Subtask 1 (S1) consists of the classi cation of headlines into safe or unsafe for incorporating an ad of a brand. The evaluation of the systems does not take into account the cultural varieties of
the Spanish language, it thus a monolingual evaluation. In this task, datasets are composed of headlines of news written in di erent version of the Spanish language, but the country of the text is not relevant for this task.
Participants were provided with the training and development subsets of L1 SANSE corpus for building the systems, and two test sets for the evaluation: the test subset of L1 SANSE corpus and the L2 SANSE corpus.
The systems presented were evaluated using the measures of Macro-Precision (M. P.), Macro-Recall (M. R.), Macro-F1 (M. F1) and Accuracy (Acc.).
Subtask 2 (S2) is similar to S1, but in this
case the aim is to evaluate the generalisation
capacity of the submitted systems. For
training their systems, participants were provided
with SANSE subsets with headlines written
only in the Spanish language spoken in Spain.
The test set was composed of headlines
written in the Spanish language spoken in di
erent countries of America. The statistics of
SENSE corpus for S2 are shown in the Table
16.
Task 4 attracted the attention of seven teams,
and most of them participated in both
levels of evaluation of the S1 and in S2. Table
2.4.3 shows the participation of the teams in
each Subtask. Five groups of the seven ones
submitted a system description paper, whose
main features will be detailed as what follows.
INGEOTEC.
S1 L1
S1 L2
ELiRF-UPVy rbnUGRy MeaningCloudy SINAIy lone wolf TNT-UA-WFU
X X X X X X X
X
X
X
X
X
X
X
S2
X
X
X
X
rbnUGR. Rodr guez Barroso, Mart
nezCamara, and
Run 2. It uses a BiLSTM8 layer as encoding layer, and its output is the concatenation of the last vector state of the two LSTM layers.
Run 3. It uses a LSTM layer as encoding layer, and its output is the concatenation of the corresponding output state vector of each input token.
The results show that the systems based on one single LSTM layer perform better than the one based on BiLSTM. Regarding the di erent results in S1 and S2 indicate that the use the entire output of the encoding layer allow to improve the generalisation capacity of the model, because the multilingual evaluation requires a higher generalisation capacity.
MeaningCloud.
SINAI.
The evaluation measures in the two Subtasks were accuracy and the macro-average 8A BiLSTM is an elaboration of two LSTM layers. System INGEOTEC run1 0.794 ELiRF UPV run2 0.787 ELiRF UPV run1 0.795 rbnUGR run1 0.784 MEANING- 0.767 CLOUD run3 rbnUGR run3 0.763 rbnUGR run2 0.774 SINAI 0.733 MEANING- 0.723 CLOUD run2 MEANINGCLOUD run1 of precision, recall and F1, and the systems were ranked according to the value of macroF1. Table 18 show the results reached by each group that submitted the description of their systems in S1 L1, S1 L2 and S2 respectively. 3
Conclusions The edition of TASS 2018 has attracted the participation of 16 systems, and the submission of 15 system description papers. Moreover, we have proposed two new challenges to the international reserach community, which are in line to the requirements of the Industry.
The submitted systems are in the line of the state-of-the-art in other similar workshops, and most of them are grounded in Deep Learning and the use of hand-crafted linguistic features. Therefore, TASS may be considered as a reference forum for setting up the state-of-the-art in semantic analysis in Spanish.
As future work, we plan to enlarge the coverage of the Spanish language of the corpus InterTASS, as well as consolidating the new challenges (Task 3 and Task 4). Moreover, we will keep working in the development of new corpora and linguistic resources for the research community.
Acknowledgments This work has been partially supported by a grant from the Fondo Europeo de Desarrollo Regional (FEDER), the projects REDES (TIN2015-65136-C2-1-R, TIN2015-65136C2-2-R) and SMART-DASCI (TIN2017
Workshop on Semantic Analysis at SEPLN (TASS 2018).
Workshop Proceedings, Sevilla, Spain, September. CEUR-WS.