Introduction

The Workshop on Semantic Analysis at the SEPLN 1 (in Spanish Taller de Análisis Semántico en la SEPLN, TASS) is the evolution of the Workshop on Sentiment Analysis at the SEPLN which is being held since 2012.

1 http://www.sepln.org/workshops/tass

The aim of the workshop is the furtherance of the research in semantic tasks on texts written in Spanish, roughly speaking in Spanish data. The edition 2018 has proposed two new challenges (Tasks 3 and 4), and provided several linguistic resources.

The processing of health data is attracting the attention of the Natural Language Processing (NLP) research community (Denecke, 2015). In this line, Task 3 proposes modelling the human language in a scenario in which Spanish electronic health documents could be machine readable from a semantic point of view. This Task 3 consists of detecting and classifying concepts for semantic relating them. Task 4 is related to the brand safety concept, which is crucial for the reputation of a brand or the company of the brand. Task 4 proposes the classification of the level of safety of a news for the publication of a ads spot of a brand according to the headline of that news.

Tasks 3 and 4 provided specific datasets for accomplishing the proposed challenge, and are described in Sections 2.3.1 and 2.4.1 respectively. Task 1 provided an extension of the InterTASS corpus, that was presented in the edition of 2017 (Martínez-Cámara et al., 2017). The main novelty of the new version of InterTASS is the incorporation of tweets written in the Spanish language spoken in Spain and in the several other countries of America. Since the difficulty of Task 2 is high, the organisation proposed the same setting of the task as in previous editions.

The paper is organised as follows: Section 2 describes all the tasks proposed in the edition of year 2018. The specific details of each Subtask are in Section 2.1, 2.2, 2.3 and 2.4 respectively. Section 3 exposes the conclusions of the paper.

Spanish Semantic Analysis Tasks

As mentioned before, TASS is a relevant workshop for semantic analysis tasks, particularly for Spanish. In 2018, new resources and challenges were introduced to evolve Sentiment Analysis systems to a semantic level. In the last editions, several research groups from different countries, like Uruguay or Costa Rica, presented their systems, and it was mandatory to make an effort to build adequate resources for their languages.

In addition, society and companies are interested in new specific challenges, and for this reason new tasks arise, while maintaining the main task (global polarity).

In this Section, we describe the four tasks of the edition of 2018, namely Section 2.1 expose the details of Task 1; Section 2.2 describes the corpus and the systems submitted to Task 2; Section 2.3 is focused on the Task 3; and Section 2.4 describes all details of Task 4.

Task 1

This task focused on the evaluation of polarity classification systems at tweet level of tweets written in Spanish.

The submitted systems had to face, as usual, the lack of context due to length of tweets written in an informal language with misspelling or emojis, even onomatopeias. But this edition brought new challenges to this task:

• Multilinguality: training, tests and development corpus contain tweets written in Spanish from Spain, Peru and Costa Rica.

• Generalization: Several corpora have been used. One of them is the development set, so it follows a similar distribution. The second corpus is the test set of the General Corpus of TASS, which was compiled some years ago, so it may be lexically and semantically different from the training and development data. Furthermore, the system will be evaluated with test sets of tweets written in the Spanish language spoken in different American countries.

The General Corpus of TASS has been provided in the same way as previous editions. Further details in (Martínez-Cámara et al., 2017).

However, International TASS Corpus (In-terTASS) is a corpus released in 2017 that has been updated for this edition with new texts. It is composed of tweets written in different varieties of Spanish (for Spain, Peru and Costa Rica), so it exhibits a large amount of lexical and even structural differences in each variant. The main purpose of compiling and using an inter-varietal corpus of Spanish for the evaluation tasks is to challenge participating systems to cope with the many faces of this language worldwide.

Datasets were annotated with 4 different polarity labels positive, negative, neutral and none), and systems had to identify the orientation of the opinion expressed in each tweet in any of those 4 polarity levels.

The Spanish variety part was released in 2017 and its description can be found in (Martínez-Cámara et al., 2017). Table 1 shows the tweets distribution for training, development (dev.) and test corpora. Subtask-4: Cross-lingual. The training could be done with any dataset, but using a different one for the evaluation, in order to test the dependency of systems on a language.

Results were submitted in a plain text file with the following format: t w e e t i d \ t p o l a r i t y Accuracy and the macro-averaged versions of Precision, Recall and F1 were used as evaluation measures. Systems were ranked by the Macro-F1 and Accuracy measures.

Analysis of the Results

For task 1 five system were presented. Most of them make use of deep learning algorithms, combining different ways of obtaining the word embeddings.

INGEOTEC. Moctezuma1 et al. (2018) present a polarity classification system based on the combination of different labelling systems. The main component is the EvoMSA system, based on genetic algorithms, which combines the outputs of the other systems. EvoMSA is based on the B4MSA system for the adjustment of the different parameters (how the text is normalised, how the tokens are calculated or how the tokens are weighted) and on the EvoDAG program that carries out the classification. As for the input systems, various systems are used based on lexicons of affectivity or aggressiveness. It also uses the algorithm of word embeddings called FastText, using the Wikipedia in Spanish to train it. Vectors are generated for each document and SVM is used for training. Their approach performs better when it is trained with tweets from Spain and test with other Spanish varieties.

RETUYT-InCo. Chiruzzo and Rosá (2018) submitted three approaches: SVM using word embedding centroids and manually crafted features, CNN using word embeddings as input, and Long Short Term Memory (LSTM) using word embeddings, trained with focus on improving the recognition of neutral tweets. In all cases, embedding improves results and LSTM has the best behaviour for neutral tweets. The use of a mixed-balanced training method for the LSTM resulted in a significant improvement in the detection of neutral tweets.

ITAINNOVA. Montanés, Aznar, and del Hoyo (2018) analyse the use of convolutional network models (CNN), LSTM, Bidirectional LSTM (BiLSTM) and a hybrid approach between CNN and LSTM. The combination CNN-LSTM has been chosen as it integrates the benefits of both models. They choose the CNN-LSTM combination because it integrates the benefits provided from both models. (DAN). In order to study the behaviour of the different models, they carry out an adjustment process. They get the best results in InterTASS-ES. However, linguistic variability affects the choice of architecture and its hyperparameters, so the application of the same system to InterTASS-CR and InterTASS-PE tasks, without making any adjustment, has not allowed to obtain results as competitive as in InterTASS-ES.

ATALAYA. Luque and Pérez (2018) presented a system that uses a weighted scheme to average the subword-aware embeddings obtained from preprocessed tweets that have been enriched with data obtained from machine translation. This novel solution involves translating tweets into another language and back into the source language, to lexically and grammatically increase them. Tables 4, 5 and 6 show the results obtained in the monolingual subtasks (Spain, Costa Rica and Peru variants).

For the cross-lingual runs, the participants selected an InterTASS dataset to train their systems and a different one to test, in order to test the dependency of systems on a lan-

Run

M. F1 Acc.

retuyt-lstm-cr-2 0.504 0.537 retuyt-svm-cr-2 0.499 0.577 retuyt-svm-cr-1 0.493 0.567 elirf-cr-run-2 0.482 0.561 retuyt-cnn-cr-1 0.477 0.569 atalaya-cr-lr-50-2 0.475 0.582 ingeotec-run1 0.474 0.522 retuyt-lstm-cr-1 0.473 0.530 retuyt-cnn-cr-2 0.469 0.563 elirf-intertass-cr-run-1 0.463 0.544 atalaya-mlp-300-sentiment 0.439 0.520 atalaya-mlp-ubav3-50-3 0.436 0.560 ingeotec-run1 0.384 0.398 elirf-cr-run-1 0.317 0.288 retuyt-cnn-pe-1 0.472 0.494 atalaya-pe-lr-50-2 0.462 0.451 retuyt-lstm-pe-2 0.443 0.488 retuyt-svm-pe-2 0.441 0.471 ingeotec-run1 0.439 0.447 elirf-intertass-pe-run-2 0.438 0.461 atalaya-mlp-sentiment-ubav3-50-3 0.437 0.520 retuyt-svm-pe-1 0.437 0.474 elirf-intertass-pe-run-1 0.435 0.440 atalaya-mlp-300-sentiment 0.429 0.395 atalaya-mlp-50-sentiment 0.427 0.501 retuyt-svm-pe-2 0.425 0.477 retuyt-cnn-pe-2 0.425 0.477 retuyt-lstm-pe-1 0.419 0.420 elirf-intertass-pe-run-1 0.225 0.210 Table 6: Task 1: InterTASS Monolingual PE guage. Tables 7, 9 and 8 show the results obtained in these cross-lingual subtasks.

The overall results, in terms of F1, obtained with the monolingual and multilingual systems for the Spanish and Costa Rica collections are quite comparable, but the one with the Peru collection fall by around 10%.

Task 2

Task 2, Aspect-based Sentiment Analysis, proposes the development of aspect-based polarity classification systems. Similar to previous editions (Martínez-Cámara et al., 2017), two datasets were used to evaluate the different approaches: Social-TV and STOM-POL. Both datasets were annotated with Run M. F1 Acc.

retuyt-svm-cross-es-2 0.471 0.555 retuyt-lstm-cross-es-2 0.470 0.466 retuyt-svm-cross-es-1 0.464 0.572 retuyt-cnn-cross-es-1 0.450 0.524 retuyt-cnn-cross-es-2 0.448 0.563 ingeotec-run1 0.445 0.530 atalaya-mlp-300-sentiment 0.441 0.485 retuyt-lstm-cross-es-1 0.438 0.498 Participants were expected to submit up to 3 experiments for each provided collection, each in a plain text file with tweet identification, aspect and polarity.

For evaluation, exact match with a single label combining "aspect-polarity" was used. Similarly to Task 1, the macro-averaged ver-sion of Precision, Recall, F1, and Accuracy were considered, and Macro-F1 was used for a final ranking of proposed systems.

Collections

The Social-TV corpus was collected during the 2014 Final of "Copa del Rey" championship in Spain. After filtering out useless information, a subset of 2,773 tweets was obtained. The details of the corpus are described in (Villena-Román et al., 2015;García-Cumbreras et al., 2016;Martínez-Cámara et al., 2017).

STOMPOL (corpus of Spanish Tweets for Opinion Mining at aspect level about POLitics) is a corpus for the task of Aspect Based Sentiment Analysis. The corpus is composed of 1,284 tweets manually annotated by two annotators, and a third one in case of disagreement. The details of the corpus are described in (Villena-Román et al., 2015;García-Cumbreras et al., 2016;Martínez-Cámara et al., 2017).

Results

Only the research group ELiRF (González, Hurtado, and Pla, 2018c) participated in this edition. They explored different approaches based on Deep Learning. Specifically, they studied the behaviour of the CNN, Attention Bidirectional Long Short Term Memory (Att-BLSTM) and Deep Averaging Networks (DAN), similar to the proposal of the team for Task 1. In order to study the performance of the different models, they carried out an adjustment process. Tables 10 and 11 show the results obtained in their experiments.

Run

M. F1 Acc.

ELiRF-UPV-run1 0.485 0.627 ELiRF-UPV-run3 0.483 0.628 ELiRF-UPV-run2 0.476 0.625 , 2018). Over the years, many eHealth challenges have taken place, such as SemEval 2 , CLEF 3 campaigns and others (Augenstein et al., 2017). These tasks have mainly dealt with identification, classification, extraction and linking of knowledge. The Task 3: eHealth Knowledge Discovery (eHealth-KD) proposes modelling the human language in a scenario in which Spanish electronic health documents could be machine readable from a semantic point of view. This task is designed to encourage the development of software technologies to automatically extract a large variety of knowledge from eHealth documents written in the Spanish language.

In order to capture the semantics of a broad range of health related text, eHealth-KD proposes the identification of two types of elements: Concepts and Actions. Concepts are key phrases that represent actors or entities which are relevant in a domain, while Actions represent how these Concepts interact with each other. Actions and Concepts can be linked by two types of relations: subject and target, which describe the main roles that a Concept can perform. Also, four specific semantic relations between Concepts are defined: is-a, part-of, property-of and same-as. Figure 1 provides an example. To simplify and normalise the extraction process, the overall task is divided into three subtasks:

• Subtask A is concerned with the extraction of the relevant key phrases.

• Subtask B is concerned with the classification of the key phrases identified in 2 International Workshop on Semantic Evaluation 3 Conference and Labs of the Evaluation Forum Subtask A as either Concept or Action.

• Subtask C is concerned with the discovery of the semantic relations between pairs of entities.

To compute the evaluation metrics for each subtask, we define the following sets for comparing the annotations between both the expected output (gold standard) and the actual output in each subtask:

Correct matches (C): in all subtasks, when one gold and one given annotation exactly match.

Partial matches (P ): in subtask A, when two key phrases have a non-empty intersection.

Missing matches (M ): in subtasks A and C, when an annotation in the gold output is not provided by the system.

Spurious matches (S): in subtasks A and C, when an annotation given by the system does not appear in the gold output.

Incorrect matches (I): in subtask B, when one assigned label is incorrect.

To measure the individual subtasks results as well as overall results, the eHealth-KD challenge proposes three evaluation scenarios.

Scenario 1. The first scenario requires all subtasks (i.e. A, B and C) to be performed sequentially. The input in this scenario consists of plain text (100 sentences), and participants must submit the three output files corresponding to subtasks A, B and C. In this scenario the overall quality of the participant systems is evaluated. So, a combined micro F 1 metric was defined, taking into account results of the three tasks:

F 1ABC = 2 • P ABC • R ABC P ABC + R ABC(1)P ABC = T ABC + 1 2 P A T ABC + P A + M A + I B + M C

(2)

R ABC = T ABC + 1 2 P A T ABC + P A + S A + I B + S C

(3)

T ABC = C A + C B + C C (4)

Scenario 2. In the second scenario only subtasks B and C are performed. Hence, participants receive plain text inputs and the corresponding outputs for subtask A (a different subset of 100 sentences). This scenario allows participants to focus on the key phrases classification, without being affected by errors related to the extraction of key phrases. Like Scenario 1, a combined micro F 1 is defined which takes into account the results for subtasks B and C:

F 1BC = 2 • P BC • R BC P BC + R BC(5)P BC = T BC T BC + I B + M C (6) R BC = T BC T BC + I B + S C(7)T BC = C B + C C (8)

Scenario 3. Finally, the third scenario evaluates only subtask C. Participants are provided with plain text inputs and the corresponding output of subtasks A and B (a final subset of another 100 sentences). In this scenario, the following metric is defined for evaluation:

F 1C = 2 • P C • Rec C P C + R C(9)P C = C C C C + S C (10) R C = C C C C + M C(11)

For competition purposes, the best system is defined as the submission that maximises the macro-average F 1 across all three scenarios:

F 1 = F 1ABC + F 1BC + F 1C 3 (12)

Corpora

For evaluation purposes, a corpus of healthrelated sentences in Spanish was manually built and tagged. The corpus consists of a selection of articles collected from the Med-linePlus 4 website. These files contain several entries related to health and medicine topics, and environmental topics strongly related to health care. Spanish language items were converted to a plain text document, processed, and manually tagged using the Brat

Analysis of the Results

eHealth-KD challenge attracted the attention of a total 31 registered teams of which six of then successfully concluded their participation. Their results are summarised in Table 13. The following tag labels are designed to provide an overview of the main characteristics of each participant system: S: Uses shallow supervised models such as CRF, logistic regression, SVM, decision trees, etc.

D: Uses deep learning models, such as LSTM or convolutional networks.

E: Uses word embeddings or other embedding models trained with external corpora.

K: Uses external knowledge bases, either explicitly or implicitly (i.e, through thirdparty tools).

R:

Uses hand crafted rules based on domain expertise.

N: Uses natural language processing techniques or features, i.e., POS-tagging, dependency parsing, etc.

Baseline description:

A baseline, trained on the training corpus, was defined. This strategy consists of a dummy approach based solely on the text of key phrases. This technique collects all training data and stores three maps: (1) key phrases associated with their most common class (either Concept or Action); (2) pairs of concepts associated with their most common relation; and (3) tuples of <Action,Concept> associated with their most common role. At prediction time, these maps are used to select a key phrase, decide its class, and predict relations and roles.

Once the shared subtask ended, the official results were published. However, some participants noticed that their systems provided duplicated outputs on some occasions. These duplicated outputs, even if correct, were being counted as spurious after the first match. To account for this duplication, the evaluation script was modified to remove duplicated outputs from the participants submissions prior to calculating the evaluation metrics. Table 14 shows this second version of the metrics, where variations in scores are highlighted in bold text. This proved not to be a significant problem, since only two participants were affected, and even though their metrics improved marginally, the overall results or the main conclusions of the shared subtask did not change.

The results of this task, eHealth-KD, show that a variety of approaches, on the whole, deal effectively with the health knowledge discovery problem. However, issues still need to be resolved to obtain highly competitive systems. The best performing submissions include classic supervised learning, deep learning and knowledge-based techniques. In subtask A, the best approach (UC3M) is based on a CRF model with pre-trained embeddings as features. This approach obtains similar scores in subtask B. In general, subtask B appears to be easier than the rest, which is understandable given that there are only two classes and there is a large correlation between word lemmas and their classes (as shown by the relatively high performance of the baseline).

Subtask C, in concordance with Scenario 3, does not exceed 45% in F-score. This reinforces the belief that this task is difficult to deal with, even after having applied novel approaches (i.e. TALP and LaBDA) based on convolutional neural networks.

The best-performing systems in each scenario highly coincide with all three task results. For Scenario 1, the top performing strategy belongs to UC3M, which achieves the best scores in subtasks A and B, and the overall best result in the shared subtask (averaged across all three scenarios), pretty close to SINAI. Likewise, the best strategy in Scenario 3 corresponds to TALP, which achieves the best score for subtask C. However, for the overall results, other participants such as SINAI and UPF-UPC achieve higher average scores, even though their performance in subtask C and Scenario 3 is practically negligible. In contrast, these teams obtain relatively high scores in subtasks A and B.

The diverse nature and complexity of the three subtasks make it difficult to design a single fair evaluation metric. For this reason, we consider that each system submission gets more accurate results related to the specific sub-problems that it tackles. Although generalisation across the three subtasks is a desirable characteristic, advances in any particular subtask are also very valuable.

In general, the most competitive approaches in individual subtasks are dominated by state-of-the-art machine learning. In the particular case of subtask C, where modern deep learning approaches seem to outperform classic techniques.

In addition, incorporating domain-specific knowledge provides a significant boost to the performance. Most participants use NLP features, either explicitly, or implicitly captured in word embeddings and other representations. An interesting phenomenon is that the best systems in subtask A do not correlate with the best systems in subtask C. This suggests that the optimal approach for either subtask is different, giving rise to an interesting research line that would explore integrated approaches to simultaneously solving these three subtasks. The overall results show that general purpose knowledge discovery in domain-specific documents is potentially a prolific research area, particularly for the Spanish language. We expect similar future initiatives to provide fruitful evaluation scenarios where researchers can deploy techniques from several domains, and compete in friendly contests to improve the state-of-the-

UC3M † SINAI † UPF-UPC † TALP † LaBDA † UH

Task 4

When news are about natural disasters, readers usually feel negative emotions (sadness, for instance), whereas when those news are about the last championship won by your favourite team, readers feel positive emotions like happiness. Moreover, it is commonly assumed in marketing that emotions aroused in the reader by news articles have an impact in the perception of the advertisements displayed along with those articles. Thus, from that marketing perspective, if a company wants to promote their brand, the ads should better be associated to (i.e., shown with) news that arouse positive emotions. The objective of Task-4 is to encourage the development of systems that can classify a news article into safe (positive emotions, so safe for ads) or unsafe (negative emotions, so better avoid ads). This task could be considered as a kind of stance classification, on the positioning of readers of news contents. The task is a strong challenge because it has to deal with the polarity of feeling (safe vs. unsafe) and to work in combination with a (pseudo) thematic classification to be able to determine the meaning of the news. For example, the reduction of traffic accidents has a negative feeling because of the accidents, but the context of reducing the numbers of accidents makes those bad news good news, hence safe news.

Corpora

The Spanish brANd Safe Emotion corpus (SANSE) corpus was specifically built for this task. RSS feeds of different online newspapers written in different varieties of Spanish (Argentina, Chile, Colombia, Cuba, Spain, USA, Mexico, Peru and Venezuela) were collected for over a month. Finally 15,152 articles were captured, containing the URL, the publication date and the headline. News summaries were also collected for several sources, but finally they were discarded to make the dataset consistent and homogeneous.

Then 2,000 articles (L1 subset) were randomly selected and were manually annotated into an emotional categorisation of SAFE or UNSAFE, from the point of view of the general public of each corresponding country. The other 13,152 articles (L2 subset) were not annotated.

As the datasets were annotated with two levels of safety: SAFE and UNSAFE, the task can be considered as a binary classification task.

The annotation was carried out by two human annotators (the two organisers of the task), and, for those cases with no agreement between the two annotators, a third annotator undid the tie. A safe headline of a news was defined as an utterance that arises a positive or neutral emotion in the reader and is not related to a controversial topics: religion, extreme wing political topics, or controversial topics (those that arise strong positive emotions to some readers but strong negative emotions to other ones). An unsafe headline was defined as an utterance that arises negative emotions on the reader. Some examples in Spanish:

Así será el nuevo pan integral en España, según una nueva ley en marcha. → SAFE This will be the new integral bread in Spain, according to a new law underway.

Casi 300 municipios de Colombia en riesgo electoral. → UNSAFE Almost 300 municipalities in Colombia at electoral risk.

The agreement of the annotation was 0.58 according to π (Scott, 1955) and k (Cohen, 1960), which may consider moderate according to Landis and Koch (1977). Although the agreement is moderate, it is close to be considered substantial, and we have also to take into account that it is a new classification task that works with a strong subjective content. We will work in making the annotations guidelines more precise in order to improve the agreement of the annotators. Besides, we hope that the participants will give us insights with the aim of improving the annotation of the data.

The L1 subset was then again divided in three subsets, specifically: training, development and test. The statistics of the three subsets are in Table 15.

Tasks

Two subtasks were proposed. Subtask 1 (S1) consists of the classification of headlines into safe or unsafe for incorporating an ad of a brand. The evaluation of the systems does not take into account the cultural varieties of the Spanish language, it thus a monolingual evaluation. In this task, datasets are composed of headlines of news written in different version of the Spanish language, but the country of the text is not relevant for this task.

Participants were provided with the training and development subsets of L1 SANSE corpus for building the systems, and two test sets for the evaluation: the test subset of L1 SANSE corpus and the L2 SANSE corpus.

The systems presented were evaluated using the measures of Macro-Precision (M. P.), Macro-Recall (M. R.), Macro-F1 (M. F1) and Accuracy (Acc.).

Subtask 2 (S2) is similar to S1, but in this case the aim is to evaluate the generalisation capacity of the submitted systems. For training their systems, participants were provided with SANSE subsets with headlines written only in the Spanish language spoken in Spain. The test set was composed of headlines written in the Spanish language spoken in different countries of America. The statistics of SENSE corpus for S2 are shown in the Table 16.

Results

Task 4 attracted the attention of seven teams, and most of them participated in both levels of evaluation of the S1 and in S2. Table 2.4.3 shows the participation of the teams in each Subtask. Five groups of the seven ones submitted a system description paper, whose main features will be detailed as what follows.

INGEOTEC. Moctezuma et al. (2018) propose an ensemble classification system (EvoMSA), which is composed of several and heterogeneous base systems and a genetic programming system (EvoDAG, (Graff et al., 2017)) that optimises the contribution of each base system in the final classification. The authors combined supervised and unsupervised system as base classification systems. The supervised ones are based on the use of the algorithm SVM with different representations of the input text, namely TF-IDF and pre-trained word vectors. The system reached the best results in the monolingual and the multilingual evaluations, however the performance of the system dropped a bit in S1 L2. Since the annotation test set of S1 L2 was conducted by a voting system of the all the submitted systems, the lower performance in S1 L2 may be caused by a different error distribution between INGEOTEC and the systems submitted by the other groups.

ELiRF UPV. González, Hurtado, and Pla (2018a) propose a deep neural network, specifically the model Deep Averaging Networks (DAN) (Iyyer et al., 2015). The authors used a set of pre-trained word embeddings for representing the news headlines. The set of pre-trained word embeddings were prepared by the authors and built upon a corpus of tweets (Hurtado, Pla, and González, 2017). The high performance reached by a set of pre-trained word embeddings built upon tweets with news headlines stands out, because the genre of news headlines and tweets are different. However, it may mean that the use of language in tweets and news headlines is similar. The results show that the systems based on one single LSTM layer perform better than the one based on BiLSTM. Regarding the different results in S1 and S2 indicate that the use the entire output of the encoding layer allow to improve the generalisation capacity of the model, because the multilingual evaluation requires a higher generalisation capacity. MeaningCloud. Herrera-Planells and Villena-Román (2018) propose three supervised systems, two of them are lineal classification systems and the other one a non-lineal classification system. The linear classification systems use XGBoost (Chen and Guestrin, 2016) as classification system. They differ in the set of features used to represent the news headlines, which are mainly built using the public APIs of the text analytic platform of MeaningCloud. The non lineal classification system is a neural network based on the use of a CNN layer. The proposal that reached higher results was the one grounded in a CNN (Run 3). SINAI. Plaza del Arco et al. (2018) propose to represent the news headlines as a vector of unigrams weighted with TF-IDF, and the number of positive and negative words according to three list of opinion bearing words. The authors used SVM as classification algorithm.

Team

The evaluation measures in the two Subtasks were accuracy and the macro-average