=Paper=
{{Paper
|id=Vol-2172/p11_meaningCloud_tass2018
|storemode=property
|title=MeaningCloud at TASS 2018: News Headlines Categorization for Brand Safety Assessment
|pdfUrl=https://ceur-ws.org/Vol-2172/p11_meaningCloud_tass2018.pdf
|volume=Vol-2172
|authors=Javier Herrera-Planells,Julio Villena-Román
|dblpUrl=https://dblp.org/rec/conf/sepln/Herrera-Planells18
}}
==MeaningCloud at TASS 2018: News Headlines Categorization for Brand Safety Assessment==
https://ceur-ws.org/Vol-2172/p11_meaningCloud_tass2018.pdf
TASS 2018: Workshop on Semantic Analysis at SEPLN, septiembre 2018, págs. 97-102
MeaningCloud at TASS 2018: News Headlines Categorization for
Brand Safety Assessment
MeaningCloud en TASS 2018: Clasificación de Titulares de Noticias para
Evaluar la Seguridad de Marca
Javier Herrera-Planells, Julio Villena-Román
MeaningCloud LLC
{jherrera, jvillena}@meaningcloud.com
Abstract: This paper describes the participation of MeaningCloud in Task 4 at TASS 2018
(Martínez-Cámara et al., 2018), which is focused on Brand Safety assessment. The
objective of systems is to predict whether ads should be hidden for specific news articles,
depending on the topics covered and potential negative emotions that could be triggered.
Based on the output of our APIs for lemmatization, topics extraction and sentiment
analytics, different Natural Language Understanding techniques combined with Machine
Learning were tested in our experiments. The experiment that achieved the best result
consisted of a Deep Learning algorithm based on Word Embeddings and CNN, trained with
features based on the headlines, plus entity extraction, and topic and sentiment analysis.
Keywords: Brand Safety, Unsafe News, Natural Language Understanding, Machine
Learning, Feature Selection, Deep Learning, MeaningCloud.
Resumen: Este artículo describe la participación de MeaningCloud en la Tarea 4 de TASS
2018 (Martínez-Cámara et al., 2018), que se centra en la evaluación de la seguridad de
marca. El objetivo de los sistemas es predecir si los anuncios deberían ocultarse para
artículos de noticias específicos, dependiendo de los temas tratados y de las posibles
emociones negativas que pudieran desencadenarse. Utilizando la salida de nuestras APIs
para extracción de entidades, lematización, clasificación temática y análisis de sentimiento,
nuestro enfoque se basó en probar diferentes técnicas de comprensión del lenguaje natural
combinadas con aprendizaje automático en diferentes experimentos. El experimento que
alcanzó el mejor resultado ha consistido en un algoritmo de Deep Learning basado en Word
Embeddings más CNN, entrenado con características basadas en el texto de los titulares,
extracción de entidades, análisis temático y de sentimiento.
Palabras clave: Seguridad de Marca, Noticias seguras e inseguridad, lenguaje natural,
aprendizaje automático, selección de características, Deep Learning, MeaningCloud.
combinations of pseudo-thematic classifications
1 Introduction and sentiment analysis are involved. For
In the online advertising context, Brand Safety example, a reduction of traffic accidents implies
refers to practices and tools allowing to ensure a negative feeling because of the mention to car
that an ad will not appear in a context that could accidents, but the reduction in number actually
affect negatively or directly damage the represents good news.
advertiser’s brand. This paper describes the participation of
An article may be considered unsafe for MeaningCloud in the Task 4 Good Or Bad News
advertising if it triggers negative feelings in the of TASS 2018 workshop (Martínez-Cámara et
reader. The creation of a system that detects al., 2018), where prediction models have been
these cases faces some challenges. First, built for the categorization of news articles
different feelings might be triggered in each headlines into two categories (Safe and Unsafe).
reader, depending on their view on topics like In this task, lexical diversity among Spanish and
religion, economy, or sports. In addition, Latin American newspapers is also considered.
ISSN 1613-0073 Copyright © 2018 by the paper's authors. Copying permitted for private and academic purposes.
Javier Herrera-Planells y Julio Villena-Román
Two subtasks had to be fulfilled for this Task In addition to this anonymized version of the
4. The first subtask required and evaluated a headline, the training algorithms were also fed,
training algorithm which was fed with a corpus separately, with the detected entities (Daesh and
of 1500 headlines in Spanish from various Níger).
countries. It was afterwards tested against two
sets of 500 and 15000 headlines in Spanish from 2.1.2 Text Lemmatization
various countries too. The second subtask
Headlines are lemmatized after the previous
evaluated the generalization capacity of the
anonymization. The MeaningCloud
algorithm between Spanish from Spain and
lemmatization, PoS and parsing API has been
Spanish from diverse American countries: the
used for this task. For the previous example, the
training was fed with 250 headlines from
lemmatized form is:
newspapers of Spain and tested against 400
headlines from newspapers of Latin America. vídeo mostrar cómo terroristorganization
The tagged corpus provided was quite well matar a 0 soldado en location.
balanced between training, development and test
sets with respect to country representation 2.1.3 Sentiment Detection
(number of instances), although slightly
unbalanced with a higher number of samples in Sentiment is detected using the MeaningCloud
the Unsafe category (64%). sentiment analysis API. For instance, given the
following raw headline:
2 Our Approach Animales mueren en zoológico de
Our approach is composed by two steps: first, Venezuela por falta de comida.
multiple features are extracted from each For this headline, a global sentiment N+ is
headline. Then, each feature vector is fed to a detected. This score belongs to a scale [P+, P,
machine learning model which finally produces Neutral, N, N+] grading from positive to
the Safe/Unsafe prediction. negative, which we map to a range 0 (P+) to 4
(N+).
2.1 Feature Generation The API also provides token-level sentiment
Features are extracted using the following public (morir with sentiment N+, por falta de comida
APIs in our text analytics platform: entity with a sentiment N) and subjectivity data (in this
extraction and anonymization, lemmatization, case, OBJECTIVE), features which have been
and sentiment and topic detection, omitted in this task.
2.1.1 Entity Extraction and Anonymization 2.1.4 Topic Detection
Entities are detected using the MeaningCloud News topics (thematic categorization) can be
topics extraction API. This service has been used detected either with the MeaningCloud text
with raw headlines as the following one: classification API, using one of the predefined
models such as IPTC for news categorization or
Vídeo muestra cómo Daesh mata a 4 IAB for advertising market, or aggregating the
soldados en Níger. thematic information returned by the topics
For this headline, the entities Daesh and extraction API for each detected entity. For this
Níger are detected along with their classes: task, we used this second approach. For instance,
Organization>TerroristOrganization and for the following raw headline:
Location>Country. En plena distensión por los Juegos
With this information, anonymized versions Olímpicos, Kim Jong-Un invitó al
of the headlines are generated to abstract from presidente de Corea del Sur a Pyongyang.
references to actual entities that could bias the
analysis. Numbers are masked too. For instance: The topics extraction API detects four
entities: Juegos Olímpicos (Event), Kim Jong-un
Vídeo muestra cómo (Person), Corea del Sur and Pyongyang (both
#TerroristOrganization# mata a 0 Location). Two of them provide thematic
soldados en #Location#. information: Juegos Olímpicos belongs to sports
98
MeaningCloud at TASS 2018: News Headlines Categorization for Brand Safety Assessment
and Kim Jong-un belongs to politics. So finally The resulting experiment for this task, trained
sports and politics are selected as topics. with 1500 samples and tested with 500 samples
(L1 corpus), had the following performance:
2.2 Classifiers for Monolingual 71.4% accuracy, 71.7% Macro-F1, 71.3%
Classification (subtask 1) Macro-Precision, and 72.2% Macro-Recall.
The confusion matrix is shown in Table 1. As
2.2.1 Run 1: Machine Learning it can be observed, the main reason for the errors
The starting point of our first experiment was a is the incorrect prediction of Unsafe news as
training set of headlines that were anonymized Safe, accounting for 19% of total errors and 32%
and lemmatized. of the errors in the Unsafe category.
We performed an iteration over all of them,
creating a list of n-grams that are frequently Actual Predicted Count
found in the Unsafe category. Each n-gram was Safe Safe 153 (30% all, 76% Safe)
assigned with a higher score if it appeared more Safe Unsafe 48 (10% all, 24% Safe)
frequently in Unsafe than in Safe Unsafe Unsafe 204 (41% all, 68% Unsafe)
headlines. Some of the top-ranked Unsafe Unsafe Safe 95 (19% all, 32% Unsafe)
n-grams in the list after the training process were Table 1: Run 1, L1 corpus confusion matrix
morir, denunciar, asesinar, caso de corrupción
and the placeholder for the anonymized entity
terroristorganization. These n-grams are then 2.2.2 Run 2: Extended Features
used to generate the features for each headline. This experiment follows the same principle as
One headline is represented by the following the first one: we generate a n-gram score list in
feature vector: our training step, and feature vectors use these
• The sum of the scores of Unsafe scores along the sentiment and topic statistics.
unigrams found in the headline, weighted This second experiment extends the n-gram
to the length of the headline in words. score features by using extra n-gram lists. These
• Scores for Unsafe bigrams, trigrams and additional lists are generated using the non-
4-grams separately, in the same way as anonymized version of the headlines. Some of
unigrams. the top scoring resulting n-grams are FARC,
• The sentiment score (0 to 4), extracted as Jones Huala or caso de Edu Saettone.
described in section 2.1.3. Using this information, which is derived from
• Features for each of the most frequent non-anonymized entities, becomes helpful when
topics, such as sports, politics or religion, categorizing headlines within a similar period.
extracted as described in section 2.1.4. This approach resulted in an increase of
performance over the previous experiment:
The most informative features, as shown by 73.2% accuracy, 72.5% Macro-F1, 72.3%
the Extra-Trees algorithm (Geurts et al., 2006), Macro-Precision, and 72.7% Macro-Recall.
were the following: unigram scores (43%), The confusion matrix is shown in Table 2.
bigram scores (21%), sentiment scores (19%), The detection of Unsafe category has noticeably
trigram scores (6%), 4-gram scores (2%), improved, though the accuracy of Safe category
politics topics (2%), football topics (1%) and has decreased.
economy topics (1%).
Then, several machine learning algorithms Actual Predicted Count
have been tested for making predictions on these Safe Safe 141 (28% all, 70% Safe)
feature vectors, including: KNN (Altman, 1992), Safe Unsafe 60 (12% all, 30% Safe)
random forests (Breiman, 2001), multilayer Unsafe Unsafe 225 (45% all, 75% Unsafe)
perceptron, logistic regression, SVM (Vapnik et Unsafe Safe 74 (15% all, 25% Unsafe)
al., 1995), XGBoost (Chen et al., 2016), and
AdaBoost (Freund et al., 2003). Table 2: Run 2, L1 corpus confusion matrix
The accuracy of the different algorithms was
evaluated using development set. XGBoost was 2.2.3 Run 3: Deep Learning
finally chosen as the top-performant algorithm
for this experiment. The final settings were a tree This experiment, opposite to the previous ones,
booster, learning rate of 0.1, minimum child feeds the machine learning algorithm with a set
weight of 1 and maximum depth of 3. of words/tokens. A deep learning model based
99
Javier Herrera-Planells y Julio Villena-Román
on word embeddings and a convolutional neural high increase in performance: 77.6% accuracy,
network is then used for making predictions. 76.7% Macro-F1, 76.7% Macro-Precision, and
The following headline will be used for 76.7% Macro-Recall.
describing the process used in this experiment: Table 3 again shows the confusion matrix.
Al menos 25 civiles muertos deja ataque
Actual Predicted Count
contra el Daesh en Siria.
Safe Safe 145 (29% all, 72% Safe)
The same features as described in the Safe Unsafe 56 (11% all, 28% Safe)
previous experiment are generated: preprocessed Unsafe Unsafe 243 (49% all, 81% Unsafe)
text, sentiment, topics and entities. All of them Unsafe Safe 56 (11% all, 19% Unsafe)
are encoded as a set of tokens:
Table 3: Run 3, L1 corpus confusion matrix
al menos 0 civil muerto dejar ataque
contra el terroristorganization en There is an improvement in both classes from
location entdaesh entsiria sentiment3 the previous experiments, most noticeable in the
topicpolitics Unsafe category. This confusion matrix is the
best among the previous ones if we take into
The information contained in these tokens is
account the risk of considering an Unsafe article
the following:
as Safe. In this case, ads would be shown by
• Anonymized and lemmatized text: al mistake.
menos 0 civil muerto dejar ataque contra
el terroristorganization en location. 2.2.4 Overall Results
• Non anonymized entities: entdaesh,
Table 4 shows the overall results for subtask 1 of
entsiria.
our three experiments, sorted by Macro-F1
• Sentiment: sentiment3, meaning a
which is the comparison metric among
sentiment with score 3 (negative, N).
participants.
• Topics found: topicpolitics.
Afterwards, a deep learning model developed Run Id Macro-F1
using the Keras framework (Chollet et al., 2015) Run 1 0.717
is trained on this set of tokens. The Run 2 0.725
implementation of the model for this experiment Run 3 0.767
has the following settings: Table 4: Overall results, L1 corpus
• Input sequences with length of 23 words
(two times 11.5, the average word Next table 5 shows the final ranking in terms
length), padded for shorter texts with of Macro-F1 for the best run by all participants,
PAD placeholders at the end. sorted by Macro-F1. Our best experiment ranked
• Embedding generation: 300-dimensional 4th among 7 participants.
embeddings for the 2241 most frequent
words (two thirds of the total 3362 Group Macro-F1
different words). UNK placeholder for INGEOTEC 0.795
words out of selected vocabulary. ELiRF-UPV 0.790
• Convolutional neural network, rbnUGR 0.774
calculating a convolution with 3 different MEANINGCLOUD 0.767
region sizes (Zhang and Wallace, 2015) SINAI 0.728
lone_wolf 0.700
and 2 filters for each region size. Kernel
TNT-UA-WFU 0.492
size of {3,4,5}x300, ReLU activation
function (Nair and Hinton, 2010) and Table 5: Subtask 1, L1 corpus team ranking
max-pooling strategy.
• Two final densely-connected layers with Finally, the results over the L2 corpus
a dropout of 0.25 (Srivastava et al., (including 13 152 headlines), tagged by pooling
2014). The second layer acts as the submissions and based on the vote of majority,
output layer using a Softmax function. are shown in Table 6. Our best experiment
ranked 4th again among all participants. The
The resulting model, trained with 1500
improvement of results with respect to the other
samples and tested with 500 samples, showed a
corpus (L1) may be not real because of the
100
MeaningCloud at TASS 2018: News Headlines Categorization for Brand Safety Assessment
pooling (the decision of the majority may be 3 Conclusions
wrong anyway).
In this paper we described a system for detecting
Group Macro-F1
headlines of news articles that might be unsafe
ELiRF-UPV 0.883
for advertising. We have incorporated different
rbnUGR 0.873 preprocessing techniques, such as text
INGEOTEC 0.866 lemmatization, entity extraction and
MEANINGCLOUD 0.793 anonymization, topic detection and sentiment
SINAI 0.773 analysis.
TNT-UA-WFU 0.544 Then, we have evaluated several
classification algorithms, from n-gram scoring to
Table 6: Subtask 1, L2 corpus team ranking embeddings and deep learning models.
Three techniques were found to significantly
2.3 Classifiers for Multilingual improve the accuracy of the model: providing
Classification (subtask 2) both the anonymized text and the non-
anonymized entities separately, include the
Run 3 was, apparently, the top performant sentiment pre-detection and the use of a deep
among the other experiments in subtask 1, so it learning approach for the model training.
was selected for subtask 2. The model was
trained with 250 headlines from newspapers of Disclaimer
Spain, and tested against 408 headlines from
newspapers of America. MeaningCloud is one of the co-organizers of
The results were the following: 65.8% TASS since the first edition in 2012, and,
accuracy, 65.1% Macro-F1, 64.7% Macro- specifically this year, of Task 4 Good Or Bad
Precision, and 65.4% Macro-Recall. News. Our participation in this task has been
The confusion matrix is shown in Table 7. completely blind, without making use of any
Obviously, results are considerably worse than information or dataset not provided to the rest of
in the first task, as the information available for the participants.
training is extremely reduced. We are also sponsoring TASS 2018 with
prizes for the best teams. Obviously, as insiders,
Actual Predicted Count we were never eligible for the prize, should our
Safe Safe 99 (24% all, 63% Safe) experiments had been the top-performant.
Safe Unsafe 57 (14% all, 37% Safe)
Unsafe Unsafe 169 (42% all, 67% Unsafe) References
Unsafe Safe 84 (20% all, 33% Unsafe) Altman, N. S. 1992. An Introduction to Kernel
Table 7: Run 3 subtask 2 confusion matrix and Nearest-Neighbor Non-Parametric
Regression. The American Statistician, 46(3),
Finally, Table 8 shows the ranking in terms 175-185.
of Macro-F1 in subtask 2 for the best run by all Breiman, L. 2001. Random Forests. Machine
participants, sorted by Macro-F1. Our best learning, 45(1), 5-32.
experiment ranked 4th among 5 participants.
Chen, T., and C. Guestrin. 2016. Xgboost: A
Group Macro-F1 Scalable Tree Boosting System. In
INGEOTEC 0.719 Proceedings of the 22nd ACM SIGKDD
ELiRF-UPV 0.699 International Conference on Knowledge
rbnUGR 0.683 Discovery and Data Mining (pp. 785-794).
MEANINGCLOUD 0.651 ACM.
ITAINNOVA 0.617
Chollet, F. 2015. Keras. GitHub.
Table 8: Subtask 2 team ranking https://github.com/keras-team/keras
Results in this subtask are, as expected, lower Cortes, C., and V. Vapnik. 1995. Support-vector
than for subtask 1, for all teams. The ranking networks. Machine learning, 20(3), 273-297.
among teams stays the same, so, apparently, the Freund, Y., R. Iyer, R.E. Schapire, and Y.
lack of information (or the lack of generalization Singer. 2003. An Efficient Boosting
of the models) affects the same to all groups. Algorithm for Combining Preferences. The
101
Javier Herrera-Planells y Julio Villena-Román
Journal of Machine Learning Research, 4
(Nov), 933-969.
Geurts, P., D. Ernst, and L. Wehenkel. 2006.
Extremely Randomized Trees. Machine
learning, 63(1), 3-42.
Martínez-Cámara, E., Y. Almeida-Cruz, M.C.
Díaz-Galiano, S. Estévez-Velarde, M.A.
García-Cumbreras, M. García-Vega, Y.
Gutiérrez, A. Montejo Ráez, A. Montoyo, R.
Muñoz, A. Piad-Morffis, and J. Villena-
Román. 2018. Overview of TASS 2018:
Opinions, Health and Emotions. In
Proceedings of TASS 2018: Workshop on
Semantic Analysis at SEPLN (TASS 2018).
CEUR Workshop Proceedings, vol 2172,
Sevilla, Spain, September 2018. CEUR-WS.
Nair, V., and G. E. Hinton. 2010. Rectified
Linear Units improve Restricted Boltzmann
Machines. In Proceedings of the 27th
International Conference on Machine
Learning (ICML-10) (pp. 807-814).
Srivastava, N., G. Hinton, A. Krizhevsky, I.
Sutskever, and R. Salakhutdinov. 2014.
Dropout: a simple way to prevent neural
networks from overfitting. The Journal of
Machine Learning Research, 15(1), 1929-
1958.
Zhang, Y., and B. Wallace. 2015. A sensitivity
analysis of (and practitioners' guide to)
convolutional neural networks for sentence
classification. CoRR 2015.
102