The Google Jigsaw team produced submissions for two of the EVALITA 2020 (Basile et al., 2020) shared tasks, based in part on the technology that powers the publicly available PerspectiveAPI comment evaluation service. We present a basic description of our submitted results and a review of the types of errors that our system made in these shared tasks.
The HaSpeeDe2 shared task consists of Italian
social media posts that have been labeled for hate
speech and stereotypes. As Jigsaw’s participation
was limited to the A and B tasks, we will be
limiting our analysis to that portion. The full details
of the dataset are available in the task guidelines
The AMI task includes both raw (natural
Twitter) and synthetic (template-generated) datasets.
The raw data consists of Italian tweets manually
labelled and balanced according to misogyny and
aggressiveness labels, while the synthetic data is
labelled only for misogyny and is intended to
measure the presence of unintended bias
While Jigsaw has hosted three separate Kaggle
competitions relevant to this competition
The models we build are based on the popular
BERT architecture
In part, our submissions explore the importance
of pre-training
A second question studied in prior work is to
what extent training generalizes across languages
gual teacher model (that is too large to practically serve in production) to a smaller CNN. Using this large teacher model, we initially compared the EVALITA hate speech and stereotype annotations against the teacher model’s scores for different attributes. The results are shown in Figure 1 for the training data. Perspective is a reasonable detector for the hate speech attribute, but performs less well for the stereotype attribute, with the identity attack model performing the best.
Using these same models on the AMI task, shown in Figure 2 for detecting misogyny proved even more challenging. In this case, the aggressiveness attribute was evaluated only on the subset of the training data labeled misogynous. In this case, the most popular attribute of “toxicity” is actually counter-indicative of the misogyny label. The best detector for both of these attributes appears to be the “threat” model.
As can be seen, the existing classifiers are all poor predictors of both attributes for this shared task. Due to errors in our initial analysis, we did not end up using any of the models used for PerspectiveAPI in our final submissions.
ategory C news tweets
Submission 1 2 1 2
hatespeech 0.68 0.64 0.72 0.77
stereotype 0.64 0.68 0.67 0.74
The Jigsaw team submitted two separate submissions that were independently trained for Tasks A and B.
Our first submission, one that did not perform very well, was based on a simple multilingual BERT model fine-tuned on 10 random splits of the training data. For each split, 10% of the data was held out to choose an appropriate equal-error-rate threshold for the resulting model.
The BERT fine-tuning system used the 12 layer
model
As Jigsaw has already developed toxicity models for the Italian language, we initially hoped that these would provide a preliminary baseline for the competition despite the independent nature of the development of the annotation guidelines. Our Italian models score comments for toxicity as well as five additional distinct toxicity attributes: severe toxicity, profanity, threats, insults, and identity attacks. We might expect some of these attributes to correlate with the HaSpeeDe2 and AMI attributes, though it is not immediately clear whether any of these correlations should be particularly strong.
The current Jigsaw PerspectiveAPI models are
typically trained via distillation from a
multilinusing stochastic gradient descent with early
stopping, which is computed using the AUC metric
computed using the 10% held out slice. This
model is implemented using Keras
To create the final submission, the decisions of the ten separate classifiers were combined in a majority voting scheme (if 5 or more models produced a positive detection, the attribute was assigned true).
Our second submission was based on a similar approach of fine-tuning a BERT-based model, but one based on a more closely matched training set.
The underlying technology we used is the same
as the Google Cloud AutoML for natural language
processing product that had been employed in
similar labeling applications
The remaining models built for this
competition and in the subsequent section are based on a
customized BERT 768-dimension 12-layer model
pretrained on 1B user-generated comments using
MLM for 125 steps. This model was then
finetuned on supervised comments in multiple
languages for six attributes: toxicity, severe
toxicity, obscene, threat, insult, and identity hate. This
model also uses a custom wordpiece model
Our hate speech and misogyny models use a fully connected final layer that combines the six output attributes and allows weight propagation through all layers of the network. Fine-tuning continues on using the supervised training data provided by the competition hosts using the ADAM optimizer with a learning rate of 1e–5.
Figure 3 displays the ROC curve for our second submission for each of the news and the tweets datasets as well as for both the hate speech and stereotype attributes.
Our second submission for HaSpeeDe2 consisted of fine-tuning a single model with the provided training data with a 10% held-out set. The custom BERT model was fine-tuned on TPUs using a relatively small batch size of 32. 4.2
AMI Our submissions for the AMI task only considered the unconstrained case, due to the use of pretrained models. All AMI models were finetuned on TPUs using the customized BERT check1.0 0.8 e t a R0.6 e v ii t s o Pe0.4 u r T 0.2 0.00.0 0.2
Tweets Hatespeech 85.5% Tweets Stereotype 82.6% News Stereotype 77.3%
News Hatespeech 75.7% 0.4 0.6 0.8 False Positive Rate 1.0 point and custom wordpiece vocabulary from Section 4.1.2. However, a larger batch-size of 128 was specified. All models were fine-tuned simultaneously on misogynous and aggressive labels using the provided data, where zero aggressiveness weights were assigned to data points with no misogynous labels.
Both submissions were based on ensembles of partitioned models evaluated on a 10% held-out test set. We explored two different ensembling techniques, which we discuss in the next section.
AMI submission 1 does not not include synthetic data. AMI submission 2 includes the synthetic data and custom biasing mitigation data selected from Wikipedia articles. Table 2 clearly shows that the inclusion of such data significantly improved the performance on Task B for submission 2. Interestingly, the inclusion of synthetic and bias mitigation data slightly improved the performance in Task A as well.
Task A B
SubmissSiocnore 1 0.738 2 0.741 1 0.649 2 0.883
The two Jigsaw models ranked in first and second place for Task A. The second submission ranked first among participants for Task B.
Both the first and second submissions for AMI
were ensembles of fine-tuned custom BERT
models constructed from partitioned training data. We
explored two ensembling techniques
Thresholds for the individual models in the majority vote and average ensemble were calculated to optimize for the point on the held-out data ROC curve where jTPR (1 FPR)j is minimized.
The majority voting model performed slightly better for both the misogynous and aggressive task on the held-out sets. As such, both submissions use majority vote.
Using the same configuration as Section 4.1.2, we partitioned the raw training data into ten randomly chosen partitions and fine-tuned nine of these using the 10% held out portion to compute thresholds. No synthetic or de-biasing data was included in this submission.
We include ROC curves for half of these models in Figure 4, to illustrate that they are similar but with some variance when used to score the test data.
1.0
B is not surprising given that no bias mitigating data or constraints were included in training.
In order to mitigate bias, we decided to augment the training data set using sentences sampled from the Italian Wikipedia articles that contain the 17 terms listed in the identity terms file provided with the test set data. These sentences were labeled as both non-misogynous and non-aggressive. 11K sentences were used for this purpose, with the term frequencies summarized in Table 3.
Identity Term donna donne femmine femmina fidanzata nonna mamma casalinga casalinghe compagne compagna mamme fidanzate nonne matrone matrona morosa
The second submission employed the same partitioning of data with a held-out set. However the unconstrained data included the raw training data, the provided synthetic data and our de-biasing term data. As with submission 1, majority vote was used with custom thresholds determined by evaluation on the held-out set.
Our first unconstrained submission for AMI achieved scores of 0.741 for Task A and 0.883 for Task B. 5
We discuss an informal analysis of the errors we observed with each of these tasks. Aside from the typical questions regarding data annotation quality, and the small sample sizes, we observed some particular instances of avoidable errors. 5.1
Our first unconstrained submission using majority vote for AMI achieved scores of 0.738 for Task A and 0.649 for Task B. The poorer score for Task Looking at the largest incongruities as shown in Table 4 it is clear that context, which is unavailable to our models, and presumably to the mod
ID Comment
11355 @user @user @user Giustissimo, non bisogna mai nascondersi nelle ideologie,
sopratutto oggi perche´ non esistono piu`. Sta di fatto, che le cose piu` aberranti
che leggi oggi sui giornali hanno sempre@a@che fare con stranieri... o rom
URL
10803 #Simone di #CasalBruciato, #Roma: “Certi rom sono cittadini italiani, ma non
sono uguali a noi. Uguali non e` il termine piu` giusto da usare”. URL
11288 I SOLDI DEI DISABILI AI MIGRANTI La regione Emilia Romagna destina
la meta` dei fondi destinati alle fasce deboli a progetti per i richiedenti asilo A
Reggio Emilia il 69% delle risorse stanziate sono state utilizzate ai richiedenti
asilo #PRIMAGLIITALIANI URL
10420 #MeNeFottoDi questi sfigati #facciamorete che continuano a giustificare ogni
crimine commesso da immigrati... non fate rete, FATE SCHIFO...
#facciamociFURBI
11189 @user Naturalmente in questo caso alla faccia dei comunisti e dei migranti
stitici!
10483 @user SCHIFOSA IPOCRITA SPONSORIZZI I MUSSULMANI E POI VOI
DARE I DIRITTI ALLE DONNE SI VEDE CHE SEI POSSEDUTA DAL
DIAVOLO SEI BUGIARDA BOLDRINA SAI SOLO PROTESTARE POI
TI CHIEDI PERCH E´ IL VERO ITALIANO TI ODIA PERCH E´ SEI UNA
SPORCA IPOCRITA
1
1
0
0
0
.00003
.00003
erators, is important for determining the author’s
intent. The use of humor and the practice of
quoting text from another author are also confounding
factors. As this task is known to be hard
The BERT model that we fine-tuned for this application is cased, and we see within our errors frequent use of all-caps text. However, lower casing the text has almost no effect on the scores, suggesting that the BERT pre-training has already linked the various cased versions of the tokens in the vocabulary.
We analyzed the frequency of word piece fragments in the data and saw no correlation between misclassification and the presence of segmented words. This suggests that vocabulary coverage in the test set does not play a significant role in explaining our systems’ errors.
Considering the sentence with the high model score for hate speech, several single terms are tagged by the model. For example the term “sfigati” occurs only once in the training data in a sentence that is marked as non-hate speech. However, this term is not in our vocabulary and gets split into pieces “sfiga##ti”, and the prefix “sfiga” appears in two out of three training examples that are marked hate speech—exactly the kind of data sparsity that leads to unwanted bias. Using a larger amount of training data, even if it creates an imbalance, is one way to address this, as we did in the case of the AMI challenge. 5.2
AMI Because we are using ensemble models trained on partitions of the training set, we observe that the highest-scoring test samples that are marked nonmisogynous and non-aggressive, as well as the lowest-scoring misogynous and aggressive comments, vary from model to model. However, we display the most frequently occurring mistakes across all ten ensembles in Table 5.
Regarding the false alarms, these comments appear to be mislabeled test instances, and there is ample support for this claim in the training data.
The first comment combines both uppercase and a missing space. While it’s true that subjunctive mode is not well represented in the training data, lower casing this sentence produces high scores.
This is also the case with the third example. The second error seems more subtle, perhaps an attempt at humor, but one with no salient misogyny terms. 6
Because the identity terms for AMI are not
observed with a high frequency in the test data, we
restrict our analysis to the synthetic data set. We
find wide variation in the performance of our
individual models, with one model exhibiting very
poor performance across the subgroups. The
summary of the AUC measurements for these models
are shown in Figure 5, Figure 6, and Figure 7 using
the technique presented in
Our particular approach to the EVALITA challenges represented an unsurprising application of what has now become a textbook technique: leveraging the resources of large pre-trained models. However, many participants achieved nearly similar performance levels in the constrained task. We regard this as a more impressive accomplishment.
Jigsaw continues to apply machine learning to support publishers and to help them host quality online conversations where readers feel safe participating. The kinds of comments these challenges tagged are some of the most concerning and pernicious online behaviors, far outside of the norms that are tolerated in other public spaces. But humans and machines both still misinterpret profanity for hostility, and tagging humor, quotations, sarcasm, and other legitimate expressions for moderation remain serious problems.
Challenges like the AMI and HasSpeede2 competitions underscore the importance of understanding the relationships between the parties in a conversation, and the participants’ intents. We are greatly encouraged that attributes that our systems do not currently capture were somewhat within the reach of our present techniques—but clearly much work remains to be done.