This paper describes the participation of the CIMAT-GTO team in the MeO endEs 2021 competition. Our main goal is to evaluate an auxiliary sentence scheme for classi cation with BERT in the o ensiveness identi cation task. The use of the auxiliary sentence has been shown to increase the e ciency of classi ers based on pre-trained BERT models in various tasks. We propose two new approaches to obtain the auxiliary sentence, the objective of the proposals is to remark the available information on the use of the words along the classes in the training corpus. The proposals S2KNNC and S2ChiN use techniques related to Nearest Neighbor and Attribute Selection by Chi-square, respectively, to construct the auxiliary sentence. Our results indicate that the auxiliary sentence scheme allows to improve the performance of the BERT-based classi er or even BERT classi er ensembles.
The violence on social media is clearly manifested in widespread polarization [
To promote the ght against this problem, Plaza-del-Arco et al [
The main goal of this paper is to evaluate an auxiliary sentence scheme for classi cation with BERT in the o ensiveness identi cation task. The recent use of the auxiliary sentence in classi ers based on the ne tuning of pre-trained BERT models has shown increased e ciency in multiple tasks. We propose two new approaches to obtain the auxiliary sentence, these proposals seek to take advantage of the information available in the training corpus on the word´s uses. The S2KNNC approach takes back the training instances where the same words are used in close contexts to be compared and considered as auxiliary sentence for the classi cation model. The S2ChiN presents an instance version with re ned information to the classi er as the auxiliary sentence in order to focus the attention on the most relevant words. Our results indicate that the auxiliary sentence scheme allows to improve the performance of the BERT-based classi er or even BERT classi er ensembles.
The remainder of the paper is structured as follows. Section 2 presents a brief description of the state of the art, it includes the aggressive text identi cation proposals that are closest to our approach and a brief description of the works that have recently been proposed to use the auxiliary sentence scheme in other tasks. Section 3 describes the three auxiliary sentence schemes explored. In Section 4 the experimental results are presented. In Section 5 some ethical issues that concern this work are discussed and Section 6 presents our main conclusions. 2
In the same way as other NLP tasks, the initial approaches to the identi
cation of o ensiveness have proposed the use of classical schemes such as BoW
with TFIDF [
In [12] an ensemble of BETO1 classi ers with ne-tuning and the use of data augmentation is proposed, in [13] the use of automatic translation is proposed in order to be able to alternate the use of pre-trained transformers in Spanish, English or multilingual. The work of Villatoro-Tello et al. [14] proposes a classier whose input is the output probability of the BETO network in addition to attributes obtained by a BoW and some metadata attributes that are speci c to the nature of the social network used in the competition.
The results obtained by the di erent approaches show that the approaches based on BETO classi ers are the most e ective, however, it has been seen some improvement by small assistance incorporating external information in the 1 We note that BETO is a model very similar to BERT, pre-trained in Spanish [18] and made available by the authors at https://github.com/dccuchile/beto training through data augmentation with external corpus or with modi ed instances versions of the same training corpus as adversary augmentation versions or augmentation with related words replaced [12].
In several text classi cation tasks that have been shown to obtain good results using transformers, such as sentiment analysis, topic classi cation, question classi cation and humor prediction, a new scheme has been proposed to help the transformer network by providing additional information. This approach focuses the BERT classi cation model to the new task, taking advantage of the fact that BERT has a 2-sentence input scheme. The second sentence, normally not used in text classi cation, introduces additional or auxiliary information [15{17]. 2.1
Auxiliary sentence schemes The idea behind the BERT auxiliary sentence schemes is to exploit the entire structure used in the BERT pre-training where two sentences are included. BERT simultaneously learns to predict the masked tokens and predict the relationship between the two sentences to infer whether they are consecutive or not (in the corpus in which the BERT language model is trained).
The two-sentence scheme was initially used for tasks where the classi cation problem requires two text sequences such as question answering (Q.A.) or semantic relationship between texts (STS), however, recently this additional entry is being used to give additional information or to guide the BERT model in single sentence classi cation tasks. In these schemes, it is important to de ne how the auxiliary sentence will be constructed in order to correctly orient the BERT model in the new task.
In [16] the auxiliary sentence scheme is used to perform sentiment analysis focused on speci c aspects. The auxiliary sentences proposed are: 1. The use of the auxiliary sentence in the form of a question to guide BERT to focus on the speci c aspect or classi cation's characteristic; the auxiliary sentence has the form: \what do you think of the <aspect>of <object of evaluation>" (e.g. \what do you think of the safety of location-1 ?"). 2. The use of the auxiliary sentence in the form of a complete answer, including the class; the auxiliary sentence has the form: \the polarity of the aspect <aspect>of <object of evaluation>is <class>" (e.g. \the polarity of the aspect safety of location-1 is positive"). 3. The use of an auxiliary sentence that only contains the keywords that BERT is required to focus on, in this case the sentence is not necessarily grammatically correct so it is called a pseudo-sentence; the auxiliary sentence has the form: \<aspect><object of evaluation><class>" (e.g. \safety location1 positive").
In [15] in the spirit of introducing some generalization, an auxiliary sentence is constructed by means of data augmentation techniques in which certain words of the original sentence are replaced.
The auxiliary sentence scheme has allowed to provide additional task information to the pre-trained transformer networks (BERT, RoBERTa or XLNet), in order to improve e ciency in tasks where the ne-tuning is performed. 3
We propose the use of the auxiliary sentence scheme under the successful BETO ne-tuning classi er ensemble [12]. We explore three construction methods of the auxiliary sentence, one taken directly from the proposed schemes in the state of the art and two new approaches. In the following subsections, we describe the three methods to obtain the auxiliary sentences and the proposed BETO ensemble. 3.1
Auxiliary sentence baseline scheme: Basic Question (S2Q) This strategy of auxiliary sentence construction was the direct adaptation of the characteristic question of the classi cation interest as it is proposed in [16]. The auxiliary sentence for the o ensive language detection identi cation case is: "Es ofensivo?"; in English: "Is this o ensive?". 3.2
Auxiliary sentence scheme based on Nearest Neighbor (S2KNNC) Following a similar motivation to the use of auxiliary sentence with augmented instances as in [17], we propose a new scheme that provides to BETO classi er an auxiliary sentence that helps to compare the sentence to classify (S1) with other sentences in which several S1's words are used in similar context and that it is con dent that their use and context make them o ensive or non-o ensive.
To obtain the auxiliary sentence, S2, for a sentence to be classi ed, S1, by the method based on the nearest neighbor (S2KNNC), we follow: Given an input to the classi er, sentence S1, we obtain its auxiliary sentence S2 getting the k-th nearest neighbor sentence with class C (Positive or Negative) in the training corpus. Finally the input for the BETO ne-tuning in S2KNNC method is the pair S1, S2 .
The auxiliary sentence is chosen with a speci c class (known because the sentence comes from the training corpus) and a speci c neighbor number K. The selection of K and class C provides us with di erent alternatives for the auxiliary sentence S2 for the same input instance S1 as we shown in examples of the Table 1.
To obtain the closest neighbor we use a BoW representation with a TFIDF weighting scheme, removing stopwords and using the Euclidean distance. This strategy allow us to obtain auxiliary sentences where the same relevant class words are used2.
The intuitive idea of using the nearest neighbor-based auxiliary sentence is to assist the classi er in learning the task by o ering a very close point of comparison where words are used in a particular class and take advantage of the S1 and S2 sentence relationship pre-training to the new task. 3.3
Auxiliary sentence scheme based on Relevant Information (S2ChiN) In our second proposal for the use of the auxiliary sentence scheme, we construct a pseudo sentence from the most relevant words for the classi cation of the original sentence S1. The S2 auxiliary sentence is a version that lters out the least relevant words to focus the classi er's attention on the most informative words for class prediction purposes.
To obtain the auxiliary sentence, S2, for a sentence to be classi ed, S1, by the method based on Chi-squared information ltering (S2ChiN), we follow: Given an instance of an input sentence to the classi er, sentence S1, we construct its auxiliary sentence S2 after eliminating all the words that are not in the N top of the ranking of the best attributes using chi-square test .
The auxiliary sentence constructed depends directly on the choice of the parameter N that is correlated with the amount of information and words that are preserved, therefore we can construct di erent alternatives of auxiliary sentence S2 for the same input instance S1 by choosing di erent thresholds value N.
The idea behind the use of the relevant information in to the auxiliary sentence is to help the classi er to focus on the words that provide (statistically) the highest con dence of the class. We expect to provide the neural network with a clue of the relevant attributes from the beginning and guide the network weight adjustment process to converge to an optimal point or at least a semi-optimal point better than when this additional information is not taken into account. 3.4
BETO classi er ensemble method
Taking into account the good performance of BETO ensembles [12], for each
strategy used in the auxiliary sentence scheme, an ensemble of classi ers is
generated by using ne-tuning pre-trained BETO model [18]. In our ensemble, the
output probability of each ne-tuning BETO model is used as attributes input
to a SVM3 classi er.
2 It should be noted that experimentally these characteristics achieve the best
performance for classi cation with the Nearest Neighbors method.
3 The linear kernel SVM from the Skit Learn library. This concatenation of methods
is previously used in [
The ensemble with the S2Q method is made up of 10 classi ers with randomly
initialized of the last linear layer of the ne tuning BETO model, all classi ers are
trained trained with the sames pairs S1, S2Q. The ensemble with the S2KNNC
method integrates 18 classi ers, 3 for each parameter con guration (each one
with linear layer random initialization) used in 6 parametric combinations with
K = [
Auxiliary sentence examples In this section we include an example of the auxiliary sentences generated for the same original sentence by the di erent methods. The Table 1 shows the example of the auxiliary sentences generated by the di erent alternatives of generation scheme for the same instance, we can see in S2KNNC with positive and negative classes that the auxiliary sentence give a good sample of the o ensive and non-ofensive use of the word "gorda" (Fat), which is the word with a possible ambiguous use. In the auxiliary sentences generated by S2ChiN, we see that the word "vista" and the whole expression "la vista gorda" are selected, the word "vista" is the most important clue for know that the word "gorda" is part if an non-o ensive expression "hacerse de la vista gorda"4. Therefore, S2ChiN gives the classi er an S2 with ltered key information to identify the non-o ensiveness of the text. 4 Close to the meaning of "turn a blind eye" and nothing related to obesity
Auxiliary sentences by method Parameters Auxiliary sentence (S2)
None "Es ofensivo?"
Class Positive Positive Negative
"gorda pelotuda" "Tu madre es muy gorda" "Ultimamente me estoy haciendo de la vista gorda... Y de la cara, la cintura, los cachetes" "<Eso es un secreto a voces! Solo las autoridades se hacen de la vista gorda. . . "
"la vista" "de la vista gorda" Method S2Q S2KNNC S2KNNC S2KNNC kth NN 1st 3rd 1st S2KNNC 3rd
Negative S2ChiN S2ChiN
N threshold for chi-square 300 1200
In this section we describe the experiments designed to compare the proposed
strategies. There are preliminary evaluations prior to those submitted in the
MeOfendEs evaluation campaign [
Pre-competition results
We perform a strati ed division on the MEXA3T training set [
Method BETO-baseline BETO-baseline S2KNNC S2KNNC S2KNNC S2KNNC S2KNNC S2KNNC S2ChiN S2ChiN S2ChiN S2ChiN S2ChiN S2Q S2KNNC S2ChiN S2ChiN S2Q S2KNNC S2ChiN S2ChiN
In Table 2 we show the F-Score measure of the rst harmonic (M-F1) obtained in the test set of our MEXA3T division. The experimental results show the di erent performances obtained by exploring speci c parameters of each model and the size of the ensemble. From the Table 2 the relevance of the ensemble strategy is observed, it is always better to use SVM. In general SVM allows to get results as good as the best individual component or better. Observing the use of a single auxiliary sentence (without ensemble) of the S2KNNC method, we nd that the auxiliary sentence from the positive class always gets a better performance than the negative one, this phenomenon is probably due to the fact that the positive class is the minority class therefore is more di cult to learn for the classi er and the auxiliary sentences is a good help. From the components of S2KNNC it is surprising that the nearest neighbor (K= 1) is not the be the best component (only for negative class). In the comparison of the S2ChiN components, we see that apparently higher thresholds N allow better performance though, in the ensemble, better results were obtained in a more limited range of information ltering.
In general, it is observed that the addition of extra information included in the auxiliary sentence pushes up the results, however, not all information addition is equally good as evidenced by the fact that the best ensembles are not those with the greater number of components. 4.2
O
cial competition results For the MeO endEs competition all models were trained with the provided set for the competition and the predictions on the test set were submitted. In Table 3 we show the Precision, Recall and F-Score for the positive class reported by the organisers.
From the Table 3 we can see the methods proposed by our CIMAT-GTO group, S2ChiN and S2KNNC. Those are in the fourth and seventh position in the rank of the F-score, the S2ChiN method obtains better performance than the S2KNNC as in the preliminary results (Section 4.1). The S2ChiN method is 3.2% below the best performance "CIMAT" group. An interesting observation is that S2ChiN is the second method with the best Recall (with a value very close to the rst place), an important factor for the o ensive identi cation task because all suspected cases must be attended, as noted in the introduction, omissions could be costly. 5
We nd it necessary to note that this proposed system has not considered offensive expressions such as racism, sexism or other expressions that could o end or harm vulnerable groups in a more serious way. Similarly, it is necessary to note that the evaluation forum makes a distinction between o ensive and vulgar expressions, therefore, groups that may be more sensitive to vulgarity may be o ended by expressions not identi ed as o ensive. Finally, we understand that language is a cultural expression and as such, it is always subject of interpretations that can always be di erent between di erent cultures. We warn that the criteria learned by the systems and used during the analysis should not be taken as a single reference, however, we believe that this reference is very useful for the tools development. Other reference criteria could be considered for the system if it is required in the future for a di erent context. 6
In this work, two new approaches were proposed for the application of auxiliary sentence schemes to the o ensiveness identi cation. We found that the auxiliary sentences scheme help to BETO classi ers in the problem addressed, moreover, it was found that some ways of obtaining auxiliary sentences are better than others. The auxiliary sentence with the ltering of the relevant information were the best option but we found that the ltering parameter N must be selected in a narrow range to obtain a good balance between the relevant information concentration and the lost information. Finally, we discover that the auxiliary statement scheme is a good strategy to introduce additional information to the BETO classi er, however, better ways to build the auxiliary sentence need to be explored to achieve more signi cant improvements in the performance of the identi cation method.
The authors thank CONACYT, INAOE and CIMAT for the computer resources provided through the INAOE Supercomputing Laboratory's Deep Learning Platform for Language Technologies (Laboratorio de Supercomputo: Plataforma de Aprendizaje Profundo) with the project "Identi cation of Aggressive and O ensive text through specialized BERT's ensembles" and CIMAT Bajio Supercomputing Laboratory (#300832). Sanchez-Vega would like to thank CONACYT for its support through projects "Algoritmos de procesamiento del lenguaje natural para la modelacion y analisis de la violencia textual con aplicacion en documentos historicos" (ID. BP-FP-20201015143044227-814705) and "Ciencia de datos aplicado al analisis de expedientes de personas desaparecidas". 12. Mario Guzman-Silverio, Angel Balderas-Paredes, Adrian Pastor Lopez-Monroy: Transformers and Data Augmentation for Aggressiveness Detection in Mexican Spanish, In: IberLEF@SEPLN 2020: 293-302 13. Mircea-Adrian Tanase, George-Eduard Zaharia, Dumitru-Clementin Cercel, Mihai Dascalu: Upb at mex-a3t 2020: Detecting aggressiveness in mexican spanish social media content by ne-tuning transformer-based models, In: IberLEF@SEPLN 2020: 236-245 14. Esau Villatoro-Tello, Gabriela Ram rez-de-la-Rosa, Sajit Kumar, Shantipriya Parida, Petr Motl cek: Idiap and UAM Participation at MEX-A3T Evaluation Campaign. In: IberLEF@SEPLN 2020: 252-257 15. Shanshan Yu , Jindian Su , Da Luo: Improving BERT-Based Text Classi cation With Auxiliary Sentence and Domain Knowledge, In: IEEE Access 7, 176600176612, 2019 16. Chi Sun, Luyao Huang, Xipeng Qiu: Utilizing BERT for Aspect-Based Sentiment
Analysis via Constructing Auxiliary Sentence. In: NAACL-HLT (1) 2019: 380-385 17. Jian Ma, Shu-Yi Xie, Meizhi Jin, Lian-Xin Jiang, Yang Mo, Jian-Ping Shen: XSYSIGMA at SemEval-2020 Task 7: Method for Predicting Headlines' Humor Based on Auxiliary Sentences with EI-BERT. In: SemEval@COLING 2020: 10771084 18. J Canete, G Chaperon, R Fuentes, J Perez: Spanish pre-trained bert model and evaluation data, In: PML4DC at ICLR 2020 19. Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova: BERT: Pretraining of Deep Bidirectional Transformers for Language Understanding. NAACLHLT (1) 2019: 4171-4186