Our team (qu) participate in the classi cation tasks of MeO endEs@IberLEF 2021. The task aims to distinguish o ensive Spanish in online communication. We only participate in subtask 3 which aims to identify the o ensive tweets (non-contextual Mexican Spanish). To solve the problem, we mainly put forward the model (XLM-R combined with LSTM). In our model, the F1 score is 0.6676 and the precision is 0.7433.
With the popularization of the Internet, online communication becomes a important part of life. While social networking brings great convenience to human beings, it also has huge risks and hidden dangers [1]. One risk arises from online comments (o ensive), which may hurt netizens, or even cause long-term harm to the victims, leading them to depression and suicide [2]. Therefore, o ensive comments can be detected and analyzed which have a positive e ect on every Internet user. Many social media and technology companies are studying the identi cation of o ensive speech in the network, and the most important thing is how to improve the e ectiveness of identi cation [3].
The main task of MeO endEs@IberLEF 2021 [4] is to analyze and detect o ensive Spanish on social networks, hoping to nd a better solution to identify o ensive Spanish and its categories, and explore the impact of metadata on the identi cation task [5]. The mission is divided into four subtasks and two di erent Spanish languages (universal Spanish and Mexican Spanish). The goal of subtask 1 is to classify generic Spanish comments (non-contextual) into multiple categories (OFP: O ensive, target is a person. OFG: O ensive, target is a group of people or collective. NOM: Non-O ensive, but with inadequate language. NO: non-o ensive). The goal of subtask 2 is also to do multiple classi cations of comments, but the di erence is that subtask 2 provide metadata. The goal of subtask 3 is to classify Mexican Spanish tweets (non-context) into o ensive and non-o ensive tweets. Subtask 4 is also a dichotomy for Mexican Spanish tweets, but the di erence between subtask 4 and subtask 3 is that subtask 4 will provide metadata. Metadata associated with tweets include date, retweet count, bear count, reply status, quote status and so on [6].
In this competition, we only take part in subtask 3 (Non-contextual binary classi cation for Mexican Spanish). For subtask 3, we mainly use the XLM-R model combined with LSTM (Long-Short Term Memory). The XLM-R model is a combination of the XLM model and RoBERTa model. Relatively speaking, the new model increases the number of model's languages and expands the number of train set, which greatly improve the performance of the downstream tasks. At the same time, the LSTM network [7] can further improve the model accuracy. Finally, we use the k-fold ensemble method to get higher scores.
The structure of the rest in this paper is as follows: The second part is the related research work on the recognition and classi cation of o ensive language. The third part is the relevant description of the data set and the construction of our model. The fourth part is the summary and analysis of the experimental results. 2
The identi cation and classi cation of o ensive language in social media have always existed. As early as 2009, Yin [8] identi ed and classi ed the harassment behaviors that appeared in Web2.0. The author's idea was to extract the features of N-Grams, TF-IDF, semantic and other features from the text at rst, and classi ed the text by SVM (Support Vector Machine). This method achieved excellent results at that time. For now, it has to be admitted that traditional machine learning methods still play a key role in the recognition and classi cation of o ensive language. However, the traditional machine learning method also has a major drawback, which relies excessively on features and model parameters selection. With the development of Deep Learning, more and more neural network models make outstanding achievements in the classi cation of o ensive language. At rst, after Kim [9] applied CNN (Convolutional Neural Networks) to text classi cation, many text recognition models related to CNN are produced in the NLP ( Natural Language Processing
dict.cnki.net). In the eld of text detection and recognition, we have to talk about the LSTM model. RNN (Recurrent Neural Network) [10] plays an important role in the eld of NLP, but it has the shortcoming of long-term dependence, and LSTM is designed for RNN's shortcomings [11]. Chakrabarty's, Zhang's and Srivastava`s experiments prove the signi cant contribution of the LSTM model in text classi cation [12].
In 2018, Google introduced a new NLP model: the BERT model [13]. The BERT model is excellent in many elds of NLP. Subsequently, the BERT model is also used in the recognition and classi cation of o ensive language in social networking. A large number of facts prove that the e ect of BERT model is indeed better than the general model. But in the cross-lingual realm, BERT still has many disadvantages.
Although the BERT model can be trained in multiple languages, it also has this irreparable disadvantage: di erent languages cannot learn from each other and communicate with each other. The XLM o ers its solution for this shortcoming. By training di erent kinds of languages under the same model, the model can absorb the information from di erent languages. [14].
RoBERTa's improvement over traditional BERT is mainly re ected in the
following three aspects: (
In subtask 3, we use the data set provided by MeO endEs@IberLEF 2021 that is collected from Mexican-Spanish data on Twitter, and the data are manually tagged (o ensive and non-o ensive). There are training data (5060 pieces), validation data (76 pieces) and test data (2183 pieces). The ratio of o ensive data items and non-o ensive data items is about 5:2, and the data distribution is uneven. 3.2
Although Spanish is the fourth most spoken language in the world, there are
relatively few resources about it. So we need to use cross-lingual model for this
task. As shown in Figure1 below, the XLM-R model+LSTM model is adopted
in this paper. The experimental steps of this model are as follows:
(
(
(
(
At the same time, we also use the comparison model, only using the XLM-R model, and not combined with LSTM. The experimental steps of this model are as follows: Firstly, we get pooler output (P O) from input data through hidden layer calculation. P O is a two-dimensional vector. Secondly, we transform P O into a three-dimensional vector and input it into Bi-GRU to get the result of a three-dimensional vector. Finally, we transform the result of three-dimensional vector into two-dimensional vector and input it into the classi er and we can get the labels of tweets. The results of the two comparative experiments (the parameters of the two models are consistent) in the development set are shown in the following table 1. As shown in the following table, the model combined with LSTM has more excellent performance than the other. In this article, the training data set only have 5,060 pieces. And the amount of data is relatively small, so we use the K-folding ensemble method to improve the performance of the model. The idea of the K-folding ensemble method comes from k-folding cross-validation. We divide the data set into K in total, use K-1 for training, and use the remaining one as a validation set. Repeat K times in total to ensure that each piece of data can become a validation set. Finally, we will accumulate these K results to nd the average value as the nal result. The K-folding ensemble method is shown in Figure 2.
The purpose of using the K-fold ensemble method is to extract di erent features of all data as much as possible in the case of existing training data, which can e ectively improve the generalization ability of the model. We use validation set to test the e ect of k-fold ensemble method. In the Table 1, compared with the method without k-fold ensemble method, the test result in F1 score increased by 0.0624. 4
4.1
We observe the data set and nd that the data downloaded from Twitter are
not processed and looked very messy. In order to enable the model to obtain
data features and get a good training model, we further process the data. The
speci c process is as follows: (
The platform used in this experiment: Intel CORE i7 CPU (16G), hard disk 1T, NVIDIA RTX 3080Ti. The operating system is Windows 10. The editor is Pychrom2020. The framework of Deep Learning is PyTorch. The setting of hyper-parameters has a great in uence on the performance of the model. The following Table 2 shows the settings of the experimental hyper-parameters: The evaluation criteria of subtask 3 are precision, recall ,and F1 score. In this mission, the partial results and the baseline shown in Table 3 below.
In the Table 3, the results of team vic gomez, team qu and the baseline of
task 3 are included. The team vic gomez win the rst place in the subtask 3.
And our team-name is qu. In the competition subtask 3, the ideal results were
not achieved. We think there may be two main reasons for the poor results: (
In this paper, we mainly propose a model (XLM-R and LSTM) for o ensive detection and classi cation in Mexican Spanish (non-contextual). And the kfold ensemble method is adopted to improve the generalization ability of the model. We think that there is still room for improvement in model performance. In the next research, we will further improve the optimization model (solve the problems caused by data imbalance by over-sampling and under-sampling), and improve the e ciency of recognition and classi cation. 13. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018) 14. Lample, G., Conneau, A.: Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291 (2019) 15. Zampieri, M., Malmasi, S., Nakov, P., Rosenthal, S., Farra, N., Kumar, R.: Semeval-2019 task 6: Identifying and categorizing o ensive language in social media (o enseval). arXiv preprint arXiv:1903.08983 (2019) 16. Majumder, P., Patel, D., Modha, S., Mandl, T.: Overview of the hasoc track at re 2019: Hate speech and o ensive content identi cation in indo-european languages. In: Working Notes of FIRE 2019 - Forum for Information Retrieval Evaluation (2019)