This article rst introduces the development of deep learning in natural language processing in recent years. Then the related work is brie y explained. We participat in the classi cation tasks of MeOffendEs@IberLEF 2021 Subtask 3: Non-contextual binary classi cation of Mexican Spanish. In this task, our work involves categorizing tweets as o ensive or non-o ensive tweets in the O endMEX corpus. The method in this article rst uses XLM to obtain semantic feature information, and then extracts the features again through convolutional neural networks. Focal Loss is used in the model to improve the classi cation e ect. In this task, our team name is Dong. The precision of our model is 0.6050, the recall is 0.5361, and the F1 score is 0.5685.
In recent years, with the rapid development of Internet-based media, a large
number of websites with social services as the main content have emerged, such
as Facebook, Twitter and Weibo. Their birth makes information dissemination
no longer centered on media, but centered on users. The cost for people to share
and obtain information has become very low, but at the same time o ensive
language is also ooding people's vision. O ensive language is text content that
can irritate individuals or groups, including hate language, personal attacks,
harassment, ridicule, etc. In recent years, research related to speech abuse has
received more and more attention. Preventing the abuse of o ensive language
is of great signi cance to maintaining social harmony. The basis for e ectively
avoiding speech abuse is to quickly, automatically and accurately identify o
ensive language. In this binary classi cation task of MeO endEs@IberLEF 2021
[
Section 2 of this article brie y introduces the progress and current situation of work related to natural language processing. Section 3 describes the data set used in this mission. Section 4 illustrates our method, which describes the preprocessing steps and main structure of the model. Section 5 describes the adjusted hyper-parameters applicable to this model and our experimental results. We summarized our work in Section 6. 2
With the application and development of deep learning in many industrial and
commercial elds and the rapid improvement of computer computing
performance, deep learning models are becoming more and more easily applicable to
various scenarios, including the task of detecting o ensive short texts. Deep
neural networks have a more complex network structure than traditional machine
learning methods, and can dig out high-dimensional feature information from
samples by self-learning the characteristics of samples. In 2017, Thomas
Davidson et al. [
This task is Subtask 3 in MeO endEs@IberLEF 2021. Our task is to classify tweets as o ensive or non-o ensive tweets in the O endMEX corpus. The training data used in our task is provided to each participant by the task organizer. These data are collected from Mexican Spanish on Twitter and have been tagged.
In order to make the model better understand the semantic relationship of sentences during the training phase, we remove punctuation marks, numbers, and emoticons during preprocessing. This approach does not a ect the e ect of the model in this classi cation task, and can reduce the di culty of training the model. BERT inserts an identi er [CLS] and a separator [SEP] into each sentence sample during training. Among them, [CLS] is added at the beginning of the sentence, and [SEP] is added at the end of the sentence. [SEP] separates two sentences, so that the sample forms a structure of [CLS] + sentence + [SEP]. 4.2
Our model includes XLM and CNN models. Since BERT was proposed, it has proved that the pre-training model is very e ective in processing natural language understanding tasks, and it has achieved good results in a variety of natural language processing tasks. We consider that BERT mainly focuses on the natural language processing tasks of a single language, so in this task we used Hugging Face's implementation of XLM model. XLM is a Cross-Lingual version developed by Facebook on the basis of BERT. XLM has the same structure as BERT, so it is also an encoder composed of multiple Encoder structures in Transformer [13].
When training XLM, we use XLM to encode any sentences into a shared embedding space. The input sequence will be masked. The process is to select 15% of the input sequence for random masking, 80% of which are replaced by [Mask], 10% are replaced by random words, and the remaining 10% are still correct words. The Fig.1 below shows the basic structure of the model.
The focus of this classi cation task is to extract the core semantic information contained in the sentence. We input the feature information extracted by XLM into CNN. We initialize it before entering CNN. The CNN layer can e ectively reduce the number of parameters, thereby reducing the di culty of training, and at the same time extract low-level local features into higher-level features. In the convolutional neural network we designed, the Batch size is set to 8, the size of the convolution kernel is 2, 3, 4, and 5, and the dropout is set to 0.2. Dropout is added to reduce the possibility of over tting. In order to accelerate convergence and alleviate the problem of gradient disappearance, we use the Batch Normalization layer to normalize each batch of data. We use the ReLU activation function inside the CNN, and use the sigmoid activation function for the nal output.
When some previous models used the traditional cross-entropy loss function, those models ignored the di erence in the degree of optimization of the model between positive and negative samples. In this task, the proportion of positive samples and negative samples in the training data set is unbalanced. Therefore, this model adoptes the Focal Loss function[14] based on the cross-entropy loss function in ne-tuning. This method can reduce the weight occupied by a large number of simple negative samples in the process of training and optimization, and put the model's focus on sparse di cult samples. This method is added to the classi cation model of this article, and and are set to 2 and 0.75 respectively, which can improve the classi cation e ect.
In this classi cation task, our model is based on PyTorch. After many experiments, the training parameter values are shown in Table 1. For this task, our model is composed of XLM and CNN, and the Adam [15] optimizer is used. In the internal and output of CNN, we use ReLU activation function and sigmoid activation function respectively.
In this task, the scores obtained by our model are shown in Table 2. The organizer's evaluation indicators for this task are precision, recall and F1 score. The precision of our model is 0.6050 (ranked 10th, 0.31 lower than the highest score). The recall is 0.5361 (ranked 9th, 0.16 lower than the highest score). The F1 score is 0.5685 (ranked 9th, 0.14 lower than the highest score).
From Table 2 we can see that although our macro precision is not high compared to model 3, our model has better stability. Comparing madol 1 and model 2, our macro recall and F1 score are relatively low. We plan to use Focal Loss to improve macro recall, but the e ect is limited. From the perspective of the data set, we nd that the amount of o ensive and non-o ensive text is very di erent. In addition, the semantics of some texts are ambiguous, which can cause interference. This situation makes our model more biased towards the side with more data when ne-tuning, and the model may have over- tting during training. These situations ultimately lead to the model's poor performance in the test set.
This article mainly describes the overall idea and optimization scheme of noncontextual o ensive detection and classi cation for Mexican Spanish. In the model implementation, we use the XLM pre-training model as a word vector model, and construct a convolutional neural network in downstream tasks to extract deep semantic information. The Focal Loss function is introduced into the model to improve the classi cation performance. Among these evaluation indicators, this method is not optimal compared with other methods, indicating that this model has certain disadvantages. Although the downstream task captures some local information through CNN, its extraction ability is limited. In the next stage of research, we can add sentence-level attention information to improve the detection results. In addition, we consider adjusting the depth of the CNN in the model and the number of epochs to improve the model's characterization ability. 13. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. In: Proceedings of the 31st International Conference on Neural Information Processing Systems. vol. 30, pp. 5998{ 6008 (2017) 14. Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollar, P.: Focal loss for dense object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence 42(2), 318{327 (2020) 15. Kingma, D., Ba, J.: Adam: A method for stochastic optimization. Computer Science (2014) 16. Miao, Y., Ji, Y., Peng, E.: Application of cnn-bigru model in chinese short text sentiment analysis. In: Proceedings of the 2019 2nd International Conference on Algorithms, Computing and Arti cial Intelligence (2019)