In this paper we describe our participation in the Aggressiveness Detection Track at the third edition of MEX-A3T. We evaluate two strategies for text classification, a traditional classifier (Logistic Regression) and a classifier based on transformers (BETO). We also study the inclusion of social media metadata features to try to get context from authors and text messages. Social media platforms are one of the most popular ways to communicate in the "Information Age", allowing their users to express and spread many kinds of ideas, from the cheerful to the not so positive side of "freedom of speech". These networks aren't immune to people that share ofensive content, users that show malicious intent and are quick to reply with aggressive manners. Anonymity, ease of access and lack of punishment for the most part, encourages these individuals to express themselves ofensively. The volume of messages that are sent daily on social media makes moderation a dificult task to be dealt with by conventional means, and as people increasingly communicate online, the need for high quality automated abusive language classifiers becomes much more profound[ 1]. One of the goals of the third edition of MEX-A3T [2] is to tackle this problem and further improve the research of this important Natural Language Processing (NLP) task, the detection of aggressive tweets in Mexican Spanish. The issue is that spotting ofensive messages and hate speech is challenging because systems cannot rely on the text content [3, 4]; for this reason, in this work we evaluate a strategy to try to give context to short texts from social media by taking into account message and author metadata. Our hypothesis is that these additional attributes are expected to better distinguish between ofensive and not-ofensive messages and improve classification scores.
1. We replaced the string of characters “&" with “&". This was necessary to get a closer representation to the text used in the original tweets. 2. We strip emojis from the tweets. 3. All words were made lowercase.
For LogisticRegression classifier, we tokenized the dataset using the TweetTokenizer utility
from NLTK [
We conducted our research using the following features: Lexical: We use word n-grams (n=1, 3) as features, this collection of terms is weighted with its term frequency (TF).
Metadata (MD): By using the Standard Twitter API platform [
Logistic Regression (LR): Considered part of the traditional approaches for most of the NLP tasks. This algorithm uses a linear regression equation that includes a function called “logistic/sigmoid function”, this function produces an “S” shaped curve that is able to tell the probability of class assignment.
BETO: BERT model trained on a big Spanish corpus. BETO [
Because of its computational afordability and flexibility at handling diferent types of inputs (including NULL values), XGBoost Classifier was selected as the blender of the information present in our proposed systems. The first step in our framework involves feeding the text part of the dataset to either LR or BETO, then the classifier returns class probabilities, and lastly these predictions are concatenated with either metadata or NaN values (for the tweets that we couldn’t retrieve their metadata online) to form the input vector for XGBoost, which outputs the final decisions.
The datasets were provided by MEX-A3T Team. Table 3 shows the distribution of training and test partitions for Spanish tweets.
Our first task was to "extend" the dataset. This process is simple: for every message in the collection a query is made using the text to search for that tweet online. Due to the nature of the task at hand, some tweets couldn’t be recovered, possibly due to deletion of posts or suspended accounts. Table 4 shows the amount of tweets that we were able to recover.
We trained two classification systems for this task, one with Logistic Regression and one using BETO, and we decided to submit a set of predictions for each system: • Run 1 consists of a XGBoost classifier fed with metadata features and Logistic Regression probabilities, trained with features from a Bag of Words of range=(1, 3) considering the term frequency of all the tokens that appear at least twice. • Run 2 is similar to Run 1, but in this case the XGBoost classifier takes metadata features and BETO probabilities as inputs.
To evaluate our experiments with the features discussed in section 2.2, we performed a 5-Fold Cross Validation on the train set for LR, and a single train-test split using BETO due to time constraints.
We performed all modeling regarding the creation of TF feature matrices, LR and XGBoost
classifiers using scikit-learn[ 12], and for the BETO model, we used the implementation described
in [
From a total of 19 registered submissions, the BETO classification was ranked 4th place (above both Bi-GRU and BoW-SVM baselines), while the LR classification was ranked 8th. Table 6 lists the top five final rankings for the Aggressiveness Identification track for 2020 (our scores appear in bold).
To breakdown our results, we started by addressing the performance of our proposal, regarding F1-score and contrasted against the rest of the competitors. Fig. 1 presents two box plots for the complete distribution of submitted results in terms of F1 ofensive and F1 non-ofensive. This analysis suggests that the outcome achieved by our proposal is competitive, been located within the first quartile for all submissions. The second part of our analysis focuses on reviewing what kind of class predictions were changed by adding metadata to our best ranked system. In the validation stage, metadata features were able to rectify 25 label assignments (including 17 true positives) and also make 27 new mistakes (with 24 false positives). Table 7 shows some examples of these results (trigger words appear in bold).
In this paper, we describe our strategy to classify aggressive and non-aggressive tweets in Mexican Spanish. In our best performing system, we use a transformers based classifier, BETO, paired with the addition of metadata features through a decision-tree-based ensemble algorithm, XGBoost. Our proposal shows to be competitive for this specific task. We noticed that metadata can be helpful to detect subtle samples of aggressiveness but also classify tweets as ofensive when swearing is present and not necessarily used to insult someone. However it is interesting to see how these additional attributes based on the behavior of users inside social media can be used to strengthen classification methods. We look forward to enhance our current framework, focusing on diferent levels of lexical and contextual analysis using state-of-the-art approaches and complementing them with metadata attributes. transformers for language understanding, arXiv preprint arXiv:1810.04805 (2018). [12] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, E. Duchesnay, Scikit-learn: Machine learning in Python, Journal of Machine Learning Research 12 (2011) 2825–2830.