This paper describes the participation of our Research Group in the shared task of MexA3T 2019. We evaluated the impact of using as principal features, the set of terms identi ed as discriminant by distinct feature selection strategies. Our main goal was to test if a condensed set of words can be indicative of the aggressiveness of a short text written in a very informal setting (i.e., tweets). Our experiments indicate that di erent feature selection techniques favor di erent aspects of the aggressiveness in a short text.
Hate speech, harassment, and cyberbullying are a few examples of how social
media can negatively a ect groups of people online. According to [
Accordingly, in this paper, we describe our proposed system for aggressiveness
identi cation in Mexican-Spanish tweets. Speci cally, we describe our
participation in the second edition of the MexA3T 2019 challenge [
Our main goal was to evaluate if a condensed set of terms, obtained through a feature selection strategy, would provide enough information to identify aggressiveness in a tweet. To that purpose, we used the traditional pipeline for text classi cation. First step is the preprocessing, we lowercased all texts, removed numbers and symbols (except for exclamation points), we used a common word to indicate a hashtag, and all emojis where described1 in text format. Second, we extracted a set of attributes to generate a representation based on the vector model. In this phase we tested the three strategies described below for selecting a relative small set of attributes. Third, we use a learning algorithm to classify tweets into two classes: aggressive and non-aggressive.
Next, we describe three di erent strategies for attribute selection. Two of them are widely used in many text classi cation task, however the third one, to the best of our knowledge, has not been used as feature selection technique before. 2.1
Document frequency is a very simple approach for selecting informative terms, aiming at lowering the dimensionality of vector representations. The idea behind this technique is to ignore terms that are used in very few documents (hapax) or used in almost all documents (known as corpus-speci c stop words), assuming its importance is locally to only those texts or they are not useful for distinguishing between classes because those terms appeared in all documents, respectively; hence, we ignore terms not tting on one of this criteria.
For our experiments we used a grid search to nd the best value for these thresholds. We search for both high and low thresholds (high indicate we ignore terms appearing in more than h% of all documents; and low indicate that we ignore terms appearing in less than l% of all documents). The best empirical result was obtained when h = 80% and l = 0%. 2.2
From information theory, mutual information (MI) of two random variables x
and y can be describe as the reduction in the uncertainty of variable x due to
1 We used an emoji library for python: https://github.com/carpedm20/emoji/
the knowledge of y [
Thus, we computed the MI value for each feature, and those with the greater MI value were used as attributes. For our experiments we selected n top features with greater MI, the best performance was obtained with n = 1000. 2.3
This technique is a linguistically motivated approach aiming to identify those
lexical markers that represent the words springing to mind in response to a
speci c topic. Lexical Availability (LA) measures the ease with which a word is
generated in a given communicative situation, and allows to obtain the mental
lexicon which represents the vocabulary ow usable of a person [
In our experiments, for combining the mental lexicon for each class we used the union, intersection and symmetric di erence of both lists. However, the best performance was obtained when we used the union of both lexicons to generate the nal vocabulary. We also search for the n top attributes with the higher value of lexical availability before combining the lists. The best performance was obtained with n = 1000 and n = 3000. 3
During the development phase of the MexA3T challenge, a training set of tweets
was released. The total number of instances for the aggressive class was 2727,
and 4973 for no-aggressive tweets, for a total of 7700 tweets in the training set.
A better description of the data can be found in [
For validating our experiments in this stage we performed a 10 fold cross validation technique. We used the F-score to measure the best performances; although we use the macro average score we also reported the F-score for the positive class (aggressive) and for the negative class (non-aggressive).
Furthermore, we used several classi ers along the development phase but we found consistent performances with Nave Bayes classi er. Thus, for all of our reported results we used the scikit-learn2 implementation of this classi er with the default parameters.
Table 1, under the validation phase column, shows the best performance for all the con gurations tested, using the three di erent feature selection techniques with some variations as described in Section 2. We also included the bag-of-word (BOW) with the top 1000 most frequent features. As we can see from the table, all feature selection strategies showed similar performances. A slighter better Fmacro was obtained with document frequency (DF) and lexical availability (LA 2 https://scikit-learn.org/stable/modules/naive bayes.html with n=3000). However, note that while DF performed similar to LA (n=3000) the size of the vocabulary for DF was approximately 15000 for each fold; while the vocabulary size for LA with n=3000 was approximately 4500 for each fold.
Validation phase
Test phase BOW (n=1000) DF (h=80%) MI (n=1000) LA (n=1000) LA (n=3000) Ensamble
F+
For the evaluation stage of the MexA3T shared task a set of 3156 tweets were given. The nal results are shown in Table 1 under the test phase column. For this stage, we also included an ensemble con guration, which assigns the class corresponding to the majority vote of the ve submissions. As we can see, similar as in the validation stage, we obtained the best f-macro with document frequency (DF) and lexical availability (LA). However, the performance for the positive class (aggressive) decreased, for all submissions, in about 30% in comparison to the validation data.
To have a better idea of the amount of unique information given by the three feature selection strategies, Table 2 indicates the size of unique sets of attributes when comparing pairs of strategies. To make a fair judgment, we examined sets with the same sizes (i.e. 1000). The last row in this table shows, on the one hand, that lexical availability (LA) has 191 unique attributes compared with the set obtained by the bag-of-words (BOW) vocabulary. On the other hand, only 109 attributes are unique in the set of BOW compared against the LA method. Similarly, when LA is compared against the MI technique, the total number of unique attributes is 212 for LA but only 131 for MI. This initial analysis illustrates that the LA method is capable of nding more diverse attributes in general. 0 179 191
MI
This paper describes our participation in the shared task of MexA3T 2019 for identify aggressive tweets written in Mexican-Spanish. We used the general pipeline for text classi cation varying the strategy used for feature selection. Our goal was to test if a condensed set of words could be indicative of the aggressiveness of a short text. Our experiments indicated that di erent feature selection techniques favor di erent aspects of the aggressiveness in a short text.
As feature work we plan to performed a deeper analysis of the set of attributes selected by the tested strategies. Our initial analysis indicates that lexical availability can extract set of attributes with a di erent linguistic meaning as opposed to bag-of-words or mutual information.
We thank UAM Cuajimalpa and CONACyT (project grant CB-2015-01-258588) for their support. The rst author thanks INAOE for the facilities given in her research visit.