This article presents the work done in our participation in the task of detecting aggressive and violent incidents on Twitter by using images and text, in the DA-VINCIS competition as part of IberLEF 2023. Due to the high impact that a violent incident generates in society and the negative effect it has on the people involved in the event, finding a solution for the detection of such events is of vital importance for government institutions to ensure the safety of the population. Precisely, our participation was focused on making use of the corpus provided by the organizers to perform the task of detecting violent incidents using exclusively textual information. Different Natural Language Processing approaches were used to solve the competition task such as bag of words, textual representations, and transformers. Our proposal obtained, for subtask 1, an f1-score value of 0.9283 in the development phase and 0.8969 in the final phase, ranking second in the development phase and eighth in the final phase. For subtask 2, an f1-score value of 0.8380 was obtained in the development phase and 0.7647 in the final phase. The team's proposal ranked tenth in the development phase and thirteenth in the final phase.
Violence, defined as the action of causing physical or psychological harm or injury to others or to
oneself, is a common problem that affects society around the world. On the other hand, violent incidents
are a manifestation of violence and can be presented in a variety of ways, such as robbery, murder,
harassment, terrorism, etc. For individuals and communities, experiencing an event of violence can
imply the loss of a feeling of security [
In this context, social networks as a popular medium of communication constitute an important part in the diffusion of violent incidents. More specifically, Twitter has served as a medium for sharing information about violent incidents that users of this social network witness or are aware of. This makes Twitter a useful tool for early detection of violent events due to its immediate and global nature. People can post real-time updates about what they are witnessing to inform authorities and other users about violence in a specific location. This can be particularly beneficial in crisis or emergency situations, where decision making, and response coordination require accurate and up-to-date information.
Tweets containing information about violent events can be difficult to identify and classify, but there are some textual characteristics that can help differentiate these types of posts. The presence in the text of violent language, explicit descriptions of violence, mentions of victims or those involved, hashtags related to violence and references to specific places or events may suggest the presence of violent incidents.
In this paper, we present the work carried out in the participation in the DA-VINCIS competition as
part of IberLEF 2023 [
Recognizing violent incidents on social media is strongly linked to understanding the context of the
texts reporting these incidents, as reported in the work presented in Karystianis et. al. [
Prabhu et al. [
In this work, a model based on BERT [
The DA-VINCIS competition [
Subtask 1: Identification of violent events. Determine whether a given tweet is associated with a violent incident from the considered categories or not (binary classification). In this two-class problem, the "positive" class consists of tweets reporting violent events from the categories of interest (accident, murder, and robbery). The negative samples are those tweets that are associated with reports of other violent events or no violent incidents.
Subtask 2: Violent event category recognition. Recognize the criminal category to which a given tweet belongs (multi-class multi-label classification). The categories considered are Accident, Murder, Robbery and Other. The category Other includes reports of violent incidents other than accident, murder, and robbery, as well as generic tweets not related to violent events.
For the evaluation of the subtask solution proposals, the competition organizers established that for
subtask 1 the metrics of Accuracy, recall and f1-score would be used. Indicating that the primary metric
would be f1-score. For subtask 2, the metrics used were Macro-average of Precision, recall and
f1score. Macro-average of f1-score was the primary evaluation metric for this subtask. The Codalab
platform [
For the development of the competency, a methodology consisting of various stages was proposed: analysis of the data set, data preprocessing, model selection and training, model testing and proposal submission. 4.1.
The organizers of the competition provided a training dataset with 2996 records, each record containing two fields, one for the text of the tweet and another with the names of the images related to that text. A sample of the dataset is shown in Table 1. Additionally, two files containing the corresponding labels to each of the competition tasks were provided.
In subtask 1, we aimed to determine whether a given tweet is associated with a violent incident from the categories considered or not (binary classification). The label file contains the values 0 (negative class) or 1 (positive class) for each record in the training dataset. The 2996 records are divided into 1277 positive and 1719 negative (see left side of Figure 2).
In subtask 2, the aim is to recognize the incident categories to which a given tweet belongs (multiclass multi-label classification). The distribution of the tweets in the various categories can be seen on the right side of Figure 1.
Texts in Each Category Other, 1719, 57%
Accident, 940, 31%
Murder, 182, 6% Robbery, 190, 6%
For the evaluation of the proposed models, the organizers provided a dataset of 582 records in the development phase and for the final phase the test file had 1153 records. 4.2.
After the data analysis, the use of emojis, symbols, special characters, hashtags, and links was identified in the textual part of the dataset; therefore, the text was cleaned in order to perform a better training and detection. For the preprocessing of the data, the following activities were conducted: • • • •
Remove symbology and special characters that do not provide context or information to the tweet.
Convert the emojis present in the text to their textual representation in English using Python’s “emoji” library. For example, for "" its textual representation would be: ":thumbs_up:".
Normalize the different users included in the tweets. Users tagged in tweets do not directly contribute to the detection of violent incidents. For example, @News would be @User. Remove the links present in the tweet. Since in this case the detection of violent incidents is focused on text analysis, the links were removed to avoid noise in the data. 4.3.
At this stage, key decisions were made on the choice of classification models. It was decided to consider the following: 1. Bag of Words (BoW): This classical approach was used to represent the text of the tweets.
This model consists of building a vocabulary from all the unique words present in the dataset,
then representing each tweet as a feature vector indicating the frequency or presence of each
word in the vocabulary. This is a simple but effective representation for training text
classification models. The EvoMSA 2.0 library was used for this approach [
For both phases of the competition (Development and Final), the data set provided was divided into 80% for training and 20% for validation. The results obtained in each phase and subtask are presented below.
Initially, we used EvoMSA combined with bag-of-words. The proposed solution was built using the dataset given for validation and published on the Codalab platform for scoring. For the f1 metric, the proposal received a score of 0.9059. Later, it was determined to use the pre-trained model based on BERT (bert-base-multilingual-cased). This approach received a score of 0.9283 on the f1 measure, placing our proposal as second place in this phase of the competition, as shown in Table 2 (our submissions were made under the user VickBat but we will refer to the results under the team name ITC).
For the final phase, work continued on the optimization of the pre-trained model based on BERT. In the final hyperparameters, we use a learning rate of 4e-5, AdamW as the optimizer, 128 as the maximum sequence value, and 8 as the training batch size. The solution proposal was generated with the data file provided to evaluate this phase. The submitted proposal obtained a value of 0.8969 in the f1-score metric and ranked eighth in the competition as can be seen in Table 3.
For this subtask, a model based on the pre-trained RoBERTa model was developed. In the development phase, the team's proposal obtained a result of 0.8375 in the Precision metric and 0.8380 in the f1 metric, ranking tenth in the stage. Table 4 shows the results of the stage.
For the final phase we worked on the optimization of the model by testing with various adjustments to the hyperparameters. In the final hyperparameters, we use a learning rate of 3e-5, AdamW as the optimizer, 512 as the maximum sequence value, and 8 as the training batch size. We sent the proposed solution to the test data set provided, and a result of 0.7760 was obtained in the Precision metric and 0.7647 under the f1 metric. The proposal was placed in the thirteenth position under the f1 metric. Table 5 shows the results of the subtask.
The results obtained show that both models based on traditional methods, such as the use of bagof-words, and models based on newer techniques, such as the use of transformers (BERT and RoBERTa), were adequate for both tasks. The transformer-based models performed slightly better in our case, but we believe that overall the different models performed competitively.
In this paper, participation in the DA-VINCIS competition as part of IberLEF 2023 in the classification of violent incidents in tweets was presented. For subtask 1, the best result was obtained using a BERT-based pre-trained model with which the team's proposal placed second in the development stage and eighth in the final phase. For subtask 2, the best result was obtained using a pretrained model based on RoBERTa with which the team's proposal ranked tenth in the development stage and thirteenth in the final phase. The results obtained were satisfactory, since a competitive level of performance was achieved in the competition tasks. As future work, it is proposed to continue with the optimization of the hyperparameters used in the classification models and to implement multimodal techniques to address the task.
We want to express our gratitude to CONAHCYT and the Tecnológico Nacional de México campus Culiacán for supporting our team to participate in the DAVINCIS@IberLEF 2023 challenge for detection of aggressive and violent incidents from social media in Spanish.