This paper describes the participation of the SINAI team in the Detection of Aggressive and Violent INCIdents from Social Media in Spanish (DA-VINCIS) task organized at the evaluation campaign IberLEF 2023. The system combines the encoding of text and images with separated pretrained neural networks. The network was fine-tuned on an augmented dataset from the provided training data. The proposed approach obtained, in task 1, an F1 score of 0.9165 and, in task 2, the second-best result, with an F1 score of 0.8733. The results suggest that data augmentation and the fusion of modality-specialized encoders are valid strategies for achieving state-of-the-art results for the automatic classification of data that combine visual and textual information.
Although chronic violence builds on historical legacies of social and political violence,
oppression, exclusion, and armed conflict, it is also molded by contemporary dynamics such as
rapidly evolving forms of governance, information technologies, climate change, the intensified
dynamics of globalization, and other factors [
Violent comments flood social networks, this new form of communication is easily accessible to most of the world’s population. Social media platforms have attempted to curb the presence of users who engage in violent comments by implementing manual tools. However, the responsibility of identifying and silencing such comments lies with the platform’s users themselves. Recognizing indications of violence within a corpus of text, especially in the absence of contextual information, is a challenging undertaking.
In the tasks proposed by DA-VINCIS lab [
DA-VINCIS task is divided into two subtasks: (1) violent event identification, which consists in the classification of tweets as violent or non-violent (binary classification); (2) violent event category recognition, which is a multi-label task to identify among an accident, murder, robbery, and other, where several labels could be selected in the same comment.
In this paper, we present a multimodal proposal, where the image and the text information are encoded by pretrained models. We propose two pretrained models, which have been fine-tuned, to encode both visual and textual information. Resulting tensors are then concatenated before passing to a feed forward network for final label prediction. To train the models, we have followed a data augmentation strategy.
The paper is organized as follows: in Section 2 a brief review of related works on methods and ideas for the detection of violence in text and multimodal systems is given. In Section 3, an explanation of the corpus and the preprocessing followed to obtain more data through augmentation for training of the model is provided. Then, in Section 4, the architecture of the proposed model is explained in more detail, describing the pretrained models used and detailing the strategy to search for hyperparameters, as well as the use of weights to deal with unbalanced classes in the dataset, as in the case of the Violent event category recognition task. In Section 5 we will show the experiments performed and the results of the best models found. Finally, in Section 6 we will draw some conclusions from the results obtained and discuss possible future work.
The widespread access to social networks in recent years has demanded detecting and removing
sensitive content to prevent minors from viewing it. This content can range from accidents,
aggressions, personal insults, sensitive news and alike. Early exposure to violent content on the
internet is related to desensitization to violence [
Types of violence such as gender-based violence are an under-reported public health problem,
and non-physical forms are recognized as psychological violence. This type of violence can
sometimes be detected through social networks. Universal Sentence Encoder (USE) has been
used in conjunction with multiple machine learning algorithms for the task of violence detection
[
The detection of violence is a novel task, most of the classifications that we can find these years
are classifications by text or images. Probabilistic models such as Latent Dirichlet Allocation
(LDA) topic models [
With the introduction of Transformers [
We use the term "modality" for each of these acquisition frameworks. It is rare that a
single modality provides complete knowledge of the phenomenon of interest. The increasing
availability of several modalities reporting on the same system introduces new degrees of
freedom [
In this paper, we will use the concatenation of textual and visual information provided by images associated with a Twitter comment in Spanish in order to classify it as violent or not.
The provided training data comprises three text files containing comments sourced from Twitter, a folder containing images, and corresponding labels for each task. Each subtask uses the same training data, with the labels being the main diference. The relationship between images and text is closely intertwined, with each tweet item accompanied by at least one image, and a maximum of four. We have 2,996 training entries, each associated with a label according to the subtask. The dataset contains a cumulative count of 4,267 images, spread across these entries. On average, each text has 1.42 images associated with it, since the median of the data is one image per entry, we can conclude that most of the texts have only one image associated with them, with 4 images being the maximum for each text. Table 2 shows the classification of the labels for each data. The first subtask exhibits a reasonable balance among its classes. However, in the case of the multi-label task, a notable imbalance exists within the murder and theft classes.
Labels of subtask 1 contain two values, (0) for non-violent entries and (1) for violent entries. For the second subtask, it is provided a vector with four possible values (accident, murder, robbery, other). These values are not exclusive of each other, the same entry can be a robbery and a murder at the same time, so we talk about a multi-label problem.
Text entries were processed as follows: • Replacing links with the tag "<URL>". • Replacing users with the tag "<USER>".
• Removing emojis.
Due to the limited training data, we have applied various data augmentation techniques. These include modifications to the original images using image modification libraries, and back-translation for the text. The details are given in the next section.
The first step for the data augmentation has been to split a tweet into several entries, the same tweet has been multiplied by the number of images associated to it, i.e. [image 1, image 2, image 3], text; would be the original entry. After splitting, each sample would have one entry for each image.
Following the previous step, it is observed that multiple images will be linked to the same text entry. To address this situation, a back-translation approach will be employed for texts that are identical within an entry but have distinct associated images. For instance, if a particular entry has repeated the same text three times, the last text in the sequence will undergo translation from Spanish to English, and vice versa, three times. This process introduces minor variations to each text within the same tweet, thereby diversifying the dataset.
Finally, each image has been used to generate three diferent versions by applying three diferent transformations, using the albumentations [13] library. The initial transformation involved applying a horizontal flip followed by a vertical flip to the image, with a 50% probability of applying this transformation. The second transformation involves converting the image to monochrome, resulting in a grayscale representation. Additionally, a blur efect is applied to the image, with a 40% probability of this transformation being implemented. These transformations can be seen in figure 1, and the result of the training dataset in Table 1 below.
The comparison between the original data and those obtained after data augmentation to improve the quality of the training set can be seen in Table 2. There is a class imbalance in subtask 2, where there are significantly fewer values for murder and robbery, compared to the other two labels. This negatively afects multi-label training. One solution is to use weights for each class. A more in-depth explanation of the weighting scheme followed for dealing with class imbalance is given in Section ??.
Text Morale #EEUU sufrió una derrota vergonzosa ante ...
Morale #USA sufrió una vergonzosa derrota contra ...
Morale #USA sufrió una vergonzosa derrota contra ...
Morale #USA sufrió una vergonzosa derrota contra ...
Tus acciones te hacen ser una bella persona ...
Tus acciones te hacen una persona hermosa, ...
...
We can split the process followed by our multimodal approach into the following steps: 1. Before training, images and texts are preprocessed separately, through tokenizers and truncating the sequences before being used as inputs to the model. The output obtained from these tokenizers will be the input together with the labels associated with each piece of data for the model. 2. The architecture is based on the use of two pretrained models that tokenize images and text separately. The results obtained are stored separately to be used as input to the pretrained models. 3. Within the multimodal model, the features obtained by the tokenizers are passed as input for each pretrained model depending on whether they are textual features or images. 4. The output tensor of each of these models is concatenated as shown in Figure 2.
Two pretrained models are used in the system architecture. The text processing task is based on the model is RoBERTa (A Robustly Optimised BERT Pretraining Approach) [14] and BEiT (BERT pretraining if Image Transformers) [15] for visual features extraction: • RoBERTa Large: RoBERTa is a self-monitored pretrained transformer model. This means that it is pretrained with the Masked language modeling (MLM) objective only with the raw texts without human supervision. It should be acknowledged that, inadvertently, the English variant of RoBERTa (the original version) was utilized instead of a version explicitly or implicitly tailored to support the Spanish language.
The use of RoBERTa Large implies the model must adapt to Spanish text, in which case performance is afected than if we had used a RoBERTa model in its Spanish version. Remarkably, the final system demonstrated satisfactory performance, indicating that RoBERTa exhibited proficiency in handling Spanish or could be efectively fine-tuned using Spanish text corpora. • BEIT Base patch16-244: This model is a pretrained BEiT model. The BEiT model is a Vision Transformer (ViT), which is a transformer encoder model (BERT-like). The model learns an internal representation of the images that can then be used to extract useful features. In addition, if you have tagged images, it can classify them.
The system architecture starts with the preprocessed data. Images and text are processed independently: texts by Roberta Tokenizer to obtain the inputs for the transformer model. Images go through BEIT Image Processor to obtain pixel values. These will be the inputs to the visual network.
The outputs of these models will be concatenated into a single tensor, which will be the input of the classification layers. This final feed forward network (the classification head) consists of four layers, each with a ReLu activation layer and a Dropout layer with a probability in the range of [0.4, 0.5], depending on the hyperparameter optimization. Each of the layers has a diferent size, decreasing by half, until reaching the last layer, to force final encoding. The size of the output layer depends on the type of subtask being addressed. For binary classification, the output size would be 1, while for multi-label classification, it would be 4. For both tasks, a ifnal sigmoid function is applied. A threshold of 0.5 over that final output will serve to solve both, binary and multi-label classification.
The choice of loss function difers based on the specific subtask the model is trained on. For subtask 1, we used binary cross entropy without weight modification because both classes (violence, non-violence) are well balanced, as seen in Table 2. For subtask 2, on the other hand, the data is unbalanced. Therefore, in the cross entropy loss function, a weight for positive samples is applied. This particular loss function can be customized by providing a vector of weights for the positive classes. Considering subtask 2, it is necessary to utilize weights, with one weight assigned to each of the classes. For a given class, positive counts are the number of samples assigned to that label and the negative count the other remaining labels (see Equation 1). Therefore, the more samples are found for a class, the lower the loss calibration weight.
During the optimization process, various ranges of values for four hyperparameters were explored following a grid search approach, as illustrated in Table 4. To identify the best values, an early stopping technique was employed, where the stopping condition was evaluated during the training process using a development split. This condition is that the macro metric F1 stops improving during three consecutive epochs. This way, the number of epochs to be used for a model with specific parameters can be reliably adjusted.
After a series of experiments, using 80% of the data as training data, and the remaining 20% of data as validation data, the best hyperparameters found for each of the subtasks according to the results obtained from validations are represented in Table 5 for subtask 1, and Table 6 for subtask 2 in Section 5.
After data augmentation, the dataset has 17,059 samples and 16 entries referring to the same text could be found in the dataset. The text might not be similar to the original due to the back-translation technique. To maintain the integrity of the dataset and prevent similar texts from being split between the test and validation sets, a manual split was performed. The goal was to ensure that all similar texts, which have undergone the back-translation process, remain within the same dataset. The split for the local experiments was 80% for training and 20% for validation. The selected validation set does not include the modifications made for data augmentation to be faithful to the real data to be finally predicted in the final phase of the competition.
After identifying the best-performing models, they were trained using the entire dataset. With these trained models, predictions were made for the test samples, and the resulting labels were submitted to the Codalab platform, as instructed by the organizers.
In Tables 5 and 6, the development dataset refers to the local validations using 80% of the data as training and the remaining 20% as validation as explained in previous sections. The validation dataset was provided by the competition for testing. The best F1-score result has been obtained with that dataset because the training set for the validation ranking is richer than with the local experiments. The best results obtained and the models’ parameters are shown in Tables 5 and 6.
The models used for the final test of the competition have been those that have given the best results in the oficial validation dataset, shown in Tables 5 and 6 above.
In the competition’s final results for the violent event identification task (subtask 1) we obtained an F1-score of 0.9165, giving us the fourth-best score. Table 7 shows the best results of the teams in the leaderboard and their scores for this task.
Finally, for the violent event category recognition task (subtask 2) we obtained an F1-score of 0.8733, obtaining the second-best score. Table 8 shows the best results for this task, as in the previous table.
The results obtained from the predictions still need to be processed. As explained above, the data with which the classifier network is trained has the following structure: [image | text].
For each tweet, if the original data has three images, three diferent predictions will be generated. A voting system has been applied to decide which predictive result is the selected one for the tweet. If for the same tweet, there are more or the same positive tags (evidence of violence) than negative tags, it will be labeled as violent. In the case of subtask 2, for each of the options (accident, robbery, murder, other), a vote is made for each of them, following the same voting rule as in subtask 1 explained above.
In this paper, we have developed a multimodal system capable of processing textual and visual information by concatenating them in order to solve classification and multi-label classification problems over the DA-VINCIS lab at IberLEF2023, which provided a corpus of tweets with Spanish images to solve each task it presents.
We have achieved 4th place in the violence classification task and 2nd place in the violence category classification task. The results are promising, and in the future, we will develop other ideas such as exploring further augmentation techniques or using diferent types of pretrained models for linking and processing textual and visual information. Actually, regarding textual information, we plan to replicate experiments using models supporting Spanish, like XLMRoBERTa [16], mDeBERTa [17] or MarIA’s project ones [18]. Also, soft voting instead of hard one applied will be explored. Besides, alternatives to a simple concatenation of features before the classification head may not be the best approach for feature combinations. In this sense, we will study other possibilities, like gating or attention, to generate multimodal encoding.
This work has been partially supported by WeLee project (1380939, FEDER Andalucía 2014-2020) funded by the Andalusian Regional Government, and projects CONSENSO (PID2021-122263OBC21), MODERATES (TED2021-130145B-I00), SocialTOX (PDC2022-133146-C21) funded by Plan Nacional I+D+i from the Spanish Government, and project PRECOM (SUBV-00016) funded by the Ministry of Consumer Afairs of the Spanish Government. Conference on Document Analysis and Recognition (ICDAR), 2019, pp. 71–77. doi:10. 1109/ICDAR.2019.00021. [13] A. Buslaev, V. I. Iglovikov, E. Khvedchenya, A. Parinov, M. Druzhinin, A. A. Kalinin, Albumentations: Fast and flexible image augmentations, Information 11 (2020). URL: https://www.mdpi.com/2078-2489/11/2/125. doi:10.3390/info11020125. [14] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, V. Stoyanov, Roberta: A robustly optimized BERT pretraining approach, CoRR abs/1907.11692 (2019). URL: http://arxiv.org/abs/1907.11692. arXiv:1907.11692. [15] H. Bao, L. Dong, S. Piao, F. Wei, Beit: Bert pre-training of image transformers, arXiv preprint arXiv:2106.08254 (2021). [16] A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, V. Stoyanov, Unsupervised cross-lingual representation learning at scale, CoRR abs/1911.02116 (2019). URL: http://arxiv.org/abs/1911.02116. arXiv:1911.02116. [17] P. He, X. Liu, J. Gao, W. Chen, DEBERTA: Decoding-Enhanced BERT with Disentangled Attention, in: International Conference on Learning Representations, 2021. URL: https: //openreview.net/forum?id=XPZIaotutsD. [18] A. G. Fandiño, J. A. Estapé, M. Pàmies, J. L. Palao, J. S. Ocampo, C. P. Carrino, C. A. Oller, C. R. Penagos, A. G. Agirre, M. Villegas, Maria: Spanish language models, Procesamiento del Lenguaje Natural 68 (2022). URL: https://upcommons.upc.edu/handle/2117/367156# .YyMTB4X9A-0.mendeley. doi:10.26342/2022-68-3.