This research paper presents our involvement in the collaborative evaluation campaign of DAVINCIS@IberLEF 2023. Our focus lies on tackling the Violent-Event Identification (VEI) task, wherein we employ a multi-modal approach that combines textual input with image captions extracted from visual data. The obtained results demonstrate a competitive performance, as we achieved the top position in the VEI task with an average F1 score of 0.92638.
multitask approach that captures pertinent information, efectively combining the VEI and VEC
tasks into a single model achieving the best results in Violent-Event-Identification (VEI). In the
VEC task, the utilization of a prompt-based approach, as detailed by [
In the 2023 edition of the DA-VINCIS@IberLEF [
The DA-VINCIS@IberLEF 2023 [
Violent-Events Categories
Trafic Accident
Murder Robbery
Other
Categories Percentage 31.38 % 6.01 % 5.24 % 57.38 %
From Table 1 we observe that sub-task 2 is quite challenging due to the low number of examples of classes Murder and Robbery. The Violent-Event-Categorization task is designed to be a multi-label task, however, the number of input texts that belong to multiple classes is around 1% of all the dataset. Hence we decided to treat the VEC problem as a multi-class classification task. Each text example is associated with one or multiple images related to the tweet content, for the training data set we have 4259 images for all the text examples.
To determine how to use the visual and textual information in a way that can be used to accurately
identify and categorize violent events, we propose multiple multi-modal approaches either
combining the outputs of the model’s representation vectors or by joining textual descriptions
of each one of the images related to a text. To extract the important information from the
images, we propose diferent approaches such as using a Convoutional Neural Network (CNN)
or a pre-trained image captioning model such as BLIP [
Original obtained caption: two ak ak ak ak ak ak ak ak ak ak ak ak ak ak ak ak ak ak and a man with a bald haircut and a bald face Caption after correction with regular expressions: two ak and a man with a bald haircut and a bald face.
The histograms depicted in Figure 1 display the lengths of tweets and captions in the
training dataset. It is evident that both tweets and captions are relatively short, thereby falling
well within the text-length limitations of the Transformer model [
The textual information contains the most relevant information about violent events
identification and categorization (See EXP6 Section 4.1.2). We use a RoBERTa [
• Concatenating CNN pooler output, with separated text models: We conduct full
ifne-tuning on two distinct RoBERTa [
• Using Captions and Text in the same sentence to train a single model: The second approach to merge the visual and the textual part is by connecting the captions and the tweet of the input data using a separator (</s></s> for RoBERTa model). This new representation is then passed to a single Transformer model and fine-tuned for each one of the two sub-tasks.
Aside from the proposed-multi modal configurations, we believe that by tuning the parameters of the loss function and randomly increasing the number of examples of the under-represented categories we can increase the performance of the proposed models. In this work, we propose an approximation of the weights that may help to reduce the impact of the class imbalance. We propose to use data oversampling as well to make sure that at least one example of the under-represented categories is contained in a training batch.
• Proposed Randomized Data Oversampling: As described in Section 2, the dataset for Violent-Event-Categorization (VEC) has a high level of imbalance. Therefore we propose a randomized data oversampling strategy. To this aim, we consider the under-represented classes for Murder and Robbery. For each positive example ∈ { , } we take a random number ∼ (0, 1) and an oversampling factor and replicate the number of copies of by . This results in a higher number of positive examples of the under-represented classes. To avoid any bias, after performing this data oversampling, we shufle the data randomly. This oversampling procedure is only applied to the training dataset. (RIGHT) the randomized data oversampling. From this figure, we see that after this oversampling procedure, the dataset has significantly more positive examples for classes Murder and Robbery. • Weighted Loss-Function: To reduce the impact of class imbalance for classes Murder and Robbery in VEC, we propose a modification of the Cross-Entropy Loss function adding weights to balance the importance of the under-represented classes. We use the expression of the Cross-Entropy Loss used by [16] that has the following formula. (, ) = {1, . . . , } where = − log ︃(
(, ) ∑︀=1 (, ) )︃ · I̸= Where is the number of classes, and correspond to the input and the target respectively.
The weights were adjusted according to the following formula.
Where is the probability of selecting a positive example of class obtained by. ≈ = ∑︀∈ I()=
|| Where represents the original dataset without data oversampling. From experimental results, the weights used in this work are presented in Table 2
Class Trafic Accident
Murder Robbery
Other
Associated Loss Weight 0.15 1.8 1.7 0.1
In this section we present the results of the proposed models and proposed strategies in a custom dataset created from the original training dataset, these experiments helped us to determine the importance of parameter tuning to boost model performance and the best multi-modal approach for each one of the tasks to solve. In this section, we present the oficial results obtained for the VEI and VEC shared tasks.
In order to evaluate the proposed models for sub-tasks 1 and 2, we created a stratified partition of the original training dataset consisting of 80% of total examples for training and the rest 20% for validation. We used all labeled examples available to train the final models for the final submission.
To test the impact of parameter tuning on model performance we conducted an experiment
to test how a Weighted Loss Function, Data Oversampling, and the combination of these two
strategies may help to improve the classification capacities of a model. Especially when there is a
high imbalance among classes like in sub-task 2. For these experiments, we set the oversampling
factor to 10. We tested the impact on the performance of the model in sub-task 2 by using a
Weighted Loss Function (EXP1), a Randomized Data Oversampling (EXP2), and the combination
of these two strategies (EXP3). To enable comparison, we conducted tests by training a model
without utilizing any of the suggested strategies to enhance its performance (EXP0). For this
comparison, we consider just the textual information and we use a RoBERTa [
Boosting Strategy
None Weighted Cross-Entropy Loss (WCEL) Randomize Data Oversampling (RDO)
RDO+WCEL
Experiment ID
EXP0 EXP1 EXP2 EXP3
From Table 3 we see that the combination of the Weighted Loss Function and the Data Oversampling increased the model capabilities (EXP3). It is worth mentioning that each one of the strategies increased significantly the classification performance being the Weighted-LossFunction (EXP1) the boosting strategy with the highest increase on its own.
In this case, we see that by using these two boosting strategies together we obtained a better performance in Violent Event Categorization. While the enhancement in model performance may not be significantly superior compared to using any of the individual proposed strategies, it is evident that each strategy contributes to the model’s ability to identify important features in a distinct manner. The Weighted-Cross-Entropy approach assigns greater significance to underrepresented classes, whereas data oversampling ensures a higher representation of examples from these classes in the training batches.
As stated in [17], model ensembles may be useful to increase the classification performance of the models. In this case, we obtained a 0.01 − 0.02 increase when using ensembles compared to not using them. The inference time is not compromised as the ensemble procedure can be done in parallel.
The DA-VINCIS@IberLEF 2023 challenge [
Model
Inception-V3[
RoBERTa-Tweet
Outputs concatenation Joining Caption and Tweet with separator
Experiment ID
EXP4 EXP5 EXP6 EXP7 EXP8
From Table 4 we see that the features obtained from the images do not provide suficient
information to either detect or categorize violent events (EXP4). However, when this
information is combined with the tweet using a separator as in our proposal (EXP8), it slightly
increases the classification performance of the text model. This may be due to the inherent
Self-attention mechanism of all the Transformer models. Self-attention is an attention
mechanism that establishes connections between various positions within a single sequence, enabling
the computation of a representation for the entire sequence [
To demonstrate this, we employ an integrated gradients approach to extract significant features from the texts and ascertain the words that are pertinent for classifying the presence of a violent event.
Figure 5 reveals that certain tokens within the caption have an influence on the model to infer a positive prediction, indicating its capability to identify the crucial features within the combined input text and generated caption. Based on the observations from Figure 5, it becomes apparent that words such as "accidente" (accident) and "carretera" (road) play a crucial role in the model’s ability to recognize a Violent Event mentioned in the tweet. Notably, some of these important words are included in the generated caption. Figure 6 illustrates a tweet in which no violent events were detected. It is noteworthy that words like "emoji" play a role in the model’s negative prediction. Similarly to the positive example, some of the significant words utilized to classify the tweet as a "Non Violent" example are present within the caption generated by the associated image.
We did two submissions for sub-task 1, the first one denoted by RUN1 corresponds to an ensemble of five models with diferent weight initialization, and the second submission ( RUN2) corresponds to a single model used for inference.
The oficial results are shown in Table 5. We see that our proposals obtained the best results among the rest of the participants either with the single model proposal or the ensemble boosting strategy.
For the second sub-task referring to Violent-Events Categorization, we did two submissions using the same model configurations as mentioned in Section 4.2. The oficial results are shown in Table 6.
The results shown in Table 6 correspond to the final results released for sub-task 2. In this task, our proposal obtained seventh place with an average F1 score of 0.842074 among all the classes. The close diference between the results obtained by our proposal and the winning team suggests that with further parameter adjustments, the F1 score could be increased. The boosting strategies presented in this study have the potential to be adjusted in order to further enhance model performance. For instance, instead of data oversampling, data augmentation techniques like using generative models such as GPT-3 [20] can be employed to increase text diversity. This approach enables the model to learn broader patterns, thereby improving the categorization of Violent Events. Similarly, in the case of Weighted-Cross-Entropy, evolutionary strategies like the Covariance Matrix Adaptation (CMA-ES) [21] algorithm can be utilized to determine optimal weights that maximize the F1 score for the under-represented classes.
According to the results obtained from the predictions made by the proposed model, the classes Murder and Robbery have the lowest F1 score of all classes in the violent event categorization. In this section, we present some misclassified examples of these two classes. The first example corresponds to an example of the Murder class that was misclassified and assigned to the Other class.
As we see from the picture associated with the text, it does not provide any useful information that could be related to a murder. The text contains words like investigación (investigation) suggesting that the content described in the tweet is not a confirmed fact.
el @usuario inició investigación por delito de homicidio, luego de un hecho registrado en la 5 de mayo, donde fallecieron dos personas ’en el área se recopilan indicios’ señala la entidad #exitosanoticias url </s></s> un grupo de personas de pie alrededor de una calle
The information shown below is labeled as a positive example of Robbery, however, the model predicted this example as Murder. It is suggested that the word disparo (shot) is mostly associated by the model with murder. The image related to the text does not provide a lot of information similar to the previous example. This shows that the visual part lacks information to complement the textual information. naucalpan edomex emoji coche de policía emoji emoji ambulancia emoji una mujer recibió un disparo durante un asalto a bordo de una combi
los agresores huyeron fue en la esquina de avenida naucalpan y calle allende, en la colonia hidalgo @usuario url </s></s> un hombre está durmiendo en el suelo en un autobús
This paper describes our participation at the DA-VINCIS@IberLEF 2023 challenge on the ViolentEvent-Identification and Violent-Event-Categorization sub-tasks. Our approaches rely on optimal parameter tuning and multi-modal strategies to solve both sub-tasks. The proposed solutions showed a great performance in the VEI task obtaining the best results in the challenge. For the Violent-Event-Categorization task, further parameter tuning is required to increase model results. Based on the experimental findings presented in this research paper, it is evident that achieving better model performance necessitates greater diversity in text generation. Consequently, we propose the incorporation of data augmentation strategies, such as back-translation or generative models, to generate data specifically for the underrepresented classes. In order to obtain more precise weights for the cross-entropy loss, we hypothesize that employing evolutionary strategies will lead to further improvements in model performance.
The authors thank CONACYT, INAOE and CIMAT for the computer resources provided through the INAOE Supercomputing Laboratory’s Deep Learning Platform for Language Technologies (Laboratorio de Supercómputo: Plataforma de Aprendizaje Profundo) with the project "Identification of Aggressive and Ofensive text through specialized BERT’s ensembles" and CIMAT Bajio Supercomputing Laboratory (#300832). Esaú Villatoro-Tello, was supported by Idiap Research Institute during the elaboration of this work. nisms, Technical Report, Cornell Aeronautical Lab Inc Bufalo NY, 1961. [16] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al., Pytorch: An imperative style, high-performance deep learning library, Advances in neural information processing systems 32 (2019). [17] J. Briskilal, C. Subalalitha, An ensemble model for classifying idioms and literal texts using bert and roberta, Information Processing & Management 59 (2022) 102756. [18] I. Loshchilov, F. Hutter, Decoupled weight decay regularization, arXiv preprint arXiv:1711.05101 (2017). [19] D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980 (2014). [20] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., Language models are few-shot learners, Advances in neural information processing systems 33 (2020) 1877–1901. [21] N. Hansen, S. D. Müller, P. Koumoutsakos, Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation (cma-es), Evolutionary computation 11 (2003) 1–18.