Multi-modal fact verification has become an important but challenging issue on social media due to the mismatch between the text and images in the misinformation of news content, which has been addressed by considering cross-modalities to identify the veracity of the news in recent years. In this paper, we propose the Pre-CoFactv2 framework with new parameter-eficient foundation models for modeling fine-grained text and input embeddings with lightening parameters, multi-modal multi-type fusion for not only capturing relations for the same and diferent modalities but also for diferent types (i.e., claim and document), and feature representations for explicitly providing metadata for each sample. In addition, we introduce a unified ensemble method to boost model performance by adjusting the importance of each trained model with not only the weights but also the powers. Extensive experiments show that Pre-CoFactv2 outperforms Pre-CoFact by a large margin and achieved new state-of-the-art results at the Factify challenge at AAAI 2023. We further illustrate model variations to verify the relative contributions of diferent components. Our team won the first prize (F1-score: 81.82%) and we made our code publicly available1.
The rapid rise of social media technology allows people to send and receive information
immediately, and also creates fertile soil for the fast spread of fake news. The proliferation of fake news
not only triggers a storm of public opinion but also manipulates public events such as elections.
For instance, there were approximately 30 million tweets from 2.2 million users on Twitter in
the five months preceding the US 2016 presidential election, and either fake or extremely biased
news was contained in 25% of these tweets [
Given a claim and the support information consisting of not only text but also images, the
fact checker aims to discriminate whether the claim entails the support. Recent approaches
have demonstrated that utilizing multi-modal contexts to detect fake news achieves better
performance than only using one modality. For instance, Wang et al. [
However, Pre-CoFact fails to finetune the pre-trained model due to the large number of parameters, which heavily depend on the post layers to learn the contextual information. Besides, we argue that additional textual features such as stopword and URL counts provide definitive descriptions and relations between inputs. In addition, the co-attention modules only capture the dependencies between modalities while ignoring the relations between diferent types of samples (i.e., claim and document). Therefore, we believe that multi-modal fact verification remains an unexplored but essential problem.
To tackle this task and the aforementioned limitations, we examine our proposed method on
the real-world Factify 2 challenge [
In this paper, we propose Pre-CoFactv2 with parameter-eficient large foundation models
with feature representations to address the challenge. We utilize large-scale pre-trained
foundation models, DeBERTa [
To summarize, the contributions of this paper are three-fold: • We propose parameter-eficient large foundation models with feature representations (Pre-CoFactv2) for multi-modal fact verification by integrating adapters in the largescale foundation models to achieve competitive performance by training only lightening parameters, and converting additional features to learn the explicit correlations between claim and document. • In addition to capturing information between modalities, we design multi-modal multitype fusions for diferent types of modalities (i.e., claim text and document images, document text and claim images), which enforces the model to distinguish between the given claim and document. • To boost the detection quality, we introduce a unified ensemble method to integrate various considerations from diverse models. This approach won first place, surpassing the second place by 1.3% and the oficial baseline by 25.9% in terms of testing score. Moreover, extensive experiments were conducted to examine and analyze the contribution and efectiveness of each module.
In recent years, multiple modalities (e.g., text and images) have been incorporated to demonstrate
the great potential for fact verification. MAVE [
Existing work focuses on the importance of learning shared representations among multimodal information, which motivates us to use attention mechanisms to achieve the purpose. However, pre-trained models with a base size are often used in the previous approaches, which cannot utilize more fine-grained features compared with the large-size model. To that end, we adopt large-size pre-trained models to efectively produce embeddings of text and images from the complex multi-modal inputs.
Transformer [
In addition to the success of PLMs in NLP tasks, Transformer has also started taking over
visual benchmarks recently. Dosovitskiy et al. [17] proposed Vision Transformer (ViT) for
pretraining with image patches, which has achieved competitive results compared to state-of-the-art
convolution neural networks on ImageNet-1K image-level classification benchmarks. Swin
Transformer (Swinv1) [18] constructs hierarchical feature maps and uses the shifted window
approach for computing self-attention, making it suitable as a general-purpose backbone for
various vision tasks such as object detection and semantic segmentation. Afterwards, Swin
Transformer v2 (Swinv2) [
To use the generic knowledge of textual and visual information for fact verification, we employ SOTA large-scale PLMs as pre-trained models instead of learning from scratch. Furthermore, we utilize adapters in large-scale PLMs to finetune PLMs with lightening parameters while improving model performance.
by = { , |=| 1 , the goal is to classify one of the five categories:
} Given a multi-modal claim denoted by = { , }|=| 1 and a fact-checking document denoted • Support_Text: the claim text is similar but images of the document and claim are not similar.
• Support_Multimodal: both the claim text and image are similar to that of the document.
• Insuficient_Text: both text and images of the claim are neither supported nor refuted by the document. • Insuficient_Multimodal: the claim text is neither supported nor refuted by the document but images are similar to the document.
• Refute: The images or text from the claim and document are completely contradictory.
Each sample contains a claim and a document, each of which includes the OCR feature, some special token count (e.g., stopword, @ URL), and word and character length.
Figure 2 illustrates an overview of the proposed Pre-CoFactv2 framework. The additional features are generated by the feature extractor from the given claim text, claim image, document text, and document image. Then, we adopt two parameter-eficient foundation models for learning in-domain knowledge from pre-trained embeddings with adapters and a multi-modal multi-type fusion module for modeling not only cross-modality (i.e., text and image) relations but also cross-type (i.e., claim and document) relations. Outputs of these embeddings are fused by the verification classifier with cross-entropy loss as well as supervised contrastive loss [ 19] to separate embeddings and find clearer boundaries.
Inspired by previous works for textual information in fake checking [20, 21], textual features are extracted from both claim and document text to enable the model to learn explicit information from diferent perspectives. Specifically, we extract statistical features from the text, namely sentence word length, and character length. Then, we calculate the stopword, @name, and URL counts to indicate the style of text content. Besides, OCR text extracted from images is adopted to get the semantics information of the image instead of pixel values only.
Benefiting from the advancement of utilizing basic knowledge as pre-trained models, we follow
[
Finetuning existing foundation models requires a large number of computation resources due to the number of parameters, but is able to learn better representations for downstream tasks which are not trained in Pre-CoFact. To that end, we design an adapter module in the foundation models, which enables us to finetune the backbone with only a few parameters but still achieves better performance than the freeze parameters.
We follow [23], which demonstrates the competitive performance of fine-tuning between lightening parameters and all backbone parameters, by adding an additional adapter layer in the output layer of Swinv2 to empower the model to capture the information of images in the downstream tasks (fake news detection in this paper). The weights of the adapter layer are computed as follows: 2( ) = ( ) ̃ + ( );̃ ( ) ̃ = ( +̃ ) + , (1) (2) (3) (4) (5) where ̃ is the output embedding before the feed-forward layer of Swin-Transformerv2, ( ) is the original feed-forward layer, and ( ) the same dimension of ( ) from the input-related weights.
produces the adapted representations with
We note that we cannot add adapter layers to our NLP foundation models since our GPU memory will OOM, but we believe that this concept can improve the performance as well.
and the others follow a similar process: Previous work only considered 1) images of claims and documents, 2) text of claims and documents, 3) images of claims and text of documents, and 4) images of claims and text of claims to produce context embeddings. However, the relations between diferent types are critical to judge the text and images across claims and documents. Therefore, we add two additional co-attention blocks to model correlations between 5) images of documents and text of claims, and 6) images of documents and text of documents. The co-attention block is a variation of the multi-head self-attention block, which takes two modalities as inputs to learn interactions and relations. Specifically, we compute the dot products of the query and the key, divide each by and apply a softmax function to the attention scores to obtain the weights on the values, which indicates the relative importance of each value for a given query.
We illustrate the computation of images of claims ( ) and documents ( ) as examples, For predicting the label of the given claims and documents, all of the 12 aggregated outputs from the multi-modal multi-type fusion, the 4 aggregated embeddings of , the 32 dimensions output from the feature extractor are concatenated as the input , , , and of the classifier: ̂ = (( (
1 )) 2 ), = = , = , = , = , = , (
, ) = ( = ( , = ( ( + ( ( )
√ , )) ) + ) ) ) sentence or image embeddings.
where , , , , , ∈ ℝ× , and
method and feed-forward network as in [
To use fewer parameters, we share weights in a co-attention block for improving performance because of allowing the model to learn common representations of the inputs. Finally, we apply the mean aggregation to fuse all the results into one embedding to represent the corresponding and is the same normalization √ , (6) (7) (8) (9) (10) (11) where 1 ∈ ℝ48× and 2 ∈ ℝ ×5. Note that uses ReLU which is the same as .
To enhance the generalization of our method, we jointly train supervised contrastive learning [19] and cross-entropy. Thus, embeddings with the same label would become closer, while embeddings with diferent labels would increase the distance. The loss function is as follows: || =1 = × −
∑ ( ̂ ) + (1 − ) × , where 0.3 for supervised contrastive loss respectively1.
indicates a supervised contrastive loss. We set 0.7 for cross-entropy loss and
To eliminate the efect of noisy data and to integrate various advantages, ensemble learning with
power weighted sum has been used to integrate the informative knowledge from diferent models
to achieve a better predictive performance of the overall model via the voting technique [
The dimension of additional features from text and image was set to 32, the embedding dimension of text and image were both set to 256, the inner dimension of the feed-forward layer was 512, and the number of heads was set to 12. The hidden dimension of the classifier was 128. The dropout rate was 0.1, and the max sequence length was 512. The batch size was 24, the learning rates were set to 5e-5 and 1e-5, the pre-training epochs were set to 10 and the training epochs were set to 15, and the seeds were tested with 42. The pre-trained DeBERTa was deberta-large2, and the Swinv2 was swinv2-base-patch4-window8-2563. All of the parameters in the two pre-trained models are first finetuned by another dataset [ 25] to enhance the model capability, and then by the Factify 2 dataset. In the feature extractor module, all images were transformed by resizing to 256, center cropping to 256, and normalizing. On the other hand, all text was normalized by replacing emojis in text strings using demoji 4, all the abbreviations are 1We empirically found that supervised contrastive loss is not beneficial in this task ( §4.2.1), thus the final model sets as 1. (12) (13) Feature Extractor
PLMCV PLMNLP
Adapters
SupConLoss expanded, and we delete the @name and URL to shorten the length of the text with meaningless words. All the experiments were conducted on a machine with AMD Ryzen Threadripper 3960X 24-Core Processor, Nvidia GeForce RTX 3090, and 252GB RAM. To evaluate the performance of the task, the weighted average F1 score was used across the 5 categories. The source code is available at https://github.com/wwweiwei/Pre-CoFactv2-AAAI-2023.
To examine the relative contribution of our proposed module, we first conduct the ablative experiments by removing each module as the variants of Pre-CoFactv2. Table 1 summarizes the results of the variations on the validation set. We can observe that adopting diferent pre-trained models (i.e., CoatNet [26] and BigBird [27]) with feature extractor (1) slightly outperforms PreCoFact. Moreover, using a pre-trained CV model with Swinv2 with an adapter and a pre-trained NLP model with Deberta-large (3) significantly improves the performance compared with PreCoFact, which illustrates the importance of not only larger PLMs and finetuning them with lightening parameters but also the multi-modal multi-type fusion. Comparing the variations (2) with (3), additional information from the feature extractor further boosts the performance by capturing explicit information instead of only semantic information from the embeddings. It is worth noting that adding contrastive loss to the embeddings after fusion fails to benefit overall performance, which suggests that calculating the contrastive loss of images and text respectively may be more efective. Nonetheless, our proposed framework signifies the ability to incorporate multi-modal claims and documents with explicit features and lightening parameters to efectively classify the veracity of the news.
To ensure the efectiveness of our unified ensemble technique, we conducted a comprehensive ablation study of variants of ensemble techniques by integrating all the predictions in diferent weights and powers. (1) is the subset of (2) when all the weight is the same, (2) is the subset of
Model
Support Text F1 (%) Support Multimodal F1 (%)
Insuficient Text F1 (%) Insuficient Multimodal F1 (%)
Refute F1 (%) (3) when all the power equals 1, and (3) is the subset of (4) when all the power is equal. Table 2 proves that the unified power weighted sum achieves the best result, which demonstrates that our proposed unified ensemble method is superior to other techniques by up to 1.7%.
We can observe that averaging all predictions improves the performance only slightly, while setting diferent importance and powers boosts the performance more. Our unified ensemble technique, in contrast, achieves the best quality compared to diferent ensemble methods. We note however that these parameters are manually tuned, which requires more costs to achieve better performance.
The results for the testing set are shown in Table 3 in terms of the weighted F1-score. Our approach achieved the state-of-the-art performance of 81.82% of the weighted F1-score, winning ifrst place in the multi-modal fact-checking challenge, and outperformed the second place and the oficial baseline by 1.3% and 25.9%, respectively. This again indicates the reasonable and efective design of Pre-CoFactv2.
To further analyze the classification details, we illustrate the confusion matrices of the validation set as well as the testing set of Pre-CoFactv2. As shown in Figure 3, we can observe that the class of refute is the most distinguishable category, while some of the classes between insuficient_text and support_text misclassify each other.
In this work, we introduce parameter-eficient large foundation models with feature representation (Pre-CoFactv2) to mitigate the issue of disseminating multi-modal fake news. With the integration of adapters with foundation models, we are able to finetune the backbones with only a few parameters while achieving better performance. In addition, feature representations provide explicit information about both text and images of claims as well as documents to clarify
Support_Text (%) Support_Multimodal (%)
Insuficient_Text (%) Insuficient_Multimodal (%)
Refute (%) Final (%)
Triple-Check Baseline the relations between them with multi-modal multi-type fusion. With the help of a unified ensemble technique, Pre-CoFactv2 is ranked first on the oficial leaderboard with an F1 score of 0.81, which outperforms the baseline by 25.9% and greatly benefits the research of multi-modal fact verification. Furthermore, extensive ablative experiments demonstrate the efectiveness of each module in our proposed framework. Major, I. Beltagy, H. Nguyen, L. Saulnier, S. Tan, P. O. Suarez, V. Sanh, H. Laurençon, Y. Jernite, J. Launay, M. Mitchell, C. Rafel, A. Gokaslan, A. Simhi, A. Soroa, A. F. Aji, A. Alfassy, A. Rogers, A. K. Nitzav, C. Xu, C. Mou, C. Emezue, C. Klamm, C. Leong, D. van Strien, D. I. Adelani, et al., BLOOM: A 176b-parameter open-access multilingual language model, CoRR abs/2211.05100 (2022). [17] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, N. Houlsby, An image is worth 16x16 words: Transformers for image recognition at scale, in: ICLR, OpenReview.net, 2021. [18] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, B. Guo, Swin transformer: Hierarchical vision transformer using shifted windows, in: ICCV, IEEE, 2021, pp. 9992–10002. [19] B. Gunel, J. Du, A. Conneau, V. Stoyanov, Supervised contrastive learning for pre-trained language model fine-tuning, in: ICLR, OpenReview.net, 2021. [20] C. Castillo, M. Mendoza, B. Poblete, Information credibility on twitter, in: WWW, ACM, 2011, pp. 675–684. [21] J. Gao, H. Hofmann, S. Oikonomou, D. Kiskovski, A. Bandhakavi, Logically at the factify 2022: Multimodal fact verification, CoRR abs/2112.09253 (2021). [22] A. F. Agarap, Deep learning using rectified linear units (relu), CoRR abs/1803.08375 (2018). [23] C. Fu, Z. Chen, Y. Lee, H. Lee, Adapterbias: Parameter-eficient token-dependent representation shift for adapters in NLP tasks, in: NAACL-HLT (Findings), Association for Computational Linguistics, 2022, pp. 2608–2621. [24] W. Wang, Y. Tang, W. Du, W. Peng, Nycu_twd@lt-edi-acl2022: Ensemble models with VADER and contrastive learning for detecting signs of depression from social media, in: LT-EDI, Association for Computational Linguistics, 2022, pp. 136–139. [25] S. Mishra, S. S, A. Bhaskar, P. Chopra, A. N. Reganti, P. Patwa, A. Das, T. Chakraborty, A. P. Sheth, A. Ekbal, FACTIFY: A multi-modal fact verification dataset, in: Proceedings of De-Factify: Workshop on Multimodal Fact Checking and Hate Speech Detection, volume 3199 of CEUR Workshop Proceedings, ceur, 2022. [26] Z. Dai, H. Liu, Q. V. Le, M. Tan, Coatnet: Marrying convolution and attention for all data sizes, in: NeurIPS, 2021, pp. 3965–3977. [27] M. Zaheer, G. Guruganesh, K. A. Dubey, J. Ainslie, C. Alberti, S. Ontañón, P. Pham, A. Ravula, Q. Wang, L. Yang, A. Ahmed, Big bird: Transformers for longer sequences, in: NeurIPS, 2020. [28] S. Suryavardan, S. Mishra, M. Chakraborty, P. Patwa, A. Rani, A. Chadha, A. Reganti, A. Das, A. Sheth, M. Chinnakotla, A. Ekbal, S. Kumar, Findings of factify 2: multimodal fake news detection, in: proceedings of defactify 2: second workshop on Multimodal Fact-Checking and Hate Speech Detection, CEUR, 2023.