This paper describes our participant system for the multi-modal fact verification (Factify) challenge at AAAI 2022. Despite the recent advance in text-based verification techniques and large pre-trained multimodal models cross vision and language, very limited work has been done in applying multimodal techniques to automate fact checking processes, particularly considering the increasing prevalence of claims and fake news about images and videos on social media. In our work, the challenge is treated as a multimodal entailment task and framed as multi-class classification. Two baseline approaches are proposed and explored including an ensemble model (combining two uni-modal models) and a multimodal attention network (modeling the interaction between image and text pair from claim and evidence document). We conduct several experiments investigating and benchmarking diferent SoTA pre-trained transformers and vision models in this work. Our best model is ranked first on the leaderboard and obtains a weighted average F-measure of 0.77 on both validation and test set. Exploratory analysis is also carried out on the Factify data set and uncovers salient patterns and issues (e.g. word overlap, visual entailment correlation, source bias) that motivates our hypothesis. Finally, we highlight challenges of the task and multimodal dataset for future research.
Rapidly growing volume of misinformation and fake news have become a pressing challenge
and cause severe consequences on society. Significant joint eforts have been undertaken by
a wide range of parties (represented by journalists, researchers, independent fact checkers)
to protect communities from false information. It has never been more important to have
a versatile ecosystem to scale up and speed up fact checking against misinformation using
technology, which can be broadly categorised into claim detection and claim validation [
Compared to text-based fact checking, multimodal verification is an under-explored area of
research. Image and text both contain rich information but reside in heterogeneous modalities.
Comparing representation learning within the same modality, cross-modal architectures need
not only learn the features for image and text to express their respective content but
importantly capture a measure for cross-modal semantic integrity [
Simple image similarity cannot resolve fine-grained images diferences and perform poorly
for adversarial images [
Relying on visual similarity analysis alone for multimodal fact verification is naturally prone to false positives, because images related to branding and advertisements (e.g., the “breaking news” image or a company’s logo) are often reused. This may cause erroneous detection when there is no real connection between them other than the reuse of a generic image. The problem becomes more complex with images exploited in disinformation on social media.
Our work in this competition is in response to current online misinformation multimodal In the demise of Union Minister Ram Vilas Paswan, the nation has lost a visionary leader. He was among the most active and longest-serving members of parliament...
... President Ram Nath Kovind said on Wednesday, ... Addressing the fourth annual convocation of the Jawaharlal Nehru University, he said said Indian scholars of today ...
Prime Minister Narendra Modi holds a meeting via video-conferencing with the Chief Ministers over #COVID19...
... Prime Minister Narendra Modi on Saturday held a video conference with ... showed Modi wearing a white mask during the interaction ... issue and has focused on solving the above challenges. Two diferent algorithms are designed for the task that is framed as a multimodal entailment prediction problem following two diferent frameworks, including an ensemble learning and an end-to-end attention network. The ensemble model approach is implemented with a decision tree classifier that combines predictions of two uni-modal models with a few data-specific heuristic features. Two uni-modal models are implemented including a 3-way text entailment model based on a State-of-the-Art (SoTA) pre-trained transformer language architectures fine-tuned on the task dataset, and a pre-trained CNN model (ResNet-50) for image similarity. A SoTA multimodal attention network for 5-way end-to-end entailment classification is implemented as an alternative solution in attempt to infer combinatorial entailment relation by combining representation of language and vision. Globe-level multimodal interactions are modeled with a popular multi-branch attention network framework in order to fuse multimodal information. Strong baselines are implemented for both 3-way and 5-way text entailment models to prove the advantage of our proposed methods. Exploratory data analysis and bias test experiments are conducted to understand the potential data issues and present the challenge of creating high-quality multimodal datasets for the real-world problem. Best results from the ensemble model were submitted for the competition.
In the remainder of this paper, we firstly present a brief overview of related work (Section 2), then task definition and our proposed methods in details (Section 3) followed by experiments on the task dataset (Section 5). Exploratory data analysis is elaborated in Section 4. Finally, the results discussion (including 3-way models and 5-way models) and conclusion are provided in Sections 6 and 7 respectively.
Text Entailment Recognising Textual Entailment (RTE) is earliest and most related work to our
Factify challenge that aims to determine an inferential relationship between natural language
hypothesis and premise. On the basis of a given sentence pair, the task is to predict 3-way labels
including Support, Refute or NotEnoughInfo. Well-known shared tasks include FEVER [
Multi/cross-modal representation learning In the field of multimodal reasoning and
matching, the success of attention mechanism in the NLP community motivated computer
vision techniques to shift from traditional twin network (typically with Siamese nets[
Relevance matching technique Relevance matching (RM) is the core problem of
information retrieval (IR) and has also been applied for the detecting the entailment relation[
Visual Entailment Visual Entailment (VE) [
We frame the Factify task as a problem of multimodal entailment, which is to reason about the relationship of a multimodal claim as hypothesis and a multimodal document as premise.
Specifically, given a multimodal hypothesis (e.g. tweet) denoted by = + and a document (typically one or more fact-checking articles) denoted by = + , both of which contain one image and a text, we aim to derive a function (, ) that infers their entailment of five categories ( "Support_Multimodal", "Support_Text", "Insuficient_Multimodal" , "Insuficient_Text" , "Refute"). The label assignment is based on the relationship conveyed by (, ), • Support_Multimodal holds if there is enough evidence in to conclude that is true and is relevant to and in the same information context, • Support_Text holds if there is enough evidence in to conclude that is true but is irrelevant to and does not provide supplemental information for , • Insuficient_Multimodal holds if the evidence in is insuficient to draw a conclusion about but is relevant to and in the same information context, • Insuficient_Text holds if the evidence in is insuficient to draw a conclusion about and is irrelevant to and does not provide supplemental information for , • otherwise, the relationship is Refute, implying that there is enough evidence in to conclude that is false and is irrelevant to Q of both visual and text content.
Recognizing entailment in natural language is a straightforward application for fact verification. In this section, we aim to study how well a SoTA textual entailment model can be fine-tuned on the textual data pairs in Factify data set and then used in a three-way RTE task. This allows us to further assess and benchmark our proposed solution of combining two uni-modal models prediction into an ensemble model for final 5-way multimodal entailment prediction.
Pretrained transformer fine-tuning : Pretrained transformer models [
Transformer-based models, such as BERT, have been one of the most successful deep learning
models for NLP, but one of their core limitations is the quadratic dependency (mainly in
terms of memory) on the sequence length due to their full attention mechanism. For that
reason, Google’s BigBird model is selected in this study, which is one of the most successful
long-sequence transformers that supports sequence length of 4000 tokens. To deal with the
limitations that other models face, BigBird uses a sparse attention mechanism that reduces the
quadratic dependency to linear[
MatchPyramid: In contrary to computationally expensive transformer models, we propose
a simple baseline text entailment model with relevance matching technique. Intuitively, an
article may be relevant to a claim if they have overlapping words or similar words. A strong
interaction model, known as MatchPyramid[
Inspired by [
Subsequently, the self-attended and feature matrix is then fed into a GRU layer in order to obtain contextual representation. Finally, dot product function is applied to build similarity matrix between two GRU output sequences for MatchPyramid model in attempt to measure the semantic relevance more accurately between claim and document with higher level of word semantics. The output of MatchPyramid is flattened into a 1D vector and fed into a fully connected multi-layer perceptron (MLP), followed by the Softmax model to perform 3-way classification.
One way to utilize multiple models is to combine uni-modal model predictions in an ensemble classifier to predict final labels. As elaborated in 3.2, 3-way textual entailment model is helpful in distinguishing three-way entailment relationship based on linguistic and semantic clues between and . To address multimodal entailment (as defined in this task 3.1), a simple relatedness measurement for visual content between and is adopted based on image pairwise similarity as a proxy of their visual entailment computed with a pre-trained CNN model (ResNet) in our approach. The same as text entailment, this is based on hypothesis and salient correlation patterns observed in this dataset that an article is relevant to a claim if the article contains images similar to the claim’s images.
Hence, an ensemble approach is proposed to combine textual entailment, visual relatedness measurement and additional data specific features. More specifically, the proposed ensemble model utilizes a basic decision tree classifier with the following feature encoding to provide end-to-end five categories classification as depicted in Fig. 1.
• Length of Text and OCR: four features of text length between (, ) are employed representing the length of , (), , and () respectively. OCR text are measured and used as independent features here (ref. 4.4). • Text Entailment: two features consisting of a numeric representations (0, 1, 2) of text entailment prediction (i.e., "insuficient", "support", "refute") along with the corresponding probabilities (Sec. 3.2) • Image Similarity: the pairwise cosine similarity score between and is computed based on the features obtained by pre-trained ResNet-50 model, • Image Domain: two features encoded with one-hot-encoding scheme on the source domain name for and (ref. 4.5).
We consider how to obtain attended multimodal information that can efectively capture
consistency and integrity of multimedia content between and . Inspired by current advance
in attention techniques [
In general, our end-to-end multimodal entailment architecture consists of embedding layer, text matching layer, multimodal matching layer, and classification layer . Formally, as input representations, Images pairs ( and ) are represented as the top layer (before softmax) of a pre-trained convolutional network and text pairs ( and ) are mapped into a vector ∈ R by a fixed word embedding layer initialised by Glove embeddings. denote as word embedding dimension (e.g., set to 50 for Glove-50). There is no restriction on the choice of the image encoder but the pre-trained ResNet-50 model is used in our experiments because of its simplicity.
In embedding layer, Let be the dimension of an image visual vector (i.e., = 2049 for ResNet-50) and Let and be the number of words in and respectively. Let ∈ R and ∈ R× be claim image embedding vector and word embedding matrix, respectively. Likewise, let ∈ R and ∈ R× be document image embedding vector and word embedding matrix, respectively. Each 2048-dim feature vector ( and ) is fed into a (non-trainable) linear layer to reduce the visual features from 2048 to 512 dimensional vector space in this work. For the embeddings of the text pair, self-attention (SDP) is applied (as specified in section 3.2 for both and before feeding into a separate GRU layer to obtain both of their context sequence representation (_ ∈ R× and _ ∈ R× ) and corresponding global representation (i.e., ifnal state), denoted by _ ∈ R and _ ∈ R. denotes the GRU output dimension.
In subsequent text matching layer, the same pipeline (as specified above for extended MatchPyramid) is applied in attempt to model the high level relevance between article content () to claim text () based on contextual word embedding interactions. Interaction feature matrix is calculated by the matrix dot product between _ and _, which is then applied by deep hierarchical convolution layers in the MatchPyramid model to extract aggregated similarity feature vector __ ∈ R and is the output dimension of flattened feature maps. The high-level matching patterns are then fed into a multi-layer perceptron (MLP) with dropout to produce the final matching score with learnable weights.
Multimodal latent interaction features are derived in multimodal matching layer which mainly
consists of visual matching layer and cross-modal attention layer. Fundamentally, multimodal
matching layer aims to find potential relevance of document visual vector ( ) to claim vectors of
either visual or text context or both ( and _), and are hence critical for predicting multimodal
entailment relations for target claim. The same as the process for text embeddings, our visual
matching layer utilises self-attention (SDP) for image pairs embeddings ( and ) in attempt to
capture important features in each image. Then, a visual similarity feature (, = ⃦⃦⃦ ⃦⃦⃦⃗ · ⃦⃦⃦ ⃦⃦⃦⃗ )
is computed by applying dot product to the two saliency guided image embeddings with 2
norm. This practice follows [
In cross-modal attention layer, multiple attention mechanisms are applied, aiming to learn to ifnd and attend to the maximum relevant elements on both feature importance and relational mapping among each self-attended visual and content vectors between claim and document. Specifically, text-image interaction feature between document image and claim text is computed , using same SDP attention: _ = (, _). Likewise, document image and document text interaction is computed via ,
_ = (, _). Note that due to the mismatched feature space between vision and content, GRU is employed initially to align image features and the text features in order to perform cross-modal learning. Finally, resulting multimodal features = (, ⊕ , ⊕ ,_ ⊕ ,_) are obtained from the merge (with concatenation) of the two intermediate layers output before feeding into a MLP with dropout to generate fused higher level features.
Finally, in the classification layer , the outputs of two MLP layers are merged along with corresponding hypothesis multimodal representation () into a combined representation with batch normalisation. is the output of a separate MLP layer for merged (concatenation) _ and normalised _ ( ∈ R+). Final representation are applied with dropout regularization before feeding to a softmax layer to output the probabilities of five categories.
Five entailment categories are balanced in both train and validation data set. As mentioned
above, both claim (hypothesis) and document(premise) contains an image and a text of vary
length. Optical character recognition (OCR) text extracted from image are not counted separately
here and combined with corresponding claim and document text respectively. Thus, large claim
word size presented in the table is due to extracted OCR text. The data details for each set are
shown in Table 3. More details about the dataset and task details can be referred in [
Word overlap is an important indication for modeling textual entailment as well as of potential data bias. Naturally, when pairing claims to evidence sentences, the word overlap ratio will be higher on average for claims with their supporting evidence. However, models relying on word-overlap will perform poorly when dealing with complexity in the real world examples (typically antonymous examples and adversarial attacks). In VITAMINC[66] and FEVER[67] dataset, this bias is deliberately minimized in order to create challenging examples that require sentence-pair inference and cannot be solved by simple word matching techniques. Here, the word overlaps distribution per class in train and val set are presented in Table 4. The data distribution indicates that evidential premise data in Factify have clearly high word overlap ratio than other two categories of insuficient evidence.
Image similarity is the most basic indicator of multimodal entailment. To empirically validate the intuition of potential similarity correlation between and (as mentioned in 3.3) in the dataset, image relatedness analysis is conducted in this section. Similar to the potential bias from data overlapping, we compute image pairwise similarity distribution with embedding space computed from pre-trained ResNet50 model over train/val set as presented in Table 5 and 6. As seen from the distribution over five categories, two text related entailment categories have clearly lower pairwise image similarity that multimodal evidential entailment categories.
Text and OCR (text) length distributions between (, ) in the train set are presented in Fig 2 and Fig 3. Clearly separable distribution patterns can be seen across five categories in claim and document text and their corresponding OCR text. As shown in Fig 2, document text length varies most for ’Support_Multimodal’ among other four entailment categories. The document length in two insuficient categories share similar range and ’Refute’ category has the least document length. The claim length distribution shows a clear bias of ’Refute’ examples towards shorter claims. While the remaining classes present similar ranges, ’Insufient_Text’ and ’Support_Multimodal’ tend to include slightly shorter claims. In comparison, OCR text for both claim and document in "Refute" category samples shows surprising longer length than other four categories. Motivated by the observation, we adopted text lengths as features in our ensemble model as illustrated in 3.3.
Source bias is one of the known and common problems in machine learning dataset [68] which
happens when most data samples are collected from the same source. This problem has several
links which mainly include selection bias, capture bias (bias happens due to particular data
collection methods), label bias and negative set bias. We are interested in probing potential
source bias in Factify datset. The potential bias in images domains over five categories brought
to our attention in data analysis. This is important since multimedia metadata have been proved
to be valuable information and signal for fact verification [
To evaluate the performance of our baseline solutions, we use weighted average F1 for benchmarking on validation set. All the experiments are implemented on a single NVIDIA A100 GPU with up to 20 GiB RAM.
We conducted hypothesis only reliance test by using hypothesis only information to train a model as baseline. This is a commonly adopted approach [71, 72] in SNLI/RTE to verify the presence of data bias. The assumption is that without any premise information, this baseline is supposed to make a random guess out of the five classes. We train two models with or without images and test the resulting accuracy for each model on val set.
Two models (Hypo and Hypo+) are implemented with similar architecture that
consists of a text processing component and/or ResNet embedding layer followed by two
fullyconnected (FC) layers. The text processing component is used to extract the text feature from
the given hypothesis. It firstly generates a sequence of word embeddings for the given claim
text. The embedding sequence is then fed into a GRU [73] to output the text context features of
dimension 300. The image processing component involves BGR to RGB conversion, resizing
images (to [
Our experiments shows that the resulting accuracy and weighted F1 achieves same value in both 0.60 and both 0.64 on val set for text and text+image models respectively, implying the existence of bias in Factify dataset. The details are presented in results section. Our proposed solutions outperform the two hypothesis only baselines.
Transformer fine-tune settings: For the best entailment model, the pre-trained BigBird model from Huggingface and the implementation for pair-wise classification finetune was used. For our experiment the model was fine-tuned for 2 epochs, using the AdamW optimizer with learning rate 2e-5 and epsilon 1e-8 with batch size 4. Maximum sentence length was set to the mean length of the input texts namely, 1396 tokens. To train the 3-way entailment model, the 5-way data categories were converted to 3-way categories including "Support" ("Support_Multimodal"+"Support_Text"), "Refute", Insuficient("Insuficient_Multimodal"+"Insuficient_Text"). OCR text from both and were excluded.
MatchPyramid baseline model settings: We use Glove embeddings with 50 dimension (GloVe 6B 50d) for text input and GRU output dimension is set to 50. The number of CNN layers is set to 2, each with kernel size 3 × 3, pooling size 5 × 10 and in valid mode (i.e., no padding). ReLu activation is placed between all convolutional layers. Convolution channels are set to 16 and 32 respectively. 2-layer MLP are set with hidden dimensions of 128 and 64 respectively. The maximum text length for and are set to 100 and 1000 respectively. Limited experiments are conducted including content context size of claim and document, global average pooling and convolution padding schemes.
The ensemble model is implemented using sklearn’s DecisionTreeClassifier class. Best pretrained transformer ("BigBird") based text entailment classifier is adopted and pairwise image similarity is computed based on ResNet-50. For image pre-processing and feature extraction, same practice introduced in 5.2 is applied. One-hot encoding function provided in Scikit-learn is utilised to converts two categorical features (i.e., URL domains of and ) as a one-hot numeric array learnt from train set. Text are pre-processed separately for BigBird model and ensemble model. No pre-processing is applied for four text length features. For the ensemble model training, we use ‘best’ split based on ‘gini’ impurity matrix as training criteria and limit the number of layers to 8 to avoid overfitting. 5.5. 5-way Multimodal classification 5-way end-to-end Multimodal Entailment model Multimodal is implemented with Keras and tensorflow(v2.4) with Adam optimiser with an adaptive learning scheduler. The initial learning rate and weight decay are both set to 0.0001. Batch size is 32 and maximum number of training epochs is set to 80. Optimal parameters and settings from MatchPyramid baseline model experiments are applied. Checkpoint callback is used to save best model that achieves best validation accuracy. ReLu activation is applied to all convolution layers and fully-connected layers. The uniform He initialization ("he_uniform") is used for all ReLU layers. Same settings (layer size, hidden dim, activation, etc) are applied for three separate MLP layers. Few parameters and architectures are experimented including vary lengths of claim and document content, MLP layer size (1-2), hidden layer dimensions of MLPs(64,256,512,768,1024), merge strategies (concatenation and multiplication) of three MLPs output for classification layer. Ablation study is conducted by removing individual sub-components including hypothesis MLP (), document crossmodal interaction(,_), and document-claim crossmodal interaction(,_). The ablation experiments show the efectiveness of full model architecture. Best model (as reported in result 6.2) is obtained at training epoch 9 and stopped at epoch 14 with optimal settings of the 3 MLPs architecture, 1-layer MLP with dimension 256, MLPs outputs merged with concatenation, and text input lengths of and with 100 and 1000 respectively.
The results of the 3-way text entailment models are presented in Table 7. To validate our model choice, we evaluate few SoTA pre-trained transformer models, including BERT, RoBerta, BigBird and LongFormer. The best performing models are BigBird and LongFormer with the overall winner being BigBird because of the slightly better results and smaller input context size required (1396 vs 1484 respectively).
For the architecture of extended MatchPyramid baseline, we have experimented with diferent
parameters such as longer context length in (including 1500, 2000, 3000), Glove model
with 300 dimension, larger GRU output dimension of 300, various pooling size ([
The results of 5-way and model on val set are presented in Table
8. Four of our baseline methods outperform all baseline models proposed by task organiser as
reported in [
It was found that diferentiating between "Insuficient_Multimodal" and "Support_Text" or between "Insuficient_Text" and "Support_Multimodal" was the most challenging task without relying data specific features. In other words, when a sample contains supporting document
text for the claim but the image is irrelevant, our model has low confidence in predicting the label as "Support_Text" or "Insuficient_Multimodal" . Likewise, when document image is relevant to claim image about same information context but document text is irrelevant, our model has low confidence in predicting the correct label. The decision is highly dependent on the annotation bias. From all the labels, the "Refute" label is the most distinguishable category and highly dependent on the text. The performance is highly consistent among all our models and participant systems in this competition (as seen in leaderboard 10). This is possibly mainly attributed to the articles samples selected from very few fact checking sources that have highly diferentiable linguistic clues (typically high frequent negative words used and same verdict sentences frequently appeared in this category such as "The claim is false").
Final test set results and competition leaderboard are presented in Table 9 and 10 respectively. Our best model ("Ensemble") outperform all competition systems and best baseline models [74]. Test result of model achieved 0.77 avg. F1 which is the same as the result on val set and 10% higher than the result of .
We described our participation in the Multimodal Fact Verification Factify Challenge with the implementation of two proposed baseline solutions including an ensemble model and an end-to-end multimodal entailment model. Ensemble model based system outperform the endto-end model on val set and test set. The best performing model in this competition combines results of 3-way text entailment classifier, visual similarity with a pre-trained CNN model and heuristics learnt from the dataset. Multimodal fusion technique is explored in this paper to model interaction between diferent modalities (i.e., text and image) in claim and document pairs and combines information from them to learn multimodal entailment relationship end-to-end. We found that multimodal entailment based system sufer from overfitting. Apart from limited train size and identified data bias, our experiments suggest that fine-grained image and text interaction model need to be explored further.
We found that the ambiguous labels in Factify dataset undermines the performance of our deep learning architecture. Creating a dataset for a complex real-word multimodal NLP problems particularly natural language inference as multimodal verification has raised emergent challenges [75, 76] and indeed a cumbersome task, and we appreciate the work by the Factify organizers, yet, a more elaborate and unbiased dataset along with well defined annotation criterion should make this dataset more suitable for benchmark. More efort is required to tackle the dataset challenge of minimising hypotheses from human annotators and make dataset better reflecting real-world challenges. As an emergent research field, we hope our extensive data analysis and proposed baseline solutions can inspire further work. [59] L. Pang, Y. Lan, J. Guo, J. Xu, X. Cheng, A study of matchpyramid models on ad-hoc retrieval, arXiv preprint arXiv:1606.04648 (2016). [60] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, I. Polosukhin, Attention is all you need, in: Advances in neural information processing systems, 2017, pp. 5998–6008. [61] Z. Li, Y. Li, H. Lu, Improve image captioning by self-attention, in: International Conference on Neural Information Processing, Springer, 2019, pp. 91–98. [62] T. Malisiewicz, A. Gupta, A. A. Efros, Ensemble of exemplar-svms for object detection and beyond, in: 2011 International conference on computer vision, IEEE, 2011, pp. 89–96. [63] M. Gharbi, T. Malisiewicz, S. Paris, F. Durand, A gaussian approximation of feature space for fast image similarity (2012). [64] L. Wang, Y. Zhang, J. Feng, On the euclidean distance of images, IEEE transactions on pattern analysis and machine intelligence 27 (2005) 1334–1339. [65] A. Pedraza, O. Deniz, G. Bueno, Really natural adversarial examples, International Journal of Machine Learning and Cybernetics (2021) 1–13. [66] T. Schuster, A. Fisch, R. Barzilay, Get your vitamin c! robust fact verification with contrastive evidence, arXiv preprint arXiv:2103.08541 (2021). [67] J. Thorne, A. Vlachos, C. Christodoulopoulos, A. Mittal, Evaluating adversarial attacks against multiple fact verification systems, Association for Computational Linguistics, 2020. [68] A. Torralba, A. A. Efros, Unbiased look at dataset bias, in: CVPR 2011, IEEE, 2011, pp.
1521–1528. [69] B.-C. Chen, L. S. Davis, Deep representation learning for metadata verification, in: 2019 IEEE Winter Applications of Computer Vision Workshops (WACVW), IEEE, 2019, pp. 73–82. [70] T. Prabhakar, A. Gupta, K. Nadig, D. George, Check mate: Prioritizing user generated multimedia content for fact-checking, in: Proceedings of the International AAAI Conference on Web and Social Media, volume 15, 2021, pp. 1025–1033. [71] S. Gururangan, S. Swayamdipta, O. Levy, R. Schwartz, S. R. Bowman, N. A. Smith, Annotation artifacts in natural language inference data, arXiv preprint arXiv:1803.02324 (2018). [72] H. T. Vu, C. Greco, A. Erofeeva, S. Jafaritazehjan, G. Linders, M. Tanti, A. Testoni, R. Bernardi, A. Gatt, Grounded textual entailment, arXiv preprint arXiv:1806.05645 (2018). [73] J. Chung, C. Gulcehre, K. Cho, Y. Bengio, Empirical evaluation of gated recurrent neural networks on sequence modeling, arXiv preprint arXiv:1412.3555 (2014). [74] P. Patwa, S. Mishra, S. Suryavardan, A. Bhaskar, P. Chopra, A. Reganti, A. Das, T. Chakraborty, A. Sheth, A. Ekbal, C. Ahuja, Benchmarking multi-modal entailment for fact verification, in: Proceedings of De-Factify: Workshop on Multimodal Fact Checking and Hate Speech Detection, CEUR, 2022. [75] R. Le Bras, S. Swayamdipta, C. Bhagavatula, R. Zellers, M. Peters, A. Sabharwal, Y. Choi, Adversarial filters of dataset biases, in: International Conference on Machine Learning, PMLR, 2020, pp. 1078–1088. [76] S. Sharma, M. Dey, K. Sinha, Evaluating gender bias in natural language inference, arXiv preprint arXiv:2105.05541 (2021).