We describe our approaches to sub-task 1A on multi-modal check-worthiness classification of the CheckThat! Lab 2023 in English. The goal was to determine whether a tweet is worth fact-checking based on its text and image content. Our submission was based on a kernel ensemble of diferent uni-modal and multi-modal classifiers. It achieved second place out of 7 teams with an F1 score of 0.708. multi-modal, claim check-worthiness, multiple kernel learning, CheckThat! The CheckThat! Lab 2023 [1] included five tasks targeting various aspects of misinformation. We describe our approach to Task 1 Check-Worthiness in Multimodal and Unimodal Contents, which contained two sub-tasks. Of the two sub-tasks, we participated specifically in sub-task 1A targeting multi-modal content. The goal was to classify a tweet consisting of both text and an image as check-worthy or not. The sub-task was ofered both in Arabic and English. We only developed methods for the English data.
The general problem of misinformation in social media has recieved a lot of interest from the
community in recent years. Apart from the CheckThat! Lab tasks there have been tasks focusing
on identifying the veracity of a claim or rumour, such as RumourEval [
The first modern systems for check-worthiness detection include ClaimBuster [
3.1. Data
The multi-modal check-worthiness sub-task is a binary classification task where a tweet
consisting of a short text and an image has to be classified as check-worthy or not. During the
development phase of the shared task, the organizers released training data ( ), validation
data ( ) and a dev-test set to be used for evaluation during development ( − ). The test
data was released shortly before the submission deadline and its labels were only released
after the submission deadline. For all our experiments, we combine the and − sets
into a single validation set . The individual systems are trained on and evaluated on
. The sizes of these sets and their label distributions are shown in Table 1. We note that
each sample contained both text and image data. The training and development data came from
the MM-Claims dataset [
Our first uni-modal system is based on the tweet text only. We first pre-process the texts by
replacing URLs 1, user handles, and sequences of emoji 2 by placeholder tokens. The text was
then lower-cased and tokenized by splitting on white-space. Tokens shorter than 2 characters
were discarded. Based on this we computed TF-IDF [
We then use these feature vectors to train a linear Support Vector Machine (SVM) [
Next, we trained another text-only system. For this we fine-tuned an electra-base-discriminator
[
For our experiments we use the provided discriminator model checkpoint from Huggingface 3
[
In section 3.2.5 we will need access to a feature vector extracted from electra-clf. For this we remove the final dense layer of electra-clf and use the model activations as feature vectors and
2For this we use the emoji package: https://github.com/carpedm20/emoji/. 3https://huggingface.co/google/electra-base-discriminator scale them to unit length. We will refer to these feature vectors as .
Our multi-modal model relies on pre-trained encoder models for each modality. For text, we use
the twitter-roberta-base 4 checkpoint from Huggingface. This is a RoBERTa [
For images, we use a Vision Transformer (ViT ) [33] that has been pre-trained on ImageNet21k [34]. We again use a checkpoint provided by Huggingface5. The model takes images at a 224 × 224 pixel resolution as input and processes them as a sequence of 16 × 16 pixel patches. This results in an output representation of size × where is the number of patches and the patch embedding dimension.
We first project both representations into a shared space of dimension ℎ using a dense
layer and relu activation for each representation. This results in representations of sizes
× ℎ and × ℎ . We then concatenate them to get a new representation of size
× ℎ where = + . We then feed this representation through a transformer encoder
[
We fine-tune this model on but keep the weights of both the RoBERTa and the ViT encoders frozen. We call the resulting model multi-modal-clf and show its hyper-parameters in Table 3.
Recent Large Language Models (LLMs) such as the GPT family [
Value We use the Language Model Query Language (LMQL) [36] to formulate the prompt and constrain the answers. We show the prompt written in LMQL in Listing 1.
from where
f a c t − c h e c k i n g ?
R e a s o n i n g : [ REASON ] o p e n a i / t e x t − d a v i n c i − 0 0 3 STOPS_AT ( REASON , ” . ” ) and ANSWER i n [ ’ Yes ’ , ’ No ’ ]
The placeholder claim is where we insert the tweet text. The placeholders ANSWER and REASON are filled in by the model. In our case we use OpenAI ’s text-davinci-003 6 model. The answer is constrained to the words Yes and No which we can directly use as predictions, which we will call gpt-answer. The reasoning is constrained to be one sentence, since it should stop generating when it produces the first full stop. We apply a similar feature extraction procedure as for text-ngram in Section 3.2.1 to these reasoning sentences. We forgo any special token replacements and use n-grams up to length 3 but keep the other parameters the same. The resulting feature vectors will be called − and call it gpt-ngram. 6https://platform.openai.com/docs/models/gpt-3-5 . We then train a linear SVM on −
We have seen that all our base models have an associated feature vector: − , , − , and − . For each of these we can define a linear kernel. The kernel value for two samples and for a given system is then defined as (, ) = () () , where () is the feature vector of system for sample . Given such a kernel , we can then train an SVM. For − and − this is equivalent to their associated classifiers text-ngram and gpt-ngram. On the other hand, for and − we will call the resulting SVM classifiers electra-kernel and multi-modal-kernel respectively.
We will include an additional ViT encoder based feature vector − . It is based on the same ViT encoder as multi-modal-clf, which also provides a pooled representation for classification, which we will use as − . We will call the resulting kernel-based SVM classifier img-untrained-kernel.
Next, we show how we combine these kernels into an ensemble. Given a set of systems , we
can define their average kernel as:
(, ) = ∑ 1
∈ ||
(, )
This is known as a fixed rule multiple kernel learning method [
In Table 4 we show our main results. Our submission achieved an F1 score of 0.708 on the test set. We note that if we use the default classification threshold 8 electra-kernel and all-kernels achieve that exact same score. This could indicate that our ensemble method is redundant. In practice, F1 scores can be sensitive to the decision threshold. In Figure 1 we show the Precision and Recall Curves for each system. They show the Precision and Recall of a system for all potential thresholds. In the plot we include lines of constant F1 in light gray. We can see that the default thresholds (black cross marks) tend to select sub-optimal operating points.
We could therefore try to find a better classification threshold. For this we can use the validation set and use the threshold which maximizes the F1 score on . The results are shown as red cross marks in Figure 1 and in the column called Tuned Threshold in Table 4. Since gpt-answer provides only binary outputs we can not change its threshold. The values for electra-clf and multi-modal-clf are missing since we did not compute their output on 9. We can see that for most systems this method selects an even worse threshold. We had already
8For SVM-based systems the default threshold is 0, for classifiers trained using cross-entropy to produce class probabilities, the default threshold is 0.5. 9This was due to time constraints. noticed this during development, where system performance varied greatly between and − , and therefore we chose the default classification threshold.
Finally, in Table 4 we also include the scores that could be achieved if we had access to the ideal threshold. We computed it by selecting the threshold which maximizes the F1 score on . Of course, in reality one never has access to this knowledge, but we include it here to show how much influence threshold selection can have on the system comparison.
In Figure 1 we can also see that the curve for submission lies above the individual kernel based systems over the most recall values. Meaning that for most fixed recalls it achieves higher precision. This indicates that our ensembling method indeed yields an improved classifier. On the other hand, we can also see that electra-clf and multi-modal-clf perform even better.
In Figure 2 we show the Receiver Operating Characteristic (ROC) curves for all our systems. We can again see that electra-clf and multi-modal-clf have the highest area under the curve (AUC), meaning that for most fixed false positive rates they have a higher true positive rate than other systems. We can also see that our ensembling method outperforms individual kernel methods.
We have laid out our solution to the CheckThat! Lab 2023 sub-task 1A on multi-modal checkworthiness classification. Our solution includes diverse components that we combine using a multiple kernel learning approach. Our submission achieved second place out of 7 teams with an F1 score of 0.708. While analysing our results, we noted that the performance measure can vary drastically based on the selected decision threshold. When considering threshold-free methods such as ROC and PR curves, we find that our ensemble indeed seems to perform better than its individual components. Nevertheless, we note that the directly fine-tuned models outperform our submission under this lens. The performance gap between electra-clf and electra-kernel as well as multi-modal-clf and multi-modal-kernel is an open question requiring further study.
This work has been funded by the Hamison project supported by the EU ERA-Net CHIST-ERA; the Swiss National Science Foundation [20CH21_209672]. tics, Online, 2020, pp. 1644–1650. URL: https://aclanthology.org/2020.findings-emnlp.148. doi:1 0 . 1 8 6 5 3 / v 1 / 2 0 2 0 . f i n d i n g s - e m n l p . 1 4 8 . [33] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, N. Houlsby, An image is worth 16x16 words: Transformers for image recognition at scale, CoRR abs/2010.11929 (2020).
URL: https://arxiv.org/abs/2010.11929. a r X i v : 2 0 1 0 . 1 1 9 2 9 . [34] T. Ridnik, E. Ben-Baruch, A. Noy, L. Zelnik-Manor, Imagenet-21k pretraining for the masses, 2021. a r X i v : 2 1 0 4 . 1 0 9 7 2 . [35] J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. H. Chi, Q. Le, D. Zhou, Chain of thought prompting elicits reasoning in large language models, CoRR abs/2201.11903 (2022). URL: https://arxiv.org/abs/2201.11903. a r X i v : 2 2 0 1 . 1 1 9 0 3 . [36] L. Beurer-Kellner, M. Fischer, M. Vechev, Prompting is programming: A query language for large language models, PLDI ’23 (2022).