Images play an important role in the perception of online news. We aim to gain more insight into the interplay of images and texts in different news domains. In this paper, we describe our method toward Image Text ReMatching based on the CLIP model with zero-shots and finetuning. Specifically, we introduce WISE-FT, a method for linear interpolation of weights between the zero-shot and finetuning models, to improve recall for re-matching. The WISE-FT has been reported to be effective to boost accuracy in CLIP on classification experiments. We obtained certain MRR and Recall with these methods.1 Online news articles in recent years have been a mixture of text and image-based articles. Images are often added to text articles to attract attention and help readers understand the article intuitively. Typically, studies of multimedia and recommendation systems assume a simple relationship between images and text. For example, in the study of image captions [1], it is assumed that the caption is a textual representation of the landscape of the image. However, news-specific studies point to a more complex relationship [2]. The NewsImages task [3] in MediaEval 2022 investigates this relationship to understand its implications for journalism and news personalization. In MediaEval 2021, Thien-Tri et al. [4] used CLIP (Contrastive Language-Image Pre-Training) [5]. Thien-Tri et al. did not train on the dataset, but we train on the dataset published by the organizer, resulting in improvement in Recall and MRR was observed. However, news data has a different relationship to image and text dataset such as MSCOCO [6]. Therefore, CLIP's inherent ability may be lost due to shifts in the data distribution caused by finetuning. As a solution we apply WISE-FT [7], which has been reported to be robust to shifts in data distribution in class classification, in our retrieval task. The paper is organized as follows: in Sec. 2, related studies; in Sec. 3, our approach; in Sec. 4, experimental results and their analysis are presented, and trends are discussed through visualization. Finally, Sec. 5 discusses the conclusions and challenges.
CLIP (Contrastive Language-Image Pre-Training) [
CLIP shows consistent accuracy in zero-shot inference across a variety of datasets. Finetuning can also improve accuracy on
specific datasets. However, shifts in the data distribution in finetuning can reduce robustness. Wortsman et al [
Three datasets are provided by the organizer in this task [
News dataset has two characteristics. First, they may differ in content from domain to domain. Second, they contain depending on when the news is released. For the first characteristic, finetuning adapted to the data in that domain is expected to improve Recall. However, for the second characteristic, there is a possibility that finetuning may degrade performance due to differences in data distribution caused by the timing of news releases. Therefore, we use WISE-FT, which has been reported to improve accuracy in classification problems, for the retrieval problem. WISE-FT is a linear interpolation of weights as shown in Equation 1. Since training is performed here for each dataset, the WISE-FT is also performed for each dataset. The hyperparameter α of the linear interpolation is also different for each dataset. 3.3
In this task, three datasets are provided by the organizer for this task: Online News portals, Twitter, and RSS news feed. We sort each dataset in chronological order, using 80% from the top as training data and the remaining 20% as validation data. The predictions for the test data were conducted by extracting features from all the test data, followed by using the features to compute cosine similarity to obtain the top 100 candidates.
In Run1, we use zero-shot prediction by CLIP pretrained on WebImageText. In Run2, we finetuned CLIP on each dataset and predict. In Run3, we apply WISE-FT to the CLIP trained on each dataset. The parameter alpha is the value with the highest Recall@1 in each validation data. Specifically, 0.5 is used for Online news portals, 0.5 for Twitter, and 0.4 for RSS news feeds. 4 4.1
The results of the submitted runs are summarized in Table 1 for Online News portals, Table2 for Twitter, and Table3 for RSS news feed. The left column shows the name of the Runs. The evaluation metrics shown are MRR@100, Recall@5, Recall@10, Recall@50, and Recall@100. Each column in Table 1 shows the results of zero-shot, finetuning, and WISE-FT of CLIP.
The analysis in this chapter uses our own split data for validation. Table 4 shows the average improvement and decrease in ranking from finetuning to WISE-FT. The tabulations here assume of an improvement from zero-shot to finetuning. It should be noted here that the size of each validation dataset is different for each dataset. Recall values improved the most for RSS news feeds, but the average increase in ranking was the lowest. The left side of Figure 1 shows a case that was worsened by finetuning but improved by WISE-FT, and the right side shows a case that was improved by finetuning but worsened by WISE-FT. Here, the original text is too long to display, so only the first sentence is displayed. In cases that were exacerbated by WISE-FT, news about the person was found. In the cases that improved with WISE-FT, news was seen where the text described the scene of the image. It is possible that WISE-FT brought general information into focus. The left side of Figure 3 shows a case that was worsened by finetuning but improved by WISE-FT, and the right side shows a case that was improved by finetuning but worsened by WISE-FT. Here, the original text is too long to display, so only the first sentence is displayed. Despite the higher accuracy compared to other datasets, the text is not a direct expression to the image in both cases.
We adopted CLIP and implemented finetuning and WISE-FT. As a result, we achieved MRR@100 score of 0.240, Recall@100 score of 0.735 for Online News portals test set, MRR@100 score of 0.476 and Recall@100 score of 0.595 for Twitter test set, and MRR@100 score of 0.455 and Recall@100 score of 0.859 for RSS news feed test set. We confirmed that the zero-shot method can obtain a constant Recall. Furthermore, in all datasets, finetuning increased Recall more than zeroshot. This indicates that CLIP pre-trained on datasets with different domains has some effect on the news dataset, and that learning to adapt to the news dataset is an effective method. On the other hand, the improvement in Recall by WISE-FT was significant only for the RSS news feed dataset. This may be because the hyperparameter in the linear interpolation of WISEFT was determined by the validation data, which did not result in optimal model weights for the test data. Alternatively, the finetunig may have over-adapted to the content within a certain time period.
Future work is needed to understand why similar images were attached to each news article. For example, the two images on the left in Figure 3 are images of buildings, but in the news articles they have different information such as place names. For this reason, the application of methods other than deep learning, such as pre-associating images to unique expressions such as place names, may improve Recall. [9] Jörg Tiedemann and Santhosh Thottingal. 2020. OPUS-MT – Building open translation services for the World. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, pages 479–480, Lisboa, Portugal. European Association for Machine Translation.