HCMUS at MediaEval 2021: Fine-tuning CLIP for Automatic News-Images Re-Matching

Abstract

Matching text and images based on their semantics have an essential role in cross-media retrieval. The NewsImages task, MediaEval2021, explores the challenge of building accurate and high-performance algorithms. We proposed diferent approaches leveraging the advantages of fine-tuning CLIP for the multi-class retrieval task. With our approach, the best-performed method reaches a recall@100 score of 0,77441.

INTRODUCTION

In the context of journalism, authors often use images to represent the main content of a particular article. A study in 2020 indicates that the textual content and accompany images might not be related [ 6 ]. Many previous studies in multimedia and recommendation system domains mostly investigate image-text pairs with simple relationships, e.g., [ 3 ]. The MediaEval 2021 NewsImages Task calls for researchers to investigate the real-world relationship of news text and images in more depth, in order to understand its implications for journalism and news recommendation system [ 1 ]

The HCMUS-team participates in the Image-Text-Re-Matching task. Particularly, given a set of image-text pairs in the wild, the task requires us to correctly re-assign images to their decoupled articles, with the aim to understand the implication of journalism in choosing illustrative images.

RELATED WORK

Learning correspondences between images and texts are challenging because of their representation discrepancies. A majority of studies focus on connecting objects with corresponding semantic words in sentences. Lee et al. [ 4 ] proposed Stack-Cross attention mechanism to find correspondence scores between objects and words. As an improvement, Liu et al. [ 5 ] introduced a graphstructured network to capture both image-sentence level relations and object-word level correspondences. On the other hand, Wang et al. [ 8 ] combines early and late fusion strategies. The incorporation helps models to learn both intra-modal and inter-modal information eficiently.

APPROACH

CLIP(Contrastive Language–Image Pre-training)[ 7 ] is proposed by Radford et al. It is a powerful pretrain-model for text-image matching tasks. In our survey, CLIP is the best choice as the baseline for ifne-tuning. The CLIP model has been trained with more than 400 million text-image pairs, and the dataset domain is huge, covering the dataset portion of the NewsImage task. The inference result of the CLIP model (without training with NewsImage dataset) for the dataset provided by the organizers is out-performance compared to the models that we built ourselves or using CLIP as the backbone and train it with NewsImage dataset. In addition, the number of text-image pairs in the NewsImage dataset is relatively small and does not represent the specificity of the dataset. Therefore, we decided not to retrain the model with the NewsImage Task dataset but just use it as an evaluation dataset and fine-tune the number of words, the preprocessing step based on the performance of the model on this dataset. Our fine-tuning takes place at a step that determines how many words to include in the model as well as which words should be kept or discarded. Basically, our approach consists of 4 steps:(1) translation, (2) text preprocessing, (3) image and text vectorization, (4) feed to CLIP, and (5) evaluation. 3.1

Translation

The language used in articles in NewsImage dataset is German, but the language in CLIP is English, so we need to translate all articles into English. Google translate is a useful API to help us do this as it is free and highly accurate. 3.2

Text preprocessing

The conventional preprocess includes dropping NA instances, converting categorical labels into numerical labels, converting all text into lowercase, etc. Additionally, we also expand contractions such as "He’s", "She’s" and remove some words like "an","a","the". We believe this extra preprocessing works will help extract even more useful information for our embedding features. Finally, Ekphrasis library [ 2 ] helps us segment words that are intentionally or unintentionally written and correct misspellings or typos for cleaner text. After the preprocessing step, we determined the number of words to be fed into the model as we realized it greatly afected the model’s performance. Basically, we gradually adjust the number of words fed into the model and observe the change of performance of model. Experimental results on its influence will be described in more detail in the Experiments and Experimental results sections. 4.1

EXPERIMENTS AND EXPERIMENTAL

RESULTS

Experiments

We have submitted five runs for this task. Basically, they are all generated from CLIP model but difer in the number of words of the article included in the model, resulting in diferent results. From run 1 to run 4, corresponding to the number of words of each article that we feed into CLIP is 10, 20, 30, and 40 words. We fine-tuned the word count of each article because during our experiments on the NewsImage set, we noticed that, as we gradually increased the number of words feed into the CLIP, the recall@1 gradually decreased while the recall@100 increased. That said, the influence of the word count in an article on the model’s performance is significant. The last run is the average ensemble submission combines all the results of run 1 to run 4 methods. In this method, all runs have the same weights of 0.25. 4.2

Experimental Results

CONCLUSION AND FUTURE WORKS

News Images is a dificult task when it requires exactly matching the image with the text for nearly 2000 pairs, but we have obtained relatively satisfactory results with 0,77441 for the MeanRecall@100 scale. This demonstrates the eficiency of the model structure, as well as the benefits that the pretrain-model brings, when the dataset used to train in the NewsImage task is not too large. In the future, we wish to investigate more methods and delve into this topic as it is a potential field that still has many problems to be solved.

ACKNOWLEDGMENTS

This work was funded by Gia Lam Urban Development and Investment Company Limited, Vingroup and supported by Vingroup Innovation Foundation (VINIF) under project code VINIF.2019.DA19

References

[1] Özlem Özgöbek Duc Tien Dang Nguyen adn Mehdi Elahi Andreas Lommatzsch , Benjamin Kille . 2021 . News Images in MediaEval 2021 . In Proc. of the MediaEval 2021 Workshop . Online. ( 2021 ). https: //multimediaeval.github.io/editions/2021/tasks/newsimages/

[2] Christos Baziotis , Nikos Pelekis, and Christos Doulkeridis . 2017 . DataStories at SemEval-2017 Task 4: Deep LSTM with Attention for Messagelevel and Topic-based Sentiment Analysis . In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval- 2017 ). Association for Computational Linguistics , Vancouver, Canada, 747 - 754 .

[3] MD. Zakir Hossain , Ferdous Sohel, Mohd Fairuz Shiratuddin, and Hamid Laga . 2019 . A Comprehensive Survey of Deep Learning for Image Captioning . ACM Comput. Surv . 51 , 6 , Article 118 (feb 2019 ), 36 pages. https://doi.org/10.1145/3295748

[4] Kuang-Huei Lee , Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He . 2018 . Stacked cross attention for image-text matching . In Proceedings of the European Conference on Computer Vision (ECCV) . 201 - 216 .

[5] Chunxiao Liu , Zhendong Mao, Tianzhu Zhang, Hongtao Xie, Bin Wang , and Yongdong Zhang . 2020 . Graph structured network for image-text matching . In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 10921 - 10930 .

[6] Nelleke Oostdijk , Hans van Halteren, Erkan Bas, ar, and Martha Larson . 2020 . The Connection between the Text and Images of News Articles: New Insights for Multimedia Analysis . In Proceedings of the 12th Language Resources and Evaluation Conference. European Language Resources Association , Marseille, France, 4343 - 4351 . https: //aclanthology.org/ 2020 .lrec- 1 . 535

[7] Alec Radford , Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever . 2021 . Learning Transferable Visual Models From Natural Language Supervision . CoRR abs/2103 .00020 ( 2021 ). arXiv: 2103 .00020 https: //arxiv.org/abs/2103.00020

[8] Yifan Wang , Xing Xu, Wei Yu , Ruicong Xu, Zuo Cao , and Heng Tao Shen. 2021 . Combine Early and Late Fusion Together: A Hybrid Fusion Framework for Image-Text Matching . In 2021 IEEE International Conference on Multimedia and Expo (ICME) . IEEE , 1- 6 .