Matching the most appropriate image to its corresponding article poses a significant challenge in this landscape. This paper explores the intricate challenge of matching headline images to news articles, utilizing the zero-shot capability of CLIP to address the complex relationship between texts and both real and AI-generated images in the MediaEval 2023 News-Images Challenge. Additionally, analyzes the ramifications of diverse translation methodologies on the eficacy of CLIP performance. The innovative approach involving key phrase extraction for CLIP input demonstrates competitive results across various benchmarks in information extraction and matching.
CEUR ceur-ws.org
In today’s Internet age, online news articles play a crucial role as fundamental sources of
information on current events, employing compelling titles and content segments to engage
and inform readers efectively. Journalists strategically integrate images to enhance content
intuitiveness, enabling a comprehensive understanding of the presented information and
captivating the reader’s attention. The MediaEval Multimedia Evaluation benchmark, with a focus
on the NewsImages task [
Understanding the interaction between text and images in news is crucial for grasping news
content creation. Recent studies challenge the notion of a simple text-image connection,
highlighting the limitations of traditional image captioning models. New dynamic
attentionbased models, like those by Messina et al. [
The fundamental concept of this architecture is to integrate both text and image as inputs by embedding them into a shared space. For the image input, each image undergoes vector embedding by the CLIP image encoder. These embeddings are then indexed using the Faiss library (as you can see in Figure 1). With the text input, we process the headline and snippet of the news (including translating text and optionally extracting keyphrases) before encoding into an embedding using the CLIP text encoder. In the end, we identify the 100 most relevant images by leveraging the K-NN (with cosine similarity) algorithm.
The dataset consists of three components, each derived from news content sourced from news
portals, including GDELT1 and GDLT2, and a news feed RT dataset. The articles from RT News
are written in German and paired with their corresponding English texts in the dataset. In
[
The section aims to analyze and extract relevant keyphrases from given inputs. The CLIP without
keyphrase approach shows suboptimal accuracy, attributed to lengthy and noisy headlines and
snippet sentences. The observation that images closely match the content in the headline and
snippet, so to enrich key information for the image query, suggests the need for the keyphrase
approach to extract more crucial entities. To address this problem, our approach uses KBIR
(Keyphrase Boundary Infilling with Replacement) [
Automatic image captioning and text-image matching have advanced significantly, typically
requiring labelled data and specialized training. CLIP, a pre-trained neural network model,
takes a unique approach by learning joint image and text representations without task-specific
optimization. Its ability to transfer knowledge to other tasks without prior training, along with
a large and varied pre-training dataset, especially with news articles collected on the internet,
makes it a suitable and attractive option for tasks such as NewsImage. In addition, this allows
the model to achieve state-of-the-art performance on tasks it hasn’t been explicitly trained
on before, ofering a promising baseline for further research [
The competition task requires participants to predict a sequentially organized list of images that closely aligns with the accompanying textual article. Evaluation employs the Mean Reciprocal Rank (MRR) metric and MeanRecall@K scores (K in 5, 10, 50, 100). Our research undergoes assessment on three datasets provided by the competition organizers, leading to distinct experimental methodologies and variations in textual input for CLIP due to inherent dissimilarities in each dataset. Consequently, our innovative approaches difer for each dataset under consideration.
The experimental findings on various lexical translation methodologies show relatively consistent results compared to utilizing the organizers’ translated text. Comparisons between translation models indicate that using Googletrans is better mBART for the RT dataset, this led to the decision to leverage Googletrans for ongoing enhancements. However, based on experimental reports, among many translation methodologies, we conclude that translator modules do not greatly afect CLIP’s performance, so future methods should consider the option of removing the translator module to reduce pipeline complexity.
In the experiment, incorporating keyphrases into the textual content that previously consisted of the headline and snippet as input for CLIP yields overall performance improvements, particularly in the MeanRecall@5 and MeanRecall@10 metrics (increases of 0.00133 and 0.00734, respectively), as shown in Table 2. The keyphrase’s ability to encapsulate main ideas helps the model focus more on crucial information and clarify images query, resulting in commendable outcomes, particularly in retrieving images within the 5th and 10th ranks.
This study tackles the demanding text-image matching task in the MediaEval 2023 NewsImages challenge, achieving notable success using the pre-trained CLIP model’s zero-shot capability. Our experiments underscore the eficacy of the model architecture and the benefits of employing a pre-trained model. We studied to experiment with the CLIP’s ability on both real and synthetic images, yielding promising outcomes for real images and proficient performance on AI-generated images. In addition, we proved that adding a translator did not improve performance, so we consider not using it in the pipeline in the future. Conversely, using key phrases showed positive signs of slightly increasing the accuracy of top-5 and top-10 image queries.
Future eforts will concentrate on implementing a more extensive approach, exploring
additional techniques, such as re-ranking strategies [