In this paper, we describe how we leverage OpenCLIP to generate automated image recommendations for online news articles for the MediaEval 2023 NewsImages task. By exploring diferent text prompting techniques, a total of five retrieval approaches were devised. Results show, however, that the bestperforming approach is an unmodified CLIP version with the raw article headline as input. We reflect on this finding and its implication for future NewsImages tasks.
In recent years, methods for aligning visual media, such as images with short textual descriptions
covering their semantic content, have enjoyed increased attention. The introduction of the first
CLIP model [
Given a non-literal relationship between the content of a news article and its corresponding image, this task is slightly diferent from the more classical problem of semantic text and image alignment. An additional complicating factor is that a news article is often substantially longer than an image caption, introducing further challenges.
In our approach outlined in this paper, we rely on an OpenCLIP model [
We opt not to fine-tune the OpenCLIP model but instead experiment with diferent ways
of how best to generate textual input from the available article data. The textual input we
generated serves as a pseudo caption for a news article. The motivation behind doing so is due to
distinct linguistic features of news headlines (e.g., frequent use of noun strings and omission of
auxiliary verbs [
The remainder of this paper is structured as follows: Section 2 describes the details of the setup of our retrieval pipeline, we then present the run evaluations in Section 3, and we conclude our paper with a discussion of these results in Section 4, together with the lessons learned and the implications for the next iterations of this task.
Our main approach for the image retrieval task is generating a representative pseudo caption from the textual content of an article. This text will then be used as an input prompt for an unmodified OpenCLIP model in order to select fitting images. The motivation behind focusing on the text prompt is twofold: First, there are stylistic diferences between article headlines and leads on the one hand and image captions—which is what CLIP was trained on—on the other hand. Second, the dataset provided for the task was too small to meaningfully fine-tune the CLIP model, which is why an unmodified version was used instead.
In total, we used five diferent text generation strategies and submitted one run for each approach. We discuss the details of each of the five approaches in the overview below. Run 1 - Raw Title For the first run, simply use the article’s title as it is stored in the provided dataset. No additional tags, leads, or outlet information were used. This approach serves as the internal baseline for assessing the performance of the subsequent approaches. Run 2 - Pre-processed Title For the second run, we included a text-cleaning pre-processing step. This included removing stop words, punctuation, and other special characters that could potentially result from encoding mismatches (e.g., ‘Ã’ or ‘¢’). Furthermore, we removed any mention of the news outlet and converted the entire text to lowercase. The motivation behind doing so is to create a structurally more caption-like input text without performing any semantic manipulation.
Run 3 - Raw Tags Run three uses the tags provided in the dataset rather than the article title.
The tags are concatenated into a string using a comma as a separator. As the RT dataset did not contain any tags, we used the article text instead. The inclusion of tags allows us to include more information, potentially on what is depicted in the image, without exceeding the token limit of the text encoder of the CLIP model.
Run 4 - T5 For the fourth run, we use a pre-trained T5 model [
Run 5 - NER-TextRank 10 For run five, we used named entity recognition provided by the
spaCy1 framework to extract relevant entities from the article title and text. The extracted
entities were scored using TextRank [
The diferent methods of the run submissions are a first approach to create pseudo captions for news headlines. However, it remains an open question what the recommendations and precise requirements are that result in an optimal rephrasing of the article title. Please see Section 4 for a more detailed discussion of alternative approaches.
Table 1 summarizes the achieved results from the five submitted runs. The numbers show that Run 1 achieved the highest scores for all Hits@k across all three task datasets; using the raw title as input to the CLIP model substantially outperformed all other approaches.
Text cleaning (Run 2) and rephrasing (Run 3) did not improve the retrieval process by producing more caption-like input text. Rewriting news article headlines and teasers into statement sentences with T5 seems to have the opposite efect, as it did not improve the raw title. Similarly, adding named entities to the input text via text augmentation processes (Run 3 and Run 5) seems to mainly introduce more noise into the retrieval process. TextRank’s text keyword extraction was especially detrimental to the retrieval tasks, resulting in the overall lowest scores (see Table 1, Run 5).
Comparing the achieved scores across datasets, we see our approach performing best on GDELT1, followed by GDELT2, and RT. The analysis of the results suggests that this is mainly due to the inclusion of AI-generated content in GDELT2 and RT. The reason for generated content performing worse might be due to the fact that the contents of the image, e.g., human subjects, can be heavily stylized in the GDELT1 and RT datasets.
Evaluations of the training runs showed that retrieving the correct image—if the matching image was AI-generated—was highly dependent on knowing the details of the model used to create the image in the first place. As this information was not communicated for the provided task datasets, this made it very dificult to include any AI-specific text rephrasing or augmentation technique to account for the characteristics of the AI images properly.
Overall, we do take the performance of the raw title baseline as an indicator for editors to select images mainly based on the headline of a news story. As such, we think generating pseudo-captions remains a worthwhile strategy to pursue.
Leading up to the submission, we explored several alternative text-image embedding approaches. Approaches included training feed-forward networks, LSTM options, and Siamese networks. We combined these approaches with various techniques to rewrite the text prompts, such as employing vector combinations of title, lead, tags, and text. Unfortunately, no approach was able to beat the baseline of using the raw article title as input for the OpenCLIP model.
We believe this shortcoming is partly due to the limited size of our training dataset when exploring alternative text-image embedding approaches. The task dataset was too small to serve as a training set for model fine-tuning. For future iterations of the NewsImages task, having access to a larger training dataset is, therefore, critical.
Looking at future iterations of the NewsImages task, we would like to highlight two possible strategies that could lead to an improvement of the retrieval pipeline. The first option to explore is creating a dedicated model that transforms article headlines into caption-like descriptions. For that, we would need to more closely investigate outlet- and story-specific requirements for rephrasing. The second option is to take the current pipeline and reverse it. The resulting workflow would start with the image selection, generating a caption for each image, and then ifnding the most closely matching title/news headline.
In concluding our working notes paper, we want to briefly comment on two shortcomings we
saw in connection with the evaluation process and goal of the task. The first point we want to
address is that by allowing one and only one image to be a valid match for a given news article,
the evaluation process seemingly implies there to be a one-to-one relationship between article
text and image (cf. [
Our second point focused on AI-generated content. We found that the inclusion of AIgenerated images in GDELT2 and RT not only led to an overall lower score compared to GDELT1, but their inclusion potentially entails a major shift in the task’s goal. Instead of focusing on providing meaningful image recommendations for article headlines, the task instead becomes more focused on trying to recreate the exact image generation pipelines.
For more details on the two highlighted aspects, please see our Quest for Insight paper [