News articles typically consist of text and images. Images plays a crucial role in catching the user's attention and emphasising the article's message. For each news text, the editor must select the best photos from the available set of recent photos, archived photos or stock images to both attract user's attention and best fitting with the news article text. The NewsImages benchmark aims to shed a light on this real-world relation between news texts and the accompanying images. The task provides datasets and evaluation components for studying this relation. The datasets includes AI-generated images as an additional research challenge. This paper describes the NewsImages task in detail, giving the explanations for the dataset and evaluation metrics. It also discusses the connections to existing research and the addressed challenges.
In the fast-paced world of digital journalism, news articles are inherently multi-modal, seamlessly intertwining text and images to convey information. Among the various components of a news article, images occupy a pivotal role. They not only serve as a visual aid; they catch the readers’ interest, compelling them to delve into the text. Furthermore, images reinforce the central message of the article, often providing context or ofering a visual perspective that words alone might fail to capture. With the rise of generative artificial intelligence, there has been a shift towards automating news article creation. This automation includes the generation of text and images that align perfectly with the content.
NewsImages task aims to support the research in understanding the relationship between news texts and their accompanying images on news portals. This relationship is full of challenges. The vast expanse of news topics, the diversity in domains, the plethora of news portals, and the myriad styles of news articles, all culminate in a complex web of considerations when matching text with images. Delving deeper into the scenario, NewsImages is driven by several pertinent questions: How can the connection between texts and images in news articles be re-established? To what extent do generated images alter this re-establishment? Are there discernible patterns or principles that guide editors when they select images for news pieces? And, in the grand scheme of automated news generation, are there innovative methods to generate better-suited images for given news texts?
Deciphering the relationship between text and images in news articles is an important task
for understanding both the creation and the perception of content in the news sector. The
depiction gap between texts and images is a major problem [
For the 5th time, the NewsImages challenge explores the aspects of multimedia content in
news. The first editions (NewsREEL Multimedia [
The NewsImages benchmark investigates the connection between textual news content and associated imagery. This year’s task draws its data from two distinct news dissemination channels: oficial publishers’ portals and RSS feeds. Participants are provided with a comprehensive training dataset, encompassing linked text-image pairs, complemented by a test dataset with disassociated pairs. The challenge mandates the development and critical evaluation of innovative methodologies to accurately re-associate news articles with corresponding images. The dataset represents a challenge with instances of images, such as conceptual stock photographs, potentially aligning with multiple articles. Participants are required to submit a prioritized list of plausible image matches, with the evaluation metric favoring early correct re-associations.
NewsImages provides a dataset compris- Batch Source Purpose No. Cases ing three parts built on news from news portals and an RSS news feed. As source GDELT-P1-a Web sites Training 8500 for the crawled web sites, we use the GDELT-P1-b Web sites Test 1500 GDELT project (https://www.gdeltproject. GDELT-P2-a Web sites Training 12 041 org/) that aggregates news from all over GDELT-P2-b Web sites Test 1500 the world. For the RT part the RSS feed RT-a RSS Feed Training 9755 rtde1 has been used. The dataset has RT-b RSS Feed Test 3000 been created using the following three steps: (1) Crawling: Crawl news items Table 1: Dataset Statistics. The dataset comes in six from the selected sources and eliminate batches. The number of cases refers to the news articles that do not consist of an article-image pairs. image and a suitable text. We use news items published in the period November 2022–August 2023. For the GDELT part the news title and the entities (extracted by GDELT from the news text for creating knowledge graphs) are used (http://data.gdeltproject.org/gkg/index.html. For the RT part, the news title and the snippet (both German originals and English machine translation of these fields) are used. (2) Cleaning: For ensuring the quality of images, we use diferent heuristics for removing duplicates, low quality images, and logos. In addition, we remove images mainly consist of text. (3) Image generation: For studying the problem of matching generated images we use Stable Difusion. We use the news article’s headline as the prompt. The generated images are used to replace some of the original images. In the three parts of the dataset, the fraction of generated images difers. Part GDELT-P1 does not contain any generated images; GDELT-P2 contains 80% generated images, and RT has 50% generated images. (4) Splitting: Each part of the dataset is split into a training and a test set as Table 1 illustrates.
The data set contains information related to articles and images. Articles’ metadata include the URL, title, and a text snippet (RT batch) or the entities extracted from the news text (GDELT batch). Image captions or image filenames must not be used in the task.
The NewsImages benchmark is designed to analyze the relation between news texts and the accompanying images. As a concrete task, the participants must assign a matching image to for each news text in the given test set. Concretely, for each news article an ordered list of 100 images must be submitted. The participants provide a text file that provides a tab separated list of 100 image IDs for each news article ID.
The participants’ submissions are evaluated against a ground truth defined by the originally crawled connection between the images and the text. The ground truth ensures that a 1:1 relation between the images and the texts exists.
The participants’ submissions are evaluated using the Mean Reciprocal Rank (MRR) [
In addition to MRR, we also compute the Average Recall (AR) at ranks for ∈ {1, 5, 10, 20, 50, 100}. AR computes the average over the recall scores calculated for each news article. The evaluation scores are computed separately for each batch.
Participants are encouraged to contribute working notes that elucidate their innovative concepts, fostering an in-depth exploration of the intricate relationship between textual content and images in news media. In this pursuit, participants have the opportunity to submit a maximum of five runs for each of the three test datasets. Each run entails a set of predictions tailored to these test datasets. We encourage participants to engage in a comprehensive comparative analysis of their various runs, encompassing assessments of quality, computational complexity, and resource utilization.
Furthermore, the discussion of results should be characterized by a nuanced consideration of the datasets’ idiosyncrasies, illuminating how the discoveries made can be extrapolated to diverse scenarios. To culminate, participants are expected to articulate their insights and reflect on their potential contributions towards advancing cutting-edge research in this field.
The linking between news texts and images is still a complicated problem due to the news domain’s diversity, editors’ habits, and readers’ expectations. The mixture of real photos, stock images, archived photos, and AI generated images makes it very challenging to extract not only concepts from images but also to understand the principles applying when selecting the images. The NewsImages challenge provides a medium-sized, real-world data set for investigating the existing principles for connecting images and texts. Participants can develop, optimize, and evaluate innovative re-matching methods for news texts and images.With the growing popularity and enhancement of AI methods for generating images, images that are more representative of the text could replace the partially matching images like stock photos. These artificial images could be used to reinforce the credibility of fake news but also avoid misinterpretation of news caused by ill-fitted stock images. Thus, understanding the relation between news texts and images remains a highly relevant and challenging research topic. News Images provides the foundation to foster the development and evaluation of innovative approaches. Acknowledgments We gratefully thank Marc Gallofré Ocaña and Sohail Ahmed Khan for supporting the data set creation. We acknowledge the contributions of the GDELT (https://www.gdeltproject.org/) project for providing the data which made the dataset creation possible.