The primary aim of the MediaEval 2023 NewsImages task is to augment the understanding of the interplay between textual and visual elements in news articles. This involves the precise alignment of textual information with its corresponding visual counterpart. News articles leverage a combination of text and images to convey information and engage readers in a multimedia format. The variety of online news articles, coupled with the complexity of the relationship between textual content and images, poses a unique challenge. The proposed approach leverages CLIP's capabilities, employing separate encoders for text (DistilBERT) and images (ResNet50), and projecting embeddings into a lower-dimensional space. The key insight emerged from efectively using the CLIP model to establish a smooth correlation between textual narratives and images. Trained on the NewsImages 2023 dataset, encompassing both real and generated images, the model's results are evaluated using the Mean Reciprocal Rank (MRR) and Precision@K metrics.
The dynamic nature of internet news items showcases a complex interplay between written
material and other multimedia components, including visuals. In contrast to domains like picture
captioning, where captions convey content that is presented directly, the relationship between
text and images in news articles is much more complex. The wide spectrum of news topics,
which includes politics, economics, sports, health, and entertainment, adds to this complexity.
The dificulty increases when timely visuals are unavailable and stock photos or even generated
images are used. The objective of this paper is to identify the intricate relationship between text
and images and match the corresponding news text-image pair, as called for by the NewsImages
2023 task [
Innovative approaches are required to understand and align textual and visual elements in
online articles for a more comprehensive news consumption experience. In recent research,
multi-modal news analysis, as explored by Cheema et al. [
For computer vision training, Radford et al. [
On the topic of self-supervised learning, Wang et al. [
Shen et al. [
Addressing the limitation of mismatching in image-text matching caused by CLIP’s global
feature contrasting, Peng et al. [
To match clippings from news articles with the most appropriate image, the CLIP (Contrastive
Language-Image Pre-training) model is used. CLIP learns a shared embedding space for images
and their corresponding textual descriptions, fostering a unified understanding of multimodal
relationships. The contrastive learning framework enhances the model’s robustness by
maximizing the similarity of correct image-text pairs and minimizing the similarity of incorrect pairs.
CLIP’s capability for zero-shot learning is invaluable in scenarios with diverse and evolving
datasets, such as news articles with varying topics. The model built uses a structure similar to
that of Shariatnia [
The image encoder model plays a critical role in the pre-training pipeline, in order to extract meaningful features from input images. Deep convolutional layers are adopted to capture hierarchical representations of the content. A pre-trained ResNet50 model is employed, to enhance the encoder’s ability to discern patterns and features. A global average pooling, and a zero-class classification configuration are utilized, to extract a fixed-size vector representation.
The text encoder model makes use of DistilBERT. A tokenizer with a maximum sequence length of 200 tokens is used. In the forward pass, the module takes in the input tokens and attention masks, computes the last hidden state through the DistilBERT model and then the final sentence embedding is derived by using the CLS token’s hidden representation from the last hidden state tensor.
The input embeddings are then projected into a 256-dimensional space. In the forward pass, a linear projection layer, followed by a GELU activation function is used. This is followed by a dropout layer and layer normalization. The CLIP model that is built receives a batch containing image data and text inputs. The image and text features are encoded separately through the respective encoders, and then the image and text embeddings are projected.
The similarity between the image and text embeddings is computed using the dot product. The dot similarity serves as a measure of alignment between the two kinds of embeddings. It is calculated by taking the dot product of the normalized embeddings, providing a scalar value that signifies the degree of similarity between the two. When the vectors are perfectly aligned with a cosine similarity of 1, it indicates a high degree of similarity between the news article and the corresponding image. However, when the vectors are orthogonal in nature, the dot product is minimal, signifying dissimilarity. The resulting similarity scores are adjusted with the softmax function and utilized to calculate the contrastive loss. A cross entropy loss is also calculated. The overall loss is calculated as the mean of the losses from the image and text modalities, and backpropagation is executed accordingly. The CLIP model is trained with ten epochs, using a batch size of 32, a projection dimension of 256 and a dropout rate of 0.1.
To assess the task’s performance, the Mean Reciprocal Rank was employed as the primary metric, complemented by Precision@K scores (with K values of 1, 5, 10, 20, 50, 100). The Mean Reciprocal Rank provides insight into the average position at which the linked image appears. Achieving an early match contributes to a higher average score, and the precision scores at various positions in the prediction list ofer a nuanced performance evaluation.
Oxford researcher to watch Barbie as ’dessert’ to Oppenheimer amid dual premiere
On the GDELT-P1 dataset, encompassing standard articles and their corresponding images, the model demonstrated a Mean Reciprocal Rank of 0.07839. In practical terms, this signifies that, on average, the model identifies the first correct match around the 13 th position in the list of predictions for each query. Notably, its performance excelled on the GDELT-P2 dataset, primarily comprising images generated by machine-learning models. Here, the model identified the first correct match at around the 10 th position. The images presented in Figure 2 depict the top three retrievals corresponding to the given prompt.
This work explores the performance of the CLIP model in the text-image matching task of NewsImages at MediaEval 2023. A fair level of accuracy was obtained using CLIP, as evaluated using the Mean Reciprocal Rank and the Precision@K scores. The model exhibits superior performance on the GDELT-P2 dataset, with the generated images compared to the GDELT-P1 dataset. To expand the scope and enhance the eficacy of the model, a viable strategy involves training the CLIP model with a language-specific text encoder. By incorporating languagespecific features and nuances, this tailored approach aims to bolster the model’s understanding and performance. Furthermore, to advance the contributions of this work, future endeavors could concentrate on delving into the intricacies of the model and implementing additional ifne-tuning techniques.