Finding a matching image for a news article is a core problem in the creation of traditional and online newspapers. The task of imagetext matching has thus become a vibrant research area in computer science. The performance of state-of-the-art image retrieval systems on various benchmarks is excellent. However, they all rely on datasets with a detailed textual description of the images or on very large training collections. In this work, we optimize imagetext matching algorithms for a small dataset based on the data of a single newspaper. Our optimized processing pipeline and the computed configurations reach precise results. The evaluation results obtained in the MediaEval NewsImages benchmark significantly outperforming the algorithms from previous years.
The process of selecting images for news articles in the multimedia
industry is crucial. Images play a significant role of the storytelling
process. They are used to attract the user’s attention, thus achieving
a high number of clicks or high average dwell time per user.
However, finding a good image that matches the news article in a single
picture is a hard task. Automating this task can provide beneficial
efects in diferent areas, e.g. leveraging the eficiency of publishing
articles, saving costs and human resources. Finding a relationship
between a text and an image is a problem that is researched in the
ifeld of recommender systems. Several papers exist that achieved
good results, but most works rely on huge generic data collections.
In this paper we develop models for a specific newspaper that has
its own image database, a diferent journalistic style and a
significantly smaller amount of data. We evaluate our models using the
data provided in the MediaEval 2021 NewsImages Challenge. A
detailed description of the dataset and the evaluation metrics are
discussed in the Task Overview paper [
Our approach is inspired by recent works in the domain of text
and image encoding as well as advanced Image-Text Matching
methods. We analyzed commonly used CNNs (pretrained on
ImageNet [
In this work we research the degree to which the textual and visual contents of a news article are related. Our developed model should be able to recommend a ranked list of related images, for a given text input. We analyze, whether state of the art image-text matching architectures like VSE work for a small and homogeneous dataset from just one newspaper. Furthermore, we research which adaptations are needed to improve the performance in the MediaEval NewsImages scenario.
The rest of this paper is organized as follows: Sec. 2 explains our approach and the implementation. In Sec. 3 we present the performance results and discuss the specific strengths of the models. Finally, we summarize our work and discuss extensions in Sec. 5. 2
Our approach follows the general architecture of Visual Semantic
Embeddings [
Image Encoding. The image encoding consists of three steps: (i)
preprocessing, (ii) feature extraction and (iii) linear mapping into
the joint embedding size. In the preprocessing, we normalize the
RGB values of the pixels and resize the images to 250 pixels. In the
second step, the preprocessed image are fed into a pretrained CNN
(VGG19 [
Title Preprocessor
Text Preprocessor
Title Embeddings
Text Embeddings Category Embeddings Category Category
Preprocessor Category Fuser r txeT cdeon
E
Title Fuser
FTuesxetr Feature Fuser Article Embedding
Image Preprocessor age rcode ImageNet pretrained VGG-19 ImnE Linear Transformation
Image Embedding
Constrastive Cosine Similarity
Text Encoding. The text encoding stands in the center of this
work. One special feature of news image retrieval is that more than
one textual input might exist. In the NewsImages task the article
title, the snippet and the article category are provided. We employ
three preprocessing steps for each textual input. We apply stop
words removal and stemming (using nltk). In order to get the same
number of word vectors for each input, we picked a constant length
and cropped or extended the input to that length. Subsequently, we
vectorized the text and compute a semantic embedding [
First Fusion Layer: The task of the first fusion layer is to reduce the three matrices to three vector representations. Embedding each textual input on a word level, yields three matrices of the sizes: ( = 5, ) for the title input, ( = 25, ) for the text input and ( = 1, ) for the category input; the word embedding size is 300.
Stacking and Second Fusion Layer: Receiving three inputs of size (1, ) for title, text and category, the next step is to fuse all three representations and transform them in the size of the joint embedding space of the size (1, ). In order to achieve that, we stack all three input representations of (1, ) which yields one vector of size (1, 3 ). Another fully connected layer of size (3, ) then maps the stacked representations to the size of the joint embedding space (1, ).
Contrastive Loss. A multitude of loss functions exist to train the
joint embedding space of article and image embedding. The loss
function should ensure with the learned model that the similarity
between an article and the true matching image is higher than
the similarity to other images and vice versa; the use of a
marginbased contrastive loss fulfills these requirements [
We tested diferent configurations focusing on finding optimal embeddings and hyperparameters.
The experimental results on the Mediaeval test set of size 3022 are shown in Table 1. The experiments reveal that the linear layer for the dimension reduction of the textual inputs outperforms adaptive max pooling in all compositions with a margin of almost 10% in the settings D and C. However, the adaptive max pooling component performed extremely well in most recent works. The reason for that seems to be the diference between pretrained word embeddings and fine-tuned word embeddings. The adaptive max pooling can consider positions in the textual input. The fully connected layer on the other hand is better suited for the pretrained embeddings because it will learn an average importance of the diferent positions.
This suggests that the linear layer instead of adaptive max pooling is more adaptive to the word embedding.
In addition, we find the models B and D difer in the performance whereas the models only difer in the used data for learning the embeddings. While model B with the word embedding trained on wiki achieves higher recall at position 5 and 10, the same model with our word embedding trained on German news article data, performs better at recall at 50 and 100. The wiki-based embedding has a more ifne-grained diferentiation between words. Thus, given a word and a slight modification, the wiki embedding is able to produce two significantly diferent representations. Furthermore, the vocabulary of the wiki embedding is much larger than the vocabulary of our custom embedding. The custom embedding performs better if we look at a large interval of the ranking (r@50, r@100) because it is better suited to embed news article words. In summary, the custom embedding provides a better representation of the articles compared to the embeddings computed based on the wiki corpus. However, when fine grained diferentiations between words are relevant, the wiki-based embedding performs better.
Model r@5 r@10 r@50 r@100 A: Word Embeddings MaxPool + wiki 1.93% 3.76% 12.59% 19.37% B: Word Embeddings Linear + wiki 4.49% 7.26% 20.99% 31.91% C: Word Embeddings MaxPool + custom 2.92% 4.60% 14.36% 24.86% D: Word Embeddings Linear + custom 3.97% 7.10% 21.57% 33.26% E: Word/Subw. Emb. Linear + wiki 2.56% 4.70 % 16.19 % 26.68% Table 1: The evaluation results obtained for the evaluation set for the analyzed models. 4
In this work we made the following contributions: First, we showed that state of the art architectures perform significantly worse on a small, non-descriptive and homogeneous dataset. Secondly, we showed that the performance of embeddings trained on large corpora such as Wikipedia, improve the performance in the top 10 retrieved images while tailored embeddings (to a specific style of a newspaper) improve the top 100 performance. Thirdly, we provide our code1. For future Mediaeval participants and other researchers for benchmarking purposes and to build upon. 5
We have investigated how to adapt state of the art image-text
matching systems to a small, homogeneous and specific dataset. We
analyzed existing and well performing image-text matching systems
like VSE, identified components which do not work well with our
dataset, and systematically tested possible substitutions for them.
Our experiment show that the non-viability of components like the
trainable word embeddings have impacts on the viability of other
components, e.g. the adaptive max pooling. We further showed that
we can successfully substitute these components in an easy way
and achieve reasonable performance on our data. Future work could
investigate other substitutions for the identified components, e.g.
optimizing the word embeddings with respect to the loss.
Furthermore, future projects could research other configurations or even
inputs for the image encoding layer as well as investigating fairness
aspects. It might be that our strategy works well for political
articles but not for sports articles. Thus, analyzing and incorporating
fairness aspects of matching and ranking [