Matching images to text plays an important role in cross-media retrieval and research has proven this to be an underestimated challenge. This problem is addressed by the MediaEval 2021 NewsImages Challenge with the goal to gain more insights into the real-world relationship of news articles and images. We develop models for re-establishing the connection of a news article to its corresponding image using datasets of a German news publisher (“task 1”). Our approaches follow the idea of pairwise similarity learning and are optimized by algorithmic hill climbing. Additionally, we employ Canonical Correlation Analysis as an approach using joint embedding learning. The evaluation shows that our approaches produce good results for the underlying image-text rematching task, yet require further optimization to yield stable prediction performance.
Multimedia content is accompanying our everyday life. News
articles are one form of multimedia that are characterized by textual
content accompanied by imagery. The assumption that a simple
relationship underlies this connection has frequently turned out
to be oversimplified in research [
The challenge of image-text retrieval has been addressed broadly
in research around multimedia analysis [
Based on the existing methods, we develop two text-image matching strategies optimized for the specific requirements of the NewsImages task. In Sec. 2 we explain the preprocessing and the steps of our strategies in detail. Sec. 3 presented the evaluation results. Finally, the overall findings are discussed in Sec. 4. 2
We develop two approaches addressing the text-image rematching task. This section discusses the steps of our approaches.
Data Preprocessing. We preprocess the provided data for
eficiently computing similarity scores. Firstly, we translate the image
labels (computed by VGG-19 trained on ImageNet) from English to
German using Google Translate [
In addition to the standard preprocessing, we integrate
Open-deWordNet [
Pairwise similarity learning & algorithmic hill climbing. Our first
approach follows the idea of pairwise similarity learning. The
objective is to compute a similarity score for each image-article pair
to be used to construct 1-to-1 matches. We implement this using
spaCy similarity from the natural language processing (nlp)
module SpaCy [
For computing the best parameter configuration, we make use
of algorithmic hill climbing. Starting with an (arbitrary) initial
conifguration, the solution is incrementally adapted [
The parameter configurations are evaluated using matrices containing pairwise similarity scores with the columns representing the image IDs, the rows the article IDs and the correct matches being located along the diagonal of the matrix. The score is computed using the variables Row counter, Column counter, and Total counter.
For each generated parameter configuration, the performance is evaluated as follows: if the similarity score for the diagonal value is higher than the other scores within its row and column, then the index of the total counter is increased by 1. If this condition is not met, then it is checked whether the diagonal value is higher than the other scores within its row and the row counter index increased by 1 dependently. If both conditions are not met, the column counter index is increased by 1. Once the values of all the 3 counting variables are set, the performance is calculated by comparing the values of the row and column counters: if the row counter is larger than the column counter, then the row counter is divided by the number of pairs (n) and returned. Otherwise the column value divided by the number of pairs (n) is returned as the ifnal score and evaluated by our hill climbing algorithm.
Canonical Correlation Analysis. As a second approach, we apply the sklearn Canonical Correlation Analysis. On the preprocessed dataset, the spaCy implementation of Word2Vec is used for computing a vector representation (having 300 dimensions) of the dataset. We use a random sample of size 1,500 data points, utilizing the article text and image labels columns. Then we split this set into train set (2/3) and validation set (1/3). Initial tests of CCA showed a poor performance; that is why, we adapted the method. We applied Kernel PCA (kPCA) to transform the data through a Radial Basis Function (RBF) expansion, limited to utmost 700 generated data dimensions. The kPCA transformation is applied on both the train and test data to keep compatible and comparable dimensions. Then, we train a CCA instance on the train data set and evaluate the model on both the train and validation set. The evaluation is performed based on the predicted vector for each of the article text vectors. Since the CCA-predicted vectors are most likely not corresponding to actual word2vec image label transformations, we compare each CCA prediction to all of the word2vec image label vectors using the cosine similarity measure, thus constructing a similarity matrix. This allows us to make 1-1 article texts to image label mappings. 3
The evaluation (by the task organizers) shows that out pairwise similarity-based approach reaches @100 = 0.21, the CCAbased approach reaches @100 = 0.24. Even though CCA outperforms the pairwise similarity-based approach in general, the recall score for k=5 and k=10 is higher with hill-climbing, indicating that low-level semantics are found more efectively using the straightforward method of pairwise similarity learning. The better evaluation scores for higher observed for CCA show that CCA performs better with regard to high-level semantic similarity.
We analyze the parameter settings for maximizing the performance for the pairwise similarity learning approach. We find that the used configuration considers for each image the 8 labels with the highest score (no label probability score has been applied). This indicates that a detailed image description is crucial for the text-image rematching task. Furthermore, we find that considering the title in addition to the article snippet does not improve the performance. Moreover, our analysis shows that the replacement of words with the first word of their synsets as well as the removal of outliers and duplicates are not activated for the final evaluation.
Analyzing the parameters used by CCA we find, that considering lemmatized article texts and image labels does not yield the optimal results. Using Kernel PCA with a RBF kernel, the R2 score, which indicates how well the regression model fits the observed data, was greatly increased for the train set (achieving values up to 99.8% with k=100). The performance on the test set reached a R2 score of 38.5% (k=100); thus this method outperformed the hill-climbing optimized pairwise similarity learning approach, that reached an R2 score of 32.6%. 4
The evaluation shows, that both approaches yield robust results for the image-text re-matching. The results slightly outperform the best results from MediaEval NewsImages 2020. The CCA-based approach reaches a recall@100 score of 23.6% on the evaluation set; the hill climbing approach based on pairwise similarity learning yields a recall@100 score of 20.6%. Due to limited resources, we have tested only a restricted set of parameter configurations; we think that a further parameter optimization will improve the performance. For our CCA model, we observe a performance diference between training and testing set, indicating the presence of overfitting. The overfitting could be tackled by applying regularization or adding a dropout layer (eliminating features with a low impact). Furthermore, a penalty-based component could be used for boosting articles considering the margin-based error. The ranking of the top-k images for an article could then be optimized significantly.
Furthermore, an alternative image labeling component should be considered to getting a more detailed image description that could be matched with the article text. This is based on the observation that we observed a better performance when considering more images labels.