Using a large dataset of headlines, excerpts, and related images, we examine the complex link between linguistic and visual aspects in news items in this study. In Mediaeval 2023, we are entrusted with identifying patterns to explain the relationships between text and visuals while taking a number of variables into account. The text features were extracted using the BERT model, and the image features were extracted using the CNN model EficientNet-b0. The extracted features of image and text are then used to train the VSE++ model. which helps us to establish the relationship between the text and the images. Our study, which places a strong emphasis on the model, attempts to clarify the intricate dynamics of the connectivity between the text and images.
According to Ali and Paccosi [
This paper by Zhang and Lu [
Yin and Chen in [3] propose a method for precise image and text retrieval in complex multimodal environments. Utilizes improved feature extraction with 2-dimensional principal component analysis (2DPCA) for images and LSTM with word vectors for text. Interactive learning through a dual-modal CAE achieves accurate cross-modal retrieval. Experimental results on multiple datasets demonstrate superior performance, surpassing other methods in mean average precision (MAP) and precision-recall rate (PR) curves. Yu and Yao in [4] introduce a cross-modal Remote Sensing (RS) image retrieval method using Graph Neural Network (GNN). Addresses the challenge of information misalignment between query text and RS images. Proposes a feature matching network with GNN to learn feature interaction and association between text and RS images. Employs text and RS image graph modules and a multi-head attention mechanism for efective fusion and matching. Experimental results on standard datasets demonstrate competitive performance. The previous edition by Lommatzsch and Kille in [5] focuses on developing innovative methodologies to accurately reassociate news articles with corresponding images, understanding the complexities of linking news texts and images using the impact of AI-generated iges.
In our exploration of image-text relations in news articles, we employ cutting-edge methods. Convolutional Neural Networks (CNNs) like EficientNet extract image features, while Bidirectional Encoder Representations from Transformers (BERT) handles text. As the features of diferent texts are of diferent length, we apply a padding process to ensure a cohesive connection between these features. Our ensemble approach combines CNNs, BERT, and the padding process to boost accuracy.
The objective of the proposed work is to fine-tune the model’s parameters to capture intricate visual patterns relevant to news articles. The proposed EficientNet model is trained on a large dataset of diverse images and the pre-trained BERT model has been trained using the dataset that consists of English text and also English translated German texts. This step equips the model with a deep understanding of language nuances, enabling it to extract meaningful features from news article texts.The padding process involves aligning the extracted image and text features to ensure compatibility and enhances the choerence of the features. The propsed work uses Visual-Semantic Embedding (VSE++) model for seamless integration of image and text features extracted from the given datasets.
The features extracted from the datasets have been elaborated below.
BERT employs a transformer-based architecture that facilitates bidirectional learning, allowing the model to capture contextual information from both preceding and subsequent words for improved language representation. EficientNet-b0 features a baseline architecture with compound scaling, systematically increasing model depth, width, and resolution to achieve an optimal balance between computational eficiency and classification accuracy in image tasks. Fine tuning of both image and text model’s representations ensures multimodal integration to enhance its capacity to extract meaningful visual features, to adapt the model to the specific nuances of the dataset at hand, enhancing its efectiveness in capturing domain-specific semantic relationships.
The integration of text and image representations is orchestrated through the Visual-Semantic Embedding (VSE++) model. Faghri and Fleet in [6] present a novel technique, VSE++, for improving visual-semantic embeddings for cross-modal retrieval by incorporating hard negatives in the loss function, resulting in significant gains in retrieval performance, as demonstrated through experiments on the MS-COCO and Flickr30K datasets. VSE++ employs a multimodal architecture that learns joint embeddings for images and text and acts as a bridge to enhance the alignment of visual and semantic representations through positive instance pairs. This model encapsulates the essence of cross-modal understanding, enabling the system to discern semantic similarities between textual descriptions and corresponding images.
We leverage BERT for textual representation, exploiting its bidirectional transformer to capture comprehensive language context. The resulting text features, crucial for tasks requiring nuanced understanding, are zero-padded for uniform length.
On the visual front, we employ EficientNet B0, a computationally eficient yet potent CNN tailored for image classification. Its adeptness at capturing diverse feature levels in images is harnessed, and these features are resized to align with the dimensions of the text features for multimodal integration. Then VSE++ model was fine tuned and used for training based on joint embeddings, comparing the features of text and images obtained for the text and corresponding images from the given datasets consisting of english and german news article’s text and images, with the required paramters used for our datasets. The hyper parameters, learning rate and number of epochs were set to 0.001 and 100 respectively and we are training the model in batches of 100 text and image features in each batch. The model weights are saved in a pytorch ifle and are the weights are added further for the successive batches.
The performance of the proposed architecture was evaluated using the metrics namely Match@N, Mean Reciprocal Rank and Mean Recall@ k.
In the evaluation of information retrieval systems based on the provided metrics, the performance of three distinct runs reveals noteworthy insights. In the context of the English datasets both runs exhibited commendable capabilities in identifying relevant results, with approximately 7% of relevant predictions within the top 100. However, one dataset outperformed its counterpart, showcasing higher mean recall values across various thresholds and a more favorable mean reciprocal rank (MRR) at 100, suggesting a superior ranking of relevant results. Conversely, a non-English dataset, demonstrated a comparatively lower performance, with only 2.67% of relevant results identified within the top 100. The MRR and recall values further indicated a reduced ability to retrieve pertinent information in comparison to the English runs.
This analysis underscores the nuanced performance of the information retrieval model across diverse datasets. While the English runs exhibited robust performance metrics, the disparities observed in the German dataset highlight potential challenges in cross-lingual information retrieval. The results emphasize the importance of considering linguistic variations and datasetspecific characteristics in system evaluations.
In summary, these findings provide valuable insights for optimizing information retrieval systems, emphasizing the need for tailored strategies to enhance performance across diverse linguistic contexts with the help of metrics- Match@N, Mean Reciprocal Rank and Mean Recall@ k.
In this competition, we have built a model based on the foundation and precedents established by previous work. But we could not get the desired output exactly due to time constraints because of unavailability of required resources due to the cyclone Michaung.
We emphasise that similarity models from multiple methods produces the best results. Future work would ideally experiment further with diferent parameters, diferent base estimators, and diferent techniques. [3] X. Yin, L. Chen, A cross-modal image and text retrieval method based on eficient feature extraction and interactive learning cae, Scientific Programming 2022 (2022) 1–12. [4] H. Yu, F. Yao, W. Lu, N. Liu, P. Li, H. You, X. Sun, Text-image matching for cross-modal remote sensing image retrieval via graph neural network, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 16 (2022) 812–824. [5] A.Lommatzsch, B.Kille, O. Özgöbek, M.Elahi, D.-T. Dang-Nguyen, Newsimages: Connecting text and images in mediaeval 2023, Working Notes Proceedings of the MediaEval 2023 Workshop, Amsterdam, The Netherlands and Online and Online, 1-2 February 2024, CEUR Workshop Proceedings, CEURWS.org (2023). [6] F. Faghri, D. J. Fleet, J. R. Kiros, S. Fidler, Vse++: Improving visual-semantic embeddings with hard negatives, arXiv preprint arXiv:1707.05612 (2017).