This study delves into the domain of multimodal learning, focusing on image-text alignment to discern common olfactory references within multilingual content. The task aims for the Multimodal Understanding of Smells in Texts and Images (MUSTI) at MediaEval '23 [1]. The goal of this task is to address the gap between multimedia analysis and multimedia representation. The task aims to predict whether a text passage and an image evoke the same smell source or not. Our research employs the Bootstrapping Language Image Pre-training (BLIP) model which is a Visual Language Pre-training (VLP) model and capable of both vision-language understanding and generative tasks. We particularly engaged the BlipForConditionalGeneration model, a variant of BLIP, for image captioning to generate textual descriptions for the input images. These generated captions are matched with the corresponding text of the images based on the similarity score. Using the obtained similarity score a binary classification is performed using a multinomial Naive Bayes classifier. Our objective is to evaluate the efectiveness of amalgamating image captioning and text classification for this task. We employed a base model using BLIP and fine-tuned the same model and achieved an F1 score of 48.93% and 55.91% respectively.
Exploring olfactory information in images and text is crucial for historical and interdisciplinary
research, shedding light on nuanced cultural contexts. Museums and galleries globally pioneer
olfactory enrichments for immersive experiences, emphasizing the interdisciplinary potential
and the importance of historically accurate olfactory settings. Automating olfactory information
extraction has not gained much importance among the researchers since it is a challenging task
to identify them in texts or images because of rare linguistic evidence in texts and its implicit
representations in images [
This paper focuses on the MUSTI subtask 1, expediting the understanding of olfactory references in multilingual text and images and forging connections between modalities. This task is a binary classification of whether an image and a text passage evoke the same smell source or not. In this study we evaluate the efectiveness of amalgamating image captioning and text classification for this task. We assess the performance of the state-of-the-art model, BLIP, for image captioning, followed by the Multinomial Naive Bayes classifier to predict the classification labels on the test data of the MUSTI challenge. We provide insights into the performances of both the base and fine-tuned versions of this model.
In Section 2, we cite related work and references. Next, we provide a detailed explanation of our approach in Section 3. Following this, the results of the models in various configurations are reported in Section 4. Finally, a summary of our evaluation and an outlook conclude this paper in Section 5.
The detection of smells, or olfaction, has traditionally been associated with human senses, but
recent interdisciplinary research has extended this concept to image and text analysis. In image
analysis, convolutional neural networks (CNNs), as demonstrated by [
In this study, the research methodology unfolds through a systematic approach, addressing the complexities inherent in matching language descriptions with visual stimuli in the context of olfactory experiences. The methodology followed is an amalgamation of image captioning using BLIP model and text classification using a Naive Bayes classifier.
The study employs a dataset comprising of image-text pairs sourced from [
For understanding smells in texts and images we propose to use the BLIP model [
The baseline model was directly used to generate image captions and the obtained captions were used further to understand the correlation between the images and texts. In the finetuned model, the processor functions were used as a wrapper to combine the two processors BERT tokenizer and BLIP image processor into a single interface, allowing the model to handle both text and image inputs seamlessly during inference and training. It applies WordPiece tokenization on text, simultaneously resizing and preprocessing raw images into the format required by the model.
In the case of fine-tuned model, the same baseline model was tuned by loading the images as batches of 17 and then the model undergoes a rigorous fine-tuning regimen of 20 epochs, facilitated by the Adam with weight decay optimizer (AdamW) with a learning rate of 5e-5.
Short captions are then decoded from the output of the model for all images obtained using the URLs. These captions are then stored and used for subsequent classification.
The Multinomial Naive Bayes classifier is chosen for its aptitude in discerning binary relationships, undergoing training for 3 epochs on the transformed training dataset, transforming texts using a CountVectorizer. The classifier is then used for binary classification based on the similarity of original text and generated captions. By ultimately assigning binary labels using a threshold, we attempt the classification task of identifying the correlation of smells across diferent modalities.
The state-of-the-art BLIP model demonstrated promising results in image captioning, generating textual descriptions for images. Subsequent integration with the Multinomial Naive Bayes classifier allowed us to construct a binary image-text similarity classifier. The performance metrics, including accuracy, precision, recall, and F1 score, were computed for both base and ifne-tuned versions of the model. These metrics provide insights into the model’s ability to correctly identify positive instances while minimizing false positives and false negatives.
Our analysis reveals the eficacy of combining image captioning with text classification for the MUSTI task. The utilization of a binary classifier helps determine the co-relation of smells across diferent modalities. The model seems to perform significantly better when fine-tuned to the MUSTI dataset, as shown in Table 1. It achieves a precision of 67.42% and displays a moderate level of success in the classification task, in contrast to the 62.2% precision of the base model. The corresponding F1-scores of 55.91% and 48.93% for the fine-tuned and base models, respectively, further highlight the nuanced performance diference.
The variation of the metrics for each class "YES" and "NO" highlights the diference in the ability of the model to predict positive and negative classes. While the model is fairly successful in identifying and predicting negative classes in both the fine-tuned and base versions, it struggles in positive class prediction.
Metric NO YES Accuracy Macro Avg Weighted Avg
Fine-tuned model Precision Recall F1-Score 0.7895 0.7000 0.7420 0.3284 0.4400 0.3761
0.6350 0.5589 0.5700 0.5591 0.6742 0.6350 0.6506
Base model Support Precision Recall F1-Score 150 0.7480 0.6333 0.6859 50 0.2466 0.3600 0.2927 200 0.5650 200 0.4973 0.4967 0.4893 200 0.6227 0.5650 0.5876
In conclusion, the fine-tuned model, with a precision of 78.9%, recall of 70%, accuracy of 74.2%, and F1-score of 55.91% on the test data, demonstrates its superior performance in the given task.
In this study, we employ the VLP framework BLIP coupled with a Multinomial Naive Bayes classifier, achieving promising results through fine-tuning MUSTI data for English. Challenges persist in automating olfactory information extraction due to limited linguistic evidence and implicit image representation. While BLIP excels for English, it falls short for multilingual data, and the use of Naive Bayes captures semantic similarity but struggles with detecting similar olfactory sources. Text tokenization, crucial for semantic understanding, may lead to information loss. As part of future work, we would like to further fine-tune the model in order to improve accuracy for the positive class and we may also explore advanced multimodal architectures and incorporate additional contextual cues to improve the model’s grasp of olfactory references.