This study, titled the Multimodal understanding of Smells in Texts and Images (MUSTI) task, aims to explore the relationship between textual descriptions and visual depictions of smells. Using machine learning techniques, specifically leveraging the CLIP model and tokenization methods, this research extracts features from both text and images to analyze and correlate olfactory elements.The approach involves a model that computes similarity between textual descriptions and images based on their scent-evoking content. This model engages in classification tasks, determining whether a given textimage pair shares a common smell source (1 for positive correlation, 0 for negative correlation). By calculating similarity scores between text and image features, it quantifies the degree of correlation based on scent-related content, enabling a nuanced understanding of connections between textual descriptions and visual representations of smells. Trained on a dataset of image-text pairs, the model outputs scores for accuracy, providing a foundation for a comprehensive analysis of scent-related associations within multimedia content.] The understanding of smells, an underrepresented dimension in multimedia analysis, serves as the focal point of the MUSTI (Multimodal Understanding of Smells in Texts and Images) task. This task aims to delve into the intricate relationship between olfactory references found in textual descriptions and visual depictions across diferent historical periods and languages. This research work outlines the task's primary objectives: MUSTI Classification, requiring participants to predict shared olfactory sources between texts and images as a binary classification problem. Utilizing the CLIP model and tokenization methods, the model extracts features from images and text and computes the cosine similarity between them based on smell-related content.
In recent years, the study of olfactory dimensions has gained traction, recognizing the
significance of smells in memory and emotions [
For the Multimodal Understanding of Smells in Texts and Images (MUSTI) task[
The MUSTI dataset, a collection of copyright-free texts and images from various repositories and archives, forms the basis for this task. Annotated with over 80 categories of smell objects and gestures, the dataset allows participants to develop models aimed at recognizing and linking olfactory references across languages and modalities.
Initially, the dataset undergoes preprocessing of the provided textual data, comprising multilingual passages in English, German, Italian, and French, using a tokenization scheme implemented through the ’bert-base-multilingual-cased’ tokenizer. To accommodate varying text lengths, the data is divided into chunk of texts that fit within the maximum sequence length compatible with the CLIP model’s input requirements. Simultaneously, images linked to the textual descriptions are acquired by parsing the provided URLs. These images undergo resizing and conversion to numerical arrays for further processing.
To create feature representations conducive to model interpretation, a combination of pixellevel information from images and token embeddings derived from textual segments are utilized by our model. Image arrays are flattened, while textual segments are tokenized and flattened, yielding joint feature representations that capture the essence of both modalities.
The proposed work employs the CLIP model which is designed to predict the image and text pairings. CLIP’s embeddings for images and text share the same space and the CLIP’s layers leverage a large-scale pretrained model which enables us to encode semantic relationships between text and images. This allows for cross-modal understanding, aiding in associating textual descriptions of smells with relevant visual representations, facilitating multimodal comprehension of olfactory concepts. The CLIP loss aims to maximize the cosine similarity between the image and text embeddings for the N genuine pairs in the batch while minimizing the cosine similarity for the N² - N incorrect pairings. This similarity measure serves as an indicator of the correspondence or shared olfactory source between the textual descriptions and the accompanying images.
The cosine similarities between text and images were computed and also an average similarity score across all segmented text chunks associated with each image is derived, providing a comprehensive assessment of the overall olfactory connection between text and image pairs. The average similarity scores are benchmarked against a threshold value to determine the presence or absence of a shared olfactory source between text and image pairs. By assigning binary labels based on the similarity threshold, the classification task (MUSTI Classification) for recognizing the co-relation of smells across modalities is facilitated.
A series of experiments was conducted, iterating over diferent stages of data preprocessing, feature extraction, and model utilization to achieve comprehensive olfactory understanding across text and image modalities. The proposed system was evaluated using precision, recall, and F1-score metrics and with overall accuracy. The results are depicted in the below table 1.The accuracy was found out to be 61.38%.
On further attempts, exclusion of title in processing the similarity led to a better accuracy of 63.47%. This could imply confusion caused due to the inclusion of title. The results of the was depicted in the following table 2
Precision
Recall F1-Score
Support
The achieved accuracy of 63.47% in predicting the presence (’YES’) or absence (’NO’) of common smell sources between text passages and images demonstrates a moderate level of success in our model’s performance. The precision, recall, and F1-scores for each class (’YES’ and ’NO’) indicate notable diferences in the model’s ability to predict positive and negative instances. The model displays a relatively higher precision of 80.75% for identifying cases where there are no common smell sources and a score of 61.54% which indicates that it misses a considerable number of actual instances where there are no common smell sources. The F1-score of 60.84% suggests a fair balance between precision and recall for the classes.
To refine the model, options include further fine-tuning CLIP and optimizing feature engineering techniques for better discernment of nuanced textual and visual olfactory relationships. Augmenting and enriching the dataset with diverse textual genres, images from various historical periods, and more smell-related annotations can significantly enhance the model’s comprehension of olfactory references across diferent contexts and time frames.