This paper explores the incorporation of olfactory data into multimodal language models, a relatively under-explored area in computational linguistics. We tackled the challenge of detecting olfactory stimuli in text and images, with a particular emphasis on multilingual contexts. Our approach involved enhancing the Large Language and Vision Assistant (LLaVA) model, through fine-tuning with a specialized dataset of around 2500 image-text pairs. By leveraging the open-source nature of LLaVA and the resource-eficient ifne-tuning techniques such as Low-rank Adapter (LoRA), our study aims to contribute to the broader exploration of adapting language models to previously under-researched sensory modalities, such as olfaction.
The field of multimodal machine learning has predominantly concentrated on text and image data. This focus is primarily because eficient representation techniques are readily available for these modalities. Conversely, other sensory dimensions, such as olfaction and gustation, have received less attention. This can be attributed partly to the challenge of incorporating their complex chemical structures into machine learning frameworks. However, it is important to note that these less-explored modalities are also implicitly present in both text and images, a facet that has been largely overlooked. The Multimodal Understanding of Smells in Texts and Images (MUSTI) addresses this gap by encouraging research in the detection of olfactory sources through texts and images in a multilingual context.
In this paper, we outline our contribution to MUSTI 2023 [
Prompt Determine if the following text and image share common elements, with a specific focus on smell sources. Look for entities such as objects, animals, fruits, or any other elements that could be potential sources of smells. Answer YES or NO. Image: <image> Text: <text> Determine if the following text and image share common elements, with a specific focus on smell sources. Look for entities such as objects, animals, fruits, or any other elements that could be potential sources of smells. If you identify any such common elements, please list them in the same language as the provided text. If no common elements are found, simply respond with ’No common elements identified.’ Image: <image> Text: <text>
Prior research in olfaction through natural language processing is limited. [
The closest line of research is last year’s shared task [
3.1. Model
Our approach employs LLaVA, which is a general-domain multimodal conversation model[
To optimize LLaVA for the designated subtasks, we transform the existing data into a format suitable for instruction-tuning. The specific prompts used in our final model is provided in Table 1. Despite the related nature of these subtasks, we ensured each prompt was self-contained. Therefore, from each text-image pair, two training instances were created. An important finding during this phase was the need to guide the model to list objects in the same language as the input text, as it tended to default to English otherwise.
During the fine-tuning, instead of updating the entire model, we used LoRA which greatly
reduces the number of learnable parameters by freezing the model and learning much smaller
projection matrices between layers [
For our study, we followed the LLaVA model’s oficial GitHub hyperparameters 2, experimenting with various LoRA settings but found no significant performance diferences. Consequently, we chose a rank and alpha value of 16. We fine-tuned the LLaVA-13B model for three epochs on our dataset, noting no further gains with extended training, likely due to dataset size limitations. The fine-tuning was completed in under 3 hours using a 40GB A100 GPU at a batch size of 4. 3.2. Data The MUSTI 2023 dataset comprises pairs of texts and images, selected to evoke olfactory experiences and sourced from historical archives. The dataset encompasses four languages: English (EN), German (DE), French (FR), and Italian (IT), and contains a total of 2,374 image-text pairs. However, the data is unbalanced, with only 593 pairs annotated as positive, meaning the existence of at least one common smell source. We created an in-house development set, constituting 10% of the total data, ensuring a similar distribution of positive and negative examples across all languages. The development set is primarily used for hyperparameter tuning and prompt tuning. Table 2 details the distribution of these pairs in the training and development sets, highlighting the number of positive and negative samples in each language.
In this section, we present and discuss the results of our models on the in-house development set allocated during the training phase, as well as the oficial results on the test sets. We participated in all three subtasks, namely i) identification of whether or not text passages and images evoke the same smell source, ii) listing them, and also iii) performing the same tasks for another language in a zero-shot setting.
During the development phase, we evaluated both the LLaVA-7B and LLaVA-13B models, before and after fine-tuning, as shown in Table 3. These models were assessed across multiple languages. The F1-macro scores revealed significant enhancements following fine-tuning. Notably, the fine-tuned LLaVA-13B model achieved a remarkable overall F1-macro score of 0.882, with particular improvements in recognizing positive samples. This suggests that the base models initially lacked the necessary knowledge for detecting potential smell sources. The LLaVA-7B model also showed competitive performance, especially considering its smaller size. 2https://github.com/haotian-liu/LLaVA/blob/main/scripts/v1_5/finetune_task_lora.sh Thanks to LoRA, the computational cost of fine-tuning was minimal, making the switch to a larger model have almost no tangible efect on the resources needed.
However, the most significant improvement was observed in subtask 2. On the development data, we noted that the plain LLaVA models scored almost 0 F-score for positive examples, i.e. when the evaluation ignored the correct classification of no common objects. This was because the models either failed to provide any list or listed all objects in the images. After fine-tuning, the performance of LLaVA-13B drastically improved to an F-score of 0.61, indicating that the model learned to discern which objects needed to be detected.
In the oficial results (Table 4), the fine-tuned LLaVA-13B model showed balanced macro precision and recall on both test and test-zero sets, performing well in identifying negative samples. The F1-Scores reached 0.776 for Subtask 1 and 0.698 for Subtask 2 in the test set. In the zero-shot scenario, performance dropped to 0.65 and 0.538 for Subtask 2, respectively. However, this is still promising, especially considering that Slovenian was not included in the pre-training or fine-tuning phases. These results highlight the fine-tuned LLaVA model’s eficacy in recognizing olfactory data, marking a notable advancement in the capabilities of multimodal language models.
In our research, we delved into the less-explored territory of integrating olfactory data into multimodal language models. Using the LLaVA model, we focused on recognizing olfactory cues in a diverse range of texts and images. By fine-tuning LLaVA with around 2500 image-text pairs and employing the Low-rank Adapter (LoRA) method, we achieved notable enhancements in the model’s ability to detect olfactory stimuli. We believe that our findings highlight the potential of multimodal language models in processing sensory information beyond conventional texts and visuals.