Correlation between olfactory information and human memory allows images and texts to rely on their content to separate from human olfactory cells and create imaginary olfactory experiences for humans. This means that images and text may contain equally rich olfactory information, and utilizing this olfactory information is inevitably limited by the distribution characteristics of image or text data, such as language gaps and long-tail distribution problems. To this end, this paper proposes a method based on target detection, which models similar olfactory information contained in images and texts into the same feature space, bridging the cross-language and cross-modal gaps, and adopts data augmentation and special sampling strategies respectively to alleviate the language imbalance of text data and the long-tail distribution of image objects.
In this paper, we delve into the MUSTI task of MediaEval2023[
The texts in the development set are composed of English, French, German, and Italian, but the proportions of these four languages are uneven (en: 795, fr: 300, de: 480, it: 799). On the other hand, the images in the development set are mostly European medieval paintings, and the content contains a large number of objects of diferent classes, such as fruits, animals, portraits, jewelry decorations, etc. There is a long-tail distribution phenomenon which causes the model’s detection performance for tail classes to drop sharply. Existing image and text retrieval research rarely involves olfactory information, so we build the model based on the characteristics of the data set and combined with traditional target detection algorithms.
As shown in Figure 1, our proposed method is divided into Stage 1 for coarse-grained matching
and Stage 2 for fine-grained matching. The first stage is used for coarse-grained classification,
that is, to detect whether there are similar olfactory objects in images and texts, using
multilingual BERT (m-BERT)[
The second stage is used for fine-grained classification, that is, detecting which objects cause similar olfactory experiences in images and texts. We counted the subtask 2 labels of all samples, and categorized objects observed across all images into 77 classes for marking bounding boxes for training YOLOv5. A hyperparameter which means an image contains objects whose total occurrence frequency in all images is less than was introduced to alleviate long-tail distribution problems in theses classes. The performance of YOLOv5 trained on training sets divided with diferent values is shown in Section3. The text feature extractor and image feature extractor trained in the first stage will be used as the initial state of the second stage to extract noun features and image features in each bounding box respectively. A pair of text features and image features of the same category are regarded as Positive samples, otherwise they are regarded as negative samples. These feature pairs will then be fed into the classifier to calculate the binary cross-entropy loss ℒ .
Images in the dataset had several obvious content categories: food, flowers, animals (including prey and herds), water-related (such as the sea, ports, rivers, whales, fish, etc.), jewelry decoration, personal portraits, crowd gatherings, etc. Most of these categories have significant frequency of specific objects and significant olfactory characteristics, so we respectively converted the images to gray scale images or retained the original color content, and respectively used fine-tuned ResNet50 as classifier and K-means algorithm for clustering, but all the performances were poor. In Stage 1, the text feature extractor of the model uses a variety of oficial HuggingFace oficial versions and the performance comparison is shown in Table 1.
In Stage 2, we select training samples through each time and then randomly select samples to ensure that the training set:validation set is 8:2, and there is at least one sample for each category in the validation set. When =0, the training set is completely randomly selected. The proportion of selected samples in the development set is 50.92% when =40, while it is 80.63% when =90. The impact of diferent values of super parameter on the performance of YOLOv5 in Stage 2 is shown in Figure 2. The larger the value, the more categories can be ensured to appear stably in the training set. However, a too large value will also cause some tail categories to appear too few times in the validation set, causing the randomness of the validation results to increase. The overall F1 score of YOLO on the head categories gradually increases as the value increases, and roughly converges when =86. But as the value increases to 90, the F1-score of the head class drops significantly. The overall detection F1 score on the tail category first increases and then decreases as the value increases, reaching the overall optimum when k=87. For extreme tail categories, a smaller value can result in a lower F1 score, while a slightly larger value will cause YOLO to no longer be able to detect extreme tail categories. Therefore, we finally select =87 to take into account the performance of all head and tail categories. In addition to text data, we also tried data augmentation on image data in Stage 2. We used Stable Difusion to try to generate some medieval-style meals and portraits, but they all have visible diferences from the real medieval paintings in the development set. Besides, it is dificult to establish efective tail data augmentation for paintings with various styles and clutter based on afine transformation. For examples, an apple may look like a peach after the color is changed, while a candle is dificult to detect after rotation because the candles in other samples are vertical and the pipes are usually in the form of smoking slanted strips.
The comparisons of the F1 score of three subtasks are shown in Table 2. The overall structure
of Yi et al. [
On subtask 2, the proposed method is far inferior to [
Thanks to the organizers of the MediaEval2023, especially to those organizers for MUSTI. This work was supported in part by the Joint Project for Innovation and Development of Shandong Natural Science Foundation (ZR2022LZH012) and Joint Project for Smart Computing of Shandong Natural Science Foundation (ZR2020LZH015).