The Multilingual Image-Text Classification (MITC) task is a specific instance of the Image-Text Classification (ITC) task, where each item to be classified consists of a visual representation and a textual description written in one of several possible languages. In this paper we proposeMM-gFun, an extension of thegFun learning architecture originally developed for cross-lingual text classification. We extend its original text-only implementation to handle perceptual modalities.
(F. Sebastiani) CEUR ceur-ws.org
Generalized Funnelling g(Fun) [
VGFs are language-dependent functions that map (monolingual) documents into
languageindependent vectorial representations (i.e., views) aligned across languages. Since each view is aligned
across languages, they can be aggregated into a single representation that is also aligned across
languages, and which can be thus fed to the meta-classifier. In gFun, for each language a set of VGFs is
instantiated to leverage diferent types of information that can be brought to bear on the training process.
In [
While the original gFun was developed for a unimodal setting (i.e., involving only textual data), in this article we extend it to be deployed in a multimodal setting involving images as well as text written in diferent languages. This is achieved by augmenting the set of VGFs originally proposed with a Visual VGFs. This module is designed to explicitly mine the correlations between the images and the target classes, which remained out of reach to the other VGFs.
The training of MM-gFun is a two-phase training. In the first phase, all the VGFs of the first-tier are tuned to produce the posterior probabilities representations of images and texts. In the second-phase, output representations of the first-tier are aggregated according to a pre-defined policy and subsequently used to train the meta-classifier. By leveraging the shared (modality-agnostic) space of the posterior probabilities, the meta-classifier can be trained on the entire collection of items, be they images or text, and independently of the textual description language.
To encode the image-class correlations, we derive image representations from a Vision Transformer (ViT). o deal with the continuous input of images, the ViT divides the image into fixed-size patches. The patches are then flattened into vectors and linearly projected to a higher-dimensional space, serving as input visual tokens for the transformer. Similarly to transformer model for texts, positional embeddings are added to the visual tokens to account for the relative positions of patches.
Dataset We base our experiments on GLAMI-1M [
From the original GLAMI-1M, we extract a smaller version with a controlled distribution of languages, both for the training and in test sets. For each language, we sample 15,000 documents in order to represent higher-resourced (or “source”, in the context of HTL) languages ℒ( = {Bulgarian (bg), Czech (cz), Hungarian (hu), Lithuanian (lt), Slovak (sk), Turkish (tr)}), and we sample 5,000 documents in order to represent lower-resourced languages (ℒ = {Estonian (ee), Spanish (es), Greek (gr), Croatian (hr), Latvian (lv), Romanian (ro), Slovenian (sl)}). Note that for for Greek, Spanish, Romanian, and Latvian languages there are fewer training examples (respectively: 3405, 2434, 3533, and 4184); in these cases we simply take them all. The final reduced dataset consists of 100,000 training items and 58,553 test items. The class distribution in the reduced version closely resembles the one of the original dataset after the sampling. This reduced version facilitates a controlled analysis of the ability to transfer knowledge from higher-resourced to lower-resourced languages.
Training Details We initialize the textual VGFs of MM-gFun from pre-trained models mDeBERTa-v3. The textual input is processed via the relative SentencePiece tokenizer, with a maximum sequence length of 32 tokens to ease the comparison with results obtained in 2[]. For the visual VGF, we initialize the component from CLIP visual encoder. Visual images are center cropped to square features of 224 pixels. All the learners in the first-tier and the metaclassifier are optimized to minimize the cross entropy loss, with AdamW and cosine annealing learning rate scheduler. We set the learning rate to0.0001 in all our experiments, and train all of the models for a maximum of 25 epochs, with a early-stopping set to 5 epochs without any increase on the accuracy on the validation set.
Results We report the results that we have obtained using the original GLAMI-1M dataset in its
entirety as well as those obtained in the reduced setting, where we balance the number of documents
across the diferent languages. In order to allow for a direct comparison with [
Few- 58.95 N/A 72.35 72.40 48.61 47.71 46.55 N/A N/A N/A 71.43 47.25 75.05
Few- 57.94 N/A 71.50 71.70 47.68 47.35 72.13 N/A N/A N/A 72.26 48.35 74.26 The results indicate that multilingual textual information encodes a stronger signal for the classification task. Indeed, multilingual models achieve strong performance, with top-1 and top-5 accuracy around 83%, significantly outperforming vision-only models, which achieve top-1 of 63 and 71%, respectively. Multimodal baselines such as AltCLIP, m-CLIP, and EmbraceNet show similar performance, but they do not measure up in terms of accuracy when compared with the text-based multilingual models. The proposed method MM-gFun achieves the best results in terms of top-1 accuracy when leveraging both textual and visual inputs. This, along with the fact thatMM-gFun improves upon its individual components (CLIP-ViT and mDeBERTa), demonstrates the ability of the method of leveraging both modalities. Consistently with our results on the full dataset, we observe that multilingual models outperform visual ones also in the few-shot setting (denoted by the “Few-” prefix), in terms of micro- and macro-averaged accuracy across languages. However, in this data-scarce scenario, all our multimodal baselines fall short in terms of performance when compared to MM-gFun and CLIP-ViT, by also exhibiting a significant drop in accuracy with respect to the data-rich scenario with respect to the other approaches. Here, MM-gFun achieves good results, improving over the unimodal textual-component by no less than 2.5 points and by 16 points with respect to the visual component.
In this paper, we have presented MM-gFun, an architecture for Heterogeneous Transfer Learning (HTL) in the task of multilingual image-text classification. This architecture is an extension of the unimodal multilingualgFun. By augmenting the set of Visual Grounding Functions (VGFs) with a specific module to represent input images, we demonstrated improved performance on multilingual image-text classification tasks. Our approach achieves superior performance compared to MM-gFun’s internal components when trained independently. We validated our hypothesis through experiments on GLAMI-1M, a multilingual and multimodal dataset of fashion product images with textual descriptions in 13 diferent languages.
This work has been supported by the FAIR and SoBigData.it projects, funded by the Italian Ministry of University and Research under the NextGenerationEU program. The authors’ opinions do not necessarily reflect those of the funding agencies.
During the preparation of this work, the authors used ChatGPT in order to: Grammar and spelling check.