Recently, multi-modal systems such as CLIP (Contrastive Language-Image Pre-training) were introduced to represent images and texts jointly in the same embedding space. These models are trained on massive amounts of image-caption pairs and show impressive performance on zero-shot image classification. However, their usage is limited to English due to their training data. Training the same model for diferent languages is non-trivial since the amount of natural data in those might not be suficient, and automatic translations of original captions might not have suficient quality, harming performance. In this paper, we present the first CLIP model for the Italian Language (CLIP-Italian), trained on more than 1.4 million image-text pairs. Results show that CLIP-Italian outperforms a multilingual CLIP model on image retrieval and zero-shot classification tasks for the Italian language.1 Sistemi multimodali come CLIP (Contrastive Language-Image Pre-training) sono stati proposti di recente al fine di ottenere rappresentazioni di immagini e testo in uno spazio latente condiviso. Questi modelli sono allenati su enormi quantità di immagini associate alle loro didascalie, e dimostrano abilità eccellenti nell'efettuare classificazioni “zero-shot”. Ciononostante, il loro utilizzo è limitato all'inglese, la lingua utilizzata durante il loro addestramento. Ottenere modelli del genere per altre lingue non è cosa da poco, poiché la quantità di dati a disposizione per queste lingue potrebbe non essere suficiente e la traduzione automatica delle didascalie inglesi originali potrebbe portare a risultati non soddisfacenti. In questo articolo presentiamo il primo modello CLIP per la lingua italiana (CLIP-Italian), addestrato con più di 1.4 milioni di immagini e rispettive didascalie. I risultati riportati dimostrano l'eficacia di CLIP-Italian per l'estrazione e la classificazione zero-shot in italiano, ottenendo risultati migliori di un modello CLIP multilingue.
# using BERT and ViT to encode raw images and texts encoded_images = clip . image_encoder ( images ) encoded_texts = clip . text_encoder ( images ) # nomalize the projections embedded_images = l2_normalization ( clip . image_projection ( encoded_images )) embedded_texts = l2_normalization ( clip . text_projection ( encoded_texts )) logits = np . dot ( embedded_images , embedded_texts .T) * logit_scale labels = np . arange (n) # correct image - text match is on the main diagonal loss_images = cross_entropy_loss ( logits , labels , axis =0) loss_texts = cross_entropy_loss ( logits , labels , axis =1) loss = ( loss_images + loss_texts ) / 2
CLIP-Italian capabilities and streamline testing.2 be used for zero-shot text-based image retrieval and zeroshot image classification by looking at the similarities Contributions. We create the largest publicly avail- between available texts and images. able multi-modal dataset for the Italian language. We use CLIP-Italian difers from the original CLIP in that enthis dataset to train and release the first CLIP image-text coders are not trained from scratch. We continue training model for the Italian language. We show that this model from checkpoints of other pre-trained models. This apperforms better than its multilingual counterpart in two proach allows us to leverage pre-training knowledge of well-established multi-modal tasks: image retrieval and existing models and remap it to new lexical items to crezero-shot image classification. Moreover, we release the ate CLIP-Italian. We extensively cover training details in model checkpoint, code, and an online demo to showcase Section 4.
CLIP-Italian capabilities.
CLIP’s architecture consists of two distinct encoders,
one for images and one for texts. At training time, all
images and texts in a mini-batch are each projected to
a 512-dimensional space. Next, vector similarities are
computed for each image-text pair, and cross-entropy loss
is applied. Finally, the average loss along the image and
text dimensions is used to update model parameters. The
loss is used to align the two 512-dimensional projection
spaces. Figure 1 briefly summarizes how the contrastive
loss is computed in CLIP. We refer the reader to [
After training, CLIP can be used without further training for a variety of diferent tasks. Since images and texts are embedded in the same space, CLIP embeddings can
https://www.comet.ml/g8a9/clip-italian/reports/clip-italian-train ing-metrics, Demo: https://huggingface.co/spaces/clip-italian/clip-i talian-demo
CLIP-Italian model.
• WIT [11] is a multilingual image-caption dataset collected from Wikipedia. We pre-process and extract the Italian subset, selecting the Reference Description captions as captions of interest. While several possible captions are available, we select those described as the most topical and highestquality captions in the original paper. • MSCOCO-IT [12].3 The captions of this dataset come from the original MSCOCO dataset [13] and are translated with Microsoft Translator. The 2017 MSCOCO training set contains more than 100K images. More than one caption is available for each image. • Conceptual Captions (CC) [14].4 In this dataset, there are more than 3 million image-caption pairs, collected from the web. All images with available URLs were downloaded, and their captions
4https://github.com/google-research-datasets/conceptual-cap tions
Captions 38% 8% 52% 2%
were translated to Italian using DeepL.5, totaling roughly 710K captions. • La Foto del Giorno (ILPOST).6 This image-caption dataset is collected from Il Post, a prominent Ital3.1. Translations Our CLIP-Italian model is based on previous pre-trained state-of-the-art models for both the vision and textual parts. We use Vision Transformer (ViT) [16] and BERTinspired [17] text encoder. We limit the sequence length to 96 tokens and use a local batch size of 128 for each of the 8 TPU cores we used. For the optimization procedure, we used the AdaBelief optimizer [18] with Adaptive Gradient Clipping (AGC) and a Cosine Annealing Schedule [19]. We run training for a maximum of 15 epochs, evaluate at the end of each epoch, and release the checkpoint with the best validation loss.
set due to the low amount of captioned images for Italian compared to the original CLIP training dataset. Instead of relying on open-source translators, we use the proprietary DeepL API to obtain readily available high-quality English captions. While this choice aims to minimize the noise in translated data, we know about the bias (e.g., gender and age) that translation systems introduce during translations [15]. Some of the captions are available Data Augmentation Following standard practices in in Figure 2. computer vision, we applied several augmentations to
To assess the translation quality, three native Italian the available images. In particular, we used random afine speakers among the authors inspected a sample of 100 transformations, perspective changes, occasional equaltranslations alongside their original English sources, rat- ization, and random changes to brightness, contrast, sating translations with scores between 1 and 4. We adopt uration, and hue. Importantly, we made sure to keep hue the following categorization for the provided scores: 1, augmentations limited to allow the model to learn color the sentence has lost its meaning, or it is not possible to definitions. understand it; 2, it is possible to get the idea, but there is something wrong; 3, good, however, a native speaker might complain about some parts of the translation; 4, correct translation.
The average score was 3.78, suggesting that the translations were good on average. We also computed an inter-rater agreement with Gwet’s AC1 using ordinal weighting, obtaining a value of 0.858. This value suggests a strong agreement between annotators.
Projection Layers Warmup Since pre-trained checkpoints were used as starting points for both the vision and the text encoders, we found it helpful to warm-up projection layers. To do so, we first train the entire network using frozen vision and text encoders until loss convergence. After this first phase, the rest of the model is unfrozen to perform end-to-end training. We always pick the model with the best evaluation loss, evaluating every 15 epochs.
6https://www.ilpost.it/foto-del-giorno/
an endless cargo of tanks on a train pulled down tracks in an empty dry landscape un carico infinito di carri armati su un treno trascinato lungo i binari in un paesaggio secco e vuoto person walking down the aisle persona che cammina lungo la navata popular rides at night at the county fair giostre popolari di notte alla fiera della contea
jected to 512-dimensional vectors to which we apply the loss defined in CLIP using logit scaling equal to 20. We empirically observed that logit scaling has a strong positive impact on model performance, suggesting that the embeddings have similar Euclidian norms and that scaling their dot similarities helped the cross entropy.
CLIP-Italian 0.3797 0.5039 0.5204 mCLIP images for each sample’s contribution. If the target image is not within them, we approximate 1/ to 0 (MRR@k).
Table 3 reports the results for the image retrieval task, in terms of MRR@k, where ∈ {1, 5, 10}. CLIP-Italian outscores mCLIP across the board. 5.2. Zero-shot Classification
system explicitly trained for the Italian language. Hence, to provide meaningful comparisons, we compare its performance to an available multilingual CLIP11 model trained with multilingual knowledge distillation [20].
The zero-shot image classification task replicates the
experiment run by Radford et al. [
Reciprocal Rank (MRR) retrieval metric. The metric computes the rank assigned to each image to be retrieved (, where = 1 is best), takes its reciprocal, and averages it across all the dataset samples ( = 1/||· ∑︀|| 1/). We consider only the first retrieved
Similarly to MRR@k, we consider a “hit” if the predicted class is within the top k retrieved and a “miss” otherwise. Similarly to the image retrieval task, we compute accuracy at (Accuracy@k) with ∈ {1, 5, 10}.
Table 4 reports the results for the zero-shot classification task. CLIP-Italian outscores mCLIP across the board.
10https://huggingface.co/openai/clip-vit-base-patch32 11https://huggingface.co/sentence-transformers/clip-ViT-B-3 2-multilingual-v1 5.3. Discussion
22.11
43.69
52.55
tasks we have been testing. Note that the performance
for zero-shot ImageNet classification of the CLIP-Italian
model (trained on 1.4 million image-text pairs) are lower
than those shown in Radford et al. [
task on the Unsplash25K dataset.12 Figure 2 shows the results of the query “due cani sulla neve” (two dogs on the snow), the model correctly finds the image, combining 12https://github.com/unsplash/datasets the concept of “snow” and the one of “two dogs”.13 We anecdotally find moderate numeracy capabilities during empirical evaluation, with suficient ability to identify up to three distinct or repeated elements inside images, with a steep drop in coherence when more than three elements are present. Given the likely low number of training points depicting more than three subjects in a scene, we impute this finding to implicit bias in the training set. Figure 3 shows a similar performance for “una coppia al tramonto” (a couple during sunset), where the model could identify two people with sunlight in the background. A similar query, but with a mountain as a background, can be found in Figure 4. Despite the overall good performances, the model is inevitably subject to limitations and biases. For example, Figure 5 shows an image of a tiny hedgehog retrieved using the query “un topolino” (a tiny mouse). We leave a more thorough exploration of biases and stereotypes learned by the CLIPItalian model to future work.
language, trained on 1.4 million image-text pairs. The model shows promising zero-shot performance in two well-established tasks, suggesting many possible future applications.
Google which provided the computational resources to train CLIP-Italian. This project has also in part received 13Note, however, that compositional understanding in CLIP is limited, see [21]
Large-scale models are dificult and costly to train, and important considerations have to be taken into account when developing them [22, 23]. We computed the cost of the diferent experiments we ran, and we estimated a total of 2,688$ for each TPU used. This result comes from the hourly cost of the TPU (8$) for 14 days; Note that we had access to a second TPU VM for part of the project and that, in this estimate, we are ignoring storage and data transfer costs. Strubell et al. [22] describe how these models can have a substantial environmental impact. As described tational Linguistics (Volume 1: Long Papers), 2023, [16] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weispp. 13028–13043. senborn, X. Zhai, T. Unterthiner, M. Dehghani, [7] Z. Huang, F. Bianchi, M. Yuksekgonul, T. J. Montine, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, J. Y. Zou, A visual–language foundation model N. Houlsby, An image is worth 16x16 words: Transfor pathology image analysis using medical twitter, formers for image recognition at scale, in: InterNature Medicine 29 (2023) 2307 – 2316. URL: https: national Conference on Learning Representations, //api.semanticscholar.org/CorpusID:260970273. 2021. URL: https://openreview.net/f orum?id=Yicb [8] D. Nozza, F. Bianchi, D. Hovy, What the MASK? FdNTTy.
making sense of language-specific bert models, [17] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, BERT: arXiv preprint arXiv:2003.02912 (2020). Pre-training of deep bidirectional transformers for [9] P. Rust, J. Pfeifer, I. Vulić, S. Ruder, I. Gurevych, language understanding, in: Proceedings of the How good is your tokenizer? on the monolingual 2019 Conference of the North American Chapter of performance of multilingual language models, in: the Association for Computational Linguistics: HuProceedings of the 59th Annual Meeting of the As- man Language Technologies, Volume 1 (Long and sociation for Computational Linguistics and the Short Papers), Association for Computational Lin11th International Joint Conference on Natural Lan- guistics, Minneapolis, Minnesota, 2019, pp. 4171– guage Processing (Volume 1: Long Papers), Associ- 4186. URL: https://www.aclweb.org/anthology/N19 ation for Computational Linguistics, Online, 2021, -1423. doi:10.18653/v1/N19-1423. pp. 3118–3135. URL: https://aclanthology.org/202 [18] J. Zhuang, T. Tang, Y. Ding, S. Tatikonda, 1.acl-long.243. doi:10.18653/v1/2021.acl-long. N. Dvornek, X. Papademetris, J. S. Duncan, Ad243. abelief optimizer: Adapting stepsizes by the [10] G. Sarti, M. Nissim, It5: Large-scale text-to-text belief in observed gradients, arXiv preprint pretraining for italian language understanding and arXiv:2010.07468 (2020).
generation, ArXiv abs/2203.03759 (2022). [19] I. Loshchilov, F. Hutter, SGDR: stochastic gradi[11] K. Srinivasan, K. Raman, J. Chen, M. Bendersky, ent descent with warm restarts, in: 5th InterM. Najork, Wit: Wikipedia-based image text national Conference on Learning Representations, dataset for multimodal multilingual machine learn- ICLR 2017, Toulon, France, April 24-26, 2017, Coning, arXiv preprint arXiv:2103.01913 (2021). ference Track Proceedings, OpenReview.net, 2017. [12] A. Scaiella, D. Croce, R. Basili, Large scale datasets URL: https://openreview.net/f orum?id=Skq89Scxx. for image and video captioning in italian, IJCoL. [20] N. Reimers, I. Gurevych, Making monolingual senItalian Journal of Computational Linguistics 5 (2019) tence embeddings multilingual using knowledge 49–60. distillation, in: Proceedings of the 2020 Conference [13] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, on Empirical Methods in Natural Language ProcessD. Ramanan, P. Dollár, C. L. Zitnick, Microsoft ing (EMNLP), Association for Computational Lincoco: Common objects in context, in: European guistics, Online, 2020, pp. 4512–4525. URL: https:// conference on computer vision, Springer, 2014, pp. www.aclweb.org/anthology/2020.emnlp-main.365. 740–755. doi:10.18653/v1/2020.emnlp-main.365. [14] P. Sharma, N. Ding, S. Goodman, R. Soricut, Con- [21] M. Yuksekgonul, F. Bianchi, P. Kalluri, D. Jurafsky, ceptual captions: A cleaned, hypernymed, image J. Zou, When and why vision-language models alt-text dataset for automatic image captioning, in: behave like bags-of-words, and what to do about Proceedings of the 56th Annual Meeting of the As- it?, in: The Eleventh International Conference on sociation for Computational Linguistics (Volume 1: Learning Representations, 2022.
Long Papers), Association for Computational Lin- [22] E. Strubell, A. Ganesh, A. McCallum, Energy and guistics, Melbourne, Australia, 2018, pp. 2556–2565. policy considerations for deep learning in NLP, in: URL: https://www.aclweb.org/anthology/P18-1238. Proceedings of the 57th Annual Meeting of the Asdoi:10.18653/v1/P18-1238. sociation for Computational Linguistics, Associa[15] D. Hovy, F. Bianchi, T. Fornaciari, “you sound just tion for Computational Linguistics, Florence, Italy, like your father” commercial machine translation 2019, pp. 3645–3650. URL: https://www.aclweb.org systems include stylistic biases, in: Proceedings /anthology/P19-1355. doi:10.18653/v1/P19-135 of the 58th Annual Meeting of the Association for 5.
Computational Linguistics, Association for Com- [23] E. M. Bender, T. Gebru, A. McMillan-Major, putational Linguistics, Online, 2020, pp. 1686–1690. S. Shmitchell, On the dangers of stochastic parrots: URL: https://www.aclweb.org/anthology/2020.ac Can language models be too big?, in: Proceedings l-main.154. doi:10.18653/v1/2020.acl-main.15 of the 2021 ACM Conference on Fairness, Account4. ability, and Transparency, 2021, pp. 610–623.