The introduction of the CLIP model signaled a breakthrough in multimodal deep learning. This paper examines whether CLIP can be fruitfully applied to a (binary) classification task in the Humanities. We focus on a historical collection of late-nineteenth century magic lantern slides from the Lucerna database. Based on the available metadata, we evaluate CLIP's performance on classifying slide images into 'exterior' and 'interior' categories. We compare the performance of several textual prompts for CLIP to two conventional mono-modal models (textual and visual) which we train and evaluate on the same stratified set of 5,244 magic lantern slides and their captions. We find that the textual and multimodal models achieve a respectable performance (∼0.80 accuracy) but are still outperformed by a vision model that was fine-tuned to the task ( ∼0.89). We flag three methodological issues that might arise from the application of CLIP in the (computational) humanities. First, the lack of (need for) labelled data makes it hard to inspect and/or interpret the performance of the model. Second, CLIP's zero-shot capability only allows for classification tasks to be simulated, which makes it doubtful if standard metrics can be used to compare its performance to text and/or image models. Third, the lack of efective prompt engineering techniques makes the performance of CLIP (highly) unstable.
Following the development of deep learning models that are trained on expressions of a single
sensory modality, mostly hearing (text) and seeing (images), researchers have recently focused
on multimodal applications: models that process and relate information from multiple
modalities [
In January 2021, the introduction of the CLIP (Contrastive Language-Image Pre-training)
signaled a breakthrough in the field of multimodal machine learning [
While computer vision models have frequently been reported to outperform humans, they
are optimized for performance on the specific task and data of the benchmark. As a result,
their performance cannot be compared to the highly-contextual vision of humans [
Building on recent discussions about the ’visual digital turn’ [
Based on the available metadata for the slides, we evaluate CLIP’s performance on
recognizing images of exterior/interior scenes. Seemingly purely visual in nature, multimodality theory
would argue that text, such as captions, play a crucial role in producing these categories [
We flag three methodological issues that might arise from a possible widespread application
of CLIP in the (computational) humanities. First, the lack of (need for) labelled data makes it
hard to inspect and/or interpret the performance of the model. Second, even if labelled data is
available, CLIP’s zero-shot capability only allows for classification tasks to be simulated. As a
result, it is doubtful whether accuracy and other standard metrics can be used to meaningfully
compare CLIP to text and/or image models. Finally, the lack of methods to find the right, let
alone the optimal, textual prompt(s) makes the performance of CLIP (highly) unstable. As
a result, ‘prompt engineering’ [
This paper is part of the larger History of Implicit Bias project at the University of Antwerp, which applies machine learning to identify patterns of (implicit) bias in several nineteenth century digital collections. Multimodal machine learning could provide a breakthrough for this kind of research, which seeks to analyze large-scale and complex patterns of meaning in (historical) data. Models like CLIP could not only ofer researchers the opportunity to study categories, such as ’the family,’ that are highly multimodal in nature, but also, in conjunction with mono-modal techniques, fleece out the distribution of diferent modalities in meaningmaking. This exploratory paper tests the robustness of CLIP to provide a sound basis for such research in the future.
The study of the magic lantern has been stimulated by the increasing digital accessibility of
lantern slides. The Lucerna Magic Lantern Web Resource was the first digital repository of
digitized lantern slides. At the time of writing, it contained 42,019 digital slides, up from 38,000
in 2019, most of them uploaded and annotated by Lucerna’s founder Richard Crangle [
To compare the performance of CLIP to mono-modal models on the exterior/interior classification task we used the ‘type of image’ field to produce a stratified .60/.20/.20 train, validation and test set of exterior and interior images with captions. As Table 1 shows, Lucerna’s slides were manually labelled for several types describing the physical setting captured on the slide. We combined the types ‘photograph of exterior location’ and ‘photograph of life models in exterior location’ to collect slides showing exterior locations (Fig. 1) and the ‘photograph of interior location’ and ‘photograph of life models in interior location’ types to collect slides of interior locations (Fig. 2). Initially, we also included the ‘photograph of life models in studio set’ in the collection of interior slides. However, as Fig. 3 and Fig. 4 show, this category often contains fictional ‘outdoor’ scenes 1 This demonstrates that seemingly binary categories, such as outdoor/indoor, often prove to be far-less rigid in actual practice. To enable comparison to a purely textual model, we only included slides with captions, discarding those without captions or with frequently recurring or generic ones, such as ‘Intro(duction)’ or ‘Title’. To create a balanced set, we included all the remaining slides of the interior category (2,622) and an equally-sized random sample of slides from the exterior category (5,244 total).
We compared the zero-shot performance of CLIP for several (apparently) binary prompts (Table 2) to a visual and a textual model (Table 3). The main advance of CLIP is that it does not need labeled training data to achieve competitive performance on a wide variety of classification tasks. However, this zero-shot capability results in the fact that we can only simulate a classification task. First, textual prompts have to be picked that are (apparent) mutually exclusive terms, phrases, or sentences. However, this does not exclude the possibility that both prompts are (un)likely textual descriptions of the same image. In contrast to models that are trained for a binary classification task, we do not ask CLIP a single question (Is this A or B?) but rather normalize the answers to two questions (Is this A?/Is this B?). Following earlier work, to calculate the accuracy of CLIP on a classification task, we use the softmax function to normalize the output of the model for the two prompts into a single probability distribution. While most deep learning models use softmax to normalize the output into a probability score, we ague that its application is conceptually diferent in the case of CLIP.
To compare CLIP’s zero-shot capabilities to mono-modal models we used relatively
simple transfer learning methods. For the vision model, we applied the fast.ai framework to
train a ResNet 18, a relatively simple convolutional neural network, pretrained on the
ImageNet dataset. Instead of manually selecting hyperparameters, for example by determining
the learning rate, we resorted to fast.ai’s default finetune method and its default parameters
(for four epochs). For the text-only model, we first used a run-of-the-mill text classification
approach [
1Copyright of Figures 1-4. Reproduced by permission via Lucerna Magic Lantern Web Resource. Figure 1: Private collection. Digital image © 2016 Anke Napp. Figure 2: The Hive. Digital image © 2018 Worcestershire County Council. Figure 3: Philip and Rosemary Banham Collection. Digital image © 2016 Philip and Rosemary Banham. Figure 4: Private collection. Digital image © 2006 Ludwig Vogl-Bienek.
We supplemented this, potentially naive, classifier with a generic, pretrained BERT for sequence classification. We started from the uncased, multilingual model from the Transformers library, which we finetuned as a binary exterior/interior classifier on the training set (we monitored on the development set via early stopping) and evaluated on the test set. The motivation for this was twofold. First, because we could bootstrap from a pretrained model, we expected the model to be able to model more subtle semantic aspects of the textual descriptions that aren’t obvious from the lexical surface level (e.g. synonyms). Second, we started from the multilingual model that is available for this architecture: because our data is not exclusively monolingual, which could have given the BERT classifier a modest edge. A drawback of this neural approach is that model criticism through feature inspection is less straightforward.
accuracy on exterior accuracy on interior accuracy on all
Starting with the performance of CLIP,
Table 2 shows that diferent prompts lead to
diferent accuracy scores. From ∼0.96 for
‘indoors’ in the ‘outdoors/indoors’ prompt, to
worse then guessing: ∼0.49 for ‘outdoor’ in
the ‘outdoor/indoor’ prompt. Similar to the
‘prompt engineering’ discussions surrounding
GPT-3 [
The diference in accuracy between ‘outdoor’ and ‘outdoors’ (Table 2) is a good example of this.
In relation to prompt engineering, Radford
et al. (2021) note that images are rarely
paired with a single word [
The limited increase in accuracy of adding
‘a photograph of’ to the prompts might be partly a result of the ‘temporal bias’ [
Looking at Table 2 we hypothesized that combining high performing words or snippets from diferent prompts might lead to better results. However, this is not the the case. While ‘exterior’ achieves high accuracy in the ‘exterior/interior’ prompt, its performance drops when combined with ‘indoors,’ which achieved high accuracy in the ‘outdoors/indoors’ combination and experiences an even more dramatic drop in accuracy when combined with ‘exterior’ (Table 2). This process can be explained by the fact that we normalize the output of the model for two prompts into a single probability distribution.
Regarding the textual models, a number of observations can be made. First, they score on par with the multimodal model, which is striking because the latter was trained nor finetuned on this specific dataset and task. Second, the visual model outperforms the textual models, suggesting that the textual modality is less relevant for this classification task. Interestingly, the word unigram model outperforms that based on character trigrams: this is an atypical result for a common text classification task and suggest that most of the useful cues in the title data is actually realised at the simple lexical level of atomic tokens. The visualization (Fig. 5) of the word unigram model’s highest weights for either class supports this hypothesis. Apart from the telltale feature ‘interior’, the indoor vocabulary is dominated by lexis related to the interior of church buildings (‘misericord’, ‘nave’, ‘choir’, etc.) – Exeter cathedral, in particular, might be over-represented in the data. The outdoor vocabulary, on the other hand, clearly points to more panoramic, landscape-related or aquatic (e.g. ‘bridge’, ‘lake’, ‘canal’, ‘harbour’) features or urban scenery (e.g. ‘street’, ‘town’, ‘gate’). The fixed expression ‘view from’ is also recognized by the model as a powerful lexical predictor of the exterior category. The fact that clear lexical clues are doing all the hard discriminatory work is also the suggested by the unimpressive performance of BERT: given its pretrained nature, in spite of the limited size of the training data set, we expected BERT to be able to harness at least some its pre-existing linguistic knowledge, but that hardly seems to be the case. Concerning prompt engineering, we hypothesized that highest weights for the two classes might result in relevant prompts for CLIP. However, as Table 2 shows, the combination street/interior does not lead to particularly good results.
Next to looking at the accuracy metric, we can use the top errors of CLIP and the visual model to compare them (Fig 6a). Clearly, the models have difficulties with diferent kinds of slides. The errors of the vision model seem the result from a lack of sky. The top error of CLIP is a result of mislabeling. While its caption (’in a Javanese home’) suggest the interior category, the image shows a family outside their house. CLIP wrongly attributed the other images to the exterior category, while they show details inside Exeter cathedral.
Multimodal models hold the promise to lead to a ‘practical revolution’ in computational
humanities research [
We would like to thank Ruben Ros for his help with the data collection and Melvin Wevers for helping setting up CLIP.