Image captioning is a deep learning task that involves computer vision methods to extract visual information from the image and also natural language processing to generate the result caption in natural language. Image captioning models, just like other deep learning models, need a large amount of training data and require a long time to train. In this work, we investigate the impact of using a smaller amount of training data on the performance of the standard image captioning model Oscar. We train Oscar on diferent sizes of the training dataset and measure its performance in terms of accuracy and computational complexity. We observe that the computational time increases linearly with the amount of data used for training. However, the accuracy does not follow this linear trend and the relative improvement diminishes as we add more data to the training. We also measure the consistency of individual sizes of the training sets and observe that the more data we use for training the more consistent the metrics are. In addition to traditional evaluation metrics, we evaluate the performance using CLIP similarity. We investigate whether it can be used as a fully-fledged metric providing a unique advantage over the traditional metrics; it does not need reference captions that had to be acquired by human annotators. Our results show a high correlation between CLIP with the other metrics. This work provides valuable insights for understanding the requirements for training efective image captioning models. We believe our results can be transferred to other models, even in other deep-learning tasks.
is not only one correct caption for an image. This is
beImage captioning is a task in computer vision that in- cause diferent individuals may consider diferent aspects
volves generating a textual description of an image. The of an image to be important, and they may therefore
degoal is to provide a comprehensive and human-like de- scribe the image in diferent ways. Because of this, there
scription of the content of an image, which can be useful is not one ideal evaluation metric that can be used to
for a variety of applications, such as enabling individu- measure the quality of a generated caption, as diferent
als with visual impairments to better understand visual metrics may be better suited for evaluating diferent
atinformation, improving the accuracy and relevance of tributes of the caption.
image search results, etc. It is a complex task because The general problem of deep learning is that it
reit requires the identification and interpretation of visual quires a large amount of data and the training process
information, as well as the generation of grammatically can be computationally intensive. In this work, we
invescorrect and fluent sentences. This requires a combined ef- tigate the relationship between the size of the training
fort of computer vision and natural language processing dataset and the performance of a standard image
captionmethods. ing model, Oscar [
The scientific community has been interested in this of the training dataset and measure the performance by
task for over a decade [
© 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License linear dependence. We also provide insight into the reCPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g ACttEribUutRion W4.0oInrtekrnsahtioonpal (PCCroBYce4.0e).dings (CEUR-WS.org) lationship between the size of the training dataset and its performance on selected metrics. Furthermore, we such as Conceptual Captions and Conceptual12m, may measure the consistency of the data for each of the met- depend on the filtering applied during collection, and rics used, and we expect that smaller subsets of the data their consistency may be harder to guarantee. These will have higher variance than larger subsets. Our re- datasets, however, ofer a larger number of images and sults will help to better understand the requirements for a greater variance. As a result, state-of-the-art image training efective image captioning models and the po- captioning models often utilize a combination of multiple tential trade-ofs between dataset size and performance. datasets in order to achieve the best performance. In this Additionally, our findings may be useful for researchers work, we chose to use the COCO Captions dataset for our and practitioners who are interested in optimizing the experiments due to its suitable size for training and also training of deep learning models in general. dividing into subsets. The COCO Captions dataset also
In addition to using state-of-the-art evaluation metrics, has a suficient number of images to allow for a robust
we also evaluate our image captioning methods on CLIP evaluation of the model’s performance.
(Contrastive Language-Image Pre-training) similarity [
We investigate whether CLIP can be used as a full-fledged 2.2. Evaluation
evaluation metric for image captioning. We find that it
has a major advantage over traditional metrics: it does The evaluation of image captions is a challenging task
not require reference labels from annotators. This means due to the inherent subjectivity of language and the
multhat CLIP can be used to evaluate image captioning mod- tiple ways in which an image can be correctly described.
els in an unsupervised or self-supervised manner, which Most evaluation metrics for image captioning compute
can be useful in situations where annotated data is not the diference between a candidate caption and a
referavailable or is too expensive to obtain. ence caption provided by human annotators. Traditional
metrics, such as BLEU [
One well-known dataset for image captioning is pares n-grams in candidate and reference captions.
HowFlickr30k [
Each of these datasets has its own advantages and semantic space. The cosine similarity can then be used
disadvantages. For instance, the Flickr30k dataset has a to compute the agreement between the input text and
good consistency and is well-suited for evaluation due to the image. Also, difusion models for generating images
the multiple reference captions provided for each image. use CLIP [16] to evaluate the generated image based on
It is a valuable feature because a single image can often text input. In image captioning, CLIP can be used to
be described in multiple ways, and it is useful to have a evaluate the generated caption. Although CLIP has not
diverse set of captions for each image to better capture been considered a standard evaluation metric for image
the range of possible descriptions. However, the quality captioning, in this study we present it as such. In this
of datasets containing images collected from the internet, study, we present it as a potential fully-fledged metric
that thoroughly assesses the semantic quality of
candi1% sub01 a group of brown cows standing in a field 25% sub01 a cow that is laying down in the grass.
1% sub02 a group of cows that are standing together. 25% sub02 a cow is standing in a field with another cow behind it.
1% sub03 a group of cows are standing in the grass. 25% sub03 a cow is standing in a field with another cow.
1% sub04 a herd of black and white cows in a field. 25% sub04 a cow with a red ear tag standing in a field.
1% sub05 a group of cows stand together in a grassy area. 25% sub05 a black and white cow standing in a field.
1% sub06 a herd of cows standing in a field. 25% sub06 a cow is standing in the grass with another cow behind it.
1% sub07 a group of cows grazing on a field. 25% sub07 a cow is standing in a field with another cow behind it.
1% sub08 a group of brown cows laying in a field 25% sub08 a cow is standing in a field of grass.
1% sub09 a couple of cows standing together in a field. 25% sub09 a cow is standing in a field with other cows.
1% sub10 two cows in a field with a fence surrounded by green grass. 25% sub10 two cows are laying down in a field.
date captions. We compute the correlation between CLIP tivation for using this specific method is that we have
preand other metrics and investigate whether CLIP can be viously used it in our own experiments and found it to be
used in this manner. A previous research study [17] has a convenient method to use. While it may not currently
conducted similar experiments, but focused on comput- be the best-performing model, Oscar is a
transformering the correlation with human judgment and comparing based method and we believe that the results of our
experit to correlations with other metrics, whereas we compute iments may be generalizable to other transformer-based
correlations with other metrics directly. or deep-learning models in the field.
To assess the performance of Oscar, we conducted two
2.3. Image Captioning Methods main experiments. The first experiment involved
measuring the time needed to train the model using various
Recent advances in image captioning have seen the amounts of data while tracking the performance on a set
widespread adoption of deep learning techniques. Early of chosen evaluation metrics. In addition to traditional
methods used convolutional neural networks (CNNs) as metrics, we also evaluated the model using CLIP
simiencoders, such as the model proposed by [18]. More re- larity [
Our experiments are based on the training and
evaluation of the image captioning model Oscar [
The authors of Oscar provide a demonstration dataset
3. Experiments of feature vectors and object tags that can be used as
input to Oscar, but do not specify the method by which
In this work, we investigate the performance and efi- these object detections are obtained. In order to generate
ciency of the image captioning method Oscar [
1–8
dataset, we developed a full pipeline that takes a source
image as input and produces a caption as output.
According to [
Analysis of the demonstration dataset provided by Oscar revealed that there are always at least 10 detections per image, with confidence scores higher than 0.2. Based on this finding, we configured the object detector in our pipeline to generate detections with confidence scores higher than 0.2, and to include detections with lower confidence scores if there are fewer than 10 detections in total. This ensures that the input to Oscar matches the format of the demonstration dataset.
In this work, we conducted experiments using the COCO
Captions [
The demonstration dataset provided by Oscar also consists of images from the COCO Captions dataset, which is split into train+val+test sets that originally belonged to the original train+val COCO Captions dataset.
We decided to follow this distribution, resulting in final train+val+test sets of 5,000+5,000+113,287 images.
To assess the efect of training data size on model performance, we selected various amounts of data from the training set to train Oscar. The sizes of the training subsets were 100%, 50%, 25%, 10%, and 1% of the original 3.3. Impact of Diferent Volumes of Data train set. For each subset size, multiple random selecon Model Performance tions were made from the full training set to measure the consistency of the selected data. The number of random In this experiment, we evaluate the performance of the selections for each subset size is shown in Table 1. The Oscar image captioning model on the COCO Captions number of data selections was chosen to provide a sufidataset. As described in Section 3.2, the dataset was split cient number of samples to measure variance while also into training, validation, and test sets, with the valida- considering the computational resources available. tion and test sets remaining unchanged for evaluation The Oscar model was trained using various sizes purposes. of training subsets for a total of 30 epochs, and the BLEU-4 CIDEr 0.316 1.076
CLIP 0.2882
elapsed time was recorded. Training was conducted using In the second experiment, we investigate whether CLIP NVIDIA GeForce GTX 1080 Ti GPUs. The relationship similarity can be used as a fully-fledged metric for evalubetween elapsed time and training subset size is shown ating image captioning tasks. Our analysis of the data, as in Figure 2. As expected, this relationship follows a linear depicted in Figure 4, revealed that CLIP exhibits behavior dependence between data size and computational time. similar to that of other metrics. To further investigate
During training, the model was evaluated on the vali- this relationship, we calculated Pearson’s correlation codation set after every 5th epoch, and the best-performing eficient between all metrics across all subsets of the data. checkpoint was saved. The CIDEr metric was used for The resulting correlations are presented in Figure 5. this evaluation because it has been found to correlate Our findings show that all metrics are highly correwell with human judgment [17] and Oscar uses it as its lated. This indicates the correct, consistent, and expected default output score. After training, the best-performing behavior of all the metrics. In addition, we observed that checkpoint was selected based on its performance on the the BLEU, METEOR, ROUGE, and CIDEr metrics tend to validation set and then evaluated on the test set. The be on average more correlated with each other than with resulting score on the test set is shown in Figure 4. SPICE or CLIP. This trend is likely due to the fact that
In order to assess the consistency of evaluation re- the former group of metrics compares the placement of sults, we measured the variability of the metric scores n-grams in candidate and reference captions, while the for each subset size. The variability is visualized in the latter two metrics do not consider syntactic content but ifgure using boxplots, which allow us to see the variance rather focus on semantics. of diferent metrics across the individual subsets. The The main takeaway is that CLIP is a viable metric for non-overlapping quarters of the boxplots indicate that image captioning evaluation which does not need referthere is a statistically significant diference in the scores ence captions. This outcome is essential since it enables depending on the subset size. This highlights the impor- hypothetical training of a captioning system without reftance of carefully considering the subset size in order to erences in an unsupervised manner. Bleu_1
1.0000 0.9996 0.9990 0.9982 0.9981 0.9990 0.9986 0.9974 0.9983 Bleu_2
COCO Caption for multiple reasons. Because we believe In our work, we conducted several experiments to ana- it has good consistency - it contains scenes of everyday lyze the training of the image captioning method Oscar. life with a limited variety of objects and because it has First, we trained the method on diferent sizes of training 5 annotations per image. Another reason is that it has data. We measured the elapsed time of the training loop good size - it is big enough to make an adequate 1% split and the performance on given metrics. The training dura- from it, but it is also small enough for 36 training runs tion has a linear relationship with the volume of data that of 30 epochs to be computed in reasonable time on our is used. Furthermore, we have measured the behavior GPUs. Lastly, the quality of the detector producing the of individual metrics based on the size of the training detections and feature vectors afects the performance. data. We measured the consistency of the data for indi- Based on our output, one can now decide to reduce the vidual subsets. We experimentally show that the models training data volume if the goal is to achieve a specific trained on smaller subsets have a higher variance of all minimum score of a metric. It can be assumed, that the the evaluation metrics than the models trained on larger behavior will be similar to other models and datasets. sets. We observe that the scores converge to some value. In our second experiment, we evaluated the correlation However, the improvement of the individual metrics is between various state-of-the-art metrics and the CLIP not linearly dependent on the amount of data used for metric, which we believe, can be used as a fully-fledged training. As we add more data for training, the improve- metric for image captioning with its huge advantage - it ment diminishes. This is afected by multiple phenomena: does not need any reference captions. Our results showed The first one is the capacity of the model itself, hence that all the metrics including CLIP are highly correlated. the convergence to a non-perfect value of the metrics. This supports CLIP’s potential use as a fully-fledged met
1–8 1.0000 ric for image captioning. Previous research [17] has also investigated the CLIP metric, focusing on the correlation with human judgment and comparing it to the correlation of other metrics. In comparing those results to ours, we found that the ranking of the correlation of individual metrics to human judgment corresponds to the ranking of the correlation of other metrics with CLIP.
sity of West Bohemia, project No. SGS-2022-017. Computational resources were supplied by the project "eInfrastruktura CZ" (e-INFRA CZ LM2018140) supported by the Ministry of Education, Youth and Sports of the Czech Republic. Also, we would like to thank RNDr. Blanka Šedivá, Ph.D. for giving us the initial idea for this research.