Image Captioning involves generating a textual description of an image, in the most accurate way possible. It requires a combination of Computer Processing techniques, which can both be enhanced individually to improve the overall performance of the model. Knowledge Distillation is a model compression technique, where a smaller network learns from the predictions of a larger network to find a more optimal convergence space. It effectively improves the performance of the smaller network, without any problems like over fitting. Typically, the performance of image captioning is measured in terms of the BLEU score and CIDER score. In this work, we have tested and recorded the performance of three different Image Captioning model architectures, in terms of a large network, small network and knowledge distilled small network to track and analyse the effects of Knowledge Distillation. The results are promising when compared with the state of art models.
Image Captioning is the task of generating a natural language description of the scenes and objects in an image. The sentences given as output must be grammatically correct and describe the image as accurately as possible. Therefore, the image captioning models must be able to recognize/describe the objects, their context within the scene, and their relationships, and frame them into a proper sentence in the target language. The image captioning tasks involves a combination of computer vision (CV) and natural language processing (NLP) techniques, where the CV parts come into play during object detection and recognition, and embedding into a feature vector. The NLP parts involve the conversion of this feature vector into the sentence, framing the words based on the object’s location, action, and features, as well as their importance in the image. The typical input and output for an Image Captioning task is seen in Figure 1. There have been tremendous advancements in the efficiency and accuracy of CV and NLP techniques; hence this has been seen as a consequential increase in the performance of image captioning models. Models like Inception and other advanced CNNs for image detection, as well as Transformers for NLP tasks, provide access to state-of-the-art models on all devices.
2020 Copyright for this paper by its authors. In this work, we have seen the performance of 3 different models used typically for image captioning. Creating a characteristic language depiction of an image has drawn in interests as of late both in light of its significance in useful applications and on the grounds that it interfaces two significant machine learning fields: CV and NLP. Existing methodologies are either top-down, that start from an image and convert it into words, or bottom up, which concoct words portraying different parts of an image and afterward consolidate them. The general approach is seen in Figure 2 below.
The Transformer and BERT models, and their applications, exhibit that models with attention
mechanisms, wherein the recurrent layers are replaced for the utilization of self-attention, offer far
better performances in sequence modelling. This alternative likewise gives unique architecture
modelling abilities, as the attention layer is utilized in a multi-layer structure differently. Image
Captioning might need to be deployed in a number of use cases, including on edge devices or in areas
with low bandwidth or low powered devices. They might not support large accurate models; hence the
concept of Knowledge Distillation (KD) has been used to improve the performance of low accuracy
models. By distilling the knowledge in an ensemble of models into a single model, Hinton et al [
Image Captioning has seen much advancement in recent years, in part due to the
independence of the modules involved, namely CV and NLP modules. Any improvements in one of
these fields can be shown to have a consequential performance improvement in the overall image
captioning task too. There have also been vast improvements in model evaluations and performance
metrics used, to ensure captions generated that are closer to the ground truth. The earliest works
involving Image Captioning include articles like [
With respect to the decoding components of image captioning, typical approaches include
greedy search and beam search. The advancements in deep learning have led to Recurrent Neural
Network-based NLP models, in order to predict the highest probable sentences for the visual
embedding. The LSTM based approach has been quite prevalent, with varying layers leading to better
performance [
The datasets used for the task are the Flickr8k and MS-COCO dataset. The Flickr8k dataset consists of 8,000 photos, each of which is accompanied by five different captions that provide clear descriptions of the important items and events. An example from the dataset is shown below in Figure 3.
The COCO dataset is a large-scale object detection, segmentation, and captioning dataset. Each of the almost 82000 images includes 5 captions for training Image Captioning models. For this work, only a small subset of around 5000 images has been used. An example is shown in Figure 4.
The best image captioning models involve usage of a SOTA-level object detection/recognition model, for the initial image processing purposes. The final layer of the detection model is passed through an encoder model, which captures it as an embedded feature vector. This is finally passed through the decoder model (typically RNN architecture), which outputs the probabilities for all 5001 words in pre-defined vocabulary.
In this work, the performances of 3 different image captioning techniques, with 3 models of varying architecture sizes for each technique, have been compared to analyze the impact of KD. The 3 models used are: 1. ResNet50 + Beam Search 2. Inception V3 + Encoder-Decoder with Attention 3. EffNetB0 + Transformer Encoder-Decoder
To analyze the effects of Knowledge Distillation on these image captioning models, we have taken a teacher and student architecture for each and trained separately. The distilled model takes untrained student architecture and learns along with the teacher’s predictions in order to more efficiently converge. So, 9 different models have been trained and tested.
ResNet50 is the SOTA-object detection model used in this technique. It is a type of ResNet model having 50 layers, with 48 Convolutional layers, a MaxPool and an AveragePool layer. The model as in Figure 5 and Figure 6 are used for feature extraction and the embeddings are passed to the encoderdecoder network (RNN) which uses a beam search algorithm in order to give the probabilities of the next word for all the words in the pre-defined dictionary. The model initially involves the ResNet50 model, whose final layer output is passed to the Encoder. The Encoder comprises of a fully connected linear/dense layer and a Normalization layer before being passed to the Decoder. The Decoder involves an embedding layer, an LSTM layer and the final fully connected layer of the vocab size for the probabilities. The below table 1 depicts the architecture details.
Because just the latest hidden state of the encoder RNN is used as the context vector for the decoder,
the traditional seq2seq model is generally unable to effectively handle extended input sequences. In
this model, we have used Inception V3 to preprocess all the images on the datasets. The captions are
tokenized and the model is trained. The model consists of Encoder-Decoder architecture as in Figure
7 is similar to [
The architecture details as in table 2 consists of the Inception V3 model for object recognition and image pre-processing. The final layer is directly passed through the dense/fully connected layer which then passes it to the decoder. The decoder consists of the Bahdanu attention layer, GRU layer for holding information for longer periods of time compared to RNN and is more efficient than LSTM, and dense/fully connected layers to predict the probabilities of the words for the image.
The Transformer Neural Network is a unique design that tries to tackle sequence-to-sequence tasks while also being able to handle long-range dependencies. One major distinction in these networks is that the input sequence may be sent in parallel, allowing the GPU to be fully exploited while also increasing training speed. The vanishing gradient issue is also overcome by a substantial margin because it is based on the multi-headed attention layer. Transformers as in Figure 8 have been successfully adapted to many deep learning tasks, easily outperforming other network architectures. 256 512 512 512 512 256 0.3 256 0.5 5001
We have evaluated the results on BLEU and CIDER metrics. Bleu is the standard evaluation metric for measuring the amount of correspondence between the network output and the ground truth. The models were tested on a split of around 1000 images. Cider metric is a consensus-based metric to measure the similarity between generated output and the set of human-translated sentences. In Tables 4 and 5, we see how the performances of state-of-the-art architectures for image captioning and our architectures performances. Despite the smaller size of the knowledge distilled models, it shows comparable performance and is far more efficient than the larger models.
In this work, we trained multiple teachers, students, and distilled models on the Image Captioning task. We used two standard datasets, namely Flickr8k and MS-COCO dataset to train the models. A comparison was made showing the results obtained for all the models, and we could see that the transformer models effectively outperformed their counterparts. We saw that in some cases, the student model outperformed teacher models of other architectures (Model 3 student vs Model 1 Teacher), whereas in other cases, the knowledge distilled model was given a boost was able to match/outperform other teachers (Model 3 distilled student vs Model 2 teacher). We can clearly see the use cases where a smaller model would be able to replace and to an extent, even outperform existing slower larger models. For future works, we can include more models in the study, as well as tuning the distillation parameters. We can also choose to study the effects of further distillation to establish a relationship function between performance and distillation.