Using a Novel Capsule Network for an Innovative Approach to Image Captioning

Abstract

Zebrafish is a popular model system for biomedical analysis, especially for compound screening in drug research. In this paper, we present a comprehensive investigation aimed at enhancing the processing pipeline for segmenting zebrafish larvae images. The emphasis is on the application of an unsupervised segmentation method for segmenting zebrafish in Optical Projection Tomography (OPT) images. We propose a novel pipeline that integrates the Transformer and U-Net, a convolutional neural network for bio-medical image segmentation, to achieve accurate segmentation of zebrafish larvae images. This accuracy is critical for precise 3D reconstruction. Leveraging transfer learning, we broaden the capabilities of our trained model to segment OPT images. This approach is intended to enhance the robustness and versatility of our pipeline, allowing it to cater to a broad range of imaging modalities beyond traditional microscopic images. The developed processing pipeline is then used for 3D reconstruction of the segmented areas, demonstrating its potential for advanced biomedical analysis. Our findings confirm the eficiency and accuracy of the proposed pipeline providing robust tools for future Zebrafish-based research, particularly in the domains of drug screening and cancer treatment.

-

CEUR ceur-ws.org

1. Introduction

Automatic image captioning is a challenging problem in computer vision, and it aims to generate rich content and human-understandable descriptions for given images [ 1 ]. The performance of image captioning models is closely related to the quality of extracted features from images. The power of the language model can help to generate accurate and meaningful descriptions related to image content. Considering the semantic relationships between the identified objects within the image is essential in the image caption generation task. However, identifying the objects (i.e., the nouns in the caption) within an image is still challenging. Moreover, finding their interaction (i.e., the verbs in the caption) is extremely dificult. In fact, ex-pressing object interaction by natural language as semantic knowledge, either as verbs or adverbial compositions, is the core issue in image captioning.

In this paper, we develop a novel method that (1) overcomes the limitations of CNNs, (2) generates descriptions with a non-restricted variety of words, and (3) is capable of describing the relationships between the objects. We CEUR Workshop Proce dings htp:/ceur-ws.org ISN1613-073

Attribution 4.0 International (CC BY 4.0).

CEUR

Workshop Proceedings (CEUR-WS.org) use a novel encoder–decoder mechanism that addresses these challenges by using a capsule network (CapsNet) [ 2 ]. The result is a set of meaningful descriptions for the image via a language model. CapsNet can efectively compensate for the shortcomings of a CNN by detecting tissue overlap characteristics [ 3 ]. In CapsNet, more salient spatial features and geometrical attributes, such as direction, size, scale, and object attributions, can be represented for each input. This aspect of CapsNet contrasts with CNN since the lack of local invariance features produces excessive variations of global discriminating outputs [ 4 ]. In addition, our model employs an external knowledge base, i.e., Wikipedia, aiming to accomplish augmented textual training data to generate more meaningful and diverse captions.

The main contributions of our work are as follows: • The development of a novel parallel structure for a capsule network can capture more comprehensive information about the objects within an image by considering their relationships. • The use of Wikipedia as an external knowledge base for enrichment of all the textual training information and generating out-of-domain representation when describing the content of the image. • The application of our framework on the MS

COCO large-scale dataset.

Using large-scale

datasets including RGB images requires a huge number of resources because of the architecture of capsule networks.

existing state-of-the-art models. • We performed a bench-marking towards a list of

In the next section, we will consider state-of-the-art approaches.

2. Related Work

considering visual and semantic attention to the image.

For example, Yang and Liu [ 12 ], introduced a method called ATT-BM-SOM to increase the readability of the syntax and optimize the syntactic structure of captions.

This framework operates based on the attention balance mechanism and the syntax optimization module and effectively fuses image information. Their model generates high-quality captions, compensating for the lack of image information selection and syntax readability. In the next section, the structure of the image caption generation models and the employed networks in our experiments will be discussed in more detail.

Image captioning is a popular research topic in computer vision and natural language processing. Generating an accurate textual explanation that describes the content of an image is accomplished by understanding the visual content of the image. Recently, the interest in image captioning has broadened with the development of benchmark datasets such as MS-COCO [ 5 ], Flickr 8K [ 6 ], and Flickr 30K [ 7 ].

Current image captioning models can be categorized 3. Materials and Image Captioning into template-based, retrieval-based, and neural networkbased models. The template-based models [ 8 ], first detect Methods all the image attributes using image classification and ob- Following the trend of current work, we use an enject detection methods. These methods generate captions coder–decoder framework to create the captions of imby filling in predefined templates from the identified ob- ages. Understanding the image requires recognizing the jects. This approach produces too flexible captions that objects, properties, and interactions in the encoder part. cannot correctly describe the relationships between at- Moreover, producing sentences to describe images in tributes [ 9 ]. Retrieval-based models [ 10 ] create a pool of the decoder requires understanding language syntax and similar images in an image database rank the retrieved semantics. images by measuring their similarities, and then change Figure 1 illustrates the employed Knowledge Discovery the found image descriptions to create a new description Database (KDD) of our model: images and descriptions for the queried image. The usefulness of this strategy is proceed separately in the data processing phase. Then severely constrained when dealing with images that are in the transformation phase, all the image and text data not in the dataset and thus not classified, i.e., unseen. are processed to create feature vectors for the language

The neural network-based models are inspired by the model. A CNN is employed for predicting the labels from success of deep neural networks in machine learning the given image. tasks and used in an encoder–decoder architecture [ 11 ]. In the text enrichment phase, we used Wikipedia to An encoder extracts image contents by a CNN, a module extract relevant information based on the predicted laassociates contents to words, and a decoder by an RNN bels of images. Then, all the data sequences are fed to the is used for language modeling and creating image cap- language model in the NLP phase for tokenizing, embedtions. Liu et al. [ 12 ] proposed an ontology to describe the ding, and making word vectors from the image captions scene construction of images. Their constructed ontology in the dataset and extracted knowledge from Wikipedia. can specify the object types and the special information After which, all the information is fed into the caption for the objects (e.g., location, velocity). This visual and predictor in the evaluation section to produce a caption special information can be transformed into meaningful given the input image. project information for generating captions using inte- The novelty of our work consists of a new variant grated computer vision and linguistic models. In [ 13 ], of the capsule network, parallelizing its basic structure authors demonstrated that a large amount of data could to capture more comprehensive information about the lead to lower estimation variance and hence lower er- objects within the image, thus leading to a more accurate ror with better prediction performance. However, data description of the input image. The primary structure quality plays an important role in the performance of the of capsule network works well on a simple dataset such model. The hypothesis is that more data may contain as MNIST, which includes images with a single object useful information. To this aim, Hossain et al. [ 14 ], pro- and only one channel. However, the network eficiency posed a method that leverages a combination of real and significantly decreases when applied to images with large synthetic data generated by the Generative Adversarial special dimensions and complex datasets such as MSNetwork (GAN). It is an eficient alternative for the tech- COCO and Flicker. niques requiring human-annotated images, as they are The presence of multiple channels and objects in the labor-intensive to generate and time-consuming. images increases the training time of the network and

Various improvements are made to captioning models leads to weak results compared to state-of-the-art [ 15 ]. to make the network more inventive and efective by This problem happens due to ineficiency in capturing the underlying information of the image. To handle this issue, we extended the baseline network by parallelizing the convolutional layers and the primary capsules of the original CapsNet, followed by a concatenation approach to extract more complex and qualified features from the images. On the other hand, parallelizing the convolution layers reduces the dimensions of the fed features to the primary capsules and accelerates the learning process.

In the proposed image captioning model, we use CNN and CapsNet architectures to incorporate visual context from an image, which is then used as the input of a machine translation, such as an RNN architecture, to generate objective sentences in the decoder part of the framework. We applied cross-entropy loss to adjust the model weights during the sequential model training. In this section, the entire model flow is described in more detail. We have applied both the Inception-V3 or VGG16 as image feature extractors. These networks are trained on the ImageNet dataset with more than one million images of 1000 classes. Training the CapsNet is done from scratch and based on 80 categories of objects in Category Caps. The details of these networks are shown in Table1. length of the vector denotes the chance of that entity existing. Unlike a convolutional network, capsules save comprehensive information about the location and pose of an entity.

Sabour et al. [ 2 ], claimed that regardless of the high capability of CNNs, this network has two main disadvantages: 1) lack of rotation invariant and 2- using a pooling layer. The former causes failure in recognizing spatial relations between the objects, and the latter causes information loss due to the maximum value selection of each region. Therefore Sabour et al. [ 2 ], proposed a capsule network to address the issues mentioned above.

There are diferent concrete components in a capsule net-work for learning the semantic representations within the image (see Figure 2) These components map construction by reconstructing the discrepancy map from the input image.

The major components of the capsule network involve the following: • Primary capsules combine the features extracted

by convolutional layers in the construction phase. • Reshaping the extracted feature maps from the

primary capsules. • Squashing is a non-linear activation function that squashes the weighted input vector of a particular capsule. This function distributes the length of the output vector be-tween 0 and 1. • The dynamic routing layer produces output capsules with high agreements by automatically grouping input capsules. The pooling layers in the capsule network are re-placed by a mechanism called “routing by agreement” in the rooting layer: the output of each capsule in the lower level is sent to the parent capsules in the higher level only if their features have a dependency. • Category capsules with a marginal classification loss and a reconstruction sub-network with a reconstruction loss for recovering the original image from capsule representations.

The operation of all these components is explained

in this section in more detail. One important aspect of 3.1. Capsule Network capsule networks is their ability to identify individual parts of objects in a single image and then represent A capsule is a set of neurons whose activity vectors in- spatial relation-ships between those parts. For example, dicate the posture characteristics of an entity and the in figure 2, the CapsNet has identified three diferent parts of objects with-in the input image (tie, child, bin). The output image on the right side of the figure

2 shows the

result of the reconstruction sub-network in the employed capsule network. Figure 3 shows the construction of a capsule and how data is routed between lower-level and higher-level capsules.

In Figure 3a, each capsule finds the appropriate parent in the next layer during the dynamic routing procedure

The input and output of capsules are vectors. Given as the prediction vector of capsule and | as the output of parent capsule in higher level will be computed by multiplying with a weighted matrix

: ̂| = ⋅ ing a component in the image even after changing the

The length of , indicates the probability of predict- viewing angle. The direction of represents several to capsule by the following function: A weighted sum over all | , and an intermediate coupling coeficient , is calculated as the total input vector

Here, the coupling coeficient , are the class-specific likelihood calculated after flattening the vectors and is computed by a routing Soft-Max function as follows: to send its output to those capsules in the above layer. Figure 3: Transferring information among the capsules from properties of that component, such as size and position. that could not predict the correct label, if the predicted [1...I] and high-level capsules (b) routing procedure.

where is loss term for one prediction, is a term equal to 1 when the class is present; otherwise, it is 0.

The upper and lower bounds of margin loss parameters, + and −, are set to 0.9 and 0.1 [ 2 ]. It means that if an entity is present with a probability above 0.9, the loss is zero; otherwise, the loss is not zero. Regarding capsules probability of all those labels is below 0.1, the margin loss is zero; otherwise, it is not zero. The parameter is set at 0.5 and is used for numerical stability to control the down weighting of the initial weights for the absent (2) classes. ‖ ⋅ ‖ in all the equations denotes 2 norm. (1) (3) (4) over the input vector in this network: = ‖ ‖2 1 + ‖ ‖

2 ‖ ‖ where is the input vector and is the normalized output between 0 and 1. The log probability is updated along with the routing mechanism by calculating the agreement between as the output of capsule in the above layer and | , as a prediction vector.

The loss function of the network for each capsule is computed as follows: = max(0, + −‖ ‖)2 +(1− ) max(0, ‖ ‖− −)2 (5) = ∑ |

=

exp( ) ∑ exp( ) where represents the log probability of connection between capsules and . As shown in Figure 3b, the value of increases when the lower-level and higher- is forward-ed to the decoder. The networks prevent overlevel capsules are consistent with their predictions and iftting by re-building the input image from the output decreases when they are inconsistent. Based on the origcapsules by minimizing the reconstruction loss as a reginal paper, this parameter is initialized at 0 in the routing ularization method in the decoder [ 16 ]. by agreement procedure. Instead of applying the ReLU

The original capsule network has been tested on the

activation function as in VGG16 and Inception-v3, the folMNIST dataset with one color channel (grayscale). Howlowing non-linear squashing function will be calculated ever, the color of objects is an important factor in object

3.2. Improved capsule network

In the improved version of the capsule network architecture, where we parallelized the convolution layers and primary capsules, the input image size is 229 × 229 × 3.

The diferent architecture of the capsule network distinguishes it compared to CNN. Except for the input and out-put layers, the capsule network consists of primary and category capsule layers. The output of the capsules detection and image captioning tasks. Therefore, we propose a parallelized capsule network that generates the descriptions of the images by passing the RGB images with three color channels through the three blocks of parallel convolutional layers and parallel primary capsules.

The three-color channels of RGB images can store information and intuitively visualize content. Therefore, color analysis is also addressed in this parallelized structure of the capsule net-work, which makes the model more informative and improves the descriptiveness of image captions by extracting more qualified features from the image [ 17 ]. Adding more convolutional layers was not logical due to the increasing model complexity computational cost. The structure of the new network has been 6. Training language model for certain epochs based presented in figure 4. on its performance on validation data. During The model steps in Figure 4 are summarized as follows: the training phase, the model predicts the next 1. Partitioning the image set into train, validation, word of each word in the caption (Figure 4q,r) and test subsets randomly After the training phase, the model is ready to evaluate 2. Applying image feature extractor models to ex- test set images by extracting visual features and predicttract visual features from the images (Figure 4a–j) ing the captions using a greedy search. Greedy search 3. Extracting external knowledge for each image by selects the word with the highest probability at each time searching the predicted labels from the previous step and uses it as the GRU input for the following time step as a query in Wikipedia and adding it to the step until the end of the sentence is reached. In the next captions that already exist for the images in the section, we will dis-cuss the details of the experiments 4. Adaptpalsyeitn(gFipgruerper4okc–emss)ing methods to contextual and the obtained results by the analyzed methods. data before feeding it to the RNN network, i.e., removing the punctuation numbers and wrap- 3.3. Gated Recurrent Unit ping each sentence around with “ssss” and “eeee” Our image captioning framework used a three-layer RNN tokens to specify the beginning and end of sen- network with a Gated Recurrent Unit cell (Chung et al., tences for the network (Figure 4n) 2014). This RNN is equipped with visual features in the 5. Transforming the textual features to the integers feature maps of CNN and CapsNet. The proposed model vector by tokenizing and embedding operations generates a description for each image by maximizing the for training by the language model (Figure 4p) probability of the current word predicted in the caption and machines. This metric measures the maximum numaccording to the following formula: where are the parameters of the proposed model = 1 to = . It means that BLEU-1 is the accuracy of ∗ = arg max ∑ log ( | ; )

( ,) where is the time step and is context vector. A two- reference sentences. This metric finds the longest substep process feeds all the text data to the RNN network. sequence of tokens between candidate and reference senand

is the correct description of image . Suppose { 0, … , −1 } is a sequence of words in transcription of length , then log ( | )

as the probability of generating a word for an image , is as follows: ∑ log ( =0 log ( | ) = | , 0, … , −1 , )

(7) The first step is tokenizing, and the second one is embedding. All the words in the sentences are converted into so-called integer token vectors during tokenizing.

This process is based on 10,000 most frequent and unique words in the image captions.

4. Experiments

This section reports the details of implementations and the results of the experiments conducted by diferent variations of models.

4.1. Dataset and Implementation Details

We use the MS-COCO dataset [ 5 ], to evaluate the proposed model in our experiments. MS-COCO contains 123,287 k images with five captions and 80 object categories for each image annotated by Amazon Mechanical Turk (AMT) workers. Since there are no available annotations for the test set, in this work, we used publicly available splits provided by Karpathy et al. [ 18 ]. We use 5000 images for validation and testing and the rest for the training set. All the models are implemented in Python version 3.6 and using the capabilities provided by Keras version 2.2.5 and TensorFlow version 1.15.0 deep learning libraries. Table 1 shows the parameters set for each network. The training was done using a machine equipped with two GeForce RTX 2080 GPU cards with 8 GB memory. The machine was installed with two GPUs, but for the experiments, only one was necessary.

4.2. Metrics

To compare our results to other baseline models, we measure the performance of the implemented models by the commonly used metrics, BLEU 1–4 [ 19 ], ROUGE [ 20 ], and METEOR [ 21 ].

BLEU is one of the popular metrics to evaluate the

correspondence between generated sentences by humans (6) ber of co-occurrence n-grams between reference and candidate sentences. Here, ‘n’ takes the value of 1, 2, 3, and 4 depending on the length of sentences. Each BLEU-N metric averages the calculated accuracy from the description created for the image with the reference description based on 1-gram, BLEU-2 is the geometric mean of the calculated accuracy based on 1-gram and 2-gram, BLEU-3 is the geometric mean of the calculated accuracy based on 1-gram, 2-gram, and 3-gram, and so on.

ROUGE evaluates the performance of generated sentences by a machine based on their similarity to the tences and calculates how many tokens from the human reference summaries were duplicated in the machinegenerated summaries. Unlike BLEU, which prioritizes precision, ROUGE is recall-oriented and can estimate correlated n-grams better than BLEU.

METEOR is the last evaluation metric in this paper. In this metric and the exact word match, the stemmed and wordnet synonym tokens are taken into account between the alignment of the candidate and the reference sentence.

4.3. Baselines

We provide two baseline approaches to verify the efectiveness of the models. The framework for the baseline is almost the same as the model in [ 11 ] as a baseline method, except that GRU replaces the LSTM language model. We used inception-V3 and VGG16 as the feature extractor method for the encoder part.

4.4. Our approaches

We assess diferent variations of our approach. CN + IncV3 utilizes the extracted features from the capsule network and inception-V3 as image features extractors. CN + VGG16 uses a VGG16 network rather than inception-V3 in the en-coder. The Wikipedia knowledge base enriches the contextualized language model in this model. So, CN + IncV3 + EK and CN + VGG16 + EK are the models that use relevant external knowledge from Wikipedia. We also have performed additional experiments to check the importance of the capsule network in describing the content of images. To that end, we implemented IncV3 + EK and VGG16 + Ek methods to verify the efectiveness of the capsule net-work for image captioning models.

5. Results and Discussions

This section discusses the results from the diferent implementations of our framework and then compares them to state-of-the-art. Table 2 reports image captioning results for diferent implementations of our method on the MS-COCO dataset. The results demonstrate that the CN + IncV3 + EK model with capsule network and inceptionV3 feature extractors can generate more human-like sentences by adding external knowledge to the language model. This model archives significantly better results in the overall metrics.

Models B1 B2 B3 B4 R M al. [24], implemented a cooperative learning mechanism to combine two image caption and image retrieval modVGG 16 (Baseline) 0.33 0.24 0.18 0.16 0.21 0.24 ules while generating a caption. Then, during a multiIncV3 (Baseline) 0.36 0.26 0.21 0.17 0.23 0.28 step refining process, they refined the image-level and CN + IncV3 0.77 0.54 0.43 0.35 0.47 0.35 object-level information to produce a meaningful caption. CCNN ++ IVnGcVG31+6 EK 00..4819 00..3704 00..2651 00..1594 00..2686 00..3445 Instead of using GRU as RNN, Yu et al. [25], proposed CN + VGG 16 + EK 0.59 0.44 0.37 0.29 0.31 0.38 a model which employed a multimodal transformer as IncV3 + EK 0.63 0.43 0.34 0.28 0.29 0.31 a language model in the decoder to generate a caption. VGG 16 + EK 0.38 0.27 0.22 0.18 0.23 0.26 Contrary to our approach, Lu et al. [26], Anderson et al.[27], have focused on important image regions. Lu et al. [26]), proposed an adaptive attention framework that

To prove the efectiveness of this model, we compare could decide whether to rely on special attention to the the result of the CN + IncV3 + EK method with state-of- image and when to attend to the textual image informathe-art research. In Table 3, the bold numbers show that tion. In [27], Anderson et al. extracted a set of salient Table 3 shows that our best model outperforms previously regions from the image by applying a bottom-up mechapublished results on the MS-COCO “Karpathy” test split nism. They also implemented a top-down mechanism to dataset. determine the distribution of attention over the image to compute feature weightings in diferent regions. Jiang et Table 3 al. [28], proposed a framework that includes a recurrent Comparison of the best result to state-of-the-art. fusion network. This fusion procedure is implemented Models B1 B2 B3 B4 R M between the encoder and decoder to exploit interactions among the represented features from the encoder part for creating a new set of vectors from decoder outputs.

Ours (Aneja et al., 2018 [22]) (Tan et al., 2019 [23]) (Wu et al., 2017 [ 11 ]) (Zhang et al., 2021 [24]) (J. Yu et al., 2019 [25]) (Lu et al., 2017 [26]) (Ande. et al., 2018 [27]) (Jiang et al., 2018 [28]) (Yang et al., 2020 [ 12 ])

Compared to our model, Aneja et al. [22], has proposed

an attention mechanism to leverage spatial features of an image to find salient objects. Tan et al. [ 23], proposed a tuning model with a small number of parameters in the RNN. Their model can produce a very sparse decoder for generating a caption preserving the performance of the method compared to their baseline. Zhang et al. Zhang et

5.1. Qualitative Results

In this section, we present some examples to show the performance of the CN + IncV3 + EK method as our best model. We used the occlusion sensitivity function to visualize and localize the most important regions of the images for the network. The occlusion function computes sensitivity maps for CNNs. Figure 5 shows some examples from our results.

As demonstrated in Figure 5, using occlusion sensitivity helps us better understand features used by the network and provide insight into the reasons for the misclassified images. These examples show that CN + IncV3 + EK is the best descriptor model as it can generate more human-like sentences for each image.

6. Conclusions

In this paper, we developed an encoder–decoder framework employing a novel parallelized capsule network as a feature extractor and the Wikipedia database as an external knowledge provider to establish if this approach can out-perform state-of-the-art solutions. We implemented diferent architectures to produce contextual knowledge from images to achieve this. Our novel approach demonstrated that using a parallel capsule network as an encoder model provided a versatile image feature extractor. Moreover, we have demonstrated that the use of external knowledge further improved the results. Our best model was trained with the capsule network and inception-V3 as a feature extractor, with caption enrichment by an external contextual description. acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, 2005, pp. 65–72. [22] J. Aneja, A. Deshpande, A. G. Schwing, Convolutional image captioning, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 5561–5570. [23] J. H. Tan, C. S. Chan, J. H. Chuah, Image captioning with sparse recurrent neural network, arXiv preprint arXiv:1908.10797 (2019). [24] W. Zhang, S. Tang, J. Su, J. Xiao, Y. Zhuang, Tell and guess: cooperative learning for natural image caption generation with hierarchical refined attention, Multimedia Tools and Applications 80 (2021) 16267–16282. [25] J. Yu, J. Li, Z. Yu, Q. Huang, Multimodal transformer with multi-view visual representation for image captioning, IEEE transactions on circuits and systems for video technology 30 (2019) 4467–4480. [26] J. Lu, C. Xiong, D. Parikh, R. Socher, Knowing when to look: Adaptive attention via a visual sentinel for image captioning, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 375–383. [27] P. Anderson, X. He, C. Buehler, D. Teney, M. Johnson, S. Gould, L. Zhang, Bottom-up and top-down attention for image captioning and visual question answering, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 6077–6086. [28] W. Jiang, L. Ma, Y.-G. Jiang, W. Liu, T. Zhang, Recurrent fusion network for image captioning, in: Proceedings of the European conference on computer vision (ECCV), 2018, pp. 499–515.

References

[1] Y. Wei , L. Wang , H. Cao , M. Shao , C. Wu , Multiattention generative adversarial network for image captioning , Neurocomputing 387 ( 2020 ) 91 - 99 .

[2] S. Sabour , N. Frosst , G. E. Hinton , Dynamic routing between capsules , Advances in neural information processing systems 30 ( 2017 ).

[3] X. Ai , J. Zhuang , Y. Wang , P. Wan , Y. Fu , Rescaps: an improved capsule network and its application in ultrasonic image classification of thyroid papillary carcinoma , Complex & Intelligent Systems ( 2021 ) 1 - 9 .

[4] G. E. Hinton , S. Sabour , N. Frosst , Matrix capsules with em routing , in: International conference on learning representations , 2018 .

[5] T.-Y. Lin , M. Maire , S. Belongie , J. Hays , P. Perona , D. Ramanan , P. Dollár , C. L. Zitnick , Microsoft coco: Common objects in context , in: Computer VisionECCV 2014 : 13th European Conference, Zurich, Switzerland, September 6- 12 , 2014 , Proceedings, Part V 13, Springer, 2014 , pp. 740 - 755 .

[6] M. Hodosh , P. Young , J. Hockenmaier , Framing image description as a ranking task: Data, models and evaluation metrics , Journal of Artificial Intelligence Research 47 ( 2013 ) 853 - 899 .

[7] P. Young , A. Lai , M. Hodosh , J. Hockenmaier , From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions, Transactions of the Association for Computational Linguistics 2 ( 2014 ) 67 - 78 .

[8] A. Farhadi , M. Hejrati , M. A. Sadeghi , P. Young , C. Rashtchian , J. Hockenmaier , D. Forsyth , Every picture tells a story: Generating sentences from images , in: Computer Vision-ECCV 2010: 11th European Conference on Computer Vision , Heraklion, Crete, Greece, September 5- 11 , 2010 , Proceedings, Part IV 11 , Springer, 2010 , pp. 15 - 29 .

[9] S. Javanmardi , A. M. Latif , M. T. Sadeghi , M. Jahanbanifard , M. Bonsangue , F. J. Verbeek , Caps captioning: a modern image captioning approach based on improved capsule network , Sensors 22 ( 2022 ) 8376 .

[10] P. Kuznetsova , V. Ordonez , A. Berg , T. Berg , Y. Choi , Generalizing image captions for image-text parallel corpus , in: Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2 : Short Papers) , 2013 , pp. 790 - 796 .

[11] Q. Wu , C. Shen , P. Wang , A. Dick , A. Van Den Hengel , Image captioning and visual question answering based on attributes and external knowledge , IEEE transactions on pattern analysis and machine intelligence 40 ( 2017 ) 1367 - 1381 .

[12] Z. Yang , Q. Liu , Att-bm-som: A framework of efectively choosing image information and optimizing syntax for image captioning , IEEE Access 8 ( 2020 ) 50565 - 50573 .

[13] D. Martens , F. Provost , Pseudo-social network targeting from consumer transaction data ( 2011 ).

[14] M. Z. Hossain , F. Sohel , M. F. Shiratuddin , H. Laga , M. Bennamoun , Text to image synthesis for improved image captioning , IEEE Access 9 ( 2021 ) 64918 - 64928 .

[15] M. K. Patrick , A. F. Adekoya , A. A. Mighty , B. Y. Edward , Capsule networks-a survey , Journal of King Saud University-computer and information sciences 34 ( 2022 ) 1295 - 1310 .

[16] B. Mandal , S. Ghosh , R. Sarkhel , N. Das , M. Nasipuri , Using dynamic routing to extract intermediate features for developing scalable capsule networks , in: 2019 Second International Conference on Advanced Computational and Communication Paradigms (ICACCP) , IEEE, 2019 , pp. 1 - 6 .

[17] S. Albawi , T. A. Mohammed , S. Al-Zawi , Understanding of a convolutional neural network , in: 2017 international conference on engineering and technology (ICET) , Ieee, 2017 , pp. 1 - 6 .

[18] A. Karpathy , L. Fei-Fei , Deep visual-semantic alignments for generating image descriptions , in: Proceedings of the IEEE conference on computer vision and pattern recognition , 2015 , pp. 3128 - 3137 .

[19] K. Papineni , S. Roukos , T. Ward , W. Zhu , A method for automatic evaluation of machine translation” , the Proceedings of ACL-2002 , ACL, Philadelphia, PA, July 2002 ( 2001 ).

[20] C.-Y. Lin , Rouge: A package for automatic evaluation of summaries , in: Text summarization branches out, 2004 , pp. 74 - 81 .

[21] S. Banerjee , A. Lavie , Meteor: An automatic metric for mt evaluation with improved correlation with human judgments , in: Proceedings of the