The analysis and classification of targets within imagery captured by aerial and space-based systems provides the US intelligence community and military geospatial intelligence (GEOINT) personnel with important insights into adversary force dispositions and intentions. It has also entered the mainstream thanks to openly available tools like Google Earth. The high resolution of space-based sensors and common use of overhead imagery in everyday life means with the exception of decoys and camouflage, an average person is now reasonably capable of identifying objects in electrooptical (EO) imagery. EO images are, however, limited by cloud coverage and daylight. About half of the time when a satellite in low earth orbit could image a target it will be night, necessitating the use of either an infrared (IR) or a synthetic aperture radar (SAR) sensor. Both IR and SAR images require a trained imagery analyst to reliably identify targets. A repetitive and time consuming task that currently requires human expertise, importantly and creatively, is an ideal problem for deep learning. Automated target recognition (ATR) seeks to reduce the total work load of analysts so that their effort can be spent on the more human-centric tasks like presenting and explaining intelligence to a decision maker. ATR is also intended to reduce the time from collection to exploitation by screening images at machine speeds rather than manually. SAR ATR is complicated by the available data to train and assess machine learning models. Unlike other image classification tasks, there is not a large and freely available amount of training data for researchers. Further, the data that is publicly available only covers a small fraction of the types of targets an effective SAR ATR system would be required to identify.
Synthetic Aperture Radar (SAR) is a radar mounted to a moving platform that uses the platform's motion to approximate the effect of a large antenna. The high resolution that can be achieved by creating a radar with an effective aperture much greater in size than is physically possible allows for radar returns to be processed into images similar to what can be achieved with a camera [19]. SAR imagery provides an important tool for the United States Intelligence Community and military geospatial intelligence (GEOINT) personnel because of its all-weather, day/night collection capability. Additionally, some wavelengths that SAR imaging systems operate in have a degree of foliage and ground penetrating capability allowing for the detection of buried objects or objects under tree cover that would not be observable by other sensors such as EO sensors.
These important advantages of SAR imaging for GEOINT analysts do come with some significant drawbacks inherent to SAR images. Because SAR images are not true optical images, they are susceptible to noise generated by constructive and destructive interference between radar reflections that appear as bright or dark spots called "speckle" in the image [19]. Also various materials and geometries will reflect the radar pulses differently, creating blobs or blurs that can obscure the objects physical dimensions. These issues, as well as problems caused The SLICY consisting of simple geometric shapes such as cylinders, edge reflectors, and corner reflectors which could be used for calibration of sensors or for modeling the propagation of radar reflections. Fig. 1 shows the example photographs and MSTAR images by class. It demonstrates the difficulties an imagery analyst would face when identifying targets in SAR imagery. The vehicles that are easily recognizable in photos become blurs in SAR images. Due to its public availability and ease of access for researchers, the data set has become the standard for SAR image Automated Target Recognition (ATR) classification research.
ATR in SAR imagery using "shallow" classification methods, which are traditional classifiers applied directly to SAR images with the breakthrough feature extraction layers as demonstrated in convolutional neural networks, produced generally good results. An SVM method proposed by [27] achieved 91% accuracy in a five-class test [27], while a Bayesian classifier reported a 95.05% accuracy in a 10-classes test [13].
In recent years, the work on classification of SAR imagery has focused on the use of CNNs. In 2015, Morgan showed that a relatively small CNN could achieve 92.1% accuracy across the 10-class of the MSTAR dataset, roughly in line with the shallow methods previously explored. Morgan's method also showed that a network trained on nine of the MSTAR target classes could be retrained to include a tenth class 10-20 times faster than training a 10-class classifier from scratch. The ability to more easily adapt the model to changes in target sets represents an advantage over shallow classification techniques [11]. This is especially valuable in a military ATR context given the fluid nature of military operations, where changes to the order of battle may necessitate updating a deployed ATR system. Malmgren-Hansen et al., explored transfer learning from a CNN pre-trained on simulated SAR images generated by using ray tracing software and detailed computer aided design models of target systems. They showed that model performance was improved, especially in cases where the amount of training data was reduced [10]. The technique of generating simulated SAR images for training could also be valuable in a military SAR ATR context where an insufficient amount of training data for some systems may exist.
The use of a linear SVM as a replacement for the softmax activation that is typically used for multiclass classifiers in neural networks has been shown to be potentially more effective for some classification tasks [22]. Transfer learning from ImageNet to MSTAR with an SVM classifier was explored by [1] in 2018. Their methodology compared the performance of an SVM classifier trained on mid-level feature data extracted from multiple layers from AlexNet, GoogLeNet, and VGG-16 neural networks without retraining the feature extracting network. Although they reported over 99% accuracy when classifying features extracted from mid-level convolutional layers from AlexNet, performance of the SVM on features from fully-connected layers did not achieve 80% accuracy. The best performance reported on from the VGG-16 architecture was 92.3% from a mid-level convolutional layer, but only 49.2% and 46.3% from features extracted in the last two fullyconnected layers [1].
CNNs require a very large amount of data to train an accurate model and it is not uncommon for data sets with tens or even hundreds of thousands of images to be used when training a model. Transfer learning presents one possible solution when training a CNN on a limited data set by leveraging knowledge from a previously learned source task to aid in learning a new target task [14]. In an image classification problem, transfer learning works by training a CNN on a data set that has a very large number of images and freezing the parameters for a certain number of layers and extracting midlevel feature representations before training further layers and the final classification layer [7].
ImageNet is an open source labeled image database organized in a branching hierarchical method of "synonym sets" or "synsets". For example, the "tank" synset is found in a tree going from vehicle to wheeled vehicle to self-propelled vehicle to armored vehicle to tank. The ImageNet database consists of over 14 million labeled images organized into over 21,000 synsets. Pre-trained ImageNets are often used in transfer learning.
Transfer learning is typically used when source and target tasks are not too dissimilar in order to avoid negative transfer. Negative transfer occurs when the features learned in the transfer learning method actually handicap the model performance [14]. However, transfer learning becomes more useful when a curious phenomenon that many deep neural networks trained on natural images learn similar features across images from different domains.
Evidence shows that low and mid-level features could represent basic ATR features in images such as texture, corners, edges, and color blobs [9], and the low and mid-level neural network feature extraction function resembles the actual biological and human neurons' function. Low and mid-level of features extracted from CNNs are likely common across even dissimilar data sets. A transfer learning approach between different domains is feasible and ATR tasks are evidently successful in cross-domain applications [2], [8]. For example, the application of transfer learning to remote sensing target detection and classification was studied [16], which showed that a CNN classifier trained on a photographic data set could be retrained to perform remote sensing classification of ships at sea with a good performance.
In practice and in cross-domain applications, very few people train an entire CNN from scratch because it is relatively rare to have a data set of sufficient size. For this reason, transfer learning with feature extraction combined with shallow classifiers are suitable choices for SAR images.
The network architecture employed in this paper was a modified VGG-16 architecture [18]. The original VGG-16 architecture is shown in Fig. 2. It consists of two or three linked convolutional/pooling blocks, three fully-connected layers, and a softmax activation in the end to determine the class label. The network employs a 3x3 kernel and a stride of one so that each pixel is the center of a convolutional step. The architecture has been modified to freeze model weights for the first two convolutional/pooling blocks (e.g., the first six layers). The model top has also been replaced with a fullyconnected layer, a dropout layer to mitigate overfitting, and two final fully-connected layers with a softmax activation for classification [16]. This is also referred as a modified VGG-16 architecture or a VGG-16 architecture in this paper. It was initialized with the ImageNet weights and had the first 3. Our method, as in Fig. 4, shows that the dense layer of 1024 features extracted are saved and used as the input to a shallow classifier.
In our experiment, the standard VGG-16 model is implemented in the Keras application program interface (API) with TensorFlow as the backend. The ImageNet weights available in the Keras are ported from the Visual Geometry Group at Oxford University that developed the VGG architecture for ILSVRC-2014 localization and classification tasks [18]. We also use, Orange, which is an open source data science and machine learning toolkit that allows users to easily manipulate data through a graphical user interface. Orange has several built-in machine learning algorithms and simplifies the data management and pre-processing requirements to allow users to experiment with approaches to machine learning and data science [3].
A CNN is trained on the 2200 training images with a 20% validation split. The training and test data were both then run through the retrained neural network. The last fully connected layer before the neural network's output was saved as a 1024dimensional vector for each image as shown in Fig. 3.
The extracted features run through the Orange workflow are pictured in Fig. 5. The precision and recall are used to compare the base CNN performance, kNN, SVM, and random forest
The baseline model, which is the modified VGG-16, is shown in Fig. 3. The modified VGG-16 without transfer learning and trained exclusively on MSTAR, resulted in an average precision and recall of 0.96. The same modified VGG-16 with full transfer learning of the convolutional layers with weights from ImageNet resulted in an average precision and recall of 0.88. Although, the transfer learning approach has the advantage of converging much more quickly than the CNN initialized with random weights, the full transfer learning of all convolutional weights and only retraining the CNN top did not match the performance of the non-transfer learning approach, suggesting some negative transfer occurs in the later convolutional layers. As shown in Fig. 6, the best performed is the modified VGG-16 with partial transfer learning in Fig. 3 and resulted an average precision and recall of 0.98. The multistep classifier using a kNN classifier in Fig. 4 was able to match the best baseline performance with an average precision and call of 0.98, while the SVM and random forest classifiers fell short of the baseline model's performance.
As described before, ATR of SAR images are typically sensitive to the noise in the images. A CNN is known to be vulnerable to the noisy models both from environmental perturbation or adversarials' deliberated manipulations [6], [21]. To study the effect, random Gaussian noise with a noise factor of 0.2 was added to the images from the data set. Fig. 7 shows an example of an original SAR image with one added noise. The feature extraction from CNN and follow on shallow classification process was then repeated without retraining the base model in order to test the robustness of the model. The baseline model (i.e., modified VGG-16 model) was then retrained on the noisy images for 30 epochs and accuracy was compared.
Neither the neural network nor any of the multistep classifiers proved robust enough to handle the addition of random noise to the images. However, after retraining the kNN and SVM multistep classifiers perform better than the modified VGG-16 with partial transfer learning.
Performance on the SLICY class is of interest because it demonstrates the model's ability to discriminate an invalid target from a valid target. All other classes, with the exception of the D7, are former Soviet Union military equipment. The D7 is a Caterpillar bulldozer. Up-armored versions of the D7 and related equipment are often used in combat engineering roles. In a military context this means they are likely to be a valid target. As demonstrated by the high precision and recall in this class across all models, valid targets are very rarely classified as a SLICY (high precision) and the random objects are not being accepted as valid targets (high recall).
The performance of the kNN classifier is also notable since the use of an SVM for classification after feature extraction is previously studied; however, little research has been done on To further explore the relations of feature extraction, transfer learning, and kNN, we ran an additional experiment where we first extracted the transfer weights of the first six layers of the VGG-16 architecture from ImageNet. Since the flatten dimension is 32x32x128=131,072, as shown in Fig. 9, we applied the unsupervised learning k-means algorithm to group the 131,072 dimension into 2048 clusters. The reasoning here is that the first six layers probably embed the best features (texture, corners, edges, and color blobs) that can be used for classification. Finally, We performed kNN and other supervised learning methods in Orange based on the 2048 dimensional train and test data. Fig. 10 shows the test data results from Orange for the VGG6-transfer-kmeans-kNN method with an average precision and recall of 0.93. The six layers of transfer learning together with k-means and kNN provide an inexpensive (without GPU or AWS, for exam-ple) and no supervised learning or no class labels required approach for SAR ATR. Recently, various learning-to-hash algorithms [24] are used to approximate the exact nearest neighbors, which translates a supervised learning problem and kNN into an index/search problem [15], and simplifies the whole recognition for the tactical operation requirements in the area of SAR ATR. If there are no class labels of SAR available, our multistep classifiers with transfer learning and kNN can provide an unsupervised classification with a high accuracy and confidence to match an object which looks like another object seen before. Currently, the analysis community does not have an established standard for the percent of correctly identified targets by an imagery analyst. Instead the analysis relies on the user's experience and confidence in their own work, providing responses such as "possible main battle tank" or "likely BMP-2", and thus a direct comparison to expert-level performance is difficult to establish. Both the baseline model employing transfer learning and the shallow classifiers using a neural network as a feature extractor performed with a high degree of accuracy and would be valuable in an operational context as an aid to GEOINT analysts.