This paper introduces a novel similarity-based positional encoding method aimed at improving the classification of medical images using Vision Transformers (ViTs). Traditional positional encoding methods focus primarily on spatial information, but they may not adequately capture the complex geometric patterns characteristic of medical images. To address this, we propose a method that utilizes convolution operations to extract geometric features, followed by a similarity matrix based on cosine similarity between image patches. This encoding is then incorporated into the ViT model, enabling it to learn more meaningful relationships beyond basic spatial positioning. The efectiveness of this method is shown through experiments on six medical imaging datasets from MedMNIST, where our approach consistently outperforms the conventional learned positional encoding. This is particularly true in datasets with prominent geometric structures like PneumoniaMNIST and BloodMNIST. The results indicate that similarity-based encoding can significantly enhance medical image classification accuracy.
Vision Transformers (ViTs) have revolutionized the field of computer vision, achieving state-of-the-art
performance on various tasks [
In this paper, we propose a novel perspective on learned positional encodings in Vision Transformers. We argue that these encodings are not merely learning "positions" in the traditional sense, but rather capturing high-level relationships and patterns within the visual data. This insight challenges the conventional understanding of positional encodings and opens up new avenues for improving ViT architectures. In particular, by considering complex medical images such as X-rays, histological or dermatological ones, we can notice that particular diseases or findings are usually visible in form of geometric patterns, which can be properly extracted using convolutions.
Building on this observation, we introduce a new encoding method based on similarity measures between image patches, after applying convolution operations. Our approach leverages the inherent structure and relationships within visual data, allowing the model to capture more meaningful representations that go beyond simple spatial positions. We show that this similarity-based encoding leads to improved performance and generalization across various computer vision tasks. Our contributions are threefold: • we show the efectiveness of our approach through extensive experiments on medical benchmark datasets, showing improved performance compared to traditional positional encodings.
This work not only advances our understanding of Vision Transformers but also paves the way for more eficient and efective visual representation learning in deep neural networks applied to medical images interpretation.
Positional encoding is a key component for Transfomer model, introduced by [
The Visual Transformer (ViT) [
This means that the spatial information within each patch is encoded into a linear sequence of values. An extra learnable parameter is preposed to the features vector, called class token; both the image patch projection and the class token have the same dimensionality.
The transformer encoder block in Vision Transformers (ViTs) is a fundamental component responsible for processing input patches and capturing global dependencies within the image. It consists of several layers, each composed of two main sub-modules: the multi-head self-attention mechanism and the position-wise feed-forward neural network. In the multi-head self-attention mechanism, the input patches are transformed into query, key, and value representations. These representations are linearly projected to multiple attention heads, which independently compute attention scores capturing the relationships between diferent patches. The attention scores are then weighted and combined to generate context-aware representations for each patch. Simultaneously, the position-wise feed-forward neural network applies a non-linear transformation to each patch’s representation, independently. This transformation enhances the model’s ability to capture complex patterns and features within the input patches.
After processing through the attention mechanism and the feed-forward network, the output representations of the patches are passed through residual connections and layer normalization, facilitating stable training and improved gradient flow. Finally, the output of the transformer encoder blocks is fed into subsequent layers or used directly for downstream tasks such as image classification. In the context of image classification, our architecture uses the class token which flows in input to a classification head, using Cross entropy loss as optimization function:
1 ∑︁ ∑︁ , log(ˆ,) ℒ = − =1 =1 (1) Where N in the number of samples, C is the number of classes, is the true label of the i-th sample for the j-th class (either 0 or 1) and ˆ is the predicted probability that the i-th sample belongs to the j-th class.
One of the crucial concept in ViTs is the positional encoding. The positional encoding was introduced
in [
In section 2 we discussed the state-of-the-art for positional encoding. While initially designed to encode absolute or relative positions of image patches, recent studies suggest that learned positional encodings in ViTs may be capturing more than just spatial locations and outperform the fixed positional encoding techniques. In the next section we propose a novel similarity-based encoding method that explicitly models relationships between image patches through convolution.
Positional encodings are a class of methods which give spatial or sequential information to the selfattention mechanisms used by Transfomer models. Positional encodings are broadly studied in the literature (see Section 2) for standard Transformer model (i.e. Transformer for natural language processing). While there are few proposals for ViTs’ positional encoding, the most of them adopt learned positional encoding, which seems to outperform the other techniques.
Learned positional encoding allows the model to implicitly capture the relationships between patches in a flexible way, adapting the encoding to the task at hand and discovering representations that go beyond explicit spatial coordinates, such as semantic relationships and interactions that are spatially informed. The learned positional encoding aims to capture how diferent parts of the image contribute to the overall context, learning patterns such as "closeness" in pixel space or object part arrangements. This can be crucial for tasks like classification and segmentation.
In medical image analysis, however, geometrical patterns (such as shapes, boundaries, and specific anatomical structures) often play a more crucial role than the general patch-to-patch correlation captured by Vision Transformers (ViT), which are unable to manage these geometrical patterns. Based on this observation, we propose a new method for positional encoding, based on convolution operations and cosine similarity between extracted features. The overall ViT architecture has not to be changed and one can exploit pre-trained models for specific task and fine-tune just the positional encoding blocks.
Let represent the input image, divided into patches; we embed each patch and we denote by ∈ R× the matrix representing the patch embeddings extracted from the image ( is the number of patches and is the dimensionality of each patch embedding). Let ∈ R× be the feature map computed by convolutions on the image . The goal is to compute a similarity matrix based on the features map, and to use this similarity matrix as a positional encoding.
The cosine similarity between two vectors and , representing the − ℎ and − ℎ patch embeddings, is given by: where s the cosine similarity between feature and patch , · represents the dot product between the embeddings of patches i and j and |||| and || || are the Euclidean norms (magnitudes) of the vectors and . The complete similarity matrix ∈ R× for all features is computed as: =
· |||| · || || = · ||||2 (2) (3) where is the transpose of and ||||2 normalizes each row of the features matrix to compute the cosine similarities row-wise. Next, we apply a linear transformation to the similarity matrix to map it to the same dimensionality as the original patch embeddings. Let’s define this transformation using a learnable weight matrix ∈ R× : = (4) where ∈ R× is the positional encoding matrix based on similarity between patches, ∈ R× is the cosine similarity matrix and ∈ R× is a learnable projection matrix that transforms the similarity information to match the dimensionality of the patch embeddings.
Since ∈ R× represents the patch embeddings, the positional ecoding can be added to to produce the final input to the transformer: ′ = + (5) where ′ ∈ R· is the modified patch embedding, which now includes the similarity-based positional encoding.
The learned convolutional filters extract geometrical information from the input images, which are essentials in medical image analysis. Through the usage of these geometrical features as encoding, we help the Transfomer architecture to learn patterns which can be useful for the specific task.
In order to compare learned positional encoding (which is the state-of-the-art encoding for ViTs) with our proposed encoding based on similarity, we have defined an architecture with fixed hyperparameters (e.g number of transfomer layer, number of heads, embedding dimension, etc.) and we have trained the network on specific medical datasets with both encoding techniques, by comparing the results.
We resort to 6 diferent datasets which are part of MedMNIST[
The images represent diferent conditions, including benign, malignant, and normal tissue. Task: binary or multi-class classification. • TissueMNIST, a dataset that focuses on histological images from human tissue samples. It contains tissue images from various organs, extracted from the HuBMAP dataset. Each image represents a section of tissue that can be classified into diferent cell types. Task: multi-class classification. • BloodMNIST, a dataset of blood smear images for the classification of blood cells. It is derived from the Atlas of Blood dataset and contains images of diferent blood cell types, which is useful for tasks related to hematology. Task: multi-class classification.
(a) BloodMNIST (b) PneumoniaMNIST (c) PathMNIST
In this section we compare the results we have obtained by considering both standard learned positional encoding and similarity positional encoding. Table 1 shows such results in term of accuracy. It is possible to see that the proposed similarity positional encoding brings to better accuracy for every benchmark dataset we have used.
For BloodMNIST, PneumoniaMNIST and PathMNIST the accuracy gap between our positional encoding and learned positional encoding is bigger than for the other datasets. This may be explained by the fact that these datasets contain more geometrical structure than the other, and since our positional encoding is designed to add to the model the ability to recognize geometric shapes, the behaviour is consistent with the hypotheses. Figure 2 shows an example of BloodMNIST’s image (2a), PneumoniaMNIST’s image (2b) and PathMNIST’s image (2c). In contrast, datasets as DermaMNIST, in which there are no such geometrical features, the resulting performance are slightly increased but overall comparable.
In this work we have proposed a new technique for positional encoding in medical image analysis. Starting from the rationale of having geometrical structures in medical images, we have developed a new encoding technique that employs convolutions to extract geometrical information from the input images and compute cosine similarity between extracted features, in such a way to use this information as positional encoding. We have shown that, in terms of accuracy, this new positional encoding outperforms the standard learned positional encoding, especially in images featuring geometric structures.
As future works we want to focus on two diferent tasks: • compare the attention masks generated by the two diferent encodings, in order to evaluate if the explainability of the model increases introducing geometrical features; • compare our positional encoding with other proposed method, listed in section 2
Andrea Santomauro’s PhD research is co-financed by ARLANIS REPLY. We also acknowledge the use of the Chameleon Cloud testbed (https://chameleoncloud.org/) that has allowed us to perform all the experiments described in the present work.