In the present paper, we present a preliminary and ongoing work concerning the problem of generating a suitable diagnostic report from radiology (x-ray) images. The task is tackled with a contrastive learning approach, where radiology images and textual reports are embedded in a common space, with the goal of getting similar (close in space) embeddings between a given image and the corresponding report. Once the embeddings for an image and the corresponding textual report are generated (through suitable ifne-tuned models), we propose to feed them to a contrastive learning engine in such a way that image and textual embeddings are pushed close in the embedding space if image and text are related, while they are moved away otherwise. Preliminary analysis shows promising results in terms of efectiveness of the contrastive learning approach, but also suggest relevant issues to be investigated such as the importance of context and the role of suitable encoder/decoder modules to properly deal with the textual generation.
Healthcare data are intrinsically multimodal [
These kinds of data are provided through diferent modalities: structured information such
as EHR, time-series data as in ECGs, images in case of x-rays, CATs or PETs, and free text in
case of diagnostic reports. Consequently, the interest in jointly exploiting diferent modalities
for machine learning tasks in healthcare applications have been growing in the last years
[
The attention-based approach is usually adopted in image-text matching problems related to
multimodal retrieval [
The general idea is to adopt a contrastive loss in such a way that, given a pair <image, text>, the corresponding embeddings are pushed close in the embedding space if image and text are related, while they are pushed away otherwise. Next sections will outlined the approach and report some preliminary results.
Contrastive Learning (CL) is a machine learning methodology where unlabeled sample data
are compared against each other, in order to teach a model which points are similar and which
are diferent. One of the most prominent application of this paradigm is the siamese network
model [
In a CL setting, a way to obtain the desired goal is to train the model with the so called
contrastive loss. Suppose that is a parametric function (with set of parameters ) to be
learned. Let 1, 2 be two input points, (1, 2) a distance function (to be learned)
between input points, and ∈ {0, 1} a binary label such that = 0 if 1, 2 are deemed
similar and = 1 otherwise. Following LeCunn’s original formulation [
1 1 (, (, 1, 2)) = (1 − ) ( )2 + max(0, − )2
2 2 For similar points ( = 0), the loss is given by their distance; on the contrary when two points are dissimilar ( = 1) they contribute to the loss function only if their distance is within the margin . The main rational behind formula 1 is that we want to bring closer similar items, and move away, in the embedding space, dissimilar items. The margin hyperparameter avoids pushing away items that are already far enough.
ℒ( ) = ∑︁ (, (, 1, 2)) =1 (1)
In the present work, contrastive loss is used in a multimodal fashion. As mentioned above, we consider the problem of generating suitable (diagnostic) reports associated with a radiology image; in this preliminary work we just consider simple x-ray images. The general idea is the following: • produce an image embedding using a suitable CNN with the x-ray image as input; • produce a text embedding using a suitable deep encoder with a free text diagnostic report as input; • learn a combined embedding model trained by means of contrastive loss, using as similar pairs x-ray images and their corresponding reports, and as dissimilar pairs x-ray images and unrelated reports Next section will outline the proposed architecture.
The learning part of the architecture we are proposing (for the training phase) is summarized
in figure 1. Following the general scheme introduced in the previous section, to produce x-ray
embeddings we adopted a particular CNN with the goal of extracting the most important
hidden features of an x-ray image; in particular we considered a pretrained version of CheXNet,
a 121-layer CNN trained on ChestX-ray14, one of the largest publicly available chest X-ray
dataset [
Once all the embeddings are obtained, we standardize them to avoid big diferences in value scale, and we generate a specific training set for the contrastive learning task (see next section). During this step, CheXNet’s parameters are fine-tuned, while we freeze the EncoderDecoder’s parameters; the aim is to map images’ embedding in the textual embedding space, so we can exploit the decoder part to generate images related reports. Figure 3 shows the report generation part of the architecture. For generating the report of a given x-ray image, we submit the latter to the fine-tuned version of the CheXNet+FCNN to generate the suitable embedding that we assume to be close to the text corresponding to the diagnostic report. The embedding is then fed to the decoder and the corresponding report is finally generated.
To perform an experimental analysis of the approach, we considered a public dataset available through the Open-i service of the National Library of Medicine, National Institutes of Health, (Bethesda, MD, USA), and containing coupled elements of x-ray images and reports [19]. Images in the dataset are grayscale DICOM images; a preprocessing step has been performed in order to assure that every image is of the same size. We also perfomed image normalization. Associated diagnostic reports are stored as XML files which contain diferent information; in this work we are interested in the findings part of the report, a free text information representing the diagnostic findings obtained by the radiologist after the inspection of the image. More than one image can be associated with a given report; for the aim of this work we consider them as independent images with the same diagnostic report. Figure 4 shows the histograms correspoding to the number of reports with a given number of associated images. We can notice that the majority of the reports have two associated images, that are front X-ray and lateral X-ray. Some reports have no images associated, and we discarded them. We also discard images having an empty findings part in the corresponding report. We finally performed further sanity checks and text normalization, finally resulting in 3851 items represented as pairs <x-ray_image, ifnding_text>. We performed the embedding generation of every image and text in each item using the architecture of figure 1, and in particular the part upstream of the contrastive engine. We get 3851 pairs <x-ray_embedding, finding_embedding >; we split this dataset of embedding pairs into 80% for training and 20% for test. The training set has been used as input of the contrastive engine of figure 1 for the fine tuning of the model. Early stopping has been used to stop training, with a final validation loss of about 2− 07.
The experimental analysis on the test set has been used to check the following issues: • Embeddings quality • Generated text quality Concerning embedding quality we measured the the distance between the generated report embedding, and the original ones. In particular, given an x-ray image, we use the architecture of figure 3 to generate the corresponding predicted report; we then compared the predicted report with the original one by measuring the L2 distance between the embeddings. Table 1 reports the average normalized distance obtained on the test set, as well as some confidence intervals on such value The average quality of the embeddings is then quite promising, even if AVG Distance 0.21045
Std_dev 0.13991 not completely satisfactory. We think that more efort should be devoted to the exploitation of multiple images associated to a given report, without considering them as independent.
Concerning the quality of generated reports, we found some problems with particular words that can be generated out of context, because of embedding values close to the ones of more suitable (in terms of context) words. This issue can be addressed in the encoder training which is based on BERT: BERT is trained on a corpus of general purpose text, and not in a specific ifeld (e.g., radiology), thus our guess is that more specific contextual information should be provided. We plan to investigate more on this issue and by trying diferent approaches such as: • Inserting prior knowledge • Using another Encoder/Decoder architecture, specific for the Medical field (e.g., MedBert [20]).
We have presented a proposal for a multimodal learning architecture focusing on a contrastive
learning approach to the automated generation of radiology reports from x-ray images. The
work is very preliminary and the results obtained so far have evidenced some issues to focus on
for future works, among which the exploitation of multiple images for the same report, the use of
contextual knowledge in the textual encoder/decoder and the room for alternative architectural
modules. In particular, we think that a closer comparison with the CLIP architecture [