Automated radiology report generation from chest X-ray (CXR) images has the potential to significantly reduce the workload of radiologists and improve diagnostic efficiency. In this paper, we propose a novel architecture that integrates a Transformer encoder with an LSTM layer to generate coherent, contextually accurate reports from CXR images. CheXNet, a DenseNet-based model pre-trained on the Chest X-ray dataset, is employed to extract 1024-dimensional feature vectors from the input images. These features are passed to the Transformer encoder, which uses multi-head attention and positional encodings to capture both local and global relationships in the data. An LSTM layer is introduced after the encoder to refine the image features and better capture sequential dependencies in the report. The Transformer decoder generates the report in an autoregressive manner, utilizing beam search during inference to improve fluency and accuracy. Experimental results show that our model achieves competitive performance across BLEU-1 to BLEU-4 scores, with a BLEU-1 score of 0.4636 and a BLEU -4 score of 0.3575, outperforming several baseline methods. The results indicate that our hybrid approach effectively balances word-level accuracy and sequence coherence, making it a robust solution for medical report generation.
Automated medical report generation has gained increasing attention in recent years, driven by the
growing volume of medical imaging data and the limited availability of radiologists. Chest X-ray
(CXR) images, one of the most commonly used diagnostic tools in healthcare, provide critical
insights into thoracic conditions, including pneumonia, lung cancer, and cardiovascular diseases [1]
. Despite their diagnostic significance, interpreting these images and drafting comprehensive
radiology reports remain time-consuming and require specialized expertise. Automating the
generation of medical reports from CXR images could significantly reduce the workload for
radiologists, accelerate diagnosis, and improve the consistency of reporting [
Several studies have explored automated report generation using various deep learning techniques.
For instance, CheXNet, a DenseNet-based model pre-trained on the Chest X-ray dataset, has shown
promising results in detecting thoracic diseases from CXR images [
In this study, we propose a novel architecture that integrates a Transformer-based model with an
LSTM layer to address these challenges. Transformers, known for their ability to capture long-range
dependencies through attention mechanisms [
1. Hybrid Transformer-LSTM Architecture: We propose a novel model that combines a Transformer encoder with an LSTM layer to address the challenges of generating coherent, sequential radiology reports from CXR images. The model utilizes both self-attention and cross-attention mechanisms, self-attention in the encoder to capture global dependencies within the image features, and cross-attention in the decoder to align the generated text with the encoded image features. This architecture leverages the strengths of both attention mechanisms and sequential modeling to improve the quality of report generation. 2. CheXNet Feature Extraction: We integrate CheXNet, a DenseNet-based model pre-trained on the Chest X-ray dataset, to extract rich 1024-dimensional feature vectors from CXR images. CheXNet's ability to detect thoracic diseases enhances the visual representation passed to the Transformer encoder. 3. Beam Search Decoding: During the inference phase, we implement beam search with varying beam widths (2, 5, and 7) to improve the fluency and accuracy of the generated reports. Beam search allows the model to explore multiple word sequences, enhancing the quality of the final output. 4. Comprehensive Evaluation: We evaluate the model’s performance using BLEU scores across multiple n-gram levels (BLEU-1, BLEU-2, BLEU-3, and BLEU-4), demonstrating its effectiveness in capturing both word-level accuracy and contextual coherence. Additionally, we compare the performance of our approach with several existing methods in the field, highlighting its advantages and areas for further improvement.
By addressing both the spatial and sequential aspects of report generation, this work presents a robust framework for automating radiology report generation from CXR images. Figure 1 illustrates the architecture of the proposed system and the flow of data throughout the model's processing stages. The paper at hand comprises the following: Section II presents the related work. Section III presents the material and methods of the work. Experiments are illustrated in Section IV. Section V presents the experimented results. Section VI presents the discussion. Finally, section VII concludes the paper.
A significant amount of research work has been conducted on automatic report generation from
radiological images. Li et al. [
Convolutional neural networks (CNNs) are a class of artificial neural networks that are widely used
for feature extraction and classification tasks, particularly in image and time-series data [
A transformer based encoder-decoder architecture is used to generate natural language reports from
CXR images. The encoder-decoder transformer is an advanced model designed for
sequence-tosequence tasks, particularly useful for transforming image data into coherent textual reports. Below,
the key components of the model are described, focusing on how the encoder and decoder work
together to accomplish this task.
3.2.1. Encoder
The encoder's role is to encode the visual information from CXR images and convert it into a set of
encoded features. Unlike traditional transformer models that rely solely on self-attention [
Mathematically, the transformation process in the encoder can be described by the self-attention mechanism, which focuses purely on the visual features.
ℎ = ( ) here 3.2.2. Decoder
represents the visual features extracted by the DenseNet from the X-ray image, and ℎ denotes the encoded hidden state for each segment of the input feature map.
The decoder generates a natural language report based on the encoded features from the encoder. It
produces a sequence of words, with each word depending on both the encoded image features and
the previously generated words in the report. To improve the decoding process, a relational memory
module is integrated into the transformer architecture [
The report generation can be formalized using the following chain rule, where the probability of
generating the next word depends on both the prior words and the encoded image features:
( | 1, 2, … , −1,
; )
(1)
(2)
where is the target text sequence (the report),
refers to the input X-ray image, and
represents the model parameters. During training, the goal is to maximize the log-likelihood of
generating the correct sequence , given the input image. At inference time, the model uses beam
search to iteratively sample words from the probability distribution, generating the report one word
at a time until it reaches a predefined maximum length or an end-of-sentence token [
Beam Search is a widely used decoding algorithm for generating sequences in tasks such as machine
translation, image captioning, and text generation [
and generates a probability distribution over the vocabulary for the next word. Rather word in the sequence: than selecting only the word with the highest probability, beam search selects the top words with the highest probabilities. These words form the beginning of different candidate sequences. The likelihood of a sequence = { , , … … . . , } is given by the product of the probabilities of each ( ) =
( | 1 , 2, … . . , −1) At each subsequent time step, the algorithm expands each of the current sequences by appending the next word, again considering the top words based on their conditional probabilities. The result is × times
× candidate sequences, which are then pruned by selecting only the top sequences with the highest cumulative scores. This process is repeated until one of the termination conditions is met, such as reaching a predefined maximum sequence length or encountering the end-of-sequence token
. The beam width plays a crucial role in controlling the trade-off between search depth and computational efficiency. A larger beam width allows the algorithm to explore a wider range of possible sequences, potentially leading to more accurate and coherent outputs. However, increasing
also results in higher computational costs, as the number of
sequences that need to be evaluated and ranked grows exponentially [
ℎ
This normalization ensures that longer, more informative sequences are not penalized simply due
to their length. Once the decoding process is complete, the sequence with the best (lowest) score is
selected as the final output. Beam search is commonly applied to generate radiology reports from
CXR images. Each image was processed to extract features, which were then fed into the captioning
model to generate descriptive reports. Different beam widths can be explored to evaluate trade-offs
between sequence diversity and computational complexity. The generated sequences are often
evaluated using BLEU scores to measure the similarity between the predicted reports and the actual
medical reports [
The Long Short-Term Memory (LSTM) layer is a type of recurrent neural network (RNN) designed to capture patterns over long sequences of data, particularly useful in tasks where maintaining a logical sequence is essential. Unlike standard RNNs, LSTMs can retain important information over extended periods, making them ideal for applications where the order of information is crucial, such as text generation or time-series forecasting. By capturing sequential dependencies, LSTMs help ensure that each generated word or phrase aligns with prior context, producing coherent and contextually accurate outputs.
Multi-Input Feature Extraction Architecture
In this study, we utilized the public OpenI dataset, which contains CXR images collected from the
Indiana University hospital network [
The dataset comprises CXR images in PNG format and corresponding radiology reports in XML
format. To extract the relevant information, we first parsed the XML files using the ElementTree
library [
Additionally, we removed numerical values and sequences of irrelevant characters, such as repeated "x" marks, as they do not provide meaningful information for text analysis. Words with fewer than two characters were filtered out, except for medically relevant terms, ensuring that only informative words remained in the processed text. Finally, for patients with multiple X-ray images, we generated image pairs to capture the relationship between different views of the same anatomical region, such as frontal and lateral chest views. When multiple images were available for a single patient, we carefully paired them to ensure that both images corresponded to the same time point or diagnostic session, maintaining consistency in the dataset. In cases where only one image was available for a patient, we duplicated the image to create a pair, treating both images as the same view. This was done to standardize the input format for the model, which requires two images per patient for proper training. The rationale behind creating image pairs lies in the need for the model to learn the spatial and anatomical correlations between different views of the same region. In medical imaging, different angles can reveal additional information or confirm findings from another view, so combining these perspectives helps the model develop a more comprehensive understanding of the patient’s condition. By exposing the model to multiple views of the same anatomical region during training, we enhance its ability to make more accurate predictions, particularly when subtle pathological findings might appear differently depending on the orientation. For each image pair, we ensured that the associated clinical findings from the radiology report were properly aligned with the visual data. This step was crucial in maintaining the integrity of the dataset, as accurate alignment between image pairs and their corresponding text annotations is vital for the success of multi-modal deep learning models. These reports provided context for the visual data, allowing the model to map specific radiological findings to certain visual patterns in the X-ray images, improving its interpretative capabilities.
This preprocessing pipeline of creating image pairs and aligning them with their corresponding textual reports facilitated the construction of a highly structured and coherent dataset. Figure 2 illustrates our proposed flowchart, outlining the preprocessing pipeline. This dataset was instrumental in the subsequent training, validation, and testing phases of the study, enabling the model to learn not only from the visual features of individual images but also from the relationships between different views and the corresponding medical context.
The resulting model was able to provide more accurate and nuanced diagnostic predictions, contributing to the overall goal of generating reliable radiology reports from CXRs. Finally, for patients with multiple images, we created image pairs by combining available images or using the same image for both positions when only one image was present. This step was essential for our model to understand the relationship between different views of the same anatomical region. Each image pair was associated with the corresponding findings from the report, ensuring proper alignment of textual and visual data. This preprocessing pipeline enabled the creation of a structured dataset that could be reliably used for the training, validation, and testing phases of the study.
In our research, the feature extraction process is fundamentally supported by the CheXNet model,
which is built on the DenseNet-121 architecture [
Before feeding the images into the model, several preprocessing steps were applied. The images
were resized to 224x224 pixels to match the input size expected by DenseNet-121, ensuring
compatibility and consistency across the dataset. Additionally, grayscale CXR images were
converted to RGB format by replicating the single grayscale channel across three channels, aligning
with the model's requirement for RGB input [
In this research, the evaluation of the model's performance was conducted using BLEU (Bilingual Evaluation Understudy) scores. BLEU is a widely recognized metric for evaluating the quality of text generated by machine learning models, particularly in tasks such as machine translation and text generation. BLEU measures the correspondence between the machine-generated output and a reference output, with higher scores indicating better alignment between the two. We used BLEU1, BLEU-2, BLEU-3, and BLEU-4 to ca pture n-gram overlaps between the predicted and actual sequences, thus providing a robust evaluation of the generated reports at different levels of granularity. BLEU-1 measures the unigram (single word) precision between the predicted and reference sequences. It evaluates the extent to which individual words from the predicted sequence appear in the reference sequence, without considering the order of the words. BLEU-2 extends this by considering bigram precision, which takes into account pairs of consecutive words, providing insight into how well the model captures short phrases or sequences of two words. BLEU-3 further evaluates the model’s ability to generate contextually coherent sequences by considering trigrams (three consecutive words), while BLEU-4 assesses the precision of four-gram sequences, offering a more comprehensive evaluation of longer and more complex phrase structures. In this study, weighted BLEU scores were used to balance the contribution of different n-gram levels. For example, BLEU-2 applies equal weight to unigrams and bigrams, ensuring that both word-level and phraselevel precision are accounted for. Similarly, BLEU-3 and BLEU-4 distribute the weights across unigrams, bigrams, trigrams, and four-grams, allowing for a more nuanced evaluation of the model's ability to generate coherent and contextually appropriate sequences. The evaluation process was conducted separately for both the cross-validation (CV) and test datasets. For each image-report pair, both the reference report and the model-generated report were preprocessed by removing punctuation and tokenizing the sequences. The BLEU scores were then calculated by comparing the n-grams between the predicted and reference sequences, allowing for an objective assessment of the model’s performance. To ensure consistency, the average BLEU scores across all test samples were computed, providing a detailed view of the model’s capability to generate accurate and coherent reports. To summarize the model’s performance, an average BLEU score was calculated by taking the arithmetic mean of BLEU-1, BLEU-2, BLEU-3, and BLEU-4. This average score serves as a composite indicator of the model’s performance across all n-gram levels, balancing both word-level precision and longer contextual coherence. This approach provides a holistic measure of the model’s ability to generate high-quality reports, combining both accuracy at the word level and fluency across longer sequences.
Finally, to account for the varying lengths of the generated sequences, the BLEU scores were normalized by dividing the cumulative score by the sequence length. This normalization ensured that longer, more informative sequences were not penalized, preventing bias toward shorter sequences and providing a fair and balanced evaluation across all reports, regardless of their length. This normalization technique contributes to a more accurate representation of the model’s true performance in generating coherent and contextually relevant outputs.
In this study, we employed an encoder-decoder Transformer architecture, combined with an LSTM layer, to automatically generate radiology reports from CXR images. Our proposed model is shown in Figure 3, the first stage of the model, the encoder, takes as input the image features extracted by CheXNet, a DenseNet-based CNN pre-trained weights on the CXR dataset. CheXNet transforms the input X-ray images into 1024-dimensional feature vectors. These vectors contain rich information regarding the visual characteristics of the images, such as anomalies, disease patterns, and other medically significant features. The CheXNet model was chosen due to its proven accuracy in identifying thoracic diseases, making it an ideal feature extractor for our report generation task. The extracted features are then passed into the Transformer encoder. The Transformer encoder consists of multiple layers of multi-head self-attention mechanisms and feed-forward networks. The selfattention mechanism allows the model to focus on different regions of the image feature set simultaneously, helping it capture global dependencies within the X-ray. Multi-head attention improves the model's ability to learn complex relationships by focusing on different parts of the input in parallel. To preserve the positional information of the image features, which the Transformer lacks due to its architecture, positional encodings are added. These encodings ensure that the spatial structure of the image is maintained throughout the encoding process. In order to better capture sequential dependencies from the image features, we introduce an LSTM layer after the Transformer encoder. The LSTM is a type of recurrent neural network (RNN) known for its ability to handle long-term dependencies and sequential data effectively. In our model, the LSTM refines the encoded image features by considering temporal patterns across the feature sequence, which is critical for generating coherent and contextually relevant reports. The LSTM’s role is particularly important as medical reports are typically sequential in nature, with certain observations depending on others.
After the LSTM has processed the image features, they are passed into the Transformer decoder, which is responsible for generating the radiology report. The decoder generates the report word by word in an autoregressive fashion, where each word is conditioned on both the previously generated words and the encoded image features. The decoder utilizes a masked multi-head self-attention mechanism to ensure that the generation process is sequential, preventing future words from influencing the current prediction. Additionally, the decoder uses a cross-attention mechanism to focus on the encoded image features while generating each word, ensuring that the generated text remains aligned with the content of the X-ray image. The model is trained using teacher forcing, where the ground truth words are provided during training to help the model learn the correct sequence of words. We optimize the model using the Adam optimizer with a learning rate of 0.001. The loss function used is categorical cross-entropy, which compares the predicted word probabilities with the ground truth words to guide the learning process. During inference, the model generates the report using beam search. Beam search allows the model to explore multiple possible word sequences at each generation step, selecting the sequence with the highest overall probability. This method improves the fluency and accuracy of the generated reports. Beam search was implemented with a beam width of 2, 5, and 7 in different experiments to assess its impact on the quality of the generated reports and to balance the trade-off between computational efficiency and generation quality.
In summary, our approach integrates the Transformer’s powerful attention mechanisms with the sequential modelling capabilities of LSTM. The combination of these techniques enables the model to effectively process the complex visual data from CXRs and generate accurate, coherent radiology reports. This architecture ensures that the generated reports reflect both global and local features of the images, resulting in a system that is both robust and highly accurate for medical report generation. All experiments were conducted on an NVIDIA GPU model Tesla V100 with memory size 16GB to accelerate model training and reduce computation time.
In the results section, we present the performance of the model based on the BLEU evaluation metrics across multiple n-gram levels. The BLEU scores provide a detailed analysis of the model’s ability to generate coherent and contextually relevant reports by comparing the predicted outputs with the reference sequences. For BLEU-1, which measures unigram precision and focuses on individual word matches, the model achieved a score of 0.4636. This indicates a relatively high level of word-level accuracy in the generated reports, reflecting the model's ability to correctly predict relevant terms that appear in the reference reports. The BLEU-2 score, which accounts for bigram precision and captures short phrases, was slightly lower at 0.4504. This suggests that while the model can accurately predict individual words, there is a slight reduction in performance when considering word pairs, indicating some challenges in maintaining short-term contextual coherence. As the evaluation extended to longer n-grams, the scores declined further. The BLEU-3 score, which assesses trigram precision, was 0.3754, indicating a more pronounced difficulty in generating contextually accurate sequences of three words. Similarly, the BLEU-4 score, which measures fourgram precision and reflects the model’s ability to capture longer and more complex phrase structures, was 0.3575. The decreasing trend in BLEU scores from unigram to four-gram precision highlights the increasing complexity the model faces in generating longer, contextually accurate sequences. To summarize the overall performance, the average BLEU score, calculated as the mean of BLEU-1, BLEU-2, BLEU-3, and BLEU-4, was 0.4117. This composite score reflects the model's general performance across different n-gram levels, balancing both word-level precision and longer sequence coherence. While the model demonstrates strong performance in word-level accuracy, as indicated by the BLEU-1 score, its ability to generate longer, more coherent phrases and sentences is more challenging, as shown by the progressively lower BLEU scores for longer n-grams. These results suggest that while the model is effective at generating relevant words and short phrases, there is room for improvement in generating longer, contextually coherent sequences, which are critical for producing high-quality, fluent reports.
The results of our study demonstrate the effectiveness of combining Transformer and LSTM architectures for generating radiology reports from CXR images, leveraging pretrained CheXNet for feature extraction. As indicated in Table 1, the model achieved a BLEU-1 score of 0.4636, highlighting its ability to accurately predict individual words relevant to medical reports. This performance suggests that the model is particularly adept at capturing important terms, which is critical for conveying key findings in radiology reports. However, the decline in performance across BLEU-2 to BLEU-4 metrics reflects the challenges the model faces in maintaining coherence in longer phrases and sentences. Specifically, the BLEU-4 score of 0.3575 points to difficulties in accurately generating complex, multi-word sequences. This limitation is expected, given the sequential nature of radiology reports, where specific findings and observations need to be described in detail and with context. The integration of the LSTM layer was designed to address such issues by capturing temporal dependencies, yet the model still struggles to consistently generate longer coherent sequences, indicating a potential area for improvement. The results also show that the inclusion of beam search during inference, with different beam widths, plays a significant role in balancing computational efficiency with the fluency and accuracy of the generated reports. By exploring multiple word sequences at each generation step, the model improves its output quality. Nevertheless, the progressively lower scores in higher n-grams suggest that there is room to further optimize this aspect of the model, possibly by exploring alternative decoding strategies or enhancing the sequential modeling of medical terminology.
A notable comparison can be made with the results of Elaanba et al. [
This study presents a novel approach for generating radiology reports from chest X-ray (CXR) images by integrating a Transformer encoder with an LSTM layer. Leveraging CheXNet for feature extraction, our model effectively captures both global and local image features, while the LSTM enhances sequential modeling, which is crucial for producing coherent medical reports. The use of beam search during inference further improves the quality and fluency of the generated reports. Experimental results demonstrate that our model achieves competitive BLEU scores, with BLEU-1 at 0.4636 and BLEU-2 at 0.4504, indicating strong performance in capturing relevant medical terminology and generating coherent short phrases. Compared to previous methods, our model achieves higher BLEU-1 and BLEU-2 scores than the 0.438 reported by Li et al. and the 0.455 reported by Jing et al. Although the BLEU-3 (0.3754) and BLEU-4 (0.3575) scores declined, reflecting the challenge of generating longer, contextually rich sequences, they remain higher than those reported in related studies, such as Chen et al.'s model with a BLEU-4 score of 0.165. These results highlight our model’s effectiveness at generating short, coherent sentences but also suggest areas for improvement in handling more complex sequences. While our approach shows significant promise, certain limitations should be noted. First, the model's reliance on a single-view CXR dataset may constrain its performance for complex cases that would benefit from multi-view imaging. Additionally, the model's computational complexity results in a relatively long runtime, which could impact real-time application feasibility. These limitations suggest opportunities for further enhancement. For future work, enhancing the LSTM layer with bidirectional LSTMs or exploring more efficient decoding strategies, such as top-k or nucleus sampling, could improve coherence in longer text sequences. Moreover, incorporating multi-view image analysis and domain-specific knowledge, such as clinical embeddings or expert annotations, may further refine report accuracy and contextual relevance. Overall, this study underscores that combining attention mechanisms with sequential modelling is a promising direction for advancing automated radiology report generation.
During the preparation of this work, we used ChatGPT to assist with paraphrasing and improving sentence clarity, and Grammarly to assist with grammar and spelling checks. All AI-generated suggestions were critically reviewed and edited by the authors to ensure accuracy, originality, and alignment with the publication’s standards. The authors take full responsibility for the content and conclusions of this work.
Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., & Summers, R. M. (2017). ChestX-ray8: Hospital-scale chest X-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 2097–2106). IEEE. doi:10.1109/CVPR.2017.369.