In this paper, we focus on describing the techniques used in the MEDIQA-MAGIC [1] task, including pre-processing, fine-tuning, and inferencing. Also, we used the mediqa-m3g-dataset [2] to train, validate, and test our model. The dataset consists of medical questions and images, with the goal of generating medical advice based on the input. To effectively extract information from images without re-training a multimodal model, we opted to use a Vision-Language Pre-Training(VLP) model to obtain image information first. We then combined the medical question with the text we've generated from the images. That way, we only requires to fine-tune the subsequent LLM to meet the needs of our downstream tasks. This approach reduces the GPU memory needed for fine-tuning and increases efficiency.
Since the release of transformers, numerous large language models (LLMs) have been introduced, including classic models like BERT and the GPT [3,4,5] series. Starting from GPT-3 [5], many models have gained widespread attention for their exceptional performance in few-shot and zero-shot learning scenarios. Among them, the recently released Llama3 [6] by Meta has achieved remarkable results across various datasets compared to other open-source LLMs.
Today's large language models (LLMs) require high-quality and extensive datasets. Therefore, finetuning the entire model without additional medical datasets becomes an impractical goal. Partial Enabled Fine-Tuning (PEFT) can achieve more desirable responses even with a small amount of data. PEFT encompasses various techniques such as "Reparameterized, Additive, Partial and Hybrid Fine-Tuning, " [7] each offering diverse fine-tuning methods.
In Reparameterized Fine-Tuning, notable methods like LoRA [8] introduce additional learnable variables in Linear Layers or Attention layers.
In Additive Fine-Tuning involves fixing LLM parameters and adding learnable parameters in front of the original prompt. Examples include prefix-tuning [9], which trains new prefixes to provide context and guide LLMs to produce appropriate answers, and Parameter-Efficient Prompt Tuning [10], which modifies the input layer context without changing the core model parameters. Adapter-based methods, such as AdapterDrop [11] and Hadamard Adapters [12], introduce adapters to adapt the model to new tasks.
In Hybrid Fine-Tuning methods, such as MAM Adapter [13] and AutoPEFT [14], establish connections between the aforementioned techniques. These methods allow for effective model fine-tuning to adapt to downstream tasks even without access to high-quality and extensive datasets.
Vision language models encompass a wide range of domains, involving both text and images. These models integrate distinct features from both modalities to perform various downstream tasks, such as automatic subtitle generation for videos and visual question answering (VQA). The vision component of these models can be traced back to the field of image classification, where labels are used to categorize each image, often with multiple labels per image. Backbone networks such as ResNet [15], EfficientNet [16], ViT [17], and Swin [18] are employed to extract features for classification purposes. Subsequently, tasks like image captioning have emerged, where backbone networks extract image features, and use Language model to generate simple text descriptions of the image content. Notable examples of this approach include the CLIP [19] and BLIP series [20,21]. In the context of our task presented in this paper-MEDIQA-MAGIC [1]-we focus on the VQA domain. Here, we extract features from images or convert them into textual descriptions, which are then processed by large language models (LLMs) to produce the desired descriptions.
Vision Question Answering (VQA) has long been an established field. Historically, models like RCNN or ResNet were used to extract image features, which were then aligned with text using models like BERT or transformers to generate answers, as seen in models like ViLBERT [22], VL-BERT [23], and Pixel-BERT [24].
One mainstream method involves using an image encoder and a text encoder, followed by a transformer to combine the two features. Notable examples of this approach include models like LLaVA [25], BEiTv3 [26], and InternVL [27].
In this paper, we describe the techniques we've employed in the MEDIQA-MAGIC [1] task, focusing on data processing, model fine-tuning, and inference. To efficiently extract information from images without re-training a multimodal model, we utilized a Vision-Language Pre-trained (VLP) model to obtain image information. This information was then combined with the text extracted from the images. Subsequently, only the downstream Large Language Model (LLM) required fine-tuning. This approach minimizes GPU memory usage during fine-tuning and enhances overall efficiency. In this task we only ues T4 16GB RAM on Colab to fine-tuning and inference.
In this section, we will provide a detailed explanation of the steps involved, including data preprocessing, model pretraining, model inference and post-processing. As illustrated in figure3.1, we adopted the approach of converting images to text using a VLP model to bridge the gap between images and text. However, VLP models without specific pretraining may not effectively convert medical images. Therefore, without using additional training datasets, we employed the more universally effective Gemini [28] for this role. For the LLM, we used the latest Llama3-8b[6] model.
We used Gemini [28] for data preprocessing to convert images into text. Gemini offers better image descriptions compared to VLP models like CLIP [19] and BLIP [20,21].
Using the LoRA method provided by UnSloth, we incorporated LoRA [8] into Llama3-8b [6] with their default parameters: r=16, dropout=0, alpha=16. Subsequently, we employed SFT (Supervised Fine-Tuning) to optimize LoRA in order to match the style of the correct answers. During fine-tuning, we did not utilize the query_content_en since most of the training data lacks substantial content. Therefore, we only used query_title_en as inputs. Additionally, for all instructions, we uniformly used "Give me a medical advice" as the prompt, then input the patient context inside query_title_en.
Initially, we used a fixed training example for one-shot learning, which meant we selected a single, consistent example from our training data. However, this approach proved ineffective due to the varying suitability of examples for different tasks. To improve this, we switched to randomly selecting an example from our training data for each one-shot attempt. This enhancement allowed us to better match the diversity of tasks and improve the model's adaptability.
During the inference phase, we not only utilize the query_title_en of the current question but also incorporate zero, one, two, or four shots randomly selected from the training data. We refrain from using more shots due to the limitations imposed by max_length. Initially, during testing, we included the text generated from image conversion in the one-shot input. However, experimental results showed a decrease in performance on validation due to the max_length constraint. Therefore, in the final After generating the text, leading and trailing whitespace is removed from the output. However, due to the limitation of max_new_tokens, it's not guaranteed that <end_token> will appear in every output. Thus, there's no assurance of complete answers in the actual response every time.
In Table 1, we observe a total of 310 training samples, out of which 271 include image data. For the validation set, there are 50 samples, with 44 containing images. The test set comprises 93 samples, all of which include images.
Regarding evaluation metrics, we employ two scoring methods: DeltaBLEU [29] and BERTScore [30]. BERTScore [30] leverages contextual embeddings from BERT pre-trained models to compute the F1 score based on the maximum cosine similarity between two sentences, with a maximum score of 100. The actual algorithm for BERTScore is shown in Equation 3. DeltaBLEU [29] Score is derived from BERTScore and incorporates weights derived from human qualitative judgments, as well as the maximum number of N-gram matches across multiple reference answers, yielding a maximum score of 1. The actual algorithm for DeltaBLEU is shown in Equation 4Precision(X, Y) = 1 π
Table 2 shows a significant difference in scores before and after fine-tuning. Without fine-tuning, the performance of one-shot learning heavily depends on the quality of the example used. Since we used random sampling from the training data to assist LLM predictions during one-shot tasks, if the sampled The image shows a close-up of a fingertip with a small, irregular-shaped, white-yellowish lump. The lump appears dry, crumbly, and slightly raised. There are no other visible skin conditions, injuries, or discolorations on the finger or surrounding area. The background suggests a well-lit, clean environment, possibly a medical setting. The focus on the isolated lump suggests concern about its nature and origin. Due to the limited information, it is difficult to determine the patient's age, gender, and overall health condition, as well as the context surrounding the appearance of the lump. 11rk8rd
No Image describe The image depicts a close-up view of a skin lesion on the back of a person's arm, just below the elbow. The lesion is roughly circular, approximately 1 centimeter in diameter, and raised with a well-defined border. It exhibits a red or pink coloration and has a scaly, whitish surface with some crusting. The surrounding skin appears normal in color and texture. There is no visible hair within the lesion, although hair is present on the surrounding skin. The setting and the individual's age and gender are unclear, as is their expression or posture, making it difficult to assess their overall condition or level of discomfort.
data were of low quality, the scores might not improve even with multiple samples. After applying LoRA fine-tuning, the model showed a trend where more shots generally resulted in higher scores. However, due to the max_token limitation of LLMs, we could not indefinitely add more shots to enhance performance. Table 4 shows the results of converting randomly selected images into text using both Gemini [28] and BLIP [21]. It is evident that BLIP [21] often fails to successfully generate text descriptions and, when it does, the descriptions tend to be overly brief. In contrast, Gemini [28] consistently uses more detailed text to describe the images. We believe that in the medical domain, providing more detailed descriptions is crucial. Therefore, we conclude that using Gemini is more effective in assisting Llama3 [6] in answering questions.
From Table 3 compares the different results obtained from testing img2Text data generated by various img2Text models. we can see that "no image text" performed well in the zero-shot setting. This is because our model was fine-tuned using LoRA without image text, and increasing the number of shots without image text did not yield higher scores. Although BLIP [21] performed well in one-shot and four-shot settings, it did not perform well in the two-shot setting. We believe this is because BLIP often fails to effectively convert images to text, so random sampling does not guarantee better results with more samples. For the results generated by Gemini [28], we observed a clear trend of improved performance with more shots. We believe this is because Gemini provides more stable and reliable results compared to BLIP [21].
At the end, to give you a better understanding of our method, we provide an example of the input and output in appendix Table 5.
After analyzing the results, we identified several limitations in our approach. First, when diagnosing, the model often mistake the disease for another similar disease. That way, the model may provide incorrect advice. Although sometimes, the advice is still correct due to the simularity of the diseases. But, more than half of the time, the advice is incorrect and may cause harm to the patient. Therefore, it's irresponsible to use our model in real-world medical applications.
Our current model has limitations in accurately distinguishing between similar diseases, often leading to incorrect advice despite sometimes being correct due to disease similarities. To address this, we propose a new method for future implementation that combines Chain of Thought (CoT) [31] reasoning with Retrieval-Augmented Generation (RAG) [32].
In this method, we aim to leverage the patient's provided symptoms to narrow down the potential diseases. The Chain of Thought approach will help the model logically deduce and refine the possible diagnoses step by step. By integrating CoT, the model will follow a structured reasoning process, improving its ability to distinguish between similar diseases.
Furthermore, we will combine this with RAG to enhance the model's access to relevant medical information. RAG will allow the model to retrieve and incorporate external medical knowledge dynamically, providing more context and supporting the CoT reasoning process.
By combining these approaches, we expect to improve the accuracy and reliability of our model's medical advice. Future work will involve implementing and testing this combined method, utilizing more advanced models like Llama3 70b [6] and GPT-4-o [33]. With these enhancements, we believe our model will better support healthcare professionals in making informed decisions.
This study demonstrated an efficient approach to the MEDIQA-MAGIC task using Llama3 and Gemini models. By fine-tuning Llama3 with LoRA and leveraging Gemini for image-to-text conversion, we significantly improved performance, achieving high DELTA-BLEU metrics. Our results highlight the effectiveness of parameter-efficient fine-tuning methods and the importance of detailed image descriptions in medical AI applications.
However, our observations from the BERTScore and manual review indicate that the model often misidentifies diseases, leading to inaccuracies in the provided answers. This poses a significant limitation in medical question answering or diagnostic assistance, as the accuracy of the information is crucial. Providing inaccurate information could mislead healthcare professionals, potentially causing harm. Therefore, despite the model's fluency in generating sentences, the issue of model hallucination remains unresolved. This necessitates further improvements in the model's ability to accurately identify diseases before it can be considered viable for commercial use in the medical field. and gender are indeterminate from the image. The skin appears to have a slightly reddish hue and several small, raised bumps. These bumps could be insect bites, folliculitis, or another type of skin irritation. There is no visible evidence of injury, swelling, or medical equipment in the image. The setting and context of the image are unclear, as it only shows a close-up of the person's leg. More information about the individual's symptoms and medical history would be needed to make a definitive diagnosis.
It is a case of eczema called dyshidrotic eczema. It is aggravated by atopic dermatitis, excessive sweating, sun exposure, and smoking. Avoid aggravating factors where possible. Potassium permanganate soaks may be useful in the acute phase. Apply topical treatments like topical steroids (usually potent or ultrapotent), pimecrolimus and tacrolimus, and regular use of emollients and moisturizers. I have also added a cream for faster healing to be applied twice daily for 2 weeks. Use white toothpaste. Avoid direct contact with food items. Take multivitamins once daily. Rest and I/V fluids are must. Antibiotics are given in case of systemic infection. The lesion can be surgically removed. Referral to a dermatologist is recommended for dermoscopic examination and skin biopsy. The 5-year survival rate is >95%.