In our participation in the MEDIQA-MAGIC[1] 2024 workshop at CLEF, we employed the mediqa-m3g-dataset[2] for fine-tuning and one-shot sampling. Our primary models for inference were Llama3 and Gemini, where Gemini served as the Vision-Language Pre-training (VLP) model and Llama3 was utilized for downstream tasks. We focused on parameter-eficient fine-tuning of Llama3 using Low-Rank Adaptation (LoRA). Our approach achieved notable results, including a DELTA-BLEU score of 4.461 and a BERTScore of 0.855, the highest in the task competition. This study underscores the eficacy of parameter-eficient fine-tuning techniques in enhancing medical visual question answering (VQA) performance.
Since the release of transformers, numerous large language models (LLMs) have been introduced,
including classic models like BERT and the GPT[
Today’s large language models (LLMs) require high-quality and extensive datasets. Therefore, finetuning the entire model without additional medical datasets becomes an impractical goal. Partial Enabled
Fine-Tuning (PEFT) can achieve more desirable responses even with a small amount of data. PEFT
encompasses various techniques such as "Reparameterized, Additive, Partial and Hybrid Fine-Tuning,"[
In Reparameterized Fine-Tuning, notable methods like LoRA[
In Additive Fine-Tuning involves fixing LLM parameters and adding learnable parameters in front of
the original prompt. Examples include prefix-tuning[
In Hybrid Fine-Tuning methods, such as MAM Adapter[
Vision language models encompass a wide range of domains, involving both text and images. These
models integrate distinct features from both modalities to perform various downstream tasks, such
as automatic subtitle generation for videos and visual question answering (VQA). The vision
component of these models can be traced back to the field of image classification, where labels are used to
categorize each image, often with multiple labels per image. Backbone networks such as ResNet[
Vision Question Answering (VQA) has long been an established field. Historically, models like RCNN
or ResNet were used to extract image features, which were then aligned with text using models like
BERT or transformers to generate answers, as seen in models like ViLBERT[
One mainstream method involves using an image encoder and a text encoder, followed by a
transformer to combine the two features. Notable examples of this approach include models like LLaVA[
In this paper, we describe the techniques we’ve employed in the MEDIQA-MAGIC[
In this section, we will provide a detailed explanation of the steps involved, including data
preprocessing, model pretraining, model inference and post-processing. As illustrated in figure3.1, we adopted
the approach of converting images to text using a VLP model to bridge the gap between images and
text. However, VLP models without specific pretraining may not efectively convert medical images.
Therefore, without using additional training datasets, we employed the more universally efective
Gemini[
We used Gemini[
Using the LoRA method provided by UnSloth, we incorporated LoRA[
Initially, we used a fixed training example for one-shot learning, which meant we selected a single, consistent example from our training data. However, this approach proved inefective due to the varying suitability of examples for diferent tasks. To improve this, we switched to randomly selecting an example from our training data for each one-shot attempt. This enhancement allowed us to better match the diversity of tasks and improve the model’s adaptability.
During the inference phase, we not only utilize the query_title_en of the current question but also incorporate zero, one, two, or four shots randomly selected from the training data. We refrain from using more shots due to the limitations imposed by max_length. Initially, during testing, we included the text generated from image conversion in the one-shot input. However, experimental results showed a decrease in performance on validation due to the max_length constraint. Therefore, in the final evaluation, only the text generated from the current question’s image conversion is included as part of the input.
After generating the text, leading and trailing whitespace is removed from the output. However, due to the limitation of max_new_tokens, it’s not guaranteed that <end_token> will appear in every output. Thus, there’s no assurance of complete answers in the actual response every time.
In Table 1, we observe a total of 310 training samples, out of which 271 include image data. For the validation set, there are 50 samples, with 44 containing images. The test set comprises 93 samples, all of which include images.
Regarding evaluation metrics, we employ two scoring methods: DeltaBLEU[
Precision(X, Y) = 1 ∑︁ max sim(, )
∈X ∈Y Recall(X, Y) = 1 ∑︁ max sim( , ) ∈Y ∈X
Precision(X, Y) · Recall(X, Y) BertF1Score(X, Y) = 2 · Precision(X, Y) + Recall(X, Y) ∆ BLEU = ∑︀ ∑︀∈n-grams(ℎ) max:∈, {, · #(ℎ, , )} ∑︀ ∑︀∈n-grams(ℎ) max {, · #(ℎ)} (1) (2) (3) (4)
Table 2 shows a significant diference in scores before and after fine-tuning. Without fine-tuning, the performance of one-shot learning heavily depends on the quality of the example used. Since we used random sampling from the training data to assist LLM predictions during one-shot tasks, if the sampled
lower eyelid. There appears to be a small, yellow bump or pustule near the inner corner of the eye. The surrounding skin is slightly red and inflamed. The person’s eye is open, but their expression and the context of the image (e.g., setting, posture) are unclear. It’s dificult to determine the person’s age and gender based on the image alone.
The image shows a close-up of a fingertip with a small, irregular-shaped, white-yellowish lump. The lump appears dry, crumbly, and slightly raised. There are no other visible skin conditions, injuries, or discolorations on the finger or surrounding area. The background suggests a well-lit, clean environment, possibly a medical setting. The focus on the isolated lump suggests concern about its nature and origin. Due to the limited information, it is dificult to determine the patient’s age, gender, and overall health condition, as well as the context surrounding the appearance of the lump.
The image depicts a close-up view of a skin lesion on the back of a person’s arm, just below the elbow. The lesion is roughly circular, approximately 1 centimeter in diameter, and raised with a well-defined border. It exhibits a red or pink coloration and has a scaly, whitish surface with some crusting. The surrounding skin appears normal in color and texture. There is no visible hair within the lesion, although hair is present on the surrounding skin. The setting and the individual’s age and gender are unclear, as is their expression or posture, making it dificult to assess their overall condition or level of discomfort. data were of low quality, the scores might not improve even with multiple samples. After applying LoRA fine-tuning, the model showed a trend where more shots generally resulted in higher scores. However, due to the max_token limitation of LLMs, we could not indefinitely add more shots to enhance performance.
Table 4 shows the results of converting randomly selected images into text using both Gemini[
From Table 3 compares the diferent results obtained from testing img2Text data generated by various
img2Text models. we can see that "no image text" performed well in the zero-shot setting. This is
because our model was fine-tuned using LoRA without image text, and increasing the number of shots
without image text did not yield higher scores. Although BLIP[
At the end, to give you a better understanding of our method, we provide an example of the input and output in appendix Table 5.
After analyzing the results, we identified several limitations in our approach. First, when diagnosing, the model often mistake the disease for another similar disease. That way, the model may provide incorrect advice. Although sometimes, the advice is still correct due to the simularity of the diseases. But, more than half of the time, the advice is incorrect and may cause harm to the patient. Therefore, it’s irresponsible to use our model in real-world medical applications.
Our current model has limitations in accurately distinguishing between similar diseases, often leading
to incorrect advice despite sometimes being correct due to disease similarities. To address this, we
propose a new method for future implementation that combines Chain of Thought (CoT)[
In this method, we aim to leverage the patient’s provided symptoms to narrow down the potential diseases. The Chain of Thought approach will help the model logically deduce and refine the possible diagnoses step by step. By integrating CoT, the model will follow a structured reasoning process, improving its ability to distinguish between similar diseases.
Furthermore, we will combine this with RAG to enhance the model’s access to relevant medical information. RAG will allow the model to retrieve and incorporate external medical knowledge dynamically, providing more context and supporting the CoT reasoning process.
By combining these approaches, we expect to improve the accuracy and reliability of our model’s
medical advice. Future work will involve implementing and testing this combined method, utilizing
more advanced models like Llama3 70b[
This study demonstrated an eficient approach to the MEDIQA-MAGIC task using Llama3 and Gemini models. By fine-tuning Llama3 with LoRA and leveraging Gemini for image-to-text conversion, we significantly improved performance, achieving high DELTA-BLEU metrics. Our results highlight the efectiveness of parameter-eficient fine-tuning methods and the importance of detailed image descriptions in medical AI applications.
However, our observations from the BERTScore and manual review indicate that the model often misidentifies diseases, leading to inaccuracies in the provided answers. This poses a significant limitation in medical question answering or diagnostic assistance, as the accuracy of the information is crucial. Providing inaccurate information could mislead healthcare professionals, potentially causing harm. Therefore, despite the model’s fluency in generating sentences, the issue of model hallucination remains unresolved. This necessitates further improvements in the model’s ability to accurately identify diseases before it can be considered viable for commercial use in the medical field.
Our sample code for model fine-tuning and inferencing is available online at the following links: • Training Sample Colab, • Testing Sample Colab.
title
text Some group of people has a genetic predisposition to these lines. It is more common in the dominant arm. Research studies are limited on this condition but the more fat that you carry on your arms, the more likely creases are to form on your skin. If you don’t want this crease, then first lose some body fat to rule out that cause. If the crease still persists, then you’ll know that they’re a permanent fixture on your arms due to your genetics. If they’re not causing you any pain or if they don’t look too abnormal for your liking , then you don’t need to worry about them. The image shows a close-up of the back of a person’s hairy leg. The individual’s age and gender are indeterminate from the image. The skin appears to have a slightly reddish hue and several small, raised bumps. These bumps could be insect bites, folliculitis, or another type of skin irritation. There is no visible evidence of injury, swelling, or medical equipment in the image. The setting and context of the image are unclear, as it only shows a close-up of the person’s leg. More information about the individual’s symptoms and medical history would be needed to make a definitive diagnosis.
excessive sweating, sun exposure, and smoking. Avoid aggravating factors where possible. Potassium permanganate soaks may be useful in the acute phase. Apply topical treatments like topical steroids (usually potent or ultrapotent), pimecrolimus and tacrolimus, and regular use of emollients and moisturizers. I have also added a cream for faster healing to be applied twice daily for 2 weeks. Use white toothpaste.
fluids are must. Antibiotics are given in case of systemic infection. The lesion can be surgically removed. Referral to a dermatologist is recommended for dermoscopic examination and skin biopsy. The 5-year survival rate is >95%.