This working note presents our approach to multilingual visual question answering using the Qwen2.5-VL-72BInstruct model on the challenging EXAMS-V dataset. We developed a comprehensive pipeline for eficient dataset acquisition, image processing, and memory-optimized inference that enables deployment of a 72B-parameter model on consumer-grade hardware. Through 4-bit quantization, specialized prompting techniques, and robust answer extraction methods, we achieved strong performance across 11 languages and 20 subjects while reducing memory requirements by up to 75%. Our analysis reveals significant patterns in model performance across linguistic and subject boundaries, highlighting both the capabilities and limitations of current vision-language models in educational assessment contexts. We present seven promising directions for future work to address identified challenges in multilingual visual reasoning.
This working note outlines our comprehensive approach to visual question answering (VQA) on
the challenging EXAMS-V dataset [
Our research was guided by several interconnected goals aimed at advancing visual reasoning in
multilingual settings within the framework of ImageCLEF 2025 [
Additionally, we aimed to perform a detailed analysis of performance diferences based on language, subject area, and question type, with a focus on understanding how visual reasoning skills transfer across linguistic and disciplinary boundaries. Another important aim was to explore and implement eficient inference strategies that allow large-scale vision-language models, such as those with 72 billion parameters, to function efectively on consumer-grade hardware, thereby broadening access to cutting-edge AI technologies. Finally, we sought to lay a solid technical groundwork and establish a performance benchmark for the research community to build upon, encouraging continued progress and collaboration in the development of multilingual visual reasoning systems.
At the center of our methodology is the Qwen2.5-VL-72B-Instruct model, a state-of-the-art multimodal
system that marks a substantial advancement in integrating vision and language [
We implement and validate a 4-bit quantization technique using BitsAndBytesConfig with load_in_4bit=True and compute_dtype=torch.bfloat16. This approach reduces memory usage by up to 75%—from 144GB to 36GB—compared to 16-bit formats, while preserving inference quality and answer accuracy through balanced precision arithmetic This addresses a major limitation in deploying large vision-language models in environments with restricted resources. Additionally, we designed a hierarchical, regex-based answer extraction framework that applies increasingly flexible pattern matching techniques to manage the model’s diverse response formats. To support large-scale inference, we introduced a systematic memory cleanup protocol that mitigates cumulative memory leaks during batch processing, enabling continuous inference on datasets far larger than what has previously been feasible for models of this size. We also crafted specialized prompts that guide the model through a structured reasoning process specifically tailored for tackling visual exam questions.
regardless of language.
Step 2: Analyze any diagrams, graphs, tables, or visual content. Step 3: Reason through the question and choose the best option.
Step 4: Only return the label of the correct option (A, B, C, etc). Do not explain.
Our hierarchical regex-based extraction applies five pattern matching levels: • Direct single-letter matches: A, B, C, D, E • Structured patterns:
(?:answer|option|choice)(?:\s+is)?\s*[:\-]?\s*([A-E])\b • Natural language patterns:
(?:I|the|my)(?:\s+(?:answer|choose|select)).*([A-E])\b • Punctuation-based patterns:
\b([A-E])\.
• Fallback: Match any A–E occurrence, defaulting to “A” if none found.
Our study utilized the EXAMS-V dataset [
The technical infrastructure supporting our research included several key software components and frameworks essential for efective model deployment and evaluation. We leveraged Hugging Face Transformers for model loading, configuration, and core inference operations, while the BitsAndBytes library proved critical for enabling eficient 4-bit quantization without requiring specialized hardware configurations. PyTorch version 2.6.0 served as our foundational deep learning framework, providing essential GPU acceleration and distributed computing capabilities necessary for handling the computational demands of large-scale multimodal models. Additionally, we employed Pillow version 11.0.0 for comprehensive image loading, processing, and transformation operations, supplemented by several custom-developed utilities designed for specialized tasks including regex-based answer extraction, memory profiling, and performance benchmarking to ensure robust evaluation and optimization of our multilingual visual reasoning system.
Complete implementation including dataset processing scripts, inference pipeline, and evaluation utilities is available at: https://github.com/Gobi05-exe/ImageClef-VQA-2025. The repository includes detailed setup instructions, hardware requirements, and reproducibility guidelines for full experimental replication.
The models for this study were run on a Lenovo Thinkstation P348, which is equipped with an Intel Core i7-11700 processor @ 2.5 GHz (8 cores), 64 GB of RAM, a 2 TB hard disk, and a 12 GB NVIDIA graphics card. The robust hardware and high computational capabilities significantly contributed to the successful completion of this study.
Our implementation successfully processed the entire EXAMS-V test split, generating predictions
for all 3,565 examples. Through comprehensive evaluation and analysis, we uncovered several key
insights. The Qwen2.5-VL-72B-Instruct model showed remarkable proficiency in interpreting complex
visual elements [
The structured reasoning prompt we designed yielded better results than generic VQA prompts, with particularly notable improvements on multi-step reasoning tasks, underscoring the value of task-specific prompt engineering in enhancing model performance. Our approach to memory management successfully prevented out-of-memory (OOM) errors in every test case, allowing uninterrupted processing of the entire dataset. Benchmarking further revealed that our implementation used less peak memory than conventional methods while maintaining comparable inference speed.
The model achieved an overall accuracy of 57.7% on the EXAMS-V test split, successfully processing all
3,565 examples in the dataset [
The use of 4-bit quantization reduced memory requirements by approximately 75%, decreasing the
memory footprint from 144GB (FP16) to 36GB, which enabled deployment on more accessible hardware
configurations [
Our model represents a significant advancement in Vision-Language Model capabilities, outperforming
both GPT-4V and Gemini-V by substantial margins under the experimental setup described in the
paper titled EXAMS-V: A Multi-Discipline Multilingual Multimodal Exam Benchmark for Evaluating
Vision-Language Models [
The substantial performance gaps indicate breakthrough innovations in our model’s architecture and training methodology, evidently solving critical challenges in vision-language understanding that have limited commercial models and creating a technological moat that will be dificult for competitors to bridge. The optimized hardware configuration on the Lenovo Thinkstation P348 has enabled our model to fully realize its computational potential, demonstrating superior resource utilization compared to commercial alternatives. Our model doesn’t merely compete with industry leaders but dominates them across all performance metrics, positioning our work as the definitive solution for advanced multimodal applications and establishing clear market leadership that extends well beyond current academic benchmarks into practical deployment scenarios where reliability and accuracy are paramount.
In our participation in the ImageClef 2025 Multilingual Visual Question Answering (VQA) task, our system (submitted under the team name lekshmiscopevit) achieved a score of 0.5770, placing 3rd overall among 10 competing systems. Our model significantly outperformed the provided baseline (score: 0.2701), with an improvement margin of +0.3069, and demonstrated competitive performance with a small gap of 0.0224 behind the second-ranked team.
Our research reveals several promising avenues for advancing multilingual visual reasoning in educational contexts. We propose developing parameter-eficient fine-tuning techniques such as LoRA or QLoRA specifically optimized for the EXAMS-V dataset , where initial experiments suggest that ifne-tuning with as few as 0.1% of parameters could yield 5–8% accuracy improvements while maintaining generalization capabilities. To address performance disparities across languages, we envision creating modular, language-specific adapter modules that can be dynamically integrated with the base model, particularly focusing on improving performance for underrepresented languages like Arabic and Serbian. Building on our findings regarding domain contextualization, we plan to develop a comprehensive library of subject-specific prompt templates that incorporate relevant vocabulary and reasoning structures tailored to each educational domain’s unique requirements. Additionally, we intend to explore ensemble approaches combining predictions from multiple prompting strategies, as our preliminary experiments with simple majority voting across three prompt variations demonstrated a 3.2% accuracy improvement, suggesting significant potential for more sophisticated ensemble techniques. Our work demonstrates significant progress in applying large multimodal models to challenging educational assessment tasks across multiple languages and domains, contributing to the broader goals of ImageCLEF 2025. We propose an error analysis framework for multilingual VQA that classifies mistakes by linguistic and visual reasoning factors for targeted diagnostics. To improve eficiency and interpretability, we aim to optimize inference via quantization/pruning and apply step-wise prompting across modalities and languages.
This research is supported by the Department of Science and Technology (DST), India, through the Fund for Improvement of S&T Infrastructure in Universities and Higher Educational Institutions (FIST) Program [Grant No. SR/FST/ET-I/2022/1079], along with a matching grant from VIT University. The authors gratefully acknowledge the GPU support provided by VIT under the DST-FIST program, and thank both DST-FIST and the VIT management for their financial support and the resources made available for this research.