Unveiling Differences: A Vision Encoder-Decoder Model for Difference Medical Visual Question Answering

The proposed architecture focuses on a Vision Encoder-Decoder scheme specifically designed to compare pairs of chest X-ray images and generate answers about changes between them. The visual component uses a Transformer-based backbone pre-trained and fine-tuned on radiograph data to extract representations from both images. These representations are augmented with a learnable indicator that signals which image is the reference and which is the current one, so that the fusion process can explicitly distinguish between the two inputs.

The text decoder is a lightweight Transformer that processes the tokenized question and integrates the differentiated visual information via cross-attention. Given the relatively limited vocabulary and structure of questions in this domain, a smaller decoder is chosen to generate precise answers without requiring large-scale pre-trained language models. At each generation step, the decoder combines the textual context (the question and previously generated tokens) with the fused visual features, producing the answer autoregressively and ensuring coherence in comparing the reference and current images.

The training strategy is organized in staged phases: first, the visual encoder is fine-tuned exclusively on radiographs to learn domain-specific features; next, the text decoder is integrated and initially trained alone with the encoder frozen, optimizing visual-text fusion in isolation; finally, a joint fine-tuning of both encoder and decoder is performed to adjust the entire system. During this process, techniques such as hard example selection and image augmentations are applied to strengthen the model’s robustness against real variations in radiographs.

Our results confirm that the proposed method clearly outperforms the state of the art in Medical VQA. In our overall evaluations, our model produces answers that are more coherent, fluent, and clinically relevant than those of previous approaches, thanks to an architecture specifically designed to interpret radiological images and generate precise diagnoses. This improvement is reflected across all quality metrics, demonstrating that our training strategy and network modifications foster a deeper understanding of the medical context and yield text that more faithfully mirrors clinical reasoning.

Beyond the aggregate numbers, the qualitative analysis of two extreme examples—both achieving maximum metric scores—reveals a crucial insight. In the first case, the model’s response perfectly matches the true diagnosis, identifying every anomaly and adequately justifying each finding. However, in the second case, despite registering the same high quantitative scores, the model commits a diagnostic error: it either assumes that certain abnormalities have resolved when they remain present.

This contrast highlights a decisive limitation of current metrics: they measure linguistic and structural similarity but fail to capture the clinical validity of the content. In a setting where automated diagnoses can directly influence medical decisions, this gap is unacceptable. Thus, our results not only establish a new benchmark in quantitative performance but also underscore the urgent need to develop evaluation metrics tailored to Medical VQA—metrics capable of verifying diagnostic accuracy and ensuring patient safety.