ExGra-Med: Extended Context Graph Alignment for Medical Vision-Language Models

Duy M. H. Nguyen; Nghiem T. Diep; Trung Q. Nguyen; Hoang-Bao Le; Tai Nguyen; Tien Nguyen; TrungTin Nguyen; Nhat Ho; Pengtao Xie; Roger Wattenhofer; James Zou; Daniel Sonntag; Mathias Niepert

Abstract

State-of-the-art medical multi-modal LLMs (med-MLLMs), such as LLaVA-Med and BioMedGPT, primarily depend on scaling model size and data volume, with training driven largely by autoregressive objectives. However, we reveal that this approach can lead to weak vision-language alignment, making these models overly dependent on costly instruction-following data. To address this, we introduce ExGra-Med, a novel multi-graph alignment framework that jointly aligns images, instruction responses, and extended captions in the latent space, advancing semantic grounding and cross-modal coherence. To scale to large LLMs (e.g., LLaMa-7B), we develop an efficient end-to-end training scheme using black-box gradient estimation, enabling fast and scalable optimization. Empirically, ExGra-Med matches LLaVA-Med’s performance using just 10% of pre-training data, achieving a 20.13% gain on VQA-RAD and approaching full-data performance. It also outperforms strong baselines like BioMedGPT and RadFM on visual chatbot and zero-shot classification tasks, demonstrating its promise for efficient, high-quality vision-language integration in medical AI.

1 Introduction

Generic Multi-Modal Large Language Models (MLLMs) such as GPT-4V (Achiam et al., 2023), LLaVa (Liu et al., 2024b), and Next-GPT (Wu et al., 2023b) unify text, image, and audio processing for tasks like captioning and visual reasoning. A key component in training MLLMs is instruction-following (IF) data (Lou et al., 2023), which involves complex, often multi-turn interactions grounded in image content (Sun et al., 2024). In the medical domain, specialized IF datasets, including medical images, clinical notes, and diagnostic criteria, have been curated to adapt general-purpose MLLMs while leveraging their pre-learned knowledge and minimizing training costs (Xie et al., 2024). For example, LLaVA-Med (Li et al., 2024) samples 600K image-text pairs from PMC-15M (Zhang et al., 2023b), using GPT-4 to generate around 60K multi-modal IF examples. The training involves two pretraining steps: (i) aligning vision encoders and language decoders via projection layers, and (ii) jointly training the model (excluding the vision encoder) on medical IF data using an auto-regressive objective. The resulting model is then fine-tuned for downstream medical tasks.

Following the above approach, most later works have focused on scaling up the amount of medical IF data (Xie et al., 2024; Zhang et al., 2023a; He et al., 2024) or increasing the model size by incorporating larger vision encoders or language decoders (Wu et al., 2023a; Jiang et al., 2024) while relying on the same standard autoregressive learning scheme. Contrary to this, we question the effectiveness of autoregressive objective functions when learning medical-MLLM with IF data. Surprisingly, our findings reveal that autoregressive learning is highly data-hungry during pre-training, i.e., without sufficient medical IF samples, model performance plummets for downstream tasks, even after fine-tuning. To illustrate this, we pre-trained LLAVA-Med using only $10 %$ of the data and compared it to the version trained on $100 %$ . Both models were fine-tuned on two medical visual question-answering tasks - VQA-RAD (Lau et al., 2018) and PathVQA (He et al., 2020) - and their average performance on open- and close-ended questions are compared. The results show a dramatic decline: from $72.64 %$ to $52.39 %$ on VQA-RAD (Figure 1) and from $64.06 %$ to $56.15 %$ on PathVQA (Table 4). This underscores the instability of medical-MLLM trained with autoregressive methods and highlights the problem that these methods require the curation of enough medical IF data to achieve satisfactory performance.

Refer to caption — Figure 1: Our ExGra-Med versus LLaVA-Med across varying instruction-following (IF) pre-training data sizes, highlighting the data-hungry behavior of auto-regressive modeling. Both models are fine-tuned on the same VQA-RAD training set after the pre-training stage at each IF rate. At 100% IF pre-training, ExGra-Med and LLaVA-Med are benchmarked against other state-of-the-art models, all fine-tuned on the same VQA-RAD training set (except GPT-4, which is evaluated without fine-tuning). Circle radius represents the number of model parameters.

To address the limitations of autoregressive training under limited instruction-following data, we propose ExGra-Med, a novel multi-graph alignment framework that strengthens cross-modal understanding in multi-modal large language models (MLLMs). At the core of our approach is the construction of three modality-specific graphs: one for visual features extracted by a vision encoder, and two for different textual variants of the instruction. These graphs represent semantic relationships within and across modalities, and we formulate a combinatorial multi-graph alignment problem to learn consistent triplet-level associations between the image, its instruction, and a semantically enriched variant. This alignment objective is jointly optimized with the autoregressive language modeling loss, enabling the model to enhance semantic depth, coherence, and instruction-following ability. To generate the enriched instruction variant, we use a frozen LLM (GPT-4 Achiam et al. (2023)) to produce a contextually extended version of each instruction that highlights key concept relationships without altering the original intent. The vision encoder and language model (LLaMa Touvron et al. (2023)) then process the image, instruction, and extension independently to produce node embeddings used in the alignment step. Unlike naive data augmentation, our use of GPT-4 enriches supervision and facilitates fine-grained graph-based correspondence learning across modalities (Figure 3).

Our method stands apart from prior multi-modal alignment approaches for LLMs (Park et al., 2024; Li et al., 2023a; Chen et al., 2023a) in two key ways. (i) Instead of merely learning projection layers between frozen vision and language models, we train the LLM directly via a multi-graph alignment framework. (ii) We also extend pairwise contrastive learning by integrating global structural graph constraints, enabling alignment not just between individual image-caption pairs but across entire datasets. This graph-based design captures both feature and relational consistency, critical for handling similar entities in medical data. While multi-graph alignment is typically non-differentiable (Rolínek et al., 2020b) and computationally intensive (Pevzner, 1992), we address these challenges using implicit maximum likelihood estimation (Niepert et al., 2021; Minervini et al., 2023). This enables efficient gradient-based training over large LLMs (e.g., LLaMa-7B) using a barycenter graph (Agueh and Carlier, 2011) for alignment, allowing our model to scale effectively while preserving strong alignment performance.

In summary, we make the following key contributions:

•

We reveal the data-demanding nature of autoregressive modeling in pre-training medical-MLLM (LLaVa-Med), showing that insufficient instruction-following data leads to significant performance drops on downstream tasks, even after fine-tuning.
•

We introduce a new multi-graph alignment objective that establishes triplet correlations among images, their instruction-following context, and their enriched versions. Additionally, we developed an efficient solver for training with LLMs and outlined theoretical properties related to distance and the shortest path in the geodesic space of multi-modal graphs.
•

We empirically demonstrate that using a small amount of pre-training data, ExGra-Med can achieve performance comparable to LLaVa-Med trained on $100 %$ data. Additionally, when trained on larger datasets, ExGra-Med outperforms several state-of-the-art med-MLLMs and multi-modal pre-training algorithms across three Medical VQA tasks, medical visual chat, and the average zero-shot image classification performance on 23 datasets.

2 Related Work

Medical Multi-Modal LLMs. Recent developments in medical-MLLM like Biomed-GPT (Zhang et al., 2023a), MedFlamingo (Moor et al., 2023), Med-Dr (He et al., 2024), LLAVA-Med (Li et al., 2024), and Med-PaLMs (Singhal et al., 2023; Tu et al., 2024) are transforming healthcare by integrating diverse data types and scaling medical instruction data. Biomed-GPT excels with multiple biomedical modalities, MedFlamingo focuses on few-shot learning for medical visual question answering, and LLAVA-Med leverages large-scale biomedical image-text pairs for improved performance. Commonly, these models emphasize scaling medical instruction data and increasing model parameters to enhance accuracy and applicability in real-world medical scenarios. In contrast, our approach examines widely used autoregressive pre-training algorithms and demonstrates that incorporating enriched-context multi-graph alignment of existing instruction samples can significantly enhance medical-MLLM performance without requiring larger models or extensive datasets.

Visual Instruction Tuning. Visual instruction tuning techniques aim to bridge the gap between frozen vision-language models and frozen LLMs trained on unimodal data, enabling them to work effectively in a multi-modal context. These methods involve (i) learning a multi-layer perceptron (MLP) layer to map embeddings from the vision model to the language model as LLaVa (Liu et al., 2024b), VideoLLM (Chen et al., 2023a); (ii) using adapter-based adjustment as LLaMa-adapter (Zhang et al., 2024b), Voxposer (Huang et al., 2023), or (iii) learning multi-modal perceiver by gated cross-attention (Alayrac et al., 2022) or Q-Former as in BLIP-2 (Li et al., 2023a). Pre-training algorithms to train these models can be combined with both auto-regressive and contrastive learning (Park et al., 2024; Zhai et al., 2023) or image-text matching as in (Li et al., 2022, 2023a). Our algorithm differs from those by focusing on directly training LLMs rather than lightweight projectors. This requires a fast solver capable of efficiently handling forward and backward passes through large-scale LLMs with extensive parameters.

Vision-language Pretraining Algorithm. Pre-training algorithms commonly applied for vision-language models, like CLIP (Radford et al., 2021), employ various strategies. Among these, generative approaches are widely used, including masked prediction in language models (Devlin, 2018; Song et al., 2020), or autoregressive algorithms that predict sequential text in LLMs (Liu et al., 2024b; Zhang et al., 2024b). Another prominent direction focuses on discriminative methods, which learn contrastive distances between image-text pairs (Liu et al., 2023; Zhai et al., 2023; Khan and Fu, 2023), optimal transport (Chen et al., 2022; Nguyen et al., 2024a), or impose clustering constraints (Park et al., 2024). Toward multi-modal learning across three or more modalities, there also exist works such as PAC-SSarto et al. (2023), GeoCLAP Khanal et al. (2023), and IMAGEBIND Girdhar et al. (2023), extending the CLIP or InfoNCE Oord et al. (2018) to align embeddings across multiple modalities simultaneously.

Our function departs from these by generalizing them into a combinatorial graph-matching formulation across cross-domain graphs. While LVM-Med (MH Nguyen et al., 2024) is the most similar to our approach, it targets alignment within vision tasks, whereas we align images, instruction-following data, and extended contextual information. We provide a more comprehensive comparison between ExGra-Med and related works in the Appendix.

Scalable Multi-Graph Alignment. Graph alignment across $K$ domains ( $K \geq 3$ ) is highly computationally intensive. Current methods, such as multi-marginal optimal transport (Lin et al., 2022; Piran et al., 2024), Wasserstein barycenters (Nguyen et al., 2024b), and multi-adjacency matrix assumptions (Bernard et al., 2019; Swoboda et al., 2019), relax the problem but are limited to small-scale tasks and require multiple solver steps, making them inefficient for LLM training. In contrast, our algorithm leverages heuristic solvers (Swoboda et al., 2017; Rolínek et al., 2020b) and modern gradient estimation techniques for black-box optimization (Niepert et al., 2021; Minervini et al., 2023), enabling scalable and efficient performance for large language models. A deeper analysis of this factor is discussed in the Ablation study.

3 Multi-graph Alignment Learning

We denote the vision encoder, projector, and large-language model (LLM) models are $f_{θ} (.), h_{ϕ} (.), g_{σ} (.)$ , respectively. Figure 3 illustrates our ExGra-Med algorithm, which learns parameters for these models by solving a triplet alignment between modalities in instruction tuning data. Below, we summarize the notations used before describing each component in detail.

Notation. Given any tensor $𝑻 = (T_{i, j, k, l})$ and matrix $𝑴 = (M_{k, l})$ , we use $𝑻 \otimes 𝑴$ to denote the tensor-matrix multiplication, i.e., the matrix $(\sum_{k, l} T_{i, j, k, l} M_{k, l})_{i, j}$ . Given $𝒀 = [𝒚_{1}, 𝒚_{2}, . . ., 𝒚_{N}] \in ℝ^{N \times d}$ , we define $𝔼 (𝒀) = \frac{1}{N} \sum_{i = 1}^{N} 𝒚_{i} \in ℝ^{d}$ . Moreover, we define the matrix scalar (or inner) product associated with the Frobenius norm between two matrices $𝑴 = (M_{i, j})$ and $𝑵 = (N_{i, j})$ as $⟨ \cdot, \cdot ⟩$ , i.e., $⟨ 𝑴, 𝑵 ⟩ = \sum_{i, j} M_{i, j} N_{i, j}$ . We write $[M] = {1, 2, \dots, M}$ for any natural number $M$ .

3.1 Extended context enriched medical instruction following data

Recent research has demonstrated that incorporating longer context significantly enhances LLMs’ ability to process complex inputs and improves instruction-following by retaining more relevant information (Liu et al., 2024c; An et al., 2024; Pawar et al., 2024). Building on this insight, we expand medical instruction-following data by generating contextually enriched paraphrased versions of existing samples, offering a complementary perspective to the original dataset. There are two key motivations for incorporating both original and extended captions in our multi-graph alignment framework. (i) First, aligning with original captions preserves precise, domain-specific details, while extended captions enhance semantic richness, leading to more robust image embeddings. (ii) Second, this approach helps the LLM generate contextually rich yet semantically consistent responses, improving alignment across diverse linguistic forms (Table 6).

In particular, a typical instruction sample includes ${𝑿_{v}, [𝑿_{q}^{1}, 𝑿_{a}^{1}], . . ., [𝑿_{q}^{L}, 𝑿_{a}^{L}]}$ where $𝑿_{v}$ is an input image, $𝑿_{q}^{l}$ a question, and $𝑿_{a}^{l}$ an answer at round $l$ in multi-round $L$ of a conversation. In the medical domain, most of the questions are generic, and the information answer usually covers the question, so we only focus on extending the answer $𝑿_{a}$ . We leverage the GPT API with a prompt to form an extended context for each $𝑿_{a}^{l}$ by:

𝑿_{a e}^{l} = G P T (𝑿_{q}^{l}, 𝑿_{a}^{l}, p r o m p t), \forall l \in [L] .

(1)

The details for $p r o m p t$ are presented in the Appendix. In short, we ask GPT to provide additional explanations for concepts that appeared in the original answer $𝑿_{a}$ while keeping the content consistent. An example output for $𝑿_{a e}^{l}$ is illustrated in Figure 2 and Table 8. It’s worth noting that other frozen LLMs like Gemini are also valid in our method (Table 4).

3.2 Multi-graph construction on vision-language embedding

Each image $𝑿_{v} \in ℝ^{3 \times H \times W}$ , where $(H, W)$ are the original spatial dimensions, is divided it into a sequence of visual patches $𝑼 = [u_{i}]_{i = 1}^{N}$ with $N = (H \times W) / U$ with $U$ the patch size. Using a pre-trained ViT model $f_{θ}$ , we extract patch-wise features as $𝑽 = f_{θ} (𝑼) \in ℝ^{N \times d_{v}}$ and apply another projector to map it into the projected embedding $𝒁 = h_{ϕ} (𝑽) \in ℝ^{N \times d}$ . We then pool the features from the image patches to define a global description as $𝒁_{v} = 𝔼 (𝒁) \in ℝ^{d}$ . For each language input $𝑿_{c}^{l} \in {𝑿_{a}^{l}, 𝑿_{a e}^{l}}$ with $c \in {a, a e}$ , we assume it has $M$ tokens, i.e., $𝑿_{c}^{l} = [𝒙_{j}]_{j = 1}^{M} \in ℝ^{M}$ , and feed it into the LLM model to extract a set of embedding $𝒁_{c}^{l} = g_{σ} ([𝒙_{j}]_{j = 1}^{M}) = [𝒆_{j}]_{j = 1}^{M} \in ℝ^{M \times d}$ . We subsequently concatenate all multi-round $L$ in each single instruction tuning to define $𝒁_{c} = \frac{1}{L} \sum_{l = 1}^{L} 𝔼 (𝒁_{c}^{l})$ which collects average text embedding of original answers ( $c = a$ ) and their longer-context extended versions ( $c = a e$ ) respectively. Though we adapt simple average pooling feature mechanisms, it remains an effective approach with a clear observed margin of separation between the distinct distributions (Table 6 in the Ablation study).

Given a batch size of $B$ instruction-tuning samples, we now construct three graphs $𝒢_{v} = (𝒱_{v}, ℰ_{v})$ , $𝒢_{a} = (𝒱_{a}, ℰ_{a})$ , and $𝒢_{a e} = (𝒱_{a e}, ℰ_{a e})$ representing for visual image features, text embedding encoded by LLM for original answers and their extended context embedding extended by GPT. Specifically, for each triplet pair ${𝑿_{v}^{(k)}, [𝑿_{a}^{l}]^{(k)}, [𝑿_{a e}^{l}]^{(k)}}_{k}, (k \in [B])$ , we add a node representing $𝑿_{v}^{(k)}$ to $𝒱_{v}$ , a node for $[𝑿_{a}^{l}]^{(k)}$ to $𝒱_{e}$ , and finally a node for $[𝑿_{a e}^{l}]^{(k)}$ to $𝒱_{a e}$ . This results in a set of nodes $𝒱_{v} = {𝑿_{v}^{(1)}, . . ., 𝑿_{v}^{(B)}}$ ; $𝒱_{c} = {[𝑿_{c}^{l}]^{(1)}, . . ., [𝑿_{c}^{l}]^{(B)}}$ for each $c \in {a, a e}$ . We equip node-level feature matrices for these graphs using their embedding computed above, i.e., $𝑭_{v} = {𝒁_{v}^{(1)}, . . ., 𝒁_{v}^{(B)}}$ , $𝑭_{c} = {𝒁_{c}^{(1)}, . . ., 𝒁_{c}^{(B)}}$ . The edges for $ℰ_{v}, ℰ_{c}$ afterward can be created through the k-nearest neighbors algorithm given the feature node matrices $𝑭_{v}, 𝑭_{c}$ . Finally, we can run a message-passing network $m_{α} (.)$ on three built graphs to learn richer node representations. This approach has proven effective for representation learning (Tang et al., 2022; Ju et al., 2024), resulting in aggregated feature-node matrices as ${\hat{𝒁}_{s}^{(1)}, . . ., \hat{𝒁}_{s}^{(B)}} = m_{α} (𝑭_{s}, ℰ_{s})$ , with $s \in {v, a, a e}$ .

3.3 Second-order graph alignment problem

We first provide background information about the second-order graph alignment problem between two arbitrary graphs $𝒢_{1} = (𝒱_{1}, ℰ_{1})$ and $𝒢_{2} = (𝒱_{2}, ℰ_{2})$ , which is known as quadratic assignment problem. It occurs in several problems in vision and computer graphics to find correspondences between two graph structures under constraints on the consistency of node features and the graph structures (Zanfir and Sminchisescu, 2018; Haller et al., 2022; Ehm et al., 2024).

We denote by $𝑽 \in {0, 1}^{| 𝒱_{1} | | 𝒱_{2} |}$ , with $| 𝒱_{1} | = M$ and $| 𝒱_{2} | = N$ , the indicator matrix of matched vertices, that is, $V_{i, j} = 1$ if a vertex $v_{i} \in 𝒱_{1}$ is matched with $v_{j} \in 𝒱_{2}$ and $V_{i, j} = 0$ otherwise. That is, $𝑽$ is a binary matrix with exactly one non-zero entry in each row and column. Similarly, we set $𝑬 \in {0, 1}^{| ℰ_{1} | | ℰ_{2} |}$ as the indicator tensor of match edges, that is, $E_{i, k, j, l} = 1$ if $V_{i, j} = 1$ and $V_{k, l} = 1$ and $E_{i, k, j, l} = 0$ otherwise. This implies that the tensor $𝑬$ is fully determined by the matrix $𝑽$ , that is, $E_{i, k, j, l} = V_{i, j} V_{k, l}$ . We also define the vertex affinity matrix and edge affinity tensor as $𝑨^{v} \in ℝ^{| 𝒱_{1} | | 𝒱_{2} |}$ and $𝑨^{e} \in ℝ^{| ℰ_{1} | | ℰ_{2} |}$ , respectively. The set $𝒜 (𝒢_{1}, 𝒢_{2})$ indicates for all admissible pairs $(𝑽, 𝑬)$ that encode a valid matching between $𝒢_{1}$ and $𝒢_{2}$ .

𝒜 (𝒢_{1}, 𝒢_{2}) = {𝑽 \in {0, 1}^{M \times N} : \sum_{i = 1}^{M} V_{i, j} = 1, \sum_{j = 1}^{N} V_{i, j} = 1} .

(2)

The second-order graph alignment (SGA) problem now is defined as:

	$SGA (𝑨^{v}, 𝑨^{e}) = a r g m i n_{𝑽 \in 𝒜 (𝒢_{1}, 𝒢_{2})} ⟨ 𝑨^{v} + 𝑨^{e} \otimes 𝑽, 𝑽 ⟩$
	$= a r g m i n_{𝑽 \in 𝒜 (𝒢_{1}, 𝒢_{2})} \sum_{i, j} A^{v}_{i, j} V_{i, j} + \sum_{i, j, k, l} A^{e}_{i, j, k, l} V_{i, j} V_{k, l} .$		(3)

3.4 Scalable Multi-graph Alignment

Our aim is to solve the graph alignment between $𝒢_{v}, 𝒢_{a}$ , and $𝒢_{a e}$ to form a triplet constraint between input image embedding, its original instruction embedding, and the extended version ones. However, solving a structure-aware graph alignment between $K$ domains ( $K \geq 3$ ) is computationally expensive. One potential solution is to perform pairwise graph alignments $(\binom{K}{2})$ times, as shown in Eq. (3.3), while applying specific constraints to maintain consistency between correspondences (Bernard et al., 2019; Swoboda et al., 2019) (Table 6). However, these approaches become impractical as $K$ increases, making them unsuitable for large-scale problems or for integrating multimodal inputs such as text, images, audio, or medical health records.

Another direction leverages the barycenter concept from optimal transport Guo et al. (2020); Altschuler and Boix-Adsera (2022), which identifies a central distribution that minimizes the weighted sum of Wasserstein distances to the given input distributions. We follow this idea to reformulate the alignment of $K$ graphs into $K$ separate alignments with a barycenter graph. Unlike previous unsupervised methods that estimate the barycenter before aligning, we directly define the barycenter using known triplet pairs across the three graphs. This significantly reduces complexity, making our solver more efficient in LLM settings.

Specifically, we define a new barycenter graph $𝒢_{b r} = (𝒱_{b r}, ℰ_{b r})$ where $𝒱_{b r} = {v_{b r}^{(1)}, . . ., v_{b r}^{(B)}}$ with $v_{b r}^{(k)} = \bar{𝑿}^{(k)} = {𝑿_{v}^{(k)}, [𝑿_{a}^{l}]^{(k)}, [𝑿_{a e}^{l}]^{(k)}}$ and a correspondence feature node as $𝑭_{b r} = \frac{1}{3} {\sum_{s} \hat{𝒁}_{s}^{(1)}, . . ., \sum_{s} \hat{𝒁}_{s}^{(B)}}$ with $s \in {v, a, a e}$ . The edge set $ℰ_{b r}$ is formed similarly to another graph by running the k-nearest neighbor on feature node $𝑭_{b r}$ . We now state the multi-graph alignment as:

SGA (𝑨_{s}^{v}, 𝑨_{s}^{e}) = a r g m i n_{𝑽_{s} \in 𝒜 (𝒢_{s}, 𝒢_{b r})} \sum_{s \in {v, a, a e}} ⟨ 𝑨_{s}^{v} + 𝑨_{s}^{e} \otimes 𝑽_{s}, 𝑽_{s} ⟩,

(4)

where $𝑽_{s}$ is the indicator matrix representing for valid mapping between $𝒢_{s}$ and $𝒢_{b r}$ , $𝑨_{s}^{v} \in ℝ^{| 𝒱_{s} | | 𝒱_{b r} |}$ and $𝑨_{s}^{e} \in ℝ^{| ℰ_{s} | | ℰ_{b r} |}$ be vertex affinity matrix and edge affinity tensor between $𝒢_{s}$ and $𝒢_{b r}$ . For e.g., $(𝑨_{s}^{v})_{i j} = d (\hat{𝒁}_{s}^{(i)}, \frac{1}{3} \sum_{s} \hat{𝒁}_{s}^{(j)})$ with $d (.)$ be a distance metric (e.g., cosine distance) measuring similarity between node $i^{t h}$ in $𝒢_{s}$ and node $j^{t h}$ in $𝒢_{b r}$ .

To address the NP-Hard nature of aligning each graph to the barycenter graph $𝒢_{c}$ , which arises from its combinatorial complexity, we employ efficient heuristic solvers utilizing Lagrange decomposition techniques (Swoboda et al., 2017; Rolínek et al., 2020b).

3.5 Backpropagation with Black-box Gradient Estimation

Given $\hat{𝑽}_{s} = SGA (𝑨_{s}^{v}, 𝑨_{s}^{e})$ be solution obtained from the solver, we aim to learn feature representation for LLMs such that $\hat{𝑽}_{s}$ be identical to true triplet alignments explicitly indicated by the barycenter graph. By denoting $𝑽_{s}^{*}$ be an optimal mapping between the graph $𝒢_{c}$ to $𝒢_{b r}$ , we compute the following total of hamming loss function:

ℒ (\hat{𝑽}_{s}, 𝑽_{s}^{*}) = \sum_{s \in {v, a, a e}} ⟨ \hat{𝑽}_{s}, (1 - 𝑽_{s}^{*}) ⟩ + ⟨ 𝑽_{s}^{*}, (1 - \hat{𝑽}_{s}) ⟩ .

(5)

However, computing the gradient of the loss function with respect to the matching problem inputs $(𝑨_{s}^{v}, 𝑨_{s}^{e})$ , i.e., $\nabla_{𝑨_{s}^{v}, 𝑨_{s}^{e}} ℒ (\hat{𝑽}_{s}, 𝑽_{s}^{*})$ , poses a challenge due to the piecewise constant nature of the graph matching objective in Eq. (4) (Pogančić et al., 2020; Rolínek et al., 2020a). To address this, we resort to the IMLE (Niepert et al., 2021; Minervini et al., 2023), a method that estimates gradients and enables backpropagation through the alignment algorithm by taking the difference between solutions of noise-perturbed alignments.

In particular, given $(ϵ, ϵ^{'}) \sim G u m b l e (0, 1)$ and for each $s \in {v, a, a e}$ , we compute:

	$(\frac{\partial ℒ}{\partial 𝑨_{s}^{v}}, \frac{\partial ℒ}{\partial 𝑨_{s}^{e}})$	$\approx \tilde{𝑽}_{s} - SGA (𝑨_{s, λ}^{v}, 𝑨_{s, λ}^{e}) where \tilde{𝑽}_{s} = SGA (𝑨_{s}^{v} + ϵ, 𝑨_{s}^{e} + ϵ^{'}),$
	$(𝑨_{s, λ}^{v}, 𝑨_{s, λ}^{e})$	$= (𝑨_{s}^{v} + ϵ, 𝑨_{s}^{e} + ϵ^{'}) - λ \nabla_{\tilde{𝑽}_{s}} ℒ (\tilde{𝑽}_{s}, 𝑽_{s}^{*}), with λ is a step size.$

3.6 Graph Distance Properties via Structure Alignment

We provide theoretical insights into the graph-matching problem in Eq.(3.3). Once the optimal matching is found, it defines a valid distance metric between graphs (Theorem1) based on joint node-edge representations. Additionally, the geodesic path connecting two graphs (Theorem 1) can be derived from these alignments, enabling dynamic formulations in optimal transport and sampling strategies in representation learning. In particular, we define a discrete distance between two graphs given a solution to the matching alignment as follows:

d_{SGA} (𝒢_{1}, 𝒢_{2})

= \min_{𝑽 \in 𝒜 (𝒢_{1}, 𝒢_{2})} (\sum_{i, j} A^{v}_{i, j} V_{i, j} + \sum_{i, j, k, l} A^{e}_{i, j, k, l} V_{i, j} V_{k, l}) .

(6)

We provide an informal definition of the space of structured graphs below to introduce the main theorems. A more formal treatment, including proofs and definitions, is presented in the Appendix.

Definition 1 (Space of all structured graphs).

$𝕊 (ℱ)$ is the space of all structured graphs defined over a node metric feature space $(ℱ, d_{f})$ , where each graph is associated with an edge structure space $(𝒮, d_{s})$ and a mixing measure $μ = \sum_{i = 1}^{N} w_{i} δ_{(f_{i}, s_{i})}$ over the product space $(ℱ \times 𝒮)$ .

[Metric properties] The distance $d_{SGA}$ in Eq. (6) defines a metric in $𝕊 (ℱ)$ .

The $d_{SGA}$ distance above is zero if and only if there exists a one-to-one mapping between the graph vertices that preserves both shortest paths and features and both graphs have the same number of vertices.

[Geodesic space] The space $𝕊 (ℱ)$ equipped with the $d_{SGA}$ distance is geodesic.

4 Experiments

4.1 Implementation Details

Table 1: Fine-tuning performance on MedVQA tasks (pre-trained 10%). Bold indicates the best values among pre-training algorithms (Sec. 4.3), excluding LLaVA-Med.

Method VQA-RAD SLAKE PathVQA Overall Open Closed Avg. Open Closed Avg. Open Closed Avg. LLaVA-Med (100%) 63.65 81.62 72.64 83.44 83.41 83.43 36.78 91.33 64.06 73.37 LLaVA-Med (10%) 43.38 $↓$ 20.3 61.4 $↓$ 20.2 52.39 $↓$ 20.3 80.94 $↓$ 2.5 80.29 $↓$ 3.1 80.62 $↓$ 2.8 24.26 $↓$ 13.69 88.03 $↓$ 3.18 56.15 $↓$ 7.91 63.05 $↓$ 10.3 InfoNCE 59.39 77.57 68.48 82.4 83.17 82.78 34.59 91.45 63.02 71.43 PLOT 16.86 26.47 21.67 37.81 56.25 47.03 11.79 81.36 46.58 38.42 SigLIP 56.99 77.94 67.47 80.86 80.53 80.69 18.08 50.85 34.465 60.88 VLAP 57.49 76.47 66.98 80.05 82.21 81.13 32.21 91.16 61.685 69.93 GeoCLAP 60.68 75.37 68.03 82.64 85.10 83.87 35.12 91.15 63.14 71.68 PAC-S 57.72 72.79 65.26 83.78 81.49 82.64 35.01 91.36 63.19 70.36 IMAGEBIND 57.31 75.74 66.53 80.79 84.13 82.46 34.61 91.42 63.02 70.67 ExGra-Med 66.02 79.04 72.52 84.92 85.10 85.01 37.25 91.45 64.34 73.96

Table 2: Fine-tuning performance on MedVQA tasks (pre-trained 40%). Bold indicate for best values among pre-training algorithms (Sec. 4.3) excluding LLaVA-Med.

Method VQA-RAD SLAKE PathVQA Overall Open Closed Avg. Open Closed Avg. Open Closed Avg. LLaVA-Med (100%) 63.65 81.62 72.64 83.44 83.41 83.43 36.78 91.33 64.06 73.37 LLaVA-Med (40%) 62.23 $↓$ 1.4 79.41 $↓$ 2.2 70.82 $↓$ 1.8 84.42 $↑$ 1.0 83.65 $↓$ 0.2 84.04 $↑$ 0.6 31.86 $↓$ 4.9 84.99 $↓$ 6.3 58.43 $↓$ 5.6 71.09 $↓$ 2.3 InfoNCE 63.11 77.57 70.34 82.68 83.89 83.29 33.58 89.62 61.6 71.74 PLOT 64.36 79.41 71.89 83.38 82.93 83.16 35.11 89.59 62.35 72.46 SigLIP 63.02 81.25 72.14 81.26 80.29 80.77 36.01 90.86 63.435 72.12 VLAP 63.17 79.04 71.11 83.38 83.89 83.64 35.62 90.83 63.225 72.66 GeoCLAP 62.28 82.72 72.5 82.64 85.2 83.92 33.2 75.05 54.13 70.18 PAC-S 63.77 79.41 71.59 84.52 85.58 85.05 27.11 85.34 56.23 70.96 IMAGEBIND 64.73 78.68 71.71 82.31 84.62 83.47 35.76 87.08 61.42 72.20 ExGra-Med 66.01 82.72 74.37 84.47 85.82 85.15 37.41 91.27 64.34 74.57

Table 3: ExGra-Med versus LLaVa-Med when pre-trained with extended captions (ext.cap). Performance is reported on VQA-RAD downstream after fine-tuning.

Method	VQA-RAD
Method	Open	Closed	Avg.
LLaVA-Med (100%)	63.65	81.62	72.64
LLaVA-Med (10%)	43.38 $↓$ 20.3	61.4 $↓$ 20.2	52.39 $↓$ 20.3
LLaVA-Med (10%) ext.cap	63.07	75.74	69.41
ExGra-Med (10%)	66.02	79.04	72.52
LLaVA-Med (40%)	62.23 $↓$ 1.4	79.41 $↓$ 2.2	70.82 $↓$ 1.8
LLaVA-Med (40%) ext.cap	63.68	79.78	71.73
ExGra-Med (40%)	66.01	82.72	74.37

Table 4: ExGra-Med results with different frozen LLMs. It shows that Gemini and Qwen is also effective within our method.

Method	VQA-RAD	SLAKE
ExGra-Med (GPT-4), $10 %$	72.52	85.01
ExGra-Med (Gemini), $10 %$	71.09	83.98
ExGra-Med (Qwen), $10 %$	70.13	84.7
LLaVa-Med (Baseline) $10 %$	52.39	80.62
ExGra-Med (GPT-4), $40 %$	74.37	85.15
ExGra-Med (Gemini), $40 %$	73.26	85.10
ExGra-Med with synonyms, $40 %$	72.39	82.93
LLaVa-Med (Baseline) $40 %$	70.82	84.04

Model Architecture Configuration. We use the LLaMA-7B large language model (Touvron et al., 2023), the CLIP-ViT-L-Patch14 visual encoder (Radford et al., 2021), and an MLP projection similar to LLaVA 1.5 (Liu et al., 2024a). Stage 1 follows the standard LLaVA-Med (Li et al., 2024) setup, while stage 2 incorporates our multi-graph alignment with autoregressive training. For multi-graph alignment, a 2-layer graph convolutional network is applied to the output of the Projection and LLM Decoder (handling both image and text modalities). We train for 1 epoch in stage 1 and 3 epochs in stage 2 using the same dataset as LLava-Med. The model is optimized using Adam (Kingma, 2014) with CosineAnnealingLR scheduler and learning rates of $2 e - 3$ and $2 e - 5$ for stages 1 and 2, respectively.

Pre-training data. We use the same dataset as LLaVA-Med (Li et al., 2024). Stage 1 includes 600K image-text pairs filtered from PMC-15M, converted into instruction-following data with simple image descriptions. Stage 2 comprises 60K image-text pairs from PMC-15M across five modalities: CXR, CT, MRI, histopathology, and gross pathology. GPT-4 is then employed to generate multi-round Q&A in a tone mimicking visual interpretation, converting these pairs into an instruction-following format.

Running-time. We train ExGra-Med using 4 A100-GPUs per with 80GB for both stages and complete the training process for stage 1 in 6.5 hours and for stage 2 in 7.5 hours. With original LLaVA-Med (version 1.5) (Li et al., 2024), the training process for stage 1 finishes in 6.5 hours, and for stage 2 finishes in 7 hours. In total, we need an extra 0.5 hour to complete the whole pre-training process compared to the LLaVa-Med.

4.2 Autoregressive Training is Data-hungry

We highlight the data-intensive nature of autoregressive training by evaluating LLaVA-Med, a state-of-the-art biomedical multimodal language model. LLaVA-Med follows a two-stage training process: Stage 1 aligns image-text tokens with biomedical concepts, and Stage 2 fine-tunes the model for instruction tasks. We pre-trained LLaVA-Med on varying data amounts (10%, 40%, 70%) and fine-tuned it on the VQA-RAD dataset. As shown in Figure 1, performance drops significantly at 10% pre-training compared to full training, revealing the autoregressive mechanism’s data dependency issue in medical-MLLMs. This highlights the challenge of weak connections between visual features and text embeddings without sufficient instruction-tuning data.

In contrast, our ExGra-Med specifically learns image-text alignment by enforcing semantic consistency across images, instruction responses, and extended contexts. Using the same setup as LLaVA-Med, ExGra-Med excels even with limited data, achieving $72.52 %$ at just $10 %$ pretraining, far above LLaVA-Med’s $52.39 %$ and consistently outperforms LLaVA-Med across instruction tuning rates ( $40 % - 100 %$ ).

Impact of Longer Training and Enriched Captions on LLaVa-Med Performance. We conduct a deeper analysis of the data-hungry phenomenon by examining LLaVa-Med when (i) trained longer in Stage 2 (with an additional hour) and when (ii) incorporating extended captions in an autoregressive manner, as done in ExGra-Med. The experiments were performed using 10% and 40% pre-training settings, followed by fine-tuning on the VQA-RAD dataset. From the results in Table 4, we draw two key conclusions. First, extended captions improve LLaVa-Med performance, particularly in the data-scarce 10% pre-training scenario. However, in both settings, ExGra-Med demonstrates superior performance with significant margins, highlighting the effectiveness of its multi-graph alignment strategy in mitigating the data-hungry issues of the autoregressive mechanism.

4.3 Multi-modal Pre-training Comparison

To further demonstrate the strengths of ExGra-Med, we compare its performance against other vision-language pre-training methods currently used for visual instruction tuning to enhance frozen vision-language models

Datasets. We test pre-trained models on three prominent biomedical VQA datasets: VQA-RAD (Lau et al., 2018), SLAKE (Liu et al., 2021), and PathVQA (He et al., 2020). VQA-RAD includes 3,515 questions across 315 radiology images, while SLAKE contains 642 radiology images from various body parts and over 7k QA pairs. PathVQA, focused on pathology, features 5k images and 32.8k questions. All datasets include open-ended (e.g., what, why, where) and closed-ended (yes/no or two-option) question types. We report performance using recall for open-ended, which evaluates the proportion of ground-truth tokens that appear in the generated sequences and accuracy for the closed-ended questions.

Baselines. We compare seven approaches, including:

•

two modalities-based methods such as InfoNCE-based methods (Khan and Fu, 2023; Liu et al., 2023), SigLIP (Zhai et al., 2023), PLOT (Chen et al., 2022), VLAP (Park et al., 2024). Among this, SigLIP adapts the Sigmoid loss on image-text pairs to break the global view of the pairwise similarities for normalization, resulting in scaling in large batch sizes. PLOT defines optimal transport as a distance between visual image patches and text embedding. In contrast, VLAP uses assignment prediction to bridge the modality gap between the visual and LLM embeddings.
•

multi-modal learning across three modalities such as image, text, voice, or augmented image—is explored by methods like PAC-S Sarto et al. (2023), GeoCLAP Khanal et al. (2023), and IMAGEBIND Girdhar et al. (2023). PAC-S integrates contrastive losses across multiple modality pairs: (image–text), (image–augmented text), and (text–augmented text). GeoCLAP applies CLIP-style contrastive learning to all cross-domain pairs, such as audio–text and text–image. Similarly, IMAGEBIND generalizes the InfoNCE loss to learn unified embeddings across diverse modalities.

We train the baselines under the same settings as ExGra-Med with varying pre-training data rates and compare their performance on downstream tasks.

Results. Tables 4–16 show our performance and baselines under $10 %$ , $40 %$ , and $100 %$ instruction-tuning data. While most contrastive baselines improve over LLaVA-Med at $10 %$ , ExGra-Med consistently outperforms all methods across settings. It excels particularly on open-ended questions requiring external knowledge and maintains stable gains across all VQA datasets. In contrast, some methods like SigLIP peak at $40 %$ (e.g., $72.14 %$ Avg on VQA-RAD) but drop notably at $100 %$ (down over $6 %$ ), whereas ExGra-Med improves further to $74.91 %$ (Avg) and $74.75 %$ (Overall).

Table 5: Comparison with other Med-MLLMs on MedVQA tasks. All models (except GPT-4) are fine-tuned on the same training set in each VQA task. These Med-MLLMs differ notably in model size, training data volume, and pre-training strategies - e.g., ExGra-Med (7B, 60K GPT-4 augmented samples) vs. MedDR (40B, 2M samples).

Method #Para VQA-RAD SLAKE PathVQA Overall Open Closed Avg. Open Closed Avg. Open Closed Avg. LLaVA-Med 7B 63.65 81.62 72.64 83.44 83.41 83.43 36.78 91.33 64.06 73.37 BiomedGPT-B 182M 60.9 81.3 71.1 84.3 89.9 87.1 28 88 58 72.07 M2I2 - 61.8 81.6 71.7 74.7 91.1 82.9 36.3 88 62.15 72.25 BioMed-CLIP 422M 67.6 79.8 73.7 82.5 89.7 86.1 Med-Dr 40B 37.5 78.9 58.2 74.2 83.4 78.8 33.5 90.2 61.85 66.28 LLaVA (general) 7B 50 65.1 57.55 78.2 63.2 70.7 7.7 63.2 35.45 54.57 GPT-4 200B 39.5 78.9 59.2 33.6 43.6 38.6 Med-MoE (Phi2) 3.6B 58.55 82.72 70.64 85.06 85.58 85.32 34.74 91.98 63.36 73.11 Med-MoE (Stable LM) 2B 50.08 80.07 65.08 83.16 83.41 83.29 33.79 91.30 62.55 70.3 ExGra-Med 7B 66.35 83.46 74.91 85.34 85.58 85.46 36.82 90.92 63.87 74.75 ExGra-Med (DCI) 7B 67.03 83.46 75.25 84.88 85.58 85.23 37.77 91.86 64.82 75.1

Table 6: ExGra-Med ablation study. Results are presented as average scores on VQA-RAD and SLAKE, using pre-trained weights on 10%, 40%, 100%.

Method	VQA-RAD	SLAKE
ExGra-Med (Full, 10%, $α = 1.0$ )	72.52	85.01
- (ii) ExGra-Med (Full, 10%, $α = 0.1$ )	65.95	82.9
- (ii) ExGra-Med (Full, 10%, $α = 0.5$ )	67.72	82.33
ExGra-Med (Full, 40%)	74.37	84.99
- (iii) ExGra-Med w/o ext. context	72.12	81.95
- (iv) ExGra-Med w/o ori. caption	72.58	82.31
- (v) ExGra-Med w/o message passing	73.90	84.29
- (vi) ExGra-Med in two stages	72.81	84.14
ExGra-Med (Full, 100%)	74.91	85.46
- (vii) ExGra-Med w/o barycenter graph	73.88	84.34

4.4 Med-VQA Comparison with Medical MLLMs

We now compare ExGra-Med pre-trained with $100 %$ data against other medical foundation models, each trained on varying datasets and employing different architectures or model sizes.
Baselines. We benchmark eight competitors, both generic or medical foundation models, including LLaVA (Liu et al., 2024b), LLaVA-Med (Li et al., 2024), Med-Flamingo (Moor et al., 2023), Med-Dr (He et al., 2024), Biomed-GPT (Zhang et al., 2023a), M2I2 (Li et al., 2023b), GPT-4o (Achiam et al., 2023) and Med-MoE (Jiang et al., 2024). Whilst LLaVA and GPT-4o have no medical background, the others are pre-trained on a variety of biomedical knowledge. With the exception of LLaVa, which we reproduced, the results for the other baselines are taken from the literature. Moreover, we also present an enhanced version, ExGra-Med + DCI, which integrates multi-scale visual features from vision encoders (Yao et al., 2024), potentially benefiting medical image analysis by considering both local (detailed) and global (contextual) features.

Results. Overall, both versions of ExGra-Med outperform baseline models (Table 6), with the DCI variant achieving the best results on PathVQA ( $64.82 %$ average, $75.1 %$ overall). Compared to LLaVA-Med, it shows notable gains: + $2.01 %$ on VQA-RAD, + $2.03 %$ on SLAKE, and + $0.76 %$ on PathVQA. Despite using only 7B parameters, both ExGra-Med models surpass the 40B Med-Dr across all datasets.

4.5 Medical Visual Chatbot Evaluation & Zero-shot Image Classification

Medical Visual Chatbot.

We present in Section E Appendix ExGra-Med results compared against several SOTA general and Med-MLLMs such as LLaVA, GPT-4o, LLaVA-Med, Med-Flamingo, Med-Dr, and Biomed-GPT. Among these, we observe that ExGra-Med is the best model across question types.

Zero-shot Image Classification as MedVQA.

Results are presented in Section F of the Appendix. In short, we outperform other models across all datasets, particularly excelling in microscopy, where it surpasses RadFM by $8.2 %$

4.6 Additional Analysis

Table 7: Results of fully finetuning vs LoRA finetuning on VQA-RAD dataset.

Method VQA-RAD Open VQA-RAD Closed VQA-RAD Avg Param LLaVa-Med (LoRa) 62.06 75.00 68.53 2.1B ExGra-Med (LoRa) 63.55 79.41 71.48 2.1B LLaVa-Med (Fully-finetuning) 63.65 81.62 72.64 7B ExGra-Med (Fully-finetuning) 66.35 83.46 74.91 7B

Potentially Hallucination in Extended Captions. We conducted a user study with five general practitioners from top public hospitals (Appendix Section K). In Stage 2 of pre-training, each expert evaluated 200 image-text pairs (1,000 total) across five modalities - chest X-ray, CT, MRI, histology, and others - rating the completeness and accuracy of GPT-4-generated extended captions. As shown in Figures 10–14, most scores ranged from 3 to 5, with few low ratings, confirming the overall consistency and quality of the extended outputs. Also, these extended captions are used only during pre-training to guide latent space alignment. They are excluded during fine-tuning on downstream tasks. As such, we argue that a small amount of noise in the extended captions should have minimal impact on overall performance, since they do not directly affect the model’s task-specific adaptations.

Other Factors. We examine ExGra-Med results under following settings:

•

(i) generalization to other frozen LLMs (GPT-4, Gemini Team et al. (2023), and Qwen3-8B LLM Yang et al. (2025)) to generate extended captions and how the method works with simple synonyms (Table 4).
•

ii) contribution of coefficient ( $α$ ) combine multi-graph alignment with auto-regressive.
•

iii) without using extended contexts, i.e., simplifying from the three graph alignment to two cross-domain alignments (image vs. original captions).
•

(iv) without using original captions, i.e., only extended ones are used.
•

(v) applying message passing to enhance node features.
•

(vi) using multi-graph alignment in both steps (default uses only Step-2).
•

(vii) solving three pair-wise graph alignments in multi-graph alignment rather than solving through a barycenter graph (Eq.(4) in Sec. 3.4).
•

(viii) using parameter-efficient finetuning with LoRa Hu et al. (2022) rather than fully finetuning on downstream tasks.

Tables 4-6 show results (i)-(vii) where the most important factors are highlighted. We further observe that ExGra-Med generalizes effectively across distinct LLM paraphrase generators such as GPT-4, Gemini, and Qwen3-8B. The stable performance across these models indicates that ExGra-Med is not tightly coupled to a specific language model architecture, but instead captures transferable alignment mechanisms applicable to a wide range of paraphrastic contexts.

In Table 7, we report the results of ExGra-Med and LLaVA-Med on the VQA-RAD dataset for the (viii) setting. As shown, ExGra-Med consistently outperforms LLaVA-Med even when both models are fine-tuned using LoRA, demonstrating the robustness of our approach under parameter-efficient adaptation. Nevertheless, as expected, LoRA-based fine-tuning yields a modest performance drop compared to full fine-tuning. We believe that to bridge this gap, pre-training on a larger-scale medical instruction-tuning dataset (e.g., the 21M samples curated from MedTrinity Xie et al. (2024)) would allow the LoRA setup to more closely match the performance of fully fine-tuned models. Additional analyses on average pooling features, k-nearest neighbors for graph construction (Section 3.2).

Current Limitations and Future Work. While our experiments have primarily focused on the LLaVa model, it is essential to validate the effectiveness and adaptability of ExGra-Med with alternative architectures, such as the Flamingo model (Alayrac et al., 2022), which has shown promising results in vision-language tasks. Expanding the evaluation to include other state-of-the-art models can provide a broader perspective on the generalizability and robustness of ExGra-Med. Furthermore, incorporating vision encoders or large language models (LLMs) that are specifically pre-trained on medical datasets (Chen et al., 2023b; MH Nguyen et al., 2024; Zhao et al., 2024; Chen et al., 2025; Zhang et al., 2024a) presents a compelling opportunity to enhance both performance and domain-specific understanding. These specialized models are designed to capture the nuanced characteristics of medical data, which could further enhance the robustness of ExGra-Med in complex biomedical scenarios. Moreover, extending our mechanism to the setting of medical visual chain-of-thought Johnson et al. (2019); Le-Duc et al. (2025) reasoning represents a promising direction for improving both the overall performance and the trustworthiness of the model.

5 Conclusion

We have shown that enforcing triplet correlations among images, instructions, and extended captions significantly enhances vision-language alignment - an area where autoregressive models like LLaVA-Med struggle, particularly under limited data and domain shift, which manifests as a strong dependence on large-scale pre-training data. To this end, we introduce ExGra-Med, a multi-graph alignment algorithm that trains efficiently, matches LLaVA-Med using only 10% of the data, and outperforms other state-of-the-art models across tasks. These results highlight that selecting an effective learning algorithm for LLMs is as crucial as scaling model size or data volume.

Acknowledment

This work was supported by Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany’s Excellence Strategy - EXC 2075 – 390740016, the DARPA ANSR program under award FA8750-23- 2-0004, the DARPA CODORD program under award HR00112590089. The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Duy M. H. Nguyen. Duy M. H. Nguyen and Daniel Sonntag are also supported by the No-IDLE project (BMBF, 01IW23002), the MASTER project (EU, 101093079), and the Endowed Chair of Applied Artificial Intelligence, Oldenburg University.

References

J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. (2023) Gpt-4 technical report. arXiv preprint arXiv:2303.08774. Cited by: §1, §1, §4.4. ↩ a b c
M. Agueh and G. Carlier (2011) Barycenters in the wasserstein space. SIAM Journal on Mathematical Analysis 43 (2), pp. 904–924. Cited by: §1. ↩ a
J. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al. (2022) Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems 35, pp. 23716–23736. Cited by: §2, §4.6. ↩ a b
J. M. Altschuler and E. Boix-Adsera (2022) Wasserstein barycenters are np-hard to compute. SIAM Journal on Mathematics of Data Science 4 (1), pp. 179–203. Cited by: §3.4. ↩ a
C. An, F. Huang, J. Zhang, S. Gong, X. Qiu, C. Zhou, and L. Kong (2024) Training-free long-context scaling of large language models. arXiv preprint arXiv:2402.17463. Cited by: §3.1. ↩ a
A. Asraf and Z. Islam (2021) COVID19 pneumonia and normal chest x-ray pa dataset. mendeley data v1 (2021). Elsevier, Amsterdam, The Netherlands, Tech. Rep. Cited by: Table 12. ↩ a
F. Bernard, J. Thunberg, P. Swoboda, and C. Theobalt (2019) Hippi: higher-order projected power iterations for scalable multi-matching. In Proceedings of the ieee/cvf international conference on computer vision, pp. 10284–10293. Cited by: §2, §3.4. ↩ a b
[8] (2023) Blood cell images. Note: https://www.kaggle.com/datasets/paultimothymooney/blood-cellsAccessed: 2024-09-30 Cited by: Table 11. ↩ a
D. Cao and F. Bernard (2023) Self-supervised learning for multimodal non-rigid 3d shape matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17735–17744. Cited by: §D.2. ↩ a
G. Chen, W. Yao, X. Song, X. Li, Y. Rao, and K. Zhang (2022) Plot: prompt learning with optimal transport for vision-language models. International Conference on Learning Representations. Cited by: 2nd item, §2, 1st item. ↩ a b c
G. Chen, Y. Zheng, J. Wang, J. Xu, Y. Huang, J. Pan, Y. Wang, Y. Wang, Y. Qiao, T. Lu, et al. (2023a) Videollm: modeling video sequence with large language models. arXiv preprint arXiv:2305.13292. Cited by: §1, §2. ↩ a b
H. Chen, R. Shukla, R. Wu, S. Yang, D. Duong-Tran, D. M. H. Nguyen, M. Niepert, C. Beeche, J. Gee, J. Duda, et al. (2025) Adapting vision-language models for 3d ct/mri understanding on pmbb via slice selection and explanation analysis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2273–2282. Cited by: §4.6. ↩ a
P. Chen (2018) Cited by: Table 12. ↩ a
Z. Chen, A. H. Cano, A. Romanou, A. Bonnet, K. Matoba, F. Salvi, M. Pagliardini, S. Fan, A. Köpf, A. Mohtashami, et al. (2023b) Meditron-70b: scaling medical pretraining for large language models. arXiv preprint arXiv:2311.16079. Cited by: §4.6. ↩ a
[15] Chest ct-scan images dataset. Note: https://tianchi.aliyun.com/dataset/93929Accessed: 2024-09-30 Cited by: Table 13. ↩ a
L. Chizat and F. Bach (2018) On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport. In Advances in Neural Information Processing Systems, Vol. 31. Cited by: §A.2. ↩ a
M. E. Chowdhury, T. Rahman, A. Khandakar, R. Mazhar, M. A. Kadir, Z. B. Mahbub, K. R. Islam, M. S. Khan, A. Iqbal, N. Al Emadi, et al. (2020) Can ai help in screening viral and covid-19 pneumonia?. Ieee Access 8, pp. 132665–132676. Cited by: Table 12. ↩ a
J. P. Cohen, P. Morrison, L. Dao, K. Roth, T. Q. Duong, and M. Ghassemi (2020) Covid-19 image data collection: prospective predictions are the future. arXiv preprint arXiv:2006.11988. Cited by: Table 12. ↩ a
[19] Covid ct dataset. Note: https://tianchi.aliyun.com/dataset/106604Accessed: 2024-09-30 Cited by: Table 13. ↩ a
[20] (2023) Covid-19 image dataset: 3 way classification - covid-19, viral pneumonia, normal. Note: https://tianchi.aliyun.com/dataset/93853 Cited by: Table 12. ↩ a
M. Cuturi (2013) Sinkhorn distances: lightspeed computation of optimal transport. Advances in neural information processing systems 26. Cited by: 2nd item. ↩ a
J. Devlin (2018) Bert: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. Cited by: §2. ↩ a
V. Ehm, M. Gao, P. Roetzer, M. Eisenberger, D. Cremers, and F. Bernard (2024) Partial-to-partial shape matching with geometric consistency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 27488–27497. Cited by: §3.3. ↩ a
R. Girdhar, A. El-Nouby, Z. Liu, M. Singh, K. V. Alwala, A. Joulin, and I. Misra (2023) Imagebind: one embedding space to bind them all. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15180–15190. Cited by: §2, 2nd item. ↩ a b
W. Guo, N. Ho, and M. Jordan (2020) Fast algorithms for computational optimal transport and wasserstein barycenter. In International Conference on Artificial Intelligence and Statistics, pp. 2088–2097. Cited by: §3.4. ↩ a
A. Gupta and R. Gupta (2019) ISBI 2019 c-nmc challenge: classification in cancer cell imaging. Select Proceedings 2. Cited by: Table 11. ↩ a
S. Haller, L. Feineis, L. Hutschenreiter, F. Bernard, C. Rother, D. Kainmüller, P. Swoboda, and B. Savchynskyy (2022) A comparative study of graph matching algorithms in computer vision. In European Conference on Computer Vision, pp. 636–653. Cited by: §3.3. ↩ a
S. He, Y. Nie, Z. Chen, Z. Cai, H. Wang, S. Yang, and H. Chen (2024) MedDr: diagnosis-guided bootstrapping for large-scale medical vision-language learning. arXiv preprint arXiv:2404.15127. Cited by: §1, §2, §4.4. ↩ a b c
X. He, Y. Zhang, L. Mou, E. Xing, and P. Xie (2020) Pathvqa: 30000+ questions for medical visual question answering. arXiv preprint arXiv:2003.10286. Cited by: §1, §4.3. ↩ a b
T. Heimann and H. Meinzer (2009) Statistical shape models for 3d medical image segmentation: a review. Medical image analysis 13 (4), pp. 543–563. Cited by: §D.2. ↩ a
E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, et al. (2022) Lora: low-rank adaptation of large language models.. ICLR 1 (2), pp. 3. Cited by: 8th item. ↩ a
Y. Hu, T. Li, Q. Lu, W. Shao, J. He, Y. Qiao, and P. Luo (2024) Omnimedvqa: a new large-scale comprehensive evaluation benchmark for medical lvlm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22170–22183. Cited by: Appendix F, Appendix F, Appendix F. ↩ a b c
W. Huang, C. Wang, R. Zhang, Y. Li, J. Wu, and L. Fei-Fei (2023) Voxposer: composable 3d value maps for robotic manipulation with language models. arXiv preprint arXiv:2307.05973. Cited by: §2. ↩ a
J. Hyun, M. Kang, D. Wee, and D. Yeung (2023) Detection recovery in online multi-object tracking with sparse graph tracker. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 4850–4859. Cited by: §D.2. ↩ a
S. Jaeger, S. Candemir, S. Antani, Y. J. Wáng, P. Lu, and G. Thoma (2014) Two public chest x-ray datasets for computer-aided screening of pulmonary diseases. Quantitative imaging in medicine and surgery 4 (6), pp. 475. Cited by: Table 12. ↩ a b
S. Javadi and S. A. Mirroshandel (2019) A novel deep learning method for automatic assessment of human sperm images. Computers in biology and medicine 109, pp. 182–194. Cited by: Table 11. ↩ a
S. Jiang, T. Zheng, Y. Zhang, Y. Jin, and Z. Liu (2024) MoE-tinymed: mixture of experts for tiny medical large vision-language models. arXiv preprint arXiv:2404.10237. Cited by: §1, §4.4. ↩ a b
H. K. Jin, H. E. Lee, and E. Kim (2024) Performance of chatgpt-3.5 and gpt-4 in national licensing examinations for medicine, pharmacy, dentistry, and nursing: a systematic review and meta-analysis. BMC Medical Education 24 (1), pp. 1013. Cited by: Appendix J. ↩ a
A. E. W. Johnson, T. J. Pollard, S. J. Berkowitz, N. R. Greenbaum, M. P. Lungren, C. Y. Deng, R. G. Mark, and S. Horng (2019) MIMIC-cxr, a de-identified publicly available database of chest radiographs with free-text reports. Scientific Data 6 (1), pp. 317. External Links: Document Cited by: §4.6. ↩ a
W. Ju, Z. Fang, Y. Gu, Z. Liu, Q. Long, Z. Qiao, Y. Qin, J. Shen, F. Sun, Z. Xiao, et al. (2024) A comprehensive survey on deep graph representation learning. Neural Networks, pp. 106207. Cited by: §3.2. ↩ a
J. N. Kather, N. Halama, and A. Marx (2018) 100,000 histological images of human colorectal cancer and healthy tissue. Zenodo. Note: Accessed: 2024-09-30 External Links: Document Cited by: Table 11. ↩ a
D. S. Kermany, M. Goldbaum, W. Cai, C. C. Valentim, H. Liang, S. L. Baxter, A. McKeown, G. Yang, X. Wu, F. Yan, et al. (2018) Identifying medical diagnoses and treatable diseases by image-based deep learning. cell 172 (5), pp. 1122–1131. Cited by: Table 13. ↩ a
Z. Khan and Y. Fu (2023) Contrastive alignment of vision to language through parameter-efficient transfer learning. International Conference on Learning Representations. Cited by: 2nd item, §2, 1st item. ↩ a b c
S. Khanal, S. Sastry, A. Dhakal, and N. Jacobs (2023) Learning tri-modal embeddings for zero-shot soundscape mapping. The British Machine Vision Conference (BMVC). Cited by: §2, 2nd item. ↩ a b
D. P. Kingma (2014) Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980. Cited by: §4.1. ↩ a
J. J. Lau, S. Gayen, A. Ben Abacha, and D. Demner-Fushman (2018) A dataset of clinically generated visual questions and answers about radiology images. Scientific data 5 (1), pp. 1–10. Cited by: §1, §4.3. ↩ a b
K. Le-Duc, P. T. H. Trinh, D. M. H. Nguyen, T. Nguyen, N. T. Diep, A. Ngo, T. Vu, T. Vuong, A. Nguyen, M. Nguyen, V. T. Hoang, K. Nguyen, H. Nguyen, C. Ngo, A. Liu, N. Ho, A. Hauschild, K. X. Nguyen, T. Nguyen-Tang, P. Xie, D. Sonntag, J. Zou, M. Niepert, and A. T. Nguyen (2025) S-chain: structured visual chain-of-thought for medicine. arXiv preprint. Cited by: §4.6. ↩ a
C. Li, C. Wong, S. Zhang, N. Usuyama, H. Liu, J. Yang, T. Naumann, H. Poon, and J. Gao (2024) Llava-med: training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems 36. Cited by: §1, §2, §4.1, §4.1, §4.1, §4.4. ↩ a b c d e f
J. Li, D. Li, S. Savarese, and S. Hoi (2023a) Blip-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730–19742. Cited by: §1, §2. ↩ a b c
J. Li, D. Li, C. Xiong, and S. Hoi (2022) Blip: bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pp. 12888–12900. Cited by: §2. ↩ a
P. Li, G. Liu, L. Tan, J. Liao, and S. Zhong (2023b) Self-supervised vision-language pretraining for medial visual question answering. In 2023 IEEE 20th International Symposium on Biomedical Imaging (ISBI), Vol. , pp. 1–5. External Links: Document Cited by: §4.4. ↩ a
T. Lin, N. Ho, M. Cuturi, and M. I. Jordan (2022) On the complexity of approximating multi-marginal optimal transport. Journal of Machine Learning Research 23, pp. 1–43. Cited by: §2. ↩ a
B. Liu, L. Zhan, L. Xu, L. Ma, Y. Yang, and X. Wu (2021) Slake: a semantically-labeled knowledge-enhanced dataset for medical visual question answering. In 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), Vol. , pp. 1650–1654. External Links: Document Cited by: §4.3. ↩ a
H. Liu, C. Li, Y. Li, and Y. J. Lee (2024a) Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26296–26306. Cited by: §4.1. ↩ a
H. Liu, C. Li, Q. Wu, and Y. J. Lee (2024b) Visual instruction tuning. Advances in neural information processing systems 36. Cited by: §1, §2, §2, §4.4. ↩ a b c d
L. Liu, X. Sun, T. Xiang, Z. Zhuang, L. Yin, and M. Tan (2023) Contrastive vision-language alignment makes efficient instruction learner. arXiv preprint arXiv:2311.17945. Cited by: 2nd item, §2, 1st item. ↩ a b c
N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang (2024c) Lost in the middle: how language models use long contexts. Transactions of the Association for Computational Linguistics 12, pp. 157–173. Cited by: §3.1. ↩ a
R. Lou, K. Zhang, and W. Yin (2023) A comprehensive survey on instruction following. arXiv preprint arXiv:2303.10475. Cited by: §1. ↩ a
X. Ma, X. Chu, Y. Wang, Y. Lin, J. Zhao, L. Ma, and W. Zhu (2024) Fused gromov-wasserstein graph mixup for graph-level classifications. Advances in Neural Information Processing Systems 36. Cited by: §D.1. ↩ a
F. Mémoli and G. Sapiro (2005) A Theoretical and Computational Framework for Isometry Invariant Recognition of Point Cloud Data. Foundations of Computational Mathematics 5 (3), pp. 313–347. External Links: ISSN 1615-3383 Cited by: §B.2. ↩ a
F. Mémoli (2011) Gromov–Wasserstein Distances and the Metric Approach to Object Matching. Foundations of Computational Mathematics 11 (4), pp. 417–487 (en). External Links: ISSN 1615-3375, 1615-3383 Cited by: §B.2. ↩ a
D. MH Nguyen, H. Nguyen, N. Diep, T. N. Pham, T. Cao, B. Nguyen, P. Swoboda, N. Ho, S. Albarqouni, P. Xie, et al. (2024) Lvm-med: learning large-scale self-supervised vision models for medical imaging via second-order graph matching. Advances in Neural Information Processing Systems 36. Cited by: §D.2, §2, §4.6. ↩ a b c
P. Minervini, L. Franceschi, and M. Niepert (2023) Adaptive perturbation-based gradient estimation for discrete latent variable models. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 9200–9208. Cited by: §1, §2, Figure 3, §3.5. ↩ a b c d
M. Moor, Q. Huang, S. Wu, M. Yasunaga, Y. Dalmia, J. Leskovec, C. Zakka, E. P. Reis, and P. Rajpurkar (2023) Med-flamingo: a multimodal medical few-shot learner. In Machine Learning for Health (ML4H), pp. 353–367. Cited by: §2, §4.4. ↩ a b
L. Nanni, M. Paci, F. L. Caetano dos Santos, H. Skottman, K. Juuti-Uusitalo, and J. Hyttinen (2016) Texture descriptors ensembles enable image-based classification of maturation of human stem cell-derived retinal pigmented epithelium. PLoS One 11 (2), pp. e0149399. Cited by: Table 11. ↩ a
D. M. Nguyen, A. T. Le, T. Q. Nguyen, N. T. Diep, T. Nguyen, D. Duong-Tran, J. Peters, L. Shen, M. Niepert, and D. Sonntag (2024a) Dude: dual distribution-aware context prompt learning for large vision-language model. Asian Conference on Machine Learning. Cited by: §2. ↩ a
D. M. Nguyen, N. Lukashina, T. Nguyen, A. T. Le, T. Nguyen, N. Ho, J. Peters, D. Sonntag, V. Zaverkin, and M. Niepert (2024b) Structure-aware e (3)-invariant molecular conformer aggregation networks. International Conference on Machine Learning. Cited by: 1st item, §D.1, §2. ↩ a b c
M. Niepert, P. Minervini, and L. Franceschi (2021) Implicit mle: backpropagating through discrete exponential family distributions. Advances in Neural Information Processing Systems 34, pp. 14567–14579. Cited by: 2nd item, §1, §2, Figure 3, §3.5. ↩ a b c d e
[69] (2023) NLM - malaria data. Note: https://lhncbc.nlm.nih.gov/LHC-research/LHC-projects/image-processing/malaria-datasheet.htmlAccessed: 2024-09-30 Cited by: Table 11. ↩ a
A. v. d. Oord, Y. Li, and O. Vinyals (2018) Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748. Cited by: §2. ↩ a
J. Park, J. Lee, and K. Sohn (2024) Bridging vision and language spaces with assignment prediction. International Conference on Learning Representations. Cited by: §1, §2, §2, 1st item. ↩ a b c d
S. Pawar, S. Tonmoy, S. Zaman, V. Jain, A. Chadha, and A. Das (2024) The what, why, and how of context length extension techniques in large language models–a detailed survey. arXiv preprint arXiv:2401.07872. Cited by: §3.1. ↩ a
P. A. Pevzner (1992) Multiple alignment, communication cost, and graph matching. SIAM Journal on Applied Mathematics 52 (6), pp. 1763–1779. Cited by: §1. ↩ a
Z. Piran, M. Klein, J. Thornton, and M. Cuturi (2024) Contrasting multiple representations with the multi-marginal matching gap. International Conference on Machine Learning. Cited by: §2. ↩ a
M. V. Pogančić, A. Paulus, V. Musil, G. Martius, and M. Rolinek (2020) Differentiation of blackbox combinatorial solvers. In International Conference on Learning Representations, Cited by: §3.5. ↩ a
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. (2021) Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748–8763. Cited by: §2, §4.1. ↩ a b
P. Rajpurkar, J. Irvin, A. Bagul, D. Ding, T. Duan, H. Mehta, B. Yang, K. Zhu, D. Laird, R. L. Ball, et al. (2017) Mura: large dataset for abnormality detection in musculoskeletal radiographs. arXiv preprint arXiv:1712.06957. Cited by: Table 12. ↩ a
M. Rolínek, V. Musil, A. Paulus, M. Vlastelica, C. Michaelis, and G. Martius (2020a) Optimizing rank-based metrics with blackbox differentiation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7620–7630. Cited by: §3.5. ↩ a
M. Rolínek, P. Swoboda, D. Zietlow, A. Paulus, V. Musil, and G. Martius (2020b) Deep graph matching via blackbox differentiation of combinatorial solvers. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXVIII, pp. 407–424. Cited by: 1st item, §1, §2, §3.4. ↩ a b c d
S. Sarto, M. Barraco, M. Cornia, L. Baraldi, and R. Cucchiara (2023) Positive-augmented contrastive learning for image and video captioning evaluation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6914–6924. Cited by: §2, 2nd item. ↩ a b
F. Shaker, S. A. Monadjemi, J. Alirezaie, and A. R. Naghsh-Nilchi (2017) A dictionary learning approach for human sperm heads classification. Computers in biology and medicine 91, pp. 181–190. Cited by: Table 11. ↩ a
K. Singhal, T. Tu, J. Gottweis, R. Sayres, E. Wulczyn, L. Hou, K. Clark, S. Pfohl, H. Cole-Lewis, D. Neal, et al. (2023) Towards expert-level medical question answering with large language models. arXiv preprint arXiv:2305.09617. Cited by: §2. ↩ a
E. Soares and P. Angelov (2020) A large dataset of real patients ct scans for covid-19 identification. Harv. Dataverse 1, pp. 1–8. Cited by: Table 13. ↩ a
K. Song, X. Tan, T. Qin, J. Lu, and T. Liu (2020) Mpnet: masked and permuted pre-training for language understanding. Advances in neural information processing systems 33, pp. 16857–16867. Cited by: §2. ↩ a
F. A. Spanhol, L. S. Oliveira, C. Petitjean, and L. Heutte (2015) A dataset for breast cancer histopathological image classification. Ieee transactions on biomedical engineering 63 (7), pp. 1455–1462. Cited by: Table 11. ↩ a
J. Suckling (1994) The mammographic images analysis society digital mammogram database. In Exerpta Medica. International Congress Series, 1994, Vol. 1069, pp. 375–378. Cited by: Table 12. ↩ a
Y. Sun, C. Liu, K. Zhou, J. Huang, R. Song, W. X. Zhao, F. Zhang, D. Zhang, and K. Gai (2024) Parrot: enhancing multi-turn instruction following for large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9729–9750. Cited by: §1. ↩ a
P. Swoboda, A. Mokarian, C. Theobalt, F. Bernard, et al. (2019) A convex relaxation for multi-graph matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11156–11165. Cited by: §2, §3.4. ↩ a b
P. Swoboda, C. Rother, H. Abu Alhaija, D. Kainmuller, and B. Savchynskyy (2017) A study of lagrangean decompositions and dual ascent solvers for graph matching. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1607–1616. Cited by: 1st item, §2, §3.4. ↩ a b c
S. Tang, F. Zhu, L. Bai, R. Zhao, C. Wang, and W. Ouyang (2022) Unifying visual contrastive learning for object recognition from a graph perspective. In European Conference on Computer Vision, pp. 649–667. Cited by: §3.2. ↩ a
G. Team, R. Anil, S. Borgeaud, J. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican, et al. (2023) Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805. Cited by: 1st item. ↩ a
H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. (2023) Llama: open and efficient foundation language models. arXiv preprint arXiv:2302.13971. Cited by: §1, §4.1. ↩ a b
T. Tu, S. Azizi, D. Driess, M. Schaekermann, M. Amin, P. Chang, A. Carroll, C. Lau, R. Tanno, I. Ktena, et al. (2024) Towards generalist biomedical ai. NEJM AI 1 (3), pp. AIoa2300138. Cited by: §2. ↩ a
L. Wang, Z. Q. Lin, and A. Wong (2020) Covid-net: a tailored deep convolutional neural network design for detection of covid-19 cases from chest x-ray images. Scientific reports 10 (1), pp. 19549. Cited by: Table 12. ↩ a
C. Wu, X. Zhang, Y. Zhang, Y. Wang, and W. Xie (2023a) Towards generalist foundation model for radiology. arXiv preprint arXiv:2308.02463. Cited by: Appendix F, §1. ↩ a b
S. Wu, H. Fei, L. Qu, W. Ji, and T. Chua (2023b) Next-gpt: any-to-any multimodal llm. arXiv preprint arXiv:2309.05519. Cited by: §1. ↩ a
[97] (2023) X-ray hand small joint classification dataset (based on bone age scoring method rus-chn). Note: https://aistudio.baidu.com/datasetdetail/69582/0Accessed: 2024-09-30 Cited by: Table 12. ↩ a
Y. Xie, C. Zhou, L. Gao, J. Wu, X. Li, H. Zhou, S. Liu, L. Xing, J. Zou, C. Xie, et al. (2024) MedTrinity-25m: a large-scale multimodal dataset with multigranular annotations for medicine. arXiv preprint arXiv:2408.02900. Cited by: §1, §1, §4.6. ↩ a b c
A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. (2025) Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: 1st item. ↩ a
H. Yao, W. Wu, T. Yang, Y. Song, M. Zhang, H. Feng, Y. Sun, Z. Li, W. Ouyang, and J. Wang (2024) Dense connector for mllms. arXiv preprint arXiv:2405.13800. Cited by: §4.4. ↩ a
A. Zanfir and C. Sminchisescu (2018) Deep learning of graph matching. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2684–2693. Cited by: §3.3. ↩ a
X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer (2023) Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11975–11986. Cited by: §2, §2, 1st item. ↩ a b c
D. Zhang, Y. Yu, J. Dong, C. Li, D. Su, C. Chu, and D. Yu (2024a) Mm-llms: recent advances in multimodal large language models. arXiv preprint arXiv:2401.13601. Cited by: §4.6. ↩ a
K. Zhang, J. Yu, Z. Yan, Y. Liu, E. Adhikarla, S. Fu, X. Chen, C. Chen, Y. Zhou, X. Li, et al. (2023a) BiomedGPT: a unified and generalist biomedical generative pre-trained transformer for vision, language, and multimodal tasks. arXiv preprint arXiv:2305.17100. Cited by: §1, §2, §4.4. ↩ a b c
R. Zhang, J. Han, C. Liu, A. Zhou, P. Lu, Y. Qiao, H. Li, and P. Gao (2024b) LLaMA-adapter: efficient fine-tuning of large language models with zero-initialized attention. In The Twelfth International Conference on Learning Representations, Cited by: §2, §2. ↩ a b
S. Zhang, Y. Xu, N. Usuyama, J. Bagga, R. Tinn, S. Preston, R. Rao, M. Wei, N. Valluri, C. Wong, et al. (2023b) Large-scale domain-specific pretraining for biomedical vision-language processing. arXiv preprint arXiv:2303.00915 2 (3), pp. 6. Cited by: §1. ↩ a
T. Zhao, Y. Gu, J. Yang, N. Usuyama, H. H. Lee, S. Kiblawi, T. Naumann, J. Gao, A. Crabtree, J. Abel, et al. (2024) A foundation model for joint segmentation, detection and recognition of biomedical objects across nine modalities. Nature methods, pp. 1–11. Cited by: §4.6. ↩ a

Supplementary Material for
“ExGra-Med: Extended Context Graph Alignment for Medical Vision-Language Models”

Appendix A Proofs of the Main Theoretical Results

In this section, we provide detailed technical proofs of our main theoretical results. To accomplish this, we first introduce the fundamental concepts and materials that will be utilized in the proofs of Theorems 1 and 1.

We consider labelled graphs as tuples of the form $𝒢 = (𝒱, ℰ, ℒ_{f}, ℒ_{s})$ , where the labelling function $ℒ_{f} : 𝒱 \mapsto ℱ$ assigns each vertex $v_{i} \in 𝒱$ to a feature $f_{i} = ℒ_{f} (v_{i})$ in some feature space $(ℱ, d_{f})$ . Similarly, we denote $ℒ_{s} : 𝒱 \mapsto 𝒮$ as a structure function which links each vertex $v_{i} \in 𝒱$ with its structure information $s_{i} = ℒ_{s} (v_{i})$ , e.g., edge information, in some structure space $(𝒮, d_{s})$ . By associating a weight to each vertex, we allow the graph $𝒢$ to be represented by a fully supported mixing measure $μ = \sum_{i = 1}^{N} w_{i} δ_{(f_{i}, s_{i})}$ over the product between feature space and structure space $ℱ \times 𝒮$ . Notably, $μ$ is not necessarily a probability measure as the summation of its weights can be different from one. We have the vertex affinity matrix between two graphs as $𝑨^{v} \in ℝ^{M \times N}$ , where $A^{v}_{i, j} = (d_{f} (f_{i}, f_{j})) i, j$ . Structural similarity is measured by pairwise distances within each graph, represented by $𝑨^{e} \in ℝ^{| ℰ_{1} | | ℰ_{2} |}$ , with $A^{e}_{i, j, k, l} = | d_{s} (s_{i}, s_{k}) - d_{s} (s_{j}, s_{l}) |$ , where $d_{s} (\cdot)$ models node distance, such as the shortest path. We then define the space of all structured graphs $(ℱ \times 𝒮, d_{f}, μ)$ over a metric feature space $(ℱ, d_{f})$ as $𝕊 (ℱ)$ , where $(𝒮, d_{s})$ is a metric structure space and $μ = \sum_{i = 1}^{N} w_{i} δ_{(f_{i}, s_{i})}$ is a mixing measure over $ℱ \times 𝒮$ .

A.1 Proof of Theorem 1

For the sake of simplicity, we denote the labeled graphs $𝒢$ and structured graphs discussed above only by $μ$ the whole structured graph.

To prove Theorem 1, for any two graphs $𝒢_{1}$ and $𝒢_{2}$ in the structured graph space $𝕊 (ℱ)$ , described respectively by their mixing measure $μ_{1} = \sum_{i = 1}^{M} w_{1 i} δ_{(f_{1 i}, s_{1 i})}$ and $μ_{2} = \sum_{j = 1}^{N} w_{2 j} δ_{(f_{2 j}, s_{2 j})}$ , respectively, we wish to prove the following properties:

1.

Positivity: $d_{SGA} (𝒢_{1}, 𝒢_{2}) > 0$ for any $𝒢_{1} \neq 𝒢_{2}$ .
2.

Equality relation: $d_{SGA} (𝒢_{1}, 𝒢_{2}) = 0$ if and only if $𝒢_{1} = 𝒢_{2}$ .
3.

Symmetry: $d_{SGA} (𝒢_{1}, 𝒢_{2}) = d_{SGA} (𝒢_{2}, 𝒢_{1})$ .
4.

Triangle inequality: $d_{SGA} (𝒢_{1}, 𝒢_{3}) \leq d_{SGA} (𝒢_{1}, 𝒢_{2}) + d_{SGA} (𝒢_{2}, 𝒢_{3})$ for any graph $𝒢_{3}$ .

Note first that 1. Positivity and 3. Symmetry hold trivially.

Proof of 2. Equality relation. The equality relation immediately follows the following Proposition 1, which is proved in Appendix B.1.

Proposition 1 (Equality relation).

For any two graphs $𝒢_{1}$ and $𝒢_{2}$ in the structured graph space $𝕊 (ℱ)$ , described respectively by their mixing measure $μ_{1} = \sum_{i = 1}^{M} w_{1 i} δ_{(f_{1 i}, s_{1 i})}$ and $μ_{2} = \sum_{j = 1}^{N} w_{2 j} δ_{(f_{2 j}, s_{2 j})}$ , it holds $d_{SGA} (𝒢_{1}, 𝒢_{2}) = 0$ if and only if $M = N$ and there exists a bijection $σ : [M] \mapsto [M]$ such that:

E1.

$\forall i \in [M] : w_{1 i} = w_{2 σ (i)}$ .
E2.

$\forall i \in [M] : f_{1 i} = f_{2 σ (i)}$ .
E3.

$\forall i, k \in [M]^{2} : d_{s} (s_{1 i}, s_{1 k}) = d_{s} (s_{2 σ (i)}, s_{2 σ (k)})$ .

Proof of 4. Triangle inequality. Let us consider two arbitrary graphs $𝒢_{1}$ and $𝒢_{2}$ , described respectively by their probability measure $μ_{1} = \sum_{i = 1}^{M} w_{1 i} δ_{(f_{1 i}, s_{1 i})}$ and $μ_{2} = \sum_{j = 1}^{N} w_{2 j} δ_{(f_{2 j}, s_{2 j})}$ . For any graph $𝒢_{3}$ described by its probability measure $μ_{3} = \sum_{i = 1}^{K} w_{3 k} δ_{(f_{3 k}, s_{3 k})}$ , we define $𝑷 \in 𝒜 (𝒢_{1}, 𝒢_{2})$ and $𝑸 \in 𝒜 (𝒢_{2}, 𝒢_{3})$ as two optimal couplings of the SGA distance between $μ_{1}$ and $μ_{2}$ and $μ_{2}$ and $μ_{3}$ , respectively, i.e.,

	$𝑷 \in 𝒜 (𝒢_{1}, 𝒢_{2}) = {𝑷 \in {0, 1}^{M \times N} : \sum_{i = 1}^{M} P_{i, j} = w_{1 j} = 1, \sum_{j = 1}^{N} P_{i, j} = w_{2 i} = 1},$
	$𝑸 \in 𝒜 (𝒢_{2}, 𝒢_{3}) = {𝑸 \in {0, 1}^{N \times K} : \sum_{j = 1}^{N} Q_{j, k} = w_{2 k} = 1, \sum_{k = 1}^{K} Q_{j, k} = w_{3 j} = 1} .$

We then construct $𝑹 = (\sum_{j} \frac{P_{i, j} Q_{j, k}}{w_{2 j}})_{i, k}$ . Then it holds that $𝑹 \in 𝒜 (𝒢_{1}, 𝒢_{3})$ . Indeed, we have

\sum_{i} R_{i, k} = \sum_{i} \sum_{j} \frac{P_{i, j} Q_{j, k}}{w_{2 j}} = \sum_{j} \sum_{i} P_{i, j} \frac{Q_{j, k}}{w_{2 j}} = \sum_{j} w_{1 j} \frac{Q_{j, k}}{w_{2 j}} = \sum_{j} Q_{j, k} = 1 .

By the suboptimality of $𝑹$ , the triangle inequalities of $d_{f}$ and $| \cdot |$ , we have

	$d_{SGA} (𝒢_{1}, 𝒢_{3})$	$\leq \sum_{i, j, k, l} [d_{f} (f_{1 i}, f_{3 j}) + \| d_{s} (s_{1 i}, s_{1 k}) - d_{s} (s_{3 j}, s_{3 l}) \|] R_{i, j} R_{k, l}$
		$= \sum_{i, j, k, l} [d_{f} (f_{1 i}, f_{3 j}) + \| d_{s} (s_{1 i}, s_{1 k}) - d_{s} (s_{3 j}, s_{3 l}) \|] \sum_{t} \frac{P_{i, t} Q_{t, j}}{w_{2 t}} \sum_{d} \frac{P_{k, d} Q_{d, l}}{w_{2 d}}$
		$= \sum_{i, j, k, l, t, d} [d_{f} (f_{1 i}, f_{3 j}) + \| d_{s} (s_{1 i}, s_{1 k}) - d_{s} (s_{3 j}, s_{3 l}) \|] \frac{P_{i, t} Q_{t, j}}{w_{2 t}} \frac{P_{k, d} Q_{d, l}}{w_{2 d}}$
		$\leq \sum_{i, j, k, l, t, d} [d_{f} (f_{1 i}, f_{2 t}) + d_{f} (f_{2 t}, f_{3 j})] \frac{P_{i, t} Q_{t, j}}{w_{2 t}} \frac{P_{k, d} Q_{d, l}}{w_{2 d}}$
		$+ \sum_{i, j, k, l, t, d} [\| d_{s} (s_{1 i}, s_{1 k}) - d_{s} (s_{2 t}, s_{2 d}) \| + \| d_{s} (s_{2 t}, s_{2 d}) - d_{s} (s_{3 j}, s_{3 l}) \|] \frac{P_{i, t} Q_{t, j}}{w_{2 t}} \frac{P_{k, d} Q_{d, l}}{w_{2 d}}$
		$= \sum_{i, j, k, l, t, d} [d_{f} (f_{1 i}, f_{2 t}) + \| d_{s} (s_{1 i}, s_{1 k}) - d_{s} (s_{2 t}, s_{2 d}) \|] \frac{P_{i, t} P_{k, d}}{w_{2 t}} \frac{Q_{t, j} Q_{d, l}}{w_{2 d}}$
		$+ \sum_{i, j, k, l, t, d} [d_{f} (f_{2 t}, f_{3 j}) + \| d_{s} (s_{2 t}, s_{2 d}) - d_{s} (s_{3 j}, s_{3 l}) \|] \frac{P_{i, t} Q_{t, j}}{w_{2 t}} \frac{P_{k, d} Q_{d, l}}{w_{2 d}}$
		$= \sum_{i, k, t, d} [d_{f} (f_{1 i}, f_{2 t}) + \| d_{s} (s_{1 i}, s_{1 k}) - d_{s} (s_{2 t}, s_{2 d}) \|] P_{i, t} P_{k, d} \sum_{j} \frac{Q_{t, j}}{w_{2 t}} \sum_{l} \frac{Q_{d, l}}{w_{2 d}}$
		$+ \sum_{j, l, t, d} [d_{f} (f_{2 t}, f_{3 j}) + \| d_{s} (s_{2 t}, s_{2 d}) - d_{s} (s_{3 j}, s_{3 l}) \|] Q_{t, j} Q_{d, l} \sum_{i} \frac{P_{i, t}}{w_{2 t}} \sum_{k} \frac{P_{k, d}}{w_{2 d}} .$

Note that we have

\sum_{j} \frac{Q_{t, j}}{w_{2 t}} = \sum_{l} \frac{Q_{d, l}}{w_{2 d}} = \sum_{i} \frac{P_{i, t}}{w_{2 t}} = \sum_{k} \frac{P_{k, d}}{w_{2 d}} = 1 .

This is how we achieve the desired result, because

	$d_{SGA} (𝒢_{1}, 𝒢_{3})$	$\leq \sum_{i, k, t, d} [d_{f} (f_{1 i}, f_{2 t}) + \| d_{s} (s_{1 i}, s_{1 k}) - d_{s} (s_{2 t}, s_{2 d}) \|] P_{i, t} P_{k, d}$
		$+ \sum_{j, l, t, d} [d_{f} (f_{2 t}, f_{3 j}) + \| d_{s} (s_{2 t}, s_{2 d}) - d_{s} (s_{3 j}, s_{3 l}) \|] Q_{t, j} Q_{d, l}$
		$= d_{SGA} (𝒢_{1}, 𝒢_{2}) + d_{SGA} (𝒢_{2}, 𝒢_{3}) (since 𝑷 and 𝑸 are the optimal plans) .$

A.2 Proof of Theorem 1

Theorem 1 enables us to characterise the optimal transport problem between two measures as a curve in the space of measures, with the objective of minimising its total length. Furthermore, this formulation is beneficial for deriving global minima results for non-convex particles in gradient descent in an optimisation context, which is a valuable application of gradient flows [16]. By definition, a geodesic between $𝒢_{1}$ and $𝒢_{2}$ is a shortest path between these two graphs. In particular, the computation of distances along constant speed geodesic paths is a relatively straightforward process, as these paths are directly embedded into the real line $ℝ$ as follows: $d_{SGA} (𝒢_{1}, 𝒢_{2}) = | t - u |^{- 1} d_{SGA} (p (u), p (t))$ , for all $0 \leq u \neq t \leq 1$ and for any path (continuous map) $p$ connect $𝒢_{1}$ to $𝒢_{2}$ such that $p (u) = 𝒢_{1}$ and $p (t) = 𝒢_{2}$ . To prove Theorem A.2, it is necessary to collect fundamental material using Definition 2 from metric geometry for a general metric space $(𝕄, d)$ .

Definition 2 (Length and geodesic spaces).

Let $(𝕄, d)$ be a metric space and two points $x, y \in 𝕄$ . We say that a path (curve) $p : [0, 1] \mapsto 𝕄$ connect or join $x$ to $y$ if $p (0) = x$ and $p (1) = y$ and $p$ is a continuous map.

We also define the length $L (p) \in ℝ$ of a path $p : [0, 1] \mapsto 𝕄$ as

L (p) : = sup \sum_{i = 1}^{n} d (p (t_{i}), p (t_{i + 1}))

where we take the supremum over all $n \geq 1$ and all $n$ -tuples $t_{1} < \dots < t_{n}$ in $[0, 1]$ .

We denote a metric space $𝕄$ as a length space if for all $x, y \in 𝕄$ , $d (x, y) = inf_{p} L (p)$ where the infimum is taken over all paths $p$ connecting $x$ to $y$ .

We call a length space as a geodesic space if for all $x, y \in 𝕄$ , there exists a path $p (x, y) : [0, 1] \mapsto 𝕄$ such that

d (x, y) = \min_{p (x, y)} L (p (x, y)) .

We also denote the path $p (x, y)$ as a geodesic between $x$ and $y$ .

Finally, we define a path $p : [0, 1] \mapsto 𝕄$ as a constant speed geodesic if and only if

d (p (u), p (t)) = | t - u | d (p (0), p (1)), \forall u, t \in [0, 1] .

For the proof of Theorem 1, we first consider an optimal coupling $𝑽^{*}$ for SGA distance between two graphs $𝒢_{1}$ and $𝒢_{2}$ , i.e.,

d_{SGA} (𝒢_{1}, 𝒢_{2}) = \min_{𝑽 \in 𝒜 (𝒢_{1}, 𝒢_{2})} O (𝑨^{v}, 𝑨^{e}, 𝑽) = O (𝑨^{v}, 𝑨^{e}, 𝑽^{*}),

described respectively by their mixing measure $μ_{0} = \sum_{i = 1}^{M} w_{0 i} δ_{(f_{0 i}, s_{0 i})}$ and $μ_{1} = \sum_{j = 1}^{N} w_{1 j} δ_{(f_{1 j}, s_{1 j})}$ . Moreover, for any $t \in [0, 1]$ we define $ν_{t} : ℱ \times 𝒮_{0} \times ℱ \times 𝒮_{1} \mapsto ℱ \times 𝒮_{0} \times 𝒮_{1}$ such that

ν_{t} (f_{0}, s_{0}, f_{1}, s_{1}) = ((1 - t) f_{0} + t f_{1}, s_{0}, s_{1}), and μ_{t} : = ν_{t} # 𝑽^{*} = \sum_{i = 1}^{M} \sum_{j = 1}^{N} V^{*}_{i, j} δ_{((1 - t) f_{0} + t f_{1}, s_{0 i}, s_{1 j})},

and on the metric space $𝒮_{0} \times 𝒮_{1}$ , we define the distance

d_{t} : = (1 - t) d_{s_{0}} \oplus t d_{s_{1}} : (1 - t) d_{s_{0}} \oplus t d_{s_{1}} ((s_{0 i}, s_{0 j}), (s_{1 k}, s_{1 l})) = (1 - t) d_{s} (s_{0 i}, s_{1 k}) + t d_{s} (s_{0 j}, s_{1 l})

for any $((s_{0 i}, s_{0 j}), (s_{1 k}, s_{1 l})) \in 𝒮_{0} \times 𝒮_{1}$ . Here, we denote $#$ the push-forward operator such that $ν_{t} # 𝑽^{*} (𝔸) = 𝑽^{*} (ν_{t}^{- 1} (𝔸))$ for any Borel sets of a $σ$ -albegra. For simplicity, we only consider $(ℱ, d_{f}) = (ℝ^{d}, ∥ \cdot ∥)$ where $∥ \cdot ∥$ is the Euclidean norm.

Then we aim to prove that $(ℱ \times 𝒮_{0} \times 𝒮_{1}, (1 - t) d_{s_{0}} \oplus t d_{s_{1}}, μ_{t})_{t \in [0, 1]}$ is a constant speed geodesic joining $(ℱ \times 𝒮_{0}, d_{s 0}, μ_{0}))$ and $(ℱ \times 𝒮_{1}, d_{s 1}, μ_{1}))$ , for arbitrary elements $(ℱ \times 𝒮_{0}, d_{s 0}, μ_{0}))$ and $(ℱ \times 𝒮_{1}, d_{s 1}, μ_{1}))$ in the metric space $(𝕊 (ℱ), d_{SGA})$ .

To do so, we consider any $u, t \in [0, 1]$ such that $u \neq t$ . By definition, we have to prove that

d_{SGA} (μ_{u}, μ_{t}) = | t - u | d_{SGA} (μ_{0}, μ_{1}) .

(7)

Indeed, to prove equation (7), we first recall that

	$μ_{u}$	$: = ν_{u} # 𝑽^{} = \sum_{i = 1}^{M} \sum_{j = 1}^{N} V^{}_{i, j} δ_{((1 - u) f_{0} + u f_{1}, s_{0 i}, s_{1 j})},$
	$μ_{t}$	$: = ν_{t} # 𝑽^{} = \sum_{i = 1}^{M} \sum_{j = 1}^{N} V^{}_{i, j} δ_{((1 - t) f_{0} + t f_{1}, s_{0 i}, s_{1 j})},$
	$d_{SGA} (μ_{0}, μ_{1})$	$= \sum_{i, j, k, l} [d_{f} (f_{0 i}, f_{1 j}) + \| d_{s} (s_{0 i}, s_{1 k}) - d_{s} (s_{0 j}, s_{1 l}) \|] V^{}_{i, j} V^{}_{k, l} .$

We then define the coupling $γ^{u, t} = (μ_{u} \times μ_{t}) # 𝑽^{*} \in 𝒜 (μ_{u}, μ_{t})$ . By the suboptimality of $γ^{u, t}$ , it holds that:

	$d_{SGA} (μ_{u}, μ_{t})$	$\leq \sum_{i, j, k, l} [d_{f} (f_{0 i}, f_{1 j}) + \| d_{t} ((s_{0 i}, s_{0 j}), (s_{1 k}, s_{1 l})) - d_{u} ((s_{0 i}, s_{0 j}), (s_{1 k}, s_{1 l})) \|] γ^{u, t}_{i, j} γ^{u, t}_{k, l}$
		$= \sum_{i, j, k, l} [d_{f} ((1 - t) f_{0 i} + t f_{1 j}, (1 - u) f_{0 i} + u f_{1 j})$
		$+ \| (1 - t) d_{s} (s_{0 i}, s_{1 k}) + t d_{s} (s_{0 j}, s_{1 l}) - (1 - u) d_{s} (s_{0 i}, s_{1 k}) - u d_{s} (s_{0 j}, s_{1 l}) \|] V^{}_{i, j} V^{}_{k, l}$
		$= \sum_{i, j, k, l} [(t - u) d_{f} (f_{0 i}, f_{1 j}) + \| (t - u) d_{s} (s_{0 i}, s_{1 k}) - (t - u) d_{s} (s_{0 j}, s_{1 l}) \|] V^{}_{i, j} V^{}_{k, l}$
		$= \| t - u \| \sum_{i, j, k, l} [d_{f} (f_{0 i}, f_{1 j}) + \| d_{s} (s_{0 i}, s_{1 k}) - d_{s} (s_{0 j}, s_{1 l}) \|] V^{}_{i, j} V^{}_{k, l}$
		$= \| t - u \| d_{SGA} (μ_{0}, μ_{1}) .$

Here, we used the fact that $d_{f}$ is the Euclidean norm, hence

d_{f} ((1 - t) f_{0 i} + t f_{1 j}, (1 - u) f_{0 i} + u f_{1 j}) = ∥ (1 - t) f_{0 i} + t f_{1 j} - (1 - u) f_{0 i} - u f_{1 j} ∥ = | t - u | d_{f} (f_{0 i}, f_{1 j}) .

Therefore, we have

d_{SGA} (μ_{u}, μ_{t})

\leq | t - u | d_{SGA} (μ_{0}, μ_{1}) .

(8)

The remaining task is to prove that

d_{SGA} (μ_{u}, μ_{t})

\geq | t - u | d_{SGA} (μ_{0}, μ_{1}) .

(9)

To show that this inequality, we note that via the triangle inequality of $d_{SGA}$ and for any $0 \leq u \leq t \leq 1$ , it holds that

	$d_{SGA} (μ_{0}, μ_{1})$	$\leq d_{SGA} (μ_{0}, μ_{u}) + d_{SGA} (μ_{u}, μ_{t}) + d_{SGA} (μ_{t}, μ_{1})$
		$\leq u d_{SGA} (μ_{0}, μ_{1}) + (t - u) d_{SGA} (μ_{0}, μ_{1}) + (1 - t) d_{SGA} (μ_{0}, μ_{1})$
		$= d_{SGA} (μ_{0}, μ_{1}) .$

Hence, for any $0 \leq u \leq t \leq 1$ , we obtain

d_{SGA} (μ_{0}, μ_{u}) + d_{SGA} (μ_{u}, μ_{t}) + d_{SGA} (μ_{t}, μ_{1}) = u d_{SGA} (μ_{0}, μ_{1}) + (t - u) d_{SGA} (μ_{0}, μ_{1}) + (1 - t) d_{SGA} (μ_{0}, μ_{1}) .

(10)

Suppose that

d_{SGA} (μ_{u}, μ_{t}) < (t - u) d_{SGA} (μ_{0}, μ_{1}) .

Then combining with the fact that

d_{SGA} (μ_{0}, μ_{u}) \leq u d_{SGA} (μ_{0}, μ_{1}), and d_{SGA} (μ_{t}, μ_{1}) \leq (1 - t) d_{SGA} (μ_{0}, μ_{1}),

we have

d_{SGA} (μ_{0}, μ_{u}) + d_{SGA} (μ_{u}, μ_{t}) + d_{SGA} (μ_{t}, μ_{1}) < u d_{SGA} (μ_{0}, μ_{1}) + (t - u) d_{SGA} (μ_{0}, μ_{1}) + (1 - t) d_{SGA} (μ_{0}, μ_{1}) .

This leads to the contradiction with the equation (10.) Hence the desired inequality in (9) holds. Finally, we obtain

d_{SGA} (μ_{u}, μ_{t})

= | t - u | d_{SGA} (μ_{0}, μ_{1}) .

(11)

Appendix B Proofs of Technical Results

B.1 Proof of Proposition 1

First, let us suppose that $d_{SGA} (𝒢_{1}, 𝒢_{2}) = 0$ . We wish to prove the existence of a bijection $σ$ satisfying E1, E2, and E3. Indeed, let $𝑽 \in 𝒜 (𝒢_{1}, 𝒢_{2})$ be any admissible transportation plan that encode a valid matching between $𝒢_{1}$ and $𝒢_{2}$ . Then we define:

	$d (s_{1 i}, s_{1 k}) = \frac{1}{2} [d_{f} (f_{1 i}, f_{1 k}) + d_{s} (s_{1 i}, s_{1 k})], \forall i, k \in [M]^{2},$		(12)
	$d (s_{2 j}, s_{2 l}) = \frac{1}{2} [d_{f} (f_{2 j}, f_{2 l}) + d_{s} (s_{2 j}, s_{2 l})], \forall j, l \in [M]^{2} .$		(13)

Recall that we then define SGM discrepancy as:

	$d_{SGA} (𝒢_{1}, 𝒢_{2})$	$= \min_{𝑽 \in 𝒜 (𝒢_{1}, 𝒢_{2})} (\sum_{i, j} A^{v}_{i, j} V_{i, j} + \sum_{i, j, k, l} A^{e}_{i, j, k, l} V_{i, j} V_{k, l}) = \min_{𝑽 \in 𝒜 (𝒢_{1}, 𝒢_{2})} O (𝑨^{v}, 𝑨^{e}, 𝑽)$
		$= \min_{𝑽 \in 𝒜 (𝒢_{1}, 𝒢_{2})} [O_{f} (𝑨^{v}, 𝑽) + O_{s} (𝑨^{e}, 𝑽)] .$		(14)

It should be recalled that the vertex affinity matrix $𝑨^{v} \in ℝ^{M \times M}$ , defined as $A^{v}_{i, j} = (d_{f} (f_{1 i}, f_{2 j}))_{i, j}$ , was introduced in the previous section. The edge affinity tensor, denoted by $𝑨^{e}$ , is defined as follows: $A^{e}_{i, j, k, l} = | d_{s} (s_{1 i}, s_{1 k}) - d_{s} (s_{2 j}, s_{2 l}) |$ .

Let $𝑽^{*}$ be the optimal coupling for $d_{SGA} (𝒢_{1}, 𝒢_{2})$ . Then we have

O_{f} (𝑨^{v}, 𝑽^{*}) + O_{s} (𝑨^{e}, 𝑽^{*}) = \min_{𝑽 \in 𝒜 (𝒢_{1}, 𝒢_{2})} [O_{f} (𝑨^{v}, 𝑽) + O_{s} (𝑨^{e}, 𝑽)] = d_{SGA} (𝒢_{1}, 𝒢_{2}) = 0 .

(15)

Since both $O_{f} (𝑨^{v}, 𝑽^{*})$ and $O_{s} (𝑨^{e}, 𝑽^{*})$ are non-negative, we conclude that $O_{f} (𝑨^{v}, 𝑽^{*}) = O_{s} (𝑨^{e}, 𝑽^{*}) = 0$ . Now we wish to use the following Lemma B.2, which is proved in Appendix B.2.

Lemma 1.

Given the definition of $\bar{A}^{e}_{i, j, k, l} = | d (s_{1 i}, s_{1 k}) - d (s_{2 j}, s_{2 l}) |$ where $d (s_{1 i}, s_{1 k})$ and $d (s_{2 j}, s_{2 l})$ are provided in equations (12) and (13), respectively, it holds that

O_{s} (\bar{𝑨}^{e}, 𝑽^{*}) = \sum_{i, j, k, l} \bar{A}^{e}_{i, j, k, l} V^{*}_{i, j} V^{*}_{k, l} = \sum_{i, j, k, l} | d (s_{1 i}, s_{1 k}) - d (s_{2 j}, s_{2 l}) | V^{*}_{i, j} V^{*}_{k, l} = 0 .

(16)

Moreover, there exists a bijective $σ : [M] \mapsto [N]$ with $M = N$ satisfies the weight and distance $d$ preserving isometry as follows:

E1.

$\forall i \in [M] : w_{1 i} = w_{2 σ (i)}$ .
E3*.

$\forall i, k \in [M]^{2} : d (s_{1 i}, s_{1 k}) = d (s_{2 σ (i)}, s_{2 σ (k)})$ .

Because we have $𝑽^{*}$ is the optimal coupling w.r.t. the distance $d$ such that

O_{s} (\bar{𝑨}^{e}, 𝑽^{*}) = \min_{𝑽 \in 𝒜 (𝒢_{1}, 𝒢_{2})} O_{s} (\bar{𝑨}^{e}, 𝑽) = 0,

(17)

$𝑽^{*}$ is supported by $σ$ and satisfies $𝑽^{*} = 𝑰_{M \times N} \times σ$ . Therefore, $O_{f} (𝑨^{v}, 𝑽^{*}) = \sum_{i, j} d_{f} (f_{1 i}, f_{2 σ (i)}) V^{*}_{i, j} = \sum_{i} d_{f} (f_{1 i}, f_{2 σ (i)}) \sum_{j} V^{*}_{i, j} = \sum_{i} d_{f} (f_{1 i}, f_{2 σ (i)}) = 0$ . Here, we used the fact that

𝑽^{*} \in 𝒜 (𝒢_{1}, 𝒢_{2}) = {𝑽 \in {0, 1}^{M \times N} : \sum_{i = 1}^{M} V_{i, j} = w_{1 j} = 1, \sum_{j = 1}^{N} V_{i, j} = w_{2 i} = 1} .

Note that $d_{f} (f_{1 i}, f_{2 σ (i)}), i \in [M]$ are all non-negative. This leads to $d_{f} (f_{1 i}, f_{2 σ (i)}) = 0, \forall i \in [M]$ . This is equivalent to $f_{1 i} = f_{2 σ (i)}, \forall i \in [M]$ since $d_{f}$ is a metric, which is the desired E2. Therefore, we also have $d_{f} (f_{1 i}, f_{1 k}) = d_{f} (f_{2 σ (i)}, f_{2 σ (k)}), \forall i, k \in [M]$ . Combining equations (12), (13), and E3*, we have

	$d (s_{1 i}, s_{1 k})$	$= \frac{1}{2} [d_{f} (f_{1 i}, f_{1 k}) + d_{s} (s_{1 i}, s_{1 k})],$		(18)
	$d (s_{2 σ (i)}, s_{σ (k)})$	$= \frac{1}{2} [d_{f} (f_{2 σ (i)}, f_{2 σ (k)}) + d_{s} (s_{2 σ (i)}, s_{2 σ (k)})], \forall i, k \in [M]^{2} .$		(19)

This leads to the desired result, i.e., E3. $d_{s} (s_{1 i}, s_{1 k}) = d_{s} (s_{2 σ (i)}, s_{2 σ (k)}), \forall i, k \in [M]^{2}$ .

Now, let us suppose that $M = N$ there exists a bijection $σ : [M] \mapsto [N]$ satisfying E1, E2, and E3. We wish to prove that $d_{SGA} (𝒢_{1}, 𝒢_{2}) = 0$ . Then we can consider the transportation plan $𝑽^{*} = 𝑰_{M \times N} \times σ$ , i.e., $𝑽^{*}$ is associated with $i \mapsto i$ and $j \mapsto σ (i)$ . Using E1, it holds that $𝑽^{*} \in 𝒜 (𝒢_{1}, 𝒢_{2})$ . Moreover, via E2 and E3, we also have

	$d_{SGA} (𝒢_{1}, 𝒢_{2})$	$= \min_{𝑽 \in 𝒜 (𝒢_{1}, 𝒢_{2})} O (𝑨^{v}, 𝑨^{e}, 𝑽) \leq \sum_{i, j} A^{v}_{i, j} V^{}_{i, j} + \sum_{i, j, k, l} A^{e}_{i, j, k, l} V^{}_{i, j} V^{*}_{k, l}$
		$= \sum_{i, j} d_{f} (f_{1 i}, f_{2 j}) V^{}_{i, j} + \sum_{i, j, k, l} \| d_{s} (s_{1 i}, s_{1 k}) - d_{s} (s_{2 j}, s_{2 l}) \| V^{}_{i, j} V^{*}_{k, l}$
		$= \sum_{i, j} d_{f} (f_{1 i}, f_{2 σ (i)}) V^{}_{i, j} + \sum_{i, j, k, l} \| d_{s} (s_{1 i}, s_{1 k}) - d_{s} (s_{2 σ (i)}, s_{2 σ (k)}) \| V^{}_{i, j} V^{*}_{k, l} = 0 .$

This leads to the desired result that $d_{SGA} (𝒢_{1}, 𝒢_{2}) = 0$ .

B.2 Proof of Lemma 1

By definitions and the triangle inequalities of the metric $d_{f}$ and $d_{s}$ , we have

$O_{s} (\bar{𝑨}^{e}, 𝑽^{*})$	$= \sum_{i, j, k, l} \| d (s_{1 i}, s_{1 k}) - d (s_{2 j}, s_{2 l}) \| V^{}_{i, j} V^{}_{k, l}$
	$= \sum_{i, j, k, l} \| \frac{1}{2} [d_{f} (f_{1 i}, f_{1 k}) + d_{s} (s_{1 i}, s_{1 k})] - \frac{1}{2} [d_{f} (f_{2 j}, f_{2 l}) + d_{s} (s_{2 j}, s_{2 l})] \| V^{}_{i, j} V^{}_{k, l}$
	$= \sum_{i, j, k, l} \| \frac{1}{2} [d_{f} (f_{1 i}, f_{1 k}) - d_{f} (f_{2 j}, f_{2 l})] + \frac{1}{2} [d_{s} (s_{1 i}, s_{1 k}) - d_{s} (s_{2 j}, s_{2 l})] \| V^{}_{i, j} V^{}_{k, l}$
	$\leq \frac{1}{2} \sum_{i, j, k, l} \| d_{f} (f_{1 i}, f_{1 k}) - d_{f} (f_{2 j}, f_{2 l}) \| V^{}_{i, j} V^{}_{k, l} + \frac{1}{2} \sum_{i, j, k, l} \| d_{s} (s_{1 i}, s_{1 k}) - d_{s} (s_{2 j}, s_{2 l}) \| V^{}_{i, j} V^{}_{k, l}$
	$= \frac{1}{2} \sum_{i, j, k, l} \| d_{f} (f_{1 i}, f_{1 k}) - d_{f} (f_{2 j}, f_{2 l}) \| V^{}_{i, j} V^{}_{k, l} + \frac{1}{2} O_{s} (𝑨^{e}, 𝑽^{*})$
	$= \frac{1}{2} \sum_{i, j, k, l} \| d_{f} (f_{1 i}, f_{1 k}) - d_{f} (f_{2 j}, f_{2 l}) \| V^{}_{i, j} V^{}_{k, l} (since O_{s} (𝑨^{e}, 𝑽^{*}) = 0) .$	(20)

Using the triangle inequality of the metric $d_{f}$ again, we have

	$d_{f} (f_{1 i}, f_{1 k}) \leq d_{f} (f_{1 i}, f_{2 j}) + d_{f} (f_{2 j}, f_{2 l}) + d_{f} (f_{2 l}, f_{1 k}),$
	$d_{f} (f_{2 j}, f_{2 l}) \leq d_{f} (f_{2 j}, f_{1 i}) + d_{f} (f_{1 i}, f_{1 k}) + d_{f} (f_{1 k}, f_{2 l}) .$

This is equivalent to

	$d_{f} (f_{1 i}, f_{1 k}) - d_{f} (f_{2 j}, f_{2 l}) \leq d_{f} (f_{1 i}, f_{2 j}) + d_{f} (f_{1 k}, f_{2 l}),$
	$d_{f} (f_{2 j}, f_{2 l}) - d_{f} (f_{1 i}, f_{1 k}) \leq d_{f} (f_{1 i}, f_{2 j}) + d_{f} (f_{1 k}, f_{2 l}) .$		(21)

We consider two sets $I_{1} = {i, j, k, l : d_{f} (f_{1 i}, f_{1 k}) - d_{f} (f_{2 j}, f_{2 l}) \leq 0}$ and $I_{2} = {i, j, k, l : d_{f} (f_{2 j}, f_{2 l}) - d_{f} (f_{1 i}, f_{1 k}) \leq 0}$ . Combining equations (B.2) and (21), it holds that

$O_{s} (\bar{𝑨}^{e}, 𝑽^{*})$	$\leq \frac{1}{2} \sum_{i, j, k, l} \| d_{f} (f_{1 i}, f_{1 k}) - d_{f} (f_{2 j}, f_{2 l}) \| V^{}_{i, j} V^{}_{k, l}$
	$= \frac{1}{2} \sum_{i, j, k, l \in I_{1}} [d_{f} (f_{2 j}, f_{2 l}) - d_{f} (f_{1 i}, f_{1 k})] V^{}_{i, j} V^{}_{k, l}$
	$+ \frac{1}{2} \sum_{i, j, k, l \in I_{2}} [d_{f} (f_{1 i}, f_{1 k}) - d_{f} (f_{2 j}, f_{2 l})] V^{}_{i, j} V^{}_{k, l}$
	$\leq \frac{1}{2} \sum_{i, j, k, l \in I_{1}} [d_{f} (f_{1 i}, f_{2 j}) + d_{f} (f_{1 k}, f_{2 l})] V^{}_{i, j} V^{}_{k, l}$
	$+ \frac{1}{2} \sum_{i, j, k, l \in I_{2}} [d_{f} (f_{1 i}, f_{2 j}) + d_{f} (f_{1 k}, f_{2 l})] V^{}_{i, j} V^{}_{k, l}$
	$= \frac{1}{2} \sum_{i, j, k, l} [d_{f} (f_{1 i}, f_{2 j}) + d_{f} (f_{1 k}, f_{2 l})] V^{}_{i, j} V^{}_{k, l}$
	$= \frac{M}{2} \sum_{i, j} d_{f} (f_{1 i}, f_{2 j}) V^{}_{i, j} + \frac{M}{2} \sum_{k, l} d_{f} (f_{1 k}, f_{2 l}) V^{}_{k, l} = M O_{f} (𝑨^{v}, 𝑽^{*}) = 0 .$	(22)

Hence, $O_{s} (\bar{𝑨}^{e}, 𝑽^{*}) = 0$ since $O_{s} (\bar{𝑨}^{e}, 𝑽^{*}) \geq 0$ . Here, we have $𝑽^{*}$ is the optimal coupling such that

O_{s} (\bar{𝑨}^{e}, 𝑽^{*}) = \min_{𝑽 \in 𝒜 (𝒢_{1}, 𝒢_{2})} O_{s} (\bar{𝑨}^{e}, 𝑽) .

(23)

Hence, in accordance with Theorem 5.1 from [61, 60], there exists an isomorphisms between the metric spaces associated with $𝒢_{1}$ and $𝒢_{2}$ , described respectively by their mixing measure $μ_{1} = \sum_{i = 1}^{M} w_{1 i} δ_{(f_{1 i}, s_{1 i})}$ and $μ_{2} = \sum_{j = 1}^{N} w_{2 j} δ_{(f_{2 j}, s_{2 j})}$ . This means that there exists a bijective with weight preserving isometry $σ : [M] \mapsto [N]$ . This implies that $M = N$ and there exists a bijective $σ : [M] \mapsto [N]$ satisfies the weight and distance $d$ preserving isometry as follows:

E1.

$\forall i \in [M] : w_{1 i} = w_{2 σ (i)}$ .
E3*.

$\forall i, k \in [M]^{2} : d (s_{1 i}, s_{1 k}) = d (s_{2 σ (i)}, s_{2 σ (k)})$ .

Appendix C Examples of Extended Contexts Generated Using GPT-4

We present several examples of enriched captions generated using the GPT-4 API in Table 8. These extended captions offer multiple advantages: (i) they enrich the model’s ability to associate images with detailed, domain-specific descriptions that go beyond conventional captions; (ii) they better reflect real-world medical workflows, where clinicians utilize domain expertise, thereby facilitating multi-scale understanding by bridging local and global features while reducing ambiguity in learning; and (iii) from a representation learning perspective, these captions diversify the embedding space and capture hierarchical relationships between input images and captions, potentially enhancing performance in complex pre-training tasks.

[Uncaptioned image] — Table 8: Example of a conversation extended with enriched caption explanations.

Appendix D Expanded Discussion of Related Work

D.1 Graph Perspective and Optimal Transport for Alignment Problems

ExGra-Med formulates the graph alignment to solve the node-to-node correspondences under edge constraints indeed can be formulated using optimal transport, namely fused Gromov-wasserstein optimal transport (FGW-OT) [67, 59]. However, two main challenges hinder us from using optimal transport in ExGra-Med:

•

First, performing the forward pass to compute alignment between graph pairs using optimal transport is computationally expensive [67], making it impractical for scaling to large-scale LLM training with hundreds of thousands of samples. This obstacle is further challenged in ExGra-Med, where three separate graph alignment problems have to be solved, significantly incurring computational costs. In contrast, adopting a graph-based formulation enables the use of heuristic solvers specifically designed for combinatorial graph matching [89, 79], providing an efficient solution to address the high computational complexity of graph alignment.
•

Second, our training loss (Hamming loss in Eq.(5) requires gradients from graph alignments to learn feature representations. Using optimal transport would necessitate backpropagation through its Sinkhorn iterations [21] (50-100), adding substantial computational cost and GPU memory usage for storing intermediate variables. Our graph-based formulation addresses the second challenge by leveraging modern gradient estimation techniques for black-box optimization [68], making the backward step efficient for LLM training.

D.2 Combinatorial Alignment for Representation Learning

Combinatorial graph alignment is a key problem in computer vision, aiming to establish correspondences between elements of two discrete sets, such as image key points or 3D mesh vertices. It is widely used in tasks like 3D vision [9], tracking [34], and shape model learning [30]. In contrastive learning, LVM-Med [62] is the most relevant to our ExGra-Med toward learning feature representation by back-propagation through a combinatorial training loss. However, there are key differences between them:

•

Motivation: ExGra-Med identifies the data-intensive nature of auto-regressive modeling in LLaVa-Med and addresses it by introducing a multi-graph alignment approach across vision, image captions, and extended captions. LVM-Med, in contrast, is designed as a single-modality pre-trained model (vision) that learns through contrastive learning between images and their augmented versions.
•

Optimization Solver: ExGra-Med integrates multiple modalities (vision, captions, and extended contexts) into a barycenter graph-based solver, making it computationally efficient. LVM-Med relies on pairwise graph matching, which becomes computationally expensive for large-scale models. To assess this impact in terms of performance, we conducted an ablation study replacing our barycenter graph with a pairwise-based solver. As shown in Table 6 (bottom row), this alternative approach resulted in lower records.
•

Graph Construction: ExGra-Med works in the vision-language domain, making augmentation complex since captions must remain semantically meaningful. It introduces extended contexts generated by LLMs (e.g., GPT-4, Gemini) to enhance representation learning. LVM-Med in the other direction works with the vision domain and thus can define two graphs - one based on input images and another on their augmented versions.
•

Theoretical Analytic: ExGra-Med introduces new theoretical insights, proving that its multi-graph distance is a valid metric and demonstrating that the shortest path in the geodesic space of multi-modal graphs enhances learning. Opposingly, LVM-Med lacks theoretical contributions, focusing primarily on empirical performance.
•

Performance Comparison: We compared ExGra-Med, which leverages a triplet alignment between images, captions, and extended captions, against the LVM-Med solver, which employs contrastive learning between images and captions. Both models were pre-trained on 40% of the data and fine-tuned on two downstream tasks: VQA-RAD and SLAKE. The results show that ExGra-Med consistently outperforms LVM-Med, achieving scores of $74.37$ vs. $72.12$ on VQA-RAD and $84.99$ vs. $81.95$ on SLAKE.

Appendix E Medical Visual Chatbot

Datasets. Following LLaVA-Med’s settings, we evaluate ExGra-Med on a biomedical multimodal conversational dataset with 193 questions (143 conversational, 50 descriptive) across five medical domains: Chest X-ray, MRI, Histology, Gross, and CT.

Table 9: Medical visual chatbot evaluation. Results are reported using GPT-4 as the scorer.

Method #Para Question Type Domain Overall Conver. Descr. CXR MRI Hist. Gross CT LLAVA 7B 39.40 26.20 41.60 33.40 38.40 32.91 33.40 36.1 LLAVA-Med 1.0 7B 47.4 33.99 51.31 36.32 45.61 41.09 44.87 43.93 LLAVA-Med 1.5 7B 46.78 34.58 54.58 36.5 41.85 40.3 45.02 43.62 MedFlamingo 8.3B 28.58 13.89 26.93 21.34 22.09 32.71 22.25 24.77 Med-Dr 40B 35.61 19.28 38.98 26.28 29.10 35.40 28.30 31.38 Biomed-GPT 182M 20.71 17.99 27.53 18.50 17.18 14.72 22.08 20.01 GPT-4o 200B 42.04 25.47 42.77 39.74 38.68 31.40 35.59 37.75 ExGra-Med 7B 48.49 34.32 58.37 36.82 46.05 45.19 38.24 44.82 ExGra-Med (DCI) 7B 48.99 34.01 59.9 32.34 51.88 42.53 38.28 45.11

Baselines. We evaluate with several SOTA multimodal large language models, including general models like LLaVA and GPT-4o, as well as medical-focused models such as LLaVA-Med and its variants, Med-Flamingo, Med-Dr, and Biomed-GPT. We use the officially provided weights for all comparison baselines without additional reproduction steps. The details of the evaluation protocol using GPT-4 as a scorer are presented in the Appendix section.

Evaluation Protocol

We evaluate the ability of models to serve as a medical visual chatbot as follows: each of the 193 novel questions in the dataset has a corresponding ground-truth answer. We generate responses to each question from the LMM. Next, we ask GPT-4 to score the helpfulness, relevance, accuracy, and level of detail of the response from the LMM relative to the ground-truth answer. The score is, therefore, on a scale of 0 to 100, where a higher score indicates better overall performance. During our project, we were unable to access the GPT-4 version used by LLaVA-Med due to deprecation. Therefore, we opt for the GPT-4o version as a judge. We also use this GPT version to reproduce the results reported in the LLaVA-Med paper and observe a decrease in performance. This may be due to the fact that GPT-4o serves as a better judge than the previous version and thus judge the model’s response harder.

Results

Table 9 shows the experimental results of ExGra-Med alongside competitive methods, with the highest scores in bold. Our two method variants—based on LLaVA 1.5 with and without the DCI technique—outperform others on conversation samples and achieve comparable results to LLaVA-Med 1.5 on description samples. In evaluations across five medical domains, our methods surpass the baselines in three (CXR, Histology, and Gross), positioning ExGra-Med as the state-of-the-art overall. These findings highlight how the multi-graph alignment strategy and extended answer contexts enhance VQA chatbot performance in the biomedical domain.

Qualitative results are shown in Table 10, where ExGra-Med’s generated responses are compared against a series of example questions and image contexts. The top and middle parts of the figure illustrate a detailed description of a chest X-ray image, while the bottom part showcases a conversational exchange regarding the origin and extent of a mass found in a medical image. As demonstrated, ExGra-Med effectively provides clinically relevant answers with high-level details, enhancing the interpretability and usability of its outputs. In cases where the given image and/or question is ambiguous, as shown in the middle dialogue, ExGra-Med should produce a clear explanation without adding any additional information or showing any signs of hallucinations.

Appendix F Zero-shot Image Classification as MedVQA

Datasets. We assess the generalization of ExGra-Med on zero-shot image classification by adapting public datasets from [32]. We convert image classification tasks into visual question answering, where the model selects the correct answer from input options based on the given image (Figure 9). We focus on three key data modalities prevalent in our pre-training: Microscopy, Computed Tomography (CT), and Chest X-Ray (CXR). This evaluation spans several downstream tasks, including 8 datasets for Microscopy, 4 for CT, and 11 for CXR, totaling 23 datasets.

Baselines. We use checkpoints from LLaVa-Med, Med-Flamingo, and RadFM [95] for zero-shot inference on the collected datasets. Notably, RadFM is pre-trained on $16 M$ 2D and 3D medical scans, while ExGra-Med is trained on just $600 K$ instruction-following data. For baseline models, we follow the prompts proposed by [32], with detailed evaluations using third-party software to align model outputs with ground-truth answers.

Evaluation method. Following [32], we use Question-answering Score as a metric to report the performance of the models. Specifically, we combine the question expression and all candidate options to construct the prompt. Our prompt template therefore is as follows:

“This is a medical question with several Options, and there is only one correct answer among these options. Please select the correct answer for the question. Remember, you can only select one option. The Question is: $⟨$ Question $⟩$ . ### The candidate Options are: $⟨$ Options $⟩$ .

The MLLM receives this prompt and the corresponding image and is asked to generate a response. We then utilize difflib, a standard Python package to compare two strings, to calculate the similarity of the response with each of the candidate options and pick the option with the largest similarity as the final prediction. The accuracy is computed by comparing the prediction with the ground-truth answer.

Results. Figure 4 illustrates the average performance of ExGra-Med across Microscopy, CT, and Chest X-Ray modalities, with the total number of images and question-answer items listed below. Detailed results for each dataset within these modality groups are provided in Tables 11, 13, and 12. Overall, ExGra-Med still outperforms other models across all datasets, especially excelling in the microscopy modality, where it exceeds the runner-up, RadFM, by $8.2 %$ . We attribute these benefits to the strong alignment between visual features and language embeddings achieved through triplet constraints, which compel the model to capture deeper semantic relationships.

Figure 9 provides several examples of microscopy and CT images. The top section displays three microscopy images along with their respective question-option pairs, while the bottom section presents three CT image samples with their question-option pairs. The ground truth correct options are highlighted in blue. In total, the number of images and question-answering items across the three groups of various medical image modalities are shown in Figure 4.

Table 11: Performance comparison on various microscopy image datasets.

Microscopy Image Dataset Method LLaVA-Med Med-Flamingo RadFM Ours CRC100k [41] 24.74 17.18 27.48 28.06 ALL Challenge [26] 29.24 13.20 39.88 27.49 BioMediTech [65] 39.14 16.08 47.84 46.97 Blood Cell [8] 21.11 15.25 16.95 29.87 BreakHis [85] 23.27 13.62 18.26 33.74 NLM-Malaria [69] 30.67 6.76 32.43 66.67 HuSHeM [81] 16.85 18.18 11.36 25.84 MHSMA [36] 29.64 39.66 38.41 39.70 Avg. 26.83 17.49 29.08 37.29

Table 12: Performance comparison across CXR datasets.

CXR Dataset Method LLaVA-Med Med-Flamingo RadFM Ours RUS CHN [97] 28.05 20.19 29.88 41.88 Mura [77] 20.70 25.91 43.47 30.19 Pulmonary Chest MC [35] 21.05 27.03 10.81 47.37 MIAS [86] 25.35 38.30 28.37 42.96 Pulmonary Chest Shenzhen [35] 26.35 32.54 36.95 19.93 COVIDx CXR-4 [94] 28.25 25.83 48.14 22.68 Knee Osteoarthritis [13] 11.20 22.24 6.19 8.69 Chest X-Ray PA [6] 29.06 38.04 38.28 49.41 CoronaHack [18] 19.74 33.67 22.99 47.81 Covid-19 tianchi [20] 16.67 45.26 33.68 30.21 Covid19 heywhale [17] 22.03 56.31 23.37 29.28 Avg. 23.18 32.01 28.84 33.67

Table 13: Performance comparison on various CT (Computed Tomography) datasets.

CT Dataset Method LLaVA-Med Med-Flamingo RadFM Ours Chest CT Scan [15] 25.72 20.00 25.06 20.09 SARS-CoV-2 CT [83] 28.79 40.92 44.55 34.95 Covid CT [19] 22.61 21.72 28.79 37.19 OCT & X-Ray 2017 [42] 27.21 28.08 29.46 47.89 Avg. 26.08 27.68 31.97 35.03

Results We provide detailed results for datasets on each data modality in Tables 11, 13, and 12.

Appendix G LLM Prompting for GPT-4 to Generate Extended Captions

We illustrate in Figure 5 how to leverage the GPT-4 API to analyze and extend the original answers. For detailed responses in specific cases, refer to Table 8.

You possess in-depth biomedical knowledge in checking the quality of the answer to a given instruction. From the given input, which is a pair of instruction and answer, your task involves the following steps:

1.

Explain why the given answer is not good for its instruction. Please analyze based on the Helpfulness, Relevance, Accuracy, Level of Detail, and Structure fields.
2.

Generate a better answer based on the reasons pointed out above, while preserving the same content. To achieve that, you may want to adjust the level of details, add bullet points, or use comprehensive words, etc. Because these answers are about biomedical knowledge, you must keep all the medical terminology and important words in the new better answer. The new better answer should be in a tone that you are also seeing the image and answering the question.
3.

Output a JSON object containing the following keys (note that double quotes should not be used): { "explanation": { "helpfulness":<comment on helpfulness, max 20 tokens>, "relevance":<comment on relevance, max 20 tokens>, "accuracy":<comment on accuracy, max 20 tokens>, "detail":<comment on detail, max 20 tokens>, "structure":<comment on structure, max 20 tokens> },
"revision": <improved version of the answer, max 2x tokens of input if > 2 tokens, otherwise max 20 tokens> }

Figure 5: Instructions provided to the system for analyzing the quality of answers based on different criteria and generating a revised response in JSON format.

Appendix H Additional Results for Multi-modal Pre-training Comparison

H.1 MedVQA datasets

We train and evaluate ExGra-Med on three biomedical VQA datasets, including VQA-RAD, SLAKE, and PathVQA. The dataset statistics are summarized in detail in Table 14.

•

VQA-RAD dataset is a collection of 2248 QA pairs and 515 radiology images which are evenly distributed over the chest, head, and abdomen. Over half of the answers are closed-ended (i.e., yes/no type), while the rest are open-ended with short phrase answers.
•

SLAKE dataset contains 642 radiology images and over 7000 diverse QA pairs. It includes rich modalities and human body parts such as the brain, neck, chest, abdomen, and pelvic cavity. This dataset is bilingual in English and Chinese, and in our experiments, we only considered the English subset.
•

PathVQA dataset contain pathology images. It has a total of 32795 QA pairs and 4315 pathology images. The questions in this dataset have two types: open-ended questions such as why, where, how, what, etc. and closed-ended questions.

Table 14: Dataset statistics for 3 medical VQA datasets: VQA-RAD, SLAKE, and PathVQA.

Dataset	VQA-RAD		SLAKE			PathVQA
Dataset	Train	Test	Train	Val	Test	Train	Val	Test
# Images	313	203	450	96	96	2599	858	858
# QA Pairs	1797	451	4919	1053	1061	19755	6279	6761
# Open	770	179	2976	631	645	9949	3144	3370
# Closed	1027	272	1943	422	416	9806	3135	3391

H.2 Results

Tables 15 and 16 present the results using 70% and 100% of the data. Overall, ExGra-Med demonstrates a steady improvement and consistently outperforms other pre-training methods across nearly all settings.

Table 15: Performance fine-tuning on MedVQA downstream datasets (pre-training 70%). Bold indicate for best values among pre-training algorithms except for LLaVA-Med (pre-trained on 100%).

Method VQA-RAD SLAKE PathVQA Overall Open Closed Avg. Open Closed Avg. Open Closed Avg. LLaVA-Med (100%) 63.65 81.62 72.64 83.44 83.41 83.43 36.78 91.33 64.06 73.37 LLaVA-Med (70%) 65.96 $↑$ 2.31 81.62 $↓$ 0 73.79 $↑$ 1.13 84.16 $↑$ 0.72 83.17 $↓$ 0.24 83.67 $↑$ 0.24 37.39 $↑$ 0.61 92.27 $↑$ 0.94 64.83 $↑$ 0.77 74.1 $↑$ 0.64 InfoNCE 64.18 77.94 71.06 70.9 82.69 76.80 33.58 88.5 61.04 69.63 PLOT 60.13 78.31 69.22 82.48 83.89 83.185 29.23 85.7 57.478 69.96 SigLIP 61.68 78.68 70.18 82.04 83.17 82.61 34.43 90.3 62.37 71.72 VLAP 64.08 79.41 71.75 84.94 85.1 85.02 36.44 91.51 63.98 73.58 ExGra-Med 67.12 81.99 74.56 84.81 84.86 84.84 37.26 91.77 64.52 74.64

Table 16: Performance fine-tuning on MedVQA downstream datasets (pre-training 100%).

Method VQA-RAD SLAKE PathVQA Overall Open Closed Avg. Open Closed Avg. Open Closed Avg. LLaVA-Med (100%) 63.65 81.62 72.64 83.44 83.41 83.43 36.78 91.33 64.06 73.37 InfoNCE 66.01 79.41 72.71 83.23 83.41 83.32 35.01 89.53 62.27 72.77 PLOT 63.58 77.21 70.4 82.44 84.86 83.65 34.45 89.97 62.21 72.09 SigLIP 57.11 74.26 65.69 85.07 83.41 84.24 36.47 89.38 62.925 70.95 VLAP 60.93 79.78 70.36 84.74 83.17 83.955 35.86 89.65 62.755 72.36 ExGra-Med 66.35 83.46 74.91 85.34 85.58 85.46 36.82 90.92 63.87 74.75

Appendix I Further Ablation Studies

I.1 $K$ Nearest Neighbor in the Graph Construction Step

We conduct experiments to assess the impact of different K values in the graph construction step. Table 17 presents model performance on the VQA-RAD dataset along with the training time for Step-2 pre-training using 10% of the data for each $K$ value. Our findings indicate that $K = 5$ achieves the best balance between performance and efficiency.

Table 17: Impact of Nearest Neighbors Count on Graph Construction. Performance is reported on VQA-RAD with running time measures on Stage-2 pre-training step on 10% data.

Settings	VQA-RAD
Settings	Open	Close	Avg.	Run Time
ExGra-Med (Full), K = 3	55.9	73.9	64.9	1h
ExGra-Med (Full), K = 5	66	79.04	72.52	1h4’
ExGra-Med (Full), K = 7	55.52	73.16	64.37	1h17’

Table 18: Comparison of pre-training algorithms with different feature embedding methods. Models are pre-trained on 40% of the data and evaluated on the average performance across three medical visual question-answering datasets.

Method	VQA-RAD	SLAKE	PathVQA
ExGra-Med	74.37	84.99	64.34
InfoNCE (avg.feature)	70.34	83.29	61.6
PLOT (optimal transport)	71.89	83.16	62.4

I.2 Feature representation analysis using average pooling for visual and language tokens

We investigate using average pooling token features in ExGra-Med with two experiments:

•

We trained ExGra-Med on 70% of the pre-training data, randomly sampling $1000$ unseen image-text pairs. The trained model extracted features using average pooling, and a box plot (Figure 6) visualized the central tendency, spread, and skewness of $1000$ positive and negative pairs. The results show: (i) the median similarity for positive pairs is significantly higher than for negative pairs, indicating clear separation; (ii) while some overlap exists in the interquartile ranges (IQRs), the shift in central tendency confirms the distinction; and (iii) outliers are present, particularly among negative pairs, but they minimally overlap with the core distribution of positive pairs.

Figure 6: Visualization of similarity values between positive and negative pairs based on features computed by ExGra-Med.
•

We compare against two pre-training algorithms, InfoNCE [43, 56] and PLOT [10]. Both utilize the same contrastive loss, but InfoNCE relies on cosine distance with averaged features, while PLOT directly computes optimal transport over sets of visual and text tokens. The results for these baselines are summarized in Table 18. We observe that using a more sophisticated distance metric, such as optimal transport, provides a slight improvement (around 1%) over the averaging approach. However, the performance gain is relatively modest. Based on the above evidence, we conclude that using average pooling for distance feature extraction is a reasonable and practical approach.

Appendix J Qualification Test on the GPT-generated Extended Captions

We adopt the GPT-4 as a tool for paraphrasing image captioning due to its improved performance compared with GPT-3.5, especially in healthcare [38]. During our implementations, we also randomly checked for a hundred samples and found consistency between extended context and original ones. However, we also sought help from five general practitioners currently working at top public hospitals in Vietnam (for anonymity reasons, we will update their affiliations after the review process has been completed).

In particular, we randomly chose $1000$ samples in Stage 2 of pre-training, covering five data modalities: chest X-ray, CT scan, MRI, histology, and others. Each doctor is assigned a specific data modality given their expertise, including 200 image-text pairs and corresponding captions. We then build an annotation tool for them to verify data where each sample is asked with two questions (i) whether the extended caption covers the original caption; and (ii) whether new concepts appearing in extended captions are correct. For (i) and (ii), doctors can rate with five levels (from 1 to 5), each indicating an increasing level of correctness (Figures 7-8).

We provide statistical correctness evaluated by general doctors for these domains in Figures 10,11,12, 13, and 14. It can be seen that most rating scores fall between 3 and 5, with only a small number of samples rated 1 or 2, validating the overall consistency of GPT-4 outputs. While concerns may arise regarding the impact of low-scoring extended captions (rated 1 or 2) on the LLM, it’s important to note that these extended captions are utilized solely for contrastive learning during pre-training to align the model’s latent space representations. They are not used in auto-regressive training, which involves predicting target ground-truth tokens. Additionally, the model is fine-tuned with the given training samples from downstream tasks after pre-training (no extended captions are used). Thus, we argue that the presence of a small number of noisy extended captions should not significantly affect the performance of the LLM.

[bib.bib75] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. (2023) Gpt-4 technical report. arXiv preprint arXiv:2303.08774. Cited by: §1, §1, §4.4. ↩ a b c

[bib.bib11] M. Agueh and G. Carlier (2011) Barycenters in the wasserstein space. SIAM Journal on Mathematical Analysis 43 (2), pp. 904–924. Cited by: §1. ↩ a

[bib.bib45] J. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al. (2022) Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems 35, pp. 23716–23736. Cited by: §2, §4.6. ↩ a b

[bib.bib31] J. M. Altschuler and E. Boix-Adsera (2022) Wasserstein barycenters are np-hard to compute. SIAM Journal on Mathematics of Data Science 4 (1), pp. 179–203. Cited by: §3.4. ↩ a

[bib.bib86] C. An, F. Huang, J. Zhang, S. Gong, X. Qiu, C. Zhou, and L. Kong (2024) Training-free long-context scaling of large language models. arXiv preprint arXiv:2402.17463. Cited by: §3.1. ↩ a

[bib.bib109] A. Asraf and Z. Islam (2021) COVID19 pneumonia and normal chest x-ray pa dataset. mendeley data v1 (2021). Elsevier, Amsterdam, The Netherlands, Tech. Rep. Cited by: Table 12. ↩ a

[bib.bib40] F. Bernard, J. Thunberg, P. Swoboda, and C. Theobalt (2019) Hippi: higher-order projected power iterations for scalable multi-matching. In Proceedings of the ieee/cvf international conference on computer vision, pp. 10284–10293. Cited by: §2, §3.4. ↩ a b

[bib.bib95] [8] (2023) Blood cell images. Note: https://www.kaggle.com/datasets/paultimothymooney/blood-cellsAccessed: 2024-09-30 Cited by: Table 11. ↩ a

[bib.bib27] D. Cao and F. Bernard (2023) Self-supervised learning for multimodal non-rigid 3d shape matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17735–17744. Cited by: §D.2. ↩ a

[bib.bib8] G. Chen, W. Yao, X. Song, X. Li, Y. Rao, and K. Zhang (2022) Plot: prompt learning with optimal transport for vision-language models. International Conference on Learning Representations. Cited by: 2nd item, §2, 1st item. ↩ a b c

[bib.bib48] G. Chen, Y. Zheng, J. Wang, J. Xu, Y. Huang, J. Pan, Y. Wang, Y. Wang, Y. Qiao, T. Lu, et al. (2023a) Videollm: modeling video sequence with large language models. arXiv preprint arXiv:2305.13292. Cited by: §1, §2. ↩ a b

[bib.bib35] H. Chen, R. Shukla, R. Wu, S. Yang, D. Duong-Tran, D. M. H. Nguyen, M. Niepert, C. Beeche, J. Gee, J. Duda, et al. (2025) Adapting vision-language models for 3d ct/mri understanding on pmbb via slice selection and explanation analysis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2273–2282. Cited by: §4.6. ↩ a

[bib.bib108] P. Chen (2018) Cited by: Table 12. ↩ a

[bib.bib62] Z. Chen, A. H. Cano, A. Romanou, A. Bonnet, K. Matoba, F. Salvi, M. Pagliardini, S. Fan, A. Köpf, A. Mohtashami, et al. (2023b) Meditron-70b: scaling medical pretraining for large language models. arXiv preprint arXiv:2311.16079. Cited by: §4.6. ↩ a

[bib.bib100] [15] Chest ct-scan images dataset. Note: https://tianchi.aliyun.com/dataset/93929Accessed: 2024-09-30 Cited by: Table 13. ↩ a

[bib.bib3] L. Chizat and F. Bach (2018) On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport. In Advances in Neural Information Processing Systems, Vol. 31. Cited by: §A.2. ↩ a

[bib.bib112] M. E. Chowdhury, T. Rahman, A. Khandakar, R. Mazhar, M. A. Kadir, Z. B. Mahbub, K. R. Islam, M. S. Khan, A. Iqbal, N. Al Emadi, et al. (2020) Can ai help in screening viral and covid-19 pneumonia?. Ieee Access 8, pp. 132665–132676. Cited by: Table 12. ↩ a

[bib.bib110] J. P. Cohen, P. Morrison, L. Dao, K. Roth, T. Q. Duong, and M. Ghassemi (2020) Covid-19 image data collection: prospective predictions are the future. arXiv preprint arXiv:2006.11988. Cited by: Table 12. ↩ a

[bib.bib102] [19] Covid ct dataset. Note: https://tianchi.aliyun.com/dataset/106604Accessed: 2024-09-30 Cited by: Table 13. ↩ a

[bib.bib111] [20] (2023) Covid-19 image dataset: 3 way classification - covid-19, viral pneumonia, normal. Note: https://tianchi.aliyun.com/dataset/93853 Cited by: Table 12. ↩ a

[bib.bib33] M. Cuturi (2013) Sinkhorn distances: lightspeed computation of optimal transport. Advances in neural information processing systems 26. Cited by: 2nd item. ↩ a

[bib.bib64] J. Devlin (2018) Bert: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. Cited by: §2. ↩ a

[bib.bib82] V. Ehm, M. Gao, P. Roetzer, M. Eisenberger, D. Cremers, and F. Bernard (2024) Partial-to-partial shape matching with geometric consistency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 27488–27497. Cited by: §3.3. ↩ a

[bib.bib116] R. Girdhar, A. El-Nouby, Z. Liu, M. Singh, K. V. Alwala, A. Joulin, and I. Misra (2023) Imagebind: one embedding space to bind them all. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15180–15190. Cited by: §2, 2nd item. ↩ a b

[bib.bib24] W. Guo, N. Ho, and M. Jordan (2020) Fast algorithms for computational optimal transport and wasserstein barycenter. In International Conference on Artificial Intelligence and Statistics, pp. 2088–2097. Cited by: §3.4. ↩ a

[bib.bib92] A. Gupta and R. Gupta (2019) ISBI 2019 c-nmc challenge: classification in cancer cell imaging. Select Proceedings 2. Cited by: Table 11. ↩ a

[bib.bib84] S. Haller, L. Feineis, L. Hutschenreiter, F. Bernard, C. Rother, D. Kainmüller, P. Swoboda, and B. Savchynskyy (2022) A comparative study of graph matching algorithms in computer vision. In European Conference on Computer Vision, pp. 636–653. Cited by: §3.3. ↩ a

[bib.bib67] S. He, Y. Nie, Z. Chen, Z. Cai, H. Wang, S. Yang, and H. Chen (2024) MedDr: diagnosis-guided bootstrapping for large-scale medical vision-language learning. arXiv preprint arXiv:2404.15127. Cited by: §1, §2, §4.4. ↩ a b c

[bib.bib68] X. He, Y. Zhang, L. Mou, E. Xing, and P. Xie (2020) Pathvqa: 30000+ questions for medical visual question answering. arXiv preprint arXiv:2003.10286. Cited by: §1, §4.3. ↩ a b

[bib.bib25] T. Heimann and H. Meinzer (2009) Statistical shape models for 3d medical image segmentation: a review. Medical image analysis 13 (4), pp. 543–563. Cited by: §D.2. ↩ a

[bib.bib55] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, et al. (2022) Lora: low-rank adaptation of large language models.. ICLR 1 (2), pp. 3. Cited by: 8th item. ↩ a

[bib.bib88] Y. Hu, T. Li, Q. Lu, W. Shao, J. He, Y. Qiao, and P. Luo (2024) Omnimedvqa: a new large-scale comprehensive evaluation benchmark for medical lvlm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22170–22183. Cited by: Appendix F, Appendix F, Appendix F. ↩ a b c

[bib.bib46] W. Huang, C. Wang, R. Zhang, Y. Li, J. Wu, and L. Fei-Fei (2023) Voxposer: composable 3d value maps for robotic manipulation with language models. arXiv preprint arXiv:2307.05973. Cited by: §2. ↩ a

[bib.bib26] J. Hyun, M. Kang, D. Wee, and D. Yeung (2023) Detection recovery in online multi-object tracking with sparse graph tracker. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 4850–4859. Cited by: §D.2. ↩ a

[bib.bib106] S. Jaeger, S. Candemir, S. Antani, Y. J. Wáng, P. Lu, and G. Thoma (2014) Two public chest x-ray datasets for computer-aided screening of pulmonary diseases. Quantitative imaging in medicine and surgery 4 (6), pp. 475. Cited by: Table 12. ↩ a b

[bib.bib99] S. Javadi and S. A. Mirroshandel (2019) A novel deep learning method for automatic assessment of human sperm images. Computers in biology and medicine 109, pp. 182–194. Cited by: Table 11. ↩ a

[bib.bib76] S. Jiang, T. Zheng, Y. Zhang, Y. Jin, and Z. Liu (2024) MoE-tinymed: mixture of experts for tiny medical large vision-language models. arXiv preprint arXiv:2404.10237. Cited by: §1, §4.4. ↩ a b

[bib.bib42] H. K. Jin, H. E. Lee, and E. Kim (2024) Performance of chatgpt-3.5 and gpt-4 in national licensing examinations for medicine, pharmacy, dentistry, and nursing: a systematic review and meta-analysis. BMC Medical Education 24 (1), pp. 1013. Cited by: Appendix J. ↩ a

[bib.bib36] A. E. W. Johnson, T. J. Pollard, S. J. Berkowitz, N. R. Greenbaum, M. P. Lungren, C. Y. Deng, R. G. Mark, and S. Horng (2019) MIMIC-cxr, a de-identified publicly available database of chest radiographs with free-text reports. Scientific Data 6 (1), pp. 317. External Links: Document Cited by: §4.6. ↩ a

[bib.bib59] W. Ju, Z. Fang, Y. Gu, Z. Liu, Q. Long, Z. Qiao, Y. Qin, J. Shen, F. Sun, Z. Xiao, et al. (2024) A comprehensive survey on deep graph representation learning. Neural Networks, pp. 106207. Cited by: §3.2. ↩ a

[bib.bib93] J. N. Kather, N. Halama, and A. Marx (2018) 100,000 histological images of human colorectal cancer and healthy tissue. Zenodo. Note: Accessed: 2024-09-30 External Links: Document Cited by: Table 11. ↩ a

[bib.bib103] D. S. Kermany, M. Goldbaum, W. Cai, C. C. Valentim, H. Liang, S. L. Baxter, A. McKeown, G. Yang, X. Wu, F. Yan, et al. (2018) Identifying medical diagnoses and treatable diseases by image-based deep learning. cell 172 (5), pp. 1122–1131. Cited by: Table 13. ↩ a

[bib.bib12] Z. Khan and Y. Fu (2023) Contrastive alignment of vision to language through parameter-efficient transfer learning. International Conference on Learning Representations. Cited by: 2nd item, §2, 1st item. ↩ a b c

[bib.bib114] S. Khanal, S. Sastry, A. Dhakal, and N. Jacobs (2023) Learning tri-modal embeddings for zero-shot soundscape mapping. The British Machine Vision Conference (BMVC). Cited by: §2, 2nd item. ↩ a b

[bib.bib91] D. P. Kingma (2014) Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980. Cited by: §4.1. ↩ a

[bib.bib69] J. J. Lau, S. Gayen, A. Ben Abacha, and D. Demner-Fushman (2018) A dataset of clinically generated visual questions and answers about radiology images. Scientific data 5 (1), pp. 1–10. Cited by: §1, §4.3. ↩ a b

[bib.bib37] K. Le-Duc, P. T. H. Trinh, D. M. H. Nguyen, T. Nguyen, N. T. Diep, A. Ngo, T. Vu, T. Vuong, A. Nguyen, M. Nguyen, V. T. Hoang, K. Nguyen, H. Nguyen, C. Ngo, A. Liu, N. Ho, A. Hauschild, K. X. Nguyen, T. Nguyen-Tang, P. Xie, D. Sonntag, J. Zou, M. Niepert, and A. T. Nguyen (2025) S-chain: structured visual chain-of-thought for medicine. arXiv preprint. Cited by: §4.6. ↩ a

[bib.bib52] C. Li, C. Wong, S. Zhang, N. Usuyama, H. Liu, J. Yang, T. Naumann, H. Poon, and J. Gao (2024) Llava-med: training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems 36. Cited by: §1, §2, §4.1, §4.1, §4.1, §4.4. ↩ a b c d e f

[bib.bib44] J. Li, D. Li, S. Savarese, and S. Hoi (2023a) Blip-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730–19742. Cited by: §1, §2. ↩ a b c

[bib.bib7] J. Li, D. Li, C. Xiong, and S. Hoi (2022) Blip: bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pp. 12888–12900. Cited by: §2. ↩ a

[bib.bib57] P. Li, G. Liu, L. Tan, J. Liao, and S. Zhong (2023b) Self-supervised vision-language pretraining for medial visual question answering. In 2023 IEEE 20th International Symposium on Biomedical Imaging (ISBI), Vol. , pp. 1–5. External Links: Document Cited by: §4.4. ↩ a

[bib.bib30] T. Lin, N. Ho, M. Cuturi, and M. I. Jordan (2022) On the complexity of approximating multi-marginal optimal transport. Journal of Machine Learning Research 23, pp. 1–43. Cited by: §2. ↩ a

[bib.bib87] B. Liu, L. Zhan, L. Xu, L. Ma, Y. Yang, and X. Wu (2021) Slake: a semantically-labeled knowledge-enhanced dataset for medical visual question answering. In 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), Vol. , pp. 1650–1654. External Links: Document Cited by: §4.3. ↩ a

[bib.bib90] H. Liu, C. Li, Y. Li, and Y. J. Lee (2024a) Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26296–26306. Cited by: §4.1. ↩ a

[bib.bib50] H. Liu, C. Li, Q. Wu, and Y. J. Lee (2024b) Visual instruction tuning. Advances in neural information processing systems 36. Cited by: §1, §2, §2, §4.4. ↩ a b c d

[bib.bib63] L. Liu, X. Sun, T. Xiang, Z. Zhuang, L. Yin, and M. Tan (2023) Contrastive vision-language alignment makes efficient instruction learner. arXiv preprint arXiv:2311.17945. Cited by: 2nd item, §2, 1st item. ↩ a b c

[bib.bib80] N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang (2024c) Lost in the middle: how language models use long contexts. Transactions of the Association for Computational Linguistics 12, pp. 157–173. Cited by: §3.1. ↩ a

[bib.bib19] R. Lou, K. Zhang, and W. Yin (2023) A comprehensive survey on instruction following. arXiv preprint arXiv:2303.10475. Cited by: §1. ↩ a

[bib.bib83] X. Ma, X. Chu, Y. Wang, Y. Lin, J. Zhao, L. Ma, and W. Zhu (2024) Fused gromov-wasserstein graph mixup for graph-level classifications. Advances in Neural Information Processing Systems 36. Cited by: §D.1. ↩ a

[bib.bib1] F. Mémoli and G. Sapiro (2005) A Theoretical and Computational Framework for Isometry Invariant Recognition of Point Cloud Data. Foundations of Computational Mathematics 5 (3), pp. 313–347. External Links: ISSN 1615-3383 Cited by: §B.2. ↩ a

[bib.bib2] F. Mémoli (2011) Gromov–Wasserstein Distances and the Metric Approach to Object Matching. Foundations of Computational Mathematics 11 (4), pp. 417–487 (en). External Links: ISSN 1615-3375, 1615-3383 Cited by: §B.2. ↩ a

[bib.bib43] D. MH Nguyen, H. Nguyen, N. Diep, T. N. Pham, T. Cao, B. Nguyen, P. Swoboda, N. Ho, S. Albarqouni, P. Xie, et al. (2024) Lvm-med: learning large-scale self-supervised vision models for medical imaging via second-order graph matching. Advances in Neural Information Processing Systems 36. Cited by: §D.2, §2, §4.6. ↩ a b c

[bib.bib21] P. Minervini, L. Franceschi, and M. Niepert (2023) Adaptive perturbation-based gradient estimation for discrete latent variable models. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 9200–9208. Cited by: §1, §2, Figure 3, §3.5. ↩ a b c d

[bib.bib77] M. Moor, Q. Huang, S. Wu, M. Yasunaga, Y. Dalmia, J. Leskovec, C. Zakka, E. P. Reis, and P. Rajpurkar (2023) Med-flamingo: a multimodal medical few-shot learner. In Machine Learning for Health (ML4H), pp. 353–367. Cited by: §2, §4.4. ↩ a b

[bib.bib94] L. Nanni, M. Paci, F. L. Caetano dos Santos, H. Skottman, K. Juuti-Uusitalo, and J. Hyttinen (2016) Texture descriptors ensembles enable image-based classification of maturation of human stem cell-derived retinal pigmented epithelium. PLoS One 11 (2), pp. e0149399. Cited by: Table 11. ↩ a

[bib.bib74] D. M. Nguyen, A. T. Le, T. Q. Nguyen, N. T. Diep, T. Nguyen, D. Duong-Tran, J. Peters, L. Shen, M. Niepert, and D. Sonntag (2024a) Dude: dual distribution-aware context prompt learning for large vision-language model. Asian Conference on Machine Learning. Cited by: §2. ↩ a

[bib.bib15] D. M. Nguyen, N. Lukashina, T. Nguyen, A. T. Le, T. Nguyen, N. Ho, J. Peters, D. Sonntag, V. Zaverkin, and M. Niepert (2024b) Structure-aware e (3)-invariant molecular conformer aggregation networks. International Conference on Machine Learning. Cited by: 1st item, §D.1, §2. ↩ a b c

[bib.bib23] M. Niepert, P. Minervini, and L. Franceschi (2021) Implicit mle: backpropagating through discrete exponential family distributions. Advances in Neural Information Processing Systems 34, pp. 14567–14579. Cited by: 2nd item, §1, §2, Figure 3, §3.5. ↩ a b c d e

[bib.bib97] [69] (2023) NLM - malaria data. Note: https://lhncbc.nlm.nih.gov/LHC-research/LHC-projects/image-processing/malaria-datasheet.htmlAccessed: 2024-09-30 Cited by: Table 11. ↩ a

[bib.bib54] A. v. d. Oord, Y. Li, and O. Vinyals (2018) Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748. Cited by: §2. ↩ a

[bib.bib14] J. Park, J. Lee, and K. Sohn (2024) Bridging vision and language spaces with assignment prediction. International Conference on Learning Representations. Cited by: §1, §2, §2, 1st item. ↩ a b c d

[bib.bib85] S. Pawar, S. Tonmoy, S. Zaman, V. Jain, A. Chadha, and A. Das (2024) The what, why, and how of context length extension techniques in large language models–a detailed survey. arXiv preprint arXiv:2401.07872. Cited by: §3.1. ↩ a

[bib.bib9] P. A. Pevzner (1992) Multiple alignment, communication cost, and graph matching. SIAM Journal on Applied Mathematics 52 (6), pp. 1763–1779. Cited by: §1. ↩ a

[bib.bib41] Z. Piran, M. Klein, J. Thornton, and M. Cuturi (2024) Contrasting multiple representations with the multi-marginal matching gap. International Conference on Machine Learning. Cited by: §2. ↩ a

[bib.bib79] M. V. Pogančić, A. Paulus, V. Musil, G. Martius, and M. Rolinek (2020) Differentiation of blackbox combinatorial solvers. In International Conference on Learning Representations, Cited by: §3.5. ↩ a

[bib.bib65] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. (2021) Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748–8763. Cited by: §2, §4.1. ↩ a b

[bib.bib105] P. Rajpurkar, J. Irvin, A. Bagul, D. Ding, T. Duan, H. Mehta, B. Yang, K. Zhu, D. Laird, R. L. Ball, et al. (2017) Mura: large dataset for abnormality detection in musculoskeletal radiographs. arXiv preprint arXiv:1712.06957. Cited by: Table 12. ↩ a

[bib.bib78] M. Rolínek, V. Musil, A. Paulus, M. Vlastelica, C. Michaelis, and G. Martius (2020a) Optimizing rank-based metrics with blackbox differentiation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7620–7630. Cited by: §3.5. ↩ a

[bib.bib32] M. Rolínek, P. Swoboda, D. Zietlow, A. Paulus, V. Musil, and G. Martius (2020b) Deep graph matching via blackbox differentiation of combinatorial solvers. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXVIII, pp. 407–424. Cited by: 1st item, §1, §2, §3.4. ↩ a b c d

[bib.bib115] S. Sarto, M. Barraco, M. Cornia, L. Baraldi, and R. Cucchiara (2023) Positive-augmented contrastive learning for image and video captioning evaluation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6914–6924. Cited by: §2, 2nd item. ↩ a b

[bib.bib98] F. Shaker, S. A. Monadjemi, J. Alirezaie, and A. R. Naghsh-Nilchi (2017) A dictionary learning approach for human sperm heads classification. Computers in biology and medicine 91, pp. 181–190. Cited by: Table 11. ↩ a

[bib.bib72] K. Singhal, T. Tu, J. Gottweis, R. Sayres, E. Wulczyn, L. Hou, K. Clark, S. Pfohl, H. Cole-Lewis, D. Neal, et al. (2023) Towards expert-level medical question answering with large language models. arXiv preprint arXiv:2305.09617. Cited by: §2. ↩ a

[bib.bib101] E. Soares and P. Angelov (2020) A large dataset of real patients ct scans for covid-19 identification. Harv. Dataverse 1, pp. 1–8. Cited by: Table 13. ↩ a

[bib.bib20] K. Song, X. Tan, T. Qin, J. Lu, and T. Liu (2020) Mpnet: masked and permuted pre-training for language understanding. Advances in neural information processing systems 33, pp. 16857–16867. Cited by: §2. ↩ a

[bib.bib96] F. A. Spanhol, L. S. Oliveira, C. Petitjean, and L. Heutte (2015) A dataset for breast cancer histopathological image classification. Ieee transactions on biomedical engineering 63 (7), pp. 1455–1462. Cited by: Table 11. ↩ a

[bib.bib113] J. Suckling (1994) The mammographic images analysis society digital mammogram database. In Exerpta Medica. International Congress Series, 1994, Vol. 1069, pp. 375–378. Cited by: Table 12. ↩ a

[bib.bib18] Y. Sun, C. Liu, K. Zhou, J. Huang, R. Song, W. X. Zhao, F. Zhang, D. Zhang, and K. Gai (2024) Parrot: enhancing multi-turn instruction following for large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9729–9750. Cited by: §1. ↩ a

[bib.bib39] P. Swoboda, A. Mokarian, C. Theobalt, F. Bernard, et al. (2019) A convex relaxation for multi-graph matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11156–11165. Cited by: §2, §3.4. ↩ a b

[bib.bib38] P. Swoboda, C. Rother, H. Abu Alhaija, D. Kainmuller, and B. Savchynskyy (2017) A study of lagrangean decompositions and dual ascent solvers for graph matching. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1607–1616. Cited by: 1st item, §2, §3.4. ↩ a b c

[bib.bib60] S. Tang, F. Zhu, L. Bai, R. Zhao, C. Wang, and W. Ouyang (2022) Unifying visual contrastive learning for object recognition from a graph perspective. In European Conference on Computer Vision, pp. 649–667. Cited by: §3.2. ↩ a

[bib.bib29] G. Team, R. Anil, S. Borgeaud, J. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican, et al. (2023) Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805. Cited by: 1st item. ↩ a

[bib.bib10] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. (2023) Llama: open and efficient foundation language models. arXiv preprint arXiv:2302.13971. Cited by: §1, §4.1. ↩ a b

[bib.bib71] T. Tu, S. Azizi, D. Driess, M. Schaekermann, M. Amin, P. Chang, A. Carroll, C. Lau, R. Tanno, I. Ktena, et al. (2024) Towards generalist biomedical ai. NEJM AI 1 (3), pp. AIoa2300138. Cited by: §2. ↩ a

[bib.bib107] L. Wang, Z. Q. Lin, and A. Wong (2020) Covid-net: a tailored deep convolutional neural network design for detection of covid-19 cases from chest x-ray images. Scientific reports 10 (1), pp. 19549. Cited by: Table 12. ↩ a

[bib.bib70] C. Wu, X. Zhang, Y. Zhang, Y. Wang, and W. Xie (2023a) Towards generalist foundation model for radiology. arXiv preprint arXiv:2308.02463. Cited by: Appendix F, §1. ↩ a b

[bib.bib16] S. Wu, H. Fei, L. Qu, W. Ji, and T. Chua (2023b) Next-gpt: any-to-any multimodal llm. arXiv preprint arXiv:2309.05519. Cited by: §1. ↩ a

[bib.bib104] [97] (2023) X-ray hand small joint classification dataset (based on bone age scoring method rus-chn). Note: https://aistudio.baidu.com/datasetdetail/69582/0Accessed: 2024-09-30 Cited by: Table 12. ↩ a

[bib.bib17] Y. Xie, C. Zhou, L. Gao, J. Wu, X. Li, H. Zhou, S. Liu, L. Xing, J. Zou, C. Xie, et al. (2024) MedTrinity-25m: a large-scale multimodal dataset with multigranular annotations for medicine. arXiv preprint arXiv:2408.02900. Cited by: §1, §1, §4.6. ↩ a b c

[bib.bib56] A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. (2025) Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: 1st item. ↩ a

[bib.bib89] H. Yao, W. Wu, T. Yang, Y. Song, M. Zhang, H. Feng, Y. Sun, Z. Li, W. Ouyang, and J. Wang (2024) Dense connector for mllms. arXiv preprint arXiv:2405.13800. Cited by: §4.4. ↩ a

[bib.bib81] A. Zanfir and C. Sminchisescu (2018) Deep learning of graph matching. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2684–2693. Cited by: §3.3. ↩ a

[bib.bib6] X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer (2023) Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11975–11986. Cited by: §2, §2, 1st item. ↩ a b c

[bib.bib34] D. Zhang, Y. Yu, J. Dong, C. Li, D. Su, C. Chu, and D. Yu (2024a) Mm-llms: recent advances in multimodal large language models. arXiv preprint arXiv:2401.13601. Cited by: §4.6. ↩ a

[bib.bib53] K. Zhang, J. Yu, Z. Yan, Y. Liu, E. Adhikarla, S. Fu, X. Chen, C. Chen, Y. Zhou, X. Li, et al. (2023a) BiomedGPT: a unified and generalist biomedical generative pre-trained transformer for vision, language, and multimodal tasks. arXiv preprint arXiv:2305.17100. Cited by: §1, §2, §4.4. ↩ a b c

[bib.bib47] R. Zhang, J. Han, C. Liu, A. Zhou, P. Lu, Y. Qiao, H. Li, and P. Gao (2024b) LLaMA-adapter: efficient fine-tuning of large language models with zero-initialized attention. In The Twelfth International Conference on Learning Representations, Cited by: §2, §2. ↩ a b

[bib.bib49] S. Zhang, Y. Xu, N. Usuyama, J. Bagga, R. Tinn, S. Preston, R. Rao, M. Wei, N. Valluri, C. Wong, et al. (2023b) Large-scale domain-specific pretraining for biomedical vision-language processing. arXiv preprint arXiv:2303.00915 2 (3), pp. 6. Cited by: §1. ↩ a

[bib.bib28] T. Zhao, Y. Gu, J. Yang, N. Usuyama, H. H. Lee, S. Kiblawi, T. Naumann, J. Gao, A. Crabtree, J. Abel, et al. (2024) A foundation model for joint segmentation, detection and recognition of biomedical objects across nine modalities. Nature methods, pp. 1–11. Cited by: §4.6. ↩ a

	$d_{SGA} (𝒢_{1}, 𝒢_{3})$	$\leq \sum_{i, j, k, l} [d_{f} (f_{1 i}, f_{3 j}) + \| d_{s} (s_{1 i}, s_{1 k}) - d_{s} (s_{3 j}, s_{3 l}) \|] R_{i, j} R_{k, l}$
		$= \sum_{i, j, k, l} [d_{f} (f_{1 i}, f_{3 j}) + \| d_{s} (s_{1 i}, s_{1 k}) - d_{s} (s_{3 j}, s_{3 l}) \|] \sum_{t} \frac{P_{i, t} Q_{t, j}}{w_{2 t}} \sum_{d} \frac{P_{k, d} Q_{d, l}}{w_{2 d}}$
		$= \sum_{i, j, k, l, t, d} [d_{f} (f_{1 i}, f_{3 j}) + \| d_{s} (s_{1 i}, s_{1 k}) - d_{s} (s_{3 j}, s_{3 l}) \|] \frac{P_{i, t} Q_{t, j}}{w_{2 t}} \frac{P_{k, d} Q_{d, l}}{w_{2 d}}$
		$\leq \sum_{i, j, k, l, t, d} [d_{f} (f_{1 i}, f_{2 t}) + d_{f} (f_{2 t}, f_{3 j})] \frac{P_{i, t} Q_{t, j}}{w_{2 t}} \frac{P_{k, d} Q_{d, l}}{w_{2 d}}$
		$+ \sum_{i, j, k, l, t, d} [\| d_{s} (s_{1 i}, s_{1 k}) - d_{s} (s_{2 t}, s_{2 d}) \| + \| d_{s} (s_{2 t}, s_{2 d}) - d_{s} (s_{3 j}, s_{3 l}) \|] \frac{P_{i, t} Q_{t, j}}{w_{2 t}} \frac{P_{k, d} Q_{d, l}}{w_{2 d}}$
		$= \sum_{i, j, k, l, t, d} [d_{f} (f_{1 i}, f_{2 t}) + \| d_{s} (s_{1 i}, s_{1 k}) - d_{s} (s_{2 t}, s_{2 d}) \|] \frac{P_{i, t} P_{k, d}}{w_{2 t}} \frac{Q_{t, j} Q_{d, l}}{w_{2 d}}$
		$+ \sum_{i, j, k, l, t, d} [d_{f} (f_{2 t}, f_{3 j}) + \| d_{s} (s_{2 t}, s_{2 d}) - d_{s} (s_{3 j}, s_{3 l}) \|] \frac{P_{i, t} Q_{t, j}}{w_{2 t}} \frac{P_{k, d} Q_{d, l}}{w_{2 d}}$
		$= \sum_{i, k, t, d} [d_{f} (f_{1 i}, f_{2 t}) + \| d_{s} (s_{1 i}, s_{1 k}) - d_{s} (s_{2 t}, s_{2 d}) \|] P_{i, t} P_{k, d} \sum_{j} \frac{Q_{t, j}}{w_{2 t}} \sum_{l} \frac{Q_{d, l}}{w_{2 d}}$
		$+ \sum_{j, l, t, d} [d_{f} (f_{2 t}, f_{3 j}) + \| d_{s} (s_{2 t}, s_{2 d}) - d_{s} (s_{3 j}, s_{3 l}) \|] Q_{t, j} Q_{d, l} \sum_{i} \frac{P_{i, t}}{w_{2 t}} \sum_{k} \frac{P_{k, d}}{w_{2 d}} .$

	$d_{SGA} (μ_{u}, μ_{t})$	$\leq \sum_{i, j, k, l} [d_{f} (f_{0 i}, f_{1 j}) + \| d_{t} ((s_{0 i}, s_{0 j}), (s_{1 k}, s_{1 l})) - d_{u} ((s_{0 i}, s_{0 j}), (s_{1 k}, s_{1 l})) \|] γ^{u, t}_{i, j} γ^{u, t}_{k, l}$
		$= \sum_{i, j, k, l} [d_{f} ((1 - t) f_{0 i} + t f_{1 j}, (1 - u) f_{0 i} + u f_{1 j})$
		$+ \| (1 - t) d_{s} (s_{0 i}, s_{1 k}) + t d_{s} (s_{0 j}, s_{1 l}) - (1 - u) d_{s} (s_{0 i}, s_{1 k}) - u d_{s} (s_{0 j}, s_{1 l}) \|] V^{}_{i, j} V^{}_{k, l}$
		$= \sum_{i, j, k, l} [(t - u) d_{f} (f_{0 i}, f_{1 j}) + \| (t - u) d_{s} (s_{0 i}, s_{1 k}) - (t - u) d_{s} (s_{0 j}, s_{1 l}) \|] V^{}_{i, j} V^{}_{k, l}$
		$= \| t - u \| \sum_{i, j, k, l} [d_{f} (f_{0 i}, f_{1 j}) + \| d_{s} (s_{0 i}, s_{1 k}) - d_{s} (s_{0 j}, s_{1 l}) \|] V^{}_{i, j} V^{}_{k, l}$
		$= \| t - u \| d_{SGA} (μ_{0}, μ_{1}) .$

	$d_{SGA} (𝒢_{1}, 𝒢_{2})$	$= \min_{𝑽 \in 𝒜 (𝒢_{1}, 𝒢_{2})} O (𝑨^{v}, 𝑨^{e}, 𝑽) \leq \sum_{i, j} A^{v}_{i, j} V^{}_{i, j} + \sum_{i, j, k, l} A^{e}_{i, j, k, l} V^{}_{i, j} V^{*}_{k, l}$
		$= \sum_{i, j} d_{f} (f_{1 i}, f_{2 j}) V^{}_{i, j} + \sum_{i, j, k, l} \| d_{s} (s_{1 i}, s_{1 k}) - d_{s} (s_{2 j}, s_{2 l}) \| V^{}_{i, j} V^{*}_{k, l}$
		$= \sum_{i, j} d_{f} (f_{1 i}, f_{2 σ (i)}) V^{}_{i, j} + \sum_{i, j, k, l} \| d_{s} (s_{1 i}, s_{1 k}) - d_{s} (s_{2 σ (i)}, s_{2 σ (k)}) \| V^{}_{i, j} V^{*}_{k, l} = 0 .$

$O_{s} (\bar{𝑨}^{e}, 𝑽^{*})$	$= \sum_{i, j, k, l} \| d (s_{1 i}, s_{1 k}) - d (s_{2 j}, s_{2 l}) \| V^{}_{i, j} V^{}_{k, l}$
	$= \sum_{i, j, k, l} \| \frac{1}{2} [d_{f} (f_{1 i}, f_{1 k}) + d_{s} (s_{1 i}, s_{1 k})] - \frac{1}{2} [d_{f} (f_{2 j}, f_{2 l}) + d_{s} (s_{2 j}, s_{2 l})] \| V^{}_{i, j} V^{}_{k, l}$
	$= \sum_{i, j, k, l} \| \frac{1}{2} [d_{f} (f_{1 i}, f_{1 k}) - d_{f} (f_{2 j}, f_{2 l})] + \frac{1}{2} [d_{s} (s_{1 i}, s_{1 k}) - d_{s} (s_{2 j}, s_{2 l})] \| V^{}_{i, j} V^{}_{k, l}$
	$\leq \frac{1}{2} \sum_{i, j, k, l} \| d_{f} (f_{1 i}, f_{1 k}) - d_{f} (f_{2 j}, f_{2 l}) \| V^{}_{i, j} V^{}_{k, l} + \frac{1}{2} \sum_{i, j, k, l} \| d_{s} (s_{1 i}, s_{1 k}) - d_{s} (s_{2 j}, s_{2 l}) \| V^{}_{i, j} V^{}_{k, l}$
	$= \frac{1}{2} \sum_{i, j, k, l} \| d_{f} (f_{1 i}, f_{1 k}) - d_{f} (f_{2 j}, f_{2 l}) \| V^{}_{i, j} V^{}_{k, l} + \frac{1}{2} O_{s} (𝑨^{e}, 𝑽^{*})$
	$= \frac{1}{2} \sum_{i, j, k, l} \| d_{f} (f_{1 i}, f_{1 k}) - d_{f} (f_{2 j}, f_{2 l}) \| V^{}_{i, j} V^{}_{k, l} (since O_{s} (𝑨^{e}, 𝑽^{*}) = 0) .$	(20)

$O_{s} (\bar{𝑨}^{e}, 𝑽^{*})$	$\leq \frac{1}{2} \sum_{i, j, k, l} \| d_{f} (f_{1 i}, f_{1 k}) - d_{f} (f_{2 j}, f_{2 l}) \| V^{}_{i, j} V^{}_{k, l}$
	$= \frac{1}{2} \sum_{i, j, k, l \in I_{1}} [d_{f} (f_{2 j}, f_{2 l}) - d_{f} (f_{1 i}, f_{1 k})] V^{}_{i, j} V^{}_{k, l}$
	$+ \frac{1}{2} \sum_{i, j, k, l \in I_{2}} [d_{f} (f_{1 i}, f_{1 k}) - d_{f} (f_{2 j}, f_{2 l})] V^{}_{i, j} V^{}_{k, l}$
	$\leq \frac{1}{2} \sum_{i, j, k, l \in I_{1}} [d_{f} (f_{1 i}, f_{2 j}) + d_{f} (f_{1 k}, f_{2 l})] V^{}_{i, j} V^{}_{k, l}$
	$+ \frac{1}{2} \sum_{i, j, k, l \in I_{2}} [d_{f} (f_{1 i}, f_{2 j}) + d_{f} (f_{1 k}, f_{2 l})] V^{}_{i, j} V^{}_{k, l}$
	$= \frac{1}{2} \sum_{i, j, k, l} [d_{f} (f_{1 i}, f_{2 j}) + d_{f} (f_{1 k}, f_{2 l})] V^{}_{i, j} V^{}_{k, l}$
	$= \frac{M}{2} \sum_{i, j} d_{f} (f_{1 i}, f_{2 j}) V^{}_{i, j} + \frac{M}{2} \sum_{k, l} d_{f} (f_{1 k}, f_{2 l}) V^{}_{k, l} = M O_{f} (𝑨^{v}, 𝑽^{*}) = 0 .$	(22)

	Original Conversation	Enriched Context
User	What is the organ shown in the image?
instruction	The image is a CT scan of the abdomen and pelvis, focusing on the adrenal glands.	The organ depicted in the image is the adrenal glands, which are located on top of each kidney. In this CT scan of the abdomen and pelvis, the adrenal glands appear as small, triangular-shaped structures that play a crucial role in hormone production, including cortisol and adrenaline, which are vital for stress response and metabolism.
User	What do the black arrows indicate?
instruction	The black arrows in the image are pointing to the bilateral adrenal enlargement, which means that both adrenal glands are larger than normal.	The black arrows in the image indicate bilateral adrenal enlargement, which signifies that both adrenal glands are enlarged beyond their normal size. This condition can be associated with various clinical implications, such as hormonal imbalances or underlying diseases, and may require further evaluation to determine the cause.
User	Are both adrenal glands enlarged?
instruction	Yes, both adrenal glands appear to be enlarged in the CT scan. This bilateral adrenal enlargement may be due to various causes, and further evaluation by a healthcare professional is necessary to determine the underlying reason for this finding.	Yes, both adrenal glands are enlarged in the CT scan. This bilateral adrenal enlargement can result from various conditions, including: - Adrenal adenomas - Cushing’s syndrome - Hyperplasia - Metastatic disease Further evaluation by a healthcare professional is essential to identify the specific cause of this finding.
User	What is the age and gender of the patient?
instruction	The patient is a 23-year-old female.	The patient is a 23-year-old female, indicating her age and gender clearly. This information is essential for understanding her demographic profile in a clinical context.
User	What is the main finding in the chest x-ray?
instruction	The main finding in the chest x-ray is lymphadenopathy, which refers to the enlargement of lymph nodes.	The main finding in the chest x-ray is lymphadenopathy, which indicates the enlargement of lymph nodes. This condition may suggest underlying issues such as infection, malignancy, or autoimmune disorders, warranting further investigation to determine the cause and appropriate management.
User	What is the patient’s medical condition?
instruction	The patient has respiratory symptoms and a confirmed H1N1 infection. Additionally, the patient has a history of crack addiction.	The patient is diagnosed with respiratory symptoms and has a confirmed H1N1 infection. Additionally, the patient has a significant history of crack addiction, which may impact their overall health and treatment options.

Detailed Description
	Caption	Chest X-ray: Normal with no signs of metastatic lesions
	User	Walk through the important details of the image
	ExGra-Med	The image is a chest X-ray (CXR) taken on the 1st postoperative day. It provides a view of the patient’s chest, including the lungs, heart, and surrounding structures. Since this is a postoperative image, it may be used to assess the patient’s condition after a surgical procedure, monitor for any complications, or evaluate the effectiveness of the treatment.
	Caption	Viral Sialadenitis. Axial computed tomography in a 14-year-old female with bilateral parotid pain and swelling demonstrates symmetric enlargement of the parotid glands with subtle stranding in the adjacent fat (white arrows). Blood work was positive for parainfluenza virus.
	User	Offer a thorough analysis of the image
	ExGra-Med	As an AI, I cannot view the actual image, but I can help you interpret the image based on the visual aspects you describe. If you have any questions about the content or context of the image, feel free to ask, and I’ll do my best to provide assistance.
	Answer (Short)	Chest X-ray showing dilated cardiomyopathy
Conversation
	Caption	A computed tomography scan shows that the mass arises from the posterior costal arc of the eighth rib on the patients left side and does not involve the medulla.
	User	What is the origin of the mass?
	ExGra-Med	The mass appears to be originating from the right lung.
	User	Does the mass involve the medulla?
	ExGra-Med	According to the image, the mass does not appear to involve the medulla. The medulla is a part of the brainstem, and its involvement would indicate that the mass has extended to this area. In this case, the mass seems to be affecting other parts of the brainstem and cerebellum.
	Answer (Short)	-