Vision Language Model Alignment in TRL ⚡️

Introduction

Vision Language Models (VLMs) are getting stronger, but aligning them to human preferences still matters. In TRL, we already showed how to post-train VLMs with Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO). This time, we’re going further.

tl;dr Here’s what’s new in TRL:

• Mixed Preference Optimization (MPO)
• Group Relative Policy Optimization (GRPO)
• Group Sequence Policy Optimization (GSPO) (a variant of GRPO)

These go beyond pairwise DPO, extracting richer signals from preference data and scaling better with modern VLMs.

We’ve also extended existing methods to support VLMs:

• Reinforce Leave One Out (RLOO)
• Online Direct Preference Optimization (Online DPO)

This enables more efficient and scalable multimodal alignment.

Finally:

• Native Supervised Fine-tuning support for Vision Language Models
• Training scripts and demo notebooks to help you get started quickly

Table of Contents

Alignment for Vision Language Models

Traditionally, you would take a base model, apply SFT to follow instructions, and then apply DPO to align it to preferential data. Previously, we adapted this approach to Vision Language Models (VLMs) and validated it on IDEFICS2, showing improvement in model responses.

DPO works by optimizing preferences between pairs of model responses using a contrastive loss: you have a chosen and a rejected answer and you optimize your preferences based on what you want and don’t want.

But in the last year, new multimodal alignment methods have gained popularity, GRPO and MPO, that can push VLM performance even further. At the end of the blog post you can find a table that showcases the differences between model responses.

Mixed Preference Optimization (MPO)

Aligning multimodal models with SFT to do reasoning tasks falls short due to distribution shift. Meanwhile, models aligned with DPO fail to generate coherent rationales and might generate repetitive responses. To address this, there’s a new technique called Mixed Preference Optimization (MPO) specifically made for multimodal models. This method is essentially an extension of DPO with multiple losses: preference loss from DPO (sigmoid), quality loss from Binary Classifier Optimization (BCO), and generation loss from SFT. According to the paper, simply switching to this combined loss results in 6.2 pts improvement in MathVista!

Since this is only modifying the loss, we added combined loss support to TRL’s DPOTrainer class. To use it, you can initialize the DPOConfig as follows:

mpo_config = DPOConfig(
    loss_type=["sigmoid", "bco_pair", "sft"], 
    loss_weights=[0.8, 0.2, 1.0], 
)

Then initialize the DPOTrainer:

mpo_trainer = DPOTrainer(
    model=model_id,
    args=mpo_config,
    processing_class=tokenizer,
    train_dataset=dataset,
)
mpo_trainer.train()

And that’s it! If you want to explore further, you can find a complete notebook example here.

Multimodal Group Relative Policy Optimization (GRPO)

Group Relative Policy Optimization (GRPO) is a cutting-edge alignment method initially introduced in DeepSeek Math paper and later integrated to DeepSeek R1, the groundbreaking LLM. It’s an addition to PPO where the policy updates are done over groups (batches of trajectories that represent how a dialogue rolls out). This feature makes it more robust to reward noise, as the noise averages out within groups. Since the model learns broader sense of a good response rather than singular high reward samples, this method also makes the model highly performant.

In TRL, we now introduce GRPO support for vision language models. We will not provide a full training script example, as you can find it in the notebook. Instead, we’ll focus on highlighting the key component and concepts.

To make the training script work effectively, we need to validate that the format of the answer is correct and that the solution itself is close to the completed parts, so we write two reward functions. In order to really see improvements in the latter reward, you would need a rather maximalist setup, where you have relatively larger models, a lot of generations, and a high-quality, diverse dataset.

import re
from math_verify import LatexExtractionConfig, parse, verify

def format_reward(completions, **kwargs):
    """Reward function that checks if the completion has a specific format."""
    pattern = r"^.*?\s*.*?$"
    matches = [re.match(pattern, content) for content in completions]
    rewards_list = [1.0 if match else 0.0 for match in matches]
    rewards = [1.0 if match else 0.0 for match in matches]
    print(completions)
    print(rewards)
    return rewards

def accuracy_reward(completions, **kwargs):
    """Reward function that checks if the completion is the same as the ground truth."""
    solutions = kwargs['solution']
    completion_contents = [completion[0]["content"] for completion in completions]
    rewards = []
    for content, solution in zip(completion_contents, solutions):
        gold_parsed = parse(solution, extraction_mode="first_match", extraction_config=[LatexExtractionConfig()])
        answer_parsed = parse(content, extraction_mode="first_match", extraction_config=[LatexExtractionConfig()])
        if len(gold_parsed) != 0:
            try:
                rewards.append(float(verify(answer_parsed, gold_parsed)))
            except Exception:
                rewards.append(0.0)
        else:
            rewards.append(1.0)
    return rewards

Then, you can initialize GRPOConfig and GRPOTrainer, pass in the reward functions we defined above and call train() to start training.

from trl import GRPOConfig, GRPOTrainer

training_args = GRPOConfig(
    learning_rate=1e-5,
    max_prompt_length=None,
    ... 
)
trainer = GRPOTrainer(
    model=model_id,
    reward_funcs=[format_reward, accuracy_reward],
    args=training_args,
    train_dataset=train_dataset,
)
trainer.train()

Explore the full notebook example here.

Group Sequence Policy Optimization (GSPO)

Group Sequence Policy Optimization (GSPO) is a RL alignment algorithm recently released by Qwen that overcomes some limitations of GRPO. It achieves a more stable training computing importance sampling weights at the sequence level instead of per-token. Its benefits are more relevant in MoE style models.

Latest TRL also introduces support for GSPO and since it’s a variant of GRPO’s loss, it comes with multimodal support. To create the trainer, the process is the same as with GRPO, but adding the following extra params (values are extracted from the paper).

from trl import GRPOConfig

training_args = GRPOConfig(
        ...
    importance_sampling_level="sequence",
    epsilon=3e-4,
    epsilon_high=4e-4,
    beta=0.0,
    loss_type="grpo",
    steps_per_generation=1,
    steps_per_generation=4
)

Comparison

Here’s a table summarizing model outputs for Qwen2.5VL-3B fine-tuned with the techniques discussed above. Note that we’ve done minimal runs on dataset subsets, and the models were fine-tuned on different datasets, so the comparison is made for vibe-check.

Further Extensions for VLMs

In addition to MPO, GRPO, and GSPO, TRL now supports Reinforce Leave One Out (RLOO) and Online Direct Preference Optimization (Online DPO) for Vision Language Models (VLMs), enabling alignment on multimodal datasets.

Reinforce Leave One Out (RLOO)

RLOO now supports common VLMs. You can find a complete training example in the rloo_visual_math.py script.

Here’s how to set up a RLOOTrainer:

trainer = RLOOTrainer(
    model=model_name,
    args=training_args,
    reward_funcs=[think_format_reward, accuracy_reward],
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
)

trainer.train()

And to launch training directly from the example script:

CUDA_VISIBLE_DEVICES=1,2 python3 examples/rloo_visual_math/rloo_visual_math.py --model_name_or_path Qwen/Qwen2.5-VL-3B-Instruct

Online Direct Preference Optimization (Online DPO)

Online DPO also supports VLMs. See the online_dpo_visual_math.py script for a simple example.

To run the example script (vLLM integration will be discussed later):

CUDA_VISIBLE_DEVICES=1,2 python3 examples/online_dpo_visual_math/online_dpo_visual_math.py --model_name_or_path Qwen/Qwen2.5-VL-3B-Instruct --use_vllm --vllm_mode server

These scripts are ready-to-run for VLM training; full parameter tuning is documented in TRL: Online DPO trainer RLOO Trainer.

Native Supervised Fine-tuning Support

Previously, SFTTrainer was partially supporting vision language models. This was primarily due to many differences across VLM implementations in transformers API. With the standardization of the transformers API, we have shipped a full support for vision language models. You can simply initialize SFTTrainer with a VLM.

from trl import SFTConfig, SFTTrainer
from datasets import load_dataset

trainer = SFTTrainer(
    model="Qwen/Qwen2.5-VL-3B-Instruct",
    args=SFTConfig(max_length=None), 
    train_dataset=load_dataset("trl-lib/llava-instruct-mix", split="train"),
)
trainer.train()

To train a VLM, you need to provide a dataset with an additional images column containing the images to be processed. You can take a look at Dataset Formats — Vision Datasets for more information on how it should look like. A good example is LLaVA Instruct Mix.

We also have a sft_visual_chat.py script that works out of the box for transformers vision language models.

vLLM Integration in TRL

vLLM is integrated in TRL to support online alignment methods where you need to generate samples during training. Running the example scripts like the following enables vLLM:

CUDA_VISIBLE_DEVICES=1,2 python3 examples/grpo_visual_math/grpo_visual_math.py --model_name_or_path Qwen/Qwen2.5-VL-3B-Instruct --use_vllm --vllm_mode colocate

There’s mainly two modes: colocate and server. colocate runs vLLM in the same process as the training loop, sharing the same GPU between training and generation, creating a vLLM LLM instance inside the GRPOTrainer. Meanwhile server requires you to serve vLLM separately in a different process where you can hit the server. You can start this server with the command:

trl vllm-serve --model Qwen/Qwen2.5-VL-3B-Instruct --tensor-parallel-size 1 

Then you can run the script as follows.

CUDA_VISIBLE_DEVICES=1,2 python3 examples/grpo_visual_math/grpo_visual_math.py --model_name_or_path Qwen/Qwen2.5-VL-3B-Instruct --use_vllm --vllm_mode server

One more tip: we have added support for using vLLM with transformers backend in TRL. You can enable it when running a script with colocate or when serving the model by passing the --vllm_model_impl transformers flag.

You can read more about vLLM integration in TRL here.

Useful Resources

Below, you can find a compilation of resources to explore the alignment of VLMs in detail. Enjoy!