Multimodal AI training can demand enormous model updates, turning every federated learning round into a heavy exchange of data. NVIDIA FLARE addresses that challenge with a framework that supports both lightweight parameter-efficient updates and full-model communication.
At the center of this approach is FedUMM, a federated training system for unified multimodal models co-developed by William & Mary and NVIDIA. Its key move is simple but powerful: exchange only lightweight LoRA adapters while keeping the BLIP backbone frozen.
A Smaller Update Can Change the Entire Training Workflow
FedUMM focuses on unified multimodal models, where training involves more than one form of data or input. Instead of sending the full model during each round, the system exchanges LoRA adapters, which carry the trainable updates while the BLIP backbone remains frozen.
That design cuts the amount of information each client must communicate. In experiments, FedUMM reduced per-client communication from 28.6 GB to 0.094 GB per round.
The difference is enormous. A client that once needed to communicate 28.6 GB per round can instead send and receive 0.094 GB, while FedUMM keeps the federated training process centered on lightweight adapter exchange.
Why does that matter? Federated learning depends on communication between participating clients and a shared training process, so the size of each update becomes a central part of the workflow. FedUMM targets that pressure by separating the frozen model backbone from the smaller pieces that carry training changes.
NVIDIA FLARE Handles More Than One Communication Pattern
NVIDIA FLARE is an open source, extensible Python SDK and framework for federated learning and collaborative computing. Its design supports both parameter-efficient communication, such as adapter-based updates, and full-model communication patterns.
That flexibility gives federated AI workflows room to match the communication method to the training task. FedUMM demonstrates the parameter-efficient path through LoRA adapters and a frozen BLIP backbone, while NVIDIA FLARE also supports workflows that communicate full models.
The framework includes several features aimed at network and memory constraints:
- Large-object externalization: NVIDIA FLARE can replace large objects in a message with lightweight references and transfer the underlying data separately.
- Tensor streaming: The framework supports streaming tensors as part of its approach to handling large model-related data.
- Disk-backed aggregation: NVIDIA FLARE can use disk-backed aggregation to address memory constraints during federated workflows.
These capabilities focus on the practical burden of moving and handling large objects. Instead of forcing every large object to travel inside a message, NVIDIA FLARE can use a lightweight reference and transfer the underlying data separately.
FedUMM Puts Federated Multimodal Training in Focus
FedUMM connects the communication strategy of parameter-efficient training with the needs of unified multimodal models. Its use of LoRA adapters over a frozen BLIP backbone creates a clear division between the model structure that stays in place and the updates that move between clients.
That division also gives the work a measurable result. The reduction from 28.6 GB to 0.094 GB per client per round shows how adapter-based communication can reduce the amount of data exchanged during federated training experiments.
FedUMM is supported by the NVIDIA Academic Grant Program, bringing NVIDIA support to the project from both the collaboration and research sides. The work was co-developed by William & Mary and NVIDIA.
The project’s authors are Ziyue Xu, Holger Roth, Zhihong Zhang, and Peter Cnudde. FedUMM received an Outstanding Student Paper Award at the FL@FM workshop at TheWebConf 2026, adding recognition to a project focused on one of federated multimodal AI’s central challenges: managing model updates without turning communication into the bottleneck.
A Broader Route for Federated AI Systems
NVIDIA FLARE’s combination of communication options and data-handling features points to a broader workflow for federated learning. Parameter-efficient methods can exchange lightweight adapters, while full-model patterns remain available when a training process requires them.
Large-object externalization, tensor streaming, and disk-backed aggregation extend that workflow beyond the update format itself. They address the network and memory constraints that arise when federated systems handle large objects and tensors.
FedUMM shows what this can look like for unified multimodal models: keep the BLIP backbone frozen, exchange LoRA adapters, and reduce per-client communication to 0.094 GB per round in experiments. NVIDIA FLARE provides the open source, extensible Python framework that supports this kind of federated learning and collaborative computing.
As federated multimodal AI develops, the size and movement of model updates will remain central design questions. FedUMM offers one direct answer, while NVIDIA FLARE supplies the communication patterns and infrastructure needed to explore both lightweight and full-model approaches.
