Personalized Federated Training of Diffusion Models with Privacy Guarantees

Diffusion models produce high-fidelity samples and have recently become the de facto approach for synthetic image generation. However, prior work shows that these models exhibit strong vulnerability to privacy attacks, including reconstruction and membership inference (e.g., Carlini et al.), which makes adoption difficult in sensitive domains such as healthcare. Unfortunately, existing approaches that apply differential privacy during training often fail to preserve the high fidelity that makes diffusion models effective. In this talk I will present a new approach for training diffusion models in the federated setting, where clients hold non-IID data and seek formal privacy guarantees. The key idea in our approach is personalization, which helps alleviate the tension between privacy and utility in federated learning. Our method exploits the coarse-to-fine refinement structure that characterizes diffusion models: a shared diffusion model learns the coarse structure that appears across clients, while client-specific models perform the finer refinements that encode client-level information. This design lets clients benefit from collaboration while preventing the shared model from reproducing any individual client’s data, since it only observes noisy privatized versions of each client’s data. The method provides formal local differential privacy guarantees for each client while empirically preserving the high fidelity of diffusion models, which allows each client to release their personalized model publicly without compromising the privacy of other clients. We also show in a toy Gaussian mixture model that collaboration in this framework improves sample quality relative to private non-collaborative training. Extensive experiments on CIFAR-10, Colorized MNIST, and CelebA support these results: the framework generates high-fidelity samples, improves performance on minority and underrepresented classes, and maintains strong protection against membership inference, memorization, and reconstruction attacks.

The talk is based on joint work with Bingqing Jiang, A F M Mahfuzul Kabir, Weitong Zhang, Difan Zou, Lingxiao Wang and will appear in CVPR 2026.