Abstract

We start with a brief overview of the current approach of federated learning (FL) and differential privacy (DP) for private training of small LMs, empowering production on-device decoding LMs in mobile keyboard applications. Recent technical advances including new BLT-DP-FTRL algorithm offers strong privacy-utility trade-offs and ease-of-use in deployment, the SI-CIFG model architecture for efficient on-device training and compatibility with DP, and synthetic data from LLMs to improve (public) pre-training. The dedication to privacy-preserving learning to improve small LMs has not only delivered substantial user benefits, but has also helped improve LLMs in mobile typing applications, bridged by synthetic data. Next, we share our exploration over the past few years on generating and using synthetic data to improve LMs. We focus on approaches adhering to the privacy principles of both data minimization and data anonymization, and show how they are making a real-world impact in small and large models. We discuss synthetic data usage in complicated training paradigms and algorithms to improve quality and efficiency of synthetic data generation. Finally, we discuss implications of recent trends in federated learning, open problems and a preliminary study on personalization.

Video Recording