Scale Learning and Reasoning Across Heterogeneous Gradients and Semantics

In federated or decentralized environments, tasks clients solve can exhibit different gradients (along with gradient statistics), or even completely different problem semantics (e.g., solving a biology problem versus a CS problem). These forms of heterogeneity pose significant challenges to training and inference. For instance, to realize the benefits of adaptive optimization, it is critical to address heterogeneous preconditioners accumulated on the client side. I first introduce an optimizer Bi^2Clip that approximates adaptive methods without maintaining preconditioners. It leverages coordinate-wise bi-directional clipping, which also helps mitigate issues such as heavy-tailed gradient noise. I then discuss an asynchronous framework based on it that can tolerate heterogeneous hardware with improved dependence on the staleness. Lastly, we shift from gradient/parameter space to semantic space, and present a federation-over-text (FoT) framework designed for heterogeneous tasks and domains. In FoT, instead of transmitting gradients or models, clients iteratively share metacognitive summaries of their local reasoning and planning processes to build a library of reusable insights. I discuss its early applications on math problems solving and ML research insight discovery. Overall, these results show that carefully treating heterogeneity can result in meaningful gains in both final performance and learning/inference efficiency.