Results 1291 - 1300 of 23832

It is well-established that gradient descent (GD) successfully trains deep neural networks that generalize, yet the precise mechanisms remain an active area of research. This presentation delves into the training dynamics of GD, examining how hyperparameters shape the optimization path. We will discuss two key phenomena, implicit regularization and training instability and present research showing how they act as a crucial bias, steering the learning process toward generalizable solutions.

The vast majority of training for supervised learning algorithms today is done via optimization, due to the maturity and speed of efficient gradient-based algorithms for implementation. A Bayesian approach to machine learning has been largely abandoned due to the failure of traditional Markov Chain Monte Carlo (MCMC) algorithms to sample from complex posterior distributions in a reasonable amount of time.

At the same time, in recent years diffusion models have proven to be a remarkably effective generative algorithm for sampling from complex and high dimensional target densities in examples such as Stable Diffusion and DALL-E. Additionally, many of the best MCMC algorithms we have today make use of auxiliary random variables, creating joint distributions between “target” variables we want to be sampling and “auxiliary” variables that are designed by the practitioner to facilitate sampling the target variable. Examples include Hamiltonian Monte Carlo with a “momentum” variable, simulated tempering with a “temperature” variable, and proximal sampling.

Combining these two ideas, diffusion models can be viewed as a mathematical object defining a series of auxiliary random variables coupled with the target variable that can facilitate sampling. Under certain conditions, we can demonstrate a mixture representation of the target density where the mixture components are well suited to traditional MCMC sampling. This structure has been named a “Log-Concave Coupling” in past research by this author. Under such a structure, a sample of the auxiliary random variable from its marginal density, followed by a sample of the target random variable from its conditional density can be accomplished efficiently by traditional MCMC methods such as Langevin Diffusion. This produces an equivalent sample of the target variable from its original target density, while only having to sample from log-concave densities in each step.

The unprecedented performance levels and energy needs of AI and ML applications require new approaches to both technology choices and to the related design methods.
Diversity and heterogeneous integration will play a major role in the realization of future systems. Nevertheless, emerging technologies can be leveraged only if supported by the corresponding design tools and flows, that may depart significantly from those used for CMOS.
While various technologies and computing paradigms have different goals and constraints, some design solutions may leverage the same fundamental properties of digital design. This brings logic synthesis and optimization again at the forefront of research as a main enabler of innovative and disruptive digital system design.

Designing high performance and energy efficient AI accelerators requires significant engineering effort, and as the rapidly evolving field of machine learning develops new models, the current approach of designing ad hoc accelerators does not scale. In this talk, I will present our ongoing research on a high-level synthesis (HLS)-based framework for design space exploration and generation of hardware accelerators for AI. Given architectural parameters, such as datatype, scaling granularity, compute parallelism and memory sizes, the framework generates a performant fabrication-ready accelerator. Accelerators generated through this framework have been taped out in several chips, targeting various workloads including convolutional neural networks and transformer networks. In this talk, I will present the generator framework, and show how we can also use it as a benchmarking tool for designs leveraging emerging technologies.