Stochastic Rounding and its Regularization Effects

Motivated by the resurgence of stochastic rounding, we consider stochastic nearness rounding of tall and thin real matrices. We provide theoretical and empirical evidence showing that, with high probability, the smallest singular value of a stochastically rounded matrix is bounded away from zero -- regardless of how close the original matrix was to being rank-deficient and even if it were rank-deficient. In other words, stochastic rounding implicitly regularizes tall-and-thin matrices so that the rounded version has full rank. We will briefly discuss the implications of such results for solving regression problems.