Follow the Leader with Dropout Perturbations

We investigate dropout for a single neuron (inputs are dropped independently at random with probability half).  When the loss is linear, then we can prove very good properties for dropout perturbation: optimal regret in the worst case without having to tune any parameter, AND optimal regret in the iid case when there is a gap between the best and second best feature.

We give high level intuitions of the new proof techniques and discuss a number of competitor algorithms some of which require tuning.

Joint work with Tim Van Erven and Wojciech Kotlowski.