We introduce a method of feature selection for non-linear Support Vector Machines. The method is based upon finding those features which minimize the so-called radius / margin bound, which is an upper-bound on the leave-one-out error. This search can be efficiently performed via gradient descent. The resulting algorithms are shown to be superior to some standard feature selection algorithms on both toy data and real-life problems.