Multilevel Optimization and Machine Learning

Kristin P. Bennett

Dept of Mathematical Sciences
Rensselaer Polytechnic Institute

Multilevel models are optimization problems that require solution of other mathematical programming problems as part of their constraints.    Machine learning tasks such as cross-validation can be naturally expressed as multilevel models.   Using cross-validation as an example, we will examine how multilevel models are formulated.   We will explore how bilevel models can be transformed to alternative forms such as mathematical programs with equilibrium constraints (MPEC), nonlinear programs, and integer programs.  We will discuss available algorithms for solving bilevel programs.   We will examine the versatility of bilevel framework for addressing different machine learning tasks such as semi-supervised learning, variable selection, variable scaling, and kernel selection.   Finally, we will conclude with a discussion of potential contributions and challenges of multilevel optimization in machine learning.