ICML 2011 workshop on unsupervised and transfer learning

Transfer Learning for Document Classification:
Sampling Informative Priors

Philemon Brakel  and Benjamin Schrauwen
Department of Electronics and Information Systems, Ghent University, Belgium

When not much labeled data is available, learning algorithms might benefit from prior domain knowledge. We used a hierarchical Bayesian logistic regression model in which the covariance matrix of a multivariate Gaussian prior over the parameters was estimated for a set of related tasks. Inference was done by using a combination of Hybrid Monte Carlo and Gibbs sampling. We demonstrate on a binary document classification task that the obtained priors contain information that is beneficial for performing the task of interest when the number of labeled examples is small. Using these priors leads to a significant improvement of performance over models with simpler regularizers. We also investigate the kind of information these priors learn to capture.