Leave-one-out cross-validation is a simple generic tool for selecting the best empirical model. When we model data empirically, for example, with a polynomial, we want to select the model which provides the best compromise between bias and variance. If the empirical model is too simple, it won’t be able to describe the data properly and the result is biased. If the model is too flexible, it will start to follow statistical noise, this leads to an increased variance. This is called overfitting and leads to poor generalization of the model.
The general steps for LOO cross-validation are:
- Remove i-th datum from input data set
- Fit model to remaining data set
- Use fitted model to predict the i-th datum and store difference to original i-th datum
- Do this for all i and compute variance of the differences
- Select model with the smallest variance
The variance computed in this way is the mean-squared-error, which consists of a bias term squared and the variance. Minimizing this thus finds the best compromise between the two terms.
I demonstrate below how to select the optimal order for a polynomial model with leave-one-out (LOO) cross validation.
