Introduction to Gaussian processes
Gaussian Processes TL;DR
- Non-parametric prior on functions;
- Benefit from properties of Gaussian random variables (closed under linear operations)
- Option to use non-linear operations or non-Gaussian likelihoods; then we approximate the likelihood (Laplace or variational inference)
- Many things in statistics and ML can be shown to be a GP (from linear regression to neural networks)
- Helpful for understanding the intuitions behind what some model actually does
- Unevenly spaced data is not a problem
- Con: Computationally demanding (but approximations available)
- Con: They are very flexible, which may require proper constraining using priors to guide the GP
What is a kernel?
For fixed nonlinear feature space mappings $\phi(\mathbb{x})$, the kernel function is given by
$$k(x, x') = \phi(x)^T \phi(x')$$
So we also have $k(x,x')=k(x',x)$.
If the feature mapping is linear $\phi(x)=x$ then the kernel function is $x^T x'$. But notice that if the feature mapping is nonlinear, the kernel still is a number of scalar products!
What is a Gaussian process?
Collection of random variables, of which a finite amount has a joint Gaussian distribution.
N(m,K)
How do we specifiy K?
E.g. squared exponential kernel
Is we increase the size of the covariance matrix (i.e. more GP random variables) we get a smoother function.
Two hyperparameters:
- sigma squared
- influences amplitude
- l: length scale
- influences smoothness of lines
So these smooth functions are sampled from a Gaussian with covariance K(x,x'), where x is e.g. 50 datapoints (in the slide).
Note that x x' is not y values; so it does not depend on the input data yet! We can sample from specific x values on the prior.
Standard Kernels
See slide.
Defining new kernels
See equations 17-26.
Bayesian linear regression as a GP
Some people argue that the prior introduces arbitrariness. But defining the likelihood is also a model assumption, just like the prior. It's better to make them explicit and understand where you assumptions may fail.
It is useful for our own understanding to cast Bayesian linear regression as an example of a very simple GP.
Weights and bias of regression line are sampled from a Gaussian.
What is a kernel?
Kernel; def see eq. 5.
Uses feature space mapping $\phi_(\mathbf{x})$.
The kernel expresses similarity.
What is "duel representation"?
Kernel trick: to move from linear models to more complicated non-linear versions, e.g from PCA to non-linear version of PCA.
Kernel for Bayesian linear regression
Define covariance
TODO: went too fast.
When useful
- When more input features than data points, GP approaches may be cheaper
- But in general the linear regression approach is more obvious
Handy properties for regression
See eq. 28 for GP on Latent error
Posterior predictive density
See eq. 29-31
f* indicates "prediction".
GP inference
See eq. 32
In simple problems GP are well behaved. But in complex scenarios,you want to leverage expert knowledge to for example restrict the length scale; e.g. we may known the function should roughly have a certain smoothness.
GP are useful for uncertainty quantification.
Possible constraints
- Additive (extrapolate for away)
- Multiplicative (goes to zero away from the data points)
- Symmetry
- Independence of other arguments
GP Classification
More in detail later
- step 1
- step 2
- step 3
Before everything was Gaussian, which means we can use many nice properties.
But with classes we end up with a non-Gaussian distribution, e.g. bernouilli for binary classes
Next weeks we discuss:
- sampling
- laplace approximation
- variational inference
The bigger picture: The Zoubin cube
See slide.
Three axes:
- Classification
- Bayesian approach
- Kernel methods
Neural Networks as GP
Neil 1994 showed that a single layer NN is a Gaussian Process, by assuming that the weights and bias are distributed according to a Gaussian, with a kernal depending on the non-linearity.
GPs and NNs potentially compensate each others weaknesses:
NNs are better for dealing with larger datasets, so this link with NNs may help Gaussians deal with more data.
The uncertainty estimated by the Gaussian process correlates with the mean squared prediction error. This means that GPs are very useful for uncertainty quantification in Neural Networks.
GPs and Bayesian optimization
When you are globally optimizing some black box function that is very expensive to evaluate (e.g. in physics), we can use Gaussian processes as priors.
Why use GPs? Pros and cons
- Flexible and diverse way of modeling non-parametric functions
- Quantify uncertainty
- Leverage Gaussian properties
- Approximation available for non-linear and non-Gaussian functions
- Combination of data-driven and mechanistic modelling
- Suitable for multi-disciplinary research
Cons:
- $O(N^3)$ computational cost of marginal likelihood evaluation/predictions
- Cholesky decomposition (?)
- practical solution: add small noise to diagonal of covariance matrix
- $O(DN^2)$ and $O(N^2)$ memory for computing kernel matrix.
Conceptual
- Not all functions can be represented as a Gaussian process
- e.g. monotonic functions cannot be represented as a GP
- Missed the 2nd one