Lecture 11: Variational Inference

When choosing an inference method there's several trade-offs leading to a sensible choice.

  • Exact and slow with MCMC
  • Faster less accurate with VI

Previous lecture we discussed sampling approaches to estimating the posterior distribution. These various algorithms work great overall, but are usually quite expensive.

Integration vs. optimization

The integration necessary to get to the posterior is expensive. Can we instead turn this 'integration' problem into an 'optimization' problem.

"Variational inference is that thing you implement while waiting for your Gibbs samples to converge" (David Blei)

Trade-offs:

  • More mathematically involved
  • No theoretical guarantees on the outcome
  • Usually fast
  • Stochastic variational inference made some type of models on larger data sets feasible (e.g. large topic models; Gaussian processes for big data)

Background: information theory

How can we quantify information?

Have random variable X. How much information is received when we observe a variable of X?

P(snows in Siberia in december) = 0.99

P(snows in NL in december) = 0.1

When we see that it shows in NL in December this gives us more information than when we observe it snows in Siberia (DUH).

This is because there is more entropy "uncertainty" in the estimate for NL.

Homework here: derive relation between entropy and the probability density. $h(x) = -log(p(x))$ (Cf. 1.6 in Bishop)

If we have some probability distribution p(x) and we approximate it with q(x), we can express how much information we lose using relative-entropy or Kullback-Leibler divergence.

Note that KL divergence is not symmetric.

VI and KL Divergence

In VI, we don't know true posterior p(x) and instead try to find an q(x) that's a good approximation, i.e. we will be looking to minimize KL-divergence.

Main challenge: we of course do not know p(x), so how can we minimize the difference?

But we can factorize the model evidence into the lower bound KL-divergence.

$$ln p(X) = L(q, theta) + KL(q||p)$$

To minimize loss of information we maximize the L instead!

L is also called the variational free energy.

But how to choose distribution q(x)?

Factorized approximations (Mean field )

Choosing q(x).

Idea: use a family of factorized distributions.

$$q(Z) = \prod_{i=1}^M q_i(Z_i)$$

This is called mean field approximation.

General idea: keep all except one of component distributions fixed, then minimize for KL, and iterate.

A property of this type of factorized approximation is that the mean is approximated well, but the variance is usually captures not well. This is a general downside to mean field approximation.

In expectation propagation we can instead minimize KL(p||q) (instead of KL(q||p)).

In this approach we still capture the mean and variance, but variance will be the same in all directions, meaning we'll most likely put a lot of probability mass in the wrong area.

KL(q||p) finds one of the local modes (in our example there are two), whereas KL(p||q) finds an average of modes?

Variational linear regression

See examples from slides.

When computing the lower bound for different M's in your regression, you can also use the lower bound as a model selection criterium, because (if we assume same priors on models?) it's also lowerbound on the log marginal likelihood.