Notes A2

  • Discretization works well in 1D;
  • Problem with variational inference in this case is that we (may?) have a bimodal distribution, but our parameterized variational distribution e.g. a Gaussian only handles the unimodal scenario
    • By discretizing we can avoid this problem.

On the notebook:

  • use rsample to allow gradient w.r.t. sampling
    • Exercise: try this without rsample and re-implement yourself.
  • scale; standard deviation parameter
  • softplus instead of exponential, because exp is unstable for big inputs
    • softplus instead has exponential decay but dot dot linear
  • in evaluate_ELBO function, we do not compute entropy in closed form and instead use it in the approximator, which may add a bit of variance.
    • But doing it this way makes VI super independent of the specific model
    • E.g. closed form solution
  • My question: Poisson likelihood doesn't have rsample, but that doesnt matter! Luca just used rsample everywhere and then chooses to detach it later, to make sure you don't actually backpropagate.
  • For ex. 2, you regenerate the data using a Poisson likelihood, you can still use a Gaussian as the variational distribution if you want!
  • Ex 2: which range of z to pick? Start pretty broad and then adjust based on exploration.
  • Ex. 2 True posterior is not going to be a Gaussian with the Poisson likelihood
    • We expect the grid to do better than VI, because of the reason above
    • Experiment: If you use way more datapoints, the posterior will tend to be Gaussian, so a VI with a Gaussian variational posterior will do better due to the central limit theorem
    • Exceptions are distros with pathological variance; e.g. Gaucy has infinite variance.

For Ex. 3 we will clearly see the multimodal posterior if we compute it by hand. This is explained in last lecture (2a). Experiment and play with this to see the variational distribution fit to either of the modes.

Ex. 4 typo: z1 gives mean, z2 gives variance. So we have two latents, one for mean, one for latent. We do two Gaussian approximations. Use mean field approximation.