Probabilistic Deep Learning

Reading material

TODO \ questions

  • (C) Exercise 2.5 Lecture notes 2b
  • Know how to derive closed form solutions for Gaussians (see lecture 0)
  • When is something intractable?
  • Closed form variational solutions (2a, end of section 2)
  • Know difference between discriminative and generative methods (also in relation to DNNs)
  • Write out error calibration formula
  • Explain reasons for intractability in various models
  • Explain problems with Bayesian deep learning

Lectures:

The lectures were given in a messed up order because the course was developed on the go. Lectures 0* came after 3.

Notes on assignments:

Main goals

  • How to integrate knowledge (e.g. mathematical equations) into deep learning components?
    • I.e. can we create hybrid models!

Concepts

Also see index machine learning

Formulas to write down on cheat sheet

  • ELBO
  • KL-divergence (both forms)
  • Reparameterization
  • Sigmoid and derivative of sigmoid (Bishop p. 205)
  • Entropy of normal distribution
    • $\frac{1}{2} log (2 \pi \sigma^2) + \frac{1}{2$}
  • Form for log of univariate and multivariate Gaussian

Univariate:

$$-\frac{1}{2 \sigma^2} \sum_{n=1}^N (x_n - \mu)^2 - \frac{N}{2} ln(\sigma^2) - \frac{N}{2}ln(2\pi)$$

  • Closed form posterior solution Gaussian inference
  • Change of variable formula (lecture 0b)

See SML-Cheatsheet.md