Probabilistic Deep Learning
Reading material
- Durr, Sick - Probabilistic Deep Learning
- Main book with notebooks: https://github.com/tensorchiefs/dl_book
- Ch. 4, 5, T.B.C.
- Goodfellow, Bengio, Courville - Deep Learning
- More for detailed background reading
TODO \ questions
- (C) Exercise 2.5 Lecture notes 2b
- Know how to derive closed form solutions for Gaussians (see lecture 0)
- When is something intractable?
- Closed form variational solutions (2a, end of section 2)
- Know difference between discriminative and generative methods (also in relation to DNNs)
- Write out error calibration formula
- Explain reasons for intractability in various models
- Explain problems with Bayesian deep learning
Lectures:
The lectures were given in a messed up order because the course was developed on the go. Lectures 0* came after 3.
- PDL-0a-Exact Bayesian Inference
- PDL-0b-Exact multivariate inference
- PDL-1-Introduction.md
- PDL-1-Probabilistic_modeling.md
- PDL-2a-Bayesian_inference_with_gradient_descent
- PDL-2b-Stochastic_variational_inference.md
- PDL-3-Variational_Supervised_Learning.md
- PDL-4-Variational timeseries analysis
- PDL 5 -Expectation_maximization_and_variation_maximum_likelihood.md
- TODO PDL 6 Amortized inference and autoencoders
- Lecture 7... bunch of papers I didn't read.
Notes on assignments:
Main goals
- How to integrate knowledge (e.g. mathematical equations) into deep learning components?
- I.e. can we create hybrid models!
Concepts
- Empirical Bayes
- Probabilistic Calibration
- Latent variables
- ELBO: evidence lower bound
- Jensen inequality
- Reparameterization trick
Also see index machine learning
Formulas to write down on cheat sheet
- ELBO
- KL-divergence (both forms)
- Reparameterization
- Sigmoid and derivative of sigmoid (Bishop p. 205)
- Entropy of normal distribution
- $\frac{1}{2} log (2 \pi \sigma^2) + \frac{1}{2$}
- Form for log of univariate and multivariate Gaussian
Univariate:
$$-\frac{1}{2 \sigma^2} \sum_{n=1}^N (x_n - \mu)^2 - \frac{N}{2} ln(\sigma^2) - \frac{N}{2}ln(2\pi)$$
- Closed form posterior solution Gaussian inference
- Change of variable formula (lecture 0b)