Lecture 5
Short and handy review of what we've discussed, including example exam question.
P. 87: Additive Gaussian problem.
p. 94; we can think of this as a multivariate observation, i.e. we see all variables at the same time.
The likelihood formula with multiple components we can think of as a single multivariate Gaussian.
See p.97: 2.113-117: study these forms, they will for sure be on the exam.
p.98: once you find an answer, how do you check if your answer makes sense?
Look at the behavior what happens when:
- S^{-1} -> 0 (very broad prior)
- Apparently we get the same maximum likelihood solution
- That makes sense: without restrictions, our MLE is fully determined by the data
- N=0
- Here you should end up with the prior; we are left with S_0^{-1}.
- N -> Inf
- Then the prior should have no impact anymore; so again same MLE solution
In regression, we are not only interested in weights, but also in how certain or uncertain we are about those weights. The Bayesian approach also encodes this uncertainty in a posterior over the weights. But we are also not interested in weights in the sense that the whole point of this exercise was to predict values. From about slide 100: getting to the predictive distribution. Here we again see that N influences the distribution around the prediction, i.e. it's certainty.
P. 103: with N->Inf, only intrinsic noise is left.
P. 104: you see that around the datapoints we have one standard deviation uncertainty; but at other spots this widens.
Bayesian model comparison
In the Bayesian approach we do not necessarily have to do train-test splits, but can test overfitting of models directly on the training data, by integration of model parameters.
Data constrains viable parameters in model parameter space.
Height at optimal solution; width posterior/width prior.
REVIEW, missed a few things here; teacher was over time and rushing.