Probabilistic Deep Learning Lecture 1: Introduction
What's the theoretical foundation for using one particular loss function over another? What guides our choice of architectures and loss functions etc.? Goal of this course is to contextualize deep learning in the context of statistics and in particular Bayesian probability.
So the first main topic is:
- How to study loss functions from the perspective of probability theory?
Then we move to problems that standard deep learning architectures do not deal with well:
- Latent variables
A typical use of a latent variable is an underlying diagnosis. We do not directly observe the underlying problem, only the symptoms.
Why do we want probability distributions over our estimations, rather than deterministic point estimates? In this case, by doing a MLE you do not acknowledge that there are other possible diagnoses that may be more unlikely, but not irrelevant.
W.r.t. AI, the future always involves a degree of uncertainty. For good decision-making, you need to take uncertainty into account. But most of contemporary AI still is not probabilistic.
In standard deep learning we do have a probabilistic estimate, e.g. using softmax and cross-entropy. But this means we interpret the last layer output as probabilities. But this does not mean that the method overall is properly probabilistically calibrated.
The problem of probabilistic calibration: nice that the output is probabilities, but do they correctly capture the probability distribution over the data?
P(Rain)=0.99, and it doesn't rain a while, then perhaps our method wasn't calibrated because it was too certain about it raining. E.g. the method may predict rain with a high accuracy, but it doesn't correctly factor in uncertainty and when it fails. So a model with good probabilistic calibration should also recognize situations where its estimate may be wrong.
This is very important. E.g. in medicine one needs to be ready for rare but destructive symptoms. What's most likely to happen is not necessarily what happens. You need probabilities for risk-assessment.
The goal of probabilistic deep learning is to provide robust probability estimates.
Bayesian Statistics Recap
Bayesian Statistics is one of the pillars of PDL.
- X: observed data
- Z: latent variables
We usually assume to know the likelihood $P(X|Z)$. This is sort of a "forward model" because we assume to know the value of the latent. Under this assumption on the latent, what's the probability of the observed data?
We want to know: $P(Z|X)$ i.e. perform the "Bayesian inversion" by applying Bayes' theorem.
What some statisticians do not like here is that Bayes' theorem also depends on $P(Z)$ i.e. a prior distribution over the latents. E.g. in the medical field we could look at the frequency of a diagnosis. This is the empirical Bayes approach, i.e. you have a frequentist prior! The controversy really is: what prior do we choose when we do not have such an empirical estimation of the prior.
You can also do meta-learning: e.g. when we need a prior for classification we can see what works in other classification problems. (Not sure I got this right). For example, in the beginning of COVID we did not have a large population of COVID-19 patients. The meta-learning approach here could be to look at the behavior of other corona viruses, e.g. the common cold.
E.g. we can study if children transfer the virus, and then have a prior that suggests they do, because they do transfer e.g. the common cold. We can still overrule this prior if the data shows otherwise with a lot of confidence.
In practice we often have little data, and then having a prior helps getting proper probabilistic calibration.
Bayesian NN
In Bayesian Neural Networks we can define the log likelihood $log(P(X|W))$ as a loss function. Cross-entropy loss is in fact a log likelihood term. This tells us "how much we fit the data" given a set of weights.
We then add a prior over weights $P(W)$ to get from a point estimate on weights to a posterior distribution over weights $P(W|X)$. To get point estimates for prediction in the end we integrate over weights. But now this prediction takes into account a proper probabilistic model and is probabilistically calibrated.
But how do we get $P(W)$?
Unless you have a lot of data, it will be difficult to get good probabilistic calibration without a good prior on the weights. Construction priors on weights is somewhat of an art rather than a science. But a lot of work has to be done here.
The issue with probabilistic calibration in deep learning: if we get softmax output and we interpret them as probabilistic, how confident are we then that these are correct?
Bayes theorem uses the integral over weights as normalizing constant, but because we have so many weights this is even more infeasible than it commonly is.
Question hour week 1
We have data points that are tuples $(x_j, y_j)$ with regressors and targets.
A neural net approximates $\hat(y_j) = f_w(x_j)$ with weights w.
We have a loss function that defines distance to targets.
$L(w)= 1/2 \sum_{j=1}^{J}( y_j - f_w(x_i))^2$
or $L(w) = - log l(w)$
So convention L(w) for the log likelihood, lower case l for regular.
likelihood $l(w) = \prod_{i} p(y_i | x_{ji}w)$ which is essentially a generative model. Could e.g. be a normal with homoscedasticity (=fixed variance), e.g. $N(w x_j; I\sigma^2)$ i.e. $N(\mu=f_w(x_j), I\sigma^2)$.
When writing out the log likelihood, we drop terms that do not depend on the parameters we are optimizing (weights w in this case). (So we drop -log 2 pi sigma^2).
In the final expression we see that the error has variance in it. We also do not usually care about this; the final expression is proportional to the SSE loss function.
So if we assume Gaussian noise, we can derive the SSE as a natural fit for linear regression.
In any case, we still have to be smart. We have to think about which generative model could be behind our data, taking into account the features of the data.