Exact Multivariate Inference
The form of Bayes' theorem is unchanged in the multivariate case. But in the multivariate case, the normalization integral will be an integral w.r.t. all variables, i.e. a multivariate integral.
You generally can do the integrals one by one, until you've completed the last integral.
In the multivariate case, we cannot discretize the integral anymore, since there's an exponential number of grid points (exponential: $M^N$).
In Bayesian notation, we usually gather all variables into a vector $\mathbf{x}$. In this case we can write the evidence in the usual way.
We are not dealing with indefinite integrals, we're just lazy and omit the integral range.
A definite integral is a number whereas an indefinite integral is function (namely the antiderivative)!
In practice, you get familiar with probability distributions and know the appropriate range.
Note that in eq. 6, we do not explicitly write $x$ as a vector. We do this because in some cases the formula works both in the univariate and multivariate case. This hold for example for the KL-divergence.
Marginalization and conditionals
Sometimes we're only interested in one variable $x_1$, but we don't know the other variables. In that case, you marginalize the other variables out. You can do the same and end up with a joint, in which case we sometimes speak of a "joint marginalized distribution".
Marginalization is very difficult in practice. But if we can sample from a joint distribution we have another approach to marginalization. From joint to marginal this is trivial.
Let's say we have $x_1,x_2,x_3 ~ P(x_1,x_2,x_3)$. If we are interested in $x_1 ~ P(x_1)$, we can sample from the joint distribution and just ignore all other variables. Done. So if we can sample from the joint, that's good news.
Bayesian inference is a special way of conditioning on probability distributions.
Change of variable formula
This is very important in this course.
Neural networks can be modeled as a differentiable transformation from input data to output.
If we start with variable x, that follows the input distribution p(x), then we can define the variable $y=f(x)$ which is the result of some differentiable transformation, which then will result in a probability distribution over y. So probabilistically speaking, NNs can be phrased in terms of a variable change in probability distributions (such that we end up with a very complex output). Because in the end, NN is just a huge method for computing transformation $f$.
NNs can thus be used to model very complex probability distributions as a transformation of input.
For variational inference of MLE, we need to be able to express these transformations mathematically.
We know how to sample from $p(z)$, namely by sampling x and then piping it through f(x). But in this case, we are interested in computing the probability density $p(z)$.
Since we know how to sample x, we can compute the expectation of z just by sampling x, pipe through f(x) and take the expectation of the output.
But knowing the expectation over z is not enough, because an infinite amount of distributions can have the same expectation.
Trick: introduce test function $\phi(z)$.
If we can compute the average of any possible transformation of $z$, then this allows me to control and fully specify the density over z.
Example: use the $\delta$ function as a test function.
If you take an integral from a to b over the delta times f(x), then this is equal to f(0) (assuming 0 is between a and b). The delta function can "extract" the value at a specific point only by having infinite mass around 0. The latter is needed because strictly speaking density is only defined over ranges, i.e. the density of a point is infinitely small, which is why we need to multiply by infinite to recover its value. Delta is not a proper function and not well-defined, but a limit case of any smooth function.
Now we use delta as the test function, which given above property, allows us to recover the density of a point $z^*$. We can recover any possible point of the probability density over $z$.
Now we can combine steps
- We can sample from $p(z)$
- We know the expectation over $z$, but not the probability density $z$
- But now we have found a way to compute the density over $z$, via the computation of an expectation.
Namely, expectation over x, of $\phi(f(x))$. Now we want to express this w.r.t. z instead of x. Well, the have
- f(x)=z.
- $p_0(x)$ can be written as the inverse $f^{-1}(z)$
- now dx needs to be written as dz
- use Jacobian matrix!
$dx = | det Df^{-1}(z) | dz$
The inverse Jacobian matrix is the matrix of partial derivatives of the inverse transformation $f^{-1}$ for all function components.
Nice thing about Jacobian is can be used to linearize transformation (...). Gives you the best linear approximation around a point from a nonlinear function (that's the whole point about the derivative).
Note that we are assuming we can compute the determinant; i.e. we are assuming square matrices, i.e. an equal dimension of input and output.
blablabla... you are you just 'looking back' to the probability of the point you used to get here.
Eq. 22: Determinant of inverse can be written in terms of the forward transformation.
Because the element of a Jacobian gives you a linear approximation, it makes sense that the infinitesimal dx depends on this Jacobian; since this "local linear approximation" is exactly what you are doing (in the limit case) with dx.
Building intuition about determinant: Example with A = [ [ a 0], [0 b]], transformation is multiplying volume with $a*b$ which happens to be the absolute determinant |A|.
So the determinant of a transformation gives you the scaling factor of the volume!
We don't have a linear transformation, rather a non-linear one. However, we can apply the volume transformation (i.e. determinant) on each local linear transformation of the volume around a single point.
Multivariate Gaussian
We use the transformation of variable to define the multivariate Gaussian.
$Det(A A^T) = Det(A)Det(A^T)$
So $Det(C)=Det(A)^2$
So to write this in terms of the determinant of A we take the square root here.
Eq. 31: This is why N/2 appears in the exponent of the determinant multivariate distribution.
C is:
- symmetric (important for interpretation because correlation is symmetric)
- positive definite
- See eq. 33
Understand this section and how we can explicitly write the density using the variable transformation. Change of variable formula can be applied to any non-linear transformation. But to go from N univariate normals to a N-dimensional multivariate distribution, we do use that the latter is the result of a linear transformation (namely through correlation, I guess?).
If you don't make that assumption you can apply the same procedure, but you'll destroy normality and won't end up with a multivariate Gaussian.