Lecture 4: Probability distributions and Bayesian inference

  • Multinomial and Dirichlet
  • Univariate Gaussian and maximum likelihood
  • Multivariate Gaussian
  • Bayes theorem for the linear Gaussian model
  • Gaussian Mixture Models

Multinomial variables

Book: 2.2

Alternative for Bernoulli distr. with x in {1,2} and using the delta function.

Rewriting into a product, it becomes easy to see that we can extend this to multinomial variables where x can instead take K mutually exclusive states.

We can encode the K states either in an ordinal encoding scheme (one number per state), or using one-hot-encoding (0-vector except 1 for the encoded value).

What is the advantage of one-hot-encoding? In this case there is no ordering. The distance between any two vectors is equal. E.g. in ordinal encoding "apple (1)" would be more distant to "orange (3)" than to "grapefruit (2)".

So we use one-K encoding.

Slide 53: We need Bayesian inference to overcome this "overfitting"; all mu's for unobserved variables are 0! This is highly unlikely!

The Dirichlet distribution is a conjugate prior of the Multinomial distribution.

Simplex: space of mu's where all mu's are positive and normalized.

Review until slide 58.

Gaussian distribution

A Gaussian is completely determined by the mean and the variance, because the mean of the Gaussian is the mean parameter, and its variance is the variance parameter.

Central Limit Theorem

Any sum of large sets of random variables is approximately Gaussian distributed.

Until 63

Bias in Maximum Likelihood estimates

The formulas on slide 63 are functions of data.

Assuming the data generation is Gaussian; then our estimates would be biased if we would on average recover the mean and variance. We do recover the mean, but we do not exactly recover the true variance.

If we draw a single data point, our mu is that point but the variance is zero. This is by definition an underestimation of the variance of the true underlying distribution. So if you do N single picks, you find a variance of 0 each time.

This is called a bias estimator.

  1. This bias disappears with more N
    • Bayesian approach counters this
    • Frequentist approaches tell you to instead take N-1 instead of N
  2. With more parameters, we have a more severe bias

Multivariate Gaussian

Now x and mu are vectors, Sigma is a square covariance matrix DxD.

Product of two independent Gaussian variables is a multivariate Gaussian with diagonal covariance matrx.

Covariance between x1 and x2 is zero, because the two Gaussians are independent. Also, covariance of x_i with itself is the variance.

So the covariance matrix here has zeros everywhere except on the diagonal, where we see t he variances of the involved random variables.

I missed something with axis-aligned.

In the following, we see how Gaussians look like with general covariance matrices (not just a diagonal one).

Intermezzo: symmetric matrices, eigenvalues, eigenvectors

See Bishop 2005 appendix C for a recap.

u1 and u2 are orthogal, i.e. their dot product is 0.

Symmetric A has a rotation such that ... eigenvalue basis.

Gaussians and covariance

The covariance matrix of a Gaussian is symmetric, so we can use the previous results.

The eigenvalues of that covariance matrix are variances in the rotated basis!

Also means that all eigenvalues of the covariance matrix must be positive.

A symmetric matrix with only positive eigenvalues is called a positive definite.

Example use of eigenvectors: Principle Component Analysis

An algorithm uses the properties we just examined.

If you have a big high-dimensional data X, and you choose N principle components, then it returns the projection of the data on the basis of eigenvectors with the largest eigenvalues. (Ref. slide 70; there we chose the largest lambda(?))

N.b. eigenvectors indicate direction; I think.

A.k.a. PCA chooses the direction of largest variance in the rotated space.

We can also leave out the eigenvectors and move only in the latent space. To "get back" to the original space we multiply with the eigenvalues (or eigenvectors?).

Example discussed in Bishop 12.1.3.

Many parameters

For multivariate Gaussians, we have a lot of parameters.

Covariance D(D+1)/2 and mean D parameters, for dimension D.

For high dimensional applications having quadratic size parameters may be too much.

So we can apply limitations/restrictions.

See p.73; I missed the formulas.

Marginal and conditional Gaussians

Partitioning; REVIEW 74.

Do we partition the variables of a multivariate? And condition on on the other? Why do we do that?

I think; go back to example of picking fruit (x1) based on a choice for box (x2). When you have a Gaussian over both random variables; it is useful to model x1|x2.

It's very convenient to work with precision matrices, which is the inverse covariance matrices.

Why? In the Gaussian we had an inverse of the covariance matrix somewhere (? Can't recall). This can now be directly rewritten as K.

Careful: matrix inverse is not component-wise.

Lot of math I did not really follow that quickly.

Conclusion: Conditional of Gaussian is itself also a Gaussian.

Bayes' theorem for linear Gaussian model

Big lambda on slide 78 is a precision matrix. (But we used K for that earlier?)

Lambda inverse on slide 79 is a way of writing a 2x2 matrix (row1; row2)

Bayesian inference for linear Gaussian model

Book: 2.3.6

Slide 80 -> assume variance is known; we try to find the mean.

Conjugate prior of Gaussian is also a Gaussian.

If $N_0 = 0$, then $\mu_{ML} = \mu_{N} = x^{-}$ (not sure you to do the stripe above the x). Does this make Bayesian with a flat prior coincide with regular maximum likelihood estimation?

They seem the same, but are different because a flat prior means a lot of uncertainty. Regular ML instead had way too much certainty in very specific values (due to overfitting).

See 2.3.6.

Yulia:

So here is another important point Wim is making, which perhaps is relevant for previous discussion. We don’t put the prior just to sum-up our subjective ideas, but it also allows us to quantify uncertainty later on in a very natural way (since we are estimating posterior of the parameters — we assume they are random variables and have a distribution, and not just ‘one true value’).