SML Lecture 3: Probability theory and applications

  • Probability recap
  • Maximum likelihood
  • Bayesian vs. frequentist perspective
  • Bayesian updating

Probability theory

Probabilities let us express a degree of uncertainty quantitatively in terms of numbers, and allows reasoning under uncertainty.

Sources of uncertainty:

  • Measurement noise
  • Influence of unobserved factors
  • Finite data
  • Missing data
  • Range of experimental settings
  • etc.

Other things we should also take into account later:

  • uncertainty in model parameters;
  • uncertainty in predictions.

First half is recap of probability concepts and Bayes' rule.

Conditional independence: P(X,Y|Z)=P(X|Z)P(Y|Z)

Probability densities

When x takes values from a continuous domain, the probability of any value of x is zero! It would be like trying to specify a probability in an interval that's "empty".

I.e. we cannot express the probability of a continuous value independent of an interval.

If we consider a CDF instead of a PDF, p(x) may be bigger than one! The only constraint is that $p(x) \leq 0$ and that it normalized to one when we integrate the area under the graph.

CDF is also non decreasing.

REVIEW: difference PDB and CDF

REVIEW mutlivariate notation. With vectors?

Multivariate densities

Probability p(x): probability of x falling in an infinitesimal voilume dx around x given by p(x)dx.

Transformation of densities

Under nonlinear change of variables, a probability transforms p(x)transforms differently from an ordinary function.

E.g. we cannot substitute one density function into another?

|dx/dy| is the Jacobian; determines the magnitude of the transformation; or something like that.

Probability density of y:

P_y(Y) = Px[f^-1(y)]/ (dy/dx)

Expectations and variances

We use the convention to express discrete expectation of f as ; just in this course to make a distinction.

Variance: variability around the mean.

$$var[f]= - ^2$$

Covariance: how they vary with respect to each other; measure of synchronicity.

Choosing a probability distribution

The age old example: Flipping a coin.

Bernouilli

Bernoulli distribution, let 1 and 0 indicate head and tails.

$$p(x=1|mu)=mu$$

$$p(x=0|mu)=1-mu$$

$$Bern(x|mu)=mu^x(1-mu)^{1-x}$$

$$E[x]=mu$$

$$var[x]= mu(1-mu)$$

See notes for derivation.

Maximum likelihood estimation

D = {x^1,...,x^N}

We assume these events are independent and come from the same distribution.

With parametrized distribution $p(x|\theta)$.

Let's take Bernouilli again. What parameters do we have to take to make the distribution fit the data best?

TODO: Copy in here MLE formula later.

We use log for convenience; underflow etc. Possible because log is monotonically increasing.

Frequentist methods consider a single true $\theta$ and data generation mechanism $p(D|\theta)$ provided by "Nature". Good old providence!

$$E\theta_{ML}=\sum_{D} p(D|\theta)\theta_{ML}(D)$$

Now fill in the Bernouilli distribution.

In this case, maximation gives ....

However, if we find {1,1,1}, our MLE will be 1! Predicting the best model says the coin will always come up head. And yet this is unlikely!

What's the problem? We have very little observations!

Our confidence in our conclusion should be low in this case.

The Bayesian approach can quantify this degree of certainty, the frequentist cannot (I think?).

Bayesian vs. frequentist

  • events are not necessarily repeatable
  • the assumption that an event is infinitely repeatable is also not practical
  • events may yield only indirect observations

Cox: common sense axioms for degree of uncertainty -> probability theory.

Inclusion of prior knowledge may be useful. Debate: is this a reflection of prior knowledge or a construct? If you choose your prior poorly you may get bad artificially certain results.

Bayesian integrals are computationally complex, so we often rely on approximations.

Frequentist data resampling is appealing (e.g. bootstrapping)

Bayesian machine learning

We express our uncertainty about probabilities by imposing a prior distribution into it. Frequentist statistiscs does not take this directly into account!

evidence; marginal distribution of the data.

In MLE we can ignore the evidence term because it's just for normalization.

Back to the amazon example. We had the Bernouilli distribution. What prior on parameters mu do we choose?

Convenient: conjugate prior.

We should also be cautious about choosing an uninformative prior, because it can lead to bad behavior of your posterior distribution.

(Interesting to me! Because you would choose the prior based on prior knowledge right? How proper is it then to choose a different prior just because it's convenient?)

If we now do the Amazing MLE analysis again (see slides), we do get the expected result.

Review, we can also do the analysis with $\deta = \mu_1 - \mu_2$ and then finding $P(\deta > 0|D)$. This directly compares the two cases.