Maximum Likelihood Estimate for a univariate Gaussian

Also see univariate closed form Gaussian inference.

First find the expression for the Gaussian log likelihood:

$$ln p(D|\mu, \sigma^2) = ln \prod_{n=1}^N \mathbb{N}(x_n | \mu, \sigma^2)$$ $$= \sum_{n=1}^N ln \mathbb{N}(x_n | \mu, \sigma^2)$$ $$= \sum_{n=1}^N ln( \frac{1}{\sqrt{ 2\pi \sigma^2}} exp{ - \frac{(x_n - \mu)^2}{2 \sigma^2}})$$ $$= N ln( \frac{1}{\sqrt{2\pi \sigma^2}}) - \sum_{n=1}^N \frac{(x_n - \mu)^2}{2 \sigma^2}$$ $$= -\frac{1}{2 \sigma^2} \sum_{n=1}^N (x_n - \mu)^2 - \frac{N}{2} ln(\sigma^2) - \frac{N}{2}ln(2\pi)$$

The next step is to find $\mu_{ML}$ by deriving with respect to $\mu$. The derivative w.r.t. $\mu$ is:

$$\frac{1}{\sigma^2} \sum_{n=1}^N (x_n - \mu)$$

Setting to zero gives the MLE:

$$\mu_{ML} = \frac{1}{N}\sum_{n=1}^N x_n$$

We see this solution does not depend on $\sigma^2$! So to find the MLE for $\sigma^2$ we can start by simply substituting in $\mu_{ML}$.

One important detail: now we find the gradient with respect to $\sigma^2$ (not $\sigma$!). So easiest is to define $v=\sigma^2$. Then computing the derivative w.r.t. $v$ and setting to zero gives:

$$\frac{1}{2 v^2} \sum_{n=1}^N (x_n - \bar{x})^2a - \frac{N}{2 v} = 0$$

Getting out the awkward term on the right:

$$\frac{N}{2 v^2}( \frac{1}{N} \sum_{n=1}^N(x_n - \bar{x})^2 - v) = 0$$

This can be zero either when $v = \sigma^2 = \infty$ or when:

$$v = \sigma^2 = \frac{1}{N} \sum_{n=1}^N (x_n - \bar{x})^2$$

Voila!

Multivariate Gaussian with diagonal covariance matrix decomposed in product of univariate Gaussians

To show this you need to use

  • $\Sigma^{-1}{ij} = \frac{1}{\sigma_i^2} I{ij}$
  • $|\Sigma|=\prod_{i=1}^D \sigma_i^2$
    • Determinant of diagonal covariance matrix is just the product of diagonal terms

Expected values

The expected value $\mathbb{E}{\mu_{ML}}=\mu$, i.e. here the MLE is unbiased. If you compute the expected value over $\sigma_{ML}^2$ however, you'll find that

$$\sigma_{ML}^2 \frac{N-1}{N} \sigma^2$$

I.e. the ML estimate of the variance underestimates the true variance and is thus a biased estimator.

To find this result, first write out the expected value over the variance by

1) pushing the expectation within the sum 2) expanding the square and again pushing all expectations as far "inside" as possible 3) solving the three component expectations * Make case distinctions where $x_{i} = x_{k}$ and $x_{i} \neq x_{k}$ * in the former case you can expand the expectation due to independence * in the latter case you'll find an expectation over the square of each datapoint, which is $\mu^2$ (see here why * To solve the expectation with a square over a sum, write out the square as an explicit multiplication and again push the expectation inside; you'll be able to reuse a result from a previous term.