Continuous probability distributions

When you have a discrete parameter space, you can associate a probability to any given value. This is not possible in the case of a continuous probability distribution, as there are infinite infinitesimal possible values. This means that the probability of having one exact value is zero. You can, however, associate a probability with any given bin when you discretize a continuous probability distribution.

The question for any given bin is then: how much probability mass of the total probability mass (i.e. 1) is located in this particular bin of interest?

In the case of a discrete probability distribution this question is trivial, because you can just sum the probability masses corresponding to the bins of interest.

In the case of continuous probability distributions we apply the same principle but deal with the edge case where the bin size is infinitesimally small. To compute the probability mass of a certain region in the continuous parameter space, we thus need a function that expresses the change in probability mass with respect to the chosen range ("bin size").

This is the probability density function .

We can show this is a special case of summing the probability mass of discrete bins.

  1. Let x be some random, continuous variable
  2. We want to compute the probability mass over some bin/interval $[x_i, x_i + \Delta x]$ (ith interval)
  3. The probability mass is $p([x_i, x_i + \Delta x])$ where p(x) is the mass per bin.
  4. We know that when we do this for all intervals, the result sums to 1 because we are dealing with a probability distribution.

$$\sum{i}^{}p([x_i, x_i + \Delta x]) = 1$$

  1. But now we want to express the probability mass as a function of the chosen bin size. The above is equivalent to:

$$\sum{i}^{}\Delta x \frac{p([x_i, x_i + \Delta x])}{\Delta x} = 1$$

  1. In the case that our bin-size approaches zero, for continuous data, we write dx instead and see this sum turn into an integral.

$$\sum{i}^{}\Delta x \frac{p([x_i, x_i + \Delta x])}{\Delta x} = \int{}^{} dx p(x) = 1$$

We thus see that the probability density function p(x) expresses the change in probability mass with respect to any change in the chosen range for the values of random variable x.