Jensen inequality

For a function $f(x)$, we have $f(x)=f(E[x])$ since the expectation over a constant is just that constant. But for a concave function, Jensen inequality states that $f(E[x]) \leq E[f(x)]$ (reverse for convex function).

Convex function: if you pick any two points and you connect them, then the connecting straight line will always be above the two points themselves.

$f(lambda x_1 + (1-z)x_z)$

Logarithm is not convex, but the negative logarithm is (log is concave).

Some function over an expectation is always bigger or equal than the expectation of the function if the function is concave.

$$f(E[x]) \geq E[f(x)]$$

For an example application, see evidence lower bound (ELBO).

Visual intuition

TODO: more mathematical insight about why Jensen Equality holds.