Kullback-Leibler Divergence

For the discrete case:

$$D_{KL}(P||Q = \sum_{x \in X}) P(x)log(\frac{P(x)}{Q(x)})$$

We see that this expresses the expectation of P(x)/Q(x) under the assumption of distribution P.

This also helps us write the KL-divergence when we are sampling from P:

$$D_{KL}(P||Q) = E_{x~P(x) [ log \frac{P(x)}{Q(x)}]}$$

Note that you always sample from the left argument of the KL-divergence (in this case P), but that you can switch around the terms of the quotient and get the KL-divergence as such:

$$D_{KL}(P||Q) = E_{x~P(x) [ log \frac{P(x)}{Q(x)}]} = - E_{x~P(x) [ log \frac{Q(x)}{P(x)}]}$$

Sometimes this is handy to do.

For example, when we want to investigate the relation of ELBO:

$$\E_{z~q(z)}[log( p(D|z)\frac{p(z)}{q(z)})]$$

and the KL-divergence:

$$D_{KL}( q(z), p(z|D))$$

... then you want to switch the arguments around.

For it's relation to cross-entropy, also see Durr Ch. 4 -building loss functions.