Cross-entropy loss

The cross-entropy loss formula is easily derived by

1) writing out the likelihood of a binary classification problem 2) taking the log likelihood 3) taking the negative log likelihood as loss

See Bishop p. 206 for the gradient, using the derivative of the sigmoid

$$\frac{ \partial \sigma }{ \partial a} = \sigma ( 1 - \sigma)$$

Also see logistic regression.

Binary cross-entropy:

$L(y_i, \hat{y_i}) = -y_i log(\hat{y_i})-(1-y_i)log(1-\hat{y_i})$

If y=1, the loss function requires $\hat{y}$ to also be large. If y=0, the loss function requires $\hat{y}$ to also be small.

Note that the cross-entropy loss includes a negative logarithm. When $\hat{y}=0$ we end up with $log(0)=Inf$.

So in practice, we can solve this by clipping the classifier output to something small like 0.001 or so.

Categorical cross-entropy loss (extension to multiple class)

$L_i=-\sum_j y_{i,j}log(\hat{y}_{i,j})$

Cross entropy and KL-divergence

 KL-divergence_equals_crossentropy.PNG

See Durr-PDL_Chbuilding_loss_functions.md