Cross-entropy loss
The cross-entropy loss formula is easily derived by
1) writing out the likelihood of a binary classification problem 2) taking the log likelihood 3) taking the negative log likelihood as loss
See Bishop p. 206 for the gradient, using the derivative of the sigmoid
$$\frac{ \partial \sigma }{ \partial a} = \sigma ( 1 - \sigma)$$
Also see logistic regression.
Binary cross-entropy:
$L(y_i, \hat{y_i}) = -y_i log(\hat{y_i})-(1-y_i)log(1-\hat{y_i})$
If y=1, the loss function requires $\hat{y}$ to also be large. If y=0, the loss function requires $\hat{y}$ to also be small.
Note that the cross-entropy loss includes a negative logarithm. When $\hat{y}=0$ we end up with $log(0)=Inf$.
So in practice, we can solve this by clipping the classifier output to something small like 0.001 or so.
Categorical cross-entropy loss (extension to multiple class)
$L_i=-\sum_j y_{i,j}log(\hat{y}_{i,j})$