Building loss functions with the likelihood approach
Chapter 4 of Probabilistic Deep Learning with Python, Keras, and TensorFlow probability, by Oliver Dürr and Beate Sick
There is "a generally valid approach for deriving the loss function when working with probabilistic models" (94) i.e. the MLE approach.
Recall: likelihood is not a valid probability because likelihoods do not sum up to 1.
Deriving loss functions for classification
Deriving cross-entropy for binary classification
- "the likelihood of a classification model for a given training example ($x_i$, $y_i$) is simply the probability the network assigns to the correct class $y_i$." (100; my emphasis)
- likelihood of whole dataset is product of individual sample probabilities, assuming they are drawn i.i.d.
- If $y \in {0,1}$ then we have a Bernouilli distribution
For Bernouilli-distributed targets, we have
$$w = argmax_w \prod_{i=1}{n} P(Y = y_i, x_i, w)$$
Which we can then split:
$$w = argmax_w \prod_{i with y_i=0}{n} P(Y = 0, x_i, w) \prod_{i with y_i=1}{n} P(Y = 1, x_i, w)$$
To avoid underflow, take the log likelihood instead.
$$w = argmax_w \sum_{i with y_i=0}{n} log P(Y = 0, x_i, w) + \sum_{i with y_i=1}{n} log P(Y = 1, x_i, w)$$
- You may divide by n to get mean log-likelihood per observation if you don't want a dependency on the amount of samples
- DL systems do typically not maximize but minimize; minimizing the negative likelihood is the same as maximizing the regular log likelihood
Adding these two in the mix, we have defined the cross-entropy for a binary classification model.
$$ cross-entropy = - \frac{1}{n} ( \sum_{i with y_i=0}{n} log(p_o(x_j)) + \sum_{i with y_i=1}{n} log(p_1(x_j)) )$$
If you do not use one-hot encoding of two output nodes, but instead a single output node then you can simply rewrite the formula above using $p_1(x_i)$ and $1-p_1(x_i)$.
For more than two classes
CPD: Conditional Probability Distribution
Multinomial distribution assigns a likelihood to each class k. E.g. with $k=10$, we express a probability for each outcome class with constraint $\sum_{k=0}^9 P(Y=k|x,w) = 1$. This is a CPD conditioned on the input data and the current set of weights.
Untrained NNs are initialized with small random weights, which will lead to a more or less uniform distribution over classes.
If we train a bit and pass an image of e.g. label 2, then the NN outputs a multinomial distribution over all classes. The ground truth label can be seen as a multinomial that assigns all probability to the single true class.
Again, recall that the (negative log) likelihood in the loss function only takes into account the probability assigned to the true class.
The NLL then is:
$$ -( \sum_{j with y_j=0} log(p_0(x_j)) + ... + \sum_{j with y_j=K-1})$$
That is, for each class you sum over the input examples $x_j$ and check the probability that is assigned to the correct class. For n input images, the loss per example is just the average of the NLL over the whole batch.
Usually, we one-hot-encode the output so that we can write this briefly as the multiclass cross-entropy:
$$- \frac{1}{n} \sum_{i=1}{n} p_{i}^{true} \dot log(p_i)$$
Notebook: http://mng.bz/OMJK
Insights from the notebook: Keras.md .
My categorical cross-entropy numpy solution:
NLL = -1 * np.mean(np.sum(np.multiply(np.log(Pred_prob), Y_train), axis=1))
Answer from PDL book:
loss=np.zeros(len(X_train))
Y=np.argmax(Y_train,axis=1)
for i in range(0,len(X_train)):
loss[i]=np.log(Pred_prob[i][Y[i]])
-np.sum(loss)/len(X_train
Relationship between NLL, cross-entropy, KL-divergence
Entropy: $-E_p( log(p) ) = - \sum_k p_k * log(p_k)$
Cross-entropy: $H(P,Q)=-E_p(log(q)) = -\sum_k p_k * log(p_k)$
We have two distributions q and p and then calculate the expected value of log(q) given distribution p.
In regression, we have MSE. If we want to do something similar in classification, we cannot just take the difference of the predicted value and the true value, because we are now working with distributions.
This is what the KL-divergence measures.
The KL-divergence is the expected value of the differences of log probabilities.
So if the KL-divergence is zero, the expected difference between target and true distribution is nil, which would be the ideal situation.
The KL-divergence is in fact the same as the cross-entropy!
So when we minimize the cross-entropy, we are also minimizing the KL-divergence!
So "KL divergence is to classification what the MSE is to regression" (p.110)
Also note that both cross-entropy and KL-divergence are not symmetric. (Swapping will also give numerical troubles since $p_{true}$ will mostly be zero; so you'll end up taking the log of 0 ).
Deriving a loss function for regression
MSE results from the max likelihood principle in regression.
We can also use the max likelihood principle to design a loss function that works with heteroscedacity .
In regression, the NN output will not be a distribution over class labels, but the parameters of a fitted continuous probability distribution. That could be a Gaussian distribution around a mean i.e. an expected value.
Q: Why is that Gaussian a "conditional" distribution?
Well, the estimate for $\mu$ depends on the data x!
Likelihood: $L(a,b) = \prod_{i=1}^n f(y_i;(a x_i _+ b), \sigma)$
$\hat(a)$ and $\hat(b)$ are the maximum likelihood estimates for which the likelihood is maximal.
The argmax over this likelihood can be easily rewritten to an argmin over the MSE assuming $\sigma$ is constant. See p. 117-118.
A Gaussian is not always the most appropriate. E.g. it may lead to negative estimates, which is not suitable for each problem. We variance can also not always be assumed to be fixed.
If you also want to model the variance, you need another NN output node. The output should be interpreted e.g. as $log(\sigma)$ so that the output variance will be non-negative.
The only thing this prevents is a closed form solution. Before, showing that the NLL for regression resulted in MSE assumed constant variance. So having non-constant variance prevents such a derivation. However, this is not problematic at all if we don't only rely on traditional statistics. It's just that now we need to do stochastic gradient descent, i.e. optimization.
N.B. to optimize the variance you don't need ground truth labels! You find the best variance automatically using the maximum likelihood approach: e.g., if your data is spread out but your variance low, you'll have a low max joint probability.