Numerical Computation | Chapter 4 of "Deep Learning" by Ian Goodfellow, Yoshua Bengio, Aaron Courville

Machine learning relies heavily on numerical computations:

  • Optimization
  • Solving linear equations
  • Evaluation continuous functions digitally, i.e. with a finite amount of memory and discretely

Overflow and underflow

Digital approximation of continuous numbers leads to rounding errors. These can accumulate over time, so machine learning algorithms need to handle this in order to work in practice, even if they work in theory.

Underflow occurs when numbers near zero are rounded to zero. Many functions shows really different behavior when an entry is zero rather than a small number. Division by zero is a problematic and obvious example; but also multiplying by 0, or taking a log of 0 are issues.

Overflow: large magnitude numbers are approximated as $\infty$

Example: softmax. $exp(x_i)$ both underflows or overflows for large negative or large positive numbers, leading to an undefined behavior.

Trick: instead evaluate $softmax(z)$ with $z=\mathbf{x} - max_i x_i$ so that the largest argument being 0. Also, at least one term in the denominator will have value of 1. (Why?) Additionally, you can implement the log softmax to prevent underflow in the numerator.

Poor conditioning

Conditioning: how fast does a function's value change for small changes in input?

Note that this issue is not due to rounding errors; it's rather the case that equations with poor conditioning result in more destructive consequences of rounding errors.

When matrices have a large difference between the largest and smallest eigenvalues, then their inversion is very sensitive to errors in the input.

Gradient-Based Optimization