SML lecture 8: Neural Networks
- Feed forward networks
- Gradient descent optimization (stochastic/conjugate)
- Backpropagation
- Regularization in neural networks
- Convolutional neural networks
Feed-forward neural networks
Feed-forward networks are basically multi-layer perceptrons.
In logistic regeression we have to choose a finite number of fixed basis functions in advance.
This suffers from the curse of dimensionality.
Two different approaches:
- Gaussian Processes; implicitly allow 'infinitely many' basis functions
- Full Bayesian treatment
- Difficult for large data size N
-
Multi-layer Perceptrons: makes the basis functions adaptive with parameters that can be learned along coefficients $w_j$
- Training is slow
- But prediction is fast, since you end up with basis functions that are fixed after they are learned.
- Can handle a lot of data (cf. Gaussian Processes).
-
Activations are linear combinations of input
- Superscript indicates layer
- $w_{ji}^{x}$ are weights
- $w_{j0}^{x}$ are bias
- h(.) is a non-linear activation function
- $z_{j}^{1}=h(a_j^1)$ are hidden units values
- Hidden unit values are inputs to the second hidden layer $M^{(2)}$
The output activation function $f(a_k^{(out)})$ requires more customization depending on your problem:
- Linear for regression problem
- Sigmoid for binary classification
- Softmax for K outputs for K classes
The choice of hidden activation function h is more of a modeling choice.
Option: skip-layer connections from input to second hidden layer. This can help training the network.
With only skip-connections, you reduce the network to logistic regression again. This is very easy to "train". So using skip-connections can help balance a trade-off with perforamance and training speed.
Slide 5: f activation is omitted because it's just the identity function.
Properties
Cf. 5.1 p.227-231
Good approximation properties.
A two layer network with linear outputs f(x)=x and h(x)=tanh(x) activation functions can approximate any function given enough hidden inputs.
E.g. an quadratic function or sign can be approximated close to perfect, even though the MLP does not contain any quadratic term etc.
Error function and output coding
We apply the typical ML approach: minimize E(w).
Depending on the problem we use different error functions and output activations.
In regression, we assume mean+Gaussian noise; we've seen this amounts to minimizing the sum of squared error; linear output function.
Classification: entropy error function, since outputs are probabilities.
Error landscape
Because we have non-linear activation functions, our error function E(w) will be non-quadratic in the parameters w.
Gradient descent approach: we take steps in the direction of the negative gradient.
Gradient descent optimization
Error functions are defined per input data point/pattern. The total error is simply the sum of errors on the individual patterns. We saw before that this held for regression. Error function is basically the log likelihood (?); when then taking the log over a product we end up with a sum. This allows for batch learning: update based on all patterns each time step.
We also have on-line learning flavours:
- sequential gradient descent: cycle through data point by point
- stochastic gradient descent: sample with replacement from a large dataset and update on that step.
- On average stochastic gradient descent still moves in the same direction as batch.
- Random fluctuations may help escape local minima
We can also take intermediate approaches: (e.g. minibatch learning).
Momentum
See slide 10-11; there are various pitfalls.
With incorrect learning rates, we may have divergence and oscillations (sign-flipping).
If we satisfy all constraints on learning rates to avoid this, we may end up with a very small learning rate. But then convergence is often very slow.
We can use momentum to alleviate this issue.
Alternatives
- Newtons method
Discussed before; review.
H must be such that eta times lambda is such that you end up exactly in a minimum in both directions. Lambda falls out of the equation so you get to a minimum in one step.
- Line search / steepest descent method
First you traverse to an optimal point in one direction; then you do the other in an orthogonal direction.
REVIEW Slide 14 for why exactly this is the case. Has something to do with inner product of some gradients being 0.
Orthogonal updates is not an advantage, because this may not be the optimal direction to traverse in. It's a suboptimal constraint.
Conjugate gradient descent
This is designed to avoid the problem with orthogonal updates in steepest gradient descent.
Hessian is contains second-order derivatives, so gradients of gradients.
We get rid of the Hessian somehow using the Polak-Ribiere rule.
Q: if multiplication delta times nabla error is 0 in both direction, what does that mean?
A: orthogonal in 2D -> zo gradient is 0. We are in a local optimum.
How to deal with beta?
Polak-Ribiere rule
Conjugate directions can be computed without computing the Hessian matrix. E.g. with Polak-Ribiere rule.
This is a batch method; this doesn't work with e.g. stochastic gradient descent.
Error back propagation
We treat biases as weights now (see footnote slide 17).
Again, the error is the sum of the error per pattern.
Backprop computes partial derivatives w.r.t. all weights and biases by iteratively applying the chain rule. This is because networks are chained/nested transformations.
We start with a forward pass because we need to know all activations a and hidden unit values z.
The error only depends on $w_{ji}^m$ via activation $a_j^m$.
So to compute the partial derivate of error w.r.t. weights we apply the chain rule: partial derivative w.r.t activation times partial derivative of activation w.r.t. the weights.
We also see that the error depends on $a_j^m$ only via activation in layer $a_k^{(m+1)}$.
N.B. I changed the order of slides here.
Regularization
Overfitting can be battled using:
- Architecture (e.g. number of hidden units)
- Weight decay
- Setting hyperparameters
Early stopping
Stop when validation error starts increasing.
REVIEW: Early stopping when starting with small initial weights has a similar effect of weight decay.
Weight sharing
Weight sharing is performed in Convolutional neural networks.
Images have very high input dimensionality, so MLP is not suitable. We also need translation invariance.
In CNNs we have:
- local receptive fields
- REVIEW weight sharing
- So deltas are also shared
Filters are learned using backpropagation, but due to weight sharing (i.e. we have a constraint on weights) the backpropagation formula for this looks slightly different.