Backpropagation
During backpropagation you may encounter out of memory errors. Where and why is all this memory used during backpropagation?
Backpropagation computes gradients of expressions through recursive application of the chain rule.
$$\frac{dz}{dx} = \frac{dz}{dy} \frac{dy}{dx}$$
This implies that you compute the partial derivatives for each intermediate computation node. You can pre-compute all gradients. During backpropagation, you can then immediately "fill in" the concrete values from the forward pass, starting from the loss at the end of the network and moving towards the inputs and weights.
This means that you always first need to do a forward pass and memorize the forward values. So this memory requirement cannot be done away with. If your network has a lot of parameters this may cause memory issues, because all parameter values need to be stored in memory.
The backpropagation algorithm is extensible and does not essentially change if your model increases the amount of parameters to the millions.