Short History of Deep Learning Revolution
- 1959 Hubel & Wiesel 1959;
- discovery of preferred orientation of neuronal regions
- and hypothesis of featural hierarchy
- 1980 Fukushima: Neocognitron
- No backprop
- Unsupervised pre-training
- 1998 Yann LeCun: LeNet5; looks like modern neural nets
- On MNIST
- Backprop
- Stochastic Gradient Descent
- 2012: AlexNet by Alex Khrizevsky
- Backprop, SGD, new: ReLu
- New: Dropout
- He had to use 2 GPUs in parallel to fit enough images in memory
- More layers
- ImageNet data, 1.5M images, 1k classes
- 2012-2013 deep learning revolution
- the availability of ImageNet can be seen as transition into modern deep learning
- As well as the availability of better GPUs
- As well as the innovations of AlexNet: ReLu, dropout, etc.
- AlexNet almost halved the error w.r.t. 2011 by going deeper
- This initiated the "Revolution of Depth"