Short History of Deep Learning Revolution

  • 1959 Hubel & Wiesel 1959;
    • discovery of preferred orientation of neuronal regions
    • and hypothesis of featural hierarchy
  • 1980 Fukushima: Neocognitron
    • No backprop
    • Unsupervised pre-training
  • 1998 Yann LeCun: LeNet5; looks like modern neural nets
    • On MNIST
    • Backprop
    • Stochastic Gradient Descent
  • 2012: AlexNet by Alex Khrizevsky
    • Backprop, SGD, new: ReLu
    • New: Dropout
    • He had to use 2 GPUs in parallel to fit enough images in memory
    • More layers
    • ImageNet data, 1.5M images, 1k classes
  • 2012-2013 deep learning revolution
    1. the availability of ImageNet can be seen as transition into modern deep learning
    2. As well as the availability of better GPUs
    3. As well as the innovations of AlexNet: ReLu, dropout, etc.
    4. AlexNet almost halved the error w.r.t. 2011 by going deeper
    5. This initiated the "Revolution of Depth"