Bengali Digit Recognition

@bengali

Trying to answer the question:

What is in your opinion the most interesting work from the literature for this challenge?

"AI Learns to Recognize Handwritten Digits: Bengali.AI Computer Vision Challenge 2018"

"a massive void was present in terms of Bengali handwritten digits dataset, let alone characters." (1)

NumbtaDB: well structured open dataset for Bengali handwritten digits. Manually cleaned and standardized (5 month process).

For the Kaggle competition multiple sources were used.

They did augmentations on parts of the ... test data. Why on the test data?

Augmentations (done by competition organizers or by competitors?):

  • Spatial transformations: Rotation, Translation, Shear, Height/Width Shift, Channel Shift, Zoom
  • Brightness, Contrast, Saturation, Hue shifts, Noise
  • Occlusions
  • Superimposition (simulate text being visible from the other side)

No augmentation performed on the training set. Why not?

Winning algorithms:

Place 1: used Resnet-34 and Resnet-50 + transfer learning + 6-way ensemble + PyTorch

For ResNet, see: Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.

Place 2: ResNeXt-50. Notable insight: had a data generation scheme called Overlay.

Place 3: 4-way ensemble, no transfer learning, trained end to end (despite using transfer learning?) DenseNet 121 with pre-trained weights from ImageNet.

Data Augmentation: Gaussian Blur, Additive Gaussian Noise, Channel-wise random scaling, Affine scaling, Translation, Rotation and Shear, Contrast Normalization, Additive Pepper noise.

Issues with label noise

@label @noise

"Large datasets are prone to having label noise both in the training and testing datasets"

See: Benoît Frènay and Michel Verleysen. Classification in the presence of label noise: a survey. IEEE transactions on neural networks and learning systems, 25(5):845–869, 2014

"Deep learning models trained with large supervised datasets are robust to mislabeled data in the training set"

URLs for references

Bengali Handwritten Character Classification using Transfer Learning on Deep Convolutional Neural Network

Source: https://arxiv.org/html/1902.11133v1

Code: https://github.com/swagato-c/bangla-hwcr-present

Abstract:

We employ Transfer Learning on ResNet 50, a state-of-the-art deep Convolutional Neural Network Model, pretrained on ImageNet dataset. We also use other techniques like a modified version of One Cycle Policy, varying the input image sizes etc. to ensure that our training occurs fast. We use the BanglaLekha-Isolated Dataset for evaluation of our technique which consists of 84 classes (50 Basic, 10 Numerals and 24 Compound Characters). We are able to achieve 96.12% accuracy in just 47 epochs on BanglaLekha-Isolated dataset. When comparing our method with that of other researchers, considering number of classes and without using Ensemble Learning, the proposed solution achieves state of the art result for Handwritten Bengali Character Recognition.

Used dataset (ILSVRC):

The BanglaLekha-Isolated Dataset as described in Section III-C was used for evaluation because of its large sample size, inferior data quality ( when compared to other available datasets which suits our purpose since it helps us better generalize ) and large variance and also because the output classes are balanced. The dataset consisted of 84 characters, which consisted of 10 Numerals, 50 basic characters and 24 Frequently Used Compound Characters.

Dit snap ik niet: je wilt toch pre-training met ImageNet en dan fine-tunen met de gepaste dataset?

"The CNN Model proposed is ResNet-50 pre-trained with ILSVRC dataset[1] and fine-tuned using Transfer Learning."

Objective:

The objective of the paper is to use current best practices to train the model effectively and in as few iterations as possible. For better performance, optimization techniques like a slightly modified version of One Cycle Policy was used. In just 47 epochs, we were able to achieve 96.12% accuracy on BanglaLekha-Isolated dataset. The dataset had 84 classes, each for every character in the BanglaLekha-Isolated Dataset.

Literature review

  • Lists several feature extraction methods, e.g. K-NN, PCA.
  • "With the advent of deep learning, it was possible to partly or fully eliminate the need for feature extraction."

Architecture choice

  • pretrained Deep CNN ResNet-50
  • chosen because of heavy reliance on 1) Batch Normalization and 2) Dropout
    • regularization: against overfitting.
  • Residual net: identity mappings/skip-connections for easier training without vanishing gradient problems.
  • Pretrained on Imagenet Large Scale Visual Recognition Challenge ( ILSVRC )
  • Transfer learning (see below)
  • AdamW optimizer (fine-tune learning rate per parameter)
    • AdamW is Adam with fixed weight decay issues!
    • Cf. https://www.fast.ai/2018/07/02/adam-weight-decay/
    • Alternative: SGD with momentum
    • Adam generally requires more regularization than SGD, so be sure to adjust your regularization hyper-parameters when switching from SGD to Adam.
  • Learning rate scheduler (modified One-Cycle-Policy )
    • or "superconvergence"
    • first linearly increase learning rate
    • then anneal with cosine

Preprocessing

  • colour-inversion
  • noise removal
  • edge-thickening filters.
  • Resize tot 224x224 (net als wij, vanwege ImageNet).
    • Verschillende input groottes, "varied from 110 px * 110 px to 220 px * 220 px."

Training

fast.ai on top of PyTorch.

  • "Note that for 224 px * 224 px, the batch size was reduced to 64. After this the bottom layers were frozen again and the model is finetuned with 128 px * 128 px images."
  • Fascinerend: ze trainen in verschillende fasen, waarbij ze de image size en batch size aanpassen (omgekeerd evenredig)

  • NO ensemble learning (which is common in most solutions we see)

Transfer learning

  • 84 output classes

  • Thus, the initial model accepted images of size 224 * 224 px and classified the images to 1000 categories. We employed Transfer Learning [32] to use this pre-trained model and modified it to classify 84 classes instead of 1000 by removing the Fully Connected (FC) layers of the original model and substituting them with new layers as given in Figure 4.

Zie figure 4 voor vervangende lagen! Kunnen wij imiteren!

Limitations

The Bengali language has more conjunct-consonant characters than just the 24 frequently used ones which are present in the BanglaLekha-Isolated (Section III-C) dataset. This means that even though the character set present in BanglaLekha-Isolated Dataset represents the major chunk of the entire Bengali corpus, it doesn't contain every single character. Hence, the performance presented is for those 84 characters only. However, to the best of our knowledge, there are no datasets which not only contains sufficient samples of all the characters in Bengali Language but is also of good quality.