Vision Transformer TF-Hub Application

Description

This repositories show how to fine-tune a Vision Transformer model from TensorFlow Hub on the Image Scene Detection dataset.

Dataset Used

A newly collected Camera Scene Classification dataset consisting of images belonging to 30 different classes. This dataset is the part of the competition which is Mobile AI Workshop @ CVPR 2021. You can find the dataset details here.

Models

These models are available on TensorFlow Hub for Vision Transformer.

Image Classifiers

• ViT-S16
• ViT-B8
• ViT-B16
• ViT-B32
• ViT-L16
• ViT-R26-S32 (light augmentation)
• ViT-R26-S32 (medium augmentation)
• ViT-R50-L32

Feature Extractors

• ViT-S16
• ViT-B8
• ViT-B16
• ViT-B32
• ViT-L16
• ViT-R26-S32 (light augmentation)
• ViT-R26-S32 (medium augmentation)
• ViT-R50-L32

Note: As we want to fine-tune our model so we used the feature-extractor model and build the image classifier.

Benchmark Results

Note: Last three results are benchmarked during thr CVPR Competition. You can find the repository here.

Notebooks

✅ ViT S/16
✅ ViT R26-S/32 (Light Augmentation)
✅ ViT R26-S/32 (Medium Augmentation)
✅ ViT B/32
⬜ ViT R50-L/32
⬜ ViT B/16
⬜ ViT L/16
⬜ ViT B/8

Links

Each directory of model contains the particular notebook, python script, metric graph, train-logs(in .csv) and TensorBoard callbacks.

References

[1] An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale by Dosovitskiy et al.

[2] How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers by Steiner et al.

[3] Vision Transformer GitHub

[4] jax2tf tool

[5] Image Classification with Vision Transformer in Keras

[6] ViT-jax2tf

[7] Vision Transformers are Robust Learners, Repository

[8] Vision Transformer TF-Hub Model Collection

Acknowledgements

• Thanks to Sayak Paul for building the models of ViT so that we can use Vision Transformer in a straight way.
• Thanks to the authors of Vision Transformers for their efforts put into open-sourcing the models.

Contributors