ConvNeXt models TensorFlow

100

13

Sayak Paul

Added on September 14, 2024

The project implements ConvNeXt models in TensorFlow and Keras and demonstrates their performance on the ImageNet-1k dataset. Additionally, it provides useful utility scripts.

ConvNeXt models TensorFlow

The project implements ConvNeXt models in TensorFlow and Keras and demonstrates their performance on the ImageNet-1k dataset. Additionally, it provides useful utility scripts.

Sayak Paul

README.md

This repository provides TensorFlow / Keras implementations of different ConvNeXt [1] variants. It also provides the TensorFlow / Keras models that have been populated with the original ConvNeXt pre-trained weights available from [2]. These models are not blackbox SavedModels i.e., they can be fully expanded into tf.keras.Model objects and one can call all the utility functions on them (example: .summary()).

As of today, all the TensorFlow / Keras variants of the models listed here are available in this repository except for the isotropic ones. This list includes the ImageNet-1k as well as ImageNet-21k models.

Refer to the "Using the models" section to get started. Additionally, here's a related blog post that jots down my experience.

Conversion

TensorFlow / Keras implementations are available in models/convnext_tf.py. Conversion utilities are in convert.py.

Models

The converted models are available on TF-Hub.

There should be a total of 15 different models each having two variants: classifier and feature extractor. You can load any model and get started like so:

import tensorflow as tf

model_gcs_path = "gs://tfhub-modules/sayakpaul/convnext_tiny_1k_224/1/uncompressed"
model = tf.keras.models.load_model(model_gcs_path)
print(model.summary(expand_nested=True))

The model names are interpreted as follows:

  • convnext_large_21k_1k_384: This means that the model was first pre-trained on the ImageNet-21k dataset and was then fine-tuned on the ImageNet-1k dataset. Resolution used during pre-training and fine-tuning: 384x384. large denotes the topology of the underlying model.
  • convnext_large_1k_224: Means that the model was pre-trained on the ImageNet-1k dataset with a resolution of 224x224.

Results

Results are on ImageNet-1k validation set (top-1 accuracy).

name original acc@1 keras acc@1
convnext_tiny_1k_224 82.1 81.312
convnext_small_1k_224 83.1 82.392
convnext_base_1k_224 83.8 83.28
convnext_base_1k_384 85.1 84.876
convnext_large_1k_224 84.3 83.844
convnext_large_1k_384 85.5 85.376
convnext_base_21k_1k_224 85.8 85.364
convnext_base_21k_1k_384 86.8 86.79
convnext_large_21k_1k_224 86.6 86.36
convnext_large_21k_1k_384 87.5 87.504
convnext_xlarge_21k_1k_224 87.0 86.732
convnext_xlarge_21k_1k_384 87.8 87.68

Differences in the results are primarily because of the differences in the library implementations especially how image resizing is implemented in PyTorch and TensorFlow. Results can be verified with the code in i1k_eval. Logs are available at this URL.

Using the models

Pre-trained models:

from models.convnext_tf import get_convnext_model

convnext_tiny = get_convnext_model()
print(convnext_tiny.summary(expand_nested=True))

To view different model configurations, refer here.

Upcoming (contributions welcome)

  • Align layer initializers (useful if someone wanted to train the models from scratch)
  • Allow the models to accept arbitrary shapes (useful for downstream tasks)
  • Convert the isotropic models as well
  • Fine-tuning notebook (thanks to awsaf49)
  • Off-the-shelf-classification notebook
  • Publish models on TF-Hub

References

[1] ConvNeXt paper: https://arxiv.org/abs/2201.03545

[2] Official ConvNeXt code: https://github.com/facebookresearch/ConvNeXt

Acknowledgements

Related Content

Kashgari

Kashgari is a production-level NLP Transfer learning framework built on top of tf.keras for text-labeling and text-classification, includes Word2Vec, BERT, and GPT2 Language Embedding.
GitHubUpdated 6 months ago

YoloV7 in Tensorflow.js

This repository is an implementation of Yolov7 using Tensorflow.js. The code runs directly on the browser and the detector was trained on the MS COCO dataset to recognizes up to 80 different classes.
GitHubUpdated 17 months ago

PixPlot

PixPlot combines Tensorflow and WebGL to visualize large image collections in an interactive web environment.
GitHubUpdated 23 months ago

Various ways of serving Stable Diffusion

This repository shows a various ways to deploy Stable Diffusion with TF Serving, Hugging Face Endpoint, and FastAPI.
GitHubUpdated 25 months ago

Hanzi Character Featurizer

In deep learning, the characteristics of Chinese characters (pronunciation characteristics, font characteristics) need to be extracted in many cases. This project provides a general character feature extraction framework, and built-in pinyin, glyph (four-corner code) and radical disassembly features.
GitHubUpdated 48 months ago

We serve cookies on this site to analyze traffic, remember your preferences, and optimize your experience.