MLOps for Vision Models from ๐Ÿค— with TFX

115

18

Chansung Park

Added on September 14, 2024

This repository shows how to build Machine Learning pipeline for a vision model (TensorFlow) from ๐Ÿค— Transformers using the TensorFlow Ecosystem.

MLOps for Vision Models from ๐Ÿค— with TFX

This repository shows how to build Machine Learning pipeline for a vision model (TensorFlow) from ๐Ÿค— Transformers using the TensorFlow Ecosystem.

Chansung Park

README.md

MLOps for Vision Models (TensorFlow) from ๐Ÿค— Transformers with TensorFlow Extended (TFX)

This repository shows how to build Machine Learning pipeline for a vision model (TensorFlow) from ๐Ÿค— Transformers using the TensorFlow Ecosystem. In particular, we use TensorFlow Extended(TFX), and there are TensorFlow Data Validation(TFDV), Transform(TFT), Model Analysis(TFMA), and Serving(TF Serving) besides TensorFlow itself internally involved.

NOTE: This is a follow-up projects of "Deploying Vision Models (TensorFlow) from ๐Ÿค— Transformers" which shows how to deploy ViT model locally, on kubernetes, and on a fully managed service Vertex AI.

We will show how to build ML pipeline with TFX in a step-by-step manner:

  • Basic ( )

    • as the first step, we show how to build ML pipeline with the most basic components, which are ExampleGen, Trainer, and Pusher. These components are responsible for injecting raw dataset into the ML pipeline, training a TensorFlow model, and deploying a trained model.

  • Intermediate ( )

    • as the second step, we show how to extend the ML pipeline from the first step by adding more components, which are SchemaGen, StatisticsGen, and Transform. These components are responsible for analyzing the structures of the dataset, analyzing the statistical traits of the features in the dataset, and data pre-processing.

  • Advanced Part 1 ( )

    • as the third step, we show how to extend the ML pipeline from the second step by adding more components, which are Resolver and Evaluator. These components are responsible for importing existing Artifacts (such as previously trained model) and comparing the performance between two models (one from the Resolver and one from the current pipeline run).

  • Advanced Part 2 ( )

    • as the fourth step, we show how to extend the ML pipeline from the third step by adding one more additional component, Tuner. This component is responsible for running a set of experiments with different sets of hyperparameters with fewer epochs, and the found best hyperparameter combination will be passed to the Trainer, and Trainer will train the model longer time with that hyperparameter combinations as the starting point.

  • ๐Ÿค— Hub Integration ( )

    • in this optional step, we show how to use custom TFX components for ๐Ÿค— Hub. In particular, we use HFModelPusher to push currently trained model to ๐Ÿค— Model Hub and HFSpacePusher to automatically deploy Gradio application to ๐Ÿค— Space Hub.

Acknowledgements

We are thankful to the ML Developer Programs team at Google that provided GCP support.

Related Content

YoloV7 in Tensorflow.js

This repository is an implementation of Yolov7 using Tensorflow.js. The code runs directly on the browser and the detector was trained on the MS COCO dataset to recognizes up to 80 different classes.
GitHubโ€ขUpdated 17 months ago

Semantic Sementation model within ML pipeline

This repository shows how to build a Machine Learning Pipeline for Semantic Segmentation task with TensorFlow Extended(TFX) and various GCP products such as Vertex Pipeline, Training, and Enpoint.
GitHubโ€ขUpdated 21 months ago

Complete MLOps System Workflow with GCP and TFX

This repository shows how to build a complete MLOps system with TensorFlow eXtended(TFX) and various GCP products.
GitHubโ€ขUpdated 30 months ago

Usage of TF based SegFormer in transformers

This repository demonstrates how to use TensorFlow based SegFormer model in ๐Ÿค— transformers package with Jupyter Notebook and Gradio application.
GitHubโ€ขUpdated 32 months ago

Continuous Adaptation for Machine Learning System

MLOps system evolves according to the changes of the world, and that is usually caused by data/concept drift. This project shows how to combine two separate pipelines, one for batch prediction and the
GitHubโ€ขUpdated 39 months ago

We serve cookies on this site to analyze traffic, remember your preferences, and optimize your experience.

AboutยทTermsยทPrivacyยทFeedback