Three Levels of MLOps

This Google Cloud source lists three levels of MLOps. The higher models offer significant benefits the more complex a project is, but are also more difficult to implement. MLOps level 2 requires collaboration between data scientists, software/platform engineers, data engineers, devops engineers. MLOps processes also require tighter integration with DevOps tooling.

Level 0: Manual process

  • Target of development is a model, not a pipeline.
  • Little concern for production, so if a model needs to go to production a lot of work is still to be done.
  • No eye for monitoring and retraining.

Suitable for situations where there is little concern for production time for good reasons:

  • Side projects
  • Studying new techniques
  • Kaggle competitions
  • Proof of concepts

Level 1: ML pipeline automation

  • Development target is ML pipeline
  • Reusable pipeline components
  • Code, artifact and experiment tracking
  • Easy retraining
  • Symmetry between development and production environments
  • Monitoring in production

Advantages:

  • Process standardization
  • Rapid prototyping
  • Fast go-to-market
  • Avoid model drift through periodic retraining
  • Possible to learn in production if a feedback loop is included
  • Easy handover of project to others

Level 2: CI/CD pipeline automation

This level builds forth on level 1 and assumes there are existing pipelines in production.

  • Development target are pipeline components
  • Continuous integration: code changes in the components are merged if tests are passed
  • Continuous deployment: if a component is updated, changes are deployed to production if tests are passed
  • Continuous training: release a new model automatically after a trigger, e.g. uploading new data

Advantages:

  • Rapid iteration on production pipelines and models
  • Facilitates A/B testing of changes by deploying to a certain part of the target audience
  • Easier collaboration across larger (>10) teams
  • Continuous improvement in production rather than only shipping periodically