Machine learning devops curriculum

Coding best practices

Topics:

Version control skills

Preparing code for production

  • Handling errors
  • Testing specifically for data science and machine learning
  • Logging best practices
  • Model drift
    • Watch for change in the distributions of business metrics, model metrics, or input features.
  • Automated vs. non-automated retraining
    • automated retraining is suitable when the model architecture does not change but you need frequent updates to counter model drift (domain specific)
    • automated training is cool in other scenarios, but e.g. hard to implement e.g. when also adding new features in the mix
    • non-automated when there is little to no model drift, or when model architecture of features need changes

Data science questions to ask in an enterprise context

Data Science Definition of Done

What is data science's definition of done (DOD)

MLOps: reproducible workflows with machine learning pipelines

TODO refactor in MLOps note.

By using MLOps we can design an orchestrated, tracked and versioned workflow that can be easily reproduced and inspected.

What is MLops?

MLOps is:

A set of best practices and methods for an efficient end-to-end development and operation of performant, scalable, reliable, automated and reproducible ML solutions in a real production setting.

MLOps drastically reduces time spent in orchestration, infrastructure and deployment. Without MLOps in an enterprise setting, you'll end up spending 90% of your time on other things than modelling, dealing with questions like:

  • Where is my data? How do I get it?
  • How do I train my model with enough computational resources?
  • How I transfer knowledge to other people?
  • How do I track experiments?
  • What did I do a month ago that resulted in this performance?

MLOps helps:

  • Remove friction and time spent on infrastructure
  • Reduce time spent on orchestration
  • More reliability in production
  • Faster time to market
  • Cheaper operations
  • More transparent/visibility of data science process

MLOps is always useful, but it's not always necessary such as in a hobby project. But you develop hugely from MLOps in... :

  • In any production setting
  • When reuse is key e.g. because your team works on similar products and projects
  • When documentation and collaboration are important
  • For complex projects

Machine Learning Pipeline

A pipeline is a sequence of components where each component is independent, modular, and reusable.

You can represent a pipeline as a Directed Acyclic Graph.

Context and history

50% of ML projects do not make it to production (source).

Early ML systems had a massive technical dept. See the 2015 paper "Hidden Technical Debt in Machine Learning Systems" by Sculley et al. link. Abstract:

Machine learning offers a fantastically powerful toolkit for building useful complex prediction systems quickly. This paper argues it is dangerous to think of these quick wins as coming for free. Using the software engineering framework of technical debt, we find it is common to incur massive ongoing maintenance

This paper can be considered the start of MLOps as its own discipline.

Data science workflows: academy vs. the wild

The data science process ideally consists of:

  • Create a clean, organized, reproducible, end-to-end machine learning pipeline from scratch
  • Transform, validate and verify your data (avoid “garbage in, garbage out”)
  • Run experiments, track data, code, and results
  • Select the best performing model and promote it to production
  • Create an artifact ready for deployment, and release your final pipeline
  • Monitor

In an Academic/Kaggle workflow we get a ready data set and a clear metric to optimize, have to worry less about production context and computational costs, and can thus spend most of our time doing elaborate modeling.

In "the wild" you have to do more. At the start, there is work in getting the problem definition clear. This includes asking ourselves the question whether we actually need ML. If software without ML solves the task, this is a considerably cheaper and less complex option. There is also significant work in collecting a data set, even before the data cleaning and preprocessing that is needed specifically for ML.

A trained ML system is also not the end point in an enterprise context. You will also need to incorporate the model in some sort of production system that provides answers to customers. You will also need to constantly monitor that the model in production is still performing and delivering the value it should.

Three levels of MLOps

See three levels of MLOps

MLOps tools

This space is very crowded with tools. To just identify some main tools and some important MLOps aspects:

  • Code tracking: GitHub, GitLad, BitBucket ...
  • Experiment tracking: Weights & Biases, MLflow, Neptune, Comet ...
  • Artifact tracking: Weights & Biases, MLflow, Neptune, Comet ...
  • Model repository: Weights & Biases, MLflow, Paperspace ...
  • ML pipelines: MLflow + Hydra, Kubeflow, Metaflow, TFX ...
  • Environment isolation: conda, docker ...
  • Orchestration: MLflow + Hydra, K8s, AWS, GCP, Azure
  • Modeling: scikit-learn, Pytorch, TensorFlow

These tools are battle-tested, simple, free or freemium, and fit in the MLOps big picture.

MLflow

See MLflow.

Feature stores

See feature store.

Data Exploration and Preparation

See EDA exploratory data analysis

Data Validation

See testing in data science

Data Provenance

Data provenance is the complete record of data for the purpose of accountability, including:

  • Origin: how was the data generated and where does it come from?
  • Movement: what other systems has the data went through?
  • Manipulation: which transformations have been applied to the data?

Data Version Control (DVC)

DVC is more than just version control on data (it does not do version control itself, but instead leverages git for this).

  • Managing data
  • Managing models
  • Creating and running pipelines
  • Tracking experiments

All artifacts are stored elsewhere, DVC just helps tracking them.

Inference pipeline as an artifact

An inference artifact is not just saving the trained model, but also the preprocessing steps that were used to create the data that was input to that specific trained model. If you do not include the preprocessing step as an inference artifact, your results will not be fully reproducible.

This is also very important for achieving development/production symmetry. By also saving the preprocessing steps as an inference artifact, we ensure that the same preprocessing that was used during the training pipeline is used in the inference pipeline. So in short, we need an inference pipeline rather than just a model to achieve development/production symmetry by doing the same preprocessing in both.

But how to store preprocessing steps as a single artifact? Examples:

  • Chain scikit-learn transformers together into a single pipeline object that can be exported
  • Pytorch: include the needed transformations in the network using "scriptable transforms" and exporting with torch.script
    • Use Sequential() class containing transforms, such as transforms.Resize([256, 256])
    • Export: s = torch.jit.script(pipe) and s.save("inference_artifact.pt")
  • scikit-learn's ColumnTransformer is useful to create a single pipeline with sub-pipelines for different columns in a pandas dataframe
    • e.g. one for categorical variables, one for numerical variables, and then pipelines for columns that require exceptional handling

If the preprocessing stage includes feature engineering, consider using a feature store.

Exporting using MLflow

A very good option for exporting inference pipelines is using the MLflow convention, which has several benefits:

  • Standard format accepted by many downstream tools
  • Several subformats or "flavors" in the same model export to suit various downstream tools
    • for sklearn you have mlflow.sklearn.save_model(); similar for pytorch, ONNX, keras, generic python function flavor for customization
  • Easily read in again when using mlflow for orchestration
  • Environment isolation because the model export contains the used environment
    • The right conda.ymls are included by default, but you can customize using the conda_env flag.
  • A data signature containing the data schema for input and output data can be provided so downstream tools can catch data schema problems
    • You can use the mlflow.models.infer_signature() function
  • Input examples can be included in the export so you can easily test the inference pipeline
  • Exports can be transformed into a Docker image with REST API to run the model/inference pipeline!

Example:

mlflow.sklearn.save_model(
  pipe,  # sklearn pipeline
  export_path,  # Path to a directory for the produced package
  signature=signature, # input and output schema
  input_example=X_test.iloc[:5]  # the first few examples
)

Test what you deploy

After exporting the model, evaluate the inference artifact on the test data set! Only if performance is sufficient, we should select the best model and mark it for production, e.g. with a particular unique tag such as prod. (Unique meaning that only one model will have this tag.)

Experimentation

The right mindset

  • Iteration: it's normal
    • E.g. you might even find bugs in the preprocessing steps and have to go back a few steps
  • Iteration requires time, so factor this in within the project deadlines
  • Be systematic: change one thing at a time
  • "Nothing is lost": every experiment matters, knowing what does not work is also valuable
  • Have a criterium for when to stop because there are diminishing returns when experimenting
  • Nothing is fixed: even when you have a trained model and are at the last step of the model development, it may be more beneficial to go all the way to the first step and collect more and/or higher quality data instead of developing a complex solution that may not even work.

Reproducible experiments

To reproduce experiments, three things are essential (one bonus):

  • The data needs to be versioned so you can always retrieve the input data for any experiment
  • The code needs to be versioned for the same reason
  • Track all experiments from the beginning of the project
  • If randomness is involved (almost always, e.g. train-validation split) then setting seeds appropriately is also needed for reproducibility

Configuration e.g. with Hydra

Releasing a pipeline

Performance Testing, incl. bias

Monitoring performance after deployment

  • Record keeping of processes, such as data ingestion
    • Source data files, ingestion date, size of the input datasets, details on the cleaning process
  • Re-training and re-deploying
  • Automatic scoring
  • Monitoring model drift
  • Hypothesis testing in model comparisons
  • Automatic reporting using APIs

Operational:

  • Timing operational processes to check for latency issues
  • Checking and resolving module dependency issues
  • Checking and resolving data stability issues

Sources

MLOps

Evaluation: