Machine learning devops curriculum
Coding best practices
Topics:
- Writing clean and modular code
- Refactoring code
- Optimizing code for efficiency
- For time and for storage
- Writing code documentation
- Inline strings, docstrings, project documentation
- Following PEP8 guidelines and using linting
- E.g. autopep8 and pylint
autopep8 --in-place --aggressive --aggressive script_name.py
Version control skills
- Git add, commit, push
- Branching
- Common version control workflows
- Code reviews
Preparing code for production
- Handling errors
- Testing specifically for data science and machine learning
- Logging best practices
- Model drift
- Watch for change in the distributions of business metrics, model metrics, or input features.
- Automated vs. non-automated retraining
- automated retraining is suitable when the model architecture does not change but you need frequent updates to counter model drift (domain specific)
- automated training is cool in other scenarios, but e.g. hard to implement e.g. when also adding new features in the mix
- non-automated when there is little to no model drift, or when model architecture of features need changes
Data science questions to ask in an enterprise context
- Data science questions to ask in an enterprise context
- Elements of a succesful ML project
- Who are ML stakeholders
Data Science Definition of Done
What is data science's definition of done (DOD)
MLOps: reproducible workflows with machine learning pipelines
TODO refactor in MLOps note.
By using MLOps we can design an orchestrated, tracked and versioned workflow that can be easily reproduced and inspected.
What is MLops?
MLOps is:
A set of best practices and methods for an efficient end-to-end development and operation of performant, scalable, reliable, automated and reproducible ML solutions in a real production setting.
MLOps drastically reduces time spent in orchestration, infrastructure and deployment. Without MLOps in an enterprise setting, you'll end up spending 90% of your time on other things than modelling, dealing with questions like:
- Where is my data? How do I get it?
- How do I train my model with enough computational resources?
- How I transfer knowledge to other people?
- How do I track experiments?
- What did I do a month ago that resulted in this performance?
MLOps helps:
- Remove friction and time spent on infrastructure
- Reduce time spent on orchestration
- More reliability in production
- Faster time to market
- Cheaper operations
- More transparent/visibility of data science process
MLOps is always useful, but it's not always necessary such as in a hobby project. But you develop hugely from MLOps in... :
- In any production setting
- When reuse is key e.g. because your team works on similar products and projects
- When documentation and collaboration are important
- For complex projects
Machine Learning Pipeline
A pipeline is a sequence of components where each component is independent, modular, and reusable.
You can represent a pipeline as a Directed Acyclic Graph.
Context and history
50% of ML projects do not make it to production (source).
Early ML systems had a massive technical dept. See the 2015 paper "Hidden Technical Debt in Machine Learning Systems" by Sculley et al. link. Abstract:
Machine learning offers a fantastically powerful toolkit for building useful complex prediction systems quickly. This paper argues it is dangerous to think of these quick wins as coming for free. Using the software engineering framework of technical debt, we find it is common to incur massive ongoing maintenance
This paper can be considered the start of MLOps as its own discipline.
Data science workflows: academy vs. the wild
The data science process ideally consists of:
- Create a clean, organized, reproducible, end-to-end machine learning pipeline from scratch
- Transform, validate and verify your data (avoid “garbage in, garbage out”)
- Run experiments, track data, code, and results
- Select the best performing model and promote it to production
- Create an artifact ready for deployment, and release your final pipeline
- Monitor
In an Academic/Kaggle workflow we get a ready data set and a clear metric to optimize, have to worry less about production context and computational costs, and can thus spend most of our time doing elaborate modeling.
In "the wild" you have to do more. At the start, there is work in getting the problem definition clear. This includes asking ourselves the question whether we actually need ML. If software without ML solves the task, this is a considerably cheaper and less complex option. There is also significant work in collecting a data set, even before the data cleaning and preprocessing that is needed specifically for ML.
A trained ML system is also not the end point in an enterprise context. You will also need to incorporate the model in some sort of production system that provides answers to customers. You will also need to constantly monitor that the model in production is still performing and delivering the value it should.
Three levels of MLOps
MLOps tools
This space is very crowded with tools. To just identify some main tools and some important MLOps aspects:
- Code tracking: GitHub, GitLad, BitBucket ...
- Experiment tracking: Weights & Biases, MLflow, Neptune, Comet ...
- Artifact tracking: Weights & Biases, MLflow, Neptune, Comet ...
- Model repository: Weights & Biases, MLflow, Paperspace ...
- ML pipelines: MLflow + Hydra, Kubeflow, Metaflow, TFX ...
- Environment isolation: conda, docker ...
- Orchestration: MLflow + Hydra, K8s, AWS, GCP, Azure
- Modeling: scikit-learn, Pytorch, TensorFlow
These tools are battle-tested, simple, free or freemium, and fit in the MLOps big picture.
MLflow
See MLflow.
Feature stores
See feature store.
Data Exploration and Preparation
See EDA exploratory data analysis
Data Validation
Data Provenance
Data provenance is the complete record of data for the purpose of accountability, including:
- Origin: how was the data generated and where does it come from?
- Movement: what other systems has the data went through?
- Manipulation: which transformations have been applied to the data?
Data Version Control (DVC)
DVC is more than just version control on data (it does not do version control itself, but instead leverages git for this).
- Managing data
- Managing models
- Creating and running pipelines
- Tracking experiments
All artifacts are stored elsewhere, DVC just helps tracking them.
Inference pipeline as an artifact
An inference artifact is not just saving the trained model, but also the preprocessing steps that were used to create the data that was input to that specific trained model. If you do not include the preprocessing step as an inference artifact, your results will not be fully reproducible.
This is also very important for achieving development/production symmetry. By also saving the preprocessing steps as an inference artifact, we ensure that the same preprocessing that was used during the training pipeline is used in the inference pipeline. So in short, we need an inference pipeline rather than just a model to achieve development/production symmetry by doing the same preprocessing in both.
But how to store preprocessing steps as a single artifact? Examples:
- Chain scikit-learn transformers together into a single pipeline object that can be exported
- Pytorch: include the needed transformations in the network using "scriptable transforms" and exporting with torch.script
- Use
Sequential()class containing transforms, such astransforms.Resize([256, 256]) - Export:
s = torch.jit.script(pipe)ands.save("inference_artifact.pt")
- Use
- scikit-learn's
ColumnTransformeris useful to create a single pipeline with sub-pipelines for different columns in a pandas dataframe- e.g. one for categorical variables, one for numerical variables, and then pipelines for columns that require exceptional handling
If the preprocessing stage includes feature engineering, consider using a feature store.
Exporting using MLflow
A very good option for exporting inference pipelines is using the MLflow convention, which has several benefits:
- Standard format accepted by many downstream tools
- Several subformats or "flavors" in the same model export to suit various downstream tools
- for
sklearnyou havemlflow.sklearn.save_model(); similar forpytorch,ONNX,keras, generic python function flavor for customization
- for
- Easily read in again when using mlflow for orchestration
- Environment isolation because the model export contains the used environment
- The right
conda.ymls are included by default, but you can customize using theconda_envflag.
- The right
- A data signature containing the data schema for input and output data can be provided so downstream tools can catch data schema problems
- You can use the
mlflow.models.infer_signature()function
- You can use the
- Input examples can be included in the export so you can easily test the inference pipeline
- Exports can be transformed into a Docker image with REST API to run the model/inference pipeline!
Example:
mlflow.sklearn.save_model(
pipe, # sklearn pipeline
export_path, # Path to a directory for the produced package
signature=signature, # input and output schema
input_example=X_test.iloc[:5] # the first few examples
)
Test what you deploy
After exporting the model, evaluate the inference artifact on the test data set!
Only if performance is sufficient, we should select the best model and mark it for production, e.g. with a particular unique tag such as prod.
(Unique meaning that only one model will have this tag.)
Experimentation
The right mindset
- Iteration: it's normal
- E.g. you might even find bugs in the preprocessing steps and have to go back a few steps
- Iteration requires time, so factor this in within the project deadlines
- Be systematic: change one thing at a time
- "Nothing is lost": every experiment matters, knowing what does not work is also valuable
- Have a criterium for when to stop because there are diminishing returns when experimenting
- Nothing is fixed: even when you have a trained model and are at the last step of the model development, it may be more beneficial to go all the way to the first step and collect more and/or higher quality data instead of developing a complex solution that may not even work.
Reproducible experiments
To reproduce experiments, three things are essential (one bonus):
- The data needs to be versioned so you can always retrieve the input data for any experiment
- The code needs to be versioned for the same reason
- Track all experiments from the beginning of the project
- If randomness is involved (almost always, e.g. train-validation split) then setting seeds appropriately is also needed for reproducibility
Configuration e.g. with Hydra
Releasing a pipeline
Performance Testing, incl. bias
Monitoring performance after deployment
- Record keeping of processes, such as data ingestion
- Source data files, ingestion date, size of the input datasets, details on the cleaning process
- Re-training and re-deploying
- Automatic scoring
- Monitoring model drift
- Hypothesis testing in model comparisons
- Automatic reporting using APIs
Operational:
- Timing operational processes to check for latency issues
- Checking and resolving module dependency issues
- Checking and resolving data stability issues
Sources
MLOps
- MLOps: Continuous delivery and automation pipelines in machine learning
- MLOps: What It Is, Why it Matters, and How To Implement It (from a Data Scientist Perspective)
- MLOps Core
- 10 Amazing MLOps Learning Resources
- MLOps-Reducing the technical debt of Machine Learning
- Create Reusable ML Modules with MLflow Projects & Docker