MLflow

A MLflow project has three main components:

  1. Project code
    • Reusable and reproducible by sticking to conventions
    • MLFlow is language-agnostic. Use the right language for the right task. You can even mix and match.
  2. Environment definition of runtime code dependencies
    • Uses Conda or Docker
    • Note that Conda is not limited to Python and is also language-agnostic
    • Note that Conda != Anaconda. Conda is the package manager, Anaconda offers an integrated set of packages.
  3. Project definition: what is contained in the project and how does the user interact with it?
    • API and CLI tools for running projects and chaining projects into workflows

Project dependencies: defining the environment

Basic example of conda.yml:

name: download_data
channels:
  - conda-forge
  - defaults
dependencies:
  - requests=2.24.0
  - pip=20.3.3
  - mlflow=1.14.1
  - hydra-core=1.0.6
  - pip:
        - wandb==0.10.21

Things to note:

  • name is the environment name
  • The order in the list of channels matters because it defines the lookup order of dependencies
  • Best practice is to list to full version of a working environment to ensure compatibility of dependencies
  • If some dependencies are not available in conda you can nest a definition e.g. with pip specific installs (see wandb above).
  • Versions are indicated with a single = for conda but == for pip

You can also use Docker.

See https://www.mlflow.org/docs/latest/projects.html#specifying-an-environment.

Conda vs. Docker

Considerations for Conda:

  • Easy to use, little DevOps skills needed
  • Self-contained, all info in conda.yml
  • Not completely isolated, still dependent on host
  • Cannot use Kubernetes (which requires Docker)

Considerations for Docker:

  • Complete isolation
    • No annoying issues when switching between Windows and Linux
  • Can run anywhere, also on Kubernetes
  • Requires more skills to setup
  • You need a separate Dockerfile in addition to the MLproject file, so your config is not self contained.
  • Running locally requires root privileges
    • Which means that adding a user to the docker group effectively gives that user root permissions

Project definition in MLproject

NOTE: the filename is MLproject without extension.

Example:

name: download_data
conda_env: conda.yml

entry_points:
  main:
    parameters:
      file_url:
        description: URL of the file to download
        type: uri
      artifact_name:
        description: Name for the W&B artifact that will be created
        type: str
        default: output.csv

    command: >-
      python download_data.py --file_url {file_url} \
                              --artifact_name {artifact_name}
  other_script:
    parameters:
        parameter_one:
          description: First parameter
          type: str
    command: julia other_script.jl {parameter_one}

Things to note:

  • You must always define a main entry point, which will serve as the default
  • Every entry point section must define the command to run
  • The parameters within brackets will be interpolated with the user input.

Running an MLflow project from CLI

From a local folder:

mlflow run /path/to/the/local/folder -P file_url=https://...

NOTE: for each parameter you provide a -P flag.

If you want to specify a different entry point, use -e:

mlflow run /path/to/the/local/folder -e other_script

From a remote Git repo:

mlflow run git@github.com/my_username/my_repo.git

Run a specific version tag or branch, e.g. to only run released code versions:

mlflow run git@github.com/my_username/my_repo.git -v 1.2.3

Running an MLflow project from a Python script

import mlflow

mlflow.run(
  # URI can be a local path or the URL to a git repository
  # here we are going to use a local path
  uri="my_project",
  # Entry point to call
  entry_point="main",
  # Parameters for that entry point
  parameters={
    "file_url": "https://...",
    "artifact_name": "my_data.csv"
  }
)

Running in the cloud

There are several options:

  • Databricks: native support by using the -b flag: mlflow run -b
  • Execute pipeline on a compute instance you spin up yourself
  • Hydra launchers
    • Supported backends: Joblib (parallel local jobs on one CPU), Ray, Redis Queue, SLURM
  • Kubernetes without Kubeflow
    • Quick and dirty but not optimal because entire pipeline is run within a single container
    • In other words, you lose the benefits of the composition of the pipeline into multiple steps
    • E.g. include miniconda in Docker build so that components can still use conda.
  • Kubernetes with Kubeflow
    • Install miniconda and mlflow in Docker image
    • You can relatively easily convert your mlflow pipeline with main.py to a Kubeflow pipeline

Setting the storage backend

By default MLflow will store results locally. One particular issue I ran into is that when running on an AzureML compute the default backend is the AzureML workspace. In this particular instance I wanted to store results locally. The mlflow Python package has the functions mlflow.get_tracking_uri() and mlflow.set_tracking_uri().

However, when running mlflow run . the backend stored in the environment variable MLFLOW_TRACKING_URI is used. So, in order to set a local file or sqlite backend, change this environment variable, e.g. with:

export MLFLOW_TRACKING_URI=file://$(pwd)/mlruns

Experiment tracking

Example:

experiment_id = mlflow.create_experiment("experiment 1")

with mlflow.run() as run:

  # Log an hyper parameter
  run.log_param("hyperparam1", 100)

  # Log a metric
  for i in range(10):
    run.log_metric("foo_metric", i)

  # Log an artifact
  path_to_my_artifact = "./my_artifact.csv"
  run.log_artifacts(path_to_my_artifact)