DP-100 Exam Preparation
From mid september this exam will be open book (access to MS Learn pages).
The exam is now open book. When taking the online exam it should now allow you to search Microsoft Learn. See this link.
For exam go to Microsoft learn and search for azure.ai.ml to ensure you will get SDKv2 code.
Review materials
TODO check the explainability dashboard (Models tab in AML studio).
TODO what about privacy settings?
Resources:
- Course manuals
- Interactive learning material
- Trainer community channel
- Relevant architecture/ guidance/best practices
- GitHub - MicrosoftLearning/mslearn-azure-ml
- Resource Center | Microsoft Azure <- FREE AZURE E-BOOKS (WHITEPAPER) + MORE
Practical exam tips:
Search for these pages, make sure to select the right category:
Azure Machine Learning documentationunder Documentation- If you can't find a page for v2 it may only exist under SDKv1 (but still valid for v2, the v2 link will fallback to v1)
azure-ai-ml Packageunder the category "Referentie" (Dutch) / "Reference" (English)- Searching for
MLClientalso works well. - Not to be confused with "Referenties" ("credentials"), very confusing...
- Searching for
-
For CLI questions, you need to search for
az mlunder "Reference".- E.g.
az ml endpoint
- E.g.
-
140 + 20 minutes
- Starts or ends with a case study. Spend max 8 minutes on case study
- Info on business and technical requirements
- Integrate info from multiple sources, determine what's important, make best decision
- answer several questions that refer to case study
- 40-60 questions
- Give yourself max 2 min per question
- Answer all questions:
- there is no penalty for guessing
- but note that some questions cannot be skipped!
- Mark items for review and take another look later using ms learn documentation
- Not just multiple-choice, also drag and drop etc.
TODO How to find this page quickly? Link: https://learn.microsoft.com/en-us/azure/machine-learning/?view=azureml-api-2
Abbreviations:
- AML or AzureML: Azure Machine Learning
Data ingestion strategy for ML projects
- Where is data stored? For AML it needs to be in Azure Cloud. Three common options are:
- Azure Blob Storage
- Azure Data Lake Storage (Gen 2)
- Azure SQL Database
- What is the data format?
- Tabular or structured?
- Semi-structured data
- Unstructured
A main feature of the cloud is that compute and storage are separated into separate services. You do not use compute to store your data. You can scale compute up and down based on usage demands, storage is persistent but accessible in different access tiers.
If the data is outside of the cloud, you need to think about how to get the data in the cloud first. AzureML assumes data is in blob storage in Azure cloud somewhere.
A coded approach would use ADF/Synapse/Fabric/Spark in Databricks pipelines for data ingestion, very dependent on where the data lives.
A "non-coded" approach could use Azure Data Factory, or Synapse Workspace GUI.
E.g. Azure Synapse Analytics allows data pipelines with transformation steps (in notebook) and writing output to other Azure services. You can build in checks and balances.
Exercise: Design a data ingestion strategy.
Example: In blob storage you could manually define containers on several levels to structure the process of incrementally "working up" data quality for use in machine learning:
- Bronze container for raw data
- Silver container: transformed data
- Gold container: can be shared with the team
In these you can just dump data. When hierarchical namespacing is enabled, you can instead "+ Add Directory" and arbitrarily nest folders to define your data namespace. With this enabled you are not using blob storage but a Data Lake. Data Lake Storage Gen2 allows hierarchical namespacing which access level control. This would also make reuse of datasets between projects easier.
The ML process: which tools to use?
Exercise: Design a model training strategy
Possible services:
- Azure Cognitive Services
- Can integrate functional APIs
- See AI-102
- Azure Synapse Analytics
- Integration with Spark and GUI data ingestion pipelining
- See DP-203
- Azure Databricks with Spark
- Azure Machine Learning
When to use AML over other tools? If your company only uses Spark, Databricks is the best choice because it's a performance wrapper around the Spark framework optimized for data usage. It's notebook-centered and facilitates working together in notebooks.
If your team already uses Synapse Analytics for data engineering, you can consider it for analysis too, but Synapse uses the open source Spark framework, but there's less freedom Spark-wise. Nasty detail: don't forget to publish your notebook.
Because Databricks and Azure Synapse Analytics notebooks are built on Spark they are useful for big data.
In AzureML, by default you run Python on a single machine, so unless you take care of parallelization yourself this won't be very suitable for big data (let's say >30 million rows). When to use: AzureML is more useful for MLOps pipelines with more control.
Algorithms
See How to select algorithms for Azure Machine Learning.
N.B. one test exam asked to "select an algorithm to use that will support the non-linearity of the model." I find this a very weird formulation, because typically I'd say you care most about the linearity of the problem, not the model. E.g. the "Two Class Support Vector Machine" is a linear model (and the wrong answer for this question) even though SVMs can solve non-linear problems since it's only linear in a hyperspace...
NOTE: this doc page is annoyingly only available in V1.
Dev environment
- Local
- Azure Data Science Virtual Machine
- Cf. Azure VM; this one is preconfigured with data science software. You can find it as a separate resource in the portal.
- Managed, but slow to start up.
- Compute instance
- Already has SDK installed
- Can also connect with visual Studio Code + AML extension
Q: You must reuse the same Python packages, environment variables, and software settings across all compute instances. You need to configure the project. What should you do?
A: Using the Environment object will guarantee you are reusing the same environment every time. Using the Workspace object will allow you to create a workspace but not to save the environment. Applying the script to set up your environment will not guarantee the environment is the same every time because it is not a systematic approach. Using the same compute instance will not allow you to reuse the environment across all compute instances.
Compute options
Resources:
In MS Learn, search for "Compute targets". There's an overview on the page "What are compute targets?" Looking at that page, I extract this rule of thumb:
- General purpose compute names start with a D
- Memory optimized compute targets start with a E
- Compute optimized starts with F
- High performance compute starts with an H
- Storage optimized starts with an L
- Memory optimized starts with an M
- GPU starts with N
Things to consider:
- CPU or GPU
- General purpose (testing and dev on smaller sets) or memory optimized (high memory-to-CPU ratio for in-memory analytics).
- Spark compute for big data
- or deploy into a container?
How do you chose the right option?
- Compute instances are ideal for experimentation
- Compute clusters are ideal for model training
- But this is not distributed computing like Spark! So not useful for distributed processing of a single data set.
- Useful for running pipeline components in parallel, e.g. via the designer
- Useful for batch inference
- Kubernetes cluster (previously "inference cluster") for real-time inference (scale up and down)
- For running pipelines in production
- Attached compute to connect to compute running in other services
Tip: When using notebooks, you can also run a notebook in a serverless Spark compute! Generally speaking you can connect to compute in other services.
Environments
Environment includes python packages, environment variables, runtime.
And which environment to use?
- Curated environment
- Custom environments specified in
- From a Docker file
- conda.yml
- Set up script
- Specify in GUI
Snippet for curated environments:
for env in ws.environments.list():
print(env.name)
env = ml_client.environments.get("AzureML-sklearn-0.24-ubuntu18.04-py37-cpu", version=44)
print(env.description, env.tags)
Note that when e.g. running a command job you can specify the curated environment by its name in the environment argument, instead of creating an Environment object like below.
E.g. environment="AzureML-pytorch-1.8-ubuntu18.04-py37-cuda11-gpu@latest".
Create an environment from a Docker file:
from azure.ai.ml.entities import Environment
env_docker_image = Environment(
image="mcr.microsoft.com/azureml/openmpi3.1.2-ubuntu18.04",
name="docker-image-example",
description="Environment created from a Docker image.",
)
ml_client.environments.create_or_update(env_docker_image)
If you have a base Docker image but want to add some extra dependencies, you can specify a conda.yml on top of the base image:
from azure.ai.ml.entities import Environment
env_docker_conda = Environment(
image="mcr.microsoft.com/azureml/openmpi3.1.2-ubuntu18.04",
conda_file="./src/conda-env.yml",
name="docker-image-plus-conda-example",
description="Environment created from a Docker image plus Conda environment.",
)
ml_client.environments.create_or_update(env_docker_conda)
Note that if you use Docker file, the environment will be rebuild for each experiment run. So if possible, register an environment so it can be reused.
Deployment
The deployment solution depends on whether the input data is real-time (score data as it comes in) or batch (score new data in batches and save results in a file or database where it can be queried later). When batching, you can schedule the model scoring or do it based on a trigger. Also consider the compute costs.
Flow:
- In both cases you need to have a model registered in the model registry. (use mlflow)
- You need a scoring script that takes the model and scores incoming data (not needed when using with mlflow)
- Required functions:
def init()to register the model e.g. as a global so it's available within the entire container. Called once.def run(json_data)to transform the incoming data to a format that works for your model. Called every time data is received.
- Create container image for the runtime environment
- If you use MLFlow in AML the previous two steps are abstracted away for you.
- Otherwise e.g. use Azure Container Instances
- Deployment from image registry & Model monitoring
- Cloud service, heavy edge, or light edge.
For the exam, be able to explain the pros and cons of using MLflow vs. custom models and deployment. For custom: manually register the model + write a scoring script with (init() and run()) + define your runtime environment.
How to deal with historical data needed by the model?
Just brainstorming here:
- Option 1: provide historical data in the request payload
- Option 2: collect the required data in the scoring script and then add the features dynamically
- Probably unfeasible!
- Option 3: pull it from another API or feature store
- Probably still too slow
- Option 4: turn into a batch problem
Blue/green deployment
This is an explanation from Red Hat source:
Blue green deployment is an application release model that gradually transfers user traffic from a previous version of an app or microservice to a nearly identical new release—both of which are running in production. The old version can be called the blue environment while the new version can be known as the green environment. Once production traffic is fully transferred from blue to green, blue can standby in case of rollback or pulled from production and updated to become the template upon which the next update is made. There are downsides to this continuous deployment model. Not all environments have the same uptime requirements or the resources to properly perform CI/CD processes like blue green. But many apps evolve to support such continuous delivery as the enterprises supporting them digitally transform.
Managed online endpoints
- For realtime; otherwise run a pipeline periodically
- In AML you have Managed Online endpoints and Kubernetes online endpoints.
- Blue/green deployment for evaluating models on unseen data
You could customize end points themselves.
- the init method which is called only once before model usage
- the execution function run() which is called every time data is received
- the class
ManagedOnlineEndpointis the output class when creating the endpoint, but cannot be changed itself
Note that a Kubernetes-endpoint can scale but requires you to manage the underlying infrastructure. A managed online endpoint both scales and abstracts away the infrastructure for you.
To deploy an online endpoint with the name of the endpoint, its instance type, its environment and its code configuration, among other things, the class ManagedOnlineDeployment needs to be used. The class OnlineEndpoint retrieves the online endpoint's property once deployed but does not allow it to deploy one.
Batch endpoints
- The main difference with a realtime managed online endpoint is that this one triggers a pipeline.
- Unlike just using a pipeline without an endpoint, you can now utilize these pipelines outside of AML because you now have an endpoint that communicates over HTTPS.
- Use the
BatchEndpointclass. - Deploy a
BatchDeploymentto deploy a model that takes the endpoint as an argument. - You should at least use a compute cluster, because the scoring can be done in parallel. The AML endpoints manage this for you.
- When you do this yourself, manage mini-batches yourself.
- You can deploy multiple models if you use the blue/green endpoint approach.
- You can convert pipelines to a scoring pipeline via the designer, I think?
You need to define several things like:
- mini_batch_size
- max concurrency per instance
- output action, like "append row".
- output file name
Logging deployment
See "Monitor web services" -> "Monitor with Application Insights". Link: https://learn.microsoft.com/en-us/azure/machine-learning/how-to-enable-data-collection?view=azureml-api-1&viewFallbackFrom=azureml-api-2
Query logs:
traces
|where message == "STDOUT" and customDimensions.["Service Name"] == "<service name>"
| project timestamp, customDimensions.Content
Endpoint testing is relatively easy to test with log analytics and blue/green development (live testing).
Combination with FastAPI?
Q: I asked whether it is common to use AzureML but build an API yourself e.g. using FastAPI?
A: It's not required, but optional. There's not much documentation that integrates the two topics but also, up to you, really.
Using FASTAPI: If you set up an endpoint and you can use the REST API to access the data in Azure but you can also just use the endpoint in the V2 SDK. See introduction of managed endpoints in AzureML
In a typical use case, you might train a machine learning model using Azure Machine Learning, then use FastAPI to create an API for your model, and finally deploy this API as an Azure Machine Learning endpoint Blog post on using FastAPI on Azure and example repo.
Azure Machine Learning workspace resource
Automatically makes:
- Storage account: blob storage
- Key vault for corporate password managing
- Application insights for real time monitoring
- Container registry
- Log analytics workspace if not existent yet
For the exam, understand bash commands to e.g. create an AML workspace automatically in a setup script.
Nowadays the Designer tool uses SDKv2 so it offers the standardized pipeline approach, graphical + code.
Example bash script to initialize an AML workspace and two compute resources with Azure CLI:
#! /usr/bin/sh
# Create random string
guid=$(cat /proc/sys/kernel/random/uuid)
suffix=${guid//[-]/}
suffix=${suffix:0:18}
# Set the necessary variables
RESOURCE_GROUP="rg-dp100-l${suffix}"
RESOURCE_PROVIDER="Microsoft.MachineLearning"
REGIONS=("eastus" "westus" "centralus" "northeurope" "westeurope")
RANDOM_REGION=${REGIONS[$RANDOM % ${#REGIONS[@]}]}
WORKSPACE_NAME="mlw-dp100-l${suffix}"
COMPUTE_INSTANCE="ci${suffix}"
COMPUTE_CLUSTER="aml-cluster"
# Register the Azure Machine Learning resource provider in the subscription
echo "Register the Machine Learning resource provider:"
az provider register --namespace $RESOURCE_PROVIDER
# Create the resource group and workspace and set to default
echo "Create a resource group and set as default:"
az group create --name $RESOURCE_GROUP --location $RANDOM_REGION
az configure --defaults group=$RESOURCE_GROUP
echo "Create an Azure Machine Learning workspace:"
az ml workspace create --name $WORKSPACE_NAME
az configure --defaults workspace=$WORKSPACE_NAME
# Create compute instance
echo "Creating a compute instance with name: " $COMPUTE_INSTANCE
az ml compute create --name ${COMPUTE_INSTANCE} --size STANDARD_DS11_V2 --type ComputeInstance
# Create compute cluster
echo "Creating a compute cluster with name: " $COMPUTE_CLUSTER
az ml compute create --name ${COMPUTE_CLUSTER} --size STANDARD_DS11_V2 --max-instances 2 --type AmlCompute
Hyperparameter tuning
AML has its own SDK and for defining a sweep job.
You can call a sweep in your training script.
Know the difference between sampling methods and termination strategies.
Define search space
Hyperparameters are either discrete or continuous and this defines how you specify the search space.
Discrete are specified using Choice or "quartile" versions of the continuous options Uniform, LogUniform, Normal, LogNormal.
The quantile versions have a Q prepended, like QUniform, QLogUniform, QNormal, QLogNormal.
Sampling methods
There are only three choices, so this is a limitation. Furthermore, not all sampling methods work with discrete and continuous parameters.
- Grid sampling: works with discrete only (Choice, Qnormal)
- Exhaustive
- Random sampling: works for both discrete and continuous
- Sobol: this is a subtype of random sampling that adds a random seed for reproduction
- Non-exhaustive
- Bayesian sampling: limited to choice, quniform, and uniform.
Sweep objective
Specify:
- primary metric
- should match the metric logged in the training script with mlflow.log_metric()
- goal: "Maximize" or "Minimize" (with a capital for some reason?!)
Termination strategies
NOTE: early termination is most useful when retraining models in production if you thoroughly understand how your parameters influence model performance, after experience and experimentation basically. This allows you to retrain models faster. Also be careful when you receive significantly new data.
When developing a model for the first time, you typically don't want to settle for a local optimimum but find the best model.
There are some limitations to strategies you can use.
- BanditPolicy
- cuts jobs under a threshold given a slack factor
- a slack factor of 0.35 (35%) means that runs that are more than 35% worse than the best performing run are terminated early
- punishing with a low
slack_factor, accepts more with a higher value
- Median stopping policy
- stop if lower than median
- TruncationSelectionPolicy
- cancels a percentage of the lowest performing jobs
- quite easily triggers stopping at local optimum
For each you need to specify:
- evaluation_interval: check policy at a particular interval
- delay_evaluation: when to start using the policy
Guides in picking a strategy:
- For a conservative policy that provides savings without terminating promising jobs, consider a Median Stopping Policy with evaluation_interval 1 and delay_evaluation 5. These are conservative settings that can provide approximately 25%-35% savings with no loss on primary metric (based on our evaluation data).
- For more aggressive savings, use Bandit Policy with a smaller allowable slack or Truncation Selection Policy with a larger truncation percentage.
See https://learn.microsoft.com/en-us/azure/machine-learning/how-to-tune-hyperparameters?view=azureml-api-2
Azure CLI: az ml
On Windows, install AML: az extension add -n ml -y
Example for creating a compute: az ml compute create --file compute.yml --resource-group rg101 --workspace my-ws-name
Jobs
You can use command from azure.ai.ml to run an arbitrary script as a standalone job.
And then use create_or_update(job).
Example:
from azure.ai.ml import Input, Output
from azure.ai.ml.constants import AssetTypes
from azure.ai.ml import command
# configure input and output
my_job_inputs = {
"local_data": Input(type=AssetTypes.URI_FILE, path="azureml:diabetes-local:1")
}
my_job_outputs = {
"datastore_data": Output(type=AssetTypes.URI_FOLDER, path="azureml://datastores/blob_training_data/paths/datastore-path")
}
# configure job
job = command(
code="./src",
command="python move-data.py --input_data ${{inputs.local_data}} --output_datastore ${{outputs.datastore_data}}",
inputs=my_job_inputs,
outputs=my_job_outputs,
environment="AzureML-sklearn-0.24-ubuntu18.04-py37-cpu@latest",
compute="aml-cluster",
display_name="move-diabetes-data",
experiment_name="move-diabetes-data"
)
# submit job
returned_job = ml_client.create_or_update(job)
aml_url = returned_job.studio_url
print("Monitor your job at", aml_url)
NOTE that when you define inputs and outputs as a dictionary as above, you can refer to it in a argparse.Namespace kind of way in the command (similar to when you define inputs and outputs of a pipeline component).
The move-data.py script should accept argument e.g. using argparse.
NOTE that the command above uses a curated environment.
Datastore
Abstraction layer that acts as a pointer to underlying storage and handles authentication.
You can change the default datastore!
Data assets
Advantages:
- Sharing data between team members
- Datastores allow you to not worry about connection strings or data paths
- Version the data asset meta data
Types (SDKv2):
uri_fileuri_foldermltable- .yml file including file paths to input data, and references to data transformations, support data schemas
- Answer from test exam: MLTable takes multiple txt files (?) with a schema definition? The MLTable backend is specified in YAML entirely, so perhaps that's what is meant with "txt files".
Initially, you may want to work with data assets in notebooks, to explore the data and experiment with machine learning models. Any URI_FILE or URI_FOLDER type data assets are read as you would normally read data. For example, to read a CSV file a data asset points to, you can use the pandas function read_csv().
A MLTable type data asset is already read by the MLTable file, which specifies the schema and how to interpret the data. Since the data is already read, you can easily convert a MLTable data asset to a pandas dataframe.
Example:
import mltable
registered_data_asset = ml_client.data.get(name='diabetes-table', version=1)
tbl = mltable.load(f"azureml:/{registered_data_asset.id}")
df = tbl.to_pandas_dataframe()
Q: Why use MLTable?
The preprocessing with mltable is very limited so we don't really use it, but the schema checking is useful. It's also quite a bit faster when you have a large amount of data and you need only a few columns / partitions / column_value==x, compared to loading everything and then filtering with pandas.
When using data assets as job inputs or outputs, make sure you specify the correct mode (e.g. download vs upload, read-only mount, write-only mount).
NOTE: the types FileDataset and TabularDataset belong to SDKv1.
Git
Very silly, but there's a whole section on "Git integration" in AML. You can get questions about the order of actions when using Git e.g. in a notebook.
- Open the terminal window in the AML Notebook tab
- Generate new SSH key
- Clone public key to Git account
- Clone git repo
Data tagging
TODO will this be covered?
The NER task in AutoML accepts tag data in txt format, not csv.
Privacy
I can't find AML docs on differential privacy.
The things I find use the package OpenDP but I can't find Microsoft docs on it.
The most important parameter to understand is epsilon, with a privacy / accuracy trade-off.
Epsilon is an inverse measure of the amount of noise added to the data, so lower epsilon -> more noise.
Or to read it otherwise, a higher epsilon stays closer to the original data (less noise).
A very low epsilon value will give high privacy, but at the cost of accuracy.
Pick a higher epsilon to keep higher accuracy while balancing privacy requirements.
Epsilon is related to a delta value that indicates the probability that a report on the data is not fully private.
V1 had "interpret" package. azureml-interpret. There seems to be a V2 version too: https://learn.microsoft.com/en-us/azure/machine-learning/how-to-machine-learning-interpretability?view=azureml-api-2
There are default explanation components offered by AzureML.
In the designer, look for microsoft_azureml_rai_.
See aka.ms/raicomponents
They only support models with a sklearn flavor.
You can explore:
- error analysis
- interpretability
- cohorts
- feature importance (aggregate vs local)
SMOTE: https://learn.microsoft.com/en-us/azure/machine-learning/component-reference/smote?view=azureml-api-2&source=recommendations
AutoML
The initial target audience were non-coders, but nowadays it has its place in data science professions too mainly as a tool for benchmarking.
AutoML hyperparameterises models and allows exploration of different algorithms and preprocessing.
Even though there are some very basic preprocessing options (e.g. scaling or imputation), generally speaking you need to preprocess your data set before putting it in AutoML! (This is an interesting point as regards to its usefulness for no-code solutions.)
Use a compute cluster so you can run jobs in parallel.
In the background MLFlow will be invoked with mlflow.autolog().
(Is this actually what limits the supported models?)
Available:
- Classification
- Regression
- Time series forecasting
- Natural Language Processing
- Computer Vision
There's a button "enable deep learning" for some categories.
Make sure to view the additional configuration.
- This allows you to specify per column name e.g. which type of imputation can be done.
- Limitation: AutoML will always normalize your data.
- There's a "Explain best model" for feature importance
- Indicate which models to try
- Specify exit criterion (e.g. time out) or metric threshold (when it's)
- Concurrency options
- Type of train-validation split; or use separate validation set
- Optionally provide a hold-out test data set
For the exam, know which default parameters we need to start a sweep (when using the coding approach).
- compute
- primary_metric
- target_column_name
- etc. see docs!
Problems with the data will be shown in the Data guardrails tab of a completed AutoML job.
For example, warnings will be raised for class imbalance, high cardinality or performed missing feature imputation.
N.B. you can turn featurization off, see azureml.train.automl.automlconfig .
MLFlow
Mostly repetition.
I had not used before:
query = "metrics.AUC > 0.8 and tags.model_type = 'LogisticRegression'"
mlflow.search_runs(exp.experiment_id, filter=query)
TODO mlflow stages?
MLOps
Maturity model: https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/mlops-maturity-model
Accelerator: https://microsoft.github.io/azureml-ops-accelerator/1-MLOpsFoundation/1-MLOpsOverview/2-MLOpsMaturityModel.html
MLOps Cloud Adoption Framework: https://learn.microsoft.com/en-us/azure/cloud-adoption-framework/ready/azure-best-practices/ai-machine-learning-mlops
Monitoring
There's a data drift monitor built in that sends you email if there's changes.
Monitoring jobs is not done in authoring pages, but you need to click the pipeline job link in the list of submitted jobs.
You can use Event Grid triggers to define actions on events in AML, like:
- Send emails on run failure and run completion
- Use an Azure function after a model is registered
- Streaming events from Azure Machine Learning to various of endpoints
- Trigger an ML pipeline when drift is detected
The events are Microsoft.MachineLearningServices.{EVENT} with values RunCompleted, ModelRegistered, ModelDeployed, DatasetDriftDetected, RunStatusChanged for EVENT.
Also check out Evidently. Idea: create a component that uploads to your Evidently dashboard.
See screenshot.
riverml concept drift: https://riverml.xyz/0.11.1/examples/concept-drift-detection/
Online-learning? https://github.com/online-ml/river
Tip: calmcode.io
Pipelines
For each component the inputs and outputs are persisted, so by set up there is a lot of redundancy. You need to think about whether you want to clean up periodically, or overwrite data.
Overwriting or deleting data however reduces auditability, so there's a trade-off!
Possible solution: mlflow can set stages to keep track of the model life cycle.
You can schedule a pipeline with the JobSchedule class.
You can give it a RecurrenceTrigger or RecurrencePattern to trigger in a timed basis.
CICD
- Setup
- Model development
- Optimisation
- Continuous integration
- Dependency management
- Continuous deployment
- Monitoring
Model monitoring:
- Performance / RAI
- Data drift
- Prediction drift
- Data qualtiy
- Compute utilisation and environment
- Unit testing
Environment
- Production team
- Testing pre-production team
- Development environment
Feature stores
(Not exam material)
See azureml examples on the feature store: https://github.com/Azure/azureml-examples/tree/main/sdk/python/featurestore_sample
Tutorial: https://learn.microsoft.com/en-us/azure/machine-learning/tutorial-get-started-with-feature-store?view=azureml-api-2&tabs=SDK-track
- Offline materialization vs online materialization
- offline data snapshots can be quickly accessed from ADLS (default blob storage is not supported?).
- The FS will do the transformations for you, but transformations have to be done in Spark, not Python!
- needs to inherit the Transformer class
- but you can use pyspark
- Currently a managed Spark compute is being released for this purpose
- You can also put precomputed features in the store
- You create "entities"