202211041154-Azure_AI_fundamentals

https://learn.microsoft.com/en-us/certifications/azure-ai-fundamentals/

Possible route:

  1. Azure Data Fundamentals certification,
  2. level up with the Azure AI Engineer Associate,
  3. Azure Data Scientist Associate,
  4. Azure Developer Associate certifications

Azure ML

  • Automated Machine Learning
  • Azure Machine Learning Designer
  • Data and compute management
  • Pipelines

Includes services:

  • Anomaly Detector
  • Computer vision
    • Pretrained: Image classification, Object detection, Semantic Segmentation, Image Analysis (e.g. extracting "tags" that describe images); Face detection, analysis, recognition; OCR
    • Custom Vision: train custom image classification and object detection models
    • Face: for face detection & recognition solutions
    • Form Recognizer: extract information from scanned forms and invoices
  • NLP
    • Language
    • Translator (text only)
    • Speech (incl. translation for spoken languages)
    • Azure Bot: platform for conversational AI, using the Bot Framework
  • Knowledge Mining
    • Azure Cognitive Search: private, enterprise search solution that builds indexes using integrated image processing and NLP for knowledge mining

Responsible AI

Challenges & Risks:

  • Bias
  • Errors may cause harm
  • Exposing sensitive data
  • Solutions may not work for everyone
  • User trust in complex systems
  • Liability for AI-driven decisions

Six principles

  • Fairness
  • Reliability and safety
  • Privacy and security
  • Inclusiveness
  • Transparency
  • Accountability

Environment

  • Azure Machine Learning Workspace is a resource for managing data, compute, code, models, artifacts.
  • Azure Machine Learning Studio manages several things, incl. compute targets
  • Compute targets are:
    • Compute Instances
    • Compute Clusters: scalable clusters of VMs for on-demand processing of experiment code
    • Inference Clusters: Deployment for predictive services using training models
    • Attached Compute: Links to existing Azure compute resources

AutoML

  • Prepare data
  • Train model
  • Evaluate performance
  • Deploy a predictive service

AutoML is available for:

  • Classification
  • Regression
  • Time series forecasting
  • In the future: NLP + Computer Vision

Azure Machine Learning Designer

  • GUI for drag-and-drop creation of ML pipelines
  • Each designer project is called a pipeline
  • A component encapsulates a step in a pipeline
  • A ML job executes a task against a compute target
    • Records of runs can be viewed in Azure ML Studio
  • If you have defined a training pipeline, you can easily define the corresponding inference pipeline, that uses the same transformations etc on new data, using a menu entry.
  • Inference pipelines can be deployed at an endpoint and connected to client applications.
    • Endpoint deployment takes a while

Not many notes, nothing new.

Classification

  • Precision
  • Recall: true positive rate
  • ROC curve (receiver operating characteristic) for calibrating probabilistic (binary) classifier
    • AUC area under the curve
    • Interpretation: The AUC is the probability that the model ranks a random positive example higher than a random negative example.
    • E.g. AUC < 0.3 is worse than random guessing

Use Score Model

Clustering

Use Assign Data to Clusters.

Cognitive Services

Cognitive Services provide access to a large set of AI resources for various tasks. Typically you can also make a separate resource without Cognitive Services. One advantage of Cognitive Services is having a single endpoint for all these resources. This simplifies administration and development if you plan on using multiple cognitive services. Pros and cons are discussed below.

Computer Vision

Pre-built models for various use cases. E.g.:

  • Image tagging
  • Brand detection
  • Face detection
  • Categorizing images, in a general hierarchy and with two domain-specific models
    • Celebrities
    • Landmarks
  • OCR

Custom Vision

Q: What's the difference between the Computer Vision and Custom Vision service?

  • Part of Azure Cognitive Services
  • So Azure Custom Vision Service requires no in depth knowledge of deep learning
  • Custom Vision resources
  • Use Custom Vision Portal or Custom Vision service programming language-specific softwae development kits (SDKs)
  • When publishing, provide the following information to developers: 1) project ID, 2) model name, 3) key (for client authentication), 4) endpoint for prediction resource.
  • Image classification, object detection

When and how to use Custom Vision or Cognitive Services?

Use two resources for the following two tasks. This is handy because this way you can track resource utilization separately for training and client applications doing prediction. For each, you can either use Custom Vision or Cognitive Services:

  1. Train on custom images
  2. Publish the model so clients can use it to generate predictions

Three approaches:

  1. Use Cognitive Services for both training and prediction
    • Easiest, because you will have one endpoint (HTTP where service is hosted) and one key (used by client applications for authentication)
  2. Custom Vision
    • Prompt: use for trainig, prediction, or both?
    • When "both" is selected, then two resources are created, one for training and one for prediction.
  3. Mix: cognitive services for one task and custom vision for the other; and vice versa.
    • The resources must be in the same region

Metrics:

  • Precision & Recall
  • Average Precision (AP)
    • This is not the average of the precision metrics for various classes
    • Instead, it's the area under the PR-curve (precision (y) x recall (x) curve)

Object detection

  • Custom Vision portal contains a GUI for tagging images
    • Also provides suggested
  • www.customvision.ai

Facial recognition

Multiple services:

  • Computer Vision: face detection + returning bounding box for faces
  • Video Indexer: detect and identify faces in video
  • Face: pre-built algorithms for detecting, recognizing and analyzing faces
    • So this is the most capable service
    • Includes things like recognition and analysis of facial landmarks

Things the Face service returns:

  • Blur: how blurred is a face (e.g. to assess whether it's in camera focus)
  • Exposure: of the specific face, not the overall image
  • Glasses: yes/no
  • Head pose: 3D head orientation
  • Noise: noise in the overall image
  • Occlusion: are objects blocking the face?

Because this service handles faces with glasses, so this is not a big impediment for face detection. Something that is a problem are extreme angles.

Because this is a sensitive technology, a limited access policy is implemented for the sake of responsible AI use. You need additional access (via an intake form; how strict is this really?) for 1) comparing faces for similarity and 2) identifying individuals in an image

Processing written or printed text and OCR

Intersection of computer vision and NLP.

The Computer Vision service provides the Read API for reading text in images, which is optimized for images with significant amounts of text or that have considerable visual noise.

The Read API works asynchronously so that chunks of large documents can be processed in parallel. The process is as follows:

  1. Submit an image to the API and retrieve an operation ID
  2. Check the status of the analysis using that operation ID
  3. When completed, retrieve the results, which are ordered hierarchically:
    • Pages; includes info on page size and orientation
    • Lines; including bounding box coordinates indicating position on the page
    • Words; including bounding box coordinates and the text itself

The Form Recognizer service reads forms (e.g. receipts or invoices) and returns structured information such as key value pairs. So this service also contains some analysis, e.g. distinguishing merchant contact information from other types of info such as the date and time of the purchase. It does:

  • Matching field names to values
  • Processing tables of data
  • Identifying specific field types: dates, telephone numbers, addresses, totals of money, etc.

Remarks:

  • You can either use a pre-built receipt model or a custom model trained on your own forms (already start with N=5 samples of training data).
  • As with all these services, you can make a separate Form Recognizer resource, or use Cognitive Services.
  • The maximum file size for forms is 50MB; pdf 17x17 inches

Language cognitive service

https://language.cognitive.azure.com/?azure-portal=true

Examples:

  • Determine text language
  • Sentiment analysis
  • Extracting key phrases
  • Entity recognition
  • Entity linking
  • Question answering from a knowledge base

Conversational Language Understanding

In Azure, this is actually part of the Language service.

Three core concepts:

  • Utterances
  • Entities: entity to which an utterance refers. Four types:
    1. Machine-learned: trained from labelled sample utterances
    2. List: hierarchy of lists and sublists, e.g. (device) -> ((synonym list for 'light'), (synonym list of 'lamp'))
    3. RegEx: e.g. handy for telephone numbers
    4. Pattern.any: entities that are used with patterns that are too complex to extract from sample utterances.
  • Intents: represents the purpose, or goal, expressed in a user's utterance

    • should concisely group utterance tasks
    • Have a None intent as a fallback
  • Authoring is the step of defining entities, intents, utterances.

    • Many prebuilt domains with predefined intents and entities are provided
    • Minimum of five different utterance examples for each intent for training, e.g. "Switch on the light"
    • You need to label entities in the defined utterances. For example an entity could be "Device". Then you would label "fan" and "light" as device.
  • Publish the model so that client applications can predict intents and entities from input user utterances

Speech cognitive service

Recognition (Text-to-speech API):

  • Acoustic model
  • Language model

Synthesis (Speech-to-text API):

  • Tokenizing text
  • Assign phonetic transcription to each word
  • Break phonetic transcription into prosodic units (phrases, clauses, sentences, etc.) in order to create phonemes
  • Phonemes are converted to audio format
  • Based on Microsoft's Universal Language Model
    • Optimized for conversation and dictation
    • Transcription is either in real-time or in batch
    • You can also build your own model

There's also a speech translation API for translating speech in one language to text or speech in another.

Translator

Note that the Translator service supports text-to-text translation, but that speech-to-text and speech-to-speech translation are supported by the Speech service. For example, when you have an input speech stream this allows you realtime translation into text (for example for closed captioning movies) or speech (realtime two-way translation of spoken conversation)

  • 60 languages supported

  • provide ISO 639-1 language codes
  • specify one from language, but you can select multiple to languages

Question Answering + Azure Bot Service

If you want to build a customer support bot you'll need both the Language and Bot Service.

  • Knowledge base using Language Service -> specifically the Question Answering service
    • Use Language Studio's custom question answering feature to create knowledge bases with question-and-answer pairs
    • That is easier than using the Language service REST API or SDK (but these are also options)
  • Bot Service

On the knowledge base:

  • Q&A-pairs can be generated from an existing FAQ or entered/extended manually.
  • An in-built NLP model matches appropriate answers to questions, even if these answers were not recorded as an exact answer to the question.
  • The knowledge based is used over a REST API. Clients require:
    • knowledge base ID
    • knowledge base endpoint
    • knowledge base authorization key

On the bot:

  • Create a bot for a specific knowledge base with the Microsoft Bot Framework SDK or using the automatic bot creation offered by the Azure Bot Service.
  • Bot can be managed in Azure Portal
    • add custom code to extend functionality
    • test the bot in an interactive test interface
    • logging, analytics, integration with other services
  • Connect the bot to multiple channels, e.g. both email and web chat.

Decision Services -> Anomaly Detector

The Anomaly Detector service is part of Decision Services. Weirdly enough, this is the only module available in the "Decision Support" part of the AI fundamentals collection of learning paths.

Anomaly Detector:

  • Requires no machine learning knowledge
  • uses the concept of a "one parameter" strategy
    • Q What is that?
  • For: historical time series data, real-time data such as streaming inputs from IOT devices and sensors.

Q: What is the purpose of specifying granularity in your JSON data object? A: it indicates the recording pattern of the data.

Details:

  • Define an upper and lower boundary using a sensitivity value
  • Data in JSON includes: granularity of the data (e.g. "hourly"), timestamp, value.
  • Maximum of 8640 data points in the same JSON object -> break up data in chunks/windows and process them in sequence

Data consistency:

  • <= 10% missing, negligible
  • 10% missing, "fill" data e.g. With linear interpolation

  • anomaly detection more reliable with even distribution; you can adjust the aggregation scale to play with the distribution

Batch vs. real-time detection:

  • Batch: apply anomaly detection to whole data series at a time
    • Creates a single statistical model based on the whole data series
    • For flat trend time series data with occasional anomalies
    • For seasonal time series data with occasional anomalies
  • Real-time:
    • Model only for the data points you send
    • Monitor data as it's created
    • Useful for anomalies that must be acted on immediately, e.g. a critical setting in a manufacturing process.