Citadel platform observability
The following observability capabilities built into the Citadel accelerator
| # | Capability | Primary question it answers | Data store |
|---|---|---|---|
| 1 | APIM Analytics Dashboard | How much are my LLMs / AI services being used? | Azure Monitor metrics |
| 2 | APIM → Application Insights | Is the gateway healthy and fast? | Application Insights |
| 3 | APIM → Log Analytics (ApiManagementGatewayLlmLogs) |
What exactly was sent and returned? | Log Analytics workspace |
| 4 | Cosmos DB usage data + Power BI | Who is spending what, and how do I charge it back? | Cosmos DB → Power BI |
1. Built-in APIM analytics
There is an APIM Azure Monitor diagnostic setting that enables built-in analytics directly into APIM. This gives insight into aggregated metrics (Monitoring -> Analytics -> Language Models):
- Requests per API, product (access contract), and operation
- Token consumption (prompt / completion / total) per model and per product
- Latency and response-code distribution
2. APIM application insights
- End-to-end request tracing with W3C correlation across the gateway → backend hop
- Failure and dependency analytics, live metrics, and the Application Map
- Response times, failure rates, and configurable request/response header capture
- LLM usage & throttling custom metrics emitted by the gateway policies — the recommended target for alerting
- Optional pre-built Azure portal dashboards
Custom metrics also land in the connected Application Insights component. This requires dimension support on app insights. TODO What is this?
The basic pattern: design policies to emit metric, they land in app insights, then you can select .
<fragment>
<!-- NOTE: emit-tokens has limited support for up to 6 custom dimensions -->
<!-- Using of optional dimensions require setting up variables before executing the policy fragment -->
<!-- Skip token emission for non-LLM requests (e.g., GET operations like listing models) -->
<choose>
<when condition="@((string)context.Variables.GetValueOrDefault<string>("requestedModel", "") != "non-llm-request")">
<llm-emit-token-metric namespace="llm-usage">
<!-- Use case or agentic platform name-->
<dimension name="productName" value="@(context.Product?.Name?.ToString() ?? "Portal-Admin")" />
<!-- Target model or deployment name -->
<dimension name="deploymentName" value="@((string)context.Variables.GetValueOrDefault<string>("requestedModel", "DefaultModel"))" />
<!-- This is the backend processed the request-->
<dimension name="Backend ID" />
<!-- This can be your agent Id or APIM subscription id-->
<dimension name="appId" value="@((string)context.Variables.GetValueOrDefault<string>("appId", context.Subscription?.Id ?? "Portal-Admin-Sub"))" />
<!-- CUSTOM DIMENSIONS: Allows defining additional context for LLM usage -->
<dimension name="customDimension1" value="@((string)context.Variables.GetValueOrDefault<string>("customDimension1", "NA"))" />
<dimension name="customDimension2" value="@((string)context.Variables.GetValueOrDefault<string>("customDimension2", "NA"))" />
</llm-emit-token-metric>
</when>
</choose>
</fragment>
<fragment>
<!--
Fragment: Raise Alert Events (comprehensive gateway alerting)
Purpose: Emits a single Application Insights custom metric ("AI Gateway Alert" in the
"ai-gateway-alerts" namespace) for a curated set of service-impacting events so
that teams can build targeted Azure Monitor alert rules. Supersedes the
throttling-only "raise-throttling-events" fragment.
Design goals:
- LOW NOISE / OPT-IN: alerting is per-category and per-scope. Platform-health categories
(throttling, backend failures) default ON; client-driven categories (auth, content-safety,
PII) default OFF and are opted-in per access contract (product) for mission-critical use cases.
- SINGLE METRIC, MANY ALERTS: one metric with an "alertType" dimension lets you slice/alert
per category, product, model, backend, and app without emitting many metric names.
Two modes (auto-detect):
1. AUTO mode (default): include in the <outbound> and <on-error> sections. The alert type is
derived from context.Response.StatusCode and context.LastError. Captures:
- throttling : 429 (llm-token-limit tokens-per-minute / backend rate limit)
- quota-exceeded : 403 token-quota exhaustion (llm-token-limit, "AITokenQuotaExceeded")
- backend-failure : 5xx from the backend / backend connectivity errors
- content-safety : llm-content-safety block (context.LastError.Source ~ content-safety)
- pii-failure : PII anonymization failure (fail-closed 502)
- auth-failure : validate-jwt failures that reach on-error
2. EXPLICIT mode: set "alertEventType" (and optionally "alertEventStatus") BEFORE including the
fragment (from a NON-fragment policy scope — a fragment cannot include another fragment).
NOTE: authorization rejections that short-circuit via <return-response> from WITHIN a
fragment (security-handler, validate-model-access) cannot include this fragment; they emit
an equivalent inline <emit-metric> at the point of rejection instead.
Configuration variables (set at API level or in the access contract / product policy):
- alertsEnabled (default "true") : master kill-switch. Set "false" to suppress ALL alerts.
- alertOnThrottling (default "true") : emit for 429 throttling events (TPM rate limit).
- alertOnQuotaExceeded (default "true") : emit for 403 token-quota exhaustion.
- alertOnBackendFailure(default "true") : emit for 5xx backend failures.
- alertOnAuthFailure (default "false") : emit for 401/403 authorization failures.
- alertOnContentSafety (default "false") : emit for content-safety blocks.
- alertOnPiiFailure (default "false") : emit for PII anonymization failures.
Emitted dimensions (emit-metric supports at most 5 custom dimensions): alertType, statusCode,
productName, deploymentName, backendId.
NOTE: backendId uses the value-based `selected-backend` variable rather than the predefined
"Backend ID" dimension, because that predefined dimension is only valid in the OUTBOUND
section and this fragment also runs in on-error. It is "NA" when routing hasn't run yet
(e.g. an auth failure before backend selection).
NOTE: deploymentName can be high-cardinality; excessive cardinality increases custom
metric cost. Keep opt-in categories scoped to the use cases that truly need them.
-->
<choose>
<when condition="@(context.Variables.GetValueOrDefault<string>("alert-event-emitted", "false") != "true")">
<!-- Resolve the alert type: explicit override (set by caller) wins, otherwise derive it. -->
<set-variable name="_alertType" value="@{
var explicitType = context.Variables.GetValueOrDefault<string>("alertEventType", "");
if (!string.IsNullOrEmpty(explicitType)) { return explicitType; }
int sc = context.Response?.StatusCode ?? 0;
string src = context.LastError?.Source ?? "";
string reason = context.LastError?.Reason ?? "";
string msg = context.LastError?.Message ?? "";
// Source-based classification takes precedence over raw status code
if (src.IndexOf("content-safety", StringComparison.OrdinalIgnoreCase) >= 0) { return "content-safety"; }
if (src.IndexOf("pii", StringComparison.OrdinalIgnoreCase) >= 0
|| reason.IndexOf("PIIAnonymization", StringComparison.OrdinalIgnoreCase) >= 0
|| msg.IndexOf("PIIAnonymization", StringComparison.OrdinalIgnoreCase) >= 0) { return "pii-failure"; }
// Token QUOTA exhaustion returns 403 (llm-token-limit token-quota), classified by APIM
// as "AITokenQuotaExceeded". Detect it BEFORE the generic 403 -> auth-failure mapping.
// On-error path: context.LastError.Reason carries the classification. Outbound path:
// fall back to the presence of a Retry-After header (set by llm-token-limit on quota/TPM
// rejections but NOT on an auth 403).
if (sc == 403 && (
reason.IndexOf("TokenQuota", StringComparison.OrdinalIgnoreCase) >= 0
|| reason.IndexOf("QuotaExceeded", StringComparison.OrdinalIgnoreCase) >= 0
|| msg.IndexOf("TokenQuota", StringComparison.OrdinalIgnoreCase) >= 0
|| !string.IsNullOrEmpty(context.Response?.Headers.GetValueOrDefault("Retry-After", ""))
|| !string.IsNullOrEmpty(context.Response?.Headers.GetValueOrDefault("retry-after", ""))
)) { return "quota-exceeded"; }
if (sc == 429) { return "throttling"; }
if (sc == 401 || sc == 403) { return "auth-failure"; }
if (sc >= 500 && sc < 600) { return "backend-failure"; }
return "none";
}" />
<!-- Resolve the status code to report (explicit override wins). -->
<set-variable name="_alertStatusCode" value="@{
var explicitStatus = context.Variables.GetValueOrDefault<string>("alertEventStatus", "");
if (!string.IsNullOrEmpty(explicitStatus)) { return explicitStatus; }
int sc = context.Response?.StatusCode ?? 0;
return sc.ToString();
}" />
<!-- Resolve whether this alert type is enabled, honoring the master switch and per-type toggle. -->
<set-variable name="_alertEmit" value="@{
string t = context.Variables.GetValueOrDefault<string>("_alertType", "none");
if (t == "none") { return "false"; }
var master = context.Variables.GetValueOrDefault<string>("alertsEnabled", "true");
if (!master.Equals("true", StringComparison.OrdinalIgnoreCase)) { return "false"; }
string toggleName;
bool dflt;
if (t == "throttling") { toggleName = "alertOnThrottling"; dflt = true; }
else if (t == "quota-exceeded") { toggleName = "alertOnQuotaExceeded"; dflt = true; }
else if (t == "backend-failure") { toggleName = "alertOnBackendFailure"; dflt = true; }
else if (t == "auth-failure") { toggleName = "alertOnAuthFailure"; dflt = false; }
else if (t == "content-safety") { toggleName = "alertOnContentSafety"; dflt = false; }
else if (t == "pii-failure") { toggleName = "alertOnPiiFailure"; dflt = false; }
else { toggleName = ""; dflt = false; }
if (string.IsNullOrEmpty(toggleName)) { return "false"; }
var v = context.Variables.GetValueOrDefault<string>(toggleName, dflt ? "true" : "false");
return v.Equals("true", StringComparison.OrdinalIgnoreCase) ? "true" : "false";
}" />
<choose>
<when condition="@(context.Variables.GetValueOrDefault<string>("_alertEmit", "false") == "true")">
<emit-metric name="AI Gateway Alert" value="1" namespace="ai-gateway-alerts">
<!-- Event category: throttling | quota-exceeded | backend-failure | auth-failure | content-safety | pii-failure -->
<dimension name="alertType" value="@(context.Variables.GetValueOrDefault<string>("_alertType", "unknown"))" />
<!-- HTTP status code associated with the event -->
<dimension name="statusCode" value="@(context.Variables.GetValueOrDefault<string>("_alertStatusCode", "0"))" />
<!-- Use case or agentic platform name (access contract / product) -->
<dimension name="productName" value="@(context.Product?.Name?.ToString() ?? "Portal-Admin")" />
<!-- Target model or deployment name (when known) -->
<dimension name="deploymentName" value="@((string)context.Variables.GetValueOrDefault<string>("requestedModel", "NA"))" />
<!-- Backend / pool selected for the request. Uses the `selected-backend`
variable (value-based) rather than the predefined "Backend ID" dimension,
because that predefined dimension is only valid in the OUTBOUND section and
this fragment also runs in on-error. Empty ("NA") when routing hasn't run
(e.g. an auth failure before backend selection). -->
<dimension name="backendId" value="@((string)context.Variables.GetValueOrDefault<string>("selected-backend", "NA"))" />
</emit-metric>
<!-- Guard against a double emission if the fragment is included more than once. -->
<set-variable name="alert-event-emitted" value="true" />
</when>
</choose>
</when>
</choose>
</fragment>
Also see e2e example on alerting
3. APIM -> Log analytics
- Per-request model, deployment, token usage, and gateway/region metadata
- Full KQL querying, joins, workbooks, and alerting over LLM traffic
- Optional prompt / completion auditing — capturing the actual request and response messages when compliance or debugging requires it
4. Cosmos DB usage data + Power BI (FinOps & chargeback)
- Cost per product (access contract), model, backend, and application
- Detailed token breakdown (cached, audio, reasoning, prediction) priced independently
- Chargeback/showback of fixed-cost capacity (e.g., PTU) via proportional distribution
- Two free-form custom dimensions per record (e.g., end-user, session, cost center)
So the flow is: APIM policy emits metric -> app insights -> logic app aggregates data -> CosmosDB -> PowerBI on top of Cosmos.
Why not app insights -> PowerBI? Probably because the retention period of app insights is low and you don't need direct data.