AI Observability: Monitoring LLMs and ML Models in Production

You built the model, deployed it, and watched it work in staging. Then it hit production and quietly started behaving differently. Outputs drifted. Latency spiked on Tuesday afternoons. A prompt that worked perfectly six weeks ago now returns borderline responses. Nobody noticed for three weeks because there were no alerts, no dashboards, no way to see what was happening inside the model at runtime.

This is the observability gap in production AI. Traditional application monitoring—uptime checks, error rates, response times—tells you when a service is down. It says nothing about whether the model is giving good answers. AI observability closes that gap by instrumenting the full lifecycle: inputs, outputs, model behaviour, user reactions, and downstream effects.

• Why AI observability differs from classic APM
• What to monitor for LLMs vs traditional ML models
• Core signals: latency, cost, quality, safety, drift
• Tooling landscape: Langfuse, Arize AI, Evidently AI, and OpenTelemetry
• Implementation patterns for production systems
• Building alerting that surfaces real problems
• Common mistakes and how to avoid them
• Frequently asked questions

Why AI Observability Is Different

Classic application monitoring answers binary questions: did the request succeed? How long did it take? AI observability asks probabilistic questions: was the response correct? Was it safe? Did it reflect the same behaviour as last week? These are not questions a health-check endpoint can answer.

Three properties make AI systems uniquely hard to monitor:

• Outputs are soft. A REST API either returns a valid response or it errors. An LLM always returns a response—but that response might be hallucinatory, off-topic, or subtly harmful. There is no HTTP 4xx for "confident but wrong."
• Inputs are unbounded. Traditional systems receive structured data. LLMs receive natural language, which means the input distribution shifts constantly as users discover new ways to phrase requests.
• Behaviour changes without a deployment. An LLM can begin refusing requests, changing tone, or degrading in accuracy because a third-party API updated the underlying model—no code change, no deploy, no alert.

What to Monitor: LLMs vs Traditional ML Models

The signals differ by model type, though the underlying philosophy is the same: instrument everything between the request entering your system and the user receiving a response.

Large Language Models

Signal	Why It Matters	How to Capture
Token count (input/output)	Direct cost driver; unexpected spikes indicate prompt injection or context bloat	API response headers / SDK callbacks
Latency by percentile	P99 latency often 5–10× median; user experience degrades before mean moves	Timing wrappers around API calls
Response quality score	Tracks whether outputs meet quality standards over time	LLM-as-judge eval or human feedback labels
Safety / guardrail hits	Spikes indicate adversarial prompts or model policy changes	Classifier output on every response
User feedback signals	Thumbs up/down, edits, regenerations are ground-truth quality signals	UI instrumentation → event stream
Retrieval quality (RAG)	Chunk relevance scores; if retrieval degrades, generation degrades	Reranker scores, context utilisation rate

Traditional ML Models

Signal	Why It Matters	Tool
Feature drift	Input distribution shift predicts future performance drop before labels arrive	Evidently AI, Arize AI
Prediction drift	Output distribution change may indicate upstream data pipeline issue	Evidently AI, WhyLabs
Model accuracy (where labels exist)	Ground truth comparison when delayed labels become available	MLflow, Arize AI
Serving latency and throughput	Infrastructure issues separate from model quality	Prometheus, Datadog
Business KPI correlation	Connect model output to revenue/conversion to catch silent regressions	Custom dashboards

Core Signals: The AI Observability Stack

Whether you are running an LLM chain or a gradient-boosted classifier, five signal categories give you meaningful coverage.

1. Latency

Track time-to-first-token (TTFT) separately from total response time for streaming LLMs. TTFT directly determines perceived responsiveness. A p95 TTFT above 2 seconds will suppress feature adoption even if total generation time is acceptable. Alert on rolling p95 rather than mean—averages hide tail behaviour.

2. Cost

Token spend per session, per user, and per feature. One misbehaving prompt template can consume more tokens in a week than your entire planned monthly budget. Set spend alerts at 60% and 90% of budget; investigate at 60%.

3. Quality

The hardest signal to capture at scale. Three practical approaches:

• LLM-as-judge: Route a sample (2–5%) of production traffic through a separate evaluation model with a rubric. Cheap to implement; biased toward the judge model's priors.
• User feedback: Explicit (thumbs) and implicit (regenerate, copy, ignore). Low coverage but high signal quality.
• Reference-based eval: Compare against a fixed golden dataset on a schedule. Catches regressions when the underlying model updates.

4. Safety

Run every output through a lightweight classifier that detects policy violations, harmful content, and PII leakage. This is a blocking check in the request path for high-stakes applications and an async check logged to your observability platform for everything else.

5. Drift

For traditional ML: monitor feature distributions using statistical tests (KS test, Population Stability Index) on a daily schedule. For LLMs: embed a sample of inputs and outputs; when the centroid of the embedding cloud moves significantly, investigate why the input or output distribution has changed.

Tooling Landscape

Langfuse

Open-source LLM engineering platform. SDK integrations for Python, JavaScript, and LangChain. Traces every step in an LLM chain with parent-child hierarchy—ideal when debugging why a multi-step agent returned a bad answer. Supports custom evaluation scores, session tracking, and a self-hosted option for teams with data residency requirements. Best for teams building LLM applications from scratch who want full observability without vendor lock-in.

Arize AI

Production ML and LLM monitoring. Strong on drift detection and embedding visualisation. The embedding projector makes it visually obvious when your input distribution drifts—you can see clusters form and separate in 3D. Supports both LLM traces and traditional ML model monitoring in the same platform. Best for organisations running a mix of LLM and traditional ML in production who want a single pane of glass.

Evidently AI

Open-source data and model monitoring. Purpose-built for traditional ML drift detection with a rich library of statistical tests and visual reports. Has added LLM evaluation capabilities. Best for data science teams that care deeply about statistical rigour in drift detection and want full control over the evaluation methodology.

OpenTelemetry + Prometheus + Grafana

For teams that already run OpenTelemetry, the opentelemetry-instrumentation-anthropic and equivalent packages emit LLM spans automatically. This routes into your existing observability stack at the cost of less LLM-specific analysis. Best for platform engineering teams that want AI signals alongside infrastructure signals in a single Grafana instance.

Implementation Patterns

Pattern 1: The Tracing Wrapper

Wrap every LLM call with a tracing decorator that captures input, output, model, latency, and token count. In Langfuse this is a context manager; in OpenTelemetry it is a span. The key discipline is to trace at the logical operation level—"answer question about invoice"—not the raw API call level, so traces are readable by product and business stakeholders.

Pattern 2: Async Evaluation Pipeline

Never evaluate every response synchronously in the request path—it doubles latency. Instead, publish a copy of the request-response pair to a queue and process evaluations asynchronously. Store scores back in your observability platform within 30 seconds. Alert thresholds fire on a rolling 15-minute window of scores.

Pattern 3: Shadow Mode for Model Updates

When a new model version becomes available, route 5% of traffic to it while logging both old and new responses. Compare quality scores, latency, and cost before promoting. This is how you avoid the silent regression that comes from a provider updating the underlying model without notice.

Pattern 4: Business Metric Correlation

The most valuable observability insight is not "quality score dropped from 0.82 to 0.79"—it is "conversion on the AI-assisted checkout fell 3% after the model update." Pipe your AI observability events into your product analytics platform (Amplitude, Mixpanel, or a data warehouse) and build dashboards that join model signals to business outcomes.

Alerting That Surfaces Real Problems

Most teams start with too many alerts on too-sensitive thresholds and end up ignoring them all. Start with three:

1. P95 latency > 2× baseline for 5 minutes. This catches serving infrastructure problems and runaway context growth.
2. Quality score < threshold for 30-minute rolling window. Calibrate the threshold during a known-good period; 1.5 standard deviations below mean is a reasonable starting point.
3. Safety classifier hit rate > 3× baseline for 10 minutes. Indicates an active adversarial attempt or a model policy change that is blocking legitimate requests.

Add cost alerts separately—set at 60% and 90% of monthly budget—and review weekly. Operational and cost alerts should go to different channels; mixing them trains engineers to ignore cost alerts.

Common Mistakes

• Monitoring infrastructure but not model behaviour. Knowing your Lambda function is healthy tells you nothing about whether the LLM is giving useful answers.
• Evaluating only happy-path samples. The worst outputs tend to come from edge-case inputs. Sample adversarially—include the inputs that scored lowest last week in your evaluation set.
• Not versioning prompts. If you change a prompt without recording the change, you cannot correlate the quality shift to its cause. Treat prompts as code: version control, code review, staged rollout.
• Setting alert thresholds on day one. You do not know what "normal" looks like until you have two to four weeks of production data. Collect data for a month before setting numeric thresholds.
• Ignoring the retrieval layer in RAG. In a retrieval-augmented system, most quality failures originate in retrieval—the wrong chunks are returned—not in generation. Monitor retrieval separately.

Frequently Asked Questions

Does observability add latency to my application?

Instrumentation overhead is typically under 5 milliseconds for synchronous tracing. Evaluations should always run asynchronously, adding zero latency to the user path. The main risk is accidental synchronous evaluation—design your pipeline explicitly to avoid this.

What sample rate should I use for evaluations?

For LLM-as-judge evaluation, 2–5% is sufficient for statistically meaningful quality trends at moderate traffic volumes (over 1,000 requests per day). For human review, 0.5–1% is realistic given reviewer bandwidth. Always sample 100% of safety-flagged responses regardless of overall rate.

Can I use the same observability stack for both LLMs and traditional ML?

Tools like Arize AI and WhyLabs support both. However, the signal types differ enough that teams often maintain separate dashboards: one for LLM trace-level observability and one for traditional ML drift and accuracy. Shared infrastructure (data warehouse, alerting platform) is sensible; shared dashboards less so.

How do I handle observability for third-party model updates?

Subscribe to model provider changelogs and run your quality evaluation suite whenever a provider announces an update. Keep a frozen evaluation dataset—at least 200 representative queries with expected outputs—so you can benchmark before and after. If quality drops, you have documented evidence to raise with the provider or to justify switching models.

Conclusion

AI observability is not optional in production—it is the difference between knowing your AI system is working and assuming it is. Start with the five core signals: latency, cost, quality, safety, and drift. Instrument synchronously but evaluate asynchronously. Pick one tool from the landscape—Langfuse for LLM chains, Arize or Evidently for traditional ML—and build the habit of reviewing dashboards weekly before the first incident forces you to.

The teams that get the most value from production AI are not the ones with the most sophisticated models. They are the ones that know, with confidence, what their models are doing every hour of every day. Halkwinds helps engineering teams build that confidence—from observability architecture to automated evaluation pipelines—so AI systems can be trusted at scale.