Data Observability: Monitoring Your Data Pipelines for Quality

Every data engineer has lived through the same nightmare: a dashboard executives rely on shows revenue dropping 40% overnight, panic ensues, and after two hours of frantic debugging you discover an upstream API changed a field from cents to dollars. The data was never "down" — it was just wrong. This is the fundamental problem data observability solves. Unlike traditional monitoring, which tells you whether a job succeeded or failed, data observability tells you whether the data itself is trustworthy. This article walks through the concepts, architecture, and practical implementation of a data observability practice that catches incidents before your stakeholders do.

• Background / Why This Matters
• Core Concepts and Architecture
• Implementation Strategy
• Scaling and Operational Considerations
• Common Mistakes / What to Avoid
• Frequently Asked Questions
• Conclusion

Background / Why This Matters

The modern data stack has made it trivial to move data. Tools like Fivetran, Airbyte, and dbt let a small team ingest hundreds of sources and build thousands of models. But this same ease of movement has created a fragility problem: the more pipelines you have, the more surface area exists for silent failure.

Traditional pipeline monitoring answers questions like "Did the DAG run?" and "Did the job exit with code 0?" Those are necessary but wildly insufficient. A job can complete successfully while:

• Loading zero rows because an upstream table was truncated
• Duplicating records because a merge key changed
• Silently dropping a column that three downstream models depend on
• Delivering data 6 hours late because a source system slowed down

Research and industry surveys consistently suggest that data teams spend a significant portion of their week — estimates vary, but often cited figures land around 30–50% — firefighting data quality issues rather than building. Worse, these issues erode trust. Once a business user catches a broken number, they stop trusting the entire platform, and adoption collapses.

Data observability shifts your posture from reactive ("someone reported a bug") to proactive ("our system flagged an anomaly at 2am and paused the downstream refresh").

Takeaway: If your only signal that data is broken is a Slack message from an angry stakeholder, you don't have observability — you have a rumor mill. The goal is to make the data itself observable, measurable, and alertable.

Core Concepts and Architecture

Data observability is typically decomposed into what practitioners call the five pillars. Understanding each helps you decide what to instrument first.

The Five Pillars

• Freshness — Is the data up to date? A table that normally updates every hour but hasn't changed in 12 hours is a red flag, even if no job "failed."
• Volume — Did the expected number of rows arrive? A table that usually gets 1M rows/day suddenly receiving 200 rows signals an upstream break.
• Schema — Did the structure change? Columns added, dropped, renamed, or retyped (schema drift) frequently break downstream logic silently.
• Distribution — Are the values within expected ranges? Null rates, cardinality, min/max, and category distributions catch subtle corruption.
• Lineage — When something breaks, which downstream assets are affected, and which upstream source caused it? Lineage turns a single alert into a blast-radius map.

Reference Architecture

A mature observability setup sits alongside your pipeline rather than inside it. The typical components are:

1. Metadata collection layer — Query logs, information schema, and warehouse system tables (Snowflake's ACCOUNT_USAGE, BigQuery's INFORMATION_SCHEMA) feed freshness and volume metrics without touching raw data.
2. Test and assertion layer — In-pipeline checks like dbt tests validate business rules at transform time (e.g., unique, not_null, accepted_values, and custom SQL assertions).
3. Anomaly detection layer — ML-driven baselining (as offered by tools like Monte Carlo) learns normal behavior for freshness and volume, so you don't hand-write thresholds for thousands of tables.
4. Lineage graph — Column-level lineage (Datafold specializes here) maps how a change in one column ripples through the DAG.
5. Alerting and incident management — Routing to Slack, PagerDuty, or Opsgenie with ownership metadata so the right team responds.

Takeaway: Don't try to build all five pillars at once. Freshness and volume give you the highest signal-to-effort ratio because they can be derived automatically from warehouse metadata.

Implementation Strategy

The right approach depends on your team's maturity, budget, and how many pipelines you run. Here's a phased strategy that avoids the common trap of buying an expensive platform before you understand your own data.

Phase 1: Instrument the Basics with What You Already Have

Before any purchase, add dbt tests to your most critical models. Start with the "golden tables" that feed executive dashboards and finance reports. Concretely:

• Add not_null and unique tests on primary keys.
• Add relationships tests to catch broken joins.
• Write singular tests for business invariants — e.g., total_revenue >= 0 or order_status in ('placed','shipped','cancelled').
• Use dbt source freshness to alert when a source table hasn't loaded within its SLA.

This costs nothing beyond engineering time and catches a surprising percentage of incidents.

Phase 2: Add Automated Anomaly Detection

Hand-written thresholds don't scale past a few dozen tables. When you're monitoring hundreds, you need a system that learns baselines automatically. This is where a platform like Monte Carlo earns its keep — it profiles historical freshness and volume, then flags deviations without you defining rules per table.

Phase 3: Establish Lineage and Ownership

Once alerts fire reliably, the next bottleneck is triage. Column-level lineage (Datafold, or the lineage features in Monte Carlo) tells you exactly which downstream reports break when an upstream column changes. Pair lineage with an ownership model — every dataset should have a named owner and an on-call rotation.

Build vs. Buy Comparison

Approach	Best For	Coverage	Ongoing Cost	Time to Value
dbt tests only	Small teams, critical models	Business rules, freshness	Engineering time	Days
dbt + Datafold	Teams shipping frequent model changes	Regression testing, column lineage	Moderate license	1–2 weeks
Monte Carlo (full platform)	Large orgs, hundreds of tables	All five pillars, ML anomaly detection	Higher license	2–4 weeks
Fully custom in-house	Unique compliance needs, strong platform team	Whatever you build	High engineering	Months

At Halkwinds, our Data & Analytics practice typically recommends starting with dbt tests plus a targeted vendor for anomaly detection, reserving fully custom builds for teams with regulatory constraints or scale that outgrows off-the-shelf tooling.

Takeaway: Layer your tooling. In-pipeline assertions (dbt tests) catch known failure modes; anomaly detection catches the unknown unknowns. You need both.

Scaling and Operational Considerations

Observability that works for 50 tables can drown you at 5,000. Scaling introduces new problems, most of which are organizational rather than technical.

Alert Fatigue Is the Silent Killer

The fastest way to kill an observability program is to generate 200 alerts a day. Teams stop reading them, and a real incident gets buried. Mitigate this by:

• Tiering assets. Only page for Tier-1 tables (finance, exec dashboards). Lower tiers get a daily digest, not a 2am page.
• Deduplicating with lineage. One upstream failure shouldn't fire 40 alerts for every affected downstream model — collapse them into a single root-cause incident.
• Tuning sensitivity. Suppress alerts during known maintenance windows and expected seasonal spikes.

Cost of Monitoring

Freshness and volume checks that query warehouse metadata are cheap. But distribution checks that scan full tables can rack up serious compute cost. On Snowflake or BigQuery, prefer approximate aggregates, sample large tables, and schedule heavy profiling during off-peak hours. Monitor your monitoring bill.

Embedding Checks in CI/CD

Catching bad data in production is good; catching it before deploy is better. Run dbt tests in CI on every pull request. Use Datafold's data diff to show reviewers exactly how a proposed model change alters row counts and value distributions before it merges. This shifts quality left, which is dramatically cheaper than incident response.

Ownership and SLAs

Define explicit data SLAs — e.g., "the daily_revenue table will be fresh by 6am ET with a 99.5% monthly reliability target." Publish these. SLAs create accountability and give you a concrete metric to improve against.

Takeaway: Scaling observability is mostly about reducing noise and assigning ownership, not adding more checks. A precise alert to the right owner beats a thousand ignored ones.

Common Mistakes / What to Avoid

Having implemented observability across a range of stacks, these are the failure patterns we see most often.

• Monitoring everything equally. Not all tables matter. A staging table's null rate is noise; the finance mart's null rate is a fire. Prioritize ruthlessly.
• Buying a platform before inst