TL;DR
- Start with four measures: throughput, latency, error rate and freshness. They show whether data is moving and whether it is current.
- A job that reports success can still deliver bad data. Add checks on row counts, schema changes and value ranges.
- Pick alerts with an owner and a next step. An alert nobody acts on is noise.
- Tools fall into four types. Most teams need more than one, and a plan matters more than the tool.
- Monitor the dashboard layer too. A stale refresh is the failure your leaders see first.
Data pipeline monitoring is the routine of watching data as it moves from source systems to the reports people use, so you learn about a problem before an executive does. It checks that jobs run, that data arrives on time, and that the numbers still make sense. This guide explains what to measure, how to set alerts, and how to start without buying a pile of tools.
What data pipeline monitoring covers
A data pipeline is a chain of steps. Data is pulled from sources such as an electronic health record, an accounting system or a CRM. It is cleaned, joined, and loaded into a warehouse. Then a dashboard reads from it.
Monitoring watches each step. It answers simple questions:
- Did the job start and finish?
- Did the expected amount of data arrive?
- Is the data recent enough to use?
- Did anything change upstream that will break the next step?
You may also see the word observability. The two terms overlap. Monitoring tells you the current state of the pipeline and the data, such as whether it is complete, accurate and fresh. IBM describes observability as the wider ability to understand a pipeline's internal state at any time. In practice, monitoring tells you something is wrong. Observability helps you see why.
The four metrics to track first
A useful starting set comes from RudderStack's guide to pipeline monitoring. It names four core measures.
| Metric | What it tells you | Example question |
|---|---|---|
| Throughput | Volume processed per unit of time | Did we load the usual number of claims today? |
| Latency | How long processing takes | Did the nightly job finish before 6 a.m.? |
| Error rate | Share of operations that fail | How many records were rejected? |
| Freshness | How current the data is | When was the sales table last updated? |
Freshness is the one business users feel most. A dashboard can look perfect and still show yesterday's numbers. If the page says "updated this morning" and the data stopped Tuesday, trust drops fast.
Add content checks
The four metrics describe movement. They do not describe meaning. A pipeline can run on time and load half the rows it should. Add checks on the data itself:
- Volume: Compare today's row count with a normal range for that weekday.
- Schema: Flag new, missing or renamed columns.
- Values: Check for nulls in required fields, negative amounts, and dates in the future.
- Duplicates: Check key fields that should be unique.
- Reconciliation: Compare totals between source and destination.
The Kestra decision guide explains why this matters. Observability platforms build a baseline of normal patterns. That lets them catch a silent failure, such as a sudden drop in row counts, even when the job technically reports success.
If you want the wider view of this topic, our data quality management guide covers rules, ownership and scoring.
Types of pipeline problems and how to catch them
Most incidents fall into a few groups. Matching each to a check keeps your setup small.
| Problem | Typical cause | Check that catches it |
|---|---|---|
| Job did not run | Scheduler or credential failure | Orchestrator alert on missed run |
| Job ran but is slow | Growing data, poor query design | Latency threshold, trend over time |
| Data is late | Upstream delay | Freshness check against a deadline |
| Data is missing | Partial extract, API limit | Row count against expected range |
| Data changed shape | Source system update | Schema change detection |
| Numbers look wrong | Logic error, bad join | Reconciliation and value range tests |
Notice that only the first row is a plain failure. The rest can pass unnoticed unless you check for them on purpose.
The four types of monitoring tools
The market is crowded, so it helps to sort tools by what they watch. The Kestra guide groups them into four categories:
- Orchestrators. They track whether jobs succeeded and how long they took.
- Data observability platforms. They watch data health, including volume and schema drift.
- Quality frameworks. They test specific rules about data content.
- Application performance monitoring tools. They watch the infrastructure underneath.
No single category covers everything, and combining them is common. A small team might use its orchestrator's alerts, a handful of quality tests, and the refresh notices built into its BI software. A larger team might add a dedicated observability platform.
A warning from Datadog's pipeline monitoring guide: the most common mistake is treating observability as a collection of tools instead of a strategy. That leaves gaps. The same guide recommends standard telemetry formats such as OpenTelemetry so your metrics, logs and traces can be correlated across systems.
Do not forget lineage
Lineage is a map of where data came from and what depends on it. It is often an afterthought. It becomes urgent the day someone wants to change a column and nobody knows which reports use it.
The open OpenLineage standard can collect this metadata automatically. The Apache Airflow OpenLineage documentation describes how it captures dependencies across DAGs and tasks. That gives you a visual map for impact analysis before you ship a change.
From reactive to proactive monitoring
Reactive monitoring tells you a job failed. Proactive monitoring tells you what is likely to fail and why. IBM's guidance recommends moving toward the second, using lineage and tracing to explain why data quality is falling downstream.
A practical way to make that move:
- Fix the noisy alerts first. Remove any that nobody acts on.
- Test at the source. Catching a bad file at ingestion is cheaper than finding it in a quarterly report.
- Track trends, not only thresholds. A job that takes 10 minutes more each week will fail eventually.
- Record every incident and its cause. Patterns show where to invest.
How to set up monitoring in six steps
You do not need a large program. Work in this order.
- List the reports that matter most. Pick the five dashboards or reports leaders use to make decisions. Start there.
- Trace each report back to its sources. Write down every table, file and system feeding it. This is a simple lineage map.
- Set a freshness deadline for each. Example: the finance dashboard must reflect the prior day by 7 a.m.
- Add volume and schema checks on the most important tables.
- Assign an owner for each alert. One named person or role, plus a short note on what to do first.
- Review monthly. Look at alerts that fired, which were real, and which checks are missing.
Keep the first version small. Ten well chosen checks beat two hundred that no one reads.
Write alerts people act on
Alert fatigue is the main reason monitoring fails. When everything pages someone, people mute the channel.
A good alert has four parts:
- What failed, in plain words
- Which reports or teams are affected
- How serious it is
- The first step to take
Split alerts by urgency. A missed executive refresh before a board meeting needs an immediate message. A small row count drift in a test table can wait for a daily summary.
Monitor the dashboard layer
Many guides stop at the warehouse. Your business users do not. They open a dashboard, and that is where a failure becomes visible.
BI tools have their own refresh jobs, and those can fail even when the warehouse is healthy. Microsoft documents that Power BI sends refresh failure notifications to the semantic model owner by email by default. That is a useful baseline. It only helps if the owner is a real person who still works on the team and still reads that inbox. Check who the owner is on your key reports today.
For Tableau and Power BI environments, also watch:
- Extract and dataset refresh status and duration
- Slow dashboards, which often signal a heavy query or a poor data model
- Visible "last updated" labels, so users can judge freshness themselves
- Changes to data sources that reports depend on
Slow dashboards are often a pipeline and modeling problem, not a visual one. Our ETL vs ELT guide explains how where you transform data affects speed and cost.
Sector notes: healthcare and finance
Healthcare
In healthcare, monitoring pipeline runs is not enough. You also need to look at data at rest. Stored records can hold duplicates or errors that a successful job never reveals. NIST's qDAR tool was built to assess the quality of data sitting in immunization information systems, which shows how seriously public health programs treat stored data quality. A duplicate patient record affects counts, outcomes and follow up.
If your pipelines carry patient data, monitoring also has to respect privacy rules. Logs and alert messages should not expose protected health information. Our guide to HIPAA compliant analytics covers this. For a worked example of a healthcare reporting build, see our patient assessment analytics dashboard case study.
Finance
Finance teams care about reliable delivery of data on a schedule. Dynatrace documents how DORA metrics from software delivery can be adapted to measure pipeline health. These are lead time for changes, deployment frequency, change failure rate and mean time to restore. The last two are especially useful: how often do changes break something, and how fast do you recover?
For month end close, add reconciliation checks that tie totals back to the general ledger. A dashboard that is off by a small amount is worse than one that is clearly late.
What we see in practice
We build dashboards in Tableau and Power BI, and we usually meet monitoring as a symptom. A leader says the numbers look off. The dashboard is rarely the cause. The cause is upstream: a source that changed a field, a refresh that quietly stopped, or a join that doubled some rows.
That is why we start engagements by diagnosing the data flow before touching visuals. A good looking dashboard on an unreliable feed only hides the problem. Our view is simple:
- Monitor what the business depends on first, not everything.
- Put the freshness time on the dashboard itself.
- Give every alert a named owner.
- Treat monitoring as part of the data strategy, with the same attention as the dashboard design.
Teams often ask whether they need a new platform. Usually not at first. Most gain a lot from clear ownership, a short list of checks, and a documented path from source to report. Tools come after that.
Get help with your pipelines and dashboards
If your reports are late, slow or hard to trust, we can look at the whole flow from source to dashboard. We work on data strategy and governance, build and tune BI dashboards, and automate reporting so refreshes are dependable. You can see our featured Tableau dashboards or book a free assessment to review where your data flow is breaking down.
Frequently asked questions
Is a data pipeline the same as ETL?
Not exactly. ETL is one kind of data pipeline, where data is extracted, transformed, then loaded. Pipelines also include ELT, streaming and simple copy jobs. Monitoring applies to all of them.
What should I monitor first?
Start with the reports leaders use for decisions. For each one, track freshness, row counts on its key tables, and refresh status in the BI tool. Add schema checks next. This covers most common failures with a small amount of work.
What is the difference between monitoring and observability?
Monitoring watches known measures and tells you when one crosses a limit. Observability adds context, such as lineage and traces, so you can find the cause of a problem you did not predict. Most teams begin with monitoring and grow into observability.
Do small teams need a dedicated monitoring tool?
Often not at the start. Orchestrator alerts, a few quality tests and BI refresh notifications can cover a lot. Consider a dedicated platform when the number of pipelines grows, or when failures keep reaching users before your team sees them.
