We checked the facts and tool details here against each vendor's own pages on October 1, 2026. Definitions and feature claims below are the vendors' own wording, and we have not tested any of them. Tools change often, so confirm anything before you spend money on it.

So, what is a data pipeline?​

Think of it as a delivery route for data. It starts in a source, such as a shop's database or a payment app. The end point is a destination, such as a warehouse (a database built for reports), a data lake (cheap file storage that many tools can read), or a lakehouse (a data lake organised so tools can query it like a warehouse).

AWS's page uses a plain picture: like water moving from a reservoir to your taps, a pipeline moves data from where it is collected to where it is stored. Its definition is a series of processing steps that prepare enterprise data for analysis. AWS sells data services, so read this as AWS's wording, not settled fact.

AWS's own definition, from its "What is a Data Pipeline?" page, October 2026.

Two words cause most of the confusion, so let's sort them out now. Pipeline is the whole path. ETL (extract, transform, load) is one way to build that path, where data gets reshaped before it lands.

International Business Machines (IBM) says ETL follows a fixed extract, transform, load order, while a data pipeline "may simply move data from one system to another" without transformations. That is IBM's definition, and other vendors draw the line differently. On IBM's reading, every ETL pipeline is a data pipeline, but not the other way round.

One more trap is product naming. In Azure Data Factory, Microsoft uses "pipeline" for a logical grouping of activities that together perform a task. That's one product's object, not the general idea.

The parts of a data pipeline​

IBM says three core stages make up a pipeline: data ingestion, data transformation and data storage. Transformation is the optional one, since a pipeline may simply move data.

Ingestion means copying data out of its sources. Cleaning it and changing its shape is transformation, and storage means putting it where people can use it.

Here's the part people forget. A fourth piece sits beside those three.

Something has to decide when each step runs, retry it when it fails, and warn a person when it keeps failing. AWS's page says a well-designed pipeline keeps working even if individual components fail, and that you must protect data accuracy despite interruptions. That is a design goal, and no vendor page proves a given tool meets it.

Most real problems live in the lower lane of that diagram. A one-off copy is easy, but a copy that runs every day for two years, through failed runs and changed columns, is the hard part.

What are data pipelines used for?​

Common uses, per IBM, include fraud detection, exploratory analysis, dashboards, machine learning and data quality monitoring and more. The use decides the speed you need. Fraud checks need seconds, while a dashboard usually doesn't.

AWS adds that pipelines handle structured data (database rows), semi-structured data (such as JSON files) and unstructured data (such as images). Check that a tool reads the shape of data you have.

Data pipeline types, compared​

Types aren't one tidy list, because they answer different questions. Batch and streaming answer "how often." ETL and ELT (extract, load, transform) answer "where does the reshaping happen."

Change data capture (CDC) answers "how do we find what changed," and reverse ETL answers "which way does the data flow."

Below are six common kinds, each with its limit.

According to IBM, batch loads batches of data at set intervals, typically off-peak. IBM says streaming pipelines, also called event-driven architectures, continuously process events from sources such as sensors or user interactions in an app. AWS adds that batch pipelines run infrequently while stream processing pipelines run continuously.

Change data capture deserves a longer note, because it's the type most people haven't met. Log-based CDC reads a log the database already writes instead of querying your tables again and again. In PostgreSQL, for example, logical decoding works by decoding the write-ahead log (the file where the database records its changes) into a stream that outside programs can read.

Confluent's page says log-based CDC "doesn't impose computational overhead on source databases". That is vendor wording that we have not tested, and the same page notes that log formats are not standardized.

It says the other common method, timestamp-based CDC, can only process soft deletes and requires changes to the existing database schema.

The source still has chores, though. In PostgreSQL, a replication slot (a bookmark that tracks a change reader's place) prevents removal of the log files it needs even when no connection is using it. The docs say this consumes storage and, in extreme cases, could shut the database down, so a slot that is no longer needed should be dropped.

The docs name a place to look: the pg_replication_slots view has a wal_status column. There, "extended" means the log has grown past max_wal_size but is still being kept. "Lost" means the slot is no longer usable.

ETL and ELT differ in where the reshaping happens. On the ELT side, IBM says it has a lower upfront cost, but unpredictable data volumes can cause unexpected costs later. IBM adds that masking personal data in transit before loading is an ETL strength, and it still calls ETL a popular choice.

Reverse ETL simply runs the other way. Fivetran's docs describe its Activations product as managed reverse ETL pipelines that deliver warehouse data into business tools. "Managed" is Fivetran's word, and Activations rows are billed on the same monthly-active-rows meter, per its usage-based pricing docs.

How to choose: four questions, not one​

The six labels above answer different questions, so freshness alone can't pick a row. Split the choice into four parts instead.

• How old can the data be? Hours or a day, minutes, or seconds (batch or streaming).
• How do we find what changed? Copy whole tables, read timestamps, or read the database log (change data capture).
• Where does the reshaping happen? Before loading or after loading (ETL or ELT).
• Which way does the data flow? Into the warehouse, or back out to business tools (reverse ETL).

Real designs combine the answers. For example, change capture can copy database changes into storage as they happen, and a scheduled job can build the reports from that data afterward. That one design mixes a change-log method, the ELT order and a batch-style schedule.

The diagram covers only the first question, timing. Start there, because you shouldn't pay for seconds when a report is read once a day.

A worked example: an online shop's orders​

This example is an illustration with invented rows, not a real customer and not a test. Picture a small shop that keeps orders in a Postgres database, where the owner wants a daily sales report. Here's each stage, plus the one thing most likely to go wrong there.

The copy step ends when rows land in the warehouse, and the SQL query that adds up sales is the next step. Here are three invented orders to show the delete problem from the second row.

That gap is why the CDC type exists. A hard delete removes the row, and per Confluent, timestamp-based capture can only process soft deletes, so a scan has nothing to find.

A script with no alerts and no retries may still move data, but it isn't yet a production-ready pipeline.

Which tools do which job​

A pipeline is a design, and tools are the parts you build it from. Each tool below does one slice, and most teams end up needing more than one.

Fivetran and dbt are now one company, because Fivetran announced the completion of its merger with dbt Labs on June 1, 2026. Check each vendor's pages for any change to how they're sold.

Before you rely on any vendor's freshness or reliability claim, ask for the worst-case delay (not the default schedule), how failed syncs and deleted rows are handled, and what exactly the bill counts.

What goes wrong, and how to guard against it​

Most failures are boring and predictable, and each one maps to a row of the example above.

• Stale data. A report is only as fresh as the last successful run plus the time the later steps take. Fivetran's docs say a failed sync is retried after the update interval or one hour, whichever is shorter, so a failure can add that wait.
• Missing deletes. Methods that look only for new or changed rows can miss rows that were removed. Reading the database log is one way around it, with the setup cost described above.
• Duplicates after a crash. PostgreSQL says that after a crash a slot might return to an earlier position, so recent changes are sent again. The receiving side must be able to handle the same message twice.
• Duplicates after a rerun. The Airflow docs say a retried task should produce the same outcome on every re-run. The word for that is idempotent, which just means safe to run twice.
• Changed columns. If the source renames or drops a column, the steps after it can fail, and nobody notices without an alert.
• Cost surprises. Billing meters count rows, events or gigabytes, and each rises differently.

Here is the rerun case with invented rows, as an illustration and not a test. Say a run that loads order 1003 (a 20-dollar sale) fails halfway and the scheduler runs it again.

The Airflow page says not to add rows blindly on a rerun, because that can create duplicates. Use an upsert (update the row if it exists, add it if not), which needs a column that identifies each row, here the order number.

That is one guard, not the whole answer, because a replayed old change must not overwrite a newer one. For scheduled batch tasks, the same page also says to read and write one named slice of data, such as one day, rather than "the latest data."

Build it yourself or buy a tool?​

Match your situation to a path below, then check the limit before you commit.

Frequently asked questions​

Conclusion​

Start with how old the data can be. Then add a scheduler, safe reruns and alerts, and check each vendor page before you spend money.

ETL (extract, transform, load) pipelines are easy to describe and easy to get wrong. An ETL pipeline is an automated process that extracts data from your source systems, transforms it (cleans and reshapes it), and loads it into a destination such as a data warehouse or data lake. In ELT (extract, load, transform) the same reshaping happens after the data lands instead of before.

Facts below come from linked vendor, project and reference pages, read on October 1, 2026. We did not build or run a pipeline for this guide, and the one worked example uses invented rows. Several of the pages are from companies that sell cloud services or data tools, so their claims are reported as their claims and we have not tested them.

What is an ETL pipeline, in plain words?​

Picture a restaurant chain that wants one report on yesterday's sales. The orders sit in a database, the refunds sit in a spreadsheet, and the delivery sales come from an app. Somebody has to collect all three, fix the mismatches, and put the result in one place.

An ETL pipeline automates that job, so nobody repeats it by hand each morning.

First it extracts (copies) the data from each source. Then it transforms (cleans, reshapes and combines) that data. Last, it loads (writes) the result into a destination where people can query it.

AWS defines ETL as combining data from multiple sources into a large, central repository called a data warehouse, using business rules to clean and organize it first. Other sources widen the target a bit. Google Cloud also names a database, a data store or a data lake as destinations.

Two ideas hide inside the word "pipeline". The process is automated, so it runs on a schedule or a trigger. And it's repeatable, so the same steps run again and again on fresh data.

The three steps of an ETL pipeline​

Extract: copy the data out​

Extraction pulls records out of the source. Sources can be databases, files, web services (APIs) and event streams. Wikipedia's ETL article adds that extraction usually includes a validation check, and that rows which fail the rules should ideally be rejected and reported back so someone can correct them.

The hard part isn't copying. It's deciding what to copy each time.

One common design is a first run that copies everything, which is called a full load. Later runs then copy only what's new or changed, which is called an incremental load. We'll get to how that works in a minute.

Transform: clean and reshape​

This is the step that makes ETL, well, ETL. Microsoft lists the usual operations as filtering, sorting, aggregating, joining, cleaning, deduplicating and validating data. In its description, the transformation applies business rules, using a specialised engine, often with staging tables that temporarily hold the data while it is processed.

Google Cloud says extracted data is first loaded into a staging area, then cleaned and put into a common format, which typically means removing duplicate, incomplete or obviously wrong records.

International Business Machines (IBM) adds one reason people still pick ETL. It says ETL can mask sensitive data while it's still in transit, before it reaches the destination. That matters if you'd rather not have private columns land anywhere they shouldn't, but it is IBM's claim and we did not test it.

Load: write it to the destination​

Loading writes the prepared data to its final home. That can be a data warehouse (a database built for reports and analysis), a data lake (cheap storage for files in many formats) or a regular database. Estuary's guide notes that loads can be full (replace everything) or incremental (add or update only what changed).

Microsoft says the three phases often overlap, so don't picture a strict assembly line. While data is still being extracted, a transformation can already work on what has arrived. Loading can start on prepared data too, instead of waiting for extraction to finish.

What surrounds the three steps​

A daily pipeline also needs something to schedule it and something to watch it.

Microsoft splits the scheduling side into two ideas. Control flow decides the order of tasks, and a task can't start until the one before it has finished with a success, failure or completion result. Data flow is the movement and reshaping of the data inside one task.

On the tooling side, Estuary's guide names Apache Airflow, Dagster and Prefect as orchestrators, the programs that run the steps on a schedule. Stripe's guide says to monitor job runtimes, row counts, error rates and freshness, and to set alerts for failures.

A worked illustration with invented rows​

This example is made up. The orders, amounts and dates are invented, and nobody ran it as a test. It only shows how the three steps and the counts fit together.

Say a shop keeps orders in a table called raw_orders. Each row has an order number, an amount in cents, a status, a customer email and a last-updated time. The goal is a clean sales table the finance team can trust.

An earlier run already loaded order 090 into it, so sales starts with one row.

Now say the last run finished at midnight on September 30. The pipeline remembers that time, and that remembered point is called a watermark. Here's each step.

Extract. Read only the rows updated after the watermark. That skips order 090, which an earlier run already handled, so three rows are extracted: 101, 102 and 103.

Transform. Reject the test order (102), turn cents into a dollar amount, and leave the email column behind so it never reaches the destination. One row is rejected and two pass.

Load. Write the two passing rows into sales with an upsert. That's just "insert this row, or update it if the order number already exists".

After the run, sales holds three rows: the old 090 plus the new 101 and 103.

Now run the same window a second time. Because the load is an upsert, 101 and 103 are updated in place and sales still holds the same three rows. A plain insert could add them again.

That repeat-safety is the idea behind the "where pipelines break" section. Two gaps show up even in this tiny case.

First, a filter only decides what gets loaded. If order 101 is later changed to a test order, the next run extracts it and rejects it, so the old 101 row stays in sales. Removing it would take a separate delete or flag step.

Second, think about when the remembered time moves forward. If it jumps to the end of the day before the load finishes and the load then fails, 101 and 103 are skipped on the next run. So the sensible order is to move it only after the load succeeds.

Both points, and the counts above, are our reasoning from this invented example, not claims from a source. The counts rule is also specific to it: each row comes out once or is rejected. Joins and aggregations change row counts in other ways.

How pipelines pick up changes​

That watermark is the simplest way to run incremental loads. Microsoft's Azure Data Factory docs describe a watermark as a column that holds the last updated time or an incrementing key. A delta load then copies the data between an old watermark and a new one.

Here's the catch: a watermark needs a column that reliably moves forward.

If a table has no such column, or someone forgets to update it, rows get missed. The query only sees rows that still exist, so it can't show you a row that was deleted. Our read: when deletes matter, the pipeline needs a different method.

The same Microsoft page lists other methods. Change Tracking is a lightweight SQL Server feature that identifies inserted, updated or deleted data. For files, it describes filtering by last-modified date, and warns that scanning huge numbers of files to copy only a few is still slow.

There's also a fuller approach that reads the database's own change log.

Three ways to build one (and what each costs you)​

Estuary's guide names three common routes. One caveat: Estuary sells a pipeline product, so treat its framing as one vendor's view.

Estuary says production needs four things beyond the three steps: a staging area, a scheduler, incremental loads and validation. Its validation list: schema checks, null thresholds and row-count reconciliation, which means comparing how many rows left the source with how many arrived.

A cheap check is a row-count comparison after each load. Our read: when each row either passes or is rejected, compare extracted against rejected plus loaded (3 = 1 + 2 in the illustration). Comparing source rows to loaded rows would raise a false alarm whenever the transform drops something on purpose.

Where ETL pipelines break (and why it's rarely the three steps)​

Our read of the sources below: trouble usually starts outside the three steps, in reruns, late data, changing sources and silent errors. Four of these are worth knowing before you build or buy.

A rerun writes duplicates. Orchestrators retry failed tasks. That's their job, and it's also where the trouble starts.

The Apache Airflow best practices say a task should produce the same outcome on every re-run. On top of that, they say you shouldn't use a plain insert during a re-run because it might lead to duplicate rows. The advice is to use an upsert instead.

Apache Airflow best practices, "Creating a task", October 2026.

That page also says to read and write in a specific partition, and never to read "the latest available data" in a task. Someone may update the input between re-runs, and then you get different outputs. It also warns against using the current time (now()) inside a task for critical computation, since that gives a different result on each run.

A task that gives the same result when run twice is called idempotent (a fancy word for "safe to repeat"). Microsoft uses the same word and says to design parallel work around partition boundaries such as date, tenant or shard key, to avoid write conflicts and allow idempotent retries.

Half-finished results. The Airflow page says to treat tasks like transactions in a database, so a task never leaves incomplete results behind. Its example: don't leave incomplete data in file or object storage at the end of a task. Nobody wants to find half a file at 9 a.m.

Late or out-of-order data. For streaming pipelines, Microsoft recommends checkpointing for at-least-once processing and idempotent transformations to handle duplicates. It also recommends watermarking for late-arriving events and out-of-order processing, and dead letter queues (a holding place) for messages that can't be processed.

That streaming "watermarking" is a different idea from the remembered time in the worked illustration. The first deals with events that show up late. The second marks where the last batch stopped.

Delay by design. Estuary says traditional batch processing is straightforward to implement but introduces delay by design. That suits a nightly report, less so fraud checks.

What people use ETL pipelines for​

Qlik lists data migration from legacy systems, centralizing data into one source of truth, enriching data by combining platforms, supporting privacy rules and preparing data sets for analytics. Stripe's guide adds examples built on joined data, such as marketing analytics that joins ad platforms, web data and customer records, and fraud detection that pulls together transaction, login and device data.

ETL pipeline vs data pipeline: not the same thing​

A data pipeline is the bigger idea: any automated movement of data from one place to another. IBM lists four common types: batch, streaming, data integration and cloud-native.

An ETL pipeline is a data pipeline that includes a transform step before loading. Not every data pipeline does that. Qlik says a data pipeline may transform after loading (ELT) or not transform at all.

Informatica says ETL pipelines run in batches while data pipelines work in real time. Google Cloud says streaming ETL pipelines now exist alongside batch ones, and IBM calls ETL with streaming technology "streaming ETL".

So batch versus real time is a choice you make. It's not what separates ETL from other pipelines.

ETL vs ELT: which order fits you?​

ELT means extract, load, transform. The only difference, per Microsoft, is where the transformation happens. In ELT it runs inside the destination, using that system's own processing power instead of a separate engine.

Microsoft's conditions are the clearest we found.

Each side has limits. Microsoft says ELT only works well when the target is powerful enough to transform the data efficiently. IBM says ELT can lower upfront cost, but unpredictable data volumes can cause unexpected costs later, and unstructured data can take time to format inside the target.

AWS says in its ETL vs ELT comparison that the two may be used together for complex analytics that use multiple data formats from varied sources. One more term you'll hear: a lakehouse is a data lake that adds table features such as reliable updates, so it can serve as an ELT target.

Two neighbours: reverse ETL and streaming​

People often mix these two up with ETL. Microsoft describes reverse ETL as moving transformed, modeled data from analytical systems such as a warehouse back into operational tools such as a customer relationship management (CRM) system, a marketing tool or a support system. It still follows extract, transform and load, only in the other direction.

Microsoft contrasts real-time streaming with ETL and ELT, which it describes as operating on datasets in scheduled batches. Streaming instead processes data as it arrives, through a message broker or event hub and a stream processor. It suits cases where low latency is critical, such as fraud detection or live dashboards.

Some sources use the phrase "streaming ETL", so how often a pipeline runs is a separate question from where the transformation happens.

So which should you pick?​

A checklist before you trust a pipeline​

Run these questions on a pipeline you've inherited, or on a tool you're about to buy. Each one maps to a failure described above.

• Does it copy everything once and then only changes, or work another way? How does it know what changed?
• Can it see deletes, or only new and updated rows?
• What happens if a run fails halfway? Can it be rerun without duplicates?
• Where does raw data sit, and who can read it? Are sensitive columns masked before loading?
• Who is alerted when a run fails or loads zero rows?
• What does a late-arriving row do to yesterday's numbers?

If a vendor can't answer these in its docs, the answer is "not published". Ask before you sign, and treat any latency, uptime, "no maintenance" or savings claim as unverified until you see it in the docs or in a test you ran.

For an alert rule, here is our read of the illustration above. Alert when a run fails, when extracted does not equal rejected plus loaded, or when a run loads zero rows on a day you expect changes.

Conclusion​

An ETL pipeline is three steps wrapped in automation. What decides whether one is any good is how it finds changes, how it handles a failed run, and how fast someone learns that something broke.

Go with ETL when you need to reshape or mask data before it lands, or when audit rules ask for it. Pick ELT when your destination is strong and you want to keep the raw data around. Plenty of teams end up using both.

Last reviewed October 2026.

FAQs​