Skip to main content

DB2 to BigQuery: Incremental Sync via Iceberg

Moving IBM DB2 LUW (Linux, UNIX and Windows) data to BigQuery usually means a licensed replication product feeding Kafka, or a nightly EXPORT and bq load. This guide does it with OLake Go.

It reads the DB2 tables with a read-only user and writes them once as Apache Iceberg tables into a BigLake catalog. BigLake is Google Cloud's Iceberg representational state transfer (REST) catalog, and a cursor column keeps the tables fresh.

The files stay in your own Cloud Storage bucket. BigQuery queries the tables through the catalog with a four-part name, with no dataset to create and no external table to declare, and DuckDB and Spark read the same tables.

By Shubham Satish BaldavaUpdated October 30, 2026

Key takeaways

  • OLake Go replicates DB2 LUW into Iceberg tables in a BigLake catalog: a full historical snapshot first, then incremental syncs on a cursor column such as UPDATED_AT, with no replication license or load job in between.
  • BigQuery reads the catalog as a data source. Every change a sync commits is visible on the next SELECT, with no refresh.
  • The same tables, written once by OLake Go, are also queryable from DuckDB and Spark. In the run on this page all three engines returned identical rows.

The short answer. To replicate DB2 to BigQuery, run OLake Go with a read-only DB2 user as the source and a BigLake catalog with a Cloud Storage bucket as the Iceberg destination, with BigQuery as the target query engine. The first sync copies each table in full (the historical snapshot); every later sync fetches the rows whose cursor column moved and upserts them. BigQuery, DuckDB and Spark then query the same Iceberg tables, with no second copy.

How DB2 data reaches BigQuery here​

The first sync reads the row and page counts DB2 keeps for each table, splits the table into chunks, and reads the chunks in parallel. It writes Parquet data files and records the highest cursor value it saw. Every later sync selects rows above that value, writes them as new data files, and adds a delete file that retires the old version of each updated row.

Each sync ends with one commit to BigLake. OLake Go uploads the data and delete files to the bucket with the service account's own credentials, then updates the table's metadata pointer in the catalog. BigQuery, DuckDB and Spark see the new snapshot on their next query; none of them keeps a copy.

The cursor is the whole contract. A row that changes without its UPDATED_AT moving is never picked up, and a deleted row stays in the Iceberg table. Set the column in every write path, and soft-delete if deletes matter.

What you need​

  • DB2 LUW 11.5.3 or later, with a read-only user, statistics on each table and a cursor column. The tables on this page ran on DB2 12.1.5 Community Edition in Docker.
  • An amd64 host for the DB2 connector. IBM's driver has no Linux arm64 build. On an Apple silicon Mac, turn on Rosetta in Docker Desktop.
  • A Google Cloud project with the BigLake API enabled, a Cloud Storage bucket for the files, and a service account for OLake Go with the BigLake Editor role on the project and Storage Object User on the bucket. The run on this page used a bucket in us-central1.
  • BigQuery in the same project. Queries run in the bucket's region, and the account running them needs the BigQuery Job User role plus read access to the catalog.
  • OLake UI from the Docker Compose install. The run on this page used OLake Go v1.1.0.

Step 1: Prepare DB2 for incremental sync​

Most readers point OLake Go at tables they already have. To follow along with the exact data from this page, create these two tables first:

The two sample tables used on this page
Sample schema, in schema DB2INST1 of database SHOP
CREATE TABLE customers (
  id INT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
  name VARCHAR(100), email VARCHAR(200), city VARCHAR(100), tier VARCHAR(20),
  created_at TIMESTAMP DEFAULT CURRENT TIMESTAMP,
  updated_at TIMESTAMP DEFAULT CURRENT TIMESTAMP);

CREATE TABLE orders (
  id INT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
  customer_id INT REFERENCES customers(id),
  product VARCHAR(100), amount DECIMAL(10,2), status VARCHAR(20),
  created_at TIMESTAMP DEFAULT CURRENT TIMESTAMP,
  updated_at TIMESTAMP DEFAULT CURRENT TIMESTAMP);

-- Then insert whatever you like. This page started with 8 customers and 40 orders.

OLake Go reads DB2 LUW (Linux, UNIX and Windows) tables through IBM's Data Server driver: a parallel full snapshot first, then incremental syncs that fetch only rows whose cursor column moved past the last value seen. The DB2 connector has no log-based change data capture (CDC), so four things must be in place before the first sync.

SettingValueWhy it matters
A read-only userCONNECT on the database plus SELECT on each tableThe snapshot and every incremental sync read the tables directly. DB2 authenticates users against the operating system or LDAP, so the account exists there first and is then granted in the database.
Table statisticsRUNSTATS on every table you syncOLake Go reads the row count and page count from SYSCAT.TABLES to split the snapshot into chunks. A table that has never had statistics collected stops the sync with stats not populated. Stale statistics are fine.
A cursor columna column that grows with every change, such as UPDATED_ATIncremental syncs select WHERE cursor > last value. Rows updated without touching the cursor are not picked up.
A primary keyon every tableOLake Go uses it to upsert changed rows into the Iceberg table. Without one, changed rows are appended.
On the DB2 database, as the instance owner or a DBADM user
-- 1. A user for OLake Go. Create the operating system account first,
--    for example: useradd -M olake && passwd olake
GRANT CONNECT ON DATABASE TO USER olake;

-- 2. Read access to the tables you sync
GRANT SELECT ON TABLE db2inst1.customers TO USER olake;
GRANT SELECT ON TABLE db2inst1.orders TO USER olake;

-- 3. Statistics on every table you sync
CALL SYSPROC.ADMIN_CMD('RUNSTATS ON TABLE db2inst1.customers AND INDEXES ALL');
CALL SYSPROC.ADMIN_CMD('RUNSTATS ON TABLE db2inst1.orders AND INDEXES ALL');

-- 4. A cursor column that your application always sets, for example
--    updated_at TIMESTAMP DEFAULT CURRENT TIMESTAMP, updated on every change

Check the result before moving on. A CARD of -1 means statistics were never collected:

sql
SELECT tabname, card, npages FROM syscat.tables WHERE tabschema = 'DB2INST1';

Version, driver and SSL details are on the DB2 connector page.

The DB2 connector image is built for amd64 only

IBM's Data Server driver has no Linux arm64 build, so olakego/source-db2 runs on amd64 hosts. On an Apple silicon Mac, turn on Rosetta in Docker Desktop and pull the image with --platform linux/amd64.

Deletes do not replicate in incremental mode

A deleted row has no cursor value to read, so it stays in the Iceberg table. If deletes matter, soft-delete in DB2 (a deleted_at column that also bumps updated_at) or schedule an occasional full refresh.

Step 2: Set up the BigLake catalog​

BigQuery reads Iceberg tables through a catalog. BigLake is the one this page uses: Google Cloud's Iceberg REST catalog, with one namespace per job and the files in your Cloud Storage bucket.

DuckDB and Spark read from the same catalog. To BigQuery it is a data source in the project, so a table OLake Go commits is queryable on the next statement.

CatalogOLake Go writesBigQuery readsVerified on this page
BigLake (Google Cloud Lakehouse)YesYes: native: namespaces map to BigQuery datasetsYes, end to end

BigLake is the Iceberg REST catalog built into Google Cloud's Lakehouse. OLake Go talks to it at https://biglake.googleapis.com/iceberg/v1/restcatalog, the table files land in a Cloud Storage bucket you own, and BigQuery reads the tables without any registration step. Writing to BigLake needs OLake Go 0.9.1 or later.

Everything below happens in the Google Cloud console, in the project that will hold the tables.

  1. Enable the API. Open APIs and services, then Enable APIs and services, and enable the BigLake API. BigQuery and Cloud Storage are on by default in most projects.
  2. Create a bucket. In Cloud Storage, create a bucket in the region where BigQuery will run the queries (us-central1 on this page). The bucket name becomes the catalog name in BigQuery, so pick one you are happy to type in SQL.
  3. Create a service account for OLake Go. In IAM and Admin, Service Accounts, create one (for example olake-biglake). Then grant it two roles: BigLake Editor on the project (IAM, Grant access, the service account as principal) and Storage Object User on the bucket (the bucket's Permissions tab).
  4. Download a JSON key. On the service account's Keys tab, Add key, Create new key, JSON. The file downloads once; OLake Go asks for its contents.
Google Cloud console IAM Grant access panel with the OLake service account as principal and the BigLake Editor role selected
The project-level grant: the service account OLake Go runs as, with the BigLake Editor role.

There is no catalog to create. BigLake creates a single-bucket catalog named after the bucket the first time OLake Go calls the endpoint with gs://<bucket> as the catalog path, in the default end-user credential mode, where every client brings its own storage credentials. The catalog shows up under BigLake, Catalogs after the first connection test.

Two things to know before the first sync:

  • The service account's BigLake Editor role is what lets OLake Go create namespaces and commit tables. Storage Object User on the bucket is what lets it write the Parquet and metadata files. Missing either one fails the connection test with a 403 from biglake.googleapis.com or from storage.googleapis.com, which tells you which grant is missing.
  • Leave the catalog in end-user mode. OLake Go 1.1.0 does not send the X-Iceberg-Access-Delegation header, so a catalog switched to vended credentials rejects its commits (see troubleshooting). Readers that want short-lived tokens from the catalog can switch the mode later with gcloud beta biglake iceberg catalogs update <bucket> --credential-mode=vended-credentials, after setting rest_access_delegation in a CLI config.
Google Cloud console Create private key dialog for the OLake service account with the JSON key type selected
The JSON key for the service account. The console downloads it once; its contents go into the OLake Go destination form.

Step 3: Connect DB2 to OLake Go​

  1. In the OLake UI, open Sources and click Create Source. Pick DB2 LUW as the connector, give the source a name and keep the OLake Go version on the latest release.
  2. Fill in the host, the port (50000 by default), the database name, and the username and password from step 1. The Enable CDC switch stays off: DB2 runs in incremental mode only.
  3. Click Set up source. OLake Go tests the connection and saves the source.

Here is the source form from the run on this page. DB2 ran in another container, so the host is host.docker.internal, with database shop on port 50000. The change data capture (CDC) switch is greyed out with Not supported, which is expected for DB2.

OLake UI Edit source form for DB2 LUW on OLake Go v1.1.0 with CDC marked not supported, host host.docker.internal, port 50000, database shop and user db2inst1
The DB2 source in OLake UI: database shop on port 50000, with CDC not supported for this connector.

The full field reference, including SSL and SSH tunnels, is on the DB2 connector page.

Step 4: Configure the Iceberg destination​

  1. Open Destinations and click Create Destination.
  2. Pick Apache Iceberg as the connector and BigLake as the catalog type.
  3. Fill in the fields below. The catalog path is the bucket, with no folder after it. Paste the whole service account JSON file into the credentials field.
  4. Click Create. OLake Go calls the BigLake endpoint, creates the catalog if it is the first contact, and writes and drops a test table in the bucket.

Here is the destination form from the run on this page. It holds the BigLake endpoint, the bucket as the catalog path, the service account JSON and the project ID.

OLake UI Edit destination form for Apache Iceberg with catalog type Big Lake, REST catalog URI https://biglake.googleapis.com/iceberg/v1/restcatalog, BigLake catalog path gs://olake-x2y-lab-bq, authentication type GCP and the project ID oauth-361809
The BigLake destination in OLake UI: the endpoint, the bucket as catalog path, GCP authentication and the project ID.
FieldValue on this pageWhat it is
REST Catalog URIhttps://biglake.googleapis.com/iceberg/v1/restcatalogThe BigLake Iceberg REST endpoint. The same for every project and region.
BigLake Catalog Pathgs://<bucket>The bucket that holds the tables. The bucket name is also the catalog name BigQuery uses. Bucket root only; a folder after it is not supported.
Authentication TypeGCPGoogle OAuth through the service account. The only choice for BigLake.
GCP Service Account JSONthe key file contentsUsed for both the catalog calls and the Cloud Storage writes.
Catalog Nameolake_icebergA label OLake Go uses in its own logs. Lowercase letters and underscores only.
GCP Auth Scopeshttps://www.googleapis.com/auth/cloud-platformThe OAuth scope on the token. Leave the default.
GCP Project IDthe project IDRoutes and bills the catalog calls. Required for multi-bucket catalogs, harmless to set for a single bucket.
Enable Arrow WritesoffKeeps the default Java writer, which is what this page was tested with.

Step 5: Create the job and run the first sync​

This job lists BigQuery, Spark and DuckDB as target query engines. OLake Go then offers only the delete formats all three read, and the run on this page used Positional Deletes. Because the DB2 connector has no CDC mode, each stream runs as Full Refresh + Incremental with UPDATED_AT as the cursor.

  1. Open Jobs and click Create Job. Give it a name and a schedule. Every Day at a fixed hour, or a cron expression, both work; the schedule is how often the change stream is drained, not how often a full copy runs.
  2. Select the source and the destination you created.
  3. Under Advanced Settings, set Target Query Engines to Google BigQuery, Apache Spark and DuckDB. OLake Go then resolves an Iceberg delete format every one of those engines can read, so you never pick one by hand.
  4. Click Next. OLake Go discovers the tables and opens the Streams page.
  5. Tick the tables to replicate. For each one, set the sync mode to Full Refresh + Incremental with UPDATED_AT as the cursor column and the ingestion mode to Upsert. Leave the Upsert Type as OLake Go resolved it (Positional Deletes in the run on this page).
  6. Click Create Job, then open the job's menu and choose Sync Now for the first run.

DB2 reports unquoted names in upper case, so the streams and columns appear that way.

OLake UI Streams Selection page with the DB2INST1 schema, CUSTOMERS and ORDERS selected, sync mode Full Refresh + Incremental, CDC options disabled, cursor field UPDATED_AT, ingestion mode Upsert and upsert type Positional Deletes on Iceberg format version V2
Each stream: Full Refresh + Incremental on UPDATED_AT, Upsert, and Positional Deletes as the upsert type.
OLake UI Job Logs and History for db2_to_bigquery showing two completed syncs, the first in 40 seconds and the second in 39 seconds
Job history: the 40 s historical snapshot, then the 39 s incremental sync.

On this page the first run copied 8 customers and 40 orders (48 records, 2.38 KB) and finished in 40 seconds, container start included. OLake Go names the BigLake namespace after the job, the database and the DB2 schema, lower-cased: db2_to_bigquery on SHOP.DB2INST1 produced db2_to_bigquery_shop_db2inst1. The Iceberg DB field at the top of the Streams page changes it before the first sync.

Step 6: Query the tables in BigQuery​

There is no registration step and no external table to declare. A BigLake catalog is a BigQuery data source in the same project: the catalog is the bucket name, the namespace OLake Go created is the dataset, and the tables are the tables. Address them with the four-part name in backticks, and run the query in the bucket's region.

BigQuery SQL editor (location: the bucket region)
SELECT id, name, city, tier
FROM `<project-id>.<bucket>.db2_to_bigquery_shop_db2inst1.customers`
ORDER BY id;

SELECT status, count(*) AS n, round(sum(amount), 2) AS total
FROM `<project-id>.<bucket>.db2_to_bigquery_shop_db2inst1.orders`
GROUP BY status ORDER BY status;

From a terminal, bq query --project_id=<project-id> --location=<region> --use_legacy_sql=false "<sql>" runs the same statements. Set the location to the bucket's region (us-central1 on this page); the default US multi-region returns Not found: Dataset for a catalog whose bucket is regional.

OLake Go adds four columns to every table: _olake_id (a hash of the primary key), _op_type, _olake_timestamp and _cdc_timestamp. _op_type is r for a row from the first snapshot, c for an insert, u for an update and d for a delete. Filter them out of dashboards, or use them to see when a row last changed.

BigQuery reads Iceberg v2 tables with positional and equality deletes, which is why the job on this page uses positional deletes. Deletion vectors on v3 tables are a preview feature in BigQuery with per-query limits, so keep the job off deletion vectors while BigQuery is a reader.

BigQuery SQL editor in the Google Cloud console running the customers query on oauth-361809.olake-x2y-lab-bq.db2_to_bigquery_shop_db2inst1.customers and returning 10 rows with id, name, city and tier
BigQuery reading the customers table through the BigLake catalog after the incremental sync: 10 rows, one per key.

Step 7: Change rows in DB2 and sync again​

To see the incremental path work, the run on this page made 12 changes in DB2 after the first sync. Every update also sets updated_at, which is what the cursor sees:

Changes made in DB2 between the two syncs
INSERT INTO customers (name, email, city, tier) VALUES ('Maya Patel', 'maya@example.com', 'Pune', 'silver');
INSERT INTO customers (name, email, city, tier) VALUES ('Lucas Silva', 'lucas@example.com', 'São Paulo', 'bronze');
INSERT INTO orders (customer_id, product, amount, status) VALUES (9, 'Laptop', 1299.00, 'placed');
INSERT INTO orders (customer_id, product, amount, status) VALUES (9, 'Mouse', 25.50, 'placed');
INSERT INTO orders (customer_id, product, amount, status) VALUES (10, 'Monitor', 349.99, 'placed');
INSERT INTO orders (customer_id, product, amount, status) VALUES (3, 'Keyboard', 89.00, 'placed');
INSERT INTO orders (customer_id, product, amount, status) VALUES (7, 'Webcam', 79.90, 'placed');
UPDATE customers SET tier = 'gold',      updated_at = CURRENT TIMESTAMP WHERE name = 'Liam Carter';
UPDATE customers SET city = 'Abu Dhabi', updated_at = CURRENT TIMESTAMP WHERE name = 'Zara Ali';
UPDATE orders    SET status = 'delivered', updated_at = CURRENT TIMESTAMP WHERE id IN (1, 4, 7);

The second sync fetched 12 rows (635 bytes) above the saved cursor and finished in 39 seconds end to end. For customers that meant one new data file with 4 rows and one positional delete file retiring the 2 old versions. OLake Go's commit report in the sync log shows addedDataFiles=1, addedRecords=4, addedPositionalDeleteFiles=1, addedPositionalDeletes=2, and orders the same shape with 8 rows and 3 deletes.

OLake UI sync log filtered to the Iceberg commit reports for customers and orders, showing addedDataFiles 1, addedPositionalDeleteFiles 1, addedRecords 4 and 8, and addedPositionalDeletes 2 and 3
The incremental sync in the OLake UI log: one data file and one positional delete file per table.

BigQuery shows the result on the next query, with no refresh and no ALTER TABLE. Liam Carter is gold, Zara Ali moved to Abu Dhabi, and the two new customers are there. In incremental mode OLake Go cannot tell an insert from an update, so the 4 rows the second sync brought carry _op_type = u and the snapshot rows keep r.

Schema changes. When a column is added in DB2, OLake Go adds it to the Iceberg table on the next sync and commits the new schema to BigLake. BigQuery shows the column on the next query.

DB2 itself may put the table in REORG PENDING after an ALTER TABLE; see troubleshooting below. Full rules are on the schema evolution page.

Query the same tables from DuckDB and Spark​

Nothing is exported, copied or refreshed: both engines read the files OLake Go wrote, through the same BigLake catalog.

DuckDB attaches BigLake as an Iceberg REST catalog with a Google access token. The catalog on this page is in end-user credential mode, so DuckDB also needs its own Cloud Storage credentials to read the files: an HMAC key for the service account (Cloud Storage, Settings, Interoperability, Create a key for a service account).

DuckDB 1.5.5
INSTALL iceberg; LOAD iceberg;
INSTALL httpfs;  LOAD httpfs;

-- gcloud auth print-access-token, or a token minted for the service account
CREATE SECRET biglake_catalog (
TYPE iceberg,
TOKEN '<access token>',
EXTRA_HTTP_HEADERS MAP {'x-goog-user-project': '<project-id>'}
);
-- the HMAC key pair for the bucket
CREATE SECRET biglake_files (TYPE gcs, KEY_ID '<hmac access id>', SECRET '<hmac secret>');

ATTACH 'gs://<bucket>' AS biglake (
TYPE iceberg,
ENDPOINT 'https://biglake.googleapis.com/iceberg/v1/restcatalog',
SECRET biglake_catalog
);

SHOW ALL TABLES;
SELECT id, name, city, tier FROM biglake.db2_to_bigquery_shop_db2inst1.customers ORDER BY id;

The access token lives an hour; CREATE OR REPLACE SECRET with a fresh one when a query fails with 401. If the catalog is in vended-credentials mode instead, drop the gcs secret and add ACCESS_DELEGATION_MODE 'vended_credentials' to the ATTACH: BigLake then hands DuckDB a short-lived storage token per table. DuckDB reads positional deletes, so the rows match BigQuery after every sync.

Google Cloud console Cloud Storage Interoperability settings with the Select a service account dialog, the OLake service account selected and the Create key button
The key for DuckDB: Cloud Storage, Settings, Interoperability, then a key for the same service account OLake Go uses.

DuckDB 1.5.5 returned the same 10 customers and the same orders roll-up as BigQuery in the run on this page. Its types were INTEGER for id, DOUBLE for amount and TIMESTAMP WITH TIME ZONE for the timestamps. The data files came through the hash-based message authentication code (HMAC) key.

An orders roll-up matched DB2 to the cent in all three engines. It gave 16 delivered orders for 8659.68, 18 placed for 10522.26, 11 shipped for 6397.25.

Data types: DB2 to Iceberg to BigQuery​

OLake Go maps DB2 types to Iceberg types, and BigQuery maps Iceberg types to its own when it reads the catalog. Decimals and times are the ones to watch.

DB2Iceberg (written by OLake Go)BigQuery showsNotes
SMALLINT, INTEGERintINT64The identity keys on this page arrived as INT64.
BIGINTlongINT64
DECIMAL, NUMERIC, DECFLOAT, FLOAT, DOUBLEdoubleFLOAT64The orders.amount sums on this page matched DB2 to the cent; round at query time when you need exact decimal arithmetic.
REALfloatFLOAT64
BOOLEANbooleanBOOL
DATE, TIMESTAMPtimestamptz (UTC)TIMESTAMPOLake Go writes every timestamp in UTC, whatever the server time zone.
TIMEstringSTRING
CHAR, VARCHAR, CLOB, GRAPHIC, VARGRAPHIC, DBCLOB, XMLstringSTRING
BLOBstringSTRINGBinary values arrive as text; keep large binaries out of the sync.

DB2 types in the table come from the DB2 connector reference. The BigQuery side comes from the BigQuery Iceberg tables page (checked 2026-10-30) and from the table schemas of this run.

Cost and maintenance​

  • Storage is your bucket: you pay Cloud Storage rates for the Parquet and delete files. The BigLake catalog itself has no per-table charge.
  • Compute. OLake Go runs wherever you run Docker, on an amd64 host for the DB2 connector. BigQuery bills on-demand queries by bytes scanned, or by slot time on an edition; a query over the catalog reads the Parquet files in the bucket.
  • DB2 load. Each incremental sync is one range scan per table on the cursor column; index UPDATED_AT so it stays cheap as the table grows. Keep RUNSTATS in your regular maintenance window so snapshot chunks stay even.
  • Maintenance. BigLake keeps every snapshot OLake Go commits in the table's metadata.json and rejects one above 1 MB, which a busy table reaches after a few thousand syncs. Expire old snapshots and compact on a schedule with OLake Fusion or with Spark's expire_snapshots and rewrite_data_files procedures through the same catalog.
  • Key rotation. The service account key is what OLake Go writes with. Rotate it from the service account's Keys tab and paste the new JSON into the destination.

Troubleshooting​

1. The source test fails on an Apple silicon Mac​

image pull registry-1.docker.io/olakego/source-db2:v1.1.0: Error response from daemon: no matching manifest for linux/arm64/v8 in the manifest list entries

The OLake UI worker pulls connector images for the host's platform, and the DB2 image exists for amd64 only. Turn on Use Rosetta for x86_64/amd64 emulation in Docker Desktop, then pull the image yourself and tag it with the name the worker asks for:

bash
docker pull --platform linux/amd64 olakego/source-db2:v1.1.0
docker tag olakego/source-db2:v1.1.0 registry-1.docker.io/olakego/source-db2:v1.1.0

On an amd64 Linux host, such as a Compute Engine VM, the error does not occur.

2. The first sync stops with "stats not populated"​

stats not populated for table[DB2INST1.CUSTOMERS]. Please run CLP command: RUNSTATS ON TABLE DB2INST1.CUSTOMERS AND INDEXES ALL

DB2 has never collected statistics for the table, and OLake Go needs them to size the snapshot chunks. Run the RUNSTATS call from step 1 for every table in the job, then sync again.

3. A sync fails with SQL0668N after an ALTER TABLE​

Adding, dropping or retyping a column can leave a DB2 table in REORG PENDING, and reads fail until it is reorganized. Run CALL SYSPROC.ADMIN_CMD('REORG TABLE db2inst1.customers'), then sync again. The DB2 connector page lists the operations that cause it.

4. The CLI picked the wrong cursor column​

discover fills cursor_field in selected_streams.json with the first column in alphabetical order, for example CITY for CUSTOMERS. Set it to UPDATED_AT for every stream before the first sync; the OLake UI asks for it on the Streams page instead.

5. BigQuery cannot find the dataset​

Not found: Dataset <project>:<bucket>.db2_to_bigquery_shop_db2inst1 was not found in location US

The query ran in a location other than the bucket's region. Set the query location to the bucket's region in the editor's More, Query settings panel, or pass --location=<region> to bq.

How this compares with other ways to move DB2 data to BigQuery
MethodFreshnessUpdates and deletesWhere the data landsWhat you run
Debezium's Db2 connector, Kafka and a BigQuery sinkSeconds, from SQL Replication capture tablesYesKafka topics, then BigQuery tablesKafka, Kafka Connect, Debezium, tables in capture mode and, per Debezium's docs, an IBM InfoSphere Data Replication license (checked Oct 2026)
Export (EXPORT TO ... OF DEL), then bq loadHours to dailyNo; full reloads onlyNative BigQuery tablesA script, a scheduler and a staging bucket
OLake Go to Iceberg on BigLake (this guide)Incremental on a schedule down to one minuteInserts and updates that move the cursor; deletes need a soft-delete columnApache Iceberg tables that BigQuery, Spark, DuckDB, Trino and Flink all readOne Docker Compose stack, open source; your bucket plus your own compute

Log-based replication is the choice when you need hard deletes and already hold the license. Writing Iceberg once avoids paying for a second copy as soon as Spark, DuckDB or a notebook needs the same rows.

When not to use this​

  • You need hard deletes replicated. Incremental sync cannot see a deleted row. Soft-delete in DB2, or use a log-based tool.
  • Your DB2 runs on z/OS or IBM i. The connector covers DB2 LUW only.
  • Every query must hit native BigQuery speed. A catalog table is read from Parquet in the bucket on every query. Keep the Iceberg tables as the source and load the hot slice into a native table on a schedule.

FAQs​

How do I replicate IBM DB2 to BigQuery?

Run OLake Go (open source, Apache 2.0) against DB2 LUW with a read-only user, and point it at a BigLake catalog with a Cloud Storage bucket. The first sync copies each table in parallel chunks; every later sync fetches only rows whose cursor column (for example UPDATED_AT) moved, and upserts them. BigQuery queries the tables through the catalog with a four-part name. No replication license, Kafka or load job is involved.

Does OLake Go do change data capture (CDC) from DB2?

No. The DB2 connector runs Full Refresh and Full Refresh + Incremental; it does not read the DB2 log or the SQL Replication capture tables. Incremental syncs select rows where a cursor column is greater than the last value seen, which captures inserts and updates that bump the cursor but not deletes.

Does BigQuery read Iceberg tables with row-level deletes?

Yes. BigQuery reads Iceberg v2 merge-on-read tables with positional and equality deletes. The run on this page used positional deletes and BigQuery returned one row per key after the updates.

Do I need to create a BigQuery dataset or an external table?

No. A BigLake catalog is a data source BigQuery reads directly: the bucket name is the catalog, the namespace OLake Go creates is the dataset, and the tables appear under it. The query uses the four-part name project.bucket.namespace.table, and there is no CREATE EXTERNAL TABLE and no metadata refresh.

Why does the first sync ask me to run RUNSTATS?

OLake Go splits each table into chunks for the parallel snapshot using the row and page counts DB2 keeps in SYSCAT.TABLES. Those counts exist only after RUNSTATS has run once on the table. Run CALL SYSPROC.ADMIN_CMD('RUNSTATS ON TABLE schema.table AND INDEXES ALL') per table; statistics that are out of date do not stop a sync.

Which DB2 data types change on the way to BigQuery?

DECIMAL, NUMERIC, DECFLOAT, FLOAT and DOUBLE become Iceberg double and show up in BigQuery as FLOAT64, so round at query time when you need exact decimal arithmetic. SMALLINT, INTEGER and BIGINT become INT64. DATE and TIMESTAMP become TIMESTAMP in UTC. TIME, XML, CLOB, BLOB and GRAPHIC types arrive as STRING.

DB2 to SnowflakeSame source, read from Snowflake.DB2 to ClickHouseSame source, read from ClickHouse.DB2 to Amazon RedshiftSame source, read from Amazon Redshift.Postgres to BigQuerySame engine, replicated from Postgres.Oracle to BigQuerySame engine, replicated from Oracle.MongoDB to BigQuerySame engine, replicated from MongoDB.MySQL to BigQuerySame engine, replicated from MySQL.Kafka to BigQuerySame engine, replicated from Kafka.MariaDB to BigQuerySame engine, replicated from MariaDB.S3 to BigQuerySame engine, replicated from S3.SQL Server to BigQuerySame engine, replicated from SQL Server.DB2 connector referenceEvery field, sync mode and troubleshooting note for the source.Query engine compatibilityWhich engines read which catalogs and delete formats.All integrationsEvery source and engine pair with a tested guide.

Deeper reading on this site: