DB2 to ClickHouse: Incremental Sync via Iceberg

Getting IBM DB2 LUW (Linux, UNIX and Windows) data into ClickHouse usually means a CSV export on a schedule or a licensed replication product feeding Kafka. This guide takes a different route.

OLake Go reads the DB2 tables with a read-only user and commits them as Apache Iceberg tables into an Apache Polaris catalog. ClickHouse then attaches that catalog as a database.

Polaris is an open source Iceberg REST (Representational State Transfer) catalog you run yourself, with the files in your own bucket. DuckDB and Trino read the same tables through the same endpoint, and a cursor column keeps them fresh.

By Shubham Satish Baldava · Updated October 30, 2026

Key takeaways

• OLake Go replicates DB2 LUW into Iceberg tables in Polaris: a full historical snapshot first, then incremental syncs on a cursor column such as UPDATED_AT, with no Kafka or replication license in between.
• ClickHouse attaches the whole Polaris catalog as one database with DataLakeCatalog, reads the files with the credentials Polaris vends, and sees each sync the moment it commits.
• The same tables, written once by OLake Go, are also queryable from DuckDB and Trino through the same catalog. In the run on this page all three engines returned identical rows.

The short answer. To replicate DB2 to ClickHouse, run OLake Go with a read-only DB2 user as the source and an Apache Polaris catalog as the Iceberg destination. The first sync copies each table in full (the historical snapshot); every later sync fetches the rows whose cursor column moved and upserts them. ClickHouse, DuckDB and Trino then query the same Iceberg tables, with no second copy.

How DB2 data reaches ClickHouse here

The first sync reads the row and page counts DB2 keeps for each table, splits the table into chunks, and reads the chunks in parallel. It writes Parquet data files and records the highest cursor value it saw. Every later sync selects rows above that value, writes them as a new data file, and adds an equality delete file that retires the old version of each key.

Each sync ends with one commit to Polaris. OLake Go asks Polaris for short-lived storage credentials, uploads the data and delete files, then posts the new snapshot. ClickHouse, DuckDB and Trino see it on their next query; none of them keeps a copy.

The cursor is the whole contract. A row that changes without its UPDATED_AT moving is never picked up, and a deleted row stays in the Iceberg table. Set the column in every write path, and soft-delete if deletes matter.

What you need

• DB2 LUW 11.5.3 or later, with a read-only user, statistics on each table and a cursor column. The tables on this page ran on DB2 12.1.5 Community Edition in Docker.
• An amd64 host for the DB2 connector. IBM's driver has no Linux arm64 build. On an Apple silicon Mac, turn on Rosetta in Docker Desktop.
• An Apache Polaris server and a bucket, both reachable from OLake Go and from ClickHouse. The run on this page used Polaris 1.8.0 and MinIO in Docker; on AWS, use an S3 bucket and an IAM role instead.
• ClickHouse 26.1 or later, where the DataLakeCatalog engine reads REST catalogs. The run on this page used the 26.8 LTS release in Docker.
• OLake UI from the Docker Compose install. The run on this page used OLake Go v1.1.0.

Step 1: Prepare DB2 for incremental sync

Most readers point OLake Go at tables they already have. To follow along with the exact data from this page, create these two tables first:

The two sample tables used on this page

Sample schema, in schema DB2INST1 of database SHOP

CREATE TABLE customers (
  id INT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
  name VARCHAR(100), email VARCHAR(200), city VARCHAR(100), tier VARCHAR(20),
  created_at TIMESTAMP DEFAULT CURRENT TIMESTAMP,
  updated_at TIMESTAMP DEFAULT CURRENT TIMESTAMP);

CREATE TABLE orders (
  id INT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
  customer_id INT REFERENCES customers(id),
  product VARCHAR(100), amount DECIMAL(10,2), status VARCHAR(20),
  created_at TIMESTAMP DEFAULT CURRENT TIMESTAMP,
  updated_at TIMESTAMP DEFAULT CURRENT TIMESTAMP);

-- Then insert whatever you like. This page started with 8 customers and 40 orders.

OLake Go reads DB2 LUW (Linux, UNIX and Windows) tables through IBM's Data Server driver: a parallel full snapshot first, then incremental syncs that fetch only rows whose cursor column moved past the last value seen. The DB2 connector has no log-based change data capture (CDC), so four things must be in place before the first sync.

Setting	Value	Why it matters
A read-only user	CONNECT on the database plus SELECT on each table	The snapshot and every incremental sync read the tables directly. DB2 authenticates users against the operating system or LDAP, so the account exists there first and is then granted in the database.
Table statistics	RUNSTATS on every table you sync	OLake Go reads the row count and page count from SYSCAT.TABLES to split the snapshot into chunks. A table that has never had statistics collected stops the sync with stats not populated. Stale statistics are fine.
A cursor column	a column that grows with every change, such as UPDATED_AT	Incremental syncs select WHERE cursor > last value. Rows updated without touching the cursor are not picked up.
A primary key	on every table	OLake Go uses it to upsert changed rows into the Iceberg table. Without one, changed rows are appended.

On the DB2 database, as the instance owner or a DBADM user

-- 1. A user for OLake Go. Create the operating system account first,
--    for example: useradd -M olake && passwd olake
GRANT CONNECT ON DATABASE TO USER olake;

-- 2. Read access to the tables you sync
GRANT SELECT ON TABLE db2inst1.customers TO USER olake;
GRANT SELECT ON TABLE db2inst1.orders TO USER olake;

-- 3. Statistics on every table you sync
CALL SYSPROC.ADMIN_CMD('RUNSTATS ON TABLE db2inst1.customers AND INDEXES ALL');
CALL SYSPROC.ADMIN_CMD('RUNSTATS ON TABLE db2inst1.orders AND INDEXES ALL');

-- 4. A cursor column that your application always sets, for example
--    updated_at TIMESTAMP DEFAULT CURRENT TIMESTAMP, updated on every change

Check the result before moving on. A CARD of -1 means statistics were never collected:

sql

SELECT tabname, card, npages FROM syscat.tables WHERE tabschema = 'DB2INST1';

Version, driver and SSL details are on the DB2 connector page.

The DB2 connector image is built for amd64 only

IBM's Data Server driver has no Linux arm64 build, so olakego/source-db2 runs on amd64 hosts. On an Apple silicon Mac, turn on Rosetta in Docker Desktop and pull the image with --platform linux/amd64.

Deletes do not replicate in incremental mode

A deleted row has no cursor value to read, so it stays in the Iceberg table. If deletes matter, soft-delete in DB2 (a deleted_at column that also bumps updated_at) or schedule an occasional full refresh.

Step 2: Set up the Polaris catalog

ClickHouse reads Iceberg tables through a catalog. Apache Polaris is the one this page uses: an open source REST catalog that you run next to your bucket.

OLake Go writes into it, and ClickHouse, DuckDB and Trino read from it with the credentials Polaris hands out. The Iceberg files sit in your own S3 or MinIO bucket; Polaris holds only the pointers and the permissions.

Catalog	OLake Go writes	ClickHouse reads	Verified on this page
Apache Polaris	Yes	Yes: DataLakeCatalog with catalog_type rest	Yes, end to end

Apache Polaris

Apache Polaris is an open source Iceberg REST catalog. It keeps the table metadata pointers, hands out short-lived storage credentials to engines, and applies its own role-based access control. OLake Go writes to it through the standard Iceberg REST API, and ClickHouse, Trino, DuckDB and Spark read from the same endpoint. The files live in an S3 bucket or an S3-compatible store that you own.

Three things need to exist before OLake Go can write: a Polaris server, a catalog inside it that points at your bucket, and a principal (a client id and secret) with the right to manage that catalog. The run on this page used Polaris 1.8.0 and MinIO in Docker; on AWS, swap the MinIO endpoint for a plain S3 bucket and an IAM role.

docker-compose.yml (Polaris and MinIO, for a local test)

services:
  minio:
    image: minio/minio
    hostname: lake-minio
    command: server /data --console-address ":9001"
    environment: { MINIO_ROOT_USER: minioadmin, MINIO_ROOT_PASSWORD: minioadmin }
    ports: ["9000:9000", "9001:9001"]
  polaris:
    image: apache/polaris:latest
    depends_on: [minio]
    ports: ["8181:8181", "8182:8182"]
    environment:
      AWS_REGION: us-east-1
      AWS_ACCESS_KEY_ID: minioadmin
      AWS_SECRET_ACCESS_KEY: minioadmin
      POLARIS_BOOTSTRAP_CREDENTIALS: POLARIS,root,<root-secret>
      polaris.realm-context.realms: POLARIS

Polaris stores its own state in memory with this setup, so it starts empty every time; a production deployment points it at Postgres. Once it is up, create the bucket, the catalog and the principal through the management API. root is the bootstrap principal; the olake principal is the one OLake Go and the engines use.

Polaris, once

P=http://localhost:8181

# 1. A bucket for the tables (MinIO; on AWS create it in the console or with aws s3 mb)
aws --endpoint-url http://localhost:9000 s3 mb s3://lakehouse

# 2. A root token
TOKEN=$(curl -s $P/api/catalog/v1/oauth/tokens -H 'Polaris-Realm: POLARIS' \
  -d grant_type=client_credentials -d client_id=root -d client_secret=<root-secret> \
  -d scope=PRINCIPAL_ROLE:ALL | jq -r .access_token)
H=(-H "Authorization: Bearer $TOKEN" -H 'Polaris-Realm: POLARIS' -H 'Content-Type: application/json')

# 3. The catalog: where the files go, and how Polaris and the engines reach the store
curl -s "${H[@]}" $P/api/management/v1/catalogs -d '{"catalog": {
  "name": "olake", "type": "INTERNAL", "readOnly": false,
  "properties": {"default-base-location": "s3://lakehouse/olake"},
  "storageConfigInfo": {"storageType": "S3", "allowedLocations": ["s3://lakehouse/olake"],
    "endpoint": "http://<minio-host>:9000/", "endpointInternal": "http://lake-minio:9000",
    "pathStyleAccess": true, "region": "us-east-1"}}}'

# 4. A principal for OLake Go and the engines. The response holds clientId and clientSecret: keep them.
curl -s "${H[@]}" $P/api/management/v1/principals -d '{"principal": {"name": "olake"}}'

# 5. Roles and the grant that lets the principal create namespaces and tables in the catalog
curl -s "${H[@]}" $P/api/management/v1/principal-roles -d '{"principalRole": {"name": "olake_role"}}'
curl -s "${H[@]}" $P/api/management/v1/catalogs/olake/catalog-roles -d '{"catalogRole": {"name": "olake_catalog_role"}}'
curl -s -X PUT "${H[@]}" $P/api/management/v1/principals/olake/principal-roles -d '{"principalRole": {"name": "olake_role"}}'
curl -s -X PUT "${H[@]}" $P/api/management/v1/principal-roles/olake_role/catalog-roles/olake -d '{"catalogRole": {"name": "olake_catalog_role"}}'
curl -s -X PUT "${H[@]}" $P/api/management/v1/catalogs/olake/catalog-roles/olake_catalog_role/grants -d '{"type": "catalog", "privilege": "CATALOG_MANAGE_CONTENT"}'

What each block is for:

• endpoint is the store address Polaris hands to engines along with vended credentials; endpointInternal is the address Polaris itself uses. On AWS S3 leave both out. On MinIO both are required, or Polaris tries to reach AWS and fails. Keep the trailing slash on endpoint: ClickHouse 26.8 and 26.10 build the object path from it, and without the slash every read fails with a 404 (the fix is explained under troubleshooting on this page).
• Polaris vends credentials through the store's STS endpoint. MinIO exposes one, so nothing else is needed. For a store without STS, add "stsUnavailable": true and give Polaris static keys to hand out.
• CATALOG_MANAGE_CONTENT lets the principal create namespaces and tables and read and write their data. Scope it to a namespace instead when several teams share the catalog.
• The principal's secret is shown once, in the response to step 4. It is what goes into OLake Go as <client_id>:<client_secret>.

Hostnames with underscores break Polaris

Polaris is a Java service, and Java's URI parser rejects hostnames that contain an underscore. A MinIO container named lab_minio makes every table creation fail with host must not be null or Expected scheme-specific part at index 5: http:. Use a hyphen in the hostname (lake-minio), as above.

You can check the principal outside OLake Go by exchanging its credentials for a token, the same call OLake Go makes on every sync:

Token check

curl -s http://localhost:8181/api/catalog/v1/oauth/tokens -H 'Polaris-Realm: POLARIS' \
  -d grant_type=client_credentials -d client_id=<client_id> -d client_secret=<client_secret> \
  -d scope=PRINCIPAL_ROLE:ALL

A good principal returns a JSON object with an access_token. Running Polaris on Kubernetes, with Postgres behind it, or with an external identity provider is covered on the Polaris catalog page.

Step 3: Connect DB2 to OLake Go

OLake UI

1. In the OLake UI, open Sources and click Create Source. Pick DB2 LUW as the connector, give the source a name and keep the OLake Go version on the latest release.
2. Fill in the host, the port (50000 by default), the database name, and the username and password from step 1. The Enable CDC switch stays off: DB2 runs in incremental mode only.
3. Click Set up source. OLake Go tests the connection and saves the source.

Here is the source form from the run on this page. DB2 ran in another container, so the host is host.docker.internal, with database shop on port 50000. The change data capture (CDC) switch is greyed out with Not supported, which is expected for DB2.

The DB2 source in OLake UI: database shop on port 50000, with CDC not supported for this connector.

The full field reference, including SSL and SSH tunnels, is on the DB2 connector page.

OLake CLI

Create source.json in your config folder:

source.json

{
    "host": "DB2 LUW-host",
    "port": 50000,
    "database": "DB2 LUW-DBName",
    "username": "DB2 LUW-Username",
    "password": "DB2 LUW-Password",
    "ssl": {
        "mode": "disable"
    },
    "max_threads": 30,
    "jdbc_url_params": {},
    "retry_count": 3
}

Then confirm OLake Go can reach the database. --platform linux/amd64 lets the amd64-only image run on an Apple silicon Mac and changes nothing on an amd64 host:

check command

docker run --pull=always \
  -v "$PWD/config:/mnt/config" \
  --platform linux/amd64 \
  olakego/source-db2:latest \
  check --config /mnt/config/source.json

A working connection returns "status": "SUCCEEDED". Every field is described on the DB2 connector page.

Step 4: Configure the Iceberg destination

Apache Polaris

OLake UI

1. Open Destinations and click Create Destination.
2. Pick Apache Iceberg as the connector and Polaris as the catalog type.
3. Choose OAuth2 as the authentication type. REST Catalog URI is the Polaris catalog endpoint (http://<polaris-host>:8181/api/catalog), Polaris Catalog Name the catalog from the setup (olake), REST Auth URI the token endpoint, and REST Credential the principal's <client_id>:<client_secret>.
4. Fill in the store: AWS Region, and for MinIO the S3 Endpoint with the AWS Access Key and AWS Secret Key. On AWS S3 leave the endpoint empty.
5. Click Create. OLake Go exchanges the credentials for a token, creates a namespace and a test table through Polaris, writes to the bucket and drops the test table again.

Here is the destination form from the run on this page. Polaris and MinIO ran in other containers, so both addresses use host.docker.internal; the credential is the olake principal from step 2.

The Polaris destination in OLake UI: OAuth2 with the principal from step 2, and the MinIO endpoint for the files.

Polaris Configuration Parameters

Parameter	Sample Value	Description
REST Catalog URI required	http://<REST_ENDPOINT>:8181/api/catalog	Endpoint URL for the Polaris REST catalog service.
Authentication Type required	oauth2	Authentication method OLake Go uses for Polaris REST catalog requests. Set to match your catalog's authentication configuration.
Polaris Catalog Name required	<POLARIS_CATALOG_NAME>	Name of the Polaris catalog to use, OLake Go passes this to the Polaris REST catalog as the warehouse identifier, so catalog operations are routed to that catalog and its pre-configured S3 storage and IAM settings.
REST Auth URI required	http://<REST_ENDPOINT>:8181/api/catalog/v1/oauth/tokens	URL of the REST catalog’s OAuth2 token endpoint. OLake Go uses this with REST Credential to request an access token during OAuth2 client-credentials authentication.
REST Credential required	<olake_user_client_id>:<olake_user_client_secret>	Specifies the client ID and secret for OAuth2, formatted as client_id:client_secret. Used with REST Auth URI when OLake Go requests an access token from the catalog’s auth service. Not needed when using REST Token or other auth types.
AWS Region required	ap-south-1	AWS Region where your S3 bucket exists. Required so OLake Go calls the correct regional S3 endpoint.
REST Scope	PRINCIPAL_ROLE:ALL	Required for OAuth2 client-credentials auth. Scopes are space-separated. Used with REST Auth URI and REST Credential for OAuth2 client-credentials auth. Values are catalog-specific. Not needed when using REST Token or other auth types.
S3 Endpoint	http://<S3_ENDPOINT>:9000	S3 API endpoint for writing Iceberg data files. Leave empty for AWS S3 but required for S3-compatible storage like MinIO.
AWS Access Key	XXX	AWS access key ID for authentication. Optional if using IAM roles or instance profiles. If using IAM: Docker Compose: add the required IAM environment variables under x-envs in your compose file. See Service Environment Variables for setup details. Kubernetes: set up pod IAM in Cloud Integration.
AWS Secret Key	XXX	AWS secret access key for authentication. Optional if using IAM roles or instance profiles. If using IAM: Docker Compose: add the required IAM environment variables under x-envs in your compose file. See Service Environment Variables for setup details. Kubernetes: set up pod IAM in Cloud Integration.
Enable Arrow Writes	false	Writes data and delete files using Apache Arrow based writer and registers them in Iceberg.
Catalog Name	olake_iceberg	Name of the Iceberg catalog OLake Go registers tables under. Defaults to olake_iceberg if left empty.

Catalog Name Supported for v0.3.5 and above

For the catalog name, OLake Go only supports lowercase letters and underscores. Spaces and special characters are not supported.

OLake CLI

Create destination.json next to source.json:

destination.json

{
  "type": "ICEBERG",
  "writer": {
    "catalog_type": "polaris",
    "catalog_name": "polaris",
    "rest_catalog_url": "http://<polaris-host>:8181/api/catalog",
    "iceberg_s3_path": "olake",
    "rest_auth_type": "oauth2",
    "oauth2_uri": "http://<polaris-host>:8181/api/catalog/v1/oauth/tokens",
    "credential": "<client_id>:<client_secret>",
    "scope": "PRINCIPAL_ROLE:ALL",
    "aws_region": "us-east-1",
    "s3_endpoint": "http://<minio-host>:9000",
    "aws_access_key": "<minio access key>",
    "aws_secret_key": "<minio secret key>",
    "arrow_writes": false
  }
}

Polaris Configuration Parameters

Parameter	Sample Value	Description
catalog_type required	polaris	Defines the catalog type used by the writer.
rest_catalog_url required	http://<REST_ENDPOINT>:8181/api/catalog	Endpoint URL for the Polaris REST catalog service.
iceberg_s3_path required	<POLARIS_CATALOG_NAME>	Name of the Polaris catalog to use, OLake Go passes this to the Polaris REST catalog as the warehouse identifier, so catalog operations are routed to that catalog and its pre-configured S3 storage and IAM settings.
rest_auth_type required	oauth2	Authentication method OLake Go uses for Polaris REST catalog requests. Set to match your catalog's authentication configuration.
oauth2_uri required	http://<REST_ENDPOINT>:8181/api/catalog/v1/oauth/tokens	URL of the REST catalog’s OAuth2 token endpoint. OLake Go uses this with REST Credential to request an access token during OAuth2 client-credentials authentication.
credential required	<olake_user_client_id>:<olake_user_client_secret>	Specifies the client ID and secret for OAuth2, formatted as client_id:client_secret. Used with REST Auth URI when OLake Go requests an access token from the catalog’s auth service.
aws_region required	ap-south-1	AWS Region where your S3 bucket exists. Required so OLake Go calls the correct regional S3 endpoint.
scope	PRINCIPAL_ROLE:ALL	Required for OAuth2 client-credentials auth. Scopes are space-separated. Used with REST Auth URI and REST Credential for OAuth2 client-credentials auth. Values are catalog-specific.
s3_endpoint	http://<S3_ENDPOINT>:9000	S3 API endpoint for writing Iceberg data files. Leave empty for AWS S3 but required for S3-compatible storage like MinIO.
aws_access_key	XXX	AWS access key ID for authentication. Optional if using IAM roles or instance profiles. If using IAM: Docker Compose: add the required IAM environment variables under x-envs in your compose file. See Service Environment Variables for setup details. Kubernetes: set up pod IAM in Cloud Integration.
aws_secret_key	XXX	AWS secret access key for authentication. Optional if using IAM roles or instance profiles. If using IAM: Docker Compose: add the required IAM environment variables under x-envs in your compose file. See Service Environment Variables for setup details. Kubernetes: set up pod IAM in Cloud Integration.
arrow_writes	false	Writes data and delete files using Apache Arrow based writer and registers them in Iceberg.
catalog_name	olake_iceberg	Name of the Iceberg catalog OLake Go registers tables under. Defaults to olake_iceberg if left empty.

Catalog Name Supported for v0.3.5 and above

For the catalog name, OLake Go only supports lowercase letters and underscores. Spaces and special characters are not supported.

Step 5: Create the job and run the first sync

OLake UI

1. Open Jobs and click Create Job. Give it a name and a schedule. Every Day at a fixed hour, or a cron expression, both work; the schedule is how often the change stream is drained, not how often a full copy runs.
2. Select the source and the destination you created.
3. Under Advanced Settings, set Target Query Engines to ClickHouse, plus DuckDB and Trino if they will read the tables too. OLake Go then resolves an Iceberg delete format every one of those engines can read, so you never pick one by hand.
4. Click Next. OLake Go discovers the tables and opens the Streams page.
5. Tick the tables to replicate. For each one, set the sync mode to Full Refresh + Incremental with UPDATED_AT as the cursor column and the ingestion mode to Upsert. Leave the Upsert Type as OLake Go resolved it (Equality Deletes in the run on this page).
6. Click Create Job, then open the job's menu and choose Sync Now for the first run.

Three target query engines were set on this job, and OLake Go resolved the delete format all three read: equality deletes, which need no index between runs. Because the DB2 connector has no CDC mode, each stream runs as Full Refresh + Incremental with UPDATED_AT as the cursor. DB2 reports unquoted names in upper case, so the streams and columns appear that way.

Each stream: Full Refresh + Incremental on UPDATED_AT, Upsert, and the upsert type OLake Go resolved (Equality Deletes in this run).

Job history: the 14 s historical snapshot, then the 12 s incremental sync.

OLake CLI

Discover the tables, choose the streams, then sync:

discover

docker run --pull=always \
-v "$PWD/config:/mnt/config" \
--platform linux/amd64 \
olakego/source-db2:latest \
discover --config /mnt/config/source.json

discover writes two files: available_streams.json (every table it found, do not edit) and selected_streams.json (the tables you want and their settings). In selected_streams.json, set sync_mode to {props.cliSyncMode ?? 'cdc'} for each table and pick the delete format your readers can open:

selected_streams.json (one selected stream)

{
"selected_streams": {
  "DB2INST1": [
    {
      "stream_name": "CUSTOMERS",
      "sync_mode": "incremental",
      "cursor_field": "UPDATED_AT",
      "normalization": true,
      "append_mode": false,
      "update_type": "eq"
    }
  ]
}
}

update_type accepts eq (equality deletes, the default), pos (positional deletes) and dv (deletion vectors). Positional deletes and deletion vectors keep a small index between runs, so give the container a persistent directory for it through OLAKE_INDEX_DB_DIR. The first sync runs without a state file; OLake Go writes one at the end of the run:

first sync

docker run --pull=always \
-v "$PWD/config:/mnt/config" \
-e OLAKE_INDEX_DB_DIR=/mnt/config/index-db \
--platform linux/amd64 \
olakego/source-db2:latest \
sync \
--config /mnt/config/source.json \
--selected-streams /mnt/config/selected_streams.json \
--available-streams /mnt/config/available_streams.json \
--destination /mnt/config/destination.json

Every later run adds --state /mnt/config/state.json to the same command, which is how OLake Go knows to replay only the changes since the last run. The full selected_streams.json reference is on the Docker CLI page; the index directory is described under index DB setup.

On this page the first run copied 8 customers and 40 orders (48 records, 2.38 KB) and finished in 14 seconds, container start included. OLake Go names the destination namespace after the job, the database and the DB2 schema, lower-cased: db2_to_clickhouse on SHOP.DB2INST1 produced db2_to_clickhouse_shop_db2inst1. The Iceberg DB field at the top of the Streams page changes it before the first sync.

Step 6: Query the tables in ClickHouse

Apache Polaris

There is no registration step. ClickHouse attaches the whole Polaris catalog as one database with the DataLakeCatalog engine, lists every table OLake Go committed, and reads the files with the credentials Polaris vends. One thing to know before the first query: ClickHouse has one level of namespace, so the Iceberg namespace and table name go together inside backticks.

clickhouse-client

SET allow_database_iceberg = 1;

CREATE DATABASE polaris
ENGINE = DataLakeCatalog('http://<polaris-host>:8181/api/catalog')
SETTINGS
catalog_type = 'rest',
warehouse = 'olake',
catalog_credential = '<client_id>:<client_secret>',
oauth_server_uri = 'http://<polaris-host>:8181/api/catalog/v1/oauth/tokens',
auth_scope = 'PRINCIPAL_ROLE:ALL',
vended_credentials = true,
region = 'us-east-1';

SHOW TABLES FROM polaris;

-- What OLake Go created (namespace = <job name>_<source database>_<source schema>)
SELECT id, name, city, tier
FROM polaris.`db2_to_clickhouse_shop_db2inst1.customers`
ORDER BY id;

vended_credentials = true makes ClickHouse read the files with the short-lived keys Polaris hands out per table, so no store keys sit in ClickHouse. To use static keys instead, replace it with storage_endpoint, aws_access_key_id and aws_secret_access_key. The database is read-only: INSERT, ALTER and RENAME DATABASE return NOT_IMPLEMENTED.

OLake Go adds four columns to every table: _olake_id (a hash of the primary key), _op_type, _olake_timestamp and _cdc_timestamp. _op_type is r for a row from the first snapshot, c for an insert, u for an update and d for a delete. Filter them out of dashboards, or use them to see when a row last changed.

ClickHouse 26.8 reading the customers table through Polaris after the incremental sync: 10 rows, one per key.

Try it on your own database

OLake Go is open source. The quickstart brings up the UI with Docker Compose in a few minutes; the Slack community answers setup questions the same day.

QuickstartJoin Slack

Step 7: Change rows in DB2 and sync again

To see the incremental path work, the run on this page made 12 changes in DB2 after the first sync. Every update also sets updated_at, which is what the cursor sees:

Changes made in DB2 between the two syncs

INSERT INTO customers (name, email, city, tier) VALUES ('Maya Patel', 'maya@example.com', 'Pune', 'silver');
INSERT INTO customers (name, email, city, tier) VALUES ('Lucas Silva', 'lucas@example.com', 'São Paulo', 'bronze');
INSERT INTO orders (customer_id, product, amount, status) VALUES (9, 'Laptop', 1299.00, 'placed');
INSERT INTO orders (customer_id, product, amount, status) VALUES (9, 'Mouse', 25.50, 'placed');
INSERT INTO orders (customer_id, product, amount, status) VALUES (10, 'Monitor', 349.99, 'placed');
INSERT INTO orders (customer_id, product, amount, status) VALUES (3, 'Keyboard', 89.00, 'placed');
INSERT INTO orders (customer_id, product, amount, status) VALUES (7, 'Webcam', 79.90, 'placed');
UPDATE customers SET tier = 'gold',      updated_at = CURRENT TIMESTAMP WHERE name = 'Liam Carter';
UPDATE customers SET city = 'Abu Dhabi', updated_at = CURRENT TIMESTAMP WHERE name = 'Zara Ali';
UPDATE orders    SET status = 'delivered', updated_at = CURRENT TIMESTAMP WHERE id IN (1, 4, 7);

The second sync fetched 12 rows (635 bytes) above the saved cursor and finished in 12 seconds end to end. For customers that meant one new data file with 4 rows and one equality delete file with a delete for each of those 4 keys.

The 2 inserts get one too, because an equality delete costs nothing when no older row matches. The commit report in the sync log shows addedDataFiles=1, addedRecords=4, addedEqualityDeleteFiles=1, addedEqualityDeletes=4, and orders the same shape with 8 rows and 8 deletes.

The incremental sync in the OLake UI log: one data file and one equality delete file per table.

ClickHouse shows the result on the next query, with no refresh. Liam Carter is gold, Zara Ali moved to Abu Dhabi, and the two new customers are there; the two retired versions are gone. In incremental mode OLake Go cannot tell an insert from an update, so the 4 rows the second sync brought carry _op_type = u and the snapshot rows keep r.

Schema changes. When a column is added in DB2, OLake Go adds it to the Iceberg table on the next sync, and ClickHouse shows it the next time the table is opened. DB2 itself may put the table in REORG PENDING after an ALTER TABLE; see troubleshooting below. Full rules are on the schema evolution page.

Query the same tables from DuckDB and Trino

Nothing is exported, copied or refreshed: both engines read the files OLake Go wrote, through the same Polaris endpoint.

DuckDB

DuckDB attaches Polaris as an Iceberg REST catalog with the principal's client id and secret; DuckDB fetches the token itself. On a laptop, where the store address Polaris hands out may not resolve, add an S3 secret for the store and turn credential vending off:

DuckDB 1.5.5

INSTALL iceberg; LOAD iceberg;
INSTALL httpfs;  LOAD httpfs;

CREATE SECRET polaris_secret (
TYPE iceberg,
CLIENT_ID '<client_id>', CLIENT_SECRET '<client_secret>',
OAUTH2_SERVER_URI 'http://<polaris-host>:8181/api/catalog/v1/oauth/tokens',
OAUTH2_SCOPE 'PRINCIPAL_ROLE:ALL'
);
-- MinIO, or any store whose address differs from the one Polaris vends
CREATE SECRET store_secret (
TYPE s3, KEY_ID '<access key>', SECRET '<secret key>',
ENDPOINT '<minio-host>:9000', URL_STYLE 'path', USE_SSL false, REGION 'us-east-1'
);

ATTACH 'olake' AS polaris (
TYPE iceberg,
ENDPOINT 'http://<polaris-host>:8181/api/catalog',
SECRET polaris_secret,
ACCESS_DELEGATION_MODE 'none'
);

SHOW ALL TABLES;
SELECT id, name, city, tier FROM polaris.db2_to_clickhouse_shop_db2inst1.customers ORDER BY id;

With Polaris on AWS S3, drop the S3 secret and the ACCESS_DELEGATION_MODE line: DuckDB then reads with the credentials Polaris vends. DuckDB reads equality and positional deletes, and deletion vectors from duckdb-iceberg 1.5.3 onward.

DuckDB 1.5.5 reading the same table through Polaris: the same 10 rows.

Trino

Trino reaches Polaris through its Iceberg connector in REST mode. One catalog properties file does it; Trino fetches the OAuth2 token with the principal's credentials and reads the files with the credentials Polaris vends.

etc/catalog/polaris.properties

connector.name=iceberg
iceberg.catalog.type=rest
iceberg.rest-catalog.uri=http://<polaris-host>:8181/api/catalog
iceberg.rest-catalog.warehouse=olake
iceberg.rest-catalog.security=OAUTH2
iceberg.rest-catalog.oauth2.credential=<client_id>:<client_secret>
iceberg.rest-catalog.oauth2.scope=PRINCIPAL_ROLE:ALL
iceberg.rest-catalog.oauth2.server-uri=http://<polaris-host>:8181/api/catalog/v1/oauth/tokens
iceberg.rest-catalog.vended-credentials-enabled=true
fs.native-s3.enabled=true
s3.region=us-east-1
# MinIO or another S3-compatible store only:
s3.endpoint=http://<minio-host>:9000
s3.path-style-access=true

trino

SHOW SCHEMAS FROM polaris;
SELECT id, name, city, tier FROM polaris.db2_to_clickhouse_shop_db2inst1.customers ORDER BY id;

Trino 484 read the tables on this page with no further settings. Equality and positional deletes are read on every current release; deletion vectors from Trino 480, where the docs still mark them experimental.

Trino 484 reading the same table through Polaris with vended credentials: the same 10 rows.

An orders roll-up matched DB2 to the cent in all three engines. It gave 16 delivered orders for 8659.68, 18 placed for 10522.26, 11 shipped for 6397.25.

Data types: DB2 to Iceberg to ClickHouse

OLake Go maps DB2 types to Iceberg types, and ClickHouse maps Iceberg types to its own. Decimals and times are the ones to watch.

DB2	Iceberg (written by OLake Go)	ClickHouse shows	Notes
SMALLINT, INTEGER	int	Nullable(Int32)	The identity keys on this page arrived as Int32.
BIGINT	long	Nullable(Int64)
DECIMAL, NUMERIC, DECFLOAT, FLOAT, DOUBLE	double	Nullable(Float64)	The orders.amount sums on this page matched DB2 to the cent; round at query time when you need exact decimal arithmetic.
REAL	float	Nullable(Float32)
BOOLEAN	boolean	Nullable(Bool)
DATE, TIMESTAMP	timestamptz (UTC)	Nullable(DateTime64(6, 'UTC'))	OLake Go writes every timestamp in UTC, whatever the server time zone.
TIME	string	Nullable(String)
CHAR, VARCHAR, CLOB, GRAPHIC, VARGRAPHIC, DBCLOB, XML	string	Nullable(String)
BLOB	string	Nullable(String)	Binary values arrive as text; keep large binaries out of the sync.

DB2 types in the table come from the DB2 connector reference. The ClickHouse side comes from the ClickHouse Iceberg table engine page (checked 2026-10-30) and from DESCRIBE on the tables of this run.

Cost and maintenance

• Storage is your bucket: you pay S3 or MinIO rates for the Parquet and delete files, and nothing to Polaris or ClickHouse for holding them.
• Compute. OLake Go runs wherever you run Docker, on an amd64 host for the DB2 connector. ClickHouse spends CPU only when you query; a DataLakeCatalog query reads Parquet over the network with no MergeTree index, so load a hot slice into a MergeTree table when a dashboard needs sub-second answers.
• DB2 load. Each incremental sync is one range scan per table on the cursor column; index UPDATED_AT so it stays cheap as the table grows. Keep RUNSTATS in your regular maintenance window so snapshot chunks stay even.
• Maintenance. Nothing compacts the tables for you. A table that gets a new small file and delete file per sync wants compaction; OLake Fusion does this for the catalogs OLake Go writes to, and Spark's rewrite_data_files works through the same Polaris endpoint.
• Credential rotation. Polaris principals do not expire by default. Rotate the olake principal's secret on your own schedule; a rotated secret fails the next sync at the token exchange, before any data moves.

Troubleshooting

1. The source test fails on an Apple silicon Mac

image pull registry-1.docker.io/olakego/source-db2:v1.1.0: Error response from daemon: no matching manifest for linux/arm64/v8 in the manifest list entries

The OLake UI worker pulls connector images for the host's platform, and the DB2 image exists for amd64 only. Turn on Use Rosetta for x86_64/amd64 emulation in Docker Desktop, then pull the image yourself and tag it with the name the worker asks for:

bash

docker pull --platform linux/amd64 olakego/source-db2:v1.1.0
docker tag olakego/source-db2:v1.1.0 registry-1.docker.io/olakego/source-db2:v1.1.0

The run on this page hit this first and passed after these two commands. On an amd64 Linux host the error does not occur.

2. The first sync stops with "stats not populated"

stats not populated for table[DB2INST1.CUSTOMERS]. Please run CLP command: RUNSTATS ON TABLE DB2INST1.CUSTOMERS AND INDEXES ALL

DB2 has never collected statistics for the table, and OLake Go needs them to size the snapshot chunks. Run the RUNSTATS call from step 1 for every table in the job, then sync again.

3. A sync fails with SQL0668N after an ALTER TABLE

Adding, dropping or retyping a column can leave a DB2 table in REORG PENDING, and reads fail until it is reorganized. Run CALL SYSPROC.ADMIN_CMD('REORG TABLE db2inst1.customers'), then sync again. The DB2 connector page lists the operations that cause it.

4. The CLI picked the wrong cursor column

discover fills cursor_field in selected_streams.json with the first column in alphabetical order. In the run on this page that was CITY for CUSTOMERS and AMOUNT for ORDERS. Set it to UPDATED_AT for every stream before the first sync; the OLake UI asks for it on the Streams page instead.

5. ClickHouse lists the tables but every query returns a 404

Code: 499. DB::Exception: Failed to get object info for key olake/.../metadata/00003-....metadata.json in bucket lakehouse: No response body.. HTTP response code: 404

The object key is doubled. ClickHouse 26.8 and 26.10 join the table location onto the storage endpoint, then fail to strip it again when the endpoint has no trailing slash. Set the Polaris catalog's endpoint to a value ending in /, for example http://minio:9000/.

6. Updated rows are missing after an incremental sync

The update did not move the cursor column. OLake Go only fetches rows whose cursor value is above the last one it saw. Set UPDATED_AT in every write path, or add a trigger that sets it on UPDATE.

How this compares with other ways to move DB2 data to ClickHouse

Method	Freshness	Updates and deletes	Where the data lands	What you run
Debezium's Db2 connector, Kafka and the ClickHouse Kafka engine	Seconds, from SQL Replication capture tables	Yes, with your own merge logic	Kafka topics, then ClickHouse tables	Kafka, Kafka Connect, Debezium, tables in capture mode and, per Debezium's docs, an IBM InfoSphere Data Replication license (checked Oct 2026)
ClickHouse ODBC table function against DB2	Live, per query	Reads the current row	Nothing is copied; every query hits DB2	IBM's ODBC driver and a DSN on every ClickHouse server
Export (EXPORT TO ... OF DEL), then INSERT ... SELECT FROM s3(...)	Hours to daily	No; full reloads only	Native ClickHouse tables	A script, a scheduler and a staging bucket
OLake Go to Iceberg on Polaris (this guide)	Incremental on a schedule down to one minute	Inserts and updates that move the cursor; deletes need a soft-delete column	Apache Iceberg tables that ClickHouse, DuckDB, Trino and Spark all read	One Docker Compose stack, open source; your bucket plus your own compute

Log-based replication through Kafka is the choice when you need seconds of latency and hard deletes. Writing Iceberg once avoids paying for a second copy as soon as another engine needs the same rows.

When not to use this

• Every query must hit MergeTree speed. A DataLakeCatalog query reads Parquet with no primary index. Keep the Iceberg tables as the source and load the hot slice into a MergeTree table on a schedule.
• You need hard deletes replicated. Incremental sync cannot see a deleted row. Soft-delete in DB2, or use a log-based tool.
• Your DB2 runs on z/OS or IBM i. The connector covers DB2 LUW only.

FAQs

How do I get IBM DB2 data into ClickHouse?

Run OLake Go (open source, Apache 2.0) against DB2 LUW with a read-only user, and point it at an Apache Polaris catalog. The first sync copies each table in parallel chunks; every later sync fetches only rows whose cursor column (for example UPDATED_AT) moved, and upserts them. ClickHouse attaches the Polaris catalog with its DataLakeCatalog engine and reads the Iceberg tables in place. No Kafka, replication license or ODBC bridge is involved.

Does OLake Go do change data capture (CDC) from DB2?

No. The DB2 connector runs Full Refresh and Full Refresh + Incremental; it does not read the DB2 log or the SQL Replication capture tables. Incremental syncs select rows where a cursor column is greater than the last value seen, which captures inserts and updates that bump the cursor but not deletes.

Does ClickHouse read the updated rows correctly?

Yes. OLake Go writes each updated row as a new data file plus an equality delete for its key, and ClickHouse reads equality deletes from 25.8. In the run on this page ClickHouse 26.8 returned one row per key after the second sync, the same 10 customers DuckDB and Trino returned.

Why does the first sync ask me to run RUNSTATS?

OLake Go splits each table into chunks for the parallel snapshot using the row and page counts DB2 keeps in SYSCAT.TABLES. Those counts exist only after RUNSTATS has run once on the table. Run CALL SYSPROC.ADMIN_CMD('RUNSTATS ON TABLE schema.table AND INDEXES ALL') per table; statistics that are out of date do not stop a sync.

Which DB2 data types change on the way to ClickHouse?

DECIMAL, NUMERIC, DECFLOAT, FLOAT and DOUBLE become Iceberg double and show up as Float64, so round at query time when you need exact decimal arithmetic. INTEGER becomes Int32 and BIGINT Int64. DATE and TIMESTAMP become DateTime64 in UTC. TIME, XML, CLOB, BLOB and GRAPHIC types arrive as String.

Why Apache Polaris rather than a ClickHouse table?

A Polaris catalog is one open endpoint that ClickHouse, DuckDB, Trino and Spark all read, with its own access control and vended storage credentials. Loading into a ClickHouse MergeTree table is faster to query but makes ClickHouse the only reader, and a second engine means a second copy. Keep the Iceberg tables as the shared source and load hot slices into MergeTree when a dashboard needs it.

How we tested this

• Run on 2026-10-30 with OLake Go v1.1.0.
• Read back with ClickHouse 26.8.36 (LTS), DataLakeCatalog engine, DuckDB 1.5.5, Trino 484.
• A 2-table DB2 LUW 12.1.5 schema (8 customers, 40 orders) synced with Full Refresh + Incremental on an UPDATED_AT cursor and equality deletes as the Iceberg update/delete format into Apache Polaris 1.8.0 on MinIO, then 7 inserts and 5 updates picked up by a second sync. ClickHouse, DuckDB and Trino returned identical rows.

Related guides

DB2 to SnowflakeSame source, read from Snowflake.DB2 to BigQuerySame source, read from BigQuery.DB2 to Amazon RedshiftSame source, read from Amazon Redshift.Postgres to ClickHouseSame engine, replicated from Postgres.MySQL to ClickHouseSame engine, replicated from MySQL.MongoDB to ClickHouseSame engine, replicated from MongoDB.SQL Server to ClickHouseSame engine, replicated from SQL Server.Kafka to ClickHouseSame engine, replicated from Kafka.DB2 connector referenceEvery field, sync mode and troubleshooting note for the source.Query engine compatibilityWhich engines read which catalogs and delete formats.All integrationsEvery source and engine pair with a tested guide.

Deeper reading on this site:

• DB2 LUW connector reference
• IBM DB2 LUW to a lakehouse with Apache Iceberg
• Equality vs positional deletes vs deletion vectors
• OLake Go ingestion benchmarks

Start with your own DB2

The quickstart runs OLake UI with Docker Compose. Create the source, the Polaris destination and the job as above, and the first sync lands your tables where ClickHouse can read them.

QuickstartJoin Slack