MongoDB to ClickHouse: CDC via Apache Iceberg

MongoDB to ClickHouse usually means ClickPipes on ClickHouse Cloud, the MongoDB table engine, or Debezium through Kafka, and tables that only ClickHouse can read. This guide takes a different route.

OLake Go reads a MongoDB change stream and commits the changes as Apache Iceberg tables into an Apache Polaris catalog. ClickHouse attaches that catalog as a database.

Polaris is an open source Iceberg REST (Representational State Transfer) catalog you run yourself, with the files in your own bucket. Nothing else is in the path: no Kafka, no Debezium, no ClickHouse Cloud. DuckDB and Trino read the same tables through the same endpoint.

By Shubham Satish Baldava · Updated October 29, 2026

Key takeaways

• OLake Go replicates MongoDB collections into Iceberg tables in Polaris: a full historical snapshot first, then change data capture (CDC) from a change stream, with no Kafka or Debezium in between.
• ClickHouse attaches the whole Polaris catalog as one database with DataLakeCatalog, reads the files with the credentials Polaris vends, and sees every change the moment a sync commits.
• The same tables, written once by OLake Go, are also queryable from DuckDB and Trino through the same catalog. In the run on this page all three engines returned identical rows.

The short answer. To replicate MongoDB to ClickHouse with change data capture (CDC), run OLake Go with a MongoDB replica set as the source. Point it at an Apache Polaris catalog as the Iceberg destination.

The first sync copies each collection in full (the historical snapshot); every later sync replays inserts, updates and deletes from the change stream. ClickHouse, DuckDB and Trino then query the same Iceberg tables, with no second copy.

How MongoDB CDC reaches ClickHouse here

Two things happen in order, and OLake Go does both on its own. The first sync opens a change stream and saves its resume token. Then it reads each collection in parallel chunks and writes Parquet data files: that is the historical load.

Every later sync resumes the stream from the saved token and replays the inserts, updates and deletes since. Nothing written during the snapshot is lost, and there is no separate backfill job.

Each sync ends with one commit to Polaris. OLake Go asks Polaris for short-lived storage credentials, uploads the data and delete files, then posts the new snapshot to the catalog. ClickHouse, DuckDB and Trino see it on their next query; none of them keeps a copy.

One thing to keep an eye on: the oplog is MongoDB's replay window. A change stream can only resume from a token that is still in the oplog.

A job paused for longer than the oplog window cannot resume and needs a fresh snapshot. Size the oplog for your longest planned pause; the source test on this page reports the current window.

What you need

• MongoDB 4.0 or later running as a replica set, self-hosted or on Atlas, with a user that can read the database and the oplog. The collections on this page ran on MongoDB 7 as a single-node replica set; step 1 covers Atlas too.
• An Apache Polaris server and a bucket, both reachable from OLake Go and from ClickHouse. The run on this page used Polaris 1.8.0 and MinIO in Docker; on AWS, use an S3 bucket and an IAM role instead.
• ClickHouse 26.1 or later, where the DataLakeCatalog engine reads REST catalogs. The run on this page used the 26.8 LTS release in Docker.
• OLake UI from the Docker Compose install. The run on this page used OLake Go v1.1.0.
• Docker, and about 40 minutes.

Step 1: Prepare MongoDB for CDC

Most readers point OLake Go at collections they already have. If you want to follow along with the exact data from this page, create these two collections first:

The two sample collections used on this page

mongosh, database shop

const now = new Date();
db.customers.insertMany([
  { _id: 1, name: 'Aarav Mehta', email: 'aarav@example.com', city: 'Mumbai', tier: 'gold', created_at: now, updated_at: now },
  // ... 8 customers with integer _id values
]);
db.orders.insertMany([
  { _id: 1, customer_id: 2, product: 'Headphones', amount: NumberDecimal('99.19'), status: 'shipped', created_at: now, updated_at: now },
  // ... 40 orders
]);
// Then insert whatever you like. This page started with 8 customers and 40 orders.

OLake Go reads MongoDB changes through change streams, which MongoDB builds on the replica set oplog. Three things must be in place before the first sync.

Setting	Value	Why it matters
Replica set	any name, even a single node	Change streams and the oplog exist only in replica set mode. Atlas clusters are already replica sets.
Oplog retention	long enough to cover your longest pause	OLake Go resumes from a change-stream token saved after each sync; if the oplog has rolled past it, the next sync needs a fresh snapshot.
A user for OLake Go	read on the database plus clusterMonitor	The snapshot reads the collections; the change stream needs oplog access, which clusterMonitor (or readAnyDatabase) grants.

mongosh, on the primary

// 1. A single-node replica set, if you do not have one (self-hosted only)
//    start mongod with --replSet rs0, then:
rs.initiate({ _id: "rs0", members: [{ _id: 0, host: "<host>:27017" }] })

// 2. A user for OLake Go
use admin
db.createUser({
  user: "olake", pwd: "<password>",
  roles: [{ role: "read", db: "shop" }, { role: "clusterMonitor", db: "admin" }]
})

// 3. Check the oplog window; OLake Go warns below 7 days
use local
db.oplog.rs.stats().maxSize      // bytes; raise with replSetResizeOplog if it is small

Check the result before moving on:

javascript

rs.status().members.map(m => m.stateStr)      // one PRIMARY
db.getSiblingDB("admin").getUsers().users.map(u => u.user)

On MongoDB Atlas the replica set and oplog are already there; create the user under Database Access with the read role on your database and clusterMonitor, and allow OLake Go's IP in Network Access. The full guide, including Atlas screenshots, is MongoDB and Atlas CDC setup.

The oplog is your replay window

Atlas and most self-hosted installs keep 24 hours or less of oplog by default. OLake Go's source test shows the current window and warns below 7 days. Size it for your longest planned pause, or run the job often enough that the token never falls out of the window.

Step 2: Set up the Polaris catalog

ClickHouse reads Iceberg tables through a catalog. Apache Polaris is the one this page uses: an open source REST catalog that you run next to your bucket.

OLake Go writes into it, and ClickHouse, DuckDB and Trino read from it with the credentials Polaris hands out. The Iceberg files sit in your own S3 or MinIO bucket; Polaris holds only the pointers and the permissions.

Catalog	OLake Go writes	ClickHouse reads	Verified on this page
Apache Polaris	Yes	Yes: DataLakeCatalog with catalog_type rest	Yes, end to end

Apache Polaris

Apache Polaris is an open source Iceberg REST catalog. It keeps the table metadata pointers, hands out short-lived storage credentials to engines, and applies its own role-based access control. OLake Go writes to it through the standard Iceberg REST API, and ClickHouse, Trino, DuckDB and Spark read from the same endpoint. The files live in an S3 bucket or an S3-compatible store that you own.

Three things need to exist before OLake Go can write: a Polaris server, a catalog inside it that points at your bucket, and a principal (a client id and secret) with the right to manage that catalog. The run on this page used Polaris 1.8.0 and MinIO in Docker; on AWS, swap the MinIO endpoint for a plain S3 bucket and an IAM role.

docker-compose.yml (Polaris and MinIO, for a local test)

services:
  minio:
    image: minio/minio
    hostname: lake-minio
    command: server /data --console-address ":9001"
    environment: { MINIO_ROOT_USER: minioadmin, MINIO_ROOT_PASSWORD: minioadmin }
    ports: ["9000:9000", "9001:9001"]
  polaris:
    image: apache/polaris:latest
    depends_on: [minio]
    ports: ["8181:8181", "8182:8182"]
    environment:
      AWS_REGION: us-east-1
      AWS_ACCESS_KEY_ID: minioadmin
      AWS_SECRET_ACCESS_KEY: minioadmin
      POLARIS_BOOTSTRAP_CREDENTIALS: POLARIS,root,<root-secret>
      polaris.realm-context.realms: POLARIS

Polaris stores its own state in memory with this setup, so it starts empty every time; a production deployment points it at Postgres. Once it is up, create the bucket, the catalog and the principal through the management API. root is the bootstrap principal; the olake principal is the one OLake Go and the engines use.

Polaris, once

P=http://localhost:8181

# 1. A bucket for the tables (MinIO; on AWS create it in the console or with aws s3 mb)
aws --endpoint-url http://localhost:9000 s3 mb s3://lakehouse

# 2. A root token
TOKEN=$(curl -s $P/api/catalog/v1/oauth/tokens -H 'Polaris-Realm: POLARIS' \
  -d grant_type=client_credentials -d client_id=root -d client_secret=<root-secret> \
  -d scope=PRINCIPAL_ROLE:ALL | jq -r .access_token)
H=(-H "Authorization: Bearer $TOKEN" -H 'Polaris-Realm: POLARIS' -H 'Content-Type: application/json')

# 3. The catalog: where the files go, and how Polaris and the engines reach the store
curl -s "${H[@]}" $P/api/management/v1/catalogs -d '{"catalog": {
  "name": "olake", "type": "INTERNAL", "readOnly": false,
  "properties": {"default-base-location": "s3://lakehouse/olake"},
  "storageConfigInfo": {"storageType": "S3", "allowedLocations": ["s3://lakehouse/olake"],
    "endpoint": "http://<minio-host>:9000/", "endpointInternal": "http://lake-minio:9000",
    "pathStyleAccess": true, "region": "us-east-1"}}}'

# 4. A principal for OLake Go and the engines. The response holds clientId and clientSecret: keep them.
curl -s "${H[@]}" $P/api/management/v1/principals -d '{"principal": {"name": "olake"}}'

# 5. Roles and the grant that lets the principal create namespaces and tables in the catalog
curl -s "${H[@]}" $P/api/management/v1/principal-roles -d '{"principalRole": {"name": "olake_role"}}'
curl -s "${H[@]}" $P/api/management/v1/catalogs/olake/catalog-roles -d '{"catalogRole": {"name": "olake_catalog_role"}}'
curl -s -X PUT "${H[@]}" $P/api/management/v1/principals/olake/principal-roles -d '{"principalRole": {"name": "olake_role"}}'
curl -s -X PUT "${H[@]}" $P/api/management/v1/principal-roles/olake_role/catalog-roles/olake -d '{"catalogRole": {"name": "olake_catalog_role"}}'
curl -s -X PUT "${H[@]}" $P/api/management/v1/catalogs/olake/catalog-roles/olake_catalog_role/grants -d '{"type": "catalog", "privilege": "CATALOG_MANAGE_CONTENT"}'

What each block is for:

• endpoint is the store address Polaris hands to engines along with vended credentials; endpointInternal is the address Polaris itself uses. On AWS S3 leave both out. On MinIO both are required, or Polaris tries to reach AWS and fails. Keep the trailing slash on endpoint: ClickHouse 26.8 and 26.10 build the object path from it, and without the slash every read fails with a 404 (the fix is explained under troubleshooting on this page).
• Polaris vends credentials through the store's STS endpoint. MinIO exposes one, so nothing else is needed. For a store without STS, add "stsUnavailable": true and give Polaris static keys to hand out.
• CATALOG_MANAGE_CONTENT lets the principal create namespaces and tables and read and write their data. Scope it to a namespace instead when several teams share the catalog.
• The principal's secret is shown once, in the response to step 4. It is what goes into OLake Go as <client_id>:<client_secret>.

Hostnames with underscores break Polaris

Polaris is a Java service, and Java's URI parser rejects hostnames that contain an underscore. A MinIO container named lab_minio makes every table creation fail with host must not be null or Expected scheme-specific part at index 5: http:. Use a hyphen in the hostname (lake-minio), as above.

You can check the principal outside OLake Go by exchanging its credentials for a token, the same call OLake Go makes on every sync:

Token check

curl -s http://localhost:8181/api/catalog/v1/oauth/tokens -H 'Polaris-Realm: POLARIS' \
  -d grant_type=client_credentials -d client_id=<client_id> -d client_secret=<client_secret> \
  -d scope=PRINCIPAL_ROLE:ALL

A good principal returns a JSON object with an access_token. Running Polaris on Kubernetes, with Postgres behind it, or with an external identity provider is covered on the Polaris catalog page.

Step 3: Connect MongoDB to OLake Go

OLake UI

1. In the OLake UI, open Sources and click Create Source. Pick MongoDB as the connector, give the source a name and keep the OLake Go version on the latest release.
2. Under Change data capture (CDC), leave Enable CDC on.
3. Under Connection details, add each host as host:port (or one SRV host with Use SRV on for Atlas), then the Database Name, Auth DB (admin), username, password and the Replica Set name. Leave Read Preference on primary unless you have secondaries to spare.
4. Click Set up source. OLake Go tests the connection, confirms that it can open a change stream, and reports the oplog window with a warning below 7 days.

Here is the source form from the run on this page. MongoDB ran in another container as a single-node replica set, so the host is host.docker.internal; the user is the one from step 1.

The MongoDB source in OLake UI: Enable CDC on, with the replica set and the user from step 1.

Set up source runs the connection test, then checks that the server is a replica set, that a change stream can be opened and how long the oplog window is. When one fails, the dialog names the setting and its recommended value.

The full field reference, including SRV strings, IAM authentication and SSH tunnels, is on the MongoDB connector page.

OLake CLI

Create source.json in your config folder. replica_set is what lets OLake Go open the change stream:

OLAKE_DIRECTORY/source.json

{
  "hosts": [
    "host1:27017",
    "host2:27017",
    "host3:27017"
  ],
  "username": "username",
  "password": "password",
  "authdb": "admin",
  "replica_set": "rs0",
  "read_preference": "secondaryPreferred",
  "srv": false,
  "database": "database",
  "max_threads": 5,
  "backoff_retry_count": 4,
  "chunking_strategy":""
}

Then confirm OLake Go can reach the database:

check command

docker run --pull=always \
  -v "$PWD/config:/mnt/config" \
  olakego/source-mongodb:latest \
  check --config /mnt/config/source.json

A working setup returns "status": "SUCCEEDED" with a prerequisites list that reports change_streams as passed and the oplog_retention window. Every field is described on the MongoDB connector page.

Step 4: Configure the Iceberg destination

Apache Polaris

OLake UI

1. Open Destinations and click Create Destination.
2. Pick Apache Iceberg as the connector and Polaris as the catalog type.
3. Choose OAuth2 as the authentication type. REST Catalog URI is the Polaris catalog endpoint (http://<polaris-host>:8181/api/catalog), Polaris Catalog Name the catalog from the setup (olake), REST Auth URI the token endpoint, and REST Credential the principal's <client_id>:<client_secret>.
4. Fill in the store: AWS Region, and for MinIO the S3 Endpoint with the AWS Access Key and AWS Secret Key. On AWS S3 leave the endpoint empty.
5. Click Create. OLake Go exchanges the credentials for a token, creates a namespace and a test table through Polaris, writes to the bucket and drops the test table again.

Here is the destination form from the run on this page. Polaris and MinIO ran in other containers, so both addresses use host.docker.internal; the credential is the olake principal from step 2.

The Polaris destination in OLake UI: OAuth2 with the principal from step 2, and the MinIO endpoint for the files.

Polaris Configuration Parameters

Parameter	Sample Value	Description
REST Catalog URI required	http://<REST_ENDPOINT>:8181/api/catalog	Endpoint URL for the Polaris REST catalog service.
Authentication Type required	oauth2	Authentication method OLake Go uses for Polaris REST catalog requests. Set to match your catalog's authentication configuration.
Polaris Catalog Name required	<POLARIS_CATALOG_NAME>	Name of the Polaris catalog to use, OLake Go passes this to the Polaris REST catalog as the warehouse identifier, so catalog operations are routed to that catalog and its pre-configured S3 storage and IAM settings.
REST Auth URI required	http://<REST_ENDPOINT>:8181/api/catalog/v1/oauth/tokens	URL of the REST catalog’s OAuth2 token endpoint. OLake Go uses this with REST Credential to request an access token during OAuth2 client-credentials authentication.
REST Credential required	<olake_user_client_id>:<olake_user_client_secret>	Specifies the client ID and secret for OAuth2, formatted as client_id:client_secret. Used with REST Auth URI when OLake Go requests an access token from the catalog’s auth service. Not needed when using REST Token or other auth types.
AWS Region required	ap-south-1	AWS Region where your S3 bucket exists. Required so OLake Go calls the correct regional S3 endpoint.
REST Scope	PRINCIPAL_ROLE:ALL	Required for OAuth2 client-credentials auth. Scopes are space-separated. Used with REST Auth URI and REST Credential for OAuth2 client-credentials auth. Values are catalog-specific. Not needed when using REST Token or other auth types.
S3 Endpoint	http://<S3_ENDPOINT>:9000	S3 API endpoint for writing Iceberg data files. Leave empty for AWS S3 but required for S3-compatible storage like MinIO.
AWS Access Key	XXX	AWS access key ID for authentication. Optional if using IAM roles or instance profiles. If using IAM: Docker Compose: add the required IAM environment variables under x-envs in your compose file. See Service Environment Variables for setup details. Kubernetes: set up pod IAM in Cloud Integration.
AWS Secret Key	XXX	AWS secret access key for authentication. Optional if using IAM roles or instance profiles. If using IAM: Docker Compose: add the required IAM environment variables under x-envs in your compose file. See Service Environment Variables for setup details. Kubernetes: set up pod IAM in Cloud Integration.
Enable Arrow Writes	false	Writes data and delete files using Apache Arrow based writer and registers them in Iceberg.
Catalog Name	olake_iceberg	Name of the Iceberg catalog OLake Go registers tables under. Defaults to olake_iceberg if left empty.

Catalog Name Supported for v0.3.5 and above

For the catalog name, OLake Go only supports lowercase letters and underscores. Spaces and special characters are not supported.

OLake CLI

Create destination.json next to source.json:

destination.json

{
  "type": "ICEBERG",
  "writer": {
    "catalog_type": "polaris",
    "catalog_name": "polaris",
    "rest_catalog_url": "http://<polaris-host>:8181/api/catalog",
    "iceberg_s3_path": "olake",
    "rest_auth_type": "oauth2",
    "oauth2_uri": "http://<polaris-host>:8181/api/catalog/v1/oauth/tokens",
    "credential": "<client_id>:<client_secret>",
    "scope": "PRINCIPAL_ROLE:ALL",
    "aws_region": "us-east-1",
    "s3_endpoint": "http://<minio-host>:9000",
    "aws_access_key": "<minio access key>",
    "aws_secret_key": "<minio secret key>",
    "arrow_writes": false
  }
}

Polaris Configuration Parameters

Parameter	Sample Value	Description
catalog_type required	polaris	Defines the catalog type used by the writer.
rest_catalog_url required	http://<REST_ENDPOINT>:8181/api/catalog	Endpoint URL for the Polaris REST catalog service.
iceberg_s3_path required	<POLARIS_CATALOG_NAME>	Name of the Polaris catalog to use, OLake Go passes this to the Polaris REST catalog as the warehouse identifier, so catalog operations are routed to that catalog and its pre-configured S3 storage and IAM settings.
rest_auth_type required	oauth2	Authentication method OLake Go uses for Polaris REST catalog requests. Set to match your catalog's authentication configuration.
oauth2_uri required	http://<REST_ENDPOINT>:8181/api/catalog/v1/oauth/tokens	URL of the REST catalog’s OAuth2 token endpoint. OLake Go uses this with REST Credential to request an access token during OAuth2 client-credentials authentication.
credential required	<olake_user_client_id>:<olake_user_client_secret>	Specifies the client ID and secret for OAuth2, formatted as client_id:client_secret. Used with REST Auth URI when OLake Go requests an access token from the catalog’s auth service.
aws_region required	ap-south-1	AWS Region where your S3 bucket exists. Required so OLake Go calls the correct regional S3 endpoint.
scope	PRINCIPAL_ROLE:ALL	Required for OAuth2 client-credentials auth. Scopes are space-separated. Used with REST Auth URI and REST Credential for OAuth2 client-credentials auth. Values are catalog-specific.
s3_endpoint	http://<S3_ENDPOINT>:9000	S3 API endpoint for writing Iceberg data files. Leave empty for AWS S3 but required for S3-compatible storage like MinIO.
aws_access_key	XXX	AWS access key ID for authentication. Optional if using IAM roles or instance profiles. If using IAM: Docker Compose: add the required IAM environment variables under x-envs in your compose file. See Service Environment Variables for setup details. Kubernetes: set up pod IAM in Cloud Integration.
aws_secret_key	XXX	AWS secret access key for authentication. Optional if using IAM roles or instance profiles. If using IAM: Docker Compose: add the required IAM environment variables under x-envs in your compose file. See Service Environment Variables for setup details. Kubernetes: set up pod IAM in Cloud Integration.
arrow_writes	false	Writes data and delete files using Apache Arrow based writer and registers them in Iceberg.
catalog_name	olake_iceberg	Name of the Iceberg catalog OLake Go registers tables under. Defaults to olake_iceberg if left empty.

Catalog Name Supported for v0.3.5 and above

For the catalog name, OLake Go only supports lowercase letters and underscores. Spaces and special characters are not supported.

Step 5: Create the job and run the first sync

OLake UI

1. Open Jobs and click Create Job. Give it a name and a schedule. Every Day at a fixed hour, or a cron expression, both work; the schedule is how often the change stream is drained, not how often a full copy runs.
2. Select the source and the destination you created.
3. Under Advanced Settings, set Target Query Engines to ClickHouse, plus DuckDB and Trino if they will read the tables too. OLake Go then resolves an Iceberg delete format every one of those engines can read, so you never pick one by hand.
4. Click Next. OLake Go discovers the collections and opens the Streams page.
5. Tick the collections to replicate. For each one, set the sync mode to Full Refresh + CDC and the ingestion mode to Upsert. Leave the Upsert Type as OLake Go resolved it (Equality Deletes in the run on this page).
6. Click Create Job, then open the job's menu and choose Sync Now for the first run.

Three target query engines were set on this job. That is the only format decision you make.

OLake Go works out which Iceberg delete format all three can read and sets it on every stream. With ClickHouse in the list it chose equality deletes, which need no index between runs.

Each collection: Full Refresh + CDC, Upsert, and the upsert type OLake Go resolved (Equality Deletes in this run). The Iceberg DB name at the top is editable.

Job history: the 15 s historical snapshot, then the 8 s change-only sync.

OLake CLI

Discover the collections, choose the streams, then sync:

discover

docker run --pull=always \
-v "$PWD/config:/mnt/config" \
olakego/source-mongodb:latest \
discover --config /mnt/config/source.json

discover writes two files: available_streams.json (every collection it found, do not edit) and selected_streams.json (the collections you want and their settings). In selected_streams.json, set sync_mode to {props.cliSyncMode ?? 'cdc'} for each collection and pick the delete format your readers can open:

selected_streams.json (one selected stream)

{
"selected_streams": {
  "shop": [
    {
      "stream_name": "customers",
      "sync_mode": "cdc",
      "normalization": true,
      "append_mode": false,
      "update_type": "eq"
    }
  ]
}
}

update_type accepts eq (equality deletes, the default), pos (positional deletes) and dv (deletion vectors). Positional deletes and deletion vectors keep a small index between runs, so give the container a persistent directory for it through OLAKE_INDEX_DB_DIR. The first sync runs without a state file; OLake Go writes one at the end of the run:

first sync

docker run --pull=always \
-v "$PWD/config:/mnt/config" \
-e OLAKE_INDEX_DB_DIR=/mnt/config/index-db \
olakego/source-mongodb:latest \
sync \
--config /mnt/config/source.json \
--selected-streams /mnt/config/selected_streams.json \
--available-streams /mnt/config/available_streams.json \
--destination /mnt/config/destination.json

Every later run adds --state /mnt/config/state.json to the same command, which is how OLake Go knows to replay only the changes since the last run. The full selected_streams.json reference is on the Docker CLI page; the index directory is described under index DB setup.

On this page the first run copied 8 customers and 40 orders (48 documents, 6.40 KB) and finished in 15 seconds, container start included. OLake Go names the destination namespace after the job and the source database, so mongodb_to_clickhouse on database shop produced mongodb_to_clickhouse_shop. The Iceberg DB field at the top of the Streams page changes it before the first sync.

Step 6: Query the tables in ClickHouse

Apache Polaris

There is no registration step. ClickHouse attaches the whole Polaris catalog as one database with the DataLakeCatalog engine, lists every table OLake Go committed, and reads the files with the credentials Polaris vends. One thing to know before the first query: ClickHouse has one level of namespace, so the Iceberg namespace and table name go together inside backticks.

clickhouse-client

SET allow_database_iceberg = 1;

CREATE DATABASE polaris
ENGINE = DataLakeCatalog('http://<polaris-host>:8181/api/catalog')
SETTINGS
catalog_type = 'rest',
warehouse = 'olake',
catalog_credential = '<client_id>:<client_secret>',
oauth_server_uri = 'http://<polaris-host>:8181/api/catalog/v1/oauth/tokens',
auth_scope = 'PRINCIPAL_ROLE:ALL',
vended_credentials = true,
region = 'us-east-1';

SHOW TABLES FROM polaris;

-- What OLake Go created (namespace = <job name>_<source database>_<source schema>)
SELECT _id, name, city, tier
FROM polaris.`mongodb_to_clickhouse_shop.customers`
ORDER BY _id;

vended_credentials = true makes ClickHouse read the files with the short-lived keys Polaris hands out per table, so no store keys sit in ClickHouse. To use static keys instead, replace it with storage_endpoint, aws_access_key_id and aws_secret_access_key. The database is read-only: INSERT, ALTER and RENAME DATABASE return NOT_IMPLEMENTED.

OLake Go adds four columns to every table: _olake_id (a hash of the primary key), _op_type, _olake_timestamp and _cdc_timestamp. _op_type is r for a row from the first snapshot, c for an insert, u for an update and d for a delete. Filter them out of dashboards, or use them to see when a row last changed.

MongoDB's _id is the key column, and a Decimal128 field such as amount arrives as String, so cast it before you sum:

Money stored as NumberDecimal

SELECT status, count() AS n, round(sum(toDecimal64(amount, 2)), 2) AS total
FROM polaris.`mongodb_to_clickhouse_shop.orders`
GROUP BY status ORDER BY status;

ClickHouse 26.8 reading the customers table through Polaris after the change sync: 10 rows, one per _id.

Try it on your own database

OLake Go is open source. The quickstart brings up the UI with Docker Compose in a few minutes; the Slack community answers setup questions the same day.

QuickstartJoin Slack

Step 7: Change documents in MongoDB and sync again

To see the change path work, the run on this page made 12 changes in MongoDB after the first sync:

Changes made in MongoDB between the two syncs

const now = new Date();
db.customers.insertMany([
  { _id: 9,  name: 'Maya Patel',  email: 'maya@example.com',  city: 'Pune',      tier: 'silver', created_at: now, updated_at: now },
  { _id: 10, name: 'Lucas Silva', email: 'lucas@example.com', city: 'São Paulo', tier: 'bronze', created_at: now, updated_at: now }
]);
db.orders.insertMany([
  { _id: 41, customer_id: 9,  product: 'Laptop',   amount: NumberDecimal('1299.00'), status: 'placed', created_at: now, updated_at: now },
  { _id: 42, customer_id: 9,  product: 'Mouse',    amount: NumberDecimal('25.50'),   status: 'placed', created_at: now, updated_at: now },
  { _id: 43, customer_id: 10, product: 'Monitor',  amount: NumberDecimal('349.99'),  status: 'placed', created_at: now, updated_at: now },
  { _id: 44, customer_id: 3,  product: 'Keyboard', amount: NumberDecimal('89.00'),   status: 'placed', created_at: now, updated_at: now },
  { _id: 45, customer_id: 7,  product: 'Webcam',   amount: NumberDecimal('79.90'),   status: 'placed', created_at: now, updated_at: now }
]);
db.customers.updateOne({ name: 'Liam Carter' }, { $set: { tier: 'gold', updated_at: now } });
db.customers.updateOne({ name: 'Zara Ali' },    { $set: { city: 'Abu Dhabi', updated_at: now } });
db.orders.updateMany({ _id: { $in: [1, 4, 7] } }, { $set: { status: 'delivered', updated_at: now } });

The second sync read 12 change events (1.60 KB) from the stream and finished in 8 seconds end to end. For customers that meant one new data file with 4 rows: 2 inserts plus the new versions of 2 updated documents. One small equality delete file retires the 2 old versions by key.

OLake Go's commit report in the sync log shows it directly: addedDataFiles=1, addedRecords=4, addedEqualityDeleteFiles=1, addedEqualityDeletes=2. The orders table has the same shape with 8 rows and 3 deletes.

The change-only sync in the OLake UI log: one data file and one equality delete file per collection.

ClickHouse shows the result on the next query, with no refresh. Liam Carter is gold, Zara Ali moved to Abu Dhabi, and the two new customers are there.

The two retired versions are gone, because ClickHouse applies the equality deletes while reading. The _op_type column shows how each row arrived: r for the snapshot, c for the inserts and u for the updates.

Schema changes. When a new field appears in the documents, OLake Go adds it as a column on the next sync. ClickHouse shows it the next time the table is opened, with nulls for older rows.

A field whose type changes in an incompatible way fails the sync rather than corrupt the table. Full rules are on the schema evolution page.

Query the same tables from DuckDB and Trino

This is the point of writing Iceberg instead of loading ClickHouse tables. Nothing is exported, copied or refreshed: both engines read the files OLake Go wrote, through the same Polaris endpoint.

DuckDB

DuckDB attaches Polaris as an Iceberg REST catalog with the principal's client id and secret; DuckDB fetches the token itself. On a laptop, where the store address Polaris hands out may not resolve, add an S3 secret for the store and turn credential vending off:

DuckDB 1.5.5

INSTALL iceberg; LOAD iceberg;
INSTALL httpfs;  LOAD httpfs;

CREATE SECRET polaris_secret (
TYPE iceberg,
CLIENT_ID '<client_id>', CLIENT_SECRET '<client_secret>',
OAUTH2_SERVER_URI 'http://<polaris-host>:8181/api/catalog/v1/oauth/tokens',
OAUTH2_SCOPE 'PRINCIPAL_ROLE:ALL'
);
-- MinIO, or any store whose address differs from the one Polaris vends
CREATE SECRET store_secret (
TYPE s3, KEY_ID '<access key>', SECRET '<secret key>',
ENDPOINT '<minio-host>:9000', URL_STYLE 'path', USE_SSL false, REGION 'us-east-1'
);

ATTACH 'olake' AS polaris (
TYPE iceberg,
ENDPOINT 'http://<polaris-host>:8181/api/catalog',
SECRET polaris_secret,
ACCESS_DELEGATION_MODE 'none'
);

SHOW ALL TABLES;
SELECT _id, name, city, tier FROM polaris.mongodb_to_clickhouse_shop.customers ORDER BY _id;

With Polaris on AWS S3, drop the S3 secret and the ACCESS_DELEGATION_MODE line: DuckDB then reads with the credentials Polaris vends. DuckDB reads equality and positional deletes, and deletion vectors from duckdb-iceberg 1.5.3 onward.

DuckDB 1.5.5 reading the same table through Polaris: the same 10 rows.

Trino

Trino reaches Polaris through its Iceberg connector in REST mode. One catalog properties file does it; Trino fetches the OAuth2 token with the principal's credentials and reads the files with the credentials Polaris vends.

etc/catalog/polaris.properties

connector.name=iceberg
iceberg.catalog.type=rest
iceberg.rest-catalog.uri=http://<polaris-host>:8181/api/catalog
iceberg.rest-catalog.warehouse=olake
iceberg.rest-catalog.security=OAUTH2
iceberg.rest-catalog.oauth2.credential=<client_id>:<client_secret>
iceberg.rest-catalog.oauth2.scope=PRINCIPAL_ROLE:ALL
iceberg.rest-catalog.oauth2.server-uri=http://<polaris-host>:8181/api/catalog/v1/oauth/tokens
iceberg.rest-catalog.vended-credentials-enabled=true
fs.native-s3.enabled=true
s3.region=us-east-1
# MinIO or another S3-compatible store only:
s3.endpoint=http://<minio-host>:9000
s3.path-style-access=true

trino

SHOW SCHEMAS FROM polaris;
SELECT _id, name, city, tier FROM polaris.mongodb_to_clickhouse_shop.customers ORDER BY _id;

Trino 484 read the tables on this page with no further settings. Equality and positional deletes are read on every current release; deletion vectors from Trino 480, where the docs still mark them experimental.

Trino 484 reading the same table through Polaris with vended credentials: the same 10 rows.

An orders roll-up matched MongoDB to the cent in all three engines, each casting amount to a decimal first. It gave 16 delivered orders for 8659.68, 18 placed for 10522.26, 11 shipped for 6397.25.

Data types: MongoDB to Iceberg to ClickHouse

OLake Go maps MongoDB's BSON (Binary JSON) types to Iceberg types, and ClickHouse maps Iceberg types to its own. Scalars pass through; everything structured arrives as text.

MongoDB type	Iceberg (written by OLake Go)	ClickHouse shows	Notes
int, timestamp	int	Nullable(Int32)	The integer _id keys on this page arrived as Int32.
long	long	Nullable(Int64)
double	double	Nullable(Float64)
boolean	boolean	Nullable(Bool)
date	timestamptz (UTC)	Nullable(DateTime64(6, 'UTC'))
decimal128	string	Nullable(String)	Money stored as NumberDecimal arrives as text. sum(toDecimal64(amount, 2)) gives the exact total; the roll-up on this page used that cast. Store doubles in MongoDB if you want a numeric column without the cast.
objectId, binData, regex, code	string	Nullable(String)	An ObjectId primary key arrives as its 24-character hex string.
object, array	string	Nullable(String)	JSON text. Parse with JSONExtract or JSONExtractString in ClickHouse.

MongoDB types in the table come from the MongoDB connector reference. The ClickHouse side comes from the ClickHouse Iceberg table engine page (checked 2026-10-29) and from DESCRIBE on the tables of this run. OLake Go adds _cdc_resume_token on MongoDB tables, the change-stream token each change came from.

Cost and maintenance

• Storage is your bucket: you pay S3 or MinIO rates for the Parquet and delete files, and nothing to Polaris or ClickHouse for holding them.
• Compute. OLake Go runs wherever you run Docker. ClickHouse spends CPU only when you query; a DataLakeCatalog query reads Parquet over the network with no MergeTree index, so load a hot slice into a MergeTree table when a dashboard needs sub-second answers.
• Maintenance. Nothing compacts the tables for you. A CDC table that gets a new small file and delete file per sync wants compaction; OLake Fusion does this for the catalogs OLake Go writes to, and Spark's rewrite_data_files works through the same Polaris endpoint.
• Oplog size. The oplog is sized in gigabytes, not hours, so a write-heavy cluster shortens the window. Check it after a busy day and resize before a planned pause.
• Credential rotation. Polaris principals do not expire by default. Rotate the olake principal's secret on your own schedule with the management API; a rotated secret fails the next sync at the token exchange, before any data moves.

Troubleshooting

1. The source test warns about the oplog

oplog_retention: current 1.3 hours, recommended >= 7 days

Not a blocker, but it is the window in which the next sync must start. The fix is on the MongoDB side: replSetResizeOplog on self-hosted, or the minimum oplog window setting on Atlas. Until then, run the job more often than the window.

2. The source test cannot open a change stream

change_streams: OLake cannot open a change stream, so CDC cannot run.

Either mongod is not running with --replSet, or the user lacks oplog access. Check rs.status() on the server and give the OLake Go user clusterMonitor (or readAnyDatabase) in admin on top of read on the database.

3. ClickHouse lists the tables but every query returns a 404

Code: 499. DB::Exception: Failed to get object info for key olake/mongodb_to_clickhouse_shop/customers/lakehouse/olake/mongodb_to_clickhouse_shop/customers/metadata/00003-....metadata.json in bucket lakehouse: No response body.. HTTP response code: 404

The object key is doubled. ClickHouse 26.8 and 26.10 join the table location onto the storage endpoint, then fail to strip it again when the endpoint has no trailing slash. Set the Polaris catalog's endpoint (and storage_endpoint in ClickHouse, if you use static keys) to a value ending in /, for example http://minio:9000/.

4. A money column will not sum

Code: 43. DB::Exception: Illegal type String of argument for aggregate function sum. (ILLEGAL_TYPE_OF_ARGUMENT)

The field is a Decimal128 in MongoDB, which arrives as String. Wrap it: sum(toDecimal64(amount, 2)). The run on this page hit this on its first roll-up.

5. The destination test fails at the token exchange

OLake Go's destination test exchanges the principal's credentials at /api/catalog/v1/oauth/tokens before anything else. Three causes cover nearly every failure.

A wrong client id or secret, a principal without a role that carries CATALOG_MANAGE_CONTENT on the catalog, or a wrong Polaris Catalog Name. Run the token check from the catalog setup; it returns the same error Polaris gives OLake Go.

How this compares with other ways to move MongoDB data to ClickHouse

Method	Freshness	Updates and deletes	Where the data lands	What you run
ClickPipes for MongoDB (ClickHouse Cloud)	CDC through change streams	Yes, into ReplacingMergeTree tables	Native ClickHouse Cloud tables, documents as JSON; ClickHouse is the only reader	Nothing to run; ClickHouse Cloud only, billed per ClickPipe
ClickHouse MongoDB table engine	Live, per query	Reads the current document	Nothing is copied; every query hits MongoDB	A MongoDB user ClickHouse can log in with
Debezium, Kafka and the ClickHouse Kafka engine	Seconds	Yes, with your own merge logic	Kafka topics, then ClickHouse tables	Kafka, Kafka Connect, Debezium and a materialized view per collection
Managed ELT (Fivetran, Estuary, Airbyte Cloud)	Minutes, by plan	Varies by tool	ClickHouse tables, a second copy you pay to move and store	Nothing; priced per row, per GB or per connector
OLake Go to Iceberg on Polaris (this guide)	CDC on a schedule down to one minute	Yes, as Iceberg row-level deletes	Apache Iceberg tables that ClickHouse, DuckDB, Trino and Spark all read	One Docker Compose stack, open source; your bucket plus your own compute

ClickPipes is the simpler choice when you are on ClickHouse Cloud, ClickHouse is the only reader and you want MergeTree query speed. Writing Iceberg once avoids paying to move and store a second copy. That matters as soon as another engine needs the same rows, or you want to own the files and the catalog.

When not to use this

• Every query must hit MergeTree speed. A DataLakeCatalog query reads Parquet with no primary index. Keep the Iceberg tables as the source and load the hot slice into a MergeTree table on a schedule.
• You need the documents kept whole. OLake Go flattens one level; deeply nested documents arrive as JSON text in one column. If every query starts with JSONExtract, ClickHouse's own JSON column type on a MergeTree table may serve you better.
• You need deletion vectors on ClickHouse 26.8. The LTS release reads positional and equality deletes only; keep the upsert type OLake Go resolves, or move to a release that reads Iceberg v3.

FAQs

How do I replicate MongoDB to ClickHouse without ClickPipes or the MongoDB table engine?

Run OLake Go (open source, Apache 2.0) against a MongoDB replica set and point it at an Apache Polaris catalog. The first sync copies each collection in full, every later sync replays only the changes from a change stream, and ClickHouse reads the result as Iceberg tables through its DataLakeCatalog engine. No Kafka, no Debezium and no ClickHouse Cloud account sit in the path.

Does ClickHouse read Iceberg tables with row-level deletes?

Yes. ClickHouse reads positional deletes, and equality deletes from 25.8. The run on this page used equality deletes, which OLake Go chose because ClickHouse, DuckDB and Trino all read them, and ClickHouse 26.8 returned one row per key after the updates. Deletion vectors (Iceberg v3) are not read by the 26.8 LTS release.

Does OLake Go copy the existing collections before it starts reading the change stream?

Yes, automatically. A job in Full Refresh + CDC mode opens the change stream, copies every selected collection in parallel chunks, then replays the stream from the token it saved. Nothing written during the copy is lost, and there is no separate backfill job.

How do nested documents arrive in ClickHouse?

Flattened one level. Top-level fields become columns; nested objects and arrays arrive as JSON text in a String column, which ClickHouse parses with JSONExtract. Decimal128 values also arrive as String, so cast them with toDecimal64 before summing. ObjectId keys arrive as their 24-character hex string.

Why Apache Polaris rather than a ClickHouse table?

A Polaris catalog is one open endpoint that ClickHouse, DuckDB, Trino and Spark all read, with its own access control and vended storage credentials. Loading into a ClickHouse MergeTree table is faster to query but makes ClickHouse the only reader, and a second engine means a second copy. Keep the Iceberg tables as the shared source and load hot slices into MergeTree when a dashboard needs it.

Does MongoDB CDC with OLake Go work on Atlas?

Yes. Atlas clusters are already replica sets with the oplog on. Create a database user with the read role on your database plus clusterMonitor, allow OLake Go's address in the Atlas network list, and paste the SRV connection string into the source. Set the Atlas minimum oplog window to cover your longest pause between syncs.

What does OLake Go need on the MongoDB side?

A replica set (a single node is fine), oplog retention longer than your longest pause between syncs, and a user with read on the database plus clusterMonitor or readAnyDatabase in admin for oplog access. Step 1 has the mongosh commands.

How we tested this

• Run on 2026-10-29 with OLake Go v1.1.0.
• Read back with ClickHouse 26.8.36 (LTS), DataLakeCatalog engine, DuckDB 1.5.5, Trino 484.
• A 2-collection MongoDB 7 replica set (8 customers, 40 orders) synced with Full Refresh + CDC and equality deletes as the Iceberg update/delete format into Apache Polaris 1.8.0 on MinIO, then 7 inserts and 5 updates replayed through a second sync. ClickHouse, DuckDB and Trino returned identical rows.

Related guides

MongoDB to SnowflakeSame source, read from Snowflake.MongoDB to BigQuerySame source, read from BigQuery.MongoDB to Amazon RedshiftSame source, read from Amazon Redshift.MongoDB to DatabricksSame source, read from Databricks.MongoDB to Apache IcebergSame source, read from Apache Iceberg.Postgres to ClickHouseSame engine, replicated from Postgres.MySQL to ClickHouseSame engine, replicated from MySQL.SQL Server to ClickHouseSame engine, replicated from SQL Server.DB2 to ClickHouseSame engine, replicated from DB2.Kafka to ClickHouseSame engine, replicated from Kafka.MongoDB connector referenceEvery field, sync mode and troubleshooting note for the source.Query engine compatibilityWhich engines read which catalogs and delete formats.All integrationsEvery source and engine pair with a tested guide.

Deeper reading on this site:

• Equality vs positional deletes vs deletion vectors
• Which engines read which delete formats
• OLake Go ingestion benchmarks

Start with your own MongoDB

The quickstart runs OLake UI with Docker Compose. Create the source, the Polaris destination and the job as above, and the first sync lands your collections where ClickHouse can read them.

QuickstartJoin Slack