Skip to main content

MongoDB to Snowflake: CDC via Apache Iceberg

MongoDB to Snowflake comes down to one choice. Copy the documents into tables Snowflake owns, or write them once as Apache Iceberg tables that Snowflake and every other engine can read. This guide does the second.

OLake Go reads MongoDB changes through change streams and commits them as Iceberg tables into a catalog Snowflake reads, one column per top-level field. With Snowflake's own Horizon catalog, the default on this page, the files sit on Snowflake-managed storage.

Nothing else is in the path: no Kafka, no Debezium, no Spark. With AWS Glue, the files sit in your S3 bucket and Snowflake reads them through a catalog integration.

By Shubham Satish BaldavaUpdated October 29, 2026

Key takeaways

  • OLake Go replicates MongoDB collections into Snowflake as Iceberg tables: a full historical snapshot first, then change data capture (CDC) from a change stream, with no Kafka or Debezium in between.
  • Through the Horizon catalog the tables land as Snowflake-managed Iceberg tables, flattened one level with a column per field. You query them in Snowsight right away, with no CREATE ICEBERG TABLE, no external volume and no refresh.
  • The same tables, written once by OLake Go, are also queryable from DuckDB and Spark through the same catalog. In the run on this page all three engines returned identical rows.

The short answer. To replicate MongoDB to Snowflake with change data capture (CDC), run OLake Go against a MongoDB replica set. Point it at the Snowflake Horizon catalog as the Iceberg destination.

The first sync copies each collection in full (the historical snapshot); every later sync replays inserts, updates and deletes from the change stream. Snowflake, DuckDB and Spark then query the same Iceberg tables, with no second copy.

How MongoDB CDC reaches Snowflake here​

Two things happen in order, and OLake Go does both on its own. The first sync opens a change stream and keeps its resume token. It then reads each collection in parallel chunks, split on _id, and writes Parquet data files: that is the historical load.

Every later sync resumes the change stream from the saved token and replays the inserts, updates and deletes since. Nothing written during the snapshot is lost, and there is no separate backfill job.

Each sync ends with one commit to Horizon. OLake Go asks Horizon for short-lived storage credentials, uploads the data and delete files, then posts the new snapshot to the catalog. Snowflake sees it the moment the commit lands; DuckDB and Spark see it on their next catalog load.

One thing to keep an eye on: the oplog is MongoDB's replay window. If the resume token has fallen out of the oplog by the time the next sync runs, OLake Go cannot resume and needs a fresh snapshot.

The source test reports the window and warns below 7 days. The single-node container on this page had 1.3 hours: fine for a demo, not for production.

What you need​

  • MongoDB 4.0 or later running as a replica set, self-hosted or on Atlas, with a user that can read the database and the oplog. The collections on this page ran on MongoDB 7 as a single-node replica set; step 1 covers Atlas too.
  • A Snowflake account where you can run the one-time setup as ACCOUNTADMIN. Snowflake-managed storage means no cloud bucket or IAM role on your side.
  • OLake UI from the Docker Compose install, with the OLake Go MongoDB source on v0.11 or later. The run on this page used v1.1.0.
  • Docker, and about 30 minutes.

Step 1: Prepare MongoDB for CDC​

Most readers point OLake Go at collections they already have. If you want to follow along with the exact data from this page, create these two collections first:

The two sample collections used on this page
mongosh, database shop
const now = new Date();
db.customers.insertMany([
  { _id: 1, name: 'Aarav Mehta', email: 'aarav@example.com', city: 'Mumbai', tier: 'gold', created_at: now, updated_at: now },
  // ... 8 customers with integer _id values
]);
db.orders.insertMany([
  { _id: 1, customer_id: 2, product: 'Headphones', amount: NumberDecimal('99.19'), status: 'shipped', created_at: now, updated_at: now },
  // ... 40 orders
]);
// Then insert whatever you like. This page started with 8 customers and 40 orders.

OLake Go reads MongoDB changes through change streams, which MongoDB builds on the replica set oplog. Three things must be in place before the first sync.

SettingValueWhy it matters
Replica setany name, even a single nodeChange streams and the oplog exist only in replica set mode. Atlas clusters are already replica sets.
Oplog retentionlong enough to cover your longest pauseOLake Go resumes from a change-stream token saved after each sync; if the oplog has rolled past it, the next sync needs a fresh snapshot.
A user for OLake Goread on the database plus clusterMonitorThe snapshot reads the collections; the change stream needs oplog access, which clusterMonitor (or readAnyDatabase) grants.
mongosh, on the primary
// 1. A single-node replica set, if you do not have one (self-hosted only)
//    start mongod with --replSet rs0, then:
rs.initiate({ _id: "rs0", members: [{ _id: 0, host: "<host>:27017" }] })

// 2. A user for OLake Go
use admin
db.createUser({
  user: "olake", pwd: "<password>",
  roles: [{ role: "read", db: "shop" }, { role: "clusterMonitor", db: "admin" }]
})

// 3. Check the oplog window; OLake Go warns below 7 days
use local
db.oplog.rs.stats().maxSize      // bytes; raise with replSetResizeOplog if it is small

Check the result before moving on:

javascript
rs.status().members.map(m => m.stateStr)      // one PRIMARY
db.getSiblingDB("admin").getUsers().users.map(u => u.user)

On MongoDB Atlas the replica set and oplog are already there; create the user under Database Access with the read role on your database and clusterMonitor, and allow OLake Go's IP in Network Access. The full guide, including Atlas screenshots, is MongoDB and Atlas CDC setup.

The oplog is your replay window

Atlas and most self-hosted installs keep 24 hours or less of oplog by default. OLake Go's source test shows the current window and warns below 7 days. Size it for your longest planned pause, or run the job often enough that the token never falls out of the window.

Step 2: Set up the catalog in Snowflake​

Snowflake reads Iceberg tables from two kinds of catalog. Horizon is Snowflake's own: OLake Go writes straight into it, the tables are Snowflake-managed, and they appear in Snowsight the moment a sync commits.

AWS Glue is a catalog outside Snowflake, so the files stay in your S3 bucket. Snowflake reaches them through a catalog integration: a read-only link to that external catalog, refreshed after each sync.

Horizon is the default on this page. Glue is the right pick when Athena, Redshift or Spark on Amazon Elastic MapReduce (EMR) will read the same tables. It keeps the files in your own bucket.

CatalogOLake Go writesSnowflake readsVerified on this page
AWS GlueYesYes: catalog integration (read-only)Documented, not yet run by us
Snowflake HorizonYesYes: native: tables are Snowflake-managed Iceberg tablesYes, end to end

Snowflake Horizon is the catalog built into every Snowflake account. OLake Go talks to it through the Iceberg REST endpoint at https://<account_identifier>.snowflakecomputing.com/polaris/api/catalog, and the tables it creates are Snowflake-managed Iceberg tables. Writing through this endpoint needs OLake Go 0.11.0 or later.

Run the following once in a Snowsight worksheet as ACCOUNTADMIN. It creates a database for the tables, a role OLake Go writes as, a service user that holds the token, and the authentication policy the token needs.

Snowflake, once per account
USE ROLE ACCOUNTADMIN;

-- A small warehouse for your own queries, and the database OLake Go writes into.
CREATE WAREHOUSE IF NOT EXISTS OLAKE_WH WAREHOUSE_SIZE = XSMALL AUTO_SUSPEND = 60 AUTO_RESUME = TRUE;
CREATE DATABASE IF NOT EXISTS OLAKE_DEMO;
ALTER DATABASE OLAKE_DEMO SET CATALOG = 'SNOWFLAKE';
ALTER DATABASE OLAKE_DEMO SET EXTERNAL_VOLUME = 'SNOWFLAKE_MANAGED';

-- The role OLake Go uses. It needs to create schemas; tables are created inside them.
CREATE ROLE IF NOT EXISTS OLAKE_ROLE;
GRANT USAGE, CREATE SCHEMA ON DATABASE OLAKE_DEMO TO ROLE OLAKE_ROLE;
GRANT USAGE ON WAREHOUSE OLAKE_WH TO ROLE OLAKE_ROLE;
GRANT ROLE OLAKE_ROLE TO ROLE SYSADMIN;

-- A service user for OLake Go. It holds the programmatic access token.
CREATE USER IF NOT EXISTS OLAKE_SVC TYPE = SERVICE DEFAULT_ROLE = OLAKE_ROLE;
GRANT ROLE OLAKE_ROLE TO USER OLAKE_SVC;

-- Snowflake issues a token to a service user only under a network policy, unless an
-- authentication policy waives that. This one waives it. Any schema can hold the policy.
USE SCHEMA OLAKE_DEMO.PUBLIC;
CREATE AUTHENTICATION POLICY IF NOT EXISTS olake_pat_policy
  PAT_POLICY = (NETWORK_POLICY_EVALUATION = ENFORCED_NOT_REQUIRED);
ALTER USER OLAKE_SVC SET AUTHENTICATION POLICY olake_pat_policy;

-- The two values OLake Go asks for.
SELECT CURRENT_ORGANIZATION_NAME() || '-' || CURRENT_ACCOUNT_NAME() AS account_identifier,
       CURRENT_REGION() AS region;

-- The token. Snowflake shows the secret once; copy it straight into OLake Go.
ALTER USER OLAKE_SVC ADD PROGRAMMATIC ACCESS TOKEN OLAKE_PAT
  ROLE_RESTRICTION = 'OLAKE_ROLE' DAYS_TO_EXPIRY = 30;

What each block is for:

  • CATALOG = 'SNOWFLAKE' and EXTERNAL_VOLUME = 'SNOWFLAKE_MANAGED' on the database make every table OLake Go creates a Snowflake-managed Iceberg table on Snowflake's own storage. No S3 bucket, IAM role or external volume on your side. To keep the files in your own bucket, create an external volume and name it here instead.
  • The authentication policy is the step people miss. Without it, the ADD PROGRAMMATIC ACCESS TOKEN line fails with Network Policy is required when creating a programmatic access token for user OLAKE_SVC. In production, attach a network policy that allows OLake Go's egress IP instead of waiving the check.
  • ROLE_RESTRICTION pins the token to OLAKE_ROLE, so a leaked token cannot act as ACCOUNTADMIN. Tokens expire after DAYS_TO_EXPIRY (15 by default, 365 at most). Rotate it before then or the sync fails with an authentication error.

You can check the token outside OLake Go by exchanging it for a session token, the same call OLake Go makes on every sync:

Token check
curl -s -X POST "https://<account_identifier>.snowflakecomputing.com/polaris/api/catalog/v1/oauth/tokens" \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  --data-urlencode grant_type=client_credentials \
  --data-urlencode scope=session:role:OLAKE_ROLE \
  --data-urlencode client_secret=<your PAT>

A good token returns a JSON object with an access_token. The other authentication methods (key-pair JWT, External OAuth, workload identity federation) are on the Horizon catalog page.

Step 3: Connect MongoDB to OLake Go​

  1. In the OLake UI, open Sources and click Create Source. Pick MongoDB as the connector, give the source a name and keep the OLake Go version on the latest release.
  2. Under Change data capture (CDC), leave Enable CDC on.
  3. Under Connection details, add each host as host:port (or one SRV host with Use SRV on for Atlas), then the Database Name, Auth DB (admin), username, password and the Replica Set name. Leave Read Preference on primary unless you have secondaries to spare.
  4. Click Set up source. OLake Go tests the connection, confirms that it can open a change stream, and reports the oplog window with a warning below 7 days.

Here is the source form from the run on this page. MongoDB ran in another container, so the host is host.docker.internal:27017; the replica set is rs0 and the user is the one from step 1.

OLake UI Edit source form for MongoDB on OLake Go v1.1.0 with Enable CDC on, host host.docker.internal:27017, database shop, auth DB admin, user olake, replica set rs0 and read preference primary
The MongoDB source in OLake UI: Enable CDC on, the replica set name and the user from step 1.

Set up source runs the connection test, confirms that a change stream can be opened, and reports the oplog window before anything is saved. The run on this page got a warning here, because the single-node container keeps only 1.3 hours of oplog.

The full field reference, including SRV strings, IAM authentication and SSH tunnels, is on the MongoDB connector page.

Step 4: Configure the Iceberg destination​

  1. Open Destinations and click Create Destination.
  2. Pick Apache Iceberg as the connector and Horizon as the catalog type.
  3. Choose Programmatic Access Token (PAT) as the authentication type and fill in the fields below.
  4. Click Create. OLake Go calls the Horizon endpoint, exchanges the token and saves the destination.
OLake UI destination form for the Snowflake Horizon catalog with the PAT authentication fields

Horizon Configuration Parameters

ParameterSample ValueDescription
Authentication Type
required		Programmatic Access Token (PAT)Authentication method OLake Go uses for Horizon Catalog REST API requests. Set to match your workspace's authentication configuration.
Horizon Iceberg REST Catalog URI
required		https://<account_identifier>.snowflakecomputing.com/polaris/api/catalogSnowflake Horizon Catalog REST API endpoint.
Snowflake Database
required		SNOWThe name of the database in Snowflake.
Session Role Scope
required		session:role:<HORIZON_ROLE>Informs Snowflake which role to use for this Horizon Catalog REST session, which controls privileges for catalog and table operations. Space-separated if providing more than one.
Programmatic Access Token
required		ver:1-hint:...Snowflake programmatic access token (PAT).
Cloud Storage Region
required		AWS_AP_SOUTH_1Snowflake's account region used to route requests to the correct storage endpoint. Run SELECT CURRENT_REGION(); to know the region
Catalog NamehorizonName of the Iceberg catalog OLake Go registers tables under. Defaults to olake_iceberg if left empty.
Enable Arrow WritesfalseWrites data and delete files using Apache Arrow based writer and registers them in Iceberg.
Catalog Name Supported for v0.3.5 and above
For the catalog name, OLake Go only supports lowercase letters and underscores. Spaces and special characters are not supported.

Step 5: Create the job and run the first sync​

  1. Open Jobs and click Create Job. Give it a name and a schedule. Every Day at a fixed hour, or a cron expression, both work; the schedule is how often the change stream is drained, not how often a full copy runs.
  2. Select the source and the destination you created.
  3. Under Advanced Settings, set Target Query Engines to Snowflake, plus DuckDB and Apache Spark if they will read the tables too. OLake Go then resolves an Iceberg delete format every one of those engines can read, so you never pick one by hand.
  4. Click Next. OLake Go discovers the tables and opens the Streams page.
  5. Tick the tables to replicate. For each one, set the sync mode to Full Refresh + CDC and the ingestion mode to Upsert. Leave the Upsert Type as OLake Go resolved it (Positional Deletes in the run on this page).
  6. Click Create Job, then open the job's menu and choose Sync Now for the first run.

Three target query engines were set on this job. That is the only format decision you make: OLake Go works out which Iceberg delete format all three can read and sets it on every stream.

OLake UI Streams Selection page with the shop database, customers and orders selected, sync mode Full Refresh + CDC, ingestion mode Upsert and upsert type Positional Deletes on Iceberg format version V2
Each stream: Full Refresh + CDC, Upsert, and the upsert type OLake Go resolved (Positional Deletes in this run). The Iceberg DB name at the top is editable.
OLake UI Job Logs and History for mongodb_to_snowflake showing two completed syncs, the first in 30 seconds and the second in 15 seconds
Job history: the 30 s historical snapshot, then the 15 s change-only sync.

On this page the first run copied 8 customers and 40 orders (48 documents) and finished in 30 seconds, container start included. OLake Go names the destination schema after the job and the source database, so mongodb_to_snowflake on database shop produced mongodb_to_snowflake_shop. The Iceberg DB field at the top of the Streams page changes it before the first sync.

Step 6: Query the tables in Snowflake​

There is no registration step. The schema and tables OLake Go committed through Horizon are already Snowflake-managed Iceberg tables in the database you configured. One thing to know before the first query: OLake Go creates them with quoted, lowercase names, so Snowflake needs the double quotes.

Snowsight
-- What OLake Go created (schema = <job name>_<source database>[_<source schema>])
SHOW ICEBERG TABLES IN SCHEMA OLAKE_DEMO."mongodb_to_snowflake_shop";

-- Quoted identifiers: the names are lowercase, as they were in the source
SELECT "_id", "name", "city", "tier"
FROM OLAKE_DEMO."mongodb_to_snowflake_shop"."customers"
ORDER BY "_id";

SHOW ICEBERG TABLES reports catalog_name = SNOWFLAKE, iceberg_table_type = MANAGED and external_volume_name = SNOWFLAKE_MANAGED: Snowflake treats them as its own managed Iceberg tables, not as external tables that need a refresh.

OLake Go adds four columns to every table: _olake_id (a hash of the primary key), _op_type, _olake_timestamp and _cdc_timestamp. _op_type is r for a row from the first snapshot, c for an insert, u for an update and d for a delete. Filter them out of dashboards, or use them to see when a row last changed.

One MongoDB-specific detail: the primary key column is _id, as in the documents, and OLake Go adds _cdc_resume_token next to its usual _olake_id, _op_type, _olake_timestamp and _cdc_timestamp columns.

Snowsight SQL file running SHOW ICEBERG TABLES on the mongodb_to_snowflake_shop schema and listing customers and orders with owner OLAKE_ROLE, external volume SNOWFLAKE_MANAGED, catalog SNOWFLAKE and table type MANAGED
SHOW ICEBERG TABLES in Snowsight: both collections are now Snowflake-managed Iceberg tables, created by OLAKE_ROLE through Horizon.
Snowsight query on OLAKE_DEMO.mongodb_to_snowflake_shop.customers returning 10 rows with _id, name, city and tier
The customers collection in Snowsight after the change sync: 10 rows, one column per field.

Step 7: Change documents in MongoDB and sync again​

To see the change path work, the run on this page made 12 changes in MongoDB after the first sync:

Changes made in MongoDB between the two syncs
const now = new Date();
db.customers.insertMany([
  { _id: 9,  name: 'Maya Patel',  email: 'maya@example.com',  city: 'Pune',      tier: 'silver', created_at: now, updated_at: now },
  { _id: 10, name: 'Lucas Silva', email: 'lucas@example.com', city: 'São Paulo', tier: 'bronze', created_at: now, updated_at: now }
]);
db.orders.insertMany([
  { _id: 41, customer_id: 9,  product: 'Laptop',   amount: NumberDecimal('1299.00'), status: 'placed', created_at: now, updated_at: now },
  { _id: 42, customer_id: 9,  product: 'Mouse',    amount: NumberDecimal('25.50'),   status: 'placed', created_at: now, updated_at: now },
  { _id: 43, customer_id: 10, product: 'Monitor',  amount: NumberDecimal('349.99'),  status: 'placed', created_at: now, updated_at: now },
  { _id: 44, customer_id: 3,  product: 'Keyboard', amount: NumberDecimal('89.00'),   status: 'placed', created_at: now, updated_at: now },
  { _id: 45, customer_id: 7,  product: 'Webcam',   amount: NumberDecimal('79.90'),   status: 'placed', created_at: now, updated_at: now }
]);
db.customers.updateOne({ name: 'Liam Carter' }, { $set: { tier: 'gold', updated_at: now } });
db.customers.updateOne({ name: 'Zara Ali' },    { $set: { city: 'Abu Dhabi', updated_at: now } });
db.orders.updateMany({ _id: { $in: [1, 4, 7] } }, { $set: { status: 'delivered', updated_at: now } });

The second sync read 12 change events (1.60 KB) from the stream and finished in 15 seconds end to end. For customers that meant one new data file with 4 rows: 2 inserts plus the new versions of 2 updated documents.

One small delete file retires the 2 old versions. OLake Go's commit report in the sync log shows it directly: addedDataFiles=1, addedRecords=4, addedPositionalDeleteFiles=1, addedPositionalDeletes=2, and orders the same shape with 8 rows and 3 deletes.

OLake UI sync log filtered to the Iceberg commit reports for orders and customers, showing addedDataFiles 1, addedPositionalDeleteFiles 1, addedRecords 8 and 4, totalEqualityDeletes 0
The change-only sync in the OLake UI log: one data file and one positional delete file per collection, zero equality deletes.

Snowflake shows the result with no refresh. Liam Carter is gold, Zara Ali moved to Abu Dhabi, and the two new customers are there. The _op_type column shows how each row arrived:

Snowsight query on customers after the CDC sync showing 10 rows with _op_type r for the snapshot rows, u for Liam Carter and Zara Ali, c for Maya Patel and Lucas Silva
Snowsight after the change sync: the two updates (u), the two inserts (c) and the untouched snapshot rows (r).

Schema changes. MongoDB has no schema, so new fields show up whenever a document carries one. OLake Go adds a column for a new top-level field on the next sync, and Snowflake shows it; documents without the field hold NULL there.

A field whose type changes between documents (a number in one, a string in another) is widened to VARCHAR. Full rules are on the schema evolution page.

Query the same tables from DuckDB and Spark​

This is the point of writing Iceberg instead of loading Snowflake tables. Nothing is exported, copied or refreshed: both engines read the files OLake Go wrote, through the same Horizon endpoint, with credentials vended by Snowflake.

DuckDB attaches Horizon as an Iceberg REST catalog. It wants a bearer token rather than the raw PAT, so exchange the PAT first (the curl call from the catalog setup), then attach:

DuckDB 1.5.5
INSTALL iceberg; LOAD iceberg;
INSTALL httpfs;  LOAD httpfs;

CREATE SECRET horizon_secret (TYPE iceberg, TOKEN '<access_token from /v1/oauth/tokens>');

ATTACH 'OLAKE_DEMO' AS horizon (
TYPE iceberg,
ENDPOINT 'https://<account_identifier>.snowflakecomputing.com/polaris/api/catalog',
SECRET horizon_secret,
ACCESS_DELEGATION_MODE 'vended_credentials'
);

SHOW ALL TABLES;
SELECT _id, name, city, tier FROM horizon.mongodb_to_snowflake_shop.customers ORDER BY _id;

The session token lives about an hour; re-run the exchange and CREATE OR REPLACE SECRET when a query fails with an authorization error. DuckDB reads deletion vectors from duckdb-iceberg 1.5.3 onward, and only attaches REST catalogs backed by S3, S3 Tables or Google Cloud Storage, so pick an AWS- or GCP-hosted Snowflake account for this path.

DuckDB 1.5.5 shell querying horizon.mongodb_to_snowflake_shop.customers and returning the same 10 rows as Snowflake
DuckDB 1.5.5 reading the Snowflake-managed table through Horizon: the same 10 rows.

An orders roll-up matched MongoDB to the cent in all three engines once amount was cast. 16 delivered orders for 8659.68, 18 placed for 10522.26, 11 shipped for 6397.25.

Data types: MongoDB to Iceberg to Snowflake​

OLake Go maps MongoDB's BSON (Binary JSON) types to Iceberg types, and Snowflake maps Iceberg types to its own. Scalars pass through; everything structured arrives as text.

MongoDB typeIceberg (written by OLake Go)Snowflake showsNotes
int, timestampintNUMBER(10,0)
longlongNUMBER(19,0)
doubledoubleFLOAT
booleanbooleanBOOLEAN
datetimestamptz (UTC)TIMESTAMP_LTZ(6)
decimal128stringVARCHARMoney stored as NumberDecimal arrives as text. SUM("amount"::NUMBER(10,2)) gives the exact total; the roll-up on this page used that cast. Store doubles in MongoDB if you want a numeric column without the cast.
objectId, binData, regex, codestringVARCHARAn ObjectId primary key arrives as its 24-character hex string.
object, arraystringVARCHARJSON text. Parse with PARSE_JSON() in Snowflake, or query :field paths after PARSE_JSON.

MongoDB types in the table come from the MongoDB connector reference; the Snowflake side from Snowflake's Iceberg data types page (checked 2026-10-14).

Cost and maintenance​

  • Storage is Snowflake's with SNOWFLAKE_MANAGED: you pay Snowflake storage rates for the Parquet and delete files. Point the database at your own external volume to keep the files in your bucket instead.
  • Compute. OLake Go runs wherever you run Docker; it does not use a Snowflake warehouse to write. Snowflake compute is spent only when you query.
  • Maintenance. Snowflake expires old snapshots and compacts manifests on Snowflake-managed tables at no cost. Automatic data-file compaction is bundled only when Snowflake alone writes the table (Snowflake-managed Iceberg table maintenance). A CDC table that gets a new small file every minute will want compaction from the writer side; OLake Fusion does this for the catalogs OLake Go writes to, and Spark's rewrite_data_files works through the same Horizon endpoint.
  • Oplog window. On Atlas, raise the minimum oplog window in the cluster settings if the job runs less often than once a day; on self-hosted, replSetResizeOplog.
  • Token rotation. The programmatic access token expires (30 days in the setup above, 365 at most). Put the renewal on a calendar; an expired token fails the next sync at the catalog handshake, before any data moves.

Troubleshooting​

1. The source test warns about the oplog​

oplog_retention: current 1.3 hours, recommended >= 7 days

Not a blocker, but it is the window in which the next sync must start. The fix is on the MongoDB side: replSetResizeOplog on self-hosted, or the minimum oplog window setting on Atlas. Until then, run the job more often than the window.

2. The source test cannot open a change stream​

change_streams: OLake cannot open a change stream, so CDC cannot run.

Either mongod is not running with --replSet, or the user lacks oplog access. Check rs.status() on the server and give the OLake Go user clusterMonitor (or readAnyDatabase) in admin on top of read on the database.

3. Snowflake cannot find the table​

SQL compilation error:
Object 'OLAKE_DEMO.MONGODB_TO_SNOWFLAKE_SHOP.CUSTOMERS' does not exist or not authorized.

Snowflake upper-cased an unquoted identifier. OLake Go creates the schema and tables with their lowercase MongoDB names, so quote them: OLAKE_DEMO."mongodb_to_snowflake_shop"."customers".

4. A money column sums to text​

Numeric value 'abc' is not recognized

decimal128 fields land as VARCHAR. Cast them in the query ("amount"::NUMBER(12,2)), or store the value as a double in MongoDB. A row with a non-numeric string in that field breaks the cast; find it with TRY_TO_NUMBER("amount") IS NULL.

5. The destination test fails at the catalog handshake​

OLake Go's destination test exchanges the token at /v1/oauth/tokens before anything else. Three causes cover nearly every failure: a service user without a network or authentication policy, a scope that does not match the token's ROLE_RESTRICTION, or an expired token. Run the curl exchange from the catalog setup; it returns the same error Snowflake gives OLake Go.

How this compares with other ways to move MongoDB data to Snowflake
MethodFreshnessUpdates and deletesWhere the data landsWhat you run
Snowflake Openflow connector for MongoDBCDC through change streamsYesNative Snowflake tables, documents as VARIANT; Snowflake is the only readerAn Openflow deployment in Snowpark Container Services or your own AWS account, billed in Snowflake credits
Debezium, Kafka and the Snowflake Kafka connectorSecondsYes, with the connector's tombstone handlingKafka topics, then Snowflake tablesKafka, Kafka Connect, Debezium and a schema registry
Managed ELT (Fivetran, Estuary, Airbyte Cloud)Minutes, by planYesSnowflake tables, a second copy you pay to move and storeNothing; priced per row, per GB or per connector
Manual export (mongoexport to JSON, then COPY INTO a VARIANT column)Hours to dailyNo; full reloads onlyFiles you controlA script and a scheduler
OLake Go to Iceberg on Horizon (this guide)CDC on a schedule down to one minuteYes, as Iceberg row-level deletesApache Iceberg tables that Snowflake, DuckDB, Spark and Trino all readOne Docker Compose stack, open source; Snowflake storage plus your own compute

Openflow is the simpler choice when Snowflake is the only reader and you want Snowflake to run the pipeline. Writing Iceberg once avoids paying to move and store a second copy as soon as another engine needs the same rows, or you want to own the files.

When not to use this​

  • Your Snowflake account is on Azure and DuckDB is a reader. DuckDB attaches REST catalogs backed by S3, S3 Tables and Google Cloud Storage only, so pick Spark or Trino as the second engine there.
  • You need the documents kept whole. OLake Go flattens one level; deeply nested documents arrive as JSON text in one column. If every query starts with PARSE_JSON, a VARIANT landing table in Snowflake may serve you better.

FAQs​

How do I replicate MongoDB to Snowflake without a managed connector?

Run OLake Go (open source, Apache 2.0) against a MongoDB replica set and point it at Snowflake's Horizon catalog. The first sync copies each collection in parallel chunks, every later sync replays only the changes from a change stream, and the collections appear in Snowflake as Snowflake-managed Iceberg tables with one column per top-level field. No Kafka, Debezium or Snowflake connector sits in the path.

Does OLake Go copy the existing collections before it starts reading change streams?

Yes, automatically. A job in Full Refresh + CDC mode opens the change stream, copies every selected collection in parallel chunks, then replays the changes from the stream's resume token. Nothing written during the copy is lost, and there is no separate backfill job.

How do MongoDB documents become Snowflake columns?

OLake Go flattens each document one level: every top-level field becomes a column, and nested objects and arrays arrive as JSON text in a VARCHAR column. Parse them in Snowflake with PARSE_JSON. Fields that appear later in some documents are added as new columns on the next sync.

What is the best way to move MongoDB data to Snowflake?

Snowflake Openflow has a MongoDB connector that lands native Snowflake tables when Snowflake is the only reader. Fivetran, Estuary and Airbyte fit teams that want many connectors under one bill. OLake Go fits when the collections should also be readable from DuckDB, Spark or Trino, and in OLake's published ingestion benchmarks it posted higher snapshot and CDC throughput than the other tools measured.

Does this work with MongoDB Atlas?

Yes. Atlas clusters are already replica sets with the oplog on. Create a database user with the read role on your database plus clusterMonitor, allow OLake Go's IP in Network Access, and use the SRV connection string with Use SRV on in the source form.

Which MongoDB types change on the way to Snowflake?

int, long, double, boolean and date map to NUMBER, FLOAT, BOOLEAN and TIMESTAMP_LTZ. Decimal128, ObjectId, binary, objects and arrays arrive as VARCHAR, so cast Decimal128 money fields before summing them. OLake Go also adds a _cdc_resume_token column that records the change-stream position each row came from.

Can I use AWS Glue instead of the Horizon catalog?

Yes. With Glue as the catalog the files stay in your S3 bucket, Athena, Redshift and EMR read them natively, and Snowflake reads them through a catalog integration as read-only tables you refresh after each sync. Horizon is the better default when Snowflake is the main reader, because the tables are Snowflake-managed and need no refresh.

MongoDB to ClickHouseSame source, read from ClickHouse.MongoDB to BigQuerySame source, read from BigQuery.MongoDB to Amazon RedshiftSame source, read from Amazon Redshift.MongoDB to DatabricksSame source, read from Databricks.MongoDB to Apache IcebergSame source, read from Apache Iceberg.Postgres to SnowflakeSame engine, replicated from Postgres.MySQL to SnowflakeSame engine, replicated from MySQL.Oracle to SnowflakeSame engine, replicated from Oracle.SQL Server to SnowflakeSame engine, replicated from SQL Server.DB2 to SnowflakeSame engine, replicated from DB2.MariaDB to SnowflakeSame engine, replicated from MariaDB.Kafka to SnowflakeSame engine, replicated from Kafka.S3 to SnowflakeSame engine, replicated from S3.MongoDB connector referenceEvery field, sync mode and troubleshooting note for the source.Query engine compatibilityWhich engines read which catalogs and delete formats.All integrationsEvery source and engine pair with a tested guide.

Deeper reading on this site: