Skip to main content

MongoDB to Databricks: CDC via Apache Iceberg

MongoDB to Databricks usually means a Debezium and Kafka pipeline, or nightly mongoexport files picked up by Auto Loader. This guide takes a different route. OLake Go reads MongoDB changes through change streams and commits them as Apache Iceberg tables into Unity Catalog, one column per top-level field.

The collections land as Unity Catalog managed Iceberg tables, with the files in the catalog's managed storage. Nothing else is in the path: no Kafka, no Debezium, no Databricks ingestion pipeline. DuckDB and Spark read the same tables through the same catalog.

By Shubham Satish BaldavaUpdated October 30, 2026

Key takeaways

  • OLake Go replicates MongoDB collections into Databricks as Iceberg tables: a full historical snapshot first, then change data capture (CDC) from a change stream, with no Kafka or Debezium in between.
  • The tables are Unity Catalog managed Iceberg tables, flattened one level with a column per field. You query them from the Databricks SQL editor or a notebook right away, with no external table and no refresh.
  • The same tables, written once by OLake Go, are also queryable from DuckDB and Spark through Unity Catalog. In the run on this page all three engines returned identical rows.

The short answer. To replicate MongoDB to Databricks with change data capture (CDC), run OLake Go against a MongoDB replica set with Unity Catalog as the Iceberg destination and Databricks as a target query engine. The first sync copies each collection in full (the historical snapshot), and every later sync replays inserts, updates and deletes from the change stream. Databricks, DuckDB and Spark then query the same Iceberg tables, with no second copy.

How MongoDB CDC reaches Databricks here​

Two things happen in order, and OLake Go does both on its own. The first sync opens a change stream and keeps its resume token, then reads each collection in parallel chunks split on _id and writes Parquet data files: that is the historical load. Every later sync resumes the change stream from the saved token and replays the inserts, updates and deletes since.

Nothing written during the snapshot is lost, and there is no separate backfill job. An update or delete retires the old row with a deletion vector, a small bitmap per data file that marks the replaced positions.

Each sync ends with one commit to Unity Catalog. Databricks sees the new snapshot on its next query, and DuckDB and Spark see it on their next catalog load.

One thing to keep an eye on: the oplog is MongoDB's replay window. If the resume token has fallen out of the oplog by the time the next sync runs, OLake Go cannot resume and needs a fresh snapshot. The source test reports the window and warns below 7 days.

What you need​

  • MongoDB 4.0 or later running as a replica set, self-hosted or on Atlas, with a user that can read the database and the oplog. The collections on this page ran on MongoDB 7 as a single-node replica set; step 1 covers Atlas too.
  • A Databricks workspace with Unity Catalog, a catalog with managed storage, and an admin who can turn on external data access and grant EXTERNAL USE SCHEMA. The run on this page used a workspace on Google Cloud.
  • OLake UI from the Docker Compose install. The run on this page used OLake Go v1.1.0.
  • Docker, and about 30 minutes.

Step 1: Prepare MongoDB for CDC​

Most readers point OLake Go at collections they already have. If you want to follow along with the exact data from this page, create these two collections first:

The two sample collections used on this page
mongosh, database shop
const now = new Date();
db.customers.insertMany([
  { _id: 1, name: 'Aarav Mehta', email: 'aarav@example.com', city: 'Mumbai', tier: 'gold', created_at: now, updated_at: now },
  // ... 8 customers with integer _id values
]);
db.orders.insertMany([
  { _id: 1, customer_id: 2, product: 'Headphones', amount: NumberDecimal('99.19'), status: 'shipped', created_at: now, updated_at: now },
  // ... 40 orders
]);
// Then insert whatever you like. This page started with 8 customers and 40 orders.

OLake Go reads MongoDB changes through change streams, which MongoDB builds on the replica set oplog. Three things must be in place before the first sync.

SettingValueWhy it matters
Replica setany name, even a single nodeChange streams and the oplog exist only in replica set mode. Atlas clusters are already replica sets.
Oplog retentionlong enough to cover your longest pauseOLake Go resumes from a change-stream token saved after each sync; if the oplog has rolled past it, the next sync needs a fresh snapshot.
A user for OLake Goread on the database plus clusterMonitorThe snapshot reads the collections; the change stream needs oplog access, which clusterMonitor (or readAnyDatabase) grants.
mongosh, on the primary
// 1. A single-node replica set, if you do not have one (self-hosted only)
//    start mongod with --replSet rs0, then:
rs.initiate({ _id: "rs0", members: [{ _id: 0, host: "<host>:27017" }] })

// 2. A user for OLake Go
use admin
db.createUser({
  user: "olake", pwd: "<password>",
  roles: [{ role: "read", db: "shop" }, { role: "clusterMonitor", db: "admin" }]
})

// 3. Check the oplog window; OLake Go warns below 7 days
use local
db.oplog.rs.stats().maxSize      // bytes; raise with replSetResizeOplog if it is small

Check the result before moving on:

javascript
rs.status().members.map(m => m.stateStr)      // one PRIMARY
db.getSiblingDB("admin").getUsers().users.map(u => u.user)

On MongoDB Atlas the replica set and oplog are already there; create the user under Database Access with the read role on your database and clusterMonitor, and allow OLake Go's IP in Network Access. The full guide, including Atlas screenshots, is MongoDB and Atlas CDC setup.

The oplog is your replay window

Atlas and most self-hosted installs keep 24 hours or less of oplog by default. OLake Go's source test shows the current window and warns below 7 days. Size it for your longest planned pause, or run the job often enough that the token never falls out of the window.

Step 2: Set up Unity Catalog in Databricks​

Databricks reads Iceberg tables through Unity Catalog, the catalog built into every Databricks workspace. OLake Go writes straight into it, so the tables become Unity Catalog managed Iceberg tables. They show up in Catalog Explorer and the SQL editor the moment a sync commits.

The files sit in the catalog's managed storage, which is backed by an external location in your own cloud account. DuckDB and Spark reach the same tables through the same Unity Catalog endpoint.

CatalogOLake Go writesDatabricks readsVerified on this page
Unity CatalogYesYes: native: Unity Catalog managed Iceberg tablesYes, end to end

Unity Catalog is the catalog built into every Databricks workspace. OLake Go writes to it through Unity Catalog's Iceberg REST (representational state transfer) endpoint at https://<workspace-url>/api/2.1/unity-catalog/iceberg-rest, and the tables it creates are Unity Catalog managed Iceberg tables. Databricks allows writes through this endpoint for managed Iceberg tables only (Databricks: access tables from Iceberg clients, checked 2026-10-30).

Four things must be in place in the workspace, once:

SettingWhereWhy it matters
External data access on the metastoreCatalog, the gear icon, Metastore, then External data accessWithout it the Iceberg REST endpoint refuses every client that is not Databricks itself.
A catalog with managed storageCatalog, Create catalog, with a storage location backed by an external locationOLake Go creates one schema per job inside it. The catalog name is the warehouse value every client uses.
EXTERNAL USE SCHEMA for the writing principalOn the catalog, so it covers the schemas OLake Go createsUnity Catalog checks this privilege, on top of the usual ones, before it serves a table to an outside engine.
A personal access token or a service principalSettings, Developer, Access tokensOLake Go sends it as a bearer token on every catalog call.
Databricks SQL editor, once per catalog (replace the principal)
-- The catalog OLake Go writes into. Its managed storage holds the Iceberg files.
CREATE CATALOG IF NOT EXISTS lakehouse MANAGED LOCATION '<external location URL>/lakehouse';

-- What the writing principal needs: browse and create schemas, then read and write tables in them
GRANT USE CATALOG, CREATE SCHEMA ON CATALOG lakehouse TO `olake@example.com`;
GRANT USE SCHEMA, CREATE TABLE, SELECT, MODIFY ON CATALOG lakehouse TO `olake@example.com`;
GRANT EXTERNAL USE SCHEMA ON CATALOG lakehouse TO `olake@example.com`;

Granting on the catalog makes the privileges inherit to every schema OLake Go creates later. EXTERNAL USE SCHEMA is not part of ALL PRIVILEGES, and only the catalog owner can grant it (Databricks: Unity Catalog privileges, checked 2026-10-30). A service principal with OAuth machine-to-machine credentials is the better choice in production. The other authentication methods are on the Unity Catalog destination page.

Databricks reads deletion vectors only

Unity Catalog accepts updates to a managed Iceberg table only as copy-on-write files or as Iceberg v3 deletion vectors. It rejects equality delete files with Can not create a Path from a null string [ErrorCode: 2015]. Set Databricks as a target query engine on the job, and OLake Go writes deletion vectors from the first sync.

Step 3: Connect MongoDB to OLake Go​

  1. In the OLake UI, open Sources and click Create Source. Pick MongoDB as the connector, give the source a name and keep the OLake Go version on the latest release.
  2. Under Change data capture (CDC), leave Enable CDC on.
  3. Under Connection details, add each host as host:port (or one SRV host with Use SRV on for Atlas), then the Database Name, Auth DB (admin), username, password and the Replica Set name. Leave Read Preference on primary unless you have secondaries to spare.
  4. Click Set up source. OLake Go tests the connection, confirms that it can open a change stream, and reports the oplog window with a warning below 7 days.

Here is the source form from the run on this page. MongoDB ran in another container, so the host is host.docker.internal:27017; the replica set is rs0 and the user is the one from step 1.

OLake UI Edit source form for MongoDB on OLake Go v1.1.0 with Enable CDC on, host host.docker.internal:27017, database shop, auth DB admin, user olake, replica set rs0 and read preference primary
The MongoDB source in OLake UI: Enable CDC on, the replica set name and the user from step 1.

Set up source runs the connection test, confirms that a change stream can be opened, and reports the oplog window before anything is saved. A single-node test container keeps a short oplog, so expect a warning there.

The full field reference, including SRV strings, IAM authentication and SSH tunnels, is on the MongoDB connector page.

Step 4: Configure the Iceberg destination​

  1. Open Destinations and click Create Destination.
  2. Pick Apache Iceberg as the connector and Unity as the catalog type.
  3. Choose Personal Access Token as the authentication type and fill in the fields below. The catalog name from the setup goes into the catalog field.
  4. Click Create. OLake Go calls the Unity Catalog endpoint with the token and saves the destination.
OLake UI destination form for an Apache Iceberg destination on the Unity catalog type with the personal access token fields

The run on this page wrote into a catalog called datazip, with a personal access token that expires after seven days. The snippets below use lakehouse; replace it with your catalog name.

Unity Configuration Parameters

ParameterSample ValueDescription
REST Catalog URL
required		https://1234567890123456.1.gcp.databricks.com/Databricks workspace URL with Unity Catalog REST API endpoint. Use your actual workspace URL.
Authentication Type
required		Personal Access TokenAuthentication method OLake Go uses for Unity Catalog REST API requests. Set to match your workspace's authentication configuration.
Unity Catalog Name
required		workspaceName of the catalog in Unity.
Token
required		dapi123456789...Databricks Personal Access Token
Catalog Nameolake_icebergName of the Iceberg catalog OLake Go registers tables under. Defaults to olake_iceberg if left empty.
Enable Arrow WritesfalseWrites data and delete files using Apache Arrow based writer and registers them in Iceberg.
Catalog Name Supported for v0.3.5 and above
For the catalog name, OLake Go only supports lowercase letters and underscores. Spaces and special characters are not supported.

Step 5: Create the job and run the first sync​

  1. Open Jobs and click Create Job. Give it a name and a schedule. Every Day at a fixed hour, or a cron expression, both work; the schedule is how often the change stream is drained, not how often a full copy runs.
  2. Select the source and the destination you created.
  3. Under Advanced Settings, set Target Query Engines to Databricks, plus DuckDB and Apache Spark if they will read the tables too. OLake Go then resolves an Iceberg delete format every one of those engines can read, so you never pick one by hand.
  4. Click Next. OLake Go discovers the collections and opens the Streams page.
  5. Tick the collections to replicate. For each one, set the sync mode to Full Refresh + CDC and the ingestion mode to Upsert. Leave the Upsert Type as OLake Go resolved it (Deletion Vector in the run on this page).
  6. Click Create Job, then open the job's menu and choose Sync Now for the first run.

Three target query engines were set on this job, and Databricks narrows the delete formats to one: deletion vectors on Iceberg v3 tables. OLake Go sets that format on every stream, so there is nothing to pick.

OLake UI Streams Selection page with the shop database, customers and orders selected, sync mode Full Refresh + CDC, ingestion mode Upsert and upsert type Deletion Vector on Iceberg format version V3
Each stream: Full Refresh + CDC, Upsert, and Deletion Vector on Iceberg v3, the format OLake Go resolved for Databricks.
OLake UI Job Logs and History for mongodb_to_databricks showing two completed syncs, the first in 23 seconds and the second in 16 seconds
Job history: the 23 s historical snapshot, then the 16 s change-only sync.

On this page the first run copied 8 customers and 40 orders (48 documents, 6.40 KB) and finished in 23 seconds, container start included. OLake Go names the Unity Catalog schema after the job and the source database, so mongodb_to_databricks on database shop produced mongodb_to_databricks_shop. The Iceberg DB field at the top of the Streams page changes it before the first sync.

Step 6: Query the tables in Databricks​

There is no registration step. The schema and tables OLake Go committed through Unity Catalog are managed Iceberg tables in the catalog you configured, and the Databricks SQL editor or a notebook reads them by their three-part name.

Databricks SQL editor
-- What OLake Go created (schema = <job name>_<source database>[_<source schema>])
SHOW TABLES IN lakehouse.mongodb_to_databricks_shop;

SELECT _id, name, city, tier
FROM lakehouse.mongodb_to_databricks_shop.customers
ORDER BY _id;

SELECT status, count(*) AS n, round(sum(CAST(amount AS DOUBLE)), 2) AS total
FROM lakehouse.mongodb_to_databricks_shop.orders
GROUP BY status ORDER BY status;

Any SQL warehouse or cluster on a recent Databricks Runtime reads them; a 2X-Small serverless warehouse is enough for checks. DESCRIBE EXTENDED on a table shows the Iceberg provider and the deletion vectors in its properties.

OLake Go adds four columns to every table: _olake_id (a hash of the primary key), _op_type, _olake_timestamp and _cdc_timestamp. _op_type is r for a row from the first snapshot, c for an insert, u for an update and d for a delete. Filter them out of dashboards, or use them to see when a row last changed.

Databricks SQL editor running SELECT on datazip.mongodb_to_databricks_shop.customers on the 2X-Small SQL warehouse, returning 10 rows with id, name, city and tier
The MongoDB customers table in the Databricks SQL editor after both syncs: 10 rows, read through Unity Catalog.

Two MongoDB details show up here. The key column is _id, as in the documents, and amount is a STRING because the source stores it as Decimal128, so the roll-up casts it before summing. OLake Go also adds _cdc_resume_token next to its usual columns.

Step 7: Change documents in MongoDB and sync again​

To see the change path work, the run on this page made 12 changes in MongoDB after the first sync:

Changes made in MongoDB between the two syncs
const now = new Date();
db.customers.insertMany([
  { _id: 9,  name: 'Maya Patel',  email: 'maya@example.com',  city: 'Pune',      tier: 'silver', created_at: now, updated_at: now },
  { _id: 10, name: 'Lucas Silva', email: 'lucas@example.com', city: 'São Paulo', tier: 'bronze', created_at: now, updated_at: now }
]);
db.orders.insertMany([
  { _id: 41, customer_id: 9,  product: 'Laptop',   amount: NumberDecimal('1299.00'), status: 'placed', created_at: now, updated_at: now },
  { _id: 42, customer_id: 9,  product: 'Mouse',    amount: NumberDecimal('25.50'),   status: 'placed', created_at: now, updated_at: now },
  { _id: 43, customer_id: 10, product: 'Monitor',  amount: NumberDecimal('349.99'),  status: 'placed', created_at: now, updated_at: now },
  { _id: 44, customer_id: 3,  product: 'Keyboard', amount: NumberDecimal('89.00'),   status: 'placed', created_at: now, updated_at: now },
  { _id: 45, customer_id: 7,  product: 'Webcam',   amount: NumberDecimal('79.90'),   status: 'placed', created_at: now, updated_at: now }
]);
db.customers.updateOne({ name: 'Liam Carter' }, { $set: { tier: 'gold', updated_at: now } });
db.customers.updateOne({ name: 'Zara Ali' },    { $set: { city: 'Abu Dhabi', updated_at: now } });
db.orders.updateMany({ _id: { $in: [1, 4, 7] } }, { $set: { status: 'delivered', updated_at: now } });

The second sync read 12 change events (1.60 KB) from the stream and finished in 16 seconds end to end. For customers that meant one new data file with 4 rows: 2 inserts plus the new versions of 2 updated documents. One deletion vector retires the 2 old versions.

OLake Go's commit report in the sync log shows it directly: addedDataFiles=1, addedRecords=4, addedDVs=1 and addedPositionalDeletes=2. The orders table has the same shape with 8 rows and 3 deleted positions, and the equality delete count stays at 0.

OLake UI sync log filtered to the Iceberg commit reports for customers and orders after the change sync, showing addedDataFiles 1, addedDVs 1, addedRecords 4 and 8, and addedPositionalDeletes 2 and 3
The change-only sync in the OLake UI log: one data file and one deletion vector per collection, zero equality deletes.

Databricks shows the result on the next query, with no refresh. Liam Carter is gold, Zara Ali moved to Abu Dhabi, and the two new customers are there. The _op_type column marks how each row arrived: r for the snapshot rows, c for the inserts and u for the updates.

Schema changes. MongoDB has no schema, so new fields show up whenever a document carries one. OLake Go adds a column for a new top-level field on the next sync and commits the new schema to Unity Catalog; documents without the field hold NULL there. A field whose type changes between documents is widened to STRING, and the full rules are on the schema evolution page.

Query the same tables from DuckDB and Spark​

Nothing is exported, copied or refreshed. Both engines read the files OLake Go wrote, through the same Unity Catalog endpoint, with storage credentials vended by Databricks.

DuckDB attaches the Unity Catalog catalog as an Iceberg REST catalog with the same token. With vended_credentials, Unity Catalog hands DuckDB short-lived storage credentials per table, so no cloud keys are configured on the DuckDB side.

DuckDB 1.5.5
INSTALL iceberg; LOAD iceberg;
INSTALL httpfs;  LOAD httpfs;
CREATE SECRET unity (TYPE iceberg, TOKEN '<personal access token>');
ATTACH 'lakehouse' AS uc (
TYPE iceberg,
ENDPOINT 'https://<workspace-url>/api/2.1/unity-catalog/iceberg-rest',
SECRET unity,
ACCESS_DELEGATION_MODE 'vended_credentials'
);
SELECT _id, name, city, tier
FROM uc.mongodb_to_databricks_shop.customers
ORDER BY _id;

DuckDB reads Iceberg v3 deletion vectors from duckdb-iceberg 1.5.3, so the rows match Databricks after every sync. The principal behind the token needs EXTERNAL USE SCHEMA, the same as OLake Go.

DuckDB 1.5.5 returned the same 10 customers and the same orders roll-up as Databricks in the run on this page.

An orders roll-up matched MongoDB to the cent in all three engines once amount was cast. It gave 16 delivered orders for 8659.68, 18 placed for 10522.26, 11 shipped for 6397.25.

Data types: MongoDB to Iceberg to Databricks​

OLake Go maps MongoDB's BSON (Binary JSON) types to Iceberg types, and Databricks maps Iceberg types to its own. Scalars pass through; everything structured arrives as text.

MongoDB typeIceberg (written by OLake Go)Databricks showsNotes
int, timestampintINTThe integer _id and customer_id fields on this page arrived as INT.
longlongBIGINT
doubledoubleDOUBLE
booleanbooleanBOOLEAN
datetimestamptz (UTC)TIMESTAMP
decimal128stringSTRINGMoney stored as NumberDecimal arrives as text. Cast it before summing; the roll-up on this page used CAST(amount AS DOUBLE).
objectId, binData, regex, codestringSTRINGAn ObjectId primary key arrives as its 24-character hex string.
object, arraystringSTRINGJSON text; parse it at query time.

MongoDB types in the table come from the MongoDB connector reference; the Databricks side comes from DESCRIBE on the tables of this run.

Cost and maintenance​

  • Storage is the cloud bucket behind the catalog's managed location. You pay your cloud's storage rates for the Parquet data files and deletion vectors.
  • Compute. OLake Go runs wherever you run Docker and does not use a Databricks warehouse or cluster to write. Databricks compute is spent only when you query; the checks on this page ran on a 2X-Small serverless SQL warehouse.
  • Maintenance. Unity Catalog runs snapshot expiration and file compaction on managed Iceberg tables when predictive optimization is on (Databricks: Iceberg tables (checked Oct 2026)). Check that it is on for the catalog OLake Go writes to. Spark's rewrite_data_files also works through the same Unity Catalog endpoint.
  • Token rotation. A personal access token expires on the date set when it was created, and an expired token fails the next sync at the first catalog call. Use a service principal with OAuth machine-to-machine credentials for production jobs.
  • Oplog window. On Atlas, raise the minimum oplog window in the cluster settings if the job runs less often than once a day; on self-hosted, use replSetResizeOplog.

Troubleshooting​

1. The source test warns about the oplog​

oplog_retention: current 1.3 hours, recommended >= 7 days

Not a blocker, but it is the window in which the next sync must start. The fix is on the MongoDB side: replSetResizeOplog on self-hosted, or the minimum oplog window setting on Atlas. Until then, run the job more often than the window.

2. The source test cannot open a change stream​

change_streams: OLake cannot open a change stream, so CDC cannot run.

Either mongod is not running with --replSet, or the user lacks oplog access. Check rs.status() on the server and give the OLake Go user clusterMonitor (or readAnyDatabase) in admin on top of read on the database.

3. Order totals come back as text​

decimal128 fields land as STRING, so a plain sum(amount) does not add money. Cast in the query, as the roll-up on this page does with CAST(amount AS DOUBLE), or cast to DECIMAL(12,2) for exact arithmetic. Storing the value as a double in MongoDB removes the cast.

4. The destination test fails with a permission error​

Three settings cover nearly every failure: External data access is off on the metastore, the principal lacks EXTERNAL USE SCHEMA on the catalog, or the catalog name in the destination is wrong. ALL PRIVILEGES does not include EXTERNAL USE SCHEMA, so a catalog owner still has to grant it explicitly. Run the grants from step 2 again for the principal that owns the token.

5. Databricks rejects a commit with "Can not create a Path from a null string"​

Can not create a Path from a null string [ErrorCode: 2015]

Unity Catalog returns this when a client commits equality delete files to a managed Iceberg table (Databricks: access tables from Iceberg clients (checked Oct 2026)). Set Databricks as a target query engine on the job, so OLake Go writes deletion vectors. Start any table that was written with another delete format again from a full refresh.

How this compares with other ways to move MongoDB data to Databricks
MethodFreshnessUpdates and deletesWhere the data landsWhat you run
Debezium, Kafka and Structured Streaming from Kafka (checked Oct 2026)SecondsYes, with your own merge logicKafka topics, then tables written by a Databricks streaming jobKafka, Kafka Connect, Debezium and a running Databricks job
mongoexport to JSON files in a bucket, then COPY INTO or Auto Loader (checked Oct 2026)Hours to dailyNo; full reloads or your own mergeTables written by a Databricks jobAn export script, a bucket and a Databricks job
OLake Go to Iceberg on Unity Catalog (this guide)CDC on a schedule down to one minuteYes, as deletion vectorsApache Iceberg tables that Databricks, DuckDB and Spark all readOne Docker Compose stack, open source; your storage plus your own compute

Log-based capture through Debezium fits when you need seconds of latency and already run Kafka. Writing Iceberg once avoids a second copy as soon as another engine needs the same rows.

When not to use this​

  • You need the documents kept whole. OLake Go flattens one level, and deeply nested documents arrive as JSON text in one column. If every query starts by parsing that text, a landing table built for semi-structured data may serve you better.
  • Every query must run on Databricks-native tables. OLake Go writes managed Iceberg tables. If a pipeline needs native Delta Lake features on these rows, load them into a Delta table downstream.

FAQs​

How do I replicate MongoDB to Databricks without a managed connector?

Run OLake Go (open source, Apache 2.0) against a MongoDB replica set and point it at Unity Catalog's Iceberg REST endpoint. The first sync copies each collection in parallel chunks, every later sync replays only the changes from a change stream, and the collections appear in Databricks as Unity Catalog managed Iceberg tables with one column per top-level field. No Kafka, Debezium or Databricks ingestion pipeline sits in the path.

Does OLake Go copy the existing collections before it starts reading change streams?

Yes, automatically. A job in Full Refresh + CDC mode opens the change stream, copies every selected collection in parallel chunks, then replays the changes from the stream's resume token. Nothing written during the copy is lost, and there is no separate backfill job.

Which Iceberg delete format does OLake Go write for Databricks?

Deletion vectors, on Iceberg v3 tables. Under Unity Catalog, Databricks reads deletion vectors and rejects commits that carry equality deletes. Setting Databricks as a target query engine on the job makes OLake Go pick deletion vectors from the first sync, so there is no format to choose.

How do MongoDB documents become Databricks columns?

OLake Go flattens each document one level: every top-level field becomes a column, and nested objects and arrays arrive as JSON text in a STRING column. Fields that appear later in some documents are added as new columns on the next sync.

Does this work with MongoDB Atlas?

Yes. Atlas clusters are already replica sets with the oplog on. Create a database user with the read role on your database plus clusterMonitor, allow OLake Go's IP in Network Access, and use the SRV connection string with Use SRV on in the source form.

Which MongoDB types change on the way to Databricks?

int, long, double, boolean and date map to INT, BIGINT, DOUBLE, BOOLEAN and TIMESTAMP. Decimal128, ObjectId, binary, objects and arrays arrive as STRING, so cast Decimal128 money fields before summing them. OLake Go also adds a _cdc_resume_token column that records the change-stream position each row came from.

MongoDB to SnowflakeSame source, read from Snowflake.MongoDB to ClickHouseSame source, read from ClickHouse.MongoDB to BigQuerySame source, read from BigQuery.MongoDB to Amazon RedshiftSame source, read from Amazon Redshift.MongoDB to Apache IcebergSame source, read from Apache Iceberg.Postgres to DatabricksSame engine, replicated from Postgres.MySQL to DatabricksSame engine, replicated from MySQL.MariaDB to DatabricksSame engine, replicated from MariaDB.Oracle to DatabricksSame engine, replicated from Oracle.SQL Server to DatabricksSame engine, replicated from SQL Server.Kafka to DatabricksSame engine, replicated from Kafka.S3 to DatabricksSame engine, replicated from S3.MongoDB connector referenceEvery field, sync mode and troubleshooting note for the source.Query engine compatibilityWhich engines read which catalogs and delete formats.All integrationsEvery source and engine pair with a tested guide.

Deeper reading on this site: