Skip to main content

MongoDB to BigQuery: CDC via Apache Iceberg

MongoDB to BigQuery usually means a Datastream stream, Debezium through Kafka, or a nightly mongoexport and bq load. This guide takes a fourth route. OLake Go reads the MongoDB change stream and commits the changes as Apache Iceberg tables into a BigLake catalog, Google Cloud's Iceberg representational state transfer (REST) catalog, with the files in your own Cloud Storage bucket.

BigQuery queries the tables through the catalog with a four-part name, with no dataset to create and no external table to declare. Nothing else is in the path: no Datastream, no Kafka, no load job. DuckDB and Spark read the same tables through the same catalog.

By Shubham Satish BaldavaUpdated October 30, 2026

Key takeaways

  • OLake Go replicates MongoDB into Iceberg tables in a BigLake catalog: a full historical snapshot first, then change data capture (CDC) from the change stream, with no Datastream or Debezium in between. Documents are flattened into columns on the way.
  • BigQuery reads the catalog as a data source. Every change a sync commits is visible on the next SELECT, with no refresh and no load job.
  • The same tables, written once by OLake Go, are also queryable from DuckDB and Spark. In the run on this page all three engines returned identical rows.

The short answer. To replicate MongoDB to BigQuery with change data capture (CDC), run OLake Go with a MongoDB replica set as the source and a BigLake catalog with a Cloud Storage bucket as the Iceberg destination, with BigQuery as the target query engine. The first sync copies each collection in full (the historical snapshot); every later sync replays inserts, updates and deletes from the change stream.

BigQuery, DuckDB and Spark then query the same Iceberg tables, with no second copy.

How MongoDB CDC reaches BigQuery here​

Two things happen in order, and OLake Go does both on its own. The first sync opens a change stream and saves its resume token. Then it reads each collection in parallel chunks and writes Parquet data files: that is the historical load.

Every later sync resumes the stream from the saved token and replays the inserts, updates and deletes since. Nothing written during the snapshot is lost, and there is no separate backfill job.

Each sync ends with one commit to BigLake. OLake Go uploads the data and delete files to the bucket with the service account's own credentials, then updates the table's metadata pointer in the catalog. BigQuery, DuckDB and Spark see the new snapshot on their next query; none of them keeps a copy.

One thing to keep an eye on: the oplog is MongoDB's replay window. A change stream can only resume from a token that is still in the oplog.

A job paused for longer than the oplog window cannot resume and needs a fresh snapshot. Size the oplog for your longest planned pause; the source test on this page reports the current window.

What you need​

  • MongoDB 4.0 or later running as a replica set, self-hosted or on Atlas, with a user that can read the database and the oplog. The collections on this page ran on MongoDB 7 as a single-node replica set in Docker; step 1 covers Atlas too.
  • A Google Cloud project with the BigLake API enabled, a Cloud Storage bucket for the files, and a service account for OLake Go with the BigLake Editor role on the project and Storage Object User on the bucket. The run on this page used a bucket in us-central1.
  • BigQuery in the same project. Queries run in the bucket's region, and the account running them needs the BigQuery Job User role plus read access to the catalog.
  • OLake UI from the Docker Compose install. The run on this page used OLake Go v1.1.0.
  • Docker, and about 30 minutes.

Step 1: Prepare MongoDB for CDC​

Most readers point OLake Go at collections they already have. If you want to follow along with the exact data from this page, create these two collections first:

The two sample collections used on this page
mongosh, database shop
const now = new Date();
db.customers.insertMany([
  { _id: 1, name: 'Aarav Mehta', email: 'aarav@example.com', city: 'Mumbai', tier: 'gold', created_at: now, updated_at: now },
  // ... 8 customers with integer _id values
]);
db.orders.insertMany([
  { _id: 1, customer_id: 2, product: 'Headphones', amount: NumberDecimal('99.19'), status: 'shipped', created_at: now, updated_at: now },
  // ... 40 orders
]);
// Then insert whatever you like. This page started with 8 customers and 40 orders.

OLake Go reads MongoDB changes through change streams, which MongoDB builds on the replica set oplog. Three things must be in place before the first sync.

SettingValueWhy it matters
Replica setany name, even a single nodeChange streams and the oplog exist only in replica set mode. Atlas clusters are already replica sets.
Oplog retentionlong enough to cover your longest pauseOLake Go resumes from a change-stream token saved after each sync; if the oplog has rolled past it, the next sync needs a fresh snapshot.
A user for OLake Goread on the database plus clusterMonitorThe snapshot reads the collections; the change stream needs oplog access, which clusterMonitor (or readAnyDatabase) grants.
mongosh, on the primary
// 1. A single-node replica set, if you do not have one (self-hosted only)
//    start mongod with --replSet rs0, then:
rs.initiate({ _id: "rs0", members: [{ _id: 0, host: "<host>:27017" }] })

// 2. A user for OLake Go
use admin
db.createUser({
  user: "olake", pwd: "<password>",
  roles: [{ role: "read", db: "shop" }, { role: "clusterMonitor", db: "admin" }]
})

// 3. Check the oplog window; OLake Go warns below 7 days
use local
db.oplog.rs.stats().maxSize      // bytes; raise with replSetResizeOplog if it is small

Check the result before moving on:

javascript
rs.status().members.map(m => m.stateStr)      // one PRIMARY
db.getSiblingDB("admin").getUsers().users.map(u => u.user)

On MongoDB Atlas the replica set and oplog are already there; create the user under Database Access with the read role on your database and clusterMonitor, and allow OLake Go's IP in Network Access. The full guide, including Atlas screenshots, is MongoDB and Atlas CDC setup.

The oplog is your replay window

Atlas and most self-hosted installs keep 24 hours or less of oplog by default. OLake Go's source test shows the current window and warns below 7 days. Size it for your longest planned pause, or run the job often enough that the token never falls out of the window.

Step 2: Set up the BigLake catalog​

BigQuery reads Iceberg tables through a catalog. BigLake is the one this page uses: it is Google Cloud's Iceberg REST catalog, OLake Go creates a namespace per job in it, and the files sit in your Cloud Storage bucket.

BigQuery, DuckDB and Spark all read from the same catalog. BigQuery treats it as a data source in the project, so a table OLake Go commits is queryable on the next statement.

CatalogOLake Go writesBigQuery readsVerified on this page
BigLake (Google Cloud Lakehouse)YesYes: native: namespaces map to BigQuery datasetsYes, end to end

BigLake is the Iceberg REST catalog built into Google Cloud's Lakehouse. OLake Go talks to it at https://biglake.googleapis.com/iceberg/v1/restcatalog, the table files land in a Cloud Storage bucket you own, and BigQuery reads the tables without any registration step. Writing to BigLake needs OLake Go 0.9.1 or later.

Everything below happens in the Google Cloud console, in the project that will hold the tables.

  1. Enable the API. Open APIs and services, then Enable APIs and services, and enable the BigLake API. BigQuery and Cloud Storage are on by default in most projects.
  2. Create a bucket. In Cloud Storage, create a bucket in the region where BigQuery will run the queries (us-central1 on this page). The bucket name becomes the catalog name in BigQuery, so pick one you are happy to type in SQL.
  3. Create a service account for OLake Go. In IAM and Admin, Service Accounts, create one (for example olake-biglake). Then grant it two roles: BigLake Editor on the project (IAM, Grant access, the service account as principal) and Storage Object User on the bucket (the bucket's Permissions tab).
  4. Download a JSON key. On the service account's Keys tab, Add key, Create new key, JSON. The file downloads once; OLake Go asks for its contents.
Google Cloud console IAM Grant access panel with the OLake service account as principal and the BigLake Editor role selected
The project-level grant: the service account OLake Go runs as, with the BigLake Editor role.

There is no catalog to create. BigLake creates a single-bucket catalog named after the bucket the first time OLake Go calls the endpoint with gs://<bucket> as the catalog path, in the default end-user credential mode, where every client brings its own storage credentials. The catalog shows up under BigLake, Catalogs after the first connection test.

Two things to know before the first sync:

  • The service account's BigLake Editor role is what lets OLake Go create namespaces and commit tables. Storage Object User on the bucket is what lets it write the Parquet and metadata files. Missing either one fails the connection test with a 403 from biglake.googleapis.com or from storage.googleapis.com, which tells you which grant is missing.
  • Leave the catalog in end-user mode. OLake Go 1.1.0 does not send the X-Iceberg-Access-Delegation header, so a catalog switched to vended credentials rejects its commits (see troubleshooting). Readers that want short-lived tokens from the catalog can switch the mode later with gcloud beta biglake iceberg catalogs update <bucket> --credential-mode=vended-credentials, after setting rest_access_delegation in a CLI config.
Google Cloud console Create private key dialog for the OLake service account with the JSON key type selected
The JSON key for the service account. The console downloads it once; its contents go into the OLake Go destination form.

Step 3: Connect MongoDB to OLake Go​

  1. In the OLake UI, open Sources and click Create Source. Pick MongoDB as the connector, give the source a name and keep the OLake Go version on the latest release.
  2. Under Change data capture (CDC), leave Enable CDC on.
  3. Under Connection details, add each host as host:port (or one SRV host with Use SRV on for Atlas), then the Database Name, Auth DB (admin), username, password and the Replica Set name. Leave Read Preference on primary unless you have secondaries to spare.
  4. Click Set up source. OLake Go tests the connection, confirms that it can open a change stream, and reports the oplog window with a warning below 7 days.

Here is the source form from the run on this page. MongoDB ran in another container, so the host is host.docker.internal:27017; the replica set is rs0 and the user is the one from step 1, authenticating against admin.

OLake UI Edit source form for MongoDB on OLake Go v1.1.0 with Enable CDC on, host host.docker.internal:27017, database shop, auth DB admin, user olake, replica set rs0 and chunking strategy Split Vector
The MongoDB source in OLake UI: Enable CDC on, replica set rs0, with the user from step 1.

Set up source runs the connection test, confirms that a change stream can be opened, and reports the oplog window before anything is saved. A window under 7 days shows as a warning, not a blocker.

The full field reference, including SRV strings, IAM authentication and SSH tunnels, is on the MongoDB connector page.

Step 4: Configure the Iceberg destination​

  1. Open Destinations and click Create Destination.
  2. Pick Apache Iceberg as the connector and BigLake as the catalog type.
  3. Fill in the fields below. The catalog path is the bucket, with no folder after it. Paste the whole service account JSON file into the credentials field.
  4. Click Create. OLake Go calls the BigLake endpoint, creates the catalog if it is the first contact, and writes and drops a test table in the bucket.

Here is the destination form from the run on this page. It holds the BigLake endpoint, the bucket as the catalog path, the service account JSON and the project ID.

OLake UI Edit destination form for Apache Iceberg with catalog type Big Lake, REST catalog URI https://biglake.googleapis.com/iceberg/v1/restcatalog, BigLake catalog path gs://olake-x2y-lab-bq, authentication type GCP and the project ID oauth-361809
The BigLake destination in OLake UI: the endpoint, the bucket as catalog path, GCP authentication and the project ID.
FieldValue on this pageWhat it is
REST Catalog URIhttps://biglake.googleapis.com/iceberg/v1/restcatalogThe BigLake Iceberg REST endpoint. The same for every project and region.
BigLake Catalog Pathgs://<bucket>The bucket that holds the tables. The bucket name is also the catalog name BigQuery uses. Bucket root only; a folder after it is not supported.
Authentication TypeGCPGoogle OAuth through the service account. The only choice for BigLake.
GCP Service Account JSONthe key file contentsUsed for both the catalog calls and the Cloud Storage writes.
Catalog Nameolake_icebergA label OLake Go uses in its own logs. Lowercase letters and underscores only.
GCP Auth Scopeshttps://www.googleapis.com/auth/cloud-platformThe OAuth scope on the token. Leave the default.
GCP Project IDthe project IDRoutes and bills the catalog calls. Required for multi-bucket catalogs, harmless to set for a single bucket.
Enable Arrow WritesoffKeeps the default Java writer, which is what this page was tested with.

Step 5: Create the job and run the first sync​

This job lists BigQuery, Spark and DuckDB as target query engines. OLake Go then offers only the delete formats all three read, and the run on this page picked Positional Deletes from that list. Each stream runs as Full Refresh + CDC, so the change stream carries every change after the snapshot.

  1. Open Jobs and click Create Job. Give it a name and a schedule. Every Day at a fixed hour, or a cron expression, both work; the schedule is how often the change stream is drained, not how often a full copy runs.
  2. Select the source and the destination you created.
  3. Under Advanced Settings, set Target Query Engines to Google BigQuery, Apache Spark and DuckDB. OLake Go then resolves an Iceberg delete format every one of those engines can read, so you never pick one by hand.
  4. Click Next. OLake Go discovers the collections and opens the Streams page.
  5. Tick the collections to replicate. For each one, set the sync mode to Full Refresh + CDC and the ingestion mode to Upsert. Leave the Upsert Type as OLake Go resolved it (Positional Deletes in the run on this page).
  6. Click Create Job, then open the job's menu and choose Sync Now for the first run.

OLake Go offers every delete format BigQuery, Spark and DuckDB all read: equality and positional deletes. Deletion vectors are not in the list because BigQuery reads them only in preview.

OLake UI Streams Selection page with the shop database, customers and orders selected, sync mode Full Refresh + CDC, ingestion mode Upsert and upsert type Positional Deletes on Iceberg format version V2
Each stream: Full Refresh + CDC, Upsert, and Positional Deletes as the upsert type. The Iceberg DB name at the top is editable.
OLake UI Job Logs and History for mongodb_to_bigquery showing two completed syncs, the first in 34 seconds and the second in 33 seconds
Job history: the 34 s historical snapshot, then the 33 s change-only sync.

On this page the first run copied 8 customers and 40 orders (48 documents, 6.40 KB) and finished in 34 seconds, container start included. OLake Go names the BigLake namespace after the job and the source database, so mongodb_to_bigquery on database shop produced mongodb_to_bigquery_shop. The Iceberg DB field at the top of the Streams page changes it before the first sync.

Step 6: Query the tables in BigQuery​

There is no registration step and no external table to declare. A BigLake catalog is a BigQuery data source in the same project: the catalog is the bucket name, the namespace OLake Go created is the dataset, and the tables are the tables. Address them with the four-part name in backticks, and run the query in the bucket's region.

BigQuery SQL editor (location: the bucket region)
SELECT _id, name, city, tier
FROM `<project-id>.<bucket>.mongodb_to_bigquery_shop.customers`
ORDER BY _id;

SELECT status, count(*) AS n, round(sum(cast(amount AS NUMERIC)), 2) AS total
FROM `<project-id>.<bucket>.mongodb_to_bigquery_shop.orders`
GROUP BY status ORDER BY status;

From a terminal, bq query --project_id=<project-id> --location=<region> --use_legacy_sql=false "<sql>" runs the same statements. Set the location to the bucket's region (us-central1 on this page); the default US multi-region returns Not found: Dataset for a catalog whose bucket is regional.

OLake Go adds four columns to every table: _olake_id (a hash of the primary key), _op_type, _olake_timestamp and _cdc_timestamp. _op_type is r for a row from the first snapshot, c for an insert, u for an update and d for a delete. Filter them out of dashboards, or use them to see when a row last changed.

BigQuery reads Iceberg v2 tables with positional and equality deletes, which is why the job on this page uses positional deletes. Deletion vectors on v3 tables are a preview feature in BigQuery with per-query limits, so keep the job off deletion vectors while BigQuery is a reader.

BigQuery SQL editor in the Google Cloud console running the customers query on the olake-x2y-lab-bq catalog and returning 10 rows with _id, name, city and tier
BigQuery reading the customers table through the BigLake catalog after the change sync: 10 rows, one per key.

The key column is _id, as in MongoDB, and amount arrives as text because the fixture stores it as Decimal128. The roll-up above therefore casts it: round(sum(cast(amount AS NUMERIC)), 2).

Step 7: Change documents in MongoDB and sync again​

To see the change path work, the run on this page made 12 changes in MongoDB after the first sync:

Changes made in MongoDB between the two syncs
const now = new Date();
db.customers.insertMany([
  { _id: 9,  name: 'Maya Patel',  email: 'maya@example.com',  city: 'Pune',      tier: 'silver', created_at: now, updated_at: now },
  { _id: 10, name: 'Lucas Silva', email: 'lucas@example.com', city: 'São Paulo', tier: 'bronze', created_at: now, updated_at: now }
]);
db.orders.insertMany([
  { _id: 41, customer_id: 9,  product: 'Laptop',   amount: NumberDecimal('1299.00'), status: 'placed', created_at: now, updated_at: now },
  { _id: 42, customer_id: 9,  product: 'Mouse',    amount: NumberDecimal('25.50'),   status: 'placed', created_at: now, updated_at: now },
  { _id: 43, customer_id: 10, product: 'Monitor',  amount: NumberDecimal('349.99'),  status: 'placed', created_at: now, updated_at: now },
  { _id: 44, customer_id: 3,  product: 'Keyboard', amount: NumberDecimal('89.00'),   status: 'placed', created_at: now, updated_at: now },
  { _id: 45, customer_id: 7,  product: 'Webcam',   amount: NumberDecimal('79.90'),   status: 'placed', created_at: now, updated_at: now }
]);
db.customers.updateOne({ name: 'Liam Carter' }, { $set: { tier: 'gold', updated_at: now } });
db.customers.updateOne({ name: 'Zara Ali' },    { $set: { city: 'Abu Dhabi', updated_at: now } });
db.orders.updateMany({ _id: { $in: [1, 4, 7] } }, { $set: { status: 'delivered', updated_at: now } });

The second sync read 12 change events (1.60 KB) from the stream and finished in 33 seconds end to end. For customers that meant one new data file with 4 rows: 2 inserts plus the new versions of 2 updated documents. One small positional delete file retires the 2 old versions.

OLake Go's commit report in the sync log shows it directly: addedDataFiles=1, addedRecords=4, addedPositionalDeleteFiles=1, addedPositionalDeletes=2. The orders table has the same shape with 8 rows and 3 deletes.

OLake UI sync log filtered to the Iceberg commit report for customers, showing addedDataFiles 1, addedPositionalDeleteFiles 1, addedRecords 4 and addedPositionalDeletes 2
The change-only sync in the OLake UI log: one data file and one positional delete file per collection.

BigQuery shows the result on the next query, with no refresh and no ALTER TABLE. Liam Carter is gold, Zara Ali moved to Abu Dhabi, and the two new customers are there.

The two retired versions are gone, because BigQuery applies the positional deletes while reading. The _op_type column shows how each row arrived: r for the snapshot, c for the inserts and u for the updates.

Schema changes. When a new field appears in the documents, OLake Go adds it as a column on the next sync and commits the new schema to BigLake in the same commit. BigQuery shows the column on the next query, with nulls for older rows.

A field whose type changes in an incompatible way fails the sync rather than corrupt the table. Full rules are on the schema evolution page.

Query the same tables from DuckDB and Spark​

This is the point of writing Iceberg instead of loading BigQuery tables. Nothing is exported, copied or refreshed: both engines read the files OLake Go wrote, through the same BigLake catalog.

DuckDB attaches BigLake as an Iceberg REST catalog with a Google access token. The catalog on this page is in end-user credential mode, so DuckDB also needs its own Cloud Storage credentials to read the files: an HMAC key for the service account (Cloud Storage, Settings, Interoperability, Create a key for a service account).

DuckDB 1.5.5
INSTALL iceberg; LOAD iceberg;
INSTALL httpfs;  LOAD httpfs;

-- gcloud auth print-access-token, or a token minted for the service account
CREATE SECRET biglake_catalog (
TYPE iceberg,
TOKEN '<access token>',
EXTRA_HTTP_HEADERS MAP {'x-goog-user-project': '<project-id>'}
);
-- the HMAC key pair for the bucket
CREATE SECRET biglake_files (TYPE gcs, KEY_ID '<hmac access id>', SECRET '<hmac secret>');

ATTACH 'gs://<bucket>' AS biglake (
TYPE iceberg,
ENDPOINT 'https://biglake.googleapis.com/iceberg/v1/restcatalog',
SECRET biglake_catalog
);

SHOW ALL TABLES;
SELECT _id, name, city, tier FROM biglake.mongodb_to_bigquery_shop.customers ORDER BY _id;

The access token lives an hour; CREATE OR REPLACE SECRET with a fresh one when a query fails with 401. If the catalog is in vended-credentials mode instead, drop the gcs secret and add ACCESS_DELEGATION_MODE 'vended_credentials' to the ATTACH: BigLake then hands DuckDB a short-lived storage token per table. DuckDB reads positional deletes, so the rows match BigQuery after every sync.

Google Cloud console Cloud Storage Interoperability settings with the Select a service account dialog, the OLake service account selected and the Create key button
The key for DuckDB: Cloud Storage, Settings, Interoperability, then a key for the same service account OLake Go uses.

DuckDB 1.5.5 returned the same 10 customers and the same orders roll-up as BigQuery in the run on this page, with sum(cast(amount AS decimal(12,2))) for the money column. It read the current metadata file of each table through the catalog and the data files through the hash-based message authentication code (HMAC) key.

An orders roll-up matched MongoDB to the cent in all three engines, each casting amount to a decimal first. It gave 16 delivered orders for 8659.68, 18 placed for 10522.26, 11 shipped for 6397.25.

Data types: MongoDB to Iceberg to BigQuery​

OLake Go maps MongoDB's BSON (Binary JSON) types to Iceberg types, and BigQuery maps Iceberg types to its own when it reads the catalog. Scalars pass through; everything structured arrives as text.

MongoDB typeIceberg (written by OLake Go)BigQuery showsNotes
int, timestampintINT64The integer _id keys on this page arrived as INT64.
longlongINT64
doubledoubleFLOAT64
booleanbooleanBOOL
datetimestamptz (UTC)TIMESTAMP
decimal128stringSTRINGMoney stored as NumberDecimal arrives as text. sum(cast(amount AS NUMERIC)) gives the exact total; the roll-up on this page used that cast. Store doubles in MongoDB if you want a numeric column without the cast.
objectId, binData, regex, codestringSTRINGAn ObjectId primary key arrives as its 24-character hex string.
object, arraystringSTRINGJSON text. Parse with PARSE_JSON into a JSON value, or pick fields with JSON_VALUE.

MongoDB types in the table come from the MongoDB connector reference. The BigQuery side comes from the BigQuery Iceberg tables page (checked 2026-10-30) and from the table schemas of this run. OLake Go adds _cdc_resume_token on MongoDB tables, the change-stream token each change came from.

Cost and maintenance​

  • Storage is your bucket: you pay Cloud Storage rates for the Parquet and delete files. The BigLake catalog itself has no per-table charge.
  • Compute. OLake Go runs wherever you run Docker. BigQuery bills on-demand queries by bytes scanned, or by slot time on an edition; a query over the catalog reads the Parquet files in the bucket, so filter on the columns you need. DuckDB and Spark cost whatever machine they run on.
  • Maintenance. A CDC table that gets a new small file and delete file per sync wants compaction, and BigLake keeps every snapshot OLake Go commits in the table's metadata.json. BigLake rejects a metadata.json above 1 MB, which a busy table reaches after a few thousand syncs. Expire old snapshots and compact on a schedule with OLake Fusion or with Spark's expire_snapshots and rewrite_data_files procedures through the same catalog.
  • Oplog size. The oplog is sized in gigabytes, not hours, so a write-heavy cluster shortens the window. Check it after a busy day and resize before a planned pause.
  • Key rotation. The service account key is what OLake Go writes with. Rotate it on your schedule from the service account's Keys tab and paste the new JSON into the destination.

Troubleshooting​

1. The source test warns about the oplog​

oplog_retention: current 1.3 hours, recommended >= 7 days

Not a blocker, but it is the window in which the next sync must start. The fix is on the MongoDB side: replSetResizeOplog on self-hosted, or the minimum oplog window setting on Atlas. Until then, run the job more often than the window.

2. The source test cannot open a change stream​

change_streams: OLake cannot open a change stream, so CDC cannot run.

Either mongod is not running with --replSet, or the user lacks oplog access. Check rs.status() on the server and give the OLake Go user clusterMonitor (or readAnyDatabase) in admin on top of read on the database.

3. A money column will not sum in BigQuery​

No matching signature for aggregate function SUM
Argument types: STRING

The field is a Decimal128 in MongoDB, which arrives as STRING. Wrap it: sum(cast(amount AS NUMERIC)). DuckDB and Spark need the same cast, to decimal(12,2).

4. The destination test fails on a catalog in vended-credentials mode​

failed to test destination: failed to create or get table: rpc error: code = Internal desc = Failed to process request: Malformed request: X-Iceberg-Access-Delegation header must be present and contain `vended-credentials` when credential mode is `CREDENTIAL_MODE_VENDED_CREDENTIALS`.

The catalog was created, or switched, with --credential-mode=vended-credentials, and OLake Go 1.1.0 does not send the header BigLake wants in that mode (olake#1369). Switch the catalog back with gcloud beta biglake iceberg catalogs update <bucket> --credential-mode=end-user, which is the default for a catalog OLake Go created itself. With the CLI, "rest_access_delegation": "vended-credentials" in destination.json makes the same test pass; the UI has no field for it yet.

5. BigQuery cannot find the dataset​

Not found: Dataset <project>:<bucket>.mongodb_to_bigquery_shop was not found in location US

The query ran in a location other than the bucket's region. Set the query location to the bucket's region in the editor's More, Query settings panel, or pass --location=<region> to bq. Leaving the location unset lets BigQuery pick it from the table.

How this compares with other ways to move MongoDB data to BigQuery
MethodFreshnessUpdates and deletesWhere the data landsWhat you run
Datastream for MongoDBNear real time, by Google's description; CDC from change streamsYesNative BigQuery datasets; BigQuery is the only readerNothing to run; a connection profile and a stream in the console
Debezium, Kafka and a BigQuery sinkSecondsYes, with the connector's upsert modeKafka topics, then BigQuery tablesKafka, Kafka Connect, Debezium and a sink connector per collection
Export (mongoexport to JSON), then bq loadHours to dailyNo; full reloads onlyNative BigQuery tablesA script, a scheduler and a staging bucket
Managed ELT (Fivetran, Estuary, Airbyte Cloud)Minutes, by planVaries by toolBigQuery tables, a second copy you pay to move and storeNothing; priced per row, per GB or per connector
OLake Go to Iceberg on BigLake (this guide)CDC on a schedule down to one minuteYes, as Iceberg row-level deletesApache Iceberg tables that BigQuery, Spark, DuckDB, Trino and Flink all readOne Docker Compose stack, open source; your bucket plus your own compute

Datastream is the simpler choice when BigQuery is the only reader and a managed Google service is the preference. Writing Iceberg once avoids paying to move and store a second copy. That matters as soon as Spark, DuckDB or a notebook needs the same rows, or you want to own the files.

When not to use this​

  • Every query must hit native BigQuery speed. A catalog table is read from Parquet in the bucket on every query. Keep the Iceberg tables as the source and load the hot slice into a native table on a schedule.
  • Deeply nested documents must stay queryable as columns. OLake Go flattens one level; nested objects and arrays arrive as JSON text. Model them with JSON_VALUE in BigQuery, or use a tool that unnests on write.
  • You want deletion vectors. BigQuery reads them only in preview, with a cap of 100,000 deletion vector entries per query. Keep the job on positional deletes while BigQuery is a reader.

FAQs​

How do I replicate MongoDB to BigQuery without Datastream or a Kafka pipeline?

Run OLake Go (open source, Apache 2.0) against a MongoDB replica set and point it at a BigLake catalog with a Cloud Storage bucket. The first sync copies each collection in full, every later sync replays only the changes from a change stream, and BigQuery queries the tables through the catalog with a four-part name. Datastream, Kafka, Debezium and a load job are not needed, and documents are flattened into columns on the way.

Does BigQuery read Iceberg tables with row-level deletes?

Yes. BigQuery reads Iceberg v2 merge-on-read tables with positional and equality deletes. The run on this page used positional deletes and BigQuery returned one row per key after the updates. Deletion vectors on Iceberg v3 tables are a preview feature with per-query limits, so keep the job on positional deletes while BigQuery is a reader.

Does OLake Go copy the existing collections before it starts reading the change stream?

Yes, automatically. A job in Full Refresh + CDC mode opens the change stream, copies every selected collection in parallel chunks, then replays the stream from the token it saved. Nothing written during the copy is lost, and there is no separate backfill job.

How do nested documents arrive in BigQuery?

Flattened one level. Top-level fields become columns; nested objects and arrays arrive as JSON text in a STRING column, which BigQuery parses with PARSE_JSON or JSON_VALUE. Decimal128 values also arrive as STRING, so cast them to NUMERIC before you sum.

Do I need to create a BigQuery dataset or an external table?

No. A BigLake catalog is a data source BigQuery reads directly: the bucket name is the catalog, the namespace OLake Go creates is the dataset, and the tables appear under it. The query uses the four-part name project.bucket.namespace.table, and there is no CREATE EXTERNAL TABLE and no metadata refresh.

Does MongoDB CDC with OLake Go work on Atlas?

Yes. Atlas clusters are already replica sets with the oplog on. Create a database user with the read role on your database plus clusterMonitor, allow OLake Go's address in the Atlas network list, and paste the cluster hosts into the source form with Use SRV on.

What does OLake Go need on the MongoDB side?

A replica set (a single node is fine), oplog retention longer than your longest pause between syncs, and a user with read on the database plus clusterMonitor or readAnyDatabase in admin for oplog access. Step 1 has the commands.

MongoDB to SnowflakeSame source, read from Snowflake.MongoDB to ClickHouseSame source, read from ClickHouse.MongoDB to Amazon RedshiftSame source, read from Amazon Redshift.MongoDB to DatabricksSame source, read from Databricks.MongoDB to Apache IcebergSame source, read from Apache Iceberg.Postgres to BigQuerySame engine, replicated from Postgres.Oracle to BigQuerySame engine, replicated from Oracle.MySQL to BigQuerySame engine, replicated from MySQL.Kafka to BigQuerySame engine, replicated from Kafka.MariaDB to BigQuerySame engine, replicated from MariaDB.S3 to BigQuerySame engine, replicated from S3.DB2 to BigQuerySame engine, replicated from DB2.SQL Server to BigQuerySame engine, replicated from SQL Server.MongoDB connector referenceEvery field, sync mode and troubleshooting note for the source.Query engine compatibilityWhich engines read which catalogs and delete formats.All integrationsEvery source and engine pair with a tested guide.

Deeper reading on this site: