MongoDB Source

MongoDB Source enables data synchronization from MongoDB to your desired destination.

info

OLake UI is live (beta)! You can now use the UI to configure your MongoDB source, discover streams, and sync data. Check it out at OLake UI regarding how to setup using Docker Compose and running it locally.

Now, you can use the UI to configure your MongoDB source, discover streams, and sync data.

Use OLake UI for MongoDB

Create a MongoDB Source in OLake UI

Follow the steps below to get started with the MongoDB Source using the OLake UI (assuming the OLake UI is running locally on localhost:8000):

1. Navigate to Sources Tab.
2. Click on + Create Source.
3. Select MongoDB as the source type from Connector type.
4. Fill in the required connection details in the form. For details regarding the connection details, refer to the MongoDB Source Configuration section on the right side of UI.
5. Click on Create ->
6. OLake will test the source connection and display the results. If the connection is successful, you will see a success message. If there are any issues, OLake will provide error messages to help you troubleshoot.

This will create a MongoDB source in OLake, now you can use this source in your Jobs Pipeline to sync data from MongoDB to Apache Iceberg or AWS S3.

Edit MongoDB Source in OLake UI

To edit an existing MongoDB source in OLake UI, follow these steps:

1. Navigate to the Sources Tab.
2. Locate the MongoDB source you want to edit from Active Sources or Inactive Sources tabs or using the search bar.
3. Click on the Edit button next to the source from the Actions tab (3 dots).
4. Update the connection details as needed in the form and Click on Save Changes.

caution

Editing a source can break pipeline.

You will see a notification saying "Due to the editing, the jobs are going to get affected".

Editing this source will affect the following jobs that are associated with this source and as a result will fail immediately. Do you still want to edit the source?

5. OLake will test the updated source connection once you hit confirm on the Source Editing Caution Modal. If the connection is successful, you will see a success message. If there are any issues, we will provide error messages to help you troubleshoot.

Jobs Associated with MongoDB Source

In the Source Edit page, you can see the list of jobs that are associated with this source. You can also see the status of each job, whether it is running, failed, or completed and can pause the job from the same screen as well.

Delete MongoDB Source in OLake UI

To delete an existing MongoDB source in OLake UI, follow these steps:

1. Navigate to the Sources Tab.
2. Locate the MongoDB source you want to delete from Active Sources or Inactive Sources tabs or using the search bar.
3. Click on the Delete button next to the source from the Actions tab (3 dots).

4. A confirmation dialog will appear asking you to confirm the deletion.
5. Click on Delete to confirm the deletion.

This will remove the MongoDB source from OLake.

note

You can also delete a source from the Source Edit page by clicking on the Delete button at the bottom of the page.

Use OLake CLI for MongoDB

To sync data TLDR:

1. Create a source.json with your MongoDB connection details.
2. Create a destination.json with your Writer (Apache Iceberg / AWS S3 / Azure ADLS / Google Cloud Storage) connection details.
3. Run discover to generate a streams.json of available streams.
4. Run sync to replicate data to your specified destination.

Below is an overview of the supported modes and writers for MongoDB data replication, along with tables summarizing the details.

Supported Modes

Our replication process supports various modes to fit different data ingestion needs.

The Full Refresh mode retrieves the entire dataset from MongoDB and is ideal for initial data loads or when a complete dataset copy is required.

In contrast, CDC (Change Data Capture) continuously tracks and synchronizes incremental changes in real time, making sure that your destination remains updated with minimal latency.

Mode	Description
Full Refresh	Fetches the complete dataset from MongoDB.
CDC (Change Data Capture)	Tracks and syncs incremental changes from MongoDB in real time.
Strict CDC (Change Data Capture)	Tracks only new changes from the current position in the MongoDB change stream, without performing an initial backfill.
Incremental	No, WIP

Supported Destinations

Destination	Supported	Docs	Comments
Apache Iceberg	Yes	Link
AWS S3	Yes	Link	Supports both plain-Parquet and Iceberg format writes; requires aws_access_key / IAM role.
Azure	Yes	Link
Google Cloud Storage	Yes	Link (Iceberg) Link (Parquet)	Any S3 protocol compliant object store can work with OLake
Local Filesystem	Yes	Link

Setup and Configuration

To run the MongoDB Driver, configure the following files with your specific credentials and settings:

• source.json: MongoDB connection details.
• streams.json: List of collections and fields to sync (generated using the Discover command).
• write.json: Configuration for the destination where the data will be written.

Place these files in your project directory before running the commands.

Source File

Add MongoDB credentials in following format in source.json file as shown here.

Commands

Discover Command

The Discover command generates json content for streams.json file, which defines the schema of the collections to be synced.

Usage

To run the Discover command, use the following syntax

OLake Docker

macOS / Linux

docker run --pull=always  \
  -v "$HOME/PATH_TO_OLAKE_DIRECTORY:/mnt/config" \
  olakego/source-mongodb:latest \
  discover \
  --config /mnt/config/source.json

CMD

docker run --pull=always  ^
  -v "%USERPROFILE%\PATH_TO_OLAKE_DIRECTORY:/mnt/config" ^
  olakego/source-mongodb:latest ^
  discover ^
  --config /mnt/config/source.json

Powershell

docker run --pull=always  `
  -v "$env:USERPROFILE\PATH_TO_OLAKE_DIRECTORY:/mnt/config" `
  olakego/source-mongodb:latest `
  discover `
  --config /mnt/config/source.json

Locally run OLake

macOS / Linux

OLAKE_BASE_PATH="$HOME/PATH_TO_OLAKE_DIRECTORY/olake/drivers/mongodb/config" && \
./build.sh driver-mongodb discover \
  --config "$OLAKE_BASE_PATH/source.json"

CMD

set "OLAKE_BASE_PATH=%USERPROFILE%\PATH_TO_OLAKE_DIRECTORY\olake\drivers\mongodb\config" && ^
./build.sh driver-mongodb discover ^
  --config "%OLAKE_BASE_PATH%\source.json"

Powershell

$OLAKE_BASE_PATH = "$env:USERPROFILE\PATH_TO_OLAKE_DIRECTORY\olake\drivers\mongodb\config"; `
./build.sh driver-mongodb discover `
  --config "$OLAKE_BASE_PATH\source.json"

info

PATH_TO_OLAKE_DIRECTORY is the absolute path where you have created the directory [as discussed above]. -v "$HOME/PATH_TO_OLAKE_DIRECTORY:/mnt/config" \ maps to -v /Users/JOHN_DOE_USERNAME/Desktop/projects/OLAKE_DIRECTORY:/mnt/config \ in macOS and Linux systems. Follow the same pattern in other systems.

Streams File

After executing the Discover command, a streams.json file is created. Read more about Streams File here.

Destination File

Read about about

1. Apache Iceberg Destination config
2. S3 writer here
3. Local Filesystem writer

Sync Command

The Sync command fetches data from MongoDB and ingests it into the destination.

OLake Docker

macOS / Linux

docker run --pull=always  \
  -v "$HOME/PATH_TO_OLAKE_DIRECTORY:/mnt/config" \
  olakego/source-mongodb:latest \
  sync \
  --config /mnt/config/source.json \
  --catalog /mnt/config/streams.json \
  --destination /mnt/config/destination.json

CMD

docker run --pull=always  ^
  -v "%USERPROFILE%\PATH_TO_OLAKE_DIRECTORY:/mnt/config" ^
  olakego/source-mongodb:latest ^
  sync ^
  --config /mnt/config/source.json ^
  --catalog /mnt/config/streams.json ^
  --destination /mnt/config/destination.json

Powershell

docker run --pull=always  `
  -v "$env:USERPROFILE\PATH_TO_OLAKE_DIRECTORY:/mnt/config" `
  olakego/source-mongodb:latest `
  sync `
  --config /mnt/config/source.json `
  --catalog /mnt/config/streams.json `
  --destination /mnt/config/destination.json

Locally run OLake

macOS / Linux

OLAKE_BASE_PATH="$HOME/PATH_TO_OLAKE_DIRECTORY/olake/drivers/mongodb/config" && \
./build.sh driver-mongodb sync \
  --config "$OLAKE_BASE_PATH/source.json" \
  --catalog "$OLAKE_BASE_PATH/streams.json" \
  --destination "$OLAKE_BASE_PATH/destination.json"

CMD

set "OLAKE_BASE_PATH=%USERPROFILE%\PATH_TO_OLAKE_DIRECTORY\olake\drivers\mongodb\config" && ^
./build.sh driver-mongodb sync ^
  --config "%OLAKE_BASE_PATH%\source.json" ^
  --catalog "%OLAKE_BASE_PATH%\streams.json" ^
  --destination "%OLAKE_BASE_PATH%\destination.json"

Powershell

$OLAKE_BASE_PATH = "$env:USERPROFILE\PATH_TO_OLAKE_DIRECTORY\olake\drivers\mongodb\config"; `
./build.sh driver-mongodb sync `
  --config "$OLAKE_BASE_PATH\source.json" `
  --catalog "$OLAKE_BASE_PATH\streams.json" `
  --destination "$OLAKE_BASE_PATH\destination.json"

To run sync with state:

OLake Docker

macOS / Linux

docker run --pull=always  \
  -v "$HOME/PATH_TO_OLAKE_DIRECTORY:/mnt/config" \
  olakego/source-mongodb:latest \
  sync \
  --config /mnt/config/source.json \
  --catalog /mnt/config/streams.json \
  --destination /mnt/config/destination.json \
  --state /mnt/config/state.json

CMD

docker run --pull=always  ^
  -v "%USERPROFILE%\PATH_TO_OLAKE_DIRECTORY:/mnt/config" ^
  olakego/source-mongodb:latest ^
  sync ^
  --config /mnt/config/source.json ^
  --catalog /mnt/config/streams.json ^
  --destination /mnt/config/destination.json ^
  --state /mnt/config/state.json

Powershell

docker run --pull=always  `
  -v "$env:USERPROFILE\PATH_TO_OLAKE_DIRECTORY:/mnt/config" `
  olakego/source-mongodb:latest `
  sync `
  --config /mnt/config/source.json `
  --catalog /mnt/config/streams.json `
  --destination /mnt/config/destination.json `
  --state /mnt/config/state.json

Locally run OLake

macOS / Linux

OLAKE_BASE_PATH="$HOME/PATH_TO_OLAKE_DIRECTORY/olake/drivers/mongodb/config" && \
./build.sh driver-mongodb sync \
  --config "$OLAKE_BASE_PATH/source.json" \
  --catalog "$OLAKE_BASE_PATH/streams.json" \
  --destination "$OLAKE_BASE_PATH/destination.json" \
  --state "$OLAKE_BASE_PATH/state.json"

CMD

set "OLAKE_BASE_PATH=%USERPROFILE%\PATH_TO_OLAKE_DIRECTORY\olake\drivers\mongodb\config" && ^
./build.sh driver-mongodb sync ^
  --config "%OLAKE_BASE_PATH%\source.json" ^
  --catalog "%OLAKE_BASE_PATH%\streams.json" ^
  --destination "%OLAKE_BASE_PATH%\destination.json" ^
  --state "%OLAKE_BASE_PATH%\state.json"

Powershell

$OLAKE_BASE_PATH = "$env:USERPROFILE\PATH_TO_OLAKE_DIRECTORY\olake\drivers\mongodb\config"; `
./build.sh driver-mongodb sync `
  --config "$OLAKE_BASE_PATH\source.json" `
  --catalog "$OLAKE_BASE_PATH\streams.json" `
  --destination "$OLAKE_BASE_PATH\destination.json" `
  --state "$OLAKE_BASE_PATH\state.json"

Find more about state file and its configuration here.

MongoDB to Iceberg Data Type Mapping

When syncing data from MongoDB to Iceberg, OLake handles data type conversions to ensure compatibility. Below is a table that outlines how MongoDB data types are mapped to Iceberg data types:

MongoDB Data Types	Iceberg Data Type
int, timestamp	int
long	bigint
No direct mapping observed	float
double	double
boolean	boolean
date	timestamp
string, object, objectId, binData (binary), code, regex (BSONRegExp), decimal128, maxKey, minKey, array, undefined	string

Changelog

Expand to review

Version	Date	Pull Request	Subject
v0.0.7	14.04.2025	https://github.com/datazip-inc/olake/pull/203
v0.0.10	31.04.2025	https://github.com/datazip-inc/olake/pull/250