Skip to main content
Last updated:|... min read
Bug iconOpen OLake issuesBug iconOpen OLake doc issue

Getting Started with OLake for MongoDB

OLake helps you replicate data from MongoDB into local or S3-based data lakes using Parquet or Iceberg table formats. This tutorial walks you through every step of the setup—from creating necessary configuration files to running your first data sync.

TLDR:​

  1. Create a config.json with your MongoDB connection details.
  2. Create a writer.json with your Writer (Apache Iceberg or AWS S3) connection details.
  3. Run discover to generate a catalog.json of available streams.
  4. Run sync to replicate data to your specified destination.

discover-sync

Introduction & Requirements​

To use OLake, ensure you have:

  • Docker installed and running on your machine.
  • MongoDB credentials (hosts, replica set name, username/password if applicable).
  • AWS S3 credentials (if you plan to write data to AWS S3).
  • Apache Iceberg and Catalog configuration credentials (if you plan to write data to Iceberg tables).

Refer here for more details on Writer requirements.

You will also need:

  • An empty directory to store OLake configuration files and outputs. This guide will refer to it as olake_directory.
note

For setting up the project locally on your system and debugging configs to be made, follow this guide - Setting up debugger in VS Code

Step 1: Prepare Your Directory​

  1. Create a new directory on your local machine. Let’s call it olake_directory:

    mkdir olake_directory

  2. Inside this folder, create two files:

    • writer.json: Specifies your output destination (local or S3).
    • config.json: Contains connection settings for MongoDB (or other databases in the future). 
    cd olake_directory
    touch writer.json
    touch config.json

Folder Structure:

olake_directory/
  ├─ writer.json
  └─ config.json

Directory

1.1 Example writer.json​

Refer to Writer config section for individual writers or refer them here..

  1. Apache Iceberg Writer
  2. AWS S3 Writer
  3. Local FileSystem Writer

1.2 Example config.json (MongoDB)​

Below is a sample config.json for connecting to a MongoDB replica set. Customize each field to match your environment.

olake_directory/config.json
{
  "hosts": [
    "host1:27017",
    "host2:27017",
    "host3:27017"
  ],
  "username": "test",
  "password": "test",
  "authdb": "admin",
  "replica-set": "rs0",
  "read-preference": "secondaryPreferred",
  "srv": true,
  "server-ram": 16,
  "database": "database",
  "max_threads": 50,
  "default_mode": "cdc",
  "backoff_retry_count": 2,
  "partition_strategy":""
}

Description of above parameters​

Refer to source configuration for more details on config.json.

Step 2: Generate a Catalog File​

OLake needs to discover which collections (streams) exist in your MongoDB. This step will create a catalog.json listing available streams, schemas, and default sync modes.

  1. Open your terminal in the same directory (say olake_directory) containing config.json and writer.json.
  2. Run the discover command using Docker:
docker run \
  -v "$HOME/PATH_TO_OLAKE_DIRECTORY:/mnt/config" \
  olakego/source-mongodb:latest \
  discover \
  --config /mnt/config/config.json
info

PATH_TO_OLAKE_DIRECTORY is the absolute path where you have created the directory [as discussed above]. -v "$HOME/PATH_TO_OLAKE_DIRECTORY:/mnt/config" \ maps to -v /Users/JOHN_DOE_USERNAME/Desktop/projects/olake-docker:/mnt/config \ in macOS and Linux systems. Follow the same pattern in other systems.

Catalog

Flag/ParameterDescription
discoverThe OLake sub-command that scans MongoDB schemas.
--config /mnt/config/config.jsonTells OLake where to find your MongoDB connection details.

2.1 Understanding the catalog.json File​

After running discover, OLake generates catalog.json in olake_directory with entries like:

olake_directory/catalog.json
{
    "selected_streams": {
        "otter_db": [
            {
                "partition_regex": "{now(),2025,YY}-{now(),06,MM}-{now(),13,DD}/{string_change_language,,}",
                "stream_name": "stream_0"
            },
            {
                "partition_regex": "{,1999,YY}-{,09,MM}-{,31,DD}/{latest_revision,,}",
                "stream_name": "stream_8"
            }
        ]
    },
    "streams": [
        {
            "stream": {
                "name": "stream_8",
                "namespace": "otter_db",
                "type_schema": { ... },
                "supported_sync_modes": [
                    "full_refresh",
                    "cdc"
                ],
                "source_defined_primary_key": [
                    "_id"
                ],
                "available_cursor_fields": [],
                "sync_mode": "cdc"
            }
        },
    // ... other streams
  ]
}
  • selected_streams: The streams / tables / collections OLake will replicate.
  • streams: Metadata for each discovered collection, including schemas and sync modes (e.g., cdc, full_refresh).
  • partition_regex: Specify the regex pattern. For more details, refer to S3 docs
tip

Exclude Streams: You can remove unneeded collections by editing selected_streams directly. For instance, deleting "customers" if you only want to sync orders.

Before (including customers):

olake_directory/catalog.json
"selected_streams": {
        "otter_db": [
            {
                "stream_name": "order",
                "partition_regex": ""
            },
            {
                "stream_name": "customer",
                "partition_regex": ""
            }
        ]
    },

After (to exclude customers):

olake_directory/catalog.json
"selected_streams": {
        "otter_db": [
            {
                "stream_name": "order",
                "partition_regex": ""  
            },
        ]
    },

Step 3: Run Your First Data Sync​

Now that you have catalog.json, it’s time to sync data from MongoDB to your specified destination (local or S3).

docker run \
  -v "$HOME/PATH_TO_OLAKE_DIRECTORY:/mnt/config" \
  olakego/source-mongodb:latest \
  sync \
  --config /mnt/config/config.json \
  --catalog /mnt/config/catalog.json \
  --destination /mnt/config/writer.json

First Data Sync

Flag/ParameterDescription
syncThe OLake sub-command that runs a data replication (snapshot + CDC).
--config /mnt/config/config.jsonMongoDB connection settings.
--catalog /mnt/config/catalog.jsonThe file detailing which streams OLake will replicate.
--destination /mnt/config/writer.jsonThe output configuration file (local or S3).
  • This command performs both the initial snapshot and, if configured, continuous CDC ("default_mode": "cdc").
  • If you only want a full one-time snapshot, set the stream’s sync_mode to "full_refresh" in catalog.json.
info

Example: If your sync_mode is "cdc", OLake will:

  1. Do a one-time full snapshot of each selected collection.
  2. Automatically begin listening to MongoDB’s oplog for near real-time changes.

When the sync finishes, you should see new files either:

  • Locally (in the volume-mapped directory).
  • On S3 (inside the specified s3_path).

Step 3.1 Synced Data​

If you are using VS Code, install a parquet reader extension to visualize the parquet file contents that will be made post sync process.

First Data Sync JSON

Step 3.2 Synced Data Normalized​

If you have turned on "normalization": true in your writer.json file, expect the below Level 1 Flattening of JSON data.

Read more about JSON flattening here - Flatten Object Types and Query Arrays in Semi-Structured Nested JSON

Running the sync command with normalization turned on​

JSON Normalized logs

Output Data Dump​

First Data Sync JSON Normalized

Step 3.3 Change output directory​

If you need to output the parquet dump to some other location, you can make changes in the writer.json file by appending the /mnt/config/my_directory

olake_directory/writer.json
{
    "type": "PARQUET",
       "writer": {
         "normalization":true, 
         "local_path": "/mnt/config/my_directory" 
    }
  }

Here, /mnt/config represents the olake_directory.

Step 4: Resume Sync with a State File​

If a sync is interrupted or you need to resume from a previous checkpoint, OLake automatically saves progress in a state.json file. Use the --state parameter to continue from that point:

docker run \
  -v "$HOME/PATH_TO_OLAKE_DIRECTORY:/mnt/config" \
  olakego/source-mongodb:latest \
  sync \
  --config /mnt/config/config.json \
  --catalog /mnt/config/catalog.json \
  --destination /mnt/config/writer.json \
  --state /mnt/config/state.json

Resume Sync with a State File

Flag/ParameterDescription
--state /mnt/config/state.jsonPoints OLake to an existing state file.
  • state.json typically includes a resume token (for MongoDB) or an offset for other databases, ensuring OLake does not reprocess records it has already synced.

A typical state.json file has the following structure:

state.json
{
  "type": "STREAM",
  "streams": [
    {
      "stream": "stream_8",
      "namespace": "otter_db",
      "sync_mode": "",
      "state": {
        "_data": "8267B34D61000000022B0429296E1404"
      }
    },
    {
      "stream": "stream_0",
      "namespace": "otter_db",
      "sync_mode": "",
      "state": {
        "_data": "8267B34D61000000022B0429296E1404"
      }
    }
  ]
}

In this example, "_data": "8267B34D6..." is a MongoDB resume token that tells OLake where to pick up the CDC stream.

For more details on the state.json configuration, refer the state docs

Debugging​

Follow the debugging instructions in this guide - Setting up debugger in VS Code

Docker Commands & Flags​

Click here for more info about Docker Commands & Flags

Next Steps & Wrap-Up​

  1. Check Your Output: Verify your Parquet files (or Iceberg tables) were created either locally or in your S3 bucket.
  2. Explore Schema Evolution: If your MongoDB documents gain new fields, OLake can adapt automatically. Watch for updated schemas in subsequent runs.
  3. Try More Destinations: OLake can also write to Iceberg on S3 (and more in the future). Update your writer config as needed.
  4. Analytics & Querying: Connect your newly created Parquet/Iceberg data to engines like Trino, Spark, or Presto for powerful querying.

Congratulations! You’ve completed your first OLake data replication. If you encounter any issues or have feedback, please visit our GitHub repository to open an issue or contribute.


Need Assistance?

If you have any questions or uncertainties about setting up OLake, contributing to the project, or troubleshooting any issues, we’re here to help. You can:

  • Email Support: Reach out to our team at hello@olake.io for prompt assistance.
  • Join our Slack Community: where we discuss future roadmaps, discuss bugs, help folks to debug issues they are facing and more.
  • Schedule a Call: If you prefer a one-on-one conversation, schedule a call with our CTO and team.

Your success with OLake is our priority. Don’t hesitate to contact us if you need any help or further clarification!

Join our growing community
GitHub
Slack
Twitter
LinkedIn
YouTube
LinkedIn
Join OLake Cloud waitlist
Low-cost
Easy Data Ingestion
Elastic Scaling
Join Waitlist