Skip to main content

Roadmap

New Connectors

  • Kafka
  • AWS S3

Observability & Monitoring Module

  • Real-time metrics and pipeline insights
  • Improved alerting and diagnostics

Upcoming Features

  • Heartbeat mechanism for sync health tracking
  • Incremental Sync Mode
  • Filtering Data in case of Full Load
  • SMTP’s for CDC
  • Helm deployments
  • Partition evolution

Detailed roadmap can be found on GitHub OLake Roadmap 2024-25

Source Connector Level Functionalities Supported

Connector Functionalities MongoDB Postgres MySQL
Full Refresh Sync Mode
Incremental Sync Mode
CDC Sync Mode
Full Parallel Processing
CDC Parallel Processing
Resumable Full Load
CDC Heartbeat

Additional Planned sources - AWS S3 | Kafka

Destination Level Functionalities Supported

Features/Functionality Local Filesystem S3 (raw dump) Iceberg Azure ADLS Google Cloud Storage
Flattening & Normalization (L1)
Partitioning
Schema ChangesWIPWIPWIP
Schema Evolution

Catalogue Support

Catalog typeDocs / exampleComments
REST – LakekeeperLINK[Officially Supported] Rust-native catalog with optimistic locking; Helm chart available for K8s.
REST - UnityLINK[Supported and tested] Unity Catalog (Databricks) with Personal Access Token authentication.
REST – GravitinoLINK[Supported, Yet to be tested] Uses standard Iceberg REST; Gravitino adds multi-cloud routing.
REST – NessieLINK[Supported, Yet to be tested] Time-travel branches/tags; supply nessie.endpoint in destination.json.
REST – PolarisLINK[Supported, Yet to be tested]
Hive MetastoreHive catalog configClassic HMS; good fit for on-prem Hadoop or EMR.
JDBC Catalog (Postgres/MySQL)JDBC catalog sampleStores Iceberg metadata in an RDBMS(Postgres); easiest to spin up locally with Postgres.
AWS Glue CatalogGlue catalog IAM & configBest choice on AWS; lets Athena, EMR, and Redshift query the same tables instantly.
Azure PurviewNot Planned, submit a request
BigLake MetastoreNot Planned, submit a request

See OLake Roadmap GitHub Discussion for more details on the kind of features we are planning to add in the future and see the current progress of those feature implementation for more details.


Need Assistance?

If you have any questions or uncertainties about setting up OLake, contributing to the project, or troubleshooting any issues, we’re here to help. You can:

  • Email Support: Reach out to our team at hello@olake.io for prompt assistance.
  • Join our Slack Community: where we discuss future roadmaps, discuss bugs, help folks to debug issues they are facing and more.
  • Schedule a Call: If you prefer a one-on-one conversation, schedule a call with our CTO and team.

Your success with OLake is our priority. Don’t hesitate to contact us if you need any help or further clarification!