Summary
The fifth OLake community meetup showcased new features, including Apache Iceberg as a destination for AWS S3 and local setups. The team demonstrated MongoDB to Iceberg sync capabilities, announced upcoming MySQL and Postgres sync features, and discussed the architecture of OLake—explaining how it syncs data from databases to data lakehouses using CDC. Performance improvements of 2-3 times faster syncs, upcoming UI functionality, orchestration tool integrations, and enhanced catalog support were also highlighted, alongside a strong call for open source contributions.
Chapters & Topics
Introduction and New Features
Priyansh Khodiyar introduced the fifth OLake community meetup, welcoming new members including Sanskar, Akash, and Harshvardhan. Shubham Baldava, co-founder and CTO of Datazip, also introduced himself and shared insights from his background in data engineering.
OLake UI Development
Priyansh discussed the ongoing development of the OLake UI led by Swati. He showcased the OLake front-end repository, noted that the master branch remains unchanged for now, and mentioned that Figma designs are available, with active work on both the initial setup and staging branches.
Apache Iceberg as Destination
Priyansh demonstrated the new feature that enables Apache Iceberg as a destination for both AWS S3 and local setups. He explained the setup process, including using Minio as a local replacement for S3 and configuring a JDBC catalog with Postgres, with detailed instructions provided in the documentation.
Iceberg Destination Functionality
Shubham explained how the Iceberg destination handles continuous processing of CDC data, achieving a sync latency of 10-20 seconds—ideal for data lakehouses. He also mentioned that the team is working on compaction solutions to manage the creation of many small files during frequent syncs.
Postgres Demo
Shubham conducted a live demo showing the process of syncing data from Postgres to Iceberg using OLake. The demonstration covered both full load and incremental syncs via CDC, including updating a record in Postgres and observing the change reflected in the Iceberg table.
Open Source Contributions
Priyansh encouraged community members to contribute to the project by exploring various repositories such as Oleg Docs and the main Oleg project. He highlighted good first issues for newcomers, while Shubham acknowledged current delays in PR reviews and promised improvements in review speed starting next week.
Action Items
- Priyansh Khodiyar will post the meeting recording on the OLake YouTube channel, publish sample videos demonstrating Postgres to Iceberg, Postgres to S3, MySQL to Iceberg, and MySQL to S3 syncs, and update documentation for building Olix Docker.
- Shubham Baldava will improve the PR review process and speed up reviews from next week.
- Community members are encouraged to check out good first issues on the OLake GitHub repository for contribution opportunities.