OLake (v0.2.10 - v0.3.4)

October 31 – November 28, 2025

🎯 What's New

Sources

1. New Source Introduction -
   Introduced Kafka as new source connector for OLake.

🔧 Bug Fixes & Stability

1. Clear destination S3 Parquet path handling -
   Fixed S3 Parquet path trimming in clear destination operations to properly handle whitespace in provided paths.

2. Increase Iceberg commit timeout -
   Extended Iceberg commit and write request timeout from 300s to 3600s to improve reliability for large data batches.

3. Clear destination stream cleanup fix -
   Corrected destination database checks so the streams are correctly detected and dropped during clear destination, ensuring complete cleanup.

4. Kafka consumer balancing -
   Updated balancer to assign consumer readers only to partitions with available data, preventing idle consumers from blocking on empty partitions and eliminating indefinite fetch waits until context cancellation.

5. LSN acknowledgment fix -
   Added a loop that repeatedly checks for the confirmed LSN to move forward, and stops after 5 minutes if it doesn’t, preventing hangs and making the process fail-safe.

6. Clear Destination Global State reset -
   When all persisted streams are dropped, mark the global state as nil; fail the sync on LSN mismatch; and fail incremental syncs when primary and secondary state remain nil throughout, preventing backfill-driven duplication.

7. Global State nil Check -
   Perform the empty-streams check only when global state is non-nil.

8. PostgreSQL initial wait timeout increase -
   Increased the initial wait time for the first PostgreSQL message by removing the previous 2400-second upper limit and raising the default timeout from 1200 seconds to 10800 seconds (3 hours) to better handle slow-start scenarios.

9. Telemetry event enrichment for syncs -
   Added extra telemetry details by including the sync ID in the sync-complete event and recording which streams ran as full refresh, improving observability of sync runs.