Use Cases for OLake

OLake is designed to facilitate efficient and reliable data replication from transactional databases to Apache Iceberg, enabling various data engineering and analytics workflows.

1. Offloading OLTP Databases for Analytics

Running complex analytical queries directly on production Online Transaction Processing (OLTP) databases can impact performance and stability for transactional workloads.

OLake provides a mechanism to replicate data from these databases (such as PostgreSQL, MySQL, and MongoDB) to an Iceberg-based data lake using Change Data Capture (CDC).

This separation allows analytical workloads to run on the Iceberg data without competing for resources with production applications. OLake supports a faster initial full load for bootstrapping the data lake and employs a Dead Letter Queue (DLQ) mechanism to handle schema type changes gracefully, contributing to pipeline reliability.

• Sync data from MySQL, Postgres, and MongoDB to Iceberg.
• Separate analytical workloads from production databases.
• Improve query performance by using systems optimized for analytics on the Iceberg data.
• Implement a Dead Letter Queue to handle unexpected schema changes during replication.
• Enable faster initial data synchronization using efficient full-load capabilities.

2. Building Open Data Stacks and Scaling Data Engineering

Organizations seeking to reduce dependency on proprietary data integration and warehousing solutions can leverage OLake as a component of an open-source data stack. By standardizing on Apache Iceberg as the data lake table format, OLake enables compatibility with a wide range of open query engines (e.g., Trino, Presto, Dremio, DuckDB, Spark).

Using an open-source connector like OLake allows engineering teams to avoid vendor lock-in, understand the system's mechanics, contribute to its development, or extend it for specific requirements. This approach supports a flexible, scalable, and open data architecture.

• Provide an open-source alternative to managed ETL/replication services.
• Enable extensibility and customization through its open codebase.
• Standardize data lake storage on the Apache Iceberg format.
• Avoid vendor lock-in associated with proprietary data formats and tools.
• Support diverse query engines for various use cases and user teams.

3. Enabling Near-Real-Time Analytics

Modern data applications often require data freshness measured in minutes, not hours. OLake facilitates near-real-time analytics by continuously replicating data from transactional databases using log-based CDC, typically achieving latency of less than 1 minute for changes to appear in Iceberg.

This continuous ingestion, combined with Iceberg features like incremental processing capabilities (facilitated by hidden partitioning) and query optimizations (metadata pruning, vectorized reads), supports fast, cost-efficient analytics on frequently updated data.

• Ingest data with low latency using log-based CDC.
• Support incremental data processing workflows with Apache Iceberg.
• Leverage Iceberg features for efficient query execution.

4. Cost-Effective Data Retention and Compliance

Storing large volumes of historical data for compliance, auditing, or future analysis can be expensive in cloud data warehouses. OLake helps address this by replicating data to Iceberg, which typically stores data on cost-effective object storage platforms (like S3, GCS).

Data stored in Iceberg via OLake remains immediately queryable through compatible engines, eliminating the need for separate rehydration steps. Apache Iceberg's support for schema evolution further aids long-term data retention by allowing changes to the data structure over time without disrupting access to historical versions.

• Store large historical datasets at lower costs compared to typical data warehouse storage.
• Maintain immediate queryability of retained data.
• Support compliance and auditing requirements with long-term, queryable data retention.
• Utilize Apache Iceberg's schema evolution capabilities to handle data structure changes over time.

5. Powering AI and ML Data Pipelines

Reliable access to fresh, high-quality structured data from transactional systems is crucial for building and training effective AI and Machine Learning models. OLake automates the process of bringing this data into an Iceberg-based lakehouse.

The continuous updates provided by OLake ensure that data used for ML feature stores or model training reflects recent activity. The compatibility of Iceberg with data processing engines commonly used in ML workflows (such as PySpark and DuckDB) facilitates integration into existing data science pipelines, supporting faster model iteration and development.

• Automate the ingestion of transactional data for ML feature stores.
• Provide a consistent data source for model training and evaluation.
• Integrate with common data processing engines used in ML workflows.

6. Simplifying Change Data Capture

Implementing robust and scalable CDC pipelines can be technically challenging. OLake offers a focused, open-source solution specifically for database-to-Iceberg replication.

OLake utilizes log-based CDC mechanisms, which typically have minimal impact on the source database's performance. It provides built-in support for handling schema evolution (changes to table structure) and includes a Dead Letter Queue to manage data type errors or other issues, enhancing the reliability of the CDC process. The architecture is designed to be compatible with open-source streaming concepts.

• Utilize log-based CDC for minimal impact on source systems.
• Support schema evolution to handle changes in table structures.
• Include a Dead Letter Queue for reliable error handling.

7. Reducing Cloud Data Warehouse Costs

Cloud data warehouses often incur significant costs based on factors like data volume stored and compute usage for querying. OLake provides a strategy to reduce these costs by enabling the offloading of raw, historical, or less frequently accessed data from the data warehouse to an Iceberg lakehouse stored on less expensive object storage.

This allows organizations to right-size their data warehouse usage for highly active or critical data, while still retaining access to the full dataset in the cost-efficient Iceberg layer.

• Reduce cloud data warehouse expenses by moving data to Iceberg.
• Leverage the lower cost of object storage for large datasets compared to proprietary warehouse storage.