Skip to main content

MySQL to Apache Iceberg: Binlog CDC into Polaris

MySQL to Apache Iceberg usually means Debezium, Kafka and a sink connector, or a Flink job that needs its own cluster. This guide takes an open source route with neither. OLake Go reads the MySQL binary log and commits the changes as Iceberg tables into an Apache Polaris catalog, with the files in your own bucket.

Polaris is an open source Iceberg REST (Representational State Transfer) catalog you run yourself. Apache Spark, DuckDB and Trino read the tables through that one endpoint, and in the run on this page all three returned the same rows.

By Shubham Satish BaldavaUpdated October 30, 2026

Key takeaways

  • OLake Go replicates MySQL into Iceberg tables in Polaris: a full historical snapshot first, then change data capture (CDC) from the binlog, with no Kafka, Debezium or Flink cluster in between.
  • Every sync is one Iceberg commit with Parquet data files and positional delete files, so updates land as row-level changes rather than duplicate rows.
  • The same tables are open to any engine that reads an Iceberg REST catalog. Spark, DuckDB and Trino returned identical rows and totals in the run on this page.

The short answer. For open source change data capture (CDC) from MySQL into Apache Iceberg, run OLake Go (Apache 2.0) with MySQL's binary log as the source and an Iceberg REST catalog such as Apache Polaris as the destination. The first sync copies each table in full; every later sync replays inserts, updates and deletes from the binlog. Spark, DuckDB and Trino then query the same tables.

How MySQL CDC reaches Apache Iceberg here​

Two things happen in order, and OLake Go does both on its own. The first sync records the current binlog position, then reads each table in parallel chunks and writes Parquet data files. Every later sync opens the binlog at the saved position and replays the inserts, updates and deletes since.

An update needs the old row retired. With positional deletes, OLake Go keeps an index of where each primary key sits (data file and row number) and writes a small delete file that points at those positions. Readers skip the retired rows without comparing keys, which is why every current Spark, DuckDB and Trino release reads this format.

One thing to keep an eye on: the binlog is MySQL's replay window. MySQL deletes binlog files after binlog_expire_logs_seconds, and a job paused for longer than that cannot resume. Keep retention longer than your longest planned pause, and alert on SHOW BINARY LOGS size if disk is tight.

What you need​

  • MySQL 8.0 or later with the binlog on in ROW format and a user that can read it. The tables on this page ran on MySQL 8.4. Hosted MySQL on Amazon Relational Database Service (RDS), Aurora, Google Cloud SQL or Azure works the same way; step 1 links the setup guides.
  • An Apache Polaris server and a bucket, both reachable from OLake Go and from your query engines. The run on this page used Polaris 1.8.0 and MinIO in Docker; on AWS, use an S3 bucket and an IAM role instead.
  • At least one engine that reads an Iceberg REST catalog. This page shows Apache Spark 3.5.6 with Iceberg 1.10.0, DuckDB 1.5.5 and Trino 484.
  • OLake UI from the Docker Compose install. The run on this page used OLake Go v1.1.0.
  • Docker, and about 40 minutes.

Step 1: Prepare MySQL for CDC​

Most readers point OLake Go at tables they already have. If you want to follow along with the exact data from this page, create these two tables first:

The two sample tables used on this page
Sample schema
CREATE TABLE customers (
  id INT AUTO_INCREMENT PRIMARY KEY,
  name VARCHAR(100), email VARCHAR(200), city VARCHAR(100), tier VARCHAR(20),
  created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
  updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP);

CREATE TABLE orders (
  id INT AUTO_INCREMENT PRIMARY KEY,
  customer_id INT, product VARCHAR(100), amount DECIMAL(10,2), status VARCHAR(20),
  created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
  updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  FOREIGN KEY (customer_id) REFERENCES customers(id));

-- Then insert whatever you like. This page started with 8 customers and 40 orders.

OLake Go reads MySQL changes from the binary log (binlog), the same stream MySQL replicas use. Four things must be in place on the server before the first sync.

SettingValueWhy it matters
log_binON with a server_idWithout a binlog MySQL records no row changes, so there is nothing to replicate. Changing it needs a restart.
binlog_formatROWStatement-based logging does not carry the row images OLake Go needs.
binlog_row_metadataFULLAdds column names and types to every event, so OLake Go can map changes to columns after a schema change.
Replication userSELECT, REPLICATION SLAVE, REPLICATION CLIENTThe snapshot reads the tables; the change stream reads the binlog.
On the MySQL server
-- 1. Binary logging: put these in my.cnf (or the parameter group on managed MySQL) and restart
--    server-id = 1
--    log_bin = mysql-bin
--    binlog_format = ROW
--    binlog_row_metadata = FULL
--    binlog_expire_logs_seconds = 604800   -- keep 7 days, so a paused job can catch up
--    gtid_mode = ON, enforce_gtid_consistency = ON   -- optional; gives microsecond _cdc_timestamp

-- 2. A user for OLake Go
CREATE USER IF NOT EXISTS 'olake'@'%' IDENTIFIED BY '<password>';
GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'olake'@'%';
FLUSH PRIVILEGES;

-- 3. Table statistics: the chunked snapshot plans its chunks from them
ANALYZE TABLE customers, orders;

Check the result before moving on:

sql
SHOW VARIABLES WHERE Variable_name IN ('log_bin', 'binlog_format', 'binlog_row_metadata', 'server_id');
SHOW GRANTS FOR 'olake'@'%';

Managed MySQL sets these through a parameter group or server setting rather than my.cnf. Each flavour has its own guide:

Where MySQL runsGuide
Amazon RDSRDS MySQL CDC setup
Amazon AuroraAurora MySQL CDC setup
Azure Database for MySQLAzure MySQL CDC setup
Google Cloud SQLCloud SQL MySQL CDC setup
Your own server or VMGeneric MySQL CDC setup
Docker, for a local testLocal MySQL with Docker Compose
Binlog retention is your replay window

MySQL deletes binlog files after binlog_expire_logs_seconds. If a job is paused longer than that, the next sync cannot find its saved position and needs a fresh full snapshot. Keep retention longer than your longest planned pause.

Step 2: Set up the Polaris catalog​

An Iceberg table is a set of files plus a catalog entry that says which metadata file is current. Apache Polaris holds those entries and the permissions; the Parquet files sit in your own S3 or MinIO bucket. OLake Go writes through Polaris, and every engine on this page reads through it with the credentials Polaris hands out.

The Polaris catalog page lists the other engines that read it. If you already run AWS Glue, Lakekeeper or another catalog, the Apache Iceberg overview shows what changes.

CatalogOLake Go writesApache Iceberg readsVerified on this page
Apache PolarisYesYes: any Iceberg REST client with OAuth2Yes, end to end

Apache Polaris is an open source Iceberg REST catalog. It keeps the table metadata pointers, hands out short-lived storage credentials to engines, and applies its own role-based access control. OLake Go writes to it through the standard Iceberg REST API, and ClickHouse, Trino, DuckDB and Spark read from the same endpoint. The files live in an S3 bucket or an S3-compatible store that you own.

Three things need to exist before OLake Go can write: a Polaris server, a catalog inside it that points at your bucket, and a principal (a client id and secret) with the right to manage that catalog. The run on this page used Polaris 1.8.0 and MinIO in Docker; on AWS, swap the MinIO endpoint for a plain S3 bucket and an IAM role.

docker-compose.yml (Polaris and MinIO, for a local test)
services:
  minio:
    image: minio/minio
    hostname: lake-minio
    command: server /data --console-address ":9001"
    environment: { MINIO_ROOT_USER: minioadmin, MINIO_ROOT_PASSWORD: minioadmin }
    ports: ["9000:9000", "9001:9001"]
  polaris:
    image: apache/polaris:latest
    depends_on: [minio]
    ports: ["8181:8181", "8182:8182"]
    environment:
      AWS_REGION: us-east-1
      AWS_ACCESS_KEY_ID: minioadmin
      AWS_SECRET_ACCESS_KEY: minioadmin
      POLARIS_BOOTSTRAP_CREDENTIALS: POLARIS,root,<root-secret>
      polaris.realm-context.realms: POLARIS

Polaris stores its own state in memory with this setup, so it starts empty every time; a production deployment points it at Postgres. Once it is up, create the bucket, the catalog and the principal through the management API. root is the bootstrap principal; the olake principal is the one OLake Go and the engines use.

Polaris, once
P=http://localhost:8181

# 1. A bucket for the tables (MinIO; on AWS create it in the console or with aws s3 mb)
aws --endpoint-url http://localhost:9000 s3 mb s3://lakehouse

# 2. A root token
TOKEN=$(curl -s $P/api/catalog/v1/oauth/tokens -H 'Polaris-Realm: POLARIS' \
  -d grant_type=client_credentials -d client_id=root -d client_secret=<root-secret> \
  -d scope=PRINCIPAL_ROLE:ALL | jq -r .access_token)
H=(-H "Authorization: Bearer $TOKEN" -H 'Polaris-Realm: POLARIS' -H 'Content-Type: application/json')

# 3. The catalog: where the files go, and how Polaris and the engines reach the store
curl -s "${H[@]}" $P/api/management/v1/catalogs -d '{"catalog": {
  "name": "olake", "type": "INTERNAL", "readOnly": false,
  "properties": {"default-base-location": "s3://lakehouse/olake"},
  "storageConfigInfo": {"storageType": "S3", "allowedLocations": ["s3://lakehouse/olake"],
    "endpoint": "http://<minio-host>:9000/", "endpointInternal": "http://lake-minio:9000",
    "pathStyleAccess": true, "region": "us-east-1"}}}'

# 4. A principal for OLake Go and the engines. The response holds clientId and clientSecret: keep them.
curl -s "${H[@]}" $P/api/management/v1/principals -d '{"principal": {"name": "olake"}}'

# 5. Roles and the grant that lets the principal create namespaces and tables in the catalog
curl -s "${H[@]}" $P/api/management/v1/principal-roles -d '{"principalRole": {"name": "olake_role"}}'
curl -s "${H[@]}" $P/api/management/v1/catalogs/olake/catalog-roles -d '{"catalogRole": {"name": "olake_catalog_role"}}'
curl -s -X PUT "${H[@]}" $P/api/management/v1/principals/olake/principal-roles -d '{"principalRole": {"name": "olake_role"}}'
curl -s -X PUT "${H[@]}" $P/api/management/v1/principal-roles/olake_role/catalog-roles/olake -d '{"catalogRole": {"name": "olake_catalog_role"}}'
curl -s -X PUT "${H[@]}" $P/api/management/v1/catalogs/olake/catalog-roles/olake_catalog_role/grants -d '{"type": "catalog", "privilege": "CATALOG_MANAGE_CONTENT"}'

What each block is for:

  • endpoint is the store address Polaris hands to engines along with vended credentials; endpointInternal is the address Polaris itself uses. On AWS S3 leave both out. On MinIO both are required, or Polaris tries to reach AWS and fails. Keep the trailing slash on endpoint: ClickHouse 26.8 and 26.10 build the object path from it, and without the slash every read fails with a 404 (the fix is explained under troubleshooting on this page).
  • Polaris vends credentials through the store's STS endpoint. MinIO exposes one, so nothing else is needed. For a store without STS, add "stsUnavailable": true and give Polaris static keys to hand out.
  • CATALOG_MANAGE_CONTENT lets the principal create namespaces and tables and read and write their data. Scope it to a namespace instead when several teams share the catalog.
  • The principal's secret is shown once, in the response to step 4. It is what goes into OLake Go as <client_id>:<client_secret>.
Hostnames with underscores break Polaris

Polaris is a Java service, and Java's URI parser rejects hostnames that contain an underscore. A MinIO container named lab_minio makes every table creation fail with host must not be null or Expected scheme-specific part at index 5: http:. Use a hyphen in the hostname (lake-minio), as above.

You can check the principal outside OLake Go by exchanging its credentials for a token, the same call OLake Go makes on every sync:

Token check
curl -s http://localhost:8181/api/catalog/v1/oauth/tokens -H 'Polaris-Realm: POLARIS' \
  -d grant_type=client_credentials -d client_id=<client_id> -d client_secret=<client_secret> \
  -d scope=PRINCIPAL_ROLE:ALL

A good principal returns a JSON object with an access_token. Running Polaris on Kubernetes, with Postgres behind it, or with an external identity provider is covered on the Polaris catalog page.

Step 3: Connect MySQL to OLake Go​

  1. In the OLake UI, open Sources and click Create Source. Pick MySQL as the connector, give the source a name and keep the OLake Go version on the latest release.
  2. Under Change data capture (CDC), leave Enable CDC on. The default Initial Wait Time is fine; raise it only if the binlog is large and the next sync times out while finding its saved position.
  3. Under Connection details, fill in the host, port, database name, username and password. Keep TLS off for a local test; on managed MySQL use the provider's certificate.
  4. Click Set up source. OLake Go tests the connection, then checks the binlog prerequisites (binlog_access, log_bin, binlog_format, binlog_row_metadata) and shows the current and recommended value for anything that fails.

Here is the source form from the run on this page. MySQL ran in another container, so the host is host.docker.internal; the user is the replication user from step 1.

OLake UI Edit source form for MySQL on OLake Go v1.1.0 with Enable CDC on, host host.docker.internal, database shop and user olake
The MySQL source in OLake UI: Enable CDC on, with the replication user from step 1.

Set up source runs the connection test and the binlog checks (binlog_access, log_bin, binlog_format, binlog_row_metadata) before anything is saved. When one fails, the dialog names the setting and its recommended value.

The full field reference, including SSL and SSH tunnels, is on the MySQL connector page.

Step 4: Configure the Iceberg destination​

  1. Open Destinations and click Create Destination.
  2. Pick Apache Iceberg as the connector and Polaris as the catalog type.
  3. Choose OAuth2 as the authentication type. REST Catalog URI is the Polaris catalog endpoint (http://<polaris-host>:8181/api/catalog), Polaris Catalog Name the catalog from the setup (olake), REST Auth URI the token endpoint, and REST Credential the principal's <client_id>:<client_secret>.
  4. Fill in the store: AWS Region, and for MinIO the S3 Endpoint with the AWS Access Key and AWS Secret Key. On AWS S3 leave the endpoint empty.
  5. Click Create. OLake Go exchanges the credentials for a token, creates a namespace and a test table through Polaris, writes to the bucket and drops the test table again.

Here is the destination form from the run on this page. Polaris and MinIO ran in other containers, so both addresses use host.docker.internal; the credential is the olake principal from step 2.

OLake UI Edit destination form for Apache Iceberg with catalog type Polaris, OAuth2 authentication, REST catalog URI on port 8181, catalog name olake and the MinIO S3 endpoint
The Polaris destination in OLake UI: OAuth2 with the principal from step 2, and the MinIO endpoint for the files.

Polaris Configuration Parameters

ParameterSample ValueDescription
REST Catalog URI
required		http://<REST_ENDPOINT>:8181/api/catalogEndpoint URL for the Polaris REST catalog service.
Authentication Type
required		oauth2Authentication method OLake Go uses for Polaris REST catalog requests. Set to match your catalog's authentication configuration.
Polaris Catalog Name
required		<POLARIS_CATALOG_NAME>Name of the Polaris catalog to use, OLake Go passes this to the Polaris REST catalog as the warehouse identifier, so catalog operations are routed to that catalog and its pre-configured S3 storage and IAM settings.
REST Auth URI
required		http://<REST_ENDPOINT>:8181/api/catalog/v1/oauth/tokensURL of the REST catalog’s OAuth2 token endpoint. OLake Go uses this with REST Credential to request an access token during OAuth2 client-credentials authentication.
REST Credential
required		<olake_user_client_id>:<olake_user_client_secret>Specifies the client ID and secret for OAuth2, formatted as client_id:client_secret. Used with REST Auth URI when OLake Go requests an access token from the catalog’s auth service. Not needed when using REST Token or other auth types.
AWS Region
required		ap-south-1AWS Region where your S3 bucket exists. Required so OLake Go calls the correct regional S3 endpoint.
REST ScopePRINCIPAL_ROLE:ALLRequired for OAuth2 client-credentials auth. Scopes are space-separated. Used with REST Auth URI and REST Credential for OAuth2 client-credentials auth. Values are catalog-specific. Not needed when using REST Token or other auth types.
S3 Endpointhttp://<S3_ENDPOINT>:9000S3 API endpoint for writing Iceberg data files. Leave empty for AWS S3 but required for S3-compatible storage like MinIO.
AWS Access KeyXXXAWS access key ID for authentication. Optional if using IAM roles or instance profiles.

If using IAM:
Docker Compose: add the required IAM environment variables under x-envs in your compose file. See Service Environment Variables for setup details.
Kubernetes: set up pod IAM in Cloud Integration.
AWS Secret KeyXXXAWS secret access key for authentication. Optional if using IAM roles or instance profiles.

If using IAM:
Docker Compose: add the required IAM environment variables under x-envs in your compose file. See Service Environment Variables for setup details.
Kubernetes: set up pod IAM in Cloud Integration.
Enable Arrow WritesfalseWrites data and delete files using Apache Arrow based writer and registers them in Iceberg.
Catalog Nameolake_icebergName of the Iceberg catalog OLake Go registers tables under. Defaults to olake_iceberg if left empty.
Catalog Name Supported for v0.3.5 and above
For the catalog name, OLake Go only supports lowercase letters and underscores. Spaces and special characters are not supported.

Step 5: Create the job and run the first sync​

  1. Open Jobs and click Create Job. Give it a name and a schedule. Every Day at a fixed hour, or a cron expression, both work; the schedule is how often the change stream is drained, not how often a full copy runs.
  2. Select the source and the destination you created.
  3. Under Advanced Settings, set Target Query Engines to Apache Spark, DuckDB and Trino, or whichever engines will read the tables. OLake Go then resolves an Iceberg delete format every one of those engines can read, so you never pick one by hand.
  4. Click Next. OLake Go discovers the tables and opens the Streams page.
  5. Tick the tables to replicate. For each one, set the sync mode to Full Refresh + CDC and the ingestion mode to Upsert. Leave the Upsert Type as OLake Go resolved it (Positional Deletes in the run on this page).
  6. Click Create Job, then open the job's menu and choose Sync Now for the first run.

Three target query engines were set on this job: Apache Spark, DuckDB and Trino. All three read every Iceberg delete format, and the run on this page used positional deletes on Iceberg format version 2 for every stream.

OLake UI Streams Selection page with the shop database, customers and orders selected, Full Refresh + CDC, Upsert and Positional Deletes on Iceberg format version V2
Each stream: Full Refresh + CDC, Upsert, and Positional Deletes on Iceberg v2. The Iceberg DB name at the top is editable.
OLake UI Job Logs and History for mysql_to_iceberg showing two completed syncs of 9 seconds each
Job history: the 9 s historical snapshot, then the 9 s change-only sync.

On this page the first run copied 8 customers and 40 orders (48 records, 2.10 KB) and finished in 9 seconds, container start included. OLake Go names the destination namespace after the job and the source database, so mysql_to_iceberg on database shop produced mysql_to_iceberg_shop. The Iceberg DB field at the top of the Streams page changes it before the first sync.

Step 6: Query the tables from DuckDB​

DuckDB is the quickest way to check the result from a laptop: one ATTACH and the whole catalog is visible.

DuckDB attaches Polaris as an Iceberg REST catalog with the principal's client id and secret; DuckDB fetches the token itself. On a laptop, where the store address Polaris hands out may not resolve, add an S3 secret for the store and turn credential vending off:

DuckDB 1.5.5
INSTALL iceberg; LOAD iceberg;
INSTALL httpfs;  LOAD httpfs;

CREATE SECRET polaris_secret (
TYPE iceberg,
CLIENT_ID '<client_id>', CLIENT_SECRET '<client_secret>',
OAUTH2_SERVER_URI 'http://<polaris-host>:8181/api/catalog/v1/oauth/tokens',
OAUTH2_SCOPE 'PRINCIPAL_ROLE:ALL'
);
-- MinIO, or any store whose address differs from the one Polaris vends
CREATE SECRET store_secret (
TYPE s3, KEY_ID '<access key>', SECRET '<secret key>',
ENDPOINT '<minio-host>:9000', URL_STYLE 'path', USE_SSL false, REGION 'us-east-1'
);

ATTACH 'olake' AS polaris (
TYPE iceberg,
ENDPOINT 'http://<polaris-host>:8181/api/catalog',
SECRET polaris_secret,
ACCESS_DELEGATION_MODE 'none'
);

SHOW ALL TABLES;
SELECT id, name, city, tier FROM polaris.mysql_to_iceberg_shop.customers ORDER BY id;

With Polaris on AWS S3, drop the S3 secret and the ACCESS_DELEGATION_MODE line: DuckDB then reads with the credentials Polaris vends. DuckDB reads equality and positional deletes, and deletion vectors from duckdb-iceberg 1.5.3 onward.

DuckDB 1.5.5 shell querying polaris.mysql_to_iceberg_shop.customers and returning 10 rows with id, name, city and tier
DuckDB 1.5.5 reading the customers table through Polaris after the change sync: 10 rows, one per key.

Step 7: Change rows in MySQL and sync again​

To see the change path work, the run on this page made 12 changes in MySQL after the first sync:

Changes made in MySQL between the two syncs
INSERT INTO customers (name, email, city, tier) VALUES
  ('Maya Patel', 'maya@example.com', 'Pune', 'silver'),
  ('Lucas Silva', 'lucas@example.com', 'São Paulo', 'bronze');
INSERT INTO orders (customer_id, product, amount, status) VALUES
  (9, 'Laptop', 1299.00, 'placed'), (9, 'Mouse', 25.50, 'placed'),
  (10, 'Monitor', 349.99, 'placed'), (3, 'Keyboard', 89.00, 'placed'), (7, 'Webcam', 79.90, 'placed');
UPDATE customers SET tier = 'gold'      WHERE name = 'Liam Carter';
UPDATE customers SET city = 'Abu Dhabi' WHERE name = 'Zara Ali';
UPDATE orders    SET status = 'delivered' WHERE id IN (1, 4, 7);

The second sync read 12 records (563 bytes) from the binlog and finished in 9 seconds end to end. For customers that meant one new data file with 4 rows: 2 inserts plus the new versions of 2 updated rows. One positional delete file retires the 2 old versions by file and row number.

OLake Go's commit report in the sync log shows it directly: addedDataFiles=1, addedRecords=4, addedPositionalDeleteFiles=1, addedPositionalDeletes=2, and no equality delete files. The orders table has the same shape with 8 rows and 3 deletes.

OLake UI sync log filtered to the Iceberg commit report for customers, showing addedDataFiles 1, addedPositionalDeleteFiles 1, addedRecords 4 and addedPositionalDeletes 2
The change-only sync in the OLake UI log: one data file and one positional delete file per table.

Every engine shows the result on its next query, with no refresh. Liam Carter is gold, Zara Ali moved to Abu Dhabi, and the two new customers are there. The _op_type column shows how each row arrived: r for the snapshot, c for the inserts and u for the updates.

Schema changes. When a column is added in MySQL, OLake Go adds it to the Iceberg table on the next sync, and readers see it the next time they load the table. Widening changes (int to bigint, float to double) are applied.

Incompatible type changes fail the sync rather than corrupt the table. binlog_row_metadata = FULL is what lets OLake Go see new column names in the binlog. Full rules are on the schema evolution page.

Query the same tables from Trino and Spark​

Nothing is exported or copied for a second engine. Trino and Spark read the files OLake Go wrote, through the same Polaris endpoint.

Trino reaches Polaris through its Iceberg connector in REST mode. One catalog properties file does it; Trino fetches the OAuth2 token with the principal's credentials and reads the files with the credentials Polaris vends.

etc/catalog/polaris.properties
connector.name=iceberg
iceberg.catalog.type=rest
iceberg.rest-catalog.uri=http://<polaris-host>:8181/api/catalog
iceberg.rest-catalog.warehouse=olake
iceberg.rest-catalog.security=OAUTH2
iceberg.rest-catalog.oauth2.credential=<client_id>:<client_secret>
iceberg.rest-catalog.oauth2.scope=PRINCIPAL_ROLE:ALL
iceberg.rest-catalog.oauth2.server-uri=http://<polaris-host>:8181/api/catalog/v1/oauth/tokens
iceberg.rest-catalog.vended-credentials-enabled=true
fs.native-s3.enabled=true
s3.region=us-east-1
# MinIO or another S3-compatible store only:
s3.endpoint=http://<minio-host>:9000
s3.path-style-access=true
trino
SHOW SCHEMAS FROM polaris;
SELECT id, name, city, tier FROM polaris.mysql_to_iceberg_shop.customers ORDER BY id;

Trino 484 read the tables on this page with no further settings. Equality and positional deletes are read on every current release; deletion vectors from Trino 480, where the docs still mark them experimental.

Trino 484 CLI querying customers in the polaris catalog schema mysql_to_iceberg_shop and returning the same 10 rows as DuckDB
Trino 484 reading the same table through Polaris: the same 10 rows.

All three engines agreed on the counts and the money: 10 customers, 45 orders and 25579.19 in total. The orders roll-up matched MySQL to the cent: 16 delivered orders for 8659.68, 18 placed for 10522.26, 11 shipped for 6397.25.

Data types: MySQL to Iceberg​

OLake Go maps MySQL types to Iceberg types. Most columns pass through unchanged; three groups are worth knowing about.

MySQLIceberg (written by OLake Go)Trino showsNotes
int, mediumint, smallint, tinyint (signed or unsigned)intinteger
bigint, bigint unsignedlongbigint
decimal(p,s), float, doubledoubledoubleOLake Go stores decimals as doubles. The orders.amount sums on this page matched MySQL to the cent in every engine; round at query time when you need exact decimal arithmetic.
datetime, timestamptimestamptz (UTC)timestamp(6) with time zonedatetime has no zone in MySQL; it is read as UTC.
char, varchar, text family, enum, json, time, bit(1)stringvarchartime arrives as text.
binary(n), varbinary, blob familyfixed[n] or binaryvarbinary

MySQL types in the table come from the MySQL connector reference; the Trino column comes from DESCRIBE on the tables of this run.

Cost and maintenance​

  • Storage is your bucket: you pay S3 or MinIO rates for the Parquet and delete files, and nothing to Polaris for holding pointers.
  • Compute. OLake Go runs wherever you run Docker. Each engine spends compute only when it queries, and none of them keeps its own copy.
  • Maintenance. Nothing compacts the tables for you. A CDC table that gets a new small file and delete file per sync wants compaction; OLake Fusion does this for the catalogs OLake Go writes to, and Spark's rewrite_data_files works through the same Polaris endpoint.
  • Binlog disk. Seven days of binlog on a busy server can be tens of gigabytes. Size the volume for it, or shorten retention once the job runs reliably every hour.
  • Credential rotation. Polaris principals do not expire by default. Rotate the olake principal's secret on your own schedule; a rotated secret fails the next sync at the token exchange, before any data moves.

Troubleshooting​

1. The first sync stops with a statistics error​

failed to get or split chunks: stats not populated for table[shop.orders]. Please run ANALYZE TABLE to update table statistics

OLake Go plans the parallel snapshot from information_schema row estimates. A freshly created or freshly loaded table has none. Run ANALYZE TABLE <table>; for each table in the job, then sync again.

2. The source test fails on the binlog​

binlog_format: current STATEMENT, recommended ROW

The test dialog names the setting. binlog_format and binlog_row_metadata can be changed at runtime with SET GLOBAL.

Put them in my.cnf or the parameter group too. Otherwise they revert on restart. log_bin itself needs a restart.

3. A new job over existing tables writes nothing​

Table[mysql_to_iceberg_shop.customers]: not found starting index snapshot id, truncating stream index before rebuild
Thread[shop.customers_min[1]-max[<nil>]]: chunk min[1] and max[<nil>] already committed, skipping
Reached the configured latest binlog position mysql-bin.000003:2809; stopping CDC sync

The run on this page hit this after a first attempt failed and the job was recreated under the same name. The Iceberg tables from that attempt were still in Polaris. OLake Go treated their snapshot chunks as already committed and skipped them.

The change sync then read 0 records. Drop the old tables (or give the job a new Iceberg DB name) before a fresh start, and the snapshot and change syncs run normally.

4. The next sync cannot find its binlog position​

idle timeout reached, exiting binlog syncer

Two causes. If binlog_expire_logs_seconds already removed the saved binlog file, run a fresh full snapshot and lengthen retention. If the file exists but the server is slow to seek to the position, raise Initial Wait Time on the source.

5. The destination test fails at the token exchange​

OLake Go's destination test exchanges the principal's credentials at /api/catalog/v1/oauth/tokens before anything else. Three causes cover nearly every failure: a wrong client id or secret, a principal without a role that carries CATALOG_MANAGE_CONTENT on the catalog, or a wrong Polaris Catalog Name. Run the token check from the catalog setup; it returns the same error Polaris gives OLake Go.

How this compares with other ways to move MySQL data to Apache Iceberg
MethodFreshnessUpdates and deletesWhat you run
Debezium, Kafka and Kafka Connect with an Iceberg sinkSet by the sink commit intervalAppended as change events, with a separate merge job for the current stateDebezium, Kafka, Kafka Connect and usually a schema registry
Flink CDC with the Iceberg Flink sinkSet by the checkpoint intervalUpserts through equality deletes on tables with a primary keyA Flink cluster with checkpoint storage
Managed ELT with an Iceberg destinationMinutes, by planVaries by toolNothing to run; priced per row, per GB or per connector
Periodic mysqldump or export to Parquet, then a load jobHoursFull reload or hand-written mergeA scheduler and your own scripts
OLake Go to Iceberg on Polaris (this guide)CDC on a schedule down to one minuteUpserts through positional deletes, equality deletes or deletion vectors, matched to the target enginesOne Docker Compose stack, open source; your bucket plus your own compute

Debezium and Kafka make sense when Kafka already carries the change events for other consumers. Flink wins when you need joins, enrichment or second-level latency on the way in. For copying MySQL tables into Iceberg with the least infrastructure, OLake Go is one job and one catalog; the trade-off is that freshness is a schedule, not a stream.

When not to use this​

  • You need sub-minute freshness. OLake Go runs on a schedule, down to one minute between syncs. A streaming pipeline is the better fit for second-level latency.
  • You need transformations on the way in. OLake Go copies tables as they are, with row filters and partitioning. Joins and enrichment belong in a query engine after the data lands, or in a stream processor before it.
  • Tables have no primary key. Inserts replicate; updates and deletes cannot be matched to a row and are not applied. Add a primary key or a unique index first.

FAQs​

What is an open source way to do CDC from MySQL into Apache Iceberg?

OLake Go is open source under Apache 2.0 and reads the MySQL binary log directly, so the pipeline is one job and one Iceberg catalog. The first sync copies the tables in full, every later sync replays inserts, updates and deletes from the binlog, and each sync lands as one Iceberg commit. Debezium with Kafka Connect and Flink CDC are the other open source routes; both need a Kafka or Flink cluster to run.

Do I need Kafka, Debezium or Spark to get MySQL changes into Iceberg?

No. OLake Go reads the binlog itself and has its own Iceberg writer, which uploads the Parquet data and delete files and commits the snapshot to the catalog. Spark, DuckDB and Trino are only readers on this page.

Which Iceberg delete format does OLake Go write for Spark, DuckDB and Trino?

Spark, DuckDB and Trino all read equality deletes, positional deletes and deletion vectors, so every format was open to this job. The run on this page used positional deletes on Iceberg format version 2. Every current release of the three engines reads them, while Trino reads deletion vectors only from release 480, where its docs still mark them experimental.

Does OLake Go copy the existing MySQL data before it starts reading the binlog?

Yes, automatically. A job in Full Refresh + CDC mode records the binlog position, copies every selected table in parallel chunks, then switches to the binlog from that position. Nothing written during the copy is lost, and there is no separate backfill job.

Does MySQL CDC with OLake Go work on Amazon RDS, Aurora or Cloud SQL?

Yes, as long as the binlog is on with row format. Managed MySQL sets binlog_format, binlog_row_metadata and retention through a parameter group or flag instead of my.cnf; step 1 links the setup guide for each. On RDS the binlog is on once automated backups are enabled; set binlog_format to ROW in the parameter group.

How much binlog does MySQL need to keep for OLake Go?

Enough to cover your longest pause. OLake Go saves the binlog position after every sync and resumes from it; if MySQL has already expired that file, the next sync cannot resume and needs a fresh snapshot. Seven days (binlog_expire_logs_seconds = 604800) is a safe default for a job that runs at least daily.

MySQL to SnowflakeSame source, read from Snowflake.MySQL to ClickHouseSame source, read from ClickHouse.MySQL to DuckDBSame source, read from DuckDB.MySQL to BigQuerySame source, read from BigQuery.MySQL to Amazon RedshiftSame source, read from Amazon Redshift.MySQL to DatabricksSame source, read from Databricks.Postgres to Apache IcebergSame engine, replicated from Postgres.MongoDB to Apache IcebergSame engine, replicated from MongoDB.Oracle to Apache IcebergSame engine, replicated from Oracle.SQL Server to Apache IcebergSame engine, replicated from SQL Server.Kafka to Apache IcebergSame engine, replicated from Kafka.MySQL connector referenceEvery field, sync mode and troubleshooting note for the source.Query engine compatibilityWhich engines read which catalogs and delete formats.All integrationsEvery source and engine pair with a tested guide.

Deeper reading on this site: