One Table, Every Engine: What It Takes to Make Iceberg Interoperable
16 min read
OLake helps companies adopt Apache Iceberg, moving operational databases into an open table format without handing the data to a proprietary warehouse on the way in.
This post is about the word open in that sentence, and why it often doesn't hold. An open table format promises that any engine implementing the spec can read your table. A table written by a CDC pipeline usually can't keep that promise: it is readable by Spark, and often by nothing else your company already pays for.
Iceberg is not at fault. The cause is a single decision the writer makes about how to record a deleted row, usually made without the information needed to make it well.
Engine and catalog support for Iceberg delete formats changes quickly. Everything in this post, including the compatibility tables, the open-source survey and the code we reference, reflects what we found up to September 2026.
The promise, and the thing that breaks it
A table format describes a table as files in object storage. Iceberg's metadata says which data files make up the table right now, what schema they follow, how they are partitioned, and which rows have been deleted. Data is Parquet, Avro or ORC. Metadata is JSON and Avro. Nothing needs a running service to interpret.
So the table is not owned by whoever wrote it. Land data once, then query it from Spark, Trino, Snowflake, DuckDB, all against the same bytes. No per-engine copy, no per-engine pipeline, no argument about which of four numbers is right.
That is the promise, and it is worth real money. But "every engine" is not something the spec can hand you. It depends on the overlap between what engines have actually built. The spec is one document; the engines are a dozen codebases at a dozen levels of maturity. Write something a reader has not implemented and you get a table that reader cannot query.
Sometimes it fails loudly, which is the good case. The bad case is an engine that returns rows you deleted.
Interoperability is the whole point. Take an example: your pipeline lands data fast, but into a table only Databricks can read. You haven't built an open lakehouse. You've built a Databricks table that happens to be stored as Parquet. You still pay the operational cost of running Iceberg, the manifests, snapshot expiry and compaction, and you still have the lock-in you adopted Iceberg to escape.
Deletes decide who can read your table
Three ways to say a row is gone
Change data capture (CDC) is mostly updates and deletes. Iceberg lets you record a deletion without rewriting the data file holding the row: you write a small side file that readers apply at scan time. There are three ways to write it, and they differ in how much the writer must know about where the row physically sits.
All three are part of Iceberg's merge-on-read (MOR) strategy: the writer records the deletion on the side and the reader merges it with the data at scan time, instead of copy-on-write rewriting the whole data file up front.
Equality delete. "Any row where id = 42 is deleted." Needs the key value, nothing else.
Positional delete. "Row 118,431 of data/00007-abc.parquet is deleted." Needs the exact file and the row's position inside it, its ordinal.
Deletion vector. Added in format version 3, where it replaces positional deletes outright: v3 tables no longer write positional delete files, and a new vector must absorb any positional deletes already present for that data file. Same information, better container: one bitmap per data file, stored in a Puffin file (Iceberg's format for blobs like this), marking every deleted ordinal at once instead of a new delete file per batch.
Equality deletes are cheap to write and expensive to read
This has nothing to do with implementation quality. It falls out of the scan-planning rules in the Iceberg spec, which decide when a delete file has to be applied to a data file.
A deletion vector applies to a data file when the file's path matches the vector's referenced_data_file and the partitions match. The vector names its file. Planning a scan over a thousand files is a thousand lookups.
A positional delete file works the same way, matched by referenced_data_file.
An equality delete has no file reference at all. It applies when the partitions match and the data file's sequence number, the counter Iceberg bumps on each commit, is lower than the delete's. So it covers everything written before it in that partition, and that is the whole scope. The delete file does not know which data files hold the rows it removes, and has no way to say.
So the reader applies every equality delete to every older data file in the partition, matching on values. One equality delete against a thousand candidate files means comparing the key against a thousand files. The min/max column statistics Iceberg stores prune some of them, but pruning starts from a set that is the whole partition. For the other two formats that set has one member by construction.
A second rule compounds it. Equality deletes written under an unpartitioned spec apply globally, to every data file in the table. Most CDC destinations are unpartitioned until somebody partitions them, so in practice every equality delete is a global delete. This is also why Dremio shows up as unsupported in the matrix below: Dremio skips global equality deletes.
Why write them at all, then? Because the writer does not know where the row is, and finding out costs something. Equality deletes are what you emit when you have no index.
The other two formats need an index
Positional deletes and vectors are nearly free to read. A sorted list of ordinals, or a bitmap, applied to one named file. No value comparison. Vectors go further and collapse many small delete files into one bitmap per data file, which is why the spec requires a new vector to absorb any existing positional deletes for that file.
All that convenience is paid for at write time, with one question the format cannot answer: for this key, which file and which ordinal?
Nothing in Iceberg's metadata maps a key to a row position. There is no API for it and no cheap derivation. Answering it from the table means scanning data files. If you want the lookup, you maintain it.
The scale is easy to underestimate. The index is not "the rows I am about to write." It is every live row in the table, because any of them could be the target of the next update. Replicate a 300 million row table and you need a 300 million entry index before the first positional delete, and you need it still there tomorrow, in a different process, on a different machine.
Who can read what, and why one bad file decides it
Which engine reads which delete format, as of September 2026:
| Engine | Equality | Positional | Deletion vectors |
|---|---|---|---|
| Apache Spark | ✓ | ✓ | ✓ |
| Apache Flink | ✓ | ✓ | ✓ |
| Trino | ✓ | ✓ | ✓ |
| DuckDB | ✓ | ✓ | ✓ |
| ClickHouse | ✓ | ✓ | ✓ |
| Presto | ✓ | ✓ | — |
| StarRocks | ✓ | ✓ | — |
| Apache Hive | ✓ | ✓ | — |
| AWS Athena | ✓ | ✓ | — |
| Google BigQuery | ✓ | ✓ | — ¹ |
| Snowflake | — | ✓ | ✓ |
| Dremio | — ² | ✓ | — ³ |
| Databricks | — | — | ✓ |
¹ Vectors in Preview only. ² Skips global equality deletes, which is what an unpartitioned table produces. ³ Vectors in Dremio Cloud only.
Two rows matter most. Snowflake does not read equality deletes at all. Databricks reads neither equality nor positional deletes, only vectors. Both names show up on a lot of enterprise lists, and between them they rule out the format streaming writers emit by default.
Here is what makes it unforgiving. A reader must resolve every delete format present in the table. There is no partial credit, because the only available fallback is to ignore a delete file it cannot parse, and ignoring one means returning deleted rows. A good engine fails the query instead.
So a table that is 99% positional deletes and 1% equality deletes is, to Snowflake, exactly as unreadable as one that is 100% equality deletes. Proportion is irrelevant. One unsupported delete file decides it.
Catalog bottlenecks
The engine is only half of it. The catalog, the service that tracks which tables exist and accepts commits to them, also decides what you are allowed to write, and that is product policy rather than anything in the spec. A managed REST catalog can happily accept table creation and appends while rejecting other operations outright.
| Catalog | Equality deletes | Positional deletes | Deletion vectors | Governance |
|---|---|---|---|---|
| Project Nessie | Yes | Yes | Yes | Limited |
| Hive Metastore | Yes | Yes | Yes | Limited |
| AWS Glue | Yes | Yes | Yes | Yes |
| Apache Polaris | Yes | Yes | Yes | Yes |
| Snowflake Horizon | No | Yes | Yes | Strong |
| Databricks Unity | No | No | Yes | Strong |
The two catalogs with the strongest governance are the two that accept the least. Both refuse equality deletes.
Read the delete columns against governance and the trade is clear. The open catalogs take anything and govern little. The managed ones govern properly and narrow what you may write, which is a reasonable product decision that happens to collide with how most CDC pipelines write.
Snowflake Horizon is the one we hit directly. It requires access delegation, and in our experience it will not accept equality-delete writes.
What open source offers today
Before building anything we went looking for something we could adopt instead. We worked through the open-source tools that write Iceberg tables, the CDC connectors and the table-maintenance jobs, checking what each one does when it has to delete a row.
Nothing writes the readable formats directly. The closest anything comes is recent: a maintenance task that landed in Apache Iceberg's Flink module in June 2026, and even that repairs tables after the fact rather than writing them correctly in the first place.
Why none of them can
Writing a positional delete or a deletion vector for a row you did not just write means answering one question: which file, which ordinal. That needs a durable index of the whole table, and no open-source ingestion tool keeps one.
So the formats every engine can read are the formats nothing open source will write for you.
Flink repairs the table afterwards, on v3 only
One offering gets close. Iceberg's Flink integration ships a convertEqualityDeletes maintenance task. The sink keeps writing equality deletes, and a separate job rewrites them into deletion vectors later.
The task repairs a table after the fact instead of writing it correctly, and it converts to vectors specifically, so it runs on format version 3 only. On v2 there is nothing on offer, and v3 brings its own problems, starting with Athena rejecting those tables outright.
One compaction and the index is gone
Before each run the job checks who else wrote to the table, by walking back through recent snapshots looking for its own marker. Find one it did not write, and it has to assume the index is stale, because another writer may have moved the files it points at. It cannot tell which entries went bad, so it throws away all of them and reads the whole table again. Compaction is such a commit, so the index is rebuilt from scratch on whatever cadence compaction runs.
What we built
You declare the readers, OLake picks the format
We turned the question around. Instead of picking a delete format and hoping readers cope, you declare which engines will read the table. OLake works out what it is allowed to write.
Writable formats are ordered cheapest first: equality, positional, vector. Equality needs no index. Positional needs a full key-to-location index. Vectors need that same index plus a v3 table. OLake checks that list against what every declared engine can read, keeps only the formats all of them handle, and picks the cheapest survivor. If nothing survives, it tells you at configuration time.
The behavior falls out of the data:
- All Spark. Equality deletes. No index, no bootstrap scan, no volume, nothing extra to operate. You do not pay for a problem you do not have.
- Add Snowflake. Equality drops out. Positional is selected and the index comes online.
- Add Databricks. Positional drops out too. Vectors are selected and the table moves to v3.
- Databricks and Dremio together. Empty. You hear about it at configuration time, not from a wrong dashboard six weeks later.
All three formats (equality, positional and deletion vector) are live in OLake (starting from version v1.0.0).
Index storage: how OLake maintains it
Two of the three formats need the same primitive: given an _olake_id, the primary-key field OLake generates, return the data file and ordinal where that row lives. The access pattern is narrow. Point lookups, no range scans, no query language, one entry per live row. Hundreds of millions of keys is normal.
A key-value store covers that. Three constraints decided which one.
It has to be embedded, because the sync is one Go binary in a container and a separate index service is another thing to deploy, secure and have fail. It has to be crash-consistent, because sync pods get killed routinely and a torn index does not fail loudly, it writes positional deletes against wrong rows. And it has to be pure Go, because we cross-compile for several platforms from one build. Last one rules out RocksDB, which is the obvious pick and the one Iceberg's docs recommend.
We use Pebble, a pure-Go LSM store written as a RocksDB replacement for CockroachDB. Same durability model, no cgo, and its crash-recovery path gets harder use under a distributed SQL database than anything we would build a test harness for. That recovery path is load-bearing here: Pebble rebuilds from its write-ahead log at open, so a hard pod kill leaves a volume it can recover from rather than a corrupt index.
Each stream has its own index. One Pebble store per stream, in its own directory, holding two logical tables:
- Paths. File path to file ID. Written once per data file.
- Rows.
_olake_idto file ID and ordinal. Written once per live row.
A lookup hits the rows table, gets a file ID, and resolves the path from the paths table. Separating them is what keeps the index small, below. Keeping stores separate means one stream's corruption or compaction storm cannot reach another's, and a single index can be dropped and rebuilt without touching the rest.
Following one row through its life. Take David, id = 2, and follow him through everything a CDC table goes through.
Insert: David's row arrives and lands in s3://a.parquet at position 1. OLake writes a.parquet to the paths table once, and David's _olake_id to the rows table pointing at that file and ordinal.
Update: A CDC record changes David's contact from +01 to +02.
OLake looks up David's _olake_id, gets back s3://a.parquet, position 1, and emits a positional delete naming exactly that file and ordinal. The updated row goes into a new file, say s3://b.parquet at position 0, and David's index entry now points there. A writer without the index knows only the key, so the best it can emit is id = 2 is deleted, an equality delete that Snowflake and Databricks will not read.
Crash: Now the sync pod is killed mid-way. A sync writes two things, the Iceberg commit and the index, and the crash can land between them:
- If Iceberg committed
b.parquetbut the index never recorded it, the next sync reconciles forward from the index's last recorded snapshot, reads the files added since, finds David inb.parquet, and fixes his entry. - If the Iceberg commit never happened, David's update never became visible to readers. The next sync replays the CDC record and writes his entry again.
Either way David's entry converges on what the table actually holds. A stale entry is never trusted, because it would send the next delete to a row that has moved.
Compaction: Later, Spark compacts the table, merging a.parquet, b.parquet and their delete files into s3://c.parquet. David's live row now sits in c.parquet at, say, position 7. On the next sync OLake reads only the newly added file, c.parquet, and updates David's entry to point there. Nothing else is rescanned. Flink's converter, at this same moment, throws away its whole index and rereads the entire table.
Keeping it small. The naive row entry stores the full file path. Object-storage paths run 100 to 200 bytes, and a few thousand of them get repeated across hundreds of millions of rows, so almost the whole index is duplicated strings.
Two things remove that. Each path is interned on first sight, stored once in the paths table against a 64-bit ID, and the row entry holds the ID instead. Both halves of the row entry are then written as uvarints: seven bits of value per byte, the eighth bit saying whether another byte follows, so small numbers cost small bytes. David's entry after compaction, file ID 3 and position 7, fits in two bytes instead of a fixed sixteen. Even a row deep in a large file, at position 118,431, needs only three bytes for its ordinal.
A 110 to 210 byte entry becomes two to six. At 2 billion records that is a ~9 GB index instead of ~30 GB.
Storage for Pebble. Pebble needs a real filesystem, so the index lives on a block volume attached to the sync pod rather than on shared network storage, where file locking is unreliable and an LSM store's habit of rewriting data during compaction gets expensive. In the Helm chart you set the volume size per job, and a job only gets a volume when its index is actually required, meaning only when the declared engines forced positional deletes or vectors.
The result
With the index in the write path, OLake writes positional deletes on a v2 table or vectors on a v3 one, whichever the declared readers resolve, and the table is correct for all of them the moment it is committed. No maintenance job, no window.
For tables that arrive with the problem already there, the same index drives a one-time repair during reconciliation, before the first record of the sync is written: existing equality delete files are rewritten as positional deletes and the sync proceeds from a clean state.
Every table is readable by the engines you declared when you created the job, which is the difference between a stack that is interoperable and one that is only open on paper.
