Iceberg Delete Format Interoperability in OLake Go: Equality, Position & Deletion Vectors

Interoperability is coming to OLake Go!

One table, written once, readable by every engine in your stack. No more working out which of the three Iceberg delete formats your downstream engines will accept, just input your destination catalog, select your query engine and OLake Go does the rest.

Before we dive into what's new, let's take a look at what we're trying to improve.

Iceberg's Three Delete Formats and What Reads Them​

Whenever a record is deleted, Iceberg doesn't immediately remove it from the data file; instead, it creates another file in which it records which row is deleted, and it's up to the reader to apply that delete when reading.

Iceberg supports three ways of doing this: Equality Deletes and Position Deletes, included in the V2 spec, and Deletion Vectors, introduced with V3.

Let's look back at what each of these means.

• Equality deletes: these are like saying "delete any row where this value matches." So, if ID=123 identifies the row, the writer records that value and never has to find where the row actually sits. A reader has to check every row for a match.
• Position deletes: these take the opposite approach and store the file path and the position of the row to delete. A reader can go straight there, so reads are cheap, but the writer has to locate the row before it can record anything, making the writes costly.
• Deletion vectors: these are like position deletes, but instead of writing a delete file for each delete, Iceberg uses a bitmap per data file to track the rows that have been deleted. They also resolve a problem that position deletes produce at scale: the accumulation of small delete files.

Now that we've set the stage on what deletion methods Iceberg supports, let's take a look at what the other side, the query engines, can read.

Query Engine	Equality deletes	Position deletes	Deletion vectors
Apache Spark	Yes	Yes	Yes
Snowflake	No	Yes	Yes
Databricks	No	No	Yes
Athena	Yes	Yes	No
Trino	Yes	Yes	No

So the first thing you do is pick a format. You start with checking what each engine on your stack can read, and take the common intersection and redo this same exercise every time someone adds a new engine.

The second problem then shows up. Everything except equality deletes requires the writer to find the row before it can delete it, meaning you're reading data back just to write a delete. That's write amplification, and it gets worse as the table grows.

Most teams don't fight this. They write in append mode and move on. Nothing to pick, or look up, and every engine can read plain data files. But you haven't avoided the cost; you've postponed it. The table now holds every version of every row, and every query has to work out which one is current. Every consumer needs the same deduplication logic.

You can make any of this work. The point is that you shouldn't have to.

What's New With OLake Go?​

OLake Go takes the format decision off your hands. All it needs is the destination catalog and the list of query engines that need read access; OLake Go works out the rest.

Here's what changes with the latest update:

• Iceberg Delete Format Support: OLake Go previously supported only equality deletes, it now supports all three Iceberg delete formats: equality deletes, position deletes, and deletion vectors.
• Writes for your engines: OLake Go now takes query engines as inputs and writes data in a format that ensures compatibility across all the specified query engines.
• Indexed writes: Position deletes and Deletion vectors are made efficient using indexing.
• No more need of append-only writes and deduplication: Append-only writes were the only workarounds for interoperability, and they came with the cost of deduplication at read time. Now, OLake Go writes data in a format that is compatible with all the selected query engines, eliminating the need for append-only writes and read-time deduplication.
• Backward-compatible migration: Tables that have already been written in equality delete format can be moved to position delete or deletion vector formats without having to rebuild from the source.