Skip to main content

Apache Spark and Apache Iceberg: OLake Go setup

Apache Spark with the Iceberg runtime reads every catalog OLake Go writes to. It also reads every Iceberg delete format.

Short answer. To read replicated tables in Spark, run OLake Go with any Iceberg catalog, then configure a Spark catalog of the matching type, such as type=glue or type=rest. Spark reads equality deletes, positional deletes and, with Iceberg 1.8 or later on the classpath, deletion vectors.

Key takeaways

  • Spark with the Iceberg runtime reads every catalog OLake Go writes to, such as AWS Glue and Polaris.
  • It reads equality deletes, positional deletes and, with Iceberg 1.8 or later, deletion vectors.
  • Spark is the cross-check reader on many guides, for example Postgres to Iceberg.

Last reviewed October 2026.

How OLake Go gets data into Apache Spark​

OLake Go uses change data capture (CDC) where the source has it and commits Iceberg tables to the catalog. Spark then reads them for batch jobs, joins or checks. On this site, Spark is the cross-check reader on many guides.

The guides below show Spark reading the tables of each run. The interoperability post compares Spark and Databricks on the same tables.

When to use it​

  • Spark jobs should transform or join tables that a warehouse also reads.
  • You need one reader that opens every catalog and delete format.

When not to use it​

  • Spark runs an Iceberg runtime older than 1.8 and the tables use deletion vectors.
  • You only need ad hoc SQL on a laptop. DuckDB attaches the same REST (representational state transfer) catalogs from a single process.

Catalogs Apache Spark reads

CatalogApache Spark readsHow
AWS GlueYestype=glue
Apache PolarisYestype=rest with OAuth2
LakekeeperYestype=rest with OAuth2
Snowflake HorizonYestype=rest with a Snowflake token
Unity CatalogYestype=rest with OAuth2
BigLake (Google Cloud Lakehouse)Yestype=rest with Google auth
Amazon S3 TablesYestype=rest with SigV4
NessieYestype=nessie
Hive MetastoreYestype=hive
JDBCYestype=jdbc
Generic RESTYestype=rest

Delete formats Apache Spark reads

Equality deletes (v2)Positional deletes (v2)Deletion vectors (v3)
YesYesYes

Spark with the Iceberg runtime reads every delete format. Deletion vectors need Iceberg 1.8 or later on the classpath. Set Apache Spark as a target query engine on the OLake Go job and OLake Go picks a format it reads; with Apache Spark as the only reader that is positional deletes (v2).

Sources: Iceberg: Spark configuration (checked 2026-10-12).

Tested guides for Apache Spark

No guide ends in Apache Spark yet.

Guides where Apache Spark reads the same tables

These pages show Apache Spark reading the same tables through the same catalog, in their cross-engine section. No second copy of the data is made.

Frequently asked questions​

Which Iceberg version does Spark need for deletion vectors?

Iceberg 1.8 or later on the classpath.

Is there a tested guide with Spark?

Yes. Spark read the tables in the cross-engine section of the guides listed below.

Which Spark catalog type matches each OLake Go catalog?

type=glue for AWS Glue, type=rest for Polaris, Lakekeeper, Unity Catalog, Horizon, BigLake and S3 Tables, and type=nessie, type=hive or type=jdbc for the rest.