Skip to main content

Dataproc and Apache Iceberg: OLake Go setup

Spark on Google Cloud Dataproc reads Iceberg tables in the BigLake catalog. BigQuery reads the same catalog.

Short answer. To process replicated tables on Dataproc, run OLake Go with the BigLake Iceberg REST (representational state transfer) catalog as the destination. Point Spark on Dataproc at it with Google authentication and Iceberg 1.10 jars, and Spark reads every delete format. If BigQuery also reads the tables, follow the BigQuery delete-format limits instead.

Key takeaways

  • Spark on Dataproc reads the Iceberg tables OLake Go commits to BigLake.
  • Spark reads every Iceberg delete format, so Dataproc adds no delete-format limit of its own.
  • Spark on Dataproc and BigQuery share one copy of the data.

Last reviewed October 2026.

How OLake Go gets data into Google Cloud Dataproc​

OLake Go uses change data capture (CDC) where the source has it. It commits Iceberg tables to BigLake. Spark on Dataproc and BigQuery then share one copy of the data.

No guide on this site runs Dataproc yet. The nine BigQuery guides write the same BigLake tables, and the BigLake setup post covers the catalog.

When to use it​

  • Spark jobs on Google Cloud should process tables that BigQuery also reads.
  • The data already lives in Google Cloud Storage.

When not to use it​

  • You register tables in Dataproc Metastore and expect BigQuery to see them. BigQuery does not see Dataproc Metastore tables.
  • BigQuery is a reader and you want deletion vectors. Those are a BigQuery preview feature.

Catalogs Google Cloud Dataproc reads

CatalogGoogle Cloud Dataproc readsHow
BigLake (Google Cloud Lakehouse)YesREST catalog with Google auth
Hive MetastoreYesDataproc Metastore (Spark only; BigQuery does not see these tables)

Delete formats Google Cloud Dataproc reads

Equality deletes (v2)Positional deletes (v2)Deletion vectors (v3)
YesYesYes

Spark on Dataproc reads every delete format with Iceberg 1.10 jars. If BigQuery shares the table, follow the BigQuery limits instead. Set Google Cloud Dataproc as a target query engine on the OLake Go job and OLake Go picks a format it reads; with Google Cloud Dataproc as the only reader that is positional deletes (v2).

Sources: Dataproc: Iceberg with the Lakehouse catalog (checked 2026-10-12).

Tested guides for Google Cloud Dataproc

No guide ends in Google Cloud Dataproc yet.

Frequently asked questions​

Does Spark on Dataproc read deletion vectors?

Yes, with Iceberg 1.10 jars on the classpath.

Can BigQuery and Dataproc share the tables?

Yes, through the BigLake catalog. Tables in Dataproc Metastore are visible to Spark only.

How does OLake Go write to BigLake?

With a service account JSON key. See the BigLake catalog page.