Google Cloud Storage (GCS) Writer Overview

OLake supports writing data to Google Cloud Storage (GCS) in parquet format.

Google Cloud Storage provides an S3-compatible interface, allowing you to use S3-compatible tools and libraries to interact with GCS buckets and objects. This interoperability enables seamless integration and ingestion of data into GCS by supporting the Amazon S3 API, which means you can use existing S3 tools and workflows with minimal changes—such as updating the endpoint to https://storage.googleapis.com and authenticating via HMAC keys. This compatibility simplifies migrations, data transfers, and tool usage across platforms.

Add these HMAC keys as Access Key and Secret Key in the destination configuration - Parquet Writer Configuration

info

HMAC (Hash-based Message Authentication Code) keys in Google Cloud Storage are used for authentication when accessing GCS resources, particularly through the S3-compatible API. They consist of an access key and a secret key, which provide a way to sign requests and verify identity without using Google account credentials. \

Refer - https://cloud.google.com/storage/docs/authentication/hmackeys

More details on how to configure and use the GCS writer will be available soon. In the meantime, you can refer to the Iceberg writer documentation for general guidance on Iceberg integration, as the principles are similar across different storage backends.