S3 Parquet

OLake Go writes Parquet through the Amazon S3 connector. Storage Type selects S3-compatible storage (AWS S3, GCS, or MinIO) or Azure. Before proceeding, review the Getting Started and Installation sections.

S3

OLake Go supports writing in parquet format directly to S3. Before proceeding with the S3 Parquet destination, we recommend reviewing the Getting Started and Installation sections.

---

Prerequisites

• Before setting up the destination, make sure you have successfully set up the source.
• AWS S3: Review the minimum required IAM Permissions for S3 Parquet Writer.
• GCS: Review GCP IAM Permission for role-based access, and create HMAC keys for your service account to use as S3 credentials. See Creation of HMAC Keys below.

---

Configuration

OLake UI

• Before setting up the destination, make sure you have successfully set up the source.

Key	Description	Data Type	Probable Values
S3 Bucket	The name of the Amazon/Google S3 bucket (without s3:// or gs://) where your output files will be stored. Ensure that the bucket exists and that you have proper access.	string	A valid S3 bucket name (e.g. "olake-s3-test")
S3 Region	The AWS/MinIO/GCS region where the specified S3 bucket is hosted.	string	AWS/GCS region codes such as "ap-south-1", "us-west-2", etc.
S3 Access Key	The AWS/MinIO/GCS HMAC access key used for authenticating S3 requests.	string	A valid AWS/GCS HMAC access key
S3 Secret Key	The AWS/MinIO/GCS HMAC secret key used for S3 authentication. This key should be kept secure.	string	A valid AWS/GCS HMAC secret key
S3 Path	The specific path (or prefix) within the S3 bucket where data files will be written. This is typically a folder path that starts with a / (e.g. "/data").	string	A valid path string
S3 Endpoint	(Optional) Custom S3-compatible endpoint. Required when using GCS HMAC keys or MinIO S3.	string	"https://storage.googleapis.com", "https://<MinIO-Storage-Endpoint>:9000"
Max File Size (MB)	(Optional) Rolls a partition into a new parquet file once the current file reaches this size in MB. Fractional values are allowed. Defaults to 512 when unset.	number	A positive number (integers e.g. 512, 128 or fractional e.g. 0.1 for 100KB)

OLake CLI

destination.json

{
    "type": "PARQUET",
    "writer": {
      "s3_bucket": "<BUCKET_NAME>",
      "s3_region": "ap-south-1",
      "s3_access_key": "XXX",
      "s3_secret_key": "XXX",
      "s3_path": "/data",
      "s3_endpoint": "http://<S3_ENDPOINT>:9000"
    }
}

Key	Description	Data Type	Probable Values
type	Specifies the output file format for writing data. Currently, only the "PARQUET" format is supported.	string	"PARQUET"
s3_bucket	The name of the Amazon S3/GCS bucket (without s3:// or gs://) where your output files will be stored. Ensure that the bucket exists and that you have proper access.	string	A valid S3 bucket name (e.g. "olake-s3-test")
s3_region	The AWS/GCS region where the specified S3 bucket is hosted.	string	AWS/GCS region codes such as "us-west-2", "ap-south-1", etc.
s3_access_key	The AWS/GCS HMAC access key used for authenticating S3 requests.	string	A valid AWS/GCS HMAC access key
s3_secret_key	The AWS/GCS HMAC secret key used for S3 authentication. This key should be kept secure.	string	A valid AWS/GCS HMAC secret key
s3_endpoint	(Optional) The custom endpoint for S3-compatible services. Required for GCS using HMAC keys.	string	"https://storage.googleapis.com"
s3_path	(Optional) The specific path (or prefix) within the S3 bucket where data files will be written. This is typically a folder path that starts with a / (e.g. "/data").	string	A valid path string
max_file_size_mb	(Optional) Rolls a partition into a new parquet file once the current file reaches this size in MB. Fractional values are allowed. Defaults to 512 when unset.	number	A positive number (integers e.g. 512, 128 or fractional e.g. 0.1 for 100KB)

---

info

1. The generated .parquet files use SNAPPY compression (Read more). Note that SNAPPY is no longer supported by S3 Select when performing queries.
2. OLake Go creates a test folder named olake_writer_test containing a single text file (.txt) with the content:
   S3 write test

   This is used to verify that you have the necessary permissions to write to S3.

Azure

OLake Go writes Parquet to Azure when Storage Type is Azure on the Amazon S3 connector. Azure supports Connection String, Shared Key, SAS Token, and Managed Identity authentication.

---

Prerequisites

• The identity must be able to list, read, write, and delete blobs in the container.

---

Configuration

Configure the following fields to write Parquet files to Azure.

Authentication

It supports multiple authentication types. Select the authentication mechanism your storage account uses.

Connection String

To use Connection String authentication, pass one connection string to OLake Go. The key can be an account key or a SAS key.

• The string must contain AccountName or BlobEndpoint, and AccountKey or SharedAccessSignature.
• When the string contains an AccountKey, OLake Go pulls that key out and signs each Blob request with it.
• A string with SharedAccessSignature is not signed with an AccountKey.
• UseDevelopmentStorage=true is rejected unless AccountName, AccountKey, and BlobEndpoint are all set.

Follow the steps in the Manage storage account access keys documentation to access the key.

OLake UI

Connection String Configuration Parameters:

Parameter	Sample Value	Description
Storage Type	Azure	Storage backend OLake Go uses to write Parquet files. Set to Azure to write to Azure Blob Storage.
Azure Auth Type required	Connection String	Authentication method OLake Go uses for Azure Blob requests. Set to match your storage account's authentication requirement.
Azure Storage Container Name required	olake-data	Name of an existing Azure storage container with permission to store output files.
Azure Path	/data	Path inside the container where data files are written.
Azure Connection String required	DefaultEndpointsProtocol=https;AccountName=<ACCOUNT_NAME>;AccountKey=<ACCOUNT_KEY>;EndpointSuffix=core.windows.net	Connection string OLake Go uses to authenticate Blob requests. The key inside it can be an Account key or a SAS key.
Max File Size (MB)	A positive number (integers e.g. 512, 128 or fractional e.g. 0.1 for 100KB)	Rolls a partition into a new parquet file once the current file reaches this size in MB. Fractional values are allowed. Defaults to 512 when unset.

OLake CLI

destination.json

{
  "type": "PARQUET",
  "writer": {
    "storage_type": "Azure",
    "azure_auth_type": "connection-string",
    "azure_container_name": "olakeauthtest",
    "azure_path": "/data",
    "azure_connection_string": "",
    "max_file_size_mb": ""
  }
}

Connection String Configuration Parameters:

Parameter	Sample Value	Description
storage_type	Azure	Storage backend OLake Go uses to write Parquet files. Set to Azure to write to Azure Blob Storage.
azure_auth_type required	connection-string	Authentication method OLake Go uses for Azure Blob requests. Set to match your storage account's authentication requirement.
azure_container_name required	olakeauthtest	Name of an existing Azure storage container with permission to store output files.
azure_path	/data	Path inside the container where data files are written.
azure_connection_string required	DefaultEndpointsProtocol=https;AccountName=<ACCOUNT_NAME>;AccountKey=<ACCOUNT_KEY>;EndpointSuffix=core.windows.net	Connection string OLake Go uses to authenticate Blob requests. The key inside it can be an Account key or a SAS key.
max_file_size_mb	A positive number (integers e.g. 512, 128 or fractional e.g. 0.1 for 100KB)	Rolls a partition into a new parquet file once the current file reaches this size in MB. Fractional values are allowed. Defaults to 512 when unset.

Shared Key

To use Shared Key authentication, copy a storage account access key in Azure and give OLake Go the raw key and the account name as separate fields. OLake Go uses the key to sign each Blob request, and Azure checks the signature. Follow the steps in the Manage storage account access keys documentation to access the key.

OLake UI

Shared Key Configuration Parameters:

Parameter	Sample Value	Description
Storage Type	Azure	Storage backend OLake Go uses to write Parquet files. Set to Azure to write to Azure Blob Storage.
Azure Auth Type required	Shared Key	Authentication method OLake Go uses for Azure Blob requests. Set to match your storage account's authentication requirement.
Azure Storage Account Name required	olakedata	Name of the Azure storage account that contains the container.
Azure Storage Container Name required	olake-data	Name of an existing Azure storage container with permission to store output files.
Azure Path	/data	Path inside the container where data files are written.
Azure Endpoint	https://<ACCOUNT_NAME>.blob.core.windows.net/	Blob service URL. Leave empty to use https://{account}.blob.core.windows.net/. Set it only when using a custom blob URL.
Azure Storage Account Key required	<ACCOUNT_KEY>	Raw storage account access key, passed separately from the account name. OLake Go uses the key to sign each Blob request.
Max File Size (MB)	A positive number (integers e.g. 512, 128 or fractional e.g. 0.1 for 100KB)	Rolls a partition into a new parquet file once the current file reaches this size in MB. Fractional values are allowed. Defaults to 512 when unset.

OLake CLI

destination.json

{
  "type": "PARQUET",
  "writer": {
    "storage_type": "Azure",
    "azure_auth_type": "shared-key",
    "azure_storage_account_name": "olakeadlsauthtest",
    "azure_container_name": "warehouse",
    "azure_path": "",
    "azure_endpoint": "",
    "azure_storage_account_key": "",
    "max_file_size_mb": ""
  }
}

Shared Key Configuration Parameters:

Parameter	Sample Value	Description
storage_type	Azure	Storage backend OLake Go uses to write Parquet files. Set to Azure to write to Azure Blob Storage.
azure_auth_type required	shared-key	Authentication method OLake Go uses for Azure Blob requests. Set to match your storage account's authentication requirement.
azure_storage_account_name required	olakeadlsauthtest	Name of the Azure storage account that contains the container.
azure_container_name required	warehouse	Name of an existing Azure storage container with permission to store output files.
azure_path	/data	Path inside the container where data files are written.
azure_endpoint	https://<ACCOUNT_NAME>.blob.core.windows.net/	Blob service URL. Leave empty to use https://{account}.blob.core.windows.net/. Set it only when using a custom blob URL.
azure_storage_account_key required	<ACCOUNT_KEY>	Raw storage account access key, passed separately from the account name. OLake Go uses the key to sign each Blob request.
max_file_size_mb	A positive number (integers e.g. 512, 128 or fractional e.g. 0.1 for 100KB)	Rolls a partition into a new parquet file once the current file reaches this size in MB. Fractional values are allowed. Defaults to 512 when unset.

SAS Token

To use SAS Token authentication, generate a shared access signature for your container in Azure and give OLake Go the token and the account name as separate fields. The token limits what OLake Go can do: it is scoped to a container, and its permissions can be set directly on the token or through a stored access policy. OLake Go appends the token to each Blob request, and Azure checks the signature and the permissions before allowing the request.

• The token must be the query string (sv=...&sig=...), with or without a leading ?. It must not be the full blob URL.
• The expiry is whatever was set when the SAS was created.

Follow the steps in the Grant limited access to Azure Storage resources using shared access signatures (SAS) documentation to create the token.

OLake UI

SAS Token Configuration Parameters:

Parameter	Sample Value	Description
Storage Type	Azure	Storage backend OLake Go uses to write Parquet files. Set to Azure to write to Azure Blob Storage.
Azure Auth Type required	SAS Token	Authentication method OLake Go uses for Azure Blob requests. Set to match your storage account's authentication requirement.
Azure Storage Account Name required	olakedata	Name of the Azure storage account that contains the container.
Azure Storage Container Name required	olake-data	Name of an existing Azure storage container the SAS token is scoped to. The container must already exist and the token must allow writes.
Azure Path	/data	Path inside the container where data files are written.
Azure Endpoint	https://<ACCOUNT_NAME>.blob.core.windows.net/	Blob service URL. Leave empty to use https://{account}.blob.core.windows.net/. Set it only when using a custom blob URL.
Azure SAS Token required	sv=2024-11-04&sr=c&sig=<SIGNATURE>	SAS query string OLake Go appends to each Blob request. Pass sv=...&sig=..., not the full blob URL.
Max File Size (MB)	A positive number (integers e.g. 512, 128 or fractional e.g. 0.1 for 100KB)	Rolls a partition into a new parquet file once the current file reaches this size in MB. Fractional values are allowed. Defaults to 512 when unset.

OLake CLI

destination.json

{
  "type": "PARQUET",
  "writer": {
    "storage_type": "Azure",
    "azure_auth_type": "sas-token",
    "azure_storage_account_name": "olakedata",
    "azure_endpoint": "https://olakedata.blob.core.windows.net/",
    "azure_path": "/data",
    "azure_sas_token": "sv=2024-11-04&sr=c&sig=<SIGNATURE>",
    "azure_container_name": "warehouse",
    "max_file_size_mb": 512
  }
}

SAS Token Configuration Parameters:

Parameter	Sample Value	Description
storage_type	Azure	Storage backend OLake Go uses to write Parquet files. Set to Azure to write to Azure Blob Storage.
azure_auth_type required	sas-token	Authentication method OLake Go uses for Azure Blob requests. Set to match your storage account's authentication requirement.
azure_storage_account_name required	olakedata	Name of the Azure storage account that contains the container.
azure_container_name required	warehouse	Name of an existing Azure storage container the SAS token is scoped to. The container must already exist and the token must allow writes.
azure_path	/data	Path inside the container where data files are written.
azure_endpoint	https://olakedata.blob.core.windows.net/	Blob service URL. Leave empty to use https://{account}.blob.core.windows.net/. Set it only when using a custom blob URL.
azure_sas_token required	sv=2024-11-04&sr=c&sig=<SIGNATURE>	SAS query string OLake Go appends to each Blob request. Pass sv=...&sig=..., not the full blob URL.
max_file_size_mb	A positive number (integers e.g. 512, 128 or fractional e.g. 0.1 for 100KB)	Rolls a partition into a new parquet file once the current file reaches this size in MB. Fractional values are allowed. Defaults to 512 when unset.

Managed Identity/ Workload Identity

OLake Go supports two Managed Identity/ Workload Identity options: Azure Virtual Machine and AKS Workload Identity.

How the Client ID field works

The Client ID entered in the OLake UI or CLI always has the highest priority. The Client ID field tells OLake Go to use the user-assigned managed identity. Leave it empty to use the VM's system-assigned identity, or, on Kubernetes, the identity of the job ServiceAccount.

OLake Go always prioritizes the Client ID you enter in OLake Go. If it is empty, it falls back to another identity only when this field is left empty. This depends on where Olake Go runs, as described below.

Virtual machine

Uses the managed identity assigned to an Azure VM. To set one up, follow Configure managed identities on Azure virtual machines.

• Client ID provided (through UI or CLI): OLake Go uses that user-assigned identity. It must already be attached to the VM.
• Client ID empty: OLake Go uses the VM's system-assigned identity.

AKS Workload Identity

Uses a workload identity on Azure Kubernetes Service. To set one up, follow Use a Microsoft Entra Workload ID with Azure Kubernetes Service.

• Client ID provided (through UI or CLI): OLake Go uses that identity (it should be federated).
• Client ID empty: OLake Go uses the client ID from the azure.workload.identity/client-id annotation on the job ServiceAccount. See Cloud Integration for Azure. If the annotation is not set (annotations: {}), the sync fails.

OLake UI

Managed Identity/ Workload Identity Configuration Parameters:

Parameter	Sample Value	Description
Storage Type	Azure	Storage backend OLake Go uses to write Parquet files. Set to Azure to write to Azure Blob Storage.
Azure Auth Type required	Managed Identity/ Workload Identity	Authentication method OLake Go uses for Azure Blob requests. Uses the identity of the VM or pod OLake Go runs on.
Azure Storage Account Name required	olakedata	Name of the Azure storage account that contains the container.
Azure Storage Container Name required	olake-data	Name of an existing Azure storage container with permission to store output files.
Azure Path	/data	Path inside the container where data files are written.
Azure Endpoint	https://<ACCOUNT_NAME>.blob.core.windows.net/	Blob service URL. Leave empty to use https://{account}.blob.core.windows.net/. Set it only when using a custom blob URL.
Azure Client ID	a1b2c3d4-e5f6-7890-abcd-ef1234567890	Client ID of a user-assigned managed identity. Leave empty for a system-assigned managed identity or AKS workload identity (AZURE_CLIENT_ID on the process).
Max File Size (MB)	A positive number (integers e.g. 512, 128 or fractional e.g. 0.1 for 100KB)	Rolls a partition into a new parquet file once the current file reaches this size in MB. Fractional values are allowed. Defaults to 512 when unset.

OLake CLI

destination.json

{
  "type": "PARQUET",
  "writer": {
    "storage_type": "Azure",
    "azure_auth_type": "managed-or-workload-identity",
    "azure_storage_account_name": "olakedata",
    "azure_container_name": "warehouse",
    "azure_path": "/data",
    "azure_endpoint": "https://olakedata.blob.core.windows.net/",
    "azure_client_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
    "max_file_size_mb": 512
  }
}

Managed Identity/ Workload Identity Configuration Parameters:

Parameter	Sample Value	Description
storage_type	Azure	Storage backend OLake Go uses to write Parquet files. Set to Azure to write to Azure Blob Storage.
azure_auth_type required	managed_or_workload_identity	Authentication method OLake Go uses for Azure Blob requests. Uses the identity of the VM or pod OLake Go runs on.
azure_storage_account_name required	olakedata	Name of the Azure storage account that contains the container.
azure_container_name required	warehouse	Name of an existing Azure storage container with permission to store output files.
azure_path	/data	Path inside the container where data files are written.
azure_endpoint	https://olakedata.blob.core.windows.net/	Blob service URL. Leave empty to use https://{account}.blob.core.windows.net/. Set it only when using a custom blob URL.
azure_client_id	a1b2c3d4-e5f6-7890-abcd-ef1234567890	Client ID of a user-assigned managed identity. Leave empty for a system-assigned managed identity or AKS workload identity (AZURE_CLIENT_ID on the process).
max_file_size_mb	512	Rolls a partition into a new parquet file once the current file reaches this size in MB. Fractional values are allowed. Defaults to 512 when unset.