Skip to main content
ParameterSample ValueDescription
iceberg_s3_paths3://warehouse/ or gs://hive-dataproc-generated-bucket/hive-warehouseDetermines the S3 path or storage location for Iceberg data. The value s3://warehouse/ represents the designated S3 bucket or directory. If using GCP, use dataproc hive metastore bucket. Use s3a:// if using Minio
aws_regionus-east-1Specifies the AWS region associated with the S3 bucket where the data is stored.
aws_access_keyXXXAWS access key with sufficient permissions for S3. Optional if using IAM role attached to running instance/pod.
aws_secret_keyXXXAWS secret key with sufficient permissions for S3. Optional if using IAM role attached to running instance/pod.
s3_endpointhttp://S3_ENDPOINTSpecifies the endpoint URL for the S3 service. This may be used when connecting to an S3-compatible storage service like MinIO running on localhost.
hive_urithrift://<hostname>:9083 or thrift://METASTORE_IP:9083Defines the URI of the Hive Metastore service that the writer will connect to for catalog interactions. METASTORE_IP will be provided by GCP's Hive dataproc metastore or thrift://localhost:9083 or thrift://host.docker.internal:9083 if you are using local setup using docker compose
s3_use_sslfalseIndicates whether SSL is enabled for S3 connections. "false" means that SSL is disabled for these communications.
s3_path_styletrueDetermines if path-style access is used for S3. "true" means that the writer will use path-style addressing instead of the default virtual-hosted style.
hive_clients5Specifies the number of Hive clients allocated for managing interactions with the Hive Metastore.
hive_sasl_enabledfalseIndicates whether SASL authentication is enabled for the Hive connection. "false" means that SASL is disabled.
iceberg_dbiceberg_dbSpecifies the name of the Iceberg database to be used by the destination configuration.


πŸ’‘ Join the OLake Community!

Got questions, ideas, or just want to connect with other data engineers?
πŸ‘‰ Join our Slack Community to get real-time support, share feedback, and shape the future of OLake together. πŸš€

Your success with OLake is our priority. Don’t hesitate to contact us if you need any help or further clarification!