Configs

This page details the configuration options for the Iceberg writer via the writer.json file. Two sample configurations are provided below along with a table that explains each key-value pair.

Using Glue Catalog

AWS S3 + Glue Configuration

writer.json

{
    "type": "ICEBERG",
    "writer": {
      "normalization": false,
      "s3_path": "s3://bucket_name/olake_iceberg/test_olake",
      "aws_region": "ap-south-1",
      "aws_access_key": "XXX",
      "aws_secret_key": "XXX",
      "database": "olake_iceberg",
      "grpc_port": 50051,
      "server_host": "localhost"
    }
}

AWS S3 + Glue Configuration Parameters

Parameter	Sample Value	Description
normalization	false	Flag to enable or disable data normalization.
s3_path	s3://bucket_name/olake_iceberg/test_olake	S3 path where the Iceberg data is stored in AWS.
aws_region	ap-south-1	AWS region where the S3 bucket and Glue catalog are located.
aws_access_key	XXX	AWS access key with sufficient permissions for S3 and Glue.
aws_secret_key	XXX	AWS secret key corresponding to the access key.
database	olake_iceberg	Name of the database to be created in AWS Glue.
grpc_port	50051	Port on which the gRPC server listens.
server_host	localhost	Host address of the gRPC server.

Using JDBC catalog with MinIO

Local Test Configuration (JDBC + Minio)

writer.json

{
  "type": "ICEBERG",
  "writer": {
    "catalog_type": "jdbc",
    "jdbc_url": "jdbc:postgresql://localhost:5432/iceberg",
    "jdbc_username": "iceberg",
    "jdbc_password": "password",
    "normalization": false,
    "iceberg_s3_path": "s3a://warehouse",
    "s3_endpoint": "http://localhost:9000",
    "s3_use_ssl": false,
    "s3_path_style": true,
    "aws_access_key": "admin",
    "aws_secret_key": "password",
    "iceberg_db": "olake_iceberg"
  }
}

Local Test Configuration Parameters

Parameter	Sample Value	Description
catalog_type	jdbc	Type of catalog. Use jdbc for a PostgreSQL-based catalog in local tests.
jdbc_url	jdbc:postgresql://localhost:5432/iceberg	JDBC URL for connecting to the PostgreSQL catalog.
jdbc_username	iceberg	Username for the PostgreSQL catalog.
jdbc_password	password	Password for the PostgreSQL catalog.
normalization	false	Flag to enable or disable data normalization.
iceberg_s3_path	s3a://warehouse	S3 path where the Iceberg data/metadata files will be stored.
s3_endpoint	http://localhost:9000	Endpoint for the S3 service (Minio in this case).
s3_use_ssl	false	Boolean flag to determine if SSL should be used when connecting to S3.
s3_path_style	true	Use path-style addressing for the S3 API.
aws_access_key	admin	AWS access key (or Minio key for local testing).
aws_secret_key	password	AWS secret key (or Minio secret for local testing).
iceberg_db	olake_iceberg	Name of the database to be created in the Iceberg catalog.