JDBC Catalog

destination.json

{
  "type": "ICEBERG",
  "writer": {
    "catalog_type": "jdbc",
    "jdbc_url": "jdbc:postgresql://host.docker.internal:5432/iceberg",
    "jdbc_username": "iceberg",
    "jdbc_password": "password",
    "normalization": false,
    "iceberg_s3_path": "s3a://warehouse",
    "s3_endpoint": "http://host.docker.internal:9000",
    "s3_use_ssl": false,
    "s3_path_style": true,
    "aws_access_key": "admin",
    "aws_region": "ap-south-1",
    "aws_secret_key": "password",
    "iceberg_db": "ICEBERG_DATABASE_NAME"
  }
}

JDBC Configuration Parameters

Parameter	Sample Value	Description
catalog_type	jdbc	Type of catalog. Use jdbc for a PostgreSQL-based catalog tests.
jdbc_url	jdbc:postgresql://host.docker.internal:5432/iceberg	JDBC URL for connecting to the PostgreSQL catalog.
jdbc_username	iceberg	Username for the PostgreSQL catalog.
jdbc_password	password	Password for the PostgreSQL catalog.
normalization	false	Flag to enable or disable data normalization.
iceberg_s3_path	s3a://warehouse	S3 path where the Iceberg data/metadata files will be stored.
s3_endpoint	http://host.docker.internal:9000	Endpoint for the S3 service (Minio in this case).
s3_use_ssl	false	Boolean flag to determine if SSL should be used when connecting to S3.
s3_path_style	true	Use path-style addressing for the S3 API.
aws_access_key	admin	AWS access key (or Minio key for local testing).
aws_secret_key	password	AWS secret key (or Minio secret for local testing).
iceberg_db	olake_iceberg	Name of the database to be created in the Iceberg catalog.

You can query the data via:

SELECT * FROM CATALOG_NAME.ICEBERG_DATABASE_NAME.TABLE_NAME;

• CATALOG_NAME can be: jdbc_catalog, hive_catalog, rest_catalog, etc.
• ICEBERG_DATABASE_NAME is the name of the Iceberg database you created / added as a value in destination.json file.

If you are build olake locally (using ./build.sh command), change the config to:

1. jdbc_url": "jdbc:postgresql://localhost:5432/iceberg"
2. s3_endpoint": "http://localhost:9000"

For S3 related permissions which is needed to write data to S3, refer to the AWS S3 Permissions documentation.

info

If you wish to test out the REST Catalog locally, you can use the docker-compose setup. The local test setup uses Minio as an S3-compatible storage and other all supported catalog types.

You can then setup local spark to run queries on the iceberg tables created in the local test setup.