REST Catalog

OLake UI

Parameter	Sample Value	Description
REST Catalog URL	http://localhost:8181/catalog	Specifies the endpoint URL for the REST catalog service that the writer will connect to.
Iceberg S3 Path	warehouse	Determines the S3 path or storage location for Iceberg data. "warehouse" represents the designated storage directory.
Iceberg Database	olake_iceberg	Specifies the name of the Iceberg database that will be used by the destination configuration.
S3 Endpoint	http://host.docker.internal:9000	Endpoint for the S3 service (Minio in this case).
AWS Region	us-east-1	Specifies the AWS region associated with the S3 bucket where the data is stored.
AWS Access Key	admin	AWS access key (or Minio key for local testing).
AWS Secret Key**	password	AWS secret key (or Minio secret for local testing).

OLake CLI

destination.json

{
  "type": "ICEBERG",
  "writer": {
    "catalog_type": "rest",
    "rest_catalog_url": "http://localhost:8181/catalog",
    "iceberg_s3_path": "warehouse",
    "iceberg_db": "ICEBERG_DATABASE_NAME"
  }
}

REST Configuration Parameters

Configuration Fields

Parameter	Sample Value	Description
catalog_type	rest, glue, jdbc	Defines the catalog type used by the writer. "rest" means the writer interacts with a RESTful catalog service.
rest_catalog_url	http://localhost:8181/catalog	Specifies the endpoint URL for the REST catalog service that the writer will connect to.
iceberg_s3_path	warehouse	Determines the S3 path or storage location for Iceberg data. "warehouse" represents the designated storage directory.
iceberg_db	olake_iceberg	Specifies the name of the Iceberg database that will be used by the destination configuration.

Authentication Fields (optional)

Parameter	Sample Value	Description
token	abc...xyz	Specifies the Bearer token sent in the Authorization header for authenticating with the REST catalog service.
oauth2_uri	https://auth.server.com/oauth/token	OAuth2 server URI for OAuth2 authentication.
rest_auth_type	oauth2	Authentication type (e.g., "oauth2").
credential	your_id:your_secret	Specifies the client ID and secret for OAuth2, formatted as client_id:client_secret.
scope	api.read api.write	OAuth2 scopes (space-separated).
rest_signing_name	s3tables	Service name for AWS Signature V4 (e.g., "s3tables").
rest_signing_region	us-east-1	Region for AWS Signature V4 signing.
rest_signing_v_4	true	Enable AWS Signature V4 signing (boolean).

You can query the data via:

SELECT * FROM CATALOG_NAME.ICEBERG_DATABASE_NAME.TABLE_NAME;

• CATALOG_NAME can be: jdbc_catalog, hive_catalog, rest_catalog, etc.
• ICEBERG_DATABASE_NAME is the name of the Iceberg database you created / added as a value in destination.json file.

For S3 related permissions which is needed to write data to S3, refer to the AWS S3 Permissions documentation.

REST Supported catalog types

Using Lakekeeper

Lakekeeper is a service that provides a unified view of all your data lakes, regardless of their location. It allows you to manage and monitor your data lakes from a single interface, making it easier to keep track of your data and ensure its integrity.

We have provided a sample configuration for Lakekeeper, which can be used to set up the service in your environment. This configuration includes all the necessary parameters and settings to get started with Lakekeeper. Reder here for more

After you have set up Lakekeeper using the docker compose provided, you can access the Lakekeeper UI at http://localhost:8181/ui.

Steps to create a Warehouse in Lakekeeper

1. Access the Warehouse Section:

   • Navigate to the Warehouse section within Lakekeeper.

2. Initiate Warehouse Creation:

   • Click on the Add Warehouse button.

3. Select the Storage Type:

   • Choose S3 as the storage option.

4. Enter AWS Credentials:

   • Provide the AWS Access Key ID as admin.
   • Provide the AWS Secret Access Key as password.

5. Configure S3 Settings:

   • Set the S3 Flavor to S3 Compatible Storage.
   • Enable the toggle for Enable path style access.
   • Enable the toggle for Enable alternative s3 protocols.

6. Specify the Bucket Details:

   • Enter the bucket name as warehouse (or the name of the bucket you created using MinIO).

7. Configure the Endpoint:

   • Provide the endpoint as http:{IP of your machine}:9000.
   • To retrieve your machine's IP address, execute the command:

     ipconfig getifaddr en0

8. Select the Bucket Region:

   • Choose the appropriate region for your bucket.

Following these steps will successfully create a warehouse in Lakekeeper.

Using S3 Tables

Click here

Using Gravitino

Coming soon!

Using Nessie

Coming soon!

Using Polaris

Coming soon!

Using Unity

Click here

info

If you wish to test out the REST Catalog locally, you can use the docker-compose setup. The local test setup uses Minio as an S3-compatible storage and other all supported catalog types.

You can then setup local spark to run queries on the iceberg tables created in the local test setup.