REST Catalog

writer.json

{
  "type": "ICEBERG",
  "writer": {
    "catalog_type": "rest",
    "normalization": false,
    "rest_catalog_url": "http://localhost:8181/catalog",
    "iceberg_s3_path": "warehouse",
    "iceberg_db": "ICEBERG_DATABASE_NAME"
  }
}

REST Configuration Parameters

Parameter	Sample Value	Description
catalog_type	rest, glue, jdbc	Defines the catalog type used by the writer. "rest" means the writer interacts with a RESTful catalog service.
normalization	false	Indicates whether data normalization is applied. "false" means that normalization is disabled.
rest_catalog_url	http://localhost:8181/catalog	Specifies the endpoint URL for the REST catalog service that the writer will connect to.
iceberg_s3_path	warehouse	Determines the S3 path or storage location for Iceberg data. "warehouse" represents the designated storage directory.
iceberg_db	olake_iceberg	Specifies the name of the Iceberg database that will be used by the writer configuration.

You can query the data via:

SELECT * FROM CATALOG_NAME.ICEBERG_DATABASE_NAME.TABLE_NAME;

• CATALOG_NAME can be: jdbc_catalog, hive_catalog, rest_catalog, etc.
• ICEBERG_DATABASE_NAME is the name of the Iceberg database you created / added as a value in writer.json file.

For S3 related permissions which is needed to write data to S3, refer to the AWS S3 Permissions documentation.

REST Supported catalog types

Using Lakekeeper

Lakekeeper is a service that provides a unified view of all your data lakes, regardless of their location. It allows you to manage and monitor your data lakes from a single interface, making it easier to keep track of your data and ensure its integrity.

We have provided a sample configuration for Lakekeeper, which can be used to set up the service in your environment. This configuration includes all the necessary parameters and settings to get started with Lakekeeper. Refer here

After you have set up Lakekeeper using the docker compose provided, you can access the Lakekeeper UI at http://localhost:8181/ui.

Steps to create a Warehouse in Lakekeeper

1. Access the Warehouse Section:

   • Navigate to the Warehouse section within Lakekeeper.

2. Initiate Warehouse Creation:

   • Click on the Add Warehouse button.

3. Select the Storage Type:

   • Choose S3 as the storage option.

4. Enter AWS Credentials:

   • Provide the AWS Access Key ID as admin.
   • Provide the AWS Secret Access Key as password.

5. Configure S3 Settings:

   • Set the S3 Flavor to S3 Compatible Storage.
   • Enable the toggle for Enable path style access.
   • Enable the toggle for Enable alternative s3 protocols.

6. Specify the Bucket Details:

   • Enter the bucket name as warehouse (or the name of the bucket you created using MinIO).

7. Configure the Endpoint:

   • Provide the endpoint as http:{IP of your machine}:9000.
   • To retrieve your machine's IP address, execute the command:

     ipconfig getifaddr en0

8. Select the Bucket Region:

   • Choose the appropriate region for your bucket.

Following these steps will successfully create a warehouse in Lakekeeper.