2. REST | OLake

Last updated:8/6/2025|... min read

REST Catalog

OLake UI
OLake CLI
REST Supported catalog types

Parameter	Sample Value	Description
REST Catalog URL	http://localhost:8181/catalog	Specifies the endpoint URL for the REST catalog service that the writer will connect to.
Iceberg S3 Path	warehouse	Determines the S3 path or storage location for Iceberg data. "warehouse" represents the designated storage directory.
Iceberg Database	olake_iceberg	Specifies the name of the Iceberg database that will be used by the destination configuration.
S3 Endpoint	`http://host.docker.internal:9000`	Endpoint for the S3 service (Minio in this case).
AWS Region	us-east-1	Specifies the AWS region associated with the S3 bucket where the data is stored.
AWS Access Key	`admin`	AWS access key (or Minio key for local testing).
AWS Secret Key**	`password`	AWS secret key (or Minio secret for local testing).

You can query the data via:

SELECT * FROM CATALOG_NAME.ICEBERG_DATABASE_NAME.TABLE_NAME;

CATALOG_NAME can be: jdbc_catalog, hive_catalog, rest_catalog, etc.
ICEBERG_DATABASE_NAME is the name of the Iceberg database you created / added as a value in destination.json file.

For S3 related permissions which is needed to write data to S3, refer to the AWS S3 Permissions documentation.

destination.json
{
  "type": "ICEBERG",
  "writer": {
    "catalog_type": "rest",
    "rest_catalog_url": "http://localhost:8181/catalog",
    "iceberg_s3_path": "warehouse",
    "iceberg_db": "ICEBERG_DATABASE_NAME"
  }
}

REST Configuration Parameters

Configuration Fields

Parameter	Sample Value	Description
catalog_type	rest, glue, jdbc	Defines the catalog type used by the writer. "rest" means the writer interacts with a RESTful catalog service.
rest_catalog_url	http://localhost:8181/catalog	Specifies the endpoint URL for the REST catalog service that the writer will connect to.
iceberg_s3_path	warehouse	Determines the S3 path or storage location for Iceberg data. "warehouse" represents the designated storage directory.
iceberg_db	olake_iceberg	Specifies the name of the Iceberg database that will be used by the destination configuration.

Authentication Fields (optional)

Parameter	Sample Value	Description
token	abc...xyz	Specifies the Bearer token sent in the Authorization header for authenticating with the REST catalog service.
oauth2_uri	https://auth.server.com/oauth/token	OAuth2 server URI for OAuth2 authentication.
rest_auth_type	oauth2	Authentication type (e.g., "oauth2").
credential	your_id:your_secret	Specifies the client ID and secret for OAuth2, formatted as client_id:client_secret.
scope	api.read api.write	OAuth2 scopes (space-separated).
rest_signing_name	s3tables	Service name for AWS Signature V4 (e.g., "s3tables").
rest_signing_region	us-east-1	Region for AWS Signature V4 signing.
rest_signing_v_4	`true`	Enable AWS Signature V4 signing (boolean).

You can query the data via:

SELECT * FROM CATALOG_NAME.ICEBERG_DATABASE_NAME.TABLE_NAME;

CATALOG_NAME can be: jdbc_catalog, hive_catalog, rest_catalog, etc.
ICEBERG_DATABASE_NAME is the name of the Iceberg database you created / added as a value in destination.json file.

For S3 related permissions which is needed to write data to S3, refer to the AWS S3 Permissions documentation.

Lakekeeper is a service that provides a unified view of all your data lakes, regardless of their location. It allows you to manage and monitor your data lakes from a single interface, making it easier to keep track of your data and ensure its integrity.

We have provided a sample configuration for Lakekeeper, which can be used to set up the service in your environment. This configuration includes all the necessary parameters and settings to get started with Lakekeeper. Reder here for more

After you have set up Lakekeeper using the docker compose provided, you can access the Lakekeeper UI at http://localhost:8181/ui.

Steps to create a Warehouse in Lakekeeper

Access the Warehouse Section:
- Navigate to the Warehouse section within Lakekeeper.
Initiate Warehouse Creation:
- Click on the Add Warehouse button.
Select the Storage Type:
- Choose S3 as the storage option.
Enter AWS Credentials:
- Provide the AWS Access Key ID as admin.
- Provide the AWS Secret Access Key as password.
Configure S3 Settings:
- Set the S3 Flavor to S3 Compatible Storage.
- Enable the toggle for Enable path style access.
- Enable the toggle for Enable alternative s3 protocols.
Specify the Bucket Details:
- Enter the bucket name as warehouse (or the name of the bucket you created using MinIO).
Configure the Endpoint:
- Provide the endpoint as http:{IP of your machine}:9000.
- To retrieve your machine's IP address, execute the command:
```
ipconfig getifaddr en0
```
Select the Bucket Region:
- Choose the appropriate region for your bucket.

Following these steps will successfully create a warehouse in Lakekeeper.

This guide helps you set up and use Project Nessie as an Iceberg REST catalog with OLake. It uses Docker Compose to spin up Nessie, MinIO (S3-compatible object store), and Postgres (for Nessie metadata storage).

Assumption:

All services involved in the sync process — including the OLake source image, Nessie, MinIO, and any other dependencies — must be part of the same Docker network.

Setup Steps:

Save the following as docker-compose.yml:

services:
  nessie:
    image: ghcr.io/projectnessie/nessie:latest
    container_name: nessie
    ports:
      - "19120:19120"
    environment:
      - nessie.version.store.type=JDBC
      - quarkus.datasource.db-kind=postgresql
      - quarkus.datasource.jdbc.url=jdbc:postgresql://postgres:5432/nessie_db
      - quarkus.datasource.username=nessie
      - quarkus.datasource.password=nessie
      - nessie.catalog.default-warehouse=warehouse
      - nessie.catalog.warehouses.warehouse.location=s3://warehouse/
      - nessie.catalog.service.s3.default-options.region=us-east-1
      - nessie.catalog.service.s3.default-options.path-style-access=true
      - nessie.catalog.service.s3.default-options.endpoint=http://minio:9000/
      - nessie.catalog.service.s3.default-options.external-endpoint=http://host.docker.internal:9000/
      - nessie.catalog.service.s3.default-options.access-key=urn:nessie-secret:quarkus:nessie.catalog.secrets.access-key
      - nessie.catalog.secrets.access-key.name=minio
      - nessie.catalog.secrets.access-key.secret=minio123
    depends_on:
      postgres:
        condition: service_healthy
      mc:
        condition: service_completed_successfully
    networks:
      - nessie-network

  minio:
    image: quay.io/minio/minio:RELEASE.2025-07-18T21-56-31Z
    container_name: minio
    ports:
      - "9000:9000"
      - "9090:9090"
    environment:
      MINIO_ROOT_USER: minio
      MINIO_ROOT_PASSWORD: minio123
      MINIO_ADDRESS: ":9000"
      MINIO_CONSOLE_ADDRESS: ":9090"
    volumes:
      - minio-data:/data
    command: server /data
    networks:
      - nessie-network
    healthcheck:
      test: ["CMD-SHELL", "curl -f http://localhost:9000/minio/health/live || exit 1"]
      interval: 5s
      timeout: 2s
      retries: 15

  mc:
    image: quay.io/minio/minio:RELEASE.2025-07-18T21-56-31Z
    container_name: mc
    depends_on:
      minio:
        condition: service_healthy
    entrypoint: /bin/bash
    restart: "no"
    command: >
      -c "mc alias set myminio http://minio:9000/ minio minio123 && 
           mc mb myminio/warehouse --ignore-existing"
    networks:
      - nessie-network

  postgres:
    image: postgres:15
    container_name: postgres
    environment:
      POSTGRES_USER: nessie
      POSTGRES_PASSWORD: nessie
      POSTGRES_DB: nessie_db
    ports:
      - "5432:5432"
    volumes:
      - postgres-data:/data
    networks:
      - nessie-network
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U nessie"]
      interval: 5s
      timeout: 2s
      retries: 10

volumes:
  minio-data:
  postgres-data:

networks:
  nessie-network:
    driver: bridge

Start the services:

docker-compose up -d

Nessie Catalog Configuration in OLake

To make OLake use Nessie as the catalog, you'll need to point OLake to the REST endpoint provided by Nessie (http://host.docker.internal:19120/iceberg/), and configure it to write data to MinIO under the warehouse bucket.

Example: writer.json Save the following config as writer.json or your equivalent config file:

{
  "type": "ICEBERG",
  "writer": {
    "catalog_type": "rest",
      "rest_catalog_url": "http://host.docker.internal:19120/iceberg/",
      "iceberg_s3_path": "s3://warehouse/",
      "iceberg_db": "iceberg_db",
      "s3_endpoint": "http://host.docker.internal:9000",
      "aws_access_key": "minio",
      "aws_secret_key": "minio123",
      "aws_region": "us-east-1",
      "no_identifier_fields": true
  }
}

info

If you wish to test out the REST Catalog locally, you can use the docker-compose setup. The local test setup uses Minio as an S3-compatible storage and other all supported catalog types.

You can then setup local spark to run queries on the iceberg tables created in the local test setup.

Need Assistance?

If you have any questions or uncertainties about setting up OLake, contributing to the project, or troubleshooting any issues, we’re here to help. You can:

Your success with OLake is our priority. Don’t hesitate to contact us if you need any help or further clarification!

REST Catalog

REST Configuration Parameters

Configuration Fields

Authentication Fields (optional)

Steps to create a Warehouse in Lakekeeper

Assumption:

Setup Steps:

Nessie Catalog Configuration in OLake

Need Assistance?

Join our growing community

GitHub

Slack

Twitter

LinkedIn

YouTube

REST Catalog​

REST Configuration Parameters​

Configuration Fields​

Authentication Fields (optional)​

Steps to create a Warehouse in Lakekeeper​

Assumption:​

Setup Steps:​

Nessie Catalog Configuration in OLake​

Need Assistance?

Join our growing community

GitHub

Slack

Twitter

LinkedIn

YouTube

REST Catalog

REST Configuration Parameters

Configuration Fields

Authentication Fields (optional)

Steps to create a Warehouse in Lakekeeper

Assumption:

Setup Steps:

Nessie Catalog Configuration in OLake