Skip to main content
Last updated:|... min read
Bug iconOpen OLake issuesBug iconOpen OLake doc issue

REST Catalog

The REST catalog is a standardized API designed to simplify the management of Apache Iceberg tables across diverse engines and programming languages. By providing a unified client interface, it eliminates the need for separate catalog integrations for engines like Spark, Flink, Trino, or languages like Java.

Built on an OpenAPI specification, the REST catalog offers a modern, flexible alternative to the Hive Metastore's Thrift interface, tailored specifically for Iceberg's architecture.

The Generic REST Catalog is the default implementation of the Apache Iceberg REST Catalog API. It provides a standard, engine-agnostic way to manage Iceberg tables without depending on a specific catalog service like Nessie, Polaris, or Unity.

Prerequisites​

Required services:

  • Object store – e.g., S3, MinIO, or another S3-compatible storage for table data and metadata.
  • Metadata database – typically PostgreSQL, with a dedicated database and user for Iceberg.
  • REST service – the Iceberg REST Catalog service.

Permissions:

  • The REST Catalog service user needs read and write permissions on the object store buckets used by Iceberg tables.
  • The catalog user should have full DDL and DML privileges on the Iceberg metadata database and tables.

Configuration​

Configure the following fields to connect OLake Go to your REST catalog.

Authentication​

The catalog supports multiple authentication types.


REST catalog setup form with required fields for catalog type, URIs, database, S3 path, AWS credentials, and region.

REST Configuration Parameters​

ParameterSample ValueDescription
REST Catalog URL
required		http://<REST_ENDPOINT>:8181Specifies the endpoint URL for the REST catalog service that the writer will connect to.
S3 Path
required		s3://<BUCKET_NAME>S3 bucket path where Iceberg table data and metadata files will be stored.
AWS Region
required		ap-south-1AWS Region where your S3 bucket exists. Required so OLake Go calls the correct regional S3 endpoint for writing Iceberg data files.
Catalog Nameolake_icebergName of the Iceberg catalog OLake Go registers tables under. Defaults to olake_iceberg if left empty.
S3 Endpointhttps://glue.ap-south-1.amazonaws.comS3 API endpoint for writing Iceberg data files. Optional for AWS S3 but required for S3-compatible storage like MinIO.
AWS Access KeyXXXAWS access key ID for authentication. Optional if using IAM roles or instance profiles.

If using IAM:
Docker Compose: add the required IAM environment variables under x-envs in your compose file. See Service Environment Variables for setup details.
Kubernetes: set up pod IAM in Cloud IAM Integration.
AWS Secret KeyXXXAWS secret access key for authentication. Optional if using IAM roles or instance profiles.

If using IAM:
Docker Compose: add the required IAM environment variables under x-envs in your compose file. See Service Environment Variables for setup details.
Kubernetes: set up pod IAM in Cloud IAM Integration.
Enable Arrow Writesfalse/trueWrites data and delete files using Apache Arrow based writer and registers them in Iceberg.
Tokenabc...xyzSpecifies the Bearer token sent in the Authorization header for authenticating with the REST catalog service.
Authentication Typenone
oauth2
sigv4
google		How OLake Go authenticates with the REST catalog. Select the method your catalog supports. This determines which authentication fields are shown and required.
REST Auth URIhttps://auth.server.com/oauth/tokenURL of the REST catalog’s OAuth2 token endpoint. OLake Go uses this with REST Credential to request an access token during OAuth2 client-credentials authentication.
REST Credential<client_id>:<client_secret>Specifies the client ID and secret for OAuth2, formatted as client_id:client_secret. Used with REST Auth URI when OLake Go requests an access token from the catalog’s auth service. Not needed when using REST Token or other auth types.
REST ScopePRINCIPAL_ROLE:ALLRequired for OAuth2 client-credentials authentication along with REST Auth URI and REST Credential. Tells the OAuth2 server what level of access to grant, space-separated if requesting more than one. Values are catalog-specific. Not needed when using REST Token or other auth types.
REST Signing Names3tablesAWS SigV4 service name used to sign REST catalog API requests. Required when REST Enable Signature V4 is turned on.
REST Signing Regionus-east-1AWS region used for SigV4 signing of REST catalog API requests. Required when REST REST Enable Signature V4 is turned on. Must match your catalog region.
REST Enable Signature V4true/falseEnable AWS Signature V4 signing.
Disable Identifier Tablestrue/falseNeeded to set true for Databricks Unity Catalog as it doesn't support identifier fields
Catalog Name Supported for v0.3.5 and above

For the catalog name, OLake Go only supports lowercase letters and underscores. Spaces and special characters are not supported.

Click Create -> to test the connection and verify that OLake Go can validate both the REST catalog endpoint and S3 compatible object storage access.

Setup For Local Testing​

Save the following docker-compose.yml which will start the following services required for an Iceberg REST Catalog.

  1. REST Catalog Service (Tabular image) – Provides the REST API for managing Iceberg tables.
  2. PostgreSQL – Serves as the metadata database to track Iceberg table metadata and schema evolution.
  3. MinIO + MinIO Client – An S3-compatible object store used to store table data and snapshots.
docker-compose.yml
version: "3.9"

services:
  rest:
    image: tabulario/iceberg-rest
    container_name: iceberg-rest
    ports:
      - 8181:8181
    volumes:
      - catalog-data:/catalog
    environment:
      AWS_ACCESS_KEY_ID: admin
      AWS_SECRET_ACCESS_KEY: password
      AWS_REGION: us-east-1
      CATALOG_WAREHOUSE: s3://warehouse/
      CATALOG_IO__IMPL: org.apache.iceberg.aws.s3.S3FileIO
      CATALOG_S3_ENDPOINT: http://minio:9090
      CATALOG_URI: jdbc:postgresql://postgres:5432/iceberg
      CATALOG_JDBC_USER: iceberg
      CATALOG_JDBC_PASSWORD: password
    networks:
      - iceberg_net
    depends_on:
      postgres:
        condition: service_healthy
      mc:
        condition: service_completed_successfully

  postgres:
    image: postgres:15
    container_name: postgres
    networks:
      - iceberg_net
    environment:
      POSTGRES_USER: iceberg
      POSTGRES_PASSWORD: password
      POSTGRES_DB: iceberg
    healthcheck:
      test: [ "CMD", "pg_isready", "-U", "iceberg", "-d", "password" ]
      interval: 2s
      timeout: 10s
      retries: 3
      start_period: 10s
    ports:
      - 5432:5432
    volumes:
      - ./data/postgres-data:/var/lib/postgresql/data

  minio:
    image: minio/minio
    hostname: minio
    container_name: minio
    ports:
      - 9090:9090
      - 9091:9091
    volumes:
      - minio-data:/data
    environment:
      MINIO_ACCESS_KEY: admin
      MINIO_SECRET_KEY: password
      MINIO_DOMAIN: minio
    command: server --address ":9090" --console-address ":9091" /data
    networks:
      iceberg_net:
        aliases:
          - warehouse.minio

  mc:
    image: minio/mc
    container_name: mc
    environment:
      AWS_ACCESS_KEY_ID: admin
      AWS_SECRET_ACCESS_KEY: password
      AWS_REGION: us-east-1
    entrypoint: >
      /bin/sh -c "
      until (/usr/bin/mc alias set minio http://minio:9090 admin password) do echo '...waiting...' && sleep 1; done;
      echo 'Ensuring warehouse bucket exists and is public...';
      if /usr/bin/mc stat minio/warehouse > /dev/null 2>&1; then
        echo 'Warehouse bucket exists, removing for fresh start...';
        /usr/bin/mc rm -r --force minio/warehouse || echo 'Failed to remove warehouse, proceeding...';
      fi;
      /usr/bin/mc mb minio/warehouse;
      /usr/bin/mc anonymous set public minio/warehouse;
      echo 'Minio warehouse bucket setup complete.';
      "
    networks:
      - iceberg_net
    depends_on:
      - minio

volumes:
  catalog-data:
  minio-data:

networks:
  iceberg_net:

Start the services:

docker-compose up -d
note

All services involved in the sync OLake Go, REST Catalog Service, MinIO, and Postgres must run in the same Docker network.

Troubleshooting​

  • Your authentication credentials are invalid ... unauthorized_client
    • Fix:
      • Ensure the correct OAuth/token or DB username/password is provided.
      • Re-issue new tokens or refresh secrets if expired.
    • Check using this command: 
      curl -H "Authorization: Bearer <token>" https://<catalog-endpoint>/v1/config
  • User: <ARN> is not authorized to perform: sts:AssumeRole
    • Fix:
      • Make sure the correct IAM role is assigned.
      • Validate role trust relationships and necessary permissions to S3 and Catalog.
    • Check using this command: 
      aws sts assume-role --role-arn <S3_role_arn> --role-session-name test-session


💡 Join the OLake Community!

Got questions, ideas, or just want to connect with other data engineers?
👉 Join our Slack Community to get real-time support, share feedback, and shape the future of OLake together. 🚀

Your success with OLake is our priority. Don’t hesitate to contact us if you need any help or further clarification!

Join our growing community

GitHub
Slack
Twitter
LinkedIn
YouTube