OLake UI Kubernetes Installation with Helm

This guide details the process for deploying OLake UI on Kubernetes using the official Helm chart.

Components

• OLake UI: Main web interface for job management and configuration
• OLake Worker: Background worker for processing data replication jobs
• PostgreSQL: Primary database for storing job data, configurations, and sync state
• Temporal: Workflow orchestration engine for managing job execution
• Elasticsearch: Search and indexing backend for Temporal workflow data
• Signup Init: One-time initialization service that creates the default admin user

Prerequisites

Ensure the following requirements are met before proceeding:

• Kubernetes 1.19+: Administrative access to a Kubernetes cluster
• Helm 3.2.0+: Helm client installed and configured. Installation Guide
• kubectl: Configured kubectl command-line tool. Installation Guide
• StorageClass: A StorageClass is required by the chart to provision persistent volumes for PostgreSQL, Elasticsearch and the shared storage volume

  # List available StorageClass
  kubectl get storageclass

• System Requirements: Minimum and recommended node specification for running Sync jobs
  • Minimum: 8 vCPU, 16 GB RAM
  • Recommended: 16 vCPU, 32 GB RAM
• Node class recommendations: The following components should be scheduled on on-demand nodes to minimize disruption:
  • nfs-server (if enabled)
  • PostgreSQL (if enabled)
  • Temporal
  • OLake workers
  • OLake UI

Quick Start

1. Add OLake Helm Repository

helm repo add olake https://datazip-inc.github.io/olake-helm
helm repo update

2. Install the Chart

AWS EKS

Basic Installation:

helm install olake olake/olake --set global.storageClass="gp2"

Others

Basic Installation:

helm install olake olake/olake

3. Access OLake UI

Forward the UI service port to local machine:

kubectl port-forward svc/olake-ui 8000:8000

Open browser and navigate to: http://localhost:8000

Default Credentials:

• Username: admin
• Password: password

tip

If OLake is installed with Ingress enabled, port-forwarding is not necessary. Access the application using the configured Ingress hostname.

Configuration Options

Updating OLake UI Version

Pull the latest images and restart the deployments without downtime:

# Restart OLake components
kubectl rollout restart deployment/olake-ui
kubectl rollout restart deployment/olake-workers

Initial User Setup

Create a Kubernetes secret to replace default credentials:

kubectl create secret generic olake-admin-credentials \
  --from-literal=username='superadmin' \
  --from-literal=password='a-very-secure-password' \
  --from-literal=email='admin@mycompany.com'

Then configure in values.yaml:

olakeUI:
  initUser:
    existingSecret: "olake-admin-credentials"
    secretKeys:
      username: "username"
      password: "password"
      email: "email"

Apply the configuration:

helm upgrade olake olake/olake -f values.yaml

Ingress Configuration

To expose OLake through an ingress controller, create a custom values file:

# values.yaml
olakeUI:
  ingress:
    enabled: true
    className: "nginx"
    hosts:
      - host: olake.example.com
        paths:
          - path: /
            pathType: Prefix
    tls:
      - secretName: olake-tls
        hosts:
          - olake.example.com

JobID-Based Node Mapping

With this powerful feature, specific data jobs can be routed to specific Kubernetes nodes, by which performance and reliability can be optimized.

Where is the JobID found? The JobID is an integer that is automatically assigned to each job created in OLake UI. The JobID can be found in the corresponding row for each job on the Jobs page.

global:
  jobMapping:
    123:
      olake.io/workload-type: "heavy"
    456:
      node-type: "high-cpu"
    789:
      olake.io/workload-type: "small"
    999: {} # Empty mapping uses default scheduling

note

• For any JobID that is not specified in the jobMapping configuration, the corresponding job's pod will be scheduled by the standard Kubernetes scheduler, which places it on any available node in the cluster.

Cloud IAM Integration

OLake's activity pods (the pods by which the actual data sync is performed) can be allowed to securely access cloud resources(AWS Glue or S3) using IAM roles.

global:
  jobServiceAccount:
    create: true
    name: "olake-job-sa"
    
    # Cloud provider IAM role associations
    annotations:
      # AWS IRSA
      eks.amazonaws.com/role-arn: "arn:aws:iam::123456789012:role/olake-job-role"
      
      # GCP Workload Identity
      iam.gke.io/gcp-service-account: "olake-job@project.iam.gserviceaccount.com"
      
      # Azure Workload Identity
      azure.workload.identity/client-id: "12345678-1234-1234-1234-123456789012"

note

Note: For detailed instructions on the creation of IAM roles and service accounts, the official documentation for AWS IRSA, GCP Workload Identity, or Azure Workload Identity should be referred to. For a minimal Glue and S3 IAM access policy please refer here.

Encryption Configuration

Configure encryption of job metadata stored in Postgres database by setting the OLAKE_SECRET_KEY environment variable in the OLake Worker:

global:
  env:
    # 1. For AWS KMS (starts with 'arn:aws:kms:'):
    OLAKE_SECRET_KEY: "arn:aws:kms:us-west-2:123456789012:key/12345678-1234-1234-1234-123456789012"
    
    # 2. For local AES-256 (any other non-empty string):
    OLAKE_SECRET_KEY: "your-secret-encryption-key"  # Auto-hashed to 256-bit key
    
    # 3. For no encryption (default, not recommended for production):
    OLAKE_SECRET_KEY: ""  # Empty = no encryption

Persistent Storage Configuration

The OLake application components (UI, Worker, and Activity Pods) require a shared ReadWriteMany (RWX) volume for coordinating pipeline state and metadata.

For production, a robust, highly-available RWX-capable storage solution such as AWS EFS, GKE Filestore, or Azure Files must be used. This is achieved by disabling the built-in NFS server and providing an existing Kubernetes StorageClass that is backed by a managed storage service. An example for using StorageClass is given below:

nfsServer:
  # 1. The development NFS server is disabled
  enabled: false
  
  # 2. An existing ReadWriteMany PersistentVolumeClaim is specified
  external:
    storageClass: "efs-csi"

note

For development and quick starts, a simple NFS server is included and enabled by default. This provides an out-of-the-box shared storage solution without any external dependencies. However, because this server runs as a single pod, it represents a single point of failure and is not recommended for production use.

External PostgreSQL Configuration

External PostgreSQL databases can be used instead of the built-in postgresql deployment. It is the primary database for storing job data, configurations, and sync state.

Requirements:

• PostgreSQL 12+ with btree_gin extension enabled
• This can be enabled with CREATE EXTENSION IF NOT EXISTS btree_gin;, then run \dx to verify if its enabled.
• Both OLake and Temporal databases created on the PostgreSQL instance
• Network connectivity from Kubernetes cluster to PostgreSQL instance

1. Create database secret:

kubectl create secret generic external-postgres-secret \
  --from-literal=host="postgres-host" \
  --from-literal=port="5432" \
  --from-literal=olake_database="olakeDB" \
  --from-literal=temporal_database="temporalDB" \
  --from-literal=username="username" \
  --from-literal=password="password" \
  --from-literal=ssl_mode="require"

2. Configure values.yaml:

postgresql:
  enabled: false
  external:
    existingSecret: "external-postgres-secret"
    secretKeys:
      host:              "host"
      port:              "port"
      olake_database:    "olake_database"
      temporal_database: "temporal_database"
      username:          "username"
      password:          "password"
      ssl_mode:          "ssl_mode"

Upgrading Chart Version

Upgrade to Latest Version

helm repo update
helm upgrade olake olake/olake

Upgrade with New Configuration

helm upgrade olake olake/olake -f new-values.yaml

Troubleshooting

Check Pod Logs

# OLake UI logs  
kubectl logs -l app.kubernetes.io/name=olake-ui -f

# Temporal server logs
kubectl logs -l app.kubernetes.io/name=temporal-server -f

Common Issues

Pods Stuck in Pending State:

• Check if your cluster has sufficient resources
• Check node selectors or affinity rules
• Verify StorageClass is available and configured correctly

Pods Stuck in CrashLoopBackOff State:

• If pod logs show error like failed to ping database, check the Database Connection Issues section below
• If pod restarts and goes into CrashLoopBackOff state, check for the resource requests and limits defined in Helm Values file
• If pod events show error like failed to mount volume, check if the nfs-server pod is up and running

Database Connection Issues:

• Verify PostgreSQL pod is running: kubectl get pods -l app.kubernetes.io/name=postgresql
• In case, setup is done with External PostgreSQL Configuration, check the following:
  • Check if the host is pointing to Writer instance and not a Reader of the database
  • Check if password contains special characters like @ or # etc. If yes, use a different password
  • Check if ssl_mode is correctly set in the kubernetes secret for external database
• Check database connectivity from other pods
• Review database credentials in secrets

Uninstallation

Remove OLake Installation

helm uninstall olake

warning

Some resources are intentionally preserved after helm uninstall to prevent accidental data loss:

• PersistentVolumeClaims (PVCs): olake-shared-storage and database PVCs are retained to preserve job data, configurations, and historical information
• NFS Server Resources: If installed using the built-in NFS server, the following resources persist:
  • Service/olake-nfs-server
  • StatefulSet/olake-nfs-server
  • ClusterRole/olake-nfs-server
  • ClusterRoleBinding/olake-nfs-server
  • StorageClass/nfs-server
  • ServiceAccount/olake-nfs-server