Docker Compose (OLake UI)

OLake UI provides a complete Docker Compose stack for running the replication sync using different sources with orchestration.

Components & Architecture

• OLake UI: Main web interface for job management and configuration
• Temporal Worker: Background worker for processing data replication jobs
• PostgreSQL: Primary database for storing job data, configurations, sync state, and Temporal visibility data
• Temporal Server: Workflow orchestration engine for managing job execution
• Temporal UI: Web interface for monitoring workflows and debugging
• Signup Init: One-time initialization service that creates the default admin user
• Elasticsearch (Legacy only): Search and indexing backend for Temporal workflow data (deprecated in new installations)

Prerequisites

The following requirements must be met before starting:

• Docker installed (Docker Desktop recommended)
• Docker Compose (included with Docker Desktop)
• Port 8000 available on the system
• System Requirements:
  • Minimum: 8 vCPU, 16 GB RAM
  • Recommended: 16 vCPU, 32 GB RAM

Quick Start

One-Command Setup

The fastest way to get OLake UI running is with a single command:

curl -sSL https://raw.githubusercontent.com/datazip-inc/olake-ui/master/docker-compose-v1.yml | docker compose -f - up -d

Want Ingestion + Maintenance in one setup?

OLake bundles both capabilities together. Get started with a single command — see the setup here.

This command will:

• Download the latest docker-compose.yml file
• Pull all required Docker images
• Start all services in the background
• Create a default admin user automatically

Access the Application

• OLake UI: http://localhost:8000

Login

The default credentials are:

• Username: admin
• Password: password

For detailed job creation instructions, see Create Jobs or Creating First Pipeline.

Service Configuration

Changing Admin Credentials

The default admin user can be customized by editing the docker-compose.yml file before starting:

x-signup-defaults:
  username: &defaultUsername "your-username"
  password: &defaultPassword "your-secure-password"
  email: &defaultEmail "your-email@example.com"

Updating OLake UI Version

To update OLake UI to the latest version, use the following commands based on your setup:

Latest (with New Features)

Upgrade Ingestion Stack

curl -sSL https://raw.githubusercontent.com/datazip-inc/olake-ui/master/docker-compose-v1.yml | docker compose -f - down && \
curl -sSL https://raw.githubusercontent.com/datazip-inc/olake-ui/master/docker-compose-v1.yml | docker compose -f - up -d

Legacy (with Elasticsearch)

curl -sSL https://raw.githubusercontent.com/datazip-inc/olake-ui/master/docker-compose.yml | docker compose -f - down && \
curl -sSL https://raw.githubusercontent.com/datazip-inc/olake-ui/master/docker-compose.yml | docker compose -f - up -d

Note: Your data and configurations will be preserved as they are stored in persistent volumes and the olake-data directory.

Upgrading Legacy to Latest

To move from a Legacy setup (with Elasticsearch) to the new Postgres-only visibility setup, follow these steps:

1. Stop and remove the existing legacy stack:

   curl -sSL https://raw.githubusercontent.com/datazip-inc/olake-ui/master/docker-compose.yml | docker compose -f - down

2. Start the new stack:

   curl -sSL https://raw.githubusercontent.com/datazip-inc/olake-ui/master/docker-compose-v1.yml | docker compose -f - up -d

Visibility Data Loss

When upgrading from the Legacy setup to the New setup, existing job logs and workflow history stored in Elasticsearch will not be visible in the new setup. Only new jobs and logs generated after the upgrade will be visible.

If you require access to historical logs and job history, continue using the Legacy Setup.

Encryption Key Configuration

OLake Go supports encryption of source and destination configurations stored in the database. Configure the encryption key in docker-compose.yml:

• Custom String: Provide any string (OLake Go generates SHA-256 hash):

  x-encryption:
    key: &encryptionKey "your-passphrase"

• AWS KMS: Use a AWS KMS key ARN (recommended for production):

  x-encryption:
    key: &encryptionKey "arn:aws:kms:us-east-1:123456789012:key/12345678-1234-1234-1234-123456789012"

• Disable Encryption: Use empty string:

  x-encryption:
    key: &encryptionKey ""  # No Encryption

Customizing Data Directory

The default data directory can be changed by modifying the host persistence path:

x-app-defaults:
  host_persistence_path: &hostPersistencePath /custom/path/to/olake-data
  worker_config_volume_details: &workerConfigVolumeDetails
    type: bind
    source: *hostPersistencePath
    target: /tmp/olake-config

This will create and use /custom/path/to/olake-data instead of the default ./olake-data directory.

External PostgreSQL Configuration

OLake Go supports using an external PostgreSQL instance instead of the built-in Postgres service included in the docker-compose file. This PostgreSQL stores all job data, configurations, sync state, and Temporal workflow data.

Within the compose file, an extension field named x-db-envs defines all database configuration values used by OLake Go and Temporal. These values can be modified to point to the external PostgreSQL instance.

x-db-envs:
  DB_HOST: &DBHost postgresql
  DB_PORT: &DBPort 5432
  DB_USER: &DBUser temporal
  DB_PASSWORD: &DBPassword temporal
  DB_SSLMODE: &DBSSLMode disable
  OLAKE_DB_NAME: &olakeDBName postgres
  TEMPORAL_DB_NAME: &temporalDBName temporal

If connection to external PostgreSQL instance is over TLS, the following variables need to be uncommented under services.temporal.env section in the compose file:

  # for TLS enabled external postgres database
  SQL_TLS: true
  SQL_TLS_DISABLE_HOST_VERIFICATION: true
  SQL_TLS_ENABLED: true
  SQL_HOST_VERIFICATION: false

Optional

A separate PostgreSQL container (postgresql service) is included by default. When an external database is used, this service is not needed and can be stopped after all the services are started and healthy.

Service Environment Variables

Key environment variables that can be customized within x-envs section. These variables are automatically injected into the olake-ui and olake-worker containers:

x-envs:
  shared: &sharedEnvs
    CONTAINER_REGISTRY_BASE: ${CONTAINER_REGISTRY_BASE:-registry-1.docker.io}
    OLAKE_SECRET_KEY: *encryptionKey
    PERSISTENT_DIR: *hostPersistencePath
    FOO: bar
    KEY: value
    NAME: example

External Temporal Configuration

External Temporal server can be used instead of the built-in Temporal deployment. It is the primary tool for workflow orchestration of olake jobs.

x-envs:
  shared: &sharedEnvs
    CONTAINER_REGISTRY_BASE: ${CONTAINER_REGISTRY_BASE:-registry-1.docker.io}
    OLAKE_SECRET_KEY: *encryptionKey
    PERSISTENT_DIR: *hostPersistencePath
    TEMPORAL_EXTERNAL: "true"
    TEMPORAL_ADDRESS: "<region>.<cloud>.api.temporal.io:7233"
    TEMPORAL_API_KEY: "<temporal-api-key>"
    TEMPORAL_NAMESPACE: "<temporal-namespace>"
    TEMPORAL_TASK_QUEUE: "<temporal-task-queue>"
    TEMPORAL_ENABLE_TLS: "true"

Private Registry Configuration

If you are pulling images from a private or self-hosted registry, add the registry settings to the shared environment block used by the compose stack.

First, log in to the registry from a terminal:

docker login registry.your-company.com -u 'robot$username' -p 'ABCD'

Then add the following variables under x-envs.shared in the compose file:

x-envs:
  shared: &sharedEnvs
    CONTAINER_REGISTRY_BASE: "registry.your-company.com"
    CONTAINER_REGISTRY_USERNAME: "<username>"
    CONTAINER_REGISTRY_PASSWORD: "<password>"
    CONTAINER_REGISTRY_INSECURE: "<true/false>"
    CONTAINER_REGISTRY_TLS_SKIP_VERIFY: "<true/false>"
    CONTAINER_REGISTRY_CA_CERT: "<PEM-encoded-CA-certificate>"

These values are used by the compose stack when resolving image pull credentials and registry TLS settings.

Data Persistence

The Docker Compose setup creates the following data storage:

OLake Go Config Directory

• olake-data: Local directory created in the current working directory
  • Contains streams configurations, connection settings, and sync state
  • Persists across container restarts and recreations
  • Used by OLake UI and Temporal Worker services

Docker Volumes

• temporal-postgresql-data: PostgreSQL database storage
  • Contains workflow execution history, job metadata and sync state
  • Used by the PostgreSQL service
• temporal-elasticsearch-data (Legacy only): Elasticsearch search index storage
  • Used by the Elasticsearch service in the legacy configuration

Log Retention

OLake Go includes an automated log retention system that helps manage disk space by automatically cleaning up old log files. This prevents log files from accumulating indefinitely and consuming excessive storage space.

What It Does

• Runs daily at midnight (00:00 server's local timezone) using a cron job
• Deletes entire log directories that are older than the configured retention period (defaults to 30 days)

Configuration

Set the LOG_RETENTION_PERIOD environment variable in the docker-compose of olake-ui to control how long logs are kept:

# In docker-compose.yml
services:
  temporal-worker:
    environment:
      LOG_RETENTION_PERIOD: "30"  # Keep logs for 30 days

Monitoring Log Cleaner

The log cleanup process can be monitored through the logs of the temporal-worker service defined in the docker-compose configuration for olake-ui.

When the log cleaner starts, it emits a log entry indicating the beginning of the process:

Log cleaner started... 

Every time the cron job runs, it emits the following log:

Running log cleaner...

As each log directory is deleted, a log entry is generated showing the full path of the directory being removed:

Deleting folder /path/to/olake/logs

Temporal Retention Period

OLake Go includes a configurable retention period for Temporal visibility data that controls how long job and sync history are stored and visible in the system. This helps manage database storage by automatically cleaning up old workflow execution history.

Configuration

Set the TEMPORAL_RETENTION_PERIOD environment variable in the temporal-worker service within the docker-compose configuration for olake-ui to control how long visibility data is retained:

# In docker-compose.yml
services:
  temporal-worker:
    environment:
      TEMPORAL_RETENTION_PERIOD: "168h" #Keeps data visible for 7 days (by default)

Configuration Constraints:

• Minimum value: 1 day
• Default value: 7 days

The retention period is specified in hours. Only job and sync history within the configured retention period will be visible in the OLake UI.

Troubleshooting

Common Issues

Port Conflicts

If port binding errors occur:

1. Check what's using the ports: lsof -i :8000 (on macOS/Linux)
2. Stop conflicting services or change ports in docker-compose.yml

Database Connection Issues

• Ensure PostgreSQL container is healthy: docker compose ps
• Check PostgreSQL logs: docker compose logs postgresql

Memory Issues

• Ensure Docker has at least 4GB RAM allocated
• Check Docker resource usage: docker stats

Permission Issues

• On Linux, ensure the user is in the docker group
• Check file permissions for the ./olake-data/ directory (or custom path if modified)

Reset Everything

The stack can be completely reset with:

docker compose down -v  # Removes containers and volumes
docker compose up -d    # Fresh start

Warning: This will delete all job data and configurations.