Skip to main content

GitOps with Docker Compose

Instead of creating a source, destination, job and selecting the streams through the OLake UI, you can describe them as plain JSON files in a folder. OLake UI watches that folder. When a file is added or changed, OLake Go applies it, and the job shows up in the UI. The folder can be a Git checkout so you get a full Git workflow.

How it works​

  1. Create a folder. It can be empty.
  2. Enable GitOps in docker-compose-v1.yml: set one environment variable, mount the folder you created above into olake-ui, and start the UI.
  3. Add JSON files to the folder at any time, in any order: a source, a destination, a job and its streams. OLake UI picks up each file as it appears. The source and destination are created in OLake UI as soon as they are added to the folder. When a job has its source, destination and streams, OLake UI creates the job and runs it on the schedule set in its frequency cron expression.

Prerequisites​

  • The following versions are required for GitOps:
    • OLake UI: 0.9.0 or later
    • OLake Go: 1.0.0 or later
    • OLake Worker: 0.5.0 or later
  • A folder to hold the JSON files.

Step 1: Enable GitOps and start OLake UI​

Create a folder on your host, for example olake-gitops. It can be empty for now.

On the olake-ui service in your docker-compose-v1.yml, set OLAKE_GITOPS_MODE and mount your folder:

yaml
services:
  olake-ui:
    environment:
      OLAKE_GITOPS_MODE: file
    volumes:
      - ./olake-gitops:/etc/olake/gitops
  • OLAKE_GITOPS_MODE: file makes OLake UI watch the mounted folder and apply changes whenever files in it change.
  • Replace ./olake-gitops with the path of your folder.

OLake UI needs read access to the folder, and write access for the state file. If the folder is read-only, point OLAKE_GITOPS_STATE_FILE at a writable path. OLake UI starts only if it can write the state file. A missing or read-only location is reported at startup. See Environment variables for all the settings.

Start OLake UI:

bash
docker compose -f docker-compose-v1.yml up -d

OLake UI is now watching the folder. Anything you add from here on is synced to the UI.

Step 2: Add your objects to the folder​

Add one JSON file per object. A complete setup needs four files: a source, a destination, a job that connects them, and the streams to sync.

olake-gitops/
├── source.json
├── destination.json
├── job.json
└── streams.json

You don't have to add them together. They can be added in any order. A job is created only after its source, destination and streams files are all in the folder.

File Layout​

  • Each file describes one object, using the same fields as the OLake UI API plus a kind.
  • The value for kind can be one of source, destination, job or streams.
  • projectId is required in every file. userId is also required for source, destination and job.
  • OLake Go reads every *.json file in the folder you mounted, and in the folders that are inside it (for instance, each job can have its own folder).
  • You can name the file anything and put it in any folder. OLake Go tells objects apart by the value of kind and name inside the file. In the case of streams.json, it considers the value of job instead of name.

A source file has "kind": "source". The other fields are:

  • name: the name of the source in OLake Go.
  • type: which source you are using, such as postgres.
  • version: the OLake Go version.
  • config: the connection configuration for the source.
source.json
{
  "kind": "source",
  "projectId": "123",
  "userId": 1,
  "name": "pg_prod",
  "type": "postgres",
  "version": "v1.0.0",
  "config": {
    "host": "postgres",
    "port": 5432,
    "database": "app",
    "username": "olake",
    "password": "password"
  }
}
Credentials

Credentials live in these files. Keep the folder private and never commit real passwords to a public repository. To keep credentials out of the files, use Secret References.

Step 3: Check the job in the UI​

Open OLake UI. Each source, destination and job appears as soon as its file is applied, and the job runs on its frequency. If something does not appear, check its status in the state file (olake-gitops/.olake-gitops-state.json) or see How failures are handled.

To change the pipeline later, edit the files. OLake Go applies the change automatically.

Credentials and secrets​

You have two ways to provide credentials in a source or destination config:

  • Plain values, as in the examples above. The password sits in the file, so this suits local testing.
  • Secret references (recommended). Use this when you want the credentials outside the config file. OLake Go reads the secrets again on an interval (default 1h, set by OLAKE_GITOPS_SECRETS_REFRESH_INTERVAL), so rotated credentials are picked up without a restart.

Secret references​

Set any string field in a source or destination config to a reference instead of the value. OLake Go fetches the value when it syncs and stores the resolved config in its database.

info

References work only for Source and Destination configs.

json
"config": {
  "host": "pg.prod.internal",
  "username": "${aws-sm:olake/prod/pg#username}",
  "password": "${aws-sm:olake/prod/pg#password}"
}

The format is ${<provider>:<path>}. These providers are supported:

ProviderExampleNotes
Environment variable${env:PG_POSTGRES}Read from the olake-ui container. Set the variable on the olake-ui service.
AWS Secrets Manager${aws-sm:olake/prod/pg#password}Secret Name
GCP Secret Manager${gcp-sm:pg-password}Secret ID. Use ${gcp-sm:projects/my-proj/secrets/pg-password/versions/3} to pin a project or version.
Azure Key Vault${azure-kv:my-vault/pg-password}Vault name, then secret name. Add /<version> to pin a version.

A few rules apply to every provider:

  • The reference must be the whole value.
  • #key reads one key of a secret that holds a JSON object, such as {"username":"...","password":"..."}. Without #key, the whole secret value is used as a string.
note

OLAKE_SECRET_KEY encrypts source and destination configs in the OLake Go database. To know how to set it in Docker Compose, see Encryption Key Configuration.

Set up your secret provider​

OLake UI reads the secrets, so set these on the olake-ui service. The worker does not need them. Pick the provider you use:

On EC2 or ECS with a role, set only the region. For any setup without role based authentication, also set access keys (testing only):

docker-compose-v1.yml
services:
  olake-ui:
    environment:
      # required when references use a secret name; not needed with a full ARN
      OLAKE_GITOPS_SECRETS_AWS_REGION: ap-south-1
      # optional, testing only: set both or neither; not needed on EC2 or ECS with a role
      OLAKE_GITOPS_SECRETS_AWS_ACCESS_KEY_ID: <access-key-id>
      OLAKE_GITOPS_SECRETS_AWS_SECRET_ACCESS_KEY: <secret-access-key>

Use it in a config:

json
"password": "${aws-sm:olake/prod/pg#password}"

When a secret cannot be read​

If a secret is missing, or OLake Go has no access to it, the source or destination becomes Failed. OLake Go does not retry it by itself. Fix the secret or the access, then resync the object.

How failures are handled​

TypeExamplesStatusWhat OLake Go does
RetryableOLake Go database or Temporal briefly unreachable; a job waiting for its source, destination or streamsPending
  • Retries every 30 seconds. After 30 minutes of an infrastructure outage, it becomes Failed.
  • A job that waits for a missing source, destination or streams object stays Pending with no time limit.
Non-retryableInvalid JSON, missing field, unknown type, wrong credentials, unsupported versionFailedLogs the reason, records it in the state file, and waits for you to change something.

Fixing a failed source, destination or streams file re-runs the dependent job. You do not need to edit the job file.

Failure indicator containers​

When an object becomes Failed, the OLake worker starts a container named olake-failed-<kind>-<name>, using the worker's own image. It prints the failure reason and keeps running, so it is visible in docker ps. OLake Go removes it automatically once the object is Ready again.

<name> is the object's name (for streams, the job name) in lower case, with any character other than a letter, digit or - replaced by -. Repeated - are merged into one, - at the start or end are removed, and the full name is cut to 63 characters. For example, the source pg_prod gives olake-failed-source-pg-prod.

bash
# List all failed objects
docker ps --filter label=olake.io/indicator=true
 
# Read the failure reason
docker logs olake-failed-source-pg-prod
tip

Set up an alert that fires whenever a container with olake.io/indicator=true exists.

Resync manually​

Editing a file already makes OLake Go re-apply it, so most fixes need nothing more. The exception is when the problem was outside the file. OLake Go still sees the same file content, so it keeps skipping the Failed object. Common cases:

  • You created a missing secret.
  • You fixed a password in the database.
  • An infrastructure outage that lasted more than 30 minutes has ended.

For these, add a reconcileRequest field to the file. It is a nudge that tells OLake Go to try again, without changing anything else about the object.

  1. Open the file of the failed object and add reconcileRequest at the top level. The value can be any number or string.

    source.json
      {
     "kind": "source",
     "reconcileRequest": 1,
     "projectId": "123",
     "userId": 1,
     "name": "pg_prod",
     "type": "postgres",
     "version": "v1.0.0",
     "config": { "host": "postgres", "port": 5432, "database": "app", "username": "olake", "password": "${env:PG_PASSWORD}" }
     }

  2. Save the file and confirm it worked. OLake Go stores the value it handled as reconcileHandled in the state file, next to the object's phase and message. The key is <kind>//<name> (for streams, use their job):

    bash
    jq '.["source//pg_prod"] | {phase, message, reconcileHandled}' olake-gitops/.olake-gitops-state.json

To resync the same object again later, change the value, for example from 1 to 2. It only has to differ from the previous one, so a counter is fine.

caution

To resync one object, use reconcileRequest. Do not delete the whole state file. If you do, OLake Go rechecks every object on the next restart and runs stream discovery for every job, which is slow. If discovery finds a change in the source, such as a new column, that job is updated and its running sync stops.

Important behavior​

  • Create and update only. Deleting a file does not delete anything in OLake Go. The source, destination or job stays as it was last applied, and GitOps no longer manages it.
  • Do not rename sources, destinations or jobs.
  • Keep objects unique. If two files describe the same kind and name, they conflict and the last one wins.
  • Stream changes clear data. When a stream's configuration changes, OLake Go clears the data for the changed streams and re-syncs them.

Environment variables​

Set these under environment of the olake-ui service only.

VariableRequired?Why
OLAKE_GITOPS_MODERequired, set to file.Turns GitOps on.
OLAKE_GITOPS_FILE_DIROptional. Default: /etc/olake/gitops.Tells OLake Go which folder to read. Set it only if your files are somewhere else.
OLAKE_GITOPS_STATE_FILEOptional. Required when the folder is mounted read-only.Tells OLake Go where to write its status file. Point it at a writable, mounted path.
OLAKE_GITOPS_SECRETS_REFRESH_INTERVALOptional. Default: 1h.How often OLake Go reads secrets again to pick up rotated values. Use a unit, such as 15m. 0 turns it off.
OLAKE_GITOPS_SECRETS_AWS_REGIONRequired when an ${aws-sm:...} reference uses a secret name. Not needed when it uses the full secret ARN.Tells OLake Go which AWS region to read from.
OLAKE_GITOPS_SECRETS_AWS_ACCESS_KEY_IDOptional, testing only. Set both AWS keys or neither. Not needed on EC2 or ECS with a role.AWS login with access keys.
OLAKE_GITOPS_SECRETS_AWS_SECRET_ACCESS_KEYOptional, testing only. Set both AWS keys or neither.AWS login with access keys.
OLAKE_GITOPS_SECRETS_GCP_PROJECTRequired when a ${gcp-sm:...} reference uses a short secret ID and OLake Go runs outside a GCE VM. Not needed with the full projects/... form, or on a GCE VM.Tells OLake Go which GCP project the secret is in.
GOOGLE_APPLICATION_CREDENTIALSRequired when OLake Go runs outside a GCE VM. Not needed on a GCE VM.Path to the GCP credential file inside the container. Mount the file too.
OLAKE_GITOPS_SECRETS_AZURE_TENANT_IDOptional, testing only. Set all three Azure values or none. Not needed on an Azure VM with a managed identity.Azure login with a service principal.
OLAKE_GITOPS_SECRETS_AZURE_CLIENT_IDOptional, testing only. Set all three Azure values or none.Azure login with a service principal.
OLAKE_GITOPS_SECRETS_AZURE_CLIENT_SECRETOptional, testing only. Set all three Azure values or none.Azure login with a service principal.
note

For production, do not use access keys or client secrets. Let OLake Go log in with the cloud identity of where it runs, as shown in Set up your secret provider.