OLake Commands and Flags

OLake provides a set of CLI commands, each designed for specific use cases.

Commands can be executed in three ways:

• Using the build.sh script
• Running the generated binary
• Through the OLake Docker CLI.

build.sh command

./build.sh driver-[SOURCE-TYPE] [COMMAND] [FLAG]

Compatibility

The ./build.sh script is a Unix shell script. It runs natively on Linux and macOS terminals.

For Windows:

• Use Git Bash, WSL (Windows Subsystem for Linux), or another Unix-like shell.
• Alternatively, use the OLake Docker CLI, which works consistently across platforms.

Explanation of placeholders:

• SOURCE-TYPE → The source driver being used.
• COMMAND → The action to perform (listed in the sections below).
• FLAG → Additional configuration for fine-tuning the command.

Using the binary

Executing build.sh creates a binary file in the configuration directory.
Once available, the binary can be called directly to run commands, without needing build.sh.

Commands

1. Check

./build.sh driver-[SOURCE-TYPE] check [FLAG]

Description:

Verifies the connection to either a source or a destination.

Required flags:

Only one among these is required:

• --config → Checks the connection to a source.
• --destination → Checks the connection to a destination.

2. Spec

./build.sh driver-[SOURCE-TYPE] spec [FLAG]

Description:

Generates a JSON Schema and UI Schema. These schemas are used by RJSF to render and validate configuration forms.
When no flag is provided, the spec of the defined SOURCE-TYPE is generated.

Optional flags:

• --destination-type → Generates the spec for a destination driver instead of the source.
  example:

  ./build.sh driver-mysql spec --destination-type iceberg

3. Discover

./build.sh driver-[SOURCE-TYPE] discover [FLAG]

Description:

Generates a streams.json file containing information about all streams in the source.

Required Flags:

• --config → Path to the source configuration file.

Optional Flags:

• --destination-database-prefix → Adds a custom prefix to the destination database name.
• --streams→ Useful when streams.json has been manually modified. This flag ensures that the existing changes are preserved, while also merging in any new updates from the database (such as newly added tables).
• --timeout → Overrides the default timeout value for the command.

4. Sync

./build.sh driver-[SOURCE-TYPE] sync [FLAG]

Description:

Used to sync the data from the source to the destination.

Required Flags:

All of these flags need to be specified:

• --config → Specifies path to the source configuration file.
• --streams → Specifies path to the streams.json file (produced by the discover command).
• --destination : Specifies path to the destination configuration file.

Optional Flags:

• --state: Specifies path to the state file.
• --clear-destination: Clears existing data for the selected streams in the destination before syncing.

---

Flags

1. Help

./build.sh driver-[DRIVER_NAME] --help

Description:

• Lists all available commands and flags for the current OLake CLI version.
• Can be run without specifying a command.
• The shorthand -h can also be used.

2. Config

./build.sh driver-[SOURCE-TYPE] [COMMAND] --config [PATH_TO_CONFIG_FILE]

Description:

• Specifies the path to the source configuration file.
• For details about configuration files for different sources, see:
  • MongoDB Config file
  • MySQL Config file
  • Oracle Config file
  • PostgreSQL Config file

3. Streams

./build.sh driver-[SOURCE-TYPE] [COMMAND] --streams [PATH_TO_STREAMS_FILE]

Description:

• Specifies the path to the streams.json file. This file is generated after the discover command.
• When used during discovery, this flag updates the existing streams.json:
  • Keeps prior manual changes.
  • Adds new streams detected in the source database.

To know more about streams.json, read Streams Config.

4. Destination

./build.sh driver-[SOURCE-TYPE] [COMMAND] --destination [PATH_TO_DESTINATION_FILE]

Description:

• Specifies the path to the destination configuration file.
• For details about destination configuration files, see:
  • AWS Glue Catalog configuration
  • REST Catalog configuration
  • JDBC Catalog configuration
  • Hive Catalog configuration
  • Parquet configuration

5. State

./build.sh driver-[SOURCE-TYPE] [COMMAND] --state [PATH_TO_STATE_FILE]

Description:

• Specifies the path to the state file.
• The state file contains metadata (such as offsets and positions) that enables:
  • Resuming interrupted syncs.
  • Continuing incremental or CDC syncs without restarting from scratch.

6. Clear Destination

./build.sh driver-[SOURCE-TYPE] [COMMAND] --clear-destination

Description:

• Clears data in the destination, only for the selected streams defined in streams.json.
• Resets the state file for those streams.

7. Destination database prefix

./build.sh driver-[SOURCE-TYPE] [COMMAND] --destination-database-prefix [PREFIX_TO_ADD]

Description:

• Adds a custom prefix to the database name created in the destination.
  Example:

  ./build.sh driver-mysql discover --config [PATH_TO_SOURCE_CONFIG_FILE] --destination-database-prefix olake
  If the source database is sales-db and the driver is mysql:
  • Default (Normalized) → mysql_sales_db
  • With prefix (Normalized) → olake_sales_db

8. Destination type

./build.sh driver-[SOURCE-TYPE] [COMMAND] --destination-type [TYPE_OF_DESTINATION]

Description:

• Used with the spec command to generate JSON Schema and UI Schema for the specified destination.
• TYPE_OF_DESTINATION can be any OLake supported destination, for example: iceberg or parquet.

9. Decryption of configuration files

./build.sh driver-[SOURCE-TYPE] [COMMAND] --encryption-key [DECRYPTION_KEY]

Description:

• Provides a key for OLake to decrypt encrypted configuration files during execution.
• Supported values include KMS keys, UUIDs, or custom strings.
• The flag must follow the encrypted file in the command.
  Example:

  ./build.sh driver-mysql check config [PATH_TO_SOURCE_CONFIG_FILE] --encryption-key hello-world
  In this case, if the source config file is encrypted, OLake uses the provided key (hello-world) to decrypt and parse it.

10. No Save

./build.sh driver-[SOURCE-TYPE] [COMMAND] --no-save

Description:

• Prevents saving of any files generated by the command. This flag is valid for all available commands.
• Example: If used with discover, the streams.json file and related logs are not saved.

11. Timeout

./build.sh driver-[SOURCE-TYPE] [COMMAND] --timeout [TIMEOUT_IN_SECONDS]

Description:

• Applies only to the discover command.
• Overrides the default timeout of 300 seconds (5 minutes).
• This is helpful when working with large datasets or slower networks where the operation may need extra time to complete.