OLake Go Commands and Flags

Streams v2 is here

• Streams config now uses two files: selected_streams.json (the file you edit) and available_streams.json (read-only, written by discover).

• Streams v2 and the new CLI flags are supported from OLake Go v1.0.0 or later.

• See the migration guide to migrate to Streams v2 from Streams v1 or the Streams v1 reference for more details on the legacy format.

OLake Go provides a set of CLI commands, each designed for specific use cases.

Commands can be executed in three ways:

• Using the build.sh script
• Running the generated binary
• Through the OLake Go Docker CLI.

build.sh command

./build.sh driver-[SOURCE-TYPE] [COMMAND] [FLAG]

Compatibility

The ./build.sh script is a Unix shell script. It runs natively on Linux and macOS terminals.

For Windows:

• Use Git Bash, WSL (Windows Subsystem for Linux), or another Unix-like shell.
• Alternatively, use the OLake Go Docker CLI, which works consistently across platforms.

Explanation of placeholders:

• SOURCE-TYPE → The source driver being used.
• COMMAND → The action to perform (listed in the sections below).
• FLAG → Additional configuration for fine-tuning the command.

Using the binary

Executing build.sh creates a binary file in the configuration directory.

Once available, the binary can be called directly to run commands, without needing build.sh.

Commands

1. Check

bash

./build.sh driver-[SOURCE-TYPE] check [FLAG]

Description:

Verifies the connection to either a source or a destination. For source connectors that support CDC, the check also evaluates CDC prerequisites:

• Required checks must pass for a successful test connection.
• Optional checks are not required. They show recommended values for good to have settings. You can fix those settings or continue without changing them.

Required flags:

Only one among these is required:

• --config → Checks the connection to a source.
• --destination → Checks the connection to a destination.

2. Spec

bash

./build.sh driver-[SOURCE-TYPE] spec [FLAG]

Description:

Generates a JSON Schema and UI Schema. These schemas are used by RJSF to render and validate configuration forms.

When no flag is provided, the spec of the defined SOURCE-TYPE is generated.

Optional flags:

• --destination-type → Generates the spec for a destination driver instead of the source.
  
• --available-query-engines → Lists the query engines OLake Go supports. Use these names as input for --target-query-engines while running Discover command.

3. Discover

bash

./build.sh driver-[SOURCE-TYPE] discover [FLAG]

Description:

Writes selected_streams.json, available_streams.json and the legacy streams.json file containing information about all streams in the source.

Required flags:

• --config → Specifies path to the source configuration file. Not needed for --difference-* runs.

Optional flags:

• --selected-streams + --available-streams → Re-run discover with both flags so available_streams reflects all tables with updated schema at the source, and selected_streams keeps the tables you already chose and your edits, including schema updates for those streams.
• --difference-selected-streams + --difference-available-streams → Pass both flags to compare current and updated stream files and writes difference_available_streams.json and difference_selected_streams.json.
• --destination-database-prefix → Adds a custom prefix to the destination database name.
• --timeout → Overrides the default timeout value for the command.
• --max-discover-threads → Max number of parallel threads for discovery of table in database.
• --target-query-engines → Comma-separated query engines (for example trino,databricks,spark) so discover lists only delete formats all of these engines support in available_streams.json and the selected_streams.json will include the higher priority delete format supported by all the engines.

4. Sync

bash

./build.sh driver-[SOURCE-TYPE] sync [FLAG]

Description:

Used to sync the data from the source to the destination.

Required flags:

All of these flags need to be specified:

• --config → Specifies path to the source configuration file.
• --selected-streams + --available-streams → Specify paths to selected_streams.json and available_streams.json (produced by the discover command).
• --destination → Specifies path to the destination configuration file.

Optional flags:

• --state → Specifies path to the state file.
• --discover-schema → Enables source schema discovery during sync.

info

➡️ You must learn about stats.json configuration. Refer to the Stats Config guide.

5. Clear Destination

bash

./build.sh driver-[SOURCE-TYPE] clear-destination [FLAG]

Description:

• Clears data in the destination, only for the streams listed in selected_streams.json (or in legacy streams.json).

Required flags:

All of these flags need to be specified:

• --selected-streams + --available-streams → Specify paths to selected_streams.json and available_streams.json (produced by the discover command).
• --destination → Specifies path to the destination configuration file.

Optional flags:

• --state → Used to reset the state file for selected streams.

---

Flags

1. Help

bash

./build.sh driver-[DRIVER_NAME] --help

Description:

• Lists all available commands and flags for the current OLake CLI version.
• Can be run without specifying a command.
• The shorthand -h can also be used.

2. Config

bash

./build.sh driver-[SOURCE-TYPE] [COMMAND] --config [PATH_TO_CONFIG_FILE]

Description:

• Specifies the path to the source configuration file.
• For details about configuration files for different sources, see:
  • MongoDB Config file
  • MySQL Config file
  • Oracle Config file
  • PostgreSQL Config file
  • S3 Config file
  • DB2 Config file
  • MSSQL Config file
  • Kafka Config file

3. Available streams

bash

./build.sh driver-[SOURCE-TYPE] [COMMAND] --selected-streams [PATH_TO_SELECTED_STREAMS_FILE] --available-streams [PATH_TO_AVAILABLE_STREAMS_FILE]

Description:

• Specifies the path to available_streams.json. Discover writes this read-only file. Do not edit it.
• Must be passed together with --selected-streams.
• Used by discover (merge, keeps your edits), sync and clear-destination.
• For the file format, see Streams configuration.

4. Selected streams

bash

./build.sh driver-[SOURCE-TYPE] [COMMAND] --selected-streams [PATH_TO_SELECTED_STREAMS_FILE] --available-streams [PATH_TO_AVAILABLE_STREAMS_FILE]

Description:

• Specifies the path to selected_streams.json. This is the file you edit. It lists the streams to sync and their settings.
• Must be passed together with --available-streams.
• Used by discover (merge, keeps your edits), sync and clear-destination.
• For the file format, see Streams configuration.

5. Destination

bash

./build.sh driver-[SOURCE-TYPE] [COMMAND] --destination [PATH_TO_DESTINATION_FILE]

Description:

• Specifies the path to the destination configuration file.
• For details about destination configuration files, see:
  • AWS Glue Catalog configuration
  • Generic REST Catalog configuration
  • Lakekeeper Catalog configuration
  • Nessie Catalog configuration
  • S3 Tables Catalog configuration
  • Unity Catalog configuration
  • Apache Polaris Catalog configuration
  • BigLake Catalog configuration
  • JDBC Catalog configuration
  • Hive Catalog configuration
  • Parquet configuration

6. State

bash

./build.sh driver-[SOURCE-TYPE] [COMMAND] --state [PATH_TO_STATE_FILE]

Description:

• Specifies the path to the state file.
• The state file contains metadata (such as offsets and positions) that enables:
  • Resuming interrupted syncs.
  • Continuing incremental or CDC syncs without restarting from scratch.
  • Storing the version in the state file for maintaining backward compatibility.

The state.json file is organized into two main sections:

1. Global State

The global section contains global state information that applies to all streams and driver-specific replication metadata that tracks the overall position in the source database's change log. The structure varies by database driver:

MySQL

MySQL uses binlog position for global state tracking to maintain the replication position across all streams.

state.json (MySQL)

{
  "type": "STREAM",
  "version": 1,
  "global": {
    "state": {
      "server_id": 261398335,
      "state": {
        "position": {
          "Name": "mysql-bin.000070",
          "Pos": 811746
        }
      }
    },
    "streams": [
      "my_db.decimal_test",
      "my_db.incr_test"
    ]
  }
}

PostgreSQL

PostgreSQL uses LSN (Log Sequence Number) for global state tracking to maintain the replication position across all streams.

state.json (PostgreSQL)

{
  "type": "STREAM",
  "version": 1,
  "global": {
    "state": {
      "lsn": "BD7/650015C8"
    },
    "streams": [
      "public.sample_data",
      "public.employees"
    ]
  }
}

MongoDB

MongoDB does not use a global state section. The state is maintained at the stream level only.

2. Streams State

The streams section is an array where each element tracks the synchronization state for a specific stream. The structure varies by database driver:

MySQL & PostgreSQL

Each stream state object contains:

state.json

{
  "stream": "table1",
  "namespace": "my_db",
  "sync_mode": "",
  "state": {
    "chunks": []
  }
}

MongoDB

MongoDB stream state includes a _data field that stores the resume token:

state.json (MongoDB)

{
  "stream": "users",
  "namespace": "public",
  "sync_mode": "",
  "state": {
    "_data": "82696F0837000000012B0429296E1404",
    "chunks": []
  }
}

State Configuration Elements

Component	Type	Example Value	Description
version	integer	1 or 0	Version 0 enables legacy, lenient handling for backward compatibility Version 1 and above enforce stricter validation and fail-fast behavior for newly created state
global	object	For postgres : "global": { "state" : { "lsn": "BD7/650015C8" }, "streams": [ "public.sample_data", "public.employees" ] }	Contains global replication metadata. Structure varies by driver: MySQL uses binlog position, PostgreSQL uses LSN, MongoDB does not have a global section.
global.state.server_id	integer	261398335	(MySQL only) The MySQL server ID used for replication tracking.
global.state.state.position	object	{"Name": "mysql-bin.000070", "Pos": 811746}	(MySQL only) Tracks the current binlog file name and position for CDC replication.
global.state.lsn	string	"BD7/650015C8"	(PostgreSQL only) Log Sequence Number (LSN) that tracks the position in the PostgreSQL write-ahead log (WAL) for CDC replication.
global.streams	array	["public.decimal_test", "public.incr_test"]	List of all streams that are being tracked in this state file.
stream	string	"decimal_test", "incr_test"	The name of the stream being tracked. Must match the stream name defined in streams.json.
namespace	string	"my_db", "public"	The namespace (database/schema) that the stream belongs to.
sync_mode	string	"", "incremental", "cdc"	The synchronization mode being used for this stream. May be empty if not explicitly set.
state.chunks	array	[]	Array tracking data chunks that have been processed. Used for resuming partial syncs and managing large data transfers.
state._data	string	"82696F0837000000012B0429296E1404"	(MongoDB only) Resume token used to track the position in MongoDB's change stream for CDC replication.

What's Next: The state file is automatically created and updated during sync operations. You can manually specify a state file using the --state flag to resume from a previous synchronization point.

7. Destination database prefix

bash

./build.sh driver-[SOURCE-TYPE] [COMMAND] --destination-database-prefix [PREFIX_TO_ADD]

Example:
./build.sh driver-mysql discover --config [PATH_TO_SOURCE_CONFIG_FILE] --destination-database-prefix olake

Description:

• Adds a custom prefix to the database name created in the destination.
• Example:

  If the source database is sales-db and the driver is mysql:

  • Default (Normalized) → mysql_sales_db
  • With prefix (Normalized) → olake_sales_db

8. Destination type

bash

./build.sh driver-[SOURCE-TYPE] [COMMAND] --destination-type [TYPE_OF_DESTINATION]

Description:

• Used with the spec command to generate JSON Schema and UI Schema for the specified destination.
• TYPE_OF_DESTINATION can be any OLake Go supported destination, for example: iceberg or parquet.

9. Decryption of configuration files

bash

./build.sh driver-[SOURCE-TYPE] [COMMAND] --encryption-key [DECRYPTION_KEY]

Example:
./build.sh driver-mysql check config [PATH_TO_SOURCE_CONFIG_FILE] --encryption-key hello-world

Description:

• Provides a key for OLake Go to decrypt encrypted configuration files during execution.
• Supported values include KMS keys, UUIDs, or custom strings.
• The flag must follow the encrypted file in the command.
• Example: In this case, if the source config file is encrypted, OLake Go uses the provided key (hello-world) to decrypt and parse it.

10. No Save

bash

./build.sh driver-[SOURCE-TYPE] [COMMAND] --no-save

Description:

• Prevents saving of any files generated by the command. This flag is valid for all available commands.
• Example: If used with discover, the generated streams files and related logs are not saved.

11. Timeout

bash

./build.sh driver-[SOURCE-TYPE] [COMMAND] --timeout [TIMEOUT_IN_SECONDS]

Description:

• Applies only to the discover command.
• Overrides the default timeout of 300 seconds (5 minutes).
• This is helpful when working with large datasets or slower networks where the operation may need extra time to complete.

12. Max discover threads

bash

./build.sh driver-[SOURCE-TYPE] [COMMAND] --max-discover-threads [NUMBER_OF_THREADS]

Description:

• Applies only to the discover command.
• Sets the maximum number of parallel threads used for discovering table schemas in the database.
• Value: Integer (mandatory when the flag is used). Minimum value is 1 (must be greater than 0).
• Default: If the flag is not provided, the value defaults to 50.

13. Discover schema

bash

./build.sh driver-[SOURCE-TYPE] [COMMAND] --discover-schema

Description:

• Applies only to the sync command.
• By default, sync skips source schema discovery to keep runs fast and trusts the streams files passed (--selected-streams and --available-streams, or the deprecated --streams) from a prior discover run.
• When this flag is passed, sync re-discovers source schema and validates configured streams against the source.

14. Difference available streams

bash

./build.sh driver-[SOURCE-TYPE] discover --selected-streams /path/selected_streams.json --available-streams /path/available_streams.json --difference-selected-streams /path/new_selected_streams.json --difference-available-streams /path/new_available_streams.json

Description:

• Used with the discover command to compare differences between two available_streams.json files, specifically an old file and a new file.
• Must be used together with --available-streams /path/old_available_streams.json to specify the path to the old available streams file.
• The --difference-available-streams flag specifies the path to the new available_streams.json that will be compared against the old one.
• Running this command generates a difference_available_streams.json file containing the differences between the old and new available streams files.

Internal testing command

This command is primarily used internally for testing purposes to compare differences between old and new available_streams.json files. It is not typically required for production use. However, if you want to check the differences between available streams, you can use this command.

15. Difference selected streams

bash

./build.sh driver-[SOURCE-TYPE] discover --selected-streams /path/selected_streams.json --available-streams /path/available_streams.json --difference-selected-streams /path/new_selected_streams.json --difference-available-streams /path/new_available_streams.json

Description:

• Used with the discover command to compare differences between two selected_streams.json files, specifically an old file and a new file.
• Must be used together with --selected-streams /path/old_selected_streams.json to specify the path to the old selected streams file.
• The --difference-selected-streams flag specifies the path to the new selected_streams.json that will be compared against the old one.
• Running this command generates a difference_selected_streams.json file containing the differences between the old and new selected streams files.

Internal testing command

This command is primarily used internally for testing purposes to compare differences between old and new selected_streams.json files. It is not typically required for production use. However, if you want to check the differences between selected streams, you can use this command.

16. Target query engines

bash

./build.sh driver-[SOURCE-TYPE] discover --config [PATH_TO_CONFIG_FILE] --target-query-engines trino,databricks,spark

Description:

• Applies only to the discover command.
• Comma-separated list of query engines you plan to use to read synced Iceberg tables (for example trino, databricks, spark).
• Discover limits each stream's available_update_types in available_streams.json to delete formats those engines can read. With multiple engines, only formats supported by all listed engines are included (the intersection).
• Use this when editing selected_streams.json so update_type choices match your query engines.
• To know more about the behavior of this flag, see Target Query Engines.

17. Available query engines

bash

./build.sh driver-[SOURCE-TYPE] spec --available-query-engines

Description:

• Used with the spec command.
• Prints the query engine identifiers OLake Go supports (for example trino, databricks, spark).
• Use these values in the comma-separated list for --target-query-engines on discover so delete format options match what your engines can read.
• For which delete formats each engine supports, see Delete format support by query engine.