Skip to main content

Overview

In addition to MySQL, OLake Go also supports MariaDB flavor as CDC source. It supports multiple sync modes and offers features like parallel chunking, checkpointing, and automatic resume for failed full loads. This connector can be used within the OLake UI or run locally via Docker for open-source workflows.

Sync Modes Supported​

  • Full Refresh
  • Full Refresh + CDC
  • CDC Only
  • Full Refresh + Incremental

Prerequisites​

Version Prerequisites​

MySQL Version: MySQL 5.7+

CDC Prerequisite​

Connection Prerequisites​

  • Read access to the tables for the MySQL user.

After initial Prerequisites are fulfilled, the configurations for MySQL can be configured.

Configuration​

1. Navigate to the Source Configuration Page​

  1. Complete the OLake UI Setup Guide
  2. After logging in to the OLake UI, select the Sources tab from the left sidebar.
  3. Click Create Source on the top right corner.
  4. Select MySQL from the connector dropdown
  5. Provide a name for this source.

2. Provide Configuration Details​

  • Enter MySQL credentials.

Change data capture (CDC)​

FieldDescriptionExample Value
Enable CDCEnables change data capture to sync like inserts, updates, and deletes. When enabled, CDC-specific fields appear below. When disabled, supports only Full Refresh and Incremental sync modes.Enabled (default)
Initial Wait TimeMaximum duration in seconds to wait before considering the binlog syncer idle.10

Connection details​

FieldDescriptionExample Value
MySQL Host requiredList of database host addresses to connect to.mysql-host
DatabaseName of the target database to use.mysql-db
Username requiredUsername for authenticating with the database.mysql-user
Password requiredPassword for the database user.mysqlpwd
Port requiredPort number on which the database server is listening.3306
JDBC URL ParametersExtra JDBC URL parameters for fine-tuning the connection.{"connectTimeout":"10000"}
SSL ConfigurationDefines whether the connection to the database is encrypted over TLS/SSL. Needed to meet security requirements and connect to databases that do not allow plain-text traffic.
  • require
  • disable
  • verify-ca
  • verify-full
Skip TLS VerificationIndicates whether to skip TLS certificate verification.false

Additional configuration​

FieldDescriptionExample Value
Max ThreadsMaximum number of parallel threads for processing or syncing data.3
Backoff Retry CountNumber of retry attempts for establishing sync with exponential backoff.3
SSH ConfigConfigure OLake Go to connect through an SSH tunnel.
  • No Tunnel
  • SSH Key Authentication
  • SSH Password Authentication

3. Test Connection​

  • Click Set up source to validate the configuration. The test performs three checks:

    Testing Source Configuration modal showing connection success, required MySQL CDC prerequisite checks, and optional CDC settings

    1. Connection validation: Verifies that OLake Go can reach the MySQL server using the provided host, port, credentials, and SSL settings.

    2. Required CDC prerequisites validation (when CDC is enabled): Evaluates required server-side settings to ensure reliable CDC operation:

      • binlog_access: User must have binlog read privileges
      • log_bin: Binary logging must be enabled
      • binlog_format: Must be set to ROW
      • binlog_row_image: Must be set to FULL

    3. Optional CDC prerequisites validation (when CDC is enabled): Evaluates optional server-side settings:

      • binlog_row_metadata: Binlog row metadata should be set to FULL
      • binlog_retention: Binlog retention must be sufficient for expected sync pauses

  • If any required check fails, the test connection will show the current and recommended values. Fix the server-side settings and retry the connection.

  • If any optional check fails, the test connection will show the current and recommended values. You can fix the server-side settings and retry, or continue and create the source without making changes.

  • If all checks pass, the MySQL source is created successfully. Jobs can then be configured using this source.

  • In case of connection failure, refer to the Troubleshooting section.

Data Type Mapping​

MySQL Data TypesDestination Data Type
int, int unsigned, mediumint, mediumint unsigned, smallint, smallint unsigned, tinyint, tinyint unsignedint
bigint, bigint unsignedbigint
float, decimal(10,2)float
double, double precision, realdouble
datetime, timestamptimestamptz
char, varchar, text, tinytext, mediumtext, longtext, enum, json, bit(1), timestring
INT UNSIGNED mapping change

From OLake Go version 0.4.1 and above, INT UNSIGNED maps to bigint in the destination. In versions earlier than 0.4.1, it maps to int.

timestamptz timezone

OLake Go always ingests timestamp data in UTC format, independent of the source timezone.

Date and Time Handling​

During transfer, values in date, time, and timestamp columns are modified to ensure valid calendar ranges and destination compatibility.

  • Case I (Year 0000):
    Source dates with year 0000 are not valid in most destinations, so we change them to the epoch start date.
    Example: 0000-05-10 → 1970-01-01
  • Case II (Year > 9999):
    Extremely large years are capped at 9999. The month and date are not affected.
    Examples: 10000-03-12 → 9999-03-12
  • Case III (Invalid month/day):
    When the month or day exceeds valid ranges (i.e. month > 12 or day > 31), or the combined date is invalid, the value is replaced with the epoch start date.
    Examples: 2024-13-15 → 1970-01-01, 2023-04-31 → 1970-01-01
Note

These rules apply to date, time, and timestamp columns during transfer.

Troubleshooting​

1. Failed to Get Current Binlog Position​

Cause: Binary logging not enabled, wrong format, or insufficient privileges.
Fix:

  • Ensure binary logging is enabled:
sql
SHOW VARIABLES LIKE 'log_bin'; // Should return `ON`
  • Ensure row-based logging:
sql
SHOW VARIABLES LIKE 'binlog_format'; // Should return `ROW`.
  • Grant required privileges:
sql
GRANT REPLICATION CLIENT, REPLICATION SLAVE ON *.* TO 'your_user'@'%';

2. Idle Timeout Reached, Exiting Binlog Syncer​

  • Cause: During CDC sync, OLake Go saves the binlog position as state. When the next CDC sync starts, OLake Go searches for this saved position in MySQL's binlog to resume from where it left off. If this search operation takes longer than the configured initial_wait_time (for example, if initial_wait_time is set to 10 seconds but searching the binlog takes 15 seconds), the timeout is triggered and the binlog syncer exits.
  • Fix: Increase the initial_wait_time value in your source configuration to allow sufficient time for OLake Go to locate the saved binlog position.

3. Column Count Mismatch: Expected X, Got Y​

  • Cause: Table schema changed after CDC started or incomplete binlog metadata.
  • Fix:
    • Ensure full binlog metadata:
    sql
    SHOW VARIABLES LIKE 'binlog_row_metadata';
    Should return FULL. To set it:
    sql
    SET GLOBAL binlog_row_metadata = 'FULL';
    Update my.cnf or my.ini for persistence.
    • Restart the connector after schema changes.

4. Failed to Get or Split Chunks​

  • Cause: Table stats not populated or table contains 0 records.

  • Fix: Run the following query to populate table statistics:

    sql
    ANALYZE TABLE <namespace>.<table_name>;

5. Failed to Load MySQL Timezone Location​

  • Cause: If you see the log message failed to load mysql timezone location %s, falling back to UTC. Set jdbc_url_params.time_zone to override: %s, it indicates that the MySQL timezone could not be loaded properly. This often occurs when timezone abbreviations (like 'IST') are used, which can lead to inconsistencies since multiple regions share the same short name (e.g., India, Israel, Ireland).
  • Fix:

    • First, check the current timezone settings:

      sql
      SELECT @@session.time_zone, @@global.time_zone, @@system_time_zone;

      If any of these returns IST or another abbreviation, this can cause inconsistency issues.

    • Option 1 (Set via JDBC using IANA timezone name):

      • Use IANA timezone names (e.g., Asia/Kolkata) instead of abbreviations. Set this in the JDBC URL parameters:
        json
        {
          "jdbc_url_params": {
            "time_zone": "Asia/Kolkata"
          }
        }
      • Verify that IANA timezones are available in your database:
        sql
        SELECT count(*) FROM mysql.time_zone_name;
        If this returns 0, the timezone tables are not present. In this case you should use Option 2 instead.

    • Option 2 (Use timezone offset):

      • If IANA timezone names are not available in your environment, use a fixed offset instead:
        json
        {
          "jdbc_url_params": {
            "time_zone": "+05:30"
          }
        }

If the issue is not listed here, post the query on Slack to get it resolved within a few hours.

Changelog​

Date of ReleaseVersionDescription
27 August 20250.1.11override default timeout in Discover