Troubleshooting OLake

info

Read debug docs here

1. File name too long & FATAL error occurred while reading records: failed to finish backfill chunk 381: main writer closed

2025-02-17T07:03:00Z ERROR main writer closed, with error: failed to create parititon file: failed to create directories[output/otter_db/stream_8/H.

Solution: The max filename length is 255 bytes and this error shows that you have excceded that limit for file creation (might happen if you partion based on STRING field that contain values that are too large). Usually in a linux system, these limits are defined at:

cat /usr/include/linux/limits.h

...
#define NAME_MAX         255    /* # chars in a file name */
#define PATH_MAX        4096    /* # chars in a path name including nul */
...

2. No encryption (SQLSTATE 28000)

FATAL failed to ping database: failed to connect to `host=product-team-testing.postgres.database.azure.com user=product_team database=postgres`: server error (FATAL: no p
g_hba.conf entry for host "4.240.65.100", user "product_team", database "postgres", no encryption (SQLSTATE 28000))

Solution: Change the ssl.mode: disable to ssl.mode: allow to resolve this issue. This is a known issue with Azure Postgres and the solution is to set the ssl.mode to allow in the connection string.

Unexpected EOF

Running Postgres Sync ran into following error: FATAL error occurred while reading records: failed to receive message from wal: receive message failed: unexpected EOF - Solution: Set the value of intial_wait_time: 10 in case if you have increased the value in source database config file. This could be because of network issues, try again in that case.

3.Connecting to Supbase

OLake can connect seamlessly to Supabase Postgres databases using any of the three connection modes that Supabase provides:

• Direct Connection – Connects directly to the Postgres instance.
  Note: By default, Supabase's direct connection supports only IPv6. IPv4 support can be enabled via the paid IPv4 add-on.

• Transaction Pooler – Uses PgBouncer in transaction pooling mode. Suitable for most use cases that do not require session-level features.

• Session Pooler – Recommended when the client network supports only IPv4. In this mode, each client gets a dedicated database session.

⚠️ CDC Support: The Direct Connection mode is required for CDC (Change Data Capture) operations, as Supabase permits replication slot creation only when connecting directly to the Postgres instance.

IP Version Compatibility

OLake fully supports both IPv4 and IPv6 connections. However, Supabase’s IP compatibility depends on the connection mode and account configuration.

To check your machine or cloud environment’s IP version, you can run:

curl -6 https://ifconfig.co/ip

The format of the returned address indicates the IP version:

• Addresses with four decimal numbers (0–255) separated by dots (.), for eg. 192.168.1.1, indicate IPv4.
• Addresses containing hexadecimal segments separated by colons (:), such as 2404:6800:4007:821::200e, indicate IPv6.

Docker IPv6 Support

By default, Docker containers support only IPv4, which may block access to Supabase’s Direct Connection if IPv6 is required. IPv6 can be enabled as follows:

1. Open the Docker Desktop Dashboard.
2. Click the Settings icon (⚙️) in the top-right corner.
3. Select Docker Engine from the left-hand sidebar.
4. In the JSON configuration editor, add the following line

"ipv6":true

5. Click Apply & Restart to save the changes and restart Docker.

RLS (Row-Level Security) Considerations

When using a read-only database role for syncing with OLake, ensure that Row-Level Security (RLS) is either disabled or properly configured. If RLS policies restrict access, sync operations may complete without error but result in zero rows being replicated.

To allow a specific role to bypass RLS entirely, execute the following command (replace <rolename> with the actual role name):

ALTER USER <rolename> WITH BYPASSRLS;