Troubleshooting OLake

1. file name too long & FATAL error occurred while reading records: failed to finish backfill chunk 381: main writer closed

2025-02-17T07:03:00Z ERROR main writer closed, with error: failed to create parititon file: failed to create directories[output/otter_db/stream_8/H.

Solution: The max filename length is 255 bytes and this error shows that you have excceded that limit for file creation (might happen if you partion based on STRING field that contain values that are too large). Usually in a linux system, these limits are defined at:

cat /usr/include/linux/limits.h

...
#define NAME_MAX         255    /* # chars in a file name */
#define PATH_MAX        4096    /* # chars in a path name including nul */
...