Troubleshooting Iceberg Writer

1. maven related issues

Iceberg JAR file not found. Building with Maven...
usage: mvn [-h] [_ ...]
mvn: error: unrecognized arguments: -Dmaven.test.skip=true

Solution:

Install the correct version of maven. Run which mvn or mvn --version to verify you have correct installation of Maven.

2. AWS region issues

Region must be specified either via environment variable (AWS_REGION) or  system property (aws.region)., software.amazon.awssdk.regions.providers.AwsProfileRegionProvider@7a7eb56e: No region provided in profile: default, software.amazon.awssdk.regions.providers.InstanceProfileRegionProvider@241d5e60: Unable to contact EC2 metadata service.

Solution

Although you should not see such issues, but in case you do, export a variable export AWS_REGION=us-east-1 or any other region to resolve this.

3. Linkage error

025-04-02T13:51:25Z ERROR [Java-Iceberg:50051] Error: LinkageError occurred while loading main class io.debezium.server.iceberg.OlakeRpcServer
2025-04-02T13:51:25Z ERROR [Java-Iceberg:50051] 	java.lang.UnsupportedClassVersionError: io/debezium/server/iceberg/OlakeRpcServer has been compiled by a more recent version of the Java Runtime (class file version 61.0), this version of the Java Runtime only recognizes class file versions up to 55.0

Solution

Make sure your java version is >17. You can check your java version by running java -version in the terminal. If you have an older version, consider updating to a newer version of Java.

Reload your shell with source ~/.zshrc after updating the Java version.

Some common issues

Below are common issues encountered when using the Iceberg Writer and steps to resolve them:

Issue	Possible Cause	Resolution Steps
gRPC Connection Fails	The gRPC server is not running or using an incorrect port.	- Ensure the Java gRPC server is running. - Verify that the port in destination.json matches the server's port.
Data Not Appearing in Spark	Misconfiguration or delayed data propagation.	- Double-check the destination.json configuration. - Connect to the spark-iceberg container and run a Spark SQL query.
Authentication Errors (S3/Glue)	Incorrect AWS credentials or insufficient permissions.	- Confirm the AWS access key and secret key are correct. - Ensure the credentials allow access to S3 and Glue.
Docker Compose Issues	Docker or Docker Compose is not properly installed or services failed to start.	- Verify Docker and Docker Compose are installed and running. - Use docker compose ps to check service statuses.

For issues not listed here, consult the logs for the Java gRPC server or Docker containers for more detailed error messages.

VSCode Debug Configuration

Set up the following configuration in your VSCode debug console:

{
    "version": "0.2.0",
    "configurations": [
        {
            "type": "java",
            "name": "OlakeRpcServer",
            "request": "launch",
            "mainClass": "io.debezium.server.iceberg.OlakeRpcServer",
            "projectName": "iceberg-sink",
            "args": [
                "{\"catalog-impl\":\"org.apache.iceberg.jdbc.JdbcCatalog\",\"catalog-name\":\"olake_iceberg\",\"io-impl\":\"org.apache.iceberg.aws.s3.S3FileIO\",\"jdbc.password\":\"password\",\"jdbc.user\":\"iceberg\",\"port\":\"50051\",\"s3.access-key-id\":\"admin\",\"s3.endpoint\":\"http://localhost:9000\",\"s3.path-style-access\":\"true\",\"s3.secret-access-key\":\"password\",\"s3.ssl-enabled\":\"false\", \"s3.region\":\"us-east-1\",\"table-namespace\":\"olake_iceberg\",\"table-prefix\":\"\",\"upsert\":\"false\",\"upsert-keep-deletes\":\"true\",\"uri\":\"jdbc:postgresql://localhost:5432/iceberg\",\"warehouse\":\"s3a://warehouse\",\"write.format.default\":\"parquet\"}"
            ]
        }
    ]
}