Troubleshooting Iceberg Writer

1. maven related issues

Iceberg JAR file not found. Building with Maven...
usage: mvn [-h] [_ ...]
mvn: error: unrecognized arguments: -Dmaven.test.skip=true

Solution:

Install the correct version of maven. Run which mvn or mvn --version to verify you have correct installation of Maven.

2. AWS region issues

Region must be specified either via environment variable (AWS_REGION) or  system property (aws.region)., software.amazon.awssdk.regions.providers.AwsProfileRegionProvider@7a7eb56e: No region provided in profile: default, software.amazon.awssdk.regions.providers.InstanceProfileRegionProvider@241d5e60: Unable to contact EC2 metadata service.

Solution

Although you should not see such issues, but in case you do, export a variable export AWS_REGION=us-east-1 or any other region to resolve this.

Some commond issues

Below are common issues encountered when using the Iceberg Writer and steps to resolve them:

Issue	Possible Cause	Resolution Steps
gRPC Connection Fails	The gRPC server is not running or using an incorrect port.	- Ensure the Java gRPC server is running. - Verify that the port in writer.json matches the server's port.
Data Not Appearing in Spark	Misconfiguration or delayed data propagation.	- Double-check the writer.json configuration. - Connect to the spark-iceberg container and run a Spark SQL query.
Authentication Errors (S3/Glue)	Incorrect AWS credentials or insufficient permissions.	- Confirm the AWS access key and secret key are correct. - Ensure the credentials allow access to S3 and Glue.
Docker Compose Issues	Docker or Docker Compose is not properly installed or services failed to start.	- Verify Docker and Docker Compose are installed and running. - Use docker compose ps to check service statuses.

For issues not listed here, consult the logs for the Java gRPC server or Docker containers for more detailed error messages.

VSCode Debug Configuration

Set up the following configuration in your VSCode debug console:

{
    "version": "0.2.0",
    "configurations": [
        {
            "type": "java",
            "name": "OlakeRpcServer",
            "request": "launch",
            "mainClass": "io.debezium.server.iceberg.OlakeRpcServer",
            "projectName": "iceberg-sink",
            "args": [
                "{\"catalog-impl\":\"org.apache.iceberg.jdbc.JdbcCatalog\",\"catalog-name\":\"olake_iceberg\",\"io-impl\":\"org.apache.iceberg.aws.s3.S3FileIO\",\"jdbc.password\":\"password\",\"jdbc.user\":\"iceberg\",\"port\":\"50051\",\"s3.access-key-id\":\"admin\",\"s3.endpoint\":\"http://localhost:9000\",\"s3.path-style-access\":\"true\",\"s3.secret-access-key\":\"password\",\"s3.ssl-enabled\":\"false\", \"s3.region\":\"us-east-1\",\"table-namespace\":\"olake_iceberg\",\"table-prefix\":\"\",\"upsert\":\"false\",\"upsert-keep-deletes\":\"true\",\"uri\":\"jdbc:postgresql://localhost:5432/iceberg\",\"warehouse\":\"s3a://warehouse\",\"write.format.default\":\"parquet\"}"
            ]
        }
    ]
}