Skip to main content

DB2 to Redshift: Incremental Sync via Iceberg

Moving IBM DB2 LUW (Linux, UNIX and Windows) data to Redshift usually means an AWS Database Migration Service (DMS) task, a licensed replication product, or a nightly EXPORT and COPY. This guide does it with OLake Go. It reads the DB2 tables with a read-only user, writes them once as Apache Iceberg tables into the AWS Glue Data Catalog, and keeps them fresh on a cursor column.

The files stay in your own S3 bucket. Redshift maps the Glue database onto an external schema and queries the tables in place, and Athena and DuckDB read the same tables through the same catalog.

By Shubham Satish BaldavaUpdated October 30, 2026

Key takeaways

  • OLake Go replicates DB2 LUW into Iceberg tables in the Glue Data Catalog: a full historical snapshot first, then incremental syncs on a cursor column such as UPDATED_AT, with no DMS or replication license in between.
  • Redshift reads the Glue database through one CREATE EXTERNAL SCHEMA statement, with no refresh. Every change a sync commits is visible on the next query, on Serverless and RA3 alike.
  • The same tables, written once by OLake Go, are also queryable from Athena and DuckDB. In the run on this page all three engines returned identical rows.

The short answer. To replicate DB2 to Redshift, run OLake Go with a read-only DB2 user as the source and the AWS Glue Data Catalog as the Iceberg destination, with Positional Deletes as the upsert type, the format Redshift reads. The first sync copies each table in full (the historical snapshot); every later sync fetches the rows whose cursor column moved and upserts them. Redshift, Athena and DuckDB then query the same Iceberg tables, with no second copy.

How DB2 data reaches Redshift here​

The first sync reads the row and page counts DB2 keeps for each table, splits the table into chunks, and reads the chunks in parallel. It writes Parquet data files and records the highest cursor value it saw. Every later sync selects rows above that value, writes them as new data files, and adds a delete file that retires the old version of each updated row.

Each sync ends with one commit to Glue. OLake Go uploads the data and delete files to the bucket, then updates the table's metadata pointer in the Glue database. Redshift, Athena and DuckDB see the new snapshot on their next query; none of them keeps a copy.

The cursor is the whole contract. A row that changes without its UPDATED_AT moving is never picked up, and a deleted row stays in the Iceberg table. Set the column in every write path, and soft-delete if deletes matter.

What you need​

  • DB2 LUW 11.5.3 or later, with a read-only user, statistics on each table and a cursor column. The tables on this page ran on DB2 12.1.5 Community Edition in Docker.
  • An amd64 host for the DB2 connector. IBM's driver has no Linux arm64 build. On an Apple silicon Mac, turn on Rosetta in Docker Desktop.
  • An AWS account with an S3 bucket for the files and an IAM user or role for OLake Go with Glue and S3 permissions. The run on this page used a bucket and a scoped IAM user in ap-southeast-1.
  • Amazon Redshift, Serverless or an RA3 cluster, with an IAM role attached that can read Glue and the bucket. The run on this page used a Serverless workgroup at 8 RPU.
  • OLake UI from the Docker Compose install. The run on this page used OLake Go v1.1.0.

Step 1: Prepare DB2 for incremental sync​

Most readers point OLake Go at tables they already have. To follow along with the exact data from this page, create these two tables first:

The two sample tables used on this page
Sample schema, in schema DB2INST1 of database SHOP
CREATE TABLE customers (
  id INT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
  name VARCHAR(100), email VARCHAR(200), city VARCHAR(100), tier VARCHAR(20),
  created_at TIMESTAMP DEFAULT CURRENT TIMESTAMP,
  updated_at TIMESTAMP DEFAULT CURRENT TIMESTAMP);

CREATE TABLE orders (
  id INT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
  customer_id INT REFERENCES customers(id),
  product VARCHAR(100), amount DECIMAL(10,2), status VARCHAR(20),
  created_at TIMESTAMP DEFAULT CURRENT TIMESTAMP,
  updated_at TIMESTAMP DEFAULT CURRENT TIMESTAMP);

-- Then insert whatever you like. This page started with 8 customers and 40 orders.

OLake Go reads DB2 LUW (Linux, UNIX and Windows) tables through IBM's Data Server driver: a parallel full snapshot first, then incremental syncs that fetch only rows whose cursor column moved past the last value seen. The DB2 connector has no log-based change data capture (CDC), so four things must be in place before the first sync.

SettingValueWhy it matters
A read-only userCONNECT on the database plus SELECT on each tableThe snapshot and every incremental sync read the tables directly. DB2 authenticates users against the operating system or LDAP, so the account exists there first and is then granted in the database.
Table statisticsRUNSTATS on every table you syncOLake Go reads the row count and page count from SYSCAT.TABLES to split the snapshot into chunks. A table that has never had statistics collected stops the sync with stats not populated. Stale statistics are fine.
A cursor columna column that grows with every change, such as UPDATED_ATIncremental syncs select WHERE cursor > last value. Rows updated without touching the cursor are not picked up.
A primary keyon every tableOLake Go uses it to upsert changed rows into the Iceberg table. Without one, changed rows are appended.
On the DB2 database, as the instance owner or a DBADM user
-- 1. A user for OLake Go. Create the operating system account first,
--    for example: useradd -M olake && passwd olake
GRANT CONNECT ON DATABASE TO USER olake;

-- 2. Read access to the tables you sync
GRANT SELECT ON TABLE db2inst1.customers TO USER olake;
GRANT SELECT ON TABLE db2inst1.orders TO USER olake;

-- 3. Statistics on every table you sync
CALL SYSPROC.ADMIN_CMD('RUNSTATS ON TABLE db2inst1.customers AND INDEXES ALL');
CALL SYSPROC.ADMIN_CMD('RUNSTATS ON TABLE db2inst1.orders AND INDEXES ALL');

-- 4. A cursor column that your application always sets, for example
--    updated_at TIMESTAMP DEFAULT CURRENT TIMESTAMP, updated on every change

Check the result before moving on. A CARD of -1 means statistics were never collected:

sql
SELECT tabname, card, npages FROM syscat.tables WHERE tabschema = 'DB2INST1';

Version, driver and SSL details are on the DB2 connector page.

The DB2 connector image is built for amd64 only

IBM's Data Server driver has no Linux arm64 build, so olakego/source-db2 runs on amd64 hosts. On an Apple silicon Mac, turn on Rosetta in Docker Desktop and pull the image with --platform linux/amd64.

Deletes do not replicate in incremental mode

A deleted row has no cursor value to read, so it stays in the Iceberg table. If deletes matter, soft-delete in DB2 (a deleted_at column that also bumps updated_at) or schedule an occasional full refresh.

Step 2: Set up the Glue catalog​

Redshift reads Iceberg tables through a catalog. AWS Glue Data Catalog is the one this page uses: OLake Go creates a Glue database per job and registers each table there. The files sit in your S3 bucket.

Redshift, Athena, Amazon Elastic MapReduce (EMR) and DuckDB all read from the same catalog. Redshift reaches it through an external schema, with the IAM role attached to the workgroup or cluster.

CatalogOLake Go writesAmazon Redshift readsVerified on this page
AWS GlueYesYes: external schema from the Glue Data CatalogYes, end to end

AWS Glue Data Catalog is the default catalog in OLake Go. The tables live as Parquet and metadata files in an S3 bucket you own, and Glue holds the pointer to the current metadata file. Athena, Redshift, EMR, Spark and DuckDB read the same tables through Glue, which is why it is the catalog to pick when more than one AWS engine will read the data.

You need:

  • An S3 bucket for the table files, in the region where your engines run.
  • An IAM user or role for OLake Go with Glue and S3 permissions. The exact policy is on the AWS Glue catalog page.
  • No Glue database in advance: OLake Go creates one per job, named after the source database and schema (for example pg_to_snowflake_shop_public).

Glue has no built-in compaction for tables written by external engines, so plan on OLake Fusion or Spark for maintenance once the change volume grows.

The run on this page gave OLake Go a scoped IAM user with this inline policy. Redshift got a role with AmazonS3ReadOnlyAccess and read access to Glue:

IAM policy for the OLake Go user (replace the bucket, region and account)
{
  "Version": "2012-10-17",
  "Statement": [
    { "Effect": "Allow", "Action": ["s3:ListBucket", "s3:GetBucketLocation"], "Resource": "arn:aws:s3:::<bucket>" },
    { "Effect": "Allow", "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:AbortMultipartUpload", "s3:ListMultipartUploadParts"], "Resource": "arn:aws:s3:::<bucket>/*" },
    { "Effect": "Allow",
      "Action": ["glue:GetDatabase", "glue:GetDatabases", "glue:CreateDatabase", "glue:GetTable", "glue:GetTables", "glue:CreateTable", "glue:UpdateTable", "glue:DeleteTable"],
      "Resource": ["arn:aws:glue:<region>:<account>:catalog", "arn:aws:glue:<region>:<account>:database/*", "arn:aws:glue:<region>:<account>:table/*/*"] }
  ]
}

Step 3: Connect DB2 to OLake Go​

  1. In the OLake UI, open Sources and click Create Source. Pick DB2 LUW as the connector, give the source a name and keep the OLake Go version on the latest release.
  2. Fill in the host, the port (50000 by default), the database name, and the username and password from step 1. The Enable CDC switch stays off: DB2 runs in incremental mode only.
  3. Click Set up source. OLake Go tests the connection and saves the source.

Here is the source form from the run on this page. DB2 ran in another container, so the host is host.docker.internal, with database shop on port 50000. The change data capture (CDC) switch is greyed out with Not supported, which is expected for DB2.

OLake UI Edit source form for DB2 LUW on OLake Go v1.1.0 with CDC marked not supported, host host.docker.internal, port 50000, database shop and user db2inst1
The DB2 source in OLake UI: database shop on port 50000, with CDC not supported for this connector.

The full field reference, including SSL and SSH tunnels, is on the DB2 connector page.

Step 4: Configure the Iceberg destination​

  1. Open Destinations and click Create Destination.
  2. Pick Apache Iceberg as the connector and AWS Glue as the catalog type.
  3. Enter the S3 path, the AWS region and the access key pair (or leave the keys empty to use the instance role).
  4. Click Create.
ParameterSample ValueDescription
S3 Path
required		s3://<BUCKET_NAME>/S3 bucket path where Iceberg table data and metadata files will be stored.
AWS Region
required		ap-south-1AWS Region where your S3 bucket and Glue Data Catalog exist. Required so OLake Go calls the correct regional S3 and Glue endpoints.
AWS Access KeyXXXAWS access key ID for authentication. Optional if using IAM roles or instance profiles.

If using IAM:
Docker Compose: add the required IAM environment variables under x-envs in your compose file. See Service Environment Variables for setup details.
Kubernetes: set up pod IAM in Cloud Integration.
AWS Secret KeyXXXAWS secret access key for authentication. Optional if using IAM roles or instance profiles.

If using IAM:
Docker Compose: add the required IAM environment variables under x-envs in your compose file. See Service Environment Variables for setup details.
Kubernetes: set up pod IAM in Cloud Integration.
S3 Endpointhttps://glue.ap-south-1.amazonaws.comS3 API endpoint for writing Iceberg data files. Optional for AWS S3 but required for S3-compatible storage like MinIO.
Custom Glue Endpoint Configurationtrue/falseEnable custom Glue catalog configuration when Glue catalog settings differ from S3 (different credentials, region, or endpoint). Shows additional Glue configuration fields. Leave off when Glue and S3 use the same AWS setup.
Enable Arrow Writestrue/falseWrites data and delete files using Apache Arrow based writer and registers them in Iceberg.

The following fields appear in the UI after Custom Glue Endpoint Configuration is turned on. Configure Glue catalog connection settings here separately from the S3 fields above.

ParameterSample ValueDescription
Glue Catalog ID123456789012AWS account ID used as the Glue Data Catalog identifier.
Glue Access KeyXXXAccess key for authenticating Glue catalog requests. Required when Glue credentials differ from S3 credentials.
Glue Secret KeyXXXSecret key for authenticating Glue catalog requests. Required when Glue credentials differ from S3 credentials.
Glue Endpointhttps://glue.ap-south-1.amazonaws.comCustom endpoint URL for AWS Glue or a Glue-compatible catalog service.
Glue Regionap-south-1Region for the Glue catalog, if different from the S3 region. Falls back to AWS Region if not set.

Here is the destination form from the run on this page. It holds the bucket path OLake Go writes under, the region of the bucket and the Glue catalog, and the access key pair of the IAM user from step 2.

OLake UI Edit destination form for Apache Iceberg with catalog type AWS Glue, an S3 path under the lab bucket, region ap-southeast-1 and an access key pair
The Glue destination in OLake UI: the S3 path, the region and the keys of the scoped IAM user.

Step 5: Create the job and run the first sync​

Redshift is not in OLake Go's target engine picker yet. This job therefore lists Athena and DuckDB as target engines, with Positional Deletes as the upsert type.

That is the format Redshift reads, and both listed engines read it too. Because the DB2 connector has no CDC mode, each stream runs as Full Refresh + Incremental with UPDATED_AT as the cursor.

  1. Open Jobs and click Create Job. Give it a name and a schedule. Every Day at a fixed hour, or a cron expression, both work; the schedule is how often the change stream is drained, not how often a full copy runs.
  2. Select the source and the destination you created.
  3. Under Advanced Settings, set Target Query Engines to AWS Athena and DuckDB (Redshift is not in the list yet; it reads what Athena reads). OLake Go then resolves an Iceberg delete format every one of those engines can read, so you never pick one by hand.
  4. Click Next. OLake Go discovers the tables and opens the Streams page.
  5. Tick the tables to replicate. For each one, set the sync mode to Full Refresh + Incremental with UPDATED_AT as the cursor column and the ingestion mode to Upsert. Leave the Upsert Type as OLake Go resolved it (Positional Deletes in the run on this page).
  6. Click Create Job, then open the job's menu and choose Sync Now for the first run.

DB2 reports unquoted names in upper case, so the streams and columns appear that way.

OLake UI Streams Selection page with the DB2INST1 schema, CUSTOMERS and ORDERS selected, sync mode Full Refresh + Incremental, CDC options disabled, cursor field UPDATED_AT, ingestion mode Upsert and upsert type Positional Deletes on Iceberg format version V2
Each stream: Full Refresh + Incremental on UPDATED_AT, Upsert, and Positional Deletes as the upsert type.
OLake UI Job Logs and History for db2_to_redshift showing two completed syncs of 16 seconds each
Job history: the 16 s historical snapshot, then the 16 s incremental sync.

On this page the first run copied 8 customers and 40 orders (48 records, 2.38 KB) and finished in 16 seconds, container start included. OLake Go names the Glue database after the job, the database and the DB2 schema, lower-cased: db2_to_redshift on SHOP.DB2INST1 produced db2_to_redshift_shop_db2inst1. The Iceberg DB field at the top of the Streams page changes it before the first sync.

Step 6: Query the tables in Redshift​

Redshift reads the Glue Data Catalog through an external schema: one statement maps a Glue database onto a schema name in your Redshift database, and every Iceberg table OLake Go commits there is queryable at once, with no refresh. Redshift Serverless and RA3 clusters both do this; the IAM role attached to the namespace or cluster needs read access to Glue and to the bucket.

Redshift query editor v2
-- Once: map the Glue database OLake Go created (schema = <job name>_<source database>_<source schema>)
CREATE EXTERNAL SCHEMA lakehouse
FROM DATA CATALOG
DATABASE 'db2_to_redshift_shop_db2inst1'
IAM_ROLE default;

-- Then query the Iceberg tables like any other schema
SELECT id, name, city, tier FROM lakehouse.customers ORDER BY id;
SELECT status, count(*) AS n, round(sum(amount), 2) AS total
FROM lakehouse.orders GROUP BY status ORDER BY status;

IAM_ROLE default uses the default role of the namespace (Serverless) or cluster; name a role ARN instead when you keep several. Redshift reads Iceberg v2 tables with positional deletes, which is why the job on this page uses that upsert type. The external schema is read-only: OLake Go is the writer, and INSERT or DELETE on these tables fails.

OLake Go adds four columns to every table: _olake_id (a hash of the primary key), _op_type, _olake_timestamp and _cdc_timestamp. _op_type is r for a row from the first snapshot, c for an insert, u for an update and d for a delete. Filter them out of dashboards, or use them to see when a row last changed.

Redshift query editor v2 in the AWS console running the customers query on the Serverless workgroup and returning 10 rows with id, name, city and tier
Redshift Serverless reading the customers table through the external schema after the incremental sync in query editor v2: 10 rows, one per key.

Step 7: Change rows in DB2 and sync again​

To see the incremental path work, the run on this page made 12 changes in DB2 after the first sync. Every update also sets updated_at, which is what the cursor sees:

Changes made in DB2 between the two syncs
INSERT INTO customers (name, email, city, tier) VALUES ('Maya Patel', 'maya@example.com', 'Pune', 'silver');
INSERT INTO customers (name, email, city, tier) VALUES ('Lucas Silva', 'lucas@example.com', 'São Paulo', 'bronze');
INSERT INTO orders (customer_id, product, amount, status) VALUES (9, 'Laptop', 1299.00, 'placed');
INSERT INTO orders (customer_id, product, amount, status) VALUES (9, 'Mouse', 25.50, 'placed');
INSERT INTO orders (customer_id, product, amount, status) VALUES (10, 'Monitor', 349.99, 'placed');
INSERT INTO orders (customer_id, product, amount, status) VALUES (3, 'Keyboard', 89.00, 'placed');
INSERT INTO orders (customer_id, product, amount, status) VALUES (7, 'Webcam', 79.90, 'placed');
UPDATE customers SET tier = 'gold',      updated_at = CURRENT TIMESTAMP WHERE name = 'Liam Carter';
UPDATE customers SET city = 'Abu Dhabi', updated_at = CURRENT TIMESTAMP WHERE name = 'Zara Ali';
UPDATE orders    SET status = 'delivered', updated_at = CURRENT TIMESTAMP WHERE id IN (1, 4, 7);

The second sync fetched 12 rows (635 bytes) above the saved cursor and finished in 16 seconds end to end. For customers that meant one new data file with 4 rows and one positional delete file retiring the 2 old versions. The commit report in the sync log shows addedDataFiles=1, addedRecords=4, addedPositionalDeleteFiles=1, addedPositionalDeletes=2, and orders the same shape with 8 rows and 3 deletes.

OLake UI sync log filtered to the Iceberg commit reports for customers and orders, showing addedDataFiles 1, addedPositionalDeleteFiles 1, addedRecords 4 and 8, and addedPositionalDeletes 2 and 3
The incremental sync in the OLake UI log: one data file and one positional delete file per table.

Redshift shows the result on the next query, with no refresh and no ALTER TABLE. Liam Carter is gold, Zara Ali moved to Abu Dhabi, and the two new customers are there. In incremental mode OLake Go cannot tell an insert from an update, so the 4 rows the second sync brought carry _op_type = u and the snapshot rows keep r.

Schema changes. When a column is added in DB2, OLake Go adds it to the Iceberg table and its Glue definition on the next sync, and Redshift shows it on the next query. DB2 itself may put the table in REORG PENDING after an ALTER TABLE; see troubleshooting below. Full rules are on the schema evolution page.

Query the same tables from Athena and DuckDB​

Nothing is exported, copied or refreshed: both engines read the files OLake Go wrote, through the same Glue database.

Athena reads the Glue Data Catalog natively: the Glue database OLake Go created shows up in the query editor's database list, and the Iceberg tables in it need no registration and no refresh. Pick a workgroup with a query result location in S3, select the database, and query.

Athena query editor (database: the one OLake Go created)
SELECT id, name, city, tier FROM db2_to_redshift_shop_db2inst1.customers ORDER BY id;
SELECT status, count(*) AS n, round(sum(amount), 2) AS total
FROM db2_to_redshift_shop_db2inst1.orders GROUP BY status ORDER BY status;

Athena engine v3 reads Iceberg v2 tables with positional and equality deletes, and cannot read v3 tables at all. Keep the job on positional or equality deletes, never deletion vectors, while Athena is a reader. From the command line, aws athena start-query-execution --work-group <workgroup> --query-execution-context Database=${props.schema ?? 'postgres_to_redshift_shop_public'} --query-string "..." runs the same queries; aws athena get-query-results returns the rows.

Athena query editor in the AWS console showing the customers query on db2_to_redshift_shop_db2inst1.customers completed with 10 rows
Athena reading the same table through Glue: the same 10 rows.

An orders roll-up matched DB2 to the cent in all three engines. It gave 16 delivered orders for 8659.68, 18 placed for 10522.26, 11 shipped for 6397.25.

Data types: DB2 to Iceberg to Redshift​

OLake Go maps DB2 types to Iceberg types, and Redshift maps Iceberg types to its own through the external schema. Decimals and times are the ones to watch.

DB2Iceberg (written by OLake Go)Redshift showsNotes
SMALLINT, INTEGERintINTEGER
BIGINTlongBIGINT
DECIMAL, NUMERIC, DECFLOAT, FLOAT, DOUBLEdoubleDOUBLE PRECISIONThe orders.amount sums on this page matched DB2 to the cent; round at query time when you need exact decimal arithmetic.
REALfloatREAL
BOOLEANbooleanBOOLEAN
DATE, TIMESTAMPtimestamptz (UTC)TIMESTAMPOLake Go writes every timestamp in UTC; Redshift shows it without a zone.
TIMEstringVARCHAR
CHAR, VARCHAR, CLOB, GRAPHIC, VARGRAPHIC, DBCLOB, XMLstringVARCHAR
BLOBstringVARCHARBinary values arrive as text; keep large binaries out of the sync.

DB2 types in the table come from the DB2 connector reference. The Redshift side comes from the Redshift Iceberg page (checked 2026-10-30) and from svv_external_columns on the tables of this run.

Cost and maintenance​

  • Storage is your bucket: you pay S3 rates for the Parquet and delete files, and nothing to Glue beyond its per-object pricing for the table entries.
  • Compute. OLake Go runs wherever you run Docker, on an amd64 host for the DB2 connector. Redshift Serverless bills RPU-hours only while queries run; a query over the external schema scans the Parquet files in S3 through Redshift Spectrum. Athena bills per terabyte scanned.
  • DB2 load. Each incremental sync is one range scan per table on the cursor column; index UPDATED_AT so it stays cheap as the table grows. Keep RUNSTATS in your regular maintenance window so snapshot chunks stay even.
  • Maintenance. Glue has no built-in compaction for tables written by external engines. A table that gets a new small file and delete file per sync wants compaction; OLake Fusion does this for the catalogs OLake Go writes to, and Spark on EMR's rewrite_data_files works through the same Glue catalog.
  • Key rotation. The IAM user's access key is what OLake Go writes with. Rotate it on your schedule, or run OLake Go on an instance with a role and leave the key fields empty.

Troubleshooting​

1. The source test fails on an Apple silicon Mac​

image pull registry-1.docker.io/olakego/source-db2:v1.1.0: Error response from daemon: no matching manifest for linux/arm64/v8 in the manifest list entries

The OLake UI worker pulls connector images for the host's platform, and the DB2 image exists for amd64 only. Turn on Use Rosetta for x86_64/amd64 emulation in Docker Desktop, then pull the image yourself and tag it with the name the worker asks for:

bash
docker pull --platform linux/amd64 olakego/source-db2:v1.1.0
docker tag olakego/source-db2:v1.1.0 registry-1.docker.io/olakego/source-db2:v1.1.0

On an amd64 Linux host, such as an EC2 instance, the error does not occur.

2. The first sync stops with "stats not populated"​

stats not populated for table[DB2INST1.CUSTOMERS]. Please run CLP command: RUNSTATS ON TABLE DB2INST1.CUSTOMERS AND INDEXES ALL

DB2 has never collected statistics for the table, and OLake Go needs them to size the snapshot chunks. Run the RUNSTATS call from step 1 for every table in the job, then sync again.

3. A sync fails with SQL0668N after an ALTER TABLE​

Adding, dropping or retyping a column can leave a DB2 table in REORG PENDING, and reads fail until it is reorganized. Run CALL SYSPROC.ADMIN_CMD('REORG TABLE db2inst1.customers'), then sync again. The DB2 connector page lists the operations that cause it.

4. The CLI picked the wrong cursor column​

discover fills cursor_field in selected_streams.json with the first column in alphabetical order, for example CITY for CUSTOMERS. Set it to UPDATED_AT for every stream before the first sync; the OLake UI asks for it on the Streams page instead.

5. The destination test fails on a table OLake Go did not create​

User: arn:aws:iam::<account>:user/<olake user> is not authorized to perform: s3:GetObject on resource: "arn:aws:s3:::<another bucket>/.../test_olake.db/test_olake/metadata/00000-....metadata.json"

OLake Go's destination test opens a probe table called test_olake.test_olake in the Glue database test_olake. An earlier run in the same account and region may have left that table behind, pointing at a bucket your user cannot read. Delete the stale table from Glue, or run OLake Go against a region whose Glue catalog does not have it.

6. Redshift cannot see the external schema's tables​

ERROR: Database 'db2_to_redshift_shop_db2inst1' does not exist in the Glue Data Catalog

The external schema was created before the first sync, or in a different region from the Glue database. OLake Go creates the Glue database on the first sync; create the external schema after it, in the workgroup's region.

How this compares with other ways to move DB2 data to Redshift
MethodFreshnessUpdates and deletesWhere the data landsWhat you run
AWS DMS with a Db2 LUW sourceCDC from the DB2 logYesNative Redshift tablesA DMS replication instance and task, a recoverable (archive-logging) database and a DBADM user
Debezium's Db2 connector, Kafka and the Redshift sinkSeconds, from SQL Replication capture tablesYesKafka topics, then Redshift tablesKafka, Kafka Connect, Debezium, tables in capture mode and, per Debezium's docs, an IBM InfoSphere Data Replication license (checked Oct 2026)
Export (EXPORT TO ... OF DEL), then COPY into RedshiftHours to dailyNo; full reloads onlyNative Redshift tablesA script, a scheduler and a staging bucket
OLake Go to Iceberg on Glue (this guide)Incremental on a schedule down to one minuteInserts and updates that move the cursor; deletes need a soft-delete columnApache Iceberg tables that Redshift, Athena, EMR, Spark and DuckDB all readOne Docker Compose stack, open source; your bucket plus your own compute

DMS is the simpler choice when you need deletes replicated and native Redshift tables are the only goal. Writing Iceberg once avoids paying to move and store a second copy as soon as Athena, EMR or a notebook needs the same rows.

When not to use this​

  • You need hard deletes replicated. Incremental sync cannot see a deleted row. Soft-delete in DB2, or use a log-based tool.
  • Your DB2 runs on z/OS or IBM i. The connector covers DB2 LUW only.
  • Every query must hit native Redshift speed. Spectrum reads Parquet from S3 on every query. Keep the Iceberg tables as the source and load the hot slice into a native table on a schedule.

FAQs​

How do I replicate IBM DB2 to Redshift without DMS?

Run OLake Go (open source, Apache 2.0) against DB2 LUW with a read-only user, and point it at the AWS Glue Data Catalog with an S3 bucket. The first sync copies each table in parallel chunks; every later sync fetches only rows whose cursor column (for example UPDATED_AT) moved, and upserts them. Redshift reads the result through an external schema over the Glue database. No DMS task, replication license or Redshift load job is needed.

Does OLake Go do change data capture (CDC) from DB2?

No. The DB2 connector runs Full Refresh and Full Refresh + Incremental; it does not read the DB2 log or the SQL Replication capture tables. Incremental syncs select rows where a cursor column is greater than the last value seen, which captures inserts and updates that bump the cursor but not deletes.

Does Redshift read Iceberg tables with row-level deletes?

Yes. Redshift reads Iceberg v2 tables with positional deletes through Redshift Spectrum, on Serverless and RA3. The run on this page used positional deletes and Redshift returned one row per key after the updates.

Is Redshift one of OLake Go's target query engines?

Not in the picker yet. On this page the job lists Athena and DuckDB as target engines and the upsert type is Positional Deletes, which is the format Redshift reads. Redshift then reads the tables through the external schema like any other Iceberg v2 table.

Why does the first sync ask me to run RUNSTATS?

OLake Go splits each table into chunks for the parallel snapshot using the row and page counts DB2 keeps in SYSCAT.TABLES. Those counts exist only after RUNSTATS has run once on the table. Run CALL SYSPROC.ADMIN_CMD('RUNSTATS ON TABLE schema.table AND INDEXES ALL') per table; statistics that are out of date do not stop a sync.

Which DB2 data types change on the way to Redshift?

DECIMAL, NUMERIC, DECFLOAT, FLOAT and DOUBLE become Iceberg double and show up in Redshift as DOUBLE PRECISION, so round at query time when you need exact decimal arithmetic. INTEGER stays INTEGER and BIGINT stays BIGINT. DATE and TIMESTAMP become TIMESTAMP in UTC. TIME, XML, CLOB, BLOB and GRAPHIC types arrive as VARCHAR.

DB2 to SnowflakeSame source, read from Snowflake.DB2 to ClickHouseSame source, read from ClickHouse.DB2 to BigQuerySame source, read from BigQuery.Postgres to Amazon RedshiftSame engine, replicated from Postgres.Oracle to Amazon RedshiftSame engine, replicated from Oracle.Kafka to Amazon RedshiftSame engine, replicated from Kafka.MySQL to Amazon RedshiftSame engine, replicated from MySQL.MongoDB to Amazon RedshiftSame engine, replicated from MongoDB.MariaDB to Amazon RedshiftSame engine, replicated from MariaDB.S3 to Amazon RedshiftSame engine, replicated from S3.SQL Server to Amazon RedshiftSame engine, replicated from SQL Server.DB2 connector referenceEvery field, sync mode and troubleshooting note for the source.Query engine compatibilityWhich engines read which catalogs and delete formats.All integrationsEvery source and engine pair with a tested guide.

Deeper reading on this site: