Skip to main content

S3 to Redshift: CSV Files as Apache Iceberg

Files land in S3 every hour or every day: exports from an application, drops from a partner, logs from a job. S3 to Redshift usually means a COPY script on a schedule, a Spectrum external table per folder, or a managed loader.

This guide takes a different route. OLake Go reads the files itself and turns each folder into an Apache Iceberg table in the AWS Glue Data Catalog. Every later run syncs only the files that are new.

Redshift maps the Glue database onto an external schema and queries the tables in place. Nothing else is in the path: no COPY, no manifest, no table definition per folder. Athena and DuckDB read the same tables through the same catalog.

By Shubham Satish BaldavaUpdated October 30, 2026

Key takeaways

  • OLake Go turns S3 folders into Iceberg tables in the Glue Data Catalog: the first sync reads every file under the prefix, and every later sync reads only files newer than the last run, by their modification time.
  • Redshift reads the Glue database through one CREATE EXTERNAL SCHEMA statement, with no refresh. Every file a sync commits is visible on the next query, on Serverless and RA3 alike.
  • The same tables, written once by OLake Go, are also queryable from Athena and DuckDB. In the run on this page all three engines returned identical rows.

The short answer. To load S3 files into Redshift as Iceberg, run OLake Go with the bucket and prefix as the source and the AWS Glue Data Catalog as the destination. Put each stream in Full Refresh + Incremental mode with _last_modified_time as the cursor.

The first sync reads every file; every later sync reads only the files added since. Redshift, Athena and DuckDB then query the same Iceberg tables, with no second copy.

How S3 files reach Redshift here​

Two things happen in order, and OLake Go does both on its own. The first sync lists everything under the prefix and makes one stream per first-level folder.

Then it reads every file in parallel: that is the historical load. At the end it saves the newest LastModified time it saw per stream.

Every later sync lists the prefix again and reads only the files whose LastModified time is newer than the saved cursor. On this page the second sync reported Filtered 1 files to process out of 2 total for each stream.

Rows are appended; a file never changes a row that an earlier file wrote. Each sync ends with one commit to Glue, and Redshift, Athena and DuckDB see the new snapshot on their next query.

One thing to keep an eye on: the cursor is the object's modification time, not anything inside the file. Uploading a corrected copy of yesterday's file gives it today's LastModified time.

The whole file is then read again, and every row in it is appended a second time. Add files, never rewrite them, and keep an id column in the data so a query can pick the latest version of a row.

What you need​

  • A source bucket, on S3 or an S3-compatible store, with files in folders under one prefix: CSV, JSON, Parquet or XML, plain or gzip. Amazon S3, MinIO and LocalStack all work; the run on this page used MinIO in Docker with two CSV folders.
  • Read-only access to the source bucket: s3:ListBucket and s3:GetObject, through a role on AWS or static keys on MinIO.
  • An AWS account with an S3 bucket for the Iceberg files and an IAM user or role for OLake Go with Glue and S3 permissions. The run on this page used a bucket and a scoped IAM user in ap-southeast-1. On AWS the source files and the Iceberg files can share one bucket under different prefixes.
  • Amazon Redshift, Serverless or an RA3 cluster, with an IAM role attached that can read Glue and the lakehouse bucket. The run on this page used a Serverless workgroup at 8 RPU.
  • OLake UI from the Docker Compose install with the OLake Go S3 source. The run on this page used v1.1.0.
  • Docker, and about 30 minutes.

Step 1: Prepare the source bucket​

Most readers point OLake Go at files they already have. If you want to follow along with the exact data from this page, upload two CSV files shaped like these first:

The two sample files used on this page

Each stream is a folder under data/, with one dated subfolder per upload. The header row names the columns, and id identifies the row so a later file can correct it.

data/customers/2026-10-30/customers.csv
id,name,email,city,tier,created_at,updated_at
1,Aarav Mehta,aarav@example.com,Mumbai,gold,2026-10-30T03:34:34Z,2026-10-30T03:34:34Z
2,Priya Sharma,priya@example.com,Bengaluru,silver,2026-10-30T03:34:34Z,2026-10-30T03:34:34Z
data/orders/2026-10-30/orders.csv
id,customer_id,product,amount,status,created_at,updated_at
1,2,Headphones,99.19,shipped,2026-10-30T03:34:34Z,2026-10-30T03:34:34Z
2,3,Monitor,178.38,delivered,2026-10-30T03:34:34Z,2026-10-30T03:34:34Z
Upload with the AWS CLI (add --endpoint-url for MinIO)
aws s3 sync ./stage s3://shop/data/
# Then add whatever you like. This page started with 8 customers and 40 orders.

There is nothing to switch on in S3. OLake Go lists the objects under a prefix, groups them into streams by their first folder, and remembers the newest LastModified time it has read. Four things decide whether that works for your bucket.

RequirementValueWhy it matters
Folder layout<prefix>/<stream>/.../fileThe first folder after the prefix is the stream name and becomes the table. Files sitting directly under the prefix belong to no stream.
File formatCSV, JSON (lines or array), Parquet or XML, plain or gzipOne format per source. Mixed buckets need one source per format, or a file pattern that picks one.
Read-only accesss3:ListBucket on the bucket, s3:GetObject on its objectsListing finds the files and tracks their LastModified time; GetObject reads them.
Files that are added, not rewrittenNew file per batchIncremental syncs read files newer than the saved cursor. A rewritten file gets a new LastModified time and is read again in full.

This is the layout the run on this page used, with one CSV file per day per stream:

Bucket layout
s3://shop/data/
├── customers/
│   └── 2026-10-29/customers.csv
└── orders/
  └── 2026-10-29/orders.csv

On AWS, attach a policy with the two read actions to the user or role OLake Go runs as. On MinIO or another S3-compatible store, a user with read access to the bucket is enough, and the endpoint goes into the source form.

IAM policy for the OLake Go user or role
{
"Version": "2012-10-17",
"Statement": [
  { "Effect": "Allow", "Action": ["s3:ListBucket"], "Resource": ["arn:aws:s3:::shop"] },
  { "Effect": "Allow", "Action": ["s3:GetObject"], "Resource": ["arn:aws:s3:::shop/*"] }
]
}

Leave the access key fields empty on AWS to use the instance role, task role or environment credentials instead of static keys. Static keys are required for MinIO and LocalStack.

LastModified is the cursor

OLake Go reads a file when its LastModified time is newer than the cursor saved after the last sync. Uploading a corrected copy of an old file therefore re-reads the whole file and appends every row again. Add new files for new data; keep old files as they are.

Step 2: Set up the Glue catalog​

Redshift reads Iceberg tables through a catalog. AWS Glue Data Catalog is the one this page uses: OLake Go creates a Glue database per job and registers each table there. The Iceberg files sit in your lakehouse bucket.

Redshift, Athena, Amazon Elastic MapReduce (EMR) and DuckDB all read from the same catalog. Redshift reaches it through an external schema, with the IAM role attached to the workgroup or cluster.

CatalogOLake Go writesAmazon Redshift readsVerified on this page
AWS GlueYesYes: external schema from the Glue Data CatalogYes, end to end

AWS Glue Data Catalog is the default catalog in OLake Go. The tables live as Parquet and metadata files in an S3 bucket you own, and Glue holds the pointer to the current metadata file. Athena, Redshift, EMR, Spark and DuckDB read the same tables through Glue, which is why it is the catalog to pick when more than one AWS engine will read the data.

You need:

  • An S3 bucket for the table files, in the region where your engines run.
  • An IAM user or role for OLake Go with Glue and S3 permissions. The exact policy is on the AWS Glue catalog page.
  • No Glue database in advance: OLake Go creates one per job, named after the source database and schema (for example pg_to_snowflake_shop_public).

Glue has no built-in compaction for tables written by external engines, so plan on OLake Fusion or Spark for maintenance once the change volume grows.

The run on this page gave OLake Go a scoped IAM user with this inline policy on the lakehouse bucket. Redshift got a role with AmazonS3ReadOnlyAccess and read access to Glue:

IAM policy for the OLake Go user (replace the bucket, region and account)
{
  "Version": "2012-10-17",
  "Statement": [
    { "Effect": "Allow", "Action": ["s3:ListBucket", "s3:GetBucketLocation"], "Resource": "arn:aws:s3:::<bucket>" },
    { "Effect": "Allow", "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:AbortMultipartUpload", "s3:ListMultipartUploadParts"], "Resource": "arn:aws:s3:::<bucket>/*" },
    { "Effect": "Allow",
      "Action": ["glue:GetDatabase", "glue:GetDatabases", "glue:CreateDatabase", "glue:GetTable", "glue:GetTables", "glue:CreateTable", "glue:UpdateTable", "glue:DeleteTable"],
      "Resource": ["arn:aws:glue:<region>:<account>:catalog", "arn:aws:glue:<region>:<account>:database/*", "arn:aws:glue:<region>:<account>:table/*/*"] }
  ]
}

Step 3: Connect S3 to OLake Go​

  1. In the OLake UI, open Sources and click Create Source. Pick S3 as the connector, give the source a name and keep the OLake Go version on the latest release.
  2. The Enable CDC toggle under Change data capture (CDC) reads not supported for S3 and stays off. Object storage has no change log; the job's Full Refresh + Incremental mode on the file modification time is what carries new data.
  3. Under Connection details, fill in the S3 Bucket Name, the AWS Region, and the Path Prefix that holds the stream folders. On AWS, leave Access Key ID and Secret Access Key empty to use the role OLake Go runs with; on MinIO or LocalStack fill them in and set Custom Endpoint to the service URL.
  4. Pick the File Format. For CSV, check the delimiter, the header toggle and the quote character; for XML, set the row identifier. File Pattern is an optional regular expression on the object key, useful when a prefix mixes formats.
  5. Click Set up source. OLake Go calls HeadBucket with the credentials; a 403 or 404 here means the key, the bucket name or the endpoint is wrong.

Here is the source form from the run on this page. MinIO ran in another container, so the custom endpoint is http://host.docker.internal:9000. The bucket is shop, the prefix data/, and the format CSV with a header row.

OLake UI Edit source form for S3 on OLake Go v1.1.0 with the Enable CDC toggle marked not supported, bucket shop, region us-east-1, static keys, path prefix data/, custom endpoint http://host.docker.internal:9000, file format CSV with delimiter comma, header on and quote character double quote
The S3 source in OLake UI: bucket, prefix, the MinIO endpoint, and the CSV options. The keys shown are MinIO defaults for a local test.

Set up source calls HeadBucket before anything is saved, so a wrong key or bucket name fails here rather than on the first sync.

The full field reference, including JSON, Parquet and XML options, is on the S3 connector page.

Step 4: Configure the Iceberg destination​

  1. Open Destinations and click Create Destination.
  2. Pick Apache Iceberg as the connector and AWS Glue as the catalog type.
  3. Enter the S3 path, the AWS region and the access key pair (or leave the keys empty to use the instance role).
  4. Click Create.
ParameterSample ValueDescription
S3 Path
required		s3://<BUCKET_NAME>/S3 bucket path where Iceberg table data and metadata files will be stored.
AWS Region
required		ap-south-1AWS Region where your S3 bucket and Glue Data Catalog exist. Required so OLake Go calls the correct regional S3 and Glue endpoints.
AWS Access KeyXXXAWS access key ID for authentication. Optional if using IAM roles or instance profiles.

If using IAM:
Docker Compose: add the required IAM environment variables under x-envs in your compose file. See Service Environment Variables for setup details.
Kubernetes: set up pod IAM in Cloud Integration.
AWS Secret KeyXXXAWS secret access key for authentication. Optional if using IAM roles or instance profiles.

If using IAM:
Docker Compose: add the required IAM environment variables under x-envs in your compose file. See Service Environment Variables for setup details.
Kubernetes: set up pod IAM in Cloud Integration.
S3 Endpointhttps://glue.ap-south-1.amazonaws.comS3 API endpoint for writing Iceberg data files. Optional for AWS S3 but required for S3-compatible storage like MinIO.
Custom Glue Endpoint Configurationtrue/falseEnable custom Glue catalog configuration when Glue catalog settings differ from S3 (different credentials, region, or endpoint). Shows additional Glue configuration fields. Leave off when Glue and S3 use the same AWS setup.
Enable Arrow Writestrue/falseWrites data and delete files using Apache Arrow based writer and registers them in Iceberg.

The following fields appear in the UI after Custom Glue Endpoint Configuration is turned on. Configure Glue catalog connection settings here separately from the S3 fields above.

ParameterSample ValueDescription
Glue Catalog ID123456789012AWS account ID used as the Glue Data Catalog identifier.
Glue Access KeyXXXAccess key for authenticating Glue catalog requests. Required when Glue credentials differ from S3 credentials.
Glue Secret KeyXXXSecret key for authenticating Glue catalog requests. Required when Glue credentials differ from S3 credentials.
Glue Endpointhttps://glue.ap-south-1.amazonaws.comCustom endpoint URL for AWS Glue or a Glue-compatible catalog service.
Glue Regionap-south-1Region for the Glue catalog, if different from the S3 region. Falls back to AWS Region if not set.

Here is the destination form from the run on this page. It holds the lakehouse bucket path OLake Go writes under, the region of the bucket and the Glue catalog, and the access key pair of the IAM user from step 2.

OLake UI Edit destination form for Apache Iceberg with catalog type AWS Glue, an S3 path under the lab bucket, region ap-southeast-1 and an access key pair
The Glue destination in OLake UI: the S3 path, the region and the keys of the scoped IAM user.

Step 5: Create the job and run the first sync​

S3 streams differ from database streams in one way on the Streams page. The ingestion mode is Append and cannot be changed, because a file carries no key to match an earlier row by.

Set the sync mode to Full Refresh + Incremental and leave the cursor on _last_modified_time, which OLake Go adds to every stream. Redshift is not in OLake Go's target engine picker yet, so this job lists Athena and DuckDB.

  1. Open Jobs and click Create Job. Give it a name and a schedule. Every Day at a fixed hour, or a cron expression, both work; the schedule is how often the change stream is drained, not how often a full copy runs.
  2. Select the source and the destination you created.
  3. Under Advanced Settings, set Target Query Engines to AWS Athena and DuckDB (Redshift is not in the list yet; it reads what Athena reads). OLake Go then resolves an Iceberg delete format every one of those engines can read, so you never pick one by hand.
  4. Click Next. OLake Go discovers the folders and opens the Streams page.
  5. Tick the folders to replicate. For each one, set the sync mode to Full Refresh + Incremental with _last_modified_time as the cursor column and the ingestion mode to Append.
  6. Click Create Job, then open the job's menu and choose Sync Now for the first run.

With Append mode there are no delete files to agree on, so the target engines only record which engines read the tables. Every engine that reads Iceberg v2 reads an append-only table, Redshift included.

OLake UI Streams Selection page with the s3 namespace, customers and orders selected, sync mode Full Refresh + Incremental, cursor field _last_modified_time, ingestion mode Append with Upsert greyed out, normalization on
Each folder: Full Refresh + Incremental on _last_modified_time, Append (Upsert is not offered for files). The Iceberg DB name at the top is editable.
OLake UI Job Logs and History for s3_to_redshift showing two completed syncs, the first in 11 seconds and the second in 10 seconds
Job history: the 11 s first read of both folders, then the 10 s sync of the two new files.

On this page the first run read the 8 customer and 40 order rows (48 records, 2.61 KB). It finished in 11 seconds, container start included. OLake Go names the Glue database after the job, the bucket and the namespace.

The namespace for every S3 stream is s3, so s3_to_redshift on bucket shop produced s3_to_redshift_shop_s3. The Iceberg DB field at the top of the Streams page changes it before the first sync.

Step 6: Query the tables in Redshift​

Redshift reads the Glue Data Catalog through an external schema: one statement maps a Glue database onto a schema name in your Redshift database, and every Iceberg table OLake Go commits there is queryable at once, with no refresh. Redshift Serverless and RA3 clusters both do this; the IAM role attached to the namespace or cluster needs read access to Glue and to the bucket.

Redshift query editor v2
-- Once: map the Glue database OLake Go created (schema = <job name>_<source database>_<source schema>)
CREATE EXTERNAL SCHEMA lakehouse
FROM DATA CATALOG
DATABASE 's3_to_redshift_shop_s3'
IAM_ROLE default;

-- Then query the Iceberg tables like any other schema
SELECT id, name, city, tier FROM lakehouse.customers ORDER BY id;
SELECT status, count(*) AS n, round(sum(amount), 2) AS total
FROM lakehouse.orders GROUP BY status ORDER BY status;

IAM_ROLE default uses the default role of the namespace (Serverless) or cluster; name a role ARN instead when you keep several. Redshift reads Iceberg v2 tables with positional deletes, which is why the job on this page uses that upsert type. The external schema is read-only: OLake Go is the writer, and INSERT or DELETE on these tables fails.

OLake Go adds four columns to every table: _olake_id (a hash of the primary key), _op_type, _olake_timestamp and _cdc_timestamp. _op_type is r for a row from the first snapshot, c for an insert, u for an update and d for a delete. Filter them out of dashboards, or use them to see when a row last changed.

Because files append, the query that most readers want is the latest version of each row. Redshift has no QUALIFY, so a ROW_NUMBER() subquery does it, ordering by the file time OLake Go stored in _last_modified_time. The id column arrived as DOUBLE PRECISION from the CSV, hence the cast:

Latest row per id
SELECT id::INT AS id, name, city, tier
FROM (
  SELECT *, ROW_NUMBER() OVER (PARTITION BY id ORDER BY _last_modified_time DESC) AS rn
  FROM lakehouse.customers
)
WHERE rn = 1
ORDER BY id;
Redshift query editor v2 in the AWS console running the latest-row-per-id customers query on the Serverless workgroup and returning 10 rows with id, name, city and tier
Redshift Serverless reading the customers table through the external schema after the second sync in query editor v2: one row per id.

Step 7: Add files to the bucket and sync again​

To see the incremental path work, the run on this page uploaded one more file per folder after the first sync. Each carries new rows and corrected copies of existing rows, the way a daily export does:

Files added between the two syncs
data/customers/2026-10-30/customers-changes.csv   4 rows: ids 9 and 10 are new, ids 3 and 8 are corrections
data/orders/2026-10-30/orders-changes.csv         8 rows: ids 41 to 45 are new, ids 1, 4 and 7 now read "delivered"

The second sync listed both folders, skipped the two files it had already read, and read the two new ones. That was 12 records (686 bytes) in 10 seconds end to end. The sync log shows the filter per stream, with the cursor it compared against, and the commit report shows one new data file per table and no delete files: addedDataFiles=1, addedRecords=4 for customers.

OLake UI sync log filtered to the lines Filtered 1 files to process out of 2 total with the cursor 2026-10-29T22:04:34Z, one line per stream
The second sync in the OLake UI log: one new file per folder, the first file skipped by its modification time.

Redshift shows the result with no refresh. The corrected rows exist twice.

_op_type tells them apart: r for the first full read, u for rows that arrived through an incremental sync. _last_modified_time carries the file time, which is what the ROW_NUMBER() query above sorts on:

SELECT id::INT, name, city, tier, _op_type, _last_modified_time FROM lakehouse.customers WHERE id IN (3, 8, 9, 10) ORDER BY 1, 6
id  name         city       tier    _op_type  _last_modified_time
3   Liam Carter  Austin     bronze  r         2026-10-29T22:04:34Z
3   Liam Carter  Austin     gold    u         2026-10-29T22:05:01Z
8   Zara Ali     Dubai      silver  r         2026-10-29T22:04:34Z
8   Zara Ali     Abu Dhabi  silver  u         2026-10-29T22:05:01Z
9   Maya Patel   Pune       silver  u         2026-10-29T22:05:01Z
10  Lucas Silva  São Paulo  bronze  u         2026-10-29T22:05:01Z

Schema changes. When a new column appears in a later file, OLake Go adds it to the Iceberg table on the next sync. The Glue table definition is updated in the same commit, and Redshift shows it on the next query, with nulls for older rows.

A column whose inferred type changes in an incompatible way fails the sync rather than corrupt the table. Full rules are on the schema evolution page.

Query the same tables from Athena and DuckDB​

This is the point of writing Iceberg instead of loading Redshift tables. Nothing is exported, copied or refreshed: both engines read the files OLake Go wrote, through the same Glue database. Athena uses the same ROW_NUMBER() subquery; DuckDB has QUALIFY.

Athena reads the Glue Data Catalog natively: the Glue database OLake Go created shows up in the query editor's database list, and the Iceberg tables in it need no registration and no refresh. Pick a workgroup with a query result location in S3, select the database, and query.

Athena query editor (database: the one OLake Go created)
SELECT id, name, city, tier FROM s3_to_redshift_shop_s3.customers ORDER BY id;
SELECT status, count(*) AS n, round(sum(amount), 2) AS total
FROM s3_to_redshift_shop_s3.orders GROUP BY status ORDER BY status;

Athena engine v3 reads Iceberg v2 tables with positional and equality deletes, and cannot read v3 tables at all. Keep the job on positional or equality deletes, never deletion vectors, while Athena is a reader. From the command line, aws athena start-query-execution --work-group <workgroup> --query-execution-context Database=${props.schema ?? 'postgres_to_redshift_shop_public'} --query-string "..." runs the same queries; aws athena get-query-results returns the rows.

Athena query editor in the AWS console running the latest-row-per-id customers query on the s3_to_redshift_shop_s3 database and returning 10 rows
Athena reading the same table through Glue with a ROW_NUMBER() subquery: the same 10 rows.

An orders roll-up over the latest row per id matched the files in all three engines. It gave 16 delivered orders for 8659.68, 18 placed for 10522.26, 11 shipped for 6397.25.

Data types: CSV to Iceberg to Redshift​

A CSV file has no types, so OLake Go infers them from a sample of rows at discover time and maps them to Iceberg types.

Redshift then maps Iceberg types to its own through the external schema. Parquet files skip the inference and keep their own types. This is what the CSV run on this page produced:

In the CSVIceberg (written by OLake Go)Redshift showsNotes
Whole numbers (id, customer_id)doubleDOUBLE PRECISIONIn this run the integer columns were inferred as floating point. Cast with ::INT in queries, or use Parquet when the type matters.
Decimals (amount)doubleDOUBLE PRECISIONThe orders.amount sums on this page matched the files to the cent in every engine; round at query time when you need exact decimal arithmetic.
ISO 8601 timestamps (created_at)timestamptz (UTC)TIMESTAMPStrings that parse as timestamps are stored as timestamps, shown without a zone.
Other textstringVARCHAR
_last_modified_timestringVARCHARThe object's LastModified time in ISO 8601, so it sorts correctly as text. Added to every row by OLake Go.

Types on the left come from the S3 connector reference. The Redshift side comes from the Redshift Iceberg page (checked 2026-10-30) and from svv_external_columns on the tables of this run. JSON files map integers to bigint and nested objects to JSON text; Parquet keeps int, bigint, date and timestamp as they are.

Cost and maintenance​

  • Storage is your lakehouse bucket: you pay S3 rates for the Parquet files, on top of the source files, and nothing to Glue beyond its per-object pricing for the table entries.
  • Compute. OLake Go runs wherever you run Docker. Redshift Serverless bills RPU-hours only while queries run; a query over the external schema scans the Parquet files in S3 through Redshift Spectrum, so filter on the columns you need. Athena bills per terabyte scanned.
  • Listing calls. Every sync lists the whole source prefix to find new files. A prefix with hundreds of thousands of objects makes that the slowest part of the run; keep the prefix to the folders OLake Go needs, or move processed files out.
  • Maintenance. Glue has no built-in compaction for tables written by external engines. A folder that gets a small file every few minutes wants compaction; OLake Fusion does this for the catalogs OLake Go writes to, and Spark on EMR's rewrite_data_files works through the same Glue catalog.
  • Key rotation. The IAM user's access key is what OLake Go writes with. Rotate it on your schedule, or run OLake Go on an instance with a role and leave the key fields empty.

Troubleshooting​

1. The source test is refused​

failed to access bucket shop: operation error S3: HeadBucket, https response error StatusCode: 403, api error Forbidden: Forbidden

The key pair cannot read the source bucket. On AWS, check that the policy from step 1 is attached to the user or role, and that the bucket policy does not deny it.

On MinIO, the access key or secret is wrong. A StatusCode: 404 with NotFound instead means the bucket name is wrong or the endpoint points at another store.

2. Discovery finds nothing​

Discovered 0 files in 0 streams (after filtering)
Stream grouping enabled at level 1 (first folder after path_prefix)
FATAL no streams found in connector

Either the prefix holds no files, or the files sit directly under it with no folder to name the stream. Move them into a folder per table (data/customers/...), check that Path Prefix ends with /, and loosen File Pattern if you set one.

3. Rows appear twice after a sync​

A file was rewritten in place, or a corrected copy was uploaded with the same rows. Its new LastModified time made OLake Go read it again, and Append mode added every row a second time. Keep the ROW_NUMBER() query from step 6 for reads, and change the upstream process to add files rather than replace them.

4. The destination test fails on a table OLake Go did not create​

User: arn:aws:iam::<account>:user/<olake user> is not authorized to perform: s3:GetObject on resource: "arn:aws:s3:::<another bucket>/.../test_olake.db/test_olake/metadata/00000-....metadata.json"

OLake Go's destination test opens a probe table called test_olake.test_olake in the Glue database test_olake. An earlier run in the same account and region may have left that table behind, pointing at a bucket your user cannot read.

The test then fails before it writes anything. Delete the stale table from Glue, or run OLake Go against a region whose Glue catalog does not have it.

5. Redshift cannot see the external schema's tables​

ERROR: Database 's3_to_redshift_shop_s3' does not exist in the Glue Data Catalog

The external schema was created before the first sync, or in a different region from the Glue database. OLake Go creates the Glue database on the first sync; create the external schema after it, in the workgroup's region.

How this compares with other ways to load S3 files into Redshift
MethodFreshnessUpdates and deletesWhere the data landsWhat you run
COPY from S3 on a scheduleBy scheduleInserts rows from files; no mergeNative Redshift tablesA scheduler, a manifest or prefix per load, and a running workgroup or cluster
Redshift auto-copy jobsMinutes after each file arrivesInserts rows from files; no mergeNative Redshift tables; Redshift is the only readerA COPY JOB per table on a provisioned cluster or Serverless workgroup
Redshift Spectrum external tables over the filesLive, per queryNoNothing is copied; Redshift reads the CSV in placeAn external table definition per folder, and a Glue entry to keep in step with the columns
Managed ELT (Fivetran, Estuary, Airbyte Cloud)Minutes, by planVaries by toolRedshift tables, a second copy you pay to move and storeNothing; priced per row, per GB or per connector
OLake Go to Iceberg on Glue (this guide)On a schedule down to one minuteNo; files are appended, with _last_modified_time for latest-row queriesApache Iceberg tables that Redshift, Athena, EMR, Spark and DuckDB all readOne Docker Compose stack, open source; your bucket plus your own compute

Auto-copy is the simpler choice when files should appear in Redshift within minutes and Redshift is the only reader. Writing Iceberg once turns CSV into typed Parquet that Athena, EMR and DuckDB read without a second copy. It also keeps the pipeline the same when a database or Kafka source is added beside the files.

When not to use this​

  • Rows must be updated or deleted at write time. Files have no key, so S3 streams only append. Use a database or Kafka source for that, or merge downstream into a native Redshift table.
  • The files are rewritten in place. Every rewrite is read again in full. Change the producer to add files, or accept duplicates and query the latest row per key.
  • The files are already Parquet and Redshift is the only reader. A Spectrum external table reads those files in place; copying them into Iceberg buys nothing until another engine needs them or the schema starts to drift.
  • Files are not CSV, JSON, Parquet or XML. Fixed-width, Excel and Avro files are not read.

FAQs​

How do I load files from S3 into Redshift without COPY or a Spectrum external table per folder?

Run OLake Go (open source, Apache 2.0) with the bucket and prefix as the source and the AWS Glue Data Catalog as the Iceberg destination. Each folder under the prefix becomes an Iceberg table in a Glue database, the first sync reads every file, and every later sync reads only files newer than the last run. Redshift reads the Glue database through one external schema, with no COPY job, no manifest and no table definition per folder.

Does OLake Go re-read the whole bucket on every sync?

No. OLake Go saves the newest LastModified time it has read per stream and, on the next sync, lists the prefix and reads only files modified after it. The run on this page read 1 of 2 files per stream on its second sync. A file that is rewritten in place gets a new LastModified time and is read again in full.

Can OLake Go update or delete rows from S3 files?

No. Files carry no primary key or change log, so S3 streams run in Append mode only: every row in every new file becomes a row in the table. When a later file corrects a row, both versions exist. Keep the latest with a ROW_NUMBER() subquery over the id and _last_modified_time columns, or merge downstream into a native Redshift table.

Which file formats does OLake Go read from S3?

CSV (with delimiter, header and quote options), JSON as line-delimited or array files, Parquet, and XML with a row identifier, each plain or gzip-compressed. One source handles one format; a file pattern narrows the keys when a prefix holds more than one. Parquet keeps its own types; CSV types are inferred from a sample of rows.

Does this work with MinIO, LocalStack or another S3-compatible store?

Yes, for the source side. Set the custom endpoint on the source and provide static keys. The run on this page used MinIO in Docker with the endpoint http://host.docker.internal:9000 as the source, and a real S3 bucket plus the Glue Data Catalog as the destination. When the source files already sit in S3, the endpoint stays empty and the keys can be left out to use an instance or task role.

How are the Redshift tables named for an S3 source?

After the folder. The first folder under the path prefix is the stream, so data/customers/ becomes a table called customers. The Glue database defaults to the job name, the bucket name and the namespace s3; the job s3_to_redshift on bucket shop produced s3_to_redshift_shop_s3. The Iceberg DB field on the Streams page changes it before the first sync. In Redshift the external schema name is yours to pick.

Is Redshift one of OLake Go's target query engines?

Not in the picker yet. On this page the job lists Athena and DuckDB as target engines. With Append mode there are no delete files to agree on, so the choice only records which engines read the tables, and Redshift reads them through the external schema like any other Iceberg table.

S3 to SnowflakeSame source, read from Snowflake.S3 to BigQuerySame source, read from BigQuery.S3 to DatabricksSame source, read from Databricks.Postgres to Amazon RedshiftSame engine, replicated from Postgres.Oracle to Amazon RedshiftSame engine, replicated from Oracle.Kafka to Amazon RedshiftSame engine, replicated from Kafka.MySQL to Amazon RedshiftSame engine, replicated from MySQL.MongoDB to Amazon RedshiftSame engine, replicated from MongoDB.MariaDB to Amazon RedshiftSame engine, replicated from MariaDB.SQL Server to Amazon RedshiftSame engine, replicated from SQL Server.DB2 to Amazon RedshiftSame engine, replicated from DB2.S3 connector referenceEvery field, sync mode and troubleshooting note for the source.Query engine compatibilityWhich engines read which catalogs and delete formats.All integrationsEvery source and engine pair with a tested guide.

Deeper reading on this site: