Skip to main content

S3 to Snowflake: CSV Files as Apache Iceberg

Files land in S3 every hour or every day: exports from an application, drops from a partner, logs from a job. S3 to Snowflake usually means a stage, a COPY INTO script or Snowpipe, and tables only Snowflake can read.

This guide takes a different route. OLake Go reads the files itself and turns each folder into an Apache Iceberg table. Every later run syncs only the files that are new.

With Snowflake's own Horizon catalog, the default on this page, the tables sit on Snowflake-managed storage. They appear in Snowsight as soon as a sync commits.

Nothing else is in the path: no stage, no pipe, no warehouse during the load. With AWS Glue, the Iceberg files sit in your own S3 bucket, next to the source files, and Snowflake reads them through a catalog integration.

By Shubham Satish BaldavaUpdated October 29, 2026

Key takeaways

  • OLake Go turns S3 folders into Snowflake Iceberg tables: the first sync reads every file under the prefix, and every later sync reads only files newer than the last run, by their modification time.
  • Through the Horizon catalog the tables land as Snowflake-managed Iceberg tables. You query them in Snowsight right away, with no stage, no COPY INTO, no external volume and no refresh.
  • The same tables, written once by OLake Go, are also queryable from DuckDB and Spark through the same catalog. In the run on this page all three engines returned identical rows.

The short answer. To load S3 files into Snowflake as Iceberg, run OLake Go with the bucket and prefix as the source. Point it at the Snowflake Horizon catalog as the destination.

Put each stream in Full Refresh + Incremental mode with _last_modified_time as the cursor. The first sync reads every file; every later sync reads only the files added since. Snowflake, DuckDB and Spark then query the same Iceberg tables, with no second copy.

How S3 files reach Snowflake here​

Two things happen in order, and OLake Go does both on its own. The first sync lists everything under the prefix and makes one stream per first-level folder.

Then it reads every file in parallel: that is the historical load. At the end it saves the newest LastModified time it saw per stream.

Every later sync lists the prefix again and reads only the files whose LastModified time is newer than the saved cursor. On this page the second sync reported Filtered 1 files to process out of 2 total for each stream. Rows are appended; a file never changes a row that an earlier file wrote.

One thing to keep an eye on: the cursor is the object's modification time, not anything inside the file. Uploading a corrected copy of yesterday's file gives it today's LastModified time.

The whole file is then read again, and every row in it is appended a second time. Add files, never rewrite them, and keep an id column in the data so a query can pick the latest version of a row.

What you need​

  • An S3 bucket, or an S3-compatible store, with files in folders under one prefix: CSV, JSON, Parquet or XML, plain or gzip. Amazon S3, MinIO and LocalStack all work; the run on this page used MinIO in Docker with two CSV folders.
  • Read-only access to the bucket: s3:ListBucket and s3:GetObject, through a role on AWS or static keys on MinIO.
  • A Snowflake account where you can run the one-time setup as ACCOUNTADMIN. Snowflake-managed storage means no cloud bucket or IAM role on your side for the destination.
  • OLake UI from the Docker Compose install with the OLake Go S3 source. The run on this page used v1.1.0.
  • Docker, and about 30 minutes.

Step 1: Prepare the S3 bucket​

Most readers point OLake Go at files they already have. If you want to follow along with the exact data from this page, upload two CSV files shaped like these first:

The two sample files used on this page

Each stream is a folder under data/, with one dated subfolder per upload. The header row names the columns, and id identifies the row so a later file can correct it.

data/customers/2026-10-29/customers.csv
id,name,email,city,tier,created_at,updated_at
1,Aarav Mehta,aarav@example.com,Mumbai,gold,2026-10-29T15:12:59Z,2026-10-29T15:12:59Z
2,Priya Sharma,priya@example.com,Bengaluru,silver,2026-10-29T15:12:59Z,2026-10-29T15:12:59Z
data/orders/2026-10-29/orders.csv
id,customer_id,product,amount,status,created_at,updated_at
1,2,Headphones,99.19,shipped,2026-10-29T15:12:59Z,2026-10-29T15:12:59Z
2,3,Monitor,178.38,delivered,2026-10-29T15:12:59Z,2026-10-29T15:12:59Z
Upload with the AWS CLI (add --endpoint-url for MinIO)
aws s3 sync ./stage s3://shop/data/
# Then add whatever you like. This page started with 8 customers and 40 orders.

There is nothing to switch on in S3. OLake Go lists the objects under a prefix, groups them into streams by their first folder, and remembers the newest LastModified time it has read. Four things decide whether that works for your bucket.

RequirementValueWhy it matters
Folder layout<prefix>/<stream>/.../fileThe first folder after the prefix is the stream name and becomes the table. Files sitting directly under the prefix belong to no stream.
File formatCSV, JSON (lines or array), Parquet or XML, plain or gzipOne format per source. Mixed buckets need one source per format, or a file pattern that picks one.
Read-only accesss3:ListBucket on the bucket, s3:GetObject on its objectsListing finds the files and tracks their LastModified time; GetObject reads them.
Files that are added, not rewrittenNew file per batchIncremental syncs read files newer than the saved cursor. A rewritten file gets a new LastModified time and is read again in full.

This is the layout the run on this page used, with one CSV file per day per stream:

Bucket layout
s3://shop/data/
├── customers/
│   └── 2026-10-29/customers.csv
└── orders/
  └── 2026-10-29/orders.csv

On AWS, attach a policy with the two read actions to the user or role OLake Go runs as. On MinIO or another S3-compatible store, a user with read access to the bucket is enough, and the endpoint goes into the source form.

IAM policy for the OLake Go user or role
{
"Version": "2012-10-17",
"Statement": [
  { "Effect": "Allow", "Action": ["s3:ListBucket"], "Resource": ["arn:aws:s3:::shop"] },
  { "Effect": "Allow", "Action": ["s3:GetObject"], "Resource": ["arn:aws:s3:::shop/*"] }
]
}

Leave the access key fields empty on AWS to use the instance role, task role or environment credentials instead of static keys. Static keys are required for MinIO and LocalStack.

LastModified is the cursor

OLake Go reads a file when its LastModified time is newer than the cursor saved after the last sync. Uploading a corrected copy of an old file therefore re-reads the whole file and appends every row again. Add new files for new data; keep old files as they are.

Step 2: Set up the catalog in Snowflake​

Snowflake reads Iceberg tables from two kinds of catalog. Horizon is Snowflake's own: OLake Go writes straight into it, the tables are Snowflake-managed, and they appear in Snowsight the moment a sync commits.

AWS Glue is a catalog outside Snowflake, so the Iceberg files stay in your S3 bucket. Snowflake reaches them through a catalog integration: a read-only link to that external catalog, refreshed after each sync.

Horizon is the default on this page. Glue is the right pick when the source files already live in S3 and you want the Iceberg tables beside them. Athena, Redshift and Spark on Amazon Elastic MapReduce (EMR) then read them as well.

CatalogOLake Go writesSnowflake readsVerified on this page
AWS GlueYesYes: catalog integration (read-only)Documented, not yet run by us
Snowflake HorizonYesYes: native: tables are Snowflake-managed Iceberg tablesYes, end to end

Snowflake Horizon is the catalog built into every Snowflake account. OLake Go talks to it through the Iceberg REST endpoint at https://<account_identifier>.snowflakecomputing.com/polaris/api/catalog, and the tables it creates are Snowflake-managed Iceberg tables. Writing through this endpoint needs OLake Go 0.11.0 or later.

Run the following once in a Snowsight worksheet as ACCOUNTADMIN. It creates a database for the tables, a role OLake Go writes as, a service user that holds the token, and the authentication policy the token needs.

Snowflake, once per account
USE ROLE ACCOUNTADMIN;

-- A small warehouse for your own queries, and the database OLake Go writes into.
CREATE WAREHOUSE IF NOT EXISTS OLAKE_WH WAREHOUSE_SIZE = XSMALL AUTO_SUSPEND = 60 AUTO_RESUME = TRUE;
CREATE DATABASE IF NOT EXISTS OLAKE_DEMO;
ALTER DATABASE OLAKE_DEMO SET CATALOG = 'SNOWFLAKE';
ALTER DATABASE OLAKE_DEMO SET EXTERNAL_VOLUME = 'SNOWFLAKE_MANAGED';

-- The role OLake Go uses. It needs to create schemas; tables are created inside them.
CREATE ROLE IF NOT EXISTS OLAKE_ROLE;
GRANT USAGE, CREATE SCHEMA ON DATABASE OLAKE_DEMO TO ROLE OLAKE_ROLE;
GRANT USAGE ON WAREHOUSE OLAKE_WH TO ROLE OLAKE_ROLE;
GRANT ROLE OLAKE_ROLE TO ROLE SYSADMIN;

-- A service user for OLake Go. It holds the programmatic access token.
CREATE USER IF NOT EXISTS OLAKE_SVC TYPE = SERVICE DEFAULT_ROLE = OLAKE_ROLE;
GRANT ROLE OLAKE_ROLE TO USER OLAKE_SVC;

-- Snowflake issues a token to a service user only under a network policy, unless an
-- authentication policy waives that. This one waives it. Any schema can hold the policy.
USE SCHEMA OLAKE_DEMO.PUBLIC;
CREATE AUTHENTICATION POLICY IF NOT EXISTS olake_pat_policy
  PAT_POLICY = (NETWORK_POLICY_EVALUATION = ENFORCED_NOT_REQUIRED);
ALTER USER OLAKE_SVC SET AUTHENTICATION POLICY olake_pat_policy;

-- The two values OLake Go asks for.
SELECT CURRENT_ORGANIZATION_NAME() || '-' || CURRENT_ACCOUNT_NAME() AS account_identifier,
       CURRENT_REGION() AS region;

-- The token. Snowflake shows the secret once; copy it straight into OLake Go.
ALTER USER OLAKE_SVC ADD PROGRAMMATIC ACCESS TOKEN OLAKE_PAT
  ROLE_RESTRICTION = 'OLAKE_ROLE' DAYS_TO_EXPIRY = 30;

What each block is for:

  • CATALOG = 'SNOWFLAKE' and EXTERNAL_VOLUME = 'SNOWFLAKE_MANAGED' on the database make every table OLake Go creates a Snowflake-managed Iceberg table on Snowflake's own storage. No S3 bucket, IAM role or external volume on your side. To keep the files in your own bucket, create an external volume and name it here instead.
  • The authentication policy is the step people miss. Without it, the ADD PROGRAMMATIC ACCESS TOKEN line fails with Network Policy is required when creating a programmatic access token for user OLAKE_SVC. In production, attach a network policy that allows OLake Go's egress IP instead of waiving the check.
  • ROLE_RESTRICTION pins the token to OLAKE_ROLE, so a leaked token cannot act as ACCOUNTADMIN. Tokens expire after DAYS_TO_EXPIRY (15 by default, 365 at most). Rotate it before then or the sync fails with an authentication error.

You can check the token outside OLake Go by exchanging it for a session token, the same call OLake Go makes on every sync:

Token check
curl -s -X POST "https://<account_identifier>.snowflakecomputing.com/polaris/api/catalog/v1/oauth/tokens" \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  --data-urlencode grant_type=client_credentials \
  --data-urlencode scope=session:role:OLAKE_ROLE \
  --data-urlencode client_secret=<your PAT>

A good token returns a JSON object with an access_token. The other authentication methods (key-pair JWT, External OAuth, workload identity federation) are on the Horizon catalog page.

Step 3: Connect S3 to OLake Go​

  1. In the OLake UI, open Sources and click Create Source. Pick S3 as the connector, give the source a name and keep the OLake Go version on the latest release.
  2. The Enable CDC toggle under Change data capture (CDC) reads not supported for S3 and stays off. Object storage has no change log; the job's Full Refresh + Incremental mode on the file modification time is what carries new data.
  3. Under Connection details, fill in the S3 Bucket Name, the AWS Region, and the Path Prefix that holds the stream folders. On AWS, leave Access Key ID and Secret Access Key empty to use the role OLake Go runs with; on MinIO or LocalStack fill them in and set Custom Endpoint to the service URL.
  4. Pick the File Format. For CSV, check the delimiter, the header toggle and the quote character; for XML, set the row identifier. File Pattern is an optional regular expression on the object key, useful when a prefix mixes formats.
  5. Click Set up source. OLake Go calls HeadBucket with the credentials; a 403 or 404 here means the key, the bucket name or the endpoint is wrong.

Here is the source form from the run on this page. MinIO ran in another container, so the custom endpoint is http://host.docker.internal:9000. The bucket is shop, the prefix data/, and the format CSV with a header row.

OLake UI Edit source form for S3 on OLake Go v1.1.0 with the Enable CDC toggle marked not supported, bucket shop, region us-east-1, static keys, path prefix data/, custom endpoint http://host.docker.internal:9000, file format CSV with delimiter comma, header on and quote character double quote
The S3 source in OLake UI: bucket, prefix, the MinIO endpoint, and the CSV options. The keys shown are MinIO defaults for a local test.

Set up source calls HeadBucket before anything is saved, so a wrong key or bucket name fails here rather than on the first sync.

The full field reference, including JSON, Parquet and XML options, is on the S3 connector page.

Step 4: Configure the Iceberg destination​

  1. Open Destinations and click Create Destination.
  2. Pick Apache Iceberg as the connector and Horizon as the catalog type.
  3. Choose Programmatic Access Token (PAT) as the authentication type and fill in the fields below.
  4. Click Create. OLake Go calls the Horizon endpoint, exchanges the token and saves the destination.
OLake UI destination form for the Snowflake Horizon catalog with the PAT authentication fields

Horizon Configuration Parameters

ParameterSample ValueDescription
Authentication Type
required		Programmatic Access Token (PAT)Authentication method OLake Go uses for Horizon Catalog REST API requests. Set to match your workspace's authentication configuration.
Horizon Iceberg REST Catalog URI
required		https://<account_identifier>.snowflakecomputing.com/polaris/api/catalogSnowflake Horizon Catalog REST API endpoint.
Snowflake Database
required		SNOWThe name of the database in Snowflake.
Session Role Scope
required		session:role:<HORIZON_ROLE>Informs Snowflake which role to use for this Horizon Catalog REST session, which controls privileges for catalog and table operations. Space-separated if providing more than one.
Programmatic Access Token
required		ver:1-hint:...Snowflake programmatic access token (PAT).
Cloud Storage Region
required		AWS_AP_SOUTH_1Snowflake's account region used to route requests to the correct storage endpoint. Run SELECT CURRENT_REGION(); to know the region
Catalog NamehorizonName of the Iceberg catalog OLake Go registers tables under. Defaults to olake_iceberg if left empty.
Enable Arrow WritesfalseWrites data and delete files using Apache Arrow based writer and registers them in Iceberg.
Catalog Name Supported for v0.3.5 and above
For the catalog name, OLake Go only supports lowercase letters and underscores. Spaces and special characters are not supported.

Step 5: Create the job and run the first sync​

S3 streams differ from database streams in one way on the Streams page. The ingestion mode is Append and cannot be changed, because a file carries no key to match an earlier row by. Set the sync mode to Full Refresh + Incremental and leave the cursor on _last_modified_time, which OLake Go adds to every stream.

  1. Open Jobs and click Create Job. Give it a name and a schedule. Every Day at a fixed hour, or a cron expression, both work; the schedule is how often the change stream is drained, not how often a full copy runs.
  2. Select the source and the destination you created.
  3. Under Advanced Settings, set Target Query Engines to Snowflake, plus DuckDB and Apache Spark if they will read the tables too. OLake Go then resolves an Iceberg delete format every one of those engines can read, so you never pick one by hand.
  4. Click Next. OLake Go discovers the folders and opens the Streams page.
  5. Tick the folders to replicate. For each one, set the sync mode to Full Refresh + Incremental with _last_modified_time as the cursor column and the ingestion mode to Append.
  6. Click Create Job, then open the job's menu and choose Sync Now for the first run.

Three target query engines were set on this job. With Append mode there are no delete files to agree on, so the choice only records which engines read the tables.

OLake UI Streams Selection page with the s3 namespace, customers and orders selected, sync mode Full Refresh + Incremental, cursor field _last_modified_time, ingestion mode Append with Upsert greyed out, normalization on
Each folder: Full Refresh + Incremental on _last_modified_time, Append (Upsert is not offered for files). The Iceberg DB name at the top is editable.
OLake UI Job Logs and History for s3_to_snowflake showing two completed syncs, the first in 26 seconds and the second in 15 seconds
Job history: the 26 s first read of both folders, then the 15 s sync of the two new files.

On this page the first run read the 8 customer and 40 order rows (48 records, 2.61 KB). It finished in 26 seconds, container start included. OLake Go names the destination schema after the job, the bucket and the namespace.

The namespace for every S3 stream is s3, so s3_to_snowflake on bucket shop produced s3_to_snowflake_shop_s3. The Iceberg DB field at the top of the Streams page changes it before the first sync.

Step 6: Query the tables in Snowflake​

There is no registration step. The schema and tables OLake Go committed through Horizon are already Snowflake-managed Iceberg tables in the database you configured. One thing to know before the first query: OLake Go creates them with quoted, lowercase names, so Snowflake needs the double quotes.

Snowsight
-- What OLake Go created (schema = <job name>_<source database>[_<source schema>])
SHOW ICEBERG TABLES IN SCHEMA OLAKE_DEMO."s3_to_snowflake_shop_s3";

-- Quoted identifiers: the names are lowercase, as they were in the source
SELECT "id", "name", "city", "tier"
FROM OLAKE_DEMO."s3_to_snowflake_shop_s3"."customers"
ORDER BY "id";

SHOW ICEBERG TABLES reports catalog_name = SNOWFLAKE, iceberg_table_type = MANAGED and external_volume_name = SNOWFLAKE_MANAGED: Snowflake treats them as its own managed Iceberg tables, not as external tables that need a refresh.

OLake Go adds four columns to every table: _olake_id (a hash of the primary key), _op_type, _olake_timestamp and _cdc_timestamp. _op_type is r for a row from the first snapshot, c for an insert, u for an update and d for a delete. Filter them out of dashboards, or use them to see when a row last changed.

Snowsight SQL file running SHOW ICEBERG TABLES on the s3_to_snowflake_shop_s3 schema and listing customers and orders with owner OLAKE_ROLE, external volume SNOWFLAKE_MANAGED, catalog SNOWFLAKE and table type MANAGED
SHOW ICEBERG TABLES in Snowsight: both folders are now Snowflake-managed tables, created by OLAKE_ROLE through Horizon.

Because files append, the query that most readers want is the latest version of each row. QUALIFY does that in one statement, ordering by the file time OLake Go stored in _last_modified_time:

Latest row per id
SELECT "id"::INT AS id, "name", "city", "tier"
FROM OLAKE_DEMO."s3_to_snowflake_shop_s3"."customers"
QUALIFY ROW_NUMBER() OVER (PARTITION BY "id" ORDER BY "_last_modified_time" DESC) = 1
ORDER BY id;
Snowsight query on OLAKE_DEMO.s3_to_snowflake_shop_s3.customers with QUALIFY ROW_NUMBER over id ordered by _last_modified_time returning 10 rows
The customers table in Snowsight after the second sync, one row per id, on an X-Small warehouse.

Step 7: Add files to the bucket and sync again​

To see the incremental path work, the run on this page uploaded one more file per folder after the first sync. Each carries new rows and corrected copies of existing rows, the way a daily export does:

Files added between the two syncs
data/customers/2026-10-29/customers-changes.csv   4 rows: ids 9 and 10 are new, ids 3 and 8 are corrections
data/orders/2026-10-29/orders-changes.csv         8 rows: ids 41 to 45 are new, ids 1, 4 and 7 now read "delivered"

The second sync listed both folders, skipped the two files it had already read, and read the two new ones. That was 12 records (686 bytes) in 15 seconds end to end. The sync log shows the filter per stream, with the cursor it compared against:

OLake UI sync log filtered to the lines Filtered 1 files to process out of 2 total with the cursor 2026-10-29T15:43:01Z, one line per stream
The second sync in the OLake UI log: one new file per folder, the first file skipped by its modification time.

Snowflake shows the result with no refresh. The corrected rows exist twice.

_op_type tells them apart: r for the first full read, u for rows that arrived through an incremental sync. _last_modified_time carries the file time, which is what the QUALIFY query above sorts on.

Snowsight query on customers for ids 3, 8, 9 and 10 returning 6 rows: two versions each for Liam Carter and Zara Ali with _op_type r and u, and one row each for Maya Patel and Lucas Silva
Both versions of the corrected rows: Liam Carter bronze then gold, Zara Ali Dubai then Abu Dhabi, each tagged with the file time it came from.

Schema changes. When a new column appears in a later file, OLake Go adds it to the Iceberg table on the next sync. Snowflake shows it with nulls for older rows.

A column whose inferred type changes in an incompatible way fails the sync rather than corrupt the table. Full rules are on the schema evolution page.

Query the same tables from DuckDB and Spark​

This is the point of writing Iceberg instead of loading Snowflake tables. Nothing is exported, copied or refreshed: both engines read the files OLake Go wrote, through the same Horizon endpoint, with credentials vended by Snowflake. The same QUALIFY pattern works in DuckDB; Spark uses a ROW_NUMBER() subquery.

DuckDB attaches Horizon as an Iceberg REST catalog. It wants a bearer token rather than the raw PAT, so exchange the PAT first (the curl call from the catalog setup), then attach:

DuckDB 1.5.5
INSTALL iceberg; LOAD iceberg;
INSTALL httpfs;  LOAD httpfs;

CREATE SECRET horizon_secret (TYPE iceberg, TOKEN '<access_token from /v1/oauth/tokens>');

ATTACH 'OLAKE_DEMO' AS horizon (
TYPE iceberg,
ENDPOINT 'https://<account_identifier>.snowflakecomputing.com/polaris/api/catalog',
SECRET horizon_secret,
ACCESS_DELEGATION_MODE 'vended_credentials'
);

SHOW ALL TABLES;
SELECT id, name, city, tier FROM horizon.s3_to_snowflake_shop_s3.customers ORDER BY id;

The session token lives about an hour; re-run the exchange and CREATE OR REPLACE SECRET when a query fails with an authorization error. DuckDB reads deletion vectors from duckdb-iceberg 1.5.3 onward, and only attaches REST catalogs backed by S3, S3 Tables or Google Cloud Storage, so pick an AWS- or GCP-hosted Snowflake account for this path.

DuckDB 1.5.5 shell querying horizon.s3_to_snowflake_shop_s3.customers with QUALIFY row_number over id and returning the same 10 rows as Snowflake
DuckDB 1.5.5 reading the Snowflake-managed table through Horizon: the same 10 rows, latest version per id.

An orders roll-up over the latest row per id matched the files in all three engines. It gave 16 delivered orders for 8659.68, 18 placed for 10522.26, 11 shipped for 6397.25.

Data types: CSV to Iceberg to Snowflake​

A CSV file has no types, so OLake Go infers them from a sample of rows at discover time and maps them to Iceberg types.

Snowflake then maps Iceberg types to its own. Parquet files skip the inference and keep their own types. This is what the CSV run on this page produced:

In the CSVIceberg (written by OLake Go)Snowflake showsNotes
Whole numbers (id, customer_id)doubleFLOATIn this run the integer columns were inferred as floating point. Cast with ::INT in queries, or use Parquet when the type matters.
Decimals (amount)doubleFLOATThe orders.amount sums on this page matched the files to the cent in every engine; round at query time when you need exact decimal arithmetic.
ISO 8601 timestamps (created_at)timestamptz (UTC)TIMESTAMP_LTZ(6)Strings that parse as timestamps are stored as timestamps; set the session zone before comparing.
Other textstringVARCHAR
_last_modified_timestringVARCHARThe object's LastModified time in ISO 8601, so it sorts correctly as text. Added to every row by OLake Go.

Types on the left come from the S3 connector reference; the Snowflake side from Snowflake's Iceberg data types page (checked 2026-10-14). JSON files map integers to bigint and nested objects to JSON text; Parquet keeps int, bigint, date and timestamp as they are.

Cost and maintenance​

  • Storage is Snowflake's with SNOWFLAKE_MANAGED: you pay Snowflake storage rates for the Iceberg data files, on top of the source files in your bucket. With Glue as the catalog, both sit in your bucket.
  • Compute. OLake Go runs wherever you run Docker; it does not use a Snowflake warehouse to write. Snowflake compute is spent only when you query.
  • Listing calls. Every sync lists the whole prefix to find new files. A prefix with hundreds of thousands of objects makes that the slowest part of the run; keep the prefix to the folders OLake Go needs, or move processed files out.
  • Maintenance. Snowflake expires old snapshots and compacts manifests on Snowflake-managed tables at no cost. Automatic data-file compaction is bundled only when Snowflake alone writes the table (Snowflake-managed Iceberg table maintenance). A folder that gets a small file every few minutes wants compaction from the writer side; OLake Fusion does this for the catalogs OLake Go writes to, and Spark's rewrite_data_files works through the same Horizon endpoint.
  • Token rotation. The programmatic access token expires (30 days in the setup above, 365 at most). Put the renewal on a calendar; an expired token fails the next sync at the catalog handshake, before any file is read.

Troubleshooting​

1. The source test is refused​

failed to access bucket shop: operation error S3: HeadBucket, https response error StatusCode: 403, api error Forbidden: Forbidden

The key pair cannot read the bucket. On AWS, check that the policy from step 1 is attached to the user or role, and that the bucket policy does not deny it.

On MinIO, the access key or secret is wrong. A StatusCode: 404 with NotFound instead means the bucket name is wrong or the endpoint points at another store.

2. Discovery finds nothing​

Discovered 0 files in 0 streams (after filtering)
Stream grouping enabled at level 1 (first folder after path_prefix)
FATAL no streams found in connector

Either the prefix holds no files, or the files sit directly under it with no folder to name the stream. Move them into a folder per table (data/customers/...), check that Path Prefix ends with /, and loosen File Pattern if you set one.

3. Snowflake cannot find the table​

SQL compilation error:
Object 'OLAKE_DEMO.S3_TO_SNOWFLAKE_SHOP_S3.CUSTOMERS' does not exist or not authorized.

Snowflake upper-cased an unquoted identifier. OLake Go creates the schema and tables with their lowercase folder names, so quote them: OLAKE_DEMO."s3_to_snowflake_shop_s3"."customers".

4. Rows appear twice after a sync​

A file was rewritten in place, or a corrected copy was uploaded with the same rows. Its new LastModified time made OLake Go read it again, and Append mode added every row a second time. Keep the QUALIFY ROW_NUMBER() query from step 6 for reads, and change the upstream process to add files rather than replace them.

5. The destination test fails at the catalog handshake​

OLake Go's destination test exchanges the token at /v1/oauth/tokens before anything else. Three causes cover nearly every failure: a service user without a network or authentication policy, a scope that does not match the token's ROLE_RESTRICTION, or an expired token. Run the curl exchange from the catalog setup; it returns the same error Snowflake gives OLake Go.

How this compares with other ways to load S3 files into Snowflake
MethodFreshnessUpdates and deletesWhere the data landsWhat you run
Snowpipe with auto-ingest from an external stageMinutes after each file arrivesInserts rows from files with COPY INTO semantics; no mergeNative Snowflake tables; Snowflake is the only readerA stage, a pipe and an S3 event notification, billed in Snowpipe credits
COPY INTO on a scheduleBy scheduleInserts rows from files; no mergeNative Snowflake tablesA task or scheduler and a running warehouse
Snowflake external tables over the filesOn refreshNoNothing is copied; Snowflake reads the files in placeA stage and a refresh per new file
Managed ELT (Fivetran, Estuary, Airbyte Cloud)Minutes, by planVaries by toolSnowflake tables, a second copy you pay to move and storeNothing; priced per row, per GB or per connector
OLake Go to Iceberg on Horizon (this guide)On a schedule down to one minuteNo; files are appended, with _last_modified_time for latest-row queriesApache Iceberg tables that Snowflake, DuckDB, Spark and Trino all readOne Docker Compose stack, open source; Snowflake storage plus your own compute

Snowpipe is the simpler choice when files should appear in Snowflake within minutes and Snowflake is the only reader. Writing Iceberg once avoids a second copy per engine as soon as DuckDB, Spark or Trino need the same rows. It also keeps the pipeline the same when a database or Kafka source is added beside the files.

When not to use this​

  • Rows must be updated or deleted at write time. Files have no key, so S3 streams only append. Use a database or Kafka source for that, or merge downstream.
  • The files are rewritten in place. Every rewrite is read again in full. Change the producer to add files, or accept duplicates and query the latest row per key.
  • The files are already Parquet and Snowflake is the only reader. Snowflake can read those files in place through an external table; copying them into Iceberg buys nothing until another engine needs them.
  • Files are not CSV, JSON, Parquet or XML. Fixed-width, Excel and Avro files are not read.

FAQs​

How do I load files from S3 into Snowflake without Snowpipe or COPY INTO?

Run OLake Go (open source, Apache 2.0) with the bucket as the source and Snowflake's Horizon catalog as the Iceberg destination. Each folder under the prefix becomes a Snowflake-managed Iceberg table, the first sync reads every file, and every later sync reads only files newer than the last run. No stage, pipe or warehouse is involved in the load.

Does OLake Go re-read the whole bucket on every sync?

No. OLake Go saves the newest LastModified time it has read per stream and, on the next sync, lists the prefix and reads only files modified after it. The run on this page read 1 of 2 files per stream on its second sync. A file that is rewritten in place gets a new LastModified time and is read again in full.

Can OLake Go update or delete rows from S3 files?

No. Files carry no primary key or change log, so S3 streams run in Append mode only: every row in every new file becomes a row in the table. When a later file corrects a row, both versions exist. Keep the latest with a QUALIFY ROW_NUMBER() query over the id and _last_modified_time columns, or merge downstream.

Which file formats does OLake Go read from S3?

CSV (with delimiter, header and quote options), JSON as line-delimited or array files, Parquet, and XML with a row identifier, each plain or gzip-compressed. One source handles one format; a file pattern narrows the keys when a prefix holds more than one. Parquet keeps its own types; CSV types are inferred from a sample of rows.

Does this work with MinIO, LocalStack or another S3-compatible store?

Yes. Set the custom endpoint on the source and provide static keys. The run on this page used MinIO in Docker with the endpoint http://host.docker.internal:9000. On AWS S3 itself the endpoint stays empty and the keys can be left out to use an instance or task role.

How are the Snowflake tables named for an S3 source?

After the folder. The first folder under the path prefix is the stream, so data/customers/ becomes a table called customers. The schema defaults to the job name, the bucket name and the namespace s3; the job s3_to_snowflake on bucket shop produced s3_to_snowflake_shop_s3. The Iceberg DB field on the Streams page changes it before the first sync.

Can I use AWS Glue instead of the Horizon catalog?

Yes. With Glue as the catalog the files stay in your S3 bucket, Athena, Redshift and EMR read them natively, and Snowflake reads them through a catalog integration as read-only tables you refresh after each sync. Horizon is the better default when Snowflake is the main reader, because the tables are Snowflake-managed and need no refresh.

S3 to BigQuerySame source, read from BigQuery.S3 to Amazon RedshiftSame source, read from Amazon Redshift.S3 to DatabricksSame source, read from Databricks.Postgres to SnowflakeSame engine, replicated from Postgres.MySQL to SnowflakeSame engine, replicated from MySQL.MongoDB to SnowflakeSame engine, replicated from MongoDB.Oracle to SnowflakeSame engine, replicated from Oracle.SQL Server to SnowflakeSame engine, replicated from SQL Server.DB2 to SnowflakeSame engine, replicated from DB2.MariaDB to SnowflakeSame engine, replicated from MariaDB.Kafka to SnowflakeSame engine, replicated from Kafka.S3 connector referenceEvery field, sync mode and troubleshooting note for the source.Query engine compatibilityWhich engines read which catalogs and delete formats.All integrationsEvery source and engine pair with a tested guide.

Deeper reading on this site: