Skip to main content

S3 to Databricks: Load Files as Apache Iceberg

Files land in S3 every hour or every day: exports from an application, drops from a partner, logs from a job. S3 to Databricks usually means an Auto Loader job on a cluster, and tables written by that job.

This guide takes a different route. OLake Go reads the files itself and turns each folder into an Apache Iceberg table in Unity Catalog. Every later run syncs only the files that are new.

The tables land as Unity Catalog managed Iceberg tables, with the files in your own cloud storage. Nothing else is in the path: no Databricks job, no cluster during the load. DuckDB and Spark read the same tables through the same catalog.

By Shubham Satish BaldavaUpdated October 30, 2026

Key takeaways

  • OLake Go turns S3 folders into Databricks Iceberg tables: the first sync reads every file under the prefix, and every later sync reads only files newer than the last run, by their modification time.
  • The tables are Unity Catalog managed Iceberg tables. You query them from the Databricks SQL editor or a notebook right away, with no external table and no refresh.
  • The same tables, written once by OLake Go, are also queryable from DuckDB and Spark through Unity Catalog. In the run on this page all three engines returned identical rows.

The short answer. To load S3 files into Databricks as Iceberg, run OLake Go with the bucket and prefix as the source and Unity Catalog as the destination. Put each stream in Full Refresh + Incremental mode with _last_modified_time as the cursor.

The first sync reads every file; every later sync reads only the files added since. Databricks, DuckDB and Spark then query the same Iceberg tables, with no second copy.

How S3 files reach Databricks here​

The first sync lists everything under the prefix and makes one stream per first-level folder. Then it reads every file in parallel: that is the historical load. At the end it saves the newest LastModified time it saw per stream.

Every later sync lists the prefix again and reads only the files whose LastModified time is newer than the saved cursor. Rows are appended; a file never changes a row that an earlier file wrote. Each sync ends with one commit to Unity Catalog, which Databricks sees on its next query.

One thing to keep an eye on: the cursor is the object's modification time, not anything inside the file. Uploading a corrected copy of yesterday's file gives it today's LastModified time, and every row in it is appended a second time. Add files, never rewrite them, and keep an id column so a query can pick the latest version of a row.

What you need​

  • An S3 bucket, or an S3-compatible store, with files in folders under one prefix: CSV, JSON, Parquet or XML, plain or gzip. Amazon S3, MinIO and LocalStack all work; the run on this page used MinIO in Docker with two CSV folders.
  • Read-only access to the bucket: s3:ListBucket and s3:GetObject, through a role on AWS or static keys on MinIO.
  • A Databricks workspace with Unity Catalog, a catalog with managed storage, and an admin who can turn on external data access and grant EXTERNAL USE SCHEMA. The run on this page used a workspace on Google Cloud.
  • OLake UI from the Docker Compose install with the OLake Go S3 source. The run on this page used v1.1.0.
  • Docker, and about 30 minutes.

Step 1: Prepare the S3 bucket​

Most readers point OLake Go at files they already have. If you want to follow along with the exact data from this page, upload two CSV files shaped like these first:

The two sample files used on this page

Each stream is a folder under data/, with one dated subfolder per upload. The header row names the columns, and id identifies the row so a later file can correct it.

data/customers/2026-10-29/customers.csv
id,name,email,city,tier,created_at,updated_at
1,Aarav Mehta,aarav@example.com,Mumbai,gold,2026-10-29T15:12:59Z,2026-10-29T15:12:59Z
2,Priya Sharma,priya@example.com,Bengaluru,silver,2026-10-29T15:12:59Z,2026-10-29T15:12:59Z
data/orders/2026-10-29/orders.csv
id,customer_id,product,amount,status,created_at,updated_at
1,2,Headphones,99.19,shipped,2026-10-29T15:12:59Z,2026-10-29T15:12:59Z
2,3,Monitor,178.38,delivered,2026-10-29T15:12:59Z,2026-10-29T15:12:59Z
Upload with the AWS CLI (add --endpoint-url for MinIO)
aws s3 sync ./stage s3://shop/data/
# Then add whatever you like. This page started with 8 customers and 40 orders.

There is nothing to switch on in S3. OLake Go lists the objects under a prefix, groups them into streams by their first folder, and remembers the newest LastModified time it has read. Four things decide whether that works for your bucket.

RequirementValueWhy it matters
Folder layout<prefix>/<stream>/.../fileThe first folder after the prefix is the stream name and becomes the table. Files sitting directly under the prefix belong to no stream.
File formatCSV, JSON (lines or array), Parquet or XML, plain or gzipOne format per source. Mixed buckets need one source per format, or a file pattern that picks one.
Read-only accesss3:ListBucket on the bucket, s3:GetObject on its objectsListing finds the files and tracks their LastModified time; GetObject reads them.
Files that are added, not rewrittenNew file per batchIncremental syncs read files newer than the saved cursor. A rewritten file gets a new LastModified time and is read again in full.

This is the layout the run on this page used, with one CSV file per day per stream:

Bucket layout
s3://shop/data/
├── customers/
│   └── 2026-10-29/customers.csv
└── orders/
  └── 2026-10-29/orders.csv

On AWS, attach a policy with the two read actions to the user or role OLake Go runs as. On MinIO or another S3-compatible store, a user with read access to the bucket is enough, and the endpoint goes into the source form.

IAM policy for the OLake Go user or role
{
"Version": "2012-10-17",
"Statement": [
  { "Effect": "Allow", "Action": ["s3:ListBucket"], "Resource": ["arn:aws:s3:::shop"] },
  { "Effect": "Allow", "Action": ["s3:GetObject"], "Resource": ["arn:aws:s3:::shop/*"] }
]
}

Leave the access key fields empty on AWS to use the instance role, task role or environment credentials instead of static keys. Static keys are required for MinIO and LocalStack.

LastModified is the cursor

OLake Go reads a file when its LastModified time is newer than the cursor saved after the last sync. Uploading a corrected copy of an old file therefore re-reads the whole file and appends every row again. Add new files for new data; keep old files as they are.

Step 2: Set up Unity Catalog in Databricks​

Databricks reads Iceberg tables through Unity Catalog, the catalog built into every Databricks workspace. OLake Go writes straight into it, so the tables become Unity Catalog managed Iceberg tables. They show up in Catalog Explorer and the SQL editor the moment a sync commits.

The files sit in the catalog's managed storage, which is backed by an external location in your own cloud account. DuckDB and Spark reach the same tables through the same Unity Catalog endpoint.

CatalogOLake Go writesDatabricks readsVerified on this page
Unity CatalogYesYes: native: Unity Catalog managed Iceberg tablesYes, end to end

Unity Catalog is the catalog built into every Databricks workspace. OLake Go writes to it through Unity Catalog's Iceberg REST (representational state transfer) endpoint at https://<workspace-url>/api/2.1/unity-catalog/iceberg-rest, and the tables it creates are Unity Catalog managed Iceberg tables. Databricks allows writes through this endpoint for managed Iceberg tables only (Databricks: access tables from Iceberg clients, checked 2026-10-30).

Four things must be in place in the workspace, once:

SettingWhereWhy it matters
External data access on the metastoreCatalog, the gear icon, Metastore, then External data accessWithout it the Iceberg REST endpoint refuses every client that is not Databricks itself.
A catalog with managed storageCatalog, Create catalog, with a storage location backed by an external locationOLake Go creates one schema per job inside it. The catalog name is the warehouse value every client uses.
EXTERNAL USE SCHEMA for the writing principalOn the catalog, so it covers the schemas OLake Go createsUnity Catalog checks this privilege, on top of the usual ones, before it serves a table to an outside engine.
A personal access token or a service principalSettings, Developer, Access tokensOLake Go sends it as a bearer token on every catalog call.
Databricks SQL editor, once per catalog (replace the principal)
-- The catalog OLake Go writes into. Its managed storage holds the Iceberg files.
CREATE CATALOG IF NOT EXISTS lakehouse MANAGED LOCATION '<external location URL>/lakehouse';

-- What the writing principal needs: browse and create schemas, then read and write tables in them
GRANT USE CATALOG, CREATE SCHEMA ON CATALOG lakehouse TO `olake@example.com`;
GRANT USE SCHEMA, CREATE TABLE, SELECT, MODIFY ON CATALOG lakehouse TO `olake@example.com`;
GRANT EXTERNAL USE SCHEMA ON CATALOG lakehouse TO `olake@example.com`;

Granting on the catalog makes the privileges inherit to every schema OLake Go creates later. EXTERNAL USE SCHEMA is not part of ALL PRIVILEGES, and only the catalog owner can grant it (Databricks: Unity Catalog privileges, checked 2026-10-30). A service principal with OAuth machine-to-machine credentials is the better choice in production. The other authentication methods are on the Unity Catalog destination page.

Databricks reads deletion vectors only

Unity Catalog accepts updates to a managed Iceberg table only as copy-on-write files or as Iceberg v3 deletion vectors. It rejects equality delete files with Can not create a Path from a null string [ErrorCode: 2015]. Set Databricks as a target query engine on the job, and OLake Go writes deletion vectors from the first sync.

Step 3: Connect S3 to OLake Go​

  1. In the OLake UI, open Sources and click Create Source. Pick S3 as the connector, give the source a name and keep the OLake Go version on the latest release.
  2. The Enable CDC toggle under Change data capture (CDC) reads not supported for S3 and stays off. Object storage has no change log; the job's Full Refresh + Incremental mode on the file modification time is what carries new data.
  3. Under Connection details, fill in the S3 Bucket Name, the AWS Region, and the Path Prefix that holds the stream folders. On AWS, leave Access Key ID and Secret Access Key empty to use the role OLake Go runs with; on MinIO or LocalStack fill them in and set Custom Endpoint to the service URL.
  4. Pick the File Format. For CSV, check the delimiter, the header toggle and the quote character; for XML, set the row identifier. File Pattern is an optional regular expression on the object key, useful when a prefix mixes formats.
  5. Click Set up source. OLake Go calls HeadBucket with the credentials; a 403 or 404 here means the key, the bucket name or the endpoint is wrong.

Here is the source form from the run on this page. MinIO ran in another container, so the custom endpoint is http://host.docker.internal:9000. The bucket is shop, the prefix data/, and the format CSV with a header row.

OLake UI Edit source form for S3 on OLake Go v1.1.0 with the Enable CDC toggle marked not supported, bucket shop, region us-east-1, static keys, path prefix data/, custom endpoint http://host.docker.internal:9000, file format CSV with delimiter comma, header on and quote character double quote
The S3 source in OLake UI: bucket, prefix, the MinIO endpoint, and the CSV options. The keys shown are MinIO defaults for a local test.

Set up source calls HeadBucket before anything is saved, so a wrong key or bucket name fails here rather than on the first sync.

The full field reference, including JSON, Parquet and XML options, is on the S3 connector page.

Step 4: Configure the Iceberg destination​

  1. Open Destinations and click Create Destination.
  2. Pick Apache Iceberg as the connector and Unity as the catalog type.
  3. Choose Personal Access Token as the authentication type and fill in the fields below. The catalog name from the setup goes into the catalog field.
  4. Click Create. OLake Go calls the Unity Catalog endpoint with the token and saves the destination.
OLake UI destination form for an Apache Iceberg destination on the Unity catalog type with the personal access token fields

The run on this page wrote into a catalog called datazip, with a personal access token that expires after seven days. The snippets below use lakehouse; replace it with your catalog name.

Unity Configuration Parameters

ParameterSample ValueDescription
REST Catalog URL
required		https://1234567890123456.1.gcp.databricks.com/Databricks workspace URL with Unity Catalog REST API endpoint. Use your actual workspace URL.
Authentication Type
required		Personal Access TokenAuthentication method OLake Go uses for Unity Catalog REST API requests. Set to match your workspace's authentication configuration.
Unity Catalog Name
required		workspaceName of the catalog in Unity.
Token
required		dapi123456789...Databricks Personal Access Token
Catalog Nameolake_icebergName of the Iceberg catalog OLake Go registers tables under. Defaults to olake_iceberg if left empty.
Enable Arrow WritesfalseWrites data and delete files using Apache Arrow based writer and registers them in Iceberg.
Catalog Name Supported for v0.3.5 and above
For the catalog name, OLake Go only supports lowercase letters and underscores. Spaces and special characters are not supported.

Step 5: Create the job and run the first sync​

S3 streams differ from database streams in one way on the Streams page. The ingestion mode is Append and cannot be changed, because a file carries no key to match an earlier row by. Set the sync mode to Full Refresh + Incremental and leave the cursor on _last_modified_time, which OLake Go adds to every stream.

  1. Open Jobs and click Create Job. Give it a name and a schedule. Every Day at a fixed hour, or a cron expression, both work; the schedule is how often the change stream is drained, not how often a full copy runs.
  2. Select the source and the destination you created.
  3. Under Advanced Settings, set Target Query Engines to Databricks, plus DuckDB and Apache Spark if they will read the tables too. OLake Go then resolves an Iceberg delete format every one of those engines can read, so you never pick one by hand.
  4. Click Next. OLake Go discovers the folders and opens the Streams page.
  5. Tick the folders to replicate. For each one, set the sync mode to Full Refresh + Incremental with _last_modified_time as the cursor column and the ingestion mode to Append.
  6. Click Create Job, then open the job's menu and choose Sync Now for the first run.

Three target query engines were set on this job. With Append mode there are no delete files to agree on, so the choice only records which engines read the tables.

OLake UI Streams Selection page with the s3 namespace, customers and orders selected, sync mode Full Refresh + Incremental, cursor field _last_modified_time, ingestion mode Append with Upsert greyed out, normalization on
Each folder: Full Refresh + Incremental on _last_modified_time, and Append. The Iceberg DB name at the top is editable.
OLake UI Job Logs and History for s3_to_databricks showing two completed syncs, the first in 23 seconds and the second in 17 seconds
Job history: the 23 s first read of both folders, then the 17 s sync of the two new files.

On this page the first run read the 8 customer and 40 order rows (48 records, 2.61 KB). It finished in 23 seconds, container start included. OLake Go names the Unity Catalog schema after the job, the bucket and the namespace, so s3_to_databricks on bucket shop produced s3_to_databricks_shop_s3.

Step 6: Query the tables in Databricks​

There is no registration step. The schema and tables OLake Go committed through Unity Catalog are managed Iceberg tables in the catalog you configured, and the Databricks SQL editor or a notebook reads them by their three-part name.

Databricks SQL editor
-- What OLake Go created (schema = <job name>_<source database>[_<source schema>])
SHOW TABLES IN lakehouse.s3_to_databricks_shop_s3;

SELECT id, name, city, tier
FROM lakehouse.s3_to_databricks_shop_s3.customers
ORDER BY id;

SELECT status, count(*) AS n, round(sum(amount), 2) AS total
FROM lakehouse.s3_to_databricks_shop_s3.orders
GROUP BY status ORDER BY status;

Any SQL warehouse or cluster on a recent Databricks Runtime reads them; a 2X-Small serverless warehouse is enough for checks. DESCRIBE EXTENDED on a table shows the Iceberg provider and the deletion vectors in its properties.

OLake Go adds four columns to every table: _olake_id (a hash of the primary key), _op_type, _olake_timestamp and _cdc_timestamp. _op_type is r for a row from the first snapshot, c for an insert, u for an update and d for a delete. Filter them out of dashboards, or use them to see when a row last changed.

Databricks SQL editor running a latest-row-per-id query on datazip.s3_to_databricks_shop_s3.customers on the 2X-Small SQL warehouse, returning 10 rows
The same table in the Databricks SQL editor: the S3 rows are append-only, so the query keeps the latest row per id and returns the 10 current customers.

Files only append, so the queries above return every row from every file: 12 customer rows and 48 order rows after the second sync. The query that most readers want is the latest version of each row. QUALIFY does that in one statement, ordering by the file time OLake Go stored in _last_modified_time:

Latest row per id
SELECT CAST(id AS INT) AS id, name, city, tier
FROM lakehouse.s3_to_databricks_shop_s3.customers
QUALIFY ROW_NUMBER() OVER (PARTITION BY id ORDER BY _last_modified_time DESC) = 1
ORDER BY id;

SELECT status, count(*) AS n, round(sum(amount), 2) AS total
FROM (
  SELECT * FROM lakehouse.s3_to_databricks_shop_s3.orders
  QUALIFY ROW_NUMBER() OVER (PARTITION BY id ORDER BY _last_modified_time DESC) = 1
)
GROUP BY status ORDER BY status;

It returned 10 customers, and 45 orders worth 25579.19 in total, on the 2X-Small serverless warehouse.

Step 7: Add files to the bucket and sync again​

To see the incremental path work, the run on this page uploaded one more file per folder after the first sync. Each carries new rows and corrected copies of existing rows, the way a daily export does:

Files added between the two syncs
data/customers/2026-10-29/customers-changes.csv   4 rows: ids 9 and 10 are new, ids 3 and 8 are corrections
data/orders/2026-10-29/orders-changes.csv         8 rows: ids 41 to 45 are new, ids 1, 4 and 7 now read "delivered"

The second sync listed both folders, skipped the two files it had already read, and read the two new ones. That was 12 records (686 bytes) in 17 seconds end to end. OLake Go's commit report shows operation=append with one data file per table, 4 rows for customers and 8 for orders, and no delete files.

OLake UI sync log filtered to the Iceberg commit reports for customers and orders after the second sync, showing operation append, addedDataFiles 1, addedRecords 4 and 8, and totalDeleteFiles 0
The second sync in the OLake UI log: one appended data file per table and no delete files.

Databricks shows the result on the next query, with no refresh. The corrected rows exist twice, and _op_type tells them apart: r for the 8 rows of the first full read, u for the 4 that arrived through the incremental sync. The latest-row query from step 6 returns Liam Carter as gold and Zara Ali in Abu Dhabi.

Schema changes. When a new column appears in a later file, OLake Go adds it to the Iceberg table on the next sync. Databricks shows it with nulls for older rows.

A column whose inferred type changes in an incompatible way fails the sync rather than corrupt the table. Full rules are on the schema evolution page.

Query the same tables from DuckDB and Spark​

Nothing is exported, copied or refreshed. Both engines read the files OLake Go wrote, through the same Unity Catalog endpoint, with storage credentials vended by Databricks. The same QUALIFY pattern works in DuckDB; Spark uses a ROW_NUMBER() subquery.

DuckDB attaches the Unity Catalog catalog as an Iceberg REST catalog with the same token. With vended_credentials, Unity Catalog hands DuckDB short-lived storage credentials per table, so no cloud keys are configured on the DuckDB side.

DuckDB 1.5.5
INSTALL iceberg; LOAD iceberg;
INSTALL httpfs;  LOAD httpfs;
CREATE SECRET unity (TYPE iceberg, TOKEN '<personal access token>');
ATTACH 'lakehouse' AS uc (
TYPE iceberg,
ENDPOINT 'https://<workspace-url>/api/2.1/unity-catalog/iceberg-rest',
SECRET unity,
ACCESS_DELEGATION_MODE 'vended_credentials'
);
SELECT id, name, city, tier
FROM uc.s3_to_databricks_shop_s3.customers
ORDER BY id;

DuckDB reads Iceberg v3 deletion vectors from duckdb-iceberg 1.5.3, so the rows match Databricks after every sync. The principal behind the token needs EXTERNAL USE SCHEMA, the same as OLake Go.

DuckDB 1.5.5 returned the same 12 customer rows as Databricks, and the same 10 with QUALIFY ROW_NUMBER() OVER (PARTITION BY id ORDER BY _last_modified_time DESC) = 1.

An orders roll-up over the latest row per id matched the files in all three engines. It gave 16 delivered orders for 8659.68, 18 placed for 10522.26, 11 shipped for 6397.25.

Data types: CSV to Iceberg to Databricks​

A CSV file has no types, so OLake Go infers them from a sample of rows at discover time and maps them to Iceberg types. Databricks then maps Iceberg types to its own. Parquet files skip the inference and keep their own types.

In the CSVIceberg (written by OLake Go)Databricks showsNotes
Whole numbers (id, customer_id)doubleDOUBLEIn this run the integer columns were inferred as floating point. Cast with CAST(id AS INT) in queries, or use Parquet when the type matters.
Decimals (amount)doubleDOUBLEThe orders.amount sums on this page matched the files to the cent in every engine; round at query time when you need exact decimal arithmetic.
ISO 8601 timestamps (created_at)timestamptz (UTC)TIMESTAMPStrings that parse as timestamps are stored as timestamps.
Other textstringSTRING
_last_modified_timestringSTRINGThe object's LastModified time in ISO 8601, so it sorts correctly as text. Added to every row by OLake Go.

Types on the left come from the S3 connector reference; the Databricks side comes from DESCRIBE on the tables of this run. JSON files map integers to bigint; Parquet keeps int, bigint, date and timestamp as they are.

Cost and maintenance​

  • Storage is your cloud bucket behind the catalog's managed location. You pay your cloud's storage rates for the Parquet data files and deletion vectors.
  • Compute. OLake Go runs wherever you run Docker and does not use a Databricks warehouse or cluster to write. Databricks compute is spent only when you query; the checks on this page ran on a 2X-Small serverless SQL warehouse.
  • Maintenance. Each sync adds a small data file, plus a deletion vector when rows change. Plan compaction from the writer side: OLake Fusion does this for the catalogs OLake Go writes to, and Spark's rewrite_data_files works through the same Unity Catalog endpoint.
  • Token rotation. A personal access token expires on the date set when it was created, and an expired token fails the next sync at the first catalog call. Use a service principal with OAuth machine-to-machine credentials for production jobs.
  • Listing calls. Every sync lists the whole prefix to find new files. A prefix with hundreds of thousands of objects makes that the slowest part of the run; keep the prefix to the folders OLake Go needs, or move processed files out.

Troubleshooting​

1. The source test is refused​

failed to access bucket shop: operation error S3: HeadBucket, https response error StatusCode: 403, api error Forbidden: Forbidden

The key pair cannot read the bucket. On AWS, check that the policy from step 1 is attached to the user or role, and that the bucket policy does not deny it.

On MinIO, the access key or secret is wrong. A StatusCode: 404 with NotFound instead means the bucket name is wrong or the endpoint points at another store.

2. Discovery finds nothing​

Discovered 0 files in 0 streams (after filtering)
Stream grouping enabled at level 1 (first folder after path_prefix)
FATAL no streams found in connector

Either the prefix holds no files, or the files sit directly under it with no folder to name the stream. Move them into a folder per table (data/customers/...), check that Path Prefix ends with /, and loosen File Pattern if you set one.

3. Rows appear twice after a sync​

A file was rewritten in place, or a corrected copy was uploaded with the same rows. Its new LastModified time made OLake Go read it again, and Append mode added every row a second time. Keep the latest-row query from step 6 for reads, and change the upstream process to add files rather than replace them.

4. The destination test fails with a permission error​

Three settings cover nearly every failure. External data access may be off on the metastore, or the principal may lack EXTERNAL USE SCHEMA on the catalog. The third is a wrong catalog name in the destination.

ALL PRIVILEGES does not include EXTERNAL USE SCHEMA, so a catalog owner still has to grant it explicitly. Run the grants from step 2 again for the principal that owns the token.

How this compares with other ways to load S3 files into Databricks
MethodFreshnessUpdates and deletesWhere the data landsWhat you run
Auto Loader (checked Oct 2026)Minutes, or continuous while the stream runsAppends new files; merge with your own codeTables written by a Databricks job or pipelineA Databricks job or pipeline running on Databricks compute
Lakeflow ConnectS3 is not among its managed file connectors, which list Google Drive and SharePoint (checked Oct 2026)Not applicableNot applicableNot applicable
OLake Go to Iceberg on Unity Catalog (this guide)On a schedule down to one minuteNo; files are appended, with _last_modified_time for latest-row queriesApache Iceberg tables that Databricks, DuckDB, Spark and Trino all readOne Docker Compose stack, open source; your storage plus your own compute

Auto Loader is the simpler choice when Databricks is the only reader and a job on Databricks compute is fine. OLake Go keeps the load off Databricks compute, and it keeps the pipeline the same when a database or Kafka source is added beside the files.

When not to use this​

  • Rows must be updated or deleted at write time. Files have no key, so S3 streams only append. Use a database or Kafka source for that, or merge downstream.
  • The files are rewritten in place. Every rewrite is read again in full. Change the producer to add files, or accept duplicates and query the latest row per key.
  • Files are not CSV, JSON, Parquet or XML. Fixed-width, Excel and Avro files are not read.

FAQs​

How do I load files from S3 into Databricks as Iceberg tables without Auto Loader?

Run OLake Go (open source, Apache 2.0) with the bucket as the source and Unity Catalog's Iceberg REST endpoint as the destination. Each folder under the prefix becomes a Unity Catalog managed Iceberg table, the first sync reads every file, and every later sync reads only files newer than the last run. No Databricks job or cluster runs during the load.

Does OLake Go re-read the whole bucket on every sync?

No. OLake Go saves the newest LastModified time it has read per stream and, on the next sync, lists the prefix and reads only files modified after it. The second sync on this page read only the two new files, 12 records in total. A file that is rewritten in place gets a new LastModified time and is read again in full.

Can OLake Go update or delete rows from S3 files?

No. Files carry no primary key or change log, so S3 streams run in Append mode only: every row in every new file becomes a row in the table. When a later file corrects a row, both versions exist. Keep the latest with a QUALIFY ROW_NUMBER() query over the id and _last_modified_time columns, or merge downstream.

Which file formats does OLake Go read from S3?

CSV (with delimiter, header and quote options), JSON as line-delimited or array files, Parquet, and XML with a row identifier, each plain or gzip-compressed. One source handles one format; a file pattern narrows the keys when a prefix holds more than one. Parquet keeps its own types; CSV types are inferred from a sample of rows.

Does this work with MinIO, LocalStack or another S3-compatible store?

Yes. Set the custom endpoint on the source and provide static keys. The run on this page used MinIO in Docker with the endpoint http://host.docker.internal:9000. On AWS S3 itself the endpoint stays empty and the keys can be left out to use an instance or task role.

How are the Databricks tables named for an S3 source?

After the folder. The first folder under the path prefix is the stream, so data/customers/ becomes a table called customers. The schema defaults to the job name, the bucket name and the namespace s3; the job s3_to_databricks on bucket shop produced s3_to_databricks_shop_s3. The Iceberg DB field on the Streams page changes it before the first sync.

S3 to SnowflakeSame source, read from Snowflake.S3 to BigQuerySame source, read from BigQuery.S3 to Amazon RedshiftSame source, read from Amazon Redshift.Postgres to DatabricksSame engine, replicated from Postgres.MySQL to DatabricksSame engine, replicated from MySQL.MariaDB to DatabricksSame engine, replicated from MariaDB.MongoDB to DatabricksSame engine, replicated from MongoDB.Oracle to DatabricksSame engine, replicated from Oracle.SQL Server to DatabricksSame engine, replicated from SQL Server.Kafka to DatabricksSame engine, replicated from Kafka.S3 connector referenceEvery field, sync mode and troubleshooting note for the source.Query engine compatibilityWhich engines read which catalogs and delete formats.All integrationsEvery source and engine pair with a tested guide.

Deeper reading on this site: