Skip to main content

SQL Server to Databricks: CDC via Apache Iceberg

SQL Server to Databricks usually means the Lakeflow Connect SQL Server connector, a Kafka pipeline, or a nightly bcp export picked up by Auto Loader. This guide takes a different route. SQL Server's own change data capture (CDC) feature fills change tables from the transaction log, and OLake Go commits those rows as Apache Iceberg tables into Unity Catalog.

The tables land as Unity Catalog managed Iceberg tables, with the files in your own cloud storage. Nothing else is in the path: no Kafka, no Debezium, no Databricks ingestion pipeline. DuckDB and Spark read the same tables through the same catalog.

By Shubham Satish BaldavaUpdated October 30, 2026

Key takeaways

  • OLake Go replicates SQL Server into Databricks as Iceberg tables: a full historical snapshot first, then CDC from the change tables SQL Server already maintains, with no Kafka or Debezium in between.
  • The tables are Unity Catalog managed Iceberg tables. You query them from the Databricks SQL editor or a notebook right away, with no external table and no refresh.
  • The same tables, written once by OLake Go, are also queryable from DuckDB and Spark through Unity Catalog. In the run on this page all three engines returned identical rows.

The short answer. To replicate SQL Server to Databricks with change data capture (CDC), enable CDC on the database and tables. Run OLake Go with a read-only login as the source and Unity Catalog as the Iceberg destination, with Databricks as a target query engine.

The first sync copies each table in full; every later sync replays inserts, updates and deletes from the change tables with deletion vectors. Databricks, DuckDB and Spark then query the same Iceberg tables.

How SQL Server CDC reaches Databricks here​

The first sync records the current log sequence number (LSN), then reads each table in parallel chunks and writes Parquet data files: that is the historical load. Every later sync reads the change tables from the saved LSN and replays the inserts, updates and deletes since. A deletion vector, a small bitmap per data file, retires the old version of each changed row.

Each sync ends with one commit to Unity Catalog. Databricks sees the new snapshot on its next query, and DuckDB and Spark see it on their next catalog load.

The change tables are SQL Server's replay window, and the Agent is what fills them. The cleanup job drops change rows older than three days by default, so a job paused longer than that needs a fresh snapshot. A stopped Agent means no new change rows at all, with no error on the SQL Server side.

What you need​

  • SQL Server 2016 SP1 or later on Developer, Standard or Enterprise edition, with CDC available and the SQL Server Agent running. The tables on this page ran on SQL Server 2022 Developer in Docker.
  • A Databricks workspace with Unity Catalog, a catalog with managed storage, and an admin who can turn on external data access and grant EXTERNAL USE SCHEMA. The run on this page used a workspace on Google Cloud.
  • OLake UI from the Docker Compose install with the OLake Go MSSQL source. The run on this page used v1.1.0.
  • Docker, and about 40 minutes.

Step 1: Prepare SQL Server for CDC​

Most readers point OLake Go at tables they already have. If you want to follow along with the exact data from this page, create these two tables first:

The two sample tables used on this page
Sample schema, database shop
CREATE TABLE dbo.customers (
  id INT IDENTITY(1,1) PRIMARY KEY,
  name NVARCHAR(100), email NVARCHAR(200), city NVARCHAR(100), tier NVARCHAR(20),
  created_at DATETIME2 DEFAULT SYSUTCDATETIME(),
  updated_at DATETIME2 DEFAULT SYSUTCDATETIME());

CREATE TABLE dbo.orders (
  id INT IDENTITY(1,1) PRIMARY KEY,
  customer_id INT REFERENCES dbo.customers(id),
  product NVARCHAR(100), amount DECIMAL(10,2), status NVARCHAR(20),
  created_at DATETIME2 DEFAULT SYSUTCDATETIME(),
  updated_at DATETIME2 DEFAULT SYSUTCDATETIME());

-- Then insert whatever you like. This page started with 8 customers and 40 orders.

OLake Go reads SQL Server changes from the change tables that SQL Server's own change data capture (CDC) feature fills from the transaction log. Four things must be in place before the first sync.

SettingValueWhy it matters
CDC on the databasesys.sp_cdc_enable_dbCreates the cdc schema and the capture jobs for the database.
CDC on each tablesys.sp_cdc_enable_table per tableCreates the capture instance whose change table OLake Go reads.
SQL Server AgentrunningThe capture job that copies log records into the change tables is an Agent job; without it nothing lands in cdc.*. On Amazon RDS and Azure SQL Managed Instance the agent is on by default.
A login for OLake GoSELECT on the table schema, on the cdc schema and on msdb.dbo.cdc_jobsThe snapshot reads the tables; the change stream reads the change tables; the source test reads the capture job definition. Without the msdb grant the test fails with no SELECT on msdb.dbo.cdc_jobs.
On the SQL Server database
-- 1. CDC on the database (Developer, Standard 2016 SP1+ or Enterprise edition)
USE shop;
EXEC sys.sp_cdc_enable_db;

-- 2. CDC on every table you sync. @role_name = NULL skips the gating role.
EXEC sys.sp_cdc_enable_table @source_schema = N'dbo', @source_name = N'customers', @role_name = NULL;
EXEC sys.sp_cdc_enable_table @source_schema = N'dbo', @source_name = N'orders',    @role_name = NULL;

-- 3. A login for OLake Go
USE master;
CREATE LOGIN olake WITH PASSWORD = '<password>';
USE shop;
CREATE USER olake FOR LOGIN olake;
GRANT SELECT ON SCHEMA :: dbo TO olake;
GRANT SELECT ON SCHEMA :: cdc TO olake;
GRANT VIEW DATABASE STATE TO olake;

-- 4. OLake Go checks the capture job definition in msdb before each sync
USE msdb;
CREATE USER olake FOR LOGIN olake;
GRANT SELECT ON dbo.cdc_jobs TO olake;

Check the result before moving on:

sql
SELECT name, is_cdc_enabled FROM sys.databases WHERE name = 'shop';   -- 1
SELECT capture_instance FROM cdc.change_tables;                        -- dbo_customers, dbo_orders
EXEC msdb.dbo.sp_help_job @job_name = N'cdc.shop_capture';             -- the Agent job exists

Hosted SQL Server uses its own procedure to enable CDC on the database: EXEC msdb.dbo.rds_cdc_enable_db 'shop' on Amazon RDS, EXEC msdb.dbo.gcloudsql_cdc_enable_db 'shop' on Cloud SQL; the table step is the same. The full walk-through is the SQL Server connector page.

Change tables keep three days by default

The cleanup job removes change rows older than three days (sys.sp_cdc_change_job @job_type = 'cleanup' changes it). A job paused longer than that loses its position and needs a fresh snapshot. Lengthen retention before a planned pause.

Step 2: Set up Unity Catalog in Databricks​

Databricks reads Iceberg tables through Unity Catalog, the catalog built into every Databricks workspace. OLake Go writes straight into it, so the tables become Unity Catalog managed Iceberg tables. They show up in Catalog Explorer and the SQL editor the moment a sync commits.

The files sit in the catalog's managed storage, which is backed by an external location in your own cloud account. DuckDB and Spark reach the same tables through the same Unity Catalog endpoint.

CatalogOLake Go writesDatabricks readsVerified on this page
Unity CatalogYesYes: native: Unity Catalog managed Iceberg tablesYes, end to end

Unity Catalog is the catalog built into every Databricks workspace. OLake Go writes to it through Unity Catalog's Iceberg REST (representational state transfer) endpoint at https://<workspace-url>/api/2.1/unity-catalog/iceberg-rest, and the tables it creates are Unity Catalog managed Iceberg tables. Databricks allows writes through this endpoint for managed Iceberg tables only (Databricks: access tables from Iceberg clients, checked 2026-10-30).

Four things must be in place in the workspace, once:

SettingWhereWhy it matters
External data access on the metastoreCatalog, the gear icon, Metastore, then External data accessWithout it the Iceberg REST endpoint refuses every client that is not Databricks itself.
A catalog with managed storageCatalog, Create catalog, with a storage location backed by an external locationOLake Go creates one schema per job inside it. The catalog name is the warehouse value every client uses.
EXTERNAL USE SCHEMA for the writing principalOn the catalog, so it covers the schemas OLake Go createsUnity Catalog checks this privilege, on top of the usual ones, before it serves a table to an outside engine.
A personal access token or a service principalSettings, Developer, Access tokensOLake Go sends it as a bearer token on every catalog call.
Databricks SQL editor, once per catalog (replace the principal)
-- The catalog OLake Go writes into. Its managed storage holds the Iceberg files.
CREATE CATALOG IF NOT EXISTS lakehouse MANAGED LOCATION '<external location URL>/lakehouse';

-- What the writing principal needs: browse and create schemas, then read and write tables in them
GRANT USE CATALOG, CREATE SCHEMA ON CATALOG lakehouse TO `olake@example.com`;
GRANT USE SCHEMA, CREATE TABLE, SELECT, MODIFY ON CATALOG lakehouse TO `olake@example.com`;
GRANT EXTERNAL USE SCHEMA ON CATALOG lakehouse TO `olake@example.com`;

Granting on the catalog makes the privileges inherit to every schema OLake Go creates later. EXTERNAL USE SCHEMA is not part of ALL PRIVILEGES, and only the catalog owner can grant it (Databricks: Unity Catalog privileges, checked 2026-10-30). A service principal with OAuth machine-to-machine credentials is the better choice in production. The other authentication methods are on the Unity Catalog destination page.

Databricks reads deletion vectors only

Unity Catalog accepts updates to a managed Iceberg table only as copy-on-write files or as Iceberg v3 deletion vectors. It rejects equality delete files with Can not create a Path from a null string [ErrorCode: 2015]. Set Databricks as a target query engine on the job, and OLake Go writes deletion vectors from the first sync.

Step 3: Connect SQL Server to OLake Go​

  1. In the OLake UI, open Sources and click Create Source. Pick MSSQL as the connector, give the source a name and keep the OLake Go version on the latest release.
  2. Under Change data capture (CDC), leave Enable CDC on. Turn on Manage Capture Instance if you want OLake Go to create a new capture instance by itself after a schema change (it then needs db_owner); leave it off to manage capture instances by hand.
  3. Under Connection details, fill in the host, port, database name, username and password. Keep SSL off for a local test; on hosted SQL Server use the provider's certificate.
  4. Click Set up source. OLake Go tests the connection, then checks that CDC is on for the database and that the tables have capture instances, and shows the current and recommended value for anything that fails.

Here is the source form from the run on this page. SQL Server ran in another container, so the host is host.docker.internal; the login is the one from step 1, with VIEW DATABASE STATE and SELECT on dbo and cdc.

OLake UI Edit source form for MSSQL on OLake Go v1.1.0 with Enable CDC on, host host.docker.internal, port 1433, database shop, user olake and SSL disabled
The SQL Server source in OLake UI: Enable CDC on, with the login from step 1.

Set up source runs the connection test and the CDC checks before anything is saved: CDC on the database, a capture instance per table, a readable capture job. When one fails, the dialog names the setting and what to run.

The full field reference, including SSL modes and SSH tunnels, is on the SQL Server connector page.

Step 4: Configure the Iceberg destination​

  1. Open Destinations and click Create Destination.
  2. Pick Apache Iceberg as the connector and Unity as the catalog type.
  3. Choose Personal Access Token as the authentication type and fill in the fields below. The catalog name from the setup goes into the catalog field.
  4. Click Create. OLake Go calls the Unity Catalog endpoint with the token and saves the destination.
OLake UI destination form for an Apache Iceberg destination on the Unity catalog type with the personal access token fields

The run on this page wrote into a catalog called datazip, with a personal access token that expires after seven days. The snippets below use lakehouse; replace it with your catalog name.

Unity Configuration Parameters

ParameterSample ValueDescription
REST Catalog URL
required		https://1234567890123456.1.gcp.databricks.com/Databricks workspace URL with Unity Catalog REST API endpoint. Use your actual workspace URL.
Authentication Type
required		Personal Access TokenAuthentication method OLake Go uses for Unity Catalog REST API requests. Set to match your workspace's authentication configuration.
Unity Catalog Name
required		workspaceName of the catalog in Unity.
Token
required		dapi123456789...Databricks Personal Access Token
Catalog Nameolake_icebergName of the Iceberg catalog OLake Go registers tables under. Defaults to olake_iceberg if left empty.
Enable Arrow WritesfalseWrites data and delete files using Apache Arrow based writer and registers them in Iceberg.
Catalog Name Supported for v0.3.5 and above
For the catalog name, OLake Go only supports lowercase letters and underscores. Spaces and special characters are not supported.

Step 5: Create the job and run the first sync​

  1. Open Jobs and click Create Job. Give it a name and a schedule. Every Day at a fixed hour, or a cron expression, both work; the schedule is how often the change stream is drained, not how often a full copy runs.
  2. Select the source and the destination you created.
  3. Under Advanced Settings, set Target Query Engines to Databricks, plus DuckDB and Apache Spark if they will read the tables too. OLake Go then resolves an Iceberg delete format every one of those engines can read, so you never pick one by hand.
  4. Click Next. OLake Go discovers the tables and opens the Streams page.
  5. Tick the tables to replicate. For each one, set the sync mode to Full Refresh + CDC and the ingestion mode to Upsert. Leave the Upsert Type as OLake Go resolved it (Deletion Vector in the run on this page).
  6. Click Create Job, then open the job's menu and choose Sync Now for the first run.

Three target query engines were set on this job, and Databricks narrows the delete formats to one: deletion vectors on Iceberg v3 tables. Leave systranschemas unticked; troubleshooting entry 2 explains why.

OLake UI Streams Selection page with the dbo schema, customers and orders selected and systranschemas unticked, sync mode Full Refresh + CDC, ingestion mode Upsert and upsert type Deletion Vector on Iceberg format version V3
Each stream: Full Refresh + CDC, Upsert, and Deletion Vector on Iceberg v3, the format OLake Go resolved for Databricks.
OLake UI Job Logs and History for sql_server_to_databricks showing two completed syncs, the first in 27 seconds and the second in 20 seconds
Job history: the 27 s historical snapshot, then the 20 s change-only sync.

On this page the first run copied 8 customers and 40 orders (48 records, 3.05 KB) and finished in 27 seconds, container start included. OLake Go names the Unity Catalog schema after the job, the source database and the schema, so sql_server_to_databricks on database shop produced sql_server_to_databricks_shop_dbo. The Iceberg DB field at the top of the Streams page changes it before the first sync.

Step 6: Query the tables in Databricks​

There is no registration step. The schema and tables OLake Go committed through Unity Catalog are managed Iceberg tables in the catalog you configured, and the Databricks SQL editor or a notebook reads them by their three-part name.

Databricks SQL editor
-- What OLake Go created (schema = <job name>_<source database>[_<source schema>])
SHOW TABLES IN lakehouse.sql_server_to_databricks_shop_dbo;

SELECT id, name, city, tier
FROM lakehouse.sql_server_to_databricks_shop_dbo.customers
ORDER BY id;

SELECT status, count(*) AS n, round(sum(amount), 2) AS total
FROM lakehouse.sql_server_to_databricks_shop_dbo.orders
GROUP BY status ORDER BY status;

Any SQL warehouse or cluster on a recent Databricks Runtime reads them; a 2X-Small serverless warehouse is enough for checks. DESCRIBE EXTENDED on a table shows the Iceberg provider and the deletion vectors in its properties.

OLake Go adds four columns to every table: _olake_id (a hash of the primary key), _op_type, _olake_timestamp and _cdc_timestamp. _op_type is r for a row from the first snapshot, c for an insert, u for an update and d for a delete. Filter them out of dashboards, or use them to see when a row last changed.

Databricks SQL editor running SELECT on datazip.sql_server_to_databricks_shop_dbo.customers on the 2X-Small SQL warehouse, returning 10 rows with id, name, city and tier
The SQL Server customers table in the Databricks SQL editor after both syncs: 10 rows, read through Unity Catalog.

Step 7: Change rows in SQL Server and sync again​

To see the change path work, the run on this page made 12 changes in SQL Server after the first sync:

Changes made in SQL Server between the two syncs
INSERT INTO dbo.customers (name, email, city, tier) VALUES
  (N'Maya Patel', 'maya@example.com', N'Pune', 'silver'),
  (N'Lucas Silva', 'lucas@example.com', N'São Paulo', 'bronze');
INSERT INTO dbo.orders (customer_id, product, amount, status) VALUES
  (9, 'Laptop', 1299.00, 'placed'), (9, 'Mouse', 25.50, 'placed'),
  (10, 'Monitor', 349.99, 'placed'), (3, 'Keyboard', 89.00, 'placed'), (7, 'Webcam', 79.90, 'placed');
UPDATE dbo.customers SET tier = 'gold',       updated_at = SYSUTCDATETIME() WHERE name = N'Liam Carter';
UPDATE dbo.customers SET city = N'Abu Dhabi', updated_at = SYSUTCDATETIME() WHERE name = N'Zara Ali';
UPDATE dbo.orders    SET status = 'delivered', updated_at = SYSUTCDATETIME() WHERE id IN (1, 4, 7);

The second sync read 12 records (855 bytes) from the change tables and finished in 20 seconds end to end. For customers that meant one new data file with 4 rows and one deletion vector marking the 2 old versions. OLake Go's commit report shows addedDataFiles=1, addedRecords=4, addedDVs=1 and addedPositionalDeletes=2, and orders the same shape with 8 rows and 3 deleted positions.

OLake UI sync log filtered to the Iceberg commit reports for orders and customers after the change-only sync, showing addedDataFiles 1, addedDVs 1, addedRecords 8 and 4, and addedPositionalDeletes 3 and 2
The change-only sync in the OLake UI log: one data file and one deletion vector per table.

Databricks shows the result on the next query, with no refresh. Liam Carter is gold, Zara Ali moved to Abu Dhabi, and the two new customers are there. The _op_type column shows how each row arrived: r for the 6 untouched snapshot rows, c for the 2 inserts and u for the 2 updates.

Schema changes. A column added in SQL Server is not part of the existing capture instance. Create a new capture instance for the table, or let OLake Go do it with Manage Capture Instance on the source.

OLake Go then adds the column to the Iceberg table on the next sync. Full rules are on the schema evolution page.

Query the same tables from DuckDB and Spark​

Nothing is exported, copied or refreshed. Both engines read the files OLake Go wrote, through the same Unity Catalog endpoint, with storage credentials vended by Databricks.

DuckDB attaches the Unity Catalog catalog as an Iceberg REST catalog with the same token. With vended_credentials, Unity Catalog hands DuckDB short-lived storage credentials per table, so no cloud keys are configured on the DuckDB side.

DuckDB 1.5.5
INSTALL iceberg; LOAD iceberg;
INSTALL httpfs;  LOAD httpfs;
CREATE SECRET unity (TYPE iceberg, TOKEN '<personal access token>');
ATTACH 'lakehouse' AS uc (
TYPE iceberg,
ENDPOINT 'https://<workspace-url>/api/2.1/unity-catalog/iceberg-rest',
SECRET unity,
ACCESS_DELEGATION_MODE 'vended_credentials'
);
SELECT id, name, city, tier
FROM uc.sql_server_to_databricks_shop_dbo.customers
ORDER BY id;

DuckDB reads Iceberg v3 deletion vectors from duckdb-iceberg 1.5.3, so the rows match Databricks after every sync. The principal behind the token needs EXTERNAL USE SCHEMA, the same as OLake Go.

DuckDB 1.5.5 returned the same 10 customers and the same orders roll-up as Databricks in the run on this page.

An orders roll-up matched SQL Server to the cent in all three engines. It gave 16 delivered orders for 8659.68, 18 placed for 10522.26, 11 shipped for 6397.25.

Data types: SQL Server to Iceberg to Databricks​

OLake Go maps SQL Server types to Iceberg types, and Databricks maps Iceberg types to its own. Numbers with a scale and the date types are the groups to know about.

SQL ServerIceberg (written by OLake Go)Databricks showsNotes
tinyint, smallint, intintINTThe IDENTITY keys on this page arrived as INT.
bigintlongBIGINT
decimal, numeric, money, smallmoney, floatdoubleDOUBLEThe orders.amount sums on this page matched SQL Server to the cent; round at query time when you need exact decimal arithmetic.
realfloatFLOAT
bitbooleanBOOLEAN
date, smalldatetime, datetime, datetime2, datetimeoffsettimestamptz (UTC)TIMESTAMPdatetime2 has no zone in SQL Server; it is read as UTC.
char, varchar, nchar, nvarchar, text, ntext, uniqueidentifier, xml, json, sql_variantstringSTRINGA uniqueidentifier arrives as its 36-character text form.
binary, varbinary, image, rowversion, geometry, geography, hierarchyidstringSTRINGBinary and spatial values arrive as text; keep large binaries out of the sync.

SQL Server types in the table come from the SQL Server connector reference; the Databricks side comes from DESCRIBE on the tables of this run. OLake Go adds _cdc_start_lsn and _cdc_seqval on SQL Server tables, the log position each change came from.

Cost and maintenance​

  • Storage is your cloud bucket behind the catalog's managed location. You pay your cloud's storage rates for the Parquet data files and deletion vectors.
  • Compute. OLake Go runs wherever you run Docker and does not use a Databricks warehouse or cluster to write. Databricks compute is spent only when you query; the checks on this page ran on a 2X-Small serverless SQL warehouse.
  • Maintenance. Each sync adds a small data file, plus a deletion vector when rows change. Plan compaction from the writer side: OLake Fusion does this for the catalogs OLake Go writes to, and Spark's rewrite_data_files works through the same Unity Catalog endpoint.
  • Token rotation. A personal access token expires on the date set when it was created, and an expired token fails the next sync at the first catalog call. Use a service principal with OAuth machine-to-machine credentials for production jobs.
  • Change table disk. Every captured change is a row in the cdc schema until the cleanup job removes it. Shorten retention once the job runs reliably every hour.

Troubleshooting​

1. The source test fails on the capture job​

required CDC prerequisites not met: cdc_capture_job ... no SELECT on msdb.dbo.cdc_jobs

OLake Go reads the capture job definition from msdb to pick a safe start position. Create a user for the login in msdb and grant SELECT ON dbo.cdc_jobs, as in step 1.

2. The first sync fails on a table you did not create​

failed to run change stream: failed in pre cdc run for driver[mssql]: CDC is not enabled for stream dbo.systranschemas

Enabling CDC or replication on a database adds dbo.systranschemas to it, and discovery lists it next to your tables. It has no capture instance, so a job that selects every stream fails before the snapshot. Untick it on the Streams page, along with any other table without a capture instance.

3. The source test passes but no changes arrive​

Two causes. The common one is timing: the capture job copies log records into the change tables every few seconds, so a sync started right after a write can read nothing and the next sync picks the rows up.

The other cause is that the SQL Server Agent is not running, so the change tables never fill. EXEC master.dbo.xp_servicecontrol 'QueryState', 'SQLServerAgent' reports Running. when it is up.

4. The destination test fails with a permission error​

Three settings cover nearly every failure. External data access may be off on the metastore, or the principal may lack EXTERNAL USE SCHEMA on the catalog. The third is a wrong catalog name in the destination.

ALL PRIVILEGES does not include EXTERNAL USE SCHEMA, so a catalog owner still has to grant it explicitly. Run the grants from step 2 again for the principal that owns the token.

5. Databricks rejects a commit with "Can not create a Path from a null string"​

Can not create a Path from a null string [ErrorCode: 2015]

Unity Catalog returns this when a client commits equality delete files to a managed Iceberg table, per Databricks: access tables from Iceberg clients (checked Oct 2026). Set Databricks as a target query engine on the job, so OLake Go writes deletion vectors, and start any table that was written with another delete format again from a full refresh.

How this compares with other ways to move SQL Server data to Databricks
MethodFreshnessUpdates and deletesWhere the data landsWhat you run
Lakeflow Connect SQL Server connectorIncremental, through SQL Server change tracking or CDC (checked Oct 2026)YesTables managed by a Databricks ingestion pipelineA Lakeflow Connect pipeline, billed on Databricks compute
Debezium, Kafka and Databricks Structured Streaming (checked Oct 2026)SecondsYes, with your own merge logicKafka topics, then tables written by a Databricks streaming jobKafka, Kafka Connect, the Debezium SQL Server connector and a running Databricks job
Export (bcp to files), then Auto Loader (checked Oct 2026)Hours to dailyNo; full reloads or your own mergeTables written by a Databricks pipelineAn export script, a bucket and a Databricks job
OLake Go to Iceberg on Unity Catalog (this guide)CDC on a schedule down to one minuteYes, as Iceberg deletion vectorsApache Iceberg tables that Databricks, DuckDB, Spark and Trino all readOne Docker Compose stack, open source; your storage plus your own compute

Lakeflow Connect is the simpler choice when Databricks is the only reader and managed ingestion is the goal. Writing Iceberg once avoids a second copy as soon as another engine needs the same rows, and keeps the ingestion off Databricks compute.

When not to use this​

  • The edition has no CDC. SQL Server Express and Web cannot enable change data capture. Full Refresh and Full Refresh + Incremental on an updated_at column still work there.
  • Tables have no primary key. Inserts replicate; updates and deletes cannot be matched to a row and are not applied. Add a primary key or a unique index first.
  • You want managed ingestion billed on Databricks. Lakeflow Connect runs inside the workspace and needs no host of your own.

FAQs​

How do I replicate SQL Server to Databricks with CDC?

Enable change data capture on the database and tables, then run OLake Go (open source, Apache 2.0) with a read-only login as the source and Unity Catalog's Iceberg REST endpoint as the destination. The first sync copies the tables in full; every later sync reads only the rows SQL Server's capture job wrote to the change tables. The tables appear in Databricks as Unity Catalog managed Iceberg tables.

How is this different from the Lakeflow Connect SQL Server connector?

Lakeflow Connect is Databricks' managed ingestion, and its overview lists SQL Server among its CDC database connectors (checked Oct 2026). It writes tables for Databricks to read. OLake Go writes open Iceberg tables once into Unity Catalog, so Databricks, DuckDB, Spark and Trino read the same files without a second copy, and the ingestion runs on your own Docker host.

Which Iceberg delete format does OLake Go write for Databricks?

Deletion vectors, on Iceberg v3 tables. Under Unity Catalog, Databricks reads deletion vectors but not Iceberg v2 positional or equality deletes. Setting Databricks as a target query engine on the job makes OLake Go pick deletion vectors from the first sync.

What does SQL Server need for OLake Go CDC?

Change data capture on the database and on each table, which needs Developer, Standard (2016 SP1 or later) or Enterprise edition, plus a running SQL Server Agent to fill the change tables. The OLake Go login needs SELECT on the table schema, on the cdc schema and on msdb.dbo.cdc_jobs.

Which SQL Server data types change on the way to Databricks?

decimal, numeric, money and float columns become Iceberg double and show up in Databricks as DOUBLE, so round at query time when you need exact decimal arithmetic. datetime, datetime2, smalldatetime and date become TIMESTAMP in UTC. nvarchar, varchar, uniqueidentifier, xml and the binary types arrive as STRING. int stays INT and bigint stays BIGINT.

How long does SQL Server keep change rows for OLake Go?

Three days by default. The CDC cleanup job removes change rows older than its retention setting, and a job paused for longer than that cannot resume and needs a fresh snapshot. Raise retention with sys.sp_cdc_change_job before a planned pause.

SQL Server to SnowflakeSame source, read from Snowflake.SQL Server to ClickHouseSame source, read from ClickHouse.SQL Server to BigQuerySame source, read from BigQuery.SQL Server to Amazon RedshiftSame source, read from Amazon Redshift.SQL Server to Apache IcebergSame source, read from Apache Iceberg.Postgres to DatabricksSame engine, replicated from Postgres.MySQL to DatabricksSame engine, replicated from MySQL.MariaDB to DatabricksSame engine, replicated from MariaDB.MongoDB to DatabricksSame engine, replicated from MongoDB.Oracle to DatabricksSame engine, replicated from Oracle.Kafka to DatabricksSame engine, replicated from Kafka.S3 to DatabricksSame engine, replicated from S3.SQL Server connector referenceEvery field, sync mode and troubleshooting note for the source.Query engine compatibilityWhich engines read which catalogs and delete formats.All integrationsEvery source and engine pair with a tested guide.

Deeper reading on this site: