Compaction Ran Out of Memory? Check Your Table and Fix It
If compaction keeps failing on a table that receives a lot of updates, the compaction executor probably doesn't have enough memory for it. This page shows how to find which of two memory problems you have, how much memory the table needs, and how to set it.
Which Problem Do You Have?
Open the failed task's reason in Runs and Logs and look for one of these errors:
OutOfMemoryError: Java heap space: the executor's memory is full. The task's data doesn't fit in the executor memory (the Java heap), so Java stops the task. See Problem A.exit code 137andOOMKilled: the pod went over its memory limit. The executor uses more memory than its pod's limit, so the pod is stopped. See Problem B.
A failed run commits nothing, so your data is safe. The table just isn't compacted until the problem is fixed.
How Compaction Uses Memory
Fusion splits a table's compaction into tasks of about the target file size, 512 MB by default. Each task runs on one Spark executor from start to end. The executor runs in a Kubernetes pod, and the pod's memory has two parts:
| Part | Set by | Holds | Problem |
|---|---|---|---|
| Executor memory (Java heap) | spark.executor.memory | Where the task works; mostly its changed keys | A, when it fills up |
| Memory overhead | spark.executor.memoryOverhead: default 10 %, at least 384 MB | Java outside the heap: class metadata, thread stacks, compiled code, garbage-collector data, network and file buffers ("VM overheads, interned strings, other native overheads" in Spark's words) | B, when it's too small |
| Pod limit | Executor memory + overhead; Spark sets it as the pod's resources.limits.memory | Everything the executor uses; Kubernetes stops the pod above it | B: usually a small overhead, sometimes a nearly full executor memory |
Sources: Spark's configuration, memory tuning and Kubernetes docs, for Spark 3.5, which Fusion uses.
Adding more executors doesn't help with either problem, because each task runs on a single executor.
Problem A: The Executor's Memory Is Full
Symptom: the failed task's reason contains:
java.lang.OutOfMemoryError: Java heap space
Why It Happens
When a table gets an update or a delete, many writers don't rewrite the old row. They write its key (primary key or identifier columns) to an equality delete file, meaning "drop any older row with this key". Each of those keys is a changed key. To learn more, see Merge-on-Read vs Copy-on-Write in Apache Iceberg and Iceberg Delete Formats for CDC.
Compaction removes those old rows, so each task keeps the changed keys of its rows in memory until it ends. The task fails when they don't fit in the executor memory, which is more likely with:
- Many updates since the last compaction
- A large target file size, so each task covers more rows
- A small executor, such as 1 GB
Insert-only tables and big data files alone don't cause it. How Much Memory You Need works out the memory for your table.
On tables synced by OLake Go, the key is _olake_id, built from the source primary key. Only streams in Upsert mode, the default, write changed keys; Append mode doesn't (see Upsert vs Append). The first sync after a full load also writes one for every inserted row. For how OLake Go picks a delete format, see Iceberg Delete Format Interoperability in OLake Go.
How Much Memory You Need
The memory a task needs depends on how many changed keys it holds and how long each key is. Find your table's inputs with the queries below, then enter them in the calculator.
The Formula
memory_needed = (changed_keys_per_task + largest_delete_file_rows) × (BASE_KEY_BYTES + max_key_length) × BUFFER
changed_keys_per_task = min(target_file_size_mb × max_rows_per_mb, changed_keys_in_partition)
Round memory_needed up to a whole GB to get the value for spark.executor.memory.
What each term means
changed_keys_in_partitionis the number of changed keys in the table's busiest partition.largest_delete_file_rowsis the number of rows in the table's largest equality delete file.max_rows_per_mbis the maximum rows per MB of any data file in the table.target_file_size_mbis the target file size compaction splits tasks by, 512 MB by default.max_key_lengthis the length of the longest key, in characters, 32 by default.changed_keys_per_taskis the number of changed keys one task holds. It's never more than the rows a task covers.BASE_KEY_BYTESis the memory each key takes before its own characters, a fixed 250 B.BUFFERis headroom for Spark and Java, a fixed × 1.3.
Find the inputs
Run the queries for the first three inputs, keep or change the two defaults, then enter all five in the calculator.
The queries only read the table's metadata tables, so they're fast and safe on any table. Run them in Fusion's Terminal or in your own query engine, and replace my_catalog.my_db.my_table with your table. They're written in Spark SQL; for other engines, see Other query engines.
changed_keys_in_partition
The changed keys in the busiest partition. It grows with every update until the next compaction, and a key that changed twice counts twice.
In Spark SQL (other engines), use equality_delete_record_count in the top row. An unpartitioned table returns one row.
/* Changed keys per partition, busiest first */
SELECT *
FROM my_catalog.my_db.my_table.partitions
ORDER BY equality_delete_record_count DESC
LIMIT 5;
largest_delete_file_rows
The rows in the largest equality delete file. A task reads it while holding its changed keys, so it adds to the peak.
In Spark SQL (other engines):
/* Rows in the largest equality delete file (content = 2) */
SELECT max(record_count) AS largest_delete_file_rows
FROM my_catalog.my_db.my_table.files
WHERE content = 2;
max_rows_per_mb
The maximum rows per MB of any data file. Times the target file size, it's the maximum rows one task can cover.
In Spark SQL (other engines):
/* The maximum rows per MB of any data file (content = 0),
leaving out files bigger than the target file size: 512 MB by default */
SELECT max(record_count * 1048576.0 / file_size_in_bytes) AS max_rows_per_mb
FROM my_catalog.my_db.my_table.files
WHERE content = 0 AND file_size_in_bytes < 512 * 1024 * 1024;
A data file bigger than the target is a task of its own. If its record_count is more than target_file_size_mb × max_rows_per_mb, use its record_count instead.
target_file_size_mb
The size Fusion aims for when it splits compaction into tasks, 128 or more. Keep the default of 512 unless you changed it in Configuration.
max_key_length
The longest key, in characters. Keep the default of 32 on tables synced by OLake Go, where the key is the 32-character _olake_id hash, unless the source table has a single-column primary key: then _olake_id is that column's value as text.
For other keys, first find the key column. In Spark SQL (other engines):
/* The key column has a non-null value_count in readable_metrics.
equality_ids holds the key columns' field IDs. */
SELECT equality_ids, readable_metrics
FROM my_catalog.my_db.my_table.files
WHERE content = 2
LIMIT 1;
Metadata stores at most 16 characters of each value, so get the longest key from the source database instead. For example, on PostgreSQL:
/* Longest primary key value in the source table (replace id and the table name) */
SELECT max(length(id::text)) AS max_key_length
FROM public.my_source_table;
Calculator
Enter the five inputs and your current executor memory. The fields start with an example table that fails on a 4g executor.
spark.executor.memory=6g, as in Fix It: Raise the Executor Memory.Worked example
The calculator's example table, on a 4g executor at the default 512 MB target file size. The queries return 20,000,000 changed keys, 1,000,000 rows in the largest delete file and 32,000 rows per MB.
- Rows per task. A task holds at most 512 MB × 32,000 rows per MB = 16,384,000 rows.
- Changed keys per task. The partition has 20,000,000 changed keys, more than one task can hold, so each task holds at most 16,384,000 of them.
- Memory needed. Add the 1,000,000 rows of the largest delete file: (16,384,000 + 1,000,000) × (250 + 32) B × 1.3 = 5.94 GB.
- Executor memory. 5.94 GB is more than 4g, so the task fails with
OutOfMemoryError: setspark.executor.memory=6g, as in fix A. - Overhead. 6g is 6 GB or less, so also set
spark.executor.memoryOverheadFactor=0.15(921 MB instead of the default 614 MB), as in fix B.
- Quick check: expect about 0.34 GB per million changed keys a task holds, with 32-character keys.
- Same unit: the calculator's GB and the
ginspark.executor.memoryare the same, so 6 GB is6g. - On the safe side: a key that changed twice counts twice, so the real need can be lower.
- Size for any number of updates: enter a very large
changed_keys_in_partition. The calculator caps it at the most rows one task can hold, which gives the most memory a task can ever need. - More than one core: with
spark.executor.coresabove 1, an executor runs that many tasks at once, so multiply the memory by the number of cores. - Overhead: for
spark.executor.memoryOverhead, see problem B.
Using another query engine?
Most engines read the same Iceberg metadata tables under a different name:
| Engine | Metadata tables |
|---|---|
| Trino, Athena | "my_table$files", "my_table$partitions" |
| Flink, StarRocks | my_table$files, my_table$partitions |
| Hive | my_db.my_table.files, my_db.my_table.partitions |
| DuckDB | iceberg_metadata('...') |
If your engine's partitions table has no equality-delete count, sum record_count from the files table where content = 2, grouped by partition. If your engine can't show Iceberg metadata (for example Snowflake, BigQuery or Databricks), use PyIceberg's table.inspect.files() and table.inspect.partitions().
Fix It: Raise the Executor Memory
Set spark.executor.memory to the value from the calculator. The next scheduled compaction picks up the change and should show Success.
- Docker setup
- Kubernetes (Helm)
On the Docker Compose setup (OLake UI docker-compose-v1.yml), edit the optimizer deployment in the local cluster:
# Run kubectl inside the Docker setup's local Kubernetes cluster
alias fusion-kubectl='docker exec -i fusion-cluster-control-plane kubectl --kubeconfig /etc/kubernetes/admin.conf'
# Save the optimizer deployment to a file
fusion-kubectl -n spark get deploy fusion-optimizer -o yaml > fo.yaml
# Change the executor memory from the default 1g to 3g
# (macOS sed; on Linux use: sed -i 's/.../.../' fo.yaml)
sed -i '' 's/spark.executor.memory=1g/spark.executor.memory=3g/' fo.yaml
# Apply the change; this restarts the optimizer
fusion-kubectl replace -f - < fo.yaml
# Wait until the new optimizer is running
fusion-kubectl -n spark rollout status deploy/fusion-optimizer
Replace 3g with your value. Make sure Docker Desktop has about 1 GB free beyond the executor memory, for example 4 GB for a 3g executor.
docker compose up recreates the stack with the default 1g, so redo this after it.
Set the value in your values file and run helm upgrade (see Customizing the Spark optimizer):
fusion:
optimizer:
spark:
extraConfig:
# Executor memory per task. The chart's default is 4g.
spark.executor.memory: "6g"
Replace 6g with your value.
If the executor memory is 6 GB or less, also apply the fix for problem B, or the pod can be stopped before its memory is full.
Verify Changes
After the next compaction shows Success in Runs and Logs, run the changed_keys_in_partition query again. equality_delete_record_count should be 0.
Problem B: The Pod Went Over Its Memory Limit
Symptom: the failed task's reason contains exit code 137 and OOMKilled, and there is no OutOfMemoryError:
The executor with id 4 exited with exit code 137(SIGKILL, possible container OOM).
termination reason: OOMKilled
A new executor pod starts for each retry, and kubectl -n <namespace> get pods shows them ending as OOMKilled.
Why It Happens
The pod's memory limit is spark.executor.memory + spark.executor.memoryOverhead. Kubernetes stops the pod when the executor goes over it, for one of two reasons:
- The overhead is too small. Java needs about 330 MB plus 3.6 % of the executor memory outside the heap, about 0.5 GB for a 4 GB executor. The default overhead, 10 % with a 384 MB minimum, is less than that for executors of 6 GB or less, so the pod is stopped even when its executor memory is half empty.
- The executor memory is nearly full. Even with enough overhead, the two together can go over the limit.
To tell which, work out memory_needed with problem A's formula:
memory_needed | Cause | Fix |
|---|---|---|
Well below spark.executor.memory | The overhead is too small | Raise the overhead |
Close to or above spark.executor.memory | The executor memory is nearly full | Raise the overhead and the executor memory |
The two fixes don't conflict, so if you're unsure, apply both.
Fix It: Raise the Overhead
If spark.executor.memory is 6 GB or less, raise the overhead with one setting (see Spark's configuration):
- Below 4 GB: set
spark.executor.memoryOverhead=512m. A percentage doesn't help here, because Spark's 384 MB minimum applies up to about 3.8 GB. - 4 GB to 6 GB: set
spark.executor.memoryOverheadFactor=0.15, 15 % instead of the default 10 %. - Above 6 GB: the default 10 % is enough.
spark.executor.memory | Overhead by default | Set | Overhead after |
|---|---|---|---|
| 1g | 384 MB | spark.executor.memoryOverhead=512m | 512 MB |
| 2g | 384 MB | spark.executor.memoryOverhead=512m | 512 MB |
| 3g | 384 MB | spark.executor.memoryOverhead=512m | 512 MB |
| 4g | 409 MB | spark.executor.memoryOverheadFactor=0.15 | 614 MB |
| 6g | 614 MB | spark.executor.memoryOverheadFactor=0.15 | 921 MB |
Set only one of the two. Spark ignores the factor when spark.executor.memoryOverhead is set.
- Docker setup
- Kubernetes (Helm)
Add the setting to the optimizer deployment in the local cluster:
# Run kubectl inside the Docker setup's local Kubernetes cluster
alias fusion-kubectl='docker exec -i fusion-cluster-control-plane kubectl --kubeconfig /etc/kubernetes/admin.conf'
# Add spark.executor.memoryOverhead=512m to the optimizer's Spark settings
# (default 384 MB below 4g); this restarts the optimizer.
# For a 4g to 6g executor, use spark.executor.memoryOverheadFactor=0.15 instead.
fusion-kubectl -n spark patch deploy fusion-optimizer --type=json -p '[
{"op":"add","path":"/spec/template/spec/containers/0/args/0","value":"--conf"},
{"op":"add","path":"/spec/template/spec/containers/0/args/1","value":"spark.executor.memoryOverhead=512m"}]'
# Wait until the new optimizer is running
fusion-kubectl -n spark rollout status deploy/fusion-optimizer
docker compose up recreates the stack with the default overhead, so redo this after it.
Add the setting to your values file and run helm upgrade (see Customizing the Spark optimizer):
fusion:
optimizer:
spark:
extraConfig:
# 4g to 6g executor (the chart's default is 4g): overhead as a share of
# the executor memory. Spark's default is 0.10.
spark.executor.memoryOverheadFactor: "0.15"
# Below 4g, use a fixed overhead instead of the line above:
# spark.executor.memoryOverhead: "512m"
Verify Changes
After the next compaction, the executor pods are no longer OOMKilled and the run shows Success in Runs and Logs. If the task now fails with java.lang.OutOfMemoryError instead, the executor memory is the limit: go to problem A.