Resource Requirements for High Partition Counts

When configuring partitions, consider the RAM requirements based on your partition count and thread configuration:

RAM Calculation (Historical Load & Parallel-table based CDC - MongoDB/Kafka sources):

• Each partition writer can consume approximately 20-50 MB of RAM when processing data
• Total RAM needed ≈ Number of threads × Number of partitions × 20-50 MB
• Example: With 10 threads and 1000 partitions, you may need 256-512 GB RAM

Instance Sizing Recommendations:

• For high partition counts (1000+ partitions with multiple threads), use a 512 GB RAM instance
• For very high partition counts, consider scaling to a 1 TB RAM instance

Mitigation Strategies:

• Reduce thread count: Lowering the number of threads reduces concurrent partition writers and RAM usage
• Optimize partition granularity: Avoid creating too many small partitions; aim for partitions with meaningful data volumes