fix(spark): cap executor cores to prevent job starvation

Spark standalone gives executors all worker cores by default.
LiveDashboard was consuming all 8 cores (2 workers x 4 cores).
- streaming: spark.executor.cores=1 (uses 1 core total)
- batch jobs: spark.executor.cores=2 (uses 4 cores total, 2 per executor)
Leaves headroom for concurrent streaming + batch.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Janis Eccarius
2026-06-17 00:44:16 +02:00
parent d3feb65eb3
commit 1b75466b3e
2 changed files with 10 additions and 0 deletions
+2
View File
@@ -53,6 +53,8 @@ spec:
- --conf
- spark.executor.instances=1
- --conf
- spark.executor.cores=1
- --conf
- spark.executor.memory=512m
- --conf
- spark.driver.memory=512m