NowChess/GitOps
4
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(spark): cap executor cores to prevent job starvation

Browse Source 
Spark standalone gives executors all worker cores by default.
LiveDashboard was consuming all 8 cores (2 workers x 4 cores).
- streaming: spark.executor.cores=1 (uses 1 core total)
- batch jobs: spark.executor.cores=2 (uses 4 cores total, 2 per executor)
Leaves headroom for concurrent streaming + batch.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Janis Eccarius
2026-06-17 00:44:16 +02:00
parent d3feb65eb3
commit 1b75466b3e
2 changed files with 10 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
spark-analytics/base/cronjobs.yaml
+8
View File
@@ -33,6 +33,8 @@ spec:
- --conf
- spark.executor.instances=2
- --conf
- spark.executor.cores=2
- --conf
- spark.executor.memory=1g
- --conf
- spark.driver.memory=512m
@@ -107,6 +109,8 @@ spec:
- --conf
- spark.executor.instances=2
- --conf
- spark.executor.cores=2
- --conf
- spark.executor.memory=1g
- --conf
- spark.driver.memory=512m
@@ -180,6 +184,8 @@ spec:
- --conf
- spark.executor.instances=2
- --conf
- spark.executor.cores=2
- --conf
- spark.executor.memory=1g
- --conf
- spark.driver.memory=512m
@@ -254,6 +260,8 @@ spec:
- --conf
- spark.executor.instances=2
- --conf
- spark.executor.cores=2
- --conf
- spark.executor.memory=1g
- --conf
- spark.driver.memory=512m
spark-analytics/base/streaming.yaml
+2
View File
@@ -53,6 +53,8 @@ spec:
- --conf
- spark.executor.instances=1
- --conf
- spark.executor.cores=1
- --conf
- spark.executor.memory=512m
- --conf
- spark.driver.memory=512m