Commit Graph

2 Commits

Author SHA1 Message Date
Janis Eccarius ab95b4f101 fix(spark): replace shared spark-events PVC with emptyDir to fix Multi-Attach error
local-path provisioner is RWO-only; attaching a single PVC to master, two
workers, history-server, and streaming pod on separate nodes causes
Multi-Attach errors. Each pod now gets its own ephemeral emptyDir for
/spark-events. Event logging confs removed from CronJobs and streaming
Deployment (logs are pod-local and do not persist across restarts).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 20:33:48 +02:00
Janis Eccarius eefa6e728a feat(spark): deploy persistent Spark cluster with Web UI
- Add spark-cluster/base with Bitnami Spark chart 9.2.10 (master +
  2 workers), spark-events PVC (RWO/local-path), and Ingress resources
  for spark.nowchess.janis-eccarius.de (master UI) and
  spark-history.nowchess.janis-eccarius.de (history placeholder)
- Add spark-cluster/eu-central-1-prod overlay and ArgoCD Application
  (automated prune + selfHeal)
- Migrate all spark-analytics CronJobs and streaming Deployment from
  K8s-native mode (k8s://...) to Spark standalone (spark://spark-master:7077);
  drop kubernetes-specific --conf flags; add eventLog confs; mount
  spark-events PVC on all job pods

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-15 22:41:33 +02:00