Rolling update deadlocked: new pod couldn't attach the RWO checkpoint
PVC while the old pod still held it, so ContainerCreating stalled for
39+ minutes. Recreate terminates the old pod first.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Spark executors crashed immediately (exit 1) because the driver
advertised its pod name as the RPC hostname, which is not DNS-resolvable
in-cluster. Added headless Service `spark-live-dashboard-driver` and
pinned spark.driver.host/port + spark.blockManager.port so workers can
connect back to the driver. Stops the JVM fork storm that saturated all
three nodes at ~100% CPU.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Nodes have only ~60-210Mi free. Requests are scheduler hints only —
pods burst to 1Gi limit once placed. 64Mi fits on all three nodes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Nodes are saturated on requested resources. Driver pods (streaming +
cronjobs) were requesting 500m CPU + 768Mi each. Limits unchanged at
1 CPU / 1Gi — pods can still burst when resources are free.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
local-path PVCs are created root-owned. The apache/spark image runs as
spark (UID 185). fsGroup: 185 makes K8s chown the mounted volume to that
group on attach, fixing the mkdir IOException on /spark-checkpoint.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
local-path provisioner is RWO-only; attaching a single PVC to master, two
workers, history-server, and streaming pod on separate nodes causes
Multi-Attach errors. Each pod now gets its own ephemeral emptyDir for
/spark-events. Event logging confs removed from CronJobs and streaming
Deployment (logs are pod-local and do not persist across restarts).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Wire five Spark jobs (OpeningBook, PlayerStats, PlayerClustering,
PlayerGraph, LiveDashboard) to Kubernetes via native Spark-on-K8s mode.
- spark-analytics/base: ServiceAccount, RBAC Role/RoleBinding (driver
executor pod management), two PVCs (10Gi output, 2Gi checkpoint),
four CronJobs for batch analytics, one Deployment for the Structured
Streaming live dashboard
- spark-analytics/eu-central-1-prod: namespace overlay pinning image
to 0.1.0 with strategic merge patches that update both the container
image field and the spark.kubernetes.container.image conf arg
- eu-central-1/argo-apps/spark-analytics: ArgoCD Application pointing
at the prod overlay with automated prune+selfHeal sync
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Janis Eccarius