With SPARK_WORKER_CORES=1 and 2 workers the cluster has 2 total cores.
LiveDashboard holds both — batch jobs wait indefinitely. 4 cores per
worker gives 8 total, enough to run streaming + batch concurrently.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
bitnami-labs GitHub Pages repo (bitnami-labs.github.io/sealed-secrets)
returns 404 — chart moved to charts.bitnami.com/bitnami. Updates chart
version from 2.16.1 to 2.5.19 (controller 0.27.1 → 0.31.0). Controller
is backward-compatible with existing sealed secrets.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Cluster stable. Re-enabling automated sync on spark-analytics and
spark-cluster. Driver DNS fix and Recreate strategy already in main,
so next sync deploys the corrected manifests.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
CPU saturation caused an unclean AOF write, corrupting the tail of
appendonly.aof.41.incr.aof at offset 32653224. Redis refuses to start.
Setting aof-load-corrupt-tail-max-size to 88 (value Redis reported) so
it truncates and loads the intact data instead of crash-looping.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Cluster saturated at 100% CPU due to Spark executor crash loop.
Disabling automated sync on spark-analytics and spark-cluster to prevent
further changes until cluster is stable.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Rolling update deadlocked: new pod couldn't attach the RWO checkpoint
PVC while the old pod still held it, so ContainerCreating stalled for
39+ minutes. Recreate terminates the old pod first.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Spark executors crashed immediately (exit 1) because the driver
advertised its pod name as the RPC hostname, which is not DNS-resolvable
in-cluster. Added headless Service `spark-live-dashboard-driver` and
pinned spark.driver.host/port + spark.blockManager.port so workers can
connect back to the driver. Stops the JVM fork storm that saturated all
three nodes at ~100% CPU.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Nodes have only ~60-210Mi free. Requests are scheduler hints only —
pods burst to 1Gi limit once placed. 64Mi fits on all three nodes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Nodes are saturated on requested resources. Driver pods (streaming +
cronjobs) were requesting 500m CPU + 768Mi each. Limits unchanged at
1 CPU / 1Gi — pods can still burst when resources are free.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
local-path PVCs are created root-owned. The apache/spark image runs as
spark (UID 185). fsGroup: 185 makes K8s chown the mounted volume to that
group on attach, fixing the mkdir IOException on /spark-checkpoint.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
local-path provisioner is RWO-only; attaching a single PVC to master, two
workers, history-server, and streaming pod on separate nodes causes
Multi-Attach errors. Each pod now gets its own ephemeral emptyDir for
/spark-events. Event logging confs removed from CronJobs and streaming
Deployment (logs are pod-local and do not persist across restarts).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
K8s injects SPARK_MASTER_PORT=tcp://<ip>:7077 because the Service is
named spark-master. Spark master parses it as integer -> NumberFormatException.
enableServiceLinks: false suppresses all Docker-link style env var injection.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
3.5.x has no official Docker Hub image. 4.0.3 is the oldest available
scala2.13 variant on Docker Hub.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Scheduler uses requests (not limits) for placement. Previous 1Gi requests
per pod caused Insufficient memory even with plenty of physical RAM.
New requests: master 256Mi, workers 256Mi, history-server 128Mi.
Limits unchanged. SPARK_WORKER_MEMORY reduced to 512m to match worker budget.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Bitnami Spark chart requires a Broadcom commercial license. Replace with
plain Deployments using apache/spark:3.5.4-scala2.13-java17-ubuntu
(Apache 2.0, same image used by the analytics jobs):
- spark-master: Deployment + ClusterIP Service (ports 7077/8080)
- spark-worker: Deployment, 2 replicas, connects to spark-master:7077
- spark-history-server: updated to apache/spark image
Ingress backend corrected to spark-master:8080.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The Bitnami spark chart 9.2.10 does not render a history server despite
historyServer.enabled=true in values. Add an explicit Deployment and
ClusterIP Service (spark-history-server-svc:18080) using the same
bitnami/spark:3.5.2 image, wired to the shared spark-events PVC.
The existing spark-history-ingress now has a valid backend.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Wire five Spark jobs (OpeningBook, PlayerStats, PlayerClustering,
PlayerGraph, LiveDashboard) to Kubernetes via native Spark-on-K8s mode.
- spark-analytics/base: ServiceAccount, RBAC Role/RoleBinding (driver
executor pod management), two PVCs (10Gi output, 2Gi checkpoint),
four CronJobs for batch analytics, one Deployment for the Structured
Streaming live dashboard
- spark-analytics/eu-central-1-prod: namespace overlay pinning image
to 0.1.0 with strategic merge patches that update both the container
image field and the spark.kubernetes.container.image conf arg
- eu-central-1/argo-apps/spark-analytics: ArgoCD Application pointing
at the prod overlay with automated prune+selfHeal sync
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add Argo Rollout (blueGreen), active/preview Services, and HPA for the
new analysis service (port 8087). Wire image pins into all three overlay
kustomizations and register the analysis Warehouse and freight entries in
all three Kargo Stages (staging -> eu-central-1-prod / htwg-1-prod).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Janis Eccarius <eccariusjanis@gmail.com>
Reviewed-on: #381
Add Argo Rollout (blueGreen), active/preview Services, and HPA for the
new analysis service (port 8087). Wire image pins into all three overlay
kustomizations and register the analysis Warehouse and freight entries in
all three Kargo Stages (staging -> eu-central-1-prod / htwg-1-prod).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Janis Eccarius
svc.kargo
Janis
shosho996