Adds three new CronJobs (spark-accuracy-blunder, spark-clock-pressure,
spark-smurf-anomaly) and registers their six output tables in the
analytics webview (accuracy_by_rating, blunder_outcome, clock_by_rating,
scramble_outcome, smurf_anomaly, flagged_smurfs).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
do-block-storage only supports ReadWriteOnce. Executor tasks failed with
`Mkdirs failed` because the executor ran on the spark-worker node which
had no mount of the spark-analytics-output PVC.
- Mount spark-analytics-output PVC on spark-worker pod at /spark-output
- Add podAffinity to all 11 analytics CronJob driver pods to schedule on
the same node as spark-worker, ensuring the RWO PVC is accessible by
both driver and executor
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds CronJobs and headless driver Services for GameLengthJob, ColorAdvantageJob,
EloDistributionJob, TimeControlJob, DailyActivityJob, RatingMismatchJob, and
TerminationStatsJob. All suspended by default; scheduled Saturday mornings.
Adds spark-analytics-webview: a stdlib Python HTTP server (ConfigMap + Deployment
+ Service) that reads CSV output from the spark-analytics-output PVC and renders
13 datasets as browsable HTML tables. Disabled (replicas=0) in base.
Staging overlay enables the webview (replicas=1), exposes it via Ingress at
spark-results-st.nowchess.janis-eccarius.de, and sets NOWCHESS_PGN_PATH on all
CronJobs to the Lichess Jan-2013 rated standard PGN dump for demo runs without
a NowChess database.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Nodes have only ~60-210Mi free. Requests are scheduler hints only —
pods burst to 1Gi limit once placed. 64Mi fits on all three nodes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Nodes are saturated on requested resources. Driver pods (streaming +
cronjobs) were requesting 500m CPU + 768Mi each. Limits unchanged at
1 CPU / 1Gi — pods can still burst when resources are free.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
local-path PVCs are created root-owned. The apache/spark image runs as
spark (UID 185). fsGroup: 185 makes K8s chown the mounted volume to that
group on attach, fixing the mkdir IOException on /spark-checkpoint.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
local-path provisioner is RWO-only; attaching a single PVC to master, two
workers, history-server, and streaming pod on separate nodes causes
Multi-Attach errors. Each pod now gets its own ephemeral emptyDir for
/spark-events. Event logging confs removed from CronJobs and streaming
Deployment (logs are pod-local and do not persist across restarts).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Wire five Spark jobs (OpeningBook, PlayerStats, PlayerClustering,
PlayerGraph, LiveDashboard) to Kubernetes via native Spark-on-K8s mode.
- spark-analytics/base: ServiceAccount, RBAC Role/RoleBinding (driver
executor pod management), two PVCs (10Gi output, 2Gi checkpoint),
four CronJobs for batch analytics, one Deployment for the Structured
Streaming live dashboard
- spark-analytics/eu-central-1-prod: namespace overlay pinning image
to 0.1.0 with strategic merge patches that update both the container
image field and the spark.kubernetes.container.image conf arg
- eu-central-1/argo-apps/spark-analytics: ArgoCD Application pointing
at the prod overlay with automated prune+selfHeal sync
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Janis Eccarius