Each dataset page now has an "Explain this dataset" button that calls the
NVIDIA NIM chat completions API (meta/llama-3.3-70b-instruct) directly from
the browser with headers + first 10 rows as context. API key is entered via
a settings modal and stored in localStorage only — never sent to the server.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Points to /spark-output/.pgn-cache so the decompressed Lichess PGN file
is written to the shared PVC, accessible by both driver and executor pods.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
do-block-storage only supports ReadWriteOnce. Executor tasks failed with
`Mkdirs failed` because the executor ran on the spark-worker node which
had no mount of the spark-analytics-output PVC.
- Mount spark-analytics-output PVC on spark-worker pod at /spark-output
- Add podAffinity to all 11 analytics CronJob driver pods to schedule on
the same node as spark-worker, ensuring the RWO PVC is accessible by
both driver and executor
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
13 separate DB connections on index page caused readiness probe timeout.
Now all counts share one connection. BrokenPipeError on probe disconnect silenced.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Handle URLs without explicit port, show DB_HOST:PORT/DB_NAME in error messages
and startup log to simplify connection debugging.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Rewrite webview serve.py to query PostgreSQL analytics tables via psycopg2
instead of reading CSV files from PVC; PVC mount removed from Deployment
- Remove isPgnMode guard from all 4 original jobs so results always write to DB
regardless of whether input comes from game_records or a Lichess PGN dump
- Add JDBC write-back to all 7 new analytics jobs (game_length_distribution,
game_length_by_result, color_advantage, elo_distribution, time_control_stats,
hourly_activity, weekly_activity, rating_mismatch, termination_stats)
- Add analytics_component_sizes JDBC write to PlayerGraphJob
- Add NOWCHESS_PGN_PATH staging patches for the 7 new CronJobs
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds CronJobs and headless driver Services for GameLengthJob, ColorAdvantageJob,
EloDistributionJob, TimeControlJob, DailyActivityJob, RatingMismatchJob, and
TerminationStatsJob. All suspended by default; scheduled Saturday mornings.
Adds spark-analytics-webview: a stdlib Python HTTP server (ConfigMap + Deployment
+ Service) that reads CSV output from the spark-analytics-output PVC and renders
13 datasets as browsable HTML tables. Disabled (replicas=0) in base.
Staging overlay enables the webview (replicas=1), exposes it via Ingress at
spark-results-st.nowchess.janis-eccarius.de, and sets NOWCHESS_PGN_PATH on all
CronJobs to the Lichess Jan-2013 rated standard PGN dump for demo runs without
a NowChess database.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Rolling update deadlocked: new pod couldn't attach the RWO checkpoint
PVC while the old pod still held it, so ContainerCreating stalled for
39+ minutes. Recreate terminates the old pod first.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Spark executors crashed immediately (exit 1) because the driver
advertised its pod name as the RPC hostname, which is not DNS-resolvable
in-cluster. Added headless Service `spark-live-dashboard-driver` and
pinned spark.driver.host/port + spark.blockManager.port so workers can
connect back to the driver. Stops the JVM fork storm that saturated all
three nodes at ~100% CPU.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Nodes have only ~60-210Mi free. Requests are scheduler hints only —
pods burst to 1Gi limit once placed. 64Mi fits on all three nodes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Nodes are saturated on requested resources. Driver pods (streaming +
cronjobs) were requesting 500m CPU + 768Mi each. Limits unchanged at
1 CPU / 1Gi — pods can still burst when resources are free.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
local-path PVCs are created root-owned. The apache/spark image runs as
spark (UID 185). fsGroup: 185 makes K8s chown the mounted volume to that
group on attach, fixing the mkdir IOException on /spark-checkpoint.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
local-path provisioner is RWO-only; attaching a single PVC to master, two
workers, history-server, and streaming pod on separate nodes causes
Multi-Attach errors. Each pod now gets its own ephemeral emptyDir for
/spark-events. Event logging confs removed from CronJobs and streaming
Deployment (logs are pod-local and do not persist across restarts).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Wire five Spark jobs (OpeningBook, PlayerStats, PlayerClustering,
PlayerGraph, LiveDashboard) to Kubernetes via native Spark-on-K8s mode.
- spark-analytics/base: ServiceAccount, RBAC Role/RoleBinding (driver
executor pod management), two PVCs (10Gi output, 2Gi checkpoint),
four CronJobs for batch analytics, one Deployment for the Structured
Streaming live dashboard
- spark-analytics/eu-central-1-prod: namespace overlay pinning image
to 0.1.0 with strategic merge patches that update both the container
image field and the spark.kubernetes.container.image conf arg
- eu-central-1/argo-apps/spark-analytics: ArgoCD Application pointing
at the prod overlay with automated prune+selfHeal sync
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Janis Eccarius
Janis