Commit Graph

18 Commits

Author SHA1 Message Date
Janis af3310297b feat(spark-analytics): per-row free-text questions in webview
Each expandable row now has a text input + Ask button alongside the
auto-analysis. Type any question about that row, press Ask or Enter, and
the answer streams in below via the existing /explain SSE path.

Server: row mode accepts an optional "question" field — when set it uses
a grounded Q&A prompt instead of the default analysis. Adds askRow() JS
handler, row-ask markup, and supporting CSS.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-24 09:47:15 +02:00
Janis 9a5b62c2ab fix(spark-analytics): remove raw newline from webview JS comment
The SEP comment contained "\n\n", which Python's triple-quoted JS block
turned into real newline characters in the emitted script. That produced
an inline-<script> SyntaxError ("string literal contains an unescaped
line break"), so every handler went undefined and clicking the analyse
buttons did nothing (ReferenceError: analyzeRow is not defined).

Replace the comment text with plain words — no backslash escapes. Verified
by exec'ing serve.py (psycopg2 stubbed) to capture the Python-evaluated
JS and parsing it with node --check.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-24 09:37:15 +02:00
Janis 9bb8f119b5 fix(spark-analytics): stream NIM AI responses to kill webview 504s
The /explain endpoint blocked on a synchronous NIM call (llama-3.3-70b,
30-60s+) while nginx's default 60s proxy-read-timeout fired first,
surfacing every AI request as a 504 Gateway Timeout.

Switch the NIM proxy to streaming (stream:true) and relay its SSE to the
browser via a new _emit() helper; the client now reads resp.body
incrementally and renders tokens as they arrive. With chunks flowing,
nginx's read-timeout resets per chunk so the 60s wall is gone, and the
frozen spinner is replaced by live text.

Ingress: disable proxy-buffering (else nginx buffers the whole response,
defeating streaming) and set proxy-read-timeout to 120s as the
inter-chunk gap budget. Server urlopen timeout 60->110s.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-24 09:22:31 +02:00
Janis Eccarius f64b838030 feat(spark-analytics): add AccuracyBlunder, ClockPressure, SmurfAnomaly jobs and webview datasets
Adds three new CronJobs (spark-accuracy-blunder, spark-clock-pressure,
spark-smurf-anomaly) and registers their six output tables in the
analytics webview (accuracy_by_rating, blunder_outcome, clock_by_rating,
scramble_outcome, smurf_anomaly, flagged_smurfs).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-24 00:18:22 +02:00
Janis Eccarius 587ea19e0e fix(spark-analytics): increase NIM timeout and trim max_tokens to avoid read timeouts
70B model at 30-50 tok/s was exceeding the 30s urllib timeout for larger
outputs. Raise server timeout to 60s, add browser AbortController at 65s
so the UI fails gracefully. Trim max_tokens (row: 300, dataset: 512) to
keep generation well inside the window.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-21 17:58:29 +02:00
Janis Eccarius 1ab9d60dd2 fix(spark-analytics): use ThreadingHTTPServer to prevent probe starvation
Single-threaded HTTPServer blocked readiness probes during long NIM API
calls (~30s), causing Kubernetes to flip the pod unhealthy until the
request completed. ThreadingHTTPServer handles each request in its own
thread so probes are never starved.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-21 17:53:31 +02:00
Janis Eccarius ac51ad09a4 feat(spark-analytics): per-row AI analysis with queue and rate-limit handling
- Add  button on each row (first 200) — clicking analyses that specific
  row with a chess-expert prompt tailored to the entry type (opening,
  player, etc.)
- Cache results per row so already-analysed rows skip re-requests
- Serialise all NIM requests through a promise queue to avoid bursting
- Pass 429 + Retry-After through from NIM to the browser; JS surfaces
  "Rate limited — retry in Ns" error and re-enables the button for retry
- Dataset-level "Analyse top 10 rows" now sends a per-row numbered prompt
  asking for chess-specific analysis of each entry rather than a vague
  dataset summary
- Store first 200 rows in window.DATASET.rows for client-side row access

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-21 17:48:10 +02:00
Janis Eccarius 7f11de365c fix(spark-analytics): proxy NIM API calls through server to avoid CORS
Browser cannot call integrate.api.nvidia.com directly — no CORS headers.
Add POST /explain endpoint: browser sends key + dataset, server calls NIM
with urllib, returns JSON. Key is never logged or stored server-side.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-21 17:36:04 +02:00
Janis Eccarius 60467d8ff6 fix(spark-analytics): escape \\n in JS userPrompt strings
Python triple-quoted string interprets \n as literal newlines,
producing invalid JS string literals and a SyntaxError that
killed the entire script block.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-21 17:25:20 +02:00
Janis Eccarius 9074c3fbd0 fix(spark-analytics): replace multiline JS template literals with concatenation
Multi-line template literals spanning unindented continuation lines break the
YAML block scalar indentation, causing kustomize to fail with MalformedYAMLError.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-21 17:19:00 +02:00
Janis Eccarius 252c543fed feat(spark-analytics): add per-user NVIDIA NIM AI explanations to webview
Each dataset page now has an "Explain this dataset" button that calls the
NVIDIA NIM chat completions API (meta/llama-3.3-70b-instruct) directly from
the browser with headers + first 10 rows as context. API key is entered via
a settings modal and stored in localStorage only — never sent to the server.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-21 17:16:26 +02:00
Janis Eccarius 283c6728d4 fix(spark-analytics): batch index COUNT queries in one connection, suppress BrokenPipeError
13 separate DB connections on index page caused readiness probe timeout.
Now all counts share one connection. BrokenPipeError on probe disconnect silenced.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-21 15:52:34 +02:00
Janis Eccarius 609a222211 fix(spark-analytics): robust JDBC URL parsing, log resolved DB host on startup
Handle URLs without explicit port, show DB_HOST:PORT/DB_NAME in error messages
and startup log to simplify connection debugging.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-21 15:47:48 +02:00
Janis Eccarius bdc3efc878 fix(spark-analytics): install psycopg2 to /tmp/pkg, set PYTHONPATH so it is importable
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-21 15:44:09 +02:00
Janis Eccarius 6765c5162d fix(spark-analytics): set HOME=/tmp so pip can write under non-root user 65534
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-21 15:42:50 +02:00
Janis Eccarius 4395415985 fix(spark-analytics): add --no-cache-dir to pip install for nonexistent home dir
User 65534 has no home directory; pip cache write to /nonexistent fails. Skip cache.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-21 15:41:41 +02:00
Janis Eccarius ca78173a56 feat(spark-analytics): switch webview to PostgreSQL, always write analytics to DB
- Rewrite webview serve.py to query PostgreSQL analytics tables via psycopg2
  instead of reading CSV files from PVC; PVC mount removed from Deployment
- Remove isPgnMode guard from all 4 original jobs so results always write to DB
  regardless of whether input comes from game_records or a Lichess PGN dump
- Add JDBC write-back to all 7 new analytics jobs (game_length_distribution,
  game_length_by_result, color_advantage, elo_distribution, time_control_stats,
  hourly_activity, weekly_activity, rating_mismatch, termination_stats)
- Add analytics_component_sizes JDBC write to PlayerGraphJob
- Add NOWCHESS_PGN_PATH staging patches for the 7 new CronJobs

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-21 15:35:32 +02:00
Janis Eccarius a190d50a09 feat(spark-analytics): add 7 CronJobs, results webview, and LiChess staging demo
Adds CronJobs and headless driver Services for GameLengthJob, ColorAdvantageJob,
EloDistributionJob, TimeControlJob, DailyActivityJob, RatingMismatchJob, and
TerminationStatsJob. All suspended by default; scheduled Saturday mornings.

Adds spark-analytics-webview: a stdlib Python HTTP server (ConfigMap + Deployment
+ Service) that reads CSV output from the spark-analytics-output PVC and renders
13 datasets as browsable HTML tables. Disabled (replicas=0) in base.

Staging overlay enables the webview (replicas=1), exposes it via Ingress at
spark-results-st.nowchess.janis-eccarius.de, and sets NOWCHESS_PGN_PATH on all
CronJobs to the Lichess Jan-2013 rated standard PGN dump for demo runs without
a NowChess database.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-21 15:09:49 +02:00