ci: bump version with Build-154

fix: modified training pipeline
feat(official-bots): add Google Colab notebook for NNUE training (NCS-111) (#81 )
2026-06-24 17:55:44 +00:00 · 2026-06-24 19:37:26 +02:00 · 2026-06-24 19:33:24 +02:00 · 2026-06-24 19:33:12 +02:00 · 2026-06-23 22:30:53 +00:00 · 2026-06-24 00:21:40 +02:00
16 changed files with 1306 additions and 141 deletions
@@ -81,3 +81,20 @@
 * **analytics:** upgrade Spark to 4.0.3 — 3.5.x has no official Docker image ([46af115](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/46af1154de34a8596cb6cb28c6fad7aba90f597c))
 * **analytics:** write decompressed PGN to shared PVC path for executor access ([a268a9a](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/a268a9acb7ba190c76e996ccf3ea3bd00e5cec92))
 ##  (2026-06-23)
 ### Features
 * **analytics:** add 7 new Spark analytics jobs and extend GameSource ([8e17c14](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/8e17c14dff740cd115011dfbf17de35083b8fe46))
 * **analytics:** add accuracy and blunder analysis job for Lichess data ([c3e7b82](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/c3e7b82ae806adf5713ce4d267c1155e73a40ff5))
 * **analytics:** add Dockerfile, CI workflow, and stable jar name for K8s deployment ([95215b6](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/95215b6a420fd526df1aa395f9b087556c8ad03b))
 * **analytics:** add PostgreSQL JDBC write-back to all four batch jobs ([0e0ea4c](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/0e0ea4c9893c6efed52e633e55d05ab3ed004502))
 * **analytics:** add Spark batch analytics module ([259b3bb](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/259b3bbb24c0f23326269b93f4b3c84012f727cd))
 * **analytics:** add Structured Streaming, MLlib clustering, GraphX jobs ([e1d80b9](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/e1d80b9331666feea191b1fd08aa762f3581c918))
 * **analytics:** always write results to PostgreSQL regardless of input source ([da0e6d1](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/da0e6d1ee2d391ecb6291396f82471eb51b1b25e))
 * **official-bots:** park expert bot on tournament server at startup ([#76](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/76)) ([751a58b](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/751a58b6061f7434115e229a7661894c76768bc2))
 ### Bug Fixes
 * **analytics:** upgrade Spark to 4.0.3 — 3.5.x has no official Docker image ([46af115](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/46af1154de34a8596cb6cb28c6fad7aba90f597c))
 * **analytics:** write decompressed PGN to shared PVC path for executor access ([a268a9a](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/a268a9acb7ba190c76e996ccf3ea3bd00e5cec92))
@@ -0,0 +1,191 @@
 package de.nowchess.analytics
 import org.apache.spark.sql.SparkSession
 import org.apache.spark.sql.expressions.Window
 import org.apache.spark.sql.functions as F
 /** Per-move accuracy & blunder analysis mined from Lichess `[%eval ...]` move annotations.
  *
  * Unlike the flat single-`groupBy` summaries (opening rates, colour advantage), this job reconstructs the *quality of
  * every move* from the engine evaluations Lichess embeds in the movetext (`{ [%eval 0.24] }`, mate scores `[%eval
  * #-3]`) and turns them into the same accuracy signals lichess.com surfaces: average centipawn loss (ACPL), and counts
  * of inaccuracies / mistakes / blunders.
  *
  * Pipeline (all Spark SQL string/array functions + window funcs — no UDFs, Catalyst-friendly):
  *   1. Keep only games carrying `[%eval` comments.
  *   2. `regexp_extract_all` pulls every eval in ply order; mate scores collapse to ±10 pawns, normal evals are clamped
  *      to ±10 so a single huge swing cannot dominate the mean. All evals are White-POV pawns.
  *   3. `posexplode` → one row per ply; a per-game window `lag` gives the eval *before* the move.
  *   4. Centipawn loss for the side that moved = how much the eval moved against them (white wants it up, black down),
  *      floored at 0 and scaled to centipawns.
  *   5. Roll up to (game, side): ACPL + inaccuracy(≥50cp) / mistake(≥100cp) / blunder(≥200cp) counts, tagged with that
  *      side's Elo and whether they won.
  *
  * Outputs (Parquet + CSV + JDBC):
  *   - `accuracy_by_rating` — ACPL, avg blunders/mistakes/inaccuracies per game and win-rate, per Elo band. Shows how
  *     move quality scales with rating.
  *   - `blunder_outcome` — win-rate bucketed by number of blunders in the game. Quantifies "one blunder costs you the
  *     game".
  *
  * Requires the eval-annotated Lichess dump (`NOWCHESS_PGN_PATH` → an evals dump); JDBC games carry no per-move evals.
  */
 object AccuracyBlunderJob:
  def main(args: Array[String]): Unit =
    val jdbcUrl   = sys.env.getOrElse("NOWCHESS_JDBC_URL", "jdbc:postgresql://localhost:5432/nowchess")
    val dbUser    = sys.env.getOrElse("NOWCHESS_DB_USER", "nowchess")
    val dbPass    = sys.env.getOrElse("NOWCHESS_DB_PASS", "nowchess")
    val outputDir = if args.length > 0 then args(0) else "/tmp/nowchess-accuracy"
    val spark = SparkSession
      .builder()
      .appName("NowChess Accuracy & Blunders")
      .getOrCreate()
    run(spark, jdbcUrl, dbUser, dbPass, outputDir)
    spark.stop()
  def run(spark: SparkSession, jdbcUrl: String, dbUser: String, dbPass: String, outputDir: String): Unit =
    val games = GameSource
      .loadExtended(spark, jdbcUrl, dbUser, dbPass)
      .select("pgn", "result", "white_elo", "black_elo")
      .filter(F.col("result").isNotNull.and(F.col("pgn").contains("[%eval")))
      .withColumn("game_id", F.monotonically_increasing_id())
    // White-POV pawn evals in ply order; mate → ±10, normal evals clamped to ±10.
    val evalStrs = F.expr("""regexp_extract_all(pgn, '\\[%eval ([^\\]]+)\\]', 1)""")
    val evalCps = F.expr(
      "transform(eval_strs, x -> CASE " +
        "WHEN x LIKE '#-%' THEN -10.0 " +
        "WHEN x LIKE '#%' THEN 10.0 " +
        "ELSE greatest(-10.0, least(10.0, cast(x as double))) END)",
    )
    val withEvals = games
      .withColumn("eval_strs", evalStrs)
      .withColumn("eval_cp", evalCps)
      .filter(F.size(F.col("eval_cp")) >= 2)
    val plies = withEvals.select(
      F.col("game_id"),
      F.col("result"),
      F.col("white_elo"),
      F.col("black_elo"),
      F.posexplode(F.col("eval_cp")).as(Seq("ply", "eval_after")),
    )
    val byGame     = Window.partitionBy("game_id").orderBy("ply")
    val mover      = F.when(F.col("ply") % 2 === 0, "white").otherwise("black")
    val evalBefore = F.coalesce(F.lag("eval_after", 1).over(byGame), F.lit(0.15))
    val cpl = F.greatest(
      F.lit(0.0),
      F.when(F.col("mover") === "white", evalBefore - F.col("eval_after"))
        .otherwise(F.col("eval_after") - evalBefore),
    ) * 100
    val moves = plies
      .withColumn("mover", mover)
      .withColumn("cpl", cpl)
    val perSide = moves
      .groupBy("game_id", "mover", "result", "white_elo", "black_elo")
      .agg(
        F.round(F.avg("cpl"), 1).as("acpl"),
        F.sum(F.when(F.col("cpl") >= 200, 1).otherwise(0)).as("blunders"),
        F.sum(F.when(F.col("cpl") >= 100 && F.col("cpl") < 200, 1).otherwise(0)).as("mistakes"),
        F.sum(F.when(F.col("cpl") >= 50 && F.col("cpl") < 100, 1).otherwise(0)).as("inaccuracies"),
      )
      .withColumn(
        "self_elo",
        F.when(F.col("mover") === "white", F.col("white_elo")).otherwise(F.col("black_elo")),
      )
      .withColumn("won", F.when(F.col("mover") === F.col("result"), 1).otherwise(0))
    writeAccuracyByRating(perSide, jdbcUrl, dbUser, dbPass, outputDir)
    writeBlunderOutcome(perSide, jdbcUrl, dbUser, dbPass, outputDir)
  private def writeAccuracyByRating(
      perSide: org.apache.spark.sql.DataFrame,
      jdbcUrl: String,
      dbUser: String,
      dbPass: String,
      outputDir: String,
  ): Unit =
    val elo = F.col("self_elo")
    val band = F
      .when(elo < 1200, "<1200")
      .when(elo < 1500, "1200–1499")
      .when(elo < 1800, "1500–1799")
      .when(elo < 2100, "1800–2099")
      .otherwise("2100+")
    val bandOrder = F
      .when(elo < 1200, 1)
      .when(elo < 1500, 2)
      .when(elo < 1800, 3)
      .when(elo < 2100, 4)
      .otherwise(5)
    val stats = perSide
      .filter(elo.isNotNull)
      .withColumn("band", band)
      .withColumn("band_order", bandOrder)
      .groupBy("band", "band_order")
      .agg(
        F.count("*").as("player_games"),
        F.round(F.avg("acpl"), 1).as("avg_acpl"),
        F.round(F.avg("blunders"), 2).as("avg_blunders"),
        F.round(F.avg("mistakes"), 2).as("avg_mistakes"),
        F.round(F.avg("inaccuracies"), 2).as("avg_inaccuracies"),
        F.round(F.avg("won"), 3).as("win_rate"),
      )
      .orderBy(F.asc("band_order"))
      .drop("band_order")
    write(stats, outputDir, "accuracy_by_rating", jdbcUrl, dbUser, dbPass, "analytics_accuracy_by_rating")
  private def writeBlunderOutcome(
      perSide: org.apache.spark.sql.DataFrame,
      jdbcUrl: String,
      dbUser: String,
      dbPass: String,
      outputDir: String,
  ): Unit =
    val b      = F.col("blunders")
    val bucket = F.when(b === 0, "0").when(b === 1, "1").when(b === 2, "2").otherwise("3+")
    val order  = F.when(b === 0, 0).when(b === 1, 1).when(b === 2, 2).otherwise(3)
    val stats = perSide
      .withColumn("blunder_bucket", bucket)
      .withColumn("bucket_order", order)
      .groupBy("blunder_bucket", "bucket_order")
      .agg(
        F.count("*").as("player_games"),
        F.round(F.avg("won"), 3).as("win_rate"),
        F.round(F.avg("acpl"), 1).as("avg_acpl"),
      )
      .orderBy(F.asc("bucket_order"))
      .drop("bucket_order")
    write(stats, outputDir, "blunder_outcome", jdbcUrl, dbUser, dbPass, "analytics_blunder_outcome")
  private def write(
      df: org.apache.spark.sql.DataFrame,
      outputDir: String,
      name: String,
      jdbcUrl: String,
      dbUser: String,
      dbPass: String,
      table: String,
  ): Unit =
    df.write.mode("overwrite").parquet(s"$outputDir/$name")
    df.write.mode("overwrite").option("header", "true").csv(s"$outputDir/${name}_csv")
    if !GameSource.isPgnMode then
      df.write
        .mode("overwrite")
        .format("jdbc")
        .option("url", jdbcUrl)
        .option("dbtable", table)
        .option("user", dbUser)
        .option("password", dbPass)
        .option("driver", "org.postgresql.Driver")
        .save()
@@ -0,0 +1,199 @@
 package de.nowchess.analytics
 import org.apache.spark.sql.SparkSession
 import org.apache.spark.sql.expressions.Window
 import org.apache.spark.sql.functions as F
 /** Time-management & clock-pressure analysis mined from Lichess `[%clk ...]` move annotations.
  *
  * Lichess records each player's remaining clock after every move (`{ [%clk 0:02:31] }`). This job reconstructs
  * per-move thinking time and remaining-time from those stamps to answer questions the existing time-control summary
  * cannot: how long do players actually think, how often do they fall into time scrambles (<10 s left), how often do
  * they flag (lose on time), and does burning the clock correlate with winning?
  *
  * Pipeline (Spark SQL string/array funcs + window funcs — no UDFs):
  *   1. `regexp_extract_all` pulls every `h:mm:ss` clock in ply order, converted to seconds.
  *   2. `posexplode` → one row per ply; even plies are White's clock, odd plies Black's.
  *   3. A per-(game,side) window `lag` gives the same side's previous clock; the difference is that move's thinking time.
  *      Remaining clock <10 s marks a time-scramble move.
  *   4. Roll up to (game, side): avg move time, scramble fraction, min clock, Elo, win flag, and whether the side lost on
  *      time (`Termination "Time forfeit"`).
  *
  * Outputs (Parquet + CSV + JDBC):
  *   - `clock_by_rating` — avg move time, scramble fraction, flag-loss rate and win-rate per Elo band.
  *   - `scramble_outcome` — win-rate bucketed by how much of the game was played in time-scramble. Quantifies the cost of
  *     time trouble.
  *
  * Requires a clock-annotated Lichess dump (`NOWCHESS_PGN_PATH`).
  */
 object ClockPressureJob:
  def main(args: Array[String]): Unit =
    val jdbcUrl   = sys.env.getOrElse("NOWCHESS_JDBC_URL", "jdbc:postgresql://localhost:5432/nowchess")
    val dbUser    = sys.env.getOrElse("NOWCHESS_DB_USER", "nowchess")
    val dbPass    = sys.env.getOrElse("NOWCHESS_DB_PASS", "nowchess")
    val outputDir = if args.length > 0 then args(0) else "/tmp/nowchess-clock-pressure"
    val spark = SparkSession
      .builder()
      .appName("NowChess Clock Pressure")
      .getOrCreate()
    run(spark, jdbcUrl, dbUser, dbPass, outputDir)
    spark.stop()
  def run(spark: SparkSession, jdbcUrl: String, dbUser: String, dbPass: String, outputDir: String): Unit =
    val games = GameSource
      .loadExtended(spark, jdbcUrl, dbUser, dbPass)
      .select("pgn", "result", "white_elo", "black_elo", "termination")
      .filter(F.col("result").isNotNull.and(F.col("pgn").contains("[%clk")))
      .withColumn("game_id", F.monotonically_increasing_id())
    val clkStrs = F.expr("""regexp_extract_all(pgn, '\\[%clk ([^\\]]+)\\]', 1)""")
    // "h:mm:ss" → seconds.
    val clkSecs = F.expr(
      "transform(clk_strs, x -> " +
        "cast(split(x, ':')[0] as double) * 3600 + " +
        "cast(split(x, ':')[1] as double) * 60 + " +
        "cast(split(x, ':')[2] as double))",
    )
    val withClk = games
      .withColumn("clk_strs", clkStrs)
      .withColumn("clk_sec", clkSecs)
      .filter(F.size(F.col("clk_sec")) >= 4)
    val plies = withClk.select(
      F.col("game_id"),
      F.col("result"),
      F.col("white_elo"),
      F.col("black_elo"),
      F.col("termination"),
      F.posexplode(F.col("clk_sec")).as(Seq("ply", "clk_after")),
    )
    val mover    = F.when(F.col("ply") % 2 === 0, "white").otherwise("black")
    val bySide   = Window.partitionBy("game_id", "mover").orderBy("ply")
    val moveTime = F.lag("clk_after", 1).over(bySide) - F.col("clk_after")
    val moves = plies
      .withColumn("mover", mover)
      .withColumn("move_time", moveTime)
    val perSide = moves
      .groupBy("game_id", "mover", "result", "white_elo", "black_elo", "termination")
      .agg(
        F.round(F.avg("move_time"), 1).as("avg_move_time"),
        F.count("*").as("moves"),
        F.round(F.min("clk_after"), 1).as("min_clk"),
        F.sum(F.when(F.col("clk_after") < 10, 1).otherwise(0)).as("scramble_moves"),
      )
      .withColumn("scramble_fraction", F.round(F.col("scramble_moves") / F.col("moves"), 3))
      .withColumn(
        "self_elo",
        F.when(F.col("mover") === "white", F.col("white_elo")).otherwise(F.col("black_elo")),
      )
      .withColumn("won", F.when(F.col("mover") === F.col("result"), 1).otherwise(0))
      .withColumn(
        "flag_loss",
        F.when(
          F.coalesce(F.col("termination"), F.lit("")).contains("Time forfeit") && F.col("won") === 0,
          1,
        ).otherwise(0),
      )
    writeClockByRating(perSide, jdbcUrl, dbUser, dbPass, outputDir)
    writeScrambleOutcome(perSide, jdbcUrl, dbUser, dbPass, outputDir)
  private def writeClockByRating(
      perSide: org.apache.spark.sql.DataFrame,
      jdbcUrl: String,
      dbUser: String,
      dbPass: String,
      outputDir: String,
  ): Unit =
    val elo = F.col("self_elo")
    val band = F
      .when(elo < 1200, "<1200")
      .when(elo < 1500, "1200–1499")
      .when(elo < 1800, "1500–1799")
      .when(elo < 2100, "1800–2099")
      .otherwise("2100+")
    val bandOrder = F
      .when(elo < 1200, 1)
      .when(elo < 1500, 2)
      .when(elo < 1800, 3)
      .when(elo < 2100, 4)
      .otherwise(5)
    val stats = perSide
      .filter(elo.isNotNull)
      .withColumn("band", band)
      .withColumn("band_order", bandOrder)
      .groupBy("band", "band_order")
      .agg(
        F.count("*").as("player_games"),
        F.round(F.avg("avg_move_time"), 1).as("avg_move_time_s"),
        F.round(F.avg("scramble_fraction"), 3).as("avg_scramble_fraction"),
        F.round(F.avg("flag_loss"), 3).as("flag_loss_rate"),
        F.round(F.avg("won"), 3).as("win_rate"),
      )
      .orderBy(F.asc("band_order"))
      .drop("band_order")
    write(stats, outputDir, "clock_by_rating", jdbcUrl, dbUser, dbPass, "analytics_clock_by_rating")
  private def writeScrambleOutcome(
      perSide: org.apache.spark.sql.DataFrame,
      jdbcUrl: String,
      dbUser: String,
      dbPass: String,
      outputDir: String,
  ): Unit =
    val sf = F.col("scramble_fraction")
    val bucket = F
      .when(sf === 0, "none")
      .when(sf < 0.05, "<5%")
      .when(sf < 0.20, "5–20%")
      .otherwise(">20%")
    val order = F
      .when(sf === 0, 0)
      .when(sf < 0.05, 1)
      .when(sf < 0.20, 2)
      .otherwise(3)
    val stats = perSide
      .withColumn("scramble_bucket", bucket)
      .withColumn("bucket_order", order)
      .groupBy("scramble_bucket", "bucket_order")
      .agg(
        F.count("*").as("player_games"),
        F.round(F.avg("won"), 3).as("win_rate"),
        F.round(F.avg("flag_loss"), 3).as("flag_loss_rate"),
      )
      .orderBy(F.asc("bucket_order"))
      .drop("bucket_order")
    write(stats, outputDir, "scramble_outcome", jdbcUrl, dbUser, dbPass, "analytics_scramble_outcome")
  private def write(
      df: org.apache.spark.sql.DataFrame,
      outputDir: String,
      name: String,
      jdbcUrl: String,
      dbUser: String,
      dbPass: String,
      table: String,
  ): Unit =
    df.write.mode("overwrite").parquet(s"$outputDir/$name")
    df.write.mode("overwrite").option("header", "true").csv(s"$outputDir/${name}_csv")
    if !GameSource.isPgnMode then
      df.write
        .mode("overwrite")
        .format("jdbc")
        .option("url", jdbcUrl)
        .option("dbtable", table)
        .option("user", dbUser)
        .option("password", dbPass)
        .option("driver", "org.postgresql.Driver")
        .save()
@@ -0,0 +1,154 @@
 package de.nowchess.analytics
 import org.apache.spark.sql.SparkSession
 import org.apache.spark.sql.expressions.Window
 import org.apache.spark.sql.functions as F
 /** Smurf / sandbagging anomaly detection via population z-scores.
  *
  * Smurfs (strong players on fresh accounts) and sandbaggers leave a statistical signature: a win-rate, an upset-rate
  * (beating higher-rated opponents) and a self-Elo climb that sit far above the population norm. This job builds those
  * three features per player, standardises each against the whole player base, and flags the players whose combined
  * deviation is extreme.
  *
  * Features per player (from each game's own/opponent Elo):
  *   - win_rate — fraction of decisive results won
  *   - upset_rate — wins vs higher-rated opponents / games vs higher-rated opponents
  *   - elo_climb — max self-Elo − min self-Elo across their games (rapid rating gain)
  *
  * Standardisation uses a single unbounded window (`Window.partitionBy()`), i.e. mean/stddev over every qualifying
  * player, so z = (x − μ) / σ. The composite anomaly score sums the three z-scores. No UDFs — pure SQL aggregates +
  * window functions, so Catalyst plans the whole job.
  *
  * Outputs (Parquet + CSV + JDBC):
  *   - `anomaly_scores` — every qualifying player with features, z-scores and composite, ranked most-anomalous first.
  *   - `flagged_smurfs` — the suspicious subset (high composite, or the classic high-winrate / few-games / steep-climb
  *     profile).
  *
  * Meaningful only when Elo is present (Lichess dump); requires `minGames` (arg 1, default 15) to avoid small-sample
  * noise.
  */
 object SmurfAnomalyJob:
  def main(args: Array[String]): Unit =
    val jdbcUrl   = sys.env.getOrElse("NOWCHESS_JDBC_URL", "jdbc:postgresql://localhost:5432/nowchess")
    val dbUser    = sys.env.getOrElse("NOWCHESS_DB_USER", "nowchess")
    val dbPass    = sys.env.getOrElse("NOWCHESS_DB_PASS", "nowchess")
    val outputDir = if args.length > 0 then args(0) else "/tmp/nowchess-smurf-anomaly"
    val minGames  = if args.length > 1 then args(1).toInt else 15
    val spark = SparkSession
      .builder()
      .appName("NowChess Smurf Anomaly Detection")
      .getOrCreate()
    run(spark, jdbcUrl, dbUser, dbPass, outputDir, minGames)
    spark.stop()
  def run(
      spark: SparkSession,
      jdbcUrl: String,
      dbUser: String,
      dbPass: String,
      outputDir: String,
      minGames: Int,
  ): Unit =
    val games = GameSource
      .loadExtended(spark, jdbcUrl, dbUser, dbPass)
      .select("white_id", "black_id", "result", "white_elo", "black_elo")
      .filter(F.col("result").isNotNull)
    val asWhite = games.select(
      F.col("white_id").as("player_id"),
      F.col("white_elo").as("self_elo"),
      F.col("black_elo").as("opp_elo"),
      F.when(F.col("result") === "white", 1).otherwise(0).as("won"),
    )
    val asBlack = games.select(
      F.col("black_id").as("player_id"),
      F.col("black_elo").as("self_elo"),
      F.col("white_elo").as("opp_elo"),
      F.when(F.col("result") === "black", 1).otherwise(0).as("won"),
    )
    val playerGames = asWhite
      .union(asBlack)
      .filter(F.col("self_elo").isNotNull.and(F.col("opp_elo").isNotNull))
    val higher = F.col("opp_elo") > F.col("self_elo")
    val features = playerGames
      .groupBy("player_id")
      .agg(
        F.count("*").as("total_games"),
        F.round(F.avg("won"), 3).as("win_rate"),
        F.round(F.avg("self_elo"), 0).as("avg_self_elo"),
        (F.max("self_elo") - F.min("self_elo")).as("elo_climb"),
        F.sum(F.when(higher, 1).otherwise(0)).as("vs_higher"),
        F.sum(F.when(higher && F.col("won") === 1, 1).otherwise(0)).as("upsets"),
      )
      .filter(F.col("total_games") >= minGames)
      .withColumn("upset_rate", F.round(F.col("upsets") / F.greatest(F.col("vs_higher"), F.lit(1)), 3))
    val all = Window.partitionBy()
    def z(col: String): org.apache.spark.sql.Column =
      val mean = F.avg(col).over(all)
      val std  = F.stddev(col).over(all)
      F.round((F.col(col) - mean) / F.when(std === 0 || std.isNull, F.lit(1.0)).otherwise(std), 2)
    val scored = features
      .withColumn("z_win_rate", z("win_rate"))
      .withColumn("z_upset_rate", z("upset_rate"))
      .withColumn("z_elo_climb", z("elo_climb"))
      .withColumn(
        "anomaly_score",
        F.round(F.col("z_win_rate") + F.col("z_upset_rate") + F.col("z_elo_climb"), 2),
      )
      .withColumn(
        "flagged",
        (F.col("anomaly_score") >= 4.0)
          .or(F.col("win_rate") >= 0.8 && F.col("total_games") < 50 && F.col("elo_climb") >= 300),
      )
    val ordered = scored
      .select(
        "player_id",
        "total_games",
        "win_rate",
        "avg_self_elo",
        "elo_climb",
        "upset_rate",
        "z_win_rate",
        "z_upset_rate",
        "z_elo_climb",
        "anomaly_score",
        "flagged",
      )
      .orderBy(F.desc("anomaly_score"))
    write(ordered, outputDir, "anomaly_scores", jdbcUrl, dbUser, dbPass, "analytics_smurf_anomaly")
    val flagged = ordered.filter(F.col("flagged") === true)
    write(flagged, outputDir, "flagged_smurfs", jdbcUrl, dbUser, dbPass, "analytics_flagged_smurfs")
  private def write(
      df: org.apache.spark.sql.DataFrame,
      outputDir: String,
      name: String,
      jdbcUrl: String,
      dbUser: String,
      dbPass: String,
      table: String,
  ): Unit =
    df.write.mode("overwrite").parquet(s"$outputDir/$name")
    df.write.mode("overwrite").option("header", "true").csv(s"$outputDir/${name}_csv")
    if !GameSource.isPgnMode then
      df.write
        .mode("overwrite")
        .format("jdbc")
        .option("url", jdbcUrl)
        .option("dbtable", table)
        .option("user", dbUser)
        .option("password", dbPass)
        .option("driver", "org.postgresql.Driver")
        .save()
@@ -1,3 +1,3 @@
 MAJOR=0
-MINOR=7
+MINOR=8
 PATCH=0
@@ -890,3 +890,152 @@
 ### Reverts
 * Revert "refactor: update metrics paths formatting in application.yml for clarity" ([3870566](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/38705663498d5f47c40dafe2f26198589ede8656))
 ##  (2026-06-23)
 ### Features
 * add initialization metrics for various services ([d438e97](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/d438e97f32bdde0bfc63c1b4a8cc810cdd093166))
 * add OpenTelemetry trace configuration with parentbased sampler ([3904d5a](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/3904d5ad8ad4930ddee65287a7bfab785a6148f5))
 * **analytics:** add Spark batch analytics module ([#70](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/70)) ([39f1657](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/39f1657e1db6e84889af338c43be8cb5c03c3ec3))
 * **config:** update application.yml for PostgreSQL and remove staging/production configurations ([2404e61](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/2404e6164c3b50ffccbea5238d636060d6abe4d6))
 * **config:** update application.yml for staging and production environments ([6113432](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/6113432a14c476a3a0dfc0d449e17d023697f2ba))
 * configure logging and add OpenTelemetry support ([#49](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/49)) ([d57c488](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/d57c4886612d1d92da0e1b79209fc83e6ef537a1))
 * **docker:** add .dockerignore and .gitignore files for build exclusions ([c987d8e](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/c987d8e258c0e6c4cfbdaa8381c64c410d7a2b83))
 * **docker:** add Dockerfiles for building Quarkus application in native and JVM modes ([3f2d2bb](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/3f2d2bb4c97fa8cddba66e1da4427c54236dfeed))
 * **docker:** add Dockerfiles for Quarkus application in JVM and native modes ([34b9933](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/34b993304670cf2aa62cd2f6460cee7b9864b08e))
 * **events:** migrate game-creation and bot flows to Redis Streams NCS-89 ([#62](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/62)) ([a24924c](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/a24924c23057db3d700a75dbc4333557789cd991))
 * NCS-78 Add Traceability to the Applications ([#46](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/46)) ([649566e](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/649566eb3fcf38f91c8896a739f74ea318af312d))
 * NCS-78 Add Traceability to the Applications ([#47](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/47)) ([87dfc6c](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/87dfc6c2bcce7f7d58fc641bd8d468a2e584c108))
 * NCS-82 add Swiss-system tournament module ([#55](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/55)) ([c5661de](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/c5661de4a0ebf4b33211f5a391840dcf744656b7))
 * **official-bots:** activate opening book in expert bot (native-safe) ([260db25](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/260db25803ec55ce99e55782791eabdc190dfed4))
 * **official-bots:** consume GameOver stream for bot cleanup ([#67](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/67)) ([db9d153](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/db9d1533912f4b41c4d1ca80ccffdde5d23d6ff6))
 * **official-bots:** make HybridBot veto actionable and use it for expert ([1df29cf](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/1df29cf3a6e21af3f396b2b7a6da67d978f941ae))
 * **official-bots:** park expert bot on tournament server at startup ([#75](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/75)) ([30295a4](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/30295a4bb95855ee8261c92278bb9ebc80ee12ee))
 * **official-bots:** resolve tournament bot token from Redis and account service ([386ddc5](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/386ddc5c19f8f893b16c6422aa5393b54c872e45))
 * **tournament:** auto-join external tournaments and publish created ones ([#77](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/77)) ([9978b7e](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/9978b7ea78eb658a225a461b9cd339386c0c14f3))
 * **tournament:** federate tournaments across clusters with DB replication ([5b000a6](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/5b000a6e5f04ea6770d1c7ab6bfdaded77a99172))
 * **tournament:** seed external server registry from env var on startup ([845dc9c](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/845dc9c2935c8bc1be42541dfaf31c9a861d3272))
 * true-microservices ([#40](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/40)) ([5909242](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/590924254e8a2754de661a57a03e43f89ceb6299))
 ### Bug Fixes
 * enable official bots to connect to external tournament server ([#71](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/71)) ([688d30e](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/688d30e2b10026923372be5fca3c63eaaee2de2a))
 * **official-bots:** configure JWT verification ([#72](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/72)) ([98c64fc](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/98c64fc0d56dc542beb31c75f4b9056d91de03cd))
 * **official-bots:** correct parkOn path from /api/bots to /api/account/bots ([1be9949](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/1be9949c0b5c6a1db535696620d77735050d6c93))
 * **official-bots:** derive tournament game color from game endpoint ([#79](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/79)) ([bfc4672](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/bfc46723e615bb9b65f7f9bba5f53877c4f079a7))
 * **official-bots:** discover tournament games by polling, not just the stream ([10113fd](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/10113fd0579b614d15870798d933bc9c495d2049))
 * **official-bots:** make botToken optional, fall back to env, fix 502 status ([f43d193](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/f43d1930d80670d810c57b54eaa3789854fa082c))
 * **official-bots:** NCS-70-auto-register official bots with account service ([#59](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/59)) ([7117a93](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/7117a93376272094d0b1a6abf2121254ce396684))
 * **official-bots:** park on external tournament servers using correct endpoint and token ([3188241](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/31882417377468b41bbe3ff94506aa4928024450))
 * **official-bots:** play games by polling state instead of NDJSON stream ([bfb15c7](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/bfb15c7299bd471d5e064a577ed10af98e2ea90a))
 * **official-bots:** play only own tournament games with correct color ([4651bb7](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/4651bb796f07a21bd013d9521b2dfe2e1078cebb))
 * **official-bots:** prioritize Redis token over stale env var in joinTournament ([83dd2d4](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/83dd2d4335ca48eb3e5aa234a75367574276ba63))
 * **official-bots:** register with tournament server directly to get correct token ([64b5d55](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/64b5d5567f110c2fe152558c7de275a1e0b30e21))
 * **official-bots:** resolve per-difficulty bot token on tournament join ([fdf4c94](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/fdf4c94811d086996447bb4657fac1d9bd6e5a93))
 * **official-bots:** resume tournaments already joined after restart ([285b73e](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/285b73efbd6dd98cec410ade9eead9881d693a8f))
 * **official-bots:** sync bots before token fetch on first startup after DB wipe ([b0ddb27](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/b0ddb274d23bca8b1b3f691ce0d643f33e0b54cd))
 ### Reverts
 * Revert "refactor: update metrics paths formatting in application.yml for clarity" ([3870566](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/38705663498d5f47c40dafe2f26198589ede8656))
 ##  (2026-06-23)
 ### Features
 * add initialization metrics for various services ([d438e97](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/d438e97f32bdde0bfc63c1b4a8cc810cdd093166))
 * add OpenTelemetry trace configuration with parentbased sampler ([3904d5a](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/3904d5ad8ad4930ddee65287a7bfab785a6148f5))
 * **analytics:** add Spark batch analytics module ([#70](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/70)) ([39f1657](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/39f1657e1db6e84889af338c43be8cb5c03c3ec3))
 * **config:** update application.yml for PostgreSQL and remove staging/production configurations ([2404e61](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/2404e6164c3b50ffccbea5238d636060d6abe4d6))
 * **config:** update application.yml for staging and production environments ([6113432](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/6113432a14c476a3a0dfc0d449e17d023697f2ba))
 * configure logging and add OpenTelemetry support ([#49](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/49)) ([d57c488](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/d57c4886612d1d92da0e1b79209fc83e6ef537a1))
 * **docker:** add .dockerignore and .gitignore files for build exclusions ([c987d8e](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/c987d8e258c0e6c4cfbdaa8381c64c410d7a2b83))
 * **docker:** add Dockerfiles for building Quarkus application in native and JVM modes ([3f2d2bb](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/3f2d2bb4c97fa8cddba66e1da4427c54236dfeed))
 * **docker:** add Dockerfiles for Quarkus application in JVM and native modes ([34b9933](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/34b993304670cf2aa62cd2f6460cee7b9864b08e))
 * **events:** migrate game-creation and bot flows to Redis Streams NCS-89 ([#62](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/62)) ([a24924c](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/a24924c23057db3d700a75dbc4333557789cd991))
 * NCS-78 Add Traceability to the Applications ([#46](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/46)) ([649566e](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/649566eb3fcf38f91c8896a739f74ea318af312d))
 * NCS-78 Add Traceability to the Applications ([#47](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/47)) ([87dfc6c](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/87dfc6c2bcce7f7d58fc641bd8d468a2e584c108))
 * NCS-82 add Swiss-system tournament module ([#55](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/55)) ([c5661de](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/c5661de4a0ebf4b33211f5a391840dcf744656b7))
 * **official-bots:** activate opening book in expert bot (native-safe) ([260db25](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/260db25803ec55ce99e55782791eabdc190dfed4))
 * **official-bots:** consume GameOver stream for bot cleanup ([#67](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/67)) ([db9d153](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/db9d1533912f4b41c4d1ca80ccffdde5d23d6ff6))
 * **official-bots:** make HybridBot veto actionable and use it for expert ([1df29cf](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/1df29cf3a6e21af3f396b2b7a6da67d978f941ae))
 * **official-bots:** park expert bot on tournament server at startup ([#75](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/75)) ([30295a4](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/30295a4bb95855ee8261c92278bb9ebc80ee12ee))
 * **official-bots:** resolve tournament bot token from Redis and account service ([386ddc5](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/386ddc5c19f8f893b16c6422aa5393b54c872e45))
 * **tournament:** auto-join external tournaments and publish created ones ([#77](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/77)) ([9978b7e](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/9978b7ea78eb658a225a461b9cd339386c0c14f3))
 * **tournament:** federate tournaments across clusters with DB replication ([5b000a6](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/5b000a6e5f04ea6770d1c7ab6bfdaded77a99172))
 * **tournament:** seed external server registry from env var on startup ([845dc9c](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/845dc9c2935c8bc1be42541dfaf31c9a861d3272))
 * true-microservices ([#40](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/40)) ([5909242](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/590924254e8a2754de661a57a03e43f89ceb6299))
 ### Bug Fixes
 * enable official bots to connect to external tournament server ([#71](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/71)) ([688d30e](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/688d30e2b10026923372be5fca3c63eaaee2de2a))
 * **official-bots:** configure JWT verification ([#72](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/72)) ([98c64fc](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/98c64fc0d56dc542beb31c75f4b9056d91de03cd))
 * **official-bots:** correct parkOn path from /api/bots to /api/account/bots ([1be9949](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/1be9949c0b5c6a1db535696620d77735050d6c93))
 * **official-bots:** derive tournament game color from game endpoint ([#79](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/79)) ([bfc4672](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/bfc46723e615bb9b65f7f9bba5f53877c4f079a7))
 * **official-bots:** discover tournament games by polling, not just the stream ([10113fd](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/10113fd0579b614d15870798d933bc9c495d2049))
 * **official-bots:** make botToken optional, fall back to env, fix 502 status ([f43d193](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/f43d1930d80670d810c57b54eaa3789854fa082c))
 * **official-bots:** NCS-70-auto-register official bots with account service ([#59](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/59)) ([7117a93](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/7117a93376272094d0b1a6abf2121254ce396684))
 * **official-bots:** park on external tournament servers using correct endpoint and token ([3188241](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/31882417377468b41bbe3ff94506aa4928024450))
 * **official-bots:** play games by polling state instead of NDJSON stream ([bfb15c7](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/bfb15c7299bd471d5e064a577ed10af98e2ea90a))
 * **official-bots:** play only own tournament games with correct color ([4651bb7](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/4651bb796f07a21bd013d9521b2dfe2e1078cebb))
 * **official-bots:** prioritize Redis token over stale env var in joinTournament ([83dd2d4](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/83dd2d4335ca48eb3e5aa234a75367574276ba63))
 * **official-bots:** register with tournament server directly to get correct token ([64b5d55](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/64b5d5567f110c2fe152558c7de275a1e0b30e21))
 * **official-bots:** resolve per-difficulty bot token on tournament join ([fdf4c94](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/fdf4c94811d086996447bb4657fac1d9bd6e5a93))
 * **official-bots:** resume tournaments already joined after restart ([285b73e](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/285b73efbd6dd98cec410ade9eead9881d693a8f))
 * **official-bots:** sync bots before token fetch on first startup after DB wipe ([b0ddb27](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/b0ddb274d23bca8b1b3f691ce0d643f33e0b54cd))
 * **official-bots:** use ThreadLocalRandom in PolyglotBook for native image ([1b30c3b](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/1b30c3be393d25712c8743d3d9057207f8bbb67c))
 ### Reverts
 * Revert "refactor: update metrics paths formatting in application.yml for clarity" ([3870566](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/38705663498d5f47c40dafe2f26198589ede8656))
 ##  (2026-06-24)
 ### Features
 * add initialization metrics for various services ([d438e97](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/d438e97f32bdde0bfc63c1b4a8cc810cdd093166))
 * add OpenTelemetry trace configuration with parentbased sampler ([3904d5a](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/3904d5ad8ad4930ddee65287a7bfab785a6148f5))
 * **analytics:** add Spark batch analytics module ([#70](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/70)) ([39f1657](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/39f1657e1db6e84889af338c43be8cb5c03c3ec3))
 * **config:** update application.yml for PostgreSQL and remove staging/production configurations ([2404e61](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/2404e6164c3b50ffccbea5238d636060d6abe4d6))
 * **config:** update application.yml for staging and production environments ([6113432](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/6113432a14c476a3a0dfc0d449e17d023697f2ba))
 * configure logging and add OpenTelemetry support ([#49](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/49)) ([d57c488](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/d57c4886612d1d92da0e1b79209fc83e6ef537a1))
 * **docker:** add .dockerignore and .gitignore files for build exclusions ([c987d8e](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/c987d8e258c0e6c4cfbdaa8381c64c410d7a2b83))
 * **docker:** add Dockerfiles for building Quarkus application in native and JVM modes ([3f2d2bb](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/3f2d2bb4c97fa8cddba66e1da4427c54236dfeed))
 * **docker:** add Dockerfiles for Quarkus application in JVM and native modes ([34b9933](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/34b993304670cf2aa62cd2f6460cee7b9864b08e))
 * **events:** migrate game-creation and bot flows to Redis Streams NCS-89 ([#62](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/62)) ([a24924c](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/a24924c23057db3d700a75dbc4333557789cd991))
 * NCS-78 Add Traceability to the Applications ([#46](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/46)) ([649566e](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/649566eb3fcf38f91c8896a739f74ea318af312d))
 * NCS-78 Add Traceability to the Applications ([#47](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/47)) ([87dfc6c](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/87dfc6c2bcce7f7d58fc641bd8d468a2e584c108))
 * NCS-82 add Swiss-system tournament module ([#55](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/55)) ([c5661de](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/c5661de4a0ebf4b33211f5a391840dcf744656b7))
 * **official-bots:** activate opening book in expert bot (native-safe) ([260db25](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/260db25803ec55ce99e55782791eabdc190dfed4))
 * **official-bots:** add Google Colab notebook for NNUE training (NCS-111) ([#81](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/81)) ([fa10852](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/fa10852bc98451d4068ec6fb9e7a486b5e53ef5c))
 * **official-bots:** consume GameOver stream for bot cleanup ([#67](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/67)) ([db9d153](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/db9d1533912f4b41c4d1ca80ccffdde5d23d6ff6))
 * **official-bots:** implement king-relative (HalfKP) encoding in NNUE (NCS-109) ([#80](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/80)) ([44f376f](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/44f376f03221f086b898741436e13c93fd314dd1))
 * **official-bots:** make HybridBot veto actionable and use it for expert ([1df29cf](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/1df29cf3a6e21af3f396b2b7a6da67d978f941ae))
 * **official-bots:** park expert bot on tournament server at startup ([#75](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/75)) ([30295a4](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/30295a4bb95855ee8261c92278bb9ebc80ee12ee))
 * **official-bots:** resolve tournament bot token from Redis and account service ([386ddc5](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/386ddc5c19f8f893b16c6422aa5393b54c872e45))
 * **tournament:** auto-join external tournaments and publish created ones ([#77](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/77)) ([9978b7e](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/9978b7ea78eb658a225a461b9cd339386c0c14f3))
 * **tournament:** federate tournaments across clusters with DB replication ([5b000a6](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/5b000a6e5f04ea6770d1c7ab6bfdaded77a99172))
 * **tournament:** seed external server registry from env var on startup ([845dc9c](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/845dc9c2935c8bc1be42541dfaf31c9a861d3272))
 * true-microservices ([#40](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/40)) ([5909242](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/590924254e8a2754de661a57a03e43f89ceb6299))
 ### Bug Fixes
 * enable official bots to connect to external tournament server ([#71](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/71)) ([688d30e](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/688d30e2b10026923372be5fca3c63eaaee2de2a))
 * modified training pipeline ([9f9140c](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/9f9140cb585345cd244a1dfee1a06e51a5f7f7a8))
 * **official-bots:** configure JWT verification ([#72](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/72)) ([98c64fc](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/98c64fc0d56dc542beb31c75f4b9056d91de03cd))
 * **official-bots:** correct parkOn path from /api/bots to /api/account/bots ([1be9949](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/1be9949c0b5c6a1db535696620d77735050d6c93))
 * **official-bots:** derive tournament game color from game endpoint ([#79](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/79)) ([bfc4672](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/bfc46723e615bb9b65f7f9bba5f53877c4f079a7))
 * **official-bots:** discover tournament games by polling, not just the stream ([10113fd](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/10113fd0579b614d15870798d933bc9c495d2049))
 * **official-bots:** make botToken optional, fall back to env, fix 502 status ([f43d193](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/f43d1930d80670d810c57b54eaa3789854fa082c))
 * **official-bots:** NCS-70-auto-register official bots with account service ([#59](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/59)) ([7117a93](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/7117a93376272094d0b1a6abf2121254ce396684))
 * **official-bots:** park on external tournament servers using correct endpoint and token ([3188241](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/31882417377468b41bbe3ff94506aa4928024450))
 * **official-bots:** play games by polling state instead of NDJSON stream ([bfb15c7](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/bfb15c7299bd471d5e064a577ed10af98e2ea90a))
 * **official-bots:** play only own tournament games with correct color ([4651bb7](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/4651bb796f07a21bd013d9521b2dfe2e1078cebb))
 * **official-bots:** prioritize Redis token over stale env var in joinTournament ([83dd2d4](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/83dd2d4335ca48eb3e5aa234a75367574276ba63))
 * **official-bots:** register with tournament server directly to get correct token ([64b5d55](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/64b5d5567f110c2fe152558c7de275a1e0b30e21))
 * **official-bots:** resolve per-difficulty bot token on tournament join ([fdf4c94](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/fdf4c94811d086996447bb4657fac1d9bd6e5a93))
 * **official-bots:** resume tournaments already joined after restart ([285b73e](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/285b73efbd6dd98cec410ade9eead9881d693a8f))
 * **official-bots:** sync bots before token fetch on first startup after DB wipe ([b0ddb27](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/b0ddb274d23bca8b1b3f691ce0d643f33e0b54cd))
 * **official-bots:** use ThreadLocalRandom in PolyglotBook for native image ([1b30c3b](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/1b30c3be393d25712c8743d3d9057207f8bbb67c))
 ### Reverts
 * Revert "refactor: update metrics paths formatting in application.yml for clarity" ([3870566](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/38705663498d5f47c40dafe2f26198589ede8656))
@@ -0,0 +1,377 @@
 {
 "nbformat": 4,
 "nbformat_minor": 5,
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.10.0"
  },
  "colab": {
   "provenance": [],
   "gpuType": "T4"
  },
  "accelerator": "GPU"
 },
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# NNUE Training Pipeline\n",
    "\n",
    "End-to-end notebook: data generation → Stockfish labeling → training → `.nbai` export.\n",
    "\n",
    "**Runtime:** GPU (T4 or better). Runtime → Change runtime type → T4 GPU.\n",
    "\n",
    "**Persistence:** Checkpoints and datasets are saved to Google Drive so training can resume after session timeout."
   ],
   "id": "intro-md"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## ⚙️ 1 — Setup"
   ],
   "id": "setup-md"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Mount Google Drive for checkpoint persistence\n",
    "from google.colab import drive\n",
    "drive.mount('/content/drive')"
   ],
   "id": "mount-drive"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "# ── Configure these paths once ───────────────────────────────────────────────\n",
    "REPO_URL     = 'https://git.janis-eccarius.de/NowChess/NowChessSystems.git'\n",
    "DRIVE_ROOT   = '/content/drive/MyDrive/NowChess'\n",
    "REPO_DIR     = f'{DRIVE_ROOT}/NowChessSystems'\n",
    "PYTHON_DIR   = f'{REPO_DIR}/modules/official-bots/python'\n",
    "# ─────────────────────────────────────────────────────────────────────────────\n",
    "\n",
    "os.makedirs(DRIVE_ROOT, exist_ok=True)\n",
    "\n",
    "if not os.path.isdir(REPO_DIR):\n",
    "    !git clone --depth=1 \"{REPO_URL}\" \"{REPO_DIR}\"\n",
    "    print('Repo cloned to Drive.')\n",
    "else:\n",
    "    !git -C \"{REPO_DIR}\" pull --ff-only\n",
    "    print('Repo updated.')"
   ],
   "id": "clone-repo"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Install Python dependencies\n",
    "!pip install -q chess tqdm rich zstandard\n",
    "\n",
    "# Stockfish for position labeling\n",
    "!apt-get install -q -y stockfish\n",
    "import shutil\n",
    "STOCKFISH_PATH = shutil.which('stockfish') or '/usr/games/stockfish'\n",
    "print(f'Stockfish: {STOCKFISH_PATH}')\n",
    "\n",
    "# Add pipeline source to path\n",
    "import sys\n",
    "sys.path.insert(0, f'{PYTHON_DIR}/src')\n",
    "sys.path.insert(0, PYTHON_DIR)\n",
    "print('Python path configured.')"
   ],
   "id": "install-deps"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## 🗄️ 2 — Data\n",
    "\n",
    "Choose **one** of the two options below:\n",
    "- **Option A** — generate FEN positions with random play, then label them with Stockfish.\n",
    "- **Option B** — upload an existing `labeled.jsonl` from your machine or Drive."
   ],
   "id": "data-md"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "\n",
    "# Paths (all on Drive so they survive session restarts)\n",
    "DATA_DIR      = Path(DRIVE_ROOT) / 'training_data'\n",
    "DATA_DIR.mkdir(parents=True, exist_ok=True)\n",
    "POSITIONS_FILE = DATA_DIR / 'positions.txt'   # raw FENs\n",
    "LABELED_FILE   = DATA_DIR / 'labeled.jsonl'   # FEN + eval pairs\n",
    "\n",
    "print(f'Data directory: {DATA_DIR}')"
   ],
   "id": "data-paths"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ── Option A: Generate + label ────────────────────────────────────────────────\n",
    "# Adjust NUM_POSITIONS to taste. 50 000 trains in ~10 min on T4;\n",
    "# 200 000+ gives better generalisation.\n",
    "NUM_POSITIONS    = 50_000\n",
    "STOCKFISH_DEPTH  = 12\n",
    "LABEL_WORKERS    = 4       # parallel Stockfish processes\n",
    "MIN_MOVE         = 5       # skip opening book moves\n",
    "MAX_MOVE         = 60\n",
    "\n",
    "from generate import play_random_game_and_collect_positions\n",
    "from label    import label_positions_with_stockfish\n",
    "\n",
    "print(f'Generating {NUM_POSITIONS:,} positions...')\n",
    "count = play_random_game_and_collect_positions(\n",
    "    str(POSITIONS_FILE),\n",
    "    total_positions=NUM_POSITIONS,\n",
    "    samples_per_game=1,\n",
    "    min_move=MIN_MOVE,\n",
    "    max_move=MAX_MOVE,\n",
    "    num_workers=4,\n",
    ")\n",
    "print(f'{count:,} positions written to {POSITIONS_FILE}')\n",
    "\n",
    "print('Labeling with Stockfish (this is the slow step)...')\n",
    "ok = label_positions_with_stockfish(\n",
    "    str(POSITIONS_FILE),\n",
    "    str(LABELED_FILE),\n",
    "    STOCKFISH_PATH,\n",
    "    depth=STOCKFISH_DEPTH,\n",
    "    num_workers=LABEL_WORKERS,\n",
    ")\n",
    "if ok:\n",
    "    print(f'Labeled dataset saved: {LABELED_FILE}')\n",
    "else:\n",
    "    print('ERROR: labeling failed')"
   ],
   "id": "option-a-generate"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ── Option B: Upload existing labeled.jsonl ───────────────────────────────────\n",
    "# Run this cell instead of Option A if you already have a labeled dataset.\n",
    "#\n",
    "# To upload from local machine:\n",
    "#   from google.colab import files\n",
    "#   uploaded = files.upload()   # pick your labeled.jsonl\n",
    "#   import shutil, os\n",
    "#   shutil.move(next(iter(uploaded)), str(LABELED_FILE))\n",
    "#\n",
    "# Or copy from Drive:\n",
    "#   import shutil\n",
    "#   shutil.copy('/content/drive/MyDrive/path/to/labeled.jsonl', str(LABELED_FILE))\n",
    "\n",
    "import os\n",
    "if LABELED_FILE.exists():\n",
    "    lines = sum(1 for _ in open(LABELED_FILE))\n",
    "    print(f'Ready: {lines:,} labeled positions at {LABELED_FILE}')\n",
    "else:\n",
    "    print('No labeled.jsonl found — run Option A first or upload one.')"
   ],
   "id": "option-b-upload"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## 🏋️ 3 — Train\n",
    "\n",
    "Standard training runs a fixed number of epochs.  \n",
    "**Burst mode** is better for Colab: it repeatedly restarts from the best checkpoint within a time budget, surviving session disconnects gracefully."
   ],
   "id": "train-md"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from train import train_nnue, burst_train, DEFAULT_HIDDEN_SIZES\n",
    "\n",
    "WEIGHTS_DIR = Path(DRIVE_ROOT) / 'weights'\n",
    "WEIGHTS_DIR.mkdir(parents=True, exist_ok=True)\n",
    "OUTPUT_FILE = str(WEIGHTS_DIR / 'nnue_weights.pt')\n",
    "\n",
    "# ── Training hyperparameters ──────────────────────────────────────────────────\n",
    "HIDDEN_SIZES      = DEFAULT_HIDDEN_SIZES   # [1536, 1024, 512, 256]\n",
    "BATCH_SIZE        = 16384\n",
    "EPOCHS            = 100\n",
    "EARLY_STOPPING    = 10                     # None to disable\n",
    "SUBSAMPLE_RATIO   = 1.0\n",
    "\n",
    "# Resume from latest checkpoint if one exists\n",
    "checkpoints = sorted(WEIGHTS_DIR.glob('nnue_weights_v*.pt'))\n",
    "CHECKPOINT = str(checkpoints[-1]) if checkpoints else None\n",
    "if CHECKPOINT:\n",
    "    print(f'Resuming from checkpoint: {CHECKPOINT}')\n",
    "else:\n",
    "    print('Starting training from scratch.')"
   ],
   "id": "train-config"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ── Standard training ─────────────────────────────────────────────────────────\n",
    "# Use this when you have a reliable long-running session.\n",
    "\n",
    "train_nnue(\n",
    "    data_file=str(LABELED_FILE),\n",
    "    output_file=OUTPUT_FILE,\n",
    "    epochs=EPOCHS,\n",
    "    batch_size=BATCH_SIZE,\n",
    "    checkpoint=CHECKPOINT,\n",
    "    use_versioning=True,\n",
    "    early_stopping_patience=EARLY_STOPPING,\n",
    "    subsample_ratio=SUBSAMPLE_RATIO,\n",
    "    hidden_sizes=HIDDEN_SIZES,\n",
    ")"
   ],
   "id": "standard-train"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ── Burst training (recommended for Colab free tier) ─────────────────────────\n",
    "# Restarts from the global best each time early stopping fires.\n",
    "# Set BURST_MINUTES to slightly less than the Colab session limit (~70 min).\n",
    "\n",
    "BURST_MINUTES      = 70\n",
    "EPOCHS_PER_SEASON  = 30\n",
    "BURST_PATIENCE     = 8\n",
    "\n",
    "burst_train(\n",
    "    data_file=str(LABELED_FILE),\n",
    "    output_file=OUTPUT_FILE,\n",
    "    duration_minutes=BURST_MINUTES,\n",
    "    epochs_per_season=EPOCHS_PER_SEASON,\n",
    "    early_stopping_patience=BURST_PATIENCE,\n",
    "    batch_size=BATCH_SIZE,\n",
    "    initial_checkpoint=CHECKPOINT,\n",
    "    use_versioning=True,\n",
    "    subsample_ratio=SUBSAMPLE_RATIO,\n",
    "    hidden_sizes=HIDDEN_SIZES,\n",
    ")"
   ],
   "id": "burst-train"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## 📦 4 — Export\n",
    "\n",
    "Convert the best `.pt` checkpoint to the `.nbai` binary format read by `NbaiLoader` in Scala."
   ],
   "id": "export-md"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from export import export_to_nbai\n",
    "\n",
    "NBAI_FILE = Path(DRIVE_ROOT) / 'nnue_weights.nbai'\n",
    "\n",
    "# Pick the latest versioned checkpoint\n",
    "checkpoints = sorted(WEIGHTS_DIR.glob('nnue_weights_v*.pt'))\n",
    "if not checkpoints:\n",
    "    raise FileNotFoundError('No checkpoints found in ' + str(WEIGHTS_DIR))\n",
    "\n",
    "latest = checkpoints[-1]\n",
    "print(f'Exporting {latest.name} → {NBAI_FILE.name}')\n",
    "\n",
    "export_to_nbai(\n",
    "    weights_file=str(latest),\n",
    "    output_file=str(NBAI_FILE),\n",
    "    trained_by='colab',\n",
    ")\n",
    "print('Export complete.')"
   ],
   "id": "export-cell"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## ⬇️ 5 — Download\n",
    "\n",
    "Download the `.nbai` weights file and the latest `.pt` checkpoint to your local machine.\n",
    "\n",
    "Place `nnue_weights.nbai` in `modules/official-bots/src/main/resources/` and rebuild the native image."
   ],
   "id": "download-md"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from google.colab import files\n",
    "\n",
    "if NBAI_FILE.exists():\n",
    "    files.download(str(NBAI_FILE))\n",
    "    print(f'Downloading {NBAI_FILE.name}')\n",
    "else:\n",
    "    print('No .nbai file found — run the Export cell first.')\n",
    "\n",
    "checkpoints = sorted(WEIGHTS_DIR.glob('nnue_weights_v*.pt'))\n",
    "if checkpoints:\n",
    "    latest = checkpoints[-1]\n",
    "    files.download(str(latest))\n",
    "    print(f'Downloading checkpoint {latest.name}')\n",
    "else:\n",
    "    print('No .pt checkpoint found.')"
   ],
   "id": "download-cell"
  }
 ]
 }
@@ -53,6 +53,11 @@ class NNUEDataset(Dataset):
        eval_val = self.evals[idx]
        features = fen_to_features(fen)
        # Board is flipped for Black-to-move in fen_to_features; negate eval
        # so the label still means "good for the side shown as White after flip"
        if ' b ' in fen:
            eval_val = -eval_val
        # Use evaluation as-is if normalized, otherwise apply sigmoid scaling
        if self.is_normalized:
            target = torch.tensor(eval_val, dtype=torch.float32)
@@ -61,38 +66,59 @@ class NNUEDataset(Dataset):
        return features, target
 # King-relative (HalfKP) encoding: two perspectives, one per side's king.
 # Each piece is encoded as:  kingSq * 768 + pieceIdx * 64 + sq
 # White perspective uses white king square; black perspective uses black king square.
 # Total input dimension = 2 × 64 × 12 × 64 = 98304.
 _HALF_SIZE = 64 * 12 * 64   # 49152 features per perspective
 INPUT_SIZE = _HALF_SIZE * 2  # 98304
 _PIECE_TO_IDX = {
    'p': 0, 'n': 1, 'b': 2, 'r': 3, 'q': 4, 'k': 5,
    'P': 6, 'N': 7, 'B': 8, 'R': 9, 'Q': 10, 'K': 11,
 }
 def fen_to_features(fen):
-    """Convert FEN to 768-dimensional binary feature vector."""
+    """Convert FEN to 98304-dim king-relative (HalfKP) feature vector.
    # Piece type to index: pawn=0, knight=1, bishop=2, rook=3, queen=4, king=5
    piece_to_idx = {'p': 0, 'n': 1, 'b': 2, 'r': 3, 'q': 4, 'k': 5,
                    'P': 6, 'N': 7, 'B': 8, 'R': 9, 'Q': 10, 'K': 11}
    features = torch.zeros(768, dtype=torch.float32)
    For Black-to-move positions the board is mirrored (ranks flipped, colours
    swapped) so the network always sees the position from the side-to-move's
    perspective.  The caller is responsible for negating the eval label to match.
    """
    features = torch.zeros(INPUT_SIZE, dtype=torch.float32)
    try:
        board = chess.Board(fen)
-
+        # Perspective flip: present all positions as if White is to move
-        # 12 piece types × 64 squares = 768
+        if board.turn == chess.BLACK:
-        for square in chess.SQUARES:
+            board = board.mirror()
-            piece = board.piece_at(square)
+        wk = board.king(chess.WHITE)
-            if piece is not None:
+        bk = board.king(chess.BLACK)
-                piece_char = piece.symbol()
+        if wk is None or bk is None:
-                if piece_char in piece_to_idx:
+            return features
-                    piece_idx = piece_to_idx[piece_char]
+        for sq in chess.SQUARES:
-                    feature_idx = piece_idx * 64 + square
+            piece = board.piece_at(sq)
-                    features[feature_idx] = 1.0
+            if piece is None:
-    except:
+                continue
            pidx = _PIECE_TO_IDX[piece.symbol()]
            # White-king perspective (indices 0 .. _HALF_SIZE-1)
            features[wk * 768 + pidx * 64 + sq] = 1.0
            # Black-king perspective (indices _HALF_SIZE .. INPUT_SIZE-1)
            features[_HALF_SIZE + bk * 768 + pidx * 64 + sq] = 1.0
    except Exception:
        pass
    return features
-DEFAULT_HIDDEN_SIZES = [1536, 1024, 512, 256]
+# Smaller hidden layers are appropriate: the L1 input is very sparse (~64 active
 # features out of 98304) so the L1 itself is cheap to update incrementally; the
 # larger capacity comes from the wider perspective encoding, not deeper layers.
 DEFAULT_HIDDEN_SIZES = [512, 256, 128]
 class NNUE(nn.Module):
    """NNUE neural network with configurable hidden layers.
-    Architecture: 768 → hidden_sizes[0] → ... → hidden_sizes[-1] → 1
+    Architecture: INPUT_SIZE → hidden_sizes[0] → ... → hidden_sizes[-1] → 1
    Layer attributes follow the naming l1, l2, ..., lN so export.py can
    infer the architecture directly from the state_dict.
    """
@@ -102,7 +128,7 @@ class NNUE(nn.Module):
        if hidden_sizes is None:
            hidden_sizes = DEFAULT_HIDDEN_SIZES
        self.hidden_sizes = list(hidden_sizes)
-        sizes = [768] + self.hidden_sizes + [1]
+        sizes = [INPUT_SIZE] + self.hidden_sizes + [1]
        num_hidden = len(self.hidden_sizes)
        for i in range(num_hidden):
@@ -1,17 +1,20 @@
 package de.nowchess.bot
 import de.nowchess.bot.bots.{ClassicalBot, HybridBot}
 import de.nowchess.bot.util.PolyglotBook
 import jakarta.enterprise.context.ApplicationScoped
 import org.jboss.logging.Logger
 object BotController:
  private val log = Logger.getLogger(classOf[BotController])
  private val openingBook = PolyglotBook.fromResource("/opening_book.bin")
  private val bots: Map[String, Bot] = Map(
    "easy"   -> ClassicalBot(BotDifficulty.Easy),
    "medium" -> ClassicalBot(BotDifficulty.Medium),
    "hard"   -> ClassicalBot(BotDifficulty.Hard),
-    "expert" -> HybridBot(BotDifficulty.Expert, vetoReporter = log.debug(_)),
+    "expert" -> HybridBot(BotDifficulty.Expert, vetoReporter = log.debug(_), book = Some(openingBook)),
  )
  def getBot(name: String): Option[Bot] = bots.get(name.toLowerCase)
@@ -23,9 +23,9 @@ object EvaluationNNUE extends Evaluation:
    nnue.copyAccumulator(parentPly, childPly)
  override def pushAccumulator(childPly: Int, move: Move, parent: GameContext, child: GameContext): Unit =
-    // Use incremental updates, but recompute from scratch every 10 plies to prevent accumulation errors
+    // Recompute every 10 plies to prevent floating-point drift; king moves always recompute internally
    if childPly % 10 == 0 then nnue.recomputeAccumulator(childPly, child.board)
-    else nnue.pushAccumulator(childPly, move, parent.board)
+    else nnue.pushAccumulator(childPly, move, parent.board, child.board)
  override def evaluateAccumulator(ply: Int, context: GameContext, hash: Long): Int =
    nnue.evaluateAtPlyWithValidation(ply, context.turn, hash, context.board)
@@ -1,17 +1,17 @@
 package de.nowchess.bot.bots.nnue
-import de.nowchess.api.board.{Board, Color, File, Piece, PieceType, Square}
+import de.nowchess.api.board.{Board, Color, Piece, PieceType, Square}
 import de.nowchess.api.game.GameContext
 import de.nowchess.api.move.{Move, MoveType, PromotionPiece}
 class NNUE(model: NbaiModel):
-  private val featureSize   = model.layers(0).inputSize
+  private val HALF_SIZE     = 49152                     // 64 king-squares × 12 piece-types × 64 piece-squares
  private val featureSize   = model.layers(0).inputSize // 98304 (= HALF_SIZE * 2) for king-relative
  private val accSize       = model.layers(0).outputSize
-  private val validateAccum = sys.env.contains("NNUE_VALIDATE") // Enable with NNUE_VALIDATE=1
+  private val validateAccum = sys.env.contains("NNUE_VALIDATE")
-  // Column-major L1 weights for cache-friendly sparse & incremental updates.
+  // Column-major L1 weights: l1WeightsT(featureIdx * accSize + outputIdx)
  // l1WeightsT(featureIdx * accSize + outputIdx) = l1Weights(outputIdx * featureSize + featureIdx)
  private val l1WeightsT: Array[Float] =
    val w = model.weights(0).weights
    val t = new Array[Float](featureSize * accSize)
@@ -23,7 +23,6 @@ class NNUE(model: NbaiModel):
  private val MAX_PLY                      = 128
  private val l1Stack: Array[Array[Float]] = Array.fill(MAX_PLY + 1)(new Array[Float](accSize))
  // Shared evaluation buffers: index i holds the output of layers(i) (all except the scalar output layer).
  private val evalBuffers: Array[Array[Float]] = model.layers.init.map(l => new Array[Float](l.outputSize))
  // ── Eval cache ───────────────────────────────────────────────────────────
@@ -36,9 +35,29 @@ class NNUE(model: NbaiModel):
  private def squareNum(sq: Square): Int = sq.rank.ordinal * 8 + sq.file.ordinal
-  private def featureIndex(piece: Piece, sqNum: Int): Int =
+  // Mirror square vertically (rank 0 ↔ rank 7) for the perspective flip
-    val colorOffset = if piece.color == Color.White then 6 else 0
+  private def flipSqNum(sqNum: Int): Int = (7 - sqNum / 8) * 8 + sqNum % 8
-    (colorOffset + piece.pieceType.ordinal) * 64 + sqNum
+
  private def pieceIdx(piece: Piece): Int =
    if piece.color == Color.White then 6 + piece.pieceType.ordinal else piece.pieceType.ordinal
  // White-king perspective: index in [0, HALF_SIZE)
  private def featureIdxWhite(piece: Piece, sqNum: Int, wkSq: Int): Int =
    wkSq * 768 + pieceIdx(piece) * 64 + sqNum
  // Black-king perspective: index in [HALF_SIZE, featureSize)
  private def featureIdxBlack(piece: Piece, sqNum: Int, bkSq: Int): Int =
    HALF_SIZE + bkSq * 768 + pieceIdx(piece) * 64 + sqNum
  private def wkSqOf(board: Board): Int =
    board.pieces
      .collectFirst { case (sq, p) if p.pieceType == PieceType.King && p.color == Color.White => squareNum(sq) }
      .getOrElse(0)
  private def bkSqOf(board: Board): Int =
    board.pieces
      .collectFirst { case (sq, p) if p.pieceType == PieceType.King && p.color == Color.Black => squareNum(sq) }
      .getOrElse(0)
  private def addColumn(l1Pre: Array[Float], featureIdx: Int): Unit =
    val offset = featureIdx * accSize
@@ -48,92 +67,96 @@ class NNUE(model: NbaiModel):
    val offset = featureIdx * accSize
    for i <- 0 until accSize do l1Pre(i) -= l1WeightsT(offset + i)
  private def addPiece(l1: Array[Float], piece: Piece, sqNum: Int, wkSq: Int, bkSq: Int): Unit =
    addColumn(l1, featureIdxWhite(piece, sqNum, wkSq))
    addColumn(l1, featureIdxBlack(piece, sqNum, bkSq))
  private def removePiece(l1: Array[Float], piece: Piece, sqNum: Int, wkSq: Int, bkSq: Int): Unit =
    subtractColumn(l1, featureIdxWhite(piece, sqNum, wkSq))
    subtractColumn(l1, featureIdxBlack(piece, sqNum, bkSq))
  // ── Accumulator init ─────────────────────────────────────────────────────
  def initAccumulator(board: Board): Unit =
    val wkSq = wkSqOf(board)
    val bkSq = bkSqOf(board)
    System.arraycopy(model.weights(0).bias, 0, l1Stack(0), 0, accSize)
-    for (sq, piece) <- board.pieces do addColumn(l1Stack(0), featureIndex(piece, squareNum(sq)))
+    for (sq, piece) <- board.pieces do addPiece(l1Stack(0), piece, squareNum(sq), wkSq, bkSq)
  // ── Accumulator push (incremental updates) ───────────────────────────────
-  def pushAccumulator(childPly: Int, move: Move, board: Board): Unit =
+  def pushAccumulator(childPly: Int, move: Move, parentBoard: Board, childBoard: Board): Unit =
    System.arraycopy(l1Stack(childPly - 1), 0, l1Stack(childPly), 0, accSize)
-    val l1 = l1Stack(childPly)
+    if isKingMove(move, parentBoard) then recomputeAccumulatorInto(l1Stack(childPly), childBoard)
-    move.moveType match
+    else applyNonKingDelta(l1Stack(childPly), move, parentBoard)
-      case MoveType.Normal(_)                                 => applyNormalDelta(l1, move, board)
+
-      case MoveType.EnPassant                                 => applyEnPassantDelta(l1, move, board)
+  private def isKingMove(move: Move, board: Board): Boolean =
-      case MoveType.CastleKingside | MoveType.CastleQueenside => applyCastleDelta(l1, move, board)
+    move.moveType == MoveType.CastleKingside ||
-      case MoveType.Promotion(p)                              => applyPromotionDelta(l1, move, p, board)
+      move.moveType == MoveType.CastleQueenside ||
      board.pieceAt(move.from).exists(_.pieceType == PieceType.King)
  def copyAccumulator(parentPly: Int, childPly: Int): Unit =
    System.arraycopy(l1Stack(parentPly), 0, l1Stack(childPly), 0, accSize)
  def recomputeAccumulator(ply: Int, board: Board): Unit =
-    System.arraycopy(model.weights(0).bias, 0, l1Stack(ply), 0, accSize)
+    recomputeAccumulatorInto(l1Stack(ply), board)
-    for (sq, piece) <- board.pieces do addColumn(l1Stack(ply), featureIndex(piece, squareNum(sq)))
+
  private def recomputeAccumulatorInto(l1: Array[Float], board: Board): Unit =
    val wkSq = wkSqOf(board)
    val bkSq = bkSqOf(board)
    System.arraycopy(model.weights(0).bias, 0, l1, 0, accSize)
    for (sq, piece) <- board.pieces do addPiece(l1, piece, squareNum(sq), wkSq, bkSq)
  def validateAccumulator(ply: Int, board: Board): Boolean =
-    // Compute what L1 should be from scratch
+    val expected = new Array[Float](accSize)
-    val expectedL1 = new Array[Float](accSize)
+    val wkSq     = wkSqOf(board)
-    System.arraycopy(model.weights(0).bias, 0, expectedL1, 0, accSize)
+    val bkSq     = bkSqOf(board)
-    for (sq, piece) <- board.pieces do addColumn(expectedL1, featureIndex(piece, squareNum(sq)))
+    System.arraycopy(model.weights(0).bias, 0, expected, 0, accSize)
-
+    for (sq, piece) <- board.pieces do addPiece(expected, piece, squareNum(sq), wkSq, bkSq)
    // Compare with actual L1
    val actual = l1Stack(ply)
-    val maxError =
+    (0 until accSize).forall(i => math.abs(actual(i) - expected(i)) < 0.001f)
      (0 until accSize).foldLeft(0f) { (currentMax, i) =>
        val error = math.abs(actual(i) - expectedL1(i))
        math.max(currentMax, error)
      }
-    maxError < 0.001f // Allow small floating-point errors
+  // ── Non-king incremental deltas ──────────────────────────────────────────
-  private def applyNormalDelta(l1: Array[Float], move: Move, board: Board): Unit =
+  private def applyNonKingDelta(l1: Array[Float], move: Move, board: Board): Unit =
-    // Extract source and destination square indices early
+    val wkSq = wkSqOf(board)
-    val fromNum = squareNum(move.from)
+    val bkSq = bkSqOf(board)
-    val toNum   = squareNum(move.to)
+    move.moveType match
      case MoveType.Normal(_)    => applyNormalDelta(l1, move, board, wkSq, bkSq)
      case MoveType.EnPassant    => applyEnPassantDelta(l1, move, board, wkSq, bkSq)
      case MoveType.Promotion(p) => applyPromotionDelta(l1, move, p, board, wkSq, bkSq)
      case _                     => () // king moves handled before this point
-    // Get the moving piece
+  private def applyNormalDelta(l1: Array[Float], move: Move, board: Board, wkSq: Int, bkSq: Int): Unit =
    board.pieceAt(move.from).foreach { mover =>
-      subtractColumn(l1, featureIndex(mover, fromNum))
+      val fromNum = squareNum(move.from)
-
+      val toNum   = squareNum(move.to)
-      // If there's a capture, subtract the captured piece
+      removePiece(l1, mover, fromNum, wkSq, bkSq)
-      board.pieceAt(move.to).foreach { cap =>
+      board.pieceAt(move.to).foreach(cap => removePiece(l1, cap, toNum, wkSq, bkSq))
-        subtractColumn(l1, featureIndex(cap, toNum))
+      addPiece(l1, mover, toNum, wkSq, bkSq)
      }
      // Add the piece to its new location
      addColumn(l1, featureIndex(mover, toNum))
    }
-  private def applyEnPassantDelta(l1: Array[Float], move: Move, board: Board): Unit =
+  private def applyEnPassantDelta(l1: Array[Float], move: Move, board: Board, wkSq: Int, bkSq: Int): Unit =
    board.pieceAt(move.from).foreach { pawn =>
      val capturedSq = Square(move.to.file, move.from.rank)
-      subtractColumn(l1, featureIndex(pawn, squareNum(move.from)))
+      removePiece(l1, pawn, squareNum(move.from), wkSq, bkSq)
-      board.pieceAt(capturedSq).foreach(cap => subtractColumn(l1, featureIndex(cap, squareNum(capturedSq))))
+      board.pieceAt(capturedSq).foreach(cap => removePiece(l1, cap, squareNum(capturedSq), wkSq, bkSq))
-      addColumn(l1, featureIndex(pawn, squareNum(move.to)))
+      addPiece(l1, pawn, squareNum(move.to), wkSq, bkSq)
    }
-  private def applyCastleDelta(l1: Array[Float], move: Move, board: Board): Unit =
+  private def applyPromotionDelta(
-    board.pieceAt(move.from).foreach { king =>
+      l1: Array[Float],
-      val rank     = move.from.rank
+      move: Move,
-      val kingside = move.moveType == MoveType.CastleKingside
+      promo: PromotionPiece,
-      val (rookFrom, rookTo) =
+      board: Board,
-        if kingside then (Square(File.H, rank), Square(File.F, rank))
+      wkSq: Int,
-        else (Square(File.A, rank), Square(File.D, rank))
+      bkSq: Int,
-      val rook = Piece(king.color, PieceType.Rook)
+  ): Unit =
      subtractColumn(l1, featureIndex(king, squareNum(move.from)))
      addColumn(l1, featureIndex(king, squareNum(move.to)))
      subtractColumn(l1, featureIndex(rook, squareNum(rookFrom)))
      addColumn(l1, featureIndex(rook, squareNum(rookTo)))
    }
  private def applyPromotionDelta(l1: Array[Float], move: Move, promo: PromotionPiece, board: Board): Unit =
    board.pieceAt(move.from).foreach { pawn =>
      val toNum = squareNum(move.to)
-      subtractColumn(l1, featureIndex(pawn, squareNum(move.from)))
+      removePiece(l1, pawn, squareNum(move.from), wkSq, bkSq)
-      board.pieceAt(move.to).foreach(cap => subtractColumn(l1, featureIndex(cap, toNum)))
+      board.pieceAt(move.to).foreach(cap => removePiece(l1, cap, toNum, wkSq, bkSq))
-      addColumn(l1, featureIndex(Piece(pawn.color, promotedType(promo)), toNum))
+      addPiece(l1, Piece(pawn.color, promotedType(promo)), toNum, wkSq, bkSq)
    }
  private def promotedType(promo: PromotionPiece): PieceType = promo match
@@ -154,7 +177,6 @@ class NNUE(model: NbaiModel):
      score
  def evaluateAtPlyWithValidation(ply: Int, turn: Color, hash: Long, board: Board): Int =
    // For debugging: validate that incremental accumulator matches recomputation
    if validateAccum && ply > 0 && ply % 10 != 0 then
      val isValid = validateAccumulator(ply, board)
      if !isValid then System.err.println(s"WARNING: NNUE accumulator diverged at ply $ply")
@@ -206,9 +228,23 @@ class NNUE(model: NbaiModel):
  private val legacyL1 = new Array[Float](accSize)
  def evaluate(context: GameContext): Int =
    // Match training: for Black-to-move positions, mirror the board (ranks flipped,
    // colours swapped) so the model always sees from the side-to-move's perspective.
    // The scoreFromOutput negation then converts back to White's absolute perspective.
    val (wkSq, bkSq, pieces, turn) =
      if context.turn == Color.Black then
        val wk = flipSqNum(bkSqOf(context.board)) // flipped Black king → new "White" king
        val bk = flipSqNum(wkSqOf(context.board)) // flipped White king → new "Black" king
        val flipped = context.board.pieces.map { case (sq, p) =>
          (sq, Piece(p.color.opposite, p.pieceType))
        }
        (wk, bk, flipped, Color.Black) // pass Black so scoreFromOutput negates the result
      else (wkSqOf(context.board), bkSqOf(context.board), context.board.pieces, context.turn)
    System.arraycopy(model.weights(0).bias, 0, legacyL1, 0, accSize)
-    for (sq, piece) <- context.board.pieces do addColumn(legacyL1, featureIndex(piece, squareNum(sq)))
+    for (sq, piece) <- pieces do
-    runL2toOutput(legacyL1, context.turn)
+      val sqNum = if turn == Color.Black then flipSqNum(squareNum(sq)) else squareNum(sq)
      addPiece(legacyL1, piece, sqNum, wkSq, bkSq)
    runL2toOutput(legacyL1, turn)
  def benchmark(): Unit =
    val context    = GameContext.initial
@@ -4,9 +4,9 @@ import de.nowchess.api.board.*
 import de.nowchess.api.game.GameContext
 import de.nowchess.api.move.{Move, MoveType, PromotionPiece}
-import java.io.{DataInputStream, FileInputStream}
+import java.io.{DataInputStream, FileInputStream, InputStream}
 import java.util.concurrent.ThreadLocalRandom
 import scala.collection.mutable
 import scala.util.Random
 /** Reads a Polyglot opening book (.bin file) and probes it for moves.
  *
@@ -16,24 +16,11 @@ import scala.util.Random
  *   - weight: 2 bytes (Short) — move weight (higher = preferred)
  *   - learn: 4 bytes (Int) — learning data (unused)
  */
-final class PolyglotBook(path: String):
+final class PolyglotBook private (entries: Map[Long, Vector[BookEntry]]):
  private val entries: Map[Long, Vector[BookEntry]] =
    try {
      val r = loadBookFile(path)
      println(s"Book loaded successfully. ${r.size} entries found.")
      r
    } catch
      case e: Exception =>
        println(s"Error loading book: $e")
        // Gracefully fail: return empty map if book cannot be loaded
        // This allows the bot to work even if the book file is missing
        scala.collection.immutable.Map.empty
  /** Probe the book for a move in the given position. Returns a weighted random move, or None if not in book. */
  def probe(context: GameContext): Option[Move] =
    val hash = PolyglotHash.hash(context)
    println(f"0x$hash%016X")
    entries.get(hash).flatMap { bookEntries =>
      if bookEntries.isEmpty then None
      else
@@ -41,24 +28,6 @@ final class PolyglotBook(path: String):
        decodeMove(entry.move, context)
    }
  private def loadBookFile(path: String): Map[Long, Vector[BookEntry]] =
    val input = DataInputStream(FileInputStream(path))
    try
      val result = mutable.Map[Long, Vector[BookEntry]]()
      while input.available() > 0 do
        val key    = input.readLong()
        val move   = input.readShort()
        val weight = input.readShort()
        input.readInt() // learning data (unused)
        val entry = BookEntry(key, move, weight)
        result.updateWith(key) {
          case Some(entries) => Some(entries :+ entry)
          case None          => Some(Vector(entry))
        }
      result.toMap
    finally input.close()
  /** Decode a packed Polyglot move short into an Option[Move].
    *
    * Bit layout of the move Short:
@@ -124,7 +93,7 @@ final class PolyglotBook(path: String):
    if entries.length == 1 then entries.head
    else
      val totalWeight = entries.map(_.weight).sum
-      val pick        = Random.nextInt(totalWeight.max(1)) // NOSONAR
+      val pick        = ThreadLocalRandom.current().nextInt(totalWeight.max(1)) // NOSONAR
      @scala.annotation.tailrec
      def select(remaining: Int, idx: Int): BookEntry =
@@ -134,4 +103,48 @@ final class PolyglotBook(path: String):
      select(pick, 0)
 object PolyglotBook:
  /** Load a book from a filesystem path. Fails gracefully to an empty book. */
  def apply(path: String): PolyglotBook =
    safeLoad(s"file $path")(FileInputStream(path))
  /** Load a book from a classpath resource (native-image safe: the resource is embedded in the binary, so no file must
    * be mounted into the pod).
    */
  def fromResource(name: String): PolyglotBook =
    Option(getClass.getResourceAsStream(name)) match
      case Some(stream) => safeLoad(s"resource $name")(stream)
      case None =>
        println(s"Error loading book: resource $name not found on classpath")
        new PolyglotBook(Map.empty)
  private def safeLoad(source: String)(stream: => InputStream): PolyglotBook =
    try
      val entries = parse(stream)
      println(s"Book loaded successfully from $source. ${entries.size} entries found.")
      new PolyglotBook(entries)
    catch
      case e: Exception =>
        println(s"Error loading book from $source: $e")
        new PolyglotBook(Map.empty)
  private def parse(stream: InputStream): Map[Long, Vector[BookEntry]] =
    val input = DataInputStream(stream)
    try
      val result = mutable.Map[Long, Vector[BookEntry]]()
      while input.available() > 0 do
        val key    = input.readLong()
        val move   = input.readShort()
        val weight = input.readShort()
        input.readInt() // learning data (unused)
        val entry = BookEntry(key, move, weight)
        result.updateWith(key) {
          case Some(entries) => Some(entries :+ entry)
          case None          => Some(Vector(entry))
        }
      result.toMap
    finally input.close()
 private case class BookEntry(key: Long, move: Short, weight: Int)
@@ -85,17 +85,17 @@ class HybridBotTest extends AnyFunSuite with Matchers:
  private val altMove  = Move(Square(File.E, Rank.R2), Square(File.E, Rank.R3), MoveType.Normal())
  private def vetoRules: RuleSet = new RuleSet:
-    private def fresh(ctx: GameContext): Boolean                          = ctx.moves.isEmpty
+    private def fresh(ctx: GameContext): Boolean                         = ctx.moves.isEmpty
-    def candidateMoves(context: GameContext)(square: Square): List[Move]  = Nil
+    def candidateMoves(context: GameContext)(square: Square): List[Move] = Nil
-    def legalMoves(context: GameContext)(square: Square): List[Move]      = Nil
+    def legalMoves(context: GameContext)(square: Square): List[Move]     = Nil
-    def allLegalMoves(context: GameContext): List[Move]                   =
+    def allLegalMoves(context: GameContext): List[Move] =
      if fresh(context) then List(mateMove, altMove) else Nil
-    def isCheck(context: GameContext): Boolean                            = false
+    def isCheck(context: GameContext): Boolean                = false
-    def isCheckmate(context: GameContext): Boolean                        = context.moves.lastOption.contains(mateMove)
+    def isCheckmate(context: GameContext): Boolean            = context.moves.lastOption.contains(mateMove)
-    def isStalemate(context: GameContext): Boolean                        = context.moves.lastOption.contains(altMove)
+    def isStalemate(context: GameContext): Boolean            = context.moves.lastOption.contains(altMove)
-    def isInsufficientMaterial(context: GameContext): Boolean             = false
+    def isInsufficientMaterial(context: GameContext): Boolean = false
-    def isFiftyMoveRule(context: GameContext): Boolean                    = false
+    def isFiftyMoveRule(context: GameContext): Boolean        = false
-    def isThreefoldRepetition(context: GameContext): Boolean              = false
+    def isThreefoldRepetition(context: GameContext): Boolean  = false
    def applyMove(context: GameContext)(move: Move): GameContext =
      context.copy(turn = context.turn.opposite, moves = context.moves :+ move)
@@ -1,3 +1,3 @@
 MAJOR=0
-MINOR=34
+MINOR=37
 PATCH=0
Author	SHA1	Message	Date
TeamCity	b4709b4a33	ci: bump version with Build-154	2026-06-24 17:55:44 +00:00
Janis Eccarius	9f9140cb58	fix: modified training pipeline Build & Test (NowChessSystems) TeamCity build finished Details	2026-06-24 19:37:26 +02:00
Janis	fa10852bc9	feat(official-bots): add Google Colab notebook for NNUE training (NCS-111) (#81 ) Build & Test (NowChessSystems) TeamCity build finished Details Adds python/NNUETraining.ipynb with five sections: - Setup: mount Drive, clone/update repo, install deps + Stockfish - Data: Option A (generate + label) or Option B (upload existing labeled.jsonl) - Train: standard epoch loop or burst mode (recommended for Colab free tier) - Export: convert best .pt checkpoint to .nbai via export.py - Download: pull .nbai and .pt to local machine via files.download Checkpoints and datasets are persisted to Google Drive so training survives session disconnects and can be resumed automatically. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Janis Eccarius <eccariusjanis@gmail.com> Reviewed-on: #81	2026-06-24 19:33:24 +02:00
Janis	44f376f032	feat(official-bots): implement king-relative (HalfKP) encoding in NNUE (NCS-109) (#80 ) Co-authored-by: Janis Eccarius <eccariusjanis@gmail.com> Reviewed-on: #80	2026-06-24 19:33:12 +02:00
TeamCity	7372867a82	ci: bump version with Build-152	2026-06-23 22:30:53 +00:00
Janis Eccarius	c3e7b82ae8	feat(analytics): add accuracy and blunder analysis job for Lichess data Build & Test (NowChessSystems) TeamCity build finished Details	2026-06-24 00:21:40 +02:00
TeamCity	e88b081947	ci: bump version with Build-151	2026-06-23 21:54:06 +00:00
Janis Eccarius	1b30c3be39	fix(official-bots): use ThreadLocalRandom in PolyglotBook for native image Build & Test (NowChessSystems) TeamCity build finished Details A stored java.util.Random field is reachable from BotController's static openingBook, so GraalVM baked it into the image heap and aborted the native build (Random in image heap has a cached seed). Use ThreadLocalRandom.current() at call time instead — no stored instance, nothing in the image heap, still thread-safe. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-23 23:42:15 +02:00
Janis Eccarius	f8ca95af3c	refactor(official-bots): use java.util.Random in PolyglotBook Build & Test (NowChessSystems) TeamCity build finished Details scala.util.Random delegates to a shared global java.util.Random, a contention point across concurrent bot games. Use a per-book java.util.Random instance instead. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-23 23:34:38 +02:00
TeamCity	4a50db0721	ci: bump version with Build-150	2026-06-23 21:27:19 +00:00
Janis Eccarius	260db25803	feat(official-bots): activate opening book in expert bot (native-safe) Build & Test (NowChessSystems) TeamCity build finished Details Load the Polyglot opening book as a classpath resource and wire it into the expert HybridBot. Previously the bot supported Option[PolyglotBook] but BotController passed None, so the book was never used. PolyglotBook.fromResource reads via getResourceAsStream so the book is embedded in the GraalVM native image instead of read from the filesystem (FileInputStream) — no file needs mounting into the pod. The filesystem apply(path) factory is kept for tests. Moved codekiddy.bin into resources as opening_book.bin. Dropped the per-probe debug println. NCS-43 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-23 23:17:52 +02:00