Compare commits
5 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 6351a19b67 | |||
| d7df76b769 | |||
| f44d3ee376 | |||
| 688d30e2b1 | |||
| 98c64fc0d5 |
@@ -569,3 +569,44 @@
|
|||||||
|
|
||||||
* Revert "feat: add authentication permissions for metrics endpoints in application.yml" ([a298417](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/a298417b9e4d68dc73bbf40be63d9484536e9f83))
|
* Revert "feat: add authentication permissions for metrics endpoints in application.yml" ([a298417](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/a298417b9e4d68dc73bbf40be63d9484536e9f83))
|
||||||
* Revert "refactor: update metrics paths formatting in application.yml for clarity" ([3870566](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/38705663498d5f47c40dafe2f26198589ede8656))
|
* Revert "refactor: update metrics paths formatting in application.yml for clarity" ([3870566](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/38705663498d5f47c40dafe2f26198589ede8656))
|
||||||
|
## (2026-06-17)
|
||||||
|
|
||||||
|
### Features
|
||||||
|
|
||||||
|
* **account:** implement token pair handling for login and refresh endpoints ([9296db8](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/9296db88b7131bbda9b9b0da65c327ef9063ee31))
|
||||||
|
* add authentication permissions for metrics endpoints in application.yml ([04edd4d](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/04edd4d6fd8a63196c36f6d67992832febc9bebb))
|
||||||
|
* add initialization metrics for various services ([d438e97](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/d438e97f32bdde0bfc63c1b4a8cc810cdd093166))
|
||||||
|
* add OpenTelemetry trace configuration with parentbased sampler ([3904d5a](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/3904d5ad8ad4930ddee65287a7bfab785a6148f5))
|
||||||
|
* **api:** define shared EventEnvelope and EventType for Redis EventBus ([#61](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/61)) ([595c172](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/595c172900da99de367c274488c3ccbeaef55882))
|
||||||
|
* **bot-platform:** migrate BotRegistry to Redis Streams consumer group ([#63](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/63)) ([0ad2e10](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/0ad2e10999213df6dd00f0c31a088c28a4dc0083))
|
||||||
|
* **config:** add H2 database configuration for testing environment ([39c9e49](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/39c9e492cef2515368c074da9406f95e9c0c9e64))
|
||||||
|
* **config:** update application.yml for PostgreSQL and remove staging/production configurations ([2404e61](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/2404e6164c3b50ffccbea5238d636060d6abe4d6))
|
||||||
|
* **config:** update application.yml for staging and production environments ([6113432](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/6113432a14c476a3a0dfc0d449e17d023697f2ba))
|
||||||
|
* configure logging and add OpenTelemetry support ([#49](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/49)) ([d57c488](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/d57c4886612d1d92da0e1b79209fc83e6ef537a1))
|
||||||
|
* **docker:** add .dockerignore and .gitignore files for build exclusions ([c987d8e](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/c987d8e258c0e6c4cfbdaa8381c64c410d7a2b83))
|
||||||
|
* **docker:** add Dockerfiles for building Quarkus application in native and JVM modes ([3f2d2bb](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/3f2d2bb4c97fa8cddba66e1da4427c54236dfeed))
|
||||||
|
* **docker:** add Dockerfiles for Quarkus application in JVM and native modes ([34b9933](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/34b993304670cf2aa62cd2f6460cee7b9864b08e))
|
||||||
|
* **events:** migrate game-creation and bot flows to Redis Streams NCS-89 ([#62](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/62)) ([a24924c](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/a24924c23057db3d700a75dbc4333557789cd991))
|
||||||
|
* **logging:** add DEBUG/INFO/WARN logging across services (NCS-72) ([#41](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/41)) ([804a4bf](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/804a4bf179e3dfb19e2be4390e7e543caf5237c6))
|
||||||
|
* NCS-78 Add Traceability to the Applications ([#46](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/46)) ([649566e](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/649566eb3fcf38f91c8896a739f74ea318af312d))
|
||||||
|
* NCS-78 Add Traceability to the Applications ([#47](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/47)) ([87dfc6c](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/87dfc6c2bcce7f7d58fc641bd8d468a2e584c108))
|
||||||
|
* NCS-82 add Swiss-system tournament module ([#55](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/55)) ([c5661de](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/c5661de4a0ebf4b33211f5a391840dcf744656b7))
|
||||||
|
* **reflection:** add native reflection configuration for tournament classes ([65bc6a7](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/65bc6a759937543df2d29905688bfa9e68d0c9d4))
|
||||||
|
* true-microservices ([#40](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/40)) ([5909242](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/590924254e8a2754de661a57a03e43f89ceb6299))
|
||||||
|
* update application.yml with new API root paths and add Micrometer and OpenTelemetry dependencies ([72ce262](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/72ce262bc491f94297700e6002fb5d0812e2cc2a))
|
||||||
|
* **ws:** migrate challenge notifications to Redis Streams ([#66](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/66)) ([55f102c](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/55f102cbaa684be94a158b16aaa42a50b36afaf3))
|
||||||
|
|
||||||
|
### Bug Fixes
|
||||||
|
|
||||||
|
* **account:** configure JDBC connection pool size to prevent exhaustion under load ([29072ef](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/29072efbfb1cfa1c3b1a85b4c1a587c971d245f9))
|
||||||
|
* **auth:** add InternalClientHeadersFactory for custom client headers management ([e279c39](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/e279c39246470156bf11e745ee72204018d4229d))
|
||||||
|
* enable official bots to connect to external tournament server ([#71](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/71)) ([688d30e](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/688d30e2b10026923372be5fca3c63eaaee2de2a))
|
||||||
|
* NCS-84 More Verbose Logging ([#51](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/51)) ([4ad92ab](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/4ad92ab23698267f8faa59c4e18388d4a0042cca))
|
||||||
|
* **official-bots:** NCS-70-auto-register official bots with account service ([#59](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/59)) ([7117a93](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/7117a93376272094d0b1a6abf2121254ce396684))
|
||||||
|
* remove unused HTTP root-path configurations from application.yml ([3ed3e59](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/3ed3e59ee456d54cd3d65ece4f36623e256b9736))
|
||||||
|
* **tests:** update token path to accessToken in ChallengeResourceTest ([354db11](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/354db11972342c47a1034303c11bccfb92e60109))
|
||||||
|
|
||||||
|
### Reverts
|
||||||
|
|
||||||
|
* Revert "feat: add authentication permissions for metrics endpoints in application.yml" ([a298417](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/a298417b9e4d68dc73bbf40be63d9484536e9f83))
|
||||||
|
* Revert "refactor: update metrics paths formatting in application.yml for clarity" ([3870566](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/38705663498d5f47c40dafe2f26198589ede8656))
|
||||||
|
|||||||
@@ -239,6 +239,7 @@ class AccountService:
|
|||||||
.subject(botId.toString)
|
.subject(botId.toString)
|
||||||
.expiresAt(Long.MaxValue)
|
.expiresAt(Long.MaxValue)
|
||||||
.claim("type", "bot")
|
.claim("type", "bot")
|
||||||
|
.claim("isBot", true)
|
||||||
.claim("name", botName)
|
.claim("name", botName)
|
||||||
.sign()
|
.sign()
|
||||||
|
|
||||||
|
|||||||
@@ -1,3 +1,3 @@
|
|||||||
MAJOR=0
|
MAJOR=0
|
||||||
MINOR=24
|
MINOR=25
|
||||||
PATCH=0
|
PATCH=0
|
||||||
|
|||||||
@@ -22,6 +22,9 @@
|
|||||||
// NOWCHESS_JDBC_URL (default: jdbc:postgresql://localhost:5432/nowchess)
|
// NOWCHESS_JDBC_URL (default: jdbc:postgresql://localhost:5432/nowchess)
|
||||||
// NOWCHESS_DB_USER (default: nowchess)
|
// NOWCHESS_DB_USER (default: nowchess)
|
||||||
// NOWCHESS_DB_PASS (default: nowchess)
|
// NOWCHESS_DB_PASS (default: nowchess)
|
||||||
|
// NOWCHESS_PGN_PATH (optional) — file or http(s) URL of a Lichess PGN dump (.pgn or .pgn.zst).
|
||||||
|
// When set, all batch jobs read games from the dump instead of PostgreSQL and
|
||||||
|
// skip JDBC write-back (Parquet/CSV output only). Demo data source.
|
||||||
|
|
||||||
plugins {
|
plugins {
|
||||||
id("scala")
|
id("scala")
|
||||||
@@ -71,6 +74,10 @@ dependencies {
|
|||||||
|
|
||||||
// PostgreSQL JDBC driver bundled so it is available on executor classpath.
|
// PostgreSQL JDBC driver bundled so it is available on executor classpath.
|
||||||
implementation("org.postgresql:postgresql:42.7.4")
|
implementation("org.postgresql:postgresql:42.7.4")
|
||||||
|
|
||||||
|
// zstd-jni: decompress Lichess .pgn.zst dumps in-process. Provided at runtime by Spark
|
||||||
|
// (it uses zstd-jni internally for shuffle/event-log compression), so compile-only here.
|
||||||
|
compileOnly("com.github.luben:zstd-jni:1.5.6-9")
|
||||||
}
|
}
|
||||||
|
|
||||||
application {
|
application {
|
||||||
|
|||||||
@@ -0,0 +1,119 @@
|
|||||||
|
package de.nowchess.analytics
|
||||||
|
|
||||||
|
import org.apache.spark.SparkFiles
|
||||||
|
import org.apache.spark.sql.DataFrame
|
||||||
|
import org.apache.spark.sql.SparkSession
|
||||||
|
import org.apache.spark.sql.functions as F
|
||||||
|
|
||||||
|
/** Normalised game-record source for the batch jobs.
|
||||||
|
*
|
||||||
|
* Every batch job consumes the same five-column shape:
|
||||||
|
* - white_id, black_id : player identifiers
|
||||||
|
* - result : one of "white", "black", "draw"
|
||||||
|
* - move_count : number of plies
|
||||||
|
* - pgn : full PGN ("[Event …]…\n\n1. e4 …"), header and movetext separated by a blank line
|
||||||
|
*
|
||||||
|
* Two backends, selected by the `NOWCHESS_PGN_PATH` environment variable:
|
||||||
|
* - unset → PostgreSQL `game_records` table (production)
|
||||||
|
* - set → a Lichess PGN dump file/URL (demo). Point it at a `lichess_db_standard_rated_*.pgn[.zst]`
|
||||||
|
* to drive every batch job from real Lichess games.
|
||||||
|
*
|
||||||
|
* Lichess parsing uses only Spark SQL string functions — no UDFs — so Catalyst can push predicates,
|
||||||
|
* matching the no-UDF approach already used in OpeningBookJob.
|
||||||
|
*/
|
||||||
|
object GameSource:
|
||||||
|
|
||||||
|
private val PgnPathEnv = "NOWCHESS_PGN_PATH"
|
||||||
|
|
||||||
|
/** True when a Lichess PGN dump is configured; jobs use this to skip JDBC write-back. */
|
||||||
|
def isPgnMode: Boolean = sys.env.contains(PgnPathEnv)
|
||||||
|
|
||||||
|
def load(spark: SparkSession, jdbcUrl: String, dbUser: String, dbPass: String): DataFrame =
|
||||||
|
sys.env.get(PgnPathEnv) match
|
||||||
|
case Some(path) => fromLichessPgn(spark, path)
|
||||||
|
case None => fromJdbc(spark, jdbcUrl, dbUser, dbPass)
|
||||||
|
|
||||||
|
def fromJdbc(spark: SparkSession, jdbcUrl: String, dbUser: String, dbPass: String): DataFrame =
|
||||||
|
spark.read
|
||||||
|
.format("jdbc")
|
||||||
|
.option("url", jdbcUrl)
|
||||||
|
.option("dbtable", "game_records")
|
||||||
|
.option("user", dbUser)
|
||||||
|
.option("password", dbPass)
|
||||||
|
.option("driver", "org.postgresql.Driver")
|
||||||
|
.option("fetchsize", "10000")
|
||||||
|
.load()
|
||||||
|
.select("white_id", "black_id", "result", "move_count", "pgn")
|
||||||
|
|
||||||
|
/** Parses a Lichess PGN dump into the normalised game shape.
|
||||||
|
*
|
||||||
|
* `path` may be:
|
||||||
|
* - an http(s)/ftp URL — fetched once via SparkContext.addFile and distributed to executors, then read
|
||||||
|
* from the local replica (no S3/PVC needed; handy for a staging demo)
|
||||||
|
* - any Hadoop-readable path (file://, hdfs://, s3a://, …)
|
||||||
|
*
|
||||||
|
* `.zst` dumps (Lichess' native format) are decompressed in-process via zstd-jni; `.gz`/`.bz2` are
|
||||||
|
* handled by Spark's text reader codecs.
|
||||||
|
*
|
||||||
|
* Records are split on the "[Event " tag that opens every game, so each row holds one complete game
|
||||||
|
* (the empty fragment before the first game is filtered out). Header tags are read with regexp_extract;
|
||||||
|
* the movetext (after the blank line) is cleaned of clock/eval comments and move numbers to count plies.
|
||||||
|
*/
|
||||||
|
def fromLichessPgn(spark: SparkSession, path: String): DataFrame =
|
||||||
|
val resolved = resolvePath(spark, path)
|
||||||
|
val record = F.col("value")
|
||||||
|
|
||||||
|
val resultTag = F.regexp_extract(record, "Result \"([^\"]*)\"", 1)
|
||||||
|
val result = F
|
||||||
|
.when(resultTag === "1-0", "white")
|
||||||
|
.when(resultTag === "0-1", "black")
|
||||||
|
.when(resultTag === "1/2-1/2", "draw")
|
||||||
|
.otherwise(F.lit(null).cast("string"))
|
||||||
|
|
||||||
|
val moveText = F.coalesce(F.split(record, "\n\n").getItem(1), F.lit(""))
|
||||||
|
val noComment = F.regexp_replace(moveText, "\\{[^}]*\\}", "")
|
||||||
|
val noResult = F.regexp_replace(noComment, "(1-0|0-1|1/2-1/2|\\*)", "")
|
||||||
|
val noNumbers = F.regexp_replace(noResult, "\\d+\\.+", " ")
|
||||||
|
val plies = F.size(F.filter(F.split(F.trim(noNumbers), "\\s+"), tok => F.length(tok) > 0))
|
||||||
|
|
||||||
|
spark.read
|
||||||
|
.option("lineSep", "[Event ")
|
||||||
|
.text(resolved)
|
||||||
|
.filter(F.length(F.trim(record)) > 0)
|
||||||
|
.select(
|
||||||
|
F.regexp_extract(record, "White \"([^\"]*)\"", 1).as("white_id"),
|
||||||
|
F.regexp_extract(record, "Black \"([^\"]*)\"", 1).as("black_id"),
|
||||||
|
result.as("result"),
|
||||||
|
plies.as("move_count"),
|
||||||
|
F.concat(F.lit("[Event "), record).as("pgn"),
|
||||||
|
)
|
||||||
|
.filter((F.col("white_id") =!= "").and(F.col("black_id") =!= ""))
|
||||||
|
|
||||||
|
/** Turns an http(s)/ftp URL into a cluster-local path by fetching it once with SparkContext.addFile,
|
||||||
|
* which distributes the file to every executor. `.zst` is decompressed in-process and the plain `.pgn`
|
||||||
|
* is redistributed. Non-URL paths are returned unchanged.
|
||||||
|
*/
|
||||||
|
private def resolvePath(spark: SparkSession, path: String): String =
|
||||||
|
if !path.matches("^(https?|ftp)://.*") then path
|
||||||
|
else
|
||||||
|
spark.sparkContext.addFile(path)
|
||||||
|
val local = SparkFiles.get(baseName(path))
|
||||||
|
if !local.endsWith(".zst") then "file://" + local
|
||||||
|
else distribute(spark, decompressZstd(local))
|
||||||
|
|
||||||
|
private def baseName(path: String): String = path.substring(path.lastIndexOf('/') + 1)
|
||||||
|
|
||||||
|
private def distribute(spark: SparkSession, localPath: String): String =
|
||||||
|
spark.sparkContext.addFile("file://" + localPath)
|
||||||
|
"file://" + SparkFiles.get(baseName(localPath))
|
||||||
|
|
||||||
|
/** Decompresses a `.zst` file to a temp `.pgn` using zstd-jni (bundled with Spark at runtime). */
|
||||||
|
private def decompressZstd(srcPath: String): String =
|
||||||
|
val out = java.io.File.createTempFile("lichess-", ".pgn")
|
||||||
|
out.deleteOnExit()
|
||||||
|
val in = com.github.luben.zstd.ZstdInputStream(
|
||||||
|
java.io.BufferedInputStream(java.io.FileInputStream(srcPath)),
|
||||||
|
)
|
||||||
|
try java.nio.file.Files.copy(in, out.toPath, java.nio.file.StandardCopyOption.REPLACE_EXISTING)
|
||||||
|
finally in.close()
|
||||||
|
out.getAbsolutePath
|
||||||
@@ -37,15 +37,8 @@ object OpeningBookJob:
|
|||||||
outputDir: String,
|
outputDir: String,
|
||||||
maxPlies: Int,
|
maxPlies: Int,
|
||||||
): Unit =
|
): Unit =
|
||||||
val games = spark.read
|
val games = GameSource
|
||||||
.format("jdbc")
|
.load(spark, jdbcUrl, dbUser, dbPass)
|
||||||
.option("url", jdbcUrl)
|
|
||||||
.option("dbtable", "game_records")
|
|
||||||
.option("user", dbUser)
|
|
||||||
.option("password", dbPass)
|
|
||||||
.option("driver", "org.postgresql.Driver")
|
|
||||||
.option("fetchsize", "10000")
|
|
||||||
.load()
|
|
||||||
.select("pgn", "result")
|
.select("pgn", "result")
|
||||||
.filter(F.col("result").isNotNull.and(F.col("pgn").isNotNull))
|
.filter(F.col("result").isNotNull.and(F.col("pgn").isNotNull))
|
||||||
|
|
||||||
@@ -79,15 +72,16 @@ object OpeningBookJob:
|
|||||||
.option("header", "true")
|
.option("header", "true")
|
||||||
.csv(s"$outputDir/opening_book_top1000")
|
.csv(s"$outputDir/opening_book_top1000")
|
||||||
|
|
||||||
top1000.write
|
if !GameSource.isPgnMode then
|
||||||
.mode("overwrite")
|
top1000.write
|
||||||
.format("jdbc")
|
.mode("overwrite")
|
||||||
.option("url", jdbcUrl)
|
.format("jdbc")
|
||||||
.option("dbtable", "analytics_opening_stats")
|
.option("url", jdbcUrl)
|
||||||
.option("user", dbUser)
|
.option("dbtable", "analytics_opening_stats")
|
||||||
.option("password", dbPass)
|
.option("user", dbUser)
|
||||||
.option("driver", "org.postgresql.Driver")
|
.option("password", dbPass)
|
||||||
.save()
|
.option("driver", "org.postgresql.Driver")
|
||||||
|
.save()
|
||||||
|
|
||||||
/** Extracts the first `maxPlies` moves from a PGN column as a space-separated string.
|
/** Extracts the first `maxPlies` moves from a PGN column as a space-separated string.
|
||||||
*
|
*
|
||||||
|
|||||||
@@ -50,15 +50,8 @@ object PlayerClusteringJob:
|
|||||||
outputDir: String,
|
outputDir: String,
|
||||||
k: Int,
|
k: Int,
|
||||||
): Unit =
|
): Unit =
|
||||||
val games = spark.read
|
val games = GameSource
|
||||||
.format("jdbc")
|
.load(spark, jdbcUrl, dbUser, dbPass)
|
||||||
.option("url", jdbcUrl)
|
|
||||||
.option("dbtable", "game_records")
|
|
||||||
.option("user", dbUser)
|
|
||||||
.option("password", dbPass)
|
|
||||||
.option("driver", "org.postgresql.Driver")
|
|
||||||
.option("fetchsize", "10000")
|
|
||||||
.load()
|
|
||||||
.select("white_id", "black_id", "result", "move_count")
|
.select("white_id", "black_id", "result", "move_count")
|
||||||
.filter(F.col("result").isNotNull)
|
.filter(F.col("result").isNotNull)
|
||||||
|
|
||||||
@@ -126,25 +119,26 @@ object PlayerClusteringJob:
|
|||||||
.option("header", "true")
|
.option("header", "true")
|
||||||
.csv(s"$outputDir/cluster_archetypes")
|
.csv(s"$outputDir/cluster_archetypes")
|
||||||
|
|
||||||
clustersDf.write
|
if !GameSource.isPgnMode then
|
||||||
.mode("overwrite")
|
clustersDf.write
|
||||||
.format("jdbc")
|
.mode("overwrite")
|
||||||
.option("url", jdbcUrl)
|
.format("jdbc")
|
||||||
.option("dbtable", "analytics_player_clusters")
|
.option("url", jdbcUrl)
|
||||||
.option("user", dbUser)
|
.option("dbtable", "analytics_player_clusters")
|
||||||
.option("password", dbPass)
|
.option("user", dbUser)
|
||||||
.option("driver", "org.postgresql.Driver")
|
.option("password", dbPass)
|
||||||
.save()
|
.option("driver", "org.postgresql.Driver")
|
||||||
|
.save()
|
||||||
|
|
||||||
archetypes.write
|
archetypes.write
|
||||||
.mode("overwrite")
|
.mode("overwrite")
|
||||||
.format("jdbc")
|
.format("jdbc")
|
||||||
.option("url", jdbcUrl)
|
.option("url", jdbcUrl)
|
||||||
.option("dbtable", "analytics_cluster_archetypes")
|
.option("dbtable", "analytics_cluster_archetypes")
|
||||||
.option("user", dbUser)
|
.option("user", dbUser)
|
||||||
.option("password", dbPass)
|
.option("password", dbPass)
|
||||||
.option("driver", "org.postgresql.Driver")
|
.option("driver", "org.postgresql.Driver")
|
||||||
.save()
|
.save()
|
||||||
|
|
||||||
private def buildPlayerStats(games: org.apache.spark.sql.DataFrame): org.apache.spark.sql.DataFrame =
|
private def buildPlayerStats(games: org.apache.spark.sql.DataFrame): org.apache.spark.sql.DataFrame =
|
||||||
val asWhite = games.select(
|
val asWhite = games.select(
|
||||||
|
|||||||
@@ -53,15 +53,8 @@ object PlayerGraphJob:
|
|||||||
dbPass: String,
|
dbPass: String,
|
||||||
outputDir: String,
|
outputDir: String,
|
||||||
): Unit =
|
): Unit =
|
||||||
val gamesRdd: RDD[Row] = spark.read
|
val gamesRdd: RDD[Row] = GameSource
|
||||||
.format("jdbc")
|
.load(spark, jdbcUrl, dbUser, dbPass)
|
||||||
.option("url", jdbcUrl)
|
|
||||||
.option("dbtable", "game_records")
|
|
||||||
.option("user", dbUser)
|
|
||||||
.option("password", dbPass)
|
|
||||||
.option("driver", "org.postgresql.Driver")
|
|
||||||
.option("fetchsize", "10000")
|
|
||||||
.load()
|
|
||||||
.select("white_id", "black_id", "result")
|
.select("white_id", "black_id", "result")
|
||||||
.filter(F.col("result").isNotNull)
|
.filter(F.col("result").isNotNull)
|
||||||
.rdd
|
.rdd
|
||||||
@@ -116,15 +109,16 @@ object PlayerGraphJob:
|
|||||||
.mode("overwrite")
|
.mode("overwrite")
|
||||||
.parquet(s"$outputDir/player_graph")
|
.parquet(s"$outputDir/player_graph")
|
||||||
|
|
||||||
result.write
|
if !GameSource.isPgnMode then
|
||||||
.mode("overwrite")
|
result.write
|
||||||
.format("jdbc")
|
.mode("overwrite")
|
||||||
.option("url", jdbcUrl)
|
.format("jdbc")
|
||||||
.option("dbtable", "analytics_player_graph")
|
.option("url", jdbcUrl)
|
||||||
.option("user", dbUser)
|
.option("dbtable", "analytics_player_graph")
|
||||||
.option("password", dbPass)
|
.option("user", dbUser)
|
||||||
.option("driver", "org.postgresql.Driver")
|
.option("password", dbPass)
|
||||||
.save()
|
.option("driver", "org.postgresql.Driver")
|
||||||
|
.save()
|
||||||
|
|
||||||
// How many players belong to each connected component?
|
// How many players belong to each connected component?
|
||||||
// A large dominant component + many singletons is the expected shape.
|
// A large dominant component + many singletons is the expected shape.
|
||||||
|
|||||||
@@ -34,15 +34,8 @@ object PlayerStatsJob:
|
|||||||
dbPass: String,
|
dbPass: String,
|
||||||
outputDir: String,
|
outputDir: String,
|
||||||
): Unit =
|
): Unit =
|
||||||
val games = spark.read
|
val games = GameSource
|
||||||
.format("jdbc")
|
.load(spark, jdbcUrl, dbUser, dbPass)
|
||||||
.option("url", jdbcUrl)
|
|
||||||
.option("dbtable", "game_records")
|
|
||||||
.option("user", dbUser)
|
|
||||||
.option("password", dbPass)
|
|
||||||
.option("driver", "org.postgresql.Driver")
|
|
||||||
.option("fetchsize", "10000")
|
|
||||||
.load()
|
|
||||||
.select("white_id", "black_id", "result", "move_count")
|
.select("white_id", "black_id", "result", "move_count")
|
||||||
.filter(F.col("result").isNotNull)
|
.filter(F.col("result").isNotNull)
|
||||||
|
|
||||||
@@ -84,12 +77,13 @@ object PlayerStatsJob:
|
|||||||
.mode("overwrite")
|
.mode("overwrite")
|
||||||
.parquet(s"$outputDir/player_stats")
|
.parquet(s"$outputDir/player_stats")
|
||||||
|
|
||||||
stats.write
|
if !GameSource.isPgnMode then
|
||||||
.mode("overwrite")
|
stats.write
|
||||||
.format("jdbc")
|
.mode("overwrite")
|
||||||
.option("url", jdbcUrl)
|
.format("jdbc")
|
||||||
.option("dbtable", "analytics_player_stats")
|
.option("url", jdbcUrl)
|
||||||
.option("user", dbUser)
|
.option("dbtable", "analytics_player_stats")
|
||||||
.option("password", dbPass)
|
.option("user", dbUser)
|
||||||
.option("driver", "org.postgresql.Driver")
|
.option("password", dbPass)
|
||||||
.save()
|
.option("driver", "org.postgresql.Driver")
|
||||||
|
.save()
|
||||||
|
|||||||
@@ -281,3 +281,32 @@
|
|||||||
### Reverts
|
### Reverts
|
||||||
|
|
||||||
* Revert "refactor: update metrics paths formatting in application.yml for clarity" ([3870566](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/38705663498d5f47c40dafe2f26198589ede8656))
|
* Revert "refactor: update metrics paths formatting in application.yml for clarity" ([3870566](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/38705663498d5f47c40dafe2f26198589ede8656))
|
||||||
|
## (2026-06-17)
|
||||||
|
|
||||||
|
### Features
|
||||||
|
|
||||||
|
* add initialization metrics for various services ([d438e97](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/d438e97f32bdde0bfc63c1b4a8cc810cdd093166))
|
||||||
|
* add OpenTelemetry trace configuration with parentbased sampler ([3904d5a](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/3904d5ad8ad4930ddee65287a7bfab785a6148f5))
|
||||||
|
* **analytics:** add Spark batch analytics module ([#70](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/70)) ([39f1657](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/39f1657e1db6e84889af338c43be8cb5c03c3ec3))
|
||||||
|
* **config:** update application.yml for PostgreSQL and remove staging/production configurations ([2404e61](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/2404e6164c3b50ffccbea5238d636060d6abe4d6))
|
||||||
|
* **config:** update application.yml for staging and production environments ([6113432](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/6113432a14c476a3a0dfc0d449e17d023697f2ba))
|
||||||
|
* configure logging and add OpenTelemetry support ([#49](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/49)) ([d57c488](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/d57c4886612d1d92da0e1b79209fc83e6ef537a1))
|
||||||
|
* **docker:** add .dockerignore and .gitignore files for build exclusions ([c987d8e](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/c987d8e258c0e6c4cfbdaa8381c64c410d7a2b83))
|
||||||
|
* **docker:** add Dockerfiles for building Quarkus application in native and JVM modes ([3f2d2bb](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/3f2d2bb4c97fa8cddba66e1da4427c54236dfeed))
|
||||||
|
* **docker:** add Dockerfiles for Quarkus application in JVM and native modes ([34b9933](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/34b993304670cf2aa62cd2f6460cee7b9864b08e))
|
||||||
|
* **events:** migrate game-creation and bot flows to Redis Streams NCS-89 ([#62](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/62)) ([a24924c](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/a24924c23057db3d700a75dbc4333557789cd991))
|
||||||
|
* NCS-78 Add Traceability to the Applications ([#46](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/46)) ([649566e](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/649566eb3fcf38f91c8896a739f74ea318af312d))
|
||||||
|
* NCS-78 Add Traceability to the Applications ([#47](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/47)) ([87dfc6c](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/87dfc6c2bcce7f7d58fc641bd8d468a2e584c108))
|
||||||
|
* NCS-82 add Swiss-system tournament module ([#55](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/55)) ([c5661de](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/c5661de4a0ebf4b33211f5a391840dcf744656b7))
|
||||||
|
* **official-bots:** consume GameOver stream for bot cleanup ([#67](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/67)) ([db9d153](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/db9d1533912f4b41c4d1ca80ccffdde5d23d6ff6))
|
||||||
|
* true-microservices ([#40](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/40)) ([5909242](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/590924254e8a2754de661a57a03e43f89ceb6299))
|
||||||
|
|
||||||
|
### Bug Fixes
|
||||||
|
|
||||||
|
* enable official bots to connect to external tournament server ([#71](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/71)) ([688d30e](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/688d30e2b10026923372be5fca3c63eaaee2de2a))
|
||||||
|
* **official-bots:** configure JWT verification ([#72](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/72)) ([98c64fc](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/98c64fc0d56dc542beb31c75f4b9056d91de03cd))
|
||||||
|
* **official-bots:** NCS-70-auto-register official bots with account service ([#59](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/59)) ([7117a93](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/7117a93376272094d0b1a6abf2121254ce396684))
|
||||||
|
|
||||||
|
### Reverts
|
||||||
|
|
||||||
|
* Revert "refactor: update metrics paths formatting in application.yml for clarity" ([3870566](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/38705663498d5f47c40dafe2f26198589ede8656))
|
||||||
|
|||||||
@@ -12,6 +12,12 @@ quarkus:
|
|||||||
enabled: true
|
enabled: true
|
||||||
log:
|
log:
|
||||||
level: INFO
|
level: INFO
|
||||||
|
mp:
|
||||||
|
jwt:
|
||||||
|
verify:
|
||||||
|
publickey:
|
||||||
|
location: ${JWT_PUBLIC_KEY_PATH:keys/public.pem}
|
||||||
|
issuer: nowchess
|
||||||
|
|
||||||
nowchess:
|
nowchess:
|
||||||
redis:
|
redis:
|
||||||
|
|||||||
@@ -0,0 +1,9 @@
|
|||||||
|
-----BEGIN PUBLIC KEY-----
|
||||||
|
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAxDsnsCAl0vQx7Vu9CLDZ
|
||||||
|
g0SG05NgUzu9T+3DTEaHGq60T2uriO8BenwyvsF3BnDqTbKf4voohZ1DNfzdbT1J
|
||||||
|
Fj8B62FrDmxcO+sp1/b5HUCJP6y2uSRCmzOHe5k7Pk1IEi72FgBpKXSRkFibRlVf
|
||||||
|
634g7mgsPZAQ9PJEsv4Qvm05T9L6+Gmq6N3bMVLKRXs4RhDhaFbYH9GtUg1eI0yH
|
||||||
|
YjGyRfqzW/nqVMstOLHt8CuPouq4p7eMzeDH3YHkxPm4GG5foCXMOd2DZrW0SCcr
|
||||||
|
7dhFeNVWzQ2m53eOhBzNQX+v3pgjVStsePhBRt2LyGfwkNzmqDgqWsMzSHRMY+cn
|
||||||
|
WQIDAQAB
|
||||||
|
-----END PUBLIC KEY-----
|
||||||
+1
@@ -2,6 +2,7 @@ package de.nowchess.bot.resource
|
|||||||
|
|
||||||
case class JoinTournamentRequest(
|
case class JoinTournamentRequest(
|
||||||
tournamentId: String,
|
tournamentId: String,
|
||||||
|
botToken: String,
|
||||||
difficulty: String,
|
difficulty: String,
|
||||||
serverUrl: Option[String],
|
serverUrl: Option[String],
|
||||||
)
|
)
|
||||||
|
|||||||
+1
-1
@@ -33,7 +33,7 @@ class TournamentJoinResource:
|
|||||||
difficulty,
|
difficulty,
|
||||||
serverUrl,
|
serverUrl,
|
||||||
)
|
)
|
||||||
player.joinTournament(req.tournamentId, difficulty, serverUrl) match
|
player.joinTournament(req.tournamentId, req.botToken, difficulty, serverUrl) match
|
||||||
case Right(botId) =>
|
case Right(botId) =>
|
||||||
val resp = JoinTournamentResponse(botId, difficulty, "joining")
|
val resp = JoinTournamentResponse(botId, difficulty, "joining")
|
||||||
Response.ok(resp).build()
|
Response.ok(resp).build()
|
||||||
|
|||||||
+1
-1
@@ -20,7 +20,7 @@ object TournamentBotConfig:
|
|||||||
tournamentId <- env.get("TOURNAMENT_ID").filter(_.nonEmpty)
|
tournamentId <- env.get("TOURNAMENT_ID").filter(_.nonEmpty)
|
||||||
token <- env.get("TOURNAMENT_BOT_TOKEN").filter(_.nonEmpty)
|
token <- env.get("TOURNAMENT_BOT_TOKEN").filter(_.nonEmpty)
|
||||||
botId <- jwtSubject(token)
|
botId <- jwtSubject(token)
|
||||||
serverUrl = env.getOrElse("TOURNAMENT_SERVER_URL", "http://localhost:8089")
|
serverUrl = env.getOrElse("TOURNAMENT_SERVER_URL", "http://141.37.123.132:8086")
|
||||||
difficulty = env.getOrElse("TOURNAMENT_BOT_DIFFICULTY", "medium")
|
difficulty = env.getOrElse("TOURNAMENT_BOT_DIFFICULTY", "medium")
|
||||||
yield TournamentBotConfig(serverUrl, tournamentId, token, botId, difficulty)
|
yield TournamentBotConfig(serverUrl, tournamentId, token, botId, difficulty)
|
||||||
|
|
||||||
|
|||||||
+38
-26
@@ -39,10 +39,11 @@ class TournamentBotGamePlayer:
|
|||||||
// scalafix:on DisableSyntax.var
|
// scalafix:on DisableSyntax.var
|
||||||
|
|
||||||
val defaultServerUrl: String =
|
val defaultServerUrl: String =
|
||||||
System.getenv().asScala.getOrElse("TOURNAMENT_SERVER_URL", "http://localhost:8089")
|
System.getenv().asScala.getOrElse("TOURNAMENT_SERVER_URL", "http://141.37.123.132:8086")
|
||||||
|
|
||||||
@PostConstruct
|
@PostConstruct
|
||||||
def initialize(): Unit =
|
def initialize(): Unit =
|
||||||
|
parkOnStartup()
|
||||||
config match
|
config match
|
||||||
case None =>
|
case None =>
|
||||||
log.info("Tournament bot disabled — set TOURNAMENT_ID and TOURNAMENT_BOT_TOKEN to enable")
|
log.info("Tournament bot disabled — set TOURNAMENT_ID and TOURNAMENT_BOT_TOKEN to enable")
|
||||||
@@ -50,11 +51,42 @@ class TournamentBotGamePlayer:
|
|||||||
log.infof("Tournament bot enabled — server=%s tournament=%s bot=%s", cfg.serverUrl, cfg.tournamentId, cfg.botId)
|
log.infof("Tournament bot enabled — server=%s tournament=%s bot=%s", cfg.serverUrl, cfg.tournamentId, cfg.botId)
|
||||||
startAsync(cfg)
|
startAsync(cfg)
|
||||||
|
|
||||||
def joinTournament(tournamentId: String, difficulty: String, serverUrl: String): Either[String, String] =
|
private def parkOnStartup(): Unit =
|
||||||
registerBot(serverUrl, difficulty) match
|
park(defaultServerUrl, "expert") match
|
||||||
case None => Left("Failed to register bot with tournament server")
|
case Some(id) => log.infof("Parked expert bot on %s as id %s", defaultServerUrl, id)
|
||||||
case Some((botId, token)) =>
|
case None => log.warnf("Failed to park expert bot on %s", defaultServerUrl)
|
||||||
val cfg = TournamentBotConfig(serverUrl, tournamentId, token, botId, difficulty)
|
|
||||||
|
private def park(serverUrl: String, difficulty: String): Option[String] =
|
||||||
|
System.getenv().asScala.get("TOURNAMENT_BOT_TOKEN").filter(_.nonEmpty).flatMap { token =>
|
||||||
|
Try {
|
||||||
|
val body = s"""{"name":"${botName(difficulty)}"}"""
|
||||||
|
val response = client
|
||||||
|
.target(serverUrl)
|
||||||
|
.path("api")
|
||||||
|
.path("bots")
|
||||||
|
.request(MediaType.APPLICATION_JSON)
|
||||||
|
.header("Authorization", s"Bearer $token")
|
||||||
|
.post(Entity.entity(body, MediaType.APPLICATION_JSON))
|
||||||
|
if response.getStatus == 201 || response.getStatus == 200 then
|
||||||
|
val id = objectMapper.readTree(response.readEntity(classOf[String])).path("id").asText()
|
||||||
|
response.close()
|
||||||
|
Option(id).filter(_.nonEmpty)
|
||||||
|
else { log.warnf("Parking bot %s returned status %d", botName(difficulty), response.getStatus); response.close(); None }
|
||||||
|
}.getOrElse(None)
|
||||||
|
}
|
||||||
|
|
||||||
|
private def botName(difficulty: String): String = s"NowChess ${difficulty.capitalize}"
|
||||||
|
|
||||||
|
def joinTournament(
|
||||||
|
tournamentId: String,
|
||||||
|
botToken: String,
|
||||||
|
difficulty: String,
|
||||||
|
serverUrl: String,
|
||||||
|
): Either[String, String] =
|
||||||
|
TournamentBotConfig.jwtSubject(botToken) match
|
||||||
|
case None => Left("Invalid bot token — could not extract subject")
|
||||||
|
case Some(botId) =>
|
||||||
|
val cfg = TournamentBotConfig(serverUrl, tournamentId, botToken, botId, difficulty)
|
||||||
if join(cfg) then
|
if join(cfg) then
|
||||||
startAsync(cfg)
|
startAsync(cfg)
|
||||||
Right(botId)
|
Right(botId)
|
||||||
@@ -65,26 +97,6 @@ class TournamentBotGamePlayer:
|
|||||||
thread.setDaemon(true)
|
thread.setDaemon(true)
|
||||||
thread.start()
|
thread.start()
|
||||||
|
|
||||||
private def registerBot(serverUrl: String, difficulty: String): Option[(String, String)] =
|
|
||||||
Try {
|
|
||||||
val name = s"NowChess ${difficulty.capitalize}"
|
|
||||||
val body = s"""{"name":"$name","isBot":true}"""
|
|
||||||
val response = client
|
|
||||||
.target(serverUrl)
|
|
||||||
.path("api")
|
|
||||||
.path("auth")
|
|
||||||
.path("register")
|
|
||||||
.request(MediaType.APPLICATION_JSON)
|
|
||||||
.post(Entity.entity(body, MediaType.APPLICATION_JSON))
|
|
||||||
if response.getStatus == 201 then
|
|
||||||
val node = objectMapper.readTree(response.readEntity(classOf[String]))
|
|
||||||
val id = node.path("id").asText()
|
|
||||||
val token = node.path("token").asText()
|
|
||||||
response.close()
|
|
||||||
if id.nonEmpty && token.nonEmpty then Some((id, token)) else None
|
|
||||||
else { log.warnf("Bot registration returned status %d", response.getStatus); response.close(); None }
|
|
||||||
}.getOrElse(None)
|
|
||||||
|
|
||||||
@PreDestroy
|
@PreDestroy
|
||||||
def cleanup(): Unit =
|
def cleanup(): Unit =
|
||||||
running = false
|
running = false
|
||||||
|
|||||||
@@ -1,3 +1,3 @@
|
|||||||
MAJOR=0
|
MAJOR=0
|
||||||
MINOR=18
|
MINOR=19
|
||||||
PATCH=0
|
PATCH=0
|
||||||
|
|||||||
Reference in New Issue
Block a user