Compare commits
6 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 343e2bdd10 | |||
| 751a58b606 | |||
| 30295a4bb9 | |||
| f44d3ee376 | |||
| 688d30e2b1 | |||
| 98c64fc0d5 |
@@ -569,3 +569,44 @@
|
||||
|
||||
* Revert "feat: add authentication permissions for metrics endpoints in application.yml" ([a298417](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/a298417b9e4d68dc73bbf40be63d9484536e9f83))
|
||||
* Revert "refactor: update metrics paths formatting in application.yml for clarity" ([3870566](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/38705663498d5f47c40dafe2f26198589ede8656))
|
||||
## (2026-06-17)
|
||||
|
||||
### Features
|
||||
|
||||
* **account:** implement token pair handling for login and refresh endpoints ([9296db8](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/9296db88b7131bbda9b9b0da65c327ef9063ee31))
|
||||
* add authentication permissions for metrics endpoints in application.yml ([04edd4d](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/04edd4d6fd8a63196c36f6d67992832febc9bebb))
|
||||
* add initialization metrics for various services ([d438e97](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/d438e97f32bdde0bfc63c1b4a8cc810cdd093166))
|
||||
* add OpenTelemetry trace configuration with parentbased sampler ([3904d5a](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/3904d5ad8ad4930ddee65287a7bfab785a6148f5))
|
||||
* **api:** define shared EventEnvelope and EventType for Redis EventBus ([#61](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/61)) ([595c172](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/595c172900da99de367c274488c3ccbeaef55882))
|
||||
* **bot-platform:** migrate BotRegistry to Redis Streams consumer group ([#63](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/63)) ([0ad2e10](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/0ad2e10999213df6dd00f0c31a088c28a4dc0083))
|
||||
* **config:** add H2 database configuration for testing environment ([39c9e49](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/39c9e492cef2515368c074da9406f95e9c0c9e64))
|
||||
* **config:** update application.yml for PostgreSQL and remove staging/production configurations ([2404e61](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/2404e6164c3b50ffccbea5238d636060d6abe4d6))
|
||||
* **config:** update application.yml for staging and production environments ([6113432](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/6113432a14c476a3a0dfc0d449e17d023697f2ba))
|
||||
* configure logging and add OpenTelemetry support ([#49](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/49)) ([d57c488](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/d57c4886612d1d92da0e1b79209fc83e6ef537a1))
|
||||
* **docker:** add .dockerignore and .gitignore files for build exclusions ([c987d8e](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/c987d8e258c0e6c4cfbdaa8381c64c410d7a2b83))
|
||||
* **docker:** add Dockerfiles for building Quarkus application in native and JVM modes ([3f2d2bb](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/3f2d2bb4c97fa8cddba66e1da4427c54236dfeed))
|
||||
* **docker:** add Dockerfiles for Quarkus application in JVM and native modes ([34b9933](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/34b993304670cf2aa62cd2f6460cee7b9864b08e))
|
||||
* **events:** migrate game-creation and bot flows to Redis Streams NCS-89 ([#62](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/62)) ([a24924c](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/a24924c23057db3d700a75dbc4333557789cd991))
|
||||
* **logging:** add DEBUG/INFO/WARN logging across services (NCS-72) ([#41](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/41)) ([804a4bf](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/804a4bf179e3dfb19e2be4390e7e543caf5237c6))
|
||||
* NCS-78 Add Traceability to the Applications ([#46](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/46)) ([649566e](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/649566eb3fcf38f91c8896a739f74ea318af312d))
|
||||
* NCS-78 Add Traceability to the Applications ([#47](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/47)) ([87dfc6c](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/87dfc6c2bcce7f7d58fc641bd8d468a2e584c108))
|
||||
* NCS-82 add Swiss-system tournament module ([#55](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/55)) ([c5661de](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/c5661de4a0ebf4b33211f5a391840dcf744656b7))
|
||||
* **reflection:** add native reflection configuration for tournament classes ([65bc6a7](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/65bc6a759937543df2d29905688bfa9e68d0c9d4))
|
||||
* true-microservices ([#40](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/40)) ([5909242](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/590924254e8a2754de661a57a03e43f89ceb6299))
|
||||
* update application.yml with new API root paths and add Micrometer and OpenTelemetry dependencies ([72ce262](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/72ce262bc491f94297700e6002fb5d0812e2cc2a))
|
||||
* **ws:** migrate challenge notifications to Redis Streams ([#66](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/66)) ([55f102c](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/55f102cbaa684be94a158b16aaa42a50b36afaf3))
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
* **account:** configure JDBC connection pool size to prevent exhaustion under load ([29072ef](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/29072efbfb1cfa1c3b1a85b4c1a587c971d245f9))
|
||||
* **auth:** add InternalClientHeadersFactory for custom client headers management ([e279c39](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/e279c39246470156bf11e745ee72204018d4229d))
|
||||
* enable official bots to connect to external tournament server ([#71](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/71)) ([688d30e](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/688d30e2b10026923372be5fca3c63eaaee2de2a))
|
||||
* NCS-84 More Verbose Logging ([#51](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/51)) ([4ad92ab](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/4ad92ab23698267f8faa59c4e18388d4a0042cca))
|
||||
* **official-bots:** NCS-70-auto-register official bots with account service ([#59](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/59)) ([7117a93](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/7117a93376272094d0b1a6abf2121254ce396684))
|
||||
* remove unused HTTP root-path configurations from application.yml ([3ed3e59](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/3ed3e59ee456d54cd3d65ece4f36623e256b9736))
|
||||
* **tests:** update token path to accessToken in ChallengeResourceTest ([354db11](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/354db11972342c47a1034303c11bccfb92e60109))
|
||||
|
||||
### Reverts
|
||||
|
||||
* Revert "feat: add authentication permissions for metrics endpoints in application.yml" ([a298417](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/a298417b9e4d68dc73bbf40be63d9484536e9f83))
|
||||
* Revert "refactor: update metrics paths formatting in application.yml for clarity" ([3870566](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/38705663498d5f47c40dafe2f26198589ede8656))
|
||||
|
||||
@@ -239,6 +239,7 @@ class AccountService:
|
||||
.subject(botId.toString)
|
||||
.expiresAt(Long.MaxValue)
|
||||
.claim("type", "bot")
|
||||
.claim("isBot", true)
|
||||
.claim("name", botName)
|
||||
.sign()
|
||||
|
||||
|
||||
@@ -1,3 +1,3 @@
|
||||
MAJOR=0
|
||||
MINOR=24
|
||||
MINOR=25
|
||||
PATCH=0
|
||||
|
||||
@@ -22,6 +22,9 @@
|
||||
// NOWCHESS_JDBC_URL (default: jdbc:postgresql://localhost:5432/nowchess)
|
||||
// NOWCHESS_DB_USER (default: nowchess)
|
||||
// NOWCHESS_DB_PASS (default: nowchess)
|
||||
// NOWCHESS_PGN_PATH (optional) — file or http(s) URL of a Lichess PGN dump (.pgn or .pgn.zst).
|
||||
// When set, all batch jobs read games from the dump instead of PostgreSQL and
|
||||
// skip JDBC write-back (Parquet/CSV output only). Demo data source.
|
||||
|
||||
plugins {
|
||||
id("scala")
|
||||
@@ -71,6 +74,10 @@ dependencies {
|
||||
|
||||
// PostgreSQL JDBC driver bundled so it is available on executor classpath.
|
||||
implementation("org.postgresql:postgresql:42.7.4")
|
||||
|
||||
// zstd-jni: decompress Lichess .pgn.zst dumps in-process. Provided at runtime by Spark
|
||||
// (it uses zstd-jni internally for shuffle/event-log compression), so compile-only here.
|
||||
compileOnly("com.github.luben:zstd-jni:1.5.6-9")
|
||||
}
|
||||
|
||||
application {
|
||||
|
||||
@@ -0,0 +1,119 @@
|
||||
package de.nowchess.analytics
|
||||
|
||||
import org.apache.spark.SparkFiles
|
||||
import org.apache.spark.sql.DataFrame
|
||||
import org.apache.spark.sql.SparkSession
|
||||
import org.apache.spark.sql.functions as F
|
||||
|
||||
/** Normalised game-record source for the batch jobs.
|
||||
*
|
||||
* Every batch job consumes the same five-column shape:
|
||||
* - white_id, black_id : player identifiers
|
||||
* - result : one of "white", "black", "draw"
|
||||
* - move_count : number of plies
|
||||
* - pgn : full PGN ("[Event …]…\n\n1. e4 …"), header and movetext separated by a blank line
|
||||
*
|
||||
* Two backends, selected by the `NOWCHESS_PGN_PATH` environment variable:
|
||||
* - unset → PostgreSQL `game_records` table (production)
|
||||
* - set → a Lichess PGN dump file/URL (demo). Point it at a `lichess_db_standard_rated_*.pgn[.zst]`
|
||||
* to drive every batch job from real Lichess games.
|
||||
*
|
||||
* Lichess parsing uses only Spark SQL string functions — no UDFs — so Catalyst can push predicates,
|
||||
* matching the no-UDF approach already used in OpeningBookJob.
|
||||
*/
|
||||
object GameSource:
|
||||
|
||||
private val PgnPathEnv = "NOWCHESS_PGN_PATH"
|
||||
|
||||
/** True when a Lichess PGN dump is configured; jobs use this to skip JDBC write-back. */
|
||||
def isPgnMode: Boolean = sys.env.contains(PgnPathEnv)
|
||||
|
||||
def load(spark: SparkSession, jdbcUrl: String, dbUser: String, dbPass: String): DataFrame =
|
||||
sys.env.get(PgnPathEnv) match
|
||||
case Some(path) => fromLichessPgn(spark, path)
|
||||
case None => fromJdbc(spark, jdbcUrl, dbUser, dbPass)
|
||||
|
||||
def fromJdbc(spark: SparkSession, jdbcUrl: String, dbUser: String, dbPass: String): DataFrame =
|
||||
spark.read
|
||||
.format("jdbc")
|
||||
.option("url", jdbcUrl)
|
||||
.option("dbtable", "game_records")
|
||||
.option("user", dbUser)
|
||||
.option("password", dbPass)
|
||||
.option("driver", "org.postgresql.Driver")
|
||||
.option("fetchsize", "10000")
|
||||
.load()
|
||||
.select("white_id", "black_id", "result", "move_count", "pgn")
|
||||
|
||||
/** Parses a Lichess PGN dump into the normalised game shape.
|
||||
*
|
||||
* `path` may be:
|
||||
* - an http(s)/ftp URL — fetched once via SparkContext.addFile and distributed to executors, then read
|
||||
* from the local replica (no S3/PVC needed; handy for a staging demo)
|
||||
* - any Hadoop-readable path (file://, hdfs://, s3a://, …)
|
||||
*
|
||||
* `.zst` dumps (Lichess' native format) are decompressed in-process via zstd-jni; `.gz`/`.bz2` are
|
||||
* handled by Spark's text reader codecs.
|
||||
*
|
||||
* Records are split on the "[Event " tag that opens every game, so each row holds one complete game
|
||||
* (the empty fragment before the first game is filtered out). Header tags are read with regexp_extract;
|
||||
* the movetext (after the blank line) is cleaned of clock/eval comments and move numbers to count plies.
|
||||
*/
|
||||
def fromLichessPgn(spark: SparkSession, path: String): DataFrame =
|
||||
val resolved = resolvePath(spark, path)
|
||||
val record = F.col("value")
|
||||
|
||||
val resultTag = F.regexp_extract(record, "Result \"([^\"]*)\"", 1)
|
||||
val result = F
|
||||
.when(resultTag === "1-0", "white")
|
||||
.when(resultTag === "0-1", "black")
|
||||
.when(resultTag === "1/2-1/2", "draw")
|
||||
.otherwise(F.lit(null).cast("string"))
|
||||
|
||||
val moveText = F.coalesce(F.split(record, "\n\n").getItem(1), F.lit(""))
|
||||
val noComment = F.regexp_replace(moveText, "\\{[^}]*\\}", "")
|
||||
val noResult = F.regexp_replace(noComment, "(1-0|0-1|1/2-1/2|\\*)", "")
|
||||
val noNumbers = F.regexp_replace(noResult, "\\d+\\.+", " ")
|
||||
val plies = F.size(F.filter(F.split(F.trim(noNumbers), "\\s+"), tok => F.length(tok) > 0))
|
||||
|
||||
spark.read
|
||||
.option("lineSep", "[Event ")
|
||||
.text(resolved)
|
||||
.filter(F.length(F.trim(record)) > 0)
|
||||
.select(
|
||||
F.regexp_extract(record, "White \"([^\"]*)\"", 1).as("white_id"),
|
||||
F.regexp_extract(record, "Black \"([^\"]*)\"", 1).as("black_id"),
|
||||
result.as("result"),
|
||||
plies.as("move_count"),
|
||||
F.concat(F.lit("[Event "), record).as("pgn"),
|
||||
)
|
||||
.filter((F.col("white_id") =!= "").and(F.col("black_id") =!= ""))
|
||||
|
||||
/** Turns an http(s)/ftp URL into a cluster-local path by fetching it once with SparkContext.addFile,
|
||||
* which distributes the file to every executor. `.zst` is decompressed in-process and the plain `.pgn`
|
||||
* is redistributed. Non-URL paths are returned unchanged.
|
||||
*/
|
||||
private def resolvePath(spark: SparkSession, path: String): String =
|
||||
if !path.matches("^(https?|ftp)://.*") then path
|
||||
else
|
||||
spark.sparkContext.addFile(path)
|
||||
val local = SparkFiles.get(baseName(path))
|
||||
if !local.endsWith(".zst") then "file://" + local
|
||||
else distribute(spark, decompressZstd(local))
|
||||
|
||||
private def baseName(path: String): String = path.substring(path.lastIndexOf('/') + 1)
|
||||
|
||||
private def distribute(spark: SparkSession, localPath: String): String =
|
||||
spark.sparkContext.addFile("file://" + localPath)
|
||||
"file://" + SparkFiles.get(baseName(localPath))
|
||||
|
||||
/** Decompresses a `.zst` file to a temp `.pgn` using zstd-jni (bundled with Spark at runtime). */
|
||||
private def decompressZstd(srcPath: String): String =
|
||||
val out = java.io.File.createTempFile("lichess-", ".pgn")
|
||||
out.deleteOnExit()
|
||||
val in = com.github.luben.zstd.ZstdInputStream(
|
||||
java.io.BufferedInputStream(java.io.FileInputStream(srcPath)),
|
||||
)
|
||||
try java.nio.file.Files.copy(in, out.toPath, java.nio.file.StandardCopyOption.REPLACE_EXISTING)
|
||||
finally in.close()
|
||||
out.getAbsolutePath
|
||||
@@ -37,15 +37,8 @@ object OpeningBookJob:
|
||||
outputDir: String,
|
||||
maxPlies: Int,
|
||||
): Unit =
|
||||
val games = spark.read
|
||||
.format("jdbc")
|
||||
.option("url", jdbcUrl)
|
||||
.option("dbtable", "game_records")
|
||||
.option("user", dbUser)
|
||||
.option("password", dbPass)
|
||||
.option("driver", "org.postgresql.Driver")
|
||||
.option("fetchsize", "10000")
|
||||
.load()
|
||||
val games = GameSource
|
||||
.load(spark, jdbcUrl, dbUser, dbPass)
|
||||
.select("pgn", "result")
|
||||
.filter(F.col("result").isNotNull.and(F.col("pgn").isNotNull))
|
||||
|
||||
@@ -79,15 +72,16 @@ object OpeningBookJob:
|
||||
.option("header", "true")
|
||||
.csv(s"$outputDir/opening_book_top1000")
|
||||
|
||||
top1000.write
|
||||
.mode("overwrite")
|
||||
.format("jdbc")
|
||||
.option("url", jdbcUrl)
|
||||
.option("dbtable", "analytics_opening_stats")
|
||||
.option("user", dbUser)
|
||||
.option("password", dbPass)
|
||||
.option("driver", "org.postgresql.Driver")
|
||||
.save()
|
||||
if !GameSource.isPgnMode then
|
||||
top1000.write
|
||||
.mode("overwrite")
|
||||
.format("jdbc")
|
||||
.option("url", jdbcUrl)
|
||||
.option("dbtable", "analytics_opening_stats")
|
||||
.option("user", dbUser)
|
||||
.option("password", dbPass)
|
||||
.option("driver", "org.postgresql.Driver")
|
||||
.save()
|
||||
|
||||
/** Extracts the first `maxPlies` moves from a PGN column as a space-separated string.
|
||||
*
|
||||
|
||||
@@ -50,15 +50,8 @@ object PlayerClusteringJob:
|
||||
outputDir: String,
|
||||
k: Int,
|
||||
): Unit =
|
||||
val games = spark.read
|
||||
.format("jdbc")
|
||||
.option("url", jdbcUrl)
|
||||
.option("dbtable", "game_records")
|
||||
.option("user", dbUser)
|
||||
.option("password", dbPass)
|
||||
.option("driver", "org.postgresql.Driver")
|
||||
.option("fetchsize", "10000")
|
||||
.load()
|
||||
val games = GameSource
|
||||
.load(spark, jdbcUrl, dbUser, dbPass)
|
||||
.select("white_id", "black_id", "result", "move_count")
|
||||
.filter(F.col("result").isNotNull)
|
||||
|
||||
@@ -126,25 +119,26 @@ object PlayerClusteringJob:
|
||||
.option("header", "true")
|
||||
.csv(s"$outputDir/cluster_archetypes")
|
||||
|
||||
clustersDf.write
|
||||
.mode("overwrite")
|
||||
.format("jdbc")
|
||||
.option("url", jdbcUrl)
|
||||
.option("dbtable", "analytics_player_clusters")
|
||||
.option("user", dbUser)
|
||||
.option("password", dbPass)
|
||||
.option("driver", "org.postgresql.Driver")
|
||||
.save()
|
||||
if !GameSource.isPgnMode then
|
||||
clustersDf.write
|
||||
.mode("overwrite")
|
||||
.format("jdbc")
|
||||
.option("url", jdbcUrl)
|
||||
.option("dbtable", "analytics_player_clusters")
|
||||
.option("user", dbUser)
|
||||
.option("password", dbPass)
|
||||
.option("driver", "org.postgresql.Driver")
|
||||
.save()
|
||||
|
||||
archetypes.write
|
||||
.mode("overwrite")
|
||||
.format("jdbc")
|
||||
.option("url", jdbcUrl)
|
||||
.option("dbtable", "analytics_cluster_archetypes")
|
||||
.option("user", dbUser)
|
||||
.option("password", dbPass)
|
||||
.option("driver", "org.postgresql.Driver")
|
||||
.save()
|
||||
archetypes.write
|
||||
.mode("overwrite")
|
||||
.format("jdbc")
|
||||
.option("url", jdbcUrl)
|
||||
.option("dbtable", "analytics_cluster_archetypes")
|
||||
.option("user", dbUser)
|
||||
.option("password", dbPass)
|
||||
.option("driver", "org.postgresql.Driver")
|
||||
.save()
|
||||
|
||||
private def buildPlayerStats(games: org.apache.spark.sql.DataFrame): org.apache.spark.sql.DataFrame =
|
||||
val asWhite = games.select(
|
||||
|
||||
@@ -53,15 +53,8 @@ object PlayerGraphJob:
|
||||
dbPass: String,
|
||||
outputDir: String,
|
||||
): Unit =
|
||||
val gamesRdd: RDD[Row] = spark.read
|
||||
.format("jdbc")
|
||||
.option("url", jdbcUrl)
|
||||
.option("dbtable", "game_records")
|
||||
.option("user", dbUser)
|
||||
.option("password", dbPass)
|
||||
.option("driver", "org.postgresql.Driver")
|
||||
.option("fetchsize", "10000")
|
||||
.load()
|
||||
val gamesRdd: RDD[Row] = GameSource
|
||||
.load(spark, jdbcUrl, dbUser, dbPass)
|
||||
.select("white_id", "black_id", "result")
|
||||
.filter(F.col("result").isNotNull)
|
||||
.rdd
|
||||
@@ -116,15 +109,16 @@ object PlayerGraphJob:
|
||||
.mode("overwrite")
|
||||
.parquet(s"$outputDir/player_graph")
|
||||
|
||||
result.write
|
||||
.mode("overwrite")
|
||||
.format("jdbc")
|
||||
.option("url", jdbcUrl)
|
||||
.option("dbtable", "analytics_player_graph")
|
||||
.option("user", dbUser)
|
||||
.option("password", dbPass)
|
||||
.option("driver", "org.postgresql.Driver")
|
||||
.save()
|
||||
if !GameSource.isPgnMode then
|
||||
result.write
|
||||
.mode("overwrite")
|
||||
.format("jdbc")
|
||||
.option("url", jdbcUrl)
|
||||
.option("dbtable", "analytics_player_graph")
|
||||
.option("user", dbUser)
|
||||
.option("password", dbPass)
|
||||
.option("driver", "org.postgresql.Driver")
|
||||
.save()
|
||||
|
||||
// How many players belong to each connected component?
|
||||
// A large dominant component + many singletons is the expected shape.
|
||||
|
||||
@@ -34,15 +34,8 @@ object PlayerStatsJob:
|
||||
dbPass: String,
|
||||
outputDir: String,
|
||||
): Unit =
|
||||
val games = spark.read
|
||||
.format("jdbc")
|
||||
.option("url", jdbcUrl)
|
||||
.option("dbtable", "game_records")
|
||||
.option("user", dbUser)
|
||||
.option("password", dbPass)
|
||||
.option("driver", "org.postgresql.Driver")
|
||||
.option("fetchsize", "10000")
|
||||
.load()
|
||||
val games = GameSource
|
||||
.load(spark, jdbcUrl, dbUser, dbPass)
|
||||
.select("white_id", "black_id", "result", "move_count")
|
||||
.filter(F.col("result").isNotNull)
|
||||
|
||||
@@ -84,12 +77,13 @@ object PlayerStatsJob:
|
||||
.mode("overwrite")
|
||||
.parquet(s"$outputDir/player_stats")
|
||||
|
||||
stats.write
|
||||
.mode("overwrite")
|
||||
.format("jdbc")
|
||||
.option("url", jdbcUrl)
|
||||
.option("dbtable", "analytics_player_stats")
|
||||
.option("user", dbUser)
|
||||
.option("password", dbPass)
|
||||
.option("driver", "org.postgresql.Driver")
|
||||
.save()
|
||||
if !GameSource.isPgnMode then
|
||||
stats.write
|
||||
.mode("overwrite")
|
||||
.format("jdbc")
|
||||
.option("url", jdbcUrl)
|
||||
.option("dbtable", "analytics_player_stats")
|
||||
.option("user", dbUser)
|
||||
.option("password", dbPass)
|
||||
.option("driver", "org.postgresql.Driver")
|
||||
.save()
|
||||
|
||||
@@ -281,3 +281,32 @@
|
||||
### Reverts
|
||||
|
||||
* Revert "refactor: update metrics paths formatting in application.yml for clarity" ([3870566](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/38705663498d5f47c40dafe2f26198589ede8656))
|
||||
## (2026-06-17)
|
||||
|
||||
### Features
|
||||
|
||||
* add initialization metrics for various services ([d438e97](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/d438e97f32bdde0bfc63c1b4a8cc810cdd093166))
|
||||
* add OpenTelemetry trace configuration with parentbased sampler ([3904d5a](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/3904d5ad8ad4930ddee65287a7bfab785a6148f5))
|
||||
* **analytics:** add Spark batch analytics module ([#70](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/70)) ([39f1657](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/39f1657e1db6e84889af338c43be8cb5c03c3ec3))
|
||||
* **config:** update application.yml for PostgreSQL and remove staging/production configurations ([2404e61](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/2404e6164c3b50ffccbea5238d636060d6abe4d6))
|
||||
* **config:** update application.yml for staging and production environments ([6113432](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/6113432a14c476a3a0dfc0d449e17d023697f2ba))
|
||||
* configure logging and add OpenTelemetry support ([#49](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/49)) ([d57c488](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/d57c4886612d1d92da0e1b79209fc83e6ef537a1))
|
||||
* **docker:** add .dockerignore and .gitignore files for build exclusions ([c987d8e](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/c987d8e258c0e6c4cfbdaa8381c64c410d7a2b83))
|
||||
* **docker:** add Dockerfiles for building Quarkus application in native and JVM modes ([3f2d2bb](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/3f2d2bb4c97fa8cddba66e1da4427c54236dfeed))
|
||||
* **docker:** add Dockerfiles for Quarkus application in JVM and native modes ([34b9933](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/34b993304670cf2aa62cd2f6460cee7b9864b08e))
|
||||
* **events:** migrate game-creation and bot flows to Redis Streams NCS-89 ([#62](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/62)) ([a24924c](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/a24924c23057db3d700a75dbc4333557789cd991))
|
||||
* NCS-78 Add Traceability to the Applications ([#46](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/46)) ([649566e](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/649566eb3fcf38f91c8896a739f74ea318af312d))
|
||||
* NCS-78 Add Traceability to the Applications ([#47](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/47)) ([87dfc6c](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/87dfc6c2bcce7f7d58fc641bd8d468a2e584c108))
|
||||
* NCS-82 add Swiss-system tournament module ([#55](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/55)) ([c5661de](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/c5661de4a0ebf4b33211f5a391840dcf744656b7))
|
||||
* **official-bots:** consume GameOver stream for bot cleanup ([#67](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/67)) ([db9d153](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/db9d1533912f4b41c4d1ca80ccffdde5d23d6ff6))
|
||||
* true-microservices ([#40](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/40)) ([5909242](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/590924254e8a2754de661a57a03e43f89ceb6299))
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
* enable official bots to connect to external tournament server ([#71](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/71)) ([688d30e](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/688d30e2b10026923372be5fca3c63eaaee2de2a))
|
||||
* **official-bots:** configure JWT verification ([#72](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/72)) ([98c64fc](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/98c64fc0d56dc542beb31c75f4b9056d91de03cd))
|
||||
* **official-bots:** NCS-70-auto-register official bots with account service ([#59](https://git.janis-eccarius.de/NowChess/NowChessSystems/issues/59)) ([7117a93](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/7117a93376272094d0b1a6abf2121254ce396684))
|
||||
|
||||
### Reverts
|
||||
|
||||
* Revert "refactor: update metrics paths formatting in application.yml for clarity" ([3870566](https://git.janis-eccarius.de/NowChess/NowChessSystems/commit/38705663498d5f47c40dafe2f26198589ede8656))
|
||||
|
||||
@@ -12,6 +12,12 @@ quarkus:
|
||||
enabled: true
|
||||
log:
|
||||
level: INFO
|
||||
mp:
|
||||
jwt:
|
||||
verify:
|
||||
publickey:
|
||||
location: ${JWT_PUBLIC_KEY_PATH:keys/public.pem}
|
||||
issuer: nowchess
|
||||
|
||||
nowchess:
|
||||
redis:
|
||||
|
||||
@@ -0,0 +1,9 @@
|
||||
-----BEGIN PUBLIC KEY-----
|
||||
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAxDsnsCAl0vQx7Vu9CLDZ
|
||||
g0SG05NgUzu9T+3DTEaHGq60T2uriO8BenwyvsF3BnDqTbKf4voohZ1DNfzdbT1J
|
||||
Fj8B62FrDmxcO+sp1/b5HUCJP6y2uSRCmzOHe5k7Pk1IEi72FgBpKXSRkFibRlVf
|
||||
634g7mgsPZAQ9PJEsv4Qvm05T9L6+Gmq6N3bMVLKRXs4RhDhaFbYH9GtUg1eI0yH
|
||||
YjGyRfqzW/nqVMstOLHt8CuPouq4p7eMzeDH3YHkxPm4GG5foCXMOd2DZrW0SCcr
|
||||
7dhFeNVWzQ2m53eOhBzNQX+v3pgjVStsePhBRt2LyGfwkNzmqDgqWsMzSHRMY+cn
|
||||
WQIDAQAB
|
||||
-----END PUBLIC KEY-----
|
||||
+1
@@ -2,6 +2,7 @@ package de.nowchess.bot.resource
|
||||
|
||||
case class JoinTournamentRequest(
|
||||
tournamentId: String,
|
||||
botToken: String,
|
||||
difficulty: String,
|
||||
serverUrl: Option[String],
|
||||
)
|
||||
|
||||
+1
-1
@@ -33,7 +33,7 @@ class TournamentJoinResource:
|
||||
difficulty,
|
||||
serverUrl,
|
||||
)
|
||||
player.joinTournament(req.tournamentId, difficulty, serverUrl) match
|
||||
player.joinTournament(req.tournamentId, req.botToken, difficulty, serverUrl) match
|
||||
case Right(botId) =>
|
||||
val resp = JoinTournamentResponse(botId, difficulty, "joining")
|
||||
Response.ok(resp).build()
|
||||
|
||||
+1
-1
@@ -20,7 +20,7 @@ object TournamentBotConfig:
|
||||
tournamentId <- env.get("TOURNAMENT_ID").filter(_.nonEmpty)
|
||||
token <- env.get("TOURNAMENT_BOT_TOKEN").filter(_.nonEmpty)
|
||||
botId <- jwtSubject(token)
|
||||
serverUrl = env.getOrElse("TOURNAMENT_SERVER_URL", "http://localhost:8089")
|
||||
serverUrl = env.getOrElse("TOURNAMENT_SERVER_URL", "http://141.37.123.132:8086")
|
||||
difficulty = env.getOrElse("TOURNAMENT_BOT_DIFFICULTY", "medium")
|
||||
yield TournamentBotConfig(serverUrl, tournamentId, token, botId, difficulty)
|
||||
|
||||
|
||||
+38
-26
@@ -39,10 +39,11 @@ class TournamentBotGamePlayer:
|
||||
// scalafix:on DisableSyntax.var
|
||||
|
||||
val defaultServerUrl: String =
|
||||
System.getenv().asScala.getOrElse("TOURNAMENT_SERVER_URL", "http://localhost:8089")
|
||||
System.getenv().asScala.getOrElse("TOURNAMENT_SERVER_URL", "http://141.37.123.132:8086")
|
||||
|
||||
@PostConstruct
|
||||
def initialize(): Unit =
|
||||
parkOnStartup()
|
||||
config match
|
||||
case None =>
|
||||
log.info("Tournament bot disabled — set TOURNAMENT_ID and TOURNAMENT_BOT_TOKEN to enable")
|
||||
@@ -50,11 +51,42 @@ class TournamentBotGamePlayer:
|
||||
log.infof("Tournament bot enabled — server=%s tournament=%s bot=%s", cfg.serverUrl, cfg.tournamentId, cfg.botId)
|
||||
startAsync(cfg)
|
||||
|
||||
def joinTournament(tournamentId: String, difficulty: String, serverUrl: String): Either[String, String] =
|
||||
registerBot(serverUrl, difficulty) match
|
||||
case None => Left("Failed to register bot with tournament server")
|
||||
case Some((botId, token)) =>
|
||||
val cfg = TournamentBotConfig(serverUrl, tournamentId, token, botId, difficulty)
|
||||
private def parkOnStartup(): Unit =
|
||||
park(defaultServerUrl, "expert") match
|
||||
case Some(id) => log.infof("Parked expert bot on %s as id %s", defaultServerUrl, id)
|
||||
case None => log.warnf("Failed to park expert bot on %s", defaultServerUrl)
|
||||
|
||||
private def park(serverUrl: String, difficulty: String): Option[String] =
|
||||
System.getenv().asScala.get("TOURNAMENT_BOT_TOKEN").filter(_.nonEmpty).flatMap { token =>
|
||||
Try {
|
||||
val body = s"""{"name":"${botName(difficulty)}"}"""
|
||||
val response = client
|
||||
.target(serverUrl)
|
||||
.path("api")
|
||||
.path("bots")
|
||||
.request(MediaType.APPLICATION_JSON)
|
||||
.header("Authorization", s"Bearer $token")
|
||||
.post(Entity.entity(body, MediaType.APPLICATION_JSON))
|
||||
if response.getStatus == 201 || response.getStatus == 200 then
|
||||
val id = objectMapper.readTree(response.readEntity(classOf[String])).path("id").asText()
|
||||
response.close()
|
||||
Option(id).filter(_.nonEmpty)
|
||||
else { log.warnf("Parking bot %s returned status %d", botName(difficulty), response.getStatus); response.close(); None }
|
||||
}.getOrElse(None)
|
||||
}
|
||||
|
||||
private def botName(difficulty: String): String = s"NowChess ${difficulty.capitalize}"
|
||||
|
||||
def joinTournament(
|
||||
tournamentId: String,
|
||||
botToken: String,
|
||||
difficulty: String,
|
||||
serverUrl: String,
|
||||
): Either[String, String] =
|
||||
TournamentBotConfig.jwtSubject(botToken) match
|
||||
case None => Left("Invalid bot token — could not extract subject")
|
||||
case Some(botId) =>
|
||||
val cfg = TournamentBotConfig(serverUrl, tournamentId, botToken, botId, difficulty)
|
||||
if join(cfg) then
|
||||
startAsync(cfg)
|
||||
Right(botId)
|
||||
@@ -65,26 +97,6 @@ class TournamentBotGamePlayer:
|
||||
thread.setDaemon(true)
|
||||
thread.start()
|
||||
|
||||
private def registerBot(serverUrl: String, difficulty: String): Option[(String, String)] =
|
||||
Try {
|
||||
val name = s"NowChess ${difficulty.capitalize}"
|
||||
val body = s"""{"name":"$name","isBot":true}"""
|
||||
val response = client
|
||||
.target(serverUrl)
|
||||
.path("api")
|
||||
.path("auth")
|
||||
.path("register")
|
||||
.request(MediaType.APPLICATION_JSON)
|
||||
.post(Entity.entity(body, MediaType.APPLICATION_JSON))
|
||||
if response.getStatus == 201 then
|
||||
val node = objectMapper.readTree(response.readEntity(classOf[String]))
|
||||
val id = node.path("id").asText()
|
||||
val token = node.path("token").asText()
|
||||
response.close()
|
||||
if id.nonEmpty && token.nonEmpty then Some((id, token)) else None
|
||||
else { log.warnf("Bot registration returned status %d", response.getStatus); response.close(); None }
|
||||
}.getOrElse(None)
|
||||
|
||||
@PreDestroy
|
||||
def cleanup(): Unit =
|
||||
running = false
|
||||
|
||||
@@ -1,3 +1,3 @@
|
||||
MAJOR=0
|
||||
MINOR=18
|
||||
MINOR=19
|
||||
PATCH=0
|
||||
|
||||
@@ -3,7 +3,6 @@ package de.nowchess.ws.resource
|
||||
import de.nowchess.ws.config.RedisConfig
|
||||
import io.micrometer.core.instrument.{Counter, Gauge, MeterRegistry}
|
||||
import io.quarkus.redis.datasource.RedisDataSource
|
||||
import io.quarkus.redis.datasource.pubsub.PubSubCommands
|
||||
import io.quarkus.websockets.next.*
|
||||
import io.smallrye.jwt.auth.principal.JWTParser
|
||||
import jakarta.annotation.PostConstruct
|
||||
@@ -34,6 +33,7 @@ class GameWebSocketResource:
|
||||
// scalafix:on DisableSyntax.var
|
||||
|
||||
private val connections = new ConcurrentHashMap[String, ConnectionMeta]()
|
||||
private val pendingAuth = ConcurrentHashMap.newKeySet[String]()
|
||||
|
||||
@PostConstruct
|
||||
def initializeMetrics(): Unit = {
|
||||
@@ -64,40 +64,60 @@ class GameWebSocketResource:
|
||||
s"${redisConfig.prefix}:game:$gameId:c2s"
|
||||
|
||||
@OnOpen
|
||||
def onOpen(connection: WebSocketConnection, handshake: HandshakeRequest): Unit =
|
||||
def onOpen(connection: WebSocketConnection): Unit =
|
||||
activeGauge
|
||||
val gameId = connection.pathParam("gameId")
|
||||
val playerId = resolvePlayerId(handshake)
|
||||
log.infof("Game WebSocket opened — gameId=%s playerId=%s", gameId, playerId.getOrElse("anonymous"))
|
||||
val gameId = connection.pathParam("gameId")
|
||||
val handler: Consumer[String] = msg => connection.sendText(msg).subscribe().`with`(_ => (), _ => ())
|
||||
val subscriber = redis.pubsub(classOf[String]).subscribe(s2cTopic(gameId), handler)
|
||||
connections.put(connection.id(), ConnectionMeta(gameId, subscriber, playerId))
|
||||
connections.put(connection.id(), ConnectionMeta(gameId, subscriber, None))
|
||||
connectionsOpened.increment()
|
||||
publishConnected(gameId, playerId)
|
||||
pendingAuth.add(connection.id())
|
||||
log.infof("Game WebSocket opened — gameId=%s connId=%s awaiting auth", gameId, connection.id())
|
||||
|
||||
@OnTextMessage
|
||||
def onTextMessage(connection: WebSocketConnection, message: String): Unit =
|
||||
messagesReceived.increment()
|
||||
Option(connections.get(connection.id())).foreach { meta =>
|
||||
val enriched = meta.playerId match
|
||||
case Some(pid) => injectPlayerId(message, pid)
|
||||
case None => message
|
||||
redis.pubsub(classOf[String]).publish(c2sTopic(meta.gameId), enriched)
|
||||
}
|
||||
if pendingAuth.remove(connection.id()) then
|
||||
val playerIdOpt =
|
||||
parseAuthToken(message)
|
||||
.flatMap(token => Try(jwtParser.parse(token)).toOption)
|
||||
.map(_.getSubject)
|
||||
playerIdOpt match
|
||||
case None =>
|
||||
log.warnf("Game WebSocket auth failed — closing connId=%s", connection.id())
|
||||
connection.close().subscribe().`with`(_ => (), _ => ())
|
||||
case Some(playerId) =>
|
||||
Option(connections.get(connection.id())).foreach { meta =>
|
||||
connections.put(connection.id(), meta.copy(playerId = Some(playerId)))
|
||||
publishConnected(meta.gameId, Some(playerId))
|
||||
}
|
||||
else
|
||||
Option(connections.get(connection.id())).foreach { meta =>
|
||||
val enriched = meta.playerId match
|
||||
case Some(pid) => injectPlayerId(message, pid)
|
||||
case None => message
|
||||
redis.pubsub(classOf[String]).publish(c2sTopic(meta.gameId), enriched)
|
||||
}
|
||||
|
||||
@OnClose
|
||||
def onClose(connection: WebSocketConnection): Unit =
|
||||
pendingAuth.remove(connection.id())
|
||||
Option(connections.remove(connection.id())).foreach { meta =>
|
||||
log.infof("Game WebSocket closed — gameId=%s", meta.gameId)
|
||||
meta.subscriber.unsubscribe(s2cTopic(meta.gameId))
|
||||
connectionsClosed.increment()
|
||||
}
|
||||
|
||||
private def resolvePlayerId(handshake: HandshakeRequest): Option[String] =
|
||||
Option(handshake.header("Authorization"))
|
||||
.filter(_.nonEmpty)
|
||||
.flatMap(token => Try(jwtParser.parse(token)).toOption)
|
||||
.map(_.getSubject)
|
||||
private def parseAuthToken(message: String): Option[String] =
|
||||
val trimmed = message.trim
|
||||
if !trimmed.contains("\"type\":\"auth\"") then None
|
||||
else
|
||||
val start = trimmed.indexOf("\"token\":\"")
|
||||
if start < 0 then None
|
||||
else
|
||||
val valueStart = start + 9
|
||||
val end = trimmed.indexOf('"', valueStart)
|
||||
if end < 0 then None else Some(trimmed.substring(valueStart, end)).filter(_.nonEmpty)
|
||||
|
||||
private def publishConnected(gameId: String, playerId: Option[String]): Unit =
|
||||
val connectedMsg = playerId match
|
||||
|
||||
@@ -32,6 +32,7 @@ class UserWebSocketResource:
|
||||
private val maxStreamLen = 1000L
|
||||
|
||||
private val connections = new ConcurrentHashMap[String, (String, WebSocketConnection)]()
|
||||
private val pendingAuth = ConcurrentHashMap.newKeySet[String]()
|
||||
|
||||
private def userStreamKey(userId: String): String =
|
||||
s"${redisConfig.prefix}:user:$userId:events:stream"
|
||||
@@ -39,29 +40,34 @@ class UserWebSocketResource:
|
||||
private def dlqKey: String = s"${redisConfig.prefix}:dlq"
|
||||
|
||||
@OnOpen
|
||||
def onOpen(connection: WebSocketConnection, handshake: HandshakeRequest): Unit =
|
||||
val userIdOpt = Option(handshake.header("Authorization"))
|
||||
.filter(_.nonEmpty)
|
||||
.flatMap(token => Try(jwtParser.parse(token)).toOption)
|
||||
.map(_.getSubject)
|
||||
def onOpen(connection: WebSocketConnection): Unit =
|
||||
pendingAuth.add(connection.id())
|
||||
|
||||
userIdOpt match
|
||||
case None =>
|
||||
log.warn("WebSocket opened with no valid JWT — closing connection")
|
||||
connection.close().subscribe().`with`(_ => (), _ => ())
|
||||
case Some(userId) =>
|
||||
log.infof("User WebSocket opened — userId=%s connId=%s", userId, connection.id())
|
||||
createGroupIfAbsent(userId, connection.id())
|
||||
connections.put(connection.id(), (userId, connection))
|
||||
executor.submit(
|
||||
new Runnable:
|
||||
def run(): Unit = pollLoop(connection.id(), userId, connection),
|
||||
)
|
||||
val connectedMsg = s"""{"type":"CONNECTED","userId":"$userId"}"""
|
||||
connection.sendText(connectedMsg).subscribe().`with`(_ => (), _ => ())
|
||||
@OnTextMessage
|
||||
def onTextMessage(connection: WebSocketConnection, message: String): Unit =
|
||||
if pendingAuth.remove(connection.id()) then
|
||||
val userIdOpt =
|
||||
parseAuthToken(message)
|
||||
.flatMap(token => Try(jwtParser.parse(token)).toOption)
|
||||
.map(_.getSubject)
|
||||
userIdOpt match
|
||||
case None =>
|
||||
log.warn("WebSocket opened with no valid JWT — closing connection")
|
||||
connection.close().subscribe().`with`(_ => (), _ => ())
|
||||
case Some(userId) =>
|
||||
log.infof("User WebSocket opened — userId=%s connId=%s", userId, connection.id())
|
||||
createGroupIfAbsent(userId, connection.id())
|
||||
connections.put(connection.id(), (userId, connection))
|
||||
executor.submit(
|
||||
new Runnable:
|
||||
def run(): Unit = pollLoop(connection.id(), userId, connection),
|
||||
)
|
||||
val connectedMsg = s"""{"type":"CONNECTED","userId":"$userId"}"""
|
||||
connection.sendText(connectedMsg).subscribe().`with`(_ => (), _ => ())
|
||||
|
||||
@OnClose
|
||||
def onClose(connection: WebSocketConnection): Unit =
|
||||
pendingAuth.remove(connection.id())
|
||||
log.infof("User WebSocket closed — connectionId=%s", connection.id())
|
||||
val userIdOpt = Option(connections.remove(connection.id())).map(_._1)
|
||||
userIdOpt.foreach { userId =>
|
||||
@@ -128,3 +134,14 @@ class UserWebSocketResource:
|
||||
) match
|
||||
case Failure(ex) => log.warnf(ex, "Failed to publish to stream %s", key)
|
||||
case Success(_) => ()
|
||||
|
||||
private def parseAuthToken(message: String): Option[String] =
|
||||
val trimmed = message.trim
|
||||
if !trimmed.contains("\"type\":\"auth\"") then None
|
||||
else
|
||||
val start = trimmed.indexOf("\"token\":\"")
|
||||
if start < 0 then None
|
||||
else
|
||||
val valueStart = start + 9
|
||||
val end = trimmed.indexOf('"', valueStart)
|
||||
if end < 0 then None else Some(trimmed.substring(valueStart, end)).filter(_.nonEmpty)
|
||||
|
||||
Reference in New Issue
Block a user