feat(analytics): add Structured Streaming, MLlib clustering, GraphX jobs

Three new Spark jobs demonstrating complementary Spark pillars:

LiveDashboardJob (Structured Streaming):
- Simulates NowChess game-over event stream via rate source
- Watermarking (45 s late-data tolerance)
- Tumbling 1-min windows → append-mode Parquet output
- Sliding 5-min/1-min windows → update-mode console output
- Checkpointing for exactly-once fault tolerance
- Production wiring comments show Kafka / spark-redis swap-in

PlayerClusteringJob (MLlib):
- Derives 4 player features from game_records via JDBC
- VectorAssembler + StandardScaler + KMeans inside a Pipeline
- ClusteringEvaluator (silhouette score) to measure quality
- Per-cluster archetype averages show what each tier represents

PlayerGraphJob (GraphX):
- Builds directed player graph (vertices=players, edges=games)
- PageRank — identifies most influential/active players
- ConnectedComponents — finds isolated player communities
- Bridges GraphX RDD results back to DataFrames via explicit schema
  (avoids spark.implicits._ which breaks Scala 3 → Spark 2.13 interop)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Janis Eccarius
2026-06-15 22:15:24 +02:00
parent 259b3bbb24
commit e1d80b9331
4 changed files with 449 additions and 0 deletions
+6
View File
@@ -62,6 +62,12 @@ dependencies {
compileOnly("org.apache.spark:spark-core_2.13:$sparkVersion") {
exclude(group = "org.slf4j", module = "slf4j-log4j12")
}
compileOnly("org.apache.spark:spark-mllib_2.13:$sparkVersion") {
exclude(group = "org.slf4j", module = "slf4j-log4j12")
}
compileOnly("org.apache.spark:spark-graphx_2.13:$sparkVersion") {
exclude(group = "org.slf4j", module = "slf4j-log4j12")
}
// PostgreSQL JDBC driver bundled so it is available on executor classpath.
implementation("org.postgresql:postgresql:42.7.4")