Janis Eccarius
a268a9acb7
Build & Test (NowChessSystems) TeamCity build finished
SparkFiles.get() on the driver returns a driver-local path. When this was passed to spark.read.text() the executor tried to open that path on its own filesystem (separate pod), silently reading 0 rows. Fix: download and decompress the Lichess PGN to NOWCHESS_PGN_CACHE_DIR (default /tmp) which must be a filesystem shared between driver and executor pods. In the k8s deployment this is the spark-analytics-output PVC mounted at /spark-output, so set NOWCHESS_PGN_CACHE_DIR=/spark-output/.pgn-cache. Also caches the decompressed file across runs — skips download if already present. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>