Enable DEBUG for io.smallrye.jwt and io.quarkus.security on the
official-bots rollout in staging only (container env, not the shared
ConfigMap) to diagnose the persistent 401. Remove after root cause.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The official-bots rollout never mounted the ncs-jwt-keys secret nor set
JWT_PUBLIC_KEY_PATH, so the service fell back to the image-baked
public.pem, which does not match the deployed signing key. Every token
failed RS256 verification -> 401.
Mount ncs-jwt-keys at /secrets/jwt and point JWT_PUBLIC_KEY_PATH there,
mirroring account/core/tournament. Verify-only, no private key.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
With SPARK_WORKER_CORES=1 and 2 workers the cluster has 2 total cores.
LiveDashboard holds both — batch jobs wait indefinitely. 4 cores per
worker gives 8 total, enough to run streaming + batch concurrently.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
bitnami-labs GitHub Pages repo (bitnami-labs.github.io/sealed-secrets)
returns 404 — chart moved to charts.bitnami.com/bitnami. Updates chart
version from 2.16.1 to 2.5.19 (controller 0.27.1 → 0.31.0). Controller
is backward-compatible with existing sealed secrets.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Cluster stable. Re-enabling automated sync on spark-analytics and
spark-cluster. Driver DNS fix and Recreate strategy already in main,
so next sync deploys the corrected manifests.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
CPU saturation caused an unclean AOF write, corrupting the tail of
appendonly.aof.41.incr.aof at offset 32653224. Redis refuses to start.
Setting aof-load-corrupt-tail-max-size to 88 (value Redis reported) so
it truncates and loads the intact data instead of crash-looping.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Cluster saturated at 100% CPU due to Spark executor crash loop.
Disabling automated sync on spark-analytics and spark-cluster to prevent
further changes until cluster is stable.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Rolling update deadlocked: new pod couldn't attach the RWO checkpoint
PVC while the old pod still held it, so ContainerCreating stalled for
39+ minutes. Recreate terminates the old pod first.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Spark executors crashed immediately (exit 1) because the driver
advertised its pod name as the RPC hostname, which is not DNS-resolvable
in-cluster. Added headless Service `spark-live-dashboard-driver` and
pinned spark.driver.host/port + spark.blockManager.port so workers can
connect back to the driver. Stops the JVM fork storm that saturated all
three nodes at ~100% CPU.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Nodes have only ~60-210Mi free. Requests are scheduler hints only —
pods burst to 1Gi limit once placed. 64Mi fits on all three nodes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Nodes are saturated on requested resources. Driver pods (streaming +
cronjobs) were requesting 500m CPU + 768Mi each. Limits unchanged at
1 CPU / 1Gi — pods can still burst when resources are free.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
local-path PVCs are created root-owned. The apache/spark image runs as
spark (UID 185). fsGroup: 185 makes K8s chown the mounted volume to that
group on attach, fixing the mkdir IOException on /spark-checkpoint.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
local-path provisioner is RWO-only; attaching a single PVC to master, two
workers, history-server, and streaming pod on separate nodes causes
Multi-Attach errors. Each pod now gets its own ephemeral emptyDir for
/spark-events. Event logging confs removed from CronJobs and streaming
Deployment (logs are pod-local and do not persist across restarts).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
K8s injects SPARK_MASTER_PORT=tcp://<ip>:7077 because the Service is
named spark-master. Spark master parses it as integer -> NumberFormatException.
enableServiceLinks: false suppresses all Docker-link style env var injection.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
3.5.x has no official Docker Hub image. 4.0.3 is the oldest available
scala2.13 variant on Docker Hub.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Scheduler uses requests (not limits) for placement. Previous 1Gi requests
per pod caused Insufficient memory even with plenty of physical RAM.
New requests: master 256Mi, workers 256Mi, history-server 128Mi.
Limits unchanged. SPARK_WORKER_MEMORY reduced to 512m to match worker budget.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Bitnami Spark chart requires a Broadcom commercial license. Replace with
plain Deployments using apache/spark:3.5.4-scala2.13-java17-ubuntu
(Apache 2.0, same image used by the analytics jobs):
- spark-master: Deployment + ClusterIP Service (ports 7077/8080)
- spark-worker: Deployment, 2 replicas, connects to spark-master:7077
- spark-history-server: updated to apache/spark image
Ingress backend corrected to spark-master:8080.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The Bitnami spark chart 9.2.10 does not render a history server despite
historyServer.enabled=true in values. Add an explicit Deployment and
ClusterIP Service (spark-history-server-svc:18080) using the same
bitnami/spark:3.5.2 image, wired to the shared spark-events PVC.
The existing spark-history-ingress now has a valid backend.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Janis
svc.kargo
Janis Eccarius