Track whether subscribeGame is being called and completing successfully
to diagnose empty game-writeback messages.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
When instance stream drops, immediately force delete the K8s pod (grace period 0). No waiting for health check or pod watch events.
Reduces failover latency and ensures stale pods don't linger.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Scale up: only if resource constrained (CPU/memory)
Scale down: only if NOT resource constrained AND game load low
Remove: triggering scale-up on unexpected instance failures
Keep: health monitoring (mark dead, delete pod, failover games) but no scaling
Prevents cascade scaling from transient health check failures.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Accidentally flipped pod matching direction in previous commits. Changed from correct instanceId.contains(podName) to incorrect podName.contains(instanceId), causing all health checks to fail.
Reverted all 3 locations to original working logic.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Pod matching used endsWith(instanceId) which failed to match any pods because instanceId is randomly generated 8-char string, not pod name suffix. All instances marked dead causing cascading health check failures.
Changed to podName.contains(instanceId) to match instanceId embedded anywhere in pod name. Reverting incomplete fix from previous commit.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
- Add lastUpdatedMs timestamp to GameCacheDto to track actual game updates instead of heartbeat time. Fix cache eviction incorrectly marking correspondence games as idle.
- Use atomic SPOP in LoadBalancer.getGamesToMove() to prevent concurrent rebalance calls from selecting same games for migration.
- Add game→instance reverse mapping (nowchess:game:$gameId:instance) to eliminate O(instances) linear scan during cache eviction.
- Fix HealthMonitor pod matching from loose contains() to reliable endsWith() to prevent matching unintended pods with similar names.
- Update FailoverService to maintain game→instance mappings when migrating games during failover.
- Update CacheEvictionManager to use game→instance mapping for O(1) lookup instead of O(n) instance scan.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Instances were being incorrectly marked DEAD because their Redis key TTL was
not being refreshed on heartbeat. HealthMonitor.checkRedisHeartbeat() checks
pttl > 0, which fails when the TTL expires even if the instance is alive and
sending regular heartbeats.
Now pexpire(key, heartbeatTtl) is called on each heartbeat to keep the key
alive. Prevents scaling messages from undercounting healthy instances.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
When pod deletion fails, instances remained in registry with state=DEAD,
preventing scale-down since avgLoad calculation counted them. Now:
- Use gracePeriod(0) for immediate pod deletion instead of 30s wait
(prevents cascade when nodes are down or pods stuck terminating)
- Remove instance from registry on deletion failure anyway
(prevents dead instances from blocking scale-down via avgLoad)
This breaks the cycle: failed deletions → scaleUp → max replicas →
more failures → more stuck instances blocking recovery.
- Wrap asInstanceOf casts with scalafix:off/on in isResourceConstrained
- Replace var with immutable List in HealthMonitor.checkInstanceHealth
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
When an instance is evicted or fails health check, immediately trigger scale-up
to replace the lost capacity. Don't wait for the next scheduled scale check.
HealthMonitor now calls autoScaler.scaleUp() when:
1. Stale instances are evicted
2. Instance fails health check and is marked dead
Ensures quick recovery from instance loss.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
AutoScaler now checks K8s pod metrics (CPU) in addition to subscription count.
Scale-up triggers if:
1. avgLoad > scaleUpThreshold * maxGamesPerCore, OR
2. Any instance has CPU > 800m
Fixes scenario where instance under heavy CPU load wouldn't scale without
high subscription count. Now responds to compute utilization, not just game count.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
checkAndScale was deciding to scale-down based on avgLoad without checking
if already at minimum replicas. This caused unnecessary scale-down attempts
that would fail and log noise.
Add check: only scale-down if instances.size > scaleMinReplicas
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Scale-down was calling failoverService.onInstanceStreamDropped synchronously
and waiting for it to complete. Failover retries for up to 30s waiting for
healthy instances, which blocks the Quarkus event loop thread.
This caused:
- Event loop blocked for 15+ seconds
- Redis health checks timing out (also on event loop)
- Scale-down operations failing
Fix: Trigger drain asynchronously without waiting. Scale-down proceeds
immediately while drain happens in background.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Critical fixes:
- Enable auto-scaling (was disabled in config)
- Add periodic cache eviction (5m interval) — CacheEvictionManager never ran
- Add periodic rebalance check (30s) — proactive load balancing
- Add 5s timeout to all gRPC calls (batchResubscribe, unsubscribe, evict)
- Use Option instead of null checks (scalafix compliance)
These gaps left the coordinator unable to:
1. Scale up when instances overloaded (scaling was disabled)
2. Clean up idle games from memory (no scheduled eviction)
3. Rebalance load proactively (only on scale-up)
4. Handle hung instances (no RPC timeouts, operations could hang forever)
Combined with prior fixes for instance metadata parsing and heartbeat TTL,
the coordinator now handles overload scenarios correctly.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
TeamCity
Janis