NowChess/GitOps
4
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

merge into: NowChess/GitOps:feat/spark-analytics
Branches Tags
NowChess/GitOps:main NowChess/GitOps:feat/tournament-bot-env-config NowChess/GitOps:kargo/promotion/eu-central-1-prod.01kv905910jc7qk9vprc946cpy.1bf7ff3 NowChess/GitOps:feat/spark-analytics
...
pull from: NowChess/GitOps:kargo/promotion/eu-central-1-prod.01kv905910jc7qk9vprc946cpy.1bf7ff3
Branches Tags
NowChess/GitOps:main NowChess/GitOps:feat/tournament-bot-env-config NowChess/GitOps:kargo/promotion/eu-central-1-prod.01kv905910jc7qk9vprc946cpy.1bf7ff3 NowChess/GitOps:feat/spark-analytics

26 Commits
feat/spark-analytics ... kargo/promotion/eu-central-1-prod.01kv905910jc7qk9vprc946cpy.1bf7ff3

Author SHA1 Message Date
Kargo 3e71ed7dda chore(kargo): update image tag official-bots to 0.18.0 on eu-central-1-prod
Delete PR Branch On Close / delete-branch (pull_request) Has been skipped
Details
2026-06-16 19:57:32 +00:00
svc.kargo 7f17905b68 chore(kargo): update image tag analysis to 0.2.0 on staging (#395) 2026-06-16 21:53:37 +02:00
svc.kargo 7b95f16361 chore(kargo): update image tag official-bots to 0.18.0 on staging (#394) 2026-06-16 21:52:58 +02:00
svc.kargo e61cf790b1 chore(kargo): update image tag tournament to 0.3.0 on staging (#393) 2026-06-16 21:49:31 +02:00
Janis Eccarius e08e8313ff fix(spark): use nginx ingress class instead of traefik 
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-06-16 20:42:14 +02:00
Janis Eccarius ab95b4f101 fix(spark): replace shared spark-events PVC with emptyDir to fix Multi-Attach error 
local-path provisioner is RWO-only; attaching a single PVC to master, two
workers, history-server, and streaming pod on separate nodes causes
Multi-Attach errors. Each pod now gets its own ephemeral emptyDir for
/spark-events. Event logging confs removed from CronJobs and streaming
Deployment (logs are pod-local and do not persist across restarts).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-06-16 20:33:48 +02:00
Janis Eccarius e20bedbff5 fix(spark): disable service link env vars on master pod 
K8s injects SPARK_MASTER_PORT=tcp://<ip>:7077 because the Service is
named spark-master. Spark master parses it as integer -> NumberFormatException.
enableServiceLinks: false suppresses all Docker-link style env var injection.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-06-16 20:14:37 +02:00
Janis Eccarius aa8d04a1cc fix(spark): upgrade image to apache/spark:4.0.3-scala2.13-java17-ubuntu 
3.5.x has no official Docker Hub image. 4.0.3 is the oldest available
scala2.13 variant on Docker Hub.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-06-16 20:11:38 +02:00
Janis Eccarius 710e2ec308 fix(spark): lower memory requests to fix scheduling on resource-constrained nodes 
Scheduler uses requests (not limits) for placement. Previous 1Gi requests
per pod caused Insufficient memory even with plenty of physical RAM.
New requests: master 256Mi, workers 256Mi, history-server 128Mi.
Limits unchanged. SPARK_WORKER_MEMORY reduced to 512m to match worker budget.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-06-16 20:02:02 +02:00
Janis Eccarius 25f230fd18 chore(spark): integrate upstream main into spark-analytics branch 2026-06-16 19:58:26 +02:00
Janis Eccarius d962dfda22 feat(spark): replace Bitnami chart with Apache Spark manifests 
Bitnami Spark chart requires a Broadcom commercial license. Replace with
plain Deployments using apache/spark:3.5.4-scala2.13-java17-ubuntu
(Apache 2.0, same image used by the analytics jobs):
- spark-master: Deployment + ClusterIP Service (ports 7077/8080)
- spark-worker: Deployment, 2 replicas, connects to spark-master:7077
- spark-history-server: updated to apache/spark image
Ingress backend corrected to spark-master:8080.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-06-16 19:48:51 +02:00
Janis 63e9d0512c feat(nci-11): add Kubernetes manifests for nowchess-analysis service (#392) 
Co-authored-by: Janis Eccarius <eccariusjanis@gmail.com>
Reviewed-on: #392
 2026-06-16 19:42:04 +02:00
svc.kargo 0442352b3e chore(kargo): update image tag frontend to 0.3.0 on eu-central-1-prod (#391) 2026-06-16 10:37:05 +02:00
svc.kargo 44b3369501 chore(kargo): update image tag tournament to 0.2.0 on eu-central-1-prod (#390) 2026-06-16 10:36:25 +02:00
svc.kargo fd62f236fa chore(kargo): update image tag core to 0.51.0 on eu-central-1-prod (#389) 2026-06-16 10:35:46 +02:00
svc.kargo d0d4370c50 chore(kargo): update image tag tournament to 0.2.0-x86_v2 on htwg-1-prod (#388) 2026-06-16 10:35:25 +02:00
svc.kargo 1dd1cacbb4 chore(kargo): update image tag core to 0.51.0-x86_v2 on htwg-1-prod (#387) 2026-06-16 10:35:20 +02:00
svc.kargo 1a3403a48e chore(kargo): update image tag analysis to 0.2.0-x86_v2 on htwg-1-prod (#386) 2026-06-16 10:35:12 +02:00
svc.kargo 9b906edfc9 chore(kargo): update image tag analysis to 0.2.0 on eu-central-1-prod (#385) 2026-06-16 10:35:05 +02:00
svc.kargo d6a05d2e1e chore(kargo): update image tag frontend to 0.3.0-x86_v2 on htwg-1-prod (#384) 2026-06-16 10:35:04 +02:00
svc.kargo 0bb5aabf98 chore(kargo): update image tag analysis to 0.2.0 on staging (#383) 2026-06-15 23:01:39 +02:00
svc.kargo 767a7f28a1 chore(kargo): update image tag tournament to 0.2.0 on staging (#382) 2026-06-15 22:23:21 +02:00
Janis 0373f61ef7 feat(nci-11): add Kubernetes manifests for nowchess-analysis service (#381) 
Add Argo Rollout (blueGreen), active/preview Services, and HPA for the
new analysis service (port 8087). Wire image pins into all three overlay
kustomizations and register the analysis Warehouse and freight entries in
all three Kargo Stages (staging -> eu-central-1-prod / htwg-1-prod).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Janis Eccarius <eccariusjanis@gmail.com>
Reviewed-on: #381
 2026-06-15 22:17:45 +02:00
shosho996 b546313e79 feat: NCI-12 kargo (#364) 
Co-authored-by: Lala, Shahd <Shahd.Lala@sybit.de>
Reviewed-on: #364
Co-authored-by: Shahd Lala <shosho996@blackhole.local>
Co-committed-by: Shahd Lala <shosho996@blackhole.local>
 2026-06-15 22:17:32 +02:00
svc.kargo afdc648347 chore(kargo): update image tag kargo to 1.10.7 (#380) 2026-06-12 18:23:35 +02:00
svc.kargo 93793671e3 chore(kargo): update image tag frontend to 0.3.0 on staging (#379) 2026-06-10 22:45:48 +02:00
15 changed files with 177 additions and 166 deletions
Expand all files Collapse all files
eu-central-1/argo-apps/kargo/kargo.yaml
+1 -1
View File
@@ -11,7 +11,7 @@ spec:
sources:
- repoURL: ghcr.io/akuity/kargo-charts
chart: kargo
targetRevision: 1.10.6
targetRevision: 1.10.7
helm:
valueFiles:
- $values/kargo/eu-central-1/values.yaml
kargo-projects/nowchess/ncs-stage.yaml
+20
View File
@@ -68,6 +68,12 @@ spec:
sources:
direct: true
requiredSoakTime: 5m0s
- origin:
kind: Warehouse
name: tournament
sources:
direct: true
requiredSoakTime: 5m0s
- origin:
kind: Warehouse
name: ws
@@ -168,6 +174,13 @@ spec:
direct: false
stages:
- staging
- origin:
kind: Warehouse
name: tournament
sources:
direct: false
stages:
- staging
- origin:
kind: Warehouse
name: ws
@@ -267,6 +280,13 @@ spec:
direct: false
stages:
- staging
- origin:
kind: Warehouse
name: tournament
sources:
direct: false
stages:
- staging
- origin:
kind: Warehouse
name: ws
kargo-projects/nowchess/ncs-warehouse.yaml
+15
View File
@@ -135,6 +135,21 @@ spec:
---
apiVersion: kargo.akuity.io/v1alpha1
kind: Warehouse
metadata:
name: tournament
namespace: nowchess-kargo
spec:
freightCreationPolicy: Automatic
interval: 5m0s
subscriptions:
- image:
discoveryLimit: 20
imageSelectionStrategy: SemVer
repoURL: ghcr.io/now-chess/now-chess-systems/tournament
strictSemvers: true
---
apiVersion: kargo.akuity.io/v1alpha1
kind: Warehouse
metadata:
name: ws
namespace: nowchess-kargo
nowchess/eu-central-1-prod/kustomization.yaml
+6 -6
View File
@@ -32,28 +32,28 @@ patches:
kind: ConfigMap
name: nowchess-frontend-env-config
images:
- name: ghcr.io/now-chess/now-chess-systems/analysis
newTag: 0.1.0
- name: ghcr.io/now-chess/now-chess-systems/account
newTag: 0.24.0
- name: ghcr.io/now-chess/now-chess-systems/analysis
newTag: 0.1.0
- name: ghcr.io/now-chess/now-chess-systems/bot-platform
newTag: 0.12.0
- name: ghcr.io/now-chess/now-chess-systems/coordinator
newTag: 0.32.0
- name: ghcr.io/now-chess/now-chess-systems/core
newTag: 0.50.0
newTag: 0.51.0
- name: ghcr.io/now-chess/now-chess-systems/frontend
newTag: 0.2.7
newTag: 0.3.0
- name: ghcr.io/now-chess/now-chess-systems/io
newTag: 0.23.0
- name: ghcr.io/now-chess/now-chess-systems/official-bots
newTag: 0.17.0
newTag: 0.18.0
- name: ghcr.io/now-chess/now-chess-systems/rule
newTag: 0.18.0
- name: ghcr.io/now-chess/now-chess-systems/store
newTag: 0.24.0
- name: ghcr.io/now-chess/now-chess-systems/tournament
newTag: 0.1.0
newTag: 0.2.0
- name: ghcr.io/now-chess/now-chess-systems/ws
newTag: 0.16.0
configurations:
nowchess/eu-central-1-staging/kustomization.yaml
+5 -5
View File
@@ -94,10 +94,10 @@ patches:
spec:
replicas: 1
images:
- name: ghcr.io/now-chess/now-chess-systems/analysis
newTag: 0.1.0
- name: ghcr.io/now-chess/now-chess-systems/account
newTag: 0.24.0
- name: ghcr.io/now-chess/now-chess-systems/analysis
newTag: 0.2.0
- name: ghcr.io/now-chess/now-chess-systems/bot-platform
newTag: 0.12.0
- name: ghcr.io/now-chess/now-chess-systems/coordinator
@@ -105,17 +105,17 @@ images:
- name: ghcr.io/now-chess/now-chess-systems/core
newTag: 0.51.0
- name: ghcr.io/now-chess/now-chess-systems/frontend
newTag: 0.2.8
newTag: 0.3.0
- name: ghcr.io/now-chess/now-chess-systems/io
newTag: 0.23.0
- name: ghcr.io/now-chess/now-chess-systems/official-bots
newTag: 0.17.0
newTag: 0.18.0
- name: ghcr.io/now-chess/now-chess-systems/rule
newTag: 0.18.0
- name: ghcr.io/now-chess/now-chess-systems/store
newTag: 0.24.0
- name: ghcr.io/now-chess/now-chess-systems/tournament
newTag: 0.1.0
newTag: 0.3.0
- name: ghcr.io/now-chess/now-chess-systems/ws
newTag: 0.16.0
configurations:
nowchess/htwg-1-prod/kustomization.yaml
+5 -3
View File
@@ -35,14 +35,16 @@ images:
newTag: 0.1.0-x86_v2
- name: ghcr.io/now-chess/now-chess-systems/account
newTag: 0.24.0-x86_v2
- name: ghcr.io/now-chess/now-chess-systems/analysis
newTag: 0.2.0-x86_v2
- name: ghcr.io/now-chess/now-chess-systems/bot-platform
newTag: 0.12.0-x86_v2
- name: ghcr.io/now-chess/now-chess-systems/coordinator
newTag: 0.32.0-x86_v2
- name: ghcr.io/now-chess/now-chess-systems/core
newTag: 0.50.0-x86_v2
newTag: 0.51.0-x86_v2
- name: ghcr.io/now-chess/now-chess-systems/frontend
newTag: 0.2.7-x86_v2
newTag: 0.3.0-x86_v2
- name: ghcr.io/now-chess/now-chess-systems/io
newTag: 0.23.0-x86_v2
- name: ghcr.io/now-chess/now-chess-systems/official-bots
@@ -52,6 +54,6 @@ images:
- name: ghcr.io/now-chess/now-chess-systems/store
newTag: 0.24.0-x86_v2
- name: ghcr.io/now-chess/now-chess-systems/tournament
newTag: 0.1.0-x86_v2
newTag: 0.2.0-x86_v2
- name: ghcr.io/now-chess/now-chess-systems/ws
newTag: 0.16.0-x86_v2
spark-analytics/base/cronjobs.yaml
-36
View File
@@ -34,10 +34,6 @@ spec:
- spark.executor.memory=1g
- --conf
- spark.driver.memory=512m
- --conf
- spark.eventLog.enabled=true
- --conf
- spark.eventLog.dir=/spark-events
- --class
- de.nowchess.analytics.OpeningBookJob
- local:///app/analytics.jar
@@ -69,15 +65,10 @@ spec:
volumeMounts:
- name: spark-output
mountPath: /spark-output
- name: spark-events
mountPath: /spark-events
volumes:
- name: spark-output
persistentVolumeClaim:
claimName: spark-analytics-output
- name: spark-events
persistentVolumeClaim:
claimName: spark-events
---
apiVersion: batch/v1
kind: CronJob
@@ -115,10 +106,6 @@ spec:
- spark.executor.memory=1g
- --conf
- spark.driver.memory=512m
- --conf
- spark.eventLog.enabled=true
- --conf
- spark.eventLog.dir=/spark-events
- --class
- de.nowchess.analytics.PlayerStatsJob
- local:///app/analytics.jar
@@ -149,15 +136,10 @@ spec:
volumeMounts:
- name: spark-output
mountPath: /spark-output
- name: spark-events
mountPath: /spark-events
volumes:
- name: spark-output
persistentVolumeClaim:
claimName: spark-analytics-output
- name: spark-events
persistentVolumeClaim:
claimName: spark-events
---
apiVersion: batch/v1
kind: CronJob
@@ -195,10 +177,6 @@ spec:
- spark.executor.memory=1g
- --conf
- spark.driver.memory=512m
- --conf
- spark.eventLog.enabled=true
- --conf
- spark.eventLog.dir=/spark-events
- --class
- de.nowchess.analytics.PlayerClusteringJob
- local:///app/analytics.jar
@@ -230,15 +208,10 @@ spec:
volumeMounts:
- name: spark-output
mountPath: /spark-output
- name: spark-events
mountPath: /spark-events
volumes:
- name: spark-output
persistentVolumeClaim:
claimName: spark-analytics-output
- name: spark-events
persistentVolumeClaim:
claimName: spark-events
---
apiVersion: batch/v1
kind: CronJob
@@ -276,10 +249,6 @@ spec:
- spark.executor.memory=1g
- --conf
- spark.driver.memory=512m
- --conf
- spark.eventLog.enabled=true
- --conf
- spark.eventLog.dir=/spark-events
- --class
- de.nowchess.analytics.PlayerGraphJob
- local:///app/analytics.jar
@@ -310,12 +279,7 @@ spec:
volumeMounts:
- name: spark-output
mountPath: /spark-output
- name: spark-events
mountPath: /spark-events
volumes:
- name: spark-output
persistentVolumeClaim:
claimName: spark-analytics-output
- name: spark-events
persistentVolumeClaim:
claimName: spark-events
spark-analytics/base/streaming.yaml
-9
View File
@@ -33,10 +33,6 @@ spec:
- spark.executor.memory=512m
- --conf
- spark.driver.memory=512m
- --conf
- spark.eventLog.enabled=true
- --conf
- spark.eventLog.dir=/spark-events
- --class
- de.nowchess.analytics.LiveDashboardJob
- local:///app/analytics.jar
@@ -67,12 +63,7 @@ spec:
volumeMounts:
- name: spark-checkpoint
mountPath: /spark-checkpoint
- name: spark-events
mountPath: /spark-events
volumes:
- name: spark-checkpoint
persistentVolumeClaim:
claimName: spark-streaming-checkpoint
- name: spark-events
persistentVolumeClaim:
claimName: spark-events
spark-analytics/eu-central-1-prod/kustomization.yaml
-20
View File
@@ -31,10 +31,6 @@ patches:
- spark.executor.memory=1g
- --conf
- spark.driver.memory=512m
- --conf
- spark.eventLog.enabled=true
- --conf
- spark.eventLog.dir=/spark-events
- --class
- de.nowchess.analytics.OpeningBookJob
- local:///app/analytics.jar
@@ -65,10 +61,6 @@ patches:
- spark.executor.memory=1g
- --conf
- spark.driver.memory=512m
- --conf
- spark.eventLog.enabled=true
- --conf
- spark.eventLog.dir=/spark-events
- --class
- de.nowchess.analytics.PlayerStatsJob
- local:///app/analytics.jar
@@ -98,10 +90,6 @@ patches:
- spark.executor.memory=1g
- --conf
- spark.driver.memory=512m
- --conf
- spark.eventLog.enabled=true
- --conf
- spark.eventLog.dir=/spark-events
- --class
- de.nowchess.analytics.PlayerClusteringJob
- local:///app/analytics.jar
@@ -132,10 +120,6 @@ patches:
- spark.executor.memory=1g
- --conf
- spark.driver.memory=512m
- --conf
- spark.eventLog.enabled=true
- --conf
- spark.eventLog.dir=/spark-events
- --class
- de.nowchess.analytics.PlayerGraphJob
- local:///app/analytics.jar
@@ -163,10 +147,6 @@ patches:
- spark.executor.memory=512m
- --conf
- spark.driver.memory=512m
- --conf
- spark.eventLog.enabled=true
- --conf
- spark.eventLog.dir=/spark-events
- --class
- de.nowchess.analytics.LiveDashboardJob
- local:///app/analytics.jar
spark-cluster/base/history-server.yaml
+4 -6
View File
@@ -18,10 +18,9 @@ spec:
spec:
containers:
- name: history-server
# Same image version the Bitnami chart deploys for master/workers.
image: docker.io/bitnami/spark:3.5.2
image: apache/spark:4.0.3-scala2.13-java17-ubuntu
command:
- /opt/bitnami/spark/bin/spark-class
- /opt/spark/bin/spark-class
- org.apache.spark.deploy.history.HistoryServer
env:
- name: SPARK_HISTORY_OPTS
@@ -32,7 +31,7 @@ spec:
resources:
requests:
cpu: 100m
memory: 512Mi
memory: 128Mi
limits:
cpu: 500m
memory: 1Gi
@@ -41,8 +40,7 @@ spec:
mountPath: /spark-events
volumes:
- name: spark-events
persistentVolumeClaim:
claimName: spark-events
emptyDir: {}
---
apiVersion: v1
kind: Service
spark-cluster/base/ingress.yaml
+4 -9
View File
@@ -6,7 +6,7 @@ metadata:
app.kubernetes.io/name: spark-cluster
app.kubernetes.io/part-of: nowchess
spec:
ingressClassName: traefik
ingressClassName: nginx
rules:
- host: spark.nowchess.janis-eccarius.de
http:
@@ -15,9 +15,9 @@ spec:
pathType: Prefix
backend:
service:
name: spark-master-svc
name: spark-master
port:
number: 80
number: 8080
---
apiVersion: networking.k8s.io/v1
kind: Ingress
@@ -26,13 +26,8 @@ metadata:
labels:
app.kubernetes.io/name: spark-cluster
app.kubernetes.io/part-of: nowchess
annotations:
# NOTE: The Bitnami spark chart 9.2.10 does not include a standalone history
# server deployment or service. This Ingress is a placeholder. To activate
# history server access, deploy a separate spark-history-server Deployment
# and Service named spark-history-server-svc on port 18080.
spec:
ingressClassName: traefik
ingressClassName: nginx
rules:
- host: spark-history.nowchess.janis-eccarius.de
http:
spark-cluster/base/kustomization.yaml
+2 -53
View File
@@ -3,58 +3,7 @@ kind: Kustomization
resources:
- pvc.yaml
- master.yaml
- workers.yaml
- history-server.yaml
- ingress.yaml
helmCharts:
- name: spark
repo: https://charts.bitnami.com/bitnami
version: "9.2.10"
releaseName: spark
namespace: nowchess
valuesInline:
master:
resources:
requests:
cpu: 500m
memory: 1Gi
limits:
cpu: "1"
memory: 2Gi
extraVolumes:
- name: spark-events
persistentVolumeClaim:
claimName: spark-events
extraVolumeMounts:
- name: spark-events
mountPath: /spark-events
worker:
replicaCount: 2
resources:
requests:
cpu: 500m
memory: 1Gi
limits:
cpu: "1"
memory: 2Gi
extraVolumes:
- name: spark-events
persistentVolumeClaim:
claimName: spark-events
extraVolumeMounts:
- name: spark-events
mountPath: /spark-events
historyServer:
enabled: true
eventsDir: /spark-events
extraVolumes:
- name: spark-events
persistentVolumeClaim:
claimName: spark-events
extraVolumeMounts:
- name: spark-events
mountPath: /spark-events
ingress:
enabled: false
service:
type: ClusterIP
spark-cluster/base/master.yaml
Executable
+64
View File
@@ -0,0 +1,64 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: spark-master
labels:
app.kubernetes.io/name: spark-master
app.kubernetes.io/part-of: nowchess
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: spark-master
template:
metadata:
labels:
app.kubernetes.io/name: spark-master
app.kubernetes.io/part-of: nowchess
spec:
# Prevent K8s from injecting SPARK_MASTER_PORT=tcp://... env vars (Docker-link
# style). Spark master parses SPARK_MASTER_PORT as an integer and crashes.
enableServiceLinks: false
containers:
- name: spark-master
image: apache/spark:4.0.3-scala2.13-java17-ubuntu
command:
- /opt/spark/bin/spark-class
- org.apache.spark.deploy.master.Master
ports:
- name: cluster
containerPort: 7077
- name: webui
containerPort: 8080
resources:
requests:
cpu: 200m
memory: 256Mi
limits:
cpu: "1"
memory: 2Gi
volumeMounts:
- name: spark-events
mountPath: /spark-events
volumes:
- name: spark-events
emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
name: spark-master
labels:
app.kubernetes.io/name: spark-master
app.kubernetes.io/part-of: nowchess
spec:
selector:
app.kubernetes.io/name: spark-master
ports:
- name: cluster
port: 7077
targetPort: cluster
- name: webui
port: 8080
targetPort: webui
type: ClusterIP
spark-cluster/base/pvc.yaml
+5 -18
View File
@@ -1,18 +1,5 @@
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: spark-events
labels:
app.kubernetes.io/name: spark-cluster
app.kubernetes.io/part-of: nowchess
spec:
# ReadWriteOnce for k3d/local-path compatibility.
# NOTE: k3d uses the local-path provisioner which only supports RWO.
# For production with multiple concurrent writers (workers + history server),
# replace with a RWX-capable storage class such as NFS or Longhorn and
# change accessModes to [ReadWriteMany].
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
# No cluster-level PVCs needed.
# Analytics output and streaming checkpoint PVCs live in spark-analytics/base/pvc.yaml.
# spark-events PVC removed: local-path provisioner is RWO-only; sharing a single PVC
# across master, workers, and history-server on different nodes causes Multi-Attach errors.
# Event logs now use emptyDir (pod-local, ephemeral) on each pod.
spark-cluster/base/workers.yaml
Executable
+46
View File
@@ -0,0 +1,46 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: spark-worker
labels:
app.kubernetes.io/name: spark-worker
app.kubernetes.io/part-of: nowchess
spec:
replicas: 2
selector:
matchLabels:
app.kubernetes.io/name: spark-worker
template:
metadata:
labels:
app.kubernetes.io/name: spark-worker
app.kubernetes.io/part-of: nowchess
spec:
containers:
- name: spark-worker
image: apache/spark:4.0.3-scala2.13-java17-ubuntu
command:
- /opt/spark/bin/spark-class
- org.apache.spark.deploy.worker.Worker
- spark://spark-master:7077
env:
- name: SPARK_WORKER_CORES
value: "1"
- name: SPARK_WORKER_MEMORY
value: "512m"
ports:
- name: webui
containerPort: 8081
resources:
requests:
cpu: 200m
memory: 256Mi
limits:
cpu: "1"
memory: 2Gi
volumeMounts:
- name: spark-events
mountPath: /spark-events
volumes:
- name: spark-events
emptyDir: {}