GitOps/DEPLOYMENT_GUIDE.md

Deployment Guide

Detailed step-by-step instructions for deploying the complete GitOps infrastructure.

Pre-Deployment Checklist

Environment Verification

# Check Kubernetes version (1.24+)
kubectl version --short

# Verify node capacity
kubectl describe nodes | grep -E "Name:|cpu:|memory:" | head -15

# Check available storage
kubectl get storageclass

# Verify ingress controller (if using)
kubectl get ingressclass

# Check for existing installations
kubectl get ns | grep -E "argocd|cert-manager|kargo"

Prerequisites Installation

1. Install kubectl (if not present)

Linux/macOS:

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
chmod +x kubectl
sudo mv kubectl /usr/local/bin/

Windows (PowerShell):

curl.exe -LO "https://dl.k8s.io/release/v1.28.0/bin/windows/amd64/kubectl.exe"

2. Install Helm

Linux/macOS:

curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

Windows (Chocolatey):

choco install kubernetes-helm

3. Install Kustomize

Linux/macOS:

curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh" | bash
sudo mv kustomize /usr/local/bin/

4. Install Kustomize Helm Plugin

# Create plugin directory
mkdir -p ~/.config/kustomize/plugin/kustomize.config.k8s.io/v1/helmchart

# Download plugin (this is typically handled by 'kustomize build --enable-helm')
# The flag --enable-helm automatically enables the plugin

5. Configure kubectl Context

# List available contexts
kubectl config get-contexts

# Switch to correct context
kubectl config use-context <context-name>

# Verify connection
kubectl cluster-info
kubectl auth can-i get pods --all-namespaces

Automated Deployment

Using the Provided Script

The quickest way to deploy is using the included deployment script:

cd scripts
chmod +x deploy-to-cluster.sh
./deploy-to-cluster.sh

What the script does:

1. Installs Cert-Manager with CRDs
2. Installs Argo CD with full configuration
3. Sets up sealed secrets configuration
4. Deploys the root Argo CD application
5. Displays final access credentials

Output Example:

----------------------------------------
 🎉 Kubernetes local cluster setup complete!
 🎉 Access ArgoCD at: https://localhost:31443
 🎉 Default login: admin / <generated-password>
----------------------------------------

Script Troubleshooting

If the script fails:

# Check for error messages
tail -f /tmp/deploy-to-cluster.log  # If logging is redirected

# Manually check what failed
kubectl get pods -A
kubectl describe pod -n <failed-namespace> <pod-name>

# Resume from specific step (modify script as needed)

Manual Step-by-Step Deployment

Use this process if you need more control or the automated script fails.

Step 1: Create Required Namespaces

# Create namespaces
kubectl create namespace argocd
kubectl create namespace cert-manager
kubectl create namespace kargo

# Label namespaces
kubectl label namespace cert-manager cert-manager.io/disable-validation=false

# Verify
kubectl get namespaces | grep -E "argocd|cert-manager|kargo"

Step 2: Install Cert-Manager

Cert-Manager must be installed before Argo CD because it provides certificate management.

# Add Helm repository
helm repo add jetstack https://charts.jetstack.io
helm repo update

# Install Cert-Manager CRDs first
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.0/cert-manager.crds.yaml

# Wait for CRDs to be available
kubectl wait --for condition=established --timeout=60s crd/certificates.cert-manager.io

# Build and apply cert-manager with kustomize
echo "Installing Cert-Manager..."
kustomize build --enable-helm cert-manager/eu-central-1 | kubectl apply -f -

# Wait for cert-manager to be ready
echo "Waiting for cert-manager to be ready..."
kubectl wait --for=condition=ready pod \
  -l app.kubernetes.io/name=cert-manager \
  -n cert-manager --timeout=300s

# Verify installation
kubectl -n cert-manager get pods
kubectl get crd | grep cert-manager

Step 3: Install Argo CD

echo "Installing Argo CD..."
kustomize build --enable-helm argocd/eu-central-1 | kubectl apply -f -

# Wait for Argo CD components
echo "Waiting for Argo CD to be ready..."
kubectl wait --for=condition=ready pod \
  -l app.kubernetes.io/name=argocd-server \
  -n argocd --timeout=300s

kubectl wait --for=condition=ready pod \
  -l app.kubernetes.io/name=argocd-application-controller \
  -n argocd --timeout=300s

# Verify installation
kubectl -n argocd get pods

Step 4: Get Argo CD Credentials

# Retrieve initial admin password
ARGO_PASSWORD=$(kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath="{.data.password}" | base64 --decode)

echo "Argo CD Admin Password: $ARGO_PASSWORD"
echo "Store this securely and change it after first login"

Step 5: Access Argo CD UI

Option A: Port Forward (Development)

# Forward local port to Argo CD server
kubectl port-forward -n argocd svc/argocd-server 8080:443

# Access at: https://localhost:8080
# Login: admin / <password from step 4>

Option B: Using NodePort (Already configured)

# Get the NodePort
kubectl -n argocd get svc argocd-server -o jsonpath='{.spec.ports[0].nodePort}'

# Access at: https://<node-ip>:<nodeport>
# Example: https://192.168.1.100:31443

Option C: Ingress (Production)

# Check if Ingress is created
kubectl -n argocd get ingress

# Get ingress hostname
kubectl -n argocd get ingress -o jsonpath='{.items[0].status.loadBalancer.ingress[0].hostname}'

# Access via configured domain
# Example: https://argo.knockout.janis-eccarius.de

Step 6: Configure Git Repository

Log into Argo CD UI and configure the Git repository:

1. Navigate to: Settings → Repositories
2. Click: "Connect Repo" → "VIA SSH"
3. Fill in details:
   • Repository URL: git@git.janis-eccarius.de:NowChess/GitOps.git
   • SSH private key: (Paste your SSH key or upload)
   • Known hosts: (Provide or auto-generate)
4. Test connection: Click "Test"
5. Save: Click "Save" and "Connect"

SSH Key Setup:

# Generate SSH key if needed
ssh-keygen -t ed25519 -C "argocd" -f ~/.ssh/argocd_key -N ""

# Add to SSH agent
ssh-add ~/.ssh/argocd_key

# Copy public key to Git provider
cat ~/.ssh/argocd_key.pub | xclip -selection clipboard  # Linux
cat ~/.ssh/argocd_key.pub | pbcopy  # macOS

Step 7: Deploy Root Application

# Apply the root Argo CD application
kubectl apply -f eu-central-1/root-apps-app.yaml

# Verify the application was created
kubectl -n argocd get application orchestration-root-app-eu-central-1

# Watch sync progress
watch "kubectl -n argocd describe application orchestration-root-app-eu-central-1"

# Or use argocd CLI
argocd app list
argocd app watch orchestration-root-app-eu-central-1

Step 8: Install Kargo (Optional)

If you want to enable progressive delivery with Kargo:

# Build and apply Kargo configuration
echo "Installing Kargo..."
kustomize build --enable-helm kargo/eu-central-1 | kubectl apply -f -

# Wait for Kargo to be ready
kubectl wait --for=condition=ready pod \
  -l app.kubernetes.io/name=kargo \
  -n kargo --timeout=300s

# Get Kargo admin password
kubectl -n kargo get secret kargo-admin-password \
  -o jsonpath="{.data.password}" | base64 --decode

# Access Kargo API
kubectl port-forward -n kargo svc/kargo-api 8443:443
# Access at: https://localhost:8443

Step 9: Install Argo Rollouts (Optional)

For progressive deployment strategies:

echo "Installing Argo Rollouts..."
kustomize build --enable-helm argo-rollouts/eu-central-1 | kubectl apply -f -

# Verify installation
kubectl -n argo-rollouts get pods

Post-Deployment Configuration

1. Update Argo CD Admin Password

# Generate bcrypt hash
PASSWORD="mynewpassword"
HASH=$(htpasswd -nbBC 10 admin "$PASSWORD" | tr -d ':\n' | sed 's/$2y/$2a/')

# Update secret
kubectl -n argocd patch secret argocd-secret \
  --type merge \
  -p "{\"data\":{\"admin.password\":\"$(echo -n $HASH | base64 -w0)\"}}"

# Logout and re-login with new password

2. Configure OIDC (Optional but Recommended)

Create/update argocd/eu-central-1/values.yaml:

configs:
  cm:
    oidc.config: |
      name: Azure AD
      issuer: https://login.microsoftonline.com/<tenant-id>/v2.0
      clientID: <your-client-id>
      clientSecret: $oidc.azuread.clientSecret
      requestedScopes:
        - openid
        - profile
        - email
        - groups

3. Set Up Monitoring/Alerts

Enable ServiceMonitors for Prometheus:

# In argocd/eu-central-1/values.yaml
controller:
  metrics:
    serviceMonitor:
      enabled: true
      interval: 30s
      
server:
  metrics:
    serviceMonitor:
      enabled: true

4. Configure Backup Strategy

# Install velero for backup
helm repo add vmware-tanzu https://vmware-tanzu.github.io/helm-charts
helm install velero vmware-tanzu/velero \
  --namespace velero \
  --create-namespace \
  --set configuration.backupStorageLocation.bucket=<bucket-name> \
  --set configuration.backupStorageLocation.provider=aws

5. Enable Network Policies (Production)

# Create network policy template
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: argocd-allow-ingress
  namespace: argocd
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: argocd-server
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          name: ingress-nginx
    ports:
    - protocol: TCP
      port: 8080

Verification Checklist

After deployment, verify all components:

# ✓ All pods running
kubectl get pods -A | grep -E "argocd|cert-manager|kargo"

# ✓ Persistent volumes mounted
kubectl get pvc -A

# ✓ Services accessible
kubectl get svc -A | grep -E "argocd|cert-manager|kargo"

# ✓ Argo CD application synced
argocd app list

# ✓ Certificates valid
kubectl get certificate -A
kubectl describe certificate -n <namespace> <cert-name>

# ✓ No pod errors
kubectl get events -A --sort-by='.lastTimestamp' | tail -20

# ✓ DNS resolution working
kubectl run -it --rm debug --image=alpine --restart=Never -- \
  nslookup argocd-server.argocd.svc.cluster.local

Deployment Validation Script

Create a validation script:

#!/bin/bash
# validate-deployment.sh

set -e

echo "Validating GitOps Deployment..."

# Check Cert-Manager
echo "✓ Checking Cert-Manager..."
kubectl -n cert-manager get pods -q | grep Running &>/dev/null || \
  (echo "✗ Cert-Manager pods not running" && exit 1)

# Check Argo CD
echo "✓ Checking Argo CD..."
kubectl -n argocd get pods -q | grep Running &>/dev/null || \
  (echo "✗ Argo CD pods not running" && exit 1)

# Check root application
echo "✓ Checking root application..."
ROOT_APP=$(kubectl -n argocd get application -o jsonpath='{.items[0].metadata.name}')
STATUS=$(kubectl -n argocd get application $ROOT_APP -o jsonpath='{.status.operationState.phase}')
echo "  Root app: $ROOT_APP (Status: $STATUS)"

# Check certificates
echo "✓ Checking certificates..."
CERTS=$(kubectl get certificate -A -o jsonpath='{.items[*].metadata.name}')
echo "  Certificates found: ${#CERTS[@]}"

echo ""
echo "✅ All validation checks passed!"

Troubleshooting Deployment

Cert-Manager CRD Issues

# If cert-manager CRDs fail to install:
kubectl get crd | grep cert-manager

# Delete and reinstall
kubectl delete crd certificates.cert-manager.io
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.0/cert-manager.crds.yaml

Pod Crash Loop

# Check pod logs
kubectl -n <namespace> logs <pod-name>

# Check pod status
kubectl -n <namespace> describe pod <pod-name>

# Check resource availability
kubectl describe nodes

Git Repository Connection Issues

# Test Git connectivity from pod
kubectl -n argocd exec -it <argocd-repo-server-pod> -- bash

# Inside pod:
ssh -v git@git.janis-eccarius.de
git clone git@git.janis-eccarius.de:NowChess/GitOps.git

Certificate Issues

# Check cert-manager controller logs
kubectl -n cert-manager logs -l app.kubernetes.io/name=cert-manager

# Check certificate status
kubectl get certificate -A -o wide
kubectl describe certificate <cert-name> -n <namespace>

# Manually trigger renewal
kubectl annotate certificate <cert-name> -n <namespace> \
  cert-manager.io/issue-temporary-certificate="true" --overwrite

Uninstall (if needed)

# Remove applications first (preserves data)
kubectl delete -f eu-central-1/root-apps-app.yaml

# Remove Kargo
helm uninstall kargo -n kargo
kubectl delete ns kargo

# Remove Argo CD
helm uninstall argocd -n argocd
kubectl delete ns argocd

# Remove Cert-Manager
helm uninstall cert-manager -n cert-manager
kubectl delete ns cert-manager
kubectl delete crd certificates.cert-manager.io

---

Last Updated: 2026-04-16 Version: 1.0