GitOps for the KnockOut-Whist Project

This repository contains the GitOps configuration for deploying and managing the KnockOut-Whist project using modern DevOps tools and practices. It leverages GitOps principles to ensure that infrastructure and application deployments are version-controlled, auditable, and automated.

Table of Contents

• Overview
• Architecture
• Prerequisites
• Quick Start
• Directory Structure
• Components
• Installation Guide
• Configuration
• Secrets Management
• Troubleshooting
• Contributing

Overview

This GitOps repository implements a complete infrastructure-as-code solution for Kubernetes deployments in the EU Central 1 region. It provides:

• Declarative Infrastructure: All configurations are defined in Git
• Automated Deployments: Argo CD automatically syncs cluster state with Git
• Progressive Delivery: Kargo manages safe, progressive deployments across environments
• Certificate Management: Automated certificate provisioning via cert-manager
• Secrets Management: Encrypted secret storage with sealed-secrets pattern
• Monitoring & Logging: Integration points for observability tools

Architecture

┌─────────────────────────────────────────────────────────────┐
│                    Kubernetes Cluster                       │
│                                                             │
│  ┌──────────────────────────────────────────────────────┐  │
│  │                   ArgoCD (Namespace)                 │  │
│  │  - Application Controller                           │  │
│  │  - Server (UI & API)                                │  │
│  │  - Repo Server                                      │  │
│  │  - Redis (Session Store)                            │  │
│  └──────────────────────────────────────────────────────┘  │
│                          ▼                                   │
│  ┌──────────────────────────────────────────────────────┐  │
│  │              Cert-Manager (Namespace)                │  │
│  │  - Webhook                                          │  │
│  │  - Controller                                       │  │
│  │  - CA Issuer                                        │  │
│  └──────────────────────────────────────────────────────┘  │
│                          ▼                                   │
│  ┌──────────────────────────────────────────────────────┐  │
│  │                Kargo (Namespace)                     │  │
│  │  - API Server                                       │  │
│  │  - Controller                                       │  │
│  │  - Webhooks Server                                  │  │
│  │  - Management Controller                            │  │
│  │  - External Webhooks Server                         │  │
│  └──────────────────────────────────────────────────────┘  │
│                                                             │
│  ┌──────────────────────────────────────────────────────┐  │
│  │          Kargo Projects (Orchestration)              │  │
│  │  - Promotion Pipelines                              │  │
│  │  - Warehouse                                        │  │
│  │  - Stages                                           │  │
│  └──────────────────────────────────────────────────────┘  │
│                                                             │
│  ┌──────────────────────────────────────────────────────┐  │
│  │         Argo Rollouts (Optional)                     │  │
│  │  - Canary Deployments                               │  │
│  │  - Progressive Rollout Controller                   │  │
│  └──────────────────────────────────────────────────────┘  │
│                                                             │
└─────────────────────────────────────────────────────────────┘
           │
           ▼
    ┌─────────────────┐
    │   Git Repo      │
    │ (This Repo)     │
    └─────────────────┘

Prerequisites

System Requirements

• Kubernetes: Version 1.24+ (tested with 1.26+)
• kubectl: Latest stable version
• Helm: 3.10+
• Kustomize: 5.0+ (with Helm plugin enabled)
• Git: 2.40+

Cluster Requirements

• Minimum 2 CPU cores, 4GB RAM (for development)
• Production: 4+ CPU cores, 8GB+ RAM recommended
• Storage class configured for persistent volumes
• Ingress controller (optional, for external access)
• cert-manager CRDs installed before cert-manager deployment

Network Requirements

• Outbound access to container registries (quay.io, ghcr.io, ecr-public.aws.com)
• Git repository access (SSH key configured)
• DNS resolution for configured domains
• Port 443 (HTTPS) for API access
• Port 6379 (Redis) for internal Argo CD communication

Quick Start

1. Clone the Repository

git clone git@git.janis-eccarius.de:NowChess/GitOps.git
cd GitOps

2. Prepare Your Kubernetes Cluster

Ensure your kubectl context points to the target cluster:

kubectl cluster-info
kubectl get nodes

3. Automatic Installation

For a complete automated setup (recommended for initial deployment):

cd scripts
chmod +x deploy-to-cluster.sh
./deploy-to-cluster.sh

This script will:

• Install Cert-Manager and required CRDs
• Install and configure Argo CD
• Configure sealed secrets for GitOps
• Deploy the root Argo CD application
• Display access credentials

4. Manual Installation (Step-by-Step)

If you prefer manual control or need to customize the installation:

Step 1: Install Cert-Manager

kustomize build --enable-helm cert-manager/eu-central-1 | kubectl apply -f -

# Wait for cert-manager to be ready
kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=cert-manager \
  -n cert-manager --timeout=300s

Step 2: Install Argo CD

kustomize build --enable-helm argocd/eu-central-1 | kubectl apply -f -

# Wait for Argo CD to be ready
kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=argocd-server \
  -n argocd --timeout=300s

Step 3: Get Initial Admin Password

kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath="{.data.password}" | base64 --decode

Step 4: Access Argo CD

Port Forward (Development):

kubectl port-forward -n argocd svc/argocd-server 8080:443
# Access at https://localhost:8080

Ingress (Production):

• If using an ingress controller, access via configured domain (e.g., argo.knockout.janis-eccarius.de)

Step 5: Deploy Root Application

kubectl apply -f eu-central-1/root-apps-app.yaml

# Monitor deployment
kubectl -n argocd get applications
argocd app list

Directory Structure

GitOps/
├── README.md                           # This file
├── scripts/
│   └── deploy-to-cluster.sh           # Automated deployment script
├── argocd/                            # Argo CD configuration
│   ├── base/                          # Base kustomization
│   │   ├── cert-manager-namespace.yaml
│   │   ├── kustomization.yaml
│   │   └── values.yaml
│   └── eu-central-1/                  # Regional overrides
│       ├── kube-devops.yaml
│       ├── kustomization.yaml
│       └── values.yaml
├── cert-manager/                      # Cert-Manager configuration
│   ├── base/
│   │   ├── cert-manager-namespace.yaml
│   │   ├── kustomization.yaml
│   │   └── values.yaml
│   └── eu-central-1/
│       ├── cert-issuer.yaml
│       ├── kustomization.yaml
├── kargo/                             # Kargo progressive delivery
│   ├── base/
│   │   ├── kustomization.yaml
│   │   └── values.yaml
│   └── eu-central-1/
│       ├── kustomization.yaml
│       └── values.yaml
├── kargo-projects/                    # Kargo project definitions
│   └── orchestration-stack/
│       ├── kustomization.yaml
│       ├── orch-project.yaml
│       ├── orch-projectconfig.yaml
│       ├── orch-promotion-template.yaml
│       ├── orch-stage.yaml
│       └── orch-warehouse.yaml
├── argo-rollouts/                     # Argo Rollouts configuration
│   └── eu-central-1/
│       ├── kube-devops.yaml
│       └── kustomization.yaml
├── eu-central-1/                      # Regional deployment root
│   ├── root-apps-app.yaml             # Root Argo CD application
│   └── argo-apps/                     # All deployed applications
│       ├── argo-rollouts/
│       ├── cert-manager/
│       ├── kargo/
│       └── kargo-projects/
├── secrets/                           # Encrypted secrets
│   ├── kustomization.yaml
│   ├── gitea/
│   ├── github/
│   └── kargo/
└── Passwords.kdbx                     # Password manager file (DO NOT COMMIT)

Components

Argo CD

Argo CD is a declarative, GitOps continuous delivery tool for Kubernetes.

Configuration Files:

• argocd/eu-central-1/values.yaml - Main Helm values

Key Features Enabled:

• Helm support with kustomize.buildOptions: --enable-helm
• Automated pruning and self-healing
• Kustomize integration
• Redis caching
• Status badge support

Access:

# Via port-forward
kubectl port-forward -n argocd svc/argocd-server 8080:443

# Get initial password
kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath="{.data.password}" | base64 --decode

Next Steps:

1. Change default admin password in Argo CD UI
2. Configure Git repository credentials
3. Set up OIDC/SSO for authentication
4. Create additional Argo CD projects for RBAC

Cert-Manager

Cert-Manager automates certificate management in Kubernetes using cert-manager and Let's Encrypt.

Configuration Files:

• cert-manager/base/values.yaml - Base Helm configuration
• cert-manager/eu-central-1/cert-issuer.yaml - Certificate issuer configuration

Certificate Issuers:

• Self-signed CA issuer for internal certificates
• Integration with Let's Encrypt for public certificates (can be configured)

Usage:

# View certificate issuers
kubectl get issuers -n cert-manager

# Monitor certificate requests
kubectl get certificaterequest -A

Kargo

Kargo is a progressive delivery platform that automates and secures multi-stage promotion of Freight across a series of Stages.

Configuration Files:

• kargo/base/values.yaml - Core Kargo configuration
• kargo/eu-central-1/values.yaml - Regional overrides
• kargo-projects/orchestration-stack/ - Project definitions

Key Features:

• Multi-stage promotion pipelines
• Integration with Argo CD and Argo Rollouts
• OIDC authentication support
• External webhook integrations
• Garbage collection for old promotions and freight

Access:

# Port forward to Kargo API
kubectl port-forward -n kargo svc/kargo-api 8443:443

# Get admin credentials
kubectl -n kargo get secret kargo-admin-password -o jsonpath="{.data.password}"

Argo Rollouts

Argo Rollouts provides advanced deployment strategies (Canary, Blue-Green, etc.).

Configuration Files:

• argo-rollouts/eu-central-1/kube-devops.yaml - Deployment spec
• argo-rollouts/eu-central-1/kustomization.yaml - Kustomization

Features:

• Canary deployments
• Blue-green deployments
• Progressive rollouts with automated analysis
• Integration with analysis tools

Installation Guide

Detailed Installation Steps

Prerequisites Verification

# Verify Kubernetes version
kubectl version --short

# Check node capacity
kubectl top nodes

# Verify storage class exists
kubectl get storageclass

# Check for existing cert-manager
kubectl get crd | grep cert-manager

Step 1: Namespace Setup

# Create namespaces
kubectl create namespace argocd
kubectl create namespace cert-manager
kubectl create namespace kargo

# Label namespaces for cert-manager
kubectl label namespace cert-manager cert-manager.io/inject-enabled=true

Step 2: Install Cert-Manager

# Add cert-manager CRDs first
kustomize build cert-manager/base | kubectl apply -f -

# Wait for CRDs
sleep 5

# Install cert-manager
kustomize build --enable-helm cert-manager/eu-central-1 | kubectl apply -f -

# Verify installation
kubectl -n cert-manager get pods
kubectl get crd | grep certmanager

Step 3: Install Argo CD

kustomize build --enable-helm argocd/eu-central-1 | kubectl apply -f -

# Wait for readiness
kubectl -n argocd wait --for=condition=ready pod \
  -l app.kubernetes.io/name=argocd-server --timeout=300s

Step 4: Configure Secrets

# Create Git repository secret
kubectl -n argocd create secret generic repo-credentials \
  --from-file=sshPrivateKey=$HOME/.ssh/id_rsa \
  --dry-run=client -o yaml | kubectl apply -f -

Step 5: Deploy Applications

# Apply root application
kubectl apply -f eu-central-1/root-apps-app.yaml

# Monitor sync status
watch "kubectl -n argocd get applications"

Post-Installation Configuration

1. Update Admin Credentials

# Generate new password hash
PASSWORD=$(echo 'mypassword' | htpasswd -nbBC 10 admin | tr -d ':\n' | sed 's/$2y/$2a/')

# Update secret
kubectl -n argocd patch secret argocd-secret -p \
  "{\"data\":{\"admin.password\":\"$(echo -n $PASSWORD | base64)\"}\
  \"admin.passwordMtime\":\"$(date -u +'%Y-%m-%dT%H:%M:%SZ' | base64)\"}"

2. Configure Git Repository

In Argo CD UI:

1. Settings → Repositories
2. Connect Repo → Via SSH
3. Enter repository URL: git@git.janis-eccarius.de:NowChess/GitOps.git
4. Upload SSH private key
5. Save and verify

3. Enable OIDC Authentication (Optional)

Update argocd/eu-central-1/values.yaml:

configs:
  cm:
    oidc.config: |
      name: <Your Provider>
      issuer: <https://provider/endpoint>
      clientID: <client-id>
      clientSecret: $oidc.clientSecret

Configuration

Regional Configuration

This repository is configured for EU Central 1 region. To add new regions:

1. Create new directory: eu-west-1/
2. Copy and adapt configuration from eu-central-1/
3. Update domain names and region-specific values
4. Create new root application for the region

Customizing Component Versions

Edit the Helm chart versions in respective kustomization.yaml files:

# Example: argocd/eu-central-1/kustomization.yaml
helmCharts:
- name: argo-cd
  repo: https://argoproj.github.io/argo-helm
  version: 5.x.x  # Update version here
  releaseName: argocd
  namespace: argocd

Resource Limits

Default resource requests/limits can be modified in values.yaml files:

# argocd/eu-central-1/values.yaml
controller:
  resources:
    requests:
      cpu: 250m
      memory: 256Mi
    limits:
      cpu: 500m
      memory: 512Mi

Secrets Management

Secret Storage Strategy

This repository uses the Sealed Secrets pattern:

1. Encrypted Storage: Secrets are encrypted in Git using sealing keys
2. Key Management: Sealing keys are stored securely outside Git
3. Decryption: Sealed secrets are automatically decrypted by the cluster

Creating New Secrets

# Install kubeseal (one-time)
wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.18.0/kubeseal-0.18.0-linux-amd64.tar.gz
tar xfz kubeseal-0.18.0-linux-amd64.tar.gz -C /usr/local/bin

# Create a new secret
kubectl -n <namespace> create secret generic my-secret \
  --from-literal=password=mysecret \
  --dry-run=client -o yaml > my-secret.yaml

# Seal it
kubeseal -f my-secret.yaml -w my-sealed-secret.yaml

# Commit sealed secret to Git
git add my-sealed-secret.yaml
git commit -m "Add sealed secret"
git push

Accessing Secret Credentials

Located in secrets/ directory:

• GitHub: secrets/github/ - GitHub access tokens
• Gitea: secrets/gitea/ - Gitea repository access
• Kargo: secrets/kargo/ - Kargo admin credentials

Never commit unencrypted secrets to Git.

Troubleshooting

Common Issues and Solutions

1. Pods Not Starting

# Check pod status and events
kubectl -n <namespace> describe pod <pod-name>

# View logs
kubectl -n <namespace> logs <pod-name>

2. Certificate Issues

# Check certificate status
kubectl get certificate -A
kubectl describe certificate -n <namespace> <cert-name>

# Check cert-manager logs
kubectl -n cert-manager logs -f deployment/cert-manager

3. Argo CD Sync Failures

# Get application status
kubectl -n argocd get application

# Detailed status
kubectl -n argocd describe application <app-name>

# Resync
argocd app sync <app-name>

4. Git Repository Connection Issues

# Check repository credentials
kubectl -n argocd get secret repo-credentials -o yaml

# Test repository access
kubectl -n argocd exec -it <argocd-repo-server-pod> -- bash
# Try: ssh -v git@git.janis-eccarius.de

Debugging Commands

# View all resources in cluster
kubectl get all -A

# Get cluster events
kubectl get events -A --sort-by='.lastTimestamp'

# Check resource quotas
kubectl describe resourcequota -A

# Monitor node status
kubectl describe nodes

Support Resources

• Argo CD Docs: https://argo-cd.readthedocs.io/
• Cert-Manager Docs: https://cert-manager.io/docs/
• Kargo Docs: https://kargo.akuity.io/
• Argo Rollouts Docs: https://argoproj.github.io/argo-rollouts/

Contributing

Making Changes

1. Create a feature branch:

   git checkout -b feature/my-feature
   

2. Make your changes and test locally:

   kustomize build <path> | kubectl apply --dry-run=client -f -
   

3. Commit with descriptive messages:

   git commit -m "feat: add new certificate issuer"
   

4. Push to repository:

   git push origin feature/my-feature
   

5. Create merge request for review

Best Practices

• Small, focused commits: Each commit should represent one logical change
• Test before committing: Use --dry-run to validate
• Document changes: Update this README for significant changes
• Use semantic versioning: Tag releases appropriately
• Follow naming conventions: Use descriptive names for branches, commits, and resources

Policy and Guidelines

• All changes must go through Git version control
• Never manually apply Kubernetes manifests to production
• Always validate with kustomize build before deployment
• Encrypt all secrets before committing
• Keep sealed-secrets keys secure (not in Git)
• Regular security audits of repository access

Additional Resources

Documentation Files

• DEPLOYMENT_GUIDE.md - Detailed deployment instructions
• ARCHITECTURE.md - System architecture overview
• TROUBLESHOOTING.md - Extended troubleshooting guide
• UPGRADE_GUIDE.md - Version upgrade procedures

Useful Commands Reference

# Kustomize build and apply
kustomize build . | kubectl apply -f -
kustomize build . | kubectl apply --dry-run=client -f -

# Watch applications
kubectl -n argocd get applications -w
argocd app list
argocd app watch <app-name>

# Get application details
argocd app get <app-name>
kubectl -n argocd describe application <app-name>

# Manual sync
argocd app sync <app-name>

# Access logs
kubectl -n argocd logs -f deployment/argocd-controller
kubectl -n cert-manager logs -f deployment/cert-manager

---

Last Updated: 2026-04-16 Maintained By: NowChess DevOps Team Repository: https://git.janis-eccarius.de/NowChess/GitOps