NowChess/GitOps
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: add comprehensive documentation and deployment scripts

Browse Source 
- Created new documentation files including TROUBLESHOOTING.md, CONFIGURATION.md, and CONTRIBUTING.md.
- Added deployment guide with automated and manual instructions.
- Introduced new YAML configurations for Kubernetes resources and Helm charts.
- Implemented a deployment script for setting up the GitOps infrastructure.
This commit is contained in:
Janis
2026-04-16 19:07:08 +02:00
commit 20d8146e8c
38 changed files with 10739 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+694
View File
@@ -0,0 +1,694 @@
# GitOps for the KnockOut-Whist Project
This repository contains the GitOps configuration for deploying and managing the KnockOut-Whist project using modern DevOps tools and practices. It leverages GitOps principles to ensure that infrastructure and application deployments are version-controlled, auditable, and automated.
## Table of Contents
- [Overview](#overview)
- [Architecture](#architecture)
- [Prerequisites](#prerequisites)
- [Quick Start](#quick-start)
- [Directory Structure](#directory-structure)
- [Components](#components)
- [Installation Guide](#installation-guide)
- [Configuration](#configuration)
- [Secrets Management](#secrets-management)
- [Troubleshooting](#troubleshooting)
- [Contributing](#contributing)
## Overview
This GitOps repository implements a complete infrastructure-as-code solution for Kubernetes deployments in the EU Central 1 region. It provides:
- **Declarative Infrastructure**: All configurations are defined in Git
- **Automated Deployments**: Argo CD automatically syncs cluster state with Git
- **Progressive Delivery**: Kargo manages safe, progressive deployments across environments
- **Certificate Management**: Automated certificate provisioning via cert-manager
- **Secrets Management**: Encrypted secret storage with sealed-secrets pattern
- **Monitoring & Logging**: Integration points for observability tools
## Architecture
```
┌─────────────────────────────────────────────────────────────┐
│ Kubernetes Cluster │
│ │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ ArgoCD (Namespace) │ │
│ │ - Application Controller │ │
│ │ - Server (UI & API) │ │
│ │ - Repo Server │ │
│ │ - Redis (Session Store) │ │
│ └──────────────────────────────────────────────────────┘ │
│ ▼ │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ Cert-Manager (Namespace) │ │
│ │ - Webhook │ │
│ │ - Controller │ │
│ │ - CA Issuer │ │
│ └──────────────────────────────────────────────────────┘ │
│ ▼ │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ Kargo (Namespace) │ │
│ │ - API Server │ │
│ │ - Controller │ │
│ │ - Webhooks Server │ │
│ │ - Management Controller │ │
│ │ - External Webhooks Server │ │
│ └──────────────────────────────────────────────────────┘ │
│ │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ Kargo Projects (Orchestration) │ │
│ │ - Promotion Pipelines │ │
│ │ - Warehouse │ │
│ │ - Stages │ │
│ └──────────────────────────────────────────────────────┘ │
│ │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ Argo Rollouts (Optional) │ │
│ │ - Canary Deployments │ │
│ │ - Progressive Rollout Controller │ │
│ └──────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────┐
│ Git Repo │
│ (This Repo) │
└─────────────────┘
```
## Prerequisites
### System Requirements
- **Kubernetes**: Version 1.24+ (tested with 1.26+)
- **kubectl**: Latest stable version
- **Helm**: 3.10+
- **Kustomize**: 5.0+ (with Helm plugin enabled)
- **Git**: 2.40+
### Cluster Requirements
- Minimum 2 CPU cores, 4GB RAM (for development)
- Production: 4+ CPU cores, 8GB+ RAM recommended
- Storage class configured for persistent volumes
- Ingress controller (optional, for external access)
- cert-manager CRDs installed before cert-manager deployment
### Network Requirements
- Outbound access to container registries (quay.io, ghcr.io, ecr-public.aws.com)
- Git repository access (SSH key configured)
- DNS resolution for configured domains
- Port 443 (HTTPS) for API access
- Port 6379 (Redis) for internal Argo CD communication
## Quick Start
### 1. Clone the Repository
```bash
git clone git@git.janis-eccarius.de:NowChess/GitOps.git
cd GitOps
```
### 2. Prepare Your Kubernetes Cluster
Ensure your kubectl context points to the target cluster:
```bash
kubectl cluster-info
kubectl get nodes
```
### 3. Automatic Installation
For a complete automated setup (recommended for initial deployment):
```bash
cd scripts
chmod +x deploy-to-cluster.sh
./deploy-to-cluster.sh
```
This script will:
- Install Cert-Manager and required CRDs
- Install and configure Argo CD
- Configure sealed secrets for GitOps
- Deploy the root Argo CD application
- Display access credentials
### 4. Manual Installation (Step-by-Step)
If you prefer manual control or need to customize the installation:
#### Step 1: Install Cert-Manager
```bash
kustomize build --enable-helm cert-manager/eu-central-1 | kubectl apply -f -
# Wait for cert-manager to be ready
kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=cert-manager \
-n cert-manager --timeout=300s
```
#### Step 2: Install Argo CD
```bash
kustomize build --enable-helm argocd/eu-central-1 | kubectl apply -f -
# Wait for Argo CD to be ready
kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=argocd-server \
-n argocd --timeout=300s
```
#### Step 3: Get Initial Admin Password
```bash
kubectl -n argocd get secret argocd-initial-admin-secret \
-o jsonpath="{.data.password}" | base64 --decode
```
#### Step 4: Access Argo CD
**Port Forward (Development)**:
```bash
kubectl port-forward -n argocd svc/argocd-server 8080:443
# Access at https://localhost:8080
```
**Ingress (Production)**:
- If using an ingress controller, access via configured domain (e.g., `argo.knockout.janis-eccarius.de`)
#### Step 5: Deploy Root Application
```bash
kubectl apply -f eu-central-1/root-apps-app.yaml
# Monitor deployment
kubectl -n argocd get applications
argocd app list
```
## Directory Structure
```
GitOps/
├── README.md # This file
├── scripts/
│ └── deploy-to-cluster.sh # Automated deployment script
├── argocd/ # Argo CD configuration
│ ├── base/ # Base kustomization
│ │ ├── cert-manager-namespace.yaml
│ │ ├── kustomization.yaml
│ │ └── values.yaml
│ └── eu-central-1/ # Regional overrides
│ ├── kube-devops.yaml
│ ├── kustomization.yaml
│ └── values.yaml
├── cert-manager/ # Cert-Manager configuration
│ ├── base/
│ │ ├── cert-manager-namespace.yaml
│ │ ├── kustomization.yaml
│ │ └── values.yaml
│ └── eu-central-1/
│ ├── cert-issuer.yaml
│ ├── kustomization.yaml
├── kargo/ # Kargo progressive delivery
│ ├── base/
│ │ ├── kustomization.yaml
│ │ └── values.yaml
│ └── eu-central-1/
│ ├── kustomization.yaml
│ └── values.yaml
├── kargo-projects/ # Kargo project definitions
│ └── orchestration-stack/
│ ├── kustomization.yaml
│ ├── orch-project.yaml
│ ├── orch-projectconfig.yaml
│ ├── orch-promotion-template.yaml
│ ├── orch-stage.yaml
│ └── orch-warehouse.yaml
├── argo-rollouts/ # Argo Rollouts configuration
│ └── eu-central-1/
│ ├── kube-devops.yaml
│ └── kustomization.yaml
├── eu-central-1/ # Regional deployment root
│ ├── root-apps-app.yaml # Root Argo CD application
│ └── argo-apps/ # All deployed applications
│ ├── argo-rollouts/
│ ├── cert-manager/
│ ├── kargo/
│ └── kargo-projects/
├── secrets/ # Encrypted secrets
│ ├── kustomization.yaml
│ ├── gitea/
│ ├── github/
│ └── kargo/
└── Passwords.kdbx # Password manager file (DO NOT COMMIT)
```
## Components
### Argo CD
Argo CD is a declarative, GitOps continuous delivery tool for Kubernetes.
**Configuration Files**:
- `argocd/eu-central-1/values.yaml` - Main Helm values
**Key Features Enabled**:
- Helm support with `kustomize.buildOptions: --enable-helm`
- Automated pruning and self-healing
- Kustomize integration
- Redis caching
- Status badge support
**Access**:
```bash
# Via port-forward
kubectl port-forward -n argocd svc/argocd-server 8080:443
# Get initial password
kubectl -n argocd get secret argocd-initial-admin-secret \
-o jsonpath="{.data.password}" | base64 --decode
```
**Next Steps**:
1. Change default admin password in Argo CD UI
2. Configure Git repository credentials
3. Set up OIDC/SSO for authentication
4. Create additional Argo CD projects for RBAC
### Cert-Manager
Cert-Manager automates certificate management in Kubernetes using cert-manager and Let's Encrypt.
**Configuration Files**:
- `cert-manager/base/values.yaml` - Base Helm configuration
- `cert-manager/eu-central-1/cert-issuer.yaml` - Certificate issuer configuration
**Certificate Issuers**:
- Self-signed CA issuer for internal certificates
- Integration with Let's Encrypt for public certificates (can be configured)
**Usage**:
```bash
# View certificate issuers
kubectl get issuers -n cert-manager
# Monitor certificate requests
kubectl get certificaterequest -A
```
### Kargo
Kargo is a progressive delivery platform that automates and secures multi-stage promotion of Freight across a series of Stages.
**Configuration Files**:
- `kargo/base/values.yaml` - Core Kargo configuration
- `kargo/eu-central-1/values.yaml` - Regional overrides
- `kargo-projects/orchestration-stack/` - Project definitions
**Key Features**:
- Multi-stage promotion pipelines
- Integration with Argo CD and Argo Rollouts
- OIDC authentication support
- External webhook integrations
- Garbage collection for old promotions and freight
**Access**:
```bash
# Port forward to Kargo API
kubectl port-forward -n kargo svc/kargo-api 8443:443
# Get admin credentials
kubectl -n kargo get secret kargo-admin-password -o jsonpath="{.data.password}"
```
### Argo Rollouts
Argo Rollouts provides advanced deployment strategies (Canary, Blue-Green, etc.).
**Configuration Files**:
- `argo-rollouts/eu-central-1/kube-devops.yaml` - Deployment spec
- `argo-rollouts/eu-central-1/kustomization.yaml` - Kustomization
**Features**:
- Canary deployments
- Blue-green deployments
- Progressive rollouts with automated analysis
- Integration with analysis tools
## Installation Guide
### Detailed Installation Steps
#### Prerequisites Verification
```bash
# Verify Kubernetes version
kubectl version --short
# Check node capacity
kubectl top nodes
# Verify storage class exists
kubectl get storageclass
# Check for existing cert-manager
kubectl get crd | grep cert-manager
```
#### Step 1: Namespace Setup
```bash
# Create namespaces
kubectl create namespace argocd
kubectl create namespace cert-manager
kubectl create namespace kargo
# Label namespaces for cert-manager
kubectl label namespace cert-manager cert-manager.io/inject-enabled=true
```
#### Step 2: Install Cert-Manager
```bash
# Add cert-manager CRDs first
kustomize build cert-manager/base | kubectl apply -f -
# Wait for CRDs
sleep 5
# Install cert-manager
kustomize build --enable-helm cert-manager/eu-central-1 | kubectl apply -f -
# Verify installation
kubectl -n cert-manager get pods
kubectl get crd | grep certmanager
```
#### Step 3: Install Argo CD
```bash
kustomize build --enable-helm argocd/eu-central-1 | kubectl apply -f -
# Wait for readiness
kubectl -n argocd wait --for=condition=ready pod \
-l app.kubernetes.io/name=argocd-server --timeout=300s
```
#### Step 4: Configure Secrets
```bash
# Create Git repository secret
kubectl -n argocd create secret generic repo-credentials \
--from-file=sshPrivateKey=$HOME/.ssh/id_rsa \
--dry-run=client -o yaml | kubectl apply -f -
```
#### Step 5: Deploy Applications
```bash
# Apply root application
kubectl apply -f eu-central-1/root-apps-app.yaml
# Monitor sync status
watch "kubectl -n argocd get applications"
```
### Post-Installation Configuration
#### 1. Update Admin Credentials
```bash
# Generate new password hash
PASSWORD=$(echo 'mypassword' | htpasswd -nbBC 10 admin | tr -d ':\n' | sed 's/$2y/$2a/')
# Update secret
kubectl -n argocd patch secret argocd-secret -p \
"{\"data\":{\"admin.password\":\"$(echo -n $PASSWORD | base64)\"}\
\"admin.passwordMtime\":\"$(date -u +'%Y-%m-%dT%H:%M:%SZ' | base64)\"}"
```
#### 2. Configure Git Repository
In Argo CD UI:
1. Settings → Repositories
2. Connect Repo → Via SSH
3. Enter repository URL: `git@git.janis-eccarius.de:NowChess/GitOps.git`
4. Upload SSH private key
5. Save and verify
#### 3. Enable OIDC Authentication (Optional)
Update `argocd/eu-central-1/values.yaml`:
```yaml
configs:
cm:
oidc.config: |
name: <Your Provider>
issuer: <https://provider/endpoint>
clientID: <client-id>
clientSecret: $oidc.clientSecret
```
## Configuration
### Regional Configuration
This repository is configured for **EU Central 1** region. To add new regions:
1. Create new directory: `eu-west-1/`
2. Copy and adapt configuration from `eu-central-1/`
3. Update domain names and region-specific values
4. Create new root application for the region
### Customizing Component Versions
Edit the Helm chart versions in respective `kustomization.yaml` files:
```yaml
# Example: argocd/eu-central-1/kustomization.yaml
helmCharts:
- name: argo-cd
repo: https://argoproj.github.io/argo-helm
version: 5.x.x # Update version here
releaseName: argocd
namespace: argocd
```
### Resource Limits
Default resource requests/limits can be modified in `values.yaml` files:
```yaml
# argocd/eu-central-1/values.yaml
controller:
resources:
requests:
cpu: 250m
memory: 256Mi
limits:
cpu: 500m
memory: 512Mi
```
## Secrets Management
### Secret Storage Strategy
This repository uses the **Sealed Secrets** pattern:
1. **Encrypted Storage**: Secrets are encrypted in Git using sealing keys
2. **Key Management**: Sealing keys are stored securely outside Git
3. **Decryption**: Sealed secrets are automatically decrypted by the cluster
### Creating New Secrets
```bash
# Install kubeseal (one-time)
wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.18.0/kubeseal-0.18.0-linux-amd64.tar.gz
tar xfz kubeseal-0.18.0-linux-amd64.tar.gz -C /usr/local/bin
# Create a new secret
kubectl -n <namespace> create secret generic my-secret \
--from-literal=password=mysecret \
--dry-run=client -o yaml > my-secret.yaml
# Seal it
kubeseal -f my-secret.yaml -w my-sealed-secret.yaml
# Commit sealed secret to Git
git add my-sealed-secret.yaml
git commit -m "Add sealed secret"
git push
```
### Accessing Secret Credentials
Located in `secrets/` directory:
- **GitHub**: `secrets/github/` - GitHub access tokens
- **Gitea**: `secrets/gitea/` - Gitea repository access
- **Kargo**: `secrets/kargo/` - Kargo admin credentials
**Never commit unencrypted secrets to Git.**
## Troubleshooting
### Common Issues and Solutions
#### 1. Pods Not Starting
```bash
# Check pod status and events
kubectl -n <namespace> describe pod <pod-name>
# View logs
kubectl -n <namespace> logs <pod-name>
```
#### 2. Certificate Issues
```bash
# Check certificate status
kubectl get certificate -A
kubectl describe certificate -n <namespace> <cert-name>
# Check cert-manager logs
kubectl -n cert-manager logs -f deployment/cert-manager
```
#### 3. Argo CD Sync Failures
```bash
# Get application status
kubectl -n argocd get application
# Detailed status
kubectl -n argocd describe application <app-name>
# Resync
argocd app sync <app-name>
```
#### 4. Git Repository Connection Issues
```bash
# Check repository credentials
kubectl -n argocd get secret repo-credentials -o yaml
# Test repository access
kubectl -n argocd exec -it <argocd-repo-server-pod> -- bash
# Try: ssh -v git@git.janis-eccarius.de
```
### Debugging Commands
```bash
# View all resources in cluster
kubectl get all -A
# Get cluster events
kubectl get events -A --sort-by='.lastTimestamp'
# Check resource quotas
kubectl describe resourcequota -A
# Monitor node status
kubectl describe nodes
```
### Support Resources
- **Argo CD Docs**: https://argo-cd.readthedocs.io/
- **Cert-Manager Docs**: https://cert-manager.io/docs/
- **Kargo Docs**: https://kargo.akuity.io/
- **Argo Rollouts Docs**: https://argoproj.github.io/argo-rollouts/
## Contributing
### Making Changes
1. Create a feature branch:
```bash
git checkout -b feature/my-feature
```
2. Make your changes and test locally:
```bash
kustomize build <path> | kubectl apply --dry-run=client -f -
```
3. Commit with descriptive messages:
```bash
git commit -m "feat: add new certificate issuer"
```
4. Push to repository:
```bash
git push origin feature/my-feature
```
5. Create merge request for review
### Best Practices
- **Small, focused commits**: Each commit should represent one logical change
- **Test before committing**: Use `--dry-run` to validate
- **Document changes**: Update this README for significant changes
- **Use semantic versioning**: Tag releases appropriately
- **Follow naming conventions**: Use descriptive names for branches, commits, and resources
### Policy and Guidelines
- All changes must go through Git version control
- Never manually apply Kubernetes manifests to production
- Always validate with `kustomize build` before deployment
- Encrypt all secrets before committing
- Keep sealed-secrets keys secure (not in Git)
- Regular security audits of repository access
## Additional Resources
### Documentation Files
- `DEPLOYMENT_GUIDE.md` - Detailed deployment instructions
- `ARCHITECTURE.md` - System architecture overview
- `TROUBLESHOOTING.md` - Extended troubleshooting guide
- `UPGRADE_GUIDE.md` - Version upgrade procedures
### Useful Commands Reference
```bash
# Kustomize build and apply
kustomize build . | kubectl apply -f -
kustomize build . | kubectl apply --dry-run=client -f -
# Watch applications
kubectl -n argocd get applications -w
argocd app list
argocd app watch <app-name>
# Get application details
argocd app get <app-name>
kubectl -n argocd describe application <app-name>
# Manual sync
argocd app sync <app-name>
# Access logs
kubectl -n argocd logs -f deployment/argocd-controller
kubectl -n cert-manager logs -f deployment/cert-manager
```
---
**Last Updated**: 2026-04-16
**Maintained By**: NowChess DevOps Team
**Repository**: https://git.janis-eccarius.de/NowChess/GitOps