The Production Homelab Thesis

There are a lot of posts about using AI to build a homelab. Almost all of them follow the same arc: someone runs a prompt, a few Docker containers appear, and the author concludes that AI is either revolutionary or over-hyped.

This series is different. Over 43 working sessions spanning several months, I built a production-grade Kubernetes platform from bare metal. GitOps-managed with FluxCD, SSO for every application, HA PostgreSQL, a software supply chain via Nexus, CI/CD pipelines, security scanning, and observability from day one. With Claude Code as the engineering co-pilot. Including every mistake.

Here’s what I want to say upfront: the interesting part isn’t the AI. It’s the methodology.

What “production” means at homelab scale

When I say production-grade, I mean specific things:

GitOps-first: Every change to the cluster is a pull request. The cluster is what the repository says it is. If you want to change a config, you change the YAML and let Flux reconcile it. You don’t kubectl edit in production. You don’t kubectl edit in the homelab either.

Identity and access management: Every application has SSO. Not “someday I’ll add auth,” from day one. Keycloak as the identity provider, OAuth2 Proxy as the application-layer gateway. One login for everything.

Secrets that aren’t secrets: Nothing is hard-coded. Nothing is base64-encoded in a YAML file and committed to git. Secrets flow from 1Password through External Secrets Operator into Kubernetes Secrets. The repository can be public and nothing sensitive would be exposed.

High availability where it matters: The PostgreSQL backend, used by 10+ applications, is a two-node primary/standby cluster with keepalived VIP failover. The control plane is three K3s servers. Tested failover. Measured recovery time (~15 seconds).

Observability from day one: The monitoring stack, kube-prometheus-stack, Loki, Grafana, was deployed before the first application. You need metrics to debug your first deployment. If you add monitoring later, you’re debugging blind during the hardest part.

CI/CD that actually validates: Every PR runs Terraform format checks, Ansible lint, kustomize build, schema validation with kubeconform, and a policy check (no :latest image tags, no wildcard chart versions, no insecure TLS skip). Breaking the pipeline means something real is wrong.

None of this is beyond reach for a homelab. The reason people don’t do it is that it takes a lot of time to set up. That’s where AI changes the equation.

The tech stack

Before going further, here’s what was built:

Infrastructure layer:

5 Proxmox nodes (mini PCs, 12-16 cores, 31-67GB RAM each)
Terraform with the bpg/proxmox provider for VM provisioning
cloud-init for first-boot configuration
PostgreSQL HA cluster (2 VMs, keepalived VIP), external datastore for K3s and 10 application databases
1Password Connect HA (2 LXC containers, keepalived VIP), secrets backend

Kubernetes layer:

K3s: 3 server nodes + 5 agent nodes, external PostgreSQL datastore
FluxCD for GitOps: bootstrapped to this repository, dependency chain enforced
MetalLB (L2 mode), ingress-nginx, cert-manager (self-signed CA + Tailscale for remote TLS)
NFS subdir provisioner (Synology NAS)
Tailscale operator: all remote access via Tailscale, no public internet exposure

Platform services:

kube-prometheus-stack (Prometheus, Alertmanager, Grafana), Loki + Promtail
Keycloak OIDC (codecentric chart, external PostgreSQL)
OAuth2 Proxy with nginx integration
External Secrets Operator with 1Password Connect ClusterSecretStore
Nexus Repository Manager: 13 proxy repositories (Docker Hub, GHCR, Quay, Helm, apt, npm, pypi, cargo, Go)

Applications (20+ deployed):

Productivity: code-server, Wiki.js, Linkwarden, Draw.io, AFFiNE
Automation: n8n, Windmill
Infrastructure: Portainer, Nexus, Dependency-Track, Homepage dashboard
Media: Plex (movies, music, photos)
Development: GitLab CE, TeamCity, JupyterLab, Coder

Looking at this list now, I went from having grandiose ideas of what I wanted my homelab to be, to now, I am legitimately running out of ideas of what to do. I wanted a CI/CD pipeline, kubernetes cluster for easier management and deployment of applications with access to my NAS. Now I’m exploring how I can use ideas, maintenance and automation driven by Slack and AFFiNE, to developer environments integrated with AI agents to automate test deployments; the possibilities feel insurmountable.

The multi-AI workflow

I didn’t use one AI tool. Here’s the actual workflow:

Claude Code (local CLI) was the primary engineering co-pilot. Every session started with /start-task, creating a git worktree, scoping the work, setting up a task list. The session ended with /finalize, knowledge capture, changelog update, commit, PR, wait for CI, merge. Claude generated Terraform HCL, Kubernetes YAML, Ansible playbooks, shell scripts, and PR descriptions. It reviewed its own work when asked. It caught errors in its earlier output when shown the failing CI logs.

OpenAI Codex (the codex CLI) handled specific targeted tasks, particularly CI pipeline work where a different model’s code generation pattern was useful. Having two AI assistants with different strengths turned out to be practically useful, not just theoretically interesting.

Claude Code Action (GitHub Actions) ran an automated PR review on every pull request, using a detailed homelab-specific review prompt: checking for hardcoded secrets, :latest image tags, incorrect API versions, missing dual ingress, Terraform template escaping, and Ansible privilege escalation issues. The same AI that wrote the code reviewed the PR.

@claude in PR comments gave on-demand assistance: “Claude, look at this CI failure” or “Claude, what would cause this Flux reconciliation error?” without switching to a local terminal.

This created a loop that looked like:

Human: /start-task → Claude: scaffold the implementation
Claude: commit + push + gh pr create
CI: GitHub Actions (Terraform, Ansible, kubeconform, policy-check)
Claude Code Action: automated PR review (security, patterns, docs gaps)
Human: review PR + Claude's review + CI results → merge or request changes

The human stayed in the decision seat for every merge. The AIs handled the majority of the generation, validation, and review.

Before Claude, 2-3 weeks of intermittent work with other tools: partial progress, nothing complete. With Claude: 2.5 weeks to finish the entire build. The token counter for that period reads ~1.5 billion tokens processed by Claude Code, 97% of them cache reads of existing project context. API-equivalent cost: ~$994. Actual cost: ~$120 in flat subscriptions ($100/month Claude Max, $20/month Codex). Post 14 has the full breakdown and what those numbers reveal about why the session methodology worked.

AI Collaboration Note

What Claude contributed: A typical session opened with something like: “We’re adding Dependency-Track to the cluster. It needs external PostgreSQL, dual ingress (nginx LAN + Tailscale), OAuth2 Proxy SSO, and a 10Gi NFS PVC for its vulnerability database. Follow the existing app deployment pattern.” Claude would scaffold all 8 manifests, write the PR description, and explain each decision it made. The multi-AI review loop — Claude Code writing code, Claude Code Action reviewing the PR, GitHub Copilot and Gemini adding additional passes — caught issues that single-pass generation missed.

Where it needed correction: Claude Code has no memory between sessions. Early on, re-explaining infrastructure conventions at the start of every session was a consistent tax: naming patterns, which StorageClass to use, why we have dual ingress, what the secrets pipeline looks like. The fix was the project CLAUDE.md file — a persistent context document Claude reads on session start. Without it, the model gave correct general-purpose answers that didn’t match the homelab-specific patterns already established.

Prompt that worked: "I'm starting a new session. Read CLAUDE.md and the recent git log. We're adding [feature]. Follow the existing app deployment pattern from [most similar existing app]."

Using a different AI tool? The session-based workflow (bounded scope, PR as the delivery unit, knowledge capture at the end) is tool-agnostic. The /start-task and /finalize commands are Claude Code-specific, but the underlying pattern (open a git worktree, do the work, open a PR, let CI validate, merge) is a practice you can implement with any AI coding assistant.

Why this series exists

I looked for content that showed AI-assisted infrastructure work at this level of complexity. I found:

Posts about deploying a few Helm charts with Copilot
Posts about using ChatGPT to generate a docker-compose.yml
Posts about “vibe coding” a side project

What I didn’t find was a documented account of using AI to build the kind of infrastructure that SREs build at work, with all the HA considerations, the security posture, the GitOps discipline, the CI/CD validation, the observability requirements. The kind of setup where the question isn’t “does it run” but “will it still run at 3am when I’m asleep.”

That’s what this series documents.

I wanted to write this series because I was not a full believer in AI being able to achieve what I wanted, at least in its given state. However, now I look at it with a new perspective. I believe there are improvements that can be made but I have developed a trust in the tool if I give it the right guidelines. I now trust that it can manage my systems and execute appropriate response commands to a system malfunction, and if not it will alert me that human intervention is required.

It’s also a story about what AI assistance actually looks like over 43 sessions. Not just the wins. The JVM truststore that took three sessions to fix. The Jetty runtime file extraction that nobody documents. The PROXMOX_VE_* environment variables that silently override your Terraform provider block. The wrong SSH username that Claude insisted on using until shown the error output.

The failures are the interesting part.

What you’ll get from this series

This series is written for practitioners: people who have spent time debugging Kubernetes reconciliation failures at midnight, who understand why “it works on my machine” is a Kubernetes incident waiting to happen, and who want to know what AI-assisted infrastructure work actually looks like over months and dozens of sessions. Not what’s theoretically possible. What happened.

If you’re running infrastructure at work, you’ll recognize the problems: keeping configuration consistent across 20+ services, preventing secrets from leaking into git, making sure the cluster at 3am looks like the cluster at 3pm. The homelab context means I made every mistake without a production SLA attached. The methodology that came out of it applies anywhere the stakes are real.

What comes next

Post 2 starts at the beginning: Proxmox cluster setup, Terraform with the bpg/proxmox provider, VM provisioning via cloud-init, and the PostgreSQL HA cluster. It also covers the first significant mistake, the SSH username that Claude got wrong, and how the PR review process caught it.

Every post in this series will follow a similar structure:

What we were trying to do
How we approached it (with code and decisions shown)
Where it failed and how we fixed it
What Claude contributed, and where human judgment was essential
Lessons you can take away without reading 43 session changelogs

The sanitized public repository with the full IaC code is available at https://github.com/EDKarlsson/homelab-as-production-iac. Every post links to the relevant directory.

The stack, summarized

Layer	Technology	Notes
Hypervisor	Proxmox VE	5 nodes, bpg/proxmox Terraform provider
IaC	Terraform	VM provisioning, provider v0.95.0
Configuration	Ansible + cloud-init	K3s cluster setup, PostgreSQL HA
Container runtime	K3s	3 servers + 5 agents, external PG datastore
GitOps	FluxCD	Bootstrapped to this repo, v2.7.5
Ingress	ingress-nginx + MetalLB	LAN access via nip.io
Remote access	Tailscale operator	All remote access, dual ingress pattern
TLS	cert-manager (self-signed CA) + Tailscale	LAN self-signed, remote *.homelab.ts.net
Identity	Keycloak	codecentric chart, external PostgreSQL
Auth gateway	OAuth2 Proxy	nginx annotations, SSO for all apps
Secrets	1Password Connect + ESO	ExternalSecret for every credential
Observability	kube-prometheus-stack + Loki	Deployed before first application
Artifact registry	Nexus Repository Manager	13 proxy repos, K3s registry mirrors
Primary AI	Claude Code	43 sessions, /start-task + /finalize workflow
Secondary AI	OpenAI Codex	Specific CI/CD tasks
PR review	Claude Code Action + Copilot/Gemini	Automated review on every PR

Next: Post 2 — Provisioning Hardware with Terraform and an AI That Made Mistakes

This series documents a real homelab build. All IP addresses, hostnames, and personal identifiers have been replaced with generic examples. The IaC code is available in the sanitized public repository.