Homelab as Production/Recap
Five Posts In: The Foundation Arc
What the first five posts of Homelab as Production are actually about
The first five posts of the Homelab as Production series are live on Medium. I’ve been thinking about what, taken together, they’re actually about.
Not Terraform. Not FluxCD. Not ingress controllers.
They’re about the order in which you make decisions, and what that ordering costs you if you get it wrong.
What “production-grade” requires you to decide first
The thesis post (Post 1) is an attempt to define terms. “Production-grade” gets used loosely. In this build it means specific things: every cluster change goes through a pull request, every application has SSO from day one, no credential ever touches git, PostgreSQL with tested HA failover, CI/CD that actually validates before merge, monitoring deployed before the first application.
None of those are decisions you can add later without pain. Adding GitOps to a live cluster means going back and recreating your ad-hoc changes as manifests. Adding SSO to 20 existing applications means touching every one of them. Adding secrets management means auditing every hard-coded value.
You either make those decisions first or you pay a retrofit tax. The homelab was a chance to pay it upfront on a system with no production SLA attached.
The first place AI got it wrong
Terraform provisioning went well, mostly. Eight K3s VMs, two PostgreSQL HA nodes, two 1Password Connect LXCs, all from HCL using the bpg/proxmox provider. Most of it worked on the first attempt, which is genuinely impressive for a provider that isn’t in the top-100 of Claude’s training examples.
Then the SSH username bug. Claude kept using the wrong username. Confidently. Even after I described the issue, it repeated the same mistake. Only when I showed the exact error output did it course-correct.
That’s a pattern worth understanding: AI assistants don’t backtrack from a confident wrong answer without evidence. Describing the problem isn’t enough. You need to show the failure output. And you need a place to document the specific ways your environment differs from the defaults (naming conventions, StorageClass names, secrets pipeline structure) so you’re not re-explaining them every session. That file became the CLAUDE.md.
Why structural discipline beats personal discipline
GitOps with FluxCD (Post 3). Once Flux was bootstrapped, the discipline became structural: there was no path to deploying something that didn’t go through git and through a PR review. Not a rule I had to enforce. A constraint the system enforced.
This matters for AI-assisted work because the temptation to shortcut is always present. “Just apply this directly” is always faster in the moment. Structural discipline removes the option. Every AI-generated change went through CI validation and a human review before merging. 185 pull requests, every one reviewed. That’s what structural discipline looks like across a 43-session project.
The invisible layer
Between “K3s cluster is running” and “applications are reachable with TLS” there’s a platform layer that most tutorials skip entirely (Post 4). MetalLB for real IP addresses, ingress-nginx as the single reverse proxy, cert-manager for TLS without internet exposure, NFS provisioner for persistent storage.
This layer also contains the most consequential architectural decision in the project: how LAN and remote access work for every application. That decision shapes the dual ingress pattern that every subsequent application follows. Making it carefully at the platform layer meant not revisiting it 20 times at the application layer.
The AI was useful here in a specific way: it presented three TLS options with explicit trade-offs before recommending anything. The “simplest” option would have broken an unstated requirement (LAN access without being on the Tailscale network). Structured options format, combined with explicit constraints upfront, produced a better answer than a direct question would have. “Here are my constraints; what are my options?” works better than “what TLS should I use?” every time.
The debugging argument
kube-prometheus-stack, Loki, and Grafana went in before the first user-facing workload (Post 5). The argument is simple: your first deployment will produce a surprise. You want debugging tools running before it does.
Within days, the monitoring stack caught a Flux Kustomization circular dependency that would have been invisible without Prometheus metrics. Hours of archaeology, avoided.
The same principle applies to the CLAUDE.md, the gotchas registry, and the PR review loop. The tooling that makes a project maintainable needs to exist before the project gets hard to maintain.
The pattern
Five posts in, the theme is clear: foundations compound. The decisions you make in the first few sessions of a long project either make the remaining sessions easier or harder. GitOps before applications. Monitoring before applications. Platform layer before workloads. Structural discipline before you need it.
The AI was most useful when the constraints were explicit. “Here’s what’s already decided, here’s what’s still open” produced better outputs than “what should I do?” every time. That’s not a limitation of the tooling. It’s how engineering conversations work when they’re going well.
All five posts are live on Medium: [MEDIUM SERIES LINK]
The IaC reference code (sanitized) is at homelab-as-production-iac. Every post links to the relevant directory.