Homelab as Production/Recap
Posts 11-16: The Hard Parts, and What They Revealed
What the final six posts of Homelab as Production are actually about
The last six posts aren’t the glamorous ones. GitLab took ten pull requests to get working. A single OIDC login button took three full sessions to appear. A CI pipeline problem required seven sequential fixes before the runner would register. These aren’t the parts of the story where everything clicks into place.
But this batch is also where the methodology got its real stress test. When the problems are layered and the AI is wrong and the failure is silent, does the process hold? The answer, mostly: yes, because the structure was doing the work that individual judgment couldn’t reliably do alone.
Post 15b is the executive summary for anyone who needs to make the case to their team. Post 16 closes the series with what 43 sessions actually taught. Those are the payoffs. The posts before them are where the cost was paid.
When one bad YAML line takes down three unrelated apps
Post 11 starts with an incident that is embarrassingly simple and surprisingly instructive. A missing indentation level in a HelmRelease, valid YAML that wasn’t a valid HelmRelease, got merged without CI. Within two minutes, Flux’s reconciliation for the entire apps Kustomization had stalled. Flux is atomic at the Kustomization level: one broken resource blocks everything in that Kustomization. Three other applications that had nothing to do with the change stopped reconciling. The fix took thirty seconds. The incident cost forty-five minutes.
The pipeline I built in response runs on every pull request: kubeconform validates all Kubernetes manifests against live CRD schemas, tflint catches Terraform mistakes before plan, ShellCheck finds shell issues before they fail at 2am, and Ansible Lint enforces task structure. The pipeline is smart about what it runs: a path filter detects which areas changed and skips irrelevant checks. Kubernetes-only changes don’t run Terraform validation. Ansible-only changes don’t run kubeconform.
The deeper point: Infrastructure as Code has a worse blast radius than application code for this category of error. A broken Kubernetes manifest can take down reconciliation for every app in a Kustomization. Invalid Terraform can wipe resources. A broken Ansible playbook runs against all cluster nodes simultaneously. The CI cost is a few hundred lines of YAML. The incident cost is measured in time that has better uses.
Ten pull requests and the review loop that found the real problems
Post 12 (GitLab on Kubernetes) is the one about collaboration structure as much as deployment.
GitLab CE went in over three sessions and ten PRs. Some of those PRs were feature work. Several were fixes for bugs that previous PRs introduced. One was a direct response to automated review feedback that made the previous PR substantially better. The most interesting part isn’t the deployment itself; it’s what the multi-AI review loop caught.
The deployment choice was raw manifests over the official Helm chart. GitLab’s cloud-native chart requires a minimum of 8 GB RAM, produces a multi-component deployment with its own ingress preferences, and adds a Helm template rendering layer between your configuration and the running pod. The Omnibus Docker image is a single container, configured via one environment variable, with a single PVC. When something goes wrong, you can read the manifest directly and understand why.
The first hard problem was OIDC with a homelab CA. GitLab CE’s OmniAuth OIDC provider fetches the Keycloak discovery endpoint at startup. The homelab Keycloak instance uses a self-signed CA. The fix: mount the CA certificate into the GitLab container and configure Ruby’s OpenSSL trust store via GITLAB_OMNIBUS_CONFIG. This isn’t in the documentation for the Docker image; it required reading the Omnibus configuration reference.
The second hard problem was the GitLab Kubernetes Runner. The Runner registration token format changed in GitLab 16. Claude’s initial configuration used the legacy format. The CI pipeline caught it immediately (runner registration failing), which led to PR 8, which introduced a different issue with allow_failure behavior when no runner matched the job tags. That cascaded through three more PRs, each fixing the specific error that the previous fix revealed. The loop, paste the CI output, get the next fix, open the next PR, held.
Three sessions, three completely distinct root causes
Post 13 is the one I’d point to if someone asked for a real example of layered debugging with AI assistance.
Dependency-Track’s OIDC login button took three sessions to appear. Not because any single problem was complicated, but because fixing each layer revealed a completely different problem underneath. Session 41: the JVM couldn’t verify the homelab CA certificate because it uses its own truststore, not the OS truststore. Session 42: CORS headers from the Nginx-served frontend weren’t reaching the API. Session 43: Dependency-Track uses Jetty internally, and Jetty’s OIDC client has a different configuration path than the Servlet-based path the documentation describes.
The AI failure worth naming directly: during Act 1, Claude proposed ALPINE_HTTPS_TRUST_ALL_CERTIFICATES=true as the fix for the JVM trust issue. The human accepted it. It was wrong. Not wrong in the sense that it doesn’t work; it does bypass TLS verification. Wrong in the sense that it’s a security bypass, not a fix, and it solved the wrong layer. The actual fix was mounting the CA bundle into the JVM truststore via keytool. That required reading the JVM certificate management documentation, not the Dependency-Track documentation.
The pattern this revealed: confidence is not calibration. Claude is often confident in wrong answers, and the failure is often silent or misleading. The mitigation is structured verification: what exactly does this setting change, and which layer of the stack does it affect?
The honest retrospective
Post 14 is the one that makes the series credible.
Forty-three sessions and 185 pull requests is enough data to say something real about what AI assistance does and doesn’t do well in infrastructure work. Post 14 is the honest version: specific failures named, patterns documented, and a genuine attempt to separate what AI genuinely accelerates from what it still can’t reliably do.
What it got right: One-to-many pattern application. Once the 7-manifest application deployment pattern was established, Claude applied it correctly across 20 applications. Human review took three minutes per app, not thirty, because you were checking values against a known structure, not reading novel configuration. It also consistently surfaced community-standard architectural choices: the Flux clusters/homelab/ directory layout matches the official reference repository exactly. The codecentric Keycloak chart over Bitnami. kube-prometheus-stack as the all-in-one monitoring choice. These are decisions that held up over months of operation.
What it got wrong: Domain-specific configuration details, particularly around security. The SSH username in the Terraform remote backend configuration defaulted to ubuntu. The remote state target was a NixOS system where the correct user was nixos. The error only surfaced during terraform apply. The 1Password Connect versus Service Account conflation produced a ClusterSecretStore that showed Ready while ExternalSecrets failed silently. The external-secrets.io/v1beta1 API version appeared consistently in generated manifests until explicitly corrected, because Claude’s training data predated the v1 promotion in ESO v2.0.
The pattern across failures: security-relevant configurations have two properties that make them harder for AI. First, correctness requires understanding what the setting actually does, not just what it’s named. Second, failures are often silent or indirect. A wrong SSH username fails at runtime, not at plan time. A wrong auth mode makes a secret store look healthy while secrets fail to sync. The failure signature doesn’t point at the cause.
The structure that made the collaboration work: CI for objective validation, automated PR review for convention consistency, human review for judgment. Each layer has different failure modes. The layers are complementary, not redundant.
Translating the methodology to a team
Post 15 addresses the question a colleague asked partway through the project: “How would we do that on the team?”
The honest answer is that the methodology has friction points at team scale. Parallel work, shared review responsibility, and the organizational pressure of “we should be using AI” all introduce failure modes that don’t exist for a single engineer. The post is structured around four practices that made AI assistance productive across 43 sessions: bounded sessions that end in a PR (not a running change to a system), the PR as the unit of review (not the AI conversation), the human review gate as non-negotiable, and knowledge capture at the end of every session.
The post uses the Dan Shapiro / Nate Jones five-level AI maturity framework (Level 0: spicy autocomplete through Level 5: dark factory) as a calibration tool. Most teams are at Level 2. The goal is Level 3 to Level 4, not Level 5. Level 5 is not the goal for infrastructure with real availability and security consequences. The human review gate isn’t inefficiency; it’s the accountability mechanism.
For engineering leaders: the executive summary
Post 15b is a standalone document for the reader who needs to make the case at work. 43 sessions. 185 pull requests. 20+ production-grade applications. All with SSO, all with secrets management, all with dual ingress, all with HA PostgreSQL behind a tested failover. Built on physical hardware. The methodology is described in terms of organizational patterns rather than homelab specifics: bounded sessions, the PR as the unit of work, the review gate, knowledge capture.
The executive summary is designed to be forwarded. If you work with engineering managers or team leads who are trying to understand how to apply AI assistance at team scale without surrendering code quality, this is the post to send them.
What 43 sessions taught
Post 16 is the series close. Not because the homelab is finished (it isn’t), but because the first phase is complete enough to document.
The three lessons that held up longest:
The platform layer is the most important investment you’ll make. Every tutorial jumps to deploying an application. Almost none spend time on MetalLB, cert-manager, and a proper NFS provisioner: the layer that makes all subsequent deployments smooth. Get it right first. A broken cert-manager configuration fails silently for weeks. A misconfigured NFS provisioner makes every new PVC a debugging exercise. Once the platform layer is solid, deploying a new application is minutes, not hours.
GitOps is a discipline, not a feature. Flux doesn’t make you use GitOps. It makes GitOps available. The discipline (every change is a PR, no kubectl edit, the cluster is what the repository says it is) has to come from you. The moments I deviated from that discipline are the moments I lost track of what the cluster was actually running.
The structure is what made the AI’s contributions trustworthy, not the AI itself. Any team can have a capable code generator. The differentiator is whether there’s a quality gate, an audit trail, and a mechanism for turning failures into future-session context. Without those, you have individual wins that don’t compound.
The numbers
43 sessions. 185 pull requests. 20+ deployed applications. Roughly 1.5 billion Claude Code tokens across the project. API-equivalent cost: approximately $994. Actual cost: $120 per month in subscriptions ($100 for Claude Max, $20 for Codex).
The cost per merged PR, at the API-equivalent rate, is about $5.37. The cost per deployed application is around $50. At subscription rates, both are an order of magnitude lower. That math isn’t the point. The point is that the methodology made the cost legible: every PR is a unit of work, every session has a scope, every failure becomes a documented gotcha. The AI’s contributions are auditable because the structure makes them auditable.
Posts 11-16 are live on Medium: [MEDIUM SERIES LINK]
The IaC reference code (sanitized) is at homelab-as-production-iac.