edkarlsson
GitHubGet in touch

Homelab as Production/Part 14 of 16

What Claude Got Wrong (And Right)

An honest retrospective on 43 sessions of AI-assisted infrastructure work

Forty-three sessions. One hundred eighty-five pull requests. Twenty-plus deployed applications. A full stack from bare metal Proxmox nodes through Terraform, Ansible, K3s, FluxCD, and a platform layer that a real engineering team would recognize as production-grade.

So: what did we actually learn about working with AI on infrastructure?

Not the marketing version. The honest version, with the specific failures named, the patterns documented, and an attempt to separate what AI genuinely accelerates from what it still can’t reliably do.

This post is that honest version.

I am a true believer in AI technology, while being conscious of the environmental and societal impact it carries. I thought it would have taken far more effort to get Claude to work in the way I wanted; the actual experience was faster and more capable than I’d expected.

What Claude Got Right

1. Rapid scaffolding of repeatable patterns

The highest-leverage moment in the entire project was the design of the 7-manifest application deployment pattern. It took one session to establish: namespace, PVC, deployment, service, nginx ingress, Tailscale ingress, ExternalSecret, kustomization entry. Once that template existed, Claude applied it correctly across every subsequent application, and the cluster has twenty of them.

The human wrote the pattern once. Claude reproduced it twenty times.

This is the clearest demonstration of what AI does well in infrastructure work: one-to-many pattern application. The first Wiki.js deployment was the hard one. Every subsequent deployment, n8n, Windmill, AFFiNE, code-server, Linkwarden, Dependency-Track, was mechanical. Claude adapted port numbers, image names, 1Password item references, and resource limits while maintaining complete structural consistency. Human review took three minutes per app, not thirty, because you were checking values, not verifying structure.

The pattern also held across file types. ExternalSecrets consistently used external-secrets.io/v1. HelmReleases consistently pinned chart versions. Tailscale ingresses consistently used the same minimal structure. After session one established the convention, Claude maintained it across forty-three sessions without being reminded.

2. Surfacing community-standard architectural choices

Several of the project’s best decisions came from Claude knowing what the broader community had converged on, before I had read the relevant documentation deeply.

The Flux repository layout is the clearest example. I described the existing directory structure, Terraform in infrastructure/, Kubernetes manifests in kubernetes/, and asked where the Flux entry point should live. Claude walked through three options with concrete trade-offs: mixing Flux into the Terraform directory (different tooling layers, bad), putting Flux inside kubernetes/ (circular dependency, awkward path references), or a separate clusters/homelab/ directory (clean separation, community-standard, ready for multi-cluster). The recommendation was Option C, matching the official Flux flux2-kustomize-helm-example reference repository exactly.

Other examples: the codecentric keycloakx Helm chart over Bitnami (more current, closer to upstream Keycloak’s own configuration conventions); kube-prometheus-stack as the all-in-one monitoring choice over assembling Prometheus, Alertmanager, and Grafana separately; the NFS subdir provisioner for storage (lightweight, good enough for homelab, doesn’t require a dedicated storage operator). In each case, Claude drew on broad community knowledge and gave a recommendation that held up over months of operation.

3. Option-presenting for consequential decisions

The better AI interactions in this project were not “generate this configuration.” They were “here are three options with trade-offs, which do you want?”

TLS strategy had three real options: self-signed CA issued by cert-manager (good for LAN, bad UX for trust), Let’s Encrypt (requires public DNS or ACME DNS challenge, complex for homelab), or Tailscale’s built-in TLS certificates (excellent UX for remote access, only works on the Tailscale network). Claude laid out all three with specific implications for each. The decision was easy once the options were clearly stated: self-signed CA for LAN nginx ingress, Tailscale for remote access, dual ingress on every application.

Database backend choices, Nexus proxy repository scope, Helm chart selection: in each case, the most useful Claude interaction was option generation and trade-off analysis, not autonomous decision-making. This is the right division of labor. Claude knows what the options are and what their properties are. The human knows which properties matter for this specific context.

4. Catching its own errors when shown output

The debugging loop that developed over the project was efficient: Claude implements something, CI runs, the human pastes the failing log output, Claude traces the root cause. This worked well.

The 7-PR GitLab CI debugging chain from Post 11 is the most extreme example. A single CI pipeline problem required seven sequential PRs to fully resolve, covering runner registration, token format changes in GitLab 16, allow_failure behavior with no matching runner, and Ansible handler ordering. Each PR fixed the specific error Claude had been shown. The human’s job was to run CI and paste the output. Claude’s job was to trace the error and propose the next fix.

This pattern held broadly across the project. When given concrete error output, a Flux reconciliation failure, a kubeconform schema violation, a CI lint error, Claude usually identified the root cause accurately on the first or second attempt. The loop was the mechanism: Claude doesn’t always get it right in the abstract, but it reliably gets it right when shown the actual failure.

5. Comprehensive PR descriptions and knowledge capture

Every pull request in this project has a clear summary, a technical decisions section explaining why choices were made, a files-changed table, and a test plan. This was generated by Claude as part of the /finalize workflow.

The value compounded over time. Months later, the changelog is comprehensive enough to reconstruct what happened in any session, what was deployed, what problems were encountered, why a particular approach was chosen over alternatives. This isn’t glamorous work. It’s also not work that anyone does consistently when doing it manually requires extra effort. Having it happen automatically as part of every session close meant it actually happened.

The gotchas registry, docs/reference/technical-gotchas.md, was built the same way. When Claude made a mistake that led to a real failure, it consistently suggested documenting the pattern. Over forty-three sessions, that became a reference covering fifty-plus confirmed issues with specific fixes, spanning Terraform, Kubernetes, Ansible, keepalived, and individual applications. The document exists because Claude proposed adding to it. The human kept accepting the proposals.

6. Cross-cutting consistency across the entire codebase

Twenty applications. All ExternalSecrets use external-secrets.io/v1. All HelmReleases have remediation.retries set. All apps have both nginx and Tailscale ingress. All image tags are pinned.

Maintaining this kind of consistency across a large codebase is exactly the kind of task that humans do poorly over time. Attention drifts. Shortcuts get taken. The fifteenth deployment drifts from the first. Claude maintained consistency because it was working from the established pattern and had no fatigue-driven incentive to cut corners.

When the automated PR review (run via Claude Code Action) flagged an external-secrets.io/v1beta1 in a new ExternalSecret, or spotted a missing Tailscale ingress on a new app, or caught an image tagged :latest, it was enforcing the same patterns that had been established in session one. The review prompt was the pattern specification. The automated review was the consistency check.

What Claude Got Wrong

1. Domain-specific configuration details

The SSH username bug from Post 2 is the canonical example. The bpg/proxmox provider uses two authentication mechanisms: Proxmox API credentials for most operations, and SSH credentials for operations the Proxmox API can’t perform. The SSH username must be the operating system root user, not the Proxmox API token ID.

Claude set it to the API token ID, terraform@pve!provider, because from looking at the provider configuration alone, that’s the identity being authenticated. The distinction between “what you authenticate as for the API” and “what you authenticate as for SSH” isn’t inferrable from the provider schema. It requires either reading the specific documentation for this edge case or having encountered it before.

The error only surfaced during terraform apply. This is the pattern: Claude’s mistakes in domain-specific configuration are often invisible until the thing actually runs. Schema validation catches missing fields. It doesn’t catch fields that are syntactically valid but semantically wrong for this specific provider.

The 1Password Connect vs. Service Account conflation (Post 7) is the same category of failure. Connect mode and Service Account mode are mutually exclusive. Claude generated configuration that set up both, OP_CONNECT_HOST for the ESO configuration and OP_SERVICE_ACCOUNT_TOKEN in the environment, because the reasoning “Connect for Terraform, service account for ESO” sounds plausible without knowing that the 1Password provider enters an ambiguous authentication state when both are present. The ClusterSecretStore showed Ready. ExternalSecrets failed silently. The failure mode only became clear through log analysis.

The external-secrets.io/v1beta1 API version is a third example. Claude’s training data included ESO documentation from before the v1 promotion in ESO v2.0. It generated v1beta1 consistently until explicitly corrected. The API version field is valid YAML. It just doesn’t match what the cluster is serving.

These are not random errors. They cluster around the same underlying cause: Claude’s knowledge of a tool’s specific configuration is bounded by its training data, and the training data doesn’t always include the latest breaking changes, the product-specific quirks, or the exact behavior of unusual edge cases in provider authentication models.

2. The ALPINE_HTTPS_TRUST_ALL_CERTIFICATES red herring

Post 13 documented the Dependency-Track OIDC integration, and there’s a failure in that story that deserves naming directly: Claude proposed ALPINE_HTTPS_TRUST_ALL_CERTIFICATES=true as the fix for the JVM TLS trust issue, and the human accepted it. It was wrong.

The Dependency-Track application has two separate TLS validation layers: the Alpine application framework’s HTTP utilities (covered by ALPINE_HTTPS_TRUST_ALL_CERTIFICATES) and the JVM’s own SSL context (not covered). The actual fix required importing the Keycloak certificate into the JVM truststore via an init container running keytool -import and pointing the JVM at the custom truststore via JAVA_TOOL_OPTIONS=-Djavax.net.ssl.trustStore=....

ALPINE_HTTPS_TRUST_ALL_CERTIFICATES=true wasn’t a made-up setting. It was documented, it was real, and it addressed a real layer of the problem. But it didn’t address the JVM layer, which is where the OIDC token validation was actually failing. The SSO button still didn’t appear. The next session spent time tracing back through the assumption.

This is the failure pattern that deserves the most attention: Claude is confident in wrong answers, and the human can’t always evaluate confidence calibration in real time. The proposed fix sounded correct. It referenced a real environment variable with a clear name. It was accepted. It was wrong.

The mitigation is explicit verification rather than trust. When Claude proposes a configuration that should fix an observed symptom, the right question is: “what exactly does this setting change, and which layer of the stack does it affect?” In this case, that question would have revealed that ALPINE_HTTPS_TRUST_ALL_CERTIFICATES covered only the Alpine HTTP utilities, not the JVM SSL context, and the OIDC flow uses the JVM for certificate validation.

3. Occasional over-engineering

Claude occasionally proposed more abstraction than the situation warranted. Helper scripts for one-time operations. Feature flags for things that could simply be changed directly. Parameterized modules before the second use case existed to justify the parameterization.

The CLAUDE.md project instructions now include a note about avoiding over-engineering, added specifically because this pattern appeared often enough to be worth naming. The instinct to generalize isn’t wrong. Good infrastructure is composable and reusable. But premature abstraction creates cognitive overhead without benefit, and in a homelab context, the cost of “this could be more general” is often higher than the benefit.

Here’s a representative excerpt from the CLAUDE.md that was in use by mid-project:

# CLAUDE.md — homelab-iac

## What this repo is
A Terraform + Ansible + FluxCD monorepo managing a 5-node Proxmox cluster
running K3s with 20+ applications. PostgreSQL HA via keepalived VIP.

## Key conventions
- Every application gets dual ingress: nginx (LAN) + Tailscale (remote)
- All secrets via 1Password Connect → ESO → Kubernetes Secret
- ExternalSecret `property:` fields must reference CUSTOM TEXT FIELDS only
  (default Login fields — username, password — are not addressable by ESO)
- All ExternalSecrets use `external-secrets.io/v1` (not v1beta1, removed in ESO v2)
- One session = one PR. Never `kubectl edit` in the cluster directly.

## Known gotchas
- `PROXMOX_VE_*` env vars silently override provider block
- `user_account` and `user_data_file_id` are mutually exclusive in bpg/proxmox
- pg_hba.conf is NOT replicated by streaming replication — apply changes on both nodes
- keepalived health check must only check pg_isready, never pg_is_in_recovery()

The gotchas section grows with the project. By session 43, it had 27 entries.

The practical correction was to ask: “does this need to be this general right now, or is there a simpler version that works for the current case?” Claude usually accepted the simpler framing when it was made explicit.

4. Context limitations in long sessions

In long sessions, three or four hours covering several interconnected changes, Claude would occasionally contradict a decision made earlier in the same session. A naming convention established at the start of the session would be violated in the fourth PR. A constraint explained at the beginning would be forgotten by the time code was being generated for the fifth task.

The /finalize workflow and MEMORY.md helped significantly but didn’t fully solve this. The real mitigation was session discipline: keep sessions bounded and focused. A session that tries to cover everything ends up with a Claude that has lost track of its own constraints. A session with a narrow scope, “today we are deploying Dependency-Track and nothing else,” has enough room in context for the relevant decisions to stay accessible throughout.

5. Security-sensitive configuration is the highest-risk category

Looking across the failures in this project, the pattern is clear: the SSH username, the auth mode conflation, the TLS trust bypass. These are all security-relevant configurations. This isn’t coincidental. Security configurations have two properties that make them harder for AI.

First, they’re often deliberately opaque. Security-sensitive settings are usually non-obvious for a reason. Making the secure path the easy path is a user interface challenge that most tools haven’t solved. The bpg/proxmox provider doesn’t label the SSH username field “this should be different from your API authentication identity.” The 1Password docs discuss Connect mode and Service Account mode in separate sections, as if they’re alternatives, not mutually exclusive configurations.

Second, failures in security configurations are often silent or misleading. A wrong SSH username fails at runtime, not at plan time. A wrong auth mode makes the secret store look healthy while secrets fail to sync. A missing certificate trust layer lets you get past some validation checks while failing at others. The failure signature is indirect.

The PR review loop was most valuable here. Not because the automated review always caught security issues, it didn’t always, but because the human review gate meant that security-relevant changes had a second pair of eyes before merging. The rule that emerged from this project: never skip human review on changes that touch authentication configuration, TLS handling, or secrets management, regardless of how confident the automated review is.

The PR Review Loop as a Quality Gate

The collaboration cycle that made this project work looks like this:

flowchart LR
    A["Human<br/>defines scope<br/>/start-task"] --> B["Claude<br/>implements<br/>commit + PR"]
    B --> C["GitHub Actions<br/>CI validation<br/>kubeconform · lint · policy"]
    C --> D["Claude Code Action<br/>automated PR review<br/>homelab-specific prompt"]
    D --> E["Human<br/>reviews PR<br/>+ both AI outputs"]
    E -->|"changes requested"| B
    E -->|"approved"| F["Human<br/>merges PR<br/>never AI"]
    F --> G["Knowledge capture<br/>/finalize<br/>gotchas registry"]
    G --> A

The human stays in the decision seat at every merge. The AIs handle generation, validation, and review.

What each layer caught:

The CI pipeline caught structural issues: invalid YAML, schema violations via kubeconform, image tags that weren’t pinned, Terraform format failures, Ansible lint errors. These are objective checks. They catch things that are definitely wrong.

The Claude Code Action automated review caught pattern violations: an ExternalSecret using v1beta1, a new application missing its Tailscale ingress, a HelmRelease without remediation.retries, a configuration-snippet annotation that would be blocked by the ingress-nginx admission webhook. These are consistency checks against established conventions. They catch things that aren’t wrong in isolation but are wrong in context.

The human review caught what the AI review missed: domain-specific gotchas (the SSH username), security concerns (the Connect/Service Account question), and architectural decisions that require judgment about tradeoffs. The human also caught the ALPINE_HTTPS_TRUST_ALL_CERTIFICATES case, but only in the next session, when the symptom hadn’t gone away. Real-time evaluation of AI confidence is genuinely difficult.

The key property of this loop is that it’s layered. No single review mechanism is sufficient. CI catches objective failures. The automated review catches convention violations. The human catches the rest. Each layer has a different failure mode, so the layers are complementary rather than redundant.

The Multi-AI Workflow in Practice

This project used four AI systems:

Claude Code (local CLI, 43 sessions) was the primary engineering co-pilot. Every session was the same loop: define scope, generate code, review output, iterate. Claude Code handled the majority of code generation across Terraform HCL, Kubernetes YAML, Ansible playbooks, shell scripts, and documentation.

OpenAI Codex handled specific targeted tasks, primarily in CI/CD work where a different model’s pattern for bash scripts was useful. It wasn’t a replacement for Claude Code but a complement. The 7-PR GitLab debugging chain used both: when Claude Code’s approach to a specific bash idiom was blocked by a subtle shell expansion issue, framing the same problem for Codex produced a different approach that was easier to validate against the specific failure mode. Having two AI models available made the debugging more effective. Different models have different blind spots.

Claude Code Action (GitHub Actions) provided automated PR review. The review prompt was the key: a homelab-specific list of things to check, no :latest tags, correct API versions, dual ingress present, Terraform template string escaping, Ansible privilege escalation at task level, no insecure TLS settings. Baking the context into the review prompt produced reviews that were meaningfully better than a generic “review this PR” request.

@claude in PR comments provided on-demand assistance for CI failure analysis without context switching to a local terminal. When a CI failure needed quick analysis, tagging Claude in the PR comment thread produced a root cause suggestion without opening a new session, loading context, and re-establishing the problem.

The workflow wasn’t “pick one AI and use it for everything.” It was “different AI tools for different points in the loop, each with a specific role.”

Session Discipline as the Real Unlock

Here is the honest claim: the biggest factor in forty-three sessions of productive work was not the AI model. It was the session structure.

Each session had a narrow, defined scope. Every change was a pull request, no kubectl apply in production, no terraform apply without a plan review, no configuration change without a Git diff. Every session ended with knowledge capture: MEMORY.md updated, technical-gotchas.md updated, CHANGELOG.md entry written, PR merged, semver tag cut. Failures were documented immediately. Patterns were named and stored. MEMORY.md is a persistent cross-session context file maintained in the .claude/ directory. Unlike CLAUDE.md (which documents project conventions) or the gotchas registry (which documents specific failures), MEMORY.md accumulates the kind of architectural decisions and running context that Claude Code needs across multiple sessions.

This structure would make human-only engineering more productive too. The session discipline isn’t a Claude Code feature. It’s a practice that Claude Code’s /start-task and /finalize commands formalized. The AI amplified the structure. The structure existed independently.

The corollary: AI without structure does not scale. When sessions ran long without a defined scope, quality declined. When changes skipped the PR process and went directly to the cluster, they were harder to undo and undocumented. When knowledge capture was deferred, context was lost and gotchas were re-learned. The structure was what made the AI’s contributions reliable over time. Remove the structure, and you have a capable code generator with no quality gate and no memory.

Bounded scope. PR as the delivery unit. Knowledge capture at session end. Failures documented immediately. These four practices, applied consistently, are what made this project work.

The natural extension of this methodology, multiple AI models coordinated as a team, each with defined roles, a shared task queue, and scoring to compare outputs, is documented in the companion Series 2.

The Numbers

The actual cost of this project is worth stating plainly.

Timeline: 2-3 weeks of intermittent progress with other tools before switching to Claude. 2.5 weeks with Claude to complete the entire build: 8 K3s VMs, HA PostgreSQL, FluxCD GitOps, platform services, SSO, secrets management, CI/CD, observability, and 20+ applications. The prior work wasn’t wasted — it built familiarity with the problem space. But the output difference is real.

Token usage (Claude-assisted phase only):

Tool Total Tokens API-Equivalent Cost
Claude Code 1,506,088,349 $801.70
Codex CLI ~662,223,346 $191.94
Total ~2.17 billion ~$993.64

Actual spend: $100/month Claude Max + $20/month Codex. For a 3-week project, roughly $90 total — against an API-equivalent of ~$994.

The more interesting number is the token distribution. Of Claude Code’s 1.5 billion tokens, 1.46 billion were cache reads. Actual new input: 241,000 tokens. Output: 815,000 tokens.

97% of every token was re-reading existing project context — the CLAUDE.md, session notes, git log summaries, changelogs — not processing new prompts. The session methodology wasn’t just organizationally useful. It was computationally efficient: the cost of re-establishing context on each turn was absorbed by prompt caching, not by new token generation.

This has a practical implication for replicating this workflow: the subscription model is the right pricing tier for context-heavy, session-based engineering work. The whole point of the CLAUDE.md-first approach is giving the model full context on every turn. On a per-token API, that discipline has a cost. On a flat subscription, it’s included. The pricing model changes how aggressively you invest in persistent context.

The hardware cost was under $2,000 for five used mini PCs, purchased before this project started. The AI tooling cost was the subscription fees above. The real investment was time — roughly 150 hours across the full project, including the prior work with other tools. The platform that came out of it runs 20+ applications in production today.

AI Collaboration Note

There is an obvious irony in this post: it was written with AI assistance. Claude drafted a post critiquing Claude’s failure modes.

The irony is worth naming, because it illustrates the actual situation with AI-assisted work in 2026. You can’t avoid using AI for this kind of work if you want to be productive at the scale this project operated. The choice isn’t “AI or no AI.” It’s “AI with what level of verification.”

The answer in this project was: always verify claims about domain-specific configuration against primary sources. Always require a human to merge security-relevant changes. Always run CI before merging. Never treat AI confidence as a proxy for correctness.

The AI that wrote the draft of this post also got the ALPINE_HTTPS_TRUST_ALL_CERTIFICATES proposal wrong. The human is the one who noticed the JVM trust layer was a separate concern. That’s the actual collaboration: AI handles the throughput, humans handle the judgment calls that require genuine expertise or that have security implications.

That division of labor, maintained consistently, is what made 185 pull requests across 43 sessions work at all.

Lessons

  • AI is highest-leverage for one-to-many pattern application. Design the pattern once with care. Let AI reproduce it across N applications. The quality of the output is bounded by the quality of the template.

  • Domain-specific configuration details are the failure category to watch. Provider authentication models, API version transitions, application-specific TLS layers, these require primary source documentation, not AI recall. Verify before committing.

  • Confidence is not calibration. Claude is often confident in wrong answers, and the failure mode is often silent or misleading. The mitigation is structured verification: what exactly does this setting change, and which layer of the stack does it affect?

  • Security-sensitive configuration requires human review, always. No automated review substitutes for a second pair of eyes on authentication configuration, TLS handling, and secrets management. The PR review loop is not a formality. It’s the quality gate.

  • The structure is what made the AI’s contributions trustworthy, not the AI itself. Any team can have a capable code generator. The differentiator is whether there’s a quality gate, an audit trail, and a mechanism for turning failures into future-session context. Without those, you have individual wins that don’t compound.

Next: Post 15 — AI-Assisted Infrastructure for Your Team — translating the homelab methodology to an engineering innovation team.

This post is part of Homelab as Production: AI-Assisted Infrastructure from Zero to GitOps, a 16-part series on building a production-grade homelab with Terraform, Ansible, FluxCD, and AI-assisted development.