Homelab as Production/Recap
Posts 6-10: The Trustworthy Arc
What the second five posts of Homelab as Production are actually about
Posts 1-5 built a cluster that ran. Posts 6-10 are about building a cluster you can trust.
The difference is sharper than it sounds. A running cluster is one where Flux reconciles and pods are green. A trustworthy cluster is one where you know what happens when a node goes down at 2am, where no credential has ever touched git, where every app requires authentication before it serves a response, and where you can pull a package from your local cache even if Docker Hub is rate-limiting you.
Posts 6-10 are the security and reliability layer. The order matters.
Authentication before applications
The lesson from Post 6 (Keycloak and OAuth2 Proxy): set up SSO before you have more than three apps, not after. The per-app cost of adding OAuth2 Proxy to an nginx ingress is two annotations. The cost of retrofitting SSO onto 20 apps that already have their own account systems, stored credentials, and user data is much higher.
The post is also the one where the most subtle AI failure in the series appears. Claude’s initial Keycloak configuration worked until I added a second hostname (Tailscale remote access). The OIDC issuer claim in the token must exactly match the issuer URL configured in every consumer. Two URLs for the same Keycloak instance means one of them is wrong, and the error message (“invalid issuer”) doesn’t clearly point to the cause. Understanding the OIDC specification, not just the tooling, was what resolved it.
The secrets pipeline as architecture
Post 7 (1Password Connect and External Secrets Operator) is the one I wish I’d read at the start of every infrastructure project.
The principle is simple: secrets live in a secrets manager, and they travel to Kubernetes through an automated sync, never through a human copy-paste or a git commit. The git repository contains only ExternalSecret manifests, which declare where secrets live without containing any values. An attacker who compromises the git history gets configuration, not credentials.
The failure mode Claude hit here was conflating two authentication modes that are mutually exclusive. The fix required auditing all environment configuration and removing the variable that shouldn’t have been set. The lesson: when a Kubernetes resource shows “Ready” but the behavior is wrong, read the controller logs, not just the status.
The pattern as a productivity multiplier
By the sixth deployment I noticed drift: each application had a slightly different manifest structure. Post 8 describes the response: a seven-manifest standard (namespace, PVC, deployment, service, LAN ingress, Tailscale ingress, ExternalSecret) that every application follows without deviation.
The leverage point is that a consistent pattern turns an AI assistant into a reliable boilerplate generator. “Deploy this app following the same pattern as Wiki.js, port 5678, PostgreSQL on this VIP, ExternalSecret from this 1Password item” produces a complete, correct manifest set in one pass. The human review is fast because you’re checking specific values, not reading novel structure. That three-minute review, multiplied across fifteen applications in a session, is what made the application layer tractable.
Your own supply chain
Post 9 (Nexus) starts with a Docker Hub rate limit error mid-afternoon and ends with thirteen proxy repositories covering container images, Helm charts, apt packages, npm, pip, Cargo, and Go modules.
The hardest problem wasn’t setting up Nexus. It was getting K3s containerd to actually route pulls through it. A subtle interaction between how containerd handles path-based registry mirrors and how Nexus expects Docker V2 API requests to arrive produced a 400 Not a docker request error that took several hours to trace to a single missing /v2 path segment in the mirror configuration. That kind of bug is only findable by inspecting the actual HTTP requests at each hop.
When something breaks while you’re asleep
Post 10 (Day-2 Operations) asks the question that separates infrastructure that runs from infrastructure you can trust: what happens automatically when something fails?
PostgreSQL HA with keepalived and a floating VIP means the answer for the database layer is “the standby promotes and the VIP moves, within fifteen seconds, without manual intervention.” The backup automation follows the same VIP: both nodes run the cron job, but only the current VIP holder actually executes it. After a failover, backups resume automatically on the next cycle without reconfiguration.
The most important section of that post is the one where I stopped PostgreSQL on the primary on purpose, watched the failover happen, and confirmed everything worked before I needed it to.
The pattern
Five more posts, the same underlying theme: production-grade infrastructure requires decisions that compound. SSO before you have too many apps to retrofit. Secrets pipeline before the first credential. Deployment pattern before the drift sets in. Supply chain resilience before you depend on upstream availability. Tested failover before you need it.
The AI was most useful when the pattern was already established and the task was replication. It was least useful when the failure mode was a subtle protocol-level interaction that required reading a specification to understand. That division of labor held across all five posts.
Posts 6-10 are live on Medium: [MEDIUM SERIES LINK]
The IaC reference code (sanitized) is at homelab-as-production-iac.