edkarlsson
GitHubGet in touch

Homelab as Production/Part 4 of 16

The Platform Layer Nobody Talks About

MetalLB, ingress-nginx, cert-manager, NFS: the boring infrastructure every app depends on

This is Part 4 of “Homelab as Production: AI-Assisted Infrastructure from Zero to GitOps.” The series covers building a full IaC-managed homelab from bare metal to 20+ production-grade applications using Terraform, Ansible, FluxCD, and Claude Code.

Most Kubernetes tutorials go straight from cluster creation to deploying a “hello world” application. They skip what actually has to exist before a real app can run: a LoadBalancer implementation, an ingress controller, a TLS certificate strategy, and somewhere to put persistent data. The gap between “K3s cluster is up” and “applications are reachable and have TLS” is not small. This post covers that gap.

This is the layer that nobody writes tutorials about because it’s not exciting. But I spent more time debugging this layer than any individual application. Get it wrong and nothing else works. Get it right and every future application deployment becomes a ten-minute exercise.

Why Platform Services Come First

The dependency graph is strict. Applications need:

  • A way to get an external IP (MetalLB gives this to Services of type: LoadBalancer)
  • A reverse proxy that routes HTTP/HTTPS traffic to the right Service (ingress-nginx)
  • TLS certificates so browsers don’t throw security warnings (cert-manager)
  • Persistent storage (NFS subdir provisioner creates PersistentVolumes on demand)

Each of these depends on the previous one. ingress-nginx requests a LoadBalancer IP from MetalLB. cert-manager creates Certificate resources that the ingress controller reads to configure TLS termination. Applications then reference the nfs-kubernetes StorageClass to claim volumes.

In the Flux GitOps structure, this plays out as an explicit dependency chain:

platform-controllers  →  platform-configs  →  apps

Platform controllers (the HelmReleases that install MetalLB, ingress-nginx, cert-manager, NFS provisioner, and the Tailscale operator) must be healthy before configs (IPAddressPool, ClusterIssuer, cluster-wide secrets) can apply. Both must be healthy before any application manifests reconcile. Breaking this ordering causes cascading reconciliation failures that can be confusing to debug because the error surfaces in the wrong layer.

MetalLB: Making LoadBalancer Services Work

K3s ships with a built-in LoadBalancer implementation called servicelb (also known as klipper-lb). For a single-node development cluster it works fine. For anything more complex, especially if you want to control which IP addresses get assigned and need clean L2 ARP behavior, it becomes a problem. servicelb and MetalLB can’t coexist cleanly. The fix is straightforward: disable servicelb when provisioning K3s.

In the K3s configuration passed via Ansible, this looks like:

# group_vars for K3s servers
k3s_server_extra_args:
  - "--disable=servicelb"

With servicelb out of the way, MetalLB takes over. MetalLB supports two modes: BGP (Border Gateway Protocol) and L2 (Layer 2/ARP). BGP is the right choice for production data centers with BGP-capable routers. In a homelab with a consumer router and a flat LAN, L2 mode is the correct call. L2 mode works by having MetalLB respond to ARP requests for the assigned IP address, making the virtual IP reachable on the local network. No BGP configuration, no router changes, no special hardware required.

The HelmRelease is straightforward:

# kubernetes/platform/controllers/metallb.yaml
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: metallb
  namespace: metallb-system
spec:
  interval: 30m
  chart:
    spec:
      chart: metallb
      version: "0.15.x"
      sourceRef:
        kind: HelmRepository
        name: metallb
        namespace: metallb-system

The IP pool configuration lives in platform/configs/ rather than platform/controllers/, because it requires the MetalLB CRDs to already exist before it can apply. This is exactly the dependency chain in action:

# kubernetes/platform/configs/metallb-config.yaml
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: default-pool
  namespace: metallb-system
spec:
  addresses:
    - 10.0.0.201-10.0.0.250
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: default
  namespace: metallb-system
spec:
  ipAddressPools:
    - default-pool

The IP range 10.0.0.201-10.0.0.250 sits above the home router’s DHCP ceiling. This is important: if your router hands out addresses up to .200, starting the MetalLB pool at .201 means no conflicts. The L2Advertisement resource tells MetalLB to use ARP mode for the specified pools. Two resources, twenty-one lines, and LoadBalancer Services start getting real IPs.

ingress-nginx: One Front Door for Everything

With MetalLB operational, the next piece is ingress-nginx. This is the reverse proxy that receives all HTTP/HTTPS traffic at a single IP and routes it to the correct application based on the hostname.

The reason to run one ingress controller rather than multiple is operational simplicity. A single LoadBalancer IP, a single set of TLS termination rules, a single place to look when an application isn’t reachable. Every application gets its own Ingress resource that declares which hostname routes to which Service, and the controller handles the actual nginx configuration.

# kubernetes/platform/controllers/ingress-nginx.yaml
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: ingress-nginx
  namespace: ingress-nginx
spec:
  interval: 30m
  chart:
    spec:
      chart: ingress-nginx
      version: "4.x"
      sourceRef:
        kind: HelmRepository
        name: ingress-nginx
        namespace: ingress-nginx
  values:
    controller:
      service:
        type: "LoadBalancer"
    admissionWebhooks:
      enabled: false

The type: LoadBalancer here is the connection point to MetalLB. When this HelmRelease applies, Kubernetes creates a LoadBalancer Service for the nginx controller, MetalLB assigns it 10.0.0.201 (the first available IP in the pool), and all subsequent Ingress resources route through that address.

Watching 10.0.0.201 appear in kubectl get svc -n ingress-nginx was the first moment the entire GitOps pipeline felt real. Before that, Flux had been reconciling resources and the cluster had been accepting manifests, but nothing had reached out to the physical network and announced itself. That address was LAN-routable. I could ping it from my laptop. A commit in a git repository had caused a real-world address assignment: no manual kubectl apply, no hand-edited config. This was the review loop working end to end: change the manifest, push, let Flux reconcile, observe the result. Every application deployment after this was proving the same loop, at a higher layer of the stack.

It has been a genuinely surprising experience to be in the customer’s shoes when requesting an application and watching Claude ask clarifying questions, figure out why I want something, provide alternatives or push back when something seemed unnecessary, then implement and deploy it. The platform layer made all of that possible.

Note the admissionWebhooks.enabled: false. This disables the validating admission webhook that ships with ingress-nginx by default. In a homelab, the webhook adds complexity without meaningful benefit, and as we’ll see later, it’s also responsible for one of the more confusing gotchas in the platform layer.

The TLS Strategy Decision

Before deploying cert-manager, I had to decide what kind of TLS certificates I actually wanted. There are three realistic options for a homelab:

Option 1: Let’s Encrypt. The standard for public-facing sites. cert-manager handles the ACME challenge automatically. The catch: Let’s Encrypt requires either port 80/443 open to the internet for HTTP-01 challenges, or a DNS provider that supports DNS-01 challenges. This homelab has no public internet exposure. That’s a deliberate design constraint, so Let’s Encrypt was out.

Option 2: Self-signed CA for LAN + Tailscale for remote. Create a private Certificate Authority inside the cluster using cert-manager’s self-signed issuer. All LAN-accessible applications get TLS from this internal CA. For remote access, use the Tailscale Kubernetes operator, which automatically provisions publicly trusted *.homelab.ts.net certificates backed by Let’s Encrypt. No ports need to be open, because Tailscale handles the DNS-01 challenge internally through its control plane.

Option 3: Tailscale as the sole ingress. Skip ingress-nginx and cert-manager entirely. Every application gets a *.homelab.ts.net hostname and a valid cert, but only works when connected to Tailscale. No LAN access without the VPN.

Claude presented all three options with their trade-offs. Option 3 was tempting for simplicity, but the requirement for LAN access without Tailscale, such as running automations or accessing tools from devices that aren’t on the tailnet, ruled it out. Option 1 was ruled out by the no-public-exposure constraint. Option 2 won.

The self-signed CA chain in cert-manager has three parts:

# kubernetes/platform/configs/cluster-issuers.yaml
---
# Step 1: A ClusterIssuer that can sign anything using a self-signed cert
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned-issuer
spec:
  selfSigned: {}
---
# Step 2: The CA certificate itself — signed by selfsigned-issuer
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: homelab-ca
  namespace: cert-manager
spec:
  isCA: true
  commonName: Homelab CA
  secretName: homelab-ca
  duration: 87600h  # 10 years
  privateKey:
    algorithm: ECDSA
    size: 256
  issuerRef:
    name: selfsigned-issuer
    kind: ClusterIssuer
---
# Step 3: The ClusterIssuer that applications reference
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: homelab-ca-issuer
spec:
  ca:
    secretName: homelab-ca

The chain works like this: selfsigned-issuer signs the homelab-ca Certificate, which is stored as a Kubernetes Secret. homelab-ca-issuer reads that Secret and uses it to sign TLS certificates for each application’s Ingress. All 20+ applications reference homelab-ca-issuer in their Ingress annotations.

The self-signed CA has a 10-year validity period. When it expires, every service using the homelab CA trust store will break simultaneously, so add a calendar reminder and document the renewal procedure before you need it.

The browser warning on LAN (because the homelab CA isn’t in the system trust store by default) is manageable. Export the CA cert once and add it to your system trust store, and the warnings disappear:

kubectl get secret homelab-ca -n cert-manager \
  -o jsonpath='{.data.tls\.crt}' | base64 -d \
  | sudo tee /usr/local/share/ca-certificates/homelab-ca.crt
sudo update-ca-certificates

For remote access via Tailscale, the TLS story is entirely different. The Tailscale Kubernetes operator provisions *.homelab.ts.net certificates automatically through Tailscale’s internal Let’s Encrypt integration. No cert-manager involvement, no ACME challenges, no ports opened. The certificates are publicly trusted out of the box.

The Dual Ingress Pattern

Every application in this homelab gets two Ingress resources. This is the most consequential architecture decision in the platform layer, because it shapes how every application is accessed for the life of the cluster.

The pattern works like this:

LAN ingress uses ingressClassName: nginx, routes through the MetalLB IP at 10.0.0.201, and uses nip.io, a wildcard DNS service that resolves any hostname containing an IP back to that IP, so no DNS server configuration is needed. TLS comes from the homelab CA issuer.

Remote ingress uses ingressClassName: tailscale, which the Tailscale operator handles. The application becomes reachable at <app>.homelab.ts.net from any device on the tailnet. TLS comes from Let’s Encrypt and is publicly trusted.

Here is the n8n workflow automation app as a concrete example of both ingresses:

# ingress.yaml — LAN access
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: n8n
  namespace: n8n
  annotations:
    cert-manager.io/cluster-issuer: homelab-ca-issuer
spec:
  ingressClassName: nginx
  rules:
    - host: n8n.10.0.0.201.nip.io
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: n8n
                port:
                  number: 5678
  tls:
    - secretName: n8n-tls
      hosts:
        - n8n.10.0.0.201.nip.io
# tailscale-ingress.yaml — remote access
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: n8n-tailscale
  namespace: n8n
spec:
  ingressClassName: tailscale
  defaultBackend:
    service:
      name: n8n
      port:
        number: 5678
  tls:
    - hosts:
        - n8n  # becomes n8n.homelab.ts.net

The Tailscale ingress is deliberately minimal. The hostname in the tls.hosts list is just n8n, and the operator appends the tailnet domain automatically. No cert-manager annotation needed. The Tailscale operator handles certificate provisioning.

This two-file pattern repeats across all 20+ applications. Familiarity with it makes adding new applications fast.

NFS Subdir Provisioner: On-Demand Persistent Volumes

Applications that need persistent storage, like databases, artifact repositories, and data pipelines, require PersistentVolumes. The question is which provisioner to use.

The homelab has a Synology NAS with an NFS export at /volume1/kubernetes. The choice came down to nfs-subdir-external-provisioner versus more capable alternatives like democratic-csi. The decision was straightforward: nfs-subdir is a single-purpose tool with a decade of production use behind it, a simple Helm chart with a handful of values, and no external dependencies. democratic-csi supports more storage protocols and advanced features, but also brings considerably more operational surface area. For a homelab NAS share, nfs-subdir is the right level of complexity.

# kubernetes/platform/controllers/nfs-provisioner.yaml
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: nfs-subdir-external-provisioner
  namespace: nfs-provisioner
spec:
  interval: 30m
  chart:
    spec:
      chart: nfs-subdir-external-provisioner
      version: "4.x"
      sourceRef:
        kind: HelmRepository
        name: nfs-subdir-external-provisioner
        namespace: nfs-provisioner
  values:
    nfs:
      server: 10.0.0.161   # Synology NAS LAN IP
      path: /volume1/kubernetes
    storageClass:
      name: nfs-kubernetes
      defaultClass: false
      reclaimPolicy: Retain
      archiveOnDelete: true

The reclaimPolicy: Retain and archiveOnDelete: true settings are important for a homelab. When a PersistentVolumeClaim is deleted, Retain keeps the underlying data on the NFS share rather than deleting it, and archiveOnDelete renames the directory rather than removing it. Accidental PVC deletion doesn’t mean data loss.

Applications then claim storage with a simple PVC referencing the StorageClass:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: n8n-data
  namespace: n8n
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: nfs-kubernetes
  resources:
    requests:
      storage: 5Gi

The provisioner creates a subdirectory on the NAS, creates a PersistentVolume bound to the NAS path, and the application has persistent storage. No manual volume management required.

AI Collaboration Note

What Claude contributed: When I described the TLS constraints (no public internet exposure, need LAN access, Tailscale already in use), Claude organized the decision space into three distinct options with explicit trade-offs before recommending anything. This was more useful than a direct recommendation because the constraints interact in non-obvious ways. The “simplest” option (Tailscale-only) would have meant losing LAN access without the VPN, which broke an unstated requirement. The structured options format made it easy to catch that.

For the self-signed CA chain, Claude produced the three-resource sequence (selfsigned-issuer → Certificate → CA ClusterIssuer) correctly on the first attempt, which reflects that cert-manager’s bootstrapping pattern is well-represented in its training data.

Where it needed correction: During the AFFiNE deployment (a collaborative knowledge base app added later in the series), I needed WebSocket support in the nginx ingress. Claude initially suggested using nginx.ingress.kubernetes.io/configuration-snippet to inject the Upgrade and Connection headers. This failed silently. The annotation was accepted by kubectl but had no effect. The admission webhook in ingress-nginx 1.9+ blocks configuration-snippet by default as a security measure, and because the webhook was disabled in this setup, the annotation was simply ignored rather than rejected with an error.

The correct approach is nginx.ingress.kubernetes.io/proxy-set-headers, which references a ConfigMap in the same namespace as the Ingress. The ConfigMap keys are header names and the values are nginx variables:

# In the Ingress
annotations:
  nginx.ingress.kubernetes.io/proxy-set-headers: "affine/websocket-headers"

# ConfigMap in the same namespace
apiVersion: v1
kind: ConfigMap
metadata:
  name: websocket-headers
  namespace: affine
data:
  Upgrade: "$http_upgrade"
  Connection: "upgrade"

The value of proxy-set-headers is <namespace>/<configmap-name>. This isn’t prominently documented, and Claude didn’t suggest it unprompted. I found it in the ingress-nginx documentation after the configuration-snippet path failed.

Prompt that worked: “The configuration-snippet annotation isn’t working for WebSocket headers. What’s the alternative approach in ingress-nginx that doesn’t require configuration-snippet?”

Using a different AI tool? The TLS options conversation works well with any AI that can reason about constraints. The key is to state your constraints explicitly before asking for a recommendation: “I cannot open ports to the internet, I need LAN access without VPN, and Tailscale is already deployed. What are my TLS options?” Vague questions get recommendations that fit the general case, not your case.

Lessons

  • The platform layer has a strict ordering. CRDs must exist before resources that consume them. Build your Flux dependency chain to match: controllers first, configs second, applications last. Violating this ordering gives you cryptic reconciliation errors in the wrong layer.

  • L2 mode MetalLB requires disabling servicelb first, not after. Deploying MetalLB while K3s’s built-in servicelb is still running causes silent IP assignment conflicts that are difficult to trace. Make --disable=servicelb part of the K3s provisioning step, not a post-hoc cleanup.

  • Stating your constraints before asking for a TLS recommendation gets a better answer. “No public exposure, Tailscale already deployed, need LAN access without VPN” leads to the self-signed CA + Tailscale dual-issuer approach immediately. Asking “what TLS should I use?” leads to Let’s Encrypt, which doesn’t fit the constraint at all.

  • The dual ingress pattern scales. Two Ingress resources per application sounds like overhead, but it’s a repeatable template. Once you have the pattern down, adding a new application’s ingress layer takes five minutes. The operational cost of two files per app is negligible. The benefit of reliable LAN and remote access to every service is substantial.

  • configuration-snippet is blocked in ingress-nginx 1.9+. If you need to inject custom nginx proxy headers, use proxy-set-headers with a ConfigMap. Don’t spend time debugging why your configuration-snippet annotation has no effect. It’s almost certainly being silently blocked by the admission webhook policy.

Next: Post 5 — Observability Before Applications — why you instrument the cluster before deploying anything else.