edkarlsson
GitHubGet in touch

Homelab as Production/Part 10 of 16

Day-2 Operations — HA, Backup, and Rolling Upgrades

The unglamorous work that makes infrastructure actually reliable

There’s a version of this homelab that runs. I had that version approximately three weeks into the project. Flux was reconciling. Applications were deploying. Dashboards were green.

Then I asked myself a question that changes everything: what happens when something breaks?

Not “what happens when I fix it manually,” but what happens automatically, while I’m asleep, before I even know it broke? That question is the dividing line between infrastructure that runs and infrastructure you can trust. Day-2 is where that line lives. It covers the unglamorous work: fault tolerance, automated backup, tested failover, and rolling upgrades that don’t take everything down at once.

Having HA is something I didn’t necessarily need, but I was getting to the point where “why not do this?” was an option.

This post covers the three day-2 concerns I spent the most time on: PostgreSQL high availability with automatic VIP failover, a backup strategy that follows failover automatically, and K3s rolling upgrades across minor versions. Each one has at least one failure story embedded in it, because that’s how I learned them.

PostgreSQL High Availability

My K3s cluster uses PostgreSQL as its external datastore rather than the embedded etcd that K3s ships with by default. The K3s HA documentation explains both options. I chose external PostgreSQL for two reasons: I was already running a PostgreSQL instance for application databases, and a single PostgreSQL with streaming replication is operationally simpler to reason about than a three-node etcd quorum.

The architecture is a two-VM cluster: one primary and one standby. Both run PostgreSQL. The primary handles all writes. The standby receives a continuous stream of WAL (Write-Ahead Log) records via async streaming replication and replays them. Every consumer, including K3s, Terraform remote state, Keycloak, and all application databases, connects exclusively to a floating VIP managed by keepalived. The VIP lives on the primary under normal conditions, and migrates automatically to the standby when the primary fails.

graph TD
    VIP["VIP: 10.0.0.44<br/>keepalived VRRP · virtual_router_id 44"]

    subgraph PRI["node-01 — VM 520 · 10.0.0.45"]
        PRI_KA["keepalived MASTER<br/>priority 100"]
        PRI_PG["PostgreSQL Primary<br/>read/write"]
        PRI_KA --- PRI_PG
    end

    subgraph STB["node-05 — VM 521 · 10.0.0.46"]
        STB_KA["keepalived BACKUP<br/>priority 90"]
        STB_PG["PostgreSQL Standby<br/>hot standby · read-only"]
        STB_KA --- STB_PG
    end

    VIP -->|"normally routes to"| PRI
    VIP -.->|"failover within ~15s"| STB
    PRI_PG -->|"streaming WAL replication<br/>async"| STB_PG
    PRI_KA <-->|"VRRP unicast"| STB_KA

    CLIENTS["Consumers<br/>K3s · Terraform · Apps<br/>(connect to VIP only)"]
    CLIENTS --> VIP

    style VIP fill:#d97706,color:#fff
    style PRI_KA fill:#16a34a,color:#fff
    style PRI_PG fill:#15803d,color:#fff
    style STB_KA fill:#6b7280,color:#fff
    style STB_PG fill:#4b5563,color:#fff
    style CLIENTS fill:#0f172a,color:#fff

The VIP is the central abstraction. Consumers never need to know which physical VM is the primary. When the primary dies, the VIP moves, and clients reconnect to the same address they’ve always used.

The Keepalived Health Check — A Subtle Design Trap

Keepalived decides which node holds the VIP through a priority election. Each node advertises its effective priority via VRRP. The health check script modifies that effective priority: if the health check fails, the node’s priority drops by a configured weight.

In my setup:

  • Primary starts at priority 100, standby at priority 90
  • The health check weight is -20
  • If the primary’s health check fails, its effective priority drops to 80 (100 - 20)
  • The standby’s priority of 90 wins the election
  • The VIP migrates

The health check script is deliberately minimal:

#!/bin/bash
pg_isready -q || exit 1
exit 0

pg_isready checks whether PostgreSQL is accepting connections. Nothing more. This one-liner is doing important work by not doing more.

Here’s the trap: it might seem useful to also check pg_is_in_recovery(), to verify that the node claiming MASTER status is actually a primary and not a standby. Claude’s initial design went down this path. The logic seemed reasonable: the health check verifies the node is a real primary, so the BACKUP only wins the election when the primary is definitively gone.

The problem is a deadlock. The standby is always in recovery. That’s what makes it a standby. If the health check tests pg_is_in_recovery() and fails when the result is t (true), then the standby’s health check always fails. Its effective priority is always 90 - 20 = 70. Even when the primary is completely dead, the standby’s effective priority (70) is lower than the failed primary’s effective priority (80). The VIP never moves.

The right design: pg_isready in the health check, pg_is_in_recovery() in the notify script, which is the script that runs after the VIP is acquired, to decide whether promotion is needed. The health check answers “is PostgreSQL running?” The notify script answers “is it already a primary, or does it need to be promoted?”

Failover Timing and Sequence

With the current configuration, total failover takes approximately 15 seconds. The breakdown:

Phase Duration What happens
Health check failure detection ~10s 2 failed checks at 5s intervals
VRRP advertisement timeout ~3s 3 missed advertisements at 1s interval
PostgreSQL promotion ~2s WAL replay completion + timeline switch

The full sequence:

sequenceDiagram
    participant C as Consumers
    participant V as VIP 10.0.0.44
    participant P as Primary .45 (MASTER)
    participant S as Standby .46 (BACKUP)

    Note over P,S: Normal operation — Primary holds VIP

    C->>V: connect
    V->>P: route
    P-->>S: streaming WAL replication

    Note over P: PostgreSQL stops (crash / maintenance)

    loop Health check fails ×2 (~10s)
        P->>P: pg_isready → fails
    end
    P->>P: priority drops: 100 - 20 = 80

    loop VRRP advertisements missed ×3 (~3s)
        P--xS: no VRRP advert
    end

    S->>S: wins election (priority 90 > 80)
    S->>V: acquire VIP
    S->>S: notify script: pg_is_in_recovery() = true → PROMOTE
    S->>S: pg_ctlcluster promote (~2s)

    Note over S: Standby is now Primary
    C->>V: connect (same address)
    V->>S: route to promoted node

Fifteen seconds of write unavailability is acceptable for a homelab. The applications mostly use connection retry logic, and the databases that are most write-sensitive (K3s datastore, Keycloak) tolerate brief interruptions gracefully.

What actually happens to in-flight write transactions during that window depends on the application. Connections drop when the primary’s PostgreSQL process stops, so any application that was mid-write will see a connection error. Applications then reconnect to the same VIP address and find the promoted standby ready to accept writes. K3s uses etcd-compatible retry semantics and tolerates a 15-second datastore interruption without losing cluster state. Applications using a PostgreSQL connection pool (PgBouncer, SQLAlchemy, JDBC) will see a brief connection error and retry transparently if retry_on_error is configured. Applications that don’t have retry logic will surface the error to the user. The operational question to answer for each stateful application, before considering the HA setup production-ready, is whether its connection pool retries transparently or whether a connection drop surfaces as an error to end users.

The pg_hba.conf Gotcha (The One I Hit After a Failover)

Streaming replication copies data pages. It does not copy configuration files.

pg_hba.conf is a configuration file.

I learned this empirically. I had added a new database and a corresponding user about a week before doing a planned maintenance failover. The primary accepted connections from the new application pod without issues. When I triggered failover and the standby was promoted, the new application started throwing authentication errors. The promoted standby didn’t have the matching pg_hba.conf entry, because pg_hba.conf changes aren’t streamed with WAL. I had edited the primary and never applied the same change to the standby.

The fix is mechanical: any time you edit pg_hba.conf on the primary, you must apply the same change on the standby and reload:

# After editing pg_hba.conf on the primary (10.0.0.45):
# SSH to standby and apply the same change
ssh k3sadmin@10.0.0.46 "sudo pg_ctlcluster 16 main reload"
# Or more explicitly:
ssh k3sadmin@10.0.0.46 "sudo -u postgres psql -c 'SELECT pg_reload_conf();'"

This isn’t a failure of PostgreSQL replication design. pg_hba.conf is intentionally not replicated, because you might want different access rules on the standby. But in practice, for a homelab HA setup where the standby must be a drop-in replacement for the primary, you want them synchronized. Make it a habit: any pg_hba.conf edit on one node means the same edit on the other node within the same session.

Backup Automation with the VIP Guard Pattern

The backup cron job is deployed to both nodes, primary and standby. It runs on both, every six hours. But only one of them actually does any work.

The backup script opens with a VIP guard:

# VIP guard: only back up on the current primary
VIP="10.0.0.44"
if ! ip addr show | grep -q "${VIP}"; then
    log "Not VIP holder. Skipping backup."
    exit 0
fi

If the VIP isn’t assigned to the local interface, the script exits immediately. Only the current VIP holder, whichever node is the active primary, proceeds to run pg_dump across all eight databases and sync the results to NFS.

Claude designed this pattern. The problem it solves is failover awareness in backup automation. If you run backups only on the primary’s static IP and the primary fails, backups stop until someone manually reconfigures the cron job. With the VIP guard pattern, backups resume automatically on the next cron cycle after failover, because the promoted standby now holds the VIP.

The current configuration backs up every six hours, retains seven days of local dumps at /var/backups/postgresql/, and syncs to a Synology NAS with thirty days of retention. Each backup run also writes a machine-readable status file:

# Check backup health on whichever node holds the VIP
ssh k3sadmin@10.0.0.44 "cat /var/backups/postgresql/backup-status.json | python3 -m json.tool"

The NFS mount uses nofail,soft,timeo=30,retrans=3. This is important. Without nofail, a boot with the NAS offline causes the mount to block and delay the node’s startup indefinitely.

Testing the Restore

A backup you’ve never restored is a backup you can’t trust. This is not a philosophical point. It’s a practical one: the failure mode of an untested backup isn’t “the restore takes longer than expected.” It’s “the restore doesn’t work at all, and you discover this during an actual incident.”

The basic restore test for PostgreSQL is straightforward. Copy the dump to a test machine (or the standby VM, which works well since it already has PostgreSQL running):

# On the standby or a test machine — restore a single database
psql -U postgres -d postgres -c "CREATE DATABASE myapp_test;"
psql -U postgres -d myapp_test < /var/backups/postgresql/myapp_$(date +%Y-%m-%d).sql

# Verify the row count on a key table
psql -U postgres -d myapp_test -c "SELECT COUNT(*) FROM users;"

For pg_dumpall archives (which back up all databases plus global objects), the restore is:

psql -U postgres -f /var/backups/postgresql/all_databases_$(date +%Y-%m-%d).sql postgres

For K3s specifically, “verified” means checking that the K3s datastore tables exist and have data:

psql -U postgres -d kubernetes -c "\dt"
psql -U postgres -d kubernetes -c "SELECT COUNT(*) FROM kine;"

The kine table is K3s’s key-value store. A successful restore shows row count greater than zero. An empty table means K3s would start with no cluster state.

After confirming the row counts, verify that an application can actually connect: run a test query against one application’s database from an application pod, or temporarily point an application at the test restore and confirm it starts without errors.

This belongs in the same session as the backup automation, not deferred to “later.” Later never comes, and later is exactly when you need to know the backup works.

The restore tests above cover a single database. The cluster-level disaster recovery scenario is different: if the K3s cluster itself is unrecoverable, all nodes lost, the GitOps repository is the recovery artifact. Restore PostgreSQL from the most recent backup, reinstall K3s pointing at the same PostgreSQL VIP, and let Flux reconcile the repository against the empty cluster. Within 10 to 15 minutes, the cluster state described in the repository is restored. This is what “the repository is the source of truth” means operationally, not just that changes go through Git, but that Git is sufficient to recreate the running system from nothing.

K3s Rolling Upgrade: The 4-Hop Strategy

K3s follows Kubernetes version skew policy: you can’t skip more than two minor versions in a single upgrade. Going from v1.28 to v1.32 in a single step isn’t supported. I did it in four sequential hops:

v1.28.3+k3s1 → v1.29.12+k3s1 → v1.30.10+k3s1 → v1.31.8+k3s1 → v1.32.12+k3s1

Each hop is a separate playbook run with the target version passed explicitly. The K3s manual upgrade documentation covers the procedure. The key constraints:

  1. Upgrade servers before agents. Agents must not run a newer version than the servers they’re connected to.
  2. Upgrade one node at a time. serial: 1 in Ansible prevents concurrent upgrades that could cause split-brain.
  3. Drain before upgrading, uncordon after the node reports Ready.

The upgrade playbook structure follows this pattern for each node:

- name: "Upgrade: K3s server nodes (serial)"
  hosts: k3s_servers
  become: true
  serial: 1
  tasks:
    - name: Drain this server node
      ansible.builtin.command: >
        k3s kubectl drain {{ inventory_hostname }}
        --ignore-daemonsets
        --delete-emptydir-data
        --timeout=120s
      delegate_to: "{{ groups['k3s_servers'][0] }}"
      when: k3s_version not in node_current_version.stdout

    - name: Run K3s installer with target version
      ansible.builtin.shell: |
        INSTALL_K3S_VERSION={{ k3s_version }} /tmp/k3s-install.sh server
      when: k3s_version not in node_current_version.stdout

    - name: Wait for node to report Ready
      ansible.builtin.command: >
        k3s kubectl get node {{ inventory_hostname }}
        -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}'
      register: node_ready
      until: node_ready.stdout == "True"
      retries: 30
      delay: 10

    - name: Uncordon this server node
      ansible.builtin.command: >
        k3s kubectl uncordon {{ inventory_hostname }}
      delegate_to: "{{ groups['k3s_servers'][0] }}"
      when: k3s_version not in node_current_version.stdout

    - name: Pause 30s before next server
      ansible.builtin.pause:
        seconds: 30
      when: k3s_version not in node_current_version.stdout

The agents phase is identical in structure, using k3s-agent instead of k3s for the systemd service name and passing K3S_URL and K3S_TOKEN to the installer.

The --forks=1 flag on the ansible-playbook command is also required, not just serial: 1. With 1Password’s SSH agent, parallel connections trigger simultaneous approval dialogs that are difficult to manage quickly enough. With --forks=1, each SSH connection is fully sequential.

Each hop took roughly 30-40 minutes across the full 8-node cluster (3 servers + 5 agents), with the 30-second pause between nodes as a stabilization buffer. Four hops across two days, with a cluster health check between each hop.

I Tested the Failover

This section exists because most people skip it.

After building the keepalived configuration and verifying replication was streaming, I stopped PostgreSQL on the primary:

ssh k3sadmin@10.0.0.45 "sudo systemctl stop postgresql"

Then I waited. Fifteen seconds later, the standby had promoted itself. The VIP had moved. I ran pg_isready -h 10.0.0.44 and got a successful response. I connected to the promoted standby and confirmed SELECT pg_is_in_recovery() returned f. I watched the application logs, saw a few connection timeout retries, then normal operation resumed.

Untested failover is not high availability. It’s the appearance of high availability. The actual test matters because it validates the specific combination of your keepalived timing parameters, your notify script, your application retry logic, and your network configuration. Testing it once isn’t the same as running it in production under real load, but it’s infinitely better than discovering the failure mode during an actual incident at 2am.

After the test, I restored the original primary as a standby via pg_basebackup from the promoted node. PostgreSQL promotion is irreversible. You can’t simply restart the old primary and have it join as a standby. You re-initialize it:

sudo -u postgres pg_basebackup \
  -h 10.0.0.46 \
  -U replicator \
  -D /var/lib/postgresql/16/main \
  -Fp -Xs -P \
  --checkpoint=fast

The --checkpoint=fast flag is important. Without it, pg_basebackup waits for PostgreSQL to reach the next scheduled checkpoint, which can take up to five minutes.

AI Collaboration Note

What Claude contributed: The VIP guard pattern for backup automation, specifically the insight that installing the cron job on both nodes but having the script check VIP ownership before executing makes backup automation automatically failover-aware without any external orchestration. This was a clean, self-contained design that I wouldn’t have arrived at as quickly on my own.

Where it needed correction: Two places. First, the initial keepalived health check design included a pg_is_in_recovery() check, which would have permanently broken failover (the standby can never win the VRRP election if its health check always fails). I caught this by tracing the math: standby effective priority = 90 - 20 = 70, which is always less than the failed primary’s 80. Second, after initial deployment, I hit the pg_hba.conf replication gap during a planned maintenance failover. Connections from one application were rejected on the promoted standby because the pg_hba.conf entry had only been applied to the primary. Claude hadn’t flagged this as a gotcha during the initial build.

Prompt that worked: "Walk me through the keepalived health check design for PostgreSQL HA. What are the failure modes if the health check checks pg_is_in_recovery() instead of only pg_isready?"

Using a different AI tool? The VIP guard pattern is a general concept. Any AI assistant can reason about it if you frame the problem as “I need a cron job that only runs on whichever of two nodes is currently active, and I need it to follow failover automatically without reconfiguration.” The key is presenting the constraint explicitly rather than asking for a generic backup solution.

Lessons

  • Day-2 is where reliability is built, not where it’s declared. Deploying a service isn’t the same as operating a service. HA, backup, and upgrade procedures aren’t features to add later. They shape the architecture from the start.

  • The keepalived health check must only answer “is PostgreSQL accepting connections?” and nothing else. Any additional check that the standby would structurally fail, like checking recovery status, creates a deadlock that silently breaks failover.

  • pg_hba.conf is the configuration file most likely to bite you during a failover. The failure is silent until the standby promotes: your application authenticates fine against the primary, then fails authentication against the promoted standby because the pg_hba.conf edit never made it there. Make editing both nodes in a single session a habit, not a reminder.

  • K3s upgrades require sequential hops across minor versions. Plan for four playbook runs to cross four minor versions, not one. The serial: 1 directive in Ansible and --forks=1 at the CLI level both matter. One controls play execution, the other controls connection concurrency.

  • Untested failover is an optimistic assumption, not a safety guarantee. The fifteen-second failover only works as described if your specific combination of keepalived timing, notify script behavior, application retry logic, and network configuration all behave as expected together. Test it once on purpose to find out if they do.

Next: Post 11 — CI/CD: Validating Infrastructure as Code — every PR goes through a pipeline. Here’s what that pipeline checks.