edkarlsson
GitHubGet in touch

Homelab as Production/Part 3 of 16

GitOps from Day One: Bootstrapping FluxCD

Why you set up GitOps before deploying any applications

Here is a mistake almost everyone makes: they get their Kubernetes cluster running, open a terminal, and start typing kubectl apply -f. It feels productive. Things appear. Then, six months later, nobody, including the person who ran those commands, can reconstruct exactly what is running or why. The cluster has drifted from any written record. Debugging means reading live cluster state and hoping it matches the intent. Rolling back means hoping your shell history is intact.

I didn’t do this. Not because I’m disciplined by nature, but because the discipline was baked into the workflow from the start: no application goes into the cluster unless it goes through Git first.

Getting FluxCD to work required understanding layers I hadn’t had to think about before: project structure, dependency ordering, the distinction between a “Flux file” and an application manifest. I tried to deploy it myself before but always got stuck on the details. This was where having an AI co-pilot stopped being a curiosity and started being load-bearing.

This post covers how I bootstrapped FluxCD before deploying a single application. It covers the prerequisite step (migrating Terraform remote state to PostgreSQL), the architectural decision about where Flux lives in the repository, the dependency chain that ensures things deploy in the right order, and a couple of real bugs that taught me how Flux actually works under pressure.

The Prerequisite: Terraform Remote State on PostgreSQL

Before FluxCD could be bootstrapped, there was a chicken-and-egg problem to solve.

Flux reads from a Git repository. To bootstrap Flux, I needed a cluster. To provision the cluster with Terraform, I needed a state backend that would survive beyond my local machine. Local .tfstate files are a liability in any collaborative or long-lived setup. Lose the laptop, lose the state.

The solution was to migrate Terraform state to a PostgreSQL backend before touching Flux at all. My homelab already had a PostgreSQL VM (provisioned in the previous session), so the migration was straightforward:

# infrastructure/backend.tf
terraform {
  backend "pg" {
    conn_str = "postgres://terraform:password@10.0.0.44/terraform_state"
  }
}

Migrating existing local state to the remote backend takes one command:

terraform init -migrate-state

Terraform detects the backend change, asks for confirmation, and copies all existing state to PostgreSQL. From that point forward, terraform plan and terraform apply read and write from the database, not the filesystem.

This matters for GitOps because it means my infrastructure state is now as durable as my PostgreSQL cluster. I can run Terraform from any machine, including CI runners, without copying state files around.

With remote state in place, Flux bootstrapping could proceed.

The Repository Structure Decision

Before running flux bootstrap, there’s a decision to make that shapes everything downstream: where does Flux live in the repository?

Three options exist:

Option Location Problem
A infrastructure/flux/ Mixes Terraform (VM provisioning) with Flux (K8s orchestration): different tools, different layers
B kubernetes/flux/ Circular. Flux lives inside the directory it manages; path references become awkward
C clusters/homelab/ Clean separation: each top-level directory has a single responsibility

Option C follows the official Flux community pattern and the flux2-kustomize-helm-example reference repository. Claude surfaced this recommendation when I described the repo layout, explaining the trade-offs with enough specificity that Option C was clearly correct.

The key insight is that the three directories serve three completely different concerns:

  • infrastructure/ answers WHERE: Terraform provisions VMs, networks, and Proxmox resources
  • kubernetes/ answers WHAT: Kubernetes manifests describe desired cluster state
  • clusters/homelab/ answers HOW: Flux Kustomizations control what gets deployed, in what order, with what intervals
graph LR
    subgraph WHERE["WHERE — infrastructure/"]
        TF["Terraform<br/>VMs · Networks<br/>Cloud-init · Proxmox"]
    end
    subgraph WHAT["WHAT — kubernetes/"]
        K8S["K8s Manifests<br/>HelmReleases<br/>Namespaces · ConfigMaps"]
    end
    subgraph HOW["HOW — clusters/homelab/"]
        FLUX["Flux Kustomizations<br/>Ordering · Dependencies<br/>Intervals · Pruning"]
    end

    TF -->|"provisions cluster"| K8S
    FLUX -->|"reconciles"| K8S

    style WHERE fill:#f3f4f6,stroke:#9ca3af
    style WHAT fill:#eff6ff,stroke:#93c5fd
    style HOW fill:#f0fdf4,stroke:#86efac

Each layer can be understood in isolation. A Terraform operator can work in infrastructure/ without knowing anything about Flux. An application developer can work in kubernetes/apps/ without knowing how Flux Kustomizations are structured. A platform engineer working in clusters/homelab/ controls deployment order and scheduling without touching application manifests.

This separation also scales naturally. Even though there’s currently one cluster, the layout is ready for a second:

clusters/
├── homelab/        ← Production (current)
└── staging/        ← Future staging cluster (same kubernetes/ manifests, different config)

Same kubernetes/ manifests, different clusters/ entry point. No restructuring required later.

The Bootstrap Command

With the directory structure decided, bootstrapping Flux is a single command:

flux bootstrap github \
  --owner=<your-github-org> \
  --repository=homelab-iac \
  --branch=main \
  --path=clusters/homelab \
  --personal

See the Flux bootstrap for GitHub documentation for full options and token scoping requirements. The GitHub PAT used here goes into 1Password immediately. It’s not left in environment variables or shell history.

What this command does:

  1. Installs the Flux controllers (source-controller, kustomize-controller, helm-controller, notification-controller) into the flux-system namespace
  2. Creates a GitRepository resource pointing to the GitHub repository
  3. Creates a root Kustomization that watches clusters/homelab/
  4. Commits the generated manifests (gotk-components.yaml, gotk-sync.yaml, kustomization.yaml) to clusters/homelab/flux-system/ in the repository

After bootstrap, the clusters/homelab/ directory looks like this:

clusters/homelab/
├── flux-system/
│   ├── gotk-components.yaml     ← Flux CRDs + controller Deployments (~560KB, auto-managed)
│   ├── gotk-sync.yaml           ← GitRepository + root Kustomization
│   └── kustomization.yaml       ← Kustomize config for flux-system resources
├── platform.yaml                ← Kustomization → ./kubernetes/platform/{controllers,configs}
├── apps.yaml                    ← Kustomization → ./kubernetes/apps
└── monitoring.yaml              ← Kustomization → ./kubernetes/platform/monitoring/...

The flux-system/ directory is managed by Flux itself. The other files, platform.yaml, apps.yaml, monitoring.yaml, are the ones I write. They’re Flux Kustomization resources that point Flux at specific directories in the repository.

The Dependency Chain

The most important design choice after bootstrap is the dependency chain between Kustomizations. Flux reconciles resources concurrently by default, which would mean cert-manager CRDs might not exist when a ClusterIssuer tries to apply, or the External Secrets Operator might not be running when an ExternalSecret tries to sync.

The solution is explicit dependsOn declarations combined with wait: true. Flux won’t reconcile a downstream Kustomization until the upstream one reports all resources as Ready.

flowchart TD
    FS["flux-system<br/>(self-managing)"]

    PC["platform-controllers<br/>cert-manager · ingress-nginx · ESO · MetalLB · Longhorn"]
    PCfg["platform-configs<br/>ClusterIssuers · ClusterSecretStore · MetalLB pools"]
    APPS["apps<br/>Application HelmReleases"]

    MC["monitoring-controllers<br/>kube-prometheus-stack · Loki · Promtail"]
    MCfg["monitoring-configs<br/>Dashboards · PodMonitors · alerts"]

    FS --> PC
    PC -->|"wait: true — CRDs must exist"| PCfg
    PCfg -->|"wait: true — ingress + certs + secrets ready"| APPS

    FS --> MC
    MC --> MCfg

    style FS fill:#6b7280,color:#fff
    style PC fill:#2563eb,color:#fff
    style PCfg fill:#1d4ed8,color:#fff
    style APPS fill:#16a34a,color:#fff
    style MC fill:#7c3aed,color:#fff
    style MCfg fill:#6d28d9,color:#fff

The chain works as follows:

  • flux-system bootstraps itself and starts watching the Git repository
  • platform-controllers deploys cert-manager, ingress-nginx, MetalLB, the External Secrets Operator, Longhorn, and the Tailscale operator. wait: true means Flux blocks here until all controller pods report Ready and all CRDs are registered.
  • platform-configs deploys ClusterIssuers, ClusterSecretStore, and MetalLB IP pools. These are resources that consume the CRDs created in the previous step, and dependsOn: platform-controllers enforces the ordering.
  • apps deploys application workloads. dependsOn: platform-configs ensures ingress, TLS issuance, and secret syncing are all available before any application tries to use them.
  • monitoring-controllers and monitoring-configs run in parallel with the platform chain but follow the same internal ordering pattern.

The Key YAML: platform.yaml

Here is the actual clusters/homelab/platform.yaml file that encodes this chain:

---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: platform-controllers
  namespace: flux-system
spec:
  interval: 1h
  retryInterval: 1m
  timeout: 5m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./kubernetes/platform/controllers
  prune: true
  wait: true
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: platform-configs
  namespace: flux-system
spec:
  dependsOn:
    - name: platform-controllers
  interval: 1h
  retryInterval: 1m
  timeout: 5m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./kubernetes/platform/configs
  prune: true

Notice that platform-configs has no wait: true. It depends on platform-controllers being fully Ready before it starts, but downstream Kustomizations (apps) enforce their own ordering separately via dependsOn: platform-configs.

The path values are relative to the repository root, not to the Kustomization file itself. This is because the GitRepository source clones the entire repository, and all paths resolve from the clone root. A common early mistake is writing paths relative to clusters/homelab/, which produces “path not found” errors from Flux.

All path values in Flux Kustomization spec.path fields resolve from the repository root.

AI Collaboration Note

AI Collaboration Note What Claude contributed: The clearest contribution here was the repository structure recommendation. When I described having Terraform in infrastructure/, Kubernetes manifests in kubernetes/, and being unsure where to put Flux, Claude walked through all three options with concrete trade-offs. It specifically called out the circular reference problem with Option B and the tooling-layer mixing problem with Option A. It also surfaced the community-standard pattern and linked to the official Flux repository structure guide, which I hadn’t read carefully enough.

The second contribution was catching a real bug. When reviewing reconciliation status after bootstrap, I noticed one Kustomization wasn’t applying its second resource. Claude asked to see the raw file contents, noticed that the kustomization.yaml file had two YAML documents separated by --- where the second document referenced a resource path that didn’t exist. Flux’s kustomize-controller parsed the first document successfully and silently ignored the second. No error, no warning, just a missing resource. The fix was a one-line path correction, but finding it required reading the file carefully rather than trusting the reconciliation status output.

Where it needed correction: The initial bootstrap command Claude suggested included --components-extra=image-reflector-controller,image-automation-controller for automated image updates. This sounded useful but added controllers I didn’t need yet, and image automation requires additional configuration to avoid unintended tag updates. I removed those flags and bootstrapped with the four standard controllers only.

Prompt that worked: "I have infrastructure/ for Terraform, kubernetes/ for manifests, and I need to decide where to put the Flux entry point. Walk me through the options with trade-offs — I want the community-standard approach."

Using a different AI tool? The structure decision prompt works with any LLM that has knowledge of FluxCD. The key is to describe your specific directory layout first, then ask for trade-off analysis rather than “what is the best option.” The latter tends to produce generic answers, while the former forces the model to reason about your actual constraints.

Flux Kustomization Atomicity

One property of Flux Kustomizations that caught me off guard is that reconciliation is atomic: if any single resource in a Kustomization fails server-side validation or apply, the entire Kustomization is blocked. No resources apply, even perfectly valid ones.

This has a direct consequence for how you structure your manifests. A single broken application can block every other application in the apps Kustomization from receiving updates. I hit this when a PVC manifest had an immutable field change. After the PVC was bound to a PV, the dynamically assigned volumeName became part of the spec and couldn’t be changed in-place. Flux’s dry-run detected the immutable field change and refused to apply anything in the apps Kustomization until the conflict was resolved.

See the Flux Kustomization API documentation for the full behavior specification.

Common triggers for atomicity blocks:

Trigger Symptom
PVC with volumeName mismatch after binding Dry-run fails with “spec is immutable”
PV with immutable field change (e.g., NFS path) Apply fails with “field is immutable”
CRD not yet registered “no matches for kind X”
Schema validation failure Validation error before apply

The practical implication for repository structure: keep things that can fail independently in separate Kustomizations. Platform controllers, platform configs, and applications are already separated. But within the apps Kustomization, all applications share the same atomic boundary. One broken app blocks all app updates until it’s fixed.

The mitigation, when you need to unblock quickly, is to temporarily comment out the broken resource from its kustomization.yaml file. Flux prunes the resource from the cluster and reconciliation proceeds. This is a legitimate escape hatch, not a workaround to be ashamed of. It’s faster than debugging under pressure, and the Git history records exactly when and why the resource was temporarily disabled.

When Flux gets stuck: recovery steps

When a Kustomization stops reconciling and you need to diagnose or unblock it quickly, these three commands cover most situations:

1. Find the blocking resource:

kubectl describe kustomization apps -n flux-system

The Status.Conditions section will show the last reconciliation error and which resource triggered it. This is almost always faster than scanning logs.

2. Force a reconciliation after fixing the resource:

flux reconcile kustomization apps --with-source

The --with-source flag forces Flux to re-fetch the Git source before reconciling, ensuring it picks up your fix rather than replaying a cached version.

3. Temporarily suspend a broken Kustomization to unblock others:

flux suspend kustomization apps
# fix the broken resource
flux resume kustomization apps

Suspending stops Flux from trying to reconcile the Kustomization entirely. Use this when one broken Kustomization is blocking others via dependsOn, and you need to let the healthy ones proceed while you debug.

These three commands belong in your runbook. If they’re not documented, you’ll be googling them at 11pm, and that’s exactly when you don’t want to be searching.

Important: If you’ve applied changes directly with kubectl apply or kubectl edit (bypassing GitOps), Flux will reconcile them away on the next sync, reverting your change. If you need a manual change to persist, either commit it to the repository first, or run flux suspend kustomization <name> to pause reconciliation while you work, then commit and resume.

Lessons

  • GitOps before applications, not after. Retrofitting GitOps onto a running cluster means reconciling drift between actual state and desired state across every resource you’ve already applied. Starting with Flux means your first application deployment is also your first GitOps deployment. The workflow is established from commit one.

  • Migrate Terraform remote state before bootstrapping Flux. They’re both infrastructure concerns, and you don’t want to bootstrap a cluster with local state that’s one laptop failure away from being unrecoverable. PostgreSQL backend is a natural choice if you already have a PostgreSQL VM.

  • The three-directory split (infrastructure/, kubernetes/, clusters/) pays off when you delegate. When you hand a session scope to an AI, or a colleague, the directory structure tells them exactly where to look and what not to touch. Separation of concerns is also separation of context.

  • dependsOn plus wait: true is not optional. Without it, Flux applies Kustomizations concurrently, and the race conditions are subtle. Resources apply successfully individually but fail at runtime because a CRD isn’t yet registered or a controller isn’t yet healthy. The dependency chain encodes the ordering that would otherwise live only in your head.

  • Flux Kustomization atomicity shapes your manifest structure. Know that one failing resource blocks the entire Kustomization. Group resources so that failures are isolated to the smallest possible blast radius. Keep the escape hatch, commenting out broken resources, in mind when you’re debugging under pressure.

Next: Post 4 — The Platform Layer Nobody Talks About — MetalLB, ingress-nginx, cert-manager, and NFS: the boring infrastructure every application depends on.