CI/CD — Validating Infrastructure as Code

The incident that finally convinced me to build a real CI pipeline was embarrassingly simple. I had been iterating on a Kubernetes HelmRelease and made a small formatting mistake, a missing spec.values indentation level. The manifest was valid YAML; it just wasn’t a valid HelmRelease. I opened a PR, merged it quickly because “it’s just a values tweak,” and within two minutes Flux was stuck:

HelmRelease/homelab/my-app: failed to install Helm release: error unmarshaling JSON...

The reconciliation for the entire apps Kustomization stalled. Flux is atomic. One broken resource in a Kustomization blocks everything in that Kustomization. Three other apps that had nothing to do with my change stopped reconciling. I spent fifteen minutes figuring out what had even changed.

As the project grew more complex, where a small indentation mistake could cascade into a cluster-wide reconciliation stall, I realized I needed a real CI/CD pipeline. This was also the point where I started thinking about leveraging Codex, Gemini, and Copilot on GitHub as automated reviewers. Always automated, always running before merge.

The fix took thirty seconds. The incident cost forty-five minutes. That math isn’t acceptable for something a schema validator would have caught in under a second.

Infrastructure as Code deserves the same CI discipline as application code. The blast radius of a bad merge is potentially worse: a broken Kubernetes manifest can take down GitOps reconciliation for every app in a Kustomization. Invalid Terraform can wipe resources. A broken Ansible playbook runs against all cluster nodes simultaneously. The cost of a CI pipeline is a few hundred lines of YAML. The cost of a bad merge is measured in incident time.

Here’s the pipeline I built, what it checks, and the failures I had to debug along the way.

The GitHub Actions Pipeline

The pipeline lives in .github/workflows/ci-testing.yml and triggers on every pull request and push to main. The first thing it does is figure out what actually changed:

name: CI - Validation and Testing

on:
  pull_request:
    branches: [main]
  push:
    branches: [main]
  workflow_dispatch:
    inputs:
      run_homelab_smoke:
        description: "Run optional self-hosted homelab smoke tests"
        required: false
        type: boolean
        default: false

concurrency:
  group: ci-${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

jobs:
  changes:
    name: Detect Changed Areas
    runs-on: ubuntu-latest
    outputs:
      terraform: ${{ steps.filter.outputs.terraform }}
      ansible: ${{ steps.filter.outputs.ansible }}
      kubernetes: ${{ steps.filter.outputs.kubernetes }}
      scripts: ${{ steps.filter.outputs.scripts }}
    steps:
      - uses: actions/checkout@v4
      - name: Filter paths
        id: filter
        uses: dorny/paths-filter@v3
        with:
          list-files: csv
          filters: |
            terraform:
              - "infrastructure/**/*.tf"
            ansible:
              - "ansible/**/*.yml"
              - "ansible/**/*.yaml"
              - "ansible/**/*.j2"
            kubernetes:
              - "clusters/**/*.yaml"
              - "kubernetes/**/*.yaml"
            scripts:
              - "scripts/**/*.sh"

The path filtering matters. If I push a documentation change, the Terraform validation job doesn’t run. If I push a pure Kubernetes manifest change, Ansible syntax checks are skipped. This keeps CI fast and avoids burning runner minutes on checks that can’t possibly be affected by a given change.

The concurrency block cancels stale runs on the same branch. If I push two commits in quick succession, only the second one needs to complete.

Terraform Validation

When Terraform files change, three checks run in sequence:

Format check. terraform fmt -check -recursive ensures every .tf file is formatted to the canonical style. This is enforced pre-commit as well, so format failures in CI mean someone bypassed the hooks.

Init and validate. The pipeline discovers every directory containing .tf files and runs terraform init -backend=false followed by terraform validate on each. The -backend=false flag skips remote state initialization. CI doesn’t have access to the PostgreSQL backend, and it doesn’t need to. terraform validate checks syntax, provider references, and module inputs without touching real infrastructure.

tflint. TFLint catches issues that terraform validate misses: deprecated or invalid attribute names, unused variables, and provider-specific rule violations. Each Terraform directory gets its own tflint --init run so plugins are downloaded correctly.

- name: Validate all Terraform directories
  shell: bash
  run: |
    set -euo pipefail
    mapfile -t tf_dirs < <(find infrastructure -name '*.tf' -printf '%h\n' | sort -u)
    for dir in "${tf_dirs[@]}"; do
      echo "==> terraform init/validate in ${dir}"
      terraform -chdir="${dir}" init -backend=false -input=false -no-color >/dev/null
      terraform -chdir="${dir}" validate -no-color
    done

- name: TFLint
  shell: bash
  run: |
    set -euo pipefail
    mapfile -t tf_dirs < <(find infrastructure -name '*.tf' -printf '%h\n' | sort -u)
    for dir in "${tf_dirs[@]}"; do
      tflint --chdir "${dir}" --init
      tflint --chdir "${dir}"
    done

Ansible Validation

Ansible failures are particularly dangerous because playbooks run against all cluster nodes at once. A syntax error in the upgrade playbook, discovered during an actual upgrade, would be a bad time.

The Ansible job installs the required collections and then runs --syntax-check against every significant playbook:

- name: Install Ansible tooling
  run: |
    python -m pip install --upgrade pip
    pip install ansible-core ansible-lint
    ansible-galaxy collection install \
      ansible.posix \
      community.general \
      artis3n.tailscale \
      onepassword.connect \
      community.postgresql

- name: Ansible syntax checks
  run: |
    ansible-playbook -i ansible/inventory/k3s.yml \
      ansible/playbooks/k3s-cluster.yml --syntax-check
    ansible-playbook -i ansible/inventory/k3s.yml \
      ansible/playbooks/k3s-upgrade.yml --syntax-check \
      -e k3s_version=v1.32.12+k3s1
    ansible-playbook -i ansible/inventory/k3s.yml \
      ansible/playbooks/pg-backup.yml --syntax-check

After syntax checks, ansible-lint runs on changed files (or the full ansible/ tree on a full run). The lint profile is “production,” which catches things like become_user without become: true at the task level, relative src: paths in template tasks, and hardcoded IPs that should be variables.

The hardcoded collection list gotcha

This one caught me off guard. When I added community.postgresql to ansible/requirements.yml, my playbooks worked locally (because I had the collection installed), and they worked when I ran ansible-galaxy collection install -r requirements.yml manually. But CI kept failing with “collection not found.”

The reason: the ansible-galaxy collection install line in the GitHub Actions workflow is hardcoded. It does not read requirements.yml. When you add a new collection to your requirements file, you must also add it to the CI YAML manually. There’s no magic auto-discovery.

This is a known limitation and an easy one to miss. The fix is simple, just update the collection install line, but the failure mode is confusing because everything works locally.

Kubernetes Validation

This is the most important job in the pipeline. Two scripts run in sequence:

validate.sh

scripts/k8s/validate.sh is adapted from the FluxCD project’s own validation script. It does three things:

YAML syntax validation. Every .yaml file in the repository gets parsed by yq. A file that isn’t valid YAML fails here.
Flux CRD validation via kubeconform. kubeconform validates Flux custom resources (HelmRelease, Kustomization, GitRepository, etc.) against the Flux OpenAPI schemas downloaded from the FluxCD release. The schema version is read directly from the clusters/homelab/flux-system/gotk-components.yaml file, so it tracks whatever version of Flux is deployed.
Kustomize overlay build. Every kustomization.yaml in the repository is built with kustomize build, and the output is piped through kubeconform for Kubernetes schema validation. This is the check that would have caught my HelmRelease indentation mistake.

echo "INFO - Validating kustomize overlays"
find . -type f -name $kustomize_config -print0 | while IFS= read -r -d $'\0' file;
  do
    echo "INFO - Validating kustomization ${file/%$kustomize_config}"
    if ! kustomize build "${file/%$kustomize_config}" "${kustomize_flags[@]}" | \
      kubeconform "${kubeconform_flags[@]}" "${kubeconform_config[@]}"; then
      exit 1
    fi
done

Kubernetes Secret resources are explicitly skipped (-skip=Secret) because secrets in this repo use SOPS field references that would fail schema validation.

policy-check.sh

scripts/ci/policy-check.sh enforces a set of drift guardrails. Rather than failing the entire pipeline when it finds an existing violation (there were several when I first wrote it), it compares current findings against allowlists. Only new violations fail CI.

The checks enforced:

Check	Why it matters
Floating image tags (`:latest`, `*-latest`)	Unpinned images cause non-deterministic deployments and break reproducibility
Wildcard Helm chart versions (`*.x`)	Same problem for charts
`insecure-skip-tls-verify: true` in kubeconfig templates	Disables certificate validation, allowing MITM attacks
`StrictHostKeyChecking=no` in automation	Disables SSH host key verification
`write-kubeconfig-mode: 0644`	Makes kubeconfig world-readable on disk
`insecure = true` in Terraform providers	Disables TLS verification for provider API calls

Each policy has a corresponding allowlist in ci/allowlists/. When I need to temporarily allow an exception (during a migration, for example), I add the specific file+line reference to the allowlist rather than disabling the check globally. The allowlist is checked into git, so the exception is visible in review.

A real allowlist entry looks like this:

# ci/allowlists/latest-tags.txt
# Format: path/to/file.yaml:search-pattern
# Each line exempts a specific occurrence from the :latest check.
# Exempt the intentional :latest for the dev-only scratchpad image
kubernetes/apps/scratchpad/deployment.yaml:image: myuser/scratchpad:latest

The format is file:pattern, one entry per line. Comments explain why the exception exists. The history of exceptions in this file, visible via git log, is as informative as the policy itself.

The first time the pipeline caught something that would have caused a real incident, it was a :latest tag on the TeamCity build agent image. I had been iterating quickly on the agent configuration and had temporarily used :latest to avoid looking up the exact version string. The intent was to pin it before merging, the kind of thing you tell yourself you’ll fix and then don’t. The policy check blocked the PR. Hard stop, not a warning. I pinned the tag, pushed again, CI passed. That took thirty seconds. What it prevented was a rolling pod restart some weeks later when Docker Hub’s content behind that :latest tag changed and the new image had a dependency mismatch that broke the build pipeline. I don’t know exactly when that would have happened, but I know it would have. The pipeline is the automated half of the review loop. The first time it earns its keep isn’t when it confirms something was fine. It’s when it stops something that wasn’t.

check_policy "floating image tags" \
  "${found_latest}" "${LATEST_ALLOWLIST}" "${new_latest}"
check_policy "wildcard Helm chart versions" \
  "${found_helm}" "${HELM_ALLOWLIST}" "${new_helm}"
check_policy "kubeconfig insecure-skip-tls-verify" \
  "${found_insecure_tls_skip}" "${INSECURE_TLS_SKIP_ALLOWLIST}" "${new_insecure_tls_skip}"

This incremental enforcement approach means I can introduce the policy checks to a repo with existing violations without immediately breaking everything. The baseline gets allowlisted, new violations are blocked, and the allowlists shrink over time as I clean things up.

Shell Script Lint

ShellCheck runs on every changed .sh file. ShellCheck catches a class of bugs that are easy to miss when writing shell: unquoted variables, incorrect subshell handling, missing pipefail, and more. Scripts in scripts/_archive/ are excluded since deprecated scripts aren’t maintained and would generate too much noise.

The check runs incrementally on changed files rather than the whole scripts/ tree, which keeps it fast on PRs that only touch a single script.

The Claude Code Review Action

Beyond static validation, every PR gets an automated code review from Claude via the Claude Code Action. The review uses anthropics/claude-code-action@v1 with a detailed homelab-specific review prompt that checks for things the static tools can’t catch:

- name: Run Claude Code Review
  uses: anthropics/claude-code-action@v1
  with:
    claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
    prompt: |
      You are reviewing a pull request for a homelab IaC monorepo.
      The stack is: Proxmox (5 nodes) → K3s Kubernetes → 20+ apps via FluxCD.
      Secrets flow through 1Password Connect → ESO → Kubernetes Secrets.

      Review the PR diff and post a structured review comment.
      Use `gh pr diff ${{ github.event.pull_request.number }}` to get the diff.

The review prompt covers five categories:

Security. No hardcoded credentials. No TLS bypass. No relaxed SSH host key checking. No unapproved Terraform providers.

Kubernetes / Flux. No :latest image tags. No wildcard chart versions. ExternalSecret resources must use external-secrets.io/v1 (not the removed v1beta1). 1Password ESO property: references must use custom text field names, because the default Login fields (username, password) aren’t addressable by ESO. New apps should have dual ingress (nginx for LAN, Tailscale for remote). Flux dependency chain ordering must be respected.

Terraform. .tftpl files must escape bash ${...} syntax as $${...}. Terraform’s templatefile() parser treats unescaped ${ as a template expression, which breaks cloud-init scripts. VMs with initialization blocks should have lifecycle { ignore_changes = [initialization] }. No PROXMOX_VE_* environment variables, since they silently override the provider block.

Ansible. Rolling operations must use serial: 1. Tasks requiring root must have become: true. No hardcoded IPs that belong in group_vars.

Documentation. Version changes should update docs/reference/version-matrix.md. New gotchas should appear in docs/reference/technical-gotchas.md. New app deployments should follow the standard pattern.

The action posts a structured review comment with findings ordered from High to Medium to Low severity, plus a clear APPROVE / REQUEST CHANGES / COMMENT decision at the end. If it finds no issues, it approves with gh pr review --approve. If it finds High or Medium issues, it requests changes.

This is genuinely useful. The review catches things like a new ExternalSecret using the old v1beta1 API version, or a HelmRelease missing install.remediation.retries. Static schema validation confirms the YAML is structurally valid; the Claude review checks whether the content follows the project’s conventions.

Beyond Claude, the project also uses GitHub Copilot and Gemini Code Assist as PR reviewers. Here is what that looks like in practice on a real PR:

Gemini Code Assist automatically posting a structured review summary on a pull request, seconds after it was opened. Gemini Code Assist reviewing PR #131 unprompted. Summary of changes, bulleted highlights, specific file-level findings — all generated automatically.

GitHub Copilot’s independent PR overview for the same pull request. Copilot’s review of the same PR. Two AI reviewers, same files, independent passes. Findings overlapped on some items, diverged on others.

The author’s tracking comment: all 6 Gemini and Copilot findings resolved in a single follow-up commit. All 6 findings from both reviewers resolved and documented in one place before requesting re-review. The table format makes it auditable.

Re-review requested after all feedback items resolved. Codex hit its usage limit on this PR and didn’t post a review. The re-review request after all items resolved. Codex ran out of quota on this PR — a real limitation of free-tier AI tooling that’s worth knowing about.

The @claude On-Demand Action

The second Claude action is more interactive. The claude.yml workflow triggers on any PR comment or review comment that contains @claude, as long as the commenter is an owner, member, or collaborator:

on:
  issue_comment:
    types: [created]
  pull_request_review_comment:
    types: [created]

jobs:
  claude:
    if: |
      github.actor != 'dependabot[bot]' &&
      (github.event_name == 'issue_comment' &&
        contains(github.event.comment.body, '@claude') &&
        (github.event.comment.author_association == 'OWNER' ||
         github.event.comment.author_association == 'MEMBER'))

The action has actions: read permission, which means it can read the CI results for the current PR. So when CI fails and the cause isn’t obvious, I can leave a comment:

@claude the kubernetes_flux job is failing but the error is buried in the kubeconform output. What's wrong?

Claude reads the workflow logs, identifies the failing manifest and the schema error, and replies in the thread. This has saved me several rounds of log-reading for failures where the root cause was a few dozen lines deep in kubeconform output.

The Pipeline at a Glance

Here’s the full CI flow:

flowchart TD
    PR["Pull Request Opened<br/>(or push to main)"]
    FILTER["Detect Changed Areas<br/>dorny/paths-filter<br/>→ terraform / ansible / k8s / scripts"]

    subgraph PARALLEL["Parallel Validation (path-filtered)"]
        TF["Terraform<br/>fmt · init/validate · tflint"]
        ANS["Ansible<br/>syntax-check · ansible-lint"]
        K8S["Kubernetes + Flux<br/>validate.sh · kubeconform"]
        SH["Shell Scripts<br/>shellcheck -x"]
    end

    POLICY["Policy Checks<br/>policy-check.sh<br/>no :latest · no wildcards<br/>no insecure TLS · no StrictHostKeyChecking=no"]
    ARTIFACTS["Build CI Artifacts<br/>bundle.tar.gz + provenance.json"]
    REVIEW["Claude Code Review<br/>anthropics/claude-code-action@v1<br/>Security · K8s · Terraform · Ansible · Docs"]
    SUMMARY["CI Summary<br/>(branch protection gate)"]
    HUMAN["Human Review<br/>+ Merge Decision"]

    PR --> FILTER
    FILTER --> PARALLEL
    K8S --> POLICY
    PR --> ARTIFACTS
    PR --> REVIEW

    TF --> SUMMARY
    ANS --> SUMMARY
    POLICY --> SUMMARY
    SH --> SUMMARY
    ARTIFACTS --> SUMMARY
    REVIEW --> HUMAN
    SUMMARY --> HUMAN

    style PR fill:#1d4ed8,color:#fff
    style SUMMARY fill:#16a34a,color:#fff
    style REVIEW fill:#7c3aed,color:#fff
    style POLICY fill:#dc2626,color:#fff
    style HUMAN fill:#92400e,color:#fff

Pre-Commit Hooks: The First Line of Defense

CI catches issues before merge, but pre-commit hooks catch them before push. The repository uses three hooks that run automatically on git commit:

terraform_fmt: formats all .tf files
terraform_validate: validates Terraform syntax and provider references
terraform_tflint: runs TFLint on changed Terraform directories

If any hook fails, the commit is rejected. This means most Terraform issues are caught before they ever reach a PR. The pre-commit hooks are configured in .pre-commit-config.yaml and are installed with pre-commit install during initial setup.

The practical value: the CI Terraform job almost never fails because the pre-commit hooks have already caught any format or syntax issues locally. CI becomes a confirmation rather than a catch.

The GitLab CI Debugging Chain

Alongside the GitHub Actions pipeline, I built a parallel GitLab CI pipeline (.gitlab-ci.yml) as part of integrating GitLab as a secondary CI platform. What followed was seven PRs of iterative failures, each one revealing a new assumption that didn’t hold in the GitLab runner environment. It’s a useful case study in why CI pipelines need to be run, not just written.

PR #170: gcompat missing in Alpine Terraform image. The GitLab Terraform job used a lightweight Alpine-based Terraform image. The 1Password Terraform provider is compiled with CGO and links against glibc. Alpine uses musl libc. Result: exec format error on provider init. Fix: install gcompat (a glibc compatibility layer for musl) in the job’s before_script.

PR #175: git not installed in publish-nexus job. The artifact publishing script calls git rev-parse --short=7 HEAD to embed the commit SHA in the artifact version. The publish job used a minimal Docker image that didn’t include git. Fix: add git to the job’s package install list. The error message (git: command not found) was clear once you found it, but it was buried in the job output.

PR #176: Tailscale DNS not resolvable from K8s executor job pods. The Nexus publish job needed to reach the internal Nexus instance. Nexus is only accessible via Tailscale. The GitLab CI executor spawns job pods inside the cluster. Those pods can’t resolve Tailscale MagicDNS hostnames. They use the cluster’s CoreDNS, which has no knowledge of the Tailscale network. Fix: override NEXUS_URL in the job to use the internal Kubernetes service URL (nexus.nexus.svc.cluster.local) instead of the Tailscale FQDN.

PR #177: allow_failure: true does not prevent pending-forever. The sbom job (software bill of materials generation) was configured with allow_failure: true because it required a homelab shell runner that wasn’t yet registered. The expectation was that the job would fail gracefully. Instead, the job sat pending forever. It was waiting for a runner that would never appear, which blocked the pipeline from completing. allow_failure: true only applies to jobs that actually run and exit with a non-zero code. A job with no matching runner never runs. Fix: add when: manual to the job, which removes it from the automatic pipeline execution and requires explicit triggering.

I had total confidence we could work through it with Claude until it was working. It was also around this point that I started thinking about writing this series, wondering how many people had actually tried this at home, given that none of it is strictly necessary.

This last one is particularly subtle. The GitLab CI documentation documents allow_failure clearly, but the interaction with “no matching runner” is easy to miss. The symptom (pipeline blocked, no error message) doesn’t obviously point to the cause.

AI Collaboration Note

What Claude contributed: The initial structure of policy-check.sh was drafted with Claude’s help, including the allowlist comparison pattern using comm -23 to find new violations that aren’t in the baseline. Claude also generated the first version of the code review prompt for claude-code-review.yml based on a description of the project’s conventions.

Where it needed correction: Claude’s first draft of the code review workflow had it triggering automatically on every PR (pull_request event). In practice, the review takes several minutes and doesn’t post to the structured GitHub review section (it posts as a comment). The trigger was changed to workflow_dispatch only, so you run it manually when you want a thorough review, rather than automatically on every PR where it adds noise. Claude also initially suggested using gh pr review --request-changes in all cases, which fails when reviewing your own PRs (GitHub doesn’t allow it). The prompt needed a fallback to --comment for self-reviews.

Prompt that worked: “Write a GitHub Actions workflow that uses the Claude Code Action to review pull requests for a homelab IaC repo. The review should check for: hardcoded secrets, :latest image tags, ExternalSecret API versions, Terraform template escaping in .tftpl files, and whether new apps have dual ingress. Post the review as a gh pr review comment with findings ordered High → Medium → Low.”

Using a different AI tool? The review prompt format is tool-agnostic. You could adapt this pattern for any AI with a GitHub Actions integration (Copilot Workspace, Gemini Code Assist, etc.). The key is giving the model explicit project conventions to check against, rather than asking for a generic “code review.” Generic reviews produce generic findings.

Lessons

A kubeconform failure in CI is a thirty-second annoyance. A kubeconform miss in production is a forty-five minute incident. The math is straightforward, and the gap between them is whether you run schema validation on every PR or trust that valid YAML is also valid Kubernetes.
Path filtering keeps CI fast and relevant. Terraform jobs shouldn’t run on Kubernetes-only changes, and vice versa. Without path filtering, every PR pays the full validation cost regardless of what changed, which slows feedback and trains engineers to ignore CI results.
allow_failure: true and “no matching runner” interact badly. A job with no matching runner doesn’t fail. It pends forever. The correct fix is when: manual, which removes the job from automatic execution. This pattern applies to both GitHub Actions and GitLab CI.
The ansible-galaxy collection list in CI is hardcoded. Adding a collection to requirements.yml doesn’t automatically add it to the CI install step. When you add a new Ansible collection, update both files.
Policy checks with allowlists enable incremental hardening. Introducing a new policy check to a repository with existing violations is safe if you allowlist the current baseline. New violations are blocked, the baseline shrinks over time, and you avoid the “fix everything now or disable the check” false dichotomy.

Next: Post 12 — GitLab on Kubernetes: 10 PRs to Make It Work — self-hosted GitLab with Keycloak OIDC, homelab CA TLS, and a Kubernetes runner.