edkarlsson
GitHubGet in touch

Homelab as Production/Part 7 of 16

Secrets Without Secrets: 1Password Connect and External Secrets Operator

Never commit a secret. Ever. The production secrets pattern at homelab scale.

Secrets Without Secrets

Every infrastructure project has a moment where it first needs a password somewhere. A database password. An API key. An OAuth client secret. And at that moment, the path of least resistance is to put it in a YAML file, commit it to git, and tell yourself you’ll “do secrets properly later.”

Later never comes.

My .env files were becoming unmanageable. I had written scripts to track secrets and environment variables, saving them into 1Password so I wouldn’t forget them, but the fear of losing access and having to redeploy everything was always there. Early on, Terraform needed passwords too, and credentials ended up saved in multiple places at once. On top of that: the constant background worry about a compromised open-source dependency or a neighbor on the WiFi. I wanted a security-first posture, not a “I’ll clean this up later” posture.

I’ve seen this in production environments at real companies. A .env file committed in an early “I’ll fix it” moment, still sitting in the main branch three years and four engineers later. Credentials base64-encoded into Kubernetes Secrets and checked in alongside the application manifests. Base64 is not encryption, it’s decoration. An API key sitting in a Helm values file because somebody needed to ship something before a deadline.

The thing about “later” is that the security posture of a system is established by its first moments. The habits that form when you’re moving fast, the shortcuts that save ten minutes during setup, become the architecture. And the architecture becomes the risk profile.

So when this homelab build reached the point where the first application needed a database password, the answer was not to put it anywhere near git. The answer was to build the secrets pipeline first, before the first application.

That pipeline is: 1Password Connect as the secrets backend, External Secrets Operator as the Kubernetes bridge, and zero secrets in the repository. Not even in a branch, not even in a commit that gets squashed.

The Production Secrets Pattern

The anti-pattern is: secrets live in the repository. Sometimes they’re base64-encoded. Sometimes they’re in an .env file that “isn’t supposed to be committed.” Sometimes they’re in a Helm values.yaml file with a comment saying # TODO: move to secret. The common thread is that the secret is adjacent to the code, and “adjacent to the code” eventually means “in the code.”

The production pattern has one rule: secrets live in a secrets manager, and they travel to Kubernetes through an automated sync, never through a human copy-paste or a git commit.

The flow looks like this:

  1. A secret is created in 1Password (manually, or by a script that runs once)
  2. 1Password Connect exposes it via a local REST API
  3. External Secrets Operator reads from that API and creates a Kubernetes Secret
  4. The application references the Kubernetes Secret via secretKeyRef or a volume mount
  5. The git repository contains only the ExternalSecret manifest, which is safe to commit because it contains no secret values, only references to where secrets live

The git repository never sees the actual values. An attacker who compromises the git history gets configuration. They do not get credentials.

1Password Connect: The HA Setup

1Password Connect is a self-hosted server that provides a REST API over your 1Password vaults. Instead of calling the 1Password cloud API from inside your cluster (which requires internet, has rate limits, and creates an external dependency), Connect runs locally and caches vault data. Everything inside the cluster talks to Connect at a local IP.

In this build, Connect runs as an HA pair of Proxmox LXC containers with keepalived providing a floating VIP:

┌─────────────────────────────────────────┐
│  VIP: 10.0.0.72:8080 (keepalived VRRP) │
└──────────────────┬──────────────────────┘
                   │ floats to active node
        ┌──────────┴───────────┐
        │                      │
┌───────────────┐    ┌───────────────┐
│  LXC CT 200   │    │  LXC CT 201   │
│  pve-node-1   │    │  pve-node-2   │
│  10.0.0.70    │    │  10.0.0.71    │
│  MASTER (100) │    │  BACKUP (90)  │
└───────────────┘    └───────────────┘

Keepalived monitors the Connect /heartbeat endpoint every 5 seconds. If the active node goes down, the VIP moves to the standby within roughly 15 seconds. All consumers, Terraform, ESO, Ansible, talk to the VIP and don’t need to know which physical node is serving.

A deliberate choice here was to run Connect outside the Kubernetes cluster. Running Connect inside K3s creates a circular dependency: ESO needs Connect to sync secrets, but Connect needs a Kubernetes Secret for its credentials file, and if that Secret doesn’t exist yet, Connect can’t start, which means ESO can’t create it. The circular dependency resolves only via careful bootstrap ordering, and fails spectacularly if the sequence goes wrong.

This wasn’t an issue when I originally ran 1Password connect, because I had already deployed, synced, and it was on a separate machine, outside the cluster.

On Proxmox LXC, Connect has an independent lifecycle. It starts with the host, before K3s boots. There’s no ordering problem. The Terraform module sets LXC boot order to 1, so Connect is available by the time K3s VMs come up and ESO begins reconciling.

The only secret that leaves the repository boundary is the Connect token itself, and it lives in an environment file that is never committed, sourced locally when running Terraform or the bootstrap script.

Connect Mode vs. Service Account Mode: The First Failure

This is where Claude initially went wrong, and it’s worth documenting because it’s a genuinely confusing split in the 1Password ecosystem.

1Password offers two ways to authenticate programmatically: Connect mode (using a Connect server and a Connect token) and Service Account mode (using a service account token that calls the 1Password cloud API directly). These are architecturally different and mutually exclusive. You can’t use both simultaneously.

When drafting the initial ESO ClusterSecretStore configuration, Claude generated YAML that referenced both OP_CONNECT_HOST and OP_SERVICE_ACCOUNT_TOKEN, setting up the environment as if both modes could coexist. The reasoning sounded plausible: “Connect for Terraform, service account for ESO.” But that’s not how it works.

When both environment variables are set, the 1Password provider enters an ambiguous state. In ESO, it silently failed. The ClusterSecretStore appeared healthy in kubectl get clustersecretstore but ExternalSecrets showed SecretSyncedError on every attempt. The logs showed authentication failures without clearly indicating why.

The fix was to audit all environment configuration and unset OP_SERVICE_ACCOUNT_TOKEN entirely. This build uses Connect mode exclusively. Connect mode uses the Connect server’s HTTP API with a JWT token. Service Account mode authenticates directly to the 1Password cloud with a different token format. They’re not compatible and not composable.

AI Collaboration Note What Claude contributed: Drafted the ClusterSecretStore YAML, the ExternalSecret pattern, and the bootstrap script structure. Also identified the ESO v2 API version change from v1beta1 to v1. Where it needed correction: Initial environment configuration conflated Connect mode and Service Account mode, setting both OP_CONNECT_HOST and OP_SERVICE_ACCOUNT_TOKEN. The human caught it during debugging when ExternalSecrets silently failed to sync despite a healthy-looking ClusterSecretStore status. Claude also generated initial manifests using external-secrets.io/v1beta1 which was removed in ESO v2.0. Prompt that worked: "The ClusterSecretStore shows Ready but all ExternalSecrets fail with SecretSyncedError. Here is the ESO controller log. Walk through what could cause authentication to fail even with a healthy store status." Using a different AI tool? The same pattern applies: ask the AI to reason from observed symptoms (healthy status, failing secrets) rather than asking it to generate configuration upfront. The failure mode of conflating authentication modes is common to any AI that has learned from documentation covering multiple versions of a tool.

The Secrets Flow

flowchart LR
    subgraph OP["1Password Cloud"]
        VAULT["Homelab Vault<br/>custom text fields only"]
    end

    subgraph LXC["Proxmox LXC — HA Pair"]
        CONNECT["1Password Connect<br/>VIP: 10.0.0.72:8080<br/>local cache + REST API"]
    end

    subgraph K8S["K3s Cluster"]
        ESO["External Secrets Operator<br/>ClusterSecretStore<br/>onepassword-connect"]
        ES["ExternalSecret CR<br/>refreshInterval: 1h"]
        SECRET["Kubernetes Secret<br/>(owned by ESO)"]
        POD["Application Pod<br/>env var or volume mount"]
    end

    VAULT -->|"Connect sync pull"| CONNECT
    CONNECT -->|"REST API — Bearer token"| ESO
    ESO -->|"reconciles"| ES
    ES -->|"creates / updates"| SECRET
    SECRET -->|"secretKeyRef / volumeMount"| POD

    style VAULT fill:#0f172a,color:#fff
    style CONNECT fill:#7c3aed,color:#fff
    style ESO fill:#2563eb,color:#fff
    style ES fill:#1d4ed8,color:#fff
    style SECRET fill:#16a34a,color:#fff
    style POD fill:#0d9488,color:#fff

The important thing to notice in this diagram: the Kubernetes Secret is ephemeral. ESO owns it, and ESO re-syncs it on every refresh interval (default: 1 hour). If you manually edit the Kubernetes Secret, ESO will overwrite your changes at the next sync. The source of truth is 1Password, not the cluster.

External Secrets Operator: The Bridge

External Secrets Operator is a Kubernetes controller that watches ExternalSecret custom resources and materializes their referenced values as standard Kubernetes Secrets. It ships as a Helm chart installed into the external-secrets namespace, part of the platform controllers layer.

The ESO setup has two components:

ClusterSecretStore is cluster-scoped and created once. It’s the connection definition: where Connect lives, which vault to use, and which Kubernetes Secret contains the Connect token.

apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: onepassword-connect
spec:
  provider:
    onepassword:
      connectHost: http://10.0.0.72:8080
      vaults:
        Homelab: 1
      auth:
        secretRef:
          connectTokenSecretRef:
            name: onepassword-connect-token
            key: token
            namespace: external-secrets

Key details: vaults is a map of vault name to priority (integer). The vault name must match exactly what’s in 1Password. The connectTokenSecretRef points to the Kubernetes Secret containing the Connect JWT. This is the one manually bootstrapped secret in the entire system.

ExternalSecret is namespace-scoped, one per application secret. It declares what to fetch from 1Password and what Kubernetes Secret to produce.

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: myapp-secrets
  namespace: myapp
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: onepassword-connect
    kind: ClusterSecretStore
  target:
    name: myapp-secrets
    creationPolicy: Owner
  data:
    - secretKey: db-password
      remoteRef:
        key: myapp
        property: db-password

The remoteRef has two fields: key is the 1Password item title (e.g., myapp), and property is the field label within that item (e.g., db-password). These are separate fields, not a path.

The Custom Fields Gotcha

This is the single most common mistake when setting up 1Password with ESO, and it took a debugging session to learn it properly.

When you create a Login item in 1Password, it comes with three built-in fields: username, password, and url. These fields are special. They’re not addressable by the property field in an ExternalSecret remoteRef.

If you create a Login item and set the password, then write an ExternalSecret with property: password, one of two things happens: the sync returns empty, or the sync fails with an error like got 0 fields matching "password" (or got 2, if there’s ambiguity). Neither is the value you wanted.

The fix is simple but not obvious: always create custom text fields for any value that ESO needs to reference. In the 1Password UI, these are “Text” type fields you add manually, with labels you choose. Give them unambiguous names: db-password, api-key, smtp-username. Avoid dots in field names because the op CLI interprets dots as section separators.

The practical rule: when you create a 1Password item for an application, ignore the default Login fields entirely. Create a dedicated set of custom text fields with whatever labels make sense. Reference those labels in property:. The Kubernetes Secret key (secretKey) is independent. Name it whatever your application expects.

The Bootstrap Script: Solving the Chicken-and-Egg Problem

There is one inescapable circular dependency in this setup: ESO needs a Kubernetes Secret containing the Connect token to authenticate to 1Password, but ESO is what manages Kubernetes Secrets from 1Password. The Connect token secret can’t be created by ESO because ESO doesn’t exist yet when that secret needs to exist.

The solution is a bootstrap script that runs exactly once, before Flux reconciles ESO for the first time. The script reads the Connect token from the local environment and creates the Kubernetes Secret using kubectl:

#!/usr/bin/env bash
set -euo pipefail

NAMESPACE="external-secrets"
SECRET_NAME="onepassword-connect-token"

# Create the namespace if it doesn't exist yet
kubectl create namespace "${NAMESPACE}" --dry-run=client -o yaml | kubectl apply -f -

# Read the Connect token from environment
TOKEN="${OP_CONNECT_TOKEN:-${TF_VAR_op_connect_token:-}}"

if [[ -z "${TOKEN}" ]]; then
  echo "ERROR: No Connect token found in environment." >&2
  echo "Set OP_CONNECT_TOKEN or source .env.d/terraform.env" >&2
  exit 1
fi

# Create the secret (idempotent via --dry-run + apply)
kubectl create secret generic "${SECRET_NAME}" \
  --namespace="${NAMESPACE}" \
  --from-literal=token="${TOKEN}" \
  --dry-run=client -o yaml | kubectl apply -f -

The --dry-run=client -o yaml | kubectl apply -f - pattern makes this idempotent. Running it twice doesn’t create a duplicate or error. It applies the same manifest. The script lives at scripts/k8s/create-eso-connect-secret.sh in the repository and is safe to commit because it contains no secrets, only the logic for injecting one from the local environment.

After this script runs, Flux can bootstrap ESO, the ClusterSecretStore can authenticate to Connect, and every subsequent ExternalSecret reconciles without any manual intervention.

The Connect token is the one and only credential that has to live outside 1Password. Everything else, every database password, every API key, every OAuth client secret, goes into 1Password and comes out via ESO.

ESO v2 and the API Version Change

If you’ve found ESO documentation or blog posts from before roughly mid-2024, they’ll show apiVersion: external-secrets.io/v1beta1. That API version was deprecated and then removed in ESO v2.0.0.

The current stable API version is external-secrets.io/v1. This applies to both ClusterSecretStore and ExternalSecret. The v1beta1 API is no longer served by default. Applying a v1beta1 manifest to a cluster running ESO v2+ will produce a no matches for kind "ExternalSecret" in version "external-secrets.io/v1beta1" error.

Claude’s initial manifests used v1beta1. Not because the information was wrong, but because the training data included earlier documentation. The fix is straightforward: update the apiVersion field in every ESO manifest. Check with kubectl api-resources | grep external-secrets to see which versions your cluster is serving.

One other ESO Helm chart detail worth knowing: the chart ships CRDs via Helm templates, not the crds/ directory. In a Flux HelmRelease, set install.crds: Skip and upgrade.crds: Skip, and let the chart’s installCRDs: true value handle CRD installation. If you use crds.create: true in the Flux HelmRelease spec, you may get conflicting CRD management.

Force-Sync: When Secrets Get Stuck

ExternalSecrets have a default refreshInterval of 1 hour. If a secret fails to sync, because Connect was temporarily down, because a new item was just created and the Connect cache hasn’t caught up, or because of a transient error, it will sit in a failed state until the next refresh interval.

You don’t have to wait an hour. Annotate the ExternalSecret with force-sync set to the current timestamp:

kubectl annotate es myapp-secrets \
  -n myapp \
  force-sync=$(date +%s) \
  --overwrite

ESO watches for changes to this annotation and triggers an immediate re-sync when it changes. The value is just a Unix timestamp. It only needs to be different from the last value. This is the go-to move whenever an ExternalSecret shows SecretSyncedError and you want immediate feedback rather than waiting for the next scheduled reconciliation.

The force-sync pattern is also useful after rotating a secret in 1Password: update the value in the vault, annotate the ExternalSecret, and the new value is in the Kubernetes Secret within seconds.

When ESO fails completely: diagnostic steps

If ExternalSecrets are failing and force-sync isn’t helping, work down this checklist before assuming the issue is in the ExternalSecret itself:

1. Check the ClusterSecretStore status:

kubectl get clustersecretstore onepassword-connect -o yaml

Look at .status.conditions. A Ready: False condition here means ESO can’t reach 1Password Connect at all. ExternalSecrets won’t sync regardless of how they’re configured.

2. Check whether the Connect token secret exists:

kubectl get secret onepassword-connect-token -n external-secrets

This is the bootstrap secret that ESO needs to authenticate to 1Password Connect. If it’s missing (e.g., the namespace was recreated or the secret was accidentally deleted), the ClusterSecretStore will show as unhealthy.

3. If the token secret is missing, re-run the bootstrap script:

./scripts/k8s/create-eso-connect-secret.sh

The script is idempotent. Running it twice won’t break anything.

4. Check Connect server health from inside the cluster:

kubectl run -it --rm debug --image=curlimages/curl --restart=Never \
  -- curl http://10.0.0.72:8080/heartbeat

If the heartbeat endpoint returns a non-200, the Connect HA pair may have lost its VIP or the active container may be down. Check the keepalived status on both Connect LXC nodes.

Documenting these steps in the gotchas registry means the next engineer on-call doesn’t start from zero. They start from a checklist.

Lessons

  • The Connect token is the only secret that exists outside 1Password. Everything else flows through the pipeline. The number of places where a credential lives in plaintext in your environment is exactly one.

  • Connect mode and Service Account mode are mutually exclusive. If OP_SERVICE_ACCOUNT_TOKEN is set in the environment alongside OP_CONNECT_HOST, ESO will enter an ambiguous authentication state. Unset one completely.

  • Default Login item fields are not addressable by ESO. The username, password, and url fields on a Login-category item in 1Password can’t be referenced by property in an ExternalSecret. Create custom text fields for everything ESO needs to sync.

  • The bootstrap script being idempotent is what makes it safe to forget about. You run it once, but if the cluster is rebuilt or the namespace is accidentally deleted, you can run it again without consequences. Idempotency is the property that turns a one-time setup step into a reliable recovery procedure.

  • Use external-secrets.io/v1, because v1beta1 is gone. Any documentation, blog post, or AI-generated YAML that references v1beta1 is out of date. Check kubectl api-resources | grep external-secrets against your ESO version before trusting older examples.

Next: Post 8 — The Application Deployment Pattern — a repeatable 7-manifest pattern that scales to 20+ applications.