edkarlsson
GitHubGet in touch

Homelab as Production/Bonus

The GPU Passthrough That Wasn't: A Hardware Limitation Story

Sometimes the right answer is that the answer doesn't exist

This is the post I didn’t plan to write. The rest of this series documents 43 sessions of AI-assisted infrastructure work where things eventually worked, where debugging arcs had resolutions, and where the right fix existed and could be found. This post is about the one where it didn’t.

This happened in the final week of the project, after the main cluster infrastructure was complete. The platform was already running 20+ applications while this node kept crashing.

The goal was straightforward: add a GPU compute node to the homelab cluster, pass the RTX 3060 through to a VM running ComfyUI (an open-source AI image generation interface), and have AI image generation available as a proper Kubernetes workload. Three sessions later, the node was removed from the Proxmox cluster entirely and the GPU was running ComfyUI natively on Ubuntu, managed by nobody.

Here is what happened.

The Hardware

The sixth node added to the cluster was a dedicated GPU compute machine: an Intel i7-7820X (Skylake-X architecture, 8 cores, 16 threads), 48 GB RAM, and two GPUs: a GTX 1080 Ti as the host display card and an RTX 3060 (GA106, Ampere architecture) as the compute target. The motherboard was an ASRock X299 Taichi, which matters more than it should.

The plan was to bind the RTX 3060 to the VFIO kernel driver on the Proxmox host, pass it through to a QEMU VM via PCIe passthrough, and install NVIDIA drivers inside the VM. Standard GPU passthrough. Well documented in the Proxmox community. Should work.

Stage 1: VFIO Setup

IOMMU was already enabled in BIOS (VT-d). The GRUB configuration needed intel_iommu=on iommu=pt:

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"
update-grub && reboot

After reboot, VFIO modules were loaded and the NVIDIA driver was blacklisted so the host would not claim the RTX 3060:

# /etc/modules-load.d/vfio.conf
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

# /etc/modprobe.d/blacklist-nvidia-passthrough.conf
blacklist nouveau
blacklist nvidia
blacklist nvidiafb

The device IDs for the RTX 3060 GPU function (10de:2504) and HDMI audio function (10de:228e) went into the VFIO PCI options:

# /etc/modprobe.d/vfio.conf
options vfio-pci ids=10de:2504,10de:228e

After regenerating initramfs and rebooting, the binding confirmed:

$ lspci -nnk -s 65:00.0
Kernel driver in use: vfio-pci

$ lspci -nnk -s 65:00.1
Kernel driver in use: vfio-pci

VFIO setup: complete. The Terraform module for the ComfyUI VM was already written with the passthrough config:

module "comfyui" {
  source                  = "./modules/comfyui"
  gpu_passthrough_enabled = true
  gpu_pci_id              = "0000:65:00.0"
  gpu_audio_pci_id        = "0000:65:00.1"
  machine_type            = "q35"
  bios_type               = "ovmf"
}

The VM started. NVIDIA drivers were installed inside the guest. And then the host kernel panicked.

(Physical power button. First of twenty-two.)

Why VFIO Failed: The Ampere + X299 Problem

The crash happened consistently, roughly three to four minutes into VM boot, during NVIDIA driver initialization inside the guest. Not during VM start. Not during IOMMU setup. During the point where the NVIDIA driver in the guest VM tries to do DMA initialization with the GPU.

Claude worked through the diagnosis systematically. Three specific hardware problems were contributing:

Problem 1: rombar and pci_map_rom()

The Proxmox default of rombar=1 tells QEMU to call pci_map_rom(), which attempts to map the GPU’s ROM BAR into the VGA legacy memory region 0x000c0000-0x000dffff. On X299, the PCI bus window for that region only covers a subset of those addresses (0x000c4000-0x000c7fff). The mapping attempt triggers a resource conflict. On some hardware this is a soft failure. On X299 with Skylake-X, it is a hard kernel panic.

Fix: rombar=0. The NVIDIA driver does not need the PCI ROM for compute workloads. It loads firmware internally.

# In the Terraform hostpci block
rombar = false

Problem 2: EFI framebuffer holding GPU BARs

The EFI stub initializes a framebuffer using the GPU’s BARs at early boot and marks them as BOOTFB reservations. When VFIO later tries to set up DMA mapping for those same BARs, the reservations block the operation. Adding initcall_blacklist=sysfb_init to the GRUB command line prevents the EFI framebuffer from initializing and claiming those BARs:

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt initcall_blacklist=sysfb_init"

Problem 3: Audio function FLR and group bus reset

This one was the fatal constraint. The RTX 3060’s HDMI audio function (65:00.1) has no Function Level Reset capability. You can verify this by checking the reset_method sysfs entry: it is empty. When VFIO needs to reset the audio function after a VM exits or before a VM starts, it falls back to a group-level bus reset, which resets everything in the IOMMU group simultaneously.

On X299/Skylake-X, that group bus reset crashes the host.

Claude wrote a udev rule to force FLR-only reset:

# /etc/udev/rules.d/99-vfio-flr.rules
ACTION=="bind", SUBSYSTEM=="pci", ATTR{vendor}=="0x10de", \
  ATTR{device}=="0x228e", ATTR{reset_method}="flr"

This prevents the fallback to bus reset. Without FLR capability on the device, this means the audio function cannot be reset at all between VM instances, which causes the NVIDIA driver in a second VM boot to fail. But at least the host does not crash.

With all three fixes applied: rombar=0, sysfb_init blacklisted, and FLR forced on the audio function, the host survived VM start. The kernel panic moved later into the boot sequence. Then it stopped happening. Progress.

Then the VM reached the point where the NVIDIA driver tries to initialize DMA. The host panicked again.

Each kernel panic meant walking to the machine, holding the power button until it died, waiting, booting back into Proxmox, and starting the debug loop over. By this point the power button on that chassis was getting a workout.

This is documented as a known incompatibility: Ampere GPU DMA initialization combined with X299 PCIe handling. The community has reports of it. There is no BIOS fix. There is no driver workaround. The hardware combination does not support reliable VFIO GPU passthrough.

VFIO was abandoned.

Stage 2: The LXC Pivot

The alternative to VFIO passthrough is to run NVIDIA drivers natively on the Proxmox host and pass the device nodes into a privileged LXC container via bind-mounts. No IOMMU. No VFIO. The host driver owns the GPU; the container gets access through the character device interface.

This required clearing the VFIO configuration and letting the NVIDIA driver claim the RTX 3060:

# Comment out GPU IDs in /etc/modprobe.d/vfio.conf
# Allow NVIDIA driver to bind instead
update-initramfs -u && reboot

Driver installation: first obstacle

The Debian-packaged NVIDIA drivers at version 550.x do not build on PVE kernel 6.17. Two separate kernel API breaks in recent upstream versions affect both the open and closed NVIDIA module:

  • Closed module: drm_framebuffer_funcs was restructured in kernel 6.12
  • Open module: dma_buf_attachment_is_dynamic was removed in kernel 6.13

The official NVIDIA .run installer at version 580.x or later contains fixes for both. The Debian package repositories had not caught up:

# Remove any existing Debian NVIDIA packages
apt-get remove --purge $(dpkg -l | grep -i nvidia | awk '{print $2}' | tr '\n' ' ') 2>/dev/null

# Install from official .run installer
./NVIDIA-Linux-x86_64-580.126.18.run --no-opengl-files --dkms --silent

After this, nvidia-smi showed both GPUs on the host correctly.

LXC device access: second set of obstacles

Getting the NVIDIA device nodes into the LXC container required writing raw LXC config entries to /etc/pve/lxc/530.conf, because the bpg/proxmox Terraform provider does not model lxc.cgroup2.devices.allow or lxc.mount.entry. A null_resource with an SSH provisioner wrote them after container creation.

The cgroup2 device allowlist and mount entries:

lxc.cgroup2.devices.allow = c 195:* rwm
lxc.cgroup2.devices.allow = c *:* rwm
lxc.mount.entry = /dev/nvidia0 dev/nvidia0 none bind,optional,create=file
lxc.mount.entry = /dev/nvidia1 dev/nvidia1 none bind,optional,create=file
lxc.mount.entry = /dev/nvidiactl dev/nvidiactl none bind,optional,create=file
lxc.mount.entry = /dev/nvidia-uvm dev/nvidia-uvm none bind,optional,create=file
lxc.mount.entry = /dev/nvidia-uvm-tools dev/nvidia-uvm-tools none bind,optional,create=file

Two more gotchas surfaced, both specific to NVIDIA 580.x and kernel 6.17:

First: nvidia-uvm device nodes (/dev/nvidia-uvm, /dev/nvidia-uvm-tools) are created lazily. They do not exist until the first userspace access. If the LXC container starts before nvidia-smi has run on the host, the bind-mounts fail silently and CUDA fails inside the container. A systemd service resolved this:

[Unit]
Description=Initialize NVIDIA UVM device nodes
Before=pve-container@530.service
After=systemd-udev-settle.service

[Service]
Type=oneshot
ExecStart=/usr/bin/nvidia-smi -L
RemainAfterExit=yes

Second: in NVIDIA 580.x, the nvidia-uvm major device number is dynamically assigned by the kernel, not the previously hardcoded value of 236. On kernel 6.17, it is assigned 510. Using c 236:* rwm in the cgroup allowlist grants access to the wrong device class entirely. The fix was c *:* rwm for a trusted privileged container, or reading the actual major at runtime:

cat /sys/module/nvidia_uvm/parameters/uvm_dev_major
# 510

Also: vendor-reset was installed on the host from a previous VFIO experiment. This kernel module exists to help AMD GPUs reset between VM instances. It has no implementation for NVIDIA. On this hardware, it caused spontaneous host crashes during unrelated operations. Removing it stopped those:

dkms remove vendor-reset/0.1.1 --all

After all of this: the LXC container started successfully. NVIDIA devices were visible inside (/dev/nvidia0, /dev/nvidia1, /dev/nvidiactl, /dev/nvidia-uvm). Docker and the NVIDIA Container Toolkit installed cleanly. The test image confirmed GPU access from inside a container:

$ docker run --rm --gpus all nvidia/cuda:12.6.0-base-ubuntu24.04 nvidia-smi
# RTX 3060 visible. CUDA 12.6.

Progress. ComfyUI was deployed via Docker Compose inside the container.

Then pct exec started crashing the host.

The Final Problem: pct exec and Kernel Panics

Across the three sessions, the machine was physically restarted twenty-two times. The rest of the project, all 43 sessions of it, required zero physical power cycles. Every one of those restarts was this node.

pct exec 530 -- apt-get install <package> causes a repeatable host kernel panic on this hardware combination. Not every time. Not immediately. But consistently during package installation workloads running inside the container.

The crash does not happen during idle operation. It does not happen on the host directly. It happens specifically during apt workloads inside the privileged LXC container on this node.

The suspected cause: an interaction between cgroup2, X299/Skylake-X PCIe handling, and kernel 6.17. The precise mechanism was never identified.

What was tried:

  • BIOS updated from P1.70 (the 2017 launch BIOS) to P2.50. No change.
  • vendor-reset removed. No change on pct exec crashes (did fix unrelated crashes).
  • SSH directly into the container instead of pct exec. This worked for package installation. But the crash pattern returned during GPU workloads at heavier load.

Two sessions of debugging without finding a reproducible root cause or a working mitigation. The pattern pointed at hardware: X299 with this kernel version, this cgroup configuration, and this workload type. Not something fixable in software.

The Decision

Three sessions in, the decision was to remove the node from the Proxmox cluster and reinstall it as a standalone Ubuntu workstation. ComfyUI runs natively on Ubuntu, with NVIDIA 580.x drivers installed directly. No Proxmox. No VFIO. No LXC. No Terraform managing it.

The Terraform modules for both the original ComfyUI VM and the LXC approach are retained in the repository, commented out, with a full history of what was tried and why each approach was abandoned. The gotchas registry has 25 entries covering VFIO rombar, FLR, EFI framebuffer, cgroup2 device major numbers, nvidia-uvm lazy init, vendor-reset, and the pct exec crash pattern.

The node runs ComfyUI. The RTX 3060 generates images. It just does not do any of it as part of the cluster.

What This Illustrates

The AI collaboration was genuinely useful throughout. Claude correctly identified the rombar issue, wrote the udev FLR rule, diagnosed the dynamic nvidia-uvm major number change in NVIDIA 580.x, created the systemd service for lazy device node initialization, and traced through each layer of the device access chain inside the LXC.

None of that was the problem. The problem was that the hardware combination has a documented incompatibility that no kernel configuration, driver version, or Terraform module can fix. X299/Skylake-X + Ampere GPU VFIO passthrough does not work reliably. X299 with cgroup2 + privileged LXC + heavy GPU workloads has a crash pattern that nobody has fully explained.

When the constraint is hardware, the fix is different hardware.

The platform runs 20+ applications with SSO, GitOps, HA PostgreSQL, and a full CI/CD pipeline. The GPU compute workload runs on a standalone machine next to it. That is not a failure of the methodology. It is what honest engineering looks like.

AI Collaboration Note

What Claude contributed: Systematic diagnosis of each VFIO failure mode. The rombar/pci_map_rom root cause analysis, the udev FLR rule, the sysfb_init blacklist, the cgroup2 dynamic major number detection, the nvidia-uvm systemd service, and the vendor-reset identification were all produced by Claude given the crash symptoms and dmesg output. The LXC approach pivot was also Claude’s suggestion after VFIO was confirmed non-viable.

Where it needed correction: Claude was optimistic about each fix. After rombar was resolved, it expected the crash to be solved. After the FLR rule was applied, it expected the crash to be solved. After vendor-reset was removed, it expected the crash to be solved. The repeated “this should fix it” responses were accurate in that each fix addressed a real issue, but none of them addressed the underlying hardware incompatibility. Human judgment was required to recognize the pattern (“three sessions, no resolution, same hardware”) and make the call to abandon the approach entirely.

Prompt that worked: “The host kernel panicked during NVIDIA driver init in the guest VM. Here is the dmesg output. What is pci_map_rom doing and why would it fail on X299 hardware?”

Using a different AI tool? GPU passthrough diagnosis is highly hardware-specific. Any AI assistant can help interpret dmesg output and suggest standard VFIO fixes. The value is in the iterative loop: share the error, get a hypothesis, test it, share the result. The bottleneck is always the hardware in front of you, not the model.

Lessons

  • VFIO passthrough is hardware-specific in ways that documentation does not always capture. The Ampere + X299/Skylake-X incompatibility affects DMA initialization during NVIDIA driver load in the guest. It is not a BIOS setting or a VFIO config option. It is a hardware limitation.

  • GPU passthrough on recent NVIDIA drivers requires specific attention to rombar, sysfb, and VGA arbitration. All three need to be addressed for Ampere GPUs on non-display passthrough. rombar=0, initcall_blacklist=sysfb_init, and options vfio-pci disable_vga=1 are the correct starting configuration.

  • NVIDIA 580.x breaks assumptions from older driver documentation. The nvidia-uvm major number is now dynamically assigned. /dev/nvidia-modeset no longer exists as a standalone device node. Use c *:* rwm for cgroup allowlists in trusted privileged containers rather than hardcoded major numbers.

  • vendor-reset is for AMD GPUs only. Building or loading it for NVIDIA passthrough provides no benefit and may cause unrelated crashes. Remove it from any NVIDIA passthrough host.

  • The LXC approach (host driver plus bind-mounted device nodes) is a valid fallback for hardware where VFIO is not reliable. It avoids IOMMU entirely and works well on hardware that supports it. On X299 in this configuration, it did not fully work either.

  • Knowing when to stop is part of the methodology. Three sessions of debugging hardware incompatibilities with no root cause identified is signal, not just noise. The gotchas registry documents what was found. The hardware runs the workload it can support.

The IaC code for both the ComfyUI VM module and the LXC module (commented out, with history) is in the homelab-as-production-iac repository under infrastructure/modules/comfyui and infrastructure/modules/comfyui-lxc.