Re: NVMe MSI-X completions lost behind VMD across package C-state exit (even PC2) on Arrow Lake client VMD [8086:7d0b]

Re: NVMe MSI-X completions lost behind VMD across package C-state exit (even PC2) on Arrow Lake client VMD [8086:7d0b]

[Bjorn Helgaas]

[+cc VMD, cpuidle folks]

On Tue, Jul 14, 2026 at 08:57:17AM -0400, Rowan Cramer wrote:
> Hi,
> 
> I have what looks like a platform/VMD interrupt-delivery bug on a
> new Arrow Lake laptop, isolated with a reliable reproducer and a
> three-arm test matrix. Reporting here because the fault sits below
> the NVMe driver, in the VMD interrupt path, and is invisible unless
> the package is allowed to idle. Happy to test patches or run any
> further diagnostics.
> 
> Summary
> -------
> NVMe MSI-X completion interrupts are intermittently lost when they
> arrive through VMD's interrupt remapper while the CPU package is in
> (or exiting) ANY idle state deeper than C1 — including plain PC2.
> The drive completes the I/O correctly (the kernel's timeout handler
> always finds the completion already in the queue: "timeout,
> completion polled"); only the interrupt vanishes. Holding PM-QoS
> /dev/cpu_dma_latency at 100 us (C1-only idle) eliminates the fault
> completely; nothing else does.
> 
> System
> ------
> - Dell 14 Premium / XPS 14 (DA14250), BIOS 1.11.0 (latest; Dell
>   ships VMD force-enabled, no BIOS toggle on this model)
> - Intel Arrow Lake; VMD controller 8086:7d0b at 0000:00:0e.0
>   (client SKU -> VMD_FEATS_CLIENT, so MSI remapping is always on;
>   no VMD_FEAT_CAN_BYPASS_MSI_REMAP, no module parameter)
> - NVMe: Micron 2650 1TB (DRAM-less/HMB), FW 26500101, at
>   10000:e1:00.0; all queues route via VMD-PCI-MSIX per
>   /proc/interrupts
> - Fault is kernel-independent: reproduced identically on 6.18.33
>   (LTS), 7.0.9 and 7.0.10 (Arch builds, clean cmdline)
> - intel_idle in ACPI mode; exposed states POLL / C1_ACPI (1 us) /
>   C2_ACPI (127 us) / C3_ACPI (1048 us)
> 
> Symptom
> -------
>   nvme nvme0: I/O tag N QID Q timeout, completion polled
> 
> Present since the machine's very first boot (3 events in the first
> 90 s of its life). Baseline ~125 events/day idle; 60-190/hr under
> gaming I/O. Each event stalls the waiting I/O for io_timeout
> seconds (we run nvme_core.io_timeout=10 as mitigation), which
> manifests as multi-second system freezes. SMART is immaculate: 0
> media errors, 0 controller error-log entries against >500 host-side
> timeouts — the device completes the I/O; the notification is lost.
> 
> Key evidence
> ------------
> 1. Load is NOT the trigger. ~4 TB of worst-case saturating stress
>    (random reads at 2+ GiB/s, plus CPU, memory-bandwidth, GPU and
>    fsync-write pressure, combined) produced ZERO events. Saturating
>    I/O never lets the platform idle.
> 
> 2. Idle->active transitions ARE the trigger. A trivial reproducer
>    (bursty random reads separated by 1-8 s idle gaps, script below)
>    produces 13-16 lost interrupts per 15 min, every run.
> 
> 3. PM-QoS bisection puts the loss boundary exactly at C1|C2:
>      /dev/cpu_dma_latency hold   idle allowed        events/15min
>      none (stock)                C1+C2+C3            13-16
>      200 us                      C1+C2               19
>      100 us                      C1 only             0
>      0 us                        none (poll)         0
> 
> 4. Ruled out: drive APST (nvme set-feature 0x0c=0: no change), link
>    ASPM (runtime-disabled via sysfs on the VMD-domain link: no
>    change; note cmdline pcie_aspm=off does not govern VMD-managed
>    links, the vmd driver enables L1.1/L1.2 itself per
>    VMD_FEAT_BIOS_PM_QUIRK), kernel version, drive health, firmware
>    updates (none available).
> 
> 5. turbostat correlation (5 s intervals alongside the reproducer):
>    the package NEVER enters PC8/PC10 on this platform under bursty
>    I/O — deepest observed is PC6 — and every lost interrupt falls
>    in intervals showing only PC2/PC3/PC6 residency:
> 
>      arm (900 s each)        Pkg%pc2/pc3/pc6/pc8/pc10   events
>      stock                   13.2 / 0.3 / 4.8 / 0 / 0     13
>      endpoint LTR clamped    19.8 /  0  /  0  / 0 / 0     14
>        to 102.4 us
>      cpu_dma_latency=100us    ~0  /  0  /  0  / 0 / 0      0
> 
> 6. LTR is irrelevant. Firmware sets the endpoint LTR to 15.7 ms
>    (0x100f100f — nonzero, so vmd's BIOS PM quirk never touches it).
>    Clamping it to 102.4 us via setpci was demonstrably honored by
>    the platform (PC3/PC6 residency dropped to zero) yet the loss
>    rate was unchanged (14 vs 13). The fault does not need deep
>    package states — losses occur with nothing deeper than PC2, with
>    cores in C6/C7.
> 
> Current mitigation
> ------------------
> A systemd unit holding /dev/cpu_dma_latency at 100 us. Fully
> effective (a full day of gaming: 1 event vs the prior 60-190/hr),
> but it costs ~2.5-4 W package power versus letting the machine
> idle properly, since it must forbid core C6/C7 and PC2 everywhere.
> 
> Questions
> ---------
> 1. Is this a known Arrow Lake (or MTL-family) VMD erratum? The
>    machine shipped this way; Windows presumably masks it via its
>    own idle policy or is equally affected but silent.
> 
> 2. The client feature set forces MSI remapping on 8086:7d0b. Is the
>    VMCONFIG_MSI_REMAP bypass (as used for 28C0) architecturally
>    possible on client VMD? I am happy to test a patch enabling
>    bypass on 7d0b, or any other diagnostic/experimental patch —
>    the reproducer gives a decisive answer in 15 minutes.
> 
> 3. If the hardware genuinely cannot deliver remapped MSIs across
>    package C-state exit, should vmd.c be constraining PM (PM-QoS /
>    LTR / DMI-quirk) on affected platforms? As it stands, every
>    affected laptop loses NVMe interrupts silently at its default
>    idle settings.
> 
> Full journals since the machine's first boot, lspci -vvv, SMART
> dumps, raw turbostat logs and the complete phase-by-phase test
> history are preserved and available on request.

Can you make these available somewhere and include a URL?

> Reproducer
> ----------
> Point ROOT at any directory with a few multi-hundred-MB files on
> the VMD-attached NVMe, run for 900 s, and count kernel "completion
> polled" events (nvme_core.io_timeout=10 makes them visible within
> 10 s):
> 
>   #!/usr/bin/env python3
>   # bursty-I/O reproducer: idle gaps let the platform enter package
>   # idle; the first read of each burst rides the wake-up.
>   import os, sys, random, time
> 
>   ROOT = os.path.expanduser("~/BG3_Game")   # adjust
>   CHUNK = 65536
>   MIN_SIZE = 200 * 1024 * 1024
> 
>   files = []
>   for dirpath, _, names in os.walk(ROOT, followlinks=True):
>       for n in names:
>           p = os.path.join(dirpath, n)
>           try:
>               s = os.path.getsize(p)
>               if s >= MIN_SIZE:
>                   files.append((p, s))
>           except OSError:
>               pass
>   if not files:
>       sys.exit("no large files under " + ROOT)
> 
>   deadline = time.time() + int(sys.argv[1])
>   bursts = 0
>   fds = {}
>   while time.time() < deadline:
>       time.sleep(random.uniform(1.0, 8.0))
>       for _ in range(random.randint(4, 40)):
>           path, size = random.choice(files)
>           fd = fds.get(path)
>           if fd is None:
>               fd = fds[path] = os.open(path, os.O_RDONLY)
>           off = random.randrange(0, max(1, size - CHUNK)) & ~4095
>           os.pread(fd, CHUNK, off)
>       bursts += 1
>   print(f"bursts={bursts}")
> 
> (Drop page caches before each run — echo 3 >
> /proc/sys/vm/drop_caches — or cached reads will suppress the rate.)
> 
> Thanks,
> Rowan Cramer

Re: NVMe MSI-X completions lost behind VMD across package C-state exit (even PC2) on Arrow Lake client VMD [8086:7d0b]

[Rowan Cramer]

Done:

  https://github.com/Laif-added/nvme-vmd-interrupt-loss

Contents (README.md maps each file to the specific claims in my
report):

  - logs/journal-kernel-all-boots.txt — complete kernel log of every
    boot since the machine's first power-on (2026-06-26 onward, ~71k
    lines; 9 boots — a 10th, 3 s aborted first-power-on boot logged
    no kernel messages), including the 3 losses in the first ~90 s
    of its life
  - logs/timeout-counts-per-boot.txt — per-boot event counts with
    kernel versions (6.18.33 / 7.0.9 / 7.0.10)
  - logs/turbostat-test-*.log — raw 5 s turbostat for the PC-state
    correlation and LTR-clamp arms
  - hw/lspci-vvv.txt — full dump incl. the VMD domain (10000:), plus
    the raw LTR register readout (0x100f100f)
  - hw/smartctl-a.txt, hw/nvme-error-log.txt — 0 media errors, 0
    controller error-log entries (serials redacted)
  - report/test-history.txt — raw phase-by-phase results P0-P12 +
    TEST A/B
  - scripts/ — the reproducer (nvme-irq-bursty.py) and every test
    driver used

One scope note: the published journals are kernel-transport only
(journalctl _TRANSPORT=kernel) to keep userspace log noise and
personal data out of a public repo. The full journals are preserved
locally; I'm happy to send them to any maintainer off-list if
something beyond the kernel messages is needed.

Standing offer from the original report still applies: the
reproducer gives a decisive answer in 15 minutes, so I can turn
around any diagnostic or experimental patch (e.g. a MSI-remap
bypass attempt on 7d0b) quickly.

Thanks,
Rowan Cramer


On Tue, Jul 14, 2026 at 1:33 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
>
> [+cc VMD, cpuidle folks]
>
> On Tue, Jul 14, 2026 at 08:57:17AM -0400, Rowan Cramer wrote:
> > Hi,
> >
> > I have what looks like a platform/VMD interrupt-delivery bug on a
> > new Arrow Lake laptop, isolated with a reliable reproducer and a
> > three-arm test matrix. Reporting here because the fault sits below
> > the NVMe driver, in the VMD interrupt path, and is invisible unless
> > the package is allowed to idle. Happy to test patches or run any
> > further diagnostics.
> >
> > Summary
> > -------
> > NVMe MSI-X completion interrupts are intermittently lost when they
> > arrive through VMD's interrupt remapper while the CPU package is in
> > (or exiting) ANY idle state deeper than C1 — including plain PC2.
> > The drive completes the I/O correctly (the kernel's timeout handler
> > always finds the completion already in the queue: "timeout,
> > completion polled"); only the interrupt vanishes. Holding PM-QoS
> > /dev/cpu_dma_latency at 100 us (C1-only idle) eliminates the fault
> > completely; nothing else does.
> >
> > System
> > ------
> > - Dell 14 Premium / XPS 14 (DA14250), BIOS 1.11.0 (latest; Dell
> >   ships VMD force-enabled, no BIOS toggle on this model)
> > - Intel Arrow Lake; VMD controller 8086:7d0b at 0000:00:0e.0
> >   (client SKU -> VMD_FEATS_CLIENT, so MSI remapping is always on;
> >   no VMD_FEAT_CAN_BYPASS_MSI_REMAP, no module parameter)
> > - NVMe: Micron 2650 1TB (DRAM-less/HMB), FW 26500101, at
> >   10000:e1:00.0; all queues route via VMD-PCI-MSIX per
> >   /proc/interrupts
> > - Fault is kernel-independent: reproduced identically on 6.18.33
> >   (LTS), 7.0.9 and 7.0.10 (Arch builds, clean cmdline)
> > - intel_idle in ACPI mode; exposed states POLL / C1_ACPI (1 us) /
> >   C2_ACPI (127 us) / C3_ACPI (1048 us)
> >
> > Symptom
> > -------
> >   nvme nvme0: I/O tag N QID Q timeout, completion polled
> >
> > Present since the machine's very first boot (3 events in the first
> > 90 s of its life). Baseline ~125 events/day idle; 60-190/hr under
> > gaming I/O. Each event stalls the waiting I/O for io_timeout
> > seconds (we run nvme_core.io_timeout=10 as mitigation), which
> > manifests as multi-second system freezes. SMART is immaculate: 0
> > media errors, 0 controller error-log entries against >500 host-side
> > timeouts — the device completes the I/O; the notification is lost.
> >
> > Key evidence
> > ------------
> > 1. Load is NOT the trigger. ~4 TB of worst-case saturating stress
> >    (random reads at 2+ GiB/s, plus CPU, memory-bandwidth, GPU and
> >    fsync-write pressure, combined) produced ZERO events. Saturating
> >    I/O never lets the platform idle.
> >
> > 2. Idle->active transitions ARE the trigger. A trivial reproducer
> >    (bursty random reads separated by 1-8 s idle gaps, script below)
> >    produces 13-16 lost interrupts per 15 min, every run.
> >
> > 3. PM-QoS bisection puts the loss boundary exactly at C1|C2:
> >      /dev/cpu_dma_latency hold   idle allowed        events/15min
> >      none (stock)                C1+C2+C3            13-16
> >      200 us                      C1+C2               19
> >      100 us                      C1 only             0
> >      0 us                        none (poll)         0
> >
> > 4. Ruled out: drive APST (nvme set-feature 0x0c=0: no change), link
> >    ASPM (runtime-disabled via sysfs on the VMD-domain link: no
> >    change; note cmdline pcie_aspm=off does not govern VMD-managed
> >    links, the vmd driver enables L1.1/L1.2 itself per
> >    VMD_FEAT_BIOS_PM_QUIRK), kernel version, drive health, firmware
> >    updates (none available).
> >
> > 5. turbostat correlation (5 s intervals alongside the reproducer):
> >    the package NEVER enters PC8/PC10 on this platform under bursty
> >    I/O — deepest observed is PC6 — and every lost interrupt falls
> >    in intervals showing only PC2/PC3/PC6 residency:
> >
> >      arm (900 s each)        Pkg%pc2/pc3/pc6/pc8/pc10   events
> >      stock                   13.2 / 0.3 / 4.8 / 0 / 0     13
> >      endpoint LTR clamped    19.8 /  0  /  0  / 0 / 0     14
> >        to 102.4 us
> >      cpu_dma_latency=100us    ~0  /  0  /  0  / 0 / 0      0
> >
> > 6. LTR is irrelevant. Firmware sets the endpoint LTR to 15.7 ms
> >    (0x100f100f — nonzero, so vmd's BIOS PM quirk never touches it).
> >    Clamping it to 102.4 us via setpci was demonstrably honored by
> >    the platform (PC3/PC6 residency dropped to zero) yet the loss
> >    rate was unchanged (14 vs 13). The fault does not need deep
> >    package states — losses occur with nothing deeper than PC2, with
> >    cores in C6/C7.
> >
> > Current mitigation
> > ------------------
> > A systemd unit holding /dev/cpu_dma_latency at 100 us. Fully
> > effective (a full day of gaming: 1 event vs the prior 60-190/hr),
> > but it costs ~2.5-4 W package power versus letting the machine
> > idle properly, since it must forbid core C6/C7 and PC2 everywhere.
> >
> > Questions
> > ---------
> > 1. Is this a known Arrow Lake (or MTL-family) VMD erratum? The
> >    machine shipped this way; Windows presumably masks it via its
> >    own idle policy or is equally affected but silent.
> >
> > 2. The client feature set forces MSI remapping on 8086:7d0b. Is the
> >    VMCONFIG_MSI_REMAP bypass (as used for 28C0) architecturally
> >    possible on client VMD? I am happy to test a patch enabling
> >    bypass on 7d0b, or any other diagnostic/experimental patch —
> >    the reproducer gives a decisive answer in 15 minutes.
> >
> > 3. If the hardware genuinely cannot deliver remapped MSIs across
> >    package C-state exit, should vmd.c be constraining PM (PM-QoS /
> >    LTR / DMI-quirk) on affected platforms? As it stands, every
> >    affected laptop loses NVMe interrupts silently at its default
> >    idle settings.
> >
> > Full journals since the machine's first boot, lspci -vvv, SMART
> > dumps, raw turbostat logs and the complete phase-by-phase test
> > history are preserved and available on request.
>
> Can you make these available somewhere and include a URL?
>
> > Reproducer
> > ----------
> > Point ROOT at any directory with a few multi-hundred-MB files on
> > the VMD-attached NVMe, run for 900 s, and count kernel "completion
> > polled" events (nvme_core.io_timeout=10 makes them visible within
> > 10 s):
> >
> >   #!/usr/bin/env python3
> >   # bursty-I/O reproducer: idle gaps let the platform enter package
> >   # idle; the first read of each burst rides the wake-up.
> >   import os, sys, random, time
> >
> >   ROOT = os.path.expanduser("~/BG3_Game")   # adjust
> >   CHUNK = 65536
> >   MIN_SIZE = 200 * 1024 * 1024
> >
> >   files = []
> >   for dirpath, _, names in os.walk(ROOT, followlinks=True):
> >       for n in names:
> >           p = os.path.join(dirpath, n)
> >           try:
> >               s = os.path.getsize(p)
> >               if s >= MIN_SIZE:
> >                   files.append((p, s))
> >           except OSError:
> >               pass
> >   if not files:
> >       sys.exit("no large files under " + ROOT)
> >
> >   deadline = time.time() + int(sys.argv[1])
> >   bursts = 0
> >   fds = {}
> >   while time.time() < deadline:
> >       time.sleep(random.uniform(1.0, 8.0))
> >       for _ in range(random.randint(4, 40)):
> >           path, size = random.choice(files)
> >           fd = fds.get(path)
> >           if fd is None:
> >               fd = fds[path] = os.open(path, os.O_RDONLY)
> >           off = random.randrange(0, max(1, size - CHUNK)) & ~4095
> >           os.pread(fd, CHUNK, off)
> >       bursts += 1
> >   print(f"bursts={bursts}")
> >
> > (Drop page caches before each run — echo 3 >
> > /proc/sys/vm/drop_caches — or cached reads will suppress the rate.)
> >
> > Thanks,
> > Rowan Cramer