Re: [PATCH v5 0/7] Global Software Interrupt Moderation (GSIM)

[Naman Jain <namjain@linux.microsoft.com>]

On Wed, Sep 09, 2026 at 04:52:27PM +0800, Fengnan wrote:
> 在 2026/9/9 16:06, Luigi Rizzo 写道:
> > On Wed, Sep 9, 2026 at 8:25 AM Christoph Hellwig <hch@infradead.org> wrote:
> >> On Sat, Sep 05, 2026 at 10:30:22PM +0200, Thomas Gleixner wrote:
> >>>> it appears that GSIM is only effective in scenarios where multi disks at very high
> >>>> IOPS; in some cases, there was a noticeable performance regression.
> >>>> If there’s something wrong with my configuration, please correct me.
> >>> So we have a NVME specific mechanism to tackle the same problem and a
> >>> more generic version which is subsystem "independent".
> >> I'm not sure they tackle the entirely same problem, although they are
> >> very related.
> > There is overlap but it is accidental.
> > The difference between the two mechanism is the following:
> >
> > - GSIM addresses hardware limitations (SoCs that create huge PCIe
> >    backpressure on each MSIx interrupt, eventually reducing the available
> >    bandwidth too much, regardless of how many CPUs can process interrupts).
> >    The goal here is to throttle the total MSIx rate, blocking them at
> >    the source (PCIe device) rather than later in the interrupt controller.
> >
> > - the NVME mechanism (and NAPI) aim to amortize interrupt processing
> >    overhead. It does so by recreating some fine grained single-source
> >    interrupt moderation (something that NICs do in hardware, but NVME
> >    does not, with 100us being way too coarse for modern SSDs) and an
> >    adaptive scheme to adjust the delay and decide when to kick in.
> That's a great summary.
>
> As for generic solutions, I've tried several approaches before, but none
> of them
> could achieve the same or close results as NVMe adaptive interrupt polling.
> I'm still experimenting with different options.
>
>
> >
> > I think there is a reason for both.
> >
> > GSIM because it is more general and addresses an issue the others cannot handle
> > the subsystem-specific scheme because they can be more efficient
> > (eg they can hook into existing helpers like the threaded irq handler,
> > call directly the handler without going through the full irq chain,
> > use more targeted heuristics to decide when to kick in).
> >
> > cheers
> > luigi

Sharing the test results with GSIM and other NVME specific approaches
here as well. Previously discussed in
https://lore.kernel.org/all/1d12109d-2f47-4ca5-af7b-66c032a379a7@linux.microsoft.com/

There are couple of implementations posted on NVME list, including mine,
to solve nvme issues. Below data also suggests that while GSIM may bring
generic improvements in certain cases, we may still need some NVME
specific changes to address hardirq workload problems.

Background: I was debugging soft lockup issues where sufficiently high FIO
workload can lockup the CPU cores on which high number of NVME PCI MSI
interrupts are affined.

System: 128 ARM64 vCPUs, two NUMA nodes (CPUs 0-63 / 64-127),
There are four data NVMe controllers, each with 14 interrupt-driven I/O 
queues. Corresponding queue vectors from the four controllers are routed
to the same CPU. Therefore:

   vectors per affected CPU:       4
   affected interrupt CPUs:       14
   total data-NVMe I/O vectors:   56

The affected CPUs are:
   0, 10, 19, 28, 37, 46, 55,
   64, 74, 83, 92, 101, 110, 119
The reproduced soft lockups happen on exactly those 14 CPUs.

The production fio command is equivalent to:
   --filename=dev1:dev2:dev3:dev4
   --numjobs=128
   --iodepth=256
   --group_reporting

so the potential total is:
   128 x 256 = 32,768 requests

The NVMe hardware queue depth is 1,024, with 1,023 usable entries.

Default GSIM changes could not prevent soft lockup. Luigi suggested
a timer push change in addition to GSIM changes, which prevents the
lockups, but has an impact on latency and throughput. These are marked as
"push" in below tables.

NVMe fallback: changes I proposed in NVME, here:
https://lore.kernel.org/all/20261009050556.2817978-1-namjain@linux.microsoft.com/
threaded NVMe IRQs: Default kernel with nvme.use_threaded_interrupts=1

High-load results are medians of three runs, you can find complete 
minimum-to-maximum range:

                               IOPS median       complete range
   NVMe fallback                 5.004M          5.002-5.004M
   threaded NVMe IRQs            3.140M          3.100-3.169M
   GSIM fixed 10 us + push       1.962M          1.935-2.025M
   GSIM fixed 30 us + push       1.790M          1.775-1.822M
   GSIM fixed 50 us + push       1.744M          1.736-1.746M

Completion latency:
                               mean        p99          p99.9
   NVMe fallback              6.533 ms    14.352 ms     16.908 ms
   threaded NVMe IRQs        10.421 ms    22.151 ms     24.510 ms
   GSIM fixed 10 us + push   16.570 ms    38.011 ms    270.533 ms
   GSIM fixed 30 us + push   18.283 ms    40.108 ms    492.831 ms
   GSIM fixed 50 us + push   18.743 ms    40.632 ms    541.065 ms

More details and numbers in my RFC series.

Regards,
Naman