Re: [BUG] RCU hang with io_uring nvme polling

[Keith Busch <kbusch@kernel.org>]

On Fri, Jul 03, 2026 at 01:20:24PM -0400, Ben Carey wrote:
> While testing the patch we decided to trace the amount of time a workload
> spends in blk_hctx_poll. We found that, for a test case with 8 jobs running for
> 10 seconds, it spent ~71% of its runtime in that function alone. We ran this
> test with an Intel Optane mounted with NVMe over PCIe target but have observed
> similar behavior on a VM, measured by:
> 
> perf record -F 99 -a -g -- \
>   fio --bs=1K --direct=1 --iodepth=1 --runtime=10 --rw=randread --time_based \
>     --ioengine=io_uring --hipri=1 --fixedbufs=0 --registerfiles=0 \
>       --sqthread_poll=0 \
>     --numjobs=8 --name=job0 --output-format=json --clocksource=clock_gettime \
>     --filename=/dev/nvme0n1
> 
> Again, this was tested with nvme.poll_queues=1, but similar behavior occurs
> with higher poll_queues, and also on a VM.
> 
> This bug seems to pollute our experimental results, and thus stands as
> something needing to be fixed for us to continue our research. Do you all think
> there's a different solution than the timeout?

What exactly do you have in mind? Shouldn't you expect to spend most of
your CPU time in the polling loop? As long as you keep the queues busy,
there's something to poll, so blk_hctx_poll is exactly where you want to
see the software be in a perf report. Seeing a high poll CPU utilization
means the software is efficient compared to the hardware. If we spend
very little time in the polling loop, then either you have incredibly
quick hardware, and let's face it, Optane SSDs are EOL and a generation
behind on link speeds so that's not gonna get there anymore, or our
software dispatch stack has an inefficiency somewhere.

If you have many pollers competing against a very low utilized queue,
then I think you have an application level problem mismatched to the
feature.

If you want to spend less time in the poll loop, then set the hybrid
poll sleep time. It should result in less polling time, but it'll push
your average latency higher.

The only thing the jiffie timeout may show a problem is when you stop
dispatching, which should only affect the time to close the ring when it
lost the polling race with a peer on the last IO it is looking for, but
should not affect individual IO latency.