[bug report] nvme-tcp poll queue causes busy loop and 100% CPU in nvme_tcp_poll() on the latest linux-block/for-next

[bug report] nvme-tcp poll queue causes busy loop and 100% CPU in nvme_tcp_poll() on the latest linux-block/for-next

[Yi Zhang]

Hi

This issue was observed on the latest linux-block/for-next during CKI
test, where enabling the poll queues leads to a tight busy polling
loop and 100% CPU usage during "nvme connect".
It seems was introduced from v6.19-rc1 and cannot reproduced on v6.18.
I will try to bisect it.
I am happy to test any proposed fixes or provide additional traces if needed.

Reproducer:
#nvme_trtype=tcp  ./check nvme/047
nvme/047 (tr=tcp) (test different queue types for fabric transports) [passed]
    runtime  1061.108s  ...  19857.364s

# ps aux | grep "nvme connect"
root        2492 99.9  0.0  15884  5520 pts/0    R+   01:28   0:13
nvme connect --traddr 127.0.0.1 --transport tcp --trsvcid 4420 --nqn
blktests-subsystem-1
--hostnqn=nqn.2014-08.org.nvmexpress:uuid:0f01fb42-9f7f-4856-b0b3-51e60b8de349
--hostid=0f01fb42-9f7f-4856-b0b3-51e60b8de349 --nr-write-queues=1
--nr-poll-queues=1 --output-format=json

# perf top -p  2492
Samples: 71K of event 'cycles:P', 4000 Hz, Event count (approx.):
39693035620 lost: 0/0 drop: 0/0
Overhead  Shared O  Symbol
  51.62%  [kernel]  [k] blk_hctx_poll
  34.64%  [kernel]  [k] nvme_tcp_poll
  13.39%  [kernel]  [k] __x86_indirect_thunk_rax
   0.02%  [kernel]  [k] native_irq_return_iret
   0.02%  [kernel]  [k] native_write_msr
   0.02%  [kernel]  [k] update_load_avg
   0.01%  [kernel]  [k] __raw_spin_lock_irqsave
   0.01%  [kernel]  [k] native_sched_clock
   0.01%  [kernel]  [k] read_tsc
   0.01%  [kernel]  [k] sched_balance_trigger
   0.01%  [kernel]  [k] __hrtimer_run_queues
   0.01%  [kernel]  [k] __update_load_avg_cfs_rq
   0.01%  [kernel]  [k] __timekeeping_advance
   0.01%  [kernel]  [k] native_read_msr
   0.01%  [kernel]  [k] ktime_get_update_offsets_now
   0.01%  [kernel]  [k] task_tick_fair
   0.01%  [kernel]  [k] native_apic_msr_eoi
   0.01%  [kernel]  [k] sugov_get_util

dmesg:
[ 5514.212238] run blktests nvme/047 at 2026-01-19 16:37:29
[ 5514.229581] loop0: detected capacity change from 0 to 2097152
[ 5514.235566] nvmet: adding nsid 1 to subsystem blktests-subsystem-1
[ 5514.244495] nvmet_tcp: enabling port 0 (127.0.0.1:4420)
[ 5514.267088] nvmet: Created nvm controller 1 for subsystem
blktests-subsystem-1 for NQN
nqn.2014-08.org.nvmexpress:uuid:0f01fb42-9f7f-4856-b0b3-51e60b8de349.
[ 5514.269066] nvme nvme0: creating 73 I/O queues.
[ 5514.276926] nvme nvme0: mapped 1/72/0 default/read/poll queues.
[ 5514.301778] nvme nvme0: new ctrl: NQN "blktests-subsystem-1", addr
127.0.0.1:4420, hostnqn:
nqn.2014-08.org.nvmexpress:uuid:0f01fb42-9f7f-4856-b0b3-51e60b8de349
[ 5515.068893] nvme nvme0: Removing ctrl: NQN "blktests-subsystem-1"
[ 5515.150669] nvmet: Created nvm controller 2 for subsystem
blktests-subsystem-1 for NQN
nqn.2014-08.org.nvmexpress:uuid:0f01fb42-9f7f-4856-b0b3-51e60b8de349.
[ 5515.152791] nvme nvme0: creating 74 I/O queues.
[ 5515.161907] nvme nvme0: mapped 1/72/1 default/read/poll queues.
[25370.530190] nvme nvme0: new ctrl: NQN "blktests-subsystem-1", addr
127.0.0.1:4420, hostnqn:
nqn.2014-08.org.nvmexpress:uuid:0f01fb42-9f7f-4856-b0b3-51e60b8de349
[25371.283929] nvme nvme0: Removing ctrl: NQN "blktests-subsystem-1"


--
Best Regards,
  Yi Zhang

Re: [bug report] nvme-tcp poll queue causes busy loop and 100% CPU in nvme_tcp_poll() on the latest linux-block/for-next

[Christoph Hellwig]

Any chance you could bisect it to the offending commit?

Re: [bug report] nvme-tcp poll queue causes busy loop and 100% CPU in nvme_tcp_poll() on the latest linux-block/for-next

[Ming Lei]

On Wed, Jan 21, 2026 at 10:50:00AM +0800, Yi Zhang wrote:
> Hi
> 
> This issue was observed on the latest linux-block/for-next during CKI
> test, where enabling the poll queues leads to a tight busy polling
> loop and 100% CPU usage during "nvme connect".
> It seems was introduced from v6.19-rc1 and cannot reproduced on v6.18.
> I will try to bisect it.

It may be related with f22ecf9c14c1 ("blk-mq: delete task running check
in blk_hctx_poll()").

I guess it only happens when `nvme connect` takes a bit long time, not sure if
it can be reproduced 100%.


Thanks,
Ming

Re: [bug report] nvme-tcp poll queue causes busy loop and 100% CPU in nvme_tcp_poll() on the latest linux-block/for-next

[Yi Zhang]

On Wed, Jan 21, 2026 at 4:53 PM Ming Lei <ming.lei@redhat.com> wrote:
>
> On Wed, Jan 21, 2026 at 10:50:00AM +0800, Yi Zhang wrote:
> > Hi
> >
> > This issue was observed on the latest linux-block/for-next during CKI
> > test, where enabling the poll queues leads to a tight busy polling
> > loop and 100% CPU usage during "nvme connect".
> > It seems was introduced from v6.19-rc1 and cannot reproduced on v6.18.
> > I will try to bisect it.
>
> It may be related with f22ecf9c14c1 ("blk-mq: delete task running check
> in blk_hctx_poll()").

This commit merged to v6.19-rc1, I will revert it and retest.

>
> I guess it only happens when `nvme connect` takes a bit long time, not sure if
> it can be reproduced 100%.

Correct, it's 100% reproduced.

>
>
> Thanks,
> Ming
>


-- 
Best Regards,
  Yi Zhang

Re: [bug report] nvme-tcp poll queue causes busy loop and 100% CPU in nvme_tcp_poll() on the latest linux-block/for-next

[Ming Lei]

On Wed, Jan 21, 2026 at 10:15:56PM +0800, Yi Zhang wrote:
> On Wed, Jan 21, 2026 at 5:03 PM Yi Zhang <yi.zhang@redhat.com> wrote:
> >
> > On Wed, Jan 21, 2026 at 4:53 PM Ming Lei <ming.lei@redhat.com> wrote:
> > >
> > > On Wed, Jan 21, 2026 at 10:50:00AM +0800, Yi Zhang wrote:
> > > > Hi
> > > >
> > > > This issue was observed on the latest linux-block/for-next during CKI
> > > > test, where enabling the poll queues leads to a tight busy polling
> > > > loop and 100% CPU usage during "nvme connect".
> > > > It seems was introduced from v6.19-rc1 and cannot reproduced on v6.18.
> > > > I will try to bisect it.
> > >
> > > It may be related with f22ecf9c14c1 ("blk-mq: delete task running check
> > > in blk_hctx_poll()").
> >
> > This commit merged to v6.19-rc1, I will revert it and retest.
> >
> 
> Hi Ming/Christoph
> 
> Confirmed the issue was introduced by this commit.
> 
> f22ecf9c14c1 blk-mq: delete task running check in blk_hctx_poll()

I guess the need_resched() in blk_hctx_poll() is still not enough, and it
may burns CPU too aggressively, especially for this in-kernel special sync
polling from passthrough request in nvmf_connect_io_queue().

Please try the following patch:

diff --git a/block/blk-mq.c b/block/blk-mq.c
index a29d8ac9d3e3..968699277c3d 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -1480,7 +1480,7 @@ EXPORT_SYMBOL_GPL(blk_rq_is_poll);
 static void blk_rq_poll_completion(struct request *rq, struct completion *wait)
 {
 	do {
-		blk_hctx_poll(rq->q, rq->mq_hctx, NULL, 0);
+		blk_hctx_poll(rq->q, rq->mq_hctx, NULL, BLK_POLL_ONESHOT);
 		cond_resched();
 	} while (!completion_done(wait));
 }
-- 
2.47.1



Thanks,
Ming