Re: [PATCH RFC] nvme-tcp: allow multiple queues per hctx

[Keith Busch <kbusch@kernel.org>]

On Mon, Sep 21, 2026 at 02:21:28PM +0200, Hannes Reinecke wrote:
> Interesting.
> We have so far refrained from similar attempts as the idea was that
> one CPU would be able to saturate the available bandwidth; additionally
> nvme_tcp_io_work() would be gobbling up any available data, so scheduling
> between two instances on the same queue would be pointless.

PCIe and RDMA can saturate the link with a single queue, but not TCP.
Note, this RFC specifically schedules multiple queues, not the same
queue. It is the same blk-mq hctx, but that fans out to different nvme
queues.
 
> So question would be: where does the speed up come from?

I have the workqueue unbounded, so if a batch of 4 requests comes in on
one thread, they get worked on in parallel on 4 different CPUs utilizing
different sockets. This also exploits NIC parallelisms via multiple RSS
buckets that wouldn't happen with single socket usage.

> In general not a bad idea. Especially if it helps to up our performance.
> But this really points to the same problem we're having with HW RAID
> HBAs: we have an issue if the per-queue I/O performance is below what
> the cpu can drive. Then it _does_ make sense to have more queues than\ CPUs,
> but the layout of which is beyond what blk-mq can handle.
> Ideally we should be able to handle that via blk-mq, too.

I'm still trying to see if we can achieve a similar result just by
making duplicate connections and letting the multipath policy handle the
spread. That does improve things significantly, but I'm getting worse
performance than this RFC, and I still don't know why yet. I didn't get
to work on this last week, so it's still on me to work out where it's
lacking.

> Nice topic for ALPSS ...

Thanks! We'll definitely touch on this next week.