Re: ublk: RFC fetch_req_multishot

[Ming Lei <ming.lei@redhat.com>]

On Thu, Apr 24, 2025 at 06:19:29PM +0000, Ofer Oshri wrote:
> Hi,
> 
> Our code uses a single io_uring per core, which is shared among all block devices - meaning each block device on a core uses the same io_uring.
> 

Can I understand you are using single io_uring for serving one hw queue of
multiple ublk device?

> Let’s say the size of the io_uring is N. Each block device submits M UBLK_U_IO_FETCH_REQ requests. As a result, with the current implementation, we can only support up to P block devices, where P = N / M. This means that when we attempt to support block device P+1, it will fail due to io_uring exhaustion.
> 

Suppose N is the SQ size, the supported count of ublk device can be much bigger
than N/M, because any SQE is freed & available after it is issued to kernel, here
the SQE should be free for reuse after one UBLK_U_IO_FETCH_REQ uring_cmd is
issued to ublk driver.

That is said you can queue arbitrary number of uring_cmd with fixed SQ
size since N is just the submission batch size.

But it needs the ublk server implementation to flush queued SQE if
io_uring_get_sqe() returns NULL.

> To address this, we’d like to propose an enhancement to the ublk driver. The idea is inspired by the multi-shot concept, where a single request allows multiple replies.
> 
> We propose adding:
> 
> 1. A method to register a pool of ublk_io commands.
> 
> 2. Introduce a new UBLK_U_IO_FETCH_REQ_MULTISHOT operation, where a pool of ublk_io commands is bound to a block device. Then, upon receiving a new BIO, the ublk driver can select a reply from the pre-registered pool and push it to the io_uring.
> 
> 3. Introduce a new UBLK_U_IO_COMMIT_REQ command to explicitly mark the completion of a request. In this case, the ublk driver returns the request to the pool.  We can retain the existing UBLK_U_IO_COMMIT_AND_FETCH_REQ command, but for multi-shot scenarios, the “FETCH” operation would simply mean returning the request to the pool.
> 
> What are your thoughts on this approach?

I think we need to understand the real problem you want to address
before digging into the uring_cmd pool concept.

1) for save memory for lots of ublk device ?

- so far, the main preallocation should be from blk-mq request, and
as Caleb mentioned, the state memory from both ublk and io_uring isn't
very big

2) need to support as many as ublk device in single io_uring context with
limited SQ/CQ size ?

- it may not be one big problem because fixed SQ size allows to issue
arbitrary number of uring_cmd

- but CQ size may limit number of completed uring_cmd for notifying
incoming ublk request, is this your problem? Jens has added ring resize
via IORING_REGISTER_RESIZE_RINGS:

https://lore.kernel.org/io-uring/20241022021159.820925-1-axboe@kernel.dk/


3) or other requirement?



Thanks,
Ming