Re: [LSF/MM/BPF TOPIC] block drivers in user space

[Ming Lei <ming.lei@redhat.com>]

On Tue, Mar 29, 2022 at 01:20:57PM -0400, Gabriel Krisman Bertazi wrote:
> Ming Lei <ming.lei@redhat.com> writes:
> 
> >> I was thinking of something like this, or having a way for the server to
> >> only operate on the fds and do splice/sendfile.  But, I don't know if it
> >> would be useful for many use cases.  We also want to be able to send the
> >> data to userspace, for instance, for userspace networking.
> >
> > I understand the big point is that how to pass the io data to ubd driver's
> > request/bio pages. But splice/sendfile just transfers data between two FDs,
> > then how can the block request/bio's pages get filled with expected data?
> > Can you explain a bit in detail?
> 
> Hi Ming,
> 
> My idea was to split the control and dataplanes in different file
> descriptors.
> 
> A queue has a fd that is mapped to a shared memory area where the
> request descriptors are.  Submission/completion are done by read/writing
> the index of the request on the shared memory area.
> 
> For the data plane, each request descriptor in the queue has an
> associated file descriptor to be used for data transfer, which is
> preallocated at queue creation time.  I'm mapping the bio linearly, from
> offset 0, on these descriptors on .queue_rq().  Userspace operates on
> these data file descriptors with regular RW syscalls, direct splice to
> another fd or pipe, or mmap it to move data around. The data is
> available on that fd until IO is completed through the queue fd.  After
> an operation is completed, the fds are reused for the next IO on that
> queue position.
> 
> Hannes has pointed out the issues with fd limits. :)

OK, thanks for the detailed explanation!

Also you may switch to map each request queue/disk into a FD, and every
request is mapped to one fixed extent of the 'file' via rq->tag since we
have max sectors limit for each request, then fd limits can be avoided.

But I am wondering if this way is friendly to userspace side implementation,
since there isn't buffer, only FDs visible to userspace.


thanks,
Ming