Re: [LSF/MM/BPF BoF]: extend UBLK to cover real storage hardware

[Ming Lei <ming.lei@redhat.com>]

On Wed, Feb 15, 2023 at 10:27:07AM -0500, Stefan Hajnoczi wrote:
> On Wed, Feb 15, 2023 at 08:51:27AM +0800, Ming Lei wrote:
> > On Mon, Feb 13, 2023 at 02:13:59PM -0500, Stefan Hajnoczi wrote:
> > > On Mon, Feb 13, 2023 at 11:47:31AM +0800, Ming Lei wrote:
> > > > On Wed, Feb 08, 2023 at 07:17:10AM -0500, Stefan Hajnoczi wrote:
> > > > > On Wed, Feb 08, 2023 at 10:12:19AM +0800, Ming Lei wrote:
> > > > > > On Mon, Feb 06, 2023 at 03:27:09PM -0500, Stefan Hajnoczi wrote:
> > > > > > > On Mon, Feb 06, 2023 at 11:00:27PM +0800, Ming Lei wrote:
> > > > > > > > Hello,
> > > > > > > > 
> > > > > > > > So far UBLK is only used for implementing virtual block device from
> > > > > > > > userspace, such as loop, nbd, qcow2, ...[1].
> > > > > > > 
> > > > > > > I won't be at LSF/MM so here are my thoughts:
> > > > > > 
> > > > > > Thanks for the thoughts, :-)
> > > > > > 
> > > > > > > 
> > > > > > > > 
> > > > > > > > It could be useful for UBLK to cover real storage hardware too:
> > > > > > > > 
> > > > > > > > - for fast prototype or performance evaluation
> > > > > > > > 
> > > > > > > > - some network storages are attached to host, such as iscsi and nvme-tcp,
> > > > > > > > the current UBLK interface doesn't support such devices, since it needs
> > > > > > > > all LUNs/Namespaces to share host resources(such as tag)
> > > > > > > 
> > > > > > > Can you explain this in more detail? It seems like an iSCSI or
> > > > > > > NVMe-over-TCP initiator could be implemented as a ublk server today.
> > > > > > > What am I missing?
> > > > > > 
> > > > > > The current ublk can't do that yet, because the interface doesn't
> > > > > > support multiple ublk disks sharing single host, which is exactly
> > > > > > the case of scsi and nvme.
> > > > > 
> > > > > Can you give an example that shows exactly where a problem is hit?
> > > > > 
> > > > > I took a quick look at the ublk source code and didn't spot a place
> > > > > where it prevents a single ublk server process from handling multiple
> > > > > devices.
> > > > > 
> > > > > Regarding "host resources(such as tag)", can the ublk server deal with
> > > > > that in userspace? The Linux block layer doesn't have the concept of a
> > > > > "host", that would come in at the SCSI/NVMe level that's implemented in
> > > > > userspace.
> > > > > 
> > > > > I don't understand yet...
> > > > 
> > > > blk_mq_tag_set is embedded into driver host structure, and referred by queue
> > > > via q->tag_set, both scsi and nvme allocates tag in host/queue wide,
> > > > that said all LUNs/NSs share host/queue tags, current every ublk
> > > > device is independent, and can't shard tags.
> > > 
> > > Does this actually prevent ublk servers with multiple ublk devices or is
> > > it just sub-optimal?
> > 
> > It is former, ublk can't support multiple devices which share single host
> > because duplicated tag can be seen in host side, then io is failed.
> 
> The kernel sees two independent block devices so there is no issue
> within the kernel.

This way either wastes memory, or performance is bad since we can't
make a perfect queue depth for each ublk device.

> 
> Userspace can do its own hw tag allocation if there are shared storage
> controller resources (e.g. NVMe CIDs) to avoid duplicating tags.
> 
> Have I missed something?

Please look at lib/sbitmap.c and block/blk-mq-tag.c and see how many
hard issues fixed/reported in the past, and how much optimization done
in this area.

In theory hw tag allocation can be done in userspace, but just hard to
do efficiently:

1) it has been proved as one hard task for sharing data efficiently in
SMP, so don't reinvent wheel in userspace, and this work could take
much more efforts than extending current ublk interface, and just
fruitless

2) two times tag allocation slows down io path much

2) even worse for userspace allocation, cause task can be killed and
no cleanup is done, so tag leak can be caused easily


Thanks, 
Ming