Re: [LSF/MM/BPF BoF]: extend UBLK to cover real storage hardware

[Stefan Hajnoczi <stefanha@redhat.com>]

[-- Attachment #1: Type: text/plain, Size: 4310 bytes --]

On Thu, Feb 16, 2023 at 08:46:56AM +0800, Ming Lei wrote:
> On Wed, Feb 15, 2023 at 10:27:07AM -0500, Stefan Hajnoczi wrote:
> > On Wed, Feb 15, 2023 at 08:51:27AM +0800, Ming Lei wrote:
> > > On Mon, Feb 13, 2023 at 02:13:59PM -0500, Stefan Hajnoczi wrote:
> > > > On Mon, Feb 13, 2023 at 11:47:31AM +0800, Ming Lei wrote:
> > > > > On Wed, Feb 08, 2023 at 07:17:10AM -0500, Stefan Hajnoczi wrote:
> > > > > > On Wed, Feb 08, 2023 at 10:12:19AM +0800, Ming Lei wrote:
> > > > > > > On Mon, Feb 06, 2023 at 03:27:09PM -0500, Stefan Hajnoczi wrote:
> > > > > > > > On Mon, Feb 06, 2023 at 11:00:27PM +0800, Ming Lei wrote:
> > > > > > > > > Hello,
> > > > > > > > > 
> > > > > > > > > So far UBLK is only used for implementing virtual block device from
> > > > > > > > > userspace, such as loop, nbd, qcow2, ...[1].
> > > > > > > > 
> > > > > > > > I won't be at LSF/MM so here are my thoughts:
> > > > > > > 
> > > > > > > Thanks for the thoughts, :-)
> > > > > > > 
> > > > > > > > 
> > > > > > > > > 
> > > > > > > > > It could be useful for UBLK to cover real storage hardware too:
> > > > > > > > > 
> > > > > > > > > - for fast prototype or performance evaluation
> > > > > > > > > 
> > > > > > > > > - some network storages are attached to host, such as iscsi and nvme-tcp,
> > > > > > > > > the current UBLK interface doesn't support such devices, since it needs
> > > > > > > > > all LUNs/Namespaces to share host resources(such as tag)
> > > > > > > > 
> > > > > > > > Can you explain this in more detail? It seems like an iSCSI or
> > > > > > > > NVMe-over-TCP initiator could be implemented as a ublk server today.
> > > > > > > > What am I missing?
> > > > > > > 
> > > > > > > The current ublk can't do that yet, because the interface doesn't
> > > > > > > support multiple ublk disks sharing single host, which is exactly
> > > > > > > the case of scsi and nvme.
> > > > > > 
> > > > > > Can you give an example that shows exactly where a problem is hit?
> > > > > > 
> > > > > > I took a quick look at the ublk source code and didn't spot a place
> > > > > > where it prevents a single ublk server process from handling multiple
> > > > > > devices.
> > > > > > 
> > > > > > Regarding "host resources(such as tag)", can the ublk server deal with
> > > > > > that in userspace? The Linux block layer doesn't have the concept of a
> > > > > > "host", that would come in at the SCSI/NVMe level that's implemented in
> > > > > > userspace.
> > > > > > 
> > > > > > I don't understand yet...
> > > > > 
> > > > > blk_mq_tag_set is embedded into driver host structure, and referred by queue
> > > > > via q->tag_set, both scsi and nvme allocates tag in host/queue wide,
> > > > > that said all LUNs/NSs share host/queue tags, current every ublk
> > > > > device is independent, and can't shard tags.
> > > > 
> > > > Does this actually prevent ublk servers with multiple ublk devices or is
> > > > it just sub-optimal?
> > > 
> > > It is former, ublk can't support multiple devices which share single host
> > > because duplicated tag can be seen in host side, then io is failed.
> > 
> > The kernel sees two independent block devices so there is no issue
> > within the kernel.
> 
> This way either wastes memory, or performance is bad since we can't
> make a perfect queue depth for each ublk device.
> 
> > 
> > Userspace can do its own hw tag allocation if there are shared storage
> > controller resources (e.g. NVMe CIDs) to avoid duplicating tags.
> > 
> > Have I missed something?
> 
> Please look at lib/sbitmap.c and block/blk-mq-tag.c and see how many
> hard issues fixed/reported in the past, and how much optimization done
> in this area.
> 
> In theory hw tag allocation can be done in userspace, but just hard to
> do efficiently:
> 
> 1) it has been proved as one hard task for sharing data efficiently in
> SMP, so don't reinvent wheel in userspace, and this work could take
> much more efforts than extending current ublk interface, and just
> fruitless
> 
> 2) two times tag allocation slows down io path much
> 
> 2) even worse for userspace allocation, cause task can be killed and
> no cleanup is done, so tag leak can be caused easily

So then it is not "the former" after all?

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]