Re: Accelerating non-standard disk types

[Raphael Norwitz <raphael.norwitz@nutanix.com>]

On Tue, May 17, 2022 at 04:29:17PM +0100, Stefan Hajnoczi wrote:
> On Mon, May 16, 2022 at 05:38:31PM +0000, Raphael Norwitz wrote:
> > Hey Stefan,
> > 
> > We've been thinking about ways to accelerate other disk types such as
> > SATA and IDE rather than translating to SCSI and using QEMU's iSCSI
> > driver, with existing and more performant backends such as SPDK. We
> > think there are some options worth exploring:
> > 
> > [1] Keep using the SCSI translation in QEMU but back vDisks with a
> > vhost-user-scsi or vhost-user-blk backend device.
> 
> If I understand correctly the idea is to have a QEMU Block Driver that
> connects to SPDK using vhost-user-scsi/blk?
>

Yes - the idea would be to introduce logic to translate SATA/IDE to SCSI
or block requests and send them via vhost-user-{scsi/blk} to SPDK or any
other vhost-user backend. Our thought is that this is doable today
whereas we may have to wait for QEMU to formally adopt libblkio before
proceeding with [3], and depending on timelines it may make sense to
implement [1] and then switch over to [3] later. Thoughts?

> > [2] Implement SATA and IDE emulation with vfio-user (likely with an SPDK
> > client?).
> 
> This is definitely the option with the lowest overhead. I'm not sure if
> implementing SATA and IDE emulation in SPDK is worth the effort for
> saving the last few cycles.
>

Agreed - it’s probably not worth exploring because of the amount of work
involved. One good argument would be that it may be better for security
in the multiprocess QEMU world, but to me that does not seem strong
enough to justify the work involved so I suggest we drop option [2].

> > [3] We've also been looking at your libblkio library. From your
> > description in
> > https://lists.gnu.org/archive/html/qemu-devel/2021-04/msg06146.html it
> > sounds like it may definitely play a role here, and possibly provide the
> > nessesary abstractions to back I/O from these emulated disks to any
> > backends we may want?
> 
> Kevin Wolf has contributed a vhost-user-blk driver for libblkio. This
> lets you achieve #1 using QEMU's libblkio Block Driver. The guest still
> sees IDE or SATA but instead of translating to iSCSI the I/O requests
> are sent over vhost-user-blk.
> 
> I suggest joining the libblkio chat and we can discuss how to set this
> up (the QEMU libblkio BlockDriver is not yet in qemu.git):
> https://matrix.to/#/#libblkio:matrix.org

Great - I have joined and will follow up there.

> 
> > We are planning to start a review of these options internally to survey
> > tradeoffs, potential timelines and practicality for these approaches. We
> > were also considering putting a submission together for KVM forum
> > describing our findings. Would you see any value in that?
> 
> I think it's always interesting to see performance results. I wonder if
> you have more cutting-edge optimizations or performance results you want
> to share at KVM Forum because IDE and SATA are more legacy/niche
> nowadays?
>

I realize I over-emphasized performance in my question - our larger goal
here is to align the data path for all disk types. We have some hope
that SATA can be sped up a bit, but it’s entirely possible that the MMIO
overhead will way outweigh and disk I/O improvements. Our thought was to
present a “Roadmap for supporting offload alternate disk types”, but
with your and Paolo’s response it seems like there isn’t enough material
to warrant a KVM talk and we should rather invest time in prototyping
and evaluating solutions.

> Stefan