Re: Re: [virtio-comment] About adding a new device type virtio-nvme

[Stefan Hajnoczi <stefanha@redhat.com>]

[-- Attachment #1: Type: text/plain, Size: 5499 bytes --]

On Thu, Jan 19, 2023 at 07:02:00PM +0800, 侯英乐 wrote:
> Thu, 19 Jan 2023 12:33:54 +0200, Max Gurtovoy wrote:
> >On 19/01/2023 12:19, 侯英乐 wrote:
> 
> >> Wed, 18 Jan 2023 12:09:59 +0200, Max Gurtovoy wrote:
> 
> >>> On 18/01/2023 5:23, 侯英乐 wrote:
> 
> >>>> On  Tue, 17 Jan 2023 19:19:59 +0200, Max Gurtovoy wrote:
> 
> >>>>> On 17/01/2023 4:04, 侯英乐 wrote:
> 
> >>>>>> On Wed, 11 Jan 2023 10:16:55 -0500, Stefan wrote:
> 
> >>>>>>>> On Wed, Jan 11, 2023 at 11:21:35AM +0800, 侯英乐 wrote:
> 
> >>>>>>>> As we know, nvme has more features than virtio-blk. For example, with the development of virtualization IO offloading to hardware, virtio-blk and NVME-OF offloading to hardware >are developing rapidly.  So if virtio and nvme are combined into Virtio-NvMe, Is it necessary to add a device type Virtio-NvMe ?
> 
> >>>>>>    
> 
> >>>>>>> Hi,
> 
> >>>>>>> In theory, yes, virtio-nvme can be done. The question is why do it?
> 
> >>>>>>> NVMe already provides a PCI hardware spec for software and hardware
> 
> >>>>>>> implementations to follow. An NVMe PCI device can be exposed to the
> 
> >>>>>>> guest and modern operating systems recognize it without requiring new
> 
> >>>>>>> drivers.
> 
> >>>>>>> The value of VIRTIO here is probably in the deep integration into the
> 
> >>>>>>> virtualization stack with vDPA, vhost, etc. A virtio-nvme device can use
> 
> >>>>>>> all these things whereas a PCI device needs to do everything from
> 
> >>>>>>> scratch.
> 
> >>>>>> The NVME technology and ecosystem are complete. However, in virtualization scenarios, NVME devices can only use PCIe pass-through . When NVME and virtio combine to connect to the vDPA ecosystem, live migration is supported.
> 
> >>>>>>> Let's not forget that virtio-blk is widely used and new commands are
> 
> >>>>>>> being added as needed. Which NVMe features are you missing in
> 
> >>>>>>> virtio-blk?
> 
> >>>>>> With the introduction of the concept of DPU, a large number of vendors are offloading virtual devices to hardware. The back-end of Virtio-blk does not support remote storage. Therefore, Virtio-Nvme-of can well combine the advantages of remote storage and virtio live migration
> 
> >>>>>>> I guess this is why virtio-nvme hasn't been done before: people who want
> 
> >>>>>> NVMe can already do NVMe PCI, people who want VIRTIO can use virtio-blk,
> 
> >>>>>>> and so there hasn't been a great need to combine VIRTIO and NVMe yet.
> 
> >>>>>>> What advantages do you see in having virtio-nvme?
> 
> >>>>>> virtio-nvme advantages :
> 
> >>>>>> 1)  live migration
> 
> >>>>>   
> 
> >>>>> This is WIP and will use VFIO live migration framework.
> 
> >>>> Yes, VFIO live migration framework is WIP,  but I still think vdpa is a friendlier framework.
> 
> >>
> 
> >>
> 
> >>> Not sure what you consider friendly ?
> 
> >> My personal opinion: VFIO live migration requires device design requirements.
> 
> >> But vDPA-based live migration, the software-abstracted vDPA device in the vDPA
> 
> >> framework can do some state recording, The design requirements for virtio devices
> 
> >> that are offloading to hardware may be lower.
> 
> >>
> 
> >>  
> 
> >>
> 
> >>
> 
> >>> The community agreed that in SR-IOV - VF migration is done via PF interface.
> 
> >>
> 
> >>
> 
> >>> Any device specific migration (e.g. vdpa/virtio)  is not as generic as
> 
> >>> VFIO migration. Also it will be maintained by a smaller group of engineers.
> 
> >>
> 
> >>> If you would like to use vdpa - I suggest using virtio-blk and not
> 
> >>> inventing virtio-nvme device that will for sure be with less feature set
> 
> >>> than pure NVMe.
> 
> >>
> 
> >>
> 
> >>> In case you're missing some feature in virtio-blk that exist in NVMe,
> 
> >>> you're welcome to submit a proposal to the technical group with that
> 
> >>> feature.
> 
> >>
> 
> >> Yes, this is good advice.
> 
> >> virtio-blk adds Fabrics related commands to enable virtio-blk to support
> 
> >> virtio-blk-of (over Fabric), I wonder if it is feasible.
> 
> > 
> 
> >I'm totally confused.
> 
> >I thought you're are trying to build some virtualized environment and
> 
> >you're looking for storage devices that support Live migration.
> 
> >How does virtio-blk-of will assist here ?
> 
> 
> 
> I would like to push virtio storage in hardware offloading scenarios, 
> enabling open source solutions that support remote storage access.
> So we're talking about the need for virtio-nvme and virtio-blk-of.

If I understand correctly you're saying the guest driver needs to speak
the same protocol as the remote storage?

That's a good idea for local storage because it avoids extra layers of
software that parses/translates commands.

However, I don't understand why it matters for remote storage because
commands needs to be parsed by the DPU and sent as messages over a
fabric anyway. Whether you go virtio-blk<->NVMeoF,
virtio-blk<->virtio-blk-of, or nvme-pci<->NVMeoF, it's still the same
path. None of them presents a significant optimization opportunity.

The main optimization is to configure some sort of RDMA to avoid copying
around I/O buffers, but the buffers only contain data and are not
protocol-specific so virtio-blk<->NVMeoF should work.

Can you explain what you wrote in a bit more detail, I don't understand
why virtio-blk-of is needed?

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]