Re: MSI for Virtio PCI transport

[Manivannan Sadhasivam <manisadhasivam.linux@gmail.com>]

On Tue, Jun 25, 2024 at 03:52:30AM -0400, Michael S. Tsirkin wrote:
> On Mon, Jun 24, 2024 at 09:49:57PM +0530, Manivannan Sadhasivam wrote:
> > Hi,
> > 
> > We are looking into adapting Virtio spec for configurable physical PCIe endpoint
> > devices to expose Virtio devices to the host machine connected over PCIe. This
> > allows us to use the existing frontend drivers on the host machine, thus
> > minimizing the development efforts. This idea is not new as some vendors like
> > NVidia have already released customized PCIe devices exposing Virtio devices to
> > the host machines. But we are working on making the configurable PCIe devices
> > running Linux kernel to expose Virtio devices using the PCI Endpoint (EP)
> > subsystem.
> > 
> > Below is the simplistic represenation of the idea with virt-net as an
> > example. But this could be extended to any supported Virtio devices:
> > 
> >            HOST                                    ENDPOINT           
> > 
> > +-----------------------------+         +-----------------------------+
> > |                             |         |                             |
> > |         Linux Kernel        |         |         Linux Kernel        |
> > |                             |         |                             |
> > |                             |         |     +------------------+    |
> > |                             |         |     |                  |    | 
> > |                             |         |     |       Modem      |    |
> > |                             |         |     |                  |    |
> > |                             |         |     +---------|--------+    |
> > |                             |         |               |             |
> > |     +------------------+    |         |     +---------|--------+    |
> > |     |                  |    |         |     |                  |    |
> > |     |     Virt-net     |    |         |     |    Virtio EPF    |    |
> > |     |                  |    |         |     |                  |    |
> > |     +---------|--------+    |         |     +---------|--------+    |
> > |               |             |         |               |             |
> > |     +---------|--------+    |         |     +---------|--------+    |
> > |     |                  |    |         |     |                  |    |
> > |     |    Virtio PCI    |    |         |     | PCI EP Subsystem |    |
> > |     |                  |    |         |     |                  |    |
> > |     +---------|--------+    |         |     +---------|--------+    |
> > | SW            |             |         | SW            |             |
> > ----------------|--------------         ----------------|--------------
> > | HW            |             |         | HW            |             |
> > |     +---------|--------+    |         |     +---------|--------+    |
> > |     |                  |    |         |     |                  |    |
> > |     |      PCIe RC     |    |         |     |     PCIe EP      |    |
> > |     |                  |    |         |     |                  |    |
> > +-----+---------|--------+----+         +-----+---------|--------+----+
> >                 |                                       |               
> >                 |                                       |               
> >                 |                                       |               
> >                 |                PCIe                   |               
> >                 -----------------------------------------               
> > 
> > While doing so, we faced an issue due to lack of MSI support defined in Virtio
> > spec for PCI transport. Currently, the PCI transport (starting from 0.9.5) has
> > only defined INTx (legacy) and MSI-X interrupts for the device to send
> > notifications to the guest. While it works well for the hypervisor to guest
> > communcation, when a physical PCIe device is used as a Virtio device, lack of
> > MSI support is hurting the performance (when there is no MSI-X).
> > 
> > Most of the physical PCIe endpoint devices support MSI interrupts over MSI-X for
> > simplicity and with Virtio not supporting MSI, falling back to legacy INTx
> > interrupts is affecting the performance.
> > 
> > First of all, INTx requires the PCIe devices to send two MSG TLPs
> > (Assert/Deassert) to emulate level triggered interrupt on the host. And there
> > could be some delay between assert and deassert messages to make sure that the
> > host recognizes it as an interrupt (level trigger). Also, the INTx interrupts
> > are limited to 1 per function, so all the notifications from device has to share
> > this single interrupt (INTA).
> > 
> > On the other hand, MSI requires only one MWr TLP from the device to host and
> > since it is a posted write, there is no delay involved. Also, a single PCIe
> > function can use upto 32 MSIs, thus making it possible to use one MSI vector per
> > virtqueue (32 is more than enough for most of the usecases).
> > 
> > So my question is, why does the Virtio spec not supporting MSI? If there are no
> > major blocker in supporting MSI, could we propose adding MSI to the Virtio spec?
> > 
> > - Mani
> > 
> > -- 
> > மணிவண்ணன் சதாசிவம்
> 
> Yes, it's possible to add - however, you also said EP requires more
> changes from virtio. So maybe we need "virtio over EP" then.

I don't think we need a separate 'Virtio over EP'. EP uses PCI transport, so
'Virtio over PCI' is fine as it is. Just because 'Virtio over PCI' was designed
based on virtual PCI devices exposed by the hypervisor, the real world
limitations were not taken into account. And that's what we are trying to add.

> Let's try to figure out the full list of issues, to see which makes
> more sense.
> 

Sure. But each one would need a separate discussion, that's why I started this
thread for MSI. Let me check with Shunsuke and come up with an exhaustive list.

- Mani

-- 
மணிவண்ணன் சதாசிவம்