Re: How to create/use RPMSG-over-VIRTIO devices in Linux

[Jason Gunthorpe <jgg@ziepe.ca>]

On Fri, Sep 20, 2024 at 08:56:07AM -0500, Doug Miller wrote:
> On 9/20/2024 7:45 AM, Jason Gunthorpe wrote:
> > On Mon, Sep 16, 2024 at 08:38:42AM -0500, Doug Miller wrote:
> > > On 9/15/2024 11:58 AM, Jason Gunthorpe wrote:
> > > > On Fri, Sep 13, 2024 at 08:39:26AM -0600, Mathieu Poirier wrote:
> > > > > KVM has nothing to do with this.  The life of a virtio device starts
> > > > > in the VMM (Virtual Machine Manager) where a backend device is created
> > > > > and a virtio MMIO entry for that device is added to the device tree
> > > > > that is fed to the VM kernel.  When the VM kernel boots the virtio
> > > > > MMIO entry in the DT is parsed as part of the normal device discovery
> > > > > process and a virtio-device is instantiated, added to the virtio-bus
> > > > > and a driver is probed.
> > > > > 
> > > > > I suggest you start looking at that process using the kvmtool and a
> > > > > simple virtio device such as virtio-rng.
> > > > I would repeat again, I think trying to create a companion virtio
> > > > device to go along with a real vPCI device and then logically
> > > > associating both of them with a single driver is going to cause so
> > > > much pain you should not do it.
> > > > 
> > > > Find a way to send your RPCs through your own vPCI device.
> > > When you say "your own vPCI device", are you referring to the virtual
> > > functions that are created by the adapter? Those are defined by the
> > > hardware specification and we don't have the ability to extend them.
> > Yes you do the VMM can extend them. You need some qemu code or a
> > vfio-cornelis or something like that.

> We can't require that SR-IOV customers use one specific VMM (qemu). 

No matter what you do, you will require some modification. The other
end of any imagined virtio will be in qemu too after all.

> you're talking about modifying the kernel virtio_pci facility, that may
> be a worthwhile effort long term but I suspect it will take a longer
> time to get adopted. Using an existing kernel facility as-is would be
> the most practical solution.

There is really nothing.

> > > What we're investigating is using RPMSG-over-VIRTIO, not using virtio
> > > devices directly.
> > I understand, and that will be very painful.
> Can you expand on what you mean by "painful"? Are you speaking from
> experience with the rpmsg interfaces (can you point to problem areas)?
> Or is this based on the fact that, as of yet, no one has come forward to
> explain exactly how to do this?

From understanding how vfio works and knowing that if you try to tie w
virtio and your PCI together it will be a huge mess.

Jason