* Re: Using RPMSG to communicate between host and guest drivers [not found] <133c1301-dd19-4cce-82dc-3e8ee145c594@cornelisnetworks.com> @ 2024-08-26 16:27 ` Dennis Dalessandro 2024-08-26 16:50 ` Mathieu Poirier 2024-08-26 23:45 ` Jason Gunthorpe 0 siblings, 2 replies; 8+ messages in thread From: Dennis Dalessandro @ 2024-08-26 16:27 UTC (permalink / raw) To: Doug Miller, linux-remoteproc Cc: Bjorn Andersson, Mathieu Poirier, OFED mailing list On 7/31/24 4:02 PM, Doug Miller wrote: > I am working on SR-IOV support for a new adapter which has shared > resources between the PF and VFs and requires an out-of-band (outside > the adapter) communication mechanism to manage those resources. I have > been looking at RPMSG as a mechanism to communicate between the driver > on a guest (VM) and the driver on the host OS (which "owns" the > resources). It appears to me that virtio is intended for communication > between guests and host, and RPMSG over virtio is what I want to use. > > Can anyone confirm that RPMSG is capable of doing what we need? If so, > I'll need some help figuring out how to use that from kernel device > drivers (I've not been able to find any examples of doing the > service/device side). If not, is there some other facility that is > better suited? Hi Bjorn and Mathieu, any advice here for Doug? Adding linux-rdma folks as that is where this will eventually target. -Denny ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Using RPMSG to communicate between host and guest drivers 2024-08-26 16:27 ` Using RPMSG to communicate between host and guest drivers Dennis Dalessandro @ 2024-08-26 16:50 ` Mathieu Poirier 2024-08-26 17:22 ` Doug Miller 2024-08-26 23:45 ` Jason Gunthorpe 1 sibling, 1 reply; 8+ messages in thread From: Mathieu Poirier @ 2024-08-26 16:50 UTC (permalink / raw) To: Dennis Dalessandro Cc: Doug Miller, linux-remoteproc, Bjorn Andersson, OFED mailing list Apologies for the late reply - this got lost in the vacation email backlog. On Mon, 26 Aug 2024 at 10:27, Dennis Dalessandro <dennis.dalessandro@cornelisnetworks.com> wrote: > > On 7/31/24 4:02 PM, Doug Miller wrote: > > I am working on SR-IOV support for a new adapter which has shared > > resources between the PF and VFs and requires an out-of-band (outside It would have been a good idea to let people know what "PF" and "VF" means to avoid confusion. > > the adapter) communication mechanism to manage those resources. I have > > been looking at RPMSG as a mechanism to communicate between the driver > > on a guest (VM) and the driver on the host OS (which "owns" the > > resources). It appears to me that virtio is intended for communication > > between guests and host, and RPMSG over virtio is what I want to use. > > Virtio is definitely the standard way to convey information between a host and a guest. You can specify as many virtqueues as needed (in-band and out-of-band) and it is widely supported. What information is conveyed by the virtqueues and how it gets conveyed is entirely up to the use case. Have a look at the specification of existing virtio drivers to get a better idea [1]. If the driver you are working with hasn't been standardised, I highly encourage you to submit a draft for it. If it has then add to the current specification. All that said, you could use RPMSG as the protocol that runs on top of the virtqueues - that should be fairly easy to do. Thanks, Mathieu [1]. https://docs.oasis-open.org/virtio/virtio/v1.2/csd01/virtio-v1.2-csd01.html > > Can anyone confirm that RPMSG is capable of doing what we need? If so, > > I'll need some help figuring out how to use that from kernel device > > drivers (I've not been able to find any examples of doing the > > service/device side). If not, is there some other facility that is > > better suited? > > Hi Bjorn and Mathieu, any advice here for Doug? Adding linux-rdma folks as that > is where this will eventually target. > > -Denny > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Using RPMSG to communicate between host and guest drivers 2024-08-26 16:50 ` Mathieu Poirier @ 2024-08-26 17:22 ` Doug Miller 2024-08-28 14:44 ` Mathieu Poirier 0 siblings, 1 reply; 8+ messages in thread From: Doug Miller @ 2024-08-26 17:22 UTC (permalink / raw) To: Mathieu Poirier, Dennis Dalessandro Cc: linux-remoteproc, Bjorn Andersson, OFED mailing list On 8/26/2024 11:50 AM, Mathieu Poirier wrote: > Apologies for the late reply - this got lost in the vacation email backlog. > > On Mon, 26 Aug 2024 at 10:27, Dennis Dalessandro > <dennis.dalessandro@cornelisnetworks.com> wrote: >> On 7/31/24 4:02 PM, Doug Miller wrote: >>> I am working on SR-IOV support for a new adapter which has shared >>> resources between the PF and VFs and requires an out-of-band (outside > It would have been a good idea to let people know what "PF" and "VF" > means to avoid confusion. "PF" refers to the Physical Function of the PCI adapter - that which exists always, regardless of whether SR-IOV is active. The "VF" refers to the virtual function(s) that are created when SR-IOV is enabled and configured. Typically, the VFs and the PF are assigned to different OS instances running in different VMs. So, the OS that owns the PF needs to be able to handle resource requests from the OSes that own the VFs (and also send notifications). > >>> the adapter) communication mechanism to manage those resources. I have >>> been looking at RPMSG as a mechanism to communicate between the driver >>> on a guest (VM) and the driver on the host OS (which "owns" the >>> resources). It appears to me that virtio is intended for communication >>> between guests and host, and RPMSG over virtio is what I want to use. >>> > Virtio is definitely the standard way to convey information between a > host and a guest. You can specify as many virtqueues as needed > (in-band and out-of-band) and it is widely supported. What > information is conveyed by the virtqueues and how it gets conveyed is > entirely up to the use case. Have a look at the specification of > existing virtio drivers to get a better idea [1]. If the driver you > are working with hasn't been standardised, I highly encourage you to > submit a draft for it. If it has then add to the current > specification. > > All that said, you could use RPMSG as the protocol that runs on top of > the virtqueues - that should be fairly easy to do. I had initially started looking at using virtio directly, but it looked like I was going to have to get a new device ID defined upstream and it would be a significant effort compared to using an existing facility. I then saw device ID VIRTIO_ID_RPMSG, which appears to be exactly what we'd have to create if we were defining a new virtio device for what we need. However, the problem has been understanding how to write code to provide the rpmsg "device" side. There does not appear to be any documentation and there is no example code to follow. It seems that the device side is typically contained in a GPU or accelerator, which was not written for a Linux kernel. So I have many questions on how (and when) to use the interfaces (rpmsg_register_device, rpmsg_create_channel, rpmsg_create_ept, rpmsg_find_device, ...). > > Thanks, > Mathieu > > [1]. https://docs.oasis-open.org/virtio/virtio/v1.2/csd01/virtio-v1.2-csd01.html > >>> Can anyone confirm that RPMSG is capable of doing what we need? If so, >>> I'll need some help figuring out how to use that from kernel device >>> drivers (I've not been able to find any examples of doing the >>> service/device side). If not, is there some other facility that is >>> better suited? >> Hi Bjorn and Mathieu, any advice here for Doug? Adding linux-rdma folks as that >> is where this will eventually target. >> >> -Denny >> External recipient ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Using RPMSG to communicate between host and guest drivers 2024-08-26 17:22 ` Doug Miller @ 2024-08-28 14:44 ` Mathieu Poirier 2024-08-28 15:34 ` Doug Miller 0 siblings, 1 reply; 8+ messages in thread From: Mathieu Poirier @ 2024-08-28 14:44 UTC (permalink / raw) To: Doug Miller Cc: Dennis Dalessandro, linux-remoteproc, Bjorn Andersson, OFED mailing list On Mon, 26 Aug 2024 at 11:22, Doug Miller <doug.miller@cornelisnetworks.com> wrote: > > On 8/26/2024 11:50 AM, Mathieu Poirier wrote: > > Apologies for the late reply - this got lost in the vacation email backlog. > > > > On Mon, 26 Aug 2024 at 10:27, Dennis Dalessandro > > <dennis.dalessandro@cornelisnetworks.com> wrote: > >> On 7/31/24 4:02 PM, Doug Miller wrote: > >>> I am working on SR-IOV support for a new adapter which has shared > >>> resources between the PF and VFs and requires an out-of-band (outside > > It would have been a good idea to let people know what "PF" and "VF" > > means to avoid confusion. > "PF" refers to the Physical Function of the PCI adapter - that which > exists always, regardless of whether SR-IOV is active. The "VF" refers > to the virtual function(s) that are created when SR-IOV is enabled and > configured. Typically, the VFs and the PF are assigned to different OS > instances running in different VMs. So, the OS that owns the PF needs to > be able to handle resource requests from the OSes that own the VFs (and > also send notifications). Thank you for the clarification. > > > >>> the adapter) communication mechanism to manage those resources. I have > >>> been looking at RPMSG as a mechanism to communicate between the driver > >>> on a guest (VM) and the driver on the host OS (which "owns" the > >>> resources). It appears to me that virtio is intended for communication > >>> between guests and host, and RPMSG over virtio is what I want to use. > >>> > > Virtio is definitely the standard way to convey information between a > > host and a guest. You can specify as many virtqueues as needed > > (in-band and out-of-band) and it is widely supported. What > > information is conveyed by the virtqueues and how it gets conveyed is > > entirely up to the use case. Have a look at the specification of > > existing virtio drivers to get a better idea [1]. If the driver you > > are working with hasn't been standardised, I highly encourage you to > > submit a draft for it. If it has then add to the current > > specification. > > > > All that said, you could use RPMSG as the protocol that runs on top of > > the virtqueues - that should be fairly easy to do. > I had initially started looking at using virtio directly, but it looked > like I was going to have to get a new device ID defined upstream and it > would be a significant effort compared to using an existing facility. I > then saw device ID VIRTIO_ID_RPMSG, which appears to be exactly what > we'd have to create if we were defining a new virtio device for what we > need. However, the problem has been understanding how to write code to > provide the rpmsg "device" side. There does not appear to be any > documentation and there is no example code to follow. It seems that the > device side is typically contained in a GPU or accelerator, which was > not written for a Linux kernel. So I have many questions on how (and > when) to use the interfaces (rpmsg_register_device, > rpmsg_create_channel, rpmsg_create_ept, rpmsg_find_device, ...). VIRTIO_ID_RPMSG is a special case - it was defined to establish a communication channel between a main processor (typically a cortex-A) and a remote processor, something like a M4 or an R5F. As such it is typically used in conjunction with the "remoteproc" subsystem. The device side you are looking for is part of the openAMP library [1]. I am not aware of an implementation of a virtio device that would use VIRTIO_ID_RPMSG in a MMIO area or a PCI config space to instantiate a generic message passing interface. [1]. https://github.com/OpenAMP/open-amp > > > > Thanks, > > Mathieu > > > > [1]. https://docs.oasis-open.org/virtio/virtio/v1.2/csd01/virtio-v1.2-csd01.html > > > >>> Can anyone confirm that RPMSG is capable of doing what we need? If so, > >>> I'll need some help figuring out how to use that from kernel device > >>> drivers (I've not been able to find any examples of doing the > >>> service/device side). If not, is there some other facility that is > >>> better suited? > >> Hi Bjorn and Mathieu, any advice here for Doug? Adding linux-rdma folks as that > >> is where this will eventually target. > >> > >> -Denny > >> > > External recipient ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Using RPMSG to communicate between host and guest drivers 2024-08-28 14:44 ` Mathieu Poirier @ 2024-08-28 15:34 ` Doug Miller 0 siblings, 0 replies; 8+ messages in thread From: Doug Miller @ 2024-08-28 15:34 UTC (permalink / raw) To: Mathieu Poirier Cc: Dennis Dalessandro, linux-remoteproc, Bjorn Andersson, OFED mailing list On 8/28/2024 9:44 AM, Mathieu Poirier wrote: > On Mon, 26 Aug 2024 at 11:22, Doug Miller > <doug.miller@cornelisnetworks.com> wrote: >> On 8/26/2024 11:50 AM, Mathieu Poirier wrote: >>> Apologies for the late reply - this got lost in the vacation email backlog. >>> >>> On Mon, 26 Aug 2024 at 10:27, Dennis Dalessandro >>> <dennis.dalessandro@cornelisnetworks.com> wrote: >>>> On 7/31/24 4:02 PM, Doug Miller wrote: >>>>> I am working on SR-IOV support for a new adapter which has shared >>>>> resources between the PF and VFs and requires an out-of-band (outside >>> It would have been a good idea to let people know what "PF" and "VF" >>> means to avoid confusion. >> "PF" refers to the Physical Function of the PCI adapter - that which >> exists always, regardless of whether SR-IOV is active. The "VF" refers >> to the virtual function(s) that are created when SR-IOV is enabled and >> configured. Typically, the VFs and the PF are assigned to different OS >> instances running in different VMs. So, the OS that owns the PF needs to >> be able to handle resource requests from the OSes that own the VFs (and >> also send notifications). > Thank you for the clarification. > >>>>> the adapter) communication mechanism to manage those resources. I have >>>>> been looking at RPMSG as a mechanism to communicate between the driver >>>>> on a guest (VM) and the driver on the host OS (which "owns" the >>>>> resources). It appears to me that virtio is intended for communication >>>>> between guests and host, and RPMSG over virtio is what I want to use. >>>>> >>> Virtio is definitely the standard way to convey information between a >>> host and a guest. You can specify as many virtqueues as needed >>> (in-band and out-of-band) and it is widely supported. What >>> information is conveyed by the virtqueues and how it gets conveyed is >>> entirely up to the use case. Have a look at the specification of >>> existing virtio drivers to get a better idea [1]. If the driver you >>> are working with hasn't been standardised, I highly encourage you to >>> submit a draft for it. If it has then add to the current >>> specification. >>> >>> All that said, you could use RPMSG as the protocol that runs on top of >>> the virtqueues - that should be fairly easy to do. >> I had initially started looking at using virtio directly, but it looked >> like I was going to have to get a new device ID defined upstream and it >> would be a significant effort compared to using an existing facility. I >> then saw device ID VIRTIO_ID_RPMSG, which appears to be exactly what >> we'd have to create if we were defining a new virtio device for what we >> need. However, the problem has been understanding how to write code to >> provide the rpmsg "device" side. There does not appear to be any >> documentation and there is no example code to follow. It seems that the >> device side is typically contained in a GPU or accelerator, which was >> not written for a Linux kernel. So I have many questions on how (and >> when) to use the interfaces (rpmsg_register_device, >> rpmsg_create_channel, rpmsg_create_ept, rpmsg_find_device, ...). > VIRTIO_ID_RPMSG is a special case - it was defined to establish a > communication channel between a main processor (typically a cortex-A) > and a remote processor, something like a M4 or an R5F. As such it is > typically used in conjunction with the "remoteproc" subsystem. The > device side you are looking for is part of the openAMP library [1]. I > am not aware of an implementation of a virtio device that would use > VIRTIO_ID_RPMSG in a MMIO area or a PCI config space to instantiate a > generic message passing interface. > > [1]. https://github.com/OpenAMP/open-amp I had been looking into OpenAMP, but was stuck on the examples which appear to be running from userland and do not appear to be what is needed for kernel modules. I'll look at the larger project to see if there is something I can use. I had assumed, and obviously could be wrong, that the fact that VIRTIO_ID_RPMSG was using virtio meant it was suited for VM-host communications. I was looking at the interfaces that are enabled via CONFIG_RPMSG and those appeared to be what was needed for the device side, although it's not clear to me how they are used. I ran across "pci-hyperv" which looks like it might be providing the same communications path, but the only use case I'm finding is mlx5 and I wonder if this is really intended for general (future) use. Also, it's not clear to me yet just how the host side of that works (yet). > >>> Thanks, >>> Mathieu >>> >>> [1]. https://docs.oasis-open.org/virtio/virtio/v1.2/csd01/virtio-v1.2-csd01.html >>> >>>>> Can anyone confirm that RPMSG is capable of doing what we need? If so, >>>>> I'll need some help figuring out how to use that from kernel device >>>>> drivers (I've not been able to find any examples of doing the >>>>> service/device side). If not, is there some other facility that is >>>>> better suited? >>>> Hi Bjorn and Mathieu, any advice here for Doug? Adding linux-rdma folks as that >>>> is where this will eventually target. >>>> >>>> -Denny >>>> >> External recipient External recipient ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Using RPMSG to communicate between host and guest drivers 2024-08-26 16:27 ` Using RPMSG to communicate between host and guest drivers Dennis Dalessandro 2024-08-26 16:50 ` Mathieu Poirier @ 2024-08-26 23:45 ` Jason Gunthorpe 2024-08-28 11:49 ` Doug Miller 1 sibling, 1 reply; 8+ messages in thread From: Jason Gunthorpe @ 2024-08-26 23:45 UTC (permalink / raw) To: Dennis Dalessandro Cc: Doug Miller, linux-remoteproc, Bjorn Andersson, Mathieu Poirier, OFED mailing list On Mon, Aug 26, 2024 at 12:27:02PM -0400, Dennis Dalessandro wrote: > On 7/31/24 4:02 PM, Doug Miller wrote: > > I am working on SR-IOV support for a new adapter which has shared > > resources between the PF and VFs and requires an out-of-band (outside > > the adapter) communication mechanism to manage those resources. I have > > been looking at RPMSG as a mechanism to communicate between the driver > > on a guest (VM) and the driver on the host OS (which "owns" the > > resources). It appears to me that virtio is intended for communication > > between guests and host, and RPMSG over virtio is what I want to use. > > > > Can anyone confirm that RPMSG is capable of doing what we need? If so, > > I'll need some help figuring out how to use that from kernel device > > drivers (I've not been able to find any examples of doing the > > service/device side). If not, is there some other facility that is > > better suited? > > Hi Bjorn and Mathieu, any advice here for Doug? Adding linux-rdma folks as that > is where this will eventually target. Typically in cases like this you'd paravirtualize some of the VF before sticking it in the VM so that there is a tidy channel between the VF driver and the VMM to do whatever this coordination is. There are many examples, but it is hard to see if you don't know the device architectures in detail. If you stick it in seperate virtio PCI device you'll have hard problems co-ordinating the two drivers. Jason ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Using RPMSG to communicate between host and guest drivers 2024-08-26 23:45 ` Jason Gunthorpe @ 2024-08-28 11:49 ` Doug Miller 2024-08-28 14:17 ` Jason Gunthorpe 0 siblings, 1 reply; 8+ messages in thread From: Doug Miller @ 2024-08-28 11:49 UTC (permalink / raw) To: Jason Gunthorpe, Dennis Dalessandro Cc: linux-remoteproc, Bjorn Andersson, Mathieu Poirier, OFED mailing list On 8/26/2024 6:45 PM, Jason Gunthorpe wrote: > On Mon, Aug 26, 2024 at 12:27:02PM -0400, Dennis Dalessandro wrote: >> On 7/31/24 4:02 PM, Doug Miller wrote: >>> I am working on SR-IOV support for a new adapter which has shared >>> resources between the PF and VFs and requires an out-of-band (outside >>> the adapter) communication mechanism to manage those resources. I have >>> been looking at RPMSG as a mechanism to communicate between the driver >>> on a guest (VM) and the driver on the host OS (which "owns" the >>> resources). It appears to me that virtio is intended for communication >>> between guests and host, and RPMSG over virtio is what I want to use. >>> >>> Can anyone confirm that RPMSG is capable of doing what we need? If so, >>> I'll need some help figuring out how to use that from kernel device >>> drivers (I've not been able to find any examples of doing the >>> service/device side). If not, is there some other facility that is >>> better suited? >> Hi Bjorn and Mathieu, any advice here for Doug? Adding linux-rdma folks as that >> is where this will eventually target. > Typically in cases like this you'd paravirtualize some of the VF > before sticking it in the VM so that there is a tidy channel between > the VF driver and the VMM to do whatever this coordination is. There > are many examples, but it is hard to see if you don't know the device > architectures in detail. Can you give more detail on how this paravirtualization is accomplished? Or point to an example? It seems to me that rpmsg would be a cleaner solution, at least until I can see how paravirtualization is implemented for comparison. > > If you stick it in seperate virtio PCI device you'll have hard > problems co-ordinating the two drivers. > > Jason External recipient ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Using RPMSG to communicate between host and guest drivers 2024-08-28 11:49 ` Doug Miller @ 2024-08-28 14:17 ` Jason Gunthorpe 0 siblings, 0 replies; 8+ messages in thread From: Jason Gunthorpe @ 2024-08-28 14:17 UTC (permalink / raw) To: Doug Miller Cc: Dennis Dalessandro, linux-remoteproc, Bjorn Andersson, Mathieu Poirier, OFED mailing list On Wed, Aug 28, 2024 at 06:49:27AM -0500, Doug Miller wrote: > > Typically in cases like this you'd paravirtualize some of the VF > > before sticking it in the VM so that there is a tidy channel between > > the VF driver and the VMM to do whatever this coordination is. There > > are many examples, but it is hard to see if you don't know the device > > architectures in detail. > > Can you give more detail on how this paravirtualization is accomplished? > Or point to an example? Not sure I have easy examples, several devices I've seen proxy things through their FW/HW. I think some of the Intel NICs were working like this. Some have full vfio-mdevs or qemu modules to intercept registers and build it. > It seems to me that rpmsg would be a cleaner solution, at least > until I can see how paravirtualization is implemented for > comparison. I think you will find it very hard to connect such a thing up throughout all the peices of software. Having two related devices across everything is not easy. Jason ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2024-08-28 15:34 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <133c1301-dd19-4cce-82dc-3e8ee145c594@cornelisnetworks.com>
2024-08-26 16:27 ` Using RPMSG to communicate between host and guest drivers Dennis Dalessandro
2024-08-26 16:50 ` Mathieu Poirier
2024-08-26 17:22 ` Doug Miller
2024-08-28 14:44 ` Mathieu Poirier
2024-08-28 15:34 ` Doug Miller
2024-08-26 23:45 ` Jason Gunthorpe
2024-08-28 11:49 ` Doug Miller
2024-08-28 14:17 ` Jason Gunthorpe
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox