How to create/use RPMSG-over-VIRTIO devices in Linux

How to create/use RPMSG-over-VIRTIO devices in Linux

[Doug Miller]

I am trying to learn how to create an RPMSG-over-VIRTIO device (service)
in order to perform communication between a host driver and a guest
driver. The RPMSG-over-VIRTIO driver (client) side is fairly well
documented and there is a good example (starting point, at least) in
samples/rpmsg/rpmsg_client_sample.c.

I see that I can create an endpoint (struct rpmsg_endpoint) using
rpmsg_create_ept(), and from there I can use rpmsg_send() et al. and the
rpmsg_rx_cb_t cb to perform the communications. However, this requires a
struct rpmsg_device and it is not clear just how to get one that is
suitable for this purpose.

It appears that one or both of rpmsg_create_channel() and
rpmsg_register_device() are needed in order to obtain a device for the
specific host-guest communications channel. At some point, a "root"
device is needed that will use virtio (VIRTIO_ID_RPMSG) such that new
subdevices can be created for each host-guest pair.

In addition, building a kernel with CONFIG_RPMSG, CONFIG_RPMSG_VIRTIO,
and CONFIG_RPMSG_NS set, and doing a modprobe virtio_rpmsg_bus, seems to
get things setup but that does not result in creation of any "root"
rpmsg-over-virtio device. Presumably, any such device would have to be
setup to use a specific range of addresses and also be tied to
virtio_rpmsg_bus to ensure that virtio is used.

It is also not clear if/how register_rpmsg_driver() will be required on
the rpmsg driver side, even though the sample code does not use it.

So, first questions are:

* Am I looking at the correct interfaces in order to create the host
rpmsg device side?
* What needs to be done to get a "root" rpmsg-over-virtio device created
(if required)?
* How is a rpmsg-over-virtio device created for each host-guest driver
pair, for use with rpmsg_create_ept()?
* Does the guest side (rpmsg driver) require any special handling to
plug-in to the host driver (rpmsg device) side? Aside from using the
correct addresses to match device side.

Thanks,
Doug

External recipient

Re: How to create/use RPMSG-over-VIRTIO devices in Linux

[Doug Miller]

On 9/3/2024 10:52 AM, Doug Miller wrote:
> I am trying to learn how to create an RPMSG-over-VIRTIO device
> (service) in order to perform communication between a host driver and
> a guest driver. The RPMSG-over-VIRTIO driver (client) side is fairly
> well documented and there is a good example (starting point, at least)
> in samples/rpmsg/rpmsg_client_sample.c.
>
> I see that I can create an endpoint (struct rpmsg_endpoint) using
> rpmsg_create_ept(), and from there I can use rpmsg_send() et al. and
> the rpmsg_rx_cb_t cb to perform the communications. However, this
> requires a struct rpmsg_device and it is not clear just how to get one
> that is suitable for this purpose.
>
> It appears that one or both of rpmsg_create_channel() and
> rpmsg_register_device() are needed in order to obtain a device for the
> specific host-guest communications channel. At some point, a "root"
> device is needed that will use virtio (VIRTIO_ID_RPMSG) such that new
> subdevices can be created for each host-guest pair.
>
> In addition, building a kernel with CONFIG_RPMSG, CONFIG_RPMSG_VIRTIO,
> and CONFIG_RPMSG_NS set, and doing a modprobe virtio_rpmsg_bus, seems
> to get things setup but that does not result in creation of any "root"
> rpmsg-over-virtio device. Presumably, any such device would have to be
> setup to use a specific range of addresses and also be tied to
> virtio_rpmsg_bus to ensure that virtio is used.
>
> It is also not clear if/how register_rpmsg_driver() will be required
> on the rpmsg driver side, even though the sample code does not use it.
>
> So, first questions are:
>
> * Am I looking at the correct interfaces in order to create the host
> rpmsg device side?
> * What needs to be done to get a "root" rpmsg-over-virtio device
> created (if required)?
> * How is a rpmsg-over-virtio device created for each host-guest driver
> pair, for use with rpmsg_create_ept()?
> * Does the guest side (rpmsg driver) require any special handling to
> plug-in to the host driver (rpmsg device) side? Aside from using the
> correct addresses to match device side.

It looks to me as though the virtio_rpmsg_bus module can create a
"rpmsg_ctl" device, which could be used to create channels from which
endpoints could be created. However, when I load the virtio_rpmsg_bus,
rpmsg_ns, and rpmsg_core modules there is no "rpmsg_ctl" device created
(this is running in the host OS, before any VMs are created/run).

Is this the correct way to use RPMSG-over-VIRTIO? If so, what actions
need to be taken to cause a "rpmsg_ctl" device to be created? What
method would be used (in a kernel driver) to get a pointer to the
"rpmsg_ctl" device, for use with rpmsg_create_channel()?

>
> Thanks,
> Doug
>

External recipient

Re: How to create/use RPMSG-over-VIRTIO devices in Linux

[Mathieu Poirier]

On Tue, Sep 10, 2024 at 08:12:07AM -0500, Doug Miller wrote:
> On 9/3/2024 10:52 AM, Doug Miller wrote:
> > I am trying to learn how to create an RPMSG-over-VIRTIO device
> > (service) in order to perform communication between a host driver and
> > a guest driver. The RPMSG-over-VIRTIO driver (client) side is fairly
> > well documented and there is a good example (starting point, at least)
> > in samples/rpmsg/rpmsg_client_sample.c.
> > 
> > I see that I can create an endpoint (struct rpmsg_endpoint) using
> > rpmsg_create_ept(), and from there I can use rpmsg_send() et al. and
> > the rpmsg_rx_cb_t cb to perform the communications. However, this
> > requires a struct rpmsg_device and it is not clear just how to get one
> > that is suitable for this purpose.
> > 
> > It appears that one or both of rpmsg_create_channel() and
> > rpmsg_register_device() are needed in order to obtain a device for the
> > specific host-guest communications channel. At some point, a "root"
> > device is needed that will use virtio (VIRTIO_ID_RPMSG) such that new
> > subdevices can be created for each host-guest pair.
> > 
> > In addition, building a kernel with CONFIG_RPMSG, CONFIG_RPMSG_VIRTIO,
> > and CONFIG_RPMSG_NS set, and doing a modprobe virtio_rpmsg_bus, seems
> > to get things setup but that does not result in creation of any "root"
> > rpmsg-over-virtio device. Presumably, any such device would have to be
> > setup to use a specific range of addresses and also be tied to
> > virtio_rpmsg_bus to ensure that virtio is used.
> > 
> > It is also not clear if/how register_rpmsg_driver() will be required
> > on the rpmsg driver side, even though the sample code does not use it.
> > 
> > So, first questions are:
> > 
> > * Am I looking at the correct interfaces in order to create the host
> > rpmsg device side?
> > * What needs to be done to get a "root" rpmsg-over-virtio device
> > created (if required)?
> > * How is a rpmsg-over-virtio device created for each host-guest driver
> > pair, for use with rpmsg_create_ept()?
> > * Does the guest side (rpmsg driver) require any special handling to
> > plug-in to the host driver (rpmsg device) side? Aside from using the
> > correct addresses to match device side.
> 
> It looks to me as though the virtio_rpmsg_bus module can create a
> "rpmsg_ctl" device, which could be used to create channels from which
> endpoints could be created. However, when I load the virtio_rpmsg_bus,
> rpmsg_ns, and rpmsg_core modules there is no "rpmsg_ctl" device created
> (this is running in the host OS, before any VMs are created/run).
>

At this time the modules stated above are all used when a main processor is
controlling a remote processor, i.e via the remoteproc subsystem.  I do not know
of an implementation where VIRTIO_ID_RPMSG is used in the context of a
host/guest scenario.  As such you will find yourself in uncharted territory.  

At some point there were discussion via the OpenAMP body to standardize the
remoteproc's subsystem establishment of virtqueues to conform to a host/guest
scenario but was abandonned.  That would have been a step in the right direction
for what you are trying to do.

> Is this the correct way to use RPMSG-over-VIRTIO? If so, what actions
> need to be taken to cause a "rpmsg_ctl" device to be created? What
> method would be used (in a kernel driver) to get a pointer to the
> "rpmsg_ctl" device, for use with rpmsg_create_channel()?
> 
> > 
> > Thanks,
> > Doug
> > 
> 
> External recipient

Re: How to create/use RPMSG-over-VIRTIO devices in Linux

[Doug Miller]

On 9/10/2024 10:13 AM, Mathieu Poirier wrote:
> On Tue, Sep 10, 2024 at 08:12:07AM -0500, Doug Miller wrote:
>> On 9/3/2024 10:52 AM, Doug Miller wrote:
>>> I am trying to learn how to create an RPMSG-over-VIRTIO device
>>> (service) in order to perform communication between a host driver and
>>> a guest driver. The RPMSG-over-VIRTIO driver (client) side is fairly
>>> well documented and there is a good example (starting point, at least)
>>> in samples/rpmsg/rpmsg_client_sample.c.
>>>
>>> I see that I can create an endpoint (struct rpmsg_endpoint) using
>>> rpmsg_create_ept(), and from there I can use rpmsg_send() et al. and
>>> the rpmsg_rx_cb_t cb to perform the communications. However, this
>>> requires a struct rpmsg_device and it is not clear just how to get one
>>> that is suitable for this purpose.
>>>
>>> It appears that one or both of rpmsg_create_channel() and
>>> rpmsg_register_device() are needed in order to obtain a device for the
>>> specific host-guest communications channel. At some point, a "root"
>>> device is needed that will use virtio (VIRTIO_ID_RPMSG) such that new
>>> subdevices can be created for each host-guest pair.
>>>
>>> In addition, building a kernel with CONFIG_RPMSG, CONFIG_RPMSG_VIRTIO,
>>> and CONFIG_RPMSG_NS set, and doing a modprobe virtio_rpmsg_bus, seems
>>> to get things setup but that does not result in creation of any "root"
>>> rpmsg-over-virtio device. Presumably, any such device would have to be
>>> setup to use a specific range of addresses and also be tied to
>>> virtio_rpmsg_bus to ensure that virtio is used.
>>>
>>> It is also not clear if/how register_rpmsg_driver() will be required
>>> on the rpmsg driver side, even though the sample code does not use it.
>>>
>>> So, first questions are:
>>>
>>> * Am I looking at the correct interfaces in order to create the host
>>> rpmsg device side?
>>> * What needs to be done to get a "root" rpmsg-over-virtio device
>>> created (if required)?
>>> * How is a rpmsg-over-virtio device created for each host-guest driver
>>> pair, for use with rpmsg_create_ept()?
>>> * Does the guest side (rpmsg driver) require any special handling to
>>> plug-in to the host driver (rpmsg device) side? Aside from using the
>>> correct addresses to match device side.
>> It looks to me as though the virtio_rpmsg_bus module can create a
>> "rpmsg_ctl" device, which could be used to create channels from which
>> endpoints could be created. However, when I load the virtio_rpmsg_bus,
>> rpmsg_ns, and rpmsg_core modules there is no "rpmsg_ctl" device created
>> (this is running in the host OS, before any VMs are created/run).
>>
> At this time the modules stated above are all used when a main processor is
> controlling a remote processor, i.e via the remoteproc subsystem.  I do not know
> of an implementation where VIRTIO_ID_RPMSG is used in the context of a
> host/guest scenario.  As such you will find yourself in uncharted territory.
>
> At some point there were discussion via the OpenAMP body to standardize the
> remoteproc's subsystem establishment of virtqueues to conform to a host/guest
> scenario but was abandonned.  That would have been a step in the right direction
> for what you are trying to do.
I was looking at some existing rpmsg code, at it appeared to me that
some adapters, like the "qcom", are creating an rpmsg device that
provides specialized methods for talking to the remote processor(s). I
have assumed this is because that hardware does not allow for running
something remotely that can utilize the virtio queues directly, and so
these rpmsg devices provide code to do the communication with their
hardware. What's not clear is whether these devices are using
rpmsg-over-virtio or if they are creating their own rpmsg facility (and
whether they even support guest-host communication).

What I'm also wondering is what needs to be done differently for virtio
when communicating guest-host vs local CPU to remote processor. I was
hoping that RPMSG-over-VIRTIO would be easily adapted to what we need.
If we have to create a new virtio device (that is nearly identical to
rpmsg), that is going to push-out SR-IOV support a great deal, plus
requiring cloning of a lot of existing code for a new purpose.

Our only other alternative is to do something to allow guest-host
communication to use the fabric loopback, which is not at all desirable
and has many issues of its own.

>
>> Is this the correct way to use RPMSG-over-VIRTIO? If so, what actions
>> need to be taken to cause a "rpmsg_ctl" device to be created? What
>> method would be used (in a kernel driver) to get a pointer to the
>> "rpmsg_ctl" device, for use with rpmsg_create_channel()?
>>
>>> Thanks,
>>> Doug
>>>
>> External recipient


External recipient

Re: How to create/use RPMSG-over-VIRTIO devices in Linux

[Mathieu Poirier]

On Tue, 10 Sept 2024 at 09:43, Doug Miller
<doug.miller@cornelisnetworks.com> wrote:
>
> On 9/10/2024 10:13 AM, Mathieu Poirier wrote:
> > On Tue, Sep 10, 2024 at 08:12:07AM -0500, Doug Miller wrote:
> >> On 9/3/2024 10:52 AM, Doug Miller wrote:
> >>> I am trying to learn how to create an RPMSG-over-VIRTIO device
> >>> (service) in order to perform communication between a host driver and
> >>> a guest driver. The RPMSG-over-VIRTIO driver (client) side is fairly
> >>> well documented and there is a good example (starting point, at least)
> >>> in samples/rpmsg/rpmsg_client_sample.c.
> >>>
> >>> I see that I can create an endpoint (struct rpmsg_endpoint) using
> >>> rpmsg_create_ept(), and from there I can use rpmsg_send() et al. and
> >>> the rpmsg_rx_cb_t cb to perform the communications. However, this
> >>> requires a struct rpmsg_device and it is not clear just how to get one
> >>> that is suitable for this purpose.
> >>>
> >>> It appears that one or both of rpmsg_create_channel() and
> >>> rpmsg_register_device() are needed in order to obtain a device for the
> >>> specific host-guest communications channel. At some point, a "root"
> >>> device is needed that will use virtio (VIRTIO_ID_RPMSG) such that new
> >>> subdevices can be created for each host-guest pair.
> >>>
> >>> In addition, building a kernel with CONFIG_RPMSG, CONFIG_RPMSG_VIRTIO,
> >>> and CONFIG_RPMSG_NS set, and doing a modprobe virtio_rpmsg_bus, seems
> >>> to get things setup but that does not result in creation of any "root"
> >>> rpmsg-over-virtio device. Presumably, any such device would have to be
> >>> setup to use a specific range of addresses and also be tied to
> >>> virtio_rpmsg_bus to ensure that virtio is used.
> >>>
> >>> It is also not clear if/how register_rpmsg_driver() will be required
> >>> on the rpmsg driver side, even though the sample code does not use it.
> >>>
> >>> So, first questions are:
> >>>
> >>> * Am I looking at the correct interfaces in order to create the host
> >>> rpmsg device side?
> >>> * What needs to be done to get a "root" rpmsg-over-virtio device
> >>> created (if required)?
> >>> * How is a rpmsg-over-virtio device created for each host-guest driver
> >>> pair, for use with rpmsg_create_ept()?
> >>> * Does the guest side (rpmsg driver) require any special handling to
> >>> plug-in to the host driver (rpmsg device) side? Aside from using the
> >>> correct addresses to match device side.
> >> It looks to me as though the virtio_rpmsg_bus module can create a
> >> "rpmsg_ctl" device, which could be used to create channels from which
> >> endpoints could be created. However, when I load the virtio_rpmsg_bus,
> >> rpmsg_ns, and rpmsg_core modules there is no "rpmsg_ctl" device created
> >> (this is running in the host OS, before any VMs are created/run).
> >>
> > At this time the modules stated above are all used when a main processor is
> > controlling a remote processor, i.e via the remoteproc subsystem.  I do not know
> > of an implementation where VIRTIO_ID_RPMSG is used in the context of a
> > host/guest scenario.  As such you will find yourself in uncharted territory.
> >
> > At some point there were discussion via the OpenAMP body to standardize the
> > remoteproc's subsystem establishment of virtqueues to conform to a host/guest
> > scenario but was abandonned.  That would have been a step in the right direction
> > for what you are trying to do.
> I was looking at some existing rpmsg code, at it appeared to me that
> some adapters, like the "qcom", are creating an rpmsg device that
> provides specialized methods for talking to the remote processor(s). I
> have assumed this is because that hardware does not allow for running
> something remotely that can utilize the virtio queues directly, and so
> these rpmsg devices provide code to do the communication with their
> hardware. What's not clear is whether these devices are using
> rpmsg-over-virtio or if they are creating their own rpmsg facility (and
> whether they even support guest-host communication).
>

The QC implementation is different and does not use virtio - there is
a special HW interface between the main and the remote processors.
That configuration is valid since RPMSG can be implemented over
anything.

> What I'm also wondering is what needs to be done differently for virtio
> when communicating guest-host vs local CPU to remote processor. I was

From a kernel/guest perspective, not much should be needed.  That said
the VMM will need to be supplemented with extra configuration
capabilities to instantiate the virtio-rpmsg device.  But that is just
off the top of my head without seriously looking at the use case.
From a virtio-bus perspective, there might be an issue if a platform
is using remote processors _and_ also instantiating VMs that
configures a virtio-rpmsg device.  Again, that is just off the top of
my head but needs to be taken into account.

> hoping that RPMSG-over-VIRTIO would be easily adapted to what we need.
> If we have to create a new virtio device (that is nearly identical to
> rpmsg), that is going to push-out SR-IOV support a great deal, plus
> requiring cloning of a lot of existing code for a new purpose.

Duplication of code would not be a viable way forward.
Reusing/enhancing/fixing what is currently available is definitely a
better option.

>
> Our only other alternative is to do something to allow guest-host
> communication to use the fabric loopback, which is not at all desirable
> and has many issues of its own.
>
> >
> >> Is this the correct way to use RPMSG-over-VIRTIO? If so, what actions
> >> need to be taken to cause a "rpmsg_ctl" device to be created? What
> >> method would be used (in a kernel driver) to get a pointer to the
> >> "rpmsg_ctl" device, for use with rpmsg_create_channel()?
> >>
> >>> Thanks,
> >>> Doug
> >>>
> >> External recipient
>
>
> External recipient

Re: How to create/use RPMSG-over-VIRTIO devices in Linux

[Doug Miller]

On 9/11/2024 11:12 AM, Mathieu Poirier wrote:
> On Tue, 10 Sept 2024 at 09:43, Doug Miller
> <doug.miller@cornelisnetworks.com> wrote:
>> On 9/10/2024 10:13 AM, Mathieu Poirier wrote:
>>> On Tue, Sep 10, 2024 at 08:12:07AM -0500, Doug Miller wrote:
>>>> On 9/3/2024 10:52 AM, Doug Miller wrote:
>>>>> I am trying to learn how to create an RPMSG-over-VIRTIO device
>>>>> (service) in order to perform communication between a host driver and
>>>>> a guest driver. The RPMSG-over-VIRTIO driver (client) side is fairly
>>>>> well documented and there is a good example (starting point, at least)
>>>>> in samples/rpmsg/rpmsg_client_sample.c.
>>>>>
>>>>> I see that I can create an endpoint (struct rpmsg_endpoint) using
>>>>> rpmsg_create_ept(), and from there I can use rpmsg_send() et al. and
>>>>> the rpmsg_rx_cb_t cb to perform the communications. However, this
>>>>> requires a struct rpmsg_device and it is not clear just how to get one
>>>>> that is suitable for this purpose.
>>>>>
>>>>> It appears that one or both of rpmsg_create_channel() and
>>>>> rpmsg_register_device() are needed in order to obtain a device for the
>>>>> specific host-guest communications channel. At some point, a "root"
>>>>> device is needed that will use virtio (VIRTIO_ID_RPMSG) such that new
>>>>> subdevices can be created for each host-guest pair.
>>>>>
>>>>> In addition, building a kernel with CONFIG_RPMSG, CONFIG_RPMSG_VIRTIO,
>>>>> and CONFIG_RPMSG_NS set, and doing a modprobe virtio_rpmsg_bus, seems
>>>>> to get things setup but that does not result in creation of any "root"
>>>>> rpmsg-over-virtio device. Presumably, any such device would have to be
>>>>> setup to use a specific range of addresses and also be tied to
>>>>> virtio_rpmsg_bus to ensure that virtio is used.
>>>>>
>>>>> It is also not clear if/how register_rpmsg_driver() will be required
>>>>> on the rpmsg driver side, even though the sample code does not use it.
>>>>>
>>>>> So, first questions are:
>>>>>
>>>>> * Am I looking at the correct interfaces in order to create the host
>>>>> rpmsg device side?
>>>>> * What needs to be done to get a "root" rpmsg-over-virtio device
>>>>> created (if required)?
>>>>> * How is a rpmsg-over-virtio device created for each host-guest driver
>>>>> pair, for use with rpmsg_create_ept()?
>>>>> * Does the guest side (rpmsg driver) require any special handling to
>>>>> plug-in to the host driver (rpmsg device) side? Aside from using the
>>>>> correct addresses to match device side.
>>>> It looks to me as though the virtio_rpmsg_bus module can create a
>>>> "rpmsg_ctl" device, which could be used to create channels from which
>>>> endpoints could be created. However, when I load the virtio_rpmsg_bus,
>>>> rpmsg_ns, and rpmsg_core modules there is no "rpmsg_ctl" device created
>>>> (this is running in the host OS, before any VMs are created/run).
>>>>
>>> At this time the modules stated above are all used when a main processor is
>>> controlling a remote processor, i.e via the remoteproc subsystem.  I do not know
>>> of an implementation where VIRTIO_ID_RPMSG is used in the context of a
>>> host/guest scenario.  As such you will find yourself in uncharted territory.
>>>
>>> At some point there were discussion via the OpenAMP body to standardize the
>>> remoteproc's subsystem establishment of virtqueues to conform to a host/guest
>>> scenario but was abandonned.  That would have been a step in the right direction
>>> for what you are trying to do.
>> I was looking at some existing rpmsg code, at it appeared to me that
>> some adapters, like the "qcom", are creating an rpmsg device that
>> provides specialized methods for talking to the remote processor(s). I
>> have assumed this is because that hardware does not allow for running
>> something remotely that can utilize the virtio queues directly, and so
>> these rpmsg devices provide code to do the communication with their
>> hardware. What's not clear is whether these devices are using
>> rpmsg-over-virtio or if they are creating their own rpmsg facility (and
>> whether they even support guest-host communication).
>>
> The QC implementation is different and does not use virtio - there is
> a special HW interface between the main and the remote processors.
> That configuration is valid since RPMSG can be implemented over
> anything.
>
>> What I'm also wondering is what needs to be done differently for virtio
>> when communicating guest-host vs local CPU to remote processor. I was
>  From a kernel/guest perspective, not much should be needed.  That said
> the VMM will need to be supplemented with extra configuration
> capabilities to instantiate the virtio-rpmsg device.  But that is just
> off the top of my head without seriously looking at the use case.
>  From a virtio-bus perspective, there might be an issue if a platform
> is using remote processors _and_ also instantiating VMs that
> configures a virtio-rpmsg device.  Again, that is just off the top of
> my head but needs to be taken into account.
I am new to rpmsg and virtio, and so my understanding of internals is
still very limited. Is there someone I can work with to determine what
needs to be done here? I am guessing that virtio either automatically
adapts to guest-host or rproc-host - in which case no changes may be
required - or else it requires a different setup and rpmsg will need to
be extended to allow for that. If there are changes to rpmsg required,
we'll want to get those submitted as soon as possible. One complication
for submitting our driver changes is that it is part of a much larger
effort to support new hardware, and it may not be possible to submit
them together with rpmsg changes.
>
>> hoping that RPMSG-over-VIRTIO would be easily adapted to what we need.
>> If we have to create a new virtio device (that is nearly identical to
>> rpmsg), that is going to push-out SR-IOV support a great deal, plus
>> requiring cloning of a lot of existing code for a new purpose.
> Duplication of code would not be a viable way forward.
> Reusing/enhancing/fixing what is currently available is definitely a
> better option.
>
>> Our only other alternative is to do something to allow guest-host
>> communication to use the fabric loopback, which is not at all desirable
>> and has many issues of its own.
>>
>>>> Is this the correct way to use RPMSG-over-VIRTIO? If so, what actions
>>>> need to be taken to cause a "rpmsg_ctl" device to be created? What
>>>> method would be used (in a kernel driver) to get a pointer to the
>>>> "rpmsg_ctl" device, for use with rpmsg_create_channel()?
>>>>
>>>>> Thanks,
>>>>> Doug
>>>>>
>>>> External recipient
>>
>> External recipient


External recipient

Re: How to create/use RPMSG-over-VIRTIO devices in Linux

[Mathieu Poirier]

On Wed, Sep 11, 2024 at 12:24:07PM -0500, Doug Miller wrote:
> On 9/11/2024 11:12 AM, Mathieu Poirier wrote:
> > On Tue, 10 Sept 2024 at 09:43, Doug Miller
> > <doug.miller@cornelisnetworks.com> wrote:
> > > On 9/10/2024 10:13 AM, Mathieu Poirier wrote:
> > > > On Tue, Sep 10, 2024 at 08:12:07AM -0500, Doug Miller wrote:
> > > > > On 9/3/2024 10:52 AM, Doug Miller wrote:
> > > > > > I am trying to learn how to create an RPMSG-over-VIRTIO device
> > > > > > (service) in order to perform communication between a host driver and
> > > > > > a guest driver. The RPMSG-over-VIRTIO driver (client) side is fairly
> > > > > > well documented and there is a good example (starting point, at least)
> > > > > > in samples/rpmsg/rpmsg_client_sample.c.
> > > > > > 
> > > > > > I see that I can create an endpoint (struct rpmsg_endpoint) using
> > > > > > rpmsg_create_ept(), and from there I can use rpmsg_send() et al. and
> > > > > > the rpmsg_rx_cb_t cb to perform the communications. However, this
> > > > > > requires a struct rpmsg_device and it is not clear just how to get one
> > > > > > that is suitable for this purpose.
> > > > > > 
> > > > > > It appears that one or both of rpmsg_create_channel() and
> > > > > > rpmsg_register_device() are needed in order to obtain a device for the
> > > > > > specific host-guest communications channel. At some point, a "root"
> > > > > > device is needed that will use virtio (VIRTIO_ID_RPMSG) such that new
> > > > > > subdevices can be created for each host-guest pair.
> > > > > > 
> > > > > > In addition, building a kernel with CONFIG_RPMSG, CONFIG_RPMSG_VIRTIO,
> > > > > > and CONFIG_RPMSG_NS set, and doing a modprobe virtio_rpmsg_bus, seems
> > > > > > to get things setup but that does not result in creation of any "root"
> > > > > > rpmsg-over-virtio device. Presumably, any such device would have to be
> > > > > > setup to use a specific range of addresses and also be tied to
> > > > > > virtio_rpmsg_bus to ensure that virtio is used.
> > > > > > 
> > > > > > It is also not clear if/how register_rpmsg_driver() will be required
> > > > > > on the rpmsg driver side, even though the sample code does not use it.
> > > > > > 
> > > > > > So, first questions are:
> > > > > > 
> > > > > > * Am I looking at the correct interfaces in order to create the host
> > > > > > rpmsg device side?
> > > > > > * What needs to be done to get a "root" rpmsg-over-virtio device
> > > > > > created (if required)?
> > > > > > * How is a rpmsg-over-virtio device created for each host-guest driver
> > > > > > pair, for use with rpmsg_create_ept()?
> > > > > > * Does the guest side (rpmsg driver) require any special handling to
> > > > > > plug-in to the host driver (rpmsg device) side? Aside from using the
> > > > > > correct addresses to match device side.
> > > > > It looks to me as though the virtio_rpmsg_bus module can create a
> > > > > "rpmsg_ctl" device, which could be used to create channels from which
> > > > > endpoints could be created. However, when I load the virtio_rpmsg_bus,
> > > > > rpmsg_ns, and rpmsg_core modules there is no "rpmsg_ctl" device created
> > > > > (this is running in the host OS, before any VMs are created/run).
> > > > > 
> > > > At this time the modules stated above are all used when a main processor is
> > > > controlling a remote processor, i.e via the remoteproc subsystem.  I do not know
> > > > of an implementation where VIRTIO_ID_RPMSG is used in the context of a
> > > > host/guest scenario.  As such you will find yourself in uncharted territory.
> > > > 
> > > > At some point there were discussion via the OpenAMP body to standardize the
> > > > remoteproc's subsystem establishment of virtqueues to conform to a host/guest
> > > > scenario but was abandonned.  That would have been a step in the right direction
> > > > for what you are trying to do.
> > > I was looking at some existing rpmsg code, at it appeared to me that
> > > some adapters, like the "qcom", are creating an rpmsg device that
> > > provides specialized methods for talking to the remote processor(s). I
> > > have assumed this is because that hardware does not allow for running
> > > something remotely that can utilize the virtio queues directly, and so
> > > these rpmsg devices provide code to do the communication with their
> > > hardware. What's not clear is whether these devices are using
> > > rpmsg-over-virtio or if they are creating their own rpmsg facility (and
> > > whether they even support guest-host communication).
> > > 
> > The QC implementation is different and does not use virtio - there is
> > a special HW interface between the main and the remote processors.
> > That configuration is valid since RPMSG can be implemented over
> > anything.
> > 
> > > What I'm also wondering is what needs to be done differently for virtio
> > > when communicating guest-host vs local CPU to remote processor. I was
> >  From a kernel/guest perspective, not much should be needed.  That said
> > the VMM will need to be supplemented with extra configuration
> > capabilities to instantiate the virtio-rpmsg device.  But that is just
> > off the top of my head without seriously looking at the use case.
> >  From a virtio-bus perspective, there might be an issue if a platform
> > is using remote processors _and_ also instantiating VMs that
> > configures a virtio-rpmsg device.  Again, that is just off the top of
> > my head but needs to be taken into account.
> I am new to rpmsg and virtio, and so my understanding of internals is
> still very limited. Is there someone I can work with to determine what
> needs to be done here? I am guessing that virtio either automatically
> adapts to guest-host or rproc-host - in which case no changes may be
> required - or else it requires a different setup and rpmsg will need to
> be extended to allow for that. If there are changes to rpmsg required,
> we'll want to get those submitted as soon as possible. One complication
> for submitting our driver changes is that it is part of a much larger
> effort to support new hardware, and it may not be possible to submit
> them together with rpmsg changes.

The virtio part won't be a problem.  In your case what is missing is the glue
that will setup the virtqueues and install the RPMSG protocol on top of them.
The 'glue' is the new virtio-rpmsg device that needs to be created.  That part
includes the creation of a new virtio device by the VMM and a kernel driver that
can be called from the virtio_bus once it has been discovered.  

Everything in the virtio and RPMSG subsystems are aleady tailored to support all
this, so no changes should be needed.  As for the VMM, I suggest to start with
kvmtool.  Lastly, none of this requires "real" hardware or your specific
hardware - it can all be done from QEMU.

> > 
> > > hoping that RPMSG-over-VIRTIO would be easily adapted to what we need.
> > > If we have to create a new virtio device (that is nearly identical to
> > > rpmsg), that is going to push-out SR-IOV support a great deal, plus
> > > requiring cloning of a lot of existing code for a new purpose.
> > Duplication of code would not be a viable way forward.
> > Reusing/enhancing/fixing what is currently available is definitely a
> > better option.
> > 
> > > Our only other alternative is to do something to allow guest-host
> > > communication to use the fabric loopback, which is not at all desirable
> > > and has many issues of its own.
> > > 
> > > > > Is this the correct way to use RPMSG-over-VIRTIO? If so, what actions
> > > > > need to be taken to cause a "rpmsg_ctl" device to be created? What
> > > > > method would be used (in a kernel driver) to get a pointer to the
> > > > > "rpmsg_ctl" device, for use with rpmsg_create_channel()?
> > > > > 
> > > > > > Thanks,
> > > > > > Doug
> > > > > > 
> > > > > External recipient
> > > 
> > > External recipient
> 
> 
> External recipient

Re: How to create/use RPMSG-over-VIRTIO devices in Linux

[Doug Miller]

On 9/12/2024 10:10 AM, Mathieu Poirier wrote:
> On Wed, Sep 11, 2024 at 12:24:07PM -0500, Doug Miller wrote:
>> On 9/11/2024 11:12 AM, Mathieu Poirier wrote:
>>> On Tue, 10 Sept 2024 at 09:43, Doug Miller
>>> <doug.miller@cornelisnetworks.com> wrote:
>>>> On 9/10/2024 10:13 AM, Mathieu Poirier wrote:
>>>>> On Tue, Sep 10, 2024 at 08:12:07AM -0500, Doug Miller wrote:
>>>>>> On 9/3/2024 10:52 AM, Doug Miller wrote:
>>>>>>> I am trying to learn how to create an RPMSG-over-VIRTIO device
>>>>>>> (service) in order to perform communication between a host driver and
>>>>>>> a guest driver. The RPMSG-over-VIRTIO driver (client) side is fairly
>>>>>>> well documented and there is a good example (starting point, at least)
>>>>>>> in samples/rpmsg/rpmsg_client_sample.c.
>>>>>>>
>>>>>>> I see that I can create an endpoint (struct rpmsg_endpoint) using
>>>>>>> rpmsg_create_ept(), and from there I can use rpmsg_send() et al. and
>>>>>>> the rpmsg_rx_cb_t cb to perform the communications. However, this
>>>>>>> requires a struct rpmsg_device and it is not clear just how to get one
>>>>>>> that is suitable for this purpose.
>>>>>>>
>>>>>>> It appears that one or both of rpmsg_create_channel() and
>>>>>>> rpmsg_register_device() are needed in order to obtain a device for the
>>>>>>> specific host-guest communications channel. At some point, a "root"
>>>>>>> device is needed that will use virtio (VIRTIO_ID_RPMSG) such that new
>>>>>>> subdevices can be created for each host-guest pair.
>>>>>>>
>>>>>>> In addition, building a kernel with CONFIG_RPMSG, CONFIG_RPMSG_VIRTIO,
>>>>>>> and CONFIG_RPMSG_NS set, and doing a modprobe virtio_rpmsg_bus, seems
>>>>>>> to get things setup but that does not result in creation of any "root"
>>>>>>> rpmsg-over-virtio device. Presumably, any such device would have to be
>>>>>>> setup to use a specific range of addresses and also be tied to
>>>>>>> virtio_rpmsg_bus to ensure that virtio is used.
>>>>>>>
>>>>>>> It is also not clear if/how register_rpmsg_driver() will be required
>>>>>>> on the rpmsg driver side, even though the sample code does not use it.
>>>>>>>
>>>>>>> So, first questions are:
>>>>>>>
>>>>>>> * Am I looking at the correct interfaces in order to create the host
>>>>>>> rpmsg device side?
>>>>>>> * What needs to be done to get a "root" rpmsg-over-virtio device
>>>>>>> created (if required)?
>>>>>>> * How is a rpmsg-over-virtio device created for each host-guest driver
>>>>>>> pair, for use with rpmsg_create_ept()?
>>>>>>> * Does the guest side (rpmsg driver) require any special handling to
>>>>>>> plug-in to the host driver (rpmsg device) side? Aside from using the
>>>>>>> correct addresses to match device side.
>>>>>> It looks to me as though the virtio_rpmsg_bus module can create a
>>>>>> "rpmsg_ctl" device, which could be used to create channels from which
>>>>>> endpoints could be created. However, when I load the virtio_rpmsg_bus,
>>>>>> rpmsg_ns, and rpmsg_core modules there is no "rpmsg_ctl" device created
>>>>>> (this is running in the host OS, before any VMs are created/run).
>>>>>>
>>>>> At this time the modules stated above are all used when a main processor is
>>>>> controlling a remote processor, i.e via the remoteproc subsystem.  I do not know
>>>>> of an implementation where VIRTIO_ID_RPMSG is used in the context of a
>>>>> host/guest scenario.  As such you will find yourself in uncharted territory.
>>>>>
>>>>> At some point there were discussion via the OpenAMP body to standardize the
>>>>> remoteproc's subsystem establishment of virtqueues to conform to a host/guest
>>>>> scenario but was abandonned.  That would have been a step in the right direction
>>>>> for what you are trying to do.
>>>> I was looking at some existing rpmsg code, at it appeared to me that
>>>> some adapters, like the "qcom", are creating an rpmsg device that
>>>> provides specialized methods for talking to the remote processor(s). I
>>>> have assumed this is because that hardware does not allow for running
>>>> something remotely that can utilize the virtio queues directly, and so
>>>> these rpmsg devices provide code to do the communication with their
>>>> hardware. What's not clear is whether these devices are using
>>>> rpmsg-over-virtio or if they are creating their own rpmsg facility (and
>>>> whether they even support guest-host communication).
>>>>
>>> The QC implementation is different and does not use virtio - there is
>>> a special HW interface between the main and the remote processors.
>>> That configuration is valid since RPMSG can be implemented over
>>> anything.
>>>
>>>> What I'm also wondering is what needs to be done differently for virtio
>>>> when communicating guest-host vs local CPU to remote processor. I was
>>>   From a kernel/guest perspective, not much should be needed.  That said
>>> the VMM will need to be supplemented with extra configuration
>>> capabilities to instantiate the virtio-rpmsg device.  But that is just
>>> off the top of my head without seriously looking at the use case.
>>>   From a virtio-bus perspective, there might be an issue if a platform
>>> is using remote processors _and_ also instantiating VMs that
>>> configures a virtio-rpmsg device.  Again, that is just off the top of
>>> my head but needs to be taken into account.
>> I am new to rpmsg and virtio, and so my understanding of internals is
>> still very limited. Is there someone I can work with to determine what
>> needs to be done here? I am guessing that virtio either automatically
>> adapts to guest-host or rproc-host - in which case no changes may be
>> required - or else it requires a different setup and rpmsg will need to
>> be extended to allow for that. If there are changes to rpmsg required,
>> we'll want to get those submitted as soon as possible. One complication
>> for submitting our driver changes is that it is part of a much larger
>> effort to support new hardware, and it may not be possible to submit
>> them together with rpmsg changes.
> The virtio part won't be a problem.  In your case what is missing is the glue
> that will setup the virtqueues and install the RPMSG protocol on top of them.
> The 'glue' is the new virtio-rpmsg device that needs to be created.  That part
> includes the creation of a new virtio device by the VMM and a kernel driver that
> can be called from the virtio_bus once it has been discovered.
I don't completely follow. Is there some KVM configuration option that
causes the virtio-rpmsg device to be created? And then our host driver
will need to be able to respond to some notification and dynamically
adapt to each VMM being started? I'm not getting a clear picture of how
this works. I'm also not clear on the responsibilities of our guest
driver(s) vs. our host driver. For virtio I saw there was the concept of
a "driver" side and a "device" side, and the guest seemed to be creating
the driver and the host created the device. The rpmsg layer seems to be
more complex in that area, so I'm not sure what actions our guest driver
with take vs. our host driver.
>
> Everything in the virtio and RPMSG subsystems are aleady tailored to support all
> this, so no changes should be needed.  As for the VMM, I suggest to start with
> kvmtool.  Lastly, none of this requires "real" hardware or your specific
> hardware - it can all be done from QEMU.
>
>>>> hoping that RPMSG-over-VIRTIO would be easily adapted to what we need.
>>>> If we have to create a new virtio device (that is nearly identical to
>>>> rpmsg), that is going to push-out SR-IOV support a great deal, plus
>>>> requiring cloning of a lot of existing code for a new purpose.
>>> Duplication of code would not be a viable way forward.
>>> Reusing/enhancing/fixing what is currently available is definitely a
>>> better option.
>>>
>>>> Our only other alternative is to do something to allow guest-host
>>>> communication to use the fabric loopback, which is not at all desirable
>>>> and has many issues of its own.
>>>>
>>>>>> Is this the correct way to use RPMSG-over-VIRTIO? If so, what actions
>>>>>> need to be taken to cause a "rpmsg_ctl" device to be created? What
>>>>>> method would be used (in a kernel driver) to get a pointer to the
>>>>>> "rpmsg_ctl" device, for use with rpmsg_create_channel()?
>>>>>>
>>>>>>> Thanks,
>>>>>>> Doug
>>>>>>>
>>>>>> External recipient
>>>> External recipient
>>
>> External recipient


External recipient

Re: How to create/use RPMSG-over-VIRTIO devices in Linux

[Mathieu Poirier]

On Fri, 13 Sept 2024 at 05:46, Doug Miller
<doug.miller@cornelisnetworks.com> wrote:
>
> On 9/12/2024 10:10 AM, Mathieu Poirier wrote:
> > On Wed, Sep 11, 2024 at 12:24:07PM -0500, Doug Miller wrote:
> >> On 9/11/2024 11:12 AM, Mathieu Poirier wrote:
> >>> On Tue, 10 Sept 2024 at 09:43, Doug Miller
> >>> <doug.miller@cornelisnetworks.com> wrote:
> >>>> On 9/10/2024 10:13 AM, Mathieu Poirier wrote:
> >>>>> On Tue, Sep 10, 2024 at 08:12:07AM -0500, Doug Miller wrote:
> >>>>>> On 9/3/2024 10:52 AM, Doug Miller wrote:
> >>>>>>> I am trying to learn how to create an RPMSG-over-VIRTIO device
> >>>>>>> (service) in order to perform communication between a host driver and
> >>>>>>> a guest driver. The RPMSG-over-VIRTIO driver (client) side is fairly
> >>>>>>> well documented and there is a good example (starting point, at least)
> >>>>>>> in samples/rpmsg/rpmsg_client_sample.c.
> >>>>>>>
> >>>>>>> I see that I can create an endpoint (struct rpmsg_endpoint) using
> >>>>>>> rpmsg_create_ept(), and from there I can use rpmsg_send() et al. and
> >>>>>>> the rpmsg_rx_cb_t cb to perform the communications. However, this
> >>>>>>> requires a struct rpmsg_device and it is not clear just how to get one
> >>>>>>> that is suitable for this purpose.
> >>>>>>>
> >>>>>>> It appears that one or both of rpmsg_create_channel() and
> >>>>>>> rpmsg_register_device() are needed in order to obtain a device for the
> >>>>>>> specific host-guest communications channel. At some point, a "root"
> >>>>>>> device is needed that will use virtio (VIRTIO_ID_RPMSG) such that new
> >>>>>>> subdevices can be created for each host-guest pair.
> >>>>>>>
> >>>>>>> In addition, building a kernel with CONFIG_RPMSG, CONFIG_RPMSG_VIRTIO,
> >>>>>>> and CONFIG_RPMSG_NS set, and doing a modprobe virtio_rpmsg_bus, seems
> >>>>>>> to get things setup but that does not result in creation of any "root"
> >>>>>>> rpmsg-over-virtio device. Presumably, any such device would have to be
> >>>>>>> setup to use a specific range of addresses and also be tied to
> >>>>>>> virtio_rpmsg_bus to ensure that virtio is used.
> >>>>>>>
> >>>>>>> It is also not clear if/how register_rpmsg_driver() will be required
> >>>>>>> on the rpmsg driver side, even though the sample code does not use it.
> >>>>>>>
> >>>>>>> So, first questions are:
> >>>>>>>
> >>>>>>> * Am I looking at the correct interfaces in order to create the host
> >>>>>>> rpmsg device side?
> >>>>>>> * What needs to be done to get a "root" rpmsg-over-virtio device
> >>>>>>> created (if required)?
> >>>>>>> * How is a rpmsg-over-virtio device created for each host-guest driver
> >>>>>>> pair, for use with rpmsg_create_ept()?
> >>>>>>> * Does the guest side (rpmsg driver) require any special handling to
> >>>>>>> plug-in to the host driver (rpmsg device) side? Aside from using the
> >>>>>>> correct addresses to match device side.
> >>>>>> It looks to me as though the virtio_rpmsg_bus module can create a
> >>>>>> "rpmsg_ctl" device, which could be used to create channels from which
> >>>>>> endpoints could be created. However, when I load the virtio_rpmsg_bus,
> >>>>>> rpmsg_ns, and rpmsg_core modules there is no "rpmsg_ctl" device created
> >>>>>> (this is running in the host OS, before any VMs are created/run).
> >>>>>>
> >>>>> At this time the modules stated above are all used when a main processor is
> >>>>> controlling a remote processor, i.e via the remoteproc subsystem.  I do not know
> >>>>> of an implementation where VIRTIO_ID_RPMSG is used in the context of a
> >>>>> host/guest scenario.  As such you will find yourself in uncharted territory.
> >>>>>
> >>>>> At some point there were discussion via the OpenAMP body to standardize the
> >>>>> remoteproc's subsystem establishment of virtqueues to conform to a host/guest
> >>>>> scenario but was abandonned.  That would have been a step in the right direction
> >>>>> for what you are trying to do.
> >>>> I was looking at some existing rpmsg code, at it appeared to me that
> >>>> some adapters, like the "qcom", are creating an rpmsg device that
> >>>> provides specialized methods for talking to the remote processor(s). I
> >>>> have assumed this is because that hardware does not allow for running
> >>>> something remotely that can utilize the virtio queues directly, and so
> >>>> these rpmsg devices provide code to do the communication with their
> >>>> hardware. What's not clear is whether these devices are using
> >>>> rpmsg-over-virtio or if they are creating their own rpmsg facility (and
> >>>> whether they even support guest-host communication).
> >>>>
> >>> The QC implementation is different and does not use virtio - there is
> >>> a special HW interface between the main and the remote processors.
> >>> That configuration is valid since RPMSG can be implemented over
> >>> anything.
> >>>
> >>>> What I'm also wondering is what needs to be done differently for virtio
> >>>> when communicating guest-host vs local CPU to remote processor. I was
> >>>   From a kernel/guest perspective, not much should be needed.  That said
> >>> the VMM will need to be supplemented with extra configuration
> >>> capabilities to instantiate the virtio-rpmsg device.  But that is just
> >>> off the top of my head without seriously looking at the use case.
> >>>   From a virtio-bus perspective, there might be an issue if a platform
> >>> is using remote processors _and_ also instantiating VMs that
> >>> configures a virtio-rpmsg device.  Again, that is just off the top of
> >>> my head but needs to be taken into account.
> >> I am new to rpmsg and virtio, and so my understanding of internals is
> >> still very limited. Is there someone I can work with to determine what
> >> needs to be done here? I am guessing that virtio either automatically
> >> adapts to guest-host or rproc-host - in which case no changes may be
> >> required - or else it requires a different setup and rpmsg will need to
> >> be extended to allow for that. If there are changes to rpmsg required,
> >> we'll want to get those submitted as soon as possible. One complication
> >> for submitting our driver changes is that it is part of a much larger
> >> effort to support new hardware, and it may not be possible to submit
> >> them together with rpmsg changes.
> > The virtio part won't be a problem.  In your case what is missing is the glue
> > that will setup the virtqueues and install the RPMSG protocol on top of them.
> > The 'glue' is the new virtio-rpmsg device that needs to be created.  That part
> > includes the creation of a new virtio device by the VMM and a kernel driver that
> > can be called from the virtio_bus once it has been discovered.
> I don't completely follow. Is there some KVM configuration option that
> causes the virtio-rpmsg device to be created? And then our host driver
> will need to be able to respond to some notification and dynamically
> adapt to each VMM being started? I'm not getting a clear picture of how
> this works. I'm also not clear on the responsibilities of our guest
> driver(s) vs. our host driver. For virtio I saw there was the concept of
> a "driver" side and a "device" side, and the guest seemed to be creating
> the driver and the host created the device. The rpmsg layer seems to be
> more complex in that area, so I'm not sure what actions our guest driver
> with take vs. our host driver.
> >

KVM has nothing to do with this.  The life of a virtio device starts
in the VMM (Virtual Machine Manager) where a backend device is created
and a virtio MMIO entry for that device is added to the device tree
that is fed to the VM kernel.  When the VM kernel boots the virtio
MMIO entry in the DT is parsed as part of the normal device discovery
process and a virtio-device is instantiated, added to the virtio-bus
and a driver is probed.

I suggest you start looking at that process using the kvmtool and a
simple virtio device such as virtio-rng.

> > Everything in the virtio and RPMSG subsystems are aleady tailored to support all
> > this, so no changes should be needed.  As for the VMM, I suggest to start with
> > kvmtool.  Lastly, none of this requires "real" hardware or your specific
> > hardware - it can all be done from QEMU.
> >
> >>>> hoping that RPMSG-over-VIRTIO would be easily adapted to what we need.
> >>>> If we have to create a new virtio device (that is nearly identical to
> >>>> rpmsg), that is going to push-out SR-IOV support a great deal, plus
> >>>> requiring cloning of a lot of existing code for a new purpose.
> >>> Duplication of code would not be a viable way forward.
> >>> Reusing/enhancing/fixing what is currently available is definitely a
> >>> better option.
> >>>
> >>>> Our only other alternative is to do something to allow guest-host
> >>>> communication to use the fabric loopback, which is not at all desirable
> >>>> and has many issues of its own.
> >>>>
> >>>>>> Is this the correct way to use RPMSG-over-VIRTIO? If so, what actions
> >>>>>> need to be taken to cause a "rpmsg_ctl" device to be created? What
> >>>>>> method would be used (in a kernel driver) to get a pointer to the
> >>>>>> "rpmsg_ctl" device, for use with rpmsg_create_channel()?
> >>>>>>
> >>>>>>> Thanks,
> >>>>>>> Doug
> >>>>>>>
> >>>>>> External recipient
> >>>> External recipient
> >>
> >> External recipient
>
>
> External recipient

Re: How to create/use RPMSG-over-VIRTIO devices in Linux

[Jason Gunthorpe]

On Fri, Sep 13, 2024 at 08:39:26AM -0600, Mathieu Poirier wrote:
> KVM has nothing to do with this.  The life of a virtio device starts
> in the VMM (Virtual Machine Manager) where a backend device is created
> and a virtio MMIO entry for that device is added to the device tree
> that is fed to the VM kernel.  When the VM kernel boots the virtio
> MMIO entry in the DT is parsed as part of the normal device discovery
> process and a virtio-device is instantiated, added to the virtio-bus
> and a driver is probed.
> 
> I suggest you start looking at that process using the kvmtool and a
> simple virtio device such as virtio-rng.

I would repeat again, I think trying to create a companion virtio
device to go along with a real vPCI device and then logically
associating both of them with a single driver is going to cause so
much pain you should not do it.

Find a way to send your RPCs through your own vPCI device.

Jason

Re: How to create/use RPMSG-over-VIRTIO devices in Linux

[Doug Miller]

On 9/15/2024 11:58 AM, Jason Gunthorpe wrote:
> On Fri, Sep 13, 2024 at 08:39:26AM -0600, Mathieu Poirier wrote:
>> KVM has nothing to do with this.  The life of a virtio device starts
>> in the VMM (Virtual Machine Manager) where a backend device is created
>> and a virtio MMIO entry for that device is added to the device tree
>> that is fed to the VM kernel.  When the VM kernel boots the virtio
>> MMIO entry in the DT is parsed as part of the normal device discovery
>> process and a virtio-device is instantiated, added to the virtio-bus
>> and a driver is probed.
>>
>> I suggest you start looking at that process using the kvmtool and a
>> simple virtio device such as virtio-rng.
> I would repeat again, I think trying to create a companion virtio
> device to go along with a real vPCI device and then logically
> associating both of them with a single driver is going to cause so
> much pain you should not do it.
>
> Find a way to send your RPCs through your own vPCI device.
>
> Jason

When you say "your own vPCI device", are you referring to the virtual
functions that are created by the adapter? Those are defined by the
hardware specification and we don't have the ability to extend them.
What we're investigating is using RPMSG-over-VIRTIO, not using virtio
devices directly. It appeared to me that rpmsg was intended to provide a
general-purpose "pipe" between two processors that otherwise don't have
a communication path. That sounds ideal for what we need.

- Doug

External recipient

Re: How to create/use RPMSG-over-VIRTIO devices in Linux

[Jason Gunthorpe]

On Mon, Sep 16, 2024 at 08:38:42AM -0500, Doug Miller wrote:
> On 9/15/2024 11:58 AM, Jason Gunthorpe wrote:
> > On Fri, Sep 13, 2024 at 08:39:26AM -0600, Mathieu Poirier wrote:
> > > KVM has nothing to do with this.  The life of a virtio device starts
> > > in the VMM (Virtual Machine Manager) where a backend device is created
> > > and a virtio MMIO entry for that device is added to the device tree
> > > that is fed to the VM kernel.  When the VM kernel boots the virtio
> > > MMIO entry in the DT is parsed as part of the normal device discovery
> > > process and a virtio-device is instantiated, added to the virtio-bus
> > > and a driver is probed.
> > > 
> > > I suggest you start looking at that process using the kvmtool and a
> > > simple virtio device such as virtio-rng.
> > I would repeat again, I think trying to create a companion virtio
> > device to go along with a real vPCI device and then logically
> > associating both of them with a single driver is going to cause so
> > much pain you should not do it.
> > 
> > Find a way to send your RPCs through your own vPCI device.
> 
> When you say "your own vPCI device", are you referring to the virtual
> functions that are created by the adapter? Those are defined by the
> hardware specification and we don't have the ability to extend them.

Yes you do the VMM can extend them. You need some qemu code or a
vfio-cornelis or something like that.

> What we're investigating is using RPMSG-over-VIRTIO, not using virtio
> devices directly. 

I understand, and that will be very painful.

Jason

Re: How to create/use RPMSG-over-VIRTIO devices in Linux

[Doug Miller]

On 9/20/2024 7:45 AM, Jason Gunthorpe wrote:
> On Mon, Sep 16, 2024 at 08:38:42AM -0500, Doug Miller wrote:
>> On 9/15/2024 11:58 AM, Jason Gunthorpe wrote:
>>> On Fri, Sep 13, 2024 at 08:39:26AM -0600, Mathieu Poirier wrote:
>>>> KVM has nothing to do with this.  The life of a virtio device starts
>>>> in the VMM (Virtual Machine Manager) where a backend device is created
>>>> and a virtio MMIO entry for that device is added to the device tree
>>>> that is fed to the VM kernel.  When the VM kernel boots the virtio
>>>> MMIO entry in the DT is parsed as part of the normal device discovery
>>>> process and a virtio-device is instantiated, added to the virtio-bus
>>>> and a driver is probed.
>>>>
>>>> I suggest you start looking at that process using the kvmtool and a
>>>> simple virtio device such as virtio-rng.
>>> I would repeat again, I think trying to create a companion virtio
>>> device to go along with a real vPCI device and then logically
>>> associating both of them with a single driver is going to cause so
>>> much pain you should not do it.
>>>
>>> Find a way to send your RPCs through your own vPCI device.
>> When you say "your own vPCI device", are you referring to the virtual
>> functions that are created by the adapter? Those are defined by the
>> hardware specification and we don't have the ability to extend them.
> Yes you do the VMM can extend them. You need some qemu code or a
> vfio-cornelis or something like that.
We can't require that SR-IOV customers use one specific VMM (qemu). If
you're talking about modifying the kernel virtio_pci facility, that may
be a worthwhile effort long term but I suspect it will take a longer
time to get adopted. Using an existing kernel facility as-is would be
the most practical solution.
>
>> What we're investigating is using RPMSG-over-VIRTIO, not using virtio
>> devices directly.
> I understand, and that will be very painful.
Can you expand on what you mean by "painful"? Are you speaking from
experience with the rpmsg interfaces (can you point to problem areas)?
Or is this based on the fact that, as of yet, no one has come forward to
explain exactly how to do this?
>
> Jason


External recipient

Re: How to create/use RPMSG-over-VIRTIO devices in Linux

[Jason Gunthorpe]

On Fri, Sep 20, 2024 at 08:56:07AM -0500, Doug Miller wrote:
> On 9/20/2024 7:45 AM, Jason Gunthorpe wrote:
> > On Mon, Sep 16, 2024 at 08:38:42AM -0500, Doug Miller wrote:
> > > On 9/15/2024 11:58 AM, Jason Gunthorpe wrote:
> > > > On Fri, Sep 13, 2024 at 08:39:26AM -0600, Mathieu Poirier wrote:
> > > > > KVM has nothing to do with this.  The life of a virtio device starts
> > > > > in the VMM (Virtual Machine Manager) where a backend device is created
> > > > > and a virtio MMIO entry for that device is added to the device tree
> > > > > that is fed to the VM kernel.  When the VM kernel boots the virtio
> > > > > MMIO entry in the DT is parsed as part of the normal device discovery
> > > > > process and a virtio-device is instantiated, added to the virtio-bus
> > > > > and a driver is probed.
> > > > > 
> > > > > I suggest you start looking at that process using the kvmtool and a
> > > > > simple virtio device such as virtio-rng.
> > > > I would repeat again, I think trying to create a companion virtio
> > > > device to go along with a real vPCI device and then logically
> > > > associating both of them with a single driver is going to cause so
> > > > much pain you should not do it.
> > > > 
> > > > Find a way to send your RPCs through your own vPCI device.
> > > When you say "your own vPCI device", are you referring to the virtual
> > > functions that are created by the adapter? Those are defined by the
> > > hardware specification and we don't have the ability to extend them.
> > Yes you do the VMM can extend them. You need some qemu code or a
> > vfio-cornelis or something like that.

> We can't require that SR-IOV customers use one specific VMM (qemu). 

No matter what you do, you will require some modification. The other
end of any imagined virtio will be in qemu too after all.

> you're talking about modifying the kernel virtio_pci facility, that may
> be a worthwhile effort long term but I suspect it will take a longer
> time to get adopted. Using an existing kernel facility as-is would be
> the most practical solution.

There is really nothing.

> > > What we're investigating is using RPMSG-over-VIRTIO, not using virtio
> > > devices directly.
> > I understand, and that will be very painful.
> Can you expand on what you mean by "painful"? Are you speaking from
> experience with the rpmsg interfaces (can you point to problem areas)?
> Or is this based on the fact that, as of yet, no one has come forward to
> explain exactly how to do this?

From understanding how vfio works and knowing that if you try to tie w
virtio and your PCI together it will be a huge mess.

Jason

Re: How to create/use RPMSG-over-VIRTIO devices in Linux

[Doug Miller]

On 9/13/2024 9:39 AM, Mathieu Poirier wrote:
> On Fri, 13 Sept 2024 at 05:46, Doug Miller
> <doug.miller@cornelisnetworks.com> wrote:
>> On 9/12/2024 10:10 AM, Mathieu Poirier wrote:
>>> On Wed, Sep 11, 2024 at 12:24:07PM -0500, Doug Miller wrote:
>>>> On 9/11/2024 11:12 AM, Mathieu Poirier wrote:
>>>>> On Tue, 10 Sept 2024 at 09:43, Doug Miller
>>>>> <doug.miller@cornelisnetworks.com> wrote:
>>>>>> On 9/10/2024 10:13 AM, Mathieu Poirier wrote:
>>>>>>> On Tue, Sep 10, 2024 at 08:12:07AM -0500, Doug Miller wrote:
>>>>>>>> On 9/3/2024 10:52 AM, Doug Miller wrote:
>>>>>>>>> I am trying to learn how to create an RPMSG-over-VIRTIO device
>>>>>>>>> (service) in order to perform communication between a host driver and
>>>>>>>>> a guest driver. The RPMSG-over-VIRTIO driver (client) side is fairly
>>>>>>>>> well documented and there is a good example (starting point, at least)
>>>>>>>>> in samples/rpmsg/rpmsg_client_sample.c.
>>>>>>>>>
>>>>>>>>> I see that I can create an endpoint (struct rpmsg_endpoint) using
>>>>>>>>> rpmsg_create_ept(), and from there I can use rpmsg_send() et al. and
>>>>>>>>> the rpmsg_rx_cb_t cb to perform the communications. However, this
>>>>>>>>> requires a struct rpmsg_device and it is not clear just how to get one
>>>>>>>>> that is suitable for this purpose.
>>>>>>>>>
>>>>>>>>> It appears that one or both of rpmsg_create_channel() and
>>>>>>>>> rpmsg_register_device() are needed in order to obtain a device for the
>>>>>>>>> specific host-guest communications channel. At some point, a "root"
>>>>>>>>> device is needed that will use virtio (VIRTIO_ID_RPMSG) such that new
>>>>>>>>> subdevices can be created for each host-guest pair.
>>>>>>>>>
>>>>>>>>> In addition, building a kernel with CONFIG_RPMSG, CONFIG_RPMSG_VIRTIO,
>>>>>>>>> and CONFIG_RPMSG_NS set, and doing a modprobe virtio_rpmsg_bus, seems
>>>>>>>>> to get things setup but that does not result in creation of any "root"
>>>>>>>>> rpmsg-over-virtio device. Presumably, any such device would have to be
>>>>>>>>> setup to use a specific range of addresses and also be tied to
>>>>>>>>> virtio_rpmsg_bus to ensure that virtio is used.
>>>>>>>>>
>>>>>>>>> It is also not clear if/how register_rpmsg_driver() will be required
>>>>>>>>> on the rpmsg driver side, even though the sample code does not use it.
>>>>>>>>>
>>>>>>>>> So, first questions are:
>>>>>>>>>
>>>>>>>>> * Am I looking at the correct interfaces in order to create the host
>>>>>>>>> rpmsg device side?
>>>>>>>>> * What needs to be done to get a "root" rpmsg-over-virtio device
>>>>>>>>> created (if required)?
>>>>>>>>> * How is a rpmsg-over-virtio device created for each host-guest driver
>>>>>>>>> pair, for use with rpmsg_create_ept()?
>>>>>>>>> * Does the guest side (rpmsg driver) require any special handling to
>>>>>>>>> plug-in to the host driver (rpmsg device) side? Aside from using the
>>>>>>>>> correct addresses to match device side.
>>>>>>>> It looks to me as though the virtio_rpmsg_bus module can create a
>>>>>>>> "rpmsg_ctl" device, which could be used to create channels from which
>>>>>>>> endpoints could be created. However, when I load the virtio_rpmsg_bus,
>>>>>>>> rpmsg_ns, and rpmsg_core modules there is no "rpmsg_ctl" device created
>>>>>>>> (this is running in the host OS, before any VMs are created/run).
>>>>>>>>
>>>>>>> At this time the modules stated above are all used when a main processor is
>>>>>>> controlling a remote processor, i.e via the remoteproc subsystem.  I do not know
>>>>>>> of an implementation where VIRTIO_ID_RPMSG is used in the context of a
>>>>>>> host/guest scenario.  As such you will find yourself in uncharted territory.
>>>>>>>
>>>>>>> At some point there were discussion via the OpenAMP body to standardize the
>>>>>>> remoteproc's subsystem establishment of virtqueues to conform to a host/guest
>>>>>>> scenario but was abandonned.  That would have been a step in the right direction
>>>>>>> for what you are trying to do.
>>>>>> I was looking at some existing rpmsg code, at it appeared to me that
>>>>>> some adapters, like the "qcom", are creating an rpmsg device that
>>>>>> provides specialized methods for talking to the remote processor(s). I
>>>>>> have assumed this is because that hardware does not allow for running
>>>>>> something remotely that can utilize the virtio queues directly, and so
>>>>>> these rpmsg devices provide code to do the communication with their
>>>>>> hardware. What's not clear is whether these devices are using
>>>>>> rpmsg-over-virtio or if they are creating their own rpmsg facility (and
>>>>>> whether they even support guest-host communication).
>>>>>>
>>>>> The QC implementation is different and does not use virtio - there is
>>>>> a special HW interface between the main and the remote processors.
>>>>> That configuration is valid since RPMSG can be implemented over
>>>>> anything.
>>>>>
>>>>>> What I'm also wondering is what needs to be done differently for virtio
>>>>>> when communicating guest-host vs local CPU to remote processor. I was
>>>>>    From a kernel/guest perspective, not much should be needed.  That said
>>>>> the VMM will need to be supplemented with extra configuration
>>>>> capabilities to instantiate the virtio-rpmsg device.  But that is just
>>>>> off the top of my head without seriously looking at the use case.
>>>>>    From a virtio-bus perspective, there might be an issue if a platform
>>>>> is using remote processors _and_ also instantiating VMs that
>>>>> configures a virtio-rpmsg device.  Again, that is just off the top of
>>>>> my head but needs to be taken into account.
>>>> I am new to rpmsg and virtio, and so my understanding of internals is
>>>> still very limited. Is there someone I can work with to determine what
>>>> needs to be done here? I am guessing that virtio either automatically
>>>> adapts to guest-host or rproc-host - in which case no changes may be
>>>> required - or else it requires a different setup and rpmsg will need to
>>>> be extended to allow for that. If there are changes to rpmsg required,
>>>> we'll want to get those submitted as soon as possible. One complication
>>>> for submitting our driver changes is that it is part of a much larger
>>>> effort to support new hardware, and it may not be possible to submit
>>>> them together with rpmsg changes.
>>> The virtio part won't be a problem.  In your case what is missing is the glue
>>> that will setup the virtqueues and install the RPMSG protocol on top of them.
>>> The 'glue' is the new virtio-rpmsg device that needs to be created.  That part
>>> includes the creation of a new virtio device by the VMM and a kernel driver that
>>> can be called from the virtio_bus once it has been discovered.
>> I don't completely follow. Is there some KVM configuration option that
>> causes the virtio-rpmsg device to be created? And then our host driver
>> will need to be able to respond to some notification and dynamically
>> adapt to each VMM being started? I'm not getting a clear picture of how
>> this works. I'm also not clear on the responsibilities of our guest
>> driver(s) vs. our host driver. For virtio I saw there was the concept of
>> a "driver" side and a "device" side, and the guest seemed to be creating
>> the driver and the host created the device. The rpmsg layer seems to be
>> more complex in that area, so I'm not sure what actions our guest driver
>> with take vs. our host driver.
> KVM has nothing to do with this.  The life of a virtio device starts
> in the VMM (Virtual Machine Manager) where a backend device is created
> and a virtio MMIO entry for that device is added to the device tree
> that is fed to the VM kernel.  When the VM kernel boots the virtio
> MMIO entry in the DT is parsed as part of the normal device discovery
> process and a virtio-device is instantiated, added to the virtio-bus
> and a driver is probed.
>
> I suggest you start looking at that process using the kvmtool and a
> simple virtio device such as virtio-rng.
Looking at the virtio-rng code in kvmtool, I must be missing something.
That looks like it is userland code and never calls into the kernel to
actually create any sort of device for VIRTIO_ID_RNG. It appears to just
add it to a private device list, and I'm not finding any place where
that list gets turned into real devices.

Are you saying that the virtio device on the host is not created until
the VM boots the guest kernel - meaning the VM kernel/driver must take
some action to cause the device to be created on the host? I was
expecting that when the host boots our driver would be creating some
sort of device or entry, and when guests boot our driver there would
register and get matched to the host device. It would really help if I
could see an end-to-end example of this working. But I need some help
identifying the various components involved.

Is it going to be necessary to modify the VMMs to get virtio_rpmsg_bus
devices created?
>>> Everything in the virtio and RPMSG subsystems are aleady tailored to support all
>>> this, so no changes should be needed.  As for the VMM, I suggest to start with
>>> kvmtool.  Lastly, none of this requires "real" hardware or your specific
>>> hardware - it can all be done from QEMU.
>>>
>>>>>> hoping that RPMSG-over-VIRTIO would be easily adapted to what we need.
>>>>>> If we have to create a new virtio device (that is nearly identical to
>>>>>> rpmsg), that is going to push-out SR-IOV support a great deal, plus
>>>>>> requiring cloning of a lot of existing code for a new purpose.
>>>>> Duplication of code would not be a viable way forward.
>>>>> Reusing/enhancing/fixing what is currently available is definitely a
>>>>> better option.
>>>>>
>>>>>> Our only other alternative is to do something to allow guest-host
>>>>>> communication to use the fabric loopback, which is not at all desirable
>>>>>> and has many issues of its own.
>>>>>>
>>>>>>>> Is this the correct way to use RPMSG-over-VIRTIO? If so, what actions
>>>>>>>> need to be taken to cause a "rpmsg_ctl" device to be created? What
>>>>>>>> method would be used (in a kernel driver) to get a pointer to the
>>>>>>>> "rpmsg_ctl" device, for use with rpmsg_create_channel()?
>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Doug
>>>>>>>>>
>>>>>>>> External recipient
>>>>>> External recipient
>>>> External recipient
>>
>> External recipient


External recipient