Re: How to create/use RPMSG-over-VIRTIO devices in Linux

[Doug Miller <doug.miller@cornelisnetworks.com>]

On 9/11/2024 11:12 AM, Mathieu Poirier wrote:
> On Tue, 10 Sept 2024 at 09:43, Doug Miller
> <doug.miller@cornelisnetworks.com> wrote:
>> On 9/10/2024 10:13 AM, Mathieu Poirier wrote:
>>> On Tue, Sep 10, 2024 at 08:12:07AM -0500, Doug Miller wrote:
>>>> On 9/3/2024 10:52 AM, Doug Miller wrote:
>>>>> I am trying to learn how to create an RPMSG-over-VIRTIO device
>>>>> (service) in order to perform communication between a host driver and
>>>>> a guest driver. The RPMSG-over-VIRTIO driver (client) side is fairly
>>>>> well documented and there is a good example (starting point, at least)
>>>>> in samples/rpmsg/rpmsg_client_sample.c.
>>>>>
>>>>> I see that I can create an endpoint (struct rpmsg_endpoint) using
>>>>> rpmsg_create_ept(), and from there I can use rpmsg_send() et al. and
>>>>> the rpmsg_rx_cb_t cb to perform the communications. However, this
>>>>> requires a struct rpmsg_device and it is not clear just how to get one
>>>>> that is suitable for this purpose.
>>>>>
>>>>> It appears that one or both of rpmsg_create_channel() and
>>>>> rpmsg_register_device() are needed in order to obtain a device for the
>>>>> specific host-guest communications channel. At some point, a "root"
>>>>> device is needed that will use virtio (VIRTIO_ID_RPMSG) such that new
>>>>> subdevices can be created for each host-guest pair.
>>>>>
>>>>> In addition, building a kernel with CONFIG_RPMSG, CONFIG_RPMSG_VIRTIO,
>>>>> and CONFIG_RPMSG_NS set, and doing a modprobe virtio_rpmsg_bus, seems
>>>>> to get things setup but that does not result in creation of any "root"
>>>>> rpmsg-over-virtio device. Presumably, any such device would have to be
>>>>> setup to use a specific range of addresses and also be tied to
>>>>> virtio_rpmsg_bus to ensure that virtio is used.
>>>>>
>>>>> It is also not clear if/how register_rpmsg_driver() will be required
>>>>> on the rpmsg driver side, even though the sample code does not use it.
>>>>>
>>>>> So, first questions are:
>>>>>
>>>>> * Am I looking at the correct interfaces in order to create the host
>>>>> rpmsg device side?
>>>>> * What needs to be done to get a "root" rpmsg-over-virtio device
>>>>> created (if required)?
>>>>> * How is a rpmsg-over-virtio device created for each host-guest driver
>>>>> pair, for use with rpmsg_create_ept()?
>>>>> * Does the guest side (rpmsg driver) require any special handling to
>>>>> plug-in to the host driver (rpmsg device) side? Aside from using the
>>>>> correct addresses to match device side.
>>>> It looks to me as though the virtio_rpmsg_bus module can create a
>>>> "rpmsg_ctl" device, which could be used to create channels from which
>>>> endpoints could be created. However, when I load the virtio_rpmsg_bus,
>>>> rpmsg_ns, and rpmsg_core modules there is no "rpmsg_ctl" device created
>>>> (this is running in the host OS, before any VMs are created/run).
>>>>
>>> At this time the modules stated above are all used when a main processor is
>>> controlling a remote processor, i.e via the remoteproc subsystem.  I do not know
>>> of an implementation where VIRTIO_ID_RPMSG is used in the context of a
>>> host/guest scenario.  As such you will find yourself in uncharted territory.
>>>
>>> At some point there were discussion via the OpenAMP body to standardize the
>>> remoteproc's subsystem establishment of virtqueues to conform to a host/guest
>>> scenario but was abandonned.  That would have been a step in the right direction
>>> for what you are trying to do.
>> I was looking at some existing rpmsg code, at it appeared to me that
>> some adapters, like the "qcom", are creating an rpmsg device that
>> provides specialized methods for talking to the remote processor(s). I
>> have assumed this is because that hardware does not allow for running
>> something remotely that can utilize the virtio queues directly, and so
>> these rpmsg devices provide code to do the communication with their
>> hardware. What's not clear is whether these devices are using
>> rpmsg-over-virtio or if they are creating their own rpmsg facility (and
>> whether they even support guest-host communication).
>>
> The QC implementation is different and does not use virtio - there is
> a special HW interface between the main and the remote processors.
> That configuration is valid since RPMSG can be implemented over
> anything.
>
>> What I'm also wondering is what needs to be done differently for virtio
>> when communicating guest-host vs local CPU to remote processor. I was
>  From a kernel/guest perspective, not much should be needed.  That said
> the VMM will need to be supplemented with extra configuration
> capabilities to instantiate the virtio-rpmsg device.  But that is just
> off the top of my head without seriously looking at the use case.
>  From a virtio-bus perspective, there might be an issue if a platform
> is using remote processors _and_ also instantiating VMs that
> configures a virtio-rpmsg device.  Again, that is just off the top of
> my head but needs to be taken into account.
I am new to rpmsg and virtio, and so my understanding of internals is
still very limited. Is there someone I can work with to determine what
needs to be done here? I am guessing that virtio either automatically
adapts to guest-host or rproc-host - in which case no changes may be
required - or else it requires a different setup and rpmsg will need to
be extended to allow for that. If there are changes to rpmsg required,
we'll want to get those submitted as soon as possible. One complication
for submitting our driver changes is that it is part of a much larger
effort to support new hardware, and it may not be possible to submit
them together with rpmsg changes.
>
>> hoping that RPMSG-over-VIRTIO would be easily adapted to what we need.
>> If we have to create a new virtio device (that is nearly identical to
>> rpmsg), that is going to push-out SR-IOV support a great deal, plus
>> requiring cloning of a lot of existing code for a new purpose.
> Duplication of code would not be a viable way forward.
> Reusing/enhancing/fixing what is currently available is definitely a
> better option.
>
>> Our only other alternative is to do something to allow guest-host
>> communication to use the fabric loopback, which is not at all desirable
>> and has many issues of its own.
>>
>>>> Is this the correct way to use RPMSG-over-VIRTIO? If so, what actions
>>>> need to be taken to cause a "rpmsg_ctl" device to be created? What
>>>> method would be used (in a kernel driver) to get a pointer to the
>>>> "rpmsg_ctl" device, for use with rpmsg_create_channel()?
>>>>
>>>>> Thanks,
>>>>> Doug
>>>>>
>>>> External recipient
>>
>> External recipient


External recipient