rpmsg: exposing the bus to userspace

rpmsg: exposing the bus to userspace

[Matteo Sartori]

Hi,

I have been working on remoteproc/rpmsg and FreeRTOS/OpenAMP for the
latest three months and recently, with my colleagues, I have started
reasoning about the best possible way to expose the kernel rpmsg bus
to applications running in user space. The idea is to end up with a
clean mechanism for the user-kernel communication, like sockets or
char devices, in order to build AMP applications on top of it.

In fact, we have been running experiments with some out-of-tree
drivers implementing exactly such interfaces, namely a driver
registering a new address family AF_RPMSG for the socket interface,
from Texas Instruments, and a char device driver representing
different rpmsg channels with /dev/rpmsgX special files, found inside
OpenAMP obsolete directory.

I am quite new to kernel development and in particular to remoteproc
subsystem, so I am asking the following questions because maybe I have
missed some past developments and discussions about this topic. I
apologize if this is the case.

- Have you ever considered the inclusion of these out-of-tree modules
in the mainline?
- If so, have you planned some work on this, or what are the reasons
for not including these modules?
- Is there already a preferred way for exposing the rpmsg bus for
generic services to userspace, or is the idea to have a different
rpmsg driver, therefore a user-kernel communication strategy, for each
different use case of the bus?

Anyway, I would be very glad to contribute in the direction of an
appropriate user-kernel interface for accessing the rpmsg bus, and at
the moment I have a working test case for the socket driver as well as
for the char driver. Given the fact I don't see the superiority of one
approach compared to the other, I would like to propose and discuss
with you both of them.

This was kind of an introductory email. I am preparing two patchsets.

Best regards,
Matteo

Re: rpmsg: exposing the bus to userspace

[Bjorn Andersson]

On Thu 04 Aug 03:33 PDT 2016, Matteo Sartori wrote:

> Hi,
> 

Hi Matteo,

> I have been working on remoteproc/rpmsg and FreeRTOS/OpenAMP for the
> latest three months and recently, with my colleagues, I have started
> reasoning about the best possible way to expose the kernel rpmsg bus
> to applications running in user space. The idea is to end up with a
> clean mechanism for the user-kernel communication, like sockets or
> char devices, in order to build AMP applications on top of it.
> 

Cool, I see this coming up on several fronts.

I'm currently finishing up a patch series where I hope to bring support
for additional backends/wire formats to rpmsg (e.g. Qualcomm SMD). One
of the users of this (the Qualcomm case) is a packet interface for some
userspace parts to communicate with specialized services on the modem
processor.

> In fact, we have been running experiments with some out-of-tree
> drivers implementing exactly such interfaces, namely a driver
> registering a new address family AF_RPMSG for the socket interface,
> from Texas Instruments, and a char device driver representing
> different rpmsg channels with /dev/rpmsgX special files, found inside
> OpenAMP obsolete directory.
> 

Going with the AF_RPMSG approach seems (at first) to map quite nicely
with the src/dst port model. It would e.g. allow us to use recvfrom() to
implement services for multiple clients.

But the src/dst are only unique for each given virtproc, so we need to
make up a mechanism for selecting among a set of rpmsg instances if more
than one remote exists. Further more I believe that the AF_RPMSG concept
would have to be integrated into the core of rpmsg and we would have two
different concepts intermixed in the implementation; the in-kernel model
with life cycle management and the user-space variant - and we need to
figure out how to make these not step on each others toes.


So I'm preferring a model where we can have a rpmsg-client that can be
attached to an endpoint and expose this explicitly to userspace.


In the Qualcomm case mentioned above, you only have defined
point-to-point channels, so spawning a char device for each one is
rather natural. But the N:1 case (a generic service in the virtio rpmsg
case) requires a recvfrom()/sendto() like interface.

So further investigation/discussions on this is needed.

> I am quite new to kernel development and in particular to remoteproc
> subsystem, so I am asking the following questions because maybe I have
> missed some past developments and discussions about this topic. I
> apologize if this is the case.
> 

There have been people looking into this, but I have not yet seen any
output heading towards the mainline kernel.

> - Have you ever considered the inclusion of these out-of-tree modules
> in the mainline?

Yes, enough times so that we will have to address this rather soon.

> - If so, have you planned some work on this, or what are the reasons
> for not including these modules?

Yes. I need to go back and review these implementations to give you a
good answer, it's not a bad idea to base the mainline solution on
existing downstream code - if it's a good fit.

> - Is there already a preferred way for exposing the rpmsg bus for
> generic services to userspace, or is the idea to have a different
> rpmsg driver, therefore a user-kernel communication strategy, for each
> different use case of the bus?
> 

The preferred way, in a lot of cases is to have a specialized kernel
driver dealing with buffer management and communication and using one of
the generic frameworks for exposing this to user space; whenever
possible.

But there are cases (even in production) where this is not feasible, so
we need to address this.

> Anyway, I would be very glad to contribute in the direction of an
> appropriate user-kernel interface for accessing the rpmsg bus, and at
> the moment I have a working test case for the socket driver as well as
> for the char driver. Given the fact I don't see the superiority of one
> approach compared to the other, I would like to propose and discuss
> with you both of them.
> 

Sounds good, thanks for waking the discussion to life again. I added
Akashi-san to the thread as well, as he's was looking at the TI patches
before.

Regards,
Bjorn

Re: rpmsg: exposing the bus to userspace

[Matteo Sartori]

Hi Bjorn,
apologies for the latency.

On 2016-08-10 22:03 GMT+02:00, Bjorn Andersson wrote:
> Going with the AF_RPMSG approach seems (at first) to map quite nicely
> with the src/dst port model. It would e.g. allow us to use recvfrom() to
> implement services for multiple clients.
>
> But the src/dst are only unique for each given virtproc, so we need to
> make up a mechanism for selecting among a set of rpmsg instances if more
> than one remote exists. Further more I believe that the AF_RPMSG concept
> would have to be integrated into the core of rpmsg and we would have two
> different concepts intermixed in the implementation; the in-kernel model
> with life cycle management and the user-space variant - and we need to
> figure out how to make these not step on each others toes.

Can you please elaborate a little more on the intermixed implementations?
For what I understand it is necessary to introduce the AF_RPSMG concept
inside the rpmsg core because e.g. after a bind() syscall on a given port
(local rpmsg address) it is important to create a new endpoint, and to register
a new callback with.
But, I don't see why there should be a user-space variant of the
AF_RPMSG concept.


> So I'm preferring a model where we can have a rpmsg-client that can be
> attached to an endpoint and expose this explicitly to userspace.

Do you refer to the possibility that a userspace process can create
and attach to a new endpoint?
>From my experience, the only method supporting this kind of userspace-
kernelspace interaction is that of sockets, where the bind() syscall semantic
can be translated into the creation of new rpmsg endpoint which subsequent
recvfrom()s can perform the desired behaviour on, that is receiving traffic
from a given port while knowing the source address of each data unit.

Do you have other userspace/kernelspace interaction models to be
considered ? Maybe, I am not aware of other possibilities.


> In the Qualcomm case mentioned above, you only have defined
> point-to-point channels, so spawning a char device for each one is
> rather natural. But the N:1 case (a generic service in the virtio rpmsg
> case) requires a recvfrom()/sendto() like interface.

I agree, the problem of using char devices is that they model only
the situation where there is a rpmsg channel underneath, that is a
communication link already installed between two ends.

In my point of view the rpmsg channel abstraction matches quite well
with char devices but there are problems with the particular mechanism
rpmsg uses to match devices with drivers.
For instance, the char driver I have tested these days spawns a
new /dev/rpmsgX every time the rpmsg bus sees a channel id matching
the one specified by the driver. If there are two rprocs that create a new
channel with the same name (eg "rpmsg-char-dev-channel") and the driver
matches such channel name, then two new char devices will be created,
/dev/rpmsg0 and /dev/rpmsg1. How do I know which char device is the
target I want to communcate with? There is no control over the minor
number assignment and it depends on the order of probe() calls the rpmsg
bus performs.

The first solution that comes to my mind is to use the information
that rpmsg and remoteproc subsystems expose to sysfs. This way
a user space program can understand the correspondence
between char devices and actual rpmsg channels and rprocs.
For this solution the code from texas is already working, given the
fact it is up to the userspace program to retrieve additional informations.

Instead, if we desire more control inside the kernel we should consider
a different creation strategy, such as a /dev/rpmsgXY scheme.
The X standing for the rproc index and the Y standing for a particular
digit representing different channels. For instance, if we have two remotes
each with a single channel in the system we could name the char devices
as /dev/rpmsg00 and /dev/rpmsg10.


To sum up, I would say the char approach is more suitable for static
configurations and is oriented to represent a rpmsg channel while the
socket approach is more dynamic, given the fact it permits generic
services to be deployed and it is a little bit detached from the rpmsg channel
concept. In my opinion, we should consider both since socket can give
more powerful communication paradigms while char devices represent a
point-to-point link and are more straightforward to use.

Best regards,
Matteo

Re: rpmsg: exposing the bus to userspace

[Bjorn Andersson]

On Thu 25 Aug 03:31 PDT 2016, Matteo Sartori wrote:

> Hi Bjorn,
> apologies for the latency.
> 

Dito.

> On 2016-08-10 22:03 GMT+02:00, Bjorn Andersson wrote:
> > Going with the AF_RPMSG approach seems (at first) to map quite nicely
> > with the src/dst port model. It would e.g. allow us to use recvfrom() to
> > implement services for multiple clients.
> >
> > But the src/dst are only unique for each given virtproc, so we need to
> > make up a mechanism for selecting among a set of rpmsg instances if more
> > than one remote exists. Further more I believe that the AF_RPMSG concept
> > would have to be integrated into the core of rpmsg and we would have two
> > different concepts intermixed in the implementation; the in-kernel model
> > with life cycle management and the user-space variant - and we need to
> > figure out how to make these not step on each others toes.
> 
> Can you please elaborate a little more on the intermixed implementations?

After studying rpmsg_proto in detail I take this comment back. After my
first read-through I interpreted the code as you expose the entire
address space to via the socket interface.

After reading the implementation more thoroughly I now see that the
rpmsg_proto implementation only exposes specific channels (i.e. the ones
named "rpmsg-proto") to userspace.

> For what I understand it is necessary to introduce the AF_RPSMG concept
> inside the rpmsg core because e.g. after a bind() syscall on a given port
> (local rpmsg address) it is important to create a new endpoint, and to register
> a new callback with.

This can be done with rpmsg_create_ept()

> But, I don't see why there should be a user-space variant of the
> AF_RPMSG concept.
> 
> 
> > So I'm preferring a model where we can have a rpmsg-client that can be
> > attached to an endpoint and expose this explicitly to userspace.
> 
> Do you refer to the possibility that a userspace process can create
> and attach to a new endpoint?

Yes

On the Qualcomm platforms we have a varying number of channels per
remoteproc that we would like to be able to access from user space.

The alternative to allowing user space to request channels to be
exported would be to list them in devicetree - but this is not a
property of the hardware.

> From my experience, the only method supporting this kind of userspace-
> kernelspace interaction is that of sockets, where the bind() syscall semantic
> can be translated into the creation of new rpmsg endpoint which subsequent
> recvfrom()s can perform the desired behaviour on, that is receiving traffic
> from a given port while knowing the source address of each data unit.
> 
> Do you have other userspace/kernelspace interaction models to be
> considered ? Maybe, I am not aware of other possibilities.
> 

There are two that comes to mind, either using a character device per
rpmsg instance that provides an ioctl interface for
exporting/unexporting endpoints or through some sysfs interface.

Per the design of rpmsg_proto we want to be able to specify both src and
dst address, and from Qualcomm SMD we want to be able to control this
with channel names. So I think the ioctl based approach would be easier.

> 
> > In the Qualcomm case mentioned above, you only have defined
> > point-to-point channels, so spawning a char device for each one is
> > rather natural. But the N:1 case (a generic service in the virtio rpmsg
> > case) requires a recvfrom()/sendto() like interface.
> 
> I agree, the problem of using char devices is that they model only
> the situation where there is a rpmsg channel underneath, that is a
> communication link already installed between two ends.
> 

So does rpmsg_proto, it only exposes communication channels after the
rpmsg core has probed upon finding a "rpmsg-proto" channel.

> In my point of view the rpmsg channel abstraction matches quite well
> with char devices but there are problems with the particular mechanism
> rpmsg uses to match devices with drivers.
> For instance, the char driver I have tested these days spawns a
> new /dev/rpmsgX every time the rpmsg bus sees a channel id matching
> the one specified by the driver. If there are two rprocs that create a new
> channel with the same name (eg "rpmsg-char-dev-channel") and the driver
> matches such channel name, then two new char devices will be created,
> /dev/rpmsg0 and /dev/rpmsg1. How do I know which char device is the
> target I want to communcate with? There is no control over the minor
> number assignment and it depends on the order of probe() calls the rpmsg
> bus performs.
> 

We can use udev rules to rename the rpmsg devices based on the channel
names, but currently we don't have a convenient name for the parent - so
we probably need to look into that some more.

So I think this is okay, from the kernels pov.

> The first solution that comes to my mind is to use the information
> that rpmsg and remoteproc subsystems expose to sysfs. This way
> a user space program can understand the correspondence
> between char devices and actual rpmsg channels and rprocs.
> For this solution the code from texas is already working, given the
> fact it is up to the userspace program to retrieve additional informations.
> 

I think this is a good way forward, perhaps we need to provide
additional attributes so that it's possible to build a user friendly dev
name.

> Instead, if we desire more control inside the kernel we should consider
> a different creation strategy, such as a /dev/rpmsgXY scheme.
> The X standing for the rproc index and the Y standing for a particular
> digit representing different channels. For instance, if we have two remotes
> each with a single channel in the system we could name the char devices
> as /dev/rpmsg00 and /dev/rpmsg10.
> 

I would prefer that we find a solution to exposing the parent name in a
way that it's possible to write a udev rule to get
/dev/rpmsg/<parent-label>/<channel-name>

Perhaps if we just add support for an optional "label" property for
controlling the remoteproc's dev->name we would have what we need there
- and solve the annoyance of the probe order dependant naming of the
remoteprocs.

> 
> To sum up, I would say the char approach is more suitable for static
> configurations and is oriented to represent a rpmsg channel while the
> socket approach is more dynamic, given the fact it permits generic
> services to be deployed and it is a little bit detached from the rpmsg channel
> concept.

I was expecting to find a socket based implementation that allows for
unbound services to be implemented in userspace, but was happy to see
that neither of the TI solutions I looked at supported this today.

Further more, the Qualcomm SMD implementation doesn't support offchannel
communication, so this simplification suits that as well.


Not supporting this simplifies the problem and we should be able to
implement this without tangling it into the rpmsg implementation too
much.

> In my opinion, we should consider both since socket can give
> more powerful communication paradigms while char devices represent a
> point-to-point link and are more straightforward to use.
> 

I agree with the view that sockets would give a more powerful model, but
unless we _really_ need that I would prefer if we don't go there.
Most use-cases I've seen does not require support for implementing
unbound services and the ability to do so in-kernel is still there.



So my suggestion is that we upon probing a rpmsg device create a
rpmsg_ctrl%d character device, which has an ioctl interface for
exporting and unexporting endpoints. We can be a little bit optimistic
and have the export create a rpmsg%d device, with the associated
rpmsg_channel_info.

Then upon getting the open() callback we issue rpmsg_create_ept() and
release() calls rpmsg_destroy_ept().

For rpmsg backends that support advertising channel state this would
mean that once the char dev is open we have an open rpmsg channel.


To be able to use the rpmsg_create_ept() interface we need to have a
rpmsg_device handle for the backend-specific rpmsg instance, so I think
the first step needs to be initiated by each rpmsg backend by creating a
"dummy" rpmsg_device and triggering the registration above.

Regards,
Bjorn

RE: rpmsg: exposing the bus to userspace

[Marek Novak]

Hello Bjorn and the community,

Bjorn mentioned a sysfs-based exporting mechanism.
I have written a kernel module, which allows exporting RPMsg endpoints to the user space.
If you want, please look at the code here: https://github.com/EmbeddedRPC/erpc-imx-demos/blob/master/MPU/setup_yocto/rpmsg_sysfs_interface.patch

I am preparing an example/demonstration repository on Github showing its use-case here:
https://github.com/EmbeddedRPC/erpc-imx-demos


Regards,
Marek NOVAK
NXP Semiconductors
Brno University of Technology

-----Original Message-----
From: linux-remoteproc-owner@vger.kernel.org [mailto:linux-remoteproc-owner@vger.kernel.org] On Behalf Of Bjorn Andersson
Sent: Wednesday, September 21, 2016 6:09 AM
To: Matteo Sartori <matteo.sartori@t3lab.it>
Cc: takahiro.akashi@linaro.org; linux-remoteproc@vger.kernel.org; Alessio Paccoia <alessio.paccoia@t3lab.it>; Michele Rodolfi <michele.rodolfi@t3lab.it>; Claudio Salati <claudio.salati@t3lab.it>
Subject: Re: rpmsg: exposing the bus to userspace

On Thu 25 Aug 03:31 PDT 2016, Matteo Sartori wrote:

> Hi Bjorn,
> apologies for the latency.
> 

Dito.

> On 2016-08-10 22:03 GMT+02:00, Bjorn Andersson wrote:
> > Going with the AF_RPMSG approach seems (at first) to map quite 
> > nicely with the src/dst port model. It would e.g. allow us to use 
> > recvfrom() to implement services for multiple clients.
> >
> > But the src/dst are only unique for each given virtproc, so we need 
> > to make up a mechanism for selecting among a set of rpmsg instances 
> > if more than one remote exists. Further more I believe that the 
> > AF_RPMSG concept would have to be integrated into the core of rpmsg 
> > and we would have two different concepts intermixed in the 
> > implementation; the in-kernel model with life cycle management and 
> > the user-space variant - and we need to figure out how to make these not step on each others toes.
> 
> Can you please elaborate a little more on the intermixed implementations?

After studying rpmsg_proto in detail I take this comment back. After my first read-through I interpreted the code as you expose the entire address space to via the socket interface.

After reading the implementation more thoroughly I now see that the rpmsg_proto implementation only exposes specific channels (i.e. the ones named "rpmsg-proto") to userspace.

> For what I understand it is necessary to introduce the AF_RPSMG 
> concept inside the rpmsg core because e.g. after a bind() syscall on a 
> given port (local rpmsg address) it is important to create a new 
> endpoint, and to register a new callback with.

This can be done with rpmsg_create_ept()

> But, I don't see why there should be a user-space variant of the 
> AF_RPMSG concept.
> 
> 
> > So I'm preferring a model where we can have a rpmsg-client that can 
> > be attached to an endpoint and expose this explicitly to userspace.
> 
> Do you refer to the possibility that a userspace process can create 
> and attach to a new endpoint?

Yes

On the Qualcomm platforms we have a varying number of channels per remoteproc that we would like to be able to access from user space.

The alternative to allowing user space to request channels to be exported would be to list them in devicetree - but this is not a property of the hardware.

> From my experience, the only method supporting this kind of userspace- 
> kernelspace interaction is that of sockets, where the bind() syscall 
> semantic can be translated into the creation of new rpmsg endpoint 
> which subsequent recvfrom()s can perform the desired behaviour on, 
> that is receiving traffic from a given port while knowing the source address of each data unit.
> 
> Do you have other userspace/kernelspace interaction models to be 
> considered ? Maybe, I am not aware of other possibilities.
> 

There are two that comes to mind, either using a character device per rpmsg instance that provides an ioctl interface for exporting/unexporting endpoints or through some sysfs interface.

Per the design of rpmsg_proto we want to be able to specify both src and dst address, and from Qualcomm SMD we want to be able to control this with channel names. So I think the ioctl based approach would be easier.

> 
> > In the Qualcomm case mentioned above, you only have defined 
> > point-to-point channels, so spawning a char device for each one is 
> > rather natural. But the N:1 case (a generic service in the virtio 
> > rpmsg
> > case) requires a recvfrom()/sendto() like interface.
> 
> I agree, the problem of using char devices is that they model only the 
> situation where there is a rpmsg channel underneath, that is a 
> communication link already installed between two ends.
> 

So does rpmsg_proto, it only exposes communication channels after the rpmsg core has probed upon finding a "rpmsg-proto" channel.

> In my point of view the rpmsg channel abstraction matches quite well 
> with char devices but there are problems with the particular mechanism 
> rpmsg uses to match devices with drivers.
> For instance, the char driver I have tested these days spawns a new 
> /dev/rpmsgX every time the rpmsg bus sees a channel id matching the 
> one specified by the driver. If there are two rprocs that create a new 
> channel with the same name (eg "rpmsg-char-dev-channel") and the 
> driver matches such channel name, then two new char devices will be 
> created,
> /dev/rpmsg0 and /dev/rpmsg1. How do I know which char device is the 
> target I want to communcate with? There is no control over the minor 
> number assignment and it depends on the order of probe() calls the 
> rpmsg bus performs.
> 

We can use udev rules to rename the rpmsg devices based on the channel names, but currently we don't have a convenient name for the parent - so we probably need to look into that some more.

So I think this is okay, from the kernels pov.

> The first solution that comes to my mind is to use the information 
> that rpmsg and remoteproc subsystems expose to sysfs. This way a user 
> space program can understand the correspondence between char devices 
> and actual rpmsg channels and rprocs.
> For this solution the code from texas is already working, given the 
> fact it is up to the userspace program to retrieve additional informations.
> 

I think this is a good way forward, perhaps we need to provide additional attributes so that it's possible to build a user friendly dev name.

> Instead, if we desire more control inside the kernel we should 
> consider a different creation strategy, such as a /dev/rpmsgXY scheme.
> The X standing for the rproc index and the Y standing for a particular 
> digit representing different channels. For instance, if we have two 
> remotes each with a single channel in the system we could name the 
> char devices as /dev/rpmsg00 and /dev/rpmsg10.
> 

I would prefer that we find a solution to exposing the parent name in a way that it's possible to write a udev rule to get /dev/rpmsg/<parent-label>/<channel-name>

Perhaps if we just add support for an optional "label" property for controlling the remoteproc's dev->name we would have what we need there
- and solve the annoyance of the probe order dependant naming of the remoteprocs.

> 
> To sum up, I would say the char approach is more suitable for static 
> configurations and is oriented to represent a rpmsg channel while the 
> socket approach is more dynamic, given the fact it permits generic 
> services to be deployed and it is a little bit detached from the rpmsg 
> channel concept.

I was expecting to find a socket based implementation that allows for unbound services to be implemented in userspace, but was happy to see that neither of the TI solutions I looked at supported this today.

Further more, the Qualcomm SMD implementation doesn't support offchannel communication, so this simplification suits that as well.


Not supporting this simplifies the problem and we should be able to implement this without tangling it into the rpmsg implementation too much.

> In my opinion, we should consider both since socket can give more 
> powerful communication paradigms while char devices represent a 
> point-to-point link and are more straightforward to use.
> 

I agree with the view that sockets would give a more powerful model, but unless we _really_ need that I would prefer if we don't go there.
Most use-cases I've seen does not require support for implementing unbound services and the ability to do so in-kernel is still there.



So my suggestion is that we upon probing a rpmsg device create a rpmsg_ctrl%d character device, which has an ioctl interface for exporting and unexporting endpoints. We can be a little bit optimistic and have the export create a rpmsg%d device, with the associated rpmsg_channel_info.

Then upon getting the open() callback we issue rpmsg_create_ept() and
release() calls rpmsg_destroy_ept().

For rpmsg backends that support advertising channel state this would mean that once the char dev is open we have an open rpmsg channel.


To be able to use the rpmsg_create_ept() interface we need to have a rpmsg_device handle for the backend-specific rpmsg instance, so I think the first step needs to be initiated by each rpmsg backend by creating a "dummy" rpmsg_device and triggering the registration above.

Regards,
Bjorn
--
To unsubscribe from this list: send the line "unsubscribe linux-remoteproc" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rpmsg: exposing the bus to userspace

[Bjorn Andersson]

On Tue 20 Sep 21:43 PDT 2016, Marek Novak wrote:

> Hello Bjorn and the community,
> 

Hi Marek

> Bjorn mentioned a sysfs-based exporting mechanism.
> I have written a kernel module, which allows exporting RPMsg endpoints
> to the user space.  If you want, please look at the code here:
> https://github.com/EmbeddedRPC/erpc-imx-demos/blob/master/MPU/setup_yocto/rpmsg_sysfs_interface.patch
> 

This is in line with my ideas, allowing user space to open arbitrary
channels.

I think we can afford replacing the originating rpmsg device with a
special rpmsg_device created straight from the individual backends - so
that we don't have to rely on a specific channel to be exposed to
"enable" this feature.

> I am preparing an example/demonstration repository on Github showing its use-case here:
> https://github.com/EmbeddedRPC/erpc-imx-demos
> 

Thanks,
Bjorn

> 
> Regards,
> Marek NOVAK
> NXP Semiconductors
> Brno University of Technology
> 
> -----Original Message-----
> From: linux-remoteproc-owner@vger.kernel.org [mailto:linux-remoteproc-owner@vger.kernel.org] On Behalf Of Bjorn Andersson
> Sent: Wednesday, September 21, 2016 6:09 AM
> To: Matteo Sartori <matteo.sartori@t3lab.it>
> Cc: takahiro.akashi@linaro.org; linux-remoteproc@vger.kernel.org; Alessio Paccoia <alessio.paccoia@t3lab.it>; Michele Rodolfi <michele.rodolfi@t3lab.it>; Claudio Salati <claudio.salati@t3lab.it>
> Subject: Re: rpmsg: exposing the bus to userspace
> 
> On Thu 25 Aug 03:31 PDT 2016, Matteo Sartori wrote:
> 
> > Hi Bjorn,
> > apologies for the latency.
> > 
> 
> Dito.
> 
> > On 2016-08-10 22:03 GMT+02:00, Bjorn Andersson wrote:
> > > Going with the AF_RPMSG approach seems (at first) to map quite 
> > > nicely with the src/dst port model. It would e.g. allow us to use 
> > > recvfrom() to implement services for multiple clients.
> > >
> > > But the src/dst are only unique for each given virtproc, so we need 
> > > to make up a mechanism for selecting among a set of rpmsg instances 
> > > if more than one remote exists. Further more I believe that the 
> > > AF_RPMSG concept would have to be integrated into the core of rpmsg 
> > > and we would have two different concepts intermixed in the 
> > > implementation; the in-kernel model with life cycle management and 
> > > the user-space variant - and we need to figure out how to make these not step on each others toes.
> > 
> > Can you please elaborate a little more on the intermixed implementations?
> 
> After studying rpmsg_proto in detail I take this comment back. After my first read-through I interpreted the code as you expose the entire address space to via the socket interface.
> 
> After reading the implementation more thoroughly I now see that the rpmsg_proto implementation only exposes specific channels (i.e. the ones named "rpmsg-proto") to userspace.
> 
> > For what I understand it is necessary to introduce the AF_RPSMG 
> > concept inside the rpmsg core because e.g. after a bind() syscall on a 
> > given port (local rpmsg address) it is important to create a new 
> > endpoint, and to register a new callback with.
> 
> This can be done with rpmsg_create_ept()
> 
> > But, I don't see why there should be a user-space variant of the 
> > AF_RPMSG concept.
> > 
> > 
> > > So I'm preferring a model where we can have a rpmsg-client that can 
> > > be attached to an endpoint and expose this explicitly to userspace.
> > 
> > Do you refer to the possibility that a userspace process can create 
> > and attach to a new endpoint?
> 
> Yes
> 
> On the Qualcomm platforms we have a varying number of channels per remoteproc that we would like to be able to access from user space.
> 
> The alternative to allowing user space to request channels to be exported would be to list them in devicetree - but this is not a property of the hardware.
> 
> > From my experience, the only method supporting this kind of userspace- 
> > kernelspace interaction is that of sockets, where the bind() syscall 
> > semantic can be translated into the creation of new rpmsg endpoint 
> > which subsequent recvfrom()s can perform the desired behaviour on, 
> > that is receiving traffic from a given port while knowing the source address of each data unit.
> > 
> > Do you have other userspace/kernelspace interaction models to be 
> > considered ? Maybe, I am not aware of other possibilities.
> > 
> 
> There are two that comes to mind, either using a character device per rpmsg instance that provides an ioctl interface for exporting/unexporting endpoints or through some sysfs interface.
> 
> Per the design of rpmsg_proto we want to be able to specify both src and dst address, and from Qualcomm SMD we want to be able to control this with channel names. So I think the ioctl based approach would be easier.
> 
> > 
> > > In the Qualcomm case mentioned above, you only have defined 
> > > point-to-point channels, so spawning a char device for each one is 
> > > rather natural. But the N:1 case (a generic service in the virtio 
> > > rpmsg
> > > case) requires a recvfrom()/sendto() like interface.
> > 
> > I agree, the problem of using char devices is that they model only the 
> > situation where there is a rpmsg channel underneath, that is a 
> > communication link already installed between two ends.
> > 
> 
> So does rpmsg_proto, it only exposes communication channels after the rpmsg core has probed upon finding a "rpmsg-proto" channel.
> 
> > In my point of view the rpmsg channel abstraction matches quite well 
> > with char devices but there are problems with the particular mechanism 
> > rpmsg uses to match devices with drivers.
> > For instance, the char driver I have tested these days spawns a new 
> > /dev/rpmsgX every time the rpmsg bus sees a channel id matching the 
> > one specified by the driver. If there are two rprocs that create a new 
> > channel with the same name (eg "rpmsg-char-dev-channel") and the 
> > driver matches such channel name, then two new char devices will be 
> > created,
> > /dev/rpmsg0 and /dev/rpmsg1. How do I know which char device is the 
> > target I want to communcate with? There is no control over the minor 
> > number assignment and it depends on the order of probe() calls the 
> > rpmsg bus performs.
> > 
> 
> We can use udev rules to rename the rpmsg devices based on the channel names, but currently we don't have a convenient name for the parent - so we probably need to look into that some more.
> 
> So I think this is okay, from the kernels pov.
> 
> > The first solution that comes to my mind is to use the information 
> > that rpmsg and remoteproc subsystems expose to sysfs. This way a user 
> > space program can understand the correspondence between char devices 
> > and actual rpmsg channels and rprocs.
> > For this solution the code from texas is already working, given the 
> > fact it is up to the userspace program to retrieve additional informations.
> > 
> 
> I think this is a good way forward, perhaps we need to provide additional attributes so that it's possible to build a user friendly dev name.
> 
> > Instead, if we desire more control inside the kernel we should 
> > consider a different creation strategy, such as a /dev/rpmsgXY scheme.
> > The X standing for the rproc index and the Y standing for a particular 
> > digit representing different channels. For instance, if we have two 
> > remotes each with a single channel in the system we could name the 
> > char devices as /dev/rpmsg00 and /dev/rpmsg10.
> > 
> 
> I would prefer that we find a solution to exposing the parent name in a way that it's possible to write a udev rule to get /dev/rpmsg/<parent-label>/<channel-name>
> 
> Perhaps if we just add support for an optional "label" property for controlling the remoteproc's dev->name we would have what we need there
> - and solve the annoyance of the probe order dependant naming of the remoteprocs.
> 
> > 
> > To sum up, I would say the char approach is more suitable for static 
> > configurations and is oriented to represent a rpmsg channel while the 
> > socket approach is more dynamic, given the fact it permits generic 
> > services to be deployed and it is a little bit detached from the rpmsg 
> > channel concept.
> 
> I was expecting to find a socket based implementation that allows for unbound services to be implemented in userspace, but was happy to see that neither of the TI solutions I looked at supported this today.
> 
> Further more, the Qualcomm SMD implementation doesn't support offchannel communication, so this simplification suits that as well.
> 
> 
> Not supporting this simplifies the problem and we should be able to implement this without tangling it into the rpmsg implementation too much.
> 
> > In my opinion, we should consider both since socket can give more 
> > powerful communication paradigms while char devices represent a 
> > point-to-point link and are more straightforward to use.
> > 
> 
> I agree with the view that sockets would give a more powerful model, but unless we _really_ need that I would prefer if we don't go there.
> Most use-cases I've seen does not require support for implementing unbound services and the ability to do so in-kernel is still there.
> 
> 
> 
> So my suggestion is that we upon probing a rpmsg device create a rpmsg_ctrl%d character device, which has an ioctl interface for exporting and unexporting endpoints. We can be a little bit optimistic and have the export create a rpmsg%d device, with the associated rpmsg_channel_info.
> 
> Then upon getting the open() callback we issue rpmsg_create_ept() and
> release() calls rpmsg_destroy_ept().
> 
> For rpmsg backends that support advertising channel state this would mean that once the char dev is open we have an open rpmsg channel.
> 
> 
> To be able to use the rpmsg_create_ept() interface we need to have a rpmsg_device handle for the backend-specific rpmsg instance, so I think the first step needs to be initiated by each rpmsg backend by creating a "dummy" rpmsg_device and triggering the registration above.
> 
> Regards,
> Bjorn
> --
> To unsubscribe from this list: send the line "unsubscribe linux-remoteproc" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rpmsg: exposing the bus to userspace

[Marek Novak]

[-- Attachment #1: Type: text/plain, Size: 11045 bytes --]

Hello Bjorn and the community,

I also support the idea of making the export of RPMsg endpoints to the user-space part of the virtio_rpmsg_bus kernel module.

As you suggest, it would allow to skip the channel concept, which is not really necessary for the communication. (RPMsg-Lite implementation is an evidence of this idea)

Thanks & Regards,
Marek

________________________________
From: Bjorn Andersson <bjorn.andersson@linaro.org>
Sent: Sep 29, 2016 7:35 PM
To: Marek Novak
Cc: Matteo Sartori; takahiro.akashi@linaro.org; linux-remoteproc@vger.kernel.org; Alessio Paccoia; Michele Rodolfi; Claudio Salati; Michal Princ; Petr Lukas
Subject: Re: rpmsg: exposing the bus to userspace

On Tue 20 Sep 21:43 PDT 2016, Marek Novak wrote:

> Hello Bjorn and the community,
>

Hi Marek

> Bjorn mentioned a sysfs-based exporting mechanism.
> I have written a kernel module, which allows exporting RPMsg endpoints
> to the user space.  If you want, please look at the code here:
> https://github.com/EmbeddedRPC/erpc-imx-demos/blob/master/MPU/setup_yocto/rpmsg_sysfs_interface.patch
>

This is in line with my ideas, allowing user space to open arbitrary
channels.

I think we can afford replacing the originating rpmsg device with a
special rpmsg_device created straight from the individual backends - so
that we don't have to rely on a specific channel to be exposed to
"enable" this feature.

> I am preparing an example/demonstration repository on Github showing its use-case here:
> https://github.com/EmbeddedRPC/erpc-imx-demos
>

Thanks,
Bjorn

>
> Regards,
> Marek NOVAK
> NXP Semiconductors
> Brno University of Technology
>
> -----Original Message-----
> From: linux-remoteproc-owner@vger.kernel.org [mailto:linux-remoteproc-owner@vger.kernel.org] On Behalf Of Bjorn Andersson
> Sent: Wednesday, September 21, 2016 6:09 AM
> To: Matteo Sartori <matteo.sartori@t3lab.it>
> Cc: takahiro.akashi@linaro.org; linux-remoteproc@vger.kernel.org; Alessio Paccoia <alessio.paccoia@t3lab.it>; Michele Rodolfi <michele.rodolfi@t3lab.it>; Claudio Salati <claudio.salati@t3lab.it>
> Subject: Re: rpmsg: exposing the bus to userspace
>
> On Thu 25 Aug 03:31 PDT 2016, Matteo Sartori wrote:
>
> > Hi Bjorn,
> > apologies for the latency.
> >
>
> Dito.
>
> > On 2016-08-10 22:03 GMT+02:00, Bjorn Andersson wrote:
> > > Going with the AF_RPMSG approach seems (at first) to map quite
> > > nicely with the src/dst port model. It would e.g. allow us to use
> > > recvfrom() to implement services for multiple clients.
> > >
> > > But the src/dst are only unique for each given virtproc, so we need
> > > to make up a mechanism for selecting among a set of rpmsg instances
> > > if more than one remote exists. Further more I believe that the
> > > AF_RPMSG concept would have to be integrated into the core of rpmsg
> > > and we would have two different concepts intermixed in the
> > > implementation; the in-kernel model with life cycle management and
> > > the user-space variant - and we need to figure out how to make these not step on each others toes.
> >
> > Can you please elaborate a little more on the intermixed implementations?
>
> After studying rpmsg_proto in detail I take this comment back. After my first read-through I interpreted the code as you expose the entire address space to via the socket interface.
>
> After reading the implementation more thoroughly I now see that the rpmsg_proto implementation only exposes specific channels (i.e. the ones named "rpmsg-proto") to userspace.
>
> > For what I understand it is necessary to introduce the AF_RPSMG
> > concept inside the rpmsg core because e.g. after a bind() syscall on a
> > given port (local rpmsg address) it is important to create a new
> > endpoint, and to register a new callback with.
>
> This can be done with rpmsg_create_ept()
>
> > But, I don't see why there should be a user-space variant of the
> > AF_RPMSG concept.
> >
> >
> > > So I'm preferring a model where we can have a rpmsg-client that can
> > > be attached to an endpoint and expose this explicitly to userspace.
> >
> > Do you refer to the possibility that a userspace process can create
> > and attach to a new endpoint?
>
> Yes
>
> On the Qualcomm platforms we have a varying number of channels per remoteproc that we would like to be able to access from user space.
>
> The alternative to allowing user space to request channels to be exported would be to list them in devicetree - but this is not a property of the hardware.
>
> > From my experience, the only method supporting this kind of userspace-
> > kernelspace interaction is that of sockets, where the bind() syscall
> > semantic can be translated into the creation of new rpmsg endpoint
> > which subsequent recvfrom()s can perform the desired behaviour on,
> > that is receiving traffic from a given port while knowing the source address of each data unit.
> >
> > Do you have other userspace/kernelspace interaction models to be
> > considered ? Maybe, I am not aware of other possibilities.
> >
>
> There are two that comes to mind, either using a character device per rpmsg instance that provides an ioctl interface for exporting/unexporting endpoints or through some sysfs interface.
>
> Per the design of rpmsg_proto we want to be able to specify both src and dst address, and from Qualcomm SMD we want to be able to control this with channel names. So I think the ioctl based approach would be easier.
>
> >
> > > In the Qualcomm case mentioned above, you only have defined
> > > point-to-point channels, so spawning a char device for each one is
> > > rather natural. But the N:1 case (a generic service in the virtio
> > > rpmsg
> > > case) requires a recvfrom()/sendto() like interface.
> >
> > I agree, the problem of using char devices is that they model only the
> > situation where there is a rpmsg channel underneath, that is a
> > communication link already installed between two ends.
> >
>
> So does rpmsg_proto, it only exposes communication channels after the rpmsg core has probed upon finding a "rpmsg-proto" channel.
>
> > In my point of view the rpmsg channel abstraction matches quite well
> > with char devices but there are problems with the particular mechanism
> > rpmsg uses to match devices with drivers.
> > For instance, the char driver I have tested these days spawns a new
> > /dev/rpmsgX every time the rpmsg bus sees a channel id matching the
> > one specified by the driver. If there are two rprocs that create a new
> > channel with the same name (eg "rpmsg-char-dev-channel") and the
> > driver matches such channel name, then two new char devices will be
> > created,
> > /dev/rpmsg0 and /dev/rpmsg1. How do I know which char device is the
> > target I want to communcate with? There is no control over the minor
> > number assignment and it depends on the order of probe() calls the
> > rpmsg bus performs.
> >
>
> We can use udev rules to rename the rpmsg devices based on the channel names, but currently we don't have a convenient name for the parent - so we probably need to look into that some more.
>
> So I think this is okay, from the kernels pov.
>
> > The first solution that comes to my mind is to use the information
> > that rpmsg and remoteproc subsystems expose to sysfs. This way a user
> > space program can understand the correspondence between char devices
> > and actual rpmsg channels and rprocs.
> > For this solution the code from texas is already working, given the
> > fact it is up to the userspace program to retrieve additional informations.
> >
>
> I think this is a good way forward, perhaps we need to provide additional attributes so that it's possible to build a user friendly dev name.
>
> > Instead, if we desire more control inside the kernel we should
> > consider a different creation strategy, such as a /dev/rpmsgXY scheme.
> > The X standing for the rproc index and the Y standing for a particular
> > digit representing different channels. For instance, if we have two
> > remotes each with a single channel in the system we could name the
> > char devices as /dev/rpmsg00 and /dev/rpmsg10.
> >
>
> I would prefer that we find a solution to exposing the parent name in a way that it's possible to write a udev rule to get /dev/rpmsg/<parent-label>/<channel-name>
>
> Perhaps if we just add support for an optional "label" property for controlling the remoteproc's dev->name we would have what we need there
> - and solve the annoyance of the probe order dependant naming of the remoteprocs.
>
> >
> > To sum up, I would say the char approach is more suitable for static
> > configurations and is oriented to represent a rpmsg channel while the
> > socket approach is more dynamic, given the fact it permits generic
> > services to be deployed and it is a little bit detached from the rpmsg
> > channel concept.
>
> I was expecting to find a socket based implementation that allows for unbound services to be implemented in userspace, but was happy to see that neither of the TI solutions I looked at supported this today.
>
> Further more, the Qualcomm SMD implementation doesn't support offchannel communication, so this simplification suits that as well.
>
>
> Not supporting this simplifies the problem and we should be able to implement this without tangling it into the rpmsg implementation too much.
>
> > In my opinion, we should consider both since socket can give more
> > powerful communication paradigms while char devices represent a
> > point-to-point link and are more straightforward to use.
> >
>
> I agree with the view that sockets would give a more powerful model, but unless we _really_ need that I would prefer if we don't go there.
> Most use-cases I've seen does not require support for implementing unbound services and the ability to do so in-kernel is still there.
>
>
>
> So my suggestion is that we upon probing a rpmsg device create a rpmsg_ctrl%d character device, which has an ioctl interface for exporting and unexporting endpoints. We can be a little bit optimistic and have the export create a rpmsg%d device, with the associated rpmsg_channel_info.
>
> Then upon getting the open() callback we issue rpmsg_create_ept() and
> release() calls rpmsg_destroy_ept().
>
> For rpmsg backends that support advertising channel state this would mean that once the char dev is open we have an open rpmsg channel.
>
>
> To be able to use the rpmsg_create_ept() interface we need to have a rpmsg_device handle for the backend-specific rpmsg instance, so I think the first step needs to be initiated by each rpmsg backend by creating a "dummy" rpmsg_device and triggering the registration above.
>
> Regards,
> Bjorn
> --
> To unsubscribe from this list: send the line "unsubscribe linux-remoteproc" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: Type: text/html, Size: 14290 bytes --]

Re: rpmsg: exposing the bus to userspace

[Bjorn Andersson]

On Thu 29 Sep 11:36 PDT 2016, Marek Novak wrote:

>    Hello Bjorn and the community,
> 

Please don't top-post.

>    I also support the idea of making the export of RPMsg endpoints to the
>    user-space part of the virtio_rpmsg_bus kernel module.
> 
>    As you suggest, it would allow to skip the channel concept, which is not
>    really necessary for the communication. (RPMsg-Lite implementation is an
>    evidence of this idea)
> 

Correct, we need the endpoints for each pipe exposed.

The purpose of the rpmsg device is to capture the life cycle of the
primary endpoint, in this case we want the control mechanism to follow
the life cycle of the virtio-rpmsg device.

Regards,
Bjorn

>    Thanks & Regards,
>    Marek
> 
>      ----------------------------------------------------------------------
> 
>    From: Bjorn Andersson <bjorn.andersson@linaro.org>
>    Sent: Sep 29, 2016 7:35 PM
>    To: Marek Novak
>    Cc: Matteo Sartori; takahiro.akashi@linaro.org;
>    linux-remoteproc@vger.kernel.org; Alessio Paccoia; Michele Rodolfi;
>    Claudio Salati; Michal Princ; Petr Lukas
>    Subject: Re: rpmsg: exposing the bus to userspace
>    On Tue 20 Sep 21:43 PDT 2016, Marek Novak wrote:
> 
>    > Hello Bjorn and the community,
>    >
> 
>    Hi Marek
> 
>    > Bjorn mentioned a sysfs-based exporting mechanism.
>    > I have written a kernel module, which allows exporting RPMsg endpoints
>    > to the user space.  If you want, please look at the code here:
>    >
>    https://github.com/EmbeddedRPC/erpc-imx-demos/blob/master/MPU/setup_yocto/rpmsg_sysfs_interface.patch
>    >
> 
>    This is in line with my ideas, allowing user space to open arbitrary
>    channels.
> 
>    I think we can afford replacing the originating rpmsg device with a
>    special rpmsg_device created straight from the individual backends - so
>    that we don't have to rely on a specific channel to be exposed to
>    "enable" this feature.
> 
>    > I am preparing an example/demonstration repository on Github showing its
>    use-case here:
>    > https://github.com/EmbeddedRPC/erpc-imx-demos
>    >
> 
>    Thanks,
>    Bjorn
> 
>    >
>    > Regards,
>    > Marek NOVAK
>    > NXP Semiconductors
>    > Brno University of Technology
>    >
>    > -----Original Message-----
>    > From: linux-remoteproc-owner@vger.kernel.org
>    [mailto:linux-remoteproc-owner@vger.kernel.org] On Behalf Of Bjorn
>    Andersson
>    > Sent: Wednesday, September 21, 2016 6:09 AM
>    > To: Matteo Sartori <matteo.sartori@t3lab.it>
>    > Cc: takahiro.akashi@linaro.org; linux-remoteproc@vger.kernel.org;
>    Alessio Paccoia <alessio.paccoia@t3lab.it>; Michele Rodolfi
>    <michele.rodolfi@t3lab.it>; Claudio Salati <claudio.salati@t3lab.it>
>    > Subject: Re: rpmsg: exposing the bus to userspace
>    >
>    > On Thu 25 Aug 03:31 PDT 2016, Matteo Sartori wrote:
>    >
>    > > Hi Bjorn,
>    > > apologies for the latency.
>    > >
>    >
>    > Dito.
>    >
>    > > On 2016-08-10 22:03 GMT+02:00, Bjorn Andersson wrote:
>    > > > Going with the AF_RPMSG approach seems (at first) to map quite
>    > > > nicely with the src/dst port model. It would e.g. allow us to use
>    > > > recvfrom() to implement services for multiple clients.
>    > > >
>    > > > But the src/dst are only unique for each given virtproc, so we need
>    > > > to make up a mechanism for selecting among a set of rpmsg instances
>    > > > if more than one remote exists. Further more I believe that the
>    > > > AF_RPMSG concept would have to be integrated into the core of rpmsg
>    > > > and we would have two different concepts intermixed in the
>    > > > implementation; the in-kernel model with life cycle management and
>    > > > the user-space variant - and we need to figure out how to make these
>    not step on each others toes.
>    > >
>    > > Can you please elaborate a little more on the intermixed
>    implementations?
>    >
>    > After studying rpmsg_proto in detail I take this comment back. After my
>    first read-through I interpreted the code as you expose the entire address
>    space to via the socket interface.
>    >
>    > After reading the implementation more thoroughly I now see that the
>    rpmsg_proto implementation only exposes specific channels (i.e. the ones
>    named "rpmsg-proto") to userspace.
>    >
>    > > For what I understand it is necessary to introduce the AF_RPSMG
>    > > concept inside the rpmsg core because e.g. after a bind() syscall on a
>    > > given port (local rpmsg address) it is important to create a new
>    > > endpoint, and to register a new callback with.
>    >
>    > This can be done with rpmsg_create_ept()
>    >
>    > > But, I don't see why there should be a user-space variant of the
>    > > AF_RPMSG concept.
>    > >
>    > >
>    > > > So I'm preferring a model where we can have a rpmsg-client that can
>    > > > be attached to an endpoint and expose this explicitly to userspace.
>    > >
>    > > Do you refer to the possibility that a userspace process can create
>    > > and attach to a new endpoint?
>    >
>    > Yes
>    >
>    > On the Qualcomm platforms we have a varying number of channels per
>    remoteproc that we would like to be able to access from user space.
>    >
>    > The alternative to allowing user space to request channels to be
>    exported would be to list them in devicetree - but this is not a property
>    of the hardware.
>    >
>    > > From my experience, the only method supporting this kind of userspace-
>    > > kernelspace interaction is that of sockets, where the bind() syscall
>    > > semantic can be translated into the creation of new rpmsg endpoint
>    > > which subsequent recvfrom()s can perform the desired behaviour on,
>    > > that is receiving traffic from a given port while knowing the source
>    address of each data unit.
>    > >
>    > > Do you have other userspace/kernelspace interaction models to be
>    > > considered ? Maybe, I am not aware of other possibilities.
>    > >
>    >
>    > There are two that comes to mind, either using a character device per
>    rpmsg instance that provides an ioctl interface for exporting/unexporting
>    endpoints or through some sysfs interface.
>    >
>    > Per the design of rpmsg_proto we want to be able to specify both src and
>    dst address, and from Qualcomm SMD we want to be able to control this with
>    channel names. So I think the ioctl based approach would be easier.
>    >
>    > >
>    > > > In the Qualcomm case mentioned above, you only have defined
>    > > > point-to-point channels, so spawning a char device for each one is
>    > > > rather natural. But the N:1 case (a generic service in the virtio
>    > > > rpmsg
>    > > > case) requires a recvfrom()/sendto() like interface.
>    > >
>    > > I agree, the problem of using char devices is that they model only the
>    > > situation where there is a rpmsg channel underneath, that is a
>    > > communication link already installed between two ends.
>    > >
>    >
>    > So does rpmsg_proto, it only exposes communication channels after the
>    rpmsg core has probed upon finding a "rpmsg-proto" channel.
>    >
>    > > In my point of view the rpmsg channel abstraction matches quite well
>    > > with char devices but there are problems with the particular mechanism
>    > > rpmsg uses to match devices with drivers.
>    > > For instance, the char driver I have tested these days spawns a new
>    > > /dev/rpmsgX every time the rpmsg bus sees a channel id matching the
>    > > one specified by the driver. If there are two rprocs that create a new
>    > > channel with the same name (eg "rpmsg-char-dev-channel") and the
>    > > driver matches such channel name, then two new char devices will be
>    > > created,
>    > > /dev/rpmsg0 and /dev/rpmsg1. How do I know which char device is the
>    > > target I want to communcate with? There is no control over the minor
>    > > number assignment and it depends on the order of probe() calls the
>    > > rpmsg bus performs.
>    > >
>    >
>    > We can use udev rules to rename the rpmsg devices based on the channel
>    names, but currently we don't have a convenient name for the parent - so
>    we probably need to look into that some more.
>    >
>    > So I think this is okay, from the kernels pov.
>    >
>    > > The first solution that comes to my mind is to use the information
>    > > that rpmsg and remoteproc subsystems expose to sysfs. This way a user
>    > > space program can understand the correspondence between char devices
>    > > and actual rpmsg channels and rprocs.
>    > > For this solution the code from texas is already working, given the
>    > > fact it is up to the userspace program to retrieve additional
>    informations.
>    > >
>    >
>    > I think this is a good way forward, perhaps we need to provide
>    additional attributes so that it's possible to build a user friendly dev
>    name.
>    >
>    > > Instead, if we desire more control inside the kernel we should
>    > > consider a different creation strategy, such as a /dev/rpmsgXY scheme.
>    > > The X standing for the rproc index and the Y standing for a particular
>    > > digit representing different channels. For instance, if we have two
>    > > remotes each with a single channel in the system we could name the
>    > > char devices as /dev/rpmsg00 and /dev/rpmsg10.
>    > >
>    >
>    > I would prefer that we find a solution to exposing the parent name in a
>    way that it's possible to write a udev rule to get
>    /dev/rpmsg/<parent-label>/<channel-name>
>    >
>    > Perhaps if we just add support for an optional "label" property for
>    controlling the remoteproc's dev->name we would have what we need there
>    > - and solve the annoyance of the probe order dependant naming of the
>    remoteprocs.
>    >
>    > >
>    > > To sum up, I would say the char approach is more suitable for static
>    > > configurations and is oriented to represent a rpmsg channel while the
>    > > socket approach is more dynamic, given the fact it permits generic
>    > > services to be deployed and it is a little bit detached from the rpmsg
>    > > channel concept.
>    >
>    > I was expecting to find a socket based implementation that allows for
>    unbound services to be implemented in userspace, but was happy to see that
>    neither of the TI solutions I looked at supported this today.
>    >
>    > Further more, the Qualcomm SMD implementation doesn't support offchannel
>    communication, so this simplification suits that as well.
>    >
>    >
>    > Not supporting this simplifies the problem and we should be able to
>    implement this without tangling it into the rpmsg implementation too much.
>    >
>    > > In my opinion, we should consider both since socket can give more
>    > > powerful communication paradigms while char devices represent a
>    > > point-to-point link and are more straightforward to use.
>    > >
>    >
>    > I agree with the view that sockets would give a more powerful model, but
>    unless we _really_ need that I would prefer if we don't go there.
>    > Most use-cases I've seen does not require support for implementing
>    unbound services and the ability to do so in-kernel is still there.
>    >
>    >
>    >
>    > So my suggestion is that we upon probing a rpmsg device create a
>    rpmsg_ctrl%d character device, which has an ioctl interface for exporting
>    and unexporting endpoints. We can be a little bit optimistic and have the
>    export create a rpmsg%d device, with the associated rpmsg_channel_info.
>    >
>    > Then upon getting the open() callback we issue rpmsg_create_ept() and
>    > release() calls rpmsg_destroy_ept().
>    >
>    > For rpmsg backends that support advertising channel state this would
>    mean that once the char dev is open we have an open rpmsg channel.
>    >
>    >
>    > To be able to use the rpmsg_create_ept() interface we need to have a
>    rpmsg_device handle for the backend-specific rpmsg instance, so I think
>    the first step needs to be initiated by each rpmsg backend by creating a
>    "dummy" rpmsg_device and triggering the registration above.
>    >
>    > Regards,
>    > Bjorn
>    > --
>    > To unsubscribe from this list: send the line "unsubscribe
>    linux-remoteproc" in the body of a message to majordomo@vger.kernel.org
>    More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: rpmsg: exposing the bus to userspace

[Marek Novak]

-----Original Message-----
From: Bjorn Andersson [mailto:bjorn.andersson@linaro.org] 
Sent: Thursday, September 29, 2016 10:08 PM
To: Marek Novak <marek.novak@nxp.com>
Cc: Matteo Sartori <matteo.sartori@t3lab.it>; takahiro.akashi@linaro.org; linux-remoteproc@vger.kernel.org; Alessio Paccoia <alessio.paccoia@t3lab.it>; Michele Rodolfi <michele.rodolfi@t3lab.it>; Claudio Salati <claudio.salati@t3lab.it>; Michal Princ <michal.princ@nxp.com>; Petr Lukas <petr.lukas@nxp.com>
Subject: Re: rpmsg: exposing the bus to userspace

On Thu 29 Sep 11:36 PDT 2016, Marek Novak wrote:

>    Hello Bjorn and the community,
> 

Please don't top-post.

>    I also support the idea of making the export of RPMsg endpoints to the
>    user-space part of the virtio_rpmsg_bus kernel module.
> 
>    As you suggest, it would allow to skip the channel concept, which is not
>    really necessary for the communication. (RPMsg-Lite implementation is an
>    evidence of this idea)
> 

Correct, we need the endpoints for each pipe exposed.

The purpose of the rpmsg device is to capture the life cycle of the primary endpoint, in this case we want the control mechanism to follow the life cycle of the virtio-rpmsg device.

Regards,
Bjorn

>    Thanks & Regards,
>    Marek
> 
>      
> ----------------------------------------------------------------------
> 
>    From: Bjorn Andersson <bjorn.andersson@linaro.org>
>    Sent: Sep 29, 2016 7:35 PM
>    To: Marek Novak
>    Cc: Matteo Sartori; takahiro.akashi@linaro.org;
>    linux-remoteproc@vger.kernel.org; Alessio Paccoia; Michele Rodolfi;
>    Claudio Salati; Michal Princ; Petr Lukas
>    Subject: Re: rpmsg: exposing the bus to userspace
>    On Tue 20 Sep 21:43 PDT 2016, Marek Novak wrote:
> 
>    > Hello Bjorn and the community,
>    >
> 
>    Hi Marek
> 
>    > Bjorn mentioned a sysfs-based exporting mechanism.
>    > I have written a kernel module, which allows exporting RPMsg endpoints
>    > to the user space.  If you want, please look at the code here:
>    >
>    https://github.com/EmbeddedRPC/erpc-imx-demos/blob/master/MPU/setup_yocto/rpmsg_sysfs_interface.patch
>    >
> 
>    This is in line with my ideas, allowing user space to open arbitrary
>    channels.
> 
>    I think we can afford replacing the originating rpmsg device with a
>    special rpmsg_device created straight from the individual backends - so
>    that we don't have to rely on a specific channel to be exposed to
>    "enable" this feature.
> 
>    > I am preparing an example/demonstration repository on Github showing its
>    use-case here:
>    > https://github.com/EmbeddedRPC/erpc-imx-demos
>    >
> 
>    Thanks,
>    Bjorn
> 
>    >
>    > Regards,
>    > Marek NOVAK
>    > NXP Semiconductors
>    > Brno University of Technology
>    >
>    > -----Original Message-----
>    > From: linux-remoteproc-owner@vger.kernel.org
>    [mailto:linux-remoteproc-owner@vger.kernel.org] On Behalf Of Bjorn
>    Andersson
>    > Sent: Wednesday, September 21, 2016 6:09 AM
>    > To: Matteo Sartori <matteo.sartori@t3lab.it>
>    > Cc: takahiro.akashi@linaro.org; linux-remoteproc@vger.kernel.org;
>    Alessio Paccoia <alessio.paccoia@t3lab.it>; Michele Rodolfi
>    <michele.rodolfi@t3lab.it>; Claudio Salati <claudio.salati@t3lab.it>
>    > Subject: Re: rpmsg: exposing the bus to userspace
>    >
>    > On Thu 25 Aug 03:31 PDT 2016, Matteo Sartori wrote:
>    >
>    > > Hi Bjorn,
>    > > apologies for the latency.
>    > >
>    >
>    > Dito.
>    >
>    > > On 2016-08-10 22:03 GMT+02:00, Bjorn Andersson wrote:
>    > > > Going with the AF_RPMSG approach seems (at first) to map quite
>    > > > nicely with the src/dst port model. It would e.g. allow us to use
>    > > > recvfrom() to implement services for multiple clients.
>    > > >
>    > > > But the src/dst are only unique for each given virtproc, so we need
>    > > > to make up a mechanism for selecting among a set of rpmsg instances
>    > > > if more than one remote exists. Further more I believe that the
>    > > > AF_RPMSG concept would have to be integrated into the core of rpmsg
>    > > > and we would have two different concepts intermixed in the
>    > > > implementation; the in-kernel model with life cycle management and
>    > > > the user-space variant - and we need to figure out how to make these
>    not step on each others toes.
>    > >
>    > > Can you please elaborate a little more on the intermixed
>    implementations?
>    >
>    > After studying rpmsg_proto in detail I take this comment back. After my
>    first read-through I interpreted the code as you expose the entire address
>    space to via the socket interface.
>    >
>    > After reading the implementation more thoroughly I now see that the
>    rpmsg_proto implementation only exposes specific channels (i.e. the ones
>    named "rpmsg-proto") to userspace.
>    >
>    > > For what I understand it is necessary to introduce the AF_RPSMG
>    > > concept inside the rpmsg core because e.g. after a bind() syscall on a
>    > > given port (local rpmsg address) it is important to create a new
>    > > endpoint, and to register a new callback with.
>    >
>    > This can be done with rpmsg_create_ept()
>    >
>    > > But, I don't see why there should be a user-space variant of the
>    > > AF_RPMSG concept.
>    > >
>    > >
>    > > > So I'm preferring a model where we can have a rpmsg-client that can
>    > > > be attached to an endpoint and expose this explicitly to userspace.
>    > >
>    > > Do you refer to the possibility that a userspace process can create
>    > > and attach to a new endpoint?
>    >
>    > Yes
>    >
>    > On the Qualcomm platforms we have a varying number of channels per
>    remoteproc that we would like to be able to access from user space.
>    >
>    > The alternative to allowing user space to request channels to be
>    exported would be to list them in devicetree - but this is not a property
>    of the hardware.
>    >
>    > > From my experience, the only method supporting this kind of userspace-
>    > > kernelspace interaction is that of sockets, where the bind() syscall
>    > > semantic can be translated into the creation of new rpmsg endpoint
>    > > which subsequent recvfrom()s can perform the desired behaviour on,
>    > > that is receiving traffic from a given port while knowing the source
>    address of each data unit.
>    > >
>    > > Do you have other userspace/kernelspace interaction models to be
>    > > considered ? Maybe, I am not aware of other possibilities.
>    > >
>    >
>    > There are two that comes to mind, either using a character device per
>    rpmsg instance that provides an ioctl interface for exporting/unexporting
>    endpoints or through some sysfs interface.
>    >
>    > Per the design of rpmsg_proto we want to be able to specify both src and
>    dst address, and from Qualcomm SMD we want to be able to control this with
>    channel names. So I think the ioctl based approach would be easier.
>    >
>    > >
>    > > > In the Qualcomm case mentioned above, you only have defined
>    > > > point-to-point channels, so spawning a char device for each one is
>    > > > rather natural. But the N:1 case (a generic service in the virtio
>    > > > rpmsg
>    > > > case) requires a recvfrom()/sendto() like interface.
>    > >
>    > > I agree, the problem of using char devices is that they model only the
>    > > situation where there is a rpmsg channel underneath, that is a
>    > > communication link already installed between two ends.
>    > >
>    >
>    > So does rpmsg_proto, it only exposes communication channels after the
>    rpmsg core has probed upon finding a "rpmsg-proto" channel.
>    >
>    > > In my point of view the rpmsg channel abstraction matches quite well
>    > > with char devices but there are problems with the particular mechanism
>    > > rpmsg uses to match devices with drivers.
>    > > For instance, the char driver I have tested these days spawns a new
>    > > /dev/rpmsgX every time the rpmsg bus sees a channel id matching the
>    > > one specified by the driver. If there are two rprocs that create a new
>    > > channel with the same name (eg "rpmsg-char-dev-channel") and the
>    > > driver matches such channel name, then two new char devices will be
>    > > created,
>    > > /dev/rpmsg0 and /dev/rpmsg1. How do I know which char device is the
>    > > target I want to communcate with? There is no control over the minor
>    > > number assignment and it depends on the order of probe() calls the
>    > > rpmsg bus performs.
>    > >
>    >
>    > We can use udev rules to rename the rpmsg devices based on the channel
>    names, but currently we don't have a convenient name for the parent - so
>    we probably need to look into that some more.
>    >
>    > So I think this is okay, from the kernels pov.
>    >
>    > > The first solution that comes to my mind is to use the information
>    > > that rpmsg and remoteproc subsystems expose to sysfs. This way a user
>    > > space program can understand the correspondence between char devices
>    > > and actual rpmsg channels and rprocs.
>    > > For this solution the code from texas is already working, given the
>    > > fact it is up to the userspace program to retrieve additional
>    informations.
>    > >
>    >
>    > I think this is a good way forward, perhaps we need to provide
>    additional attributes so that it's possible to build a user friendly dev
>    name.
>    >
>    > > Instead, if we desire more control inside the kernel we should
>    > > consider a different creation strategy, such as a /dev/rpmsgXY scheme.
>    > > The X standing for the rproc index and the Y standing for a particular
>    > > digit representing different channels. For instance, if we have two
>    > > remotes each with a single channel in the system we could name the
>    > > char devices as /dev/rpmsg00 and /dev/rpmsg10.
>    > >
>    >
>    > I would prefer that we find a solution to exposing the parent name in a
>    way that it's possible to write a udev rule to get
>    /dev/rpmsg/<parent-label>/<channel-name>
>    >
>    > Perhaps if we just add support for an optional "label" property for
>    controlling the remoteproc's dev->name we would have what we need there
>    > - and solve the annoyance of the probe order dependant naming of the
>    remoteprocs.
>    >
>    > >
>    > > To sum up, I would say the char approach is more suitable for static
>    > > configurations and is oriented to represent a rpmsg channel while the
>    > > socket approach is more dynamic, given the fact it permits generic
>    > > services to be deployed and it is a little bit detached from the rpmsg
>    > > channel concept.
>    >
>    > I was expecting to find a socket based implementation that allows for
>    unbound services to be implemented in userspace, but was happy to see that
>    neither of the TI solutions I looked at supported this today.
>    >
>    > Further more, the Qualcomm SMD implementation doesn't support offchannel
>    communication, so this simplification suits that as well.
>    >
>    >
>    > Not supporting this simplifies the problem and we should be able to
>    implement this without tangling it into the rpmsg implementation too much.
>    >
>    > > In my opinion, we should consider both since socket can give more
>    > > powerful communication paradigms while char devices represent a
>    > > point-to-point link and are more straightforward to use.
>    > >
>    >
>    > I agree with the view that sockets would give a more powerful model, but
>    unless we _really_ need that I would prefer if we don't go there.
>    > Most use-cases I've seen does not require support for implementing
>    unbound services and the ability to do so in-kernel is still there.
>    >
>    >
>    >
>    > So my suggestion is that we upon probing a rpmsg device create a
>    rpmsg_ctrl%d character device, which has an ioctl interface for exporting
>    and unexporting endpoints. We can be a little bit optimistic and have the
>    export create a rpmsg%d device, with the associated rpmsg_channel_info.
>    >
>    > Then upon getting the open() callback we issue rpmsg_create_ept() and
>    > release() calls rpmsg_destroy_ept().
>    >
>    > For rpmsg backends that support advertising channel state this would
>    mean that once the char dev is open we have an open rpmsg channel.
>    >
>    >
>    > To be able to use the rpmsg_create_ept() interface we need to have a
>    rpmsg_device handle for the backend-specific rpmsg instance, so I think
>    the first step needs to be initiated by each rpmsg backend by creating a
>    "dummy" rpmsg_device and triggering the registration above.
>    >
>    > Regards,
>    > Bjorn
>    > --
>    > To unsubscribe from this list: send the line "unsubscribe
>    linux-remoteproc" in the body of a message to majordomo@vger.kernel.org
>    More majordomo info at  http://vger.kernel.org/majordomo-info.html

Hi Bjorn,

Sorry for the top posting...

You said that the RPMsg device should follow the life cycle of the actual remote processor.
I can imagine it is possible (on nowadays used like that) to detect the boot-up of the 
remote processor thanks to the name service message being received.
But what if the remote core resets from some reason? The RPMsg device would
still be there and in current implementation we cannot detect the remote core going down.
To detect it, it would require some platform-specific mechanism based either
on polling some register or something interrupt-based. The only generic solution is now
to try to send a message and if no answer is received. We can assume the second core is down.
But personally, I don't support this solution.

I was thinking again about making the sysfs exporting mechanism part of the virtio_rpmsg_bus kernel module.
Aren't there any other technique than sysfs to export endpoints?
Also, if we agree, sysfs is the most standard way to achieve this, exporting an endpoint as a character device might not
be the only possibility. I have noticed somebody exporting endpoints as sockets:
https://git.ti.com/rpmsg/rpmsg/blobs/ace6f9cfeea0ac958b6b0a53059fd0a72635e695/net/rpmsg/rpmsg_proto.c

I can imagine for some purposes it is better to export it as character device with either streaming or message-based 
receiving policy as I did it in here: 
https://github.com/EmbeddedRPC/erpc-imx-demos/blob/master/MPU/setup_yocto/rpmsg_sysfs_interface.patch

But somebody might prefer the socket way. 
Maybe we should provide some technique to specify the way user wants to export it. Either as a character device with
different receiving policies or as a socket. But this would require to take multiple implementations and put them
somehow together. I would prefer making this maybe a separate kernel module, since the code of the base kernel module
would grow considerably...

What do you think?

Regards,
Marek

Re: rpmsg: exposing the bus to userspace

[Bjorn Andersson]

On Tue 04 Oct 01:05 PDT 2016, Marek Novak wrote:

[..]
> On Thu 29 Sep 11:36 PDT 2016, Marek Novak wrote:
> 
> >    Hello Bjorn and the community,
> > 
> 
> Please don't top-post.
> 
> >    I also support the idea of making the export of RPMsg endpoints to the
> >    user-space part of the virtio_rpmsg_bus kernel module.
> > 
> >    As you suggest, it would allow to skip the channel concept, which is not
> >    really necessary for the communication. (RPMsg-Lite implementation is an
> >    evidence of this idea)
> > 
> 
> Correct, we need the endpoints for each pipe exposed.
> 
> The purpose of the rpmsg device is to capture the life cycle of the primary endpoint, in this case we want the control mechanism to follow the life cycle of the virtio-rpmsg device.
> 
> Regards,
> Bjorn
[..]
> Hi Bjorn,
> 
> Sorry for the top posting...
> 
> You said that the RPMsg device should follow the life cycle of the
> actual remote processor.
> I can imagine it is possible (on nowadays used like that) to detect
> the boot-up of the remote processor thanks to the name service message
> being received.
> But what if the remote core resets from some reason? The RPMsg device
> would still be there and in current implementation we cannot detect
> the remote core going down.
> To detect it, it would require some platform-specific mechanism based
> either on polling some register or something interrupt-based. The only
> generic solution is now to try to send a message and if no answer is
> received. We can assume the second core is down.  But personally, I
> don't support this solution.

The virtio RPMSG device is registered and unregistered based on the
underlaying provider, in the case of the parent being remoteproc this is
done when we boot or shut down the remoteproc.

So tapping into rpmsg_probe() and rpmsg_remove() would give you exactly
this behavior.

> 
> I was thinking again about making the sysfs exporting mechanism part
> of the virtio_rpmsg_bus kernel module.
> Aren't there any other technique than sysfs to export endpoints?

As I've looked at this some more and discussed it with others I think it
makes more sense to expose a control-device per virtio rpmsg instance
and then use a ioctl based interface for creating the per endpoint
interfaces.

> Also, if we agree, sysfs is the most standard way to achieve this,
> exporting an endpoint as a character device might not be the only
> possibility. I have noticed somebody exporting endpoints as sockets:
> https://git.ti.com/rpmsg/rpmsg/blobs/ace6f9cfeea0ac958b6b0a53059fd0a72635e695/net/rpmsg/rpmsg_proto.c

I've seen this too. The benefit of using the socket API is that it
becomes more natural to support the recvfrom() and sendto() interfaces,
to be able to implement unbound services in user space.

This does however require us to invent additional mechanisms for
deterministic addressing, as the remoteproc ids are not static and rpmsg
ports/addresses are not unique in the system.

Without this the socket API is just another choice of how to transfer
chunks of data to and from the kernel.

> 
> I can imagine for some purposes it is better to export it as character
> device with either streaming or message-based receiving policy as I
> did it in here:
> https://github.com/EmbeddedRPC/erpc-imx-demos/blob/master/MPU/setup_yocto/rpmsg_sysfs_interface.patch
> 

A clear benefit of using a character based approach is that it would
allow udev to control permissions on a per-endpoint basis, without any
additional logic.

> But somebody might prefer the socket way.

Of course, there's always going to be people who wants the bread sliced
the other way :)

> Maybe we should provide some technique to specify the way user wants
> to export it. Either as a character device with different receiving
> policies or as a socket. But this would require to take multiple
> implementations and put them somehow together. I would prefer making
> this maybe a separate kernel module, since the code of the base kernel
> module would grow considerably...

In my view the char device has some merits that makes it my preferred
solution and I think we should make sure mainline has a single
implementation - I don't want to maintain two "identical" solutions.

> 
> What do you think?
> 

I think we should go for a chardev based approach integrated with the
rpmsg framework allowing user space to create and expose endpoints.

But please do bring any practical issues with such an approach to the
table, so we make sure that we get a good solution integrated.

Regards,
Bjorn

rpmsg: exposing the bus to userspace

[Matteo Sartori]

Hi,

I have been working on remoteproc/rpmsg and FreeRTOS/OpenAMP for the
latest three months and recently, with my colleagues, I have started
reasoning about the best possible way to expose the kernel rpmsg bus
to applications running in user space. The idea is to end up with a
clean mechanism for the user-kernel communication, like sockets or
char devices, in order to build AMP applications on top of it.

In fact, we have been running experiments with some out-of-tree
drivers implementing exactly such interfaces, namely a driver
registering a new address family AF_RPMSG for the socket interface,
from Texas Instruments, and a char device driver representing
different rpmsg channels with /dev/rpmsgX special files, found inside
OpenAMP obsolete directory.

I am quite new to kernel development and in particular to remoteproc
subsystem, so I am asking the following questions because maybe I have
missed some past developments and discussions about this topic. I
apologize if this is the case.

- Have you ever considered the inclusion of these out-of-tree modules
  in the mainline?
- If so, have you planned some work on this, or what are the reasons
  for not including these modules?
- Is there already a preferred way for exposing the rpmsg bus for
  generic services to userspace, or is more suitable the idea to have 
  a different rpmsg driver, therefore a user-kernel communication strategy, 
  for each different use case of the bus?

Anyway, I would be very glad to contribute in the direction of an
appropriate user-kernel interface for accessing the rpmsg bus, and at
the moment I have a working test case for the socket driver as well as
for the char driver. Given the fact I don't see the superiority of one
approach compared to the other, I would like to propose and discuss
with you both of them.

This was kind of an introductory email. I am preparing two patchsets.

Best regards,
Matteo