[Xenomai-help] migration scenario to xenomai

[Xenomai-help] migration scenario to xenomai

[Guenter Ebermann]

Hi,

At the moment I am thinking about a possible migration of a automotive bus
measuring/simulation application and some drivers from eCos to Linux/Xenomai.
The move is primarily motivated by the huge software and device driver support
under linux.

However some device drivers which interfaces our custom devices in FPGA must be
ported to linux or xenomai user-space or xenomai-rtdm.

But I dont want to write device drivers in kernel-space, because this would
involve some serious restructuring of our current drivers and software
structure and a possible performance penalty because of unneeded data copying.
It is also harder to debug a kernel-driver. In eCos there is no
user/kernel space
seperation and our custom device drivers access the registers based in FPGA
directly. The interface of these drivers to the application is not via a device
file but over direct function invocation.

So I thought of moving this part of our communication stack which has hard-real
time requirement (FlexRay, CAN, LIN) into a seperate process which only lives
in the xenomai domain (it will communicate with low-prio linux processes via
non-blocking fifo or such). But this raises two important questions: Can I
access the FPGA registers directly from a xenomai task in the user space?  Or
do I have to write a minimal linux or xenomai-rtdm device driver which maps
FPGA registers from kernel to user space?

-- Guenter

Re: [Xenomai-help] migration scenario to xenomai

[Thomas Lockhart]

> So I thought of moving this part of our communication stack which has hard-real
> time requirement (FlexRay, CAN, LIN) into a seperate process which only lives
> in the xenomai domain (it will communicate with low-prio linux processes via
> non-blocking fifo or such). But this raises two important questions: Can I
> access the FPGA registers directly from a xenomai task in the user space?  Or
> do I have to write a minimal linux or xenomai-rtdm device driver which maps
> FPGA registers from kernel to user space?

You can do direct register access from userland; we do this with several 
device drivers in our system.

We are thinking about trying to refactor a driver to use RTDM due to the 
recent discussions on losing support for userland interrupt handling, 
but for our purposes the current support for interrupts in userland has 
worked fine.

We use pci_scan_bus() and friends for finding the device, and mmap to 
get it into userland memory.

hth

                         - Tom

Re: [Xenomai-help] migration scenario to xenomai

[Guenter Ebermann]

Hi Thomas,

Thank you very much for your help.

Thomas Lockhart wrote:
> We are thinking about trying to refactor a driver to use RTDM due to the
> recent discussions on losing support for userland interrupt handling, but for
> our purposes the current support for interrupts in userland has worked fine.

If I understand you correctly you are using the rt_intr_*() functions from the
Native xenomai API in user space.

> We use pci_scan_bus() and friends for finding the device, and mmap to get it
> into userland memory.

Ok. This means your devices are on the pci bus and you have written a kernel
driver which maps the pci address space of your card to the user space using
mmap. I once did the same thing for a PCI data aquisition host driver under
linux. But, please correct me if I am wrong, such a minimal pci-memory-mapping
driver cant gain advantage of the xenomai rtdm or other xenomai api. Because
the whole mapping stuff will happen only once during module init and the user
space application (which hopefully only stays in the xenomai domain) will take
over after that.

But the whole PCI stuff leads me to another question:
I know the PCI-host infrastructure of the linux kernel - device enumeration and
driver structure. But AFAIK the PCI infrastructure of the linux kernel "only"
deals with the use case where linux is the master/host on the PCI bus.  Our
aquisition device must be a bus master PCI slave/device - this means it is
plugged into a normal PC via PCI-express. Therefor linux on the card is not
allowed to make bus enumeration. We will have to port our own PCI-device-driver
which uses the DMA controller to make PCI-burst-transfers to host memory. Is
there a infrastructure I dont know of, which can help us in this task? E.g.: A
existing general purpose DMA Api which I only have to extend to the DMA
controller (I think it is a kind of Freescale Bestcomm) of our new CPU.

-Guenter

Re: [Xenomai-help] migration scenario to xenomai

[Wolfgang Denk]

Dear Guenter Ebermann,

In message <C7EFC96D-FF5D-4150-A63D-1FFF25412FE7@domain.hid> you wrote:
> 
> I know the PCI-host infrastructure of the linux kernel - device enumeration and
> driver structure. But AFAIK the PCI infrastructure of the linux kernel "only"
> deals with the use case where linux is the master/host on the PCI bus.  Our
> aquisition device must be a bus master PCI slave/device - this means it is
> plugged into a normal PC via PCI-express. Therefor linux on the card is not
> allowed to make bus enumeration. We will have to port our own PCI-device-driver
> which uses the DMA controller to make PCI-burst-transfers to host memory. Is
> there a infrastructure I dont know of, which can help us in this task? E.g.: A
> existing general purpose DMA Api which I only have to extend to the DMA
> controller (I think it is a kind of Freescale Bestcomm) of our new CPU.

You do not want to use a MPC5200 in any new product. Actually, you do
not want to use the BestComm in any product.

As for PCIe support: both U-Boot and Linux support initialization of
hardware with respective capabilities as root complex or as end point.
For example, the AMCC PPC 440SPe is running in some storage controller
applications, where for example two of the PCIe busses are configured
as root complex and the third is used in end point mode.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@domain.hid
"There was no difference between  the  behavior  of  a  god  and  the
operations of pure chance..."   - Thomas Pynchon, _Gravity's Rainbow_

Re: [Xenomai-help] migration scenario to xenomai

[Gilles Chanteperdrix]

Guenter Ebermann wrote:
> Hi,
> 
> At the moment I am thinking about a possible migration of a automotive bus
> measuring/simulation application and some drivers from eCos to Linux/Xenomai.
> The move is primarily motivated by the huge software and device driver support
> under linux.
> 
> However some device drivers which interfaces our custom devices in FPGA must be
> ported to linux or xenomai user-space or xenomai-rtdm.
> 
> But I dont want to write device drivers in kernel-space, because this would
> involve some serious restructuring of our current drivers and software
> structure and a possible performance penalty because of unneeded data copying.
> It is also harder to debug a kernel-driver. In eCos there is no
> user/kernel space
> seperation and our custom device drivers access the registers based in FPGA
> directly. The interface of these drivers to the application is not via a device
> file but over direct function invocation.
> 
> So I thought of moving this part of our communication stack which has hard-real
> time requirement (FlexRay, CAN, LIN) into a seperate process which only lives
> in the xenomai domain (it will communicate with low-prio linux processes via
> non-blocking fifo or such). But this raises two important questions: Can I
> access the FPGA registers directly from a xenomai task in the user space?  Or
> do I have to write a minimal linux or xenomai-rtdm device driver which maps
> FPGA registers from kernel to user space?

You can access hardware registers in user-space by mmaping the memory
mapped ones with /dev/mem, or using iopl/ioperm and in[lwb]/out[lwb] for
the old x86 I/O registers.

However, if this can help for a first order port to Xenomai,
implementing an RTDM driver later is better on the long run. By limiting
the interface between the application and low-level code to a well
defined interface, it will make it possible to use different
applications with the same drivers, or use different hardware with the
same application at will, things that are often hard when the low-level
code is mixed with the application code.

And you could even get the drivers you develop merged into Xenomai, if
you are willing to publish them.

-- 
					    Gilles.

Re: [Xenomai-help] migration scenario to xenomai

[Guenter Ebermann]

Hi Gilles,

Am 07.08.2010 um 09:46 schrieb Gilles Chanteperdrix:
> You can access hardware registers in user-space by mmaping the memory
> mapped ones with /dev/mem, or using iopl/ioperm and in[lwb]/out[lwb] for
> the old x86 I/O registers.

Ok, thank you very much. I will look into using /dev/mem (we dont need x86 I/O
- we use powerpc arch).

> However, if this can help for a first order port to Xenomai,
> implementing an RTDM driver later is better on the long run. By limiting
> the interface between the application and low-level code to a well
> defined interface, it will make it possible to use different
> applications with the same drivers, or use different hardware with the
> same application at will, things that are often hard when the low-level
> code is mixed with the application code.

The xenomai RTDM API is made solely for kernel space, if i understood the docs
correctly?

Yeah, I understand your point with generic interface and device driver
separation. And I really like the way this happens under linux/xenomai. I am
also willing to write new drivers which are usable by a broader public using
these design principles. But the communication stack I am gona use (AUTOSAR
FlexRay stack) does not follow them. AUTOSAR (www.autosar.org) specs defines
the Interfaces and layers of the whole communcation system for cars. It does
not involve kernel/userspace separation, but it also does not mix the low-level
code with application code. Low level code is only implemented in the device
drivers (CAN, LIN, FlexRay), but the interface to the upper level stack is not
generic (as in linux with device drivers) - it is different for each kind of
lower layer device (CAN, LIN, FlexRay) to gain the best performance from each
of them.

So I will try to keep the AUTOSAR part in a high prio process (deadlines to
fill communication buffers in-time must be met) in user space which stays in
the xenomai domain.

For all other things I will try to use the linux design philosophy and RTDM
drivers.

> And you could even get the drivers you develop merged into Xenomai, if
> you are willing to publish them.

I will do my best to publish drivers which are not our core business and which
will be usable by a broader public. E.g. a driver for device-PCI-communication,
or other drivers which are missing yet for our target processor. And of course
bug-fixes or extensions of existing drivers.

We will like to use the rt-socket-can MSCAN driver from xenomai. I already
looked in the source and liked it very much (I maintained, extended and fixed
a MSCAN driver for eCos). On top of it I would like to write a small AUTOSAR
Can Driver API in userspace to link it with the rest of the stack. 

BTW. Is anyone of you using Xenomai on Freescale QorIP P2020?
I have seen it mentioned in the supported hardware area on the xenomai
home page.

-Guenter

Re: [Xenomai-help] migration scenario to xenomai

[Gilles Chanteperdrix]

Guenter Ebermann wrote:
> Hi Gilles,
> 
> Am 07.08.2010 um 09:46 schrieb Gilles Chanteperdrix:
>> You can access hardware registers in user-space by mmaping the memory
>> mapped ones with /dev/mem, or using iopl/ioperm and in[lwb]/out[lwb] for
>> the old x86 I/O registers.
> 
> Ok, thank you very much. I will look into using /dev/mem (we dont need x86 I/O
> - we use powerpc arch).
> 
>> However, if this can help for a first order port to Xenomai,
>> implementing an RTDM driver later is better on the long run. By limiting
>> the interface between the application and low-level code to a well
>> defined interface, it will make it possible to use different
>> applications with the same drivers, or use different hardware with the
>> same application at will, things that are often hard when the low-level
>> code is mixed with the application code.
> 
> The xenomai RTDM API is made solely for kernel space, if i understood the docs
> correctly?
> 
> Yeah, I understand your point with generic interface and device driver
> separation. And I really like the way this happens under linux/xenomai. I am
> also willing to write new drivers which are usable by a broader public using
> these design principles. But the communication stack I am gona use (AUTOSAR
> FlexRay stack) does not follow them. AUTOSAR (www.autosar.org) specs defines
> the Interfaces and layers of the whole communcation system for cars. It does
> not involve kernel/userspace separation, but it also does not mix the low-level
> code with application code. Low level code is only implemented in the device
> drivers (CAN, LIN, FlexRay), but the interface to the upper level stack is not
> generic (as in linux with device drivers) - it is different for each kind of
> lower layer device (CAN, LIN, FlexRay) to gain the best performance from each
> of them.

Well, it looks to me like you can implement the interface between driver
and user-space with standard posix interface (especially ioctl), and
then implement the AUTOSAR interfaces as a user-space library relying on
this posix interface.

-- 
					    Gilles.

Re: [Xenomai-help] migration scenario to xenomai

[Gilles Chanteperdrix]

Guenter Ebermann wrote:
> Hi Gilles,
> 
> Gilles Chanteperdrix wrote:
>> Guenter Ebermann wrote:
>>> Yeah, I understand your point with generic interface and device driver
>>> separation. And I really like the way this happens under linux/xenomai. I am
>>> also willing to write new drivers which are usable by a broader public using
>>> these design principles. But the communication stack I am gona use (AUTOSAR
>>> FlexRay stack) does not follow them. AUTOSAR (www.autosar.org) specs defines
>>> the Interfaces and layers of the whole communcation system for cars. It does
>>> not involve kernel/userspace separation, but it also does not mix the low-level
>>> code with application code. Low level code is only implemented in the device
>>> drivers (CAN, LIN, FlexRay), but the interface to the upper level stack is not
>>> generic (as in linux with device drivers) - it is different for each kind of
>>> lower layer device (CAN, LIN, FlexRay) to gain the best performance from each
>>> of them.
>> Well, it looks to me like you can implement the interface between driver
>> and user-space with standard posix interface (especially ioctl), and
>> then implement the AUTOSAR interfaces as a user-space library relying on
>> this posix interface.
> 
> Yes, this will work for the send direction for LIN and CAN very fine. But
> receive operation happens via function calls (RxIndications) from driver to
> upper layer. Also send direction of FLEXRAY is a call from driver to upper
> layer with a request to fill the buffers.
> 
> Can I make callbacks from xenomai rtdm into a xenomai app in user space?
> Or can I register C-callbacks with ioctrl at driver?

The support to have this is almost in place, we call it user-space
signals, because this is essentially the way posix signals are
implemented, but it applies to any type of user-space callback triggered
by a kernel-space action.

With the current method, the callback would be called only when
returning from a system call, but with the "mayday" support, implemented
in Xenomai 2.5.4, we have a way to have the callbacks being able to
interrupt any piece of code. The support we have currently is only the
core support, we need some window-dressing for the RTDM skin, though.

But all this can be sorted out pretty quickly if need be.

-- 
					    Gilles.

Re: [Xenomai-help] migration scenario to xenomai

[Philippe Gerum]

On Sat, 2010-08-07 at 13:02 +0200, Gilles Chanteperdrix wrote:
> Guenter Ebermann wrote:
> > Hi Gilles,
> > 
> > Gilles Chanteperdrix wrote:
> >> Guenter Ebermann wrote:
> >>> Yeah, I understand your point with generic interface and device driver
> >>> separation. And I really like the way this happens under linux/xenomai. I am
> >>> also willing to write new drivers which are usable by a broader public using
> >>> these design principles. But the communication stack I am gona use (AUTOSAR
> >>> FlexRay stack) does not follow them. AUTOSAR (www.autosar.org) specs defines
> >>> the Interfaces and layers of the whole communcation system for cars. It does
> >>> not involve kernel/userspace separation, but it also does not mix the low-level
> >>> code with application code. Low level code is only implemented in the device
> >>> drivers (CAN, LIN, FlexRay), but the interface to the upper level stack is not
> >>> generic (as in linux with device drivers) - it is different for each kind of
> >>> lower layer device (CAN, LIN, FlexRay) to gain the best performance from each
> >>> of them.
> >> Well, it looks to me like you can implement the interface between driver
> >> and user-space with standard posix interface (especially ioctl), and
> >> then implement the AUTOSAR interfaces as a user-space library relying on
> >> this posix interface.
> > 
> > Yes, this will work for the send direction for LIN and CAN very fine. But
> > receive operation happens via function calls (RxIndications) from driver to
> > upper layer. Also send direction of FLEXRAY is a call from driver to upper
> > layer with a request to fill the buffers.
> > 
> > Can I make callbacks from xenomai rtdm into a xenomai app in user space?
> > Or can I register C-callbacks with ioctrl at driver?
> 
> The support to have this is almost in place, we call it user-space
> signals, because this is essentially the way posix signals are
> implemented, but it applies to any type of user-space callback triggered
> by a kernel-space action.
> 
> With the current method, the callback would be called only when
> returning from a system call, but with the "mayday" support, implemented
> in Xenomai 2.5.4, we have a way to have the callbacks being able to
> interrupt any piece of code. The support we have currently is only the
> core support, we need some window-dressing for the RTDM skin, though.
> 
> But all this can be sorted out pretty quickly if need be.

Until this is available, a straightforward approach would be to create
some server thread of high priority in the receiving application,
waiting for indications on a blocking ioctl() call dequeuing them from
kernel space, then running the userland callback as required.

-- 
Philippe.