* Re: Port of a RTAI based code
2025-05-28 16:13 Port of a RTAI based code Emmanuel Pacaud
@ 2025-05-30 7:55 ` Jan Kiszka
2025-05-30 20:51 ` Emmanuel Pacaud
2025-06-02 15:20 ` Philippe Gerum
1 sibling, 1 reply; 6+ messages in thread
From: Jan Kiszka @ 2025-05-30 7:55 UTC (permalink / raw)
To: Emmanuel Pacaud, xenomai
On 28.05.25 18:13, Emmanuel Pacaud wrote:
> Hi,
>
> A bit of context: we are happily running a bunch of control loops for the [Virgo gravitational wave detector](https://en.wikipedia.org/wiki/Virgo_interferometer) since 2008 using a realtime code based on RTAI. This realtime code runs on DELL R7x0 servers, exchanging data through optical fibers using custom PCIe boards. We are able to implement loops up to 50kHz (data reception, packet parsing, computation, data packing and data emission). Each iteration of the loop is triggered by an interruption generated by the PCIe board. Data reception/emission and packet parsing/packing is done in the interrupt handler, the computation being done in one or more user space process called from the interrupt handler. It works fine, but the latest kernel supported by RTAI being 5.4.x, we want to port the code base to Xenomai.
>
> The way things are done currently relies on the fact RTAI schedules the user space processes as soon as we are waking them using `rt_sem_signal()`. If the process runs on the same CPU, `rt_sem_signal()` returns only when the computing task is done.
>
> In pseudocode, the kernel code is:
>
> ```
> irq_handler()
> {
> parse_imput_DMA_buffer() /* Data packets are parsed and stored in a shared memory */
>
> foreach user_space_tasks:
> rt_sem_signal(task_sem)
>
> fill_output_DMA_buffer()
> ask_PCI_board_for_DMA_transfer()
> acknowledge_interrupt()
> }
> ```
>
> In the user space process:
>
> ```
> task_thread()
> {
> while (!stop_task):
> rt_sem_wait(task_sem)
> do_computing()
>
> }
> ```
>
> I'm currently working on the port of this code to xenomai4, trying to minimize the refactoring.
>
> Would it be possible to obtain a similar behaviour using evl ? I understood the user space processes will not be scheduled during the interrupt handler execution. So I suppose I have split the interrupt handler and implement the process waking part in a kernel thread. But what would be the right approach to wake the user space processes from the kernel space ?
>
You will need to write an EVL kernel driver
(https://v4.xenomai.org/core/kernel-api/) that implement some OOB read
or ioctl which your EVL application can call. That driver will then
block the application in the kernel using an in-kernel OOB mechanism -
e.g. https://v4.xenomai.org/core/kernel-api/semaphore/. The driver also
implements an OOB interrupt handler that will signal that evl semaphore
(e.g.) and, thus, unblock the userspace thread.
Jan
--
Siemens AG, Foundational Technologies
Linux Expert Center
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: Port of a RTAI based code
2025-05-28 16:13 Port of a RTAI based code Emmanuel Pacaud
2025-05-30 7:55 ` Jan Kiszka
@ 2025-06-02 15:20 ` Philippe Gerum
2025-06-03 6:58 ` Philippe Gerum
1 sibling, 1 reply; 6+ messages in thread
From: Philippe Gerum @ 2025-06-02 15:20 UTC (permalink / raw)
To: Emmanuel Pacaud; +Cc: xenomai
Emmanuel Pacaud <emmanuel.pacaud@lapp.in2p3.fr> writes:
> [1. text/markdown]
> Hi,
>
> A bit of context: we are happily running a bunch of control loops for the [Virgo gravitational wave detector](https://en.wikipedia.org/wiki/Virgo_interferometer) since 2008 using a realtime code based on RTAI. This realtime code runs on DELL R7x0 servers, exchanging data through optical fibers using custom PCIe boards. We are able to implement loops up to 50kHz (data reception, packet parsing, computation, data packing and data emission). Each iteration of the loop is triggered by an interruption generated by the PCIe board. Data reception/emission and packet parsing/packing is done in the interrupt handler, the computation being done in one or more user space process called from the interrupt handler. It works fine, but the latest kernel supported by RTAI being 5.4.x, we want to port the code base to Xenomai.
>
> The way things are done currently relies on the fact RTAI schedules the user space processes as soon as we are waking them using `rt_sem_signal()`. If the process runs on the same CPU, `rt_sem_signal()` returns only when the computing task is done.
>
> In pseudocode, the kernel code is:
>
> ```
> irq_handler()
> {
> parse_imput_DMA_buffer() /* Data packets are parsed and stored in a shared memory */
>
> foreach user_space_tasks:
> rt_sem_signal(task_sem)
>
> fill_output_DMA_buffer()
> ask_PCI_board_for_DMA_transfer()
> acknowledge_interrupt()
> }
> ```
>
> In the user space process:
>
> ```
> task_thread()
> {
> while (!stop_task):
> rt_sem_wait(task_sem)
> do_computing()
>
> }
> ```
>
> I'm currently working on the port of this code to xenomai4, trying to minimize the refactoring.
>
> Would it be possible to obtain a similar behaviour using evl ?
Definitely not. The ability for a task to preempt an IRQ handler is an
RTAI peculiarity.
> I understood the user space processes will not be scheduled during the interrupt handler execution. So I suppose I have split the interrupt handler and implement the process waking part in a kernel thread. But what would be the right approach to wake the user space processes from the kernel space ?
>
> Emmanuel.
A possible way would be to implement a token chain using an evl wait
queue, see below. Basically, every processing task in userland would
call some out-of-band ioctl() (i.e. "rt" mode for RTAI), passing its
own, unique token/label/index, in order to wait for the next bulk of
data to process after each IRQ. When the kernel observes that all user
tasks have handled the last IRQ, the finalization work can run for the
latter (i.e. DMA xfer).
Some notes about the code below:
- it is fairly defensive, in that we don't trust the hardware for not
sending us an IRQ in the middle of a processing sequence, i.e. until
we have finalized it. Maybe it's overkill, maybe not.
- it has to be defensive against the user task doing silly things, such
as passing an invalid task token. You could also implement some sort
of binding operation (also using some ioctl request), which would
assign a unique identifier the kernel could retrieve later on - in
that case, you would not have to trust the user code, past the binding
call.
- AFAIU your original code, you could use a semaphore instead of a wait
queue, since this code expects the wake up sequence to be perfect
(i.e. Each IRQ wakes up exactly one thread in turn only once, the task
priority seems to guarantee the execution order, tasks never
block/suspend while running their processing round, and no rogue IRQ
is expected).
- proper finalization of each processing sequence depends on every user
task to complete its work, issuing IOC_TASK_WAIT when done. OTOH, the
original code expects the task not to relinquish the CPU until it is
done with the current processing round. In the evl implementation, you
could still use locks or whatever evl syscall that might block the
caller.
- it's happily untested. Hopefully, it's correct.
You may want to check drivers/evl and kernel/evl for examples using the
evl kernel API, along with the doc at [1].
HTH,
[1] https://v4.xenomai.org/core/kernel-api/
--
#include <evl/wait.h>
static struct evl_wait chain_wait = EVL_WAIT_INITIALIZER(chain_wait);
static long chain_map;
/* Say we have four processing tasks, labelled from index #0 to #3 */
#define NR_TASKS 4
irq_handler(...)
{
irqreturn_t ret = IRQ_NONE;
unsigned long flags;
raw_spin_lock_irqsave(&chain_wait.wchan.lock, flags);
if (WARN_ONCE_ONCE(chain_map)) /* Out of sequence? bad.. */
goto out;
chain_map = -1L >> (BITS_PER_LONG - NR_TASKS);
evl_wake_up_head(&chain_wait);
ret = IRQ_HANDLED;
out:
raw_spin_unlock_irqrestore(&chain_wait.wchan.lock, flags);
return ret;
}
long oob_ioctl(..., unsigned in cmd, long arg)
{
long this_task_index;
unsigned long flags;
int ret;
switch (cmd) {
case IOC_TASK_WAIT: /* Or whatever you name it. */
this_task_index = arg;
if (this_task_index >= NR_TASKS)
return -EINVAL;
ret = evl_wait_event(&chain_wait, test_bit(this_task_index, &chain_map));
if (ret)
return ret;
raw_spin_lock_irqsave(&chain_wait.wchan.lock, flags);
__clear_bit(this_task_index, &chain_map);
if (chain_map) {
evl_wake_up_head(&chain_wait);
raw_spin_unlock_irqrestore(&chain_wait.wchan.lock, flags);
evl_schedule();
} else {
raw_spin_unlock_irqrestore(&chain_wait.wchan.lock, flags);
/* Finalize sequence: DMA out + ack */
}
break;
...
}
return 0;
}
user_task(...)
{
long ret;
for (;;) {
ret = oob_ioctl(fd, IOC_TASK_WAIT, this_task_index);
...
}
}
--
Philippe.
^ permalink raw reply [flat|nested] 6+ messages in thread