Re: Port of a RTAI based code

[Philippe Gerum <rpm@xenomai.org>]

Emmanuel Pacaud <emmanuel.pacaud@lapp.in2p3.fr> writes:

> [1. text/markdown]
> Hi,
>
> A bit of context: we are happily running a bunch of control loops for the [Virgo gravitational wave detector](https://en.wikipedia.org/wiki/Virgo_interferometer) since 2008 using a realtime code based on RTAI. This realtime code runs on DELL R7x0 servers, exchanging data through optical fibers using custom PCIe boards. We are able to implement loops up to 50kHz (data reception, packet parsing, computation, data packing and data emission). Each iteration of the loop is triggered by an interruption generated by the PCIe board. Data reception/emission and packet parsing/packing is done in the interrupt handler, the computation being done in one or more user space process called from the interrupt handler. It works fine, but the latest kernel supported by RTAI being 5.4.x, we want to port the code base to Xenomai.
>
> The way things are done currently relies on the fact RTAI schedules the user space processes as soon as we are waking them using `rt_sem_signal()`. If the process runs on the same CPU, `rt_sem_signal()` returns only when the computing task is done.
>
> In pseudocode, the kernel code is:
>
> ```
> irq_handler() 
> {
>   parse_imput_DMA_buffer() /* Data packets are parsed and stored in a shared memory */
>
>   foreach user_space_tasks:
>     rt_sem_signal(task_sem)
>
>   fill_output_DMA_buffer()
>   ask_PCI_board_for_DMA_transfer()
>   acknowledge_interrupt()
> }
> ```
>
> In the user space process:
>
> ```
> task_thread()
> {
>   while (!stop_task):
>     rt_sem_wait(task_sem)
>     do_computing()
>
> }
> ```
>
> I'm currently working on the port of this code to xenomai4, trying to minimize the refactoring.
>
> Would it be possible to obtain a similar behaviour using evl ?

Definitely not. The ability for a task to preempt an IRQ handler is an
RTAI peculiarity.

> I understood the user space processes will not be scheduled during the interrupt handler execution. So I suppose I have split the interrupt handler and implement the process waking part in a kernel thread. But what would be the right approach to wake the user space processes from the kernel space ?
>
> Emmanuel.

A possible way would be to implement a token chain using an evl wait
queue, see below. Basically, every processing task in userland would
call some out-of-band ioctl() (i.e. "rt" mode for RTAI), passing its
own, unique token/label/index, in order to wait for the next bulk of
data to process after each IRQ. When the kernel observes that all user
tasks have handled the last IRQ, the finalization work can run for the
latter (i.e. DMA xfer).

Some notes about the code below:

- it is fairly defensive, in that we don't trust the hardware for not
  sending us an IRQ in the middle of a processing sequence, i.e. until
  we have finalized it. Maybe it's overkill, maybe not.

- it has to be defensive against the user task doing silly things, such
  as passing an invalid task token. You could also implement some sort
  of binding operation (also using some ioctl request), which would
  assign a unique identifier the kernel could retrieve later on - in
  that case, you would not have to trust the user code, past the binding
  call.

- AFAIU your original code, you could use a semaphore instead of a wait
  queue, since this code expects the wake up sequence to be perfect
  (i.e. Each IRQ wakes up exactly one thread in turn only once, the task
  priority seems to guarantee the execution order, tasks never
  block/suspend while running their processing round, and no rogue IRQ
  is expected).

- proper finalization of each processing sequence depends on every user
  task to complete its work, issuing IOC_TASK_WAIT when done. OTOH, the
  original code expects the task not to relinquish the CPU until it is
  done with the current processing round. In the evl implementation, you
  could still use locks or whatever evl syscall that might block the
  caller.

- it's happily untested. Hopefully, it's correct.

You may want to check drivers/evl and kernel/evl for examples using the
evl kernel API, along with the doc at [1].

HTH,

[1] https://v4.xenomai.org/core/kernel-api/

--

#include <evl/wait.h>

static struct evl_wait chain_wait = EVL_WAIT_INITIALIZER(chain_wait);

static long chain_map;

/* Say we have four processing tasks, labelled from index #0 to #3 */
#define NR_TASKS  4

irq_handler(...)
{
	irqreturn_t ret = IRQ_NONE;
	unsigned long flags;

	raw_spin_lock_irqsave(&chain_wait.wchan.lock, flags);

	if (WARN_ONCE_ONCE(chain_map)) /* Out of sequence? bad.. */
		goto out;

	chain_map = -1L >> (BITS_PER_LONG - NR_TASKS);
	evl_wake_up_head(&chain_wait);
	ret = IRQ_HANDLED;
out:
	raw_spin_unlock_irqrestore(&chain_wait.wchan.lock, flags);

	return ret;
}

long oob_ioctl(..., unsigned in cmd, long arg)
{
	long this_task_index;
	unsigned long flags;
	int ret;

	switch (cmd) {
	case IOC_TASK_WAIT: /* Or whatever you name it. */
		this_task_index = arg;
		if (this_task_index >= NR_TASKS)
			return -EINVAL;
		ret = evl_wait_event(&chain_wait, test_bit(this_task_index, &chain_map));
		if (ret)
			return ret;
		raw_spin_lock_irqsave(&chain_wait.wchan.lock, flags);
		__clear_bit(this_task_index, &chain_map);
		if (chain_map) {
			evl_wake_up_head(&chain_wait);
			raw_spin_unlock_irqrestore(&chain_wait.wchan.lock, flags);
                        evl_schedule();
                } else {
			raw_spin_unlock_irqrestore(&chain_wait.wchan.lock, flags);
			/* Finalize sequence: DMA out + ack */
		}
		break;
	...
	}

	return 0;
}


user_task(...)
{
	long ret;

	for (;;) {
		ret = oob_ioctl(fd, IOC_TASK_WAIT, this_task_index);
		...
	}
}

-- 
Philippe.