Taking first steps with EVL

Taking first steps with EVL

[Byron Jacquot]

Hello Everyone,

I'm evaluating some of the newer options for doing real-time audio
processing.  I've been playing with the Xenomai-based ElkOS - some online
discussion led me to EVL.

I'm working with a Rasbperry Pi 4B - I started with a Raspbian Lite card,
and have been building EVL kernels from the v5.12-evl-rebase branch.

I've got a grasp on the basics - I cloned & built the kernel, then ran `evl
test`, discovered missing configuration parameters, added them, and now the
tests all pass.  I've poked at Hectic and Latmus, too.

I also noticed a boot-time message about pipeline capable Pi4 IRQ chips,
and found the following discussion of a patch to fix it, so added that and
eliminated the message.
https://evlproject.org/pipermail/evl/2021-February/000300.html

I'm pulling the basics of application and driver development together
also.  I'm getting to know EVL thread spawning, diagnosing inappropriate
system calls, etc.  I've got a stub of a character device driver with the
OOB calls attached, and my application successfully attached and using
those calls, partitioning the RT and non-RT access.  Much of this was
informed by looking at both sides  (user/driver) of Hectic and Latmus.  So
far, so good!

Now I'm looking to get audio in & out of the system - it looks like there
are two possible paths I could follow, and wanted to see if anyone here
might be able to offer steering and guidance.

The first path would be to port the RTDM driver from ElkOS - unwind the
RTDM parts, and replace them with EVL ones.  I'm reasonably familiar with
that driver.  Is my understanding correct, that the heart of it is really
setting up the DMA IRQ with the OOB flag, and the OOB DMA is functional on
the bcm2835/Pi4?

The other path would be the ALSA patches described below, plus porting work
based on it for the BCM2711 I2S port & my converters.  My application
doesn't need ALSA, but if it's a useful stepping stone, it's not precluded,
either.
https://xenomai.org/pipermail/xenomai/2020-September/043502.html

Does one of these seem like a shorter path to proof of concept than the
other?

Thank you,

Byron Jacquot

Re: Taking first steps with EVL

[Philippe Gerum]

Byron Jacquot via Xenomai <xenomai@xenomai.org> writes:

> Hello Everyone,
>
> I'm evaluating some of the newer options for doing real-time audio
> processing.  I've been playing with the Xenomai-based ElkOS - some online
> discussion led me to EVL.
>
> I'm working with a Rasbperry Pi 4B - I started with a Raspbian Lite card,
> and have been building EVL kernels from the v5.12-evl-rebase branch.
>
> I've got a grasp on the basics - I cloned & built the kernel, then ran `evl
> test`, discovered missing configuration parameters, added them, and now the
> tests all pass.  I've poked at Hectic and Latmus, too.
>
> I also noticed a boot-time message about pipeline capable Pi4 IRQ chips,
> and found the following discussion of a patch to fix it, so added that and
> eliminated the message.
> https://evlproject.org/pipermail/evl/2021-February/000300.html
>

Mmh, it looks like this patch which is present in v5.10.y-dovetail fell
through the cracks when rebasing on top of v5.11. I just merged it back
to the Dovetail tip at v5.13. Thanks for the hint.

> I'm pulling the basics of application and driver development together
> also.  I'm getting to know EVL thread spawning, diagnosing inappropriate
> system calls, etc.  I've got a stub of a character device driver with the
> OOB calls attached, and my application successfully attached and using
> those calls, partitioning the RT and non-RT access.  Much of this was
> informed by looking at both sides  (user/driver) of Hectic and Latmus.  So
> far, so good!
>
> Now I'm looking to get audio in & out of the system - it looks like there
> are two possible paths I could follow, and wanted to see if anyone here
> might be able to offer steering and guidance.
>
> The first path would be to port the RTDM driver from ElkOS - unwind the
> RTDM parts, and replace them with EVL ones.  I'm reasonably familiar with
> that driver.  Is my understanding correct, that the heart of it is really
> setting up the DMA IRQ with the OOB flag, and the OOB DMA is functional on
> the bcm2835/Pi4?
>

Correct. I've been using this DMA backend in out-of-band mode to run the
oob-spi tidbit (libevl/tidbits), illustrating how to run low jitter
transfers over SPI with EVL on the RPI series (zero/2b/3b/4b) [1].

> The other path would be the ALSA patches described below, plus porting work
> based on it for the BCM2711 I2S port & my converters.  My application
> doesn't need ALSA, but if it's a useful stepping stone, it's not precluded,
> either.
> https://xenomai.org/pipermail/xenomai/2020-September/043502.html
>
> Does one of these seem like a shorter path to proof of concept than the
> other?
>

In a nutshell, option #1 (porting your driver from RTDM to the EVL
kernel API) might be the most straightforward route, since both APIs
share many semantics and your code is known to work already.

Option #2 would require to adapt the i2s and codec drivers backing the
PCM operations, like the couple of top commits at [2] do. You would also
need a user space interface to talk to the oob-capable Alsa stack.  For
this, either direct [oob_]ioctl() would do, or I could share some code
marginally extending tinyalsa [3] to enable pcm_readi()/pcm_writei() for
the oob audio path. That route would require to give more thought to the
problem (not necessarily more code though), but would make the full
range of Alsa features available to control your hardware
(e.g. configuration, muxing etc), in addition to having oob capabilities
as well.

If you application will never need Alsa features, and your existing
driver is doing the job today, then #1 may be a no-brainer. If you want
to benefit from the richness of the Alsa API in addition to
high-frequency, low jitter audio transfers at some point, then #2 is
better.

[1] https://evlproject.org/core/oob-drivers/spi/
[2] https://source.denx.de/Xenomai/xenomai4/linux-evl/-/commits/wip/audio/
[3] https://github.com/tinyalsa/tinyalsa

-- 
Philippe.

Re: Taking first steps with EVL

[Byron Jacquot]

Hello Philippe,

Thank you for the response!  I've had a chance to put your advice into a
practical form.

As my ultimate application doesn't need Alsa, I'm moving forward with
porting the RTDM driver.  I've got a crudely hacked version working, and
I'm working on getting some nontrivial application code to stand on top of
it.

A first question I've got: is there a way to tell if my DMA IRQs are IB or
OOB?  /proc/interrupts seems to be counting them regardless of the flag
setting, and the performance appears to be very good either way.

I've seen the recommendations for isolcpus & thread affinity - would you
recommend that for the IRQs, also?

Similarly, I'm watching the numbers in
/sys/devices/virtual/thread/*/stats.  The in-band switch count is constant
after init, but the CPU % utilization number (last number in the row)
doesn't seem to be reflective of the load.  I'm doing trivial work in my
application (catching the return of the OOB ioctl, copying input buffers to
output), and it's showing numbers in the 200 to 300 range.  Does that seem
right?  Looking at the nanoseconds consumed field, it looks like it might
be a count of milliseconds consumed per second?

Thank you,

Byron Jacquot

Re: Taking first steps with EVL

[Philippe Gerum]

Byron Jacquot <bjacquot@cardinalpeak.com> writes:

> Hello Philippe,
>
> Thank you for the response!  I've had a chance to put your advice into a
> practical form.
>
> As my ultimate application doesn't need Alsa, I'm moving forward with
> porting the RTDM driver.  I've got a crudely hacked version working, and
> I'm working on getting some nontrivial application code to stand on top of
> it.
>

Ok.

> A first question I've got: is there a way to tell if my DMA IRQs are IB or
> OOB?  /proc/interrupts seems to be counting them regardless of the flag
> setting, and the performance appears to be very good either way.
>

I pushed a change to display the "oob" label before the IRQ action field
when an interrupt is routed to the out-of-band stage. This is on top of
v5.13, but should apply cleanly to v5.12 as well. [1]

note: the dovetail/evl branches on top of v5.12 do not receive any fix
at the moment: in order to keep the maintenance effort tractable for me,
I'm maintaining the rebase branches for Dovetail and EVL for arm, arm64
and x86_64 on top of the current LTS (v5.10.y ATM) and the latest kernel
release (until superseded) only. Jan is maintaining a merge branch for
v5.10.y, picking fixes from the corresponding rebase branch.

> I've seen the recommendations for isolcpus & thread affinity - would you
> recommend that for the IRQs, also?

If the execution of all real-time threads on this CPU are paced by such
IRQ, and the IRQ handler is fairly simple code (granted, as it should
be) then running both on the same CPU may be the best option, saving
remote wake ups.

If non-related real-time threads are sharing the CPU, then picking the
right placement depends on whether it is acceptable to temporarily
preempt the execution of these threads for handling these IRQs. If not,
then moving the IRQ out of the way, at the expense of mere local wake
ups would make sense. In that case, an inter-processor IRQ would be
involved in resuming a thread sitting on a different CPU.

You can determine the number of IPIs EVL sent for waking up remote
threads as follows, looking at the 'RWA' field:

$ evl ps -s
/root # evl ps -s
CPU   PID   ISW     CTXSW     SYS       RWA       STAT     NAME
  0   212   0       4789399   4789400   0          Wt      timer-responder:207
  0   214   1       2         1         0          W       test-sitter:207

>
> Similarly, I'm watching the numbers in
> /sys/devices/virtual/thread/*/stats.  The in-band switch count is constant
> after init, but the CPU % utilization number (last number in the row)
> doesn't seem to be reflective of the load.  I'm doing trivial work in my
> application (catching the return of the OOB ioctl, copying input buffers to
> output), and it's showing numbers in the 200 to 300 range.

%CPU in the 200-300 range? Houston, we have a problem.

> Does that seem
> right?  Looking at the nanoseconds consumed field, it looks like it might
> be a count of milliseconds consumed per second?
>

What does this command say?

$ evl ps -t


> Thank you,
>
> Byron Jacquot

[1] https://xenomai.org/pipermail/xenomai/2021-July/045938.html

-- 
Philippe.