Where to start with Kernel config for Atom (64bit) Xenomai4 (evl)

Where to start with Kernel config for Atom (64bit) Xenomai4 (evl)

[Steub, Peter]

Hi all,

I like to get evl running on a quadcore Atom box. I started with a standard ubuntu 20.04 installation and build a kernel from ,v5.15.y-evl-rebase' branch with evl enabled and things that ,evl check -file' told me disabled. 
Good news: the kernel runs and I do have a evl api.
Bad news: latmus tells me a worst jitter of 500µs right away.
Also ,evl test' tells me:
*** stack smashing detected ***: terminated
Aborted (core dumped)
** poll-multiple: BROKEN

I have a Xenomai3 setup on this same machine that has a 30µs jitter. 

On the evlproject.org site there is a list of ports including a x86_64 atom with all the boxes checked. Is there anywhere more information about these ports, are the .config files somewhere to be found, to be used as a starting point? I'm also interested in the Raspberry PI port, but I don't like to try it all out again for myself, a kernel-config or in case of the PI even a binary or a hole Image would be great!
Also I'm kind of lost on documentation, all I find leads back to the evlproject.org main site. Where can I find what to do if say "poll-multible" failed? I have the feeling I am still missing a big junk of documentation! Can anyone please point me at it!

Thank you

Peter Steub

Re: Where to start with Kernel config for Atom (64bit) Xenomai4 (evl)

[Philippe Gerum]

"Steub, Peter via Xenomai" <xenomai@xenomai.org> writes:

> Hi all,
>
> I like to get evl running on a quadcore Atom box. I started with a
> standard ubuntu 20.04 installation

Which will almost never work out of the box on x86. Some options
commonly selected there will likely cause issues with any real-time
infrastructure.

> and build a kernel from ,v5.15.y-evl-rebase' branch with evl enabled and things that ,evl check -file' told me disabled. 

What do you mean? "evl check" is about verifying a configuration, not
giving any runtime status. The output is a set of hints about kernel
options that should appear as indicated in the submitted .config file,
but do not.

> Good news: the kernel runs and I do have a evl api.
> Bad news: latmus tells me a worst jitter of 500µs right away.

You may want to check this [1] first. A common issue causing large
latency peaks is with ACPI_PROCESSOR_IDLE as documented.

> Also ,evl test' tells me:
> *** stack smashing detected ***: terminated
> Aborted (core dumped)
> ** poll-multiple: BROKEN

# gdb /usr/evl/tests/poll-multiple

may help, running it then asking for a backtrace when it breaks. I could
have a look at the backtrace output to figure out what happens, since I
cannot reproduce this on any of my fixtures ATM, so this may depend on a
glibc setup.

>
> I have a Xenomai3 setup on this same machine that has a 30µs jitter. 
>

You should expect the same from EVL/x4, a bit better than x3 under SMP
stress due to the improved locking model.

> On the evlproject.org site there is a list of ports including a x86_64
> atom with all the boxes checked. Is there anywhere more information
> about these ports, are the .config files somewhere to be found, to be
> used as a starting point?

In short, no. But the document at [1] helps in removing known issues
from stock kernel configurations.

> I'm also interested in the Raspberry PI
> port, but I don't like to try it all out again for myself, a
> kernel-config or in case of the PI even a binary or a hole Image would
> be great!

Which PI? All PI kernels can be built using the default mainline config
for the architecture at hand. This is actually the ones I'm using.
e.g. ARCH=arm64 defconfig for PI3/64bit and PI4/64bit, or
multi_v7_defconfig for 32bit (including PI0 and PI2).

> Also I'm kind of lost on documentation, all I find leads back to the
> evlproject.org main site.

Everything related to EVL and xenomai4 is hosted at evlproject.org ATM.

> Where can I find what to do if say "poll-multible" failed? I have the feeling I am still missing a big junk of documentation! Can anyone please point me at it!
>

This means a non-reg test fails, so either the test code itself is
broken, or something it exercises is broken. In this case, #1 is most
likely given the error report. What you can do is to help in
understanding why this breaks: see above regarding the gdb command.

> Thank you
>
> Peter Steub

[1] https://evlproject.org/core/caveat/

-- 
Philippe.

WG: Where to start with Kernel config for Atom (64bit) Xenomai4 (evl)

[Steub, Peter]

> > Hi all,
> >
> > I like to get evl running on a quadcore Atom box. I started with a 
> > standard ubuntu 20.04 installation
> 
> Which will almost never work out of the box on x86. Some options 
> commonly selected there will likely cause issues with any real-time infrastructure.

Can you give me an example, the Xenomai3 setup that I use very successfully, is based on a Ubuntu 16.04 desktop, although it is pretty old already, so I may have forgotten what I did exactly back then to make it work.

> > and build a kernel from ,v5.15.y-evl-rebase' branch with evl enabled 
> > and things that ,evl check -file' told me disabled.
> 
> What do you mean? "evl check" is about verifying a configuration, not 
> giving any runtime status. The output is a set of hints about kernel 
> options that should appear as indicated in the submitted .config file, but do not.

Exactly, I did run "evl check" on the kernel config before I build the kernel. So ACPI_PROCESSOR_IDLE is deactivated as well.

> > Also ,evl test' tells me:
> > *** stack smashing detected ***: terminated Aborted (core dumped)
> > ** poll-multiple: BROKEN
> 
> # gdb /usr/evl/tests/poll-multiple
> 
> may help, running it then asking for a backtrace when it breaks. I 
> could have a look at the backtrace output to figure out what happens, 
> since I cannot reproduce this on any of my fixtures ATM, so this may 
> depend on a glibc setup.

(gdb) run
Starting program: /usr/evl/tests/poll-multiple [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff7d8b700 (LWP 3404)]
[New Thread 0x7ffff7d7a700 (LWP 3412)]
[Thread 0x7ffff7d8b700 (LWP 3404) exited]
*** stack smashing detected ***: terminated [Thread 0x7ffff7d7a700 (LWP 3412) exited]

Thread 1 "poll-multi:3400" received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) backtrace
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007ffff7db1859 in __GI_abort () at abort.c:79
#2  0x00007ffff7e1c29e in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7ffff7f4608f "*** %s ***: terminated\n") at ../sysdeps/posix/libc_fatal.c:155
#3  0x00007ffff7ebeaea in __GI___fortify_fail (msg=msg@entry=0x7ffff7f46077 "stack smashing detected") at fortify_fail.c:26
#4  0x00007ffff7ebeab6 in __stack_chk_fail () at stack_chk_fail.c:24
#5  0x0000555555557941 in main (argc=1, argv=0x7fffffffe558) at ../../home/bm_user/nas/libevl-master/tests/poll-multiple.c:163

> > I'm also interested in the Raspberry PI port, but I don't like to 
> > try it all out again for myself, a kernel-config or in case of the 
> > PI even a binary or a hole Image would be great!
> 
> Which PI? All PI kernels can be built using the default mainline 
> config for the architecture at hand. This is actually the ones I'm using.
> e.g. ARCH=arm64 defconfig for PI3/64bit and PI4/64bit, or 
> multi_v7_defconfig for 32bit (including PI0 and PI2).

Ok I started over with the kernel and now that I have done it, it seems obvious. Maybe someone could put a small list like this on the "Building the core" site so the next one who does not know how to start knows:
# ARCH=x86_64 make defconfig
# make menuconfig
	-> Processor type and features -> EVL real-time core				Enabled
	-> Processor type and features -> CPU core prioroties scheduler support	Disable
	-> Power management -> CPU Frequency scaling 				Disable
	-> Power management -> ACPI support -> Processor 				Disable
	->Kernel hacking -> Tracers 							Disable
	#######Add addition Hardware support######### # make clean # make bindeb-pkg (or just make...)

The result looks very promising, on an isolated core I get worst user values at around 3, on a non isolated core I get like 20. I haven't tested for long and not with different kind of loads but still it looks a lot better than before.  Now I need to port my application from X3 to X4 but those questions deserve a separate email.

Thank you

Peter Steub
> 
> [1] https://evlproject.org/core/caveat/
> 
> --
> Philippe.