xenomai 3.1/ipipe-core-4.19.128-cip28-arm-10.patch

xenomai 3.1/ipipe-core-4.19.128-cip28-arm-10.patch

[Robert Berger]

Hi,

I just took the latest and greatest stuff and built it with some Yocto 
dunfell/3.1.1 SDK for some 32 bit ARM i.mx6.

Xenomai/cobalt v3.1 -- #5714ceede (2020-02-04 16:06:11 +0100)

When I run xeno-test I get this:

./xeno-test
++ echo 0
++ testdir=/usr/local/xenomai/bin
++ /usr/local/xenomai/bin/smokey --run random_alloc_rounds=64 
pattern_check_rounds=64
arith OK
...
posix_clock OK
posix_cond OK
posix_fork OK
[  466.998532] [Xenomai] bad syscall <0x197>
posix_mutex OK
posix_select OK
...

0x197 (407) is not avail in 4.19[1], but in 5.4[2]

[1] 
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/arch/arm/tools/syscall.tbl?h=v4.19.131#n416

[2] 
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/arch/arm/tools/syscall.tbl?h=v5.4.50#n424 


The SDK was built against 5.4 kernel headers (which is the default for 
Yocto dunfell/3.1.1).

If I run a small C app (without xenomai) which calls this:

static void xeno_sleep_ms(unsigned int ms)
{                               /* < 1000 */
         struct timespec ts;

         ts.tv_sec = 0;
         ts.tv_nsec = ms * 1000000;
         clock_nanosleep(CLOCK_MONOTONIC, 0, &ts, NULL);
}

all is good both with 4.19.x and 4.5.kernel.

Can someone tell me tell me please where the bad syscall comes from?

It looks like someone calls clock_nanosleep_time64, but xenomai is 
developed against 4.19.x where this sycall does not exist, so my first 
idea would be that it comes from the glibc which was built against 5.4 
kernel headers. But the xenomai test case triggers it! I suspect some 
sleep_ms(1) in some weird combination with something but I am not sure 
about it.

[  251.080655] [Xenomai] bad syscall <0x197>

407     common  clock_nanosleep_time64          sys_clock_nanosleep

Please point me towards where to search further.

Regards,

Robert

Re: xenomai 3.1/ipipe-core-4.19.128-cip28-arm-10.patch

[Robert Berger]

Hi,

Sorry there was a typo in the previous mail

On 04/07/2020 21:59, Robert Berger wrote:
> 
> all is good both with 4.19.x and 4.5.kernel.
> 

should be 4.19.x and 5.4.x kernels

Regards,

Robert

xenomai 3.1/ipipe-core-4.19.128-cip28-arm-10.patch

[Robert Berger]

Hi,

Now I am a bit puzzled ;)

I rebuilt the rootfs against 4.14.x kernel headers, at least so I think ;)

bitbake -s | grep headers
linux-libc-headers                                   :4.19-r0 

nativesdk-linux-libc-headers                         :4.19-r0

... but still get the same syscall error:

root@multi-v7-ml:~# /usr/bin/smokey --run=16
[  448.996933] [Xenomai] bad syscall <0x197>
posix_mutex OK
root@multi-v7-ml:~#

0x197 is not present in a 4.19 kernel, so the error makes sense, but who 
calls it? It looks like something in the posix_mutex test triggers it.

Regards,

Robert

Re: xenomai 3.1/ipipe-core-4.19.128-cip28-arm-10.patch

[Greg Gallagher]

On Mon., Jul. 6, 2020, 7:29 p.m. Robert Berger via Xenomai <
xenomai@xenomai.org> wrote:

> Hi,
>
> Now I am a bit puzzled ;)
>
> I rebuilt the rootfs against 4.14.x kernel headers, at least so I think ;)
>
> bitbake -s | grep headers
> linux-libc-headers                                   :4.19-r0
>
> nativesdk-linux-libc-headers                         :4.19-r0
>
> ... but still get the same syscall error:
>
> root@multi-v7-ml:~# /usr/bin/smokey --run=16
> [  448.996933] [Xenomai] bad syscall <0x197>
> posix_mutex OK
> root@multi-v7-ml:~#
>
> 0x197 is not present in a 4.19 kernel, so the error makes sense, but who
> calls it? It looks like something in the posix_mutex test triggers it.
>
> Regards,
>
> Robert
>
Hi,
  I'll take a look this week with a build I've been testing the ipipe with.

Greg

>

Re: xenomai 3.1/ipipe-core-4.19.128-cip28-arm-10.patch

[Robert Berger]

Hi,

Please find my comments in-line

On 07/07/2020 02:49, Greg Gallagher wrote:
> 
> Hi,
>    I'll take a look this week with a build I've been testing the ipipe 
> with.
> 
> Greg
> 

Thanks!

I am happy to test whatever you tell me.

What I am trying to figure out is whether it's a xenomai or a Yocto 
problem;)

Regards,

Robert

Re: xenomai 3.1/ipipe-core-4.19.128-cip28-arm-10.patch

[Robert Berger]

Hi Greg,

I digged a bit deeper and it looks like although I tell Yocto/BitBake to 
use 4.19.x kernel headers it auto generates this:

/unix/sysv/linux/arm/arch-syscall.h:#define __NR_clock_nanosleep_time64 407

Will try something more drastic to see what happens.

Regards,

Robert

Re: xenomai 3.1/ipipe-core-4.19.128-cip28-arm-10.patch

[Greg Gallagher]

Hi,

On Tue, Jul 7, 2020 at 12:30 PM Robert Berger <xenomai.list@gmail.com> wrote:
>
> Hi Greg,
>
> I digged a bit deeper and it looks like although I tell Yocto/BitBake to
> use 4.19.x kernel headers it auto generates this:
>
> /unix/sysv/linux/arm/arch-syscall.h:#define __NR_clock_nanosleep_time64 407

I tested on the none cip branch 4.19.128 ipipe-arm (since i had that
on hand) and it passes with no issue.  I'll test again on the latest
cip 4.19.y ipipe-arm branch later tonight but I don't expect to get
the above error.  It looks like this syscall exists in the 5.4 kernel
but not the 4.19 one (as you mentioned before).  I haven't had a
chance to look into this too deeply but from the little bit I've
looked at, it seems that in the newer kernels there's been some
changes around clock_nanosleep. Yocto seems to be expecting that the
kernel you are using is going to be 5.4 and is generating headers
based on that.  So apps think that 407 is a valid system call but our
4.19 kernel doesn't know what system call that is.  I don't have a lot
of yocto experience but you could try moving to an older yocto branch
that supports 4.19 kernels and see what happens.

-Greg

>
> Will try something more drastic to see what happens.
>
> Regards,
>
> Robert

Re: xenomai 3.1/ipipe-core-4.19.128-cip28-arm-10.patch

[Robert Berger]

Hi,

My comments are in-line

On 07/07/2020 20:48, Greg Gallagher wrote:
> Yocto seems to be expecting that the
> kernel you are using is going to be 5.4 and is generating headers
> based on that.  So apps think that 407 is a valid system call but our
> 4.19 kernel doesn't know what system call that is.  I don't have a lot
> of yocto experience but you could try moving to an older yocto branch
> that supports 4.19 kernels and see what happens.

Well ;)

Every OE/YP release comes with some default kernel version and one would 
think the kernel headers of this kernel version are used to build the 
glibc, which kind of is the case, but it's 2 different recipes.

So although I use kernel version 4.19.x to apply the xenomai patch it 
still uses 5.4 kernel headers by default. I knew that and added my own 
4.19 kernel-headers recipe. But it looks like it was not used to build 
glibc, since I see some auto generated file with the 407 syscall.

I am still fighting trying to build a glibc with 4.19 kernel headers 
(which I think I managed).

I'll keep you updated.

> 
> -Greg

Thanks,

Robert

> 
>>
>> Will try something more drastic to see what happens.
>>
>> Regards,
>>
>> Robert

Re: xenomai 3.1/ipipe-core-4.19.128-cip28-arm-10.patch

[Robert Berger]

Hi,

I just had a chat on the Yocto IRC and they claim, that I should not 
rebuild glibc against different kernel headers. That's yet another no no ;)

Which version of glibc do you use?

I have version 2.31.

01:38:11) RP: RobertBerger: one llbc can be run against multiple kernels 
and multiple kernel versions
(01:39:26) rber@freenode: Yep, but I thought it's built against a kernel 
version/some number of syscalls and you can upgrade the kernel 
underneath, but not downgrade
(01:40:26) rber@freenode: If you downgrade the kernel potentially 
syscalls are being called which are not available in the old kernel.
(01:41:27) RP: RobertBerger: that isn't how it works
(01:41:42) rber@freenode: Ah OK ;)
(01:41:50) rber@freenode: So you say it's a runtime thing?
(01:42:46) RP: RobertBerger: the libc adapts to the kernel version its 
running against. That kernel is in the range of "latest linux kernel 
when the libc was released" to OLDEST_KERNEL
(01:43:07) rber@freenode: OK

Regards,

Robert

Re: xenomai 3.1/ipipe-core-4.19.128-cip28-arm-10.patch

[Robert Berger]

It looks like either we hit a glibc issue, or something is not clear to 
me in the xenomai test case.

The glibc has a fallback for the unknown syscall. clock_nanosleep_time64

Is this what the xenomai testcase tries to tell us?

Than that is not an error, but fine;)

Regards,

Robert

Re: xenomai 3.1/ipipe-core-4.19.128-cip28-arm-10.patch

[Greg Gallagher]

HI,

On Tue, Jul 7, 2020 at 7:04 PM Robert Berger <xenomai.list@gmail.com> wrote:
>
> It looks like either we hit a glibc issue, or something is not clear to
> me in the xenomai test case.
>
I'm using glibc 2.25.  The other thing to keep in mind is that the
message isn't coming from the Linux kernel, it's coming from Cobalt.
So it's possible cobalt is having some issues with a newer libc.  I'll
have time tonight to sit down with ftrace and see if I can get a
better idea of what's going on.


> The glibc has a fallback for the unknown syscall. clock_nanosleep_time64
>
> Is this what the xenomai testcase tries to tell us?
>
> Than that is not an error, but fine;)
>
> Regards,
>
> Robert
>
-Greg

Re: xenomai 3.1/ipipe-core-4.19.128-cip28-arm-10.patch

[Robert Berger]

Hi,

My comments are in-line.

On 08/07/2020 15:41, Greg Gallagher wrote:

> I'm using glibc 2.25.  The other thing to keep in mind is that the
> message isn't coming from the Linux kernel, it's coming from Cobalt.
> So it's possible cobalt is having some issues with a newer libc.  I'll
> have time tonight to sit down with ftrace and see if I can get a
> better idea of what's going on.
> 

I traced the issue a bit more down.

The problematic testcase is protect_trylock (posix-mutex.c:1063) since 
it calls __real_usleep()(lib/cobalt/wrappers.c:553); which causes 
[Xenomai] bad syscall <0x197>

So yes, it looks like it's a combination of Cobalt and a new glibc.

I am really surprised that no one else stumbled across this issue, since 
I can't even remember which Yocto Version used 4.19 or older kernel headers.

My hack to use 4.19 kernel headers didn't help, since, as you say, it 
does not seem to be related to this issue.

So I guess the ball is in Xenomai land ;)

Having said that I am happy to test whatever you come up with and I 
guess I could also provide some remote test setup where you can play and 
see the issue.

Regards,

Robert

Re: xenomai 3.1/ipipe-core-4.19.128-cip28-arm-10.patch

[Greg Gallagher]

Hi,

On Wed, Jul 8, 2020 at 5:19 PM Robert Berger <xenomai.list@gmail.com> wrote:
>
> Hi,
>
> My comments are in-line.
>
> On 08/07/2020 15:41, Greg Gallagher wrote:
>
> > I'm using glibc 2.25.  The other thing to keep in mind is that the
> > message isn't coming from the Linux kernel, it's coming from Cobalt.
> > So it's possible cobalt is having some issues with a newer libc.  I'll
> > have time tonight to sit down with ftrace and see if I can get a
> > better idea of what's going on.
> >
>
> I traced the issue a bit more down.
>
> The problematic testcase is protect_trylock (posix-mutex.c:1063) since
> it calls __real_usleep()(lib/cobalt/wrappers.c:553); which causes
> [Xenomai] bad syscall <0x197>
>
> So yes, it looks like it's a combination of Cobalt and a new glibc.
>
> I am really surprised that no one else stumbled across this issue, since
> I can't even remember which Yocto Version used 4.19 or older kernel headers.
>
> My hack to use 4.19 kernel headers didn't help, since, as you say, it
> does not seem to be related to this issue.
>
> So I guess the ball is in Xenomai land ;)
>
> Having said that I am happy to test whatever you come up with and I
> guess I could also provide some remote test setup where you can play and
> see the issue.
>
> Regards,
>
> Robert

I tested the latest ipipe-arm 4.19.y-cip branch using the latest gcc
from Linaro (7.5) and I can't reproduce the error.  I'm assuming it's
because it's still using a 2.25 libc.  To investigate further I'll
have to build a newer tool chain and try the test again.  I'll have
time later this week to continue the investigation.  For now it seems
using libc 2.25 avoids the error.

thanks

Greg

Re: xenomai 3.1/ipipe-core-4.19.128-cip28-arm-10.patch

[Jan Kiszka]

On 08.07.20 01:04, Robert Berger via Xenomai wrote:
> It looks like either we hit a glibc issue, or something is not clear to 
> me in the xenomai test case.
> 
> The glibc has a fallback for the unknown syscall. clock_nanosleep_time64
> 
> Is this what the xenomai testcase tries to tell us?
> 
> Than that is not an error, but fine;)
> 

If glibc is probing the existence of the new syscall by calling it, we 
may silence the warning path. To my current reading, it must have been 
triggered from a Xenomai thread in primary domain - are we sure that 
this is correct behavior? Could you set a breakpoint on glibc call and 
provide a backtrace?

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: xenomai 3.1/ipipe-core-4.19.128-cip28-arm-10.patch

[Robert Berger]

Hi,

My comments are in-line.

On 13/07/2020 20:08, Jan Kiszka wrote:
> On 08.07.20 01:04, Robert Berger via Xenomai wrote:
>> It looks like either we hit a glibc issue, or something is not clear 
>> to me in the xenomai test case.
>>
>> The glibc has a fallback for the unknown syscall. clock_nanosleep_time64
>>
>> Is this what the xenomai testcase tries to tell us?
>>
>> Than that is not an error, but fine;)
>>
> 
> If glibc is probing the existence of the new syscall by calling it, we 
> may silence the warning path. To my current reading, it must have been 
> triggered from a Xenomai thread in primary domain - are we sure that 
> this is correct behavior? Could you set a breakpoint on glibc call and 
> provide a backtrace?

When clock_nanosleep() is called from Linux the glibc first calls the 
clock_nanosleep_time64() syscall (which fails on a 4.19 kernel and 
succeeds on a 5.4 kernel). In case it fails glibc calls the old 
clock_nanosleep() syscall and all is good.

The problem here is that the testcase protect_trylock 
(posix-mutex.c:1063) calls __real_usleep()(lib/cobalt/wrappers.c:553); 
which causes [Xenomai] bad syscall <0x197>.

I guess __real_usleep should call the real POSIX implementation.

__weak
int __real_usleep(useconds_t usec)
{
         return usleep(usec);
}

usleep calls __nanosleep and this seems to cause the bad syscall.

Thread 1 "smokey" hit Breakpoint 3, usleep (useconds=useconds@entry=1) 
at ../sysdeps/posix/usleep.c:26
26        struct timespec ts = { .tv_sec = (long int) (useconds / 1000000),
(gdb) bt
#0  usleep (useconds=useconds@entry=1) at ../sysdeps/posix/usleep.c:26
#1  0xb6f81140 in __real_usleep (usec=usec@entry=1) at wrappers.c:555
#2  0x00418bf8 in protect_trylock () at posix-mutex.c:933
#3  run_posix_mutex (t=<optimized out>, argc=<optimized out>, 
argv=<optimized out>) at posix-mutex.c:1063
#4  run_posix_mutex (t=<optimized out>, argc=<optimized out>, 
argv=<optimized out>) at posix-mutex.c:1027
#5  0x004041c4 in main (argc=1, argv=0x438750) at main.c:34
(gdb) l
21      #include <unistd.h>
22
23      int
24      usleep (useconds_t useconds)
25      {
26        struct timespec ts = { .tv_sec = (long int) (useconds / 1000000),
27                               .tv_nsec = (long int) (useconds % 
1000000) * 1000ul };
28
29        /* Note the usleep() is a cancellation point.  But since we call
30           nanosleep() which itself is a cancellation point we do not have
(gdb) n
32        return __nanosleep (&ts, NULL);
(gdb) n
[ 2168.741990] [Xenomai] bad syscall <0x197>
protect_trylock () at posix-mutex.c:935
935             if (!__T(ret, pthread_mutex_trylock(&mutex)))
(gdb)

Just FYI this code with usleep() seems to work without xenomai:

int my_usleep(unsigned int millisec)
{
         int rc;
         rc = usleep(millisec);
         if (rc < 0) {
                 perror("my_usleep: ");
         }
         return rc;
}

int main(int argc, char *argv[])
{
         argc = argc;
         argv = argv;

         unsigned int millisec = 1;
         int rc;

         rc = my_usleep(millisec);

         exit(rc);
}


> 
> Jan
> 

Regards,

Robert