Re: [OE-core] Toolchain test results

Re: [OE-core] Toolchain test results

[Richard Purdie]

On Tue, 2023-07-18 at 10:14 +0100, Richard Purdie via
lists.openembedded.org wrote:
> qemuarm has ~350 failures
> qemuarm64 has ~350 failures
> qemux86-64 has ~4000 (3900 in glibc)
> qemux86 has ~4000 (3500 in glibc)
> qemuppc has ~600 failures
> qemumips64 has ~5000 failures (all over)
> qemumips has ~1600 failures
> 
> Anuj: Can Intel look into the glibc test failures on x86?

I realised the glibc issues were due to the network being disabled for
the tests and have sent a patch to fix that. That reduces the failures
from ~3900 to ~330. We should really try and reduce that further but it
is a start!

Cheers,

Richard

Re: [OE-core] Toolchain test results

[Mittal, Anuj]

On Thu, 2023-07-20 at 12:26 +0100, Richard Purdie wrote:
> On Tue, 2023-07-18 at 10:14 +0100, Richard Purdie via
> lists.openembedded.org wrote:
> > qemuarm has ~350 failures
> > qemuarm64 has ~350 failures
> > qemux86-64 has ~4000 (3900 in glibc)
> > qemux86 has ~4000 (3500 in glibc)
> > qemuppc has ~600 failures
> > qemumips64 has ~5000 failures (all over)
> > qemumips has ~1600 failures
> > 
> > Anuj: Can Intel look into the glibc test failures on x86?
> 
> I realised the glibc issues were due to the network being disabled
> for
> the tests and have sent a patch to fix that. That reduces the
> failures
> from ~3900 to ~330. We should really try and reduce that further but
> it
> is a start!
> 

A lot of locale/iconv tests seemed to be failing when calling write
with large buffers/files over NFS. Some of others were triggering OOM.

I ran the tests again after making a few changes:

https://autobuilder.yocto.io/pub/non-release/20230726-11/testresults/qemux86-64-tc/

After switching NFS mount to TCP and increasing the memory available to
1024, the number of tests failed came down to 69.

=======================================================================
=======================================
qemux86-64 PTest Result Summary
=======================================================================
=======================================
-----------------------------------------------------------------------
---------------------------------------
Recipe           | Passed       | Failed   | Skipped     | Time(s)
-----------------------------------------------------------------------
---------------------------------------
binutils         | 289          | 0        | 8           | -
binutils-gas     | 1582         | 4        | 1           | -
binutils-ld      | 1622         | 6        | 111         | -
gcc              | 149696       | 28       | 27599       | -
gcc-g++          | 219260       | 30       | 21108       | -
gcc-libatomic    | 27           | 1        | 27          | -
gcc-libgomp      | 3426         | 1        | 1962        | -
gcc-libitm       | 24           | 1        | 24          | -
gcc-libstdc++-v3 | 9695         | 32       | 5459        | -
glibc            | 4984         | 69       | 147         | -
rust             | 15768        | 0        | 554         | -
-----------------------------------------------------------------------
---------------------------------------

Thanks,

Anuj

Re: [OE-core] Toolchain test results

[Khem Raj]

On Tue, Jul 25, 2023 at 11:00 PM Anuj Mittal <anuj.mittal@intel.com> wrote:
>
> On Thu, 2023-07-20 at 12:26 +0100, Richard Purdie wrote:
> > On Tue, 2023-07-18 at 10:14 +0100, Richard Purdie via
> > lists.openembedded.org wrote:
> > > qemuarm has ~350 failures
> > > qemuarm64 has ~350 failures
> > > qemux86-64 has ~4000 (3900 in glibc)
> > > qemux86 has ~4000 (3500 in glibc)
> > > qemuppc has ~600 failures
> > > qemumips64 has ~5000 failures (all over)
> > > qemumips has ~1600 failures
> > >
> > > Anuj: Can Intel look into the glibc test failures on x86?
> >
> > I realised the glibc issues were due to the network being disabled
> > for
> > the tests and have sent a patch to fix that. That reduces the
> > failures
> > from ~3900 to ~330. We should really try and reduce that further but
> > it
> > is a start!
> >
>
> A lot of locale/iconv tests seemed to be failing when calling write
> with large buffers/files over NFS. Some of others were triggering OOM.
>
> I ran the tests again after making a few changes:
>
> https://autobuilder.yocto.io/pub/non-release/20230726-11/testresults/qemux86-64-tc/
>
> After switching NFS mount to TCP and increasing the memory available to
> 1024, the number of tests failed came down to 69.
>

This is a nice, thanks for doing it. I looked quickly at your results
especially glibc part
and it seems some of remaining failures are in nss module. I could not
see detail logs
why those tests were failing but few things to check is if we install

libnss-db and glibc-extra-nss packages into image or not. some of
these tests edit
/etc/nsswitch.conf, so are these tests running in parallel and racing.

> =======================================================================
> =======================================
> qemux86-64 PTest Result Summary
> =======================================================================
> =======================================
> -----------------------------------------------------------------------
> ---------------------------------------
> Recipe           | Passed       | Failed   | Skipped     | Time(s)
> -----------------------------------------------------------------------
> ---------------------------------------
> binutils         | 289          | 0        | 8           | -
> binutils-gas     | 1582         | 4        | 1           | -
> binutils-ld      | 1622         | 6        | 111         | -
> gcc              | 149696       | 28       | 27599       | -
> gcc-g++          | 219260       | 30       | 21108       | -
> gcc-libatomic    | 27           | 1        | 27          | -
> gcc-libgomp      | 3426         | 1        | 1962        | -
> gcc-libitm       | 24           | 1        | 24          | -
> gcc-libstdc++-v3 | 9695         | 32       | 5459        | -
> glibc            | 4984         | 69       | 147         | -
> rust             | 15768        | 0        | 554         | -
> -----------------------------------------------------------------------
> ---------------------------------------
>
> Thanks,
>
> Anuj
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#184858): https://lists.openembedded.org/g/openembedded-core/message/184858
> Mute This Topic: https://lists.openembedded.org/mt/100212267/1997914
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [raj.khem@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [OE-core] Toolchain test results

[Mittal, Anuj]

On Tue, 2023-07-25 at 23:29 -0700, Khem Raj wrote:
> On Tue, Jul 25, 2023 at 11:00 PM Anuj Mittal <anuj.mittal@intel.com>
> wrote:
> > 
> > On Thu, 2023-07-20 at 12:26 +0100, Richard Purdie wrote:
> > > On Tue, 2023-07-18 at 10:14 +0100, Richard Purdie via
> > > lists.openembedded.org wrote:
> > > > qemuarm has ~350 failures
> > > > qemuarm64 has ~350 failures
> > > > qemux86-64 has ~4000 (3900 in glibc)
> > > > qemux86 has ~4000 (3500 in glibc)
> > > > qemuppc has ~600 failures
> > > > qemumips64 has ~5000 failures (all over)
> > > > qemumips has ~1600 failures
> > > > 
> > > > Anuj: Can Intel look into the glibc test failures on x86?
> > > 
> > > I realised the glibc issues were due to the network being
> > > disabled
> > > for
> > > the tests and have sent a patch to fix that. That reduces the
> > > failures
> > > from ~3900 to ~330. We should really try and reduce that further
> > > but
> > > it
> > > is a start!
> > > 
> > 
> > A lot of locale/iconv tests seemed to be failing when calling write
> > with large buffers/files over NFS. Some of others were triggering
> > OOM.
> > 
> > I ran the tests again after making a few changes:
> > 
> > https://autobuilder.yocto.io/pub/non-release/20230726-11/testresults/qemux86-64-tc/
> > 
> > After switching NFS mount to TCP and increasing the memory
> > available to
> > 1024, the number of tests failed came down to 69.
> > 
> 
> This is a nice, thanks for doing it. I looked quickly at your results
> especially glibc part
> and it seems some of remaining failures are in nss module. I could
> not
> see detail logs
> why those tests were failing but few things to check is if we install

All the nss tests are failing here:

https://github.com/bminor/glibc/blob/master/support/test-container.c#L842

Could be a NFS problem as well. I have not looked at it in much detail
yet.

Thanks,

Anuj

Re: [OE-core] Toolchain test results

[Mittal, Anuj]

On Wed, 2023-07-26 at 06:45 +0000, Anuj Mittal wrote:
> On Tue, 2023-07-25 at 23:29 -0700, Khem Raj wrote:
> > On Tue, Jul 25, 2023 at 11:00 PM Anuj Mittal
> > <anuj.mittal@intel.com>
> > wrote:
> > > 
> > > On Thu, 2023-07-20 at 12:26 +0100, Richard Purdie wrote:
> > > > On Tue, 2023-07-18 at 10:14 +0100, Richard Purdie via
> > > > lists.openembedded.org wrote:
> > > > > qemuarm has ~350 failures
> > > > > qemuarm64 has ~350 failures
> > > > > qemux86-64 has ~4000 (3900 in glibc)
> > > > > qemux86 has ~4000 (3500 in glibc)
> > > > > qemuppc has ~600 failures
> > > > > qemumips64 has ~5000 failures (all over)
> > > > > qemumips has ~1600 failures
> > > > > 
> > > > > Anuj: Can Intel look into the glibc test failures on x86?
> > > > 
> > > > I realised the glibc issues were due to the network being
> > > > disabled
> > > > for
> > > > the tests and have sent a patch to fix that. That reduces the
> > > > failures
> > > > from ~3900 to ~330. We should really try and reduce that
> > > > further
> > > > but
> > > > it
> > > > is a start!
> > > > 
> > > 
> > > A lot of locale/iconv tests seemed to be failing when calling
> > > write
> > > with large buffers/files over NFS. Some of others were triggering
> > > OOM.
> > > 
> > > I ran the tests again after making a few changes:
> > > 
> > > https://autobuilder.yocto.io/pub/non-release/20230726-11/testresults/qemux86-64-tc/
> > > 
> > > After switching NFS mount to TCP and increasing the memory
> > > available to
> > > 1024, the number of tests failed came down to 69.
> > > 
> > 
> > This is a nice, thanks for doing it. I looked quickly at your
> > results
> > especially glibc part
> > and it seems some of remaining failures are in nss module. I could
> > not
> > see detail logs
> > why those tests were failing but few things to check is if we
> > install
> 
> All the nss tests are failing here:
> 
> https://github.com/bminor/glibc/blob/master/support/test-container.c#L842
> 

I took a look at this and these are failing because of two
reasons. We're looking at the wrong path when creating that file
because of OBJDIR_PATH we set here:

https://git.yoctoproject.org/poky/tree/meta/recipes-core/glibc/glibc/0022-Avoid-hardcoded-build-time-paths-in-the-output-binar.patch

That path is not correct for glibc-testsuite.

Next, the test tries to acquire a lock to a file on NFS which times out
because the user space server we use doesn't support file locking with
NLM.

After reverting this patch for testing and disabling locking, all the
tests that need the test container and some others pass:

https://autobuilder.yocto.io/pub/non-release/20230727-9/testresults/qemux86-64-tc/

glibc            | 5016         | 40       | 144         | -

Thanks,

Anuj

Re: [OE-core] Toolchain test results

[Richard Purdie]

On Wed, 2023-07-26 at 06:00 +0000, Mittal, Anuj wrote:
> On Thu, 2023-07-20 at 12:26 +0100, Richard Purdie wrote:
> > On Tue, 2023-07-18 at 10:14 +0100, Richard Purdie via
> > lists.openembedded.org wrote:
> > > qemuarm has ~350 failures
> > > qemuarm64 has ~350 failures
> > > qemux86-64 has ~4000 (3900 in glibc)
> > > qemux86 has ~4000 (3500 in glibc)
> > > qemuppc has ~600 failures
> > > qemumips64 has ~5000 failures (all over)
> > > qemumips has ~1600 failures
> > > 
> > > Anuj: Can Intel look into the glibc test failures on x86?
> > 
> > I realised the glibc issues were due to the network being disabled
> > for
> > the tests and have sent a patch to fix that. That reduces the
> > failures
> > from ~3900 to ~330. We should really try and reduce that further but
> > it
> > is a start!
> > 
> 
> A lot of locale/iconv tests seemed to be failing when calling write
> with large buffers/files over NFS. Some of others were triggering OOM.
> 
> I ran the tests again after making a few changes:
> 
> https://autobuilder.yocto.io/pub/non-release/20230726-11/testresults/qemux86-64-tc/
> 
> After switching NFS mount to TCP and increasing the memory available to
> 1024, the number of tests failed came down to 69.
> 
> =======================================================================
> =======================================
> qemux86-64 PTest Result Summary
> =======================================================================
> =======================================
> -----------------------------------------------------------------------
> ---------------------------------------
> Recipe           | Passed       | Failed   | Skipped     | Time(s)
> -----------------------------------------------------------------------
> ---------------------------------------
> binutils         | 289          | 0        | 8           | -
> binutils-gas     | 1582         | 4        | 1           | -
> binutils-ld      | 1622         | 6        | 111         | -
> gcc              | 149696       | 28       | 27599       | -
> gcc-g++          | 219260       | 30       | 21108       | -
> gcc-libatomic    | 27           | 1        | 27          | -
> gcc-libgomp      | 3426         | 1        | 1962        | -
> gcc-libitm       | 24           | 1        | 24          | -
> gcc-libstdc++-v3 | 9695         | 32       | 5459        | -
> glibc            | 4984         | 69       | 147         | -
> rust             | 15768        | 0        | 554         | -
> -----------------------------------------------------------------------
> ---------------------------------------


That looks like really good progress, thanks Anuj!

I think we might rebuild M2 to allow for some of the resulttool, ltp
and other test fixes to make it in. I want to look into this 32 bit arm
regression first to see what is going on there. If you're able to send
some of these out we should pull them in and improve the failures for
glibc too.

Cheers,

Richard