Re: [OE-core] [PATCH 2/2] glibc: ptest: Add support for running glibc test suite with ptest

["?ukasz Majewski" <lukma@denx.de>]

[-- Attachment #1: Type: text/plain, Size: 7344 bytes --]

Hi Richard,

> On Tue, 2021-08-24 at 09:36 +0200, ?ukasz Majewski wrote:
> > Hi Khem,
> >   
> > > On Mon, Aug 23, 2021 at 1:09 PM Lukasz Majewski <lukma@denx.de>
> > > wrote:  
> > > > 
> > > > On Mon, 23 Aug 2021 12:52:44 -0700
> > > > Khem Raj <raj.khem@gmail.com> wrote:
> > > >    
> > > > > On Mon, Aug 23, 2021 at 11:24 AM Lukasz Majewski
> > > > > <lukma@denx.de> wrote:
> > > > >    
> > > > > > Hi Khem,
> > > > > >    
> > > > > > > On 8/23/21 8:08 AM, ?ukasz Majewski wrote:    
> > > > > > > > This patch introduces new recipe - namely 'glibc-tests',
> > > > > > > > which builds and installs glibc test suite to OE/Yocto
> > > > > > > > built image.
> > > > > > > > 
> > > > > > > > It reuses code from already available 'glibc-testsuite'
> > > > > > > > recipe, which is run with 'bitbake glibc-testsuite -c
> > > > > > > > check' and uses qemu to execute remotely (via SSH)
> > > > > > > > tests on some emulated machine.
> > > > > > > > 
> > > > > > > > This recipe installs eligible tests on some rootfs
> > > > > > > > image. Afterwards, either all of them or only time
> > > > > > > > related subset, those tests can be executed on the real
> > > > > > > > hardware, to facilitate validation of this platform
> > > > > > > > with for example Y2038 problem compliance.
> > > > > > > > 
> > > > > > > > By default all tests are executed, with 'ptest-runner
> > > > > > > > glibc-tests'. To test only time related subset - one
> > > > > > > > needs to call: cd /usr/lib/glibc-tests/ptest/; rm
> > > > > > > > run-ptest; \ ln -s run-ptest-time run-ptest; cd -;
> > > > > > > > ptest-runner glibc-tests
> > > > > > > > 
> > > > > > > > To facilitate debugging, source files are provided by
> > > > > > > > default with the unstripped debugging symbols. Such
> > > > > > > > approach would reduce the already complex recipe (as it
> > > > > > > > inherits base glibc one), so there is no need to also
> > > > > > > > install *-dbg and *-src packages.    
> > > > > > > 
> > > > > > > 
> > > > > > > does it have to be a separate recipe I wonder if we can
> > > > > > > have it built as part of glibc itself controlled via
> > > > > > > ptest knob    
> > > > > > 
> > > > > > I've followed the glibc-testsuite recipe to provide tests
> > > > > > for ptest. Test creation is similar to it, but doesn't
> > > > > > require QEMU run (tests are executed on the HW).
> > > > > > 
> > > > > > Another rationale was to have some kind of "features"
> > > > > > separation in different file (liked glibc-mtrace), which is
> > > > > > easier to maintain and review.
> > > > > > 
> > > > > > Last but not least - separate recipe (and built binaries)
> > > > > > allow some kind of magic with selection of used test
> > > > > > programs (this may be useful if one would like to have
> > > > > > different sets of tests in different packages)    
> > > > > 
> > > > > That’s seems ok I think the names are quite confusing now not
> > > > > that it was not so much better before but now we have
> > > > > glibc-tests and glibc-testsuites which do same things but in
> > > > > very different way maybe glibc-testsuite should be renamed to
> > > > > something like glibc-tests-crosd or some such    
> > > > 
> > > > I think that glibc-testsuite_2.34.bb shall be renamed to
> > > > glibc-tests-qemu_2.34.bb as it is more descriptive.    
> > > 
> > > is it only runnable in qemu ? I thought we could use a read board
> > > as well with IP address  
> > 
> > It looks like by default the QEMU is used in this recipe. 
> > 
> > This recipe provides custom check-test-wrapper, which can log
> > into remote board via ssh (when TOOLCHAIN_TEST_TARGET = "ssh").
> > 
> > This looks a bit awkward for me, as the goal would be to run built
> > tests on target (exact) HW without the need of SSH.
> > 
> > It is also easier to debug the code with the real HW.  
> 
> I'm a bit worried we're going around in circles with this. We did
> have a different form of tests a while back but I was told those
> didn't work well and they were replaced with the current setup which
> worked with the driver in the glibc source code.

I've looked into the glibc-testing.inc from SHA1:
0a42ae7f7ca8fcebe4630b701e50e2352f9b7c3b

There glibc tests were built and copied on the target via NFS.

> This was done around
> the time we added support for the other toolchain test suites
> (binutils+gcc).

I'm not familiar with the "binutils+gcc" issue, which was solved in the
past. Could you share any reference?

> We ended up with options to use full system qemu or
> user mode qemu emulation depending on the speed of the target (e.g.
> with kvm or not).

From what I see in the recipe - you either run user mode QEMU by
default (at least when I do run it):

NOTE: make PARALLELMFLAGS=-j 8 SHELL=/bin/bash -i
QEMU_SYSROOT=/home/lukma/work/yocto/y2038/build/tmp/work/armv7at2hf-neon-poky-linux-gnueabi/glibc-testsuite/2.34-r0/recipe-sysroot
QEMU_OPTIONS=qemu-arm -r 5.1.0  SSH_HOST=localhost SSH_HOS T_USER=root
SSH_HOST_PORT=2222
test-wrapper=/home/lukma/work/yocto/y2038/build/tmp/work/armv7at2hf-neon-poky-linux-gnueabi/glibc-testsuite/2.34-r0/check-test-wrapper
user check

Or you log to the board via SSH (the 'ssh' mode).

> 
> Has this patchset been run with all the tests in glibc or just a
> subset?

This patch set allows running all tests [*], or (preferably in my use
case) only a time related ones. The latter is to validate Y2038 code,
which testing has two issues:

1. It cannot be tested with QEMU user mode [1]. Instead "linux
timespaces" were suggested, but it lacks support for CLOCK_REALTIME
manipulation for those tests [2].

2. Testing with SSH is possible, but is not as robust as expected (i.e.
imposes SSHD on the target board, and ETH connection).

> 
> I'm always very nervous about having two ways to do similar things.

Technically, I just reuse the glibc-testsuite_2.34.bb to the point
where tests are _only_ built. Then I pack those binaries and install on
the target's rootfs to be run with ptest framework.

Such approach has several advantages:

- No need for SSH (which can hang, as you need sshfs in user space for
  the setup)

- Run on the real HW (and environment) with the same version of glibc
  on the real target.

- Sources and debugging symbols available on the target, so easier GDB
  debugging.

- Using ptest framework to execute tests

- Time related tests are "self contained", so could be executed without
  built data.

At least for Y2038 validation such approach seems more appealing.

> 
> Cheers,
> 
> Richard
> 

Links:
[1] - https://www.mail-archive.com/qemu-discuss@nongnu.org/msg05898.html
[2] -
https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg2361284.html

Note:
[*] - some glibc tests rely on the built directory to provide correct
data/environment for execution. Time related tests are (and should be)
self contained, so those could be just executed as binaries.

Best regards,

Lukasz Majewski

--

DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email: lukma@denx.de

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 499 bytes --]