Re: [OE-core] [PATCH 2/2] glibc: ptest: Add support for running glibc test suite with ptest

["Richard Purdie" <richard.purdie@linuxfoundation.org>]

On Tue, 2021-08-24 at 09:36 +0200, ?ukasz Majewski wrote:
> Hi Khem,
> 
> > On Mon, Aug 23, 2021 at 1:09 PM Lukasz Majewski <lukma@denx.de> wrote:
> > > 
> > > On Mon, 23 Aug 2021 12:52:44 -0700
> > > Khem Raj <raj.khem@gmail.com> wrote:
> > >  
> > > > On Mon, Aug 23, 2021 at 11:24 AM Lukasz Majewski <lukma@denx.de>
> > > > wrote:
> > > >  
> > > > > Hi Khem,
> > > > >  
> > > > > > On 8/23/21 8:08 AM, ?ukasz Majewski wrote:  
> > > > > > > This patch introduces new recipe - namely 'glibc-tests',
> > > > > > > which builds and installs glibc test suite to OE/Yocto
> > > > > > > built image.
> > > > > > > 
> > > > > > > It reuses code from already available 'glibc-testsuite'
> > > > > > > recipe, which is run with 'bitbake glibc-testsuite -c
> > > > > > > check' and uses qemu to execute remotely (via SSH) tests on
> > > > > > > some emulated machine.
> > > > > > > 
> > > > > > > This recipe installs eligible tests on some rootfs image.
> > > > > > > Afterwards, either all of them or only time related subset,
> > > > > > > those tests can be executed on the real hardware, to
> > > > > > > facilitate validation of this platform with for example
> > > > > > > Y2038 problem compliance.
> > > > > > > 
> > > > > > > By default all tests are executed, with 'ptest-runner
> > > > > > > glibc-tests'. To test only time related subset - one needs
> > > > > > > to call: cd /usr/lib/glibc-tests/ptest/; rm run-ptest; \
> > > > > > > ln -s run-ptest-time run-ptest; cd -; ptest-runner
> > > > > > > glibc-tests
> > > > > > > 
> > > > > > > To facilitate debugging, source files are provided by
> > > > > > > default with the unstripped debugging symbols. Such
> > > > > > > approach would reduce the already complex recipe (as it
> > > > > > > inherits base glibc one), so there is no need to also
> > > > > > > install *-dbg and *-src packages.  
> > > > > > 
> > > > > > 
> > > > > > does it have to be a separate recipe I wonder if we can have
> > > > > > it built as part of glibc itself controlled via ptest knob  
> > > > > 
> > > > > I've followed the glibc-testsuite recipe to provide tests for
> > > > > ptest. Test creation is similar to it, but doesn't require QEMU
> > > > > run (tests are executed on the HW).
> > > > > 
> > > > > Another rationale was to have some kind of "features"
> > > > > separation in different file (liked glibc-mtrace), which is
> > > > > easier to maintain and review.
> > > > > 
> > > > > Last but not least - separate recipe (and built binaries) allow
> > > > > some kind of magic with selection of used test programs (this
> > > > > may be useful if one would like to have different sets of tests
> > > > > in different packages)  
> > > > 
> > > > That’s seems ok I think the names are quite confusing now not
> > > > that it was not so much better before but now we have glibc-tests
> > > > and glibc-testsuites which do same things but in very different
> > > > way maybe glibc-testsuite should be renamed to something like
> > > > glibc-tests-crosd or some such  
> > > 
> > > I think that glibc-testsuite_2.34.bb shall be renamed to
> > > glibc-tests-qemu_2.34.bb as it is more descriptive.  
> > 
> > is it only runnable in qemu ? I thought we could use a read board as
> > well with IP address
> 
> It looks like by default the QEMU is used in this recipe. 
> 
> This recipe provides custom check-test-wrapper, which can log
> into remote board via ssh (when TOOLCHAIN_TEST_TARGET = "ssh").
> 
> This looks a bit awkward for me, as the goal would be to run built
> tests on target (exact) HW without the need of SSH.
> 
> It is also easier to debug the code with the real HW.

I'm a bit worried we're going around in circles with this. We did have a
different form of tests a while back but I was told those didn't work well and
they were replaced with the current setup which worked with the driver in the
glibc source code. This was done around the time we added support for the other
toolchain test suites (binutils+gcc). We ended up with options to use full
system qemu or user mode qemu emulation depending on the speed of the target
(e.g. with kvm or not).

Has this patchset been run with all the tests in glibc or just a subset?

I'm always very nervous about having two ways to do similar things.

Cheers,

Richard