Re: [OE-core] [PATCH 2/2] glibc: ptest: Add support for running glibc test suite with ptest

["?ukasz Majewski" <lukma@denx.de>]

[-- Attachment #1: Type: text/plain, Size: 14631 bytes --]

Hi Nathan,

> On Fri, 27 Aug 2021 at 19:20, Lukasz Majewski <lukma@denx.de> wrote:
> >
> > Hi Nathan,
> >  
> > > On Thu, 26 Aug 2021 at 23:38, Richard Purdie
> > > <richard.purdie@linuxfoundation.org> wrote:  
> > > >
> > > > On Thu, 2021-08-26 at 14:27 +0200, Lukasz Majewski wrote:  
> > > > > Hi Richard, Khem,
> > > > >  
> > > > > > Hi Richard,
> > > > > >  
> > > > > > > On Tue, 2021-08-24 at 09:36 +0200, ?ukasz Majewski wrote:
> > > > > > >  
> > > > > > > > Hi Khem,
> > > > > > > >  
> > > > > > > > > On Mon, Aug 23, 2021 at 1:09 PM Lukasz Majewski
> > > > > > > > > <lukma@denx.de> wrote:  
> > > > > > > > > >
> > > > > > > > > > On Mon, 23 Aug 2021 12:52:44 -0700
> > > > > > > > > > Khem Raj <raj.khem@gmail.com> wrote:
> > > > > > > > > >  
> > > > > > > > > > > On Mon, Aug 23, 2021 at 11:24 AM Lukasz Majewski
> > > > > > > > > > > <lukma@denx.de> wrote:
> > > > > > > > > > >  
> > > > > > > > > > > > Hi Khem,
> > > > > > > > > > > >  
> > > > > > > > > > > > > On 8/23/21 8:08 AM, ?ukasz Majewski wrote:  
> > > > > > > > > > > > > > This patch introduces new recipe - namely
> > > > > > > > > > > > > > 'glibc-tests', which builds and installs
> > > > > > > > > > > > > > glibc test suite to OE/Yocto built image.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > It reuses code from already available
> > > > > > > > > > > > > > 'glibc-testsuite' recipe, which is run with
> > > > > > > > > > > > > > 'bitbake glibc-testsuite -c check' and uses
> > > > > > > > > > > > > > qemu to execute remotely (via SSH) tests on
> > > > > > > > > > > > > > some emulated machine.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > This recipe installs eligible tests on some
> > > > > > > > > > > > > > rootfs image. Afterwards, either all of
> > > > > > > > > > > > > > them or only time related subset, those
> > > > > > > > > > > > > > tests can be executed on the real hardware,
> > > > > > > > > > > > > > to facilitate validation of this platform
> > > > > > > > > > > > > > with for example Y2038 problem compliance.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > By default all tests are executed, with
> > > > > > > > > > > > > > 'ptest-runner glibc-tests'. To test only
> > > > > > > > > > > > > > time related subset - one needs to call: cd
> > > > > > > > > > > > > > /usr/lib/glibc-tests/ptest/; rm run-ptest;
> > > > > > > > > > > > > > \ ln -s run-ptest-time run-ptest; cd -;
> > > > > > > > > > > > > > ptest-runner glibc-tests
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > To facilitate debugging, source files are
> > > > > > > > > > > > > > provided by default with the unstripped
> > > > > > > > > > > > > > debugging symbols. Such approach would
> > > > > > > > > > > > > > reduce the already complex recipe (as it
> > > > > > > > > > > > > > inherits base glibc one), so there is no
> > > > > > > > > > > > > > need to also install *-dbg and *-src
> > > > > > > > > > > > > > packages.  
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > does it have to be a separate recipe I wonder
> > > > > > > > > > > > > if we can have it built as part of glibc
> > > > > > > > > > > > > itself controlled via ptest knob  
> > > > > > > > > > > >
> > > > > > > > > > > > I've followed the glibc-testsuite recipe to
> > > > > > > > > > > > provide tests for ptest. Test creation is
> > > > > > > > > > > > similar to it, but doesn't require QEMU run
> > > > > > > > > > > > (tests are executed on the HW).
> > > > > > > > > > > >
> > > > > > > > > > > > Another rationale was to have some kind of
> > > > > > > > > > > > "features" separation in different file (liked
> > > > > > > > > > > > glibc-mtrace), which is easier to maintain and
> > > > > > > > > > > > review.
> > > > > > > > > > > >
> > > > > > > > > > > > Last but not least - separate recipe (and built
> > > > > > > > > > > > binaries) allow some kind of magic with
> > > > > > > > > > > > selection of used test programs (this may be
> > > > > > > > > > > > useful if one would like to have different sets
> > > > > > > > > > > > of tests in different packages)  
> > > > > > > > > > >
> > > > > > > > > > > That’s seems ok I think the names are quite
> > > > > > > > > > > confusing now not that it was not so much better
> > > > > > > > > > > before but now we have glibc-tests and
> > > > > > > > > > > glibc-testsuites which do same things but in very
> > > > > > > > > > > different way maybe glibc-testsuite should be
> > > > > > > > > > > renamed to something like glibc-tests-crosd or
> > > > > > > > > > > some such  
> > > > > > > > > >
> > > > > > > > > > I think that glibc-testsuite_2.34.bb shall be
> > > > > > > > > > renamed to glibc-tests-qemu_2.34.bb as it is more
> > > > > > > > > > descriptive.  
> > > > > > > > >
> > > > > > > > > is it only runnable in qemu ? I thought we could use a
> > > > > > > > > read board as well with IP address  
> > > > > > > >
> > > > > > > > It looks like by default the QEMU is used in this
> > > > > > > > recipe.
> > > > > > > >
> > > > > > > > This recipe provides custom check-test-wrapper, which
> > > > > > > > can log into remote board via ssh (when
> > > > > > > > TOOLCHAIN_TEST_TARGET = "ssh").
> > > > > > > >
> > > > > > > > This looks a bit awkward for me, as the goal would be to
> > > > > > > > run built tests on target (exact) HW without the need of
> > > > > > > > SSH.
> > > > > > > >
> > > > > > > > It is also easier to debug the code with the real HW.  
> > > > > > >
> > > > > > > I'm a bit worried we're going around in circles with
> > > > > > > this. We did have a different form of tests a while back
> > > > > > > but I was told those didn't work well and they were
> > > > > > > replaced with the current setup which worked with the
> > > > > > > driver in the glibc source code.  
> > > > > >
> > > > > > I've looked into the glibc-testing.inc from SHA1:
> > > > > > 0a42ae7f7ca8fcebe4630b701e50e2352f9b7c3b
> > > > > >
> > > > > > There glibc tests were built and copied on the target via
> > > > > > NFS. 
> > > > > > > This was done around
> > > > > > > the time we added support for the other toolchain test
> > > > > > > suites (binutils+gcc).  
> > > > > >
> > > > > > I'm not familiar with the "binutils+gcc" issue, which was
> > > > > > solved in the past. Could you share any reference?
> > > > > >  
> > > > > > > We ended up with options to use full system qemu or
> > > > > > > user mode qemu emulation depending on the speed of the
> > > > > > > target (e.g. with kvm or not).  
> > > > > >
> > > > > > From what I see in the recipe - you either run user mode
> > > > > > QEMU by default (at least when I do run it):
> > > > > >
> > > > > > NOTE: make PARALLELMFLAGS=-j 8 SHELL=/bin/bash -i
> > > > > > QEMU_SYSROOT=/home/lukma/work/yocto/y2038/build/tmp/work/armv7at2hf-neon-poky-linux-gnueabi/glibc-testsuite/2.34-r0/recipe-sysroot
> > > > > > QEMU_OPTIONS=qemu-arm -r 5.1.0  SSH_HOST=localhost SSH_HOS
> > > > > > T_USER=root SSH_HOST_PORT=2222
> > > > > > test-wrapper=/home/lukma/work/yocto/y2038/build/tmp/work/armv7at2hf-neon-poky-linux-gnueabi/glibc-testsuite/2.34-r0/check-test-wrapper
> > > > > > user check
> > > > > >
> > > > > > Or you log to the board via SSH (the 'ssh' mode).
> > > > > >  
> > > > > > >
> > > > > > > Has this patchset been run with all the tests in glibc or
> > > > > > > just a subset?  
> > > > > >
> > > > > > This patch set allows running all tests [*], or (preferably
> > > > > > in my use case) only a time related ones. The latter is to
> > > > > > validate Y2038 code, which testing has two issues:
> > > > > >
> > > > > > 1. It cannot be tested with QEMU user mode [1]. Instead
> > > > > > "linux timespaces" were suggested, but it lacks support for
> > > > > > CLOCK_REALTIME manipulation for those tests [2].  
> > >
> > > So just a note, the qemu user mode testing is not a complete
> > > solution. There are a number of problems with it, but in general
> > > it is still quite useful for validating general functionally,
> > > especially when implementing a new platform.  
> >
> > +1
> >  
> > > A big one is due to the ability to run
> > > tests in parallel with user mode, it easily reduces test run
> > > times by an order of magnitude (compared to qemu system, which
> > > for some platforms is faster than real hardware).  
> >
> > +1
> >  
> > >
> > > The reason why qemu user mode is the default behaviour is because
> > > running "bitbake glibc-testsuite -c check" will work without any
> > > external dependencies (the remote machine, NFS/sshfs, etc.).  
> >
> > +1
> >  
> > >  
> > > > > >
> > > > > > 2. Testing with SSH is possible, but is not as robust as
> > > > > > expected (i.e. imposes SSHD on the target board, and ETH
> > > > > > connection).  
> > > > > > >
> > > > > > > I'm always very nervous about having two ways to do
> > > > > > > similar things.  
> > > > > >
> > > > > > Technically, I just reuse the glibc-testsuite_2.34.bb to the
> > > > > > point where tests are _only_ built. Then I pack those
> > > > > > binaries and install on the target's rootfs to be run with
> > > > > > ptest framework.  
> > >
> > > So it appears the make variable "run-built-tests" excludes the
> > > building of some tests.  
> >
> > As fair as I remember, the 'run-built-tests=no' switch tells the
> > glibc build system to not execute built tests.
> >  
> > > So I am not sure if the complete test suite
> > > can be built in this way.  
> >
> > I'm pretty sure that all tests required for 'check' are build.  
> 
> If you look at some of the Makefiles in glibc, it conditionally
> includes/excludes some tests from being built based on the value of
> run-built-tests. For example:
> 
> https://sourceware.org/git/?p=glibc.git;a=blob;f=localedata/Makefile;h=f585e0dd4115a99493e843aeed464090da19665f;hb=HEAD#l141
> 

I see. Thanks for pointing this out.

> >  
> > > Though I can see some value in having the
> > > glibc-testsuite recipe being able to only build the tests for
> > > development purposes (e.g. TOOLCHAIN_TEST_TARGET = "build"?).
> > >
> > > When I had implemented the testsuite recipe the glibc testsuite
> > > relied quite heavily on the environment it was built and run in,
> > > in particular around configuration. That is probably still an
> > > issue.  
> >
> > Indeed, some (a few?) tests rely on the build environment in which
> > they were build. I do personally, regard this as a some kind of a
> > bug. I do believe, that when you build test for a particular
> > version of glibc (2.34 in this case), one shall expect that it can
> > be run only by linking the proper library.
> >
> > This is the case for "time" related tests. On my test system with
> > this patch applied, all of them are executed with no errors. Those
> > are self contained.
> >  
> > >
> > > It would be interesting to know how much of the test suite can be
> > > run/passes and how that compares to the results of existing
> > > system/user emulation with cross execution using the glibc
> > > makefile.  
> >
> > I can provide exact numbers when I finish porting more Y2038 related
> > patches to newest glibc -master.  
> 
> Also something to keep in mind is that sometimes the pass/fail of the
> test binary is not what is being validated by the test. For example
> this test uses mtrace, and compares the results of the output mtrace
> file, not the test itself.
> 
> https://sourceware.org/git/?p=glibc.git;a=blob;f=localedata/Makefile;h=f585e0dd4115a99493e843aeed464090da19665f;hb=HEAD#l462
> 

Ok. I see.

My main objective is to use time related programs, which rely on the
interfaces provided by the kernel. 

From the above POV, those tests are a "standalone" ones.

> Regards,
> Nathan
> 
> >  
> > >  
> > > > > >
> > > > > > Such approach has several advantages:
> > > > > >
> > > > > > - No need for SSH (which can hang, as you need sshfs in user
> > > > > > space for the setup)  
> > >
> > > I had not tested with sshfs that likely has quite low
> > > performance, the OEQA tests rely on NFS for QEMU system emulation
> > > and that is what I had been using for hardware testing.  
> >
> > I had some issues with NFS - either I couldn't place some OE/Yocto
> > build directory in it or glibc's build directory was not working
> > with it correctly.
> >
> > That was why I had to use the sshfs instead, which is _really_ slow
> > and often breaks.
> >  
> > > However it would be interesting
> > > to see if virtiofs could improve the performance for testing with
> > > QEMU system emulation.
> > >  
> > > > > >
> > > > > > - Run on the real HW (and environment) with the same
> > > > > > version of glibc on the real target.
> > > > > >
> > > > > > - Sources and debugging symbols available on the target, so
> > > > > > easier GDB debugging.
> > > > > >
> > > > > > - Using ptest framework to execute tests
> > > > > >
> > > > > > - Time related tests are "self contained", so could be
> > > > > > executed without built data.
> > > > > >
> > > > > > At least for Y2038 validation such approach seems more
> > > > > > appealing.  
> > > > >
> > > > > Do you have any more feedback regarding this patch?  
> > > >
> > > > We're at feature freeze and this wasn't a planned change so it
> > > > is too late for this release cycle, I'm struggling with the
> > > > things that were planned.
> > > >
> > > > I need to spend some time looking at what the other code is
> > > > doing and how we're using it on the autobuilder and whether we
> > > > want to change to use the ptest approach or not.
> > > >
> > > > I added Nathan Rossi to cc since he worked on the other code for
> > > > this and may have more insight.  
> > >
> > > Thanks for cc'ing, I somehow managed to miss the thread.
> > >
> > > Regards,
> > > Nathan  
> >
> >
> > Best regards,
> >
> > Lukasz Majewski
> >
> > --
> >
> > DENX Software Engineering GmbH,      Managing Director: Wolfgang
> > Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell,
> > Germany Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email:
> > lukma@denx.de  




Best regards,

Lukasz Majewski

--

DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email: lukma@denx.de

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 499 bytes --]