eglibc testing?

eglibc testing?

[Phil Blundell]

I just ran "make check" under qemu on my eglibc-2.13 build (from oe-core
master) and got rather a lot of failures.  It wasn't immediately obvious
to me whether these were libc bugs, qemu infelicities, oe-related
harness problems, or some kind of local finger trouble.

Before I spend too much time investigating this, are other people seeing
the same effect?  What approach do the Yocto folks use for glibc
testing: are they running the testsuite directly on target hardware, or
via qemu, or something else?

The gcc testsuite is next on my list to look at and, again, I'd be
interested to hear what approach other folks are using there and what
sort of results they are currently getting.

thanks

p.

Re: eglibc testing?

[Khem Raj]

On 6/16/2011 3:15 AM, Phil Blundell wrote:
> I just ran "make check" under qemu on my eglibc-2.13 build (from oe-core
> master) and got rather a lot of failures.  It wasn't immediately obvious
> to me whether these were libc bugs, qemu infelicities, oe-related
> harness problems, or some kind of local finger trouble.
>
> Before I spend too much time investigating this, are other people seeing
> the same effect?  What approach do the Yocto folks use for glibc
> testing: are they running the testsuite directly on target hardware, or
> via qemu, or something else?

I nfsmount eglibc builddir on target and run the testsuite. I see aroung 
10-15 fails. Havent run the testsuite in a while

>
> The gcc testsuite is next on my list to look at and, again, I'd be
> interested to hear what approach other folks are using there and what
> sort of results they are currently getting.
>

I have a script that I use to launch cross testing needs a bit of setup
like passwordless ssh access to target. Results are pretty sane I have 
posted results to oe-ml occasionally.

I can start some testsuite once my build finishes today and post.

> thanks
>
> p.
>
>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Re: eglibc testing?

[Phil Blundell]

On Thu, 2011-06-16 at 06:41 -0700, Khem Raj wrote:
> I nfsmount eglibc builddir on target and run the testsuite. I see aroung 
> 10-15 fails. Havent run the testsuite in a while

Maybe the right thing to do with eglibc would be to build a rootfs
containing the testsuite, run it up in qemu-system-arm, and then
engineer some IPC mechanism to get the test results back to the host.
That way you would at least be testing something close to the real
target kernel, rather than the qemu syscall emulation which I suspect is
to blame for some of my failures.

For gcc it'd probably be fine to just run the individual tests under
qemu directly since they don't tend to have a lot of OS dependency.

p.

Re: eglibc testing?

[Khem Raj]

On 06/16/2011 07:29 AM, Phil Blundell wrote:
> On Thu, 2011-06-16 at 06:41 -0700, Khem Raj wrote:
>> I nfsmount eglibc builddir on target and run the testsuite. I see aroung
>> 10-15 fails. Havent run the testsuite in a while
>
> Maybe the right thing to do with eglibc would be to build a rootfs
> containing the testsuite, run it up in qemu-system-arm, and then
> engineer some IPC mechanism to get the test results back to the host.

using nfs isnt bad either since the tests excercise libraries from /lib
on target IOW installed eglibc.

> That way you would at least be testing something close to the real
> target kernel, rather than the qemu syscall emulation which I suspect is
> to blame for some of my failures.
>
> For gcc it'd probably be fine to just run the individual tests under
> qemu directly since they don't tend to have a lot of OS dependency.
>

Yes I think I should put together my mechanism somewhere and post it
and dejaGNU setup I use for cross testing

> p.
>
>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Re: eglibc testing?

[Phil Blundell]

On Thu, 2011-06-16 at 16:18 -0700, Khem Raj wrote:
> On 06/16/2011 07:29 AM, Phil Blundell wrote:
> > On Thu, 2011-06-16 at 06:41 -0700, Khem Raj wrote:
> >> I nfsmount eglibc builddir on target and run the testsuite. I see aroung
> >> 10-15 fails. Havent run the testsuite in a while
> >
> > Maybe the right thing to do with eglibc would be to build a rootfs
> > containing the testsuite, run it up in qemu-system-arm, and then
> > engineer some IPC mechanism to get the test results back to the host.
> 
> using nfs isnt bad either since the tests excercise libraries from /lib
> on target IOW installed eglibc.

True, though I can't think of any instances where the libc behaviour
will change depending on where it's installed. 

The thing about nfs is that it's awkward to set up in an automated way.
What I want is to arrive at a situation where you can just do:

$ bitbake -c check micro-base-image (or whatever)

and have it build all the necessary components, run their respective
testsuites, and generate a report of both the current status and any
regressions.  Ideally I want this to be doable without a lot of
installation-specific fiddling around with networking configuration and
suchlike, and I definitely want an arrangement which doesn't rely on
having target hardware on hand.  Practically speaking I think that means
it basically has to be qemu of some kind or another.

> Yes I think I should put together my mechanism somewhere and post it
> and dejaGNU setup I use for cross testing

That would be cool.

p.

Re: eglibc testing?

[Richard Purdie]

On Fri, 2011-06-17 at 11:13 +0100, Phil Blundell wrote:
> On Thu, 2011-06-16 at 16:18 -0700, Khem Raj wrote:
> > On 06/16/2011 07:29 AM, Phil Blundell wrote:
> > > On Thu, 2011-06-16 at 06:41 -0700, Khem Raj wrote:
> > >> I nfsmount eglibc builddir on target and run the testsuite. I see aroung
> > >> 10-15 fails. Havent run the testsuite in a while
> > >
> > > Maybe the right thing to do with eglibc would be to build a rootfs
> > > containing the testsuite, run it up in qemu-system-arm, and then
> > > engineer some IPC mechanism to get the test results back to the host.
> > 
> > using nfs isnt bad either since the tests excercise libraries from /lib
> > on target IOW installed eglibc.
> 
> True, though I can't think of any instances where the libc behaviour
> will change depending on where it's installed. 
> 
> The thing about nfs is that it's awkward to set up in an automated way.

The qemu scripts in OE-Core use usermode NFS by default for booting qemu
images...

> What I want is to arrive at a situation where you can just do:
> 
> $ bitbake -c check micro-base-image (or whatever)
> 
> and have it build all the necessary components, run their respective
> testsuites, and generate a report of both the current status and any
> regressions.  Ideally I want this to be doable without a lot of
> installation-specific fiddling around with networking configuration and
> suchlike, and I definitely want an arrangement which doesn't rely on
> having target hardware on hand.  Practically speaking I think that means
> it basically has to be qemu of some kind or another.
> 
> > Yes I think I should put together my mechanism somewhere and post it
> > and dejaGNU setup I use for cross testing
> 
> That would be cool.

We have the automated image QA tests in OE-Core already so perhaps we
could extend these to handle the tests somehow?

Cheers,

Richard

Re: eglibc testing?

[Phil Blundell]

On Fri, 2011-06-17 at 11:25 +0100, Richard Purdie wrote:
> On Fri, 2011-06-17 at 11:13 +0100, Phil Blundell wrote:
> > The thing about nfs is that it's awkward to set up in an automated way.
> 
> The qemu scripts in OE-Core use usermode NFS by default for booting qemu
> images...

Yeah, indeed, and I think the mechanism that they use to do this
supports my assessment of "awkward".  But presumably it does at least
work and I guess that might be the easiest way to do glibc testing as
well.  For gcc testing I think a dedicated qemu harness is probably a
better answer.

p.

Re: eglibc testing?

[Phil Blundell]

On Thu, 2011-06-16 at 16:18 -0700, Khem Raj wrote:
> Yes I think I should put together my mechanism somewhere and post it
> and dejaGNU setup I use for cross testing

Did you get a chance to do this at all?

thanks

p.

Re: eglibc testing?

[Khem Raj]

On 06/30/2011 03:32 AM, Phil Blundell wrote:
> On Thu, 2011-06-16 at 16:18 -0700, Khem Raj wrote:
>> Yes I think I should put together my mechanism somewhere and post it
>> and dejaGNU setup I use for cross testing
>
> Did you get a chance to do this at all?
>

so far no. Let me collect them and make sure they work

> thanks
>
> p.
>
>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Re: eglibc testing?

[Phil Blundell]

On Thu, 2011-06-30 at 07:29 -0700, Khem Raj wrote:
> On 06/30/2011 03:32 AM, Phil Blundell wrote:
> > On Thu, 2011-06-16 at 16:18 -0700, Khem Raj wrote:
> >> Yes I think I should put together my mechanism somewhere and post it
> >> and dejaGNU setup I use for cross testing
> >
> > Did you get a chance to do this at all?
> >
> 
> so far no. Let me collect them and make sure they work

Just a quick reminder about this :-)

p.

Re: eglibc testing?

[Khem Raj]

On Thursday, August 04, 2011 10:24:00 AM Phil Blundell wrote:
> On Thu, 2011-06-30 at 07:29 -0700, Khem Raj wrote:
> > On 06/30/2011 03:32 AM, Phil Blundell wrote:
> > > On Thu, 2011-06-16 at 16:18 -0700, Khem Raj wrote:
> > >> Yes I think I should put together my mechanism somewhere and post
> > >> it
> > >> and dejaGNU setup I use for cross testing
> > > 
> > > Did you get a chance to do this at all?
> > 
> > so far no. Let me collect them and make sure they work
> 
> Just a quick reminder about this :-)
> 

Sorry Phil, I owe it to you :) I have not finished it yet.

> p.
> 
> 
> 
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
-- 
Khem Raj

Re: eglibc testing?

[Khem Raj]

Phil Blundell <pb@...> writes:

> 
> I just ran "make check" under qemu on my eglibc-2.13 build (from oe-core
> master) and got rather a lot of failures.  It wasn't immediately obvious
> to me whether these were libc bugs, qemu infelicities, oe-related
> harness problems, or some kind of local finger trouble.
> 
> Before I spend too much time investigating this, are other people seeing
> the same effect?  What approach do the Yocto folks use for glibc
> testing: are they running the testsuite directly on target hardware, or
> via qemu, or something else?
> 
> The gcc testsuite is next on my list to look at and, again, I'd be
> interested to hear what approach other folks are using there and what
> sort of results they are currently getting.
> 


Hi Phil

I have finally found some time to cobble up howto cross test eglibc
on top of oe-core
I have tried to document it here
http://sakrah.homelinux.org/blog/2011/12/cross-testing-eglibc-regression-
testsuite-using-openembedded-coreqemu/

Right now I see around 34 failures on arm out of 1100 odd tests it ran
Let me know if you have any feedback on it.

I will also post the cross gcc testing howto sometimes soon

> thanks
> 
> p.
>