OE breakages, was: Re: fakeroot fails to build

OE breakages, was: Re: fakeroot fails to build

[Koen Kooi]

On 25-02-09 16:14, Mike (mwester) wrote:

> ((Also, are there any non-angstrom distros still left in OE, besides
> SlugOS?  I really feel like I'm swimming upstream all the time with OE
> -- things break for me so often that just don't seem to affect anyone
> else...))

Things break often<period>

I'm 'fortunate' enough to be able to spend 2 hours every day fixing 
breakages for angstrom, so I can get a working demo for my presentation 
at the bossaconference. Thanks to all the people in the beagleboard 
community I get notified of breakages pretty much the instant they 
happen, which makes finding the cause a lot easier.

This also means that my 'OE time' is almost completely filled with 
bugfixing, instead of working that is fun. I've been trying to push for 
better testing frameworks and removing the taboo stamp from the word 
'review'.

The review bit seems to gaining traction, but 'testing' is still a alien 
and mystic concepts for a lot of OE committers. Worse, some people turn 
*off* testing frameworks like insane.bbclass!

Maybe it's time for OE to consider whether it really wants a zillion 
commits per day or only a few tested commits per day. Having a ton of 
active developers means nothing when people using OE can never get a 
build completed due to all the small, but fatal bugs appearing lately.

I'm close to finishing the new stable branch manifesto which lists 
options and procedures to make it easier to do releases of OE (not 
builds, releases of the metadata itself) and track regressions, but it 
would need about 3 or four people spending a few hours on it every week, 
which sadly might not be feasible, since the .dev branch is too nice and 
shiny compared to any branch.

regards,

Koen

Re: OE breakages, was: Re: fakeroot fails to build

[Bernhard Guillon]

Koen Kooi wrote:
>
> Maybe it's time for OE to consider whether it really wants a zillion 
> commits per day or only a few tested commits per day. Having a ton of 
> active developers means nothing when people using OE can never get a 
> build completed due to all the small, but fatal bugs appearing lately.
>
We maybe can add a tested on foo machines with git revision bar and 
distro x in version y to every commit.
I think most developers are testing their commits but no one can test 
them in every possible combination.

Best regards
Bernhard Guillon

Re: OE breakages, was: Re: fakeroot fails to build

[Frans Meulenbroeks]

> On 25-02-09 16:14, Mike (mwester) wrote:
>
>> ((Also, are there any non-angstrom distros still left in OE, besides
>> SlugOS?  I really feel like I'm swimming upstream all the time with OE
>> -- things break for me so often that just don't seem to affect anyone
>> else...))
>
> Things break often<period>
>
> I'm 'fortunate' enough to be able to spend 2 hours every day fixing
> breakages for angstrom, so I can get a working demo for my presentation
> at the bossaconference. Thanks to all the people in the beagleboard
> community I get notified of breakages pretty much the instant they
> happen, which makes finding the cause a lot easier.
>
> This also means that my 'OE time' is almost completely filled with
> bugfixing, instead of working that is fun. I've been trying to push for
> better testing frameworks and removing the taboo stamp from the word
> 'review'.
>
> The review bit seems to gaining traction, but 'testing' is still a alien
> and mystic concepts for a lot of OE committers. Worse, some people turn
> *off* testing frameworks like insane.bbclass!
>
> Maybe it's time for OE to consider whether it really wants a zillion
> commits per day or only a few tested commits per day. Having a ton of
> active developers means nothing when people using OE can never get a
> build completed due to all the small, but fatal bugs appearing lately.
>
> I'm close to finishing the new stable branch manifesto which lists
> options and procedures to make it easier to do releases of OE (not
> builds, releases of the metadata itself) and track regressions, but it
> would need about 3 or four people spending a few hours on it every week,
> which sadly might not be feasible, since the .dev branch is too nice and
> shiny compared to any branch.

Well I'm happy to follow whatever procedure is needed (and if I do
something wrong by accident, please educate me (no need for anyone to
clean up my mess, I'm more than happy to do that myself, provided I
understand the problem and the proper way to fix it).

The problem I am facing is somewhat different.
I've added packages which build and test fine in my environment but
for some reason they do not build properly on tinderbox. E.g. the
latest version of mpd seems to pick up that there are some residual
elements of docbook present so it assumes docbook is there (but
docbook is in packages/nonworking)!.
This is beyond my capabilities to fix.

BTW, I noticed that there are a lot of bugs in bugzilla about packages
that do not build. Most of these are old. Guess the bugs are fixed but
no-one seems to close them.

Frans.

Re: OE breakages, was: Re: fakeroot fails to build

[Michael 'Mickey' Lauer]

Am Donnerstag, den 26.02.2009, 18:20 +0100 schrieb Frans Meulenbroeks:
> > Maybe it's time for OE to consider whether it really wants a zillion
> > commits per day or only a few tested commits per day.

I think we want both, given the right branches. Let me again propose a
structure such as org.openembedded.{stable,testing,unstable} where only
cherry-picking from the more unstable to the lesser unstable branch is
allowed.

Note that this doesn't conflict with your idea of reviews, someone/some
people still need to decide which revs are good to cherry-pick.

> BTW, I noticed that there are a lot of bugs in bugzilla about packages
> that do not build. Most of these are old. Guess the bugs are fixed but
> no-one seems to close them.

I regularly scan the bugtracker doing my share of cleaning, more people
should do that.

-- 
:M:

Re: OE breakages, was: Re: fakeroot fails to build

[Leon Woestenberg]

Hello,

On Thu, Feb 26, 2009 at 6:20 PM, Frans Meulenbroeks
<fransmeulenbroeks@gmail.com> wrote:
>> On 25-02-09 16:14, Mike (mwester) wrote:
>
> The problem I am facing is somewhat different.
> I've added packages which build and test fine in my environment but
> for some reason they do not build properly on tinderbox.
>
That often indicates a leak to the host and/or missing DEPENDS.

The build machines that run test builds, should do so with the minimal
amount of host requirements.

Easily testable using virtual machines with freshly installed Linux,
as most machines grow their installed package base during their
lifecycle.

Regards,
-- 
Leon

Re: OE breakages, was: Re: fakeroot fails to build

[Tom Rini]

On Fri, Feb 27, 2009 at 05:06:25PM +0100, Leon Woestenberg wrote:
> Hello,
> 
> On Thu, Feb 26, 2009 at 6:20 PM, Frans Meulenbroeks
> <fransmeulenbroeks@gmail.com> wrote:
> >> On 25-02-09 16:14, Mike (mwester) wrote:
> >
> > The problem I am facing is somewhat different.
> > I've added packages which build and test fine in my environment but
> > for some reason they do not build properly on tinderbox.
> >
> That often indicates a leak to the host and/or missing DEPENDS.
> 
> The build machines that run test builds, should do so with the minimal
> amount of host requirements.
> 
> Easily testable using virtual machines with freshly installed Linux,
> as most machines grow their installed package base during their
> lifecycle.

Also, chroots work for this.

-- 
Tom Rini

Re: OE breakages, was: Re: fakeroot fails to build

[Frans Meulenbroeks]

2009/2/27 Tom Rini <trini@kernel.crashing.org>:
> On Fri, Feb 27, 2009 at 05:06:25PM +0100, Leon Woestenberg wrote:
>> Hello,
>>
>> On Thu, Feb 26, 2009 at 6:20 PM, Frans Meulenbroeks
>> <fransmeulenbroeks@gmail.com> wrote:
>> >> On 25-02-09 16:14, Mike (mwester) wrote:
>> >
>> > The problem I am facing is somewhat different.
>> > I've added packages which build and test fine in my environment but
>> > for some reason they do not build properly on tinderbox.
>> >
>> That often indicates a leak to the host and/or missing DEPENDS.
>>
>> The build machines that run test builds, should do so with the minimal
>> amount of host requirements.
>>
>> Easily testable using virtual machines with freshly installed Linux,
>> as most machines grow their installed package base during their
>> lifecycle.
>
> Also, chroots work for this.
>

True but that is not what we have at the moment. E.g the mpd problem I
mentioned earlier in this thread is probably caused by some docbook
residue.

(and I agree that it is easy to miss a DEPENDS because of accumulated
installed packages).

FM

Re: OE breakages, was: Re: fakeroot fails to build

[Rodrigo Vivi]

> We maybe can add a tested on foo machines with git revision bar and distro x
> in version y to every commit.

Unfortunately it is so difficult to really test automatically every
commit because OE are receiving almost (more than?)  50 (please
correct if I'm wrong) commits by day and the right and more reliable
way to do that is every build from scratch to avoid have missing
dependencies that had already been built before.

> I think most developers are testing their commits but no one can test them
> in every possible combination.

Yes, it is difficult to test all combinations mainly because we work
with lots of machines and distros.

To do that we could maybe have a kind of distributed applicatioin
(scripts?) to build some distro for some machine spread all over the
world.

Let me explain (maunder?):

In the OE server we would have some kind of way to register the
machine that will do the tests for given distro and machine

i.e:

[koen-ip, angstrom, beagleboard]
[my-ip, mamona, nokia800]

Then, when a commit was received it would send notifications giving
the hash number to all remote clients that would put this commit in a
queue that would be built...
Maybe it is not necessary build from scrach everytime but once in a week (day?)
And the result would be sent back to the server to generate a report...

Well, but this test procedure would take so much time and maybe it by
itself doesn't prove anything because the commit has already been
done.
But maybe the reports could be used to guide the person that would
maintain the stable branch...

Anyway we can also deviate back this discussion to the commits and
change the commits policy. Have you already thought about to adopt a
commit policy like the linux kernel one?

Well don't take me bad (or serious) I'm just causing entropy to
generate a brainstorm where we can get ideas to make our life easier
and more "stable" ;)

-- 
Rodrigo Vivi
INdT - Instituto Nokia de Tecnologia
Blog: http://blog.vivi.eng.br
GPG: 0x905BE242 @ wwwkeys.pgp.net

Re: OE breakages, was: Re: fakeroot fails to build

[Tom Rini]

On Fri, Feb 27, 2009 at 06:56:16PM +0100, Frans Meulenbroeks wrote:
> 2009/2/27 Tom Rini <trini@kernel.crashing.org>:
> > On Fri, Feb 27, 2009 at 05:06:25PM +0100, Leon Woestenberg wrote:
> >> Hello,
> >>
> >> On Thu, Feb 26, 2009 at 6:20 PM, Frans Meulenbroeks
> >> <fransmeulenbroeks@gmail.com> wrote:
> >> >> On 25-02-09 16:14, Mike (mwester) wrote:
> >> >
> >> > The problem I am facing is somewhat different.
> >> > I've added packages which build and test fine in my environment but
> >> > for some reason they do not build properly on tinderbox.
> >> >
> >> That often indicates a leak to the host and/or missing DEPENDS.
> >>
> >> The build machines that run test builds, should do so with the minimal
> >> amount of host requirements.
> >>
> >> Easily testable using virtual machines with freshly installed Linux,
> >> as most machines grow their installed package base during their
> >> lifecycle.
> >
> > Also, chroots work for this.
> >
> 
> True but that is not what we have at the moment. E.g the mpd problem I
> mentioned earlier in this thread is probably caused by some docbook
> residue.
> 
> (and I agree that it is easy to miss a DEPENDS because of accumulated
> installed packages).

If you start with a very minimal chroot and an empty build output dir,
yes, you would catch these things.  I think scratchbox would help to
automate this kind of thing, but for my needs I simply unpack my chroot
base and go.

-- 
Tom Rini

Re: OE breakages, was: Re: fakeroot fails to build

[Frans Meulenbroeks]

2009/2/27 Tom Rini <trini@kernel.crashing.org>:
> On Fri, Feb 27, 2009 at 05:06:25PM +0100, Leon Woestenberg wrote:
>> Hello,
>>
>> On Thu, Feb 26, 2009 at 6:20 PM, Frans Meulenbroeks
>> <fransmeulenbroeks@gmail.com> wrote:
>> >> On 25-02-09 16:14, Mike (mwester) wrote:
>> >
>> > The problem I am facing is somewhat different.
>> > I've added packages which build and test fine in my environment but
>> > for some reason they do not build properly on tinderbox.
>> >
>> That often indicates a leak to the host and/or missing DEPENDS.
>>
>> The build machines that run test builds, should do so with the minimal
>> amount of host requirements.
>>
>> Easily testable using virtual machines with freshly installed Linux,
>> as most machines grow their installed package base during their
>> lifecycle.
>
> Also, chroots work for this.
>
> --
> Tom Rini

chroot and virtual machines indeed could help a lot.
Actually I think that chroot could be a big help porting pacakges. I
see too often that things go wrong because host files are being used
somewhere. Definitely some packages are an almost untangible knot of
problems (try to make a .bb fiel for e.g. cdrtools, I ended up moving
to cdrkit (probably better anyway, but cdrtools demonstrates the
problem quite nicely).

Not sure whether it will be easy to make a build env with chroot
(although I know debian does something like that).

Frans

Re: OE breakages, was: Re: fakeroot fails to build

[Tom Rini]

On Sat, Feb 28, 2009 at 11:54:46AM +0100, Frans Meulenbroeks wrote:
[snip]
> Not sure whether it will be easy to make a build env with chroot
> (although I know debian does something like that).

Having done this, yes.  It's basically using deboostrap to setup a new
system, then for Ubuntu use aptitude and mark everything auto-installed
except ubuntu-minimal and ubuntu-standard.  Then use (and update)
the OEAndYourDistro wiki page + bitbake to see what other host stuff you
must bring in.  Vanilla Debian would be, I think, just a different set
of meta packages.

-- 
Tom Rini

Re: OE breakages, was: Re: fakeroot fails to build

[Frans Meulenbroeks]

2009/2/28 Tom Rini <trini@kernel.crashing.org>:
> On Sat, Feb 28, 2009 at 11:54:46AM +0100, Frans Meulenbroeks wrote:
> [snip]
>> Not sure whether it will be easy to make a build env with chroot
>> (although I know debian does something like that).
>
> Having done this, yes.  It's basically using deboostrap to setup a new
> system, then for Ubuntu use aptitude and mark everything auto-installed
> except ubuntu-minimal and ubuntu-standard.  Then use (and update)
> the OEAndYourDistro wiki page + bitbake to see what other host stuff you
> must bring in.  Vanilla Debian would be, I think, just a different set
> of meta packages.
>
> --
> Tom Rini
>

I've used debootstrap in the past, but was not too sure whether it
could also do a cross build.
If it can I would say such a solution would make it a lot easier to
port packages.

What also could speed up starting from scratch is if you can download
a binary build environment.
Currently if you remove your tmp dir it takes hours and hours to build
things like gcc, glibc, etc etc

Frans.

Re: OE breakages, was: Re: fakeroot fails to build

[Tom Rini]

On Sat, Feb 28, 2009 at 07:07:55PM +0100, Frans Meulenbroeks wrote:
> 2009/2/28 Tom Rini <trini@kernel.crashing.org>:
> > On Sat, Feb 28, 2009 at 11:54:46AM +0100, Frans Meulenbroeks wrote:
> > [snip]
> >> Not sure whether it will be easy to make a build env with chroot
> >> (although I know debian does something like that).
> >
> > Having done this, yes.  It's basically using deboostrap to setup a new
> > system, then for Ubuntu use aptitude and mark everything auto-installed
> > except ubuntu-minimal and ubuntu-standard.  Then use (and update)
> > the OEAndYourDistro wiki page + bitbake to see what other host stuff you
> > must bring in.  Vanilla Debian would be, I think, just a different set
> > of meta packages.
> 
> I've used debootstrap in the past, but was not too sure whether it
> could also do a cross build.
> If it can I would say such a solution would make it a lot easier to
> port packages.

It sounds like you want something slightly different, a target env setup
on your host.  In a limited sense (ARM only) OpenBossa folks have done
this.

> What also could speed up starting from scratch is if you can download
> a binary build environment.
> Currently if you remove your tmp dir it takes hours and hours to build
> things like gcc, glibc, etc etc

Depends on your build box :)  I've been playing with PARALLEL_MAKE and
BB_NUMBER_THREADS and on a dual dual core box (2 dual core
processors) -j 4 / 4 gives me my SDK targets (Linux and MinGW) in under
2 hours.  I'm still tweaking the numbers however to see if I can get it
down to 90min.

Tangentally, packaged staging _can_ help here, but it's not I believe,
relocatible (unless you assume-provided some -native packages and
possibly we've fixed what we're doing wrong with gcc configure args).
IOW, if you build in, /OE/tmp you can't re-use the packaged staging
packages in /home/me/OE/tmp

-- 
Tom Rini

Re: OE breakages, was: Re: fakeroot fails to build

[Koen Kooi]

On 28-02-09 19:29, Tom Rini wrote:

> Tangentally, packaged staging _can_ help here, but it's not I believe,
> relocatible (unless you assume-provided some -native packages and
> possibly we've fixed what we're doing wrong with gcc configure args).
> IOW, if you build in, /OE/tmp you can't re-use the packaged staging
> packages in /home/me/OE/tmp

You can, packaged staging has 2 dirs: one for relocatable packages and 
one for non-relocatable packages. That's how I understand it.

regards,

Koen

Re: OE breakages, was: Re: fakeroot fails to build

[Tom Rini]

On Sun, Mar 01, 2009 at 06:09:14PM +0100, Koen Kooi wrote:
> On 28-02-09 19:29, Tom Rini wrote:
>
>> Tangentally, packaged staging _can_ help here, but it's not I believe,
>> relocatible (unless you assume-provided some -native packages and
>> possibly we've fixed what we're doing wrong with gcc configure args).
>> IOW, if you build in, /OE/tmp you can't re-use the packaged staging
>> packages in /home/me/OE/tmp
>
> You can, packaged staging has 2 dirs: one for relocatable packages and  
> one for non-relocatable packages. That's how I understand it.

Compiled for target packages and runs-on-host packages.  To me, that's a
non-relocatible solution.

-- 
Tom Rini