a new problem with sstate

a new problem with sstate

[Tian, Kevin]

When looking into the problem why some prebuilts can't be reused, I've struggled with
another issue these days. It's more severe because the 2nd build can't even succeed
when prebuilt is used.

The failure happened randomly on eglibc, eglibc-initial, and gcc-runtime. The error log
shows that either gcc libraries or eglibc headers are not correctly installed when building
those recipes. This first led me to think about potential dependency problem among 
those recipes. However this only happens when prebuilt is used. A fresh build just 
succeeds.

Finally it turns out from two factors:

	o sstate.bbclass has special handling about -initial and -intermediate recipes. If 
a complete (e.g. gcc-cross or eglibc) setscene function has been invoked already, 
installation for those special sstate packages is skipped, while still marked as 
accelerate-able

	o eglibc and gcc-runtime happen to have new changes between two builds and
thus require rebuild

Then the problem comes:
	- eglibc depends on gcc-cross-intermediate
	- gcc-cross-intermediate is marked as accelerated but no sstate installation actually
happens, which is simply a nop
	- gcc-cross is assumed to provide required bits, which depends on eglibc instead
and thus hasn't been started

Then eglibc compilation fails because libgcc is missing

So the key problem is about libc/gcc bootstrap process. We have plan in 1.0 to improve
bootstrap process, by having initial/intermediate bits installed to its own locations, which
should solve this problem too. However before that feature is ready, we still need
some workaround to have current sstate/prebuilt really usable, since it's very likely to
have eglibc/gcc recipe revised occasionally.

I have three options in mind:

	a) disable special handling in sstate.bbclass for -initial and -intermediate recipes, i.e.
always have all sstate packages installed. I think it's perhaps OK, but it may increase 
build time a bit and I may overlook something behind

	b) have the whole libc/gcc bootstrap process as a bundle when handling the sstate
task, i.e. any one stage failing to reuse prebuilt results in all relevant recipes excluding
from the acceleration list. However bitbake now executes tasks in  separate threads.
So I'm not sure how easily that cross-recipe info may be retrieved there. Perhaps some
database and postfunc is required

	c) always touch all gcc/libc recipes if any of them is changed to ensure all recipes
rebuilt from scratch. This however increases build time a lot and not convenient

I personally prefer to a), but need more experiments to ensure it working correctly,
but would like to get your comments too. :-)

Thanks
Kevin

Re: a new problem with sstate

[Joshua Lock]

On Wed, 2010-11-17 at 21:21 +0800, Tian, Kevin wrote:
> When looking into the problem why some prebuilts can't be reused, I've struggled with
> another issue these days. It's more severe because the 2nd build can't even succeed
> when prebuilt is used.
> 
> The failure happened randomly on eglibc, eglibc-initial, and gcc-runtime. The error log
> shows that either gcc libraries or eglibc headers are not correctly installed when building
> those recipes. This first led me to think about potential dependency problem among 
> those recipes. However this only happens when prebuilt is used. A fresh build just 
> succeeds.
> 
> Finally it turns out from two factors:
> 
> 	o sstate.bbclass has special handling about -initial and -intermediate recipes. If 
> a complete (e.g. gcc-cross or eglibc) setscene function has been invoked already, 
> installation for those special sstate packages is skipped, while still marked as 
> accelerate-able

The special casing in the sstate class was added to work around the fact
that the toolchain bootstrap overwrites some pieces of the sysroot
(bug#239).

http://bugzilla.yoctoproject.org/show_bug.cgi?id=239

If this is still not working is there anything stopping us from
expanding the workaround Richard started in
ecf2eb1efa145d5c8f350697ec605ea58beb9ba7 as a temporary fix until bug
239 is fixed?

http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=ecf2eb1efa145d5c8f350697ec605ea58beb9ba7

Cheers,
Joshua
-- 
Joshua Lock
        Intel Open Source Technology Centre

Re: a new problem with sstate

[Tom Rini]

On 11/17/2010 07:28 AM, Joshua Lock wrote:
> On Wed, 2010-11-17 at 21:21 +0800, Tian, Kevin wrote:
>> When looking into the problem why some prebuilts can't be reused, I've struggled with
>> another issue these days. It's more severe because the 2nd build can't even succeed
>> when prebuilt is used.
>>
>> The failure happened randomly on eglibc, eglibc-initial, and gcc-runtime. The error log
>> shows that either gcc libraries or eglibc headers are not correctly installed when building
>> those recipes. This first led me to think about potential dependency problem among
>> those recipes. However this only happens when prebuilt is used. A fresh build just
>> succeeds.
>>
>> Finally it turns out from two factors:
>>
>> 	o sstate.bbclass has special handling about -initial and -intermediate recipes. If
>> a complete (e.g. gcc-cross or eglibc) setscene function has been invoked already,
>> installation for those special sstate packages is skipped, while still marked as
>> accelerate-able
>
> The special casing in the sstate class was added to work around the fact
> that the toolchain bootstrap overwrites some pieces of the sysroot
> (bug#239).
>
> http://bugzilla.yoctoproject.org/show_bug.cgi?id=239
>
> If this is still not working is there anything stopping us from
> expanding the workaround Richard started in
> ecf2eb1efa145d5c8f350697ec605ea58beb9ba7 as a temporary fix until bug
> 239 is fixed?
>
> http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=ecf2eb1efa145d5c8f350697ec605ea58beb9ba7

With old style packaged-staging we workaround this by doing:

# We want to be certain that the scene is set for us only after it's set for
# our dependencies, to avoid problems with pstage package install order.
do_setscene[deptask] = "do_setscene"

Can something similar be done for the sstate way of the world?

-- 
Tom Rini
Mentor Graphics Corporation

Re: a new problem with sstate

[Tian, Kevin]

>From: Joshua Lock
>Sent: Wednesday, November 17, 2010 10:29 PM
>
>On Wed, 2010-11-17 at 21:21 +0800, Tian, Kevin wrote:
>> When looking into the problem why some prebuilts can't be reused, I've struggled with
>> another issue these days. It's more severe because the 2nd build can't even succeed
>> when prebuilt is used.
>>
>> The failure happened randomly on eglibc, eglibc-initial, and gcc-runtime. The error log
>> shows that either gcc libraries or eglibc headers are not correctly installed when building
>> those recipes. This first led me to think about potential dependency problem among
>> those recipes. However this only happens when prebuilt is used. A fresh build just
>> succeeds.
>>
>> Finally it turns out from two factors:
>>
>> 	o sstate.bbclass has special handling about -initial and -intermediate recipes. If
>> a complete (e.g. gcc-cross or eglibc) setscene function has been invoked already,
>> installation for those special sstate packages is skipped, while still marked as
>> accelerate-able
>
>The special casing in the sstate class was added to work around the fact
>that the toolchain bootstrap overwrites some pieces of the sysroot
>(bug#239).
>
>http://bugzilla.yoctoproject.org/show_bug.cgi?id=239

Yes, I can see the reason for that special case.

>
>If this is still not working is there anything stopping us from
>expanding the workaround Richard started in
>ecf2eb1efa145d5c8f350697ec605ea58beb9ba7 as a temporary fix until bug
>239 is fixed?
>
>http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=ecf2eb1efa145d5c8f350697ec6
>05ea58beb9ba7
>

I think that works for most case, if there's no change to libc/gcc bootstrap recipes.
If there happens to be one or several changed, the bootstrap process is actually
broken since some of them may be a null installation while marked finished. This
allows other tasks depending on them to continue but finally failed to find necessary
bits.

I'm thinking about expanding that. It'd be good to not use prebuilt for all libc/gcc
bootstrap recipes once any of them is changed. That way the normal bootstrap process
is started from scratch. However I haven't found a right way to achieve that yet. :/

Thanks
Kevin

Re: a new problem with sstate

[Tian, Kevin]

>From: Tom Rini
>Sent: Wednesday, November 17, 2010 10:31 PM
>
>On 11/17/2010 07:28 AM, Joshua Lock wrote:
>> On Wed, 2010-11-17 at 21:21 +0800, Tian, Kevin wrote:
>>> When looking into the problem why some prebuilts can't be reused, I've struggled with
>>> another issue these days. It's more severe because the 2nd build can't even succeed
>>> when prebuilt is used.
>>>
>>> The failure happened randomly on eglibc, eglibc-initial, and gcc-runtime. The error log
>>> shows that either gcc libraries or eglibc headers are not correctly installed when building
>>> those recipes. This first led me to think about potential dependency problem among
>>> those recipes. However this only happens when prebuilt is used. A fresh build just
>>> succeeds.
>>>
>>> Finally it turns out from two factors:
>>>
>>> 	o sstate.bbclass has special handling about -initial and -intermediate recipes. If
>>> a complete (e.g. gcc-cross or eglibc) setscene function has been invoked already,
>>> installation for those special sstate packages is skipped, while still marked as
>>> accelerate-able
>>
>> The special casing in the sstate class was added to work around the fact
>> that the toolchain bootstrap overwrites some pieces of the sysroot
>> (bug#239).
>>
>> http://bugzilla.yoctoproject.org/show_bug.cgi?id=239
>>
>> If this is still not working is there anything stopping us from
>> expanding the workaround Richard started in
>> ecf2eb1efa145d5c8f350697ec605ea58beb9ba7 as a temporary fix until bug
>> 239 is fixed?
>>
>>
>http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=ecf2eb1efa145d5c8f350697ec6
>05ea58beb9ba7
>
>With old style packaged-staging we workaround this by doing:
>
># We want to be certain that the scene is set for us only after it's set for
># our dependencies, to avoid problems with pstage package install order.
>do_setscene[deptask] = "do_setscene"
>
>Can something similar be done for the sstate way of the world?
>

Good to know that. I need to look into the detail to understand whether it
works for the purpose here. :-)

Thanks
Kevin

Re: a new problem with sstate

[Tian, Kevin]

>From: Tian, Kevin
>Sent: Thursday, November 18, 2010 1:16 PM
>
>>From: Tom Rini
>>Sent: Wednesday, November 17, 2010 10:31 PM
>>
>>On 11/17/2010 07:28 AM, Joshua Lock wrote:
>>> On Wed, 2010-11-17 at 21:21 +0800, Tian, Kevin wrote:
>>>> When looking into the problem why some prebuilts can't be reused, I've struggled with
>>>> another issue these days. It's more severe because the 2nd build can't even succeed
>>>> when prebuilt is used.
>>>>
>>>> The failure happened randomly on eglibc, eglibc-initial, and gcc-runtime. The error log
>>>> shows that either gcc libraries or eglibc headers are not correctly installed when
>building
>>>> those recipes. This first led me to think about potential dependency problem among
>>>> those recipes. However this only happens when prebuilt is used. A fresh build just
>>>> succeeds.
>>>>
>>>> Finally it turns out from two factors:
>>>>
>>>> 	o sstate.bbclass has special handling about -initial and -intermediate recipes. If
>>>> a complete (e.g. gcc-cross or eglibc) setscene function has been invoked already,
>>>> installation for those special sstate packages is skipped, while still marked as
>>>> accelerate-able
>>>
>>> The special casing in the sstate class was added to work around the fact
>>> that the toolchain bootstrap overwrites some pieces of the sysroot
>>> (bug#239).
>>>
>>> http://bugzilla.yoctoproject.org/show_bug.cgi?id=239
>>>
>>> If this is still not working is there anything stopping us from
>>> expanding the workaround Richard started in
>>> ecf2eb1efa145d5c8f350697ec605ea58beb9ba7 as a temporary fix until bug
>>> 239 is fixed?
>>>
>>>
>>http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=ecf2eb1efa145d5c8f350697e
>c6
>>05ea58beb9ba7
>>
>>With old style packaged-staging we workaround this by doing:
>>
>># We want to be certain that the scene is set for us only after it's set for
>># our dependencies, to avoid problems with pstage package install order.
>>do_setscene[deptask] = "do_setscene"
>>
>>Can something similar be done for the sstate way of the world?
>>
>
>Good to know that. I need to look into the detail to understand whether it
>works for the purpose here. :-)
>

I think this would work. With the dependency considered, then the sstate setscene
tasks become sorted. And then we could add a check to see whether previous 
libc/gcc bootstrap stages succeed on sstate installation. If any fails due to rebuild 
required, then later stages are not marked accelerable which will be rebuilt too.

However my desktop was powered down unexpectedly last night, which triggers
other issues in sstate whitelist handling which makes whole prebuilt not usable
any more after reboot. I'll need to fix that first and then try your suggestion later.

Thanks
Kevin

Re: a new problem with sstate

[Richard Purdie]

On Wed, 2010-11-17 at 07:31 -0700, Tom Rini wrote:
> With old style packaged-staging we workaround this by doing:
> 
> # We want to be certain that the scene is set for us only after it's set for
> # our dependencies, to avoid problems with pstage package install order.
> do_setscene[deptask] = "do_setscene"
> 
> Can something similar be done for the sstate way of the world?

Unfortunately not. sstate differs as the code tries to be clever about
the dependencies and reverse walks the dependency tree. This means that
if:

A => B => C => D => E

where "=>" means "depends on" then sstate will try and install a sstate
package for A, then B, then C and *stop* once it finds a match.

This fixes one of the annoyances of packaged-staging where if you're
building an image and have all the ipks installed, it would still
install the cross toolchain.

Cheers,

Richard

Re: a new problem with sstate

[Tom Rini]

On 11/28/2010 05:20 AM, Richard Purdie wrote:
> On Wed, 2010-11-17 at 07:31 -0700, Tom Rini wrote:
>> With old style packaged-staging we workaround this by doing:
>>
>> # We want to be certain that the scene is set for us only after it's set for
>> # our dependencies, to avoid problems with pstage package install order.
>> do_setscene[deptask] = "do_setscene"
>>
>> Can something similar be done for the sstate way of the world?
>
> Unfortunately not. sstate differs as the code tries to be clever about
> the dependencies and reverse walks the dependency tree. This means that
> if:
>
> A =>  B =>  C =>  D =>  E
>
> where "=>" means "depends on" then sstate will try and install a sstate
> package for A, then B, then C and *stop* once it finds a match.
>
> This fixes one of the annoyances of packaged-staging where if you're
> building an image and have all the ipks installed, it would still
> install the cross toolchain.

Er, so you're saying sstate has a 3 steps back limit?  Or am I just 
misunderstanding the example?  That would sound like other things are a 
bit hard to get done with sstate?  Or should stuff like bitbake 
virtual/kernel ; bitbake -c menuconfig virtual/kernel ; bitbake 
virtual/kernel (roughly...) still work?

-- 
Tom Rini
Mentor Graphics Corporation

Re: a new problem with sstate

[Richard Purdie]

On Sun, 2010-11-28 at 15:12 -0700, Tom Rini wrote:
> On 11/28/2010 05:20 AM, Richard Purdie wrote:
> > On Wed, 2010-11-17 at 07:31 -0700, Tom Rini wrote:
> >> With old style packaged-staging we workaround this by doing:
> >>
> >> # We want to be certain that the scene is set for us only after it's set for
> >> # our dependencies, to avoid problems with pstage package install order.
> >> do_setscene[deptask] = "do_setscene"
> >>
> >> Can something similar be done for the sstate way of the world?
> >
> > Unfortunately not. sstate differs as the code tries to be clever about
> > the dependencies and reverse walks the dependency tree. This means that
> > if:
> >
> > A =>  B =>  C =>  D =>  E
> >
> > where "=>" means "depends on" then sstate will try and install a sstate
> > package for A, then B, then C and *stop* once it finds a match.
> >
> > This fixes one of the annoyances of packaged-staging where if you're
> > building an image and have all the ipks installed, it would still
> > install the cross toolchain.
> 
> Er, so you're saying sstate has a 3 steps back limit?  Or am I just 
> misunderstanding the example?

You're misunderstanding it :/. It could stop at any point it finds a
match, I just happened to pick three steps.

Putting it a different way, my point was that there is a "setscene" per
task but the setscene dependencies represent those of the "parent" task,
but are searched backwards compared to the normal logic. So setscene
tasks don't have dependencies in their own right, its linked to the
parent task the represent.

>   That would sound like other things are a 
> bit hard to get done with sstate?  Or should stuff like bitbake 
> virtual/kernel ; bitbake -c menuconfig virtual/kernel ; bitbake 
> virtual/kernel (roughly...) still work?

This would all work just fine...

Cheers,

Richard

Re: a new problem with sstate

[Tom Rini]

On 11/28/2010 04:06 PM, Richard Purdie wrote:
> On Sun, 2010-11-28 at 15:12 -0700, Tom Rini wrote:
>> On 11/28/2010 05:20 AM, Richard Purdie wrote:
>>> On Wed, 2010-11-17 at 07:31 -0700, Tom Rini wrote:
>>>> With old style packaged-staging we workaround this by doing:
>>>>
>>>> # We want to be certain that the scene is set for us only after it's set for
>>>> # our dependencies, to avoid problems with pstage package install order.
>>>> do_setscene[deptask] = "do_setscene"
>>>>
>>>> Can something similar be done for the sstate way of the world?
>>>
>>> Unfortunately not. sstate differs as the code tries to be clever about
>>> the dependencies and reverse walks the dependency tree. This means that
>>> if:
>>>
>>> A =>   B =>   C =>   D =>   E
>>>
>>> where "=>" means "depends on" then sstate will try and install a sstate
>>> package for A, then B, then C and *stop* once it finds a match.
>>>
>>> This fixes one of the annoyances of packaged-staging where if you're
>>> building an image and have all the ipks installed, it would still
>>> install the cross toolchain.
>>
>> Er, so you're saying sstate has a 3 steps back limit?  Or am I just
>> misunderstanding the example?
>
> You're misunderstanding it :/. It could stop at any point it finds a
> match, I just happened to pick three steps.

Figured.

> Putting it a different way, my point was that there is a "setscene" per
> task but the setscene dependencies represent those of the "parent" task,
> but are searched backwards compared to the normal logic. So setscene
> tasks don't have dependencies in their own right, its linked to the
> parent task the represent.

Right.  So, to restate the problem, in pstaging we have the problem of 
the gcc/glibc dance not forcing themselves to be run forward in the 
correct order, but we can fix that with some manual deps.  But in the 
sstate of the world the problem is that we might find the wrong thing to 
provide what we want and pick the wrong gcc or glibc sstate (and get a 
-initial rather than finial)?

-- 
Tom Rini
Mentor Graphics Corporation

Re: a new problem with sstate

[Richard Purdie]

On Sun, 2010-11-28 at 16:21 -0700, Tom Rini wrote:
> > Putting it a different way, my point was that there is a "setscene" per
> > task but the setscene dependencies represent those of the "parent" task,
> > but are searched backwards compared to the normal logic. So setscene
> > tasks don't have dependencies in their own right, its linked to the
> > parent task the represent.
> 
> Right.  So, to restate the problem, in pstaging we have the problem of 
> the gcc/glibc dance not forcing themselves to be run forward in the 
> correct order, but we can fix that with some manual deps.  But in the 
> sstate of the world the problem is that we might find the wrong thing to 
> provide what we want and pick the wrong gcc or glibc sstate (and get a 
> -initial rather than finial)?

Without the change Kevin was talking about, it would always install the
gcc and glibc sstate but usually backwards, i.e. it would install glibc,
then overwrite it with glibc-initial, similarly, gcc-cross would be
overwritten with gcc-cross-intermediate which in turn would be
overwritten by gcc-cross-initial.

These things cause us pain in other ways, e.g. trying to bitbake
gcc-cross-intermediate and smashing the toolchain to pieces. or bumping
PR on glibc and glibc-initial and watching things break.

I maintain a belief that any file in the sysroots should come from one
recipe only and that we should fix the toolchain bootstrap to use
multiple sysroots as intermediate steps. Hopefully once that is done,
this problem goes away along with all the other related problems.

Cheers,

Richard