Re: sstate status

Re: sstate status

[Paul Eggleton]

On Wednesday 08 December 2010 11:41:22 you wrote:
> Do we still need those variable whitelisting changes? Based on original
> branch you pushed (basically RP's branch rebased on master imo), I think it
> basically works at least for a single recipe build though above weird issue
> exists for image build.
>
> Of course we need keep some bitbake specific variables in whitelist, such as
> DATE, TIME and several others you added in last commit.

Aren't many of those variables still exposed via preserved_envvars_list()?
 
> On the other hand, did you observe my issue on this new push? I want to
> understand its root cause, which is really funny. 

I have to admit I didn't look closely enough to be sure - I was mainly testing between two different build machines which don't share the same native arch, so the native packages were rebuilt anyway. I'm starting a fresh build now with my new branch to see if I can reproduce your results.

Cheers,
Paul

Re: sstate status

[Paul Eggleton]

On Wednesday 08 December 2010 14:02:15 Paul Eggleton wrote:
> I have to admit I didn't look closely enough to be sure - I was mainly
> testing between two different build machines which don't share the same
> native arch, so the native packages were rebuilt anyway. I'm starting a
> fresh build now with my new branch to see if I can reproduce your results.

FYI my build just finished and it's definitely rebuilding a whole bunch of packages (and not just native ones). More concerningly it seems to have rebuilt git-native and the hash has actually changed, although bitbake-diffsigs doesn't seem to be able to tell me what is different other than the hash itself.

Cheers,
Paul

Re: sstate status

[Tian, Kevin]

>From: Paul Eggleton [mailto:paul.eggleton@linux.intel.com]
>Sent: Wednesday, December 08, 2010 10:06 PM
>
>On Wednesday 08 December 2010 14:02:15 Paul Eggleton wrote:
>> I have to admit I didn't look closely enough to be sure - I was mainly
>> testing between two different build machines which don't share the same
>> native arch, so the native packages were rebuilt anyway. I'm starting a
>> fresh build now with my new branch to see if I can reproduce your results.
>
>FYI my build just finished and it's definitely rebuilding a whole bunch of packages (and not
>just native ones). More concerningly it seems to have rebuilt git-native and the hash has
>actually changed, although bitbake-diffsigs doesn't seem to be able to tell me what is
>different other than the hash itself.
>

It's a little different from my case, though I also observed lots of packages rebuilt
besides native ones. How did you check the hash has been changed? In my case git-native
simply overrides sstate package with same checksum.

Also I found that git-native.do_populate_setsecene is not scheduled at all, which may be
the reason for the rebuilt, but I haven't got the cause.

Thanks
Kevin

Re: sstate status

[Paul Eggleton]

On Wednesday 08 December 2010 14:15:45 Tian, Kevin wrote:
> It's a little different from my case, though I also observed lots of packages
> rebuilt besides native ones. 

This is what I saw as well.

> How did you check the hash has been changed? In my case git-native
> simply overrides sstate package with same checksum.

I hacked together a shell script that looks for .siginfo files with the same package/arch/task but a different hash; if it finds these then it uses bitbake-diffsigs to compare them.

As for the sstate packages being overwritten, I assume this is happening in my case as I can see the packages being rebuilt in the bitbake output.

> Also I found that git-native.do_populate_setsecene is not scheduled at all,
> which may be the reason for the rebuilt, but I haven't got the cause.

Hmm, then it could be unrelated to the general problem.

You didn't see this behaviour with your tk/sstate branch I presume?

Cheers,
Paul

Re: sstate status

[Tian, Kevin]

>From: Paul Eggleton [mailto:paul.eggleton@linux.intel.com]
>Sent: Wednesday, December 08, 2010 10:50 PM
>
>On Wednesday 08 December 2010 14:15:45 Tian, Kevin wrote:
>> It's a little different from my case, though I also observed lots of packages
>> rebuilt besides native ones.
>
>This is what I saw as well.
>
>> How did you check the hash has been changed? In my case git-native
>> simply overrides sstate package with same checksum.
>
>I hacked together a shell script that looks for .siginfo files with the same package/arch/task
>but a different hash; if it finds these then it uses bitbake-diffsigs to compare them.
>
>As for the sstate packages being overwritten, I assume this is happening in my case as I
>can see the packages being rebuilt in the bitbake output.
>
>> Also I found that git-native.do_populate_setsecene is not scheduled at all,
>> which may be the reason for the rebuilt, but I haven't got the cause.
>
>Hmm, then it could be unrelated to the general problem.
>
>You didn't see this behaviour with your tk/sstate branch I presume?
>

I think so. I just observed this behavior after my vacation last week. But it may also
come from other changes, not exactly from sstate related ones imo. There're some
bitbake changes in the middle I think.

Thanks
Kevin 

Re: sstate status

[Gary Thomas]

On 12/08/2010 07:06 AM, Paul Eggleton wrote:
> On Wednesday 08 December 2010 14:02:15 Paul Eggleton wrote:
>> I have to admit I didn't look closely enough to be sure - I was mainly
>> testing between two different build machines which don't share the same
>> native arch, so the native packages were rebuilt anyway. I'm starting a
>> fresh build now with my new branch to see if I can reproduce your results.
>
> FYI my build just finished and it's definitely rebuilding a whole bunch of packages (and not just native ones). More concerningly it seems to have rebuilt git-native and the hash has actually changed, although bitbake-diffsigs doesn't seem to be able to tell me what is different other than the hash itself.

I'm running my own experiments on this branch.  It seems that
something more fundamental is broken.  I first ran
   % bitbake poky-image-minimal
which seemed to go OK although the resulting image.tar.bz2 looks
surprisingly empty.  I've not investigated this fully.

Then in the same tree, nothing else touched, I ran
   % bitbake poky-image-sato
which generated a ton of messages like these:
   NOTE: package task-base-1.0-r69: task do_populate_sysroot_setscene: Started
   NOTE: Staging package /tmp/sstate_testing/sstate-cache/sstate-task-base-qemuarm-poky-linux-gnueabi-1.0-r69-armv5te-1-f2de7ef77366ff6028dbbf54a0206318_populate-sysroot.tgz does not
exist
   NOTE: package task-base-1.0-r69: task do_package_setscene: Started
   NOTE: Staging package /tmp/sstate_testing/sstate-cache/sstate-task-base-qemuarm-poky-linux-gnueabi-1.0-r69-armv5te-1-82eff67eab7faa49e63393c34ce384dd_package.tgz does not exist
etc.  I also have seen that if I abort this build, there are
literally hundreds of setscene tasks pending...

I've put the logs at
   http://www.mlbassoc.com/poky/poky-image-minimal-from-paule-branch
   http://www.mlbassoc.com/poky/poky-image-sato-from-paule-branch

Query: what sort of machine do you use for this testing?  I have a
Core-2 quad @2.4GHz with 3GB of RAM and it takes me around 5 hours
to make a full test.

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: sstate status

[Gary Thomas]

On 12/08/2010 07:41 AM, Gary Thomas wrote:
> On 12/08/2010 07:06 AM, Paul Eggleton wrote:
>> On Wednesday 08 December 2010 14:02:15 Paul Eggleton wrote:
>>> I have to admit I didn't look closely enough to be sure - I was mainly
>>> testing between two different build machines which don't share the same
>>> native arch, so the native packages were rebuilt anyway. I'm starting a
>>> fresh build now with my new branch to see if I can reproduce your results.
>>
>> FYI my build just finished and it's definitely rebuilding a whole bunch of packages (and not just native ones). More concerningly it seems to have rebuilt git-native and the hash
>> has actually changed, although bitbake-diffsigs doesn't seem to be able to tell me what is different other than the hash itself.
>
> I'm running my own experiments on this branch. It seems that
> something more fundamental is broken. I first ran
> % bitbake poky-image-minimal
> which seemed to go OK although the resulting image.tar.bz2 looks
> surprisingly empty. I've not investigated this fully.
>
> Then in the same tree, nothing else touched, I ran
> % bitbake poky-image-sato
> which generated a ton of messages like these:
> NOTE: package task-base-1.0-r69: task do_populate_sysroot_setscene: Started
> NOTE: Staging package /tmp/sstate_testing/sstate-cache/sstate-task-base-qemuarm-poky-linux-gnueabi-1.0-r69-armv5te-1-f2de7ef77366ff6028dbbf54a0206318_populate-sysroot.tgz does not
> exist
> NOTE: package task-base-1.0-r69: task do_package_setscene: Started
> NOTE: Staging package /tmp/sstate_testing/sstate-cache/sstate-task-base-qemuarm-poky-linux-gnueabi-1.0-r69-armv5te-1-82eff67eab7faa49e63393c34ce384dd_package.tgz does not exist
> etc. I also have seen that if I abort this build, there are
> literally hundreds of setscene tasks pending...
>
> I've put the logs at
> http://www.mlbassoc.com/poky/poky-image-minimal-from-paule-branch
> http://www.mlbassoc.com/poky/poky-image-sato-from-paule-branch

BTW you'll notice that the second build failed (it wasn't done when
I posted this originally):

NOTE: package portmap-6.0-r7: task do_install: Started
ERROR: Task failed: ('function do_install failed', '/tmp/sstate_testing/tmp/work/armv5te-poky-linux-gnueabi/portmap-6.0-r7/temp/log.do_install.26036')
ERROR: Logfile of failure stored in: /tmp/sstate_testing/tmp/work/armv5te-poky-linux-gnueabi/portmap-6.0-r7/temp/log.do_install.26036
Log data follows:
| NOTE: make -e MAKEFLAGS= install DESTDIR=/tmp/sstate_testing/tmp/work/armv5te-poky-linux-gnueabi/portmap-6.0-r7/image
| install -o root -g root -m 0755 portmap /tmp/sstate_testing/tmp/work/armv5te-poky-linux-gnueabi/portmap-6.0-r7/image/sbin
| install: cannot change ownership of `/tmp/sstate_testing/tmp/work/armv5te-poky-linux-gnueabi/portmap-6.0-r7/image/sbin/portmap': Operation not permitted
| make: *** [install] Error 1
| FATAL: oe_runmake failed
| ERROR: Task failed: ('function do_install failed', '/tmp/sstate_testing/tmp/work/armv5te-poky-linux-gnueabi/portmap-6.0-r7/temp/log.do_install.26036')
NOTE: package portmap-6.0-r7: task do_install: Failed

My build host is Fedora 13 with SELinux disabled.

> Query: what sort of machine do you use for this testing? I have a
> Core-2 quad @2.4GHz with 3GB of RAM and it takes me around 5 hours
> to make a full test.
>

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: sstate status

[Paul Eggleton]

On Wednesday 08 December 2010 14:41:11 you wrote:
> Query: what sort of machine do you use for this testing?  I have a
> Core-2 quad @2.4GHz with 3GB of RAM and it takes me around 5 hours
> to make a full test.

Well, it depends on what you use as a "full test". For most of my testing I just use poky-image-minimal, although I really ought to be doing poky-image-sato or something a bit more substantial on a regular basis. poky-image-minimal takes about 25 mins on my build machine (a Core i7 870 with 6GB RAM).

Cheers,
Paul

Re: sstate status

[Paul Eggleton]

On Wednesday 08 December 2010 15:55:57 Paul Eggleton wrote:
> poky-image-minimal takes about 25 mins on my build machine (a Core i7 870
> with 6GB RAM).

Actually that is somewhat of an underestimation. "time bitbake poky-image-minimal" tells me it actually takes about 57 minutes.

Cheers,
Paul

sstate status

[Paul Eggleton]

Hi all,

I've been doing some tests with sstate, and I've managed to get to a stage where I can share the SSTATE_DIR between my two machines (one running Fedora 14 and the other Kubuntu 10.10) and almost get a successful build from a clean TMPDIR. My last test got to poky-image-minimal.do_rootfs and then failed due to some kind of package dependency issue:

--------- snip ---------
| Processing rpm...
| Processing zypper...
| error: Failed dependencies:
|       elfutils >= 0.148 is needed by rpm-5.1.10-r7.i586
|       libaugeas0 >= 0.7.3 is needed by zypper-1.4.7+git0+9eb0e248e06c8d20ad054be2439149d9ede37531-r1.i586
| ERROR: Task failed: ('function do_rootfs failed', '/home/pokystuff/tmp/work/qemux86-poky-linux/poky-image-minimal-1.0-r0/temp/log.do_rootfs.6874')
--------- snip ---------

A "bitbake -c clean augeas elfutils" and then "bitbake poky-image-minimal" again went through OK (and built the aforementioned recipes from sstate) , so I'm not sure what went wrong here.

My current state of work is in the paule/sstate contrib branch, which is based on Kevin's tk/sstate branch. Kevin, I think the sooner we can move this towards master the better, then we can get some wider testing.

FYI Joshua has made bug #507 into a tracking bug for current sstate issues:
http://bugzilla.pokylinux.org/show_bug.cgi?id=507

Cheers,
Paul

Re: sstate status

[Richard Purdie]

Hi,

On Fri, 2010-12-03 at 16:34 +0000, Paul Eggleton wrote:
> I've been doing some tests with sstate, and I've managed to get to a
> stage where I can share the SSTATE_DIR between my two machines (one
> running Fedora 14 and the other Kubuntu 10.10) and almost get a
> successful build from a clean TMPDIR. My last test got to
> poky-image-minimal.do_rootfs and then failed due to some kind of
> package dependency issue:
> 
> --------- snip ---------
> | Processing rpm...
> | Processing zypper...
> | error: Failed dependencies:
> |       elfutils >= 0.148 is needed by rpm-5.1.10-r7.i586
> |       libaugeas0 >= 0.7.3 is needed by zypper-1.4.7+git0
> +9eb0e248e06c8d20ad054be2439149d9ede37531-r1.i586
> | ERROR: Task failed: ('function do_rootfs failed',
> '/home/pokystuff/tmp/work/qemux86-poky-linux/poky-image-minimal-1.0-r0/temp/log.do_rootfs.6874')
> --------- snip ---------
> 
> A "bitbake -c clean augeas elfutils" and then "bitbake
> poky-image-minimal" again went through OK (and built the
> aforementioned recipes from sstate) , so I'm not sure what went wrong
> here.
> 
> My current state of work is in the paule/sstate contrib branch, which
> is based on Kevin's tk/sstate branch. Kevin, I think the sooner we can
> move this towards master the better, then we can get some wider
> testing.
> 
> FYI Joshua has made bug #507 into a tracking bug for current sstate issues:
> http://bugzilla.pokylinux.org/show_bug.cgi?id=507

Kevin/Paul:

I notice a number of patches in your branches which shouldn't be needed
if we use the two patches at the head of: 

http://git.pokylinux.org/cgit/cgit.cgi/poky-contrib/log/?h=rpurdie/tweaks2

Could you please test using those two changes, or let me know if those
changes cause problems.

Cheers,

Richard

Re: sstate status

[Tian, Kevin]

>From: Richard Purdie
>Sent: Saturday, December 04, 2010 6:12 AM
>
>Hi,
>
>On Fri, 2010-12-03 at 16:34 +0000, Paul Eggleton wrote:
>> I've been doing some tests with sstate, and I've managed to get to a
>> stage where I can share the SSTATE_DIR between my two machines (one
>> running Fedora 14 and the other Kubuntu 10.10) and almost get a
>> successful build from a clean TMPDIR. My last test got to
>> poky-image-minimal.do_rootfs and then failed due to some kind of
>> package dependency issue:
>>
>> --------- snip ---------
>> | Processing rpm...
>> | Processing zypper...
>> | error: Failed dependencies:
>> |       elfutils >= 0.148 is needed by rpm-5.1.10-r7.i586
>> |       libaugeas0 >= 0.7.3 is needed by zypper-1.4.7+git0
>> +9eb0e248e06c8d20ad054be2439149d9ede37531-r1.i586
>> | ERROR: Task failed: ('function do_rootfs failed',
>>
>'/home/pokystuff/tmp/work/qemux86-poky-linux/poky-image-minimal-1.0-r0/temp/log.
>do_rootfs.6874')
>> --------- snip ---------
>>
>> A "bitbake -c clean augeas elfutils" and then "bitbake
>> poky-image-minimal" again went through OK (and built the
>> aforementioned recipes from sstate) , so I'm not sure what went wrong
>> here.
>>
>> My current state of work is in the paule/sstate contrib branch, which
>> is based on Kevin's tk/sstate branch. Kevin, I think the sooner we can
>> move this towards master the better, then we can get some wider
>> testing.
>>
>> FYI Joshua has made bug #507 into a tracking bug for current sstate issues:
>> http://bugzilla.pokylinux.org/show_bug.cgi?id=507
>
>Kevin/Paul:
>
>I notice a number of patches in your branches which shouldn't be needed
>if we use the two patches at the head of:
>
>http://git.pokylinux.org/cgit/cgit.cgi/poky-contrib/log/?h=rpurdie/tweaks2
>
>Could you please test using those two changes, or let me know if those
>changes cause problems.
>

Sure. However my remote machine in office is down unexpectedly. I'll verify it later
when going to office next week.

BTW, on my branch there're also some bug fixes out of environmental variables issue,
which could you take a look whether they're correct?

http://git.pokylinux.org/cgit/cgit.cgi/poky-contrib/commit/?h=tk/sstate&id=7a6fc816766ea925591a0cf6ae17383a953ae061, siggen.py: set 'runtaskdeps' correctly

http://git.pokylinux.org/cgit/cgit.cgi/poky-contrib/commit/?h=tk/sstate&id=0a4c46dc2c122ef5c1057380e79b75e302583b4a, siggen.py: fix python error when comparing sstate generated from different srcpath

http://git.pokylinux.org/cgit/cgit.cgi/poky-contrib/commit/?h=tk/sstate&id=7d6dd0b5717b565f478c88d53cf5e53e139f1141, siggen.py: fix the wrong usage on BB_TASKHASH_WHITELIST

Thanks
Kevin

Re: sstate status

[Richard Purdie]

On Sun, 2010-12-05 at 19:10 +0800, Tian, Kevin wrote:
> Sure. However my remote machine in office is down unexpectedly. I'll verify it later
> when going to office next week.
> 
> BTW, on my branch there're also some bug fixes out of environmental variables issue,
> which could you take a look whether they're correct?
> 
> http://git.pokylinux.org/cgit/cgit.cgi/poky-contrib/commit/?h=tk/sstate&id=7a6fc816766ea925591a0cf6ae17383a953ae061, siggen.py: set 'runtaskdeps' correctly

I've merged this one, thanks.

> http://git.pokylinux.org/cgit/cgit.cgi/poky-contrib/commit/?h=tk/sstate&id=0a4c46dc2c122ef5c1057380e79b75e302583b4a, siggen.py: fix python error when comparing sstate generated from different srcpath

I understand the need for this but I don't like the implementation :)

We can't use OEROOT here as its not a standard variable and is
deprecated. I was initially thinking POKYBASE but that isn't right
either in bitbake. Hmm.

Chris, any suggestions? I'm actually struggling a bit to come up with a
variable that represents a "head" of the metadata :/. Perhaps filename
would be enough in this case?

I'd also suggest we remove this path when we save the siginfo file, not
when we load it and do the comparison.

> http://git.pokylinux.org/cgit/cgit.cgi/poky-contrib/commit/?h=tk/sstate&id=7d6dd0b5717b565f478c88d53cf5e53e139f1141, siggen.py: fix the wrong usage on BB_TASKHASH_WHITELIST

Why the initial:

if self.twl and not self.twl.search(dataCache.pkg_fn[fn]):

?

I suspect it should just be:

dep_fn = re.search("(?P<fn>.*)\..*", dep).group('fn')
if self.twl.search(dataCache.pkg_fn[dep_fn]):
    #bb.note("Skipping %s" % dep)
    continue

?

For native/cross tasks, I still expect then to have dependencies and be
rebuilt if something changes, I just don't expect target packages to
change when native/cross ones do.

Cheers,

Richard

Re: sstate status

[Tian, Kevin]

>From: Richard Purdie [mailto:rpurdie@linux.intel.com]
>Sent: Monday, December 06, 2010 9:30 AM
>
>On Sun, 2010-12-05 at 19:10 +0800, Tian, Kevin wrote:
>> Sure. However my remote machine in office is down unexpectedly. I'll verify it later
>> when going to office next week.
>>
>> BTW, on my branch there're also some bug fixes out of environmental variables issue,
>> which could you take a look whether they're correct?
>>
>>
>http://git.pokylinux.org/cgit/cgit.cgi/poky-contrib/commit/?h=tk/sstate&id=7a6fc816766
>ea925591a0cf6ae17383a953ae061, siggen.py: set 'runtaskdeps' correctly
>
>I've merged this one, thanks.

Thanks

>
>>
>http://git.pokylinux.org/cgit/cgit.cgi/poky-contrib/commit/?h=tk/sstate&id=0a4c46dc2c1
>22ef5c1057380e79b75e302583b4a, siggen.py: fix python error when comparing sstate
>generated from different srcpath
>
>I understand the need for this but I don't like the implementation :)
>
>We can't use OEROOT here as its not a standard variable and is
>deprecated. I was initially thinking POKYBASE but that isn't right
>either in bitbake. Hmm.

Could you elaborate why POKYBASE can't be used too? Because LAYERDIR is a
one-time expansion action? If we're sure that POKYBASE is expanded before
base signature generation, then it could work.

>
>Chris, any suggestions? I'm actually struggling a bit to come up with a
>variable that represents a "head" of the metadata :/. Perhaps filename
>would be enough in this case?
>
>I'd also suggest we remove this path when we save the siginfo file, not
>when we load it and do the comparison.

Yes, that's a better approach. I'll go that approach.

>
>>
>http://git.pokylinux.org/cgit/cgit.cgi/poky-contrib/commit/?h=tk/sstate&id=7d6dd0b5717
>b565f478c88d53cf5e53e139f1141, siggen.py: fix the wrong usage on
>BB_TASKHASH_WHITELIST
>
>Why the initial:
>
>if self.twl and not self.twl.search(dataCache.pkg_fn[fn]):
>
>?
>
>I suspect it should just be:
>
>dep_fn = re.search("(?P<fn>.*)\..*", dep).group('fn')
>if self.twl.search(dataCache.pkg_fn[dep_fn]):
>    #bb.note("Skipping %s" % dep)
>    continue
>
>?
>
>For native/cross tasks, I still expect then to have dependencies and be
>rebuilt if something changes, I just don't expect target packages to
>change when native/cross ones do.
>

That's my intention too. Above logic is to filter out native/cross dependencies.
If we do that filter for native/cross tasks too, that means all dependencies 
would be wiped out which is not what we want since they all fall into the
match group. That's why I add the initial condition to restrict the filtering on
on target packages. :-)

Thanks
Kevin

Re: sstate status

[Richard Purdie]

On Tue, 2010-12-07 at 15:46 +0800, Tian, Kevin wrote: 
> >From: Richard Purdie [mailto:rpurdie@linux.intel.com]

> >http://git.pokylinux.org/cgit/cgit.cgi/poky-contrib/commit/?h=tk/sstate&id=0a4c46dc2c1
> >22ef5c1057380e79b75e302583b4a, siggen.py: fix python error when comparing sstate
> >generated from different srcpath
> >
> >I understand the need for this but I don't like the implementation :)
> >
> >We can't use OEROOT here as its not a standard variable and is
> >deprecated. I was initially thinking POKYBASE but that isn't right
> >either in bitbake. Hmm.
> 
> Could you elaborate why POKYBASE can't be used too? Because LAYERDIR is a
> one-time expansion action? If we're sure that POKYBASE is expanded before
> base signature generation, then it could work.

POKYBASE is poky specific and not a bitbake variable. Using it in
generic bitbake code is therefore a layer violation.

> >Chris, any suggestions? I'm actually struggling a bit to come up with a
> >variable that represents a "head" of the metadata :/. Perhaps filename
> >would be enough in this case?
> >
> >I'd also suggest we remove this path when we save the siginfo file, not
> >when we load it and do the comparison.
> 
> Yes, that's a better approach. I'll go that approach.

Thanks, I'll wait for an updated patch.

> That's my intention too. Above logic is to filter out native/cross dependencies.
> If we do that filter for native/cross tasks too, that means all dependencies 
> would be wiped out which is not what we want since they all fall into the
> match group. That's why I add the initial condition to restrict the filtering on
> on target packages. :-)

Ok, this makes sense. I merged it after adding in some comments to the
code.

I've asked Paul to pull together a branch with all the changes in for
sstate and see how we stand.


Cheers,

Richard

Re: sstate status

[Tian, Kevin]

>From: Richard Purdie [mailto:rpurdie@linux.intel.com]
>Sent: Tuesday, December 07, 2010 11:03 PM
>
>On Tue, 2010-12-07 at 15:46 +0800, Tian, Kevin wrote:
>> >From: Richard Purdie [mailto:rpurdie@linux.intel.com]
>
>> >http://git.pokylinux.org/cgit/cgit.cgi/poky-contrib/commit/?h=tk/sstate&id=0a4c46dc
>2c1
>> >22ef5c1057380e79b75e302583b4a, siggen.py: fix python error when comparing sstate
>> >generated from different srcpath
>> >
>> >I understand the need for this but I don't like the implementation :)
>> >
>> >We can't use OEROOT here as its not a standard variable and is
>> >deprecated. I was initially thinking POKYBASE but that isn't right
>> >either in bitbake. Hmm.
>>
>> Could you elaborate why POKYBASE can't be used too? Because LAYERDIR is a
>> one-time expansion action? If we're sure that POKYBASE is expanded before
>> base signature generation, then it could work.
>
>POKYBASE is poky specific and not a bitbake variable. Using it in
>generic bitbake code is therefore a layer violation.

Got it.

>
>> >Chris, any suggestions? I'm actually struggling a bit to come up with a
>> >variable that represents a "head" of the metadata :/. Perhaps filename
>> >would be enough in this case?
>> >
>> >I'd also suggest we remove this path when we save the siginfo file, not
>> >when we load it and do the comparison.
>>
>> Yes, that's a better approach. I'll go that approach.
>
>Thanks, I'll wait for an updated patch.
>
>> That's my intention too. Above logic is to filter out native/cross dependencies.
>> If we do that filter for native/cross tasks too, that means all dependencies
>> would be wiped out which is not what we want since they all fall into the
>> match group. That's why I add the initial condition to restrict the filtering on
>> on target packages. :-)
>
>Ok, this makes sense. I merged it after adding in some comments to the
>code.
>
>I've asked Paul to pull together a branch with all the changes in for
>sstate and see how we stand.
>
>

OK, I'll wait for that new branch to have another try.

Thanks
Kevin

Re: sstate status

[Paul Eggleton]

On Friday 03 December 2010 22:11:45 Richard Purdie wrote:
> I notice a number of patches in your branches which shouldn't be needed
> if we use the two patches at the head of: 
> 
> http://git.pokylinux.org/cgit/cgit.cgi/poky-contrib/log/?h=rpurdie/tweaks2
> 
> Could you please test using those two changes, or let me know if those
> changes cause problems.

I just tested with your branch; it definitely obviates the need to exclude large amounts of variables, however BB_TASKHASH is still being included in the hash and this causes many mismatches.

Cheers,
Paul

Re: sstate status

[Richard Purdie]

Hi Paul,

On Mon, 2010-12-06 at 16:06 +0000, Paul Eggleton wrote:
> On Friday 03 December 2010 22:11:45 Richard Purdie wrote:
> > I notice a number of patches in your branches which shouldn't be needed
> > if we use the two patches at the head of: 
> > 
> > http://git.pokylinux.org/cgit/cgit.cgi/poky-contrib/log/?h=rpurdie/tweaks2
> > 
> > Could you please test using those two changes, or let me know if those
> > changes cause problems.
> 
> I just tested with your branch; it definitely obviates the need to
> exclude large amounts of variables, however BB_TASKHASH is still being
> included in the hash and this causes many mismatches.

Agreed, I'll take that patch on its own, I just want to move carefully
to the end result.

Can you rebase your sstate branch onto master, pull in the above changes
not already in master and lets see where we are?

Cheers,

Richard

Re: sstate status

[Paul Eggleton]

On Tuesday 07 December 2010 12:46:36 Richard Purdie wrote:
> Agreed, I'll take that patch on its own, I just want to move carefully
> to the end result.
> 
> Can you rebase your sstate branch onto master, pull in the above changes
> not already in master and lets see where we are?

Done - pushed to contrib paule/sstate. I have omitted Kevin's bitbake-diffsigs srcpath fix which is still being worked on.

Cheers,
Paul

Re: sstate status

[Tian, Kevin]

>From: Paul Eggleton
>Sent: Wednesday, December 08, 2010 12:37 AM
>
>On Tuesday 07 December 2010 12:46:36 Richard Purdie wrote:
>> Agreed, I'll take that patch on its own, I just want to move carefully
>> to the end result.
>>
>> Can you rebase your sstate branch onto master, pull in the above changes
>> not already in master and lets see where we are?
>
>Done - pushed to contrib paule/sstate. I have omitted Kevin's bitbake-diffsigs srcpath fix
>which is still being worked on.
>

The weird behavior I saw yesterday is still there: lots of recipes (such as gzip-native) are
rebuilt when I build poky-image-minimal, which however finally ends up to override my
sstate cache with same checksums. 

If I end the build when it finished setscene stage, and then simply build tasks in the runqueue
such as gzip-native, git-native, . It just succeeds by reusing sstate package.

Quite strange... Still look into it now.

Thanks
Kevin

Re: sstate status

[Paul Eggleton]

On Wednesday 08 December 2010 11:22:40 Tian, Kevin wrote:
> The weird behavior I saw yesterday is still there: lots of recipes (such as gzip-native) are
> rebuilt when I build poky-image-minimal, which however finally ends up to override my
> sstate cache with same checksums. 
> 
> If I end the build when it finished setscene stage, and then simply build tasks in the runqueue
> such as gzip-native, git-native, . It just succeeds by reusing sstate package.
> 
> Quite strange... Still look into it now.

Oops, it appears that I pushed the wrong branch to paule/sstate yesterday. I've re-pushed a new one which actually includes the variable whitelisting changes.

Sorry about that!

Cheers,
Paul

Re: sstate status

[Tian, Kevin]

>From: Paul Eggleton [mailto:paul.eggleton@linux.intel.com]
>Sent: Wednesday, December 08, 2010 7:32 PM
>
>On Wednesday 08 December 2010 11:22:40 Tian, Kevin wrote:
>> The weird behavior I saw yesterday is still there: lots of recipes (such as gzip-native) are
>> rebuilt when I build poky-image-minimal, which however finally ends up to override my
>> sstate cache with same checksums.
>>
>> If I end the build when it finished setscene stage, and then simply build tasks in the
>runqueue
>> such as gzip-native, git-native, . It just succeeds by reusing sstate package.
>>
>> Quite strange... Still look into it now.
>
>Oops, it appears that I pushed the wrong branch to paule/sstate yesterday. I've re-pushed
>a new one which actually includes the variable whitelisting changes.

Do we still need those variable whitelisting changes? Based on original branch you pushed
(basically RP's branch rebased on master imo), I think it basically works at least for a single
recipe build though above weird issue exists for image build.

Of course we need keep some bitbake specific variables in whitelist, such as DATE, TIME and
several others you added in last commit.

On the other hand, did you observe my issue on this new push? I want to understand its
root cause, which is really funny. 

>
>Sorry about that!
>

No problem. :-)

Thanks
Kevin

Re: sstate status

[Gary Thomas]

On 12/03/2010 09:34 AM, Paul Eggleton wrote:
> Hi all,
>
> I've been doing some tests with sstate, and I've managed to get to a stage where I can share the SSTATE_DIR between my two machines (one running Fedora 14 and the other Kubuntu 10.10) and almost get a successful build from a clean TMPDIR. My last test got to poky-image-minimal.do_rootfs and then failed due to some kind of package dependency issue:
>
> --------- snip ---------
> | Processing rpm...
> | Processing zypper...
> | error: Failed dependencies:
> |       elfutils>= 0.148 is needed by rpm-5.1.10-r7.i586
> |       libaugeas0>= 0.7.3 is needed by zypper-1.4.7+git0+9eb0e248e06c8d20ad054be2439149d9ede37531-r1.i586
> | ERROR: Task failed: ('function do_rootfs failed', '/home/pokystuff/tmp/work/qemux86-poky-linux/poky-image-minimal-1.0-r0/temp/log.do_rootfs.6874')
> --------- snip ---------
>
> A "bitbake -c clean augeas elfutils" and then "bitbake poky-image-minimal" again went through OK (and built the aforementioned recipes from sstate) , so I'm not sure what went wrong here.
>
> My current state of work is in the paule/sstate contrib branch, which is based on Kevin's tk/sstate branch. Kevin, I think the sooner we can move this towards master the better, then we can get some wider testing.
>
> FYI Joshua has made bug #507 into a tracking bug for current sstate issues:
> http://bugzilla.pokylinux.org/show_bug.cgi?id=507

I gave this a simple try and it failed miserably.  Working from a copy of
Paul's branch in poky-contrib, here's what I did:

* On machine A (x86 Fedora 11)
   % bitbake poky-image-minimal

* On machine B (x86 Fedora 13)
   -- Copy sstate-cache from machine A to /tmp/sstate-cache (rsync with
      all perms & timestamps preserved)
   -- Using the identical conf/local.conf file from machine A, except
      I set up SSTATE_MIRRORS like this
          SSTATE_MIRRORS ?= "\
          file://.* file:///tmp/sstate-cache/"
   % bitbake poky-image-minimal

Fell apart pretty badly.  Look at the output
   http://www.mlbassoc.com/poky/poky_with_sstate.out

Also, it seemed to be rebuilding a lot of packages.  IMO, there's
no reason it should have to rebuild _any_

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: sstate status

[Gary Thomas]

On 12/03/2010 04:31 PM, Gary Thomas wrote:
> On 12/03/2010 09:34 AM, Paul Eggleton wrote:
>> Hi all,
>>
>> I've been doing some tests with sstate, and I've managed to get to a
>> stage where I can share the SSTATE_DIR between my two machines (one
>> running Fedora 14 and the other Kubuntu 10.10) and almost get a
>> successful build from a clean TMPDIR. My last test got to
>> poky-image-minimal.do_rootfs and then failed due to some kind of
>> package dependency issue:
>>
>> --------- snip ---------
>> | Processing rpm...
>> | Processing zypper...
>> | error: Failed dependencies:
>> | elfutils>= 0.148 is needed by rpm-5.1.10-r7.i586
>> | libaugeas0>= 0.7.3 is needed by
>> zypper-1.4.7+git0+9eb0e248e06c8d20ad054be2439149d9ede37531-r1.i586
>> | ERROR: Task failed: ('function do_rootfs failed',
>> '/home/pokystuff/tmp/work/qemux86-poky-linux/poky-image-minimal-1.0-r0/temp/log.do_rootfs.6874')
>>
>> --------- snip ---------
>>
>> A "bitbake -c clean augeas elfutils" and then "bitbake
>> poky-image-minimal" again went through OK (and built the
>> aforementioned recipes from sstate) , so I'm not sure what went wrong
>> here.
>>
>> My current state of work is in the paule/sstate contrib branch, which
>> is based on Kevin's tk/sstate branch. Kevin, I think the sooner we can
>> move this towards master the better, then we can get some wider testing.
>>
>> FYI Joshua has made bug #507 into a tracking bug for current sstate
>> issues:
>> http://bugzilla.pokylinux.org/show_bug.cgi?id=507
>
> I gave this a simple try and it failed miserably. Working from a copy of
> Paul's branch in poky-contrib, here's what I did:
>
> * On machine A (x86 Fedora 11)
> % bitbake poky-image-minimal
>
> * On machine B (x86 Fedora 13)
> -- Copy sstate-cache from machine A to /tmp/sstate-cache (rsync with
> all perms & timestamps preserved)
> -- Using the identical conf/local.conf file from machine A, except
> I set up SSTATE_MIRRORS like this
> SSTATE_MIRRORS ?= "\
> file://.* file:///tmp/sstate-cache/"
> % bitbake poky-image-minimal
>
> Fell apart pretty badly. Look at the output
> http://www.mlbassoc.com/poky/poky_with_sstate.out
>
> Also, it seemed to be rebuilding a lot of packages. IMO, there's
> no reason it should have to rebuild _any_
>

One thing that seems to be significant is the base build directory path.
In the case above, they did not match, i.e.
   * On machine A, BUILDDIR=/tmp/sstate_testing
   * On machine B, BUILDDIR=/home/local/my_sstate_test
I tried it this way as it matches my [customer's] expected usage.

I've rerun the experiment again where I used identical paths on both
machines and the results were vastly improved.  It still takes a horrible
length of time (64 minutes on a fast 4 processor box) and rebuilds way to
many packages for just replaying what's already been done though (IMO).
Results are at
   http://www.mlbassoc.com/poky/poky_with_sstate_same_paths.out

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: sstate status

[Tian, Kevin]

>From: Gary Thomas
>Sent: Saturday, December 04, 2010 8:48 PM
>
>On 12/03/2010 04:31 PM, Gary Thomas wrote:
>> On 12/03/2010 09:34 AM, Paul Eggleton wrote:
>>> Hi all,
>>>
>>> I've been doing some tests with sstate, and I've managed to get to a
>>> stage where I can share the SSTATE_DIR between my two machines (one
>>> running Fedora 14 and the other Kubuntu 10.10) and almost get a
>>> successful build from a clean TMPDIR. My last test got to
>>> poky-image-minimal.do_rootfs and then failed due to some kind of
>>> package dependency issue:
>>>
>>> --------- snip ---------
>>> | Processing rpm...
>>> | Processing zypper...
>>> | error: Failed dependencies:
>>> | elfutils>= 0.148 is needed by rpm-5.1.10-r7.i586
>>> | libaugeas0>= 0.7.3 is needed by
>>> zypper-1.4.7+git0+9eb0e248e06c8d20ad054be2439149d9ede37531-r1.i586
>>> | ERROR: Task failed: ('function do_rootfs failed',
>>>
>'/home/pokystuff/tmp/work/qemux86-poky-linux/poky-image-minimal-1.0-r0/temp/log.
>do_rootfs.6874')
>>>
>>> --------- snip ---------
>>>
>>> A "bitbake -c clean augeas elfutils" and then "bitbake
>>> poky-image-minimal" again went through OK (and built the
>>> aforementioned recipes from sstate) , so I'm not sure what went wrong
>>> here.
>>>
>>> My current state of work is in the paule/sstate contrib branch, which
>>> is based on Kevin's tk/sstate branch. Kevin, I think the sooner we can
>>> move this towards master the better, then we can get some wider testing.
>>>
>>> FYI Joshua has made bug #507 into a tracking bug for current sstate
>>> issues:
>>> http://bugzilla.pokylinux.org/show_bug.cgi?id=507
>>
>> I gave this a simple try and it failed miserably. Working from a copy of
>> Paul's branch in poky-contrib, here's what I did:
>>
>> * On machine A (x86 Fedora 11)
>> % bitbake poky-image-minimal
>>
>> * On machine B (x86 Fedora 13)
>> -- Copy sstate-cache from machine A to /tmp/sstate-cache (rsync with
>> all perms & timestamps preserved)
>> -- Using the identical conf/local.conf file from machine A, except
>> I set up SSTATE_MIRRORS like this
>> SSTATE_MIRRORS ?= "\
>> file://.* file:///tmp/sstate-cache/"
>> % bitbake poky-image-minimal
>>
>> Fell apart pretty badly. Look at the output
>> http://www.mlbassoc.com/poky/poky_with_sstate.out
>>
>> Also, it seemed to be rebuilding a lot of packages. IMO, there's
>> no reason it should have to rebuild _any_
>>
>
>One thing that seems to be significant is the base build directory path.
>In the case above, they did not match, i.e.
>   * On machine A, BUILDDIR=/tmp/sstate_testing
>   * On machine B, BUILDDIR=/home/local/my_sstate_test
>I tried it this way as it matches my [customer's] expected usage.

I think in usual case the directory paths would mismatch. So let's stick to that
configuration to expose more issues. In my previous debug, I have made it
in a good shape but would now turn to RP's branch to try more experiments.

It would be good if you also try that one:
http://git.pokylinux.org/cgit/cgit.cgi/poky-contrib/log/?h=rpurdie/tweaks2

>
>I've rerun the experiment again where I used identical paths on both
>machines and the results were vastly improved.  It still takes a horrible
>length of time (64 minutes on a fast 4 processor box) and rebuilds way to
>many packages for just replaying what's already been done though (IMO).
>Results are at
>   http://www.mlbassoc.com/poky/poky_with_sstate_same_paths.out
>

In my run ~25min are consumed just for sstate check/installation on a NHM 4-core box. 
then several minutes for rootfs generation if two builds totally match. the slow sstate
phase comes from same reason as a slow build - the 'exec' overhead. I use 'sato' as
the target.

In your case, however there're still some packages rebuilt, such as gzip-native,
unifdef-native, eglibc-initial, ncurses, etc. and also gcc related packages are rebuilt 
which just consumes time. Has your source tree changed between the two builds?

Thanks
Kevin

Re: sstate status

[Gary Thomas]

On 12/05/2010 04:19 AM, Tian, Kevin wrote:
>> From: Gary Thomas
>> Sent: Saturday, December 04, 2010 8:48 PM
>>
>> On 12/03/2010 04:31 PM, Gary Thomas wrote:
>>> On 12/03/2010 09:34 AM, Paul Eggleton wrote:
>>>> Hi all,
>>>>
>>>> I've been doing some tests with sstate, and I've managed to get to a
>>>> stage where I can share the SSTATE_DIR between my two machines (one
>>>> running Fedora 14 and the other Kubuntu 10.10) and almost get a
>>>> successful build from a clean TMPDIR. My last test got to
>>>> poky-image-minimal.do_rootfs and then failed due to some kind of
>>>> package dependency issue:
>>>>
>>>> --------- snip ---------
>>>> | Processing rpm...
>>>> | Processing zypper...
>>>> | error: Failed dependencies:
>>>> | elfutils>= 0.148 is needed by rpm-5.1.10-r7.i586
>>>> | libaugeas0>= 0.7.3 is needed by
>>>> zypper-1.4.7+git0+9eb0e248e06c8d20ad054be2439149d9ede37531-r1.i586
>>>> | ERROR: Task failed: ('function do_rootfs failed',
>>>>
>> '/home/pokystuff/tmp/work/qemux86-poky-linux/poky-image-minimal-1.0-r0/temp/log.
>> do_rootfs.6874')
>>>>
>>>> --------- snip ---------
>>>>
>>>> A "bitbake -c clean augeas elfutils" and then "bitbake
>>>> poky-image-minimal" again went through OK (and built the
>>>> aforementioned recipes from sstate) , so I'm not sure what went wrong
>>>> here.
>>>>
>>>> My current state of work is in the paule/sstate contrib branch, which
>>>> is based on Kevin's tk/sstate branch. Kevin, I think the sooner we can
>>>> move this towards master the better, then we can get some wider testing.
>>>>
>>>> FYI Joshua has made bug #507 into a tracking bug for current sstate
>>>> issues:
>>>> http://bugzilla.pokylinux.org/show_bug.cgi?id=507
>>>
>>> I gave this a simple try and it failed miserably. Working from a copy of
>>> Paul's branch in poky-contrib, here's what I did:
>>>
>>> * On machine A (x86 Fedora 11)
>>> % bitbake poky-image-minimal
>>>
>>> * On machine B (x86 Fedora 13)
>>> -- Copy sstate-cache from machine A to /tmp/sstate-cache (rsync with
>>> all perms&  timestamps preserved)
>>> -- Using the identical conf/local.conf file from machine A, except
>>> I set up SSTATE_MIRRORS like this
>>> SSTATE_MIRRORS ?= "\
>>> file://.* file:///tmp/sstate-cache/"
>>> % bitbake poky-image-minimal
>>>
>>> Fell apart pretty badly. Look at the output
>>> http://www.mlbassoc.com/poky/poky_with_sstate.out
>>>
>>> Also, it seemed to be rebuilding a lot of packages. IMO, there's
>>> no reason it should have to rebuild _any_
>>>
>>
>> One thing that seems to be significant is the base build directory path.
>> In the case above, they did not match, i.e.
>>    * On machine A, BUILDDIR=/tmp/sstate_testing
>>    * On machine B, BUILDDIR=/home/local/my_sstate_test
>> I tried it this way as it matches my [customer's] expected usage.
>
> I think in usual case the directory paths would mismatch. So let's stick to that
> configuration to expose more issues. In my previous debug, I have made it
> in a good shape but would now turn to RP's branch to try more experiments.
>
> It would be good if you also try that one:
> http://git.pokylinux.org/cgit/cgit.cgi/poky-contrib/log/?h=rpurdie/tweaks2
>
>>
>> I've rerun the experiment again where I used identical paths on both
>> machines and the results were vastly improved.  It still takes a horrible
>> length of time (64 minutes on a fast 4 processor box) and rebuilds way to
>> many packages for just replaying what's already been done though (IMO).
>> Results are at
>>    http://www.mlbassoc.com/poky/poky_with_sstate_same_paths.out
>>
>
> In my run ~25min are consumed just for sstate check/installation on a NHM 4-core box.
> then several minutes for rootfs generation if two builds totally match. the slow sstate
> phase comes from same reason as a slow build - the 'exec' overhead. I use 'sato' as
> the target.
>
> In your case, however there're still some packages rebuilt, such as gzip-native,
> unifdef-native, eglibc-initial, ncurses, etc. and also gcc related packages are rebuilt
> which just consumes time. Has your source tree changed between the two builds?

Do, identical source trees for all three runs.

I'll try it again later today with Richard's tree.

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: sstate status

[Gary Thomas]

On 12/05/2010 05:02 AM, Gary Thomas wrote:
> On 12/05/2010 04:19 AM, Tian, Kevin wrote:
>>> From: Gary Thomas
>>> Sent: Saturday, December 04, 2010 8:48 PM
>>>
>>> On 12/03/2010 04:31 PM, Gary Thomas wrote:
>>>> On 12/03/2010 09:34 AM, Paul Eggleton wrote:
>>>>> Hi all,
>>>>>
>>>>> I've been doing some tests with sstate, and I've managed to get to a
>>>>> stage where I can share the SSTATE_DIR between my two machines (one
>>>>> running Fedora 14 and the other Kubuntu 10.10) and almost get a
>>>>> successful build from a clean TMPDIR. My last test got to
>>>>> poky-image-minimal.do_rootfs and then failed due to some kind of
>>>>> package dependency issue:
>>>>>
>>>>> --------- snip ---------
>>>>> | Processing rpm...
>>>>> | Processing zypper...
>>>>> | error: Failed dependencies:
>>>>> | elfutils>= 0.148 is needed by rpm-5.1.10-r7.i586
>>>>> | libaugeas0>= 0.7.3 is needed by
>>>>> zypper-1.4.7+git0+9eb0e248e06c8d20ad054be2439149d9ede37531-r1.i586
>>>>> | ERROR: Task failed: ('function do_rootfs failed',
>>>>>
>>> '/home/pokystuff/tmp/work/qemux86-poky-linux/poky-image-minimal-1.0-r0/temp/log.
>>> do_rootfs.6874')
>>>>>
>>>>> --------- snip ---------
>>>>>
>>>>> A "bitbake -c clean augeas elfutils" and then "bitbake
>>>>> poky-image-minimal" again went through OK (and built the
>>>>> aforementioned recipes from sstate) , so I'm not sure what went wrong
>>>>> here.
>>>>>
>>>>> My current state of work is in the paule/sstate contrib branch, which
>>>>> is based on Kevin's tk/sstate branch. Kevin, I think the sooner we can
>>>>> move this towards master the better, then we can get some wider testing.
>>>>>
>>>>> FYI Joshua has made bug #507 into a tracking bug for current sstate
>>>>> issues:
>>>>> http://bugzilla.pokylinux.org/show_bug.cgi?id=507
>>>>
>>>> I gave this a simple try and it failed miserably. Working from a copy of
>>>> Paul's branch in poky-contrib, here's what I did:
>>>>
>>>> * On machine A (x86 Fedora 11)
>>>> % bitbake poky-image-minimal
>>>>
>>>> * On machine B (x86 Fedora 13)
>>>> -- Copy sstate-cache from machine A to /tmp/sstate-cache (rsync with
>>>> all perms& timestamps preserved)
>>>> -- Using the identical conf/local.conf file from machine A, except
>>>> I set up SSTATE_MIRRORS like this
>>>> SSTATE_MIRRORS ?= "\
>>>> file://.* file:///tmp/sstate-cache/"
>>>> % bitbake poky-image-minimal
>>>>
>>>> Fell apart pretty badly. Look at the output
>>>> http://www.mlbassoc.com/poky/poky_with_sstate.out
>>>>
>>>> Also, it seemed to be rebuilding a lot of packages. IMO, there's
>>>> no reason it should have to rebuild _any_
>>>>
>>>
>>> One thing that seems to be significant is the base build directory path.
>>> In the case above, they did not match, i.e.
>>> * On machine A, BUILDDIR=/tmp/sstate_testing
>>> * On machine B, BUILDDIR=/home/local/my_sstate_test
>>> I tried it this way as it matches my [customer's] expected usage.
>>
>> I think in usual case the directory paths would mismatch. So let's stick to that
>> configuration to expose more issues. In my previous debug, I have made it
>> in a good shape but would now turn to RP's branch to try more experiments.
>>
>> It would be good if you also try that one:
>> http://git.pokylinux.org/cgit/cgit.cgi/poky-contrib/log/?h=rpurdie/tweaks2
>>
>>>
>>> I've rerun the experiment again where I used identical paths on both
>>> machines and the results were vastly improved. It still takes a horrible
>>> length of time (64 minutes on a fast 4 processor box) and rebuilds way to
>>> many packages for just replaying what's already been done though (IMO).
>>> Results are at
>>> http://www.mlbassoc.com/poky/poky_with_sstate_same_paths.out
>>>
>>
>> In my run ~25min are consumed just for sstate check/installation on a NHM 4-core box.
>> then several minutes for rootfs generation if two builds totally match. the slow sstate
>> phase comes from same reason as a slow build - the 'exec' overhead. I use 'sato' as
>> the target.
>>
>> In your case, however there're still some packages rebuilt, such as gzip-native,
>> unifdef-native, eglibc-initial, ncurses, etc. and also gcc related packages are rebuilt
>> which just consumes time. Has your source tree changed between the two builds?
>
> No, identical source trees for all three runs.
>
> I'll try it again later today with Richard's tree.
>

I've now rerun this with Richard's tweak2 changes, building in completely
different tree paths.  While it did work, I don't see the point as it took
even longer the second time (using the sstate cache) and generated just as
much work :-(  Something is still not working as desired.

Original 'bitbake poky-image-minimal' into /tmp/sstate_testing
   time - 111 minutes
   space - 22.9 GB (in tmp)

Rerun, using the sstate-cache from the first run
   time - 114 minutes
   space - 21.7 GB

I think this last line from the rerun test is telling:
   NOTE: Tasks Summary: Attempted 1621 tasks of which 282 didn't need to be rerun and 0 failed.

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: sstate status

[Tian, Kevin]

>From: Gary Thomas [mailto:gary@mlbassoc.com]
>Sent: Monday, December 06, 2010 7:16 PM
>
>On 12/05/2010 05:02 AM, Gary Thomas wrote:
>> On 12/05/2010 04:19 AM, Tian, Kevin wrote:
>>>> From: Gary Thomas
>>>> Sent: Saturday, December 04, 2010 8:48 PM
>>>>
>>>> On 12/03/2010 04:31 PM, Gary Thomas wrote:
>>>>> On 12/03/2010 09:34 AM, Paul Eggleton wrote:
>>>>>> Hi all,
>>>>>>
>>>>>> I've been doing some tests with sstate, and I've managed to get to a
>>>>>> stage where I can share the SSTATE_DIR between my two machines (one
>>>>>> running Fedora 14 and the other Kubuntu 10.10) and almost get a
>>>>>> successful build from a clean TMPDIR. My last test got to
>>>>>> poky-image-minimal.do_rootfs and then failed due to some kind of
>>>>>> package dependency issue:
>>>>>>
>>>>>> --------- snip ---------
>>>>>> | Processing rpm...
>>>>>> | Processing zypper...
>>>>>> | error: Failed dependencies:
>>>>>> | elfutils>= 0.148 is needed by rpm-5.1.10-r7.i586
>>>>>> | libaugeas0>= 0.7.3 is needed by
>>>>>> zypper-1.4.7+git0+9eb0e248e06c8d20ad054be2439149d9ede37531-r1.i586
>>>>>> | ERROR: Task failed: ('function do_rootfs failed',
>>>>>>
>>>>
>'/home/pokystuff/tmp/work/qemux86-poky-linux/poky-image-minimal-1.0-r0/temp/log.
>>>> do_rootfs.6874')
>>>>>>
>>>>>> --------- snip ---------
>>>>>>
>>>>>> A "bitbake -c clean augeas elfutils" and then "bitbake
>>>>>> poky-image-minimal" again went through OK (and built the
>>>>>> aforementioned recipes from sstate) , so I'm not sure what went wrong
>>>>>> here.
>>>>>>
>>>>>> My current state of work is in the paule/sstate contrib branch, which
>>>>>> is based on Kevin's tk/sstate branch. Kevin, I think the sooner we can
>>>>>> move this towards master the better, then we can get some wider testing.
>>>>>>
>>>>>> FYI Joshua has made bug #507 into a tracking bug for current sstate
>>>>>> issues:
>>>>>> http://bugzilla.pokylinux.org/show_bug.cgi?id=507
>>>>>
>>>>> I gave this a simple try and it failed miserably. Working from a copy of
>>>>> Paul's branch in poky-contrib, here's what I did:
>>>>>
>>>>> * On machine A (x86 Fedora 11)
>>>>> % bitbake poky-image-minimal
>>>>>
>>>>> * On machine B (x86 Fedora 13)
>>>>> -- Copy sstate-cache from machine A to /tmp/sstate-cache (rsync with
>>>>> all perms& timestamps preserved)
>>>>> -- Using the identical conf/local.conf file from machine A, except
>>>>> I set up SSTATE_MIRRORS like this
>>>>> SSTATE_MIRRORS ?= "\
>>>>> file://.* file:///tmp/sstate-cache/"
>>>>> % bitbake poky-image-minimal
>>>>>
>>>>> Fell apart pretty badly. Look at the output
>>>>> http://www.mlbassoc.com/poky/poky_with_sstate.out
>>>>>
>>>>> Also, it seemed to be rebuilding a lot of packages. IMO, there's
>>>>> no reason it should have to rebuild _any_
>>>>>
>>>>
>>>> One thing that seems to be significant is the base build directory path.
>>>> In the case above, they did not match, i.e.
>>>> * On machine A, BUILDDIR=/tmp/sstate_testing
>>>> * On machine B, BUILDDIR=/home/local/my_sstate_test
>>>> I tried it this way as it matches my [customer's] expected usage.
>>>
>>> I think in usual case the directory paths would mismatch. So let's stick to that
>>> configuration to expose more issues. In my previous debug, I have made it
>>> in a good shape but would now turn to RP's branch to try more experiments.
>>>
>>> It would be good if you also try that one:
>>> http://git.pokylinux.org/cgit/cgit.cgi/poky-contrib/log/?h=rpurdie/tweaks2
>>>
>>>>
>>>> I've rerun the experiment again where I used identical paths on both
>>>> machines and the results were vastly improved. It still takes a horrible
>>>> length of time (64 minutes on a fast 4 processor box) and rebuilds way to
>>>> many packages for just replaying what's already been done though (IMO).
>>>> Results are at
>>>> http://www.mlbassoc.com/poky/poky_with_sstate_same_paths.out
>>>>
>>>
>>> In my run ~25min are consumed just for sstate check/installation on a NHM 4-core box.
>>> then several minutes for rootfs generation if two builds totally match. the slow sstate
>>> phase comes from same reason as a slow build - the 'exec' overhead. I use 'sato' as
>>> the target.
>>>
>>> In your case, however there're still some packages rebuilt, such as gzip-native,
>>> unifdef-native, eglibc-initial, ncurses, etc. and also gcc related packages are rebuilt
>>> which just consumes time. Has your source tree changed between the two builds?
>>
>> No, identical source trees for all three runs.
>>
>> I'll try it again later today with Richard's tree.
>>
>
>I've now rerun this with Richard's tweak2 changes, building in completely
>different tree paths.  While it did work, I don't see the point as it took
>even longer the second time (using the sstate cache) and generated just as
>much work :-(  Something is still not working as desired.
>
>Original 'bitbake poky-image-minimal' into /tmp/sstate_testing
>   time - 111 minutes
>   space - 22.9 GB (in tmp)
>
>Rerun, using the sstate-cache from the first run
>   time - 114 minutes
>   space - 21.7 GB
>
>I think this last line from the rerun test is telling:
>   NOTE: Tasks Summary: Attempted 1621 tasks of which 282 didn't need to be rerun and
>0 failed.

I have " Attempted 1621 tasks of which 953 didn't need to be rerun and 0 failed."
It should come from the fact that I use same source directory this time. I have one fix
for difference source tree in my previous branch, which is still under discussion with RP
and not in RP's branch yet. So sorry for inconvenience. I should raise the info earlier.

Although that, I took a quick look of my build. The thing is a bit weird. It's similar to
your previous result with Paul's branch, that many recipes are rebuilt though they're
claimed to be accelerable, such as gzip-native, unifdef-native, etc. The result is that
new sstate packages are generated to simply override old one. They DO have same
checksum because only one version exists with new timestamp.

I didn't look into recent commits yet. It looks that the checksum calculated at setscene
stage is different than the one finally written into sstate package.

Thanks
Kevin

Re: sstate status

[Tian, Kevin]

>From: Paul Eggleton [mailto:paul.eggleton@linux.intel.com]
>Sent: Saturday, December 04, 2010 12:35 AM
>
>Hi all,
>
>I've been doing some tests with sstate, and I've managed to get to a stage where I can
>share the SSTATE_DIR between my two machines (one running Fedora 14 and the other
>Kubuntu 10.10) and almost get a successful build from a clean TMPDIR. My last test got to
>poky-image-minimal.do_rootfs and then failed due to some kind of package dependency
>issue:
>
>--------- snip ---------
>| Processing rpm...
>| Processing zypper...
>| error: Failed dependencies:
>|       elfutils >= 0.148 is needed by rpm-5.1.10-r7.i586
>|       libaugeas0 >= 0.7.3 is needed by
>zypper-1.4.7+git0+9eb0e248e06c8d20ad054be2439149d9ede37531-r1.i586
>| ERROR: Task failed: ('function do_rootfs failed',
>'/home/pokystuff/tmp/work/qemux86-poky-linux/poky-image-minimal-1.0-r0/temp/log.
>do_rootfs.6874')
>--------- snip ---------
>
>A "bitbake -c clean augeas elfutils" and then "bitbake poky-image-minimal" again went
>through OK (and built the aforementioned recipes from sstate) , so I'm not sure what went
>wrong here.
>
>My current state of work is in the paule/sstate contrib branch, which is based on Kevin's
>tk/sstate branch. Kevin, I think the sooner we can move this towards master the better,
>then we can get some wider testing.

Thanks for your fixes. I made some mistakes when occasionally rebase and smash my 
commits with master and it's good that you caught them. I agree that we need make it
into master as early as possible. I saw Richard has pushed some different enhancements.
Let's give it some tests then.

Thanks
Kevin

>
>FYI Joshua has made bug #507 into a tracking bug for current sstate issues:
>http://bugzilla.pokylinux.org/show_bug.cgi?id=507
>