Poky Project Discussions
 help / color / mirror / Atom feed
* update about SSTATE status
@ 2010-11-21 13:02 Tian, Kevin
  2010-11-21 18:17 ` Richard Purdie
  0 siblings, 1 reply; 5+ messages in thread
From: Tian, Kevin @ 2010-11-21 13:02 UTC (permalink / raw)
  To: poky@pokylinux.org

Hi, all,

Recently I took some time to try out SSTATE prebuilt feature, and now I think it's time
to summarize issues I've seen and also some enhancements I've made. All the changes
are currently in poky-contrib:tk/sstate branch, which are still under test and on a slightly
old commit (at Nov.13).
http://git.pokylinux.org/cgit/cgit.cgi/poky-contrib/log/?h=tk/sstate

I use the sstate_cache generated from the 1st build as the SSTATE_DIR for the 2nd 
build, and then check the SSTATE use status in 2nd build. Most issues below I think
apply to a MIRROR mode too.

I've observed SSTATE broken in below scenarios:

a) ~31 recipes fail to reuse sstate packages when the 1st build and the 2nd build are
continuously build, when using same source tree
https://lists.yoctoproject.org/pipermail/poky/2010-November/000174.html

b) more recipes fail if the 2nd build was started in next day after when the 1st build
was generated. This is related to DATE/TIME variables referenced by some tasks.

c) My build machine suffered from a power down, and then all STATE packages can't
be reused after reboot. Comparison between two builds show that some shell 
variables change, e.g: GNOME_KEYRING_SOCKET, SSH_AUTH_SOCK, etc.

d) no STATE package from the 1st build could be reused when the two builds use
different source trees, though two source trees are identical with only difference on
the path. This is related to OEROOT

e) If there're changes on libc/gcc packages between the 1st build and the 2nd build,
the 2nd build will always fail, either on gcc-runtime or eglibc, or other libc/gcc
bootstrap packages. The reason is that sstate_installation skips -initial/-intermediate
packages which breaks bootstrap process.
https://lists.yoctoproject.org/pipermail/poky/2010-November/000270.html

The major problem for above issues is that many undesired variables which are
bound to build environment are counted into signature generation. Bitbake takes
a list of environmental variables from the shell and then convert into internal
variables. We need to separate those from bitbake internal variables. So I split
BB_ENV_WHITELIST and BB_ENV_EXTRAWHITE into two groups:

	BB_ENV_WHITELIST_BB/BB_ENV_EXTRAWHITE_BB
	BB_ENV_WHITELIST_SHELL/BB_ENV_EXTRAWHITE_SHELL

The 1st group covers variables defined internally in bitbake, while the other group
points to the true external environmental variables. This way we can always wipe
the _SHELL group from the dependency chain, while use BB_BASEHASH_WHITELIST
to control bitbake variables:
http://git.pokylinux.org/cgit/cgit.cgi/poky-contrib/commit/?h=tk/sstate&id=8e1b6a292b3c796c5c0273de76d2e83271c83e26

Above looks somewhat ugly, but it should improve current situation a lot. We
do need a better way to mark such variable attribute efficiently though.

In the meantime, there're other two enhancements which are also helpful:

  - update-alternative expands an absolute path when appending to do_install, 
which impacts signature:
http://git.pokylinux.org/cgit/cgit.cgi/poky-contrib/commit/?h=tk/sstate&id=a09df20442bcfc033b955992948e2dc2f6cbb0b4

  - BBCLASSEXTEND is not covered by BB_TASKHASH_WHITELIST, i.e 
"virtual:native:" and "virtual:nativesdk:":
http://git.pokylinux.org/cgit/cgit.cgi/poky-contrib/commit/?h=tk/sstate&id=07697ff94ff1829ef16effb48c73a2312af6a2ff

The remaining thing in the list:

- more tests on my branch
- rebase to latest master
- address incremental build failure when libc/gcc recipes are changed. Tom Rini
gave a suggestion to use a similar 'deptask' as in previous stage. I need more
thought on that and not sure whether it can solve it completely. If there's no clean 
way, we may wait until libc/gcc bootstrap process is changed (sysroot to different 
places) which is a 1.0 task, and then suggest people to always wipe out related 
sstate packages when seeing those errors

Comments are welcomed.

Thanks
Kevin


^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: update about SSTATE status
  2010-11-21 13:02 update about SSTATE status Tian, Kevin
@ 2010-11-21 18:17 ` Richard Purdie
  2010-11-22  7:45   ` Tian, Kevin
  0 siblings, 1 reply; 5+ messages in thread
From: Richard Purdie @ 2010-11-21 18:17 UTC (permalink / raw)
  To: Tian, Kevin; +Cc: poky@pokylinux.org

Hi Kevin,

On Sun, 2010-11-21 at 21:02 +0800, Tian, Kevin wrote:
> Recently I took some time to try out SSTATE prebuilt feature, and now I think it's time
> to summarize issues I've seen and also some enhancements I've made. All the changes
> are currently in poky-contrib:tk/sstate branch, which are still under test and on a slightly
> old commit (at Nov.13).
> http://git.pokylinux.org/cgit/cgit.cgi/poky-contrib/log/?h=tk/sstate
> 
> I use the sstate_cache generated from the 1st build as the SSTATE_DIR for the 2nd 
> build, and then check the SSTATE use status in 2nd build. Most issues below I think
> apply to a MIRROR mode too.

This is great analysis, thanks!

> I've observed SSTATE broken in below scenarios:
> 
> a) ~31 recipes fail to reuse sstate packages when the 1st build and the 2nd build are
> continuously build, when using same source tree
> https://lists.yoctoproject.org/pipermail/poky/2010-November/000174.html
> 
> b) more recipes fail if the 2nd build was started in next day after when the 1st build
> was generated. This is related to DATE/TIME variables referenced by some tasks.

I'd be interested to see which recipes are doing this...

> c) My build machine suffered from a power down, and then all STATE packages can't
> be reused after reboot. Comparison between two builds show that some shell 
> variables change, e.g: GNOME_KEYRING_SOCKET, SSH_AUTH_SOCK, etc.

We should whitelist the ones we don't mind if they change.

> d) no STATE package from the 1st build could be reused when the two builds use
> different source trees, though two source trees are identical with only difference on
> the path. This is related to OEROOT

I was sure I tested this and the checksums matched :/. Which variables
are changing to cause this - we really need to track them down...

> e) If there're changes on libc/gcc packages between the 1st build and the 2nd build,
> the 2nd build will always fail, either on gcc-runtime or eglibc, or other libc/gcc
> bootstrap packages. The reason is that sstate_installation skips -initial/-intermediate
> packages which breaks bootstrap process.
> https://lists.yoctoproject.org/pipermail/poky/2010-November/000270.html
> 
> The major problem for above issues is that many undesired variables which are
> bound to build environment are counted into signature generation. Bitbake takes
> a list of environmental variables from the shell and then convert into internal
> variables. We need to separate those from bitbake internal variables. So I split
> BB_ENV_WHITELIST and BB_ENV_EXTRAWHITE into two groups:
> 
> 	BB_ENV_WHITELIST_BB/BB_ENV_EXTRAWHITE_BB
> 	BB_ENV_WHITELIST_SHELL/BB_ENV_EXTRAWHITE_SHELL
> 
> The 1st group covers variables defined internally in bitbake, while the other group
> points to the true external environmental variables. This way we can always wipe
> the _SHELL group from the dependency chain, while use BB_BASEHASH_WHITELIST
> to control bitbake variables:
> http://git.pokylinux.org/cgit/cgit.cgi/poky-contrib/commit/?h=tk/sstate&id=8e1b6a292b3c796c5c0273de76d2e83271c83e26
> 
> Above looks somewhat ugly, but it should improve current situation a lot. We
> do need a better way to mark such variable attribute efficiently though.

I'm afraid I don't like this solution, its too complicated. Why don't we
just whitelist the variables we are happy shouldn't influence the
checksums even if they are present in the environment?

To be honest there are several of these we should only export to
"interactive" tasks anyway so perhaps marking some as unexport and then
exporting them in devshell and the interactive patch handler would be a
better approach?

> In the meantime, there're other two enhancements which are also helpful:
> 
>   - update-alternative expands an absolute path when appending to do_install, 
> which impacts signature:
> http://git.pokylinux.org/cgit/cgit.cgi/poky-contrib/commit/?h=tk/sstate&id=a09df20442bcfc033b955992948e2dc2f6cbb0b4

Yes, this looks correct, nice catch!

>   - BBCLASSEXTEND is not covered by BB_TASKHASH_WHITELIST, i.e 
> "virtual:native:" and "virtual:nativesdk:":
> http://git.pokylinux.org/cgit/cgit.cgi/poky-contrib/commit/?h=tk/sstate&id=07697ff94ff1829ef16effb48c73a2312af6a2ff

Again, good catch!

I've merged these two.

> The remaining thing in the list:
> 
> - more tests on my branch
> - rebase to latest master
> - address incremental build failure when libc/gcc recipes are changed. Tom Rini
> gave a suggestion to use a similar 'deptask' as in previous stage. I need more
> thought on that and not sure whether it can solve it completely. If there's no clean 
> way, we may wait until libc/gcc bootstrap process is changed (sysroot to different 
> places) which is a 1.0 task, and then suggest people to always wipe out related 
> sstate packages when seeing those errors

Poky's sstate code workd very differently to packaged staging, I don't
think dependencies can help us in the sstate case. We perhaps just need
to improve the logic somehow. I need to think about the problem in more
detail to come up with better direction suggestions.

Cheers,

Richard



^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: update about SSTATE status
  2010-11-21 18:17 ` Richard Purdie
@ 2010-11-22  7:45   ` Tian, Kevin
  2010-11-22 11:29     ` Tian, Kevin
  0 siblings, 1 reply; 5+ messages in thread
From: Tian, Kevin @ 2010-11-22  7:45 UTC (permalink / raw)
  To: Richard Purdie; +Cc: poky@pokylinux.org

>From: Richard Purdie [mailto:rpurdie@linux.intel.com]
>Sent: Monday, November 22, 2010 2:17 AM
>
>Hi Kevin,
>
>On Sun, 2010-11-21 at 21:02 +0800, Tian, Kevin wrote:
>> Recently I took some time to try out SSTATE prebuilt feature, and now I think it's time
>> to summarize issues I've seen and also some enhancements I've made. All the changes
>> are currently in poky-contrib:tk/sstate branch, which are still under test and on a slightly
>> old commit (at Nov.13).
>> http://git.pokylinux.org/cgit/cgit.cgi/poky-contrib/log/?h=tk/sstate
>>
>> I use the sstate_cache generated from the 1st build as the SSTATE_DIR for the 2nd
>> build, and then check the SSTATE use status in 2nd build. Most issues below I think
>> apply to a MIRROR mode too.
>
>This is great analysis, thanks!

I'd like to suggest "bitbake -S" which is really great to figure out the very cause for the
signature difference. It did take me quite some time to dump debug info before using it.
Thanks Richard to create this neat option from the start. :-)

>
>> I've observed SSTATE broken in below scenarios:
>>
>> a) ~31 recipes fail to reuse sstate packages when the 1st build and the 2nd build are
>> continuously build, when using same source tree
>> https://lists.yoctoproject.org/pipermail/poky/2010-November/000174.html
>>
>> b) more recipes fail if the 2nd build was started in next day after when the 1st build
>> was generated. This is related to DATE/TIME variables referenced by some tasks.
>
>I'd be interested to see which recipes are doing this...

It's referred by patch_do_patch, which is one example:
                        if srcdate == "now":
                                srcdate = bb.data.getVar('DATE', d, 1)

Other tasks like do_distro_check refer to DATETIME, which is however not counted in
normal task dependency and thus should be OK.

Frankly speaking I think this DATE issue should cause wider problem than what I have
observed, since do_patch task is invoked for many recipes. Perhaps it's due to my
messed build directories when tackling multiple issues in parallel, and thus my memory 
is not accurate. But anyway the DATE variable does need filtered out. :-)

>
>> c) My build machine suffered from a power down, and then all STATE packages can't
>> be reused after reboot. Comparison between two builds show that some shell
>> variables change, e.g: GNOME_KEYRING_SOCKET, SSH_AUTH_SOCK, etc.
>
>We should whitelist the ones we don't mind if they change.

I'm just afraid that this set is volatile to various host systems, and thus maintain such a
list would be a headache. That's why I'm trying to adopt a way to define all environmental
variables coming from shell instead and then whitelist all of them out.

>
>> d) no STATE package from the 1st build could be reused when the two builds use
>> different source trees, though two source trees are identical with only difference on
>> the path. This is related to OEROOT
>
>I was sure I tested this and the checksums matched :/. Which variables
>are changing to cause this - we really need to track them down...

Sorry that I don't have such info from existing debug log. There I just recorded dep
variables used in checksum calculation and also the task data itself. However I didn't 
see direct invocation on OEROOT. I'll figure it out later.

>
>> e) If there're changes on libc/gcc packages between the 1st build and the 2nd build,
>> the 2nd build will always fail, either on gcc-runtime or eglibc, or other libc/gcc
>> bootstrap packages. The reason is that sstate_installation skips -initial/-intermediate
>> packages which breaks bootstrap process.
>> https://lists.yoctoproject.org/pipermail/poky/2010-November/000270.html
>>
>> The major problem for above issues is that many undesired variables which are
>> bound to build environment are counted into signature generation. Bitbake takes
>> a list of environmental variables from the shell and then convert into internal
>> variables. We need to separate those from bitbake internal variables. So I split
>> BB_ENV_WHITELIST and BB_ENV_EXTRAWHITE into two groups:
>>
>> 	BB_ENV_WHITELIST_BB/BB_ENV_EXTRAWHITE_BB
>> 	BB_ENV_WHITELIST_SHELL/BB_ENV_EXTRAWHITE_SHELL
>>
>> The 1st group covers variables defined internally in bitbake, while the other group
>> points to the true external environmental variables. This way we can always wipe
>> the _SHELL group from the dependency chain, while use BB_BASEHASH_WHITELIST
>> to control bitbake variables:
>>
>http://git.pokylinux.org/cgit/cgit.cgi/poky-contrib/commit/?h=tk/sstate&id=8e1b6a292b3
>c796c5c0273de76d2e83271c83e26
>>
>> Above looks somewhat ugly, but it should improve current situation a lot. We
>> do need a better way to mark such variable attribute efficiently though.
>
>I'm afraid I don't like this solution, its too complicated. Why don't we
>just whitelist the variables we are happy shouldn't influence the
>checksums even if they are present in the environment?

All variables coming from shell (but not bitbake defined variables) belong to that category,
imho, which is why I want to mark those variables automatically.

If you think the simple whitelist is acceptable, I can do that. It leaves two issues though:
   o We need sync any change on preserved shell variables into this whitelist manually
   o User is allowed to use BB_ENV_EXTRAWHITE to preserve more shell variables, which
     are dynamic

But perhaps it's fine to use simple whitelist as a quick workaround for now.

>
>To be honest there are several of these we should only export to
>"interactive" tasks anyway so perhaps marking some as unexport and then
>exporting them in devshell and the interactive patch handler would be a
>better approach?

Do you have an example list of them? I'm still unclear when those shell variables
are used, and thus need time to understand them.

>
>> In the meantime, there're other two enhancements which are also helpful:
>>
>>   - update-alternative expands an absolute path when appending to do_install,
>> which impacts signature:
>>
>http://git.pokylinux.org/cgit/cgit.cgi/poky-contrib/commit/?h=tk/sstate&id=a09df20442b
>cfc033b955992948e2dc2f6cbb0b4
>
>Yes, this looks correct, nice catch!
>
>>   - BBCLASSEXTEND is not covered by BB_TASKHASH_WHITELIST, i.e
>> "virtual:native:" and "virtual:nativesdk:":
>>
>http://git.pokylinux.org/cgit/cgit.cgi/poky-contrib/commit/?h=tk/sstate&id=07697ff94ff1
>829ef16effb48c73a2312af6a2ff
>
>Again, good catch!
>
>I've merged these two.
>
>> The remaining thing in the list:
>>
>> - more tests on my branch
>> - rebase to latest master
>> - address incremental build failure when libc/gcc recipes are changed. Tom Rini
>> gave a suggestion to use a similar 'deptask' as in previous stage. I need more
>> thought on that and not sure whether it can solve it completely. If there's no clean
>> way, we may wait until libc/gcc bootstrap process is changed (sysroot to different
>> places) which is a 1.0 task, and then suggest people to always wipe out related
>> sstate packages when seeing those errors
>
>Poky's sstate code workd very differently to packaged staging, I don't
>think dependencies can help us in the sstate case. We perhaps just need
>to improve the logic somehow. I need to think about the problem in more
>detail to come up with better direction suggestions.
>

OK. 

Thanks
Kevin

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: update about SSTATE status
  2010-11-22  7:45   ` Tian, Kevin
@ 2010-11-22 11:29     ` Tian, Kevin
  2010-11-28 12:37       ` Richard Purdie
  0 siblings, 1 reply; 5+ messages in thread
From: Tian, Kevin @ 2010-11-22 11:29 UTC (permalink / raw)
  To: Tian, Kevin, Richard Purdie; +Cc: poky@pokylinux.org

>From: Tian, Kevin
>Sent: Monday, November 22, 2010 3:46 PM
>
>
>>
>>> d) no STATE package from the 1st build could be reused when the two builds use
>>> different source trees, though two source trees are identical with only difference on
>>> the path. This is related to OEROOT
>>
>>I was sure I tested this and the checksums matched :/. Which variables
>>are changing to cause this - we really need to track them down...
>
>Sorry that I don't have such info from existing debug log. There I just recorded dep
>variables used in checksum calculation and also the task data itself. However I didn't
>see direct invocation on OEROOT. I'll figure it out later.
>

I think the reason is simple. OEROOT is exported as shell variable by poky-env-internal.
All preserved shell variables are included in dependencies of shell tasks:

def build_dependencies(key, keys, shelldeps, d):
    deps = set()
    try:
        if d.getVarFlag(key, "func"):
            if d.getVarFlag(key, "python"):
                parsedvar = d.expandWithRefs(d.getVar(key, False), key)
                parser = bb.codeparser.PythonParser()
                parser.parse_python(parsedvar.value)
                deps = deps | parser.references
            else:
                parsedvar = d.expandWithRefs(d.getVar(key, False), key)
                parser = bb.codeparser.ShellParser()
                parser.parse_shell(parsedvar.value)
                deps = deps | shelldeps

Thanks
Kevin


^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: update about SSTATE status
  2010-11-22 11:29     ` Tian, Kevin
@ 2010-11-28 12:37       ` Richard Purdie
  0 siblings, 0 replies; 5+ messages in thread
From: Richard Purdie @ 2010-11-28 12:37 UTC (permalink / raw)
  To: Tian, Kevin; +Cc: poky@pokylinux.org

On Mon, 2010-11-22 at 19:29 +0800, Tian, Kevin wrote:
> >From: Tian, Kevin
> >
> >Sorry that I don't have such info from existing debug log. There I just recorded dep
> >variables used in checksum calculation and also the task data itself. However I didn't
> >see direct invocation on OEROOT. I'll figure it out later.
> >
> 
> I think the reason is simple. OEROOT is exported as shell variable by poky-env-internal.
> All preserved shell variables are included in dependencies of shell tasks:

Ah, good find. Since nothing actually needs OEROOT anymore, lets remove
it :). I'll queue some patches.

Cheers,

Richard



^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2010-11-28 21:19 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2010-11-21 13:02 update about SSTATE status Tian, Kevin
2010-11-21 18:17 ` Richard Purdie
2010-11-22  7:45   ` Tian, Kevin
2010-11-22 11:29     ` Tian, Kevin
2010-11-28 12:37       ` Richard Purdie

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox