sstate_clean() overzealous?

sstate_clean() overzealous?

[Phil Blundell]

The sstate_clean() function in sstate.bbclass is doing (inter alia):

    stfile = d.getVar("STAMP", True) + ".do_" + ss['task']
    [...]
    oe.path.remove(stfile + ".*")
    oe.path.remove(stfile + "_setscene" + ".*")

which means that, for tasks which set the stamp-extra-info flag to
${MACHINE}, it ends up blowing away the stamps for all machines rather
than just the current one.  The net effect of this seems to be that
there is no way to have the setscene stamps populated for more than one
MACHINE at any time, and hence all those tasks get rerun every time you
change MACHINE even if nothing else has been altered.

Is this behaviour deliberate?  It is certainly a little bit annoying but
I don't understand the internals of sstate well enough to judge what
exactly that glob is trying to accomplish.

thanks

p.

Re: sstate_clean() overzealous?

[McClintock Matthew-B29882]

On Mon, Oct 3, 2011 at 11:47 AM, Phil Blundell <philb@gnu.org> wrote:
> The sstate_clean() function in sstate.bbclass is doing (inter alia):
>
>    stfile = d.getVar("STAMP", True) + ".do_" + ss['task']
>    [...]
>    oe.path.remove(stfile + ".*")
>    oe.path.remove(stfile + "_setscene" + ".*")
>
> which means that, for tasks which set the stamp-extra-info flag to
> ${MACHINE}, it ends up blowing away the stamps for all machines rather
> than just the current one.  The net effect of this seems to be that
> there is no way to have the setscene stamps populated for more than one
> MACHINE at any time, and hence all those tasks get rerun every time you
> change MACHINE even if nothing else has been altered.
>
> Is this behaviour deliberate?  It is certainly a little bit annoying but
> I don't understand the internals of sstate well enough to judge what
> exactly that glob is trying to accomplish.

I've noticed that changing the MACHINE will also invalidate the
-native stuff as well.

-M

Re: sstate_clean() overzealous?

[Khem Raj]

On 10/3/2011 2:11 PM, McClintock Matthew-B29882 wrote:
> On Mon, Oct 3, 2011 at 11:47 AM, Phil Blundell<philb@gnu.org>  wrote:
>> The sstate_clean() function in sstate.bbclass is doing (inter alia):
>>
>>     stfile = d.getVar("STAMP", True) + ".do_" + ss['task']
>>     [...]
>>     oe.path.remove(stfile + ".*")
>>     oe.path.remove(stfile + "_setscene" + ".*")
>>
>> which means that, for tasks which set the stamp-extra-info flag to
>> ${MACHINE}, it ends up blowing away the stamps for all machines rather
>> than just the current one.  The net effect of this seems to be that
>> there is no way to have the setscene stamps populated for more than one
>> MACHINE at any time, and hence all those tasks get rerun every time you
>> change MACHINE even if nothing else has been altered.
>>
>> Is this behaviour deliberate?  It is certainly a little bit annoying but
>> I don't understand the internals of sstate well enough to judge what
>> exactly that glob is trying to accomplish.
>
> I've noticed that changing the MACHINE will also invalidate the
> -native stuff as well.

hmmm so we can not share sstate parts for two machines in same tmpdir ?

>
> -M
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Re: sstate_clean() overzealous?

[McClintock Matthew-B29882]

On Mon, Oct 3, 2011 at 4:18 PM, Khem Raj <raj.khem@gmail.com> wrote:
> hmmm so we can not share sstate parts for two machines in same tmpdir ?

I'm not 100% sure I'm doing things properly but this is what I did:

$ bitbake pseudo-native
$ cp sstate-cache/* ~/sstate-cache
$ rm -rf tmp/
$ bitbake pseudo-native

Now sstate-cache is verified to be working.... then I changed machines and ran

$ cp ~/sstate-cache/* sstate-cache/
$ bitbake pseudo-native

bitbake starts rebuilding everything, and added -DDD I see SState
cache is missing.

-M

Re: sstate_clean() overzealous?

[Saul Wold]

On 10/03/2011 02:11 PM, McClintock Matthew-B29882 wrote:
> On Mon, Oct 3, 2011 at 11:47 AM, Phil Blundell<philb@gnu.org>  wrote:
>> The sstate_clean() function in sstate.bbclass is doing (inter alia):
>>
>>     stfile = d.getVar("STAMP", True) + ".do_" + ss['task']
>>     [...]
>>     oe.path.remove(stfile + ".*")
>>     oe.path.remove(stfile + "_setscene" + ".*")
>>
>> which means that, for tasks which set the stamp-extra-info flag to
>> ${MACHINE}, it ends up blowing away the stamps for all machines rather
>> than just the current one.  The net effect of this seems to be that
>> there is no way to have the setscene stamps populated for more than one
>> MACHINE at any time, and hence all those tasks get rerun every time you
>> change MACHINE even if nothing else has been altered.
>>
>> Is this behaviour deliberate?  It is certainly a little bit annoying but
>> I don't understand the internals of sstate well enough to judge what
>> exactly that glob is trying to accomplish.
>
> I've noticed that changing the MACHINE will also invalidate the
> -native stuff as well.
>
I am not sure that's correct, since we use sstate with the Autobuilder 
and it's shared across different hosts building different MACHINEs, we 
don't see the -natives getting build, do you have any examples?

Sau!

> -M
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>

Re: sstate_clean() overzealous?

[McClintock Matthew-B29882]

On Mon, Oct 3, 2011 at 4:24 PM, Saul Wold <saul.wold@intel.com> wrote:
> I am not sure that's correct, since we use sstate with the Autobuilder and
> it's shared across different hosts building different MACHINEs, we don't see
> the -natives getting build, do you have any examples?

I'm building with targets that are not upstream, so I can't comment on
the upstream variant. There are different TUNES though. I'm actually
currently in the process of looking at why sstate-cache is not being
reused.

Can you point at the autobuilder stuff? Is it going from
fsl-mpc8315e-rdb to and x86_64 target? Or between two x86_64 targets?

-M

Re: sstate_clean() overzealous?

[McClintock Matthew-B29882]

On Mon, Oct 3, 2011 at 4:29 PM, Matthew McClintock <msm@freescale.com> wrote:
> On Mon, Oct 3, 2011 at 4:24 PM, Saul Wold <saul.wold@intel.com> wrote:
>> I am not sure that's correct, since we use sstate with the Autobuilder and
>> it's shared across different hosts building different MACHINEs, we don't see
>> the -natives getting build, do you have any examples?
>
> I'm building with targets that are not upstream, so I can't comment on
> the upstream variant. There are different TUNES though. I'm actually
> currently in the process of looking at why sstate-cache is not being
> reused.
>
> Can you point at the autobuilder stuff? Is it going from
> fsl-mpc8315e-rdb to and x86_64 target? Or between two x86_64 targets?

For example autoconf-native:

$ bitbake-diffsigs
../sstate-cache/sstate-autoconf-native-x86_64-linux-2.68-r2-x86_64-2-*siginfo
[mattsm@right build_p5020ds_release (master *$)]$ bitbake-diffsigs
../sstate-cache/sstate-autoconf-native-x86_64-linux-2.68-r2-x86_64-2-*siginfo
[snip]
basehash changed from bab02dfa833aca4810b0fae8c4ab48f9 to
e156f3fb86236bfc0262e6287bd73784
List of dependencies for variable baselib changed from
set(['BASELIB']) to set(['DEFAULTTUNE'])
Dependency on variable BASELIB was added
Dependency on Variable DEFAULTTUNE was removed
Variable baselib value changed from ${BASELIB} to
${@d.getVar('BASE_LIB_tune-' + (d.getVar('DEFAULTTUNE', True) or
'INVALID'), True) or 'lib'}
Hash for dependent task
virtual:native:/local/home/mattsm/git/poky/meta/recipes-devtools/autoconf/autoconf_2.68.bb.do_install
changed from e74616029727b6e61b00273425128f1d to
478e7900a2310f2669708e6909883865

Back tracking this appears to be a multilib issue since I have that
enabled on one of the -native packages:

../meta/conf/bitbake.conf:baselib = "${BASELIB}"
../meta/conf/multilib.conf:baselib = "${@d.getVar('BASE_LIB_tune-' +
(d.getVar('DEFAULTTUNE', True) or 'INVALID'), True) or 'lib'}"

Which causes the other differences....

-M

Re: sstate_clean() overzealous?

[Richard Purdie]

On Mon, 2011-10-03 at 17:47 +0100, Phil Blundell wrote:
> The sstate_clean() function in sstate.bbclass is doing (inter alia):
> 
>     stfile = d.getVar("STAMP", True) + ".do_" + ss['task']
>     [...]
>     oe.path.remove(stfile + ".*")
>     oe.path.remove(stfile + "_setscene" + ".*")
> 
> which means that, for tasks which set the stamp-extra-info flag to
> ${MACHINE}, it ends up blowing away the stamps for all machines rather
> than just the current one.  The net effect of this seems to be that
> there is no way to have the setscene stamps populated for more than one
> MACHINE at any time, and hence all those tasks get rerun every time you
> change MACHINE even if nothing else has been altered.
> 
> Is this behaviour deliberate? 

No, its not intended to be wiping out all the different machines.

>  It is certainly a little bit annoying but
> I don't understand the internals of sstate well enough to judge what
> exactly that glob is trying to accomplish.

The trouble is we now support a variety of different stamp file formats
depending on what settings you have enabled (and metadata you're using).

Clean can be called from two sources, do_clean, or just before a sstate
controlled task is about to be run. In either case we want to clear out
any previous "state" including any stamp files that might have
represented this task since we're about to invalidate anything they did.
This means wiping out "normal" stamps and any setscene variants of those
stamps. We also want to remove any stamps representing other hashes for
this task since there is no version installed.

If the BasicHash signature generator code is enabled this adds hashes to
the stamp files. This turns the stamps into the form:

<stampfile>.<hash>
or in the case of machine specific tasks:
<stampfile>.<hash>.<machine>

I think we need to teach this clean function about stamp-extra-info and
instead of stfile + ".*", remove stfile + ".*." + stamp-extra-info if
stamp-extra-info is set. That would likely fix the problem you're
seeing.

Cheers,

Richard

Re: sstate_clean() overzealous?

[Phil Blundell]

On Tue, 2011-10-04 at 11:54 +0100, Richard Purdie wrote:
> I think we need to teach this clean function about stamp-extra-info and
> instead of stfile + ".*", remove stfile + ".*." + stamp-extra-info if
> stamp-extra-info is set. That would likely fix the problem you're
> seeing.

Okay, thanks.  I applied this change locally:

diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index 91f209a..7f38800 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -258,10 +258,15 @@ def sstate_clean(ss, d):
             bb.utils.unlockfile(lock)
 
     stfile = d.getVar("STAMP", True) + ".do_" + ss['task']
+    extrainf = d.getVarFlag("do_" + ss['task'], 'stamp-extra-info')
     oe.path.remove(stfile)
     oe.path.remove(stfile + "_setscene")
-    oe.path.remove(stfile + ".*")
-    oe.path.remove(stfile + "_setscene" + ".*")
+    if extrainf:
+        oe.path.remove(stfile + ".*." + extrainf)
+       oe.path.remove(stfile + "_setscene" + ".*." + extrainf)
+    else:
+        oe.path.remove(stfile + ".*")
+       oe.path.remove(stfile + "_setscene" + ".*")
 
 CLEANFUNCS += "sstate_cleanall"
 
and it does indeed seem to resolve the problem I was seeing before.

p.

Re: sstate_clean() overzealous?

[Richard Purdie]

On Tue, 2011-10-04 at 16:03 +0100, Phil Blundell wrote:
> On Tue, 2011-10-04 at 11:54 +0100, Richard Purdie wrote:
> > I think we need to teach this clean function about stamp-extra-info and
> > instead of stfile + ".*", remove stfile + ".*." + stamp-extra-info if
> > stamp-extra-info is set. That would likely fix the problem you're
> > seeing.
> 
> Okay, thanks.  I applied this change locally:
> 
> diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
> index 91f209a..7f38800 100644
> --- a/meta/classes/sstate.bbclass
> +++ b/meta/classes/sstate.bbclass
> @@ -258,10 +258,15 @@ def sstate_clean(ss, d):
>              bb.utils.unlockfile(lock)
>  
>      stfile = d.getVar("STAMP", True) + ".do_" + ss['task']
> +    extrainf = d.getVarFlag("do_" + ss['task'], 'stamp-extra-info')
>      oe.path.remove(stfile)
>      oe.path.remove(stfile + "_setscene")
> -    oe.path.remove(stfile + ".*")
> -    oe.path.remove(stfile + "_setscene" + ".*")
> +    if extrainf:
> +        oe.path.remove(stfile + ".*." + extrainf)
> +       oe.path.remove(stfile + "_setscene" + ".*." + extrainf)

Thinking further about this it may need to be ".*" instead of ".*.". I
suspect with the above it wouldn't be wiping out some of the stamps it
should be with OE-Core and the Basic siggen code. BasicHash would work
though.

Cheers,

Richard

Re: sstate_clean() overzealous?

[Richard Purdie]

On Mon, 2011-10-03 at 22:08 +0000, McClintock Matthew-B29882 wrote:
> On Mon, Oct 3, 2011 at 4:29 PM, Matthew McClintock <msm@freescale.com> wrote:
> > On Mon, Oct 3, 2011 at 4:24 PM, Saul Wold <saul.wold@intel.com> wrote:
> >> I am not sure that's correct, since we use sstate with the Autobuilder and
> >> it's shared across different hosts building different MACHINEs, we don't see
> >> the -natives getting build, do you have any examples?
> >
> > I'm building with targets that are not upstream, so I can't comment on
> > the upstream variant. There are different TUNES though. I'm actually
> > currently in the process of looking at why sstate-cache is not being
> > reused.
> >
> > Can you point at the autobuilder stuff? Is it going from
> > fsl-mpc8315e-rdb to and x86_64 target? Or between two x86_64 targets?
> 
> For example autoconf-native:
> 
> $ bitbake-diffsigs
> ../sstate-cache/sstate-autoconf-native-x86_64-linux-2.68-r2-x86_64-2-*siginfo
> [mattsm@right build_p5020ds_release (master *$)]$ bitbake-diffsigs
> ../sstate-cache/sstate-autoconf-native-x86_64-linux-2.68-r2-x86_64-2-*siginfo
> [snip]
> basehash changed from bab02dfa833aca4810b0fae8c4ab48f9 to
> e156f3fb86236bfc0262e6287bd73784
> List of dependencies for variable baselib changed from
> set(['BASELIB']) to set(['DEFAULTTUNE'])
> Dependency on variable BASELIB was added
> Dependency on Variable DEFAULTTUNE was removed
> Variable baselib value changed from ${BASELIB} to
> ${@d.getVar('BASE_LIB_tune-' + (d.getVar('DEFAULTTUNE', True) or
> 'INVALID'), True) or 'lib'}
> Hash for dependent task
> virtual:native:/local/home/mattsm/git/poky/meta/recipes-devtools/autoconf/autoconf_2.68.bb.do_install
> changed from e74616029727b6e61b00273425128f1d to
> 478e7900a2310f2669708e6909883865
> 
> Back tracking this appears to be a multilib issue since I have that
> enabled on one of the -native packages:
> 
> ../meta/conf/bitbake.conf:baselib = "${BASELIB}"
> ../meta/conf/multilib.conf:baselib = "${@d.getVar('BASE_LIB_tune-' +
> (d.getVar('DEFAULTTUNE', True) or 'INVALID'), True) or 'lib'}"
> 

This is bad and shouldn't be happening, particularly for the native
packages. I did accidentally commit part of a work in progress in this
commit:

http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=ea0b7440ec5a8663b66f56a19f0b38e803b523d3

I didn't revert it since I thought it was a good idea and fixed some
problems I was seeing.

I suspect a baselib = "lib" in native.bbclass might be a good idea
too...

Cheers,

Richard