Is sstate broken

Is sstate broken

[Gary Thomas]

I tried to run an experiment with SSTATE_MIRRORS on the latest master.
 From what I can tell, this is 100% not working, although I may just
not have set it up properly.  Here's what I did:

* Build Poky for my platform in /local/xyz_ppc_poky
* Set up to build exact same machine/config/etc in /local/xyz_test
   pointing SSTATE_MIRRORS at /local/xyz_ppc_poky/sstate-cache by
   adding these lines to local.conf (only configuration change)
     SSTATE_MIRRORS ?= "\
     file://.* file:///local/xyz_ppc_poky/sstate-cache/"

The end result is that both builds built (from scratch) the same
set of packages.  There seems to be no reuse of anything from
the SSTATE_MIRRORS.

$ du /local/xyz_ppc_poky/* -s
20      /local/xyz_ppc_poky/conf
1456    /local/xyz_ppc_poky/downloads
4       /local/xyz_ppc_poky/pseudodone
1010392 /local/xyz_ppc_poky/sstate-cache
14434076        /local/xyz_ppc_poky/tmp

$ du -s /local/xyz_test
20      /local/xyz_test/conf
1456    /local/xyz_test/downloads
4       /local/xyz_test/pseudodone
1010276 /local/xyz_test/sstate-cache
14425860       /local/xyz_test/ tmp

The full build log of the second run is at http://www.mlbassoc.com/poky/build_ppc_with_sstate
Notice the many lines which say that a particular step failed.  I can't see the reason, nor
can I find any corresponding log file to help me out.  An example is:
   ERROR: Task 0 (virtual:native:/home/local/poky-amltd/meta/recipes-devtools/pseudo/pseudo_1.0.bb, do_unpack) failed with exit code '1'

Am I missing something?  Now that the build with BB_NO_NETWORK is working well,
this is my last hurdle to making Poky truly useful for my customers.

Thanks

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: Is sstate broken

[Zhai, Edwin]

Gary,

Thanks for trying this.
sstate function is a little bit fragile recently. I'll look at it after 
fixing some other bugs.



Gary Thomas wrote:
>
> I tried to run an experiment with SSTATE_MIRRORS on the latest master.
>  From what I can tell, this is 100% not working, although I may just
> not have set it up properly.  Here's what I did:
>
> * Build Poky for my platform in /local/xyz_ppc_poky
> * Set up to build exact same machine/config/etc in /local/xyz_test
>    pointing SSTATE_MIRRORS at /local/xyz_ppc_poky/sstate-cache by
>    adding these lines to local.conf (only configuration change)
>      SSTATE_MIRRORS ?= "\
>      file://.* file:///local/xyz_ppc_poky/sstate-cache/"
>
> The end result is that both builds built (from scratch) the same
> set of packages.  There seems to be no reuse of anything from
> the SSTATE_MIRRORS.
>
> $ du /local/xyz_ppc_poky/* -s
> 20      /local/xyz_ppc_poky/conf
> 1456    /local/xyz_ppc_poky/downloads
> 4       /local/xyz_ppc_poky/pseudodone
> 1010392 /local/xyz_ppc_poky/sstate-cache
> 14434076        /local/xyz_ppc_poky/tmp
>
> $ du -s /local/xyz_test
> 20      /local/xyz_test/conf
> 1456    /local/xyz_test/downloads
> 4       /local/xyz_test/pseudodone
> 1010276 /local/xyz_test/sstate-cache
> 14425860       /local/xyz_test/ tmp
>
> The full build log of the second run is at 
> http://www.mlbassoc.com/poky/build_ppc_with_sstate
> Notice the many lines which say that a particular step failed.  I 
> can't see the reason, nor
> can I find any corresponding log file to help me out.  An example is:
>    ERROR: Task 0 
> (virtual:native:/home/local/poky-amltd/meta/recipes-devtools/pseudo/pseudo_1.0.bb, 
> do_unpack) failed with exit code '1'
>
> Am I missing something?  Now that the build with BB_NO_NETWORK is 
> working well,
> this is my last hurdle to making Poky truly useful for my customers.
>
> Thanks
>
> -- 
> ------------------------------------------------------------
> Gary Thomas                 |  Consulting for the
> MLB Associates              |    Embedded world
> ------------------------------------------------------------
> _______________________________________________
> poky mailing list
> poky@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/poky
>

Re: Is sstate broken

[Gary Thomas]

On 02/27/2011 11:07 PM, Zhai, Edwin wrote:
> Gary,
>
> Thanks for trying this.
> sstate function is a little bit fragile recently. I'll look at it after fixing some other bugs.

Any progress on this?

Should I file this as a bug?

> Gary Thomas wrote:
>>
>> I tried to run an experiment with SSTATE_MIRRORS on the latest master.
>> From what I can tell, this is 100% not working, although I may just
>> not have set it up properly. Here's what I did:
>>
>> * Build Poky for my platform in /local/xyz_ppc_poky
>> * Set up to build exact same machine/config/etc in /local/xyz_test
>> pointing SSTATE_MIRRORS at /local/xyz_ppc_poky/sstate-cache by
>> adding these lines to local.conf (only configuration change)
>> SSTATE_MIRRORS ?= "\
>> file://.* file:///local/xyz_ppc_poky/sstate-cache/"
>>
>> The end result is that both builds built (from scratch) the same
>> set of packages. There seems to be no reuse of anything from
>> the SSTATE_MIRRORS.
>>
>> $ du /local/xyz_ppc_poky/* -s
>> 20 /local/xyz_ppc_poky/conf
>> 1456 /local/xyz_ppc_poky/downloads
>> 4 /local/xyz_ppc_poky/pseudodone
>> 1010392 /local/xyz_ppc_poky/sstate-cache
>> 14434076 /local/xyz_ppc_poky/tmp
>>
>> $ du -s /local/xyz_test
>> 20 /local/xyz_test/conf
>> 1456 /local/xyz_test/downloads
>> 4 /local/xyz_test/pseudodone
>> 1010276 /local/xyz_test/sstate-cache
>> 14425860 /local/xyz_test/ tmp
>>
>> The full build log of the second run is at http://www.mlbassoc.com/poky/build_ppc_with_sstate
>> Notice the many lines which say that a particular step failed. I can't see the reason, nor
>> can I find any corresponding log file to help me out. An example is:
>> ERROR: Task 0 (virtual:native:/home/local/poky-amltd/meta/recipes-devtools/pseudo/pseudo_1.0.bb, do_unpack) failed with exit code '1'
>>
>> Am I missing something? Now that the build with BB_NO_NETWORK is working well,
>> this is my last hurdle to making Poky truly useful for my customers.
>>
>> Thanks
>>
>> --
>> ------------------------------------------------------------
>> Gary Thomas | Consulting for the
>> MLB Associates | Embedded world
>> ------------------------------------------------------------
>> _______________________________________________
>> poky mailing list
>> poky@yoctoproject.org
>> https://lists.yoctoproject.org/listinfo/poky
>>

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: Is sstate broken

[Zhai, Edwin]

Gary Thomas wrote:
>
> On 02/27/2011 11:07 PM, Zhai, Edwin wrote:
> > Gary,
> >
> > Thanks for trying this.
> > sstate function is a little bit fragile recently. I'll look at it 
> after fixing some other bugs.
>
> Any progress on this?
>
> Should I file this as a bug?
>

We have one http://bugzilla.pokylinux.org/show_bug.cgi?id=788


> > Gary Thomas wrote:
> >>
> >> I tried to run an experiment with SSTATE_MIRRORS on the latest master.
> >> From what I can tell, this is 100% not working, although I may just
> >> not have set it up properly. Here's what I did:
> >>
> >> * Build Poky for my platform in /local/xyz_ppc_poky
> >> * Set up to build exact same machine/config/etc in /local/xyz_test
> >> pointing SSTATE_MIRRORS at /local/xyz_ppc_poky/sstate-cache by
> >> adding these lines to local.conf (only configuration change)
> >> SSTATE_MIRRORS ?= "\
> >> file://.* file:///local/xyz_ppc_poky/sstate-cache/"
> >>
> >> The end result is that both builds built (from scratch) the same
> >> set of packages. There seems to be no reuse of anything from
> >> the SSTATE_MIRRORS.
> >>
> >> $ du /local/xyz_ppc_poky/* -s
> >> 20 /local/xyz_ppc_poky/conf
> >> 1456 /local/xyz_ppc_poky/downloads
> >> 4 /local/xyz_ppc_poky/pseudodone
> >> 1010392 /local/xyz_ppc_poky/sstate-cache
> >> 14434076 /local/xyz_ppc_poky/tmp
> >>
> >> $ du -s /local/xyz_test
> >> 20 /local/xyz_test/conf
> >> 1456 /local/xyz_test/downloads
> >> 4 /local/xyz_test/pseudodone
> >> 1010276 /local/xyz_test/sstate-cache
> >> 14425860 /local/xyz_test/ tmp
> >>
> >> The full build log of the second run is at 
> http://www.mlbassoc.com/poky/build_ppc_with_sstate
> >> Notice the many lines which say that a particular step failed. I 
> can't see the reason, nor
> >> can I find any corresponding log file to help me out. An example is:
> >> ERROR: Task 0 
> (virtual:native:/home/local/poky-amltd/meta/recipes-devtools/pseudo/pseudo_1.0.bb, 
> do_unpack) failed with exit code '1'
>
> >>
> >> Am I missing something? Now that the build with BB_NO_NETWORK is 
> working well,
> >> this is my last hurdle to making Poky truly useful for my customers.
> >>
> >> Thanks
> >>
> >> --
> >> ------------------------------------------------------------
> >> Gary Thomas | Consulting for the
> >> MLB Associates | Embedded world
> >> ------------------------------------------------------------
> >> _______________________________________________
> >> poky mailing list
> >> poky@yoctoproject.org
> >> https://lists.yoctoproject.org/listinfo/poky
> >>
>
> -- 
> ------------------------------------------------------------
> Gary Thomas                 |  Consulting for the
> MLB Associates              |    Embedded world
> ------------------------------------------------------------
>

Re: Is sstate broken

[Gary Thomas]

On 03/03/2011 06:59 PM, Zhai, Edwin wrote:
>
>
> Gary Thomas wrote:
>>
>> On 02/27/2011 11:07 PM, Zhai, Edwin wrote:
>> > Gary,
>> >
>> > Thanks for trying this.
>> > sstate function is a little bit fragile recently. I'll look at it after fixing some other bugs.
>>
>> Any progress on this?
>>
>> Should I file this as a bug?
>>
>
> We have one http://bugzilla.pokylinux.org/show_bug.cgi?id=788

Yes, that addresses the random errors (which don't seem to have any affect)

However, even without those errors, the sstate information is not being
used at all.  In my experiment, a new tree (with the old sstate pointed
to by SSTATE_MIRRORS) is executing a full build every time.

Did I miss something?

>
>
>> > Gary Thomas wrote:
>> >>
>> >> I tried to run an experiment with SSTATE_MIRRORS on the latest master.
>> >> From what I can tell, this is 100% not working, although I may just
>> >> not have set it up properly. Here's what I did:
>> >>
>> >> * Build Poky for my platform in /local/xyz_ppc_poky
>> >> * Set up to build exact same machine/config/etc in /local/xyz_test
>> >> pointing SSTATE_MIRRORS at /local/xyz_ppc_poky/sstate-cache by
>> >> adding these lines to local.conf (only configuration change)
>> >> SSTATE_MIRRORS ?= "\
>> >> file://.* file:///local/xyz_ppc_poky/sstate-cache/"
>> >>
>> >> The end result is that both builds built (from scratch) the same
>> >> set of packages. There seems to be no reuse of anything from
>> >> the SSTATE_MIRRORS.
>> >>
>> >> $ du /local/xyz_ppc_poky/* -s
>> >> 20 /local/xyz_ppc_poky/conf
>> >> 1456 /local/xyz_ppc_poky/downloads
>> >> 4 /local/xyz_ppc_poky/pseudodone
>> >> 1010392 /local/xyz_ppc_poky/sstate-cache
>> >> 14434076 /local/xyz_ppc_poky/tmp
>> >>
>> >> $ du -s /local/xyz_test
>> >> 20 /local/xyz_test/conf
>> >> 1456 /local/xyz_test/downloads
>> >> 4 /local/xyz_test/pseudodone
>> >> 1010276 /local/xyz_test/sstate-cache
>> >> 14425860 /local/xyz_test/ tmp
>> >>
>> >> The full build log of the second run is at http://www.mlbassoc.com/poky/build_ppc_with_sstate
>> >> Notice the many lines which say that a particular step failed. I can't see the reason, nor
>> >> can I find any corresponding log file to help me out. An example is:
>> >> ERROR: Task 0 (virtual:native:/home/local/poky-amltd/meta/recipes-devtools/pseudo/pseudo_1.0.bb, do_unpack) failed with exit code '1'
>>
>> >>
>> >> Am I missing something? Now that the build with BB_NO_NETWORK is working well,
>> >> this is my last hurdle to making Poky truly useful for my customers.
>> >>
>> >> Thanks
>> >>
>> >> --
>> >> ------------------------------------------------------------
>> >> Gary Thomas | Consulting for the
>> >> MLB Associates | Embedded world
>> >> ------------------------------------------------------------
>> >> _______________________________________________
>> >> poky mailing list
>> >> poky@yoctoproject.org
>> >> https://lists.yoctoproject.org/listinfo/poky
>> >>
>>
>> --
>> ------------------------------------------------------------
>> Gary Thomas | Consulting for the
>> MLB Associates | Embedded world
>> ------------------------------------------------------------
>>

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: Is sstate broken

[Richard Purdie]

On Fri, 2011-03-04 at 04:34 -0700, Gary Thomas wrote:
> On 03/03/2011 06:59 PM, Zhai, Edwin wrote:
> > Gary Thomas wrote:
> >> On 02/27/2011 11:07 PM, Zhai, Edwin wrote:
> >> > Thanks for trying this.
> >> > sstate function is a little bit fragile recently. I'll look at it after fixing some other bugs.
> >>
> >> Any progress on this?
> >>
> >> Should I file this as a bug?
> >>
> >
> > We have one http://bugzilla.pokylinux.org/show_bug.cgi?id=788
> 
> Yes, that addresses the random errors (which don't seem to have any affect)
> 
> However, even without those errors, the sstate information is not being
> used at all.  In my experiment, a new tree (with the old sstate pointed
> to by SSTATE_MIRRORS) is executing a full build every time.
> 
> Did I miss something?

This should be fixed by the couple of recent changes in master, if you
could check this is working I'd appreciate it as we are getting close to
the release and I'd like this to be working.

Cheers,

Richard

Re: Is sstate broken

[Gary Thomas]

On 03/08/11 18:29, Richard Purdie wrote:
> On Fri, 2011-03-04 at 04:34 -0700, Gary Thomas wrote:
>> On 03/03/2011 06:59 PM, Zhai, Edwin wrote:
>>> Gary Thomas wrote:
>>>> On 02/27/2011 11:07 PM, Zhai, Edwin wrote:
>>>>> Thanks for trying this.
>>>>> sstate function is a little bit fragile recently. I'll look at it after fixing some other bugs.
>>>>
>>>> Any progress on this?
>>>>
>>>> Should I file this as a bug?
>>>>
>>>
>>> We have one http://bugzilla.pokylinux.org/show_bug.cgi?id=788
>>
>> Yes, that addresses the random errors (which don't seem to have any affect)
>>
>> However, even without those errors, the sstate information is not being
>> used at all.  In my experiment, a new tree (with the old sstate pointed
>> to by SSTATE_MIRRORS) is executing a full build every time.
>>
>> Did I miss something?
>
> This should be fixed by the couple of recent changes in master, if you
> could check this is working I'd appreciate it as we are getting close to
> the release and I'd like this to be working.

I updated to master (fab742bd4693ed3092690a20dd32d53fe27c3d4c) and tried again.
I don't see any difference - the run using the sstate cache as a mirror
seems to do all the same work as without.  Here's how I tested it.

* Build original tree
   % POKYCONF=meta-cobra3530p60/conf . /tmp/poky-amltd/poky-init-build-env /local/p60_step1
   ... adjust conf/local.conf
   % bitbake amltd-console-image

* Rebuild, using previous result for SSTATE_MIRRORS
   % POKYCONF=meta-cobra3530p60/conf . /tmp/poky-amltd/poky-init-build-env /local/p60_step2
   ... adjust conf/local.conf
   % bitbake amltd-console-image

The only difference between the two runs is enabling SSTATE_MIRRORS in local.conf
   %  diff -u /local/p60_step?/conf/local.conf
    --- /local/p60_step1/conf/local.conf    2011-03-09 08:28:18.266933061 -0700
    +++ /local/p60_step2/conf/local.conf    2011-03-09 09:57:51.365932951 -0700
    @@ -53,4 +53,7 @@
     IMAGE_LINGUAS ?= "en-us"

     # Minimize feature set
     DISTRO_FEATURES ?= "alsa"
    +SSTATE_MIRRORS ?= "\
    +file://.* file:///local/p60_step1/sstate-cache/"

The results seem to have gone through all the same steps (or nearly so).  The output
from the runs is at
   http://www.mlbassoc.com/poky/build.step1
   http://www.mlbassoc.com/poky/build.step2

Comparing the two build trees:
   % ls /local/p60_step1/tmp/work/*/*/temp/log.do_compile | wc
     144     144   12521
   % ls /local/p60_step2/tmp/work/*/*/temp/log.do_compile | wc
     143     143   12427
   % du -s /local/p60_step?
   15229296        /local/p60_step1
   15162760        /local/p60_step2

I know this procedure used to work (or at least close).  Am I doing
something wrong?

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: Is sstate broken

[Richard Purdie]

On Wed, 2011-03-09 at 11:42 -0700, Gary Thomas wrote:
> I updated to master (fab742bd4693ed3092690a20dd32d53fe27c3d4c) and tried again.
> I don't see any difference - the run using the sstate cache as a mirror
> seems to do all the same work as without.  Here's how I tested it.
> 
> * Build original tree
>    % POKYCONF=meta-cobra3530p60/conf . /tmp/poky-amltd/poky-init-build-env /local/p60_step1
>    ... adjust conf/local.conf
>    % bitbake amltd-console-image
> 
> * Rebuild, using previous result for SSTATE_MIRRORS
>    % POKYCONF=meta-cobra3530p60/conf . /tmp/poky-amltd/poky-init-build-env /local/p60_step2
>    ... adjust conf/local.conf
>    % bitbake amltd-console-image
> 
> The only difference between the two runs is enabling SSTATE_MIRRORS in local.conf
>    %  diff -u /local/p60_step?/conf/local.conf
>     --- /local/p60_step1/conf/local.conf    2011-03-09 08:28:18.266933061 -0700
>     +++ /local/p60_step2/conf/local.conf    2011-03-09 09:57:51.365932951 -0700
>     @@ -53,4 +53,7 @@
>      IMAGE_LINGUAS ?= "en-us"
> 
>      # Minimize feature set
>      DISTRO_FEATURES ?= "alsa"
>     +SSTATE_MIRRORS ?= "\
>     +file://.* file:///local/p60_step1/sstate-cache/"
> 
> The results seem to have gone through all the same steps (or nearly so).  The output
> from the runs is at
>    http://www.mlbassoc.com/poky/build.step1
>    http://www.mlbassoc.com/poky/build.step2
> 
> Comparing the two build trees:
>    % ls /local/p60_step1/tmp/work/*/*/temp/log.do_compile | wc
>      144     144   12521
>    % ls /local/p60_step2/tmp/work/*/*/temp/log.do_compile | wc
>      143     143   12427
>    % du -s /local/p60_step?
>    15229296        /local/p60_step1
>    15162760        /local/p60_step2
> 
> I know this procedure used to work (or at least close).  Am I doing
> something wrong?

You're not doing anything wrong and this is the same scenario I've been
testing with. After I'd fixed the origin problem it created a problem
with file urls containing globing. I managed to break the original patch
with the globing fix. The good news is the problem is simple and I've
pushed a fix.

It should work *much* better than you've seen above, not needing to
rerun many tasks so if its working it will be very obvious.

Cheers,

Richard

Re: [poky] Is sstate broken

[Koen Kooi]

Op 9 mrt 2011, om 20:04 heeft Richard Purdie het volgende geschreven:

> On Wed, 2011-03-09 at 11:42 -0700, Gary Thomas wrote:
>> I updated to master (fab742bd4693ed3092690a20dd32d53fe27c3d4c) and tried again.
>> I don't see any difference - the run using the sstate cache as a mirror
>> seems to do all the same work as without.  Here's how I tested it.
>> 
>> * Build original tree
>>   % POKYCONF=meta-cobra3530p60/conf . /tmp/poky-amltd/poky-init-build-env /local/p60_step1
>>   ... adjust conf/local.conf
>>   % bitbake amltd-console-image
>> 
>> * Rebuild, using previous result for SSTATE_MIRRORS
>>   % POKYCONF=meta-cobra3530p60/conf . /tmp/poky-amltd/poky-init-build-env /local/p60_step2
>>   ... adjust conf/local.conf
>>   % bitbake amltd-console-image
>> 
>> The only difference between the two runs is enabling SSTATE_MIRRORS in local.conf
>>   %  diff -u /local/p60_step?/conf/local.conf
>>    --- /local/p60_step1/conf/local.conf    2011-03-09 08:28:18.266933061 -0700
>>    +++ /local/p60_step2/conf/local.conf    2011-03-09 09:57:51.365932951 -0700
>>    @@ -53,4 +53,7 @@
>>     IMAGE_LINGUAS ?= "en-us"
>> 
>>     # Minimize feature set
>>     DISTRO_FEATURES ?= "alsa"
>>    +SSTATE_MIRRORS ?= "\
>>    +file://.* file:///local/p60_step1/sstate-cache/"
>> 
>> The results seem to have gone through all the same steps (or nearly so).  The output
>> from the runs is at
>>   http://www.mlbassoc.com/poky/build.step1
>>   http://www.mlbassoc.com/poky/build.step2
>> 
>> Comparing the two build trees:
>>   % ls /local/p60_step1/tmp/work/*/*/temp/log.do_compile | wc
>>     144     144   12521
>>   % ls /local/p60_step2/tmp/work/*/*/temp/log.do_compile | wc
>>     143     143   12427
>>   % du -s /local/p60_step?
>>   15229296        /local/p60_step1
>>   15162760        /local/p60_step2
>> 
>> I know this procedure used to work (or at least close).  Am I doing
>> something wrong?
> 
> You're not doing anything wrong and this is the same scenario I've been
> testing with. After I'd fixed the origin problem it created a problem
> with file urls containing globing. I managed to break the original patch
> with the globing fix. The good news is the problem is simple and I've
> pushed a fix.

Is that fix in oe-core as well?

regards,

Koen

Re: Is sstate broken

[Koen Kooi]

Op 9 mrt 2011, om 20:04 heeft Richard Purdie het volgende geschreven:

> On Wed, 2011-03-09 at 11:42 -0700, Gary Thomas wrote:
>> I updated to master (fab742bd4693ed3092690a20dd32d53fe27c3d4c) and tried again.
>> I don't see any difference - the run using the sstate cache as a mirror
>> seems to do all the same work as without.  Here's how I tested it.
>> 
>> * Build original tree
>>   % POKYCONF=meta-cobra3530p60/conf . /tmp/poky-amltd/poky-init-build-env /local/p60_step1
>>   ... adjust conf/local.conf
>>   % bitbake amltd-console-image
>> 
>> * Rebuild, using previous result for SSTATE_MIRRORS
>>   % POKYCONF=meta-cobra3530p60/conf . /tmp/poky-amltd/poky-init-build-env /local/p60_step2
>>   ... adjust conf/local.conf
>>   % bitbake amltd-console-image
>> 
>> The only difference between the two runs is enabling SSTATE_MIRRORS in local.conf
>>   %  diff -u /local/p60_step?/conf/local.conf
>>    --- /local/p60_step1/conf/local.conf    2011-03-09 08:28:18.266933061 -0700
>>    +++ /local/p60_step2/conf/local.conf    2011-03-09 09:57:51.365932951 -0700
>>    @@ -53,4 +53,7 @@
>>     IMAGE_LINGUAS ?= "en-us"
>> 
>>     # Minimize feature set
>>     DISTRO_FEATURES ?= "alsa"
>>    +SSTATE_MIRRORS ?= "\
>>    +file://.* file:///local/p60_step1/sstate-cache/"
>> 
>> The results seem to have gone through all the same steps (or nearly so).  The output
>> from the runs is at
>>   http://www.mlbassoc.com/poky/build.step1
>>   http://www.mlbassoc.com/poky/build.step2
>> 
>> Comparing the two build trees:
>>   % ls /local/p60_step1/tmp/work/*/*/temp/log.do_compile | wc
>>     144     144   12521
>>   % ls /local/p60_step2/tmp/work/*/*/temp/log.do_compile | wc
>>     143     143   12427
>>   % du -s /local/p60_step?
>>   15229296        /local/p60_step1
>>   15162760        /local/p60_step2
>> 
>> I know this procedure used to work (or at least close).  Am I doing
>> something wrong?
> 
> You're not doing anything wrong and this is the same scenario I've been
> testing with. After I'd fixed the origin problem it created a problem
> with file urls containing globing. I managed to break the original patch
> with the globing fix. The good news is the problem is simple and I've
> pushed a fix.

Is that fix in oe-core as well?

regards,

Koen

Re: [poky] Is sstate broken

[Khem Raj]

On Wed, Mar 9, 2011 at 11:30 AM, Koen Kooi <koen@dominion.thruhere.net> wrote:
>
> Op 9 mrt 2011, om 20:04 heeft Richard Purdie het volgende geschreven:
>
>> On Wed, 2011-03-09 at 11:42 -0700, Gary Thomas wrote:
>>> I updated to master (fab742bd4693ed3092690a20dd32d53fe27c3d4c) and tried again.
>>> I don't see any difference - the run using the sstate cache as a mirror
>>> seems to do all the same work as without.  Here's how I tested it.
>>>
>>> * Build original tree
>>>   % POKYCONF=meta-cobra3530p60/conf . /tmp/poky-amltd/poky-init-build-env /local/p60_step1
>>>   ... adjust conf/local.conf
>>>   % bitbake amltd-console-image
>>>
>>> * Rebuild, using previous result for SSTATE_MIRRORS
>>>   % POKYCONF=meta-cobra3530p60/conf . /tmp/poky-amltd/poky-init-build-env /local/p60_step2
>>>   ... adjust conf/local.conf
>>>   % bitbake amltd-console-image
>>>
>>> The only difference between the two runs is enabling SSTATE_MIRRORS in local.conf
>>>   %  diff -u /local/p60_step?/conf/local.conf
>>>    --- /local/p60_step1/conf/local.conf    2011-03-09 08:28:18.266933061 -0700
>>>    +++ /local/p60_step2/conf/local.conf    2011-03-09 09:57:51.365932951 -0700
>>>    @@ -53,4 +53,7 @@
>>>     IMAGE_LINGUAS ?= "en-us"
>>>
>>>     # Minimize feature set
>>>     DISTRO_FEATURES ?= "alsa"
>>>    +SSTATE_MIRRORS ?= "\
>>>    +file://.* file:///local/p60_step1/sstate-cache/"
>>>
>>> The results seem to have gone through all the same steps (or nearly so).  The output
>>> from the runs is at
>>>   http://www.mlbassoc.com/poky/build.step1
>>>   http://www.mlbassoc.com/poky/build.step2
>>>
>>> Comparing the two build trees:
>>>   % ls /local/p60_step1/tmp/work/*/*/temp/log.do_compile | wc
>>>     144     144   12521
>>>   % ls /local/p60_step2/tmp/work/*/*/temp/log.do_compile | wc
>>>     143     143   12427
>>>   % du -s /local/p60_step?
>>>   15229296        /local/p60_step1
>>>   15162760        /local/p60_step2
>>>
>>> I know this procedure used to work (or at least close).  Am I doing
>>> something wrong?
>>
>> You're not doing anything wrong and this is the same scenario I've been
>> testing with. After I'd fixed the origin problem it created a problem
>> with file urls containing globing. I managed to break the original patch
>> with the globing fix. The good news is the problem is simple and I've
>> pushed a fix.
>
> Is that fix in oe-core as well?

I guess its hardening in yocto and eventually will make into oe-core
which is good.

Re: Is sstate broken

[Khem Raj]

On Wed, Mar 9, 2011 at 11:30 AM, Koen Kooi <koen@dominion.thruhere.net> wrote:
>
> Op 9 mrt 2011, om 20:04 heeft Richard Purdie het volgende geschreven:
>
>> On Wed, 2011-03-09 at 11:42 -0700, Gary Thomas wrote:
>>> I updated to master (fab742bd4693ed3092690a20dd32d53fe27c3d4c) and tried again.
>>> I don't see any difference - the run using the sstate cache as a mirror
>>> seems to do all the same work as without.  Here's how I tested it.
>>>
>>> * Build original tree
>>>   % POKYCONF=meta-cobra3530p60/conf . /tmp/poky-amltd/poky-init-build-env /local/p60_step1
>>>   ... adjust conf/local.conf
>>>   % bitbake amltd-console-image
>>>
>>> * Rebuild, using previous result for SSTATE_MIRRORS
>>>   % POKYCONF=meta-cobra3530p60/conf . /tmp/poky-amltd/poky-init-build-env /local/p60_step2
>>>   ... adjust conf/local.conf
>>>   % bitbake amltd-console-image
>>>
>>> The only difference between the two runs is enabling SSTATE_MIRRORS in local.conf
>>>   %  diff -u /local/p60_step?/conf/local.conf
>>>    --- /local/p60_step1/conf/local.conf    2011-03-09 08:28:18.266933061 -0700
>>>    +++ /local/p60_step2/conf/local.conf    2011-03-09 09:57:51.365932951 -0700
>>>    @@ -53,4 +53,7 @@
>>>     IMAGE_LINGUAS ?= "en-us"
>>>
>>>     # Minimize feature set
>>>     DISTRO_FEATURES ?= "alsa"
>>>    +SSTATE_MIRRORS ?= "\
>>>    +file://.* file:///local/p60_step1/sstate-cache/"
>>>
>>> The results seem to have gone through all the same steps (or nearly so).  The output
>>> from the runs is at
>>>   http://www.mlbassoc.com/poky/build.step1
>>>   http://www.mlbassoc.com/poky/build.step2
>>>
>>> Comparing the two build trees:
>>>   % ls /local/p60_step1/tmp/work/*/*/temp/log.do_compile | wc
>>>     144     144   12521
>>>   % ls /local/p60_step2/tmp/work/*/*/temp/log.do_compile | wc
>>>     143     143   12427
>>>   % du -s /local/p60_step?
>>>   15229296        /local/p60_step1
>>>   15162760        /local/p60_step2
>>>
>>> I know this procedure used to work (or at least close).  Am I doing
>>> something wrong?
>>
>> You're not doing anything wrong and this is the same scenario I've been
>> testing with. After I'd fixed the origin problem it created a problem
>> with file urls containing globing. I managed to break the original patch
>> with the globing fix. The good news is the problem is simple and I've
>> pushed a fix.
>
> Is that fix in oe-core as well?

I guess its hardening in yocto and eventually will make into oe-core
which is good.

Re: [poky] Is sstate broken

[Richard Purdie]

On Wed, 2011-03-09 at 13:51 -0800, Khem Raj wrote:
> On Wed, Mar 9, 2011 at 11:30 AM, Koen Kooi <koen@dominion.thruhere.net> wrote:
> >
> > Op 9 mrt 2011, om 20:04 heeft Richard Purdie het volgende geschreven:
> >
> >> On Wed, 2011-03-09 at 11:42 -0700, Gary Thomas wrote:
> >>> I updated to master (fab742bd4693ed3092690a20dd32d53fe27c3d4c) and tried again.
> >>> I don't see any difference - the run using the sstate cache as a mirror
> >>> seems to do all the same work as without.  Here's how I tested it.
> >>>
> >>> * Build original tree
> >>>   % POKYCONF=meta-cobra3530p60/conf . /tmp/poky-amltd/poky-init-build-env /local/p60_step1
> >>>   ... adjust conf/local.conf
> >>>   % bitbake amltd-console-image
> >>>
> >>> * Rebuild, using previous result for SSTATE_MIRRORS
> >>>   % POKYCONF=meta-cobra3530p60/conf . /tmp/poky-amltd/poky-init-build-env /local/p60_step2
> >>>   ... adjust conf/local.conf
> >>>   % bitbake amltd-console-image
> >>>
> >>> The only difference between the two runs is enabling SSTATE_MIRRORS in local.conf
> >>>   %  diff -u /local/p60_step?/conf/local.conf
> >>>    --- /local/p60_step1/conf/local.conf    2011-03-09 08:28:18.266933061 -0700
> >>>    +++ /local/p60_step2/conf/local.conf    2011-03-09 09:57:51.365932951 -0700
> >>>    @@ -53,4 +53,7 @@
> >>>     IMAGE_LINGUAS ?= "en-us"
> >>>
> >>>     # Minimize feature set
> >>>     DISTRO_FEATURES ?= "alsa"
> >>>    +SSTATE_MIRRORS ?= "\
> >>>    +file://.* file:///local/p60_step1/sstate-cache/"
> >>>
> >>> The results seem to have gone through all the same steps (or nearly so).  The output
> >>> from the runs is at
> >>>   http://www.mlbassoc.com/poky/build.step1
> >>>   http://www.mlbassoc.com/poky/build.step2
> >>>
> >>> Comparing the two build trees:
> >>>   % ls /local/p60_step1/tmp/work/*/*/temp/log.do_compile | wc
> >>>     144     144   12521
> >>>   % ls /local/p60_step2/tmp/work/*/*/temp/log.do_compile | wc
> >>>     143     143   12427
> >>>   % du -s /local/p60_step?
> >>>   15229296        /local/p60_step1
> >>>   15162760        /local/p60_step2
> >>>
> >>> I know this procedure used to work (or at least close).  Am I doing
> >>> something wrong?
> >>
> >> You're not doing anything wrong and this is the same scenario I've been
> >> testing with. After I'd fixed the origin problem it created a problem
> >> with file urls containing globing. I managed to break the original patch
> >> with the globing fix. The good news is the problem is simple and I've
> >> pushed a fix.
> >
> > Is that fix in oe-core as well?
> 
> I guess its hardening in yocto and eventually will make into oe-core
> which is good.

Its a bitbake change and will head into bitbake as soon as we've got
some positive test reports back from Gary and Yocto's QA people.

Cheers,

Richard

Re: Is sstate broken

[Richard Purdie]

On Wed, 2011-03-09 at 13:51 -0800, Khem Raj wrote:
> On Wed, Mar 9, 2011 at 11:30 AM, Koen Kooi <koen@dominion.thruhere.net> wrote:
> >
> > Op 9 mrt 2011, om 20:04 heeft Richard Purdie het volgende geschreven:
> >
> >> On Wed, 2011-03-09 at 11:42 -0700, Gary Thomas wrote:
> >>> I updated to master (fab742bd4693ed3092690a20dd32d53fe27c3d4c) and tried again.
> >>> I don't see any difference - the run using the sstate cache as a mirror
> >>> seems to do all the same work as without.  Here's how I tested it.
> >>>
> >>> * Build original tree
> >>>   % POKYCONF=meta-cobra3530p60/conf . /tmp/poky-amltd/poky-init-build-env /local/p60_step1
> >>>   ... adjust conf/local.conf
> >>>   % bitbake amltd-console-image
> >>>
> >>> * Rebuild, using previous result for SSTATE_MIRRORS
> >>>   % POKYCONF=meta-cobra3530p60/conf . /tmp/poky-amltd/poky-init-build-env /local/p60_step2
> >>>   ... adjust conf/local.conf
> >>>   % bitbake amltd-console-image
> >>>
> >>> The only difference between the two runs is enabling SSTATE_MIRRORS in local.conf
> >>>   %  diff -u /local/p60_step?/conf/local.conf
> >>>    --- /local/p60_step1/conf/local.conf    2011-03-09 08:28:18.266933061 -0700
> >>>    +++ /local/p60_step2/conf/local.conf    2011-03-09 09:57:51.365932951 -0700
> >>>    @@ -53,4 +53,7 @@
> >>>     IMAGE_LINGUAS ?= "en-us"
> >>>
> >>>     # Minimize feature set
> >>>     DISTRO_FEATURES ?= "alsa"
> >>>    +SSTATE_MIRRORS ?= "\
> >>>    +file://.* file:///local/p60_step1/sstate-cache/"
> >>>
> >>> The results seem to have gone through all the same steps (or nearly so).  The output
> >>> from the runs is at
> >>>   http://www.mlbassoc.com/poky/build.step1
> >>>   http://www.mlbassoc.com/poky/build.step2
> >>>
> >>> Comparing the two build trees:
> >>>   % ls /local/p60_step1/tmp/work/*/*/temp/log.do_compile | wc
> >>>     144     144   12521
> >>>   % ls /local/p60_step2/tmp/work/*/*/temp/log.do_compile | wc
> >>>     143     143   12427
> >>>   % du -s /local/p60_step?
> >>>   15229296        /local/p60_step1
> >>>   15162760        /local/p60_step2
> >>>
> >>> I know this procedure used to work (or at least close).  Am I doing
> >>> something wrong?
> >>
> >> You're not doing anything wrong and this is the same scenario I've been
> >> testing with. After I'd fixed the origin problem it created a problem
> >> with file urls containing globing. I managed to break the original patch
> >> with the globing fix. The good news is the problem is simple and I've
> >> pushed a fix.
> >
> > Is that fix in oe-core as well?
> 
> I guess its hardening in yocto and eventually will make into oe-core
> which is good.

Its a bitbake change and will head into bitbake as soon as we've got
some positive test reports back from Gary and Yocto's QA people.

Cheers,

Richard

Re: Is sstate broken

[Gary Thomas]

On 03/09/11 12:04, Richard Purdie wrote:
> On Wed, 2011-03-09 at 11:42 -0700, Gary Thomas wrote:
>> I updated to master (fab742bd4693ed3092690a20dd32d53fe27c3d4c) and tried again.
>> I don't see any difference - the run using the sstate cache as a mirror
>> seems to do all the same work as without.  Here's how I tested it.
>>
>> * Build original tree
>>     % POKYCONF=meta-cobra3530p60/conf . /tmp/poky-amltd/poky-init-build-env /local/p60_step1
>>     ... adjust conf/local.conf
>>     % bitbake amltd-console-image
>>
>> * Rebuild, using previous result for SSTATE_MIRRORS
>>     % POKYCONF=meta-cobra3530p60/conf . /tmp/poky-amltd/poky-init-build-env /local/p60_step2
>>     ... adjust conf/local.conf
>>     % bitbake amltd-console-image
>>
>> The only difference between the two runs is enabling SSTATE_MIRRORS in local.conf
>>     %  diff -u /local/p60_step?/conf/local.conf
>>      --- /local/p60_step1/conf/local.conf    2011-03-09 08:28:18.266933061 -0700
>>      +++ /local/p60_step2/conf/local.conf    2011-03-09 09:57:51.365932951 -0700
>>      @@ -53,4 +53,7 @@
>>       IMAGE_LINGUAS ?= "en-us"
>>
>>       # Minimize feature set
>>       DISTRO_FEATURES ?= "alsa"
>>      +SSTATE_MIRRORS ?= "\
>>      +file://.* file:///local/p60_step1/sstate-cache/"
>>
>> The results seem to have gone through all the same steps (or nearly so).  The output
>> from the runs is at
>>     http://www.mlbassoc.com/poky/build.step1
>>     http://www.mlbassoc.com/poky/build.step2
>>
>> Comparing the two build trees:
>>     % ls /local/p60_step1/tmp/work/*/*/temp/log.do_compile | wc
>>       144     144   12521
>>     % ls /local/p60_step2/tmp/work/*/*/temp/log.do_compile | wc
>>       143     143   12427
>>     % du -s /local/p60_step?
>>     15229296        /local/p60_step1
>>     15162760        /local/p60_step2
>>
>> I know this procedure used to work (or at least close).  Am I doing
>> something wrong?
>
> You're not doing anything wrong and this is the same scenario I've been
> testing with. After I'd fixed the origin problem it created a problem
> with file urls containing globing. I managed to break the original patch
> with the globing fix. The good news is the problem is simple and I've
> pushed a fix.
>
> It should work *much* better than you've seen above, not needing to
> rerun many tasks so if its working it will be very obvious.

Yes, indeed it does!  I just reran step2 from scratch and it finished
in only 13 minutes (with a lot of other things on the build machine so
this is quite good).  There's still quite a lot of stuff in the build
tree:
   % du -s /local/p60_step2
   7285332 /local/p60_step2
but that's only about 1/2 of before.  The results seem very close:
   % ls -l /local/p60_step2/tmp/deploy/images/
   total 10748
   -rwxrw-r--  1 gthomas gthomas 5446150 Mar  9 13:15 amltd-console-image-cobra3530p60-20110309200339.rootfs.tar.bz2
   lrwxrwxrwx  1 gthomas gthomas      62 Mar  9 13:15 amltd-console-image-cobra3530p60.tar.bz2 -> amltd-console-image-cobra3530p60-20110309200339.rootfs.tar.bz2
   drwxrwxr-x 91 gthomas gthomas    4096 Mar  9 13:08 licenses
   -rw-rw-r--  1 gthomas gthomas 3077911 Mar  9 09:24 modules-2.6.37-r0-cobra3530p60.tgz
   -rw-r--r--  1 gthomas gthomas 2468564 Mar  9 09:24 uImage-2.6.37-r0-cobra3530p60-20110309153435.bin
   lrwxrwxrwx  1 gthomas gthomas      48 Mar  9 09:24 uImage-cobra3530p60.bin -> uImage-2.6.37-r0-cobra3530p60-20110309153435.bin
   % ls -l /local/p60_step1/tmp/deploy/images/
   total 10748
   -rwxrw-r--  1 gthomas gthomas 5446783 Mar  9 09:53 amltd-console-image-cobra3530p60-20110309153435.rootfs.tar.bz2
   lrwxrwxrwx  1 gthomas gthomas      62 Mar  9 09:53 amltd-console-image-cobra3530p60.tar.bz2 -> amltd-console-image-cobra3530p60-20110309153435.rootfs.tar.bz2
   drwxrwxr-x 92 gthomas gthomas    4096 Mar  9 09:06 licenses
   -rw-rw-r--  1 gthomas gthomas 3077911 Mar  9 09:24 modules-2.6.37-r0-cobra3530p60.tgz
   -rw-r--r--  1 gthomas gthomas 2468564 Mar  9 09:24 uImage-2.6.37-r0-cobra3530p60-20110309153435.bin
   lrwxrwxrwx  1 gthomas gthomas      48 Mar  9 09:24 uImage-cobra3530p60.bin -> uImage-2.6.37-r0-cobra3530p60-20110309153435.bin

I unpacked the resulting image tarballs and they match, modulo the
package install times in the opkg database.

Looks like it's working correctly now :-)

Can you explain what the sstate files are and how I might make use of them?
In particular, I'd like to be able to hand my customers a sstate cache which
holds the majority of the tools they might need to speed startup.  Sending
the whole thing is a bit problematic as it's quite large, so if I can winnow
it a bit it would help.

Thanks for getting this back in order

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------