sstate cache not used when adding custom task

sstate cache not used when adding custom task

[Andreas Fenkart]

Hi,

A custom task in a bbclass causes all recipes including it to be
rebuilt every time. Link to full bbclass attached below.

I already stripped that task down to this


    addtask check after do_compile before do_build
    addons_do_check() {
        :
    }

If I remove the addtask line, the recipe is not rerun:

$ bitbake recipe_name
NOTE: Tasks Summary: Attempted 2746 tasks of which 2746 didn't need to
be rerun and all succeeded.

With the 'addtask check' all task of the recipe from fetch till
rm_work_all is executed every time.

From the cooker log, do_populate_sysroot is the last task running
before do_build was, hence I changed the addtask line to this:
addtask check after do_populate_sysroot before do_build

Then the recipe will not start from do_fetch on rerun, but will
reuse the previous do_populate_sysroot run. Hence only the last 4
tasks are run (do_check, do_build, do_rm_work, do_rm_work_all)

I experimented with bitbake-diffsig/bitbake-dumpsig, but the hash
of the stamp files (do_check) are identical on each run, of
course its content too. Only the timestamp of the stamp file
changes each run.
I'm confused what triggers the rebuild. I looked into bitbake
trying to understand how the task dependency is calculated, how
the sstate files are used. But it's quite a biest and will take
me quite some time to understand.

Any hints or regarding the issue or simply where to put a print
in bitbake to get me started was welcome.

This is still yocto 2.0.2 as we missed a couple of upgrades.

The full bbclass with the do_check task
https://git.digitalstrom.org/dss-oe/dss-oe/blob/master/yocto/dS/meta-digitalstrom-devel/classes/addons.bbclass#L46

/Andreas

Re: sstate cache not used when adding custom task

[Joshua Lock]

On Thu, 2017-07-13 at 10:52 +0200, Andreas Fenkart wrote:
> Hi,
> 
> A custom task in a bbclass causes all recipes including it to be
> rebuilt every time. Link to full bbclass attached below.
> 
> I already stripped that task down to this
> 
> 
>     addtask check after do_compile before do_build
>     addons_do_check() {
>         :
>     }
> 
> If I remove the addtask line, the recipe is not rerun:

The shared state machinery doesn't know how to capture your task. There
are some guidelines on adding shared state wrapping to a task in the
Yocto Project reference manual Shared State Cache section:

http://www.yoctoproject.org/docs/latest/ref-manual/ref-manual.html#shar
ed-state-cache
http://www.yoctoproject.org/docs/latest/ref-manual/ref-manual.html#shar
ed-state

Joshua

> $ bitbake recipe_name
> NOTE: Tasks Summary: Attempted 2746 tasks of which 2746 didn't need
> to
> be rerun and all succeeded.
> 
> With the 'addtask check' all task of the recipe from fetch till
> rm_work_all is executed every time.
> 
> From the cooker log, do_populate_sysroot is the last task running
> before do_build was, hence I changed the addtask line to this:
> addtask check after do_populate_sysroot before do_build
> 
> Then the recipe will not start from do_fetch on rerun, but will
> reuse the previous do_populate_sysroot run. Hence only the last 4
> tasks are run (do_check, do_build, do_rm_work, do_rm_work_all)
> 
> I experimented with bitbake-diffsig/bitbake-dumpsig, but the hash
> of the stamp files (do_check) are identical on each run, of
> course its content too. Only the timestamp of the stamp file
> changes each run.
> I'm confused what triggers the rebuild. I looked into bitbake
> trying to understand how the task dependency is calculated, how
> the sstate files are used. But it's quite a biest and will take
> me quite some time to understand.
> 
> Any hints or regarding the issue or simply where to put a print
> in bitbake to get me started was welcome.
> 
> This is still yocto 2.0.2 as we missed a couple of upgrades.
> 
> The full bbclass with the do_check task
> https://git.digitalstrom.org/dss-oe/dss-oe/blob/master/yocto/dS/meta-
> digitalstrom-devel/classes/addons.bbclass#L46
> 
> /Andreas

Re: sstate cache not used when adding custom task

[Andreas Fenkart]

Hi,

2017-07-13 11:30 GMT+02:00 Joshua Lock <joshua.g.lock@linux.intel.com>:
> On Thu, 2017-07-13 at 10:52 +0200, Andreas Fenkart wrote:
>> Hi,
>>
>> A custom task in a bbclass causes all recipes including it to be
>> rebuilt every time. Link to full bbclass attached below.
>>
>> I already stripped that task down to this
>>
>>
>>     addtask check after do_compile before do_build
>>     addons_do_check() {
>>         :
>>     }
>>
>> If I remove the addtask line, the recipe is not rerun:
>
> The shared state machinery doesn't know how to capture your task. There
> are some guidelines on adding shared state wrapping to a task in the
> Yocto Project reference manual Shared State Cache section:
>
> http://www.yoctoproject.org/docs/latest/ref-manual/ref-manual.html#shar
> ed-state-cache
> http://www.yoctoproject.org/docs/latest/ref-manual/ref-manual.html#shar
> ed-state
>
> Joshua

That did the job, thank you

>
>> $ bitbake recipe_name
>> NOTE: Tasks Summary: Attempted 2746 tasks of which 2746 didn't need
>> to
>> be rerun and all succeeded.
>>
>> With the 'addtask check' all task of the recipe from fetch till
>> rm_work_all is executed every time.
>>
>> From the cooker log, do_populate_sysroot is the last task running
>> before do_build was, hence I changed the addtask line to this:
>> addtask check after do_populate_sysroot before do_build
>>
>> Then the recipe will not start from do_fetch on rerun, but will
>> reuse the previous do_populate_sysroot run. Hence only the last 4
>> tasks are run (do_check, do_build, do_rm_work, do_rm_work_all)
>>
>> I experimented with bitbake-diffsig/bitbake-dumpsig, but the hash
>> of the stamp files (do_check) are identical on each run, of
>> course its content too. Only the timestamp of the stamp file
>> changes each run.
>> I'm confused what triggers the rebuild. I looked into bitbake
>> trying to understand how the task dependency is calculated, how
>> the sstate files are used. But it's quite a biest and will take
>> me quite some time to understand.
>>
>> Any hints or regarding the issue or simply where to put a print
>> in bitbake to get me started was welcome.
>>
>> This is still yocto 2.0.2 as we missed a couple of upgrades.
>>
>> The full bbclass with the do_check task
>> https://git.digitalstrom.org/dss-oe/dss-oe/blob/master/yocto/dS/meta-
>> digitalstrom-devel/classes/addons.bbclass#L46
>>
>> /Andreas