How to force BitBake to track dependency to METADATA_REVISION?

How to force BitBake to track dependency to METADATA_REVISION?

[Kucharczyk, Bartlomiej (Nokia - PL/Wroclaw)]

Hello!

I am trying to use METADATA_REVISION variable (defined in metadata_scm.bbclass) in my recipe, but BitBake does not rebuild the task when git revision of the repository changes.
Setting [vardeps] does not help.

I suspect this may be related to the fact that METADATA_REVISION is defined this way:
	METADATA_REVISION ?= "${@base_detect_revision(d)}"
And BitBake does not expand this to the proper value, thus does not see any difference.

Does anyone have any suggestion how to tackle this issue, and force BitBake to properly track dependency to METADATA_REVISION value?

My original task is:
Package feeds/SDK installers are versioned (git metadata revision). I am trying to create an opkg.conf file which will contain such versioned package feeds URL in SDK installer. For build reproducibility, and user convenience.

Thanks in advance!
Bartek

Re: How to force BitBake to track dependency to METADATA_REVISION?

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 1417 bytes --]

On Thu, May 5, 2016 at 11:48 AM, Kucharczyk, Bartlomiej (Nokia -
PL/Wroclaw) <bartlomiej.kucharczyk@nokia.com> wrote:

> I am trying to use METADATA_REVISION variable (defined in
> metadata_scm.bbclass) in my recipe, but BitBake does not rebuild the task
> when git revision of the repository changes.
> Setting [vardeps] does not help.
>
> I suspect this may be related to the fact that METADATA_REVISION is
> defined this way:
>         METADATA_REVISION ?= "${@base_detect_revision(d)}"
> And BitBake does not expand this to the proper value, thus does not see
> any difference.
>
> Does anyone have any suggestion how to tackle this issue, and force
> BitBake to properly track dependency to METADATA_REVISION value?
>
> My original task is:
> Package feeds/SDK installers are versioned (git metadata revision). I am
> trying to create an opkg.conf file which will contain such versioned
> package feeds URL in SDK installer. For build reproducibility, and user
> convenience.
>

Bitbake tracks unexpanded forms of variables, not expanded. So it's
literally storing '${@base_detect_revision(d)}" in the checksum, not the
actual revision.

METADATA_REVISION[vardepvalue] = "${METADATA_REVISION}" in your recipe will
do it.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 1894 bytes --]

Re: How to force BitBake to track dependency to METADATA_REVISION?

[Kucharczyk, Bartlomiej (Nokia - PL/Wroclaw)]

> Bitbake tracks unexpanded forms of variables, not expanded. So it's literally storing '${@base_detect_revision(d)}" in the checksum, not the actual revision.
> METADATA_REVISION[vardepvalue] = "${METADATA_REVISION}" in your recipe will do it.

Hello! 

Thanks for your answer. I've tried to use vardepvalue, but it does not work... After I build it for the second time (after bumping git revision), nothing is rebuilt.

    $ bitbake test
    ...
    NOTE: Tasks Summary: Attempted 251 tasks of which 0 didn't need to be rerun and all succeeded.
    ...
    $ echo "" >> ../README
    $ git commit -m"test" ../README
    $ bitbake test
    ...
    NOTE: Tasks Summary: Attempted 251 tasks of which 251 didn't need to be rerun and all succeeded.
    ...

My test recipe is:

    $ cat ../meta/recipes-devtools/test/test_1.0.bb
    LICENSE = "CLOSED"
    INHIBIT_DEFAULT_DEPS = "1"
    METADATA_REVISION[vardepvalue] = "${METADATA_REVISION}"
    do_compile(){
        echo "${METADATA_REVISION}" > "${S}/testfile"
    }

Any suggestions?

Bartek

Re: How to force BitBake to track dependency to METADATA_REVISION?

[Richard Purdie]

On Fri, 2016-05-06 at 10:10 +0000, Kucharczyk, Bartlomiej (Nokia -
PL/Wroclaw) wrote:
> > Bitbake tracks unexpanded forms of variables, not expanded. So it's
> > literally storing '${@base_detect_revision(d)}" in the checksum,
> > not the actual revision.
> > METADATA_REVISION[vardepvalue] = "${METADATA_REVISION}" in your
> > recipe will do it.
> 
> Hello! 
> 
> Thanks for your answer. I've tried to use vardepvalue, but it does
> not work... After I build it for the second time (after bumping git
> revision), nothing is rebuilt.
> 
>     $ bitbake test
>     ...
>     NOTE: Tasks Summary: Attempted 251 tasks of which 0 didn't need
> to be rerun and all succeeded.
>     ...
>     $ echo "" >> ../README
>     $ git commit -m"test" ../README
>     $ bitbake test
>     ...
>     NOTE: Tasks Summary: Attempted 251 tasks of which 251 didn't need
> to be rerun and all succeeded.
>     ...
> 
> My test recipe is:
> 
>     $ cat ../meta/recipes-devtools/test/test_1.0.bb
>     LICENSE = "CLOSED"
>     INHIBIT_DEFAULT_DEPS = "1"
>     METADATA_REVISION[vardepvalue] = "${METADATA_REVISION}"
>     do_compile(){
>         echo "${METADATA_REVISION}" > "${S}/testfile"
>     }
> 
> Any suggestions?

The problem is probably the cache. It has no knowledge of when it needs
to reparse to see if this value has changed. If you set BB_DONT_CACHE =
"1" in the recipe, does that help?

Cheers,

Richard

Re: How to force BitBake to track dependency to METADATA_REVISION?

[Kucharczyk, Bartlomiej (Nokia - PL/Wroclaw)]

> The problem is probably the cache. It has no knowledge of when it needs
> to reparse to see if this value has changed. If you set BB_DONT_CACHE =
> "1" in the recipe, does that help?

Yes! That helps.
Thanks for those hints!

What BB_DONT_CACHE does exactly? I found it undocumented. ;-)
I understand this is config cache, not the 'sstate cache'.

Bartek

Re: How to force BitBake to track dependency to METADATA_REVISION?

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 842 bytes --]

On Fri, May 6, 2016 at 6:32 AM, Kucharczyk, Bartlomiej (Nokia - PL/Wroclaw)
<bartlomiej.kucharczyk@nokia.com> wrote:

> > The problem is probably the cache. It has no knowledge of when it needs
> > to reparse to see if this value has changed. If you set BB_DONT_CACHE =
> > "1" in the recipe, does that help?
>
> Yes! That helps.
> Thanks for those hints!
>
> What BB_DONT_CACHE does exactly? I found it undocumented. ;-)
> I understand this is config cache, not the 'sstate cache'.


It's not the config cache or the 'sstate' cache', it's the parse cache.
Recipes are reparsed when the recipe files or classes change. They aren't
re-parsed every time by default.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 1275 bytes --]