RFC: exposing information about the SRC_URI(s)/branch via buildhistory (or similar mechanism)

RFC: exposing information about the SRC_URI(s)/branch via buildhistory (or similar mechanism)

[chris.laplante]

[-- Attachment #1: Type: text/plain, Size: 1472 bytes --]

Hello all,

Most of my team's closed source recipes use something like the following:

SRC_URI = "git://git@host/path;protocol=ssh;branch=${BRANCH}"
SRCREV = "${AUTOREV}"
BRANCH ??= "master"

(BRANCH is just a convention we use to make the SRC_URI branch easy to override.)

This makes nightly builds convenient because we always build from the latest. For release versions, we can use SRCREV_pn-recipe and/or BRANCH_pn-recipe overrides in local.conf. We get the SRCREV overrides using buildhistory-collect-srcrevs. But buildhistory has no notion or tracking of SRC_URI or branches, so currently we use a script that generates the BRANCH overrides.

I'm interesting in adding SRC_URI support to buildhistory (or a similar mechanism), and would like to get some input.

Option 1) The easiest way, I think, is to just generate SRC_URI_pn- overrides with variable expansion.

Option 2) I think it could be useful to introduce BRANCH as a convention. Currently the "branch" SRC_URI parameter implicitly defaults to "master". I could foresee it implicitly defaulting to "${BRANCH}", with BRANCH ??= "master" to replicate existing functionality. To handle multiple source-controlled SRC_URIs, we'd have to do something similar to how SRCREV has a "name" override. With this option, I wouldn't think it would be necessary to generate SRC_URI overrides; just BRANCH overrides.

Option 3) A combination of 1 and 2?

Looking forward to input.

Thanks,
Chris

[-- Attachment #2: Type: text/html, Size: 4612 bytes --]

Re: [bitbake-devel] RFC: exposing information about the SRC_URI(s)/branch via buildhistory (or similar mechanism)

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 2503 bytes --]

Apologies, but I think you should drop the practice of using AUTOREV. This
completely destroys reproducibility of builds, and makes them susceptible
to global breakage when someone pushes a broken commit into one of the
component repositories.

Yes, bumping SRCREV is annoying, but a) it can be automated; b) you can
catch and reject any problems with the build at that point, so you always
have a nightly that works.

AUTOREV was never meant for the project level, it is a facility strictly
for local development.

Alex

On Thu, 1 Aug 2019 at 18:51, chris.laplante--- via bitbake-devel <
bitbake-devel@lists.openembedded.org> wrote:

> Hello all,
>
>
>
> Most of my team’s closed source recipes use something like the following:
>
>
>
> SRC_URI = "git://git@host/path;protocol=ssh;branch=${BRANCH}"
>
> SRCREV = “${AUTOREV}”
>
> BRANCH ??= “master”
>
>
>
> (BRANCH is just a convention we use to make the SRC_URI branch easy to
> override.)
>
>
>
> This makes nightly builds convenient because we always build from the
> latest. For release versions, we can use SRCREV_pn-recipe and/or
> BRANCH_pn-recipe overrides in local.conf. We get the SRCREV overrides using
> buildhistory-collect-srcrevs. But buildhistory has no notion or tracking of
> SRC_URI or branches, so currently we use a script that generates the BRANCH
> overrides.
>
>
>
> I’m interesting in adding SRC_URI support to buildhistory (or a similar
> mechanism), and would like to get some input.
>
>
>
> Option 1) The easiest way, I think, is to just generate SRC_URI_pn-
> overrides with variable expansion.
>
>
>
> Option 2) I think it could be useful to introduce BRANCH as a convention.
> Currently the “branch” SRC_URI parameter implicitly defaults to “master”. I
> could foresee it implicitly defaulting to “${BRANCH}”, with BRANCH ??=
> “master” to replicate existing functionality. To handle multiple
> source-controlled SRC_URIs, we’d have to do something similar to how SRCREV
> has a “name” override. With this option, I wouldn’t think it would be
> necessary to generate SRC_URI overrides; just BRANCH overrides.
>
>
>
> Option 3) A combination of 1 and 2?
>
>
>
> Looking forward to input.
>
>
>
> Thanks,
>
> Chris
> --
> _______________________________________________
> bitbake-devel mailing list
> bitbake-devel@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/bitbake-devel
>

[-- Attachment #2: Type: text/html, Size: 4065 bytes --]

Re: [bitbake-devel] RFC: exposing information about the SRC_URI(s)/branch via buildhistory (or similar mechanism)

[chris.laplante]

[-- Attachment #1: Type: text/plain, Size: 3048 bytes --]

We’re well aware of the limitations and possibility of failed builds, as I said we hardcode revisions using SRCREV overrides when necessary. It’s been working well for us so far. So, respectfully, I think that’s not relevant to the discussion.

Chris

From: Alexander Kanavin <alex.kanavin@gmail.com>
Sent: Thursday, August 01, 2019 12:59 PM
To: LAPLANTE,CHRIS (Agilent USA) <chris.laplante@agilent.com>
Cc: Openembedded-core@lists.openembedded.org; bitbake-devel@lists.openembedded.org
Subject: Re: [bitbake-devel] RFC: exposing information about the SRC_URI(s)/branch via buildhistory (or similar mechanism)

Apologies, but I think you should drop the practice of using AUTOREV. This completely destroys reproducibility of builds, and makes them susceptible to global breakage when someone pushes a broken commit into one of the component repositories.

Yes, bumping SRCREV is annoying, but a) it can be automated; b) you can catch and reject any problems with the build at that point, so you always have a nightly that works.

AUTOREV was never meant for the project level, it is a facility strictly for local development.

Alex

On Thu, 1 Aug 2019 at 18:51, chris.laplante--- via bitbake-devel <bitbake-devel@lists.openembedded.org<mailto:bitbake-devel@lists.openembedded.org>> wrote:
Hello all,

Most of my team’s closed source recipes use something like the following:

SRC_URI = "git://git@host/path;protocol=ssh;branch=${BRANCH}"
SRCREV = “${AUTOREV}”
BRANCH ??= “master”

(BRANCH is just a convention we use to make the SRC_URI branch easy to override.)

This makes nightly builds convenient because we always build from the latest. For release versions, we can use SRCREV_pn-recipe and/or BRANCH_pn-recipe overrides in local.conf. We get the SRCREV overrides using buildhistory-collect-srcrevs. But buildhistory has no notion or tracking of SRC_URI or branches, so currently we use a script that generates the BRANCH overrides.

I’m interesting in adding SRC_URI support to buildhistory (or a similar mechanism), and would like to get some input.

Option 1) The easiest way, I think, is to just generate SRC_URI_pn- overrides with variable expansion.

Option 2) I think it could be useful to introduce BRANCH as a convention. Currently the “branch” SRC_URI parameter implicitly defaults to “master”. I could foresee it implicitly defaulting to “${BRANCH}”, with BRANCH ??= “master” to replicate existing functionality. To handle multiple source-controlled SRC_URIs, we’d have to do something similar to how SRCREV has a “name” override. With this option, I wouldn’t think it would be necessary to generate SRC_URI overrides; just BRANCH overrides.

Option 3) A combination of 1 and 2?

Looking forward to input.

Thanks,
Chris
--
_______________________________________________
bitbake-devel mailing list
bitbake-devel@lists.openembedded.org<mailto:bitbake-devel@lists.openembedded.org>
http://lists.openembedded.org/mailman/listinfo/bitbake-devel

[-- Attachment #2: Type: text/html, Size: 8793 bytes --]

Re: [bitbake-devel] RFC: exposing information about the SRC_URI(s)/branch via buildhistory (or similar mechanism)

[Mark Hatle]

On 8/1/19 11:51 AM, chris.laplante--- via bitbake-devel wrote:
> Hello all,
> 
>  
> 
> Most of my team’s closed source recipes use something like the following:
> 
>  
> 
> SRC_URI = "git://git@host/path;protocol=ssh;branch=${BRANCH}"
> 
> SRCREV = “${AUTOREV}”
> 
> BRANCH ??= “master”
> 
>  
> 
> (BRANCH is just a convention we use to make the SRC_URI branch easy to override.)
> 
>  
> 
> This makes nightly builds convenient because we always build from the latest.
> For release versions, we can use SRCREV_pn-recipe and/or BRANCH_pn-recipe
> overrides in local.conf. We get the SRCREV overrides using
> buildhistory-collect-srcrevs. But buildhistory has no notion or tracking of
> SRC_URI or branches, so currently we use a script that generates the BRANCH
> overrides.
> 
>  
> 
> I’m interesting in adding SRC_URI support to buildhistory (or a similar
> mechanism), and would like to get some input.
> 
>  
> 
> Option 1) The easiest way, I think, is to just generate SRC_URI_pn- overrides
> with variable expansion.
> 
>  
> 
> Option 2) I think it could be useful to introduce BRANCH as a convention.
> Currently the “branch” SRC_URI parameter implicitly defaults to “master”. I
> could foresee it implicitly defaulting to “${BRANCH}”, with BRANCH ??= “master”
> to replicate existing functionality. To handle multiple source-controlled
> SRC_URIs, we’d have to do something similar to how SRCREV has a “name” override.
> With this option, I wouldn’t think it would be necessary to generate SRC_URI
> overrides; just BRANCH overrides.
> 
>  
> 
> Option 3) A combination of 1 and 2?
> 

I've had some thoughts of capturing that same information, as well as
automatically generating a changelog (from short-commit entries) from the last
build to the current one use standard git commands... but I've never had time to
implement anything.

When I talked with RP about this a while back, he mentioned it should be
possible to extend the functionality of buildhistory with an additional class.
So the user could opt into extended functionality by including more then one
buildhistory class.

So my recommendation, store the URL you pulled from along with the SRCREV (if
it's not already been stored).  This can then be used later to investigate -- or
extended [internally or externally] to create changelogs.

--Mark

> 
> Looking forward to input.
> 
>  
> 
> Thanks,
> 
> Chris
> 
> 

Re: UNVERIFIED SENDER Re: [bitbake-devel] RFC: exposing information about the SRC_URI(s)/branch via buildhistory (or similar mechanism)

[chris.laplante]

> I've had some thoughts of capturing that same information, as well as
> automatically generating a changelog (from short-commit entries) from the last
> build to the current one use standard git commands... but I've never had time to
> implement anything.
> 
> When I talked with RP about this a while back, he mentioned it should be
> possible to extend the functionality of buildhistory with an additional class.
> So the user could opt into extended functionality by including more then one
> buildhistory class.
> 
> So my recommendation, store the URL you pulled from along with the SRCREV (if
> it's not already been stored).  This can then be used later to investigate -- or
> extended [internally or externally] to create changelogs.

Wouldn't it be a bit overkill to add a new class just to handle SRC_URI? I would think it would just be a little extra code inside write_latest_srcrev.

Chris 

Re: UNVERIFIED SENDER Re: [bitbake-devel] RFC: exposing information about the SRC_URI(s)/branch via buildhistory (or similar mechanism)

[Mark Hatle]

On 8/1/19 1:03 PM, chris.laplante@agilent.com wrote:
>> I've had some thoughts of capturing that same information, as well as
>> automatically generating a changelog (from short-commit entries) from the last
>> build to the current one use standard git commands... but I've never had time to
>> implement anything.
>>
>> When I talked with RP about this a while back, he mentioned it should be
>> possible to extend the functionality of buildhistory with an additional class.
>> So the user could opt into extended functionality by including more then one
>> buildhistory class.
>>
>> So my recommendation, store the URL you pulled from along with the SRCREV (if
>> it's not already been stored).  This can then be used later to investigate -- or
>> extended [internally or externally] to create changelogs.
> 
> Wouldn't it be a bit overkill to add a new class just to handle SRC_URI? I would think it would just be a little extra code inside write_latest_srcrev.

Ordinarily I'd say yes.  But RP had concerns that these changes add up and
unless everyone wants them it could prove to be a maintenance problem.  At the
time he wanted a system for people to opt in to new functionality.  (I'm not
trying to speak for him, just repeating what I was told when I asked him a
similar question.)   In my case though I was suggesting much more functionality
then just this one item.

--Mark

> Chris 
> 

Re: UNVERIFIED SENDER Re: UNVERIFIED SENDER Re: [bitbake-devel] RFC: exposing information about the SRC_URI(s)/branch via buildhistory (or similar mechanism)

[chris.laplante]

> > Wouldn't it be a bit overkill to add a new class just to handle SRC_URI? I would think it would just be a little extra code inside
> write_latest_srcrev.
> 
> Ordinarily I'd say yes.  But RP had concerns that these changes add up and
> unless everyone wants them it could prove to be a maintenance problem.  At the
> time he wanted a system for people to opt in to new functionality.  (I'm not
> trying to speak for him, just repeating what I was told when I asked him a
> similar question.)   In my case though I was suggesting much more functionality
> then just this one item.

Makes sense. I definitely like the idea of the automatic changelog.

For now, as a quick prototype of the basic functionality (capturing SRC_URI), I will do as the bbclass suggests and extend BUILDHISTORY_PRESERVE.

Thanks,
Chris

Re: RFC: exposing information about the SRC_URI(s)/branch via buildhistory (or similar mechanism)

[Mikko.Rapeli]

Hi,

On Thu, Aug 01, 2019 at 04:51:38PM +0000, chris.laplante--- via Openembedded-core wrote:
<snip>
> I'm interesting in adding SRC_URI support to buildhistory (or a similar mechanism), and would like to get some input.

Yes to this.

Also would be nice if there was an easy way to add bitbake variables to
buildhistory.

I've patched our tree so that SRC_URI, LICENSE and CVE_PRODUCT are archived
in buildhistory. SRC_URI has many uses and changes and patches can
be easily identified. Same with LICENSE, any changes trigger a review.
CVE_PRODUCT is exported so that we can do QA check to make sure mapping
from CVE_PRODUCT for non CLOSED licenses exists to NVD database product
names (maintaining a white list of recipes which don't have any CVEs yet).

We've also changed the SDK name to be stable across builds and added
DISTRO to the path. In our case IMAGE_NAME and SDK_NAME will include
git tree tag and hash if tree is dirty, which changes buildhistory SDK paths
for every build with different input.

I could submit the patches if there is interest in them.

Cheers,

-Mikko

Re: RFC: exposing information about the SRC_URI(s)/branch via buildhistory (or similar mechanism)

[William Bourque]

I'd be curious to see the patches.
It's definitely something we could use here; we used to have nightly
build checking the build using AUTOREV as well.

On Fri, Aug 2, 2019 at 3:43 AM <Mikko.Rapeli@bmw.de> wrote:
>
> Hi,
>
> On Thu, Aug 01, 2019 at 04:51:38PM +0000, chris.laplante--- via Openembedded-core wrote:
> <snip>
> > I'm interesting in adding SRC_URI support to buildhistory (or a similar mechanism), and would like to get some input.
>
> Yes to this.
>
> Also would be nice if there was an easy way to add bitbake variables to
> buildhistory.
>
> I've patched our tree so that SRC_URI, LICENSE and CVE_PRODUCT are archived
> in buildhistory. SRC_URI has many uses and changes and patches can
> be easily identified. Same with LICENSE, any changes trigger a review.
> CVE_PRODUCT is exported so that we can do QA check to make sure mapping
> from CVE_PRODUCT for non CLOSED licenses exists to NVD database product
> names (maintaining a white list of recipes which don't have any CVEs yet).
>
> We've also changed the SDK name to be stable across builds and added
> DISTRO to the path. In our case IMAGE_NAME and SDK_NAME will include
> git tree tag and hash if tree is dirty, which changes buildhistory SDK paths
> for every build with different input.
>
> I could submit the patches if there is interest in them.
>
> Cheers,
>
> -Mikko
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: RFC: exposing information about the SRC_URI(s)/branch via buildhistory (or similar mechanism)

[chris.laplante]

> I'd be curious to see the patches.
> It's definitely something we could use here; we used to have nightly
> build checking the build using AUTOREV as well.
> 
> On Fri, Aug 2, 2019 at 3:43 AM <Mikko.Rapeli@bmw.de> wrote:
> >
> > Hi,
> >
> > On Thu, Aug 01, 2019 at 04:51:38PM +0000, chris.laplante--- via Openembedded-core wrote:
> > <snip>
> > > I'm interesting in adding SRC_URI support to buildhistory (or a similar mechanism), and would like to get some input.
> >
> > Yes to this.
> >
> > Also would be nice if there was an easy way to add bitbake variables to
> > buildhistory.
> >
> > I've patched our tree so that SRC_URI, LICENSE and CVE_PRODUCT are archived
> > in buildhistory. SRC_URI has many uses and changes and patches can
> > be easily identified. Same with LICENSE, any changes trigger a review.
> > CVE_PRODUCT is exported so that we can do QA check to make sure mapping
> > from CVE_PRODUCT for non CLOSED licenses exists to NVD database product
> > names (maintaining a white list of recipes which don't have any CVEs yet).
> >
> > We've also changed the SDK name to be stable across builds and added
> > DISTRO to the path. In our case IMAGE_NAME and SDK_NAME will include
> > git tree tag and hash if tree is dirty, which changes buildhistory SDK paths
> > for every build with different input.
> >
> > I could submit the patches if there is interest in them.

I like the idea of including LICENSE and CVE_PRODUCT as well. I will look into to making it extensible via a variable, e.g. BUILDHISTORY_PACKAGE_EXTRA_VARIABLES or something.

Chris

Re: RFC: exposing information about the SRC_URI(s)/branch via buildhistory (or similar mechanism)

[Richard Purdie]

On Fri, 2019-08-02 at 07:37 +0000, Mikko.Rapeli@bmw.de wrote:
> Hi,
> 
> On Thu, Aug 01, 2019 at 04:51:38PM +0000, chris.laplante--- via
> Openembedded-core wrote:
> <snip>
> > I'm interesting in adding SRC_URI support to buildhistory (or a
> > similar mechanism), and would like to get some input.
> 
> Yes to this.
> 
> Also would be nice if there was an easy way to add bitbake variables
> to buildhistory.
> 
> I've patched our tree so that SRC_URI, LICENSE and CVE_PRODUCT are
> archived in buildhistory. SRC_URI has many uses and changes and
> patches can be easily identified. Same with LICENSE, any changes
> trigger a review. CVE_PRODUCT is exported so that we can do QA check
> to make sure mapping from CVE_PRODUCT for non CLOSED licenses exists
> to NVD database product names (maintaining a white list of recipes
> which don't have any CVEs yet).

I think this supports my point about being more interested in patches
allowing people to extend/customise buildhistory than just adding X.

Whilst we want to have good defaults, there are always going to be
niche cases for people wanting to extend it...

Cheers,

Richard

Re: RFC: exposing information about the SRC_URI(s)/branch via buildhistory (or similar mechanism)

[chris.laplante]

> > I've patched our tree so that SRC_URI, LICENSE and CVE_PRODUCT are
> > archived in buildhistory. SRC_URI has many uses and changes and
> > patches can be easily identified. Same with LICENSE, any changes
> > trigger a review. CVE_PRODUCT is exported so that we can do QA check
> > to make sure mapping from CVE_PRODUCT for non CLOSED licenses exists
> > to NVD database product names (maintaining a white list of recipes
> > which don't have any CVEs yet).
> 
> I think this supports my point about being more interested in patches
> allowing people to extend/customise buildhistory than just adding X.
> 
> Whilst we want to have good defaults, there are always going to be
> niche cases for people wanting to extend it...


Agreed. Then we can implement our BRANCH scheme without polluting the core code with it. 

Chris

Re: [bitbake-devel] RFC: exposing information about the SRC_URI(s)/branch via buildhistory (or similar mechanism)

[chris.laplante]

> > I think this supports my point about being more interested in patches
> > allowing people to extend/customise buildhistory than just adding X.
> >
> > Whilst we want to have good defaults, there are always going to be
> > niche cases for people wanting to extend it...
> 
> 
> Agreed. Then we can implement our BRANCH scheme without polluting the core code with it.

Richard, et al. - 

I have created the cpl/buildhistory branch on poky-contrib to share a draft of my ideas for making buildhistory more modular and extensible. Would appreciate any comments.

The first change was to break up buildhistory.bbclass. Previously it was ~1000 lines, one of the biggest bbclasses in meta/classes. Now, buildhistory just contains common functionality and plumbing for the buildhistory features. Each feature listed in BUILDHISTORY_FEATURES causes a class to be automatically inherited, with a naming convention of "buildhistory-{feature}". I created buildhistory-package, buildhistory-task, buildhistory-sdk, and buildhistory-image.

Users could extend buildhistory by inheriting from buildhistory directly (e.g. if they have a completely new type of buildhistory to implement), or they can inherit from a feature class. For example, buildhistory-my-custom-packaging.bbclass. Then, they would use something like the following:

    BUILDHISTORY_FEATURES_remove = "package"
    BUILDHISTORY_FEATURES_append = " my-custom-packaging"

Alternatively, they could just add "buildhistory-my-custom-packaging" to INHERIT. 

Some features need to do additional things in the buildhistory event handler, e.g. "task" writes task signatures during commit. For this, I created BUILDHISTORY_COMMIT_PREFUNCS, which nicely decouples "task" from the base buildhistory class.

The only other big change (for now) on this branch is to abstract the buildhistory-packaging feature a bit. I do this by introducing several "hooks":

    BUILDHISTORY_PACKAGE_HOOKS - Python functions or BB functions run at the end of buildhistory_emit_pkghistory. Currently used to invoke buildhistory_list_pkg_files. 
    BUILDHISTORY_PACKAGE_RECIPEHISTORY_HOOKS - Python functions invoked in write_recipehistory, providing an opportunity to write additional lines to the per-recipe "latest" file. 
    BUILDHISTORY_PACKAGE_PACKAGEHISTORY_HOOKS - Python functions invoked in write_packagehistory providing an opportunity to write additional lines to the per-package "latest" file.
    BUILDHISTORY_PACKAGE_SRCREV_HOOKS - Python functions invoked in write_latest_srcrev, called once per srcrev-supporting SRC_URI. 
    BUILDHISTORY_PACKAGE_TAGSRCREV_HOOKS - Python functions invoked in write_latest_srcrev, called once per tag srcrev SRC_URI.

Additionally, I have re-implemented the latest_srcrev functionality in terms of BUILDHISTORY_PACKAGE_SRCREV_HOOKS and BUILDHISTORY_PACKAGE_TAGSRCREV_HOOKS.

An example usage of BUILDHISTORY_PACKAGE_SRCREV_HOOKS could be the following:

    def branch_hook(d, name, srcrev, ud, total_count, f): 
        if total_count == 1:
            f.write('BRANCH = "{0}"\n'.format(ud.branches[name]))
        else:
            f.write('BRANCH_{0} = "{1}"\n'.format(name, ud.branches[name]))

    BUILDHISTORY_PACKAGE_SRCREV_HOOKS_append = " branch_hook"


Things I'm currently unsure about:
    1. Variable naming - I thought about calling them _PREFUNCS or _POSTFUNCS instead of _HOOKS, or something like that. But the problem is that some of them can only be Python functions, not BB functions. Which leads me to:
    2. Most hooks are intended to be Python functions that are found via looking in globals(). If someone has a better solution I'd be interested. But I also don't think this one is necessarily bad.
    3. I'm trying to err on the side of not adding too many hooks for the first release. Hopefully the set I have chosen to add is seen as useful.
    4. I wonder if I should create a "buildhistory" directory under "classes" into which I would move the feature classes. To avoid polluting the "classes" directory.

Additional stuff to explore:
    1. Adding a buildhistory-logs feature class, which archives the log.do_{TASK} for a user-configurable selection of tasks. Default would be compile, configure, and install. 
    2. Adding other extension points where it makes sense.

Thanks,
Chris