[Buildroot] [PATCH 1/2] git: fix handling of git repos using master as version

[Buildroot] [PATCH 1/2] git: fix handling of git repos using master as version

[Robert Beckett]

Since changing to git init followed by fetch to initialise
git cache, when using master branch as the target version
for a package, git refuses to fetch in to currently checkout
out branch or master.

git init always creates a master branch, so this will always
be an issue for any package targeting master.
Add -u option to git fetch command to allow it to ignore that
check. We know that it is okay to overwrite master as we know that
it either contains nothing after the initial git init command, or
is tracking the remote master after a successful fetch.
We always want it to overwrite it.

Signed-off-by: Robert Beckett <bbeckett@netvu.org.uk>
---
 support/download/git | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/support/download/git b/support/download/git
index 11bb52c..c33e692 100755
--- a/support/download/git
+++ b/support/download/git
@@ -122,7 +122,7 @@ _git fetch origin -t
 # below, if there is an issue anyway. Since most of the cset we're gonna
 # have to clone are not such special refs, consign the output to oblivion
 # so as not to alarm unsuspecting users, but still trace it as a warning.
-if ! _git fetch origin "'${cset}:${cset}'" >/dev/null 2>&1; then
+if ! _git fetch -u origin "'${cset}:${cset}'" >/dev/null 2>&1; then
     printf "Could not fetch special ref '%s'; assuming it is not special.\n" "${cset}"
 fi
 
-- 
2.7.4

[Buildroot] [PATCH 2/2] dl-wrapper: Fix support for URIs containing '+'

[Robert Beckett]

'+' is a valid character in a url. The current dl-wrapper gets the
URI scheme by dropping everything after the last '+' character, with
the intension of finding 'git' from e.g. 'git+https://uri'.

If a uri has a '+' anywhere in it, it ends up using too much of the
string as a scheme, and fails to match the handler properly.

An example of where this form of URI is used is when using deploy tokens
in gitlab. It uses a form like https://<username>:<password>@gitlab.com/<group>/<repo.git>
where username for deploy token is of the form 'gitlab+deploy-token-<number>'.

Use the %% operator to search backwards until the last '+' character when
dropping the rest of the string as we know that the first '+'
in the string should be the scheme.

Signed-off-by: Robert Beckett <bbeckett@netvu.org.uk>
---
 support/download/dl-wrapper | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper
index 8d6365e..4059c37 100755
--- a/support/download/dl-wrapper
+++ b/support/download/dl-wrapper
@@ -88,7 +88,7 @@ main() {
     download_and_check=0
     rc=1
     for uri in "${uris[@]}"; do
-        backend=${uri%+*}
+        backend=${uri%%+*}
         case "${backend}" in
             git|svn|cvs|bzr|file|scp|hg) ;;
             *) backend="wget" ;;
-- 
2.7.4

[Buildroot] [PATCH 1/2] git: fix handling of git repos using master as version

[Thomas Petazzoni]

Hello,

On Mon,  4 Jun 2018 14:32:29 +0100, Robert Beckett wrote:
> Since changing to git init followed by fetch to initialise
> git cache, when using master branch as the target version
> for a package, git refuses to fetch in to currently checkout
> out branch or master.

I think we don't want to support using branches as Git versions, as it
doesn't make sense, because it won't behave as the user expects.

If you use a branch as _VERSION for a package, Buildroot will only
download once, and never update to newer commits in this branch,
because it will already have a cached tarball in its DL_DIR.

So I'm not sure we want to add some code to support a use case that we
don't want support :-)

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 1/2] git: fix handling of git repos using master as version

[Bob Beckett]

Is this philosophy set in stone? I can conceive of useful situations where
branches are preferable.
During large projects, where the rootfs is being developed at the same time
as some custom packages,
it is quite useful to have a branch as the target. Usually, I just do a
dirclean and delete the tar when I know
something has changed.

If you wanted to make it more predictable, then maybe name the tar based on
the version (branch name in this case) plus the sha, this way new tar
packages would be created if the branch advances and also if a tag (or
other supported refspec) changes via a rebase etc.

The particular thing I like to avoid is multiple version bumps in the
package spec and tags in the package source repos during development. When
many people are working on package project, expecting them all to take time
to publish tags for every little change gets unrealistic.

The question I would ask is is there any harm in supporting that if it can
be done reliably and predictably?

Anyway, if this design is locked down, then I will just maintain this patch
locally for us to use :)

Thanks

Bob


On Mon, Jun 4, 2018 at 3:33 PM, Thomas Petazzoni <
thomas.petazzoni@bootlin.com> wrote:

> Hello,
>
> On Mon,  4 Jun 2018 14:32:29 +0100, Robert Beckett wrote:
> > Since changing to git init followed by fetch to initialise
> > git cache, when using master branch as the target version
> > for a package, git refuses to fetch in to currently checkout
> > out branch or master.
>
> I think we don't want to support using branches as Git versions, as it
> doesn't make sense, because it won't behave as the user expects.
>
> If you use a branch as _VERSION for a package, Buildroot will only
> download once, and never update to newer commits in this branch,
> because it will already have a cached tarball in its DL_DIR.
>
> So I'm not sure we want to add some code to support a use case that we
> don't want support :-)
>
> Thomas
> --
> Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
> Embedded Linux and Kernel engineering
> https://bootlin.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20180604/8f042d7f/attachment.html>

[Buildroot] [PATCH 1/2] git: fix handling of git repos using master as version

[Yann E. MORIN]

On 2018-06-04 15:59 +0100, Bob Beckett spake thusly:
> Is this philosophy set in stone? I can conceive of useful situations where branches are preferable.

The problem, as Thomas quickly explained, is that it does not behave as
users expect.

First, Buildroot creates a local tarball of the git reository. We do
that for various reasons, one beiong to be able to prepare the
compliance archive (make legal-info).

So, if you run "make clean; make" then the local archive already exists
and Buildroot will not download it again, i.e. it will nt update your
git tree.

Now, if you run "make distclean", then yes, it removes the download
directory that you get in "buildroot/dl/", and so you will re-download
again, and that will be the new branch you'll get.

However, it is possible to override the download location, if you export
BR2_DL_DIR in the environment, to tell Buildroot where to look for, and
store new downloads, as a way to cache downloads for those users behind
slow connections (not everyone has ADSL-class connection, even less so
gigabit-class fibers). If BR2_DL_DIR is set, then Buildroot will never
remove it, even with "make distclean". So, you're back to square one,
where the local tarball still represent an old state of your branch.

Ergo, it does not behave as you would think it would.

The second problem, is that using a branch name makes the build
non-reproducible.

Today, you'll get a set of source from your branch, but maybe tomorrow
it will point to something else, or even have disapeared (which is the
least problematic issue, because the build would obviously fail, while
it could succeed and result in a flawed state with a new branch).

Ergo, it does not behave as you would think it would.

So, using a branch name is flawed in two ways. We can't add support for
using branch names in any reliable and predictable manner.

> During large projects, where the rootfs is being developed at the same time as some custom packages,
> it is quite useful to have a branch as the target. Usually, I just do a dirclean and delete the tar when I know
> something has changed.
> 
> If you wanted to make it more predictable, then maybe name the tar based on the version (branch name in this case) plus the sha,

But we can't know the sha1 before we git checkout the branch, so we
don;t know if we already have it localy, so we don;t know we can use a
cached version, so this undermines the very reason to have a local cache
to start with...

> this way new tar packages would be created if the branch advances and also if a tag (or other supported refspec) changes via a
> rebase etc.

Changing a tag is bad. Repeat after me: changing a tag is bad! ;-)

Yes, git allows that. Guns also allow one to shoot in ones' own foot.
That does not mean this is a good idea... ;-p

> The particular thing I like to avoid is multiple version bumps in the package spec and tags in the package source repos during
> development. When many people are working on package project, expecting them all to take time to publish tags for every little
> change gets unrealistic.
> 
> The question I would ask is is there any harm in supporting that if it can be done reliably and predictably?
> Anyway, if this design is locked down, then I will just maintain this patch locally for us to use :)

And as I hope I clearly explained above, there is no wau to make it work
predictably and reliably.

Regards,
Yann E. MORIN.

> Thanks
> 
> Bob
> On Mon, Jun 4, 2018 at 3:33 PM, Thomas Petazzoni < [1]thomas.petazzoni@bootlin.com> wrote:
> 
>   Hello,
>   On Mon,? 4 Jun 2018 14:32:29 +0100, Robert Beckett wrote:
>   > Since changing to git init followed by fetch to initialise
>   > git cache, when using master branch as the target version
>   > for a package, git refuses to fetch in to currently checkout
>   > out branch or master.
> 
>   I think we don't want to support using branches as Git versions, as it
>   doesn't make sense, because it won't behave as the user expects.
> 
>   If you use a branch as _VERSION for a package, Buildroot will only
>   download once, and never update to newer commits in this branch,
>   because it will already have a cached tarball in its DL_DIR.
> 
>   So I'm not sure we want to add some code to support a use case that we
>   don't want support :-)
>   Thomas
>   --
>   Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
>   Embedded Linux and Kernel engineering
>   [2]https://bootlin.com
> 
> Links:
> 1. mailto:thomas.petazzoni at bootlin.com
> 2. https://bootlin.com

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 2/2] dl-wrapper: Fix support for URIs containing '+'

[Yann E. MORIN]

Robert, All,

On 2018-06-04 14:32 +0100, Robert Beckett spake thusly:
> '+' is a valid character in a url. The current dl-wrapper gets the
> URI scheme by dropping everything after the last '+' character, with
> the intension of finding 'git' from e.g. 'git+https://uri'.
> 
> If a uri has a '+' anywhere in it, it ends up using too much of the
> string as a scheme, and fails to match the handler properly.
> 
> An example of where this form of URI is used is when using deploy tokens
> in gitlab. It uses a form like https://<username>:<password>@gitlab.com/<group>/<repo.git>
> where username for deploy token is of the form 'gitlab+deploy-token-<number>'.
> 
> Use the %% operator to search backwards until the last '+' character when
> dropping the rest of the string as we know that the first '+'
> in the string should be the scheme.
> 
> Signed-off-by: Robert Beckett <bbeckett@netvu.org.uk>

Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>

Regards,
Yann E. MORIN.

> ---
>  support/download/dl-wrapper | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper
> index 8d6365e..4059c37 100755
> --- a/support/download/dl-wrapper
> +++ b/support/download/dl-wrapper
> @@ -88,7 +88,7 @@ main() {
>      download_and_check=0
>      rc=1
>      for uri in "${uris[@]}"; do
> -        backend=${uri%+*}
> +        backend=${uri%%+*}
>          case "${backend}" in
>              git|svn|cvs|bzr|file|scp|hg) ;;
>              *) backend="wget" ;;
> -- 
> 2.7.4
> 

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 1/2] git: fix handling of git repos using master as version

[Thomas Petazzoni]

Hello,

Yann already answered on some of the aspects, but I wanted to add one
more thing.

On Mon, 4 Jun 2018 15:59:29 +0100, Bob Beckett wrote:

> If you wanted to make it more predictable, then maybe name the tar
> based on the version (branch name in this case) plus the sha, this way
> new tar packages would be created if the branch advances and also if a
> tag (or other supported refspec) changes via a rebase etc.

But then at every build you need to go and check the Git repository to
see if a new version on this branch is available. That's clearly not
something we want to do for all packages.

> The particular thing I like to avoid is multiple version bumps in the
> package spec and tags in the package source repos during development. When
> many people are working on package project, expecting them all to take time
> to publish tags for every little change gets unrealistic.

During development, we would normally expect people to use
<pkg>_OVERRIDE_SRCDIR or <pkg>_SITE_METHOD = local, which allows a
package to use locally available source code instead of fetching it as
a tarball or from Github.

> Anyway, if this design is locked down, then I will just maintain this patch
> locally for us to use :)

The design is never locked down, Buildroot is open sourced, and
influenced by all the people who use it and contribute to it. However,
we are obviously careful to maintain a behavior and semantic that makes
the most sense. And as Yann pointed out, reproducibility is a key thing.

That being said, Angelo is proposing something like a "taint" flag that
tells the user "warning, your build is not reproducible". Perhaps we
could extend the Git logic so that packages can say "always fetch the
latest commit from this branch", and if they do that mark the build as
tainted.

But I'm still unsure when should this re-fetch should happen. When you
completely rebuild from scratch ? When you do "make <pkg>-rebuild" ?

If we want to alter the Buildroot design, we need to agree on a
behavior that globally makes sense.

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 1/2] git: fix handling of git repos using master as version

[Bob Beckett]

Thanks for the more in depth explanation.

I agree with all of the reasons outlined for the purpose of being purely a
reproducible build manager, which buildroot only ever aimed to be.

However, people do use it during development, and with a reasonably large
number of custom packages all in development at the same time, using the
_OVERRIDE_SRCDIR method to test latest versions of multiple packages
becomes very unwieldy.
My solution so far has been to point the version at the branch that active
development is taking place on, and removing the build directories and
packages for the specific projects each time.
When I am not the one doing the development on each package's source, but
am developing the rootfs, this means I dont have to manually keep updating
my local git repositories for each project.
Once each package has hit their first release version, they do get tagged
and the release branch for the project's BR2_EXTERNAL directory gets
updated with those versions, and each version thereafter, but the
development branch for the external directory persistently stays on the
development branch head for each package.

Regarding not knowing the sha before a checkout, would git ls-remote not
suffice for this? e.g.

$ git ls-remote git at gitlab.com:adhlinux/buildroot/buildroot.git master
407fb2fe202aaeb273e4986dc88c30596a7fe8db    refs/heads/master
407fb2fe202aaeb273e4986dc88c30596a7fe8db    refs/remotes/upstream/master

It would allow you to check for changes in sha without doing a new fetch.

Oh well, not to worry. Ill keep it as a local patch, as I suspect I am
treading old ground of "buildroot is not a development platform" (Ive seen
this discussion come up multiple times before).

Regards

Bob



On Mon, Jun 4, 2018 at 4:44 PM, Yann E. MORIN <yann.morin.1998@free.fr>
wrote:

> On 2018-06-04 15:59 +0100, Bob Beckett spake thusly:
> > Is this philosophy set in stone? I can conceive of useful situations
> where branches are preferable.
>
> The problem, as Thomas quickly explained, is that it does not behave as
> users expect.
>
> First, Buildroot creates a local tarball of the git reository. We do
> that for various reasons, one beiong to be able to prepare the
> compliance archive (make legal-info).
>
> So, if you run "make clean; make" then the local archive already exists
> and Buildroot will not download it again, i.e. it will nt update your
> git tree.
>
> Now, if you run "make distclean", then yes, it removes the download
> directory that you get in "buildroot/dl/", and so you will re-download
> again, and that will be the new branch you'll get.
>
> However, it is possible to override the download location, if you export
> BR2_DL_DIR in the environment, to tell Buildroot where to look for, and
> store new downloads, as a way to cache downloads for those users behind
> slow connections (not everyone has ADSL-class connection, even less so
> gigabit-class fibers). If BR2_DL_DIR is set, then Buildroot will never
> remove it, even with "make distclean". So, you're back to square one,
> where the local tarball still represent an old state of your branch.
>
> Ergo, it does not behave as you would think it would.
>
> The second problem, is that using a branch name makes the build
> non-reproducible.
>
> Today, you'll get a set of source from your branch, but maybe tomorrow
> it will point to something else, or even have disapeared (which is the
> least problematic issue, because the build would obviously fail, while
> it could succeed and result in a flawed state with a new branch).
>
> Ergo, it does not behave as you would think it would.
>
> So, using a branch name is flawed in two ways. We can't add support for
> using branch names in any reliable and predictable manner.
>
> > During large projects, where the rootfs is being developed at the same
> time as some custom packages,
> > it is quite useful to have a branch as the target. Usually, I just do a
> dirclean and delete the tar when I know
> > something has changed.
> >
> > If you wanted to make it more predictable, then maybe name the tar based
> on the version (branch name in this case) plus the sha,
>
> But we can't know the sha1 before we git checkout the branch, so we
> don;t know if we already have it localy, so we don;t know we can use a
> cached version, so this undermines the very reason to have a local cache
> to start with...
>
> > this way new tar packages would be created if the branch advances and
> also if a tag (or other supported refspec) changes via a
> > rebase etc.
>
> Changing a tag is bad. Repeat after me: changing a tag is bad! ;-)
>
> Yes, git allows that. Guns also allow one to shoot in ones' own foot.
> That does not mean this is a good idea... ;-p
>
> > The particular thing I like to avoid is multiple version bumps in the
> package spec and tags in the package source repos during
> > development. When many people are working on package project, expecting
> them all to take time to publish tags for every little
> > change gets unrealistic.
> >
> > The question I would ask is is there any harm in supporting that if it
> can be done reliably and predictably?
> > Anyway, if this design is locked down, then I will just maintain this
> patch locally for us to use :)
>
> And as I hope I clearly explained above, there is no wau to make it work
> predictably and reliably.
>
> Regards,
> Yann E. MORIN.
>
> > Thanks
> >
> > Bob
> > On Mon, Jun 4, 2018 at 3:33 PM, Thomas Petazzoni < [1]
> thomas.petazzoni at bootlin.com> wrote:
> >
> >   Hello,
> >   On Mon,  4 Jun 2018 14:32:29 +0100, Robert Beckett wrote:
> >   > Since changing to git init followed by fetch to initialise
> >   > git cache, when using master branch as the target version
> >   > for a package, git refuses to fetch in to currently checkout
> >   > out branch or master.
> >
> >   I think we don't want to support using branches as Git versions, as it
> >   doesn't make sense, because it won't behave as the user expects.
> >
> >   If you use a branch as _VERSION for a package, Buildroot will only
> >   download once, and never update to newer commits in this branch,
> >   because it will already have a cached tarball in its DL_DIR.
> >
> >   So I'm not sure we want to add some code to support a use case that we
> >   don't want support :-)
> >   Thomas
> >   --
> >   Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
> >   Embedded Linux and Kernel engineering
> >   [2]https://bootlin.com
> >
> > Links:
> > 1. mailto:thomas.petazzoni at bootlin.com
> > 2. https://bootlin.com
>
> --
> .-----------------.--------------------.------------------.-
> -------------------.
> |  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics'
> conspiracy: |
> | +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___
>      |
> | +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is
> no  |
> | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v
>  conspiracy.  |
> '------------------------------^-------^------------------^-
> -------------------'
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20180604/c2226836/attachment.html>

[Buildroot] [PATCH 1/2] git: fix handling of git repos using master as version

[Bob Beckett]

Maybe a variable _IN_DEVELOPMENT or something could be added to a package's
makefile to indicate that the version is always considered out of date, and
should be re-fetched.
That way development branches for package specs could have the package spec
committed with that, while release branches do not and are expected to have
a non-branch version specifier.

With this strategy you would be acknowledging that you are not getting
reproducibility, but you are still getting reliable builds (the 2 reasons I
use buildroot).
The test for taint would then be to check if any packages have an
_IN_DEVELOPMENT variable set.

Given the current state, if you specify a branch of master, you get a
warning that it doesnt appear to be a special version, but that is all, and
the package doesnt build at all as it failed to get the source in the first
place.
I think if there is going to be an enforcement of not using branches, then
the build should probably fail in a more explicit way e.g. check to see if
the version is a branch. If it is, then fail the build with a message
telling the user that branches are not to be used as versions (perhaps
unless something is set?). When I hit this issue it took me a bit of
hunting to figure out what was going on. The current state neither does
what you would expect nor anything that is usable in this specific setup.




On Mon, Jun 4, 2018 at 5:23 PM, Thomas Petazzoni <
thomas.petazzoni@bootlin.com> wrote:

> Hello,
>
> Yann already answered on some of the aspects, but I wanted to add one
> more thing.
>
> On Mon, 4 Jun 2018 15:59:29 +0100, Bob Beckett wrote:
>
> > If you wanted to make it more predictable, then maybe name the tar
> > based on the version (branch name in this case) plus the sha, this way
> > new tar packages would be created if the branch advances and also if a
> > tag (or other supported refspec) changes via a rebase etc.
>
> But then at every build you need to go and check the Git repository to
> see if a new version on this branch is available. That's clearly not
> something we want to do for all packages.
>
> > The particular thing I like to avoid is multiple version bumps in the
> > package spec and tags in the package source repos during development.
> When
> > many people are working on package project, expecting them all to take
> time
> > to publish tags for every little change gets unrealistic.
>
> During development, we would normally expect people to use
> <pkg>_OVERRIDE_SRCDIR or <pkg>_SITE_METHOD = local, which allows a
> package to use locally available source code instead of fetching it as
> a tarball or from Github.
>
> > Anyway, if this design is locked down, then I will just maintain this
> patch
> > locally for us to use :)
>
> The design is never locked down, Buildroot is open sourced, and
> influenced by all the people who use it and contribute to it. However,
> we are obviously careful to maintain a behavior and semantic that makes
> the most sense. And as Yann pointed out, reproducibility is a key thing.
>
> That being said, Angelo is proposing something like a "taint" flag that
> tells the user "warning, your build is not reproducible". Perhaps we
> could extend the Git logic so that packages can say "always fetch the
> latest commit from this branch", and if they do that mark the build as
> tainted.
>
> But I'm still unsure when should this re-fetch should happen. When you
> completely rebuild from scratch ? When you do "make <pkg>-rebuild" ?
>
> If we want to alter the Buildroot design, we need to agree on a
> behavior that globally makes sense.
>
> Thomas
> --
> Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
> Embedded Linux and Kernel engineering
> https://bootlin.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20180604/f7f3e071/attachment.html>

[Buildroot] [PATCH 1/2] git: fix handling of git repos using master as version

[Gary Bisson]

Hi Bob

On Mon, Jun 04, 2018 at 05:32:41PM +0100, Bob Beckett wrote:
> Thanks for the more in depth explanation.
> 
> I agree with all of the reasons outlined for the purpose of being purely a
> reproducible build manager, which buildroot only ever aimed to be.
> 
> However, people do use it during development, and with a reasonably large
> number of custom packages all in development at the same time, using the
> _OVERRIDE_SRCDIR method to test latest versions of multiple packages
> becomes very unwieldy.
> My solution so far has been to point the version at the branch that active
> development is taking place on, and removing the build directories and
> packages for the specific projects each time.
> When I am not the one doing the development on each package's source, but
> am developing the rootfs, this means I dont have to manually keep updating
> my local git repositories for each project.
> Once each package has hit their first release version, they do get tagged
> and the release branch for the project's BR2_EXTERNAL directory gets
> updated with those versions, and each version thereafter, but the
> development branch for the external directory persistently stays on the
> development branch head for each package.
> 
> Regarding not knowing the sha before a checkout, would git ls-remote not
> suffice for this? e.g.
> 
> $ git ls-remote git at gitlab.com:adhlinux/buildroot/buildroot.git master
> 407fb2fe202aaeb273e4986dc88c30596a7fe8db    refs/heads/master
> 407fb2fe202aaeb273e4986dc88c30596a7fe8db    refs/remotes/upstream/master

Yes 'ls-remote' is actually a good option. You could have the following
package version during development:
FOO_VERSION = $(shell git ls-remote URL branch | awk '{ print $$1 }')

Then it would pick up any change automatically until you finish your dev
and change the FOO_VERSION to a proper ID.

> It would allow you to check for changes in sha without doing a new fetch.
> 
> Oh well, not to worry. Ill keep it as a local patch, as I suspect I am
> treading old ground of "buildroot is not a development platform" (Ive seen
> this discussion come up multiple times before).

Yes that will most likely not be included inside BR infrastructure but that
should be a perfectly fine option for your custom package.

Regards,
Gary

[Buildroot] [PATCH 1/2] git: fix handling of git repos using master as version

[Bob Beckett]

On Mon, Jun 4, 2018 at 5:52 PM, Gary Bisson <gary.bisson@boundarydevices.com
> wrote:

> Hi Bob
>
> On Mon, Jun 04, 2018 at 05:32:41PM +0100, Bob Beckett wrote:
> > Thanks for the more in depth explanation.
> >
> > I agree with all of the reasons outlined for the purpose of being purely
> a
> > reproducible build manager, which buildroot only ever aimed to be.
> >
> > However, people do use it during development, and with a reasonably large
> > number of custom packages all in development at the same time, using the
> > _OVERRIDE_SRCDIR method to test latest versions of multiple packages
> > becomes very unwieldy.
> > My solution so far has been to point the version at the branch that
> active
> > development is taking place on, and removing the build directories and
> > packages for the specific projects each time.
> > When I am not the one doing the development on each package's source, but
> > am developing the rootfs, this means I dont have to manually keep
> updating
> > my local git repositories for each project.
> > Once each package has hit their first release version, they do get tagged
> > and the release branch for the project's BR2_EXTERNAL directory gets
> > updated with those versions, and each version thereafter, but the
> > development branch for the external directory persistently stays on the
> > development branch head for each package.
> >
> > Regarding not knowing the sha before a checkout, would git ls-remote not
> > suffice for this? e.g.
> >
> > $ git ls-remote git at gitlab.com:adhlinux/buildroot/buildroot.git master
> > 407fb2fe202aaeb273e4986dc88c30596a7fe8db    refs/heads/master
> > 407fb2fe202aaeb273e4986dc88c30596a7fe8db    refs/remotes/upstream/master
>
> Yes 'ls-remote' is actually a good option. You could have the following
> package version during development:
> FOO_VERSION = $(shell git ls-remote URL branch | awk '{ print $$1 }')
>
> Then it would pick up any change automatically until you finish your dev
> and change the FOO_VERSION to a proper ID.
>
>
Thats actually a cracking idea. I like it (and I'm not sure why I didnt
think of it when suggesting it for detecting outdated named head versions
:) )


> > It would allow you to check for changes in sha without doing a new fetch.
> >
> > Oh well, not to worry. Ill keep it as a local patch, as I suspect I am
> > treading old ground of "buildroot is not a development platform" (Ive
> seen
> > this discussion come up multiple times before).
>
> Yes that will most likely not be included inside BR infrastructure but that
> should be a perfectly fine option for your custom package.
>
> Regards,
> Gary
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20180604/9fb1398c/attachment.html>

[Buildroot] [PATCH 1/2] git: fix handling of git repos using master as version

[Henrique Marks]

Hello All, 

> De: "Bob Beckett" <bbeckett@netvu.org.uk>
> Para: "Gary Bisson" <gary.bisson@boundarydevices.com>
> Cc: "Yann E. MORIN" <yann.morin.1998@free.fr>, "Thomas Petazzoni"
> <thomas.petazzoni@bootlin.com>, "buildroot" <buildroot@buildroot.org>
> Enviadas: Segunda-feira, 4 de junho de 2018 13:54:30
> Assunto: Re: [Buildroot] [PATCH 1/2] git: fix handling of git repos using master
> as version

> On Mon, Jun 4, 2018 at 5:52 PM, Gary Bisson < [
> mailto:gary.bisson at boundarydevices.com | gary.bisson at boundarydevices.com ] >
> wrote:

>> Hi Bob

>> On Mon, Jun 04, 2018 at 05:32:41PM +0100, Bob Beckett wrote:
>> > Thanks for the more in depth explanation.

>> > I agree with all of the reasons outlined for the purpose of being purely a
>> > reproducible build manager, which buildroot only ever aimed to be.

>> > However, people do use it during development, and with a reasonably large
>> > number of custom packages all in development at the same time, using the
>> > _OVERRIDE_SRCDIR method to test latest versions of multiple packages
>> > becomes very unwieldy.
>> > My solution so far has been to point the version at the branch that active
>> > development is taking place on, and removing the build directories and
>> > packages for the specific projects each time.
>> > When I am not the one doing the development on each package's source, but
>> > am developing the rootfs, this means I dont have to manually keep updating
>> > my local git repositories for each project.
>> > Once each package has hit their first release version, they do get tagged
>> > and the release branch for the project's BR2_EXTERNAL directory gets
>> > updated with those versions, and each version thereafter, but the
>> > development branch for the external directory persistently stays on the
>> > development branch head for each package.

>> > Regarding not knowing the sha before a checkout, would git ls-remote not
>> > suffice for this? e.g.

>> > $ git ls-remote git at gitlab.com:adhlinux/buildroot/buildroot.git master
>> > 407fb2fe202aaeb273e4986dc88c30596a7fe8db refs/heads/master
>> > 407fb2fe202aaeb273e4986dc88c30596a7fe8db refs/remotes/upstream/master

>> Yes 'ls-remote' is actually a good option. You could have the following
>> package version during development:
>> FOO_VERSION = $(shell git ls-remote URL branch | awk '{ print $$1 }')

>> Then it would pick up any change automatically until you finish your dev
>> and change the FOO_VERSION to a proper ID.

> Thats actually a cracking idea. I like it (and I'm not sure why I didnt think of
> it when suggesting it for detecting outdated named head versions :) )

>> > It would allow you to check for changes in sha without doing a new fetch.

>> > Oh well, not to worry. Ill keep it as a local patch, as I suspect I am
>> > treading old ground of "buildroot is not a development platform" (Ive seen
>> > this discussion come up multiple times before).

>> Yes that will most likely not be included inside BR infrastructure but that
>> should be a perfectly fine option for your custom package.

>> Regards,
>> Gary

> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

We use buildroot for development AND for integration. 

For development, you take your code, download the source and use OVERRIDE_SRCDIR as Thomas described. It works just fine. 

For integration, every developer submits the working code to a Robot, that puts the SHA in the version field in the .mk file. 

No use for branches, because sometimes the HEAD of the branch is not suitable either for development or integration. 

But, just to make a point about differences when using buildroot this way: we are not too much interested in instrumenting buildroot, with those scripts that check the files, because they affect development (pkg-rebuild) and integration (overall time). 

So, to please both the project and people using it for development/integration, i think the ideal would be to have the choice of not instrumenting a build, preferably on defconfigs. 

Just to point an important (and noticeable) difference when you are developing with buildroot. 

Regards, 

-- 
Dr. Henrique Marks 
henrique.marks at datacom.ind.br 
R. Am?rica, 1000 - Eldorado do Sul - RS 
CEP: 92990-000 - Brasil 
Fone: +55 51 3933 3000 - Ramal 3466 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20180604/8d2fa0f2/attachment.html>

[Buildroot] [PATCH 2/2] dl-wrapper: Fix support for URIs containing '+'

[Thomas Petazzoni]

Hello,

On Mon,  4 Jun 2018 14:32:30 +0100, Robert Beckett wrote:
> '+' is a valid character in a url. The current dl-wrapper gets the
> URI scheme by dropping everything after the last '+' character, with
> the intension of finding 'git' from e.g. 'git+https://uri'.
> 
> If a uri has a '+' anywhere in it, it ends up using too much of the
> string as a scheme, and fails to match the handler properly.
> 
> An example of where this form of URI is used is when using deploy tokens
> in gitlab. It uses a form like https://<username>:<password>@gitlab.com/<group>/<repo.git>
> where username for deploy token is of the form 'gitlab+deploy-token-<number>'.
> 
> Use the %% operator to search backwards until the last '+' character when
> dropping the rest of the string as we know that the first '+'
> in the string should be the scheme.
> 
> Signed-off-by: Robert Beckett <bbeckett@netvu.org.uk>
> ---
>  support/download/dl-wrapper | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)

Applied to master, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 1/2] git: fix handling of git repos using master as version

[Arnout Vandecappelle]

 [Please don't top-post, but reply inline.]

On 04-06-18 18:51, Bob Beckett wrote:
> Maybe a variable _IN_DEVELOPMENT or something could be added to a package's
> makefile to indicate that the version is always considered out of date, and
> should be re-fetched.

 We're never going to consider something *always* out of date, I think. We might
only consider it out-of-date when you do an explicit foo-rebuild.

> That way development branches for package specs could have the package spec
> committed with that, while release branches do not and are expected to have a
> non-branch version specifier.
> 
> With this strategy you would be acknowledging that you are not getting
> reproducibility, but you are still getting reliable builds (the 2 reasons I use
> buildroot).
> The test for taint would then be to check if any packages have an
> _IN_DEVELOPMENT variable set.

 I'm not so fond of the _IN_DEVELOPMENT idea. What I like more is to extend
_OVERRIDE_SRCDIR to support repositories in addition to local files. It would
then check out directly into the build directory, without passing through DL_DIR
or a tarball.

 This would probably be a rather complicated change however.


> Given the current state, if you specify a branch of master, you get a warning
> that it doesnt appear to be a special version, but that is all, and the package
> doesnt build at all as it failed to get the source in the first place> I think if there is going to be an enforcement of not using branches, then the
> build should probably fail in a more explicit way e.g. check to see if the
> version is a branch. If it is, then fail the build with a message telling the
> user that branches are not to be used as versions

 I completely agree with this one, we should simply fail on branches.

 Regards,
 Arnout


> (perhaps unless something is
> set?). When I hit this issue it took me a bit of hunting to figure out what was
> going on. The current state neither does what you would expect nor anything that
> is usable in this specific setup.
> 
> 
> 
> 
> On Mon, Jun 4, 2018 at 5:23 PM, Thomas Petazzoni <thomas.petazzoni@bootlin.com
> <mailto:thomas.petazzoni@bootlin.com>> wrote:
> 
>     Hello,
> 
>     Yann already answered on some of the aspects, but I wanted to add one
>     more thing.
> 
>     On Mon, 4 Jun 2018 15:59:29 +0100, Bob Beckett wrote:
> 
>     > If you wanted to make it more predictable, then maybe name the tar
>     > based on the version (branch name in this case) plus the sha, this way
>     > new tar packages would be created if the branch advances and also if a
>     > tag (or other supported refspec) changes via a rebase etc.
> 
>     But then at every build you need to go and check the Git repository to
>     see if a new version on this branch is available. That's clearly not
>     something we want to do for all packages.
> 
>     > The particular thing I like to avoid is multiple version bumps in the
>     > package spec and tags in the package source repos during development. When
>     > many people are working on package project, expecting them all to take time
>     > to publish tags for every little change gets unrealistic.
> 
>     During development, we would normally expect people to use
>     <pkg>_OVERRIDE_SRCDIR or <pkg>_SITE_METHOD = local, which allows a
>     package to use locally available source code instead of fetching it as
>     a tarball or from Github.
> 
>     > Anyway, if this design is locked down, then I will just maintain this patch
>     > locally for us to use :)
> 
>     The design is never locked down, Buildroot is open sourced, and
>     influenced by all the people who use it and contribute to it. However,
>     we are obviously careful to maintain a behavior and semantic that makes
>     the most sense. And as Yann pointed out, reproducibility is a key thing.
> 
>     That being said, Angelo is proposing something like a "taint" flag that
>     tells the user "warning, your build is not reproducible". Perhaps we
>     could extend the Git logic so that packages can say "always fetch the
>     latest commit from this branch", and if they do that mark the build as
>     tainted.
> 
>     But I'm still unsure when should this re-fetch should happen. When you
>     completely rebuild from scratch ? When you do "make <pkg>-rebuild" ?
> 
>     If we want to alter the Buildroot design, we need to agree on a
>     behavior that globally makes sense.
> 
>     Thomas
>     -- 
>     Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
>     Embedded Linux and Kernel engineering
>     https://bootlin.com
> 
> 
> 
> 
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 1/2] git: fix handling of git repos using master as version

[Arnout Vandecappelle]

On 04-06-18 18:52, Gary Bisson wrote:
> Hi Bob
> 
> On Mon, Jun 04, 2018 at 05:32:41PM +0100, Bob Beckett wrote:
>> Thanks for the more in depth explanation.
>>
>> I agree with all of the reasons outlined for the purpose of being purely a
>> reproducible build manager, which buildroot only ever aimed to be.
>>
>> However, people do use it during development, and with a reasonably large
>> number of custom packages all in development at the same time, using the
>> _OVERRIDE_SRCDIR method to test latest versions of multiple packages
>> becomes very unwieldy.
>> My solution so far has been to point the version at the branch that active
>> development is taking place on, and removing the build directories and
>> packages for the specific projects each time.
>> When I am not the one doing the development on each package's source, but
>> am developing the rootfs, this means I dont have to manually keep updating
>> my local git repositories for each project.
>> Once each package has hit their first release version, they do get tagged
>> and the release branch for the project's BR2_EXTERNAL directory gets
>> updated with those versions, and each version thereafter, but the
>> development branch for the external directory persistently stays on the
>> development branch head for each package.
>>
>> Regarding not knowing the sha before a checkout, would git ls-remote not
>> suffice for this? e.g.
>>
>> $ git ls-remote git at gitlab.com:adhlinux/buildroot/buildroot.git master
>> 407fb2fe202aaeb273e4986dc88c30596a7fe8db    refs/heads/master
>> 407fb2fe202aaeb273e4986dc88c30596a7fe8db    refs/remotes/upstream/master
> 
> Yes 'ls-remote' is actually a good option. 

 It's a bit racy though. By the time you do the actual fetch, the remote branch
may already have been updated by someone else.

 Regards,
 Arnout


You could have the following
> package version during development:
> FOO_VERSION = $(shell git ls-remote URL branch | awk '{ print $$1 }')
> 
> Then it would pick up any change automatically until you finish your dev
> and change the FOO_VERSION to a proper ID.
> 
>> It would allow you to check for changes in sha without doing a new fetch.
>>
>> Oh well, not to worry. Ill keep it as a local patch, as I suspect I am
>> treading old ground of "buildroot is not a development platform" (Ive seen
>> this discussion come up multiple times before).
> 
> Yes that will most likely not be included inside BR infrastructure but that
> should be a perfectly fine option for your custom package.
> 
> Regards,
> Gary
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 1/2] git: fix handling of git repos using master as version

[Trent Piepho]

On Mon, 2018-06-04 at 17:54 +0100, Bob Beckett wrote:
> > 
> > > 
> > > Regarding not knowing the sha before a checkout, would git ls-
> > remote not
> > > suffice for this? e.g.
> > > 
> > > $ git ls-remote git at gitlab.com:adhlinux/buildroot/buildroot.git
> > master
> > > 407fb2fe202aaeb273e4986dc88c30596a7fe8db    refs/heads/master
> > > 407fb2fe202aaeb273e4986dc88c30596a7fe8db   
> > refs/remotes/upstream/master
> > 
> > Yes 'ls-remote' is actually a good option. You could have the
> > following
> > package version during development:
> > FOO_VERSION = $(shell git ls-remote URL branch | awk '{ print $$1
> > }')
> > 
> > Then it would pick up any change automatically until you finish
> > your dev
> > and change the FOO_VERSION to a proper ID.
> > 
> > 
> 
> Thats actually a cracking idea. I like it (and I'm not sure why I
> didnt think of it when suggesting it for detecting outdated named
> head versions :) )

You will lose reproducibility of course.  There won't be any way to
recreate last week's nightly build since you'll get the current master
of every package when you try.  I would hate to try to track down a
regression in such a setup.

You'll also get builds that change from minute to minute as new commits
appear on any package's master branch.  I would think this would be
very annoying for developers.  I expect that my source code is only
changed when I run git pull, not every time I run make.

I also find that integration is not so easy as this.  Often commits in
one package require changes to another package.  Pulling in new commits
 to the product the instant they are merged in one package will often
break.

If you don't like updating package versions, here's a design that
avoids that and also doesn't suffer from the above problems.

product
-mypackage
--source code
-br-extern
--packages
---mypackage
----mypackage.mk
-----[MYPACKAGE_SITE_METHOD = local]
-----[MYPACKAGE_SITE = ../mypackage]
--configs
---myproduct_defconfig
-buildroot (submodule)
Makefile

The top level, "product", is a git repo.  There is a sub-directory for
each of the product's packages that contains the source.  One develops
the code by editing it in this location, and git commit/pull operations
on done on the code here as part of the product repo.

The by-extern directory is also part of the product repo.  It has the
buildroot files for the packages.  They are setup to use the "local"
SITE_METHOD and a relative path that points to the mypackage directory
that is part of the product repo, described previously.

buildroot itself is a git submodule of the product repo.

There is (probably) a top-level Makefile that has targets to do the
initial make command to buildroot, e.g. make -C buildroot
BR2_EXTERNAL=$(CURDIR)/br-extern O=$(CURDIR)/output/myproduct
myproduct_defconfig

To build, one git clones (recursively) the product repo.  This gives
you all the product source code and the buildroot version that will
build it.

Checking out an old version gives you the old code (any maybe an older
buildroot version too).

One develops by editing the source in this repo check out.  There is
nothing to setup (e.g. srcdir_override) to be in development mode.  One
commits to git directly from the checkout.

Once a commit is merged to master, it's part of the product.  There is
no additional step to update a git hash in a .mk file.

There is no difficulty if a single commit, or a series of commits part
of a single pull request, touch multiple packages (assume mypackage is
not the only package..).  This is how integration is done.  Multiple
commits to multiple packages could get merged into a topic branch, then
merged to master when the whole thing is ready.

A drawback here is that buildroot will not automatically rebuild a
package when the code is updated.  It's necessary to use the target
"mypackage-rebuild" to get that to happen.  A target that does this for
every package with SIT_METHOD == "local" is very handy.

I think buildroot could be made automatically detect changes by
comparing the timestamp of the .stamp_rsynced file vs the local
MYPACKAGE_SITE directory's timestamp.

[Buildroot] [PATCH 1/2] git: fix handling of git repos using master as version

[Baruch Siach]

Hi Arnout,

On Tue, Jun 05, 2018 at 12:19:41AM +0200, Arnout Vandecappelle wrote:
> On 04-06-18 18:52, Gary Bisson wrote:
> > On Mon, Jun 04, 2018 at 05:32:41PM +0100, Bob Beckett wrote:
> >> Thanks for the more in depth explanation.
> >>
> >> I agree with all of the reasons outlined for the purpose of being purely a
> >> reproducible build manager, which buildroot only ever aimed to be.
> >>
> >> However, people do use it during development, and with a reasonably large
> >> number of custom packages all in development at the same time, using the
> >> _OVERRIDE_SRCDIR method to test latest versions of multiple packages
> >> becomes very unwieldy.
> >> My solution so far has been to point the version at the branch that active
> >> development is taking place on, and removing the build directories and
> >> packages for the specific projects each time.
> >> When I am not the one doing the development on each package's source, but
> >> am developing the rootfs, this means I dont have to manually keep updating
> >> my local git repositories for each project.
> >> Once each package has hit their first release version, they do get tagged
> >> and the release branch for the project's BR2_EXTERNAL directory gets
> >> updated with those versions, and each version thereafter, but the
> >> development branch for the external directory persistently stays on the
> >> development branch head for each package.
> >>
> >> Regarding not knowing the sha before a checkout, would git ls-remote not
> >> suffice for this? e.g.
> >>
> >> $ git ls-remote git at gitlab.com:adhlinux/buildroot/buildroot.git master
> >> 407fb2fe202aaeb273e4986dc88c30596a7fe8db    refs/heads/master
> >> 407fb2fe202aaeb273e4986dc88c30596a7fe8db    refs/remotes/upstream/master
> > 
> > Yes 'ls-remote' is actually a good option. 
> 
>  It's a bit racy though. By the time you do the actual fetch, the remote branch
> may already have been updated by someone else.

Someone might update the branch while you are fetching as well. As long as you 
fetch a valid commit ID that is known to have been at the branch tip at some 
point in the recent past, it should be fine from the user perspective.

baruch

> You could have the following
> > package version during development:
> > FOO_VERSION = $(shell git ls-remote URL branch | awk '{ print $$1 }')
> > 
> > Then it would pick up any change automatically until you finish your dev
> > and change the FOO_VERSION to a proper ID.
> > 
> >> It would allow you to check for changes in sha without doing a new fetch.
> >>
> >> Oh well, not to worry. Ill keep it as a local patch, as I suspect I am
> >> treading old ground of "buildroot is not a development platform" (Ive seen
> >> this discussion come up multiple times before).
> > 
> > Yes that will most likely not be included inside BR infrastructure but that
> > should be a perfectly fine option for your custom package.
> > 
> > Regards,
> > Gary

-- 
     http://baruch.siach.name/blog/                  ~. .~   Tk Open Systems
=}------------------------------------------------ooO--U--Ooo------------{=
   - baruch at tkos.co.il - tel: +972.52.368.4656, http://www.tkos.co.il -

[Buildroot] [PATCH 1/2] git: fix handling of git repos using master as version

[Thomas Petazzoni]

Hello,

On Tue, 5 Jun 2018 00:19:41 +0200, Arnout Vandecappelle wrote:

> >> $ git ls-remote git at gitlab.com:adhlinux/buildroot/buildroot.git master
> >> 407fb2fe202aaeb273e4986dc88c30596a7fe8db    refs/heads/master
> >> 407fb2fe202aaeb273e4986dc88c30596a7fe8db    refs/remotes/upstream/master  
> > 
> > Yes 'ls-remote' is actually a good option.   
> 
>  It's a bit racy though. By the time you do the actual fetch, the remote branch
> may already have been updated by someone else.

And ?

Even if you get the latest commit in the branch, by the time the build
is finished, someone might have pushed some other changes.

There's nothing racy here: you are going to build the latest commit at
the moment of git ls-remote. There might obviously be further commits
made later on, either during the build or after the build. Whether
those commits were made right between ls-remote and the actual build,
or during the build, or after the build doesn't really matter.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 1/2] git: fix handling of git repos using master as version

[Thomas Petazzoni]

Hello,

On Tue, 5 Jun 2018 00:18:39 +0200, Arnout Vandecappelle wrote:

> On 04-06-18 18:51, Bob Beckett wrote:
> > Maybe a variable _IN_DEVELOPMENT or something could be added to a package's
> > makefile to indicate that the version is always considered out of date, and
> > should be re-fetched.  
> 
>  We're never going to consider something *always* out of date, I think. We might
> only consider it out-of-date when you do an explicit foo-rebuild.

Well, never say "never" :-)

During the development phase, if you have 50 or 100 in-house
components, it may be annoying to do a "foo-rebuild" for all of them in
order to get their latest version. But this can probably be resolved by
having a custom target (implemented locally by people who want that)
that triggers the rebuild of all packages they are interested in.

> > That way development branches for package specs could have the package spec
> > committed with that, while release branches do not and are expected to have a
> > non-branch version specifier.
> > 
> > With this strategy you would be acknowledging that you are not getting
> > reproducibility, but you are still getting reliable builds (the 2 reasons I use
> > buildroot).
> > The test for taint would then be to check if any packages have an
> > _IN_DEVELOPMENT variable set.  
> 
>  I'm not so fond of the _IN_DEVELOPMENT idea. What I like more is to extend
> _OVERRIDE_SRCDIR to support repositories in addition to local files. It would
> then check out directly into the build directory, without passing through DL_DIR
> or a tarball.
> 
>  This would probably be a rather complicated change however.

Is it that complicated to add a:

	cd $(pkg_OVERRIDE_SRCDIR); git pull

before doing the rsync of the source code ? Of course, we don't want to
do this unconditionally as soon as <pkg>_OVERRIDE_SRCDIR is set, so
perhaps a separate boolean variable is needed.

> > Given the current state, if you specify a branch of master, you get a warning
> > that it doesnt appear to be a special version, but that is all, and the package
> > doesnt build at all as it failed to get the source in the first place> I think if there is going to be an enforcement of not using branches, then the
> > build should probably fail in a more explicit way e.g. check to see if the
> > version is a branch. If it is, then fail the build with a message telling the
> > user that branches are not to be used as versions  
> 
>  I completely agree with this one, we should simply fail on branches.

Agreed.

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 1/2] git: fix handling of git repos using master as version

[Arnout Vandecappelle]

On 05-06-18 05:26, Baruch Siach wrote:
> Hi Arnout,
> 
> On Tue, Jun 05, 2018 at 12:19:41AM +0200, Arnout Vandecappelle wrote:
>> On 04-06-18 18:52, Gary Bisson wrote:
>>> On Mon, Jun 04, 2018 at 05:32:41PM +0100, Bob Beckett wrote:
>>>> Thanks for the more in depth explanation.
>>>>
>>>> I agree with all of the reasons outlined for the purpose of being purely a
>>>> reproducible build manager, which buildroot only ever aimed to be.
>>>>
>>>> However, people do use it during development, and with a reasonably large
>>>> number of custom packages all in development at the same time, using the
>>>> _OVERRIDE_SRCDIR method to test latest versions of multiple packages
>>>> becomes very unwieldy.
>>>> My solution so far has been to point the version at the branch that active
>>>> development is taking place on, and removing the build directories and
>>>> packages for the specific projects each time.
>>>> When I am not the one doing the development on each package's source, but
>>>> am developing the rootfs, this means I dont have to manually keep updating
>>>> my local git repositories for each project.
>>>> Once each package has hit their first release version, they do get tagged
>>>> and the release branch for the project's BR2_EXTERNAL directory gets
>>>> updated with those versions, and each version thereafter, but the
>>>> development branch for the external directory persistently stays on the
>>>> development branch head for each package.
>>>>
>>>> Regarding not knowing the sha before a checkout, would git ls-remote not
>>>> suffice for this? e.g.
>>>>
>>>> $ git ls-remote git at gitlab.com:adhlinux/buildroot/buildroot.git master
>>>> 407fb2fe202aaeb273e4986dc88c30596a7fe8db    refs/heads/master
>>>> 407fb2fe202aaeb273e4986dc88c30596a7fe8db    refs/remotes/upstream/master
>>>
>>> Yes 'ls-remote' is actually a good option. 
>>
>>  It's a bit racy though. By the time you do the actual fetch, the remote branch
>> may already have been updated by someone else.
> 
> Someone might update the branch while you are fetching as well. As long as you 
> fetch a valid commit ID that is known to have been at the branch tip at some 
> point in the recent past, it should be fine from the user perspective.

 Indeed, I was too quick, I hadn't understood that the idea was to use the
ls-remote result to set the version to check out.

> 
> baruch
> 
>> You could have the following
>>> package version during development:
>>> FOO_VERSION = $(shell git ls-remote URL branch | awk '{ print $$1 }')

 This might actually work pretty well... Better than my _OVERRIDE_SRCDIR
extension idea. So maybe something like

ifneq ($$($(2)_OVERRIDE_GIT_REPO),)
$(2)_VERSION = $$($(2)_OVERRIDE_GIT_REF)
$(2)_SITE = $$($(2)_OVERRIDE_GIT_REPO)
$(2)_SITE_METHOD = git
endif

and of course defaulting OVERRIDE_GIT_REF to master and defaulting
$(2)_OVERRIDE_GIT_REPO to $(3)_OVERRIDE_GIT_REPO.

 Regards,
 Arnout

>>>
>>> Then it would pick up any change automatically until you finish your dev
>>> and change the FOO_VERSION to a proper ID.
>>>
>>>> It would allow you to check for changes in sha without doing a new fetch.
>>>>
>>>> Oh well, not to worry. Ill keep it as a local patch, as I suspect I am
>>>> treading old ground of "buildroot is not a development platform" (Ive seen
>>>> this discussion come up multiple times before).
>>>
>>> Yes that will most likely not be included inside BR infrastructure but that
>>> should be a perfectly fine option for your custom package.
>>>
>>> Regards,
>>> Gary
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF