[Buildroot] [PATCH 0/4 v3] core/download: add support for git sub-modules (branch yem/git)

[Buildroot] [PATCH 0/4 v3] core/download: add support for git sub-modules (branch yem/git)

[Yann E. MORIN]

Hello All!

This little series adds support for packages coming from a git clone,
with git sub-modules.

The commit logs are pretty detailed, but roughly, it goes as thus:

  - we can't get sub-modules in a bare clone, so we no longer use bare
    clones, but use full clones (with a working copy);

  - git-archive does not support submodules, so we generate archives
    manually;

  - we check out the working copy to the correct cset, then we retrieve
    the submodules;

  - we expose a new package variable to request sub-modules.


Changes v2 -> v3:
  - added reviewd-by and tested-by tags
  - typoes fixed
  - rebased

Changes v1 -> v2:
  - properly accept -r in the download wrapper  (Aleksandar)
  - split the non-bare-git and manual-archive changes in two separate
    patches, as they are not semantically related
  - some of my usual typoes fixed...


Regards,
Yann E. MORIN.


The following changes since commit a542d0fd55e961e3c5805f12bba490d3dcc15d33

  erlang: enable compilation on mipsel (2016-07-01 00:44:36 +0200)


are available in the git repository at:

  https://git.buildroot.org/~ymorin/git/buildroot

for you to fetch changes up to b9061693f4d7a2c7030190efd686c988c1278e43

  core/pkg-infra: download git submodules if the package wants them (2016-07-01 10:53:32 +0200)


----------------------------------------------------------------
Yann E. MORIN (4):
      support/download/git: do not use bare clones
      support/download/git: do not use git archive, handle it manually
      support/download/git: add support for submodules
      core/pkg-infra: download git submodules if the package wants them

 package/pkg-download.mk     |  1 +
 package/pkg-generic.mk      |  8 +++++++
 support/download/dl-wrapper |  7 ++++---
 support/download/git        | 51 ++++++++++++++++++++++++++++++++++++++-------
 4 files changed, 57 insertions(+), 10 deletions(-)

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 1/4 v3] support/download/git: do not use bare clones

[Yann E. MORIN]

Currently, we are using bare clones, so as to minimise the disk usage,
most notably for largeish repositories such as the one for the Linux
kernel, which can go beyond the 1GiB barrier.

However, this precludes updating (and thus using) the submodules, if
any, of the repositories, as a working copy is required to use
submodules (becaue we need to know the list of submodules, where to find
them, where to clone them, what cset to checkout, and all those is
dependent upon the checked out cset of the father repository).

Switch to using /plain/ clones with a working copy.

This means that the extra refs used by some forges (like pull-requests
for Github, or changes for gerrit...) are no longer fetched as part of
the clone, because git does not offer to do a mirror clone when there is
a working copy.

Instead, we have to fetch those special refs by hand. Since there is no
easy solution to know whether the cset the user asked for is such a
special ref or not, we just try to always fetch the cset requested by
the user; if this fails, we assume that this is not a special ref (most
probably, it is a sha1) and we defer the check to the archive creation,
which would fail if the requested cset is missing anyway.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Tested-by: Matt Weber <matt@thewebers.ws>
Reviewed-by: Matt Weber <matt@thewebers.ws>
---
 support/download/git | 21 ++++++++++++++++++---
 1 file changed, 18 insertions(+), 3 deletions(-)

diff --git a/support/download/git b/support/download/git
index 314b388..2d98923 100755
--- a/support/download/git
+++ b/support/download/git
@@ -41,7 +41,7 @@ _git() {
 git_done=0
 if [ -n "$(_git ls-remote "'${repo}'" "'${cset}'" 2>&1)" ]; then
     printf "Doing shallow clone\n"
-    if _git clone ${verbose} --depth 1 -b "'${cset}'" --bare "'${repo}'" "'${basename}'"; then
+    if _git clone ${verbose} --depth 1 -b "'${cset}'" "'${repo}'" "'${basename}'"; then
         git_done=1
     else
         printf "Shallow clone failed, falling back to doing a full clone\n"
@@ -49,10 +49,25 @@ if [ -n "$(_git ls-remote "'${repo}'" "'${cset}'" 2>&1)" ]; then
 fi
 if [ ${git_done} -eq 0 ]; then
     printf "Doing full clone\n"
-    _git clone ${verbose} --mirror "'${repo}'" "'${basename}'"
+    _git clone ${verbose} "'${repo}'" "'${basename}'"
+fi
+
+pushd "${basename}" >/dev/null
+
+# Try to get the special refs exposed by some forges (pull-requests for
+# github, changes for gerrit...). There is no easy way to know whether
+# the cset the user passed us is such a special ref or a tag or a sha1
+# or whatever else. We'll eventually fail at checking out that cset,
+# below, if there is an issue anyway. Since most of the cset we're gonna
+# have to clone are not such special refs, consign the output to oblivion
+# so as not to alarm unsuspecting users, but still trace it as a warning.
+if ! _git fetch origin "'${cset}:${cset}'" >/dev/null 2>&1; then
+    printf "Could not fetch special ref '%s'; assuming it is not special.\n" "${cset}"
 fi
 
-GIT_DIR="${basename}" \
 _git archive --prefix="'${basename}/'" -o "'${output}.tmp'" --format=tar "'${cset}'"
 
+# Not really required, but here for consistency
+popd >/dev/null
+
 gzip -n <"${output}.tmp" >"${output}"
-- 
2.7.4

[Buildroot] [PATCH 1/4 v3] support/download/git: do not use bare clones

[Peter Korsgaard]

>>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:

 > Currently, we are using bare clones, so as to minimise the disk usage,
 > most notably for largeish repositories such as the one for the Linux
 > kernel, which can go beyond the 1GiB barrier.

 > However, this precludes updating (and thus using) the submodules, if
 > any, of the repositories, as a working copy is required to use
 > submodules (becaue we need to know the list of submodules, where to find
 > them, where to clone them, what cset to checkout, and all those is
 > dependent upon the checked out cset of the father repository).

 > Switch to using /plain/ clones with a working copy.

 > This means that the extra refs used by some forges (like pull-requests
 > for Github, or changes for gerrit...) are no longer fetched as part of
 > the clone, because git does not offer to do a mirror clone when there is
 > a working copy.

 > Instead, we have to fetch those special refs by hand. Since there is no
 > easy solution to know whether the cset the user asked for is such a
 > special ref or not, we just try to always fetch the cset requested by
 > the user; if this fails, we assume that this is not a special ref (most
 > probably, it is a sha1) and we defer the check to the archive creation,
 > which would fail if the requested cset is missing anyway.

FYI, I did a try with:

git clone -c remote.origin.fetch='+refs/*:refs/*' <url> to be able to do
this in one go, but it doesn't work. git clone still only fetches
refs/heads and you need to run a git pull afterwards, so it doesn't
really buy us much.

We could in theory do git init + git fetch instead of git clone, but
that doesn't look any simpler than what you propose here.

As sub modules are quite rare and we know when they are requested (using
foo_GIT_SUBMODULES), we COULD keep the existing code path around for
"normal" repos, but that would mean we would need separate code paths
for the clone/archive steps for the two variants, so that is probably
not worth it.

Committed, thanks.

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH 2/4 v3] support/download/git: do not use git archive, handle it manually

[Yann E. MORIN]

We currently use git-archive to generate the tarball. This is all handy
and dandy, but git-archive does not support submodules. In the follow-up
patch, we're going to handle submodules, so we would not be able to use
git-archive.

Instead, we manually generate the archive:
  - extract the tree to the requested cset,
  - get the date of the commit to store in the archive,
  - store only numeric owners,
  - store owner and group as 0 (zero, although any arbitrary value would
    have been fine, as long as it's a constant),
  - sort the files to store in the archive.

We also get rid of the .git directory, because there is no reason to
keep it in the context of Buildroot. Some people would love to keep it
so as to speed up later downloads when updating a package, but that is
not really doable. For example:
  - use current Buildroot
  - it would need foo-12345, so do a clone and keep the .git in the
    generated tarball
  - update Buildroot
  - it would need foo-98765
For that second clone, how could we know we would have to first extract
foo-12345 ? So, the .git in the archive is pretty much useless for
Buildroot.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Tested-by: Matt Weber <matt@thewebers.ws>
Reviewed-by: Matt Weber <matt@thewebers.ws>
---
 support/download/git | 17 ++++++++++++++---
 1 file changed, 14 insertions(+), 3 deletions(-)

diff --git a/support/download/git b/support/download/git
index 2d98923..54dd2c7 100755
--- a/support/download/git
+++ b/support/download/git
@@ -65,9 +65,20 @@ if ! _git fetch origin "'${cset}:${cset}'" >/dev/null 2>&1; then
     printf "Could not fetch special ref '%s'; assuming it is not special.\n" "${cset}"
 fi
 
-_git archive --prefix="'${basename}/'" -o "'${output}.tmp'" --format=tar "'${cset}'"
+# Checkout the required changeset.
+_git checkout -q "'${cset}'"
+
+# Get date of commit to generate a reproducible archive.
+# %cD is RFC2822, so it's fully qualified, with TZ and all.
+date="$( _git show --no-patch --pretty=format:%cD )"
+
+# We do not need the .git dir and other gitfiles to generate the tarball
+find . \( -name .git -o -name .gitmodules -o -name .gitignore \) \
+       -exec rm -rf {} +
 
-# Not really required, but here for consistency
 popd >/dev/null
 
-gzip -n <"${output}.tmp" >"${output}"
+# Generate the archive, sort with the C locale so that it is reproducible.
+tar cf - --numeric-owner --owner=0 --group=0 --mtime="${date}" \
+         -T <(find "${basename}" -not -type d |LC_ALL=C sort) \
+|gzip -n >"${output}"
-- 
2.7.4

[Buildroot] [PATCH 2/4 v3] support/download/git: do not use git archive, handle it manually

[Peter Korsgaard]

>>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:

 > We currently use git-archive to generate the tarball. This is all handy
 > and dandy, but git-archive does not support submodules. In the follow-up
 > patch, we're going to handle submodules, so we would not be able to use
 > git-archive.

 > Instead, we manually generate the archive:
 >   - extract the tree to the requested cset,
 >   - get the date of the commit to store in the archive,
 >   - store only numeric owners,
 >   - store owner and group as 0 (zero, although any arbitrary value would
 >     have been fine, as long as it's a constant),
 >   - sort the files to store in the archive.

 > We also get rid of the .git directory, because there is no reason to
 > keep it in the context of Buildroot. Some people would love to keep it
 > so as to speed up later downloads when updating a package, but that is
 > not really doable. For example:
 >   - use current Buildroot
 >   - it would need foo-12345, so do a clone and keep the .git in the
 >     generated tarball
 >   - update Buildroot
 >   - it would need foo-98765
 > For that second clone, how could we know we would have to first extract
 > foo-12345 ? So, the .git in the archive is pretty much useless for
 > Buildroot.

I've wondered if it wouldn't have been simpler to keep git archive and
then just append the git-archive output of the individual submodules
(like we do for the generated documentation in make release) with git
submodule foreach, E.G:

https://ttboj.wordpress.com/2015/07/23/git-archive-with-submodules-and-tar-magic/

Something like:

git archive --prefix=foo/ ... > $TAR
git submodule foreach --recursive \
    'git archive --prefix=foo/$path/ $sha1 > $TAR.tmp && tar Af $TAR $TAR.tmp'
rm $TAR.tmp

But we can do that later if needed.

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH 2/4 v3] support/download/git: do not use git archive, handle it manually

[Ben Boeckel]

On Sat, 02 Jul, 2016 at 17:08:04 GMT, Peter Korsgaard wrote:
> I've wondered if it wouldn't have been simpler to keep git archive and
> then just append the git-archive output of the individual submodules
> (like we do for the generated documentation in make release) with git
> submodule foreach, E.G:
>
> https://ttboj.wordpress.com/2015/07/23/git-archive-with-submodules-and-tar-magic/
>
> Something like:
>
> git archive --prefix=foo/ ... > $TAR
> git submodule foreach --recursive \
>     'git archive --prefix=foo/$path/ $sha1 > $TAR.tmp && tar Af $TAR $TAR.tmp'
> rm $TAR.tmp
>
> But we can do that later if needed.

Here's another approach:

    https://gitlab.kitware.com/paraview/paraview/blob/master/Utilities/Maintenance/create_tarballs.bash#L176

You set up a custom index file and then use `read-tree -i` to read in
the submodules. IMO, using `git archive` should be a requirement because
of export-subst attributes which cause `$Format:$` bits to be replaced
in the archived files.

--Ben

[Buildroot] [PATCH 3/4 v3] support/download/git: add support for submodules

[Yann E. MORIN]

Some git repositories may be split into a master repository and
submodules. Up until now, we did not have support for submodules,
because we were using bare clones, in which it is not possible to
update the list of submodules.

Now that we are using plain clones with a working copy, we can retrieve
the submdoules.

Add an option to the git downlaod helper to kick the update of
submodules, so that they are only fetched for those packages that
require them. Also document the existing -q option at the same time.

Submodules have a .git file at their root, which contains the path to
the real .git directory of the master repository. Since we remove it,
there is no point in keeping those .git files either.

Note: this is currently unused, but will be enabled with the follow-up
patch , that adds the necessary parts in the pkg-generic and pkg-download
infrastructures.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Tested-by: Matt Weber <matt@thewebers.ws>
Reviewed-by: Matt Weber <matt@thewebers.ws>
---
 support/download/git | 17 ++++++++++++++---
 1 file changed, 14 insertions(+), 3 deletions(-)

diff --git a/support/download/git b/support/download/git
index 54dd2c7..44a5682 100755
--- a/support/download/git
+++ b/support/download/git
@@ -6,15 +6,20 @@ set -e
 # Download helper for git, to be called from the download wrapper script
 #
 # Call it as:
-#   .../git [-q] OUT_FILE REPO_URL CSET BASENAME
+#   .../git [-q] [-r] OUT_FILE REPO_URL CSET BASENAME
+#
+#   -q  Be quiet.
+#   -r  Clone and archive sub-modules.
 #
 # Environment:
 #   GIT      : the git command to call
 
 verbose=
-while getopts :q OPT; do
+recurse=0
+while getopts :qr OPT; do
     case "${OPT}" in
     q)  verbose=-q; exec >/dev/null;;
+    r)  recurse=1;;
     \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
     esac
 done
@@ -65,13 +70,19 @@ if ! _git fetch origin "'${cset}:${cset}'" >/dev/null 2>&1; then
     printf "Could not fetch special ref '%s'; assuming it is not special.\n" "${cset}"
 fi
 
-# Checkout the required changeset.
+# Checkout the required changeset, so that we can update the required
+# submodules.
 _git checkout -q "'${cset}'"
 
 # Get date of commit to generate a reproducible archive.
 # %cD is RFC2822, so it's fully qualified, with TZ and all.
 date="$( _git show --no-patch --pretty=format:%cD )"
 
+# There might be submodules, so fetch them.
+if [ ${recurse} -eq 1 ]; then
+    _git submodule update --init --recursive
+fi
+
 # We do not need the .git dir and other gitfiles to generate the tarball
 find . \( -name .git -o -name .gitmodules -o -name .gitignore \) \
        -exec rm -rf {} +
-- 
2.7.4

[Buildroot] [PATCH 3/4 v3] support/download/git: add support for submodules

[Peter Korsgaard]

>>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:

 > Some git repositories may be split into a master repository and
 > submodules. Up until now, we did not have support for submodules,
 > because we were using bare clones, in which it is not possible to
 > update the list of submodules.

 > Now that we are using plain clones with a working copy, we can retrieve
 > the submdoules.

 > Add an option to the git downlaod helper to kick the update of
 > submodules, so that they are only fetched for those packages that
 > require them. Also document the existing -q option at the same time.

 > Submodules have a .git file at their root, which contains the path to
 > the real .git directory of the master repository. Since we remove it,
 > there is no point in keeping those .git files either.

 > Note: this is currently unused, but will be enabled with the follow-up
 > patch , that adds the necessary parts in the pkg-generic and pkg-download
 > infrastructures.

 > Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
 > Tested-by: Matt Weber <matt@thewebers.ws>
 > Reviewed-by: Matt Weber <matt@thewebers.ws>

Committed, thanks.

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH 4/4 v3] core/pkg-infra: download git submodules if the package wants them

[Yann E. MORIN]

Add a new package variable that packages can set to specify that they
need git submodules.

Only accept this option if the download method is git, as we can not get
submodules via an http download (via wget).

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Aleksandar Simeonov <aleksandar@barix.com>
Tested-by: Matt Weber <matt@thewebers.ws>
Reviewed-by: Matt Weber <matt@thewebers.ws>
Tested-By: Nicolas Cavallari <nicolas.cavallari@green-communications.fr>

---
Changes v1 -> v2:
  - properly accept the -r in the download wrapper  (Aleksandar)
---
 package/pkg-download.mk     | 1 +
 package/pkg-generic.mk      | 8 ++++++++
 support/download/dl-wrapper | 7 ++++---
 3 files changed, 13 insertions(+), 3 deletions(-)

diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index a0f694d..315203f 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -76,6 +76,7 @@ export BR_NO_CHECK_HASH_FOR =
 define DOWNLOAD_GIT
 	$(EXTRA_ENV) $(DL_WRAPPER) -b git \
 		-o $(DL_DIR)/$($(PKG)_SOURCE) \
+		$(if $($(PKG)_GIT_SUBMODULES),-r) \
 		$(QUIET) \
 		-- \
 		$($(PKG)_SITE) \
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index b712c31..fa404ca 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -468,6 +468,14 @@ ifndef $(2)_SITE_METHOD
  endif
 endif
 
+# Do not accept to download git submodule if not using the git method
+ifneq ($$($(2)_GIT_SUBMODULES),)
+ ifneq ($$($(2)_SITE_METHOD),git)
+  $$(error $(2) declares having git sub-modules, but does not use the \
+	   'git' method (uses '$$($(2)_SITE_METHOD)' instead))
+ endif
+endif
+
 ifeq ($$($(2)_SITE_METHOD),local)
 ifeq ($$($(2)_OVERRIDE_SRCDIR),)
 $(2)_OVERRIDE_SRCDIR = $$($(2)_SITE)
diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper
index ef2d872..f944b71 100755
--- a/support/download/dl-wrapper
+++ b/support/download/dl-wrapper
@@ -21,15 +21,16 @@ set -e
 
 main() {
     local OPT OPTARG
-    local backend output hfile quiet
+    local backend output hfile recurse quiet
 
     # Parse our options; anything after '--' is for the backend
-    while getopts :hb:o:H:q OPT; do
+    while getopts :hb:o:H:rq OPT; do
         case "${OPT}" in
         h)  help; exit 0;;
         b)  backend="${OPTARG}";;
         o)  output="${OPTARG}";;
         H)  hfile="${OPTARG}";;
+        r)  recurse="-r";;
         q)  quiet="-q";;
         :)  error "option '%s' expects a mandatory argument\n" "${OPTARG}";;
         \?) error "unknown option '%s'\n" "${OPTARG}";;
@@ -82,7 +83,7 @@ main() {
     # If the backend fails, we can just remove the temporary directory to
     # remove all the cruft it may have left behind. Then we just exit in
     # error too.
-    if ! "${OLDPWD}/support/download/${backend}" ${quiet} "${tmpf}" "${@}"; then
+    if ! "${OLDPWD}/support/download/${backend}" ${quiet} ${recurse} "${tmpf}" "${@}"; then
         rm -rf "${tmpd}"
         exit 1
     fi
-- 
2.7.4

[Buildroot] [PATCH 4/4 v3] core/pkg-infra: download git submodules if the package wants them

[Peter Korsgaard]

>>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:

 > Add a new package variable that packages can set to specify that they
 > need git submodules.

 > Only accept this option if the download method is git, as we can not get
 > submodules via an http download (via wget).

 > Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
 > Cc: Aleksandar Simeonov <aleksandar@barix.com>
 > Tested-by: Matt Weber <matt@thewebers.ws>
 > Reviewed-by: Matt Weber <matt@thewebers.ws>
 > Tested-By: Nicolas Cavallari <nicolas.cavallari@green-communications.fr>

 > ---
 > Changes v1 -> v2:
 >   - properly accept the -r in the download wrapper  (Aleksandar)

Committed, thanks.

Will you also send a patch to update the user manual?

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH 4/4 v3] core/pkg-infra: download git submodules if the package wants them

[Yann E. MORIN]

Peter, All,

On 2016-07-02 19:11 +0200, Peter Korsgaard spake thusly:
> >>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:
>  > Add a new package variable that packages can set to specify that they
>  > need git submodules.
[--SNIP--]
> Will you also send a patch to update the user manual?

Say "please", and I'll see what I can do... ;-)

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 4/4 v3] core/pkg-infra: download git submodules if the package wants them

[Peter Korsgaard]

>>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:

 > Peter, All,
 > On 2016-07-02 19:11 +0200, Peter Korsgaard spake thusly:
 >> >>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:
 >> > Add a new package variable that packages can set to specify that they
 >> > need git submodules.
 > [--SNIP--]
 >> Will you also send a patch to update the user manual?

 > Say "please", and I'll see what I can do... ;-)

Pretty please with sugar on top! ;)

-- 
Venlig hilsen,
Peter Korsgaard