[Buildroot] [PATCH 1/1] support: git: Use git-archive to generate download tarball

[Buildroot] [PATCH 1/1] support: git: Use git-archive to generate download tarball

[Andrew Jeffery]

Using tar's --transform option with the previously specified regex
breaks relative symlinks, as the target path starts with the sequence
'./'. Avoid the need for --transform and indirectly generate a tar via
`git archive` to preserve accuracy of archive content.

Signed-off-by: Andrew Jeffery <andrew@aj.id.au>
---
 support/download/git | 20 +-------------------
 1 file changed, 1 insertion(+), 19 deletions(-)

diff --git a/support/download/git b/support/download/git
index 11bb52c1e13b..3b671d8adcef 100755
--- a/support/download/git
+++ b/support/download/git
@@ -178,24 +178,6 @@ if [ ${recurse} -eq 1 ]; then
     _git submodule update --init --recursive
 fi
 
-# Generate the archive, sort with the C locale so that it is reproducible.
-# We do not want the .git dir; we keep other .git files, in case they are the
-# only files in their directory.
-# The .git dir would generate non reproducible tarballs as it depends on
-# the state of the remote server. It also would generate large tarballs
-# (gigabytes for some linux trees) when a full clone took place.
-find . -not -type d \
-	-and -not -path "./.git/*" >"${output}.list"
-LC_ALL=C sort <"${output}.list" >"${output}.list.sorted"
-
-# Create GNU-format tarballs, since that's the format of the tarballs on
-# sources.buildroot.org and used in the *.hash files
-tar cf - --transform="s#^\./#${basename}/#" \
-         --numeric-owner --owner=0 --group=0 --mtime="${date}" --format=gnu \
-         -T "${output}.list.sorted" >"${output}.tar"
-gzip -6 -n <"${output}.tar" >"${output}"
-
-rm -f "${output}.list"
-rm -f "${output}.list.sorted"
+_git archive --format=tar --prefix="${basename}"/ HEAD | gzip -6 > "${output}"
 
 popd >/dev/null
-- 
2.17.1

[Buildroot] [PATCH 1/1] support: git: Use git-archive to generate download tarball

[Thomas Petazzoni]

Hello Andrew,

On Tue, 14 Aug 2018 21:35:32 +0930, Andrew Jeffery wrote:
> Using tar's --transform option with the previously specified regex
> breaks relative symlinks, as the target path starts with the sequence
> './'. Avoid the need for --transform and indirectly generate a tar via
> `git archive` to preserve accuracy of archive content.
> 
> Signed-off-by: Andrew Jeffery <andrew@aj.id.au>

See commit 3abd5ba4243489b821d44e407528020cd8a138f8
("support/download/git: do not use git archive, handle it manually")

    We currently use git-archive to generate the tarball. This is all handy
    and dandy, but git-archive does not support submodules. In the follow-up
    patch, we're going to handle submodules, so we would not be able to use
    git-archive.
    
    Instead, we manually generate the archive:
      - extract the tree to the requested cset,
      - get the date of the commit to store in the archive,
      - store only numeric owners,
      - store owner and group as 0 (zero, although any arbitrary value would
        have been fine, as long as it's a constant),
      - sort the files to store in the archive.
    
    We also get rid of the .git directory, because there is no reason to
    keep it in the context of Buildroot. Some people would love to keep it
    so as to speed up later downloads when updating a package, but that is
    not really doable. For example:
      - use current Buildroot
      - it would need foo-12345, so do a clone and keep the .git in the
        generated tarball
      - update Buildroot
      - it would need foo-98765
    For that second clone, how could we know we would have to first extract
    foo-12345 ? So, the .git in the archive is pretty much useless for
    Buildroot.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 1/1] support: git: Use git-archive to generate download tarball

[Andrew Jeffery]

Hi Thomas,

On Tue, 14 Aug 2018, at 21:59, Thomas Petazzoni wrote:
> Hello Andrew,
> 
> On Tue, 14 Aug 2018 21:35:32 +0930, Andrew Jeffery wrote:
> > Using tar's --transform option with the previously specified regex
> > breaks relative symlinks, as the target path starts with the sequence
> > './'. Avoid the need for --transform and indirectly generate a tar via
> > `git archive` to preserve accuracy of archive content.
> > 
> > Signed-off-by: Andrew Jeffery <andrew@aj.id.au>
> 
> See commit 3abd5ba4243489b821d44e407528020cd8a138f8
> ("support/download/git: do not use git archive, handle it manually")
> 
>     We currently use git-archive to generate the tarball. This is all handy
>     and dandy, but git-archive does not support submodules. In the follow-up
>     patch, we're going to handle submodules, so we would not be able to use
>     git-archive.

Agh, righto. I thought there must be some reason for avoiding it though didn't
look too deeply. I guess I should find some other way around the breakage I'm
seeing.

Thanks for the pointer.

Andrew