"git archve --format=tar" output changed from 1.8.1 to 1.8.2.1

"git archve --format=tar" output changed from 1.8.1 to 1.8.2.1

[Greg KH]

Hi,

The way we upload the Linux kernel to kernel.org involves creating a tar
archive, signing the archive, and then just uploading the signature.
The server then checks out the repo based on the tag, generates the tar
archive and checks the signature to make sure they match.

A few days ago I released the 3.0.61 kernel, and it turned out that I
couldn't upload the kernel release because 'git archive' now creates a
binary file that differs from an older version of git.

I tracked this down to commit 22f0dcd9634a818a0c83f23ea1a48f2d620c0546
(archive-tar: split long paths more carefully).  The diff of a hex dump
of the tar archives shows the following difference:

--- old_git_archive	2013-01-31 17:31:24.466343388 +0100
+++ new_git_archive	2013-01-31 17:32:21.509674417 +0100
@@ -19239998,8 +19239998,8 @@
 125943d0:0000 0000 0000 0000 0000 0000 0000 0000  ................
 125943e0:0000 0000 0000 0000 0000 0000 0000 0000  ................
 125943f0:0000 0000 0000 0000 0000 0000 0000 0000  ................
-12594400:0000 0000 0000 0000 0000 0000 0000 0000  ................
-12594410:0000 0000 0000 0000 0000 0000 0000 0000  ................
+12594400:7765 7374 6272 6964 6765 2d6f 6d61 7033  westbridge-omap3
+12594410:2d70 6e61 6e64 2d68 616c 2f00 0000 0000  -pnand-hal/.....
 12594420:0000 0000 0000 0000 0000 0000 0000 0000  ................
 12594430:0000 0000 0000 0000 0000 0000 0000 0000  ................
 12594440:0000 0000 0000 0000 0000 0000 0000 0000  ................
@@ -19240025,8 +19240025,8 @@
 12594580:2f61 7374 6f72 6961 2f61 7263 682f 6172  /astoria/arch/ar
 12594590:6d2f 706c 6174 2d6f 6d61 702f 696e 636c  m/plat-omap/incl
 125945a0:7564 652f 6d61 6368 2f77 6573 7462 7269  ude/mach/westbri
-125945b0:6467 652f 7765 7374 6272 6964 6765 2d6f  dge/westbridge-o
-125945c0:6d61 7033 2d70 6e61 6e64 2d68 616c 0000  map3-pnand-hal..
+125945b0:6467 6500 0000 0000 0000 0000 0000 0000  dge.............
+125945c0:0000 0000 0000 0000 0000 0000 0000 0000  ................
 125945d0:0000 0000 0000 0000 0000 0000 0000 0000  ................
 125945e0:0000 0000 0000 0000 0000 0000 0000 0000  ................
 125945f0:0000 0000 0000 0000 0000 0000 0000 0000  ................

Interestingly, the output of uncompressing the tar archives is
identical, so the data is correct, but the binary isn't.

Now keeping binary compatibility of tar archive files isn't really a big
deal, but, the commit to git that causes this seems a bit odd, is it
really needed?  Or can we just fix the version of tar with NetBSD
instead?  :)

Any ideas?

thanks,

greg k-h

Re: "git archve --format=tar" output changed from 1.8.1 to 1.8.2.1

[Junio C Hamano]

Greg KH <gregkh@linuxfoundation.org> writes:

> The way we upload the Linux kernel to kernel.org involves creating a tar
> archive, signing the archive, and then just uploading the signature.
> The server then checks out the repo based on the tag, generates the tar
> archive and checks the signature to make sure they match.
> 
> A few days ago I released the 3.0.61 kernel, and it turned out that I
> couldn't upload the kernel release because 'git archive' now creates a
> binary file that differs from an older version of git.
> ...
> Now keeping binary compatibility of tar archive files isn't really a big
> deal, but, the commit to git that causes this seems a bit odd, is it
> really needed?  Or can we just fix the version of tar with NetBSD
> instead?  :)
>
> Any ideas?

How about fixing kup to teach the "let's cheat and let the other end
run 'git archive', if the resulting archive and GPG signature
locally created does match, we do not have to transfer the tarball
itself" trick a fall-back mode that says "but if the signature does
not match, then transfer the bulk used to create the signature to
the remote anyway".  This fallback can and should of course be
useful for the compressed patch transfer.

Re: "git archve --format=tar" output changed from 1.8.1 to 1.8.2.1

[Greg KH]

On Thu, Jan 31, 2013 at 09:32:12AM -0800, Junio C Hamano wrote:
> Greg KH <gregkh@linuxfoundation.org> writes:
> 
> > The way we upload the Linux kernel to kernel.org involves creating a tar
> > archive, signing the archive, and then just uploading the signature.
> > The server then checks out the repo based on the tag, generates the tar
> > archive and checks the signature to make sure they match.
> > 
> > A few days ago I released the 3.0.61 kernel, and it turned out that I
> > couldn't upload the kernel release because 'git archive' now creates a
> > binary file that differs from an older version of git.
> > ...
> > Now keeping binary compatibility of tar archive files isn't really a big
> > deal, but, the commit to git that causes this seems a bit odd, is it
> > really needed?  Or can we just fix the version of tar with NetBSD
> > instead?  :)
> >
> > Any ideas?
> 
> How about fixing kup to teach the "let's cheat and let the other end
> run 'git archive', if the resulting archive and GPG signature
> locally created does match, we do not have to transfer the tarball
> itself" trick a fall-back mode that says "but if the signature does
> not match, then transfer the bulk used to create the signature to
> the remote anyway".  This fallback can and should of course be
> useful for the compressed patch transfer.

Ugh, uploading a 431Mb file, over a flaky wireless connection (I end up
doing lots of kernel releases while traveling), would be a horrible
change.  I'd rather just keep using the same older version of git that
kernel.org is running instead.

thanks,

greg k-h

Re: "git archve --format=tar" output changed from 1.8.1 to 1.8.2.1

[Konstantin Ryabitsev]

[-- Attachment #1: Type: text/plain, Size: 850 bytes --]

On 31/01/13 12:41 PM, Greg KH wrote:
> Ugh, uploading a 431Mb file, over a flaky wireless connection (I end up
> doing lots of kernel releases while traveling), would be a horrible
> change.  I'd rather just keep using the same older version of git that
> kernel.org is running instead.

Well, we do accept compressed archives, so you would be uploading about
80MB instead of 431MB, but that would still be a problem for anyone
releasing large tarballs over unreliable connections. I know you
routinely do 2-3 releases at once, so that would still mean uploading
120-180MB.

I don't have immediate statistics on how many people release using "kup
--tar", but I know that at least you and Linus rely on that exclusively.


Regards,
-- 
Konstantin Ryabitsev
Systems Administrator
Linux Foundation, kernel.org
Montréal, Québec


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 730 bytes --]

Re: "git archve --format=tar" output changed from 1.8.1 to 1.8.2.1

[Junio C Hamano]

Greg KH <gregkh@linuxfoundation.org> writes:

> On Thu, Jan 31, 2013 at 09:32:12AM -0800, Junio C Hamano wrote:
>
>> How about fixing kup to teach the "let's cheat and let the other end
>> run 'git archive', if the resulting archive and GPG signature
>> locally created does match, we do not have to transfer the tarball
>> itself" trick a fall-back mode that says "but if the signature does
>> not match, then transfer the bulk used to create the signature to
>> the remote anyway".  This fallback can and should of course be
>> useful for the compressed patch transfer.
>
> Ugh, uploading a 431Mb file, over a flaky wireless connection (I end up
> doing lots of kernel releases while traveling), would be a horrible
> change.  I'd rather just keep using the same older version of git that
> kernel.org is running instead.

Then how about fixing kup to try both versions of Git?  There will
be people who run different versions of Git anyway, and kup should
not be preventing Git from helping people on other platforms, or
improving its output in general.

Re: "git archve --format=tar" output changed from 1.8.1 to 1.8.2.1

[René Scharfe]

Am 31.01.2013 18:28, schrieb Greg KH:
> I tracked this down to commit 22f0dcd9634a818a0c83f23ea1a48f2d620c0546
> (archive-tar: split long paths more carefully).  The diff of a hex dump
> of the tar archives shows the following difference:
> 
> --- old_git_archive	2013-01-31 17:31:24.466343388 +0100
> +++ new_git_archive	2013-01-31 17:32:21.509674417 +0100
> @@ -19239998,8 +19239998,8 @@
>  125943d0:0000 0000 0000 0000 0000 0000 0000 0000  ................
>  125943e0:0000 0000 0000 0000 0000 0000 0000 0000  ................
>  125943f0:0000 0000 0000 0000 0000 0000 0000 0000  ................
> -12594400:0000 0000 0000 0000 0000 0000 0000 0000  ................
> -12594410:0000 0000 0000 0000 0000 0000 0000 0000  ................
> +12594400:7765 7374 6272 6964 6765 2d6f 6d61 7033  westbridge-omap3
> +12594410:2d70 6e61 6e64 2d68 616c 2f00 0000 0000  -pnand-hal/.....
>  12594420:0000 0000 0000 0000 0000 0000 0000 0000  ................
>  12594430:0000 0000 0000 0000 0000 0000 0000 0000  ................
>  12594440:0000 0000 0000 0000 0000 0000 0000 0000  ................
> @@ -19240025,8 +19240025,8 @@
>  12594580:2f61 7374 6f72 6961 2f61 7263 682f 6172  /astoria/arch/ar
>  12594590:6d2f 706c 6174 2d6f 6d61 702f 696e 636c  m/plat-omap/incl
>  125945a0:7564 652f 6d61 6368 2f77 6573 7462 7269  ude/mach/westbri
> -125945b0:6467 652f 7765 7374 6272 6964 6765 2d6f  dge/westbridge-o
> -125945c0:6d61 7033 2d70 6e61 6e64 2d68 616c 0000  map3-pnand-hal..
> +125945b0:6467 6500 0000 0000 0000 0000 0000 0000  dge.............
> +125945c0:0000 0000 0000 0000 0000 0000 0000 0000  ................
>  125945d0:0000 0000 0000 0000 0000 0000 0000 0000  ................
>  125945e0:0000 0000 0000 0000 0000 0000 0000 0000  ................
>  125945f0:0000 0000 0000 0000 0000 0000 0000 0000  ................

This is the only directory in the repository whose path is long enough to
make a difference with the patch, 105 characters in total:

drivers/staging/westbridge/astoria/arch/arm/plat-omap/include/mach/westbridge/westbridge-omap3-pnand-hal/

Five characters less and you wouldn't notice a thing.  It contains
"westbridge" thrice, so I think it's cheating just to reach that
length, though. ;-)

> Interestingly, the output of uncompressing the tar archives is
> identical, so the data is correct, but the binary isn't.

The path is split differently between two header fields, that's all.

> Now keeping binary compatibility of tar archive files isn't really a big
> deal, but, the commit to git that causes this seems a bit odd, is it
> really needed?  Or can we just fix the version of tar with NetBSD
> instead?  :)

Apart from Junio's suggestion, I can't think of a practical solution.

You could downgrade your git to a version before the fix.  A downside is
that you won't be able to extract the archive on NetBSD without getting
an error message (but the contents would be intact, except perhaps for
permission bits of the directory above).

You could upgrade the kernel.org version of git, but that might cause the
same problem for other maintainers with long directory paths who in their
repositories who still use git without the fix.

You could make the path shorter.  Won't help at all with the release you
just did, of course.

I don't know if other tar implementations freak out when they see an
empty name field.  NetBSD's tar might seem a bit too strict here, but
overall I think it's right in complaining.

What makes the commit odd, by the way?

Thanks,
René

Re: "git archve --format=tar" output changed from 1.8.1 to 1.8.2.1

[Greg KH]

On Thu, Jan 31, 2013 at 07:33:52PM +0100, René Scharfe wrote:
> Am 31.01.2013 18:28, schrieb Greg KH:
> > I tracked this down to commit 22f0dcd9634a818a0c83f23ea1a48f2d620c0546
> > (archive-tar: split long paths more carefully).  The diff of a hex dump
> > of the tar archives shows the following difference:
> > 
> > --- old_git_archive	2013-01-31 17:31:24.466343388 +0100
> > +++ new_git_archive	2013-01-31 17:32:21.509674417 +0100
> > @@ -19239998,8 +19239998,8 @@
> >  125943d0:0000 0000 0000 0000 0000 0000 0000 0000  ................
> >  125943e0:0000 0000 0000 0000 0000 0000 0000 0000  ................
> >  125943f0:0000 0000 0000 0000 0000 0000 0000 0000  ................
> > -12594400:0000 0000 0000 0000 0000 0000 0000 0000  ................
> > -12594410:0000 0000 0000 0000 0000 0000 0000 0000  ................
> > +12594400:7765 7374 6272 6964 6765 2d6f 6d61 7033  westbridge-omap3
> > +12594410:2d70 6e61 6e64 2d68 616c 2f00 0000 0000  -pnand-hal/.....
> >  12594420:0000 0000 0000 0000 0000 0000 0000 0000  ................
> >  12594430:0000 0000 0000 0000 0000 0000 0000 0000  ................
> >  12594440:0000 0000 0000 0000 0000 0000 0000 0000  ................
> > @@ -19240025,8 +19240025,8 @@
> >  12594580:2f61 7374 6f72 6961 2f61 7263 682f 6172  /astoria/arch/ar
> >  12594590:6d2f 706c 6174 2d6f 6d61 702f 696e 636c  m/plat-omap/incl
> >  125945a0:7564 652f 6d61 6368 2f77 6573 7462 7269  ude/mach/westbri
> > -125945b0:6467 652f 7765 7374 6272 6964 6765 2d6f  dge/westbridge-o
> > -125945c0:6d61 7033 2d70 6e61 6e64 2d68 616c 0000  map3-pnand-hal..
> > +125945b0:6467 6500 0000 0000 0000 0000 0000 0000  dge.............
> > +125945c0:0000 0000 0000 0000 0000 0000 0000 0000  ................
> >  125945d0:0000 0000 0000 0000 0000 0000 0000 0000  ................
> >  125945e0:0000 0000 0000 0000 0000 0000 0000 0000  ................
> >  125945f0:0000 0000 0000 0000 0000 0000 0000 0000  ................
> 
> This is the only directory in the repository whose path is long enough to
> make a difference with the patch, 105 characters in total:
> 
> drivers/staging/westbridge/astoria/arch/arm/plat-omap/include/mach/westbridge/westbridge-omap3-pnand-hal/
> 
> Five characters less and you wouldn't notice a thing.  It contains
> "westbridge" thrice, so I think it's cheating just to reach that
> length, though. ;-)

Yeah, that file path was bad, which is just one reason why it was
deleted from the kernel in newer versions :)

> > Interestingly, the output of uncompressing the tar archives is
> > identical, so the data is correct, but the binary isn't.
> 
> The path is split differently between two header fields, that's all.

Ok, thanks for the explaination, I didn't realize that.

> > Now keeping binary compatibility of tar archive files isn't really a big
> > deal, but, the commit to git that causes this seems a bit odd, is it
> > really needed?  Or can we just fix the version of tar with NetBSD
> > instead?  :)
> 
> Apart from Junio's suggestion, I can't think of a practical solution.
> 
> You could downgrade your git to a version before the fix.  A downside is
> that you won't be able to extract the archive on NetBSD without getting
> an error message (but the contents would be intact, except perhaps for
> permission bits of the directory above).
> 
> You could upgrade the kernel.org version of git, but that might cause the
> same problem for other maintainers with long directory paths who in their
> repositories who still use git without the fix.
> 
> You could make the path shorter.  Won't help at all with the release you
> just did, of course.

What I ended up doing was just to revert your patch, generating a tar
archive that matches what the version on kernel.org.

And originally I now recall that this was something we were worried
about, but we put off dealing with it until it caused problems :)

> I don't know if other tar implementations freak out when they see an
> empty name field.  NetBSD's tar might seem a bit too strict here, but
> overall I think it's right in complaining.

Ok, thanks, I now agree.

> What makes the commit odd, by the way?

Sorry, I was originally thinking that you were working around a bug in
the NetBSD version of tar, not making it "more correct", which is
arguably the right thing to do here.

So I'll work with Konstantine to ensure we both are using the same
version of git in the future, it's our kernel.org infrastructure issue
here, not a git one, sorry for the noise.

greg k-h

Re: "git archve --format=tar" output changed from 1.8.1 to 1.8.2.1

[Greg KH]

On Thu, Jan 31, 2013 at 10:16:27AM -0800, Junio C Hamano wrote:
> Greg KH <gregkh@linuxfoundation.org> writes:
> 
> > On Thu, Jan 31, 2013 at 09:32:12AM -0800, Junio C Hamano wrote:
> >
> >> How about fixing kup to teach the "let's cheat and let the other end
> >> run 'git archive', if the resulting archive and GPG signature
> >> locally created does match, we do not have to transfer the tarball
> >> itself" trick a fall-back mode that says "but if the signature does
> >> not match, then transfer the bulk used to create the signature to
> >> the remote anyway".  This fallback can and should of course be
> >> useful for the compressed patch transfer.
> >
> > Ugh, uploading a 431Mb file, over a flaky wireless connection (I end up
> > doing lots of kernel releases while traveling), would be a horrible
> > change.  I'd rather just keep using the same older version of git that
> > kernel.org is running instead.
> 
> Then how about fixing kup to try both versions of Git?  There will
> be people who run different versions of Git anyway, and kup should
> not be preventing Git from helping people on other platforms, or
> improving its output in general.

I think the combinations of different versions of git that would have to
be installed on kernel.org to handle stuff like this as things change
over time, wouldn't be worth it.

The number of people this affects right now is only one (me), given that
the offending file is not in Linus's tree right now, so he doesn't have
issues with uploading new releases.

So I'll just work to ensure I have the same version of git in place if I
ever run into this problem again.  Or just break down and do
full-compressed tarballs instead, if I'm in a place where I have a good
network connection.

thanks,

greg k-h

Re: "git archve --format=tar" output changed from 1.8.1 to 1.8.2.1

[Greg KH]

On Thu, Jan 31, 2013 at 12:52:15PM -0500, Konstantin Ryabitsev wrote:
> On 31/01/13 12:41 PM, Greg KH wrote:
> > Ugh, uploading a 431Mb file, over a flaky wireless connection (I end up
> > doing lots of kernel releases while traveling), would be a horrible
> > change.  I'd rather just keep using the same older version of git that
> > kernel.org is running instead.
> 
> Well, we do accept compressed archives, so you would be uploading about
> 80MB instead of 431MB, but that would still be a problem for anyone
> releasing large tarballs over unreliable connections. I know you
> routinely do 2-3 releases at once, so that would still mean uploading
> 120-180MB.

That would mean I can't do kernel releases while on ferry rides, which
is probably a good thing in the end :)

> I don't have immediate statistics on how many people release using "kup
> --tar", but I know that at least you and Linus rely on that exclusively.

What causes you to upgrade the version of git on the server?  Are you
relying on packages for a distro, or is this "hand installed" by
yourself?  As long as I stay in lock-step with your updates, all should
be fine.

Oh, maybe we can report back to the user, the version of git that is
being used on the server, if the checksums don't match, so that I know
to at least see if my version is different from yours?

thanks,

greg k-h

Re: "git archve --format=tar" output changed from 1.8.1 to 1.8.2.1

[Junio C Hamano]

Greg KH <gregkh@linuxfoundation.org> writes:

>> Then how about fixing kup to try both versions of Git?  There will
>> be people who run different versions of Git anyway, and kup should
>> not be preventing Git from helping people on other platforms, or
>> improving its output in general.
>
> I think the combinations of different versions of git that would have to
> be installed on kernel.org to handle stuff like this as things change
> over time, wouldn't be worth it.

You make it sound as if you have to pick the right one among 47
different versions, but I think over the lifetime of Git there was
only one time that output from "diff" would have affected the kup's
trick to avoid the transmission of patch text, and another that
output from "tar-tree" (aka "archive --format=tar") would have
afffected the transmission of tarballs.  I do think it is feasible
if "kup" wanted to.

> The number of people this affects right now is only one (me), given that
> the offending file is not in Linus's tree right now, so he doesn't have
> issues with uploading new releases.

As a tree grows larger over time, it may be just a matter of time
for somebody else to be hit by another deep path, though.

Re: "git archve --format=tar" output changed from 1.8.1 to 1.8.2.1

[Greg KH]

On Sun, Feb 03, 2013 at 09:05:55PM -0800, Junio C Hamano wrote:
> Greg KH <gregkh@linuxfoundation.org> writes:
> > The number of people this affects right now is only one (me), given that
> > the offending file is not in Linus's tree right now, so he doesn't have
> > issues with uploading new releases.
> 
> As a tree grows larger over time, it may be just a matter of time
> for somebody else to be hit by another deep path, though.

I agree, and over time, everyone will have updated to a version of git
newer than 1.8.2.1 so we all will be fine again :)

thanks,

greg k-h