"git archive" seems to be broken wrt zip files

"git archive" seems to be broken wrt zip files

[Linus Torvalds]

So I wouldn't ever have noticed on my own, but now that I've tried
github for the kernel, somebody else reported that the downloaded zip
file (seriously? the kernel as a zip file?) is corrupt.

And it doesn't really seem to be a github issue. I can re-create it
with a simple

   git archive --format=zip HEAD -o ../kernel.zip

on my kernel repository: the end result does not unzip correctly:

   mkdir temp-directory
   cd temp-directory
   unzip kernel.zip
   ...
     inflating: virt/kvm/iommu.c
     inflating: virt/kvm/irq_comm.c
     inflating: virt/kvm/kvm_main.c
   finishing deferred symbolic links:
     arch/microblaze/boot/dts/system.dts -> ../../platform/generic/system.dts
     drivers/scsi/aic94xx/aic94xx_reg.h -> /*^J * Aic94xx SAS/SATA
driver hardware registers definitions.[ rest of the file ]
   symlink error: File name too long

iow, for some reason that "drivers/scsi/aic94xx/aic94xx_reg.h" file
seems to have been encoded as a symlink.

Anybody seen this?

                          Linus

Re: "git archive" seems to be broken wrt zip files

[Jeff King]

On Sat, Sep 10, 2011 at 09:58:08PM -0700, Linus Torvalds wrote:

> So I wouldn't ever have noticed on my own, but now that I've tried
> github for the kernel, somebody else reported that the downloaded zip
> file (seriously? the kernel as a zip file?) is corrupt.
> 
> And it doesn't really seem to be a github issue. I can re-create it
> with a simple
> 
>    git archive --format=zip HEAD -o ../kernel.zip
> 
> on my kernel repository: the end result does not unzip correctly:

Hmm. I can easily replicate the problem here, but interestingly it does
not happen with sub-trees like:

  git archive --format=zip HEAD:drivers -o ../kernel.zip

Going back in history, I can replicate it with René's 62cdce1
(git-archive --format=zip: add symlink support, 2006-10-07). So there's
nothing to bisect.

Cc'ing René.

-Peff

Re: "git archive" seems to be broken wrt zip files

[Jeff King]

On Sun, Sep 11, 2011 at 02:22:06AM -0400, Jeff King wrote:

> Hmm. I can easily replicate the problem here, but interestingly it does
> not happen with sub-trees like:
> 
>   git archive --format=zip HEAD:drivers -o ../kernel.zip
> 
> Going back in history, I can replicate it with René's 62cdce1
> (git-archive --format=zip: add symlink support, 2006-10-07). So there's
> nothing to bisect.

Weirder still. I get roughly the same output as you:

  finishing deferred symbolic links:
    arch/microblaze/boot/dts/system.dts -> ../../platform/generic/system.dts
    drivers/scsi/aic94xx/aic94xx_reg.h -> /*^J * Aic94xx SAS/SATA driver...

But looking at the generated file with zipinfo, I see:

  $ zipinfo kernel.zip
  ...
  lrwxrwxrwx  2.3 unx       33 b- stor 11-Aug-25 14:02 arch/microblaze/boot/dts/system.dts
  ...
  -rw----     0.0 fat    10470 b- defN 11-Aug-25 14:02 drivers/scsi/aic94xx/aic94xx_reg.h

IOW, the zip file looks right. I wonder if this is actually a bug in
"unzip".

-Peff

Re: "git archive" seems to be broken wrt zip files

[Andreas Schwab]

Jeff King <peff@peff.net> writes:

> IOW, the zip file looks right. I wonder if this is actually a bug in
> "unzip".

It is.  This only happens if you have more then 16k entries and when one
of the 16k entry infos is reused it happend to be previously used for a
symlink entry.

Here's a patch for unzip60 for reference:

--- process.c
+++ process.c
@@ -1751,6 +1751,12 @@ int process_cdir_file_hdr(__G)    /* ret
         = (G.crec.general_purpose_bit_flag & (1 << 11)) == (1 << 11);
 #endif
 
+#ifdef SYMLINKS
+    /* Initialize the symlink flag, may be set by the platform-specific
+       mapattr function.  */
+    G.pInfo->symlink = 0;
+#endif
+
     return PK_COOL;
 
 } /* end function process_cdir_file_hdr() */

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: "git archive" seems to be broken wrt zip files

[René Scharfe]

Am 11.09.2011 15:14, schrieb Andreas Schwab:
> Jeff King <peff@peff.net> writes:
> 
>> IOW, the zip file looks right. I wonder if this is actually a bug in
>> "unzip".
> 
> It is.  This only happens if you have more then 16k entries and when one
> of the 16k entry infos is reused it happend to be previously used for a
> symlink entry.
> 
> Here's a patch for unzip60 for reference:

Oh, thanks, now I can stop scratching my head while trying to find my
way around unzip's source and its amazing OS compatibility code.

On Windows (the main target for ZIP file support in git archive), 7-Zip,
the unzip from msysgit and the native ZIP support extract the archive
without any issues (except for 13 files whose names only differ
upper/lower case of some chars to 13 other files), by the way.  Since
they lack symlink support, the target path of
arch/microblaze/boot/dts/system.dts is simply written to a regular file,
though.

René

Re: "git archive" seems to be broken wrt zip files

[Linus Torvalds]

Re-sent as this clearly never seems to have gone anywhere due to me
not realizing that the LF problems meant that while incoming email
worked fine, going out through smpt.linux-foundation.org didn't work.

                     Linus

On Sun, Sep 11, 2011 at 11:07 AM, Linus Torvalds
<torvalds@linux-foundation.org> wrote:
> On Sun, Sep 11, 2011 at 6:14 AM, Andreas Schwab <schwab@linux-m68k.org> wrote:
>>
>> It is.  This only happens if you have more then 16k entries and when one
>> of the 16k entry infos is reused it happend to be previously used for a
>> symlink entry.
>
> Thanks for the explanation.
>
>> Here's a patch for unzip60 for reference:
>
> Is this problem known to the unzip developers? Is it actively
> maintained any more? The 6.0 release seems to have been from April
> 2009, and the web page seems to have a "last updated" of September
> 2009.. Is there some other unofficial archive for bugfixes like this?
> Distros?
>
>                            Linus
>

Re: "git archive" seems to be broken wrt zip files

[Andreas Schwab]

Linus Torvalds <torvalds@linux-foundation.org> writes:

>> Is this problem known to the unzip developers?

I have already sent them the patch and they have acknowledged it.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."