Endianness bug in git-svn or called component?

Endianness bug in git-svn or called component?

[Nathaniel W Filardo]

[-- Attachment #1: Type: text/plain, Size: 1454 bytes --]

On this machine,

mirrors hydra:/tank0/mirrors/misc% uname -a
FreeBSD hydra.priv.oc.ietfng.org 9.0-CURRENT FreeBSD 9.0-CURRENT #13: Sat Nov 14 19:40:25 EST 2009 root@hydra.priv.oc.ietfng.org:/systank/obj/systank/src/sys/NWFKERN  sparc64

attempting to fetch from an svn source yields the following error:

rs hydra:/tank0/mirrors/misc% git svn init -s https://joshua.svn.sourceforge.net/svnroot/joshua test-joshua
Initialized empty Git repository in /tank0/mirrors/misc/test-joshua/.git/                                       
mirrors hydra:/tank0/mirrors/misc% cd test-joshua                                                               
mirrors hydra:/tank0/mirrors/misc/test-joshua% git svn fetch
        A       scripts/distributedLM/config.template       
[...]
        A       build.xml
r1 = fe84a7d821ec6d92da75133ac7ad1deb476b6484 (refs/remotes/trunk)
error: index uses  extension, which we do not understand
fatal: index file corrupt
write-tree: command returned error: 128

Even on previously initialized and fetched git-svn repos, "git svn fetch"
produces the same response.

This is the same symptom as an older bug, but I have not attempted to track
down what's going wrong this time in hopes that somebody more familiar can
fix it faster; see
http://kerneltrap.org/mailarchive/git/2006/5/28/205750/thread#mid-205750 .

If there's anything more that would be helpful to know, please just ask.

Thanks.
--nwf;

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: Endianness bug in git-svn or called component?

[Andreas Schwab]

Nathaniel W Filardo <nwf@cs.jhu.edu> writes:

> On this machine,
>
> mirrors hydra:/tank0/mirrors/misc% uname -a
> FreeBSD hydra.priv.oc.ietfng.org 9.0-CURRENT FreeBSD 9.0-CURRENT #13: Sat Nov 14 19:40:25 EST 2009 root@hydra.priv.oc.ietfng.org:/systank/obj/systank/src/sys/NWFKERN  sparc64
>
> attempting to fetch from an svn source yields the following error:
>
> rs hydra:/tank0/mirrors/misc% git svn init -s https://joshua.svn.sourceforge.net/svnroot/joshua test-joshua
> Initialized empty Git repository in /tank0/mirrors/misc/test-joshua/.git/                                       
> mirrors hydra:/tank0/mirrors/misc% cd test-joshua                                                               
> mirrors hydra:/tank0/mirrors/misc/test-joshua% git svn fetch
>         A       scripts/distributedLM/config.template       
> [...]
>         A       build.xml
> r1 = fe84a7d821ec6d92da75133ac7ad1deb476b6484 (refs/remotes/trunk)
> error: index uses  extension, which we do not understand
> fatal: index file corrupt
> write-tree: command returned error: 128

I could not reproduce that on powerpc (both 32bit and 64bit).

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Endianness bug in git-svn or called component?

[Nathaniel W Filardo]

[-- Attachment #1: Type: text/plain, Size: 1618 bytes --]

On Fri, Dec 04, 2009 at 09:16:40PM +0100, Andreas Schwab wrote:
> Nathaniel W Filardo <nwf@cs.jhu.edu> writes:
> 
> > On this machine,
> >
> > mirrors hydra:/tank0/mirrors/misc% uname -a
> > FreeBSD hydra.priv.oc.ietfng.org 9.0-CURRENT FreeBSD 9.0-CURRENT #13: Sat Nov 14 19:40:25 EST 2009 root@hydra.priv.oc.ietfng.org:/systank/obj/systank/src/sys/NWFKERN  sparc64
> >
> > attempting to fetch from an svn source yields the following error:
> >
> > rs hydra:/tank0/mirrors/misc% git svn init -s https://joshua.svn.sourceforge.net/svnroot/joshua test-joshua
> > Initialized empty Git repository in /tank0/mirrors/misc/test-joshua/.git/                                       
> > mirrors hydra:/tank0/mirrors/misc% cd test-joshua                                                               
> > mirrors hydra:/tank0/mirrors/misc/test-joshua% git svn fetch
> >         A       scripts/distributedLM/config.template       
> > [...]
> >         A       build.xml
> > r1 = fe84a7d821ec6d92da75133ac7ad1deb476b6484 (refs/remotes/trunk)
> > error: index uses  extension, which we do not understand
> > fatal: index file corrupt
> > write-tree: command returned error: 128
> 
> I could not reproduce that on powerpc (both 32bit and 64bit).
> 
> Andreas.

Hm.  I seem to have forgotten to mention that I am running

% git --version
git version 1.6.5.3

built from the FreeBSD ports tree, in case that matters, using

% gcc --version
gcc (GCC) 4.2.1 20070719  [FreeBSD]

Knowing nothing of git's internals, how should I start to investigate what's
going on here?

Thanks again.
--nwf;

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: Endianness bug in git-svn or called component?

[Nathaniel W Filardo]

[-- Attachment #1: Type: text/plain, Size: 2214 bytes --]

On Fri, Dec 04, 2009 at 03:29:28PM -0500, Nathaniel W Filardo wrote:
> On Fri, Dec 04, 2009 at 09:16:40PM +0100, Andreas Schwab wrote:
> > Nathaniel W Filardo <nwf@cs.jhu.edu> writes:
> > 
> > > On this machine,
> > >
> > > mirrors hydra:/tank0/mirrors/misc% uname -a
> > > FreeBSD hydra.priv.oc.ietfng.org 9.0-CURRENT FreeBSD 9.0-CURRENT #13: Sat Nov 14 19:40:25 EST 2009 root@hydra.priv.oc.ietfng.org:/systank/obj/systank/src/sys/NWFKERN  sparc64
> > >
> > > attempting to fetch from an svn source yields the following error:
> > >
> > > rs hydra:/tank0/mirrors/misc% git svn init -s https://joshua.svn.sourceforge.net/svnroot/joshua test-joshua
> > > Initialized empty Git repository in /tank0/mirrors/misc/test-joshua/.git/                                       
> > > mirrors hydra:/tank0/mirrors/misc% cd test-joshua                                                               
> > > mirrors hydra:/tank0/mirrors/misc/test-joshua% git svn fetch
> > >         A       scripts/distributedLM/config.template       
> > > [...]
> > >         A       build.xml
> > > r1 = fe84a7d821ec6d92da75133ac7ad1deb476b6484 (refs/remotes/trunk)
> > > error: index uses  extension, which we do not understand
> > > fatal: index file corrupt
> > > write-tree: command returned error: 128
> > 
> > I could not reproduce that on powerpc (both 32bit and 64bit).
> > 
> > Andreas.

I got some free time and tracked it down.  The following one-line delta
fixes this issue for me; AIUI on sparc64 "unsigned long" is 8 bits and in
the wrong endianness for the memcpy trick to work as written?  I could be
sligntly off in my assessment of the problem, tho'.

index 1bbaf1c..9033dd3 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -1322,7 +1322,7 @@ int read_index_from(struct index_state *istate, const char *path)
                 * extension name (4-byte) and section length
                 * in 4-byte network byte order.
                 */
-               unsigned long extsize;
+               uint32_t extsize;
                memcpy(&extsize, (char *)mmap + src_offset + 4, 4);
                extsize = ntohl(extsize);
                if (read_index_extension(istate,

--nwf;

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

[RESEND] [PATCH] Endianness bug in index cache logic

[Nathaniel W Filardo]

[-- Attachment #1: Type: text/plain, Size: 2214 bytes --]

On Fri, Dec 04, 2009 at 03:29:28PM -0500, Nathaniel W Filardo wrote:
> On Fri, Dec 04, 2009 at 09:16:40PM +0100, Andreas Schwab wrote:
> > Nathaniel W Filardo <nwf@cs.jhu.edu> writes:
> > 
> > > On this machine,
> > >
> > > mirrors hydra:/tank0/mirrors/misc% uname -a
> > > FreeBSD hydra.priv.oc.ietfng.org 9.0-CURRENT FreeBSD 9.0-CURRENT #13: Sat Nov 14 19:40:25 EST 2009 root@hydra.priv.oc.ietfng.org:/systank/obj/systank/src/sys/NWFKERN  sparc64
> > >
> > > attempting to fetch from an svn source yields the following error:
> > >
> > > rs hydra:/tank0/mirrors/misc% git svn init -s https://joshua.svn.sourceforge.net/svnroot/joshua test-joshua
> > > Initialized empty Git repository in /tank0/mirrors/misc/test-joshua/.git/                                       
> > > mirrors hydra:/tank0/mirrors/misc% cd test-joshua                                                               
> > > mirrors hydra:/tank0/mirrors/misc/test-joshua% git svn fetch
> > >         A       scripts/distributedLM/config.template       
> > > [...]
> > >         A       build.xml
> > > r1 = fe84a7d821ec6d92da75133ac7ad1deb476b6484 (refs/remotes/trunk)
> > > error: index uses  extension, which we do not understand
> > > fatal: index file corrupt
> > > write-tree: command returned error: 128
> > 
> > I could not reproduce that on powerpc (both 32bit and 64bit).
> > 
> > Andreas.

I got some free time and tracked it down.  The following one-line delta
fixes this issue for me; AIUI on sparc64 "unsigned long" is 8 bits and in
the wrong endianness for the memcpy trick to work as written?  I could be
sligntly off in my assessment of the problem, tho'.

index 1bbaf1c..9033dd3 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -1322,7 +1322,7 @@ int read_index_from(struct index_state *istate, const char *path)
                 * extension name (4-byte) and section length
                 * in 4-byte network byte order.
                 */
-               unsigned long extsize;
+               uint32_t extsize;
                memcpy(&extsize, (char *)mmap + src_offset + 4, 4);
                extsize = ntohl(extsize);
                if (read_index_extension(istate,

--nwf;

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: [RESEND] [PATCH] Endianness bug in index cache logic

[Erik Faye-Lund]

On Sun, Dec 27, 2009 at 7:11 AM, Nathaniel W Filardo <nwf@cs.jhu.edu> wrote:
> I got some free time and tracked it down.  The following one-line delta
> fixes this issue for me; AIUI on sparc64 "unsigned long" is 8 bits and in

You mean 8 bytes, right?

-- 
Erik "kusma" Faye-Lund

Re: [RESEND] [PATCH] Endianness bug in index cache logic

[Nathaniel W Filardo]

[-- Attachment #1: Type: text/plain, Size: 378 bytes --]

On Sun, Dec 27, 2009 at 01:39:24PM +0100, Erik Faye-Lund wrote:
> On Sun, Dec 27, 2009 at 7:11 AM, Nathaniel W Filardo <nwf@cs.jhu.edu> wrote:
> > I got some free time and tracked it down.  The following one-line delta
> > fixes this issue for me; AIUI on sparc64 "unsigned long" is 8 bits and in
> 
> You mean 8 bytes, right?

Yes, sorry.  How embarrassing. :)
--nwf;

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: [RESEND] [PATCH] Endianness bug in index cache logic

[Junio C Hamano]

Nathaniel W Filardo <nwf@cs.jhu.edu> writes:

> I got some free time and tracked it down.  The following one-line delta
> fixes this issue for me; AIUI on sparc64 "unsigned long" is 8 bits and in
> the wrong endianness for the memcpy trick to work as written?  I could be
> sligntly off in my assessment of the problem, tho'.
>
> index 1bbaf1c..9033dd3 100644
> --- a/read-cache.c
> +++ b/read-cache.c
> @@ -1322,7 +1322,7 @@ int read_index_from(struct index_state *istate, const char *path)
>                  * extension name (4-byte) and section length
>                  * in 4-byte network byte order.
>                  */
> -               unsigned long extsize;
> +               uint32_t extsize;
>                 memcpy(&extsize, (char *)mmap + src_offset + 4, 4);
>                 extsize = ntohl(extsize);
>                 if (read_index_extension(istate,

Good catch.

The original is broken on big endian 64-bit platforms, and your sparc64
indeed is one.  The code expects to see a signature (4-byte) at src_offset
followed by length (4-byte int in network byte order) and it is trying to
read read the latter in extsize.

Thanks for the fix.  The bug dates back to late April 2006, and I am kind
of surprised that nobody reported this since then (perhaps nobody runs git
on big endian 64-bit boxes?).

Re: [RESEND] [PATCH] Endianness bug in index cache logic

[Tomas Carnecky]

On 12/27/09 7:24 PM, Junio C Hamano wrote:
> Nathaniel W Filardo<nwf@cs.jhu.edu>  writes:
>
>> I got some free time and tracked it down.  The following one-line delta
>> fixes this issue for me; AIUI on sparc64 "unsigned long" is 8 bits and in
>> the wrong endianness for the memcpy trick to work as written?  I could be
>> sligntly off in my assessment of the problem, tho'.
>>
>> index 1bbaf1c..9033dd3 100644
>> --- a/read-cache.c
>> +++ b/read-cache.c
>> @@ -1322,7 +1322,7 @@ int read_index_from(struct index_state *istate, const char *path)
>>                   * extension name (4-byte) and section length
>>                   * in 4-byte network byte order.
>>                   */
>> -               unsigned long extsize;
>> +               uint32_t extsize;
>>                  memcpy(&extsize, (char *)mmap + src_offset + 4, 4);
>>                  extsize = ntohl(extsize);
>>                  if (read_index_extension(istate,
>
> Good catch.
>
> The original is broken on big endian 64-bit platforms, and your sparc64
> indeed is one.  The code expects to see a signature (4-byte) at src_offset
> followed by length (4-byte int in network byte order) and it is trying to
> read read the latter in extsize.
>
> Thanks for the fix.  The bug dates back to late April 2006, and I am kind
> of surprised that nobody reported this since then (perhaps nobody runs git
> on big endian 64-bit boxes?).

Both the native Sun compiler as well as GCC default to 32bit binaries, 
even if the system is capable of running 64bit binaries (unlike for 
example Linux where x86_64-pc-linux-gnu-gcc produces 64bit binaries by 
default). And the git makefile doesn't use -m64, so my guess is that 
nobody bothered changing CFLAGS on sparc64 systems (I have access to a 
couple such systems and I never changed CFLAGS because git always worked 
out of the box). Though I don't know what the default is on other big 
endian systems.

tom