fatal: serious inflate inconsistency

fatal: serious inflate inconsistency

[Thomas Glanzmann]

Hello,
and here is another one I guess: ;-(

        (tomcat-05) [/usr/src] git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
        Initialized empty Git repository in /usr/src/linux-2.6/.git/
        remote: Generating pack...
        remote: Done counting 496911 objects.
        remote: Deltifying 496911 objects...
        remote:  100% (496911/496911) done
        Indexing 496911 objects...
        remote: Total 496911 (delta 401539), reused 493712 (delta 398653)
        100% (496911/496911) done
        Resolving 401539 deltas...
        fatal: serious inflate inconsistency
        fatal: index-pack died with error code 128
        fetch-pack from
        'git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git' failed.
        (tomcat-05) [/usr/src] git version
        git version 1.5.2.1

Greetings,
        Thomas

Re: fatal: serious inflate inconsistency

[Nicolas Pitre]

On Fri, 8 Jun 2007, Thomas Glanzmann wrote:

> Hello,
> and here is another one I guess: ;-(
> 
>         (tomcat-05) [/usr/src] git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
>         Initialized empty Git repository in /usr/src/linux-2.6/.git/
>         remote: Generating pack...
>         remote: Done counting 496911 objects.
>         remote: Deltifying 496911 objects...
>         remote:  100% (496911/496911) done
>         Indexing 496911 objects...
>         remote: Total 496911 (delta 401539), reused 493712 (delta 398653)
>         100% (496911/496911) done
>         Resolving 401539 deltas...
>         fatal: serious inflate inconsistency
>         fatal: index-pack died with error code 128
>         fetch-pack from
>         'git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git' failed.
>         (tomcat-05) [/usr/src] git version
>         git version 1.5.2.1

This is bad.

index-pack works like this:

1) it receives data over the network:
   1a) it writes a copy of that data to disk to create a pack
   1b) it deflates that data in parallel to compute its SHA1

2) when done, it resolves deltas and to do so:
   2a) it deflates the base object data from 1a
   2b) it applies the delta over the base object data then compute its 
       SHA1

In your case it seems that 1B and 2A don't produce the same deflated 
data somehow.

Is it reproducible?
Are you using Linux?  What kernel version?
Are you using NFS?

If it isn't reproducible then you might possibly have been the victim of 
faulty hardware.


Nicolas

Re: fatal: serious inflate inconsistency

[Thomas Glanzmann]

Hello,

> Is it reproducible?

I can try. But that produces 150 Mbyte Traffic on Kernel.org

> Are you using Linux?  What kernel version?

Yes, I do. Debian etch; 2.6.20

> Are you using NFS?

Nope.

> If it isn't reproducible then you might possibly have been the victim
> of faulty hardware.

I give it a try and come back to you. I am off the net for 1 hour.

        Thomas

Re: fatal: serious inflate inconsistency

[Thomas Glanzmann]

Hello,

> I give it a try and come back to you. I am off the net for 1 hour.

        (tomcat-05) [/usr/src] git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
        Initialized empty Git repository in /usr/src/linux-2.6/.git/
        remote: Generating pack...
        remote: Done counting 496911 objects.
        remote: Deltifying 496911 objects...
        remote:  100% (496911/496911) done
        Indexing 496911 objects...
        remote: Total 496911 (delta 401539), reused 493712 (delta 398653)
        100% (496911/496911) done
        Resolving 401539 deltas...
        fatal: serious inflate inconsistency
        fatal: index-pack died with error code 128
        fetch-pack from 'git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git' failed.

yes, it is reproducable. Again I am off the net for an hour.

        Thomas

Re: fatal: serious inflate inconsistency

[Thomas Glanzmann]

Hello again,

> yes, it is reproducable. Again I am off the net for an hour.

I used the debian packages that built a friend of mine:

        deb http://rmdir.de/~michael/git/ ./

If you want to reproduce it by yourself.

        Thomas

Re: fatal: serious inflate inconsistency

[Nicolas Pitre]

On Fri, 8 Jun 2007, Thomas Glanzmann wrote:

> Hello,
> 
> > I give it a try and come back to you. I am off the net for 1 hour.
> 
>         (tomcat-05) [/usr/src] git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
>         Initialized empty Git repository in /usr/src/linux-2.6/.git/
>         remote: Generating pack...
>         remote: Done counting 496911 objects.
>         remote: Deltifying 496911 objects...
>         remote:  100% (496911/496911) done
>         Indexing 496911 objects...
>         remote: Total 496911 (delta 401539), reused 493712 (delta 398653)
>         100% (496911/496911) done
>         Resolving 401539 deltas...
>         fatal: serious inflate inconsistency
>         fatal: index-pack died with error code 128
>         fetch-pack from 'git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git' failed.
> 
> yes, it is reproducable. Again I am off the net for an hour.

I can't reproduce that.

$ git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
Initialized empty Git repository in /home/nico/git/tst/linux-2.6/.git/
remote: Generating pack...
remote: Done counting 496911 objects.
remote: Deltifying 496911 objects...
remote:  100% (496911/496911) done
Indexing 496911 objects...
remote: Total 496911 (delta 401539), reused 493712 (delta 398653)
 100% (496911/496911) done
Resolving 401539 deltas...
 100% (401539/401539) done
Checking 22413 files out...
 100% (22413/22413) done
$ git version
git version 1.5.2.1

> I used the debian packages that built a friend of mine:
>
>         deb http://rmdir.de/~michael/git/ ./
>
> If you want to reproduce it by yourself.

I don't use Debian, sorry.


Nicolas

Re: fatal: serious inflate inconsistency

[Linus Torvalds]

On Fri, 8 Jun 2007, Thomas Glanzmann wrote:
> 
> yes, it is reproducable. Again I am off the net for an hour.

Could you compile your own git? If so, please try this patch on top of 
current git, to make the error message a bit more informative.

It does sound like a corrupt stream, but maybe debian has some strange 
zlib version that git doesn't like.

		Linus
---
 index-pack.c |    8 ++++++--
 1 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/index-pack.c b/index-pack.c
index 82c8da3..542687b 100644
--- a/index-pack.c
+++ b/index-pack.c
@@ -277,9 +277,13 @@ static void *get_data_from_pack(struct object_entry *obj)
 	stream.avail_in = len;
 	inflateInit(&stream);
 	while ((st = inflate(&stream, Z_FINISH)) == Z_OK);
+	if (st != Z_STREAM_END)
+		die("serious inflate inconsistency: %d (%s)",
+			st, stream.msg ? stream.msg : "no message");
+	if (stream.total_out != obj->size)
+		die("serious inflate inconsistency: got %lu bytes, expected %lu",
+			stream.total_out, obj->size);
 	inflateEnd(&stream);
-	if (st != Z_STREAM_END || stream.total_out != obj->size)
-		die("serious inflate inconsistency");
 	free(src);
 	return data;
 }

Re: fatal: serious inflate inconsistency

[Thomas Glanzmann]

Hello Linus,

> Could you compile your own git? If so, please try this patch on top of 
> current git, to make the error message a bit more informative.

        (tomcat-05) [/usr/src] scp 131.188.30.102:/var/tmp/sithglan/git-core-1.5.2.1/../git-core_1.5.2.1-1_i386.deb .
        Warning: Permanently added '131.188.30.102' (RSA) to the list of known hosts.
        git-core_1.5.2.1-1_i386.deb                                                    100% 3511KB   1.7MB/s   00:02
        (tomcat-05) [/usr/src] / dpkg -i git-core_1.5.2.1-1_i386.deb
        (Reading database ... 13273 files and directories currently installed.)
        Preparing to replace git-core 1:1.5.2.1-1 (using git-core_1.5.2.1-1_i386.deb) ...
        Unpacking replacement git-core ...
        Setting up git-core (1.5.2.1-1) ...
        (tomcat-05) [/usr/src] git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
        Initialized empty Git repository in /usr/src/linux-2.6/.git/
        remote: Generating pack...
        remote: Done counting 496911 objects.
        remote: Deltifying 496911 objects...
        remote:  100% (496911/496911) done
        Indexing 496911 objects...
        remote: Total 496911 (delta 401539), reused 493712 (delta 398653)
        100% (496911/496911) done
        Resolving 401539 deltas...
        fatal: serious inflate inconsistency: -3 (invalid distance too far back)
        fatal: index-pack died with error code 128
        fetch-pack from 'git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git' failed.

I talked to Michael and he is certain that the machine has a memory
corruption the next thing I am going to do is compiling a kernel in an
endless loop. Again if somone needs shell access just send me your ssh
key.

The complete build log is on (well not the complete log the first line is
missing because my screen backlog has _only_ 10000 lines (a memory issue)):

The first line is:
wget http://ftp.debian.org/debian/pool/main/g/git-core/git-core_1.5.2.1-1.dsc

The rest is here:

http://wwwcip.informatik.uni-erlangen.de/~sithglan/screen-exchange.bz2 (32k)

Thanks,
        Thomas

Re: fatal: serious inflate inconsistency

[Linus Torvalds]

On Fri, 8 Jun 2007, Thomas Glanzmann wrote:
>
>         fatal: serious inflate inconsistency: -3 (invalid distance too far back)

Whee. That's Z_DATA_ERROR, and yeah, I think the only thing that causes it 
is a corrupt zlib input stream.

> I talked to Michael and he is certain that the machine has a memory
> corruption the next thing I am going to do is compiling a kernel in an
> endless loop. Again if somone needs shell access just send me your ssh
> key.

I'd not expect memory corruption to be that *repeatable*. But the message 
certainly implies zlib data stream corruption _somewhere_, although the 
fact that it seems to be so repeatable does make me suspect program error.

Of course, it could be a hard bit-error in memory, but even then it would 
kind of have to hit the same page allocation history each time to be 
repeatable. And the page cache is the only thing that is that sticky under 
Linux, so it would have to be something like the zlib static data or code 
that had hit the memory corruption.

Sounds strange. I wonder if we might have finally hit a case of different 
versions of zlib acting differently?

But it _could_ just be a git bug too.

I don't think it's on the kernel.org side (or we would have had more 
reports of this), but maybe there is some wild pointer thing inside git, 
and nobody else noticed because it depends on some specific memory 
allocation pattern (which in turn depends on things like kernel memory 
layout choices, and on libc 'malloc()' implementation etc).

I'll build with ElectricFence, and see if it says anything for me. It 
passes all the tests, but fetching the kernel is almost certainly going to 
do things that we don't test for..

			Linus

Re: fatal: serious inflate inconsistency

[Linus Torvalds]

On Fri, 8 Jun 2007, Linus Torvalds wrote:
> 
> I'll build with ElectricFence, and see if it says anything for me. It 
> passes all the tests, but fetching the kernel is almost certainly going to 
> do things that we don't test for..

It passes smaller clones fine with ElectricFence (and bigger ones fail 
because EF uses *so* much memory for its checking that you cannot actually 
complete them), so I don't think this is a git bug. Sounds like either the 
hw problems are real, or there is some zlib (or other library) issue..

		Linus

Re: fatal: serious inflate inconsistency

[Thomas Glanzmann]

Hello Linus,

> It passes smaller clones fine with ElectricFence (and bigger ones fail 
> because EF uses *so* much memory for its checking that you cannot actually 
> complete them), so I don't think this is a git bug. Sounds like either the 
> hw problems are real, or there is some zlib (or other library) issue..

        (mailrelay) [/var/tmp] git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
        Initialized empty Git repository in /var/tmp/linux-2.6/.git/
        remote: Generating pack...
        remote: Done counting 496911 objects.
        remote: Deltifying 496911 objects...
        remote:  100% (496911/496911) done
        Indexing 496911 objects...
        remote: Total 496911 (delta 401539), reused 493712 (delta 398653)
        100% (496911/496911) done
        Resolving 401539 deltas...
        100% (401539/401539) done
        Checking 22413 files out...
        100% (22413/22413) done

I have a very similar machine. For this one it works. It has 32 bit kernel and
userland. Debian etch again. v2.6.20.

Does someone know a command for debian that checks the checksums for every
installed packages and the file that are included in that package?
I know there is one for rpm so there must be one for debian, too.

        Thomas

Re: fatal: serious inflate inconsistency

[Marco Roeland]

On Friday June 8th 2007 at 21:33 Thomas Glanzmann wrote:

> Does someone know a command for debian that checks the checksums for every
> installed packages and the file that are included in that package?

Try debsums. Without arguments it checks all installed packages and
their files, but you can also check individual packages.
-- 
Marco Roeland

Re: fatal: serious inflate inconsistency

[Thomas Glanzmann]

Hello,
the OS is not corrupt. So I am going to compile a few kernels and wait
for gcc to segfault.

        Thomas

Re: fatal: serious inflate inconsistency

[Thomas Glanzmann]

Hello again,
the machine that had the issues just got down because of a broken power
supply and took half the rack with it. So it was definitive a hardware
fault. Sorry for all the trouble. And the bottom line is: It was _not_
git, it was broken hardware!

        Thomas

Re: fatal: serious inflate inconsistency

[Linus Torvalds]

On Sat, 9 Jun 2007, Thomas Glanzmann wrote:
>
> the machine that had the issues just got down because of a broken power
> supply and took half the rack with it. So it was definitive a hardware
> fault. Sorry for all the trouble. And the bottom line is: It was _not_
> git, it was broken hardware!

Good. That was one of the design goals for git (ie the "you know you can 
trust the data" thing relies on very strong protection at all levels, 
even in the presense of disk/memory/cpu corruption).

I think we have a pretty good record on this: the only real git-induced 
corruption I can remember was the embarrassing bug with use-after-free of 
the CRLF auto-translation. But let's not get complacent.

		Linus

Re: fatal: serious inflate inconsistency

[Martin Langhoff]

On 6/9/07, Thomas Glanzmann <thomas@glanzmann.de> wrote:
> I have a very similar machine. For this one it works. It has 32 bit kernel and
> userland. Debian etch again. v2.6.20.

Wanted to mention -- we've been using git on debian sarge/etch on
several arches a lot and never seen any lib compatiblity problem.

> Does someone know a command for debian that checks the checksums

apt-get install debsums
man debsums # sorry, I don't use it often enough to remember ;-)

cheers,


m

Re: fatal: serious inflate inconsistency

[Thomas Glanzmann]

Hello,
the machine came back with a new power supply. And guess what:

(tomcat-05) [/usr/src] git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
Initialized empty Git repository in /usr/src/linux-2.6/.git/
remote: Generating pack...
remote: Done counting 499634 objects.
remote: Deltifying 499634 objects...
remote:  100% (499634/499634) done
Indexing 499634 objects...
remote: Total 499634 (delta 403845), reused 495761 (delta 400337)
100% (499634/499634) done
Resolving 403845 deltas...
100% (403845/403845) done
Checking 22411 files out...
100% (22411/22411) done

        Thomas

fatal: serious inflate inconsistency

[bob]

	From: 	  kranki@mac.com
	Subject: 	fatal: serious inflate inconsistency
	Date: 	November 8, 2007 10:56:02 PM EST
	To: 	  majordomo@vger.kernel.org

Hi, list.

Sorry, but I am not that familiar with git's internal workings,
but here is a failure that I can consistently create.  I am
running MacOSX 10.4.10 with git compiled from source.
Here is the problem output that I am receiving:

apple07:~/rmwHtmlOld bob$ git version
git version 1.5.3.5
apple07:~/rmwHtmlOld bob$ git init
Initialized empty Git repository in .git/
apple07:~/rmwHtmlOld bob$ git add .
apple07:~/rmwHtmlOld bob$ git commit --all
Created initial commit 49b4183:   Initial commit
10993 files changed, 3615639 insertions(+), 0 deletions(-)
create mode 100644 .gitignore
create mode 100755 bk.sh
.
.
.
create mode 100755 ter/webalizer/webalizer.hist
apple07:~/rmwHtmlOld bob$ cd ..
apple07:~ bob$ mkdir rmwHtml.git
apple07:~ bob$ cd rmwHtml.git/
apple07:~/rmwHtml.git bob$ git --bare init --share
usage: git-init [-q | --quiet] [--template=<template-directory>] [-- 
shared]
apple07:~/rmwHtml.git bob$ git --bare init --shared
Initialized empty shared Git repository in /Users/bob/rmwHtml.git/
apple07:~/rmwHtml.git bob$ git --bare fetch ../rmwHtmlOld master:master
warning: no common commits
remote: Generating pack...
remote: Done counting 11402 objects.
remote: Deltifying 11402 objects...
remote:  100% (11402/11402) done
Indexing 11402 objects...
100% (11402/11402) done
Resolving 3356 deltas...
fatal: serious inflate inconsistency
remote: Total 11402 (delta 3356), reused 0 (delta 0)
fatal: index-pack died with error code 128
fatal: Fetch failure: ../rmwHtmlOld
apple07:~/rmwHtml.git bob$

The ... represent about 11,000 files that I did not feel needed to be  
included.
The total directory being put in the repository is about 4gig.

There are two files that are about 1.0gig each.  I had them as one which
in MacOSX 10.4.10 showed as 1.88gig and think that this may be the
problem, but don't know why since my searches show that git should
handle files up to 2gig.  If I am wrong and there is a lesser limit,  
then
I apologize.

I have this directory, rmwHtmlOld, saved as a tarball and can recreate
the above as needed.

I apologize if I am doing something wrong that I am not aware of.
I use git for several different projects and have no problem with them.
I would appreciate any help that you can provide.  Thank you for
your time.

Re: fatal: serious inflate inconsistency

[Nicolas Pitre]

On Fri, 9 Nov 2007, bob wrote:

> Sorry, but I am not that familiar with git's internal workings,
> but here is a failure that I can consistently create.  I am
> running MacOSX 10.4.10 with git compiled from source.
> Here is the problem output that I am receiving:
> 
> remote: Generating pack...
> remote: Done counting 11402 objects.
> remote: Deltifying 11402 objects...
> remote:  100% (11402/11402) done
> Indexing 11402 objects...
> 100% (11402/11402) done
> Resolving 3356 deltas...
> fatal: serious inflate inconsistency

A similar problem (if not the same problem) has been reported on MacOSX 
in the past.  The conclusion was that either the gcc version or zlib 
version on MacOSX was bad and updating them fixed it.  I don't remember 
the details now but you should be able to find them in the mail archive 
(or maybe someone else remembers).


Nicolas

Re: fatal: serious inflate inconsistency

[bob]

Thank you, Nicolas.  I will look into installing a new
zlib.

I moved the two large files into their own repository
and then both repositories worked correctly.  So,
it must be something related to the size, hopefully,
in zlib.

Again, thanks for the reponse.

On Nov 9, 2007, at 11:11 AM, Nicolas Pitre wrote:

> On Fri, 9 Nov 2007, bob wrote:
>
>> Sorry, but I am not that familiar with git's internal workings,
>> but here is a failure that I can consistently create.  I am
>> running MacOSX 10.4.10 with git compiled from source.
>> Here is the problem output that I am receiving:
>>
>> remote: Generating pack...
>> remote: Done counting 11402 objects.
>> remote: Deltifying 11402 objects...
>> remote:  100% (11402/11402) done
>> Indexing 11402 objects...
>> 100% (11402/11402) done
>> Resolving 3356 deltas...
>> fatal: serious inflate inconsistency
>
> A similar problem (if not the same problem) has been reported on  
> MacOSX
> in the past.  The conclusion was that either the gcc version or zlib
> version on MacOSX was bad and updating them fixed it.  I don't  
> remember
> the details now but you should be able to find them in the mail  
> archive
> (or maybe someone else remembers).
>
>
> Nicolas
> -
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html