git packs

git packs

[bob]

When a repository is packed such as for a clone or fetch, is there  
just one pack file created that is used for the transfer?

Re: git packs

[Nicolas Pitre]

On Fri, 9 Nov 2007, bob wrote:

> When a repository is packed such as for a clone or fetch, is there just one
> pack file created that is used for the transfer?

Yes.

And modern Git is able to handle packs larger than 4GB too, assuming it 
is compiled using a toolchain with large file support.


Nicolas

Re: git packs

[bob]

When you say toolchain, are you referring to the compiler
and associated libraries or are you referring to OS programs
such as ls, md5, cat, etc or both?

The reason that I ask is that I have been playing different
scenarios using git 1.5.3.5 under MacOSX 10.4.10 mostly
all day and every time that

A) 	a file approaches or exceeds 2gig on an 'add', it
	results in:
	
	fatal: Out of memory? mmap failed: Cannot allocate memory



B) 	the repository size less the .git subdirectory approaches
	4gig on a 'fetch' it results in:

	Resolving 3356 deltas...
	fatal: serious inflate inconsistency: -3 (unknown compression method)
	fatal: index-pack died with error code 128
	fatal: Fetch failure: ../rmwHtmlOld

	Under B, building the initial repository works fine.

(I added a patch the Linus Torvalds gave out when a previous inflate  
problem
was being researched.)  Also, I have been looking in the source
in particular in builtin-add.c builtin-pack-objects.c and associated  
headers
and see int and unsigned long being used a lot, but not any unsigned  
long
longs.  I have been testing on my laptop which has a 32-bit Intel  
Core Duo.
Also, I have run the same tests on a dual quad-core Intel processor
which is 64 bit, (but not sure that Apple uses the 64 bits in  
10.4.10).  I
get the same results as above.

The zlib is at the latest revision of 1.2.3 and gcc is at 4.0.1
which from what I can tell supports large files, because 'off_t' is 8  
bytes
which is the size used for a 'stat' file size.

I am just wondering if these size limitations exist for MacOSX
or maybe I am doing something wrong (which is probably
the case).


On Nov 10, 2007, at 12:13 AM, Nicolas Pitre wrote:

> On Fri, 9 Nov 2007, bob wrote:
>
>> When a repository is packed such as for a clone or fetch, is there  
>> just one
>> pack file created that is used for the transfer?
>
> Yes.
>
> And modern Git is able to handle packs larger than 4GB too,  
> assuming it
> is compiled using a toolchain with large file support.
>
>
> Nicolas
> -

Re: git packs

[Luke Lu]

On Nov 9, 2007, at 10:00 PM, bob wrote:
> When you say toolchain, are you referring to the compiler
> and associated libraries or are you referring to OS programs
> such as ls, md5, cat, etc or both?
>
> The reason that I ask is that I have been playing different
> scenarios using git 1.5.3.5 under MacOSX 10.4.10 mostly
> all day and every time that
>
> A) 	a file approaches or exceeds 2gig on an 'add', it
> 	results in:
> 	
> 	fatal: Out of memory? mmap failed: Cannot allocate memory
>
>
>
> B) 	the repository size less the .git subdirectory approaches
> 	4gig on a 'fetch' it results in:
>
> 	Resolving 3356 deltas...
> 	fatal: serious inflate inconsistency: -3 (unknown compression method)
> 	fatal: index-pack died with error code 128
> 	fatal: Fetch failure: ../rmwHtmlOld
>
> 	Under B, building the initial repository works fine.
>
> (I added a patch the Linus Torvalds gave out when a previous  
> inflate problem
> was being researched.)  Also, I have been looking in the source
> in particular in builtin-add.c builtin-pack-objects.c and  
> associated headers
> and see int and unsigned long being used a lot, but not any  
> unsigned long
> longs.  I have been testing on my laptop which has a 32-bit Intel  
> Core Duo.
> Also, I have run the same tests on a dual quad-core Intel processor
> which is 64 bit, (but not sure that Apple uses the 64 bits in  
> 10.4.10).  I
> get the same results as above.
>
> The zlib is at the latest revision of 1.2.3 and gcc is at 4.0.1
> which from what I can tell supports large files, because 'off_t' is  
> 8 bytes
> which is the size used for a 'stat' file size.

mmap(2), which git uses by default, is subject to vm limits  
(typically <2GB), regardless of large file support. file `which git`  
will probably tell you that it's a Mach-O executable i386 instead of  
x86_64. In order to get 64 bit binaries on Mactel boxes, you'll need  
the -m64 flag for gcc. I suspect that compiling with NO_MMAP option  
work as well.

__Luke

Re: git packs

[Linus Torvalds]

On Sat, 10 Nov 2007, bob wrote:
> 
> The reason that I ask is that I have been playing different
> scenarios using git 1.5.3.5 under MacOSX 10.4.10 mostly
> all day and every time that
> 
> A) 	a file approaches or exceeds 2gig on an 'add', it
> 	results in:
> 		fatal: Out of memory? mmap failed: Cannot allocate memory

Git wants to handle single files as one single entity, so single big files 
really do end up being very painful. The costs of compressing them and 
generating deltas would probably get prohibitively high *anyway*, but it 
does mean that if you have gigabyte files, you do want a 64-bit VM.

I thought OS X could do 64 bits these days. Maybe not.

Anyway, that explains the "cannot allocate memory". Git simply wants to 
mmap the whole file. You don't have enough VM space for it.

(And if you seriously want to work with multi-gigabyte files, git probbaly 
isn't going to perform wonderfully well, even if it *should* work fine if 
you just have a full 64-bit environment that allows the mmap).

> B) 	the repository size less the .git subdirectory approaches
> 	4gig on a 'fetch' it results in:
> 
> 	Resolving 3356 deltas...
> 	fatal: serious inflate inconsistency: -3 (unknown compression method)

That sounds really broken. I'm not seeing what would cause that, apart 
from some really bad data corruption and/or broken zlib implementation. 
But if the pack-file really is 2GB+ in size, I could imagine some sign 
issues cropping up.

git will generally use "unsigned long" (which is probably just 32-bit on 
your setup), but since git in those circumstances would be limited by the 
size of the VM _anyway_, that's not really much of a limitation (although 
probably broken on the crazy Windows "LLP64" model). But maybe we have 
some place where we use a signed thing, or zlib does, and I could see that 
causing breakage.

But that code-sequence really should never even come *close* to the 31-bit 
limit, as long as the individual objects themselves aren't bigger than the 
available VM space (and git currently assumes "unsigned long" is 
sufficiently big to cover the VM space, which is not technically correct, 
but should be fine on OS X too).

That said, we should use "off_t" in that function. I suspect we have a 
number of people (read: me) who have grown too used to living in a 64-bit 
world..

> I have been testing on my laptop which has a 32-bit Intel Core Duo.

Ok, so you're 32-bit limited even if there is were to be some 64-bit 
support for OS X.

> Also, I have run the same tests on a dual quad-core Intel processor
> which is 64 bit, (but not sure that Apple uses the 64 bits in 10.4.10).  I
> get the same results as above.

I'm pretty sure OS X defaults to a 32-bit environment, but has at least 
*some* 64-bit support. It would definitely need to be enabled explicitly 
(since they made the *insane* decision to move over to Intel laptop chips 
six months before they got 64-bit support! Somebody at Apple is a total 
idiot, and should get fired).

So it would be interesting to hear if a 64-bit build would make a 
difference.

> The zlib is at the latest revision of 1.2.3 and gcc is at 4.0.1
> which from what I can tell supports large files, because 'off_t' is 8 bytes
> which is the size used for a 'stat' file size.

See above: single files are size-limited, but with a large off_t like 
yours, you should be fine. Except we may have screwed up.

> I am just wondering if these size limitations exist for MacOSX
> or maybe I am doing something wrong (which is probably
> the case).

We *have* had issues with broken implementations of "pread()" on some 
systems.  

You could try setting NO_PREAD in the Makefile and compiling with the 
compatibility function.. That's the only thing that comes to mind as being 
worth trying in that area.

And if you have some script to generate the repository (ie you aren't 
using "live data", but are testing the limits of the system), if you can 
make that available, so that people with non-OSX environments can test, 
that would be interesting..

I certainly have some 32-bit environments too (old linux boxes), but I'm 
too lazy to write a test-case, so I was hoping you'd be using some simple 
scripts that I could just test and see if I can see the behaviour you 
describe myself.

That said, I have worked with a 3GB pack-file (one of the KDE trial 
repos). That worked fine. But git does tend to want a *lot* of memory for 
really big repositories, so I suspect that if you actually work with 2GB+ 
pack-files, you'll be wanting a 64-bit environment just because you'll be 
wanting more than 2GB of physical RAM in order to be able to access it 
efficiently.

			Linus

Re: git packs

[Linus Torvalds]

On Fri, 9 Nov 2007, Linus Torvalds wrote:
> 
> That said, I have worked with a 3GB pack-file (one of the KDE trial 
> repos). That worked fine. But git does tend to want a *lot* of memory for 
> really big repositories, so I suspect that if you actually work with 2GB+ 
> pack-files, you'll be wanting a 64-bit environment just because you'll be 
> wanting more than 2GB of physical RAM in order to be able to access it 
> efficiently.

Just double-checked. Yes, sirree. You definitely want 4GB+ if you are 
cloning a 3GB git pack-file. The "git-pack-objects" phase not only is 
going to walk all over the pack-file, it's going to add its own memory 
footprint on top of that just keeping track of all the objects.

So I doubt 2GB+ pack-files are all that practical on 32-bit hosts. At 
least not with the kind of performance behaviour *I* would accept.

(Of course, since git packs things pretty damn well, it would need to be a 
really really big project to be a 2GB+ pack-file, or just contain a lot of 
generally large non-deltable binary data file - one scenario where git 
definitely doesn't work wonderfully well, although I doubt many other 
SCM's do either..)

			Linus

Re: git packs

[Linus Torvalds]

On Fri, 9 Nov 2007, Luke Lu wrote:
> 
> mmap(2), which git uses by default, is subject to vm limits (typically <2GB),
> regardless of large file support. file `which git` will probably tell you that
> it's a Mach-O executable i386 instead of x86_64. In order to get 64 bit
> binaries on Mactel boxes, you'll need the -m64 flag for gcc. I suspect that
> compiling with NO_MMAP option work as well.

Even with NO_MMAP, git will still want to read in source files in their 
entirety (just with regular reads). So you'll still be VM size-limited: 
the mmap() will just be replaced with a malloc+read in order to avoid 
some broken windows mmap() behaviour.

But hearing whether -m64 makes a difference would be interesting. I'm 
hoping OS X is LP64, not some insane half-way thing like Vista is.

		Linus

Re: git packs

[bob]

I will try a few things and see if I can get a script put together
that generates the inflate problem.  The data that I am
using is a backup of my original repository.  So, I can
play all that I want.  But it would be a lot easier if I
could just generate some files using dd or something.

I'll also try the 64-bit compile on my Mac Pro and see
if it works.

My only reason for keeping this directory under git is
that I find git so easy to work with across multiple
machines.  I own 10+ computers and I use git to
provide easy transfers/updates from machine to
machine normally without any issues and a great
deal of reliability.

This directory is a website that I use
internally to keep track of things important to me.
For instance, the one large file is a movie of the
inside of my house before they sheet-rocked it
so that later I would have an easier time finding
things in the walls.  There is some html and php
that I wrote in it which I did want versioned.

Maybe I should just drop back to using two
directories, one of the large files which are
static anyway and a git repo for the html and php.
I was just trying to keep everything in the
repo for simplicity.

No matter which direction that I decide, I will still
try to provide the script.  Thank you both (Luke Lu)
for the feedback.


On Nov 10, 2007, at 1:38 AM, Linus Torvalds wrote:

>
>
> On Sat, 10 Nov 2007, bob wrote:
>>
>> The reason that I ask is that I have been playing different
>> scenarios using git 1.5.3.5 under MacOSX 10.4.10 mostly
>> all day and every time that
>>
>> A) 	a file approaches or exceeds 2gig on an 'add', it
>> 	results in:
>> 		fatal: Out of memory? mmap failed: Cannot allocate memory
>
> Git wants to handle single files as one single entity, so single  
> big files
> really do end up being very painful. The costs of compressing them and
> generating deltas would probably get prohibitively high *anyway*,  
> but it
> does mean that if you have gigabyte files, you do want a 64-bit VM.
>
> I thought OS X could do 64 bits these days. Maybe not.
>
> Anyway, that explains the "cannot allocate memory". Git simply  
> wants to
> mmap the whole file. You don't have enough VM space for it.
>
> (And if you seriously want to work with multi-gigabyte files, git  
> probbaly
> isn't going to perform wonderfully well, even if it *should* work  
> fine if
> you just have a full 64-bit environment that allows the mmap).
>
>> B) 	the repository size less the .git subdirectory approaches
>> 	4gig on a 'fetch' it results in:
>>
>> 	Resolving 3356 deltas...
>> 	fatal: serious inflate inconsistency: -3 (unknown compression  
>> method)
>
> That sounds really broken. I'm not seeing what would cause that, apart
> from some really bad data corruption and/or broken zlib  
> implementation.
> But if the pack-file really is 2GB+ in size, I could imagine some sign
> issues cropping up.
>
> git will generally use "unsigned long" (which is probably just 32- 
> bit on
> your setup), but since git in those circumstances would be limited  
> by the
> size of the VM _anyway_, that's not really much of a limitation  
> (although
> probably broken on the crazy Windows "LLP64" model). But maybe we have
> some place where we use a signed thing, or zlib does, and I could  
> see that
> causing breakage.
>
> But that code-sequence really should never even come *close* to the  
> 31-bit
> limit, as long as the individual objects themselves aren't bigger  
> than the
> available VM space (and git currently assumes "unsigned long" is
> sufficiently big to cover the VM space, which is not technically  
> correct,
> but should be fine on OS X too).
>
> That said, we should use "off_t" in that function. I suspect we have a
> number of people (read: me) who have grown too used to living in a  
> 64-bit
> world..
>
>> I have been testing on my laptop which has a 32-bit Intel Core Duo.
>
> Ok, so you're 32-bit limited even if there is were to be some 64-bit
> support for OS X.
>
>> Also, I have run the same tests on a dual quad-core Intel processor
>> which is 64 bit, (but not sure that Apple uses the 64 bits in  
>> 10.4.10).  I
>> get the same results as above.
>
> I'm pretty sure OS X defaults to a 32-bit environment, but has at  
> least
> *some* 64-bit support. It would definitely need to be enabled  
> explicitly
> (since they made the *insane* decision to move over to Intel laptop  
> chips
> six months before they got 64-bit support! Somebody at Apple is a  
> total
> idiot, and should get fired).
>
> So it would be interesting to hear if a 64-bit build would make a
> difference.
>
>> The zlib is at the latest revision of 1.2.3 and gcc is at 4.0.1
>> which from what I can tell supports large files, because 'off_t'  
>> is 8 bytes
>> which is the size used for a 'stat' file size.
>
> See above: single files are size-limited, but with a large off_t like
> yours, you should be fine. Except we may have screwed up.
>
>> I am just wondering if these size limitations exist for MacOSX
>> or maybe I am doing something wrong (which is probably
>> the case).
>
> We *have* had issues with broken implementations of "pread()" on some
> systems.
>
> You could try setting NO_PREAD in the Makefile and compiling with the
> compatibility function.. That's the only thing that comes to mind  
> as being
> worth trying in that area.
>
> And if you have some script to generate the repository (ie you aren't
> using "live data", but are testing the limits of the system), if  
> you can
> make that available, so that people with non-OSX environments can  
> test,
> that would be interesting..
>
> I certainly have some 32-bit environments too (old linux boxes),  
> but I'm
> too lazy to write a test-case, so I was hoping you'd be using some  
> simple
> scripts that I could just test and see if I can see the behaviour you
> describe myself.
>
> That said, I have worked with a 3GB pack-file (one of the KDE trial
> repos). That worked fine. But git does tend to want a *lot* of  
> memory for
> really big repositories, so I suspect that if you actually work  
> with 2GB+
> pack-files, you'll be wanting a 64-bit environment just because  
> you'll be
> wanting more than 2GB of physical RAM in order to be able to access it
> efficiently.
>
> 			Linus

Re: git packs

[David Brown]

On Fri, Nov 09, 2007 at 10:58:16PM -0800, Linus Torvalds wrote:

>But hearing whether -m64 makes a difference would be interesting. I'm 
>hoping OS X is LP64, not some insane half-way thing like Vista is.

Some casual tests with printf and sizeof makes it look like it is.  At
least sizeof (void *) and sizeof (long) are both 8.

David

Re: git packs

[bob]

I compiled git under MacOSX 10.4.10 withr:

A) -m64 -arch ppc64 on a dual G5
B) -m64 -arch x86_64 on the dual quad-core

In both case, the link phase failed because there
was no 64-bit version of libz, libssl, libiconv and
libcrypto to link with.

I then installed MacOSX 10.5 (Leopard) which
was just released last month on the dual
quad-core machine with -m64 -arch x86_64.
git compiled and linked successfully.  However,
it failed in the "git add ." which was the
second command after "git init".  The message
was fairly cryptic, "Bus error".

I am guessing that the "Bus error" is an Apple
problem and it did produce a crashreport.  So,
I am going to submit it to Apple since it is easily
reproducible.

Anyway, those are the results.

I am now going to split out the jpg(s), pdf(s)
and (.mov) files from the repository  and just
manage them external to the git repository
which will fix my git problem.

I will report again when I get git running on
Leopard.

Re: git packs

[David Brown]

On Sat, Nov 10, 2007 at 12:40:16PM -0500, bob wrote:

> I am guessing that the "Bus error" is an Apple
> problem and it did produce a crashreport.  So,
> I am going to submit it to Apple since it is easily
> reproducible.

The crash report is probably not all that useful to Apple, since it occurs
in a program you have built.  Perhaps you should send the stack trace from
the crash report here and people might have some ideas about what might be
the problem.

David

Re: git packs

[bob]

It is fairly disappointing as far as indicating the problem.  Here is  
the entire report since it was so short.


============ Begin =================
Process:         git [82703]
Path:            git
Identifier:      git
Version:         ??? (???)
Code Type:       X86-64 (Native)
Parent Process:  bash [261]

Date/Time:       2007-11-10 11:23:33.976 -0500
OS Version:      Mac OS X 10.5 (9A581)
Report Version:  6

Exception Type:  EXC_BAD_ACCESS (SIGBUS)
Exception Codes: KERN_PROTECTION_FAILURE at 0x00007fff5fc00000
Crashed Thread:  Unknown

Error Formulating Crash Report:
*** -[NSCFDictionary setObject:forKey:]: attempt to insert nil value  
(key: VMUSignaturePath)
0x9501626b
0x9586009b
0x9501604b
0x9501608a
0x900d65e8
0x00078308
0x00087eb8
0x0008800e
0x000850a2
0x00002d90
0x000093be
0x0000b57c
0x0000b0c9
0x919f8793
0x0000a7d0
0x919cc075
0x919cbf32

Backtrace not available

Unknown thread crashed with X86 Thread State (64-bit):
   rax: 0x000000000000003b  rbx: 0x00000001191c9338  rcx:  
0x0000000000000392  rdx: 0x00000000aff4bc3b
   rdi: 0x00007fff5fc00000  rsi: 0x0000000040000003  rbp:  
0x00007fff5fbff180  rsp: 0x00007fff5fbff160
    r8: 0x0000000000000000   r9: 0x058487f0858487f0  r10:  
0x000000002cb27436  r11: 0x000000000ee2afe3
   r12: 0x00000000181c84c0  r13: 0x00007fff5fbff1a0  r14:  
0x000000010000000c  r15: 0x0000000101000000
   rip: 0x00007fff80543ca5  rfl: 0x0000000000010202  cr2:  
0x00007fff5fc00000

Binary images description not available
============ end =================

Maybe there was another option to specifiy other than "-m64 -arch  
x86_64".


On Nov 10, 2007, at 12:45 PM, David Brown wrote:

> On Sat, Nov 10, 2007 at 12:40:16PM -0500, bob wrote:
>
>> I am guessing that the "Bus error" is an Apple
>> problem and it did produce a crashreport.  So,
>> I am going to submit it to Apple since it is easily
>> reproducible.
>
> The crash report is probably not all that useful to Apple, since it  
> occurs
> in a program you have built.  Perhaps you should send the stack  
> trace from
> the crash report here and people might have some ideas about what  
> might be
> the problem.
>
> David
> -
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: git packs

[Nicolas Pitre]

On Sat, 10 Nov 2007, bob wrote:

> I will try a few things and see if I can get a script put together
> that generates the inflate problem.  The data that I am
> using is a backup of my original repository.  So, I can
> play all that I want.  But it would be a lot easier if I
> could just generate some files using dd or something.

Please see the patch I just posted to the list.  That should fix your 
problem.  I even included a small script to create a repository 
confirming the problem and allowing to test the fix.


Nicolas

Re: git packs

[Derek Fawcus]

On Sat, Nov 10, 2007 at 01:01:46PM -0500, bob wrote:
> It is fairly disappointing as far as indicating the problem.  Here is  
> the entire report since it was so short.

> Error Formulating Crash Report:
> *** -[NSCFDictionary setObject:forKey:]: attempt to insert nil value  
> (key: VMUSignaturePath)

huh?  The above looks like an ObjC invocation.  Last I checked,  git was C.
So why is that in the frame?

DF

Re: git packs

[bob]

Well, there are actually two problems here.  The first is that
crashreporter crashed trying to create the crash report of
the git failure.  The second was the error in git itself.
Hope that helps.

I am submitting an Apple bugreport on the first and
Nicolas Pitre is probably correct in his fix.  When I
was looking at index-pack.c everything was using 32
bit unsigned and signed number where off_t was
64-bit which is what a stat() would return.  My problem
is that I was not familiar enough with git internals to
figure out the solution.  My review was the first time
that I ever looked at git source.

Thanks, Nicolas

On Nov 11, 2007, at 6:09 AM, Derek Fawcus wrote:

> On Sat, Nov 10, 2007 at 01:01:46PM -0500, bob wrote:
>> It is fairly disappointing as far as indicating the problem.  Here is
>> the entire report since it was so short.
>
>> Error Formulating Crash Report:
>> *** -[NSCFDictionary setObject:forKey:]: attempt to insert nil value
>> (key: VMUSignaturePath)
>
> huh?  The above looks like an ObjC invocation.  Last I checked,   
> git was C.
> So why is that in the frame?
>
> DF

Re: git packs

[bob]

I applied the patch and these commands:

cd rmwHtmlOld
rm -fr .git
git init
git config core.compression 0
git add .

I then got the same error as before, "Bus error".  Rats!

Then I modified your script since I do not have seq or
your test-genrandom.  I substituted:

dd count=XX  if=/dev/random of=file_$i

where XX is adjusted to meet dd's requirements.  Also,
I found after searching for a while, that the following
works just like your seq command:

xyzzy="1 2 3 4"
for i in $xyzzy
do
...
done

Your script then ran flawlessly.

I looked through index-pack.c some more, but it is
very hard to figure it out without doing a lot of research
since there doesn't seem to be anything that describes
the layout of a pack.  The link towards the end of the user's
manual doesn't work for me.

The difference between your test and my data is that
instead of having a few large files, I have 11,500 files
of varying sizes.  On average though, the file size is
about 370k.

HTH

On Nov 10, 2007, at 11:35 PM, Nicolas Pitre wrote:

> On Sat, 10 Nov 2007, bob wrote:
>
>> I will try a few things and see if I can get a script put together
>> that generates the inflate problem.  The data that I am
>> using is a backup of my original repository.  So, I can
>> play all that I want.  But it would be a lot easier if I
>> could just generate some files using dd or something.
>
> Please see the patch I just posted to the list.  That should fix your
> problem.  I even included a small script to create a repository
> confirming the problem and allowing to test the fix.
>
>
> Nicolas
> -
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: git packs

[Nicolas Pitre]

On Sun, 11 Nov 2007, bob wrote:

> I applied the patch and these commands:
> 
> cd rmwHtmlOld
> rm -fr .git
> git init
> git config core.compression 0
> git add .

Note that I did "git config core.compression 0" simply to disable 
zlib compression altogether when creating the test repo just so it gets 
created faster.  even then, auto-generating and cloning a 8GB test 
repository isn't particularly quick.

> I then got the same error as before, "Bus error".  Rats!

Do you get that with a 32-bit or 64-bit build of Git?

> Then I modified your script since I do not have seq or
> your test-genrandom.

test-genrandom is built with Git.  It is just not installed anywhere.

> I substituted:
> 
> dd count=XX  if=/dev/random of=file_$i
> 
> where XX is adjusted to meet dd's requirements.  Also,

Again I used test-genrandom instead of /dev/random or /dev/urandom 
simply because the former is much faster.

> I found after searching for a while, that the following
> works just like your seq command:
> 
> xyzzy="1 2 3 4"
> for i in $xyzzy
> do
> ...
> done
> 
> Your script then ran flawlessly.

However 'seq -w 1 2 63' should be replaced  with "01 03 05 07 09 11 13 
15" and so on up to 63, and 'seq -w 2 2 64' is "02 04 06 08 10 12 16" 
and so on.

> I looked through index-pack.c some more, but it is
> very hard to figure it out without doing a lot of research
> since there doesn't seem to be anything that describes
> the layout of a pack.  The link towards the end of the user's
> manual doesn't work for me.

Look at Documentation/technical/pack-format.txt in the Git source tree.

> The difference between your test and my data is that
> instead of having a few large files, I have 11,500 files
> of varying sizes.  On average though, the file size is
> about 370k.

Are you saying that the test repo with big files works for you but not 
your own data set?

Would you please recap what your problem is?

With my one line patch you should not get the "serious inflate 
inconsistency" error anymore.  The bus error must be another issue.


Nicolas

Re: git packs

[Martin Langhoff]

On Nov 11, 2007 6:40 AM, bob <kranki@mac.com> wrote:
> I am now going to split out the jpg(s), pdf(s)
> and (.mov) files from the repository  and just
> manage them external to the git repository
> which will fix my git problem.

For version-less synchronization of large files I'd suggest unison (as
a smarter rsync, it's excellent) or the newfangled tra.

hth,


martin

Re: git packs

[bob]

On Nov 11, 2007, at 11:21 PM, Nicolas Pitre wrote:

> On Sun, 11 Nov 2007, bob wrote:
>
>> I applied the patch and these commands:
>>
>> cd rmwHtmlOld
>> rm -fr .git
>> git init
>> git config core.compression 0
>> git add .
>
> Note that I did "git config core.compression 0" simply to disable
> zlib compression altogether when creating the test repo just so it  
> gets
> created faster.  even then, auto-generating and cloning a 8GB test
> repository isn't particularly quick.
	I wasn't sure why you used that, but figured that I should put it
	in, because you did.  And you are very correct that these tests
	can take a very long time.
>
>> I then got the same error as before, "Bus error".  Rats!
>
> Do you get that with a 32-bit or 64-bit build of Git?
	On that machine, I changed the config.mak to generate nothing
	but a 64-bit build.  And activity monitor shows it as 64-bit
	when I am running it. Also, off_t is 64 bit on regular
	MacOSX and as a 64-bit program supported by the
	latest MacOSX.
>
>> Then I modified your script since I do not have seq or
>> your test-genrandom.
>
> test-genrandom is built with Git.  It is just not installed anywhere.
>
>> I substituted:
>>
>> dd count=XX  if=/dev/random of=file_$i
>>
>> where XX is adjusted to meet dd's requirements.  Also,
>
> Again I used test-genrandom instead of /dev/random or /dev/urandom
> simply because the former is much faster.
	I didn't know that it existed, since I am not that familiar with the
	internals and source directory for git.
>
>> I found after searching for a while, that the following
>> works just like your seq command:
>>
>> xyzzy="1 2 3 4"
>> for i in $xyzzy
>> do
>> ...
>> done
>>
>> Your script then ran flawlessly.
>
> However 'seq -w 1 2 63' should be replaced  with "01 03 05 07 09 11 13
> 15" and so on up to 63, and 'seq -w 2 2 64' is "02 04 06 08 10 12 16"
> and so on.
	That I missed but the files added up to quite a bit over 8gigs,
	because my count calculation was off.  So, I thought that it
	was adequate for the testing.  Based on above, it was
	quite a few files less as well.
>
>> I looked through index-pack.c some more, but it is
>> very hard to figure it out without doing a lot of research
>> since there doesn't seem to be anything that describes
>> the layout of a pack.  The link towards the end of the user's
>> manual doesn't work for me.
>
> Look at Documentation/technical/pack-format.txt in the Git source  
> tree.
Thanks.
>
>> The difference between your test and my data is that
>> instead of having a few large files, I have 11,500 files
>> of varying sizes.  On average though, the file size is
>> about 370k.
>
> Are you saying that the test repo with big files works for you but not
> your own data set?
	That is correct, but you are probably correct that this is a separate
	problem, because it is only occurring on the 64-bit side.  The 32-bit
	side is running without error.
>
> Would you please recap what your problem is?
	With more testing last night, I believe that the problem on the 32-bit
	side has been fixed.  But the 'Bus error' on 64-bit side persists.  I
	agree that I was combining it and probably shouldn't have.
> With my one line patch you should not get the "serious inflate
> inconsistency" error anymore.  The bus error must be another issue.
I no longer do on the 32-bit side.
>
>
> Nicolas



I am going to wait on the 64-bit problem until MacOSX 10.5 (Leopard)
settles down a bit.  It has a few bugs in it.  I will submit the  
crashreporter
issue to Apple. Currently, I don't have the time to commit a lot to  
git right
now.

As Apple comes out with new subreleases, I will retry my data.
If it doesn't get resolved after a subrelease or two, then I dig  
deeper and
hopefully have more time to devote to it at that point.

Thank you, I do appreciate your help and your patience.