http protocol, cloning git.git, fails (too many open files)

http protocol, cloning git.git, fails (too many open files)

[Randal L. Schwartz]

    $ mkdir foo
    $ cd foo
    $ cg-clone http://www.kernel.org/pub/scm/git/git.git git.new
    Fetching head...
    Fetching objects...
    progress: 1 objects, 191 bytes
progress: 2 objects, 519 bytes
progress: 3 objects, 5851 bytes
progress: 4 objects, 7671 bytes
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    progress: 5 objects, 8274 bytes
progress: 6 objects, 12989 bytes
progress: 7 objects, 16698 bytes
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    progress: 8 objects, 28639 bytes
progress: 9 objects, 37700 bytes
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    progress: 10 objects, 38341 bytes
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    progress: 11 objects, 39097 bytes
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    progress: 12 objects, 43396 bytes
progress: 13 objects, 44694 bytes
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    progress: 14 objects, 45036 bytes
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    progress: 15 objects, 45511 bytes
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    progress: 16 objects, 47485 bytes
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    progress: 17 objects, 51016 bytes
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    progress: 18 objects, 51368 bytes
progress: 19 objects, 54438 bytes
progress: 20 objects, 55478 bytes
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    progress: 21 objects, 63175 bytes
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    progress: 22 objects, 64613 bytes
progress: 23 objects, 66548 bytes
progress: 24 objects, 68347 bytes
progress: 25 objects, 70471 bytes
progress: 26 objects, 72662 bytes
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    progress: 27 objects, 75536 bytes
progress: 28 objects, 82929 bytes
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    progress: 29 objects, 85551 bytes
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    progress: 30 objects, 85834 bytes
progress: 31 objects, 86902 bytes
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    progress: 32 objects, 87980 bytes
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    progress: 33 objects, 88242 bytes
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    progress: 34 objects, 90131 bytes
progress: 35 objects, 92595 bytes
progress: 36 objects, 99562 bytes
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    progress: 37 objects, 99997 bytes
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    progress: 38 objects, 101100 bytes
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    progress: 39 objects, 101673 bytes
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    progress: 40 objects, 102357 bytes
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    progress: 41 objects, 104042 bytes
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    progress: 42 objects, 104924 bytes
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    progress: 43 objects, 105887 bytes
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    progress: 44 objects, 112614 bytes
progress: 45 objects, 114909 bytes
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    progress: 46 objects, 125043 bytes
progress: 47 objects, 132911 bytes
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    progress: 48 objects, 135339 bytes
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    progress: 49 objects, 140171 bytes
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    progress: 50 objects, 142244 bytes
progress: 51 objects, 142320 bytes
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    progress: 52 objects, 144201 bytes
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    progress: 53 objects, 147742 bytes
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    progress: 54 objects, 152880 bytes
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    progress: 55 objects, 154845 bytes
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    progress: 56 objects, 160035 bytes
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    progress: 57 objects, 162525 bytes
progress: 58 objects, 172189 bytes
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    progress: 59 objects, 175374 bytes
progress: 60 objects, 179154 bytes
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    progress: 61 objects, 183166 bytes
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    progress: 62 objects, 184979 bytes
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    progress: 63 objects, 185402 bytes
progress: 64 objects, 185658 bytes
progress: 65 objects, 186589 bytes
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
    error: Couldn't create temporary file .git/objects/be/63132804252e9a2f94ce1166ef9fa319a845fd.temp for .git/objects/be/63132804252e9a2f94ce1166ef9fa319a845fd: Too many open files

    error: Couldn't create temporary file .git/objects/df/a87fe1191862b2c650e55fdd727802ef00cf34.temp for .git/objects/df/a87fe1191862b2c650e55fdd727802ef00cf34: Too many open files

    error: Couldn't create temporary file .git/objects/51/cf86a5843acc3b6bc3d8c4be9fec0fdd0a0df5.temp for .git/objects/51/cf86a5843acc3b6bc3d8c4be9fec0fdd0a0df5: Too many open files

    error: Couldn't create temporary file .git/objects/c9/9db9dd79315dff4ac19c79b35275cd02397e60.temp for .git/objects/c9/9db9dd79315dff4ac19c79b35275cd02397e60: Too many open files

    error: Couldn't create temporary file .git/objects/2b/2c6b77afd53870423d4bad9191ed669bc55660.temp for .git/objects/2b/2c6b77afd53870423d4bad9191ed669bc55660: Too many open files

    error: Couldn't create temporary file .git/objects/dc/c168dbbfaf5c72d20cbbfba92c817f40467238.temp for .git/objects/dc/c168dbbfaf5c72d20cbbfba92c817f40467238: Too many open files

    error: Couldn't create temporary file .git/objects/07/203659d0dae619f78773d3caea6b58907a1ba5.temp for .git/objects/07/203659d0dae619f78773d3caea6b58907a1ba5: Too many open files

    progress: 66 objects, 187452 bytes

And at this point, it hung.  I had to ^C.

-- 
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!

Re: http protocol, cloning git.git, fails (too many open files)

[Junio C Hamano]

merlyn@stonehenge.com (Randal L. Schwartz) writes:

>     $ mkdir foo
>     $ cd foo
>     $ cg-clone http://www.kernel.org/pub/scm/git/git.git git.new
>     Fetching head...
>     Fetching objects...
>     progress: 1 objects, 191 bytes
> progress: 2 objects, 519 bytes
> progress: 3 objects, 5851 bytes
> progress: 4 objects, 7671 bytes
>...
>
>     progress: 66 objects, 187452 bytes
>
> And at this point, it hung.  I had to ^C.

This sounds like the problem the patch Pasky posted last night
addresses.

However, one thing puzzles me.  I just tried to reproduce it by
doing this:

	$ rm -fr git-http
        $ ulimit -n 16
        $ git clone http://www.kernel.org/pub/scm/git/git.git git-http

and it did not fail on my Linux box.  This is from the tip of
the master tree (essentially the same as 0.99.9h aka 1.0rc1),
built without Pasky's patch.  Maybe the number of in-transit
requests in your case is much higher with your connection while
waiting for a packfile to arrive than my setup.

Could you please try with the patch Nick posted (which covers
less) and then with Pasky's patch?

Re: http protocol, cloning git.git, fails (too many open files)

[walt]

On Sat, 2005-11-12 at 15:21 -0800, Junio C Hamano wrote:
[...]
> However, one thing puzzles me.  I just tried to reproduce it by
> doing this:
> 
> 	$ rm -fr git-http
>         $ ulimit -n 16
>         $ git clone http://www.kernel.org/pub/scm/git/git.git git-http
> 
> and it did not fail on my Linux box...

I've never seen it on Linux, but I do see it on NetBSD every time
with ulimit -n 64 (the default):

> Could you please try with the patch Nick posted (which covers
> less) and then with Pasky's patch?

Neither patch fixes the 'too many open files' error, but Pasky's
patch produces this debugging output:

[...]
progress: 46 objects, 105660 bytes
error: Couldn't create temporary
file .git/objects/ee/baf3aaffa72996ac8bdbc8e9498814cd99f506.temp
for .git/objects/ee/baf3aaffa72996ac8bdbc8e9498814cd99f506: Too many
open files

progress: 48 objects, 108247 bytes
error: fd leakage in release: 9
error: fd leakage in release: 11
error: fd leakage in release: 3
error: fd leakage in release: 13
error: fd leakage in release: 6
error: fd leakage in release: 12
error: fd leakage in release: 14
error: fd leakage in release: 8
error: fd leakage in release: 16
error: fd leakage in release: 15
error: fd leakage in release: 10
error: fd leakage in release: 5
error: fd leakage in release: 21
error: fd leakage in release: 17
error: fd leakage in release: 22
error: fd leakage in release: 19
error: fd leakage in release: 20
error: fd leakage in release: 25
error: fd leakage in release: 26
error: Could not read e8700aa756a3296ede4403db349dc64586935139
Waiting for
http://www.kernel.org/pub/scm/git/git.git/objects/dd/d5823df728bf213425533dcd2e47d61cbccf9e
cg-fetch: objects fetch failed
cg-clone: fetch failed

Re: http protocol, cloning git.git, fails (too many open files)

[wa1ter]

On Sun, 13 Nov 2005, walt wrote:

> On Sat, 2005-11-12 at 15:21 -0800, Junio C Hamano wrote:
> [...]
> > However, one thing puzzles me.  I just tried to reproduce it by
> > doing this:
> >
> > 	$ rm -fr git-http
> >         $ ulimit -n 16
> >         $ git clone http://www.kernel.org/pub/scm/git/git.git git-http
> >
> > and it did not fail on my Linux box...

> I've never seen it on Linux...

Aha!  I do indeed see it on linux with ulimit -n 64 but *only* when I
do the clone with cg-clone.  When I use 'git clone' it works just fine.

So, what does cogito do differently that accounts for more open files?

Re: http protocol, cloning git.git, fails (too many open files)

[Petr Baudis]

Dear diary, on Mon, Nov 14, 2005 at 01:42:33AM CET, I got a letter
where wa1ter@myrealbox.com said that...
> 
> 
> On Sun, 13 Nov 2005, walt wrote:
> 
> > On Sat, 2005-11-12 at 15:21 -0800, Junio C Hamano wrote:
> > [...]
> > > However, one thing puzzles me.  I just tried to reproduce it by
> > > doing this:
> > >
> > > 	$ rm -fr git-http
> > >         $ ulimit -n 16
> > >         $ git clone http://www.kernel.org/pub/scm/git/git.git git-http
> > >
> > > and it did not fail on my Linux box...
> 
> > I've never seen it on Linux...
> 
> Aha!  I do indeed see it on linux with ulimit -n 64 but *only* when I
> do the clone with cg-clone.  When I use 'git clone' it works just fine.
> 
> So, what does cogito do differently that accounts for more open files?

Cogito uses the same logic and underlying GIT fetching commands for
cloning as well as for fetching, while git-clone uses some custom logic
for cloning ("dumb http clone"). (In fact, is there a point in carrying
it further now that git-http-fetch got as smart as it got? You'll have
less code, and more importantly see bugs in the fetchers otherwise only
I see with Cogito. ;-)

-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
VI has two modes: the one in which it beeps and the one in which
it doesn't.

Re: http protocol, cloning git.git, fails (too many open files)

[Junio C Hamano]

Petr Baudis <pasky@suse.cz> writes:

> Dear diary, on Mon, Nov 14, 2005 at 01:42:33AM CET, I got a letter
> where wa1ter@myrealbox.com said that...
>> 
>> So, what does cogito do differently that accounts for more open files?
>
> Cogito uses the same logic and underlying GIT fetching commands for
> cloning as well as for fetching,...

Ah, you are absolutely right.  I've forgotten about that "dumb
http support" in git-clone.sh.  That was done back when packs
were introduced but http-fetch was not told about them as a
stop-gap measure.

No wonder we do not see the same problem from git-clone.  Thanks
for reminding me.

Re: http protocol, cloning git.git, fails (too many open files)

[wa1ter]

On Mon, 14 Nov 2005, Petr Baudis wrote:

> Date: Mon, 14 Nov 2005 02:09:40 +0100
> From: Petr Baudis <pasky@suse.cz>
> Cc: git@vger.kernel.org
> Newsgroups: gmane.comp.version-control.git
> Subject: Re: http protocol, cloning git.git, fails (too many open files)
>
> Dear diary, on Mon, Nov 14, 2005 at 01:42:33AM CET, I got a letter
> where wa1ter@myrealbox.com said that...

> > So, what does cogito do differently...

> Cogito uses the same logic and underlying GIT fetching commands for
> cloning as well as for fetching, while git-clone uses some custom logic
> for cloning ("dumb http clone")...

BTW, I just noticed one big difference between git clone and cg-clone:

git clone stores the URL in .git/remotes/origin while cg-clone puts
it in .git/branches/origin.  When I do a cg-update after a git clone
then cg-update can't find the URL.