* http protocol, cloning git.git, fails (too many open files)
@ 2005-11-12 22:25 Randal L. Schwartz
2005-11-12 23:21 ` Junio C Hamano
0 siblings, 1 reply; 7+ messages in thread
From: Randal L. Schwartz @ 2005-11-12 22:25 UTC (permalink / raw)
To: git
$ mkdir foo
$ cd foo
$ cg-clone http://www.kernel.org/pub/scm/git/git.git git.new
Fetching head...
Fetching objects...
progress: 1 objects, 191 bytes
progress: 2 objects, 519 bytes
progress: 3 objects, 5851 bytes
progress: 4 objects, 7671 bytes
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
progress: 5 objects, 8274 bytes
progress: 6 objects, 12989 bytes
progress: 7 objects, 16698 bytes
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
progress: 8 objects, 28639 bytes
progress: 9 objects, 37700 bytes
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
progress: 10 objects, 38341 bytes
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
progress: 11 objects, 39097 bytes
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
progress: 12 objects, 43396 bytes
progress: 13 objects, 44694 bytes
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
progress: 14 objects, 45036 bytes
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
progress: 15 objects, 45511 bytes
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
progress: 16 objects, 47485 bytes
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
progress: 17 objects, 51016 bytes
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
progress: 18 objects, 51368 bytes
progress: 19 objects, 54438 bytes
progress: 20 objects, 55478 bytes
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
progress: 21 objects, 63175 bytes
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
progress: 22 objects, 64613 bytes
progress: 23 objects, 66548 bytes
progress: 24 objects, 68347 bytes
progress: 25 objects, 70471 bytes
progress: 26 objects, 72662 bytes
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
progress: 27 objects, 75536 bytes
progress: 28 objects, 82929 bytes
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
progress: 29 objects, 85551 bytes
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
progress: 30 objects, 85834 bytes
progress: 31 objects, 86902 bytes
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
progress: 32 objects, 87980 bytes
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
progress: 33 objects, 88242 bytes
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
progress: 34 objects, 90131 bytes
progress: 35 objects, 92595 bytes
progress: 36 objects, 99562 bytes
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
progress: 37 objects, 99997 bytes
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
progress: 38 objects, 101100 bytes
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
progress: 39 objects, 101673 bytes
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
progress: 40 objects, 102357 bytes
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
progress: 41 objects, 104042 bytes
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
progress: 42 objects, 104924 bytes
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
progress: 43 objects, 105887 bytes
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
progress: 44 objects, 112614 bytes
progress: 45 objects, 114909 bytes
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
progress: 46 objects, 125043 bytes
progress: 47 objects, 132911 bytes
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
progress: 48 objects, 135339 bytes
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
progress: 49 objects, 140171 bytes
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
progress: 50 objects, 142244 bytes
progress: 51 objects, 142320 bytes
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
progress: 52 objects, 144201 bytes
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
progress: 53 objects, 147742 bytes
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
progress: 54 objects, 152880 bytes
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
progress: 55 objects, 154845 bytes
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
progress: 56 objects, 160035 bytes
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
progress: 57 objects, 162525 bytes
progress: 58 objects, 172189 bytes
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
progress: 59 objects, 175374 bytes
progress: 60 objects, 179154 bytes
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
progress: 61 objects, 183166 bytes
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
progress: 62 objects, 184979 bytes
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
progress: 63 objects, 185402 bytes
progress: 64 objects, 185658 bytes
progress: 65 objects, 186589 bytes
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
error: Couldn't create temporary file .git/objects/be/63132804252e9a2f94ce1166ef9fa319a845fd.temp for .git/objects/be/63132804252e9a2f94ce1166ef9fa319a845fd: Too many open files
error: Couldn't create temporary file .git/objects/df/a87fe1191862b2c650e55fdd727802ef00cf34.temp for .git/objects/df/a87fe1191862b2c650e55fdd727802ef00cf34: Too many open files
error: Couldn't create temporary file .git/objects/51/cf86a5843acc3b6bc3d8c4be9fec0fdd0a0df5.temp for .git/objects/51/cf86a5843acc3b6bc3d8c4be9fec0fdd0a0df5: Too many open files
error: Couldn't create temporary file .git/objects/c9/9db9dd79315dff4ac19c79b35275cd02397e60.temp for .git/objects/c9/9db9dd79315dff4ac19c79b35275cd02397e60: Too many open files
error: Couldn't create temporary file .git/objects/2b/2c6b77afd53870423d4bad9191ed669bc55660.temp for .git/objects/2b/2c6b77afd53870423d4bad9191ed669bc55660: Too many open files
error: Couldn't create temporary file .git/objects/dc/c168dbbfaf5c72d20cbbfba92c817f40467238.temp for .git/objects/dc/c168dbbfaf5c72d20cbbfba92c817f40467238: Too many open files
error: Couldn't create temporary file .git/objects/07/203659d0dae619f78773d3caea6b58907a1ba5.temp for .git/objects/07/203659d0dae619f78773d3caea6b58907a1ba5: Too many open files
progress: 66 objects, 187452 bytes
And at this point, it hung. I had to ^C.
--
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: http protocol, cloning git.git, fails (too many open files)
2005-11-12 22:25 http protocol, cloning git.git, fails (too many open files) Randal L. Schwartz
@ 2005-11-12 23:21 ` Junio C Hamano
2005-11-14 0:01 ` walt
0 siblings, 1 reply; 7+ messages in thread
From: Junio C Hamano @ 2005-11-12 23:21 UTC (permalink / raw)
To: Randal L. Schwartz; +Cc: git
merlyn@stonehenge.com (Randal L. Schwartz) writes:
> $ mkdir foo
> $ cd foo
> $ cg-clone http://www.kernel.org/pub/scm/git/git.git git.new
> Fetching head...
> Fetching objects...
> progress: 1 objects, 191 bytes
> progress: 2 objects, 519 bytes
> progress: 3 objects, 5851 bytes
> progress: 4 objects, 7671 bytes
>...
>
> progress: 66 objects, 187452 bytes
>
> And at this point, it hung. I had to ^C.
This sounds like the problem the patch Pasky posted last night
addresses.
However, one thing puzzles me. I just tried to reproduce it by
doing this:
$ rm -fr git-http
$ ulimit -n 16
$ git clone http://www.kernel.org/pub/scm/git/git.git git-http
and it did not fail on my Linux box. This is from the tip of
the master tree (essentially the same as 0.99.9h aka 1.0rc1),
built without Pasky's patch. Maybe the number of in-transit
requests in your case is much higher with your connection while
waiting for a packfile to arrive than my setup.
Could you please try with the patch Nick posted (which covers
less) and then with Pasky's patch?
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: http protocol, cloning git.git, fails (too many open files)
2005-11-12 23:21 ` Junio C Hamano
@ 2005-11-14 0:01 ` walt
2005-11-14 0:42 ` wa1ter
0 siblings, 1 reply; 7+ messages in thread
From: walt @ 2005-11-14 0:01 UTC (permalink / raw)
To: git
On Sat, 2005-11-12 at 15:21 -0800, Junio C Hamano wrote:
[...]
> However, one thing puzzles me. I just tried to reproduce it by
> doing this:
>
> $ rm -fr git-http
> $ ulimit -n 16
> $ git clone http://www.kernel.org/pub/scm/git/git.git git-http
>
> and it did not fail on my Linux box...
I've never seen it on Linux, but I do see it on NetBSD every time
with ulimit -n 64 (the default):
> Could you please try with the patch Nick posted (which covers
> less) and then with Pasky's patch?
Neither patch fixes the 'too many open files' error, but Pasky's
patch produces this debugging output:
[...]
progress: 46 objects, 105660 bytes
error: Couldn't create temporary
file .git/objects/ee/baf3aaffa72996ac8bdbc8e9498814cd99f506.temp
for .git/objects/ee/baf3aaffa72996ac8bdbc8e9498814cd99f506: Too many
open files
progress: 48 objects, 108247 bytes
error: fd leakage in release: 9
error: fd leakage in release: 11
error: fd leakage in release: 3
error: fd leakage in release: 13
error: fd leakage in release: 6
error: fd leakage in release: 12
error: fd leakage in release: 14
error: fd leakage in release: 8
error: fd leakage in release: 16
error: fd leakage in release: 15
error: fd leakage in release: 10
error: fd leakage in release: 5
error: fd leakage in release: 21
error: fd leakage in release: 17
error: fd leakage in release: 22
error: fd leakage in release: 19
error: fd leakage in release: 20
error: fd leakage in release: 25
error: fd leakage in release: 26
error: Could not read e8700aa756a3296ede4403db349dc64586935139
Waiting for
http://www.kernel.org/pub/scm/git/git.git/objects/dd/d5823df728bf213425533dcd2e47d61cbccf9e
cg-fetch: objects fetch failed
cg-clone: fetch failed
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: http protocol, cloning git.git, fails (too many open files)
2005-11-14 0:01 ` walt
@ 2005-11-14 0:42 ` wa1ter
2005-11-14 1:09 ` Petr Baudis
0 siblings, 1 reply; 7+ messages in thread
From: wa1ter @ 2005-11-14 0:42 UTC (permalink / raw)
To: git
On Sun, 13 Nov 2005, walt wrote:
> On Sat, 2005-11-12 at 15:21 -0800, Junio C Hamano wrote:
> [...]
> > However, one thing puzzles me. I just tried to reproduce it by
> > doing this:
> >
> > $ rm -fr git-http
> > $ ulimit -n 16
> > $ git clone http://www.kernel.org/pub/scm/git/git.git git-http
> >
> > and it did not fail on my Linux box...
> I've never seen it on Linux...
Aha! I do indeed see it on linux with ulimit -n 64 but *only* when I
do the clone with cg-clone. When I use 'git clone' it works just fine.
So, what does cogito do differently that accounts for more open files?
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: http protocol, cloning git.git, fails (too many open files)
2005-11-14 0:42 ` wa1ter
@ 2005-11-14 1:09 ` Petr Baudis
2005-11-14 1:52 ` Junio C Hamano
2005-11-14 13:09 ` wa1ter
0 siblings, 2 replies; 7+ messages in thread
From: Petr Baudis @ 2005-11-14 1:09 UTC (permalink / raw)
To: wa1ter; +Cc: git
Dear diary, on Mon, Nov 14, 2005 at 01:42:33AM CET, I got a letter
where wa1ter@myrealbox.com said that...
>
>
> On Sun, 13 Nov 2005, walt wrote:
>
> > On Sat, 2005-11-12 at 15:21 -0800, Junio C Hamano wrote:
> > [...]
> > > However, one thing puzzles me. I just tried to reproduce it by
> > > doing this:
> > >
> > > $ rm -fr git-http
> > > $ ulimit -n 16
> > > $ git clone http://www.kernel.org/pub/scm/git/git.git git-http
> > >
> > > and it did not fail on my Linux box...
>
> > I've never seen it on Linux...
>
> Aha! I do indeed see it on linux with ulimit -n 64 but *only* when I
> do the clone with cg-clone. When I use 'git clone' it works just fine.
>
> So, what does cogito do differently that accounts for more open files?
Cogito uses the same logic and underlying GIT fetching commands for
cloning as well as for fetching, while git-clone uses some custom logic
for cloning ("dumb http clone"). (In fact, is there a point in carrying
it further now that git-http-fetch got as smart as it got? You'll have
less code, and more importantly see bugs in the fetchers otherwise only
I see with Cogito. ;-)
--
Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
VI has two modes: the one in which it beeps and the one in which
it doesn't.
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: http protocol, cloning git.git, fails (too many open files)
2005-11-14 1:09 ` Petr Baudis
@ 2005-11-14 1:52 ` Junio C Hamano
2005-11-14 13:09 ` wa1ter
1 sibling, 0 replies; 7+ messages in thread
From: Junio C Hamano @ 2005-11-14 1:52 UTC (permalink / raw)
To: Petr Baudis; +Cc: git
Petr Baudis <pasky@suse.cz> writes:
> Dear diary, on Mon, Nov 14, 2005 at 01:42:33AM CET, I got a letter
> where wa1ter@myrealbox.com said that...
>>
>> So, what does cogito do differently that accounts for more open files?
>
> Cogito uses the same logic and underlying GIT fetching commands for
> cloning as well as for fetching,...
Ah, you are absolutely right. I've forgotten about that "dumb
http support" in git-clone.sh. That was done back when packs
were introduced but http-fetch was not told about them as a
stop-gap measure.
No wonder we do not see the same problem from git-clone. Thanks
for reminding me.
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: http protocol, cloning git.git, fails (too many open files)
2005-11-14 1:09 ` Petr Baudis
2005-11-14 1:52 ` Junio C Hamano
@ 2005-11-14 13:09 ` wa1ter
1 sibling, 0 replies; 7+ messages in thread
From: wa1ter @ 2005-11-14 13:09 UTC (permalink / raw)
To: git
On Mon, 14 Nov 2005, Petr Baudis wrote:
> Date: Mon, 14 Nov 2005 02:09:40 +0100
> From: Petr Baudis <pasky@suse.cz>
> Cc: git@vger.kernel.org
> Newsgroups: gmane.comp.version-control.git
> Subject: Re: http protocol, cloning git.git, fails (too many open files)
>
> Dear diary, on Mon, Nov 14, 2005 at 01:42:33AM CET, I got a letter
> where wa1ter@myrealbox.com said that...
> > So, what does cogito do differently...
> Cogito uses the same logic and underlying GIT fetching commands for
> cloning as well as for fetching, while git-clone uses some custom logic
> for cloning ("dumb http clone")...
BTW, I just noticed one big difference between git clone and cg-clone:
git clone stores the URL in .git/remotes/origin while cg-clone puts
it in .git/branches/origin. When I do a cg-update after a git clone
then cg-update can't find the URL.
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2005-11-14 13:15 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2005-11-12 22:25 http protocol, cloning git.git, fails (too many open files) Randal L. Schwartz
2005-11-12 23:21 ` Junio C Hamano
2005-11-14 0:01 ` walt
2005-11-14 0:42 ` wa1ter
2005-11-14 1:09 ` Petr Baudis
2005-11-14 1:52 ` Junio C Hamano
2005-11-14 13:09 ` wa1ter
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).