Problem with git-daemon and mmap.

Problem with git-daemon and mmap.

[Tim Ansell]

Hello,

I am trying to use git-daemon on a Debian stable machine. We converted
our repositories from darcs to git using tailor (which seemed to work
well). Most of the repositories work fine, however our web repository
which is quite large (about ~126Mb) is causing problems when trying to
clone via git-daemon.

The repository is public, you can find our gitweb here
 http://git.thousandparsec.net/gitweb/gitweb.cgi?p=web.git;a=summary

You can clone using the following URLs,
 http://git.thousandparsec.net/git/web.git
 git://git.thousandparsec.net/git/web.git
 git+ssh://<username>@git.thousandparsec.net/git/web.git

I can clone the repository fine locally (using both cg-clone and
git-clone) and with both git+ssh:// and http://

When trying to clone via git-daemon the git-daemon process dies with an
the following error (in the git-daemon logs - timestamps trimmed to make
the log more readable),

[30957] Connection from 203.122.246.117:20108
[30957] Extended attributes (29 bytes) exist
<host=git.thousandparsec.net>
[30957] Request upload-pack for '/git/web.git'
fatal: Out of memory? mmap failed: Bad file descriptor
error: git-upload-pack: git-rev-list died with error.
fatal: git-upload-pack: aborting due to possible repository corruption
on the remote side.
[30957] Disconnected (with error)

On the client side you see,
<snip>
remote: Generating pack...
remote: aborting due to possible repository corruption on the remote
side.
fatal: early EOF
fatal: index-pack died with error code 128
<snip>

After chatting with people on #git (on freenode) I was able to produce a
strace for this, you can find a copy of this strace log here,
http://www.thousandparsec.net/~tim/strace.log

The machine as only 512mb of Ram. It isn't running much else apart from
a few websites served via apache. Here is some more (hopefully useful)
information about the machine,
verbal:~# cat /proc/meminfo 
MemTotal:       500536 kB
MemFree:         20556 kB
<snip>
SwapTotal:     1542232 kB
SwapFree:      1542200 kB

verbal:~# dpkg --list | grep git
ii  cogito                           0.18.2-1
version control system
ii  git-core                         1.5.1.1-1
fast, scalable, distributed revision control
ii  git-daemon-run                   1.5.1.1-1
fast, scalable, distributed revision control
ii  gitweb                           1.5.1.1-1
fast, scalable, distributed revision control

verbal:~# git --version
git version 1.5.1.1

Hopefully this gives you enough information to track down this bug. I'm
happy to provide more information if needed. I'll also be on #git, but
I'm in Australia so you might have a hard time catching me.

 
Thank you for your time.

Tim Ansell
--
http://blog.mithis.net/ - Mithro Rants about Stuff

Re: Problem with git-daemon and mmap.

[Gerrit Pape]

On Wed, Apr 25, 2007 at 04:21:15PM +0930, Tim Ansell wrote:
> I am trying to use git-daemon on a Debian stable machine. We converted
> our repositories from darcs to git using tailor (which seemed to work
> well). Most of the repositories work fine, however our web repository
> which is quite large (about ~126Mb) is causing problems when trying to
> clone via git-daemon.

> The machine as only 512mb of Ram. It isn't running much else apart from
> a few websites served via apache. Here is some more (hopefully useful)

> ii  git-daemon-run                   1.5.1.1-1
> fast, scalable, distributed revision control

> Hopefully this gives you enough information to track down this bug. I'm
> happy to provide more information if needed. I'll also be on #git, but
> I'm in Australia so you might have a hard time catching me.

In the debian package version 1.4.4.4, git-daemon's memory by default is
limited to 64MB.  I changed that with 1.5.0, but maybe you chose to not
accept the changes to the run script when upgrading to 1.5.5.1-1.

Check /var/service/git-daemon/run, if it has 'exec chpst -m64000000
git-daemon ...'.  If so, remove 'chpst -m64000000', and restart the
git-daemon service through 'sv restart git-daemon'.

HTH, Gerrit.

Re: Problem with git-daemon and mmap.

[Shawn O. Pearce]

Tim Ansell <mithro@mithis.com> wrote:
> [30957] Request upload-pack for '/git/web.git'
> fatal: Out of memory? mmap failed: Bad file descriptor
> error: git-upload-pack: git-rev-list died with error.
> fatal: git-upload-pack: aborting due to possible repository corruption
> on the remote side.

Owwww....  I don't know what your bug is, but your bug is showing my
own bug.  Filtering the strace to the rev-list child shows this gem:

20769 open("./objects/pack/pack-1931b39bd648fd595248e234f43717569ec5c354.pack", O_RDONLY|O_LARGEFILE <unfinished ...>
20769 <... open resumed> )              = 5

OK, so the packfile is fd 5...

20769 mmap2(NULL, 33554432, PROT_READ, MAP_PRIVATE, 5, 0 <unfinished ...>
20769 <... mmap2 resumed> )             = 0xb5e2d000

and we mapped one 32 MiB window from it at position 0...

20769 mmap2(NULL, 31020635, PROT_READ, MAP_PRIVATE, 5, 0x6000 <unfinished ...>
20769 <... mmap2 resumed> )             = -1 ENOMEM (Cannot allocate memory)

And we asked for another window further into the file.  But got denied.

Now where are we in the code?  We're down inside use_pack(), after
we have called unuse_one_window() enough times to make sure we stay
within our allowed maximum window size.  Since we didn't unmap the
prior window at 0xb5e2d000 we aren't exceeding the current limit
(which probably the defaults).

But we're actually down inside xmmap()...

So we release the window we do have (by calling release_pack_memory),
assuming memory pressure...

20769 munmap(0xb5e2d000, 33554432 <unfinished ...>
20769 <... munmap resumed> )            = 0
20769 close(5 <unfinished ...>
20769 <... close resumed> )             = 0

And that was the last window in this packfile.  So we closed it.
Way to go us.  Our xmmap does not expect release_pack_memory
to close the fd its about to map...

20769 mmap2(NULL, 31020635, PROT_READ, MAP_PRIVATE, 5, 0x6000 <unfinished ...>
20769 <... mmap2 resumed> )             = -1 EBADF (Bad file descriptor)

And so the Linux kernel happily tells us f' off.

20769 write(2, "fatal: ", 7 <unfinished ...>
20769 <... write resumed> )             = 7
20769 write(2, "Out of memory? mmap failed: Bad "..., 47 <unfinished ...>
20769 <... write resumed> )             = 47

And we report the bad file descriptor error, and not the ENOMEM.


I'm not really sure why you blew out the memory and couldn't mmap
a second window here; you really should have been able to given
that there was only one 32 MiB window mapped, and this was fairly
earily in the process' life.  You can try tuning the packing code
down to a smaller window size (see core.packedGitWindowSize and
core.packedGitLimit).

But that strace above shows a secondary bug when we get into this
sort of situation.  Not pretty.

-- 
Shawn.

Re: Problem with git-daemon and mmap.

[Tim Ansell]

<snip>

> In the debian package version 1.4.4.4, git-daemon's memory by default is
> limited to 64MB.  I changed that with 1.5.0, but maybe you chose to not
> accept the changes to the run script when upgrading to 1.5.5.1-1.
> 
> Check /var/service/git-daemon/run, if it has 'exec chpst -m64000000
> git-daemon ...'.  If so, remove 'chpst -m64000000', and restart the
> git-daemon service through 'sv restart git-daemon'.

Doh!

It appears that after upgrading the git-daemon-run from 1.4.4 to 1.5.1,
I didn't do a "sv restart git-daemon".

After running the command I can now do a checkout form the web
repository, yay!

It appears to use 128m of virtual memory when checking out the
repository (which is why it would be hitting the 64Mb limit).

Would adding a "chpst -m256000000" be a good idea?

Thanks for your help.

Tim Ansell

Re: Problem with git-daemon and mmap.

[Gerrit Pape]

On Wed, Apr 25, 2007 at 05:31:13PM +0930, Tim Ansell wrote:
> > In the debian package version 1.4.4.4, git-daemon's memory by default is
> > limited to 64MB.  I changed that with 1.5.0, but maybe you chose to not
> > accept the changes to the run script when upgrading to 1.5.5.1-1.
> > 
> > Check /var/service/git-daemon/run, if it has 'exec chpst -m64000000
> > git-daemon ...'.  If so, remove 'chpst -m64000000', and restart the
> > git-daemon service through 'sv restart git-daemon'.

> It appears that after upgrading the git-daemon-run from 1.4.4 to 1.5.1,
> I didn't do a "sv restart git-daemon".

Probably the package upgrade should have done this for you, my fault.

> After running the command I can now do a checkout form the web
> repository, yay!
> 
> It appears to use 128m of virtual memory when checking out the
> repository (which is why it would be hitting the 64Mb limit).
> 
> Would adding a "chpst -m256000000" be a good idea?

I personally think so, yes, limiting the memory for long running
processes (services) generally is a good idea.  But finding a good
default for git-daemon isn't easy, as it depends on the repositories it
serves, and requires lots of memory per se.  So on Debian, it's now
unlimited by default.

Regards, Gerrit.

Re: Problem with git-daemon and mmap.

[Alex Riesen]

On 4/25/07, Tim Ansell <mithro@mithis.com> wrote:
> which is quite large (about ~126Mb) is causing problems when trying to
> clone via git-daemon.

126Mb sounds so _small_ compared to the repos seen lately (Firefox,
KDE, hell, my own work is long in Gb ballpark!). You're lucky :)

Re: Problem with git-daemon and mmap.

[Linus Torvalds]

On Wed, 25 Apr 2007, Shawn O. Pearce wrote:
> 
> And that was the last window in this packfile.  So we closed it.
> Way to go us.  Our xmmap does not expect release_pack_memory
> to close the fd its about to map...

Yeah, that looks like a real bug - it means that if we only have enough 
memory to have _one_ window open, the whole pack-file windowing doesn't 
work.

Now, in this case, it looks like Tim's problem was just some way too tight 
memory limits by his Debian install, and is fixed for him now, but this 
does seem to be a beauty wart that should be fixed.

> But that strace above shows a secondary bug when we get into this
> sort of situation.  Not pretty.

Yeah. 

	ulimit -S -v 65536
	git clone linux-2.6 test-linux-2.6

or similar should be able to reproduce it, if you just set the pack-file 
window to something like 32MB. Maybe.

			Linus