Seeing high CPU usage during git-upload-pack

Seeing high CPU usage during git-upload-pack

[David Ebbo]

We’re using git-upload-pack in a simple git server. When the repo is
on a slow share, we’re noticing that git-upload-pack is hugging the
CPU during the whole pack-objects operation.

More details:

- git-upload-pack gets launched (with params git-upload-pack
--stateless-rpc //path/to/repo)
- it itself launches a git process (with params pack-objects --revs
--all --stdout --progress --delta-base-offset). This git process is
the one that does all the disk I/O, but it's not the one using up the
CPU time.
- git-upload-pack appears to be waiting for the git process, and is at
high CPU during that time, suggesting that it's doing some kind of
busy wait. To further test that, I stopped the git process in the
debugger, preventing it from making progress. At that point,
git-upload-pack pegs the CPU forever.

Would someone familiar with the sources be able to comment on this
busy wait behavior, and on whether it could conceivably be done in a
way that doesn’t use up some much CPU?

Thanks,
David

Re: Seeing high CPU usage during git-upload-pack

[Nguyen Thai Ngoc Duy]

On Wed, May 9, 2012 at 2:01 PM, David Ebbo <david.ebbo@gmail.com> wrote:
> We’re using git-upload-pack in a simple git server. When the repo is
> on a slow share, we’re noticing that git-upload-pack is hugging the
> CPU during the whole pack-objects operation.

What OS is it run on? What git version?

> More details:
>
> - git-upload-pack gets launched (with params git-upload-pack
> --stateless-rpc //path/to/repo)
> - it itself launches a git process (with params pack-objects --revs
> --all --stdout --progress --delta-base-offset). This git process is
> the one that does all the disk I/O, but it's not the one using up the
> CPU time.
> - git-upload-pack appears to be waiting for the git process, and is at
> high CPU during that time, suggesting that it's doing some kind of
> busy wait. To further test that, I stopped the git process in the
> debugger, preventing it from making progress. At that point,
> git-upload-pack pegs the CPU forever.

How about stopping upload-pack and see where it stops? There's a main
loop in upload-pack.c, create_pack_file() that is only active when
data comes. I looked but failed to see how it becomes a busywait loop.
Maybe you can put some debugging there.

Setting env var GIT_TRACE_PACKET and GIT_DEBUG_SEND_PACK before
running upload-pack might also help.

> Would someone familiar with the sources be able to comment on this
> busy wait behavior, and on whether it could conceivably be done in a
> way that doesn’t use up some much CPU?
-- 
Duy

Re: Seeing high CPU usage during git-upload-pack

[Nguyen Thai Ngoc Duy]

Hi,

I don't see anything private in the mail, so I CC git@vger again. It's
a good thing to keep git@vger in the loop because there will be more
(experienced) people to help you out.

On Wed, May 9, 2012 at 9:37 PM, David Ebbo <david.ebbo@gmail.com> wrote:
> Thanks Duy for your looking at this.
>
> On Wed, May 9, 2012 at 3:37 AM, Nguyen Thai Ngoc Duy <pclouds@gmail.com> wrote:
>>
>> What OS is it run on? What git version?
>>
>
> This is using 1.7.9 on Windows (msysgit).

Ah... msysgit reimplements poll() function, in compat/win32/poll.c. If
it goes rouge, you could get an active loop. But this is just my
suspicion. I don't know anything about msysgit's poll implementation
so I can be completely wrong here.

>> How about stopping upload-pack and see where it stops? There's a main
>> loop in upload-pack.c, create_pack_file() that is only active when
>> data comes. I looked but failed to see how it becomes a busywait loop.
>> Maybe you can put some debugging there.
>>
>> Setting env var GIT_TRACE_PACKET and GIT_DEBUG_SEND_PACK before
>> running upload-pack might also help.
>>
>
> I'll try those env vars, and will debug as needed (I haven't done that
> yet, so I'll need to get set up <g>)
>
> Thanks for the pointers!
-- 
Duy

Re: Seeing high CPU usage during git-upload-pack

[David Ebbo]

On Wed, May 9, 2012 at 8:11 AM, Nguyen Thai Ngoc Duy <pclouds@gmail.com> wrote:
>
> Ah... msysgit reimplements poll() function, in compat/win32/poll.c. If
> it goes rouge, you could get an active loop. But this is just my
> suspicion. I don't know anything about msysgit's poll implementation
> so I can be completely wrong here.
>

Sorry for the delayed replay, and thanks for pointing out that this is
msysgit specific and not general to all platform. I guess the next
step is to discuss it on their forum (msysgit@googlegroups.com).

thanks!
David