Deadlock during remote update

Deadlock during remote update

[Heald, Mike]

Hi,

We have a cron job that runs remote update on a number of repositories. Sometimes, the processes deadlock and we have to go -TERM them. Here's a breakdown of what state the processes end up in when the deadlock happens, from one of our production systems yesterday:

31629 git --git-dir=/var/lib/jeepyb/openstack/nova/.git --work-tree=/var/lib/jeepyb/openstack/nova remote update --prune
    wait4'ing for 31630
31630 git fetch --prune --multiple --all
    wait4'ing for 31637
31637 git fetch --append --prune upstream
    waiting for read from pipe:[170381707] (fd 25)
31638 git-remote-https upstream https://git.openstack.org/openstack/nova
    has pipe:[170381707] (fd 1), waiting for read from pipe:[170384472]
31642 git fetch-pack --stateless-rpc --lock-pack --include-tag --thin --no-progress https://git.openstack.org/openstack/nova/  efs/heads/master
    has pipe:[170384472] (fd 1), waiting for read from pipe:[170384471] (fd 0) which fd 7 on 31638

31638 and 31642 are both waiting to read from a pipe that the other has, and isn't writing to.

This is git version 1.7.9.5, OS is ubuntu precise.

Any ideas, or any way I can get more useful info?

Thanks,
Mike

Re: Deadlock during remote update

[Duy Nguyen]

On Fri, Feb 27, 2015 at 6:03 PM, Heald, Mike <mike.heald@hp.com> wrote:
> Hi,
>
> We have a cron job that runs remote update on a number of repositories. Sometimes, the processes deadlock and we have to go -TERM them. Here's a breakdown of what state the processes end up in when the deadlock happens, from one of our production systems yesterday:
>
> 31629 git --git-dir=/var/lib/jeepyb/openstack/nova/.git --work-tree=/var/lib/jeepyb/openstack/nova remote update --prune
>     wait4'ing for 31630
> 31630 git fetch --prune --multiple --all
>     wait4'ing for 31637
> 31637 git fetch --append --prune upstream
>     waiting for read from pipe:[170381707] (fd 25)
> 31638 git-remote-https upstream https://git.openstack.org/openstack/nova
>     has pipe:[170381707] (fd 1), waiting for read from pipe:[170384472]
> 31642 git fetch-pack --stateless-rpc --lock-pack --include-tag --thin --no-progress https://git.openstack.org/openstack/nova/  efs/heads/master
>     has pipe:[170384472] (fd 1), waiting for read from pipe:[170384471] (fd 0) which fd 7 on 31638
>
> 31638 and 31642 are both waiting to read from a pipe that the other has, and isn't writing to.
>
> This is git version 1.7.9.5, OS is ubuntu precise.

There's this commit 2a45520 (remote-curl: always parse incoming refs -
2013-02-20) that fixes a deadlock between remote-http(s) and
fetch-pack. I'm not sure if it's the same problem. Jeff  knows more
about this. The fix is available since v1.8.3
-- 
Duy

Re: Deadlock during remote update

[Jeff King]

On Fri, Feb 27, 2015 at 06:27:28PM +0700, Duy Nguyen wrote:

> > 31638 git-remote-https upstream https://git.openstack.org/openstack/nova
> >     has pipe:[170381707] (fd 1), waiting for read from pipe:[170384472]
> > 31642 git fetch-pack --stateless-rpc --lock-pack --include-tag --thin --no-progress https://git.openstack.org/openstack/nova/  efs/heads/master
> >     has pipe:[170384472] (fd 1), waiting for read from pipe:[170384471] (fd 0) which fd 7 on 31638
> >
> > 31638 and 31642 are both waiting to read from a pipe that the other has, and isn't writing to.
> >
> > This is git version 1.7.9.5, OS is ubuntu precise.
> 
> There's this commit 2a45520 (remote-curl: always parse incoming refs -
> 2013-02-20) that fixes a deadlock between remote-http(s) and
> fetch-pack. I'm not sure if it's the same problem. Jeff  knows more
> about this. The fix is available since v1.8.3

You are more optimistic about my memory than I am. :)

Given the symptoms of the deadlock above, I agree it is a likely
culprit. There was a spree of deadlock fixes around the same time, but I
think that is the only one that was found on the client side of a fetch.

-Peff