Question about `git gc` locking files

Question about `git gc` locking files

[Calvin Wan]

Recently, after upgrading to 2.47.0, we had internal reports of users
erroring out with:

fatal: cannot lock ref 'HEAD': Unable to create
'<filepath>/.git/HEAD.lock': File exists.

We believe this is due to "(98077d06) run-command: fix detaching when
running auto maintenance", since we have neither `gc.autoDetach` nor
`maintenance.autoDetach` set. When this bug was fixed, the maintenance
runs that triggered during usage of the external tool, repo[1], would
lock the HEAD file in the Android manifest repository thereby erroring
out `repo`. Additionally, long running maintenance tasks would also
cause users to frequently run into this issue when using git commands
that are written to HEAD.

We can fix this easily temporarily by pushing out config changes to
run in the foreground, however, I was under the impression that `git
gc`, whether invoked normally or through `git maintenance`, would be
able to run in parallel with other git commands and therefore not
lock. There is no mention of this in the documentation for `git gc`,
but I do see it in the `git maintenance` documentation. So should `git
gc` be locking the HEAD file in the first place? And if so, is there a
way for `git gc` to have less of a dependence on HEAD.lock?

Thanks,
Calvin

[1] https://gerrit.googlesource.com/git-repo

Re: Question about `git gc` locking files

[Patrick Steinhardt]

On Mon, Oct 21, 2024 at 03:55:45PM -0700, Calvin Wan wrote:
> Recently, after upgrading to 2.47.0, we had internal reports of users
> erroring out with:
> 
> fatal: cannot lock ref 'HEAD': Unable to create
> '<filepath>/.git/HEAD.lock': File exists.
> 
> We believe this is due to "(98077d06) run-command: fix detaching when
> running auto maintenance", since we have neither `gc.autoDetach` nor
> `maintenance.autoDetach` set.

git-maintenance(1) detaches itself by default unless told not to via the
config keys that you mention.

> When this bug was fixed, the maintenance runs that triggered during
> usage of the external tool, repo[1], would lock the HEAD file in the
> Android manifest repository thereby erroring out `repo`. Additionally,
> long running maintenance tasks would also cause users to frequently
> run into this issue when using git commands that are written to HEAD.

It is a bit surprising that HEAD would need to be locked in the first
place. As far as I am aware, the only step where we end up locking refs
in the context of git-gc(1) would be when we decide to repack refs via
git-pack-refs(1). And that command shouldn't ever end up packing the
HEAD file, as that loose reference must exist

Digging a bit deeper surfaces that it's `git reflog expire --all` that
causes us to lock HEAD, which is... unfortunate. Seemingly, relfogs are
locked by locking the corresponding reference.

> We can fix this easily temporarily by pushing out config changes to
> run in the foreground, however, I was under the impression that `git
> gc`, whether invoked normally or through `git maintenance`, would be
> able to run in parallel with other git commands and therefore not
> lock. There is no mention of this in the documentation for `git gc`,
> but I do see it in the `git maintenance` documentation. So should `git
> gc` be locking the HEAD file in the first place? And if so, is there a
> way for `git gc` to have less of a dependence on HEAD.lock?

So what seems to be happening is that you have two processes racing with
each other: one that is trying to expire entries from your "HEAD"
reflog, and the one invoked by the user to update "HEAD". By default,
Git will wait up to 100ms for the "HEAD" lock to be released, but it
seems like expiring the reflog for your "HEAD" takes longer than that.
You can work around that issue by increasing "core.filesRefLockTimeout".

But this whole thing uncovers an issue with git-maintenance(1) itself.
The above commit fixed how git-maintenance(1) behaves such that we
detach at the correct point in time. But what it neglects is that some
tasks are more special than others and should be run synchronously
whereas others can be run asynchronously. Packing refs and expiring the
reflog are tasks that should be run synchronously to minimize the impact
on users.

This all demonstrates that git-maintenance(1) needs to get some more
love. You have uncovered this issue with git-gc(1) as configured task,
but we have a similar fundamental issue with the git-pack-refs(1)
subtask. So I guess we'll have to classify those subtasks into two
categories, where one category needs to be executed before detaching
itself and another category can be executed asynchronously after we have
detached.

Patrick

Re: Question about `git gc` locking files

[Taylor Blau]

On Tue, Oct 22, 2024 at 03:03:21PM +0200, Patrick Steinhardt wrote:
> > When this bug was fixed, the maintenance runs that triggered during
> > usage of the external tool, repo[1], would lock the HEAD file in the
> > Android manifest repository thereby erroring out `repo`. Additionally,
> > long running maintenance tasks would also cause users to frequently
> > run into this issue when using git commands that are written to HEAD.
>
> It is a bit surprising that HEAD would need to be locked in the first
> place. As far as I am aware, the only step where we end up locking refs
> in the context of git-gc(1) would be when we decide to repack refs via
> git-pack-refs(1). And that command shouldn't ever end up packing the
> HEAD file, as that loose reference must exist
>
> Digging a bit deeper surfaces that it's `git reflog expire --all` that
> causes us to lock HEAD, which is... unfortunate. Seemingly, relfogs are
> locked by locking the corresponding reference.

Makes sense.

> > We can fix this easily temporarily by pushing out config changes to
> > run in the foreground, however, I was under the impression that `git
> > gc`, whether invoked normally or through `git maintenance`, would be
> > able to run in parallel with other git commands and therefore not
> > lock. There is no mention of this in the documentation for `git gc`,
> > but I do see it in the `git maintenance` documentation. So should `git
> > gc` be locking the HEAD file in the first place? And if so, is there a
> > way for `git gc` to have less of a dependence on HEAD.lock?
>
> So what seems to be happening is that you have two processes racing with
> each other: one that is trying to expire entries from your "HEAD"
> reflog, and the one invoked by the user to update "HEAD". By default,
> Git will wait up to 100ms for the "HEAD" lock to be released, but it
> seems like expiring the reflog for your "HEAD" takes longer than that.
> You can work around that issue by increasing "core.filesRefLockTimeout".
>
> But this whole thing uncovers an issue with git-maintenance(1) itself.
> The above commit fixed how git-maintenance(1) behaves such that we
> detach at the correct point in time. But what it neglects is that some
> tasks are more special than others and should be run synchronously
> whereas others can be run asynchronously. Packing refs and expiring the
> reflog are tasks that should be run synchronously to minimize the impact
> on users.
>
> This all demonstrates that git-maintenance(1) needs to get some more
> love. You have uncovered this issue with git-gc(1) as configured task,
> but we have a similar fundamental issue with the git-pack-refs(1)
> subtask. So I guess we'll have to classify those subtasks into two
> categories, where one category needs to be executed before detaching
> itself and another category can be executed asynchronously after we have
> detached.

Perhaps Stolee (CC'd) has some thoughts here?

Thanks,
Taylor

Re: Question about `git gc` locking files

[lilydjwg]

On Tue, Oct 22, 2024 at 03:03:21PM +0200, Patrick Steinhardt wrote:
> On Mon, Oct 21, 2024 at 03:55:45PM -0700, Calvin Wan wrote:
>> Recently, after upgrading to 2.47.0, we had internal reports of users
>> erroring out with:
>> 
>> fatal: cannot lock ref 'HEAD': Unable to create
>> '<filepath>/.git/HEAD.lock': File exists.
>> 
>> We believe this is due to "(98077d06) run-command: fix detaching when
>> running auto maintenance", since we have neither `gc.autoDetach` nor
>> `maintenance.autoDetach` set.
> 
> git-maintenance(1) detaches itself by default unless told not to via the
> config keys that you mention.
> 
>> When this bug was fixed, the maintenance runs that triggered during
>> usage of the external tool, repo[1], would lock the HEAD file in the
>> Android manifest repository thereby erroring out `repo`. Additionally,
>> long running maintenance tasks would also cause users to frequently
>> run into this issue when using git commands that are written to HEAD.
> 
> It is a bit surprising that HEAD would need to be locked in the first
> place. As far as I am aware, the only step where we end up locking refs
> in the context of git-gc(1) would be when we decide to repack refs via
> git-pack-refs(1). And that command shouldn't ever end up packing the
> HEAD file, as that loose reference must exist
> 
> Digging a bit deeper surfaces that it's `git reflog expire --all` that
> causes us to lock HEAD, which is... unfortunate. Seemingly, relfogs are
> locked by locking the corresponding reference.
> 
>> We can fix this easily temporarily by pushing out config changes to
>> run in the foreground, however, I was under the impression that `git
>> gc`, whether invoked normally or through `git maintenance`, would be
>> able to run in parallel with other git commands and therefore not
>> lock. There is no mention of this in the documentation for `git gc`,
>> but I do see it in the `git maintenance` documentation. So should `git
>> gc` be locking the HEAD file in the first place? And if so, is there a
>> way for `git gc` to have less of a dependence on HEAD.lock?
> 
> So what seems to be happening is that you have two processes racing with
> each other: one that is trying to expire entries from your "HEAD"
> reflog, and the one invoked by the user to update "HEAD". By default,
> Git will wait up to 100ms for the "HEAD" lock to be released, but it
> seems like expiring the reflog for your "HEAD" takes longer than that.
> You can work around that issue by increasing "core.filesRefLockTimeout".
> 
> But this whole thing uncovers an issue with git-maintenance(1) itself.
> The above commit fixed how git-maintenance(1) behaves such that we
> detach at the correct point in time. But what it neglects is that some
> tasks are more special than others and should be run synchronously
> whereas others can be run asynchronously. Packing refs and expiring the
> reflog are tasks that should be run synchronously to minimize the impact
> on users.
> 
> This all demonstrates that git-maintenance(1) needs to get some more
> love. You have uncovered this issue with git-gc(1) as configured task,
> but we have a similar fundamental issue with the git-pack-refs(1)
> subtask. So I guess we'll have to classify those subtasks into two
> categories, where one category needs to be executed before detaching
> itself and another category can be executed asynchronously after we have
> detached.

Hi, any news on this issue? (If not, I'd like to get notified when
there is.)

This issue has been happening to me too. I have automation tasks fail
occasionally in the last few months, and has only traced to "git gc"
(instead of another concurrent task of our own) just now. It is the
"git pull" command which fails for the most of time.

-- 
Best regards,
lilydjwg