Re: Question about `git gc` locking files

[lilydjwg <lilydjwg@gmail.com>]

On Tue, Oct 22, 2024 at 03:03:21PM +0200, Patrick Steinhardt wrote:
> On Mon, Oct 21, 2024 at 03:55:45PM -0700, Calvin Wan wrote:
>> Recently, after upgrading to 2.47.0, we had internal reports of users
>> erroring out with:
>> 
>> fatal: cannot lock ref 'HEAD': Unable to create
>> '<filepath>/.git/HEAD.lock': File exists.
>> 
>> We believe this is due to "(98077d06) run-command: fix detaching when
>> running auto maintenance", since we have neither `gc.autoDetach` nor
>> `maintenance.autoDetach` set.
> 
> git-maintenance(1) detaches itself by default unless told not to via the
> config keys that you mention.
> 
>> When this bug was fixed, the maintenance runs that triggered during
>> usage of the external tool, repo[1], would lock the HEAD file in the
>> Android manifest repository thereby erroring out `repo`. Additionally,
>> long running maintenance tasks would also cause users to frequently
>> run into this issue when using git commands that are written to HEAD.
> 
> It is a bit surprising that HEAD would need to be locked in the first
> place. As far as I am aware, the only step where we end up locking refs
> in the context of git-gc(1) would be when we decide to repack refs via
> git-pack-refs(1). And that command shouldn't ever end up packing the
> HEAD file, as that loose reference must exist
> 
> Digging a bit deeper surfaces that it's `git reflog expire --all` that
> causes us to lock HEAD, which is... unfortunate. Seemingly, relfogs are
> locked by locking the corresponding reference.
> 
>> We can fix this easily temporarily by pushing out config changes to
>> run in the foreground, however, I was under the impression that `git
>> gc`, whether invoked normally or through `git maintenance`, would be
>> able to run in parallel with other git commands and therefore not
>> lock. There is no mention of this in the documentation for `git gc`,
>> but I do see it in the `git maintenance` documentation. So should `git
>> gc` be locking the HEAD file in the first place? And if so, is there a
>> way for `git gc` to have less of a dependence on HEAD.lock?
> 
> So what seems to be happening is that you have two processes racing with
> each other: one that is trying to expire entries from your "HEAD"
> reflog, and the one invoked by the user to update "HEAD". By default,
> Git will wait up to 100ms for the "HEAD" lock to be released, but it
> seems like expiring the reflog for your "HEAD" takes longer than that.
> You can work around that issue by increasing "core.filesRefLockTimeout".
> 
> But this whole thing uncovers an issue with git-maintenance(1) itself.
> The above commit fixed how git-maintenance(1) behaves such that we
> detach at the correct point in time. But what it neglects is that some
> tasks are more special than others and should be run synchronously
> whereas others can be run asynchronously. Packing refs and expiring the
> reflog are tasks that should be run synchronously to minimize the impact
> on users.
> 
> This all demonstrates that git-maintenance(1) needs to get some more
> love. You have uncovered this issue with git-gc(1) as configured task,
> but we have a similar fundamental issue with the git-pack-refs(1)
> subtask. So I guess we'll have to classify those subtasks into two
> categories, where one category needs to be executed before detaching
> itself and another category can be executed asynchronously after we have
> detached.

Hi, any news on this issue? (If not, I'd like to get notified when
there is.)

This issue has been happening to me too. I have automation tasks fail
occasionally in the last few months, and has only traced to "git gc"
(instead of another concurrent task of our own) just now. It is the
"git pull" command which fails for the most of time.

-- 
Best regards,
lilydjwg