[Question] .git folder file updates for changing head commit

[Question] .git folder file updates for changing head commit

[John Garry]

Hi guys,

I have a question about which files in the .git folder are updated as we 
change the head commit. I could check the codebase myself but prob will 
make a mistake and maybe some expert would be so kind as to just kindly 
tell me.

For building the linux perf tool we use the git head commit id as part 
of the tool version sting. To save time in re-building, the Makefile 
rule has a dependency on .git/HEAD for rebuilding. An alternative 
approach would be to compare git log output to check current versus 
previous build head commit, but that is seen as inefficient time-wise.

The problem is that actions like git cherry-pick and git reset --hard 
HEAD^ may not update .git/HEAD (so don't trigger a rebuild).

Is there some more suitable file(s) which we could use as a dependency? 
 From my limited experimentation, .git/index seems to always update when 
the changing head commit.

Thanks,
john

Re: [Question] .git folder file updates for changing head commit

[Taylor Blau]

On Wed, Mar 23, 2022 at 03:19:06PM +0000, John Garry wrote:
> For building the linux perf tool we use the git head commit id as part of
> the tool version sting. To save time in re-building, the Makefile rule has a
> dependency on .git/HEAD for rebuilding. An alternative approach would be to
> compare git log output to check current versus previous build head commit,
> but that is seen as inefficient time-wise.

Having a Makefile recipe that depends on $GIT_DIR/HEAD seems strange to
me.

Presumably your Makefile rules would map out which parts of your program
depend on each other, and would get invalidated when the source itself
changes, no?

Perhaps you also care about the commit you're building from in order to
embed something into your program. But it seems like you could inject
the output of "git rev-parse HEAD" when you construct the version
identifier whenever you do need to rebuild.

Thanks,
Taylor

Re: [Question] .git folder file updates for changing head commit

[John Garry]

On 23/03/2022 20:25, Taylor Blau wrote:
> On Wed, Mar 23, 2022 at 03:19:06PM +0000, John Garry wrote:
>> For building the linux perf tool we use the git head commit id as part of
>> the tool version sting. To save time in re-building, the Makefile rule has a
>> dependency on .git/HEAD for rebuilding. An alternative approach would be to
>> compare git log output to check current versus previous build head commit,
>> but that is seen as inefficient time-wise.
> Having a Makefile recipe that depends on $GIT_DIR/HEAD seems strange to
> me.
> 

Yeah, as mentioned, the idea was the we can check a rebuild on this 
rather than running a relatively time-consuming command like "git 
describe". Wasn't my idea :)

> Presumably your Makefile rules would map out which parts of your program
> depend on each other, and would get invalidated when the source itself
> changes, no?

Correct, they do. It's just the part which constructs the app version 
tag which depends on this.

> 
> Perhaps you also care about the commit you're building from in order to
> embed something into your program. But it seems like you could inject
> the output of "git rev-parse HEAD" when you construct the version
> identifier whenever you do need to rebuild.
> 

ok, so I think that we may need to run a git command always one way or 
another.

Thanks,
John

Re: [Question] .git folder file updates for changing head commit

[Ævar Arnfjörð Bjarmason]

On Wed, Mar 23 2022, Taylor Blau wrote:

> On Wed, Mar 23, 2022 at 03:19:06PM +0000, John Garry wrote:
>> For building the linux perf tool we use the git head commit id as part of
>> the tool version sting. To save time in re-building, the Makefile rule has a
>> dependency on .git/HEAD for rebuilding. An alternative approach would be to
>> compare git log output to check current versus previous build head commit,
>> but that is seen as inefficient time-wise.
>
> Having a Makefile recipe that depends on $GIT_DIR/HEAD seems strange to
> me.
>
> Presumably your Makefile rules would map out which parts of your program
> depend on each other, and would get invalidated when the source itself
> changes, no?
>
> Perhaps you also care about the commit you're building from in order to
> embed something into your program. But it seems like you could inject
> the output of "git rev-parse HEAD" when you construct the version
> identifier whenever you do need to rebuild.

Our very own build process for git.git relies on this, see how version.o
needs a GIT-VERSION-FILE, which we generate by shelling out and
including.

It is unfortunate that we don't have an advertised way to do this, with
the ref backend I think trickery using $(file) to read the HEAD will
work to do it in pure-make, i.e. you'd need to parse it to see if it's a
symref, then depend on the target it points to.

Or you could recursively depend on a glob of the whole refspace for
generating the version file ...

It would be nice if we had a way to guarantee that we'd write some file
on HEAD updates, AFAIK not even the new reference-transaction hook will
do that (due to "git reset --hard" and friends).

And yes, this does actually matter. There's a huge performance
difference between a Makefile that needs to shell out for every little
thing.

I've been optimizing our own Makefile incrementally from running in
~500ms down to 100-200ms for noop runs over over the last few
months. It's possible to get it down to 10-20ms at least by getting rid
of the remaining shell-outs.

That makes a big difference when e.g. using "make" with "git rebase -i
-x".

Re: [Question] .git folder file updates for changing head commit

[John Garry]

On 27/03/2022 16:48, Ævar Arnfjörð Bjarmason wrote:
> On Wed, Mar 23 2022, Taylor Blau wrote:
> 
>> On Wed, Mar 23, 2022 at 03:19:06PM +0000, John Garry wrote:
>>> For building the linux perf tool we use the git head commit id as part of
>>> the tool version sting. To save time in re-building, the Makefile rule has a
>>> dependency on .git/HEAD for rebuilding. An alternative approach would be to
>>> compare git log output to check current versus previous build head commit,
>>> but that is seen as inefficient time-wise.
>> Having a Makefile recipe that depends on $GIT_DIR/HEAD seems strange to
>> me.
>>
>> Presumably your Makefile rules would map out which parts of your program
>> depend on each other, and would get invalidated when the source itself
>> changes, no?
>>
>> Perhaps you also care about the commit you're building from in order to
>> embed something into your program. But it seems like you could inject
>> the output of "git rev-parse HEAD" when you construct the version
>> identifier whenever you do need to rebuild.
> Our very own build process for git.git relies on this, see how version.o
> needs a GIT-VERSION-FILE, which we generate by shelling out and
> including.

I see.

> 
> It is unfortunate that we don't have an advertised way to do this, with
> the ref backend I think trickery using $(file) to read the HEAD will
> work to do it in pure-make, i.e. you'd need to parse it to see if it's a
> symref, then depend on the target it points to.
> 
> Or you could recursively depend on a glob of the whole refspace for
> generating the version file ...
> 
> It would be nice if we had a way to guarantee that we'd write some file
> on HEAD updates, AFAIK not even the new reference-transaction hook will
> do that (due to "git reset --hard" and friends).
> 
> And yes, this does actually matter. There's a huge performance
> difference between a Makefile that needs to shell out for every little
> thing.
> 

I'm not sure if I mentioned it before, but the main motivation for 
adding the dependency was that "git describe" may be slow. We have since 
moved away from that (using "describe", which I hope turns out to be 
ok), but your GIT-VERSION-GEN still uses it. I just found that "git 
describe" is unreliable for us as it depends on tags being pushed to a 
clone git.

> I've been optimizing our own Makefile incrementally from running in
> ~500ms down to 100-200ms for noop runs over over the last few
> months. It's possible to get it down to 10-20ms at least by getting rid
> of the remaining shell-outs.
> 
> That makes a big difference when e.g. using "make" with "git rebase -i
> -x".
> .

ok, thanks,
John

Re: [Question] .git folder file updates for changing head commit

[Glen Choo]

Hi John, I'm not a big expert on Make (or even Git :p) but think I can
answer some of your questions.

John Garry <john.garry@huawei.com> writes:

> Hi guys,
>
> I have a question about which files in the .git folder are updated as we 
> change the head commit. I could check the codebase myself but prob will 
> make a mistake and maybe some expert would be so kind as to just kindly 
> tell me.
>
> For building the linux perf tool we use the git head commit id as part 
> of the tool version sting. To save time in re-building, the Makefile 
> rule has a dependency on .git/HEAD for rebuilding. An alternative 
> approach would be to compare git log output to check current versus 
> previous build head commit, but that is seen as inefficient time-wise.

You're on the right track because if .git/HEAD doesn't change, the HEAD
is still pointing to either the same commit (if in detached HEAD) or
branch (if not in detached HEAD).

The problem is that when HEAD points to a branch, the branch's 'head
commit' can change, implicitly changing the commit that .git/HEAD
actually points to.

> The problem is that actions like git cherry-pick and git reset --hard 
> HEAD^ may not update .git/HEAD (so don't trigger a rebuild).

That's what's happening here. If your HEAD is pointing to
"refs/heads/main", "cherry-pick" and "reset --hard" will change where
"refs/heads/main" points, but HEAD still points to "refs/heads/main".

Branches are often located in "loose form" under .git/refs/heads/*,
*but* refs can also be stored in a "packed form" under .git/packed-refs
(https://git-scm.com/docs/git-pack-refs). There is also a relatively
new ref storage format called reftable that I'm not familiar with, but
presumably stores its refs in other locations too.

All of this is to say that depending on specific files under .git/
exposes you to a lot of Git internals that really aren't meant for
external consumption. I suggest finding a different approach than
watching Git-internal files for changes.

> Is there some more suitable file(s) which we could use as a dependency? 
>  From my limited experimentation, .git/index seems to always update when 
> the changing head commit.

I think users typically expect the index (and therefore .git/index) to
change when the head commit changes, but this isn't always true e.g.
"reset --soft" will move the branch without changing the index.

P.S. Even if you knew the value of .git/index or .git/HEAD, that
wouldn't tell you whether or not the files in your working directory
have changed, so I'm not sure if you want to use either as Makefile
dependency.

>
> Thanks,
> john

Re: [Question] .git folder file updates for changing head commit

[John Garry]

On 23/03/2022 20:50, Glen Choo wrote:
> Hi John, I'm not a big expert on Make (or even Git :p) but think I can
> answer some of your questions.
> 
> John Garry <john.garry@huawei.com> writes:
> 
>> Hi guys,
>>
>> I have a question about which files in the .git folder are updated as we
>> change the head commit. I could check the codebase myself but prob will
>> make a mistake and maybe some expert would be so kind as to just kindly
>> tell me.
>>
>> For building the linux perf tool we use the git head commit id as part
>> of the tool version sting. To save time in re-building, the Makefile
>> rule has a dependency on .git/HEAD for rebuilding. An alternative
>> approach would be to compare git log output to check current versus
>> previous build head commit, but that is seen as inefficient time-wise.
> 
> You're on the right track because if .git/HEAD doesn't change, the HEAD
> is still pointing to either the same commit (if in detached HEAD) or
> branch (if not in detached HEAD).
> 
> The problem is that when HEAD points to a branch, the branch's 'head
> commit' can change, implicitly changing the commit that .git/HEAD
> actually points to.
> 

ok

>> The problem is that actions like git cherry-pick and git reset --hard
>> HEAD^ may not update .git/HEAD (so don't trigger a rebuild).
> 
> That's what's happening here. If your HEAD is pointing to
> "refs/heads/main", "cherry-pick" and "reset --hard" will change where
> "refs/heads/main" points, but HEAD still points to "refs/heads/main".
> 
> Branches are often located in "loose form" under .git/refs/heads/*,
> *but* refs can also be stored in a "packed form" under .git/packed-refs
> (https://git-scm.com/docs/git-pack-refs). There is also a relatively
> new ref storage format called reftable that I'm not familiar with, but
> presumably stores its refs in other locations too.
> 
> All of this is to say that depending on specific files under .git/
> exposes you to a lot of Git internals that really aren't meant for
> external consumption. I suggest finding a different approach than
> watching Git-internal files for changes.

Thanks for the detailed explanation.

I agree with your suggestion to find a new method (which may be 
reverting to the old method to just run proper git commands).

> 
>> Is there some more suitable file(s) which we could use as a dependency?
>>   From my limited experimentation, .git/index seems to always update when
>> the changing head commit.
> 
> I think users typically expect the index (and therefore .git/index) to
> change when the head commit changes, but this isn't always true e.g.
> "reset --soft" will move the branch without changing the index.

ok, good to know. I was going to try that...

> 
> P.S. Even if you knew the value of .git/index or .git/HEAD, that
> wouldn't tell you whether or not the files in your working directory
> have changed, so I'm not sure if you want to use either as Makefile
> dependency.
> 

ok, thanks again for all this info,
John

Re: [Question] .git folder file updates for changing head commit

[Andreas Schwab]

On Mär 23 2022, John Garry wrote:

> For building the linux perf tool we use the git head commit id as part of
> the tool version sting. To save time in re-building, the Makefile rule has
> a dependency on .git/HEAD for rebuilding.

There is no guarantee that .git is a directory.  In a separate worktree
.git is actually a gitfile.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: [Question] .git folder file updates for changing head commit

[John Garry]

On 23/03/2022 21:28, Andreas Schwab wrote:
> On Mär 23 2022, John Garry wrote:
> 
>> For building the linux perf tool we use the git head commit id as part of
>> the tool version sting. To save time in re-building, the Makefile rule has
>> a dependency on .git/HEAD for rebuilding.
> There is no guarantee that .git is a directory.  In a separate worktree
> .git is actually a gitfile.

ok, thanks. I'll keep that in mind.

John