Re: Consist timestamps within a checkout/clone

["Ævar Arnfjörð Bjarmason" <avarab@gmail.com>]

On Mon, Oct 31 2022, Mark Hills wrote:

> On Mon, 31 Oct 2022, Ævar Arnfjörð Bjarmason wrote:
>
>> 
>> On Mon, Oct 31 2022, Mark Hills wrote:
>> 
>> > Our use case: we commit some compiled objects to the repo, where compiling 
>> > is either slow or requires software which is not always available.
>> >
>> > Since upgrading Git 2.26.3 -> 2.32.4 (as part of Alpine Linux OS upgrade) 
>> > we are noticing a change in build behaviour.
>> >
>> > Now, after a "git clone" we find the Makefile intermittently attempting 
>> > (and failing) some builds that are not intended.
>> >
>> > Indeed, Make is acting reasonably as the source file is sometimes 
>> > marginally newer than the destination (both checked out by Git), example 
>> > below.
>> >
>> > I've never had to consider consistency timestamps within a Git checkout 
>> > until now.
>> >
>> > It's entirely possible there's _never_ a guarantee of consistency here.
>> >
>> > But then something has certainly changed in practice, as this fault has 
>> > gone from never happening to now every couple of days.
>> >
>> > Imaginging I can't be the first person to encounter this, I searched for 
>> > existing threads or docs, but overwhemingly the results were question of 
>> > Git tracking the timestamps (as part of the commit) which this is not; 
>> > it's consistency within one checkout.
>> >
>> > $ git clone --depth 1 file:///path/to/repo.git
>> >
>> > $ stat winner.jpeg
>> >   File: winner.jpeg
>> >   Size: 258243          Blocks: 520        IO Block: 4096   regular file
>> > Device: fd07h/64775d    Inode: 33696       Links: 1
>> > Access: (0644/-rw-r--r--)  Uid: (  106/ luthier)   Gid: (  106/ luthier)
>> > Access: 2022-10-31 16:05:17.756858496 +0000
>> > Modify: 2022-10-31 16:05:17.756858496 +0000
>> > Change: 2022-10-31 16:05:17.756858496 +0000
>> >  Birth: -
>> >
>> > $ stat winner.svg
>> >   File: winner.svg
>> >   Size: 52685           Blocks: 112        IO Block: 4096   regular file
>> > Device: fd07h/64775d    Inode: 33697       Links: 1
>> > Access: (0644/-rw-r--r--)  Uid: (  106/ luthier)   Gid: (  106/ luthier)
>> > Access: 2022-10-31 16:05:17.766859030 +0000
>> > Modify: 2022-10-31 16:05:17.766859030 +0000
>> > Change: 2022-10-31 16:05:17.766859030 +0000
>> >  Birth: -
>> >
>> > Elsewhere in the repository, it's clear the timestamps are not consistent:
>> >
>> > $ stat Makefile
>> >   File: Makefile
>> >   Size: 8369            Blocks: 24         IO Block: 4096   regular file
>> > Device: fd07h/64775d    Inode: 33655       Links: 1
>> > Access: (0644/-rw-r--r--)  Uid: (  106/ luthier)   Gid: (  106/ luthier)
>> > Access: 2022-10-31 16:05:51.628660212 +0000
>> > Modify: 2022-10-31 16:05:17.746857963 +0000
>> > Change: 2022-10-31 16:05:17.746857963 +0000
>> >  Birth: -
>> 
>> I think you're almost certainly running into the parallel checkout,
>> which is new in that revision range. Try tweaking checkout.workers and
>> checkout.thresholdForParallelism (see "man git-config").
>
> Thanks, it will be interesting to try this and I'll report back.

FWIW I was under the impression that we'd made it the default, so unless
you opted-in it's probably not that.

>> I can't say without looking at the code/Makefile (and even then, I don't
>> have time to dig here:), but if I had to bet I'd say that your
>> dependencies have probably always been broken with these checked-in
>> files, but they happend to work out if they were checked out in sorted
>> order.
>>
>> And now with the parallel checkout they're not guaranteed to do that, as
>> some workers will "race ahead" and finish in an unpredictable order.
>
> These are very simple Makefile rules, I don't think these dependencies are 
> broken; but your theory is in good alignment with the observed behaviour.
>
> For example, the rule from the recent case above is:
>
>   %.jpeg:         %.png
>                   convert $< $(IMFLAGS) $@
>
>   %.png:          %.svg
>                   inkscape --export-type=png --export-filename=$@ $<

Grom a glance those don't seem broken to me, but I don't know how it
interacts with your built assets.

So e.g. if you are checking in your *.jpeg files those will be more
recent than either the *.png or source *.svn, so they won't be built.

This is fast getting out of scope of Git-specific advice, but you should
run "make --debug" (there's also sub-debug flags) to see if make's idea
of the dependency graph matches yours.