Consist timestamps within a checkout/clone

Consist timestamps within a checkout/clone

[Mark Hills]

Our use case: we commit some compiled objects to the repo, where compiling 
is either slow or requires software which is not always available.

Since upgrading Git 2.26.3 -> 2.32.4 (as part of Alpine Linux OS upgrade) 
we are noticing a change in build behaviour.

Now, after a "git clone" we find the Makefile intermittently attempting 
(and failing) some builds that are not intended.

Indeed, Make is acting reasonably as the source file is sometimes 
marginally newer than the destination (both checked out by Git), example 
below.

I've never had to consider consistency timestamps within a Git checkout 
until now.

It's entirely possible there's _never_ a guarantee of consistency here.

But then something has certainly changed in practice, as this fault has 
gone from never happening to now every couple of days.

Imaginging I can't be the first person to encounter this, I searched for 
existing threads or docs, but overwhemingly the results were question of 
Git tracking the timestamps (as part of the commit) which this is not; 
it's consistency within one checkout.

$ git clone --depth 1 file:///path/to/repo.git

$ stat winner.jpeg
  File: winner.jpeg
  Size: 258243          Blocks: 520        IO Block: 4096   regular file
Device: fd07h/64775d    Inode: 33696       Links: 1
Access: (0644/-rw-r--r--)  Uid: (  106/ luthier)   Gid: (  106/ luthier)
Access: 2022-10-31 16:05:17.756858496 +0000
Modify: 2022-10-31 16:05:17.756858496 +0000
Change: 2022-10-31 16:05:17.756858496 +0000
 Birth: -

$ stat winner.svg
  File: winner.svg
  Size: 52685           Blocks: 112        IO Block: 4096   regular file
Device: fd07h/64775d    Inode: 33697       Links: 1
Access: (0644/-rw-r--r--)  Uid: (  106/ luthier)   Gid: (  106/ luthier)
Access: 2022-10-31 16:05:17.766859030 +0000
Modify: 2022-10-31 16:05:17.766859030 +0000
Change: 2022-10-31 16:05:17.766859030 +0000
 Birth: -

Elsewhere in the repository, it's clear the timestamps are not consistent:

$ stat Makefile
  File: Makefile
  Size: 8369            Blocks: 24         IO Block: 4096   regular file
Device: fd07h/64775d    Inode: 33655       Links: 1
Access: (0644/-rw-r--r--)  Uid: (  106/ luthier)   Gid: (  106/ luthier)
Access: 2022-10-31 16:05:51.628660212 +0000
Modify: 2022-10-31 16:05:17.746857963 +0000
Change: 2022-10-31 16:05:17.746857963 +0000
 Birth: -

-- 
Mark

Re: Consist timestamps within a checkout/clone

[Andreas Schwab]

On Okt 31 2022, Mark Hills wrote:

> It's entirely possible there's _never_ a guarantee of consistency here.

I don't think the order in which git writes the individual files is
defined in any way.  Thus depending on the precision of the time stamps
in the file system whether a file ends up newer than another one may
vary each time due to timing differences.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Consist timestamps within a checkout/clone

[Ævar Arnfjörð Bjarmason]

On Mon, Oct 31 2022, Mark Hills wrote:

> Our use case: we commit some compiled objects to the repo, where compiling 
> is either slow or requires software which is not always available.
>
> Since upgrading Git 2.26.3 -> 2.32.4 (as part of Alpine Linux OS upgrade) 
> we are noticing a change in build behaviour.
>
> Now, after a "git clone" we find the Makefile intermittently attempting 
> (and failing) some builds that are not intended.
>
> Indeed, Make is acting reasonably as the source file is sometimes 
> marginally newer than the destination (both checked out by Git), example 
> below.
>
> I've never had to consider consistency timestamps within a Git checkout 
> until now.
>
> It's entirely possible there's _never_ a guarantee of consistency here.
>
> But then something has certainly changed in practice, as this fault has 
> gone from never happening to now every couple of days.
>
> Imaginging I can't be the first person to encounter this, I searched for 
> existing threads or docs, but overwhemingly the results were question of 
> Git tracking the timestamps (as part of the commit) which this is not; 
> it's consistency within one checkout.
>
> $ git clone --depth 1 file:///path/to/repo.git
>
> $ stat winner.jpeg
>   File: winner.jpeg
>   Size: 258243          Blocks: 520        IO Block: 4096   regular file
> Device: fd07h/64775d    Inode: 33696       Links: 1
> Access: (0644/-rw-r--r--)  Uid: (  106/ luthier)   Gid: (  106/ luthier)
> Access: 2022-10-31 16:05:17.756858496 +0000
> Modify: 2022-10-31 16:05:17.756858496 +0000
> Change: 2022-10-31 16:05:17.756858496 +0000
>  Birth: -
>
> $ stat winner.svg
>   File: winner.svg
>   Size: 52685           Blocks: 112        IO Block: 4096   regular file
> Device: fd07h/64775d    Inode: 33697       Links: 1
> Access: (0644/-rw-r--r--)  Uid: (  106/ luthier)   Gid: (  106/ luthier)
> Access: 2022-10-31 16:05:17.766859030 +0000
> Modify: 2022-10-31 16:05:17.766859030 +0000
> Change: 2022-10-31 16:05:17.766859030 +0000
>  Birth: -
>
> Elsewhere in the repository, it's clear the timestamps are not consistent:
>
> $ stat Makefile
>   File: Makefile
>   Size: 8369            Blocks: 24         IO Block: 4096   regular file
> Device: fd07h/64775d    Inode: 33655       Links: 1
> Access: (0644/-rw-r--r--)  Uid: (  106/ luthier)   Gid: (  106/ luthier)
> Access: 2022-10-31 16:05:51.628660212 +0000
> Modify: 2022-10-31 16:05:17.746857963 +0000
> Change: 2022-10-31 16:05:17.746857963 +0000
>  Birth: -

I think you're almost certainly running into the parallel checkout,
which is new in that revision range. Try tweaking checkout.workers and
checkout.thresholdForParallelism (see "man git-config").

I can't say without looking at the code/Makefile (and even then, I don't
have time to dig here:), but if I had to bet I'd say that your
dependencies have probably always been broken with these checked-in
files, but they happend to work out if they were checked out in sorted
order.

And now with the parallel checkout they're not guaranteed to do that, as
some workers will "race ahead" and finish in an unpredictable order.

But that's all just a guess, perhaps it has nothing to do with parallel
checkout, such dependency issues are sensitive to all sorts of other
things, e.g. maybe git got slightly faster (or slower), so now files
that were always on different seconds (or the same) aren't in the state
they were in before...

Re: Consist timestamps within a checkout/clone

[Taylor Blau]

On Mon, Oct 31, 2022 at 09:21:20PM +0100, Ævar Arnfjörð Bjarmason wrote:
> I think you're almost certainly running into the parallel checkout,
> which is new in that revision range. Try tweaking checkout.workers and
> checkout.thresholdForParallelism (see "man git-config").
>
> I can't say without looking at the code/Makefile (and even then, I don't
> have time to dig here:), but if I had to bet I'd say that your
> dependencies have probably always been broken with these checked-in
> files, but they happend to work out if they were checked out in sorted
> order.
>
> And now with the parallel checkout they're not guaranteed to do that, as
> some workers will "race ahead" and finish in an unpredictable order.

Doesn't checkout.thresholdForParallelism only matter when
checkout.workers != 1?

So what you wrote seems like a reasonable explanation, but only if the
original reporter set checkout.workers to imply the non-sequential
behavior in the first place.

That said...

  - I also don't know off-hand of a place where we've defined the order
    where Git will checkout files in the working copy. So depending on
    that behavior isn't a safe thing to do.

  - Committing build artifacts into your repository is generally
    discouraged.

So while I'd guess that setting `checkout.workers` back to "1" (if it
wasn't already) will probably restore the existing behavior, counting
on that behavior in the first place is wrong.

Thanks,
Taylor

Re: Consist timestamps within a checkout/clone

[Mark Hills]

[-- Attachment #1: Type: text/plain, Size: 3133 bytes --]

On Mon, 31 Oct 2022, Taylor Blau wrote:

> On Mon, Oct 31, 2022 at 09:21:20PM +0100, Ævar Arnfjörð Bjarmason wrote:
> > I think you're almost certainly running into the parallel checkout,
> > which is new in that revision range. Try tweaking checkout.workers and
> > checkout.thresholdForParallelism (see "man git-config").
> >
> > I can't say without looking at the code/Makefile (and even then, I don't
> > have time to dig here:), but if I had to bet I'd say that your
> > dependencies have probably always been broken with these checked-in
> > files, but they happend to work out if they were checked out in sorted
> > order.
> >
> > And now with the parallel checkout they're not guaranteed to do that, as
> > some workers will "race ahead" and finish in an unpredictable order.
> 
> Doesn't checkout.thresholdForParallelism only matter when
> checkout.workers != 1?
> 
> So what you wrote seems like a reasonable explanation, but only if the
> original reporter set checkout.workers to imply the non-sequential
> behavior in the first place.
> 
> That said...
> 
>   - I also don't know off-hand of a place where we've defined the order
>     where Git will checkout files in the working copy. So depending on
>     that behavior isn't a safe thing to do.
> 
>   - Committing build artifacts into your repository is generally
>     discouraged.

If it's undefined and never implemented this is reasonable.

But "generally" is a caveat, so while I agree with the statement it also 
implies there's valid cases outside of that. Ones which used to work, too.

Here are some useful cases I have seen for the combination of build rule + 
checked in file:

- part of a build requires licensed software that's not always available

- part of the build requires large memory that other builders generally do 
  not have available

- part of the build process uses a different platform or some other system 
  requirement

- to fetch data eg. from a URL, with a record of the URL/automation but 
  also a copy of the file as a record and for offline use

So it's useful, to retain repeatable automation but not always build from 
square one.

Generally discouraged to check in build results yes, but I've found it 
very practical.
 
> So while I'd guess that setting `checkout.workers` back to "1" (if it 
> wasn't already) will probably restore the existing behavior, counting on 
> that behavior in the first place is wrong.

I think perhaps the tail is wagging the dog here, though.

It's 'wrong' because it doesn't work; but I haven't seen anything to make 
me think this is fundamentally or theoretically flawed.

If we had a transactional file system we'd reasonably expect a checkout to 
be an atomic operation -- same timestamp on the files created in that 
step. A discrepancy in timestamps would be considered incorrect; it would 
imply an 'order' to the checkout which, as you say, is order-less.

Sowhat could be the bad outcomes if Git created files stamped with the 
point in time of the "git checkout"?

> Thanks,
> Taylor
> 
> 

-- 
Mark

RE: Consist timestamps within a checkout/clone

[rsbecker]

On October 31, 2022 6:31 PM, Mark Hills wrote:
>On Mon, 31 Oct 2022, Taylor Blau wrote:
>> On Mon, Oct 31, 2022 at 09:21:20PM +0100, Ævar Arnfjörð Bjarmason wrote:
>> > I think you're almost certainly running into the parallel checkout,
>> > which is new in that revision range. Try tweaking checkout.workers
>> > and checkout.thresholdForParallelism (see "man git-config").
>> >
>> > I can't say without looking at the code/Makefile (and even then, I
>> > don't have time to dig here:), but if I had to bet I'd say that your
>> > dependencies have probably always been broken with these checked-in
>> > files, but they happend to work out if they were checked out in
>> > sorted order.
>> >
>> > And now with the parallel checkout they're not guaranteed to do
>> > that, as some workers will "race ahead" and finish in an unpredictable order.
>>
>> Doesn't checkout.thresholdForParallelism only matter when
>> checkout.workers != 1?
>>
>> So what you wrote seems like a reasonable explanation, but only if the
>> original reporter set checkout.workers to imply the non-sequential
>> behavior in the first place.
>>
>> That said...
>>
>>   - I also don't know off-hand of a place where we've defined the order
>>     where Git will checkout files in the working copy. So depending on
>>     that behavior isn't a safe thing to do.
>>
>>   - Committing build artifacts into your repository is generally
>>     discouraged.
>
>If it's undefined and never implemented this is reasonable.
>
>But "generally" is a caveat, so while I agree with the statement it also implies
>there's valid cases outside of that. Ones which used to work, too.
>
>Here are some useful cases I have seen for the combination of build rule +
>checked in file:
>
>- part of a build requires licensed software that's not always available
>
>- part of the build requires large memory that other builders generally do
>  not have available
>
>- part of the build process uses a different platform or some other system
>  requirement
>
>- to fetch data eg. from a URL, with a record of the URL/automation but
>  also a copy of the file as a record and for offline use
>
>So it's useful, to retain repeatable automation but not always build from square
>one.
>
>Generally discouraged to check in build results yes, but I've found it very practical.
>
>> So while I'd guess that setting `checkout.workers` back to "1" (if it
>> wasn't already) will probably restore the existing behavior, counting
>> on that behavior in the first place is wrong.
>
>I think perhaps the tail is wagging the dog here, though.
>
>It's 'wrong' because it doesn't work; but I haven't seen anything to make me think
>this is fundamentally or theoretically flawed.
>
>If we had a transactional file system we'd reasonably expect a checkout to be an
>atomic operation -- same timestamp on the files created in that step. A
>discrepancy in timestamps would be considered incorrect; it would imply an 'order'
>to the checkout which, as you say, is order-less.
>
>Sowhat could be the bad outcomes if Git created files stamped with the point in
>time of the "git checkout"?

Timestamps are written based on when git modifies the file in the working directory. This actually ensures that automation does work. If intermediate contents are checked into repositories (I have people who do this for very justifiable regulatory reasons), the build has to make sure that there are appropriate separations of timestamps (a.k.a. 1 second) at a minimum on UNIX-ish systems. On some other boxes that do not even have timestamps for files (you know who you are) this is moot.

However, there is a use case for maintaining timestamps - specifically for debuggers that check timestamps of source files. It is a big pain to make this work in git - but I script around this by setting the timestamps of files to the commit time when doing release builds, and allowing users to set the timestamp to the same for debugging. It helps but should not change the semantics of dev builds.

-Randall

Re: Consist timestamps within a checkout/clone

[Ævar Arnfjörð Bjarmason]

On Mon, Oct 31 2022, Mark Hills wrote:

> On Mon, 31 Oct 2022, Taylor Blau wrote:
>
>> On Mon, Oct 31, 2022 at 09:21:20PM +0100, Ævar Arnfjörð Bjarmason wrote:
>> > I think you're almost certainly running into the parallel checkout,
>> > which is new in that revision range. Try tweaking checkout.workers and
>> > checkout.thresholdForParallelism (see "man git-config").
>> >
>> > I can't say without looking at the code/Makefile (and even then, I don't
>> > have time to dig here:), but if I had to bet I'd say that your
>> > dependencies have probably always been broken with these checked-in
>> > files, but they happend to work out if they were checked out in sorted
>> > order.
>> >
>> > And now with the parallel checkout they're not guaranteed to do that, as
>> > some workers will "race ahead" and finish in an unpredictable order.
>> 
>> Doesn't checkout.thresholdForParallelism only matter when
>> checkout.workers != 1?
>> 
>> So what you wrote seems like a reasonable explanation, but only if the
>> original reporter set checkout.workers to imply the non-sequential
>> behavior in the first place.
>> 
>> That said...
>> 
>>   - I also don't know off-hand of a place where we've defined the order
>>     where Git will checkout files in the working copy. So depending on
>>     that behavior isn't a safe thing to do.
>> 
>>   - Committing build artifacts into your repository is generally
>>     discouraged.
>
> If it's undefined and never implemented this is reasonable.
>
> But "generally" is a caveat, so while I agree with the statement it also 
> implies there's valid cases outside of that. Ones which used to work, too.
>
> Here are some useful cases I have seen for the combination of build rule + 
> checked in file:
>
> - part of a build requires licensed software that's not always available
>
> - part of the build requires large memory that other builders generally do 
>   not have available
>
> - part of the build process uses a different platform or some other system 
>   requirement
>
> - to fetch data eg. from a URL, with a record of the URL/automation but 
>   also a copy of the file as a record and for offline use
>
> So it's useful, to retain repeatable automation but not always build from 
> square one.
>
> Generally discouraged to check in build results yes, but I've found it 
> very practical.
>  
>> So while I'd guess that setting `checkout.workers` back to "1" (if it 
>> wasn't already) will probably restore the existing behavior, counting on 
>> that behavior in the first place is wrong.
>
> I think perhaps the tail is wagging the dog here, though.
>
> It's 'wrong' because it doesn't work; but I haven't seen anything to make 
> me think this is fundamentally or theoretically flawed.
>
> If we had a transactional file system we'd reasonably expect a checkout to 
> be an atomic operation -- same timestamp on the files created in that 
> step. A discrepancy in timestamps would be considered incorrect; it would 
> imply an 'order' to the checkout which, as you say, is order-less.
>
> Sowhat could be the bad outcomes if Git created files stamped with the 
> point in time of the "git checkout"?

I agree that it's practical in some scenarios, including checking in
built assets.

But those that are doing that need to be aware that combining that sort
of thing with source control tends to upend your build system's idea of
the world.

E.g. until recently in git.git we had a po/git.pot in-tree, which is a
"compiled file" (although a plain-text one) that was checked in, and
dealing with that in make's dependency graph was a (minor) pain
sometimes.

Re: Consist timestamps within a checkout/clone

[Mark Hills]

[-- Attachment #1: Type: text/plain, Size: 4582 bytes --]

On Mon, 31 Oct 2022, Ævar Arnfjörð Bjarmason wrote:

> 
> On Mon, Oct 31 2022, Mark Hills wrote:
> 
> > Our use case: we commit some compiled objects to the repo, where compiling 
> > is either slow or requires software which is not always available.
> >
> > Since upgrading Git 2.26.3 -> 2.32.4 (as part of Alpine Linux OS upgrade) 
> > we are noticing a change in build behaviour.
> >
> > Now, after a "git clone" we find the Makefile intermittently attempting 
> > (and failing) some builds that are not intended.
> >
> > Indeed, Make is acting reasonably as the source file is sometimes 
> > marginally newer than the destination (both checked out by Git), example 
> > below.
> >
> > I've never had to consider consistency timestamps within a Git checkout 
> > until now.
> >
> > It's entirely possible there's _never_ a guarantee of consistency here.
> >
> > But then something has certainly changed in practice, as this fault has 
> > gone from never happening to now every couple of days.
> >
> > Imaginging I can't be the first person to encounter this, I searched for 
> > existing threads or docs, but overwhemingly the results were question of 
> > Git tracking the timestamps (as part of the commit) which this is not; 
> > it's consistency within one checkout.
> >
> > $ git clone --depth 1 file:///path/to/repo.git
> >
> > $ stat winner.jpeg
> >   File: winner.jpeg
> >   Size: 258243          Blocks: 520        IO Block: 4096   regular file
> > Device: fd07h/64775d    Inode: 33696       Links: 1
> > Access: (0644/-rw-r--r--)  Uid: (  106/ luthier)   Gid: (  106/ luthier)
> > Access: 2022-10-31 16:05:17.756858496 +0000
> > Modify: 2022-10-31 16:05:17.756858496 +0000
> > Change: 2022-10-31 16:05:17.756858496 +0000
> >  Birth: -
> >
> > $ stat winner.svg
> >   File: winner.svg
> >   Size: 52685           Blocks: 112        IO Block: 4096   regular file
> > Device: fd07h/64775d    Inode: 33697       Links: 1
> > Access: (0644/-rw-r--r--)  Uid: (  106/ luthier)   Gid: (  106/ luthier)
> > Access: 2022-10-31 16:05:17.766859030 +0000
> > Modify: 2022-10-31 16:05:17.766859030 +0000
> > Change: 2022-10-31 16:05:17.766859030 +0000
> >  Birth: -
> >
> > Elsewhere in the repository, it's clear the timestamps are not consistent:
> >
> > $ stat Makefile
> >   File: Makefile
> >   Size: 8369            Blocks: 24         IO Block: 4096   regular file
> > Device: fd07h/64775d    Inode: 33655       Links: 1
> > Access: (0644/-rw-r--r--)  Uid: (  106/ luthier)   Gid: (  106/ luthier)
> > Access: 2022-10-31 16:05:51.628660212 +0000
> > Modify: 2022-10-31 16:05:17.746857963 +0000
> > Change: 2022-10-31 16:05:17.746857963 +0000
> >  Birth: -
> 
> I think you're almost certainly running into the parallel checkout,
> which is new in that revision range. Try tweaking checkout.workers and
> checkout.thresholdForParallelism (see "man git-config").

Thanks, it will be interesting to try this and I'll report back.
 
> I can't say without looking at the code/Makefile (and even then, I don't
> have time to dig here:), but if I had to bet I'd say that your
> dependencies have probably always been broken with these checked-in
> files, but they happend to work out if they were checked out in sorted
> order.
>
> And now with the parallel checkout they're not guaranteed to do that, as
> some workers will "race ahead" and finish in an unpredictable order.

These are very simple Makefile rules, I don't think these dependencies are 
broken; but your theory is in good alignment with the observed behaviour.

For example, the rule from the recent case above is:

  %.jpeg:         %.png
                  convert $< $(IMFLAGS) $@

  %.png:          %.svg
                  inkscape --export-type=png --export-filename=$@ $<

As you suggest, perhaps the Git implementation previously ran checked out 
in some kind of time order then this happens to fulfil a useful behaviour.

Specificaly with build artefacts. These are likely to have been added to 
the repo after the source file. This could have been providing some 
pratical and useful tendency of ordering.

> But that's all just a guess, perhaps it has nothing to do with parallel
> checkout, such dependency issues are sensitive to all sorts of other
> things, e.g. maybe git got slightly faster (or slower), so now files
> that were always on different seconds (or the same) aren't in the state
> they were in before...

Hopefully I'll get to some experiments to narrow this down.

Thanks

-- 
Mark

Re: Consist timestamps within a checkout/clone

[Ævar Arnfjörð Bjarmason]

On Mon, Oct 31 2022, Mark Hills wrote:

> On Mon, 31 Oct 2022, Ævar Arnfjörð Bjarmason wrote:
>
>> 
>> On Mon, Oct 31 2022, Mark Hills wrote:
>> 
>> > Our use case: we commit some compiled objects to the repo, where compiling 
>> > is either slow or requires software which is not always available.
>> >
>> > Since upgrading Git 2.26.3 -> 2.32.4 (as part of Alpine Linux OS upgrade) 
>> > we are noticing a change in build behaviour.
>> >
>> > Now, after a "git clone" we find the Makefile intermittently attempting 
>> > (and failing) some builds that are not intended.
>> >
>> > Indeed, Make is acting reasonably as the source file is sometimes 
>> > marginally newer than the destination (both checked out by Git), example 
>> > below.
>> >
>> > I've never had to consider consistency timestamps within a Git checkout 
>> > until now.
>> >
>> > It's entirely possible there's _never_ a guarantee of consistency here.
>> >
>> > But then something has certainly changed in practice, as this fault has 
>> > gone from never happening to now every couple of days.
>> >
>> > Imaginging I can't be the first person to encounter this, I searched for 
>> > existing threads or docs, but overwhemingly the results were question of 
>> > Git tracking the timestamps (as part of the commit) which this is not; 
>> > it's consistency within one checkout.
>> >
>> > $ git clone --depth 1 file:///path/to/repo.git
>> >
>> > $ stat winner.jpeg
>> >   File: winner.jpeg
>> >   Size: 258243          Blocks: 520        IO Block: 4096   regular file
>> > Device: fd07h/64775d    Inode: 33696       Links: 1
>> > Access: (0644/-rw-r--r--)  Uid: (  106/ luthier)   Gid: (  106/ luthier)
>> > Access: 2022-10-31 16:05:17.756858496 +0000
>> > Modify: 2022-10-31 16:05:17.756858496 +0000
>> > Change: 2022-10-31 16:05:17.756858496 +0000
>> >  Birth: -
>> >
>> > $ stat winner.svg
>> >   File: winner.svg
>> >   Size: 52685           Blocks: 112        IO Block: 4096   regular file
>> > Device: fd07h/64775d    Inode: 33697       Links: 1
>> > Access: (0644/-rw-r--r--)  Uid: (  106/ luthier)   Gid: (  106/ luthier)
>> > Access: 2022-10-31 16:05:17.766859030 +0000
>> > Modify: 2022-10-31 16:05:17.766859030 +0000
>> > Change: 2022-10-31 16:05:17.766859030 +0000
>> >  Birth: -
>> >
>> > Elsewhere in the repository, it's clear the timestamps are not consistent:
>> >
>> > $ stat Makefile
>> >   File: Makefile
>> >   Size: 8369            Blocks: 24         IO Block: 4096   regular file
>> > Device: fd07h/64775d    Inode: 33655       Links: 1
>> > Access: (0644/-rw-r--r--)  Uid: (  106/ luthier)   Gid: (  106/ luthier)
>> > Access: 2022-10-31 16:05:51.628660212 +0000
>> > Modify: 2022-10-31 16:05:17.746857963 +0000
>> > Change: 2022-10-31 16:05:17.746857963 +0000
>> >  Birth: -
>> 
>> I think you're almost certainly running into the parallel checkout,
>> which is new in that revision range. Try tweaking checkout.workers and
>> checkout.thresholdForParallelism (see "man git-config").
>
> Thanks, it will be interesting to try this and I'll report back.

FWIW I was under the impression that we'd made it the default, so unless
you opted-in it's probably not that.

>> I can't say without looking at the code/Makefile (and even then, I don't
>> have time to dig here:), but if I had to bet I'd say that your
>> dependencies have probably always been broken with these checked-in
>> files, but they happend to work out if they were checked out in sorted
>> order.
>>
>> And now with the parallel checkout they're not guaranteed to do that, as
>> some workers will "race ahead" and finish in an unpredictable order.
>
> These are very simple Makefile rules, I don't think these dependencies are 
> broken; but your theory is in good alignment with the observed behaviour.
>
> For example, the rule from the recent case above is:
>
>   %.jpeg:         %.png
>                   convert $< $(IMFLAGS) $@
>
>   %.png:          %.svg
>                   inkscape --export-type=png --export-filename=$@ $<

Grom a glance those don't seem broken to me, but I don't know how it
interacts with your built assets.

So e.g. if you are checking in your *.jpeg files those will be more
recent than either the *.png or source *.svn, so they won't be built.

This is fast getting out of scope of Git-specific advice, but you should
run "make --debug" (there's also sub-debug flags) to see if make's idea
of the dependency graph matches yours.

Re: Consist timestamps within a checkout/clone

[Matheus Tavares]

> On Mon, Oct 31 2022, Mark Hills wrote:
>
> > On Mon, 31 Oct 2022, Ævar Arnfjörð Bjarmason wrote:
> >
> >>
> >> On Mon, Oct 31 2022, Mark Hills wrote:
> >>
> >> > Our use case: we commit some compiled objects to the repo, where compiling
> >> > is either slow or requires software which is not always available.
> >> >
> >> > Since upgrading Git 2.26.3 -> 2.32.4 (as part of Alpine Linux OS upgrade)
> >> > we are noticing a change in build behaviour.
> >> >
> >> > Now, after a "git clone" we find the Makefile intermittently attempting
> >> > (and failing) some builds that are not intended.
> >> >
> >> > Indeed, Make is acting reasonably as the source file is sometimes
> >> > marginally newer than the destination (both checked out by Git), example
> >> > below.
> >> >
> >> > I've never had to consider consistency timestamps within a Git checkout
> >> > until now.
> >> >
> >> > It's entirely possible there's _never_ a guarantee of consistency here.
> >> >
> >> > But then something has certainly changed in practice, as this fault has
> >> > gone from never happening to now every couple of days.
> >> >
> >> > Imaginging I can't be the first person to encounter this, I searched for
> >> > existing threads or docs, but overwhemingly the results were question of
> >> > Git tracking the timestamps (as part of the commit) which this is not;
> >> > it's consistency within one checkout.
> >> >
> >> > $ git clone --depth 1 file:///path/to/repo.git
> >> >
> >> > $ stat winner.jpeg
> >> >   File: winner.jpeg
> >> >   Size: 258243          Blocks: 520        IO Block: 4096   regular file
> >> > Device: fd07h/64775d    Inode: 33696       Links: 1
> >> > Access: (0644/-rw-r--r--)  Uid: (  106/ luthier)   Gid: (  106/ luthier)
> >> > Access: 2022-10-31 16:05:17.756858496 +0000
> >> > Modify: 2022-10-31 16:05:17.756858496 +0000
> >> > Change: 2022-10-31 16:05:17.756858496 +0000
> >> >  Birth: -
> >> >
> >> > $ stat winner.svg
> >> >   File: winner.svg
> >> >   Size: 52685           Blocks: 112        IO Block: 4096   regular file
> >> > Device: fd07h/64775d    Inode: 33697       Links: 1
> >> > Access: (0644/-rw-r--r--)  Uid: (  106/ luthier)   Gid: (  106/ luthier)
> >> > Access: 2022-10-31 16:05:17.766859030 +0000
> >> > Modify: 2022-10-31 16:05:17.766859030 +0000
> >> > Change: 2022-10-31 16:05:17.766859030 +0000
> >> >  Birth: -
> >> >
> >> > Elsewhere in the repository, it's clear the timestamps are not consistent:
> >> >
> >> > $ stat Makefile
> >> >   File: Makefile
> >> >   Size: 8369            Blocks: 24         IO Block: 4096   regular file
> >> > Device: fd07h/64775d    Inode: 33655       Links: 1
> >> > Access: (0644/-rw-r--r--)  Uid: (  106/ luthier)   Gid: (  106/ luthier)
> >> > Access: 2022-10-31 16:05:51.628660212 +0000
> >> > Modify: 2022-10-31 16:05:17.746857963 +0000
> >> > Change: 2022-10-31 16:05:17.746857963 +0000
> >> >  Birth: -
> >>
> >> I think you're almost certainly running into the parallel checkout,
> >> which is new in that revision range. Try tweaking checkout.workers and
> >> checkout.thresholdForParallelism (see "man git-config").

This does look like something you would see with parallel checkout, yes.
But...

> > Thanks, it will be interesting to try this and I'll report back.
>
> FWIW I was under the impression that we'd made it the default, so unless
> you opted-in it's probably not that.

... it indeed should be disabled by default. It seems Mark didn't
manually enable parallel checkout, as the original message only mentions
the git upgrade as a changing factor. And Alpine's git installation
script for 2.32.4 [1] doesn't seem to change our defaults either.

Perhaps, it just happens that 2.32.4 changed the checkout processing
time slightly so that each entry is finished a bit slower (or the system
was overloaded at that moment?). Anyways, the creation order (based on
the mtimes) looks correct to me from a sequential-checkout point of
view: first Makefile, than winner.jpeg, and finally winner.svg. That's
the order in which these files would appear in the index, which is the
order followed by sequential checkout.

[1]: https://git.alpinelinux.org/aports/tree/main/git/APKBUILD?h=3.14-stable&id=0f3285f2cfcb8362460002c27e219fadbf18c885

Re: Consist timestamps within a checkout/clone

[Matheus Tavares]

[Oops, I accidentally sent this from my business account. I'm
quote-replying it now from my correct personal account just in case
the original falls under spam folders for "spoofing".]

On Wed, Nov 2, 2022 at 11:16 AM Matheus Tavares
<matheus.bernardino@usp.br> wrote:
>
> > On Mon, Oct 31 2022, Mark Hills wrote:
> >
> > > On Mon, 31 Oct 2022, Ævar Arnfjörð Bjarmason wrote:
> > >
> > >>
> > >> On Mon, Oct 31 2022, Mark Hills wrote:
> > >>
> > >> > Our use case: we commit some compiled objects to the repo, where compiling
> > >> > is either slow or requires software which is not always available.
> > >> >
> > >> > Since upgrading Git 2.26.3 -> 2.32.4 (as part of Alpine Linux OS upgrade)
> > >> > we are noticing a change in build behaviour.
> > >> >
> > >> > Now, after a "git clone" we find the Makefile intermittently attempting
> > >> > (and failing) some builds that are not intended.
> > >> >
> > >> > Indeed, Make is acting reasonably as the source file is sometimes
> > >> > marginally newer than the destination (both checked out by Git), example
> > >> > below.
> > >> >
> > >> > I've never had to consider consistency timestamps within a Git checkout
> > >> > until now.
> > >> >
> > >> > It's entirely possible there's _never_ a guarantee of consistency here.
> > >> >
> > >> > But then something has certainly changed in practice, as this fault has
> > >> > gone from never happening to now every couple of days.
> > >> >
> > >> > Imaginging I can't be the first person to encounter this, I searched for
> > >> > existing threads or docs, but overwhemingly the results were question of
> > >> > Git tracking the timestamps (as part of the commit) which this is not;
> > >> > it's consistency within one checkout.
> > >> >
> > >> > $ git clone --depth 1 file:///path/to/repo.git
> > >> >
> > >> > $ stat winner.jpeg
> > >> >   File: winner.jpeg
> > >> >   Size: 258243          Blocks: 520        IO Block: 4096   regular file
> > >> > Device: fd07h/64775d    Inode: 33696       Links: 1
> > >> > Access: (0644/-rw-r--r--)  Uid: (  106/ luthier)   Gid: (  106/ luthier)
> > >> > Access: 2022-10-31 16:05:17.756858496 +0000
> > >> > Modify: 2022-10-31 16:05:17.756858496 +0000
> > >> > Change: 2022-10-31 16:05:17.756858496 +0000
> > >> >  Birth: -
> > >> >
> > >> > $ stat winner.svg
> > >> >   File: winner.svg
> > >> >   Size: 52685           Blocks: 112        IO Block: 4096   regular file
> > >> > Device: fd07h/64775d    Inode: 33697       Links: 1
> > >> > Access: (0644/-rw-r--r--)  Uid: (  106/ luthier)   Gid: (  106/ luthier)
> > >> > Access: 2022-10-31 16:05:17.766859030 +0000
> > >> > Modify: 2022-10-31 16:05:17.766859030 +0000
> > >> > Change: 2022-10-31 16:05:17.766859030 +0000
> > >> >  Birth: -
> > >> >
> > >> > Elsewhere in the repository, it's clear the timestamps are not consistent:
> > >> >
> > >> > $ stat Makefile
> > >> >   File: Makefile
> > >> >   Size: 8369            Blocks: 24         IO Block: 4096   regular file
> > >> > Device: fd07h/64775d    Inode: 33655       Links: 1
> > >> > Access: (0644/-rw-r--r--)  Uid: (  106/ luthier)   Gid: (  106/ luthier)
> > >> > Access: 2022-10-31 16:05:51.628660212 +0000
> > >> > Modify: 2022-10-31 16:05:17.746857963 +0000
> > >> > Change: 2022-10-31 16:05:17.746857963 +0000
> > >> >  Birth: -
> > >>
> > >> I think you're almost certainly running into the parallel checkout,
> > >> which is new in that revision range. Try tweaking checkout.workers and
> > >> checkout.thresholdForParallelism (see "man git-config").
>
> This does look like something you would see with parallel checkout, yes.
> But...
>
> > > Thanks, it will be interesting to try this and I'll report back.
> >
> > FWIW I was under the impression that we'd made it the default, so unless
> > you opted-in it's probably not that.
>
> ... it indeed should be disabled by default. It seems Mark didn't
> manually enable parallel checkout, as the original message only mentions
> the git upgrade as a changing factor. And Alpine's git installation
> script for 2.32.4 [1] doesn't seem to change our defaults either.
>
> Perhaps, it just happens that 2.32.4 changed the checkout processing
> time slightly so that each entry is finished a bit slower (or the system
> was overloaded at that moment?). Anyways, the creation order (based on
> the mtimes) looks correct to me from a sequential-checkout point of
> view: first Makefile, than winner.jpeg, and finally winner.svg. That's
> the order in which these files would appear in the index, which is the
> order followed by sequential checkout.
>
> [1]: https://git.alpinelinux.org/aports/tree/main/git/APKBUILD?h=3.14-stable&id=0f3285f2cfcb8362460002c27e219fadbf18c885

Re: Consist timestamps within a checkout/clone

[Marc Branchaud]

On 2022-10-31 15:01, Mark Hills wrote:
> Our use case: we commit some compiled objects to the repo, where compiling
> is either slow or requires software which is not always available.
> 
> Since upgrading Git 2.26.3 -> 2.32.4 (as part of Alpine Linux OS upgrade)
> we are noticing a change in build behaviour.
> 
> Now, after a "git clone" we find the Makefile intermittently attempting
> (and failing) some builds that are not intended.
> 
> Indeed, Make is acting reasonably as the source file is sometimes
> marginally newer than the destination (both checked out by Git), example
> below.

A fix for this was proposed in 2018 and dismissed [1].

Back then, the problem was that as Git wrote files into a directory 
sometimes the clock would tick over at a bad time, and we'd end up with 
some files being "newer" than others.  This would sour Make runs as you 
describe.

Nominally this is caused by putting generated files in the repo, but 
many times that is unavoidable (e.g. you're forking an upstream that 
puts automake-generated stuff in the repo).

IMHO, dismissing the problem back then was a mistake.  At the time I 
advocated teaching Git to give all the files it touches (creates or 
modifies) in a directory the same mtime (e.g. the time at the start of 
the checkout operation).

Instead the decision was to do nothing in Git, and instead let people 
create their own post-checkout hooks to touch the files.  I (and others) 
argued this was inadequate, to no avail.

		M.

[1] https://public-inbox.org/git/20180413170129.15310-1-mgorny@gentoo.org/#r

Re: Consist timestamps within a checkout/clone

[Ævar Arnfjörð Bjarmason]

On Tue, Nov 01 2022, Marc Branchaud wrote:

> On 2022-10-31 15:01, Mark Hills wrote:
>> Our use case: we commit some compiled objects to the repo, where compiling
>> is either slow or requires software which is not always available.
>> Since upgrading Git 2.26.3 -> 2.32.4 (as part of Alpine Linux OS
>> upgrade)
>> we are noticing a change in build behaviour.
>> Now, after a "git clone" we find the Makefile intermittently
>> attempting
>> (and failing) some builds that are not intended.
>> Indeed, Make is acting reasonably as the source file is sometimes
>> marginally newer than the destination (both checked out by Git), example
>> below.
>
> A fix for this was proposed in 2018 and dismissed [1].
>
> Back then, the problem was that as Git wrote files into a directory
> sometimes the clock would tick over at a bad time, and we'd end up
> with some files being "newer" than others.  This would sour Make runs
> as you describe.
>
> Nominally this is caused by putting generated files in the repo, but
> many times that is unavoidable (e.g. you're forking an upstream that 
> puts automake-generated stuff in the repo).
>
> IMHO, dismissing the problem back then was a mistake.  At the time I
> advocated teaching Git to give all the files it touches (creates or 
> modifies) in a directory the same mtime (e.g. the time at the start of
> the checkout operation).
>
> Instead the decision was to do nothing in Git, and instead let people
> create their own post-checkout hooks to touch the files.  I (and
> others) argued this was inadequate, to no avail.
>
> 		M.
>
> [1] https://public-inbox.org/git/20180413170129.15310-1-mgorny@gentoo.org/#r

I think that's the wrong take-away from that thread. Maybe a patch for
this will get rejected in the end, but in that case it wasn't because
the git project is never going to take a patch like this.

Maybe it won't, but:

 * That commit has no tests
 * It's clearly controversial behavior, so *if* we add it I think it's
   better to make it opt-in configurable.
 * Once that's done, you'd need doc changes etc. for that.

Now, maybe a sufficiently polished version would also be "meh" for
whatever reason, I just think it's premature to say that a change in
this direction would never be accepted.

That being said, I do wonder if software in the wild is being
monkeypatched to work around issues with make (or make-like tools)
whether such a change isn't better advocated in e.g. GNU make itself.

If it added "B" to "MAKEFLAGS" if it detected:

 * I'm in a git repository
 * It's the first time I'm running here, or "nothing is built yet"
 * My dependency graph would be different with "-B"

Wouldn't that be what people who want this feature are after?

It's not like it's SCM-agnostic, it already goes to significant trouble
to cater to RCS and SCCS of all things, so I don't see why they'd
categorically reject a patch to cater to modern VCS's.

And, unlike Gike, GNU make wouldn't need to guess that munging
timestamps would fix it, it can compute both versions of the dependency
graph, so it would know...

Re: Consist timestamps within a checkout/clone

[Marc Branchaud]

On 2022-11-02 10:45, Ævar Arnfjörð Bjarmason wrote:
> 
> On Tue, Nov 01 2022, Marc Branchaud wrote:
>>
>> Instead the decision was to do nothing in Git, and instead let people
>> create their own post-checkout hooks to touch the files.  I (and others)
>> argued this was inadequate, to no avail.
>> >> [1] 
https://public-inbox.org/git/20180413170129.15310-1-mgorny@gentoo.org/#r
> 
> I think that's the wrong take-away from that thread. Maybe a patch for
> this will get rejected in the end, but in that case it wasn't because
> the git project is never going to take a patch like this.
> 
> Maybe it won't, but:
> 
>   * That commit has no tests
>   * It's clearly controversial behavior, so *if* we add it I think it's
>     better to make it opt-in configurable.
>   * Once that's done, you'd need doc changes etc. for that.
> 
> Now, maybe a sufficiently polished version would also be "meh" for
> whatever reason, I just think it's premature to say that a change in
> this direction would never be accepted.

I did not say that it would never be accepted; perhaps I should have 
said "outcome" instead of "decision".  That 2018 thread barely discussed 
changes to the patch itself.  The patch's writer (not me) didn't pursue 
the work, after its frosty initial reception.

Past discussions around these proposals have been negative, which 
discourages people from polishing a submission as you suggest.  I hope 
that this time is different.  (Before you suggest I submit a patch, I 
sadly don't have the time to hack on Git these days.)

> That being said, I do wonder if software in the wild is being
> monkeypatched to work around issues with make (or make-like tools)
> whether such a change isn't better advocated in e.g. GNU make itself.
> 
> If it added "B" to "MAKEFLAGS" if it detected:
> 
>   * I'm in a git repository
>   * It's the first time I'm running here, or "nothing is built yet"
>   * My dependency graph would be different with "-B"
> 
> Wouldn't that be what people who want this feature are after?
> 
> It's not like it's SCM-agnostic, it already goes to significant trouble
> to cater to RCS and SCCS of all things, so I don't see why they'd
> categorically reject a patch to cater to modern VCS's.
> 
> And, unlike Gike, GNU make wouldn't need to guess that munging
> timestamps would fix it, it can compute both versions of the dependency
> graph, so it would know...

Fair points about advocating for changes in make.  However, Gnu make 
isn't the only flavour out there.  Our builds use both BSD's and Gnu's 
makes, for example.  (Also, BSD make has a completely different 
interpretation of -B, and does not have any flag that mirrors Gnu make's 
-B.)

Git is really the ideal place to solve this problem, instead of playing 
whack-a-mole with build tools and upstream projects.  Making the 
behaviour opt-in is perfectly reasonable.

		M.

Re: Consist timestamps within a checkout/clone

[Erik Cervin Edin]

I have little to add on the underlying issue or non-issue but some
ideas on how to solve your problem

On Mon, Oct 31, 2022 at 8:39 PM Mark Hills <mark@xwax.org> wrote:
>
> ...
> Indeed, Make is acting reasonably as the source file is sometimes
> marginally newer than the destination (both checked out by Git), example
> below.
>
> I've never had to consider consistency timestamps within a Git checkout
> until now.
>
> It's entirely possible there's _never_ a guarantee of consistency here.

If your makefile depends on checkout, why not
  git ls-files | xargs touch
or if this done in an environment where there's not a fresh clone each
time, maybe
  git diff HEAD --name-only --diff-filter=AM | xargs touch
or something along those lines

Re: Consist timestamps within a checkout/clone

[Ævar Arnfjörð Bjarmason]

On Tue, Nov 01 2022, Erik Cervin Edin wrote:

> I have little to add on the underlying issue or non-issue but some
> ideas on how to solve your problem
>
> On Mon, Oct 31, 2022 at 8:39 PM Mark Hills <mark@xwax.org> wrote:
>>
>> ...
>> Indeed, Make is acting reasonably as the source file is sometimes
>> marginally newer than the destination (both checked out by Git), example
>> below.
>>
>> I've never had to consider consistency timestamps within a Git checkout
>> until now.
>>
>> It's entirely possible there's _never_ a guarantee of consistency here.
>
> If your makefile depends on checkout, why not
>   git ls-files | xargs touch
> or if this done in an environment where there's not a fresh clone each
> time, maybe
>   git diff HEAD --name-only --diff-filter=AM | xargs touch
> or something along those lines

I believe you might be trying to re-invent "make -B" :)

Re: Consist timestamps within a checkout/clone

[Erik Cervin Edin]

On Tue, Nov 1, 2022 at 4:53 PM Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:
>
> I believe you might be trying to re-invent "make -B" :)

True, in the simple case, but if you
  git diff HEAD --name-only --diff-filter=AM | xargs touch
that should consolidate the modified times on disk of the files of that commit

It needs a bit more work, something like
  pre_checkout=$(git rev-parse HEAD)
  git checkout XYX &&
  git diff pre_checkout...XYZ --name-only --diff-filter=AM | xargs touch
but something like that can work around the inconsistent ordered
modified times after a checkout