(bug?) Inconsistent workdir file timestamps after initial clone.

(bug?) Inconsistent workdir file timestamps after initial clone.

[Marc Branchaud]

Hi all,

Occasionally when doing a fresh clone of a repo, if the clock ticks at just
the wrong time the checked-out files end up with different timestamps.

The effect of this can be that, when "make" is run in the workdir it'll
decide that some files are out of date and try to rebuild them.

(In our particular case, our automated build-bot cloned a submodule of some
third-party (i.e. not our) code, where a Makefile.in got an earlier timestamp
than its dependent Makefile.am, so "configure && make" then tried to rebuild
Makefile.in and the build failed because our build environment has the wrong
version of automake.)

I'm completely unfamiliar with the clone-and-checkout parts of git's code, so
my first question really is if someone more familiar with the code could look
at it (or at least point me to it) to verify whether or not such inconsistent
timestamps are possible.

If someone can please confirm that timestamps will always be consistent on
the initial checkout of a clone, then I'll have to hunt for a different cause
of our build failure.

However, if inconsistent timestamps are possible, I'd like to suggest that
this should be fixed.  (I'd learn the code and write a patch myself, but as
some of you may know I haven't had very much time for git hacking lately.)

Thanks!

		M.

Re: (bug?) Inconsistent workdir file timestamps after initial clone.

[Junio C Hamano]

Marc Branchaud <marcnarc@xiplink.com> writes:

> Occasionally when doing a fresh clone of a repo, if the clock ticks at just
> the wrong time the checked-out files end up with different timestamps.
>
> The effect of this can be that, when "make" is run in the workdir it'll
> decide that some files are out of date and try to rebuild them.
>
> (In our particular case, our automated build-bot cloned a submodule of some
> third-party (i.e. not our) code, where a Makefile.in got an earlier timestamp
> than its dependent Makefile.am, so "configure && make" then tried to rebuild
> Makefile.in and the build failed because our build environment has the wrong
> version of automake.)

Even if you somehow arrange Makefile.in and Makefile.am to have the
same timestamp, wouldn't it be up to your "make" to decide which one
is newer?  Certainly Makefile.in is not newer than Makefile.am, and
it is free to try rebuilding it.

Also if you do this after any operation:

    $ rm Makefile.am
    $ git checkout Makefile.am

you will have Makefile.am that is newer than your Makefile.in and
you will end up attempting to rebuild it.

The timestamp of a working tree file records the time at which it
was created in your working tree.  It does not have any relation to
the commit or author timestamp of the commit you check it out of.
If this command:

    $ git checkout @{1.dacade.ago} Makefile.am

gave your Makefile.am an ancient timestamp, it will break your
build.

While not including files that can be rebuilt from the source may be
the ideal solution, I've seen projects hide rules to rebuild such a
"generated but needs special tools to build" and/or a "generated but
normal developers do not have any business rebuilding" file (in your
case, Makefile.in) in their Makefiles from the normal targets (like
"make all") for this exact reason, when they choose to distribute
such files by including in their commits.

Re: (bug?) Inconsistent workdir file timestamps after initial clone.

[Marc Branchaud]

On 12-12-11 04:27 PM, Junio C Hamano wrote:
> Marc Branchaud <marcnarc@xiplink.com> writes:
> 
>> Occasionally when doing a fresh clone of a repo, if the clock ticks at just
>> the wrong time the checked-out files end up with different timestamps.
>>
>> The effect of this can be that, when "make" is run in the workdir it'll
>> decide that some files are out of date and try to rebuild them.
>>
>> (In our particular case, our automated build-bot cloned a submodule of some
>> third-party (i.e. not our) code, where a Makefile.in got an earlier timestamp
>> than its dependent Makefile.am, so "configure && make" then tried to rebuild
>> Makefile.in and the build failed because our build environment has the wrong
>> version of automake.)
> 
> Even if you somehow arrange Makefile.in and Makefile.am to have the
> same timestamp, wouldn't it be up to your "make" to decide which one
> is newer?  Certainly Makefile.in is not newer than Makefile.am, and
> it is free to try rebuilding it.

Well, the makes I've used don't rebuild anything after a "touch *".  I think
it would surprise a lot of people if their make did rebuild files when their
timestamps matched.

> Also if you do this after any operation:
> 
>     $ rm Makefile.am
>     $ git checkout Makefile.am
> 
> you will have Makefile.am that is newer than your Makefile.in and
> you will end up attempting to rebuild it.

Yes, of course.  I would never expect otherwise.

> The timestamp of a working tree file records the time at which it
> was created in your working tree.  It does not have any relation to
> the commit or author timestamp of the commit you check it out of.
> If this command:
> 
>     $ git checkout @{1.dacade.ago} Makefile.am
> 
> gave your Makefile.am an ancient timestamp, it will break your
> build.

Yes, I agree.

My point is that the initial checkout into an empty working directory should
create all files with the same timestamp.

Or, to be a bit more precise, whenever git-checkout *creates* files in the
work dir, *all* the created files should have the *same* timestamp (i.e. the
current time measured at the start of the checkout's execution, not some
bizarro other time specified by some arcane heuristic).

The more I think about it, the more I think it's sloppy for git-checkout to
just let the filesystem assign the exact current time to created files.  A
checkout theoretically should be atomic -- you really shouldn't try to play
with any of the files in your workdir while a checkout is underway.  It's
impractical to really make checkouts atomic, but I think the end result of a
checkout should as much as possible look like the checkout happened all at
one time.

> While not including files that can be rebuilt from the source may be
> the ideal solution, I've seen projects hide rules to rebuild such a
> "generated but needs special tools to build" and/or a "generated but
> normal developers do not have any business rebuilding" file (in your
> case, Makefile.in) in their Makefiles from the normal targets (like
> "make all") for this exact reason, when they choose to distribute
> such files by including in their commits.

I prefer to use the third-party code as-is, without hacking it, to have
smooth upgrades in the future.

		M.

Re: (bug?) Inconsistent workdir file timestamps after initial clone.

[Junio C Hamano]

Marc Branchaud <marcnarc@xiplink.com> writes:

> My point is that the initial checkout into an empty working directory should
> create all files with the same timestamp.
>
> Or, to be a bit more precise, whenever git-checkout *creates* files in the
> work dir, *all* the created files should have the *same* timestamp (i.e. the
> current time measured at the start of the checkout's execution, not some
> bizarro other time specified by some arcane heuristic).

My knee-jerk reaction is that it is insane to do so, but what other
SCM does such a thing? Even "tar xf" wouldn't do that, I think.

>> While not including files that can be rebuilt from the source may be
>> the ideal solution, I've seen projects hide rules to rebuild such a
>> "generated but needs special tools to build" and/or a "generated but
>> normal developers do not have any business rebuilding" file (in your
>> case, Makefile.in) in their Makefiles from the normal targets (like
>> "make all") for this exact reason, when they choose to distribute
>> such files by including in their commits.
>
> I prefer to use the third-party code as-is, without hacking it, to have
> smooth upgrades in the future.

Then perhaps take the complaints to that third-party upstream, not
here?

Re: (bug?) Inconsistent workdir file timestamps after initial clone.

[Marc Branchaud]

On 12-12-11 05:30 PM, Junio C Hamano wrote:
> Marc Branchaud <marcnarc@xiplink.com> writes:
> 
>> My point is that the initial checkout into an empty working directory should
>> create all files with the same timestamp.
>>
>> Or, to be a bit more precise, whenever git-checkout *creates* files in the
>> work dir, *all* the created files should have the *same* timestamp (i.e. the
>> current time measured at the start of the checkout's execution, not some
>> bizarro other time specified by some arcane heuristic).
> 
> My knee-jerk reaction is that it is insane to do so, but what other
> SCM does such a thing?

I'm lucky enough to just care about git these days.

> Even "tar xf" wouldn't do that, I think.

"tar xf" uses the timestamps that are stored in the tar file.  I see this as
an argument against git's exact-current-time-per-file approach: even the tar
guys understand that it's insane.

>>> While not including files that can be rebuilt from the source may be
>>> the ideal solution, I've seen projects hide rules to rebuild such a
>>> "generated but needs special tools to build" and/or a "generated but
>>> normal developers do not have any business rebuilding" file (in your
>>> case, Makefile.in) in their Makefiles from the normal targets (like
>>> "make all") for this exact reason, when they choose to distribute
>>> such files by including in their commits.
>>
>> I prefer to use the third-party code as-is, without hacking it, to have
>> smooth upgrades in the future.
> 
> Then perhaps take the complaints to that third-party upstream, not
> here?

Well, I thought that while I wait for some dozen-or-so projects to accept
changes to their builds, it might be nice for git to solve this problem for
me.  It is, after all, an effect of the way git operates.

		M.

Re: (bug?) Inconsistent workdir file timestamps after initial clone.

[Torsten Bögershausen]

 
On 11.12.12 23:30, Junio C Hamano wrote:
> Marc Branchaud <marcnarc@xiplink.com> writes:
> 
>> My point is that the initial checkout into an empty working directory should
>> create all files with the same timestamp.
>>
>> Or, to be a bit more precise, whenever git-checkout *creates* files in the
>> work dir, *all* the created files should have the *same* timestamp (i.e. the
>> current time measured at the start of the checkout's execution, not some
>> bizarro other time specified by some arcane heuristic).
> 
> My knee-jerk reaction is that it is insane to do so, but what other
> SCM does such a thing? Even "tar xf" wouldn't do that, I think.
> 


ClearCase is doing such a thing.

You need to check out a file to make it writable:
"cleartool checkout main.c"
[hack hack]
If you after some hacking don't like your changes at all,
you run 
"cleartool unco main.c" (Undo checkout)
(In git we just use "git checkout")

While in ClearCase the timestamp of your file jumps back to where
it was before the checkout, it gets the current timestamp in git.

One consequence is that ClearCase users may wish to use "ClearMake"
rather then make.

A better make (which records all timestamps somewhere) would be helpful.

>>> While not including files that can be rebuilt from the source may be
>>> the ideal solution, I've seen projects hide rules to rebuild such a
>>> "generated but needs special tools to build" and/or a "generated but
>>> normal developers do not have any business rebuilding" file (in your
>>> case, Makefile.in) in their Makefiles from the normal targets (like
>>> "make all") for this exact reason, when they choose to distribute
>>> such files by including in their commits.
>>
>> I prefer to use the third-party code as-is, without hacking it, to have
>> smooth upgrades in the future.
> 
> Then perhaps take the complaints to that third-party upstream, not
> here?
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

RE: (bug?) Inconsistent workdir file timestamps after initial clone.

[Pyeron, Jason J CTR (US)]

[-- Attachment #1: Type: text/plain, Size: 3051 bytes --]

> -----Original Message-----
> From: Torsten Bögershausen
> Sent: Wednesday, December 12, 2012 12:19 PM
> 
> 
> 
> On 11.12.12 23:30, Junio C Hamano wrote:
> > Marc Branchaud <marcnarc@xiplink.com> writes:
> >
> >> My point is that the initial checkout into an empty working
> directory should
> >> create all files with the same timestamp.
> >>
> >> Or, to be a bit more precise, whenever git-checkout *creates* files
> in the
> >> work dir, *all* the created files should have the *same* timestamp
> (i.e. the
> >> current time measured at the start of the checkout's execution, not
> some
> >> bizarro other time specified by some arcane heuristic).
> >
> > My knee-jerk reaction is that it is insane to do so, but what other
> > SCM does such a thing? Even "tar xf" wouldn't do that, I think.
> >
> 
> 
> ClearCase is doing such a thing.
> 
> You need to check out a file to make it writable:
> "cleartool checkout main.c"
> [hack hack]
> If you after some hacking don't like your changes at all,
> you run
> "cleartool unco main.c" (Undo checkout)
> (In git we just use "git checkout")
> 
> While in ClearCase the timestamp of your file jumps back to where
> it was before the checkout, it gets the current timestamp in git.

I do think that a user preference should decide if git uses metadata timestamps or current timestamp on file operations. I don’t think that the current time is proper as a default operation.

> 
> One consequence is that ClearCase users may wish to use "ClearMake"
> rather then make.
> 
> A better make (which records all timestamps somewhere) would be
> helpful.

That is why I always do "make clean" after a rollback. Do you really expect build managers to handle a bi-directional, with regards to time, SDLC? Build managers have worked hard to ensure incremental builds, it is silly to think that they should be re worked.


> 
> >>> While not including files that can be rebuilt from the source may
> be
> >>> the ideal solution, I've seen projects hide rules to rebuild such a
> >>> "generated but needs special tools to build" and/or a "generated
> but
> >>> normal developers do not have any business rebuilding" file (in
> your
> >>> case, Makefile.in) in their Makefiles from the normal targets (like
> >>> "make all") for this exact reason, when they choose to distribute
> >>> such files by including in their commits.
> >>
> >> I prefer to use the third-party code as-is, without hacking it, to
> have
> >> smooth upgrades in the future.
> >
> > Then perhaps take the complaints to that third-party upstream, not
> > here?
> > --
> > To unsubscribe from this list: send the line "unsubscribe git" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >
> 
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5615 bytes --]