Implementing $Date$ substitution - problem with git status

Implementing $Date$ substitution - problem with git status

[Peter Krefting]

Hi!

I finally got around to moving my web site repository over from CVS to Git, 
and to not have to redo everything about how it is set up, I set up a filter 
that fakes CVS' handling of "$Date$" keywords. Since I cannot access any 
information about the file from the "smudge" filter, I set up a "clean" 
filter that edits the $Date$ token on "git add":

   #!/usr/bin/perl -wTp -i.bak
   if (/\$Date:?[^\$]*\$/)
   {
     my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime();
     my $now = sprintf("%04d-%02d-%02d %02d:%02d:%02d", $year + 1900,
                       $mon + 1, $mday, $hour, $min, $sec);
     s/\$Date:?[^\$]*\$/\$Date: ${now} \$/;
   }

In .gitattributes I have lines like:

   *.html filter=date crlf=true ident

and filter.date.clean is set to point to the script above in .git/config.

This works fine, except that I have some problems with files that I cannot 
get to be "clean" in git status. "git diff" shows only a difference in the 
"$Date$" token, which is different each time I run "git diff" (the "+" line 
shows the current time):

   diff --git a/.htaccess b/.htaccess
   index 2dfbc32..d8ac343 100644
   --- a/.htaccess
   +++ b/.htaccess
   @@ -2,7 +2,7 @@
    # -----------------------------------------------------

    # $Id$
   -# $Date: 2009-08-09 19:52:30 $
   +# $Date: 2009-08-09 20:31:56 $
   [...]

Strangely, the $Id$ line above (which is handled by me adding the "ident" 
attribute to all text files), does not exhibit the same problem.

Doing "git reset --hard" or "git checkout master filename" does not help, 
the file is still believed to be modified by git.


Does anyone know if there is a way to get this to work, or am I screwed for 
trying to do things I'm not supposed to? :-)

-- 
\\// Peter - http://www.softwolves.pp.se/

Re: Implementing $Date$ substitution - problem with git status

[Johannes Sixt]

On Sonntag, 9. August 2009, Peter Krefting wrote:
> I finally got around to moving my web site repository over from CVS to Git,
> and to not have to redo everything about how it is set up, I set up a
> filter that fakes CVS' handling of "$Date$" keywords. Since I cannot access
> any information about the file from the "smudge" filter, I set up a "clean"
> filter that edits the $Date$ token on "git add":
>
>    #!/usr/bin/perl -wTp -i.bak
>    if (/\$Date:?[^\$]*\$/)
>    {
>      my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) =
> localtime(); my $now = sprintf("%04d-%02d-%02d %02d:%02d:%02d", $year +
> 1900, $mon + 1, $mday, $hour, $min, $sec);
>      s/\$Date:?[^\$]*\$/\$Date: ${now} \$/;
>    }
>
> In .gitattributes I have lines like:
>
>    *.html filter=date crlf=true ident
>
> and filter.date.clean is set to point to the script above in .git/config.
>
> This works fine, except that I have some problems with files that I cannot
> get to be "clean" in git status. "git diff" shows only a difference in the
> "$Date$" token, which is different each time I run "git diff" (the "+" line
> shows the current time):
>
>    diff --git a/.htaccess b/.htaccess
>    index 2dfbc32..d8ac343 100644
>    --- a/.htaccess
>    +++ b/.htaccess
>    @@ -2,7 +2,7 @@
>     # -----------------------------------------------------
>
>     # $Id$
>    -# $Date: 2009-08-09 19:52:30 $
>    +# $Date: 2009-08-09 20:31:56 $
>    [...]

The clean filter is applied before 'git diff' generates the diff. Since you 
have unstaged changes (I take [...] to mean this), you see a new date on 
every invocation.

After 'git add .htaccess' subsequent 'git diff' should not show any changes.

> Strangely, the $Id$ line above (which is handled by me adding the "ident"
> attribute to all text files), does not exhibit the same problem.
>
> Doing "git reset --hard" or "git checkout master filename" does not help,
> the file is still believed to be modified by git.

Now, that's an entirely different problem, and I think that there is a bug. I 
have observed this as well with my own clean filter sometimes, but not 
always. I haven't found a recipe that reliably exhibits the problem.

-- Hannes

Re: Implementing $Date$ substitution - problem with git status

[Peter Krefting]

Johannes Sixt:

> The clean filter is applied before 'git diff' generates the diff. Since you
> have unstaged changes (I take [...] to mean this), you see a new date on
> every invocation.

Sorry for being unclear; no, the $Date$ line is the only change that shows 
up. I removed the surrounding context because I didn't want to publish them.

On another machine, there is another file that exhibits the same problem.

> After 'git add .htaccess' subsequent 'git diff' should not show any changes.

But then I need to check it in again, and I'd like to avoid that, if possible.


Is it possible to have a filter run on "commit" instead of "add"? (Or is 
"commit" just moving changes from the index to the repository, making that 
impossible?)

>> Doing "git reset --hard" or "git checkout master filename" does not 
>> help, the file is still believed to be modified by git.
>
> Now, that's an entirely different problem, and I think that there is a 
> bug. I have observed this as well with my own clean filter sometimes, but 
> not always. I haven't found a recipe that reliably exhibits the problem.

It seems to be triggered by using my date script and doing

   git add file
   git commit file
   git reset --hard file

the last command gives me a local copy with the keyword expanded, but marked 
dirty.


Is it possible to have "git add" expand the keyword (both the Git-internal 
"ident", i.e "$Id$", and my "$Date$")? Would duplicating the "clean" filter 
as a "smudge" one accomplish that?

-- 
\\// Peter - http://www.softwolves.pp.se/

Re: Implementing $Date$ substitution - problem with git status

[Peter Krefting]

Johannes Sixt:

>> Doing "git reset --hard" or "git checkout master filename" does not 
>> help, the file is still believed to be modified by git.

> Now, that's an entirely different problem, and I think that there is a 
> bug. I have observed this as well with my own clean filter sometimes, but 
> not always. I haven't found a recipe that reliably exhibits the problem.

After som examination, it seems to be caused by the way I imported the CVS 
history: I kept all the $ keywords expanded in history (so that if I check 
out an old version from Git, it looks like it did in CVS). This means that 
still in the latest revision of several files, I have "$Id$" lines checked 
in in CVS format.

However, as I have set the "ident" attribute, Git wants to expand it itself 
and check in files with "$Id$". When I do a reset, it seems it records the 
entry as clean against a version stored with just "$Id$", but the record in 
history has an expanded "$Id$", and the entry is thus never deemed clean.

I can probably work around this by removing the "$Id$" attributes, or by 
removing the "ident" rule.

-- 
\\// Peter - http://www.softwolves.pp.se/

Problems with filters and git status - reproduction steps

[Peter Krefting]

Peter Krefting:

> However, as I have set the "ident" attribute, Git wants to expand it itself 
> and check in files with "$Id$". When I do a reset, it seems it records the 
> entry as clean against a version stored with just "$Id$", but the record in 
> history has an expanded "$Id$", and the entry is thus never deemed clean.

Actually, that is not the case. It seems to be the filter that causes the 
problems, all by itself. I cannot seem to reproduce this *reliably*. I tried 
setting up a minimal repository and a reproduction recipe, but I get 
different behaviour when I perform the steps manually, and when I run it 
from a shell script.

Here is the reproduction recipe:

#!/bin/bash
# Reproduction recipe for $Date$ dirty issue

ORIGINREPO=git://git.debian.org/users/peterk/gitfilterproblem.git
DESTINATIONREPO=gitfilterproblem-testrepo

# Set up repository
echo -- Cloning
git clone ${ORIGINREPO} ${DESTINATIONREPO}
cd ${DESTINATIONREPO}

# Status should be clean
echo -- After cloning, status should be clean
git status

# Set up filter
echo -- Set up filter, status should be clean
./reposetup.sh 
git status

# Create tracking branch
echo -- Create tracking branch for changed file, status should be clean
git branch --track changed-text-files origin/changed-text-files 
git status

# Merge the branch
echo -- Merge the changes from the branch, now status gets unclean
git merge changed-text-files
echo
git status
git diff

-- 
\\// Peter - http://www.softwolves.pp.se/

Re: Problems with filters and git status - reproduction steps

[Michael J Gruber]

Peter Krefting venit, vidit, dixit 12.08.2009 10:36:
> ORIGINREPO=git://git.debian.org/users/peterk/gitfilterproblem.git
> DESTINATIONREPO=gitfilterproblem-testrepo
> 
> # Set up repository
> echo -- Cloning
> git clone ${ORIGINREPO} ${DESTINATIONREPO}
> cd ${DESTINATIONREPO}
> 
> # Status should be clean
> echo -- After cloning, status should be clean
> git status
> 
> # Set up filter
> echo -- Set up filter, status should be clean
> ./reposetup.sh 
> git status

I get "Changed but not updated" already here!

Also, what's your git version? There have been some fix-ups recently
regarding the interaction between filters/textconv and assumptions of
the code about dirtiness of the worktree.

> 
> # Create tracking branch
> echo -- Create tracking branch for changed file, status should be clean
> git branch --track changed-text-files origin/changed-text-files 
> git status
> 
> # Merge the branch
> echo -- Merge the changes from the branch, now status gets unclean
> git merge changed-text-files
> echo
> git status
> git diff

Do you really want the date in the checked-in version of the file? I
would assume otherwise. Then your clean filter should really be the
smudge filter, and you would need a clean filter to go with it (remove
the date and restore the keyword).

Michael

Re: Problems with filters and git status - reproduction steps

[Peter Krefting]

Michael J Gruber:

> I get "Changed but not updated" already here!

With git 1.5.6.5, I do get that. With git 1.6.4, I seem to only get it after 
the last step.

I have 1.5.6.5 on the server with the master repo, and am running 1.6.4 on 
the client (although when I ran the recipe through a shell script, I got the 
unclean status earlier, which made me think it ran an earlier version I have 
installed in another directory).

> Do you really want the date in the checked-in version of the file?

Yes. This way, the checked out copy that makes my web server can do its job 
without requiring any of the filters being installed. And the files in 
history are marked as necessary.

> I would assume otherwise. Then your clean filter should really be the 
> smudge filter, and you would need a clean filter to go with it (remove the 
> date and restore the keyword).

The problem with that approach is that the "smudge" filter does not have 
access to the file name, and so can not look up the last change date of the 
file it is re-writing. And I want the last time the file was *changed*, not 
the time it was checked out.

Also, I imported my entire CVS history with keywords expanded to allow for 
this (and "checked out" the Git-generated tree over my CVS check-out to not 
have Git update all the time-stamps).

-- 
\\// Peter - http://www.softwolves.pp.se/

Implementing $Date$ substitution - solution using pre-commit

[Peter Krefting]

Hi!

So, my filter-based approach at doing RCS-like $Date$ expansion didn't work 
out. Since I wanted to have the Date expand on check-in, not on check-out, 
and be expanded in history, I had made it into a "clean" filter. But due to 
how Git internally checks if files are clean, that did not work properly.

The solution to this was instead to make the Date expansion into a 
pre-commit hook. Using .gitattributes, I set "datereplace=true" on all the 
files I want to have "$Date$" expand in, and then install the following 
file as .git/hooks/pre-commit:

===8< pre-commit >8===
#!/bin/bash -e

# Find base commit
if git-rev-parse --verify HEAD >/dev/null 2>&1
then
 	against=HEAD
else
 	# Initial commit: diff against an empty tree object
 	against=4b825dc642cb6eb9a060e54bf8d69288fbee4904
fi

# Set up Date substitution
export NOW=$(date +"%Y-%m-%d %H:%M:%S")
for file in $(git diff-index --cached --diff-filter=AM --name-only $against); do
   if git check-attr datereplace -- "$file" | grep 'datereplace: true' > /dev/null; then
     perl -w -i.bak -e 'my $now = $ENV{"NOW"};
$now =~ s/[^-:0-9 ]//g;
while (<>)
{
  if (/\$Date:?[^\$]*\$/)
  {
    s/\$Date:?[^\$]*\$/\$Date: ${now} \$/;
  }
  print
}' "$file"
     git update-index --add "$file"
   fi
done

exit 0
===8< pre-commit >8===

This has the added bonus over a filter that it all files committed at the 
same time will have the same date stamp, whereas the filter would expand on 
the time "git add" is executed.

-- 
\\// Peter - http://www.softwolves.pp.se/