git with custom diff for commits

git with custom diff for commits

[Gerald Gutierrez]

Hello folks,

I do a nightly mysqldump of a database and check it into a git
repository. mysqldump generates a timestamp as part of that output
which is causing git to think that the file changes every night when
it really doesn't. The timestamp is simply in an SQL comment.

So what I'd like to do is teach git to ignore that particular SQL
timestamp comment. I've tried to set up an external diff script that
runs diff -I "<<sql timestamp comment>>" that effectively ignores the
timestamp. While this works with "git diff", it seems when git
commits, it still sees the differences.

How do I properly teach git to ignore these types of differences?

Gerald.

Re: git with custom diff for commits

[Johannes Schindelin]

Hi,

On Mon, 17 Dec 2007, Gerald Gutierrez wrote:

> I do a nightly mysqldump of a database and check it into a git 
> repository. mysqldump generates a timestamp as part of that output which 
> is causing git to think that the file changes every night when it really 
> doesn't. The timestamp is simply in an SQL comment.
> 
> So what I'd like to do is teach git to ignore that particular SQL 
> timestamp comment. I've tried to set up an external diff script that 
> runs diff -I "<<sql timestamp comment>>" that effectively ignores the 
> timestamp. While this works with "git diff", it seems when git commits, 
> it still sees the differences.
> 
> How do I properly teach git to ignore these types of differences?

You might be interested in reading Documentation/gitattributes.txt, look 
for "diff driver".

Hth,
Dscho

Re: git with custom diff for commits

[Matthieu Moy]

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

> Hi,
>
> On Mon, 17 Dec 2007, Gerald Gutierrez wrote:
>
>> I do a nightly mysqldump of a database and check it into a git 
>> repository. mysqldump generates a timestamp as part of that output which 
>> is causing git to think that the file changes every night when it really 
>> doesn't. The timestamp is simply in an SQL comment.
>> 
>> So what I'd like to do is teach git to ignore that particular SQL 
>> timestamp comment. I've tried to set up an external diff script that 
>> runs diff -I "<<sql timestamp comment>>" that effectively ignores the 
>> timestamp. While this works with "git diff", it seems when git commits, 
>> it still sees the differences.
>> 
>> How do I properly teach git to ignore these types of differences?
>
> You might be interested in reading Documentation/gitattributes.txt, look 
> for "diff driver".

It will show an empty output for "git diff", but I doubt thit will
change anything at commit time. Probably the "filter" thing on the
same file (also "man gitattributes") can help though.

-- 
Matthieu

Re: git with custom diff for commits

[Johannes Schindelin]

Hi,

On Tue, 18 Dec 2007, Matthieu Moy wrote:

> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
> 
> > On Mon, 17 Dec 2007, Gerald Gutierrez wrote:
> >
> >> I've tried to set up an external diff script that runs diff -I "<<sql 
> >> timestamp comment>>" that effectively ignores the timestamp. While 
> >> this works with "git diff", it seems when git commits, it still sees 
> >> the differences.
> >> 
> >> How do I properly teach git to ignore these types of differences?
> >
> > You might be interested in reading Documentation/gitattributes.txt, 
> > look for "diff driver".
> 
> It will show an empty output for "git diff", but I doubt thit will 
> change anything at commit time. Probably the "filter" thing on the same 
> file (also "man gitattributes") can help though.

Ah, right.  I completely missed that you were talking about git-commit, 
not git-log on git commits.

Yes, setting up a "clean" filter that removes the timestamps is probably 
the reasonable thing to do here.

Sorry for the noise,
Dscho

Re: git with custom diff for commits

[Junio C Hamano]

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

>> It will show an empty output for "git diff", but I doubt thit will 
>> change anything at commit time. Probably the "filter" thing on the same 
>> file (also "man gitattributes") can help though.
>
> Ah, right.  I completely missed that you were talking about git-commit, 
> not git-log on git commits.
>
> Yes, setting up a "clean" filter that removes the timestamps is probably 
> the reasonable thing to do here.

I wouldn't do filters for something like that.  Can you guarantee that
the output from corresopnding smudge filter will load cleanly back to
the mysql database?

Just do not make the commit if you made only meaningless changes and
nothing else.  pre-commit hook would probably be a good place to do so.

RE: git with custom diff for commits

[Gerald Gutierrez]

 
> Ah, right.  I completely missed that you were talking about 
> git-commit, not git-log on git commits.
> 
> Yes, setting up a "clean" filter that removes the timestamps 
> is probably the reasonable thing to do here.

I read about the filter too, but have no idea how to implement it. Any
examples that I could follow?

Thanks,
Gerald.

Re: git with custom diff for commits

[Matthieu Moy]

Junio C Hamano <gitster@pobox.com> writes:

> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
>
>>> It will show an empty output for "git diff", but I doubt thit will 
>>> change anything at commit time. Probably the "filter" thing on the same 
>>> file (also "man gitattributes") can help though.
>>
>> Ah, right.  I completely missed that you were talking about git-commit, 
>> not git-log on git commits.
>>
>> Yes, setting up a "clean" filter that removes the timestamps is probably 
>> the reasonable thing to do here.
>
> I wouldn't do filters for something like that.  Can you guarantee that
> the output from corresopnding smudge filter will load cleanly back to
> the mysql database?

The original poster said the date was an SQL comment, so, it should be
safe to strip it. That said, precommit hook is probably a good or
better solution. At least, if you're not sure you didn't make a
mistake writting it, you'll notice either a good commit or no commit
at all. With filters, you might well notice you corrupted the commit
too late :-(.

-- 
Matthieu

Re: git with custom diff for commits

[Junio C Hamano]

Matthieu Moy <Matthieu.Moy@imag.fr> writes:

> Junio C Hamano <gitster@pobox.com> writes:
> ...
>> I wouldn't do filters for something like that.  Can you guarantee that
>> the output from corresopnding smudge filter will load cleanly back to
>> the mysql database?
>
> The original poster said the date was an SQL comment, so, it should be
> safe to strip it. That said, precommit hook is probably a good or
> better solution. At least, if you're not sure you didn't make a
> mistake writting it, you'll notice either a good commit or no commit
> at all. With filters, you might well notice you corrupted the commit
> too late :-(.

Yeah.  After writing the message you are responding to, I realized that
I already do something like that to manage html/man branches that are
autogenerated, and it is exactly the same situation as Gerald is in.

The autogeneration of documentation and updating of html/man branches is
controlled by dodoc.sh script in 'todo' branch, installed as post-update
hook in my public repository.  It builds the documentation in a checkout
of 'master' branch, installs the result into another checkout of 'html'
and 'man' branches, and commits.

The twist is in the "install" step (this is different from my suggestion
of preventing the commit in pre-commit hook).  When the newly generated
documentation files are different from the previously generated ones
only by datestamp, they are not installed into 'html'/'man' checkouts,
and such a change is not included in the commits.

So my updated suggestion would be:

 - Keep a checkout of that mysql dump;

 - Make the dump procedure to dump to that checkout area;

 - Run "git diff HEAD", perhaps with the external diff that filters out
   the cruft as suggested in the thread, and run "git checkout $path"
   for paths that had only meaningless changes;

 - Run "git commit" if the above step did not nuke all changes.

RE: git with custom diff for commits

[Gerald Gutierrez]

> 
> So my updated suggestion would be:
> 
>  - Keep a checkout of that mysql dump;
> 
>  - Make the dump procedure to dump to that checkout area;
> 
>  - Run "git diff HEAD", perhaps with the external diff that 
> filters out
>    the cruft as suggested in the thread, and run "git checkout $path"
>    for paths that had only meaningless changes;
> 
>  - Run "git commit" if the above step did not nuke all changes.
> 

I ended up doing the following, which sounds similar to the above. Instead
of doing mysqldump into data.sql, it goes into data.sql.2 which I compare
with the checked in data.sql using "diff -I <timestamp RE>". If there are no
differences, I delete data.sql.2. If there are differences, I move
data.sql.2 into data.sql and check in. Perhaps not as elegant but certainly
works.

Gerald.

Re: git with custom diff for commits

[Junio C Hamano]

"Gerald Gutierrez" <ggmlfs@gmail.com> writes:

> I ended up doing the following, which sounds similar to the above. Instead
> of doing mysqldump into data.sql, it goes into data.sql.2 which I compare
> with the checked in data.sql using "diff -I <timestamp RE>". If there are no
> differences, I delete data.sql.2. If there are differences, I move
> data.sql.2 into data.sql and check in. Perhaps not as elegant but certainly
> works.

Heh, that's essentially how automated html/man branches are managed ;-)

RE: git with custom diff for commits

[Gerald Gutierrez]

Yes, but wouldn't it be slick to actually teach git's internal diff to do
things like GNU diff can, like the ignore option -I, case insensitivity,
etc. I thought that's what the external diff capability is for, but it is
not so.

Gerald.

> -----Original Message-----
> From: Junio C Hamano [mailto:gitster@pobox.com] 
> Sent: Tuesday, December 18, 2007 12:40 PM
> To: Gerald Gutierrez
> Cc: 'Junio C Hamano'; 'Matthieu Moy'; 'Johannes Schindelin'; 
> git@vger.kernel.org
> Subject: Re: git with custom diff for commits
> 
> "Gerald Gutierrez" <ggmlfs@gmail.com> writes:
> 
> > I ended up doing the following, which sounds similar to the above. 
> > Instead of doing mysqldump into data.sql, it goes into data.sql.2 
> > which I compare with the checked in data.sql using "diff -I 
> <timestamp 
> > RE>". If there are no differences, I delete data.sql.2. If 
> there are 
> > differences, I move
> > data.sql.2 into data.sql and check in. Perhaps not as elegant but 
> > certainly works.
> 
> Heh, that's essentially how automated html/man branches are 
> managed ;-)

RE: git with custom diff for commits

[Johannes Schindelin]

Hi,

On Tue, 18 Dec 2007, Gerald Gutierrez wrote:

> Yes, but wouldn't it be slick to actually teach git's internal diff to 
> do things like GNU diff can, like the ignore option -I, case 
> insensitivity, etc. I thought that's what the external diff capability 
> is for, but it is not so.

I disagree.  Your statement is misleading when you say you want the 
internal diff to do things like GNU diff can.

What you want to do _in fact_ is not only modify the diff _output_, but 
the commit _input_.

And I am not so keen on such filters.  If I do not want to commit,  I do 
not change it to begin with.  Yes, your mysqldump is a special case here.  
But it is special anyway, as it is not source code.

Ciao,
Dscho

Re: git with custom diff for commits

[Junio C Hamano]

"Gerald Gutierrez" <ggmlfs@gmail.com> writes:

> I thought that's what the external diff capability is for,...

It certainly is.  Your external diff gets all the information it needs
to generate whatever custom diff computation it may want to do.

Re: git with custom diff for commits

[Matthieu Moy]

"Gerald Gutierrez" <ggmlfs@gmail.com> writes:

> Yes, but wouldn't it be slick to actually teach git's internal diff to do
> things like GNU diff can, like the ignore option -I, case insensitivity,
> etc. I thought that's what the external diff capability is for, but it is
> not so.

Don't confuse diff and commit.

Git is really snapshot oriented. When you commit, you don't say
"record the changes I made since HEAD", you say "record the current
state of the working tree, and also record the fact that this state is
based on HEAD".

See what a commit object looks like:

$ git cat-file -p 183f84365d 
tree 3f9d576b4adc78188a411104bc21159d459fa3f4
parent b9c506f7d9f05a630fa7e31b77a9cf5081d7dbba
author Shun Kei Leung <kevinlsk@gmail.com> 1195614079 +0800
committer Junio C Hamano <gitster@pobox.com> 1195630064 -0800

git-p4: Fix typo in --detect-labels

Signed-off-by: Kevin Leung <kevinlsk@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
$ _

The "tree" tells you the state of the project as recorded by commit,
and the "parent" tells you where the HEAD was pointing to at commit
time. That's really the raw information recorded by git.

Now, "diff" comes into the picture for two things: 1) storage
efficiency (you prefer a 1Mb pack file to a 1Gb uncompressed
repository), but that doesn't change the concept, and 2) user
interface.

When you say "git show HEAD", for example, git will recompute a diff,
and show it to you because this is what you expect. But this
information is computed at "git show" time, not at "git commit" time.


What you seem to expect is to tell git "commit the changes except this
portion", what you have to tell git is actually "commit the new state,
but not if this portion is the only change". That sounds identical,
but it's actually a bit different.

-- 
Matthieu

RE: git with custom diff for commits

[Gerald Gutierrez]

 
> -----Original Message-----
> From: Johannes Schindelin [mailto:Johannes.Schindelin@gmx.de] 
> Sent: Tuesday, December 18, 2007 1:27 PM
> To: Gerald Gutierrez
> Cc: 'Junio C Hamano'; 'Matthieu Moy'; git@vger.kernel.org
> Subject: RE: git with custom diff for commits
> 
> Hi,
> 
> On Tue, 18 Dec 2007, Gerald Gutierrez wrote:
> 
> > Yes, but wouldn't it be slick to actually teach git's 
> internal diff to 
> > do things like GNU diff can, like the ignore option -I, case 
> > insensitivity, etc. I thought that's what the external diff 
> capability 
> > is for, but it is not so.
> 
> I disagree.  Your statement is misleading when you say you 
> want the internal diff to do things like GNU diff can.
> 
> What you want to do _in fact_ is not only modify the diff 
> _output_, but the commit _input_.
> 
> And I am not so keen on such filters.  If I do not want to 
> commit,  I do not change it to begin with.  Yes, your 
> mysqldump is a special case here.  
> But it is special anyway, as it is not source code.

I don't understand how it is a benefit to have "git commit" and "git diff"
do different diff functions. It confuses me. A scenario: I keep doing git
diff's and it says there is no difference, but then git commit keeps telling
me there is.

While I understand it was built for the kernel and everybody likes the plain
text format, it is not a stretch of the imagination to have files that are
semantically identical but have byte-wise differences. OpenDocument files,
image files, XML, etc. Cases can be made either way that "the same" means
semantically or byte-wise. It's perfectly fine that the default is
byte-wise. But, if the user takes the time to build a custom diff, then
wouldn't it be reasonable to assume that user means for git to do semantic
comparisons for diff, commit and other git functions, even if there are
inconsequential byte-wise differences in the file itself?

Gerald.

RE: git with custom diff for commits

[Johannes Schindelin]

Hi,

On Tue, 18 Dec 2007, Gerald Gutierrez wrote:

> I don't understand how it is a benefit to have "git commit" and "git 
> diff" do different diff functions. It confuses me.

Matthieu said it better than I ever could.

Ciao,
Dscho

RE: git with custom diff for commits

[Gerald Gutierrez]

 
> What you seem to expect is to tell git "commit the changes 
> except this portion", what you have to tell git is actually 
> "commit the new state, but not if this portion is the only 
> change". That sounds identical, but it's actually a bit different.

It's not what I'm saying.

Here's what the git-commit manpage on kernel.org says: "git-commit - Record
changes to the repository".

What I am saying is that if the user defines a custom diff, it is reasonable
to believe that the user wants to teach git a new definition of "changes".
If that new definition says that the files have changed, then go ahead and
commit the new state. If that definition says there are no changes, then
there is nothing to commit. In other words:

if (user_redefined_changed)
  if (user_defn_of_changed(work, head)) commit(work);
else
  if (builtin_changed(work, head)) commit(work);

As it is, it's not a big deal to me. Instead of teaching git the different
concept of "changes", I teach the script that runs git instead. But teaching
git directly just seems cleaner and more intuitive.

Gerald.

RE: git with custom diff for commits

[Gerald Gutierrez]

 
> if (user_redefined_changed)
>   if (user_defn_of_changed(work, head)) commit(work); else
>   if (builtin_changed(work, head)) commit(work);

That didn't come out quite right. It should be:

if (user_redefined_changed) {
  if (user_defn_of_changed(work, head)) { commit(work); }
} else {
  if (builtin_changed(work, head)) { commit(work); }
}

RE: git with custom diff for commits

[Johannes Schindelin]

Hi,

On Tue, 18 Dec 2007, Gerald Gutierrez wrote:

[who said that?  (I know, but would you know in a few weeks?  Right.)]

> > What you seem to expect is to tell git "commit the changes except this 
> > portion", what you have to tell git is actually "commit the new state, 
> > but not if this portion is the only change". That sounds identical, 
> > but it's actually a bit different.
> 
> It's not what I'm saying.
> 
> Here's what the git-commit manpage on kernel.org says: "git-commit - 
> Record changes to the repository".

Yes.  Changes, as in "take this _file_ instead".  Not "edit this file, 
remove those lines, add these here, etc.".

So I think that you were saying exactly what Matthieu suggested.

Ciao,
Dscho

Re: git with custom diff for commits

[Martin Langhoff]

On Dec 19, 2007 11:48 AM, Johannes Schindelin
<Johannes.Schindelin@gmx.de> wrote:
> Yes.  Changes, as in "take this _file_ instead".  Not "edit this file,
> remove those lines, add these here, etc.".

Exactly. GIT is a "content tracker" that doesn't care about any
semantic meaning in your files. Here's the file, git will store it, as
is. No conversions, not fancy interpretations. Strict, safe and fast.
(There's a small exception there for DOS-style newlines, which was
much resisted.)

You can add external machinery that is aware of your content semantics
-- as you've done with calling GNU diff with ignore patterns to decide
whether to commit or not. But when you tell GIT to commit something,
there's no guessing or transformation involved.

A lot of what GIT achieves is based on that founding principle. Remove
it, and we're toast. For starters, the internal machinery is based on
SHA1 digests that change if you flip a single byte. That's what makes
git fast and realiable. It's not merely an end-user thing.

cheers,


m

Re: git with custom diff for commits

[Matthieu Moy]

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

>> Here's what the git-commit manpage on kernel.org says: "git-commit - 
>> Record changes to the repository".
>
> Yes.  Changes, as in "take this _file_ instead".  Not "edit this file, 
> remove those lines, add these here, etc.".

That said, I think the wording of the man page could be better.
Perhaps "Record new snapshot to the repository" or "Record current
state of the working tree to the repository".

I can't find something wich is both accurate and newbie friendly ...
("accurate" would probably be "Record index content into the
repository")

-- 
Matthieu

Re: git with custom diff for commits

[Matthieu Moy]

"Gerald Gutierrez" <ggmlfs@gmail.com> writes:

> if (user_redefined_changed)
>   if (user_defn_of_changed(work, head)) commit(work);
> else
>   if (builtin_changed(work, head)) commit(work);

That would lead to even less intuitive semantics.

Suppose your working tree has two changes, like

--- a/file
+++ b/file
@@ ... @@
-old ignored line
+new ignored line
@@ ... @@
-this is relevant
+change

What would git be supposed to do if you ignore anything containing
"ignored"?

If git commits, it will commit both changes. People checking-out the
new revision will get "new ignored line" in the file. So, a portion of
the diff that you wanted to ignore has actually gone into the
repository. IOW, whether "new ignored line" would go into the
repository depends on the presence of another unrelated change.

Perhaps you would have expected this, but I wouldn't, and I bet many
other users wouldn't either. And usually, when you can't have people
agree on a common reasonable semantics, the answer is "don't put it in
git, let people script the semantics they want".


Take another example : I use an external diff tool to work with
OpenDocument files. It converts them into text, and calls "git diff"
on them. My diff driver therefore ignores formatting changes. If I
made only content changes, I get no change with this diff driver.
Still, if I commit, I expect git to record my new formatting.

-- 
Matthieu

Re: git with custom diff for commits

[Johannes Schindelin]

Hi,

On Wed, 19 Dec 2007, Matthieu Moy wrote:

> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
> 
> >> Here's what the git-commit manpage on kernel.org says: "git-commit - 
> >> Record changes to the repository".
> >
> > Yes.  Changes, as in "take this _file_ instead".  Not "edit this file, 
> > remove those lines, add these here, etc.".
> 
> That said, I think the wording of the man page could be better.
> Perhaps "Record new snapshot to the repository" or "Record current
> state of the working tree to the repository".

How about "record a new revision"?

Ciao,
Dscho

Re: git with custom diff for commits

[Matthieu Moy]

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

> On Wed, 19 Dec 2007, Matthieu Moy wrote:
>
>> That said, I think the wording of the man page could be better.
>> Perhaps "Record new snapshot to the repository" or "Record current
>> state of the working tree to the repository".
>
> How about "record a new revision"?

Sounds good, yes.

-- 
Matthieu