Interaction between clean/smudge and git status

Interaction between clean/smudge and git status

[Sergio Callegari]

Hi,

I have tried for the first time the .gitattributes filter option, setting a
clean and a smudge filter for a certain type of files.

What makes me wonder is that using filters, after a clean checkout git status
says that everything is changed.

My filter is a very short script that operates on zip files.
The clean command re-zips them so that the content is merely stored.
The smudge one re-zips them so that the content is deflated again.

This kind of filter helps very much the git repacking when zip files or
openoffice files are around.

But unfortunately, with it git shows everything as changed, which is not that
nice.

Is this the expected behaviour of the smudge filter?
Unfortunately I have been able to find very little documentation on it (only a
bit in the gitattributes man page), so maybe I am missing something.

Re: Interaction between clean/smudge and git status

[Johannes Sixt]

Sergio Callegari schrieb:
> I have tried for the first time the .gitattributes filter option, setting a
> clean and a smudge filter for a certain type of files.
> 
> What makes me wonder is that using filters, after a clean checkout git status
> says that everything is changed.
...
> Is this the expected behaviour of the smudge filter?

I've observed this, too, and I don't think it is expected behavior. But it
hasn't annoyed me enough to look at it in depth. Eventually I will, and I
hope to find out what's wrong. ;)

-- Hannes

Re: Interaction between clean/smudge and git status

[Junio C Hamano]

Johannes Sixt <j.sixt@viscovery.net> writes:

> Sergio Callegari schrieb:
>> I have tried for the first time the .gitattributes filter option, setting a
>> clean and a smudge filter for a certain type of files.
>> 
>> What makes me wonder is that using filters, after a clean checkout git status
>> says that everything is changed.

What is to "re-zip"?  You have a .zip file that contains a single file in
your work tree, and the index and the tree objects record that single file
deflated?  When you "check out" from the index, you run smudge to create a
new .zip file with that single file?

>> Is this the expected behaviour of the smudge filter?
>
> I've observed this, too, and I don't think it is expected behavior. But it
> hasn't annoyed me enough to look at it in depth. Eventually I will, and I
> hope to find out what's wrong. ;)

Are you recreating the .zip file in the filter in such a way that a file
with the same contents results in byte-to-byte identical .zip file?
Otherwise as far as git is concerned you have changed the file in the work
tree.

Re: Interaction between clean/smudge and git status

[Johannes Sixt]

Junio C Hamano schrieb:
> Johannes Sixt <j.sixt@viscovery.net> writes:
>> I've observed this, too, and I don't think it is expected behavior. But it
>> hasn't annoyed me enough to look at it in depth. Eventually I will, and I
>> hope to find out what's wrong. ;)
> 
> Are you recreating the .zip file in the filter in such a way that a file
> with the same contents results in byte-to-byte identical .zip file?
> Otherwise as far as git is concerned you have changed the file in the work
> tree.

In my case, there is only a "clean" filter, and it rearranges the lines of
a particular type of text files in a canonical way. Hmm, I can't reproduce
the unwanted behavior in quick test now. I'll come back to this issue if
it shows up again.

-- Hannes

Re: Interaction between clean/smudge and git status

[Sergio Callegari]

Junio C Hamano <gitster <at> pobox.com> writes:


> 
> What is to "re-zip"?  You have a .zip file that contains a single file in
> your work tree, and the index and the tree objects record that single file
> deflated?  When you "check out" from the index, you run smudge to create a
> new .zip file with that single file?

I have a zip file that contains a collection of files (or I have an openoffice
file that is just the same).  The program that creates the zip file uses default
compression.  In this way, things managed by git result in objects that cannot
be deltified one against the other very well by git repack.

my re-zip script takes a zip file on stdin, unpacks everything in a temporary
directory, then recreates the archive with a different compression level and
puts it out on stdout.  When the compression is 0, things are merely put in the
archive. In this way the files managed git result in objects that do deltify
well one against the other and in much smaller packs.

> Are you recreating the .zip file in the filter in such a way that a file
> with the same contents results in byte-to-byte identical .zip file?
> Otherwise as far as git is concerned you have changed the file in the work
> tree.

And here you are right!!!
I thought that re-zip script was repeatable in behaviour, but it is not
(probably because things like file dates change when files are unpacked in the
temporary dir and dates get stored).

I absolutely overlooked that.

Then to do what I want to do, I need to work at a lower level, I cannot just
unzip and zip again.

Thanks

Re: Interaction between clean/smudge and git status

[Sergio Callegari]

Sergio Callegari <scallegari <at> arces.unibo.it> writes:

> 
> Junio C Hamano <gitster <at> pobox.com> writes:
> 
> 
> > Are you recreating the .zip file in the filter in such a way that a file
> > with the same contents results in byte-to-byte identical .zip file?
> > Otherwise as far as git is concerned you have changed the file in the work
> > tree.
> 
> And here you are right!!!
> I thought that re-zip script was repeatable in behaviour, but it is not
> (probably because things like file dates change when files are unpacked in the
> temporary dir and dates get stored).
> 
> I absolutely overlooked that.
> 

OK, here is a testcase too...

mkdir TEST
git init
# create zip file x.zip
git add x.zip
git commit

In this git commit the clean filter runs.

rm x.zip

git checkout x.zip

or 

git reset --hard

In this checkout the smudge filter runs

git status

It says that x.zip is changed

And yes, in some sense it is changed, because it is a different file than the
one I had before the check in. But no, in some other sense it is not changed,
because it is the file that I have just checked out (it has not been touched
after the checkout).

Not that if I had only the clean filter and not the smudge one, then the same
would have happened. 


So I think that I see your point: if the clean/smudge filters always provide the
same output independently from when they are run, then I get the message about
the changed file at most once, when I check in for the first time the "cleaned"
file.  

And this behaviour makes sense: to say that nothing has changed, git wants
things to be identical.  However it is a bit counterintuitive, because one would
think that something that has just been freshly checked out should not be
considered as changed anyway.

I wonder if this comes from the fact that when git status is run, git compares
the workdir file with the index and the index contains information on the file
as it was before the last checkin.  When filters exist, wouldn't it make sense
to have the index hold information on the files as they are after the checkout?

Sergio