Applying .gitattributes text/eol changes

Applying .gitattributes text/eol changes

[Marc Strapetz]

I'm looking for an unobtrusive way to apply (committed) changes for
text/eol attributes to the working tree. For instance, after having
changed "*.txt eol=crlf" to "*.txt eol=lf", all *.txt files should be
converted from CRLF to LF endings. The only advice I found so far is to
remove .git/index and do a reset --hard. The disadvantage of this
approach is that every file will be touched:

- although the content does not change, timestamps will be changed. This
makes tools like IDEs assume that the file content has been changed.
(Even if the timestamps would be properly reset, the replacement of the
files would have triggered system file change notifications and I'd
expect various tools to still reload these files)

- there will be warnings for files which are locked by other processes
(at least on Windows). I'm usually seeing this for JAR files which are
not affected by eol-attribute changes at all.

One solution I could think of which might be helpful in other situations
as well would be to have an "--unobtrusive" option for reset which would
only replace a file if the content has actually been changed.

Marc.

Re: Applying .gitattributes text/eol changes

[Marc Strapetz]

Any ideas on this?

Thanks,
Marc.


On 03.01.2011 18:18, Marc Strapetz wrote:
> I'm looking for an unobtrusive way to apply (committed) changes for
> text/eol attributes to the working tree. For instance, after having
> changed "*.txt eol=crlf" to "*.txt eol=lf", all *.txt files should be
> converted from CRLF to LF endings. The only advice I found so far is to
> remove .git/index and do a reset --hard. The disadvantage of this
> approach is that every file will be touched:
> 
> - although the content does not change, timestamps will be changed. This
> makes tools like IDEs assume that the file content has been changed.
> (Even if the timestamps would be properly reset, the replacement of the
> files would have triggered system file change notifications and I'd
> expect various tools to still reload these files)
> 
> - there will be warnings for files which are locked by other processes
> (at least on Windows). I'm usually seeing this for JAR files which are
> not affected by eol-attribute changes at all.
> 
> One solution I could think of which might be helpful in other situations
> as well would be to have an "--unobtrusive" option for reset which would
> only replace a file if the content has actually been changed.
> 
> Marc.
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: Applying .gitattributes text/eol changes

[Michael J Gruber]

Marc Strapetz venit, vidit, dixit 03.01.2011 18:18:
> I'm looking for an unobtrusive way to apply (committed) changes for
> text/eol attributes to the working tree. For instance, after having
> changed "*.txt eol=crlf" to "*.txt eol=lf", all *.txt files should be
> converted from CRLF to LF endings. The only advice I found so far is to
> remove .git/index and do a reset --hard. The disadvantage of this
> approach is that every file will be touched:
> 
> - although the content does not change, timestamps will be changed. This

The bytewise content does change.

> makes tools like IDEs assume that the file content has been changed.

It may be that the content is semantically equivalent.

> (Even if the timestamps would be properly reset, the replacement of the
> files would have triggered system file change notifications and I'd
> expect various tools to still reload these files)
> 
> - there will be warnings for files which are locked by other processes
> (at least on Windows). I'm usually seeing this for JAR files which are
> not affected by eol-attribute changes at all.
> 
> One solution I could think of which might be helpful in other situations
> as well would be to have an "--unobtrusive" option for reset which would
> only replace a file if the content has actually been changed.

How about

git ls-files \*.txt | xargs touch -a
git ls-files \*.txt | git checkout

?

Re: Applying .gitattributes text/eol changes

[Marc Strapetz]

On 11.01.2011 13:11, Michael J Gruber wrote:
> Marc Strapetz venit, vidit, dixit 03.01.2011 18:18:
>> I'm looking for an unobtrusive way to apply (committed) changes for
>> text/eol attributes to the working tree. For instance, after having
>> changed "*.txt eol=crlf" to "*.txt eol=lf", all *.txt files should be
>> converted from CRLF to LF endings. The only advice I found so far is to
>> remove .git/index and do a reset --hard. The disadvantage of this
>> approach is that every file will be touched:
>>
>> - although the content does not change, timestamps will be changed. This
> 
> The bytewise content does change.

The content has only changed for *.txt files, but the timestamps of
*all* files are updated. I guess (but didn't verify from code), that in
case of missing .git/index, Git freshly writes all working tree files,
ignoring already existing files which already have the correct content.
Maybe this behavior is by intention and makes sense in some cases. In my
case it has adverse effects on IDEs and probably other tools which are
monitoring the file system.

>> One solution I could think of which might be helpful in other situations
>> as well would be to have an "--unobtrusive" option for reset which would
>> only replace a file if the content has actually been changed.
> 
> How about
> 
> git ls-files \*.txt | xargs touch -a
> git ls-files \*.txt | git checkout

That won't be helpful as it requires me to know what has changed.

--
Best regards,
Marc Strapetz
=============
syntevo GmbH
http://www.syntevo.com
http://blog.syntevo.com

Re: Applying .gitattributes text/eol changes

[Michael J Gruber]

Marc Strapetz venit, vidit, dixit 11.01.2011 15:02:
> On 11.01.2011 13:11, Michael J Gruber wrote:
>> Marc Strapetz venit, vidit, dixit 03.01.2011 18:18:
>>> I'm looking for an unobtrusive way to apply (committed) changes for
>>> text/eol attributes to the working tree. For instance, after having
>>> changed "*.txt eol=crlf" to "*.txt eol=lf", all *.txt files should be
>>> converted from CRLF to LF endings. The only advice I found so far is to
>>> remove .git/index and do a reset --hard. The disadvantage of this
>>> approach is that every file will be touched:
>>>
>>> - although the content does not change, timestamps will be changed. This
>>
>> The bytewise content does change.
> 
> The content has only changed for *.txt files, but the timestamps of
> *all* files are updated. I guess (but didn't verify from code), that in

Well, sure...

> case of missing .git/index, Git freshly writes all working tree files,
> ignoring already existing files which already have the correct content.
> Maybe this behavior is by intention and makes sense in some cases. In my
> case it has adverse effects on IDEs and probably other tools which are
> monitoring the file system.

...but changing gitattributes is something you don't do routinely in
your workflow; so, at worst there would be an occasional unnecessary run
of your build process.

> 
>>> One solution I could think of which might be helpful in other situations
>>> as well would be to have an "--unobtrusive" option for reset which would
>>> only replace a file if the content has actually been changed.
>>
>> How about
>>
>> git ls-files \*.txt | xargs touch -a
>> git ls-files \*.txt | git checkout
> 
> That won't be helpful as it requires me to know what has changed.

But you do know that only (at most) *.txt have changed!

Michael

Re: Applying .gitattributes text/eol changes

[Marc Strapetz]

>> case of missing .git/index, Git freshly writes all working tree files,
>> ignoring already existing files which already have the correct content.
>> Maybe this behavior is by intention and makes sense in some cases. In my
>> case it has adverse effects on IDEs and probably other tools which are
>> monitoring the file system.
> 
> ...but changing gitattributes is something you don't do routinely in
> your workflow; so, at worst there would be an occasional unnecessary run
> of your build process.

Our Git-SVN bridge does it, potentially on every pull. This is why we
currently need to run "rm .git/index && git reset --hard" after every
pull, resp. every checkout (switching to another commit may result in
changed .gitattributes as well).

If a "git checkout" would (optionally) make sure that all EOLs are
properly set according to .gitattributes, the problem would be resolved.
As this might be not so easy to implement, my suggestion was to make
"git reset --hard" work more unobtrusive. I think we can provide a
corresponding patch, if it has chances to get accepted.

--
Best regards,
Marc Strapetz
=============
syntevo GmbH
http://www.syntevo.com
http://blog.syntevo.com



On 13.01.2011 14:23, Michael J Gruber wrote:
> Marc Strapetz venit, vidit, dixit 11.01.2011 15:02:
>> On 11.01.2011 13:11, Michael J Gruber wrote:
>>> Marc Strapetz venit, vidit, dixit 03.01.2011 18:18:
>>>> I'm looking for an unobtrusive way to apply (committed) changes for
>>>> text/eol attributes to the working tree. For instance, after having
>>>> changed "*.txt eol=crlf" to "*.txt eol=lf", all *.txt files should be
>>>> converted from CRLF to LF endings. The only advice I found so far is to
>>>> remove .git/index and do a reset --hard. The disadvantage of this
>>>> approach is that every file will be touched:
>>>>
>>>> - although the content does not change, timestamps will be changed. This
>>>
>>> The bytewise content does change.
>>
>> The content has only changed for *.txt files, but the timestamps of
>> *all* files are updated. I guess (but didn't verify from code), that in
> 
> Well, sure...
> 
>> case of missing .git/index, Git freshly writes all working tree files,
>> ignoring already existing files which already have the correct content.
>> Maybe this behavior is by intention and makes sense in some cases. In my
>> case it has adverse effects on IDEs and probably other tools which are
>> monitoring the file system.
> 
> ...but changing gitattributes is something you don't do routinely in
> your workflow; so, at worst there would be an occasional unnecessary run
> of your build process.
> 
>>
>>>> One solution I could think of which might be helpful in other situations
>>>> as well would be to have an "--unobtrusive" option for reset which would
>>>> only replace a file if the content has actually been changed.
>>>
>>> How about
>>>
>>> git ls-files \*.txt | xargs touch -a
>>> git ls-files \*.txt | git checkout
>>
>> That won't be helpful as it requires me to know what has changed.
> 
> But you do know that only (at most) *.txt have changed!
> 
> Michael
> 
> 

Re: Applying .gitattributes text/eol changes

[Michael J Gruber]

Marc Strapetz venit, vidit, dixit 13.01.2011 15:28:
>>> case of missing .git/index, Git freshly writes all working tree files,
>>> ignoring already existing files which already have the correct content.
>>> Maybe this behavior is by intention and makes sense in some cases. In my
>>> case it has adverse effects on IDEs and probably other tools which are
>>> monitoring the file system.
>>
>> ...but changing gitattributes is something you don't do routinely in
>> your workflow; so, at worst there would be an occasional unnecessary run
>> of your build process.
> 
> Our Git-SVN bridge does it, potentially on every pull. This is why we
> currently need to run "rm .git/index && git reset --hard" after every
> pull, resp. every checkout (switching to another commit may result in
> changed .gitattributes as well).

OK, now you're telling us what this is about ;)

> If a "git checkout" would (optionally) make sure that all EOLs are
> properly set according to .gitattributes, the problem would be resolved.
> As this might be not so easy to implement, my suggestion was to make
> "git reset --hard" work more unobtrusive. I think we can provide a
> corresponding patch, if it has chances to get accepted.

There have been other cases where git update-index --really-refresh
wasn't enough. You might want to check whether that is a suitable "patch
attack vector". This might be useful not only for you but also for others.

Michael

Re: Applying .gitattributes text/eol changes

[Marc Strapetz]

>> If a "git checkout" would (optionally) make sure that all EOLs are
>> properly set according to .gitattributes, the problem would be resolved.
>> As this might be not so easy to implement, my suggestion was to make
>> "git reset --hard" work more unobtrusive. I think we can provide a
>> corresponding patch, if it has chances to get accepted.
> 
> There have been other cases where git update-index --really-refresh
> wasn't enough. You might want to check whether that is a suitable "patch
> attack vector". This might be useful not only for you but also for others.

So your suggestion is to fix "git update-index --really-refresh", so
it's a replacement for "rm .git/index"? This sounds reasonable,
especially as "rm .git/index" is something one feels not comfortable
about when performing the first time ;-)

Anyway, I'm still wondering if it will resolve the "git reset --hard"
problem of re-checking out every file, even if content is already
identical in the working tree. I think that part has to be fixed, too.

What do you think about "git checkout --fix-eols" option as an
alternative? Its uses cases are more limited, though.

--
Best regards,
Marc Strapetz
=============
syntevo GmbH
http://www.syntevo.com
http://blog.syntevo.com



On 13.01.2011 15:37, Michael J Gruber wrote:
> Marc Strapetz venit, vidit, dixit 13.01.2011 15:28:
>>>> case of missing .git/index, Git freshly writes all working tree files,
>>>> ignoring already existing files which already have the correct content.
>>>> Maybe this behavior is by intention and makes sense in some cases. In my
>>>> case it has adverse effects on IDEs and probably other tools which are
>>>> monitoring the file system.
>>>
>>> ...but changing gitattributes is something you don't do routinely in
>>> your workflow; so, at worst there would be an occasional unnecessary run
>>> of your build process.
>>
>> Our Git-SVN bridge does it, potentially on every pull. This is why we
>> currently need to run "rm .git/index && git reset --hard" after every
>> pull, resp. every checkout (switching to another commit may result in
>> changed .gitattributes as well).
> 
> OK, now you're telling us what this is about ;)
> 
>> If a "git checkout" would (optionally) make sure that all EOLs are
>> properly set according to .gitattributes, the problem would be resolved.
>> As this might be not so easy to implement, my suggestion was to make
>> "git reset --hard" work more unobtrusive. I think we can provide a
>> corresponding patch, if it has chances to get accepted.
> 
> There have been other cases where git update-index --really-refresh
> wasn't enough. You might want to check whether that is a suitable "patch
> attack vector". This might be useful not only for you but also for others.
> 
> Michael
> 
> 

Re: Applying .gitattributes text/eol changes

[Junio C Hamano]

Marc Strapetz <marc.strapetz@syntevo.com> writes:

> So your suggestion is to fix "git update-index --really-refresh", so

The option is about telling git: "Earlier I promised I wouldn't touch
these paths by setting their assume-unchanged bit, but I touched them.
Please refresh the cached stat information in the index, ignoring the
promise I didn't keep."

I do not think it is a good idea to conflate your "Everything is suspect
because smudge filter has changed; please recompute all" request into the
same option.  People who use assume-unchanged would probably want "Please
rescan because I changed smudge filter" request to be carried out while
still honoring the assume-unchanged bit they set earlier.

> Anyway, I'm still wondering if it will resolve the "git reset --hard"
> problem of re-checking out every file, even if content is already
> identical in the working tree. I think that part has to be fixed, too.

There is not much to fix there. If you removed the index, then there is no
information to tell you that "content is already identical" unless you
actually check things out and compare.  By the time you found it out, you
already have done the checkout.

IOW, the current code does:

	open object
        read from the object
        deflate and write to the destination file

while your "fix" needs to look like this:

	open object
        read from the object
        deflate and write to a temporary file
        open the existing file
        read from the file and compare it to the temporary we just wrote
        if same, delete, otherwise rename the temporary file.

just for the rare case where there is an untracked file that the user is
willing to overwrite (we are discussing "rm .git/index && reset --hard"
here) happens to have the same contents.  Not a good enough reason to add
unwelcome complexity to the codepath.

> What do you think about "git checkout --fix-eols" option as an
> alternative? Its uses cases are more limited, though.

What does it do?  "git checkout --fix-eols $path" will overwrite $path
with the data at $path in the index?  Perhaps you can use the "-f" option.

Adding an option to "checkout" might be better than update-index from the
UI point of view, but the issue is not just "eols".  "eol" is a mere
special case of smudge filter that controls how the contents from the
repository are modified before getting written out to the working tree.

Re: Applying .gitattributes text/eol changes

[Michael J Gruber]

Junio C Hamano venit, vidit, dixit 14.01.2011 00:30:
> Marc Strapetz <marc.strapetz@syntevo.com> writes:
> 
>> So your suggestion is to fix "git update-index --really-refresh", so
> 
> The option is about telling git: "Earlier I promised I wouldn't touch
> these paths by setting their assume-unchanged bit, but I touched them.
> Please refresh the cached stat information in the index, ignoring the
> promise I didn't keep."
> 
> I do not think it is a good idea to conflate your "Everything is suspect
> because smudge filter has changed; please recompute all" request into the
> same option.  People who use assume-unchanged would probably want "Please
> rescan because I changed smudge filter" request to be carried out while
> still honoring the assume-unchanged bit they set earlier.

What I meant was to introduce a new option --refresh-stat or something.
We have solved the "stale stat info problem" (changed dev nums after
reboot) in a different way meanwhile, but I think there was a different
case (can't come up with the thread right now) where something like this
could have helped.

> 
>> Anyway, I'm still wondering if it will resolve the "git reset --hard"
>> problem of re-checking out every file, even if content is already
>> identical in the working tree. I think that part has to be fixed, too.
> 
> There is not much to fix there. If you removed the index, then there is no
> information to tell you that "content is already identical" unless you
> actually check things out and compare.  By the time you found it out, you
> already have done the checkout.
> 
> IOW, the current code does:
> 
> 	open object
>         read from the object
>         deflate and write to the destination file
> 
> while your "fix" needs to look like this:
> 
> 	open object
>         read from the object
>         deflate and write to a temporary file
>         open the existing file
>         read from the file and compare it to the temporary we just wrote
>         if same, delete, otherwise rename the temporary file.
> 
> just for the rare case where there is an untracked file that the user is
> willing to overwrite (we are discussing "rm .git/index && reset --hard"
> here) happens to have the same contents.  Not a good enough reason to add
> unwelcome complexity to the codepath.
> 
>> What do you think about "git checkout --fix-eols" option as an
>> alternative? Its uses cases are more limited, though.
> 
> What does it do?  "git checkout --fix-eols $path" will overwrite $path
> with the data at $path in the index?  Perhaps you can use the "-f" option.
> 
> Adding an option to "checkout" might be better than update-index from the
> UI point of view, but the issue is not just "eols".  "eol" is a mere
> special case of smudge filter that controls how the contents from the
> repository are modified before getting written out to the working tree.

Exactly, this is a more general issue. The typical answer to these
issues is that you change attributes and filters only occasionally, so
the cost of a rm .git/index && git reset --hard is irrelevant. But still
there should be a less scary way of (really) refreshing the index. Also
note that the cost of the command itself is only a part of the picture -
in the OP's case (which is a bit convoluted, of course) "cost" is really
command execution + the cost of the consequences (rebuilding triggered
by unnecessary touches). For the typical use cases, the existing options
and command paths do the perfectly sane and efficient thing.

Michael

Re: Applying .gitattributes text/eol changes

[Marc Strapetz]

>> What do you think about "git checkout --fix-eols" option as an
>> alternative? Its uses cases are more limited, though.
> 
> What does it do?  "git checkout --fix-eols $path" will overwrite $path
> with the data at $path in the index?  Perhaps you can use the "-f" option.
> 
> Adding an option to "checkout" might be better than update-index from the
> UI point of view, but the issue is not just "eols".  "eol" is a mere
> special case of smudge filter that controls how the contents from the
> repository are modified before getting written out to the working tree.

So there could be a "--thoroughly" option which will skip the stat check
(checkout_entry) and instead perform the procedure already outlined for
rebase:

> 	open object
>         read from the object
>         deflate and write to a temporary file
>         open the existing file
>         read from the file and compare it to the temporary we just wrote
>         if same, delete, otherwise rename the temporary file.

AFAIU, this change will effect mainly checkout struct, checkout_entry
and write_entry. write_entry already deals with temporary files, so that
shouldn't be too complex!?

--
Best regards,
Marc Strapetz
=============
syntevo GmbH
http://www.syntevo.com
http://blog.syntevo.com




On 14.01.2011 00:30, Junio C Hamano wrote:
> Marc Strapetz <marc.strapetz@syntevo.com> writes:
> 
>> So your suggestion is to fix "git update-index --really-refresh", so
> 
> The option is about telling git: "Earlier I promised I wouldn't touch
> these paths by setting their assume-unchanged bit, but I touched them.
> Please refresh the cached stat information in the index, ignoring the
> promise I didn't keep."
> 
> I do not think it is a good idea to conflate your "Everything is suspect
> because smudge filter has changed; please recompute all" request into the
> same option.  People who use assume-unchanged would probably want "Please
> rescan because I changed smudge filter" request to be carried out while
> still honoring the assume-unchanged bit they set earlier.
> 
>> Anyway, I'm still wondering if it will resolve the "git reset --hard"
>> problem of re-checking out every file, even if content is already
>> identical in the working tree. I think that part has to be fixed, too.
> 
> There is not much to fix there. If you removed the index, then there is no
> information to tell you that "content is already identical" unless you
> actually check things out and compare.  By the time you found it out, you
> already have done the checkout.
> 
> IOW, the current code does:
> 
> 	open object
>         read from the object
>         deflate and write to the destination file
> 
> while your "fix" needs to look like this:
> 
> 	open object
>         read from the object
>         deflate and write to a temporary file
>         open the existing file
>         read from the file and compare it to the temporary we just wrote
>         if same, delete, otherwise rename the temporary file.
> 
> just for the rare case where there is an untracked file that the user is
> willing to overwrite (we are discussing "rm .git/index && reset --hard"
> here) happens to have the same contents.  Not a good enough reason to add
> unwelcome complexity to the codepath.
> 
>> What do you think about "git checkout --fix-eols" option as an
>> alternative? Its uses cases are more limited, though.
> 
> What does it do?  "git checkout --fix-eols $path" will overwrite $path
> with the data at $path in the index?  Perhaps you can use the "-f" option.
> 
> Adding an option to "checkout" might be better than update-index from the
> UI point of view, but the issue is not just "eols".  "eol" is a mere
> special case of smudge filter that controls how the contents from the
> repository are modified before getting written out to the working tree.
> 
> 
> 
>