How to simulate a real checkout to test a new smudge filter?

How to simulate a real checkout to test a new smudge filter?

[john smith]

I am looking for a way to force smudge filter to run by simulating a
real life checkout. Let's say I just created a new branch and did not
modify any files but want to test my new smudge filter. According to
some answers such as
https://stackoverflow.com/questions/22909620/git-smudge-clean-filter-between-branches
and
https://stackoverflow.com/questions/21652242/git-re-checkout-files-after-creating-smudge-filter
it should be possible by running:

git checkout HEAD --

but in doesn't work with git 2.9.0. Method suggested in accepted
answer here
https://stackoverflow.com/questions/17223527/how-do-i-force-git-to-checkout-the-master-branch-and-remove-carriage-returns-aft
works but I don't like because it seems fragile. Is there a safe way
to do what I want to do in Git still today?

-- 
<wempwer@gmail.com>

Re: How to simulate a real checkout to test a new smudge filter?

[Torsten Bögershausen]

On 06.09.16 19:47, john smith wrote:
> I am looking for a way to force smudge filter to run by simulating a
> real life checkout. Let's say I just created a new branch and did not
> modify any files but want to test my new smudge filter. According to
> some answers such as
> https://stackoverflow.com/questions/22909620/git-smudge-clean-filter-between-branches
> and
> https://stackoverflow.com/questions/21652242/git-re-checkout-files-after-creating-smudge-filter
> it should be possible by running:
>
> git checkout HEAD --
>
> but in doesn't work with git 2.9.0. Method suggested in accepted
> answer here
> https://stackoverflow.com/questions/17223527/how-do-i-force-git-to-checkout-the-master-branch-and-remove-carriage-returns-aft
> works but I don't like because it seems fragile. Is there a safe way
> to do what I want to do in Git still today?
>
It depends what you mean with "safe way".

git checkout, git checkout -f or other combinations will only

overwrite/rewrite the files in the working tree, if, and only if,

git comes to the conclusion that "git add" will do something,

like replace a blob for a file in the index.

(And by running "rm .git/index git will evaluate the "clean" filters,

and the CRLF->LF conversion).


If you want to test a smudge filter, simply remove the file:

mv file /tmp && git checkout file

Re: How to simulate a real checkout to test a new smudge filter?

[john smith]

On 9/6/16, Torsten Bögershausen <tboegi@web.de> wrote:
> On 06.09.16 19:47, john smith wrote:
>> I am looking for a way to force smudge filter to run by simulating a
>> real life checkout. Let's say I just created a new branch and did not
>> modify any files but want to test my new smudge filter. According to
>> some answers such as
>> https://stackoverflow.com/questions/22909620/git-smudge-clean-filter-between-branches
>> and
>> https://stackoverflow.com/questions/21652242/git-re-checkout-files-after-creating-smudge-filter
>> it should be possible by running:
>>
>> git checkout HEAD --
>>
>> but in doesn't work with git 2.9.0. Method suggested in accepted
>> answer here
>> https://stackoverflow.com/questions/17223527/how-do-i-force-git-to-checkout-the-master-branch-and-remove-carriage-returns-aft
>> works but I don't like because it seems fragile. Is there a safe way
>> to do what I want to do in Git still today?
>>
> It depends what you mean with "safe way".

I want to store all my dotfiles in a single repoitory.  The problem is
that that some specific pieces of these files are different on
different machines.  I have a special .conf file that is different on
every branch and contains machine-specific definitions of some
variables such as EMAIL or SMTP server.  In my smudge filter I call a
script which parses .conf file and replace all template variable
definitions saved in the given file with correct definitions.  For
example in my ~/.bashrc I have this on all branches:

export EMAIL="@EMAIL@"

and in my .conf file on `home' branch

EMAIL=home@address.com

and on `work' branch:

EMAIL=work@address.com

And in .gitattributes on both branches:

bash/.bashrc     filter=make-usable

I also have single `master' branch that only contains template
dotfiles and no .conf.  When setting up a new machine I could just
create a new branch off master branch and add a new .conf.

In turn, clean filter replace all correct definitions in the given
dotfiles back into template definitions.

I'd prefer smudge/clean filters instead of `make' scripts etc. to
convert template dotfiles into something usable and back because
filters:

1. could be run automatically

2. do not modify files as shown by `git show HEAD:<file>' and
therefore no files are reported as modified by git status and also
there are not conflicts when merging master into work/home branch.

I have problems because with point 1 because apparently smudge filter
is not run automatically every time when branch is changed if files
listed in .gitattributes do not change. As the last resort I could
force smudge/clean filter to run just to keep advantage specified in
point 2.

-- 
<wempwer@gmail.com>

Re: How to simulate a real checkout to test a new smudge filter?

[Michael J Gruber]

john smith venit, vidit, dixit 06.09.2016 23:01:
> On 9/6/16, Torsten Bögershausen <tboegi@web.de> wrote:
>> On 06.09.16 19:47, john smith wrote:
>>> I am looking for a way to force smudge filter to run by simulating a
>>> real life checkout. Let's say I just created a new branch and did not
>>> modify any files but want to test my new smudge filter. According to
>>> some answers such as
>>> https://stackoverflow.com/questions/22909620/git-smudge-clean-filter-between-branches
>>> and
>>> https://stackoverflow.com/questions/21652242/git-re-checkout-files-after-creating-smudge-filter
>>> it should be possible by running:
>>>
>>> git checkout HEAD --
>>>
>>> but in doesn't work with git 2.9.0. Method suggested in accepted
>>> answer here
>>> https://stackoverflow.com/questions/17223527/how-do-i-force-git-to-checkout-the-master-branch-and-remove-carriage-returns-aft
>>> works but I don't like because it seems fragile. Is there a safe way
>>> to do what I want to do in Git still today?
>>>
>> It depends what you mean with "safe way".
> 
> I want to store all my dotfiles in a single repoitory.  The problem is
> that that some specific pieces of these files are different on
> different machines.  I have a special .conf file that is different on
> every branch and contains machine-specific definitions of some
> variables such as EMAIL or SMTP server.  In my smudge filter I call a
> script which parses .conf file and replace all template variable
> definitions saved in the given file with correct definitions.  For
> example in my ~/.bashrc I have this on all branches:
> 
> export EMAIL="@EMAIL@"
> 
> and in my .conf file on `home' branch
> 
> EMAIL=home@address.com
> 
> and on `work' branch:
> 
> EMAIL=work@address.com
> 
> And in .gitattributes on both branches:
> 
> bash/.bashrc     filter=make-usable
> 
> I also have single `master' branch that only contains template
> dotfiles and no .conf.  When setting up a new machine I could just
> create a new branch off master branch and add a new .conf.
> 
> In turn, clean filter replace all correct definitions in the given
> dotfiles back into template definitions.
> 
> I'd prefer smudge/clean filters instead of `make' scripts etc. to
> convert template dotfiles into something usable and back because
> filters:
> 
> 1. could be run automatically
> 
> 2. do not modify files as shown by `git show HEAD:<file>' and
> therefore no files are reported as modified by git status and also
> there are not conflicts when merging master into work/home branch.
> 
> I have problems because with point 1 because apparently smudge filter
> is not run automatically every time when branch is changed if files
> listed in .gitattributes do not change. As the last resort I could
> force smudge/clean filter to run just to keep advantage specified in
> point 2.

I'm afraid smudge/clean are not a good match for your use case:
smudge can do anything that clean removes again, i.e.: if you smudge a
blob, then clean it, it has to be the same blob.

Your smudge filter seems to give different results for the same blob
depending on other variables (your .conf), but git doesn't care as long
as clean output is the same. Also, git cannot possibly know when "it is
time" to rerun smudge.

Maybe "source"-ing .conf files would be an alternative approach for you,
or using hooks?

Cheers
Michael

Re: How to simulate a real checkout to test a new smudge filter?

[Jakub Narębski]

W dniu 06.09.2016 o 23:01, john smith pisze:

> I'd prefer smudge/clean filters instead of `make' scripts etc. to
> convert template dotfiles into something usable and back because
> filters:
> 
> 1. could be run automatically
> 
> 2. do not modify files as shown by `git show HEAD:<file>' and
> therefore no files are reported as modified by git status and also
> there are not conflicts when merging master into work/home branch.
> 
> I have problems because with point 1 because apparently smudge filter
> is not run automatically every time when branch is changed if files
> listed in .gitattributes do not change. As the last resort I could
> force smudge/clean filter to run just to keep advantage specified in
> point 2.

Couldn't you use post-checkout hook plus clean filter instead of
clean/smudge filter pair, if the smudge part depends on the branch?

Or make post-checkout hook invoke smudge filter... though
`git cat-file --filters` is not in any released version, I think...

Best,
-- 
Jakub Narębski

Re: How to simulate a real checkout to test a new smudge filter?

[john smith]

On 9/8/16, Jakub Narębski <jnareb@gmail.com> wrote:
> W dniu 06.09.2016 o 23:01, john smith pisze:
>
>> I'd prefer smudge/clean filters instead of `make' scripts etc. to
>> convert template dotfiles into something usable and back because
>> filters:
>>
>> 1. could be run automatically
>>
>> 2. do not modify files as shown by `git show HEAD:<file>' and
>> therefore no files are reported as modified by git status and also
>> there are not conflicts when merging master into work/home branch.
>>
>> I have problems because with point 1 because apparently smudge filter
>> is not run automatically every time when branch is changed if files
>> listed in .gitattributes do not change. As the last resort I could
>> force smudge/clean filter to run just to keep advantage specified in
>> point 2.
>
> Couldn't you use post-checkout hook plus clean filter instead of
> clean/smudge filter pair, if the smudge part depends on the branch?
>
> Or make post-checkout hook invoke smudge filter... though
> `git cat-file --filters` is not in any released version, I think...

I moved smudge-filter functionality to post-checkout hook and only
left clean filter. Now post-checkout hooks is called every time when
changing branches but it's not working together with clean filter when
changing branches.  For example, when I have this on master in
bash/.bashrc:

export EMAIL="@EMAIL@"

After moving to `work' it becomes this thanks to post-checkout hook
which replaces @EMAIL@ with correct definition:

export EMAIL="work@address.com"

But when I move to `home' branch from `work' I still get this:

export EMAIL="work@address.com"

post-checkout didn't work here because it can't see @EMAIL@ part in
bash/.bashrc because clean filter has not been run. I need to run this
single command to force `clean' filter to run to replace
work@address.com with @EMAIL@ and let post-checkout replace it with
correct value:

git checkout .

And now I get the correct definition:

export EMAIL="home@address.com"

So it seems that clean filter is only run when checking out paths but
not when checking out branches.  Is my thinking correct?
-- 
<wempwer@gmail.com>

Re: How to simulate a real checkout to test a new smudge filter?

[Junio C Hamano]

john smith <wempwer@gmail.com> writes:

> So it seems that clean filter is only run when checking out paths but
> not when checking out branches.  Is my thinking correct?

If the new branch you are going to and your current branch record
different contents for a path, that path will have to be checked out
to the working tree, and smudge will be involved in the process to
turn the "clean" contents stored in a Git object needs to be smudged
for use in the outside world.  If both branches have the same
contents, then there is no need to overwrite the path in the working
tree with the same thing, so it is not touched.

The clean and smudge operations should look _only_ at the contents
they are filtering, and nothing else, and the clean/smudge filtering
mechanism is designed to support that use case.  It is not designed
to do things like embedding the name of the branch that is being
checked out into the result.

Re: How to simulate a real checkout to test a new smudge filter?

[john smith]

On 9/10/16, Junio C Hamano <gitster@pobox.com> wrote:
> The clean and smudge operations should look _only_ at the contents
> they are filtering, and nothing else, and the clean/smudge filtering
> mechanism is designed to support that use case.  It is not designed
> to do things like embedding the name of the branch that is being
> checked out into the result.

Ok, I think I get it. It was actually my original problem with smudge
filters because I wanted them to be run after *every* checkout even if
file contents stayed the same (hence the subject).

Now Jakub suggested post-checkout hook in conjunction with only clean
filter for my problem of managing dotfiles but I think I don't fully
get it.  The problem is that in the scenario presented in my last
e-mail clean filter is run in the situation which doesn't like a
checkin to me.  Is `git checkout <PATH>' doing a *checkin*" under the
hood so that the clean filter is called?  What does actually `checkin'
mean in Git?  I thought that checkin it's the same as committing a
file into the repository.

-- 
<wempwer@gmail.com>

Re: How to simulate a real checkout to test a new smudge filter?

[Junio C Hamano]

john smith <wempwer@gmail.com> writes:

> ... get it.  The problem is that in the scenario presented in my last
> e-mail clean filter is run in the situation which doesn't like a
> checkin to me.  Is `git checkout <PATH>' doing a *checkin*" under the
> hood so that the clean filter is called?  What does actually `checkin'
> mean in Git?  I thought that checkin it's the same as committing a
> file into the repository.

I do not know what Jakub meant.

When we need to see if we need to update a working tree file, and
when we cannot tell if you modified the working tree file since you
checked it out, we may have to run the clean filter (and other
conversions collectively called "convert-to-git") on the contents in
it and see if the resulting blob matches what is recorded in the
index (if they match, there is no need to update the working tree
file; if they don't match, we either report that you have a local
modification, or we overwrite it with the contents, depending on
what operation we are doing for the user).

Re: How to simulate a real checkout to test a new smudge filter?

[Jakub Narębski]

W dniu 10.09.2016 o 01:07, john smith pisze:
> On 9/10/16, Junio C Hamano <gitster@pobox.com> wrote:
>> The clean and smudge operations should look _only_ at the contents
>> they are filtering, and nothing else, and the clean/smudge filtering
>> mechanism is designed to support that use case.  It is not designed
>> to do things like embedding the name of the branch that is being
>> checked out into the result.
> 
> Ok, I think I get it. It was actually my original problem with smudge
> filters because I wanted them to be run after *every* checkout even if
> file contents stayed the same (hence the subject).

And that's not the case, for performance reasons.

> 
> Now Jakub suggested post-checkout hook in conjunction with only clean
> filter for my problem of managing dotfiles but I think I don't fully
> get it.  The problem is that in the scenario presented in my last
> e-mail clean filter is run in the situation which doesn't like a
> checkin to me.  Is `git checkout <PATH>' doing a *checkin*" under the
> hood so that the clean filter is called?  What does actually `checkin'
> mean in Git?  I thought that checkin it's the same as committing a
> file into the repository.

I was wrong, I'm sorry. My mistake.

You would need post-checkout hook together with clean / smudge filters
(though you could get by without smudge filter, at least in theory...).
The `post-checkout` hook could run e.g. "git checkout -- '*.conf'"
to force use of smudge filter, after checking that it was a change
of branch ("git checkout <commit-ish>").  See githooks(5) for details
(last, third parameter passed to hook script is 1, rather than 0).

Unfortunately I don't see a way to query .gitattributes files to
find out all patterns that match specific attribute; you would need
to duplicate configuration:

  .gitattributes:
  *.conf filter=transform

  .git/config
  [filter "transform"]
  	clean  = replace-with-placeholder %f
  	smudge = expand-with-branchname %f

  .git/hooks/post-checkout
  #!/bin/sh

  test "$3" -eq "1" && git checkout -- '*.conf'

Or something like that.
-- 
Jakub Narębski

Re: How to simulate a real checkout to test a new smudge filter?

[john smith]

On 9/10/16, Jakub Narębski <jnareb@gmail.com> wrote:
> You would need post-checkout hook together with clean / smudge filters
> (though you could get by without smudge filter, at least in theory...).
> The `post-checkout` hook could run e.g. "git checkout -- '*.conf'"
> to force use of smudge filter, after checking that it was a change
> of branch ("git checkout <commit-ish>").  See githooks(5) for details
> (last, third parameter passed to hook script is 1, rather than 0).

Neat idea.  I could still have an advantage of filter as git status
wouldn't show that files are changed and there would be no merge
conflicts and additionally filters would run after every checkout even
if files stayed the same.  But are sure that `git checkout -- <FILE>'
actually forces checkout and runs smudge filter?  It doesn't work for
me neither with 2.9.0 nor 2.10.0.85.gcda1bbd.  For a minimal example,
let's say that I have this in .gitattributes:

a     filter=test-filter

And this in .git/config:

smudge = touch /tmp/SMUDGE_RUN && cat

And also have a tracked file named `a' in the repository. Now this
should create /tmp/SMUDGE_RUN:

git checkout  -- a

but it doesn't.

This also doesn't work:

git checkout -f -- a

But this does:

git checkout HEAD^ && git co -

> Unfortunately I don't see a way to query .gitattributes files to
> find out all patterns that match specific attribute; you would need
> to duplicate configuration:
>
>   .gitattributes:
>   *.conf filter=transform
>
>   .git/config
>   [filter "transform"]
>   	clean  = replace-with-placeholder %f
>   	smudge = expand-with-branchname %f
>
>   .git/hooks/post-checkout
>   #!/bin/sh
>
>   test "$3" -eq "1" && git checkout -- '*.conf'

It's a secondary issue, but - in my repository I only have one .conf
file on every branch with real definitions of machine-specific
variables and a number of various config files for various programs
such as bash/.bashrc or screen/.screenrc so in my case it would be:

  .gitattributes:
  bash/.basrc filter=transform
  git/.gitconfig filter=transform
  (...)

  .git/config
  [filter "transform"]
        clean  = replace-with-placeholder %f
        smudge = expand-with-branchname %f

  .git/hooks/post-checkout
  #!/bin/sh

  test "$3" -eq "1" && (
       git checkout -- bash/.bashrc &&
       git checkout -- git/.gitconfig
       (...)
       )

Of course a list of files in .git/hooks/post-checkout could be
generated at runtime from first column of .gitattributes.

-- 
<wempwer@gmail.com>