[PATCH] Documentation: enhance gitignore whitelist example

[PATCH] Documentation: enhance gitignore whitelist example

[Eric Blake]

I was trying to whitelist a single file pattern in a directory
that I was otherwise content to ignore, but when I tried:

/m4/
!/m4/virt-*.m4

then 'git add' kept warning me that I had to use -f.  I finally
figured out that ignoring a directory is much different than ignoring
all files in a directory, when it comes to later negation patterns:

/m4/*
!/m4/virt-*.m4

Improving the documentation will help others learn from my mistake.

Signed-off-by: Eric Blake <eblake@redhat.com>
---
 Documentation/gitignore.txt |   19 +++++++++++++++++--
 1 files changed, 17 insertions(+), 2 deletions(-)

diff --git a/Documentation/gitignore.txt b/Documentation/gitignore.txt
index 2e7328b..2f49989 100644
--- a/Documentation/gitignore.txt
+++ b/Documentation/gitignore.txt
@@ -70,7 +70,9 @@ PATTERN FORMAT
  - An optional prefix '!' which negates the pattern; any
    matching file excluded by a previous pattern will become
    included again.  If a negated pattern matches, this will
-   override lower precedence patterns sources.
+   override lower precedence patterns sources.  However, a
+   file negation does not override a path that has already
+   been excluded by a directory match.

  - If the pattern ends with a slash, it is removed for the
    purpose of the following description, but it would only find
@@ -87,7 +89,8 @@ PATTERN FORMAT

  - Otherwise, git treats the pattern as a shell glob suitable
    for consumption by fnmatch(3) with the FNM_PATHNAME flag:
-   wildcards in the pattern will not match a / in the pathname.
+   wildcards in the pattern will not match a / in the pathname,
+   and do not ignore files with a leading . in the pathname.
    For example, "Documentation/{asterisk}.html" matches
    "Documentation/git.html" but not "Documentation/ppc/ppc.html"
    or "tools/perf/Documentation/perf.html".
@@ -116,8 +119,11 @@ EXAMPLES
     [...]
     # Untracked files:
     [...]
+    #       Documentation/build
     #       Documentation/foo.html
     #       Documentation/gitignore.html
+    #       build/log
+    #       build/.file
     #       file.o
     #       lib.a
     #       src/internal.o
@@ -125,6 +131,10 @@ EXAMPLES
     $ cat .git/info/exclude
     # ignore objects and archives, anywhere in the tree.
     *.[oa]
+    # ignore files in the immediate child directory build,
+    /build/*
+    # except for the log.
+    !/build/log
     $ cat Documentation/.gitignore
     # ignore generated html files,
     *.html
@@ -134,10 +144,15 @@ EXAMPLES
     [...]
     # Untracked files:
     [...]
+    #       Documentation/build
     #       Documentation/foo.html
+    #       build/log
     [...]
 --------------------------------------------------------------

+Note that using `!/build/log' works with an earlier `/build/*' but
+would have no effect if there were an earlier `/build/'.
+
 Another example:

 --------------------------------------------------------------
-- 
1.7.4

Re: [PATCH] Documentation: enhance gitignore whitelist example

[Jonathan Nieder]

Eric Blake wrote:

> I was trying to whitelist a single file pattern in a directory
> that I was otherwise content to ignore, but when I tried:
> 
> /m4/
> !/m4/virt-*.m4
> 
> then 'git add' kept warning me that I had to use -f.  I finally
> figured out that ignoring a directory is much different than ignoring
> all files in a directory, when it comes to later negation patterns:
> 
> /m4/*
> !/m4/virt-*.m4
> 
> Improving the documentation will help others learn from my mistake.
> 
> Signed-off-by: Eric Blake <eblake@redhat.com>

Yes.

Acked-by: Jonathan Nieder <jrnieder@gmail.com>

Cc-ing Hannes, in case he has thoughts on how to explain this more
intuitively.

> ---
>  Documentation/gitignore.txt |   19 +++++++++++++++++--
>  1 files changed, 17 insertions(+), 2 deletions(-)
> 
> diff --git a/Documentation/gitignore.txt b/Documentation/gitignore.txt
> index 2e7328b..2f49989 100644
> --- a/Documentation/gitignore.txt
> +++ b/Documentation/gitignore.txt
> @@ -70,7 +70,9 @@ PATTERN FORMAT
>   - An optional prefix '!' which negates the pattern; any
>     matching file excluded by a previous pattern will become
>     included again.  If a negated pattern matches, this will
> -   override lower precedence patterns sources.
> +   override lower precedence patterns sources.  However, a
> +   file negation does not override a path that has already
> +   been excluded by a directory match.
> 
>   - If the pattern ends with a slash, it is removed for the
>     purpose of the following description, but it would only find
> @@ -87,7 +89,8 @@ PATTERN FORMAT
> 
>   - Otherwise, git treats the pattern as a shell glob suitable
>     for consumption by fnmatch(3) with the FNM_PATHNAME flag:
> -   wildcards in the pattern will not match a / in the pathname.
> +   wildcards in the pattern will not match a / in the pathname,
> +   and do not ignore files with a leading . in the pathname.
>     For example, "Documentation/{asterisk}.html" matches
>     "Documentation/git.html" but not "Documentation/ppc/ppc.html"
>     or "tools/perf/Documentation/perf.html".
> @@ -116,8 +119,11 @@ EXAMPLES
>      [...]
>      # Untracked files:
>      [...]
> +    #       Documentation/build
>      #       Documentation/foo.html
>      #       Documentation/gitignore.html
> +    #       build/log
> +    #       build/.file
>      #       file.o
>      #       lib.a
>      #       src/internal.o
> @@ -125,6 +131,10 @@ EXAMPLES
>      $ cat .git/info/exclude
>      # ignore objects and archives, anywhere in the tree.
>      *.[oa]
> +    # ignore files in the immediate child directory build,
> +    /build/*
> +    # except for the log.
> +    !/build/log
>      $ cat Documentation/.gitignore
>      # ignore generated html files,
>      *.html
> @@ -134,10 +144,15 @@ EXAMPLES
>      [...]
>      # Untracked files:
>      [...]
> +    #       Documentation/build
>      #       Documentation/foo.html
> +    #       build/log
>      [...]
>  --------------------------------------------------------------
> 
> +Note that using `!/build/log' works with an earlier `/build/*' but
> +would have no effect if there were an earlier `/build/'.
> +
>  Another example:
> 
>  --------------------------------------------------------------
> -- 
> 1.7.4
> 

Re: [PATCH] Documentation: enhance gitignore whitelist example

[Junio C Hamano]

Eric Blake <eblake@redhat.com> writes:

> I was trying to whitelist a single file pattern in a directory
> that I was otherwise content to ignore, but when I tried:
>
> /m4/
> !/m4/virt-*.m4

Please always indent displayed examples in commit log messages for
readability.

> then 'git add' kept warning me that I had to use -f.  I finally
> figured out that ignoring a directory is much different than ignoring
> all files in a directory, when it comes to later negation patterns:
>
> /m4/*
> !/m4/virt-*.m4
>
> Improving the documentation will help others learn from my mistake.
>
> Signed-off-by: Eric Blake <eblake@redhat.com>
> ---
>  Documentation/gitignore.txt |   19 +++++++++++++++++--
>  1 files changed, 17 insertions(+), 2 deletions(-)
>
> diff --git a/Documentation/gitignore.txt b/Documentation/gitignore.txt
> index 2e7328b..2f49989 100644
> --- a/Documentation/gitignore.txt
> +++ b/Documentation/gitignore.txt
> @@ -70,7 +70,9 @@ PATTERN FORMAT
>   - An optional prefix '!' which negates the pattern; any
>     matching file excluded by a previous pattern will become
>     included again.  If a negated pattern matches, this will
> -   override lower precedence patterns sources.
> +   override lower precedence patterns sources.  However, a
> +   file negation does not override a path that has already
> +   been excluded by a directory match.

It may be better to say "However X doesn't do Y" than not saying anything,
but can't we phrase this more like "If you want to do Y, you need to do Z
also/instead"?  It would be much more useful for people who are looking
for a way to do Y if you didn't stop at saying "X is not the way to do
it", and said "Do X and Z if you want to achieve Y", no?

On the other hand, if you are trying to explain why X doesn't do Y, the
above would need a bit more explanation (e.g. "when a directory matches an
ignore pattern, it tells git not to descend into the directory to find
ignored or unignored paths in it" or something like that).

> @@ -125,6 +131,10 @@ EXAMPLES
>      $ cat .git/info/exclude
>      # ignore objects and archives, anywhere in the tree.
>      *.[oa]
> +    # ignore files in the immediate child directory build,
> +    /build/*
> +    # except for the log.
> +    !/build/log

In the patch form it is clear these two lines go together, but the
correspondence does not stand out in the text after the patch is applied.

Perhaps doing it like this would make it clearer?

> +    # ignore files in the immediate child directory build, ...
> +    /build/*
> +    # ... except for the log.
> +    !/build/log

Re: [PATCH] Documentation: enhance gitignore whitelist example

[Johannes Sixt]

On Dienstag, 5. April 2011, Jonathan Nieder wrote:
> Eric Blake wrote:
> > @@ -70,7 +70,9 @@ PATTERN FORMAT
> >   - An optional prefix '!' which negates the pattern; any
> >     matching file excluded by a previous pattern will become
> >     included again.  If a negated pattern matches, this will
> > -   override lower precedence patterns sources.
> > +   override lower precedence patterns sources.  However, a
> > +   file negation does not override a path that has already
> > +   been excluded by a directory match.

I don't think this is the right place to explain this caveat. Here we describe 
the format and behavior of the patterns in a rather formal manner.

> > @@ -87,7 +89,8 @@ PATTERN FORMAT
> >
> >   - Otherwise, git treats the pattern as a shell glob suitable
> >     for consumption by fnmatch(3) with the FNM_PATHNAME flag:
> > -   wildcards in the pattern will not match a / in the pathname.
> > +   wildcards in the pattern will not match a / in the pathname,
> > +   and do not ignore files with a leading . in the pathname.

I don't think this is correct. * matches .gitignore. I tried it.

> > @@ -116,8 +119,11 @@ EXAMPLES
> >      [...]
> >      # Untracked files:
> >      [...]
> > +    #       Documentation/build
> >      #       Documentation/foo.html
> >      #       Documentation/gitignore.html
> > +    #       build/log
> > +    #       build/.file
> >      #       file.o
> >      #       lib.a
> >      #       src/internal.o
> > @@ -125,6 +131,10 @@ EXAMPLES
> >      $ cat .git/info/exclude
> >      # ignore objects and archives, anywhere in the tree.
> >      *.[oa]
> > +    # ignore files in the immediate child directory build,
> > +    /build/*
> > +    # except for the log.
> > +    !/build/log

Doesn't this example give the false impression that you could do

	/foo/*
	!/foo/bar/baz

and have foo/bar/baz not ignored? But it is still ignored.

I propose a paragraph like this in the NOTES section:

--- 8< ---
When a directory is ignored, it is not possible to un-ignore a single file 
somewhere in the directory using another pattern. E.g., with the patterns

--------------
/build/
!/build/tests/results
--------------

the file "build/tests/results" is still ignored because when a directory is 
ignored, its contents are never investigated. In a situation where a few 
exceptions in an otherwise ignored hierarchy are needed, the recommended 
procedure is to specify to ignore the root of the hierarchy and then to 'git 
add -f' the exceptional files. Subsequent changes to the files will not be 
ignored.
--- 8< ---

-- Hannes

Re: [PATCH] Documentation: enhance gitignore whitelist example

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 2184 bytes --]

On 04/05/2011 03:15 PM, Johannes Sixt wrote:
>>> @@ -87,7 +89,8 @@ PATTERN FORMAT
>>>
>>>   - Otherwise, git treats the pattern as a shell glob suitable
>>>     for consumption by fnmatch(3) with the FNM_PATHNAME flag:
>>> -   wildcards in the pattern will not match a / in the pathname.
>>> +   wildcards in the pattern will not match a / in the pathname,
>>> +   and do not ignore files with a leading . in the pathname.
> 
> I don't think this is correct. * matches .gitignore. I tried it.

That was my point.  * _does_ match .gitignore, even though for normal
shell globs, FNM_PERIOD is set and * does not match .gitignore.  That
is, while in the shell 'dir/*' only matches non-dot files, in .gitignore
it matches all files including dot-files.

Any ideas for a better way to word that?

> I propose a paragraph like this in the NOTES section:
> 
> --- 8< ---
> When a directory is ignored, it is not possible to un-ignore a single file 
> somewhere in the directory using another pattern. E.g., with the patterns
> 
> --------------
> /build/
> !/build/tests/results
> --------------
> 
> the file "build/tests/results" is still ignored because when a directory is 
> ignored, its contents are never investigated. In a situation where a few 
> exceptions in an otherwise ignored hierarchy are needed, the recommended 
> procedure is to specify to ignore the root of the hierarchy and then to 'git 
> add -f' the exceptional files. Subsequent changes to the files will not be 
> ignored.

Yeah, but then you have to 'git add -f path/to/file' them every time you
change them, or use the sledgehammer of 'git add .'.

Does it make any better sense to document:

  /build/*
  !/build/*/
  /build/*/*
  !/build/foo/baz

which ignores all files in build, then un-ignores directories, then
ignores all files in subdirectories of build except for the desired
multi-level file under build?  At which point you no longer need 'git
add -f', but can simply do 'git add build' to pick up /build/foo/baz in
one go without warning?

-- 
Eric Blake   eblake@redhat.com    +1-801-349-2682
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 619 bytes --]

Re: [PATCH] Documentation: enhance gitignore whitelist example

[Junio C Hamano]

Johannes Sixt <j6t@kdbg.org> writes:

>> > @@ -87,7 +89,8 @@ PATTERN FORMAT
>> >
>> >   - Otherwise, git treats the pattern as a shell glob suitable
>> >     for consumption by fnmatch(3) with the FNM_PATHNAME flag:
>> > -   wildcards in the pattern will not match a / in the pathname.
>> > +   wildcards in the pattern will not match a / in the pathname,
>> > +   and do not ignore files with a leading . in the pathname.
>
> I don't think this is correct. * matches .gitignore. I tried it.

It is badly phrased to begin with.

It shouldn't say "do not ignore", which is determined by the presense or
the lack of the leading '!' before the pattern.  The pattern either
matches or does not match paths that begin with dot.

I think we pass FNM_PATHNAME but not FNM_PERIOD, so * would match a period
hence ".gitignore".  What Eric wrote may be correct in the sense that "the
pattern matcher does not _ignore_ .gitignore as not matching but catches it
with '*'", but as the result the path ends up getting ignored ;-)

> I propose a paragraph like this in the NOTES section:

I think it makes sense to have something like that, but please state it
like "to do Y, do X and Z" (optionally followed by "with just X, Y won't
happen because..., hence you need to do Z as well"), instead of starting
with "you cannot do Y with only X".

IOW, start from a positive and helpful recipe, and then explain why "you
cannot do Y with only X" after that.

> --- 8< ---
> When a directory is ignored, it is not possible to un-ignore a single file 
> somewhere in the directory using another pattern. E.g., with the patterns
>
> --------------
> /build/
> !/build/tests/results
> --------------
>
> the file "build/tests/results" is still ignored because when a directory is 
> ignored, its contents are never investigated. In a situation where a few 
> exceptions in an otherwise ignored hierarchy are needed, the recommended 
> procedure is to specify to ignore the root of the hierarchy and then to 'git 
> add -f' the exceptional files. Subsequent changes to the files will not be 
> ignored.
> --- 8< ---

Re: [PATCH] Documentation: enhance gitignore whitelist example

[Jonathan Nieder]

Eric Blake wrote:

> Yeah, but then you have to 'git add -f path/to/file' them every time you
> change them

No, I don't believe that's true.

 $ git add -f git.o
 $ >git.o
 $ git add git.o

.gitignore only protects against starting to track a file that was
previously untracked.

I do tend to use exceptions in .gitignore anyway, since (1) it allows,
as a sanity check,

 $ git ls-files -i --exclude-standard

and (2) to import from or compare to a tarball, one can do

 $ git rm -rf .
 $ git add .

Re: [PATCH] Documentation: enhance gitignore whitelist example

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 2822 bytes --]

On 04/05/2011 03:41 PM, Jonathan Nieder wrote:
> Eric Blake wrote:
> 
>> Yeah, but then you have to 'git add -f path/to/file' them every time you
>> change them
> 
> No, I don't believe that's true.
> 
>  $ git add -f git.o
>  $ >git.o
>  $ git add git.o

Aha - it's that pesky dir/ vs. dir/* biting me, yet again:

$ mkdir -p /tmp/blah
$ cd /tmp/blah
$ git init
Initialized empty Git repository in /tmp/blah/.git/
$ mkdir sub
$ > sub/file
$ git add sub/file
$ git commit -a -m 'one'
[master (root-commit) 645ee5a] one
 0 files changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 sub/file
$ printf 'sub/*\n!sub/file\n' > .gitignore
$ touch sub/file2
$ echo hi > sub/file
$ git status
# On branch master
# Changes not staged for commit:
#   (use "git add <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working
directory)
#
#	modified:   sub/file
#
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#	.gitignore
no changes added to commit (use "git add" and/or "git commit -a")
$ git add sub
$ git status
# On branch master
# Changes to be committed:
#   (use "git reset HEAD <file>..." to unstage)
#
#	modified:   sub/file
#
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#	.gitignore
$ git reset
Unstaged changes after reset:
M	sub/file
$ printf 'sub/\n!sub/file\n' > .gitignore
$ git add sub
The following paths are ignored by one of your .gitignore files:
sub
Use -f if you really want to add them.
fatal: no files added
$ git add sub/file
The following paths are ignored by one of your .gitignore files:
sub
Use -f if you really want to add them.
fatal: no files added
$ git status
# On branch master
# Changes not staged for commit:
#   (use "git add <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working
directory)
#
#	modified:   sub/file
#
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#	.gitignore
no changes added to commit (use "git add" and/or "git commit -a")
$ git add .
$ git status
# On branch master
# Changes to be committed:
#   (use "git reset HEAD <file>..." to unstage)
#
#	new file:   .gitignore
#	modified:   sub/file
#

> 
> .gitignore only protects against starting to track a file that was
> previously untracked.

Not quite.  When filtering a directory, it also protects against changes
to tracked files in that directory.  And that is what has been throwing
me off, which is why we need a doc change (or possibly even a behavior
change).

-- 
Eric Blake   eblake@redhat.com    +1-801-349-2682
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 619 bytes --]

Re: [PATCH] Documentation: enhance gitignore whitelist example

[Junio C Hamano]

Eric Blake <eblake@redhat.com> writes:

> On 04/05/2011 03:15 PM, Johannes Sixt wrote:
>>>> @@ -87,7 +89,8 @@ PATTERN FORMAT
>>>>
>>>>   - Otherwise, git treats the pattern as a shell glob suitable
>>>>     for consumption by fnmatch(3) with the FNM_PATHNAME flag:
>>>> -   wildcards in the pattern will not match a / in the pathname.
>>>> +   wildcards in the pattern will not match a / in the pathname,
>>>> +   and do not ignore files with a leading . in the pathname.
>> 
>> I don't think this is correct. * matches .gitignore. I tried it.
>
> That was my point.  * _does_ match .gitignore, even though for normal
> shell globs, FNM_PERIOD is set and * does not match .gitignore.  That
> is, while in the shell 'dir/*' only matches non-dot files, in .gitignore
> it matches all files including dot-files.
>
> Any ideas for a better way to word that?

Instead of "and do not ignore files with a leading", say "but will match
a dot '.'".  You are talking about the rule for wildcards to match or not
to match the pathname and there is no room for the word "ignore" to come
into play in this sentence.