[RFC/PATCH 0/5] gitignore.5 clarifications

[RFC/PATCH 0/5] gitignore.5 clarifications

[Jonathan Nieder]

Hi,

Y.G. noticed that the explanation of '/' handling in the gitignore
docs are hard to follow.  Worse, it's completely wrong in a number
of ways.

Here's a series to fix a few of those.  Impact:

 - remove some complete nonsense (how did I mislead myself into
   thinking patterns without a '/' were anchored?)

 - address the frequently asked question "how do I un-ignore part of a
   directory I have ignored?"

 - relive what seems to be a bug (even when the "foo/" directory is
   ignored, a "git add foo/bar" should add new changes to the index
   when that file is already tracked)

 - some minor clarity improvements

At this point, I can hardly trust myself, so careful review for
correctness and clarity would be very welcome.

Incorporates material from
http://thread.gmane.org/gmane.comp.version-control.git/170907/focus=170916

Johannes Sixt (1):
  Documentation/gitignore: explain how to un-ignore part of a directory

Jonathan Nieder (3):
  Documentation/gitignore: "foo/" patterns match directories, not files
    under them
  Documentation: clarify effect of '/' in gitignore(5) patterns
  Documentation: unanchored gitignore patterns match basename

 Documentation/gitignore.txt |   48 ++++++++++++++++++++++++++++++------------
 1 files changed, 34 insertions(+), 14 deletions(-)

[PATCH 1/4] Documentation/gitignore: "foo/" patterns match directories, not files under them

[Jonathan Nieder]

The gitignore(5) manpage says that "foo/" will match a directory foo
and paths underneath it.  But that is completely false: as Johannes
Sixt likes to remind us, patterns with a trailing '/' match the named
directory, not files under that directory.  For example, the following
.gitignore file

	/build/
	!/build/tests/results

does not un-ignore build/tests/results since it was never ignored in
the first place; and commands like "git status" will not notice
changes to build/tests/results because git doesn't enter the (ignored)
build/ directory.

Correct the manual to just say that "foo/" matches the directory
"foo", and make the wording a little clearer in other ways while at
it.

Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 Documentation/gitignore.txt |   14 ++++++++------
 1 files changed, 8 insertions(+), 6 deletions(-)

diff --git a/Documentation/gitignore.txt b/Documentation/gitignore.txt
index 2e7328b8..5b070bf0 100644
--- a/Documentation/gitignore.txt
+++ b/Documentation/gitignore.txt
@@ -72,12 +72,14 @@ PATTERN FORMAT
    included again.  If a negated pattern matches, this will
    override lower precedence patterns sources.
 
- - If the pattern ends with a slash, it is removed for the
-   purpose of the following description, but it would only find
-   a match with a directory.  In other words, `foo/` will match a
-   directory `foo` and paths underneath it, but will not match a
-   regular file or a symbolic link `foo` (this is consistent
-   with the way how pathspec works in general in git).
+ - If the pattern ends with a slash, it will only match
+   directories.  In other words, `foo/` will match a
+   directory `foo` but will not match a regular file or a
+   symbolic link `foo` (this is consistent with the way
+   pathspecs work in general in git).
++
+The trailing slash is removed before applying the remaining
+rules.
 
  - If the pattern does not contain a slash '/', git treats it as
    a shell glob pattern and checks for a match against the
-- 
1.7.8.rc0

[PATCH 2/4] Documentation: clarify effect of '/' in gitignore(5) patterns

[Jonathan Nieder]

The introduction of directory-only matches in v1.5.5-rc0~208^2~1
(gitignore(5): Allow "foo/" in ignore list to match directory "foo",
2008-01-31) was a small, incremental change to gitignore syntax that
did not affect the rest of the rules in any major way.  A '/' at the
end of a pattern means "match directories only" and does not otherwise
affect the pattern.  And that is how the gitignore(5) manpage explains
the syntax.

However, to a person parsing an unfamiliar gitignore entry like foo/,
it is too easy to notice the later (old) rule that describes how
patterns containing a slash are anchored and miss that the slash
should have been stripped off before considering whether the rule
applies.

Let's just explicitly say that patterns are only anchored if they
contain a slash that is not at the end of the pattern, avoiding this
confusion.  A more graceful presentation of this material may be
possible, but for now the goal is to get the facts clear --- we can
refactor the text to scan well without losing its meaning later.

(While at it, tweak the wording for clarity and add an example.)

Reported-by: Y.G. <yamomo1@hotmail.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 Documentation/gitignore.txt |   24 +++++++++++++-----------
 1 files changed, 13 insertions(+), 11 deletions(-)

diff --git a/Documentation/gitignore.txt b/Documentation/gitignore.txt
index 5b070bf0..c7c948dd 100644
--- a/Documentation/gitignore.txt
+++ b/Documentation/gitignore.txt
@@ -77,19 +77,21 @@ PATTERN FORMAT
    directory `foo` but will not match a regular file or a
    symbolic link `foo` (this is consistent with the way
    pathspecs work in general in git).
-+
-The trailing slash is removed before applying the remaining
-rules.
 
- - If the pattern does not contain a slash '/', git treats it as
-   a shell glob pattern and checks for a match against the
-   pathname relative to the location of the `.gitignore` file
-   (relative to the toplevel of the work tree if not from a
-   `.gitignore` file).
+ - If the pattern does not contain a slash '/' at the beginning
+   or in the middle, git treats it as a shell glob pattern and
+   matches the entire pathname including slashes, relative to the
+   location of the `.gitignore` file (or relative to the toplevel
+   of the work tree if the pattern is not from a `.gitignore`
+   file), against it.
+   For example, "{asterisk}.html" matches HTML files in the
+   directory containing the `.gitignore` file and in its
+   subdirectories.
 
- - Otherwise, git treats the pattern as a shell glob suitable
-   for consumption by fnmatch(3) with the FNM_PATHNAME flag:
-   wildcards in the pattern will not match a / in the pathname.
+ - If the pattern contains a slash '/' at the beginning or in the
+   middle, git imitates the behavior of fnmatch(3) with the
+   FNM_PATHNAME flag: wildcards in the pattern will not match a /
+   in the pathname.
    For example, "Documentation/{asterisk}.html" matches
    "Documentation/git.html" but not "Documentation/ppc/ppc.html"
    or "tools/perf/Documentation/perf.html".
-- 
1.7.8.rc0

[PATCH 3/4] Documentation: unanchored gitignore patterns match basename

[Jonathan Nieder]

The rule described by v1.7.1.1~31^2 (gitignore.5: Clarify matching
rules, 2010-03-05) is just false: simple gitignore patterns without a
slash like "foo" and "cscope*" have always matched files in all
directories, not just the toplevel, and a question mark cannot be used
to match the slash separating path components.

For example:

	foo/ - matches directories named "foo" throughout the tree
	Documentation?foo - does not match Documentation/foo

Reported-by: Y.G. <yamomo1@hotmail.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 Documentation/gitignore.txt |   19 ++++++++++---------
 1 files changed, 10 insertions(+), 9 deletions(-)

diff --git a/Documentation/gitignore.txt b/Documentation/gitignore.txt
index c7c948dd..e5715a27 100644
--- a/Documentation/gitignore.txt
+++ b/Documentation/gitignore.txt
@@ -80,18 +80,19 @@ PATTERN FORMAT
 
  - If the pattern does not contain a slash '/' at the beginning
    or in the middle, git treats it as a shell glob pattern and
-   matches the entire pathname including slashes, relative to the
-   location of the `.gitignore` file (or relative to the toplevel
-   of the work tree if the pattern is not from a `.gitignore`
-   file), against it.
-   For example, "{asterisk}.html" matches HTML files in the
-   directory containing the `.gitignore` file and in its
-   subdirectories.
+   checks if the pathname with leading path components
+   removed matches it.
+   For example, "x{asterisk}.html" matches HTML files whose
+   filename begins with an "x" in the directory containing
+   the `.gitignore` file and in its subdirectories.
 
  - If the pattern contains a slash '/' at the beginning or in the
    middle, git imitates the behavior of fnmatch(3) with the
-   FNM_PATHNAME flag: wildcards in the pattern will not match a /
-   in the pathname.
+   FNM_PATHNAME flag.  The pattern is used to match the entire
+   pathname, relative to the location of the `.gitignore` file
+   (or relative to the toplevel of the work tree if the pattern
+   is not from a `.gitignore` file).  Wildcards in the pattern
+   do not match a / in the pathname.
    For example, "Documentation/{asterisk}.html" matches
    "Documentation/git.html" but not "Documentation/ppc/ppc.html"
    or "tools/perf/Documentation/perf.html".
-- 
1.7.8.rc0

[PATCH 4/4] Documentation/gitignore: explain how to un-ignore part of a directory

[Jonathan Nieder]

From: Johannes Sixt <j6t@kdbg.org>
Date: Tue, 5 Apr 2011 23:15:54 +0200

Trying to whitelist a single file pattern in a directory that I was
otherwise content to ignore, if I try

	/m4/
	!/m4/virt-*.m4

then 'git add' will keep warning me that I have to use -f.  That is
because ignoring a directory is much different than ignoring all files
in a directory, when it comes to later negation patterns:

	/m4/*
	!/m4/virt-*.m4

However, this example is misleading because it gives the false
impression that you could do

	/foo/*
	!/foo/bar/baz

and have foo/bar/baz not ignored, and it is still ignored.  Add a
paragraph in the NOTES section explaining the general rule and
suggesting ignoring the directory and using "git add -f" to track
exceptions to cope with it.

[jn: change description distilled from messages by Eric and Hannes
to the mailing list.

As Eric noticed, while the "git add -f" trick works well with
commands like "git status" and "git add -u", plain "git add
subdir/<path>" refuses to update the index with changes from an
ignored subdirectory.  When describing the trick, explicitly list a
few commands that already behave appropriately to avoid confusion.]

Reported-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
That's the end of the series.  Thoughts?

 Documentation/gitignore.txt |   15 +++++++++++++++
 1 files changed, 15 insertions(+), 0 deletions(-)

diff --git a/Documentation/gitignore.txt b/Documentation/gitignore.txt
index e5715a27..8b6f601e 100644
--- a/Documentation/gitignore.txt
+++ b/Documentation/gitignore.txt
@@ -113,6 +113,21 @@ use 'git update-index {litdd}assume-unchanged'.
 To stop tracking a file that is currently tracked, use
 'git rm --cached'.
 
+When a directory is ignored, it is not possible to un-ignore a single file
+somewhere in the directory using another pattern. E.g., with the patterns
+
+--------------
+/build/
+!/build/tests/results
+--------------
+
+the file "build/tests/results" is still ignored because when a directory is
+ignored, its contents are never investigated. In a situation where a few
+exceptions in an otherwise ignored hierarchy are needed, the recommended
+procedure is to specify to ignore the root of the hierarchy and then to 'git
+add -f' the exceptional files. Subsequent changes to the files will not be
+ignored by 'git status', 'git add .', 'git add -u', or 'git commit -a'.
+
 EXAMPLES
 --------
 
-- 
1.7.8.rc0

Re: [PATCH 1/4] Documentation/gitignore: "foo/" patterns match directories, not files under them

[Nguyen Thai Ngoc Duy]

2011/11/7 Jonathan Nieder <jrnieder@gmail.com>:
> The gitignore(5) manpage says that "foo/" will match a directory foo
> and paths underneath it.

If git ignores a directory, then it essentially ignores all paths
underneath it, doesn't it?

> But that is completely false: as Johannes
> Sixt likes to remind us, patterns with a trailing '/' match the named
> directory, not files under that directory.  For example, the following
> .gitignore file
>
>        /build/
>        !/build/tests/results
>
> does not un-ignore build/tests/results since it was never ignored in
> the first place; and commands like "git status" will not notice
> changes to build/tests/results because git doesn't enter the (ignored)
> build/ directory.

I haven't checked but I think it's because when a directory is
ignored, git just stops checking further ignore rules. So "build" _is_
ignored, too strongly that it does not care if some files may need to
be un-ignored later on.

I remember the argument was, because ignore rules are distributed
across .gitignore files, we would need to go into ignored directories
for collecting potential un-ignore rules (for example "!results" on
build/tests/.gitignore) and that just does not make much sense because
we always have to go into ignored directories.

But in your example, where we know we have negated rules, we should
follow the rules and ignore all but build/tests/results.

> Correct the manual to just say that "foo/" matches the directory
> "foo", and make the wording a little clearer in other ways while at
> it.

I haven't not read the next patches, maybe you have mentioned this
already. We should make clear that git does not look for negated rules
once a directory is ignored.

Your example however demonstrates a bug that should be fixed in my
opinion. So maybe one or two lines under BUGS section.

> Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
> ---
>  Documentation/gitignore.txt |   14 ++++++++------
>  1 files changed, 8 insertions(+), 6 deletions(-)
>
> diff --git a/Documentation/gitignore.txt b/Documentation/gitignore.txt
> index 2e7328b8..5b070bf0 100644
> --- a/Documentation/gitignore.txt
> +++ b/Documentation/gitignore.txt
> @@ -72,12 +72,14 @@ PATTERN FORMAT
>    included again.  If a negated pattern matches, this will
>    override lower precedence patterns sources.
>
> - - If the pattern ends with a slash, it is removed for the
> -   purpose of the following description, but it would only find
> -   a match with a directory.  In other words, `foo/` will match a
> -   directory `foo` and paths underneath it, but will not match a
> -   regular file or a symbolic link `foo` (this is consistent
> -   with the way how pathspec works in general in git).
> + - If the pattern ends with a slash, it will only match
> +   directories.  In other words, `foo/` will match a
> +   directory `foo` but will not match a regular file or a
> +   symbolic link `foo` (this is consistent with the way
> +   pathspecs work in general in git).

Looks good.

> ++
> +The trailing slash is removed before applying the remaining
> +rules.

Why does the trailing slash of a rule affect the remaining rules?
-- 
Duy

Re: [PATCH 2/4] Documentation: clarify effect of '/' in gitignore(5) patterns

[Nguyen Thai Ngoc Duy]

2011/11/7 Jonathan Nieder <jrnieder@gmail.com>:
> The introduction of directory-only matches in v1.5.5-rc0~208^2~1
> (gitignore(5): Allow "foo/" in ignore list to match directory "foo",
> 2008-01-31) was a small, incremental change to gitignore syntax that
> did not affect the rest of the rules in any major way.  A '/' at the
> end of a pattern means "match directories only" and does not otherwise
> affect the pattern.  And that is how the gitignore(5) manpage explains
> the syntax.
>
> However, to a person parsing an unfamiliar gitignore entry like foo/,
> it is too easy to notice the later (old) rule that describes how
> patterns containing a slash are anchored and miss that the slash
> should have been stripped off before considering whether the rule
> applies.

I think I make this mistake too. Documenting it is one way. Another way is t

>
> Let's just explicitly say that patterns are only anchored if they
> contain a slash that is not at the end of the pattern, avoiding this
> confusion.  A more graceful presentation of this material may be
> possible, but for now the goal is to get the facts clear --- we can
> refactor the text to scan well without losing its meaning later.

I think the first slash in 'foo//' may function as the anchor and it's
not very clear to me if it's in the middle of pattern of at the end.
Of course we could just change the code to strip all trailing slashes.
-- 
Duy

Re: [PATCH 3/4] Documentation: unanchored gitignore patterns match basename

[Nguyen Thai Ngoc Duy]

2011/11/7 Jonathan Nieder <jrnieder@gmail.com>:
> The rule described by v1.7.1.1~31^2 (gitignore.5: Clarify matching
> rules, 2010-03-05) is just false: simple gitignore patterns without a
> slash like "foo" and "cscope*" have always matched files in all
> directories, not just the toplevel, and a question mark cannot be used
> to match the slash separating path components.
>
> For example:
>
>        foo/ - matches directories named "foo" throughout the tree
>        Documentation?foo - does not match Documentation/foo
>
> Reported-by: Y.G. <yamomo1@hotmail.com>
> Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>

Acked-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
-- 
Duy

Re: [PATCH 4/4] Documentation/gitignore: explain how to un-ignore part of a directory

[Nguyen Thai Ngoc Duy]

2011/11/7 Jonathan Nieder <jrnieder@gmail.com>:
> From: Johannes Sixt <j6t@kdbg.org>
> Date: Tue, 5 Apr 2011 23:15:54 +0200
>
> Trying to whitelist a single file pattern in a directory that I was
> otherwise content to ignore, if I try
>
>        /m4/
>        !/m4/virt-*.m4
>
> then 'git add' will keep warning me that I have to use -f.  That is
> because ignoring a directory is much different than ignoring all files
> in a directory, when it comes to later negation patterns:
>
>        /m4/*
>        !/m4/virt-*.m4
>

gitignore.txt is also referred in sparse checkout. And (un)fortunately
the former also works in sparse checkout. I don't know, may be you
could put this in a subsection or something reference-able so we could
mention in sparse checkout document that this part of gitignore.txt
does not apply to sparse checkout?
-- 
Duy

Re: [PATCH 1/4] Documentation/gitignore: "foo/" patterns match directories, not files under them

[Junio C Hamano]

Nguyen Thai Ngoc Duy <pclouds@gmail.com> writes:

> 2011/11/7 Jonathan Nieder <jrnieder@gmail.com>:
>> The gitignore(5) manpage says that "foo/" will match a directory foo
>> and paths underneath it.
>
> If git ignores a directory, then it essentially ignores all paths
> underneath it, doesn't it?
>
>> But that is completely false: as Johannes
>> Sixt likes to remind us, patterns with a trailing '/' match the named
>> directory, not files under that directory.  For example, the following
>> .gitignore file
>>
>>        /build/
>>        !/build/tests/results
>>
>> does not un-ignore build/tests/results since it was never ignored in
>> the first place; and commands like "git status" will not notice
>> changes to build/tests/results because git doesn't enter the (ignored)
>> build/ directory.
>
> I haven't checked but I think it's because when a directory is
> ignored, git just stops checking further ignore rules. So "build" _is_
> ignored, too strongly that it does not care if some files may need to
> be un-ignored later on.
>
> I remember the argument was, because ignore rules are distributed
> across .gitignore files, we would need to go into ignored directories
> for collecting potential un-ignore rules (for example "!results" on
> build/tests/.gitignore) and that just does not make much sense because
> we always have to go into ignored directories.
>
> But in your example, where we know we have negated rules, we should
> follow the rules and ignore all but build/tests/results.
>
>> Correct the manual to just say that "foo/" matches the directory
>> "foo", and make the wording a little clearer in other ways while at
>> it.
>
> I haven't not read the next patches, maybe you have mentioned this
> already. We should make clear that git does not look for negated rules
> once a directory is ignored.
>
> Your example however demonstrates a bug that should be fixed in my
> opinion. So maybe one or two lines under BUGS section.
>
>> Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
>> ---
>>  Documentation/gitignore.txt |   14 ++++++++------
>>  1 files changed, 8 insertions(+), 6 deletions(-)
>>
>> diff --git a/Documentation/gitignore.txt b/Documentation/gitignore.txt
>> index 2e7328b8..5b070bf0 100644
>> --- a/Documentation/gitignore.txt
>> +++ b/Documentation/gitignore.txt
>> @@ -72,12 +72,14 @@ PATTERN FORMAT
>>    included again.  If a negated pattern matches, this will
>>    override lower precedence patterns sources.
>>
>> - - If the pattern ends with a slash, it is removed for the
>> -   purpose of the following description, but it would only find
>> -   a match with a directory.  In other words, `foo/` will match a
>> -   directory `foo` and paths underneath it, but will not match a
>> -   regular file or a symbolic link `foo` (this is consistent
>> -   with the way how pathspec works in general in git).
>> + - If the pattern ends with a slash, it will only match
>> +   directories.  In other words, `foo/` will match a
>> +   directory `foo` but will not match a regular file or a
>> +   symbolic link `foo` (this is consistent with the way
>> +   pathspecs work in general in git).
>
> Looks good.

Or just remove "In other words, ..." that is bogus. Everything before that
is correct.

>> ++
>> +The trailing slash is removed before applying the remaining
>> +rules.
>
> Why does the trailing slash of a rule affect the remaining rules?

Later rule makes a path with and without a slash in it work differently.
A single token "foo/" acts as if there is no trailing slash to match
any directory in the hierarchy, e.g. it matches a directory "frotz/foo".