Gitignore matching "git add" wildcard prevents operation

Gitignore matching "git add" wildcard prevents operation

[Marko Poutiainen]

It seems that that there is a bug related to how Git handles the add
command with wildcards if the wildcard also matches a file that is ignored.

E.g (I activate the *.[ao] rule when editing .git/info/exclude):

mep@Blackbird:~$ cd /tmp
mep@Blackbird:/tmp$ mkdir git
mep@Blackbird:/tmp$ cd git
mep@Blackbird:/tmp/git$ git init
Initialized empty Git repository in /tmp/git/.git/
mep@Blackbird:/tmp/git$ nano -w .git/info/exclude
mep@Blackbird:/tmp/git$ touch a.o a.c a.h
mep@Blackbird:/tmp/git$ git add a.*
The following paths are ignored by one of your .gitignore files:
a.o
Use -f if you really want to add them.
fatal: no files added
mep@Blackbird:/tmp/git$ git status
# On branch master
#
# Initial commit
#
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#       a.c
#       a.h
#       a.o
nothing added to commit but untracked files present (use "git add" to track)

Furthermore:

mep@Blackbird:/tmp/git$ git add .
mep@Blackbird:/tmp/git$ git commit -m "foo"
[master (root-commit) 43da825] foo
 0 files changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 a.c
 create mode 100644 a.h
mep@Blackbird:/tmp/git$ nano a.c
mep@Blackbird:/tmp/git$ nano a.h
mep@Blackbird:/tmp/git$ git add a.*
The following paths are ignored by one of your .gitignore files:
a.o
Use -f if you really want to add them.
fatal: no files added
mep@Blackbird:/tmp/git$ git status
# On branch master
# Changed but not updated:
#   (use "git add <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working
directory)
#
#       modified:   a.c
#       modified:   a.h
#
no changes added to commit (use "git add" and/or "git commit -a")

Re: Gitignore matching "git add" wildcard prevents operation

[Jeff King]

On Thu, Jan 14, 2010 at 08:18:40PM +0200, Marko Poutiainen wrote:

> It seems that that there is a bug related to how Git handles the add
> command with wildcards if the wildcard also matches a file that is ignored.
> 
> E.g (I activate the *.[ao] rule when editing .git/info/exclude):
> 
> mep@Blackbird:~$ cd /tmp
> mep@Blackbird:/tmp$ mkdir git
> mep@Blackbird:/tmp$ cd git
> mep@Blackbird:/tmp/git$ git init
> Initialized empty Git repository in /tmp/git/.git/
> mep@Blackbird:/tmp/git$ nano -w .git/info/exclude
> mep@Blackbird:/tmp/git$ touch a.o a.c a.h
> mep@Blackbird:/tmp/git$ git add a.*
> The following paths are ignored by one of your .gitignore files:
> a.o
> Use -f if you really want to add them.
> fatal: no files added

It's not a bug. That is working as designed. If "git add" encounters a
problem with any of the files you gave it on the command line (and
remember, the shell expands the wildcard, so git literally sees the
three files on the command line) then it aborts the entire operation.

Now whether that is useful behavior in this case, I don't know. If you
were truly manually specifying the files, then it would probably not be
a big deal to simply remove the offending file. But there is no way to
use the shell wildcard and get the behavior you want (not even a "git
add --really-ignore-my-ignores a.*").

I don't think it would be right to silently ignore them, because it may
be the sign of an error. It could be downgraded to a warning, like:

  $ git add *.a
  warning: a.o is ignored, not adding

The downside would be that somebody who really _did_ want to add such a
file but forgot to use "-f" is going to be less likely to notice it, as
the command does not barf. So maybe such behavior should be triggered
with a command-line option. I dunno.

-Peff

Re: Gitignore matching "git add" wildcard prevents operation

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> ... But there is no way to
> use the shell wildcard and get the behavior you want (not even a "git
> add --really-ignore-my-ignores a.*").

Perhaps you want to run

	$ git add 'a.*'

Notice that the wildcard is protected from the shell.

Re: Gitignore matching "git add" wildcard prevents operation

[Jeff King]

On Thu, Jan 14, 2010 at 12:21:34PM -0800, Junio C Hamano wrote:

> Jeff King <peff@peff.net> writes:
> 
> > ... But there is no way to
> > use the shell wildcard and get the behavior you want (not even a "git
> > add --really-ignore-my-ignores a.*").
> 
> Perhaps you want to run
> 
> 	$ git add 'a.*'
> 
> Notice that the wildcard is protected from the shell.

Ugh. You're right that it does work, but I don't expect users to make
the intuitive jump from the OP's problem to this solution (I certainly
didn't). In particular:

  1. Most programs don't take their own globs. Without knowing that git
     can do so, there is no reason to discover it in this instance. I
     can see searching the manpage for options, but not for a discussion
     of globbing behavior.

  2. They would have to know that using a git-glob will magically change
     the error-checking behavior.

-Peff

Re: Gitignore matching "git add" wildcard prevents operation

[Marko Poutiainen]

Jeff King wrote:
> On Thu, Jan 14, 2010 at 08:18:40PM +0200, Marko Poutiainen wrote:
> 
>> It seems that that there is a bug related to how Git handles the add
>> command with wildcards if the wildcard also matches a file that is ignored.
>>
>> E.g (I activate the *.[ao] rule when editing .git/info/exclude):
>>
>> mep@Blackbird:~$ cd /tmp
>> mep@Blackbird:/tmp$ mkdir git
>> mep@Blackbird:/tmp$ cd git
>> mep@Blackbird:/tmp/git$ git init
>> Initialized empty Git repository in /tmp/git/.git/
>> mep@Blackbird:/tmp/git$ nano -w .git/info/exclude
>> mep@Blackbird:/tmp/git$ touch a.o a.c a.h
>> mep@Blackbird:/tmp/git$ git add a.*
>> The following paths are ignored by one of your .gitignore files:
>> a.o
>> Use -f if you really want to add them.
>> fatal: no files added
> 
> It's not a bug. That is working as designed. If "git add" encounters a
> problem with any of the files you gave it on the command line (and
> remember, the shell expands the wildcard, so git literally sees the
> three files on the command line) then it aborts the entire operation.
> 
> Now whether that is useful behavior in this case, I don't know. If you
> were truly manually specifying the files, then it would probably not be
> a big deal to simply remove the offending file. But there is no way to
> use the shell wildcard and get the behavior you want (not even a "git
> add --really-ignore-my-ignores a.*").
> 
> I don't think it would be right to silently ignore them, because it may
> be the sign of an error. It could be downgraded to a warning, like:
> 
>   $ git add *.a
>   warning: a.o is ignored, not adding
> 
> The downside would be that somebody who really _did_ want to add such a
> file but forgot to use "-f" is going to be less likely to notice it, as
> the command does not barf. So maybe such behavior should be triggered
> with a command-line option. I dunno.

Well, it then the interface just isn't consistent, because if that's the
reason, then why does

$git add .

then work in this case? . is still just a type of wildcard, so it fail
in a similar way, shouldn't it?

Your idea of giving a warning is good as is the command-line option. In
any case, in my opinion this feature can make some operations more
complex than they should be. For instance, in my example I would have to
separately do git add to both files. What if there are more than two
files to add and I get this error every time I try to add the files?

Re: Gitignore matching "git add" wildcard prevents operation

[Nicolas Sebrecht]

The 14/01/10, Jeff King wrote:

>   1. Most programs don't take their own globs. Without knowing that git
>      can do so, there is no reason to discover it in this instance. I
>      can see searching the manpage for options, but not for a discussion
>      of globbing behavior.
> 
>   2. They would have to know that using a git-glob will magically change
>      the error-checking behavior.

Not sure. This isn't a Git-particular issue.

Users may hit this with a lot of other unix tools (sed, grep, find,
etc). So, we can expect either
  they already know the issue;
or
  they are discovering it using Git.

Most of the tools I talk about do have a manual section about globbing.
Users could learn globs with Git too and expect the same behaviour
somewhere else.


-- 
Nicolas Sebrecht

Re: Gitignore matching "git add" wildcard prevents operation

[Nicolas Sebrecht]

The 14/01/10, Marko Poutiainen wrote:
> 
> Well, it then the interface just isn't consistent,

But it is...

> Well, it then the interface just isn't consistent, because if that's the
> reason, then why does
> 
> $git add .
> 
> then work in this case? . is still just a type of wildcard, so it fail
> in a similar way, shouldn't it?

...because the dot isn't a wildcard. It's a path.

So, we expect 'git add' to recusively work and resolve ignored paths
from .gitignore and so.

-- 
Nicolas Sebrecht

Re: Gitignore matching "git add" wildcard prevents operation

[Jeff King]

On Fri, Jan 15, 2010 at 04:34:19PM +0100, Nicolas Sebrecht wrote:

> >   1. Most programs don't take their own globs. Without knowing that git
> >      can do so, there is no reason to discover it in this instance. I
> >      can see searching the manpage for options, but not for a discussion
> >      of globbing behavior.
> > 
> >   2. They would have to know that using a git-glob will magically change
> >      the error-checking behavior.
> 
> Not sure. This isn't a Git-particular issue.
> 
> Users may hit this with a lot of other unix tools (sed, grep, find,
> etc). So, we can expect either
>   they already know the issue;
> or
>   they are discovering it using Git.

I don't understand what you mean. How does "sed" do its own globbing of
the command line?

> Most of the tools I talk about do have a manual section about globbing.
> Users could learn globs with Git too and expect the same behaviour
> somewhere else.

Sure. git-add(1) talks about globbing, too, and it even has a sentence
that explains exactly what is happening in Marko's case:

   The git add command will not add ignored files by default. If any
   ignored files were explicitly specified on the command line, git add
   will fail with a list of ignored files. Ignored files reached by
   directory recursion or filename globbing performed by Git (quote your
   globs before the shell) will be silently ignored.

My complaint was more that as a user, I am not likely to see this
problem and think "I'll bet git-specific globbing can solve it." And
when I look in the manual, I am more likely to look for a command-line
option that helps me rather than to read all of the text (though the
fact that we specifically mention "ignore" in the same paragraph means
that at least a user searching for "ignore" in the man page will come up
with this paragraph).

-Peff

Re: Gitignore matching "git add" wildcard prevents operation

[Nicolas Sebrecht]

The 15/01/10, Jeff King wrote:
> On Fri, Jan 15, 2010 at 04:34:19PM +0100, Nicolas Sebrecht wrote:
> 
> > >   1. Most programs don't take their own globs. Without knowing that git
> > >      can do so, there is no reason to discover it in this instance. I
> > >      can see searching the manpage for options, but not for a discussion
> > >      of globbing behavior.
> > > 
> > >   2. They would have to know that using a git-glob will magically change
> > >      the error-checking behavior.
> > 
> > Not sure. This isn't a Git-particular issue.
> > 
> > Users may hit this with a lot of other unix tools (sed, grep, find,
> > etc). So, we can expect either
> >   they already know the issue;
> > or
> >   they are discovering it using Git.
> 
> I don't understand what you mean. How does "sed" do its own globbing of
> the command line?

Well, we are in the same dilemma as the other tools. The internal
globbing rules are explained in the related man page.

> > Most of the tools I talk about do have a manual section about globbing.
> > Users could learn globs with Git too and expect the same behaviour
> > somewhere else.
> 
> Sure. git-add(1) talks about globbing, too

Oops, I was missing that; thanks.

> My complaint was more that as a user, I am not likely to see this
> problem and think "I'll bet git-specific globbing can solve it."

Yes. My point is that we are not talking about a Git specific issue.
What you're raising here is true whatever the command is. So, as long as
we clearly explain how 'git add' works, we are fine. Don't we?

>                                                                  And
> when I look in the manual, I am more likely to look for a command-line
> option that helps me rather than to read all of the text

True. All I can see is to improve the man page with a dedicated section
"Globbing" instead of loosing it in a "random" place.

-- 
Nicolas Sebrecht

Re: Gitignore matching "git add" wildcard prevents operation

[Jeff King]

On Fri, Jan 15, 2010 at 05:11:07PM +0100, Nicolas Sebrecht wrote:

> > I don't understand what you mean. How does "sed" do its own globbing of
> > the command line?
> 
> Well, we are in the same dilemma as the other tools. The internal
> globbing rules are explained in the related man page.

Maybe I wasn't clear: to my knowledge, "sed" does not do any globbing
itself.  How is this the same situation?

Of course other commands like sed will be fed the expansion of a shell
glob, and there may be times when you want to feed a subset of an
expansion. But that is not my complaint; my complaint was mainly that
git's solution to this is not easily discoverable by an uninformed user.
Most other commands don't even have a solution (you would have to solve
it in the shell to pass the desired expansion to the program).

> > when I look in the manual, I am more likely to look for a command-line
> > option that helps me rather than to read all of the text
> 
> True. All I can see is to improve the man page with a dedicated section
> "Globbing" instead of loosing it in a "random" place.

I don't think that would help. The problem is that the user knows they
have an issue with ignored files. The solution is custom globbing, but
they don't know that. So making globbing more prominent doesn't help,
since they will be looking for ignores. You would need to have an
"ignore" section that mentions globbing.

To be clear: I do not have an actual solution, and my initial message
was mostly just grumbling. We _do_ mention globbing and ignores in the
same paragraph, as I quoted earlier. So that is probably enough for a
diligent user to come up with the solution, or at least enough that
trying to improve on it will have diminishing returns.

You could even argue that I was not being such a diligent user in my
initial response. :)

-Peff

Re: Gitignore matching "git add" wildcard prevents operation

[Nicolas Sebrecht]

The 15/01/10, Jeff King wrote:
> On Fri, Jan 15, 2010 at 05:11:07PM +0100, Nicolas Sebrecht wrote:
> 
> > > I don't understand what you mean. How does "sed" do its own globbing of
> > > the command line?
> > 
> > Well, we are in the same dilemma as the other tools. The internal
> > globbing rules are explained in the related man page.
> 
> Maybe I wasn't clear: to my knowledge, "sed" does not do any globbing
> itself.

"sed" does /pattern matching/ which is the meaning I give to /globbing/.

Sure, "sed" doesn't do /shell globbing/ but is fed from it. We obviously
agree here. :-)

>            But that is not my complaint; my complaint was mainly that
> git's solution to this is not easily discoverable by an uninformed user.

I agree in that it is not easily discoverable. But I don't think Git's
solution is worse than any other tool. The "discoverabiliy" looks equal
to any other tool for me. As you said, to do a good job we should be
able to know what was the original command-line, which isn't possible.


-- 
Nicolas Sebrecht

Re: Gitignore matching "git add" wildcard prevents operation

[Junio C Hamano]

Nicolas Sebrecht <nicolas.s.dev@gmx.fr> writes:

>> My complaint was more that as a user, I am not likely to see this
>> problem and think "I'll bet git-specific globbing can solve it."
>
> Yes. My point is that we are not talking about a Git specific issue.

Actually it is.  The thing is, git subcommands do *not* take individual
filenames in general from their command line.  They _always_ take pathspecs,
find files in the work tree that match the pathspecs (but excludes ignored
ones), and work on them [*1*].

Other commands usually don't do that.

So "git add ." and "git add 'a.*'" behave exactly the same way.  Both find
files that match their respective pathspecs.  The former knows that "."
matches "all files that exist in the cwd (excluding ignored ones)" and the
latter knows that "a.*" matches "all files that fnmatch(3) 'a.*' glob
(excluding ignored ones)".  Hence you will not get "don't try to add a
file that you said you want to ignore; use -f if you mean it".


[Footnote]

*1* There are exceptions, such as parameters to the "--file" option, and
"git blame".