refining .gitignores

refining .gitignores

[Bruce Stephens]

How do I get a list of files (in HEAD, say) that would be ignored by
the .gitignore files (and the other usual settings)?

It feels like something like this ought to work:

     git ls-files -z | xargs -0 git ls-files --ignored

But listing its arguments that are ignored by .gitignore (etc.)
doesn't seem to be what "git ls-files --ignored" does.  Or at least,
not quite as straightforwardly as that.

The motivation is (obviously) that I fear some of the .gitignore
patterns are too broad, and a reasonable check is that none of the
files that are already committed would be caught by the patterns.

Re: refining .gitignores

[Alex Riesen]

Bruce Stephens, Wed, Nov 14, 2007 23:36:06 +0100:
> How do I get a list of files (in HEAD, say) that would be ignored by
> the .gitignore files (and the other usual settings)?
> 
> It feels like something like this ought to work:
> 
>      git ls-files -z | xargs -0 git ls-files --ignored
> 
> But listing its arguments that are ignored by .gitignore (etc.)
> doesn't seem to be what "git ls-files --ignored" does.  Or at least,
> not quite as straightforwardly as that.

git ls-files --exclude-per-directory=.gitignore -X .git/info/exclude -i -o

> The motivation is (obviously) that I fear some of the .gitignore
> patterns are too broad, and a reasonable check is that none of the
> files that are already committed would be caught by the patterns.

git ls-files --exclude-per-directory=.gitignore -X .git/info/exclude -i

(IOW, remove the -o aka --others)

Re: refining .gitignores

[Bruce Stephens]

Alex Riesen <raa.lkml@gmail.com> writes:

> Bruce Stephens, Wed, Nov 14, 2007 23:36:06 +0100:
>> How do I get a list of files (in HEAD, say) that would be ignored by
>> the .gitignore files (and the other usual settings)?
>> 
>> It feels like something like this ought to work:
>> 
>>      git ls-files -z | xargs -0 git ls-files --ignored
>> 
>> But listing its arguments that are ignored by .gitignore (etc.)
>> doesn't seem to be what "git ls-files --ignored" does.  Or at least,
>> not quite as straightforwardly as that.
>
> git ls-files --exclude-per-directory=.gitignore -X .git/info/exclude -i -o

That doesn't seem to work.

For example, if I add '*.c' to .gitignores in git.git, I can't seem to
get that command to display any .c files.

Run on its own, it displays lots of files, but no .c files.  Run with
an argument (such as builtin-add.c), it displays nothing.

>> The motivation is (obviously) that I fear some of the .gitignore
>> patterns are too broad, and a reasonable check is that none of the
>> files that are already committed would be caught by the patterns.
>
> git ls-files --exclude-per-directory=.gitignore -X .git/info/exclude -i

That also doesn't seem to do quite what I want, and probably in the
same way.  I see git add returns non-zero error status if a file is
ignored, so I can do it with

excluded=()
for f in $(git ls-files)
do
    git add $f || excluded=($excluded $f)
done

But that feels kind of clunky.  I feel I'm missing something basic
about how git ls-files is intended to work, or something.

Re: refining .gitignores

[Alex Riesen]

Bruce Stephens, Thu, Nov 15, 2007 12:39:45 +0100:
> Alex Riesen <raa.lkml@gmail.com> writes:
> > Bruce Stephens, Wed, Nov 14, 2007 23:36:06 +0100:
> >> How do I get a list of files (in HEAD, say) that would be ignored by
> >> the .gitignore files (and the other usual settings)?
> >> 
> >> It feels like something like this ought to work:
> >> 
> >>      git ls-files -z | xargs -0 git ls-files --ignored
> >> 
> >> But listing its arguments that are ignored by .gitignore (etc.)
> >> doesn't seem to be what "git ls-files --ignored" does.  Or at least,
> >> not quite as straightforwardly as that.
> >
> > git ls-files --exclude-per-directory=.gitignore -X .git/info/exclude -i -o
> 
> That doesn't seem to work.
> 
> For example, if I add '*.c' to .gitignores in git.git, I can't seem to
> get that command to display any .c files.

.gitignore? Without "s"?
Maybe your .c files are already added to index? Otherwise you have to
use the second form. It shows known-to-Git ignored files.

> Run on its own, it displays lots of files, but no .c files.  Run with
> an argument (such as builtin-add.c), it displays nothing.

because the *are* in the index. "-o" means "others", as in "not Git".

> >> The motivation is (obviously) that I fear some of the .gitignore
> >> patterns are too broad, and a reasonable check is that none of the
> >> files that are already committed would be caught by the patterns.
> >
> > git ls-files --exclude-per-directory=.gitignore -X .git/info/exclude -i
> 
> That also doesn't seem to do quite what I want, and probably in the
> same way. ...

It shows ignored files from the "known-to-Git" fileset.

Re: refining .gitignores

[Bruce Stephens]

Alex Riesen <raa.lkml@gmail.com> writes:

[...]

> .gitignore? Without "s"?

Yes, my mistake.  I added '*.c' to ".gitignore".

> Maybe your .c files are already added to index? Otherwise you have to
> use the second form. It shows known-to-Git ignored files.

They are in the index.  What I want is a list of files which are known
to git, which are matched by the default rules (in particular the
.gitignore files).

So that should be this?

    git ls-files --exclude-per-directory=.gitignore -i

But that shows nothing at all, and it still shows nothing if I add a
"builtin-add.c" as an argument to it (this file exists, and is in the
index).

"git add builtin-add.c" fails, complaining (correctly) that the path
is ignored by one of my .gitignore files.

We're obviously talking past one another somehow, or I have a broken
build of git?

[...]

Re: refining .gitignores

[Linus Torvalds]

On Thu, 15 Nov 2007, Bruce Stephens wrote:
>
> They are in the index.  What I want is a list of files which are known
> to git, which are matched by the default rules (in particular the
> .gitignore files).

Ahh.

*All* the .gitignore rules are purely about files that git does not 
already know about. Once git tracks a pathname, the ignore rules simply 
do not matter.

Files that are in the index are simply never ignored. There's no way to 
ignore them, and in fact, the whole "git add -f" thing is a way to add 
files to the index and override the exclude rules - at which point git 
then tracks them regardless of any such exclude rules.

			Linus

Re: refining .gitignores

[Bruce Stephens]

Linus Torvalds <torvalds@linux-foundation.org> writes:

> On Thu, 15 Nov 2007, Bruce Stephens wrote:
>>
>> They are in the index.  What I want is a list of files which are known
>> to git, which are matched by the default rules (in particular the
>> .gitignore files).
>
> Ahh.
>
> *All* the .gitignore rules are purely about files that git does not 
> already know about. Once git tracks a pathname, the ignore rules simply 
> do not matter.
>
> Files that are in the index are simply never ignored. There's no way to 
> ignore them, and in fact, the whole "git add -f" thing is a way to add 
> files to the index and override the exclude rules - at which point git 
> then tracks them regardless of any such exclude rules.

OK.  I think I sort of understood that, but it's nice to have it
clearly stated.

I'm not worried so much about files that already exist.  I'm worried
that we may in the future create files that we want to store in git
and which are ignored.  If we do that there's a chance we'll miss them
(because "git gui", "git status", etc., won't bring attention to
them).

So it seemed valuable to try to make the .gitignore patterns precise:
so that they ignore all the generated files that our builds produce,
while not matching any files that git knows about.

That certainly wouldn't guarantee that new files wouldn't be caught by
the patterns, but it ought to lower the probability of this, since
there's a decent chance that such files will either be not touched by
the patterns (because they're *.java or something), or they'll fit
into one of the already handled exceptions (Doxyfile.local is one:
most *.local files are build products, but Doxyfile.local aren't).

I guess if you're starting off with some project in git then this
isn't such an issue: you evolve the patterns and the build process and
so on in parallel.  So maybe there's not such a demand for finding
this information, and those of us who do want to do it are best using
the "git add" loop approach?

Re: refining .gitignores

[Alex Riesen]

Bruce Stephens, Thu, Nov 15, 2007 21:28:06 +0100:
> Alex Riesen <raa.lkml@gmail.com> writes:
> 
> [...]
> 
> > .gitignore? Without "s"?
> 
> Yes, my mistake.  I added '*.c' to ".gitignore".
> 
> > Maybe your .c files are already added to index? Otherwise you have to
> > use the second form. It shows known-to-Git ignored files.
> 
> They are in the index.  What I want is a list of files which are known
> to git, which are matched by the default rules (in particular the
> .gitignore files).
> 
> So that should be this?
> 
>     git ls-files --exclude-per-directory=.gitignore -i

Yes, so it seems. The per-directory ignore-files don't work.
"--exclude-from=" (aka -X) still does.

> But that shows nothing at all, and it still shows nothing if I add a
> "builtin-add.c" as an argument to it (this file exists, and is in the
> index).
> 
> "git add builtin-add.c" fails, complaining (correctly) that the path
> is ignored by one of my .gitignore files.
> 
> We're obviously talking past one another somehow, or I have a broken
> build of git?

No, I honestly believed that Git-known files can be ignored. According
to Linus I must be wrong, but I have an excuse:

    $ mkdir aaa
    $ cd aaa
    $ git init
    Initialized empty Git repository in .git/
    $ uname >abc
    $ git add .
    $ ls > .gitignore
    $ git ls-files -X .gitignore
    $ git ls-files -X .gitignore -i
    abc

Re: refining .gitignores

[Linus Torvalds]

On Thu, 15 Nov 2007, Alex Riesen wrote:
> 
> No, I honestly believed that Git-known files can be ignored. According
> to Linus I must be wrong, but I have an excuse:

I do think we're not necessarily entirely consistent wrt ignore files.

In particular, git-ls-files itself actually is pretty special, because it 
has two totally different modes:

 - you can ask for "other" files (and this is where you'd be expected to 
   use .gitignore)

 - you can just list the files git knows about (and this is where you'd 
   generally be expected to *not* use .gitignore)

..and to make matters more interesting, "git add" used to do the former 
(it was just a thin script around "git-ls-files -o") but then was later 
enhanced to also take already-known files into account, so I am not at all 
surprised if we actually get confused because we used to have a fairly 
clear separation of the two cases but then we started mixing them up.

So while I think that ".gitignore" *should* only affect files that we 
don't already know about (ie effectively only file lists that come from 
"readdir()", not from internal git data structures), I would not be at all 
surprised if there are bugs.

		Linus