Fwd: Unexpected behavior of ls-files command when using --others --exclude-from, and a .gitignore file which resides in a subdirectory

Fwd: Unexpected behavior of ls-files command when using --others --exclude-from, and a .gitignore file which resides in a subdirectory

[Raúl Núñez de Arenas Coronado]

Hi there!

I'm not subscribed to the list, sorry, so I would be more than
grateful if I'm CC'd, but I'll check the mailing list archive from
time to time for replies. Thanks in advance.

It is a common practice for Python virtual environments, pytest and
other tools (not always Python related) to add a '.gitignore' file in
directories these tools create on a repository that should NOT be
committed or tracked. By adding a .gitignore file containing a single
"*", the entire directory is ignored by git. So far, so good.

But when using 'git ls-files --others --exclude-from=<file>', when
<file> is one of those .gitignore files present in a subdir, makes the
command use the patterns in that .gitgnore (in this case, the "*")
against ALL files that would otherwise be listed by using '--others'.

In short: using 'git ls-files --others
--exclude-from=subdir/.gitignore' results in an empty listing if
subdir/.gitignore contains '*". IMHO that pattern should be applied to
the subdir contents and not to the contents of the current directory.
That would be consistent with how git uses .gitignore files in
subfolders.

The obvious solution is to use --exclude-per-directory but it is
deprecated in favor of --exclude-standard and --exclude-standard shows
the same behaviour of --exclude-from=subdir/.gitignore!!!

Is there any way, not using --exclude-per-directory, of listing all
untracked files in the current repository, but ignoring those matching
.gitignore files with the patterns matching the corresponding
subdirectory?

Is this a bug or intended behaviour?

Thanks a lot in advance :)

-- 
Raúl Núñez de Arenas Coronado
.

Re: Fwd: Unexpected behavior of ls-files command when using --others --exclude-from, and a .gitignore file which resides in a subdirectory

[Junio C Hamano]

Raúl Núñez de Arenas Coronado <raulnac@gmail.com> writes:

> In short: using 'git ls-files --others
> --exclude-from=subdir/.gitignore' results in an empty listing if
> subdir/.gitignore contains '*". IMHO that pattern should be applied to
> the subdir contents ...

I do not think so.

Imagine what would happen then if you did

    $ cp subdir/.gitignore /var/tmp/1
    $ git ls-files --others --exclude-from=/var/tmp/1

in such a repository?  The "--exclude-from" option is used to name
the contents (set of patterns) that should be used and the path of
the file that happens to contain the contents does not matter.  So
you should get the same output as the ls-files command that was told
to use "--exclude-from=subdir/.gitignore".

Re: Fwd: Unexpected behavior of ls-files command when using --others --exclude-from, and a .gitignore file which resides in a subdirectory

[Raúl Núñez de Arenas Coronado]

Hi Junio, and thanks a lot for your reply :)

El lun, 22 ene 2024 a las 21:52, Junio C Hamano (<gitster@pobox.com>) escribió:
> Raúl Núñez de Arenas Coronado <raulnac@gmail.com> writes:
>
> > In short: using 'git ls-files --others
> > --exclude-from=subdir/.gitignore' results in an empty listing if
> > subdir/.gitignore contains '*". IMHO that pattern should be applied to
> > the subdir contents ...
>
> I do not think so.
>
> Imagine what would happen then if you did
>
>     $ cp subdir/.gitignore /var/tmp/1
>     $ git ls-files --others --exclude-from=/var/tmp/1

I understand this use case, yes. I was thinking about what git itself
does when ignoring files, especially when dealing with .gitgnore files
in subdirectories, but clearly this needs a different policy, yes.

What I needed from this command is a way of backing up some ignored
files. These files should not go into the repository, but I'm using
them temporarily for development so it is a good idea to back them up.
I'll just backup the entire repository instead, is not a big deal :))

Again, thanks for your reply and your explanation! It's a huge honor
to be replied to by no less than the Git team leather!

-- 
Raúl Núñez de Arenas Coronado
.

Re: Fwd: Unexpected behavior of ls-files command when using --others --exclude-from, and a .gitignore file which resides in a subdirectory

[Junio C Hamano]

Raúl Núñez de Arenas Coronado <raulnac@gmail.com> writes:

> ... I was thinking about what git itself
> does when ignoring files, especially when dealing with .gitgnore files
> in subdirectories, but clearly this needs a different policy, yes.

What "git" internally does is the equivalent of using the
"--exclude-per-directory" option to honor ".gitignore" in the
subdirectories, and in addition use ".git/info/exclude" as another
source, pretty much the same way as "--exclude-from" reads it.

> What I needed from this command is a way of backing up some ignored
> files. These files should not go into the repository, but I'm using
> them temporarily for development so it is a good idea to back them up.
> I'll just backup the entire repository instead, is not a big deal :))

The current mechanism can be used to list "ignored" files that must
be left out of the project (e.g. the most obvious one being object
files "*.o") but these "ignored" files are considered "expendable"
(i.e. Git is allowed to remove it as needed, e.g., when switching
branches and need to make room---if you have an untracked file F
that is ignored, checking out a branch that has a file F/G requires
the file F to disappear so that a directory can be created there).

We have been discussing to extend the mechanism so that we can have
"precious" files, which also will be left out of the project (e.g.,
"git add ." will not add to the index, just like an ignored file)
but are not considered "expendable".  If file F is "precious":

 - "git add ." will not add F to the index

 - "git status" will not say F is left untracked and can be added

 - "git clean -f" will not remove it.

 - checking out a branch with a tracked file F/G will *fail*, to
   prevent the loss of file.

No code yet, but the design consensus is almost there.  Once it
appears, you should be able to say "give me the list of tracked and
precious paths---I do not care about ignored ones that are expendable,
because they can be recreated mechanically and that is why they are
marked ignored", and from such a list of files, you should be able
to drive your back-up solution.

[jc: cc'ed those who are involved in the "precious" discussion].

Re: Fwd: Unexpected behavior of ls-files command when using --others --exclude-from, and a .gitignore file which resides in a subdirectory

[Raúl Núñez de Arenas Coronado]

Hi Junio!

El lun, 22 ene 2024 a las 22:42, Junio C Hamano (<gitster@pobox.com>) escribió:
> We have been discussing to extend the mechanism so that we can have
> "precious" files, which also will be left out of the project (e.g.,
> "git add ." will not add to the index, just like an ignored file)
> but are not considered "expendable".  If file F is "precious":
>
>  - "git add ." will not add F to the index
>
>  - "git status" will not say F is left untracked and can be added
>
>  - "git clean -f" will not remove it.
>
>  - checking out a branch with a tracked file F/G will *fail*, to
>    prevent the loss of file.

And that is exactly the concept I'm handling here: files that should
not be tracked BUT that are not expendable. You explained it concisely
and perfectly :)))

I'll wait until that is implemented, and in the meantime I have a
couple solutions I want to try, like using .gitprecious files and 'git
ls-files --other --ignored --exclude-from=.gitprecious'. I have more
ideas.

Thanks a lot for the help and the explanation!

-- 
Raúl Núñez de Arenas Coronado
.

Re: Fwd: Unexpected behavior of ls-files command when using --others --exclude-from, and a .gitignore file which resides in a subdirectory

[Jeff King]

On Mon, Jan 22, 2024 at 09:45:46PM +0100, Raúl Núñez de Arenas Coronado wrote:

> But when using 'git ls-files --others --exclude-from=<file>', when
> <file> is one of those .gitignore files present in a subdir, makes the
> command use the patterns in that .gitgnore (in this case, the "*")
> against ALL files that would otherwise be listed by using '--others'.
>
> In short: using 'git ls-files --others
> --exclude-from=subdir/.gitignore' results in an empty listing if
> subdir/.gitignore contains '*". IMHO that pattern should be applied to
> the subdir contents and not to the contents of the current directory.
> That would be consistent with how git uses .gitignore files in
> subfolders.

I think Junio covered this with his example, and everything is behaving
as intended (my mental model is that "--exclude-from" is something like
.git/info/exclude or the core.excludesFile option).

But...

> The obvious solution is to use --exclude-per-directory but it is
> deprecated in favor of --exclude-standard and --exclude-standard shows
> the same behaviour of --exclude-from=subdir/.gitignore!!!

...I'm not sure what's going on here. I would think that both
--exclude-standard and --exclude-per-directory would do what you want.

For example, I get:

  [setup]
  $ git init
  $ mkdir subdir
  $ echo '*' >subdir/.gitignore
  $ git add -f subdir/.gitignore && git commit -m "add gitignore"
  $ touch subdir/file file

  [no exclusions]
  $ git ls-files -o
  file
  subdir/file

  [use .gitignore]
  $ git ls-files --exclude-per-directory=.gitignore -o
  file

  [using standard excludes]
  $ git ls-files --exclude-standard -o
  file

Do you get different results from that toy repo? If not, then what is
different about your main repo? Do you perhaps have a stray "*" match
somewhere in .git/info/exclude, etc? Or are you still providing
--exclude-from in addition to --exclude-standard?

-Peff

PS I hadn't realized that --exclude-per-directory had been marked as
   deprecated. I do agree with e750951e74 (ls-files: guide folks to
   --exclude-standard over other --exclude* options, 2023-01-13) in its
   goal of guiding people to the easiest option, but I don't know that
   there has been any discussion about removing the other ones.

Re: Fwd: Unexpected behavior of ls-files command when using --others --exclude-from, and a .gitignore file which resides in a subdirectory

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> PS I hadn't realized that --exclude-per-directory had been marked as
>    deprecated. I do agree with e750951e74 (ls-files: guide folks to
>    --exclude-standard over other --exclude* options, 2023-01-13) in its
>    goal of guiding people to the easiest option, but I don't know that
>    there has been any discussion about removing the other ones.

I do not think there is any value in _removing_ the perfectly well
working --exclude* options, even though I think --exclude-standard
should be what users and scriptors should be using if they want to
emulate what Git does internally.

Re: Fwd: Unexpected behavior of ls-files command when using --others --exclude-from, and a .gitignore file which resides in a subdirectory

[Jeff King]

On Mon, Jan 22, 2024 at 01:45:05PM -0800, Junio C Hamano wrote:

> Jeff King <peff@peff.net> writes:
> 
> > PS I hadn't realized that --exclude-per-directory had been marked as
> >    deprecated. I do agree with e750951e74 (ls-files: guide folks to
> >    --exclude-standard over other --exclude* options, 2023-01-13) in its
> >    goal of guiding people to the easiest option, but I don't know that
> >    there has been any discussion about removing the other ones.
> 
> I do not think there is any value in _removing_ the perfectly well
> working --exclude* options, even though I think --exclude-standard
> should be what users and scriptors should be using if they want to
> emulate what Git does internally.

Yeah, mostly I was surprised that e750951e74 used as strong a word as
"deprecated".

-Peff

Re: Fwd: Unexpected behavior of ls-files command when using --others --exclude-from, and a .gitignore file which resides in a subdirectory

[Elijah Newren]

On Mon, Jan 22, 2024 at 1:59 PM Jeff King <peff@peff.net> wrote:
>
> On Mon, Jan 22, 2024 at 01:45:05PM -0800, Junio C Hamano wrote:
>
> > Jeff King <peff@peff.net> writes:
> >
> > > PS I hadn't realized that --exclude-per-directory had been marked as
> > >    deprecated. I do agree with e750951e74 (ls-files: guide folks to
> > >    --exclude-standard over other --exclude* options, 2023-01-13) in its
> > >    goal of guiding people to the easiest option, but I don't know that
> > >    there has been any discussion about removing the other ones.

I'm not aware of any discussion either.  I certainly had no plans to remove it.

> > I do not think there is any value in _removing_ the perfectly well
> > working --exclude* options, even though I think --exclude-standard
> > should be what users and scriptors should be using if they want to
> > emulate what Git does internally.
>
> Yeah, mostly I was surprised that e750951e74 used as strong a word as
> "deprecated".

Well, here's what I was thinking.  First, based on wikipedia's
definition of deprecated:

```
In several fields, especially computing, deprecation is the
discouragement of use of some terminology, feature, design, or
practice, typically because it has been superseded or is no longer
considered efficient or safe, without completely removing it or
prohibiting its use. Typically, deprecated materials are not
completely removed to ensure legacy compatibility or back up practice
in case new methods are not functional in an odd scenario.

It can also imply that a feature, design, or practice will be removed
or discontinued entirely in the future.
```

Since "can also imply" != "does also imply", and based on the commit
message of 491a7575f (particularly the part that quotes dcf0c16ef1,
both of which were prior work that informed the under discussion
e750951e74), I thought the use of deprecated was perfectly applicable
here.

Re: Fwd: Unexpected behavior of ls-files command when using --others --exclude-from, and a .gitignore file which resides in a subdirectory

[Junio C Hamano]

Elijah Newren <newren@gmail.com> writes:

> It can also imply that a feature, design, or practice will be removed
> or discontinued entirely in the future.
> ```
>
> Since "can also imply" != "does also imply", and based on the commit
> message of 491a7575f (particularly the part that quotes dcf0c16ef1,
> both of which were prior work that informed the under discussion
> e750951e74), I thought the use of deprecated was perfectly applicable
> here.

I think we've seen confusions before, more than just once in the
past, caused by the verb "deprecate" in our docs.  People, after
reading "X is deprecated; use Y instead", always assumed that X will
eventually be removed, even in contexts where we did not mean it.

------ >8 ----------- >8 ----------- >8 ----------- >8 ------
Subject: ls-files: avoid the verb "deprecate" for individual options

When e750951e74f updated the documentation to give greater
visibility to the --exclude-standard option, it marked the
`--exclude-per-directory` option as "deprecated".  While it
technically is correct that being deprecated does not necessarily
mean it is planned to be removed later, it seems to cause confusion
to readers, especially when we merely mean

    The option Y can be used to achieve the same thing as the option
    X, so to those of you who aren't familiar with either X or Y, we
    would recommend to use Y.

This is especially true for `--exclude-standard` vs the combination
of more granular `--exclude-from` and `--exclude-per-directory`
options.  It is true that one common combination of the granular
options can be obtained by just giving the former, but that does not
necessarily mean a more granular control is not necessary.

State why we recommend looking at `--exclude-standard` when we do so
in the description of `--exclude-per-directory`, instead of saying
that the option is deprecated.  Also, spell out the recipe to emulate
what `--exclude-standard` does, so that the users can give it minute
tweaks (like "not reading the global exclusion file from core.excludes").

Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
 Documentation/git-ls-files.txt | 24 +++++++++++++++++++-----
 1 file changed, 19 insertions(+), 5 deletions(-)

diff --git c/Documentation/git-ls-files.txt w/Documentation/git-ls-files.txt
index f65a8cd91d..93a0111cfb 100644
--- c/Documentation/git-ls-files.txt
+++ w/Documentation/git-ls-files.txt
@@ -119,8 +119,10 @@ OPTIONS
 
 --exclude-per-directory=<file>::
 	Read additional exclude patterns that apply only to the
-	directory and its subdirectories in <file>.  Deprecated; use
-	--exclude-standard instead.
+	directory and its subdirectories in <file>.  If you are
+	trying to emulate the way how Porcelain commands work,
+	using the `--exclude-standard` instead is easier and more
+	thorough.
 
 --exclude-standard::
 	Add the standard Git exclusions: .git/info/exclude, .gitignore
@@ -298,9 +300,8 @@ traversing the directory tree and finding files to show when the
 flags --others or --ignored are specified.  linkgit:gitignore[5]
 specifies the format of exclude patterns.
 
-Generally, you should just use --exclude-standard, but for historical
-reasons the exclude patterns can be specified from the following
-places, in order:
+These exclude patterns can be specified from the following places,
+in order:
 
   1. The command-line flag --exclude=<pattern> specifies a
      single pattern.  Patterns are ordered in the same order
@@ -322,6 +323,19 @@ top of the directory tree.  A pattern read from a file specified
 by --exclude-per-directory is relative to the directory that the
 pattern file appears in.
 
+Generally, you should be able to use `--exclude-standard` when you
+want the exclude rules applied the same way as what Porcelain
+commands do.  To emulate what `--exclude-standard` specifies, you
+can give `--exclude-per-directory=.gitignore`, and then specify:
+
+  1. The file specified by the `core.excludesfile` configuration
+     variable, if exists, or the `$XDG_CONFIG_HOME/git/ignore` file.
+
+  2. The `$GIT_DIR/info/exclude` file 
+
+via the `--exclude-from=` option.
+
+
 SEE ALSO
 --------
 linkgit:git-read-tree[1], linkgit:gitignore[5]

Re: Fwd: Unexpected behavior of ls-files command when using --others --exclude-from, and a .gitignore file which resides in a subdirectory

[Jeff King]

On Wed, Jan 24, 2024 at 09:06:51AM -0800, Junio C Hamano wrote:

> ------ >8 ----------- >8 ----------- >8 ----------- >8 ------
> Subject: ls-files: avoid the verb "deprecate" for individual options
> 
> When e750951e74f updated the documentation to give greater
> visibility to the --exclude-standard option, it marked the
> `--exclude-per-directory` option as "deprecated".  While it
> technically is correct that being deprecated does not necessarily
> mean it is planned to be removed later, it seems to cause confusion
> to readers, especially when we merely mean
> 
>     The option Y can be used to achieve the same thing as the option
>     X, so to those of you who aren't familiar with either X or Y, we
>     would recommend to use Y.
> 
> This is especially true for `--exclude-standard` vs the combination
> of more granular `--exclude-from` and `--exclude-per-directory`
> options.  It is true that one common combination of the granular
> options can be obtained by just giving the former, but that does not
> necessarily mean a more granular control is not necessary.
> 
> State why we recommend looking at `--exclude-standard` when we do so
> in the description of `--exclude-per-directory`, instead of saying
> that the option is deprecated.  Also, spell out the recipe to emulate
> what `--exclude-standard` does, so that the users can give it minute
> tweaks (like "not reading the global exclusion file from core.excludes").

This perfectly sums up the situation from my viewpoint. And I think the
changes you suggest not only remove the confusion over the term, they
lay out the possible options for the user much better.

The patch looks good to me, modulo one minor comment:

> diff --git c/Documentation/git-ls-files.txt w/Documentation/git-ls-files.txt
> index f65a8cd91d..93a0111cfb 100644
> --- c/Documentation/git-ls-files.txt
> +++ w/Documentation/git-ls-files.txt
> @@ -119,8 +119,10 @@ OPTIONS
>  
>  --exclude-per-directory=<file>::
>  	Read additional exclude patterns that apply only to the
> -	directory and its subdirectories in <file>.  Deprecated; use
> -	--exclude-standard instead.
> +	directory and its subdirectories in <file>.  If you are
> +	trying to emulate the way how Porcelain commands work,
> +	using the `--exclude-standard` instead is easier and more
> +	thorough.

As a native English speaker, the phrasing here is awkward to me. I'd
probably say "...emulate the way that Porcelain commands work". Or
alternatively, "emulate how Porcelain commands work".

-Peff

Re: Fwd: Unexpected behavior of ls-files command when using --others --exclude-from, and a .gitignore file which resides in a subdirectory

[Raúl Núñez de Arenas Coronado]

Hi Jeff, and thanks for your reply :)

El lun, 22 ene 2024 a las 22:34, Jeff King (<peff@peff.net>) escribió:
> For example, I get:
>
>   [setup]
>   $ git init
>   $ mkdir subdir
>   $ echo '*' >subdir/.gitignore
>   $ git add -f subdir/.gitignore && git commit -m "add gitignore"
>   $ touch subdir/file file
>
>   [no exclusions]
>   $ git ls-files -o
>   file
>   subdir/file
>
>   [use .gitignore]
>   $ git ls-files --exclude-per-directory=.gitignore -o
>   file
>
>   [using standard excludes]
>   $ git ls-files --exclude-standard -o
>   file
>
> Do you get different results from that toy repo? If not, then what is
> different about your main repo? Do you perhaps have a stray "*" match
> somewhere in .git/info/exclude, etc? Or are you still providing
> --exclude-from in addition to --exclude-standard?

The difference lies in how I deal in my computer with ignored files. I
have some files in all my repos which have to be ignored always, so I
have that pattern in my core.excludes file. BUT those files have to be
backed up on my system, just in case my local copy of the repo is lost
for some reason, as they are files I need for development in my
personal machine.

If I use --exclude-standard, no matter if those files are being
ignored by core.excludes OR .git/info/exclude, they won't appear in
that 'list other files' command output.

So, my previous idea was ignore ONLY those files ignored by the
repository .gitignore file. That way, the other files that go in my
core.excludes or .git/info/exclude will be listed and backed up. The
problem here is that some of the repository gitignored files (files
related to building, testing, etc.) have their own .gitignore file
containing '*", but they are NOT present in the repo .gitignore. This
is a common practice for Python virtual environments, for example.

Summing up: or I end up backing up those directories that have their
own .gitignore file (not the end of the world) or I use a different
solution to decide which files are ignored but "precious" (I love that
concept, see Junio C Hamano message for that) OR I add those
directories to the .gitignore present in the repo. This last idea I
don't like very much, as they are particular to my system, not the
repo, for example, *I* may decide to use ESLint and node_modules for
some particular web page I'm writing, but the related configuration
should not go into the repository and should not be ignored in
.gitignore, but in .git/info/exclude. I won't backup node_modules, of
course, it is going to be recreated, but I may want to back up
packages.json and .eslintrc. And both of those files will go into
.git/info/exclude, so if I use --exclude-standard, they won't be
backed up.

Since as both Junio and you correctly pointed, this is in fact the
expected, I'll try to find a different solution while the "precious"
concept is finally implemented. I have a couple of ideas, like using a
.giprecious file which my backup script can process easily, containing
the untracked files to back-up.

Thanks for the toy repo, helped me test the differences with my setup!

-- 
Raúl Núñez de Arenas Coronado
.

Re: Fwd: Unexpected behavior of ls-files command when using --others --exclude-from, and a .gitignore file which resides in a subdirectory

[Jeff King]

On Tue, Jan 23, 2024 at 06:40:10AM +0100, Raúl Núñez de Arenas Coronado wrote:

> > Do you get different results from that toy repo? If not, then what is
> > different about your main repo? Do you perhaps have a stray "*" match
> > somewhere in .git/info/exclude, etc? Or are you still providing
> > --exclude-from in addition to --exclude-standard?
> 
> The difference lies in how I deal in my computer with ignored files. I
> have some files in all my repos which have to be ignored always, so I
> have that pattern in my core.excludes file. BUT those files have to be
> backed up on my system, just in case my local copy of the repo is lost
> for some reason, as they are files I need for development in my
> personal machine.
> 
> If I use --exclude-standard, no matter if those files are being
> ignored by core.excludes OR .git/info/exclude, they won't appear in
> that 'list other files' command output.

Ah, OK. I understand your case better now. Then yes, I think
--exclude-per-directory is the right thing for you, since it lets you
just trigger parts of the exclusion mechanism, and not all. I would say
to ignore the deprecation mention in the manpage. ;)

(I think this also gives an interesting use case arguing for continuing
to support those more-specific exclusion options for ls-files).

If you are mostly just using core.excludesFile (and not
.git/info/exclude), then I suspect:

  git -c core.excludesFile /dev/null ls-files -o --exclude-standard

would work for you, too (but it sounds like you might also be using
.git/info/exclude).

-Peff

Re: Fwd: Unexpected behavior of ls-files command when using --others --exclude-from, and a .gitignore file which resides in a subdirectory

[Raúl Núñez de Arenas Coronado]

Hi Jeff!

El mié, 24 ene 2024 a las 2:09, Jeff King (<peff@peff.net>) escribió:
> (I think this also gives an interesting use case arguing for continuing
> to support those more-specific exclusion options for ls-files).
>
> If you are mostly just using core.excludesFile (and not
> .git/info/exclude), then I suspect:
>
>   git -c core.excludesFile /dev/null ls-files -o --exclude-standard
>
> would work for you, too (but it sounds like you might also be using
> .git/info/exclude).

In my case, since I'm also using .git/info/exclude in some repos, this
won't work as-is, but you gave me an idea, and I think I can get what
I need until the "precious" mechanism is implemented. It requires a
bit of hacking in the backup script but is not a big deal and it will
work for all my current repos.

Thanks for the help!

-- 
Raúl Núñez de Arenas Coronado
.