git ls-files wildcard behavior considered harmful

git ls-files wildcard behavior considered harmful

[Joey Hess]

I was very surprised to learn the other day that git ls-files 'foo*'
will expand wildcards (including character classes), in the absence of
expansion by the shell. (git version 2.1.4)

joey@darkstar:~/tmp/aaa>git ls-files 'foo*bar'
foo*bar
foobazbar
joey@darkstar:~/tmp/aaa>git ls-files '[abc]'
[abc]
a
b

As far as I can see this behavior is not documented on the man page,
except for a tiny mention in the --with-tree documentation, where
it says "<file> (i.e. path pattern)".

Since I wanted to avoid this wildcard expension, I tried slash-escaping
the wildcard characters. This works:

joey@darkstar:~/tmp/aaa>git ls-files 'foo\*bar'
foo*bar
joey@darkstar:~/tmp/aaa>git ls-files '\[abc\]'
[abc]

But, there is a weird behavior here with subdirectories. While normally
ls-files would recurse, slash-escaped wildcard characters in the directory
name prevent recursion.

joey@darkstar:~/tmp/aaa>git ls-files 'foo[d]'
foo[d]/subfile
food
joey@darkstar:~/tmp/aaa>git ls-files 'foo\[d\]'
joey@darkstar:~/tmp/aaa>

The above example shows a case where it's impossible to get ls-files
to only list files in a directory and not other files that match the
wildcard. This seems like it must be a bug, and it means it's impossible
to reliably work around the wildcard expansion behavior.

I suspect that this wildcard expansion behavior is useful somewhere.
But from the perspective of using ls-files as plumbing, where you want
to get out some subset of what was put in, it's not nice.

-- 
see shy jo

Re: git ls-files wildcard behavior considered harmful

[Duy Nguyen]

On Tue, Mar 31, 2015 at 6:04 AM, Joey Hess <id@joeyh.name> wrote:
> I was very surprised to learn the other day that git ls-files 'foo*'
> will expand wildcards (including character classes), in the absence of
> expansion by the shell. (git version 2.1.4)
>
> joey@darkstar:~/tmp/aaa>git ls-files 'foo*bar'
> foo*bar
> foobazbar
> joey@darkstar:~/tmp/aaa>git ls-files '[abc]'
> [abc]
> a
> b
>
> As far as I can see this behavior is not documented on the man page,
> except for a tiny mention in the --with-tree documentation, where
> it says "<file> (i.e. path pattern)".
>
> Since I wanted to avoid this wildcard expension, I tried slash-escaping
> the wildcard characters.

You can do "git --literal-pathspecs ls-files ..." or set GIT_LITERAL_PATHSPECS.
-- 
Duy

Re: git ls-files wildcard behavior considered harmful

[Joey Hess]

[-- Attachment #1: Type: text/plain, Size: 367 bytes --]

Duy Nguyen wrote:
> You can do "git --literal-pathspecs ls-files ..." or set GIT_LITERAL_PATHSPECS.

Thanks! --literal-pathspecs does allow getting around this.

Now I'm wondering what other parts of plumbing might be doing globbing
that I did not anticipate. Maybe I should set the environment variable
so I don't need to worry about it..

-- 
see shy jo

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]

Re: git ls-files wildcard behavior considered harmful

[Jeff King]

On Mon, Mar 30, 2015 at 07:36:14PM -0400, Joey Hess wrote:

> Duy Nguyen wrote:
> > You can do "git --literal-pathspecs ls-files ..." or set GIT_LITERAL_PATHSPECS.
> 
> Thanks! --literal-pathspecs does allow getting around this.
> 
> Now I'm wondering what other parts of plumbing might be doing globbing
> that I did not anticipate. Maybe I should set the environment variable
> so I don't need to worry about it..

Pretty much everything takes magic patterns[1]. Pathspecs given to diff,
path limiters for pruning, etc. If you are scripting with raw filenames
(e.g., feeding the filenames out of another tool), I recommend setting
GIT_LITERAL_PATHSPECS everywhere. That's what we do on the server side
at GitHub (and is the reason I implemented --literal-pathspecs in the
first place).

-Peff

[1] Note that globbing is just part of this. Names starting with colon
    are also magical, e.g. ":/foo", or even ":(literal)foo".  These are
    documented in the "pathspec" definition of "git help glossary".

Re: git ls-files wildcard behavior considered harmful

[Duy Nguyen]

On Tue, Mar 31, 2015 at 6:36 AM, Joey Hess <id@joeyh.name> wrote:
> Duy Nguyen wrote:
>> You can do "git --literal-pathspecs ls-files ..." or set GIT_LITERAL_PATHSPECS.
>
> Thanks! --literal-pathspecs does allow getting around this.
>
> Now I'm wondering what other parts of plumbing might be doing globbing
> that I did not anticipate. Maybe I should set the environment variable
> so I don't need to worry about it..

We're just a bit sloppy in documentation. ls-files learned about
wildcards since 56fc510 ([PATCH] git-ls-files: generalized pathspecs -
2005-08-21). Two months later, git-ls-files.txt is updated to say
"<file>" (which is not entirely correct even if we ignore wildcards,
ls-files can take directories as well). With latest git, I think only
git-mv and git-read-tree go with plain paths. It's probably best to
set GIT_LITERAL_PATHSPECS for scripting like Jeff said.
-- 
Duy

Re: git ls-files wildcard behavior considered harmful

[Jonathan Nieder]

Hi Joey,

Joey Hess wrote:

> Since I wanted to avoid this wildcard expension, I tried slash-escaping
> the wildcard characters. This works:
>
> joey@darkstar:~/tmp/aaa>git ls-files 'foo\*bar'
> foo*bar

Does 'git --noglob-pathspecs' help?

Curious,
Jonathan