--batch or some --paths-file for very long lists of paths

--batch or some --paths-file for very long lists of paths

[Yaroslav Halchenko]

Dear Git Gurus,

In DataLad (https://datalad.org) we are doing "our own" analysis of what
specific files (not entire directories) should git and git-annex operate
on.  Obviously, in large repositories (and we have with >100k files)
that might require invoking  git add  or  git diff  etc with a long list
of paths specified in the command line.  For that we often split
invocation into multiple and even resort to   git commit --amend  to
combine multiple commits then into a single one.

But I wondered if may be there is already some trick to make such
commands as   status, diff, add, commit   to operate on arbitrarily long
list of paths passed to that git command somehow.

Note that gitglossary (at least in 2.35.1 git on debian) says that 

	Pathspecs are used on the command line of "git ls-files", "git
    ls-tree", "git add", "git grep", "git diff", "git checkout", and many other
    commands ...

but 

	$> git ls-tree -h | head -n1
	usage: git ls-tree [<options>] <tree-ish> [<path>...]

so it is <path> not the <pathspec> like (why in stderr this time?)

	$> git commit -h 2>&1 | head -n1
	usage: git commit [<options>] [--] <pathspec>...

So if in both cases it is pathspec, may be pathspec could support some
other magical keyword like :(filelist)/tmp/mylonglistofpaths ?

Thanks in advance for your time and thoughts,
-- 
Yaroslav O. Halchenko
Center for Open Neuroscience     http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
WWW:   http://www.linkedin.com/in/yarik        

Re: --batch or some --paths-file for very long lists of paths

[Phillip Wood]

Hi Yaroslav

On 04/08/2022 23:59, Yaroslav Halchenko wrote:
> Dear Git Gurus,
> 
> In DataLad (https://datalad.org) we are doing "our own" analysis of what
> specific files (not entire directories) should git and git-annex operate
> on.  Obviously, in large repositories (and we have with >100k files)
> that might require invoking  git add  or  git diff  etc with a long list
> of paths specified in the command line.  For that we often split
> invocation into multiple and even resort to   git commit --amend  to
> combine multiple commits then into a single one.
> 
> But I wondered if may be there is already some trick to make such
> commands as   status, diff, add, commit   to operate on arbitrarily long
> list of paths passed to that git command somehow.

A number of porcelain commands have a --pathspec-from-file option that 
takes a file with a list of pathspecs or reads them from stdin. When 
combined with --pathspec-file-nul this handles paths containing newline 
correctly or you can quote them without this option. You can pass 
--literal-pathspecs if you have a list of paths rather than pathspecs.

At the plumbing level you can use "update-index" to add/delete/update 
paths in the index which will read paths from stdin and "checkout-index" 
will also read paths from stdin.

The diff family do not have any support for --pathspec-from-file at the 
moment but I'd be happy to see someone implement it (I think it would be 
fairly straight forward).

> Note that gitglossary (at least in 2.35.1 git on debian) says that
> 
> 	Pathspecs are used on the command line of "git ls-files", "git
>      ls-tree", "git add", "git grep", "git diff", "git checkout", and many other
>      commands ...
> 
> but
> 
> 	$> git ls-tree -h | head -n1
> 	usage: git ls-tree [<options>] <tree-ish> [<path>...]
> 
> so it is <path> not the <pathspec> like (why in stderr this time?)
> 
> 	$> git commit -h 2>&1 | head -n1
> 	usage: git commit [<options>] [--] <pathspec>...
> 
> So if in both cases it is pathspec, may be pathspec could support some
> other magical keyword like :(filelist)/tmp/mylonglistofpaths ?

I like that path magic idea, but as we already have --pathspec-from-file 
I think we'd be better improving support for that.

Best Wishes

Phillip

> Thanks in advance for your time and thoughts,