[RFC] Refspec patterns with * in the middle

[RFC] Refspec patterns with * in the middle

[Daniel Barkalow]

I've got an annoying repository where all of the branches upstream[*] have 
names, for a project "my-proj" like:

some/constant/stuff/$VERSION/junk/my-proj

I'd like to be able to use refspecs like:

 fetch = some/constant/stuff/*/junk/my-proj:refs/remotes/origin/*

I've written an implementation (which mainly involved having only one 
place do the matching and replacement for pattern refspecs, and then 
making that one place a little more clever), so that's not hard. But we 
currently prohibit refspecs like this, and I think we may want to prohibit 
some patterns of this general form, in order to keep typos from doing 
surprising things.

My use case is actually, more precisely:

some/constant/stuff/$PROJ-$NUMBER/junk/my-proj

Where $NUMBER is the version number, and $PROJ is usually, but not quite 
always "my-proj"; the exception being that it might be effectively a 
superproject. So I'd like to have:

 fetch = some/constant/stuff/my-proj-*/junk/my-proj:refs/remotes/origin/*

But I can live with remote branches like "my-proj-2.4" instead of "2.4".

I think it would make sense, and limit typo damage, to say that the * can 
only expand to something with a '/' in it if the star has a slash or the 
end of the string on each side.

What are other people's thoughts? Should I have a config option for how 
flexible matching is permitted? Any particular constraints I should have 
in general?

[*]
(This is actually a Perforce upstream, but I want to keep the mapping from 
Perforce into git simple and not grow another pattern-matching format, so 
I'm just copying Perforce paths as ref names, and then I want the match 
them with the usual code)

	-Daniel
*This .sig left intentionally blank*

Re: [RFC] Refspec patterns with * in the middle

[Jay Soffian]

On Sun, Mar 1, 2009 at 6:42 PM, Daniel Barkalow <barkalow@iabervon.org> wrote:
> I've got an annoying repository where all of the branches upstream[*] have
> names, for a project "my-proj" like:
>
> some/constant/stuff/$VERSION/junk/my-proj
>
> I'd like to be able to use refspecs like:
>
>  fetch = some/constant/stuff/*/junk/my-proj:refs/remotes/origin/*
>
> I've written an implementation (which mainly involved having only one
> place do the matching and replacement for pattern refspecs, and then
> making that one place a little more clever), so that's not hard. But we
> currently prohibit refspecs like this, and I think we may want to prohibit
> some patterns of this general form, in order to keep typos from doing
> surprising things.
>
> My use case is actually, more precisely:
>
> some/constant/stuff/$PROJ-$NUMBER/junk/my-proj
>
> Where $NUMBER is the version number, and $PROJ is usually, but not quite
> always "my-proj"; the exception being that it might be effectively a
> superproject. So I'd like to have:
>
>  fetch = some/constant/stuff/my-proj-*/junk/my-proj:refs/remotes/origin/*
>
> But I can live with remote branches like "my-proj-2.4" instead of "2.4".
>
> I think it would make sense, and limit typo damage, to say that the * can
> only expand to something with a '/' in it if the star has a slash or the
> end of the string on each side.

That seems more confusing that just saying: '*' matches everything but
the path separator ('/'), and whatever it matches on the LHS of the
':' is what it expands to on the RHS. I'm not sure how a typo would
damage anything, but this could always be enabled with
core.refspec.glob_anywhere or some such.

I think regex support is too much:

  fetch = some/constant/stuff/(my-proj-[^/]*)/junk/my-proj:refs/remotes/origin/\1

(Which in a git config, I think may need a double-backslash, but I
forget what the config parser does.)

j.

Re: [RFC] Refspec patterns with * in the middle

[Junio C Hamano]

Daniel Barkalow <barkalow@iabervon.org> writes:

> My use case is actually, more precisely:
>
> some/constant/stuff/$PROJ-$NUMBER/junk/my-proj
>
> Where $NUMBER is the version number, and $PROJ is usually, but not quite 
> always "my-proj"; the exception being that it might be effectively a 
> superproject. So I'd like to have:
>
>  fetch = some/constant/stuff/my-proj-*/junk/my-proj:refs/remotes/origin/*

... and expect "some/constant/stuff/my-proj-2.5/junk/my-proj" to be mapped
to "refs/remotes/origin/2.5"?  I think it does not look too bad.

> But I can live with remote branches like "my-proj-2.4" instead of "2.4".
>
> I think it would make sense, and limit typo damage, to say that the * can 
> only expand to something with a '/' in it if the star has a slash or the 
> end of the string on each side.

I do not understand what you mean by "* can only expand to something with
a '/' in it if ..." part.  None of the examples in your message have a
case where the asterisk matches across directory boundaries, and I thought
you would simply say "* does not match /" and be done with that.

What scenario do you have in mind that wants to match a slash with an
asterisk?

Re: [RFC] Refspec patterns with * in the middle

[Jay Soffian]

On Mon, Mar 2, 2009 at 12:08 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Daniel Barkalow <barkalow@iabervon.org> writes:
>
>> My use case is actually, more precisely:
>>
>> some/constant/stuff/$PROJ-$NUMBER/junk/my-proj
>>
>> Where $NUMBER is the version number, and $PROJ is usually, but not quite
>> always "my-proj"; the exception being that it might be effectively a
>> superproject. So I'd like to have:
>>
>>  fetch = some/constant/stuff/my-proj-*/junk/my-proj:refs/remotes/origin/*
>
> ... and expect "some/constant/stuff/my-proj-2.5/junk/my-proj" to be mapped
> to "refs/remotes/origin/2.5"?  I think it does not look too bad.
>
>> But I can live with remote branches like "my-proj-2.4" instead of "2.4".
>>
>> I think it would make sense, and limit typo damage, to say that the * can
>> only expand to something with a '/' in it if the star has a slash or the
>> end of the string on each side.
>
> I do not understand what you mean by "* can only expand to something with
> a '/' in it if ..." part.  None of the examples in your message have a
> case where the asterisk matches across directory boundaries, and I thought
> you would simply say "* does not match /" and be done with that.
>
> What scenario do you have in mind that wants to match a slash with an
> asterisk?

I think he means the following are valid:

 - foo/bar/*/baz
 - foo/bar/baz/*

But the following is not:

 - foo/bar*/baz

IOW, '*' can only appear as a non-terminating symbol if it is bounded
by '/' on each side.

j.

Re: [RFC] Refspec patterns with * in the middle

[Daniel Barkalow]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1809 bytes --]

On Mon, 2 Mar 2009, Jay Soffian wrote:

> On Mon, Mar 2, 2009 at 12:08 PM, Junio C Hamano <gitster@pobox.com> wrote:
> > Daniel Barkalow <barkalow@iabervon.org> writes:
> >
> >> My use case is actually, more precisely:
> >>
> >> some/constant/stuff/$PROJ-$NUMBER/junk/my-proj
> >>
> >> Where $NUMBER is the version number, and $PROJ is usually, but not quite
> >> always "my-proj"; the exception being that it might be effectively a
> >> superproject. So I'd like to have:
> >>
> >>  fetch = some/constant/stuff/my-proj-*/junk/my-proj:refs/remotes/origin/*
> >
> > ... and expect "some/constant/stuff/my-proj-2.5/junk/my-proj" to be mapped
> > to "refs/remotes/origin/2.5"?  I think it does not look too bad.
> >
> >> But I can live with remote branches like "my-proj-2.4" instead of "2.4".
> >>
> >> I think it would make sense, and limit typo damage, to say that the * can
> >> only expand to something with a '/' in it if the star has a slash or the
> >> end of the string on each side.
> >
> > I do not understand what you mean by "* can only expand to something with
> > a '/' in it if ..." part.  None of the examples in your message have a
> > case where the asterisk matches across directory boundaries, and I thought
> > you would simply say "* does not match /" and be done with that.
> >
> > What scenario do you have in mind that wants to match a slash with an
> > asterisk?
> 
> I think he means the following are valid:
> 
>  - foo/bar/*/baz
>  - foo/bar/baz/*
> 
> But the following is not:
> 
>  - foo/bar*/baz
> 
> IOW, '*' can only appear as a non-terminating symbol if it is bounded
> by '/' on each side.

You have my criterion right, but I want that to be valid, but only match 
things like "foo/bar-something/baz", not "foo/bar-a/b/baz".

	-Daniel
*This .sig left intentionally blank*

Re: [RFC] Refspec patterns with * in the middle

[Jay Soffian]

On Mon, Mar 2, 2009 at 1:25 PM, Daniel Barkalow <barkalow@iabervon.org> wrote:
>> But the following is not:
>>
>>  - foo/bar*/baz
>>
>> IOW, '*' can only appear as a non-terminating symbol if it is bounded
>> by '/' on each side.
>
> You have my criterion right, but I want that to be valid, but only match
> things like "foo/bar-something/baz", not "foo/bar-a/b/baz".

Ah, that makes sense. Perhaps use "**" to mean matching across path
components which is what rsync does:

 o  a '*' matches any non-empty path component (it stops at slashes).
 o  use '**' to match anything, including slashes.

?

j.

Re: [RFC] Refspec patterns with * in the middle

[Junio C Hamano]

Jay Soffian <jaysoffian@gmail.com> writes:

> On Mon, Mar 2, 2009 at 1:25 PM, Daniel Barkalow <barkalow@iabervon.org> wrote:
>>> But the following is not:
>>>
>>>  - foo/bar*/baz
>>>
>>> IOW, '*' can only appear as a non-terminating symbol if it is bounded
>>> by '/' on each side.
>>
>> You have my criterion right, but I want that to be valid, but only match
>> things like "foo/bar-something/baz", not "foo/bar-a/b/baz".
>
> Ah, that makes sense. Perhaps use "**" to mean matching across path
> components which is what rsync does:
>
>  o  a '*' matches any non-empty path component (it stops at slashes).
>  o  use '**' to match anything, including slashes.
>
> ?

I personally do not think that makes much sense (and I find ** ugly, too).

We traditionally supported '*' only at the end, and it always has meant
"match through the end, including slashes".

Requiring 'match including slashes' to be spelled as '**' only when it is
not at the end is unnecessarily confusing.

Is there a valid use case when * wants to match across directory
boundaries when it is not at the end?  I offhand do not think of a sane
one.

So, it might make it easiest to understand if we say * usually does not
match slash, except when it is used at the end immediately after a slash,
in which case it means "match through the end".

Re: [RFC] Refspec patterns with * in the middle

[Jay Soffian]

On Mon, Mar 2, 2009 at 5:39 PM, Junio C Hamano <gitster@pobox.com> wrote:
> I personally do not think that makes much sense (and I find ** ugly, too).
>
> We traditionally supported '*' only at the end, and it always has meant
> "match through the end, including slashes".

Oh, good point, I had not thought that through very well.

> Requiring 'match including slashes' to be spelled as '**' only when it is
> not at the end is unnecessarily confusing.
>
> Is there a valid use case when * wants to match across directory
> boundaries when it is not at the end?  I offhand do not think of a sane
> one.
>
> So, it might make it easiest to understand if we say * usually does not
> match slash, except when it is used at the end immediately after a slash,
> in which case it means "match through the end".

That seems simple enough.

j.

Re: [RFC] Refspec patterns with * in the middle

[Daniel Barkalow]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1959 bytes --]

On Mon, 2 Mar 2009, Junio C Hamano wrote:

> Jay Soffian <jaysoffian@gmail.com> writes:
> 
> > On Mon, Mar 2, 2009 at 1:25 PM, Daniel Barkalow <barkalow@iabervon.org> wrote:
> >>> But the following is not:
> >>>
> >>>  - foo/bar*/baz
> >>>
> >>> IOW, '*' can only appear as a non-terminating symbol if it is bounded
> >>> by '/' on each side.
> >>
> >> You have my criterion right, but I want that to be valid, but only match
> >> things like "foo/bar-something/baz", not "foo/bar-a/b/baz".
> >
> > Ah, that makes sense. Perhaps use "**" to mean matching across path
> > components which is what rsync does:
> >
> >  o  a '*' matches any non-empty path component (it stops at slashes).
> >  o  use '**' to match anything, including slashes.
> >
> > ?
> 
> I personally do not think that makes much sense (and I find ** ugly, too).
> 
> We traditionally supported '*' only at the end, and it always has meant
> "match through the end, including slashes".
> 
> Requiring 'match including slashes' to be spelled as '**' only when it is
> not at the end is unnecessarily confusing.
> 
> Is there a valid use case when * wants to match across directory
> boundaries when it is not at the end?  I offhand do not think of a sane
> one.

Maybe:

/refs/imported/$GROUP/$USER/project -> $GROUP/$USER

or

/refs/imported/sandbox/$USER/$TOPIC/project -> $USER/$TOPIC

The issue, in my case, is importing from a system where branches contain 
projects instead of projects containing branches (and everything is a 
single namespace). So I want to match an insane (for us) LHS with a sane 
RHS to get stuff into reasonable shape. I don't really care about any 
patterns where the branch identifier is multiple components, but I 
wouldn't be surprised if somebody did.

Oh, and it looks like "?" is reserved and currently unused, so we could 
have * match one or more full path components, and ? match partial path 
components.

	-Daniel
*This .sig left intentionally blank*

Re: [RFC] Refspec patterns with * in the middle

[Junio C Hamano]

Daniel Barkalow <barkalow@iabervon.org> writes:

> The issue, in my case, is importing from a system where branches contain 
> projects instead of projects containing branches (and everything is a 
> single namespace). So I want to match an insane (for us) LHS with a sane 
> RHS to get stuff into reasonable shape. I don't really care about any 
> patterns where the branch identifier is multiple components, but I 
> wouldn't be surprised if somebody did.

Isn't this just getting more and more insane?  Is this really worth
supporting, I have to wonder...

> Oh, and it looks like "?" is reserved and currently unused, so we could 
> have * match one or more full path components, and ? match partial path 
> components.

Well, "?" is not allowed exactly because it often is used to match a
single character by things like for-each-ref.

Re: [RFC] Refspec patterns with * in the middle

[David Kastrup]

Junio C Hamano <gitster@pobox.com> writes:

> Daniel Barkalow <barkalow@iabervon.org> writes:
>
>> The issue, in my case, is importing from a system where branches contain 
>> projects instead of projects containing branches (and everything is a 
>> single namespace). So I want to match an insane (for us) LHS with a sane 
>> RHS to get stuff into reasonable shape. I don't really care about any 
>> patterns where the branch identifier is multiple components, but I 
>> wouldn't be surprised if somebody did.
>
> Isn't this just getting more and more insane?  Is this really worth
> supporting, I have to wonder...
>
>> Oh, and it looks like "?" is reserved and currently unused, so we could 
>> have * match one or more full path components, and ? match partial path 
>> components.
>
> Well, "?" is not allowed exactly because it often is used to match a
> single character by things like for-each-ref.

Let me present a use case where the current matching rules have changed
working well at some point of time.

We have a product called dsp which sits in Subversion under the top
server repository as

dsp/{trunk,branches,tags}

Then we have projects that have a structure including

projekte/some-project/{trunk,branches,tags}/dsp

which usually is an external link to dsp/trunk.  However, sometimes a
project needs its own local variations of dsp, in which case
projekte/some-project/trunk/dsp becomes a properly populated copy of
dsp/trunk.

Now in git, I want to track those non-externals as branches of the dsp
checkout: that way I get cherry-picking and merging and stuff like that.
To have this work out, I need to have the dsp git project pulling stuff
from any projekte/some-project/trunk/dsp as a branch called
some-project.

I had a git-svn configuration at one point of time that did that, but
the matching rules have changed since then.

So in this case, the existing Subversion structure (which makes quite
good sense) makes wildcards in non-terminal positions desirable.

-- 
David Kastrup

Re: [RFC] Refspec patterns with * in the middle

[Daniel Barkalow]

On Mon, 2 Mar 2009, Junio C Hamano wrote:

> Daniel Barkalow <barkalow@iabervon.org> writes:
> 
> > My use case is actually, more precisely:
> >
> > some/constant/stuff/$PROJ-$NUMBER/junk/my-proj
> >
> > Where $NUMBER is the version number, and $PROJ is usually, but not quite 
> > always "my-proj"; the exception being that it might be effectively a 
> > superproject. So I'd like to have:
> >
> >  fetch = some/constant/stuff/my-proj-*/junk/my-proj:refs/remotes/origin/*
> 
> ... and expect "some/constant/stuff/my-proj-2.5/junk/my-proj" to be mapped
> to "refs/remotes/origin/2.5"?  I think it does not look too bad.

Yeah.

> > But I can live with remote branches like "my-proj-2.4" instead of "2.4".
> >
> > I think it would make sense, and limit typo damage, to say that the * can 
> > only expand to something with a '/' in it if the star has a slash or the 
> > end of the string on each side.
> 
> I do not understand what you mean by "* can only expand to something with
> a '/' in it if ..." part.  None of the examples in your message have a
> case where the asterisk matches across directory boundaries, and I thought
> you would simply say "* does not match /" and be done with that.
> 
> What scenario do you have in mind that wants to match a slash with an
> asterisk?

Backwards compatibility? Currently, refs/heads/*:refs/remotes/origin/* 
will match refs/heads/js/notes with refs/remotes/origin/js/notes, because 
it simply copies everything after the fixed prefix.

I was assuming that that particular matching is an existing requirement 
(and that you used it regularly to share topic-maintenance work between 
machines).

So my idea was that a path component of "*" would be able to match 
multiple levels, while a path component of "something-*" or "*-something" 
can only match a single path component that starts or ends with the right 
thing. Looking at it now, I think it might be more sane to use "%" for the 
second case, and say: "*" must be used as a full path component, and 
matches one or more path components; "%" may be used as part of a path 
component, and doesn't match the path separator; you can only use one such 
character, and both sides of the refspec have to agree on which.

I think the problem cases were typos like:

refs/heads*:refs/remotes/origin/* (used to cause problems)

and

refs/heads/*:refs/remotes/origin* (gives you the originmaster remote?)

	-Daniel
*This .sig left intentionally blank*