[feature request] Is it possible to have git tag can be sorted and filtered by semver?

[feature request] Is it possible to have git tag can be sorted and filtered by semver?

[Thaina Yu]

Currently there is `ls-remote` and `tag` that has option to
--sort=version:refname which is really useful when we have tag as
version format and let it sort as numeric version

But it could be better if we can also sort and filter the version in
the range of semver format. So we can use git system as dependency
management system natively

So I would like to propose a new sort option semver:refname

`git ls-remote --sort=semver:refname myRepoUrl refs/tags/v{^0.*.*}`
where brace {} can be used to define semver versioning ranges for the
patterns in addition to glob filter

Re: [feature request] Is it possible to have git tag can be sorted and filtered by semver?

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 1605 bytes --]

On 2024-07-22 at 16:58:11, Thaina Yu wrote:
> Currently there is `ls-remote` and `tag` that has option to
> --sort=version:refname which is really useful when we have tag as
> version format and let it sort as numeric version
> 
> But it could be better if we can also sort and filter the version in
> the range of semver format. So we can use git system as dependency
> management system natively
> 
> So I would like to propose a new sort option semver:refname
> 
> `git ls-remote --sort=semver:refname myRepoUrl refs/tags/v{^0.*.*}`
> where brace {} can be used to define semver versioning ranges for the
> patterns in addition to glob filter

Assuming we add such a feature, how does sorting by SemVer differ from
the current version sorting?  That is, where is the current version
sorting deficient for SemVer?  Also, what do you want to happen when a
tag doesn't meet SemVer requirements (note that the "v" prefix is not
allowed in SemVer, although it's customary in tags)?

As for the special range syntax, I think the typical suggestion is to
filter the output of ls-remote or for-each-ref by piping it to a
suitable program.  Perl or Ruby are common choices here, and both could
easily parse SemVer tags.  For example:

  git for-each-ref --sort=v:refname refs/tags/ |
  ruby -ne 'if %r[\trefs/tags/v(\d+)\.(\d+)\.(\d+)$]; ver = Regexp.last_match[1..3].map(&:to_i); puts $_ if [[2, 6, 3], ver, [2, 15, 2]].sort[1] == ver; end'

Git is intentionally designed to support this kind of shell scripting.
-- 
brian m. carlson (they/them or he/him)
Toronto, Ontario, CA

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

RE: [feature request] Is it possible to have git tag can be sorted and filtered by semver?

[rsbecker]

On Monday, July 22, 2024 5:31 PM, brian m. carlson wrote:
>On 2024-07-22 at 16:58:11, Thaina Yu wrote:
>> Currently there is `ls-remote` and `tag` that has option to
>> --sort=version:refname which is really useful when we have tag as
>> version format and let it sort as numeric version
>>
>> But it could be better if we can also sort and filter the version in
>> the range of semver format. So we can use git system as dependency
>> management system natively
>>
>> So I would like to propose a new sort option semver:refname
>>
>> `git ls-remote --sort=semver:refname myRepoUrl refs/tags/v{^0.*.*}`
>> where brace {} can be used to define semver versioning ranges for the
>> patterns in addition to glob filter
>
>Assuming we add such a feature, how does sorting by SemVer differ from the
>current version sorting?  That is, where is the current version sorting deficient for
>SemVer?  Also, what do you want to happen when a tag doesn't meet SemVer
>requirements (note that the "v" prefix is not allowed in SemVer, although it's
>customary in tags)?

Currently, tags would be sorted as follows (simple example):
1.10.0
1.2.0
1.9.1

With semver, the tags would be:
1.2.0
1.9.1
1.10.0

My take is that this, if implemented, would need to be more general, and include prefix and suffix handling, so:
v1.2.0
v1.9.0
v1.10.0

should sort as appropriate. We might need something like v({semver}), or a more general regex-like (prefix){0,1}(semver){1}(suffix){0,1}.

While at it, having a reverse sort would also be useful. For platforms that have semver-util, this can be trivially scripted. For exotics, no such luck, as semver-util is not especially portable, not for lack of trying.

>As for the special range syntax, I think the typical suggestion is to filter the output of
>ls-remote or for-each-ref by piping it to a suitable program.  Perl or Ruby are
>common choices here, and both could easily parse SemVer tags.  For example:
>
>  git for-each-ref --sort=v:refname refs/tags/ |
>  ruby -ne 'if %r[\trefs/tags/v(\d+)\.(\d+)\.(\d+)$]; ver =
>Regexp.last_match[1..3].map(&:to_i); puts $_ if [[2, 6, 3], ver, [2, 15, 2]].sort[1] ==
>ver; end'
>
>Git is intentionally designed to support this kind of shell scripting.

I think implementing both wrapped semver and reverse sort in git tag might be useful for large projects, like git and OpenSSL where the number of tags is large. It would make finding time-ordered releases somewhat easier.

--Randall

Re: [feature request] Is it possible to have git tag can be sorted and filtered by semver?

[Junio C Hamano]

<rsbecker@nexbridge.com> writes:

> Currently, tags would be sorted as follows (simple example):
> 1.10.0
> 1.2.0
> 1.9.1
>
> With semver, the tags would be:
> 1.2.0
> 1.9.1
> 1.10.0
>
> My take is that this, if implemented, would need to be more general, and include prefix and suffix handling, so:
> v1.2.0
> v1.9.0
> v1.10.0

Without implemeting anything, wouldn't "--sort=version:refname"
suggested by Brian work just fine?

In the git repository:

    $ git tag --sort=version:refname | grep -e 'v2\.[0-9][0-9]*\.[0-9][0-9]*$'
    v2.0.0
    ...
    v2.0.5
    v2.1.0
    v2.1.1
    v2.1.2
    v2.1.3
    v2.1.4
    v2.2.0
    ...
    v2.9.0
    ...
    v2.9.5
    v2.10.0
    v2.10.1
    v2.10.2
    v2.10.3
    v2.10.4
    v2.10.5
    v2.11.0
    ...
    v2.45.1
    v2.45.2

sorting 2.1.x series a lot earlier than 2.10.x series that is sorted
after 2.9.x series.

Isn't it exactly what the original requester wants to see?

One mechanism I find it lacking is a mechanism to tell the tool that
-preW in X.Y.Z-preW is a suffix that signals that X.Y.Z-preW comes
before X.Y.Z, while -postW in X.Y.Z-postW is a suffix that signals
that X.Y.Z-postW comes after X.Y.Z.  Using such a mechanism, we
could say v2.1.0-rc0 comes before v2.1.0-rc1, which in turn comes
before v2.1.0-rc2, which in turn comes before v2.1.0 proper.

But other than the ordering of the final release and -rcX, I think
"--sort=version:refname" does the right thing already.

Re: [feature request] Is it possible to have git tag can be sorted and filtered by semver?

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 4951 bytes --]

On 2024-07-22 at 23:14:03, rsbecker@nexbridge.com wrote:
> On Monday, July 22, 2024 5:31 PM, brian m. carlson wrote:
> >Assuming we add such a feature, how does sorting by SemVer differ from the
> >current version sorting?  That is, where is the current version sorting deficient for
> >SemVer?  Also, what do you want to happen when a tag doesn't meet SemVer
> >requirements (note that the "v" prefix is not allowed in SemVer, although it's
> >customary in tags)?
> 
> Currently, tags would be sorted as follows (simple example):
> 1.10.0
> 1.2.0
> 1.9.1

I agree that this happens without any --sort option.

> With semver, the tags would be:
> 1.2.0
> 1.9.1
> 1.10.0
> 
> My take is that this, if implemented, would need to be more general, and include prefix and suffix handling, so:
> v1.2.0
> v1.9.0
> v1.10.0
> 
> should sort as appropriate. We might need something like v({semver}), or a more general regex-like (prefix){0,1}(semver){1}(suffix){0,1}.

However, this is the behaviour I see with --sort=version:refname (or
v:refname).  For example, the command I provided below running in the
Git repository sorts v2.9.5 before v2.10.0, which I believe is how this
was supposed to work.  Of course, I could be totally off base, which is
why I'm asking for clarification so I can understand better.

I think it's also worth asking what happens for tags that don't match
that still.  For example, let's assume for the moment that Git used
SemVer.  I have added tags in my Git repo for when I send a series, like
so:

  7a9ba024ccdc440095537cf53ce69a5749798165 commit refs/tags/sent/credential-alwaysauth/v1
  a74efc1699038e898960c2c55185f32aade6a88a commit refs/tags/sent/credential-alwaysauth/v2
  ac45947b34d003f827d15a8623c0125fb12ec261 commit refs/tags/sent/credential-alwaysauth/v3

Those clearly don't meet SemVer and will need to be sorted _somehow_
(before or after SemVer tags?), and users will want to know how.

> While at it, having a reverse sort would also be useful. For platforms that have semver-util, this can be trivially scripted. For exotics, no such luck, as semver-util is not especially portable, not for lack of trying.

I think we have this by using the `-` prefix, such as
`--sort=-version:refname`.  I agree this is very useful, though, since
it's an O(n) operation to reverse the list, which, as you mention below,
might be large.  If we add SemVer sorting, we'll definitely want it to
work nicely with reverse sorting.

> >As for the special range syntax, I think the typical suggestion is to filter the output of
> >ls-remote or for-each-ref by piping it to a suitable program.  Perl or Ruby are
> >common choices here, and both could easily parse SemVer tags.  For example:
> >
> >  git for-each-ref --sort=v:refname refs/tags/ |
> >  ruby -ne 'if %r[\trefs/tags/v(\d+)\.(\d+)\.(\d+)$]; ver =
> >Regexp.last_match[1..3].map(&:to_i); puts $_ if [[2, 6, 3], ver, [2, 15, 2]].sort[1] ==
> >ver; end'
> >
> >Git is intentionally designed to support this kind of shell scripting.
> 
> I think implementing both wrapped semver and reverse sort in git tag might be useful for large projects, like git and OpenSSL where the number of tags is large. It would make finding time-ordered releases somewhat easier.

My worry is that a special range syntax isn't going to cover all the
possible needs.  For example, I might know that [v2.5.1, v4.0.0) will
work (despite the incompatible version at v3, my code isn't affected),
but it's very difficult to express that via a range, since what I want
is basically a tuple sort on [2, 5, 1] <= ver < [4, 0, 0].  That's easy
in Ruby (well, it's not pretty[0], but I did it above), but you can't
simply specify a range in any one field, because v2.6.0 and v3.0.5 would
both match the pattern.

This kind of thing is very common with lots of Ruby gems, which, while
they have a backwards incompatible change, don't change all that much,
so several major versions might be acceptable.  In that context, it's
likely that some sort of external filtering might be more general and
robust and meet people's needs better.

However, I'm open to being convinced otherwise if some compelling syntax
comes up.  I'm just asking questions here because I'm not super sure
what the use cases are and I think more information about the proposal
might help people decide better.

[0] That's because Ruby supports comparisons on Array via <=> (and thus
sorting), but it doesn't implement < and > themselves (although == is
implemented).  Therefore, I had to sort the three versions, and if the
given version was still in the middle afterwards, then it was in range.

Another, less pretty (in my view) option is to call <=> directly and
compare against -1 or 1.  If there are any Rubyists on the list with
better ideas, I'm open to hear them.
-- 
brian m. carlson (they/them or he/him)
Toronto, Ontario, CA

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [feature request] Is it possible to have git tag can be sorted and filtered by semver?

[Thaina Yu]

The sorting output are not very differ from `--sort=v:refname` the key
difference is ability to also filter with the semver range

As suggested, the glob pattern start to have a problem filtering out
the range of tags numerically and so we need to pipe the output to
another system such as grep. Which is not supported in windows and
become harder to do a tooling system that need to piped to separate
platform specific tools. We also need to convert semver into regex
such that it handle ranges we want properly. So I would like to
request it to be option of the git itself

As for the syntax. I propose the patterns `refs/tags/v{
__semverranges__ }` so that `v` may be prefix or not is up to user,
outside of brace `{}` is just normal glob and only inside will be
matched as semver. The suffix such as `-alpha0` or `-pre1` or `-rc3`
should also be sorted with semver rules and filter out all other
unmatched character (except git tag system suffix such as `^{}`)

On Tue, 23 Jul 2024 at 07:05, brian m. carlson
<sandals@crustytoothpaste.net> wrote:
>
> On 2024-07-22 at 23:14:03, rsbecker@nexbridge.com wrote:
> > On Monday, July 22, 2024 5:31 PM, brian m. carlson wrote:
> > >Assuming we add such a feature, how does sorting by SemVer differ from the
> > >current version sorting?  That is, where is the current version sorting deficient for
> > >SemVer?  Also, what do you want to happen when a tag doesn't meet SemVer
> > >requirements (note that the "v" prefix is not allowed in SemVer, although it's
> > >customary in tags)?
> >
> > Currently, tags would be sorted as follows (simple example):
> > 1.10.0
> > 1.2.0
> > 1.9.1
>
> I agree that this happens without any --sort option.
>
> > With semver, the tags would be:
> > 1.2.0
> > 1.9.1
> > 1.10.0
> >
> > My take is that this, if implemented, would need to be more general, and include prefix and suffix handling, so:
> > v1.2.0
> > v1.9.0
> > v1.10.0
> >
> > should sort as appropriate. We might need something like v({semver}), or a more general regex-like (prefix){0,1}(semver){1}(suffix){0,1}.
>
> However, this is the behaviour I see with --sort=version:refname (or
> v:refname).  For example, the command I provided below running in the
> Git repository sorts v2.9.5 before v2.10.0, which I believe is how this
> was supposed to work.  Of course, I could be totally off base, which is
> why I'm asking for clarification so I can understand better.
>
> I think it's also worth asking what happens for tags that don't match
> that still.  For example, let's assume for the moment that Git used
> SemVer.  I have added tags in my Git repo for when I send a series, like
> so:
>
>   7a9ba024ccdc440095537cf53ce69a5749798165 commit refs/tags/sent/credential-alwaysauth/v1
>   a74efc1699038e898960c2c55185f32aade6a88a commit refs/tags/sent/credential-alwaysauth/v2
>   ac45947b34d003f827d15a8623c0125fb12ec261 commit refs/tags/sent/credential-alwaysauth/v3
>
> Those clearly don't meet SemVer and will need to be sorted _somehow_
> (before or after SemVer tags?), and users will want to know how.
>
> > While at it, having a reverse sort would also be useful. For platforms that have semver-util, this can be trivially scripted. For exotics, no such luck, as semver-util is not especially portable, not for lack of trying.
>
> I think we have this by using the `-` prefix, such as
> `--sort=-version:refname`.  I agree this is very useful, though, since
> it's an O(n) operation to reverse the list, which, as you mention below,
> might be large.  If we add SemVer sorting, we'll definitely want it to
> work nicely with reverse sorting.
>
> > >As for the special range syntax, I think the typical suggestion is to filter the output of
> > >ls-remote or for-each-ref by piping it to a suitable program.  Perl or Ruby are
> > >common choices here, and both could easily parse SemVer tags.  For example:
> > >
> > >  git for-each-ref --sort=v:refname refs/tags/ |
> > >  ruby -ne 'if %r[\trefs/tags/v(\d+)\.(\d+)\.(\d+)$]; ver =
> > >Regexp.last_match[1..3].map(&:to_i); puts $_ if [[2, 6, 3], ver, [2, 15, 2]].sort[1] ==
> > >ver; end'
> > >
> > >Git is intentionally designed to support this kind of shell scripting.
> >
> > I think implementing both wrapped semver and reverse sort in git tag might be useful for large projects, like git and OpenSSL where the number of tags is large. It would make finding time-ordered releases somewhat easier.
>
> My worry is that a special range syntax isn't going to cover all the
> possible needs.  For example, I might know that [v2.5.1, v4.0.0) will
> work (despite the incompatible version at v3, my code isn't affected),
> but it's very difficult to express that via a range, since what I want
> is basically a tuple sort on [2, 5, 1] <= ver < [4, 0, 0].  That's easy
> in Ruby (well, it's not pretty[0], but I did it above), but you can't
> simply specify a range in any one field, because v2.6.0 and v3.0.5 would
> both match the pattern.
>
> This kind of thing is very common with lots of Ruby gems, which, while
> they have a backwards incompatible change, don't change all that much,
> so several major versions might be acceptable.  In that context, it's
> likely that some sort of external filtering might be more general and
> robust and meet people's needs better.
>
> However, I'm open to being convinced otherwise if some compelling syntax
> comes up.  I'm just asking questions here because I'm not super sure
> what the use cases are and I think more information about the proposal
> might help people decide better.
>
> [0] That's because Ruby supports comparisons on Array via <=> (and thus
> sorting), but it doesn't implement < and > themselves (although == is
> implemented).  Therefore, I had to sort the three versions, and if the
> given version was still in the middle afterwards, then it was in range.
>
> Another, less pretty (in my view) option is to call <=> directly and
> compare against -1 or 1.  If there are any Rubyists on the list with
> better ideas, I'm open to hear them.
> --
> brian m. carlson (they/them or he/him)
> Toronto, Ontario, CA