Testing for existence of a remote branch from a script

Testing for existence of a remote branch from a script

[Chris Packham]

Hi,

I look after some scripts we use at $dayjob for pushing changes though
our review system.

For some of our repositories we operate a triangular workflow where
changes are fetched from one branch (e.g. 'foo') but are pushed to a
different one ('foo_incoming'). Our CI system runs to test the changes
and when they pass 'foo_incoming' is merged (fast-forward most of the
time) into 'foo'.

The problem I have is not all our projects use this workflow so I've
tried to automate the detection of this. My script does something like

  br=$(git rev-parse --symbolic-full-name
refs/remotes/origin/foo_incoming -- 2>/dev/null || echo
refs/remotes/origin/foo)

The '--' is necessary because if foo_incoming doesn't exist then there
is extra output on stdout that puts off users. But when foo_incoming
does exist then br gets set to `refs/remotes/origin/foo_incoming\n--`.

Is there a better way of checking for the existence of a remote branch?

Thanks,
Chris

Re: Testing for existence of a remote branch from a script

[Torsten Bögershausen]

On Mon, Jan 06, 2025 at 05:40:36PM +1300, Chris Packham wrote:
> Hi,
>
> I look after some scripts we use at $dayjob for pushing changes though
> our review system.
>
> For some of our repositories we operate a triangular workflow where
> changes are fetched from one branch (e.g. 'foo') but are pushed to a
> different one ('foo_incoming'). Our CI system runs to test the changes
> and when they pass 'foo_incoming' is merged (fast-forward most of the
> time) into 'foo'.
>
> The problem I have is not all our projects use this workflow so I've
> tried to automate the detection of this. My script does something like
>
>   br=$(git rev-parse --symbolic-full-name
> refs/remotes/origin/foo_incoming -- 2>/dev/null || echo
> refs/remotes/origin/foo)
>
> The '--' is necessary because if foo_incoming doesn't exist then there
> is extra output on stdout that puts off users. But when foo_incoming
> does exist then br gets set to `refs/remotes/origin/foo_incoming\n--`.
>
> Is there a better way of checking for the existence of a remote branch?

I may have missed something, would that work:
git fetch -p
git branch -r

Re: Testing for existence of a remote branch from a script

[Junio C Hamano]

Torsten Bögershausen <tboegi@web.de> writes:

>> changes are fetched from one branch (e.g. 'foo') but are pushed to a
>> different one ('foo_incoming'). Our CI system runs to test the changes
>> and when they pass 'foo_incoming' is merged (fast-forward most of the
>> time) into 'foo'.
>> ...
>> Is there a better way of checking for the existence of a remote branch?
>
> I may have missed something, would that work:
> git fetch -p
> git branch -r

Or "git ls-remote origin refs/heads/foo-incoming" and see if it
yields anything?

The "workflow" makes me wonder how it would be bootstrapped, though.

When you add a new branch, "bar", to be pre-reviewed before getting
merged at the server side, you want people to push to
"bar-incoming".  Before the very initial update to "bar-incoming"
that pushes into "bar-incoming" and creates it, there wouldn't be
"bar-incoming" on the remote side, would there?  So "see if
foo-incoming exists and change the behaviour on the client end
accordingly" may not be a strategy to pursue.

If we wanted to support the "workflow" natively, the way we would do
so would be to introduce a protocol capability that allows the
server side to advertise:

    If you want to update branch B, you are not allowed to do so
    directly.  Instead, you are expected to push your changes to
    update 'refs/for/B'

and then "git push" on the client end would notice the capability
and turns "git push origin B" (or, more likely, the user is on local
branch B that is to build on the remote branch B from 'origin', and
"git push" with no arguments would do the right thing) into such an
update.

I am not suggesting that we jump to the above immediately.  But the
reason why I am bringing it up is because The "how would I see if
they have foo-incoming?" smells like seeking a way to implement such
a custom capability advertisement outside Git.

A few random thoughts.

 - Would it be useful if we introduced the ability to advertise
   "custom capabilities" from the receiving end of the connection,
   that does not affect how the rest of Git behaves at all?  It
   would be sort of the reverse of --server-option, which is a
   mechanism to let the client to tell the other side out of band
   information that the rest of Git is oblivious.

   The other side of course needs a way to inspect what capabilities
   are advertised.  For "--server-option", I do not think our server
   end does anything special, but other implementations can act on
   them.  This new thing can start the same way.

 - If this is a poor-man's custom capability advertisement, perhaps
   the server end can create a ref "refs/capabilities/incoming"
   (whose value does not really matter) and your client side can see
   if there is such a ref with "ls-remote"?  That may be a more
   robust thing to do instead, perhaps, as you do not need to worry
   about "What about a new branch 'bar'?" bootstrapping issues.

Re: Testing for existence of a remote branch from a script

[Theodore Ts'o]

On Mon, Jan 06, 2025 at 07:30:32AM -0800, Junio C Hamano wrote:
>  - Would it be useful if we introduced the ability to advertise
>    "custom capabilities" from the receiving end of the connection,
>    that does not affect how the rest of Git behaves at all?

So if enhancing the git server's functionality (either via git
ls-remotes or some other operation) is on the table, one of the things
that I would really love is some way of asking the question is "git
commit <SHA hash>" in the remote repository reachable via some branch
or git tag?", and optionally, "which git branch/tag should be fetched
if the testing infrastructure wants to be able to test that specific
git commit ID?"

I could solve that problem by doing a "git fetch", but in my case, the
specific repository in question might not even be on the local client
where the user is executing "gce-xfstests -c ext4/4k -g auto --repo
stable-rc.git --commit 47c2f92131c4".  (This command launches a VM
which actually builds the kernel, and I'd like to be able to give
immediate feedback about whether the command will fail due to the user
typo'ing the git commit id without either waiting for the Google Cloud
VM to launch and then fail, or doing a "git fetch" and determining
whether the git commit is there.  I might be executing this command
while on a very slow networking link, such as when I'm in an airplane
or on a cruise ship, and so the cost of doing the "git fetch" might be
prohibitive.)

Thanks,

						- Ted

Re: Testing for existence of a remote branch from a script

[Junio C Hamano]

"Theodore Ts'o" <tytso@mit.edu> writes:

> So if enhancing the git server's functionality (either via git
> ls-remotes or some other operation) is on the table, ...

Enhancements that do not require breaking backward compatibility is
always on the table ;-).

> one of the things
> that I would really love is some way of asking the question is "git
> commit <SHA hash>" in the remote repository reachable via some branch
> or git tag?", and optionally, "which git branch/tag should be fetched
> if the testing infrastructure wants to be able to test that specific
> git commit ID?"

Both sounds like a useful thing to do, but I wonder how generic
these should be and at the same time how common a narrowed-down
feature would suffice.  If we try to make it generally very useful,
at some point, we'd cross the line where we'd be better off doing
"run ssh and execute these Git commands" over the wire X-<.

There are server-side-minded folks who are extending "cat-file --batch"
to allow you to ask about objects you do not have but the other end
has, if I am not mistaken,  by the "remote-object-info" feature?

I wonder if these more advanced "info" about objects you mentioned
fit into the picture well as part of it.

Thanks.

Re: Testing for existence of a remote branch from a script

[Chris Packham]

On Tue, Jan 7, 2025 at 4:30 AM Junio C Hamano <gitster@pobox.com> wrote:
>
> Torsten Bögershausen <tboegi@web.de> writes:
>
> >> changes are fetched from one branch (e.g. 'foo') but are pushed to a
> >> different one ('foo_incoming'). Our CI system runs to test the changes
> >> and when they pass 'foo_incoming' is merged (fast-forward most of the
> >> time) into 'foo'.
> >> ...
> >> Is there a better way of checking for the existence of a remote branch?
> >
> > I may have missed something, would that work:
> > git fetch -p
> > git branch -r
>
> Or "git ls-remote origin refs/heads/foo-incoming" and see if it
> yields anything?

git ls-remote --exit-code origin refs/heads/foo_incoming would work
for me. And I think it would save me a fetch to get past one
bootstrapping issue.

>
> The "workflow" makes me wonder how it would be bootstrapped, though.
>

Currently we require the first person to know what they're doing and
do the initial push as `HEAD:refs/heads/foo_incoming` the CI tooling
we have set up knows to create `foo` if it doesn't already exist.
That's probably not too bad because for an unborn branch you'd need to
say `HEAD:refs/heads/foo` anyway before subsequent pushes would work
without the refspec.

Our workflow does require some discipline as that first push should
just be either an existing commit (when branching in an existing repo)
or a largely empty "Initial commit" for a new repository. That all
relies on a "gentlepersons's agreement" that the first push to one of
these branches doesn't require review because there's nothing really
to enforce that policy. Pushing to the non-incoming branch is
restricted to the CI user via a hook on the server for plain git or
with access permissions for gerrit.

> When you add a new branch, "bar", to be pre-reviewed before getting
> merged at the server side, you want people to push to
> "bar-incoming".  Before the very initial update to "bar-incoming"
> that pushes into "bar-incoming" and creates it, there wouldn't be
> "bar-incoming" on the remote side, would there?  So "see if
> foo-incoming exists and change the behaviour on the client end
> accordingly" may not be a strategy to pursue.

I know git has some tooling for a triangular workflow
(branch.<name>.pushRemote) I'm not sure if that can be setup to push
to a differently named branch. Some kind of branch.<name>.pushRefSpec
might help if it could support *:*_incoming.

The main thing I'm scripting on top of is Gerrit's magic refs/for/* so
even if git supported our version of a triangular workflow workflow
I'd still need something to decide when to use refs/for/foo or
refs/for/foo_incoming when dealing with Gerrit.

> If we wanted to support the "workflow" natively, the way we would do
> so would be to introduce a protocol capability that allows the
> server side to advertise:
>
>     If you want to update branch B, you are not allowed to do so
>     directly.  Instead, you are expected to push your changes to
>     update 'refs/for/B'
>
> and then "git push" on the client end would notice the capability
> and turns "git push origin B" (or, more likely, the user is on local
> branch B that is to build on the remote branch B from 'origin', and
> "git push" with no arguments would do the right thing) into such an
> update.

For the most part we're happy with using a pre-receive hook on the
server to block updates when required. We're able to put out an error
message that tells the user to push to foo_incoming.

Having some kind of config that made `git push` do the right thing
would be great as it would save the user some typing. I don't know how
much advertising from the git server this would really require (we
could easily update our reject hook to advertise some `git config`
settings).

>
> I am not suggesting that we jump to the above immediately.  But the
> reason why I am bringing it up is because The "how would I see if
> they have foo-incoming?" smells like seeking a way to implement such
> a custom capability advertisement outside Git.
>
> A few random thoughts.
>
>  - Would it be useful if we introduced the ability to advertise
>    "custom capabilities" from the receiving end of the connection,
>    that does not affect how the rest of Git behaves at all?  It
>    would be sort of the reverse of --server-option, which is a
>    mechanism to let the client to tell the other side out of band
>    information that the rest of Git is oblivious.
>
>    The other side of course needs a way to inspect what capabilities
>    are advertised.  For "--server-option", I do not think our server
>    end does anything special, but other implementations can act on
>    them.  This new thing can start the same way.
>
>  - If this is a poor-man's custom capability advertisement, perhaps
>    the server end can create a ref "refs/capabilities/incoming"
>    (whose value does not really matter) and your client side can see
>    if there is such a ref with "ls-remote"?  That may be a more
>    robust thing to do instead, perhaps, as you do not need to worry
>    about "What about a new branch 'bar'?" bootstrapping issues.
>

Re: Testing for existence of a remote branch from a script

[Theodore Ts'o]

On Mon, Jan 06, 2025 at 05:40:36PM +1300, Chris Packham wrote:
> I look after some scripts we use at $dayjob for pushing changes though
> our review system.
> ..
> Is there a better way of checking for the existence of a remote branch?

In gce-xfstests[1] I do this via "git ls-remote":

validate_branch_name()
{
    if test -z "$GIT_REPO"
    then
        echo "GIT_REPO is neither found in the config file nor provided with --repo"
        exit 1
    fi
    if [[ "$GIT_REPO" != *".git" ]]; then
        GIT_REPO="$GIT_REPO.git"
    fi
    if ! git ls-remote "$GIT_REPO" > /dev/null; then
	echo -e "Repo not found: $GIT_REPO\n"
	exit 1
    elif ! git ls-remote --heads  --exit-code "$GIT_REPO" $1 > /dev/null; then
	echo -e "$1 is not a valid branch of $GIT_REPO"
	exit 1
    fi
}

See run-fstests/util/parse_cli[2] for this function, and a related
function, validate_commit_name if you also want to accept git tags.
(The validate_commit_name function isn't perfect, since I don't want
to fetch the full git repo to validate a SHA hash specifier, but it's
good enough to validate typo'ed tag or branch names.)

[1] https://thunk.org/gce-xfstests
[2] https://github.com/tytso/xfstests-bld/blob/master/run-fstests/util/parse_cli

Cheers,

						- Ted