[PATCH] ls-remote: add "--diff" option to show only refs that differ

[PATCH] ls-remote: add "--diff" option to show only refs that differ

[Linus Torvalds]

From: Linus Torvalds <torvalds@linux-foundation.org>
Date: Thu, 2 Feb 2017 11:37:49 -0800
Subject: [PATCH] ls-remote: add "--diff" option to show only refs that differ

My main use of "git ls-remote" tends to be to check what the other end
has when some pull request goes wrong (they forgot to push, or they used
the wrong ref name or whatever), and it ends up being hard to see all
the relevant data from the noise of people just having the same basic
tags etc from upstream.

So this adds a "--diff" option that shows only the refs that are
different from the local repository.  So when somebody asks me to pull,
I can now just trivially look at what they have that isn't already my
basic branches and tags.

Note that "--diff" implies "--refs" (ie it also disables showing peeled
tags).

Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
---

This is not a big deal, but maybe others have the same issues I've had. 
And maybe nobody else ever uses "git ls-remote". I dunno.

Also, I considered adding this feature as a more generic flag to 
"check_ref_type()" (ie add a REF_NONLOCAL option to complete the existing 
REF_NORMAL/REF_HEAD/etc flags), but that would have been a more involved 
patch and I'm not convinced it makes much sense for any other use, so I 
made it a specific local hack to ls-remote instead.

Comments?

 builtin/ls-remote.c | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/builtin/ls-remote.c b/builtin/ls-remote.c
index 66cdd45cc..23469c3a6 100644
--- a/builtin/ls-remote.c
+++ b/builtin/ls-remote.c
@@ -2,6 +2,7 @@
 #include "cache.h"
 #include "transport.h"
 #include "remote.h"
+#include "refs.h"
 
 static const char * const ls_remote_usage[] = {
 	N_("git ls-remote [--heads] [--tags] [--refs] [--upload-pack=<exec>]\n"
@@ -31,6 +32,16 @@ static int tail_match(const char **pattern, const char *path)
 	return 0;
 }
 
+static int has_ref_locally(const struct ref *ref)
+{
+	unsigned char sha1[20];
+
+	if (!resolve_ref_unsafe(ref->name, RESOLVE_REF_READING, sha1, NULL))
+		return 0;
+
+	return !hashcmp(ref->old_oid.hash, sha1);
+}
+
 int cmd_ls_remote(int argc, const char **argv, const char *prefix)
 {
 	const char *dest = NULL;
@@ -39,6 +50,7 @@ int cmd_ls_remote(int argc, const char **argv, const char *prefix)
 	int quiet = 0;
 	int status = 0;
 	int show_symref_target = 0;
+	int diff = 0;
 	const char *uploadpack = NULL;
 	const char **pattern = NULL;
 
@@ -62,6 +74,8 @@ int cmd_ls_remote(int argc, const char **argv, const char *prefix)
 			    N_("exit with exit code 2 if no matching refs are found"), 2),
 		OPT_BOOL(0, "symref", &show_symref_target,
 			 N_("show underlying ref in addition to the object pointed by it")),
+		OPT_BOOL(0, "diff", &diff,
+			 N_("show only refs that differ from local refs")),
 		OPT_END()
 	};
 
@@ -98,6 +112,8 @@ int cmd_ls_remote(int argc, const char **argv, const char *prefix)
 	if (transport_disconnect(transport))
 		return 1;
 
+	if (diff)
+		flags |= REF_NORMAL;
 	if (!dest && !quiet)
 		fprintf(stderr, "From %s\n", *remote->url);
 	for ( ; ref; ref = ref->next) {
@@ -105,6 +121,8 @@ int cmd_ls_remote(int argc, const char **argv, const char *prefix)
 			continue;
 		if (!tail_match(pattern, ref->name))
 			continue;
+		if (diff && has_ref_locally(ref))
+			continue;
 		if (show_symref_target && ref->symref)
 			printf("ref: %s\t%s\n", ref->symref, ref->name);
 		printf("%s\t%s\n", oid_to_hex(&ref->old_oid), ref->name);

Re: [PATCH] ls-remote: add "--diff" option to show only refs that differ

[Junio C Hamano]

Linus Torvalds <torvalds@linux-foundation.org> writes:

> My main use of "git ls-remote" tends to be to check what the other end
> has when some pull request goes wrong (they forgot to push, or they used
> the wrong ref name or whatever), and it ends up being hard to see all
> the relevant data from the noise of people just having the same basic
> tags etc from upstream.
>
> So this adds a "--diff" option that shows only the refs that are
> different from the local repository.  So when somebody asks me to pull,
> I can now just trivially look at what they have that isn't already my
> basic branches and tags.

Most downstream folks seem to care about refs/remotes/origin/$branch
and I think in that context "git ls-remote --diff [origin]" that
compares their refs/heads/* and refs/remotes/origin/* would make
sense.  Your has_ref_locally() seems to return true by comparing
their value with the value of the local ref without any the fetch
refspec mapping.

When one contributor asks you to pull refs/heads/master you want to
go and see if it is different from refs/heads/master you have?

> Comments?
>
> +static int has_ref_locally(const struct ref *ref)
> +{
> +	unsigned char sha1[20];
> +
> +	if (!resolve_ref_unsafe(ref->name, RESOLVE_REF_READING, sha1, NULL))
> +		return 0;
> +
> +	return !hashcmp(ref->old_oid.hash, sha1);
> +}
> +
> @@ -105,6 +121,8 @@ int cmd_ls_remote(int argc, const char **argv, const char *prefix)
>  			continue;
>  		if (!tail_match(pattern, ref->name))
>  			continue;
> +		if (diff && has_ref_locally(ref))
> +			continue;
>  		if (show_symref_target && ref->symref)
>  			printf("ref: %s\t%s\n", ref->symref, ref->name);
>  		printf("%s\t%s\n", oid_to_hex(&ref->old_oid), ref->name);

Re: [PATCH] ls-remote: add "--diff" option to show only refs that differ

[Linus Torvalds]

On Thu, Feb 2, 2017 at 12:03 PM, Junio C Hamano <gitster@pobox.com> wrote:
>
> Most downstream folks seem to care about refs/remotes/origin/$branch
> and I think in that context "git ls-remote --diff [origin]" that
> compares their refs/heads/* and refs/remotes/origin/* would make
> sense.

Hmm. Maybe. The main target for noise reduction for me was actually
all the shared tags.

Which doesn't have that issue.

Also, I've never ever used "git ls-remote" on origin. Do people
actually do that? Why would a regular user ever use ls-remote in the
first place?

The only reason I've ever had for using ls-remote is exactly because
the remote is somehow "odd", and the _normal_ flow didn't work, so you
want to start investigating. So by definition (at least for me),
ls-remote is not part of a good normal flow.

So I kind of see where you are coming from, but I don't really see
that as being a normal workflow for me - or really anybody.

What I think *your* use case is would be more for a workflow along the lines of

   # update the remote data
   git fetch [origin]

   # have some way to just see what branches are not the same
   git status --all

or something ("git status" already talks about the status of the
current branch vs the origin branch).

> Your has_ref_locally() seems to return true by comparing
> their value with the value of the local ref without any the fetch
> refspec mapping.

Right. Because I see the use of "ls-remote" being mostly for
maintainer pulls, and the "origin" in many ways would be the other way
around (and you wouldn't even know what the name of said origin would
be locally).

I basically don't see downstream contributor doing ls-remote, it's a
upstream maintainer command.

But that may be a lack of imagination on my part.

> When one contributor asks you to pull refs/heads/master you want to
> go and see if it is different from refs/heads/master you have?

No. What happens is that people ask me to do something like

    git pull ..some-target.. tags/for-linus-3

and the pull fails because there is no such tag. That's when I go "ok,
they screwed up, let's see what they *meant* for me to pull", and I go
"git ls-remote".

In other words, I don't see anybody ever using git ls-remote if they
already know what the remote is. That's why I don't see "origin" to be
an issue - origin is by definition somethinig you trust, and you just
fetch and pull from.

So the only reason I've ever had for using ls-remote is literally "ok,
what the hell is going on at that remote repo".

And then it is generally a bare repository, and it generally does
*not* have remote branches in it at all.  But it *does* generally end
up having all the basic branches and tags (not always, but it's very
common).

Which is why I as a maintainer then want to just weed out anything
that is already my usual branches that everybody downstream already
has.

I agree that it's a specialized case, but I also think it's the _main_
case for ls-remote in the first place (apart from some scripting to
check for updates or whatever).

But maybe more people use ls-remote than I think they do (and in
different ways than what I envision).

                 Linus

Re: [PATCH] ls-remote: add "--diff" option to show only refs that differ

[Junio C Hamano]

Linus Torvalds <torvalds@linux-foundation.org> writes:

> I basically don't see downstream contributor doing ls-remote, it's a
> upstream maintainer command.
>
> But that may be a lack of imagination on my part.

I actually share that perception.  For the "downstream wonders about
the state of the origin" usecase, I would rather recommend "fetch",
either without "-n" (when the downstream does not value the current
state of refs/remotes/*) or with "-n" (when it does for whatever
reason).

>> When one contributor asks you to pull refs/heads/master you want to
>> go and see if it is different from refs/heads/master you have?
>
> No. What happens is that people ask me to do something like
>
>     git pull ..some-target.. tags/for-linus-3
>
> and the pull fails because there is no such tag. That's when I go "ok,
> they screwed up, let's see what they *meant* for me to pull", and I go
> "git ls-remote".

In that context, I fully agree that "--diff --tags" would help.  The
copies of your tags they have there would overwhelm what you are
really looking for in the output from the command.

And if they asked you to pull "for-linus-3" branch, which is buried
in many other branches (perhaps their publishing repository they ask
you to pull from is also serving as their back-up place, and the
local branches they use before coming up with something pull-able
are all there), then "--diff --refs" would still help by culling
tags that originated from you.

> I agree that it's a specialized case, but I also think it's the _main_
> case for ls-remote in the first place (apart from some scripting to
> check for updates or whatever).
>
> But maybe more people use ls-remote than I think they do (and in
> different ways than what I envision).

You and I are not the only folks in the world, but I agree with you
in thinking that "ls-remote" is not something you would use on
'origin' as a downstream contributor or a consumer.  

Another case I can think of that "--diff" would help is when you are
inspecting your own mirror (but that can be seen as a special case
of the "they have copies of yours plus their own", if you think of
your mirror as "them" and the difference is "being stale").

Re: [PATCH] ls-remote: add "--diff" option to show only refs that differ

[Linus Torvalds]

On Thu, Feb 2, 2017 at 1:05 PM, Junio C Hamano <gitster@pobox.com> wrote:
>
> Another case I can think of that "--diff" would help is when you are
> inspecting your own mirror (but that can be seen as a special case
> of the "they have copies of yours plus their own", if you think of
> your mirror as "them" and the difference is "being stale").

Yeah, that's actually what I did for some testing (not having stale
branches, but just to check the expected differences for my upstream
kernel repo with my local pulls that I haven't pushed out yet).

The actual real use-case is something that only happens for me only
very occasionally. I end up sending out "did you forget to push"
emails perhaps a couple of times every release, but every time I do I
will have gone and done a ls-remote on their repo..

                     Linus