Determining whether you have a commit locally, in a partial clone?

Determining whether you have a commit locally, in a partial clone?

[Tao Klerks]

Hi folks,

I "discovered" today that when you're in a partial clone, naive tests
to check for whether you have a commit locally no longer work - they
fetch the commit on-demand:

git cat-file -t SOME_HASH_NOT_IN_REFSPEC
git rev-list SOME_HASH_NOT_IN_REFSPEC

I didn't realize this until today: even commits can be "filtered out"
by partial clone, so any reference to a commit that is not found
locally must be resolved transparently via jit-fetch.

I'm optimizing some stuff for users, so I need to know whether a given
commit exists locally or not... but I can't seem to figure out how!

I tried using "git rev-list"'s "--exclude-promisor-objects" option,
but I guess I don't understand what that's supposed to do. In my case
it just made a simple check like "git rev-list
--exclude-promisor-objects SOME_HASH_NOT_IN_REFSPEC" take forever (10
mins and counting).

I confirmed that removing (commenting out) the
"remote.origin.promisor" and "remote.origin.partialclonefilter" config
keys achieves my objective, but I can't figure out how to do it
safely; "-c remote.origin.promisor=false -c
remote.origin.partialclonefilter=" does *not* seem to work. The
existence of a "remote.origin.partialclonefilter" value, even if it is
empty, appears to override the "remote.origin.promisor=false" setting.

As far as I can tell, config values cannot be unset with "-c" - in
fact I see that credential.helper was granted special support for
empty string as a way of signalling "no credential helper" by Jeff
King in 2016.

So I guess I have two questions:
* Is there any way to run a single git command in a "don't use
promisors" context?
* Is the fact that "-c remote.origin.partialclonefilter=" doesn't work
for temporarily unsetting the filter a bug/issue to be resolved?

Thanks,
Tao

Re: Determining whether you have a commit locally, in a partial clone?

[Tao Klerks]

On Tue, Jun 20, 2023 at 1:26 PM Tao Klerks <tao@klerks.biz> wrote:
>
> * Is there any way to run a single git command in a "don't use
> promisors" context?

My apologies for the self-reply, I did find a workaround here:

git -c remote.origin.url log SOME_HASH_NOT_IN_REFSPEC

I don't understand what's happening here at all, because setting
"remote.origin.url" to "True" in this way works to prevent the
normally configured URL from kicking in, but setting a different value
(or empty value) does not. The following will end up fetching from the
originally-configured URL, completely ignoring the passed-in config:

git -c remote.origin.url=BADURL log SOME_HASH_NOT_IN_REFSPEC

I'm still interested to know whether there's a less-hacky way of
determining "do I have this commit locally?", of course, but fwiw I am
unblocked.

Re: Determining whether you have a commit locally, in a partial clone?

[Junio C Hamano]

Tao Klerks <tao@klerks.biz> writes:

> My apologies for the self-reply, I did find a workaround here:
>
> git -c remote.origin.url log SOME_HASH_NOT_IN_REFSPEC
>
> I don't understand what's happening here at all, because setting
> "remote.origin.url" to "True" in this way works to prevent the
> normally configured URL from kicking in, ...

Interesting.  This happens inside remote.c:handle_config() where
git_config_string() is used to reject a non-string value given to
the "remote.<name>.url" variable and abort the process to read from
the configuration file by returning -1 and causes whoever uses that
configuration value to die.  As the command line configuration is
read first, aborting the configuration reader early would mean that
the configured values would not even be read.

I am not sure why this does not cause the entire thing to die,
though.  It requires further digging, for which I do not have time
for right now..

> ... but setting a different value
> (or empty value) does not.

Most likely, this is because you are giving a syntactically correct
value so the usual "last one wins" logic kicks in.

In hindsight, I think (1) the first one should probably fail the
"git log" process (not just the lazy fetch subprocess), and (2)
there should be an explicit way, e.g. giving an empty string, to
"clear" the list of .url accumulated so far.

(2) may look something silly like this:

 remote.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git c/remote.c w/remote.c
index 0764fca0db..ecc146856a 100644
--- c/remote.c
+++ w/remote.c
@@ -64,12 +64,22 @@ static const char *alias_url(const char *url, struct rewrites *r)
 
 static void add_url(struct remote *remote, const char *url)
 {
+	if (!*url) {
+		remote->url_nr = 0;
+		return;
+	}
+
 	ALLOC_GROW(remote->url, remote->url_nr + 1, remote->url_alloc);
 	remote->url[remote->url_nr++] = url;
 }
 
 static void add_pushurl(struct remote *remote, const char *pushurl)
 {
+	if (!*pushurl) {
+		remote->pushurl_nr = 0;
+		return;
+	}
+
 	ALLOC_GROW(remote->pushurl, remote->pushurl_nr + 1, remote->pushurl_alloc);
 	remote->pushurl[remote->pushurl_nr++] = pushurl;
 }

Re: Determining whether you have a commit locally, in a partial clone?

[Tao Klerks]

On Tue, Jun 20, 2023 at 8:31 PM Junio C Hamano <gitster@pobox.com> wrote:
>
> Tao Klerks <tao@klerks.biz> writes:
>
> > My apologies for the self-reply, I did find a workaround here:
> >
> > git -c remote.origin.url log SOME_HASH_NOT_IN_REFSPEC
> >
> > I don't understand what's happening here at all, because setting
> > "remote.origin.url" to "True" in this way works to prevent the
> > normally configured URL from kicking in, ...
>
> Interesting.  This happens inside remote.c:handle_config() where
> git_config_string() is used to reject a non-string value given to
> the "remote.<name>.url" variable and abort the process to read from
> the configuration file by returning -1 and causes whoever uses that
> configuration value to die.  As the command line configuration is
> read first, aborting the configuration reader early would mean that
> the configured values would not even be read.
>
> I am not sure why this does not cause the entire thing to die,
> though.  It requires further digging, for which I do not have time
> for right now..

I'm reasonably sure this is because the process that reads this config
value is a subprocess, and that subprocess dying was actually
*helping* me in this case.

>
> > ... but setting a different value
> > (or empty value) does not.
>
> Most likely, this is because you are giving a syntactically correct
> value so the usual "last one wins" logic kicks in.
>

I don't understand - surely what's provided "-c" should beat what's in
the repo's .config? "Last one wins" is exactly what I would expect,
but don't seem to be seeing...

> In hindsight, I think (1) the first one should probably fail the
> "git log" process (not just the lazy fetch subprocess), and (2)
> there should be an explicit way, e.g. giving an empty string, to
> "clear" the list of .url accumulated so far.
>
> (2) may look something silly like this:
>
>  remote.c | 10 ++++++++++
>  1 file changed, 10 insertions(+)
>
> diff --git c/remote.c w/remote.c
> index 0764fca0db..ecc146856a 100644
> --- c/remote.c
> +++ w/remote.c
> @@ -64,12 +64,22 @@ static const char *alias_url(const char *url, struct rewrites *r)
>
>  static void add_url(struct remote *remote, const char *url)
>  {
> +       if (!*url) {
> +               remote->url_nr = 0;
> +               return;
> +       }
> +
>         ALLOC_GROW(remote->url, remote->url_nr + 1, remote->url_alloc);
>         remote->url[remote->url_nr++] = url;
>  }
>
>  static void add_pushurl(struct remote *remote, const char *pushurl)
>  {
> +       if (!*pushurl) {
> +               remote->pushurl_nr = 0;
> +               return;
> +       }
> +
>         ALLOC_GROW(remote->pushurl, remote->pushurl_nr + 1, remote->pushurl_alloc);
>         remote->pushurl[remote->pushurl_nr++] = pushurl;
>  }
>


Thank you, I'll play with this when I get the chance.

Re: Determining whether you have a commit locally, in a partial clone?

[Jeff King]

On Tue, Jun 20, 2023 at 11:31:01AM -0700, Junio C Hamano wrote:

> In hindsight, I think (1) the first one should probably fail the
> "git log" process (not just the lazy fetch subprocess), and (2)
> there should be an explicit way, e.g. giving an empty string, to
> "clear" the list of .url accumulated so far.

Agreed. I think in general that any "list" config that is not doing
last-one-wins should have let a blank entry reset the list.

> (2) may look something silly like this:

I know you said "silly", but in case anybody wants to follow up on
this...

> diff --git c/remote.c w/remote.c
> index 0764fca0db..ecc146856a 100644
> --- c/remote.c
> +++ w/remote.c
> @@ -64,12 +64,22 @@ static const char *alias_url(const char *url, struct rewrites *r)
>  
>  static void add_url(struct remote *remote, const char *url)
>  {
> +	if (!*url) {
> +		remote->url_nr = 0;
> +		return;
> +	}

...it probably needs to free() the entries from 0..url_nr, and then also
free the unused empty-string "url".

But I think there's some questionable memory-ownership stuff going on
here. The main config caller passes in a newly allocated string, handing
over ownership to this list (despite the "const" parameter!). But in
add_url_alias(), we take a single allocated "url" string and put it in
both the "url" and "pushurl" lists. The former goes through alias_url()
which sometimes allocates a new string and sometimes does not.

Probably both of these would be better off as strvecs or string_lists.

-Peff

Re: Determining whether you have a commit locally, in a partial clone?

[Tao Klerks]

On Tue, Jun 20, 2023 at 2:04 PM Tao Klerks <tao@klerks.biz> wrote:
>
> On Tue, Jun 20, 2023 at 1:26 PM Tao Klerks <tao@klerks.biz> wrote:
> >
> > * Is there any way to run a single git command in a "don't use
> > promisors" context?
>
> My apologies for the self-reply, I did find a workaround here:
>
> git -c remote.origin.url log SOME_HASH_NOT_IN_REFSPEC
>

Another self-reply unfortunately: This workaround stops working in git 2.39 :(

I haven't understood exactly what's going on, but I guess the fetch
failure was explicitly curtailing *something* extremely expensive that
gets to go forward anyway as of 2.39.

I imagine it's related to the changelog entry "Remove error detection
from a function that fetches from promisor remotes":
https://github.com/gitgitgadget/git/blob/79bdd48716a4c455bdc8ffd91d57a18d5cd55baa/Documentation/RelNotes/2.39.0.txt

I'm back to begging for any hints here: Any idea how I can determine
whether a given commit object exists locally, *without causing it to
be fetched by the act of checking for it?*

Thanks,
Tao

Re: Determining whether you have a commit locally, in a partial clone?

[Jeff King]

On Tue, Jun 20, 2023 at 09:12:24PM +0200, Tao Klerks wrote:

> I'm back to begging for any hints here: Any idea how I can determine
> whether a given commit object exists locally, *without causing it to
> be fetched by the act of checking for it?*

This is not very efficient, but:

  git cat-file --batch-check='%(objectname)' --batch-all-objects --unordered |
  grep $some_sha1

will tell you whether we have the object locally.

I don't work with partial clones often, but it feels like being able to
say:

  git --no-partial-fetch cat-file ...

would be a useful primitive to have. The implementation might start
something like this:

diff --git a/object-file.c b/object-file.c
index 7c1af5c8db..494cdd7706 100644
--- a/object-file.c
+++ b/object-file.c
@@ -1555,6 +1555,14 @@ void disable_obj_read_lock(void)
 
 int fetch_if_missing = 1;
 
+static int allow_lazy_fetch(void)
+{
+	static int ret = -1;
+	if (ret < 0)
+		ret = git_env_bool("GIT_PARTIAL_FETCH", 1);
+	return ret;
+}
+
 static int do_oid_object_info_extended(struct repository *r,
 				       const struct object_id *oid,
 				       struct object_info *oi, unsigned flags)
@@ -1622,6 +1630,7 @@ static int do_oid_object_info_extended(struct repository *r,
 
 		/* Check if it is a missing object */
 		if (fetch_if_missing && repo_has_promisor_remote(r) &&
+		    allow_lazy_fetch() &&
 		    !already_retried &&
 		    !(flags & OBJECT_INFO_SKIP_FETCH_OBJECT)) {
 			promisor_remote_get_direct(r, real, 1);

and then have git.c populate the environment variable, similar to how we
handle --literal-pathspecs, etc.

That fetch_if_missing kind of does the same thing, but it's mostly
controlled by programs themselves which try to handle missing remote
objects specially. It does seem like you might be able to bend it to
your will here, though. I think without any patches that:

  git rev-list --objects --exclude-promisor-objects $oid

will tell you whether we have the object or not (since it turns off
fetch_if_missing, and thus will either succeed, printing nothing, or
bail if the object can't be found). It feels like --missing=error should
function similarly, but it seems to still lazy-fetch (I guess since it's
the default, the point is to just find truly unavailable objects). Using
--missing=print disables the lazy-fetch, but it seems to bail
immediately if you ask it about a missing object (I didn't dig, but my
guess is that --missing is mostly about objects we traverse, not the
initial tips).

-Peff

Re: Determining whether you have a commit locally, in a partial clone?

[Tao Klerks]

On Wed, Jun 21, 2023 at 8:45 AM Jeff King <peff@peff.net> wrote:
>
> On Tue, Jun 20, 2023 at 09:12:24PM +0200, Tao Klerks wrote:
>
> > I'm back to begging for any hints here: Any idea how I can determine
> > whether a given commit object exists locally, *without causing it to
> > be fetched by the act of checking for it?*
>
> This is not very efficient, but:
>
>   git cat-file --batch-check='%(objectname)' --batch-all-objects --unordered |
>   grep $some_sha1
>
> will tell you whether we have the object locally.
>

Thanks so much for your help!

in Windows (msys or git bash) this is still very slow in my repo with
6,500,000 local objects - around 60s - but in linux on the same repo
it's quite a lot faster, at 5s. A large proportion of my users are on
Windows though, so I don't think this will be "good enough" for my
purposes, when I often need to check for the existence of dozens or
even hundreds of commits.

> I don't work with partial clones often, but it feels like being able to
> say:
>
>   git --no-partial-fetch cat-file ...
>
> would be a useful primitive to have.

It feels that way to me, yes!

On the other hand, I find very little demand for it when I search "the
internet" - or I don't know how to search for it.


> The implementation might start
> something like this:
>
> diff --git a/object-file.c b/object-file.c
> index 7c1af5c8db..494cdd7706 100644
> --- a/object-file.c
> +++ b/object-file.c
> @@ -1555,6 +1555,14 @@ void disable_obj_read_lock(void)
>
>  int fetch_if_missing = 1;
>
> +static int allow_lazy_fetch(void)
> +{
> +       static int ret = -1;
> +       if (ret < 0)
> +               ret = git_env_bool("GIT_PARTIAL_FETCH", 1);
> +       return ret;
> +}
> +
>  static int do_oid_object_info_extended(struct repository *r,
>                                        const struct object_id *oid,
>                                        struct object_info *oi, unsigned flags)
> @@ -1622,6 +1630,7 @@ static int do_oid_object_info_extended(struct repository *r,
>
>                 /* Check if it is a missing object */
>                 if (fetch_if_missing && repo_has_promisor_remote(r) &&
> +                   allow_lazy_fetch() &&
>                     !already_retried &&
>                     !(flags & OBJECT_INFO_SKIP_FETCH_OBJECT)) {
>                         promisor_remote_get_direct(r, real, 1);
>
> and then have git.c populate the environment variable, similar to how we
> handle --literal-pathspecs, etc.
>
> That fetch_if_missing kind of does the same thing, but it's mostly
> controlled by programs themselves which try to handle missing remote
> objects specially.

Thanks, I will play with this if I get the chance. That said, I don't
control my users' distributions of Git, so on a purely practical basis
I'm looking for something that will work in git 2.39 to whatever
future version would introduce such a capability. (before 2.39, the
"set remote to False" hack works)

> It does seem like you might be able to bend it to
> your will here, though. I think without any patches that:
>
>   git rev-list --objects --exclude-promisor-objects $oid
>
> will tell you whether we have the object or not (since it turns off
> fetch_if_missing, and thus will either succeed, printing nothing, or
> bail if the object can't be found).

This behaves in a way that I don't understand:

In the repo that I'm working in, this command runs successfully
*without fetching*, but it takes a *very* long time - 300+ seconds -
much longer than even the "inefficient" 'cat-file'-based printing of
all (6.5M) local object ids that you proposed above. I haven't
attempted to understand what's going on in there (besides running with
GIT_TRACE2_PERF, which showed nothing interesting), but the idea that
git would have to work super-hard to find an object by its ID seems
counter to everything I know about it. Would there be value in my
trying to understand & reproduce this in a shareable repo, or is there
already an explanation as to why this command could/should ever do
non-trivial work, even in the largest partial repos?

> It feels like --missing=error should
> function similarly, but it seems to still lazy-fetch (I guess since it's
> the default, the point is to just find truly unavailable objects). Using
> --missing=print disables the lazy-fetch, but it seems to bail
> immediately if you ask it about a missing object (I didn't dig, but my
> guess is that --missing is mostly about objects we traverse, not the
> initial tips).

Woah, "--missing=print" seems to work!!!

The following gives me the commit hash if I have it locally, and an
error otherwise - consistently across linux and windows, git versions
2.41, 2.39, 2.38, and 2.36 - without fetching, and without crazy
CPU-churning:

git rev-list --missing=print -1 $oid

Thank you thank you thank you!

I feel like I should try to work something into the doc about this,
but I'm not sure how to express this: "--missing=error is the default,
but it doesn't actually error out when you're explicitly asking about
a missing commit, it fetches it instead - but --missing=print actually
*does* error out if you explicitly ask about a missing commit" seems
like a strange thing to be saying.

Thanks again for finding me an efficient working strategy here!

Re: Determining whether you have a commit locally, in a partial clone?

[Jeff King]

On Wed, Jun 21, 2023 at 12:10:33PM +0200, Tao Klerks wrote:

> > This is not very efficient, but:
> >
> >   git cat-file --batch-check='%(objectname)' --batch-all-objects --unordered |
> >   grep $some_sha1
> >
> > will tell you whether we have the object locally.
> >
> 
> Thanks so much for your help!
> 
> in Windows (msys or git bash) this is still very slow in my repo with
> 6,500,000 local objects - around 60s - but in linux on the same repo
> it's quite a lot faster, at 5s. A large proportion of my users are on
> Windows though, so I don't think this will be "good enough" for my
> purposes, when I often need to check for the existence of dozens or
> even hundreds of commits.

Yeah, it's just a lot of object names to print, most of which you don't
care about. :)

The more efficient thing would be to open the actual pack .idx files and
look for the names via binary search. I don't think you can convince git
to do that, though I suspect you could write a trivial libgit2 program
that does.

> > I don't work with partial clones often, but it feels like being able to
> > say:
> >
> >   git --no-partial-fetch cat-file ...
> >
> > would be a useful primitive to have.
> 
> It feels that way to me, yes!
> 
> On the other hand, I find very little demand for it when I search "the
> internet" - or I don't know how to search for it.

I think partial clones are still new enough that not many people are
using them heavily. And when they do, not managing the partial state at
a very advanced level; I think tools for pruning locally cached objects
(which you could refetch) is only just being worked on now.

> > It does seem like you might be able to bend it to
> > your will here, though. I think without any patches that:
> >
> >   git rev-list --objects --exclude-promisor-objects $oid
> >
> > will tell you whether we have the object or not (since it turns off
> > fetch_if_missing, and thus will either succeed, printing nothing, or
> > bail if the object can't be found).
> 
> This behaves in a way that I don't understand:
> 
> In the repo that I'm working in, this command runs successfully
> *without fetching*, but it takes a *very* long time - 300+ seconds -
> much longer than even the "inefficient" 'cat-file'-based printing of
> all (6.5M) local object ids that you proposed above. I haven't
> attempted to understand what's going on in there (besides running with
> GIT_TRACE2_PERF, which showed nothing interesting), but the idea that
> git would have to work super-hard to find an object by its ID seems
> counter to everything I know about it. Would there be value in my
> trying to understand & reproduce this in a shareable repo, or is there
> already an explanation as to why this command could/should ever do
> non-trivial work, even in the largest partial repos?

I think it's actually doing the gigantic traversal (and just limiting it
when it sees objects that are not available). You probably want
"--no-walk" at least, but really you don't even want to walk the trees
of any commits you specify (so you'd want to omit "--objects" if you are
asking about a commit, and otherwise include it, which is slightly
awkward).

> > It feels like --missing=error should
> > function similarly, but it seems to still lazy-fetch (I guess since it's
> > the default, the point is to just find truly unavailable objects). Using
> > --missing=print disables the lazy-fetch, but it seems to bail
> > immediately if you ask it about a missing object (I didn't dig, but my
> > guess is that --missing is mostly about objects we traverse, not the
> > initial tips).
> 
> Woah, "--missing=print" seems to work!!!
> 
> The following gives me the commit hash if I have it locally, and an
> error otherwise - consistently across linux and windows, git versions
> 2.41, 2.39, 2.38, and 2.36 - without fetching, and without crazy
> CPU-churning:
> 
> git rev-list --missing=print -1 $oid
> 
> Thank you thank you thank you!

Hmph, I thought I tried that before and it didn't work, but it seems to
work for me now. I guess I was hoping to have it print the missing
object rather than exiting with an error, but if you do one object at a
time then the error is sufficient signal. :)

You might want "--objects" if you're going to ask about non-commits.
Though it might not be necessary. I suspect Git would bail trying to
look up the object in the first place if we don't have it, and if we do
have it then it just becomes a silent noop.

> I feel like I should try to work something into the doc about this,
> but I'm not sure how to express this: "--missing=error is the default,
> but it doesn't actually error out when you're explicitly asking about
> a missing commit, it fetches it instead - but --missing=print actually
> *does* error out if you explicitly ask about a missing commit" seems
> like a strange thing to be saying.

I think we are relying on the side effect that everything except
--missing=error will turn off auto-fetching. I don't know if that's
something we'd want to document. It seems reasonable to me that we might
later change the implementation so that we kick in the --missing
behavior only after parsing the initial list of traversal tips (I mean,
I don't know why we would do that in particular, but it seems like the
kind of thing we'd want to reserve as an implementation detail subject
to change).

I do think in the long run that a big "--do-not-lazy-fetch" flag would
be the right solution to let the user tell us what they want.

> Thanks again for finding me an efficient working strategy here!

I'm glad it worked. I was mostly just thinking out loud. ;)

-Peff