How to check new commit availability without full fetch?

How to check new commit availability without full fetch?

[Leo Razoumov]

Hi List,
I am trying to find a way to check availability of new commits
*before* doing fetch or pull. Unfortunately, neither fetch nor pull
take "--dry-run" option (unlike push)
I am sure I am not the only one with such an itch.

Any help and advice are greatly appreciated.

--Leo--

Re: How to check new commit availability without full fetch?

[Nicolas Pitre]

On Sun, 10 Jan 2010, Leo Razoumov wrote:

> Hi List,
> I am trying to find a way to check availability of new commits
> *before* doing fetch or pull. Unfortunately, neither fetch nor pull
> take "--dry-run" option (unlike push)

But... _Why_ do you want/need to do that?

You could use ls-remote to see what the remote branch is pointing to, 
e.g.:

	git ls-remote origin master

and compare with the local view of that remote branch:

	git show-ref origin/master

And if both SHA1 strings match then there is nothing new to fetch.

> I am sure I am not the only one with such an itch.

Maybe you are. There is very little point knowing that the remote repo 
has new commits if you're not going to fetch them, so I don't understand 
why you need this.


Nicolas

Re: How to check new commit availability without full fetch?

[Junio C Hamano]

Nicolas Pitre <nico@fluxnic.net> writes:

>> I am sure I am not the only one with such an itch.
>
> Maybe you are. There is very little point knowing that the remote repo 
> has new commits if you're not going to fetch them, so I don't understand 
> why you need this.

A feel good factor is in play?  IOW, "I am short of time, so I won't be
able to really afford to 'git pull' and test the result of re-integrating
my changes to what happened on the other end.  If I can learn that there
is nothing happening over there, then I won't have to do anything and know
that I am up to date."

Re: How to check new commit availability without full fetch?

[Nicolas Pitre]

On Sun, 10 Jan 2010, Junio C Hamano wrote:

> Nicolas Pitre <nico@fluxnic.net> writes:
> 
> >> I am sure I am not the only one with such an itch.
> >
> > Maybe you are. There is very little point knowing that the remote repo 
> > has new commits if you're not going to fetch them, so I don't understand 
> > why you need this.
> 
> A feel good factor is in play?  IOW, "I am short of time, so I won't be
> able to really afford to 'git pull' and test the result of re-integrating
> my changes to what happened on the other end.  If I can learn that there
> is nothing happening over there, then I won't have to do anything and know
> that I am up to date."

Just do a fetch then.  If the fetch progress display looks like if it is 
going to take a while then just interrupt it and go home.  If the fetch 
looks trivial then just merge it.  In any case, the "feel good" factor 
can't be that great by only knowing if the remote has changed or not.  

Well maybe if it hasn't changed then you know right away how to feel 
about it (equally with a fetch in that case), and if the remote is 
indeed different then you can't tell whether the changes are trivial or 
not without actually fetching them.


Nicolas

Re: How to check new commit availability without full fetch?

[Leo Razoumov]

On 2010-01-10, Nicolas Pitre <nico@fluxnic.net> wrote:
> On Sun, 10 Jan 2010, Junio C Hamano wrote:
>  >
>  > A feel good factor is in play?  IOW, "I am short of time, so I won't be
>  > able to really afford to 'git pull' and test the result of re-integrating
>  > my changes to what happened on the other end.  If I can learn that there
>  > is nothing happening over there, then I won't have to do anything and know
>  > that I am up to date."
>
>
> Just do a fetch then.  If the fetch progress display looks like if it is
>  going to take a while then just interrupt it and go home.  If the fetch
>  looks trivial then just merge it.  In any case, the "feel good" factor
>  can't be that great by only knowing if the remote has changed or not.
>

Forced interruption is not such a good idea. I would favor a
non-destructive way to monitor availability of remote commits.

BTW, pull and push are in a way symmetric operations. Is there any
deep reason why push supports --dry-run but pull/fetch does not??

--Leo--

Re: How to check new commit availability without full fetch?

[Tay Ray Chuan]

Hi,

On Mon, Jan 11, 2010 at 9:36 AM, Leo Razoumov <slonik.az@gmail.com> wrote:
> On 2010-01-10, Nicolas Pitre <nico@fluxnic.net> wrote:
>> On Sun, 10 Jan 2010, Junio C Hamano wrote:
>>  >
>>  > A feel good factor is in play?  IOW, "I am short of time, so I won't be
>>  > able to really afford to 'git pull' and test the result of re-integrating
>>  > my changes to what happened on the other end.  If I can learn that there
>>  > is nothing happening over there, then I won't have to do anything and know
>>  > that I am up to date."
>>
>>
>> Just do a fetch then.  If the fetch progress display looks like if it is
>>  going to take a while then just interrupt it and go home.  If the fetch
>>  looks trivial then just merge it.  In any case, the "feel good" factor
>>  can't be that great by only knowing if the remote has changed or not.
>>
>
> Forced interruption is not such a good idea. I would favor a
> non-destructive way to monitor availability of remote commits.

By default, when you add a remote (with git remote add), git sets up
the fetch refspec in your config that looks like

  [remote "foo"]
    url = git://foo.com/git/foo.git
    fetch = refs/heads/*:refs/remotes/foo/*

That is to say, branches on the remote repo will be fetched into a
"safe" area, refs/remotes/foo/, away from the branches that you
normally work with in refs/heads/.

However, if you have a different config and you're fetching directly
into refs/heads/, then I can see why you would want to "peek" first
with --dry-run before fetching. Are you doing this?

> BTW, pull and push are in a way symmetric operations. Is there any
> deep reason why push supports --dry-run but pull/fetch does not??

It's more accurate to say that push and fetch are symmetric, because
pull is fetch with merge or rebase tacked on.

Even then, push and fetch are not _that_ symmetric...

-- 
Cheers,
Ray Chuan

Re: How to check new commit availability without full fetch?

[Nicolas Pitre]

On Sun, 10 Jan 2010, Leo Razoumov wrote:

> On 2010-01-10, Nicolas Pitre <nico@fluxnic.net> wrote:
> > On Sun, 10 Jan 2010, Junio C Hamano wrote:
> >  >
> >  > A feel good factor is in play?  IOW, "I am short of time, so I won't be
> >  > able to really afford to 'git pull' and test the result of re-integrating
> >  > my changes to what happened on the other end.  If I can learn that there
> >  > is nothing happening over there, then I won't have to do anything and know
> >  > that I am up to date."
> >
> >
> > Just do a fetch then.  If the fetch progress display looks like if it is
> >  going to take a while then just interrupt it and go home.  If the fetch
> >  looks trivial then just merge it.  In any case, the "feel good" factor
> >  can't be that great by only knowing if the remote has changed or not.
> >
> 
> Forced interruption is not such a good idea. I would favor a
> non-destructive way to monitor availability of remote commits.

You still don't answer my question though.  Again, _why_ do you need to 
know about remote commit availability without fetching them?

> BTW, pull and push are in a way symmetric operations. Is there any
> deep reason why push supports --dry-run but pull/fetch does not??

Pushing involves resource usage on your own machine while 
pulling/fetching involves the remote machine.  Your choice to "waste" 
CPU cycles on your own machine is not the same as having anybody do the 
same on a central server.


Nicolas

Re: How to check new commit availability without full fetch?

[Junio C Hamano]

Tay Ray Chuan <rctay89@gmail.com> writes:

> By default, when you add a remote (with git remote add), git sets up
> the fetch refspec in your config that looks like
>
>   [remote "foo"]
>     url = git://foo.com/git/foo.git
>     fetch = refs/heads/*:refs/remotes/foo/*
>
> That is to say, branches on the remote repo will be fetched into a
> "safe" area, refs/remotes/foo/, away from the branches that you
> normally work with in refs/heads/.
>
> However, if you have a different config and you're fetching directly
> into refs/heads/, then I can see why you would want to "peek" first
> with --dry-run before fetching.

I don't.  Until all the objects are safely transferred, none of the refs
are updated, whether they are directly slurped into local branch namespace
or remote tracking branch namespace.  So no matter what the configuration
is, interrupted transfer, forced or otherwise, is safe.

Re: How to check new commit availability without full fetch?

[Michael Witten]

On Sun, Jan 10, 2010 at 8:08 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Until all the objects are safely transferred, none of the refs
> are updated, whether they are directly slurped into local branch namespace
> or remote tracking branch namespace.  So no matter what the configuration
> is, interrupted transfer, forced or otherwise, is safe.

I suggest adding this kind of information to the fetch/pull (and I
suppose push) documentation.

Re: How to check new commit availability without full fetch?

[Dmitry Potapov]

On Sun, Jan 10, 2010 at 08:36:44PM -0500, Leo Razoumov wrote:
> 
> BTW, pull and push are in a way symmetric operations.

Not really... 'pull' = 'fetch' + 'merge', while 'push' only propagates
changes without any merging. You can say 'fetch' and 'push' are in a way
symmetric operations, but this symmetry is limited due to difference in
usage between local and remote branches.

> Is there any
> deep reason why push supports --dry-run but pull/fetch does not??

I guess it is because no one needs it. 'push' has --dry-run, because
it updates local references in a remote repository. So, you may want
to be sure that you are pushing the right thing. On the other hand,
I see no reason to have --dry-run for 'fetch', because it updates only
remote references, making them to point to the current state of the
corresponding branches. 'fetch' does not change any local branch, so
I see no reason for --dry-run.

What use case do you have in mind that needs --dry-run for 'fetch'?


Dmitry

Re: How to check new commit availability without full fetch?

[Junio C Hamano]

Michael Witten <mfwitten@gmail.com> writes:

> I suggest adding this kind of information to the fetch/pull (and I
> suppose push) documentation.

Sounds good.  Pleaes make it so.

Re: How to check new commit availability without full fetch?

[Robin Rosenberg]

söndagen den 10 januari 2010 12.12.09 skrev  Leo Razoumov:
> Hi List,
> I am trying to find a way to check availability of new commits
> *before* doing fetch or pull. Unfortunately, neither fetch nor pull
> take "--dry-run" option (unlike push)

Fetch has --dry-run. It's a fairly new option. The drawback is that it
still does the fetch, but it does not update the refs. If you re.run it
again it'll be quicker.

A faster option is to use ls-remote, but you'll have to parse the data
yourself and compare with your remote refs to see what refs has changed,
and that will not tell you /what/ the changes are.

-- robin

Re: How to check new commit availability without full fetch?

[Junio C Hamano]

Robin Rosenberg <robin.rosenberg@dewire.com> writes:

> söndagen den 10 januari 2010 12.12.09 skrev  Leo Razoumov:
>> Hi List,
>> I am trying to find a way to check availability of new commits
>> *before* doing fetch or pull. Unfortunately, neither fetch nor pull
>> take "--dry-run" option (unlike push)
>
> Fetch has --dry-run. It's a fairly new option. The drawback is that it
> still does the fetch, but it does not update the refs. If you re.run it
> again it'll be quicker.

Doesn't that worry us if it really is quicker?

If --dry-run doesn't update the refs, why do the objects that were
transferred by them not get asked the next time?  There must be a bug
somewhere, but it is getting late already, so I'll leave it to experts in
the transfer area to figure it out...

Re: How to check new commit availability without full fetch?

[Leo Razoumov]

On 2010-01-10, Nicolas Pitre <nico@fluxnic.net> wrote:
>
> You still don't answer my question though.  Again, _why_ do you need to
>  know about remote commit availability without fetching them?
>

I use git to track almost all my data (code and otherwise) and spread
it between several computers. I end up with several local repos having
the same local branches. It happens once in a while that I fetch into
a given remote/foo from several local foo branches from different
machines and the operation fails. It happens because the commits have
not been yet consistently distributed among the repos. To do the
forensics and figure out who should update whom first I need a quick
and non-destructive way to fetch dry-run.

--Leo--

Re: How to check new commit availability without full fetch?

[Nicolas Pitre]

On Mon, 11 Jan 2010, Leo Razoumov wrote:

> On 2010-01-10, Nicolas Pitre <nico@fluxnic.net> wrote:
> >
> > You still don't answer my question though.  Again, _why_ do you need to
> >  know about remote commit availability without fetching them?
> >
> 
> I use git to track almost all my data (code and otherwise) and spread
> it between several computers. I end up with several local repos having
> the same local branches. It happens once in a while that I fetch into
> a given remote/foo from several local foo branches from different
> machines and the operation fails. It happens because the commits have
> not been yet consistently distributed among the repos. To do the
> forensics and figure out who should update whom first I need a quick
> and non-destructive way to fetch dry-run.

There is probably something awkward about your setup then.

Normally you should have a remote description for any of the remote 
repositories you fetch from.  So if you have, say, remote machine_a with 
repo foo, machine_b with repo bar, and machine_c with repo baz, then 
fetching any of those will _only_ mirror locally the state of those 
remote repositories.  There is no ordering required as there can't be 
any conflicts in the mere fact of mirroring what the other guys have.  
That's what remote tracking branches are for: they follow the state of a 
remote repository and are never altered by local changes.  And you can 
have as many of those as you wish and they will never conflict with each 
other as each remote description is independent. And this is true 
whether or not the remote repository lives on the same machine (that 
would be a remote directory in that case).

And even if most if not all those remotes are actually copies of each 
others, then the first fetch to occur will transfer the new objects 
while fetching the other ones will simply notice that the required 
objects are already available locally and only the ref will be updated.

The ordering comes into play when it is time to _merge_ those remote 
branches into, say, the local master branch.  That's why it is probably 
a good thing to use fetch+merge instead of pull in this case.  But this 
is then a local matter and nothing that depends on the fetch ordering.


Nicolas

Re: How to check new commit availability without full fetch?

[Leo Razoumov]

On 2010-01-11, Nicolas Pitre <nico@fluxnic.net> wrote:
> On Mon, 11 Jan 2010, Leo Razoumov wrote:
>
>  > On 2010-01-10, Nicolas Pitre <nico@fluxnic.net> wrote:
>  > >
>  > > You still don't answer my question though.  Again, _why_ do you need to
>  > >  know about remote commit availability without fetching them?
>  > >
>  >
>  > I use git to track almost all my data (code and otherwise) and spread
>  > it between several computers. I end up with several local repos having
>  > the same local branches. It happens once in a while that I fetch into
>  > a given remote/foo from several local foo branches from different
>  > machines and the operation fails. It happens because the commits have
>  > not been yet consistently distributed among the repos. To do the
>  > forensics and figure out who should update whom first I need a quick
>  > and non-destructive way to fetch dry-run.
>
>
> There is probably something awkward about your setup then.
>
>  Normally you should have a remote description for any of the remote
>  repositories you fetch from.  So if you have, say, remote machine_a with
>  repo foo, machine_b with repo bar, and machine_c with repo baz, then
>  fetching any of those will _only_ mirror locally the state of those
>  remote repositories.  There is no ordering required as there can't be
>  any conflicts in the mere fact of mirroring what the other guys have.
>  That's what remote tracking branches are for: they follow the state of a
>  remote repository and are never altered by local changes.  And you can
>  have as many of those as you wish and they will never conflict with each
>  other as each remote description is independent. And this is true
>  whether or not the remote repository lives on the same machine (that
>  would be a remote directory in that case).
>

Setup might be, indeed, awkward but it handles very diverse tasks.
As I said in my earlier emails different repos fetch into the *same* remote/foo.
So there could be conflicts and using fetch -f could cause loss of data.

Before switching to git I used mercurial for the same purpose and it
has command that are equivalent to fetch --dry-run.

--Leo--

Re: How to check new commit availability without full fetch?

[Nicolas Pitre]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 912 bytes --]

On Mon, 11 Jan 2010, Junio C Hamano wrote:

> Robin Rosenberg <robin.rosenberg@dewire.com> writes:
> 
> > söndagen den 10 januari 2010 12.12.09 skrev  Leo Razoumov:
> >> Hi List,
> >> I am trying to find a way to check availability of new commits
> >> *before* doing fetch or pull. Unfortunately, neither fetch nor pull
> >> take "--dry-run" option (unlike push)
> >
> > Fetch has --dry-run. It's a fairly new option. The drawback is that it
> > still does the fetch, but it does not update the refs. If you re.run it
> > again it'll be quicker.
> 
> Doesn't that worry us if it really is quicker?
> 
> If --dry-run doesn't update the refs, why do the objects that were
> transferred by them not get asked the next time?  There must be a bug
> somewhere, but it is getting late already, so I'll leave it to experts in
> the transfer area to figure it out...

What about builtin-fetch.c:quickfetch() ?


Nicolas

Re: How to check new commit availability without full fetch?

[Junio C Hamano]

Nicolas Pitre <nico@fluxnic.net> writes:

> On Mon, 11 Jan 2010, Junio C Hamano wrote:
>
>> Robin Rosenberg <robin.rosenberg@dewire.com> writes:
>> 
>> > söndagen den 10 januari 2010 12.12.09 skrev  Leo Razoumov:
>> >> Hi List,
>> >> I am trying to find a way to check availability of new commits
>> >> *before* doing fetch or pull. Unfortunately, neither fetch nor pull
>> >> take "--dry-run" option (unlike push)
>> >
>> > Fetch has --dry-run. It's a fairly new option. The drawback is that it
>> > still does the fetch, but it does not update the refs. If you re.run it
>> > again it'll be quicker.
>> 
>> Doesn't that worry us if it really is quicker?
>> 
>> If --dry-run doesn't update the refs, why do the objects that were
>> transferred by them not get asked the next time?  There must be a bug
>> somewhere, but it is getting late already, so I'll leave it to experts in
>> the transfer area to figure it out...
>
> What about builtin-fetch.c:quickfetch() ?

Ahh, you are right.  It walks from objects the remote side told us are at
the tip, and stops at what we know are complete (i.e. reachable from our
tip of objects); immediately after --dry-run slurped objects, the next
fetch will prove everything is locally available and complete before going
over the network.

But either I am very confused or the use of fields from "struct ref" is
unintuitive in this codepath.

Why does it feed ref->old_sha1?  We are feeding _their_ tip commits to:

    rev-list --objects --stdin --not --all

and expecting it to report failure when some of their tip commits lead to
what we don't have yet.  The reason why we have old_sha1[] vs new_sha1[]
is because we want to report what changed from what, and also to protect
us from simultaneous updates by doing compare-and-swap using the value we
read from our refs when we started in old_sha1[], so I would have expected
that ref_map elements would have _their_ commits on the new_sha1[] side,
but apparently that is not what is happening, and it has been this way for
a long time.  The use of old_sha1[] came from 4191c35 (git-fetch: avoid
local fetching from alternate (again), 2007-11-11), so it is a lot more
likely that I am confused than the code is wrong and nobody noticed so
far.

What am I missing?

Re: How to check new commit availability without full fetch?

[Andreas Schwab]

Leo Razoumov <slonik.az@gmail.com> writes:

> Hi List,
> I am trying to find a way to check availability of new commits
> *before* doing fetch or pull. Unfortunately, neither fetch nor pull
> take "--dry-run" option (unlike push)

Try git remote show origin.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: How to check new commit availability without full fetch?

[Nicolas Pitre]

On Mon, 11 Jan 2010, Junio C Hamano wrote:

> Ahh, you are right.  It walks from objects the remote side told us are at
> the tip, and stops at what we know are complete (i.e. reachable from our
> tip of objects); immediately after --dry-run slurped objects, the next
> fetch will prove everything is locally available and complete before going
> over the network.
> 
> But either I am very confused or the use of fields from "struct ref" is
> unintuitive in this codepath.
> 
> Why does it feed ref->old_sha1?  We are feeding _their_ tip commits to:
> 
>     rev-list --objects --stdin --not --all
> 
> and expecting it to report failure when some of their tip commits lead to
> what we don't have yet.  The reason why we have old_sha1[] vs new_sha1[]
> is because we want to report what changed from what, and also to protect
> us from simultaneous updates by doing compare-and-swap using the value we
> read from our refs when we started in old_sha1[], so I would have expected
> that ref_map elements would have _their_ commits on the new_sha1[] side,
> but apparently that is not what is happening, and it has been this way for
> a long time.  The use of old_sha1[] came from 4191c35 (git-fetch: avoid
> local fetching from alternate (again), 2007-11-11), so it is a lot more
> likely that I am confused than the code is wrong and nobody noticed so
> far.

Very confusing indeed.  I first discovered about quickfetch() myself 
when I fixed shallow clone leading to commit 86386829.

If old_sha1[] was our refs then quickfetch() would always succeed and 
we'd never fetch anything.

> What am I missing?

Digging a bit, it looks like get_remote_heads() is storing the remote's 
heads into old_sha1.  And so is performed in get_refs_from_bundle(), and 
in insert_packed_refs() from get_refs_via_rsync(), etc.

Looking at the struct ref definition, we can see:

struct ref {
        struct ref *next;
        unsigned char old_sha1[20];
        unsigned char new_sha1[20];
        [...]
        struct ref *peer_ref; /* when renaming */
        [...]
};



And apparently store_updated_refs() ends up using that peer_ref like 
this:

        for (rm = ref_map; rm; rm = rm->next) {
                struct ref *ref = NULL;

                if (rm->peer_ref) {
                        ref = xcalloc(1, sizeof(*ref) + strlen(rm->peer_ref->name) + 1);
                        strcpy(ref->name, rm->peer_ref->name);
                        hashcpy(ref->old_sha1, rm->peer_ref->old_sha1);
                        hashcpy(ref->new_sha1, rm->old_sha1);
                        ref->force = rm->peer_ref->force;
                }

So.... Doesn't this all look like a total mess of needless (and even 
leaked in this case) allocations and duplications, besides being 
completely unintuitive?  Both hashcpy() above certainly throw my sense 
of logic aside...


Nicolas