how to fetch entire heirarchy of refs from a remote?

how to fetch entire heirarchy of refs from a remote?

[Jacob Keller]

Hi,

Say I have a repository which has a series of refs in a heirarchy such
as a gerrit repository with changes specified something like:

refs/changes/96/55596/3
refs/changes/96/55596/4
refs/changes/96/57496/1
refs/changes/96/57496/2
refs/changes/96/69796/1
refs/changes/96/69796/2
refs/changes/96/71696/1
refs/changes/96/71696/2
refs/changes/97/40197/1
refs/changes/97/40197/2
refs/changes/97/40197/3
refs/changes/97/40197/4
...
refs/changes/97/71697/2
refs/changes/97/71697/3
refs/changes/98/47298/1
refs/changes/98/47298/2
refs/changes/98/47298/3
refs/changes/98/57298/1
refs/changes/98/57298/2
refs/changes/98/73598/1
refs/changes/99/44099/1
refs/changes/99/69299/1


Is it possible to specify a refspec such that all of these will be
fetched? I tried doing

+refs/changes/*:refs/changes/*

but this doesn't work since the * can only be one portion of a refspec

The problem is that I would like to be able to pull all of these
changes in a git mirror so that I can reduce network traffic I use to
the gerrit server, by updating my local copy once and using the local
copy on the other tasks I need. This is necessary since the gerrit
server is (a) far away and (b) closes my connection when I have too
many requests.

But, git clone --mirror and git fetch have failed to pull every ref,
and only end up with heads and tags.

Regards,
Jake

Re: how to fetch entire heirarchy of refs from a remote?

[Junio C Hamano]

Jacob Keller <jacob.keller@gmail.com> writes:

> refs/changes/98/47298/3
> refs/changes/98/57298/1
> refs/changes/98/57298/2
> refs/changes/98/73598/1
> refs/changes/99/44099/1
> refs/changes/99/69299/1
>
>
> Is it possible to specify a refspec such that all of these will be
> fetched? I tried doing
>
> +refs/changes/*:refs/changes/*
>
> but this doesn't work since the * can only be one portion of a refspec

I am not sure if I follow.  With refs/heads/*:refs/remotes/origin/*
(e.g. following git://github.com/gitster/git for example), I am
reasonably sure you would get my refs/heads/jk/notes-dwim-doc and
other topics you authored without the slash between jk and
notes-dwim-doc getting in the way.

Re: how to fetch entire heirarchy of refs from a remote?

[Jacob Keller]

On Thu, Jan 7, 2016 at 3:15 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Jacob Keller <jacob.keller@gmail.com> writes:
>
>> refs/changes/98/47298/3
>> refs/changes/98/57298/1
>> refs/changes/98/57298/2
>> refs/changes/98/73598/1
>> refs/changes/99/44099/1
>> refs/changes/99/69299/1
>>
>>
>> Is it possible to specify a refspec such that all of these will be
>> fetched? I tried doing
>>
>> +refs/changes/*:refs/changes/*
>>
>> but this doesn't work since the * can only be one portion of a refspec
>
> I am not sure if I follow.  With refs/heads/*:refs/remotes/origin/*
> (e.g. following git://github.com/gitster/git for example), I am
> reasonably sure you would get my refs/heads/jk/notes-dwim-doc and
> other topics you authored without the slash between jk and
> notes-dwim-doc getting in the way.
>
>

That appears to be correct, but.. why doesn't:

git fetch origin +refs/changes/*:refs/changes/*

seem to anything at all?

Ideally this owuld fetch all the remote refs in refs/changes and stick
them into my refs, but it appears to do nothing at all.

Regards,
Jake

Re: how to fetch entire heirarchy of refs from a remote?

[Jacob Keller]

On Thu, Jan 7, 2016 at 3:15 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Jacob Keller <jacob.keller@gmail.com> writes:
>
>> refs/changes/98/47298/3
>> refs/changes/98/57298/1
>> refs/changes/98/57298/2
>> refs/changes/98/73598/1
>> refs/changes/99/44099/1
>> refs/changes/99/69299/1
>>
>>
>> Is it possible to specify a refspec such that all of these will be
>> fetched? I tried doing
>>
>> +refs/changes/*:refs/changes/*
>>
>> but this doesn't work since the * can only be one portion of a refspec
>
> I am not sure if I follow.  With refs/heads/*:refs/remotes/origin/*
> (e.g. following git://github.com/gitster/git for example), I am
> reasonably sure you would get my refs/heads/jk/notes-dwim-doc and
> other topics you authored without the slash between jk and
> notes-dwim-doc getting in the way.
>
>


Infact, even running

git fetch origin +refs/changes/98/73598/1:refs/changes/98/73598/1

does not end up placing anything into refs/changes?

Is it because it happens to not be known by git? That seems rather
short sighted...

Regards,
Jake

Re: how to fetch entire heirarchy of refs from a remote?

[Jacob Keller]

On Thu, Jan 7, 2016 at 3:06 PM, Jacob Keller <jacob.keller@gmail.com> wrote:
>
> But, git clone --mirror and git fetch have failed to pull every ref,
> and only end up with heads and tags.
>
> Regards,
> Jake

It turns out that it *does* pull the refs, but they don't appear to
show up inside the "refs" folder.. is there a reason for this?

ie:

ls .git/refs/changes

shows nothing,but

git ls-remote file:///path/to/repo

does show them? Any particular reason for why this is? I am confused
why refs aren't showing up inside the refs folder...

Regards,
Jake

Re: how to fetch entire heirarchy of refs from a remote?

[Jacob Keller]

On Thu, Jan 7, 2016 at 3:06 PM, Jacob Keller <jacob.keller@gmail.com> wrote:
> Hi,
>
> Say I have a repository which has a series of refs in a heirarchy such
> as a gerrit repository with changes specified something like:
>
> refs/changes/96/55596/3
> refs/changes/96/55596/4
> refs/changes/96/57496/1
> refs/changes/96/57496/2
> refs/changes/96/69796/1
> refs/changes/96/69796/2
> refs/changes/96/71696/1
> refs/changes/96/71696/2
> refs/changes/97/40197/1
> refs/changes/97/40197/2
> refs/changes/97/40197/3
> refs/changes/97/40197/4
> ...
> refs/changes/97/71697/2
> refs/changes/97/71697/3
> refs/changes/98/47298/1
> refs/changes/98/47298/2
> refs/changes/98/47298/3
> refs/changes/98/57298/1
> refs/changes/98/57298/2
> refs/changes/98/73598/1
> refs/changes/99/44099/1
> refs/changes/99/69299/1
>
>
> Is it possible to specify a refspec such that all of these will be
> fetched? I tried doing
>
> +refs/changes/*:refs/changes/*
>
> but this doesn't work since the * can only be one portion of a refspec
>
> The problem is that I would like to be able to pull all of these
> changes in a git mirror so that I can reduce network traffic I use to
> the gerrit server, by updating my local copy once and using the local
> copy on the other tasks I need. This is necessary since the gerrit
> server is (a) far away and (b) closes my connection when I have too
> many requests.
>
> But, git clone --mirror and git fetch have failed to pull every ref,
> and only end up with heads and tags.
>
> Regards,
> Jake

So, sorry for the list spam. Things work correctly, it just is using
packed-refs, which is a feature I wasn't aware of :) Git already did
what I expected but in a way I did not, so I was very confused.

Regards,
Jake

Re: how to fetch entire heirarchy of refs from a remote?

[Junio C Hamano]

Jacob Keller <jacob.keller@gmail.com> writes:

> On Thu, Jan 7, 2016 at 3:06 PM, Jacob Keller <jacob.keller@gmail.com> wrote:
>>
>> But, git clone --mirror and git fetch have failed to pull every ref,
>> and only end up with heads and tags.
>>
>> Regards,
>> Jake
>
> It turns out that it *does* pull the refs, but they don't appear to
> show up inside the "refs" folder.. is there a reason for this?
>
> ie:
>
> ls .git/refs/changes
>
> shows nothing,but
>
> git ls-remote file:///path/to/repo
>
> does show them? Any particular reason for why this is? I am confused
> why refs aren't showing up inside the refs folder...

Does

	git for-each-ref refs/changes/

in your local resulting repository show them?

There is nothing that says "ls -R .git/refs/" is the way to
enumerate available refs (in fact it is a _wrong_ way that is
guaranteed to give you wrong results).

Hint: .git/packed-refs

Re: how to fetch entire heirarchy of refs from a remote?

[Jacob Keller]

On Thu, Jan 7, 2016 at 3:45 PM, Junio C Hamano <gitster@pobox.com> wrote:
>> does show them? Any particular reason for why this is? I am confused
>> why refs aren't showing up inside the refs folder...
>
> Does
>
>         git for-each-ref refs/changes/
>
> in your local resulting repository show them?
>
> There is nothing that says "ls -R .git/refs/" is the way to
> enumerate available refs (in fact it is a _wrong_ way that is
> guaranteed to give you wrong results).
>
> Hint: .git/packed-refs
>

Yep I figured that out :) Now I know better!

Regards,
Jake