mirroring and development with three levels of repositories?

mirroring and development with three levels of repositories?

[John Clemens]

Apologies in advance for the long question.  I'm obviously doing something
wrong, but I'm not sure what and searching hasn't provided me with anything.
If there is a git-users mailing list this would be more appropriate for,
please tell me.

Short question:
It it possible for a bare repository to have both local (company-wide) and
remote (mirroring upstream) branches, as well as serving those branches to our
devs? If so, how?

Long question:

We're working on a project that's a collaboration between several companies.
The full tree for everyone is stored on a central server that, for firewall
reasons, only one person can access in the company.  So, that person creates
and updates nightly a local mirror of the central server.  We then have a few
branches of our own that we do development on inside the company, and
periodically merge those branches into the main ones and push them upstream.

In mercurial, we do the branching by cloning the local mirror on our internal
server, having the devs push and pull from that repo, and then pushing those
changes to our local mirror, which gets pushed to the central server.

Now the central server has some things using git, and the git repo has
6 branches within it, as opposed to individual repos.  The setup looks
like this:

+----------+
| upstream |  <--- contains tree with 6 branches
+----------+
    |
------------- <---- firewall
    |
+--------------+
| Local Mirror | <--- "git clone --mirror" from upstream.
+--------------+
    |
    |---------|
  +-----+     +-----+
  | dev |     | dev |
  +-----+     +-----+

I would like to create two company-wide branches here that remain local,
one that branches from upstream's HEAD, and one that branches from one
of the existing branches on upstream (call it branch-a). I then want the devs
to be able to git clone our mirror, and then switch to our company-local branch
and hack away, putting thier changes on the company-wide server.

I thought I'd do this in git by the following:

(on client/dev machine):
$ git clone http://local-server/local-mirror.git

For later reference:
$ git remote show origin
* remote origin
  Fetch URL: http://local-server/local-mirror.git
  Push  URL: http://local-server/local-mirror.git
  HEAD branch: xxx/stable
  Remote branches:
    feature             tracked
    branch-a            tracked
    xxx/master          tracked
    xxx/feature1        tracked
    xxx/feature2        tracked
    xxx/stable          tracked
  Local branch configured for 'git pull':
    xxx/stable merges with remote xxx/stable
  Local ref configured for 'git push':
    xxx/stable pushes to xxx/stable (up to date)

Then, we create a new branch and push it to the server:

$ git branch company-name/test1
$ git push origin company-name/test1

So far, so good, gitweb shows the new beanch in the mirror, pointing to
HEAD.  Now lets create and push the other new company-wide branch:

$ git checkout -b company-name/branch-a origin/branch-a
$ git push origin company-name/branch-a

Again, all seems well.  gitweb shows the new company-name/* branches on the
local server.

However, when you do a new clone of the local mirror, it now fails:

$ git clone http://local-server/local-mirror.git
Initialized empty Git repository in /home/clemej/git/local-mirror/.git/
warning: remote HEAD refers to nonexistent ref, unable to checkout.

$ git branch -a
remotes/origin/company-name/test1
remotes/origin/company-name/branch-a

Note the distinct absense of all the other branches in the repository.

$ git remote show origin
* remote origin
  Fetch URL: http://local-server/local-mirror.git
  Push  URL: http://lodql-server/local-mirror.git
  HEAD branch: (unknown)
  Remote branches:
    company-name/test1       tracked
    company-name/branch-a    tracked

Jumping over to the server, we see:

$ cd local-mirror.git/
$ git remote show origin
* remote origin
  Fetch URL: http://upstream/tree.git
  Push  URL: http://upstream/tree.git
  HEAD branch: xxx/stable
  Remote branches:
    company-name/test1       stale (use 'git remote prune' to remove)
    company-name/branch-a    stale (use 'git remote prune' to remove)
    feature             tracked
    branch-a            tracked
    xxx/master          tracked
    xxx/feature1        tracked
    xxx/feature2        tracked
    xxx/stable          tracked
  Local refs will be mirrored by 'git push'

So, I'm obviously not thinking about this the right way.  It it possible
for a bare repository to have both local (company-wide) and remote (mirroring
upstream) branches, as well as serving those branches to our devs?  If so, how?

Thanks for your time,
john.c

-- 
John Clemens <clemej@gmail.com>

Re: mirroring and development with three levels of repositories?

[Enrico Weigelt]

* John Clemens <clemej@gmail.com> wrote:

> Short question:
> It it possible for a bare repository to have both local (company-wide) and
> remote (mirroring upstream) branches, as well as serving those branches to our
> devs? If so, how?

Yes. Just set up the proper remotes in the config files (same as w/
non-bare repos). The main problem for you might be that the hub-repo's
remotes wont show up as remotes in it's clones - that's because per
default only remote's refs/heads/* namespace is mapped into local's
refs/remotes/*.

Most convenient would probably doing this in the local hub repo.
Add a proper fetch statement in the upstream remote's config section.
Suppose the upstream's remote name is "origin":


[remote "origin"]
    url = ...
    fetch = +refs/heads/*:refs/heads/origin/*
    fetch = +refs/tags/*:refs/tags/origin/*

This will make the upstream's heads and tags as they were pushed
to the hub repo directly, but with the "origin/" prefix.

> $ git clone http://local-server/local-mirror.git
> Initialized empty Git repository in /home/clemej/git/local-mirror/.git/
> warning: remote HEAD refers to nonexistent ref, unable to checkout.

What does the refs/HEAD file tell in the local mirror repo ?
 

cu
-- 
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 phone:  +49 36207 519931  email: weigelt@metux.de
 mobile: +49 151 27565287  icq:   210169427         skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

Re: mirroring and development with three levels of repositories?

[John Clemens]

On Tue, Sep 28, 2010 at 1:37 PM, Enrico Weigelt <weigelt@metux.de> wrote:
> * John Clemens <clemej@gmail.com> wrote:
>
>> Short question:
>> It it possible for a bare repository to have both local (company-wide) and
>> remote (mirroring upstream) branches, as well as serving those branches to our
>> devs? If so, how?
>
> Yes. Just set up the proper remotes in the config files (same as w/
> non-bare repos). The main problem for you might be that the hub-repo's
> remotes wont show up as remotes in it's clones - that's because per
> default only remote's refs/heads/* namespace is mapped into local's
> refs/remotes/*.
>
> Most convenient would probably doing this in the local hub repo.
> Add a proper fetch statement in the upstream remote's config section.
> Suppose the upstream's remote name is "origin":
>
>
> [remote "origin"]
>    url = ...
>    fetch = +refs/heads/*:refs/heads/origin/*
>    fetch = +refs/tags/*:refs/tags/origin/*

Hmm.. I do actually see the branches on dev machine.  When I clone from our
local mirror, everything works fine UNTIL I create a new branch and push it
to our local mirror.  From that moment on, all new clones fail (more precisely,
they do download all the blobs, but the refs only point to the newly created
branches, and HEAD is messed up.

config on the local mirror is this:

[remote "origin"]
        fetch = +refs/*:refs/*
        mirror = true
        url = xxx

presumably, fetch is recursive, so this should catch everything from upstream?

So, the problem appears to be that creating a new branch in the local mirror
repo (level 2 in the diagram), messes up all the upstream refs, even though
they still exist in packed-refs.  Any new clones clone to the dev machine
(level 3) only see the new branches, and not any of the other in the
main mirror.

> This will make the upstream's heads and tags as they were pushed
> to the hub repo directly, but with the "origin/" prefix.
>
>> $ git clone http://local-server/local-mirror.git
>> Initialized empty Git repository in /home/clemej/git/local-mirror/.git/
>> warning: remote HEAD refers to nonexistent ref, unable to checkout.
>
> What does the refs/HEAD file tell in the local mirror repo ?

On the local mirror, HEAD is:
ref: refs/heads/xxx/stable

On any new clone, HEAD still points to refs/heads/master (presumably
because the checkout failed).

Thanks,
john.c

-- 
John Clemens <clemej@gmail.com>

Re: mirroring and development with three levels of repositories?

[Enrico Weigelt]

* John Clemens <clemej@gmail.com> wrote:
> > [remote "origin"]
> >    url = ...
> >    fetch = +refs/heads/*:refs/heads/origin/*
> >    fetch = +refs/tags/*:refs/tags/origin/*
> 
> Hmm.. I do actually see the branches on dev machine.  When I clone from our
> local mirror, everything works fine UNTIL I create a new branch and push it
> to our local mirror.  From that moment on, all new clones fail (more precisely,
> they do download all the blobs, but the refs only point to the newly created
> branches, and HEAD is messed up.
> 
> config on the local mirror is this:
> 
> [remote "origin"]
>         fetch = +refs/*:refs/*
>         mirror = true
>         url = xxx

Ah, maybe it corrupts refs/HEAD somehow ? What does it point to ?

Perhaps you better don't fetch the whole refs/* but refs/heads/* and
refs/tags/* namespaces separately (as described above).

> On the local mirror, HEAD is:
> ref: refs/heads/xxx/stable

Does that ref exist there ? 
What does it tell on the upstream (big central) repo ?


cu
-- 
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 phone:  +49 36207 519931  email: weigelt@metux.de
 mobile: +49 151 27565287  icq:   210169427         skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

Re: mirroring and development with three levels of repositories?

[John Clemens]

Wait.  I just set up some dummy repositories in a working directory
that mirror what we're trying to do, and it all worked.  I created a
dummy "upstream" repo with some branches, then a local mirror repo
with 'git clone --mirror'.  I'm able to check out from the local
mirror, add "local" branches and push them to the mirror repo, and
able to then commit to those "local" branches, no problem.

Since it all seems to work locally,  I suspect a problem with our
webdav git server.  I'll check that out tomorrow.  Thanks for the help
and sorry for the noise.

john.c

On Tue, Sep 28, 2010 at 2:41 PM, Enrico Weigelt <weigelt@metux.de> wrote:
> * John Clemens <clemej@gmail.com> wrote:
>> > [remote "origin"]
>> >    url = ...
>> >    fetch = +refs/heads/*:refs/heads/origin/*
>> >    fetch = +refs/tags/*:refs/tags/origin/*
>>
>> Hmm.. I do actually see the branches on dev machine.  When I clone from our
>> local mirror, everything works fine UNTIL I create a new branch and push it
>> to our local mirror.  From that moment on, all new clones fail (more precisely,
>> they do download all the blobs, but the refs only point to the newly created
>> branches, and HEAD is messed up.
>>
>> config on the local mirror is this:
>>
>> [remote "origin"]
>>         fetch = +refs/*:refs/*
>>         mirror = true
>>         url = xxx
>
> Ah, maybe it corrupts refs/HEAD somehow ? What does it point to ?
>
> Perhaps you better don't fetch the whole refs/* but refs/heads/* and
> refs/tags/* namespaces separately (as described above).
>
>> On the local mirror, HEAD is:
>> ref: refs/heads/xxx/stable
>
> Does that ref exist there ?
> What does it tell on the upstream (big central) repo ?
>
>
> cu
> --
> ----------------------------------------------------------------------
>  Enrico Weigelt, metux IT service -- http://www.metux.de/
>
>  phone:  +49 36207 519931  email: weigelt@metux.de
>  mobile: +49 151 27565287  icq:   210169427         skype: nekrad666
> ----------------------------------------------------------------------
>  Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
> ----------------------------------------------------------------------
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>



-- 
John Clemens <clemej@gmail.com>