sharing between local "work" and "nightly build" git repos

sharing between local "work" and "nightly build" git repos

[David Frech]

I'd like to have the following setup: a ~david/git directory, where I
am free to work on things, and a ~david/nightly-git, where a cron job
is going build and test a nightly "next" branch.

I'd like to share as much as possible between the two repos. My naive
first attempt was to clone the local repo (~david/git) using -l and -s
(which I admit I do not completely understand). This sort of worked,
but one issue is that doing a "git pull" in nightly is going to pull
from the *locally*-cloned repo, not from the main git. Another is that
a checkout in nightly failed with the obscure error:

[david@tashtego ~/git-nightly]% git checkout -b nightly-next next
git checkout: updating paths is incompatible with switching branches/forcing
Did you intend to checkout 'next' which can not be resolved as commit?

I assume this is because too much state is being shared the repos, and
something is unfinished in the "git" directory.

I'd love some pointers on how to:

* share as many objects as possible
* share as little state as possible
* make git pull pull from remote in both repos.

Cheers,

- David
-- 
If I have not seen farther, it is because I have stood in the
footsteps of giants.

Re: sharing between local "work" and "nightly build" git repos

[Junio C Hamano]

"David Frech" <nimblemachines@gmail.com> writes:

> I'd like to share as much as possible between the two repos. My naive
> first attempt was to clone the local repo (~david/git) using -l and -s
> (which I admit I do not completely understand). This sort of worked,
> but one issue is that doing a "git pull" in nightly is going to pull
> from the *locally*-cloned repo, not from the main git. Another is that
> a checkout in nightly failed with the obscure error:
>
> [david@tashtego ~/git-nightly]% git checkout -b nightly-next next
> git checkout: updating paths is incompatible with switching branches/forcing
> Did you intend to checkout 'next' which can not be resolved as commit?
>
> I assume this is because too much state is being shared the repos, and
> something is unfinished in the "git" directory.

You assumed wrong.  "-l and -s" does not have to do anything
with the above symptoms.

git-clone helps people by setting up the new repository to
follow where you cloned from, but there is no rule that you
cannot change it.  Look at .git/config in the nightly repository
and find [remote "origin"] section; update its URL to whichever
repository you would want to track from and you are done.

"git checkout -b nightly-next next" is telling git to:

 - create a new branch nightly-next starting from 'next'
 - check it out

You most likely would want to fork off of "origin/next", not
'next' which probably does not exist in your repository.

If you are willing to redo the nightly repository from scratch,
I would probably recommend using --reference option when
cloning, like this:

 $ git clone --reference ~david/git git://git.kernel.org/... ~/nightly-git
 $ cd ~/nightly-git
 $ git checkout --track -b next origin/next

Then a nightly update would go like this:

 $ cd ~/nightly-git
 $ git pull origin next
 $ make clean
 $ make test || barf

Re: sharing between local "work" and "nightly build" git repos

[David Frech]

On 7/12/07, Junio C Hamano <gitster@pobox.com> wrote:
> "David Frech" <nimblemachines@gmail.com> writes:
> If you are willing to redo the nightly repository from scratch,
> I would probably recommend using --reference option when
> cloning, like this:
>
>  $ git clone --reference ~david/git git://git.kernel.org/... ~/nightly-git
>  $ cd ~/nightly-git
>  $ git checkout --track -b next origin/next
>
> Then a nightly update would go like this:
>
>  $ cd ~/nightly-git
>  $ git pull origin next
>  $ make clean
>  $ make test || barf

This makes sense, including the --track, which I neglected to set.

Could you, for my sake and the list's (if others are as confused) be
clearer about the distinctions among -l, -s, and --reference? Exactly
what they do, and their orthogonality (or lack of) really isn't clear
from reading the man page.

Thanks for your help! I'll set this up and hopefully start running it tonight.

If the build or test barfs, who should get the mail?

BTW, I figured out a sneaky way (using filters in Gmail) to send it
thru my Gmail account, so I won't have any issues with sending via
SMTP from a dynamic IP.

Cheers,

- David
-- 
If I have not seen farther, it is because I have stood in the
footsteps of giants.

Re: sharing between local "work" and "nightly build" git repos

[David Frech]

On 7/12/07, David Frech <nimblemachines@gmail.com> wrote:
> On 7/12/07, Junio C Hamano <gitster@pobox.com> wrote:
> > Then a nightly update would go like this:
> >
> >  $ cd ~/nightly-git
> >  $ git pull origin next
> >  $ make clean
> >  $ make test || barf

One more thing: would it make sense to do "make -k test" so that *all*
failures (if >1) show up?

Cheers,
- David
-- 
If I have not seen farther, it is because I have stood in the
footsteps of giants.

Re: sharing between local "work" and "nightly build" git repos

[Alex Riesen]

David Frech, Fri, Jul 13, 2007 02:33:19 +0200:
> On 7/12/07, David Frech <nimblemachines@gmail.com> wrote:
> >On 7/12/07, Junio C Hamano <gitster@pobox.com> wrote:
> >> Then a nightly update would go like this:
> >>
> >>  $ cd ~/nightly-git
> >>  $ git pull origin next
> >>  $ make clean
> >>  $ make test || barf
> 
> One more thing: would it make sense to do "make -k test" so that *all*
> failures (if >1) show up?

Yes, definitely. You'll find you will see failures which you know are
already fixed, just didn't reach the nightly-build yet.

BTW, do _not_ send the errors to this mailing list. It just too easy
gets out of control. And consider saving the build log somewhere after
doing a _completely_ clean build, i.e. like this:

    #!/bin/bash
    cd ~/nightly-git || exit
    echo 'Subject: Nightly build git next ' $(date) >/tmp/mail.msg
    echo >>/tmp/mail.msg
    {
	rm -rf * # it does not remove .git
	git reset --hard
	git pull origin next || exit 1
	make || exit 1
	make -k test|| exit 1
    } &>> /tmp/mail.msg

    $ nightly-build || sendmail local-user@localhost < /tmp/mail.msg

It is simplier to find out what went wrong if you know what state did
you have before doing the test. And the state after rm -rf * is very
simple to predict.

Re: sharing between local "work" and "nightly build" git repos

[Junio C Hamano]

"David Frech" <nimblemachines@gmail.com> writes:

> Could you, for my sake and the list's (if others are as confused) be
> clearer about the distinctions among -l, -s, and --reference? Exactly
> what they do, and their orthogonality (or lack of) really isn't clear
> from reading the man page.

Clone "-l" makes the new repository share the initial set of
objects from the original by hardlinking each and every file
under .git/objects/.  After "-l" clone, two repositories are
truly independent, except for the fact that the new one
obviously treats the old one as its "origin".

There are a few implications:

 - New objects created in the new repository is not visible in
   the original repository and vice versa.  If you do
   development in the new repository, and push the results back
   to the original, the push will involve object transfer.  If
   your original one is updated (perhaps with git-pull from the
   upstream), you would need to update the clone separately.

 - You can safely rewind the refs in the original repository and
   run git-prune there without worrying about corrupting the
   cloned repository.

Clone "-s" does not make hardlinked copy like "-l" makes.  It
creates an entry in .git/objects/info/alternates file in the new
repository, that instructs git that any object missing in the
new repository should be looked up in the cloned repository.
New objects created in the original will be shared with the
cloned repository because of this, but the sharing does not
happen in the other direction.  Also if you run "rebase" (or
"reset --hard" to rewind the tip of branches) in the original
repository and then run git-gc, it would cause some objects to
become unneeded in the original repository and they can get
discarded.  If your cloned repository depends in them, you will
be in trouble.  So you would need to be a bit careful when you
use "-s" --- the rule of thumb is not to rewind the branches, or
if you do, make sure the cloned ones are kept in sync
(i.e. rewind the commits lost in the original the same way in
the clone).

The "--reference" option uses the same mechanism as "-s" uses to
share objects from a local repository.  It is useful if you have
a repository of a different project that is related to the
project you are cloning from outside.  For example, if you have
a copy of LInus's Linux 2.6 kernel repository locally, and if
you want to clone 2.6.x.y maintenance "stable" repository, you
know the two repositories are alike and share a lot of objects,
so you would:

    $ git clone --reference ../linus-2.6/ git://.../linux-2.6.x.y/ stable.git

To this option, because it uses the same "alternate object
store" mechanism as "-s" option does, the same "be careful
rewinding the branches in the original" caveat applies.  Because
Linus never rewind his branches, the above clone is safe.

For the purpose of "keep rebuilding 'next' from git.git
repository", because my 'next' will never rewind by social
convention, you do not have to worry about the "original
repository being rewound and losing objects" problem, either, as
long as 'next' is the branch you rebuild in the "nightly"
repository.