keeping remote repo checked out?

keeping remote repo checked out?

[James Cloos]

I am testing git as a replacement for how I was using bk to manage web
sites.  As such, I created a new git repo on the server, populated it
with the old data (ignoring history; there wasn't much relevant on
that site) and cloned that to the laptop.

Editing on the server works well.  I can add, checking and a
subsequent pull on the laptop grabs the changes.

But any edits on the laptop's copy are not checked out when I push
them to the server.

Is it possible to arrange that a push does a checkout on the remote
the same way a pull does on the local?

-JimC
-- 
James H. Cloos, Jr. <cloos@jhcloos.com>

Re: keeping remote repo checked out?

[Junio C Hamano]

James Cloos <cloos@jhcloos.com> writes:

> Is it possible to arrange that a push does a checkout on the remote
> the same way a pull does on the local?

Creative use of hooks/post-update would solve that.

However, you should be very careful if you sometimes edit on
server and sometimes push from other machine to the server on
the same branch on the server.

Re: keeping remote repo checked out?

[Petr Baudis]

Dear diary, on Mon, Nov 28, 2005 at 08:48:26AM CET, I got a letter
where Junio C Hamano <junkio@cox.net> said that...
> James Cloos <cloos@jhcloos.com> writes:
> 
> > Is it possible to arrange that a push does a checkout on the remote
> > the same way a pull does on the local?
> 
> Creative use of hooks/post-update would solve that.
> 
> However, you should be very careful if you sometimes edit on
> server and sometimes push from other machine to the server on
> the same branch on the server.

Why? At worst you will get files with conflict markers on the server,
which isn't that huge problem and just what you have to expect when you
do this kind of thing. I could imagine having an option in the config
file describing whether the repository is "raw" or has the working copy
associated, and in the later case have the updating code in the default
post-update hook.

PS: If someone is bored, it would be very useful to have a manual page
for the git configuration file, containing the list of all options.

-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
VI has two modes: the one in which it beeps and the one in which
it doesn't.

Re: keeping remote repo checked out?

[Junio C Hamano]

Petr Baudis <pasky@suse.cz> writes:

> Dear diary, on Mon, Nov 28, 2005 at 08:48:26AM CET, I got a letter
> where Junio C Hamano <junkio@cox.net> said that...
>> James Cloos <cloos@jhcloos.com> writes:
>> 
>> > Is it possible to arrange that a push does a checkout on the remote
>> > the same way a pull does on the local?
>> 
>> Creative use of hooks/post-update would solve that.
>> 
>> However, you should be very careful if you sometimes edit on
>> server and sometimes push from other machine to the server on
>> the same branch on the server.
>
> Why? At worst you will get files with conflict markers on the server,
> which isn't that huge problem and just what you have to expect when you
> do this kind of thing.

Both of us are right and talking about the same thing.  You are
right that as long as hooks/post-update is done correctly the
one who works on the server should not have to worry.  I am
right that the hooks/post-update for that setup needs to be done
very carefully ;-).

Here are the things whoever is doing the hooks/post-update for
this particular setup needs to think about.

 - is it safe to assume that the guy working on the server
   working tree never switch branches?  otherwise, what to do if
   the working tree has different branch checked out when push
   called post-update?

 - should it allow forced-push that sets HEAD to non descendant
   of the current HEAD?  In a shared repository setup,
   disallowing forced-push is a good discipline.  OTOH, if this
   is primarily used as an installation mechanism to a remote
   hosting site, allowing forced-push may be ok.

 - should it do 'git-checkout', 'git-reset --hard HEAD', or
   'git-pull . branch_to_push_into'?  The former two pretty much
   assumes no development happens on the server repository and
   git push is used primarily as an installation mechanism.  The
   latter is to keep a branch, other than "master" that is
   always checked out on the server machine, and have people
   push into a different branch and merge with it automatically
   when a push happens.  what would you do when a merge conflict
   happens?

On a tangent, the last point brings up an interesting
shared-repo usage pattern.  When you have a shared central
repository like CVS, you could arrange things this way:

 * On the shared repository, prepare one branch per developer
   who is pushing into it, plus "master".

 * The developer pull from "master" and work in her private
   repository.  The changes are pushed into her own branch on
   the shared repository machine.

 * The the shared repository machine tries to merge "master"
   into developer branches when the developer branch head is
   updated.  If it does not trivially resolve, the developer
   branch head is left as the last push by the developer
   (i.e. not recording the merge).  If it does resolve, it is
   checked out into playpen area, built and testsuite run, and
   if all look well, "master" is updated with the result of the
   merge.  A notification is sent to the developer to tell which
   of the above actions happened.

   This does not have to happen in the update hook and you
   probably would not want to because it would be a lenthy
   operation.  The update hook can be used to maintain recent
   push log for a cron job that runs the "merge-test-integrate"
   procedure to decide which branch is interesting.

I am not sure if people would find this useful, though.

Re: keeping remote repo checked out?

[Petr Baudis]

Dear diary, on Mon, Nov 28, 2005 at 08:35:23PM CET, I got a letter
where Junio C Hamano <junkio@cox.net> said that...
> Both of us are right and talking about the same thing.  You are
> right that as long as hooks/post-update is done correctly the
> one who works on the server should not have to worry.  I am
> right that the hooks/post-update for that setup needs to be done
> very carefully ;-).

Aha, then I misunderstood what you wrote before - sorry. Obviously,
you are right. ;-)

> Here are the things whoever is doing the hooks/post-update for
> this particular setup needs to think about.
> 
>  - is it safe to assume that the guy working on the server
>    working tree never switch branches?  otherwise, what to do if
>    the working tree has different branch checked out when push
>    called post-update?

I wouldn't do anything. Working copy reflects the HEAD; if you don't
update HEAD, you needn't touch the working copy.

>  - should it allow forced-push that sets HEAD to non descendant
>    of the current HEAD?  In a shared repository setup,
>    disallowing forced-push is a good discipline.  OTOH, if this
>    is primarily used as an installation mechanism to a remote
>    hosting site, allowing forced-push may be ok.

I would just leave this on the particular head policy.

>  - should it do 'git-checkout', 'git-reset --hard HEAD', or
>    'git-pull . branch_to_push_into'?  The former two pretty much
>    assumes no development happens on the server repository and
>    git push is used primarily as an installation mechanism.

Files should be removed properly, which pretty much excludes the former
two, I think.

>    The latter is to keep a branch, other than "master" that is
>    always checked out on the server machine, and have people
>    push into a different branch and merge with it automatically
>    when a push happens.  what would you do when a merge conflict
>    happens?

This seems weird and overcomplicated, and it seems to mirror that GIT
does not want to deal with trees containing local modifications - which
is fine, but I don't think you should go over huge hoops in the workflow
to adjust to that. Just verify in the pre-update hook that the tree
contains no local modifications, and disallow the push otherwise. Cogito
can then use own update hooks which will enable it to handle local
changes more gracefully (albeit still not ideally).

> On a tangent, the last point brings up an interesting
> shared-repo usage pattern.  When you have a shared central
> repository like CVS, you could arrange things this way:
..snip..
> I am not sure if people would find this useful, though.

It is certainly quite interesting idea. I'm not sure how useful in
practice is it either, though. ;-)

-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
VI has two modes: the one in which it beeps and the one in which
it doesn't.

Re: keeping remote repo checked out?

[Junio C Hamano]

Petr Baudis <pasky@suse.cz> writes:

>> Here are the things whoever is doing the hooks/post-update for
>> this particular setup needs to think about.
>> 
>>  - is it safe to assume that the guy working on the server
>>    working tree never switch branches?  otherwise, what to do if
>>    the working tree has different branch checked out when push
>>    called post-update?
>
> I wouldn't do anything. Working copy reflects the HEAD; if you don't
> update HEAD, you needn't touch the working copy.
>
>>  - should it allow forced-push that sets HEAD to non descendant
>>    of the current HEAD?  In a shared repository setup,
>>    disallowing forced-push is a good discipline.  OTOH, if this
>>    is primarily used as an installation mechanism to a remote
>>    hosting site, allowing forced-push may be ok.
>
> I would just leave this on the particular head policy.

In case it was not obvious, I was not outlining what I would
want as a git-wide policy.  It was meant as an example of things
people need to think about, IOW, what their head policy should
be, if they want to use git in a workflow that magically checks
things out when a push happens.

>>  - should it do 'git-checkout', 'git-reset --hard HEAD', or
>>    'git-pull . branch_to_push_into'?  The former two pretty much
>>    assumes no development happens on the server repository and
>>    git push is used primarily as an installation mechanism.
>
> Files should be removed properly, which pretty much excludes the former
> two, I think.

Unless you screw with the branch head under a checked out tree,
git-checkout would remove files properly, and "git-reset --hard"
would do the right thing even when you update the branch head of
a checked out tree.  The last one is interesting and might be
useful.

Re: keeping remote repo checked out?

[Linus Torvalds]

On Mon, 28 Nov 2005, Petr Baudis wrote:
> 
> >  - should it do 'git-checkout', 'git-reset --hard HEAD', or
> >    'git-pull . branch_to_push_into'?  The former two pretty much
> >    assumes no development happens on the server repository and
> >    git push is used primarily as an installation mechanism.
> 
> Files should be removed properly, which pretty much excludes the former
> two, I think.

There are major locking issues with two people pushing at the same time.

The git logic does the right thing for a non-checked-out branch, but it 
can do the right thing exactly _because_ it's not checked out. It can 
create all the new objects whether the actual push succeeds or not, and it 
can do the final switch-over as a single atomic file operation.

The same is most emphatically _not_ true of "git checkout".

In other words, you need to add your very own locking for the shared 
checked-out tree, and you need to (_within_ that lock) separately remember 
what the _previous_ tree was that was checked out, because you can't just 
rely on the previous head as reported by git, because that will have been 
done ourside your "checked out lock".

And locking really isn't trivial. Especially if you _also_ do this with 
multiple clients touching the same checked-out repo over NFS (which I'd 
seriously suggest you not do).

There's a really good reason why the git core does _not_ do this. It's 
damn complicated.

		Linus

Re: keeping remote repo checked out?

[Daniel Barkalow]

On Mon, 28 Nov 2005, Linus Torvalds wrote:

> On Mon, 28 Nov 2005, Petr Baudis wrote:
> > 
> > >  - should it do 'git-checkout', 'git-reset --hard HEAD', or
> > >    'git-pull . branch_to_push_into'?  The former two pretty much
> > >    assumes no development happens on the server repository and
> > >    git push is used primarily as an installation mechanism.
> > 
> > Files should be removed properly, which pretty much excludes the former
> > two, I think.
> 
> There are major locking issues with two people pushing at the same time.
> 
> The git logic does the right thing for a non-checked-out branch, but it 
> can do the right thing exactly _because_ it's not checked out. It can 
> create all the new objects whether the actual push succeeds or not, and it 
> can do the final switch-over as a single atomic file operation.
> 
> The same is most emphatically _not_ true of "git checkout".
> 
> In other words, you need to add your very own locking for the shared 
> checked-out tree, and you need to (_within_ that lock) separately remember 
> what the _previous_ tree was that was checked out, because you can't just 
> rely on the previous head as reported by git, because that will have been 
> done ourside your "checked out lock".

We really ought to keep track of what the current checked out tree is, 
independant of the head that it's supposed to match. Then it would be safe 
to push to a head that's checked out (it wouldn't update it, but the 
system would understand what's going on).

The rest should be easy from there, just by locking on that file, since 
you probably don't care about the race between checking out the head and 
updating the head. (You would want to have the case for not getting the 
lock loop until it can get the lock, rather than aborting, however, so 
that the last version does get checked out even if checkout wins with push 
but the hook runs before checkout completes.)

	-Daniel
*This .sig left intentionally blank*

Re: keeping remote repo checked out?

[Linus Torvalds]

On Mon, 28 Nov 2005, Daniel Barkalow wrote:
> 
> We really ought to keep track of what the current checked out tree is, 
> independant of the head that it's supposed to match. Then it would be safe 
> to push to a head that's checked out (it wouldn't update it, but the 
> system would understand what's going on).

Well, that's what branches are all about. 

You really have two choices:
 - do it all yourself
 - use git branches

but if you use a special git branch for the checked-out state, then you 
have to realize that nobody can push directly to it.

You'd have a trigger that triggers on a push to _another_ branch, and then 
the trigger basically does an automatic "git pull" from that branch (and 
updates the checked-out state as part of that).

But you _still_ need to do the locking. That's very fundamental. There is 
no locking in a "git push", exactly because it can push in parallel, by 
design. And you _cannot_ check out a tree in parallel, so you have to do 
locking for that.

There's no way you can avoid the locking. You can do shortcuts: for 
example, you could avoid any explicit locking by having the "checkout" 
just happen once an hour, with an automatic "pull" from the changed 
branch. Again, that means that the branch that is checked out can NOT be 
one of the branches that people push to.

		Linus

Re: keeping remote repo checked out?

[Daniel Barkalow]

On Mon, 28 Nov 2005, Linus Torvalds wrote:

> On Mon, 28 Nov 2005, Daniel Barkalow wrote:
> > 
> > We really ought to keep track of what the current checked out tree is, 
> > independant of the head that it's supposed to match. Then it would be safe 
> > to push to a head that's checked out (it wouldn't update it, but the 
> > system would understand what's going on).
> 
> Well, that's what branches are all about. 
> 
> You really have two choices:
>  - do it all yourself
>  - use git branches
> 
> but if you use a special git branch for the checked-out state, then you 
> have to realize that nobody can push directly to it.

The idea is to have something which is private, to match the working tree, 
which is obviously private (because it doesn't even exist from the point 
of view of the database). Of course, you don't want most of the branch 
mechanism, with HEAD pointing to a file, because when you commit, you want 
to update something else, and when you check out, you want to check out 
something else. You want to use a normal git branch, but still have git 
keep track of what it thinks should be in your working tree, regardless of 
what the branch you're on does, so that checkout doesn't get confused if 
somebody's pushed to the branch you're on.

Then, if someone pushes to a branch that's checked out, git doesn't lose 
track of what it put in the working tree, so it can handle the "git 
checkout" that you do afterwards to update the working tree. Making 
everything work okay if multiple "git checkout"s happen at the same time 
is future work, and will definitely involve locking.

	-Daniel
*This .sig left intentionally blank*

Re: keeping remote repo checked out?

[Linus Torvalds]

On Mon, 28 Nov 2005, Daniel Barkalow wrote:
> > 
> > but if you use a special git branch for the checked-out state, then you 
> > have to realize that nobody can push directly to it.
> 
> The idea is to have something which is private, to match the working tree, 
> which is obviously private (because it doesn't even exist from the point 
> of view of the database).

Sure. It's private because it has its own name, and you can make _sure_ 
that nobody tries to push to it by having a trigger that refuses to update 
that branch.

Now, if you want to make it also be _invisible_ (so that when others don't 
clone, they don't see what the checked-out state was when they cloned), 
you'd have to make some new rules to git. We could have rules like "branch 
names that start with '.' aren't listed", which would effectively do 
exactly that..

			Linus

[PATCH] Re: keeping remote repo checked out?

[Daniel Barkalow]

On Mon, 28 Nov 2005, Linus Torvalds wrote:

> On Mon, 28 Nov 2005, Daniel Barkalow wrote:
> > > 
> > > but if you use a special git branch for the checked-out state, then you 
> > > have to realize that nobody can push directly to it.
> > 
> > The idea is to have something which is private, to match the working tree, 
> > which is obviously private (because it doesn't even exist from the point 
> > of view of the database).
> 
> Sure. It's private because it has its own name, and you can make _sure_ 
> that nobody tries to push to it by having a trigger that refuses to update 
> that branch.
> 
> Now, if you want to make it also be _invisible_ (so that when others don't 
> clone, they don't see what the checked-out state was when they cloned), 
> you'd have to make some new rules to git. We could have rules like "branch 
> names that start with '.' aren't listed", which would effectively do 
> exactly that..

I was planning to keep it in .git (instead of .git/refs/...), because it's 
information about the working tree, not properly part of the repository. 
It's like MERGE_HEAD in being about what the user is in the middle of.

I actually have a bit of a patch now, which only updates checkout and 
commit (anything that updates HEAD should also update CHECKED_OUT, and 
some things probably ought to read CHECKED_OUT instead of HEAD), but 
should serve to give an idea of what I mean. Essentially, if somebody's 
messed with your HEAD, than "git checkout" will update to that, and "git 
commit" will give an error message until you do (I added a check so it 
happens before you write a message, but the third argument to 
"git-update-ref HEAD" would also complain).

Obviously, not for application, since it breaks things like "git resolve" 
which are supposed to change what your working tree contains and aren't 
updated to make CHECKED_OUT match.

---

Keep a private copy of the commit whose tree corresponds to the
git-provided state of the working tree, in case somebody changes the
branch we're on behind our backs.

Signed-off-by: Daniel Barkalow <barkalow@iabervon.org>

---

 git-checkout.sh |    5 ++++-
 git-commit.sh   |   10 +++++++++-
 2 files changed, 13 insertions(+), 2 deletions(-)

applies-to: fe523a4df93cce3e5c5b0266b9d3f1cbea009afa
8831e2595f29c90a259586250d5c6d044e9d7c8c
diff --git a/git-checkout.sh b/git-checkout.sh
index 4cf30e2..503b34f 100755
--- a/git-checkout.sh
+++ b/git-checkout.sh
@@ -5,7 +5,8 @@ usage () {
     die "usage: git checkout [-f] [-b <new_branch>] [<branch>] [<paths>...]"
 }
 
-old=$(git-rev-parse HEAD)
+old=$(git-rev-parse --revs-only CHECKED_OUT)
+[ -z $old ] && old=$(git-rev-parse HEAD)
 new=
 force=
 branch=
@@ -117,6 +118,8 @@ else
     git-read-tree -m -u $old $new
 fi
 
+git-update-ref CHECKED_OUT $new
+
 # 
 # Switch the HEAD pointer to the new branch if it we
 # checked out a branch head, and remove any potential
diff --git a/git-commit.sh b/git-commit.sh
index 3d250ec..e73003b 100755
--- a/git-commit.sh
+++ b/git-commit.sh
@@ -108,6 +108,13 @@ t,*)
 esac || exit 1
 git-update-index -q --refresh || exit 1
 
+current=$(git-rev-parse --revs-only CHECKED_OUT)
+if test ! -z "$current" -a "$current" != "$(git-rev-parse HEAD)"
+then
+	echo "Changes have been pushed to this branch since you checked it out"
+	exit 1
+fi
+
 case "$verify" in
 t)
 	if test -x "$GIT_DIR"/hooks/pre-commit
@@ -232,7 +239,8 @@ then
 	tree=$(git-write-tree) &&
 	commit=$(cat "$GIT_DIR"/COMMIT_MSG | git-commit-tree $tree $PARENTS) &&
 	git-update-ref HEAD $commit $current &&
-	rm -f -- "$GIT_DIR/MERGE_HEAD"
+	rm -f -- "$GIT_DIR/MERGE_HEAD" &&
+	git-update-ref CHECKED_OUT $commit
 else
 	echo >&2 "* no commit message?  aborting commit."
 	false
---
0.99.9.GIT

Re: [PATCH] Re: keeping remote repo checked out?

[Linus Torvalds]

On Mon, 28 Nov 2005, Daniel Barkalow wrote:
> 
> I was planning to keep it in .git (instead of .git/refs/...), because it's 
> information about the working tree, not properly part of the repository. 
> It's like MERGE_HEAD in being about what the user is in the middle of.

That's fine.

However, your patch isn't.

Your patch just creates this eternal confusion of "why is HEAD different 
from CHECKED_OUT". 

It's really easy to do what you want to do _without_ any tool changes:

	git-rev-parse HEAD > .git/CHECKED_OUT
	rm .git/HEAD
	ln -s refs/heads/../../CHECKED_OUT .git/HEAD

and you're done. No tool changes necessary.

Now "HEAD" always points to the checked-out thing, but a "git clone" won't 
ever actually clone your CHECKED_OUT branch.

		Linus

Re: [PATCH] Re: keeping remote repo checked out?

[Daniel Barkalow]

On Mon, 28 Nov 2005, Linus Torvalds wrote:

> On Mon, 28 Nov 2005, Daniel Barkalow wrote:
> > 
> > I was planning to keep it in .git (instead of .git/refs/...), because it's 
> > information about the working tree, not properly part of the repository. 
> > It's like MERGE_HEAD in being about what the user is in the middle of.
> 
> That's fine.
> 
> However, your patch isn't.
> 
> Your patch just creates this eternal confusion of "why is HEAD different 
> from CHECKED_OUT". 
> 
> It's really easy to do what you want to do _without_ any tool changes:
> 
> 	git-rev-parse HEAD > .git/CHECKED_OUT
> 	rm .git/HEAD
> 	ln -s refs/heads/../../CHECKED_OUT .git/HEAD
> 
> and you're done. No tool changes necessary.
> 
> Now "HEAD" always points to the checked-out thing, but a "git clone" won't 
> ever actually clone your CHECKED_OUT branch.

But CHECKED_OUT isn't a branch. The whole point of it is that you stay on 
some real branch, so you can actually use the tools to affect the 
repository (i.e., the things under refs/). The reason to keep CHECKED_OUT 
is so that you can tell if the branch you're on changes out from under 
you.

For example, I want to have a repository where I push into the 
"production" branch, and I want to be able to get these changes in the 
working tree that the repository is in. If I have HEAD point to 
refs/heads/production, git doesn't know any more what's in the working 
tree, so it can't do the two-way merge. If I have HEAD point to 
CHECKED_OUT, git doesn't know any more what I want to get, so it can't do 
the two-way merge. That is, what I want to have work is:

  server$ git checkout production
  work$ git push server:production
  server$ git checkout

and this should leave the working tree on server with the changes pushed 
from work. Currently, I have to make sure server *isn't* tracking the 
branch I care about when the push happens, so that it doesn't lose track 
of its state, and then switch to the branch to get the changes, and copy 
the branch to a new branch, and switch to that so the next branch won't 
mess up the state.

HEAD is a symbolic ref to the ref in the repository that is the current 
branch. CHECKED_OUT is a ref to the hash that's in the working tree. 
Usually they match, because usually nothing is changing current branch 
without changing the working tree (or making the current branch match 
changes in the working tree), but the point of the patch is to respond 
sensibly to cases where this stops being true, because those can actually 
be part of desirable usage patterns.

	-Daniel
*This .sig left intentionally blank*

Re: [PATCH] Re: keeping remote repo checked out?

[Junio C Hamano]

Daniel Barkalow <barkalow@iabervon.org> writes:

> the two-way merge. That is, what I want to have work is:
>
>   server$ git checkout production
>   work$ git push server:production
>   server$ git checkout
>
> and this should leave the working tree on server with the changes pushed 
> from work.

I think the approach Linus was suggesting before he suggested
the funny refs/heads/../.. symlink idea (honestly, I think he
was half joking) is cleaner, easier to understand, easier to
implement and generally makes more sense.  Your work cycle would
become like this:

    server$ git checkout production
    work$ git checkout master
    work$ git pull server production ;# merge into work's master
    work$ git push server master:receive_from_work
    server$ git pull . receive_from_work ;# merge into server's production

and you make sure receive_from_work is not checked out on server
(or production is never pushed into) and always do fast forward
and nothing else.

Re: [PATCH] Re: keeping remote repo checked out?

[Daniel Barkalow]

On Mon, 28 Nov 2005, Junio C Hamano wrote:

> Daniel Barkalow <barkalow@iabervon.org> writes:
> 
> > the two-way merge. That is, what I want to have work is:
> >
> >   server$ git checkout production
> >   work$ git push server:production
> >   server$ git checkout
> >
> > and this should leave the working tree on server with the changes pushed 
> > from work.
> 
> I think the approach Linus was suggesting before he suggested
> the funny refs/heads/../.. symlink idea (honestly, I think he
> was half joking) is cleaner, easier to understand, easier to
> implement and generally makes more sense.  Your work cycle would
> become like this:
> 
>     server$ git checkout production
>     work$ git checkout master
>     work$ git pull server production ;# merge into work's master
>     work$ git push server master:receive_from_work
>     server$ git pull . receive_from_work ;# merge into server's production
> 
> and you make sure receive_from_work is not checked out on server
> (or production is never pushed into) and always do fast forward
> and nothing else.

That's what I'm doing currently, actually (with work pushing to 
"production", but server having checked out "deploy"), and I find it 
annoying to have to do the pull each time and have a separate head. It 
also means that, if the stuff on the server is set up as a hook, it'll 
have to do the locking against concurrent changes to the working tree in 
pull, which is much more complicated than checkout.

It's also somewhat non-intuitive, and leaves a surprising way for the user 
to confuse the system in a way that it can't detect.

	-Daniel
*This .sig left intentionally blank*

Re: [PATCH] Re: keeping remote repo checked out?

[Junio C Hamano]

Daniel Barkalow <barkalow@iabervon.org> writes:

>>     server$ git checkout production
>>     work$ git checkout master
>>     work$ git pull server production ;# merge into work's master
>>     work$ git push server master:receive_from_work
>>     server$ git pull . receive_from_work ;# merge into server's production
>> 
>> and you make sure receive_from_work is not checked out on server
>> (or production is never pushed into) and always do fast forward
>> and nothing else.
>
> That's what I'm doing currently, actually (with work pushing to 
> "production", but server having checked out "deploy"), and I find it 
> annoying to have to do the pull each time and have a separate head. It 
> also means that, if the stuff on the server is set up as a hook, it'll 
> have to do the locking against concurrent changes to the working tree in 
> pull, which is much more complicated than checkout.

This depends on what is on the checked-out working tree and how
it is used, but one use pattern I was imagining was actually
much more elaborate.  I would set up two non-naked repositories
on the server; let's call them treeR and treeP (reception and
publish).

Each work machine push into its own reception branch of treeR
repository (so you have N reception branches for N workers).  On
the server machine, a daemon monitors the reception branch heads
(inotify or idle poll loop looking at mtime; the details do not
matter), and run the git pull into "master" of treeR.  My daemon
might want to forbid anything but fast forward for this pull, or
might allow any clean merge.  It compiles and runs testsuites
next.  If anything fails during the above process, the owner of
the reception branch is notified about the failure; I might even
want to forcibly rewind her reception branch in this case to
"master" when this happens.

When everything goes well, the daemon goes to treeP and pulls
from treeR's master into "deploy", which is checked out.  This
procedure is the only thing that touches treeP, so this pull is
guaranteed to be a fast forward.

The serialization happens by having a single such daemon
running; no need to worry about locking anymore.

Re: [PATCH] Re: keeping remote repo checked out?

[Matthias Urlichs]

Hi, Junio C Hamano wrote:

> When everything goes well, the daemon goes to treeP and pulls
> from treeR's master into "deploy", which is checked out.

Why do you need a separate tree for that? -EOVERKILL.
I just use a separate index file for the production tree (and a well-known
name in refs/heads that's associated with it, for which the commit hook
blocks updates).

-- 
Matthias Urlichs   |   {M:U} IT Design @ m-u-it.de   |  smurf@smurf.noris.de
Disclaimer: The quote was selected randomly. Really. | http://smurf.noris.de
 - -
Nothing is so important that nothing else is important.

Re: [PATCH] Re: keeping remote repo checked out?

[Junio C Hamano]

Matthias Urlichs <smurf@smurf.noris.de> writes:

> Hi, Junio C Hamano wrote:
>
>> When everything goes well, the daemon goes to treeP and pulls
>> from treeR's master into "deploy", which is checked out.
>
> Why do you need a separate tree for that? -EOVERKILL.

Yes that is overkill, but treeR is used for merge and the scheme
allows that merge to be real merge not just fast forwards ---
you need a checked out tree for that.  We can use multiple
checked out tree sharing the same .git/ directory with separate
index files instead.

Re: [PATCH] Re: keeping remote repo checked out?

[Daniel Barkalow]

On Mon, 28 Nov 2005, Junio C Hamano wrote:

> Daniel Barkalow <barkalow@iabervon.org> writes:
> 
> >>     server$ git checkout production
> >>     work$ git checkout master
> >>     work$ git pull server production ;# merge into work's master
> >>     work$ git push server master:receive_from_work
> >>     server$ git pull . receive_from_work ;# merge into server's production
> >> 
> >> and you make sure receive_from_work is not checked out on server
> >> (or production is never pushed into) and always do fast forward
> >> and nothing else.
> >
> > That's what I'm doing currently, actually (with work pushing to 
> > "production", but server having checked out "deploy"), and I find it 
> > annoying to have to do the pull each time and have a separate head. It 
> > also means that, if the stuff on the server is set up as a hook, it'll 
> > have to do the locking against concurrent changes to the working tree in 
> > pull, which is much more complicated than checkout.
> 
> This depends on what is on the checked-out working tree and how
> it is used, but one use pattern I was imagining was actually
> much more elaborate.  I would set up two non-naked repositories
> on the server; let's call them treeR and treeP (reception and
> publish).

This is quite elaborate.

> Each work machine push into its own reception branch of treeR
> repository (so you have N reception branches for N workers).  On
> the server machine, a daemon monitors the reception branch heads
> (inotify or idle poll loop looking at mtime; the details do not
> matter), and run the git pull into "master" of treeR.  My daemon
> might want to forbid anything but fast forward for this pull, or
> might allow any clean merge. It compiles and runs testsuites
> next.  If anything fails during the above process, the owner of
> the reception branch is notified about the failure; I might even
> want to forcibly rewind her reception branch in this case to
> "master" when this happens.

This seems maximally inconvenient. If you lose a race, you only find out 
later, your reception branch is screwed up, and you have no way of finding 
out in advance that this is going to happen? It's much nicer to have 
people all push to a single branch, and tell them if they lose (i.e., the 
change wouldn't be a fast-forward).

> When everything goes well, the daemon goes to treeP and pulls
> from treeR's master into "deploy", which is checked out.  This
> procedure is the only thing that touches treeP, so this pull is
> guaranteed to be a fast forward.
> 
> The serialization happens by having a single such daemon
> running; no need to worry about locking anymore.

I'm not sure what the big deal about locking is. We do locking for 
updating everything else; I don't see a problem with having a lock for 
when git updates the working tree, so long as it doesn't need to hold some 
other lock as well at the same time (in which case we'd have to think 
about lock ordering).

(Actually, adding CHECKED_OUT would also allow pulling the current branch 
cleanly without all the special-casing in get-fetch.sh and git-pull.sh; it 
would have the predictable effect of leaving your working tree not up to 
date, but the ref it follows up to date, and the usual two-way merge in a 
pull would obviously do the right thing. Or you could just fetch, find out 
what will change with "git diff", and get the changes with "git 
checkout". In general, it would fix aliasing problems between 
refs/heads/* and HEAD.)

Re: [PATCH] Re: keeping remote repo checked out?

[Junio C Hamano]

Daniel Barkalow <barkalow@iabervon.org> writes:

> This seems maximally inconvenient. If you lose a race, you only find out 
> later, your reception branch is screwed up, and you have no way of finding 
> out in advance that this is going to happen?

I am not sure what you mean by "your reception branch is screwed
up".  We _could_ rewind that reception branch after acceptance
test fails but I did not mention that because I haven't thought
it through.

This was actually designed to reduce the chance of getting a
race in the first place, so I am not sure if it makes things
inconvenient.

Alice and Bob starts out from the central repository commit O
(for origin), and make progress independently.  They have one
commit on top of O each, A and B where A~1 === O and B~1 === O.

Alice pushes first, the central repository has O and accepts
because updating from O to A is a fast forward.  Bob tries to
push, and in the classic CVS-style shared repository setup in
the tutorial, this is prevented because this is not a fast
forward.  Bob needs to first pull and merge A and B to create C
(C^1 === B, C^2 === A) and push that.  This succeeds.

Instead, the approach allows you to choose to do the merge on
the server side unattended, as part of the acceptance check.

Alice pushes, and both her reception branch and the reception
repository master becomes A (fast forward).  When Bob pushes B
to his reception branch, we attempt to pull it into reception
repository master --- we do not have to fail this pull even if
it is not a fast forward (we could choose to fail it). reception
repository master becomes C which is a merge between A and B.
The result of this automerge needs to be checked for sanity, so
before this C (new reception repository master) is pulled into
the deployment repository, there is a validation step (this step
can do things other than validation; e.g. making tarballs for
distribution, automatic tagging).  While all that is happening,
other people can be pushing into their own reception branches,
waiting for their turn.  And we guarantee that what is moved to
the deployment repository has been tested "clean" (depending on
the quality of test, that is).

If the central repository acceptance policy is simple enough,
namely, if it takes whatever an individiual developer with push
access says is good at the face value, then we do not need any
of the above, and a simple "fast forward only" is good enough,
far simpler to explain and understand.

On the other hand, at places where management already has rules
that require any update to the central repository to first pass
a test suite, the acceptance test can take time to complete ---
which enlarges the race window --- and more importantly when a
push that passed a simple "fast forward only" rule fails, we
somehow need to prevent that failed head from leaking out to the
public.  That is cumbersome to arrange if we only use a single
reception branch.

In case it was not obvious, the reception repository is meant to
be push-only, and all the fetching for public consumption are
meant to happen from the deployed repository "master".  Most
importantly, reception branches in reception repository are not
to be pulled by individual developers.

Re: [PATCH] Re: keeping remote repo checked out?

[Daniel Barkalow]

On Tue, 29 Nov 2005, Junio C Hamano wrote:

> Daniel Barkalow <barkalow@iabervon.org> writes:
> 
> > This seems maximally inconvenient. If you lose a race, you only find out 
> > later, your reception branch is screwed up, and you have no way of finding 
> > out in advance that this is going to happen?
> 
> I am not sure what you mean by "your reception branch is screwed
> up".  We _could_ rewind that reception branch after acceptance
> test fails but I did not mention that because I haven't thought
> it through.
> 
> This was actually designed to reduce the chance of getting a
> race in the first place, so I am not sure if it makes things
> inconvenient.
> 
> Alice and Bob starts out from the central repository commit O
> (for origin), and make progress independently.  They have one
> commit on top of O each, A and B where A~1 === O and B~1 === O.
> 
> Alice pushes first, the central repository has O and accepts
> because updating from O to A is a fast forward.  Bob tries to
> push, and in the classic CVS-style shared repository setup in
> the tutorial, this is prevented because this is not a fast
> forward.  Bob needs to first pull and merge A and B to create C
> (C^1 === B, C^2 === A) and push that.  This succeeds.
> 
> Instead, the approach allows you to choose to do the merge on
> the server side unattended, as part of the acceptance check.
> 
> Alice pushes, and both her reception branch and the reception
> repository master becomes A (fast forward).  When Bob pushes B
> to his reception branch, we attempt to pull it into reception
> repository master --- we do not have to fail this pull even if
> it is not a fast forward (we could choose to fail it). reception
> repository master becomes C which is a merge between A and B.

If the merge works automatically, then Bob can do it without any trouble. 
If it doesn't work automatically, then Bob has to do it. But if Bob might 
have to do a merge, he can't leave for the day until his commit has gone 
all the way through, in which case he might as well do any merges while he 
waits. In my mind, the critical issue for developers is time to success or 
failure, and that's longer in your scheme. Perhaps a better idea is to 
have a script on Bob's end react to failure of the push (due to not being 
a fast-forward) by automatically pulling the thing that it's supposed to 
merge with, and push again if it worked. (If it failed, Bob fixes it, and 
then pushes again.)

> The result of this automerge needs to be checked for sanity, so
> before this C (new reception repository master) is pulled into
> the deployment repository, there is a validation step (this step
> can do things other than validation; e.g. making tarballs for
> distribution, automatic tagging).  While all that is happening,
> other people can be pushing into their own reception branches,
> waiting for their turn.  And we guarantee that what is moved to
> the deployment repository has been tested "clean" (depending on
> the quality of test, that is).
> 
> If the central repository acceptance policy is simple enough,
> namely, if it takes whatever an individiual developer with push
> access says is good at the face value, then we do not need any
> of the above, and a simple "fast forward only" is good enough,
> far simpler to explain and understand.
> 
> On the other hand, at places where management already has rules
> that require any update to the central repository to first pass
> a test suite, the acceptance test can take time to complete ---
> which enlarges the race window --- and more importantly when a
> push that passed a simple "fast forward only" rule fails, we
> somehow need to prevent that failed head from leaking out to the
> public.  That is cumbersome to arrange if we only use a single
> reception branch.

I think the right thing is to only allow pushes after something passes 
testing, probably be having people push to tags, which the test server 
tests and then pushes to the common upstream. The developers have to wait 
for the testing, but can go home when that passes.

I think there is the possibility for something really clever where, when 
you try to submit, if there's something in the testing area, it rejects 
you, you merge with the thing in the testing area, wait for the test ahead 
of you to complete, and submit either the merge (if the previous thing 
passed) or your original (if the previous thing failed). That way, you can 
overlap the time that you spend waiting for the test farm to be free with 
the time you spend merging against things that are getting in ahead of 
you. (Of course, if the merge with the testing thing isn't automatic, you 
might cancel it and do something else until you know whether that thing 
will be rejected.)

	-Daniel
*This .sig left intentionally blank*

Re: [PATCH] Re: keeping remote repo checked out?

[Matthias Urlichs]

Hi, Daniel Barkalow wrote:

> That's what I'm doing currently, actually (with work pushing to 
> "production", but server having checked out "deploy"), and I find it 
> annoying to have to do the pull each time and have a separate head.

Why? I think "what's currently deployed" and "the latest commit on the
production branch" are two distinct concepts, and should be treated as
such.

That way, a clean push (update of PRODUCTION, controlled by the update
hook) and an update of DEPLOY (the fast-forward merge (unless you need
to roll back something) from your production branch, controlled by the
post-update hook) are also distinct, can be handled individually.

Personally I just use "git-read-tree -m -u deploy master && \
cat refs/heads/master > refs/heads/deploy" in the post-update trigger.
I don't care about HEAD at all; nobody is supposed to change that
directory manually anyway.

-- 
Matthias Urlichs   |   {M:U} IT Design @ m-u-it.de   |  smurf@smurf.noris.de
Disclaimer: The quote was selected randomly. Really. | http://smurf.noris.de
 - -
Bringing computers into the home won't change either one, but may
revitalize the corner saloon.

Re: keeping remote repo checked out?

[James Cloos]

>>>>> "Junio" == Junio C Hamano <junkio@cox.net> writes:

Junio> Creative use of hooks/post-update would solve that.

I'll have to look into that.

Junio> However, you should be very careful if you sometimes edit on
Junio> server and sometimes push from other machine to the server on
Junio> the same branch on the server.

I always pull before editing on the laptop....

Thanks,

-JimC
-- 
James H. Cloos, Jr. <cloos@jhcloos.com>

Re: keeping remote repo checked out?

[Junio C Hamano]

James Cloos <cloos@jhcloos.com> writes:

>>>>>> "Junio" == Junio C Hamano <junkio@cox.net> writes:
>
> Junio> Creative use of hooks/post-update would solve that.
>
> I'll have to look into that.

Documentation/howto/rebuild-from-update-hook.txt may help.

I push into the master repository, and its post-update hook
checks out the updated "master" head to rebuild the HTML
documentation that is shown at

	http://kernel.org/pub/software/scm/git/docs/

The checkout happens not in the master repository but in a
separate repository by pulling into the latter, since the master
is a naked repository without a working tree associated with it.