git.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [SCRIPT] cg-rpush & locking
@ 2005-05-31 19:00 Tony Lindgren
  2005-05-31 23:16 ` Nicolas Pitre
  2005-06-02 19:15 ` Dan Holmsand
  0 siblings, 2 replies; 13+ messages in thread
From: Tony Lindgren @ 2005-05-31 19:00 UTC (permalink / raw)
  To: git; +Cc: Matthias Urlichs

[-- Attachment #1: Type: text/plain, Size: 859 bytes --]

Hello all,

Attached is a little script we're using for pushing changes to the
linux-omap tree. I modified it from an earlier script done by
Matthias Urlichs.

It uses rsync over ssh and should work with rsync write access too.

In order to remove lock files using rsync, I've added
a .git/locks subdirectory that contains the lock file.

The reason for this is that it allows replacing the lock directory
with an empty directory using rsync. This removes the lock file
after the remote update is done.

Currently the local lock has some issues with chained pushes...
If somebody is pushing from a remote repo to the local repo while
the local repo is being pushed to some other remote repo, the lock
may get removed.

Of course the lock does not protect from local changes either.

Anybody have any better ideas for locking that also works with
rsync?

Tony

[-- Attachment #2: cg-rpush --]
[-- Type: text/plain, Size: 3901 bytes --]

#!/bin/sh
#
# Pushes changes from the local git repo to remote repo.
#
# Copyright (C) 2005 Tony Lindgren <tony@atomide.com>
#
# Some parts based on an earlier push script, 
# Copyright (C) 2005 Matthias Urlichs.
#
# Takes the remote repo's name or url as parameter.
#
# Most likely the remote repo is not rsync writable, but
# you can use rsync over ssh for the push.
#
# When using rsync over ssh, you must use the real repo
# path on the server and an ssh key to avoid typing in
# the password multiple times. For example:
#
# $ export RSYNC_FLAGS="-z --progress"
# $ export RSYNC_RSH="ssh -i /home/user/.ssh/my-git-key"
# $ cg-rpush some.machine:/home/git/repo
#

. ${COGITO_LIB}cg-Xlib

name=$1

BRANCHES=".git/branches"
HEADS=".git/refs/heads"
OBJECTS=".git/objects"
LOCKS=".git/locks"
REMOTE_LOCK="write_lock"
TMP="/tmp"

uri=""

function usage () {
	echo "Usage: [RSYNC_FLAGS=\"-e ssh\"] $0 some.machine:/home/git/repo"
	exit 1
}

function die () {
	if [ -f $LOCKS/$REMOTE_LOCK ]; then
		rm -f $LOCKS/$REMOTE_LOCK
	fi
	echo cg-rpush: $@ >&2
	exit 1
}

function clean_locks () {
	rm -f $LOCKS/$REMOTE_LOCK
	rsync $RSYNC_FLAGS -r --delete $LOCKS/ $uri/$LOCKS
}

function validate_input () {
	[ "$name" ] || usage
	if [ ! -d ".git" ]; then
		die "Could not find local .git directory"
	fi
	uri=$(cat $BRANCHES/$name 2>/dev/null)
	[ "$uri" ] || uri=$name
	echo $uri
}

#
# We must use a lock directory to allow removing the remote lock
# files with rsync by copying over it with an empty directory.
# Creating the remote lock file should be safe. However, please note
# that we must also be careful not to remove local .git/locks/write_lock
# in case somebody is pushing to our local repo from a remote machine.
# Currently the local lock file creation can conflict with a lock
# file creation from a remote machine to our local machine.
#
function lock_files () {
	echo "Attempting to to create a write lock on remote..."
	if [ ! -d $LOCKS ]; then
		mkdir $LOCKS;
	fi
	if [ -f $LOCKS/$REMOTE_LOCK ]; then
		echo "Local write_lock already exists: $LOCKS/$REMOTE_LOCK"
		exit 1
	fi
	lock_stamp="$USER@$HOSTNAME $(date)"
	echo $lock_stamp > $LOCKS/$REMOTE_LOCK
	rsync $RSYNC_FLAGS -r --ignore-existing $LOCKS/ $uri/$LOCKS

	# Check what the remote .git/locks/write_lock has
	tmpfile=$TMP/remote_lock_$RANDOM
	rsync $RSYNC_FLAGS "$uri/$LOCKS/$REMOTE_LOCK" $tmpfile
	remote_stamp=$(cat $tmpfile)
	rm -f $tmpfile
	if [ "$remote_stamp" != "$lock_stamp" ]; then
		die "Remote locked by $remote_stamp, please try again later"
	fi
}

function check_remote_version () {
	echo "Getting remote version..."
	tmpfile=$TMP/remote_head_$RANDOM
	rsync $RSYNC_FLAGS -Lr "$uri/$HEADS/master" $tmpfile
	remote_head=$(cat $tmpfile)
	rm -f $tmpfile
	if [ -z "$remote_head" ]; then
		clean_locks
		die "Remote repository does not have $uri/$HEADS/master"
	fi
	echo "Remote head is at: $remote_head"
	if [ "$(git-cat-file -t "$remote_head" 2>/dev/null)" != "commit" ]; then
		clean_locks
		die "Remote is ahead, please do a pull first"
	fi
}

function push_git_objects () {
	echo "Pushing .git/objects..."
	rsync $RSYNC_FLAGS --ignore-existing --whole-file -v -r \
		"$OBJECTS/" "$uri/$OBJECTS/"
}

function update_remote_head () {
	local_head=$(cat $HEADS/master)
	echo "Updating remote head to: $local_head"
	rsync $RSYNC_FLAGS -Lr $HEADS/master "$uri/$HEADS/master"
}

function print_note () {
	echo "Remote updated successfully"
	echo "NOTE: Not updating checked out remote files in case they"
	echo "have been edited locally on the remote machine."
	echo "To sync checked out files on remote, you can run cg-cancel"
	echo "on remote machine. You can check for uncommitted changes"
	echo "on remote with cg-diff first, which should only show"
	echo "changes done in this push."
}

#
# Main program
#
uri=$(validate_input)
lock_files
check_remote_version
push_git_objects
update_remote_head
clean_locks
print_note
exit

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [SCRIPT] cg-rpush & locking
  2005-05-31 19:00 [SCRIPT] cg-rpush & locking Tony Lindgren
@ 2005-05-31 23:16 ` Nicolas Pitre
  2005-06-01  6:51   ` Thomas Glanzmann
  2005-06-02 19:15 ` Dan Holmsand
  1 sibling, 1 reply; 13+ messages in thread
From: Nicolas Pitre @ 2005-05-31 23:16 UTC (permalink / raw)
  To: Tony Lindgren; +Cc: git, Matthias Urlichs

On Tue, 31 May 2005, Tony Lindgren wrote:

> Anybody have any better ideas for locking that also works with
> rsync?

Why do you need a lock at all?

Just update your HEAD reference last when you push and get it first when 
you pull.


Nicolas

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [SCRIPT] cg-rpush & locking
  2005-05-31 23:16 ` Nicolas Pitre
@ 2005-06-01  6:51   ` Thomas Glanzmann
  2005-06-01 16:55     ` Tony Lindgren
  2005-06-02  2:58     ` Linus Torvalds
  0 siblings, 2 replies; 13+ messages in thread
From: Thomas Glanzmann @ 2005-06-01  6:51 UTC (permalink / raw)
  To: Nicolas Pitre; +Cc: Tony Lindgren, git, Matthias Urlichs

Hello,

> Why do you need a lock at all?

> Just update your HEAD reference last when you push and get it first when 
> you pull.

consider the following scenario: Two people push at the same time. One
HEAD gets actually written, but both think that their changes got
upstream. Of course the 'upstream' tree is consitent, but incomplete.
That is why we need a lock. And the lock should be obtained before the
remote HEAD is retrieved, I think the following scenario is how to
handle it:

	1. acquire remote lock
	2. get remote HEAD
	3. if remote HEAD is ahead (not included in our history) abort
	   and free lock.
	4. push objects
	5. update remote HEAD with local
	6. free remote lock.

	Thomas

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [SCRIPT] cg-rpush & locking
  2005-06-01  6:51   ` Thomas Glanzmann
@ 2005-06-01 16:55     ` Tony Lindgren
  2005-06-02  2:58     ` Linus Torvalds
  1 sibling, 0 replies; 13+ messages in thread
From: Tony Lindgren @ 2005-06-01 16:55 UTC (permalink / raw)
  To: Thomas Glanzmann; +Cc: Nicolas Pitre, git, Matthias Urlichs

* Thomas Glanzmann <sithglan@stud.uni-erlangen.de> [050531 23:56]:
> Hello,
> 
> > Why do you need a lock at all?
> 
> > Just update your HEAD reference last when you push and get it first when 
> > you pull.
> 
> consider the following scenario: Two people push at the same time. One
> HEAD gets actually written, but both think that their changes got
> upstream. Of course the 'upstream' tree is consitent, but incomplete.
> That is why we need a lock. And the lock should be obtained before the
> remote HEAD is retrieved, I think the following scenario is how to
> handle it:
> 
> 	1. acquire remote lock
> 	2. get remote HEAD
> 	3. if remote HEAD is ahead (not included in our history) abort
> 	   and free lock.
> 	4. push objects
> 	5. update remote HEAD with local
> 	6. free remote lock.

Yes, that's basically what the script does. We have several people
committing patches.

Tony

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [SCRIPT] cg-rpush & locking
  2005-06-01  6:51   ` Thomas Glanzmann
  2005-06-01 16:55     ` Tony Lindgren
@ 2005-06-02  2:58     ` Linus Torvalds
  2005-06-02  6:39       ` Daniel Barkalow
  1 sibling, 1 reply; 13+ messages in thread
From: Linus Torvalds @ 2005-06-02  2:58 UTC (permalink / raw)
  To: Thomas Glanzmann; +Cc: Nicolas Pitre, Tony Lindgren, git, Matthias Urlichs



On Wed, 1 Jun 2005, Thomas Glanzmann wrote:
> 
> 	1. acquire remote lock
> 	2. get remote HEAD
> 	3. if remote HEAD is ahead (not included in our history) abort
> 	   and free lock.
> 	4. push objects
> 	5. update remote HEAD with local
> 	6. free remote lock.

You really need a specialized client at the other end, because regardless 
of locking, you want to write the objects atomically (ie download them 
into a temp-file, and then do the "rename" thing to make them show up 
all-or-nothing).

Also, I'd suggest a slight modification to avoid keeping the lock for a 
long time, namely to have the lock protect just a quick "compare and 
exchange". So the algorithm would become:

	1. read remote HEAD
	2. if remote HEAD isn't in our history, abort with "remote is 
	   ahead"
	3. calculate the objects needed to push locally
	4. push them (but accept the possibility that the remote may
	   already have them, so have the protocol able to say "got that
	   one already"). Make this use the atomic write on the other end.
	5. do an atomic compare-and-exchange of the remote head with the 
	   new one (ie only switch the remote HEAD if it still matches 
	   what we were expecting it to be)

Hmm?

		Linus

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [SCRIPT] cg-rpush & locking
  2005-06-02  2:58     ` Linus Torvalds
@ 2005-06-02  6:39       ` Daniel Barkalow
  2005-06-02  7:14         ` Matthias Urlichs
  0 siblings, 1 reply; 13+ messages in thread
From: Daniel Barkalow @ 2005-06-02  6:39 UTC (permalink / raw)
  To: Linus Torvalds
  Cc: Thomas Glanzmann, Nicolas Pitre, Tony Lindgren, git,
	Matthias Urlichs

On Wed, 1 Jun 2005, Linus Torvalds wrote:

> 
> 
> On Wed, 1 Jun 2005, Thomas Glanzmann wrote:
> > 
> > 	1. acquire remote lock
> > 	2. get remote HEAD
> > 	3. if remote HEAD is ahead (not included in our history) abort
> > 	   and free lock.
> > 	4. push objects
> > 	5. update remote HEAD with local
> > 	6. free remote lock.
> 
> You really need a specialized client at the other end, because regardless 
> of locking, you want to write the objects atomically (ie download them 
> into a temp-file, and then do the "rename" thing to make them show up 
> all-or-nothing).
> 
> Also, I'd suggest a slight modification to avoid keeping the lock for a 
> long time, namely to have the lock protect just a quick "compare and 
> exchange". So the algorithm would become:
> 
> 	1. read remote HEAD
> 	2. if remote HEAD isn't in our history, abort with "remote is 
> 	   ahead"
> 	3. calculate the objects needed to push locally
> 	4. push them (but accept the possibility that the remote may
> 	   already have them, so have the protocol able to say "got that
> 	   one already"). Make this use the atomic write on the other end.
> 	5. do an atomic compare-and-exchange of the remote head with the 
> 	   new one (ie only switch the remote HEAD if it still matches 
> 	   what we were expecting it to be)
> 
> Hmm?

If the lock is only to protect against someone else modifying HEAD after
we've checked that it is our starting point and before we modify it,
there's no reason not to hold the lock while pushing; it wouldn't block
anything other than someone doing a quick push in the middle of our long
one, and thereby causing us to dump a lot of useless objects on the
server (which will become obsolete as we will need to do the merge and
push a different version).

The key is that people can still download the old version until the new
version is there, regardless of the lock; they'll get data about to
go stale, but they would have anyway had they been a few seconds
earlier. The main annoyance would be that you'd be blocked from pushing,
and then have to poll for the other upload to finish before you'd be able
to pull, do the merge, and then push your changes; you want to have the
client watch for the resolution of the other transfer one way or the
other, since you're in the current state precisely because you lost on
getting the lock and now definitely need the next version.

	-Daniel
*This .sig left intentionally blank*


^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [SCRIPT] cg-rpush & locking
  2005-06-02  6:39       ` Daniel Barkalow
@ 2005-06-02  7:14         ` Matthias Urlichs
  2005-06-02  7:32           ` Tony Lindgren
  0 siblings, 1 reply; 13+ messages in thread
From: Matthias Urlichs @ 2005-06-02  7:14 UTC (permalink / raw)
  To: Daniel Barkalow
  Cc: Linus Torvalds, Thomas Glanzmann, Nicolas Pitre, Tony Lindgren,
	git

Hi,

Daniel Barkalow:
> If the lock is only to protect against someone else modifying HEAD after
> we've checked that it is our starting point and before we modify it,
> there's no reason not to hold the lock while pushing; it wouldn't block
> anything other than someone doing a quick push in the middle of our long
> one, and thereby causing us to dump a lot of useless objects on the
> server (which will become obsolete as we will need to do the merge and
> push a different version).
> 
The objects we push aren't going to be obsolete. The server needs them
anyway, because our HEAD refers to them.

What if the connection dies in the middle of a push? You then sit there
waiting for it, and the lock, to time out. OTOH, an atomic cmpxchg on
the server can't block and can't timeout.

> you want to have the
> client watch for the resolution of the other transfer one way or the
> other, since you're in the current state precisely because you lost on
> getting the lock and now definitely need the next version.
> 
I disagree. Given that you need to wait for the upload to finish anyway
(whether you know it or not ;-) it makes sense to spend the time
actually uploading -- upload speed is frequently lower than download
for individuals.

-- 
Matthias Urlichs   |   {M:U} IT Design @ m-u-it.de   |  smurf@smurf.noris.de
Disclaimer: The quote was selected randomly. Really. | http://smurf.noris.de
 - -
Who was that masked man?

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [SCRIPT] cg-rpush & locking
  2005-06-02  7:14         ` Matthias Urlichs
@ 2005-06-02  7:32           ` Tony Lindgren
  2005-06-02 10:04             ` Matthias Urlichs
  2005-06-02 14:50             ` Linus Torvalds
  0 siblings, 2 replies; 13+ messages in thread
From: Tony Lindgren @ 2005-06-02  7:32 UTC (permalink / raw)
  To: Matthias Urlichs
  Cc: Daniel Barkalow, Linus Torvalds, Thomas Glanzmann, Nicolas Pitre,
	Tony Lindgren, git

On Thu, Jun 02, 2005 at 09:14:53AM +0200, Matthias Urlichs wrote:
> Hi,
> 
> Daniel Barkalow:
> > If the lock is only to protect against someone else modifying HEAD after
> > we've checked that it is our starting point and before we modify it,
> > there's no reason not to hold the lock while pushing; it wouldn't block
> > anything other than someone doing a quick push in the middle of our long
> > one, and thereby causing us to dump a lot of useless objects on the
> > server (which will become obsolete as we will need to do the merge and
> > push a different version).
> > 
> The objects we push aren't going to be obsolete. The server needs them
> anyway, because our HEAD refers to them.

I don't think locking for the duration of the push really is a problem.
It is unlikely that there would be so many people pushing that it would
cause inconvenience... Of course it would be nice to optimize it if
possible.

> What if the connection dies in the middle of a push? You then sit there
> waiting for it, and the lock, to time out. OTOH, an atomic cmpxchg on
> the server can't block and can't timeout.
> 
> > you want to have the
> > client watch for the resolution of the other transfer one way or the
> > other, since you're in the current state precisely because you lost on
> > getting the lock and now definitely need the next version.
> > 
> I disagree. Given that you need to wait for the upload to finish anyway
> (whether you know it or not ;-) it makes sense to spend the time
> actually uploading -- upload speed is frequently lower than download
> for individuals.

I would assume the biggest problem for most people is how they can push
through a firewall. From that point of view it would make sense to do
the push as a cgi script rather than something over ssh. And with a
cgi script you can of course optimize the locking and use tmp files
before renaming which are a bit hard to do with rsync.

Tony

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [SCRIPT] cg-rpush & locking
  2005-06-02  7:32           ` Tony Lindgren
@ 2005-06-02 10:04             ` Matthias Urlichs
  2005-06-02 14:50             ` Linus Torvalds
  1 sibling, 0 replies; 13+ messages in thread
From: Matthias Urlichs @ 2005-06-02 10:04 UTC (permalink / raw)
  To: Tony Lindgren
  Cc: Daniel Barkalow, Linus Torvalds, Thomas Glanzmann, Nicolas Pitre,
	git

[-- Attachment #1: Type: text/plain, Size: 1238 bytes --]

Hi,

Tony Lindgren:
> I don't think locking for the duration of the push really is a problem.
> It is unlikely that there would be so many people pushing that it would
> cause inconvenience... Of course it would be nice to optimize it if
> possible.
> 
Since an 'atomic' cmpxchg operation is actually easier to program than a
lockfile with timeout handling etc., I would hope so. ;-)

> I would assume the biggest problem for most people is how they can push
> through a firewall. From that point of view it would make sense to do
> the push as a cgi script rather than something over ssh.

When in doubt, do both ... I'd certainly prefer ssh if at all possible.

> And with a cgi script you can of course optimize the locking and use
> tmp files before renaming which are a bit hard to do with rsync.
> 
rsync is going away anyway for git usage (long-term), I'd assume. It
certainly becomes more and more ineffective the more our history is
growing.

-- 
Matthias Urlichs   |   {M:U} IT Design @ m-u-it.de   |  smurf@smurf.noris.de
Disclaimer: The quote was selected randomly. Really. | http://smurf.noris.de
 - -
No one can guarantee the actions of another.
		-- Spock, "Day of the Dove", stardate unknown

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [SCRIPT] cg-rpush & locking
  2005-06-02  7:32           ` Tony Lindgren
  2005-06-02 10:04             ` Matthias Urlichs
@ 2005-06-02 14:50             ` Linus Torvalds
  2005-06-02 17:54               ` Tony Lindgren
  2005-06-02 19:12               ` Daniel Barkalow
  1 sibling, 2 replies; 13+ messages in thread
From: Linus Torvalds @ 2005-06-02 14:50 UTC (permalink / raw)
  To: Tony Lindgren
  Cc: Matthias Urlichs, Daniel Barkalow, Thomas Glanzmann,
	Nicolas Pitre, git



On Thu, 2 Jun 2005, Tony Lindgren wrote:
> 
> I don't think locking for the duration of the push really is a problem.
> It is unlikely that there would be so many people pushing that it would
> cause inconvenience... Of course it would be nice to optimize it if
> possible.

I don't think the locking is a _huge_ issue - the only real problem I had
with locking with BK was that readers could block a writer (ie I couldn't
push to my public tree if there were people downloading from it), and git
doesn't have that problem. I only push to trees that are my private ones
anyway.

The real reason I'd prefer to not do locking is that _if_ the remote tree
is actually more than just a CVS-like "public repository", ie if somebody
actually does _development_ in the remote tree (hey, it may be crazy, but
git makes this usage pattern possible), then we should eventually plan on
having all of the regular "git-commit-script" and "git-pull-script" etc
_also_ do locking, since they also change HEAD.

And that's going to be a lot easier if we only do a cmp-xchg and fail at
the end (this concurrent change and "remote repo" usage is really quite
wrong, since it basically mean that you consider somebody's development
tree to be "public", and that's not what git is all about, but whatever..)

But this is not a huge issue. The most important thing is to make sure
that the new HEAD is written last, regardless, so that at least local
readers (including things like "fsck" that can take a _loong_ time) always
see consistent state.

> I would assume the biggest problem for most people is how they can push
> through a firewall. From that point of view it would make sense to do
> the push as a cgi script rather than something over ssh. And with a
> cgi script you can of course optimize the locking and use tmp files
> before renaming which are a bit hard to do with rsync.

Me personally, I want ssh as a major option. There are tons of machines 
(every single of my own ones) that I use that don't let anything but ssh 
through.

But having alternatives is good. But ssh should be the first and primary 
one, since it also means that there can't be any new security issues (ie 
you won't be opening up any new holes by installing git on the remote 
machine).

		Linus

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [SCRIPT] cg-rpush & locking
  2005-06-02 14:50             ` Linus Torvalds
@ 2005-06-02 17:54               ` Tony Lindgren
  2005-06-02 19:12               ` Daniel Barkalow
  1 sibling, 0 replies; 13+ messages in thread
From: Tony Lindgren @ 2005-06-02 17:54 UTC (permalink / raw)
  To: Linus Torvalds
  Cc: Matthias Urlichs, Daniel Barkalow, Thomas Glanzmann,
	Nicolas Pitre, git

* Linus Torvalds <torvalds@osdl.org> [050602 07:50]:
> 
> 
> On Thu, 2 Jun 2005, Tony Lindgren wrote:
> > 
> > I don't think locking for the duration of the push really is a problem.
> > It is unlikely that there would be so many people pushing that it would
> > cause inconvenience... Of course it would be nice to optimize it if
> > possible.
> 
> I don't think the locking is a _huge_ issue - the only real problem I had
> with locking with BK was that readers could block a writer (ie I couldn't
> push to my public tree if there were people downloading from it), and git
> doesn't have that problem. I only push to trees that are my private ones
> anyway.
> 
> The real reason I'd prefer to not do locking is that _if_ the remote tree
> is actually more than just a CVS-like "public repository", ie if somebody
> actually does _development_ in the remote tree (hey, it may be crazy, but
> git makes this usage pattern possible), then we should eventually plan on
> having all of the regular "git-commit-script" and "git-pull-script" etc
> _also_ do locking, since they also change HEAD.

Or perhaps a more likely scenario would be automated pulls from other trees
running as cronjobs on the remote server.

> And that's going to be a lot easier if we only do a cmp-xchg and fail at
> the end (this concurrent change and "remote repo" usage is really quite
> wrong, since it basically mean that you consider somebody's development
> tree to be "public", and that's not what git is all about, but whatever..)
> 
> But this is not a huge issue. The most important thing is to make sure
> that the new HEAD is written last, regardless, so that at least local
> readers (including things like "fsck" that can take a _loong_ time) always
> see consistent state.
> 
> > I would assume the biggest problem for most people is how they can push
> > through a firewall. From that point of view it would make sense to do
> > the push as a cgi script rather than something over ssh. And with a
> > cgi script you can of course optimize the locking and use tmp files
> > before renaming which are a bit hard to do with rsync.
> 
> Me personally, I want ssh as a major option. There are tons of machines 
> (every single of my own ones) that I use that don't let anything but ssh 
> through.

Yeah I prefer ssh too in general. Many corporate firewalls don't allow
ssh through though.

> But having alternatives is good. But ssh should be the first and primary 
> one, since it also means that there can't be any new security issues (ie 
> you won't be opening up any new holes by installing git on the remote 
> machine).

Yeah. Doing it as cgi also has some issues getting the file
permissions right so the repo would still work for ssh users too...

Is anybody planning to work on this? I'm pretty much out of
time and will probably be happy with the rsync script for now.

Tony

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [SCRIPT] cg-rpush & locking
  2005-06-02 14:50             ` Linus Torvalds
  2005-06-02 17:54               ` Tony Lindgren
@ 2005-06-02 19:12               ` Daniel Barkalow
  1 sibling, 0 replies; 13+ messages in thread
From: Daniel Barkalow @ 2005-06-02 19:12 UTC (permalink / raw)
  To: Linus Torvalds
  Cc: Tony Lindgren, Matthias Urlichs, Thomas Glanzmann, Nicolas Pitre,
	git

On Thu, 2 Jun 2005, Linus Torvalds wrote:

> The real reason I'd prefer to not do locking is that _if_ the remote tree
> is actually more than just a CVS-like "public repository", ie if somebody
> actually does _development_ in the remote tree (hey, it may be crazy, but
> git makes this usage pattern possible), then we should eventually plan on
> having all of the regular "git-commit-script" and "git-pull-script" etc
> _also_ do locking, since they also change HEAD.

It might be okay to have a single tree to which is applied patches from
emails sent by non-maintainers, pulls from non-maintainers with accessible
repositories, and pushes from home repositories of maintainers. It makes
sense to deal well with different ways to get refined commits into the
public repository. Of course, you probably don't want to carry out merges
there, so the cases where pull or commit loses the race are quite
different; you want to throw it back to the author for rebasing.

	-Daniel
*This .sig left intentionally blank*


^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [SCRIPT] cg-rpush & locking
  2005-05-31 19:00 [SCRIPT] cg-rpush & locking Tony Lindgren
  2005-05-31 23:16 ` Nicolas Pitre
@ 2005-06-02 19:15 ` Dan Holmsand
  1 sibling, 0 replies; 13+ messages in thread
From: Dan Holmsand @ 2005-06-02 19:15 UTC (permalink / raw)
  To: git; +Cc: Matthias Urlichs

[-- Attachment #1: Type: text/plain, Size: 1658 bytes --]

Tony Lindgren wrote:
> Anybody have any better ideas for locking that also works with
> rsync?

Since this seems to be Push Day on the git list, here's my feeble 
attempt at the same thing (attached cg-push script).

I do stuff in this order:

     1. read remote HEAD
     2. check that a merge from local HEAD to remote would be
        fast-forward, otherwise tell people to pull and merge.
     3. push objects using --ignore-existing (or equivalent)
     4. write lock file with --ignore-existing. The lock file
        contains, in particular, the HEAD to be written.
     5. read remote HEAD (again) and the lock file. Bail if HEAD
        changed since the first read, or if the lock file isn't
        the one we attempted to write.
     6. write remote HEAD and delete lock file in using rsync's
        --delete-after

This should always be safe, since rsync (as I understand the man page) 
always writes to temp files, and then renames into place. Checking for 
fast-forward-mergability assures that other peoples changes don't get lost.

cg-push determines uri and foreign branch name using the same rules as 
cg-pull, which is real nice as it allows you to do:

$ cg-clone me@myserver.example.com:git-repos/myrepo.git mystuff
$ cd mystuff
# write some stuff
$ cg-commit
$ cg-push

and later

$ cg-update
# write more stuff
$ cg-commit
$ cg-push

and even later

$ cg-branch-add pub me@myserver.example.com:public-repos/myrepo.git
$ cg-push pub

So, you can work pretty much exactly as you would do in CVS or svn, if 
you're so inclined, safely sharing a common repository among many users.

Which is kinda neat, if I may say so myself...

/dan

[-- Attachment #2: cg-push --]
[-- Type: text/plain, Size: 6219 bytes --]

#! /usr/bin/env bash
#
# Push changes to a remote git repository
#
# Copyright (c) Dan Holmsand, 2005.
#
# Based on cg-pull,
# Copyright (c) Petr Baudis, 2005.
#
# Takes the branch name as an argument, defaulting to "origin" (see
# `cg-branch-add` for some description of branch names).
#
# Takes one optional option: --force, that makes cg-push write objects
# regardless of lock-files and remote state. Use with care...
#
# cg-push supports two types of location specifiers:
#
# 1. local paths - simple directory names of git repositories
# 2. rsync - using the "[user@]machine:some/path" syntax
#
# Typical use would look something like this:
#
#	# clone a remote branch:
#	cg-clone me@myserver.example.com:repo.git myrepo
#	cd myrepo
#
#	# make some changes, and then do:
#	cg-commit
#	cg-push
#	
# cg-push is safe to use even if multiple users concurrently push
# to the same repository. Here's how this works:
#
# First, cg-push checks that the local repository is fully merged with
# the remote one (this will always be the case if there is only one
# user). Otherwise, you need to cg-update from the remote repository
# before you can cg-push to it.
#
# Then, cg-push writes all the object files that are missing in the
# remote repository.
#
# To finish, cg-push writes a lock file to the remote site (ignoring
# any preexisting lock file), checks that our lock file actually got 
# written, checks that the remote head is still the same (i.e. that 
# the remote site hasn't been updated while we were copying objects), 
# writes the new remote head and removes the lock.
#
# The head of the local repository is also updated in the process (as if
# the remote branch had been cg-pull'ed).
#
# cg-push requires that there already is a repository in place at the
# remote location, but it actually only checks that it has a 
# "refs/heads" subdirectory. So, creating a remote repo ready for
# cg-pushing is as easy "mkdir -p repo.git/heads/refs" at the remote
# location.

# TODO: Write tags as well.

. ${COGITO_LIB}cg-Xlib

force=
if [ "$1" = --force ]; then
	force=1; shift
fi

name=$1
[ "$name" ] || { [ -s $_git/refs/heads/origin ] && name=origin; }
[ "$name" ] || die "what to push to?"
uri=$(cat "$_git/branches/$name" 2>/dev/null) || die "unknown branch: $name"

rembranch=master
if echo "$uri" | grep -q '#'; then
	rembranch=$(echo $uri | cut -d '#' -f 2)
	uri=$(echo $uri | cut -d '#' -f 1)
fi

case $uri in
	*:*)
	readhead=rsync_readhead
	writeobjects=rsync_writeobjects
	writehead=rsync_writehead
	lock=rsync_lock
	;;
	*)
	if [ -d "$uri" ]; then
		[ -d "$uri/.git" ] && uri=$uri/.git
		readhead=local_readhead
		writeobjects=local_writeobjects
		writehead=local_writehead
		lock=local_lock
	else
		die "Don't know how to push to $uri"
	fi
	;;
esac

tmpd=$(mktemp -d -t cgpush.XXXXXX) || exit 1
trap "rm -rf $tmpd" SIGTERM EXIT

cid=$(commit-id) || exit 1
lock_msg="locked by $USER@$HOSTNAME on $(date) for writing $cid"
unset locked remhead

rsync_readhead() {
	rm -f "$tmpd/*" || return 1
	rsync $RSYNC_FLAGS --include="$rembranch" --include="$rembranch.lock" \
		--exclude='*' -r "$uri/refs/heads/" "$tmpd/" >&2 || 
		die "Fetching heads from $uri failed. Aborting."
	if [ "$locked" ]; then
		[ -s "$tmpd/$rembranch.lock" ] ||
		die "Couldn't acquire lock. Aborting."

		local rem_lock_msg=$(cat "$tmpd/$rembranch.lock")
		[ "$lock_msg" = "$rem_lock_msg" ] ||
		die "Remote is locked ($rem_lock_msg)."
	fi
	[ ! -e "$tmpd/$rembranch" ] || cat "$tmpd/$rembranch"
}

rsync_writeobjects() {
	[ -d "$_git/objects/" ] || die "no objects to copy"
	rsync $RSYNC_FLAGS -vr --ignore-existing --whole-file \
		"$_git/objects/" "$uri/objects/" 
}

rsync_lock() {
	echo "$lock_msg" > $tmpd/new_head_lock_file || return 1
	rsync $RSYNC_FLAGS --ignore-existing --whole-file \
		$tmpd/new_head_lock_file "$uri/refs/heads/$rembranch.lock"
}

rsync_writehead() {
	local heads=$tmpd/newhead
	mkdir $heads && echo "$1" > "$heads/$rembranch" || return 1
	rsync $RSYNC_FLAGS --include="$rembranch" --include="$rembranch.lock" \
		--exclude='*' --delete-after -r $heads/ "$uri/refs/heads/" 
}

local_readhead() {
	local lheads=$uri/refs/heads
	[ -d "$lheads" ] || die "no remote heads found at $uri"
	[ ! -e "$lheads/$rembranch" ] || cat "$lheads/$rembranch" 
}

local_writeobjects() {
	[ -d "$_git/objects/" ] || die "no objects to copy"
	[ -d "$uri/objects" ] || 
		GIT_DIR=$uri GIT_OBJECT_DIRECTORY=$uri/objects git-init-db ||
		die "git-init-db failed"
        # Note: We could use git-local-pull here, but this is safer
	# (git-*-pull don't react well to failures or kills), and
	# has the same semantics as rsync pushing.
	local dest=$(cd "$uri/objects" && pwd) || exit 1
	( cd "$_git/objects" && find -type f | while read f; do
		[ -f "$dest/$f" ] && continue
		ln "$f" "$dest/$f" 2>/dev/null || 
		cp "$f" "$dest/$f" || exit 1
	done ) 
}

local_lock() {
	([ "$force" ] || set -C 
	echo "$lockmsg" > "$uri/refs/heads/$rembranch.lock") 2>/dev/null
}

local_writehead() {
	local head=$uri/refs/heads/$rembranch
	echo "$1" > "$head.new" && mv "$head.new" "$head" &&
	rm "$head.lock"
}

echo "Checking remote repository"
remhead=$($readhead) || exit 1
[ "$remhead" ] || echo "Creating new branch"

if [ "$remhead" -a -z "$force" ]; then
	if [ "$remhead" = "$cid" ]; then
		echo "Remote branch \`$name' is already pushed" 
		exit 0
	fi
	git-cat-file commit "$remhead" &> /dev/null ||
	die "You need to pull from $name first. Aborting."

	base=$(git-merge-base "$remhead" "$cid") && [ "$base" ] ||
	die "You need to merge $name. Aborting."

	if [ "$base" = "$cid" ]; then
		echo "No changes to push"; exit 0
	fi

	[ "$base" = "$remhead" ] || 
	die "You need to merge $name first. Aborting." 
fi

echo "Writing objects"
$writeobjects || die "Failed to write objects. Aborting."

echo
echo "Writing new head"
$lock || die "Couldn't acquire lock on remote. Aborting."

if [ ! "$force" ]; then
	locked=1
	remhead2=$($readhead) || die "Aborting."
	[ "$remhead" = "$remhead2" ] || 
		die "Remote head changed during copy. Aborting."
fi

$writehead "$cid" || die "WARNING: Error writing remote head. Aborting."
echo "Push to $name succeeded"

echo "$cid" > "$_git/refs/heads/$name"
echo "Updated local head for $name to $cid"


^ permalink raw reply	[flat|nested] 13+ messages in thread

end of thread, other threads:[~2005-06-02 19:18 UTC | newest]

Thread overview: 13+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2005-05-31 19:00 [SCRIPT] cg-rpush & locking Tony Lindgren
2005-05-31 23:16 ` Nicolas Pitre
2005-06-01  6:51   ` Thomas Glanzmann
2005-06-01 16:55     ` Tony Lindgren
2005-06-02  2:58     ` Linus Torvalds
2005-06-02  6:39       ` Daniel Barkalow
2005-06-02  7:14         ` Matthias Urlichs
2005-06-02  7:32           ` Tony Lindgren
2005-06-02 10:04             ` Matthias Urlichs
2005-06-02 14:50             ` Linus Torvalds
2005-06-02 17:54               ` Tony Lindgren
2005-06-02 19:12               ` Daniel Barkalow
2005-06-02 19:15 ` Dan Holmsand

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).