backup or mirror a repository

backup or mirror a repository

[Dan Farina]

Hello list,

I have been poking around on IRC and gmane, but haven't yet seen a
solution to my problem:

What I really want is something semantically identical to a recursive
copy of my entire git repository on one machine sent to another machine.
Right now I can't simply use "rsync -ar --delete ..." because the pack
changes names, thus fooling rsync and sending a bunch of bits over the
network.

I saw git-mirror on gmane and asked around in IRC, but the former seems
to have died sometime last November and the latter told me I'd probably
have to write some glue to process git show-ref and git ls-remote. This
seems viable, but suboptimal. Another user chimed in that they also had
the same use case as myself.


fdr

(Justifications: I am super paranoid, so I want to push my local
branches to another disk to be safe, but then they proliferate over time
since deleted local branches are not also deleted on the target
repository. I also want to work on multiple machines seamlessly, and as
such doing a bunch of "git checkout -r ..." when I need to change
branches is also suboptimal. As I said, I can get what I want by simply
using rsync, so this is entirely about efficiency by exploiting git's
structure)

Re: backup or mirror a repository

[Johannes Schindelin]

Hi,

On Thu, 27 Sep 2007, Dan Farina wrote:

> I have been poking around on IRC and gmane, but haven't yet seen a 
> solution to my problem:

I guess you're looking for "git remote add --mirror origin <url>".  AFAIK 
this is not in any released version yet, though.

Ciao,
Dscho

Re: backup or mirror a repository

[Dan Farina]

On Thu, 2007-09-27 at 22:56 +0100, Johannes Schindelin wrote:
> I guess you're looking for "git remote add --mirror origin <url>".  AFAIK 
> this is not in any released version yet, though.

I just tried a bleeding edge git with --mirror.

It's close, except (unless there is some other provision to take care of
this) remote branch deletes. (Which, after more gmane trolling, seemed
to be "git remote rm", which doesn't seem present just yet).

Of course, it is completely possible I'm using it incorrectly. In brief,
I did "git init" "git remote add --mirror backup <url>" "git
pull" (perhaps I should have used "git remote update"?) And the behavior
of this is more or less what I wanted, as far as sending branches.
Convenient removal of branches and propagating that removal still stumps
me. Is there a preferred way to do this yet?

Thanks,
fdr

Re: backup or mirror a repository

[Johannes Schindelin]

Hi,

On Thu, 27 Sep 2007, Dan Farina wrote:

> On Thu, 2007-09-27 at 22:56 +0100, Johannes Schindelin wrote:
> > I guess you're looking for "git remote add --mirror origin <url>".  
> > AFAIK this is not in any released version yet, though.
> 
> I just tried a bleeding edge git with --mirror.
> 
> It's close, except (unless there is some other provision to take care of 
> this) remote branch deletes. (Which, after more gmane trolling, seemed 
> to be "git remote rm", which doesn't seem present just yet).

git remote rm is about removing the remote nick, not a remote branch.

> Of course, it is completely possible I'm using it incorrectly. In brief, 
> I did "git init" "git remote add --mirror backup <url>" "git pull"

"git pull" without further parameters accesses the remote "origin", not 
"backup", and it merges _one_ branch into the current branch.

What you should have done is "git fetch backup".

Or use "origin" instead of "backup" right from the start, and then use 
"git fetch".

But no pull.

> (perhaps I should have used "git remote update"?) And the behavior of 
> this is more or less what I wanted, as far as sending branches. 
> Convenient removal of branches and propagating that removal still stumps 
> me. Is there a preferred way to do this yet?

You want to remove branches from the local repo which are no longer there 
on the remote side?

Then "git remote prune" is for you.  For details, please see the man page.

Ciao,
Dscho

Re: backup or mirror a repository

[Dan Farina]

On Fri, 2007-09-28 at 02:56 +0100, Johannes Schindelin wrote:
> Hi,
> 
> On Thu, 27 Sep 2007, Dan Farina wrote:

> git remote rm is about removing the remote nick, not a remote branch.

Ah, I thought for some reason there'd be a "remove" (since the other
operations have long names) and, much like "ls-remote" there'd be some
"rm-remote." An insane presupposition of mine from having too little
information.


> What you should have done is "git fetch backup".
> 
> Or use "origin" instead of "backup" right from the start, and then use 
> "git fetch".
> 
> But no pull.

I did use "pull backup" (my error for being sloppy and omitting it from
my mail) and it does work provided I do the dirty thing of doing a
subsequent reset --hard (I thought perhaps things were not yet
fully-there). Fetch does it perfectly, as you say, thanks!

> You want to remove branches from the local repo which are no longer there 
> on the remote side?
> 
> Then "git remote prune" is for you.  For details, please see the man page.
> 

I did look at prune and update, but my problem is the opposite: I want
something that will remove branches from the remote repo when they no
longer exist locally. As-is over time I will proliferate little local
branches unless I occasionally sit down and delete branches by operating
directly on the bare backup repository. (and then use prune on the
remote nodes)

Thanks,
fdr

Re: backup or mirror a repository

[Junio C Hamano]

Dan Farina <drfarina@gmail.com> writes:

> I did look at prune and update, but my problem is the opposite: I want
> something that will remove branches from the remote repo when they no
> longer exist locally. As-is over time I will proliferate little local
> branches unless I occasionally sit down and delete branches by operating
> directly on the bare backup repository. (and then use prune on the
> remote nodes)

The "git remote add --mirror" setup is about setting up the
local repository _AS_ the backup of the remote.  In other words,
the contents come from the remote by fetching from it and safely
kept away from disaster on the local side.  And for that,
"remote prune" is a perfect thing to do.

I think what you are asking for is an opposite, a backup remote
site you would push into.  That is not what "remote add --mirror"
is about.

You can almost do it with

	git push --all $remote

except there is no way to automagically remove the branch you
removed from the local repository.  For that, we would need a
new --mirror option to "git-push".

I think it is trivial to do for native transports, as we first
get the list of all refs from the remote side before starting
the transfer.  You need to change the last parameter called
'all' to remote.c::match_refs() into an enum ('push_all' being
one of choices), introduce another enum 'push_mirror', and teach
it to "match" the remote (i.e. dst) ref that does not have
corresponding entry on our side (i.e. src) with an empty object
name to mark it removed.  Then the part marked as "Finally, tell
the other end!"  in send-pack.c::send_pack() will take care of
the actual removal.

Re: backup or mirror a repository

[Dan Farina]

On Thu, 2007-09-27 at 23:27 -0700, Junio C Hamano wrote:
> You can almost do it with
> 
> 	git push --all $remote
> 
> except there is no way to automagically remove the branch you
> removed from the local repository.  For that, we would need a
> new --mirror option to "git-push".

Yup! I actually was looking for such a thing in the git-push man page,
but was unsuccessful. I was living with git-push --all for a little
while before I thought I'd ask. Unfortunately, the preferred setup
(having the backup machine actively perform fetch and prune) is not very
nice for me due to firewalls.

> 
> I think it is trivial to do for native transports, as we first
> get the list of all refs from the remote side before starting
> the transfer.  You need to change the last parameter called
> 'all' to remote.c::match_refs() into an enum ('push_all' being
> one of choices), introduce another enum 'push_mirror', and teach
> it to "match" the remote (i.e. dst) ref that does not have
> corresponding entry on our side (i.e. src) with an empty object
> name to mark it removed.  Then the part marked as "Finally, tell
> the other end!"  in send-pack.c::send_pack() will take care of
> the actual removal.

I should have a look, but if someone else wants to work on this they
shouldn't block on me: work already made me get hit by a bus on one
project (luckily I wasn't important), and the storm is not over.

Thanks,
fdr

Re: backup or mirror a repository

[Dmitry Potapov]

On Thu, Sep 27, 2007 at 11:27:06PM -0700, Junio C Hamano wrote:
> The "git remote add --mirror" setup is about setting up the
> local repository _AS_ the backup of the remote.  In other words,
> the contents come from the remote by fetching from it and safely
> kept away from disaster on the local side.  And for that,
> "remote prune" is a perfect thing to do.

I have tried to do that but I am getting a warning:
$ git remote prune origin
Warning: unrecognized mapping in remotes.origin.fetch: +refs/*:refs/*
and no branch is removed.

I suspect that the change that introduced --mirror option for the 'add'
command did not adjust the prune procedure to handle the new situation
properly. Or is just me doing something wrong?

Anyway, because the official released git still does not have --mirror
and it could be difficult to convince the admin to use the bleading edge
version of Git for secure backup, I wrote a simple script instead, which
provides the same functionality but using the released version of Git.
Maybe, someone else finds it useful.

$ cat git-mirror
#!/bin/sh

usage() {
	echo >&2 "Usage: $0 URL"
	exit 1
}

set -e

test $# -eq 1 || usage
URL="$1"

# Initialize Git bare directory
git --bare init

# Initialize the mirror
git remote add origin "$URL"
git fetch
rmdir refs/heads
ln -s remotes/origin refs/heads

# Adding creating a script that will be run by cron
cat <<EOF > git-mirror-sync
cd "$PWD"
git fetch
git remote prune origin
EOF
chmod +x git-mirror-sync

# Adding git-mirror-sync to cron
echo "Please, add $PWD/git-mirror-sync to cron"

#########################################################

Dmitry

Re: backup or mirror a repository

[Johannes Schindelin]

Hi,

On Thu, 18 Oct 2007, Dmitry Potapov wrote:

> On Thu, Sep 27, 2007 at 11:27:06PM -0700, Junio C Hamano wrote:
> > The "git remote add --mirror" setup is about setting up the
> > local repository _AS_ the backup of the remote.  In other words,
> > the contents come from the remote by fetching from it and safely
> > kept away from disaster on the local side.  And for that,
> > "remote prune" is a perfect thing to do.
> 
> I have tried to do that but I am getting a warning:
> $ git remote prune origin
> Warning: unrecognized mapping in remotes.origin.fetch: +refs/*:refs/*
> and no branch is removed.
> 
> I suspect that the change that introduced --mirror option for the 'add' 
> command did not adjust the prune procedure to handle the new situation 
> properly. Or is just me doing something wrong?

No, you're right.  I did not anticipate git-remote to be written the way 
it is.  After fiddling with it for several hours, I am giving up for now.  
IMHO the script is too married to the idea that the remote branches live 
in refs/remotes/<remote>/*.  Probably it would be very easy by now to 
implement it as a builtin, using remote.[ch].

Ciao,
Dscho