rsync a *bunch* of git repos

rsync a *bunch* of git repos

[Jason]

All,

I have a home directory with a lot of git repos in it.  Possibly over a
hundred.  This is a good thing.  :-)  However, I'm in the process of
changing OS's and need to move my home directory out of my root
partition into its own partition.

The nuts and bolts of this aren't difficult, the problem is I don't have
a complete understanding of how git stores data.  I've heard in the
past that it uses a lot of hardlinks and softlinks.  I need to make
sure, that once I transfer the data, and reboot the machine with the new
partition mounted under /home, that all my git repos will be okay.

Going through the git repos one by one would be long and prone to
errors, so I'm looking at alternatives.  Does this look ok?

## From single user mode, or live cd
$ su -
# mkfs.ext4 /dev/sda3
# mount /dev/sda3 /mnt/home
# rsync --progress -avu /home/ /mnt/home/
# umount /mnt/home
# mv /home /home.orig
# mkdir /home

### Sanity check?
# mount /dev/sda3 /home
# rsync -avun /home.orig/ /home/

That'll check that rsync thinks things are ok, but what about git?  Is
there a git sanity check I could run to detect that three hardlinks
still point to the same inode?

One alternative may be to boot to a live CD, delete everything from the
root partition except /home, then mv /home/* -> / .  Basically, use my
current root partition as the new /home partition.

thx,

Jason.

Re: rsync a *bunch* of git repos

[Seth Robertson]

In message <20120110211548.GD10255@titan.lakedaemon.net>, Jason writes:

    The nuts and bolts of this aren't difficult, the problem is I don't have
    a complete understanding of how git stores data.  I've heard in the
    past that it uses a lot of hardlinks and softlinks.  I need to make
    sure, that once I transfer the data, and reboot the machine with the new
    partition mounted under /home, that all my git repos will be okay.

Under most circumstances, git will do the right thing.  Only if you
use specific flags on clone (like --shared or --reference) might
something go wrong, and as the manual page explains, you can use
git-repack to undo these.

The real solution is, after you rsync, to:

for f in */.git; do (cd $f; git fsck >&/dev/null || echo "$f is BAD!!"); done

If you get no output, you should be golden.  If you get output, check
to make sure the repo you are copying from is good as well.

					-Seth Robertson

Re: rsync a *bunch* of git repos

[Philip Oakley]

From: "Seth Robertson" <in-gitvger@baka.org> Sent: Tuesday, January 10, 2012 
9:24 PM
> In message <20120110211548.GD10255@titan.lakedaemon.net>, Jason writes:
>
>    The nuts and bolts of this aren't difficult, the problem is I don't 
> have
>    a complete understanding of how git stores data.  I've heard in the
>    past that it uses a lot of hardlinks and softlinks.  I need to make
>    sure, that once I transfer the data, and reboot the machine with the 
> new
>    partition mounted under /home, that all my git repos will be okay.
>
> Under most circumstances, git will do the right thing.  Only if you
> use specific flags on clone (like --shared or --reference) might
> something go wrong, and as the manual page explains, you can use
> git-repack to undo these.

I think there is an exception for certain git submodule setups with local 
repos, where the gitdir link is hard coded to the absolute machine path. 
There has been some discussion on the list recently about trying to cover 
this case.

>
> The real solution is, after you rsync, to:
>
> for f in */.git; do (cd $f; git fsck >&/dev/null || echo "$f is BAD!!"); 
> done
>
> If you get no output, you should be golden.  If you get output, check
> to make sure the repo you are copying from is good as well.
>
> -Seth Robertson

Re: rsync a *bunch* of git repos

[Junio C Hamano]

"Philip Oakley" <philipoakley@iee.org> writes:

> I think there is an exception for certain git submodule setups with
> local repos, where the gitdir link is hard coded to the absolute
> machine path. There has been some discussion on the list recently
> about trying to cover this case.

But if you read the original post carefully, all repositories involved are
under /home before the transition, and they will move to the exact same
place under /home anyway, so I do not think the discussion you have in
mind would apply to this case.

The only thing that is needed after the move would be to run

	git update-index --refresh

in all of the repositories, I think.

Re: rsync a *bunch* of git repos

[Sergio]

Junio C Hamano <gitster <at> pobox.com> writes:

> 
> "Philip Oakley" <philipoakley <at> iee.org> writes:
> 
> > I think there is an exception for certain git submodule setups with
> > local repos, where the gitdir link is hard coded to the absolute
> > machine path. There has been some discussion on the list recently
> > about trying to cover this case.
> 
> But if you read the original post carefully, all repositories involved are
> under /home before the transition, and they will move to the exact same
> place under /home anyway, so I do not think the discussion you have in
> mind would apply to this case.
> 
> The only thing that is needed after the move would be to run
> 
> 	git update-index --refresh
> 
> in all of the repositories, I think.
> 
> 

I keep a desktop and a laptop in sync with unison (which uses rsync) and I
confirm that git is entirely happy about that.

the git update-index --refresh is necessary, otherwise git status will
incorrectly report dirty trees.

I wonder if it would not make sense to incorporate the update-index --refresh in
the git status command.

As an aside: git works fine when repos are transferred with rsync, but git packs
are not rsync friendly nor friendly with backup strategies using binary deltas.

Re: rsync a *bunch* of git repos

[Neal Kreitzinger]

On 1/11/2012 7:22 AM, Sergio wrote:

>
> As an aside: git works fine when repos are transferred with rsync, but git packs
> are not rsync friendly nor friendly with backup strategies using binary deltas.
>
All these answers are assuming that your source git repos are quiet 
during the rsync.  If they are not quiet during the rsync then you are 
going to need a more tailored rsync to do things in a certain order. 
See this thread for details: 
http://thread.gmane.org/gmane.comp.version-control.git/168699

v/r,
neal

Re: rsync a *bunch* of git repos

[Jason]

Hi Seth,

On Tue, Jan 10, 2012 at 04:24:50PM -0500, Seth Robertson wrote:
> 
> In message <20120110211548.GD10255@titan.lakedaemon.net>, Jason writes:
> 
>     The nuts and bolts of this aren't difficult, the problem is I don't have
>     a complete understanding of how git stores data.  I've heard in the
>     past that it uses a lot of hardlinks and softlinks.  I need to make
>     sure, that once I transfer the data, and reboot the machine with the new
>     partition mounted under /home, that all my git repos will be okay.
> 
> Under most circumstances, git will do the right thing.  Only if you
> use specific flags on clone (like --shared or --reference) might
> something go wrong, and as the manual page explains, you can use
> git-repack to undo these.
> 
> The real solution is, after you rsync, to:
> 
> for f in */.git; do (cd $f; git fsck >&/dev/null || echo "$f is BAD!!"); done

Okay, I've been using the new system for a week or so now, and
everything looks good.  Thanks!

Before the transfer, I went ahead and did the above to make sure the
repos were good to begin with.  All 279 of them were fine, both before
and after.

thx,

Jason.

Re: rsync a *bunch* of git repos

[Andreas T.Auer]

On 10.01.2012 22:15 Jason wrote:
> The nuts and bolts of this aren't difficult, the problem is I don't have
> a complete understanding of how git stores data.  I've heard in the
> past that it uses a lot of hardlinks and softlinks.  I need to make
> sure, that once I transfer the data, and reboot the machine with the new
> partition mounted under /home, that all my git repos will be okay.
>
>    
As far as I know git uses hardlinks to save diskspace, but it doesn't 
rely on hardlinks to work. It also works with filesystems that don't 
support hardlinks.
But without hardlinks you will probably waste disk space so I recommend 
the -H parameter to rsync, which preserve hardlinks, if possible. If you 
have hundreds of git repos that are cloned from each other, than this 
should make a difference.