A system administration use case for git

A system administration use case for git

[Jon Seymour]

Hello all, it's been quite a while.

And, no, I don't care that you ripped out git-rev-list --merge-order,
it was fun while it lasted and it was still a cool  (if under
appreciated)  algorithm :-)

I've got a system administration use case that I know git does 99.9%
of. I am wondering if the last 0.1%.

It'd be nice to do this on arbitrary (non-git-controlled) file system tree:

* calculate the git hashes of the tree (without making copies of the
files in the tree)
* archive the tree hashes
* rsync the tree hashes to another place
* work out which files aren't available in the other place's git repo
* rsync those files the the other place

Is there a way to easily achieve this with git's existing tool set or
a higher layer?

jon.

Re: A system administration use case for git

[Martin Langhoff]

On Wed, Apr 22, 2009 at 10:33 AM, Jon Seymour <jon.seymour@gmail.com> wrote:
> Hello all, it's been quite a while.
> It'd be nice to do this on arbitrary (non-git-controlled) file system tree:
>
> * calculate the git hashes of the tree (without making copies of the
> files in the tree)
> * archive the tree hashes
> * rsync the tree hashes to another place
> * work out which files aren't available in the other place's git repo
> * rsync those files the the other place

Steps:

1 - rsync to the "other place"
2 - use the git repo in that "other place"
3 - if the tree hashes are needed "here", copy them from the "other
place" git to here.

rsync + git = awesomenesssssss




-- 
 martin.langhoff@gmail.com
 martin@laptop.org -- School Server Architect
 - ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 - http://wiki.laptop.org/go/User:Martinlanghoff

A system administration use case for git

[Jon Seymour]

Martin,

One disadvantage of that approach is if the file system is very large
and only has a few deltas, then I effectively have to have two copies
of the reference file system - one in the GIT repo and one that I can
damage on a regular basis with rsync for the purposes calculating the
git deltas.   I know I can then use git to repair the damage, but then
the reference has to be protected from concurrent access.

I'd prefer not to have to maintain a copy of the tree that has to be
"damaged" on a regular basis and also if I didn't have to maintain a
copy of the git objects for identical files.

In an ideal world, storage requirements at the other place would be
those of the reference file system + those of the various deltas, but
no more.

jon seymour

On Wed, Apr 22, 2009 at 6:48 PM, Martin Langhoff
<martin.langhoff@gmail.com> wrote:
> On Wed, Apr 22, 2009 at 10:33 AM, Jon Seymour <jon.seymour@gmail.com> wrote:
>> Hello all, it's been quite a while.
>> It'd be nice to do this on arbitrary (non-git-controlled) file system tree:
>>
>> * calculate the git hashes of the tree (without making copies of the
>> files in the tree)
>> * archive the tree hashes
>> * rsync the tree hashes to another place
>> * work out which files aren't available in the other place's git repo
>> * rsync those files the the other place
>
> Steps:
>
> 1 - rsync to the "other place"
> 2 - use the git repo in that "other place"
> 3 - if the tree hashes are needed "here", copy them from the "other
> place" git to here.
>
> rsync + git = awesomenesssssss
>
>
>
>
> --
>  martin.langhoff@gmail.com
>  martin@laptop.org -- School Server Architect
>  - ask interesting questions
>  - don't get distracted with shiny stuff  - working code first
>  - http://wiki.laptop.org/go/User:Martinlanghoff
>

Re: A system administration use case for git

[Martin Langhoff]

On Wed, Apr 22, 2009 at 11:22 AM, Jon Seymour <jon.seymour@gmail.com> wrote:
> One disadvantage of that approach is if the file system is very large
> and only has a few deltas, then I effectively have to have two copies
> of the reference file system - one in the GIT repo and one that I can

You could minimise the on-disk footprint -- and protect it from
concurrent access (concurrent change) by using a hardlinked tree on
the destination side. rsync knows to break hardlinks, etc.

Currently, you can't "rsync into git" which would save you that step.
It's a ton of work to do that -- if anyone is planning on working on
something like that, perhaps writing directly into the fast-import
protocol is a good shortcut.

I'd like to have something like that for my OLPC School Server, which
could benefit from using git as the backup backend -- it currently
uses hardlinked directories.

> In an ideal world, storage requirements at the other place would be
> those of the reference file system + those of the various deltas, but
> no more.

rsync + hardlinked trees + git gets you quite close to that.

cheers,



m
-- 
 martin.langhoff@gmail.com
 martin@laptop.org -- School Server Architect
 - ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 - http://wiki.laptop.org/go/User:Martinlanghoff

Re: A system administration use case for git

[Jon Seymour]

Martin,

Thanks for the info about hard-linked trees...they may well do exactly
what I need - thank you!

jon.

On Wed, Apr 22, 2009 at 8:07 PM, Martin Langhoff
<martin.langhoff@gmail.com> wrote:
> On Wed, Apr 22, 2009 at 11:22 AM, Jon Seymour <jon.seymour@gmail.com> wrote:
>> One disadvantage of that approach is if the file system is very large
>> and only has a few deltas, then I effectively have to have two copies
>> of the reference file system - one in the GIT repo and one that I can
>
> You could minimise the on-disk footprint -- and protect it from
> concurrent access (concurrent change) by using a hardlinked tree on
> the destination side. rsync knows to break hardlinks, etc.
>
> Currently, you can't "rsync into git" which would save you that step.
> It's a ton of work to do that -- if anyone is planning on working on
> something like that, perhaps writing directly into the fast-import
> protocol is a good shortcut.
>
> I'd like to have something like that for my OLPC School Server, which
> could benefit from using git as the backup backend -- it currently
> uses hardlinked directories.
>
>> In an ideal world, storage requirements at the other place would be
>> those of the reference file system + those of the various deltas, but
>> no more.
>
> rsync + hardlinked trees + git gets you quite close to that.
>
> cheers,
>
>
>
> m
> --
>  martin.langhoff@gmail.com
>  martin@laptop.org -- School Server Architect
>  - ask interesting questions
>  - don't get distracted with shiny stuff  - working code first
>  - http://wiki.laptop.org/go/User:Martinlanghoff
>

Re: A system administration use case for git

[Uwe Kleine-König]

Hello Jon,

On Wed, Apr 22, 2009 at 06:33:12PM +1000, Jon Seymour wrote:
> * calculate the git hashes of the tree (without making copies of the
> files in the tree)
You can hand-craft a tree object:

	for f in $filelist; do
		printf "100644 %s\x00" $f;

		git hash-object $f |
			perl -n -e 'chomp; for $c (split(/(.{2})/)) { printf("%c", hex($c)) if $c }';
	done |
		git hash-object -t tree --stdin


You get some extra points for directories that include subdirs :-)

There is a problem though, I think it has to do with the order of the
files in the tree object.  Here is a real-life example from the Linux
kernel (I choosed the usr subdirectory, because it's small and doesn't
contain subdirs):

	ukleinek@cepheus:~/gsrc/linux-2.6/usr$ for f in .gitignore Kconfig Makefile gen_init_cpio.c initramfs_data.S initramfs_data.bz2.S initramfs_data.gz.S initramfs_data.lzma.S; do printf "100644 %s\x00" $f; git hash-object $f | perl -n -e 'chomp; for $c (split(/(.{2})/)) { printf("%c", hex($c)) if $c }'; done | git hash-object -t tree --stdin
	64f2ca854bc19fd29479b198be11beba56a26b1e

	ukleinek@cepheus:~/gsrc/linux-2.6/usr$ git rev-parse HEAD:usr
	64f2ca854bc19fd29479b198be11beba56a26b1e

There is a practical problem though:  The filelist has to be sorted in a
way that is not provided by ls, so:

	ukleinek@cepheus:~/gsrc/linux-2.6/usr$ for f in $(ls -A); do printf "100644 %s\x00" $f; git hash-object $f | perl -n -e 'chomp; for $c (split(/(.{2})/)) { printf("%c", hex($c)) if $c }'; done | git hash-object -t tree -w --stdin
	a0a6efb3f1de956badc7607c7d372cc325a18846

	ukleinek@cepheus:~/gsrc/linux-2.6/usr$ git ls-tree a0a6efb3f1de956badc7607c7d372cc325a18846 | wc -l
	0

doesn't work :-/

This would make a nice plumbing:

	git hash-tree $directory

Best regards
Uwe

-- 
Pengutronix e.K.                              | Uwe Kleine-König            |
Industrial Linux Solutions                    | http://www.pengutronix.de/  |

Re: A system administration use case for git

[Johannes Sixt]

Uwe Kleine-König schrieb:
> There is a practical problem though:  The filelist has to be sorted in a
> way that is not provided by ls, so:
> 
> 	ukleinek@cepheus:~/gsrc/linux-2.6/usr$ for f in $(ls -A); do printf "100644 %s\x00" $f; git hash-object $f | perl -n -e 'chomp; for $c (split(/(.{2})/)) { printf("%c", hex($c)) if $c }'; done | git hash-object -t tree -w --stdin
> 	a0a6efb3f1de956badc7607c7d372cc325a18846

Does ... $(LANG=C ls -A) ... make a difference for you?

But note that this is still not the whole story because, IIUC, in tree
objects directories are sorted as if they had a slash appended; i.e.
directory "foo" is sorted *after* file "foo.c", but *before* file "foo0".

-- Hannes

Re: A system administration use case for git

[Uwe Kleine-König]

Hi Hannes,

On Thu, Apr 23, 2009 at 12:38:09PM +0200, Johannes Sixt wrote:
> Uwe Kleine-König schrieb:
> > There is a practical problem though:  The filelist has to be sorted in a
> > way that is not provided by ls, so:
> > 
> > 	ukleinek@cepheus:~/gsrc/linux-2.6/usr$ for f in $(ls -A); do printf "100644 %s\x00" $f; git hash-object $f | perl -n -e 'chomp; for $c (split(/(.{2})/)) { printf("%c", hex($c)) if $c }'; done | git hash-object -t tree -w --stdin
> > 	a0a6efb3f1de956badc7607c7d372cc325a18846
> 
> Does ... $(LANG=C ls -A) ... make a difference for you?
oh, up to now I thought that C and en_US.UTF-8 use the same sorting.  So
yes, it does it right then.

Best regards and thanks
Uwe

-- 
Pengutronix e.K.                              | Uwe Kleine-König            |
Industrial Linux Solutions                    | http://www.pengutronix.de/  |