Re: [PATCH] git-repack: create new packs inside $PACKDIR, not cwd

Re: [PATCH] git-repack: create new packs inside $PACKDIR, not cwd

[Martin Langhoff (CatalystIT)]

Martin Langhoff wrote:

> Avoid failing when cwd is !writable by writing the
> packfiles directly in the $PACKDIR.

Urgh. At the very top of the script there's

rm -f .tmp-pack-*
PACKDIR="$GIT_OBJECT_DIRECTORY/pack"

which relies on cwd again. Hmm.


m
-- 
-----------------------------------------------------------------------
Martin @ Catalyst .Net .NZ  Ltd, PO Box 11-053, Manners St,  Wellington
WEB: http://catalyst.net.nz/           PHYS: Level 2, 150-154 Willis St
OFFICE: +64(4)916-7224                              MOB: +64(21)364-017
       Make things as simple as possible, but no simpler - Einstein
-----------------------------------------------------------------------

-- 
VGER BF report: U 0.804452

[PATCH] git-repack: create new packs inside $PACKDIR, not cwd

[Martin Langhoff]

Avoid failing when cwd is !writable by writing the
packfiles directly in the $PACKDIR.

Without this, git-repack was failing when run from crontab
by non-root user accounts. For large repositories, this
also makes the mv operation a lot cheaper, and avoids leaving
temp packfiles around the fs upon failure.

Signed-off-by: Martin Langhoff <martin@catalyst.net.nz>
---
 git-repack.sh |    9 +++++----
 1 files changed, 5 insertions(+), 4 deletions(-)

diff --git a/git-repack.sh b/git-repack.sh
index 584a732..ccc8e43 100755
--- a/git-repack.sh
+++ b/git-repack.sh
@@ -42,11 +42,13 @@ case ",$all_into_one," in
 	    find . -type f \( -name '*.pack' -o -name '*.idx' \) -print`
 	;;
 esac
+
+mkdir -p "$PACKDIR" || exit
 pack_objects="$pack_objects $local $quiet $no_reuse_delta$extra"
 name=$( { git-rev-list --objects --all $rev_list ||
 	  echo "git-rev-list died with exit code $?"
 	} |
-	git-pack-objects --non-empty $pack_objects .tmp-pack) ||
+	git-pack-objects --non-empty $pack_objects "$PACKDIR/.tmp-pack") ||
 	exit 1
 if [ -z "$name" ]; then
 	echo Nothing new to pack.
@@ -54,7 +56,6 @@ else
 	if test "$quiet" != '-q'; then
 	    echo "Pack pack-$name created."
 	fi
-	mkdir -p "$PACKDIR" || exit
 
 	for sfx in pack idx
 	do
@@ -64,8 +65,8 @@ else
 				"$PACKDIR/old-pack-$name.$sfx"
 		fi
 	done &&
-	mv -f .tmp-pack-$name.pack "$PACKDIR/pack-$name.pack" &&
-	mv -f .tmp-pack-$name.idx  "$PACKDIR/pack-$name.idx" &&
+	mv -f "$PACKDIR/.tmp-pack-$name.pack" "$PACKDIR/pack-$name.pack" &&
+	mv -f "$PACKDIR/.tmp-pack-$name.idx"  "$PACKDIR/pack-$name.idx" &&
 	test -f "$PACKDIR/pack-$name.pack" &&
 	test -f "$PACKDIR/pack-$name.idx" || {
 		echo >&2 "Couldn't replace the existing pack with updated one."
-- 
1.4.2.gdfe7


-- 
VGER BF report: S 1

Re: [PATCH] git-repack: create new packs inside $PACKDIR, not cwd

[Martin Waitz]

[-- Attachment #1: Type: text/plain, Size: 447 bytes --]

hoi :)

On Mon, Sep 04, 2006 at 05:42:32PM +1200, Martin Langhoff wrote:
> Avoid failing when cwd is !writable by writing the
> packfiles directly in the $PACKDIR.

what if other GIT commands are being called while you repack?
Wouldn't they try to use the new packs, even while they are not
completely written?

Perhaps it is better to create a new subdirectory $PACKDIR/.tmp/
and create the new pack files there?

-- 
Martin Waitz

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [PATCH] git-repack: create new packs inside $PACKDIR, not cwd

[Junio C Hamano]

Martin Waitz <tali@admingilde.org> writes:

> hoi :)
>
> On Mon, Sep 04, 2006 at 05:42:32PM +1200, Martin Langhoff wrote:
>> Avoid failing when cwd is !writable by writing the
>> packfiles directly in the $PACKDIR.
>
> what if other GIT commands are being called while you repack?
> Wouldn't they try to use the new packs, even while they are not
> completely written?

Given that a new pack is created first by writing out .pack and
then .idx, and that the using side ignores .pack without
corresponding .idx, we are talking about very small window, but
you are right.

Writing into $cwd was certainly a carelessness; we tend to use
$GIT_DIR/ for this kind of thing.



-- 
VGER BF report: U 0.926108

Re: [PATCH] git-repack: create new packs inside $PACKDIR, not cwd

[Junio C Hamano]

Junio C Hamano <junkio@cox.net> writes:

> Writing into $cwd was certainly a carelessness; we tend to use
> $GIT_DIR/ for this kind of thing.

In other words...

-- >8 --
From: Martin Langhoff <martin@catalyst.net.nz>
Date: Mon, 4 Sep 2006 17:42:32 +1200
Subject: [PATCH] git-repack: create new packs inside $GIT_DIR, not cwd

Avoid failing when cwd is !writable by writing the
packfiles in $GIT_DIR, which is more in line with other commands.

Without this, git-repack was failing when run from crontab
by non-root user accounts. For large repositories, this
also makes the mv operation a lot cheaper, and avoids leaving
temp packfiles around the fs upon failure.

Signed-off-by: Martin Langhoff <martin@catalyst.net.nz>
Signed-off-by: Junio C Hamano <junkio@cox.net>
---
 git-repack.sh |   11 +++++++----
 1 files changed, 7 insertions(+), 4 deletions(-)

diff --git a/git-repack.sh b/git-repack.sh
index 584a732..b525fc5 100755
--- a/git-repack.sh
+++ b/git-repack.sh
@@ -24,8 +24,10 @@ do
 	shift
 done
 
-rm -f .tmp-pack-*
 PACKDIR="$GIT_OBJECT_DIRECTORY/pack"
+PACKTMP="$GIT_DIR/.tmp-$$-pack"
+rm -f "$PACKTMP"-*
+trap 'rm -f "$PACKTMP"-*' 0 1 2 3 15
 
 # There will be more repacking strategies to come...
 case ",$all_into_one," in
@@ -42,11 +44,12 @@ case ",$all_into_one," in
 	    find . -type f \( -name '*.pack' -o -name '*.idx' \) -print`
 	;;
 esac
+
 pack_objects="$pack_objects $local $quiet $no_reuse_delta$extra"
 name=$( { git-rev-list --objects --all $rev_list ||
 	  echo "git-rev-list died with exit code $?"
 	} |
-	git-pack-objects --non-empty $pack_objects .tmp-pack) ||
+	git-pack-objects --non-empty $pack_objects "$PACKTMP") ||
 	exit 1
 if [ -z "$name" ]; then
 	echo Nothing new to pack.
@@ -64,8 +67,8 @@ else
 				"$PACKDIR/old-pack-$name.$sfx"
 		fi
 	done &&
-	mv -f .tmp-pack-$name.pack "$PACKDIR/pack-$name.pack" &&
-	mv -f .tmp-pack-$name.idx  "$PACKDIR/pack-$name.idx" &&
+	mv -f "$PACKTMP-$name.pack" "$PACKDIR/pack-$name.pack" &&
+	mv -f "$PACKTMP-$name.idx"  "$PACKDIR/pack-$name.idx" &&
 	test -f "$PACKDIR/pack-$name.pack" &&
 	test -f "$PACKDIR/pack-$name.idx" || {
 		echo >&2 "Couldn't replace the existing pack with updated one."
-- 
1.4.2.g99d7d



-- 
VGER BF report: U 0.870206

Re: [PATCH] git-repack: create new packs inside $PACKDIR, not cwd

[Martin Langhoff (CatalystIT)]

Junio C Hamano wrote:

> In other words...

Can't be offline 2 hs to read a book... ;-) Actually, I had thought the 
pack reading code would focus on filenames following pack-<id>.pack 
pattern and corresponding idx files, and that .tmp-* was safe to have 
there. My bad.

BTW, I think there's a small error.

...

> --- a/git-repack.sh
> +++ b/git-repack.sh
> @@ -24,8 +24,10 @@ do
>  	shift
>  done
>  
> -rm -f .tmp-pack-*
>  PACKDIR="$GIT_OBJECT_DIRECTORY/pack"
> +PACKTMP="$GIT_DIR/.tmp-$$-pack"
> +rm -f "$PACKTMP"-*
> +trap 'rm -f "$PACKTMP"-*' 0 1 2 3 15

Your packtmp includes $$ which means that rm -f "$PACKTMP" will only 
clear out old packs only if the pid of the old-and-probably-dead process 
matches ours... and then a hyphen.

so instead I propose...

+trap 'rm -f "$GIT_DIR/.tmp-*-pack"' 0 1 2 3 15

cheers,


martin
-- 
-----------------------------------------------------------------------
Martin @ Catalyst .Net .NZ  Ltd, PO Box 11-053, Manners St,  Wellington
WEB: http://catalyst.net.nz/           PHYS: Level 2, 150-154 Willis St
OFFICE: +64(4)916-7224                              MOB: +64(21)364-017
       Make things as simple as possible, but no simpler - Einstein
-----------------------------------------------------------------------

-- 
VGER BF report: U 0.900798

Re: [PATCH] git-repack: create new packs inside $PACKDIR, not cwd

[Junio C Hamano]

"Martin Langhoff (CatalystIT)" <martin@catalyst.net.nz> writes:

> BTW, I think there's a small error.
>
> Your packtmp includes $$ which means that rm -f "$PACKTMP" will only
> clear out old packs..

That was deliberate.  I hate programs that clean things up
behind user's back.  The first "rm" is to get rid of what would
collide with what we are going to do (i.e. protecting ourselves)
and "trap rm" is to make sure we do not leave the cruft we know
we are going to create.  I'd rather leave other people's cruft
around, unless the purpose of the command is to clean things up,
and repack is hardly that.





-- 
VGER BF report: H 0.149712

Re: [PATCH] git-repack: create new packs inside $PACKDIR, not cwd

[Martin Langhoff (CatalystIT)]

Junio C Hamano wrote:

> "Martin Langhoff (CatalystIT)" <martin@catalyst.net.nz> writes:
> 
> 
>>BTW, I think there's a small error.
>>
>>Your packtmp includes $$ which means that rm -f "$PACKTMP" will only
>>clear out old packs..
> 
> 
> That was deliberate.  I hate programs that clean things up
> behind user's back.  The first "rm" is to get rid of what would
> collide with what we are going to do (i.e. protecting ourselves)
> and "trap rm" is to make sure we do not leave the cruft we know
> we are going to create.  I'd rather leave other people's cruft
> around, unless the purpose of the command is to clean things up,
> and repack is hardly that.
> 

Ah, ok. I misunderstood the use of trap -- of course, re-reading the man 
pages, it makes sense.

A-ok with me, then, and sorry about the noise.



martin
-- 
-----------------------------------------------------------------------
Martin @ Catalyst .Net .NZ  Ltd, PO Box 11-053, Manners St,  Wellington
WEB: http://catalyst.net.nz/           PHYS: Level 2, 150-154 Willis St
OFFICE: +64(4)916-7224                              MOB: +64(21)364-017
       Make things as simple as possible, but no simpler - Einstein
-----------------------------------------------------------------------

-- 
VGER BF report: H 0.0618878

Re: [PATCH] git-repack: create new packs inside $PACKDIR, not cwd

[Junio C Hamano]

"Martin Langhoff (CatalystIT)" <martin@catalyst.net.nz> writes:

> Ah, ok. I misunderstood the use of trap -- of course, re-reading the
> man pages, it makes sense.

However the more I think about it your original idea of using
.tmp-pack directly in .git/objects/pack/ *should* have worked
(modulo two repack instances using the same name, which is fixed
by $$ there).  Not insisting on "pack-" prefix I consider is a
feature, but I do not see a reason to take a file that begin
with a dot.

Well, probably too late to deprecate, maybe not.  I dunno.