[RFC] git-fetch - repack in the background after fetching

[RFC] git-fetch - repack in the background after fetching

[Martin Langhoff]

Check whether we have a large set of unpacked objects and repack
after the fetch, but don't for the user to wait for us. Conditional
on core.autorepack =! no.

Having ' handle concurrent pruning of packed objects'
(637cdd9d1d997fca34a1fc668fed1311e30fe95f) from Jeff King it should
be safe to repack and prune in the background.

Signed-off-by: Martin Langhoff <martin@catalyst.net.nz>

---

This is a follow up to a similar patch earlier
http://www.gelato.unsw.edu.au/archives/git/0605/21401.html -- is there 
interest in making GIT more friendly to users who don't know or care
about packing and repacking their repos?

I loathe to do this conditionally only on the count of unpacked
objects. If there's a quick'n'dirty way of asking portably whether
the machine is busy or otherwise resource-constrained (ie: on battery)
it should use it to avoid running repack at inconvenient times.

---
 git-fetch.sh |    9 +++++++++
 1 files changed, 9 insertions(+), 0 deletions(-)

diff --git a/git-fetch.sh b/git-fetch.sh
index 48818f8..7211318 100755
--- a/git-fetch.sh
+++ b/git-fetch.sh
@@ -427,3 +427,12 @@ case ",$update_head_ok,$orig_head," in
 	fi
 	;;
 esac
+
+if test "$(git-repo-config --get core.autorepack)" != 'no'
+then
+	if test $(git rev-list --unpacked --all | wc -l) -gt 1000
+	then
+		echo "Repacking in the background"
+		nice git repack -a -d -q &
+	fi
+fi
-- 
1.4.1.rc1.g59c8

Re: [RFC] git-fetch - repack in the background after fetching

[Junio C Hamano]

Martin Langhoff <martin@catalyst.net.nz> writes:

> This is a follow up to a similar patch earlier
> http://www.gelato.unsw.edu.au/archives/git/0605/21401.html -- is there 
> interest in making GIT more friendly to users who don't know or care
> about packing and repacking their repos?

I would be a bit worried about the niced background repack
racing against another instance of itself spawned by the same
parent.

> I loathe to do this conditionally only on the count of unpacked
> objects. If there's a quick'n'dirty way of asking portably whether
> the machine is busy or otherwise resource-constrained (ie: on battery)
> it should use it to avoid running repack at inconvenient times.

count-objects might be lighter weight than rev-list --unpacked.

If you mean to make core.autorepack to be boolean, checking for
string 'no' is not the right way.

	git repo-config --bool --get core.autorepack

But it does not matter if that variable is a string that is
almost always true unless the value is "no".

[PATCH] Repack should try to prevent itself from running twice, concurrently.

[Ryan Anderson]

Signed-off-by: Ryan Anderson <ryan@michonline.com>
---
 git-repack.sh |   11 +++++++++++
 1 files changed, 11 insertions(+), 0 deletions(-)

diff --git a/git-repack.sh b/git-repack.sh
index eb75c8c..20f9b55 100755
--- a/git-repack.sh
+++ b/git-repack.sh
@@ -24,6 +24,15 @@ do
 	shift
 done
 
+if [ -f $GIT_DIR/repack.lock ]
+then
+	echo "Existing repack job appears to be running."
+	echo "Remove $GIT_DIR/repack.lock if this is not the case."
+	exit 1
+else
+	echo $$ > $GIT_DIR/repack.lock
+fi
+
 rm -f .tmp-pack-*
 PACKDIR="$GIT_OBJECT_DIRECTORY/pack"
 
@@ -83,3 +92,5 @@ case "$no_update_info" in
 t) : ;;
 *) git-update-server-info ;;
 esac
+
+rm $GIT_DIR/repack.lock
-- 
1.4.1.rc1.gacb70

Re: [PATCH] Repack should try to prevent itself from running twice, concurrently.

[Johannes Schindelin]

Hi,

On Sun, 25 Jun 2006, Ryan Anderson wrote:

> +if [ -f $GIT_DIR/repack.lock ]
> +then
> +	echo "Existing repack job appears to be running."
> +	echo "Remove $GIT_DIR/repack.lock if this is not the case."
> +	exit 1
> +else
> +	echo $$ > $GIT_DIR/repack.lock
> +fi

It is not like it is being an atomic operation, but then, we are not going 
to call repack multiple times a second. I'd say it is sufficient.

Ciao,
Dscho

Re: [RFC] git-fetch - repack in the background after fetching

[Linus Torvalds]

On Sat, 24 Jun 2006, Martin Langhoff wrote:
>
> Check whether we have a large set of unpacked objects and repack
> after the fetch, but don't for the user to wait for us. Conditional
> on core.autorepack =! no.

I don't think this is safe.

It's also done stupidly.

Instead of askign how many unpacked objects we have with the (expensive) 
git-rev-list, why not just do

	ls "$GIT_DIR/objects/00" | wc -l

which is pretty much guaranteed to be faster and easier.

However, the more worrisome thing about background repacking is that while 
it should be safe against normal users, if you have two _repacks_ at the 
same time, they can decide to remove each others packs. Yeah, yeah, that's 
pretty damn unlikely, but hey, "pretty damn unlikely" is not "impossible".

Also, I think you'd want to repack with "-l", in case the thing is set up 
with an alternate object directory.

			Linus

Re: [RFC] git-fetch - repack in the background after fetching

[Johannes Schindelin]

Hi,

On Sat, 24 Jun 2006, Linus Torvalds wrote:

> However, the more worrisome thing about background repacking is that while 
> it should be safe against normal users, if you have two _repacks_ at the 
> same time, they can decide to remove each others packs. Yeah, yeah, that's 
> pretty damn unlikely, but hey, "pretty damn unlikely" is not "impossible".

Why not introduce a lock file for repack?

Ciao,
Dscho

Re: [RFC] git-fetch - repack in the background after fetching

[Linus Torvalds]

On Sun, 25 Jun 2006, Johannes Schindelin wrote:
> 
> On Sat, 24 Jun 2006, Linus Torvalds wrote:
> 
> > However, the more worrisome thing about background repacking is that while 
> > it should be safe against normal users, if you have two _repacks_ at the 
> > same time, they can decide to remove each others packs. Yeah, yeah, that's 
> > pretty damn unlikely, but hey, "pretty damn unlikely" is not "impossible".
> 
> Why not introduce a lock file for repack?

You can do that. The problem is, lock-files are really hard to do 
right, and portably. Especially from scripts.

But _I_ think the basic issue is that it's wrong to even try to do this 
background repack.

Git does explicit repacking. That's just how it is. If the worry is that 
people forget to pack often enough, why not just have the "git pull" 
script _tell_ the user, something like

	if [lots of unpacked objects]; then
		echo "You've got a boatload of unpacked objects now."
		echo "Maybe you'd like to repack using"
		echo "   git repack -a -d"
		echo "Thank you for not smoking"
	fi >&2

which is educational on so many levels.

		Linus