[PATCH] Teach receive-pack how to keep pack files when unpacklooseobjects = 0.

[PATCH] Teach receive-pack how to keep pack files when unpacklooseobjects = 0.

[Shawn Pearce]

Since keeping a pushed pack or exploding it into loose objects should
be a local repository decision this teaches receive-pack to decide
if it should call unpack-objects or index-pack --stdin --fix-thin
based on the setting of receive.unpackLooseObjects.

If receive.unpackLooseObjects is true (which it is by default for
now) then we unpack-objects as we have in the past.

If it is false then we call index-pack and ask it to include our
pid and hostname in the .keep file to make it easier to identify
why a given pack has been kept in the repository.

Currently this leaves every received pack as a kept pack.  We really
don't want that as received packs will tend to be small.  Instead we
want to delete the .keep file automatically after all refs have
been updated.  That is being left as room for future improvement.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
---
 Documentation/config.txt |    9 ++++++++-
 cache.h                  |    1 +
 receive-pack.c           |   35 ++++++++++++++++++++++++++++++++---
 sha1_file.c              |    2 +-
 4 files changed, 42 insertions(+), 5 deletions(-)

diff --git a/Documentation/config.txt b/Documentation/config.txt
index d9e73da..4eab9e8 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -301,7 +301,14 @@ imap::
 	The configuration variables in the 'imap' section are described
 	in gitlink:git-imap-send[1].
 
-receive.denyNonFastforwads::
+receive.unpackLooseObjects::
+	If set to true, git-receive-pack will unpack each received
+	object into a loose object in the repository.  If set to
+	false then the received pack file will be kept as is (but
+	may have delta bases appended onto the end).  Large pushes
+	into a repository will generally go faster if this is false.
+
+receive.denyNonFastForwards::
 	If set to true, git-receive-pack will deny a ref update which is
 	not a fast forward. Use this to prevent such an update via a push,
 	even if that push is forced. This configuration variable is
diff --git a/cache.h b/cache.h
index e997a85..6cb7e1d 100644
--- a/cache.h
+++ b/cache.h
@@ -376,6 +376,7 @@ extern struct packed_git *parse_pack_ind
 						char *idx_path);
 
 extern void prepare_packed_git(void);
+extern void reprepare_packed_git(void);
 extern void install_packed_git(struct packed_git *pack);
 
 extern struct packed_git *find_sha1_pack(const unsigned char *sha1, 
diff --git a/receive-pack.c b/receive-pack.c
index 7e154c5..6e377f0 100644
--- a/receive-pack.c
+++ b/receive-pack.c
@@ -7,9 +7,8 @@
 
 static const char receive_pack_usage[] = "git-receive-pack <git-dir>";
 
-static const char *unpacker[] = { "unpack-objects", NULL };
-
 static int deny_non_fast_forwards = 0;
+static int unpack_loose_objects = 1;
 static int report_status;
 
 static char capabilities[] = "report-status";
@@ -25,6 +24,12 @@ static int receive_pack_config(const cha
 		return 0;
 	}
 
+	if (strcmp(var, "receive.unpacklooseobjects") == 0)
+	{
+		unpack_loose_objects = git_config_bool(var, value);
+		return 0;
+	}
+
 	return 0;
 }
 
@@ -229,7 +234,31 @@ static void read_head_info(void)
 
 static const char *unpack()
 {
-	int code = run_command_v_opt(1, unpacker, RUN_GIT_CMD);
+	static const char *unpacker[] = { "unpack-objects", NULL };
+	int code;
+	
+	if (unpack_loose_objects)
+		code = run_command_v_opt(1, unpacker, RUN_GIT_CMD);
+	else {
+		const char *keeper[5], *why_kept;
+		char my_host[255], keep_arg[128 + 255];
+
+		if (gethostname(my_host, sizeof(my_host)))
+			strcpy(my_host, "localhost");
+		snprintf(keep_arg, sizeof(keep_why),
+			"--keep=receive-pack %i on %s",
+			getpid(), my_host);
+		why_kept = keep_arg + 7;
+
+		keeper[0] = "index-pack";
+		keeper[1] = "--stdin";
+		keeper[2] = "--fix-thin";
+		keeper[3] = keep_arg;
+		keeper[4] = NULL;
+		code = run_command_v_opt(1, keeper, RUN_GIT_CMD);
+		if (!code)
+			reprepare_packed_git();
+	}
 
 	switch (code) {
 	case 0:
diff --git a/sha1_file.c b/sha1_file.c
index 5e6c8b8..7bda2d4 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -663,7 +663,7 @@ void prepare_packed_git(void)
 	prepare_packed_git_run_once = 1;
 }
 
-static void reprepare_packed_git(void)
+void reprepare_packed_git(void)
 {
 	prepare_packed_git_run_once = 0;
 	prepare_packed_git();
-- 
1.4.3.3.g7d63

Re: [PATCH] Teach receive-pack how to keep pack files when unpacklooseobjects = 0.

[Junio C Hamano]

Shawn Pearce <spearce@spearce.org> writes:

> Currently this leaves every received pack as a kept pack.  We really
> don't want that as received packs will tend to be small.

Until this is resolved, I feel there should be some way to
control the behaviour, so while I agree with the general
direction, I think the patch to revert the sender's wish should
come at the end.

Re: [PATCH] Teach receive-pack how to keep pack files when unpacklooseobjects = 0.

[Junio C Hamano]

Shawn Pearce <spearce@spearce.org> writes:

> Since keeping a pushed pack or exploding it into loose objects should
> be a local repository decision this teaches receive-pack to decide
> if it should call unpack-objects or index-pack --stdin --fix-thin
> based on the setting of receive.unpackLooseObjects.

One thing you can cheaply do is to tell the number of new
commits that is coming to receive-pack from send-pack when it
sends the old..new pairs before it sends the packfile payload.
It would be just a single internal rev-list call inside
send-pack, which is reasonably cheap.

If the receiving end knows how to process that new information
(invent a "send-count" protocol extension and send it just like
we already send "report-status" request), send one extra packet
after flushing the list of old..new from send-pack to
receive-pack, to tell what the number of commits are, and make a
matching change in receive-pack.

Then, instead of receive.unpackLooseObjects being a boolean, you
can have it as a ceiling to decide if you have more than 100
commits you would keep it packed and otherwise you would
explode.  That would be very specific to the projects' size
(width of the tree) and style (huge commits vs lots of small
changes).

Re: [PATCH] Teach receive-pack how to keep pack files when unpacklooseobjects = 0.

[Nicolas Pitre]

On Mon, 30 Oct 2006, Junio C Hamano wrote:

> One thing you can cheaply do is to tell the number of new
> commits that is coming to receive-pack from send-pack when it
> sends the old..new pairs before it sends the packfile payload.
> It would be just a single internal rev-list call inside
> send-pack, which is reasonably cheap.

Well, it could even be avoided.

> If the receiving end knows how to process that new information
> (invent a "send-count" protocol extension and send it just like
> we already send "report-status" request), send one extra packet
> after flushing the list of old..new from send-pack to
> receive-pack, to tell what the number of commits are, and make a
> matching change in receive-pack.

I don't like this much since a commit can carry along very little or 
very large amount of objects.  You really want to decide on the number 
of objects.

Why not just parse the pack header in receive-pack / fetch-pack, and 
decide on the first-hand information?  Sure the pack header is then 
gone, but then the only thing that is needed is an extra flag to both 
unpack-objects and index-pack to tell them that we've already parsed the 
pack header and that the pack version is x and the number of objects is 
y.  Simply something like --pack_header=x,y.  No protocol extension 
needed, no extra rev-list, no reliance on the remote server providing 
the needed info.

Re: [PATCH] Teach receive-pack how to keep pack files when unpacklooseobjects = 0.

[Junio C Hamano]

Nicolas Pitre <nico@cam.org> writes:

> Why not just parse the pack header in receive-pack / fetch-pack, and 
> decide on the first-hand information?  Sure the pack header is then 
> gone, but then the only thing that is needed is an extra flag to both 
> unpack-objects and index-pack to tell them that we've already parsed the 
> pack header and that the pack version is x and the number of objects is 
> y.  Simply something like --pack_header=x,y.  No protocol extension 
> needed, no extra rev-list, no reliance on the remote server providing 
> the needed info.

I like it.

Because that approach assumes recieve-pack and unpack-objects
and index-pack are from the same vintage (otherwise your
receive-pack would need to have a way to see if unpack-objects
and index-pack would grok --pack_header argument), we could even
get away without passing the pack version if we wanted to.

Re: [PATCH] Teach receive-pack how to keep pack files when unpacklooseobjects = 0.

[Shawn Pearce]

Junio C Hamano <junkio@cox.net> wrote:
> Until this is resolved, I feel there should be some way to
> control the behaviour, so while I agree with the general
> direction, I think the patch to revert the sender's wish should
> come at the end.
 
I disagree, but I'm not the maintainer.  :-)

I reverted it before making my changes as I was editing the same
lines as c7740a modified.  Therefore reverting c7740a after my
changes would no longer be a clean revert, making it slightly
harder to actuallly do the revert as you would need to manually
fix up receive-pack.c.

-- 

Re: [PATCH] Teach receive-pack how to keep pack files when unpacklooseobjects = 0.

[Shawn Pearce]

Junio C Hamano <junkio@cox.net> wrote:
> Nicolas Pitre <nico@cam.org> writes:
> 
> > Why not just parse the pack header in receive-pack / fetch-pack, and 
> > decide on the first-hand information?  Sure the pack header is then 
> > gone, but then the only thing that is needed is an extra flag to both 
> > unpack-objects and index-pack to tell them that we've already parsed the 
> > pack header and that the pack version is x and the number of objects is 
> > y.  Simply something like --pack_header=x,y.  No protocol extension 
> > needed, no extra rev-list, no reliance on the remote server providing 
> > the needed info.
> 
> I like it.
> 
> Because that approach assumes recieve-pack and unpack-objects
> and index-pack are from the same vintage (otherwise your
> receive-pack would need to have a way to see if unpack-objects
> and index-pack would grok --pack_header argument), we could even
> get away without passing the pack version if we wanted to.

Heh.  On Saturday I almost did exactly what Nico proposes above.
But I thought both of you would find the --pack_header=x,y option
too brittle and would reject the change.

But since all three of us liked the same idea I'll code it up
and resend my receive-pack patch using Nico's suggestion instead.
Hopefully I'll get it out tomorrow.

BTW I think we do need to pass the pack version in the option.
If we ever do increment the pack version its going to be after this
option is introduced so supporting the option does not imply that
the callee is able to parse the pack file without knowing what
version the file is, especially if the callee supports both the
current version and the new version.

-- 

Re: [PATCH] Teach receive-pack how to keep pack files when unpacklooseobjects = 0.

[Junio C Hamano]

Shawn Pearce <spearce@spearce.org> writes:

>> Because that approach assumes recieve-pack and unpack-objects
>> and index-pack are from the same vintage (otherwise your
>> receive-pack would need to have a way to see if unpack-objects
>> and index-pack would grok --pack_header argument), we could even
>> get away without passing the pack version if we wanted to.
>...
> BTW I think we do need to pass the pack version in the option.

We are in agreement; I mentioned "we could ... if we wanted to"
because I wanted to see if you are paying attention ;-).

Re: [PATCH] Teach receive-pack how to keep pack files when unpacklooseobjects = 0.

[Shawn Pearce]

Junio C Hamano <junkio@cox.net> wrote:
> Shawn Pearce <spearce@spearce.org> writes:
> 
> >> Because that approach assumes recieve-pack and unpack-objects
> >> and index-pack are from the same vintage (otherwise your
> >> receive-pack would need to have a way to see if unpack-objects
> >> and index-pack would grok --pack_header argument), we could even
> >> get away without passing the pack version if we wanted to.
> >...
> > BTW I think we do need to pass the pack version in the option.
> 
> We are in agreement; I mentioned "we could ... if we wanted to"
> because I wanted to see if you are paying attention ;-).

Hah.  Trying to keep me on my toes aren't you.  :-)

I almost have the change finished. I'll send it before I go to bed.

--