Recent and near future backward incompatibilities

Recent and near future backward incompatibilities

[Junio C Hamano]

It was brought to my attention that the public git.git
repository cannot be cloned with older versions of git.  More
precisely, packs generated with post 16854571 (NOT contained in
v1.4.2.3 but in the current "master" and more importantly in
v1.4.3-rc3 which I tagged tonight) can contain deltas that are
not compatible with the version of git before d60fc1c8, which
means that v1.1.6 and older (v1.2.0 and later are Ok).

The older version of git did not know anything about version 3
delta (which can express larger copies from source when
computing a delta) before d60fc1c8, and barfs if it is fed a
pack that contains such a delta.  We have generated only version
2 delta for quite a while until very recently post v1.4.2.3.

The thing is, I made a mistake to repack my public repository
with more recent git (I work with "next" usually, and with
"master" after we go into -rcX cycle to prepare for the
release).  Although the version of git that kernel.org runs is
still v1.4.2.3 which means its pack-objects does not produce
version 3 delta by itself, the problem is that the delta reuse
logic happily copies out whatever delta is in existing packs.

I already repacked the public repository with an older git,
v1.4.2.3, using 'git-repack -a -d -f' to fix this problem.

One thing we can and should immediately do is to revert 16854571
for now until we decide how to resolve this issue cleanly.

These are what needs to happen but one of them is quite tricky:

 - the reusing of delta is what makes pack-objects practical,
   and it is expensive to look into existing delta to see if it
   is version 2 or version 3 before deciding to reuse each delta
   data, so even if we update pack-objects so that we can tell
   it to generate a pack that contains only version 2 deltas, it
   would be very expensive to do so and may not be practical.  I
   am not sure how to resolve this issue efficiently right now;
   we need a bit of thinking.

 - so instead we could just say a public repository that needs
   to interoperate with older clients should not keep packs with
   version 3 delta, at least for now.  deltifying from loose
   objects are done afresh every time pack-objects is run, and
   it is easier to control.

 - git-pack-objects needs to be updated so that we can control
   whether we generate version 2 or version 3 delta.

 - we need to add .git/config item that tells pack-objects to
   never generate version 3 delta for that particular
   repository.  This is similar to the way we would need to
   control the use of delta-base-offset representation currently
   cooking in "next".

 - the on-the-wire protocol between fetch-pack and upload-pack
   needs to be updated so that the server side can tell if the
   client side is prepared to handle version 3 delta in the
   pack, and this needs to be passed to pack-objects to control
   its operation.  This part I can see how to fix.

We may have a similar issue when enabling generation of loose
objects with new style headers.  This is already controlled with
the core.legacyheaders configuration item.

[PATCH] pack-objects: use of version 3 delta is now optional.

[Junio C Hamano]

This introduces a new configuration item, pack.deltaversion, to
control whether pack-objects is allowed to use version 3 delta.
By default, we keep generating version 2 delta (and version 2
packfile format) to be compatible with git earlier than v1.2.0.

This configuration affects the command in the following ways:

 - the resulting packfile will have the specified version;

 - when generating delta, larger copies are allowed only when
   deltaversion is 3;

 - the logic to reuse delta from existing packs refuses to reuse
   delta from packs that uses delta version 3 when the
   configuration is set to 2.

Signed-off-by: Junio C Hamano <junkio@cox.net>
---

 * Nico, I'd really appreciate it if you can eyeball this
   patch.  For the upcoming v1.4.3, we'd at least want to
   revert:

     commit 16854571aae6302f457c5fbee41ac64669b09595
     Author: Nicolas Pitre <nico@cam.org>
     Date:   Thu Sep 21 00:11:59 2006 -0400

     move pack creation to version 3

     It's been quite a while now that GIT is able to read version 3 packs.
     Let's create them at last.

     Signed-off-by: Nicolas Pitre <nico@cam.org>
     Signed-off-by: Junio C Hamano <junkio@cox.net>

   but I'd like to allow people to explicitly tell what version
   is to be created by default per repository to avoid future
   problems.

   For now it would be advised to leave this configuration empty
   in public repositories, which would cause them to have
   version 2 packs.  People who know their repositories are only
   used with git v1.2.0 or newer can use the configuration to
   allow version 3 packs.

   We should later add a command line override to pack-objects
   to explicitly say which delta version to be used, so that
   pack protocol can negotiate the allowed delta version between
   client and the server and pass that command line option when
   upload-pack runs pack-objects.

 builtin-pack-objects.c |   16 ++++++++++++----
 cache.h                |    1 +
 delta.h                |    2 ++
 diff-delta.c           |   15 +++++++++++++--
 environment.c          |    3 +++
 sha1_file.c            |    1 +
 6 files changed, 32 insertions(+), 6 deletions(-)

diff --git a/builtin-pack-objects.c b/builtin-pack-objects.c
index 96c069a..4d2147b 100644
--- a/builtin-pack-objects.c
+++ b/builtin-pack-objects.c
@@ -456,7 +456,7 @@ static void write_pack_file(void)
 		fprintf(stderr, "Writing %d objects.\n", nr_result);
 
 	hdr.hdr_signature = htonl(PACK_SIGNATURE);
-	hdr.hdr_version = htonl(PACK_VERSION);
+	hdr.hdr_version = htonl(delta_version);
 	hdr.hdr_entries = htonl(nr_result);
 	sha1write(f, &hdr, sizeof(hdr));
 	offset = sizeof(hdr);
@@ -914,12 +914,15 @@ static void check_object(struct object_e
 		/* Check if it is delta, and the base is also an object
 		 * we are going to pack.  If so we will reuse the existing
 		 * delta.
+		 *
+		 * Also make sure that we do not reuse delta from an existing
+		 * pack that uses higher delta version than allowed.
 		 */
 		if (!no_reuse_delta &&
 		    entry->in_pack_type == OBJ_DELTA &&
 		    (base_entry = locate_object_entry(base)) &&
-		    (!base_entry->preferred_base)) {
-
+		    (!base_entry->preferred_base) &&
+		    entry->in_pack->pack_version <= delta_version) {
 			/* Depth value does not matter - find_deltas()
 			 * will never consider reused delta as the
 			 * base object to deltify other objects
@@ -1326,10 +1329,15 @@ static void setup_progress_signal(void)
 
 static int git_pack_config(const char *k, const char *v)
 {
-	if(!strcmp(k, "pack.window")) {
+	if (!strcmp(k, "pack.window")) {
 		window = git_config_int(k, v);
 		return 0;
 	}
+	if (!strcmp(k, "pack.deltaversion")) {
+		delta_version = git_config_int(k, v);
+		if (!pack_version_ok(htonl(delta_version)))
+			die("value %s for '%s' not allowed", v, k);
+	}
 	return git_default_config(k, v);
 }
 
diff --git a/cache.h b/cache.h
index c354701..724c09a 100644
--- a/cache.h
+++ b/cache.h
@@ -337,6 +337,7 @@ extern struct packed_git {
 	unsigned int pack_last_used;
 	unsigned int pack_use_cnt;
 	int pack_local;
+	int pack_version;
 	unsigned char sha1[20];
 	/* something like ".git/objects/pack/xxxxx.pack" */
 	char pack_name[FLEX_ARRAY]; /* more */
diff --git a/delta.h b/delta.h
index 7b3f86d..55af3d2 100644
--- a/delta.h
+++ b/delta.h
@@ -1,6 +1,8 @@
 #ifndef DELTA_H
 #define DELTA_H
 
+extern int delta_version;
+
 /* opaque object for delta index */
 struct delta_index;
 
diff --git a/diff-delta.c b/diff-delta.c
index fa16d06..2f6dcfb 100644
--- a/diff-delta.c
+++ b/diff-delta.c
@@ -253,10 +253,13 @@ create_delta(const struct delta_index *i
 	int inscnt;
 	const unsigned char *ref_data, *ref_top, *data, *top;
 	unsigned char *out;
+	unsigned int ref_size_limit;
 
 	if (!trg_buf || !trg_size)
 		return NULL;
 
+	ref_size_limit = (delta_version > 2) ? 0xffffff : 0x10000;
+
 	outpos = 0;
 	outsize = 8192;
 	if (max_size && outsize >= max_size)
@@ -308,8 +311,8 @@ create_delta(const struct delta_index *i
 				continue;
 			if (ref_size > top - src)
 				ref_size = top - src;
-			if (ref_size > 0x10000)
-				ref_size = 0x10000;
+			if (ref_size > ref_size_limit)
+				ref_size = ref_size_limit;
 			if (ref_size <= msize)
 				break;
 			while (ref_size-- && *src++ == *ref)
@@ -318,6 +321,8 @@ create_delta(const struct delta_index *i
 				/* this is our best match so far */
 				msize = ref - entry->ptr;
 				moff = entry->ptr - ref_data;
+				if (delta_version > 2 && msize >= 0x10000)
+					break; /* this is good enough */
 			}
 		}
 
@@ -381,6 +386,12 @@ create_delta(const struct delta_index *i
 			if (msize & 0xff) { out[outpos++] = msize; i |= 0x10; }
 			msize >>= 8;
 			if (msize & 0xff) { out[outpos++] = msize; i |= 0x20; }
+			if (delta_version > 2) {
+				msize >>= 8;
+				if (msize & 0xff) {
+					out[outpos++] = msize; i |= 0x40;
+				}
+			}
 
 			*op = i;
 		}
diff --git a/environment.c b/environment.c
index 63b1d15..e266f83 100644
--- a/environment.c
+++ b/environment.c
@@ -8,6 +8,7 @@
  * are.
  */
 #include "cache.h"
+#include "pack.h"
 
 char git_default_email[MAX_GITNAME];
 char git_default_name[MAX_GITNAME];
@@ -25,6 +26,8 @@ const char *apply_default_whitespace;
 int zlib_compression_level = Z_DEFAULT_COMPRESSION;
 int pager_in_use;
 int pager_use_color = 1;
+/* by default we allow 2 but up to PACK_VERSION is allowed */
+int delta_version = 2;
 
 static const char *git_dir;
 static char *git_object_dir, *git_index_file, *git_refs_dir, *git_graft_file;
diff --git a/sha1_file.c b/sha1_file.c
index d111be7..6653182 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -527,6 +527,7 @@ int use_packed_git(struct packed_git *p)
 			    p->pack_size - 20)) {
 			die("packfile %s does not match index.", p->pack_name);
 		}
+		p->pack_version = ntohl(hdr->hdr_version);
 	}
 	p->pack_last_used = pack_used_ctr++;
 	p->pack_use_cnt++;
-- 
1.4.3.rc3.ga5ce

Re: [PATCH] pack-objects: use of version 3 delta is now optional.

[Jakub Narebski]

Junio C Hamano wrote:

> This introduces a new configuration item, pack.deltaversion, to
> control whether pack-objects is allowed to use version 3 delta.

Documentation, please?
-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git

Re: [PATCH] pack-objects: use of version 3 delta is now optional.

[Nicolas Pitre]

On Sun, 15 Oct 2006, Junio C Hamano wrote:

> This introduces a new configuration item, pack.deltaversion, to
> control whether pack-objects is allowed to use version 3 delta.
> By default, we keep generating version 2 delta (and version 2
> packfile format) to be compatible with git earlier than v1.2.0.
> 
> This configuration affects the command in the following ways:
> 
>  - the resulting packfile will have the specified version;
> 
>  - when generating delta, larger copies are allowed only when
>    deltaversion is 3;
> 
>  - the logic to reuse delta from existing packs refuses to reuse
>    delta from packs that uses delta version 3 when the
>    configuration is set to 2.
> 
> Signed-off-by: Junio C Hamano <junkio@cox.net>

I'd suggest to drop this altogether.  See my previous email for my 
reasoning on this issue.  I think this should be done another way.

If anything, maybe this patch can be added before v1.4.3 is released:

diff --git a/fetch-pack.c b/fetch-pack.c
index 7d23a80..1688417 100644
--- a/fetch-pack.c
+++ b/fetch-pack.c
@@ -165,9 +165,10 @@ static int find_common(int fd[2], unsign
 			continue;
 		}
 
-		packet_write(fd[1], "want %s%s%s\n", sha1_to_hex(remote),
+		packet_write(fd[1], "want %s%s%s%s\n", sha1_to_hex(remote),
 			     (multi_ack ? " multi_ack" : ""),
-			     (use_thin_pack ? " thin-pack" : ""));
+			     (use_thin_pack ? " thin-pack" : ""),
+			     " packv3");
 		fetching++;
 	}
 	packet_flush(fd[1]);
diff --git a/upload-pack.c b/upload-pack.c
index 979e583..8e57316 100644
--- a/upload-pack.c
+++ b/upload-pack.c
@@ -218,7 +218,7 @@ static int receive_needs(void)
 
 static int send_ref(const char *refname, const unsigned char *sha1)
 {
-	static char *capabilities = "multi_ack thin-pack";
+	static char *capabilities = "multi_ack thin-pack packv3";
 	struct object *o = parse_object(sha1);
 
 	if (!o)

This way pack v3 could be fed to GIT v1.4.3 and above whenever we add 
back pack v3 generation, and a pack converted to v2 from any v3 on the 
fly when that capability is not present.


Nicolas

Re: [PATCH] pack-objects: use of version 3 delta is now optional.

[Junio C Hamano]

Nicolas Pitre <nico@cam.org> writes:

> On Sun, 15 Oct 2006, Junio C Hamano wrote:
>
>> This introduces a new configuration item, pack.deltaversion, to
>> control whether pack-objects is allowed to use version 3 delta.
>> By default, we keep generating version 2 delta (and version 2
>> packfile format) to be compatible with git earlier than v1.2.0.
>...
> I'd suggest to drop this altogether.  See my previous email for my 
> reasoning on this issue.  I think this should be done another way.

I'll think about it a bit.

> If anything, maybe this patch can be added before v1.4.3 is released:
>...
> This way pack v3 could be fed to GIT v1.4.3 and above whenever we add 
> back pack v3 generation, and a pack converted to v2 from any v3 on the 
> fly when that capability is not present.

I think that is sensible.  I also was thinking that we should
call the current one packv3 and the one with delta-base-offset
packv4.

Re: [PATCH] pack-objects: use of version 3 delta is now optional.

[Jakub Narebski]

Junio C Hamano wrote:

> I think that is sensible.  I also was thinking that we should
> call the current one packv3 and the one with delta-base-offset
> packv4.

Just curious: what was the difference between packv1 and packv2,
and packv3 and packv4?
-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git

Re: [PATCH] pack-objects: use of version 3 delta is now optional.

[Nicolas Pitre]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1310 bytes --]

On Sun, 15 Oct 2006, Jakub Narebski wrote:

> Junio C Hamano wrote:
> 
> > I think that is sensible.  I also was thinking that we should
> > call the current one packv3 and the one with delta-base-offset
> > packv4.
> 
> Just curious: what was the difference between packv1 and packv2,
> and packv3 and packv4?

Pack v1 was really short-lived (one day or two).  It used a different 
encoding for object size and delta size than what exists today.  When 
the current encoding was adopted the pack version was bumped to 2 to 
make sure anyone, if any, who might have started to rely upon packs in 
those early days would not end up trying to use incompatible pack data.  
Backward compatibility was not a concern at all back then of course.  
So for all practical purposes just consider that pack version 1 never 
existed.

Pack version 3 simply redefined one bit in the delta encoding that was 
never used.  The former definition of the bit was implemented in the 
decode part, but attempts to use it in the encode part turned up to be 
way too costly for really really poor benefits.  for details just have a 
look at commit d60fc1c8649f80c006b9f493c542461e81608d4b.

As for pack v4... My opinion is that nothing justifies it so far.  So if 
I can convince Junio there shouldn't be any v4 just yet.


Nicolas

Re: [PATCH] pack-objects: use of version 3 delta is now optional.

[Junio C Hamano]

Nicolas Pitre <nico@cam.org> writes:

> As for pack v4... My opinion is that nothing justifies it so far.  So if 
> I can convince Junio there shouldn't be any v4 just yet.

The only concern I have is the commit walkers (rsync has the
same problem as well but we honestly do not care).  They just
grab existing packs and try to use them.  I have been wondering
if it might be safer to mark the delta-base-offset encoded packs
v4 to make sure the clients would get "I know only v2 and v3 but
you fed me v4" message.

People who own public repositories and who care about commit
walkers need to refrain from using delta-base-offset when
repacking (which can be done via configuration mechanism
already).

If download goes over git native protocol, there is no such
worry -- they can pack using delta-base-offset and older clients
will be fed a pack with 20-byte base object name just fine, so my
worry is really only about commit walkers.

Re: [PATCH] pack-objects: use of version 3 delta is now optional.

[Nicolas Pitre]

On Sun, 15 Oct 2006, Junio C Hamano wrote:

> Nicolas Pitre <nico@cam.org> writes:
> 
> > As for pack v4... My opinion is that nothing justifies it so far.  So if 
> > I can convince Junio there shouldn't be any v4 just yet.
> 
> The only concern I have is the commit walkers (rsync has the
> same problem as well but we honestly do not care).  They just
> grab existing packs and try to use them.  I have been wondering
> if it might be safer to mark the delta-base-offset encoded packs
> v4 to make sure the clients would get "I know only v2 and v3 but
> you fed me v4" message.

It'll get "this pack contains an unknown object type" kind of message, 
which is almost as good IMHO, with the same end result.


Nicolas

Re: [PATCH] pack-objects: use of version 3 delta is now optional.

[Nicolas Pitre]

On Sun, 15 Oct 2006, Junio C Hamano wrote:

> Nicolas Pitre <nico@cam.org> writes:
> 
> > If anything, maybe this patch can be added before v1.4.3 is released:
> >...
> > This way pack v3 could be fed to GIT v1.4.3 and above whenever we add 
> > back pack v3 generation, and a pack converted to v2 from any v3 on the 
> > fly when that capability is not present.
> 
> I think that is sensible.  I also was thinking that we should
> call the current one packv3 and the one with delta-base-offset
> packv4.

I think we should not.  The pack version should be tied to incompatible 
pack data to prevent older GIT versions from misinterpreting newer 
packs.  The delta block copy encoding is a perfect example of that where 
a bit changed meaning.

The delta-base-offset case included a new object type that wasn't used 
before hence there is no room for confusion, and yet that new delta 
object could be encoded according to pack version 2 or pack version 3 
which makes it orthogonal to the pack version itself.


Nicolas

Re: [PATCH] pack-objects: use of version 3 delta is now optional.

[A Large Angry SCM]

Nicolas Pitre wrote:
> On Sun, 15 Oct 2006, Junio C Hamano wrote:
> 
>> Nicolas Pitre <nico@cam.org> writes:
>>
>>> If anything, maybe this patch can be added before v1.4.3 is released:
>>> ...
>>> This way pack v3 could be fed to GIT v1.4.3 and above whenever we add 
>>> back pack v3 generation, and a pack converted to v2 from any v3 on the 
>>> fly when that capability is not present.
>> I think that is sensible.  I also was thinking that we should
>> call the current one packv3 and the one with delta-base-offset
>> packv4.
> 
> I think we should not.  The pack version should be tied to incompatible 
> pack data to prevent older GIT versions from misinterpreting newer 
> packs.  The delta block copy encoding is a perfect example of that where 
> a bit changed meaning.
> 
> The delta-base-offset case included a new object type that wasn't used 
> before hence there is no room for confusion, and yet that new delta 
> object could be encoded according to pack version 2 or pack version 3 
> which makes it orthogonal to the pack version itself.

It's not a new object type. It's a new object _encoding_ method.

Re: [PATCH] pack-objects: use of version 3 delta is now optional.

[Nicolas Pitre]

On Sun, 15 Oct 2006, A Large Angry SCM wrote:

> Nicolas Pitre wrote:
> > 
> > The delta-base-offset case included a new object type that wasn't used
> > before hence there is no room for confusion, and yet that new delta object
> > could be encoded according to pack version 2 or pack version 3 which makes
> > it orthogonal to the pack version itself.
> 
> It's not a new object type. It's a new object _encoding_ method.

Not at all.  If it was only a question of encoding method, then both of 
them could always be interchangeable and supersede the other, which is 
not the case here.


Nicolas

Re: [PATCH] pack-objects: use of version 3 delta is now optional.

[Linus Torvalds]

On Sun, 15 Oct 2006, Junio C Hamano wrote:
> 
> I think that is sensible.  I also was thinking that we should
> call the current one packv3 and the one with delta-base-offset
> packv4.

Quite frankly, I wonder if the pure "copy size extension" (aka "v3") thing 
is really worth it at all. 

I mean, seriously, how much does it buy us? A couple of bytes per every 
64kB of delta copied? And the downside is that you can't re-use the deltas 
with old clients and/or you have to re-create a "v2" delta at run-time 
from a v3 delta by inflating, fixing and deflating it.

So I would suggest:

 - call the delta-base-offset thing the "v3" pack format.

 - forget about the current "v3 delta" entirely. We might as well continue 
   to support reading it, but there's no point in actually ever generating 
   it. 

In other words, I think the current situation in top-of-master is the 
right situation. There's simply no point in adding code to convert v3 to 
v2 on the fly - even if it's not rocket science, it's just not _worth_ it.

(You could also have the extended copy deltas in v3-only, and only send it 
to clients that you know supports it. However, the "convert to v2" format 
issue still rears its ugly head, and as a result I just don't think it's 
_ever_ worth it).

		Linus

Re: [PATCH] pack-objects: use of version 3 delta is now optional.

[Nicolas Pitre]

On Sun, 15 Oct 2006, Linus Torvalds wrote:

> 
> 
> On Sun, 15 Oct 2006, Junio C Hamano wrote:
> > 
> > I think that is sensible.  I also was thinking that we should
> > call the current one packv3 and the one with delta-base-offset
> > packv4.
> 
> Quite frankly, I wonder if the pure "copy size extension" (aka "v3") thing 
> is really worth it at all. 
> 
> I mean, seriously, how much does it buy us? A couple of bytes per every 
> 64kB of delta copied? And the downside is that you can't re-use the deltas 
> with old clients and/or you have to re-create a "v2" delta at run-time 
> from a v3 delta by inflating, fixing and deflating it.

Right.  This is why I suggested Junio to just drop it for now.  Let's 
just wait some more until this is just not an issue any longer, say in a 
year from now when all major distributions have switched to a GIT 
version that can read V3.

If until then we find the saving really worth the backward compatibility 
v3-to-v2 conversion then we could reconsider.  But I don't think it is 
worth it just yet.

In the mean time, if Junio adds the patch I posted yesterday advertising 
the pack version capability over the native protocol then it'll help us 
make things forward compatible if ever we decide to go with generating 
packs v3 sooner.


Nicolas

Re: [PATCH] pack-objects: use of version 3 delta is now optional.

[Junio C Hamano]

Nicolas Pitre <nico@cam.org> writes:

> On Sun, 15 Oct 2006, Linus Torvalds wrote:
>
>> Quite frankly, I wonder if the pure "copy size extension" (aka "v3") thing 
>> is really worth it at all. 
>> 
>> I mean, seriously, how much does it buy us? A couple of bytes per every 
>> 64kB of delta copied? And the downside is that you can't re-use the deltas 
>> with old clients and/or you have to re-create a "v2" delta at run-time 
>> from a v3 delta by inflating, fixing and deflating it.
>
>...
> In the mean time, if Junio adds the patch I posted yesterday advertising 
> the pack version capability over the native protocol then it'll help us 
> make things forward compatible if ever we decide to go with generating 
> packs v3 sooner.

I've thought about this, but we hopefully would have ofs-delta
capability exchanged soon after 1.4.3, and that would be an
enough advertisement that the client is recent enough; although
it is technically incorrect to tie these two independent
features together, the improvement between v2 and v3 is dubious
so maybe that is the easiest.

Re: [PATCH] pack-objects: use of version 3 delta is now optional.

[Nicolas Pitre]

On Tue, 17 Oct 2006, Junio C Hamano wrote:

> Nicolas Pitre <nico@cam.org> writes:
> 
> > On Sun, 15 Oct 2006, Linus Torvalds wrote:
> >
> >> Quite frankly, I wonder if the pure "copy size extension" (aka "v3") thing 
> >> is really worth it at all. 
> >> 
> >> I mean, seriously, how much does it buy us? A couple of bytes per every 
> >> 64kB of delta copied? And the downside is that you can't re-use the deltas 
> >> with old clients and/or you have to re-create a "v2" delta at run-time 
> >> from a v3 delta by inflating, fixing and deflating it.
> >
> >...
> > In the mean time, if Junio adds the patch I posted yesterday advertising 
> > the pack version capability over the native protocol then it'll help us 
> > make things forward compatible if ever we decide to go with generating 
> > packs v3 sooner.
> 
> I've thought about this, but we hopefully would have ofs-delta
> capability exchanged soon after 1.4.3, and that would be an
> enough advertisement that the client is recent enough; although
> it is technically incorrect to tie these two independent
> features together, the improvement between v2 and v3 is dubious
> so maybe that is the easiest.

Fair enough.


Nicolas

Re: [PATCH] pack-objects: use of version 3 delta is now optional.

[A Large Angry SCM]

Junio C Hamano wrote:
[...]
> 
> I think that is sensible.  I also was thinking that we should
> call the current one packv3 and the one with delta-base-offset
> packv4.

+1

Re: Recent and near future backward incompatibilities

[Horst H. von Brand]

Junio C Hamano <junkio@cox.net> wrote:
> It was brought to my attention that the public git.git
> repository cannot be cloned with older versions of git. [...]

There seem to be a bunch of incompatible changes comming up... how about
scheduling them for a 2.0 version, soon(ish)?
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                    Fono: +56 32 2654431
Universidad Tecnica Federico Santa Maria             +56 32 2654239
Casilla 110-V, Valparaiso, Chile               Fax:  +56 32 2797513

Re: Recent and near future backward incompatibilities

[Nicolas Pitre]

On Sat, 14 Oct 2006, Junio C Hamano wrote:

> It was brought to my attention that the public git.git
> repository cannot be cloned with older versions of git.  More
> precisely, packs generated with post 16854571 (NOT contained in
> v1.4.2.3 but in the current "master" and more importantly in
> v1.4.3-rc3 which I tagged tonight) can contain deltas that are
> not compatible with the version of git before d60fc1c8, which
> means that v1.1.6 and older (v1.2.0 and later are Ok).

Ahhhhhh.  DAMN !

> One thing we can and should immediately do is to revert 16854571
> for now until we decide how to resolve this issue cleanly.
> 
> These are what needs to happen but one of them is quite tricky:
> 
>  - the reusing of delta is what makes pack-objects practical,
>    and it is expensive to look into existing delta to see if it
>    is version 2 or version 3 before deciding to reuse each delta
>    data, so even if we update pack-objects so that we can tell
>    it to generate a pack that contains only version 2 deltas, it
>    would be very expensive to do so and may not be practical.

Why not?  After all users of GIT versions that don't understand pack 
version 3 should be a very small minority by now.

>    am not sure how to resolve this issue efficiently right now;
>    we need a bit of thinking.

Actually it doesn't have to be that expensive to convert deltas v2 to 
deltas v3 on the fly.  They can be inflated, parsed, the copy ops that 
exceed 0x10000 converted into multiple ops of smaller copy blocks, then 
deflated.  This is certainly much less costly than rematching deltas 
from scratch.

Well I'd say you just revert pack v3 generation patch for now and 
release v1.4.3 without it.  Pack v3 generation can wait a bit longer 
until we implement the above or users of GIT that can read packs v2 only 
are so few that we shouldn't care anymore and tell them to use an 
intermediate version of GIT in order to clone the latest.  It is not 
like if that makes such a big difference on pack size anyway (much less 
than delta with offsets to base actually).

>  - we need to add .git/config item that tells pack-objects to
>    never generate version 3 delta for that particular
>    repository.  This is similar to the way we would need to
>    control the use of delta-base-offset representation currently
>    cooking in "next".

This is different. The delta-base-offset representation is decided at 
run time every time a pack is generated and regardless if delta data is 
being reused from another pack or regenerated afresh, and so with no 
cost.  So this is no issue for users of old GIT versions since the 
native GIT protocol already handle it in a backward compatible manner.

The only issue here concerns users that don't use the native GIT 
protocol.  But in this case they have two options: either they switch to 
the native protocol, or they upgrade to the latest GIT version which 
can always be pulled with the native GIT protocol.

> We may have a similar issue when enabling generation of loose
> objects with new style headers.  This is already controlled with
> the core.legacyheaders configuration item.

Sure, but those are never passed through the native GIT protocol which 
makes it a much less critical issue.

Not being able to upgrade to the latest GIT in order to actually cope 
with the new format because the primary GIT repository started to feed 
old GIT versions with that new format unconditionally really is a 
problem though.

I think we should not bend backward too much with repository 
compatibility issues as long as there is no interoperability issues at 
the protocol level.  But the GIT protocol must always remain 
interoperable with whatever GIT version still in use.


Nicolas

Re: Recent and near future backward incompatibilities

[Junio C Hamano]

Nicolas Pitre <nico@cam.org> writes:

> Actually it doesn't have to be that expensive to convert deltas v2 to 
> deltas v3 on the fly.  They can be inflated, parsed, the copy ops that 
> exceed 0x10000 converted into multiple ops of smaller copy blocks, then 
> deflated.  This is certainly much less costly than rematching deltas 
> from scratch.

True, when I think about it.

>>  - we need to add .git/config item that tells pack-objects to
>>    never generate version 3 delta for that particular
>>    repository.  This is similar to the way we would need to
>>    control the use of delta-base-offset representation currently
>>    cooking in "next".
>
> This is different. The delta-base-offset representation is decided at 
> run time every time a pack is generated and regardless if delta data is 
> being reused from another pack or regenerated afresh, and so with no 
> cost.  So this is no issue for users of old GIT versions since the 
> native GIT protocol already handle it in a backward compatible manner.
>
> The only issue here concerns users that don't use the native GIT 
> protocol.  But in this case they have two options: either they switch to 
> the native protocol, or they upgrade to the latest GIT version which 
> can always be pulled with the native GIT protocol.

True again.  Thanks.

Re: Recent and near future backward incompatibilities

[Theodore Tso]

On Sat, Oct 14, 2006 at 11:29:17PM -0700, Junio C Hamano wrote:
> It was brought to my attention that the public git.git
> repository cannot be cloned with older versions of git.  More
> precisely, packs generated with post 16854571 (NOT contained in
> v1.4.2.3 but in the current "master" and more importantly in
> v1.4.3-rc3 which I tagged tonight) can contain deltas that are
> not compatible with the version of git before d60fc1c8, which
> means that v1.1.6 and older (v1.2.0 and later are Ok).

By the way, note that Ubuntu Dapper (the current stable version of
Ubuntu) is shipped with git version 1.1.3, and that incompatibility
extends not to the git repository, but also the Linux-2.6 repostiory
at

git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git

						- Ted

Re: Recent and near future backward incompatibilities

[Linus Torvalds]

On Sun, 15 Oct 2006, Theodore Tso wrote:
> 
> By the way, note that Ubuntu Dapper (the current stable version of
> Ubuntu) is shipped with git version 1.1.3, and that incompatibility
> extends not to the git repository, but also the Linux-2.6 repostiory
> at
> 
> git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git

That should have been fixed earlier today, since I forced a repack when 
this was discovered.

So if somebody still can't pull it with old git tools, please holler.

That said, Ubuntu should definitely upgrade.

		Linus

Re: Recent and near future backward incompatibilities

[Stephen Hemminger]

On Sun, 15 Oct 2006 16:52:04 -0700 (PDT)
Linus Torvalds <torvalds@osdl.org> wrote:

> 
> 
> On Sun, 15 Oct 2006, Theodore Tso wrote:
> > 
> > By the way, note that Ubuntu Dapper (the current stable version of
> > Ubuntu) is shipped with git version 1.1.3, and that incompatibility
> > extends not to the git repository, but also the Linux-2.6 repostiory
> > at
> > 
> > git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
> 
> That should have been fixed earlier today, since I forced a repack when 
> this was discovered.
> 
> So if somebody still can't pull it with old git tools, please holler.
> 
> That said, Ubuntu should definitely upgrade.
> 
> 		Linus

They made some nice about putting a new version in the backport
repository.