v1.5.4 plans

v1.5.4 plans

[Junio C Hamano]

Please do not take this as the final decision made by the Emperor, whose
subjects now must follow.  This is a sanity-check to see if everybody is
on the same page.

I am not the Emperor anyway ;-)

Deprecation and Removal
-----------------------

 * We have already removed svnimport without giving a deprecation notice
   in the release notes of the previous feature release, which was bad.
   Maybe the users will forgive us.  Maybe not.

 * As discussed on the list, v1.5.4 will ship with the dashed form of
   commands (e.g. "git-commit") on users' PATH by default.  However we
   will move them outside the normal PATH (exact location needs to be
   decided by checking FHS first, something like /usr/libexec/git-core)
   in v1.5.5 so the release notes to v1.5.4 will declare deprecation
   (see the top of Documentation/RelNotes-1.5.4.txt).  We might want to
   keep copies of dashed form Porcelains in /usr/bin but that discussion
   is towards v1.5.5 (post v1.5.4, not now).

 * We also will give deprecation warning for the following features and
   commands in the release notes to v1.5.4, and remove them in v1.5.5:

   - lost-found (use fsck --lost-found);
   - post-update hook (use post-receive hook);
   - peek-remote (use ls-remote)


Topics not in 'master' yet but should be in v1.5.4
--------------------------------------------------

I think the following should go in, along with what we already have in
'master':

 * git-commit in C (Kristian and others)
 * git-add --patch (Wincent)
 * git-prune --expire (Dscho)
 * git-add --interactive coloring (Dan Zwell)
 * whitespace error classes in diff and patch, using gitattributes (Bruce and me)
 * cvsserver runs post-receive (Michael Witten)
 * git-rebase -i gives chance to rerere (Dscho)
 * git-rebase gives more appropriate help text (Wincent)
 * make refspec matching logic in git-push and git-fetch saner (Steffen Prohaska)
 * work-tree related minor fixes (Nguyen and Dscho)
 * allow update hook to munge commit (Steven Grimm)

I'd like to explicitly exclude topics about the following, although I
think there might be worthwhile ones among them to think about in the
longer term:

 * removing dashed form from the filesystem
 * teaching diff family about fileglob pathspecs
 * making it possible to omit leading paths from diff family output when
   run from a subdirectory

Re: v1.5.4 plans

[Jakub Narebski]

Does this mean we should be entering feature freeze?
Or at least feature freeze for 'master'?

-- 
Jakub Narebski

Re: v1.5.4 plans

[Junio C Hamano]

Jakub Narebski <jnareb@gmail.com> writes:

> Does this mean we should be entering feature freeze?
> Or at least feature freeze for 'master'?

I said "these are not in master but should be before v1.5.4", so master
is not frozen yet in that sense, but on the other hand, I want to stop
taking anything other than I explicitly listed to 'master' from now on.

So please take it as a "rfc feature freeze notice".

Re: v1.5.4 plans

[David Symonds]

On Dec 3, 2007 9:04 AM, Junio C Hamano <gitster@pobox.com> wrote:
> Please do not take this as the final decision made by the Emperor, whose
> subjects now must follow.  This is a sanity-check to see if everybody is
> on the same page.
>
> I am not the Emperor anyway ;-)
>

> Topics not in 'master' yet but should be in v1.5.4
> --------------------------------------------------
>
> I think the following should go in, along with what we already have in
> 'master':

Can we add the git-status/git-checkout relative path stuff that's
currently been sitting in 'next'? It would be a good step forward for
usability.


Dave.

Re: v1.5.4 plans

[Junio C Hamano]

"David Symonds" <dsymonds@gmail.com> writes:

> On Dec 3, 2007 9:04 AM, Junio C Hamano <gitster@pobox.com> wrote:
>> Please do not take this as the final decision made by the Emperor, whose
>> subjects now must follow.  This is a sanity-check to see if everybody is
>> on the same page.
>>
>> I am not the Emperor anyway ;-)
>>
>
>> Topics not in 'master' yet but should be in v1.5.4
>> --------------------------------------------------
>>
>> I think the following should go in, along with what we already have in
>> 'master':
>
> Can we add the git-status/git-checkout relative path stuff that's
> currently been sitting in 'next'? It would be a good step forward for
> usability.

I think checkout from subdirectory with relative was merged on November
18th to master, with d577bc58.  Relative path output for git-status is
part of the "git-commit in C" series, which is planned to go in.

But now you mention it, I realize that I ran "git-topic.perl" (found in
my 'todo' branch) without "--all" option when I made that list, and I
missed stuff fully merged to 'next'.  Sorry.

Here is a corrected list.

Topics not in 'master' yet but should be in v1.5.4
--------------------------------------------------

I think the following should go in, along with what we already have in
'master':

 * git-commit in C (Kristian and others)
 * git-add --patch (Wincent)
 * git-prune --expire (Dscho)
 * git-add --interactive coloring (Dan Zwell)
 * whitespace error classes in diff and patch, using gitattributes (Bruce and me)
 * cvsserver runs post-receive (Michael Witten)
 * git-rebase -i gives chance to rerere (Dscho)
 * git-rebase gives more appropriate help text (Wincent)
 * make refspec matching logic in git-push and git-fetch saner (Steffen Prohaska)
 * work-tree related minor fixes (Nguyen and Dscho)
 * allow update hook to munge commit (Steven Grimm)
 * git-fast-export (Dscho)
 * Add commitdiff to gitweb grep page (Denis Cheng)
 * "git pull --rebase" (Dscho)
 * "git config --get-color" (me)
 * "color.diff = true" means "auto" (me)
 * Rewrite "export VAR=VAL" to "VAR=VAL; export VAR" (Dscho)
 * run correct perl in Documentation (me, waiting for Merlyn)

[PATCH] Fix quote_path when called with negative length.

[Pierre Habouzit]

[-- Attachment #1: Type: text/plain, Size: 3238 bytes --]

When the len passed was -1, relative paths shortening was broken, resulting
in too long paths.

Signed-off-by: Pierre Habouzit <madcoder@debian.org>
---

    On Sun, Dec 02, 2007 at 11:39:59PM +0000, David Symonds wrote:
    > On Dec 3, 2007 9:04 AM, Junio C Hamano <gitster@pobox.com> wrote:
    > > Please do not take this as the final decision made by the Emperor, whose
    > > subjects now must follow.  This is a sanity-check to see if everybody is
    > > on the same page.
    > >
    > > I am not the Emperor anyway ;-)
    > >
    > 
    > > Topics not in 'master' yet but should be in v1.5.4
    > > --------------------------------------------------
    > >
    > > I think the following should go in, along with what we already have in
    > > 'master':
    > 
    > Can we add the git-status/git-checkout relative path stuff that's
    > currently been sitting in 'next'? It would be a good step forward for
    > usability.

    Speaking of which, there is this irritating bug in git status that
    let it show too long paths in the first chunk (the "tracked files"
    one).

    The previous version of the function was avoiding very hard to
    compute "in" length, and had quite convoluted code because of that.
    I now compute it at the beginning. The real issue was the:

 		while (prefix[off] && off < len && prefix[off] == in[off])

    line, when len is negative, the shortening never happens. I could
    have fixed it using ((len < 0 && in[off]) || off < len), but I
    disliked the resulting code, so I went for this.

    -- 
    ·O·  Pierre Habouzit
    ··O                                                madcoder@debian.org
    OOO                                                http://www.madism.org

 wt-status.c |   31 +++++++++++++------------------
 1 files changed, 13 insertions(+), 18 deletions(-)

diff --git a/wt-status.c b/wt-status.c
index 0e0439f..eb2cbea 100644
--- a/wt-status.c
+++ b/wt-status.c
@@ -84,30 +84,25 @@ static void wt_status_print_trailer(struct wt_status *s)
 static char *quote_path(const char *in, int len,
 		struct strbuf *out, const char *prefix)
 {
-	if (len > 0)
-		strbuf_grow(out, len);
+	int pos = 0;
+
+	if (len < 0)
+		len = strlen(in);
+	strbuf_grow(out, len);
 	strbuf_setlen(out, 0);
 
 	if (prefix) {
 		int off = 0;
 		while (prefix[off] && off < len && prefix[off] == in[off])
-			if (prefix[off] == '/') {
-				prefix += off + 1;
-				in += off + 1;
-				len -= off + 1;
-				off = 0;
-			} else
-				off++;
-
-		for (; *prefix; prefix++)
-			if (*prefix == '/')
+			if (prefix[off++] == '/')
+				pos = off;
+		while (prefix[off])
+			if (prefix[off++] == '/')
 				strbuf_addstr(out, "../");
 	}
 
-	for (; (len < 0 && *in) || len > 0; in++, len--) {
-		int ch = *in;
-
-		switch (ch) {
+	for (; pos < len; pos++) {
+		switch (in[pos]) {
 		case '\n':
 			strbuf_addstr(out, "\\n");
 			break;
@@ -115,8 +110,8 @@ static char *quote_path(const char *in, int len,
 			strbuf_addstr(out, "\\r");
 			break;
 		default:
-			strbuf_addch(out, ch);
-			continue;
+			strbuf_addch(out, in[pos]);
+			break;
 		}
 	}
 
-- 
1.5.3.7.2065.g3d18-dirty


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Many things pushed out to 'master'

[Junio C Hamano]

I've merged a handful topics that have been cooking in 'next' to
'master'.  Except for a few big topics still in 'next', this brings the
tip of 'master' much closer to what will become 1.5.4.

As always has been promised, the tip of 'master' is designed to be more
stable than any released version without introducing regression, and we
need to test how true that is from time to time ;-).

Please keep the fixes flowing.  The next batch will be "commit in C" and
"add --patch" series.

Re: Many things pushed out to 'master'

[Johannes Schindelin]

Hi,

On Mon, 3 Dec 2007, Junio C Hamano wrote:

> I've merged a handful topics that have been cooking in 'next' to 
> 'master'.  Except for a few big topics still in 'next', this brings the 
> tip of 'master' much closer to what will become 1.5.4.

I usually run off next + patches, so I do not know if fast-export already 
made it to "master".

But I remembered that you requested a mode for signed tags where they 
would just be copied.  I just realised while implementing "verbatim" that 
"ignore" does just that.  Maybe we should rename this mode to "verbatim"?

And maybe you want to make it the default (even if I think that this will 
result in more surprise moments than the current mode which aborts).

Ciao,
Dscho

Re: [PATCH] Fix quote_path when called with negative length.

[Jeff King]

On Mon, Dec 03, 2007 at 10:06:52AM +0100, Pierre Habouzit wrote:

>     Speaking of which, there is this irritating bug in git status that
>     let it show too long paths in the first chunk (the "tracked files"
>     one).

It was annoying me, too. See the thread 'quote_path: fix collapsing of
relative paths'.

-Peff

Re: v1.5.4 plans

[Nicolas Pitre]

On Sun, 2 Dec 2007, Junio C Hamano wrote:

> Please do not take this as the final decision made by the Emperor, whose
> subjects now must follow.  This is a sanity-check to see if everybody is
> on the same page.
> 
> I am not the Emperor anyway ;-)

Emperor of the Rising Sun.  ;-)

> Deprecation and Removal
> -----------------------
> 
>  * We also will give deprecation warning for the following features and
>    commands in the release notes to v1.5.4, and remove them in v1.5.5:
> 
>    - lost-found (use fsck --lost-found);
>    - post-update hook (use post-receive hook);
>    - peek-remote (use ls-remote)

Two things I would like to see in the next version (1.5.5) as well, for 
which we could provide early warnings now:

 - repack.usedeltabaseoffset defaulting to true

 - pack.indexversion defaulting to 2


Nicolas

Re: Many things pushed out to 'master'

[Junio C Hamano]

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

> But I remembered that you requested a mode for signed tags where they 
> would just be copied.  I just realised while implementing "verbatim" that 
> "ignore" does just that.  Maybe we should rename this mode to "verbatim"?
>
> And maybe you want to make it the default (even if I think that this will 
> result in more surprise moments than the current mode which aborts).

I did not hear others agreeing with me, so let's respect the original
author's thinking.

Re: Many things pushed out to 'master'

[Johannes Schindelin]

Hi,

On Mon, 3 Dec 2007, Junio C Hamano wrote:

> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
> 
> > But I remembered that you requested a mode for signed tags where they 
> > would just be copied.  I just realised while implementing "verbatim" 
> > that "ignore" does just that.  Maybe we should rename this mode to 
> > "verbatim"?
> >
> > And maybe you want to make it the default (even if I think that this 
> > will result in more surprise moments than the current mode which 
> > aborts).
> 
> I did not hear others agreeing with me, so let's respect the original 
> author's thinking.

But the original author already admitted that the original naming was 
stupid...

Ciao,
Dscho

Re: Many things pushed out to 'master'

[Junio C Hamano]

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

> Hi,
>
> On Mon, 3 Dec 2007, Junio C Hamano wrote:
>
>> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
>> 
>> > But I remembered that you requested a mode for signed tags where they 
>> > would just be copied.  I just realised while implementing "verbatim" 
>> > that "ignore" does just that.  Maybe we should rename this mode to 
>> > "verbatim"?
>> >
>> > And maybe you want to make it the default (even if I think that this 
>> > will result in more surprise moments than the current mode which 
>> > aborts).
>> 
>> I did not hear others agreeing with me, so let's respect the original 
>> author's thinking.
>
> But the original author already admitted that the original naming was 
> stupid...

Ok, send in updates, please.

Re: v1.5.4 plans

[Junio C Hamano]

Nicolas Pitre <nico@cam.org> writes:

> Two things I would like to see in the next version (1.5.5) as well, for 
> which we could provide early warnings now:
>
>  - repack.usedeltabaseoffset defaulting to true
>
>  - pack.indexversion defaulting to 2

I think the former would be sensible, the latter I fear might be a bit
too new but I do not recall the exact version dependency.

Could you draft a patch to ReleaseNotes to explain the consequences of
these changes using ordinary user's vocabulary, like:

	Starting v1.5.5, repack.usedeltabaseoffset will default to true,
	which will give denser packfile (i.e. more efficient storage).
	The downside is that git older than version X will not be able
	to use a repository packed using this setting.

[PATCH] fast-export: rename the signed tag mode 'ignore' to 'verbatim'

[Johannes Schindelin]

The name 'verbatim' describes much better what this mode does with
signed tags.  While at it, fix the documentation what it actually
does.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---

	On Mon, 3 Dec 2007, Junio C Hamano wrote:

	> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
	> 
	> > But the original author already admitted that the original 
	> > naming was stupid...
	> 
	> Ok, send in updates, please.

	Alright...

 Documentation/git-fast-export.txt |    4 ++--
 builtin-fast-export.c             |    8 ++++----
 2 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/Documentation/git-fast-export.txt b/Documentation/git-fast-export.txt
index 073ff7f..fd3d571 100644
--- a/Documentation/git-fast-export.txt
+++ b/Documentation/git-fast-export.txt
@@ -26,14 +26,14 @@ OPTIONS
 	Insert 'progress' statements every <n> objects, to be shown by
 	gitlink:git-fast-import[1] during import.
 
---signed-tags=(ignore|warn|strip|abort)::
+--signed-tags=(verbatim|warn|strip|abort)::
 	Specify how to handle signed tags.  Since any transformation
 	after the export can change the tag names (which can also happen
 	when excluding revisions) the signatures will not match.
 +
 When asking to 'abort' (which is the default), this program will die
 when encountering a signed tag.  With 'strip', the tags will be made
-unsigned, with 'ignore', they will be silently ignored (i.e. not exported)
+unsigned, with 'verbatim', they will be silently exported
 and with 'warn', they will be exported, but you will see a warning.
 
 
diff --git a/builtin-fast-export.c b/builtin-fast-export.c
index 72be45d..2136aad 100755
--- a/builtin-fast-export.c
+++ b/builtin-fast-export.c
@@ -23,15 +23,15 @@ static const char *fast_export_usage[] = {
 };
 
 static int progress;
-static enum { IGNORE, WARN, STRIP, ABORT } signed_tag_mode = ABORT;
+static enum { VERBATIM, WARN, STRIP, ABORT } signed_tag_mode = ABORT;
 
 static int parse_opt_signed_tag_mode(const struct option *opt,
 				     const char *arg, int unset)
 {
 	if (unset || !strcmp(arg, "abort"))
 		signed_tag_mode = ABORT;
-	else if (!strcmp(arg, "ignore"))
-		signed_tag_mode = IGNORE;
+	else if (!strcmp(arg, "verbatim") || !strcmp(arg, "ignore"))
+		signed_tag_mode = VERBATIM;
 	else if (!strcmp(arg, "warn"))
 		signed_tag_mode = WARN;
 	else if (!strcmp(arg, "strip"))
@@ -270,7 +270,7 @@ static void handle_tag(const char *name, struct tag *tag)
 				warning ("Exporting signed tag %s",
 					 sha1_to_hex(tag->object.sha1));
 				/* fallthru */
-			case IGNORE:
+			case VERBATIM:
 				break;
 			case STRIP:
 				message_size = signature + 1 - message;
-- 
1.5.3.7.2120.g1a32

Re: [PATCH] fast-export: rename the signed tag mode 'ignore' to 'verbatim'

[Johannes Schindelin]

Hi,

On Mon, 3 Dec 2007, Johannes Schindelin wrote:

> The name 'verbatim' describes much better what this mode does with
> signed tags.  While at it, fix the documentation what it actually
> does.
> 
> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
> ---

Maybe you want to squash this, too (sorry, I sent the patch too soon, 
although the mode "ignore" is still accepted, and thus, the test 
succeeded):

-- snipsnap --
[PATCH] fast-export: test "verbatim", not "ignore"

The signed-tag-mode "ignore" was renamed to "verbatim", and although we 
still accept "ignore" as a synonym, it is better to test "verbatim".

Signed-off-by: Johannes Schindelin <Johannes.Schindelin@gmx.de>

---

 t/t9301-fast-export.sh |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/t/t9301-fast-export.sh b/t/t9301-fast-export.sh
index e9c9fe6..f09bfb1 100755
--- a/t/t9301-fast-export.sh
+++ b/t/t9301-fast-export.sh
@@ -106,9 +106,9 @@ test_expect_success 'signed-tags=abort' '
 
 '
 
-test_expect_success 'signed-tags=ignore' '
+test_expect_success 'signed-tags=verbatim' '
 
-	git fast-export --signed-tags=ignore sign-your-name > output &&
+	git fast-export --signed-tags=verbatim sign-your-name > output &&
 	grep PGP output
 
 '

Re: v1.5.4 plans

[Russell]

On Dec 3, 2007 7:04 AM, Junio C Hamano <gitster@pobox.com> wrote:
>  * We have already removed svnimport without giving a deprecation notice
>    in the release notes of the previous feature release, which was bad.
>    Maybe the users will forgive us.  Maybe not.

Ah, that explains that.  I was in the middle of importing the open2x
project into a git repo.  It's a large tree which looks like it
includes several copies of linux 2.4, and importing is taking several
days.  Occasionally the svn connection times out or something, and I
just restart it and it continues.  In the middle of that I built and
installed git 1.5.3.7 and was surprised when git-svnimport wasn't
there the next time I tried to restart it.  Back to 1.5.3.6 for now.

I see there's a thread about using a git-svnimport tree with git-svn,
so I'll do that.

Oh, and you're forgiven.  :)



-- 
Virus found in this message.

[PATCH] provide advance warning of some future pack default changes

[Nicolas Pitre]

Signed-off-by: Nicolas Pitre <nico@cam.org>

---

On Mon, 3 Dec 2007, Junio C Hamano wrote:

> Nicolas Pitre <nico@cam.org> writes:
> 
> > Two things I would like to see in the next version (1.5.5) as well, for 
> > which we could provide early warnings now:
> >
> >  - repack.usedeltabaseoffset defaulting to true
> >
> >  - pack.indexversion defaulting to 2
> 
> I think the former would be sensible, the latter I fear might be a bit
> too new but I do not recall the exact version dependency.
> 
> Could you draft a patch to ReleaseNotes to explain the consequences of
> these changes using ordinary user's vocabulary, like:
> 
> 	Starting v1.5.5, repack.usedeltabaseoffset will default to true,
> 	which will give denser packfile (i.e. more efficient storage).
> 	The downside is that git older than version X will not be able
> 	to use a repository packed using this setting.

Here it is.

diff --git a/Documentation/RelNotes-1.5.4.txt b/Documentation/RelNotes-1.5.4.txt
index 6645565..d6fd3dd 100644
--- a/Documentation/RelNotes-1.5.4.txt
+++ b/Documentation/RelNotes-1.5.4.txt
@@ -43,6 +43,17 @@ Deprecation notices
  * "git peek-remote" is deprecated, as "git ls-remote" was written in C
    and works for all transports, and will be removed in the future.
 
+ * From v1.5.5, the repack.usedeltabaseoffset config option will default
+   to true, which will give denser packfile (i.e. more efficient storage).
+   The downside is that git older than version 1.4.4 will not be able
+   to directly use a repository packed using this setting.
+
+ * From v1.5.5, the pack.indexversion config option will default to 2,
+   which is slightly more efficient, and makes repacking more immune to
+   data corruptions.  Git older than version 1.5.2 may revert to version 1
+   of the pack index with a manual "git index-pack" to be able to directly
+   access corresponding pack files.
+
 
 Updates since v1.5.3
 --------------------

Re: [PATCH] provide advance warning of some future pack default changes

[Junio C Hamano]

Thanks.

Deprecating versions before 1.5.2 (May 20 2007) feels a bit too quick,
but seven month is almost an eternity in git timescale, and by now
anything older than 1.5.2 can safely be called prehistoric.  Will apply.

Re: [PATCH] provide advance warning of some future pack default changes

[linux]

>+ * From v1.5.5, the pack.indexversion config option will default to 2,
>+   which is slightly more efficient, and makes repacking more immune to
>+   data corruptions.  Git older than version 1.5.2 may revert to version 1
>+   of the pack index with a manual "git index-pack" to be able to directly
>+   access corresponding pack files.

You might want to mention that it's slightly more TIME efficient,
but takes 16% more space (28 bytes per object rather than 24).

If it helps, I documented the v2 index file format (a lot stolen
from commit c553ca25bd60dc9fd50b8bc7bd329601b81cee66 message).
(Public domain, copyright abandoned, if it breaks you get to keep both
pieces, yadda yadda.)

diff --git a/Documentation/technical/pack-format.txt b/Documentation/technical/pack-format.txt
index e5b31c8..a80baa4 100644
--- a/Documentation/technical/pack-format.txt
+++ b/Documentation/technical/pack-format.txt
@@ -1,9 +1,9 @@
 GIT pack format
 ===============
 
-= pack-*.pack file has the following format:
+= pack-*.pack files have the following format:
 
-   - The header appears at the beginning and consists of the following:
+   - A header appears at the beginning and consists of the following:
 
      4-byte signature:
          The signature is: {'P', 'A', 'C', 'K'}
@@ -34,18 +34,14 @@ GIT pack format
 
   - The trailer records 20-byte SHA1 checksum of all of the above.
 
-= pack-*.idx file has the following format:
+= Original (version 1) pack-*.idx files have the following format:
 
   - The header consists of 256 4-byte network byte order
     integers.  N-th entry of this table records the number of
     objects in the corresponding pack, the first byte of whose
-    object name are smaller than N.  This is called the
+    object name is less than or equal to N.  This is called the
     'first-level fan-out' table.
 
-    Observation: we would need to extend this to an array of
-    8-byte integers to go beyond 4G objects per pack, but it is
-    not strictly necessary.
-
   - The header is followed by sorted 24-byte entries, one entry
     per object in the pack.  Each entry is:
 
@@ -55,10 +51,6 @@ GIT pack format
 
     20-byte object name.
 
-    Observation: we would definitely need to extend this to
-    8-byte integer plus 20-byte object name to handle a packfile
-    that is larger than 4GB.
-
   - The file is concluded with a trailer:
 
     A copy of the 20-byte SHA1 checksum at the end of
@@ -68,31 +60,30 @@ GIT pack format
 
 Pack Idx file:
 
-	idx
-	    +--------------------------------+
-	    | fanout[0] = 2                  |-.
-	    +--------------------------------+ |
+	--  +--------------------------------+
+fanout	    | fanout[0] = 2 (for example)    |-.
+table	    +--------------------------------+ |
 	    | fanout[1]                      | |
 	    +--------------------------------+ |
 	    | fanout[2]                      | |
 	    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
-	    | fanout[255]                    | |
-	    +--------------------------------+ |
-main	    | offset                         | |
-index	    | object name 00XXXXXXXXXXXXXXXX | |
-table	    +--------------------------------+ |
-	    | offset                         | |
-	    | object name 00XXXXXXXXXXXXXXXX | |
-	    +--------------------------------+ |
-	  .-| offset                         |<+
-	  | | object name 01XXXXXXXXXXXXXXXX |
-	  | +--------------------------------+
-	  | | offset                         |
-	  | | object name 01XXXXXXXXXXXXXXXX |
-	  | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-	  | | offset                         |
-	  | | object name FFXXXXXXXXXXXXXXXX |
-	  | +--------------------------------+
+	    | fanout[255] = total objects    |---.
+	--  +--------------------------------+ | |
+main	    | offset                         | | |
+index	    | object name 00XXXXXXXXXXXXXXXX | | |
+table	    +--------------------------------+ | |
+	    | offset                         | | |
+	    | object name 00XXXXXXXXXXXXXXXX | | |
+	    +--------------------------------+<+ |
+	  .-| offset                         |   |
+	  | | object name 01XXXXXXXXXXXXXXXX |   |
+	  | +--------------------------------+   |
+	  | | offset                         |   |
+	  | | object name 01XXXXXXXXXXXXXXXX |   |
+	  | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~   |
+	  | | offset                         |   |
+	  | | object name FFXXXXXXXXXXXXXXXX |   |
+	--| +--------------------------------+<--+
 trailer	  | | packfile checksum              |
 	  | +--------------------------------+
 	  | | idxfile checksum               |
@@ -116,3 +107,40 @@ Pack file entry: <+
 	  20-byte base object name SHA1 (the size above is the
 		size of the delta data that follows).
           delta data, deflated.
+
+
+= Version 2 pack-*.idx files support packs larger than 4 GiB, and
+  have some other reorganizations.  They have the format:
+
+  - A 4-byte magic number '\377tOc' which is an unreasonable
+    fanout[0] value.
+
+  - A 4-byte version number (= 2)
+
+  - A 256-entry fan-out table just like v1.
+
+  - A table of sorted 20-byte SHA1 object names.  These are
+    packed together without offset values to reduce the cache
+    footprint of the binary search for a specific object name.
+
+  - A table of 4-byte CRC32 values of the packed object data.
+    This is new in v2 so compressed data can be copied directly
+    from pack to pack during repacking withough undetected
+    data corruption.
+
+  - A table of 4-byte offset values (in network byte order).
+    These are usually 31-bit pack file offsets, but large
+    offsets are encoded as an index into the next table with
+    the msbit set.
+
+  - A table of 8-byte offset entries (empty for pack files less
+    than 2 GiB).  Pack files are organized with heavily used
+    objects toward the front, so most object references should
+    not need to refer to this table.
+
+  - The same trailer as a v1 pack file:
+
+    A copy of the 20-byte SHA1 checksum at the end of
+    corresponding packfile.
+
+    20-byte SHA1-checksum of all of the above.

Re: [PATCH] provide advance warning of some future pack default changes

[Jakub Narebski]

Nicolas Pitre <nico@cam.org> writes:

> + * From v1.5.5, the repack.usedeltabaseoffset config option will default
> +   to true, which will give denser packfile (i.e. more efficient storage).
> +   The downside is that git older than version 1.4.4 will not be able
> +   to directly use a repository packed using this setting.
> +
> + * From v1.5.5, the pack.indexversion config option will default to 2,
> +   which is slightly more efficient, and makes repacking more immune to
> +   data corruptions.  Git older than version 1.5.2 may revert to version 1
> +   of the pack index with a manual "git index-pack" to be able to directly
> +   access corresponding pack files.

Which means what? Local clone with shortcut (hardlinking and remotes)?
Dumb protocols (http, ftp, rsync)?

-- 
Jakub Narebski
Poland
ShadeHawk on #git

Re: [PATCH] provide advance warning of some future pack default changes

[Nicolas Pitre]

On Thu, 13 Dec 2007, Junio C Hamano wrote:

> Thanks.
> 
> Deprecating versions before 1.5.2 (May 20 2007) feels a bit too quick,
> but seven month is almost an eternity in git timescale, and by now
> anything older than 1.5.2 can safely be called prehistoric.  Will apply.

Well, index version 1 is not gone.  It's only the version used by 
default that will change, which can be overriden with a config variable.

And even if that wasn't done, then any old Git version can just blow 
away the new index and recreate it.  So it is not like if you actually 
needed a recent version of Git to convert the repo back to the old 
format.

And all this will be effective in 1.5.5 which is still a few months 
ahead.


Nicolas

Re: [PATCH] provide advance warning of some future pack default changes

[Nicolas Pitre]

On Fri, 14 Dec 2007, Jakub Narebski wrote:

> Nicolas Pitre <nico@cam.org> writes:
> 
> > + * From v1.5.5, the repack.usedeltabaseoffset config option will default
> > +   to true, which will give denser packfile (i.e. more efficient storage).
> > +   The downside is that git older than version 1.4.4 will not be able
> > +   to directly use a repository packed using this setting.
> > +
> > + * From v1.5.5, the pack.indexversion config option will default to 2,
> > +   which is slightly more efficient, and makes repacking more immune to
> > +   data corruptions.  Git older than version 1.5.2 may revert to version 1
> > +   of the pack index with a manual "git index-pack" to be able to directly
> > +   access corresponding pack files.
> 
> Which means what? Local clone with shortcut (hardlinking and remotes)?
> Dumb protocols (http, ftp, rsync)?

Right, or simply shared repo over NFS or the like.

The 1.5.5 release notes will contain a note reminding people to set the 
corresponding config variables if they wish to retain the legacy 
behaviors.


Nicolas

Re: [PATCH] provide advance warning of some future pack default changes

[Nicolas Pitre]

On Fri, 14 Dec 2007, linux@horizon.com wrote:

> >+ * From v1.5.5, the pack.indexversion config option will default to 2,
> >+   which is slightly more efficient, and makes repacking more immune to
> >+   data corruptions.  Git older than version 1.5.2 may revert to version 1
> >+   of the pack index with a manual "git index-pack" to be able to directly
> >+   access corresponding pack files.
> 
> You might want to mention that it's slightly more TIME efficient,
> but takes 16% more space (28 bytes per object rather than 24).

Well, sure, but not now.  This is just an advance warning of what 
the next release after this one will do.

> If it helps, I documented the v2 index file format (a lot stolen
> from commit c553ca25bd60dc9fd50b8bc7bd329601b81cee66 message).
> (Public domain, copyright abandoned, if it breaks you get to keep both
> pieces, yadda yadda.)

If anything:

Acked-by: Nicolas Pitre <nico@cam.org>


> diff --git a/Documentation/technical/pack-format.txt b/Documentation/technical/pack-format.txt
> index e5b31c8..a80baa4 100644
> --- a/Documentation/technical/pack-format.txt
> +++ b/Documentation/technical/pack-format.txt
> @@ -1,9 +1,9 @@
>  GIT pack format
>  ===============
>  
> -= pack-*.pack file has the following format:
> += pack-*.pack files have the following format:
>  
> -   - The header appears at the beginning and consists of the following:
> +   - A header appears at the beginning and consists of the following:
>  
>       4-byte signature:
>           The signature is: {'P', 'A', 'C', 'K'}
> @@ -34,18 +34,14 @@ GIT pack format
>  
>    - The trailer records 20-byte SHA1 checksum of all of the above.
>  
> -= pack-*.idx file has the following format:
> += Original (version 1) pack-*.idx files have the following format:
>  
>    - The header consists of 256 4-byte network byte order
>      integers.  N-th entry of this table records the number of
>      objects in the corresponding pack, the first byte of whose
> -    object name are smaller than N.  This is called the
> +    object name is less than or equal to N.  This is called the
>      'first-level fan-out' table.
>  
> -    Observation: we would need to extend this to an array of
> -    8-byte integers to go beyond 4G objects per pack, but it is
> -    not strictly necessary.
> -
>    - The header is followed by sorted 24-byte entries, one entry
>      per object in the pack.  Each entry is:
>  
> @@ -55,10 +51,6 @@ GIT pack format
>  
>      20-byte object name.
>  
> -    Observation: we would definitely need to extend this to
> -    8-byte integer plus 20-byte object name to handle a packfile
> -    that is larger than 4GB.
> -
>    - The file is concluded with a trailer:
>  
>      A copy of the 20-byte SHA1 checksum at the end of
> @@ -68,31 +60,30 @@ GIT pack format
>  
>  Pack Idx file:
>  
> -	idx
> -	    +--------------------------------+
> -	    | fanout[0] = 2                  |-.
> -	    +--------------------------------+ |
> +	--  +--------------------------------+
> +fanout	    | fanout[0] = 2 (for example)    |-.
> +table	    +--------------------------------+ |
>  	    | fanout[1]                      | |
>  	    +--------------------------------+ |
>  	    | fanout[2]                      | |
>  	    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
> -	    | fanout[255]                    | |
> -	    +--------------------------------+ |
> -main	    | offset                         | |
> -index	    | object name 00XXXXXXXXXXXXXXXX | |
> -table	    +--------------------------------+ |
> -	    | offset                         | |
> -	    | object name 00XXXXXXXXXXXXXXXX | |
> -	    +--------------------------------+ |
> -	  .-| offset                         |<+
> -	  | | object name 01XXXXXXXXXXXXXXXX |
> -	  | +--------------------------------+
> -	  | | offset                         |
> -	  | | object name 01XXXXXXXXXXXXXXXX |
> -	  | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> -	  | | offset                         |
> -	  | | object name FFXXXXXXXXXXXXXXXX |
> -	  | +--------------------------------+
> +	    | fanout[255] = total objects    |---.
> +	--  +--------------------------------+ | |
> +main	    | offset                         | | |
> +index	    | object name 00XXXXXXXXXXXXXXXX | | |
> +table	    +--------------------------------+ | |
> +	    | offset                         | | |
> +	    | object name 00XXXXXXXXXXXXXXXX | | |
> +	    +--------------------------------+<+ |
> +	  .-| offset                         |   |
> +	  | | object name 01XXXXXXXXXXXXXXXX |   |
> +	  | +--------------------------------+   |
> +	  | | offset                         |   |
> +	  | | object name 01XXXXXXXXXXXXXXXX |   |
> +	  | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~   |
> +	  | | offset                         |   |
> +	  | | object name FFXXXXXXXXXXXXXXXX |   |
> +	--| +--------------------------------+<--+
>  trailer	  | | packfile checksum              |
>  	  | +--------------------------------+
>  	  | | idxfile checksum               |
> @@ -116,3 +107,40 @@ Pack file entry: <+
>  	  20-byte base object name SHA1 (the size above is the
>  		size of the delta data that follows).
>            delta data, deflated.
> +
> +
> += Version 2 pack-*.idx files support packs larger than 4 GiB, and
> +  have some other reorganizations.  They have the format:
> +
> +  - A 4-byte magic number '\377tOc' which is an unreasonable
> +    fanout[0] value.
> +
> +  - A 4-byte version number (= 2)
> +
> +  - A 256-entry fan-out table just like v1.
> +
> +  - A table of sorted 20-byte SHA1 object names.  These are
> +    packed together without offset values to reduce the cache
> +    footprint of the binary search for a specific object name.
> +
> +  - A table of 4-byte CRC32 values of the packed object data.
> +    This is new in v2 so compressed data can be copied directly
> +    from pack to pack during repacking withough undetected
> +    data corruption.
> +
> +  - A table of 4-byte offset values (in network byte order).
> +    These are usually 31-bit pack file offsets, but large
> +    offsets are encoded as an index into the next table with
> +    the msbit set.
> +
> +  - A table of 8-byte offset entries (empty for pack files less
> +    than 2 GiB).  Pack files are organized with heavily used
> +    objects toward the front, so most object references should
> +    not need to refer to this table.
> +
> +  - The same trailer as a v1 pack file:
> +
> +    A copy of the 20-byte SHA1 checksum at the end of
> +    corresponding packfile.
> +
> +    20-byte SHA1-checksum of all of the above.
> -
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 


Nicolas

Re: [PATCH] provide advance warning of some future pack default changes

[Joel Becker]

On Fri, Dec 14, 2007 at 08:38:51AM -0500, Nicolas Pitre wrote:
> On Fri, 14 Dec 2007, Jakub Narebski wrote:
> > Which means what? Local clone with shortcut (hardlinking and remotes)?
> > Dumb protocols (http, ftp, rsync)?
> 
> Right, or simply shared repo over NFS or the like.
> 
> The 1.5.5 release notes will contain a note reminding people to set the 
> corresponding config variables if they wish to retain the legacy 
> behaviors.

	We've seen that release notes are a poor way to communicate
this.  What will happen to a 1.4.4 user when they try to access the
repository?  Corruption, cryptic error message, or clean "this repo is
not compatible" message?

Joel

-- 

"Depend on the rabbit's foot if you will, but remember, it didn't
 help the rabbit."
	- R. E. Shay

Joel Becker
Principal Software Developer
Oracle
E-mail: joel.becker@oracle.com
Phone: (650) 506-8127

Re: [PATCH] provide advance warning of some future pack default changes

[Nicolas Pitre]

On Fri, 14 Dec 2007, Joel Becker wrote:

> On Fri, Dec 14, 2007 at 08:38:51AM -0500, Nicolas Pitre wrote:
> > On Fri, 14 Dec 2007, Jakub Narebski wrote:
> > > Which means what? Local clone with shortcut (hardlinking and remotes)?
> > > Dumb protocols (http, ftp, rsync)?
> > 
> > Right, or simply shared repo over NFS or the like.
> > 
> > The 1.5.5 release notes will contain a note reminding people to set the 
> > corresponding config variables if they wish to retain the legacy 
> > behaviors.
> 
> 	We've seen that release notes are a poor way to communicate
> this.  What will happen to a 1.4.4 user when they try to access the
> repository?  Corruption, cryptic error message, or clean "this repo is
> not compatible" message?

There won't be any corruption.

In the best case there will be a message along "x is not supported by 
this version of Git -- please consider upgrading".  In the worst case 
it'll say "x is bad".

But you know what? repositories with the change affecting 1.4.4 users 
are _already_ out there and no one complained recently.  Anyone pushing 
changes over the native Git protocol is already using deltabaseoffset as 
the native protocol negociate that capability in its handshake, and 
these days we keep packs as is on the receiving side when they're large 
enough.


Nicolas

Re: [PATCH] provide advance warning of some future pack default changes

[Joel Becker]

On Fri, Dec 14, 2007 at 05:34:49PM -0500, Nicolas Pitre wrote:
> On Fri, 14 Dec 2007, Joel Becker wrote:
> > 	We've seen that release notes are a poor way to communicate
> > this.  What will happen to a 1.4.4 user when they try to access the
> > repository?  Corruption, cryptic error message, or clean "this repo is
> > not compatible" message?
> 
> There won't be any corruption.
> 
> In the best case there will be a message along "x is not supported by 
> this version of Git -- please consider upgrading".  In the worst case 
> it'll say "x is bad".

	That would be excellent, especially the former message.
 
> But you know what? repositories with the change affecting 1.4.4 users 
> are _already_ out there and no one complained recently.  Anyone pushing 

	I did, as did people I work with.  It's on git-list, even.  I'm
pretty sure it corrupted too.

Joel

-- 

"Practice random acts of kindness and senseless acts of beauty."

 Oh, and don't forget where your towel is.

Joel Becker
Principal Software Developer
Oracle
E-mail: joel.becker@oracle.com
Phone: (650) 506-8127

Re: [PATCH] provide advance warning of some future pack default changes

[Nicolas Pitre]

On Fri, 14 Dec 2007, Joel Becker wrote:

> On Fri, Dec 14, 2007 at 05:34:49PM -0500, Nicolas Pitre wrote:
> > But you know what? repositories with the change affecting 1.4.4 users 
> > are _already_ out there and no one complained recently.  Anyone pushing 
> 
> 	I did, as did people I work with.  It's on git-list, even.  I'm
> pretty sure it corrupted too.

Could you please give me a reference to such message, so to verify that 
we're actually talking about the same thing?


Nicolas

Re: [PATCH] provide advance warning of some future pack default changes

[Joel Becker]

On Fri, Dec 14, 2007 at 05:46:14PM -0500, Nicolas Pitre wrote:
> On Fri, 14 Dec 2007, Joel Becker wrote:
> 
> > On Fri, Dec 14, 2007 at 05:34:49PM -0500, Nicolas Pitre wrote:
> > > But you know what? repositories with the change affecting 1.4.4 users 
> > > are _already_ out there and no one complained recently.  Anyone pushing 
> > 
> > 	I did, as did people I work with.  It's on git-list, even.  I'm
> > pretty sure it corrupted too.
> 
> Could you please give me a reference to such message, so to verify that 
> we're actually talking about the same thing?

	The relevant message is:

Message-ID: <7vveaindgp.fsf@gitster.siamese.dyndns.org>

See the paragraphs at the bottom.  The thread, started by me, begins
with:

Message-ID: <20070910205429.GE27837@tasint.org>

Joel

-- 

"You must remember this:
 A kiss is just a kiss,
 A sigh is just a sigh.
 The fundamental rules apply
 As time goes by."

Joel Becker
Principal Software Developer
Oracle
E-mail: joel.becker@oracle.com
Phone: (650) 506-8127

Re: [PATCH] provide advance warning of some future pack default changes

[Nicolas Pitre]

On Fri, 14 Dec 2007, Joel Becker wrote:

> On Fri, Dec 14, 2007 at 05:46:14PM -0500, Nicolas Pitre wrote:
> > On Fri, 14 Dec 2007, Joel Becker wrote:
> > 
> > > On Fri, Dec 14, 2007 at 05:34:49PM -0500, Nicolas Pitre wrote:
> > > > But you know what? repositories with the change affecting 1.4.4 users 
> > > > are _already_ out there and no one complained recently.  Anyone pushing 
> > > 
> > > 	I did, as did people I work with.  It's on git-list, even.  I'm
> > > pretty sure it corrupted too.
> > 
> > Could you please give me a reference to such message, so to verify that 
> > we're actually talking about the same thing?
> 
> 	The relevant message is:
> 
> Message-ID: <7vveaindgp.fsf@gitster.siamese.dyndns.org>
> 
> See the paragraphs at the bottom.  The thread, started by me, begins
> with:
> 
> Message-ID: <20070910205429.GE27837@tasint.org>

I don't have such emails in my mail folders anymore.


Nicolas

Re: [PATCH] provide advance warning of some future pack default changes

[Johannes Schindelin]

Hi,

On Fri, 14 Dec 2007, Nicolas Pitre wrote:

> On Fri, 14 Dec 2007, Joel Becker wrote:
> 
> > On Fri, Dec 14, 2007 at 05:46:14PM -0500, Nicolas Pitre wrote:
> > > On Fri, 14 Dec 2007, Joel Becker wrote:
> > > 
> > > > On Fri, Dec 14, 2007 at 05:34:49PM -0500, Nicolas Pitre wrote:
> > > > > But you know what? repositories with the change affecting 1.4.4 users 
> > > > > are _already_ out there and no one complained recently.  Anyone pushing 
> > > > 
> > > > 	I did, as did people I work with.  It's on git-list, even.  I'm
> > > > pretty sure it corrupted too.
> > > 
> > > Could you please give me a reference to such message, so to verify that 
> > > we're actually talking about the same thing?
> > 
> > 	The relevant message is:
> > 
> > Message-ID: <7vveaindgp.fsf@gitster.siamese.dyndns.org>
> > 
> > See the paragraphs at the bottom.  The thread, started by me, begins
> > with:
> > 
> > Message-ID: <20070910205429.GE27837@tasint.org>
> 
> I don't have such emails in my mail folders anymore.

GMane does not seem to have it, either:

	http://mid.gmane.org/20070910205429.GE27837@tasint.org

returns "No such article".

Ciao,
Dscho

Re: [PATCH] provide advance warning of some future pack default changes

[Junio C Hamano]

Nicolas Pitre <nico@cam.org> writes:

> On Fri, 14 Dec 2007, Joel Becker wrote:
>
>> > Could you please give me a reference to such message, so to verify that 
>> > we're actually talking about the same thing?
>> 
>> 	The relevant message is:
>> 
>> Message-ID: <7vveaindgp.fsf@gitster.siamese.dyndns.org>
>> 
>> See the paragraphs at the bottom.  The thread, started by me, begins
>> with:
>> 
>> Message-ID: <20070910205429.GE27837@tasint.org>
>
> I don't have such emails in my mail folders anymore.

-- >8 --

Date:	Mon, 10 Sep 2007 13:54:29 -0700
From:	Joel Becker <Joel.Becker@oracle.com>
To:	git@vger.kernel.org
Subject: Remote branches and better documentation
Message-ID: <20070910205429.GE27837@tasint.org>
Sender:	git-owner@vger.kernel.org

Junio et al,
	Git is a fast moving target, so some of this obviously needs a
grain of salt.  However, I'd like to make a couple of humble suggestions
and ask one simple question.
	First, the question:  Is there a syntax to git clone that
creates the old-style branches?  That is, you get all the branches
locally, for people that either haven't learned "git branch -r" or have
existing scripts that expect the branch to exist?  I can't find anything
in the git clone manpage.
	The suggestions are pretty simple.  First, when behavior is
changed invisibly (as the remote branch stuff was), can we note it in
the documentation?  I don't mean the ChangeLog, I mean the manpage.  I
personally already knew about "branch -r" because I read this list.  A
coworker of mine, who just uses git, spent an hour trying to find his
branches after a clone with git 1.5.  He thought his clone had failed.
He read the manpage, and there was no big "Hey, those of you used to
the old behavior, it changed!".  The single sentence about "remote
tracking branches" clearly isn't enough for folks that don't follow the
development side.  If we're going to take the liberty of changing
expected behavior silently, we should be giving it its own section in
the manpage.
	The second suggestion is related.  When an invisible change has
made the repository incompatible with older versions, we should make
sure that things behave.  We had some repositories cloned via 1.4.2.  Do
some work with 1.5.0.6 (on a different machine), then go back to the
machine with 1.4.2, and 1.4.2 doesn't work.  In fact, it can mess things
up.  He was doing simple things: pull from Linus, switch branches, etc.
If this is going to be incompatible, then the newer stuff should at
least warn about it, if not outright prevent 1.4 from running.
	These sorts of things make fast-moving changes workable.

Joel

-- >8 --

Date:	Mon, 10 Sep 2007 19:27:34 -0700
Message-ID: <7vveaindgp.fsf@gitster.siamese.dyndns.org>
Sender:	git-owner@vger.kernel.org

Joel Becker <Joel.Becker@oracle.com> writes:

> On Tue, Sep 11, 2007 at 02:05:34AM +0200, Wincent Colaiuta wrote:
>> But that's precisely the group release notes are for; existing users who 
>> need to be informed of any changes to the way things work.
>
> 	No one reads the changelogs of 100 packages updated via "yum
> update".  Heck, they don't even see the list of packages.  They just
> switch to a different desktop while it runs.

Distros are not something under my control, so I cannot help you
much there.

> 	Then there's the user that doesn't administer the system.  They
> don't even know the version changed.  It Just Breaks, and they don't
> know why.

That's a valid concern, but I am not sure how you would want to
address that issue.  Design constraints are:

 - you cannot change the old software that is not updated on the
   user's box;

 - you cannot afford to write something to the repository to
   mark the latest version that mucked with the repository every
   time any operation happens;

We _could_ check presence of $HOME/.knows-git-version-X.Y.Z file
every time we run (that's just a single stat(2) call that cannot
be too expensive) and if there isn't one, ask the user if he has
read the release notes and understood the backward compatibility
issues if there is any, and refuse to run until getting a
satisfactory answer.

But I personally do not think that would be an improvement.

After reviewing Release Notes for v1.5.0, I do not think we
could have done much better, unfortunately.

    As of git v1.5.0 there are some optional features that changes
    the repository to allow data to be stored and transferred more
    efficiently.  These features are not enabled by default, as they
    will make the repository unusable with older versions of git.
    Specifically, the available options are:

     - There is a configuration variable core.legacyheaders that
       changes the format of loose objects so that they are more
       efficient to pack and to send out of the repository over git
       native protocol, since v1.4.2.  However, loose objects
       written in the new format cannot be read by git older than
       that version; people fetching from your repository using
       older clients over dumb transports (e.g. http) using older
       versions of git will also be affected.

       To let git use the new loose object format, you have to
       set core.legacyheaders to false.

     - Since v1.4.3, configuration repack.usedeltabaseoffset allows
       packfile to be created in more space efficient format, which
       cannot be read by git older than that version.

       To let git use the new format for packfiles, you have to
       set repack.usedeltabaseoffset to true.

    The above two new features are not enabled by default and you
    have to explicitly ask for them, because they make repositories
    unreadable by older versions of git, and in v1.5.0 we still do
    not enable them by default for the same reason.  We will change
    this default probably 1 year after 1.4.2's release, when it is
    reasonable to expect everybody to have new enough version of
    git.

     - 'git pack-refs' appeared in v1.4.4; this command allows tags
       to be accessed much more efficiently than the traditional
       'one-file-per-tag' format.  Older git-native clients can
       still fetch from a repository that packed and pruned refs
       (the server side needs to run the up-to-date version of git),
       but older dumb transports cannot.  Packing of refs is done by
       an explicit user action, either by use of "git pack-refs
       --prune" command or by use of "git gc" command.

So everything was opt in and clearly marked as such.  You may
not have read it, distros may not have shown it, but then that
is something we cannot do much about, unfortunately.

I think there was _one_ honest slippage though.  Fetching from
1.5.0 peer by 1.5.0 client could (after doing content
negotiation between both ends as a protection measure) create a
packfile that cannot be read by older 1.4 clients.  Obviously
you cannot expect that kind of "protection" to work across set
of machines with mixed versions sharing a repository over NFS,
and that probably is a mistake we can learn from.

Re: [PATCH] provide advance warning of some future pack default changes

[Nicolas Pitre]

On Fri, 14 Dec 2007, Joel Becker wrote:

> On Fri, Dec 14, 2007 at 05:46:14PM -0500, Nicolas Pitre wrote:
> > On Fri, 14 Dec 2007, Joel Becker wrote:
> > 
> > > On Fri, Dec 14, 2007 at 05:34:49PM -0500, Nicolas Pitre wrote:
> > > > But you know what? repositories with the change affecting 1.4.4 users 
> > > > are _already_ out there and no one complained recently.  Anyone pushing 
> > > 
> > > 	I did, as did people I work with.  It's on git-list, even.  I'm
> > > pretty sure it corrupted too.
> > 
> > Could you please give me a reference to such message, so to verify that 
> > we're actually talking about the same thing?
> 
> 	The relevant message is:
> 
> Message-ID: <7vveaindgp.fsf@gitster.siamese.dyndns.org>
> 
> See the paragraphs at the bottom.  The thread, started by me, begins
> with:
> 
> Message-ID: <20070910205429.GE27837@tasint.org>

OK.  From those emails Junio forwarded to me, I don't see any case for 
actual _corruptions_.  Git does indeed refuse to work with unknown pack 
index or unknown objects in a pack.  Really old versions were not overly 
clueful as to why they refused to work, but they should never corrupt a 
pack which, for all purposes, is always read-only anyway.


Nicolas

Re: [PATCH] provide advance warning of some future pack default changes

[Joel Becker]

On Fri, Dec 14, 2007 at 09:23:38PM -0500, Nicolas Pitre wrote:
> On Fri, 14 Dec 2007, Joel Becker wrote:
> > 	The relevant message is:
> > 
> > Message-ID: <7vveaindgp.fsf@gitster.siamese.dyndns.org>
> > 
> > See the paragraphs at the bottom.  The thread, started by me, begins
> > with:
> > 
> > Message-ID: <20070910205429.GE27837@tasint.org>
> 
> OK.  From those emails Junio forwarded to me, I don't see any case for 
> actual _corruptions_.  Git does indeed refuse to work with unknown pack 
> index or unknown objects in a pack.  Really old versions were not overly 
> clueful as to why they refused to work, but they should never corrupt a 
> pack which, for all purposes, is always read-only anyway.

	You may not see a case for actual corruptions, but my coworker
updated his tree on a box with 1.5.x, then tried to work on a box with
1.4.x (I think 1.4.2 back then), and ended up with a tree that was
unusable.  He had to re-clone, and I think he got lucky recovering
pending changes (probably using 1.5.x on the branches with the changes,
as master was what got broken).
	My point is not that change is always bad, but that we should
really look forward to what we're doing, and that "it's in the release
notes" is not sufficient if an older git gives an incomprehensible error
or a silent problem.  I was responding to the cavalier statement "well,
it's in the release notes, so don't worry about older versions".

Joel

-- 

"Vote early and vote often." 
        - Al Capone

Joel Becker
Principal Software Developer
Oracle
E-mail: joel.becker@oracle.com
Phone: (650) 506-8127

Re: [PATCH] provide advance warning of some future pack default changes

[Nicolas Pitre]

On Mon, 17 Dec 2007, Joel Becker wrote:

> On Fri, Dec 14, 2007 at 09:23:38PM -0500, Nicolas Pitre wrote:
> > On Fri, 14 Dec 2007, Joel Becker wrote:
> > > 	The relevant message is:
> > > 
> > > Message-ID: <7vveaindgp.fsf@gitster.siamese.dyndns.org>
> > > 
> > > See the paragraphs at the bottom.  The thread, started by me, begins
> > > with:
> > > 
> > > Message-ID: <20070910205429.GE27837@tasint.org>
> > 
> > OK.  From those emails Junio forwarded to me, I don't see any case for 
> > actual _corruptions_.  Git does indeed refuse to work with unknown pack 
> > index or unknown objects in a pack.  Really old versions were not overly 
> > clueful as to why they refused to work, but they should never corrupt a 
> > pack which, for all purposes, is always read-only anyway.
> 
> 	You may not see a case for actual corruptions, but my coworker
> updated his tree on a box with 1.5.x, then tried to work on a box with
> 1.4.x (I think 1.4.2 back then), and ended up with a tree that was
> unusable.  He had to re-clone, and I think he got lucky recovering
> pending changes (probably using 1.5.x on the branches with the changes,
> as master was what got broken).

I still claim that there wasn't any corruptions.

Just for fun, just edit some document with Microsoft Office 95, then 
open the same document with Office 2007 and save it with default 
settings.  Now try to open it back with Office 95.  It won't work.  
Does that mean that the document got corrupted?

> 	My point is not that change is always bad, but that we should
> really look forward to what we're doing, and that "it's in the release
> notes" is not sufficient if an older git gives an incomprehensible error
> or a silent problem.  I was responding to the cavalier statement "well,
> it's in the release notes, so don't worry about older versions".

Your allegation of corruptions is cavalier just as well.

I'm telling you that there won't be any such corruption.  Just like in 
the M$ Office case, it is expected that newer versions make data 
unusable by older versions at some point -- that's the inevitable side 
effect of progress.

And we cannot always anticipate what kind of incompatibility will be 
worth making in the future, so it is hard to come with proper error 
messages in all cases today.

Recent enough Git versions do suggest upgrading when they refuse to 
access repositories though, and later Git versions can be configured to 
remain compatible with old Git versions.  And we also document it.

And when you still cannot figure it out on your own, then there is that 
free support available on a public mailing list, and even an IRC 
channel.

So I don't see how we could do better in that regard.  Carving the 
repository format in stone to keep ancient versions working forever is 
_not_ a solution.


Nicolas

Re: [PATCH] provide advance warning of some future pack default changes

[Joel Becker]

On Mon, Dec 17, 2007 at 03:41:24PM -0500, Nicolas Pitre wrote:
> On Mon, 17 Dec 2007, Joel Becker wrote:
> > 	You may not see a case for actual corruptions, but my coworker
> > updated his tree on a box with 1.5.x, then tried to work on a box with
> > 1.4.x (I think 1.4.2 back then), and ended up with a tree that was
> > unusable.  He had to re-clone, and I think he got lucky recovering
> > pending changes (probably using 1.5.x on the branches with the changes,
> > as master was what got broken).
> 
> I still claim that there wasn't any corruptions.
> 
> Just for fun, just edit some document with Microsoft Office 95, then 
> open the same document with Office 2007 and save it with default 
> settings.  Now try to open it back with Office 95.  It won't work.  
> Does that mean that the document got corrupted?

	No, but when you try to re-open it with Office 2007, you expect
it to work, don't you?  His master was messed up even for 1.5.x.  It was
now months ago, so I don't quite remember all the details, but I think
you'd agree that "1.5.x no longer works" is not correct.

> I'm telling you that there won't be any such corruption.  Just like in 
> the M$ Office case, it is expected that newer versions make data 
> unusable by older versions at some point -- that's the inevitable side 
> effect of progress.

	Sure, we're not complaining about that.  We complain some about
the fast pace (at the time he had his problem, 1.4 installs were not
unusual, and Junio's response suggested that "I use NFS" wasn't strongly
considered as a use case), but more we complain about the obscurity of
the reason.  If it's obvious what happened (not the specifics, just
"please upgrade" or "repository format changed" or something), the user
moves along.

> And we cannot always anticipate what kind of incompatibility will be 
> worth making in the future, so it is hard to come with proper error 
> messages in all cases today.

	How hard is it?  We have core.repositoryformatversion.  We
undoubtably have headers on our files.  As an example, an older version
should be able to ascertain 1) this is a pack file 2) I don't know how
to read it.  Thus, it should always be able to tell the user as such.
This is different from reporting "invalid pack file" or "corrupt pack
file", or "garbage in tree".  Filesystems, as an example, set
compatibility bits or version levels.  When an old kernel tries to mount
it, it does not say "corrupt filesystem", it says "this filesystem has a
feature I don't understand, I'm going to be nice and not do anything,
please upgrade".  This is clear, even though the older kernel doesn't
have any specifics about what the new feature is.

> So I don't see how we could do better in that regard.  Carving the 
> repository format in stone to keep ancient versions working forever is 
> _not_ a solution.

	Once again, we're not asking for that.  We're asking that you
think ahead to what can change, and plan for it, so you can tell the
user.  If the user has a clear idea where to go next, the can solve the
rest themselves.
	Look, not everyone reads this mailing list.  No one outside of
this list reads the Release Notes.  They get their upgrade via yum or
apt-get, along with 100 other packages.  You can't assume that 3 months
of feature discussion here is going to be known to your average user.

Joel

-- 

"Vote early and vote often." 
        - Al Capone

Joel Becker
Principal Software Developer
Oracle
E-mail: joel.becker@oracle.com
Phone: (650) 506-8127

Re: [PATCH] provide advance warning of some future pack default changes

[Junio C Hamano]

Nicolas Pitre <nico@cam.org> writes:

> On Mon, 17 Dec 2007, Joel Becker wrote:
>
>> 	You may not see a case for actual corruptions, but my coworker
>> updated his tree on a box with 1.5.x, then tried to work on a box with
>> 1.4.x (I think 1.4.2 back then), and ended up with a tree that was
>> unusable.  He had to re-clone, and I think he got lucky recovering
>> pending changes (probably using 1.5.x on the branches with the changes,
>> as master was what got broken).
>
> I still claim that there wasn't any corruptions.
> ...
> Your allegation of corruptions is cavalier just as well.
>
> I'm telling you that there won't be any such corruption.  Just like in 
> the M$ Office case, it is expected that newer versions make data 
> unusable by older versions at some point -- that's the inevitable side 
> effect of progress.

This is mostly spilt milk under the bridge now, but I have to mildly
disagree.

If we had core.usedeltabaseoffset instead of repack.usedeltabaseoffset,
and made the format negotiation in fetch-pack protocol pay attention to
that variable, Joel's coworker did not have to suffer if the repository
explicitly asked OFS_DELTA not to be used.

Instead we unconditionally said "if you are downloading with the new
client, we assume you would never be using older client to access that
repository locally, if you did so, you are screwed."

IOW, I think e4fe4b8ef7cdde842a9e5e2594d0fba1367d9dd3 (let the GIT
native protocol use offsets to delta base when possible) could have been
a bit more careful in this respect.

Re: [PATCH] provide advance warning of some future pack default changes

[J. Bruce Fields]

On Mon, Dec 17, 2007 at 01:13:18PM -0800, Joel Becker wrote:
> 	Sure, we're not complaining about that.  We complain some about
> the fast pace (at the time he had his problem, 1.4 installs were not
> unusual, and Junio's response suggested that "I use NFS" wasn't strongly
> considered as a use case), but more we complain about the obscurity of
> the reason.  If it's obvious what happened (not the specifics, just
> "please upgrade" or "repository format changed" or something), the user
> moves along.

By the way, just as a data point: I do keep some git repositories on
NFS, and access them from multiple machines with different git versions
(not on purpose--it's just that the machines don't all run the same
distro, so it'd be extra work to give them all the same version).  I
don't use anything older than 1.5.0.  If the repository became unusable
on one of those machines without warning it'd be annoying.

---b.

Re: [PATCH] provide advance warning of some future pack default changes

[Nicolas Pitre]

On Mon, 17 Dec 2007, Junio C Hamano wrote:

> This is mostly spilt milk under the bridge now, but I have to mildly
> disagree.
> 
> If we had core.usedeltabaseoffset instead of repack.usedeltabaseoffset,
> and made the format negotiation in fetch-pack protocol pay attention to
> that variable, Joel's coworker did not have to suffer if the repository
> explicitly asked OFS_DELTA not to be used.
> 
> Instead we unconditionally said "if you are downloading with the new
> client, we assume you would never be using older client to access that
> repository locally, if you did so, you are screwed."
> 
> IOW, I think e4fe4b8ef7cdde842a9e5e2594d0fba1367d9dd3 (let the GIT
> native protocol use offsets to delta base when possible) could have been
> a bit more careful in this respect.

Probably.  But this can hardly be called a "corruption" since nothing 
was actually lost, rather an incompatibility problem.

If, on the other hand, the latest Git version wasn't able to read it 
either then this is a different matter entirely.


Nicolas

Re: [PATCH] provide advance warning of some future pack default changes

[Nicolas Pitre]

On Mon, 17 Dec 2007, J. Bruce Fields wrote:

> By the way, just as a data point: I do keep some git repositories on
> NFS, and access them from multiple machines with different git versions
> (not on purpose--it's just that the machines don't all run the same
> distro, so it'd be extra work to give them all the same version).  I
> don't use anything older than 1.5.0.  If the repository became unusable
> on one of those machines without warning it'd be annoying.

What the v1.5.5 release notes will say is that you'll have to set 
pack.indexversion=1 to remain compatible with pre-1.5.2 Git versions.  

And even if you forget about it then there'll be a simple way to regain 
compatibility after the facts.


Nicolas

Re: [PATCH] provide advance warning of some future pack default changes

[J. Bruce Fields]

On Mon, Dec 17, 2007 at 04:52:16PM -0500, Nicolas Pitre wrote:
> On Mon, 17 Dec 2007, J. Bruce Fields wrote:
> 
> > By the way, just as a data point: I do keep some git repositories on
> > NFS, and access them from multiple machines with different git versions
> > (not on purpose--it's just that the machines don't all run the same
> > distro, so it'd be extra work to give them all the same version).  I
> > don't use anything older than 1.5.0.  If the repository became unusable
> > on one of those machines without warning it'd be annoying.
> 
> What the v1.5.5 release notes will say is that you'll have to set 
> pack.indexversion=1 to remain compatible with pre-1.5.2 Git versions.  

Is there any reason not to make pack.indexversion=1 the default (for
preexisting repositories at the very least) and suggest in the release
notes that people set something else if they want the features the new
version provides?

--b.

> And even if you forget about it then there'll be a simple way to regain 
> compatibility after the facts.

Re: [PATCH] provide advance warning of some future pack default changes

[Nicolas Pitre]

On Mon, 17 Dec 2007, J. Bruce Fields wrote:

> On Mon, Dec 17, 2007 at 04:52:16PM -0500, Nicolas Pitre wrote:
> > On Mon, 17 Dec 2007, J. Bruce Fields wrote:
> > 
> > > By the way, just as a data point: I do keep some git repositories on
> > > NFS, and access them from multiple machines with different git versions
> > > (not on purpose--it's just that the machines don't all run the same
> > > distro, so it'd be extra work to give them all the same version).  I
> > > don't use anything older than 1.5.0.  If the repository became unusable
> > > on one of those machines without warning it'd be annoying.
> > 
> > What the v1.5.5 release notes will say is that you'll have to set 
> > pack.indexversion=1 to remain compatible with pre-1.5.2 Git versions.  
> 
> Is there any reason not to make pack.indexversion=1 the default (for
> preexisting repositories at the very least) and suggest in the release
> notes that people set something else if they want the features the new
> version provides?

That's already the case now.

But the thing is that index version 2 is better and actually plug a flaw 
in the repacking process where unnoticed corruption could be repacked 
otherwise.  So for better repo integrity sake, it has to become the 
default at some point.


Nicolas

Re: [PATCH] provide advance warning of some future pack default changes

[Junio C Hamano]

"J. Bruce Fields" <bfields@fieldses.org> writes:

> On Mon, Dec 17, 2007 at 04:52:16PM -0500, Nicolas Pitre wrote:
>> On Mon, 17 Dec 2007, J. Bruce Fields wrote:
>> 
>> > By the way, just as a data point: I do keep some git repositories on
>> > NFS, and access them from multiple machines with different git versions
>> > (not on purpose--it's just that the machines don't all run the same
>> > distro, so it'd be extra work to give them all the same version).  I
>> > don't use anything older than 1.5.0.  If the repository became unusable
>> > on one of those machines without warning it'd be annoying.
>> 
>> What the v1.5.5 release notes will say is that you'll have to set 
>> pack.indexversion=1 to remain compatible with pre-1.5.2 Git versions.  
>
> Is there any reason not to make pack.indexversion=1 the default (for
> preexisting repositories at the very least) and suggest in the release
> notes that people set something else if they want the features the new
> version provides?

That's a judgement call.

Pack-idx format v2 is by design much safer in the face of bitflip (do we
have a test case to make sure this is indeed true?).  But from the end
user's point of view, all the usual "I do not want to be forced to
update that old box I do not want to touch" applies.

And the people who needs to suffer from the dilemma are only the ones
who access a single repository across NFS with git from different
vintage.  If that is a minority and/or tends to be more clueful people,
the inconvenience factor may be outweighed by the advantage v2 offers,
and pushing adoption of v2 harder the way Nico is driving at would
generally be a good thing.

Re: [PATCH] provide advance warning of some future pack default changes

[J. Bruce Fields]

On Mon, Dec 17, 2007 at 02:17:01PM -0800, Junio C Hamano wrote:
> "J. Bruce Fields" <bfields@fieldses.org> writes:
> 
> > On Mon, Dec 17, 2007 at 04:52:16PM -0500, Nicolas Pitre wrote:
> >> On Mon, 17 Dec 2007, J. Bruce Fields wrote:
> >> 
> >> > By the way, just as a data point: I do keep some git repositories on
> >> > NFS, and access them from multiple machines with different git versions
> >> > (not on purpose--it's just that the machines don't all run the same
> >> > distro, so it'd be extra work to give them all the same version).  I
> >> > don't use anything older than 1.5.0.  If the repository became unusable
> >> > on one of those machines without warning it'd be annoying.
> >> 
> >> What the v1.5.5 release notes will say is that you'll have to set 
> >> pack.indexversion=1 to remain compatible with pre-1.5.2 Git versions.  
> >
> > Is there any reason not to make pack.indexversion=1 the default (for
> > preexisting repositories at the very least) and suggest in the release
> > notes that people set something else if they want the features the new
> > version provides?
> 
> That's a judgement call.

Sure.  And I'm totally unfamiliar with the details here, so don't my let
my judgement weigh too heavily.

> Pack-idx format v2 is by design much safer in the face of bitflip (do we
> have a test case to make sure this is indeed true?).  But from the end
> user's point of view, all the usual "I do not want to be forced to
> update that old box I do not want to touch" applies.
> 
> And the people who needs to suffer from the dilemma are only the ones
> who access a single repository across NFS with git from different
> vintage.

Hm.  We tell people to set up public repo's by doing something like:

	git clone --bare ~/proj proj.git
	touch proj.git/git-daemon-export-ok
	scp -r proj.git example.com:

Is that going to hit the same problem if the public server has an older
git version?  (Servers do tend to be on longer upgrade cycles; the
public server I use was on something 1.4ish till about a month ago.)

--b.

Re: [PATCH] provide advance warning of some future pack default changes

[Junio C Hamano]

"J. Bruce Fields" <bfields@fieldses.org> writes:

> Hm.  We tell people to set up public repo's by doing something like:
>
> 	git clone --bare ~/proj proj.git
> 	touch proj.git/git-daemon-export-ok
> 	scp -r proj.git example.com:
>
> Is that going to hit the same problem if the public server has an older
> git version?

It will, but I think you should teach people --mirror pushing these
days, which was specifically invented for priming the public
repository.

That way, the administrator at example.com, as long as he initializes an
empty repository with suitable daemon-export-ok and necessary hooks
(which can be automated via templates), does not even have to allow you
a full ssh access.

Re: [PATCH] provide advance warning of some future pack default changes

[Nicolas Pitre]

On Mon, 17 Dec 2007, Junio C Hamano wrote:

> Pack-idx format v2 is by design much safer in the face of bitflip (do we
> have a test case to make sure this is indeed true?).

t5302 provides a good demonstration of that.


Nicolas

Re: [PATCH] provide advance warning of some future pack default changes

[J. Bruce Fields]

On Mon, Dec 17, 2007 at 02:55:15PM -0800, Junio C Hamano wrote:
> "J. Bruce Fields" <bfields@fieldses.org> writes:
> 
> > Hm.  We tell people to set up public repo's by doing something like:
> >
> > 	git clone --bare ~/proj proj.git
> > 	touch proj.git/git-daemon-export-ok
> > 	scp -r proj.git example.com:
> >
> > Is that going to hit the same problem if the public server has an older
> > git version?
> 
> It will, but I think you should teach people --mirror pushing these
> days, which was specifically invented for priming the public
> repository.
>
> That way, the administrator at example.com, as long as he initializes an
> empty repository with suitable daemon-export-ok and necessary hooks
> (which can be automated via templates), does not even have to allow you
> a full ssh access.

So the basic instructions would be something like this?:

	ssh example.com "git init --bare myproj.git"
	# (or ask your admin to do the previous step)
	git add remote example example.com:myproj.git
	git push --mirror example

OK, that's neat, thanks.

On the backwards-compatibility issue, though: this won't help the large
number of people who learned to just clone a bare repo and copy it
around, since they aren't of their own initiative going to seek out new
ways of doing things that they think they already know how to do.

--b.

Re: [PATCH] provide advance warning of some future pack default changes

[Junio C Hamano]

Nicolas Pitre <nico@cam.org> writes:

> On Mon, 17 Dec 2007, Junio C Hamano wrote:
> ...
>> Instead we unconditionally said "if you are downloading with the new
>> client, we assume you would never be using older client to access that
>> repository locally, if you did so, you are screwed."
>> 
>> IOW, I think e4fe4b8ef7cdde842a9e5e2594d0fba1367d9dd3 (let the GIT
>> native protocol use offsets to delta base when possible) could have been
>> a bit more careful in this respect.
>
> Probably.  But this can hardly be called a "corruption" since nothing 
> was actually lost, rather an incompatibility problem.

It is not a corruption, but the distinction doesn't matter much to the
end user who wants to get the job done with the data right now.  The
data that was made inaccessible is inaccessible.  The only difference is
that it is recoverable once the user upgrades, but that may be painful,
even though it may be rewarding afterwards and worth doing so, and the
user may not be able to afford doing so right at that moment.

Re: [PATCH] provide advance warning of some future pack default changes

[Mark Fasheh]

Hi,

	Just to "out" myself, I'm the "co-worker" whose name Joel has been
(politely) keeping anonymous.

On Mon, Dec 17, 2007 at 03:41:24PM -0500, Nicolas Pitre wrote:
> > 	You may not see a case for actual corruptions, but my coworker
> > updated his tree on a box with 1.5.x, then tried to work on a box with
> > 1.4.x (I think 1.4.2 back then), and ended up with a tree that was
> > unusable.  He had to re-clone, and I think he got lucky recovering
> > pending changes (probably using 1.5.x on the branches with the changes,
> > as master was what got broken).
> 
> I still claim that there wasn't any corruptions.

The following description is really vague because this was a while ago:

Something made my ocfs2.git tree unusable in that I could no longer do
common tasks, such as git-log, etc without getting messages about corrupted
refs.

I wish I had saved off some of the messages. Sorry.

I had to re-create my git tree several times before I learned by deduction
that it was the older versions of git on some of the machines that were
writing some sort of incompatible format.


> Just for fun, just edit some document with Microsoft Office 95, then 
> open the same document with Office 2007 and save it with default 
> settings.  Now try to open it back with Office 95.  It won't work.  
> Does that mean that the document got corrupted?

Boy, I hope Microsoft Office isn't our bar for compatiblity here...
	--Mark

--
Mark Fasheh
Senior Software Developer, Oracle
mark.fasheh@oracle.com

Re: [PATCH] provide advance warning of some future pack default changes

[Mark Fasheh]

On Mon, Dec 17, 2007 at 04:41:19PM -0800, Junio C Hamano wrote:
> It is not a corruption, but the distinction doesn't matter much to the
> end user who wants to get the job done with the data right now.  The
> data that was made inaccessible is inaccessible.  The only difference is
> that it is recoverable once the user upgrades, but that may be painful,
> even though it may be rewarding afterwards and worth doing so, and the
> user may not be able to afford doing so right at that moment.

Junio, I agree 100% with your description here. This is all about user
experience and data which is silently made inaccessible makes them feel
pretty bad.
	--Mark

--
Mark Fasheh
Senior Software Developer, Oracle
mark.fasheh@oracle.com

Re: [PATCH] provide advance warning of some future pack default changes

[Nicolas Pitre]

On Mon, 17 Dec 2007, Junio C Hamano wrote:

> Nicolas Pitre <nico@cam.org> writes:
> 
> > On Mon, 17 Dec 2007, Junio C Hamano wrote:
> > ...
> >> Instead we unconditionally said "if you are downloading with the new
> >> client, we assume you would never be using older client to access that
> >> repository locally, if you did so, you are screwed."
> >> 
> >> IOW, I think e4fe4b8ef7cdde842a9e5e2594d0fba1367d9dd3 (let the GIT
> >> native protocol use offsets to delta base when possible) could have been
> >> a bit more careful in this respect.
> >
> > Probably.  But this can hardly be called a "corruption" since nothing 
> > was actually lost, rather an incompatibility problem.
> 
> It is not a corruption, but the distinction doesn't matter much to the
> end user who wants to get the job done with the data right now.  The
> data that was made inaccessible is inaccessible.  The only difference is
> that it is recoverable once the user upgrades, but that may be painful,
> even though it may be rewarding afterwards and worth doing so, and the
> user may not be able to afford doing so right at that moment.

Sure, but at some point that's something users mixing versions should be 
ready to cope with.  We try to make it as painless as possible of 
course.

Data corruption is usually something you just cannot recover from 
(unless you have backups).  And if mixing different tool versions 
actually cause data corruption then this is a much much more serious 
issue and that must be avoided.

So at some point the distinction must be made, and if using an old 
version of Git on a repo created by a new version actually produces data 
corruption as Joel seemed to imply, then we must really take it 
seriously.  OTOH, compatibility issues are usually much less of a pain 
to fix.


Nicolas

Re: [PATCH] provide advance warning of some future pack default changes

[Nicolas Pitre]

On Mon, 17 Dec 2007, Mark Fasheh wrote:

> Hi,
> 
> 	Just to "out" myself, I'm the "co-worker" whose name Joel has been
> (politely) keeping anonymous.
> 
> On Mon, Dec 17, 2007 at 03:41:24PM -0500, Nicolas Pitre wrote:
> > > 	You may not see a case for actual corruptions, but my coworker
> > > updated his tree on a box with 1.5.x, then tried to work on a box with
> > > 1.4.x (I think 1.4.2 back then), and ended up with a tree that was
> > > unusable.  He had to re-clone, and I think he got lucky recovering
> > > pending changes (probably using 1.5.x on the branches with the changes,
> > > as master was what got broken).
> > 
> > I still claim that there wasn't any corruptions.
> 
> The following description is really vague because this was a while ago:
> 
> Something made my ocfs2.git tree unusable in that I could no longer do
> common tasks, such as git-log, etc without getting messages about corrupted
> refs.
> 
> I wish I had saved off some of the messages. Sorry.
> 
> I had to re-create my git tree several times before I learned by deduction
> that it was the older versions of git on some of the machines that were
> writing some sort of incompatible format.

Next time please don't hesitate to post your issue on this list.  The 
fix could have been so obvious to many people on the list, saving you 
time and frustration.  In your case I think the "fix" would have 
consisted of simply running "git repack -a -d" on the machine with the 
most recent Git version.

And if it was a case of real corruption then we certainly would have 
liked to know about it ASAP.

> > Just for fun, just edit some document with Microsoft Office 95, then 
> > open the same document with Office 2007 and save it with default 
> > settings.  Now try to open it back with Office 95.  It won't work.  
> > Does that mean that the document got corrupted?
> 
> Boy, I hope Microsoft Office isn't our bar for compatiblity here...

We can do better of course, like allowing you to still produce the old 
format with a new version of Git.  But sometimes format changes are 
unavoidable for many good reasons.


Nicolas

Re: [PATCH] provide advance warning of some future pack default changes

[Martin Langhoff]

On Dec 18, 2007 4:23 PM, Nicolas Pitre <nico@cam.org> wrote:
> Sure, but at some point that's something users mixing versions should be
> ready to cope with.  We try to make it as painless as possible of
> course.

I have to say I agree with the "apparently minor updates should not
break cross-version compat". And I think it's a communication issue
around the version numbering. The fact that this will be introduced
with a v1.5.5 is, IMHO, a good part of the problem.

If cvs 1.11 doesn't talk with 1.12 I'll say there are nuts - minor
revisions should interoperate with end users not even thinking about
it. But 1.5.5 has in its changelog lots of deprecations and interop
changes.

It's not good communication to label it 1.5.5.

Other than that, it's an _amazing_ thing, and I'm in love with git.
But the version number is a bit of a lie -- and is bound to confuse
and anger end users.

cheers,


martin

Re: [PATCH] provide advance warning of some future pack default changes

[Nicolas Pitre]

On Tue, 18 Dec 2007, Martin Langhoff wrote:

> On Dec 18, 2007 4:23 PM, Nicolas Pitre <nico@cam.org> wrote:
> > Sure, but at some point that's something users mixing versions should be
> > ready to cope with.  We try to make it as painless as possible of
> > course.
> 
> I have to say I agree with the "apparently minor updates should not
> break cross-version compat". And I think it's a communication issue
> around the version numbering. The fact that this will be introduced
> with a v1.5.5 is, IMHO, a good part of the problem.
> 
> If cvs 1.11 doesn't talk with 1.12 I'll say there are nuts - minor
> revisions should interoperate with end users not even thinking about
> it. But 1.5.5 has in its changelog lots of deprecations and interop
> changes.
> 
> It's not good communication to label it 1.5.5.

I agree.  Might be time for 1.6.0?


Nicolas

Re: [PATCH] provide advance warning of some future pack default changes

[Junio C Hamano]

"Martin Langhoff" <martin.langhoff@gmail.com> writes:

> If cvs 1.11 doesn't talk with 1.12 I'll say there are nuts - minor
> revisions should interoperate with end users not even thinking about
> it. But 1.5.5 has in its changelog lots of deprecations and interop
> changes.
>
> It's not good communication to label it 1.5.5.

There indeed are handful scheduled removals.  I do not mind declaring
that 1.6.0 comes after 1.5.4, or just relabel the removal schedule for
1.6.0 and keep the scheduled change on hold a bit longer.

By the way, I'd appreciate an Ack or comment on the recent pserver
authentication enhancements in c934dca22ee07cb3ca146a249bdb73ab0f30b2b1
(Authentication support for pserver); I do not mind merging this in
1.5.4 as the change is fairly isolated and should not affect people who
do not use the feature.

Re: [PATCH] provide advance warning of some future pack default changes

[Jakub Narebski]

Junio C Hamano wrote:
> "Martin Langhoff" <martin.langhoff@gmail.com> writes:
> 
>> If cvs 1.11 doesn't talk with 1.12 I'll say there are nuts - minor
>> revisions should interoperate with end users not even thinking about
>> it. But 1.5.5 has in its changelog lots of deprecations and interop
>> changes.
>>
>> It's not good communication to label it 1.5.5.
> 
> There indeed are handful scheduled removals.  I do not mind declaring
> that 1.6.0 comes after 1.5.4, or just relabel the removal schedule for
> 1.6.0 and keep the scheduled change on hold a bit longer.

By the way, I wonder if there would be packv4 in time for 1.6.0;
perhaps not enabled by default.

-- 
Jakub Narebski
Poland

Re: [PATCH] provide advance warning of some future pack default changes

[Jeff King]

On Mon, Dec 17, 2007 at 09:01:49PM -0800, Junio C Hamano wrote:

> There indeed are handful scheduled removals.  I do not mind declaring
> that 1.6.0 comes after 1.5.4, or just relabel the removal schedule for
> 1.6.0 and keep the scheduled change on hold a bit longer.

I can think of two other user-visible changes which have been discussed
that might warrant such a version bump:
  - option parsing tweaks (hopefully these should be minor, but it is
    clear that we cannot be 100% consistent while retaining the
    identical previous behavior)
  - moving dashed forms out of paths

-Peff

Re: [PATCH] provide advance warning of some future pack default changes

[Johannes Schindelin]

Hi,

On Tue, 18 Dec 2007, Jakub Narebski wrote:

> By the way, I wonder if there would be packv4 in time for 1.6.0; perhaps 
> not enabled by default.

Sure!  If someone undertakes the massive amount of work it takes to bring 
packv4 off!

But if that is done, I do not see why it should be off by default.

Ciao,
Dscho

Re: [PATCH] provide advance warning of some future pack default changes

[Johannes Schindelin]

Hi,

On Tue, 18 Dec 2007, Jeff King wrote:

> On Mon, Dec 17, 2007 at 09:01:49PM -0800, Junio C Hamano wrote:
> 
> > There indeed are handful scheduled removals.  I do not mind declaring 
> > that 1.6.0 comes after 1.5.4, or just relabel the removal schedule for 
> > 1.6.0 and keep the scheduled change on hold a bit longer.
> 
> I can think of two other user-visible changes which have been discussed 
> that might warrant such a version bump:
>
>   - option parsing tweaks (hopefully these should be minor, but it is
>     clear that we cannot be 100% consistent while retaining the
>     identical previous behavior)

IMHO this does not warrant a version bump.  It should be mostly 
behind-the-scenes, after all.

>   - moving dashed forms out of paths

Playing it safe, and waiting with this after announcing it more obviously, 
is something that I appreciate.  Too many scripts can break, and I am sure 
quite a few of mine will; I simply do not have the time right now to audit 
them.

Ciao,
Dscho

Re: [PATCH] provide advance warning of some future pack default changes

[Jeff King]

On Tue, Dec 18, 2007 at 12:06:23PM +0000, Johannes Schindelin wrote:

> >   - option parsing tweaks (hopefully these should be minor, but it is
> >     clear that we cannot be 100% consistent while retaining the
> >     identical previous behavior)
> 
> IMHO this does not warrant a version bump.  It should be mostly 
> behind-the-scenes, after all.

Yes, it should be, but I think there will be a few user-visible fallouts
(like "--abbrev $foo" in scripts should now be "--abbrev-default $foo"
for safety).

-Peff

Re: [PATCH] provide advance warning of some future pack default changes

[Johannes Schindelin]

Hi,

On Tue, 18 Dec 2007, Jeff King wrote:

> On Tue, Dec 18, 2007 at 12:06:23PM +0000, Johannes Schindelin wrote:
> 
> > >   - option parsing tweaks (hopefully these should be minor, but it is
> > >     clear that we cannot be 100% consistent while retaining the
> > >     identical previous behavior)
> > 
> > IMHO this does not warrant a version bump.  It should be mostly 
> > behind-the-scenes, after all.
> 
> Yes, it should be, but I think there will be a few user-visible fallouts
> (like "--abbrev $foo" in scripts should now be "--abbrev-default $foo"
> for safety).

But we are on our way to fix this, no?  IOW this warrants not a version 
bump, but an extended feature freeze/bug fix period (like Junio suggested, 
until January).

Ciao,
Dscho

Re: [PATCH] provide advance warning of some future pack default changes

[Jakub Narebski]

Johannes Schindelin wrote:
> On Tue, 18 Dec 2007, Jeff King wrote:
> 
>>   - moving dashed forms out of paths
> 
> Playing it safe, and waiting with this after announcing it more obviously, 
> is something that I appreciate.  Too many scripts can break, and I am sure 
> quite a few of mine will; I simply do not have the time right now to audit 
> them.

We could do it IMVHO in two (or two an a half :-)) steps:

1. Decide where separate exec-path area should be, following FHS. Create
   it during install. Install helper scripts there, moving it out of PATH.
   Test those tools which use helper scripts (helper commands), which
   should be _much_ easier than testing whole git for "moving dashed forms
   out of path" breakage.

2. Move dashed forms out of PATH, perhaps leaving (or with option of
   leaving) dashed forms of porcelain in PATH. Test all scripts and tests
   ;-)
   
I think that the first step can be done before 1.6.0, perhaps even
before 1.5.4
-- 
Jakub Narebski
Poland

Re: [PATCH] provide advance warning of some future pack default changes

[Nicolas Pitre]

On Tue, 18 Dec 2007, Jakub Narebski wrote:

> Junio C Hamano wrote:
> > "Martin Langhoff" <martin.langhoff@gmail.com> writes:
> > 
> >> If cvs 1.11 doesn't talk with 1.12 I'll say there are nuts - minor
> >> revisions should interoperate with end users not even thinking about
> >> it. But 1.5.5 has in its changelog lots of deprecations and interop
> >> changes.
> >>
> >> It's not good communication to label it 1.5.5.
> > 
> > There indeed are handful scheduled removals.  I do not mind declaring
> > that 1.6.0 comes after 1.5.4, or just relabel the removal schedule for
> > 1.6.0 and keep the scheduled change on hold a bit longer.

I think Git development is dynamic enough to justify 1.6.0 right after 
1.5.4.

> By the way, I wonder if there would be packv4 in time for 1.6.0;
> perhaps not enabled by default.

I don't think so.  First, if packv4 actually happens, it might justify 
v2.0.0 and not v1.6.0.

But so far there were steady improvement made to the system even with 
the current pack format, so the return on the investment for packv4 is 
diminishing.  The largest road block for packv4 at the moment is a 
complete refactoring of the tree walking code.


Nicolas

Re: [PATCH] provide advance warning of some future pack default changes

[Jeff King]

On Tue, Dec 18, 2007 at 01:30:49PM +0000, Johannes Schindelin wrote:

> > Yes, it should be, but I think there will be a few user-visible fallouts
> > (like "--abbrev $foo" in scripts should now be "--abbrev-default $foo"
> > for safety).
> 
> But we are on our way to fix this, no?  IOW this warrants not a version 
> bump, but an extended feature freeze/bug fix period (like Junio suggested, 
> until January).

I think the resolution seems to be that we will now support "--abbrev
foo", though we didn't in the past. Because the "foo" here is optional,
the old "git log --abbrev HEAD" is ambiguous. In this case we'll see
that "HEAD" isn't a number and DWIM. But that means a script trying to
be unambiguous should use "git log --abbrev-default $foo" to make sure
that "$foo" doesn't accidentally match as a number.

So there will be user-visible changes (though I don't expect them to be
huge...there simply aren't that many variables with optional arguments).

-Peff

Re: [PATCH] provide advance warning of some future pack default changes

[Nicolas Pitre]

On Tue, 18 Dec 2007, Jeff King wrote:

> So there will be user-visible changes (though I don't expect them to be
> huge...there simply aren't that many variables with optional arguments).

OTOH, there are quite a bunch of changes affecting the user experience.  
Many of the feedback messages printed by Git were completely revamped, 
starting with the progress display to the fetch summary.


Nicolas

Re: [PATCH] provide advance warning of some future pack default changes

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> I can think of two other user-visible changes which have been discussed
> that might warrant such a version bump:
>
>   - option parsing tweaks (hopefully these should be minor, but it is
>     clear that we cannot be 100% consistent while retaining the
>     identical previous behavior)

This could have a fallout, like *-default disambiguation which scripts
did not have to implement.

>   - moving dashed forms out of paths

This is already planned for 1.5.5 and it is not among "other
user-visible changes".  Technically the use of git-foo form without
preparing the environment has not been supported for quite some time,
but people have come to rely on it and I'd agree this warrants a 1.6.0.