[JGIT PATCH 00/10] Support writing pack index version 2

[JGIT PATCH 00/10] Support writing pack index version 2

[Shawn O. Pearce]

This series adds support for writing pack index v2 formatted files
out of IndexPack, and thus any pack we fetch over the network.  It
also abstracts out the index formatting functions so we can reuse
them efficiently inside of PackWriter to support creating packs on
the local disk, or to upload packs directly over a dumb transport.

I started down this path because we're missing index v2 support and
that is likely to become a default in the near future for C Git,
and because I want to support push over sftp style URLs now that
Marek has a pack writing implementation.

----

The following changes since commit 535041bba0836a3488fbd465adb171a2c70c9415:
  Florian Koeberle (1):
        Added the package fnmatch and two exceptions.

are available in the git repository at:

  repo.or.cz:/srv/git/egit/spearce.git index-v2

Shawn O. Pearce (10):
      Extract inner ObjectEntry from IndexPack class
      Make ObjectEntry's position field private
      Rename ObjectEntry to PackedObjectInfo
      Document PackedObjectInfo and make it public for reuse
      Refactor pack index writing to a common API
      Reuse the magic tOc constant for pack index headers
      Add 64 bit network byte order encoding to NB
      Compute packed object entry CRC32 data during IndexPack
      Add support for writing pack index v2 files
      Default IndexPack to honor pack.indexversion configuration

 .../src/org/spearce/jgit/lib/CoreConfig.java       |   11 +
 .../src/org/spearce/jgit/lib/PackIndex.java        |    3 +-
 .../src/org/spearce/jgit/lib/PackIndexWriter.java  |  267 ++++++++++++++++++++
 .../org/spearce/jgit/lib/PackIndexWriterV1.java    |   78 ++++++
 .../org/spearce/jgit/lib/PackIndexWriterV2.java    |  101 ++++++++
 .../src/org/spearce/jgit/pgm/IndexPack.java        |    4 +
 .../src/org/spearce/jgit/transport/IndexPack.java  |  225 ++++++++++-------
 .../spearce/jgit/transport/PackedObjectInfo.java   |  109 ++++++++
 org.spearce.jgit/src/org/spearce/jgit/util/NB.java |   37 +++
 9 files changed, 743 insertions(+), 92 deletions(-)
 create mode 100644 org.spearce.jgit/src/org/spearce/jgit/lib/PackIndexWriter.java
 create mode 100644 org.spearce.jgit/src/org/spearce/jgit/lib/PackIndexWriterV1.java
 create mode 100644 org.spearce.jgit/src/org/spearce/jgit/lib/PackIndexWriterV2.java
 create mode 100644 org.spearce.jgit/src/org/spearce/jgit/transport/PackedObjectInfo.java

[JGIT PATCH 01/10] Extract inner ObjectEntry from IndexPack class

[Shawn O. Pearce]

This type is useful not just for IndexPack but also for PackWriter
as it is essentially the index record (ObjectId and offset).  Both
data items are necessary to compute a valid index record, so we can
unify the index writing code if we have a common type.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
---
 .../src/org/spearce/jgit/transport/IndexPack.java  |   10 ----
 .../org/spearce/jgit/transport/ObjectEntry.java    |   51 ++++++++++++++++++++
 2 files changed, 51 insertions(+), 10 deletions(-)
 create mode 100644 org.spearce.jgit/src/org/spearce/jgit/transport/ObjectEntry.java

diff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/IndexPack.java b/org.spearce.jgit/src/org/spearce/jgit/transport/IndexPack.java
index bec211c..e182cfc 100644
--- a/org.spearce.jgit/src/org/spearce/jgit/transport/IndexPack.java
+++ b/org.spearce.jgit/src/org/spearce/jgit/transport/IndexPack.java
@@ -54,7 +54,6 @@ import java.util.zip.Deflater;
 import java.util.zip.Inflater;
 
 import org.spearce.jgit.errors.CorruptObjectException;
-import org.spearce.jgit.lib.AnyObjectId;
 import org.spearce.jgit.lib.BinaryDelta;
 import org.spearce.jgit.lib.Constants;
 import org.spearce.jgit.lib.InflaterCache;
@@ -715,15 +714,6 @@ public class IndexPack {
 				+ dfe.getMessage());
 	}
 
-	private static class ObjectEntry extends ObjectId {
-		final long pos;
-
-		ObjectEntry(final long headerOffset, final AnyObjectId id) {
-			super(id);
-			pos = headerOffset;
-		}
-	}
-
 	private static class UnresolvedDelta {
 		final long position;
 
diff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/ObjectEntry.java b/org.spearce.jgit/src/org/spearce/jgit/transport/ObjectEntry.java
new file mode 100644
index 0000000..58d2eb2
--- /dev/null
+++ b/org.spearce.jgit/src/org/spearce/jgit/transport/ObjectEntry.java
@@ -0,0 +1,51 @@
+/*
+ * Copyright (C) 2008, Robin Rosenberg <robin.rosenberg@dewire.com>
+ * Copyright (C) 2008, Shawn O. Pearce <spearce@spearce.org>
+ *
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * - Redistributions of source code must retain the above copyright
+ *   notice, this list of conditions and the following disclaimer.
+ *
+ * - Redistributions in binary form must reproduce the above
+ *   copyright notice, this list of conditions and the following
+ *   disclaimer in the documentation and/or other materials provided
+ *   with the distribution.
+ *
+ * - Neither the name of the Git Development Community nor the
+ *   names of its contributors may be used to endorse or promote
+ *   products derived from this software without specific prior
+ *   written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND
+ * CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES,
+ * INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+ * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
+ * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+ * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+ * CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+ * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
+ * ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+package org.spearce.jgit.transport;
+
+import org.spearce.jgit.lib.AnyObjectId;
+import org.spearce.jgit.lib.ObjectId;
+
+class ObjectEntry extends ObjectId {
+	final long pos;
+
+	ObjectEntry(final long headerOffset, final AnyObjectId id) {
+		super(id);
+		pos = headerOffset;
+	}
+}
-- 
1.5.6.74.g8a5e

[JGIT PATCH 02/10] Make ObjectEntry's position field private

[Shawn O. Pearce]

PackWriter calls this getOffset() and hides the field as a private
field as part of its related ObjectToPack class.  We do that here
as part of ObjectEntry's API so we can later replace that part of
ObjectToPack by inheriting from ObjectEntry.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
---
 .../src/org/spearce/jgit/transport/IndexPack.java  |   11 ++++++-----
 .../org/spearce/jgit/transport/ObjectEntry.java    |   12 ++++++++++--
 2 files changed, 16 insertions(+), 7 deletions(-)

diff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/IndexPack.java b/org.spearce.jgit/src/org/spearce/jgit/transport/IndexPack.java
index e182cfc..19a4b7b 100644
--- a/org.spearce.jgit/src/org/spearce/jgit/transport/IndexPack.java
+++ b/org.spearce.jgit/src/org/spearce/jgit/transport/IndexPack.java
@@ -281,8 +281,9 @@ public class IndexPack {
 	}
 
 	private void resolveDeltas(final ObjectEntry oe) throws IOException {
-		if (baseById.containsKey(oe) || baseByPos.containsKey(new Long(oe.pos)))
-			resolveDeltas(oe.pos, Constants.OBJ_BAD, null, oe);
+		if (baseById.containsKey(oe)
+				|| baseByPos.containsKey(new Long(oe.getOffset())))
+			resolveDeltas(oe.getOffset(), Constants.OBJ_BAD, null, oe);
 	}
 
 	private void resolveDeltas(final long pos, int type, byte[] data,
@@ -381,7 +382,7 @@ public class IndexPack {
 			writeWhole(def, typeCode, data);
 			end = packOut.getFilePointer();
 
-			resolveChildDeltas(oe.pos, typeCode, data, oe);
+			resolveChildDeltas(oe.getOffset(), typeCode, data, oe);
 			if (progress.isCancelled())
 				throw new IOException("Download cancelled during indexing");
 		}
@@ -458,9 +459,9 @@ public class IndexPack {
 			}
 			for (int i = 0; i < entryCount; i++) {
 				final ObjectEntry oe = entries[i];
-				if (oe.pos >>> 1 > Integer.MAX_VALUE)
+				if (oe.getOffset() >>> 1 > Integer.MAX_VALUE)
 					throw new IOException("Pack too large for index version 1");
-				NB.encodeInt32(rawoe, 0, (int) oe.pos);
+				NB.encodeInt32(rawoe, 0, (int) oe.getOffset());
 				oe.copyRawTo(rawoe, 4);
 				os.write(rawoe);
 				d.update(rawoe);
diff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/ObjectEntry.java b/org.spearce.jgit/src/org/spearce/jgit/transport/ObjectEntry.java
index 58d2eb2..10a8f4d 100644
--- a/org.spearce.jgit/src/org/spearce/jgit/transport/ObjectEntry.java
+++ b/org.spearce.jgit/src/org/spearce/jgit/transport/ObjectEntry.java
@@ -42,10 +42,18 @@ import org.spearce.jgit.lib.AnyObjectId;
 import org.spearce.jgit.lib.ObjectId;
 
 class ObjectEntry extends ObjectId {
-	final long pos;
+	private long offset;
 
 	ObjectEntry(final long headerOffset, final AnyObjectId id) {
 		super(id);
-		pos = headerOffset;
+		offset = headerOffset;
+	}
+
+	/**
+	 * @return offset in pack when object has been already written, or -1 if it
+	 *         has not been written yet
+	 */
+	long getOffset() {
+		return offset;
 	}
 }
-- 
1.5.6.74.g8a5e

[JGIT PATCH 03/10] Rename ObjectEntry to PackedObjectInfo

[Shawn O. Pearce]

Technically this class contains information about a packed object,
data which is necessary to access it or to create the index record
to support random access to the object's information.  Calling it
just ObjectEntry is no longer a very sufficient name.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
---
 .../src/org/spearce/jgit/transport/IndexPack.java  |   26 ++++----
 .../org/spearce/jgit/transport/ObjectEntry.java    |   59 --------------------
 .../spearce/jgit/transport/PackedObjectInfo.java   |   59 ++++++++++++++++++++
 3 files changed, 72 insertions(+), 72 deletions(-)
 delete mode 100644 org.spearce.jgit/src/org/spearce/jgit/transport/ObjectEntry.java
 create mode 100644 org.spearce.jgit/src/org/spearce/jgit/transport/PackedObjectInfo.java

diff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/IndexPack.java b/org.spearce.jgit/src/org/spearce/jgit/transport/IndexPack.java
index 19a4b7b..ad4bcae 100644
--- a/org.spearce.jgit/src/org/spearce/jgit/transport/IndexPack.java
+++ b/org.spearce.jgit/src/org/spearce/jgit/transport/IndexPack.java
@@ -129,7 +129,7 @@ public class IndexPack {
 
 	private long objectCount;
 
-	private ObjectEntry[] entries;
+	private PackedObjectInfo[] entries;
 
 	private int deltaCount;
 
@@ -208,7 +208,7 @@ public class IndexPack {
 			try {
 				readPackHeader();
 
-				entries = new ObjectEntry[(int) objectCount];
+				entries = new PackedObjectInfo[(int) objectCount];
 				baseById = new ObjectIdMap<ArrayList<UnresolvedDelta>>();
 				baseByPos = new HashMap<Long, ArrayList<UnresolvedDelta>>();
 
@@ -280,14 +280,14 @@ public class IndexPack {
 		progress.endTask();
 	}
 
-	private void resolveDeltas(final ObjectEntry oe) throws IOException {
+	private void resolveDeltas(final PackedObjectInfo oe) throws IOException {
 		if (baseById.containsKey(oe)
 				|| baseByPos.containsKey(new Long(oe.getOffset())))
 			resolveDeltas(oe.getOffset(), Constants.OBJ_BAD, null, oe);
 	}
 
 	private void resolveDeltas(final long pos, int type, byte[] data,
-			ObjectEntry oe) throws IOException {
+			PackedObjectInfo oe) throws IOException {
 		position(pos);
 		int c = readFromFile();
 		final int typeCode = (c >> 4) & 7;
@@ -331,7 +331,7 @@ public class IndexPack {
 			objectDigest.update((byte) 0);
 			objectDigest.update(data);
 			tempObjectId.fromRaw(objectDigest.digest(), 0);
-			oe = new ObjectEntry(pos, tempObjectId);
+			oe = new PackedObjectInfo(pos, tempObjectId);
 			entries[entryCount++] = oe;
 		}
 
@@ -339,7 +339,7 @@ public class IndexPack {
 	}
 
 	private void resolveChildDeltas(final long pos, int type, byte[] data,
-			ObjectEntry oe) throws IOException {
+			PackedObjectInfo oe) throws IOException {
 		final ArrayList<UnresolvedDelta> a = baseById.remove(oe);
 		final ArrayList<UnresolvedDelta> b = baseByPos.remove(new Long(pos));
 		int ai = 0, bi = 0;
@@ -374,9 +374,9 @@ public class IndexPack {
 			final ObjectLoader ldr = repo.openObject(baseId);
 			final byte[] data = ldr.getBytes();
 			final int typeCode = ldr.getType();
-			final ObjectEntry oe;
+			final PackedObjectInfo oe;
 
-			oe = new ObjectEntry(end, baseId);
+			oe = new PackedObjectInfo(end, baseId);
 			entries[entryCount++] = oe;
 			packOut.seek(end);
 			writeWhole(def, typeCode, data);
@@ -432,9 +432,9 @@ public class IndexPack {
 	}
 
 	private void growEntries() {
-		final ObjectEntry[] ne;
+		final PackedObjectInfo[] ne;
 
-		ne = new ObjectEntry[(int) objectCount + baseById.size()];
+		ne = new PackedObjectInfo[(int) objectCount + baseById.size()];
 		System.arraycopy(entries, 0, ne, 0, entryCount);
 		entries = ne;
 	}
@@ -458,7 +458,7 @@ public class IndexPack {
 				d.update(rawoe, 0, 4);
 			}
 			for (int i = 0; i < entryCount; i++) {
-				final ObjectEntry oe = entries[i];
+				final PackedObjectInfo oe = entries[i];
 				if (oe.getOffset() >>> 1 > Integer.MAX_VALUE)
 					throw new IOException("Pack too large for index version 1");
 				NB.encodeInt32(rawoe, 0, (int) oe.getOffset());
@@ -578,7 +578,7 @@ public class IndexPack {
 		objectDigest.update((byte) 0);
 		inflateFromInput(true);
 		tempObjectId.fromRaw(objectDigest.digest(), 0);
-		entries[entryCount++] = new ObjectEntry(pos, tempObjectId);
+		entries[entryCount++] = new PackedObjectInfo(pos, tempObjectId);
 	}
 
 	// Current position of {@link #bOffset} within the entire file.
@@ -739,7 +739,7 @@ public class IndexPack {
 		final MessageDigest d = Constants.newMessageDigest();
 		final byte[] oeBytes = new byte[Constants.OBJECT_ID_LENGTH];
 		for (int i = 0; i < entryCount; i++) {
-			final ObjectEntry oe = entries[i];
+			final PackedObjectInfo oe = entries[i];
 			oe.copyRawTo(oeBytes, 0);
 			d.update(oeBytes);
 		}
diff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/ObjectEntry.java b/org.spearce.jgit/src/org/spearce/jgit/transport/ObjectEntry.java
deleted file mode 100644
index 10a8f4d..0000000
--- a/org.spearce.jgit/src/org/spearce/jgit/transport/ObjectEntry.java
+++ /dev/null
@@ -1,59 +0,0 @@
-/*
- * Copyright (C) 2008, Robin Rosenberg <robin.rosenberg@dewire.com>
- * Copyright (C) 2008, Shawn O. Pearce <spearce@spearce.org>
- *
- * All rights reserved.
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above copyright
- *   notice, this list of conditions and the following disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- *   copyright notice, this list of conditions and the following
- *   disclaimer in the documentation and/or other materials provided
- *   with the distribution.
- *
- * - Neither the name of the Git Development Community nor the
- *   names of its contributors may be used to endorse or promote
- *   products derived from this software without specific prior
- *   written permission.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND
- * CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES,
- * INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
- * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
- * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
- * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
- * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
- * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
- * CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
- * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
- * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
- * ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
- */
-
-package org.spearce.jgit.transport;
-
-import org.spearce.jgit.lib.AnyObjectId;
-import org.spearce.jgit.lib.ObjectId;
-
-class ObjectEntry extends ObjectId {
-	private long offset;
-
-	ObjectEntry(final long headerOffset, final AnyObjectId id) {
-		super(id);
-		offset = headerOffset;
-	}
-
-	/**
-	 * @return offset in pack when object has been already written, or -1 if it
-	 *         has not been written yet
-	 */
-	long getOffset() {
-		return offset;
-	}
-}
diff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/PackedObjectInfo.java b/org.spearce.jgit/src/org/spearce/jgit/transport/PackedObjectInfo.java
new file mode 100644
index 0000000..eaedee9
--- /dev/null
+++ b/org.spearce.jgit/src/org/spearce/jgit/transport/PackedObjectInfo.java
@@ -0,0 +1,59 @@
+/*
+ * Copyright (C) 2008, Robin Rosenberg <robin.rosenberg@dewire.com>
+ * Copyright (C) 2008, Shawn O. Pearce <spearce@spearce.org>
+ *
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * - Redistributions of source code must retain the above copyright
+ *   notice, this list of conditions and the following disclaimer.
+ *
+ * - Redistributions in binary form must reproduce the above
+ *   copyright notice, this list of conditions and the following
+ *   disclaimer in the documentation and/or other materials provided
+ *   with the distribution.
+ *
+ * - Neither the name of the Git Development Community nor the
+ *   names of its contributors may be used to endorse or promote
+ *   products derived from this software without specific prior
+ *   written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND
+ * CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES,
+ * INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+ * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
+ * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+ * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+ * CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+ * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
+ * ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+package org.spearce.jgit.transport;
+
+import org.spearce.jgit.lib.AnyObjectId;
+import org.spearce.jgit.lib.ObjectId;
+
+class PackedObjectInfo extends ObjectId {
+	private long offset;
+
+	PackedObjectInfo(final long headerOffset, final AnyObjectId id) {
+		super(id);
+		offset = headerOffset;
+	}
+
+	/**
+	 * @return offset in pack when object has been already written, or -1 if it
+	 *         has not been written yet
+	 */
+	long getOffset() {
+		return offset;
+	}
+}
-- 
1.5.6.74.g8a5e

[JGIT PATCH 04/10] Document PackedObjectInfo and make it public for reuse

[Shawn O. Pearce]

Classes outside of transport may wish to use this abstraction,
so we mark it public, expose the offset field, and document
as much of the API as possible.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
---
 .../spearce/jgit/transport/PackedObjectInfo.java   |   33 ++++++++++++++++++--
 1 files changed, 30 insertions(+), 3 deletions(-)

diff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/PackedObjectInfo.java b/org.spearce.jgit/src/org/spearce/jgit/transport/PackedObjectInfo.java
index eaedee9..58feada 100644
--- a/org.spearce.jgit/src/org/spearce/jgit/transport/PackedObjectInfo.java
+++ b/org.spearce.jgit/src/org/spearce/jgit/transport/PackedObjectInfo.java
@@ -41,7 +41,14 @@ package org.spearce.jgit.transport;
 import org.spearce.jgit.lib.AnyObjectId;
 import org.spearce.jgit.lib.ObjectId;
 
-class PackedObjectInfo extends ObjectId {
+/**
+ * Description of an object stored in a pack file, including offset.
+ * <p>
+ * When objects are stored in packs Git needs the ObjectId and the offset
+ * (starting position of the object data) to perform random-access reads of
+ * objects from the pack. This extension of ObjectId includes the offset.
+ */
+public class PackedObjectInfo extends ObjectId {
 	private long offset;
 
 	PackedObjectInfo(final long headerOffset, final AnyObjectId id) {
@@ -50,10 +57,30 @@ class PackedObjectInfo extends ObjectId {
 	}
 
 	/**
-	 * @return offset in pack when object has been already written, or -1 if it
+	 * Create a new structure to remember information about an object.
+	 * 
+	 * @param id
+	 *            the identity of the object the new instance tracks.
+	 */
+	public PackedObjectInfo(final AnyObjectId id) {
+		super(id);
+	}
+
+	/**
+	 * @return offset in pack when object has been already written, or 0 if it
 	 *         has not been written yet
 	 */
-	long getOffset() {
+	public long getOffset() {
 		return offset;
 	}
+
+	/**
+	 * Set the offset in pack when object has been written to.
+	 * 
+	 * @param offset
+	 *            offset where written object starts
+	 */
+	public void setOffset(final long offset) {
+		this.offset = offset;
+	}
 }
-- 
1.5.6.74.g8a5e

[JGIT PATCH 05/10] Refactor pack index writing to a common API

[Shawn O. Pearce]

To repack a repository or to push a pack file over a dumb transport
we need to create a .idx file to match the .pack.  This means
the index writing features of transport.IndexPack are needed by
PackWriter, and/or dumb protocol transports.

This refactoring creates a PackIndexWriter class structure that
mirrors the PackIndex (reader) structure we already have in place
for PackIndexV1 and PackIndexV2.

At present only PackIndexV1 is supported as we do not have access
to the CRC data necessary to write PackIndexV2 style files.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
---
 .../src/org/spearce/jgit/lib/PackIndexWriter.java  |  243 ++++++++++++++++++++
 .../org/spearce/jgit/lib/PackIndexWriterV1.java    |   78 +++++++
 .../src/org/spearce/jgit/transport/IndexPack.java  |   37 +---
 3 files changed, 330 insertions(+), 28 deletions(-)
 create mode 100644 org.spearce.jgit/src/org/spearce/jgit/lib/PackIndexWriter.java
 create mode 100644 org.spearce.jgit/src/org/spearce/jgit/lib/PackIndexWriterV1.java

diff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/PackIndexWriter.java b/org.spearce.jgit/src/org/spearce/jgit/lib/PackIndexWriter.java
new file mode 100644
index 0000000..473e6cf
--- /dev/null
+++ b/org.spearce.jgit/src/org/spearce/jgit/lib/PackIndexWriter.java
@@ -0,0 +1,243 @@
+/*
+ * Copyright (C) 2008, Robin Rosenberg <robin.rosenberg@dewire.com>
+ * Copyright (C) 2008, Shawn O. Pearce <spearce@spearce.org>
+ *
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * - Redistributions of source code must retain the above copyright
+ *   notice, this list of conditions and the following disclaimer.
+ *
+ * - Redistributions in binary form must reproduce the above
+ *   copyright notice, this list of conditions and the following
+ *   disclaimer in the documentation and/or other materials provided
+ *   with the distribution.
+ *
+ * - Neither the name of the Git Development Community nor the
+ *   names of its contributors may be used to endorse or promote
+ *   products derived from this software without specific prior
+ *   written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND
+ * CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES,
+ * INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+ * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
+ * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+ * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+ * CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+ * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
+ * ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+package org.spearce.jgit.lib;
+
+import java.io.BufferedOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.security.DigestOutputStream;
+import java.util.List;
+
+import org.spearce.jgit.transport.PackedObjectInfo;
+import org.spearce.jgit.util.NB;
+
+/**
+ * Creates a table of contents to support random access by {@link PackFile}.
+ * <p>
+ * Pack index files (the <code>.idx</code> suffix in a pack file pair)
+ * provides random access to any object in the pack by associating an ObjectId
+ * to the byte offset within the pack where the object's data can be read.
+ */
+public abstract class PackIndexWriter {
+	/**
+	 * Create a new writer for the oldest (most widely understood) format.
+	 * <p>
+	 * This method selects an index format that can accurate describe the
+	 * supplied objects and that will be the most compatible format with older
+	 * Git implementations.
+	 * <p>
+	 * Index version 1 is widely recognized by all Git implementations, but
+	 * index version 2 (and later) is not as well recognized as it was
+	 * introduced more than a year later. Index version 1 can only be used if
+	 * the resulting pack file is under 4 gigabytes in size; packs larger than
+	 * that limit must use index version 2.
+	 * 
+	 * @param dst
+	 *            the stream the index data will be written to. If not already
+	 *            buffered it will be automatically wrapped in a buffered
+	 *            stream. Callers are always responsible for closing the stream.
+	 * @param objs
+	 *            the objects the caller needs to store in the index. Entries
+	 *            will be examined until a format can be conclusively selected.
+	 * @return a new writer to output an index file of the requested format to
+	 *         the supplied stream.
+	 * @throws IllegalArgumentException
+	 *             no recognized pack index version can support the supplied
+	 *             objects. This is likely a bug in the implementation.
+	 */
+	@SuppressWarnings("fallthrough")
+	public static PackIndexWriter createOldestPossible(final OutputStream dst,
+			final List<PackedObjectInfo> objs) {
+		int version = 1;
+		LOOP: for (final PackedObjectInfo oe : objs) {
+			switch (version) {
+			case 1:
+				if (PackIndexWriterV1.canStore(oe))
+					continue;
+				version = 2;
+			case 2:
+				break LOOP;
+			}
+		}
+		return createVersion(dst, version);
+	}
+
+	/**
+	 * Create a new writer instance for a specific index format version.
+	 * 
+	 * @param dst
+	 *            the stream the index data will be written to. If not already
+	 *            buffered it will be automatically wrapped in a buffered
+	 *            stream. Callers are always responsible for closing the stream.
+	 * @param version
+	 *            index format version number required by the caller. Exactly
+	 *            this formatted version will be written.
+	 * @return a new writer to output an index file of the requested format to
+	 *         the supplied stream.
+	 * @throws IllegalArgumentException
+	 *             the version requested is not supported by this
+	 *             implementation.
+	 */
+	public static PackIndexWriter createVersion(final OutputStream dst,
+			final int version) {
+		switch (version) {
+		case 1:
+			return new PackIndexWriterV1(dst);
+		default:
+			throw new IllegalArgumentException(
+					"Unsupported pack index version " + version);
+		}
+	}
+
+	/** The index data stream we are responsible for creating. */
+	protected final DigestOutputStream out;
+
+	/** A temporary buffer for use during IO to {link #out}. */
+	protected final byte[] tmp;
+
+	/** The entries this writer must pack. */
+	protected List<PackedObjectInfo> entries;
+
+	/** SHA-1 checksum for the entire pack data. */
+	protected byte[] packChecksum;
+
+	/**
+	 * Create a new writer instance.
+	 * 
+	 * @param dst
+	 *            the stream this instance outputs to. If not already buffered
+	 *            it will be automatically wrapped in a buffered stream.
+	 */
+	protected PackIndexWriter(final OutputStream dst) {
+		out = new DigestOutputStream(dst instanceof BufferedOutputStream ? dst
+				: new BufferedOutputStream(dst), Constants.newMessageDigest());
+		tmp = new byte[4 + Constants.OBJECT_ID_LENGTH];
+	}
+
+	/**
+	 * Write all object entries to the index stream.
+	 * <p>
+	 * After writing the stream passed to the factory is flushed but remains
+	 * open. Callers are always responsible for closing the output stream.
+	 * 
+	 * @param toStore
+	 *            sorted list of objects to store in the index. The caller must
+	 *            have previously sorted the list using {@link PackedObjectInfo}'s
+	 *            native {@link Comparable} implementation.
+	 * @param packDataChecksum
+	 *            checksum signature of the entire pack data content. This is
+	 *            traditionally the last 20 bytes of the pack file's own stream.
+	 * @throws IOException
+	 *             an error occurred while writing to the output stream, or this
+	 *             index format cannot store the object data supplied.
+	 */
+	public void write(final List<PackedObjectInfo> toStore,
+			final byte[] packDataChecksum) throws IOException {
+		entries = toStore;
+		packChecksum = packDataChecksum;
+		writeImpl();
+		out.flush();
+	}
+
+	/**
+	 * Writes the index file to {@link #out}.
+	 * <p>
+	 * Implementations should go something like:
+	 * 
+	 * <pre>
+	 * writeFanOutTable();
+	 * for (final PackedObjectInfo po : entries)
+	 * 	writeOneEntry(po);
+	 * writeChecksumFooter();
+	 * </pre>
+	 * 
+	 * <p>
+	 * Where the logic for <code>writeOneEntry</code> is specific to the index
+	 * format in use. Additional headers/footers may be used if necessary and
+	 * the {@link #entries} collection may be iterated over more than once if
+	 * necessary. Implementors therefore have complete control over the data.
+	 * 
+	 * @throws IOException
+	 *             an error occurred while writing to the output stream, or this
+	 *             index format cannot store the object data supplied.
+	 */
+	protected abstract void writeImpl() throws IOException;
+
+	/**
+	 * Output the standard 256 entry first-level fan-out table.
+	 * <p>
+	 * The fan-out table is 4 KB in size, holding 256 32-bit unsigned integer
+	 * counts. Each count represents the number of objects within this index
+	 * whose {@link ObjectId#getFirstByte()} matches the count's position in the
+	 * fan-out table.
+	 * 
+	 * @throws IOException
+	 *             an error occurred while writing to the output stream.
+	 */
+	protected void writeFanOutTable() throws IOException {
+		final int[] fanout = new int[256];
+		for (final PackedObjectInfo po : entries)
+			fanout[po.getFirstByte() & 0xff]++;
+		for (int i = 1; i < 256; i++)
+			fanout[i] += fanout[i - 1];
+		for (final int n : fanout) {
+			NB.encodeInt32(tmp, 0, n);
+			out.write(tmp, 0, 4);
+		}
+	}
+
+	/**
+	 * Output the standard two-checksum index footer.
+	 * <p>
+	 * The standard footer contains two checksums (20 byte SHA-1 values):
+	 * <ol>
+	 * <li>Pack data checksum - taken from the last 20 bytes of the pack file.</li>
+	 * <li>Index data checksum - checksum of all index bytes written, including
+	 * the pack data checksum above.</li>
+	 * </ol>
+	 * 
+	 * @throws IOException
+	 *             an error occurred while writing to the output stream.
+	 */
+	protected void writeChecksumFooter() throws IOException {
+		out.write(packChecksum);
+		out.on(false);
+		out.write(out.getMessageDigest().digest());
+	}
+}
diff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/PackIndexWriterV1.java b/org.spearce.jgit/src/org/spearce/jgit/lib/PackIndexWriterV1.java
new file mode 100644
index 0000000..b790a28
--- /dev/null
+++ b/org.spearce.jgit/src/org/spearce/jgit/lib/PackIndexWriterV1.java
@@ -0,0 +1,78 @@
+/*
+ * Copyright (C) 2008, Robin Rosenberg <robin.rosenberg@dewire.com>
+ * Copyright (C) 2008, Shawn O. Pearce <spearce@spearce.org>
+ *
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * - Redistributions of source code must retain the above copyright
+ *   notice, this list of conditions and the following disclaimer.
+ *
+ * - Redistributions in binary form must reproduce the above
+ *   copyright notice, this list of conditions and the following
+ *   disclaimer in the documentation and/or other materials provided
+ *   with the distribution.
+ *
+ * - Neither the name of the Git Development Community nor the
+ *   names of its contributors may be used to endorse or promote
+ *   products derived from this software without specific prior
+ *   written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND
+ * CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES,
+ * INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+ * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
+ * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+ * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+ * CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+ * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
+ * ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+package org.spearce.jgit.lib;
+
+import java.io.IOException;
+import java.io.OutputStream;
+
+import org.spearce.jgit.transport.PackedObjectInfo;
+import org.spearce.jgit.util.NB;
+
+/**
+ * Creates the version 1 (old style) pack table of contents files.
+ * 
+ * @see PackIndexWriter
+ * @see PackIndexV1
+ */
+class PackIndexWriterV1 extends PackIndexWriter {
+	static boolean canStore(final PackedObjectInfo oe) {
+		// We are limited to 4 GB per pack as offset is 32 bit unsigned int.
+		//
+		return oe.getOffset() >>> 1 < Integer.MAX_VALUE;
+	}
+
+	PackIndexWriterV1(final OutputStream dst) {
+		super(dst);
+	}
+
+	@Override
+	protected void writeImpl() throws IOException {
+		writeFanOutTable();
+
+		for (final PackedObjectInfo oe : entries) {
+			if (!canStore(oe))
+				throw new IOException("Pack too large for index version 1");
+			NB.encodeInt32(tmp, 0, (int) oe.getOffset());
+			oe.copyRawTo(tmp, 4);
+			out.write(tmp);
+		}
+
+		writeChecksumFooter();
+	}
+}
diff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/IndexPack.java b/org.spearce.jgit/src/org/spearce/jgit/transport/IndexPack.java
index ad4bcae..60e0bce 100644
--- a/org.spearce.jgit/src/org/spearce/jgit/transport/IndexPack.java
+++ b/org.spearce.jgit/src/org/spearce/jgit/transport/IndexPack.java
@@ -38,7 +38,6 @@
 
 package org.spearce.jgit.transport;
 
-import java.io.BufferedOutputStream;
 import java.io.EOFException;
 import java.io.File;
 import java.io.FileOutputStream;
@@ -49,6 +48,7 @@ import java.security.MessageDigest;
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.HashMap;
+import java.util.List;
 import java.util.zip.DataFormatException;
 import java.util.zip.Deflater;
 import java.util.zip.Inflater;
@@ -61,6 +61,7 @@ import org.spearce.jgit.lib.MutableObjectId;
 import org.spearce.jgit.lib.ObjectId;
 import org.spearce.jgit.lib.ObjectIdMap;
 import org.spearce.jgit.lib.ObjectLoader;
+import org.spearce.jgit.lib.PackIndexWriter;
 import org.spearce.jgit.lib.ProgressMonitor;
 import org.spearce.jgit.lib.Repository;
 import org.spearce.jgit.util.NB;
@@ -441,34 +442,14 @@ public class IndexPack {
 
 	private void writeIdx() throws IOException {
 		Arrays.sort(entries, 0, entryCount);
-		final int[] fanout = new int[256];
-		for (int i = 0; i < entryCount; i++)
-			fanout[entries[i].getFirstByte() & 0xff]++;
-		for (int i = 1; i < 256; i++)
-			fanout[i] += fanout[i - 1];
-
-		final BufferedOutputStream os = new BufferedOutputStream(
-				new FileOutputStream(dstIdx), BUFFER_SIZE);
+		List<PackedObjectInfo> list = Arrays.asList(entries);
+		if (entryCount < entries.length)
+			list = list.subList(0, entryCount);
+
+		final FileOutputStream os = new FileOutputStream(dstIdx);
 		try {
-			final byte[] rawoe = new byte[4 + Constants.OBJECT_ID_LENGTH];
-			final MessageDigest d = Constants.newMessageDigest();
-			for (int i = 0; i < 256; i++) {
-				NB.encodeInt32(rawoe, 0, fanout[i]);
-				os.write(rawoe, 0, 4);
-				d.update(rawoe, 0, 4);
-			}
-			for (int i = 0; i < entryCount; i++) {
-				final PackedObjectInfo oe = entries[i];
-				if (oe.getOffset() >>> 1 > Integer.MAX_VALUE)
-					throw new IOException("Pack too large for index version 1");
-				NB.encodeInt32(rawoe, 0, (int) oe.getOffset());
-				oe.copyRawTo(rawoe, 4);
-				os.write(rawoe);
-				d.update(rawoe);
-			}
-			os.write(packcsum);
-			d.update(packcsum);
-			os.write(d.digest());
+			PackIndexWriter.createOldestPossible(os, list)
+					.write(list, packcsum);
 		} finally {
 			os.close();
 		}
-- 
1.5.6.74.g8a5e

[JGIT PATCH 06/10] Reuse the magic tOc constant for pack index headers

[Shawn O. Pearce]

We need this constant to detect version 2 index files at read time,
but we also need it to create version 2 index files.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
---
 .../src/org/spearce/jgit/lib/PackIndex.java        |    3 ++-
 .../src/org/spearce/jgit/lib/PackIndexWriter.java  |    3 +++
 2 files changed, 5 insertions(+), 1 deletions(-)

diff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/PackIndex.java b/org.spearce.jgit/src/org/spearce/jgit/lib/PackIndex.java
index 3935d4f..ef4b13d 100644
--- a/org.spearce.jgit/src/org/spearce/jgit/lib/PackIndex.java
+++ b/org.spearce.jgit/src/org/spearce/jgit/lib/PackIndex.java
@@ -42,6 +42,7 @@ import java.io.File;
 import java.io.FileInputStream;
 import java.io.FileNotFoundException;
 import java.io.IOException;
+import java.util.Arrays;
 import java.util.Iterator;
 
 import org.spearce.jgit.util.NB;
@@ -104,7 +105,7 @@ public abstract class PackIndex implements Iterable<PackIndex.MutableEntry> {
 	}
 
 	private static boolean isTOC(final byte[] h) {
-		return h[0] == -1 && h[1] == 't' && h[2] == 'O' && h[3] == 'c';
+		return Arrays.equals(h, PackIndexWriter.TOC);
 	}
 
 	/**
diff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/PackIndexWriter.java b/org.spearce.jgit/src/org/spearce/jgit/lib/PackIndexWriter.java
index 473e6cf..c9b27d2 100644
--- a/org.spearce.jgit/src/org/spearce/jgit/lib/PackIndexWriter.java
+++ b/org.spearce.jgit/src/org/spearce/jgit/lib/PackIndexWriter.java
@@ -55,6 +55,9 @@ import org.spearce.jgit.util.NB;
  * to the byte offset within the pack where the object's data can be read.
  */
 public abstract class PackIndexWriter {
+	/** Magic constant indicating post-version 1 format. */
+	protected static final byte[] TOC = { -1, 't', 'O', 'c' };
+
 	/**
 	 * Create a new writer for the oldest (most widely understood) format.
 	 * <p>
-- 
1.5.6.74.g8a5e

[JGIT PATCH 07/10] Add 64 bit network byte order encoding to NB

[Shawn O. Pearce]

The pack index v2 file format uses a 64 bit network byte order
element for the 64 bit offset table.  We already have code to
read these elements, but we do not yet have code to write such
an element to an output file.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
---
 org.spearce.jgit/src/org/spearce/jgit/util/NB.java |   37 ++++++++++++++++++++
 1 files changed, 37 insertions(+), 0 deletions(-)

diff --git a/org.spearce.jgit/src/org/spearce/jgit/util/NB.java b/org.spearce.jgit/src/org/spearce/jgit/util/NB.java
index 61877b8..6bb65d4 100644
--- a/org.spearce.jgit/src/org/spearce/jgit/util/NB.java
+++ b/org.spearce.jgit/src/org/spearce/jgit/util/NB.java
@@ -172,6 +172,43 @@ public final class NB {
 		intbuf[offset] = (byte) v;
 	}
 
+	/**
+	 * Write a 64 bit integer as a sequence of 8 bytes (network byte order).
+	 * 
+	 * @param intbuf
+	 *            buffer to write the 48bytes of data into.
+	 * @param offset
+	 *            position within the buffer to begin writing to. This position
+	 *            and the next 7 bytes after it (for a total of 8 bytes) will be
+	 *            replaced.
+	 * @param v
+	 *            the value to write.
+	 */
+	public static void encodeInt64(final byte[] intbuf, final int offset, long v) {
+		intbuf[offset + 7] = (byte) v;
+		v >>>= 8;
+
+		intbuf[offset + 6] = (byte) v;
+		v >>>= 8;
+
+		intbuf[offset + 5] = (byte) v;
+		v >>>= 8;
+
+		intbuf[offset + 4] = (byte) v;
+		v >>>= 8;
+
+		intbuf[offset + 3] = (byte) v;
+		v >>>= 8;
+
+		intbuf[offset + 2] = (byte) v;
+		v >>>= 8;
+
+		intbuf[offset + 1] = (byte) v;
+		v >>>= 8;
+
+		intbuf[offset] = (byte) v;
+	}
+
 	private NB() {
 		// Don't create instances of a static only utility.
 	}
-- 
1.5.6.74.g8a5e

[JGIT PATCH 08/10] Compute packed object entry CRC32 data during IndexPack

[Shawn O. Pearce]

In order to create a pack index using the v2 file format we must
have the CRC32 data available for every object we discovered in
that pack stream.  We now compute this on the fly as we read the
object entries in from the input stream.  Always running the CRC
computation isn't that expensive and we can be certain we would
always have the information necessary to create a v2 index file.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
---
 .../src/org/spearce/jgit/transport/IndexPack.java  |  135 ++++++++++++++------
 .../spearce/jgit/transport/PackedObjectInfo.java   |   25 ++++-
 2 files changed, 117 insertions(+), 43 deletions(-)

diff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/IndexPack.java b/org.spearce.jgit/src/org/spearce/jgit/transport/IndexPack.java
index 60e0bce..047f0dc 100644
--- a/org.spearce.jgit/src/org/spearce/jgit/transport/IndexPack.java
+++ b/org.spearce.jgit/src/org/spearce/jgit/transport/IndexPack.java
@@ -49,6 +49,7 @@ import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.HashMap;
 import java.util.List;
+import java.util.zip.CRC32;
 import java.util.zip.DataFormatException;
 import java.util.zip.Deflater;
 import java.util.zip.Inflater;
@@ -136,6 +137,8 @@ public class IndexPack {
 
 	private int entryCount;
 
+	private final CRC32 crc = new CRC32();
+
 	private ObjectIdMap<ArrayList<UnresolvedDelta>> baseById;
 
 	private HashMap<Long, ArrayList<UnresolvedDelta>> baseByPos;
@@ -282,13 +285,15 @@ public class IndexPack {
 	}
 
 	private void resolveDeltas(final PackedObjectInfo oe) throws IOException {
+		final int oldCRC = oe.getCRC();
 		if (baseById.containsKey(oe)
 				|| baseByPos.containsKey(new Long(oe.getOffset())))
-			resolveDeltas(oe.getOffset(), Constants.OBJ_BAD, null, oe);
+			resolveDeltas(oe.getOffset(), oldCRC, Constants.OBJ_BAD, null, oe);
 	}
 
-	private void resolveDeltas(final long pos, int type, byte[] data,
-			PackedObjectInfo oe) throws IOException {
+	private void resolveDeltas(final long pos, final int oldCRC, int type,
+			byte[] data, PackedObjectInfo oe) throws IOException {
+		crc.reset();
 		position(pos);
 		int c = readFromFile();
 		final int typeCode = (c >> 4) & 7;
@@ -309,14 +314,14 @@ public class IndexPack {
 			data = inflateFromFile((int) sz);
 			break;
 		case Constants.OBJ_OFS_DELTA: {
-			c = readFromInput() & 0xff;
+			c = readFromFile() & 0xff;
 			while ((c & 128) != 0)
-				c = readFromInput() & 0xff;
+				c = readFromFile() & 0xff;
 			data = BinaryDelta.apply(data, inflateFromFile((int) sz));
 			break;
 		}
 		case Constants.OBJ_REF_DELTA: {
-			fillFromInput(20);
+			crc.update(buf, fillFromFile(20), 20);
 			use(20);
 			data = BinaryDelta.apply(data, inflateFromFile((int) sz));
 			break;
@@ -325,6 +330,9 @@ public class IndexPack {
 			throw new IOException("Unknown object type " + typeCode + ".");
 		}
 
+		final int crc32 = (int) crc.getValue();
+		if (oldCRC != crc32)
+			throw new IOException("Corruption detected re-reading at " + pos);
 		if (oe == null) {
 			objectDigest.update(Constants.encodedTypeString(type));
 			objectDigest.update((byte) ' ');
@@ -332,7 +340,8 @@ public class IndexPack {
 			objectDigest.update((byte) 0);
 			objectDigest.update(data);
 			tempObjectId.fromRaw(objectDigest.digest(), 0);
-			oe = new PackedObjectInfo(pos, tempObjectId);
+
+			oe = new PackedObjectInfo(pos, crc32, tempObjectId);
 			entries[entryCount++] = oe;
 		}
 
@@ -349,20 +358,24 @@ public class IndexPack {
 				final UnresolvedDelta ad = a.get(ai);
 				final UnresolvedDelta bd = b.get(bi);
 				if (ad.position < bd.position) {
-					resolveDeltas(ad.position, type, data, null);
+					resolveDeltas(ad.position, ad.crc, type, data, null);
 					ai++;
 				} else {
-					resolveDeltas(bd.position, type, data, null);
+					resolveDeltas(bd.position, bd.crc, type, data, null);
 					bi++;
 				}
 			}
 		}
 		if (a != null)
-			while (ai < a.size())
-				resolveDeltas(a.get(ai++).position, type, data, null);
+			while (ai < a.size()) {
+				final UnresolvedDelta ad = a.get(ai++);
+				resolveDeltas(ad.position, ad.crc, type, data, null);
+			}
 		if (b != null)
-			while (bi < b.size())
-				resolveDeltas(b.get(bi++).position, type, data, null);
+			while (bi < b.size()) {
+				final UnresolvedDelta bd = b.get(bi++);
+				resolveDeltas(bd.position, bd.crc, type, data, null);
+			}
 	}
 
 	private void fixThinPack(final ProgressMonitor progress) throws IOException {
@@ -377,10 +390,11 @@ public class IndexPack {
 			final int typeCode = ldr.getType();
 			final PackedObjectInfo oe;
 
-			oe = new PackedObjectInfo(end, baseId);
+			crc.reset();
+			writeWhole(def, typeCode, data);
+			oe = new PackedObjectInfo(end, (int) crc.getValue(), baseId);
 			entries[entryCount++] = oe;
 			packOut.seek(end);
-			writeWhole(def, typeCode, data);
 			end = packOut.getFilePointer();
 
 			resolveChildDeltas(oe.getOffset(), typeCode, data, oe);
@@ -403,12 +417,16 @@ public class IndexPack {
 			buf[hdrlen++] = (byte) (sz & 0x7f);
 			sz >>>= 7;
 		}
+		crc.update(buf, 0, hdrlen);
 		packOut.write(buf, 0, hdrlen);
 		def.reset();
 		def.setInput(data);
 		def.finish();
-		while (!def.finished())
-			packOut.write(buf, 0, def.deflate(buf));
+		while (!def.finished()) {
+			final int datlen = def.deflate(buf);
+			crc.update(buf, 0, datlen);
+			packOut.write(buf, 0, datlen);
+		}
 	}
 
 	private void fixHeaderFooter() throws IOException {
@@ -493,6 +511,7 @@ public class IndexPack {
 	private void indexOneObject() throws IOException {
 		final long pos = position();
 
+		crc.reset();
 		int c = readFromInput();
 		final int typeCode = (c >> 4) & 7;
 		long sz = c & 15;
@@ -511,11 +530,11 @@ public class IndexPack {
 			whole(typeCode, pos, sz);
 			break;
 		case Constants.OBJ_OFS_DELTA: {
-			c = readFromInput() & 0xff;
+			c = readFromInput();
 			long ofs = c & 127;
 			while ((c & 128) != 0) {
 				ofs += 1;
-				c = readFromInput() & 0xff;
+				c = readFromInput();
 				ofs <<= 7;
 				ofs += (c & 127);
 			}
@@ -525,25 +544,24 @@ public class IndexPack {
 				r = new ArrayList<UnresolvedDelta>(8);
 				baseByPos.put(base, r);
 			}
-			r.add(new UnresolvedDelta(pos));
-			deltaCount++;
 			inflateFromInput(false);
+			r.add(new UnresolvedDelta(pos, (int) crc.getValue()));
+			deltaCount++;
 			break;
 		}
 		case Constants.OBJ_REF_DELTA: {
 			c = fillFromInput(20);
-			final byte[] ref = new byte[20];
-			System.arraycopy(buf, c, ref, 0, 20);
+			crc.update(buf, c, 20);
+			final ObjectId base = ObjectId.fromRaw(buf, c);
 			use(20);
-			final ObjectId base = ObjectId.fromRaw(ref);
 			ArrayList<UnresolvedDelta> r = baseById.get(base);
 			if (r == null) {
 				r = new ArrayList<UnresolvedDelta>(8);
 				baseById.put(base, r);
 			}
-			r.add(new UnresolvedDelta(pos));
-			deltaCount++;
 			inflateFromInput(false);
+			r.add(new UnresolvedDelta(pos, (int) crc.getValue()));
+			deltaCount++;
 			break;
 		}
 		default:
@@ -559,7 +577,9 @@ public class IndexPack {
 		objectDigest.update((byte) 0);
 		inflateFromInput(true);
 		tempObjectId.fromRaw(objectDigest.digest(), 0);
-		entries[entryCount++] = new PackedObjectInfo(pos, tempObjectId);
+
+		final int crc32 = (int) crc.getValue();
+		entries[entryCount++] = new PackedObjectInfo(pos, crc32, tempObjectId);
 	}
 
 	// Current position of {@link #bOffset} within the entire file.
@@ -579,15 +599,19 @@ public class IndexPack {
 		if (bAvail == 0)
 			fillFromInput(1);
 		bAvail--;
-		return buf[bOffset++] & 0xff;
+		final int b = buf[bOffset++] & 0xff;
+		crc.update(b);
+		return b;
 	}
 
 	// Consume exactly one byte from the buffer and return it.
 	private int readFromFile() throws IOException {
 		if (bAvail == 0)
-			fillFromFile();
+			fillFromFile(1);
 		bAvail--;
-		return buf[bOffset++] & 0xff;
+		final int b = buf[bOffset++] & 0xff;
+		crc.update(b);
+		return b;
 	}
 
 	// Consume cnt bytes from the buffer.
@@ -615,13 +639,21 @@ public class IndexPack {
 	}
 
 	// Ensure at least need bytes are available in in {@link #buf}.
-	private int fillFromFile() throws IOException {
-		if (bAvail == 0) {
-			final int next = packOut.read(buf, 0, buf.length);
+	private int fillFromFile(final int need) throws IOException {
+		if (bAvail < need) {
+			int next = bOffset + bAvail;
+			int free = buf.length - next;
+			if (free + bAvail < need) {
+				if (bAvail > 0)
+					System.arraycopy(buf, bOffset, buf, 0, bAvail);
+				bOffset = 0;
+				next = bAvail;
+				free = buf.length - next;
+			}
+			next = packOut.read(buf, next, free);
 			if (next <= 0)
 				throw new EOFException("Packfile is truncated.");
-			bAvail = next;
-			bOffset = 0;
+			bAvail += next;
 		}
 		return bOffset;
 	}
@@ -642,11 +674,15 @@ public class IndexPack {
 		try {
 			final byte[] dst = objectData;
 			int n = 0;
+			int p = -1;
 			while (!inf.finished()) {
 				if (inf.needsInput()) {
-					final int p = fillFromInput(1);
+					if (p >= 0) {
+						crc.update(buf, p, bAvail);
+						use(bAvail);
+					}
+					p = fillFromInput(1);
 					inf.setInput(buf, p, bAvail);
-					use(bAvail);
 				}
 
 				int free = dst.length - n;
@@ -661,7 +697,11 @@ public class IndexPack {
 			}
 			if (digest)
 				objectDigest.update(dst, 0, n);
-			use(-inf.getRemaining());
+			n = bAvail - inf.getRemaining();
+			if (n > 0) {
+				crc.update(buf, p, n);
+				use(n);
+			}
 		} catch (DataFormatException dfe) {
 			throw corrupt(dfe);
 		} finally {
@@ -674,15 +714,23 @@ public class IndexPack {
 		try {
 			final byte[] dst = new byte[sz];
 			int n = 0;
+			int p = -1;
 			while (!inf.finished()) {
 				if (inf.needsInput()) {
-					final int p = fillFromFile();
+					if (p >= 0) {
+						crc.update(buf, p, bAvail);
+						use(bAvail);
+					}
+					p = fillFromFile(1);
 					inf.setInput(buf, p, bAvail);
-					use(bAvail);
 				}
 				n += inf.inflate(dst, n, sz - n);
 			}
-			use(-inf.getRemaining());
+			n = bAvail - inf.getRemaining();
+			if (n > 0) {
+				crc.update(buf, p, n);
+				use(n);
+			}
 			return dst;
 		} catch (DataFormatException dfe) {
 			throw corrupt(dfe);
@@ -699,8 +747,11 @@ public class IndexPack {
 	private static class UnresolvedDelta {
 		final long position;
 
-		UnresolvedDelta(final long headerOffset) {
+		final int crc;
+
+		UnresolvedDelta(final long headerOffset, final int crc32) {
 			position = headerOffset;
+			crc = crc32;
 		}
 	}
 
diff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/PackedObjectInfo.java b/org.spearce.jgit/src/org/spearce/jgit/transport/PackedObjectInfo.java
index 58feada..eae34df 100644
--- a/org.spearce.jgit/src/org/spearce/jgit/transport/PackedObjectInfo.java
+++ b/org.spearce.jgit/src/org/spearce/jgit/transport/PackedObjectInfo.java
@@ -51,9 +51,13 @@ import org.spearce.jgit.lib.ObjectId;
 public class PackedObjectInfo extends ObjectId {
 	private long offset;
 
-	PackedObjectInfo(final long headerOffset, final AnyObjectId id) {
+	private int crc;
+
+	PackedObjectInfo(final long headerOffset, final int packedCRC,
+			final AnyObjectId id) {
 		super(id);
 		offset = headerOffset;
+		crc = packedCRC;
 	}
 
 	/**
@@ -83,4 +87,23 @@ public class PackedObjectInfo extends ObjectId {
 	public void setOffset(final long offset) {
 		this.offset = offset;
 	}
+
+	/**
+	 * @return the 32 bit CRC checksum for the packed data.
+	 */
+	public int getCRC() {
+		return crc;
+	}
+
+	/**
+	 * Record the 32 bit CRC checksum for the packed data.
+	 * 
+	 * @param crc
+	 *            checksum of all packed data (including object type code,
+	 *            inflated length and delta base reference) as computed by
+	 *            {@link java.util.zip.CRC32}.
+	 */
+	public void setCRC(final int crc) {
+		this.crc = crc;
+	}
 }
-- 
1.5.6.74.g8a5e

[JGIT PATCH 09/10] Add support for writing pack index v2 files

[Shawn O. Pearce]

The v2 format is more robust for delta reuse as it has a CRC
element that covers the entire packed representation, permitting
more efficient delta-reuse during packing.  It also can address
objects in pack files larger than 4 GB, making it a better format
for the future.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
---
 .../src/org/spearce/jgit/lib/PackIndexWriter.java  |   21 ++++
 .../org/spearce/jgit/lib/PackIndexWriterV2.java    |  101 ++++++++++++++++++++
 .../src/org/spearce/jgit/pgm/IndexPack.java        |    4 +
 .../src/org/spearce/jgit/transport/IndexPack.java  |   22 ++++-
 4 files changed, 146 insertions(+), 2 deletions(-)
 create mode 100644 org.spearce.jgit/src/org/spearce/jgit/lib/PackIndexWriterV2.java

diff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/PackIndexWriter.java b/org.spearce.jgit/src/org/spearce/jgit/lib/PackIndexWriter.java
index c9b27d2..2d9d822 100644
--- a/org.spearce.jgit/src/org/spearce/jgit/lib/PackIndexWriter.java
+++ b/org.spearce.jgit/src/org/spearce/jgit/lib/PackIndexWriter.java
@@ -122,6 +122,8 @@ public abstract class PackIndexWriter {
 		switch (version) {
 		case 1:
 			return new PackIndexWriterV1(dst);
+		case 2:
+			return new PackIndexWriterV2(dst);
 		default:
 			throw new IllegalArgumentException(
 					"Unsupported pack index version " + version);
@@ -203,6 +205,25 @@ public abstract class PackIndexWriter {
 	protected abstract void writeImpl() throws IOException;
 
 	/**
+	 * Output the version 2 (and later) TOC header, with version number.
+	 * <p>
+	 * Post version 1 all index files start with a TOC header that makes the
+	 * file an invalid version 1 file, and then includes the version number.
+	 * This header is necessary to recognize a version 1 from a version 2
+	 * formatted index.
+	 * 
+	 * @param version
+	 *            version number of this index format being written.
+	 * @throws IOException
+	 *             an error occurred while writing to the output stream.
+	 */
+	protected void writeTOC(final int version) throws IOException {
+		out.write(TOC);
+		NB.encodeInt32(tmp, 0, version);
+		out.write(tmp, 0, 4);
+	}
+
+	/**
 	 * Output the standard 256 entry first-level fan-out table.
 	 * <p>
 	 * The fan-out table is 4 KB in size, holding 256 32-bit unsigned integer
diff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/PackIndexWriterV2.java b/org.spearce.jgit/src/org/spearce/jgit/lib/PackIndexWriterV2.java
new file mode 100644
index 0000000..8fa4d1a
--- /dev/null
+++ b/org.spearce.jgit/src/org/spearce/jgit/lib/PackIndexWriterV2.java
@@ -0,0 +1,101 @@
+/*
+ * Copyright (C) 2008, Shawn O. Pearce <spearce@spearce.org>
+ *
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * - Redistributions of source code must retain the above copyright
+ *   notice, this list of conditions and the following disclaimer.
+ *
+ * - Redistributions in binary form must reproduce the above
+ *   copyright notice, this list of conditions and the following
+ *   disclaimer in the documentation and/or other materials provided
+ *   with the distribution.
+ *
+ * - Neither the name of the Git Development Community nor the
+ *   names of its contributors may be used to endorse or promote
+ *   products derived from this software without specific prior
+ *   written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND
+ * CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES,
+ * INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+ * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
+ * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+ * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+ * CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+ * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
+ * ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+package org.spearce.jgit.lib;
+
+import java.io.IOException;
+import java.io.OutputStream;
+
+import org.spearce.jgit.transport.PackedObjectInfo;
+import org.spearce.jgit.util.NB;
+
+/**
+ * Creates the version 2 pack table of contents files.
+ * 
+ * @see PackIndexWriter
+ * @see PackIndexV2
+ */
+class PackIndexWriterV2 extends PackIndexWriter {
+	PackIndexWriterV2(final OutputStream dst) {
+		super(dst);
+	}
+
+	@Override
+	protected void writeImpl() throws IOException {
+		writeTOC(2);
+		writeFanOutTable();
+		writeObjectNames();
+		writeCRCs();
+		writeOffset32();
+		writeOffset64();
+		writeChecksumFooter();
+	}
+
+	private void writeObjectNames() throws IOException {
+		for (final PackedObjectInfo oe : entries)
+			oe.copyRawTo(out);
+	}
+
+	private void writeCRCs() throws IOException {
+		for (final PackedObjectInfo oe : entries) {
+			NB.encodeInt32(tmp, 0, oe.getCRC());
+			out.write(tmp, 0, 4);
+		}
+	}
+
+	private void writeOffset32() throws IOException {
+		int o64 = 0;
+		for (final PackedObjectInfo oe : entries) {
+			final long o = oe.getOffset();
+			if (o < Integer.MAX_VALUE)
+				NB.encodeInt32(tmp, 0, (int) o);
+			else
+				NB.encodeInt32(tmp, 0, (1 << 31) | o64++);
+			out.write(tmp, 0, 4);
+		}
+	}
+
+	private void writeOffset64() throws IOException {
+		for (final PackedObjectInfo oe : entries) {
+			final long o = oe.getOffset();
+			if (o > Integer.MAX_VALUE) {
+				NB.encodeInt64(tmp, 0, o);
+				out.write(tmp, 0, 8);
+			}
+		}
+	}
+}
diff --git a/org.spearce.jgit/src/org/spearce/jgit/pgm/IndexPack.java b/org.spearce.jgit/src/org/spearce/jgit/pgm/IndexPack.java
index 5a82a35..60926c1 100644
--- a/org.spearce.jgit/src/org/spearce/jgit/pgm/IndexPack.java
+++ b/org.spearce.jgit/src/org/spearce/jgit/pgm/IndexPack.java
@@ -47,10 +47,13 @@ class IndexPack extends TextBuiltin {
 	void execute(final String[] args) throws Exception {
 		boolean fixThin = false;
 		int argi = 0;
+		int version = 0;
 		for (; argi < args.length; argi++) {
 			final String a = args[argi];
 			if ("--fix-thin".equals(a))
 				fixThin = true;
+			else if (a.startsWith("--index-version="))
+				version = Integer.parseInt(a.substring(a.indexOf('=') + 1));
 			else if ("--".equals(a)) {
 				argi++;
 				break;
@@ -69,6 +72,7 @@ class IndexPack extends TextBuiltin {
 		in = new BufferedInputStream(System.in);
 		ip = new org.spearce.jgit.transport.IndexPack(db, in, base);
 		ip.setFixThin(fixThin);
+		ip.setIndexVersion(version);
 		ip.index(new TextProgressMonitor());
 	}
 }
diff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/IndexPack.java b/org.spearce.jgit/src/org/spearce/jgit/transport/IndexPack.java
index 047f0dc..06ef7cc 100644
--- a/org.spearce.jgit/src/org/spearce/jgit/transport/IndexPack.java
+++ b/org.spearce.jgit/src/org/spearce/jgit/transport/IndexPack.java
@@ -125,6 +125,8 @@ public class IndexPack {
 
 	private boolean fixThin;
 
+	private int outputVersion;
+
 	private final File dstPack;
 
 	private final File dstIdx;
@@ -185,6 +187,18 @@ public class IndexPack {
 	}
 
 	/**
+	 * Set the pack index file format version this instance will create.
+	 * 
+	 * @param version
+	 *            the version to write. The special version 0 designates the
+	 *            oldest (most compatible) format available for the objects.
+	 * @see PackIndexWriter
+	 */
+	public void setIndexVersion(final int version) {
+		outputVersion = version;
+	}
+
+	/**
 	 * Configure this index pack instance to make a thin pack complete.
 	 * <p>
 	 * Thin packs are sometimes used during network transfers to allow a delta
@@ -466,8 +480,12 @@ public class IndexPack {
 
 		final FileOutputStream os = new FileOutputStream(dstIdx);
 		try {
-			PackIndexWriter.createOldestPossible(os, list)
-					.write(list, packcsum);
+			final PackIndexWriter iw;
+			if (outputVersion <= 0)
+				iw = PackIndexWriter.createOldestPossible(os, list);
+			else
+				iw = PackIndexWriter.createVersion(os, outputVersion);
+			iw.write(list, packcsum);
 		} finally {
 			os.close();
 		}
-- 
1.5.6.74.g8a5e

[JGIT PATCH 10/10] Default IndexPack to honor pack.indexversion configuration

[Shawn O. Pearce]

Users may already desire to create only v2 pack index files, as
the extra CRC code makes repacking faster due to quicker delta
reuse code paths being available.  However we still must default
to version 0 to select oldest version available to improve our
changes of being compatible with really old Git executables.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
---
 .../src/org/spearce/jgit/lib/CoreConfig.java       |   11 +++++++++++
 .../src/org/spearce/jgit/transport/IndexPack.java  |    4 +++-
 2 files changed, 14 insertions(+), 1 deletions(-)

diff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/CoreConfig.java b/org.spearce.jgit/src/org/spearce/jgit/lib/CoreConfig.java
index 01d4210..2dd8aea 100644
--- a/org.spearce.jgit/src/org/spearce/jgit/lib/CoreConfig.java
+++ b/org.spearce.jgit/src/org/spearce/jgit/lib/CoreConfig.java
@@ -48,8 +48,11 @@ public class CoreConfig {
 
 	private final int compression;
 
+	private final int packIndexVersion;
+
 	CoreConfig(final RepositoryConfig rc) {
 		compression = rc.getInt("core", "compression", DEFAULT_COMPRESSION);
+		packIndexVersion = rc.getInt("pack", "indexversion", 0);
 	}
 
 	/**
@@ -59,4 +62,12 @@ public class CoreConfig {
 	public int getCompression() {
 		return compression;
 	}
+
+	/**
+	 * @return the preferred pack index file format; 0 for oldest possible.
+	 * @see org.spearce.jgit.transport.IndexPack
+	 */
+	public int getPackIndexVersion() {
+		return packIndexVersion;
+	}
 }
diff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/IndexPack.java b/org.spearce.jgit/src/org/spearce/jgit/transport/IndexPack.java
index 06ef7cc..8083cc8 100644
--- a/org.spearce.jgit/src/org/spearce/jgit/transport/IndexPack.java
+++ b/org.spearce.jgit/src/org/spearce/jgit/transport/IndexPack.java
@@ -102,7 +102,9 @@ public class IndexPack {
 		final File base;
 
 		base = new File(objdir, n.substring(0, n.length() - suffix.length()));
-		return new IndexPack(db, is, base);
+		final IndexPack ip = new IndexPack(db, is, base);
+		ip.setIndexVersion(db.getConfig().getCore().getPackIndexVersion());
+		return ip;
 	}
 
 	private final Repository repo;
-- 
1.5.6.74.g8a5e

[JGIT PATCH 06/10 v2] Reuse the magic tOc constant for pack index headers

[Shawn O. Pearce]

We need this constant to detect version 2 index files at read time,
but we also need it to create version 2 index files.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
---

 Fixed isTOC test to return true when h.length == 8, which
 is always the case as it has both the TOC and the version.

 Also rebased into my branch:
 
   repo.or.cz:/srv/git/egit/spearce.git index-v2

 With this patch in the series all tests pass again.

 .../src/org/spearce/jgit/lib/PackIndex.java        |    6 +++++-
 .../src/org/spearce/jgit/lib/PackIndexWriter.java  |    3 +++
 2 files changed, 8 insertions(+), 1 deletions(-)

diff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/PackIndex.java b/org.spearce.jgit/src/org/spearce/jgit/lib/PackIndex.java
index 3935d4f..c5718fa 100644
--- a/org.spearce.jgit/src/org/spearce/jgit/lib/PackIndex.java
+++ b/org.spearce.jgit/src/org/spearce/jgit/lib/PackIndex.java
@@ -104,7 +104,11 @@ public abstract class PackIndex implements Iterable<PackIndex.MutableEntry> {
 	}
 
 	private static boolean isTOC(final byte[] h) {
-		return h[0] == -1 && h[1] == 't' && h[2] == 'O' && h[3] == 'c';
+		final byte[] toc = PackIndexWriter.TOC;
+		for (int i = 0; i < toc.length; i++)
+			if (h[i] != toc[i])
+				return false;
+		return true;
 	}
 
 	/**
diff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/PackIndexWriter.java b/org.spearce.jgit/src/org/spearce/jgit/lib/PackIndexWriter.java
index 473e6cf..c9b27d2 100644
--- a/org.spearce.jgit/src/org/spearce/jgit/lib/PackIndexWriter.java
+++ b/org.spearce.jgit/src/org/spearce/jgit/lib/PackIndexWriter.java
@@ -55,6 +55,9 @@ import org.spearce.jgit.util.NB;
  * to the byte offset within the pack where the object's data can be read.
  */
 public abstract class PackIndexWriter {
+	/** Magic constant indicating post-version 1 format. */
+	protected static final byte[] TOC = { -1, 't', 'O', 'c' };
+
 	/**
 	 * Create a new writer for the oldest (most widely understood) format.
 	 * <p>
-- 
1.5.6.74.g8a5e

-- 
Shawn.

Re: [JGIT PATCH 00/10] Support writing pack index version 2

[Robin Rosenberg]

Ocular review looks fine, but nevertheless some tests break.

-- robin

org.spearce.jgit--All-Tests
org.spearce.jgit.lib.PackIndexV2Test
testIteratorMethodsContract(org.spearce.jgit.lib.PackIndexV2Test)
java.io.IOException: Unreadable pack index: /home/me/SW/EGIT.contrib/org.spearce.jgit.test/tst/pack-34be9032ac282b11fa9babdc2b2a93ca996c9c2f.idxV2
	at org.spearce.jgit.lib.PackIndex.open(PackIndex.java:95)
	at org.spearce.jgit.lib.PackIndexTest.setUp(PackIndexTest.java:54)
	at junit.framework.TestCase.runBare(TestCase.java:128)
	at junit.framework.TestResult$1.protect(TestResult.java:106)
	at junit.framework.TestResult.runProtected(TestResult.java:124)
	at junit.framework.TestResult.run(TestResult.java:109)
	at junit.framework.TestCase.run(TestCase.java:120)
	at junit.framework.TestSuite.runTest(TestSuite.java:230)
	at junit.framework.TestSuite.run(TestSuite.java:225)
	at org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestReference.run(JUnit3TestReference.java:130)
	at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:460)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:673)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:386)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:196)
Caused by: java.io.EOFException: Short read of block.
	at org.spearce.jgit.util.NB.readFully(NB.java:67)
	at org.spearce.jgit.lib.PackIndexV1.<init>(PackIndexV1.java:75)
	at org.spearce.jgit.lib.PackIndex.open(PackIndex.java:91)
	... 14 more

testIteratorReturnedValues1(org.spearce.jgit.lib.PackIndexV2Test) 
	same trace

testCompareEntriesOffsetsWithFindOffsets(org.spearce.jgit.lib.PackIndexV2Test)
	same trace

testIteratorReturnedValues2(org.spearce.jgit.lib.PackIndexV2Test)
	same trace

Together with Marek's packwrite patches the list gets even longer (his patches alone are "green"). I'm
not including them here.

-- robin

Re: [JGIT PATCH 00/10] Support writing pack index version 2

[Shawn O. Pearce]

Robin Rosenberg <robin.rosenberg.lists@dewire.com> wrote:
> 
> Ocular review looks fine, but nevertheless some tests break.

Dammit.
 
> org.spearce.jgit.lib.PackIndexV2Test
> testIteratorMethodsContract(org.spearce.jgit.lib.PackIndexV2Test)
> java.io.IOException: Unreadable pack index: /home/me/SW/EGIT.contrib/org.spearce.jgit.test/tst/pack-34be9032ac282b11fa9babdc2b2a93ca996c9c2f.idxV2
> 	at org.spearce.jgit.lib.PackIndex.open(PackIndex.java:95)
...
> 	at org.spearce.jgit.lib.PackIndexV1.<init>(PackIndexV1.java:75)

Yea, OK, I know what that is.  I busted PackIndex.isTOC.

I asked it to compare a 4 byte array (TOC only) to an 8 byte array
(TOC + version) and of course 4 != 8 so it fails.  Thus all V2
index files look like V1 files.  Only they aren't; and then all
hell breaks loose when we start treating parts of the V2 index as
different sections of the V1 index.

I'll post a replacement patch in a minute.

> Together with Marek's packwrite patches the list gets even longer (his patches alone are "green"). I'm
> not including them here.

That's because Marek's branch includes additional tests for features
he added to PackIndexV2.  But all PackIndexV2 reading is busted.

-- 
Shawn.