[PATCH JGIT] Allow writeObject() write to OutputStream

[PATCH JGIT] Allow writeObject() write to OutputStream

[Daniel Cheng (aka SDiZ)]

Signed-off-by: Daniel Cheng (aka SDiZ) <j16sdiz+freenet@gmail.com>
---

This patch make factor out the object writing code in ObjectWriter,
allow it to write to any OutputStream. 
Subclass class may then override
  writeObject(final int type, long len, InputStream is, boolean store)
to make it write to alternative locations.

There are some discussion on devl@freenetproject.org to use raw 
(uncompressed) object to freenet. This patch allow the testing.

 .../src/org/spearce/jgit/lib/ObjectWriter.java     |   93 +++++++++++---------
 1 files changed, 50 insertions(+), 43 deletions(-)

diff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/ObjectWriter.java b/org.spearce.jgit/src/org/spearce/jgit/lib/ObjectWriter.java
index 546cc68..97acae4 100644
--- a/org.spearce.jgit/src/org/spearce/jgit/lib/ObjectWriter.java
+++ b/org.spearce.jgit/src/org/spearce/jgit/lib/ObjectWriter.java
@@ -45,6 +45,7 @@
 import java.io.FileOutputStream;
 import java.io.IOException;
 import java.io.InputStream;
+import java.io.OutputStream;
 import java.io.OutputStreamWriter;
 import java.security.MessageDigest;
 import java.util.zip.Deflater;
@@ -297,8 +298,52 @@ public ObjectId computeBlobSha1(final long len, final InputStream is)
 		return writeObject(Constants.OBJ_BLOB, len, is, false);
 	}
 
-	ObjectId writeObject(final int type, long len, final InputStream is,
-			boolean store) throws IOException {
+	protected ObjectId writeObject(final int type, long len,
+			final InputStream is, final OutputStream deflateStream)
+			throws IOException {
+		md.reset();
+
+		byte[] header;
+		int n;
+
+		header = Constants.encodedTypeString(type);
+		md.update(header);
+		if (deflateStream != null)
+			deflateStream.write(header);
+
+		md.update((byte) ' ');
+		if (deflateStream != null)
+			deflateStream.write((byte) ' ');
+
+		header = Constants.encodeASCII(len);
+		md.update(header);
+		if (deflateStream != null)
+			deflateStream.write(header);
+
+		md.update((byte) 0);
+		if (deflateStream != null)
+			deflateStream.write((byte) 0);
+
+		while (len > 0
+				&& (n = is.read(buf, 0, (int) Math.min(len, buf.length))) > 0) {
+			md.update(buf, 0, n);
+			if (deflateStream != null)
+				deflateStream.write(buf, 0, n);
+			len -= n;
+		}
+
+		if (len != 0)
+			throw new IOException("Input did not match supplied length. " + len
+					+ " bytes are missing.");
+
+		if (deflateStream != null)
+			deflateStream.close();
+
+		return ObjectId.fromRaw(md.digest());
+	}
+
+	protected ObjectId writeObject(final int type, long len,
+			final InputStream is, boolean store) throws IOException {
 		final File t;
 		final DeflaterOutputStream deflateStream;
 		final FileOutputStream fileStream;
@@ -312,7 +357,6 @@ ObjectId writeObject(final int type, long len, final InputStream is,
 			fileStream = null;
 		}
 
-		md.reset();
 		if (store) {
 			def.reset();
 			deflateStream = new DeflaterOutputStream(fileStream, def);
@@ -320,46 +364,9 @@ ObjectId writeObject(final int type, long len, final InputStream is,
 			deflateStream = null;
 
 		try {
-			byte[] header;
-			int n;
-
-			header = Constants.encodedTypeString(type);
-			md.update(header);
-			if (deflateStream != null)
-				deflateStream.write(header);
-
-			md.update((byte) ' ');
-			if (deflateStream != null)
-				deflateStream.write((byte) ' ');
-
-			header = Constants.encodeASCII(len);
-			md.update(header);
-			if (deflateStream != null)
-				deflateStream.write(header);
-
-			md.update((byte) 0);
-			if (deflateStream != null)
-				deflateStream.write((byte) 0);
-
-			while (len > 0
-					&& (n = is.read(buf, 0, (int) Math.min(len, buf.length))) > 0) {
-				md.update(buf, 0, n);
-				if (deflateStream != null)
-					deflateStream.write(buf, 0, n);
-				len -= n;
-			}
-
-			if (len != 0)
-				throw new IOException("Input did not match supplied length. "
-						+ len + " bytes are missing.");
-
-			if (deflateStream != null ) {
-				deflateStream.close();
-				if (t != null)
-					t.setReadOnly();
-			}
-
-			id = ObjectId.fromRaw(md.digest());
+			id = writeObject(type, len, is, deflateStream);
+			if (t != null)
+				t.setReadOnly();
 		} finally {
 			if (id == null && deflateStream != null) {
 				try {
-- 
1.6.2

Re: [PATCH JGIT] Allow writeObject() write to OutputStream

[Shawn O. Pearce]

"Daniel Cheng (aka SDiZ)" <j16sdiz+freenet@gmail.com> wrote:
> 
> This patch make factor out the object writing code in ObjectWriter,
> allow it to write to any OutputStream. 
> Subclass class may then override
>   writeObject(final int type, long len, InputStream is, boolean store)
> to make it write to alternative locations.
> 
> There are some discussion on devl@freenetproject.org to use raw 
> (uncompressed) object to freenet. This patch allow the testing.

Ok, I understand the code as-is, but I'm not sure I understand the
reasoning for the change, or where you are trying to go with it.

Are you guys talking about making every object a loose object on
freenet, and avoiding pack files?  Or making a form of JGit that
access a Repository directly stored on freenode?

I ask because there's some folks starting to talk about putting JGit
onto a distributed hash table sort of system like Hadoop HBase,
to allow the underlying storage to scale efficiently for really
big hosting sites.  I would rather see a pooling of effort here
than folks going in different directions.
 
-- 
Shawn.

Re: [PATCH JGIT] Allow writeObject() write to OutputStream

[Daniel Cheng]

On Fri, Mar 13, 2009 at 11:08 PM, Shawn O. Pearce <spearce@spearce.org> wrote:
> "Daniel Cheng (aka SDiZ)" <j16sdiz+freenet@gmail.com> wrote:
>>
>> This patch make factor out the object writing code in ObjectWriter,
>> allow it to write to any OutputStream.
>> Subclass class may then override
>>   writeObject(final int type, long len, InputStream is, boolean store)
>> to make it write to alternative locations.
>>
>> There are some discussion on devl@freenetproject.org to use raw
>> (uncompressed) object to freenet. This patch allow the testing.
>
> Ok, I understand the code as-is, but I'm not sure I understand the
> reasoning for the change, or where you are trying to go with it.

In freenet (and most content-addressable network), file can be dropped
out when unused.
We need some method to "heal" the lost data.
The easiest way is to do this is re-insert the very same file we used
originally.

Pack files may change on different compression parameter, object order, etc.
It need some tricks to get the original file.
Loose object are immutable, so it is easier to use loose objects.

Nothing have finalized yet, we are just evaluating different approaches.

I know loose object are always larger, and have very large number of them.
So it may backfire ....

> Are you guys talking about making every object a loose object on
> freenet, and avoiding pack files?  Or making a form of JGit that
> access a Repository directly stored on freenode?
>
> I ask because there's some folks starting to talk about putting JGit
> onto a distributed hash table sort of system like Hadoop HBase,
> to allow the underlying storage to scale efficiently for really
> big hosting sites.  I would rather see a pooling of effort here
> than folks going in different directions.
>
> --
> Shawn.
>