* [JGIT PATCH 00/13] Add receive-pack server side support
@ 2008-12-23 0:27 Shawn O. Pearce
2008-12-23 0:27 ` [JGIT PATCH 01/13] Fix invalid "double checked locking" in InflaterCache Shawn O. Pearce
0 siblings, 1 reply; 17+ messages in thread
From: Shawn O. Pearce @ 2008-12-23 0:27 UTC (permalink / raw)
To: Robin Rosenberg; +Cc: git
This is a simple implementation of the receive-pack part of the
send-pack/receive-pack protocol used by push. Most of the logic
the server implements can be customized by implementing your
own PreReceiveHook implementation.
I'm actually going to use this within Gerrit 2 so users can push
into the special ref name "refs/changes/new" and have an anonymous
topic branch be created instead.
A basic "jgit daemon" for the git:// protocol is also included.
This daemon still has some major limitations over "git daemon":
- Only "receive-pack" is supported. We do not have a Java
version of "upload-pack" so we do not support it here.
- Virtual hosting is not supported; the "host=" header (if sent
by the client) is discarded.
- User escapes ("git://host/~user/foo.git") are not honored.
- No receive timeouts. Evil clients can connect and do nothing,
pinning the server thread indefinitely. This can cause the
server to consume resources until it crashes.
- No connection limits. Evil clients can DOS the server by just
opening a ton of connections.
- "jgit daemon" must be started within a git repository, even if
it won't publish that repository over the network. This is due
to a limitation in the way TextBuiltin initializes itself.
Providing "jgit receive-pack" should be trivial if the last item
is fixed, permitting jgit commands to be executed from outside of
a Git repository.
JSch does not have an SSH server implementation, so we cannot easily
provide an SSH daemon for our new push support. Apache MINA SSHD has
a nice daemon under development, and I plan on using it in Gerrit 2.
Perhaps someone will eventually contribute an extension library
for JGit which provides the binding glue necessary for MINA SSHD
to invoke JGit.
JGit's ReceivePack class does not invoke the standard hooks honored
by "git receive-pack", because doing so would require executing
an external process. There's enough data available in memory that
someone could easily contribute a PreReceiveHook and PostRecieveHook
implementation that honored these.
The early part of the series has some code cleanups to ensure we
are thread-safe, and to reduce the thrasing on the InflaterCache.
Shawn O. Pearce (13):
Fix invalid "double checked locking" in InflaterCache
Cleanup stupid release of the cached Inflater in IndexPack
Cache an Inflater inside a WindowCursor and reuse it as much as
possible
Make RefDatabase thread-safe
Make PackFile thread-safe
Make Repository thread-safe
Don't open a PackFile multiple times on scanForPacks
Expose RepositoryConfig.getBoolean so applications can use it
Add AnyObjectId.copyTo(StringBuilder)
Add compare-and-swap semantics to RefUpdate
Allow null new ObjectId during RefUpdate.delete
Implement the git-receive-pack process in Java
Add basic git daemon support to publish receive-pack
.../services/org.spearce.jgit.pgm.TextBuiltin | 1 +
.../src/org/spearce/jgit/pgm/Daemon.java | 125 +++
.../src/org/spearce/jgit/lib/AnyObjectId.java | 17 +-
.../jgit/lib/DeltaRefPackedObjectLoader.java | 2 +-
.../src/org/spearce/jgit/lib/InflaterCache.java | 31 +-
.../src/org/spearce/jgit/lib/PackFile.java | 14 +-
.../src/org/spearce/jgit/lib/RefDatabase.java | 29 +-
.../src/org/spearce/jgit/lib/RefUpdate.java | 32 +-
.../src/org/spearce/jgit/lib/Repository.java | 90 ++-
.../src/org/spearce/jgit/lib/RepositoryConfig.java | 4 +-
.../src/org/spearce/jgit/lib/WindowCursor.java | 33 +-
.../src/org/spearce/jgit/lib/WindowedFile.java | 17 +-
.../src/org/spearce/jgit/revwalk/RevWalk.java | 4 +-
.../src/org/spearce/jgit/transport/Daemon.java | 309 ++++++++
.../org/spearce/jgit/transport/DaemonClient.java | 106 +++
.../org/spearce/jgit/transport/DaemonService.java | 120 +++
.../src/org/spearce/jgit/transport/IndexPack.java | 17 +-
.../org/spearce/jgit/transport/PacketLineIn.java | 12 +
.../spearce/jgit/transport/PostReceiveHook.java | 77 ++
.../org/spearce/jgit/transport/PreReceiveHook.java | 94 +++
.../org/spearce/jgit/transport/ReceiveCommand.java | 223 ++++++
.../org/spearce/jgit/transport/ReceivePack.java | 793 ++++++++++++++++++++
.../spearce/jgit/transport/TransportGitAnon.java | 3 +-
23 files changed, 2060 insertions(+), 93 deletions(-)
create mode 100644 org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Daemon.java
create mode 100644 org.spearce.jgit/src/org/spearce/jgit/transport/Daemon.java
create mode 100644 org.spearce.jgit/src/org/spearce/jgit/transport/DaemonClient.java
create mode 100644 org.spearce.jgit/src/org/spearce/jgit/transport/DaemonService.java
create mode 100644 org.spearce.jgit/src/org/spearce/jgit/transport/PostReceiveHook.java
create mode 100644 org.spearce.jgit/src/org/spearce/jgit/transport/PreReceiveHook.java
create mode 100644 org.spearce.jgit/src/org/spearce/jgit/transport/ReceiveCommand.java
create mode 100644 org.spearce.jgit/src/org/spearce/jgit/transport/ReceivePack.java
^ permalink raw reply [flat|nested] 17+ messages in thread* [JGIT PATCH 01/13] Fix invalid "double checked locking" in InflaterCache 2008-12-23 0:27 [JGIT PATCH 00/13] Add receive-pack server side support Shawn O. Pearce @ 2008-12-23 0:27 ` Shawn O. Pearce 2008-12-23 0:27 ` [JGIT PATCH 02/13] Cleanup stupid release of the cached Inflater in IndexPack Shawn O. Pearce 0 siblings, 1 reply; 17+ messages in thread From: Shawn O. Pearce @ 2008-12-23 0:27 UTC (permalink / raw) To: Robin Rosenberg; +Cc: git We move the Inflater allocation out of the synchronized block as it requires JNI code to build the libz structure. This permits other threads to get into the Inflater allocation faster. We also fix the release implementation to validate the array size only after we enter the lock, rather than before. This prevents us from releasing an inflater due to a stale copy of the open count being in our processor cache. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> --- .../src/org/spearce/jgit/lib/InflaterCache.java | 31 ++++++++++--------- 1 files changed, 16 insertions(+), 15 deletions(-) diff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/InflaterCache.java b/org.spearce.jgit/src/org/spearce/jgit/lib/InflaterCache.java index f0f9139..578911c 100644 --- a/org.spearce.jgit/src/org/spearce/jgit/lib/InflaterCache.java +++ b/org.spearce.jgit/src/org/spearce/jgit/lib/InflaterCache.java @@ -59,13 +59,18 @@ * * @return an available inflater. Never null. */ - public synchronized static Inflater get() { + public static Inflater get() { + final Inflater r = getImpl(); + return r != null ? r : new Inflater(false); + } + + private synchronized static Inflater getImpl() { if (openInflaterCount > 0) { final Inflater r = inflaterCache[--openInflaterCount]; inflaterCache[openInflaterCount] = null; return r; } - return new Inflater(false); + return null; } /** @@ -76,23 +81,19 @@ public synchronized static Inflater get() { * does nothing. */ public static void release(final Inflater i) { - if (i == null) - return; - - if (openInflaterCount == SZ) { - i.end(); - return; + if (i != null) { + i.reset(); + if (releaseImpl(i)) + i.end(); } - - i.reset(); - releaseImpl(i); } - private static synchronized void releaseImpl(final Inflater i) { - if (openInflaterCount == SZ) - i.end(); - else + private static synchronized boolean releaseImpl(final Inflater i) { + if (openInflaterCount < SZ) { inflaterCache[openInflaterCount++] = i; + return false; + } + return true; } private InflaterCache() { -- 1.6.1.rc4.301.g5497a ^ permalink raw reply related [flat|nested] 17+ messages in thread
* [JGIT PATCH 02/13] Cleanup stupid release of the cached Inflater in IndexPack 2008-12-23 0:27 ` [JGIT PATCH 01/13] Fix invalid "double checked locking" in InflaterCache Shawn O. Pearce @ 2008-12-23 0:27 ` Shawn O. Pearce 2008-12-23 0:27 ` [JGIT PATCH 03/13] Cache an Inflater inside a WindowCursor and reuse it as much as possible Shawn O. Pearce 0 siblings, 1 reply; 17+ messages in thread From: Shawn O. Pearce @ 2008-12-23 0:27 UTC (permalink / raw) To: Robin Rosenberg; +Cc: git The try/finally idiom avoids placing the Inflater on the stack, which means it goes unreachable slightly faster. Its also much less obtuse to read. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> --- .../src/org/spearce/jgit/transport/IndexPack.java | 8 +++++--- 1 files changed, 5 insertions(+), 3 deletions(-) diff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/IndexPack.java b/org.spearce.jgit/src/org/spearce/jgit/transport/IndexPack.java index 2752bb0..3e2187c 100644 --- a/org.spearce.jgit/src/org/spearce/jgit/transport/IndexPack.java +++ b/org.spearce.jgit/src/org/spearce/jgit/transport/IndexPack.java @@ -320,9 +320,11 @@ public void index(final ProgressMonitor progress) throws IOException { writeIdx(); } finally { - final Inflater inf = inflater; - inflater = null; - InflaterCache.release(inf); + try { + InflaterCache.release(inflater); + } finally { + inflater = null; + } progress.endTask(); if (packOut != null) -- 1.6.1.rc4.301.g5497a ^ permalink raw reply related [flat|nested] 17+ messages in thread
* [JGIT PATCH 03/13] Cache an Inflater inside a WindowCursor and reuse it as much as possible 2008-12-23 0:27 ` [JGIT PATCH 02/13] Cleanup stupid release of the cached Inflater in IndexPack Shawn O. Pearce @ 2008-12-23 0:27 ` Shawn O. Pearce 2008-12-23 0:27 ` [JGIT PATCH 04/13] Make RefDatabase thread-safe Shawn O. Pearce 0 siblings, 1 reply; 17+ messages in thread From: Shawn O. Pearce @ 2008-12-23 0:27 UTC (permalink / raw) To: Robin Rosenberg; +Cc: git By caching the Inflater within the WindowCursor we can improve performance associated with reading objects from the pack files. Each read can use the cached Inflater, especially when chasing down a delta chain. This avoids locking on the global cache. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> --- .../jgit/lib/DeltaRefPackedObjectLoader.java | 2 +- .../src/org/spearce/jgit/lib/PackFile.java | 7 +++- .../src/org/spearce/jgit/lib/Repository.java | 7 +++- .../src/org/spearce/jgit/lib/WindowCursor.java | 33 ++++++++++++++------ .../src/org/spearce/jgit/lib/WindowedFile.java | 10 +----- .../src/org/spearce/jgit/revwalk/RevWalk.java | 4 +- .../src/org/spearce/jgit/transport/IndexPack.java | 9 ++++- 7 files changed, 47 insertions(+), 25 deletions(-) diff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/DeltaRefPackedObjectLoader.java b/org.spearce.jgit/src/org/spearce/jgit/lib/DeltaRefPackedObjectLoader.java index 042d3a8..b126bbd 100644 --- a/org.spearce.jgit/src/org/spearce/jgit/lib/DeltaRefPackedObjectLoader.java +++ b/org.spearce.jgit/src/org/spearce/jgit/lib/DeltaRefPackedObjectLoader.java @@ -54,7 +54,7 @@ DeltaRefPackedObjectLoader(final WindowCursor curs, final PackFile pr, } protected PackedObjectLoader getBaseLoader() throws IOException { - final PackedObjectLoader or = pack.get(deltaBase); + final PackedObjectLoader or = pack.get(curs, deltaBase); if (or == null) throw new MissingObjectException(deltaBase, "delta base"); return or; diff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/PackFile.java b/org.spearce.jgit/src/org/spearce/jgit/lib/PackFile.java index 8ebd440..6cd85b1 100644 --- a/org.spearce.jgit/src/org/spearce/jgit/lib/PackFile.java +++ b/org.spearce.jgit/src/org/spearce/jgit/lib/PackFile.java @@ -120,7 +120,12 @@ public boolean hasObject(final AnyObjectId id) { * the pack file or the index could not be read. */ public PackedObjectLoader get(final AnyObjectId id) throws IOException { - return get(new WindowCursor(), id); + final WindowCursor wc = new WindowCursor(); + try { + return get(wc, id); + } finally { + wc.release(); + } } /** diff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/Repository.java b/org.spearce.jgit/src/org/spearce/jgit/lib/Repository.java index a319c00..ff36a3d 100644 --- a/org.spearce.jgit/src/org/spearce/jgit/lib/Repository.java +++ b/org.spearce.jgit/src/org/spearce/jgit/lib/Repository.java @@ -251,7 +251,12 @@ public boolean hasObject(final AnyObjectId objectId) { */ public ObjectLoader openObject(final AnyObjectId id) throws IOException { - return openObject(new WindowCursor(),id); + final WindowCursor wc = new WindowCursor(); + try { + return openObject(wc, id); + } finally { + wc.release(); + } } /** diff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/WindowCursor.java b/org.spearce.jgit/src/org/spearce/jgit/lib/WindowCursor.java index 0f4dab9..7aad081 100644 --- a/org.spearce.jgit/src/org/spearce/jgit/lib/WindowCursor.java +++ b/org.spearce.jgit/src/org/spearce/jgit/lib/WindowCursor.java @@ -46,6 +46,8 @@ /** Temporary buffer large enough for at least one raw object id. */ final byte[] tempId = new byte[Constants.OBJECT_ID_LENGTH]; + private Inflater inf; + ByteWindow window; Object handle; @@ -98,16 +100,8 @@ int copy(final WindowedFile provider, long position, final byte[] dstbuf, * data to. * @param dstoff * current offset within <code>dstbuf</code> to inflate into. - * @param inf - * the inflater to feed input to. The caller is responsible for - * initializing the inflater as multiple windows may need to - * supply data to the same inflater to completely decompress - * something. * @return updated <code>dstoff</code> based on the number of bytes - * successfully copied into <code>dstbuf</code> by - * <code>inf</code>. If the inflater is not yet finished then - * another window's data must still be supplied as input to finish - * decompression. + * successfully inflated into <code>dstbuf</code>. * @throws IOException * this cursor does not match the provider or id and the proper * window could not be acquired through the provider's cache. @@ -116,8 +110,12 @@ int copy(final WindowedFile provider, long position, final byte[] dstbuf, * stream corruption is likely. */ int inflate(final WindowedFile provider, long position, - final byte[] dstbuf, int dstoff, final Inflater inf) + final byte[] dstbuf, int dstoff) throws IOException, DataFormatException { + if (inf == null) + inf = InflaterCache.get(); + else + inf.reset(); for (;;) { pin(provider, position); dstoff = window.inflate(handle, position, dstbuf, dstoff, inf); @@ -138,5 +136,20 @@ private void pin(final WindowedFile provider, final long position) public void release() { window = null; handle = null; + try { + InflaterCache.release(inf); + } finally { + inf = null; + } + } + + /** + * @param curs cursor to release; may be null. + * @return always null. + */ + public static WindowCursor release(final WindowCursor curs) { + if (curs != null) + curs.release(); + return null; } } diff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/WindowedFile.java b/org.spearce.jgit/src/org/spearce/jgit/lib/WindowedFile.java index f28524f..454f98b 100644 --- a/org.spearce.jgit/src/org/spearce/jgit/lib/WindowedFile.java +++ b/org.spearce.jgit/src/org/spearce/jgit/lib/WindowedFile.java @@ -45,7 +45,6 @@ import java.nio.MappedByteBuffer; import java.nio.channels.FileChannel.MapMode; import java.util.zip.DataFormatException; -import java.util.zip.Inflater; /** * Read-only cached file access. @@ -240,13 +239,8 @@ public void copyToStream(long position, final byte[] buf, long cnt, void readCompressed(final long position, final byte[] dstbuf, final WindowCursor curs) throws IOException, DataFormatException { - final Inflater inf = InflaterCache.get(); - try { - if (curs.inflate(this, position, dstbuf, 0, inf) != dstbuf.length) - throw new EOFException("Short compressed stream at " + position); - } finally { - InflaterCache.release(inf); - } + if (curs.inflate(this, position, dstbuf, 0) != dstbuf.length) + throw new EOFException("Short compressed stream at " + position); } /** diff --git a/org.spearce.jgit/src/org/spearce/jgit/revwalk/RevWalk.java b/org.spearce.jgit/src/org/spearce/jgit/revwalk/RevWalk.java index d7e4c58..b1571ab 100644 --- a/org.spearce.jgit/src/org/spearce/jgit/revwalk/RevWalk.java +++ b/org.spearce.jgit/src/org/spearce/jgit/revwalk/RevWalk.java @@ -651,7 +651,7 @@ else if (!(c instanceof RevTree)) if ((t.flags & PARSED) != 0) return t; - final ObjectLoader ldr = db.openObject(t); + final ObjectLoader ldr = db.openObject(curs, t); if (ldr == null) throw new MissingObjectException(t, Constants.TYPE_TREE); if (ldr.getType() != Constants.OBJ_TREE) @@ -680,7 +680,7 @@ public RevObject parseAny(final AnyObjectId id) throws MissingObjectException, IOException { RevObject r = objects.get(id); if (r == null) { - final ObjectLoader ldr = db.openObject(id); + final ObjectLoader ldr = db.openObject(curs, id); if (ldr == null) throw new MissingObjectException(id.toObjectId(), "unknown"); final byte[] data = ldr.getCachedBytes(); diff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/IndexPack.java b/org.spearce.jgit/src/org/spearce/jgit/transport/IndexPack.java index 3e2187c..82cd615 100644 --- a/org.spearce.jgit/src/org/spearce/jgit/transport/IndexPack.java +++ b/org.spearce.jgit/src/org/spearce/jgit/transport/IndexPack.java @@ -68,6 +68,7 @@ import org.spearce.jgit.lib.PackIndexWriter; import org.spearce.jgit.lib.ProgressMonitor; import org.spearce.jgit.lib.Repository; +import org.spearce.jgit.lib.WindowCursor; import org.spearce.jgit.util.NB; /** Indexes Git pack files for local use. */ @@ -173,6 +174,8 @@ public static IndexPack create(final Repository db, final InputStream is) /** If {@link #fixThin} this is the last byte of the original checksum. */ private long originalEOF; + private WindowCursor readCurs; + /** * Create a new pack indexer utility. * @@ -189,6 +192,7 @@ public IndexPack(final Repository db, final InputStream src, repo = db; in = src; inflater = InflaterCache.get(); + readCurs = new WindowCursor(); buf = new byte[BUFFER_SIZE]; objectData = new byte[BUFFER_SIZE]; objectDigest = Constants.newMessageDigest(); @@ -325,6 +329,7 @@ public void index(final ProgressMonitor progress) throws IOException { } finally { inflater = null; } + readCurs = WindowCursor.release(readCurs); progress.endTask(); if (packOut != null) @@ -461,7 +466,7 @@ private void fixThinPack(final ProgressMonitor progress) throws IOException { final Deflater def = new Deflater(Deflater.DEFAULT_COMPRESSION, false); long end = originalEOF; for (final ObjectId baseId : new ArrayList<ObjectId>(baseById.keySet())) { - final ObjectLoader ldr = repo.openObject(baseId); + final ObjectLoader ldr = repo.openObject(readCurs, baseId); if (ldr == null) continue; final byte[] data = ldr.getBytes(); @@ -715,7 +720,7 @@ private void verifySafeObject(final AnyObjectId id, final int type, } } - final ObjectLoader ldr = repo.openObject(id); + final ObjectLoader ldr = repo.openObject(readCurs, id); if (ldr != null) { final byte[] existingData = ldr.getCachedBytes(); if (ldr.getType() != type || !Arrays.equals(data, existingData)) { -- 1.6.1.rc4.301.g5497a ^ permalink raw reply related [flat|nested] 17+ messages in thread
* [JGIT PATCH 04/13] Make RefDatabase thread-safe 2008-12-23 0:27 ` [JGIT PATCH 03/13] Cache an Inflater inside a WindowCursor and reuse it as much as possible Shawn O. Pearce @ 2008-12-23 0:27 ` Shawn O. Pearce 2008-12-23 0:27 ` [JGIT PATCH 05/13] Make PackFile thread-safe Shawn O. Pearce 0 siblings, 1 reply; 17+ messages in thread From: Shawn O. Pearce @ 2008-12-23 0:27 UTC (permalink / raw) To: Robin Rosenberg; +Cc: git This is necessary to support a thread-safe Repository. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> --- .../src/org/spearce/jgit/lib/RefDatabase.java | 29 +++++++++++--------- 1 files changed, 16 insertions(+), 13 deletions(-) diff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/RefDatabase.java b/org.spearce.jgit/src/org/spearce/jgit/lib/RefDatabase.java index 4cf6e08..87f26bf 100644 --- a/org.spearce.jgit/src/org/spearce/jgit/lib/RefDatabase.java +++ b/org.spearce.jgit/src/org/spearce/jgit/lib/RefDatabase.java @@ -93,7 +93,7 @@ RefDatabase(final Repository r) { clearCache(); } - void clearCache() { + synchronized void clearCache() { looseRefs = new HashMap<String, Ref>(); looseRefsMTime = new HashMap<String, Long>(); packedRefs = new HashMap<String, Ref>(); @@ -139,9 +139,11 @@ RefUpdate newUpdate(final String name) throws IOException { } void stored(final String origName, final String name, final ObjectId id, final long time) { - looseRefs.put(name, new Ref(Ref.Storage.LOOSE, origName, name, id)); - looseRefsMTime.put(name, time); - setModified(); + synchronized (this) { + looseRefs.put(name, new Ref(Ref.Storage.LOOSE, origName, name, id)); + looseRefsMTime.put(name, time); + setModified(); + } db.fireRefsMaybeChanged(); } @@ -157,11 +159,13 @@ void stored(final String origName, final String name, final ObjectId id, final l void link(final String name, final String target) throws IOException { final byte[] content = Constants.encode("ref: " + target + "\n"); lockAndWriteFile(fileForRef(name), content); - setModified(); + synchronized (this) { + setModified(); + } db.fireRefsMaybeChanged(); } - void setModified() { + private void setModified() { lastRefModification = refModificationCounter++; } @@ -210,7 +214,7 @@ Ref readRef(final String partialName) throws IOException { return avail; } - private void readPackedRefs(final Map<String, Ref> avail) { + private synchronized void readPackedRefs(final Map<String, Ref> avail) { refreshPackedRefs(); avail.putAll(packedRefs); } @@ -229,7 +233,7 @@ private void readLooseRefs(final Map<String, Ref> avail, } } - private void readOneLooseRef(final Map<String, Ref> avail, + private synchronized void readOneLooseRef(final Map<String, Ref> avail, final String origName, final String refName, final File ent) { // Unchanged and cached? Don't read it again. // @@ -323,8 +327,8 @@ private Ref readRefBasic(final String name, final int depth) throws IOException return readRefBasic(name, name, depth); } - private Ref readRefBasic(final String origName, final String name, final int depth) - throws IOException { + private synchronized Ref readRefBasic(final String origName, + final String name, final int depth) throws IOException { // Prefer loose ref to packed ref as the loose // file can be more up-to-date than a packed one. // @@ -408,7 +412,7 @@ private Ref readRefBasic(final String origName, final String name, final int dep return ref; } - private void refreshPackedRefs() { + private synchronized void refreshPackedRefs() { final long currTime = packedRefsFile.lastModified(); final long currLen = currTime == 0 ? 0 : packedRefsFile.length(); if (currTime == packedRefsLastModified && currLen == packedRefsLength) @@ -479,7 +483,7 @@ private void lockAndWriteFile(File file, byte[] content) throws IOException { throw new ObjectWritingException("Unable to write " + name); } - void removePackedRef(String name) throws IOException { + synchronized void removePackedRef(String name) throws IOException { packedRefs.remove(name); writePackedRefs(); } @@ -508,5 +512,4 @@ private static BufferedReader openReader(final File fileLocation) return new BufferedReader(new InputStreamReader(new FileInputStream( fileLocation), Constants.CHARSET)); } - } -- 1.6.1.rc4.301.g5497a ^ permalink raw reply related [flat|nested] 17+ messages in thread
* [JGIT PATCH 05/13] Make PackFile thread-safe 2008-12-23 0:27 ` [JGIT PATCH 04/13] Make RefDatabase thread-safe Shawn O. Pearce @ 2008-12-23 0:27 ` Shawn O. Pearce 2008-12-23 0:27 ` [JGIT PATCH 06/13] Make Repository thread-safe Shawn O. Pearce 0 siblings, 1 reply; 17+ messages in thread From: Shawn O. Pearce @ 2008-12-23 0:27 UTC (permalink / raw) To: Robin Rosenberg; +Cc: git We really should try to avoid allocating the reverse index within the pack object itself, and instead make the applications that do use this data manage it themselves. This way the index could be released in memory when its no longer used, and the locking could be bypassed entirely. Since most of PackFile is dependent upon the WindowedFile we can mostly ignore synchronization in this class. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> --- .../src/org/spearce/jgit/lib/PackFile.java | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/PackFile.java b/org.spearce.jgit/src/org/spearce/jgit/lib/PackFile.java index 6cd85b1..ca5681b 100644 --- a/org.spearce.jgit/src/org/spearce/jgit/lib/PackFile.java +++ b/org.spearce.jgit/src/org/spearce/jgit/lib/PackFile.java @@ -341,7 +341,7 @@ private long findEndOffset(final long startOffset) return getReverseIdx().findNextOffset(startOffset, maxOffset); } - private PackReverseIndex getReverseIdx() { + private synchronized PackReverseIndex getReverseIdx() { if (reverseIdx == null) reverseIdx = new PackReverseIndex(idx); return reverseIdx; -- 1.6.1.rc4.301.g5497a ^ permalink raw reply related [flat|nested] 17+ messages in thread
* [JGIT PATCH 06/13] Make Repository thread-safe 2008-12-23 0:27 ` [JGIT PATCH 05/13] Make PackFile thread-safe Shawn O. Pearce @ 2008-12-23 0:27 ` Shawn O. Pearce 2008-12-23 0:27 ` [JGIT PATCH 07/13] Don't open a PackFile multiple times on scanForPacks Shawn O. Pearce 0 siblings, 1 reply; 17+ messages in thread From: Shawn O. Pearce @ 2008-12-23 0:27 UTC (permalink / raw) To: Robin Rosenberg; +Cc: git The Repository class itself is now thread-safe. Thread safety is ensured by copying values onto the caller stack prior to use. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> --- .../src/org/spearce/jgit/lib/Repository.java | 75 ++++++++++++------- 1 files changed, 47 insertions(+), 28 deletions(-) diff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/Repository.java b/org.spearce.jgit/src/org/spearce/jgit/lib/Repository.java index ff36a3d..89ad991 100644 --- a/org.spearce.jgit/src/org/spearce/jgit/lib/Repository.java +++ b/org.spearce.jgit/src/org/spearce/jgit/lib/Repository.java @@ -46,6 +46,7 @@ import java.io.FilenameFilter; import java.io.IOException; import java.util.ArrayList; +import java.util.Arrays; import java.util.Collection; import java.util.Collections; import java.util.HashMap; @@ -79,20 +80,22 @@ * </ul> * </li> * </ul> - * + * <p> + * This class is thread-safe. + * <p> * This implementation only handles a subtly undocumented subset of git features. * */ public class Repository { private final File gitDir; - private final File[] objectsDirs; - private final RepositoryConfig config; private final RefDatabase refs; - private PackFile[] packs; + private File[] objectDirectoryList; + + private PackFile[] packFileList; private GitIndex index; @@ -111,18 +114,19 @@ public Repository(final File d) throws IOException { gitDir = d.getAbsoluteFile(); try { - objectsDirs = readObjectsDirs(FS.resolve(gitDir, "objects"), - new ArrayList<File>()).toArray(new File[0]); + objectDirectoryList = readObjectsDirs( + FS.resolve(gitDir, "objects"), new ArrayList<File>()) + .toArray(new File[0]); } catch (IOException e) { IOException ex = new IOException("Cannot find all object dirs for " + gitDir); ex.initCause(e); throw ex; } refs = new RefDatabase(this); - packs = new PackFile[0]; + packFileList = new PackFile[0]; config = new RepositoryConfig(this); - final boolean isExisting = objectsDirs[0].exists(); + final boolean isExisting = objectDirectoryList[0].exists(); if (isExisting) { getConfig().load(); final String repositoryFormatVersion = getConfig().getString( @@ -138,7 +142,8 @@ public Repository(final File d) throws IOException { scanForPacks(); } - private Collection<File> readObjectsDirs(File objectsDir, Collection<File> ret) throws IOException { + private static Collection<File> readObjectsDirs(File objectsDir, + Collection<File> ret) throws IOException { ret.add(objectsDir); final File altFile = FS.resolve(objectsDir, "info/alternates"); if (altFile.exists()) { @@ -160,7 +165,7 @@ public Repository(final File d) throws IOException { * * @throws IOException */ - public void create() throws IOException { + public synchronized void create() throws IOException { if (gitDir.exists()) { throw new IllegalStateException("Repository already exists: " + gitDir); @@ -169,9 +174,9 @@ public void create() throws IOException { gitDir.mkdirs(); refs.create(); - objectsDirs[0].mkdirs(); - new File(objectsDirs[0], "pack").mkdir(); - new File(objectsDirs[0], "info").mkdir(); + objectDirectoryList[0].mkdirs(); + new File(objectDirectoryList[0], "pack").mkdir(); + new File(objectDirectoryList[0], "info").mkdir(); new File(gitDir, "branches").mkdir(); new File(gitDir, "remotes").mkdir(); @@ -182,6 +187,14 @@ public void create() throws IOException { getConfig().save(); } + private synchronized File[] objectsDirs(){ + return objectDirectoryList; + } + + private synchronized PackFile[] packs(){ + return packFileList; + } + /** * @return GIT_DIR */ @@ -193,7 +206,7 @@ public File getDirectory() { * @return the directory containing the objects owned by this repository. */ public File getObjectsDirectory() { - return objectsDirs[0]; + return objectsDirs()[0]; } /** @@ -217,6 +230,7 @@ public File toFile(final AnyObjectId objectId) { final String n = objectId.name(); String d=n.substring(0, 2); String f=n.substring(2); + final File[] objectsDirs = objectsDirs(); for (int i=0; i<objectsDirs.length; ++i) { File ret = new File(new File(objectsDirs[i], d), f); if (ret.exists()) @@ -231,6 +245,7 @@ public File toFile(final AnyObjectId objectId) { * known shared repositories. */ public boolean hasObject(final AnyObjectId objectId) { + final PackFile[] packs = packs(); int k = packs.length; if (k > 0) { do { @@ -271,6 +286,7 @@ public ObjectLoader openObject(final AnyObjectId id) */ public ObjectLoader openObject(final WindowCursor curs, final AnyObjectId id) throws IOException { + final PackFile[] packs = packs(); int k = packs.length; if (k > 0) { do { @@ -341,7 +357,7 @@ public ObjectLoader openObject(final WindowCursor curs, final AnyObjectId id) void openObjectInAllPacks(final AnyObjectId objectId, final Collection<PackedObjectLoader> resultLoaders, final WindowCursor curs) throws IOException { - for (PackFile pack : packs) { + for (PackFile pack : packs()) { final PackedObjectLoader loader = pack.get(curs, objectId); if (loader != null) resultLoaders.add(loader); @@ -766,11 +782,10 @@ public void close() { closePacks(); } - void closePacks() { - for (int k = packs.length - 1; k >= 0; k--) { - packs[k].close(); - } - packs = new PackFile[0]; + synchronized void closePacks() { + for (int k = packFileList.length - 1; k >= 0; k--) + packFileList[k].close(); + packFileList = new PackFile[0]; } /** @@ -795,11 +810,13 @@ public void openPack(final File pack, final File idx) throws IOException { throw new IllegalArgumentException("Pack " + pack + "does not match index " + idx); - final PackFile[] cur = packs; - final PackFile[] arr = new PackFile[cur.length + 1]; - System.arraycopy(cur, 0, arr, 1, cur.length); - arr[0] = new PackFile(this, idx, pack); - packs = arr; + synchronized (this) { + final PackFile[] cur = packFileList; + final PackFile[] arr = new PackFile[cur.length + 1]; + System.arraycopy(cur, 0, arr, 1, cur.length); + arr[0] = new PackFile(this, idx, pack); + packFileList = arr; + } } /** @@ -808,11 +825,13 @@ public void openPack(final File pack, final File idx) throws IOException { */ public void scanForPacks() { final ArrayList<PackFile> p = new ArrayList<PackFile>(); - for (int i=0; i<objectsDirs.length; ++i) - scanForPacks(new File(objectsDirs[i], "pack"), p); + for (final File d : objectsDirs()) + scanForPacks(new File(d, "pack"), p); final PackFile[] arr = new PackFile[p.size()]; p.toArray(arr); - packs = arr; + synchronized (this) { + packFileList = arr; + } } private void scanForPacks(final File packDir, Collection<PackFile> packList) { -- 1.6.1.rc4.301.g5497a ^ permalink raw reply related [flat|nested] 17+ messages in thread
* [JGIT PATCH 07/13] Don't open a PackFile multiple times on scanForPacks 2008-12-23 0:27 ` [JGIT PATCH 06/13] Make Repository thread-safe Shawn O. Pearce @ 2008-12-23 0:27 ` Shawn O. Pearce 2008-12-23 0:27 ` [JGIT PATCH 08/13] Expose RepositoryConfig.getBoolean so applications can use it Shawn O. Pearce 0 siblings, 1 reply; 17+ messages in thread From: Shawn O. Pearce @ 2008-12-23 0:27 UTC (permalink / raw) To: Robin Rosenberg; +Cc: git If we already have a given PackFile open in this repository then we should reuse the existing PackFile handle. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> --- .../src/org/spearce/jgit/lib/PackFile.java | 5 +++++ .../src/org/spearce/jgit/lib/Repository.java | 8 +++++++- .../src/org/spearce/jgit/lib/WindowedFile.java | 7 ++++++- 3 files changed, 18 insertions(+), 2 deletions(-) diff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/PackFile.java b/org.spearce.jgit/src/org/spearce/jgit/lib/PackFile.java index ca5681b..fc1b6ea 100644 --- a/org.spearce.jgit/src/org/spearce/jgit/lib/PackFile.java +++ b/org.spearce.jgit/src/org/spearce/jgit/lib/PackFile.java @@ -94,6 +94,11 @@ final PackedObjectLoader resolveBase(final WindowCursor curs, final long ofs) return reader(curs, ofs); } + /** @return the File object which locates this pack on disk. */ + public File getPackFile() { + return pack.getFile(); + } + /** * Determine if an object is contained within the pack file. * <p> diff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/Repository.java b/org.spearce.jgit/src/org/spearce/jgit/lib/Repository.java index 89ad991..8f6e96e 100644 --- a/org.spearce.jgit/src/org/spearce/jgit/lib/Repository.java +++ b/org.spearce.jgit/src/org/spearce/jgit/lib/Repository.java @@ -825,6 +825,7 @@ synchronized (this) { */ public void scanForPacks() { final ArrayList<PackFile> p = new ArrayList<PackFile>(); + p.addAll(Arrays.asList(packs())); for (final File d : objectsDirs()) scanForPacks(new File(d, "pack"), p); final PackFile[] arr = new PackFile[p.size()]; @@ -843,7 +844,7 @@ public boolean accept(final File baseDir, final String n) { } }); if (idxList != null) { - for (final String indexName : idxList) { + SCAN: for (final String indexName : idxList) { final String n = indexName.substring(0, indexName.length() - 4); final File idxFile = new File(packDir, n + ".idx"); final File packFile = new File(packDir, n + ".pack"); @@ -856,6 +857,11 @@ public boolean accept(final File baseDir, final String n) { continue; } + for (final PackFile p : packList) { + if (packFile.equals(p.getPackFile())) + continue SCAN; + } + try { packList.add(new PackFile(this, idxFile, packFile)); } catch (IOException ioe) { diff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/WindowedFile.java b/org.spearce.jgit/src/org/spearce/jgit/lib/WindowedFile.java index 454f98b..f640c42 100644 --- a/org.spearce.jgit/src/org/spearce/jgit/lib/WindowedFile.java +++ b/org.spearce.jgit/src/org/spearce/jgit/lib/WindowedFile.java @@ -101,13 +101,18 @@ public long length() { return length; } + /** @return the absolute file object this file reads from. */ + public File getFile() { + return fPath.getAbsoluteFile(); + } + /** * Get the path name of this file. * * @return the absolute path name of the file. */ public String getName() { - return fPath.getAbsolutePath(); + return getFile().getPath(); } /** -- 1.6.1.rc4.301.g5497a ^ permalink raw reply related [flat|nested] 17+ messages in thread
* [JGIT PATCH 08/13] Expose RepositoryConfig.getBoolean so applications can use it 2008-12-23 0:27 ` [JGIT PATCH 07/13] Don't open a PackFile multiple times on scanForPacks Shawn O. Pearce @ 2008-12-23 0:27 ` Shawn O. Pearce 2008-12-23 0:27 ` [JGIT PATCH 09/13] Add AnyObjectId.copyTo(StringBuilder) Shawn O. Pearce 0 siblings, 1 reply; 17+ messages in thread From: Shawn O. Pearce @ 2008-12-23 0:27 UTC (permalink / raw) To: Robin Rosenberg; +Cc: git There's no reason for these to be hidden from application level code. We expose getInt because it is a popular type; getBoolean should also be exposed because it is also a popular type. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> --- .../src/org/spearce/jgit/lib/RepositoryConfig.java | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/RepositoryConfig.java b/org.spearce.jgit/src/org/spearce/jgit/lib/RepositoryConfig.java index 3291bba..7df90cd 100644 --- a/org.spearce.jgit/src/org/spearce/jgit/lib/RepositoryConfig.java +++ b/org.spearce.jgit/src/org/spearce/jgit/lib/RepositoryConfig.java @@ -210,7 +210,7 @@ public int getInt(final String section, String subsection, * @return true if any value or defaultValue is true, false for missing or * explicit false */ - protected boolean getBoolean(final String section, final String name, + public boolean getBoolean(final String section, final String name, final boolean defaultValue) { return getBoolean(section, null, name, defaultValue); } @@ -229,7 +229,7 @@ protected boolean getBoolean(final String section, final String name, * @return true if any value or defaultValue is true, false for missing or * explicit false */ - protected boolean getBoolean(final String section, String subsection, + public boolean getBoolean(final String section, String subsection, final String name, final boolean defaultValue) { String n = getRawString(section, subsection, name); if (n == null) -- 1.6.1.rc4.301.g5497a ^ permalink raw reply related [flat|nested] 17+ messages in thread
* [JGIT PATCH 09/13] Add AnyObjectId.copyTo(StringBuilder) 2008-12-23 0:27 ` [JGIT PATCH 08/13] Expose RepositoryConfig.getBoolean so applications can use it Shawn O. Pearce @ 2008-12-23 0:27 ` Shawn O. Pearce 2008-12-23 0:27 ` [JGIT PATCH 10/13] Add compare-and-swap semantics to RefUpdate Shawn O. Pearce 0 siblings, 1 reply; 17+ messages in thread From: Shawn O. Pearce @ 2008-12-23 0:27 UTC (permalink / raw) To: Robin Rosenberg; +Cc: git This makes it easier to append an ObjectId onto a StringBuilder. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> --- .../src/org/spearce/jgit/lib/AnyObjectId.java | 17 ++++++++++++++++- 1 files changed, 16 insertions(+), 1 deletions(-) diff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/AnyObjectId.java b/org.spearce.jgit/src/org/spearce/jgit/lib/AnyObjectId.java index 943d916..3ec00ea 100644 --- a/org.spearce.jgit/src/org/spearce/jgit/lib/AnyObjectId.java +++ b/org.spearce.jgit/src/org/spearce/jgit/lib/AnyObjectId.java @@ -387,7 +387,22 @@ public void copyTo(final Writer w) throws IOException { */ public void copyTo(final char[] tmp, final Writer w) throws IOException { toHexCharArray(tmp); - w.write(tmp); + w.write(tmp, 0, STR_LEN); + } + + /** + * Copy this ObjectId to a StringBuilder in hex format. + * + * @param tmp + * temporary char array to buffer construct into before writing. + * Must be at least large enough to hold 2 digits for each byte + * of object id (40 characters or larger). + * @param w + * the string to append onto. + */ + public void copyTo(final char[] tmp, final StringBuilder w) { + toHexCharArray(tmp); + w.append(tmp, 0, STR_LEN); } private char[] toHexCharArray() { -- 1.6.1.rc4.301.g5497a ^ permalink raw reply related [flat|nested] 17+ messages in thread
* [JGIT PATCH 10/13] Add compare-and-swap semantics to RefUpdate 2008-12-23 0:27 ` [JGIT PATCH 09/13] Add AnyObjectId.copyTo(StringBuilder) Shawn O. Pearce @ 2008-12-23 0:27 ` Shawn O. Pearce 2008-12-23 0:27 ` [JGIT PATCH 11/13] Allow null new ObjectId during RefUpdate.delete Shawn O. Pearce 0 siblings, 1 reply; 17+ messages in thread From: Shawn O. Pearce @ 2008-12-23 0:27 UTC (permalink / raw) To: Robin Rosenberg; +Cc: git This permits the caller to set the value it expects to find in the Ref upon obtaining the update lock. If the ref value doesn't match then the update is aborted with Result.LOCK_FAILURE, so the caller can recover gracefully without making unexpected changes. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> --- .../src/org/spearce/jgit/lib/RefUpdate.java | 30 ++++++++++++++++++++ 1 files changed, 30 insertions(+), 0 deletions(-) diff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/RefUpdate.java b/org.spearce.jgit/src/org/spearce/jgit/lib/RefUpdate.java index 235c2fd..0c9ce91 100644 --- a/org.spearce.jgit/src/org/spearce/jgit/lib/RefUpdate.java +++ b/org.spearce.jgit/src/org/spearce/jgit/lib/RefUpdate.java @@ -149,6 +149,9 @@ /** Old value of the ref, obtained after we lock it. */ private ObjectId oldValue; + /** If non-null, the value {@link #oldValue} must have to continue. */ + private ObjectId expValue; + /** Result of the update operation. */ private Result result = Result.NOT_ATTEMPTED; @@ -190,6 +193,27 @@ public void setNewObjectId(final AnyObjectId id) { } /** + * @return the expected value of the ref after the lock is taken, but before + * update occurs. Null to avoid the compare and swap test. Use + * {@link ObjectId#zeroId()} to indicate expectation of a + * non-existant ref. + */ + public ObjectId getExpectedOldObjectId() { + return expValue; + } + + /** + * @param id + * the expected value of the ref after the lock is taken, but + * before update occurs. Null to avoid the compare and swap test. + * Use {@link ObjectId#zeroId()} to indicate expectation of a + * non-existant ref. + */ + public void setExpectedOldObjectId(final AnyObjectId id) { + expValue = id != null ? id.toObjectId() : null; + } + + /** * Check if this update wants to forcefully change the ref. * * @return true if this update should ignore merge tests. @@ -370,6 +394,12 @@ private Result updateImpl(final RevWalk walk, final Store store) return Result.LOCK_FAILURE; try { oldValue = db.idOf(getName()); + if (expValue != null) { + final ObjectId o; + o = oldValue != null ? oldValue : ObjectId.zeroId(); + if (!expValue.equals(o)) + return Result.LOCK_FAILURE; + } if (oldValue == null) return store.store(lock, Result.NEW); -- 1.6.1.rc4.301.g5497a ^ permalink raw reply related [flat|nested] 17+ messages in thread
* [JGIT PATCH 11/13] Allow null new ObjectId during RefUpdate.delete 2008-12-23 0:27 ` [JGIT PATCH 10/13] Add compare-and-swap semantics to RefUpdate Shawn O. Pearce @ 2008-12-23 0:27 ` Shawn O. Pearce 2008-12-23 0:27 ` [JGIT PATCH 12/13] Implement the git-receive-pack process in Java Shawn O. Pearce 0 siblings, 1 reply; 17+ messages in thread From: Shawn O. Pearce @ 2008-12-23 0:27 UTC (permalink / raw) To: Robin Rosenberg; +Cc: git If we are deleting a ref we really don't care about what the new ObjectId value should be; it just doesn't matter. If we didn't set the value we should consider it the same as if we had a MissingObjectException, which means treat the value as null and perform a force update test. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> --- .../src/org/spearce/jgit/lib/RefUpdate.java | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/RefUpdate.java b/org.spearce.jgit/src/org/spearce/jgit/lib/RefUpdate.java index 0c9ce91..1417f2c 100644 --- a/org.spearce.jgit/src/org/spearce/jgit/lib/RefUpdate.java +++ b/org.spearce.jgit/src/org/spearce/jgit/lib/RefUpdate.java @@ -424,7 +424,7 @@ private Result updateImpl(final RevWalk walk, final Store store) private static RevObject safeParse(final RevWalk rw, final AnyObjectId id) throws IOException { try { - return rw.parseAny(id); + return id != null ? rw.parseAny(id) : null; } catch (MissingObjectException e) { // We can expect some objects to be missing, like if we are // trying to force a deletion of a branch and the object it -- 1.6.1.rc4.301.g5497a ^ permalink raw reply related [flat|nested] 17+ messages in thread
* [JGIT PATCH 12/13] Implement the git-receive-pack process in Java 2008-12-23 0:27 ` [JGIT PATCH 11/13] Allow null new ObjectId during RefUpdate.delete Shawn O. Pearce @ 2008-12-23 0:27 ` Shawn O. Pearce 2008-12-23 0:27 ` [JGIT PATCH 13/13] Add basic git daemon support to publish receive-pack Shawn O. Pearce 0 siblings, 1 reply; 17+ messages in thread From: Shawn O. Pearce @ 2008-12-23 0:27 UTC (permalink / raw) To: Robin Rosenberg; +Cc: git This implementation provides a basic git-receive-pack service within Java. Two hooks APIs are supported before and after commands are executed within the connection, allowing daemons to customize the behavior of the updates. Logic to bind the ReceivePack class to a pipe or network socket is omitted, as it depends on the transport. SSH servers will need a pure Java SSH implementation such as Apache MINA SSHD. Anonymous push over git:// requires a basic git-daemon functionality. Local pipe access might use pure-Java pipes, or System.in/System.out. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> --- .../org/spearce/jgit/transport/PacketLineIn.java | 12 + .../spearce/jgit/transport/PostReceiveHook.java | 77 ++ .../org/spearce/jgit/transport/PreReceiveHook.java | 94 +++ .../org/spearce/jgit/transport/ReceiveCommand.java | 223 ++++++ .../org/spearce/jgit/transport/ReceivePack.java | 793 ++++++++++++++++++++ 5 files changed, 1199 insertions(+), 0 deletions(-) create mode 100644 org.spearce.jgit/src/org/spearce/jgit/transport/PostReceiveHook.java create mode 100644 org.spearce.jgit/src/org/spearce/jgit/transport/PreReceiveHook.java create mode 100644 org.spearce.jgit/src/org/spearce/jgit/transport/ReceiveCommand.java create mode 100644 org.spearce.jgit/src/org/spearce/jgit/transport/ReceivePack.java diff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/PacketLineIn.java b/org.spearce.jgit/src/org/spearce/jgit/transport/PacketLineIn.java index f87517d..ef218be 100644 --- a/org.spearce.jgit/src/org/spearce/jgit/transport/PacketLineIn.java +++ b/org.spearce.jgit/src/org/spearce/jgit/transport/PacketLineIn.java @@ -111,6 +111,18 @@ String readString() throws IOException { return RawParseUtils.decode(Constants.CHARSET, raw, 0, len); } + String readStringNoLF() throws IOException { + int len = readLength(); + if (len == 0) + return ""; + + len -= 4; // length header (4 bytes) + + final byte[] raw = new byte[len]; + NB.readFully(in, raw, 0, len); + return RawParseUtils.decode(Constants.CHARSET, raw, 0, len); + } + private void readLF() throws IOException { if (in.read() != '\n') throw new IOException("Protocol error: expected LF"); diff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/PostReceiveHook.java b/org.spearce.jgit/src/org/spearce/jgit/transport/PostReceiveHook.java new file mode 100644 index 0000000..060efb3 --- /dev/null +++ b/org.spearce.jgit/src/org/spearce/jgit/transport/PostReceiveHook.java @@ -0,0 +1,77 @@ +/* + * Copyright (C) 2008, Google Inc. + * + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials provided + * with the distribution. + * + * - Neither the name of the Git Development Community nor the + * names of its contributors may be used to endorse or promote + * products derived from this software without specific prior + * written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND + * CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, + * INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES + * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR + * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT + * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; + * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER + * CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, + * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF + * ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +package org.spearce.jgit.transport; + +import java.util.Collection; + +/** + * Hook invoked by {@link ReceivePack} after all updates are executed. + * <p> + * The hook is called after all commands have been processed. Only commands with + * a status of {@link ReceiveCommand.Result#OK} are passed into the hook. To get + * all commands within the hook, see {@link ReceivePack#getAllCommands()}. + * <p> + * Any post-receive hook implementation should not update the status of a + * command, as the command has already completed or failed, and the status has + * already been returned to the client. + * <p> + * Hooks should execute quickly, as they block the server and the client from + * completing the connection. + */ +public interface PostReceiveHook { + /** A simple no-op hook. */ + public static final PostReceiveHook NULL = new PostReceiveHook() { + public void onPostReceive(final ReceivePack rp, + final Collection<ReceiveCommand> commands) { + // Do nothing. + } + }; + + /** + * Invoked after all commands are executed and status has been returned. + * + * @param rp + * the process handling the current receive. Hooks may obtain + * details about the destination repository through this handle. + * @param commands + * unmodifiable set of successfully completed commands. May be + * the empty set. + */ + public void onPostReceive(ReceivePack rp, + Collection<ReceiveCommand> commands); +} diff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/PreReceiveHook.java b/org.spearce.jgit/src/org/spearce/jgit/transport/PreReceiveHook.java new file mode 100644 index 0000000..0af466d --- /dev/null +++ b/org.spearce.jgit/src/org/spearce/jgit/transport/PreReceiveHook.java @@ -0,0 +1,94 @@ +/* + * Copyright (C) 2008, Google Inc. + * + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials provided + * with the distribution. + * + * - Neither the name of the Git Development Community nor the + * names of its contributors may be used to endorse or promote + * products derived from this software without specific prior + * written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND + * CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, + * INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES + * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR + * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT + * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; + * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER + * CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, + * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF + * ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +package org.spearce.jgit.transport; + +import java.util.Collection; + +/** + * Hook invoked by {@link ReceivePack} before any updates are executed. + * <p> + * The hook is called with any commands that are deemed valid after parsing them + * from the client and applying the standard receive configuration options to + * them: + * <ul> + * <li><code>receive.denyDenyDeletes</code></li> + * <li><code>receive.denyNonFastForwards</code></li> + * </ul> + * This means the hook will not receive a non-fast-forward update command if + * denyNonFastForwards is set to true in the configuration file. To get all + * commands within the hook, see {@link ReceivePack#getAllCommands()}. + * <p> + * As the hook is invoked prior to the commands being executed, the hook may + * choose to block any command by setting its result status with + * {@link ReceiveCommand#setResult(ReceiveCommand.Result)}. + * <p> + * The hook may also choose to perform the command itself (or merely pretend + * that it has performed the command), by setting the result status to + * {@link ReceiveCommand.Result#OK}. + * <p> + * Hooks should run quickly, as they block the caller thread and the client + * process from completing. + * <p> + * Hooks may send optional messages back to the client via methods on + * {@link ReceivePack}. Implementors should be aware that not all network + * transports support this output, so some (or all) messages may simply be + * discarded. These messages should be advisory only. + */ +public interface PreReceiveHook { + /** A simple no-op hook. */ + public static final PreReceiveHook NULL = new PreReceiveHook() { + public void onPreReceive(final ReceivePack rp, + final Collection<ReceiveCommand> commands) { + // Do nothing. + } + }; + + /** + * Invoked just before commands are executed. + * <p> + * See the class description for how this method can impact execution. + * + * @param rp + * the process handling the current receive. Hooks may obtain + * details about the destination repository through this handle. + * @param commands + * unmodifiable set of valid commands still pending execution. + * May be the empty set. + */ + public void onPreReceive(ReceivePack rp, Collection<ReceiveCommand> commands); +} diff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/ReceiveCommand.java b/org.spearce.jgit/src/org/spearce/jgit/transport/ReceiveCommand.java new file mode 100644 index 0000000..b07b976 --- /dev/null +++ b/org.spearce.jgit/src/org/spearce/jgit/transport/ReceiveCommand.java @@ -0,0 +1,223 @@ +/* + * Copyright (C) 2008, Google Inc. + * + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials provided + * with the distribution. + * + * - Neither the name of the Git Development Community nor the + * names of its contributors may be used to endorse or promote + * products derived from this software without specific prior + * written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND + * CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, + * INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES + * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR + * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT + * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; + * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER + * CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, + * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF + * ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +package org.spearce.jgit.transport; + +import org.spearce.jgit.lib.ObjectId; +import org.spearce.jgit.lib.Ref; + +/** + * A command being processed by {@link ReceivePack}. + * <p> + * This command instance roughly translates to the server side representation of + * the {@link RemoteRefUpdate} created by the client. + */ +public class ReceiveCommand { + /** Type of operation requested. */ + public static enum Type { + /** Create a new ref; the ref must not already exist. */ + CREATE, + + /** + * Update an existing ref with a fast-forward update. + * <p> + * During a fast-forward update no changes will be lost; only new + * commits are inserted into the ref. + */ + UPDATE, + + /** + * Update an existing ref by potentially discarding objects. + * <p> + * The current value of the ref is not fully reachable from the new + * value of the ref, so a successful command may result in one or more + * objects becoming unreachable. + */ + UPDATE_NONFASTFORWARD, + + /** Delete an existing ref; the ref should already exist. */ + DELETE; + } + + /** Result of the update command. */ + public static enum Result { + /** The command has not yet been attempted by the server. */ + NOT_ATTEMPTED, + + /** The server is configured to deny creation of this ref. */ + REJECTED_NOCREATE, + + /** The server is configured to deny deletion of this ref. */ + REJECTED_NODELETE, + + /** The update is a non-fast-forward update and isn't permitted. */ + REJECTED_NONFASTFORWARD, + + /** The update affects <code>HEAD</code> and cannot be permitted. */ + REJECTED_CURRENT_BRANCH, + + /** + * One or more objects aren't in the repository. + * <p> + * This is severe indication of either repository corruption on the + * server side, or a bug in the client wherein the client did not supply + * all required objects during the pack transfer. + */ + REJECTED_MISSING_OBJECT, + + /** Other failure; see {@link ReceiveCommand#getMessage()}. */ + REJECTED_OTHER_REASON, + + /** The ref could not be locked and updated atomically; try again. */ + LOCK_FAILURE, + + /** The change was completed successfully. */ + OK; + } + + private final ObjectId oldId; + + private final ObjectId newId; + + private final String name; + + private Type type; + + private Ref ref; + + private Result status; + + private String message; + + /** + * Create a new command for {@link ReceivePack}. + * + * @param oldId + * the old object id; must not be null. Use + * {@link ObjectId#zeroId()} to indicate a ref creation. + * @param newId + * the new object id; must not be null. Use + * {@link ObjectId#zeroId()} to indicate a ref deletion. + * @param name + * name of the ref being affected. + */ + public ReceiveCommand(final ObjectId oldId, final ObjectId newId, + final String name) { + this.oldId = oldId; + this.newId = newId; + this.name = name; + + type = Type.UPDATE; + if (ObjectId.zeroId().equals(oldId)) + type = Type.CREATE; + if (ObjectId.zeroId().equals(newId)) + type = Type.DELETE; + status = Result.NOT_ATTEMPTED; + } + + /** @return the old value the client thinks the ref has. */ + public ObjectId getOldId() { + return oldId; + } + + /** @return the requested new value for this ref. */ + public ObjectId getNewId() { + return newId; + } + + /** @return the name of the ref being updated. */ + public String getRefName() { + return name; + } + + /** @return the type of this command; see {@link Type}. */ + public Type getType() { + return type; + } + + /** @return the ref, if this was advertised by the connection. */ + public Ref getRef() { + return ref; + } + + /** @return the current status code of this command. */ + public Result getResult() { + return status; + } + + /** @return the message associated with a failure status. */ + public String getMessage() { + return message; + } + + /** + * Set the status of this command. + * + * @param s + * the new status code for this command. + */ + public void setResult(final Result s) { + setResult(s, null); + } + + /** + * Set the status of this command. + * + * @param s + * new status code for this command. + * @param m + * optional message explaining the new status. + */ + public void setResult(final Result s, final String m) { + status = s; + message = m; + } + + void setRef(final Ref r) { + ref = r; + } + + void setType(final Type t) { + type = t; + } + + @Override + public String toString() { + return getType().name() + ": " + getOldId().name() + " " + + getNewId().name() + " " + getRefName(); + } +} diff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/ReceivePack.java b/org.spearce.jgit/src/org/spearce/jgit/transport/ReceivePack.java new file mode 100644 index 0000000..95519b8 --- /dev/null +++ b/org.spearce.jgit/src/org/spearce/jgit/transport/ReceivePack.java @@ -0,0 +1,793 @@ +/* + * Copyright (C) 2008, Google Inc. + * + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials provided + * with the distribution. + * + * - Neither the name of the Git Development Community nor the + * names of its contributors may be used to endorse or promote + * products derived from this software without specific prior + * written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND + * CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, + * INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES + * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR + * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT + * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; + * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER + * CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, + * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF + * ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +package org.spearce.jgit.transport; + +import java.io.BufferedWriter; +import java.io.EOFException; +import java.io.IOException; +import java.io.InputStream; +import java.io.OutputStream; +import java.io.OutputStreamWriter; +import java.io.PrintWriter; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashSet; +import java.util.Iterator; +import java.util.List; +import java.util.Map; +import java.util.Set; + +import org.spearce.jgit.errors.MissingObjectException; +import org.spearce.jgit.errors.PackProtocolException; +import org.spearce.jgit.lib.Constants; +import org.spearce.jgit.lib.NullProgressMonitor; +import org.spearce.jgit.lib.ObjectId; +import org.spearce.jgit.lib.Ref; +import org.spearce.jgit.lib.RefComparator; +import org.spearce.jgit.lib.RefUpdate; +import org.spearce.jgit.lib.Repository; +import org.spearce.jgit.lib.RepositoryConfig; +import org.spearce.jgit.revwalk.ObjectWalk; +import org.spearce.jgit.revwalk.RevCommit; +import org.spearce.jgit.revwalk.RevObject; +import org.spearce.jgit.revwalk.RevWalk; +import org.spearce.jgit.transport.ReceiveCommand.Result; + +/** + * Implements the server side of a push connection, receiving objects. + */ +public class ReceivePack { + static final String CAPABILITY_REPORT_STATUS = BasePackPushConnection.CAPABILITY_REPORT_STATUS; + + static final String CAPABILITY_DELETE_REFS = BasePackPushConnection.CAPABILITY_DELETE_REFS; + + /** Database we write the stored objects into. */ + private final Repository db; + + /** Revision traversal support over {@link #db}. */ + private final RevWalk walk; + + /** Should an incoming transfer validate objects? */ + private boolean checkReceivedObjects; + + /** Should an incoming transfer permit create requests? */ + private boolean allowCreates; + + /** Should an incoming transfer permit delete requests? */ + private boolean allowDeletes; + + /** Should an incoming transfer permit non-fast-forward requests? */ + private boolean allowNonFastForwards; + + /** Hook to validate the update commands before execution. */ + private PreReceiveHook preReceive; + + /** Hook to report on the commands after execution. */ + private PostReceiveHook postReceive; + + private InputStream rawIn; + + private OutputStream rawOut; + + private PacketLineIn pckIn; + + private PacketLineOut pckOut; + + private PrintWriter msgs; + + /** The refs we advertised as existing at the start of the connection. */ + private Map<String, Ref> refs; + + /** Capabilities requested by the client. */ + private Set<String> enabledCapablities; + + /** Commands to execute, as received by the client. */ + private List<ReceiveCommand> commands; + + /** An exception caught while unpacking and fsck'ing the objects. */ + private Throwable unpackError; + + /** if {@link #enabledCapablities} has {@link #CAPABILITY_REPORT_STATUS} */ + private boolean reportStatus; + + /** + * Create a new pack receive for an open repository. + * + * @param into + * the destination repository. + */ + public ReceivePack(final Repository into) { + db = into; + walk = new RevWalk(db); + + final RepositoryConfig cfg = db.getConfig(); + checkReceivedObjects = cfg.getBoolean("receive", "fsckobjects", false); + allowCreates = true; + allowDeletes = !cfg.getBoolean("receive", "denydeletes", false); + allowNonFastForwards = !cfg.getBoolean("receive", + "denynonfastforwards", false); + preReceive = PreReceiveHook.NULL; + postReceive = PostReceiveHook.NULL; + } + + /** @return the repository this receive completes into. */ + public final Repository getRepository() { + return db; + } + + /** @return the RevWalk instance used by this connection. */ + public final RevWalk getRevWalk() { + return walk; + } + + /** + * @return true if this instance will verify received objects are formatted + * correctly. Validating objects requires more CPU time on this side + * of the connection. + */ + public boolean isCheckReceivedObjects() { + return checkReceivedObjects; + } + + /** + * @param check + * true to enable checking received objects; false to assume all + * received objects are valid. + */ + public void setCheckReceivedObjects(final boolean check) { + checkReceivedObjects = check; + } + + /** @return true if the client can request refs to be created. */ + public boolean isAllowCreates() { + return allowCreates; + } + + /** + * @param canCreate + * true to permit create ref commands to be processed. + */ + public void setAllowCreates(final boolean canCreate) { + allowCreates = canCreate; + } + + /** @return true if the client can request refs to be deleted. */ + public boolean isAllowDeletes() { + return allowDeletes; + } + + /** + * @param canDelete + * true to permit delete ref commands to be processed. + */ + public void setAllowDeletes(final boolean canDelete) { + allowDeletes = canDelete; + } + + /** + * @return true if the client can request non-fast-forward updates of a ref, + * possibly making objects unreachable. + */ + public boolean isAllowNonFastForwards() { + return allowNonFastForwards; + } + + /** + * @param canRewind + * true to permit the client to ask for non-fast-forward updates + * of an existing ref. + */ + public void setAllowNonFastForwards(final boolean canRewind) { + allowNonFastForwards = canRewind; + } + + /** @return get the hook invoked before updates occur. */ + public PreReceiveHook getPreReceiveHook() { + return preReceive; + } + + /** + * Set the hook which is invoked prior to commands being executed. + * <p> + * Only valid commands (those which have no obvious errors according to the + * received input and this instance's configuration) are passed into the + * hook. The hook may mark a command with a result of any value other than + * {@link Result#NOT_ATTEMPTED} to block its execution. + * <p> + * The hook may be called with an empty command collection if the current + * set is completely invalid. + * + * @param h + * the hook instance; may be null to disable the hook. + */ + public void setPreReceiveHook(final PreReceiveHook h) { + preReceive = h != null ? h : PreReceiveHook.NULL; + } + + /** @return get the hook invoked after updates occur. */ + public PostReceiveHook getPostReceiveHook() { + return postReceive; + } + + /** + * Set the hook which is invoked after commands are executed. + * <p> + * Only successful commands (type is {@link Result#OK}) are passed into the + * hook. The hook may be called with an empty command collection if the + * current set all resulted in an error. + * + * @param h + * the hook instance; may be null to disable the hook. + */ + public void setPostReceiveHook(final PostReceiveHook h) { + postReceive = h != null ? h : PostReceiveHook.NULL; + } + + /** @return all of the command received by the current request. */ + public List<ReceiveCommand> getAllCommands() { + return Collections.unmodifiableList(commands); + } + + /** + * Send an error message to the client, if it supports receiving them. + * <p> + * If the client doesn't support receiving messages, the message will be + * discarded, with no other indication to the caller or to the client. + * <p> + * {@link PreReceiveHook}s should always try to use + * {@link ReceiveCommand#setResult(Result, String)} with a result status of + * {@link Result#REJECTED_OTHER_REASON} to indicate any reasons for + * rejecting an update. Messages attached to a command are much more likely + * to be returned to the client. + * + * @param what + * string describing the problem identified by the hook. The + * string must not end with an LF, and must not contain an LF. + */ + public void sendError(final String what) { + sendMessage("error", what); + } + + /** + * Send a message to the client, if it supports receiving them. + * <p> + * If the client doesn't support receiving messages, the message will be + * discarded, with no other indication to the caller or to the client. + * + * @param what + * string describing the problem identified by the hook. The + * string must not end with an LF, and must not contain an LF. + */ + public void sendMessage(final String what) { + sendMessage("remote", what); + } + + private void sendMessage(final String type, final String what) { + if (msgs != null) + msgs.println(type + ": " + what); + } + + /** + * Execute the receive task on the socket. + * + * @param input + * raw input to read client commands and pack data from. Caller + * must ensure the input is buffered, otherwise read performance + * may suffer. + * @param output + * response back to the Git network client. Caller must ensure + * the output is buffered, otherwise write performance may + * suffer. + * @param messages + * secondary "notice" channel to send additional messages out + * through. When run over SSH this should be tied back to the + * standard error channel of the command execution. For most + * other network connections this should be null. + * @throws IOException + */ + public void receive(final InputStream input, final OutputStream output, + final OutputStream messages) throws IOException { + try { + rawIn = input; + rawOut = output; + + pckIn = new PacketLineIn(rawIn); + pckOut = new PacketLineOut(rawOut); + if (messages != null) { + msgs = new PrintWriter(new BufferedWriter( + new OutputStreamWriter(messages, Constants.CHARSET), + 8192)) { + @Override + public void println() { + print('\n'); + } + }; + } + + enabledCapablities = new HashSet<String>(); + commands = new ArrayList<ReceiveCommand>(); + + service(); + } finally { + try { + if (msgs != null) { + msgs.flush(); + } + } finally { + rawIn = null; + rawOut = null; + pckIn = null; + pckOut = null; + msgs = null; + refs = null; + enabledCapablities = null; + commands = null; + } + } + } + + private void service() throws IOException { + sendAdvertisedRefs(); + recvCommands(); + if (!commands.isEmpty()) { + enableCapabilities(); + + if (needPack()) { + try { + receivePack(); + if (isCheckReceivedObjects()) + checkConnectivity(); + unpackError = null; + } catch (IOException err) { + unpackError = err; + } catch (RuntimeException err) { + unpackError = err; + } catch (Error err) { + unpackError = err; + } + } + + if (unpackError == null) { + validateCommands(); + executeCommands(); + } + + if (reportStatus) { + sendStatusReport(true, new Reporter() { + void sendString(final String s) throws IOException { + pckOut.writeString(s + "\n"); + } + }); + pckOut.end(); + } else if (msgs != null) { + sendStatusReport(false, new Reporter() { + void sendString(final String s) throws IOException { + msgs.println(s); + } + }); + msgs.flush(); + } + + postReceive.onPostReceive(this, filterCommands(Result.OK)); + } + } + + private void sendAdvertisedRefs() throws IOException { + refs = db.getAllRefs(); + + final StringBuilder m = new StringBuilder(100); + final char[] idtmp = new char[2 * Constants.OBJECT_ID_LENGTH]; + final Iterator<Ref> i = RefComparator.sort(refs.values()).iterator(); + { + if (i.hasNext()) { + final Ref r = i.next(); + format(m, idtmp, r.getObjectId(), r.getOrigName()); + } else { + format(m, idtmp, ObjectId.zeroId(), "capabilities^{}"); + } + m.append('\0'); + m.append(' '); + m.append(CAPABILITY_DELETE_REFS); + m.append(' '); + m.append(CAPABILITY_REPORT_STATUS); + m.append(' '); + writeAdvertisedRef(m); + } + + while (i.hasNext()) { + final Ref r = i.next(); + format(m, idtmp, r.getObjectId(), r.getOrigName()); + writeAdvertisedRef(m); + } + pckOut.end(); + } + + private void format(final StringBuilder m, final char[] idtmp, + final ObjectId id, final String name) { + m.setLength(0); + id.copyTo(idtmp, m); + m.append(' '); + m.append(name); + } + + private void writeAdvertisedRef(final StringBuilder m) throws IOException { + m.append('\n'); + pckOut.writeString(m.toString()); + } + + private void recvCommands() throws IOException { + for (;;) { + String line; + try { + line = pckIn.readStringNoLF(); + } catch (EOFException eof) { + if (commands.isEmpty()) + return; + throw eof; + } + + if (commands.isEmpty()) { + final int nul = line.indexOf('\0'); + if (nul >= 0) { + for (String c : line.substring(nul + 1).split(" ")) + enabledCapablities.add(c); + line = line.substring(0, nul); + } + } + + if (line.length() == 0) + break; + if (line.length() < 83) { + final String m = "error: invalid protocol: wanted 'old new ref'"; + sendError(m); + throw new PackProtocolException(m); + } + + final ObjectId oldId = ObjectId.fromString(line.substring(0, 40)); + final ObjectId newId = ObjectId.fromString(line.substring(41, 81)); + final String name = line.substring(82); + final ReceiveCommand cmd = new ReceiveCommand(oldId, newId, name); + cmd.setRef(refs.get(cmd.getRefName())); + commands.add(cmd); + } + } + + private void enableCapabilities() { + reportStatus = enabledCapablities.contains(CAPABILITY_REPORT_STATUS); + } + + private boolean needPack() { + for (final ReceiveCommand cmd : commands) { + if (cmd.getType() != ReceiveCommand.Type.DELETE) + return true; + } + return false; + } + + private void receivePack() throws IOException { + final IndexPack ip = IndexPack.create(db, rawIn); + ip.setFixThin(true); + ip.setObjectChecking(isCheckReceivedObjects()); + ip.index(NullProgressMonitor.INSTANCE); + ip.renameAndOpenPack(); + } + + private void checkConnectivity() throws IOException { + final ObjectWalk ow = new ObjectWalk(db); + for (final ReceiveCommand cmd : commands) { + if (cmd.getResult() != Result.NOT_ATTEMPTED) + continue; + if (cmd.getType() == ReceiveCommand.Type.DELETE) + continue; + ow.markStart(ow.parseAny(cmd.getNewId())); + } + for (final Ref ref : refs.values()) + ow.markUninteresting(ow.parseAny(ref.getObjectId())); + ow.checkConnectivity(); + } + + private void validateCommands() { + for (final ReceiveCommand cmd : commands) { + final Ref ref = cmd.getRef(); + if (cmd.getResult() != Result.NOT_ATTEMPTED) + continue; + + if (cmd.getType() == ReceiveCommand.Type.DELETE + && !isAllowDeletes()) { + // Deletes are not supported on this repository. + // + cmd.setResult(Result.REJECTED_NODELETE); + continue; + } + + if (cmd.getType() == ReceiveCommand.Type.CREATE) { + if (!isAllowCreates()) { + cmd.setResult(Result.REJECTED_NOCREATE); + continue; + } + + if (ref != null && !isAllowNonFastForwards()) { + // Creation over an existing ref is certainly not going + // to be a fast-forward update. We can reject it early. + // + cmd.setResult(Result.REJECTED_NONFASTFORWARD); + continue; + } + + if (ref != null) { + // A well behaved client shouldn't have sent us an + // update command for a ref we advertised to it. + // + cmd.setResult(Result.REJECTED_OTHER_REASON, "ref exists"); + continue; + } + } + + if (cmd.getType() == ReceiveCommand.Type.DELETE && ref != null + && !ObjectId.zeroId().equals(cmd.getOldId()) + && !ref.getObjectId().equals(cmd.getOldId())) { + // Delete commands can be sent with the old id matching our + // advertised value, *OR* with the old id being 0{40}. Any + // other requested old id is invalid. + // + cmd.setResult(Result.REJECTED_OTHER_REASON, + "invalid old id sent"); + continue; + } + + if (cmd.getType() == ReceiveCommand.Type.UPDATE) { + if (ref == null) { + // The ref must have been advertised in order to be updated. + // + cmd.setResult(Result.REJECTED_OTHER_REASON, "no such ref"); + continue; + } + + if (!ref.getObjectId().equals(cmd.getOldId())) { + // A properly functioning client will send the same + // object id we advertised. + // + cmd.setResult(Result.REJECTED_OTHER_REASON, + "invalid old id sent"); + continue; + } + + // Is this possibly a non-fast-forward style update? + // + RevObject oldObj, newObj; + try { + oldObj = walk.parseAny(cmd.getOldId()); + } catch (IOException e) { + cmd.setResult(Result.REJECTED_MISSING_OBJECT, cmd + .getOldId().name()); + continue; + } + + try { + newObj = walk.parseAny(cmd.getNewId()); + } catch (IOException e) { + cmd.setResult(Result.REJECTED_MISSING_OBJECT, cmd + .getNewId().name()); + continue; + } + + if (oldObj instanceof RevCommit && newObj instanceof RevCommit) { + try { + if (!walk.isMergedInto((RevCommit) oldObj, + (RevCommit) newObj)) { + cmd + .setType(ReceiveCommand.Type.UPDATE_NONFASTFORWARD); + } + } catch (MissingObjectException e) { + cmd.setResult(Result.REJECTED_MISSING_OBJECT, e + .getMessage()); + } catch (IOException e) { + cmd.setResult(Result.REJECTED_OTHER_REASON); + } + } else { + cmd.setType(ReceiveCommand.Type.UPDATE_NONFASTFORWARD); + } + } + + if (!cmd.getRefName().startsWith(Constants.R_REFS) + || !Repository.isValidRefName(cmd.getRefName())) { + cmd.setResult(Result.REJECTED_OTHER_REASON, "funny refname"); + } + } + } + + private void executeCommands() { + preReceive.onPreReceive(this, filterCommands(Result.NOT_ATTEMPTED)); + for (final ReceiveCommand cmd : filterCommands(Result.NOT_ATTEMPTED)) + execute(cmd); + } + + private void execute(final ReceiveCommand cmd) { + try { + final RefUpdate ru = db.updateRef(cmd.getRefName()); + switch (cmd.getType()) { + case DELETE: + if (!ObjectId.zeroId().equals(cmd.getOldId())) { + // We can only do a CAS style delete if the client + // didn't bork its delete request by sending the + // wrong zero id rather than the advertised one. + // + ru.setExpectedOldObjectId(cmd.getOldId()); + } + ru.setForceUpdate(true); + status(cmd, ru.delete(walk)); + break; + + case CREATE: + case UPDATE: + case UPDATE_NONFASTFORWARD: + ru.setForceUpdate(isAllowNonFastForwards()); + ru.setExpectedOldObjectId(cmd.getOldId()); + ru.setNewObjectId(cmd.getNewId()); + ru.setRefLogMessage("push", true); + status(cmd, ru.update(walk)); + break; + } + } catch (IOException err) { + cmd.setResult(Result.REJECTED_OTHER_REASON, "lock error: " + + err.getMessage()); + } + } + + private void status(final ReceiveCommand cmd, final RefUpdate.Result result) { + switch (result) { + case NOT_ATTEMPTED: + cmd.setResult(Result.NOT_ATTEMPTED); + break; + + case LOCK_FAILURE: + case IO_FAILURE: + cmd.setResult(Result.LOCK_FAILURE); + break; + + case NO_CHANGE: + case NEW: + case FORCED: + case FAST_FORWARD: + cmd.setResult(Result.OK); + break; + + case REJECTED: + cmd.setResult(Result.REJECTED_NONFASTFORWARD); + break; + + case REJECTED_CURRENT_BRANCH: + cmd.setResult(Result.REJECTED_CURRENT_BRANCH); + break; + + default: + cmd.setResult(Result.REJECTED_OTHER_REASON, result.name()); + break; + } + } + + private List<ReceiveCommand> filterCommands(final Result want) { + final List<ReceiveCommand> r = new ArrayList<ReceiveCommand>(commands + .size()); + for (final ReceiveCommand cmd : commands) { + if (cmd.getResult() == want) + r.add(cmd); + } + return r; + } + + private void sendStatusReport(final boolean forClient, final Reporter out) + throws IOException { + if (unpackError != null) { + out.sendString("unpack error " + unpackError.getMessage()); + if (forClient) { + for (final ReceiveCommand cmd : commands) { + out.sendString("ng " + cmd.getRefName() + + " n/a (unpacker error)"); + } + } + return; + } + + if (forClient) + out.sendString("unpack ok"); + for (final ReceiveCommand cmd : commands) { + if (cmd.getResult() == Result.OK) { + if (forClient) + out.sendString("ok " + cmd.getRefName()); + continue; + } + + final StringBuilder r = new StringBuilder(); + r.append("ng "); + r.append(cmd.getRefName()); + r.append(" "); + + switch (cmd.getResult()) { + case NOT_ATTEMPTED: + r.append("server bug; ref not processed"); + break; + + case REJECTED_NOCREATE: + r.append("creation prohibited"); + break; + + case REJECTED_NODELETE: + r.append("deletion prohibited"); + break; + + case REJECTED_NONFASTFORWARD: + r.append("non-fast forward"); + break; + + case REJECTED_CURRENT_BRANCH: + r.append("branch is currently checked out"); + break; + + case REJECTED_MISSING_OBJECT: + if (cmd.getMessage() == null) + r.append("missing object(s)"); + else if (cmd.getMessage().length() == 2 * Constants.OBJECT_ID_LENGTH) + r.append("object " + cmd.getMessage() + " missing"); + else + r.append(cmd.getMessage()); + break; + + case REJECTED_OTHER_REASON: + if (cmd.getMessage() == null) + r.append("unspecified reason"); + else + r.append(cmd.getMessage()); + break; + + case LOCK_FAILURE: + r.append("failed to lock"); + break; + + case OK: + // We shouldn't have reached this case (see 'ok' case above). + continue; + } + out.sendString(r.toString()); + } + } + + static abstract class Reporter { + abstract void sendString(String s) throws IOException; + } +} -- 1.6.1.rc4.301.g5497a ^ permalink raw reply related [flat|nested] 17+ messages in thread
* [JGIT PATCH 13/13] Add basic git daemon support to publish receive-pack 2008-12-23 0:27 ` [JGIT PATCH 12/13] Implement the git-receive-pack process in Java Shawn O. Pearce @ 2008-12-23 0:27 ` Shawn O. Pearce 2009-01-03 23:48 ` Robin Rosenberg 0 siblings, 1 reply; 17+ messages in thread From: Shawn O. Pearce @ 2008-12-23 0:27 UTC (permalink / raw) To: Robin Rosenberg; +Cc: git The git:// daemon service receives anonymous TCP connections and runs commands as they are received. Currently we only support the server portion of send-pack/receive-pack, so that is the only service registered in our Daemon class. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> --- .../services/org.spearce.jgit.pgm.TextBuiltin | 1 + .../src/org/spearce/jgit/pgm/Daemon.java | 125 ++++++++ .../src/org/spearce/jgit/transport/Daemon.java | 309 ++++++++++++++++++++ .../org/spearce/jgit/transport/DaemonClient.java | 106 +++++++ .../org/spearce/jgit/transport/DaemonService.java | 120 ++++++++ .../spearce/jgit/transport/TransportGitAnon.java | 3 +- 6 files changed, 662 insertions(+), 2 deletions(-) create mode 100644 org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Daemon.java create mode 100644 org.spearce.jgit/src/org/spearce/jgit/transport/Daemon.java create mode 100644 org.spearce.jgit/src/org/spearce/jgit/transport/DaemonClient.java create mode 100644 org.spearce.jgit/src/org/spearce/jgit/transport/DaemonService.java diff --git a/org.spearce.jgit.pgm/src/META-INF/services/org.spearce.jgit.pgm.TextBuiltin b/org.spearce.jgit.pgm/src/META-INF/services/org.spearce.jgit.pgm.TextBuiltin index e2e7938..5fb0953 100644 --- a/org.spearce.jgit.pgm/src/META-INF/services/org.spearce.jgit.pgm.TextBuiltin +++ b/org.spearce.jgit.pgm/src/META-INF/services/org.spearce.jgit.pgm.TextBuiltin @@ -1,4 +1,5 @@ org.spearce.jgit.pgm.Branch +org.spearce.jgit.pgm.Daemon org.spearce.jgit.pgm.DiffTree org.spearce.jgit.pgm.Fetch org.spearce.jgit.pgm.Glog diff --git a/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Daemon.java b/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Daemon.java new file mode 100644 index 0000000..aafc82e --- /dev/null +++ b/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Daemon.java @@ -0,0 +1,125 @@ +/* + * Copyright (C) 2008, Google Inc. + * + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials provided + * with the distribution. + * + * - Neither the name of the Git Development Community nor the + * names of its contributors may be used to endorse or promote + * products derived from this software without specific prior + * written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND + * CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, + * INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES + * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR + * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT + * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; + * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER + * CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, + * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF + * ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +package org.spearce.jgit.pgm; + +import java.io.File; +import java.net.InetSocketAddress; +import java.util.ArrayList; +import java.util.List; + +import org.kohsuke.args4j.Argument; +import org.kohsuke.args4j.Option; +import org.spearce.jgit.lib.Repository; +import org.spearce.jgit.transport.DaemonService; + +@Command(common = true, usage = "Export repositories over git://") +class Daemon extends TextBuiltin { + @Option(name = "--port", metaVar = "PORT", usage = "port number to listen on") + int port = org.spearce.jgit.transport.Daemon.DEFAULT_PORT; + + @Option(name = "--listen", metaVar = "HOSTNAME", usage = "hostname (or ip) to listen on") + String host; + + @Option(name = "--enable", metaVar = "SERVICE", usage = "enable the service in all repositories", multiValued = true) + final List<String> enable = new ArrayList<String>(); + + @Option(name = "--disable", metaVar = "SERVICE", usage = "disable the service in all repositories", multiValued = true) + final List<String> disable = new ArrayList<String>(); + + @Option(name = "--allow-override", metaVar = "SERVICE", usage = "configure the service in daemon.servicename", multiValued = true) + final List<String> canOverride = new ArrayList<String>(); + + @Option(name = "--forbid-override", metaVar = "SERVICE", usage = "configure the service in daemon.servicename", multiValued = true) + final List<String> forbidOverride = new ArrayList<String>(); + + @Argument(metaVar = "DIRECTORY", usage = "directories to export") + final List<File> directory = new ArrayList<File>(); + + @Override + protected void run() throws Exception { + final org.spearce.jgit.transport.Daemon d; + + d = new org.spearce.jgit.transport.Daemon( + host != null ? new InetSocketAddress(host, port) + : new InetSocketAddress(port)); + + for (final String n : enable) + service(d, n).setEnabled(true); + for (final String n : disable) + service(d, n).setEnabled(false); + + for (final String n : canOverride) + service(d, n).setOverridable(true); + for (final String n : forbidOverride) + service(d, n).setOverridable(false); + + if (directory.isEmpty()) { + export(d, db); + } else { + for (final File f : directory) { + out.println("Exporting " + f.getAbsolutePath()); + d.exportDirectory(f); + } + } + d.start(); + out.println("Listening on " + d.getAddress()); + } + + private DaemonService service(final org.spearce.jgit.transport.Daemon d, + final String n) { + final DaemonService svc = d.getService(n); + if (svc == null) + throw die("Service '" + n + "' not supported"); + return svc; + } + + private void export(final org.spearce.jgit.transport.Daemon daemon, + final Repository repo) { + File d = repo.getDirectory(); + String name = d.getName(); + while (name.equals(".git") || name.equals(".")) { + d = d.getParentFile(); + name = d.getName(); + } + if (!name.endsWith(".git")) + name += ".git"; + + out.println("Exporting current repository as \"" + name + "\""); + daemon.exportRepository(name, repo); + } +} diff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/Daemon.java b/org.spearce.jgit/src/org/spearce/jgit/transport/Daemon.java new file mode 100644 index 0000000..c225740 --- /dev/null +++ b/org.spearce.jgit/src/org/spearce/jgit/transport/Daemon.java @@ -0,0 +1,309 @@ +/* + * Copyright (C) 2008, Google Inc. + * + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials provided + * with the distribution. + * + * - Neither the name of the Git Development Community nor the + * names of its contributors may be used to endorse or promote + * products derived from this software without specific prior + * written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND + * CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, + * INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES + * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR + * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT + * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; + * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER + * CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, + * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF + * ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +package org.spearce.jgit.transport; + +import java.io.BufferedInputStream; +import java.io.BufferedOutputStream; +import java.io.File; +import java.io.IOException; +import java.io.InterruptedIOException; +import java.net.InetSocketAddress; +import java.net.ServerSocket; +import java.net.Socket; +import java.net.SocketAddress; +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashMap; +import java.util.Map; +import java.util.regex.Pattern; + +import org.spearce.jgit.lib.Repository; + +/** Basic daemon for the anonymous <code>git://</code> transport protocol. */ +public class Daemon { + /** 9418: IANA assigned port number for Git. */ + public static final int DEFAULT_PORT = 9418; + + private static final int BACKLOG = 5; + + private static final Pattern SAFE_REPOSITORY_NAME = Pattern + .compile("^[A-Za-z][A-Za-z0-9/_ -]+(\\.git)?$"); + + private InetSocketAddress myAddress; + + private final DaemonService[] services; + + private final ThreadGroup processors; + + private Map<String, Repository> exports; + + private Collection<File> exportBase; + + private boolean run; + + private Thread acceptThread; + + /** Configure a daemon to listen on any available network port. */ + public Daemon() { + this(null); + } + + /** + * Configure a new daemon for the specified network address. + * + * @param addr + * address to listen for connections on. If null, any available + * port will be chosen on all network interfaces. + */ + public Daemon(final InetSocketAddress addr) { + myAddress = addr; + exports = new HashMap<String, Repository>(); + exportBase = new ArrayList<File>(); + processors = new ThreadGroup("Git-Daemon"); + + services = new DaemonService[] { new DaemonService("receive-pack", + "receivepack") { + @Override + protected void execute(final DaemonClient dc, final Repository db) + throws IOException { + final ReceivePack rp = new ReceivePack(db); + rp.receive(dc.getInputStream(), dc.getOutputStream(), null); + } + } }; + } + + /** @return the address connections are received on. */ + public synchronized InetSocketAddress getAddress() { + return myAddress; + } + + /** + * Lookup a supported service so it can be reconfigured. + * + * @param name + * name of the service; e.g. "receive-pack"/"git-receive-pack" or + * "upload-pack"/"git-upload-pack". + * @return the service; null if this daemon implementation doesn't support + * the requested service type. + */ + public synchronized DaemonService getService(String name) { + if (!name.startsWith("git-")) + name = "git-" + name; + for (final DaemonService s : services) { + if (s.getCommandName().equals(name)) + return s; + } + return null; + } + + /** + * Add a single repository to the set that is exported by this daemon. + * <p> + * The existence (or lack-thereof) of <code>git-daemon-export-ok</code> is + * ignored by this method. The repository is always published. + * + * @param name + * name the repository will be published under. + * @param db + * the repository instance. + */ + public void exportRepository(final String name, final Repository db) { + synchronized (exports) { + exports.put(name, db); + } + } + + /** + * Recursively export all Git repositories within a directory. + * + * @param dir + * the directory to export. This directory must not itself be a + * git repository, but any directory below it which has a file + * named <code>git-daemon-export-ok</code> will be published. + */ + public void exportDirectory(final File dir) { + synchronized (exportBase) { + exportBase.add(dir); + } + } + + /** + * Start this daemon on a background thread. + * + * @throws IOException + * the server socket could not be opened. + * @throws IllegalStateException + * the daemon is already running. + */ + public synchronized void start() throws IOException { + if (acceptThread != null) + throw new IllegalStateException("Daemon already running"); + + final ServerSocket listenSock = new ServerSocket( + myAddress != null ? myAddress.getPort() : 0, BACKLOG, + myAddress != null ? myAddress.getAddress() : null); + myAddress = (InetSocketAddress) listenSock.getLocalSocketAddress(); + + run = true; + acceptThread = new Thread(processors, "Git-Daemon-Accept") { + public void run() { + while (isRunning()) { + try { + startClient(listenSock.accept()); + } catch (InterruptedIOException e) { + // Test again to see if we should keep accepting. + } catch (IOException e) { + break; + } + } + + try { + listenSock.close(); + } catch (IOException err) { + // + } finally { + synchronized (Daemon.this) { + acceptThread = null; + } + } + } + }; + acceptThread.start(); + } + + /** @return true if this daemon is receiving connections. */ + public synchronized boolean isRunning() { + return run; + } + + /** Stop this daemon. */ + public synchronized void stop() { + if (acceptThread != null) { + run = false; + acceptThread.interrupt(); + } + } + + private void startClient(final Socket s) { + final DaemonClient dc = new DaemonClient(this); + + final SocketAddress peer = s.getRemoteSocketAddress(); + if (peer instanceof InetSocketAddress) + dc.setRemoteAddress(((InetSocketAddress) peer).getAddress()); + + new Thread(processors, "Git-Daemon-Client " + peer.toString()) { + public void run() { + try { + dc.execute(new BufferedInputStream(s.getInputStream()), + new BufferedOutputStream(s.getOutputStream())); + } catch (IOException e) { + // Ignore unexpected IO exceptions from clients + e.printStackTrace(); + } finally { + try { + s.getInputStream().close(); + } catch (IOException e) { + // Ignore close exceptions + } + try { + s.getOutputStream().close(); + } catch (IOException e) { + // Ignore close exceptions + } + } + } + }.start(); + } + + synchronized DaemonService matchService(final String cmd) { + for (final DaemonService d : services) { + if (d.handles(cmd)) + return d; + } + return null; + } + + Repository openRepository(String name) { + if (!name.startsWith("/")) + return null; + name = name.substring(1); + + Repository db; + synchronized (exports) { + db = exports.get(name); + if (db != null) + return db; + + db = exports.get(name + ".git"); + if (db != null) + return db; + } + + if (SAFE_REPOSITORY_NAME.matcher(name).matches()) { + final File[] search; + synchronized (exportBase) { + search = exportBase.toArray(new File[exportBase.size()]); + } + for (final File f : search) { + db = openRepository(new File(f, name)); + if (db != null) + return db; + + db = openRepository(new File(f, name + ".git")); + if (db != null) + return db; + + db = openRepository(new File(f, name + "/.git")); + if (db != null) + return db; + } + } + return null; + } + + private Repository openRepository(final File d) { + if (d.isDirectory() && new File(d, "git-daemon-export-ok").exists()) { + try { + return new Repository(d); + } catch (IOException err) { + // Ignore + } + } + return null; + } +} diff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/DaemonClient.java b/org.spearce.jgit/src/org/spearce/jgit/transport/DaemonClient.java new file mode 100644 index 0000000..636cf22 --- /dev/null +++ b/org.spearce.jgit/src/org/spearce/jgit/transport/DaemonClient.java @@ -0,0 +1,106 @@ +/* + * Copyright (C) 2008, Google Inc. + * + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials provided + * with the distribution. + * + * - Neither the name of the Git Development Community nor the + * names of its contributors may be used to endorse or promote + * products derived from this software without specific prior + * written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND + * CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, + * INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES + * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR + * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT + * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; + * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER + * CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, + * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF + * ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +package org.spearce.jgit.transport; + +import java.io.IOException; +import java.io.InputStream; +import java.io.OutputStream; +import java.net.InetAddress; + +/** Active network client of {@link Daemon}. */ +public class DaemonClient { + private final Daemon daemon; + + private InetAddress peer; + + private InputStream rawIn; + + private OutputStream rawOut; + + DaemonClient(final Daemon d) { + daemon = d; + } + + void setRemoteAddress(final InetAddress ia) { + peer = ia; + } + + /** @return the daemon which spawned this client. */ + public Daemon getDaemon() { + return daemon; + } + + /** @return Internet address of the remote client. */ + public InetAddress getRemoteAddress() { + return peer; + } + + /** @return input stream to read from the connected client. */ + public InputStream getInputStream() { + return rawIn; + } + + /** @return output stream to send data to the connected client. */ + public OutputStream getOutputStream() { + return rawOut; + } + + void execute(final InputStream in, final OutputStream out) + throws IOException { + rawIn = in; + rawOut = out; + + String cmd = new PacketLineIn(rawIn).readStringNoLF(); + if (cmd == null || cmd.length() == 0) + return; + + final int nul = cmd.indexOf('\0'); + if (nul >= 0) { + // Newer clients hide a "host" header behind this byte. + // Currently we don't use it for anything, so we ignore + // this portion of the command. + // + cmd = cmd.substring(0, nul); + } + + final DaemonService srv = getDaemon().matchService(cmd); + if (srv == null) + return; + srv.execute(this, cmd); + } +} diff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/DaemonService.java b/org.spearce.jgit/src/org/spearce/jgit/transport/DaemonService.java new file mode 100644 index 0000000..775a506 --- /dev/null +++ b/org.spearce.jgit/src/org/spearce/jgit/transport/DaemonService.java @@ -0,0 +1,120 @@ +/* + * Copyright (C) 2008, Google Inc. + * + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials provided + * with the distribution. + * + * - Neither the name of the Git Development Community nor the + * names of its contributors may be used to endorse or promote + * products derived from this software without specific prior + * written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND + * CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, + * INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES + * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR + * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT + * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; + * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER + * CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, + * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF + * ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +package org.spearce.jgit.transport; + +import java.io.IOException; + +import org.spearce.jgit.lib.Repository; + +/** A service exposed by {@link Daemon} over anonymous <code>git://</code>. */ +public abstract class DaemonService { + private final String command; + + private final String config; + + private boolean enabled; + + private boolean overridable; + + protected DaemonService(final String cmdName, final String cfgName) { + command = cmdName.startsWith("git-") ? cmdName : "git-" + cmdName; + config = cfgName; + overridable = true; + } + + /** @return is this service enabled for invocation? */ + public boolean isEnabled() { + return enabled; + } + + /** + * @param on + * true to allow this service to be used; false to deny it. + */ + public void setEnabled(final boolean on) { + enabled = on; + } + + /** @return can this service be configured in the repository config file? */ + public boolean isOverridable() { + return overridable; + } + + /** + * @param on + * true to permit repositories to override this service's enabled + * state with the <code>daemon.servicename</code> config setting. + */ + public void setOverridable(final boolean on) { + overridable = on; + } + + /** @return name of the command requested by clients. */ + public String getCommandName() { + return command; + } + + /** + * Determine if this service can handle the requested command. + * + * @param commandLine + * input line from the client. + * @return true if this command can accept the given command line. + */ + public boolean handles(final String commandLine) { + return command.length() + 1 < commandLine.length() + && commandLine.charAt(command.length()) == ' ' + && commandLine.startsWith(command); + } + + void execute(final DaemonClient client, final String commandLine) + throws IOException { + final String name = commandLine.substring(command.length() + 1); + final Repository db = client.getDaemon().openRepository(name); + if (db == null) + return; + boolean on = isEnabled(); + if (isOverridable()) + on = db.getConfig().getBoolean("daemon", config, on); + if (on) + execute(client, db); + } + + protected abstract void execute(DaemonClient client, Repository db) + throws IOException; +} diff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/TransportGitAnon.java b/org.spearce.jgit/src/org/spearce/jgit/transport/TransportGitAnon.java index a80c335..a11f293 100644 --- a/org.spearce.jgit/src/org/spearce/jgit/transport/TransportGitAnon.java +++ b/org.spearce.jgit/src/org/spearce/jgit/transport/TransportGitAnon.java @@ -56,8 +56,7 @@ * source projects, as there are no authentication or authorization overheads. */ class TransportGitAnon extends PackTransport { - /** IANA assigned port number for Git. */ - static final int GIT_PORT = 9418; + static final int GIT_PORT = Daemon.DEFAULT_PORT; static boolean canHandle(final URIish uri) { return "git".equals(uri.getScheme()); -- 1.6.1.rc4.301.g5497a ^ permalink raw reply related [flat|nested] 17+ messages in thread
* Re: [JGIT PATCH 13/13] Add basic git daemon support to publish receive-pack 2008-12-23 0:27 ` [JGIT PATCH 13/13] Add basic git daemon support to publish receive-pack Shawn O. Pearce @ 2009-01-03 23:48 ` Robin Rosenberg 2009-01-05 2:46 ` [PATCH] Permit a wider range of repository names in jgit daemon requests Shawn O. Pearce 0 siblings, 1 reply; 17+ messages in thread From: Robin Rosenberg @ 2009-01-03 23:48 UTC (permalink / raw) To: Shawn O. Pearce; +Cc: git tisdag 23 december 2008 01:27:23 skrev Shawn O. Pearce: > + private static final Pattern SAFE_REPOSITORY_NAME = Pattern > + .compile("^[A-Za-z][A-Za-z0-9/_ -]+(\\.git)?$"); This restriction is too strict. Wouldn't any path not containing ".." be valid? In particular this did not work with my "EGIT.contrib" repo. I have a lot of repos with names llike "name.purpose". Just adding '.' to the character set isn't really enough. -- robin ^ permalink raw reply [flat|nested] 17+ messages in thread
* [PATCH] Permit a wider range of repository names in jgit daemon requests 2009-01-03 23:48 ` Robin Rosenberg @ 2009-01-05 2:46 ` Shawn O. Pearce 2009-01-05 23:07 ` Robin Rosenberg 0 siblings, 1 reply; 17+ messages in thread From: Shawn O. Pearce @ 2009-01-05 2:46 UTC (permalink / raw) To: Robin Rosenberg; +Cc: git The earlier restriction was too narrow for some applications, for example repositories named "jgit.dev" and "jgit.test" are perfectly valid Git repositories and should still be able to be served by the daemon. By blocking out only uses of ".." as a path component and Windows UNC paths (by blocking "\") we can reasonably prevent the client from escaping the base dirctories configured in the daemon. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> --- Robin Rosenberg <robin.rosenberg@dewire.com> wrote: > tisdag 23 december 2008 01:27:23 skrev Shawn O. Pearce: > > + private static final Pattern SAFE_REPOSITORY_NAME = Pattern > > + .compile("^[A-Za-z][A-Za-z0-9/_ -]+(\\.git)?$"); > > This restriction is too strict. Wouldn't any path not containing ".." be valid? In particular this did not work with my "EGIT.contrib" repo. I > have a lot of repos with names llike "name.purpose". Just adding > '.' to the character set isn't really enough. Yup. .../src/org/spearce/jgit/transport/Daemon.java | 42 +++++++++---------- 1 files changed, 20 insertions(+), 22 deletions(-) diff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/Daemon.java b/org.spearce.jgit/src/org/spearce/jgit/transport/Daemon.java index 646c88d..d39fd04 100644 --- a/org.spearce.jgit/src/org/spearce/jgit/transport/Daemon.java +++ b/org.spearce.jgit/src/org/spearce/jgit/transport/Daemon.java @@ -51,7 +51,6 @@ import java.util.Collection; import java.util.HashMap; import java.util.Map; -import java.util.regex.Pattern; import org.spearce.jgit.lib.Repository; @@ -62,9 +61,6 @@ private static final int BACKLOG = 5; - private static final Pattern SAFE_REPOSITORY_NAME = Pattern - .compile("^[A-Za-z][A-Za-z0-9/_ -]+(\\.git)?$"); - private InetSocketAddress myAddress; private final DaemonService[] services; @@ -292,24 +288,26 @@ synchronized (exports) { return db; } - if (SAFE_REPOSITORY_NAME.matcher(name).matches()) { - final File[] search; - synchronized (exportBase) { - search = exportBase.toArray(new File[exportBase.size()]); - } - for (final File f : search) { - db = openRepository(new File(f, name)); - if (db != null) - return db; - - db = openRepository(new File(f, name + ".git")); - if (db != null) - return db; - - db = openRepository(new File(f, name + "/.git")); - if (db != null) - return db; - } + if (name.startsWith("../") || name.contains("/../") + || name.contains("\\")) + return null; + + final File[] search; + synchronized (exportBase) { + search = exportBase.toArray(new File[exportBase.size()]); + } + for (final File f : search) { + db = openRepository(new File(f, name)); + if (db != null) + return db; + + db = openRepository(new File(f, name + ".git")); + if (db != null) + return db; + + db = openRepository(new File(f, name + "/.git")); + if (db != null) + return db; } return null; } -- 1.6.1.94.g9388 -- Shawn. ^ permalink raw reply related [flat|nested] 17+ messages in thread
* Re: [PATCH] Permit a wider range of repository names in jgit daemon requests 2009-01-05 2:46 ` [PATCH] Permit a wider range of repository names in jgit daemon requests Shawn O. Pearce @ 2009-01-05 23:07 ` Robin Rosenberg 0 siblings, 0 replies; 17+ messages in thread From: Robin Rosenberg @ 2009-01-05 23:07 UTC (permalink / raw) To: Shawn O. Pearce; +Cc: git måndag 05 januari 2009 03:46:22 skrev Shawn O. Pearce: > The earlier restriction was too narrow for some applications, for > example repositories named "jgit.dev" and "jgit.test" are perfectly > valid Git repositories and should still be able to be served by > the daemon. > > By blocking out only uses of ".." as a path component and Windows > UNC paths (by blocking "\") we can reasonably prevent the client > from escaping the base dirctories configured in the daemon. > > + if (name.startsWith("../") || name.contains("/../") > + || name.contains("\\")) //host/share also works as UNC path (even the DOS commands support it, provided you quote the paths) and if you block // shuldn't '/', and '[A-Z]:' also be blocked? \\ is a UNC-prefix only at the beginning of a path so if / need not be filtered, nor does //. Inside a path \\ is the same as \ AFAIK (except directly after the drive letter. This should probablybe factored out into a utilty so we can have a simple unit test for it. -- robin ^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2009-01-05 23:09 UTC | newest] Thread overview: 17+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2008-12-23 0:27 [JGIT PATCH 00/13] Add receive-pack server side support Shawn O. Pearce 2008-12-23 0:27 ` [JGIT PATCH 01/13] Fix invalid "double checked locking" in InflaterCache Shawn O. Pearce 2008-12-23 0:27 ` [JGIT PATCH 02/13] Cleanup stupid release of the cached Inflater in IndexPack Shawn O. Pearce 2008-12-23 0:27 ` [JGIT PATCH 03/13] Cache an Inflater inside a WindowCursor and reuse it as much as possible Shawn O. Pearce 2008-12-23 0:27 ` [JGIT PATCH 04/13] Make RefDatabase thread-safe Shawn O. Pearce 2008-12-23 0:27 ` [JGIT PATCH 05/13] Make PackFile thread-safe Shawn O. Pearce 2008-12-23 0:27 ` [JGIT PATCH 06/13] Make Repository thread-safe Shawn O. Pearce 2008-12-23 0:27 ` [JGIT PATCH 07/13] Don't open a PackFile multiple times on scanForPacks Shawn O. Pearce 2008-12-23 0:27 ` [JGIT PATCH 08/13] Expose RepositoryConfig.getBoolean so applications can use it Shawn O. Pearce 2008-12-23 0:27 ` [JGIT PATCH 09/13] Add AnyObjectId.copyTo(StringBuilder) Shawn O. Pearce 2008-12-23 0:27 ` [JGIT PATCH 10/13] Add compare-and-swap semantics to RefUpdate Shawn O. Pearce 2008-12-23 0:27 ` [JGIT PATCH 11/13] Allow null new ObjectId during RefUpdate.delete Shawn O. Pearce 2008-12-23 0:27 ` [JGIT PATCH 12/13] Implement the git-receive-pack process in Java Shawn O. Pearce 2008-12-23 0:27 ` [JGIT PATCH 13/13] Add basic git daemon support to publish receive-pack Shawn O. Pearce 2009-01-03 23:48 ` Robin Rosenberg 2009-01-05 2:46 ` [PATCH] Permit a wider range of repository names in jgit daemon requests Shawn O. Pearce 2009-01-05 23:07 ` Robin Rosenberg
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).