[jgit] index v2 pull request

[jgit] index v2 pull request

[Shawn O. Pearce]

So I've started to hack on jgit again, to build out more of the lower
level library.  

This particular series adds support for pack file index v2, which
Nico added to C Git several months ago and is slated to become the
default pack index file format in a future release of Git, due to
its increased error checking during delta reuse.

I also cleaned up the pack file access code to be shorter and more
concise.  Unfortunately the overall series diffstat doesn't show it
that way, but I think its now easier to follow how the control flows.

I've improved performance for packfile data access by adding a delta
base cache, like we did in C Git months ago.  This cache is really
necessary on modern repositories as pack-objects is tending to use
longer delta chains, due to the reader side now having this delta
base cache.  Profiling was showing jgit was spending a very large
amount of time in unpacking tree deltas.

We now actually use the following .git/config options to control
how the window cache limits work:

  core.packedGitWindowSize
  core.packedGitLimit
  core.deltaBaseCacheLimit
  core.packedGitMMAP

The configuration file parser now recognizes the "k/m/g" suffixes
that are comonly used in the first three parameters.  The last
option is a boolean and is unique to jgit.

The window cache and delta base cache also exploit SoftReferences to
make the caches memory-sensitive.  This way the JVM's GC can whack
our caches back if its running low on memory.  Its not quite the
same as C Git's concept of "malloc failed, flush out some windows
and retry" but its reasonably close.

Finally we now use a single WindowCache and delta base cache for
the entire JVM when running in an Eclipse workbench.  If the user
has multiple repositories open at once (say different projects)
they need to co-habitate in the JVM's virtual address space.  By
sharing the same caches the repositories can evict each other's
windows when they need to make room.

----

The following changes since commit 3cd4f3f9119b79750bbfc542119451a668c462e3:
  Robin Rosenberg (1):
        Readded qualifier to version 0.3.1

are available in the git repository at:

  repo.or.cz:/srv/git/egit/spearce.git master

Shawn O. Pearce (41):
      Make the absolute path of a WindowedFile available to callers
      Include the pack index file name when reporting a bad header
      Report pack file open failures to the console
      Refactor index reading code from PackFile to PackIndex
      Rename PackIndex.readIndexHeader to loadVersion1
      Read pack index files using FileInputStream and not WindowedFile
      Don't bother checking the index file length after reading it
      Move index:pack object count checking into PackFile class
      Refactor PackIndex version 1 handling code into PackIndexV1
      Fully document the PackIndex API
      Hoist readFully and decodeUInt32 up from PackIndexV1 to PackIndex
      Refactor PackIndex.readFully to support reading into the middle
      Read the pack index header to perform automatic version detection
      Add complete reading support for pack index v2
      Stop creating "new format" loose objects as C Git no longer likes them
      Refactor RepositoryConfig to be stacked many levels deep
      Don't make unit tests depend upon compressed file lengths
      Corrected name of the method that reloads the packed-refs data
      Cleanup misspelling in Javadoc of Repository class
      Extract StGitPatch from Repository to its own top-level type
      Refactor RepositoryState into its own top-level type
      Refactor PackFile construction to take the idxFile path from scan
      Shuffle the length of a WindowedFile down into its provider
      Defer opening pack files until we actually need object data
      Use Java 5 bit twiddling APIs rather than hand-rolling bit counting
      Support reading config file integers with k/m/g suffixes
      Honor core.packedGitWindowSize, packedGitLimit, packedGitMMAP config
      Enable easy access to the user's ~/.gitconfig
      Use a single WindowCache for the entire Eclipse workspace
      Modify the WindowedFile API to pass around a WindowCursor during reads
      Change ByteWindow implementations to hold data by SoftReferences
      Refactor WindowedFile/WindowCache so all windows are a uniform size
      Simplify WindowCache and WindowedFile internal APIs to be private
      Use a proper ReferenceQueue to avoid duplpicate window entries
      New RepositoryConfig getInt,getBoolean helper methods with no subsection
      Change delta packed object loaders to use a PackedObjectLoader for base
      Implement core.deltaBaseCacheLimit to improve unpacking performance
      Remove unnecessary bitwise AND masks in readTree
      Precount the number of tree entries during readTree
      Change the ObjectLoader.getType method to return int
      Fix ArrayIndexOutOfBoundsException in WindowCache

 .../egit/core/op/ConnectProviderOperation.java     |    4 +-
 .../spearce/egit/core/project/GitProjectData.java  |   27 +++-
 .../src/org/spearce/egit/ui/GitHistoryPage.java    |    2 +-
 .../internal/decorators/GitResourceDecorator.java  |    2 +-
 .../tst/org/spearce/jgit/lib/ReadTreeTest.java     |    4 +-
 .../org/spearce/jgit/lib/RepositoryConfigTest.java |    9 +-
 .../tst/org/spearce/jgit/lib/T0003_Basic.java      |   90 ---------
 .../tst/org/spearce/jgit/lib/T0004_PackReader.java |   11 +-
 org.spearce.jgit/META-INF/MANIFEST.MF              |    3 +-
 .../src/org/spearce/jgit/lib/ByteArrayWindow.java  |   27 +--
 .../src/org/spearce/jgit/lib/ByteBufferWindow.java |   21 +--
 .../src/org/spearce/jgit/lib/ByteWindow.java       |   42 +++--
 .../src/org/spearce/jgit/lib/Constants.java        |   21 ++
 .../src/org/spearce/jgit/lib/CoreConfig.java       |   61 +++++-
 .../jgit/lib/DeltaOfsPackedObjectLoader.java       |    2 +-
 .../spearce/jgit/lib/DeltaPackedObjectLoader.java  |   29 ++-
 .../jgit/lib/DeltaRefPackedObjectLoader.java       |    4 +-
 .../src/org/spearce/jgit/lib/ObjectLoader.java     |    6 +-
 .../src/org/spearce/jgit/lib/ObjectWriter.java     |   40 +---
 .../src/org/spearce/jgit/lib/PackFile.java         |  155 +++++----------
 .../src/org/spearce/jgit/lib/PackIndex.java        |  165 +++++++++++++++
 .../src/org/spearce/jgit/lib/PackIndexV1.java      |   84 ++++++++
 .../src/org/spearce/jgit/lib/PackIndexV2.java      |  125 ++++++++++++
 .../org/spearce/jgit/lib/PackedObjectLoader.java   |   13 +-
 .../src/org/spearce/jgit/lib/Repository.java       |  211 +++++---------------
 .../src/org/spearce/jgit/lib/RepositoryConfig.java |  172 +++++++++++-----
 .../src/org/spearce/jgit/lib/RepositoryState.java  |  108 ++++++++++
 .../src/org/spearce/jgit/lib/Tree.java             |   40 ++--
 .../org/spearce/jgit/lib/UnpackedObjectCache.java  |  156 +++++++++++++++
 .../org/spearce/jgit/lib/UnpackedObjectLoader.java |   20 +--
 .../spearce/jgit/lib/WholePackedObjectLoader.java  |   12 +-
 .../src/org/spearce/jgit/lib/WindowCache.java      |  151 +++++++++++---
 .../src/org/spearce/jgit/lib/WindowCursor.java     |   67 ++++++
 .../src/org/spearce/jgit/lib/WindowProvider.java   |   75 -------
 .../src/org/spearce/jgit/lib/WindowedFile.java     |  207 +++++++++++--------
 .../src/org/spearce/jgit/stgit/StGitPatch.java     |   52 +++++
 36 files changed, 1459 insertions(+), 759 deletions(-)
 create mode 100644 org.spearce.jgit/src/org/spearce/jgit/lib/PackIndex.java
 create mode 100644 org.spearce.jgit/src/org/spearce/jgit/lib/PackIndexV1.java
 create mode 100644 org.spearce.jgit/src/org/spearce/jgit/lib/PackIndexV2.java
 create mode 100644 org.spearce.jgit/src/org/spearce/jgit/lib/RepositoryState.java
 create mode 100644 org.spearce.jgit/src/org/spearce/jgit/lib/UnpackedObjectCache.java
 create mode 100644 org.spearce.jgit/src/org/spearce/jgit/lib/WindowCursor.java
 delete mode 100644 org.spearce.jgit/src/org/spearce/jgit/lib/WindowProvider.java
 create mode 100644 org.spearce.jgit/src/org/spearce/jgit/stgit/StGitPatch.java

-- 
Shawn.

Re: [jgit] index v2 pull request

[Jakub Narebski]

Shawn O. Pearce wrote:

> Unfortunately the overall series diffstat doesn't show it
> that way, but I think its now easier to follow how the control flows.

Isn't it the place then to put --dirstat instead of --diffstat?
-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git

Re: [jgit] index v2 pull request

[Shawn O. Pearce]

Jakub Narebski <jnareb@gmail.com> wrote:
> Shawn O. Pearce wrote:
> 
> > Unfortunately the overall series diffstat doesn't show it
> > that way, but I think its now easier to follow how the control flows.
> 
> Isn't it the place then to put --dirstat instead of --diffstat?

Hmm.  Well, jgit is basically all in one directory right now,
so --dirstat wouldn't have shown a nice neat organization the
way you think it would.

I've started to create a new directory in jgit that contains
revision walking machinary.  So once that is complete then
yea, --dirstat may show the two different parts nicely, but
even then it still isn't very useful on this project.

-- 
Shawn.

Re: [jgit] index v2 pull request

[Robin Rosenberg]

Den Saturday 08 March 2008 03.50.27 skrev Shawn O. Pearce:
> So I've started to hack on jgit again, to build out more of the lower
> level library.
Welcome back Shawn!

> This particular series adds support for pack file index v2, which
> Nico added to C Git several months ago and is slated to become the
> default pack index file format in a future release of Git, due to
> its increased error checking during delta reuse.
This i great news. I've been thinking about it, praying that you'd come
and do the work. :)

> We now actually use the following .git/config options to control
> how the window cache limits work:
>
>   core.packedGitWindowSize
>   core.packedGitLimit
>   core.deltaBaseCacheLimit
>   core.packedGitMMAP

Will the net effect, in practice, be the same on Jgit as on C Git?

>   repo.or.cz:/srv/git/egit/spearce.git master
Something missing in the formatting?

> Shawn O. Pearce (41):
>       Make the absolute path of a WindowedFile available to callers
>       Include the pack index file name when reporting a bad header
>       Report pack file open failures to the console
>       Refactor index reading code from PackFile to PackIndex
>       Rename PackIndex.readIndexHeader to loadVersion1
>       Read pack index files using FileInputStream and not WindowedFile
>       Don't bother checking the index file length after reading it
>       Move index:pack object count checking into PackFile class
>       Refactor PackIndex version 1 handling code into PackIndexV1
>       Fully document the PackIndex API
>       Hoist readFully and decodeUInt32 up from PackIndexV1 to PackIndex
>       Refactor PackIndex.readFully to support reading into the middle
>       Read the pack index header to perform automatic version detection
>       Add complete reading support for pack index v2
Yum, yum. My 15 minute review so far likes it.
>       Stop creating "new format" loose objects as C Git no longer likes
> them Refactor RepositoryConfig to be stacked many levels deep
And there goes the unit test for reading them...

>       Don't make unit tests depend upon compressed file lengths
>       Corrected name of the method that reloads the packed-refs data
>       Cleanup misspelling in Javadoc of Repository class
>       Extract StGitPatch from Repository to its own top-level type
I'm considering dropping stacked git support. We should have much
more of it and I don't use it myself anymore to what support we have
gets very little testing.

>       Refactor RepositoryState into its own top-level type
\ No newline at end of file

>       Refactor PackFile construction to take the idxFile path from scan
>       Shuffle the length of a WindowedFile down into its provider
>       Defer opening pack files until we actually need object data
>       Use Java 5 bit twiddling APIs rather than hand-rolling bit counting
I'm surprised *you* didn't know about them bitcounting algorithm java uses 
before. It's pretty common.

>       Support reading config file integers with k/m/g suffixes
>       Honor core.packedGitWindowSize, packedGitLimit, packedGitMMAP config
>       Enable easy access to the user's ~/.gitconfig
>       Use a single WindowCache for the entire Eclipse workspace
>       Modify the WindowedFile API to pass around a WindowCursor during reads 
>       Change ByteWindow implementations to hold data by SoftReferences 
>       Refactor WindowedFile/WindowCache so all windows are a uniform size
>       Simplify WindowCache and WindowedFile internal APIs to be private Use 
a proper ReferenceQueue to avoid duplpicate window entries
>       New  RepositoryConfig getInt,getBoolean helper methods with no 
subsection Change delta packed object loaders to use a PackedObjectLoader for 
base Implement core.deltaBaseCacheLimit to improve unpacking performance
>       Remove unnecessary bitwise AND masks in readTree 
Good catch
>       Precount the number of tree entries during readTree
this one too
>       Change the ObjectLoader.getType method to return int
>       Fix ArrayIndexOutOfBoundsException in WindowCache

Very good work (as always) Shawn. I've only read the code so far. I'll do some 
testing too before pushing. Do you have any unit tests for the v2 pack format 
(config reading too for that matter)?

-- robin

Re: [jgit] index v2 pull request

[Imran M Yousuf]

I would like to volunteer to work in JGit;
can someone let me know where I can pick some tasks to implement?

Best regards,

Imran

On Mon, Mar 10, 2008 at 5:51 AM, Robin Rosenberg
<robin.rosenberg@dewire.com> wrote:
> Den Saturday 08 March 2008 03.50.27 skrev Shawn O. Pearce:
>
> > So I've started to hack on jgit again, to build out more of the lower
>  > level library.
>  Welcome back Shawn!
>
>
>  > This particular series adds support for pack file index v2, which
>  > Nico added to C Git several months ago and is slated to become the
>  > default pack index file format in a future release of Git, due to
>  > its increased error checking during delta reuse.
>  This i great news. I've been thinking about it, praying that you'd come
>  and do the work. :)
>
>
>  > We now actually use the following .git/config options to control
>  > how the window cache limits work:
>  >
>  >   core.packedGitWindowSize
>  >   core.packedGitLimit
>  >   core.deltaBaseCacheLimit
>  >   core.packedGitMMAP
>
>  Will the net effect, in practice, be the same on Jgit as on C Git?
>
>
>  >   repo.or.cz:/srv/git/egit/spearce.git master
>  Something missing in the formatting?
>
>
>  > Shawn O. Pearce (41):
>  >       Make the absolute path of a WindowedFile available to callers
>  >       Include the pack index file name when reporting a bad header
>  >       Report pack file open failures to the console
>  >       Refactor index reading code from PackFile to PackIndex
>  >       Rename PackIndex.readIndexHeader to loadVersion1
>  >       Read pack index files using FileInputStream and not WindowedFile
>  >       Don't bother checking the index file length after reading it
>  >       Move index:pack object count checking into PackFile class
>  >       Refactor PackIndex version 1 handling code into PackIndexV1
>  >       Fully document the PackIndex API
>  >       Hoist readFully and decodeUInt32 up from PackIndexV1 to PackIndex
>  >       Refactor PackIndex.readFully to support reading into the middle
>  >       Read the pack index header to perform automatic version detection
>  >       Add complete reading support for pack index v2
>  Yum, yum. My 15 minute review so far likes it.
>
> >       Stop creating "new format" loose objects as C Git no longer likes
>  > them Refactor RepositoryConfig to be stacked many levels deep
>  And there goes the unit test for reading them...
>
>
>  >       Don't make unit tests depend upon compressed file lengths
>  >       Corrected name of the method that reloads the packed-refs data
>  >       Cleanup misspelling in Javadoc of Repository class
>  >       Extract StGitPatch from Repository to its own top-level type
>  I'm considering dropping stacked git support. We should have much
>  more of it and I don't use it myself anymore to what support we have
>  gets very little testing.
>
>
>  >       Refactor RepositoryState into its own top-level type
>  \ No newline at end of file
>
>
>  >       Refactor PackFile construction to take the idxFile path from scan
>  >       Shuffle the length of a WindowedFile down into its provider
>  >       Defer opening pack files until we actually need object data
>  >       Use Java 5 bit twiddling APIs rather than hand-rolling bit counting
>  I'm surprised *you* didn't know about them bitcounting algorithm java uses
>  before. It's pretty common.
>
>
>  >       Support reading config file integers with k/m/g suffixes
>  >       Honor core.packedGitWindowSize, packedGitLimit, packedGitMMAP config
>  >       Enable easy access to the user's ~/.gitconfig
>  >       Use a single WindowCache for the entire Eclipse workspace
>  >       Modify the WindowedFile API to pass around a WindowCursor during reads
>  >       Change ByteWindow implementations to hold data by SoftReferences
>  >       Refactor WindowedFile/WindowCache so all windows are a uniform size
>  >       Simplify WindowCache and WindowedFile internal APIs to be private Use
>  a proper ReferenceQueue to avoid duplpicate window entries
>  >       New  RepositoryConfig getInt,getBoolean helper methods with no
>  subsection Change delta packed object loaders to use a PackedObjectLoader for
>  base Implement core.deltaBaseCacheLimit to improve unpacking performance
>  >       Remove unnecessary bitwise AND masks in readTree
>  Good catch
>
> >       Precount the number of tree entries during readTree
>  this one too
>
> >       Change the ObjectLoader.getType method to return int
>  >       Fix ArrayIndexOutOfBoundsException in WindowCache
>
>  Very good work (as always) Shawn. I've only read the code so far. I'll do some
>  testing too before pushing. Do you have any unit tests for the v2 pack format
>  (config reading too for that matter)?
>
>  -- robin
>
>
> --
>  To unsubscribe from this list: send the line "unsubscribe git" in
>  the body of a message to majordomo@vger.kernel.org
>  More majordomo info at  http://vger.kernel.org/majordomo-info.html
>



-- 
Imran M Yousuf
Entrepreneur & Software Engineer
Smart IT Engineering
Dhaka, Bangladesh
Email: imran@smartitengineering.com
Mobile: +880-1711402557

Re: [jgit] index v2 pull request

[Robin Rosenberg]

Den Monday 10 March 2008 08.32.21 skrev Imran M Yousuf:
> I would like to volunteer to work in JGit;
> can someone let me know where I can pick some tasks to implement?
>

You can pick about anything you want that noone is working one. Pick something
that you feel is withing your grasp, or something simple outside. We need
more test cases (doesn't require much programming skills, but thinking). Unit
testing of Eclipse stuff falls into the same category. I have litte experience 
here, but I think there are lots of examples in Eclipse itself if one starts 
to poke around.

Diff against workspace/index doesn't work presently. The reworked 
indexdiffwalker or the iterator could be used for that. Please tell me if you 
start hacking on this as it's on my listl, just not yet. Should be reasonably 
hard, involves both jgit and eclipse programming.

There are some issues with selecting versions in the history view on Windows.

Copy diff as git patch to the clipboard and vice verse. Send email.

Harder and might involve more changes all over (that either I or Shawn will 
object to) is things like rename detection targeted fo r

Interface to quickdiff machinery for committing specific hunks.

Blame (even without rename detection) would be useful. Look at the CVS
annotate for inspiration. 

Support for git submodules. Will touch the core parts all over the place, but 
probably not in very intrusive way. Will require lots of test cases.

Not really git, but highlighting of trailing whitespace is on my wish list. I 
can't find a plugin that does that.

Adjust the commit dialog for amend so it shows all to-be commited changed 
files when amending.

The number of choices is infinite.

Ant, Cruisecontrol, Maven, etc-tasks.

Pick a pet.

-- robin

Re: [jgit] index v2 pull request

[Imran M Yousuf]

On Tue, Mar 11, 2008 at 3:53 AM, Robin Rosenberg
<robin.rosenberg@dewire.com> wrote:
> Den Monday 10 March 2008 08.32.21 skrev Imran M Yousuf:
>
> > I would like to volunteer to work in JGit;
>  > can someone let me know where I can pick some tasks to implement?
>  >
>
>  You can pick about anything you want that noone is working one. Pick something
>  that you feel is withing your grasp, or something simple outside. We need
>  more test cases (doesn't require much programming skills, but thinking). Unit
>  testing of Eclipse stuff falls into the same category. I have litte experience
>  here, but I think there are lots of examples in Eclipse itself if one starts
>  to poke around.
>
>  Diff against workspace/index doesn't work presently. The reworked
>  indexdiffwalker or the iterator could be used for that. Please tell me if you
>  start hacking on this as it's on my listl, just not yet. Should be reasonably
>  hard, involves both jgit and eclipse programming.
>
>  There are some issues with selecting versions in the history view on Windows.
>
>  Copy diff as git patch to the clipboard and vice verse. Send email.
>
>  Harder and might involve more changes all over (that either I or Shawn will
>  object to) is things like rename detection targeted fo r
>
>  Interface to quickdiff machinery for committing specific hunks.
>
>  Blame (even without rename detection) would be useful. Look at the CVS
>  annotate for inspiration.
>
>  Support for git submodules. Will touch the core parts all over the place, but
>  probably not in very intrusive way. Will require lots of test cases.
>
>  Not really git, but highlighting of trailing whitespace is on my wish list. I
>  can't find a plugin that does that.
>
>  Adjust the commit dialog for amend so it shows all to-be commited changed
>  files when amending.
>
>  The number of choices is infinite.
>
>  Ant, Cruisecontrol, Maven, etc-tasks.

Maven: Does this mean mavenizing the project? If so I would start it
this weekend. If you were referring to GIT Maven SCM please also let
me know.
Cruisecontrol: I would instead like to setup Hudson; I find it to be
quite user friendly and flexible at the same time. I would also like
to set up it for JGit. Once we JGit has more features like DIff remote
fetching and stuff we can also build a hudson plugin :).

I am currently looking into .git/config format as Shawn suggested.

- Imran

>
>  Pick a pet.
>
>  -- robin
>
>



-- 
Imran M Yousuf
Entrepreneur & Software Engineer
Smart IT Engineering
Dhaka, Bangladesh
Email: imran@smartitengineering.com
Mobile: +880-1711402557

Re: [jgit] index v2 pull request

[Robin Rosenberg]

Den Wednesday 12 March 2008 03.52.05 skrev Imran M Yousuf:
> Maven: Does this mean mavenizing the project? If so I would start it
> this weekend. If you were referring to GIT Maven SCM please also let
> me know.
Not to build using, but create SCM functions for those so thay can use
git without <exec> tasks. Git isn't available in any of those, while much
worse SCM's have support.

> I am currently looking into .git/config format as Shawn suggested.

That's a good one too. I have code for parsing the remotes specs, though
e.g. "+refs/heads/master/*:refs/remotes/origin/*", but nothing for the "branch" config. I haven't used it yet, so it's a rip-out.	

-- robin

>From 042fa0bef8b5d3ec8f5b1e385766ca61528e72aa Mon Sep 17 00:00:00 2001
From: Robin Rosenberg <robin.rosenberg@dewire.com>
Date: Wed, 5 Mar 2008 23:27:57 +0100
Subject: [PATCH] Add support for parsing remote specs

Signed-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>
---
 .../tst/org/spearce/jgit/lib/RemoteSpecTest.java   |   95 +++++++++++++
 .../src/org/spearce/jgit/lib/RemoteSpec.java       |  142 ++++++++++++++++++++
 .../src/org/spearce/jgit/lib/Repository.java       |   12 ++
 3 files changed, 249 insertions(+), 0 deletions(-)
 create mode 100644 org.spearce.jgit.test/tst/org/spearce/jgit/lib/RemoteSpecTest.java
 create mode 100644 org.spearce.jgit/src/org/spearce/jgit/lib/RemoteSpec.java

diff --git a/org.spearce.jgit.test/tst/org/spearce/jgit/lib/RemoteSpecTest.java b/org.spearce.jgit.test/tst/org/spearce/jgit/lib/RemoteSpecTest.java
new file mode 100644
index 0000000..817e697
--- /dev/null
+++ b/org.spearce.jgit.test/tst/org/spearce/jgit/lib/RemoteSpecTest.java
@@ -0,0 +1,95 @@
+/*
+ *  Copyright (C) 2007  Robin Rosenberg
+ *
+ *  This library is free software; you can redistribute it and/or
+ *  modify it under the terms of the GNU General Public
+ *  License, version 2, as published by the Free Software Foundation.
+ *
+ *  This library is distributed in the hope that it will be useful,
+ *  but WITHOUT ANY WARRANTY; without even the implied warranty of
+ *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ *  General Public License for more details.
+ *
+ *  You should have received a copy of the GNU General Public
+ *  License along with this library; if not, write to the Free Software
+ *  Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301
+ */
+package org.spearce.jgit.lib;
+
+import java.io.File;
+import java.io.FileOutputStream;
+
+/**
+ * Test parsing of git remotes
+ */
+public class RemoteSpecTest extends RepositoryTestCase {
+
+	/**
+	 * Test simplest case
+	 *
+	 * @throws Exception
+	 *
+	 */
+	public void testSimplestOk() throws Exception {
+		RemoteSpec spec = new RemoteSpec("their", "git://foo.bar/zip.git",
+				"refs/heads/master:refs/heads/origin",null);
+		assertEquals("refs/heads/master", spec.getFetchRemoteRef());
+		assertEquals("refs/heads/origin", spec.getFetchLocalRef());
+		assertFalse(spec.isFetchMatchAny());
+		assertFalse(spec.isFetchOverwriteAlways());
+	}
+
+	/**
+	 * Test a standard case
+	 *
+	 * @throws Exception
+	 */
+	public void testStandardOk() throws Exception {
+		RemoteSpec spec = new RemoteSpec("their", "git://example.com/zip.git",
+				"+refs/heads/master/*:refs/remotes/origin/*",null);
+		assertEquals("git://example.com/zip.git", spec.getUrl());
+		assertEquals("refs/heads/master", spec.getFetchRemoteRef());
+		assertEquals("refs/remotes/origin", spec.getFetchLocalRef());
+		assertTrue(spec.isFetchMatchAny());
+		assertTrue(spec.isFetchOverwriteAlways());
+	}
+
+	/**
+	 * Test a <quote>safer</quote> almost standard case
+	 *
+	 * @throws Exception
+	 */
+	public void testNonStandardSaferOk() throws Exception {
+		RemoteSpec spec = new RemoteSpec("their", "git://example.com/zip.git",
+				"refs/heads/master/*:refs/remotes/origin/*",null);
+		assertEquals("git://example.com/zip.git", spec.getUrl());
+		assertEquals("refs/heads/master", spec.getFetchRemoteRef());
+		assertEquals("refs/remotes/origin", spec.getFetchLocalRef());
+		assertTrue(spec.isFetchMatchAny());
+		assertFalse(spec.isFetchOverwriteAlways());
+	}
+
+	/**
+	 * Test a case copied from a real Git repo
+	 *
+	 * @throws Exception
+	 */
+	public void testReadFromConfig() throws Exception {
+		File file = new File(db.getDirectory(),"config");
+		FileOutputStream stream = new FileOutputStream(file,true);
+		try {
+			stream.write(("[remote \"spearce\"]\n"+
+		"url = http://www.spearce.org/projects/scm/egit.git\n"+
+		"fetch = +refs/heads/*:refs/remotes/spearce/*\n").getBytes());
+		} finally {
+			stream.close();
+		}
+		db.getConfig().load();
+		RemoteSpec remoteSpec = db.getRemoteSpec("spearce");
+		assertEquals("http://www.spearce.org/projects/scm/egit.git", remoteSpec.getUrl());
+		assertEquals("refs/heads", remoteSpec.getFetchRemoteRef());
+		assertEquals("refs/remotes/spearce", remoteSpec.getFetchLocalRef());
+		assertTrue(remoteSpec.isFetchMatchAny());
+		assertTrue(remoteSpec.isFetchOverwriteAlways());
+	}
+}
diff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/RemoteSpec.java b/org.spearce.jgit/src/org/spearce/jgit/lib/RemoteSpec.java
new file mode 100644
index 0000000..a6a3b6c
--- /dev/null
+++ b/org.spearce.jgit/src/org/spearce/jgit/lib/RemoteSpec.java
@@ -0,0 +1,142 @@
+/*
+ *  Copyright (C) 2007  Robin Rosenberg
+ *
+ *  This library is free software; you can redistribute it and/or
+ *  modify it under the terms of the GNU General Public
+ *  License, version 2, as published by the Free Software Foundation.
+ *
+ *  This library is distributed in the hope that it will be useful,
+ *  but WITHOUT ANY WARRANTY; without even the implied warranty of
+ *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ *  General Public License for more details.
+ *
+ *  You should have received a copy of the GNU General Public
+ *  License along with this library; if not, write to the Free Software
+ *  Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301
+ */
+package org.spearce.jgit.lib;
+
+/**
+ * Information about how to synchronize with a remote Git repository.
+ *
+ * A remote is stored in the <GIT_DIR>/config as
+ *
+ * <pre>
+ *  [remote &quot;name&quot;]
+ *     url = URL:ish
+ *     fetch = [+]remoteref:localref
+ * </pre>
+ *
+ * There are more variants but we do not support them here yet.
+ */
+public class RemoteSpec {
+
+	static class Info {
+		boolean overwriteAlways;
+
+		boolean matchAny;
+
+		String remoteRef;
+
+		String localRef;
+	}
+
+	Info fetch = new Info();
+
+	Info push = null;
+
+	private final String name;
+
+	private final String url;
+
+	/**
+	 * @return name of remote. This is a local short identifier
+	 */
+	public String getName() {
+		return name;
+	}
+
+	/**
+	 * @return the URL:ish location of the remote Git repository
+	 */
+	public String getUrl() {
+		return url;
+	}
+
+	/**
+	 * @return the local ref part used for fetch heads info
+	 */
+	public String getFetchLocalRef() {
+		return fetch.localRef;
+	}
+
+	/**
+	 * @return the remote ref part used for fetching refs from the remote repo
+	 */
+	public String getFetchRemoteRef() {
+		return fetch.remoteRef;
+	}
+
+	/**
+	 * @return whether the fetch matches all branches under the ref or just the
+	 *         named ref
+	 */
+	public boolean isFetchMatchAny() {
+		return fetch.matchAny;
+	}
+
+	/**
+	 * @return whether the tracking branch is always updated, or only when the
+	 *         update is a fast forward
+	 */
+	public boolean isFetchOverwriteAlways() {
+		return fetch.overwriteAlways;
+	}
+
+	/**
+	 * Create a representation of a git remote specification.
+	 *
+	 * @param name A local short identifier
+	 * @param url The URL:ish used for fetching / pushing
+	 * @param fetchPattern refspec for fetching
+	 * @param pushPattern refspec for pushing or null
+	 */
+	public RemoteSpec(String name, String url, String fetchPattern,
+			String pushPattern) {
+		this.name = name;
+		this.url = url;
+		parse(fetchPattern, fetch);
+		if (pushPattern != null) {
+			push = new Info();
+			parse(pushPattern, push);
+		}
+	}
+
+	private void parse(String fetchSpec, Info info) {
+		int p = 0;
+		if (fetchSpec.charAt(p) == '+') {
+			info.overwriteAlways = true;
+			++p;
+		}
+		int cp = fetchSpec.indexOf(':');
+		if (cp < 0)
+			throw new IllegalArgumentException("Bad remote format " + fetchSpec);
+		info.remoteRef = fetchSpec.substring(p, cp);
+		info.localRef = fetchSpec.substring(cp + 1);
+		if (info.remoteRef.endsWith("/*")) {
+			info.matchAny = true;
+			info.remoteRef = info.remoteRef.substring(0, info.remoteRef
+					.length() - 2);
+		}
+		if (info.localRef.endsWith("/*")) {
+			if (!info.matchAny)
+				throw new IllegalArgumentException("Bad remote format "
+						+ fetchSpec);
+			info.localRef = info.localRef.substring(0,
+					info.localRef.length() - 2);
+		} else
+			if (info.matchAny)
+				throw new IllegalArgumentException("Bad remote format " + fetchSpec);
+
+	}
+}
diff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/Repository.java b/org.spearce.jgit/src/org/spearce/jgit/lib/Repository.java
index b3fa12e..821633d 100644
--- a/org.spearce.jgit/src/org/spearce/jgit/lib/Repository.java
+++ b/org.spearce.jgit/src/org/spearce/jgit/lib/Repository.java
@@ -1132,4 +1132,16 @@ public class Repository {
 		relName = relName.replace(File.separatorChar, '/');
 		return relName;
 	}
+
+	/**
+	 * @param name
+	 *            The "remote" name in this repo
+	 * @return information about how a remote repository is beging tracked
+	 */
+	public RemoteSpec getRemoteSpec(String name) {
+		String url = getConfig().getString("remote."+name, null, "url");
+		String fetchPattern = getConfig().getString("remote."+name, null, "fetch");
+		String pushPattern = getConfig().getString("remote."+name, null, "push");
+		return new RemoteSpec(name, url, fetchPattern, pushPattern);
+	}
 }
-- 
1.5.4.3

Re: [jgit] index v2 pull request

[Shawn O. Pearce]

Robin Rosenberg <robin.rosenberg@dewire.com> wrote:
> Den Wednesday 12 March 2008 03.52.05 skrev Imran M Yousuf:
> 
> > I am currently looking into .git/config format as Shawn suggested.
> 
> That's a good one too. I have code for parsing the remotes specs, though
> e.g. "+refs/heads/master/*:refs/remotes/origin/*", but nothing for the "branch" config. I haven't used it yet, so it's a rip-out.	

Your RemoteSpec is missing the fact that some people have more than
one URL, fetch, and/or push lines in the same remote.  :-)
 
> + * <pre>
> + *  [remote &quot;name&quot;]
> + *     url = URL:ish
> + *     fetch = [+]remoteref:localref
> + * </pre>
> + *
> + * There are more variants but we do not support them here yet.

Or is that what you mean by the above statement?

In any case the remote spec needs a list of URLs a list of the
fetch/push Info objects.  That changes the public API of the
RemoteSpec class rather significantly.

> +public class RemoteSpec {
> +
> +	static class Info {
> +		boolean overwriteAlways;
> +
> +		boolean matchAny;
> +
> +		String remoteRef;
> +
> +		String localRef;
> +	}
> +
> +	Info fetch = new Info();
> +
> +	Info push = null;
> +
> +	private final String name;
> +
> +	private final String url;

-- 
Shawn.

Re: [jgit] index v2 pull request

[Imran M Yousuf]

On Wed, Mar 12, 2008 at 1:07 PM, Robin Rosenberg
<robin.rosenberg@dewire.com> wrote:
> Den Wednesday 12 March 2008 03.52.05 skrev Imran M Yousuf:
>
> > Maven: Does this mean mavenizing the project? If so I would start it
>  > this weekend. If you were referring to GIT Maven SCM please also let
>  > me know.
>  Not to build using, but create SCM functions for those so thay can use
>  git without <exec> tasks. Git isn't available in any of those, while much
>  worse SCM's have support.
>

I know a team who are currently working with the GIT SCM plugin for
Maven; I think they are using CLI for the purpose. I am more
interested in GIT VCS Module for NB6; as I have some other projects
which will be using it :).

>
>  > I am currently looking into .git/config format as Shawn suggested.
>
>  That's a good one too. I have code for parsing the remotes specs, though
>  e.g. "+refs/heads/master/*:refs/remotes/origin/*", but nothing for the "branch" config. I haven't used it yet, so it's a rip-out.

I will have a look at it; currently we have a long tasklist in a
project; I will have a look at them as soon as I can make some time
:).

- Imran

>
>  -- robin
>
>  From 042fa0bef8b5d3ec8f5b1e385766ca61528e72aa Mon Sep 17 00:00:00 2001
>  From: Robin Rosenberg <robin.rosenberg@dewire.com>
>  Date: Wed, 5 Mar 2008 23:27:57 +0100
>  Subject: [PATCH] Add support for parsing remote specs
>
>  Signed-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>
>  ---
>   .../tst/org/spearce/jgit/lib/RemoteSpecTest.java   |   95 +++++++++++++
>   .../src/org/spearce/jgit/lib/RemoteSpec.java       |  142 ++++++++++++++++++++
>   .../src/org/spearce/jgit/lib/Repository.java       |   12 ++
>   3 files changed, 249 insertions(+), 0 deletions(-)
>   create mode 100644 org.spearce.jgit.test/tst/org/spearce/jgit/lib/RemoteSpecTest.java
>   create mode 100644 org.spearce.jgit/src/org/spearce/jgit/lib/RemoteSpec.java
>
>  diff --git a/org.spearce.jgit.test/tst/org/spearce/jgit/lib/RemoteSpecTest.java b/org.spearce.jgit.test/tst/org/spearce/jgit/lib/RemoteSpecTest.java
>  new file mode 100644
>  index 0000000..817e697
>  --- /dev/null
>  +++ b/org.spearce.jgit.test/tst/org/spearce/jgit/lib/RemoteSpecTest.java
>  @@ -0,0 +1,95 @@
>  +/*
>  + *  Copyright (C) 2007  Robin Rosenberg
>  + *
>  + *  This library is free software; you can redistribute it and/or
>  + *  modify it under the terms of the GNU General Public
>  + *  License, version 2, as published by the Free Software Foundation.
>  + *
>  + *  This library is distributed in the hope that it will be useful,
>  + *  but WITHOUT ANY WARRANTY; without even the implied warranty of
>  + *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
>  + *  General Public License for more details.
>  + *
>  + *  You should have received a copy of the GNU General Public
>  + *  License along with this library; if not, write to the Free Software
>  + *  Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301
>  + */
>  +package org.spearce.jgit.lib;
>  +
>  +import java.io.File;
>  +import java.io.FileOutputStream;
>  +
>  +/**
>  + * Test parsing of git remotes
>  + */
>  +public class RemoteSpecTest extends RepositoryTestCase {
>  +
>  +       /**
>  +        * Test simplest case
>  +        *
>  +        * @throws Exception
>  +        *
>  +        */
>  +       public void testSimplestOk() throws Exception {
>  +               RemoteSpec spec = new RemoteSpec("their", "git://foo.bar/zip.git",
>  +                               "refs/heads/master:refs/heads/origin",null);
>  +               assertEquals("refs/heads/master", spec.getFetchRemoteRef());
>  +               assertEquals("refs/heads/origin", spec.getFetchLocalRef());
>  +               assertFalse(spec.isFetchMatchAny());
>  +               assertFalse(spec.isFetchOverwriteAlways());
>  +       }
>  +
>  +       /**
>  +        * Test a standard case
>  +        *
>  +        * @throws Exception
>  +        */
>  +       public void testStandardOk() throws Exception {
>  +               RemoteSpec spec = new RemoteSpec("their", "git://example.com/zip.git",
>  +                               "+refs/heads/master/*:refs/remotes/origin/*",null);
>  +               assertEquals("git://example.com/zip.git", spec.getUrl());
>  +               assertEquals("refs/heads/master", spec.getFetchRemoteRef());
>  +               assertEquals("refs/remotes/origin", spec.getFetchLocalRef());
>  +               assertTrue(spec.isFetchMatchAny());
>  +               assertTrue(spec.isFetchOverwriteAlways());
>  +       }
>  +
>  +       /**
>  +        * Test a <quote>safer</quote> almost standard case
>  +        *
>  +        * @throws Exception
>  +        */
>  +       public void testNonStandardSaferOk() throws Exception {
>  +               RemoteSpec spec = new RemoteSpec("their", "git://example.com/zip.git",
>  +                               "refs/heads/master/*:refs/remotes/origin/*",null);
>  +               assertEquals("git://example.com/zip.git", spec.getUrl());
>  +               assertEquals("refs/heads/master", spec.getFetchRemoteRef());
>  +               assertEquals("refs/remotes/origin", spec.getFetchLocalRef());
>  +               assertTrue(spec.isFetchMatchAny());
>  +               assertFalse(spec.isFetchOverwriteAlways());
>  +       }
>  +
>  +       /**
>  +        * Test a case copied from a real Git repo
>  +        *
>  +        * @throws Exception
>  +        */
>  +       public void testReadFromConfig() throws Exception {
>  +               File file = new File(db.getDirectory(),"config");
>  +               FileOutputStream stream = new FileOutputStream(file,true);
>  +               try {
>  +                       stream.write(("[remote \"spearce\"]\n"+
>  +               "url = http://www.spearce.org/projects/scm/egit.git\n"+
>  +               "fetch = +refs/heads/*:refs/remotes/spearce/*\n").getBytes());
>  +               } finally {
>  +                       stream.close();
>  +               }
>  +               db.getConfig().load();
>  +               RemoteSpec remoteSpec = db.getRemoteSpec("spearce");
>  +               assertEquals("http://www.spearce.org/projects/scm/egit.git", remoteSpec.getUrl());
>  +               assertEquals("refs/heads", remoteSpec.getFetchRemoteRef());
>  +               assertEquals("refs/remotes/spearce", remoteSpec.getFetchLocalRef());
>  +               assertTrue(remoteSpec.isFetchMatchAny());
>  +               assertTrue(remoteSpec.isFetchOverwriteAlways());
>  +       }
>  +}
>  diff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/RemoteSpec.java b/org.spearce.jgit/src/org/spearce/jgit/lib/RemoteSpec.java
>  new file mode 100644
>  index 0000000..a6a3b6c
>  --- /dev/null
>  +++ b/org.spearce.jgit/src/org/spearce/jgit/lib/RemoteSpec.java
>  @@ -0,0 +1,142 @@
>  +/*
>  + *  Copyright (C) 2007  Robin Rosenberg
>  + *
>  + *  This library is free software; you can redistribute it and/or
>  + *  modify it under the terms of the GNU General Public
>  + *  License, version 2, as published by the Free Software Foundation.
>  + *
>  + *  This library is distributed in the hope that it will be useful,
>  + *  but WITHOUT ANY WARRANTY; without even the implied warranty of
>  + *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
>  + *  General Public License for more details.
>  + *
>  + *  You should have received a copy of the GNU General Public
>  + *  License along with this library; if not, write to the Free Software
>  + *  Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301
>  + */
>  +package org.spearce.jgit.lib;
>  +
>  +/**
>  + * Information about how to synchronize with a remote Git repository.
>  + *
>  + * A remote is stored in the <GIT_DIR>/config as
>  + *
>  + * <pre>
>  + *  [remote &quot;name&quot;]
>  + *     url = URL:ish
>  + *     fetch = [+]remoteref:localref
>  + * </pre>
>  + *
>  + * There are more variants but we do not support them here yet.
>  + */
>  +public class RemoteSpec {
>  +
>  +       static class Info {
>  +               boolean overwriteAlways;
>  +
>  +               boolean matchAny;
>  +
>  +               String remoteRef;
>  +
>  +               String localRef;
>  +       }
>  +
>  +       Info fetch = new Info();
>  +
>  +       Info push = null;
>  +
>  +       private final String name;
>  +
>  +       private final String url;
>  +
>  +       /**
>  +        * @return name of remote. This is a local short identifier
>  +        */
>  +       public String getName() {
>  +               return name;
>  +       }
>  +
>  +       /**
>  +        * @return the URL:ish location of the remote Git repository
>  +        */
>  +       public String getUrl() {
>  +               return url;
>  +       }
>  +
>  +       /**
>  +        * @return the local ref part used for fetch heads info
>  +        */
>  +       public String getFetchLocalRef() {
>  +               return fetch.localRef;
>  +       }
>  +
>  +       /**
>  +        * @return the remote ref part used for fetching refs from the remote repo
>  +        */
>  +       public String getFetchRemoteRef() {
>  +               return fetch.remoteRef;
>  +       }
>  +
>  +       /**
>  +        * @return whether the fetch matches all branches under the ref or just the
>  +        *         named ref
>  +        */
>  +       public boolean isFetchMatchAny() {
>  +               return fetch.matchAny;
>  +       }
>  +
>  +       /**
>  +        * @return whether the tracking branch is always updated, or only when the
>  +        *         update is a fast forward
>  +        */
>  +       public boolean isFetchOverwriteAlways() {
>  +               return fetch.overwriteAlways;
>  +       }
>  +
>  +       /**
>  +        * Create a representation of a git remote specification.
>  +        *
>  +        * @param name A local short identifier
>  +        * @param url The URL:ish used for fetching / pushing
>  +        * @param fetchPattern refspec for fetching
>  +        * @param pushPattern refspec for pushing or null
>  +        */
>  +       public RemoteSpec(String name, String url, String fetchPattern,
>  +                       String pushPattern) {
>  +               this.name = name;
>  +               this.url = url;
>  +               parse(fetchPattern, fetch);
>  +               if (pushPattern != null) {
>  +                       push = new Info();
>  +                       parse(pushPattern, push);
>  +               }
>  +       }
>  +
>  +       private void parse(String fetchSpec, Info info) {
>  +               int p = 0;
>  +               if (fetchSpec.charAt(p) == '+') {
>  +                       info.overwriteAlways = true;
>  +                       ++p;
>  +               }
>  +               int cp = fetchSpec.indexOf(':');
>  +               if (cp < 0)
>  +                       throw new IllegalArgumentException("Bad remote format " + fetchSpec);
>  +               info.remoteRef = fetchSpec.substring(p, cp);
>  +               info.localRef = fetchSpec.substring(cp + 1);
>  +               if (info.remoteRef.endsWith("/*")) {
>  +                       info.matchAny = true;
>  +                       info.remoteRef = info.remoteRef.substring(0, info.remoteRef
>  +                                       .length() - 2);
>  +               }
>  +               if (info.localRef.endsWith("/*")) {
>  +                       if (!info.matchAny)
>  +                               throw new IllegalArgumentException("Bad remote format "
>  +                                               + fetchSpec);
>  +                       info.localRef = info.localRef.substring(0,
>  +                                       info.localRef.length() - 2);
>  +               } else
>  +                       if (info.matchAny)
>  +                               throw new IllegalArgumentException("Bad remote format " + fetchSpec);
>  +
>  +       }
>  +}
>  diff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/Repository.java b/org.spearce.jgit/src/org/spearce/jgit/lib/Repository.java
>  index b3fa12e..821633d 100644
>  --- a/org.spearce.jgit/src/org/spearce/jgit/lib/Repository.java
>  +++ b/org.spearce.jgit/src/org/spearce/jgit/lib/Repository.java
>  @@ -1132,4 +1132,16 @@ public class Repository {
>                 relName = relName.replace(File.separatorChar, '/');
>                 return relName;
>         }
>  +
>  +       /**
>  +        * @param name
>  +        *            The "remote" name in this repo
>  +        * @return information about how a remote repository is beging tracked
>  +        */
>  +       public RemoteSpec getRemoteSpec(String name) {
>  +               String url = getConfig().getString("remote."+name, null, "url");
>  +               String fetchPattern = getConfig().getString("remote."+name, null, "fetch");
>  +               String pushPattern = getConfig().getString("remote."+name, null, "push");
>  +               return new RemoteSpec(name, url, fetchPattern, pushPattern);
>  +       }
>   }
>  --
>  1.5.4.3
>
>



-- 
Imran M Yousuf
Entrepreneur & Software Engineer
Smart IT Engineering
Dhaka, Bangladesh
Email: imran@smartitengineering.com
Mobile: +880-1711402557

Re: [jgit] index v2 pull request

[Shawn O. Pearce]

Imran M Yousuf <imyousuf@gmail.com> wrote:
> I would like to volunteer to work in JGit;
> can someone let me know where I can pick some tasks to implement?

One of the areas that I think is really weak in JGit and that we
need to do a _good_ fetch/push implementation is the branch model.

For example, if you look at Repository.getBranches() it returns
to the caller a Collection<String>.

But in Git a branch is a heck of a lot more data, and that data is
relevant to the end user when we are talking about fetch and merge.
There are many configuration options stored in .git/config for a
branch, and these are (today) created by git-branch and git-checkout
automatically as the user creates and deletes branches.

JGit has none of this model.  It thinks all that a branch is is a
String.  Sad.

Another thing that bothers me is the packed refs cache.  We toss
away the peeled information (the "^" lines), but that data is very
useful when you are talking about fetch as well as a few other
types of operations, like plotting tag labels onto a history graph
(such as how gitk does it).

The Ref class was started as a means of wrapping up the various
important bits of data about a ref (of which branches are a subclass
and annotated tags with peeled data is another) but I suspect it
didn't quite do everything so these string APIs got created.

The above is a farily small task, one that any good OO programmer
should be able to tackle, but it can be challenging if you are
new to Git plumbing as you'll have to learn what are the important
parts of a branch/ref/annotated tag.

As a starting point look at the config file format in C Git and
see what is possible in a "branch" and "remote" section; that
data needs to be pretty readily available for any Ref, if such
data exists.  The backward mapping of Ref->Remote (which does
not exist yet, unless Robin created it) is quite relevant for
tracking branches.

-- 
Shawn.

Re: [jgit] index v2 pull request

[Imran M Yousuf]

On Tue, Mar 11, 2008 at 6:35 AM, Shawn O. Pearce <spearce@spearce.org> wrote:
> Imran M Yousuf <imyousuf@gmail.com> wrote:
>  > I would like to volunteer to work in JGit;
>  > can someone let me know where I can pick some tasks to implement?
>
>  One of the areas that I think is really weak in JGit and that we
>  need to do a _good_ fetch/push implementation is the branch model.
>
>  For example, if you look at Repository.getBranches() it returns
>  to the caller a Collection<String>.
>
>  But in Git a branch is a heck of a lot more data, and that data is
>  relevant to the end user when we are talking about fetch and merge.
>  There are many configuration options stored in .git/config for a
>  branch, and these are (today) created by git-branch and git-checkout
>  automatically as the user creates and deletes branches.
>
>  JGit has none of this model.  It thinks all that a branch is is a
>  String.  Sad.
>
>  Another thing that bothers me is the packed refs cache.  We toss
>  away the peeled information (the "^" lines), but that data is very
>  useful when you are talking about fetch as well as a few other
>  types of operations, like plotting tag labels onto a history graph
>  (such as how gitk does it).
>
>  The Ref class was started as a means of wrapping up the various
>  important bits of data about a ref (of which branches are a subclass
>  and annotated tags with peeled data is another) but I suspect it
>  didn't quite do everything so these string APIs got created.
>
>  The above is a farily small task, one that any good OO programmer
>  should be able to tackle, but it can be challenging if you are
>  new to Git plumbing as you'll have to learn what are the important
>  parts of a branch/ref/annotated tag.
>
>  As a starting point look at the config file format in C Git and
>  see what is possible in a "branch" and "remote" section; that
>  data needs to be pretty readily available for any Ref, if such
>  data exists.  The backward mapping of Ref->Remote (which does
>  not exist yet, unless Robin created it) is quite relevant for
>  tracking branches.

I will have a look at them and will give you and update; if I have any
questions, I hope you and team wont mind :).

>
>  --
>  Shawn.
>



-- 
Imran M Yousuf
Entrepreneur & Software Engineer
Smart IT Engineering
Dhaka, Bangladesh
Email: imran@smartitengineering.com
Mobile: +880-1711402557

Re: [jgit] index v2 pull request

[Shawn O. Pearce]

Robin Rosenberg <robin.rosenberg@dewire.com> wrote:
> Den Saturday 08 March 2008 03.50.27 skrev Shawn O. Pearce:
> 
> > We now actually use the following .git/config options to control
> > how the window cache limits work:
> >
> >   core.packedGitWindowSize
> >   core.packedGitLimit
> >   core.deltaBaseCacheLimit
> >   core.packedGitMMAP
> 
> Will the net effect, in practice, be the same on Jgit as on C Git?

Yes.  The options actually translate precisely into jgit and are
the exact same knobs as they are in C Git.  That shouldn't be a
surprise as the damn knobs were written in both places by the
same person (me).

Under an Eclipse workspace we read these options _only_ from the
user's ~/.gitconfig.  The values in the repository config file are
not honored as there is only one window cache and one delta base
case for the entire Eclipse workspace.  Given that all projects
in the workspace must share the same virtual address space this is
the only sane configuration.

> >   repo.or.cz:/srv/git/egit/spearce.git master
> Something missing in the formatting?

Heh.  That's the SSH URL.  I ran git-request-pull against my
remote rather than the public git:// URL for the repository.
Sorry about that.

> >       Stop creating "new format" loose objects as C Git no longer likes them
>
> And there goes the unit test for reading them...

Uhm, yeaaah.  I forgot about that.  Actually that's why the object
size changed in that test that I deleted the assert from.  It was
no longer a new-style loose object and thus had to use a few extra
bytes for the type code.  *sigh*

I am pretty sure that C Git also has the same need-egg-no-chicken
problem we now have in jgit.  It has code to read a new style
loose object but has no code to actually create those anymore.
So it cannot actually do a test for the new style reader either.
 
> >       Don't make unit tests depend upon compressed file lengths

Yea, its this commit I was just talking about above.  Needing to
delete this length assert was because of the new format breakage
above.

> >       Extract StGitPatch from Repository to its own top-level type
>
> I'm considering dropping stacked git support. We should have much
> more of it and I don't use it myself anymore to what support we have
> gets very little testing.

I've never used it.  I played around with it in my early git days,
then got into the porcelain and plumbing business with pg, git-gui,
jgit, egit... ;-)
 
> >       Refactor RepositoryState into its own top-level type
> \ No newline at end of file

Arrrrrrrgh.  Damnit Eclipse!  Do what I want, not what you want!
I'll fix it with a follow-up commit.  Unless you really want me
to rewrite that branch.  I'm suspecting not as you are talking
about pushing it below.

> >       Use Java 5 bit twiddling APIs rather than hand-rolling bit counting
>
> I'm surprised *you* didn't know about them bitcounting algorithm java uses 
> before. It's pretty common.

Heh.  Yea, I was surprised too.  Its fixed now.  ;-)
 
> >       Remove unnecessary bitwise AND masks in readTree 
> Good catch
> >       Precount the number of tree entries during readTree
> this one too

These were micro-optimizations while I was trying to figure out
why the hell the History view cost so much.  We spend about 95%
of our time in readTree.  With the above changes we _still_ spend
about 95% of our time in readTree.

I'm rewriting things so we don't do that.  We'll see how well
that works for me.

> >       Change the ObjectLoader.getType method to return int
> >       Fix ArrayIndexOutOfBoundsException in WindowCache
> 
> Very good work (as always) Shawn. I've only read the code so far. I'll do some 
> testing too before pushing. Do you have any unit tests for the v2 pack format 
> (config reading too for that matter)?

Nope, I didn't write any yet.  :-)

A good v2 pack index test probably requires having a v2 pack
index writer.  We don't have that yet.  Plus its a "which is
correct?" game.

I did manually test the pack index v2 reader by pulling history from
a tree that I know for a fact was fully repacked (nothing loose)
and fully repacked with index v2 (I double checked the header
with dd|od).

What I did fail to test was the 64 bit offset table.  Its a pita as
nobody has a packfile that large.  Nico left some back door debugging
options in index-pack to generate an index with a mixture of offset
types but I forgot to manually rebuild my .idx file with it.

-- 
Shawn.