jgit performance update

jgit performance update

[Shawn Pearce]

With the help of Robin Rosenberg I've been able to make jgit's log
operation run (on average) within a few milliseconds of core Git.

Walking the 50,000 most recent commits from the Mozilla trunk[1]:

  $ time git rev-list --max-count=50000 HEAD >/dev/null

  core Git:  1.882s (average)
  jgit:      1.932s (average)

  (times are with hot cache and from repeated executions)

I think that is actually pretty good given that jgit is written
in Java using a fairly object-oriented design and has to deal with
some of the limitations of the language.

One of the biggest annoyances has been the fact that although Java
1.4 offers a way to mmap a file into the process, the overhead to
access that data seems to be far higher than just reading the file
content into a very large byte array, especially if we are going
to access that file content multiple times.  So jgit performs worse
than core Git early on while it copies everything from the OS buffer
cache into the Java process, but then performs reasonably well once
the internal cache is hot.  On the other hand using the mmap call
reduces early latency but hurts the access times so much that we're
talking closer to 3s average read times for the same log operation.

Anyway, jgit is now hopefully fast enough that we can start to build
some real functionality on top of it, and not need to wait several
minutes for answers from those features while debugging them.  :)


**1** This is the pack file from Jon Smirl's import attempt.

-- 

Re: jgit performance update

[Robin Rosenberg]

söndag 03 december 2006 05:59 skrev Shawn Pearce:
> With the help of Robin Rosenberg I've been able to make jgit's log
> operation run (on average) within a few milliseconds of core Git.
>
> Walking the 50,000 most recent commits from the Mozilla trunk[1]:
>
>   $ time git rev-list --max-count=50000 HEAD >/dev/null
>
>   core Git:  1.882s (average)
>   jgit:      1.932s (average)
>
>   (times are with hot cache and from repeated executions)
Nice indeed. That was a ten-fold improvement for getting my full history. 

So, just go on to the next case. I added filtering on filenames (yes, 
CVS-induced brain damage, I should track the content. next version. filenames 
are so much handier to work with). That gives me 4.5s to retrieve a filtered 
history (from 10800 commits).Half of the time is spent in re-sorting tree 
entries. Is that really necessary?

> I think that is actually pretty good given that jgit is written
> in Java using a fairly object-oriented design and has to deal with
> some of the limitations of the language.
Most of java's slowness comes from the programmers using it. (Lutz Prechelt. 
Technical opinion: comparing Java vs. C/C++ efficiency differences to 
interpersonal differences. ACM, Vol 42,#10, 1999)

> One of the biggest annoyances has been the fact that although Java 
> 1.4 offers a way to mmap a file into the process, the overhead to
> access that data seems to be far higher than just reading the file
> content into a very large byte array, especially if we are going
> to access that file content multiple times.  So jgit performs worse
> than core Git early on while it copies everything from the OS buffer
> cache into the Java process, but then performs reasonably well once
> the internal cache is hot.  On the other hand using the mmap call
> reduces early latency but hurts the access times so much that we're
> talking closer to 3s average read times for the same log operation.

Have you tried that with difference JVM's?

Re: jgit performance update

[Jakub Narebski]

Robin Rosenberg wrote:

> söndag 03 december 2006 05:59 skrev Shawn Pearce:
>
>> With the help of Robin Rosenberg I've been able to make jgit's log
>> operation run (on average) within a few milliseconds of core Git.
>>
>> Walking the 50,000 most recent commits from the Mozilla trunk[1]:
>>
>>   $ time git rev-list --max-count=50000 HEAD >/dev/null
>>
>>   core Git:  1.882s (average)
>>   jgit:      1.932s (average)
>>
>>   (times are with hot cache and from repeated executions)
> Nice indeed. That was a ten-fold improvement for getting my full history. 
> 
> So, just go on to the next case. I added filtering on filenames (yes, 
> CVS-induced brain damage, I should track the content. next version. filenames 
> are so much handier to work with).

Git uses <path> as _revision limiter_, not as output filter. Shouldn't
jgit do the same?

P.S. What is the status of --follow option?
-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git

Re: jgit performance update

[Robin Rosenberg]

söndag 03 december 2006 15:19 skrev Jakub Narebski:
> Robin Rosenberg wrote:
> > CVS-induced brain damage, I should track the content. next version.
> > filenames are so much handier to work with).
>
> Git uses <path> as _revision limiter_, not as output filter. Shouldn't
> jgit do the same?
It's egit, i.e. the eclipse plugin I'm referring to so it's a user interface 
thing and it uses the path name.  

Re: jgit performance update

[Shawn Pearce]

Robin Rosenberg <robin.rosenberg.lists@dewire.com> wrote:
> söndag 03 december 2006 15:19 skrev Jakub Narebski:
> > Robin Rosenberg wrote:
> > > CVS-induced brain damage, I should track the content. next version.
> > > filenames are so much handier to work with).
> >
> > Git uses <path> as _revision limiter_, not as output filter. Shouldn't
> > jgit do the same?
> It's egit, i.e. the eclipse plugin I'm referring to so it's a user interface 
> thing and it uses the path name.  

Jakub's point is that "git log -- a" lists only the revisions
which affect path 'a'.  Once those revisions have been selected
then output begins.  The 'a' gets reapplied as an output filter to
only show changes relevant to file 'a', but its very much a revision
filter during the revision listing process.

-- 

Re: jgit performance update

[Shawn Pearce]

Robin Rosenberg <robin.rosenberg.lists@dewire.com> wrote:
> So, just go on to the next case. I added filtering on filenames (yes, 
> CVS-induced brain damage, I should track the content. next version. filenames 
> are so much handier to work with). That gives me 4.5s to retrieve a filtered 
> history (from 10800 commits).Half of the time is spent in re-sorting tree 
> entries. Is that really necessary?

Yea, I was looking at that code while doing the other performance
improvements and thought it might start to become a bottleneck. I
guess I was right.

What is happening here is jgit wants to store the items in the tree
in name ordering, but Git stores the items in the tree sorted such
that subtrees sort with a '/' on the end of their name.  This is a
different ordering...

The reason I'm resorting them is so we can find an entry without
knowing what its type is first.  Looks like that's going to have
to change somehow.
 
> Most of java's slowness comes from the programmers using it. (Lutz Prechelt. 
> Technical opinion: comparing Java vs. C/C++ efficiency differences to 
> interpersonal differences. ACM, Vol 42,#10, 1999)

Yes, that was clearly the case here with jgit!  :-)

_This_ programmer made jgit slow.  Learned from the mistake, and
made it faster.
 
> > One of the biggest annoyances has been the fact that although Java 
> > 1.4 offers a way to mmap a file into the process, the overhead to
> > access that data seems to be far higher than just reading the file
> > content into a very large byte array, especially if we are going
> > to access that file content multiple times.  So jgit performs worse
> > than core Git early on while it copies everything from the OS buffer
> > cache into the Java process, but then performs reasonably well once
> > the internal cache is hot.  On the other hand using the mmap call
> > reduces early latency but hurts the access times so much that we're
> > talking closer to 3s average read times for the same log operation.
> 
> Have you tried that with difference JVM's?

No, I'm on Mac OS X so I don't have a huge JVM selection (that I
know of).  And I haven't tried jgit or egit on any other system yet.

-- 

Re: jgit performance update

[Linus Torvalds]

On Sat, 2 Dec 2006, Shawn Pearce wrote:
>
> With the help of Robin Rosenberg I've been able to make jgit's log
> operation run (on average) within a few milliseconds of core Git.

Very good. Are we any closer to actually having an eclipse plugin then?

Not that I've ever actually used eclipse, but maybe I should try it, just 
to see what those strange user-land people actually do. I'll be a 
veritable Jane Goodall..

> Walking the 50,000 most recent commits from the Mozilla trunk[1]:
> 
>   $ time git rev-list --max-count=50000 HEAD >/dev/null
> 
>   core Git:  1.882s (average)
>   jgit:      1.932s (average)
> 
>   (times are with hot cache and from repeated executions)

Now, the _interesting_ case in many ways is not "--max-count", but the 
revision limiter. It _should_ be equally fast, but if you've done 
something wrong, it won't be.

IOW, try to find a point far enough back in time to get about the same 
number of commits, and then do

	time git rev-list <thatpoint>..HEAD >/dev/null

because one of the things you want to handle is ranges, more so than 
simple counts. And that is not only the much more common case, it also 
triggers a few cases that you probably didn't trigger with the regular 
"list the first 50 thousand commits" case.

> One of the biggest annoyances has been the fact that although Java
> 1.4 offers a way to mmap a file into the process, the overhead to
> access that data seems to be far higher than just reading the file
> content into a very large byte array, especially if we are going
> to access that file content multiple times.

That must suck for big packed repositories. What JVM and other environment 
are you using?

Also, I have to say, one of the reasons I'm interested in your project is 
that I've never done any Java programming, because quite frankly, I've 
never had any reason what-so-ever to do so. But if there is some simple 
setup, and you have jgit exposed somewhere as a git archive, I'd love to 
take a look, if only to finally learn more about Java.

Re: jgit performance update

[Jakub Narebski]

Linus Torvalds wrote:

> Also, I have to say, one of the reasons I'm interested in your project is 
> that I've never done any Java programming, because quite frankly, I've 
> never had any reason what-so-ever to do so. But if there is some simple 
> setup, and you have jgit exposed somewhere as a git archive, I'd love to 
> take a look, if only to finally learn more about Java.

GitWiki tells us about egit/jgit repository at
  http://www.spearce.org/projects/scm/egit.git
-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git

Re: jgit performance update

[Juergen Stuber]

Hi Jakub,

Jakub Narebski <jnareb@gmail.com> writes:
>
> GitWiki tells us about egit/jgit repository at
>   http://www.spearce.org/projects/scm/egit.git

I tried to access that with git 1.4.4.1 from Debian but 

% git clone http://www.spearce.org/projects/scm/egit.git

hangs, the first time after "walk e339766abc2b919e7bb396cae22ddef065821381",
the second time after "walk 9eec90ec5da239e063eaff6305d77294dc03396e"
which is the "walk" line just before it.

There's also the following error shortly after the start:

error: File bc01ab9e5fcd26918d7a334207183fa57ff1ce50 (http://www.spearce.org/projects/scm/egit.git/objects/75/1c8f2e504c40d1c41ebbd87d8f8968529e9c30) corrupt


Jürgen

-- 
Jürgen Stuber <juergen@jstuber.net>
http://www.jstuber.net/
gnupg key fingerprint = 2767 CA3C 5680 58BA 9A91  23D9 BED6 9A7A AF9E 68B4

Re: jgit performance update

[Robin Rosenberg]

söndag 03 december 2006 23:42 skrev Juergen Stuber:
> Hi Jakub,
>
> Jakub Narebski <jnareb@gmail.com> writes:
> > GitWiki tells us about egit/jgit repository at
> >   http://www.spearce.org/projects/scm/egit.git
>
> I tried to access that with git 1.4.4.1 from Debian but
>
> % git clone http://www.spearce.org/projects/scm/egit.git
>
> hangs, the first time after "walk
> e339766abc2b919e7bb396cae22ddef065821381", the second time after "walk
> 9eec90ec5da239e063eaff6305d77294dc03396e" which is the "walk" line just
> before it.
Works fine here. (git 1.4.4.gf05d).
>
> There's also the following error shortly after the start:
>
> error: File bc01ab9e5fcd26918d7a334207183fa57ff1ce50
> (http://www.spearce.org/projects/scm/egit.git/objects/75/1c8f2e504c40d1c41e
>bbd87d8f8968529e9c30) corrupt

Unfortunately, messages about corrupt objects are "normal" with clone over 
http. I'm not sure it has to be that way though. Run git-fsck-objects to make 
sure there are no errors. The hangs aren't normal.

-- robin

Re: jgit performance update

[Jakub Narebski]

Robin Rosenberg wrote:

> söndag 03 december 2006 23:42 skrev Juergen Stuber:
>>
>> Jakub Narebski <jnareb@gmail.com> writes:
>>>
>>> GitWiki tells us about egit/jgit repository at
>>>   http://www.spearce.org/projects/scm/egit.git
>>
>> I tried to access that with git 1.4.4.1 from Debian but
>>
>> % git clone http://www.spearce.org/projects/scm/egit.git
>>
>> hangs, the first time after "walk
>> e339766abc2b919e7bb396cae22ddef065821381", the second time after "walk
>> 9eec90ec5da239e063eaff6305d77294dc03396e" which is the "walk" line just
>> before it.
>
> Works fine here. (git 1.4.4.gf05d).
Works fine here. (git 1.4.4.1)

>> There's also the following error shortly after the start:
>>
>> error: File bc01ab9e5fcd26918d7a334207183fa57ff1ce50
>> (http://www.spearce.org/projects/scm/egit.git/objects/75/1c8f2e504c40d1c41e
>>bbd87d8f8968529e9c30) corrupt
> 
> Unfortunately, messages about corrupt objects are "normal" with clone over 
> http. I'm not sure it has to be that way though. Run git-fsck-objects to make 
> sure there are no errors. The hangs aren't normal.

I got:
$ git clone http://www.spearce.org/projects/scm/egit.git
[...]
 got 73ed47b2bb1fa5978f7368775979e5c85d354c5a
 error: File 2332eacf114debb7a27d138811197f06eb262551 
 (http://www.spearce.org/projects/scm/egit.git/objects/75/1c8f2e504c40d1c41ebbd87d8f8968529e9c30) corrupt
 Getting pack list for http://www.spearce.org/projects/scm/egit.git/
 got afefbe09bacc08adb75fb46200a973001c6b02de
[...]
 walk c1f287cb19b9910af19756cf29c08b1fda75da8c
 Some loose object were found to be corrupt, but they might be just
 a false '404 Not Found' error message sent with incorrect HTTP
 status code.  Suggest running git fsck-objects.
 got eab86de8ac23e2e77878835007724146fdd83796
$ git fsck-objects --unreachable --full --strict   ;# returns no errors

-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git

Re: jgit performance update

[Shawn Pearce]

Jakub Narebski <jnareb@gmail.com> wrote:
> I got:
> $ git clone http://www.spearce.org/projects/scm/egit.git
> [...]
>  got 73ed47b2bb1fa5978f7368775979e5c85d354c5a
>  error: File 2332eacf114debb7a27d138811197f06eb262551 
>  (http://www.spearce.org/projects/scm/egit.git/objects/75/1c8f2e504c40d1c41ebbd87d8f8968529e9c30) corrupt
>  Getting pack list for http://www.spearce.org/projects/scm/egit.git/
>  got afefbe09bacc08adb75fb46200a973001c6b02de
> [...]
>  walk c1f287cb19b9910af19756cf29c08b1fda75da8c
>  Some loose object were found to be corrupt, but they might be just
>  a false '404 Not Found' error message sent with incorrect HTTP
>  status code.  Suggest running git fsck-objects.
>  got eab86de8ac23e2e77878835007724146fdd83796
> $ git fsck-objects --unreachable --full --strict   ;# returns no errors

Right.  My web server doesn't send back 404 messages when an object
isn't found (but is packed).  Thus the error...

I've setup a project on repo.or.cz:

  http://repo.or.cz/w/egit.git


-- 

Re: jgit performance update

[Juergen Stuber]

Hej Robin,

Robin Rosenberg <robin.rosenberg.lists@dewire.com> writes:
> söndag 03 december 2006 23:42 skrev Juergen Stuber:
>> Jakub Narebski <jnareb@gmail.com> writes:
>> > GitWiki tells us about egit/jgit repository at
>> >   http://www.spearce.org/projects/scm/egit.git
>>
>> I tried to access that with git 1.4.4.1 from Debian but
>>
>> % git clone http://www.spearce.org/projects/scm/egit.git
>>
>> hangs, the first time after "walk
>> e339766abc2b919e7bb396cae22ddef065821381", the second time after "walk
>> 9eec90ec5da239e063eaff6305d77294dc03396e" which is the "walk" line just
>> before it.
> Works fine here. (git 1.4.4.gf05d).

now it works fine for me, too.


Tack

Jürgen

-- 
Jürgen Stuber <juergen@jstuber.net>
http://www.jstuber.net/
gnupg key fingerprint = 2767 CA3C 5680 58BA 9A91  23D9 BED6 9A7A AF9E 68B4

Re: jgit performance update

[Shawn Pearce]

Linus Torvalds <torvalds@osdl.org> wrote:
> Very good. Are we any closer to actually having an eclipse plugin then?

We have some parts in place.  But nothing that's user-ready.
 
> > Walking the 50,000 most recent commits from the Mozilla trunk[1]:
> 
> Now, the _interesting_ case in many ways is not "--max-count", but the 
> revision limiter. It _should_ be equally fast, but if you've done 
> something wrong, it won't be.

We haven't implemented a rev-list equivalent yet.  That's a major
feature which is missing.  The --max-count test was simple to put
together.  I really need to start thinking about replicating some
of the features of the revision functions in core Git.

> > One of the biggest annoyances has been the fact that although Java
> > 1.4 offers a way to mmap a file into the process, the overhead to
> > access that data seems to be far higher than just reading the file
> > content into a very large byte array, especially if we are going
> > to access that file content multiple times.
> 
> That must suck for big packed repositories. What JVM and other environment 
> are you using?

Mac OS 10.4.8 / Java 1.4.2.  It appears as though Sun isn't going
to fix the mmap performance problems as they can't do it "securely".
So its likely to be an issue anywhere jgit gets used...

> Also, I have to say, one of the reasons I'm interested in your project is 
> that I've never done any Java programming, because quite frankly, I've 
> never had any reason what-so-ever to do so. But if there is some simple 
> setup, and you have jgit exposed somewhere as a git archive, I'd love to 
> take a look, if only to finally learn more about Java.

Of course its in Git.  :-)

There's no webpage here, but you can clone it:

  http://www.spearce.org/projects/scm/egit.git

The code you would recognize most is in org.spearce.jgit/.../lib.
E.g.  BinaryDelta.java came from patch-delta.c in core Git.
Repository has some of the functionality of sha1_file.c, the pack
reading code is in PackFile.java.

-- 

Re: jgit performance update

[sf]

Shawn Pearce wrote:
...
> One of the biggest annoyances has been the fact that although Java
> 1.4 offers a way to mmap a file into the process,

Be careful with mmap:
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4724038


Regards

Stephan

Re: jgit performance update

[Shawn Pearce]

sf <sf-gmane@stephan-feder.de> wrote:
> Shawn Pearce wrote:
> ...
> > One of the biggest annoyances has been the fact that although Java
> > 1.4 offers a way to mmap a file into the process,
> 
> Be careful with mmap:
> http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4724038

Thanks. That particular bug has been of some discussion on
#git before.  I'm planning on making it a configuration flag in
.git/config for the user to decide how much pain they want: mmap
(with all its huge downsides) or read into byte[] (with all the
memory footprint that requires).

--