[PATCH JGIT] Method invokes inefficient new String(String) constructor

[PATCH JGIT] Method invokes inefficient new String(String) constructor

[Yann Simon]

>From FindBugs:
Using the java.lang.String(String) constructor wastes memory because
the object so constructed will be functionally indistinguishable from
the String passed as a parameter. Just use the argument String directly.

Signed-off-by: Yann Simon <yann.simon.fr@gmail.com>
---
 .../src/org/spearce/jgit/lib/RefDatabase.java      |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/RefDatabase.java b/org.spearce.jgit/src/org/spearce/jgit/lib/RefDatabase.java
index 87f26bf..49da538 100644
--- a/org.spearce.jgit/src/org/spearce/jgit/lib/RefDatabase.java
+++ b/org.spearce.jgit/src/org/spearce/jgit/lib/RefDatabase.java
@@ -447,7 +447,7 @@ private synchronized void refreshPackedRefs() {
 
 					final int sp = p.indexOf(' ');
 					final ObjectId id = ObjectId.fromString(p.substring(0, sp));
-					final String name = new String(p.substring(sp + 1));
+					final String name = p.substring(sp + 1);
 					last = new Ref(Ref.Storage.PACKED, name, name, id);
 					newPackedRefs.put(last.getName(), last);
 				}
-- 
1.6.1.2

Re: [PATCH JGIT] Method invokes inefficient new String(String) constructor

[Shawn O. Pearce]

Yann Simon <yann.simon.fr@gmail.com> wrote:
> From FindBugs:
> Using the java.lang.String(String) constructor wastes memory because
> the object so constructed will be functionally indistinguishable from
> the String passed as a parameter. Just use the argument String directly.
> 
> Signed-off-by: Yann Simon <yann.simon.fr@gmail.com>
> ---
>  .../src/org/spearce/jgit/lib/RefDatabase.java      |    2 +-
>  1 files changed, 1 insertions(+), 1 deletions(-)
> 
> diff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/RefDatabase.java b/org.spearce.jgit/src/org/spearce/jgit/lib/RefDatabase.java
> index 87f26bf..49da538 100644
> --- a/org.spearce.jgit/src/org/spearce/jgit/lib/RefDatabase.java
> +++ b/org.spearce.jgit/src/org/spearce/jgit/lib/RefDatabase.java
> @@ -447,7 +447,7 @@ private synchronized void refreshPackedRefs() {
>  
>  					final int sp = p.indexOf(' ');
>  					final ObjectId id = ObjectId.fromString(p.substring(0, sp));
> -					final String name = new String(p.substring(sp + 1));
> +					final String name = p.substring(sp + 1);
>  					last = new Ref(Ref.Storage.PACKED, name, name, id);
>  					newPackedRefs.put(last.getName(), last);

I had a specific reason for forcing a new String object here.

The line in question, p, is from the packed-refs file and
contains the entire SHA-1 in hex form at the beginning of it.
We've converted that into binary as an ObjectId, it uses 1/4 the
space of the string portion.

The Ref object, its ObjectId, and its name string, are going to be
cached in a Map, probably long-term.  We're better off shedding the
80 bytes of memory used to hold the hex SHA-1 then risk substring()
deciding its "faster" to reuse the char[] then to make a copy of it.

-- 
Shawn.

Re: [PATCH JGIT] Method invokes inefficient new String(String) constructor

[Yann Simon]

2009/3/19 Shawn O. Pearce <spearce@spearce.org>:
> The line in question, p, is from the packed-refs file and
> contains the entire SHA-1 in hex form at the beginning of it.
> We've converted that into binary as an ObjectId, it uses 1/4 the
> space of the string portion.
>
> The Ref object, its ObjectId, and its name string, are going to be
> cached in a Map, probably long-term.  We're better off shedding the
> 80 bytes of memory used to hold the hex SHA-1 then risk substring()
> deciding its "faster" to reuse the char[] then to make a copy of it.

Thany you for the explanation.
I learn something, and my tests confirm it.
The p.substring(...) can keep the entire array of char by only
updating the intern offset value.
new String(p.subtring(...)) make sure that the variable contains only
the final chars.

Yann

Re: [PATCH JGIT] Method invokes inefficient new String(String) constructor

[Yann Simon]

2009/3/19 Shawn O. Pearce <spearce@spearce.org>:
> Yann Simon <yann.simon.fr@gmail.com> wrote:
>> From FindBugs:
>> Using the java.lang.String(String) constructor wastes memory because
>> the object so constructed will be functionally indistinguishable from
>> the String passed as a parameter. Just use the argument String directly.
>>
>> Signed-off-by: Yann Simon <yann.simon.fr@gmail.com>
>> ---
>>  .../src/org/spearce/jgit/lib/RefDatabase.java      |    2 +-
>>  1 files changed, 1 insertions(+), 1 deletions(-)
>>
>> diff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/RefDatabase.java b/org.spearce.jgit/src/org/spearce/jgit/lib/RefDatabase.java
>> index 87f26bf..49da538 100644
>> --- a/org.spearce.jgit/src/org/spearce/jgit/lib/RefDatabase.java
>> +++ b/org.spearce.jgit/src/org/spearce/jgit/lib/RefDatabase.java
>> @@ -447,7 +447,7 @@ private synchronized void refreshPackedRefs() {
>>
>>                                       final int sp = p.indexOf(' ');
>>                                       final ObjectId id = ObjectId.fromString(p.substring(0, sp));
>> -                                     final String name = new String(p.substring(sp + 1));
>> +                                     final String name = p.substring(sp + 1);
>>                                       last = new Ref(Ref.Storage.PACKED, name, name, id);
>>                                       newPackedRefs.put(last.getName(), last);
>
> I had a specific reason for forcing a new String object here.
>
> The line in question, p, is from the packed-refs file and
> contains the entire SHA-1 in hex form at the beginning of it.
> We've converted that into binary as an ObjectId, it uses 1/4 the
> space of the string portion.
>
> The Ref object, its ObjectId, and its name string, are going to be
> cached in a Map, probably long-term.  We're better off shedding the
> 80 bytes of memory used to hold the hex SHA-1 then risk substring()
> deciding its "faster" to reuse the char[] then to make a copy of it.

However, using the trick newString = new String(aString.substring(),
i) does not work on all JVM.
With an IBM JVM, the newString will still contain the original array of chars.

Another solution that work on all JVM could be:
newString = new String(aString.substring(i).toCharArray())
Or
newString = new String(aString.toCharArray(), i, aString.length() - i)

I like the latter one.

Yann

[PATCH] FindBugs: don't use new String(String) in RefDatabase

[Shawn O. Pearce]

>From FindBugs:
  Using the java.lang.String(String) constructor wastes memory
  because the object so constructed will be functionally
  indistinguishable from the String passed as a parameter. Just
  use the argument String directly.

Actually, here we want to get a new String object that covers only
the portion of the source string that we are selected out.

The line in question, p, is from the packed-refs file and contains
the entire SHA-1 in hex form at the beginning of it.  We have already
converted that into binary as an ObjectId, which uses 1/4 the space
of the string portion.

The Ref object, its ObjectId, and its name string, are going to be
cached in a Map, probably long-term, as the packed-refs file does
not change frequently.  We are better off shedding the 80 bytes of
memory used to hold the hex SHA-1 then risk substring() deciding its
"better" to reuse the same char[] internally.

By creating a new StringBuilder of the exact required capacity for
the name, and then copying in the region of characters we really
want, we defeat the reuse substring() would normally perform, at
the tiny cost of an extra StringBuilder temporary.  Some JITs are
able to stack allocate that here, making it a trivial cost.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
CC: Yann Simon <yann.simon.fr@gmail.com>
---
 Yann Simon <yann.simon.fr@gmail.com> wrote:
 > 2009/3/19 Shawn O. Pearce <spearce@spearce.org>:
 > > Yann Simon <yann.simon.fr@gmail.com> wrote:
 > >> From FindBugs:
 > >> Using the java.lang.String(String) constructor wastes [...]
 > 
 > However, using the trick newString = new String(aString.substring(),
 > i) does not work on all JVM.
 > With an IBM JVM, the newString will still contain the original array of chars.
 > 
 > Another solution that work on all JVM could be:
 > newString = new String(aString.substring(i).toCharArray())
 > Or
 > newString = new String(aString.toCharArray(), i, aString.length() - i)
 > 
 > I like the latter one.

 I prefer this.  It should always do what we want, and at a lower
 temporary memory footprint (one less copy of the name).  IIRC Robin
 rejected it earlier because it wasn't obvious what we were doing. I
 say hogwash, its clear as mud.
 
 .../src/org/spearce/jgit/lib/RefDatabase.java      |    6 +++++-
 1 files changed, 5 insertions(+), 1 deletions(-)

diff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/RefDatabase.java b/org.spearce.jgit/src/org/spearce/jgit/lib/RefDatabase.java
index 6d4f374..383877f 100644
--- a/org.spearce.jgit/src/org/spearce/jgit/lib/RefDatabase.java
+++ b/org.spearce.jgit/src/org/spearce/jgit/lib/RefDatabase.java
@@ -438,7 +438,7 @@ private synchronized void refreshPackedRefs() {
 
 					final int sp = p.indexOf(' ');
 					final ObjectId id = ObjectId.fromString(p.substring(0, sp));
-					final String name = new String(p.substring(sp + 1));
+					final String name = copy(p, sp + 1, p.length());
 					last = new Ref(Ref.Storage.PACKED, name, name, id);
 					newPackedRefs.put(last.getName(), last);
 				}
@@ -460,6 +460,10 @@ private synchronized void refreshPackedRefs() {
 		}
 	}
 
+	private static String copy(final String src, final int off, final int end) {
+		return new StringBuilder(end - off).append(src, off, end).toString();
+	}
+
 	private void lockAndWriteFile(File file, byte[] content) throws IOException {
 		String name = file.getName();
 		final LockFile lck = new LockFile(file);
-- 
1.6.3.3.507.gc6b5a

Re: [PATCH] FindBugs: don't use new String(String) in RefDatabase

[Yann Simon]

2009/7/10 Shawn O. Pearce <spearce@spearce.org>:
>  Yann Simon <yann.simon.fr@gmail.com> wrote:
>  >
>  > However, using the trick newString = new String(aString.substring(),
>  > i) does not work on all JVM.
>  > With an IBM JVM, the newString will still contain the original array of chars.
>  >
>  > Another solution that work on all JVM could be:
>  > newString = new String(aString.substring(i).toCharArray())
>  > Or
>  > newString = new String(aString.toCharArray(), i, aString.length() - i)
>  >
>  > I like the latter one.
>
>  I prefer this.  It should always do what we want, and at a lower
>  temporary memory footprint (one less copy of the name).  IIRC Robin
>  rejected it earlier because it wasn't obvious what we were doing. I
>  say hogwash, its clear as mud.
>
> +       private static String copy(final String src, final int off, final int end) {
> +               return new StringBuilder(end - off).append(src, off, end).toString();
> +       }
> +

This method is quite clear.
One line javadoc would make it even clearer... :p (and maybe make Robin happy)

And you're right: by using a StringBuilder, we need one less arraycopy.

After committing your change, we can remove the entry to silent FindBugs.
(commit 21c3d82824075cd1f140b3bcf252dfaffe0fc96c)

Yann

[JGIT PATCH v2] FindBugs: don't use new String(String) in RefDatabase

[Shawn O. Pearce]

>From FindBugs:
  Using the java.lang.String(String) constructor wastes memory
  because the object so constructed will be functionally
  indistinguishable from the String passed as a parameter. Just
  use the argument String directly.

Actually, here we want to get a new String object that covers only
the portion of the source string that we are selected out.

The line in question, p, is from the packed-refs file and contains
the entire SHA-1 in hex form at the beginning of it.  We have already
converted that into binary as an ObjectId, which uses 1/4 the space
of the string portion.

The Ref object, its ObjectId, and its name string, are going to be
cached in a Map, probably long-term, as the packed-refs file does
not change frequently.  We are better off shedding the 80 bytes of
memory used to hold the hex SHA-1 then risk substring() deciding its
"better" to reuse the same char[] internally.

By creating a new StringBuilder of the exact required capacity for
the name, and then copying in the region of characters we really
want, we defeat the reuse substring() would normally perform, at
the tiny cost of an extra StringBuilder temporary.  Some JITs are
able to stack allocate that here, making it a trivial cost.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
CC: Yann Simon <yann.simon.fr@gmail.com>
---
 Yann Simon <yann.simon.fr@gmail.com> wrote:
 > This method is quite clear.
 > One line javadoc would make it even clearer... :p (and maybe make Robin happy)
 
 Javadoc is overrated.  Private utility methods like this that are one
 line long don't need documentation.  The rationale for why this line
 does what it does is something that `git blame` can answer better.
  
 > And you're right: by using a StringBuilder, we need one less arraycopy.
 > 
 > After committing your change, we can remove the entry to silent FindBugs.
 > (commit 21c3d82824075cd1f140b3bcf252dfaffe0fc96c)
 
 My patch is updated (below).  Thanks, I forgot about that filter.

 .../findBugs/FindBugsExcludeFilter.xml             |    7 -------
 .../src/org/spearce/jgit/lib/RefDatabase.java      |    6 +++++-
 2 files changed, 5 insertions(+), 8 deletions(-)

diff --git a/org.spearce.jgit/findBugs/FindBugsExcludeFilter.xml b/org.spearce.jgit/findBugs/FindBugsExcludeFilter.xml
index 2af9348..a553170 100644
--- a/org.spearce.jgit/findBugs/FindBugsExcludeFilter.xml
+++ b/org.spearce.jgit/findBugs/FindBugsExcludeFilter.xml
@@ -1,12 +1,5 @@
 <?xml version="1.0" encoding="UTF-8" ?>
 <FindBugsFilter>
-     <!-- Silence inefficient new String(String) constructor warning, see http://thread.gmane.org/gmane.comp.version-control.git/117831/focus=117937 -->
-     <Match>
-       <Class name="org.spearce.jgit.lib.RefDatabase" />
-       <Method name="refreshPackedRefs" />
-       <Bug pattern="DM_STRING_CTOR" />
-     </Match>
-
      <!-- Silence PackFile.mmap calls GC, we need to force it to remove stale
           memory mapped segments if the JVM heap is out of address space.
        -->
diff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/RefDatabase.java b/org.spearce.jgit/src/org/spearce/jgit/lib/RefDatabase.java
index 6d4f374..383877f 100644
--- a/org.spearce.jgit/src/org/spearce/jgit/lib/RefDatabase.java
+++ b/org.spearce.jgit/src/org/spearce/jgit/lib/RefDatabase.java
@@ -438,7 +438,7 @@ private synchronized void refreshPackedRefs() {
 
 					final int sp = p.indexOf(' ');
 					final ObjectId id = ObjectId.fromString(p.substring(0, sp));
-					final String name = new String(p.substring(sp + 1));
+					final String name = copy(p, sp + 1, p.length());
 					last = new Ref(Ref.Storage.PACKED, name, name, id);
 					newPackedRefs.put(last.getName(), last);
 				}
@@ -460,6 +460,10 @@ private synchronized void refreshPackedRefs() {
 		}
 	}
 
+	private static String copy(final String src, final int off, final int end) {
+		return new StringBuilder(end - off).append(src, off, end).toString();
+	}
+
 	private void lockAndWriteFile(File file, byte[] content) throws IOException {
 		String name = file.getName();
 		final LockFile lck = new LockFile(file);
-- 
1.6.4.rc0.117.g28cb

Re: [JGIT PATCH v2] FindBugs: don't use new String(String) in RefDatabase

[Robin Rosenberg]

måndag 13 juli 2009 16:53:08 skrev "Shawn O. Pearce" <spearce@spearce.org>:
>  Yann Simon <yann.simon.fr@gmail.com> wrote:
>  > This method is quite clear.
>  > One line javadoc would make it even clearer... :p (and maybe make Robin happy)
>  
>  Javadoc is overrated.  Private utility methods like this that are one
>  line long don't need documentation.  The rationale for why this line
>  does what it does is something that `git blame` can answer better.

Sorry Yann, I'm with Shawn here. A comment wouldn't add anything here since
the method is pretty simple. Besides javadocs are for API's so an inline comment
would be better, but I don't think it is necessary.

Shawn, any references for the ability of JIT's abilities to stack allocate in this context? For
me learning, will commit anyway.

-- robina

Re: [JGIT PATCH v2] FindBugs: don't use new String(String) in RefDatabase

[Shawn O. Pearce]

Robin Rosenberg <robin.rosenberg.lists@dewire.com> wrote:
> Shawn, any references for the ability of JIT's abilities to stack allocate in this context? For
> me learning, will commit anyway.

See [1] for example.  I read a presentation from a HotSpot engineer
at Sun a year or two ago that talked about it as a feature in the
Sun Java 6 runtime, but I can't track that down now.

Its a pretty simple concept.  Folks realized that some types,
e.g. StringBuilder, are often used only within a single stack
frame, and that escape analysis can be used to prove that the
StringBuilder instance is only visible within that stack frame.
Doing a stack allocation instead of a heap allocation would allow
the JVM to avoid creating unnecessary garbage.

Ah, according to [2] the feature is only in 6u14 and later, and is
an option you still need to enable on the command line.  But its
the direction the Sun JVM is going.  I imagine it would be on by
default in the future, various performance tests seem to indicate
its a worthwhile optimization.

[1] http://www.ibm.com/developerworks/java/library/j-jtp09275.html?ca=dgr-lnxw01JavaUrbanLegends
[2] http://java.sun.com/javase/6/webnotes/6u14.html

-- 
Shawn.

Re: [JGIT PATCH v2] FindBugs: don't use new String(String) in RefDatabase

[Robin Rosenberg]

tisdag 21 juli 2009 17:03:37 skrev "Shawn O. Pearce" <spearce@spearce.org>:
> Robin Rosenberg <robin.rosenberg.lists@dewire.com> wrote:
> > Shawn, any references for the ability of JIT's abilities to stack allocate in this context? For
> > me learning, will commit anyway.
> 
> See [1] for example.  I read a presentation from a HotSpot engineer
> at Sun a year or two ago that talked about it as a feature in the
> Sun Java 6 runtime, but I can't track that down now.
>
...
> 
> [1] http://www.ibm.com/developerworks/java/library/j-jtp09275.html?ca=dgr-lnxw01JavaUrbanLegends
> [2] http://java.sun.com/javase/6/webnotes/6u14.html
> 

Thanks. HotSpot rules.

-- robin

[PATCH JGIT] Computation of average could overflow

[Sohn, Matthias]

The code computes the average of two integers using either division or
signed right shift, and then uses the result as the
index of an array. If the values being averaged are very large, this can
overflow (resulting in the computation of a negative
average). Assuming that the result is intended to be nonnegative, you
can use an unsigned right shift instead. In other
words, rather that using (low+high)/2, use (low+high) >>> 1

This bug exists in many earlier implementations of binary search and
merge sort. Martin Buchholz found and fixed it
(http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6412541) in the JDK
libraries, and Joshua Bloch widely publicized
the bug pattern
(http://googleresearch.blogspot.com/2006/06/extra-extra-read-all-about-i
t-nearly.html).

Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
---
 .../src/org/spearce/jgit/dircache/DirCache.java    |    2 +-
 .../src/org/spearce/jgit/lib/Tree.java             |    2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git
a/org.spearce.jgit/src/org/spearce/jgit/dircache/DirCache.java
b/org.spearce.jgit/src/org/spearce/jgit/dircache/DirCache.java
index 58da014..fa906fa 100644
--- a/org.spearce.jgit/src/org/spearce/jgit/dircache/DirCache.java
+++ b/org.spearce.jgit/src/org/spearce/jgit/dircache/DirCache.java
@@ -593,7 +593,7 @@ int findEntry(final byte[] p, final int pLen) {
 		int low = 0;
 		int high = entryCnt;
 		do {
-			int mid = (low + high) >> 1;
+			int mid = (low + high) >>> 1;
 			final int cmp = cmp(p, pLen,
sortedEntries[mid]);
 			if (cmp < 0)
 				high = mid;
diff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/Tree.java
b/org.spearce.jgit/src/org/spearce/jgit/lib/Tree.java
index 0ecd04d..ff9e666 100644
--- a/org.spearce.jgit/src/org/spearce/jgit/lib/Tree.java
+++ b/org.spearce.jgit/src/org/spearce/jgit/lib/Tree.java
@@ -136,7 +136,7 @@ private static final int binarySearch(final
TreeEntry[] entries,
 		int high = entries.length;
 		int low = 0;
 		do {
-			final int mid = (low + high) / 2;
+			final int mid = (low + high) >>> 1;
 			final int cmp =
compareNames(entries[mid].getNameUTF8(), nameUTF8,
 					nameStart, nameEnd,
TreeEntry.lastChar(entries[mid]), nameUTF8last);
 			if (cmp < 0)
-- 
1.6.2.2.1669.g7eaf8

[PATCH JGIT] Method invokes inefficient new String(String) constructor

[Sohn, Matthias]

Using the java.lang.String(String) constructor wastes memory because the
object so constructed will be functionally
indistinguishable from the String passed as a parameter.  Just use the
argument String directly.

Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
---
 .../src/org/spearce/jgit/lib/RefDatabase.java      |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/RefDatabase.java
b/org.spearce.jgit/src/org/spearce/jgit/lib/RefDatabase.java
index 87f26bf..49da538 100644
--- a/org.spearce.jgit/src/org/spearce/jgit/lib/RefDatabase.java
+++ b/org.spearce.jgit/src/org/spearce/jgit/lib/RefDatabase.java
@@ -447,7 +447,7 @@ private synchronized void refreshPackedRefs() {
 
 					final int sp = p.indexOf(' ');
 					final ObjectId id =
ObjectId.fromString(p.substring(0, sp));
-					final String name = new
String(p.substring(sp + 1));
+					final String name =
p.substring(sp + 1);
 					last = new
Ref(Ref.Storage.PACKED, name, name, id);
 
newPackedRefs.put(last.getName(), last);
 				}
-- 
1.6.2.2.1669.g7eaf8

Re: [PATCH JGIT] Method invokes inefficient new String(String) constructor

[Shawn O. Pearce]

"Sohn, Matthias" <matthias.sohn@sap.com> wrote:
> Using the java.lang.String(String) constructor wastes memory because the
> object so constructed will be functionally
> indistinguishable from the String passed as a parameter.  Just use the
> argument String directly.

NAK.

Like the Long.valueOf() case this came up before:

http://thread.gmane.org/gmane.comp.version-control.git/113739/focus=113787
 
-- 
Shawn.