[JGIT] maven build fails on OS X

[JGIT] maven build fails on OS X

[Nigel Magnay]

I don't know if this is known about - I get failures running the m2
build for jgit:

Failed tests:
  testParse_implicit_mixed_encoded(org.spearce.jgit.revwalk.RevCommitParseTest)
  testParse_explicit_bad_encoded(org.spearce.jgit.revwalk.RevCommitParseTest)
  testParse_implicit_mixed_encoded(org.spearce.jgit.revwalk.RevTagParseTest)
  testParse_explicit_bad_encoded(org.spearce.jgit.revwalk.RevTagParseTest)
  testDequote_OctalAll(org.spearce.jgit.util.QuotedStringGitPathStyleTest)
  testDequote_Latin1(org.spearce.jgit.util.QuotedStringGitPathStyleTest)
  testDequote_RawLatin1(org.spearce.jgit.util.QuotedStringGitPathStyleTest)
  testGetText_DiffCc(org.spearce.jgit.patch.GetTextTest)

Curiously, the tests work in eclipse and from ./make_jgit.sh - I
suspect some environment setting with the mac's charset.

I can try and fix it if it's new, but I wanted to check first..

Re: [JGIT] maven build fails on OS X

[Shawn O. Pearce]

Nigel Magnay <nigel.magnay@gmail.com> wrote:
> I don't know if this is known about - I get failures running the m2
> build for jgit:

Yup.
 
> Failed tests:
>   testParse_implicit_mixed_encoded(org.spearce.jgit.revwalk.RevCommitParseTest)
>   testParse_explicit_bad_encoded(org.spearce.jgit.revwalk.RevCommitParseTest)
>   testParse_implicit_mixed_encoded(org.spearce.jgit.revwalk.RevTagParseTest)
>   testParse_explicit_bad_encoded(org.spearce.jgit.revwalk.RevTagParseTest)
>   testDequote_OctalAll(org.spearce.jgit.util.QuotedStringGitPathStyleTest)
>   testDequote_Latin1(org.spearce.jgit.util.QuotedStringGitPathStyleTest)
>   testDequote_RawLatin1(org.spearce.jgit.util.QuotedStringGitPathStyleTest)
>   testGetText_DiffCc(org.spearce.jgit.patch.GetTextTest)
> 
> Curiously, the tests work in eclipse and from ./make_jgit.sh - I
> suspect some environment setting with the mac's charset.

make_jgit.sh doesn't run tests.  But Eclipse does, and as you state,
I have also seen these pass in Eclipse but fail from m2.

What's even more curious is look at the surefire report files for
these tests.  The actual result looks correct, the expected result
looks wrong.  I think m2 miscompiled the Java sources for the tests
somehow and produced incorrect literal strings which are used for
the expected values in the tests.

I don't see these failures on Linux, from either Eclipse or Maven.

> I can try and fix it if it's new, but I wanted to check first..

I've looked at it and gave up.  I don't know what Maven is doing here
on the Mac.  Maybe fresh eyes will have a better chance at fixing it.

My experience with Maven is it works about 5% of the time, and the
other 95% of the time you have to work around it by skipping tests,
or by writing massive blocks of XML in your pom.xml file, or by
redesigning your entire project directory structure and revision
control system to use SVN instead of Git.

-- 
Shawn.

Re: [JGIT] maven build fails on OS X

[Nigel Magnay]

> I've looked at it and gave up.  I don't know what Maven is doing here
> on the Mac.  Maybe fresh eyes will have a better chance at fixing it.
>
> My experience with Maven is it works about 5% of the time, and the
> other 95% of the time you have to work around it by skipping tests,
> or by writing massive blocks of XML in your pom.xml file, or by
> redesigning your entire project directory structure and revision
> control system to use SVN instead of Git.

Ok. It's not (just) a Maven problem as I can get the same tests to
fail in Eclipse by setting the Console Encoding to MacRoman (which is
the default Charset.defaultCharset() when running a java app on OS X).

Doing some digging, just on testGetText_DiffCc :- where it fails, I
look at the output of each doing getBytes() - so will be in MacRoman -
(cols are byte no, exp before the replace, after the replace and the
result of fh.getScriptText. I get :

94 32 32,32
95 43 43,43 ++
96 116 116,116 tt
97 101 101,101 ee
98 115 115,115 ss
99 116 116,116 tt
100 32 32,32
101 -127 -127,-59 !!!!!!!!!!!!!!!!!!!!!!!!!!!! ??
102 110 110,110 nn
103 103 103,103 gg
104 115 115,115 ss
105 116 116,116 tt
106 114 114,114 rr
107 -102 -102,-10 !!!!!!!!!!!!!!!!!!!!!!!!!!!! ??
108 109 109,109 mm
109 10 10,10

-10 (F6) is ^, and -59 (C5) is some wavy lines..

However, I'm slightly confused - possibly because I don't know what
/should/ be the case. In

 exp.replace("\303\205ngstr\303\266m", "\u00c5ngstr\u00f6m")

is \303\205 really meant to represent U+00C3 and U+0085 ? Shouldn't
the replace be being done on bytes rather than strings?

Re: [JGIT] maven build fails on OS X

[Robin Rosenberg]

fredag 06 februari 2009 16:31:55 skrev Shawn O. Pearce:
> Nigel Magnay <nigel.magnay@gmail.com> wrote:
> > I don't know if this is known about - I get failures running the m2
> > build for jgit:
> 
> Yup.
>  
> > Failed tests:
> >   testParse_implicit_mixed_encoded(org.spearce.jgit.revwalk.RevCommitParseTest)
> >   testParse_explicit_bad_encoded(org.spearce.jgit.revwalk.RevCommitParseTest)
> >   testParse_implicit_mixed_encoded(org.spearce.jgit.revwalk.RevTagParseTest)
> >   testParse_explicit_bad_encoded(org.spearce.jgit.revwalk.RevTagParseTest)
> >   testDequote_OctalAll(org.spearce.jgit.util.QuotedStringGitPathStyleTest)
> >   testDequote_Latin1(org.spearce.jgit.util.QuotedStringGitPathStyleTest)
> >   testDequote_RawLatin1(org.spearce.jgit.util.QuotedStringGitPathStyleTest)
> >   testGetText_DiffCc(org.spearce.jgit.patch.GetTextTest)
> > 
> > Curiously, the tests work in eclipse and from ./make_jgit.sh - I
> > suspect some environment setting with the mac's charset.
> 
> make_jgit.sh doesn't run tests.  But Eclipse does, and as you state,
> I have also seen these pass in Eclipse but fail from m2.
> 
> What's even more curious is look at the surefire report files for
> these tests.  The actual result looks correct, the expected result
> looks wrong.  I think m2 miscompiled the Java sources for the tests
> somehow and produced incorrect literal strings which are used for
> the expected values in the tests.
> 
> I don't see these failures on Linux, from either Eclipse or Maven.

One thing to suspect could be the better-than-everybody-else decomposed
unicode causing our tests to decompose too. I don't have a Mac so Ic
cannot test my hypothesis myself, but consider

	testDequote_RawLatin1(org.spearce.jgit.util.QuotedStringGitPathStyleTest)

Here we intend to feed the dequote an ISO-Latin encoded string. Our decoder
will first try to decode it as UTF-8, which should fail, then we try to decode it
as the platform default (Linear A, Hieroglyphs, MacRoman or something not
known to living mankind). A problem is that Git *forces* us to guess the encoding,
and in some situations you'd need human intelligence to figure it out. Software
with a little more hindsight recognize this and declare UTF-8 to be the one
and only encoding. JGit always encodes things as UTF-8 for this reason, but
we cannot trust the input. These test cases are supposed to prove that we
can guess things correctly at least some of the time.. Using "platform
default" may be the wrong thing if it does not match what a C Git user on
this platform would encounter. 

Sorry about the long explanation, but I didn't have the time to write a short
one.

-- robin



> 
> > I can try and fix it if it's new, but I wanted to check first..
> 
> I've looked at it and gave up.  I don't know what Maven is doing here
> on the Mac.  Maybe fresh eyes will have a better chance at fixing it.
> 
> My experience with Maven is it works about 5% of the time, and the
> other 95% of the time you have to work around it by skipping tests,
> or by writing massive blocks of XML in your pom.xml file, or by
> redesigning your entire project directory structure and revision
> control system to use SVN instead of Git.
> 

Re: [JGIT] maven build fails on OS X

[Robin Rosenberg]

fredag 06 februari 2009 21:08:18 skrev Nigel Magnay:
> > I've looked at it and gave up.  I don't know what Maven is doing here
> > on the Mac.  Maybe fresh eyes will have a better chance at fixing it.
> >
> > My experience with Maven is it works about 5% of the time, and the
> > other 95% of the time you have to work around it by skipping tests,
> > or by writing massive blocks of XML in your pom.xml file, or by
> > redesigning your entire project directory structure and revision
> > control system to use SVN instead of Git.
> 
> Ok. It's not (just) a Maven problem as I can get the same tests to
> fail in Eclipse by setting the Console Encoding to MacRoman (which is
> the default Charset.defaultCharset() when running a java app on OS X).

Ok you found it. I just sent a reply...

> However, I'm slightly confused - possibly because I don't know what
> /should/ be the case. In
> 
>  exp.replace("\303\205ngstr\303\266m", "\u00c5ngstr\u00f6m")

The text is "Ångström".

> is \303\205 really meant to represent U+00C3 and U+0085 ? Shouldn't
> the replace be being done on bytes rather than strings?

Yes, but we fake the RAW strings by only really using the lower byte of
each code point. arrays of bytes are really hard to read in source code. 
The MacRoman is the problem. What encoding does C Git on Mac use?
Isn't that UTF-8, in which case our use of platform default as fallback is
wrong. Gah...

-- robin

Re: [JGIT] maven build fails on OS X

[Nigel Magnay]

> Here we intend to feed the dequote an ISO-Latin encoded string. Our decoder
> will first try to decode it as UTF-8, which should fail, then we try to decode it
> as the platform default (Linear A, Hieroglyphs, MacRoman or something not
> known to living mankind). A problem is that Git *forces* us to guess the encoding,
> and in some situations you'd need human intelligence to figure it out. Software
> with a little more hindsight recognize this and declare UTF-8 to be the one
> and only encoding. JGit always encodes things as UTF-8 for this reason, but
> we cannot trust the input. These test cases are supposed to prove that we
> can guess things correctly at least some of the time.. Using "platform
> default" may be the wrong thing if it does not match what a C Git user on
> this platform would encounter.
>

I got your reply exaclty as I found the same thing :-). I'm not sure
that this is the common cause of all test failures, but it is for
testGetText_DiffCc :-

When it parses the first string in extractFileLines, it passes to
RawParseUtils.decode which, as you outlined, does a "Try UTF-8, Try
caller suggestion, try Charset.defaultCharset(), then finally back out
to ISO-8859-1.

On most platforms, this is Try UTF-8 (fail), caller suggestion (also
UTF-8, skip), try default (UTF-8, skip) - finally parse ISO-8859-1.

On the mac, this is Try UTF-8 (fail), caller suggestion (UTF-8, skip),
try default (MacRoman) --> succeed.

I'm less clear on what the right way to fix it is (or if it's just the
tests that need to somehow force the system file.encoding?