Re: RFC: x86: kill binutils 2.16.x?

Re: RFC: x86: kill binutils 2.16.x?

[Kyle Moffett]

On Thu, Mar 3, 2011 at 03:30, Ingo Molnar <mingo@elte.hu> wrote:
> This is how specific GAS functionality is tested in arch/powerpc:
>
> =C2=A0 =C2=A0 =C2=A0 =C2=A0@if ! /bin/echo dssall | $(AS) -many -o $(TOUT=
) >/dev/null 2>&1 ; then \
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0echo -n '*** ${VER=
SION}.${PATCHLEVEL} kernels no longer build ' ; \
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0echo 'correctly wi=
th old versions of binutils.' ; \
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0echo '*** Please u=
pgrade your binutils to 2.12.1 or newer' ; \
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0false ; \
> =C2=A0 =C2=A0 =C2=A0 =C2=A0fi
>
> This would also be a 'constructive' (and safest) way of blacklisting binu=
tils: we'd
> really only exclude binutils that is truly buggy.

Hrm... well... actually...

It's funny that you brought up this particular case.  While I agree
that it's good in general, it's causing problems for me building a
kernel using a recent e500 gcc/binutils (triplet
"powerpc-linux-gnuspe").

Specifically the e500 doesn't have a normal PowerPC FPU, it has a
custom FPU built using extended integer registers instead, and it
happens to borrow the AltiVec opcode range to do it.

When trying to port Debian to the platform we were getting SIGILL's
all over the place until binutils got updated to reject all of the
unsupported opcodes on this particular platform.  Now of course we get
build errors, but that's a lot easier to debug and fix. :-D

Basically, binutils no longer supports "-many" (because too many
opcodes conflict), and the test itself would fail anyways (because
"dssall" is not valid on "any" PowerPC).

So while I think that it is entirely reasonable to add a similar test
for buggy x86 binutils, I'm actually about to send a patch to remove
that particular check from the powerpc Makefile.  Since the "required"
binutils 2.12.1 was released in May 2002 (almost 9 years ago) it's
probably not even worth testing for anymore.

Cheers,
Kyle Moffett

Re: RFC: x86: kill binutils 2.16.x?

[Benjamin Herrenschmidt]

On Tue, 2011-03-08 at 14:57 -0500, Kyle Moffett wrote:
> 
> Specifically the e500 doesn't have a normal PowerPC FPU, it has a
> custom FPU built using extended integer registers instead, and it
> happens to borrow the AltiVec opcode range to do it.
> 
> When trying to port Debian to the platform we were getting SIGILL's
> all over the place until binutils got updated to reject all of the
> unsupported opcodes on this particular platform.  Now of course we get
> build errors, but that's a lot easier to debug and fix. :-D
> 
> Basically, binutils no longer supports "-many" (because too many
> opcodes conflict), and the test itself would fail anyways (because
> "dssall" is not valid on "any" PowerPC). 

Note that this freescale "SPE" fiasco is just that ... a fiasco :-) I
don't think there's that many cases of opcode overlap outside of it.

Now regarding the kernel, the best is probably for nasty cases like that
to use hand coded opcodes (see ppc-opcodes.h) and stick to a more
"generic" setting for binutils, since it should be possible to build
kernels that support multiple types of BookE CPUs with different
floating point units.

Cheers,
Ben.

Re: RFC: x86: kill binutils 2.16.x?

[Scott Wood]

On Wed, 9 Mar 2011 08:28:36 +1100
Benjamin Herrenschmidt <benh@kernel.crashing.org> wrote:

> On Tue, 2011-03-08 at 14:57 -0500, Kyle Moffett wrote:
> > 
> > Specifically the e500 doesn't have a normal PowerPC FPU, it has a
> > custom FPU built using extended integer registers instead, and it
> > happens to borrow the AltiVec opcode range to do it.
> > 
> > When trying to port Debian to the platform we were getting SIGILL's
> > all over the place until binutils got updated to reject all of the
> > unsupported opcodes on this particular platform.  Now of course we get
> > build errors, but that's a lot easier to debug and fix. :-D
> > 
> > Basically, binutils no longer supports "-many" (because too many
> > opcodes conflict), and the test itself would fail anyways (because
> > "dssall" is not valid on "any" PowerPC). 
> 
> Note that this freescale "SPE" fiasco is just that ... a fiasco :-) I
> don't think there's that many cases of opcode overlap outside of it.
> 
> Now regarding the kernel, the best is probably for nasty cases like that
> to use hand coded opcodes (see ppc-opcodes.h) and stick to a more
> "generic" setting for binutils, since it should be possible to build
> kernels that support multiple types of BookE CPUs with different
> floating point units.

Combined kernels between e500v1/2 and e500mc are currently not supported
for other reasons (current kconfig doesn't prohibit it, but it doesn't
work), such as a cache line size difference (a hardcoded L1_CACHE_BYTES is
used in various places for loops), an inability to enable SPE when e500mc
is enabled, etc.

Kumar recently internally said we're not going to bother making it work.
I'm inclined to agree, given you can't even run the same userspace (unless
you don't use hard floating point at all).  It's one thing to not want to
require a separate kernel for each board, but there's a point of
diminishing returns.

-Scott

Re: RFC: x86: kill binutils 2.16.x?

[Kyle Moffett]

On Tue, Mar 8, 2011 at 16:28, Benjamin Herrenschmidt
<benh@kernel.crashing.org> wrote:
> On Tue, 2011-03-08 at 14:57 -0500, Kyle Moffett wrote:
>> Specifically the e500 doesn't have a normal PowerPC FPU, it has a
>> custom FPU built using extended integer registers instead, and it
>> happens to borrow the AltiVec opcode range to do it.
>>
>> When trying to port Debian to the platform we were getting SIGILL's
>> all over the place until binutils got updated to reject all of the
>> unsupported opcodes on this particular platform. =C2=A0Now of course we =
get
>> build errors, but that's a lot easier to debug and fix. :-D
>>
>> Basically, binutils no longer supports "-many" (because too many
>> opcodes conflict), and the test itself would fail anyways (because
>> "dssall" is not valid on "any" PowerPC).
>
> Note that this freescale "SPE" fiasco is just that ... a fiasco :-) I
> don't think there's that many cases of opcode overlap outside of it.

I absolutely agree on the "fiasco" part, although I'm pretty sure that
there's a large number of incompatible ARM variants (even 16-bit vs.
32-bit opcodes).  Unfortunately there's a lot of shipped hardware to
be supported and maintained...


> Now regarding the kernel, the best is probably for nasty cases like that
> to use hand coded opcodes (see ppc-opcodes.h) and stick to a more
> "generic" setting for binutils, since it should be possible to build
> kernels that support multiple types of BookE CPUs with different
> floating point units.

The problem is not with the kernel compile itself, but with the 2.12
"dssall" binutils test.  Basically, recent binutils treats e500 as
effectively a separate architecture that happens to share *most* of
the opcodes with regular PowerPC.  Any opcode which is not understood
by the e500 chip is either convert to an equivalent opcode which is
understood (IE: lwsync =3D> sync), or failed with an error.  This means
that the kernel compile aborts early telling me to upgrade to a newer
version of binutils.

This was *critical* for getting an actual Debian distribution
bootstrapped on the e500 cores, because so much software assumes
PowerPC =3D=3D altivec (ffmpeg), hardcodes 'asm("lwsync")' for memory
barriers (80+ packages in Debian), or includes hand-coded
floating-point ASM instructions (libffi).  Noisy build errors are
better than silent runtime failures any day of the week.

At the very least that test needs to be turned off if
CONFIG_ALTIVEC=3Dn, because the kernel builds and runs fine otherwise.

Cheers,
Kyle Moffett

Re: RFC: x86: kill binutils 2.16.x?

[Benjamin Herrenschmidt]

On Tue, 2011-03-08 at 16:59 -0500, Kyle Moffett wrote:
> 
> The problem is not with the kernel compile itself, but with the 2.12
> "dssall" binutils test.  Basically, recent binutils treats e500 as
> effectively a separate architecture that happens to share *most* of
> the opcodes with regular PowerPC. 

This is bogus. Newer e500 don't have that SPE crap afaik and BookI and
II of the architecture have been converged. In fact, Scott, don't newer
FSL chips also support real lwsync ?

>  Any opcode which is not understood
> by the e500 chip is either convert to an equivalent opcode which is
> understood (IE: lwsync => sync), or failed with an error.  This means
> that the kernel compile aborts early telling me to upgrade to a newer
> version of binutils.

This is more bogosity in binutils. lwsync is designed to fallback as
sync if not supported in -HW-, binutils shouldn't silently swallow it.

Or is it that FSL failed on the original e500 and make lwsync actually
trap ?

> This was *critical* for getting an actual Debian distribution
> bootstrapped on the e500 cores, because so much software assumes
> PowerPC == altivec (ffmpeg), hardcodes 'asm("lwsync")' for memory
> barriers (80+ packages in Debian), or includes hand-coded
> floating-point ASM instructions (libffi).  Noisy build errors are
> better than silent runtime failures any day of the week.
> 
> At the very least that test needs to be turned off if
> CONFIG_ALTIVEC=n, because the kernel builds and runs fine otherwise.

I think the right thing is to keep that as e500-legacy or something,
because afaik, newer e500's don't have most of these issues and could be
treated as "normal" powerpc again.

Cheers,
Ben.

Re: RFC: x86: kill binutils 2.16.x?

[Kyle Moffett]

On Tue, Mar 8, 2011 at 18:13, Benjamin Herrenschmidt
<benh@kernel.crashing.org> wrote:
> On Tue, 2011-03-08 at 16:59 -0500, Kyle Moffett wrote:
>>
>> The problem is not with the kernel compile itself, but with the 2.12
>> "dssall" binutils test. =C2=A0Basically, recent binutils treats e500 as
>> effectively a separate architecture that happens to share *most* of
>> the opcodes with regular PowerPC.
>
> This is bogus. Newer e500 don't have that SPE crap afaik and BookI and
> II of the architecture have been converged. In fact, Scott, don't newer
> FSL chips also support real lwsync ?

When I'm talking about "e500" I'm specifically referring to the SPE
stuff.  The "e500mc" cores are a whole different beast with a regular
FPU, but those have their own Kconfig option: "PPC_E500MC".

Actually, looking at it again, the "PPC_E500MC" depends on "E500", it
should select "PPC_FSL_BOOK3E" instead.  There are probably bugs
lurking here.

There really is fundamentally no good way to build a single system
image that supports both E500 (spe-based) and E500MC (classic FPU).
You can sort-of run classic FPU emulation on the E500, but the
performance is terrifyingly bad.

The naming confusion really is really bad too, IMO.

>> =C2=A0Any opcode which is not understood
>> by the e500 chip is either convert to an equivalent opcode which is
>> understood (IE: lwsync =3D> sync), or failed with an error. =C2=A0This m=
eans
>> that the kernel compile aborts early telling me to upgrade to a newer
>> version of binutils.
>
> This is more bogosity in binutils. lwsync is designed to fallback as
> sync if not supported in -HW-, binutils shouldn't silently swallow it.
>
> Or is it that FSL failed on the original e500 and make lwsync actually
> trap ?

Yeah... for some reason FreeScale made the "lwsync" operation trap on
e500 (again, only talking about "e500" with SPE, not "e500mc").

Unfortunately it's used frequently enough that there's a very
noticeable performance gain (~5% for some programs) by substituting it
in binutils, so that's what happens.


>> This was *critical* for getting an actual Debian distribution
>> bootstrapped on the e500 cores, because so much software assumes
>> PowerPC =3D=3D altivec (ffmpeg), hardcodes 'asm("lwsync")' for memory
>> barriers (80+ packages in Debian), or includes hand-coded
>> floating-point ASM instructions (libffi). =C2=A0Noisy build errors are
>> better than silent runtime failures any day of the week.
>>
>> At the very least that test needs to be turned off if
>> CONFIG_ALTIVEC=3Dn, because the kernel builds and runs fine otherwise.
>
> I think the right thing is to keep that as e500-legacy or something,
> because afaik, newer e500's don't have most of these issues and could be
> treated as "normal" powerpc again.

There is a separate "-me500mc" option for GAS that does the right
thing, but the kernel currently does not seem to use it.

arch/powerpc/Makefile has this:
    cpu-as-$(CONFIG_4xx) +=3D -Wa,-m405
    cpu-as-$(CONFIG_6xx) +=3D -Wa,-maltivec
    cpu-as-$(CONFIG_POWER4) +=3D -Wa,-maltivec
    cpu-as-$(CONFIG_E500) +=3D -Wa,-me500
    cpu-as-$(CONFIG_E200) +=3D -Wa,-me200

As it is, I strongly suspect that the kernel is broken for
CONFIG_E500MC with current development releases of binutils.

The real "solution" is that "e500" needs to be treated as an almost
entirely different architecture from "all other powerpc (including
e500mc)".  Userspace is only ABI-compatible if you use "-msoft-float"
on both sides.

I've been dealing with it for a while now, it really is a fiasco all around=
...

After a month of wrestling with the official Debian "powerpc" port we
gave up and created a new port "powerpcspe" specifically for the
"e500" SPE-based chips.  Then we tripped over 3 relatively major GCC
bugs which had been lurking since e500 support was initially added.

See http://wiki.debian.org/PowerPCSPEPort for a very out-of-date
status on that whole thing.

Cheers,
Kyle Moffett

Re: RFC: x86: kill binutils 2.16.x?

[Segher Boessenkool]

> The problem is not with the kernel compile itself, but with the 2.12
> "dssall" binutils test.  Basically, recent binutils treats e500 as
> effectively a separate architecture that happens to share *most* of
> the opcodes with regular PowerPC.  Any opcode which is not understood
> by the e500 chip is either convert to an equivalent opcode which is
> understood (IE: lwsync => sync), or failed with an error.  This means
> that the kernel compile aborts early telling me to upgrade to a newer
> version of binutils.

$ echo dssall | powerpc-linux-as -many -me500
$ powerpc-linux-objdump -d a.out | grep 0:
   0:   7e 00 06 6c     dssall
$ powerpc-linux-as --version | head -1
GNU assembler (GNU Binutils) 2.21.51.20110309


What version of binutils does not work?  (I also checked with
-me500x2, -me500mc, -mspe, and various combinations.  lwsync
is indeed converted to a regular sync (well, "msync") for e500
and e500x2).


Segher