arm kernel configurations

arm kernel configurations

[Phil Blundell]

On the subject of gta01 performance I thought it might be worth pointing
out a couple of common kernel inefficiencies:

- out of the box, the arm kernel always compiles itself with APCS-style
frame pointers enabled.  This means that you lose a general purpose
register, and makes the function prologue and epilogue sequences more
heavyweight.  For production use I would always recommend turning off
frame pointers (which involves patching arch/arm/Kconfig.debug) in order
to get a smaller and faster kernel.

- if your userspace is pure EABI, make sure that you have
CONFIG_OABI_COMPAT turned off.  With this option enabled, the kernel
needs to inspect all SWI instructions to see if they might be OABI
syscalls: on machines with Harvard caches, this will almost certainly
involve hauling a cacheline in from main memory, which is slow and leads
to poor dcache utilisation (since you are loading eight words of which
only one is likely to be any use).

- if your userspace doesn't depend on load-rotate (most doesn't
nowadays) then you can hard-wire the alignment trap behaviour to be
SIGBUS and avoid reloading cp15 on every kernel entry.  This saves
another two lines in the dcache.  I don't have a pretty patch for this
yet but contributions would be welcome :-)

These three changes make a measurable difference to the kernel benchmark
results.  For example, syscall entry/exit drops from about 490ns to
about 390ns on my gta01, an improvement of more than 20%.  The cost of
stat() drops from 36.8us to 33.2us, an improvement of 14%.

share and enjoy

p.

Re: arm kernel configurations

[Tom Talpey]

At 11:26 AM 10/29/2008, Phil Blundell wrote:
>- if your userspace is pure EABI, make sure that you have
>CONFIG_OABI_COMPAT turned off.

My kernel(s) have all hung during exec of init when trying this.
Are you able to turn this off with non-thumb userspace?

One side effect of turning off OABI_COMPAT is that the floating
point emulation also vanishes. I get the same behavior if I leave
OABI_COMPAT on and play with fp settings.

This is mostly with 2.6.27 on my iPAQ h1940 (s3c2410). Same
as GTA01, right?

Tom.

Re: arm kernel configurations

[Phil Blundell]

On Thu, 2008-10-30 at 10:40 -0400, Tom Talpey wrote:
> At 11:26 AM 10/29/2008, Phil Blundell wrote:
> >- if your userspace is pure EABI, make sure that you have
> >CONFIG_OABI_COMPAT turned off.
> 
> My kernel(s) have all hung during exec of init when trying this.
> Are you able to turn this off with non-thumb userspace?

Yes, it doesn't require Thumb.  Are you sure that your userspace really
is compiled for EABI and not OABI?

> One side effect of turning off OABI_COMPAT is that the floating
> point emulation also vanishes. I get the same behavior if I leave
> OABI_COMPAT on and play with fp settings.

That sounds rather like some part of your userspace has been built with
hard-fpa floating point.  FPA is basically incompatible with EABI (and I
think all extant EABI configs in OE are soft-float) so this would also
indicate that you are actually using OABI.

> This is mostly with 2.6.27 on my iPAQ h1940 (s3c2410). Same
> as GTA01, right?

Right.

p.

Re: arm kernel configurations

[Tom Talpey]

At 11:08 AM 10/30/2008, Phil Blundell wrote:
>On Thu, 2008-10-30 at 10:40 -0400, Tom Talpey wrote:
>> At 11:26 AM 10/29/2008, Phil Blundell wrote:
>> >- if your userspace is pure EABI, make sure that you have
>> >CONFIG_OABI_COMPAT turned off.
>> 
>> My kernel(s) have all hung during exec of init when trying this.
>> Are you able to turn this off with non-thumb userspace?
>
>Yes, it doesn't require Thumb.  Are you sure that your userspace really
>is compiled for EABI and not OABI?

Yes:

>>[tmt@tmt bin]$ ./arm-angstrom-linux-gnueabi-gcc -v
>>Using built-in specs.
>>Target: arm-angstrom-linux-gnueabi
>>Configured with: /home/tmt/armdev/H1940/oe/build-h1940-gpe/tmp/work/armv4t-angstrom-linux-gnueabi/gcc-cross-4.2.4-r1/gcc-4.2.4/configure --build=i686-linux --host=i686-linux --target=arm-angstrom-linux-gnueabi --prefix=/home/tmt/armdev/H1940/oe/build-h1940-gpe/tmp/cross --exec_prefix=/home/tmt/armdev/H1940/oe/build-h1940-gpe/tmp/cross --bindir=/home/tmt/armdev/H1940/oe/build-h1940-gpe/tmp/cross/bin --sbindir=/home/tmt/armdev/H1940/oe/build-h1940-gpe/tmp/cross/bin --libexecdir=/home/tmt/armdev/H1940/oe/build-h1940-gpe/tmp/cross/libexec --datadir=/home/tmt/armdev/H1940/oe/build-h1940-gpe/tmp/cross/share --sysconfdir=/home/tmt/armdev/H1940/oe/build-h1940-gpe/tmp/cross/etc --sharedstatedir=/home/tmt/armdev/H1940/oe/build-h1940-gpe/tmp/cross/com --localstatedir=/home/tmt/armdev/H1940/oe/build-h1940-gpe/tmp/cross/var --libdir=/home/tmt/armdev/H1940/oe/build-h1940-gpe/tmp/cross/lib --includedir=/home/tmt/armdev/H1940/oe/build-h1940-gpe/tmp/cross/include --oldincludedir=/home/tmt/armdev/H1940/oe/build-h1940-gpe/tmp/cross/include --infodir=/home/tmt/armdev/H1940/oe/build-h1940-gpe/tmp/cross/share/info --mandir=/home/tmt/armdev/H1940/oe/build-h1940-gpe/tmp/cross/share/man --with-gnu-ld --enable-shared --enable-target-optspace --enable-languages=c,c++,fortran --enable-threads=posix --enable-multilib --enable-c99 --enable-long-long --enable-symvers=gnu --enable-libstdcxx-pch --program-prefix=arm-angstrom-linux-gnueabi- --enable-libssp --disable-bootstrap --disable-libgomp --disable-libmudflap --with-float=soft --with-local-prefix=/home/tmt/armdev/H1940/oe/build-h1940-gpe/tmp/staging/arm-angstrom-linux-gnueabi/usr --with-gxx-include-dir=/home/tmt/armdev/H1940/oe/build-h1940-gpe/tmp/staging/arm-angstrom-linux-gnueabi//usr/include/c++ --with-sysroot=/home/tmt/armdev/H1940/oe/build-h1940-gpe/tmp/staging/arm-angstrom-linux-gnueabi --with-build-sysroot=/home/tmt/armdev/H1940/oe/build-h1940-gpe/tmp/staging/arm-angstrom-linux-gnueabi --disable-libunwind-exceptions --with-mpfr=/home/tmt/armdev/H1940/oe/build-h1940-gpe/tmp/staging/i686-linux/usr --enable-__cxa_atexit
>>Thread model: posix
>>gcc version 4.2.4
>>[tmt@tmt bin]$ 

>> One side effect of turning off OABI_COMPAT is that the floating
>> point emulation also vanishes. I get the same behavior if I leave
>> OABI_COMPAT on and play with fp settings.
>
>That sounds rather like some part of your userspace has been built with
>hard-fpa floating point.  FPA is basically incompatible with EABI (and I
>think all extant EABI configs in OE are soft-float) so this would also
>indicate that you are actually using OABI.

As above - the specs do include "--with-float=soft". I guess I'll have to
dig some more. Do I also need -mfloat-abi=softfp in the machine.conf like
some of the other arm's?

BTW, on the subject of turning off the kernel frame pointer, I certainly agree
it's worth it. For one thing, the tiny screen never manages to show me enough
of a backtrace on oops anyway.

But the Kconfig comment does warn that RMK will kill us! ;-)

Tom.  

Re: arm kernel configurations

[Phil Blundell]

On Thu, 2008-10-30 at 12:45 -0400, Tom Talpey wrote:
> As above - the specs do include "--with-float=soft". I guess I'll have to
> dig some more. Do I also need -mfloat-abi=softfp in the machine.conf like
> some of the other arm's?

I don't think so.  If I remember right, -mfloat-abi=softfp is for the
case where you want to use hard floating point instructions (for speed)
but still want to retain compatibility with a soft-float ABI.

I wonder if something has gone wrong with your glibc configuration.  Try
disassembling one of your binaries and see if you can find any reference
to "wfs" or "rfs" instructions.  Also have a look in libc.so.6 itself to
see what the syscall instructions look like: for OABI you will see
something like "swi 0x900004", whereas for EABI it will look more like
"mov ip, r7; mov r7, #4; swi 0; mov r7, ip".

> But the Kconfig comment does warn that RMK will kill us! ;-)

Heh.  True, but only if he finds out :-)

p.

Re: arm kernel configurations

[Tom Talpey]

At 01:13 PM 10/30/2008, Phil Blundell wrote:
>On Thu, 2008-10-30 at 12:45 -0400, Tom Talpey wrote:
>> As above - the specs do include "--with-float=soft". I guess I'll have to
>> dig some more. Do I also need -mfloat-abi=softfp in the machine.conf like
>> some of the other arm's?
>
>I don't think so.  If I remember right, -mfloat-abi=softfp is for the
>case where you want to use hard floating point instructions (for speed)
>but still want to retain compatibility with a soft-float ABI.
>
>I wonder if something has gone wrong with your glibc configuration.  Try
>disassembling one of your binaries and see if you can find any reference
>to "wfs" or "rfs" instructions.  Also have a look in libc.so.6 itself to
>see what the syscall instructions look like: for OABI you will see
>something like "swi 0x900004", whereas for EABI it will look more like
>"mov ip, r7; mov r7, #4; swi 0; mov r7, ip".

Looks like EABI to me then, objdump -d libc.so.6 sample syscall:

0002a0c0 <kill>:
   2a0c0:       e1a0c007        mov     ip, r7
   2a0c4:       e3a07025        mov     r7, #37 ; 0x25
   2a0c8:       ef000000        svc     0x00000000
   2a0cc:       e1a0700c        mov     r7, ip
   2a0d0:       e3700a01        cmn     r0, #4096       ; 0x1000
   2a0d4:       312fff1e        bxcc    lr
   2a0d8:       eaffaa98        b       14b40 <__libc_start_main+0x248>
   2a0dc:       e1a00000        nop                     (mov r0,r0)

And no wfs/rfs at all in any binaries. Hmm, let me try the no-OABI
kernel again. Maybe I can get more status from it. Basically it would
say "Freeing init memory" then nothing more, while with-OABI it would
cruise.

Tom.

Re: arm kernel configurations

[Tom Talpey]

At 02:09 PM 10/30/2008, Tom Talpey wrote:
>At 01:13 PM 10/30/2008, Phil Blundell wrote:
>>I wonder if something has gone wrong with your glibc configuration.  Try
>>disassembling one of your binaries and see if you can find any reference
>
>And no wfs/rfs at all in any binaries. Hmm, let me try the no-OABI
>kernel again. Maybe I can get more status from it. Basically it would
>say "Freeing init memory" then nothing more, while with-OABI it would
>cruise.

Interesting. This time it booted all the way to GPE. But, hid2hci
faulted and the kernel trapped on it. Nothing strange at all about
that binary, and the same one works fine on the OABI kernel.

Well, bottom line I don't think I'm doing anything wrong, but I can
say leaving out the OABI option always gives me trouble. Any light
you can shine would be great!


>root@h1940:~# dmesg
>Linux version 2.6.27.4-h1940-tmt (tmt@tmt) (gcc version 4.2.4) #14 PREEMPT Thu Oct 30 14:20:25 EDT 2008
>CPU: ARM920T [41129200] revision 0 (ARMv4T), cr=c0007177
>Machine: IPAQ-H1940
>...
>VFS: Mounted root (ext2 filesystem).
>Freeing init memory: 120K
>udevd version 124 started
>Unable to handle kernel paging request at virtual address 560000b8
>pgd = c3b38000
>[560000b8] *pgd=00000000
>Internal error: Oops: 5 [#1] PREEMPT
>CPU: 0    Not tainted  (2.6.27.4-h1940-tmt #14)
>PC is at do_coredump+0x24/0x74c
>LR is at get_signal_to_deliver+0x344/0x3a0
>pc : [<c0081004>]    lr : [<c0044ec8>]    psr: 60000013
>sp : c34a3db0  ip : c3b52e00  fp : c38ffaa0
>r10: c3b52e00  r9 : c3bc9ff0  r8 : c34a3f68
>r7 : c34a3ee8  r6 : 00000000  r5 : 000000dc  r4 : c34a2000
>r3 : c346f6a0  r2 : c34a3fb0  r1 : 0000000b  r0 : 56000000
>Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
>Control: c000717f  Table: 33b38000  DAC: 00000015
>Process hid2hci (pid: 976, stack limit = 0xc34a2268)
>Stack: (0xc34a3db0 to 0xc34a4000)
>3da0:                                     00000090 c02f2ca8 ffffffff c34a3dc8 
>3dc0: c00797f8 c00330cc 00000004 00000020 c34a3dec c34a3de0 c003473c c0033178 
>3de0: c34a3e14 c34a3df0 c0034b18 c00330cc c0043068 60000093 c3b43bcc c3b43bc0 
>3e00: c34a3e50 0000000b c34a3fb0 c34a3e18 c00433b4 c0034a70 c346f6a0 80000013 
>3e20: 0000000b c3bca0cc c34a3fb0 c34a3e38 c0043694 c00330cc c34a3fb0 c34a3e48 
>3e40: c00795f8 c00330cc c3b43bc0 c34a3ee8 c346f8f4 0000000b c34a3f68 c3bc9ff0 
>3e60: c34a3ee8 c0043f98 c3ae5000 c346f8dc 400367cc c0044074 c346f8dc c0044280 
>3e80: c34a2000 00000001 c3bcb000 0000000b 000000dc c34a2000 c34a3ee8 c34a3f68 
>3ea0: c3bc9ff0 c3b52e00 c34a3fb0 c0044ec8 00000000 00000000 400367cc c34a3fb0 
>3ec0: 40041174 00000000 c34a2000 00000000 2a00a984 c00291d4 00030001 c34a3fb0 
>3ee0: 00000000 c346f8dc 0000000b 00000000 00030001 00000000 c34a3e70 c3ae5000 
>3f00: c3ae5000 c00847a4 2a00a984 c00261e4 c0304788 00000107 00000001 00000000 
>3f20: 00000000 00000010 00000001 00000000 00000002 c34b4000 c34a3fac 40038480 
>3f40: c38000dc 00024801 00000000 c3b43b40 c3bcbe80 00000003 2a00a984 c34a3f68 
>3f60: 2a00a984 c34a3f70 c0061548 c00330cc 00024800 00000138 fffffffe fffffffe 
>3f80: c3ae5000 c0079e1c f4000000 ffffffff 400367cc bec19c1c 40041174 00000000 
>3fa0: c34a2000 00000000 2a00a984 c0026c8c 00000000 00000003 00000140 4018897c 
>3fc0: 40188600 400367cc bec19c1c 40041174 000010e8 2a00b008 00000000 2a00a984 
>3fe0: 40041174 bec18b20 40038d7c 40031dac 60000010 ffffffff 00000000 00000000 
>Backtrace: 
>Code: e3a06000 e593c234 e593b0ac e3a00456 (e590f0b8) 
>---[ end trace 44c9aec1cc73ba24 ]---
>Alignment trap: gpe-confd (1082) PC=0x401d71a8 Instr=0x15840000 Address=0xbe9401e3 FSR 0x813
>g_ether gadget: full speed config #1: CDC Ethernet (ECM)
>root@h1940:~# 
>root@h1940:~# 


Thanks,

Tom.

Re: arm kernel configurations

[Phil Blundell]

On Thu, 2008-10-30 at 14:09 -0400, Tom Talpey wrote:
>    2a0c0:       e1a0c007        mov     ip, r7
>    2a0c4:       e3a07025        mov     r7, #37 ; 0x25
>    2a0c8:       ef000000        svc     0x00000000
>    2a0cc:       e1a0700c        mov     r7, ip

Yeah, that is definitely an EABI syscall.

> And no wfs/rfs at all in any binaries. Hmm, let me try the no-OABI
> kernel again. Maybe I can get more status from it. Basically it would
> say "Freeing init memory" then nothing more, while with-OABI it would
> cruise.

Odd.  Might also be worth enabling CONFIG_DEBUG_USER (and passing
user_debug=255 on the command line) to see if you can get any more
detail about what's going wrong.

p.

Re: arm kernel configurations

[Phil Blundell]

On Thu, 2008-10-30 at 14:37 -0400, Tom Talpey wrote:
> Interesting. This time it booted all the way to GPE. But, hid2hci
> faulted and the kernel trapped on it. Nothing strange at all about
> that binary, and the same one works fine on the OABI kernel.

That's very strange indeed.  Does it do the same thing every time or is
it more random than that?

> >Unable to handle kernel paging request at virtual address 560000b8

This doesn't seem to have any obvious relation at all to the OABI
setting.  There's no reason why the kernel should be trying to access
that memory address during a core dump.  At a first appearance I would
say that you have some kind of hardware problem (misconfigured memory or
the like) but obviously in that case you would expect it to fail under
OABI too.  So, all in all, very weird.

p.

Re: arm kernel configurations

[Tom Talpey]

At 03:01 PM 10/30/2008, Phil Blundell wrote:
>On Thu, 2008-10-30 at 14:37 -0400, Tom Talpey wrote:
>> Interesting. This time it booted all the way to GPE. But, hid2hci
>> faulted and the kernel trapped on it. Nothing strange at all about
>> that binary, and the same one works fine on the OABI kernel.
>
>That's very strange indeed.  Does it do the same thing every time or is
>it more random than that?

Perfectly repeatable.

>> >Unable to handle kernel paging request at virtual address 560000b8
>
>This doesn't seem to have any obvious relation at all to the OABI
>setting.  There's no reason why the kernel should be trying to access
>that memory address during a core dump.  At a first appearance I would
>say that you have some kind of hardware problem (misconfigured memory or
>the like) but obviously in that case you would expect it to fail under
>OABI too.  So, all in all, very weird.

I don't think it's hardware, because it works fine at all other times (i.e.
booting other kernels and no other changes). And, there's nothing really
to mess up on the H1940 configuration-wise.

The good news for me is that I'm having really good success running a
new 2.6.27 kernel. I've done a ton of stuff to it, but having the new
upstream s3c mci driver is a huge boon to making it work properly. I also
got sound working, by writing some code to use the uda1380 codec driver
already upstream, and turning on "alsa" in the oe machine.conf.

Except for a bootloader, reading the battery level and this OABI strangeness
(plus a couple of other kernel config glitches) I dare say it's a complete port.
Almost ready to share it...

Tom.