[PATCH] ARM: move __fixup_smp out of init section

[PATCH] ARM: move __fixup_smp out of init section

[Rob Herring]

From: Rob Herring <robh@kernel.org>

With large kernel builds such as allyesconfig exceeding maximum relative
branch offsets, the init section will be too far away to branch to
directly. This causes veneers to be added by the compiler, but veneers
don't work before the MMU is enabled. Fix this by moving __fixup_smp to
the .head.text section as it is not very big.

Signed-off-by: Rob Herring <robh@kernel.org>
---
 arch/arm/kernel/head.S | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/arm/kernel/head.S b/arch/arm/kernel/head.S
index 914616e..78e1443 100644
--- a/arch/arm/kernel/head.S
+++ b/arch/arm/kernel/head.S
@@ -475,7 +475,7 @@ ENDPROC(__turn_mmu_on)
 
 
 #ifdef CONFIG_SMP_ON_UP
-	__INIT
+	__HEAD
 __fixup_smp:
 	and	r3, r9, #0x000f0000	@ architecture version
 	teq	r3, #0x000f0000		@ CPU ID supported?
-- 
1.8.3.2

[PATCH] ARM: move __fixup_smp out of init section

[Arnd Bergmann]

On Tuesday 25 February 2014 12:16:33 Rob Herring wrote:
> From: Rob Herring <robh@kernel.org>
> 
> With large kernel builds such as allyesconfig exceeding maximum relative
> branch offsets, the init section will be too far away to branch to
> directly. This causes veneers to be added by the compiler, but veneers
> don't work before the MMU is enabled. Fix this by moving __fixup_smp to
> the .head.text section as it is not very big.
> 
> Signed-off-by: Rob Herring <robh@kernel.org>

This looks good to me, but I have some related questions:

* I needed to use -mlong-calls for large kernels. Did you manage without?

* Do you (or anyone) know how to force the use of a veneer? I see a problem
  with some driver calling __do_div_asm from IIRC an init section, and that
  creates a link error when the kernel gets too big

* If FUNCTION_TRACER is enabled, we get calls from every function using
 'bl __gnu_mcount_nc'. Do you think it's possible to fix those?

* Same thing but simpler for svc_preempt calling preempt_schedule_irq
  and lookup_processor_type calling __lookup_processor_type. In those
  cases I guess we should be able to trivially rewrite the assembly
  to jump through an extra register.

The complete patch I have is at http://pastebin.com/5yVaUC4u, but it's
likely that it's completely broken ;-)

	Arnd

[PATCH] ARM: move __fixup_smp out of init section

[Rob Herring]

On Tue, Feb 25, 2014 at 12:32 PM, Arnd Bergmann <arnd@arndb.de> wrote:
> On Tuesday 25 February 2014 12:16:33 Rob Herring wrote:
>> From: Rob Herring <robh@kernel.org>
>>
>> With large kernel builds such as allyesconfig exceeding maximum relative
>> branch offsets, the init section will be too far away to branch to
>> directly. This causes veneers to be added by the compiler, but veneers
>> don't work before the MMU is enabled. Fix this by moving __fixup_smp to
>> the .head.text section as it is not very big.
>>
>> Signed-off-by: Rob Herring <robh@kernel.org>
>
> This looks good to me, but I have some related questions:
>
> * I needed to use -mlong-calls for large kernels. Did you manage without?

Yes, but I started with allyesconfig and then disabled things until it
would link. This was mainly PCI, various debug, and filesystems. This
particular failure you don't see until you try to boot.

I also saw intermittent inconsistent kallsym errors which
KALLSYMS_EXTRA_PASS=1 did not fix.

> * Do you (or anyone) know how to force the use of a veneer? I see a problem
>   with some driver calling __do_div_asm from IIRC an init section, and that
>   creates a link error when the kernel gets too big

That is surprising given that I saw veneers be inserted for some asm
to asm calls. Maybe it is something I turned off or something to do
with the types of sections.

> * If FUNCTION_TRACER is enabled, we get calls from every function using
>  'bl __gnu_mcount_nc'. Do you think it's possible to fix those?

Put it in the middle of the image? ;)

> * Same thing but simpler for svc_preempt calling preempt_schedule_irq
>   and lookup_processor_type calling __lookup_processor_type. In those
>   cases I guess we should be able to trivially rewrite the assembly
>   to jump through an extra register.

I did see that lookup_processor_type had a veneer in my build.

Rob

[PATCH] ARM: move __fixup_smp out of init section

[Arnd Bergmann]

On Tuesday 25 February 2014 15:05:01 Rob Herring wrote:
> On Tue, Feb 25, 2014 at 12:32 PM, Arnd Bergmann <arnd@arndb.de> wrote:
> > On Tuesday 25 February 2014 12:16:33 Rob Herring wrote:
> >> From: Rob Herring <robh@kernel.org>
> >>
> >> With large kernel builds such as allyesconfig exceeding maximum relative
> >> branch offsets, the init section will be too far away to branch to
> >> directly. This causes veneers to be added by the compiler, but veneers
> >> don't work before the MMU is enabled. Fix this by moving __fixup_smp to
> >> the .head.text section as it is not very big.
> >>
> >> Signed-off-by: Rob Herring <robh@kernel.org>
> >
> > This looks good to me, but I have some related questions:
> >
> > * I needed to use -mlong-calls for large kernels. Did you manage without?
> 
> Yes, but I started with allyesconfig and then disabled things until it
> would link. This was mainly PCI, various debug, and filesystems. This
> particular failure you don't see until you try to boot.
> 
> I also saw intermittent inconsistent kallsym errors which
> KALLSYMS_EXTRA_PASS=1 did not fix.

Yes, I saw those too. With XIP_KERNEL enabled, I've seen a randconfig case
that needed 12 extra passes before converging. Without that option,
it's much better but there are still cases that need more than 3 passes
and I haven't gotten to the bottom of that. It seems to be the same
configuration that refuses to link at all with ld.gold.

> > * Do you (or anyone) know how to force the use of a veneer? I see a problem
> >   with some driver calling __do_div_asm from IIRC an init section, and that
> >   creates a link error when the kernel gets too big
> 
> That is surprising given that I saw veneers be inserted for some asm
> to asm calls. Maybe it is something I turned off or something to do
> with the types of sections.
> 
> > * If FUNCTION_TRACER is enabled, we get calls from every function using
> >  'bl __gnu_mcount_nc'. Do you think it's possible to fix those?
> 
> Put it in the middle of the image? ;)
> 
> > * Same thing but simpler for svc_preempt calling preempt_schedule_irq
> >   and lookup_processor_type calling __lookup_processor_type. In those
> >   cases I guess we should be able to trivially rewrite the assembly
> >   to jump through an extra register.
> 
> I did see that lookup_processor_type had a veneer in my build.

Strange. Could it be that -mlong-calls turns off veneers?

Which gcc and binutils version do you use?

	Arnd

[PATCH] ARM: move __fixup_smp out of init section

[Rob Herring]

On Tue, Feb 25, 2014 at 3:10 PM, Arnd Bergmann <arnd@arndb.de> wrote:
> On Tuesday 25 February 2014 15:05:01 Rob Herring wrote:
>> On Tue, Feb 25, 2014 at 12:32 PM, Arnd Bergmann <arnd@arndb.de> wrote:
>> > On Tuesday 25 February 2014 12:16:33 Rob Herring wrote:
>> >> From: Rob Herring <robh@kernel.org>
>> >>
>> >> With large kernel builds such as allyesconfig exceeding maximum relative
>> >> branch offsets, the init section will be too far away to branch to
>> >> directly. This causes veneers to be added by the compiler, but veneers
>> >> don't work before the MMU is enabled. Fix this by moving __fixup_smp to
>> >> the .head.text section as it is not very big.
>> >>
>> >> Signed-off-by: Rob Herring <robh@kernel.org>
>> >
>> > This looks good to me, but I have some related questions:
>> >
>> > * I needed to use -mlong-calls for large kernels. Did you manage without?
>>
>> Yes, but I started with allyesconfig and then disabled things until it
>> would link. This was mainly PCI, various debug, and filesystems. This
>> particular failure you don't see until you try to boot.
>>
>> I also saw intermittent inconsistent kallsym errors which
>> KALLSYMS_EXTRA_PASS=1 did not fix.
>
> Yes, I saw those too. With XIP_KERNEL enabled, I've seen a randconfig case
> that needed 12 extra passes before converging. Without that option,
> it's much better but there are still cases that need more than 3 passes
> and I haven't gotten to the bottom of that. It seems to be the same
> configuration that refuses to link at all with ld.gold.
>
>> > * Do you (or anyone) know how to force the use of a veneer? I see a problem
>> >   with some driver calling __do_div_asm from IIRC an init section, and that
>> >   creates a link error when the kernel gets too big
>>
>> That is surprising given that I saw veneers be inserted for some asm
>> to asm calls. Maybe it is something I turned off or something to do
>> with the types of sections.
>>
>> > * If FUNCTION_TRACER is enabled, we get calls from every function using
>> >  'bl __gnu_mcount_nc'. Do you think it's possible to fix those?
>>
>> Put it in the middle of the image? ;)
>>
>> > * Same thing but simpler for svc_preempt calling preempt_schedule_irq
>> >   and lookup_processor_type calling __lookup_processor_type. In those
>> >   cases I guess we should be able to trivially rewrite the assembly
>> >   to jump through an extra register.
>>
>> I did see that lookup_processor_type had a veneer in my build.
>
> Strange. Could it be that -mlong-calls turns off veneers?
>
> Which gcc and binutils version do you use?

It's the arm cross compiler in ubuntu 13.10:

$ arm-linux-gnueabihf-ld -v
GNU ld (GNU Binutils for Ubuntu) 2.23.52.20130913
$ arm-linux-gnueabihf-gcc -v
Using built-in specs.
COLLECT_GCC=arm-linux-gnueabihf-gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc-cross/arm-linux-gnueabihf/4.8/lto-wrapper
Target: arm-linux-gnueabihf
Configured with: ../src/configure -v --with-pkgversion='Ubuntu/Linaro
4.8.1-10ubuntu7'
--with-bugurl=file:///usr/share/doc/gcc-4.8/README.Bugs
--enable-languages=c,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr
--program-suffix=-4.8 --enable-shared --enable-linker-build-id
--libexecdir=/usr/lib --without-included-gettext
--enable-threads=posix
--with-gxx-include-dir=/usr/arm-linux-gnueabihf/include/c++/4.8.1
--libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu
--enable-libstdcxx-debug --enable-libstdcxx-time=yes
--enable-gnu-unique-object --disable-libitm --disable-libquadmath
--enable-plugin --with-system-zlib --disable-browser-plugin
--enable-java-awt=gtk --enable-gtk-cairo
--with-java-home=/usr/lib/jvm/java-1.5.0-gcj-4.8-armhf-cross/jre
--enable-java-home
--with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-4.8-armhf-cross
--with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-4.8-armhf-cross
--with-arch-directory=arm
--with-ecj-jar=/usr/share/java/eclipse-ecj.jar --disable-libgcj
--enable-objc-gc --enable-multiarch --enable-multilib
--disable-sjlj-exceptions --with-arch=armv7-a --with-fpu=vfpv3-d16
--with-float=hard --with-mode=thumb --disable-werror
--enable-checking=release --build=x86_64-linux-gnu
--host=x86_64-linux-gnu --target=arm-linux-gnueabihf
--program-prefix=arm-linux-gnueabihf-
--includedir=/usr/arm-linux-gnueabihf/include
Thread model: posix
gcc version 4.8.1 (Ubuntu/Linaro 4.8.1-10ubuntu7)

Rob

[PATCH] ARM: move __fixup_smp out of init section

[Arnd Bergmann]

On Tuesday 25 February 2014 15:46:15 Rob Herring wrote:
> 
> It's the arm cross compiler in ubuntu 13.10:
> 
> $ arm-linux-gnueabihf-ld -v
> GNU ld (GNU Binutils for Ubuntu) 2.23.52.20130913
> $ arm-linux-gnueabihf-gcc -v
> gcc version 4.8.1 (Ubuntu/Linaro 4.8.1-10ubuntu7)
> 

Ok, I'm using the exact same one here.

	Arnd

[PATCH] ARM: move __fixup_smp out of init section

[Russell King - ARM Linux]

On Tue, Feb 25, 2014 at 12:16:33PM -0600, Rob Herring wrote:
> From: Rob Herring <robh@kernel.org>
> 
> With large kernel builds such as allyesconfig exceeding maximum relative
> branch offsets, the init section will be too far away to branch to
> directly. This causes veneers to be added by the compiler, but veneers

Don't you mean the linker?  The compiler isn't involved here.  Can
you show the disassembly of the veneer please?

I'm confused why the linker would add a veneer for this, but not for
the stuff built by the compiler.

-- 
FTTC broadband for 0.8mile line: now at 9.7Mbps down 460kbps up... slowly
improving, and getting towards what was expected from it.

[PATCH] ARM: move __fixup_smp out of init section

[Rob Herring]

On Wed, Feb 26, 2014 at 6:24 AM, Russell King - ARM Linux
<linux@arm.linux.org.uk> wrote:
> On Tue, Feb 25, 2014 at 12:16:33PM -0600, Rob Herring wrote:
>> From: Rob Herring <robh@kernel.org>
>>
>> With large kernel builds such as allyesconfig exceeding maximum relative
>> branch offsets, the init section will be too far away to branch to
>> directly. This causes veneers to be added by the compiler, but veneers
>
> Don't you mean the linker?  The compiler isn't involved here.  Can
> you show the disassembly of the veneer please?

Here is the call to fixup_smp:

c0008054:       eb000058        bl      c00081bc <__vet_atags>
c0008058:       eb00006a        bl      c0008208 <____fixup_smp_veneer>
c000805c:       eb000042        bl      c000816c <__fixup_pv_table>
c0008060:       eb000007        bl      c0008084 <__create_page_tables>

Here's the veneer:

c0008208 <____fixup_smp_veneer>:
c0008208:       e51ff004        ldr     pc, [pc, #-4]   ; c000820c
<____fixup_smp_veneer+0x4>
c000820c:       c2659260        .word   0xc2659260

>
> I'm confused why the linker would add a veneer for this, but not for
> the stuff built by the compiler.

There are 1413 other symbols with veneer in my image. So it is not
that the linker can't create them, but why it cannot for some as the
image gets bigger. Does "text" in a section name have any functional
impact on the linker? It seems veneers between .head.text and and
init.text are created, but between *.text and .fixup, .proc.info, etc.
they don't get created.

Rob