[Buildroot] Conditional setup of TARGET_ABI

[Buildroot] Conditional setup of TARGET_ABI

[Alexey Brodkin]

Hello,

I'm facing a problem which is both quite simple and obvious as well as
there's a really good solution to it... that's in theory but on practice
it's not that simple.

So the problem is linkage of Qt5Webkit library failing like that:
------------------------>8------------------------
.../output/host/lib/gcc/arc-buildroot-linux-uclibc/7.1.1/crtbeginS.o: In function
'__do_global_dtors_aux':

crtstuff.c:(.text+0x66): relocation truncated to fit: R_ARC_S25W_PCREL
against symbol `__st_r13_to_r15' defined in .text section in
.../output/host/lib/gcc/arc-buildroot-linux-uclibc/7.1.1/libgcc.a(_millicodethunk_st.o)
------------------------>8------------------------

Let me explain what is encrypted in the log above.

1. __do_global_dtors_aux() gets built as a part of libgcc and is supposed to
? ?be then used as it is with all the apps we build later on.
? ?And what's also important our GCC by default tries to use as compact
? ?relocations as possible to save memory footprint, instruction cache
? ?utilization etc. So given libgcc is very compact GCC simply decides
? ?to use so-called R_ARC_S25W_PCREL relocation which means +-16MiB
? ?offset from the current location.

2. Qt5WebKit [as well as all other libs and apps] uses functions from libgcc.
? ?In that case it's our __do_global_dtors_aux() which in its turn uses another
? ?function called __st_r13_to_r15(). And from (1) we know that distance
? ?between them should be less than 16MiB. Now given libQt5WebKit is much larger
? ?that means if the functions in question end-up placed closer to opposite ends
? ?of libQtxxx they won't be reachable to each other... and that's what we see here.

Now what kind of solutions could be applied to that problem?

1. Make sure problematic functions are placed close to each other.
? ?But that will require changes in Qt's build infrastructure... almost
? ?for nothing as then there comes another dependency like that,
? ?so it's an obvious no-go.

2. The best fix is so-called link-time relaxation. This if exists and works
? ?just solves the problem. But the problem is we don't have it yet... even though
? ?it's a long-standing item on out todo list.

3. Not very good but at least feasible and simple solution to rebuild libgcc
? ?such that longer relocations are used. That could be simply done with passing
? ?"-mlong-calls" to the GCC.
? ?But given most of the time shorter relocations work perfectly fine for us
? ?we'd prefer to not use longer relocations all the time... especially knowing
? ?that by default we use so-called "millicode" functions (that for example save and
? ?restore core registers on entry and exit of functions) from libgcc for each and
? ?every function we build it will be pretty significant performance hit system-wise.

? ?But strictly speaking it might be any other library but not only libgcc and so
? ?it IMHO makes sense to build everything with "-mlong-calls".

As a shorter summary:
IMHO the short term solution is to append "-mlong-calls" to TARGET_ABI so this
gcc option gets applied to everything we build for the target like that:
---------------------->8---------------------
ifeq ($(BR2_arc)$(BR2_PACKAGE_QT5WEBKIT),yy)
TARGET_ABI += -mlong-calls
endif
---------------------->8---------------------

Any thoughts?

-Alexey

[Buildroot] Conditional setup of TARGET_ABI

[Thomas Petazzoni]

Hello,

On Fri, 17 Nov 2017 08:23:06 +0000, Alexey Brodkin wrote:

> I'm facing a problem which is both quite simple and obvious as well as
> there's a really good solution to it... that's in theory but on practice
> it's not that simple.

First of all, thanks for summarizing the problem!

> So the problem is linkage of Qt5Webkit library failing like that:
> ------------------------>8------------------------  
> .../output/host/lib/gcc/arc-buildroot-linux-uclibc/7.1.1/crtbeginS.o: In function
> '__do_global_dtors_aux':
> 
> crtstuff.c:(.text+0x66): relocation truncated to fit: R_ARC_S25W_PCREL
> against symbol `__st_r13_to_r15' defined in .text section in
> .../output/host/lib/gcc/arc-buildroot-linux-uclibc/7.1.1/libgcc.a(_millicodethunk_st.o)
> ------------------------>8------------------------  
> 
> Let me explain what is encrypted in the log above.
> 
> 1. __do_global_dtors_aux() gets built as a part of libgcc and is supposed to
> ? ?be then used as it is with all the apps we build later on.
> ? ?And what's also important our GCC by default tries to use as compact
> ? ?relocations as possible to save memory footprint, instruction cache
> ? ?utilization etc. So given libgcc is very compact GCC simply decides
> ? ?to use so-called R_ARC_S25W_PCREL relocation which means +-16MiB
> ? ?offset from the current location.
> 
> 2. Qt5WebKit [as well as all other libs and apps] uses functions from libgcc.
> ? ?In that case it's our __do_global_dtors_aux() which in its turn uses another
> ? ?function called __st_r13_to_r15(). And from (1) we know that distance
> ? ?between them should be less than 16MiB. Now given libQt5WebKit is much larger
> ? ?that means if the functions in question end-up placed closer to opposite ends
> ? ?of libQtxxx they won't be reachable to each other... and that's what we see here.

What isn't clear to me is where are those two functions? One is in
crtbeginS.o, so it ends up linked into the program .text section, while
the other one is in libgcc, correct?

How can the size of QtWebkit have an influence on the distance between
those functions? How can the linker now about the relative locations of
libraries, since they are loaded at runtime by the dynamic linker? Or
perhaps this issue only occurs with BR2_STATIC_LIBS=y ?

> Now what kind of solutions could be applied to that problem?
> 
> 1. Make sure problematic functions are placed close to each other.
> ? ?But that will require changes in Qt's build infrastructure... almost
> ? ?for nothing as then there comes another dependency like that,
> ? ?so it's an obvious no-go.
> 
> 2. The best fix is so-called link-time relaxation. This if exists and works
> ? ?just solves the problem. But the problem is we don't have it yet... even though
> ? ?it's a long-standing item on out todo list.
> 
> 3. Not very good but at least feasible and simple solution to rebuild libgcc
> ? ?such that longer relocations are used. That could be simply done with passing
> ? ?"-mlong-calls" to the GCC.
> ? ?But given most of the time shorter relocations work perfectly fine for us
> ? ?we'd prefer to not use longer relocations all the time... especially knowing
> ? ?that by default we use so-called "millicode" functions (that for example save and
> ? ?restore core registers on entry and exit of functions) from libgcc for each and
> ? ?every function we build it will be pretty significant performance hit system-wise.
> 
> ? ?But strictly speaking it might be any other library but not only libgcc and so
> ? ?it IMHO makes sense to build everything with "-mlong-calls".
> 
> As a shorter summary:
> IMHO the short term solution is to append "-mlong-calls" to TARGET_ABI so this
> gcc option gets applied to everything we build for the target like that:
> ---------------------->8---------------------  
> ifeq ($(BR2_arc)$(BR2_PACKAGE_QT5WEBKIT),yy)
> TARGET_ABI += -mlong-calls
> endif
> ---------------------->8---------------------  
> 
> Any thoughts?

Yes, one thought: this proposed solution of tweaking TARGET_ABI doesn't
work with pre-built external toolchains. Indeed, when you build the
toolchain, you have no idea that it will be used for building "large"
libraries such as Qt5Webkit.

So perhaps we should handle this more explicitly, i.e have an
architecture/toolchain option to enable the use of "long calls", so
that a toolchain can be built with it. Conversely, in the external
toolchain support, have a config knob that allows to say "my toolchain
uses long calls or not". And then packages can depend on it.

Is it possible to easily verify if an external toolchain was built with
-mlong-calls or not?

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [arc-buildroot] Re: Conditional setup of TARGET_ABI

[Alexey Brodkin]

Hi Claus,

On Fri, 2017-11-17 at 10:07 +0000, Claudiu Zissulescu wrote:
> Hello,
> 
> > 
> > > 
> > > 2. Qt5WebKit [as well as all other libs and apps] uses functions from libgcc.
> > > ? ?In that case it's our __do_global_dtors_aux() which in its turn uses
> > another
> > > 
> > > ? ?function called __st_r13_to_r15(). And from (1) we know that distance
> > > ? ?between them should be less than 16MiB. Now given libQt5WebKit is
> > much larger
> > > 
> > > ? ?that means if the functions in question end-up placed closer to opposite
> > ends
> > > 
> > > ? ?of libQtxxx they won't be reachable to each other... and that's what we
> > see here.
> > 
> > What isn't clear to me is where are those two functions? One is in
> > crtbeginS.o, so it ends up linked into the program .text section, while
> > the other one is in libgcc, correct?
> 
> Those functions are some helpers made to replace a normal context save/restore with a call to them. The main purpose is to get a better size figure.
> This technique is used when one builds a binary using -Os compiler option, otherwise, this technique will not be used but only on request.
> 
> > 
> > > 
> > > Any thoughts?
> 
> In my opinion the millicode option should be only used for bare metal applications. For Linux like systems where the size of a binary can be quite
> large, having this technique on, may lead to all kind of complications like the one in question. Thus, my suggestion is to fully disable this option
> for all our architectures in the toolchain. This also means you need to take no action at your side ;)

So your suggestion is to pass "-mno-millicode" option to our GCC for building everything, right?
That will effectively inline millicode stuff right in each and every function compiled by GCC, correct?

-Alexey

[Buildroot] Conditional setup of TARGET_ABI

[Trent Piepho]

On Fri, 2017-11-17 at 08:23 +0000, Alexey Brodkin wrote:
> 
> Let me explain what is encrypted in the log above.
> 
> 1. __do_global_dtors_aux() gets built as a part of libgcc and is supposed to
>    be then used as it is with all the apps we build later on.
>    And what's also important our GCC by default tries to use as compact
>    relocations as possible to save memory footprint, instruction cache
>    utilization etc. So given libgcc is very compact GCC simply decides
>    to use so-called R_ARC_S25W_PCREL relocation which means +-16MiB
>    offset from the current location.
> 
> 2. Qt5WebKit [as well as all other libs and apps] uses functions from libgcc.
>    In that case it's our __do_global_dtors_aux() which in its turn uses another
>    function called __st_r13_to_r15(). And from (1) we know that distance
>    between them should be less than 16MiB. Now given libQt5WebKit is much larger
>    that means if the functions in question end-up placed closer to opposite ends
>    of libQtxxx they won't be reachable to each other... and that's what we see here.

I don't know if this works with current ld, maybe some things have
changed and the assumptions no longer hold.  But what I've been able to
do is to incrementally link multiple objects into one object, so that
the relative distance between code in those objects is fixed at the
incremental link step.  Then the combined object can be come part of a
static library and then linked into a much larger DSO or executable. 
Because ld treats each (section,object) blob as indivisible, the code
inside the incremetally linked objected will not be moved throughout
the larger final product.

In your case, I think one would need to link crtbeginS.o with
_millicodethunk_st.o to produce a replacement crtbeginS.o file.