Prelink problems -- need help!

Prelink problems -- need help!

[Mark Hatle]

I'm cross posting this in an effort to ask for help for the cross-prelink
project.  (If you are unaware of what the prelinker does, it goes through and
adjusts the relocation addresses to predefined values in an attempt to speed up
executable load times.  For embedded systems, this can be fairly critical in
meeting boot and runtime load performance targets.)

As some people might be aware, the upstream 'prelink' project that was
maintained by Jakub Jelinek at Red Hat appears to have been abandoned.

I've been the maintainer for the Yocto Project's 'prelink-cross' project for a
few years, but my work has been dealing with compilation issues and other things
that have to do with using the prelinker in a cross compilation environment.

This means that unfortunately I'm in a situation where I am having problems
trying to fix the problems in the prelinker itself, but I am not an ELF expert.
 Currently the cross prelinker appears to be broken for pretty much every
architecture and I need help trying to resolve these problems.

Issues I know of right now:

Introduction of RTYPE_CLASS_EXTERN_PROTECTED_DATA appears to be causing failures
on ARM32, x86 and x86_64.

Some possibly unrelated issue is causing various items to relocate incorrectly
(IFUNCs perhaps) on ARMv7 builds.

Power is no longer working properly, most likely due to an optimization in the
way the PPC GOT table(s) are setup.

Power64 hasn't been working properly for a while, possibly due to the switch to
ELFv2.

MIPS and MIPS64 also do not appear to be working, and I've not found a possible
cause.

As you can see, pretty much everything is currently broken and I'm struggling to
find community people with the right skill sets to help me resolve the issues.

The current state of my work in progress is available at:

git://git.yoctoproject.org/prelink-cross

branch cross_prelink_staging

If you think you can help, please let me know.
--Mark

Re: Prelink problems -- need help!

[Phil Blundell]

[Cc list trimmed]

On Mon, 2015-10-26 at 09:28 -0500, Mark Hatle wrote:
> As you can see, pretty much everything is currently broken and I'm
> struggling to
> find community people with the right skill sets to help me resolve
> the issues.

I think it's true that prelink is a technology which has largely had
its day, and I could well imagine that upstream Red Hat is no longer
very interested in it.  https://fedorahosted.org/fesco/ticket/1183 has
a summary of some of the reasons why.

Before investing any significant amount of time in fixing up prelink
(and from what you've described it sounds like it might indeed be a
significant effort to get it working again on all platforms) it might
be a good idea to make sure we are still convinced that prelink
provides useful and meaningful benefits for OE-core users.

If the answer to that is yes then I probably could spare a bit of time
to look at the ARM and MIPS issues at least. 

p.

Re: Prelink problems -- need help!

[Mark Hatle]

On 10/26/15 11:10 AM, Phil Blundell wrote:
> [Cc list trimmed]
> 
> On Mon, 2015-10-26 at 09:28 -0500, Mark Hatle wrote:
>> As you can see, pretty much everything is currently broken and I'm
>> struggling to
>> find community people with the right skill sets to help me resolve
>> the issues.
> 
> I think it's true that prelink is a technology which has largely had
> its day, and I could well imagine that upstream Red Hat is no longer
> very interested in it.  https://fedorahosted.org/fesco/ticket/1183 has
> a summary of some of the reasons why.
> 
> Before investing any significant amount of time in fixing up prelink
> (and from what you've described it sounds like it might indeed be a
> significant effort to get it working again on all platforms) it might
> be a good idea to make sure we are still convinced that prelink
> provides useful and meaningful benefits for OE-core users.
> 

I do think prelink has a purpose.  Especially for embedded systems.

The biggest reason I see against prelink is ASLR.  I agree 100% that if you want
ASLR (Address Randomization) you should not be using the prelinker.

Usually secondary reasons include if you are doing field upgrades at a package
level -- it means you also need the ability to prelink on the device which cause
have other performance ramifications and such during this upgrade process.

I agree with some of the information on the Fedora hosted link you pointed to,
but disagree with others.

While some of the modern hashing techniques and such do improve run-time dynamic
link performance, there is still a hit that we must take.  For devices that need
quick boot times, quick startup, or are memory constrained, the prelinker can
still help.  (Memory usage on very small systems is a good example.  Memory
usage can be reduced in larger applications by reducing the number of
Copy-on-write pages required to handle the relocation information.)

I probably would not use the prelinker on a workstation/server any longer.  I
might not use it on a 'larger' embedded system, but on a more traditional
embedded system I do think it still has merit.

(Probably a discussion for the future, due to the ASLR, is should the prelinker
be enabled by default -- even if it works -- but that is a discussion for later.)

> If the answer to that is yes then I probably could spare a bit of time
> to look at the ARM and MIPS issues at least.

I can certainly use any help you have.  I have someone looking into the ARM
issues with me.  It APPEARS the primary behavior there has to do with IFUNC
behavior to things like memcpy has changed, causing the prelinker to do the
wrong thing(s).  But I have few additional details at this time.

I've reached out to parts of the MIPS community for help, but we'll see if I get
any type of response.  (MIPS, especially 32-bit, I'm very surprised has broken
recently.  I haven't seen many changes in regard to the way binaries are loaded
on MIPS -- so the issue must be related to a more fundamental optimization or
bug in the prelink.)

Thanks for the offer, let me know if you find anything.  I'll certainly respond
as I make progress trying to go through the issues.

--Mark

> p.
> 

Re: Prelink problems -- need help!

[Phil Blundell]

On Mon, 2015-10-26 at 11:45 -0500, Mark Hatle wrote:
> 
> 
> While some of the modern hashing techniques and such do improve run
> -time dynamic
> link performance, there is still a hit that we must take.  For
> devices that need
> quick boot times, quick startup, or are memory constrained, the
> prelinker can
> still help.  (Memory usage on very small systems is a good example. 
>  Memory
> usage can be reduced in larger applications by reducing the number of
> Copy-on-write pages required to handle the relocation information.)

Do you have any concrete data on how much of a boot time speedup,
and/or what reduction in dirty pages, you see on a modern image with
prelink versus the same image without?

p.

Re: Prelink problems -- need help!

[Mark Hatle]

On 10/26/15 11:50 AM, Phil Blundell wrote:
> On Mon, 2015-10-26 at 11:45 -0500, Mark Hatle wrote:
>>
>>
>> While some of the modern hashing techniques and such do improve run
>> -time dynamic
>> link performance, there is still a hit that we must take.  For
>> devices that need
>> quick boot times, quick startup, or are memory constrained, the
>> prelinker can
>> still help.  (Memory usage on very small systems is a good example. 
>>  Memory
>> usage can be reduced in larger applications by reducing the number of
>> Copy-on-write pages required to handle the relocation information.)
> 
> Do you have any concrete data on how much of a boot time speedup,
> and/or what reduction in dirty pages, you see on a modern image with
> prelink versus the same image without?

-last- time I got concrete numbers on a complex boot process.  It was on the
order of 1-5% boot time.. and a fairly large number of pages saved.. (which
reduced fragmentation as well as saved memory.)

We're not talking huge numbers for each, there is a statistics mode in glibc
that can be enabled to show the number of relocations and some of the costs
involved.  Once this is working again, I will be able to run that and give more
concrete numbers.

--Mark

> p.
> 

Re: Prelink problems -- need help!

[Phil Blundell]

On Mon, 2015-10-26 at 11:55 -0500, Mark Hatle wrote:
> -last- time I got concrete numbers on a complex boot process.  It was on the
> order of 1-5% boot time.. and a fairly large number of pages saved.. (which
> reduced fragmentation as well as saved memory.)

I'm slightly surprised at the 5% figure.  Martin Jansa posted some
statistics in another thread earlier today which showed an improvement
from prelink of just under 0.5% (0.24 seconds saved out of 49.51 seconds
total) and that's more like what I would have expected.  What sort of
binaries were you running on the system you measured?

I also don't entirely understand the fragmentation argument.  We're not
talking about a no-mmu configuration here, are we?

p.

Re: Prelink problems -- need help!

[Mark Hatle]

On 10/26/15 4:18 PM, Phil Blundell wrote:
> On Mon, 2015-10-26 at 11:55 -0500, Mark Hatle wrote:
>> -last- time I got concrete numbers on a complex boot process.  It was on the
>> order of 1-5% boot time.. and a fairly large number of pages saved.. (which
>> reduced fragmentation as well as saved memory.)
> 
> I'm slightly surprised at the 5% figure.  Martin Jansa posted some
> statistics in another thread earlier today which showed an improvement
> from prelink of just under 0.5% (0.24 seconds saved out of 49.51 seconds
> total) and that's more like what I would have expected.  What sort of
> binaries were you running on the system you measured?

At the time, it was systemd, X11, a custom control module and a custom UI
program.  The custom UI program was written in C++ and had a heck of a lot of
libraries attached, and lots of RPC via dbus.

> I also don't entirely understand the fragmentation argument.  We're not
> talking about a no-mmu configuration here, are we?

Fragmentation takes time and can slow down memory allocation.  We've got both
the kernel page allocation, as well as the internal glibc memory allocation.
They work together, but are not the same.  (glibc will grab pages of memory from
the kernel and then dish them out.  As this memory fragments, it has to pull
more from the kernel to compensate... as well as free blocks to get more
contiguous memory allocated).. It's a bit of a viscous cycle of some small systems.

--Mark

> p.
> 
> 

Re: Prelink problems -- need help!

[Khem Raj]

[-- Attachment #1: Type: text/plain, Size: 1777 bytes --]


> On Oct 26, 2015, at 9:55 AM, Mark Hatle <mark.hatle@windriver.com> wrote:
> 
> On 10/26/15 11:50 AM, Phil Blundell wrote:
>> On Mon, 2015-10-26 at 11:45 -0500, Mark Hatle wrote:
>>> 
>>> 
>>> While some of the modern hashing techniques and such do improve run
>>> -time dynamic
>>> link performance, there is still a hit that we must take.  For
>>> devices that need
>>> quick boot times, quick startup, or are memory constrained, the
>>> prelinker can
>>> still help.  (Memory usage on very small systems is a good example.
>>> Memory
>>> usage can be reduced in larger applications by reducing the number of
>>> Copy-on-write pages required to handle the relocation information.)
>> 
>> Do you have any concrete data on how much of a boot time speedup,
>> and/or what reduction in dirty pages, you see on a modern image with
>> prelink versus the same image without?
> 
> -last- time I got concrete numbers on a complex boot process.  It was on the
> order of 1-5% boot time.. and a fairly large number of pages saved.. (which
> reduced fragmentation as well as saved memory.)

How is fragmentation getting better ? if dynamic linker was improving the page
locality then it would be possible but that does not seem to be the case here.

> 
> We're not talking huge numbers for each, there is a statistics mode in glibc
> that can be enabled to show the number of relocations and some of the costs
> involved.  Once this is working again, I will be able to run that and give more
> concrete numbers.
> 
> --Mark
> 
>> p.
>> 
> 
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core


[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 211 bytes --]

Re: Prelink problems -- need help!

[Mark Hatle]

On 10/29/15 1:32 PM, Khem Raj wrote:
> 
>> On Oct 26, 2015, at 9:55 AM, Mark Hatle <mark.hatle@windriver.com> wrote:
>>
>> On 10/26/15 11:50 AM, Phil Blundell wrote:
>>> On Mon, 2015-10-26 at 11:45 -0500, Mark Hatle wrote:
>>>>
>>>>
>>>> While some of the modern hashing techniques and such do improve run
>>>> -time dynamic
>>>> link performance, there is still a hit that we must take.  For
>>>> devices that need
>>>> quick boot times, quick startup, or are memory constrained, the
>>>> prelinker can
>>>> still help.  (Memory usage on very small systems is a good example.
>>>> Memory
>>>> usage can be reduced in larger applications by reducing the number of
>>>> Copy-on-write pages required to handle the relocation information.)
>>>
>>> Do you have any concrete data on how much of a boot time speedup,
>>> and/or what reduction in dirty pages, you see on a modern image with
>>> prelink versus the same image without?
>>
>> -last- time I got concrete numbers on a complex boot process.  It was on the
>> order of 1-5% boot time.. and a fairly large number of pages saved.. (which
>> reduced fragmentation as well as saved memory.)
> 
> How is fragmentation getting better ? if dynamic linker was improving the page
> locality then it would be possible but that does not seem to be the case here.

With no relocations, you don't end up doing COW pages.  No COW pages means less
memory used for running many of the small quick Linux scripts at easy boot.

These are issues I've seen on older slower systems.  It's not that big of a deal
with a lot of the newer ones.

However, someone pointed out to me earlier today that with more and more IFUNCs
being used across different platforms, that it's likely the COW page savings are
reduced from what they used to be....  If the IFUNCs are 'local to the same
page', then it should still help reduce the overall count...  but I'm not sure
that is something that is controlled in the normal case.

--Mark

>>
>> We're not talking huge numbers for each, there is a statistics mode in glibc
>> that can be enabled to show the number of relocations and some of the costs
>> involved.  Once this is working again, I will be able to run that and give more
>> concrete numbers.
>>
>> --Mark
>>
>>> p.
>>>
>>
>> --
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.openembedded.org/mailman/listinfo/openembedded-core
> 

Re: Prelink problems -- need help!

[Mark Hatle]

On 10/26/15 11:45 AM, Mark Hatle wrote:
> On 10/26/15 11:10 AM, Phil Blundell wrote:
>> [Cc list trimmed]
>>
>> On Mon, 2015-10-26 at 09:28 -0500, Mark Hatle wrote:
>>> As you can see, pretty much everything is currently broken and I'm
>>> struggling to
>>> find community people with the right skill sets to help me resolve
>>> the issues.
>>
>> I think it's true that prelink is a technology which has largely had
>> its day, and I could well imagine that upstream Red Hat is no longer
>> very interested in it.  https://fedorahosted.org/fesco/ticket/1183 has
>> a summary of some of the reasons why.
>>
>> Before investing any significant amount of time in fixing up prelink
>> (and from what you've described it sounds like it might indeed be a
>> significant effort to get it working again on all platforms) it might
>> be a good idea to make sure we are still convinced that prelink
>> provides useful and meaningful benefits for OE-core users.
>>
> 
> I do think prelink has a purpose.  Especially for embedded systems.
> 
> The biggest reason I see against prelink is ASLR.  I agree 100% that if you want
> ASLR (Address Randomization) you should not be using the prelinker.
> 
> Usually secondary reasons include if you are doing field upgrades at a package
> level -- it means you also need the ability to prelink on the device which cause
> have other performance ramifications and such during this upgrade process.
> 
> I agree with some of the information on the Fedora hosted link you pointed to,
> but disagree with others.
> 
> While some of the modern hashing techniques and such do improve run-time dynamic
> link performance, there is still a hit that we must take.  For devices that need
> quick boot times, quick startup, or are memory constrained, the prelinker can
> still help.  (Memory usage on very small systems is a good example.  Memory
> usage can be reduced in larger applications by reducing the number of
> Copy-on-write pages required to handle the relocation information.)
> 
> I probably would not use the prelinker on a workstation/server any longer.  I
> might not use it on a 'larger' embedded system, but on a more traditional
> embedded system I do think it still has merit.
> 
> (Probably a discussion for the future, due to the ASLR, is should the prelinker
> be enabled by default -- even if it works -- but that is a discussion for later.)
> 
>> If the answer to that is yes then I probably could spare a bit of time
>> to look at the ARM and MIPS issues at least.
> 
> I can certainly use any help you have.  I have someone looking into the ARM
> issues with me.  It APPEARS the primary behavior there has to do with IFUNC
> behavior to things like memcpy has changed, causing the prelinker to do the
> wrong thing(s).  But I have few additional details at this time.
> 
> I've reached out to parts of the MIPS community for help, but we'll see if I get
> any type of response.  (MIPS, especially 32-bit, I'm very surprised has broken
> recently.  I haven't seen many changes in regard to the way binaries are loaded
> on MIPS -- so the issue must be related to a more fundamental optimization or
> bug in the prelink.)
> 
> Thanks for the offer, let me know if you find anything.  I'll certainly respond
> as I make progress trying to go through the issues.

I just wanted to update folks on where I'm at.

On the Intel IA32 architecture, the problem appears to have been triggered by
the ELF_RTYPE_CLASS_EXTERN_PROTECTED_DATA that went in after March of this year.
 I've not been able to track down what is causing the bad behavior -- but fully
dropping support for that class type allows the prelink to "mostly work".

On ARM the issues appear to have some problem related to IFUNCs, but I still
don't have any data beyond that at this point.

> --Mark
> 
>> p.
>>
> 

Re: Prelink status

[Mark Hatle]

I just wanted to update everyone on where we are at.  (I won't bore anyone with
any followups past this.)

IA32 (32-bit and 64-bit):

We believe the problems are resolved.  The error was a combination of prior
patch merge issues and a bug in glibc 2.22.  A patch for the glibc 2.22 has
already been sent to the GLIBC community and I have a change ready for OE.


ARM:

The same glibc 2.22 bug affected ARM as well as IA32.

Also it appears that a patch that went in last fall to enable IFUNC on arm
somehow had a hunk dropped from it.  Now that we've identified and restored this
hunk, it appears to be working again.


MIPS32/MIPS64 (o32, n32, n64):

A new section was added, SHT_MIPS_ABIFLAGS, this was confusing the prelinker
causing it to refused to continue.  A patch has gone in to instruct the
prelinker how to handle this new section (ignore it).


Power (32-bit):

We are still investigating changes that have happened in the past year+ to
binutils and glibc for Power systems.

Power (64-bit):

It is likely this will not work anytime soon.  The ABI has changed and is no
longer compatible with the previous methods of prelinking.  This is fairly low
priority to be fixed, as it's been broken for a while now.


If anyone has any expertise on Power 32-bit or 64-bit ELF, I'm still looking for
help there to get these issues resolved.

--Mark

On 10/26/15 9:28 AM, Mark Hatle wrote:
> I'm cross posting this in an effort to ask for help for the cross-prelink
> project.  (If you are unaware of what the prelinker does, it goes through and
> adjusts the relocation addresses to predefined values in an attempt to speed up
> executable load times.  For embedded systems, this can be fairly critical in
> meeting boot and runtime load performance targets.)
> 
> As some people might be aware, the upstream 'prelink' project that was
> maintained by Jakub Jelinek at Red Hat appears to have been abandoned.
> 
> I've been the maintainer for the Yocto Project's 'prelink-cross' project for a
> few years, but my work has been dealing with compilation issues and other things
> that have to do with using the prelinker in a cross compilation environment.
> 
> This means that unfortunately I'm in a situation where I am having problems
> trying to fix the problems in the prelinker itself, but I am not an ELF expert.
>  Currently the cross prelinker appears to be broken for pretty much every
> architecture and I need help trying to resolve these problems.
> 
> Issues I know of right now:
> 
> Introduction of RTYPE_CLASS_EXTERN_PROTECTED_DATA appears to be causing failures
> on ARM32, x86 and x86_64.
> 
> Some possibly unrelated issue is causing various items to relocate incorrectly
> (IFUNCs perhaps) on ARMv7 builds.
> 
> Power is no longer working properly, most likely due to an optimization in the
> way the PPC GOT table(s) are setup.
> 
> Power64 hasn't been working properly for a while, possibly due to the switch to
> ELFv2.
> 
> MIPS and MIPS64 also do not appear to be working, and I've not found a possible
> cause.
> 
> As you can see, pretty much everything is currently broken and I'm struggling to
> find community people with the right skill sets to help me resolve the issues.
> 
> The current state of my work in progress is available at:
> 
> git://git.yoctoproject.org/prelink-cross
> 
> branch cross_prelink_staging
> 
> If you think you can help, please let me know.
> --Mark
>