ARM: relocation out of range (when loading a module)

ARM: relocation out of range (when loading a module)

[Rabin Vincent]

On Tue, Jan 11, 2011 at 12:04, Alexander Holler <holler@ahsoftware.de> wrote:
> Point is that I would like to understand why including an initramfs has such
> consequences.
>
> I have to search for some pictures which are explaining the memory layout
> (never had any interested in that before).
>
> I assume some memory near the kernel is reserved for the modules and an
> initramfs will come inbetween the kernel and the reserved memory for
> modules.

Modules get placed 16MiB below PAGE_OFFSET, and when you add another
~16MiB initramfs between them and the kernel symbols, you exceed the
offset range of the branch instruction.

It's possible to hack around this by placing the initramfs at the end of
the kernel image rather than at the beginning with the rest of the init
data.  Something like the below should work, although you should also
probably take care of alignment and also have this section freed when
the rest of the init data is freed.

diff --git a/arch/arm/kernel/vmlinux.lds.S b/arch/arm/kernel/vmlinux.lds.S
index b16c079..05ad361 100644
--- a/arch/arm/kernel/vmlinux.lds.S
+++ b/arch/arm/kernel/vmlinux.lds.S
@@ -46,7 +46,6 @@ SECTIONS
 		INIT_CALLS
 		CON_INITCALL
 		SECURITY_INITCALL
-		INIT_RAM_FS

 #ifndef CONFIG_XIP_KERNEL
 		__init_begin = _stext;
@@ -170,6 +169,11 @@ SECTIONS

 		_edata = .;
 	}
+
+	.initramfs : {
+		INIT_RAM_FS
+	}
+
 	_edata_loc = __data_loc + SIZEOF(.data);

 #ifdef CONFIG_HAVE_TCM

ARM: relocation out of range (when loading a module)

[Russell King - ARM Linux]

On Tue, Jan 11, 2011 at 09:16:38PM +0530, Rabin Vincent wrote:
> It's possible to hack around this by placing the initramfs at the end of
> the kernel image rather than at the beginning with the rest of the init
> data.  Something like the below should work, although you should also
> probably take care of alignment and also have this section freed when
> the rest of the init data is freed.

You're then running into problems as _sdata.._edata is copied to RAM on
XIP kernels, and you really don't want to waste time copying the
initramfs to RAM.

ARM: relocation out of range (when loading a module)

[Alexander Holler]

Hello,

Am 11.01.2011 16:59, schrieb Russell King - ARM Linux:
> On Tue, Jan 11, 2011 at 09:16:38PM +0530, Rabin Vincent wrote:
>> It's possible to hack around this by placing the initramfs at the end of
>> the kernel image rather than at the beginning with the rest of the init
>> data.  Something like the below should work, although you should also
>> probably take care of alignment and also have this section freed when
>> the rest of the init data is freed.
>
> You're then running into problems as _sdata.._edata is copied to RAM on
> XIP kernels, and you really don't want to waste time copying the
> initramfs to RAM.


Thanks to all for the provided informations. I'm now seeing some light. ;)

However, looking at the vmlinux.lds.S I think I should better not try to 
fix that, I would likely do something wrong because of missing knowledge 
about all those sections.

I can offer a patch wich adds a TODO to vmlinux.lds.S, but if someone 
else feels the need to fix that, feel free to do so. ;)

Regards,

Alexander


-------------------------------------------------------------------------
 From 2ce2934b6a55ac34734781d1a49569d79d0fdcda Mon Sep 17 00:00:00 2001
From: Alexander Holler <holler@ahsoftware.de>
Date: Wed, 12 Jan 2011 02:49:52 +0100
Subject: [PATCH] ARM: Add TODO to move INIT_RAM_FS to another point.

Signed-off-by: Alexander Holler <holler@ahsoftware.de>
---
  arch/arm/kernel/vmlinux.lds.S |    6 ++++++
  1 files changed, 6 insertions(+), 0 deletions(-)

diff --git a/arch/arm/kernel/vmlinux.lds.S b/arch/arm/kernel/vmlinux.lds.S
index cead889..5da3479 100644
--- a/arch/arm/kernel/vmlinux.lds.S
+++ b/arch/arm/kernel/vmlinux.lds.S
@@ -62,6 +62,12 @@ SECTIONS
                 INIT_CALLS
                 CON_INITCALL
                 SECURITY_INITCALL
+               /*
+                * TODO: The size of INIT_RAM_FS could easily reach a
+                * point (~16MB) when loading modules will fail because
+                * relocations will be out of range. So this place here
+                * isn't the best one.
+                */
                 INIT_RAM_FS

  #ifndef CONFIG_XIP_KERNEL
-- 
1.7.3.4
-------------------------------------------------------------------------

ARM: relocation out of range (when loading a module)

[Dave Martin]

On Tue, Jan 11, 2011 at 9:00 PM, Alexander Holler <holler@ahsoftware.de> wrote:
> Hello,

[...]

> ? ? ? ? ? ? ? ?INIT_CALLS
> ? ? ? ? ? ? ? ?CON_INITCALL
> ? ? ? ? ? ? ? ?SECURITY_INITCALL
> + ? ? ? ? ? ? ? /*
> + ? ? ? ? ? ? ? ?* TODO: The size of INIT_RAM_FS could easily reach a
> + ? ? ? ? ? ? ? ?* point (~16MB) when loading modules will fail because
> + ? ? ? ? ? ? ? ?* relocations will be out of range. So this place here
> + ? ? ? ? ? ? ? ?* isn't the best one.
> + ? ? ? ? ? ? ? ?*/
> ? ? ? ? ? ? ? ?INIT_RAM_FS

In general, do we expect always to be able to avoid the situation
where branches in the kernel may need to cover too large a range ...
and is there any strategy for working aroung it?

If we have problems branching from the modules area into vmlinux, we
could possibly build modules with -fPIC : this would remove the
restriction on branch range, though there would also be some
performance impact for the modules...

Cheers
---Dave

ARM: relocation out of range (when loading a module)

[Russell King - ARM Linux]

On Wed, Jan 12, 2011 at 10:05:19AM -0600, Dave Martin wrote:
> In general, do we expect always to be able to avoid the situation
> where branches in the kernel may need to cover too large a range ...
> and is there any strategy for working aroung it?

It's not that big a problem - 99.999999% of setups never run into the
problem.  It's only those who use large initramfs's built into their
kernel image at present, and that is ultimately solvable.

I don't think we need to litter code with TODO comments.

> If we have problems branching from the modules area into vmlinux, we
> could possibly build modules with -fPIC : this would remove the
> restriction on branch range, though there would also be some
> performance impact for the modules...

That also brings in issues with GOT tables and the like, and also requires
different build options for modules and the main kernel.  I don't think
kbuild is setup to do that - and I'd argue that it's unnecessary if we
fix the layout of the kernel image.

As I said a few emails ago, shuffling sections around in the image is
not as trivial as it looks on the face of it as we make assumptions
about what is in _stext.._etext, _sdata.._edata, whether there's anything
between _etext.._sdata, and other symbolic ranges.

At the moment, _stext.._etext + _sdata.._edata covers the entire kernel
image - with the init sections at the start of _stext.  Putting the init
sections between _etext and _sdata makes a hole in the middle of that,
which may be suboptimal for page allocation.  It also means that the
range no longer covers all kernel stuff.  Also whether _stext = _text,
_sdata = _data, etc.

Another technicality is that some of these ranges are used for stuff like
DMA API debugging:

        if (overlap(addr, len, _text, _etext) ||
            overlap(addr, len, __start_rodata, __end_rodata))
                err_printk(dev, NULL, "DMA-API: device driver maps memory from kernel text or rodata [addr=%p] [len=%lu]\n", addr, len);

The last pieces of the puzzle is whether we have anything that implicitly
relies on the init section being low down and freeing its pages (eg, as
a way of stopping that memory being used for non-DMA stuff.)

I do have a patch which shuffles some of this stuff around, but I'm not
entirely happy with it yet.

ARM: relocation out of range (when loading a module)

[Nicolas Pitre]

On Wed, 12 Jan 2011, Dave Martin wrote:

> In general, do we expect always to be able to avoid the situation
> where branches in the kernel may need to cover too large a range ...
> and is there any strategy for working aroung it?

Normally we try to keep the code close together.  And at least for the 
.text section, the linker is able to insert indirect branch veneers when 
the range is too large.  But in the case of modules, it's the in-kernel 
code that perform the final symbol relocation.

> If we have problems branching from the modules area into vmlinux, we
> could possibly build modules with -fPIC : this would remove the
> restriction on branch range, though there would also be some
> performance impact for the modules...

If we really needed to do such thing, that would be even better to 
simply have the kernel create those indirect veneers dynamically.  And 
in fact, Russell had that working and he posted the corresponding patch 
many years ago, but the module placement was made so that the indirect 
branches were unnecessary.

Even now, it's probably a better idea to rework the section layout and 
preserve the ability to perform direct branches into the kernel from 
modules as much as possible.


Nicolas

ARM: relocation out of range (when loading a module)

[Russell King - ARM Linux]

On Wed, Jan 12, 2011 at 01:28:23PM -0500, Nicolas Pitre wrote:
> If we really needed to do such thing, that would be even better to 
> simply have the kernel create those indirect veneers dynamically.  And 
> in fact, Russell had that working and he posted the corresponding patch 
> many years ago, but the module placement was made so that the indirect 
> branches were unnecessary.

Actually, it's something we used to do in 2.2 days when modules were
prepared and linked in userspace before being uploaded into kernel
space.  This allowed the module to be inteligently linked - so the
indirect branches were created only when they were necessary.

When the new kernel-based module linker happened, this presented a
chicken and egg problem with that approach, which give us a choice:
either place the module within range of the kernel text and guarantee
that the kernel text is reachable, or _always_ indirect every module
branch through a jump table even if it was reachable from where the
module was placed.

The decision was made to go with the former, so the latter never got
implemented.

Then came along the embedded initrd/initramfs idea which rather buggered
the scheme when large initramfs are embedded into the image.

As the overall feeling at the time was "don't use large initrds" it's
something I've never really cared about - and I'm still of the opinion
that 16MB of compressed initrd/initramfs is rather silly.

ARM: relocation out of range (when loading a module)

[Nicolas Pitre]

On Wed, 12 Jan 2011, Russell King - ARM Linux wrote:

> Then came along the embedded initrd/initramfs idea which rather buggered
> the scheme when large initramfs are embedded into the image.
> 
> As the overall feeling at the time was "don't use large initrds" it's
> something I've never really cared about - and I'm still of the opinion
> that 16MB of compressed initrd/initramfs is rather silly.

It is... but we have more than 32MB of RAM total now, and people are 
running standard distributions on ARM these days, such as Fedora or 
Ubuntu, including their corresponding initrd that may contain lots of 
modules, splashscreen data, etc.  So it might be a good idea to think 
about fixing this limitation before it comes back again.


Nicolas

ARM: relocation out of range (when loading a module)

[Sachin Verma]

Hello,

On Thu, Jan 13, 2011 at 12:25 AM, Nicolas Pitre
<nicolas.pitre@linaro.org> wrote:
> On Wed, 12 Jan 2011, Russell King - ARM Linux wrote:
>
>> Then came along the embedded initrd/initramfs idea which rather buggered
>> the scheme when large initramfs are embedded into the image.
>>
>> As the overall feeling at the time was "don't use large initrds" it's
>> something I've never really cared about - and I'm still of the opinion
>> that 16MB of compressed initrd/initramfs is rather silly.
>
> It is... but we have more than 32MB of RAM total now, and people are
> running standard distributions on ARM these days, such as Fedora or
> Ubuntu, including their corresponding initrd that may contain lots of
> modules, splashscreen data, etc. ?So it might be a good idea to think
> about fixing this limitation before it comes back again.
>

in fact it is not just the initrd/initramfs size, i am facing issues
with loading modules on MMU-less envt(Cortex M3).
I have 128MB of Physical RAM and i am using a 4MB initrd to boot my
system. when i try to load kernel modules i get relocation errors.
As MMU-less systems are allowing modules to be loaded anywhere from
RAM and not from any predefined limits of MODULES_VADDR and
MODULES_END.

Do you have any pointers on how to tackle this situation for MMU less
environments?

Regards,
Sachin.

ARM: relocation out of range (when loading a module)

[Catalin Marinas]

On 25 January 2011 08:44, Sachin Verma <imschnvrm@gmail.com> wrote:
> On Thu, Jan 13, 2011 at 12:25 AM, Nicolas Pitre
> <nicolas.pitre@linaro.org> wrote:
>> On Wed, 12 Jan 2011, Russell King - ARM Linux wrote:
>>
>>> Then came along the embedded initrd/initramfs idea which rather buggered
>>> the scheme when large initramfs are embedded into the image.
>>>
>>> As the overall feeling at the time was "don't use large initrds" it's
>>> something I've never really cared about - and I'm still of the opinion
>>> that 16MB of compressed initrd/initramfs is rather silly.
>>
>> It is... but we have more than 32MB of RAM total now, and people are
>> running standard distributions on ARM these days, such as Fedora or
>> Ubuntu, including their corresponding initrd that may contain lots of
>> modules, splashscreen data, etc. ?So it might be a good idea to think
>> about fixing this limitation before it comes back again.
>>
>
> in fact it is not just the initrd/initramfs size, i am facing issues
> with loading modules on MMU-less envt(Cortex M3).
> I have 128MB of Physical RAM and i am using a 4MB initrd to boot my
> system. when i try to load kernel modules i get relocation errors.
> As MMU-less systems are allowing modules to be loaded anywhere from
> RAM and not from any predefined limits of MODULES_VADDR and
> MODULES_END.
>
> Do you have any pointers on how to tackle this situation for MMU less
> environments?

I don't think there is a simple solution. As a hack, you could define
DMA and Normal Zones in your system and force module allocations from
the DMA zone (which would be close to the kernel).

I'm not sure whether the kernel could be compile with a larger memory
model, but then you have to improve the kernel to handle new
relocation types.

-- 
Catalin

ARM: relocation out of range (when loading a module)

[Nicolas Pitre]

On Tue, 25 Jan 2011, Sachin Verma wrote:

> Hello,
> 
> On Thu, Jan 13, 2011 at 12:25 AM, Nicolas Pitre
> <nicolas.pitre@linaro.org> wrote:
> > On Wed, 12 Jan 2011, Russell King - ARM Linux wrote:
> >
> >> As the overall feeling at the time was "don't use large initrds" it's
> >> something I've never really cared about - and I'm still of the opinion
> >> that 16MB of compressed initrd/initramfs is rather silly.
> >
> > It is... but we have more than 32MB of RAM total now, and people are
> > running standard distributions on ARM these days, such as Fedora or
> > Ubuntu, including their corresponding initrd that may contain lots of
> > modules, splashscreen data, etc. ?So it might be a good idea to think
> > about fixing this limitation before it comes back again.
> >
> 
> in fact it is not just the initrd/initramfs size, i am facing issues
> with loading modules on MMU-less envt(Cortex M3).
> I have 128MB of Physical RAM and i am using a 4MB initrd to boot my
> system. when i try to load kernel modules i get relocation errors.
> As MMU-less systems are allowing modules to be loaded anywhere from
> RAM and not from any predefined limits of MODULES_VADDR and
> MODULES_END.
> 
> Do you have any pointers on how to tackle this situation for MMU less
> environments?

The MMU-less kernel should still favor allocations close to the kernel 
text for modules, and anything else away from the kernel going 
downwards.

Otherwise a veneer should be created by the module symbol resolver such 
that if the branch distance to reach, say, printk is too large, then the 
following code would have to be dynamically generated right next to the 
module:

	ldr	pc, [pc, #-4]
	.word	<far_away_symbol>

Then, in your module, you patch the branch relocation for printk so that 
it branches to the code above instead, and then store the address of 
printk at the location represented by the .word directive.


Nicolas

ARM: relocation out of range (when loading a module)

[Russell King - ARM Linux]

On Thu, Jan 27, 2011 at 12:43:54AM -0500, Nicolas Pitre wrote:
> The MMU-less kernel should still favor allocations close to the kernel 
> text for modules, and anything else away from the kernel going 
> downwards.
> 
> Otherwise a veneer should be created by the module symbol resolver such 
> that if the branch distance to reach, say, printk is too large, then the 
> following code would have to be dynamically generated right next to the 
> module:
> 
> 	ldr	pc, [pc, #-4]
> 	.word	<far_away_symbol>
> 
> Then, in your module, you patch the branch relocation for printk so that 
> it branches to the code above instead, and then store the address of 
> printk at the location represented by the .word directive.

What you're suggesting is what we used to do with the old user-space
module tools, which would've been nice to carry forwards to the new
module code.  I never found a way to do it.

The problems:
1. Where do you create those veneers?
2. How many veneers do you allocate space for?
3. How do you determine that you need a veneer?

While you can say "next to the module" for (1), you can only do that at
the point in time when the space for the module is allocated, and you
need to know at that point how much space you require.

For (2), you could always allocate space for one veneer per symbol present
in the module, but that's very wasteful.

(3) is almost impossible to know ahead of time as you don't have the
relocations, realistically you have to allocate one veneer per symbol,
and as you don't know whether it's a data or code symbol, you'll have
to allocate one veneer for every symbol in a module.

I really don't like it, and I don't see that this is sanely solvable
without giving architectures much more control over module loading,
which I don't think will ever happen.  It's probably simpler to build
modules with whatever that magic option is to tell GCC to always generate
'far call' veneers for everything rather than trying to 'fix' the kernel
module loader.

ARM: relocation out of range (when loading a module)

[Nicolas Pitre]

On Thu, 10 Feb 2011, Russell King - ARM Linux wrote:

> On Thu, Jan 27, 2011 at 12:43:54AM -0500, Nicolas Pitre wrote:
> > The MMU-less kernel should still favor allocations close to the kernel 
> > text for modules, and anything else away from the kernel going 
> > downwards.
> > 
> > Otherwise a veneer should be created by the module symbol resolver such 
> > that if the branch distance to reach, say, printk is too large, then the 
> > following code would have to be dynamically generated right next to the 
> > module:
> > 
> > 	ldr	pc, [pc, #-4]
> > 	.word	<far_away_symbol>
> > 
> > Then, in your module, you patch the branch relocation for printk so that 
> > it branches to the code above instead, and then store the address of 
> > printk at the location represented by the .word directive.
> 
> What you're suggesting is what we used to do with the old user-space
> module tools, which would've been nice to carry forwards to the new
> module code.  I never found a way to do it.
> 
> The problems:
> 1. Where do you create those veneers?
> 2. How many veneers do you allocate space for?
> 3. How do you determine that you need a veneer?
> 
> While you can say "next to the module" for (1), you can only do that at
> the point in time when the space for the module is allocated, and you
> need to know at that point how much space you require.

You would have to guess of course.  Having a guess of 1/2 the module 
size should be pretty safe.  So allocating 3/2 the space in 
module_alloc(), and then suffice to free the unused portion in 
module_finalize().

> For (2), you could always allocate space for one veneer per symbol present
> in the module, but that's very wasteful.
> 
> (3) is almost impossible to know ahead of time as you don't have the
> relocations, realistically you have to allocate one veneer per symbol,
> and as you don't know whether it's a data or code symbol, you'll have
> to allocate one veneer for every symbol in a module.

I don't think you may know the number of symbols in advance either 
anyway.

> I really don't like it, and I don't see that this is sanely solvable
> without giving architectures much more control over module loading,
> which I don't think will ever happen.  It's probably simpler to build
> modules with whatever that magic option is to tell GCC to always generate
> 'far call' veneers for everything rather than trying to 'fix' the kernel
> module loader.

Well, this is certainly possible indeed to just use -mlong-calls to 
build modules which would solve the issue right away, and with a smaller 
cost than if each function calls were going through a veneer, but with a 
higher cost than a relocatable direct branch.  So this could be decided 
with a Kconfig option.


Nicolas

ARM: relocation out of range (when loading a module)

[Dave Martin]

On Thu, Feb 10, 2011 at 7:41 PM, Nicolas Pitre <nicolas.pitre@linaro.org> wrote:
> On Thu, 10 Feb 2011, Russell King - ARM Linux wrote:
>
>> On Thu, Jan 27, 2011 at 12:43:54AM -0500, Nicolas Pitre wrote:
>> > The MMU-less kernel should still favor allocations close to the kernel
>> > text for modules, and anything else away from the kernel going
>> > downwards.
>> >
>> > Otherwise a veneer should be created by the module symbol resolver such
>> > that if the branch distance to reach, say, printk is too large, then the
>> > following code would have to be dynamically generated right next to the
>> > module:
>> >
>> > ? ? ldr ? ? pc, [pc, #-4]
>> > ? ? .word ? <far_away_symbol>
>> >
>> > Then, in your module, you patch the branch relocation for printk so that
>> > it branches to the code above instead, and then store the address of
>> > printk at the location represented by the .word directive.
>>
>> What you're suggesting is what we used to do with the old user-space
>> module tools, which would've been nice to carry forwards to the new
>> module code. ?I never found a way to do it.
>>
>> The problems:
>> 1. Where do you create those veneers?
>> 2. How many veneers do you allocate space for?
>> 3. How do you determine that you need a veneer?
>>
>> While you can say "next to the module" for (1), you can only do that at
>> the point in time when the space for the module is allocated, and you
>> need to know at that point how much space you require.
>
> You would have to guess of course. ?Having a guess of 1/2 the module
> size should be pretty safe. ?So allocating 3/2 the space in
> module_alloc(), and then suffice to free the unused portion in
> module_finalize().
>
>> For (2), you could always allocate space for one veneer per symbol present
>> in the module, but that's very wasteful.
>>
>> (3) is almost impossible to know ahead of time as you don't have the
>> relocations, realistically you have to allocate one veneer per symbol,
>> and as you don't know whether it's a data or code symbol, you'll have
>> to allocate one veneer for every symbol in a module.
>
> I don't think you may know the number of symbols in advance either
> anyway.

You could probably cook up a good upper bound based on the size of the
kernel and the number of symbols in the module: i.e., assume that
every undefined symbol in the module needs to be fixed up to point at
the most distant symbol in the kernel.

For people with normal-sized kernels, this bound will probably work
out as zero most of the time (i.e., the current situation).  For
people with big kernels, or when many modules are already loaded, it
may work out at 100% -- but that's the price to pay for guaranteed
preallocation of the space required for the veneers.  And anyway, you
may really need a substantial chunk of those veneers in such cases.

---Dave

ARM: relocation out of range (when loading a module)

[Russell King - ARM Linux]

On Fri, Feb 11, 2011 at 09:31:04AM +0000, Dave Martin wrote:
> You could probably cook up a good upper bound based on the size of the
> kernel and the number of symbols in the module: i.e., assume that
> every undefined symbol in the module needs to be fixed up to point at
> the most distant symbol in the kernel.
> 
> For people with normal-sized kernels, this bound will probably work
> out as zero most of the time (i.e., the current situation).  For
> people with big kernels, or when many modules are already loaded, it
> may work out at 100% -- but that's the price to pay for guaranteed
> preallocation of the space required for the veneers.  And anyway, you
> may really need a substantial chunk of those veneers in such cases.

I think it's going to be easier just to re-order the kernel image link
order to solve that problem.  That just leaves uclinux...

ARM: relocation out of range (when loading a module)

[Dave Martin]

On Fri, Feb 11, 2011 at 9:38 AM, Russell King - ARM Linux
<linux@arm.linux.org.uk> wrote:
> On Fri, Feb 11, 2011 at 09:31:04AM +0000, Dave Martin wrote:
>> You could probably cook up a good upper bound based on the size of the
>> kernel and the number of symbols in the module: i.e., assume that
>> every undefined symbol in the module needs to be fixed up to point at
>> the most distant symbol in the kernel.
>>
>> For people with normal-sized kernels, this bound will probably work
>> out as zero most of the time (i.e., the current situation). ?For
>> people with big kernels, or when many modules are already loaded, it
>> may work out at 100% -- but that's the price to pay for guaranteed
>> preallocation of the space required for the veneers. ?And anyway, you
>> may really need a substantial chunk of those veneers in such cases.
>
> I think it's going to be easier just to re-order the kernel image link
> order to solve that problem. ?That just leaves uclinux...
>

But if the kernel is big, or there are many modules, changing the
order can't solve the problem surely?  Or is the problem purely caused
by having the initramfs in the way, and we think the amount of actual
code will never be big enough to cause a problem?

Cheers
---Dave

ARM: relocation out of range (when loading a module)

[Nicolas Pitre]

On Fri, 11 Feb 2011, Dave Martin wrote:

> On Fri, Feb 11, 2011 at 9:38 AM, Russell King - ARM Linux
> <linux@arm.linux.org.uk> wrote:
> > On Fri, Feb 11, 2011 at 09:31:04AM +0000, Dave Martin wrote:
> >> You could probably cook up a good upper bound based on the size of the
> >> kernel and the number of symbols in the module: i.e., assume that
> >> every undefined symbol in the module needs to be fixed up to point at
> >> the most distant symbol in the kernel.
> >>
> >> For people with normal-sized kernels, this bound will probably work
> >> out as zero most of the time (i.e., the current situation). ?For
> >> people with big kernels, or when many modules are already loaded, it
> >> may work out at 100% -- but that's the price to pay for guaranteed
> >> preallocation of the space required for the veneers. ?And anyway, you
> >> may really need a substantial chunk of those veneers in such cases.
> >
> > I think it's going to be easier just to re-order the kernel image link
> > order to solve that problem. ?That just leaves uclinux...
> >
> 
> But if the kernel is big, or there are many modules, changing the
> order can't solve the problem surely?  Or is the problem purely caused
> by having the initramfs in the way, and we think the amount of actual
> code will never be big enough to cause a problem?

Right now the biggest problem comes from the size of the initramfs.  The 
solution for that is simply to move it elsewhere towards the end of the 
kernel image instead of at the beginning.  And I think we're good for a 
couple years before the actual kernel code size becomes an issue.


Nicolas

ARM: relocation out of range (when loading a module)

[Nicolas Pitre]

On Fri, 11 Feb 2011, Dave Martin wrote:

> On Thu, Feb 10, 2011 at 7:41 PM, Nicolas Pitre <nicolas.pitre@linaro.org> wrote:
> > On Thu, 10 Feb 2011, Russell King - ARM Linux wrote:
> >
> >> On Thu, Jan 27, 2011 at 12:43:54AM -0500, Nicolas Pitre wrote:
> >> > The MMU-less kernel should still favor allocations close to the kernel
> >> > text for modules, and anything else away from the kernel going
> >> > downwards.
> >> >
> >> > Otherwise a veneer should be created by the module symbol resolver such
> >> > that if the branch distance to reach, say, printk is too large, then the
> >> > following code would have to be dynamically generated right next to the
> >> > module:
> >> >
> >> > ? ? ldr ? ? pc, [pc, #-4]
> >> > ? ? .word ? <far_away_symbol>
> >> >
> >> > Then, in your module, you patch the branch relocation for printk so that
> >> > it branches to the code above instead, and then store the address of
> >> > printk at the location represented by the .word directive.
> >>
> >> What you're suggesting is what we used to do with the old user-space
> >> module tools, which would've been nice to carry forwards to the new
> >> module code. ?I never found a way to do it.
> >>
> >> The problems:
> >> 1. Where do you create those veneers?
> >> 2. How many veneers do you allocate space for?
> >> 3. How do you determine that you need a veneer?
> >>
> >> While you can say "next to the module" for (1), you can only do that at
> >> the point in time when the space for the module is allocated, and you
> >> need to know at that point how much space you require.
> >
> > You would have to guess of course. ?Having a guess of 1/2 the module
> > size should be pretty safe. ?So allocating 3/2 the space in
> > module_alloc(), and then suffice to free the unused portion in
> > module_finalize().
> >
> >> For (2), you could always allocate space for one veneer per symbol present
> >> in the module, but that's very wasteful.
> >>
> >> (3) is almost impossible to know ahead of time as you don't have the
> >> relocations, realistically you have to allocate one veneer per symbol,
> >> and as you don't know whether it's a data or code symbol, you'll have
> >> to allocate one veneer for every symbol in a module.
> >
> > I don't think you may know the number of symbols in advance either
> > anyway.
> 
> You could probably cook up a good upper bound based on the size of the
> kernel and the number of symbols in the module: i.e., assume that
> every undefined symbol in the module needs to be fixed up to point at
> the most distant symbol in the kernel.

Sure... It is just that the memory allocation is currently done before 
the number of symbols in the module is known.  Changing that would 
require non trivial changes in the generic module loading code which 
potentially would affect all architectures, and therefore I don't think 
we want to go there.

The other solution would be to determine the number of objects in need 
of a veneer in apply_relocate(), allocate a replacement area for the 
module, copy everything over, and then create the veneers close to the 
module.  But 1) the second allocation may fail, and 2) this will change 
the distance from the kernel potentially requiring more veneers than 
initially determined, and 3) the generic module code might still have 
pointer references into the old allocation area (didn't check but that 
can be expected).  That's just too messy.

> For people with normal-sized kernels, this bound will probably work
> out as zero most of the time (i.e., the current situation).  For
> people with big kernels, or when many modules are already loaded, it
> may work out at 100% -- but that's the price to pay for guaranteed
> preallocation of the space required for the veneers.  And anyway, you
> may really need a substantial chunk of those veneers in such cases.

I still think that compiling modules with -mlong-calls, and making that 
conditional on a kernel config option so only those who need it will 
have it, is the simplest solution.  Nothing in the kernel code would 
need to be changed.


Nicolas

ARM: relocation out of range (when loading a module)

[Dave Martin]

On Fri, Feb 11, 2011 at 1:51 PM, Nicolas Pitre <nicolas.pitre@linaro.org> wrote:
> On Fri, 11 Feb 2011, Dave Martin wrote:
>
>> On Thu, Feb 10, 2011 at 7:41 PM, Nicolas Pitre <nicolas.pitre@linaro.org> wrote:
>> > On Thu, 10 Feb 2011, Russell King - ARM Linux wrote:
>> >
>> >> On Thu, Jan 27, 2011 at 12:43:54AM -0500, Nicolas Pitre wrote:
>> >> > The MMU-less kernel should still favor allocations close to the kernel
>> >> > text for modules, and anything else away from the kernel going
>> >> > downwards.
>> >> >
>> >> > Otherwise a veneer should be created by the module symbol resolver such
>> >> > that if the branch distance to reach, say, printk is too large, then the
>> >> > following code would have to be dynamically generated right next to the
>> >> > module:
>> >> >
>> >> > ? ? ldr ? ? pc, [pc, #-4]
>> >> > ? ? .word ? <far_away_symbol>
>> >> >
>> >> > Then, in your module, you patch the branch relocation for printk so that
>> >> > it branches to the code above instead, and then store the address of
>> >> > printk at the location represented by the .word directive.
>> >>
>> >> What you're suggesting is what we used to do with the old user-space
>> >> module tools, which would've been nice to carry forwards to the new
>> >> module code. ?I never found a way to do it.
>> >>
>> >> The problems:
>> >> 1. Where do you create those veneers?
>> >> 2. How many veneers do you allocate space for?
>> >> 3. How do you determine that you need a veneer?
>> >>
>> >> While you can say "next to the module" for (1), you can only do that at
>> >> the point in time when the space for the module is allocated, and you
>> >> need to know at that point how much space you require.
>> >
>> > You would have to guess of course. ?Having a guess of 1/2 the module
>> > size should be pretty safe. ?So allocating 3/2 the space in
>> > module_alloc(), and then suffice to free the unused portion in
>> > module_finalize().
>> >
>> >> For (2), you could always allocate space for one veneer per symbol present
>> >> in the module, but that's very wasteful.
>> >>
>> >> (3) is almost impossible to know ahead of time as you don't have the
>> >> relocations, realistically you have to allocate one veneer per symbol,
>> >> and as you don't know whether it's a data or code symbol, you'll have
>> >> to allocate one veneer for every symbol in a module.
>> >
>> > I don't think you may know the number of symbols in advance either
>> > anyway.
>>
>> You could probably cook up a good upper bound based on the size of the
>> kernel and the number of symbols in the module: i.e., assume that
>> every undefined symbol in the module needs to be fixed up to point at
>> the most distant symbol in the kernel.
>
> Sure... It is just that the memory allocation is currently done before
> the number of symbols in the module is known. ?Changing that would
> require non trivial changes in the generic module loading code which
> potentially would affect all architectures, and therefore I don't think
> we want to go there.
>
> The other solution would be to determine the number of objects in need
> of a veneer in apply_relocate(), allocate a replacement area for the
> module, copy everything over, and then create the veneers close to the
> module. ?But 1) the second allocation may fail, and 2) this will change
> the distance from the kernel potentially requiring more veneers than
> initially determined, and 3) the generic module code might still have
> pointer references into the old allocation area (didn't check but that
> can be expected). ?That's just too messy.
>
>> For people with normal-sized kernels, this bound will probably work
>> out as zero most of the time (i.e., the current situation). ?For
>> people with big kernels, or when many modules are already loaded, it
>> may work out at 100% -- but that's the price to pay for guaranteed
>> preallocation of the space required for the veneers. ?And anyway, you
>> may really need a substantial chunk of those veneers in such cases.
>
> I still think that compiling modules with -mlong-calls, and making that
> conditional on a kernel config option so only those who need it will
> have it, is the simplest solution. ?Nothing in the kernel code would
> need to be changed.

Yes, this sounds like a reasonable solution for now ... since most
people don't need this turned on anyway, and those that do need it
know who they are.

 if we can also move the initramfs somewhere less obstructive in the
future, then so much the better.

Cheers
---Dave

ARM: relocation out of range (when loading a module)

[Russell King - ARM Linux]

On Fri, Feb 11, 2011 at 02:25:28PM +0000, Dave Martin wrote:
>  if we can also move the initramfs somewhere less obstructive in the
> future, then so much the better.

This may do it, and it might even work.  Not particularly well tested
yet.  It'd be loads easier if we didn't have to worry about XIP.

 arch/arm/kernel/etm.c         |    2 +-
 arch/arm/kernel/smp.c         |    4 +-
 arch/arm/kernel/vmlinux.lds.S |  133 +++++++++++++++++++++-------------------
 arch/arm/mm/init.c            |    2 +-
 4 files changed, 74 insertions(+), 67 deletions(-)

diff --git a/arch/arm/kernel/etm.c b/arch/arm/kernel/etm.c
index 11db628..ca477d2 100644
--- a/arch/arm/kernel/etm.c
+++ b/arch/arm/kernel/etm.c
@@ -114,7 +114,7 @@ static int trace_start(struct tracectx *t)
 		return -EFAULT;
 	}
 
-	etm_setup_address_range(t, 1, (unsigned long)_stext,
+	etm_setup_address_range(t, 1, (unsigned long)_text,
 			(unsigned long)_etext, 0, 0);
 	etm_writel(t, 0, ETMR_TRACEENCTRL2);
 	etm_writel(t, 0, ETMR_TRACESSCTRL);
diff --git a/arch/arm/kernel/smp.c b/arch/arm/kernel/smp.c
index 4539ebc..8ad77d3 100644
--- a/arch/arm/kernel/smp.c
+++ b/arch/arm/kernel/smp.c
@@ -95,7 +95,7 @@ int __cpuinit __cpu_up(unsigned int cpu)
 #ifndef CONFIG_HOTPLUG_CPU
 		identity_mapping_add(pgd, __pa(__init_begin), __pa(__init_end));
 #endif
-		identity_mapping_add(pgd, __pa(_stext), __pa(_etext));
+		identity_mapping_add(pgd, __pa(_text), __pa(_etext));
 		identity_mapping_add(pgd, __pa(_sdata), __pa(_edata));
 	}
 
@@ -143,7 +143,7 @@ int __cpuinit __cpu_up(unsigned int cpu)
 #ifndef CONFIG_HOTPLUG_CPU
 		identity_mapping_del(pgd, __pa(__init_begin), __pa(__init_end));
 #endif
-		identity_mapping_del(pgd, __pa(_stext), __pa(_etext));
+		identity_mapping_del(pgd, __pa(_text), __pa(_etext));
 		identity_mapping_del(pgd, __pa(_sdata), __pa(_edata));
 	}
 
diff --git a/arch/arm/kernel/vmlinux.lds.S b/arch/arm/kernel/vmlinux.lds.S
index 45b5651..9cd098b 100644
--- a/arch/arm/kernel/vmlinux.lds.S
+++ b/arch/arm/kernel/vmlinux.lds.S
@@ -32,46 +32,98 @@ jiffies = jiffies_64 + 4;
 
 SECTIONS
 {
+	/*
+	 * unwind exit sections must be discarded before the rest of the
+	 * unwind sections get included.
+	 */
+	/DISCARD/ : {
+		*(.ARM.exidx.exit.text)
+		*(.ARM.extab.exit.text)
+		ARM_CPU_DISCARD(*(.ARM.exidx.cpuexit.text))
+		ARM_CPU_DISCARD(*(.ARM.extab.cpuexit.text))
+#ifndef CONFIG_HOTPLUG
+		*(.ARM.exidx.devexit.text)
+		*(.ARM.extab.devexit.text)
+#endif
+#ifndef CONFIG_MMU
+		*(.fixup)
+		*(__ex_table)
+#endif
+#ifndef CONFIG_SMP_ON_UP
+		*(.alt.smp.init)
+#endif
+	}
+
 #ifdef CONFIG_XIP_KERNEL
 	. = XIP_VIRT_ADDR(CONFIG_XIP_PHYS_ADDR);
 #else
 	. = PAGE_OFFSET + TEXT_OFFSET;
 #endif
 
-	.init : {			/* Init code and data		*/
-		_stext = .;
-		_sinittext = .;
-			HEAD_TEXT
-			INIT_TEXT
-		_einittext = .;
+	_text = .;
+	HEAD_TEXT_SECTION
+
+	.text : {			/* Real text segment		*/
+		_stext = .;		/* Text and read-only data	*/
+		__exception_text_start = .;
+		*(.exception.text)
+		__exception_text_end = .;
+		IRQENTRY_TEXT
+		TEXT_TEXT
+		SCHED_TEXT
+		LOCK_TEXT
+		KPROBES_TEXT
+#ifdef CONFIG_MMU
+		*(.fixup)
+#endif
+		*(.gnu.warning)
+		*(.glue_7)
+		*(.glue_7t)
+		. = ALIGN(4);
+		*(.got)			/* Global offset table		*/
+		ARM_CPU_KEEP(PROC_INFO)
+		_etext = .;		/* End of text and rodata section */
+	}
+
+#ifndef CONFIG_XIP_KERNEL
+	. = ALIGN(PAGE_SIZE);
+	__init_begin = .;
+#endif
+	INIT_TEXT_SECTION(0)
+	.init.proc.info : {
 		ARM_CPU_DISCARD(PROC_INFO)
+	}
+	.init.arch.info : {
 		__arch_info_begin = .;
 			*(.arch.info.init)
 		__arch_info_end = .;
+	}
+	.init.tagtable : {
 		__tagtable_begin = .;
 			*(.taglist.init)
 		__tagtable_end = .;
+	}
 #ifdef CONFIG_SMP_ON_UP
+	.init.smpalt : {
 		__smpalt_begin = .;
 			*(.alt.smp.init)
 		__smpalt_end = .;
+	}
 #endif
-
+	.init.pv_table : {
 		__pv_table_begin = .;
 			*(.pv_table)
 		__pv_table_end = .;
-
+	}
+	.init.data : {
+#ifndef CONFIG_XIP_KERNEL
+		INIT_DATA
+#endif
 		INIT_SETUP(16)
-
 		INIT_CALLS
 		CON_INITCALL
 		SECURITY_INITCALL
 		INIT_RAM_FS
-
-#ifndef CONFIG_XIP_KERNEL
-		__init_begin = _stext;
-		INIT_DATA
-#endif
 	}
 
 	PERCPU(PAGE_SIZE)
@@ -79,49 +131,10 @@ SECTIONS
 #ifndef CONFIG_XIP_KERNEL
 	. = ALIGN(PAGE_SIZE);
 	__init_end = .;
-#endif
 
-	/*
-	 * unwind exit sections must be discarded before the rest of the
-	 * unwind sections get included.
-	 */
-	/DISCARD/ : {
-		*(.ARM.exidx.exit.text)
-		*(.ARM.extab.exit.text)
-		ARM_CPU_DISCARD(*(.ARM.exidx.cpuexit.text))
-		ARM_CPU_DISCARD(*(.ARM.extab.cpuexit.text))
-#ifndef CONFIG_HOTPLUG
-		*(.ARM.exidx.devexit.text)
-		*(.ARM.extab.devexit.text)
+	_data = .;
+	_sdata = .;
 #endif
-#ifndef CONFIG_MMU
-		*(.fixup)
-		*(__ex_table)
-#endif
-	}
-
-	.text : {			/* Real text segment		*/
-		_text = .;		/* Text and read-only data	*/
-			__exception_text_start = .;
-			*(.exception.text)
-			__exception_text_end = .;
-			IRQENTRY_TEXT
-			TEXT_TEXT
-			SCHED_TEXT
-			LOCK_TEXT
-			KPROBES_TEXT
-#ifdef CONFIG_MMU
-			*(.fixup)
-#endif
-			*(.gnu.warning)
-			*(.rodata)
-			*(.rodata.*)
-			*(.glue_7)
-			*(.glue_7t)
-		. = ALIGN(4);
-		*(.got)			/* Global offset table		*/
-			ARM_CPU_KEEP(PROC_INFO)
-	}
 
 	RO_DATA(PAGE_SIZE)
 
@@ -142,8 +155,6 @@ SECTIONS
 	}
 #endif
 
-	_etext = .;			/* End of text and rodata section */
-
 #ifdef CONFIG_XIP_KERNEL
 	__data_loc = ALIGN(4);		/* location in binary */
 	. = PAGE_OFFSET + TEXT_OFFSET;
@@ -153,8 +164,10 @@ SECTIONS
 #endif
 
 	.data : AT(__data_loc) {
+#ifdef CONFIG_XIP_KERNEL
 		_data = .;		/* address in memory */
 		_sdata = .;
+#endif
 
 		/*
 		 * first, the init task union, aligned
@@ -259,12 +272,6 @@ SECTIONS
 
 	/* Default discards */
 	DISCARDS
-
-#ifndef CONFIG_SMP_ON_UP
-	/DISCARD/ : {
-		*(.alt.smp.init)
-	}
-#endif
 }
 
 /*
diff --git a/arch/arm/mm/init.c b/arch/arm/mm/init.c
index cddd684..9e3c3c3 100644
--- a/arch/arm/mm/init.c
+++ b/arch/arm/mm/init.c
@@ -294,7 +294,7 @@ void __init arm_memblock_init(struct meminfo *mi, struct machine_desc *mdesc)
 #ifdef CONFIG_XIP_KERNEL
 	memblock_reserve(__pa(_sdata), _end - _sdata);
 #else
-	memblock_reserve(__pa(_stext), _end - _stext);
+	memblock_reserve(__pa(_text), _end - _text);
 #endif
 #ifdef CONFIG_BLK_DEV_INITRD
 	if (phys_initrd_size &&

ARM: relocation out of range (when loading a module)

[Nicolas Pitre]

On Fri, 11 Feb 2011, Russell King - ARM Linux wrote:

> On Fri, Feb 11, 2011 at 02:25:28PM +0000, Dave Martin wrote:
> >  if we can also move the initramfs somewhere less obstructive in the
> > future, then so much the better.
> 
> This may do it, and it might even work.  Not particularly well tested
> yet.  It'd be loads easier if we didn't have to worry about XIP.

Is anyone actually still using XIP these days?  Now that NOR flash is 
not that much used anymore, if maintaining XIP support is more trouble 
than it is worth we could simply deprecate it.  The linker script has 
become quite more hairy than it used to be, and from a quick look I'm 
not even sure if all the stuff in there is properly located for XIP.

Otherwise, for the __init stuff move:

Acked-by: Nicolas Pitre <nicolas.pitre@linaro.org>


Nicolas

ARM: relocation out of range (when loading a module)

[Alexander Holler]

Hello,

Am 12.01.2011 19:42, schrieb Russell King - ARM Linux:

> As the overall feeling at the time was "don't use large initrds" it's
> something I've never really cared about - and I'm still of the opinion
> that 16MB of compressed initrd/initramfs is rather silly.

Hmm, sorry, I have to disagree. ;)
E.g. 12MB out of the 16MB in the initramfs I've used are modules.
I've also created some larger initramfs wich included  X11 and such, and 
I find it very convenient to use the initramfs to include a rootfs in 
the the kernel and so only have to handle one file for all. E.g. just 
load the kernel with the included rootfs via tftp and you are done.

Sure this can all be done using an external initrd, but that is much 
less comfortable to use.

At least I like that feature since it is available and won't miss it now. ;)

Regards,

Alexander

ARM: relocation out of range (when loading a module)

[Russell King - ARM Linux]

On Thu, Jan 13, 2011 at 06:50:05AM +0100, Alexander Holler wrote:
> Hello,
>
> Am 12.01.2011 19:42, schrieb Russell King - ARM Linux:
>
>> As the overall feeling at the time was "don't use large initrds" it's
>> something I've never really cared about - and I'm still of the opinion
>> that 16MB of compressed initrd/initramfs is rather silly.
>
> Hmm, sorry, I have to disagree. ;)
> E.g. 12MB out of the 16MB in the initramfs I've used are modules.
> I've also created some larger initramfs wich included  X11 and such, and  
> I find it very convenient to use the initramfs to include a rootfs in  
> the the kernel and so only have to handle one file for all. E.g. just  
> load the kernel with the included rootfs via tftp and you are done.

Oddly, that's why I wrote the bootp wrapper, which predates the
currently broken initrd-in-kernel idea.

> Sure this can all be done using an external initrd, but that is much  
> less comfortable to use.
>
> At least I like that feature since it is available and won't miss it now. ;)

The feature isn't available if it doesn't work.  It's not going to get
fixed for 2.6.38 - it'll be 2.6.39 due to the timing and complexity of
the problem.

ARM: relocation out of range (when loading a module)

[Alexander Holler]

Am 13.01.2011 11:04, schrieb Russell King - ARM Linux:

>> At least I like that feature since it is available and won't miss it now. ;)
>
> The feature isn't available if it doesn't work.  It's not going to get
> fixed for 2.6.38 - it'll be 2.6.39 due to the timing and complexity of
> the problem.

I just wanted to express that I like that feature a lot and I didn't 
wanted to ask for a quick fix nor did I wanted to make pressure.

I've tried a the patch from Rabin Vincent (along with an alignment) and 
it works. It might not be the best solution and I certainly waste some 
memory (no free), but I don't care.

Regards,

Alexander

ARM: relocation out of range (when loading a module)

[Matthieu CASTET]

Russell King - ARM Linux a ?crit :
> On Tue, Jan 11, 2011 at 09:16:38PM +0530, Rabin Vincent wrote:
>> It's possible to hack around this by placing the initramfs at the end of
>> the kernel image rather than at the beginning with the rest of the init
>> data.  Something like the below should work, although you should also
>> probably take care of alignment and also have this section freed when
>> the rest of the init data is freed.
> 
> You're then running into problems as _sdata.._edata is copied to RAM on
> XIP kernels, and you really don't want to waste time copying the
> initramfs to RAM.
> 
But in this case initramfs is after edata and before bss.
So where is the problem ?


Matthieu

ARM: relocation out of range (when loading a module)

[Russell King - ARM Linux]

On Wed, Jan 12, 2011 at 05:25:28PM +0100, Matthieu CASTET wrote:
> Russell King - ARM Linux a ?crit :
>> On Tue, Jan 11, 2011 at 09:16:38PM +0530, Rabin Vincent wrote:
>>> It's possible to hack around this by placing the initramfs at the end of
>>> the kernel image rather than at the beginning with the rest of the init
>>> data.  Something like the below should work, although you should also
>>> probably take care of alignment and also have this section freed when
>>> the rest of the init data is freed.
>>
>> You're then running into problems as _sdata.._edata is copied to RAM on
>> XIP kernels, and you really don't want to waste time copying the
>> initramfs to RAM.
>>
> But in this case initramfs is after edata and before bss.

Right, so that isn't a concern.

> So where is the problem ?

There is a problem with it, as noted by Rabin.  The initramfs will
never be freed.  Let's try to find a proper solution to this rather
than everyone sticking their half-baked ideas at it.  Let's see if
we can rearrange the _entire_ layout in a way that sorts this out
once and for all without making things even more complicated than
they already are today.

We've already got enough variables to think about here without having
yet more freeable sections scattered in the middle of the existing
layout.