ARM: mm: Could I change module space size or place modules in vmalloc area?

ARM: mm: Could I change module space size or place modules in vmalloc area?

[Gioh Kim]

Hi,

I run out of module space because I have several big driver modules.
I know I can strip the modules to decrease size but I need debug info now.

The default size of module is 16MB and the size is statically defined in the
header file. 
But a description for the module space size tells that it can be
configurable at most 32MB.

I have changed the module space size to 18MB and tested my platform.
It has been looking good.

I am not sure my patch is proper solution.
Anyway, could I configure the module space size?

Or could I place the modules into vmalloc area?



Signed-off-by: Gioh Kim <gioh.kim@lge.com>
---
 arch/arm/Kconfig              |    4 ++++
 arch/arm/include/asm/memory.h |   10 +++++++---
 2 files changed, 11 insertions(+), 3 deletions(-)

diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig index c1f1a7e..cf1fb55
100644
--- a/arch/arm/Kconfig
+++ b/arch/arm/Kconfig
@@ -2257,6 +2257,10 @@ config ARM_CPU_SUSPEND

 endmenu

+config MODULES_AREA_SIZE
+       int
+       default 0x1000000
+
 source "net/Kconfig"

 source "drivers/Kconfig"
diff --git a/arch/arm/include/asm/memory.h b/arch/arm/include/asm/memory.h
index 6976b03..3396758 100644
--- a/arch/arm/include/asm/memory.h
+++ b/arch/arm/include/asm/memory.h
@@ -32,13 +32,17 @@

 #ifdef CONFIG_MMU

+#if CONFIG_MODULES_AREA_SIZE > SZ_32M
+#error Too much space for modules
+#endif
+
 /*
  * PAGE_OFFSET - the virtual address of the start of the kernel image
  * TASK_SIZE - the maximum size of a user space task.
  * TASK_UNMAPPED_BASE - the lower boundary of the mmap VM area
  */
-#define PAGE_OFFSET            UL(CONFIG_PAGE_OFFSET)
-#define TASK_SIZE              (UL(CONFIG_PAGE_OFFSET) - UL(SZ_16M))
+#define PAGE_OFFSET    UL(CONFIG_PAGE_OFFSET)
+#define TASK_SIZE      (UL(CONFIG_PAGE_OFFSET) -
UL(CONFIG_MODULES_AREA_SIZE))
 #define TASK_UNMAPPED_BASE     ALIGN(TASK_SIZE / 3, SZ_16M)

 /*
@@ -51,7 +55,7 @@
  * and PAGE_OFFSET - it must be within 32MB of the kernel text.
  */
 #ifndef CONFIG_THUMB2_KERNEL
-#define MODULES_VADDR          (PAGE_OFFSET - SZ_16M)
+#define MODULES_VADDR          (PAGE_OFFSET - CONFIG_MODULES_AREA_SIZE)
 #else
 /* smaller range for Thumb-2 symbols relocation (2^24)*/
 #define MODULES_VADDR          (PAGE_OFFSET - SZ_8M)
--
1.7.9.5

ARM: mm: Could I change module space size or place modules in vmalloc area?

[Baruch Siach]

Hi Gioh,

On Thu, Jan 02, 2014 at 07:04:13PM +0900, Gioh Kim wrote:
> I run out of module space because I have several big driver modules.
> I know I can strip the modules to decrease size but I need debug info now.

Are you sure you need the debug info in kernel memory? I don't think the 
kernel is actually able to parse DWARF. You can load stripped binaries into 
the kernel, and still use the debug info with whatever tool you have.

baruch

-- 
     http://baruch.siach.name/blog/                  ~. .~   Tk Open Systems
=}------------------------------------------------ooO--U--Ooo------------{=
   - baruch at tkos.co.il - tel: +972.2.679.5364, http://www.tkos.co.il -

ARM: mm: Could I change module space size or place modules in vmalloc area?

[Gioh Kim]

Thank you for reply.

> -----Original Message-----
> From: Baruch Siach [mailto:baruch at tkos.co.il]
> Sent: Thursday, January 02, 2014 7:14 PM
> To: Gioh Kim
> Cc: Russell King; linux-mm at kvack.org; linux-arm-kernel; HyoJun Im
> Subject: Re: ARM: mm: Could I change module space size or place modules in
> vmalloc area?
> 
> Hi Gioh,
> 
> On Thu, Jan 02, 2014 at 07:04:13PM +0900, Gioh Kim wrote:
> > I run out of module space because I have several big driver modules.
> > I know I can strip the modules to decrease size but I need debug info
> now.
> 
> Are you sure you need the debug info in kernel memory? I don't think the
> kernel is actually able to parse DWARF. You can load stripped binaries
> into the kernel, and still use the debug info with whatever tool you have.

I agree you but driver developers of another team don't agree.
I don't know why but they say they will strip drivers later :-(
So I need to increase modules space size.


> 
> baruch
> 
> --
>      http://baruch.siach.name/blog/                  ~. .~   Tk Open
Systems
> =}------------------------------------------------ooO--U--Ooo------------
> {=
>    - baruch at tkos.co.il - tel: +972.2.679.5364, http://www.tkos.co.il -

ARM: mm: Could I change module space size or place modules in vmalloc area?

[Russell King - ARM Linux]

On Fri, Jan 03, 2014 at 09:39:31AM +0900, Gioh Kim wrote:
> Thank you for reply.
> 
> > -----Original Message-----
> > From: Baruch Siach [mailto:baruch at tkos.co.il]
> > Sent: Thursday, January 02, 2014 7:14 PM
> > To: Gioh Kim
> > Cc: Russell King; linux-mm at kvack.org; linux-arm-kernel; HyoJun Im
> > Subject: Re: ARM: mm: Could I change module space size or place modules in
> > vmalloc area?
> > 
> > Hi Gioh,
> > 
> > On Thu, Jan 02, 2014 at 07:04:13PM +0900, Gioh Kim wrote:
> > > I run out of module space because I have several big driver modules.
> > > I know I can strip the modules to decrease size but I need debug info
> > now.
> > 
> > Are you sure you need the debug info in kernel memory? I don't think the
> > kernel is actually able to parse DWARF. You can load stripped binaries
> > into the kernel, and still use the debug info with whatever tool you have.
> 
> I agree you but driver developers of another team don't agree.
> I don't know why but they say they will strip drivers later :-(
> So I need to increase modules space size.

ARM can only branch relatively within +/- 32MB.  Hence, with a module
space of 16MB, modules can reach up to a maximum 16MB into the direct-
mapped kernel image.  As module space increases in size, so that figure
decreases.  So, if module space were to be 40MB, the maximum size of the
kernel binary would be 8MB.

You want to look at a line similar to this:

      .text : 0xc0008000 - 0xc031eda0   (3164 kB)

Also, note this:

    modules : 0xbf000000 - 0xc0000000   (  16 MB)

If the difference between the lowest module address (0xbf000000) and the
highest of .text is greater than 32MB, it's impossible to load modules -
they will fail to link.

What is the size of your kernel text? (show us the line(s) like the above.)

Thanks.

-- 
FTTC broadband for 0.8mile line: 5.8Mbps down 500kbps up.  Estimation
in database were 13.1 to 19Mbit for a good line, about 7.5+ for a bad.
Estimate before purchase was "up to 13.2Mbit".

ARM: mm: Could I change module space size or place modules in vmalloc area?

[Gioh Kim]

> -----Original Message-----
> From: Russell King - ARM Linux [mailto:linux at arm.linux.org.uk]
> Sent: Friday, January 03, 2014 9:47 AM
> To: Gioh Kim
> Cc: 'Baruch Siach'; linux-mm at kvack.org; 'linux-arm-kernel'; 'HyoJun Im'
> Subject: Re: ARM: mm: Could I change module space size or place modules in
> vmalloc area?
> 
> On Fri, Jan 03, 2014 at 09:39:31AM +0900, Gioh Kim wrote:
> > Thank you for reply.
> >
> > > -----Original Message-----
> > > From: Baruch Siach [mailto:baruch at tkos.co.il]
> > > Sent: Thursday, January 02, 2014 7:14 PM
> > > To: Gioh Kim
> > > Cc: Russell King; linux-mm at kvack.org; linux-arm-kernel; HyoJun Im
> > > Subject: Re: ARM: mm: Could I change module space size or place
> > > modules in vmalloc area?
> > >
> > > Hi Gioh,
> > >
> > > On Thu, Jan 02, 2014 at 07:04:13PM +0900, Gioh Kim wrote:
> > > > I run out of module space because I have several big driver modules.
> > > > I know I can strip the modules to decrease size but I need debug
> > > > info
> > > now.
> > >
> > > Are you sure you need the debug info in kernel memory? I don't think
> > > the kernel is actually able to parse DWARF. You can load stripped
> > > binaries into the kernel, and still use the debug info with whatever
> tool you have.
> >
> > I agree you but driver developers of another team don't agree.
> > I don't know why but they say they will strip drivers later :-( So I
> > need to increase modules space size.
> 
> ARM can only branch relatively within +/- 32MB.  Hence, with a module
> space of 16MB, modules can reach up to a maximum 16MB into the direct-
> mapped kernel image.  As module space increases in size, so that figure
> decreases.  So, if module space were to be 40MB, the maximum size of the
> kernel binary would be 8MB.
> 
> You want to look at a line similar to this:
> 
>       .text : 0xc0008000 - 0xc031eda0   (3164 kB)
> 
> Also, note this:
> 
>     modules : 0xbf000000 - 0xc0000000   (  16 MB)
> 
> If the difference between the lowest module address (0xbf000000) and the
> highest of .text is greater than 32MB, it's impossible to load modules -
> they will fail to link.
> 
> What is the size of your kernel text? (show us the line(s) like the
above.)

My kernel size is about 8MB as below.
[    0.000000]     modules : 0x7f000000 - 0x80000000   (  16 MB)
[    0.000000]       .text : 0x80008000 - 0x8085d2b0   (8533 kB)

I think I can increase the modules space size into 20MB.
Thanks for your reply.

> 
> Thanks.
> 
> --
> FTTC broadband for 0.8mile line: 5.8Mbps down 500kbps up.  Estimation in
> database were 13.1 to 19Mbit for a good line, about 7.5+ for a bad.
> Estimate before purchase was "up to 13.2Mbit".

ARM: mm: Could I change module space size or place modules in vmalloc area?

[Jianguo Wu]

On 2014/1/3 8:47, Russell King - ARM Linux wrote:

> On Fri, Jan 03, 2014 at 09:39:31AM +0900, Gioh Kim wrote:
>> Thank you for reply.
>>
>>> -----Original Message-----
>>> From: Baruch Siach [mailto:baruch at tkos.co.il]
>>> Sent: Thursday, January 02, 2014 7:14 PM
>>> To: Gioh Kim
>>> Cc: Russell King; linux-mm at kvack.org; linux-arm-kernel; HyoJun Im
>>> Subject: Re: ARM: mm: Could I change module space size or place modules in
>>> vmalloc area?
>>>
>>> Hi Gioh,
>>>
>>> On Thu, Jan 02, 2014 at 07:04:13PM +0900, Gioh Kim wrote:
>>>> I run out of module space because I have several big driver modules.
>>>> I know I can strip the modules to decrease size but I need debug info
>>> now.
>>>
>>> Are you sure you need the debug info in kernel memory? I don't think the
>>> kernel is actually able to parse DWARF. You can load stripped binaries
>>> into the kernel, and still use the debug info with whatever tool you have.
>>
>> I agree you but driver developers of another team don't agree.
>> I don't know why but they say they will strip drivers later :-(
>> So I need to increase modules space size.
> 
> ARM can only branch relatively within +/- 32MB.  Hence, with a module
> space of 16MB, modules can reach up to a maximum 16MB into the direct-
> mapped kernel image.  As module space increases in size, so that figure
> decreases.  So, if module space were to be 40MB, the maximum size of the
> kernel binary would be 8MB.
> 

Hi Russell ,Arnd or Will,

I encountered the same situation in arm64, I loaded 80+ modules in arm64, and
run out of module address space(64M). Why the module space is restricted to 64M,
can it be expanded?  

In commit 257cb2519(arm64: Loadable modules), it mentioned that " Loadable modules
are loaded 64MB below the kernel image due to branch relocation restrictions",
"branch relocation restrictions" is a AArch64 instruction set restrictions?

Thanks,
Jianguo Wu.

> You want to look at a line similar to this:
> 
>       .text : 0xc0008000 - 0xc031eda0   (3164 kB)
> 
> Also, note this:
> 
>     modules : 0xbf000000 - 0xc0000000   (  16 MB)
> 
> If the difference between the lowest module address (0xbf000000) and the
> highest of .text is greater than 32MB, it's impossible to load modules -
> they will fail to link.
> 
> What is the size of your kernel text? (show us the line(s) like the above.)
> 
> Thanks.
> 

ARM: mm: Could I change module space size or place modules in vmalloc area?

[Will Deacon]

On Sat, Apr 26, 2014 at 03:12:40AM +0100, Jianguo Wu wrote:
> On 2014/1/3 8:47, Russell King - ARM Linux wrote:
> 
> > On Fri, Jan 03, 2014 at 09:39:31AM +0900, Gioh Kim wrote:
> >> Thank you for reply.
> >>
> >>> -----Original Message-----
> >>> From: Baruch Siach [mailto:baruch at tkos.co.il]
> >>> Sent: Thursday, January 02, 2014 7:14 PM
> >>> To: Gioh Kim
> >>> Cc: Russell King; linux-mm at kvack.org; linux-arm-kernel; HyoJun Im
> >>> Subject: Re: ARM: mm: Could I change module space size or place modules in
> >>> vmalloc area?
> >>>
> >>> Hi Gioh,
> >>>
> >>> On Thu, Jan 02, 2014 at 07:04:13PM +0900, Gioh Kim wrote:
> >>>> I run out of module space because I have several big driver modules.
> >>>> I know I can strip the modules to decrease size but I need debug info
> >>> now.
> >>>
> >>> Are you sure you need the debug info in kernel memory? I don't think the
> >>> kernel is actually able to parse DWARF. You can load stripped binaries
> >>> into the kernel, and still use the debug info with whatever tool you have.
> >>
> >> I agree you but driver developers of another team don't agree.
> >> I don't know why but they say they will strip drivers later :-(
> >> So I need to increase modules space size.
> > 
> > ARM can only branch relatively within +/- 32MB.  Hence, with a module
> > space of 16MB, modules can reach up to a maximum 16MB into the direct-
> > mapped kernel image.  As module space increases in size, so that figure
> > decreases.  So, if module space were to be 40MB, the maximum size of the
> > kernel binary would be 8MB.
> > 
> 
> Hi Russell ,Arnd or Will,
> 
> I encountered the same situation in arm64, I loaded 80+ modules in arm64, and
> run out of module address space(64M). Why the module space is restricted to 64M,
> can it be expanded?  

The module space is restricted to 64M on AArch64 because the range of the BL
instruction is += 128M. In order to call kernel functions, we need to ensure
that this range is large enough and therefore place the modules 64M below the
kernel text, allowing 64M for modules and 64M for the kernel text. We could
probably improve this a bit by assuming a maximum size for the kernel text.

If we want to remove the problem altogether, we'd need to hack the module
loader to insert trampolines (fiddly) or somehow persuade the tools to use
indirect branches (BLR) for all calls (inefficient).

Will

ARM: mm: Could I change module space size or place modules in vmalloc area?

[Arnd Bergmann]

On Tuesday 29 April 2014 12:19:46 Will Deacon wrote:
> On Sat, Apr 26, 2014 at 03:12:40AM +0100, Jianguo Wu wrote:
> > On 2014/1/3 8:47, Russell King - ARM Linux wrote:
> > > ARM can only branch relatively within +/- 32MB.  Hence, with a module
> > > space of 16MB, modules can reach up to a maximum 16MB into the direct-
> > > mapped kernel image.  As module space increases in size, so that figure
> > > decreases.  So, if module space were to be 40MB, the maximum size of the
> > > kernel binary would be 8MB.
> > > 
> > 
> > Hi Russell ,Arnd or Will,
> > 
> > I encountered the same situation in arm64, I loaded 80+ modules in arm64, and
> > run out of module address space(64M). Why the module space is restricted to 64M,
> > can it be expanded?  
> 
> The module space is restricted to 64M on AArch64 because the range of the BL
> instruction is += 128M. In order to call kernel functions, we need to ensure
> that this range is large enough and therefore place the modules 64M below the
> kernel text, allowing 64M for modules and 64M for the kernel text. We could
> probably improve this a bit by assuming a maximum size for the kernel text.
> 
> If we want to remove the problem altogether, we'd need to hack the module
> loader to insert trampolines (fiddly) or somehow persuade the tools to use
> indirect branches (BLR) for all calls (inefficient).

Well, there might also be a bug involved. Loading 80 modules should never
take up 64MB. The typical size of a loadable module should be a few dozen
kilobytes, although we have a few modules that are hundreds of kilobytes.

Jianguo Wu, can you send the defconfig you were using? Did you have
some debugging option enabled that increased the module size?

	Arnd

ARM: mm: Could I change module space size or place modules in vmalloc area?

[Will Deacon]

On Tue, Apr 29, 2014 at 12:35:34PM +0100, Arnd Bergmann wrote:
> On Tuesday 29 April 2014 12:19:46 Will Deacon wrote:
> > On Sat, Apr 26, 2014 at 03:12:40AM +0100, Jianguo Wu wrote:
> > > On 2014/1/3 8:47, Russell King - ARM Linux wrote:
> > > > ARM can only branch relatively within +/- 32MB.  Hence, with a module
> > > > space of 16MB, modules can reach up to a maximum 16MB into the direct-
> > > > mapped kernel image.  As module space increases in size, so that figure
> > > > decreases.  So, if module space were to be 40MB, the maximum size of the
> > > > kernel binary would be 8MB.
> > > > 
> > > 
> > > Hi Russell ,Arnd or Will,
> > > 
> > > I encountered the same situation in arm64, I loaded 80+ modules in arm64, and
> > > run out of module address space(64M). Why the module space is restricted to 64M,
> > > can it be expanded?  
> > 
> > The module space is restricted to 64M on AArch64 because the range of the BL
> > instruction is += 128M. In order to call kernel functions, we need to ensure
> > that this range is large enough and therefore place the modules 64M below the
> > kernel text, allowing 64M for modules and 64M for the kernel text. We could
> > probably improve this a bit by assuming a maximum size for the kernel text.
> > 
> > If we want to remove the problem altogether, we'd need to hack the module
> > loader to insert trampolines (fiddly) or somehow persuade the tools to use
> > indirect branches (BLR) for all calls (inefficient).
> 
> Well, there might also be a bug involved. Loading 80 modules should never
> take up 64MB. The typical size of a loadable module should be a few dozen
> kilobytes, although we have a few modules that are hundreds of kilobytes.
> 
> Jianguo Wu, can you send the defconfig you were using? Did you have
> some debugging option enabled that increased the module size?

If 64K pages are in use we could end up with a tonne of fragmentation but
yes, worth looking at the .config.

Will

ARM: mm: Could I change module space size or place modules in vmalloc area?

[Arnd Bergmann]

On Thursday 02 January 2014, Gioh Kim wrote:

> I run out of module space because I have several big driver modules.
> I know I can strip the modules to decrease size but I need debug info now.
> 
> The default size of module is 16MB and the size is statically defined in the
> header file. 
> But a description for the module space size tells that it can be
> configurable at most 32MB.
> 
> I have changed the module space size to 18MB and tested my platform.
> It has been looking good.
> 
> I am not sure my patch is proper solution.
> Anyway, could I configure the module space size?
> 
> Or could I place the modules into vmalloc area?
> 

Aside from the good comments that Russell made, I would remark that the
fact that you need multiple megabytes worth of modules indicates that you
are doing something wrong. Can you point to a git tree containing those
modules?

	Arnd

ARM: mm: Could I change module space size or place modules in vmalloc area?

[Russell King - ARM Linux]

On Fri, Jan 03, 2014 at 01:10:09PM +0100, Arnd Bergmann wrote:
> Aside from the good comments that Russell made, I would remark that the
> fact that you need multiple megabytes worth of modules indicates that you
> are doing something wrong. Can you point to a git tree containing those
> modules?

>From the comments which have been made, one point that seems to have
been identified is that if this module is first stripped and then
loaded, it can load, but if it's unstripped, it's too big.  This sounds
suboptimal to me - the debug info shouldn't be loaded into the kernel.

However, I guess there's bad interactions with module signing if you
don't do this and the module was signed with the debug info present,
so I don't think there's a good solution for this.

-- 
FTTC broadband for 0.8mile line: 5.8Mbps down 500kbps up.  Estimation
in database were 13.1 to 19Mbit for a good line, about 7.5+ for a bad.
Estimate before purchase was "up to 13.2Mbit".

ARM: mm: Could I change module space size or place modules in vmalloc area?

[Arnd Bergmann]

On Friday 03 January 2014, Russell King - ARM Linux wrote:
> On Fri, Jan 03, 2014 at 01:10:09PM +0100, Arnd Bergmann wrote:
> > Aside from the good comments that Russell made, I would remark that the
> > fact that you need multiple megabytes worth of modules indicates that you
> > are doing something wrong. Can you point to a git tree containing those
> > modules?
> 
> From the comments which have been made, one point that seems to have
> been identified is that if this module is first stripped and then
> loaded, it can load, but if it's unstripped, it's too big.  This sounds
> suboptimal to me - the debug info shouldn't be loaded into the kernel.

Reading the layout_and_allocate() function, that is probably the
intention already, and if something goes wrong there on ARM, it could be
fixed up in an arch specific module_frob_arch_sections() function.

> However, I guess there's bad interactions with module signing if you
> don't do this and the module was signed with the debug info present,
> so I don't think there's a good solution for this.

My point was another anyway: I can't think of any good reason why
you would end up with this many modules on any sane system. The only
cases I've seen so far are

- modules written in C++, with libstdc++ linked into the module
- a closed-source platform port hidden in a loadable module that
  contains all the device drivers and subsystems while ignoring the
  infrastructure we have in the kernel, and the possible legal
  implications.
- a bug in the module using large arrays that should just be
  dynamically allocated.
- device firmware statically linked into the module rather than
  loaded using request_firmware.

In each of these cases, the real answer is to fix the code they are
trying to load to do things in a more common way, especially if the
intention is to eventually merge the code upstream. It is of course
possible that they are indeed trying something valid, that's why
I asked to see the source code.

	Arnd

ARM: mm: Could I change module space size or place modules in vmalloc area?

[Gioh Kim]

I am sorry to read your mail so late.
My module had been a proprietary driver so that I requested to strip it and
got small size driver.

Thank you for attention.


> -----Original Message-----
> From: Arnd Bergmann [mailto:arnd at arndb.de]
> Sent: Friday, January 03, 2014 10:24 PM
> To: linux-arm-kernel at lists.infradead.org
> Cc: Russell King - ARM Linux; HyoJun Im; linux-mm at kvack.org; Gioh Kim
> Subject: Re: ARM: mm: Could I change module space size or place modules in
> vmalloc area?
> 
> On Friday 03 January 2014, Russell King - ARM Linux wrote:
> > On Fri, Jan 03, 2014 at 01:10:09PM +0100, Arnd Bergmann wrote:
> > > Aside from the good comments that Russell made, I would remark that
> > > the fact that you need multiple megabytes worth of modules indicates
> > > that you are doing something wrong. Can you point to a git tree
> > > containing those modules?
> >
> > From the comments which have been made, one point that seems to have
> > been identified is that if this module is first stripped and then
> > loaded, it can load, but if it's unstripped, it's too big.  This
> > sounds suboptimal to me - the debug info shouldn't be loaded into the
> kernel.
> 
> Reading the layout_and_allocate() function, that is probably the intention
> already, and if something goes wrong there on ARM, it could be fixed up in
> an arch specific module_frob_arch_sections() function.
> 
> > However, I guess there's bad interactions with module signing if you
> > don't do this and the module was signed with the debug info present,
> > so I don't think there's a good solution for this.
> 
> My point was another anyway: I can't think of any good reason why you
> would end up with this many modules on any sane system. The only cases
> I've seen so far are
> 
> - modules written in C++, with libstdc++ linked into the module
> - a closed-source platform port hidden in a loadable module that
>   contains all the device drivers and subsystems while ignoring the
>   infrastructure we have in the kernel, and the possible legal
>   implications.
> - a bug in the module using large arrays that should just be
>   dynamically allocated.
> - device firmware statically linked into the module rather than
>   loaded using request_firmware.
> 
> In each of these cases, the real answer is to fix the code they are trying
> to load to do things in a more common way, especially if the intention is
> to eventually merge the code upstream. It is of course possible that they
> are indeed trying something valid, that's why I asked to see the source
> code.
> 
> 	Arnd