poc: link modules into one .ko

poc: link modules into one .ko

[Jan Engelhardt]

I went over the suggestion of collapsing module files again [1]. For 
fun and research, instead of combining specific modules, I have set out 
to combine everything into a single module.

[1] http://marc.info/?l=netfilter-devel&m=121148631320895&w=2

All .ko modules together clock in at 1037772 bytes of diskspace, and 
they are likely to use just as much kernel memory when loaded (strip a 
few ELF stuff, and then round up to the next PAGE_SIZE again).

After the patch, the stats are much different:
-rw-r--r-- 1 jengelh 160910 2008-09-06 18:15 xtables_ext.ko
-rw-r--r-- 1 jengelh  57967 2008-09-06 18:15 xtables_xct.ko
-rw-r--r-- 1 jengelh  27295 2008-09-06 18:15 xtables_xnat.ko
-rw-r--r-- 1 jengelh  47550 2008-09-06 18:15 xtables_xv6.ko

293722 bytes, that is approximately a 70% reduction in disk and memory 
usage. (There are three modules not in this POC xtables_*.ko.) However, 
as a result, currently *all* extensions are loaded, which might incur 
extra memory being used.

The proof-of-concept patch is not too pretty, but it extrapolates one 
way to deal with the module flood (which has been inflating compile 
times and whatnot), and it does so without moving tons of code around. I 
think the only way to shove off more bytes is to revamp the API a bit 
and even try to reduce the amount of .o files during build.

The four xtables_*.ko files represent (no dependencies), (dependency on 
conntrack), (dependency on NAT) and (dependency on IPv6). Of course we 
should be keeping an eye on users who want a tiny system and not pull in 
unneeded dependencies - hence the split into actually four .ko files.

Interested parties are directed to the top commit of the koredux 
("reduction of number of .ko files") branch:

  git://dev.medozas.de/linux koredux
  http://dev.medozas.de/gitweb.cgi?p=linux;h=koredux;a=shortlog

Re: poc: link modules into one .ko

[Patrick McHardy]

Jan Engelhardt wrote:
> I went over the suggestion of collapsing module files again [1]. For 
> fun and research, instead of combining specific modules, I have set out 
> to combine everything into a single module.
> 
> [1] http://marc.info/?l=netfilter-devel&m=121148631320895&w=2
> 
> All .ko modules together clock in at 1037772 bytes of diskspace, and 
> they are likely to use just as much kernel memory when loaded (strip a 
> few ELF stuff, and then round up to the next PAGE_SIZE again).
> 
> After the patch, the stats are much different:
> -rw-r--r-- 1 jengelh 160910 2008-09-06 18:15 xtables_ext.ko
> -rw-r--r-- 1 jengelh  57967 2008-09-06 18:15 xtables_xct.ko
> -rw-r--r-- 1 jengelh  27295 2008-09-06 18:15 xtables_xnat.ko
> -rw-r--r-- 1 jengelh  47550 2008-09-06 18:15 xtables_xv6.ko
> 
> 293722 bytes, that is approximately a 70% reduction in disk and memory 
> usage. (There are three modules not in this POC xtables_*.ko.) However, 
> as a result, currently *all* extensions are loaded, which might incur 
> extra memory being used.
> 
> The proof-of-concept patch is not too pretty, but it extrapolates one 
> way to deal with the module flood (which has been inflating compile 
> times and whatnot), and it does so without moving tons of code around. I 
> think the only way to shove off more bytes is to revamp the API a bit 
> and even try to reduce the amount of .o files during build.

Its really not pretty :) While I agree that this does make sense
for the commonly used modules and distribution kernels, I don't
think it belongs here. A kbuild option to combine related modules
would be preferrable in my opinion.