[PATCH] Compress kernel modules on installation.

[PATCH] Compress kernel modules on installation.

[Steve Brokenshire]

Hi,

(I've sent this to the linux-kbuild and linux-kernel lists as this
patch modifies the Makefile.modinst file. I also don't subscribe to the 
linux-kbuild and linux-kernel mailing lists so can I have any replies
CC'ed to me please)

This patch allows kernel modules to be compressed when 'make
modules_install' is run after being copied to
the /lib/module/<version>/<...> directory which is useful if you have
module-init-tools installed with --enable-zlib. This patch adds an
option (MODULE_COMPRESS) to the kernel configuration file (specifically
init/Kconfig) so that the kernel modules will compressed if
MODULE_COMPRESS is set.

When MODULE_COMPRESS is set the output of 'make modules_install' will
go as the following:

INSTALL  drivers/fs/xfs/xfs.ko
COMPRESS drivers/fs/xfs/xfs.ko
INSTALL  drivers/fs/fat/fat.ko
COMPRESS drivers/fs/fat/fat.ko
...

I've tested my patch on kernel versions 2.6.24.2, 2.6.24-git12, 2.6.23.14 and
2.6.19 and they compile, install and compress into the respective
module directories without any errors.

I've also tested this with the uvcvideo (linux-uvc) kernel module (from
the SVN branch and with kernel versions 2.6.24.2, 2.6.24-git12, 2.6.23.14 and
2.6.19) as that uses Kbuild properly when installing the module and
after installing the uvcvideo the module is then compressed.
Unfortunately, I couldn't find any other kernel modules which used the
Kbuild system for installing their kernel modules. :(

I've included include/config/auto.conf in Makefile.modinst so that it
can check if MODULE_COMPRESS is set when installing the kernel modules.

Unfortunately when I ran mkinitrd (CentOS version) to create the initrd
image to go with the kernel, I get errors saying that certain kernel
modules don't exist despite the fact they do actually exist. When I
pass --allow-missing to mkinitrd, it seems to go fine but when booting
up with the system with the new initrd image I get error messages
saying that the modules don't exist.

A workaround is to compile the modules, don't have MODULE_COMPRESS set 
in .config, install the modules, run mkinitrd and copy it to /boot, set 
MODULE_COMPRESS in .config and then install the modules again but
compressed.

That's about it really. The only showstopper I feel is mkinitrd not
working properly with the compressed kernel modules.

Thanks,
Steve

---

Signed-off-by: Steve Brokenshire <sbrokenshire@xestia.co.uk>

diff -u -r -up a/init/Kconfig b/init/Kconfig
--- a/init/Kconfig	2008-02-02 16:34:29.000000000 +0000
+++ b/init/Kconfig	2008-02-03 09:55:52.000000000 +0000
@@ -725,6 +725,30 @@ config MODULE_FORCE_UNLOAD
 	  rmmod).  This is mainly for kernel developers and desperate users.
 	  If unsure, say N.
 
+config MODULE_COMPRESS
+	bool "Compress kernel modules on installation"
+	depends on MODULES
+	help
+	  This option compresses the kernel modules when 'make
+	  modules_install' is run.
+
+	  The modules will be compressed into the gzip (GNU zip) format
+	  and will use less space than an uncompressed kernel module would.
+
+	  When a kernel module is installed from outside of the main kernel
+	  source and uses the Kbuild system for installing modules then that
+	  kernel module will also be compressed when it is installed.
+
+	  When running mkinitrd you will find that an error message
+	  appears saying that it cannot find a certain kernel module.
+	  As a workaround, unset CONFIG_MODULE_COMPRESS, build the modules
+	  and install them, run mkinitrd and create the initrd image, place
+	  the initrd image in the correct place for booting, set
+	  CONFIG_MODULE_COMPRESS and then install the modules again.
+
+	  This options requires the module-init-tools package to be 
+	  configured with --enable-zlib.
+
 config MODVERSIONS
 	bool "Module versioning support"
 	depends on MODULES
diff -u -r -up a/scripts/Makefile.modinst b/scripts/Makefile.modinst
--- a/scripts/Makefile.modinst	2008-02-02 16:34:33.000000000 +0000
+++ b/scripts/Makefile.modinst	2008-02-02 17:21:46.000000000 +0000
@@ -5,6 +5,7 @@
 PHONY := __modinst
 __modinst:
 
+include include/config/auto.conf
 include scripts/Kbuild.include
 
 #
@@ -16,8 +17,15 @@ PHONY += $(modules)
 __modinst: $(modules)
 	@:
 
-quiet_cmd_modules_install = INSTALL $@
-      cmd_modules_install = mkdir -p $(2); cp $@ $(2) ; $(mod_strip_cmd) $(2)/$(notdir $@)
+quiet_cmd_modules_install = INSTALL  $@
+      cmd_modules_install = mkdir -p $(2); \
+				cp $@ $(2) ; \
+				$(mod_strip_cmd) $(2)/$(notdir $@)
+
+quiet_cmd_modules_compress = COMPRESS $@
+      cmd_modules_compress = gzip --best -c $(2)/`basename $@` \
+				 > $(2)/`basename $@`.gz; \
+				rm $(2)/`basename $@`
 
 # Modules built outside the kernel source tree go into extra by default
 INSTALL_MOD_DIR ?= extra
@@ -27,7 +35,8 @@ modinst_dir = $(if $(KBUILD_EXTMOD),$(ex
 
 $(modules):
 	$(call cmd,modules_install,$(MODLIB)/$(modinst_dir))
-
+	$(if $(CONFIG_MODULE_COMPRESS), \
+		$(call cmd,modules_compress,$(MODLIB)/$(modinst_dir)))
 
 # Declare the contents of the .PHONY variable as phony.  We keep that
 # information in a variable se we can use it in if_changed and friends.

Re: [PATCH] Compress kernel modules on installation.

[Oleg Verych]

On Mon, Feb 25, 2008 at 10:42 PM, Steve Brokenshire:
> Hi,
>
>  (I've sent this to the linux-kbuild and linux-kernel lists as this
>  patch modifies the Makefile.modinst file. I also don't subscribe to the
>  linux-kbuild and linux-kernel mailing lists so can I have any replies
>  CC'ed to me please)

And what if i like bzip2 (yea sometimes better) or 7zip (better if !EMBEDDED)
or whatever?

It's pure user/distro question.
_______

Re: [PATCH] Compress kernel modules on installation.

[Willy Tarreau]

On Mon, Feb 25, 2008 at 11:17:23PM +0100, Oleg Verych wrote:
> On Mon, Feb 25, 2008 at 10:42 PM, Steve Brokenshire:
> > Hi,
> >
> >  (I've sent this to the linux-kbuild and linux-kernel lists as this
> >  patch modifies the Makefile.modinst file. I also don't subscribe to the
> >  linux-kbuild and linux-kernel mailing lists so can I have any replies
> >  CC'ed to me please)
> 
> And what if i like bzip2 (yea sometimes better) or 7zip (better if !EMBEDDED)
> or whatever?
> 
> It's pure user/distro question.

not exactly, as only gzip is supported by module-init-tools. However,
if you come up with a patch to implement lzma or equivalent for
module-init-tools, it may be useful.

Willy

Re: [PATCH] Compress kernel modules on installation.

[Oleg Verych]

On Mon, Feb 25, 2008 at 11:19 PM, Willy Tarreau:
>
> On Mon, Feb 25, 2008 at 11:17:23PM +0100, Oleg Verych wrote:
>  > On Mon, Feb 25, 2008 at 10:42 PM, Steve Brokenshire:
>  > > Hi,
>  > >
>  > >  (I've sent this to the linux-kbuild and linux-kernel lists as this
>  > >  patch modifies the Makefile.modinst file. I also don't subscribe to the
>  > >  linux-kbuild and linux-kernel mailing lists so can I have any replies
>  > >  CC'ed to me please)
>  >
>  > And what if i like bzip2 (yea sometimes better) or 7zip (better if !EMBEDDED)
>  > or whatever?
>  >
>  > It's pure user/distro question.
>
>  not exactly, as only gzip is supported by module-init-tools. However,
>  if you come up with a patch to implement lzma or equivalent for
>  module-init-tools, it may be useful.

Oh, come on! It's userspace, i have scripts managing existence/compression
on per-file basis on my comp. If most distros just drop thousands of useless
(sometimes compressed) stuff to your harddisk, it's not kernel's or
modules-init-tools'
problem.

Also having compressed over compressed stuff, like on initramfs booting image
with such modules (if by defaults) may cause bigger file, more CPU overhead.

It is userspace and all that policy stuff, etc., etc.

Want to make a minidistro on lkml? Well, i see klibc working fine already.
_______

Re: [PATCH] Compress kernel modules on installation.

[Willy Tarreau]

On Mon, Feb 25, 2008 at 11:32:44PM +0100, Oleg Verych wrote:
> On Mon, Feb 25, 2008 at 11:19 PM, Willy Tarreau:
> >
> > On Mon, Feb 25, 2008 at 11:17:23PM +0100, Oleg Verych wrote:
> >  > On Mon, Feb 25, 2008 at 10:42 PM, Steve Brokenshire:
> >  > > Hi,
> >  > >
> >  > >  (I've sent this to the linux-kbuild and linux-kernel lists as this
> >  > >  patch modifies the Makefile.modinst file. I also don't subscribe to the
> >  > >  linux-kbuild and linux-kernel mailing lists so can I have any replies
> >  > >  CC'ed to me please)
> >  >
> >  > And what if i like bzip2 (yea sometimes better) or 7zip (better if !EMBEDDED)
> >  > or whatever?
> >  >
> >  > It's pure user/distro question.
> >
> >  not exactly, as only gzip is supported by module-init-tools. However,
> >  if you come up with a patch to implement lzma or equivalent for
> >  module-init-tools, it may be useful.
> 
> Oh, come on! It's userspace,

and your point is ... ?

> i have scripts managing existence/compression
> on per-file basis on my comp.

I'm definitely happy for you.

> If most distros just drop thousands of useless
> (sometimes compressed) stuff to your harddisk, it's not kernel's or
> modules-init-tools' problem.

who said there was a problem ?

> Also having compressed over compressed stuff, like on initramfs booting image
> with such modules (if by defaults) may cause bigger file, more CPU overhead.

Already known and obvious, thanks. But that's unrelated to the initial post.

> It is userspace and all that policy stuff, etc., etc.

A makefile is userspace too. I still fail to see your point.
 
> Want to make a minidistro on lkml? Well, i see klibc working fine already.

Huh? drinking before posting does no good it seems... :-)

Steve proposed a simple patch to ease modules compression during the
build process. Obviously it's userspace (as all the build process,
mind you). And there's nothing dirty nor funny there. Maybe you just
don't understand what it's useful for, but in this case it simply means
you don't need this feature :-)

Willy

Re: [PATCH] Compress kernel modules on installation.

[Willy Tarreau]

On Mon, Feb 25, 2008 at 09:42:09PM +0000, Steve Brokenshire wrote:
> Hi,
> 
> (I've sent this to the linux-kbuild and linux-kernel lists as this
> patch modifies the Makefile.modinst file. I also don't subscribe to the 
> linux-kbuild and linux-kernel mailing lists so can I have any replies
> CC'ed to me please)
> 
> This patch allows kernel modules to be compressed when 'make
> modules_install' is run after being copied to
> the /lib/module/<version>/<...> directory which is useful if you have
> module-init-tools installed with --enable-zlib. This patch adds an
> option (MODULE_COMPRESS) to the kernel configuration file (specifically
> init/Kconfig) so that the kernel modules will compressed if
> MODULE_COMPRESS is set.
> 
> When MODULE_COMPRESS is set the output of 'make modules_install' will
> go as the following:
> 
> INSTALL  drivers/fs/xfs/xfs.ko
> COMPRESS drivers/fs/xfs/xfs.ko
> INSTALL  drivers/fs/fat/fat.ko
> COMPRESS drivers/fs/fat/fat.ko
> ...
> 
> I've tested my patch on kernel versions 2.6.24.2, 2.6.24-git12, 2.6.23.14 and
> 2.6.19 and they compile, install and compress into the respective
> module directories without any errors.
> 
> I've also tested this with the uvcvideo (linux-uvc) kernel module (from
> the SVN branch and with kernel versions 2.6.24.2, 2.6.24-git12, 2.6.23.14 and
> 2.6.19) as that uses Kbuild properly when installing the module and
> after installing the uvcvideo the module is then compressed.
> Unfortunately, I couldn't find any other kernel modules which used the
> Kbuild system for installing their kernel modules. :(
> 
> I've included include/config/auto.conf in Makefile.modinst so that it
> can check if MODULE_COMPRESS is set when installing the kernel modules.
> 
> Unfortunately when I ran mkinitrd (CentOS version) to create the initrd
> image to go with the kernel, I get errors saying that certain kernel
> modules don't exist despite the fact they do actually exist. When I
> pass --allow-missing to mkinitrd, it seems to go fine but when booting
> up with the system with the new initrd image I get error messages
> saying that the modules don't exist.
> 
> A workaround is to compile the modules, don't have MODULE_COMPRESS set 
> in .config, install the modules, run mkinitrd and copy it to /boot, set 
> MODULE_COMPRESS in .config and then install the modules again but
> compressed.
> 
> That's about it really. The only showstopper I feel is mkinitrd not
> working properly with the compressed kernel modules.

Have you tried keeping the module names intact (.ko, not .ko.gz) ?
It's what I was doing with modutils in 2.4 and what I'm still doing
with module-init-tools in 2.6. While I don't particularly use mkinitrd,
I think that keeping the name intact is preferable and should help.

Regards,
Willy

Re: [PATCH] Compress kernel modules on installation.

[Adrian Bunk]

On Mon, Feb 25, 2008 at 11:21:38PM +0100, Willy Tarreau wrote:
> On Mon, Feb 25, 2008 at 09:42:09PM +0000, Steve Brokenshire wrote:
> > Hi,
> > 
> > (I've sent this to the linux-kbuild and linux-kernel lists as this
> > patch modifies the Makefile.modinst file. I also don't subscribe to the 
> > linux-kbuild and linux-kernel mailing lists so can I have any replies
> > CC'ed to me please)
> > 
> > This patch allows kernel modules to be compressed when 'make
> > modules_install' is run after being copied to
> > the /lib/module/<version>/<...> directory which is useful if you have
> > module-init-tools installed with --enable-zlib. This patch adds an
> > option (MODULE_COMPRESS) to the kernel configuration file (specifically
> > init/Kconfig) so that the kernel modules will compressed if
> > MODULE_COMPRESS is set.
> > 
> > When MODULE_COMPRESS is set the output of 'make modules_install' will
> > go as the following:
> > 
> > INSTALL  drivers/fs/xfs/xfs.ko
> > COMPRESS drivers/fs/xfs/xfs.ko
> > INSTALL  drivers/fs/fat/fat.ko
> > COMPRESS drivers/fs/fat/fat.ko
> > ...
> > 
> > I've tested my patch on kernel versions 2.6.24.2, 2.6.24-git12, 2.6.23.14 and
> > 2.6.19 and they compile, install and compress into the respective
> > module directories without any errors.
> > 
> > I've also tested this with the uvcvideo (linux-uvc) kernel module (from
> > the SVN branch and with kernel versions 2.6.24.2, 2.6.24-git12, 2.6.23.14 and
> > 2.6.19) as that uses Kbuild properly when installing the module and
> > after installing the uvcvideo the module is then compressed.
> > Unfortunately, I couldn't find any other kernel modules which used the
> > Kbuild system for installing their kernel modules. :(
> > 
> > I've included include/config/auto.conf in Makefile.modinst so that it
> > can check if MODULE_COMPRESS is set when installing the kernel modules.
> > 
> > Unfortunately when I ran mkinitrd (CentOS version) to create the initrd
> > image to go with the kernel, I get errors saying that certain kernel
> > modules don't exist despite the fact they do actually exist. When I
> > pass --allow-missing to mkinitrd, it seems to go fine but when booting
> > up with the system with the new initrd image I get error messages
> > saying that the modules don't exist.
> > 
> > A workaround is to compile the modules, don't have MODULE_COMPRESS set 
> > in .config, install the modules, run mkinitrd and copy it to /boot, set 
> > MODULE_COMPRESS in .config and then install the modules again but
> > compressed.
> > 
> > That's about it really. The only showstopper I feel is mkinitrd not
> > working properly with the compressed kernel modules.
> 
> Have you tried keeping the module names intact (.ko, not .ko.gz) ?
> It's what I was doing with modutils in 2.4 and what I'm still doing
> with module-init-tools in 2.6. While I don't particularly use mkinitrd,
> I think that keeping the name intact is preferable and should help.

How would you see if, and if yes with what program, a module was 
compressed if the name is kept intact?

I'd assume stuff like mkinitrd simply also needs to gain knowledge about 
compressed modules.

> Regards,
> Willy

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: [PATCH] Compress kernel modules on installation.

[Willy Tarreau]

On Tue, Feb 26, 2008 at 11:14:55AM +0200, Adrian Bunk wrote:
> On Mon, Feb 25, 2008 at 11:21:38PM +0100, Willy Tarreau wrote:
> > On Mon, Feb 25, 2008 at 09:42:09PM +0000, Steve Brokenshire wrote:
> > > Hi,
> > > 
> > > (I've sent this to the linux-kbuild and linux-kernel lists as this
> > > patch modifies the Makefile.modinst file. I also don't subscribe to the 
> > > linux-kbuild and linux-kernel mailing lists so can I have any replies
> > > CC'ed to me please)
> > > 
> > > This patch allows kernel modules to be compressed when 'make
> > > modules_install' is run after being copied to
> > > the /lib/module/<version>/<...> directory which is useful if you have
> > > module-init-tools installed with --enable-zlib. This patch adds an
> > > option (MODULE_COMPRESS) to the kernel configuration file (specifically
> > > init/Kconfig) so that the kernel modules will compressed if
> > > MODULE_COMPRESS is set.
> > > 
> > > When MODULE_COMPRESS is set the output of 'make modules_install' will
> > > go as the following:
> > > 
> > > INSTALL  drivers/fs/xfs/xfs.ko
> > > COMPRESS drivers/fs/xfs/xfs.ko
> > > INSTALL  drivers/fs/fat/fat.ko
> > > COMPRESS drivers/fs/fat/fat.ko
> > > ...
> > > 
> > > I've tested my patch on kernel versions 2.6.24.2, 2.6.24-git12, 2.6.23.14 and
> > > 2.6.19 and they compile, install and compress into the respective
> > > module directories without any errors.
> > > 
> > > I've also tested this with the uvcvideo (linux-uvc) kernel module (from
> > > the SVN branch and with kernel versions 2.6.24.2, 2.6.24-git12, 2.6.23.14 and
> > > 2.6.19) as that uses Kbuild properly when installing the module and
> > > after installing the uvcvideo the module is then compressed.
> > > Unfortunately, I couldn't find any other kernel modules which used the
> > > Kbuild system for installing their kernel modules. :(
> > > 
> > > I've included include/config/auto.conf in Makefile.modinst so that it
> > > can check if MODULE_COMPRESS is set when installing the kernel modules.
> > > 
> > > Unfortunately when I ran mkinitrd (CentOS version) to create the initrd
> > > image to go with the kernel, I get errors saying that certain kernel
> > > modules don't exist despite the fact they do actually exist. When I
> > > pass --allow-missing to mkinitrd, it seems to go fine but when booting
> > > up with the system with the new initrd image I get error messages
> > > saying that the modules don't exist.
> > > 
> > > A workaround is to compile the modules, don't have MODULE_COMPRESS set 
> > > in .config, install the modules, run mkinitrd and copy it to /boot, set 
> > > MODULE_COMPRESS in .config and then install the modules again but
> > > compressed.
> > > 
> > > That's about it really. The only showstopper I feel is mkinitrd not
> > > working properly with the compressed kernel modules.
> > 
> > Have you tried keeping the module names intact (.ko, not .ko.gz) ?
> > It's what I was doing with modutils in 2.4 and what I'm still doing
> > with module-init-tools in 2.6. While I don't particularly use mkinitrd,
> > I think that keeping the name intact is preferable and should help.
> 
> How would you see if, and if yes with what program, a module was 
> compressed if the name is kept intact?

depmod/modinfo/insmod/modprobe already know it. And quite honnestly,
I don't know about any other program which really needs to process
those files once installed. Well, maybe ksymoops, but I'd have to
check, as I don't recall having ever been annoyed with this.

> I'd assume stuff like mkinitrd simply also needs to gain knowledge about 
> compressed modules.

I'm not that sure. At most it might want to uncompress them to gain some
space, but someone who uses compressed modules for later use in an initrd
is probably missing something.

Regards,
Willy

Re: [PATCH] Compress kernel modules on installation.

[Adrian Bunk]

On Tue, Feb 26, 2008 at 11:22:03AM +0100, Willy Tarreau wrote:
> On Tue, Feb 26, 2008 at 11:14:55AM +0200, Adrian Bunk wrote:
> > On Mon, Feb 25, 2008 at 11:21:38PM +0100, Willy Tarreau wrote:
>...
> > > Have you tried keeping the module names intact (.ko, not .ko.gz) ?
> > > It's what I was doing with modutils in 2.4 and what I'm still doing
> > > with module-init-tools in 2.6. While I don't particularly use mkinitrd,
> > > I think that keeping the name intact is preferable and should help.
> > 
> > How would you see if, and if yes with what program, a module was 
> > compressed if the name is kept intact?
> 
> depmod/modinfo/insmod/modprobe already know it. And quite honnestly,
> I don't know about any other program which really needs to process
> those files once installed. Well, maybe ksymoops, but I'd have to
> check, as I don't recall having ever been annoyed with this.
>...

depmod/modinfo/insmod/modprobe know only if you compile 
module-init-tools with zlib support.

And what about the busybox versions?

A different name would e.g.:
- easily allow proper error handling if the userspace modules program 
  doesn't support the compression used
- better scale to support additional compressions
- give the user a hint what is happening and what might be the problem
  when anythig goes wrong

> Regards,
> Willy

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: [PATCH] Compress kernel modules on installation.

[Willy Tarreau]

On Tue, Feb 26, 2008 at 02:30:16PM +0200, Adrian Bunk wrote:
> On Tue, Feb 26, 2008 at 11:22:03AM +0100, Willy Tarreau wrote:
> > On Tue, Feb 26, 2008 at 11:14:55AM +0200, Adrian Bunk wrote:
> > > On Mon, Feb 25, 2008 at 11:21:38PM +0100, Willy Tarreau wrote:
> >...
> > > > Have you tried keeping the module names intact (.ko, not .ko.gz) ?
> > > > It's what I was doing with modutils in 2.4 and what I'm still doing
> > > > with module-init-tools in 2.6. While I don't particularly use mkinitrd,
> > > > I think that keeping the name intact is preferable and should help.
> > > 
> > > How would you see if, and if yes with what program, a module was 
> > > compressed if the name is kept intact?
> > 
> > depmod/modinfo/insmod/modprobe already know it. And quite honnestly,
> > I don't know about any other program which really needs to process
> > those files once installed. Well, maybe ksymoops, but I'd have to
> > check, as I don't recall having ever been annoyed with this.
> >...
> 
> depmod/modinfo/insmod/modprobe know only if you compile 
> module-init-tools with zlib support.
> 
> And what about the busybox versions?
> 
> A different name would e.g.:
> - easily allow proper error handling if the userspace modules program 
>   doesn't support the compression used
> - better scale to support additional compressions
> - give the user a hint what is happening and what might be the problem
>   when anythig goes wrong

I agree, but right now module.dep references existing files with their
real names. Maybe something should define exactly what it should contain
(eg: module_name.ko even if .ko.gz is used) so that all tools relying on
it do not stop after noticing that the file referenced there does not
exist.

regards,
Willy

Re: [PATCH] Compress kernel modules on installation.

[Sam Ravnborg]

On Mon, Feb 25, 2008 at 09:42:09PM +0000, Steve Brokenshire wrote:
> Hi,
> 
> (I've sent this to the linux-kbuild and linux-kernel lists as this
> patch modifies the Makefile.modinst file. I also don't subscribe to the 
> linux-kbuild and linux-kernel mailing lists so can I have any replies
> CC'ed to me please)
> 
> This patch allows kernel modules to be compressed when 'make
> modules_install' is run after being copied to
> the /lib/module/<version>/<...> directory which is useful if you have
> module-init-tools installed with --enable-zlib. This patch adds an
> option (MODULE_COMPRESS) to the kernel configuration file (specifically
> init/Kconfig) so that the kernel modules will compressed if
> MODULE_COMPRESS is set.
> 
> When MODULE_COMPRESS is set the output of 'make modules_install' will
> go as the following:
> 
> INSTALL  drivers/fs/xfs/xfs.ko
> COMPRESS drivers/fs/xfs/xfs.ko
> INSTALL  drivers/fs/fat/fat.ko
> COMPRESS drivers/fs/fat/fat.ko
> ...
> 
> I've tested my patch on kernel versions 2.6.24.2, 2.6.24-git12, 2.6.23.14 and
> 2.6.19 and they compile, install and compress into the respective
> module directories without any errors.
> 
> I've also tested this with the uvcvideo (linux-uvc) kernel module (from
> the SVN branch and with kernel versions 2.6.24.2, 2.6.24-git12, 2.6.23.14 and
> 2.6.19) as that uses Kbuild properly when installing the module and
> after installing the uvcvideo the module is then compressed.
> Unfortunately, I couldn't find any other kernel modules which used the
> Kbuild system for installing their kernel modules. :(
> 
> I've included include/config/auto.conf in Makefile.modinst so that it
> can check if MODULE_COMPRESS is set when installing the kernel modules.

Should we mention this possibility in Documentation/kbuild/modules.txt also?

Otherwise from a quick scan it looks OK.
As module-init-tols are our userspae counterpart we should align the
functionality with that tool.
I assume this gives us a simple choice of compression method and naming.

	Sam

Re: [PATCH] Compress kernel modules on installation.

[Steve Brokenshire]

On Tue, 26 Feb 2008 12:28:40 +0100
Sam Ravnborg <sam@ravnborg.org> wrote:

> On Mon, Feb 25, 2008 at 09:42:09PM +0000, Steve Brokenshire wrote:
> > Hi,
> > 
> > (I've sent this to the linux-kbuild and linux-kernel lists as this
> > patch modifies the Makefile.modinst file. I also don't subscribe to the 
> > linux-kbuild and linux-kernel mailing lists so can I have any replies
> > CC'ed to me please)
> > 
> > This patch allows kernel modules to be compressed when 'make
> > modules_install' is run after being copied to
> > the /lib/module/<version>/<...> directory which is useful if you have
> > module-init-tools installed with --enable-zlib. This patch adds an
> > option (MODULE_COMPRESS) to the kernel configuration file (specifically
> > init/Kconfig) so that the kernel modules will compressed if
> > MODULE_COMPRESS is set.
> > 
> > When MODULE_COMPRESS is set the output of 'make modules_install' will
> > go as the following:
> > 
> > INSTALL  drivers/fs/xfs/xfs.ko
> > COMPRESS drivers/fs/xfs/xfs.ko
> > INSTALL  drivers/fs/fat/fat.ko
> > COMPRESS drivers/fs/fat/fat.ko
> > ...
> > 
> > I've tested my patch on kernel versions 2.6.24.2, 2.6.24-git12, 2.6.23.14 and
> > 2.6.19 and they compile, install and compress into the respective
> > module directories without any errors.
> > 
> > I've also tested this with the uvcvideo (linux-uvc) kernel module (from
> > the SVN branch and with kernel versions 2.6.24.2, 2.6.24-git12, 2.6.23.14 and
> > 2.6.19) as that uses Kbuild properly when installing the module and
> > after installing the uvcvideo the module is then compressed.
> > Unfortunately, I couldn't find any other kernel modules which used the
> > Kbuild system for installing their kernel modules. :(
> > 
> > I've included include/config/auto.conf in Makefile.modinst so that it
> > can check if MODULE_COMPRESS is set when installing the kernel modules.
> 
> Should we mention this possibility in Documentation/kbuild/modules.txt also?
> 
> Otherwise from a quick scan it looks OK.
> As module-init-tols are our userspae counterpart we should align the
> functionality with that tool.
> I assume this gives us a simple choice of compression method and naming.
> 
> 	Sam

(Sorry for my lazy replying...)

I think it should be mentioned in Documentation/kbuild/modules.txt at
Chapter 2.2. The following should be added to the modules_install
target part:

If MODULES_COMPRESS is set when the modules_install target is run then
the module is compressed after it has been copied to 
/lib/modules/<kernel-version>. Compressed modules require 
module-init-tools installed with --zlib-enabled.

Does that sound right?

Anyway, as it currently is with the patch, MODULE_COMPRESS isn't as scalable as it perhaps could be and I've been thinking I should try and make a better version of the patch which is scalable (So depending on how module-init-tools is updated in the future each option would also give which version of module-init-tools is required to use that compression method in the init/KConfig file)

Steve

Re: [PATCH] Compress kernel modules on installation.

[Jan Engelhardt]

On Tuesday 2008-02-26 12:28, Sam Ravnborg wrote:
>> 
>> (I've sent this to the linux-kbuild and linux-kernel lists as this
>> patch modifies the Makefile.modinst file. I also don't subscribe to the 
>> linux-kbuild and linux-kernel mailing lists so can I have any replies
>> CC'ed to me please)
>> 
>> This patch allows kernel modules to be compressed when 'make
>> modules_install' is run after being copied to
>> the /lib/module/<version>/<...> directory which is useful if you have
>> module-init-tools installed with --enable-zlib. This patch adds an
>> option (MODULE_COMPRESS) to the kernel configuration file (specifically
>> init/Kconfig) so that the kernel modules will compressed if
>> MODULE_COMPRESS is set.

I recently started compressing my kernel modules and that saved me
at least 70 MB of disk space on mostlyallmodconfig.
(And no, the argument of disks being cheap is not so true with
CF or SSD.)
Distro is lazy and wants to wait for upstream to have it,
so is there any chance of getting this proposal in?


Jan

Re: [PATCH] Compress kernel modules on installation.

[Sam Ravnborg]

On Fri, Dec 26, 2008 at 08:30:19PM +0100, Jan Engelhardt wrote:
> 
> On Tuesday 2008-02-26 12:28, Sam Ravnborg wrote:
> >> 
> >> (I've sent this to the linux-kbuild and linux-kernel lists as this
> >> patch modifies the Makefile.modinst file. I also don't subscribe to the 
> >> linux-kbuild and linux-kernel mailing lists so can I have any replies
> >> CC'ed to me please)
> >> 
> >> This patch allows kernel modules to be compressed when 'make
> >> modules_install' is run after being copied to
> >> the /lib/module/<version>/<...> directory which is useful if you have
> >> module-init-tools installed with --enable-zlib. This patch adds an
> >> option (MODULE_COMPRESS) to the kernel configuration file (specifically
> >> init/Kconfig) so that the kernel modules will compressed if
> >> MODULE_COMPRESS is set.
> 
> I recently started compressing my kernel modules and that saved me
> at least 70 MB of disk space on mostlyallmodconfig.
> (And no, the argument of disks being cheap is not so true with
> CF or SSD.)
> Distro is lazy and wants to wait for upstream to have it,
> so is there any chance of getting this proposal in?

Steve said he wanted to try to make the solution more
scalable so I am awaiting a new patch.

	Sam

Re: [PATCH] Compress kernel modules on installation.

[Jan Engelhardt]

On Friday 2008-12-26 20:48, Sam Ravnborg wrote:
>> >> 
>> >> This patch allows kernel modules to be compressed when 'make
>> >> modules_install' is run after being copied to
>> >> the /lib/module/<version>/<...> directory which is useful if you have
>> >> module-init-tools installed with --enable-zlib. This patch adds an
>> >> option (MODULE_COMPRESS) to the kernel configuration file (specifically
>> >> init/Kconfig) so that the kernel modules will compressed if
>> >> MODULE_COMPRESS is set.
>> 
>> I recently started compressing my kernel modules and that saved me
>> at least 70 MB of disk space on mostlyallmodconfig.
>> (And no, the argument of disks being cheap is not so true with
>> CF or SSD.)
>> Distro is lazy and wants to wait for upstream to have it,
>> so is there any chance of getting this proposal in?
>
>Steve said he wanted to try to make the solution more
>scalable so I am awaiting a new patch.

Hm, all I needed was this patch. It might fire up some people,
but it's got all the scalability I could think of..


commit b4a3e1c610c99d4e8b543b97fd722076c6f7c5dd
Author: Jan Engelhardt <jengelh@medozas.de>
Date:   Wed Dec 10 20:39:21 2008 +0100

build: install modules compressed
---
 scripts/Makefile.modinst |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/scripts/Makefile.modinst b/scripts/Makefile.modinst
index efa5d94..c3421a1 100644
--- a/scripts/Makefile.modinst
+++ b/scripts/Makefile.modinst
@@ -17,7 +17,7 @@ __modinst: $(modules)
 	@:
 
 quiet_cmd_modules_install = INSTALL $@
-      cmd_modules_install = mkdir -p $(2); cp $@ $(2) ; $(mod_strip_cmd) $(2)/$(notdir $@)
+      cmd_modules_install = mkdir -p $(2); cp $@ $(2) ; $(mod_strip_cmd) $(2)/$(notdir $@); gzip -9f $(2)/$(notdir $@)
 
 # Modules built outside the kernel source tree go into extra by default
 INSTALL_MOD_DIR ?= extra

Re: [PATCH] Compress kernel modules on installation.

[Sam Ravnborg]

On Fri, Dec 26, 2008 at 08:50:34PM +0100, Jan Engelhardt wrote:
> 
> On Friday 2008-12-26 20:48, Sam Ravnborg wrote:
> >> >> 
> >> >> This patch allows kernel modules to be compressed when 'make
> >> >> modules_install' is run after being copied to
> >> >> the /lib/module/<version>/<...> directory which is useful if you have
> >> >> module-init-tools installed with --enable-zlib. This patch adds an
> >> >> option (MODULE_COMPRESS) to the kernel configuration file (specifically
> >> >> init/Kconfig) so that the kernel modules will compressed if
> >> >> MODULE_COMPRESS is set.
> >> 
> >> I recently started compressing my kernel modules and that saved me
> >> at least 70 MB of disk space on mostlyallmodconfig.
> >> (And no, the argument of disks being cheap is not so true with
> >> CF or SSD.)
> >> Distro is lazy and wants to wait for upstream to have it,
> >> so is there any chance of getting this proposal in?
> >
> >Steve said he wanted to try to make the solution more
> >scalable so I am awaiting a new patch.
> 
> Hm, all I needed was this patch. It might fire up some people,
> but it's got all the scalability I could think of..

Jan - there is obviously no way I could apply this patch
so late in the cycle. 
The original patch that made this a CONFIG option is
then much better as we avoid forcing new and untested
behaviour on the users.

We all know that compressing the modules are simple.
And unless someone comes up with *very* good arguments
then we should just use gzip with default parameters.

If we go for the "keep the .ko extension but compress"
then someone needs to answer the obvious questions:
- will this break on a typical distribution
- will this break busybox users

	Sam

Re: [PATCH] Compress kernel modules on installation.

[Jan Engelhardt]

On Friday 2008-12-26 23:57, Sam Ravnborg wrote:
>>>
>>>Steve said [in Feb 2008] he wanted to try to make the solution
>>>more scalable so I am awaiting a new patch.
>> 
>>Hm, all I needed was this patch. It might fire up some people,
>>but it's got all the scalability I could think of..
>
>Jan - there is obviously no way I could apply this patch
>so late in the cycle. 

2.6.29 just started, did not it. Even if not, just queue it for
the next.

>The original patch that made this a CONFIG option is
>then much better as we avoid forcing new and untested
>behaviour on the users.
>
>We all know that compressing the modules are simple.
>And unless someone comes up with *very* good arguments
>then we should just use gzip with default parameters.

Besides the -9 flag, where would there be nondefault
parameters?

>If we go for the "keep the .ko extension but compress"
>then someone needs to answer the obvious questions:
>
>- will this break on a typical distribution

No; module-init-tools already uses gzopen even on
uncompressed files.

>- will this break busybox users

I did not see any gzip support in there, so the
answer is likely "yes, as usual".

Re: [PATCH] Compress kernel modules on installation.

[Sam Ravnborg]

On Fri, Dec 26, 2008 at 08:50:34PM +0100, Jan Engelhardt wrote:
> 
> On Friday 2008-12-26 20:48, Sam Ravnborg wrote:
> >> >> 
> >> >> This patch allows kernel modules to be compressed when 'make
> >> >> modules_install' is run after being copied to
> >> >> the /lib/module/<version>/<...> directory which is useful if you have
> >> >> module-init-tools installed with --enable-zlib. This patch adds an
> >> >> option (MODULE_COMPRESS) to the kernel configuration file (specifically
> >> >> init/Kconfig) so that the kernel modules will compressed if
> >> >> MODULE_COMPRESS is set.
> >> 
> >> I recently started compressing my kernel modules and that saved me
> >> at least 70 MB of disk space on mostlyallmodconfig.
> >> (And no, the argument of disks being cheap is not so true with
> >> CF or SSD.)
> >> Distro is lazy and wants to wait for upstream to have it,
> >> so is there any chance of getting this proposal in?
> >
> >Steve said he wanted to try to make the solution more
> >scalable so I am awaiting a new patch.
> 
> Hm, all I needed was this patch. It might fire up some people,
> but it's got all the scalability I could think of..

Can I ask you to redo this patch.
I want a way to swith off compression, but it should default be enable.
So if KBUILD_MODULE_NOCOMPRESS is undefined we will compress the modules
when we install them.

Something like:

      cmd_modules_install =                                               \
         mkdir -p $(2); cp $@ $(2) ;                                      \
         $(mod_strip_cmd) $(2)/$(notdir $@)                               \
         $(if $(KBUILD_MODULES_NOCOMPRESS),, ;gzip -9f $(2)/$(notdir $@))

And add documentation to Documentation/kbuild/kbuild.txt too...

Thanks,
	Sam

Re: [PATCH] Compress kernel modules on installation.

[Arjan van de Ven]

On Sun, 18 Jan 2009 22:04:16 +0100
Sam Ravnborg <sam@ravnborg.org> wrote:

> On Fri, Dec 26, 2008 at 08:50:34PM +0100, Jan Engelhardt wrote:
> > 
> > On Friday 2008-12-26 20:48, Sam Ravnborg wrote:
> > >> >> 
> > >> >> This patch allows kernel modules to be compressed when 'make
> > >> >> modules_install' is run after being copied to
> > >> >> the /lib/module/<version>/<...> directory which is useful if
> > >> >> you have module-init-tools installed with --enable-zlib. This
> > >> >> patch adds an option (MODULE_COMPRESS) to the kernel
> > >> >> configuration file (specifically init/Kconfig) so that the
> > >> >> kernel modules will compressed if MODULE_COMPRESS is set.
> > >> 
> > >> I recently started compressing my kernel modules and that saved
> > >> me at least 70 MB of disk space on mostlyallmodconfig.
> > >> (And no, the argument of disks being cheap is not so true with
> > >> CF or SSD.)
> > >> Distro is lazy and wants to wait for upstream to have it,
> > >> so is there any chance of getting this proposal in?
> > >
> > >Steve said he wanted to try to make the solution more
> > >scalable so I am awaiting a new patch.
> > 
> > Hm, all I needed was this patch. It might fire up some people,
> > but it's got all the scalability I could think of..
>  
> Can I ask you to redo this patch.
> I want a way to swith off compression, but it should default be
> enable. So if KBUILD_MODULE_NOCOMPRESS is undefined we will compress
> the modules when we install them.
> 

especially since the decompression is an option for the module tools,
it should be an option for the kernel... 
since it does add a dependency for the module tools (and one that you
might not need; for example I disabled it just to remove a build
requires)


-- 
Arjan van de Ven 	Intel Open Source Technology Centre
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

Re: [PATCH] Compress kernel modules on installation.

[Michael Tokarev]

Arjan van de Ven wrote:
> On Sun, 18 Jan 2009 22:04:16 +0100
> Sam Ravnborg <sam@ravnborg.org> wrote:
[]
>> Can I ask you to redo this patch.
>> I want a way to swith off compression, but it should default be
>> enable. So if KBUILD_MODULE_NOCOMPRESS is undefined we will compress
>> the modules when we install them.
> 
> especially since the decompression is an option for the module tools,
> it should be an option for the kernel... 
> since it does add a dependency for the module tools (and one that you
> might not need; for example I disabled it just to remove a build
> requires)

By the way, I for one use busybox for initramfs stuff, since a long
time ago.  And it (still?) does not support decompression of kernel
modules, as of the latest (1.13) version...  I don't think I'm alone
using busybox this way, and it looks like decompression support
should be added to it first, and next to the kernel.. ;)

/mjt

Re: [PATCH] Compress kernel modules on installation.

[Jan Engelhardt]

On Sunday 2009-01-18 22:15, Michael Tokarev wrote:
>Arjan van de Ven wrote:
>> On Sun, 18 Jan 2009 22:04:16 +0100
>> Sam Ravnborg <sam@ravnborg.org> wrote:
>[]
>>> Can I ask you to redo this patch.
>>> I want a way to swith off compression, but it should default be
>>> enable. So if KBUILD_MODULE_NOCOMPRESS is undefined we will compress
>>> the modules when we install them.
>> 
>> especially since the decompression is an option for the module tools,
>> it should be an option for the kernel... 
>> since it does add a dependency for the module tools (and one that you
>> might not need; for example I disabled it just to remove a build
>> requires)
>
>By the way, I for one use busybox for initramfs stuff, since a long
>time ago.  And it (still?) does not support decompression of kernel
>modules, as of the latest (1.13) version...  I don't think I'm alone
>using busybox this way, and it looks like decompression support
>should be added to it first, and next to the kernel.. ;)

If we went by that metric, the kernel would still be in its infancy.

Re: [PATCH] Compress kernel modules on installation.

[Michael Tokarev]

Jan Engelhardt wrote:
>> []it looks like decompression support
>> should be added to it first, and next to the kernel.. ;)
> 
> If we went by that metric, the kernel would still be in its infancy.

Hey, note the smile at the end of my statement! :)

/mjt

Re: [PATCH] Compress kernel modules on installation.

[Jan Engelhardt]

On Sunday 2009-01-18 22:04, Sam Ravnborg wrote:
>
>Can I ask you to redo this patch.
>I want a way to swith off compression, but it should default be enable.
>So if KBUILD_MODULE_NOCOMPRESS is undefined we will compress the modules
>when we install them.
>
>And add documentation to Documentation/kbuild/kbuild.txt too...

changes to previous:
- added choice for compress (none, -1, -2, ..., -9)
- documentation

parent ad2507ab58aa5a3e50b0482889882ff3adf6ef33 (v2.6.29-rc2-24-gad2507a)
commit 3fc20a79496c3c5c8d4b906ea7e2496de448002d
Author: Jan Engelhardt <jengelh@medozas.de>
Date:   Mon Jan 19 21:18:39 2009 +0100

kbuild: install modules gzipped

This saves lots of disk space. Yes we had discussions about gzip -6
having a better compression-per-time ratio, but the point is that
people doing the compression in the first place often do not care
about the amount of compression time as long as the final filesize
will be smaller (that's what LZMA does over bzip2, and what bzip2
does over gzip). The KBUILD_MODULES_COMPRESS make variable can be set
over the command line to tune the compression level to your need,
including disabling it entirely.

Signed-off-by: Jan Engelhardt <jengelh@medozas.de>
---
 Documentation/kbuild/kbuild.txt |   11 +++++++++++
 scripts/Makefile.modinst        |    6 +++++-
 2 files changed, 16 insertions(+), 1 deletions(-)

diff --git a/Documentation/kbuild/kbuild.txt b/Documentation/kbuild/kbuild.txt
index 923f9dd..453ba6c 100644
--- a/Documentation/kbuild/kbuild.txt
+++ b/Documentation/kbuild/kbuild.txt
@@ -125,6 +125,17 @@ KBUILD_EXTRA_SYMBOLS
 For modules use symbols from another modules.
 See more details in modules.txt.
 
+KBUILD_MODULES_COMPRESS
+--------------------------------------------------
+By default, kbuild will compress modules with gzip -9 compression to
+reduce disk space. The KBUILD_MODULES_COMPRESS make parameter can be
+used to set an alternate compression level or disable compression
+entirely.
+Example for using the standard gzip compression level:
+	make KBUILD_MODULES_COMPRESS="-6"
+Disabling compression:
+	make KBUILD_MODULES_COMPRESS=""
+
 ALLSOURCE_ARCHS
 --------------------------------------------------
 For tags/TAGS/cscope targets, you can specify more than one archs
diff --git a/scripts/Makefile.modinst b/scripts/Makefile.modinst
index 18b4bf8..e0708b1 100644
--- a/scripts/Makefile.modinst
+++ b/scripts/Makefile.modinst
@@ -16,9 +16,13 @@ PHONY += $(modules)
 __modinst: $(modules)
 	@:
 
+KBUILD_MODULES_COMPRESS  ?= -9
 quiet_cmd_modules_install = INSTALL $@
       cmd_modules_install = mkdir -p $(2); \
-			    $(mod_strip_cmd) $@ $(2)/$(notdir $@) || cp $@ $(2)
+			    $(mod_strip_cmd) $@ $(2)/$(notdir $@) || cp $@ $(2); \
+			    $(if ${KBUILD_MODULES_COMPRESS}, \
+			    	gzip -f ${KBUILD_MODULES_COMPRESS} \
+			    	$(2)/$(notdir $@), )
 
 # Modules built outside the kernel source tree go into extra by default
 INSTALL_MOD_DIR ?= extra
-- 
# Created with git-export-patch

Re: [PATCH] Compress kernel modules on installation.

[Steve Brokenshire]

On Fri, 26 Dec 2008 20:48:33 +0100
Sam Ravnborg <sam@ravnborg.org> wrote:

> On Fri, Dec 26, 2008 at 08:30:19PM +0100, Jan Engelhardt wrote:
> > 
> > On Tuesday 2008-02-26 12:28, Sam Ravnborg wrote:
> > >> 
> > >> (I've sent this to the linux-kbuild and linux-kernel lists as this
> > >> patch modifies the Makefile.modinst file. I also don't subscribe to the 
> > >> linux-kbuild and linux-kernel mailing lists so can I have any replies
> > >> CC'ed to me please)
> > >> 
> > >> This patch allows kernel modules to be compressed when 'make
> > >> modules_install' is run after being copied to
> > >> the /lib/module/<version>/<...> directory which is useful if you have
> > >> module-init-tools installed with --enable-zlib. This patch adds an
> > >> option (MODULE_COMPRESS) to the kernel configuration file (specifically
> > >> init/Kconfig) so that the kernel modules will compressed if
> > >> MODULE_COMPRESS is set.
> > 
> > I recently started compressing my kernel modules and that saved me
> > at least 70 MB of disk space on mostlyallmodconfig.
> > (And no, the argument of disks being cheap is not so true with
> > CF or SSD.)
> > Distro is lazy and wants to wait for upstream to have it,
> > so is there any chance of getting this proposal in?
> 
> Steve said he wanted to try to make the solution more
> scalable so I am awaiting a new patch.
> 
> 	Sam
> 

Yikes! I do apologise. I have been a little bit busy lately and that my
old PC decided to fail on the 24th December so I've been using that as
an excuse for panic buying, building and installing a 64-bit system but
I do have an updated patch available that is scalable (in that... if
new compression formats are used in the future they can be easily added
by adding text to init/Kconfig and code to Makefile.modinst).

I have noticed the patch from Jan for specifying settings for gzip. I
don't see why something like this can't be added to init/Kconfig...

config MODULE_COMPRESS_OPTIONS
	string "Compression method command line options"
	depends on MODULES
	help
	  This option specifies the command line options to be used for
	  the selected compression method.

	  Please refer to the selected compression method's documentation 
	  on which options should be used.

	  If unsure, leave this option blank.

And perhaps alter for Makefile.modinst (after applying the patch below)...

quiet_cmd_modules_compress_gzip = COMPRESS $@
      cmd_modules_compress_gzip = gzip $(CONFIG_MODULE_COMPRESS_OPTIONS) \
				 -c $(2)/`basename $@` \ 
				 > $(2)/`basename $@`.gz; \
				rm $(2)/`basename $@`

I'll supply the patch for what's been done so far (that and it's gone
3am here...) so the above makes a bit more sense. I should point out
that in the patch there should be an additional blank line after
endchoice for init/Kconfig but adding that blank line with the plus
sign seems to cause the patching process to fail so I had to take it
out and that this patch works with 2.6.27.4 (which is the kernel I am
currently using) with a slight fuzz and I haven't chance to test it
with the latest kernel.

I'll have a go at incorporating what I've said regarding
MODULE_COMPRESS_OPTIONS into another patch using the latest kernel (and
add the standard 'If unsure' message to MODULE_COMPRESS as well).

Steve

Patch Follows...

================================

Signed-off-by: Steve Brokenshire <sbrokenshire@xestia.co.uk>

--- a/scripts/Makefile.modinst	2008-02-11 05:51:11.000000000 +0000
+++ b/scripts/Makefile.modinst	2008-05-29 19:30:51.000000000 +0100
@@ -5,6 +5,7 @@
 PHONY := __modinst
 __modinst:
 
+include include/config/auto.conf
 include scripts/Kbuild.include
 
 #
@@ -16,8 +17,15 @@
 __modinst: $(modules)
 	@:
 
-quiet_cmd_modules_install = INSTALL $@
-      cmd_modules_install = mkdir -p $(2); cp $@ $(2) ; $(mod_strip_cmd) $(2)/$(notdir $@)
+quiet_cmd_modules_install = INSTALL  $@
+      cmd_modules_install = mkdir -p $(2); \
+				cp $@ $(2) ; \
+				$(mod_strip_cmd) $(2)/$(notdir $@)
+
+quiet_cmd_modules_compress_gzip = COMPRESS $@
+      cmd_modules_compress_gzip = gzip --best -c $(2)/`basename $@` \
+				 > $(2)/`basename $@`.gz; \
+				rm $(2)/`basename $@`
 
 # Modules built outside the kernel source tree go into extra by default
 INSTALL_MOD_DIR ?= extra
@@ -26,8 +34,11 @@
 modinst_dir = $(if $(KBUILD_EXTMOD),$(ext-mod-dir),kernel/$(@D))
 
 $(modules):
+
 	$(call cmd,modules_install,$(MODLIB)/$(modinst_dir))
 
+	$(if $(CONFIG_MODULE_COMPRESS_GZIP), \
+		$(call cmd,modules_compress_gzip,$(MODLIB)/$(modinst_dir)))
 
 # Declare the contents of the .PHONY variable as phony.  We keep that
 # information in a variable se we can use it in if_changed and friends.
--- a/init/Kconfig	2008-02-11 05:51:11.000000000 +0000
+++ b/init/Kconfig	2008-05-29 19:05:34.000000000 +0100
@@ -716,6 +716,45 @@
 	  rmmod).  This is mainly for kernel developers and desperate users.
 	  If unsure, say N.
 
+config MODULE_COMPRESS
+	bool "Compress kernel modules on installation"
+	depends on MODULES
+	help
+	  This option compresses the kernel modules when 'make
+	  modules_install' is run.
+
+	  The modules will be compressed into the selected compression 
+	  format with Gzip being the default compression format.
+
+	  When a kernel module is installed from outside of the main kernel
+	  source and uses the Kbuild system for installing modules then that
+	  kernel module will also be compressed when it is installed.
+
+	  When running mkinitrd you will find that an error message
+	  appears saying that it cannot find a certain kernel module.
+	  As a workaround, unset CONFIG_MODULE_COMPRESS, build the modules
+	  and install them, run mkinitrd and create the initrd image, place
+	  the initrd image in the correct place for booting, set
+	  CONFIG_MODULE_COMPRESS and then install the modules again.
+
+	  This options requires the module-init-tools package to be 
+	  configured with --enable-zlib (if using gzip which is the
+	  default compression method).
+
+choice
+	depends on MODULE_COMPRESS
+	prompt "Kernel module compression method"
+	default MODULE_COMPRESS_GZIP
+
+config MODULE_COMPRESS_GZIP
+	bool "Gzip compression"
+	help
+	  Compresses the kernel modules using the Gzip (GNU zip) 
+	  compression format.
+
+	  This option requires gzip to be installed.
+	  
+endchoice
 config MODVERSIONS
 	bool "Module versioning support"
 	depends on MODULES
--- a/Documentation/kbuild/modules.txt	2008-02-11 05:51:11.000000000 +0000
+++ b/Documentation/kbuild/modules.txt	2008-05-29 19:35:30.000000000 +0100
@@ -96,6 +96,11 @@
 		Installation default is in /lib/modules/<kernel-version>/extra,
 		but may be prefixed with INSTALL_MOD_PATH - see separate
 		chapter.
+		If MODULES_COMPRESS is set when the modules_install target is 
+		run then the module is compressed after it has been 
+		copied to /lib/modules/<kernel-version>. Compressed modules 
+		using the default Gzip compression format will require 
+		module-init-tools installed with --zlib-enabled.
 
 	make -C $KDIR M=`pwd` clean
 		Remove all generated files for the module - the kernel

Re: [PATCH] Compress kernel modules on installation.

[Steve Brokenshire]

On Tue, 20 Jan 2009 03:30:47 +0000
Steve Brokenshire <sbrokenshire@xestia.co.uk> wrote:

> On Fri, 26 Dec 2008 20:48:33 +0100
> Sam Ravnborg <sam@ravnborg.org> wrote:
> 
> > On Fri, Dec 26, 2008 at 08:30:19PM +0100, Jan Engelhardt wrote:
> > > 
> > > On Tuesday 2008-02-26 12:28, Sam Ravnborg wrote:
> > > >> 
> > > >> (I've sent this to the linux-kbuild and linux-kernel lists as this
> > > >> patch modifies the Makefile.modinst file. I also don't subscribe to the 
> > > >> linux-kbuild and linux-kernel mailing lists so can I have any replies
> > > >> CC'ed to me please)
> > > >> 
> > > >> This patch allows kernel modules to be compressed when 'make
> > > >> modules_install' is run after being copied to
> > > >> the /lib/module/<version>/<...> directory which is useful if you have
> > > >> module-init-tools installed with --enable-zlib. This patch adds an
> > > >> option (MODULE_COMPRESS) to the kernel configuration file (specifically
> > > >> init/Kconfig) so that the kernel modules will compressed if
> > > >> MODULE_COMPRESS is set.
> > > 
> > > I recently started compressing my kernel modules and that saved me
> > > at least 70 MB of disk space on mostlyallmodconfig.
> > > (And no, the argument of disks being cheap is not so true with
> > > CF or SSD.)
> > > Distro is lazy and wants to wait for upstream to have it,
> > > so is there any chance of getting this proposal in?
> > 
> > Steve said he wanted to try to make the solution more
> > scalable so I am awaiting a new patch.
> > 
> > 	Sam
> > 
> 
> (Stuff I mentioned earlier has been snipped)

I've done some modifications to the previous patch and that there is
now an option to specify the options for the selected compression
format in init/Kconfig using MODULE_COMPRESS_OPTIONS and also some
minor tidying up.

It didn't go quite as planned from the previous post for altering
Makefile.modinst as gzip doesn't like "" when there is nothing in
MODULE_COMPRESS_OPTIONS or when some options are set with the quotation
marks so when MODULE_COMPRESS_OPTIONS just has "" then nothing will
happen (as there would be no value in MODCOMPOPT) and if there is
something other than "" then strip the quotation marks from the start
and the end then place the result in MODCOMPOPT which is then used by
the selected compression method.

I think I've covered everything...

Steve


Patch follows...

================================
Signed-off-by: Steve Brokenshire <sbrokenshire@xestia.co.uk>

--- a/scripts/Makefile.modinst	2009-01-18 18:45:37.000000000 +0000
+++ b/scripts/Makefile.modinst	2009-01-25 13:35:53.000000000 +0000
@@ -5,6 +5,7 @@
 PHONY := __modinst
 __modinst:
 
+include include/config/auto.conf
 include scripts/Kbuild.include
 
 #
@@ -16,8 +17,21 @@ PHONY += $(modules)
 __modinst: $(modules)
 	@:
 
-quiet_cmd_modules_install = INSTALL $@
-      cmd_modules_install = mkdir -p $(2); cp $@ $(2) ; $(mod_strip_cmd) $(2)/$(notdir $@)
+ifeq ($(CONFIG_MODULE_COMPRESS_OPTIONS), "")
+else
+ MODCOMPOPT = $(shell echo -n $(CONFIG_MODULE_COMPRESS_OPTIONS))
+endif
+
+quiet_cmd_modules_install = INSTALL  $@
+      cmd_modules_install = mkdir -p $(2); \
+				cp $@ $(2) ; \
+				$(mod_strip_cmd) $(2)/$(notdir $@)
+
+quiet_cmd_modules_compress_gzip = COMPRESS $@
+      cmd_modules_compress_gzip = gzip $(MODCOMPOPT) -c \
+				 $(2)/`basename $@` \
+				 > $(2)/`basename $@`.gz; \
+				 rm $(2)/`basename $@`
 
 # Modules built outside the kernel source tree go into extra by default
 INSTALL_MOD_DIR ?= extra
@@ -26,8 +40,11 @@ ext-mod-dir = $(INSTALL_MOD_DIR)$(subst 
 modinst_dir = $(if $(KBUILD_EXTMOD),$(ext-mod-dir),kernel/$(@D))
 
 $(modules):
+
 	$(call cmd,modules_install,$(MODLIB)/$(modinst_dir))
 
+	$(if $(CONFIG_MODULE_COMPRESS_GZIP), \
+		$(call cmd,modules_compress_gzip,$(MODLIB)/$(modinst_dir)))
 
 # Declare the contents of the .PHONY variable as phony.  We keep that
 # information in a variable se we can use it in if_changed and friends.
--- a/init/Kconfig	2009-01-18 18:45:37.000000000 +0000
+++ b/init/Kconfig	2009-01-25 15:08:39.000000000 +0000
@@ -887,6 +887,62 @@ config MODULE_FORCE_UNLOAD
 	  rmmod).  This is mainly for kernel developers and desperate users.
 	  If unsure, say N.
 
+config MODULE_COMPRESS
+	bool "Compress kernel modules on installation"
+	depends on MODULES
+	help
+	  This option compresses the kernel modules when 'make
+	  modules_install' is run.
+
+	  The modules will be compressed into the selected compression 
+	  format with gzip being the default compression format.
+
+	  When a kernel module is installed from outside of the main kernel
+	  source and uses the Kbuild system for installing modules then that
+	  kernel module will also be compressed when it is installed.
+
+	  When running mkinitrd you will find that an error message
+	  appears saying that it cannot find a certain kernel module.
+	  As a workaround, unset CONFIG_MODULE_COMPRESS, build the modules
+	  and install them, run mkinitrd and create the initrd image, place
+	  the initrd image in the correct place for booting, set
+	  CONFIG_MODULE_COMPRESS and then install the modules again.
+
+	  This option requires the module-init-tools package to be 
+	  configured with --enable-zlib (if using gzip which is the
+	  default compression format).
+
+	  If unsure, say N.
+
+config MODULE_COMPRESS_OPTIONS
+	string "Compression format command line options"
+	depends on MODULE_COMPRESS
+	help
+	  This option specifies the command line options to be used for
+	  the selected compression format.
+
+	  Please refer to the selected compression format's documentation
+	  on which options should be used.
+
+	  If unsure, leave this option blank.
+
+choice
+	prompt "Kernel module compression format"
+	depends on MODULE_COMPRESS
+	default MODULE_COMPRESS_GZIP
+
+config MODULE_COMPRESS_GZIP
+	bool "gzip compression"
+	help
+	  Compresses the kernel modules using the gzip (GNU zip) 
+	  compression format.
+
+	  This option requires gzip to be installed.
+
+	  If unsure, leave this option selected.
+	  
+endchoice
+
 config MODVERSIONS
 	bool "Module versioning support"
 	help
--- a/Documentation/kbuild/modules.txt	2009-01-18 18:45:37.000000000 +0000
+++ b/Documentation/kbuild/modules.txt	2009-01-25 15:10:27.000000000 +0000
@@ -96,6 +96,13 @@ when building an external module.
 		Installation default is in /lib/modules/<kernel-version>/extra,
 		but may be prefixed with INSTALL_MOD_PATH - see separate
 		chapter.
+		If MODULES_COMPRESS is set when the modules_install target is 
+		run then the module is compressed after it has been 
+		copied to /lib/modules/<kernel-version>. Compressed modules 
+		using the default gzip compression format will require 
+		module-init-tools installed with --zlib-enabled.
+		Any options set in MODULE_COMPRESS_OPTIONS will be 
+		passed to the selected compression format.
 
 	make -C $KDIR M=`pwd` clean
 		Remove all generated files for the module - the kernel

Re: [PATCH] Compress kernel modules on installation.

[Jan Engelhardt]

On Sunday 2009-01-25 17:51, Steve Brokenshire wrote:
>+
>+quiet_cmd_modules_install = INSTALL  $@
>+      cmd_modules_install = mkdir -p $(2); \
>+				cp $@ $(2) ; \
>+				$(mod_strip_cmd) $(2)/$(notdir $@)
>+
>+quiet_cmd_modules_compress_gzip = COMPRESS $@
>+      cmd_modules_compress_gzip = gzip $(MODCOMPOPT) -c \
>+				 $(2)/`basename $@` \
>+				 > $(2)/`basename $@`.gz; \
>+				 rm $(2)/`basename $@`

That is *three* extra shell invocations you got there
for something make can compute itself using $(@F) iirc.

Re: [PATCH] Compress kernel modules on installation.

[Steve Brokenshire]

On Sun, 25 Jan 2009 19:02:21 +0100 (CET)
Jan Engelhardt <jengelh@medozas.de> wrote:

> 
> On Sunday 2009-01-25 17:51, Steve Brokenshire wrote:
> >+
> >+quiet_cmd_modules_install = INSTALL  $@
> >+      cmd_modules_install = mkdir -p $(2); \
> >+				cp $@ $(2) ; \
> >+				$(mod_strip_cmd) $(2)/$(notdir $@)
> >+
> >+quiet_cmd_modules_compress_gzip = COMPRESS $@
> >+      cmd_modules_compress_gzip = gzip $(MODCOMPOPT) -c \
> >+				 $(2)/`basename $@` \
> >+				 > $(2)/`basename $@`.gz; \
> >+				 rm $(2)/`basename $@`
> 
> That is *three* extra shell invocations you got there
> for something make can compute itself using $(@F) iirc.

Ahhh. Didn't know that. Thanks. :)

Steve

Full patch (with modifications) follows...

================================
Signed-off-by: Steve Brokenshire <sbrokenshire@xestia.co.uk>

--- a/scripts/Makefile.modinst	2009-01-18 18:45:37.000000000 +0000
+++ b/scripts/Makefile.modinst	2009-01-25 18:39:50.000000000 +0000
@@ -5,6 +5,7 @@
 PHONY := __modinst
 __modinst:
 
+include include/config/auto.conf
 include scripts/Kbuild.include
 
 #
@@ -16,8 +17,21 @@ PHONY += $(modules)
 __modinst: $(modules)
 	@:
 
-quiet_cmd_modules_install = INSTALL $@
-      cmd_modules_install = mkdir -p $(2); cp $@ $(2) ; $(mod_strip_cmd) $(2)/$(notdir $@)
+ifeq ($(CONFIG_MODULE_COMPRESS_OPTIONS), "")
+else
+ MODCOMPOPT = $(shell echo -n $(CONFIG_MODULE_COMPRESS_OPTIONS))
+endif
+
+quiet_cmd_modules_install = INSTALL  $@
+      cmd_modules_install = mkdir -p $(2); \
+				cp $@ $(2) ; \
+				$(mod_strip_cmd) $(2)/$(notdir $@)
+
+quiet_cmd_modules_compress_gzip = COMPRESS $@
+      cmd_modules_compress_gzip = gzip $(MODCOMPOPT) -c \
+				 $(2)/$(@F) \
+				 > $(2)/$(@F).gz; \
+				 rm $(2)/$(@F)
 
 # Modules built outside the kernel source tree go into extra by default
 INSTALL_MOD_DIR ?= extra
@@ -26,8 +40,11 @@ ext-mod-dir = $(INSTALL_MOD_DIR)$(subst 
 modinst_dir = $(if $(KBUILD_EXTMOD),$(ext-mod-dir),kernel/$(@D))
 
 $(modules):
+
 	$(call cmd,modules_install,$(MODLIB)/$(modinst_dir))
 
+	$(if $(CONFIG_MODULE_COMPRESS_GZIP), \
+		$(call cmd,modules_compress_gzip,$(MODLIB)/$(modinst_dir)))
 
 # Declare the contents of the .PHONY variable as phony.  We keep that
 # information in a variable se we can use it in if_changed and friends.
--- a/init/Kconfig	2009-01-18 18:45:37.000000000 +0000
+++ b/init/Kconfig	2009-01-25 16:07:47.000000000 +0000
@@ -887,6 +887,62 @@ config MODULE_FORCE_UNLOAD
 	  rmmod).  This is mainly for kernel developers and desperate users.
 	  If unsure, say N.
 
+config MODULE_COMPRESS
+	bool "Compress kernel modules on installation"
+	depends on MODULES
+	help
+	  This option compresses the kernel modules when 'make
+	  modules_install' is run.
+
+	  The modules will be compressed into the selected compression 
+	  format with gzip being the default compression format.
+
+	  When a kernel module is installed from outside of the main kernel
+	  source and uses the Kbuild system for installing modules then that
+	  kernel module will also be compressed when it is installed.
+
+	  When running mkinitrd you will find that an error message
+	  appears saying that it cannot find a certain kernel module.
+	  As a workaround, unset CONFIG_MODULE_COMPRESS, build the modules
+	  and install them, run mkinitrd and create the initrd image, place
+	  the initrd image in the correct place for booting, set
+	  CONFIG_MODULE_COMPRESS and then install the modules again.
+
+	  This option requires the module-init-tools package to be 
+	  configured with --enable-zlib (if using gzip which is the
+	  default compression format).
+
+	  If unsure, say N.
+
+config MODULE_COMPRESS_OPTIONS
+	string "Compression format command line options"
+	depends on MODULE_COMPRESS
+	help
+	  This option specifies the command line options to be used for
+	  the selected compression format.
+
+	  Please refer to the selected compression format's documentation
+	  on which options should be used.
+
+	  If unsure, leave this option blank.
+
+choice
+	prompt "Kernel module compression format"
+	depends on MODULE_COMPRESS
+	default MODULE_COMPRESS_GZIP
+
+config MODULE_COMPRESS_GZIP
+	bool "gzip compression"
+	help
+	  Compresses the kernel modules using the gzip (GNU zip) 
+	  compression format.
+
+	  This option requires gzip to be installed.
+
+	  If unsure, leave this option selected.
+	  
+endchoice
+
 config MODVERSIONS
 	bool "Module versioning support"
 	help
--- a/Documentation/kbuild/modules.txt	2009-01-18 18:45:37.000000000 +0000
+++ b/Documentation/kbuild/modules.txt	2009-01-25 16:07:47.000000000 +0000
@@ -96,6 +96,13 @@ when building an external module.
 		Installation default is in /lib/modules/<kernel-version>/extra,
 		but may be prefixed with INSTALL_MOD_PATH - see separate
 		chapter.
+		If MODULES_COMPRESS is set when the modules_install target is 
+		run then the module is compressed after it has been 
+		copied to /lib/modules/<kernel-version>. Compressed modules 
+		using the default gzip compression format will require 
+		module-init-tools installed with --zlib-enabled.
+		Any options set in MODULE_COMPRESS_OPTIONS will be 
+		passed to the selected compression format.
 
 	make -C $KDIR M=`pwd` clean
 		Remove all generated files for the module - the kernel