[PATCH] make: add modules_update target

[PATCH] make: add modules_update target

[Kylene Jo Hall]

Here is a a patch that adds a kernel build target called
"modules_update".  

The existing "modules_install" target blows away the entire 
/lib/modules/`uname -r`/kernel directory and copies out every single
module when called at the top level.

This new "modules_update" target only copies out modules that have
changed, using "cp -u".  This less zealous method is a more efficient
approach to module installation for kernel developers working on single,
or small numbers of modules.  

Where "make modules_install" was taking ~ 3 minutes, "make
modules_update" is taking < 1 minute, as a rough benchmark.

Signed-off-by: Kylie Hall <kjhall@us.ibm.com>
Signed-off-by: Dustin Kirkland <dustin.kirkland@us.ibm.com>
---
 Makefile                 |   34 ++++++++++++++++++++++++++++++++++
 scripts/Makefile.modupdt |   29 +++++++++++++++++++++++++++++
 2 files changed, 63 insertions(+)

--- linux-2.6.15-rc5/Makefile	2005-12-03 23:10:42.000000000 -0600
+++ linux-2.6.15-rc5-mm3/Makefile	2006-04-13 12:05:26.000000000 -0500
@@ -922,6 +919,40 @@ modules: $(vmlinux-dirs) $(if $(KBUILD_B
 .PHONY: modules_prepare
 modules_prepare: prepare scripts
 
+# Target to update modules
+.PHONY: modules_update
+modules_update: _modupdt_ _modupdt_post
+
+.PHONY: _modupdt_
+_modupdt_:
+	@if [ -z "`$(DEPMOD) -V 2>/dev/null | grep module-init-tools`" ]; then \
+		echo "Warning: you may need to install module-init-tools"; \
+		echo "See http://www.codemonkey.org.uk/docs/post-halloween-2.6.txt";\
+		sleep 1; \
+	fi
+	@rm -f $(MODLIB)/source
+	@mkdir -p $(MODLIB)/kernel
+	@ln -s $(srctree) $(MODLIB)/source
+	@if [ ! $(objtree) -ef  $(MODLIB)/build ]; then \
+		rm -f $(MODLIB)/build ; \
+		ln -s $(objtree) $(MODLIB)/build ; \
+	fi
+	$(Q)$(MAKE) -rR -f $(srctree)/scripts/Makefile.modupdt
+
+# If System.map exists, run depmod.  This deliberately does not have a
+# dependency on System.map since that would run the dependency tree on
+# vmlinux.  This depmod is only for convenience to give the initial
+# boot a modules.dep even before / is mounted read-write.  However the
+# boot script depmod is the master version.
+ifeq "$(strip $(INSTALL_MOD_PATH))" ""
+depmod_opts	:=
+else
+depmod_opts	:= -b $(INSTALL_MOD_PATH) -r
+endif
+.PHONY: _modupdt_post
+_modupdt_post: _modupdt_
+	if [ -r System.map -a -x $(DEPMOD) ]; then $(DEPMOD) -ae -F System.map $(depmod_opts) $(KERNELRELEASE); fi
+
 # Target to install modules
 .PHONY: modules_install
 modules_install: _modinst_ _modinst_post
--- linux-2.6.15-rc5/scripts/Makefile.modupdt	1969-12-31 18:00:00.000000000 -0600
+++ linux-2.6.15-rc5-mm3/scripts/Makefile.modupdt	2006-04-13 13:37:24.000000000 -0500
@@ -0,0 +1,29 @@
+# ==========================================================================
+# Installing modules
+# ==========================================================================
+
+.PHONY: __modupdt
+__modupdt:
+
+include scripts/Kbuild.include
+
+#
+
+__modules := $(sort $(shell grep -h '\.ko' /dev/null $(wildcard $(MODVERDIR)/*.mod)))
+modules := $(patsubst %.o,%.ko,$(wildcard $(__modules:.ko=.o)))
+
+.PHONY: $(modules)
+__modupdt: $(modules)
+	@:
+
+      cmd_modules_update = mkdir -p $(2); cp -vu $@ $(2) | $(AWK) -F"'" '{print $$1}' | sed "s/\`/  UPDATE /"
+
+# Modules built outside the kernel source tree go into extra by default
+INSTALL_MOD_DIR ?= extra
+ext-mod-dir = $(INSTALL_MOD_DIR)$(subst $(KBUILD_EXTMOD),,$(@D))
+
+modinst_dir = $(if $(KBUILD_EXTMOD),$(ext-mod-dir),kernel/$(@D))
+
+$(modules):
+	$(call cmd,modules_update,$(MODLIB)/$(modinst_dir))
+

Re: [PATCH] make: add modules_update target

[Theodore Ts'o]

On Fri, Apr 14, 2006 at 10:06:56AM -0500, Kylene Jo Hall wrote:
> Here is a a patch that adds a kernel build target called
> "modules_update".  
> 
> The existing "modules_install" target blows away the entire 
> /lib/modules/`uname -r`/kernel directory and copies out every single
> module when called at the top level.
> 
> This new "modules_update" target only copies out modules that have
> changed, using "cp -u".  This less zealous method is a more efficient
> approach to module installation for kernel developers working on single,
> or small numbers of modules.  

Hi Kylene,

This works as long as the .config hasn't been changed so that some
configuration options haven't been changed so that a driver which had
been previously built as a module is now built into the kernel.  In
that case, you really want to make sure the no-longer applicable .ko
file has been removed from the system.  If the developer knows that to
be true, they can use your proposed modules_update without any problems.

As a suggestion, something that might be worth trying would be to
change to modules_install so that it uses cp -u, but also so that it
tries to delete all files that could have previously installed as
modules (by using the obj-y list).  This should hopefully speed up
modules_install, and make it do the right thing all the time.

Regards,

						- Ted

Re: [PATCH] make: add modules_update target

[Avi Kivity]

Theodore Ts'o wrote:
> On Fri, Apr 14, 2006 at 10:06:56AM -0500, Kylene Jo Hall wrote:
>   
>> This new "modules_update" target only copies out modules that have
>> changed, using "cp -u".  This less zealous method is a more efficient
>> approach to module installation for kernel developers working on single,
>> or small numbers of modules.  
>>     
>
> Hi Kylene,
>
> This works as long as the .config hasn't been changed so that some
> configuration options haven't been changed so that a driver which had
> been previously built as a module is now built into the kernel.  In
> that case, you really want to make sure the no-longer applicable .ko
> file has been removed from the system.  If the developer knows that to
> be true, they can use your proposed modules_update without any problems.
>
> As a suggestion, something that might be worth trying would be to
> change to modules_install so that it uses cp -u, but also so that it
> tries to delete all files that could have previously installed as
> modules (by using the obj-y list).  This should hopefully speed up
> modules_install, and make it do the right thing all the time.
>
>   
How about using rsync with --delete as a substitute for cp (if rsync is 
available)?

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: [PATCH] make: add modules_update target

[Dustin Kirkland]

On Fri, 2006-04-14 at 21:00 +0300, Avi Kivity wrote:
> How about using rsync with --delete as a substitute for cp (if rsync is 
> available)?

I thought about this, but a "grep -r rsync" didn't turn up any previous
hits in the kernel build process.  I didn't want to introduce this as a
new dependency for kernel building, if it's possible to avoid...


:-Dustin

Re: [PATCH] make: add modules_update target

[Avi Kivity]

Dustin Kirkland wrote:
> On Fri, 2006-04-14 at 21:00 +0300, Avi Kivity wrote:
>   
>> How about using rsync with --delete as a substitute for cp (if rsync is 
>> available)?
>>     
>
> I thought about this, but a "grep -r rsync" didn't turn up any previous
> hits in the kernel build process.  I didn't want to introduce this as a
> new dependency for kernel building, if it's possible to avoid...
>   
Use rsync only if it is available:

    rsync-available := $(shell rsync --version > /dev/null 2>&1 && echo y)
    copy := $(if $(rsync-available), rsync --delete, cp)

    modules_install:
               [...]
               $(copy) source target

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: [PATCH] make: add modules_update target

[Dustin Kirkland]

On Fri, 2006-04-14 at 22:02 +0300, Avi Kivity wrote:
> Use rsync only if it is available:
> 
>     rsync-available := $(shell rsync --version > /dev/null 2>&1 && echo y)
>     copy := $(if $(rsync-available), rsync --delete, cp)
> 
>     modules_install:
>                [...]
>                $(copy) source target

Actually, rsync --delete is not a viable option either.  If you first
build a kernel with a particular item built as a module, and then
afterward rebuild with the same item as built-in (or not at all),
the .ko file remains in your kernel build tree (ie, it won't be deleted
on the source such that the --delete would have your desired effect).

Furthermore, rsync's performance is considerably worse in my testing
than "cp -u", almost back to the original performance of the "rm -rf, cp
all" of the original modules_install.  (Note that I tested, rsync's
default, checksum, mod-times, and size-only.)

:-Dustin

Re: [PATCH] make: add modules_update target

[Dustin Kirkland]

On Fri, 2006-04-14 at 13:02 -0400, Theodore Ts'o wrote: 
> This works as long as the .config hasn't been changed so that some
> configuration options haven't been changed so that a driver which had
> been previously built as a module is now built into the kernel.  In
> that case, you really want to make sure the no-longer applicable .ko
> file has been removed from the system.  If the developer knows that to
> be true, they can use your proposed modules_update without any problems.
> 
> As a suggestion, something that might be worth trying would be to
> change to modules_install so that it uses cp -u, but also so that it
> tries to delete all files that could have previously installed as
> modules (by using the obj-y list).  This should hopefully speed up
> modules_install, and make it do the right thing all the time.

Ted-

So I found the obj-y list a little tough to work with, as the objects
listed there do not contain fully qualified paths describing their
locations.

Here's another approach...

What you would think of some logic such that 

        if .config file has changed since last modules_install
                call normal brute force modules_install target
        else
                call this new, more efficient modules_update target
        
I was looking through the top level Makefile for a flag or some such
that would determine if a .config file has changed since last build, to
no avail.

It would be pretty easy to maintain an md5sum/sha1sum signature of
the .config used to perform a modules install/update (.config.md5), and
compare the signature of the current .conf with the previous to
determine change.

It looks like it may not be easy to drop in modules_update as a more
efficient alternative to modules_install, but note that is not the patch
that Kylie submitted...  

I suggest that it might be sufficient to document modules_update as a
handy, more efficient development build target when someone is working
on one particular module-built aspect of the kernel.  And clearly note
that to perform a full, clean /lib/modules/`uname -r`/kernel
synchronization with a given build tree, the more thorough
modules_install target is preferred.

Thoughts?

:-Dustin

Re: [PATCH] make: add modules_update target

[Sam Ravnborg]

On Fri, Apr 14, 2006 at 07:33:39PM -0500, Dustin Kirkland wrote:
> It looks like it may not be easy to drop in modules_update as a more
> efficient alternative to modules_install, but note that is not the patch
> that Kylie submitted...
The problem to be solved is the long time it takes to do
"make modules_install" when working on a single module.
Instead of bringing in more or less complex solutions what about
extending "make dir/module.ko" to include the installation of the
module.

Something like:
"make MI=1 dir/module.ko"
where MI=1 tells us to install the said module.

I'm not particular found of the syntax - anyone with a better proposal?

Untested sample patch below.

	Sam

diff --git a/Makefile b/Makefile
index fc8e08c..0c0649c 100644
--- a/Makefile
+++ b/Makefile
@@ -1312,6 +1312,11 @@ # Modules
 	$(Q)$(MAKE) KBUILD_MODULES=$(if $(CONFIG_MODULES),1)   \
 	$(build)=$(build-dir) $(@:.ko=.o)
 	$(Q)$(MAKE) -rR -f $(srctree)/scripts/Makefile.modpost
+ifneq ($(MI),)
+	cp $@ $(MODLIB)/kernel/$(dir $@)
+	if [ -r System.map -a -x $(DEPMOD) ]; then \
+            $(DEPMOD) -ae -F System.map $(depmod_opts) $(KERNELRELEASE); fi
+endif
 
 # FIXME Should go into a make.lib or something 
 # ===========================================================================

Re: [PATCH] make: add modules_update target

[Theodore Ts'o]

On Sat, Apr 15, 2006 at 10:40:58AM +0200, Sam Ravnborg wrote:
> The problem to be solved is the long time it takes to do
> "make modules_install" when working on a single module.
> Instead of bringing in more or less complex solutions what about
> extending "make dir/module.ko" to include the installation of the
> module.
> 
> Something like:
> "make MI=1 dir/module.ko"
> where MI=1 tells us to install the said module.

Um, wouldn't that imply that either (a) the compile is being done as
root, or (b) the /lib/modules/* is writeable by a non-root userid?  I
suppose the install command could be prefixed by sudo, but that seems
awkward (and not everyone uses sudo).

						- Ted

Re: [PATCH] make: add modules_update target

[Sam Ravnborg]

On Sat, Apr 15, 2006 at 11:02:08AM -0400, Theodore Ts'o wrote:
> On Sat, Apr 15, 2006 at 10:40:58AM +0200, Sam Ravnborg wrote:
> > The problem to be solved is the long time it takes to do
> > "make modules_install" when working on a single module.
> > Instead of bringing in more or less complex solutions what about
> > extending "make dir/module.ko" to include the installation of the
> > module.
> > 
> > Something like:
> > "make MI=1 dir/module.ko"
> > where MI=1 tells us to install the said module.
> 
> Um, wouldn't that imply that either (a) the compile is being done as
> root, or (b) the /lib/modules/* is writeable by a non-root userid?  I
> suppose the install command could be prefixed by sudo, but that seems
> awkward (and not everyone uses sudo).

kbuild has support for the above scenario already - I just forgot.
Say you are hacking ext3.
Do a successfull make and install all modules.
Manually remove the ext3 module from /lib/modules/...
And use the external module support in kbuild like this:

# Got to relevant directory
$> cd fs/ext3

# To build the module:
$> make -C ../.. M=`pwd`

# To install the module:
$> make -C ../.. M=`pwd` modules_install

	Sam