Build system runs ld more often than needed

Build system runs ld more often than needed

[Jean Delvare]

Hi Sam,

I have noticed the following problem:

khali@arrakis:~/src/linux-2.6.16-git> make modules
  CHK     include/linux/version.h
  Building modules, stage 2.
  MODPOST
khali@arrakis:~/src/linux-2.6.16-git> touch drivers/media/video/zoran_card.c 
khali@arrakis:~/src/linux-2.6.16-git> make modules
  CHK     include/linux/version.h
  CC [M]  drivers/media/video/zoran_card.o
  LD [M]  drivers/media/video/zr36067.o
  LD [M]  drivers/media/video/msp3400.o
  LD [M]  drivers/media/video/tuner.o
  Building modules, stage 2.
  MODPOST
  LD [M]  drivers/media/video/msp3400.ko
  LD [M]  drivers/media/video/tuner.ko
  LD [M]  drivers/media/video/zr36067.ko
khali@arrakis:~/src/linux-2.6.16-git>

See how unrelated modules are linked again?

I investigated further and it seems to happen whenever a given Makefile
has more than one composite object definition. In the case of
drivers/media/video, the following composite objects are defined:

zoran-objs      :=	zr36120.o zr36120_i2c.o zr36120_mem.o
zr36067-objs	:=	zoran_procfs.o zoran_device.o \
			zoran_driver.o zoran_card.o
tuner-objs	:=	tuner-core.o tuner-types.o tuner-simple.o \
			mt20xx.o tda8290.o tea5767.o

msp3400-objs	:=	msp3400-driver.o msp3400-kthreads.o

I have the following enabled:

CONFIG_VIDEO_MSP3400=m
CONFIG_VIDEO_ZORAN=m
CONFIG_VIDEO_TUNER=m

So msp3400 and tuner are relinked whenever I make a change to the
zr36067 driver.

I was able to reproduce the problem in drivers/scsi so it's not local
to the media/video area. I was also able to reproduce the problem back
to Linux 2.6.0 so the problem is not new.

Can this be looked upon?

Thanks,
-- 
Jean Delvare

Re: Build system runs ld more often than needed

[Sam Ravnborg]

On Mon, Mar 27, 2006 at 02:38:48PM +0200, Jean Delvare wrote:
> Hi Sam,
> 
> I have noticed the following problem:
> 
> khali@arrakis:~/src/linux-2.6.16-git> make modules
>   CHK     include/linux/version.h
>   Building modules, stage 2.
>   MODPOST
> khali@arrakis:~/src/linux-2.6.16-git> touch drivers/media/video/zoran_card.c 
> khali@arrakis:~/src/linux-2.6.16-git> make modules
>   CHK     include/linux/version.h
>   CC [M]  drivers/media/video/zoran_card.o
>   LD [M]  drivers/media/video/zr36067.o
>   LD [M]  drivers/media/video/msp3400.o
>   LD [M]  drivers/media/video/tuner.o
>   Building modules, stage 2.
>   MODPOST
>   LD [M]  drivers/media/video/msp3400.ko
>   LD [M]  drivers/media/video/tuner.ko
>   LD [M]  drivers/media/video/zr36067.ko
> khali@arrakis:~/src/linux-2.6.16-git>
> 
> See how unrelated modules are linked again?

This is an unfortunate side-effect. Following snippet from
scripts/Makfile.build explains it:

# We would rather have a list of rules like
# 	foo.o: $(foo-objs)
# but that's not so easy, so we rather make all composite objects depend
# on the set of all their parts
$(multi-used-y) : %.o: $(multi-objs-y) FORCE
	$(call if_changed,link_multi-y)

The problem is that we have no easy way to say that this specific
module depends on this list of .o files.
With make 3.81 this would be possible utilising $(eval ...),
but the benefit is too low to introduce such a dependency.

> 
> I investigated further and it seems to happen whenever a given Makefile
> has more than one composite object definition. In the case of
> drivers/media/video, the following composite objects are defined:
> 
> zoran-objs      :=	zr36120.o zr36120_i2c.o zr36120_mem.o
> zr36067-objs	:=	zoran_procfs.o zoran_device.o \
> 			zoran_driver.o zoran_card.o
> tuner-objs	:=	tuner-core.o tuner-types.o tuner-simple.o \
> 			mt20xx.o tda8290.o tea5767.o
> 
> msp3400-objs	:=	msp3400-driver.o msp3400-kthreads.o
> 
> I have the following enabled:
> 
> CONFIG_VIDEO_MSP3400=m
> CONFIG_VIDEO_ZORAN=m
> CONFIG_VIDEO_TUNER=m
> 
> So msp3400 and tuner are relinked whenever I make a change to the
> zr36067 driver.

You could do:
make drivers/media/video/zr36067.ko
to avoid linking the others.

But kbuild will unfortunately link to much modules for a normal build.

One day I may try to generate one big Makefile and then it could
be addressed but I have yet to find 'the' reason to invest time in that
task.

	Sam

Re: Build system runs ld more often than needed

[Jean Delvare]

Hi Sam,

> > See how unrelated modules are linked again?
> 
> This is an unfortunate side-effect. Following snippet from
> scripts/Makfile.build explains it:
> 
> # We would rather have a list of rules like
> # 	foo.o: $(foo-objs)
> # but that's not so easy, so we rather make all composite objects depend
> # on the set of all their parts
> $(multi-used-y) : %.o: $(multi-objs-y) FORCE
> 	$(call if_changed,link_multi-y)
> 
> The problem is that we have no easy way to say that this specific
> module depends on this list of .o files.

OK, I see. Indeed, we had a similar problem in quilt. The kernel build
system is a bit too complex for me to propose a solution, but maybe if
I show you two rules I wrote for a similar case in the quilt Makefile,
you'll see if the same can be done for the kernel.

$(COMPAT_SYMLINKS:%=compat/%) :: Makefile
	$(call VIRTUAL_SYMLINK, \
		$($(shell echo $@ | $(AWK) '{split($$1, ar, "/"); print toupper(ar[2])}')), \
		$(strip $@))
	@chmod +x $(strip $@)

install-compat-symlink-% :: install-compat1
	ln -sf $($(shell echo $* | $(AWK) '{print toupper($$1)}'))	\
	       $(BUILD_ROOT)$(datadir)/$(PACKAGE)/compat/$*

Basically, the idea is to use awk (or any suitable tool, for that
matter) to extract and/or transform the relevant part of either the
target ($@) or the stem ($*), so as to build a variable name from it,
and then access that variable.

Given that composite targets each have an associated variable listing
the parts they must be made of, maybe it is possible to use the same
method for them? Or maybe it's too much hassle for the thin benefit,
it's up to you.

> With make 3.81 this would be possible utilising $(eval ...),
> but the benefit is too low to introduce such a dependency.

GNU make 3.80 already has $(eval ...), but having a dependency even on
3.80 is no good, agreed.

Thanks,
-- 
Jean Delvare

Re: Build system runs ld more often than needed

[Sam Ravnborg]

On Mon, Mar 27, 2006 at 06:17:28PM +0200, Jean Delvare wrote:
 
> Given that composite targets each have an associated variable listing
> the parts they must be made of, maybe it is possible to use the same
> method for them? Or maybe it's too much hassle for the thin benefit,
> it's up to you.

Too much hassle considering the small benefit.
Especially since this will slow down the allmodconfig case even more
than today. The most time spent during a kernel build is in make
doing nothing other than checking prerequisites.

	Sam