[RFC/CFT] Separate obj/src dir

[RFC/CFT] Separate obj/src dir

[Sam Ravnborg]

Based on some initial work by Kai Germaschewski I have made a
working prototype of separate obj/src tree.

Usage example:
#src located in ~/bk/linux-2.5.sepobj
mkdir ~/compile/v2.5
cd ~/compile/v2.5
sh ../../kb/v2.5/kbuild

Prints out:
SRCTREE=/home/sam/bk/linux-2.5.sepobj
OBJTREE=/home/sam/compile/v2.5
And then the normal make process starts after a short period.

The kbuild shell script takes a verbatim copy of all Makefiles,
all Kconfig files and all defconfigs. I did not even look into
using symlinks, I was not sure how they work across NFS
and the like.

The VPATH feature of make is used to let make locate the
src in the directory where the kbuild script is located.

I had to do some trikery with the CFLAGS to make it work.
The processing done in the flags define in Makefile.build is
rather unpleasent and I have another solution in mind I will
give a try soon.
I will try to include files via a directory in the root
of OBJTREE and then create symlinks towards the directories
present in SRCTREE - but need to play a bit more with that.

Another drawback is that when a .h file exist in the
SRCTREE but not in the OBJTREE the generated dependencies
will point out the .h file located in SRCTREE.
This happens for generated .h files, and therefore a simple
check is made in kbuild to check that the SRCTREE is
cleaned/mrpropered.

kconfig did not have an option to read the Kconfig files +
defconfig from somewhere else than current directory,
therefore they are copied. But that should be trivial to do.
Possible solutions:
1) Command line option:
	-r path-to-rrot-of-tree
2) Deduct it from the argument given, but then kconfig
   needs to know a bit too much about the kernel src tree.
3) Utilise the environment variable $(srctree), which is
   anyway valid.

Comments expected...

	Sam


===== Makefile 1.346 vs edited =====
--- 1.346/Makefile	Sat Nov  9 05:08:32 2002
+++ edited/Makefile	Mon Nov 18 23:07:59 2002
@@ -136,13 +136,18 @@
 export quiet Q KBUILD_VERBOSE
 
 #	Paths to obj / src tree
-
+ifneq ($(wildcard .tmp_make_config),)
+include .tmp_make_config
+src     := $(srctree)
+obj     := .
+VPATH   := $(srctree)
+else
 src	:= .
 obj	:= .
 srctree := .
 objtree := .
-
-export srctree objtree
+endif
+export srctree objtree VPATH
 
 # 	Make variables (CC, etc...)
 
@@ -304,7 +309,7 @@
 	set -e
 	$(if $(filter .tmp_kallsyms%,$^),,
 	  echo '  Generating build number'
-	  . scripts/mkversion > .tmp_version
+	  $(CONFIG_SHELL) $(src)/scripts/mkversion > .tmp_version
 	  mv -f .tmp_version .version
 	  $(Q)$(MAKE) -f scripts/Makefile.build obj=init
 	)
@@ -406,7 +411,11 @@
 
 include/asm:
 	@echo '  Making asm->asm-$(ARCH) symlink'
+ifeq ($(srctree),$(objtree))
 	@ln -s asm-$(ARCH) $@
+else
+	@ln -s $(src)/include/asm-$(ARCH) $@
+endif
 
 # 	Split autoconf.h into include/linux/config/*
 
@@ -585,7 +594,7 @@
 	tar -cvz $(RCS_TAR_IGNORE) -f $(KERNELPATH).tar.gz $(KERNELPATH)/. ; \
 	rm $(KERNELPATH) ; \
 	cd $(TOPDIR) ; \
-	. scripts/mkversion > .version ; \
+	$(CONFIG_SHELL) $(src)/scripts/mkversion > .version ; \
 	rpm -ta $(TOPDIR)/../$(KERNELPATH).tar.gz ; \
 	rm $(TOPDIR)/../$(KERNELPATH).tar.gz
 
===== scripts/Makefile.build 1.11 vs edited =====
--- 1.11/scripts/Makefile.build	Thu Nov 14 17:08:38 2002
+++ edited/scripts/Makefile.build	Tue Nov 19 20:39:41 2002
@@ -84,26 +84,34 @@
 $(multi-objs-y:.o=.s)   : modname = $(modname-multi)
 $(multi-objs-y:.o=.lst) : modname = $(modname-multi)
 
+# Allow gcc to locate header files in srctree, if we use separate objtree
+ifeq ($(srctree),$(objtree))
+flags = $($(1))
+else
+flags = -I$(obj) -I$(srctree)/$(src) $($(1)) $(patsubst -I%,-I$(srctree)/%,$(filter -I%,$($(1))))
+endif
+
 quiet_cmd_cc_s_c = CC $(quiet_modtag)  $@
-cmd_cc_s_c       = $(CC) $(c_flags) -S -o $@ $< 
+cmd_cc_s_c       = $(CC) $(call flags,c_flags) -S -o $@ $< 
 
 %.s: %.c FORCE
 	$(call if_changed_dep,cc_s_c)
 
 quiet_cmd_cc_i_c = CPP $(quiet_modtag) $@
-cmd_cc_i_c       = $(CPP) $(c_flags)   -o $@ $<
+cmd_cc_i_c       = $(CPP) $(call flags,c_flags) -o $@ $<
 
 %.i: %.c FORCE
 	$(call if_changed_dep,cc_i_c)
 
 quiet_cmd_cc_o_c = CC $(quiet_modtag)  $@
-cmd_cc_o_c       = $(CC) $(c_flags) -c -o $@ $<
+cmd_cc_o_c       = $(CC) $(call flags,c_flags) -c -o $@ $<
 
 %.o: %.c FORCE
 	$(call if_changed_dep,cc_o_c)
 
 quiet_cmd_cc_lst_c = MKLST   $@
-cmd_cc_lst_c       = $(CC) $(c_flags) -g -c -o $*.o $< && sh scripts/makelst $*.o System.map $(OBJDUMP) > $@
+cmd_cc_lst_c       = $(CC) $(call flags,c_flags) -g -c -o $*.o $< && \
+		     sh $(src)/scripts/makelst $*.o System.map $(OBJDUMP) > $@
 
 %.lst: %.c FORCE
 	$(call if_changed_dep,cc_lst_c)
@@ -116,17 +124,14 @@
 $(real-objs-m)      : modkern_aflags := $(AFLAGS_MODULE)
 $(real-objs-m:.o=.s): modkern_aflags := $(AFLAGS_MODULE)
 
-a_flags = -Wp,-MD,$(depfile) $(AFLAGS) $(NOSTDINC_FLAGS) \
-	  $(modkern_aflags) $(EXTRA_AFLAGS) $(AFLAGS_$(*F).o)
-
 quiet_cmd_as_s_S = CPP $(quiet_modtag) $@
-cmd_as_s_S       = $(CPP) $(a_flags)   -o $@ $< 
+cmd_as_s_S       = $(CPP) $(call flags,a_flags) -o $@ $< 
 
 %.s: %.S FORCE
 	$(call if_changed_dep,as_s_S)
 
 quiet_cmd_as_o_S = AS $(quiet_modtag)  $@
-cmd_as_o_S       = $(CC) $(a_flags) -c -o $@ $<
+cmd_as_o_S       = $(CC) $(call flags,a_flags) -c -o $@ $<
 
 %.o: %.S FORCE
 	$(call if_changed_dep,as_o_S)
@@ -217,8 +222,7 @@
 # Create executable from a single .c file
 # host-csingle -> Executable
 quiet_cmd_host-csingle 	= HOSTCC  $@
-      cmd_host-csingle	= $(HOSTCC) -Wp,-MD,$(depfile) \
-			  $(HOSTCFLAGS) $(HOST_EXTRACFLAGS) \
+      cmd_host-csingle	= $(HOSTCC) $(call flags,hostc_flags) \
 			  $(HOST_LOADLIBES) -o $@ $<
 $(host-csingle): %: %.c FORCE
 	$(call if_changed_dep,host-csingle)
@@ -235,9 +239,7 @@
 # Create .o file from a single .c file
 # host-cobjs -> .o
 quiet_cmd_host-cobjs	= HOSTCC  $@
-      cmd_host-cobjs	= $(HOSTCC) -Wp,-MD,$(depfile) \
-			  $(HOSTCFLAGS) $(HOST_EXTRACFLAGS) \
-			  $(HOSTCFLAGS_$(@F)) -c -o $@ $<
+      cmd_host-cobjs	= $(HOSTCC) $(call flags,hostc_flags) -c -o $@ $<
 $(host-cobjs): %.o: %.c FORCE
 	$(call if_changed_dep,host-cobjs)
 
@@ -253,18 +255,14 @@
 
 # Create .o file from a single .cc (C++) file
 quiet_cmd_host-cxxobjs	= HOSTCXX $@
-      cmd_host-cxxobjs	= $(HOSTCXX) -Wp,-MD,$(depfile) \
-			  $(HOSTCXXFLAGS) $(HOST_EXTRACXXFLAGS) \
-			  $(HOSTCXXFLAGS_$(@F)) -c -o $@ $<
+      cmd_host-cxxobjs	= $(HOSTCXX) $(call flags,hostcxx_flags) -c -o $@ $<
 $(host-cxxobjs): %.o: %.cc FORCE
 	$(call if_changed_dep,host-cxxobjs)
 
 # Compile .c file, create position independent .o file
 # host-cshobjs -> .o
 quiet_cmd_host-cshobjs	= HOSTCC  -fPIC $@
-      cmd_host-cshobjs	= $(HOSTCC) -Wp,-MD,$(depfile) -fPIC\
-			  $(HOSTCFLAGS) $(HOST_EXTRACFLAGS) \
-			  $(HOSTCFLAGS_$(@F)) -c -o $@ $<
+      cmd_host-cshobjs	= $(HOSTCC) $(call flags,hostc_flags) -fPIC -c -o $@ $<
 $(host-cshobjs): %.o: %.c FORCE
 	$(call if_changed_dep,host-cshobjs)
 
===== scripts/Makefile.lib 1.3 vs edited =====
--- 1.3/scripts/Makefile.lib	Wed Oct 30 03:52:07 2002
+++ edited/scripts/Makefile.lib	Tue Nov 19 20:45:25 2002
@@ -128,6 +128,14 @@
 c_flags        = -Wp,-MD,$(depfile) $(CFLAGS) $(NOSTDINC_FLAGS) \
 	         $(modkern_cflags) $(EXTRA_CFLAGS) $(CFLAGS_$(*F).o) \
 	         $(basename_flags) $(modname_flags) $(export_flags) 
+a_flags        = -Wp,-MD,$(depfile) $(AFLAGS) $(NOSTDINC_FLAGS) \
+		 $(modkern_aflags) $(EXTRA_AFLAGS) $(AFLAGS_$(*F).o)
+hostc_flags    = -Wp,-MD,$(depfile) \
+		 $(HOSTCFLAGS) $(HOST_EXTRACFLAGS) \
+		 $(HOSTCFLAGS_$(*F).o)
+hostcxx_flags  = -Wp,-MD,$(depfile) \
+	 	 $(HOSTCXXFLAGS) $(HOST_EXTRACXXFLAGS) \
+		 $(HOSTCXXFLAGS_$(*F).o)
 
 # Finds the multi-part object the current object will be linked into
 modname-multi = $(subst $(space),_,$(strip $(foreach m,$(multi-used),\
@@ -139,7 +147,7 @@
 quiet_cmd_shipped = SHIPPED $@
 cmd_shipped = cat $< > $@
 
-%:: %_shipped
+$(obj)/%:: $(src)/%_shipped
 	$(call cmd,shipped)
 
 # Commands useful for building a boot image
--- /dev/null	2002-08-31 01:31:37.000000000 +0200
+++ kbuild	2002-11-19 20:07:35.000000000 +0100
@@ -0,0 +1,28 @@
+#!/bin/sh
+OBJTREE=$PWD
+cd `dirname $0`
+SRCTREE=$PWD
+cd $OBJTREE
+echo OBJTREE $OBJTREE
+echo SRCTREE $SRCTREE
+if [ "$SRCTREE" != "$OBJTREE" ]; then
+  if [ -f $SRCTREE/.config -o -d $SRCTREE/include/asm ]; then
+    echo '$SRCTREE contains generated files, please run "make mrproper" in the SRCTREE'
+  else
+    for a in `cd $SRCTREE; find -type d`; do
+      mkdir -p $a;
+    done
+    for a in `cd $SRCTREE; find -name Makefile\* -o -name Kconfig\* -o -name defconfig`; do
+      cp -f $SRCTREE/$a $a
+    done
+
+    ( echo "srctree := $SRCTREE";
+      echo "objtree := $OBJTREE";
+    ) > .tmp_make_config
+    touch Rules.make
+  make $*
+  fi
+else
+  rm -f .tmp_make_config
+  make $*
+fi

Re: [RFC/CFT] Separate obj/src dir

[Richard B. Johnson]

On Tue, 19 Nov 2002, Sam Ravnborg wrote:

> Based on some initial work by Kai Germaschewski I have made a
> working prototype of separate obj/src tree.
> 
> Usage example:
> #src located in ~/bk/linux-2.5.sepobj
> mkdir ~/compile/v2.5
> cd ~/compile/v2.5
> sh ../../kb/v2.5/kbuild

[SNIPPED...]

I have a question; "What problem is this supposed to solve?"
This looks like a M$ism to me. Real source trees don't
look like this. If you don't have write access to the source-
code tree, you are screwed on a real project anyway. That's
why we have CVS, tar and other tools to provide a local copy.

Cheers,
Dick Johnson
Penguin : Linux version 2.4.18 on an i686 machine (797.90 BogoMips).
   Bush : The Fourth Reich of America

Re: [RFC/CFT] Separate obj/src dir

[Sam Ravnborg]

On Tue, Nov 19, 2002 at 03:22:45PM -0500, Richard B. Johnson wrote:
> I have a question; "What problem is this supposed to solve?"

Two problems (at least):

1) You want to compile your kernel based on two different configurations,
but sharing the same src. No need to have a duplicate of all src.
- There are other ways to do this using symlinks

2) You have the src located on a read-only filesystem.
I have been told this is the case for some SCM systems.

People has requested this feature at several occasions, and here
it is based on the current build system.
It's not ready for inclusion (obviously), and you shall
also see this as a way to check that this is considered usefull
by someone.

	Sam

Re: [RFC/CFT] Separate obj/src dir

[Richard B. Johnson]

On Tue, 19 Nov 2002, Sam Ravnborg wrote:

> On Tue, Nov 19, 2002 at 03:22:45PM -0500, Richard B. Johnson wrote:
> > I have a question; "What problem is this supposed to solve?"
> 
> Two problems (at least):
> 
> 1) You want to compile your kernel based on two different configurations,
> but sharing the same src. No need to have a duplicate of all src.
> - There are other ways to do this using symlinks
> 
> 2) You have the src located on a read-only filesystem.
> I have been told this is the case for some SCM systems.
> 
> People has requested this feature at several occasions, and here
> it is based on the current build system.
> It's not ready for inclusion (obviously), and you shall
> also see this as a way to check that this is considered usefull
> by someone.
> 
> 	Sam

Hmmm. If your source is located on a read-only file-system, you
can't modify it. You are therefore doomed to use only "known working"
distributions that are known to work with all possible module
combinations (these don't exist). So you get to compile the kernel
just as a test exercise or a gimmick like; "what did you do today?..."
answer; "I compiled the kernel..." This seems to not be very practical
since the purpose of compiling the kernel was to add something or
fix something. Now, its just to see if it compiles.

Different configurations are handled with different ".config"
files.

I think all you did was increase the compile time by writing
output files to different directories than the ones currently
in cache. There are a lot of negatives. It would be a shame for
you to waste a great deal of time on something that would not
be accepted into the distribution. Remember the earlier `make modules`
where the new objects went into a separate directory with sym-links?
That got changed. I think it got changed for good reasons.

Cheers,
Dick Johnson
Penguin : Linux version 2.4.18 on an i686 machine (797.90 BogoMips).
   Bush : The Fourth Reich of America

Re: [RFC/CFT] Separate obj/src dir

[Kai Germaschewski]

On Tue, 19 Nov 2002, Richard B. Johnson wrote:

> I think all you did was increase the compile time by writing
> output files to different directories than the ones currently
> in cache. There are a lot of negatives. It would be a shame for
> you to waste a great deal of time on something that would not
> be accepted into the distribution. Remember the earlier `make modules`
> where the new objects went into a separate directory with sym-links?
> That got changed. I think it got changed for good reasons.

Well, this isn't necessarily for everybody, if it's of no use for you, 
that's okay. Note that the patch doesn't (unless it has bugs ;) change the 
normal behavior at all. It really only adds a script to set up a separate 
objdir. If you don't want that, don't use that script ;)

--Kai

Re: [RFC/CFT] Separate obj/src dir

[Sam Ravnborg]

On Tue, Nov 19, 2002 at 03:46:28PM -0500, Richard B. Johnson wrote:
> Different configurations are handled with different ".config"
> files.
And different .config files results in different kernels.
Please note that .config files are also located in OBJTREE.

Cosider something like the following:

~/bk/linux-2.5
~/compile/arm	  <= Used to check that the kernel compiles for ARM
~/compile/allmod  <= My config with a lot of modules
~/compile/machine <= The config I use on my machine
~/compile/work    <= That's my sandbox

All configurations share the same src.
During the last copule of days we have seen some header file
clean-ups. It would have been nice if they tried compiling in
all four configurations listed above.
But if I switch .config then it is often a recompile of everything,
whereas the above setup can minimize it.
The header file cleanup is maybe not the best example because touching
a few key header files requires recompilation of everything anyway.

But my point is that there is a good use of different configurations
based on the same src.

Others that have asked for separate obj dir may step in here,
explaining why they thougt it was good.

	Sam

Re: [RFC/CFT] Separate obj/src dir

[David Woodhouse]

sam@ravnborg.org said:
>  Others that have asked for separate obj dir may step in here,
> explaining why they thougt it was good. 

Why? The only person arguing against it is RBJ, and people only ever read 
his mails for the amusement value.

Of course it's a useful feature.

--
dwmw2

Re: [RFC/CFT] Separate obj/src dir

[Miles Bader]

David Woodhouse <dwmw2@infradead.org> writes:
> Of course it's a useful feature.

Definitely.  I already use home-grown scripts to maintain multiple
object-trees, with the sources symlinked to a single source tree,
because I generally maintain several different platforms simultaneously.

Having one source tree makes things _much_ easier, as I don't have to
keep track of changes in a bunch of separate trees (it also saves a bit
of space, which is nice)!

-Miles
-- 
Suburbia: where they tear out the trees and then name streets after them.

Re: [RFC/CFT] Separate obj/src dir

[Brad Hards]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Wed, 20 Nov 2002 07:54, Sam Ravnborg wrote:
> But my point is that there is a good use of different configurations
> based on the same src.
I think that your example for testing is the most valid one.

In development, you normally have different source trees (hardlinked if you 
don't have a terabyte of hard drive space to spare), and use an editor that 
breaks hard-links (eg, emacs). You might as well build in the source 
directory, since you'll likely keep reworking it.

In release testing (aka release engineering, or more accurately: release 
fumbling-in-the-dark), you need to test a few different configurations. Sure, 
you could just build a set of symlink or hardlink trees, but it'd be very 
useful to be able to "make multiconfigs" and have a representative set of 
kernels built (either for later testing, or at least to ensure that the new 
kernel will build without modules, without networking, without IDE, and so 
on). Sure, it will take a while if you build everything, but that is why 
God^WTridge gave us ccache.

Brad

- -- 
http://linux.conf.au. 22-25Jan2003. Perth, Aust. I'm registered. Are you?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE92rNGW6pHgIdAuOMRAmcDAKCnzBOh6/6+zouOlM2bi1z2mcEmSACghEnp
MzXw5OsNmbZfzBMVGrFmNr0=
=7DkQ
-----END PGP SIGNATURE-----

Re: [RFC/CFT] Separate obj/src dir

[Daniel Jacobowitz]

On Tue, Nov 19, 2002 at 03:46:28PM -0500, Richard B. Johnson wrote:
> On Tue, 19 Nov 2002, Sam Ravnborg wrote:
> 
> > On Tue, Nov 19, 2002 at 03:22:45PM -0500, Richard B. Johnson wrote:
> > > I have a question; "What problem is this supposed to solve?"
> > 
> > Two problems (at least):
> > 
> > 1) You want to compile your kernel based on two different configurations,
> > but sharing the same src. No need to have a duplicate of all src.
> > - There are other ways to do this using symlinks
> > 
> > 2) You have the src located on a read-only filesystem.
> > I have been told this is the case for some SCM systems.
> > 
> > People has requested this feature at several occasions, and here
> > it is based on the current build system.
> > It's not ready for inclusion (obviously), and you shall
> > also see this as a way to check that this is considered usefull
> > by someone.
> > 
> > 	Sam
> 
> Hmmm. If your source is located on a read-only file-system, you
> can't modify it. You are therefore doomed to use only "known working"
> distributions that are known to work with all possible module
> combinations (these don't exist). So you get to compile the kernel
> just as a test exercise or a gimmick like; "what did you do today?..."
> answer; "I compiled the kernel..." This seems to not be very practical
> since the purpose of compiling the kernel was to add something or
> fix something. Now, its just to see if it compiles.
> 
> Different configurations are handled with different ".config"
> files.
> 
> I think all you did was increase the compile time by writing
> output files to different directories than the ones currently
> in cache. There are a lot of negatives. It would be a shame for
> you to waste a great deal of time on something that would not
> be accepted into the distribution. Remember the earlier `make modules`
> where the new objects went into a separate directory with sym-links?
> That got changed. I think it got changed for good reasons.

There are plenty of other good reasons for this.  Offhand:

- Speeds up grepping through the source tree for things if you don't
have to look at object files.  SCCS dirs still make this a pain, I may
hack grep -r to optionally skip them.

- Guarantees that doing a build should not affect anything in the srcdir;
this is handy when preparing release tarballs etc.

- Multiple builds from the same source (different architectures,
configs, compilers, whatever) without copying the source.

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer

Re: [RFC/CFT] Separate obj/src dir

[Geert Uytterhoeven]

On Tue, 19 Nov 2002, Daniel Jacobowitz wrote:
> On Tue, Nov 19, 2002 at 03:46:28PM -0500, Richard B. Johnson wrote:
> > On Tue, 19 Nov 2002, Sam Ravnborg wrote:
> > 
> > > On Tue, Nov 19, 2002 at 03:22:45PM -0500, Richard B. Johnson wrote:
> > > > I have a question; "What problem is this supposed to solve?"
> > > 
> > > Two problems (at least):
> > > 
> > > 1) You want to compile your kernel based on two different configurations,
> > > but sharing the same src. No need to have a duplicate of all src.
> > > - There are other ways to do this using symlinks
> > > 
> > > 2) You have the src located on a read-only filesystem.
> > > I have been told this is the case for some SCM systems.
> > > 
> > > People has requested this feature at several occasions, and here
> > > it is based on the current build system.
> > > It's not ready for inclusion (obviously), and you shall
> > > also see this as a way to check that this is considered usefull
> > > by someone.
> > > 
> > > 	Sam
> > 
> > Hmmm. If your source is located on a read-only file-system, you
> > can't modify it. You are therefore doomed to use only "known working"
> > distributions that are known to work with all possible module
> > combinations (these don't exist). So you get to compile the kernel
> > just as a test exercise or a gimmick like; "what did you do today?..."
> > answer; "I compiled the kernel..." This seems to not be very practical
> > since the purpose of compiling the kernel was to add something or
> > fix something. Now, its just to see if it compiles.
> > 
> > Different configurations are handled with different ".config"
> > files.
> > 
> > I think all you did was increase the compile time by writing
> > output files to different directories than the ones currently
> > in cache. There are a lot of negatives. It would be a shame for
> > you to waste a great deal of time on something that would not
> > be accepted into the distribution. Remember the earlier `make modules`
> > where the new objects went into a separate directory with sym-links?
> > That got changed. I think it got changed for good reasons.
> 
> There are plenty of other good reasons for this.  Offhand:
> 
> - Speeds up grepping through the source tree for things if you don't
> have to look at object files.  SCCS dirs still make this a pain, I may
> hack grep -r to optionally skip them.
> 
> - Guarantees that doing a build should not affect anything in the srcdir;
> this is handy when preparing release tarballs etc.

Or when you store multiple versions of the source tree using cp -rl, patch, and
same.  I do not want to accidentally touch any file in hardlinked trees, since
it would screw up the file in all trees.

Currently I use separate trees for compiling, which (A) consumes more disk
space, and (B) is much slower to maintain (e.g. diff knows about hard links).

If I can compile in a separate directory, I can directly use one of the
hardlinked trees as the source tree.

Gr{oetje,eeting}s,

						Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
							    -- Linus Torvalds

Re: [RFC/CFT] Separate obj/src dir

[Henning P. Schmiedehausen]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 733 bytes --]

"Richard B. Johnson" <root@chaos.analogic.com> writes:

>Hmmm. If your source is located on a read-only file-system, you
>can't modify it. You are therefore doomed to use only "known working"

You modify it by other means than open()/read()/write()/close()

	Regards
		Henning


-- 

"In einem Abwägungsprozess, wollen wir weiter regieren, hat sich die
SPD und die Bundesregierung und auch der Bundesfinanzminister fürs
Weiterregieren entschieden und gegen die Ehrlichkeit" -- Oswald
Metzger, Bündnis '90/Die Grünen, 12.11.2002

-- 
Henning Schmiedehausen     "Interpol und Deutsche Bank, FBI und
hps@intermeta.de            Scotland Yard, Flensburg und das BKA
henning@forge.franken.de    haben unsere Daten da." -- Kraftwerk, 1981

Re: [RFC/CFT] Separate obj/src dir

[Larry McVoy]

On Tue, Nov 19, 2002 at 03:22:45PM -0500, Richard B. Johnson wrote:
> On Tue, 19 Nov 2002, Sam Ravnborg wrote:
> 
> > Based on some initial work by Kai Germaschewski I have made a
> > working prototype of separate obj/src tree.
> > 
> > Usage example:
> > #src located in ~/bk/linux-2.5.sepobj
> > mkdir ~/compile/v2.5
> > cd ~/compile/v2.5
> > sh ../../kb/v2.5/kbuild
> 
> [SNIPPED...]
> 
> I have a question; "What problem is this supposed to solve?"
> This looks like a M$ism to me. Real source trees don't
> look like this. If you don't have write access to the source-
> code tree, you are screwed on a real project anyway. That's
> why we have CVS, tar and other tools to provide a local copy.

It can be really nice to maintain a bunch of different architectures at
the same time from the same tree.  It also makes it really easy to 
"clean" a tree.

On the other hand, I do wonder whether ccache could be used to get the
same effect.  Sam?
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: [RFC/CFT] Separate obj/src dir

[Sam Ravnborg]

On Tue, Nov 19, 2002 at 12:31:15PM -0800, Larry McVoy wrote:
> It can be really nice to maintain a bunch of different architectures at
> the same time from the same tree.  It also makes it really easy to 
> "clean" a tree.
> 
> On the other hand, I do wonder whether ccache could be used to get the
> same effect.  Sam?
I have never taken the time to look into ccache, so I dunno.

	Sam

Re: [RFC/CFT] Separate obj/src dir

[Kai Germaschewski]

On Tue, 19 Nov 2002, Larry McVoy wrote:

> On Tue, Nov 19, 2002 at 03:22:45PM -0500, Richard B. Johnson wrote:
> > On Tue, 19 Nov 2002, Sam Ravnborg wrote:
> > 
> > > Based on some initial work by Kai Germaschewski I have made a
> > > working prototype of separate obj/src tree.
> > > 
> > > Usage example:
> > > #src located in ~/bk/linux-2.5.sepobj
> > > mkdir ~/compile/v2.5
> > > cd ~/compile/v2.5
> > > sh ../../kb/v2.5/kbuild
> > 
> > [SNIPPED...]
> > 
> > I have a question; "What problem is this supposed to solve?"
> > This looks like a M$ism to me. Real source trees don't
> > look like this. If you don't have write access to the source-
> > code tree, you are screwed on a real project anyway. That's
> > why we have CVS, tar and other tools to provide a local copy.
> 
> It can be really nice to maintain a bunch of different architectures at
> the same time from the same tree.  It also makes it really easy to 
> "clean" a tree.
> 
> On the other hand, I do wonder whether ccache could be used to get the
> same effect.  Sam?

ccache helps a lot if you change your .config back to one previously 
compiled, but it still doesn't offer you the option to keep multiple 
.configs around at the same time.

Also, of course it doesn't help with keeping the source tree clean of 
non-source files ;)

Wrt the original patch, I like it, one preliminary comment is that I think
symlinks are nicer than copying. They are faster, shouldn't cause any
trouble on NFS, make uses "stat" and not "lstat", so it gets the
timestamps right, too. And if you edit a Makefile/Kconfig in the source
tree, you rather want that to take effect immediately, I guess ;)

--Kai

Re: [RFC/CFT] Separate obj/src dir

[Sam Ravnborg]

On Tue, Nov 19, 2002 at 02:48:09PM -0600, Kai Germaschewski wrote:
> Wrt the original patch, I like it, one preliminary comment is that I think
> symlinks are nicer than copying. They are faster, shouldn't cause any
> trouble on NFS, make uses "stat" and not "lstat", so it gets the
> timestamps right, too. And if you edit a Makefile/Kconfig in the source
> tree, you rather want that to take effect immediately, I guess ;)

Second try on the script.
Create symlinks as suggested, and optimised find a liitle.

	Sam

#!/bin/sh
#
# This script is used to build a kernel from a separate directory.
# The location of this script is assumed to be the root of
# the kernel tree.
# Usage:
# kernel src located in:
# ~/kernelsrc
# compile in:
# ~/compile
# cd ~/compile   <= Change to the directory where the compile shall take place
# ../kernelsrc/kbuild
#
# Arguments to kbuild is the same as used to make in the kernel build
# kbuild prints out SRCTREE and OBJTREE when started, and then makes a mirror
# of relevant files from the kernelsrc.

# files we do not care about in the kernel src
RCS_FIND_IGNORE="-name SCCS -o -name BitKeeper -o -name .svn -o -name CVS"

OBJTREE=$PWD
cd `dirname $0`
SRCTREE=$PWD
cd $OBJTREE
echo OBJTREE $OBJTREE
echo SRCTREE $SRCTREE
if [ "$SRCTREE" != "$OBJTREE" ]; then
  if [ -f $SRCTREE/.config -o -d $SRCTREE/include/asm ]; then
    echo '$SRCTREE contains generated files, please run "make mrproper" in the SRCTREE'
  else
    for a in `cd $SRCTREE; \
      find \( $RCS_FIND_IGNORE \) -prune -o -name Makefile\* -o -name Kconfig\* -o -name defconfig`; do
      if [ ! -d `dirname $a` ]; then
        mkdir -p $a
      fi
      ln -fs $SRCTREE/$a $a
    done

    ( echo "srctree := $SRCTREE";
      echo "objtree := $OBJTREE";
    ) > .tmp_make_config
    touch Rules.make
  make $*
  fi
else
  rm -f .tmp_make_config
  make $*
fi

Re: [RFC/CFT] Separate obj/src dir

[Henning P. Schmiedehausen]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 1333 bytes --]

"Richard B. Johnson" <root@chaos.analogic.com> writes:

>On Tue, 19 Nov 2002, Sam Ravnborg wrote:

>> Based on some initial work by Kai Germaschewski I have made a
>> working prototype of separate obj/src tree.
>> 
>> Usage example:
>> #src located in ~/bk/linux-2.5.sepobj
>> mkdir ~/compile/v2.5
>> cd ~/compile/v2.5
>> sh ../../kb/v2.5/kbuild

>[SNIPPED...]

>I have a question; "What problem is this supposed to solve?"
>This looks like a M$ism to me. Real source trees don't
>look like this. If you don't have write access to the source-
>code tree, you are screwed on a real project anyway. That's
>why we have CVS, tar and other tools to provide a local copy.

Having Trees read-only checked out? Having Trees on "pseudo filesystems"
backed by a SCM? 

This is the same thing we do in Java Land for ages. I personally like it
buy your taste may vary.

	Regards
		Henning

-- 

"In einem Abwägungsprozess, wollen wir weiter regieren, hat sich die
SPD und die Bundesregierung und auch der Bundesfinanzminister fürs
Weiterregieren entschieden und gegen die Ehrlichkeit" -- Oswald
Metzger, Bündnis '90/Die Grünen, 12.11.2002

-- 
Henning Schmiedehausen     "Interpol und Deutsche Bank, FBI und
hps@intermeta.de            Scotland Yard, Flensburg und das BKA
henning@forge.franken.de    haben unsere Daten da." -- Kraftwerk, 1981

Re: Separate obj/src dir

[Brian Jackson]

Sam Ravnborg writes: 

<snip>
> Another drawback is that when a .h file exist in the
> SRCTREE but not in the OBJTREE the generated dependencies
> will point out the .h file located in SRCTREE.
> This happens for generated .h files, and therefore a simple
> check is made in kbuild to check that the SRCTREE is
> cleaned/mrpropered.

I wonder how hard it would be to do this for other files types. It would be 
sort of handy to be able to copy a single file out of the source tree into 
the build tree, and have the build use the copy in the build tree. Example: 
you want to test a one liner in drivers/scsi/sd.c, you could just copy sd.c 
into the build tree, and make the change and test it out. That could be a 
huge space savings. That would help out those of us that are stuck with tiny 
hard drives in our laptops :) 

> 
> kconfig did not have an option to read the Kconfig files +
> defconfig from somewhere else than current directory,
> therefore they are copied. But that should be trivial to do.
> Possible solutions:
> 1) Command line option:
> 	-r path-to-rrot-of-tree
> 2) Deduct it from the argument given, but then kconfig
>    needs to know a bit too much about the kernel src tree.
> 3) Utilise the environment variable $(srctree), which is
>    anyway valid. 
> 
> Comments expected...

Excellent, I never really thought about it until I read this email, but it 
makes perfect sense(to me anyways). 

> 
> 	Sam 
> 
<snip>

Re: Separate obj/src dir

[Sam Ravnborg]

On Tue, Nov 19, 2002 at 02:51:54PM -0600, Brian Jackson wrote:
> Sam Ravnborg writes: 
> 
> I wonder how hard it would be to do this for other files types. It would be 
> sort of handy to be able to copy a single file out of the source tree into 
> the build tree, and have the build use the copy in the build tree. Example: 
> you want to test a one liner in drivers/scsi/sd.c, you could just copy sd.c 
> into the build tree, and make the change and test it out. That could be a 
> huge space savings. That would help out those of us that are stuck with 
> tiny hard drives in our laptops :) 
It actually works. But that is a side-effect and not something I had
planned.
I tried your example and kbuild uses the sd.c located in OBJTREE if
present. make searches lokal directory before VPATH directory.
And when sd.c is deleted the commandline changes and the original file
is compiled again.

But I can see several drawbacks with this, especially dealing with .h
files, or .c files included by other .c files.
So this is not a trustable feature.

	Sam

Re: Separate obj/src dir

[Simon Fowler]

[-- Attachment #1: Type: text/plain, Size: 1303 bytes --]

On Tue, Nov 19, 2002 at 02:51:54PM -0600, Brian Jackson wrote:
> Sam Ravnborg writes: 
> 
> <snip>
> >Another drawback is that when a .h file exist in the
> >SRCTREE but not in the OBJTREE the generated dependencies
> >will point out the .h file located in SRCTREE.
> >This happens for generated .h files, and therefore a simple
> >check is made in kbuild to check that the SRCTREE is
> >cleaned/mrpropered.
> 
> I wonder how hard it would be to do this for other files types. It would be 
> sort of handy to be able to copy a single file out of the source tree into 
> the build tree, and have the build use the copy in the build tree. Example: 
> you want to test a one liner in drivers/scsi/sd.c, you could just copy sd.c 
> into the build tree, and make the change and test it out. That could be a 
> huge space savings. That would help out those of us that are stuck with 
> tiny hard drives in our laptops :) 
> 
For that you probably want to use the hardlinked trees approach.
Just do a cp -al linux-2.5 scratch; then change your file and build
from the copy.

Simon

-- 
PGP public key Id 0x144A991C, or http://himi.org/stuff/himi.asc
(crappy) Homepage: http://himi.org
doe #237 (see http://www.lemuria.org/DeCSS) 
My DeCSS mirror: ftp://himi.org/pub/mirrors/css/ 

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Separate obj/src dir

[Pete Zaitcev]

> On Tue, Nov 19, 2002 at 02:51:54PM -0600, Brian Jackson wrote:
>> Sam Ravnborg writes:=20
>> the build tree, and have the build use the copy in the build tree. Exampl=
> e:=20
>> you want to test a one liner in drivers/scsi/sd.c, you could just copy sd=
> .c=20
>> into the build tree, and make the change and test it out. That could be a=
>=20
>> huge space savings. That would help out those of us that are stuck with=
>=20
>> tiny hard drives in our laptops :)=20
>>=20

> For that you probably want to use the hardlinked trees approach.
> Just do a cp -al linux-2.5 scratch; then change your file and build
> from the copy.
> 
> Simon

One word of caution to vi users: you WILL forget to unlink one day.
So, to protect me from myself, every time I untar a Linus' tree,
I immediately do:
 find linux-2.5.foo -type f | xargs chmod a-w
Then, the attached script reduces the typing. Run "deal dir/file.c"
before editing as if it was "bk edit dir/file.c".
Once you are done I always do:
 diff -urN -X dontdiff linux-2.5.48 linux-2.5.48-sparc > x.diff
This is a blazingly fast with hardlinked tree (cp -al).

-- Pete

#!/bin/sh
# This script dealiases a hardlinked file for editing.

set -e

if [ $# != 1 ]; then
  echo "Usage: deal file" >&2
  exit 1
fi

f=$1

mv $f $f~
cp $f~ $f
chmod u+w $f

Re: [RFC/CFT] Separate obj/src dir

[Alex Riesen]

On Tue, Nov 19, 2002 at 09:11:10PM +0100, Sam Ravnborg wrote:
> The kbuild shell script takes a verbatim copy of all Makefiles,
> all Kconfig files and all defconfigs. I did not even look into
> using symlinks, I was not sure how they work across NFS
> and the like.

But Kconfigs and defconfigs belong to the sources, don't they?

Suppose you have a main source tree and multiple objdirs with a
purpose to test different .configs. Now if you update the main
tree (including it's Kconfigs), objdirs are broken. The copies
of Kconfigs are obsoleted and maybe unrelated at all. Also makefiles.

Re: [RFC/CFT] Separate obj/src dir

[Alex Riesen]

On Wed, Nov 20, 2002 at 02:10:03PM +0100, Alex Riesen wrote:
> On Tue, Nov 19, 2002 at 09:11:10PM +0100, Sam Ravnborg wrote:
> > The kbuild shell script takes a verbatim copy of all Makefiles,
> > all Kconfig files and all defconfigs. I did not even look into
> > using symlinks, I was not sure how they work across NFS
> > and the like.
> 
> But Kconfigs and defconfigs belong to the sources, don't they?
> 
> Suppose you have a main source tree and multiple objdirs with a
> purpose to test different .configs. Now if you update the main
> tree (including it's Kconfigs), objdirs are broken. The copies
> of Kconfigs are obsoleted and maybe unrelated at all. Also makefiles.

Nevermind, found your script with symlinks.