[PATCH] Don't mkdir -p include/config in the sourcetree when using O=outputdir option

[PATCH] Don't mkdir -p include/config in the sourcetree when using O=outputdir option

[John Kacur]

>From ee63a191e50a33f6c47a00dc8f69b009a5406890 Mon Sep 17 00:00:00 2001
From: John Kacur <jkacur@redhat.com>
Date: Wed, 6 Jan 2010 16:18:02 +0100
Subject: [PATCH] Don't mkdir -p include/config in the sourcetree when using O=outputdir option

Problem: When using the option to put output files in a separate directory, make oldconfig (or related steps like silentoldconfig) create the include/config dierectory in the source tree. This causes the prepare3 check to fail, requiring the user to run make mrproper in the source tree to continue with the make.

Solution: Add the $(objtree) directory to the %config target.

The following is an example of how the Makefile currently can fail.
[jkacur@tycho rt.linux.git]$ make mrproper
[jkacur@tycho rt.linux.git]$ mkdir /bld/2.6.33-rc3
[jkacur@tycho rt.linux.git]$ cp /bld/config /bld/2.6.33-rc3/.config
[jkacur@tycho rt.linux.git]$ make O=/bld/2.6.33-rc3/ oldconfig
  HOSTCC  scripts/basic/fixdep
  HOSTCC  scripts/basic/docproc
  HOSTCC  scripts/basic/hash
  GEN     /bld/2.6.33-rc3/Makefile
  HOSTCC  scripts/kconfig/conf.o
  HOSTCC  scripts/kconfig/kxgettext.o
  HOSTCC  scripts/kconfig/zconf.tab.o
  HOSTLD  scripts/kconfig/conf
scripts/kconfig/conf -o arch/x86/Kconfig
[jkacur@tycho rt.linux.git]$ make O=/bld/2.6.33-rc3/
  GEN     /bld/2.6.33-rc3/Makefile
scripts/kconfig/conf -s arch/x86/Kconfig
  Using /home/jkacur/rt.linux.git as source for kernel
  /home/jkacur/rt.linux.git is not clean, please run 'make mrproper'
  in the '/home/jkacur/rt.linux.git' directory.
make[1]: *** [prepare3] Error 1
make: *** [sub-make] Error 2
[jkacur@tycho rt.linux.git]$ ls -ld include/config/
drwxrwxr-x 2 jkacur jkacur 6 2010-01-06 15:58 include/config/

Signed-off-by: John Kacur <jkacur@redhat.com>
---
 Makefile |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/Makefile b/Makefile
index 1231094..7d8d722 100644
--- a/Makefile
+++ b/Makefile
@@ -458,7 +458,7 @@ config: scripts_basic outputmakefile FORCE
 	$(Q)$(MAKE) $(build)=scripts/kconfig $@
 
 %config: scripts_basic outputmakefile FORCE
-	$(Q)mkdir -p include/linux include/config
+	$(Q)mkdir -p $(objtree)/include/linux include/config
 	$(Q)$(MAKE) $(build)=scripts/kconfig $@
 
 else
-- 
1.6.0.6

Re: [PATCH] Don't mkdir -p include/config in the sourcetree when using O=outputdir option

[Michal Marek]

John Kacur napsal(a):
> The following is an example of how the Makefile currently can fail.
> [jkacur@tycho rt.linux.git]$ make mrproper
> [jkacur@tycho rt.linux.git]$ mkdir /bld/2.6.33-rc3
> [jkacur@tycho rt.linux.git]$ cp /bld/config /bld/2.6.33-rc3/.config
> [jkacur@tycho rt.linux.git]$ make O=/bld/2.6.33-rc3/ oldconfig
>   HOSTCC  scripts/basic/fixdep
>   HOSTCC  scripts/basic/docproc
>   HOSTCC  scripts/basic/hash
>   GEN     /bld/2.6.33-rc3/Makefile
>   HOSTCC  scripts/kconfig/conf.o
>   HOSTCC  scripts/kconfig/kxgettext.o
>   HOSTCC  scripts/kconfig/zconf.tab.o
>   HOSTLD  scripts/kconfig/conf
> scripts/kconfig/conf -o arch/x86/Kconfig
> [jkacur@tycho rt.linux.git]$ make O=/bld/2.6.33-rc3/
>   GEN     /bld/2.6.33-rc3/Makefile
> scripts/kconfig/conf -s arch/x86/Kconfig
>   Using /home/jkacur/rt.linux.git as source for kernel
>   /home/jkacur/rt.linux.git is not clean, please run 'make mrproper'
>   in the '/home/jkacur/rt.linux.git' directory.
> make[1]: *** [prepare3] Error 1
> make: *** [sub-make] Error 2
> [jkacur@tycho rt.linux.git]$ ls -ld include/config/
> drwxrwxr-x 2 jkacur jkacur 6 2010-01-06 15:58 include/config/
> 
> Signed-off-by: John Kacur <jkacur@redhat.com>
> ---
>  Makefile |    2 +-
>  1 files changed, 1 insertions(+), 1 deletions(-)
> 
> diff --git a/Makefile b/Makefile
> index 1231094..7d8d722 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -458,7 +458,7 @@ config: scripts_basic outputmakefile FORCE
>  	$(Q)$(MAKE) $(build)=scripts/kconfig $@
>  
>  %config: scripts_basic outputmakefile FORCE
> -	$(Q)mkdir -p include/linux include/config
> +	$(Q)mkdir -p $(objtree)/include/linux include/config
>  	$(Q)$(MAKE) $(build)=scripts/kconfig $@

Can you double-check that this patch really fixes your issue and that
without this patch it fails again? Because 1) I can't reproduce this
behaviour and I also don't see how it could f, 2) your problem is a
include/config file somehow created in the source tree, but you patch
the creation of include/linux. And if there really needs to be a fix,
the same should be applied to the identical block above.

thanks,
Michal

Re: [PATCH] Don't mkdir -p include/config in the sourcetree when using O=outputdir option

[John Kacur]

On Wed, Jan 6, 2010 at 8:34 PM, Michal Marek <mmarek@suse.cz> wrote:
> John Kacur napsal(a):
>> The following is an example of how the Makefile currently can fail.
>> [jkacur@tycho rt.linux.git]$ make mrproper
>> [jkacur@tycho rt.linux.git]$ mkdir /bld/2.6.33-rc3
>> [jkacur@tycho rt.linux.git]$ cp /bld/config /bld/2.6.33-rc3/.config
>> [jkacur@tycho rt.linux.git]$ make O=/bld/2.6.33-rc3/ oldconfig
>>   HOSTCC  scripts/basic/fixdep
>>   HOSTCC  scripts/basic/docproc
>>   HOSTCC  scripts/basic/hash
>>   GEN     /bld/2.6.33-rc3/Makefile
>>   HOSTCC  scripts/kconfig/conf.o
>>   HOSTCC  scripts/kconfig/kxgettext.o
>>   HOSTCC  scripts/kconfig/zconf.tab.o
>>   HOSTLD  scripts/kconfig/conf
>> scripts/kconfig/conf -o arch/x86/Kconfig
>> [jkacur@tycho rt.linux.git]$ make O=/bld/2.6.33-rc3/
>>   GEN     /bld/2.6.33-rc3/Makefile
>> scripts/kconfig/conf -s arch/x86/Kconfig
>>   Using /home/jkacur/rt.linux.git as source for kernel
>>   /home/jkacur/rt.linux.git is not clean, please run 'make mrproper'
>>   in the '/home/jkacur/rt.linux.git' directory.
>> make[1]: *** [prepare3] Error 1
>> make: *** [sub-make] Error 2
>> [jkacur@tycho rt.linux.git]$ ls -ld include/config/
>> drwxrwxr-x 2 jkacur jkacur 6 2010-01-06 15:58 include/config/
>>
>> Signed-off-by: John Kacur <jkacur@redhat.com>
>> ---
>>  Makefile |    2 +-
>>  1 files changed, 1 insertions(+), 1 deletions(-)
>>
>> diff --git a/Makefile b/Makefile
>> index 1231094..7d8d722 100644
>> --- a/Makefile
>> +++ b/Makefile
>> @@ -458,7 +458,7 @@ config: scripts_basic outputmakefile FORCE
>>       $(Q)$(MAKE) $(build)=scripts/kconfig $@
>>
>>  %config: scripts_basic outputmakefile FORCE
>> -     $(Q)mkdir -p include/linux include/config
>> +     $(Q)mkdir -p $(objtree)/include/linux include/config
>>       $(Q)$(MAKE) $(build)=scripts/kconfig $@
>
> Can you double-check that this patch really fixes your issue and that
> without this patch it fails again? Because 1) I can't reproduce this
> behaviour and I also don't see how it could f, 2) your problem is a
> include/config file somehow created in the source tree, but you patch
> the creation of include/linux. And if there really needs to be a fix,
> the same should be applied to the identical block above.
>
> thanks,

Hi Michal

Yes, I doubled checked that the problem exists, and it is quite easy for me to
reproduce. However, my proposed solution is also broken, please don't
apply, I'm still investigating.

I am surprised that you are not able to reproduce the problem though.
Did you follow the steps I that I outlined?

Thanks, John

Re: [PATCH] Don't mkdir -p include/config in the sourcetree when using O=outputdir option

[Michal Marek]

John Kacur napsal(a):
> Yes, I doubled checked that the problem exists, and it is quite easy for me to
> reproduce. However, my proposed solution is also broken, please don't
> apply, I'm still investigating.
> 
> I am surprised that you are not able to reproduce the problem though.
> Did you follow the steps I that I outlined?

I did

$ git checkout v2.6.33-rc3
$ mkdir /bld/2.6.33-rc3
$ zcat /proc/config.gz >/bld/2.6.33-rc3/.config
$ make O=/bld/2.6.33-rc3/ oldconfig
...
hit Enter several times
...
$ ls -ld include/config /bld/2.6.33-rc3/include/config/
ls: cannot access include/config: No such file or directory
drwxr-xr-x 2 mmarek users 4096 2010-01-06 21:35
/bld/2.6.33-rc3/include/config//
$ make O=/bld/2.6.33-rc3/
  GEN     /bld/2.6.33-rc3/Makefile
scripts/kconfig/conf -s arch/x86/Kconfig
  Using /home/mmarek/linux-2.6 as source for kernel
  GEN     /bld/2.6.33-rc3/Makefile
  CHK     include/linux/version.h
  UPD     include/linux/version.h
  CHK     include/generated/utsrelease.h
  UPD     include/generated/utsrelease.h
  CC      kernel/bounds.s
... continues building ...


The mkdir command has been in the Makefile since 2006 (c955cca), so I
think that your problem is caused by something else. Try a make V=1
build with a read-only source tree to see which mkdir command fails.

Michal

Re: [PATCH] Don't mkdir -p include/config in the sourcetree when using O=outputdir option

[John Kacur]

On Wed, Jan 6, 2010 at 9:45 PM, Michal Marek <mmarek@suse.cz> wrote:
> John Kacur napsal(a):
>> Yes, I doubled checked that the problem exists, and it is quite easy for me to
>> reproduce. However, my proposed solution is also broken, please don't
>> apply, I'm still investigating.
>>
>> I am surprised that you are not able to reproduce the problem though.
>> Did you follow the steps I that I outlined?
>
> I did
>
> $ git checkout v2.6.33-rc3
> $ mkdir /bld/2.6.33-rc3
> $ zcat /proc/config.gz >/bld/2.6.33-rc3/.config
> $ make O=/bld/2.6.33-rc3/ oldconfig
> ...
> hit Enter several times
> ...
> $ ls -ld include/config /bld/2.6.33-rc3/include/config/
> ls: cannot access include/config: No such file or directory
> drwxr-xr-x 2 mmarek users 4096 2010-01-06 21:35
> /bld/2.6.33-rc3/include/config//
> $ make O=/bld/2.6.33-rc3/
>  GEN     /bld/2.6.33-rc3/Makefile
> scripts/kconfig/conf -s arch/x86/Kconfig
>  Using /home/mmarek/linux-2.6 as source for kernel
>  GEN     /bld/2.6.33-rc3/Makefile
>  CHK     include/linux/version.h
>  UPD     include/linux/version.h
>  CHK     include/generated/utsrelease.h
>  UPD     include/generated/utsrelease.h
>  CC      kernel/bounds.s
> ... continues building ...
>
>
> The mkdir command has been in the Makefile since 2006 (c955cca), so I
> think that your problem is caused by something else. Try a make V=1
> build with a read-only source tree to see which mkdir command fails.
>

Michal: The length of time that a bug exists in a piece of code is no
argument for
it's correctness!

I tried to debug this using
make -d V=1 O=/bld/2.6.33-rc3
and discovered that the bug did not always occur, but did sometimes.
This leads me to think that it is timing sensitive, and indeed the
problem shows up
more easily on a machine with multiple processors.

Is it possible that the step with mkdir -p can sometimes occur in a
Makefile parallel process
before the cd KBUILD_OUTPUT occurs? That would explain why it doesn't
reliably reproduce.

In any case, the idea behind my original patch is good, just specify
the full path
using the objtree variable and the problem goes away. The first round of my
implementation was simply lacking.

Following up this note with a patch.

John

Re: [PATCH] Don't mkdir -p include/config in the sourcetree when using O=outputdir option

[John Kacur]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 3580 bytes --]



On Wed, 6 Jan 2010, John Kacur wrote:

> On Wed, Jan 6, 2010 at 9:45 PM, Michal Marek <mmarek@suse.cz> wrote:
> > John Kacur napsal(a):
> >> Yes, I doubled checked that the problem exists, and it is quite easy for me to
> >> reproduce. However, my proposed solution is also broken, please don't
> >> apply, I'm still investigating.
> >>
> >> I am surprised that you are not able to reproduce the problem though.
> >> Did you follow the steps I that I outlined?
> >
> > I did
> >
> > $ git checkout v2.6.33-rc3
> > $ mkdir /bld/2.6.33-rc3
> > $ zcat /proc/config.gz >/bld/2.6.33-rc3/.config
> > $ make O=/bld/2.6.33-rc3/ oldconfig
> > ...
> > hit Enter several times
> > ...
> > $ ls -ld include/config /bld/2.6.33-rc3/include/config/
> > ls: cannot access include/config: No such file or directory
> > drwxr-xr-x 2 mmarek users 4096 2010-01-06 21:35
> > /bld/2.6.33-rc3/include/config//
> > $ make O=/bld/2.6.33-rc3/
> >  GEN     /bld/2.6.33-rc3/Makefile
> > scripts/kconfig/conf -s arch/x86/Kconfig
> >  Using /home/mmarek/linux-2.6 as source for kernel
> >  GEN     /bld/2.6.33-rc3/Makefile
> >  CHK     include/linux/version.h
> >  UPD     include/linux/version.h
> >  CHK     include/generated/utsrelease.h
> >  UPD     include/generated/utsrelease.h
> >  CC      kernel/bounds.s
> > ... continues building ...
> >
> >
> > The mkdir command has been in the Makefile since 2006 (c955cca), so I
> > think that your problem is caused by something else. Try a make V=1
> > build with a read-only source tree to see which mkdir command fails.
> >
> 
> Michal: The length of time that a bug exists in a piece of code is no
> argument for
> it's correctness!
> 
> I tried to debug this using
> make -d V=1 O=/bld/2.6.33-rc3
> and discovered that the bug did not always occur, but did sometimes.
> This leads me to think that it is timing sensitive, and indeed the
> problem shows up
> more easily on a machine with multiple processors.
> 
> Is it possible that the step with mkdir -p can sometimes occur in a
> Makefile parallel process
> before the cd KBUILD_OUTPUT occurs? That would explain why it doesn't
> reliably reproduce.
> 
> In any case, the idea behind my original patch is good, just specify
> the full path
> using the objtree variable and the problem goes away. The first round of my
> implementation was simply lacking.
> 
> Following up this note with a patch.
> 
> John
> 

>From b9db45e49b36e589de410f1c34fe77087fa36796 Mon Sep 17 00:00:00 2001
From: John Kacur <jkacur@redhat.com>
Date: Wed, 6 Jan 2010 23:16:38 +0100
Subject: [PATCH] Kbuild: Use $(objtree) for mkdir -p include/linux include/config

This fixes a problem where make sometimes creates the directories
include/linux include/config
in the local source tree, even though O=/outputdir was specified.

The solution is to give an absolute path using $(objtree)

Signed-off-by: John Kacur <jkacur@redhat.com>
---
 Makefile |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/Makefile b/Makefile
index 1231094..2f3ba92 100644
--- a/Makefile
+++ b/Makefile
@@ -454,11 +454,11 @@ include $(srctree)/arch/$(SRCARCH)/Makefile
 export KBUILD_DEFCONFIG KBUILD_KCONFIG
 
 config: scripts_basic outputmakefile FORCE
-	$(Q)mkdir -p include/linux include/config
+	$(Q)mkdir -p $(objtree)/include/linux $(objtree)/include/config
 	$(Q)$(MAKE) $(build)=scripts/kconfig $@
 
 %config: scripts_basic outputmakefile FORCE
-	$(Q)mkdir -p include/linux include/config
+	$(Q)mkdir -p $(objtree)/include/linux $(objtree)/include/config
 	$(Q)$(MAKE) $(build)=scripts/kconfig $@
 
 else
-- 
1.6.0.6

Re: [PATCH] Don't mkdir -p include/config in the sourcetree when using O=outputdir option

[Michal Marek]

On 6.1.2010 23:30, John Kacur wrote:
> On Wed, Jan 6, 2010 at 9:45 PM, Michal Marek <mmarek@suse.cz> wrote:
> Michal: The length of time that a bug exists in a piece of code is no
> argument for
> it's correctness!

>From your original post I though it fails reliably for you. If it's a
race, then you're right, the time it was introduced is irrelevant :-).


> I tried to debug this using
> make -d V=1 O=/bld/2.6.33-rc3
> and discovered that the bug did not always occur, but did sometimes.
> This leads me to think that it is timing sensitive, and indeed the
> problem shows up
> more easily on a machine with multiple processors.
> 
> Is it possible that the step with mkdir -p can sometimes occur in a
> Makefile parallel process
> before the cd KBUILD_OUTPUT occurs? That would explain why it doesn't
> reliably reproduce.

OK, I still don't see how it fails, but adding $(objtree) there is a
valid change and if it fixes your problem, let's apply it.

Michal