Re: Odd build breakage in 4.9-rc7

Re: Odd build breakage in 4.9-rc7

[Prarit Bhargava]

On 11/30/2016 01:18 PM, Linus Torvalds wrote:
> On Wed, Nov 30, 2016 at 9:24 AM, Jarod Wilson <jarod@redhat.com> wrote:
>>
>> Now, if I omit the -j8 and do a single-threaded build, then things work
>> fine. Prarit bisected this failure to commit
>> 9a6fb28a355d2609ace4dab4e6425442c647894d, and indeed, when reverting that
>> patch and the two that follow it from rc7, parallel make works again.
> 
> I seriously doubt that commit really makes a difference, and I think
> it was just random luck.
> 
> Do you perhaps have ccache installed?
> 
> Because ccache at some point broke dependency generation of "gcc -MM"
> that the kernel build system uses, giving those random "No such file"
> build errors.
> 
> Try uninstalling ccache and see if that helps.
> 

I reported this last week on LKML and I'm building on a freshly installed
systems that do not have ccache installed.

For example,

[root@intel-brickland-04 linux]# rpm -q ccache
package ccache is not installed
[root@intel-brickland-04 linux]# which ccache
/usr/bin/which: no ccache in
(/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/libexec/git-core:/root/bin)

In my case I tracked this to commit 3637efb00864 ("x86/mce: Add PCI quirks to
identify Xeons with machine check recovery") which adds the include for
generated/autoksyms.h.

Searching LKML and I came across a report from Ken Moffat from a month ago:

http://marc.info/?l=linux-kernel&m=147794681124332&w=2

Also cc'ing linux-kbuild.

P.

>               Linus
> 

Re: Odd build breakage in 4.9-rc7

[Linus Torvalds]

On Wed, Nov 30, 2016 at 10:28 AM, Prarit Bhargava <prarit@redhat.com> wrote:
]>
> In my case I tracked this to commit 3637efb00864 ("x86/mce: Add PCI quirks to
> identify Xeons with machine check recovery") which adds the include for
> generated/autoksyms.h.

Ok, that at least makes some sense. The other blamed commit did not
seem to possibly make a difference.

> Searching LKML and I came across a report from Ken Moffat from a month ago:
>
> http://marc.info/?l=linux-kernel&m=147794681124332&w=2

Does a "make clean" get rid of it forever? Or does it come back?

If it's a one-time dependency issue that is because some header
dependency addition that the automatic dependency generator hadn't
caught, that might explain a bisection failure too: once the file
happens to get rebuilt (and the dependencies re-done), it starts
working even though the "happens to be rebuilt" had nothing to do with
the original bug.

               Linus

Re: Odd build breakage in 4.9-rc7

[Prarit Bhargava]

On 11/30/2016 01:36 PM, Linus Torvalds wrote:
> On Wed, Nov 30, 2016 at 10:28 AM, Prarit Bhargava <prarit@redhat.com> wrote:
> ]>
>> In my case I tracked this to commit 3637efb00864 ("x86/mce: Add PCI quirks to
>> identify Xeons with machine check recovery") which adds the include for
>> generated/autoksyms.h.
> 
> Ok, that at least makes some sense. The other blamed commit did not
> seem to possibly make a difference.
> 
>> Searching LKML and I came across a report from Ken Moffat from a month ago:
>>
>> http://marc.info/?l=linux-kernel&m=147794681124332&w=2
> 
> Does a "make clean" get rid of it forever? Or does it come back?

It comes back.  The steps to reproduce this are:

1.  checkout latest linux.git
2.  make -j112

(IOW, it occurs 100% of the time for me on a clean tree.)

To work around the bug I have to do

1.  checkout latest linux.git
2.  comment out the include for generated/autoksyms.h at include/linux/export.h:81
3.  compile with -j112

This fails loudly, but then I do

4.  uncomment the include for generated/autoksyms.h at include/linux/export.h:81
5.  make -j112

and this completes with a bootable kernel AFAICT.

> 
> If it's a one-time dependency issue that is because some header
> dependency addition that the automatic dependency generator hadn't
> caught, that might explain a bisection failure too: once the file
> happens to get rebuilt (and the dependencies re-done), it starts
> working even though the "happens to be rebuilt" had nothing to do with
> the original bug.

Hopefully the linux-kbuild folks might be able to point us in the right
direction for a fix.

P.

Re: Odd build breakage in 4.9-rc7

[Jarod Wilson]

On 2016-11-30 1:50 PM, Prarit Bhargava wrote:
>
> On 11/30/2016 01:36 PM, Linus Torvalds wrote:
>> On Wed, Nov 30, 2016 at 10:28 AM, Prarit Bhargava <prarit@redhat.com> wrote:
>> ]>
>>> In my case I tracked this to commit 3637efb00864 ("x86/mce: Add PCI quirks to
>>> identify Xeons with machine check recovery") which adds the include for
>>> generated/autoksyms.h.
>>
>> Ok, that at least makes some sense. The other blamed commit did not
>> seem to possibly make a difference.
>>
>>> Searching LKML and I came across a report from Ken Moffat from a month ago:
>>>
>>> http://marc.info/?l=linux-kernel&m=147794681124332&w=2
>>
>> Does a "make clean" get rid of it forever? Or does it come back?
>
> It comes back.  The steps to reproduce this are:
>
> 1.  checkout latest linux.git
> 2.  make -j112
>
> (IOW, it occurs 100% of the time for me on a clean tree.)
>
> To work around the bug I have to do
>
> 1.  checkout latest linux.git
> 2.  comment out the include for generated/autoksyms.h at include/linux/export.h:81
> 3.  compile with -j112
>
> This fails loudly, but then I do
>
> 4.  uncomment the include for generated/autoksyms.h at include/linux/export.h:81
> 5.  make -j112
>
> and this completes with a bootable kernel AFAICT.

In my case, I first noticed this with rpm builds, which are unpacking a 
tarball every time. I did have ccache installed, but have removed it, to 
no avail. I still get this failure. Reverting these three patches...

9a6fb28a355d2609ace4dab4e6425442c647894d
3637efb00864f465baebd49464e58319fd295b65
ffb173e657fa8123bffa2a169e124b4bca0b5bc4

...gets me a working build every time. So I'm not so sure these patches 
are completely innocent. They may not be directly at fault, but 
certainly seem to be involved in causing things to get tripped up.

>> If it's a one-time dependency issue that is because some header
>> dependency addition that the automatic dependency generator hadn't
>> caught, that might explain a bisection failure too: once the file
>> happens to get rebuilt (and the dependencies re-done), it starts
>> working even though the "happens to be rebuilt" had nothing to do with
>> the original bug.
>
> Hopefully the linux-kbuild folks might be able to point us in the right
> direction for a fix.

Indeed. I'm mostly clueless here. Well, mostly clueless, period, but 
even more so here. And with the CONFIG_MODVERSIONS part of my original 
mail, which I guess I should also re-test with ccache uninstalled...

-- 
Jarod Wilson
jarod@redhat.com

Re: Odd build breakage in 4.9-rc7

[Linus Torvalds]

On Wed, Nov 30, 2016 at 10:50 AM, Prarit Bhargava <prarit@redhat.com> wrote:
>
> It comes back.  The steps to reproduce this are:
>
> 1.  checkout latest linux.git
> 2.  make -j112
>
> (IOW, it occurs 100% of the time for me on a clean tree.)

I suspect it's not new, it's just that you are able to hit the timing
just right (and the new include presumable makes that just be much
easier).

The rules for generating include/generated/autoksyms.h aren't new,
they go back to at least 4.7.

Adding Nico to the cc, since he's the person to blame for autokeysyms,
and might have some idea why the dependency wouldn't be done in time.

              Linus

Re: Odd build breakage in 4.9-rc7

[Nicolas Pitre]

On Wed, 30 Nov 2016, Linus Torvalds wrote:

> On Wed, Nov 30, 2016 at 10:50 AM, Prarit Bhargava <prarit@redhat.com> wrote:
> >
> > It comes back.  The steps to reproduce this are:
> >
> > 1.  checkout latest linux.git
> > 2.  make -j112
> >
> > (IOW, it occurs 100% of the time for me on a clean tree.)

I don't have access to such hardware where -j112 could ever make sense.  :-)
In other words, I can't reproduce regardless of the -j value I try.

> I suspect it's not new, it's just that you are able to hit the timing
> just right (and the new include presumable makes that just be much
> easier).

Here's the best fix I can think of. I can't convince myself any other 
location would be 100% safe.  Obviously I can't confirm if this actually 
fixes anything.

----- >8
Subject: kbuild: make sure autoksyms.h exists early

Some people are able to trigger a race where autoksyms.h is used before
its empty version is even created. Let's create it at the same time as
the directory holding it is created.

Signed-off-by: Nicolas Pitre <nico@linaro.org>

diff --git a/Makefile b/Makefile
index 694111b43c..9f9c3b577c 100644
--- a/Makefile
+++ b/Makefile
@@ -1019,8 +1019,6 @@ prepare2: prepare3 prepare-compiler-check outputmakefile asm-generic
 prepare1: prepare2 $(version_h) include/generated/utsrelease.h \
                    include/config/auto.conf
 	$(cmd_crmodverdir)
-	$(Q)test -e include/generated/autoksyms.h || \
-	    touch   include/generated/autoksyms.h
 
 archprepare: archheaders archscripts prepare1 scripts_basic
 
diff --git a/scripts/kconfig/Makefile b/scripts/kconfig/Makefile
index ebced77deb..90a091b6ae 100644
--- a/scripts/kconfig/Makefile
+++ b/scripts/kconfig/Makefile
@@ -35,6 +35,8 @@ nconfig: $(obj)/nconf
 
 silentoldconfig: $(obj)/conf
 	$(Q)mkdir -p include/config include/generated
+	$(Q)test -e include/generated/autoksyms.h || \
+	    touch   include/generated/autoksyms.h
 	$< $(silent) --$@ $(Kconfig)
 
 localyesconfig localmodconfig: $(obj)/streamline_config.pl $(obj)/conf

Re: Odd build breakage in 4.9-rc7

[Prarit Bhargava]

On 11/30/2016 05:41 PM, Nicolas Pitre wrote:
> On Wed, 30 Nov 2016, Linus Torvalds wrote:
> 
>> On Wed, Nov 30, 2016 at 10:50 AM, Prarit Bhargava <prarit@redhat.com> wrote:
>>>
>>> It comes back.  The steps to reproduce this are:
>>>
>>> 1.  checkout latest linux.git
>>> 2.  make -j112
>>>
>>> (IOW, it occurs 100% of the time for me on a clean tree.)
> 
> I don't have access to such hardware where -j112 could ever make sense.  :-)

:)  I could push the builds onto the -j256 but I'm doing other stuff over there. :)

> In other words, I can't reproduce regardless of the -j value I try.
> 
>> I suspect it's not new, it's just that you are able to hit the timing
>> just right (and the new include presumable makes that just be much
>> easier).
> 
> Here's the best fix I can think of. I can't convince myself any other 
> location would be 100% safe.  Obviously I can't confirm if this actually 
> fixes anything.
> 
> ----- >8
> Subject: kbuild: make sure autoksyms.h exists early
> 

I'm building with this patch on top of latest now.  I will put it in a tight
loop and clear the drop_caches between builds to see if I can make it fail.

Thanks Nicolas -- your help is very much appreciated.

P.

Re: Odd build breakage in 4.9-rc7

[Prarit Bhargava]

On 11/30/2016 05:41 PM, Nicolas Pitre wrote:
> On Wed, 30 Nov 2016, Linus Torvalds wrote:
> 
>> On Wed, Nov 30, 2016 at 10:50 AM, Prarit Bhargava <prarit@redhat.com> wrote:
>>>
>>> It comes back.  The steps to reproduce this are:
>>>
>>> 1.  checkout latest linux.git
>>> 2.  make -j112
>>>
>>> (IOW, it occurs 100% of the time for me on a clean tree.)
> 
> I don't have access to such hardware where -j112 could ever make sense.  :-)
> In other words, I can't reproduce regardless of the -j value I try.
> 
>> I suspect it's not new, it's just that you are able to hit the timing
>> just right (and the new include presumable makes that just be much
>> easier).
> 
> Here's the best fix I can think of. I can't convince myself any other 
> location would be 100% safe.  Obviously I can't confirm if this actually 
> fixes anything.
> 
> ----- >8
> Subject: kbuild: make sure autoksyms.h exists early
> 
> Some people are able to trigger a race where autoksyms.h is used before
> its empty version is even created. Let's create it at the same time as
> the directory holding it is created.
> 
> Signed-off-by: Nicolas Pitre <nico@linaro.org>
> 
> diff --git a/Makefile b/Makefile
> index 694111b43c..9f9c3b577c 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -1019,8 +1019,6 @@ prepare2: prepare3 prepare-compiler-check outputmakefile asm-generic
>  prepare1: prepare2 $(version_h) include/generated/utsrelease.h \
>                     include/config/auto.conf
>  	$(cmd_crmodverdir)
> -	$(Q)test -e include/generated/autoksyms.h || \
> -	    touch   include/generated/autoksyms.h
>  
>  archprepare: archheaders archscripts prepare1 scripts_basic
>  
> diff --git a/scripts/kconfig/Makefile b/scripts/kconfig/Makefile
> index ebced77deb..90a091b6ae 100644
> --- a/scripts/kconfig/Makefile
> +++ b/scripts/kconfig/Makefile
> @@ -35,6 +35,8 @@ nconfig: $(obj)/nconf
>  
>  silentoldconfig: $(obj)/conf
>  	$(Q)mkdir -p include/config include/generated
> +	$(Q)test -e include/generated/autoksyms.h || \
> +	    touch   include/generated/autoksyms.h
>  	$< $(silent) --$@ $(Kconfig)
>  
>  localyesconfig localmodconfig: $(obj)/streamline_config.pl $(obj)/conf
> 

The testing was successful.

After testing an hour of builds with different -j values, I'm no longer seeing
any compile issues when this patch is applied.  When I remove the patch the
compile error returns so I'm going to say that this patch fixed it.

Thanks again Nicolas.

Tested-by: Prarit Bhargava <prarit@redhat.com>

P.

Re: Odd build breakage in 4.9-rc7

[Jarod Wilson]

On 2016-12-01 9:03 AM, Prarit Bhargava wrote:
>
>
> On 11/30/2016 05:41 PM, Nicolas Pitre wrote:
>> On Wed, 30 Nov 2016, Linus Torvalds wrote:
>>
>>> On Wed, Nov 30, 2016 at 10:50 AM, Prarit Bhargava <prarit@redhat.com> wrote:
>>>>
>>>> It comes back.  The steps to reproduce this are:
>>>>
>>>> 1.  checkout latest linux.git
>>>> 2.  make -j112
>>>>
>>>> (IOW, it occurs 100% of the time for me on a clean tree.)
>>
>> I don't have access to such hardware where -j112 could ever make sense.  :-)
>> In other words, I can't reproduce regardless of the -j value I try.
>>
>>> I suspect it's not new, it's just that you are able to hit the timing
>>> just right (and the new include presumable makes that just be much
>>> easier).
>>
>> Here's the best fix I can think of. I can't convince myself any other
>> location would be 100% safe.  Obviously I can't confirm if this actually
>> fixes anything.
>>
>> ----- >8
>> Subject: kbuild: make sure autoksyms.h exists early
>>
>> Some people are able to trigger a race where autoksyms.h is used before
>> its empty version is even created. Let's create it at the same time as
>> the directory holding it is created.
>>
>> Signed-off-by: Nicolas Pitre <nico@linaro.org>
>>
>> diff --git a/Makefile b/Makefile
>> index 694111b43c..9f9c3b577c 100644
>> --- a/Makefile
>> +++ b/Makefile
>> @@ -1019,8 +1019,6 @@ prepare2: prepare3 prepare-compiler-check outputmakefile asm-generic
>>  prepare1: prepare2 $(version_h) include/generated/utsrelease.h \
>>                     include/config/auto.conf
>>  	$(cmd_crmodverdir)
>> -	$(Q)test -e include/generated/autoksyms.h || \
>> -	    touch   include/generated/autoksyms.h
>>
>>  archprepare: archheaders archscripts prepare1 scripts_basic
>>
>> diff --git a/scripts/kconfig/Makefile b/scripts/kconfig/Makefile
>> index ebced77deb..90a091b6ae 100644
>> --- a/scripts/kconfig/Makefile
>> +++ b/scripts/kconfig/Makefile
>> @@ -35,6 +35,8 @@ nconfig: $(obj)/nconf
>>
>>  silentoldconfig: $(obj)/conf
>>  	$(Q)mkdir -p include/config include/generated
>> +	$(Q)test -e include/generated/autoksyms.h || \
>> +	    touch   include/generated/autoksyms.h
>>  	$< $(silent) --$@ $(Kconfig)
>>
>>  localyesconfig localmodconfig: $(obj)/streamline_config.pl $(obj)/conf
>>
>
> The testing was successful.
>
> After testing an hour of builds with different -j values, I'm no longer seeing
> any compile issues when this patch is applied.  When I remove the patch the
> compile error returns so I'm going to say that this patch fixed it.
>
> Thanks again Nicolas.
>
> Tested-by: Prarit Bhargava <prarit@redhat.com>

Looks good here as well, can do parallel make w/o reverting Tony's 
patches again.

Tested-by: Jarod Wilson <jarod@redhat.com>

-- 
Jarod Wilson
jarod@redhat.com

Re: Odd build breakage in 4.9-rc7

[Nicolas Pitre]

On Wed, 30 Nov 2016, Prarit Bhargava wrote:

> 
> 
> On 11/30/2016 01:36 PM, Linus Torvalds wrote:
> > On Wed, Nov 30, 2016 at 10:28 AM, Prarit Bhargava <prarit@redhat.com> wrote:
> > ]>
> >> In my case I tracked this to commit 3637efb00864 ("x86/mce: Add PCI quirks to
> >> identify Xeons with machine check recovery") which adds the include for
> >> generated/autoksyms.h.
> > 
> > Ok, that at least makes some sense. The other blamed commit did not
> > seem to possibly make a difference.
> > 
> >> Searching LKML and I came across a report from Ken Moffat from a month ago:
> >>
> >> http://marc.info/?l=linux-kernel&m=147794681124332&w=2
> > 
> > Does a "make clean" get rid of it forever? Or does it come back?
> 
> It comes back.  The steps to reproduce this are:
> 
> 1.  checkout latest linux.git
> 2.  make -j112
> 
> (IOW, it occurs 100% of the time for me on a clean tree.)
> 
> To work around the bug I have to do
> 
> 1.  checkout latest linux.git
> 2.  comment out the include for generated/autoksyms.h at include/linux/export.h:81
> 3.  compile with -j112
> 
> This fails loudly, but then I do
> 
> 4.  uncomment the include for generated/autoksyms.h at include/linux/export.h:81

A simpler workaround is simply:

	touch include/generated/autoksyms.h

But hopefully the patch I just sent would fix it for good.


Nicolas