3.4-rc smpboot: do_boot_cpu failed(-1) to wakeup CPU#1

3.4-rc smpboot: do_boot_cpu failed(-1) to wakeup CPU#1

[shrybman]

All 3.4-rc kernels I have tried displayed this issue.


This report looks similar https://lkml.org/lkml/2015/9/26/165.
I also have a Core 2 cpu on a Asus P5B (not deluxe) board.



...
 ACPI: Core revision 20150818
 ACPI: 1 ACPI AML tables successfully acquired and loaded
 Mount-cache hash table entries: 8192 (order: 4, 65536 bytes)
 Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes)
 CPU: Physical Processor ID: 0
 CPU: Processor Core ID: 0
 mce: CPU supports 6 MCE banks
 CPU0: Thermal monitoring enabled (TM2)
 process: using mwait in idle threads
 Last level iTLB entries: 4KB 128, 2MB 4, 4MB 4
 Last level dTLB entries: 4KB 256, 2MB 0, 4MB 32, 1GB 0
 Freeing SMP alternatives memory: 20K (ffffffff819b5000 - ffffffff819ba000)
 ftrace: allocating 18583 entries in 73 pages
 ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
 smpboot: CPU0: Intel(R) Core(TM)2 CPU          6400  @ 2.13GHz (fam: 06, model: 0f, stepping: 06)
 Performance Events: PEBS fmt0-, 4-deep LBR, Core2 events, Intel PMU driver.
 perf_event_intel: PEBS disabled due to CPU errata
 ... version:                2
 ... bit width:              40
 ... generic registers:      2
 ... value mask:             000000ffffffffff
 ... max period:             000000007fffffff
 ... fixed-purpose events:   3
 ... event mask:             0000000700000003
 x86: Booting SMP configuration:
 .... node  #0, CPUs:      #1
 smpboot: do_boot_cpu failed(-1) to wakeup CPU#1
 x86: Booted up 1 node, 1 CPUs
 smpboot: Total of 1 processors activated (4272.42 BogoMIPS)
 devtmpfs: initialized


I took a guess and got lucky.


Reverting "x86/smpboot: Remove APIC.wait_for_init_deassert and atomic init_deasserted" resolved the issue.


Regards,


Shane

Re: 3.4-rc smpboot: do_boot_cpu failed(-1) to wakeup CPU#1

[shrybman]

> All 3.4-rc kernels I have tried displayed this issue.


Stupid typo, that should be kernel v4.3-rc.

Re: 3.4-rc smpboot: do_boot_cpu failed(-1) to wakeup CPU#1

[Donald Parsons]

On Wed, 2015-10-14 at 12:20 -0400, shrybman wrote:
> All 3.4-rc kernels I have tried displayed this issue.

> This report looks similar https://lkml.org/lkml/2015/9/26/165.
> I also have a Core 2 cpu on a Asus P5B (not deluxe) board.

> ...
>  ACPI: Core revision 20150818
>  ACPI: 1 ACPI AML tables successfully acquired and loaded
>  Mount-cache hash table entries: 8192 (order: 4, 65536 bytes)
>  Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes)
>  CPU: Physical Processor ID: 0
>  CPU: Processor Core ID: 0
>  mce: CPU supports 6 MCE banks
>  CPU0: Thermal monitoring enabled (TM2)
>  process: using mwait in idle threads
>  Last level iTLB entries: 4KB 128, 2MB 4, 4MB 4
>  Last level dTLB entries: 4KB 256, 2MB 0, 4MB 32, 1GB 0
>  Freeing SMP alternatives memory: 20K (ffffffff819b5000 - ffffffff819ba000)
>  ftrace: allocating 18583 entries in 73 pages
>  ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
>  smpboot: CPU0: Intel(R) Core(TM)2 CPU          6400  @ 2.13GHz (fam: 06, model: 0f, stepping: 06)
>  Performance Events: PEBS fmt0-, 4-deep LBR, Core2 events, Intel PMU driver.
>  perf_event_intel: PEBS disabled due to CPU errata
>  ... version:                2
>  ... bit width:              40
>  ... generic registers:      2
>  ... value mask:             000000ffffffffff
>  ... max period:             000000007fffffff
>  ... fixed-purpose events:   3
>  ... event mask:             0000000700000003
>  x86: Booting SMP configuration:
>  .... node  #0, CPUs:      #1
>  smpboot: do_boot_cpu failed(-1) to wakeup CPU#1
>  x86: Booted up 1 node, 1 CPUs
>  smpboot: Total of 1 processors activated (4272.42 BogoMIPS)
>  devtmpfs: initialized
> I took a guess and got lucky.
> Reverting "x86/smpboot: Remove APIC.wait_for_init_deassert and atomic init_deasserted" resolved the issue.


As a followup, I updated compiler to gcc-5.2 (no help or change).
I also tried suggested /vmlinuz-4.3.0-rc3 parameter in grub: 
        "cpu_init_udelay=10000"
which did not help getting missing CPU back online.

So all linux-4.3.0-rc[15] lose the second CPU core.

Don

Re[2]: 3.4-rc smpboot: do_boot_cpu failed(-1) to wakeup CPU#1

[shrybman]

..snip..

> >  smpboot: Total of 1 processors activated (4272.42 BogoMIPS)
> >  devtmpfs: initialized
> > I took a guess and got lucky.
> > Reverting "x86/smpboot: Remove APIC.wait_for_init_deassert and atomic init_deasserted" resolved the issue.
> 
> 
> As a followup, I updated compiler to gcc-5.2 (no help or change).
> I also tried suggested /vmlinuz-4.3.0-rc3 parameter in grub: 
>         "cpu_init_udelay=10000"
> which did not help getting missing CPU back online.


Did you try reverting the "x86/smpboot: Remove APIC.wait_for_init_deassert
and atomic init_deasserted"  patch?

RE: Re[2]: 3.4-rc smpboot: do_boot_cpu failed(-1) to wakeup CPU#1

[Brown, Len]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 957 bytes --]

Donald, Shane,
Thanks for reporting this.

> > I also tried suggested /vmlinuz-4.3.0-rc3 parameter in grub:
> >         "cpu_init_udelay=10000"
> > which did not help getting missing CPU back online.

right, if the issue is caused by the patch below,
that cmdline will not help.

> Did you try reverting the "x86/smpboot: Remove APIC.wait_for_init_deassert
> and atomic init_deasserted"  patch?

Yes, please let me know if reverting that patch helps you too.

You have similar hardware:

Shane:

smpboot: CPU0: Intel(R) Core(TM)2 CPU          6400  @ 2.13GHz (fam: 06, model: 0f, stepping: 06)

Donald:

CPU : Intel Core 2 CPU 6600 @ 2.4GHz

I think I can get ahold of a core2 6xxx box tomorrow.
Please send me your .config directly, and I'll see if I can reproduce the issue.

thanks,
-Len

ÿôèº{.nÇ+‰·Ÿ®‰­†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±þG«éÿŠ{ayº\x1dʇڙë,j\a­¢f£¢·hšïêÿ‘êçz_è®\x03(­éšŽŠÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?™¨è­Ú&£ø§~á¶iO•æ¬z·švØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?–I¥

RE: Re[2]: 3.4-rc smpboot: do_boot_cpu failed(-1) to wakeup CPU#1

[Donald Parsons]

On Thu, 2015-10-15 at 02:09 +0000, Brown, Len wrote:
> Donald, Shane,
> Thanks for reporting this.
> 
> > > I also tried suggested /vmlinuz-4.3.0-rc3 parameter in grub:
> > >         "cpu_init_udelay=10000"
> > > which did not help getting missing CPU back online.
> 
> right, if the issue is caused by the patch below,
> that cmdline will not help.
> 
> > Did you try reverting the "x86/smpboot: Remove APIC.wait_for_init_deassert
> > and atomic init_deasserted"  patch?
> 
> Yes, please let me know if reverting that patch helps you too.

How?  Please send a patch or git cmd(s).    I have the
    git/stable/linux-stable.git  on my PC.   Thanks.


> You have similar hardware:
> 
> Shane:
> 
> smpboot: CPU0: Intel(R) Core(TM)2 CPU          6400  @ 2.13GHz (fam: 06, model: 0f, stepping: 06)
> 
> Donald:
> 
> CPU : Intel Core 2 CPU 6600 @ 2.4GHz

Mine is also same (fam: 06, model: 0f, stepping: 06).

> I think I can get ahold of a core2 6xxx box tomorrow.
> Please send me your .config directly, and I'll see if I can reproduce the issue.

Will do.

Don

RE: Re[2]: 3.4-rc smpboot: do_boot_cpu failed(-1) to wakeup CPU#1

[Brown, Len]

> > > Did you try reverting the "x86/smpboot: Remove
> APIC.wait_for_init_deassert
> > > and atomic init_deasserted"  patch?
> >
> > Yes, please let me know if reverting that patch helps you too.
> 
> How?  Please send a patch or git cmd(s).    I have the
>     git/stable/linux-stable.git  on my PC.   Thanks.

git log calls it this:

commit 656bba306827a44ed73b3f93f75bb3147de17fae
Author: Len Brown <len.brown@intel.com>
Date:   Sun Aug 16 11:45:48 2015 -0400

    x86/smpboot: Remove APIC.wait_for_init_deassert and atomic init_deasserted

So you want to simply do this:

$ git revert 656bba306827a44ed73b3f93f75bb3147de17fae

build and test.

cheers,
-Len

RE: Re[2]: 3.4-rc smpboot: do_boot_cpu failed(-1) to wakeup CPU#1

[Donald Parsons]

On Thu, 2015-10-15 at 02:53 +0000, Brown, Len wrote:
> > > > Did you try reverting the "x86/smpboot: Remove
> > APIC.wait_for_init_deassert
> > > > and atomic init_deasserted"  patch?
> > >
> > > Yes, please let me know if reverting that patch helps you too.
> > 
> > How?  Please send a patch or git cmd(s).    I have the
> >     git/stable/linux-stable.git  on my PC.   Thanks.
> 
> git log calls it this:
> 
> commit 656bba306827a44ed73b3f93f75bb3147de17fae
> Author: Len Brown <len.brown@intel.com>
> Date:   Sun Aug 16 11:45:48 2015 -0400
> 
>     x86/smpboot: Remove APIC.wait_for_init_deassert and atomic init_deasserted
> 
> So you want to simply do this:
> 
> $ git revert 656bba306827a44ed73b3f93f75bb3147de17fae
> 
> build and test.

I did the revert in linux-stable (last tag being v4.3-rc4) gave a revert
description so it would be applied.

built and tested.  Result:  did not help, still missing the second core.

I am sure the revert happened but I am uncertain as to how to build in a
git tree.  I think I did it correctly
   cp .config into   linux-stable   tree
   edit Makefile to add -rev to: EXTRAVERSION = -rc4-rev
   make oldconfig
   make -j3 >> ../mak/mak_430-rc4-rev.log 2>&1 &

The before/after kernels are:
    3965280 Oct  4 15:20 vmlinuz-4.3.0-rc4
    3965408 Oct 15 13:45 vmlinuz-4.3.0-rc4-rev
The reverted vmlinuz is 128 bytes bigger;  .config's are identical.  I
see the revert adds back in some code so sizes seem appropriate.

Don

RE: Re[2]: 3.4-rc smpboot: do_boot_cpu failed(-1) to wakeup CPU#1

[Brown, Len]

> I did the revert in linux-stable (last tag being v4.3-rc4) gave a revert
> description so it would be applied.
> 
> built and tested.  Result:  did not help, still missing the second core.

Same result here.

upstream failed to bring up CPU #1 on 5/5 boots

    Revert "x86/smpboot: Remove APIC.wait_for_init_deassert and atomic init_deasserted"

    This reverts commit 656bba306827a44ed73b3f93f75bb3147de17fae.

Still fails the same way.

Adding "cpu_init_udelay=10000"

does not help.

commence bisect...

cheers,
-Len

Re[4]: 3.4-rc smpboot: do_boot_cpu failed(-1) to wakeup CPU#1

[shrybman]

> Donald, Shane,
> Thanks for reporting this.

Sure! Thanks for looking into it.

> You have similar hardware:
> 
> Shane:
> 
> smpboot: CPU0: Intel(R) Core(TM)2 CPU          6400  @ 2.13GHz (fam: 06, model: 0f, stepping: 06)
> 
> Donald:
> 
> CPU : Intel Core 2 CPU 6600 @ 2.4GHz
> 
> I think I can get ahold of a core2 6xxx box tomorrow.


Maybe the mobo/bios? We both have the same Asus P5B as well. Donald said he has the deluxe version while I have the plain version.

RE: Re[4]: 3.4-rc smpboot: do_boot_cpu failed(-1) to wakeup CPU#1

[Brown, Len]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 661 bytes --]

> > You have similar hardware:
> >
> > Shane:
> >
> > smpboot: CPU0: Intel(R) Core(TM)2 CPU          6400  @ 2.13GHz (fam: 06,
> model: 0f, stepping: 06)
> >
> > Donald:
> >
> > CPU : Intel Core 2 CPU 6600 @ 2.4GHz
> >
> > I think I can get ahold of a core2 6xxx box tomorrow.

Intel(R) Core(TM)2 CPU         E6800  @ 2.93GHz

is working for me with latest upstream.
(It is on an Intel D975 XBX motherboard)

please send me the .config you are using
that fails and I'll try that.

thanks,
-Len

ÿôèº{.nÇ+‰·Ÿ®‰­†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±þG«éÿŠ{ayº\x1dʇڙë,j\a­¢f£¢·hšïêÿ‘êçz_è®\x03(­éšŽŠÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?™¨è­Ú&£ø§~á¶iO•æ¬z·švØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?–I¥

Re[6]: 3.4-rc smpboot: do_boot_cpu failed(-1) to wakeup CPU#1

[shrybman]

[-- Attachment #1: Type: text/plain, Size: 49 bytes --]

I sent my config directly but here it is again.

[-- Attachment #2: config-4.3.0-rc5.c-00055-gdacea3b.gz --]
[-- Type: application/gzip, Size: 21876 bytes --]

RE: Re[6]: 3.4-rc smpboot: do_boot_cpu failed(-1) to wakeup CPU#1

[Brown, Len]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 571 bytes --]

try booting upstream with "cpu_init_udelay=1".
If it works, then it actually implicates this commit:

a9bcaa02a5104ace6a9d9e4a9cd9192a9e7744d6
("x86/smpboot: Remove SIPI delays from cpu_up()")

Unfortunately the commit message for that on is erroneous --
"cpu_init_udelay=10000" is actually a NO-OP,
because that matches the compiled-in default.
Indeed, any non-zero value bug 10000 should work.

thanks,
-Len

ÿôèº{.nÇ+‰·Ÿ®‰­†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±þG«éÿŠ{ayº\x1dʇڙë,j\a­¢f£¢·hšïêÿ‘êçz_è®\x03(­éšŽŠÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?™¨è­Ú&£ø§~á¶iO•æ¬z·švØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?–I¥

Re[6]: 3.4-rc smpboot: do_boot_cpu failed(-1) to wakeup CPU#1

[shrybman]

> try booting upstream with "cpu_init_udelay=1".
> If it works, then it actually implicates this commit:
> 
> a9bcaa02a5104ace6a9d9e4a9cd9192a9e7744d6
> ("x86/smpboot: Remove SIPI delays from cpu_up()")
> 
> Unfortunately the commit message for that on is erroneous --
> "cpu_init_udelay=10000" is actually a NO-OP,
> because that matches the compiled-in default.
> Indeed, any non-zero value bug 10000 should work.


That does seem to work. Heh, I tried that with 10000 last week.


Thanks,


Shane

Re: Re[6]: 3.4-rc smpboot: do_boot_cpu failed(-1) to wakeup CPU#1

[itpenguin]

I'm a little bit confused about this thread, because I'm facing a similiar
problem on my system:

I have a CentOS 7.7 AltArch i386 installation:
Linux erbusze 3.10.0-1062.18.1.el7.centos.plus.i686 #1 SMP Wed Mar 18
12:57:13 UTC 2020 i686 i686 i386 GNU/Linux

on this CPU:
smpboot: CPU0: Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz (fam: 06, model: 5e,
stepping: 03)

with these boot errors:
[    0.160538] Kernel panic - not syncing: Fatal exception
[   10.166450] smpboot: do_boot_cpu failed(-1) to wakeup CPU#1

I tried several different values of cpu_init_udelay in the kernel command
boot line, but with no changes to the described behaviour.



--
Sent from: http://linux-kernel.2935.n7.nabble.com/

RE: Re[4]: 3.4-rc smpboot: do_boot_cpu failed(-1) to wakeup CPU#1

[Brown, Len]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 1858 bytes --]

> > > You have similar hardware:
> > >
> > > Shane:
> > >
> > > smpboot: CPU0: Intel(R) Core(TM)2 CPU          6400  @ 2.13GHz (fam:
> 06,
> > model: 0f, stepping: 06)
> > >
> > > Donald:
> > >
> > > CPU : Intel Core 2 CPU 6600 @ 2.4GHz
> > >
> > > I think I can get ahold of a core2 6xxx box tomorrow.
> 
> Intel(R) Core(TM)2 CPU         E6800  @ 2.93GHz
> 
> is working for me with latest upstream.
> (It is on an Intel D975 XBX motherboard)

Good news - I reproduced the failure on a similar box, an Intel D975xbx2:

[    0.000000] Linux version 4.3.0-rc5+ (lenb@z87) (gcc version 4.9.2 20150212 (Red Hat 4.9.2-6) (GCC) ) #375 SMP Thu Oct 15 18:17:04 EDT 2015
...
[    0.084000] smpboot: CPU0: Intel(R) Core(TM)2 Quad CPU           @ 2.66GHz (family: 0x6, model: 0xf, stepping: 0x7)
[    0.084000] Performance Events: PEBS fmt0-, 4-deep LBR, Core2 events, Intel PMU driver.
[    0.084000] perf_event_intel: PEBS disabled due to CPU errata
[    0.084000] ... version:                2
[    0.084000] ... bit width:              40
[    0.084000] ... generic registers:      2
[    0.084000] ... value mask:             000000ffffffffff
[    0.084000] ... max period:             000000007fffffff
[    0.084000] ... fixed-purpose events:   3
[    0.084000] ... event mask:             0000000700000003
[    0.084000] x86: Booting SMP configuration:
[    0.084000] .... node  #0, CPUs:      #1
[   10.080003] smpboot: do_boot_cpu failed(-1) to wakeup CPU#1
[   10.080175] NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter.
[   10.080334]  #2 #3
[   10.084017] x86: Booted up 1 node, 3 CPUs
[   10.084120] smpboot: Total of 3 processors activated (16001.58 BogoMIPS)

ÿôèº{.nÇ+‰·Ÿ®‰­†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±þG«éÿŠ{ayº\x1dʇڙë,j\a­¢f£¢·hšïêÿ‘êçz_è®\x03(­éšŽŠÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?™¨è­Ú&£ø§~á¶iO•æ¬z·švØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?–I¥