* WARNING: at arch/x86/kernel/apic/apic.c:1239 setup_local_APIC+...
@ 2011-10-14 13:55 Borislav Petkov
2011-10-14 16:56 ` Borislav Petkov
0 siblings, 1 reply; 9+ messages in thread
From: Borislav Petkov @ 2011-10-14 13:55 UTC (permalink / raw)
To: Tejun Heo; +Cc: Andreas Herrmann, Conny Seidel, Ingo Molnar, LKML
[-- Attachment #1: Type: text/plain, Size: 3949 bytes --]
Hi,
we're hitting the warning below
<arch/x86/kernel/apic/apic.c>:
...
i = early_per_cpu(x86_cpu_to_logical_apicid, cpu);
WARN_ON(i != BAD_APICID && i != logical_smp_processor_id()); <--- HERE
/* always use the value from LDR */
early_per_cpu(x86_cpu_to_logical_apicid, cpu) =
logical_smp_processor_id();
during 32-bit testing, config is attached.
acb8bc09c6185e4d3d582d0076aaa6a89f19d8c5 added this warning
and triggers on the second part of the condition. I've dumped
x86_cpu_to_logical_apicid and logical_smp_processor_id and here's what I
get on a 32-cores box:
...
[ 4.268493] CPU 31 irqstacks, hard=e5b1e000 soft=e5b20000
[ 4.269358] #31 Ok.
[ 4.270358] smpboot cpu 31: start_ip = 80000
[ 0.003999] Initializing CPU#31
[ 0.003999] ------------[ cut here ]------------
[ 0.003999] WARNING: at arch/x86/kernel/apic/apic.c:1239 setup_local_APIC+0x12c/0x3bd()
[ 0.003999] Hardware name: None
[ 0.003999] i: -2147483648, lg_smp_id: 79
[ 0.003999] Modules linked in:
[ 0.003999] Pid: 0, comm: kworker/0:1 Tainted: G W 3.1.0-rc9-37cf9516-linus+ #1
[ 0.003999] Call Trace:
[ 0.003999] [<c1035066>] warn_slowpath_common+0x65/0x7a
[ 0.003999] [<c10350ee>] warn_slowpath_fmt+0x2b/0x2f
[ 0.003999] [<c16603bb>] setup_local_APIC+0x12c/0x3bd
[ 0.003999] [<c165fcae>] start_secondary+0x9f/0x18f
[ 0.003999] ---[ end trace 4eaa2a86a8e2da41 ]---
That's right, x86_cpu_to_logical_apicid wraps around so if I were to
boot a 32-bit kernel on a 64 cores box, I'd get each two cores to have
the same logical apic id.
Oh, and logical_smp_processor_id(), i.e. logical APIC ID starts at 0x20,
acc. to SRAT:
[ 0.000000] SRAT: PXM 0 -> APIC 0x20 -> Node 0
[ 0.000000] SRAT: PXM 0 -> APIC 0x21 -> Node 0
[ 0.000000] SRAT: PXM 0 -> APIC 0x22 -> Node 0
[ 0.000000] SRAT: PXM 0 -> APIC 0x23 -> Node 0
[ 0.000000] SRAT: PXM 0 -> APIC 0x24 -> Node 0
[ 0.000000] SRAT: PXM 0 -> APIC 0x25 -> Node 0
[ 0.000000] SRAT: PXM 0 -> APIC 0x26 -> Node 0
[ 0.000000] SRAT: PXM 0 -> APIC 0x27 -> Node 0
[ 0.000000] SRAT: PXM 1 -> APIC 0x28 -> Node 1
[ 0.000000] SRAT: PXM 1 -> APIC 0x29 -> Node 1
[ 0.000000] SRAT: PXM 1 -> APIC 0x2a -> Node 1
[ 0.000000] SRAT: PXM 1 -> APIC 0x2b -> Node 1
[ 0.000000] SRAT: PXM 1 -> APIC 0x2c -> Node 1
[ 0.000000] SRAT: PXM 1 -> APIC 0x2d -> Node 1
[ 0.000000] SRAT: PXM 1 -> APIC 0x2e -> Node 1
[ 0.000000] SRAT: PXM 1 -> APIC 0x2f -> Node 1
[ 0.000000] SRAT: PXM 2 -> APIC 0x40 -> Node 2
[ 0.000000] SRAT: PXM 2 -> APIC 0x41 -> Node 2
[ 0.000000] SRAT: PXM 2 -> APIC 0x42 -> Node 2
[ 0.000000] SRAT: PXM 2 -> APIC 0x43 -> Node 2
[ 0.000000] SRAT: PXM 2 -> APIC 0x44 -> Node 2
[ 0.000000] SRAT: PXM 2 -> APIC 0x45 -> Node 2
[ 0.000000] SRAT: PXM 2 -> APIC 0x46 -> Node 2
[ 0.000000] SRAT: PXM 2 -> APIC 0x47 -> Node 2
[ 0.000000] SRAT: PXM 3 -> APIC 0x48 -> Node 3
[ 0.000000] SRAT: PXM 3 -> APIC 0x49 -> Node 3
[ 0.000000] SRAT: PXM 3 -> APIC 0x4a -> Node 3
[ 0.000000] SRAT: PXM 3 -> APIC 0x4b -> Node 3
[ 0.000000] SRAT: PXM 3 -> APIC 0x4c -> Node 3
[ 0.000000] SRAT: PXM 3 -> APIC 0x4d -> Node 3
[ 0.000000] SRAT: PXM 3 -> APIC 0x4e -> Node 3
[ 0.000000] SRAT: PXM 3 -> APIC 0x4f -> Node 3
The warning triggers IMHO because we switch to bigsmp APIC in
default_setup_apic_routing() but the early x86_cpu_to_logical_apicid
enumeration comes from default_x86_32_early_logical_apicid()
which gets called as part of the default apic deal as part of
generic_processor_info().
So, in the end, and AFAICR, the warning triggers because we're comparing
logical APIC IDs from the APIC Logical Destination Register (0xD0) which
have been assigned by BIOS with "1 << cpu" shifted values which wrap on
32-bit.
I'd very much like to know why?
Thanks.
--
Regards/Gruss,
Boris.
Operating Systems Research Center
Advanced Micro Devices, Inc.
[-- Attachment #2: config.gz --]
[-- Type: application/octet-stream, Size: 16878 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: WARNING: at arch/x86/kernel/apic/apic.c:1239 setup_local_APIC+...
2011-10-14 13:55 WARNING: at arch/x86/kernel/apic/apic.c:1239 setup_local_APIC+ Borislav Petkov
@ 2011-10-14 16:56 ` Borislav Petkov
2011-10-14 17:55 ` Tejun Heo
0 siblings, 1 reply; 9+ messages in thread
From: Borislav Petkov @ 2011-10-14 16:56 UTC (permalink / raw)
To: Tejun Heo; +Cc: Herrmann3, Andreas, Seidel, Conny, Ingo Molnar, LKML
On Fri, Oct 14, 2011 at 09:55:16AM -0400, Borislav Petkov wrote:
> So, in the end, and AFAICR, the warning triggers because we're comparing
> logical APIC IDs from the APIC Logical Destination Register (0xD0) which
> have been assigned by BIOS with "1 << cpu" shifted values which wrap on
> 32-bit.
>
> I'd very much like to know why?
Update: upon a second thought, we think we know why: The WARN_ON
cannot stomach the switch to bigsmp apic because, on the one hand,
bigsmp_init_apic_ldr() "normalizes" the APIC ids by using the BIOS
values (x86_bios_cpu_apicid) but OTOH, x86_cpu_to_logical_apicid
contains the early percpu value which got assigned by the default apic,
i.e. the shifting thing as it is spec'ed for the LDR register.
So, the way we see it, the WARN_ON should know whether we switched apics
or should be removed completely. Btw, why was it added in the first
place, the commit message doesn't quite explain why.
Thanks.
--
Regards/Gruss,
Boris.
Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach
GM: Alberto Bozzo
Reg: Dornach, Landkreis Muenchen
HRB Nr. 43632 WEEE Registernr: 129 19551
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: WARNING: at arch/x86/kernel/apic/apic.c:1239 setup_local_APIC+...
2011-10-14 16:56 ` Borislav Petkov
@ 2011-10-14 17:55 ` Tejun Heo
2011-10-15 22:12 ` Borislav Petkov
2011-10-17 7:23 ` Jan Beulich
0 siblings, 2 replies; 9+ messages in thread
From: Tejun Heo @ 2011-10-14 17:55 UTC (permalink / raw)
To: Borislav Petkov
Cc: Herrmann3, Andreas, Seidel, Conny, Ingo Molnar, LKML, JBeulich
Hello,
(cc'ing Jan Beulich)
On Fri, Oct 14, 2011 at 06:56:04PM +0200, Borislav Petkov wrote:
> On Fri, Oct 14, 2011 at 09:55:16AM -0400, Borislav Petkov wrote:
> > So, in the end, and AFAICR, the warning triggers because we're comparing
> > logical APIC IDs from the APIC Logical Destination Register (0xD0) which
> > have been assigned by BIOS with "1 << cpu" shifted values which wrap on
> > 32-bit.
> >
> > I'd very much like to know why?
>
> Update: upon a second thought, we think we know why: The WARN_ON
> cannot stomach the switch to bigsmp apic because, on the one hand,
> bigsmp_init_apic_ldr() "normalizes" the APIC ids by using the BIOS
> values (x86_bios_cpu_apicid) but OTOH, x86_cpu_to_logical_apicid
> contains the early percpu value which got assigned by the default apic,
> i.e. the shifting thing as it is spec'ed for the LDR register.
>
> So, the way we see it, the WARN_ON should know whether we switched apics
> or should be removed completely. Btw, why was it added in the first
> place, the commit message doesn't quite explain why.
Jan hit similar problem a while back.
http://thread.gmane.org/gmane.linux.kernel/1180954
I can't seem to find the patch in mainline. Jan, Ingo? Borislav,
would the patch there resolve the warning you're seeing too?
Anyways, to summarize, the early apic thing isn't necessary anymore
and can be ripped out and the warning is pretty much bogus at this
point. I'll rip it out soonish.
Thank you.
--
tejun
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: WARNING: at arch/x86/kernel/apic/apic.c:1239 setup_local_APIC+...
2011-10-14 17:55 ` Tejun Heo
@ 2011-10-15 22:12 ` Borislav Petkov
2011-10-16 22:07 ` Conny Seidel
2011-10-17 7:23 ` Jan Beulich
1 sibling, 1 reply; 9+ messages in thread
From: Borislav Petkov @ 2011-10-15 22:12 UTC (permalink / raw)
To: Tejun Heo
Cc: Herrmann3, Andreas, Seidel, Conny, Ingo Molnar, LKML,
JBeulich@novell.com
On Fri, Oct 14, 2011 at 01:55:56PM -0400, Tejun Heo wrote:
> Jan hit similar problem a while back.
>
> http://thread.gmane.org/gmane.linux.kernel/1180954
>
> I can't seem to find the patch in mainline. Jan, Ingo? Borislav,
> would the patch there resolve the warning you're seeing too?
There's even a -v2 at http://marc.info/?l=linux-kernel&m=131722471001895
We'll run that one to test.
> Anyways, to summarize, the early apic thing isn't necessary anymore
> and can be ripped out and the warning is pretty much bogus at this
> point. I'll rip it out soonish.
Ok, thanks.
--
Regards/Gruss,
Boris.
Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach
GM: Alberto Bozzo
Reg: Dornach, Landkreis Muenchen
HRB Nr. 43632 WEEE Registernr: 129 19551
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: WARNING: at arch/x86/kernel/apic/apic.c:1239 setup_local_APIC+...
2011-10-15 22:12 ` Borislav Petkov
@ 2011-10-16 22:07 ` Conny Seidel
2011-10-17 7:23 ` Borislav Petkov
0 siblings, 1 reply; 9+ messages in thread
From: Conny Seidel @ 2011-10-16 22:07 UTC (permalink / raw)
To: Borislav Petkov
Cc: Tejun Heo, Herrmann3, Andreas, Ingo Molnar, LKML,
JBeulich@novell.com
[-- Attachment #1: Type: text/plain, Size: 1378 bytes --]
On Sat, 15 Oct 2011 18:12:15 -0400
Borislav Petkov <bp@amd64.org> wrote:
>On Fri, Oct 14, 2011 at 01:55:56PM -0400, Tejun Heo wrote:
>> Jan hit similar problem a while back.
>>
>> http://thread.gmane.org/gmane.linux.kernel/1180954
>>
>> I can't seem to find the patch in mainline. Jan, Ingo? Borislav,
>> would the patch there resolve the warning you're seeing too?
>
>There's even a -v2 at
>http://marc.info/?l=linux-kernel&m=131722471001895
>
>We'll run that one to test.
Looks good. The warning is gone.
>> Anyways, to summarize, the early apic thing isn't necessary anymore
>> and can be ripped out and the warning is pretty much bogus at this
>> point. I'll rip it out soonish.
>
>Ok, thanks.
>
--
regards,
Conny Seidel
##################################################################
# Email : conny.seidel@amd.com GnuPG-Key : 0xA6AB055D #
# Fingerprint: 17C4 5DB2 7C4C C1C7 1452 8148 F139 7C09 A6AB 055D #
##################################################################
# Advanced Micro Devices GmbH Einsteinring 24 85609 Dornach #
# General Managers: Alberto Bozzo #
# Registration: Dornach, Landkr. Muenchen; Registerger. Muenchen #
# HRB Nr. 43632 #
##################################################################
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: WARNING: at arch/x86/kernel/apic/apic.c:1239 setup_local_APIC+...
2011-10-16 22:07 ` Conny Seidel
@ 2011-10-17 7:23 ` Borislav Petkov
2011-10-17 7:38 ` Ingo Molnar
0 siblings, 1 reply; 9+ messages in thread
From: Borislav Petkov @ 2011-10-17 7:23 UTC (permalink / raw)
To: Ingo Molnar
Cc: Conny Seidel, Tejun Heo, Herrmann3, Andreas, LKML,
JBeulich@novell.com, x86
On Sun, Oct 16, 2011 at 06:07:59PM -0400, Conny Seidel wrote:
> On Sat, 15 Oct 2011 18:12:15 -0400
> Borislav Petkov <bp@amd64.org> wrote:
>
> >On Fri, Oct 14, 2011 at 01:55:56PM -0400, Tejun Heo wrote:
> >> Jan hit similar problem a while back.
> >>
> >> http://thread.gmane.org/gmane.linux.kernel/1180954
> >>
> >> I can't seem to find the patch in mainline. Jan, Ingo? Borislav,
> >> would the patch there resolve the warning you're seeing too?
> >
> >There's even a -v2 at
> >http://marc.info/?l=linux-kernel&m=131722471001895
> >
> >We'll run that one to test.
>
> Looks good. The warning is gone.
Good. Ingo, you can have our
Acked-by: Borislav Petkov <borislav.petkov@amd.com>
Tested-by: Conny Seidel <conny.seidel@amd.com>
for it, it being the -v2 at http://marc.info/?l=linux-kernel&m=131722471001895
Thanks to all for their work.
--
Regards/Gruss,
Boris.
Operating Systems Research Center
Advanced Micro Devices, Inc.
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: WARNING: at arch/x86/kernel/apic/apic.c:1239 setup_local_APIC+...
2011-10-17 7:23 ` Borislav Petkov
@ 2011-10-17 7:38 ` Ingo Molnar
2011-10-17 7:53 ` Borislav Petkov
0 siblings, 1 reply; 9+ messages in thread
From: Ingo Molnar @ 2011-10-17 7:38 UTC (permalink / raw)
To: Borislav Petkov
Cc: Conny Seidel, Tejun Heo, Herrmann3, Andreas, LKML,
JBeulich@novell.com, x86
* Borislav Petkov <bp@amd64.org> wrote:
> On Sun, Oct 16, 2011 at 06:07:59PM -0400, Conny Seidel wrote:
> > On Sat, 15 Oct 2011 18:12:15 -0400
> > Borislav Petkov <bp@amd64.org> wrote:
> >
> > >On Fri, Oct 14, 2011 at 01:55:56PM -0400, Tejun Heo wrote:
> > >> Jan hit similar problem a while back.
> > >>
> > >> http://thread.gmane.org/gmane.linux.kernel/1180954
> > >>
> > >> I can't seem to find the patch in mainline. Jan, Ingo? Borislav,
> > >> would the patch there resolve the warning you're seeing too?
> > >
> > >There's even a -v2 at
> > >http://marc.info/?l=linux-kernel&m=131722471001895
> > >
> > >We'll run that one to test.
> >
> > Looks good. The warning is gone.
>
> Good. Ingo, you can have our
>
> Acked-by: Borislav Petkov <borislav.petkov@amd.com>
> Tested-by: Conny Seidel <conny.seidel@amd.com>
>
> for it, it being the -v2 at http://marc.info/?l=linux-kernel&m=131722471001895
Yeah, it's still lined up in -tip:x86/apic for a v3.2 merge.
Thanks,
Ingo
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: WARNING: at arch/x86/kernel/apic/apic.c:1239 setup_local_APIC+...
2011-10-17 7:38 ` Ingo Molnar
@ 2011-10-17 7:53 ` Borislav Petkov
0 siblings, 0 replies; 9+ messages in thread
From: Borislav Petkov @ 2011-10-17 7:53 UTC (permalink / raw)
To: Ingo Molnar
Cc: Seidel, Conny, Tejun Heo, Herrmann3, Andreas, LKML,
JBeulich@novell.com, x86
On Mon, Oct 17, 2011 at 03:38:11AM -0400, Ingo Molnar wrote:
> Yeah, it's still lined up in -tip:x86/apic for a v3.2 merge.
Cool,
thanks.
--
Regards/Gruss,
Boris.
Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach
GM: Alberto Bozzo
Reg: Dornach, Landkreis Muenchen
HRB Nr. 43632 WEEE Registernr: 129 19551
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: WARNING: at arch/x86/kernel/apic/apic.c:1239 setup_local_APIC+...
2011-10-14 17:55 ` Tejun Heo
2011-10-15 22:12 ` Borislav Petkov
@ 2011-10-17 7:23 ` Jan Beulich
1 sibling, 0 replies; 9+ messages in thread
From: Jan Beulich @ 2011-10-17 7:23 UTC (permalink / raw)
To: Borislav Petkov, Tejun Heo
Cc: Andreas Herrmann3, Conny Seidel, Ingo Molnar, LKML
>>> On 14.10.11 at 19:55, Tejun Heo <tj@kernel.org> wrote:
> Hello,
>
> (cc'ing Jan Beulich)
>
> On Fri, Oct 14, 2011 at 06:56:04PM +0200, Borislav Petkov wrote:
>> On Fri, Oct 14, 2011 at 09:55:16AM -0400, Borislav Petkov wrote:
>> > So, in the end, and AFAICR, the warning triggers because we're comparing
>> > logical APIC IDs from the APIC Logical Destination Register (0xD0) which
>> > have been assigned by BIOS with "1 << cpu" shifted values which wrap on
>> > 32-bit.
>> >
>> > I'd very much like to know why?
>>
>> Update: upon a second thought, we think we know why: The WARN_ON
>> cannot stomach the switch to bigsmp apic because, on the one hand,
>> bigsmp_init_apic_ldr() "normalizes" the APIC ids by using the BIOS
>> values (x86_bios_cpu_apicid) but OTOH, x86_cpu_to_logical_apicid
>> contains the early percpu value which got assigned by the default apic,
>> i.e. the shifting thing as it is spec'ed for the LDR register.
>>
>> So, the way we see it, the WARN_ON should know whether we switched apics
>> or should be removed completely. Btw, why was it added in the first
>> place, the commit message doesn't quite explain why.
>
> Jan hit similar problem a while back.
>
> http://thread.gmane.org/gmane.linux.kernel/1180954
>
> I can't seem to find the patch in mainline.
Ingo let me know at the end of September that v2 got applied to -tip;
at that point, I took it that it would make only 3.2 (and perhaps then
earlier ones via -stable).
Jan
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2011-10-17 7:54 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2011-10-14 13:55 WARNING: at arch/x86/kernel/apic/apic.c:1239 setup_local_APIC+ Borislav Petkov
2011-10-14 16:56 ` Borislav Petkov
2011-10-14 17:55 ` Tejun Heo
2011-10-15 22:12 ` Borislav Petkov
2011-10-16 22:07 ` Conny Seidel
2011-10-17 7:23 ` Borislav Petkov
2011-10-17 7:38 ` Ingo Molnar
2011-10-17 7:53 ` Borislav Petkov
2011-10-17 7:23 ` Jan Beulich
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox