Re: [patch] x86: 2.6.31-rc7 crash due to buggy flat_phys_pkg_id

[Yinghai Lu <yinghai@kernel.org>]

Ravikiran G Thirumalai wrote:
> 2.6.31-rc7 does not boot on vSMPowered systems.  The sched domains
> seem to build incorrectly with error messages of the sort:
> 
> [    8.501108] CPU31: Thermal monitoring enabled (TM1)
> [    8.501127] CPU 31 MCA banks SHD:2 SHD:3 SHD:5 SHD:6 SHD:8
> [    8.650254] CPU31: Intel(R) Xeon(R) CPU           E5540  @ 2.53GHz
> stepping 04
> [    8.710324] Brought up 32 CPUs
> [    8.713916] Total of 32 processors activated (162314.96 BogoMIPS).
> [    8.721489] ERROR: parent span is not a superset of domain->span
> [    8.727686] ERROR: domain->groups does not contain CPU0
> [    8.733091] ERROR: groups don't span domain->span
> [    8.737975] ERROR: domain->cpu_power not set
> [    8.742416]
> 
> This is followed by oopsen in the scheduler code
> with NULL pointer deference in find_busiest_group.
> 
> Git bisection pointed to the following commit:
> 
> commit 2759c3287de27266e06f1f4e82cbd2d65f6a044c
> x86: don't call read_apic_id if !cpu_has_apic
> 
> Upon examining the history of the commit, the above commit seems to be a fix
> for:
> 
> commit 4797f6b021a3fa399942245d07a1feb30df81bb8
> x86: read apic ID in the !acpi_lapic case
> 
> However, there appears to be bug in the commit
> 2759c3287de27266e06f1f4e82cbd2d65f6a044c, where flat_phys_pkg_id
> uses initial apic id instead of hard_smp_processor_id() on SMP machines.
> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=2759c3287de27266e06f1f4e82cbd2d65f6a044c
> 
> This patch fixes the bug and causes vSMPowered systems to boot up
> correctly,
> 
> Signed-off-by: Ravikiran Thirumalai <kiran@scalex86.org>
> Cc: Yinghai Lu <yinghai@kernel.org>
> 
> Index: linux-2.6.31-rc6/arch/x86/kernel/apic/apic_flat_64.c
> ===================================================================
> --- linux-2.6.31-rc6.orig/arch/x86/kernel/apic/apic_flat_64.c	2009-08-21 12:42:16.000000000 -0700
> +++ linux-2.6.31-rc6/arch/x86/kernel/apic/apic_flat_64.c	2009-08-21 14:12:21.654837472 -0700
> @@ -161,7 +161,8 @@ static int flat_apic_id_registered(void)
>  
>  static int flat_phys_pkg_id(int initial_apic_id, int index_msb)
>  {
> -	return initial_apic_id >> index_msb;
> +	return cpu_has_apic ? hard_smp_processor_id() >> index_msb :
> +	    initial_apic_id >> index_msb;
>  }

it seems we should use initial apic id here.

can you check which calling for vsmp cause the problem?
arch/x86/kernel/cpu/addon_cpuid_features.c:     c->cpu_core_id = apic->phys_pkg_id(c->initial_apicid, ht_mask_width)
arch/x86/kernel/cpu/addon_cpuid_features.c:     c->phys_proc_id = apic->phys_pkg_id(c->initial_apicid, core_plus_mask_width
);
arch/x86/kernel/cpu/addon_cpuid_features.c:     c->apicid = apic->phys_pkg_id(c->initial_apicid, 0);
arch/x86/kernel/cpu/common.c:   c->phys_proc_id = apic->phys_pkg_id(c->initial_apicid, index_msb);
arch/x86/kernel/cpu/common.c:   c->cpu_core_id = apic->phys_pkg_id(c->initial_apicid, index_msb) &
arch/x86/kernel/cpu/common.c:           c->apicid = apic->phys_pkg_id(c->initial_apicid, 0);
arch/x86/kernel/cpu/common.c:   c->apicid = apic->phys_pkg_id(c->initial_apicid, 0);

YH