RE: 2.6.12-rc5 is broken in nvidia Ck804 Opteron MB/with dual cor e dual way

RE: 2.6.12-rc5 is broken in nvidia Ck804 Opteron MB/with dual cor e dual way

[YhLu]

andi,

in arch/x86_64/kernel/smboot.c, function detect_siblings(),

because smp_num_siblings is always =1,  so several lines can be removed.

        for_each_online_cpu (cpu) {
                struct cpuinfo_x86 *c = cpu_data + cpu;
*R*                int siblings = 0;
                int i;
*R*                if (smp_num_siblings > 1) {
*R*                        for_each_online_cpu (i) {
*R*                                if (cpu_core_id[cpu] == cpu_core_id[i]) {
*R*                                        siblings++;
*R*                                        cpu_set(i, cpu_sibling_map[cpu]);
*R*                                }
*R*                        }
*R*                } else {
*R*                      siblings++;
                        cpu_set(cpu, cpu_sibling_map[cpu]);
*R*                }


 *R*               if (siblings != smp_num_siblings) {
 *R*                       printk(KERN_WARNING
 *R*              "WARNING: %d siblings found for CPU%d, should be %d\n",
 *R*                              siblings, cpu, smp_num_siblings);
 *R*                       smp_num_siblings = siblings;
 *R*               }

Also I found the workaround for hang on second node is
adding one line in setup.c
        /* Low order bits define the core id (index of core in socket) */
        cpu_core_id[cpu] = phys_proc_id[cpu] & ((1 << bits)-1);
        /* Convert the APIC ID into the socket ID */
        phys_proc_id[cpu] >>= bits;
+        printk(KERN_INFO "  CPU %d(%d)  phys_proc_id %d  Core %d\n", 
+               cpu, c->x86_num_cores, phys_proc_id[cpu], cpu_core_id[cpu]);

My compile environment SUSE 9.2 gcc version is 3.3.3???

YH

RE: 2.6.12-rc5 is broken in nvidia Ck804 Opteron MB/with dual cor e dual way

[YhLu]

Brought up 4 CPUs
    CPU 0, cpu_sibling_map[0]= 1 
    CPU 0, cpu_core_map[0]= 3 
    CPU 1, cpu_sibling_map[1]= 2 
    CPU 1, cpu_core_map[1]= 3 
    CPU 2, cpu_sibling_map[2]= 4 
    CPU 2, cpu_core_map[2]= c 
    CPU 3, cpu_sibling_map[3]= 8 
    CPU 3, cpu_core_map[3]= c 
are the cpu_sibling_map[] right? 

YH

> -----Original Message-----
> From: YhLu 
> Sent: Wednesday, June 01, 2005 8:17 PM
> To: Andi Kleen
> Cc: linux-kernel@vger.kernel.org
> Subject: RE: 2.6.12-rc5 is broken in nvidia Ck804 Opteron 
> MB/with dual cor e dual way
> 
> andi,
> 
> in arch/x86_64/kernel/smboot.c, function detect_siblings(),
> 
> because smp_num_siblings is always =1,  so several lines can 
> be removed.
> 
>         for_each_online_cpu (cpu) {
>                 struct cpuinfo_x86 *c = cpu_data + cpu;
> *R*                int siblings = 0;
>                 int i;
> *R*                if (smp_num_siblings > 1) {
> *R*                        for_each_online_cpu (i) {
> *R*                                if (cpu_core_id[cpu] == 
> cpu_core_id[i]) {
> *R*                                        siblings++;
> *R*                                        cpu_set(i, 
> cpu_sibling_map[cpu]);
> *R*                                }
> *R*                        }
> *R*                } else {
> *R*                      siblings++;
>                         cpu_set(cpu, cpu_sibling_map[cpu]);
> *R*                }
> 
> 
>  *R*               if (siblings != smp_num_siblings) {
>  *R*                       printk(KERN_WARNING
>  *R*              "WARNING: %d siblings found for CPU%d, 
> should be %d\n",
>  *R*                              siblings, cpu, smp_num_siblings);
>  *R*                       smp_num_siblings = siblings;
>  *R*               }
> 
> Also I found the workaround for hang on second node is adding 
> one line in setup.c
>         /* Low order bits define the core id (index of core 
> in socket) */
>         cpu_core_id[cpu] = phys_proc_id[cpu] & ((1 << bits)-1);
>         /* Convert the APIC ID into the socket ID */
>         phys_proc_id[cpu] >>= bits;
> +        printk(KERN_INFO "  CPU %d(%d)  phys_proc_id %d  Core %d\n", 
> +               cpu, c->x86_num_cores, phys_proc_id[cpu], 
> + cpu_core_id[cpu]);
> 
> My compile environment SUSE 9.2 gcc version is 3.3.3???
> 
> YH
> 
> -
> To unsubscribe from this list: send the line "unsubscribe 
> linux-kernel" in the body of a message to 
> majordomo@vger.kernel.org More majordomo info at  
> http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

Re: 2.6.12-rc5 is broken in nvidia Ck804 Opteron MB/with dual cor e dual way

[Andi Kleen]

On Wed, Jun 01, 2005 at 08:16:35PM -0700, YhLu wrote:
> andi,
> 
> in arch/x86_64/kernel/smboot.c, function detect_siblings(),
> 
> because smp_num_siblings is always =1,  so several lines can be removed.

What do you mean? On intel systems with HyperThreading it is > 1.
> 
>         for_each_online_cpu (cpu) {
>                 struct cpuinfo_x86 *c = cpu_data + cpu;
> *R*                int siblings = 0;
>                 int i;
> *R*                if (smp_num_siblings > 1) {
> *R*                        for_each_online_cpu (i) {
> *R*                                if (cpu_core_id[cpu] == cpu_core_id[i]) {
> *R*                                        siblings++;
> *R*                                        cpu_set(i, cpu_sibling_map[cpu]);
> *R*                                }
> *R*                        }
> *R*                } else {
> *R*                      siblings++;
>                         cpu_set(cpu, cpu_sibling_map[cpu]);
> *R*                }
> 
> 
>  *R*               if (siblings != smp_num_siblings) {
>  *R*                       printk(KERN_WARNING
>  *R*              "WARNING: %d siblings found for CPU%d, should be %d\n",
>  *R*                              siblings, cpu, smp_num_siblings);
>  *R*                       smp_num_siblings = siblings;
>  *R*               }
> 
> Also I found the workaround for hang on second node is
> adding one line in setup.c
>         /* Low order bits define the core id (index of core in socket) */
>         cpu_core_id[cpu] = phys_proc_id[cpu] & ((1 << bits)-1);
>         /* Convert the APIC ID into the socket ID */
>         phys_proc_id[cpu] >>= bits;
> +        printk(KERN_INFO "  CPU %d(%d)  phys_proc_id %d  Core %d\n", 
> +               cpu, c->x86_num_cores, phys_proc_id[cpu], cpu_core_id[cpu]);

That would just change the timing.

-Andi

Re: 2.6.12-rc5 is broken in nvidia Ck804 Opteron MB/with dual cor e dual way

[Andi Kleen]

On Wed, Jun 01, 2005 at 08:26:00PM -0700, YhLu wrote:
> Brought up 4 CPUs
>     CPU 0, cpu_sibling_map[0]= 1 
>     CPU 0, cpu_core_map[0]= 3 
>     CPU 1, cpu_sibling_map[1]= 2 
>     CPU 1, cpu_core_map[1]= 3 
>     CPU 2, cpu_sibling_map[2]= 4 
>     CPU 2, cpu_core_map[2]= c 
>     CPU 3, cpu_sibling_map[3]= 8 
>     CPU 3, cpu_core_map[3]= c 
> are the cpu_sibling_map[] right? 

Yes it is correct. A CPU is always a sibling of itself.

-Andi

RE: 2.6.12-rc5 is broken in nvidia Ck804 Opteron MB/with dual cor e dual way

[YhLu]

So core1 is not the sibling of core0?

YH 

> -----Original Message-----
> From: Andi Kleen [mailto:ak@muc.de] 
> Sent: Thursday, June 02, 2005 11:52 AM
> To: YhLu
> Cc: linux-kernel@vger.kernel.org
> Subject: Re: 2.6.12-rc5 is broken in nvidia Ck804 Opteron 
> MB/with dual cor e dual way
> 
> On Wed, Jun 01, 2005 at 08:26:00PM -0700, YhLu wrote:
> > Brought up 4 CPUs
> >     CPU 0, cpu_sibling_map[0]= 1 
> >     CPU 0, cpu_core_map[0]= 3 
> >     CPU 1, cpu_sibling_map[1]= 2 
> >     CPU 1, cpu_core_map[1]= 3 
> >     CPU 2, cpu_sibling_map[2]= 4 
> >     CPU 2, cpu_core_map[2]= c 
> >     CPU 3, cpu_sibling_map[3]= 8 
> >     CPU 3, cpu_core_map[3]= c
> > are the cpu_sibling_map[] right? 
> 
> Yes it is correct. A CPU is always a sibling of itself.
> 
> -Andi
> 

RE: 2.6.12-rc5 is broken in nvidia Ck804 Opteron MB/with dual cor e dual way

[YhLu]

Really?,  smp_num_siblings is global variable and initially is set 1.

YH

> -----Original Message-----
> From: Andi Kleen [mailto:ak@muc.de] 
> Sent: Thursday, June 02, 2005 11:51 AM
> To: YhLu
> Cc: linux-kernel@vger.kernel.org
> Subject: Re: 2.6.12-rc5 is broken in nvidia Ck804 Opteron 
> MB/with dual cor e dual way
> 
> On Wed, Jun 01, 2005 at 08:16:35PM -0700, YhLu wrote:
> > andi,
> > 
> > in arch/x86_64/kernel/smboot.c, function detect_siblings(),
> > 
> > because smp_num_siblings is always =1,  so several lines 
> can be removed.
> 
> What do you mean? On intel systems with HyperThreading it is > 1.

Re: 2.6.12-rc5 is broken in nvidia Ck804 Opteron MB/with dual cor e dual way

[Dave Jones]

On Thu, Jun 02, 2005 at 11:56:25AM -0700, YhLu wrote:
 > Really?,  smp_num_siblings is global variable and initially is set 1.

Why shouldn't it be ? It would only make sense to make it
non-global if we supported a configuration where different
CPUs had different numbers of siblings, which we don't.
(And I'm fairly sure that AMD/Intel don't/won't either)

		Dave

Re: 2.6.12-rc5 is broken in nvidia Ck804 Opteron MB/with dual cor e dual way

[Ashok Raj]

On Thu, Jun 02, 2005 at 11:56:25AM -0700, YhLu wrote:
> 
>    Really?,  smp_num_siblings is global variable and initially is set 1.
> 
>    YH
> 
But detect_ht() can override it.. thats just the start value.

try cscope :-)

Cheers,
ashok

RE: 2.6.12-rc5 is broken in nvidia Ck804 Opteron MB/with dual cor e dual way

[YhLu]

        cpuid(1, &eax, &ebx, &ecx, &edx);
        smp_num_siblings = (ebx & 0xff0000) >> 16;

For amd dual core, smp_num_siblings is set to 1, and it mean has two cores.

                seq_printf(m, "siblings\t: %d\n",
                                c->x86_num_cores * smp_num_siblings);

for Intel it would be 
	c->x86_num_cores  is 2 and smp_num_siblings is 2 too....
	so every core will be HT....


Function 0000_0001[EBX]
EBX[23:16] Logical Processor Count. If CPUID Fn[8000_0001, 0000_0001][EDX:
HTT, ECX:
CMPLegacy] = 11b, then this field indicates the number of CPU cores in the
processor.
Otherwise, this field is reserved.

what is intel value about cpuid(1) ebx [23:16], when the CPU is dual core,
but HT is disabled.
1?

YH

> -----Original Message-----
> From: Ashok Raj [mailto:ashok.raj@intel.com] 
> Sent: Thursday, June 02, 2005 12:07 PM
> To: YhLu
> Cc: Andi Kleen; linux-kernel@vger.kernel.org
> Subject: Re: 2.6.12-rc5 is broken in nvidia Ck804 Opteron 
> MB/with dual cor e dual way
> 
> On Thu, Jun 02, 2005 at 11:56:25AM -0700, YhLu wrote:
> > 
> >    Really?,  smp_num_siblings is global variable and 
> initially is set 1.
> > 
> >    YH
> > 
> But detect_ht() can override it.. thats just the start value.
> 
> try cscope :-)
> 
> Cheers,
> ashok
> 

RE: 2.6.12-rc5 is broken in nvidia Ck804 Opteron MB/with dual cor e dual way

[YhLu]

in  detect_ht
        if (!cpu_has(c, X86_FEATURE_HT) || cpu_has(c,
X86_FEATURE_CMP_LEGACY))
                return;

cpu_has(c, X86_FEATURE_CMP_LEGACY) for AMD is 1, so smp_num_siblings is not
changed and it is still initial value. (1)....

YH

> -----Original Message-----
> From: YhLu 
> Sent: Thursday, June 02, 2005 1:42 PM
> To: Ashok Raj
> Cc: Andi Kleen; linux-kernel@vger.kernel.org
> Subject: RE: 2.6.12-rc5 is broken in nvidia Ck804 Opteron 
> MB/with dual cor e dual way
> 
>         cpuid(1, &eax, &ebx, &ecx, &edx);
>         smp_num_siblings = (ebx & 0xff0000) >> 16;
> 
> For amd dual core, smp_num_siblings is set to 1, and it mean 
> has two cores.
> 
>                 seq_printf(m, "siblings\t: %d\n",
>                                 c->x86_num_cores * smp_num_siblings);
> 
> for Intel it would be 
> 	c->x86_num_cores  is 2 and smp_num_siblings is 2 too....
> 	so every core will be HT....
> 
> 
> Function 0000_0001[EBX]
> EBX[23:16] Logical Processor Count. If CPUID Fn[8000_0001, 
> 0000_0001][EDX:
> HTT, ECX:
> CMPLegacy] = 11b, then this field indicates the number of CPU 
> cores in the processor.
> Otherwise, this field is reserved.
> 
> what is intel value about cpuid(1) ebx [23:16], when the CPU 
> is dual core, but HT is disabled.
> 1?
> 
> YH
> 
> > -----Original Message-----
> > From: Ashok Raj [mailto:ashok.raj@intel.com]
> > Sent: Thursday, June 02, 2005 12:07 PM
> > To: YhLu
> > Cc: Andi Kleen; linux-kernel@vger.kernel.org
> > Subject: Re: 2.6.12-rc5 is broken in nvidia Ck804 Opteron 
> MB/with dual 
> > cor e dual way
> > 
> > On Thu, Jun 02, 2005 at 11:56:25AM -0700, YhLu wrote:
> > > 
> > >    Really?,  smp_num_siblings is global variable and
> > initially is set 1.
> > > 
> > >    YH
> > > 
> > But detect_ht() can override it.. thats just the start value.
> > 
> > try cscope :-)
> > 
> > Cheers,
> > ashok
> > 
> -
> To unsubscribe from this list: send the line "unsubscribe 
> linux-kernel" in the body of a message to 
> majordomo@vger.kernel.org More majordomo info at  
> http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

RE: 2.6.12-rc5 is broken in nvidia Ck804 Opteron MB/with dual cor e dual way

[YhLu]

So siblings should be in one core...? that make sense....

YH 

> -----Original Message-----
> From: Andi Kleen [mailto:ak@muc.de] 
> Sent: Thursday, June 02, 2005 11:52 AM
> To: YhLu
> Cc: linux-kernel@vger.kernel.org
> Subject: Re: 2.6.12-rc5 is broken in nvidia Ck804 Opteron 
> MB/with dual cor e dual way
> 
> On Wed, Jun 01, 2005 at 08:26:00PM -0700, YhLu wrote:
> > Brought up 4 CPUs
> >     CPU 0, cpu_sibling_map[0]= 1 
> >     CPU 0, cpu_core_map[0]= 3 
> >     CPU 1, cpu_sibling_map[1]= 2 
> >     CPU 1, cpu_core_map[1]= 3 
> >     CPU 2, cpu_sibling_map[2]= 4 
> >     CPU 2, cpu_core_map[2]= c 
> >     CPU 3, cpu_sibling_map[3]= 8 
> >     CPU 3, cpu_core_map[3]= c
> > are the cpu_sibling_map[] right? 
> 
> Yes it is correct. A CPU is always a sibling of itself.
> 
> -Andi
> 

RE: 2.6.12-rc5 is broken in nvidia Ck804 Opteron MB/with dual cor e dual way

[YhLu]

Can you duplicate the timing problem?

YH

> > Also I found the workaround for hang on second node is 
> adding one line 
> > in setup.c
> >         /* Low order bits define the core id (index of core 
> in socket) */
> >         cpu_core_id[cpu] = phys_proc_id[cpu] & ((1 << bits)-1);
> >         /* Convert the APIC ID into the socket ID */
> >         phys_proc_id[cpu] >>= bits;
> > +        printk(KERN_INFO "  CPU %d(%d)  phys_proc_id %d  
> Core %d\n", 
> > +               cpu, c->x86_num_cores, phys_proc_id[cpu], 
> > + cpu_core_id[cpu]);
> 
> That would just change the timing.
> 
> -Andi
> 

Re: 2.6.12-rc5 is broken in nvidia Ck804 Opteron MB/with dual cor e dual way

[Andi Kleen]

On Thu, Jun 02, 2005 at 01:42:00PM -0700, YhLu wrote:
>         cpuid(1, &eax, &ebx, &ecx, &edx);
>         smp_num_siblings = (ebx & 0xff0000) >> 16;
> 
> For amd dual core, smp_num_siblings is set to 1, and it mean has two cores.
> 
>                 seq_printf(m, "siblings\t: %d\n",
>                                 c->x86_num_cores * smp_num_siblings);

Yes; that is done so that the scheduler does not set up
SMT scheduling, which is suboptimal for dual core.

Also some time ago the scheduler domain setup tended to break
with SMT and NUMA combined, but that is probably fixed
now. But it also made it advisable to not set the sibling count.

> 
> for Intel it would be 
> 	c->x86_num_cores  is 2 and smp_num_siblings is 2 too....
> 	so every core will be HT....

If that was true, then a true DC+HT machine would report 4.

I doubt it is, but Suresh can probably clarify.

> Function 0000_0001[EBX]
> EBX[23:16] Logical Processor Count. If CPUID Fn[8000_0001, 0000_0001][EDX:
> HTT, ECX:
> CMPLegacy] = 11b, then this field indicates the number of CPU cores in the
> processor.
> Otherwise, this field is reserved.
> 
> what is intel value about cpuid(1) ebx [23:16], when the CPU is dual core,
> but HT is disabled.
> 1?

-Andi

Re: 2.6.12-rc5 is broken in nvidia Ck804 Opteron MB/with dual cor e dual way

[Andi Kleen]

On Thu, Jun 02, 2005 at 02:33:06PM -0700, YhLu wrote:
> So siblings should be in one core...? that make sense....

Yes. The whole terminology is confusing for historical reasons.
If we started again with the naming game there would be probably 
different names.

To make it even more confusing while it is different internally
in the kernel, CPUID and /proc/cpuinfo fake them to the same
to the world to be compatible with old software who e.g.
reads this for licensing purposes.

BTW I doubt these are related to the hang you are chasing
because if you grep the kernel cpu_core_map is never used
in the kernel at all except for cpuinfo. This means
it is unlikely to cause any hangs.

-Andi

Re: 2.6.12-rc5 is broken in nvidia Ck804 Opteron MB/with dual cor e dual way

[Siddha, Suresh B]

On Fri, Jun 03, 2005 at 06:09:32PM +0200, Andi Kleen wrote:
> On Thu, Jun 02, 2005 at 01:42:00PM -0700, YhLu wrote:
> > 
> > for Intel it would be 
> > 	c->x86_num_cores  is 2 and smp_num_siblings is 2 too....
> > 	so every core will be HT....
> 
> If that was true, then a true DC+HT machine would report 4.
> 
> I doubt it is, but Suresh can probably clarify.

On  a DC+HT, x86_num_cores will be 2 and kernel will set smp_num_siblings 
to 2 (even though number of logical processors per physical package as
seen by cpuid.1.EBX.bits[23:16] will be 4).
Remember, smp_num_siblings in kernel will just represent HT logical siblings.

If CPU is just DC capable, cpuid.1.EBX.bits[23:16] will return '2' and
number of cores computation from cpuid.4.EAX.bits[31:26] will be 2. 
And kernel will set x86_num_cores to 2 and smp_num_siblings will be set to 1

If CPU is just HT capable,cpuid.1.EBX.bits[23:16] will return '2' and
number of cores computation from cpuid.4.EAX.bits[31:26] will be 1.
And kernel will set x86_num_cores to 1 and smp_num_siblings will be set to 2.

> > Function 0000_0001[EBX]
> > EBX[23:16] Logical Processor Count. If CPUID Fn[8000_0001, 0000_0001][EDX:
> > HTT, ECX:
> > CMPLegacy] = 11b, then this field indicates the number of CPU cores in the
> > processor.
> > Otherwise, this field is reserved.
> > 
> > what is intel value about cpuid(1) ebx [23:16], when the CPU is dual core,
> > but HT is disabled.
> > 1?

If CPU is DC+HT capable, but HT is disabled in bios, cpuid.1.EBX.bits[23:16] 
will still be 4 and during bootup, kernel will figure out that no HT
siblings came up and will reset smp_num_siblings to 1.

I am planning to do more cleanup post 2.6.12, so that smp_num_siblings will
be moved to per cpuinfo.

thanks,
suresh

Re: 2.6.12-rc5 is broken in nvidia Ck804 Opteron MB/with dual cor e dual way

[Siddha, Suresh B]

On Thu, Jun 02, 2005 at 03:03:50PM -0400, Dave Jones wrote:
> On Thu, Jun 02, 2005 at 11:56:25AM -0700, YhLu wrote:
>  > Really?,  smp_num_siblings is global variable and initially is set 1.
> 
> Why shouldn't it be ? It would only make sense to make it
> non-global if we supported a configuration where different
> CPUs had different numbers of siblings, which we don't.
> (And I'm fairly sure that AMD/Intel don't/won't either)

It will help if someone uses "maxcpus" boot param or if someone
wants to disable/enable certain logical siblings using HOTPLUG.

That is conceptually more cleaner too. Will do it post 2.6.12.

thanks,
suresh