RE: 2.6.13-rc2 with dual way dual core ck804 MB

RE: 2.6.13-rc2 with dual way dual core ck804 MB

[YhLu]

andi,

please refer the patch, it will move cpu_set(, cpu_callin_map) from
smi_callin to start_secondary.

--- /home/yhlu/xx1/linux-2.6.13-rc2/arch/x86_64/kernel/smpboot.c.orig
2005-07-06 18:41:16.789767168 -0700
+++ /home/yhlu/xx1/linux-2.6.13-rc2/arch/x86_64/kernel/smpboot.c
2005-07-06 18:45:11.923021480 -0700
@@ -442,7 +442,7 @@
        /*
         * Allow the master to continue.
         */
-       cpu_set(cpuid, cpu_callin_map);
+//     cpu_set(cpuid, cpu_callin_map); // moved to start_secondary by yhlu
 }

 static inline void set_cpu_sibling_map(int cpu)
@@ -529,8 +529,11 @@
        /* Wait for TSC sync to not schedule things before.
           We still process interrupts, which could see an inconsistent
           time in that window unfortunately. */
+
        tsc_sync_wait();

+       cpu_set(smp_processor_id(), cpu_callin_map); // moved from
smp_callin by yhlu
+
        cpu_idle();
 }

the other solution will be change cpu_callin_map to cpu_online_map in
do_boot_cpu

                /*
                 * allow APs to start initializing.
                 */
                Dprintk("Before Callout %d.\n", cpu);
                cpu_set(cpu, cpu_callout_map);
                Dprintk("After Callout %d.\n", cpu);

                /*
                 * Wait 5s total for a response
                 */
                for (timeout = 0; timeout < 50000; timeout++) {
                        if (cpu_isset(cpu, cpu_callin_map))
--------------------------> cpu_online_map
                                break;  /* It has booted */
                        udelay(100);
                }

                if (cpu_isset(cpu, cpu_callin_map)) {
--------------------------------> cpu_online_map
                        /* number CPUs logically, starting from 1 (BSP is 0)
*/
                        Dprintk("CPU has booted.\n");
                } else {
                        boot_error = 1;
                        if (*((volatile unsigned char
*)phys_to_virt(SMP_TRAMPOLINE_BASE))
                                        == 0xA5)
                                /* trampoline started but...? */
                                printk("Stuck ??\n");
                        else
                                /* trampoline code not run */
                                printk("Not responding.\n");
#if APIC_DEBUG
                        inquire_remote_apic(apicid);
#endif
                }


the result will be

Booting processor 1/1 rip 6000 rsp ffff81013ff89f58
Initializing CPU#1
masked ExtINT on CPU#1
Calibrating delay using timer specific routine.. 4422.98 BogoMIPS
(lpj=8845965)
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 1024K (64 bytes/line)
CPU 1(2) -> Node 0 -> Core 1
 stepping 00
CPU 1: Syncing TSC to CPU 0.
sync_master: 1 smp_processor_id() = 00, boot_cpu_id= 00
sync_master: 2 smp_processor_id() = 00, boot_cpu_id= 00
CPU 1: synchronized TSC with CPU 0 (last diff 0 cycles, maxerr 595 cycles)
---------------------> it is in right place.
Booting processor 2/2 rip 6000 rsp ffff81023ff1df58
Initializing CPU#2
masked ExtINT on CPU#2
Calibrating delay using timer specific routine.. 4422.99 BogoMIPS
(lpj=8845997)
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 1024K (64 bytes/line)
CPU 2(2) -> Node 1 -> Core 0
 stepping 00
CPU 2: Syncing TSC to CPU 0.
sync_master: 1 smp_processor_id() = 00, boot_cpu_id= 00
sync_master: 1 smp_processor_id() = 01, boot_cpu_id= 00
sync_master: 2 smp_processor_id() = 00, boot_cpu_id= 00
CPU 2: synchronized TSC with CPU 0 (last diff -4 cycles, maxerr 1097 cycles)
Booting processor 3/3 rip 6000 rsp ffff81013ff53f58
Initializing CPU#3
masked ExtINT on CPU#3
Calibrating delay using timer specific routine.. 4423.03 BogoMIPS
(lpj=8846075)
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 1024K (64 bytes/line)
CPU 3(2) -> Node 1 -> Core 1
 stepping 00
CPU 3: Syncing TSC to CPU 0.
sync_master: 1 smp_processor_id() = 00, boot_cpu_id= 00
sync_master: 1 smp_processor_id() = 01, boot_cpu_id= 00
sync_master: 1 smp_processor_id() = 02, boot_cpu_id= 00
sync_master: 2 smp_processor_id() = 00, boot_cpu_id= 00
CPU 3: synchronized TSC with CPU 0 (last diff -4 cycles, maxerr 1097 cycles)
Brought up 4 CPUs


> -----Original Message-----
> From: YhLu 
> Sent: Wednesday, July 06, 2005 3:25 PM
> To: Andi Kleen
> Cc: Peter Buckingham; linux-kernel@vger.kernel.org
> Subject: 2.6.13-rc2 with dual way dual core ck804 MB
> 
> andi,
> 
> the core1/node0 take a long while to get TSC synchronized. Is 
> it normal?
> i guess
> "CPU 1: synchronized TSC with CPU 0"  should be just after 
> "CPU 1: Syncing TSC to CPU0"
> 
> YH
> 
> 
> cpu 1: setting up apic clock
> cpu 1: enabling apic timer
> CPU 1: Syncing TSC to CPU 0.
> CPU has booted.
> waiting for cpu 1
> 
> cpu 2: setting up apic clock
> cpu 2: enabling apic timer
> CPU 2: Syncing TSC to CPU 0.
> CPU 2: synchronized TSC with CPU 0 (last diff -4 cycles, 
> maxerr 1097 cycles) CPU has booted.
> waiting for cpu 2
> 
> cpu 3: setting up apic clock
> cpu 3: enabling apic timer
> CPU 3: Syncing TSC to CPU 0.
> CPU 3: synchronized TSC with CPU 0 (last diff 1 cycles, 
> maxerr 1087 cycles) CPU has booted.
> waiting for cpu 3
> 
> testing NMI watchdog ... CPU#1: NMI appears to be stuck (1->1)!
> checking if image is initramfs...<6>CPU 1: synchronized TSC 
> with CPU 0 (last diff 0 cycles, maxerr 595 cycles) it isn't 
> (no cpio magic); looks like an initrd
> 
> 
> the
> -
> To unsubscribe from this list: send the line "unsubscribe 
> linux-kernel" in the body of a message to 
> majordomo@vger.kernel.org More majordomo info at  
> http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

Re: 2.6.13-rc2 with dual way dual core ck804 MB

[Mike Waychison]

YhLu wrote:
> andi,
> 
> please refer the patch, it will move cpu_set(, cpu_callin_map) from
> smi_callin to start_secondary.


This patch fixes an apparent race / lockup on our 2-way dual cores (when 
applied against 2.6.12.3).  The machine was locking up after 
"Initializing CPU#2".

Mike Waychison

> 
> --- /home/yhlu/xx1/linux-2.6.13-rc2/arch/x86_64/kernel/smpboot.c.orig
> 2005-07-06 18:41:16.789767168 -0700
> +++ /home/yhlu/xx1/linux-2.6.13-rc2/arch/x86_64/kernel/smpboot.c
> 2005-07-06 18:45:11.923021480 -0700
> @@ -442,7 +442,7 @@
>         /*
>          * Allow the master to continue.
>          */
> -       cpu_set(cpuid, cpu_callin_map);
> +//     cpu_set(cpuid, cpu_callin_map); // moved to start_secondary by yhlu
>  }
> 
>  static inline void set_cpu_sibling_map(int cpu)
> @@ -529,8 +529,11 @@
>         /* Wait for TSC sync to not schedule things before.
>            We still process interrupts, which could see an inconsistent
>            time in that window unfortunately. */
> +
>         tsc_sync_wait();
> 
> +       cpu_set(smp_processor_id(), cpu_callin_map); // moved from
> smp_callin by yhlu
> +
>         cpu_idle();
>  }
> 
> the other solution will be change cpu_callin_map to cpu_online_map in
> do_boot_cpu
> 
>                 /*
>                  * allow APs to start initializing.
>                  */
>                 Dprintk("Before Callout %d.\n", cpu);
>                 cpu_set(cpu, cpu_callout_map);
>                 Dprintk("After Callout %d.\n", cpu);
> 
>                 /*
>                  * Wait 5s total for a response
>                  */
>                 for (timeout = 0; timeout < 50000; timeout++) {
>                         if (cpu_isset(cpu, cpu_callin_map))
> --------------------------> cpu_online_map
>                                 break;  /* It has booted */
>                         udelay(100);
>                 }
> 
>                 if (cpu_isset(cpu, cpu_callin_map)) {
> --------------------------------> cpu_online_map
>                         /* number CPUs logically, starting from 1 (BSP is 0)
> */
>                         Dprintk("CPU has booted.\n");
>                 } else {
>                         boot_error = 1;
>                         if (*((volatile unsigned char
> *)phys_to_virt(SMP_TRAMPOLINE_BASE))
>                                         == 0xA5)
>                                 /* trampoline started but...? */
>                                 printk("Stuck ??\n");
>                         else
>                                 /* trampoline code not run */
>                                 printk("Not responding.\n");
> #if APIC_DEBUG
>                         inquire_remote_apic(apicid);
> #endif
>                 }
> 
> 
> the result will be
> 
> Booting processor 1/1 rip 6000 rsp ffff81013ff89f58
> Initializing CPU#1
> masked ExtINT on CPU#1
> Calibrating delay using timer specific routine.. 4422.98 BogoMIPS
> (lpj=8845965)
> CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
> CPU: L2 Cache: 1024K (64 bytes/line)
> CPU 1(2) -> Node 0 -> Core 1
>  stepping 00
> CPU 1: Syncing TSC to CPU 0.
> sync_master: 1 smp_processor_id() = 00, boot_cpu_id= 00
> sync_master: 2 smp_processor_id() = 00, boot_cpu_id= 00
> CPU 1: synchronized TSC with CPU 0 (last diff 0 cycles, maxerr 595 cycles)
> ---------------------> it is in right place.
> Booting processor 2/2 rip 6000 rsp ffff81023ff1df58
> Initializing CPU#2
> masked ExtINT on CPU#2
> Calibrating delay using timer specific routine.. 4422.99 BogoMIPS
> (lpj=8845997)
> CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
> CPU: L2 Cache: 1024K (64 bytes/line)
> CPU 2(2) -> Node 1 -> Core 0
>  stepping 00
> CPU 2: Syncing TSC to CPU 0.
> sync_master: 1 smp_processor_id() = 00, boot_cpu_id= 00
> sync_master: 1 smp_processor_id() = 01, boot_cpu_id= 00
> sync_master: 2 smp_processor_id() = 00, boot_cpu_id= 00
> CPU 2: synchronized TSC with CPU 0 (last diff -4 cycles, maxerr 1097 cycles)
> Booting processor 3/3 rip 6000 rsp ffff81013ff53f58
> Initializing CPU#3
> masked ExtINT on CPU#3
> Calibrating delay using timer specific routine.. 4423.03 BogoMIPS
> (lpj=8846075)
> CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
> CPU: L2 Cache: 1024K (64 bytes/line)
> CPU 3(2) -> Node 1 -> Core 1
>  stepping 00
> CPU 3: Syncing TSC to CPU 0.
> sync_master: 1 smp_processor_id() = 00, boot_cpu_id= 00
> sync_master: 1 smp_processor_id() = 01, boot_cpu_id= 00
> sync_master: 1 smp_processor_id() = 02, boot_cpu_id= 00
> sync_master: 2 smp_processor_id() = 00, boot_cpu_id= 00
> CPU 3: synchronized TSC with CPU 0 (last diff -4 cycles, maxerr 1097 cycles)
> Brought up 4 CPUs
> 
> 
> 
>>-----Original Message-----
>>From: YhLu 
>>Sent: Wednesday, July 06, 2005 3:25 PM
>>To: Andi Kleen
>>Cc: Peter Buckingham; linux-kernel@vger.kernel.org
>>Subject: 2.6.13-rc2 with dual way dual core ck804 MB
>>
>>andi,
>>
>>the core1/node0 take a long while to get TSC synchronized. Is 
>>it normal?
>>i guess
>>"CPU 1: synchronized TSC with CPU 0"  should be just after 
>>"CPU 1: Syncing TSC to CPU0"
>>
>>YH
>>
>>
>>cpu 1: setting up apic clock
>>cpu 1: enabling apic timer
>>CPU 1: Syncing TSC to CPU 0.
>>CPU has booted.
>>waiting for cpu 1
>>
>>cpu 2: setting up apic clock
>>cpu 2: enabling apic timer
>>CPU 2: Syncing TSC to CPU 0.
>>CPU 2: synchronized TSC with CPU 0 (last diff -4 cycles, 
>>maxerr 1097 cycles) CPU has booted.
>>waiting for cpu 2
>>
>>cpu 3: setting up apic clock
>>cpu 3: enabling apic timer
>>CPU 3: Syncing TSC to CPU 0.
>>CPU 3: synchronized TSC with CPU 0 (last diff 1 cycles, 
>>maxerr 1087 cycles) CPU has booted.
>>waiting for cpu 3
>>
>>testing NMI watchdog ... CPU#1: NMI appears to be stuck (1->1)!
>>checking if image is initramfs...<6>CPU 1: synchronized TSC 
>>with CPU 0 (last diff 0 cycles, maxerr 595 cycles) it isn't 
>>(no cpio magic); looks like an initrd
>>
>>
>>the
>>-
>>To unsubscribe from this list: send the line "unsubscribe 
>>linux-kernel" in the body of a message to 
>>majordomo@vger.kernel.org More majordomo info at  
>>http://vger.kernel.org/majordomo-info.html
>>Please read the FAQ at  http://www.tux.org/lkml/
>>

Re: [discuss] Re: 2.6.13-rc2 with dual way dual core ck804 MB

[Andi Kleen]

On Wed, Aug 10, 2005 at 04:14:19PM -0700, Mike Waychison wrote:
> YhLu wrote:
> >andi,
> >
> >please refer the patch, it will move cpu_set(, cpu_callin_map) from
> >smi_callin to start_secondary.
> 
> 
> This patch fixes an apparent race / lockup on our 2-way dual cores (when 
> applied against 2.6.12.3).  The machine was locking up after 
> "Initializing CPU#2".

The real solution for this issue is the smp_call_function_single patch from Eric
that I reposted yesterday. Yh's patch just changed the timing slightly.


-Andi

Re: [discuss] Re: 2.6.13-rc2 with dual way dual core ck804 MB

[yhlu]

andi,

you can see the difference with the patch
Booting processor 1/1 rip 6000 rsp ffff810181c61f58
Initializing CPU#1
masked ExtINT on CPU#1
Calibrating delay using timer specific routine.. 4000.31 BogoMIPS (lpj=8000624)
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 1024K (64 bytes/line)
 stepping 0a
CPU 1: Syncing TSC to CPU 0.
CPU 1: synchronized TSC with CPU 0 (last diff 0 cycles, maxerr 886 cycles)
Booting processor 2/2 rip 6000 rsp ffff81017ffa3f58
Initializing CPU#2
masked ExtINT on CPU#2
Calibrating delay using timer specific routine.. 4000.30 BogoMIPS (lpj=8000605)
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 1024K (64 bytes/line)
 stepping 0a
CPU 2: Syncing TSC to CPU 0.
CPU 2: synchronized TSC with CPU 0 (last diff 1 cycles, maxerr 901 cycles)
Booting processor 3/3 rip 6000 rsp ffff8101fffa9f58
Initializing CPU#3
masked ExtINT on CPU#3
Calibrating delay using timer specific routine.. 4000.31 BogoMIPS (lpj=8000622)
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 1024K (64 bytes/line)
 stepping 0a
CPU 3: Syncing TSC to CPU 0.
CPU 3: synchronized TSC with CPU 0 (last diff -3 cycles, maxerr 1504 cycles)
Brought up 4 CPUs


without the patch
Booting processor 1/4 APIC 0x1
Initializing CPU#1
masked ExtINT on CPU#1
Calibrating delay using timer specific routine.. 4000.30 BogoMIPS (lpj=8000608)
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 1024K (64 bytes/line)
CPU 1(1) -> Node 1 -> Core 0
 stepping 0a
CPU 1: Syncing TSC to CPU 0.
Booting processor 2/4 APIC 0x2
Initializing CPU#2
masked ExtINT on CPU#2
CPU 1: synchronized TSC with CPU 0 (last diff 1 cycles, maxerr 893 cycles)
Calibrating delay using timer specific routine.. 4000.36 BogoMIPS (lpj=8000724)
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 1024K (64 bytes/line)
CPU 2(1) -> Node 2 -> Core 0
 stepping 0a
CPU 2: Syncing TSC to CPU 0.
Booting processor 3/4 APIC 0x3
Initializing CPU#3
masked ExtINT on CPU#3
CPU 2: synchronized TSC with CPU 0 (last diff 0 cycles, maxerr 904 cycles)
Calibrating delay using timer specific routine.. 4000.16 BogoMIPS (lpj=8000335)
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 1024K (64 bytes/line)
CPU 3(1) -> Node 3 -> Core 0
 stepping 0a
CPU 3: Syncing TSC to CPU 0.
Brought up 4 CPUs
time.c: Using PIT/TSC based timekeeping.
testing NMI watchdog ... OK.
checking if image is initramfs...<6>CPU 3: synchronized TSC with CPU 0
(last diff -18 cycles, maxerr 1504 cycles)
it isn't (no cpio magic); looks like an initrd

So my patch still can be used with Eric's, It just serialize the
TSC_SYNC between cpu.

I wonder it you can refine to make TSC_SYNC serialize that beteen CPU.
That will make
CPU X:synchronized TSC ... 
in fixed postion and timming.

YH


On 8/10/05, Andi Kleen <ak@suse.de> wrote:
> On Wed, Aug 10, 2005 at 04:14:19PM -0700, Mike Waychison wrote:
> > YhLu wrote:
> > >andi,
> > >
> > >please refer the patch, it will move cpu_set(, cpu_callin_map) from
> > >smi_callin to start_secondary.
> >
> >
> > This patch fixes an apparent race / lockup on our 2-way dual cores (when
> > applied against 2.6.12.3).  The machine was locking up after
> > "Initializing CPU#2".
> 
> The real solution for this issue is the smp_call_function_single patch from Eric
> that I reposted yesterday. Yh's patch just changed the timing slightly.
> 
> 
> -Andi
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: [discuss] Re: 2.6.13-rc2 with dual way dual core ck804 MB

[Andi Kleen]

> So my patch still can be used with Eric's, It just serialize the
> TSC_SYNC between cpu.
> 
> I wonder it you can refine to make TSC_SYNC serialize that beteen CPU.
> That will make
> CPU X:synchronized TSC ... 
> in fixed postion and timming.

Why would we want that? 

Boot time is critical so it's better to do things asynchronous.

-Andi

Re: [discuss] Re: 2.6.13-rc2 with dual way dual core ck804 MB

[yhlu]

In LinuxBIOS, we could init_ecc asynchronous and the time reduced from
8x to 2.1x for 8 ways system. 1x mean 5s for 4G in one cpu. If 16G
will take 20s.

for TSC_SYNC asynchronous maybe you can get back 0.1s...

YH

On 8/10/05, Andi Kleen <ak@suse.de> wrote:
> > So my patch still can be used with Eric's, It just serialize the
> > TSC_SYNC between cpu.
> >
> > I wonder it you can refine to make TSC_SYNC serialize that beteen CPU.
> > That will make
> > CPU X:synchronized TSC ...
> > in fixed postion and timming.
> 
> Why would we want that?
> 
> Boot time is critical so it's better to do things asynchronous.
> 
> -Andi
>

Re: [discuss] Re: 2.6.13-rc2 with dual way dual core ck804 MB

[yhlu]

I wonder if you can make the bsp can start the APs callin in the same
time, and make it asynchronous, So you make spare 2s or more.

YH

On 8/10/05, yhlu <yhlu.kernel@gmail.com> wrote:
> In LinuxBIOS, we could init_ecc asynchronous and the time reduced from
> 8x to 2.1x for 8 ways system. 1x mean 5s for 4G in one cpu. If 16G
> will take 20s.
> 
> for TSC_SYNC asynchronous maybe you can get back 0.1s...
> 
> YH
> 
> On 8/10/05, Andi Kleen <ak@suse.de> wrote:
> > > So my patch still can be used with Eric's, It just serialize the
> > > TSC_SYNC between cpu.
> > >
> > > I wonder it you can refine to make TSC_SYNC serialize that beteen CPU.
> > > That will make
> > > CPU X:synchronized TSC ...
> > > in fixed postion and timming.
> >
> > Why would we want that?
> >
> > Boot time is critical so it's better to do things asynchronous.
> >
> > -Andi
> >
>

Re: [discuss] Re: 2.6.13-rc2 with dual way dual core ck804 MB

[Andi Kleen]

On Wed, Aug 10, 2005 at 05:23:31PM -0700, yhlu wrote:
> I wonder if you can make the bsp can start the APs callin in the same
> time, and make it asynchronous, So you make spare 2s or more.

The setting of cpu_callin_map in the AP could be moved earlier yes.
But it's not entirely trivial because there are some races to consider.

And the 1s quiet period on the AP could be probably also reduced
on modern systems. I doubt it is needed on Xeons or Opterons.

-Andi

Re: [discuss] Re: 2.6.13-rc2 with dual way dual core ck804 MB

[yhlu]

Yes, I mean more aggressive

static void __init smp_init(void)
{
        unsigned int i;

        /* FIXME: This should be done in userspace --RR */
        for_each_present_cpu(i) {
                if (num_online_cpus() >= max_cpus)
                        break;
                if (!cpu_online(i))
                        cpu_up(i);
        }


let cpu_up take one array instead of one int.

So  in do_boot_cpu() of smpboot.c
                /*
                 * Wait 5s total for a response
                 */
                for (timeout = 0; timeout < 50000; timeout++) {
                        if (cpu_isset(cpu, cpu_callin_map))
                                break;  /* It has booted */
                        udelay(100);
                }

could wait all be cpu_callin_map is set.

then we can spare more time.

YH


On 8/10/05, Andi Kleen <ak@suse.de> wrote:
> On Wed, Aug 10, 2005 at 05:23:31PM -0700, yhlu wrote:
> > I wonder if you can make the bsp can start the APs callin in the same
> > time, and make it asynchronous, So you make spare 2s or more.
> 
> The setting of cpu_callin_map in the AP could be moved earlier yes.
> But it's not entirely trivial because there are some races to consider.
> 
> And the 1s quiet period on the AP could be probably also reduced
> on modern systems. I doubt it is needed on Xeons or Opterons.
> 
> -Andi
>

Re: [discuss] Re: 2.6.13-rc2 with dual way dual core ck804 MB

[Andi Kleen]

On Wed, Aug 10, 2005 at 05:43:23PM -0700, yhlu wrote:
> Yes, I mean more aggressive
> 
> static void __init smp_init(void)
> {
>         unsigned int i;
> 
>         /* FIXME: This should be done in userspace --RR */
>         for_each_present_cpu(i) {
>                 if (num_online_cpus() >= max_cpus)
>                         break;
>                 if (!cpu_online(i))
>                         cpu_up(i);
>         }
> 
> 
> let cpu_up take one array instead of one int.

It can be done already by just not starting the CPUs and
then do it multithreaded from user space using sysfs with
the CPU hotplug infrastructure. Unfortunately cpu_up
right now has a global semaphore, so it won't save you any
time. However it could be done in parallel with other 
startup jobs.

-Andi

Re: [discuss] Re: 2.6.13-rc2 with dual way dual core ck804 MB

[yhlu]

andi,

is it possible for
after the AP1 call_in is done and before AP1 get in tsc_sync_wait
The AP2 call_in done.  and then AP1 get in tsc_sync_wait and before it
done, AP2 get in tsc_sync_wait too.

sync_master can not figure out from AP1 or AP2 because only have
go[MASTER] and go{SLAVE].

YH

On 8/10/05, Andi Kleen <ak@suse.de> wrote:
> On Wed, Aug 10, 2005 at 05:43:23PM -0700, yhlu wrote:
> > Yes, I mean more aggressive
> >
> > static void __init smp_init(void)
> > {
> >         unsigned int i;
> >
> >         /* FIXME: This should be done in userspace --RR */
> >         for_each_present_cpu(i) {
> >                 if (num_online_cpus() >= max_cpus)
> >                         break;
> >                 if (!cpu_online(i))
> >                         cpu_up(i);
> >         }
> >
> >
> > let cpu_up take one array instead of one int.
> 
> It can be done already by just not starting the CPUs and
> then do it multithreaded from user space using sysfs with
> the CPU hotplug infrastructure. Unfortunately cpu_up
> right now has a global semaphore, so it won't save you any
> time. However it could be done in parallel with other
> startup jobs.
> 
> -Andi
>

Re: [discuss] Re: 2.6.13-rc2 with dual way dual core ck804 MB

[yhlu]

andi,

it seems ia64 is after done with the tsc_sync then set the callin_map.

YH

        if (!(sal_platform_features & IA64_SAL_PLATFORM_FEATURE_ITC_DRIFT)) {
                /*
                 * Synchronize the ITC with the BP.  Need to do this
after irqs are
                 * enabled because ia64_sync_itc() calls
smp_call_function_single(), which
                 * calls spin_unlock_bh(), which calls
spin_unlock_bh(), which calls
                 * local_bh_enable(), which bugs out if irqs are not enabled...
                 */
                Dprintk("Going to syncup ITC with BP.\n");
                ia64_sync_itc(0);
        }

        /*
         * Get our bogomips.
         */
        ia64_init_itm();
        calibrate_delay();
        local_cpu_data->loops_per_jiffy = loops_per_jiffy;

#ifdef CONFIG_IA32_SUPPORT
        ia32_gdt_init();
#endif

        /*
         * Allow the master to continue.
         */
        cpu_set(cpuid, cpu_callin_map);


On 8/11/05, yhlu <yhlu.kernel@gmail.com> wrote:
> andi,
> 
> is it possible for
> after the AP1 call_in is done and before AP1 get in tsc_sync_wait
> The AP2 call_in done.  and then AP1 get in tsc_sync_wait and before it
> done, AP2 get in tsc_sync_wait too.
> 
> sync_master can not figure out from AP1 or AP2 because only have
> go[MASTER] and go{SLAVE].
> 
> YH
> 
> On 8/10/05, Andi Kleen <ak@suse.de> wrote:
> > On Wed, Aug 10, 2005 at 05:43:23PM -0700, yhlu wrote:
> > > Yes, I mean more aggressive
> > >
> > > static void __init smp_init(void)
> > > {
> > >         unsigned int i;
> > >
> > >         /* FIXME: This should be done in userspace --RR */
> > >         for_each_present_cpu(i) {
> > >                 if (num_online_cpus() >= max_cpus)
> > >                         break;
> > >                 if (!cpu_online(i))
> > >                         cpu_up(i);
> > >         }
> > >
> > >
> > > let cpu_up take one array instead of one int.
> >
> > It can be done already by just not starting the CPUs and
> > then do it multithreaded from user space using sysfs with
> > the CPU hotplug infrastructure. Unfortunately cpu_up
> > right now has a global semaphore, so it won't save you any
> > time. However it could be done in parallel with other
> > startup jobs.
> >
> > -Andi
> >
>

Re: [discuss] Re: 2.6.13-rc2 with dual way dual core ck804 MB

[Andi Kleen]

On Thu, Aug 11, 2005 at 11:59:21PM -0700, yhlu wrote:
> andi,
> 
> is it possible for
> after the AP1 call_in is done and before AP1 get in tsc_sync_wait
> The AP2 call_in done.  and then AP1 get in tsc_sync_wait and before it
> done, AP2 get in tsc_sync_wait too.
> 
> sync_master can not figure out from AP1 or AP2 because only have
> go[MASTER] and go{SLAVE].

Ok, you're right. It's better to move it to before callin map.

-Andi

Re: [discuss] Re: 2.6.13-rc2 with dual way dual core ck804 MB

[yhlu]

good, I will produce one patch next week.

YH

On 8/12/05, Andi Kleen <ak@suse.de> wrote:
> On Thu, Aug 11, 2005 at 11:59:21PM -0700, yhlu wrote:
> > andi,
> >
> > is it possible for
> > after the AP1 call_in is done and before AP1 get in tsc_sync_wait
> > The AP2 call_in done.  and then AP1 get in tsc_sync_wait and before it
> > done, AP2 get in tsc_sync_wait too.
> >
> > sync_master can not figure out from AP1 or AP2 because only have
> > go[MASTER] and go{SLAVE].
> 
> Ok, you're right. It's better to move it to before callin map.
> 
> -Andi
>

Re: [discuss] Re: 2.6.13-rc2 with dual way dual core ck804 MB

[Andi Kleen]

On Fri, Aug 12, 2005 at 09:18:07AM -0700, yhlu wrote:
> good, I will produce one patch next week.

I already did it in my tree.

-Andi

Re: [discuss] Re: 2.6.13-rc2 with dual way dual core ck804 MB

[yhlu]

Oh.

On 8/12/05, Andi Kleen <ak@suse.de> wrote:
> On Fri, Aug 12, 2005 at 09:18:07AM -0700, yhlu wrote:
> > good, I will produce one patch next week.
> 
> I already did it in my tree.
> 
> -Andi
>

Re: [discuss] Re: 2.6.13-rc2 with dual way dual core ck804 MB

[Mike Waychison]

Andi Kleen wrote:
> On Wed, Aug 10, 2005 at 04:14:19PM -0700, Mike Waychison wrote:
> 
>>YhLu wrote:
>>
>>>andi,
>>>
>>>please refer the patch, it will move cpu_set(, cpu_callin_map) from
>>>smi_callin to start_secondary.
>>
>>
>>This patch fixes an apparent race / lockup on our 2-way dual cores (when 
>>applied against 2.6.12.3).  The machine was locking up after 
>>"Initializing CPU#2".
> 
> 
> The real solution for this issue is the smp_call_function_single patch from Eric
> that I reposted yesterday. Yh's patch just changed the timing slightly.
> 
> 

Indeed.

I had a report here that the smp_call_function_single patch wasn't 
fixing our problem.  I just tried it myself and it appears to also fix 
the problem.

Thanks,

Mike Waychison

Re: 2.6.13-rc2 with dual way dual core ck804 MB

[Peter Buckingham]

Mike Waychison wrote:
> This patch fixes an apparent race / lockup on our 2-way dual cores (when 
> applied against 2.6.12.3).  The machine was locking up after 
> "Initializing CPU#2".

the better ways is to use the patch from Eric that Andi posted to stable 
yesterday:

	http://x86-64.org/lists/discuss/msg06943.html

peter

RE: 2.6.13-rc2 with dual way dual core ck804 MB

[YhLu]

[-- Attachment #1: Type: text/plain, Size: 6509 bytes --]

Andi,

Can you look at the patch? It could solve the timing problem.

I also move your ext_apic_id patch into 2.6.13 manaully. 

YH

> -----Original Message-----
> From: YhLu 
> Sent: Wednesday, July 06, 2005 5:51 PM
> To: Andi Kleen
> Cc: 'Peter Buckingham'; linux-kernel@vger.kernel.org; 
> 'discuss@x86-64.org'
> Subject: RE: 2.6.13-rc2 with dual way dual core ck804 MB
> 
> andi,
> 
> please refer the patch, it will move cpu_set(, 
> cpu_callin_map) from smi_callin to start_secondary.
> 
> --- 
> /home/yhlu/xx1/linux-2.6.13-rc2/arch/x86_64/kernel/smpboot.c.o
> rig   2005-07-06 18:41:16.789767168 -0700
> +++ 
> /home/yhlu/xx1/linux-2.6.13-rc2/arch/x86_64/kernel/smpboot.c  
>       2005-07-06 18:45:11.923021480 -0700
> @@ -442,7 +442,7 @@
>         /*
>          * Allow the master to continue.
>          */
> -       cpu_set(cpuid, cpu_callin_map);
> +//     cpu_set(cpuid, cpu_callin_map); // moved to 
> start_secondary by yhlu
>  }
> 
>  static inline void set_cpu_sibling_map(int cpu) @@ -529,8 +529,11 @@
>         /* Wait for TSC sync to not schedule things before.
>            We still process interrupts, which could see an 
> inconsistent
>            time in that window unfortunately. */
> +
>         tsc_sync_wait();
> 
> +       cpu_set(smp_processor_id(), cpu_callin_map); // moved from 
> + smp_callin by yhlu
> +
>         cpu_idle();
>  }
> 
> the other solution will be change cpu_callin_map to 
> cpu_online_map in do_boot_cpu
> 
>                 /*
>                  * allow APs to start initializing.
>                  */
>                 Dprintk("Before Callout %d.\n", cpu);
>                 cpu_set(cpu, cpu_callout_map);
>                 Dprintk("After Callout %d.\n", cpu);
> 
>                 /*
>                  * Wait 5s total for a response
>                  */
>                 for (timeout = 0; timeout < 50000; timeout++) {
>                         if (cpu_isset(cpu, cpu_callin_map)) 
> --------------------------> cpu_online_map
>                                 break;  /* It has booted */
>                         udelay(100);
>                 }
> 
>                 if (cpu_isset(cpu, cpu_callin_map)) { 
> --------------------------------> cpu_online_map
>                         /* number CPUs logically, starting 
> from 1 (BSP is 0) */
>                         Dprintk("CPU has booted.\n");
>                 } else {
>                         boot_error = 1;
>                         if (*((volatile unsigned char 
> *)phys_to_virt(SMP_TRAMPOLINE_BASE))
>                                         == 0xA5)
>                                 /* trampoline started but...? */
>                                 printk("Stuck ??\n");
>                         else
>                                 /* trampoline code not run */
>                                 printk("Not responding.\n"); 
> #if APIC_DEBUG
>                         inquire_remote_apic(apicid); #endif
>                 }
> 
> 
> the result will be
> 
> Booting processor 1/1 rip 6000 rsp ffff81013ff89f58 
> Initializing CPU#1 masked ExtINT on CPU#1 Calibrating delay 
> using timer specific routine.. 4422.98 BogoMIPS (lpj=8845965)
> CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
> CPU: L2 Cache: 1024K (64 bytes/line)
> CPU 1(2) -> Node 0 -> Core 1
>  stepping 00
> CPU 1: Syncing TSC to CPU 0.
> sync_master: 1 smp_processor_id() = 00, boot_cpu_id= 00
> sync_master: 2 smp_processor_id() = 00, boot_cpu_id= 00 CPU 
> 1: synchronized TSC with CPU 0 (last diff 0 cycles, maxerr 
> 595 cycles) ---------------------> it is in right place.
> Booting processor 2/2 rip 6000 rsp ffff81023ff1df58 
> Initializing CPU#2 masked ExtINT on CPU#2 Calibrating delay 
> using timer specific routine.. 4422.99 BogoMIPS (lpj=8845997)
> CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
> CPU: L2 Cache: 1024K (64 bytes/line)
> CPU 2(2) -> Node 1 -> Core 0
>  stepping 00
> CPU 2: Syncing TSC to CPU 0.
> sync_master: 1 smp_processor_id() = 00, boot_cpu_id= 00
> sync_master: 1 smp_processor_id() = 01, boot_cpu_id= 00
> sync_master: 2 smp_processor_id() = 00, boot_cpu_id= 00 CPU 
> 2: synchronized TSC with CPU 0 (last diff -4 cycles, maxerr 
> 1097 cycles) Booting processor 3/3 rip 6000 rsp 
> ffff81013ff53f58 Initializing CPU#3 masked ExtINT on CPU#3 
> Calibrating delay using timer specific routine.. 4423.03 
> BogoMIPS (lpj=8846075)
> CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
> CPU: L2 Cache: 1024K (64 bytes/line)
> CPU 3(2) -> Node 1 -> Core 1
>  stepping 00
> CPU 3: Syncing TSC to CPU 0.
> sync_master: 1 smp_processor_id() = 00, boot_cpu_id= 00
> sync_master: 1 smp_processor_id() = 01, boot_cpu_id= 00
> sync_master: 1 smp_processor_id() = 02, boot_cpu_id= 00
> sync_master: 2 smp_processor_id() = 00, boot_cpu_id= 00 CPU 
> 3: synchronized TSC with CPU 0 (last diff -4 cycles, maxerr 
> 1097 cycles) Brought up 4 CPUs
> 
> 
> > -----Original Message-----
> > From: YhLu
> > Sent: Wednesday, July 06, 2005 3:25 PM
> > To: Andi Kleen
> > Cc: Peter Buckingham; linux-kernel@vger.kernel.org
> > Subject: 2.6.13-rc2 with dual way dual core ck804 MB
> > 
> > andi,
> > 
> > the core1/node0 take a long while to get TSC synchronized. Is it 
> > normal?
> > i guess
> > "CPU 1: synchronized TSC with CPU 0"  should be just after "CPU 1: 
> > Syncing TSC to CPU0"
> > 
> > YH
> > 
> > 
> > cpu 1: setting up apic clock
> > cpu 1: enabling apic timer
> > CPU 1: Syncing TSC to CPU 0.
> > CPU has booted.
> > waiting for cpu 1
> > 
> > cpu 2: setting up apic clock
> > cpu 2: enabling apic timer
> > CPU 2: Syncing TSC to CPU 0.
> > CPU 2: synchronized TSC with CPU 0 (last diff -4 cycles, 
> maxerr 1097 
> > cycles) CPU has booted.
> > waiting for cpu 2
> > 
> > cpu 3: setting up apic clock
> > cpu 3: enabling apic timer
> > CPU 3: Syncing TSC to CPU 0.
> > CPU 3: synchronized TSC with CPU 0 (last diff 1 cycles, maxerr 1087 
> > cycles) CPU has booted.
> > waiting for cpu 3
> > 
> > testing NMI watchdog ... CPU#1: NMI appears to be stuck (1->1)!
> > checking if image is initramfs...<6>CPU 1: synchronized TSC 
> with CPU 0 
> > (last diff 0 cycles, maxerr 595 cycles) it isn't (no cpio magic); 
> > looks like an initrd
> > 
> > 
> > the
> > -
> > To unsubscribe from this list: send the line "unsubscribe 
> > linux-kernel" in the body of a message to majordomo@vger.kernel.org 
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
> > Please read the FAQ at  http://www.tux.org/lkml/
> > 


[-- Attachment #2: 2.6.13.rc2_ext_apic_id_dual_core_timing.patch --]
[-- Type: application/octet-stream, Size: 5435 bytes --]

diff -uNr /home/yhlu/xx1/linux-2.6.13-rc2.orig/arch/x86_64/kernel/genapic.c /home/yhlu/xx1/linux-2.6.13-rc2/arch/x86_64/kernel/genapic.c
--- /home/yhlu/xx1/linux-2.6.13-rc2.orig/arch/x86_64/kernel/genapic.c	2005-07-05 20:46:33.000000000 -0700
+++ /home/yhlu/xx1/linux-2.6.13-rc2/arch/x86_64/kernel/genapic.c	2005-07-06 15:37:52.932607728 -0700
@@ -31,25 +31,20 @@
 
 extern struct genapic apic_cluster;
 extern struct genapic apic_flat;
+extern struct genapic apic_physflat;
 
 struct genapic *genapic = &apic_flat;
 
-
 /*
  * Check the APIC IDs in bios_cpu_apicid and choose the APIC mode.
  */
-void __init clustered_apic_check(void)
+void __init extended_apic_check(void)
 {
 	long i;
 	u8 clusters, max_cluster;
 	u8 id;
 	u8 cluster_cnt[NUM_APIC_CLUSTERS];
-
-	if (boot_cpu_data.x86_vendor == X86_VENDOR_AMD) {
-		/* AMD always uses flat mode right now */
-		genapic = &apic_flat;
-		goto print;
-	}
+	int num_cpus = 0;
 
 #if defined(CONFIG_ACPI_BUS)
 	/*
@@ -65,10 +60,19 @@
 
 	memset(cluster_cnt, 0, sizeof(cluster_cnt));
 
+        /* Count how many CPUs the BIOS told us about, but not
+          more than what the user specified */
 	for (i = 0; i < NR_CPUS; i++) {
 		id = bios_cpu_apicid[i];
-		if (id != BAD_APICID)
-			cluster_cnt[APIC_CLUSTERID(id)]++;
+		if (id == BAD_APICID)
+			continue;
+		cluster_cnt[APIC_CLUSTERID(id)]++;
+		num_cpus++;
+	}
+
+	if (boot_cpu_data.x86_vendor == X86_VENDOR_AMD) {
+		genapic = num_cpus > 8 ? &apic_physflat : &apic_flat;
+		goto print;
 	}
 
 	clusters = 0;
diff -uNr /home/yhlu/xx1/linux-2.6.13-rc2.orig/arch/x86_64/kernel/genapic_flat.c /home/yhlu/xx1/linux-2.6.13-rc2/arch/x86_64/kernel/genapic_flat.c
--- /home/yhlu/xx1/linux-2.6.13-rc2.orig/arch/x86_64/kernel/genapic_flat.c	2005-07-05 20:46:33.000000000 -0700
+++ /home/yhlu/xx1/linux-2.6.13-rc2/arch/x86_64/kernel/genapic_flat.c	2005-07-06 15:41:04.414498056 -0700
@@ -2,7 +2,7 @@
  * Copyright 2004 James Cleverdon, IBM.
  * Subject to the GNU Public License, v.2
  *
- * Flat APIC subarch code.  Maximum 8 CPUs, logical delivery.
+ * Flat APIC subarch code.  Logical delivery.
  *
  * Hacked for x86-64 by James Cleverdon from i386 architecture code by
  * Martin Bligh, Andi Kleen, James Bottomley, John Stultz, and
@@ -195,6 +195,53 @@
 	.phys_pkg_id = phys_pkg_id,
 };
 
+
+/* 
+ * Physflat mode is used when there are more than 8 CPUs on a AMD system.
+ * We cannot use logical delivery in this case because the mask
+ * overflows, so use physical mode.
+ */
+
+static cpumask_t physflat_target_cpus(void)
+{
+       return cpumask_of_cpu(0);
+}
+
+static void physflat_send_IPI_mask(cpumask_t cpumask, int vector)
+{
+       send_IPI_mask_sequence(cpumask, vector);
+}
+
+static unsigned int physflat_cpu_mask_to_apicid(cpumask_t cpumask)
+{
+       int cpu;
+
+       /*
+        * We're using fixed IRQ delivery, can only return one phys APIC ID.
+        * May as well be the first.
+        */
+       cpu = first_cpu(cpumask);
+       if ((unsigned)cpu < NR_CPUS)
+               return x86_cpu_to_apicid[cpu];
+       else
+               return BAD_APICID;
+}
+
+struct genapic apic_physflat =  {
+       .name = "physical flat",
+       .int_delivery_mode = dest_LowestPrio,
+       .int_dest_mode = (APIC_DEST_PHYSICAL != 0),
+       .int_delivery_dest = APIC_DEST_PHYSICAL | APIC_DM_LOWEST,
+       .target_cpus = physflat_target_cpus,
+       .apic_id_registered = flat_apic_id_registered,
+       .init_apic_ldr = flat_init_apic_ldr,/*not needed, but shouldn't hurt*/
+       .send_IPI_all = flat_send_IPI_all,
+       .send_IPI_allbutself = flat_send_IPI_allbutself,
+       .send_IPI_mask = physflat_send_IPI_mask,
+       .cpu_mask_to_apicid = physflat_cpu_mask_to_apicid,
+       .phys_pkg_id = phys_pkg_id,
+};
+
 static int __init print_ipi_mode(void)
 {
 	printk ("Using IPI %s mode\n", no_broadcast ? "No-Shortcut" :
diff -uNr /home/yhlu/xx1/linux-2.6.13-rc2.orig/arch/x86_64/kernel/mpparse.c /home/yhlu/xx1/linux-2.6.13-rc2/arch/x86_64/kernel/mpparse.c
--- /home/yhlu/xx1/linux-2.6.13-rc2.orig/arch/x86_64/kernel/mpparse.c	2005-07-05 20:46:33.000000000 -0700
+++ /home/yhlu/xx1/linux-2.6.13-rc2/arch/x86_64/kernel/mpparse.c	2005-07-06 15:32:47.211084464 -0700
@@ -341,7 +341,7 @@
 			}
 		}
 	}
-	clustered_apic_check();
+	extended_apic_check();
 	if (!num_processors)
 		printk(KERN_ERR "SMP mptable: no processors registered!\n");
 	return num_processors;
diff -uNr /home/yhlu/xx1/linux-2.6.13-rc2.orig/arch/x86_64/kernel/smpboot.c /home/yhlu/xx1/linux-2.6.13-rc2/arch/x86_64/kernel/smpboot.c
--- /home/yhlu/xx1/linux-2.6.13-rc2.orig/arch/x86_64/kernel/smpboot.c	2005-07-05 20:46:33.000000000 -0700
+++ /home/yhlu/xx1/linux-2.6.13-rc2/arch/x86_64/kernel/smpboot.c	2005-07-06 18:45:11.923021480 -0700
@@ -442,7 +442,7 @@
 	/*
 	 * Allow the master to continue.
 	 */
-	cpu_set(cpuid, cpu_callin_map);
+//	cpu_set(cpuid, cpu_callin_map); // moved to start_secondary by yhlu
 }
 
 static inline void set_cpu_sibling_map(int cpu)
@@ -529,8 +529,11 @@
 	/* Wait for TSC sync to not schedule things before.
 	   We still process interrupts, which could see an inconsistent
 	   time in that window unfortunately. */
+
 	tsc_sync_wait();
 
+	cpu_set(smp_processor_id(), cpu_callin_map); // moved from smp_callin by yhlu
+
 	cpu_idle();
 }
 

2.6.13-rc2 with dual way dual core ck804 MB

[YhLu]

andi,

the core1/node0 take a long while to get TSC synchronized. Is it normal?
i guess 
"CPU 1: synchronized TSC with CPU 0"  should be just after "CPU 1: Syncing
TSC to CPU0"

YH


cpu 1: setting up apic clock
cpu 1: enabling apic timer
CPU 1: Syncing TSC to CPU 0.
CPU has booted.
waiting for cpu 1

cpu 2: setting up apic clock
cpu 2: enabling apic timer
CPU 2: Syncing TSC to CPU 0.
CPU 2: synchronized TSC with CPU 0 (last diff -4 cycles, maxerr 1097 cycles)
CPU has booted.
waiting for cpu 2

cpu 3: setting up apic clock
cpu 3: enabling apic timer
CPU 3: Syncing TSC to CPU 0.
CPU 3: synchronized TSC with CPU 0 (last diff 1 cycles, maxerr 1087 cycles)
CPU has booted.
waiting for cpu 3

testing NMI watchdog ... CPU#1: NMI appears to be stuck (1->1)!
checking if image is initramfs...<6>CPU 1: synchronized TSC with CPU 0 (last
diff 0 cycles, maxerr 595 cycles)
it isn't (no cpio magic); looks like an initrd


the