[PATCH] SMP dom0 boot fix

[PATCH] SMP dom0 boot fix

[Kamble, Nitin A]

[-- Attachment #1: Type: text/plain, Size: 1524 bytes --]

Keir,
    This patch reverts part of the 7402 patch. The reverted part is
conversion from unsigned long to uint32_t for evtchn_pending,
evtchn_mask.
  With this patch on top of the latest tip (changeset:
7502:e70ea9465b310e1cba9678ac4e9ad534bb8b670a) it can boot SMP dom0 on
x86_64.

Thanks & Regards,
Nitin
------------------------------------------------------------------------
-----------
Open Source Technology Center, Intel Corp

>-----Original Message-----
>From: Ryan Harper [mailto:ryanh@us.ibm.com]
>Sent: Thursday, October 27, 2005 4:50 PM
>To: Kamble, Nitin A
>Cc: Keir Fraser; Ian Pratt; xen-devel
>Subject: Re: [Xen-devel] RE: [PATCH] [VT]long event-channel pending and
>mask arrays
>
>* Kamble, Nitin A <nitin.a.kamble@intel.com> [2005-10-27 18:34]:
>> Keir,
>>  You are right. It can be changed to long but it is not necessary.
>>
>> The issue in SMP dom0 for x86_64 got introduced before your SMP code
>> restructuring (merge smpboot.c) rev 7415.
>>
>> The rev 7412 has my fix for SMP dom0/domU. I have tested that fix
here
>> which was working fine. That patch is pulled in the tree with many
other
>> patches on the same day. With your 7402 patch (long masks) it is
>> breaking SMP dom0 differently. If I checkout the 7412 and then revert
>
>I checked out 7401 and added in 7412 and can confirm that x86_64 DOM0
>SMP boots.
>
>--
>Ryan Harper
>Software Engineer; Linux Technology Center
>IBM Corp., Austin, Tx
>(512) 838-9253   T/L: 678-9253
>ryanh@us.ibm.com

[-- Attachment #2: long_to_uint32_masks.patch --]
[-- Type: application/octet-stream, Size: 8125 bytes --]

diff -r e70ea9465b310e1cba9678ac4e9ad534bb8b670a linux-2.6-xen-sparse/arch/ia64/xen/drivers/evtchn_ia64.c
--- a/linux-2.6-xen-sparse/arch/ia64/xen/drivers/evtchn_ia64.c	Thu Oct 27 09:53:02 2005
+++ b/linux-2.6-xen-sparse/arch/ia64/xen/drivers/evtchn_ia64.c	Thu Oct 27 17:43:03 2005
@@ -94,7 +94,7 @@
 
 irqreturn_t evtchn_interrupt(int irq, void *dev_id, struct pt_regs *regs)
 {
-    unsigned long  l1, l2;
+    u32  l1, l2;
     unsigned int   l1i, l2i, port;
     irqreturn_t (*handler)(int, void *, struct pt_regs *);
     shared_info_t *s = HYPERVISOR_shared_info;
@@ -108,14 +108,14 @@
     while ( l1 != 0 )
     {
         l1i = __ffs(l1);
-        l1 &= ~(1UL << l1i);
+        l1 &= ~(1 << l1i);
 
         while ( (l2 = s->evtchn_pending[l1i] & ~s->evtchn_mask[l1i]) != 0 )
         {
             l2i = __ffs(l2);
-            l2 &= ~(1UL << l2i);
+            l2 &= ~(1 << l2i);
 
-            port = (l1i * BITS_PER_LONG) + l2i;
+            port = (l1i << 5) + l2i;
             if ( (handler = evtchns[port].handler) != NULL )
 	    {
 		clear_evtchn(port);
diff -r e70ea9465b310e1cba9678ac4e9ad534bb8b670a linux-2.6-xen-sparse/arch/xen/kernel/evtchn.c
--- a/linux-2.6-xen-sparse/arch/xen/kernel/evtchn.c	Thu Oct 27 09:53:02 2005
+++ b/linux-2.6-xen-sparse/arch/xen/kernel/evtchn.c	Thu Oct 27 17:43:03 2005
@@ -72,7 +72,7 @@
 #ifdef CONFIG_SMP
 
 static u8 cpu_evtchn[NR_EVENT_CHANNELS];
-static unsigned long cpu_evtchn_mask[NR_CPUS][NR_EVENT_CHANNELS/BITS_PER_LONG];
+static u32 cpu_evtchn_mask[NR_CPUS][NR_EVENT_CHANNELS/32];
 
 #define active_evtchns(cpu,sh,idx)		\
 	((sh)->evtchn_pending[idx] &		\
@@ -137,7 +137,7 @@
 /* NB. Interrupts are disabled on entry. */
 asmlinkage void evtchn_do_upcall(struct pt_regs *regs)
 {
-	unsigned long  l1, l2;
+	u32  l1, l2;
 	unsigned int   l1i, l2i, port;
 	int            irq, cpu = smp_processor_id();
 	shared_info_t *s = HYPERVISOR_shared_info;
@@ -149,13 +149,13 @@
 	l1 = xchg(&vcpu_info->evtchn_pending_sel, 0);
 	while (l1 != 0) {
 		l1i = __ffs(l1);
-		l1 &= ~(1UL << l1i);
+		l1 &= ~(1 << l1i);
         
 		while ((l2 = active_evtchns(cpu, s, l1i)) != 0) {
 			l2i = __ffs(l2);
-			l2 &= ~(1UL << l2i);
+			l2 &= ~(1 << l2i);
             
-			port = (l1i * BITS_PER_LONG) + l2i;
+			port = (l1i << 5) + l2i;
 			if ((irq = evtchn_to_irq[port]) != -1)
 				do_IRQ(irq, regs);
 			else
diff -r e70ea9465b310e1cba9678ac4e9ad534bb8b670a linux-2.6-xen-sparse/include/asm-xen/evtchn.h
--- a/linux-2.6-xen-sparse/include/asm-xen/evtchn.h	Thu Oct 27 09:53:02 2005
+++ b/linux-2.6-xen-sparse/include/asm-xen/evtchn.h	Thu Oct 27 17:43:03 2005
@@ -100,8 +100,7 @@
 	 * masked.
 	 */
 	if (synch_test_bit(port, &s->evtchn_pending[0]) && 
-	    !synch_test_and_set_bit(port / BITS_PER_LONG,
-				    &vcpu_info->evtchn_pending_sel)) {
+	    !synch_test_and_set_bit(port>>5, &vcpu_info->evtchn_pending_sel)) {
 		vcpu_info->evtchn_upcall_pending = 1;
 		if (!vcpu_info->evtchn_upcall_mask)
 			force_evtchn_callback();
diff -r e70ea9465b310e1cba9678ac4e9ad534bb8b670a xen/arch/ia64/vmx/vmx_support.c
--- a/xen/arch/ia64/vmx/vmx_support.c	Thu Oct 27 09:53:02 2005
+++ b/xen/arch/ia64/vmx/vmx_support.c	Thu Oct 27 17:43:03 2005
@@ -49,7 +49,7 @@
 	  */
 	if (test_and_clear_bit(port,
 		&d->shared_info->evtchn_pending[0])) {
-	    clear_bit(port/BITS_PER_LONG, &v->vcpu_info->evtchn_pending_sel);
+	    clear_bit(port>>5, &v->vcpu_info->evtchn_pending_sel);
 	    clear_bit(0, &v->vcpu_info->evtchn_upcall_pending);
 	    vmx_io_assist(v);
 	}
@@ -67,7 +67,7 @@
 	     * nothing losed. Next loop will check I/O channel to fix this
 	     * window.
 	     */
-	    clear_bit(port/BITS_PER_LONG, &v->vcpu_info->evtchn_pending_sel);
+	    clear_bit(porti>>5, &v->vcpu_info->evtchn_pending_sel);
 	}
 	else
 	    break;
@@ -139,8 +139,8 @@
     /* Clear indicator specific to interrupt delivered from DM */
     if (test_and_clear_bit(port,
 		&d->shared_info->evtchn_pending[0])) {
-	if (!d->shared_info->evtchn_pending[port/BITS_PER_LONG])
-	    clear_bit(port/BITS_PER_LONG, &v->vcpu_info->evtchn_pending_sel);
+	if (!d->shared_info->evtchn_pending[port >> 5])
+	    clear_bit(port>>5, &v->vcpu_info->evtchn_pending_sel);
 
 	if (!v->vcpu_info->evtchn_pending_sel)
 	    clear_bit(0, &v->vcpu_info->evtchn_upcall_pending);
diff -r e70ea9465b310e1cba9678ac4e9ad534bb8b670a xen/arch/x86/vmx_io.c
--- a/xen/arch/x86/vmx_io.c	Thu Oct 27 09:53:02 2005
+++ b/xen/arch/x86/vmx_io.c	Thu Oct 27 17:43:03 2005
@@ -686,8 +686,8 @@
     int port = iopacket_port(d);
 
     /* evtchn_pending_sel bit is shared by other event channels. */
-    if (!d->shared_info->evtchn_pending[port/BITS_PER_LONG])
-        clear_bit(port/BITS_PER_LONG, &v->vcpu_info->evtchn_pending_sel);
+    if (!d->shared_info->evtchn_pending[port>>5])
+        clear_bit(port>>5, &v->vcpu_info->evtchn_pending_sel);
 
     /* Note: VMX domains may need upcalls as well. */
     if (!v->vcpu_info->evtchn_pending_sel)
@@ -727,7 +727,7 @@
             break;
         /* Events other than IOPACKET_PORT might have woken us up. In that
            case, safely go back to sleep. */
-        clear_bit(port/BITS_PER_LONG, &current->vcpu_info->evtchn_pending_sel);
+        clear_bit(port>>5, &current->vcpu_info->evtchn_pending_sel);
         clear_bit(0, &current->vcpu_info->evtchn_upcall_pending);
     } while(1);
 }
diff -r e70ea9465b310e1cba9678ac4e9ad534bb8b670a xen/common/keyhandler.c
--- a/xen/common/keyhandler.c	Thu Oct 27 09:53:02 2005
+++ b/xen/common/keyhandler.c	Thu Oct 27 17:43:03 2005
@@ -136,7 +136,7 @@
                             &d->shared_info->evtchn_pending[0]),
                    test_bit(v->virq_to_evtchn[VIRQ_DEBUG], 
                             &d->shared_info->evtchn_mask[0]),
-                   test_bit(v->virq_to_evtchn[VIRQ_DEBUG]/BITS_PER_LONG, 
+                   test_bit(v->virq_to_evtchn[VIRQ_DEBUG]>>5, 
                             &v->vcpu_info->evtchn_pending_sel));
             send_guest_virq(v, VIRQ_DEBUG);
         }
diff -r e70ea9465b310e1cba9678ac4e9ad534bb8b670a xen/include/public/xen.h
--- a/xen/include/public/xen.h	Thu Oct 27 09:53:02 2005
+++ b/xen/include/public/xen.h	Thu Oct 27 17:43:03 2005
@@ -262,9 +262,8 @@
 
 /*
  * Event channel endpoints per domain:
- *  1024 if a long is 32 bits; 4096 if a long is 64 bits.
- */
-#define NR_EVENT_CHANNELS (sizeof(unsigned long) * sizeof(unsigned long) * 64)
+ */
+#define NR_EVENT_CHANNELS 1024
 
 /*
  * Per-VCPU information goes here. This will be cleaned up more when Xen 
@@ -298,7 +297,7 @@
      */
     uint8_t evtchn_upcall_pending;
     uint8_t evtchn_upcall_mask;
-    unsigned long evtchn_pending_sel;
+    uint32_t evtchn_pending_sel;
 #ifdef __ARCH_HAS_VCPU_INFO
     arch_vcpu_info_t arch;
 #endif
@@ -364,11 +363,11 @@
      * 
      * To expedite scanning of pending notifications, any 0->1 pending
      * transition on an unmasked channel causes a corresponding bit in a
-     * per-vcpu selector word to be set. Each bit in the selector covers a
-     * 'C long' in the PENDING bitfield array.
+     * per-vcpu 32-bit selector to be set. Each bit in the selector covers a
+     * 32-bit in the PENDING bitfield array.
      */
-    unsigned long evtchn_pending[sizeof(unsigned long) * 8];
-    unsigned long evtchn_mask[sizeof(unsigned long) * 8];
+    uint32_t evtchn_pending[32];
+    uint32_t evtchn_mask[32];
 
     /*
      * Wallclock time: updated only by control software. Guests should base
diff -r e70ea9465b310e1cba9678ac4e9ad534bb8b670a xen/include/xen/event.h
--- a/xen/include/xen/event.h	Thu Oct 27 09:53:02 2005
+++ b/xen/include/xen/event.h	Thu Oct 27 17:43:03 2005
@@ -30,8 +30,7 @@
     /* These four operations must happen in strict order. */
     if ( !test_and_set_bit(port, &s->evtchn_pending[0]) &&
          !test_bit        (port, &s->evtchn_mask[0])    &&
-         !test_and_set_bit(port / BITS_PER_LONG,
-                           &v->vcpu_info->evtchn_pending_sel) &&
+         !test_and_set_bit(port>>5, &v->vcpu_info->evtchn_pending_sel) &&
          !test_and_set_bit(0, &v->vcpu_info->evtchn_upcall_pending) )
     {
         evtchn_notify(v);

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: [PATCH] SMP dom0 boot fix

[Keir Fraser]

On 28 Oct 2005, at 02:05, Kamble, Nitin A wrote:

>     This patch reverts part of the 7402 patch. The reverted part is
> conversion from unsigned long to uint32_t for evtchn_pending,
> evtchn_mask.

That was the original *point* of the patch, so yours is basically the 
anti-patch.

I can take a look... what is the bad SMP dom0 behaviour you are seeing 
(is there a bugzilla reference)?

  -- Keir

Re: Re: [PATCH] SMP dom0 boot fix

[Ryan Harper]

* Keir Fraser <Keir.Fraser@cl.cam.ac.uk> [2005-10-28 09:20]:
> 
> On 28 Oct 2005, at 02:05, Kamble, Nitin A wrote:
> 
> >    This patch reverts part of the 7402 patch. The reverted part is
> >conversion from unsigned long to uint32_t for evtchn_pending,
> >evtchn_mask.
> 
> That was the original *point* of the patch, so yours is basically the 
> anti-patch.
> 
> I can take a look... what is the bad SMP dom0 behaviour you are seeing 
> (is there a bugzilla reference)?

I'll file a bug if Nitin doesn't beat me to it.  I was seeing
smp_call_function() blocking while it waited for a function to return
from being invoked on the second processor in early boot.  

It would block forever installing the deadline io scheduler, digging
deeper, it was after the second processor was up, and the io schedulers
call kmem_cache_create() in mm/slab.c, which would eventually result in
a call to smp_call_function() and passed in wait=1 which forces the
function to block until it has run on all online processors.

WARK: CPU0 in enable_cpucache()
WARK: CPU0 do_tune_cpucache() in
WARK: CPU0 calling all cpus with do_ccupdate_local()
WARK: CPU0 smp_call_function_all_cpus()
WARK: CPU0 running do_ccupdate_local()
WARK: CPU0 invoking smp_call_function()

At this point send_IPI_allbutself() has been invoked and the system
just sits and waits on CPU1 to run the function.  But, CPU1's
evtchn_upcall_mask was set (1), so I'm guessing the pending interrupt
is never acknowledged.  

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com

Re: Re: [PATCH] SMP dom0 boot fix

[Keir Fraser]

On 28 Oct 2005, at 15:59, Ryan Harper wrote:

> At this point send_IPI_allbutself() has been invoked and the system
> just sits and waits on CPU1 to run the function.  But, CPU1's
> evtchn_upcall_mask was set (1), so I'm guessing the pending interrupt
> is never acknowledged.

Okay, the good news is that's the same bug I was able to repro last 
week. Turns out that CPU1's upcall mask is getting weirdly set under 
its feet. Since it's waiting on the big kernel lock, which is held by 
CPU0, which is waiting for acknowledgement of an interrupt in CPU1, we 
have a deadlock.

Given the problem is in that one changeset, this can't be hard to track 
down now.

  -- Keir

Re: Re: [PATCH] SMP dom0 boot fix

[Keir Fraser]

On 28 Oct 2005, at 16:15, Keir Fraser wrote:

>
> On 28 Oct 2005, at 15:59, Ryan Harper wrote:
>
>> At this point send_IPI_allbutself() has been invoked and the system
>> just sits and waits on CPU1 to run the function.  But, CPU1's
>> evtchn_upcall_mask was set (1), so I'm guessing the pending interrupt
>> is never acknowledged.
>
> Okay, the good news is that's the same bug I was able to repro last 
> week. Turns out that CPU1's upcall mask is getting weirdly set under 
> its feet. Since it's waiting on the big kernel lock, which is held by 
> CPU0, which is waiting for acknowledgement of an interrupt in CPU1, we 
> have a deadlock.
>
> Given the problem is in that one changeset, this can't be hard to 
> track down now.

Now fixed in our staging tree. sizeof_vcpu_shift in 
arch/xen/x86_64/xen_entry.S should be 4, not 3.

  -- Keir

Re: Re: [PATCH] SMP dom0 boot fix

[Ryan Harper]

* Keir Fraser <Keir.Fraser@cl.cam.ac.uk> [2005-10-28 10:41]:
> 
> On 28 Oct 2005, at 16:15, Keir Fraser wrote:
> 
> >
> >On 28 Oct 2005, at 15:59, Ryan Harper wrote:
> >
> >>At this point send_IPI_allbutself() has been invoked and the system
> >>just sits and waits on CPU1 to run the function.  But, CPU1's
> >>evtchn_upcall_mask was set (1), so I'm guessing the pending interrupt
> >>is never acknowledged.
> >
> >Okay, the good news is that's the same bug I was able to repro last 
> >week. Turns out that CPU1's upcall mask is getting weirdly set under 
> >its feet. Since it's waiting on the big kernel lock, which is held by 
> >CPU0, which is waiting for acknowledgement of an interrupt in CPU1, we 
> >have a deadlock.
> >
> >Given the problem is in that one changeset, this can't be hard to 
> >track down now.
> 
> Now fixed in our staging tree. sizeof_vcpu_shift in 
> arch/xen/x86_64/xen_entry.S should be 4, not 3.

That fixed it for me here.  Thanks!

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com

Re: Re: [PATCH] SMP dom0 boot fix

[Ky Srinivasan]

Thanks Kier. With this fix applied, I am able to boot SMP dom0. The box
I am testing on is x86_64 machine with two hardware threads. However, if
I turn on SMT in the Linux configuration, the kernel takes a fault in
early startup (a NULL pointer reference at find_busiest_group +144). It
appears that the sched domain hierarchy is not correctly set up  here.
Looking at the new smpboot.c, is turning on SMT support no longer
valid?

K. Y
 
>>> Keir Fraser <Keir.Fraser@cl.cam.ac.uk> 10/28/05 11:45 am >>> 

On 28 Oct 2005, at 16:15, Keir Fraser wrote:

>
> On 28 Oct 2005, at 15:59, Ryan Harper wrote:
>
>> At this point send_IPI_allbutself() has been invoked and the system
>> just sits and waits on CPU1 to run the function.  But, CPU1's
>> evtchn_upcall_mask was set (1), so I'm guessing the pending
interrupt
>> is never acknowledged.
>
> Okay, the good news is that's the same bug I was able to repro last 
> week. Turns out that CPU1's upcall mask is getting weirdly set under

> its feet. Since it's waiting on the big kernel lock, which is held by

> CPU0, which is waiting for acknowledgement of an interrupt in CPU1,
we 
> have a deadlock.
>
> Given the problem is in that one changeset, this can't be hard to 
> track down now.

Now fixed in our staging tree. sizeof_vcpu_shift in 
arch/xen/x86_64/xen_entry.S should be 4, not 3.

  --  Keir


_______________________________________________
Xen- devel mailing list
Xen- devel@lists.xensource.com
http://lists.xensource.com/xen- devel

RE: Re: [PATCH] SMP dom0 boot fix

[Puthiyaparambil, Aravindh]

Keir,

I have not tried a SMP Dom0 but I have been unable to bring up any
x86_64 SMP DomUs for a while now. I don't know the symptoms of the SMP
Dom0 issue so I can't say for sure that they are connected. Here is the
bug report.

http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=347

FYI, x86_32 PAE SMP DomUs are working.

Aravindh

> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
> bounces@lists.xensource.com] On Behalf Of Keir Fraser
> Sent: Friday, October 28, 2005 3:32 AM
> To: Kamble, Nitin A
> Cc: Ian Pratt; xen-devel; Ryan Harper
> Subject: [Xen-devel] Re: [PATCH] SMP dom0 boot fix
> 
> 
> On 28 Oct 2005, at 02:05, Kamble, Nitin A wrote:
> 
> >     This patch reverts part of the 7402 patch. The reverted part is
> > conversion from unsigned long to uint32_t for evtchn_pending,
> > evtchn_mask.
> 
> That was the original *point* of the patch, so yours is basically the
> anti-patch.
> 
> I can take a look... what is the bad SMP dom0 behaviour you are seeing
> (is there a bugzilla reference)?
> 
>   -- Keir
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

RE: Re: [PATCH] SMP dom0 boot fix

[Kamble, Nitin A]

>> Now fixed in our staging tree. sizeof_vcpu_shift in
>> arch/xen/x86_64/xen_entry.S should be 4, not 3.
>
>That fixed it for me here.  Thanks!

That's Great!!! Keir, Thanks for fixing.

Thanks & Regards,
Nitin
------------------------------------------------------------------------
-----------
Open Source Technology Center, Intel Corp

RE: Re: [PATCH] SMP dom0 boot fix

[Ian Pratt]

> Thanks Kier. With this fix applied, I am able to boot SMP 
> dom0. The box I am testing on is x86_64 machine with two 
> hardware threads. However, if I turn on SMT in the Linux 
> configuration, the kernel takes a fault in early startup (a 
> NULL pointer reference at find_busiest_group +144). It 
> appears that the sched domain hierarchy is not correctly set up  here.
> Looking at the new smpboot.c, is turning on SMT support no 
> longer valid?

I don't think we want particularly want to expose CPU sibling
relationships to guests, at least not ones that don't have all their
vcpu's pinned. I think a better approach would probably just to have the
guest scheduler favour low numbered VCPUs and leave Xen to do the maping
to logical CPUs. This would get us most of the benefit of SMT
scheduling, the only trick we'd be missing is the reduced migration
penalty for moving a process between logical CPUs sharing an L2 cache.
If we felt this was a big deal, we'd need some way of reflecting the
current relationship via shared info or such like.  Eitherway, no good
can come of enabling SMT in the guest right now, though we should fix
the crash.

Ian 

RE: Re: [PATCH] SMP dom0 boot fix

[Ky Srinivasan]

Thanks Ian. I agree with your analysis.

K. Y
 
>>> "Ian Pratt" <m+Ian.Pratt@cl.cam.ac.uk> 10/28/05 7:52 pm >>> 
> Thanks Kier. With this fix applied, I am able to boot SMP 
> dom0. The box I am testing on is x86_64 machine with two 
> hardware threads. However, if I turn on SMT in the Linux 
> configuration, the kernel takes a fault in early startup (a 
> NULL pointer reference at find_busiest_group +144). It 
> appears that the sched domain hierarchy is not correctly set up 
here.
> Looking at the new smpboot.c, is turning on SMT support no 
> longer valid?

I don't think we want particularly want to expose CPU sibling
relationships to guests, at least not ones that don't have all their
vcpu's pinned. I think a better approach would probably just to have
the
guest scheduler favour low numbered VCPUs and leave Xen to do the
maping
to logical CPUs. This would get us most of the benefit of SMT
scheduling, the only trick we'd be missing is the reduced migration
penalty for moving a process between logical CPUs sharing an L2 cache.
If we felt this was a big deal, we'd need some way of reflecting the
current relationship via shared info or such like.  Eitherway, no good
can come of enabling SMT in the guest right now, though we should fix
the crash.

Ian 

RE: Re: [PATCH] SMP dom0 boot fix

[Ky Srinivasan]

Thanks Ian. I agree with your analysis.

K. Y
 
>>> "Ian Pratt" <m+Ian.Pratt@cl.cam.ac.uk> 10/28/05 7:52 pm >>> 
> Thanks Kier. With this fix applied, I am able to boot SMP 
> dom0. The box I am testing on is x86_64 machine with two 
> hardware threads. However, if I turn on SMT in the Linux 
> configuration, the kernel takes a fault in early startup (a 
> NULL pointer reference at find_busiest_group +144). It 
> appears that the sched domain hierarchy is not correctly set up 
here.
> Looking at the new smpboot.c, is turning on SMT support no 
> longer valid?

I don't think we want particularly want to expose CPU sibling
relationships to guests, at least not ones that don't have all their
vcpu's pinned. I think a better approach would probably just to have
the
guest scheduler favour low numbered VCPUs and leave Xen to do the
maping
to logical CPUs. This would get us most of the benefit of SMT
scheduling, the only trick we'd be missing is the reduced migration
penalty for moving a process between logical CPUs sharing an L2 cache.
If we felt this was a big deal, we'd need some way of reflecting the
current relationship via shared info or such like.  Eitherway, no good
can come of enabling SMT in the guest right now, though we should fix
the crash.

Ian