From mboxrd@z Thu Jan  1 00:00:00 1970
From: Dave Winchell <dwinchell@virtualiron.com>
Subject: [PATCH] add memory barriers to IPI related code
Date: Thu, 23 Oct 2008 14:23:13 -0400
Message-ID: <4900C111.1070709@virtualiron.com>
Mime-Version: 1.0
Content-Type: multipart/mixed; boundary="------------030507030602040406080507"
Return-path: <xen-devel-bounces@lists.xensource.com>
List-Unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>,
	<mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
List-Post: <mailto:xen-devel@lists.xensource.com>
List-Help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-Subscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>,
	<mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
Sender: xen-devel-bounces@lists.xensource.com
Errors-To: xen-devel-bounces@lists.xensource.com
To: xen-devel <xen-devel@lists.xensource.com>
Cc: "dan.magenheimer@oracle.com" <dan.magenheimer@oracle.com>, Dave Winchell <dwinchell@virtualiron.com>, "Langsdorf,
	Mark" <mark.langsdorf@amd.com>, Keir Fraser <keir.fraser@eu.citrix.com>
List-Id: xen-devel@lists.xenproject.org

This is a multi-part message in MIME format.
--------------030507030602040406080507
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Hi Keir,

We have been seeing some hangs in call function and
tlb shootdown. In our case the machine was an Intel Tylersburg.

One failure mode is this early return in smp_call_function_interrupt:

     if ( !cpu_isset(smp_processor_id(), call_data->selected) )
         return;

Without a mb(), the target processor takes this path as it doesn't yet 
see the update
to call_data->selected from the initiating processor. Then the initiator 
waits for ever
to get an acknowledgment from this processor.

On the initiating side, I added the mb()'s to the send_IPI functions, 
x2apic already
having one.

On the target side, I added the mb()'s in the individual target functions.

Regards,
Dave


--------------030507030602040406080507
Content-Type: text/plain;
 name="p.ipi.xen.unst.12.20"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
 filename="p.ipi.xen.unst.12.20"

diff -r 4129f0f2f2ba xen/arch/x86/smp.c
--- a/xen/arch/x86/smp.c	Fri Oct 17 14:15:37 2008 +0100
+++ b/xen/arch/x86/smp.c	Mon Oct 20 10:53:24 2008 -0400
@@ -93,6 +93,8 @@ void send_IPI_mask_flat(cpumask_t cpumas
     /* An IPI with no target generates a send accept error from P5/P6 APICs. */
     WARN_ON(mask == 0);
 
+    mb();
+
     local_irq_save(flags);
 
     /*
@@ -123,6 +125,8 @@ void send_IPI_mask_phys(cpumask_t mask, 
 {
     unsigned long cfg, flags;
     unsigned int query_cpu;
+
+    mb();
 
     local_irq_save(flags);
 
@@ -160,6 +164,7 @@ static unsigned int flush_flags;
 
 fastcall void smp_invalidate_interrupt(void)
 {
+    mb();
     ack_APIC_irq();
     perfc_incr(ipis);
     irq_enter();
@@ -337,17 +342,22 @@ void smp_send_stop(void)
 
 fastcall void smp_event_check_interrupt(struct cpu_user_regs *regs)
 {
+    mb();
     ack_APIC_irq();
     perfc_incr(ipis);
 }
 
 fastcall void smp_call_function_interrupt(struct cpu_user_regs *regs)
 {
-    void (*func)(void *info) = call_data->func;
-    void *info = call_data->info;
-
+    void (*func)(void *info);
+    void *info;
+
+    mb();
     ack_APIC_irq();
     perfc_incr(ipis);
+
+    func = call_data->func;
+    info = call_data->info;
 
     if ( !cpu_isset(smp_processor_id(), call_data->selected) )
         return;

--------------030507030602040406080507
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

--------------030507030602040406080507--