[PATCH 0/6] xen,xend,tools: Add NUMA support to Xen

All of lore.kernel.org
 help / color / mirror / Atom feed

* [PATCH 0/6] xen,xend,tools: Add NUMA support to Xen
@ 2006-07-31 19:09 Ryan Harper
  2006-08-01  7:46 ` Tristan Gingold
                   ` (2 more replies)
  0 siblings, 3 replies; 11+ messages in thread
From: Ryan Harper @ 2006-07-31 19:09 UTC (permalink / raw)
  To: xen-devel

I've respun the NUMA patches against 10874 and I'm re-submitting them
with the optimizations mentioned [1]previously on the list.  There was a
request to see the overhead on non-numa/single-node machines.  I've
re-run those benchmarks (ballooning up from small mem to multi-gig) as
well as timing the initially domain increase_reservation time to gauge
the overhead when allocating from the heap.

The results for these were done on an x460, single node, with 6G ram.
The balloon test was from 512M to ~6G.  The NUMA patches include all of
the previous optimizations.

Balloon Up:
Without NUMA    With NUMA    NUMA Delta
Try1: 2943ms    2937ms       -6ms
Try2: 2996ms    2992ms       -4ms
Try3: 2971ms    2968ms       -3ms

Increase reservation
Without NUMA:
MemSize  128M  512M 1G   2G    3G    4G
------------------------------------------
Try1:    10ms  38ms 76ms 160ms 237ms 308ms
Try2:     9ms  37ms 76ms 151ms 232ms 308ms
Try3:     9ms  37ms 76ms 151ms 226ms 311ms

With NUMA:
MemSize  128M  512M 1G   2G    3G    4G
------------------------------------------
Try1:    10ms  42ms 84ms 170ms 260ms 346ms
Try2:    10ms  42ms 83ms 167ms 255ms 346ms
Try3:    10ms  42ms 83ms 168ms 251ms 346ms

The only other change has been to the infrastructure patch in which I
added the call to Xen's command line parsing to pick up the numa=
options.  This allows even on NUMA systems to disable NUMA parsing and
fake the number of nodes (numa=off fake=1, would fake 1 node).

1. http://lists.xensource.com/archives/html/xen-devel/2006-07/msg00384.html

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [PATCH 0/6] xen,xend,tools: Add NUMA support to Xen
  2006-07-31 19:09 [PATCH 0/6] xen,xend,tools: Add NUMA support to Xen Ryan Harper
@ 2006-08-01  7:46 ` Tristan Gingold
  2006-08-01 15:40   ` Ryan Harper
  2006-08-01  9:29 ` [PATCH 0/6] xen,xend,tools: Add NUMA support to Xen Tristan Gingold
  2006-08-02  6:14 ` Tristan Gingold
  2 siblings, 1 reply; 11+ messages in thread
From: Tristan Gingold @ 2006-08-01  7:46 UTC (permalink / raw)
  To: Ryan Harper, xen-devel

Le Lundi 31 Juillet 2006 21:09, Ryan Harper a écrit :
> I've respun the NUMA patches against 10874 and I'm re-submitting them
> with the optimizations mentioned [1]previously on the list.  There was a
> request to see the overhead on non-numa/single-node machines.  I've
> re-run those benchmarks (ballooning up from small mem to multi-gig) as
> well as timing the initially domain increase_reservation time to gauge
> the overhead when allocating from the heap.
Hi,

I am trying to use your patch on ia64.

In asm-x86/topology.h, you wrote:

extern unsigned int cpu_to_node[];
extern cpumask_t     node_to_cpumask[];

#define cpu_to_node(cpu)		(cpu_to_node[cpu])
#define parent_node(node)		(node)
#define node_to_first_cpu(node)  (__ffs(node_to_cpumask[node]))
#define node_to_cpumask(node)    (node_to_cpumask[node])

I think cpu_to_node and node_to_cpumask must be either a variable or a macro, 
but not both! (ia64 defines cpu_to_node as a macro).

Tristan.

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [PATCH 0/6] xen,xend,tools: Add NUMA support to Xen
  2006-08-01  7:46 ` Tristan Gingold
@ 2006-08-01 15:40   ` Ryan Harper
  2006-08-02  5:59     ` Tristan Gingold
  0 siblings, 1 reply; 11+ messages in thread
From: Ryan Harper @ 2006-08-01 15:40 UTC (permalink / raw)
  To: Tristan Gingold; +Cc: xen-devel

* Tristan Gingold <Tristan.Gingold@bull.net> [2006-08-01 02:43]:
> Le Lundi 31 Juillet 2006 21:09, Ryan Harper a écrit :
> > I've respun the NUMA patches against 10874 and I'm re-submitting them
> > with the optimizations mentioned [1]previously on the list.  There was a
> > request to see the overhead on non-numa/single-node machines.  I've
> > re-run those benchmarks (ballooning up from small mem to multi-gig) as
> > well as timing the initially domain increase_reservation time to gauge
> > the overhead when allocating from the heap.
> Hi,
> 
> I am trying to use your patch on ia64.

Thanks for testing these out on ia64.

> 
> In asm-x86/topology.h, you wrote:
> 
> extern unsigned int cpu_to_node[];
> extern cpumask_t     node_to_cpumask[];
> 
> #define cpu_to_node(cpu)		(cpu_to_node[cpu])
> #define parent_node(node)		(node)
> #define node_to_first_cpu(node)  (__ffs(node_to_cpumask[node]))
> #define node_to_cpumask(node)    (node_to_cpumask[node])
> 
> I think cpu_to_node and node_to_cpumask must be either a variable or a macro, 
> but not both! (ia64 defines cpu_to_node as a macro).

I'm not sure about this, but the definition of both the variable and
macro come from Linux, for example in

linux/include/asm-x86_64/topology.h

extern unsigned char cpu_to_node[];
extern cpumask_t     node_to_cpumask[];

#ifdef CONFIG_ACPI_NUMA
extern int __node_distance(int, int);
#define node_distance(a,b) __node_distance(a,b)
/* #else fallback version */
#endif

#define cpu_to_node(cpu)      (cpu_to_node[cpu])
#define parent_node(node)     (node)
#define node_to_first_cpu(node)  (first_cpu(node_to_cpumask[node]))
#define node_to_cpumask(node)    (node_to_cpumask[node])

AFAIK, this isn't an issue.

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [PATCH 0/6] xen,xend,tools: Add NUMA support to Xen
  2006-08-01 15:40   ` Ryan Harper
@ 2006-08-02  5:59     ` Tristan Gingold
  2006-08-02 22:29       ` Ryan Harper
  0 siblings, 1 reply; 11+ messages in thread
From: Tristan Gingold @ 2006-08-02  5:59 UTC (permalink / raw)
  To: xen-devel

Le Mardi 01 Août 2006 17:40, Ryan Harper a écrit :
> * Tristan Gingold <Tristan.Gingold@bull.net> [2006-08-01 02:43]:
> > Le Lundi 31 Juillet 2006 21:09, Ryan Harper a écrit :
> > > I've respun the NUMA patches against 10874 and I'm re-submitting them
> > > with the optimizations mentioned [1]previously on the list.  There was
> > > a request to see the overhead on non-numa/single-node machines.  I've
> > > re-run those benchmarks (ballooning up from small mem to multi-gig) as
> > > well as timing the initially domain increase_reservation time to gauge
> > > the overhead when allocating from the heap.
> >
> > Hi,
> >
> > I am trying to use your patch on ia64.
>
> Thanks for testing these out on ia64.
>
> > In asm-x86/topology.h, you wrote:
> >
> > extern unsigned int cpu_to_node[];
> > extern cpumask_t     node_to_cpumask[];
> >
> > #define cpu_to_node(cpu)		(cpu_to_node[cpu])
> > #define parent_node(node)		(node)
> > #define node_to_first_cpu(node)  (__ffs(node_to_cpumask[node]))
> > #define node_to_cpumask(node)    (node_to_cpumask[node])
> >
> > I think cpu_to_node and node_to_cpumask must be either a variable or a
> > macro, but not both! (ia64 defines cpu_to_node as a macro).
>
> I'm not sure about this, but the definition of both the variable and
> macro come from Linux, for example in
[...]
> AFAIK, this isn't an issue.
Except you are using both versions mainly the macro but at least the variable 
once in page_alloc.c:

/* Allocate 2^@order contiguous pages. */
struct page_info *alloc_heap_pages(unsigned int zone, unsigned int cpu,
                                   unsigned int order)
{
    unsigned int i,j, node = cpu_to_node[cpu], num_nodes = num_online_nodes();
    unsigned int request = (1UL << order);

This was a problem for ia64.
Furthermore you define the variable in xen/numa.h:
extern unsigned int cpu_to_node[];
#include <xen/cpumask.h>
extern cpumask_t node_to_cpumask[];

Which one is the API ?

Tristan.

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [PATCH 0/6] xen,xend,tools: Add NUMA support to Xen
  2006-08-02  5:59     ` Tristan Gingold
@ 2006-08-02 22:29       ` Ryan Harper
  2006-08-03  6:05         ` Tristan Gingold
  2006-08-05  0:33         ` [PATCH 0/6] xen, xend, tools: Add NUMA support to Xen Issues on the ES7000 Subrahmanian, Raj
  0 siblings, 2 replies; 11+ messages in thread
From: Ryan Harper @ 2006-08-02 22:29 UTC (permalink / raw)
  To: Tristan Gingold; +Cc: xen-devel

* Tristan Gingold <Tristan.Gingold@bull.net> [2006-08-02 00:56]:
> Le Mardi 01 Août 2006 17:40, Ryan Harper a écrit :
> > * Tristan Gingold <Tristan.Gingold@bull.net> [2006-08-01 02:43]:
> > > Le Lundi 31 Juillet 2006 21:09, Ryan Harper a écrit :
> > > > I've respun the NUMA patches against 10874 and I'm re-submitting them
> > > > with the optimizations mentioned [1]previously on the list.  There was
> > > > a request to see the overhead on non-numa/single-node machines.  I've
> > > > re-run those benchmarks (ballooning up from small mem to multi-gig) as
> > > > well as timing the initially domain increase_reservation time to gauge
> > > > the overhead when allocating from the heap.
> > >
> > > Hi,
> > >
> > > I am trying to use your patch on ia64.
> >
> > Thanks for testing these out on ia64.
> >
> > > In asm-x86/topology.h, you wrote:
> > >
> > > extern unsigned int cpu_to_node[];
> > > extern cpumask_t     node_to_cpumask[];
> > >
> > > #define cpu_to_node(cpu)		(cpu_to_node[cpu])
> > > #define parent_node(node)		(node)
> > > #define node_to_first_cpu(node)  (__ffs(node_to_cpumask[node]))
> > > #define node_to_cpumask(node)    (node_to_cpumask[node])
> > >
> > > I think cpu_to_node and node_to_cpumask must be either a variable or a
> > > macro, but not both! (ia64 defines cpu_to_node as a macro).
> >
> > I'm not sure about this, but the definition of both the variable and
> > macro come from Linux, for example in
> [...]
> > AFAIK, this isn't an issue.
> Except you are using both versions mainly the macro but at least the variable 
> once in page_alloc.c:
> 
> /* Allocate 2^@order contiguous pages. */
> struct page_info *alloc_heap_pages(unsigned int zone, unsigned int cpu,
>                                    unsigned int order)
> {
>     unsigned int i,j, node = cpu_to_node[cpu], num_nodes = num_online_nodes();
>     unsigned int request = (1UL << order);
> 
> This was a problem for ia64.
> Furthermore you define the variable in xen/numa.h:
> extern unsigned int cpu_to_node[];
> #include <xen/cpumask.h>
> extern cpumask_t node_to_cpumask[];
> 
> Which one is the API ?

If ia64 is already using macros, then we should use the macros.  I
should be able to remove those externs from numa.h and include the
asm/topology.h (not in xen/numa.h since that is the same thing)
in the .c files , like page_alloc.c, and use the macros instead.

Something like this (compiled on x86_64):

diff -r 083e69a85080 xen/arch/x86/dom0_ops.c
--- a/xen/arch/x86/dom0_ops.c	Mon Jul 31 10:53:59 2006 -0500
+++ b/xen/arch/x86/dom0_ops.c	Wed Aug 02 10:36:28 2006 -0500
@@ -26,6 +26,7 @@
 #include <asm/processor.h>
 #include <public/sched_ctl.h>
 #include <asm/numa.h>
+#include <asm/topology.h>
 
 #include <asm/mtrr.h>
 #include "cpu/mtrr/mtrr.h"
@@ -220,7 +221,7 @@ long arch_do_dom0_op(struct dom0_op *op,
         memset(node_to_cpu_64, 0, sizeof(node_to_cpu_64));
         for ( i=0; i<pi->nr_nodes; i++) {
             for ( j=0; j<num_online_cpus(); j++)
-                if ( cpu_isset(j, node_to_cpumask[i]) )
+                if ( cpu_isset(j, node_to_cpumask(i)) )
                     node_to_cpu_64[i] |= (u64)1 << j;
 
             if ( copy_to_guest_offset(op->u.physinfo.node_to_cpu, 
diff -r 083e69a85080 xen/common/page_alloc.c
--- a/xen/common/page_alloc.c	Mon Jul 31 10:53:59 2006 -0500
+++ b/xen/common/page_alloc.c	Wed Aug 02 10:32:27 2006 -0500
@@ -35,6 +35,7 @@
 #include <xen/domain_page.h>
 #include <xen/keyhandler.h>
 #include <asm/numa.h>
+#include <asm/topology.h>
 #include <asm/page.h>
 
 /*
@@ -317,7 +318,7 @@ struct page_info *alloc_heap_pages(unsig
 struct page_info *alloc_heap_pages(unsigned int zone, unsigned int cpu,
                                    unsigned int order)
 {
-    unsigned int i,j, node = cpu_to_node[cpu], num_nodes = num_online_nodes();
+    unsigned int i,j, node = cpu_to_node(cpu), num_nodes = num_online_nodes();
     unsigned int request = (1UL << order);
     struct page_info *pg;
 
diff -r 083e69a85080 xen/include/xen/numa.h
--- a/xen/include/xen/numa.h	Mon Jul 31 10:53:59 2006 -0500
+++ b/xen/include/xen/numa.h	Wed Aug 02 17:24:39 2006 -0500
@@ -23,8 +23,4 @@
 /* needed for drivers/acpi/numa.c */
 #define NR_NODE_MEMBLKS (MAX_NUMNODES*2)
 
-extern unsigned int cpu_to_node[];
-#include <xen/cpumask.h>
-extern cpumask_t node_to_cpumask[];
-
 #endif /* _XEN_NUMA_H */


-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [PATCH 0/6] xen,xend,tools: Add NUMA support to Xen
  2006-08-02 22:29       ` Ryan Harper
@ 2006-08-03  6:05         ` Tristan Gingold
  2006-08-05  0:33         ` [PATCH 0/6] xen, xend, tools: Add NUMA support to Xen Issues on the ES7000 Subrahmanian, Raj
  1 sibling, 0 replies; 11+ messages in thread
From: Tristan Gingold @ 2006-08-03  6:05 UTC (permalink / raw)
  To: Ryan Harper; +Cc: xen-devel

Le Jeudi 03 Août 2006 00:29, Ryan Harper a écrit :
> * Tristan Gingold <Tristan.Gingold@bull.net> [2006-08-02 00:56]:
> > Le Mardi 01 Août 2006 17:40, Ryan Harper a écrit :
> > > * Tristan Gingold <Tristan.Gingold@bull.net> [2006-08-01 02:43]:
> > > > Le Lundi 31 Juillet 2006 21:09, Ryan Harper a écrit :
> > > > > I've respun the NUMA patches against 10874 and I'm re-submitting
> > > > > them with the optimizations mentioned [1]previously on the list. 
> > > > > There was a request to see the overhead on non-numa/single-node
> > > > > machines.  I've re-run those benchmarks (ballooning up from small
> > > > > mem to multi-gig) as well as timing the initially domain
> > > > > increase_reservation time to gauge the overhead when allocating
> > > > > from the heap.
> > > >
> > > > Hi,
> > > >
> > > > I am trying to use your patch on ia64.
> > >
> > > Thanks for testing these out on ia64.
> > >
> > > > In asm-x86/topology.h, you wrote:
> > > >
> > > > extern unsigned int cpu_to_node[];
> > > > extern cpumask_t     node_to_cpumask[];
> > > >
> > > > #define cpu_to_node(cpu)		(cpu_to_node[cpu])
> > > > #define parent_node(node)		(node)
> > > > #define node_to_first_cpu(node)  (__ffs(node_to_cpumask[node]))
> > > > #define node_to_cpumask(node)    (node_to_cpumask[node])
> > > >
> > > > I think cpu_to_node and node_to_cpumask must be either a variable or
> > > > a macro, but not both! (ia64 defines cpu_to_node as a macro).
> > >
> > > I'm not sure about this, but the definition of both the variable and
> > > macro come from Linux, for example in
> >
> > [...]
> >
> > > AFAIK, this isn't an issue.
> >
> > Except you are using both versions mainly the macro but at least the
> > variable once in page_alloc.c:
> >
> > /* Allocate 2^@order contiguous pages. */
> > struct page_info *alloc_heap_pages(unsigned int zone, unsigned int cpu,
> >                                    unsigned int order)
> > {
> >     unsigned int i,j, node = cpu_to_node[cpu], num_nodes =
> > num_online_nodes(); unsigned int request = (1UL << order);
> >
> > This was a problem for ia64.
> > Furthermore you define the variable in xen/numa.h:
> > extern unsigned int cpu_to_node[];
> > #include <xen/cpumask.h>
> > extern cpumask_t node_to_cpumask[];
> >
> > Which one is the API ?
>
> If ia64 is already using macros, then we should use the macros.  I
> should be able to remove those externs from numa.h and include the
> asm/topology.h (not in xen/numa.h since that is the same thing)
> in the .c files , like page_alloc.c, and use the macros instead.
>
> Something like this (compiled on x86_64):
[...]
Yes, this would be fine for me.

Thank you for your NUMA work,
Tristan.

^ permalink raw reply	[flat|nested] 11+ messages in thread

* RE: [PATCH 0/6] xen, xend, tools: Add NUMA support to Xen Issues on the ES7000
  2006-08-02 22:29       ` Ryan Harper
  2006-08-03  6:05         ` Tristan Gingold
@ 2006-08-05  0:33         ` Subrahmanian, Raj
  2006-08-07 13:23           ` Ryan Harper
  1 sibling, 1 reply; 11+ messages in thread
From: Subrahmanian, Raj @ 2006-08-05  0:33 UTC (permalink / raw)
  To: Ryan Harper; +Cc: xen-devel

Ryan,
I tried out these patches on an ES7000. 
It's an x86-64 two-node machine with 16 procesors and 32 GB of memory
total. 

I applied all of the six patches you had posted to the latest changeset
as of today(10946:e1a2a8029e9f).

Xen finds the SLIT and SRAT tables. I was able to bring up and down
multiple DomUs. 

I then tried running a set of tests (bringing up domus and running LTP
and Kernbench) and I found that the system was hanging. It wasn't
crashed (I could ping it etc). But it was running so slow that ssh
wouldn't work, even though I could use telnet to get on port 22.

I found out by accident booting into Dom0 and starting xm top hangs the
system in a similar fashion.

Nothing appears on the serial port.

I have attached the serial output.
I am trying to figure out why it is happening.

Is anyone else seeing similar issues? Are there any suggestions on the
approach I can take to debugging this?
Thanks
Raj
Xen Development Team
Unisys Corp.

 \ \/ /___ _ __   |___ / / _ \    _   _ _ __  ___| |_ __ _| |__ | | ___ 
  \  // _ \ '_ \    |_ \| | | |__| | | | '_ \/ __| __/ _` | '_ \| |/ _ \
  /  \  __/ | | |  ___) | |_| |__| |_| | | | \__ \ || (_| | |_) | |  __/
 /_/\_\___|_| |_| |____(_)___/    \__,_|_| |_|___/\__\__,_|_.__/|_|\___|
                                                                        
 http://www.cl.cam.ac.uk/netos/xen
 University of Cambridge Computer Laboratory

 Xen version 3.0-unstable (root@site) (gcc version 4.1.0 (SUSE Linux))
Thu Aug  3 13:49:43 EDT 2006
 Latest ChangeSet: Fri Aug 04 11:36:07 2006 +0100 10946:e1a2a8029e9f

(XEN) Command line: /boot/xen.gz com1=115200,8n1 apic_verbosity=debug
(XEN) Physical RAM map:
(XEN)  0000000000000000 - 000000000009dc00 (usable)
(XEN)  000000000009dc00 - 00000000000a0000 (reserved)
(XEN)  00000000000ce000 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 0000000002670000 (usable)
(XEN)  0000000002670000 - 00000000026d7000 (ACPI data)
(XEN)  00000000026d7000 - 0000000002700000 (ACPI NVS)
(XEN)  0000000002700000 - 00000000f0000000 (usable)
(XEN)  00000000f8000000 - 00000000fec00000 (reserved)
(XEN)  00000000fffc0000 - 0000000100000000 (reserved)
(XEN)  0000000100000000 - 0000001008000000 (usable)
(XEN) System RAM: 65407MB (66976820kB)
(XEN) ACPI: RSDP (v000 PTLTD                                 ) @
0x00000000000f8650
(XEN) ACPI: RSDT (v001 PTLTD    RSDT   0x06040000  LTP 0x06040000) @
0x000000000267de53
(XEN) ACPI: FADT (v004 UNISYS ZORRO    0x06040000 PTL  0x00000008) @
0x00000000026d5f7c
(XEN) ACPI: OEM1 (v001 UNISYS   OEM1   0x06040000  LTP 0x06040000) @
0x00000000026d6070
(XEN) ACPI: OEM2 (v001 UNISYS   OEM2   0x06040000  LTP 0x06040000) @
0x00000000026d60a0
(XEN) ACPI: WSPT (v001 UNISYS   WSPT   0x06040000  LTP 0x06040000) @
0x00000000026d6c9c
(XEN) ACPI: SRAT (v001 UNISYS   SRAT   0x06040000  LTP 0x06040000) @
0x00000000026d6cc4
(XEN) ACPI: MADT (v001 PTLTD  	 APIC   0x06040000  LTP 0x06040000) @
0x00000000026d6e6c
(XEN) ACPI: SPCR (v001 PTLTD  $UCRTBL$ 0x06040000 PTL  0x00000001) @
0x00000000026d6fb0
(XEN) ACPI: DSDT (v001  Intel  870 SMP 0x06040000 MSFT 0x02000001) @
0x0000000000000000
(XEN) SRAT: PXM 2 -> APIC 28 -> Node 0
(XEN) SRAT: PXM 2 -> APIC 26 -> Node 0
(XEN) SRAT: PXM 2 -> APIC 20 -> Node 0
(XEN) SRAT: PXM 2 -> APIC 18 -> Node 0
(XEN) SRAT: PXM 2 -> APIC 30 -> Node 0
(XEN) SRAT: PXM 2 -> APIC 24 -> Node 0
(XEN) SRAT: PXM 2 -> APIC 22 -> Node 0
(XEN) SRAT: PXM 2 -> APIC 16 -> Node 0
(XEN) SRAT: PXM 1 -> APIC 12 -> Node 1
(XEN) SRAT: PXM 1 -> APIC 10 -> Node 1
(XEN) SRAT: PXM 1 -> APIC 4 -> Node 1
(XEN) SRAT: PXM 1 -> APIC 2 -> Node 1
(XEN) SRAT: PXM 1 -> APIC 14 -> Node 1
(XEN) SRAT: PXM 1 -> APIC 8 -> Node 1
(XEN) SRAT: PXM 1 -> APIC 6 -> Node 1
(XEN) SRAT: PXM 1 -> APIC 0 -> Node 1
(XEN) SRAT: Node 1 PXM 1 0-100000
(XEN) SRAT: Node 1 PXM 1 0-808000000
(XEN) SRAT: Node 0 PXM 2 808000000-1008000000
(XEN) NUMA: Using 27 for the hash shift.
(XEN) Reserving non-aligned node boundary @ mfn 8421376
(XEN) Xen heap: 11MB (12280kB)
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) found SMP MP-table at 000f8680
(XEN) DMI present.
(XEN) Using APIC driver default
(XEN) ACPI: Local APIC address 0xfee00000
(XEN) ACPI: OEM1 (v001 UNISYS   OEM1   0x06040000  LTP 0x06040000) @
0x00000000026d6070
(XEN) ACPI: DSDT (v001  Intel  870 SMP 0x06040000 MSFT 0x02000001) @
0x0000000000000000
(XEN) Switched to APIC driver `es7000'.
(XEN) ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
(XEN) Processor #0 15:4 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x06] enabled)
(XEN) Processor #6 15:4 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x02] lapic_id[0x08] enabled)
(XEN) Processor #8 15:4 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x03] lapic_id[0x0e] enabled)
(XEN) Processor #14 15:4 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x04] lapic_id[0x02] enabled)
(XEN) Processor #2 15:4 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x05] lapic_id[0x04] enabled)
(XEN) Processor #4 15:4 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x06] lapic_id[0x0a] enabled)
(XEN) Processor #10 15:4 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x07] lapic_id[0x0c] enabled)
(XEN) Processor #12 15:4 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x08] lapic_id[0x10] enabled)
(XEN) Processor #16 15:4 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x09] lapic_id[0x16] enabled)
(XEN) Processor #22 15:4 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x18] enabled)
(XEN) Processor #24 15:4 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x1e] enabled)
(XEN) Processor #30 15:4 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x12] enabled)
(XEN) Processor #18 15:4 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x14] enabled)
(XEN) Processor #20 15:4 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x1a] enabled)
(XEN) Processor #26 15:4 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x1c] enabled)
(XEN) Processor #28 15:4 APIC version 20
(XEN) ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x04] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x05] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x06] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x07] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x08] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x09] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x0a] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x0b] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x0c] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x0d] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x0e] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x0f] high edge lint[0x1])
(XEN) ACPI: IOAPIC (id[0x10] address[0xfec00000] gsi_base[0])
(XEN) IOAPIC[0]: apic_id 16, version 32, address 0xfec00000, GSI 0-23
(XEN) ACPI: IOAPIC (id[0x1a] address[0xfec04000] gsi_base[72])
(XEN) IOAPIC[1]: apic_id 26, version 32, address 0xfec04000, GSI 72-95
(XEN) ACPI: IOAPIC (id[0x1b] address[0xfec05000] gsi_base[96])
(XEN) IOAPIC[2]: apic_id 27, version 32, address 0xfec05000, GSI 96-119
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge)
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
(XEN) ACPI: IRQ0 used by override.
(XEN) ACPI: IRQ2 used by override.
(XEN) ACPI: IRQ9 used by override.
(XEN) Enabling APIC mode:  Phys.  Using 3 I/O APICs
(XEN) Using ACPI (MADT) for SMP configuration information
(XEN) mapped APIC to ffff828bfffff000 (fee00000)
(XEN) mapped IOAPIC to ffff828bffffe000 (fec00000)
(XEN) mapped IOAPIC to ffff828bffffd000 (fec04000)
(XEN) mapped IOAPIC to ffff828bffffc000 (fec05000)
(XEN) Initializing CPU#0
(XEN) Detected 3000.207 MHz processor.
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 2048K
(XEN) CPU: Physical Processor ID: 0
(XEN) CPU: Processor Core ID: 0
(XEN) VMXON is done
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#0.
(XEN) CPU0: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU0: Thermal monitoring enabled
(XEN) CPU0: Intel Genuine Intel(R) CPU 3.00GHz stepping 08
(XEN) Getting VERSION: 50014
(XEN) Getting VERSION: 50014
(XEN) Getting ID: 0
(XEN) Getting LVT0: 10700
(XEN) Getting LVT1: 10400
(XEN) ES7000: Enabling APIC mode.
(XEN) masked ExtINT on CPU#0
(XEN) Mapping cpu 0 to node 1
(XEN) Booting processor 1/6 eip 90000
(XEN) Initializing CPU#1
(XEN) masked ExtINT on CPU#1
(XEN) Mapping cpu 1 to node 1
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 2048K
(XEN) CPU: Physical Processor ID: 1
(XEN) CPU: Processor Core ID: 1
(XEN) VMXON is done
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#1.
(XEN) CPU1: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU1: Thermal monitoring enabled
(XEN) CPU1: Intel Genuine Intel(R) CPU 3.00GHz stepping 08
(XEN) Booting processor 2/8 eip 90000
(XEN) Initializing CPU#2
(XEN) masked ExtINT on CPU#2
(XEN) Mapping cpu 2 to node 1
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 2048K
(XEN) CPU: Physical Processor ID: 2
(XEN) CPU: Processor Core ID: 0
(XEN) VMXON is done
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#2.
(XEN) CPU2: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU2: Thermal monitoring enabled
(XEN) CPU2: Intel Genuine Intel(R) CPU 3.00GHz stepping 08
(XEN) Booting processor 3/14 eip 90000
(XEN) Initializing CPU#3
(XEN) masked ExtINT on CPU#3
(XEN) Mapping cpu 3 to node 1
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 2048K
(XEN) CPU: Physical Processor ID: 3
(XEN) CPU: Processor Core ID: 1
(XEN) VMXON is done
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#3.
(XEN) CPU3: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU3: Thermal monitoring enabled
(XEN) CPU3: Intel Genuine Intel(R) CPU 3.00GHz stepping 08
(XEN) Booting processor 4/2 eip 90000
(XEN) Initializing CPU#4
(XEN) masked ExtINT on CPU#4
(XEN) Mapping cpu 4 to node 1
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 2048K
(XEN) CPU: Physical Processor ID: 0
(XEN) CPU: Processor Core ID: 1
(XEN) VMXON is done
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#4.
(XEN) CPU4: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU4: Thermal monitoring enabled
(XEN) CPU4: Intel Genuine Intel(R) CPU 3.00GHz stepping 08
(XEN) Booting processor 5/4 eip 90000
(XEN) Initializing CPU#5
(XEN) masked ExtINT on CPU#5
(XEN) Mapping cpu 5 to node 1
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 2048K
(XEN) CPU: Physical Processor ID: 1
(XEN) CPU: Processor Core ID: 0
(XEN) VMXON is done
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#5.
(XEN) CPU5: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU5: Thermal monitoring enabled
(XEN) CPU5: Intel Genuine Intel(R) CPU 3.00GHz stepping 08
(XEN) Booting processor 6/10 eip 90000
(XEN) Initializing CPU#6
(XEN) masked ExtINT on CPU#6
(XEN) Mapping cpu 6 to node 1
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 2048K
(XEN) CPU: Physical Processor ID: 2
(XEN) CPU: Processor Core ID: 1
(XEN) VMXON is done
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#6.
(XEN) CPU6: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU6: Thermal monitoring enabled
(XEN) CPU6: Intel Genuine Intel(R) CPU 3.00GHz stepping 08
(XEN) Booting processor 7/12 eip 90000
(XEN) Initializing CPU#7
(XEN) masked ExtINT on CPU#7
(XEN) Mapping cpu 7 to node 1
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 2048K
(XEN) CPU: Physical Processor ID: 3
(XEN) CPU: Processor Core ID: 0
(XEN) VMXON is done
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#7.
(XEN) CPU7: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU7: Thermal monitoring enabled
(XEN) CPU7: Intel Genuine Intel(R) CPU 3.00GHz stepping 08
(XEN) Booting processor 8/16 eip 90000
(XEN) Initializing CPU#8
(XEN) masked ExtINT on CPU#8
(XEN) Mapping cpu 8 to node 0
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 2048K
(XEN) CPU: Physical Processor ID: 4
(XEN) CPU: Processor Core ID: 0
(XEN) VMXON is done
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#8.
(XEN) CPU8: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU8: Thermal monitoring enabled
(XEN) CPU8: Intel Genuine Intel(R) CPU 3.00GHz stepping 08
(XEN) Booting processor 9/22 eip 90000
(XEN) Initializing CPU#9
(XEN) masked ExtINT on CPU#9
(XEN) Mapping cpu 9 to node 0
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 2048K
(XEN) CPU: Physical Processor ID: 5
(XEN) CPU: Processor Core ID: 1
(XEN) VMXON is done
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#9.
(XEN) CPU9: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU9: Thermal monitoring enabled
(XEN) CPU9: Intel Genuine Intel(R) CPU 3.00GHz stepping 08
(XEN) Booting processor 10/24 eip 90000
(XEN) Initializing CPU#10
(XEN) masked ExtINT on CPU#10
(XEN) Mapping cpu 10 to node 0
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 2048K
(XEN) CPU: Physical Processor ID: 6
(XEN) CPU: Processor Core ID: 0
(XEN) VMXON is done
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#10.
(XEN) CPU10: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU10: Thermal monitoring enabled
(XEN) CPU10: Intel Genuine Intel(R) CPU 3.00GHz stepping 08
(XEN) Booting processor 11/30 eip 90000
(XEN) Initializing CPU#11
(XEN) masked ExtINT on CPU#11
(XEN) Mapping cpu 11 to node 0
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 2048K
(XEN) CPU: Physical Processor ID: 7
(XEN) CPU: Processor Core ID: 1
(XEN) VMXON is done
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#11.
(XEN) CPU11: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU11: Thermal monitoring enabled
(XEN) CPU11: Intel Genuine Intel(R) CPU 3.00GHz stepping 08
(XEN) Booting processor 12/18 eip 90000
(XEN) Initializing CPU#12
(XEN) masked ExtINT on CPU#12
(XEN) Mapping cpu 12 to node 0
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 2048K
(XEN) CPU: Physical Processor ID: 4
(XEN) CPU: Processor Core ID: 1
(XEN) VMXON is done
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#12.
(XEN) CPU12: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU12: Thermal monitoring enabled
(XEN) CPU12: Intel Genuine Intel(R) CPU 3.00GHz stepping 08
(XEN) Booting processor 13/20 eip 90000
(XEN) Initializing CPU#13
(XEN) masked ExtINT on CPU#13
(XEN) Mapping cpu 13 to node 0
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 2048K
(XEN) CPU: Physical Processor ID: 5
(XEN) CPU: Processor Core ID: 0
(XEN) VMXON is done
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#13.
(XEN) CPU13: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU13: Thermal monitoring enabled
(XEN) CPU13: Intel Genuine Intel(R) CPU 3.00GHz stepping 08
(XEN) Booting processor 14/26 eip 90000
(XEN) Initializing CPU#14
(XEN) masked ExtINT on CPU#14
(XEN) Mapping cpu 14 to node 0
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 2048K
(XEN) CPU: Physical Processor ID: 6
(XEN) CPU: Processor Core ID: 1
(XEN) VMXON is done
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#14.
(XEN) CPU14: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU14: Thermal monitoring enabled
(XEN) CPU14: Intel Genuine Intel(R) CPU 3.00GHz stepping 08
(XEN) Booting processor 15/28 eip 90000
(XEN) Initializing CPU#15
(XEN) masked ExtINT on CPU#15
(XEN) Mapping cpu 15 to node 0
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 2048K
(XEN) CPU: Physical Processor ID: 7
(XEN) CPU: Processor Core ID: 0
(XEN) VMXON is done
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#15.
(XEN) CPU15: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU15: Thermal monitoring enabled
(XEN) CPU15: Intel Genuine Intel(R) CPU 3.00GHz stepping 08
(XEN) Total of 16 processors activated.
(XEN) ENABLING IO-APIC IRQs
(XEN)  -> Using new ACK method
(XEN) init IO_APIC IRQs
(XEN)  IO-APIC (apicid-pin) 16-0, 16-16, 16-17, 16-18, 16-19, 16-20,
16-21, 16-22, 16-23, 26-0, 26-1, 26-2, 26-3, 26-4, 26-5, 26-6, 26-7,
26-8, 26-9, 26-10, 26-11, 26-12, 26-13, 26-14, 26-15, 26-16, 26-17,
26-18, 26-19, 26-20, 26-21, 26-22, 26-23, 27-0, 27-1, 27-2, 27-3, 27-4,
27-5, 27-6, 27-7, 27-8, 27-9, 27-10, 27-11, 27-12, 27-13, 27-14, 27-15,
27-16, 27-17, 27-18, 27-19, 27-20, 27-21, 27-22, 27-23 not connected.
(XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=0 pin2=0
(XEN) number of MP IRQ sources: 15.
(XEN) number of IO-APIC #16 registers: 24.
(XEN) number of IO-APIC #26 registers: 24.
(XEN) number of IO-APIC #27 registers: 24.
(XEN) testing the IO APIC.......................
(XEN) IO APIC #16......
(XEN) .... register #00: 00000000
(XEN) .......    : physical APIC id: 00
(XEN) .......    : Delivery Type: 0
(XEN) .......    : LTS          : 0
(XEN) .... register #01: 00178020
(XEN) .......     : max redirection entries: 0017
(XEN) .......     : PRQ implemented: 1
(XEN) .......     : IO APIC version: 0020
(XEN) .... register #02: 00000000
(XEN) .......     : arbitration: 00
(XEN) .... register #03: 00000001
(XEN) .......     : Boot DT    : 1
(XEN) .... IRQ redirection table:
(XEN)  NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect:   
(XEN)  00 000 00  1    0    0   0   0    0    0    00
(XEN)  01 000 00  0    0    0   0   0    0    0    28
(XEN)  02 000 00  0    0    0   0   0    0    0    F0
(XEN)  03 000 00  0    0    0   0   0    0    0    30
(XEN)  04 000 00  0    0    0   0   0    0    0    F1
(XEN)  05 000 00  0    0    0   0   0    0    0    38
(XEN)  06 000 00  0    0    0   0   0    0    0    40
(XEN)  07 000 00  0    0    0   0   0    0    0    48
(XEN)  08 000 00  0    0    0   0   0    0    0    50
(XEN)  09 000 00  1    1    0   0   0    0    0    58
(XEN)  0a 000 00  0    0    0   0   0    0    0    60
(XEN)  0b 000 00  0    0    0   0   0    0    0    68
(XEN)  0c 000 00  0    0    0   0   0    0    0    70
(XEN)  0d 000 00  0    0    0   0   0    0    0    78
(XEN)  0e 000 00  0    0    0   0   0    0    0    88
(XEN)  0f 000 00  0    0    0   0   0    0    0    90
(XEN)  10 000 00  1    0    0   0   0    0    0    00
(XEN)  11 000 00  1    0    0   0   0    0    0    00
(XEN)  12 000 00  1    0    0   0   0    0    0    00
(XEN)  13 000 00  1    0    0   0   0    0    0    00
(XEN)  14 000 00  1    0    0   0   0    0    0    00
(XEN)  15 000 00  1    0    0   0   0    0    0    00
(XEN)  16 000 00  1    0    0   0   0    0    0    00
(XEN)  17 000 00  1    0    0   0   0    0    0    00
(XEN) IO APIC #26......
(XEN) .... register #00: 00000000
(XEN) .......    : physical APIC id: 00
(XEN) .......    : Delivery Type: 0
(XEN) .......    : LTS          : 0
(XEN) .... register #01: 00178020
(XEN) .......     : max redirection entries: 0017
(XEN) .......     : PRQ implemented: 1
(XEN) .......     : IO APIC version: 0020
(XEN) .... register #02: 00000000
(XEN) .......     : arbitration: 00
(XEN) .... register #03: 00000001
(XEN) .......     : Boot DT    : 1
(XEN) .... IRQ redirection table:
(XEN)  NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect:   
(XEN)  00 000 00  1    0    0   0   0    0    0    00
(XEN)  01 000 00  1    0    0   0   0    0    0    00
(XEN)  02 000 00  1    0    0   0   0    0    0    00
(XEN)  03 000 00  1    0    0   0   0    0    0    00
(XEN)  04 000 00  1    0    0   0   0    0    0    00
(XEN)  05 000 00  1    0    0   0   0    0    0    00
(XEN)  06 000 00  1    0    0   0   0    0    0    00
(XEN)  07 000 00  1    0    0   0   0    0    0    00
(XEN)  08 000 00  1    0    0   0   0    0    0    00
(XEN)  09 000 00  1    0    0   0   0    0    0    00
(XEN)  0a 000 00  1    0    0   0   0    0    0    00
(XEN)  0b 000 00  1    0    0   0   0    0    0    00
(XEN)  0c 000 00  1    0    0   0   0    0    0    00
(XEN)  0d 000 00  1    0    0   0   0    0    0    00
(XEN)  0e 000 00  1    0    0   0   0    0    0    00
(XEN)  0f 000 00  1    0    0   0   0    0    0    00
(XEN)  10 000 00  1    0    0   0   0    0    0    00
(XEN)  11 000 00  1    0    0   0   0    0    0    00
(XEN)  12 000 00  1    0    0   0   0    0    0    00
(XEN)  13 000 00  1    0    0   0   0    0    0    00
(XEN)  14 000 00  1    0    0   0   0    0    0    00
(XEN)  15 000 00  1    0    0   0   0    0    0    00
(XEN)  16 000 00  1    0    0   0   0    0    0    00
(XEN)  17 000 00  1    0    0   0   0    0    0    00
(XEN) IO APIC #27......
(XEN) .... register #00: 00000000
(XEN) .......    : physical APIC id: 00
(XEN) .......    : Delivery Type: 0
(XEN) .......    : LTS          : 0
(XEN) .... register #01: 00178020
(XEN) .......     : max redirection entries: 0017
(XEN) .......     : PRQ implemented: 1
(XEN) .......     : IO APIC version: 0020
(XEN) .... register #02: 00000000
(XEN) .......     : arbitration: 00
(XEN) .... register #03: 00000001
(XEN) .......     : Boot DT    : 1
(XEN) .... IRQ redirection table:
(XEN)  NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect:   
(XEN)  00 000 00  1    0    0   0   0    0    0    00
(XEN)  01 000 00  1    0    0   0   0    0    0    00
(XEN)  02 000 00  1    0    0   0   0    0    0    00
(XEN)  03 000 00  1    0    0   0   0    0    0    00
(XEN)  04 000 00  1    0    0   0   0    0    0    00
(XEN)  05 000 00  1    0    0   0   0    0    0    00
(XEN)  06 000 00  1    0    0   0   0    0    0    00
(XEN)  07 000 00  1    0    0   0   0    0    0    00
(XEN)  08 000 00  1    0    0   0   0    0    0    00
(XEN)  09 000 00  1    0    0   0   0    0    0    00
(XEN)  0a 000 00  1    0    0   0   0    0    0    00
(XEN)  0b 000 00  1    0    0   0   0    0    0    00
(XEN)  0c 000 00  1    0    0   0   0    0    0    00
(XEN)  0d 000 00  1    0    0   0   0    0    0    00
(XEN)  0e 000 00  1    0    0   0   0    0    0    00
(XEN)  0f 000 00  1    0    0   0   0    0    0    00
(XEN)  10 000 00  1    0    0   0   0    0    0    00
(XEN)  11 000 00  1    0    0   0   0    0    0    00
(XEN)  12 000 00  1    0    0   0   0    0    0    00
(XEN)  13 000 00  1    0    0   0   0    0    0    00
(XEN)  14 000 00  1    0    0   0   0    0    0    00
(XEN)  15 000 00  1    0    0   0   0    0    0    00
(XEN)  16 000 00  1    0    0   0   0    0    0    00
(XEN)  17 000 00  1    0    0   0   0    0    0    00
(XEN) Using vector-based indexing
(XEN) IRQ to pin mappings:
(XEN) IRQ240 -> 0:2
(XEN) IRQ40 -> 0:1
(XEN) IRQ48 -> 0:3
(XEN) IRQ241 -> 0:4
(XEN) IRQ56 -> 0:5
(XEN) IRQ64 -> 0:6
(XEN) IRQ72 -> 0:7
(XEN) IRQ80 -> 0:8
(XEN) IRQ88 -> 0:9
(XEN) IRQ96 -> 0:10
(XEN) IRQ104 -> 0:11
(XEN) IRQ112 -> 0:12
(XEN) IRQ120 -> 0:13
(XEN) IRQ136 -> 0:14
(XEN) IRQ144 -> 0:15
(XEN) .................................... done.
(XEN) Using local APIC timer interrupts.
(XEN) calibrating APIC timer ...
(XEN) ..... CPU clock speed is 3000.0389 MHz.
(XEN) ..... host bus clock speed is 200.0025 MHz.
(XEN) ..... bus_scale = 0x0000CCD7
(XEN) checking TSC synchronization across 16 CPUs: 
(XEN) CPU#0 had 683915 usecs TSC skew, fixed it up.
(XEN) CPU#1 had 683926 usecs TSC skew, fixed it up.
(XEN) CPU#2 had 683932 usecs TSC skew, fixed it up.
(XEN) CPU#3 had 683927 usecs TSC skew, fixed it up.
(XEN) CPU#4 had 683926 usecs TSC skew, fixed it up.
(XEN) CPU#5 had 683934 usecs TSC skew, fixed it up.
(XEN) CPU#6 had 683926 usecs TSC skew, fixed it up.
(XEN) CPU#7 had 683927 usecs TSC skew, fixed it up.
(XEN) CPU#8 had -683927 usecs TSC skew, fixed it up.
(XEN) CPU#9 had -683928 usecs TSC skew, fixed it up.
(XEN) CPU#10 had -683927 usecs TSC skew, fixed it up.
(XEN) CPU#11 had -683927 usecs TSC skew, fixed it up.
(XEN) CPU#12 had -683928 usecs TSC skew, fixed it up.
(XEN) CPU#13 had -683927 usecs TSC skew, fixed it up.
(XEN) CPU#14 had -683928 usecs TSC skew, fixed it up.
(XEN) CPU#15 had -683921 usecs TSC skew, fixed it up.
(XEN) Platform timer is 1.193MHz PIT
(XEN) CPU 0 APIC 0 -> Node 1
(XEN) CPU 1 APIC 6 -> Node 1
(XEN) CPU 2 APIC 8 -> Node 1
(XEN) CPU 3 APIC 14 -> Node 1
(XEN) CPU 4 APIC 2 -> Node 1
(XEN) CPU 5 APIC 4 -> Node 1
(XEN) CPU 6 APIC 10 -> Node 1
(XEN) CPU 7 APIC 12 -> Node 1
(XEN) CPU 8 APIC 16 -> Node 0
(XEN) CPU 9 APIC 22 -> Node 0
(XEN) CPU 10 APIC 24 -> Node 0
(XEN) CPU 11 APIC 30 -> Node 0
(XEN) CPU 12 APIC 18 -> Node 0
(XEN) CPU 13 APIC 20 -> Node 0
(XEN) CPU 14 APIC 26 -> Node 0
(XEN) CPU 15 APIC 28 -> Node 0
(XEN) Brought up 16 CPUs
(XEN) Machine check exception polling timer started.
(XEN) *** LOADING DOMAIN 0 ***
(XEN) Domain 0 kernel supports features = { 0000000f }.
(XEN) Domain 0 kernel requires features = { 00000000 }.
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN)  Dom0 alloc.:   0000000030000000->0000000040000000 (16444393 pages
to be allocated)
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN)  Loaded kernel: ffffffff80100000->ffffffff804b4f08
(XEN)  Init. ramdisk: ffffffff804b5000->ffffffff80b8de00
(XEN)  Phys-Mach map: ffffffff80b8e000->ffffffff88983f48
(XEN)  Start info:    ffffffff88984000->ffffffff88985000
(XEN)  Page tables:   ffffffff88985000->ffffffff889ce000
(XEN)  Boot stack:    ffffffff889ce000->ffffffff889cf000
(XEN)  TOTAL:         ffffffff80000000->ffffffff88c00000
(XEN)  ENTRY ADDRESS: ffffffff80100000
(XEN) Dom0 has maximum 16 VCPUs
(XEN) Initrd len 0x6d8e00, start at 0xffffffff804b5000
(XEN) Scrubbing Free RAM:
........................................................................
........................................................................
........................................................................
........................................................................
........................................................................
........................................................................
........................................................................
........................................................................
........................................................................
.........done.
(XEN) Xen trace buffers: disabled
(XEN) Xen is relinquishing VGA console.
(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch
input to Xen).
kernel direct mapping tables up to fbf3e9000 @ 8a0d000-10847000
Bootdata ok (command line is root=/dev/sda1 vga=0x314 selinux=0
resume=/dev/sda2  splash=native showopts)
Linux version 2.6.16.13-xen (root@m1132-xenunstable) (gcc version 4.1.0
(SUSE Linux)) #1 SMP Thu Aug 3 14:13:58 EDT 2006
BIOS-provided physical RAM map:
 Xen: 0000000000000000 - 0000000fbf3e9000 (usable)
ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x06] enabled)
ACPI: LAPIC (acpi_id[0x02] lapic_id[0x08] enabled)
ACPI: LAPIC (acpi_id[0x03] lapic_id[0x0e] enabled)
ACPI: LAPIC (acpi_id[0x04] lapic_id[0x02] enabled)
ACPI: LAPIC (acpi_id[0x05] lapic_id[0x04] enabled)
ACPI: LAPIC (acpi_id[0x06] lapic_id[0x0a] enabled)
ACPI: LAPIC (acpi_id[0x07] lapic_id[0x0c] enabled)
ACPI: LAPIC (acpi_id[0x08] lapic_id[0x10] enabled)
ACPI: LAPIC (acpi_id[0x09] lapic_id[0x16] enabled)
ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x18] enabled)
ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x1e] enabled)
ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x12] enabled)
ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x14] enabled)
ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x1a] enabled)
ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x1c] enabled)
ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x04] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x05] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x06] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x07] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x08] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x09] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x0a] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x0b] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x0c] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x0d] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x0e] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x0f] high edge lint[0x1])
ACPI: IOAPIC (id[0x10] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 16, version 32, address 0xfec00000, GSI 0-23
ACPI: IOAPIC (id[0x1a] address[0xfec04000] gsi_base[72])
IOAPIC[1]: apic_id 26, version 32, address 0xfec04000, GSI 72-95
ACPI: IOAPIC (id[0x1b] address[0xfec05000] gsi_base[96])
IOAPIC[2]: apic_id 27, version 32, address 0xfec05000, GSI 96-119
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge)
ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
Setting APIC routing to xen
Using ACPI (MADT) for SMP configuration information
Allocating PCI resources starting at f0800000 (gap: f0000000:8000000)
Built 1 zonelists
Kernel command line: root=/dev/sda1 vga=0x314 selinux=0
resume=/dev/sda2  splash=native showopts
Initializing CPU#0
PID hash table entries: 4096 (order: 12, 131072 bytes)
Xen reported: 3000.010 MHz processor.
Console: colour VGA+ 80x25
Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes)
Inode-cache hash table entries: 4194304 (order: 13, 33554432 bytes)
Software IO TLB enabled: 
 Aperture:     64 megabytes
 Kernel range: 0xffff88004fa29000 - 0xffff880053a29000
PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
Memory: 64698932k/66047908k available (1898k kernel code, 1340164k
reserved, 808k data, 168k init)
Calibrating delay using timer specific routine.. 6001.70 BogoMIPS
(lpj=30008520)
Security Framework v1.0.0 initialized
Capability LSM initialized
Mount-cache hash table entries: 256
CPU: Trace cache: 12K uops, L1 D cache: 16K
CPU: L2 cache: 2048K
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 0
Initializing CPU#1
Initializing CPU#2
Initializing CPU#3
Initializing CPU#4
Initializing CPU#5
Initializing CPU#6
Initializing CPU#7
Initializing CPU#8
Initializing CPU#9
Initializing CPU#10
Initializing CPU#11
Initializing CPU#12
Initializing CPU#13
Initializing CPU#14
Brought up 16 CPUs
Initializing CPU#15
migration_cost=1555
checking if image is initramfs... it is
Freeing initrd memory: 7011k freed
DMI present.
Grant table initialized
NET: Registered protocol family 16
ACPI: bus type pci registered
PCI: Using configuration type 1
ACPI: Subsystem revision 20060127
ACPI: Interpreter enabled
ACPI: Using IOAPIC for interrupt routing
ACPI: Device [MBDV] status [00000008]: functional but not present;
setting present
ACPI: PCI Root Bridge [MBDV] (0000:ff)
ACPI: PCI Root Bridge [S1H0] (0000:00)
PCI quirk: region 0d00-0d7f claimed by ICH4 ACPI/GPIO/TCO
PCI quirk: region 0e80-0ebf claimed by ICH4 GPIO
PCI: Ignoring BAR0-3 of IDE controller 0000:00:1f.1
PCI: Transparent bridge - 0000:00:1e.0
ACPI: PCI Interrupt Link [LNKA] (IRQs 3 10 *11 14 15)
ACPI: PCI Interrupt Link [LNKB] (IRQs 3 *10 11 14 15)
ACPI: PCI Interrupt Link [LNKC] (IRQs 3 10 *11 14 15)
ACPI: PCI Interrupt Link [LNKD] (IRQs 3 10 *11 14 15)
ACPI: PCI Interrupt Link [LNKE] (IRQs 3 10 11 14 15) *0, disabled.
ACPI: PCI Interrupt Link [LNKF] (IRQs 3 10 11 14 15) *12
ACPI: PCI Interrupt Link [LNKG] (IRQs 3 10 11 14 15) *12
ACPI: PCI Interrupt Link [LNKH] (IRQs 3 *6 10 11 14 15)
ACPI: PCI Root Bridge [S1H2] (0000:0f)
PCI: Enable I/O Space to 1 KB Granularity
PCI: Enable I/O Space to 1 KB Granularity
ACPI: PCI Interrupt Link [LNKA] (IRQs *10)
Linux Plug and Play Support v0.97 (c) Adam Belay
pnp: PnP ACPI init
pnp: PnP ACPI: found 8 devices
xen_mem: Initialising balloon driver.
PCI: Using ACPI for IRQ routing
PCI: If a device doesn't work, try "pci=routeirq".  If it helps, post a
report
PCI: Bridge: 0000:00:1e.0
  IO window: 1000-1fff
  MEM window: f6000000-f7efffff
  PREFETCH window: f0800000-f08fffff
PCI: Bridge: 0000:0f:1d.0
  IO window: 2000-23ff
  MEM window: f4f00000-f4ffffff
  PREFETCH window: disabled.
PCI: Failed to allocate mem resource #6:100000@f4000000 for 0000:16:01.1
PCI: Bridge: 0000:0f:1f.0
  IO window: 2400-27ff
  MEM window: f4d00000-f4efffff
  PREFETCH window: f3e00000-f3ffffff
IA-32 Microcode Update Driver: v1.14-xen <tigran@veritas.com>
IA32 emulation $Id: sys_ia32.c,v 1.32 2002/03/24 13:02:28 ak Exp $
audit: initializing netlink socket (disabled)
audit(1154632323.818:1): initialized
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
Initializing Cryptographic API
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
Real Time Clock Driver v1.12ac
Non-volatile memory driver v1.2
PNP: No PS/2 controller found. Probing ports directly.
serio: i8042 AUX port at 0x60,0x64 irq 12
serio: i8042 KBD port at 0x60,0x64 irq 1
RAMDISK driver initialized: 16 RAM disks of 16384K size 1024 blocksize
loop: loaded (max 8 devices)
Xen virtual console successfully installed as ttyS0
Event-channel device installed.
blkif_init: reqs=64, pages=704, mmap_vstart=0xffff880fbe400000
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with
idebus=xx
mice: PS/2 mouse device common for all mice
md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27
md: bitmap version 4.39
NET: Registered protocol family 2
IP route cache hash table entries: 524288 (order: 10, 4194304 bytes)
TCP established hash table entries: 262144 (order: 10, 4194304 bytes)
TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
TCP: Hash tables configured (established 262144 bind 65536)
TCP reno registered
NET: Registered protocol family 1
NET: Registered protocol family 17
scsi_mod: no version for "struct_module" found: kernel tainted.
SCSI subsystem initialized
ICH4: IDE controller at PCI slot 0000:00:1f.1
GSI 16 sharing vector 0x98 and IRQ 16
ACPI: PCI Interrupt 0000:00:1f.1[A] -> GSI 18 (level, low) -> IRQ 16
ICH4: chipset revision 2
ICH4: not 100% native mode: will probe irqs later
    ide0: BM-DMA at 0x0880-0x0887, BIOS settings: hda:DMA, hdb:pio
    ide1: BM-DMA at 0x0888-0x088f, BIOS settings: hdc:pio, hdd:pio
hda: TOSHIBA CD/DVDW SD-R6472, ATAPI CD/DVD-ROM drive
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
Fusion MPT base driver 3.03.07
Copyright (c) 1999-2005 LSI Logic Corporation
Fusion MPT SPI Host driver 3.03.07
GSI 17 sharing vector 0xA0 and IRQ 17
ACPI: PCI Interrupt 0000:16:01.0[A] -> GSI 96 (level, low) -> IRQ 17
mptbase: Initiating ioc0 bringup
ioc0: 53C1030: Capabilities={Initiator,Target}
scsi0 : ioc0: LSI53C1030, FwRev=01032700h, Ports=1, MaxQ=255, IRQ=17
  Vendor: UNISYS    Model: 014228ST118202LC  Rev: B603
  Type:   Direct-Access                      ANSI SCSI revision: 02
SCSI device sda: 35833870 512-byte hdwr sectors (18347 MB)
sda: Write Protect is off
SCSI device sda: drive cache: write back w/ FUA
SCSI device sda: 35833870 512-byte hdwr sectors (18347 MB)
sda: Write Protect is off
SCSI device sda: drive cache: write back w/ FUA
 sda: sda1 sda2
sd 0:0:0:0: Attached scsi disk sda
sd 0:0:0:0: Attached scsi generic sg0 type 0
GSI 18 sharing vector 0xA8 and IRQ 18
ACPI: PCI Interrupt 0000:16:01.1[B] -> GSI 97 (level, low) -> IRQ 18
mptbase: Initiating ioc1 bringup
ioc1: 53C1030: Capabilities={Initiator,Target}
scsi1 : ioc1: LSI53C1030, FwRev=01032700h, Ports=1, MaxQ=255, IRQ=18
ReiserFS: sda1: found reiserfs format "3.6" with standard journal
ReiserFS: sda1: using ordered data mode
ReiserFS: sda1: journal params: device sda1, size 8192, journal first
block 18, max trans len 1024, max batch 900, max commit age 30, max
trans age 30
ReiserFS: sda1: checking transaction log (sda1)
ReiserFS: sda1: Using r5 hash to sort names
Adding 2176796k swap on /dev/sda2.  Priority:-1 extents:1
across:2176796k
Intel(R) PRO/1000 Network Driver - version 6.3.9-k4-NAPI
Copyright (c) 1999-2005 Intel Corporation.
GSI 19 sharing vector 0xB0 and IRQ 19
ACPI: PCI Interrupt 0000:10:01.0[A] -> GSI 72 (level, low) -> IRQ 19
e1000: 0000:10:01.0: e1000_probe: (PCI-X:133MHz:64-bit)
08:00:0b:1e:1a:14
e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection
GSI 20 sharing vector 0xB8 and IRQ 20
ACPI: PCI Interrupt 0000:10:01.1[B] -> GSI 73 (level, low) -> IRQ 20
e1000: 0000:10:01.1: e1000_probe: (PCI-X:133MHz:64-bit)
08:00:0b:1e:1a:15
e1000: eth1: e1000_probe: Intel(R) PRO/1000 Network Connection
device-mapper: 4.5.0-ioctl (2005-10-04) initialised: dm-devel@redhat.com
ACPI: Power Button (FF) [PWRF]
e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex
(XEN) mtrr: type mismatch for f6000000,400000 old: uncachable new:
write-combining
(XEN) mtrr: type mismatch for f6000000,400000 old: uncachable new:
write-combining
[-- MARK -- Fri Aug  4 19:00:00 2006]

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [PATCH 0/6] xen, xend, tools: Add NUMA support to Xen Issues on the ES7000
  2006-08-05  0:33         ` [PATCH 0/6] xen, xend, tools: Add NUMA support to Xen Issues on the ES7000 Subrahmanian, Raj
@ 2006-08-07 13:23           ` Ryan Harper
  2006-08-07 19:29             ` Subrahmanian, Raj
  0 siblings, 1 reply; 11+ messages in thread
From: Ryan Harper @ 2006-08-07 13:23 UTC (permalink / raw)
  To: Subrahmanian, Raj; +Cc: xen-devel

* Subrahmanian, Raj <raj.subrahmanian@unisys.com> [2006-08-04 19:34]:
> Ryan,
> I tried out these patches on an ES7000. 
> It's an x86-64 two-node machine with 16 procesors and 32 GB of memory
> total. 

Thanks for trying the patches out.

> 
> I applied all of the six patches you had posted to the latest changeset
> as of today(10946:e1a2a8029e9f).
> 
> Xen finds the SLIT and SRAT tables. I was able to bring up and down
> multiple DomUs. 
> 
> I then tried running a set of tests (bringing up domus and running LTP
> and Kernbench) and I found that the system was hanging. It wasn't
> crashed (I could ping it etc). But it was running so slow that ssh
> wouldn't work, even though I could use telnet to get on port 22.
> 
> I found out by accident booting into Dom0 and starting xm top hangs the
> system in a similar fashion.

That doesn't look good.  And if you roll back the patches, doing the
above works without issue?  I'll see if I can lock up my box using xm
top.

> 
> Nothing appears on the serial port.
> 
> I have attached the serial output.
> I am trying to figure out why it is happening.

In any case, I'd be interested in seeing the 'H' and 'u' (heapinfo and
numa topology) keyhandlers.  Also, from dom0, running a xen_numastat -H
-d -1 (dump the heap and the memory to node mapping of all running
domains).


-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com

^ permalink raw reply	[flat|nested] 11+ messages in thread

* RE: [PATCH 0/6] xen, xend, tools: Add NUMA support to Xen Issues on the ES7000
  2006-08-07 13:23           ` Ryan Harper
@ 2006-08-07 19:29             ` Subrahmanian, Raj
  0 siblings, 0 replies; 11+ messages in thread
From: Subrahmanian, Raj @ 2006-08-07 19:29 UTC (permalink / raw)
  To: Ryan Harper; +Cc: xen-devel

Ryan, 
> > Ryan,
> > I tried out these patches on an ES7000. 
> > It's an x86-64 two-node machine with 16 procesors and 32 GB 
> of memory 
> > total.
> 
> Thanks for trying the patches out.
We have a vested interest in making sure this works too :-)

> > I found out by accident booting into Dom0 and starting xm top hangs 
> > the system in a similar fashion.
> That doesn't look good.  And if you roll back the patches, 
> doing the above works without issue?  I'll see if I can lock 
> up my box using xm top.
Yes. We run LTP and Kernbench on DomUs of different sizes and these
tests complete on the main-line tree. 

> > 
> > Nothing appears on the serial port.
> > 
> > I have attached the serial output.
> > I am trying to figure out why it is happening.
> 
> In any case, I'd be interested in seeing the 'H' and 'u' 
> (heapinfo and numa topology) keyhandlers.  Also, from dom0, 
> running a xen_numastat -H -d -1 (dump the heap and the memory 
> to node mapping of all running domains).
I will do this and get back to you.

Thanks
Raj

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [PATCH 0/6] xen,xend,tools: Add NUMA support to Xen
  2006-07-31 19:09 [PATCH 0/6] xen,xend,tools: Add NUMA support to Xen Ryan Harper
  2006-08-01  7:46 ` Tristan Gingold
@ 2006-08-01  9:29 ` Tristan Gingold
  2006-08-02  6:14 ` Tristan Gingold
  2 siblings, 0 replies; 11+ messages in thread
From: Tristan Gingold @ 2006-08-01  9:29 UTC (permalink / raw)
  To: Ryan Harper, xen-devel

Le Lundi 31 Juillet 2006 21:09, Ryan Harper a écrit :
> I've respun the NUMA patches against 10874 and I'm re-submitting them
> with the optimizations mentioned [1]previously on the list.  There was a
> request to see the overhead on non-numa/single-node machines.  I've
> re-run those benchmarks (ballooning up from small mem to multi-gig) as
> well as timing the initially domain increase_reservation time to gauge
> the overhead when allocating from the heap.
Hi,

one more comment:
in dom0_ops.c:
+    case DOM0_AVAILHEAP:
+    {
+        ret = -EINVAL;
+        if ( op->u.availheap.node >= num_online_nodes() )
+            break;
+        if ( op->u.availheap.zone >= NR_ZONES )
+            break;
+
+        /* indicate the number of zones/nodes queried.
+         * NB: -1 is wild card for all zones/nodes */
+        ( op->u.availheap.zone < 0 ) ?
+            (op->u.availheap.nr_zones=NR_ZONES) :
+            (op->u.availheap.nr_zones=1);
+
+        ( op->u.availheap.node < 0 ) ?
+            (op->u.availheap.nr_nodes=num_online_nodes()) :
+            (op->u.availheap.nr_nodes=1);

I am not sure this corresponds to the Xen style.
Either use an if statement or a variable assignment:
op->u.availheap.nr_nodes = ( op->u.availheap.node < 0 ?)  num_online_nodes() : 
1;

Tristan.

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [PATCH 0/6] xen,xend,tools: Add NUMA support to Xen
  2006-07-31 19:09 [PATCH 0/6] xen,xend,tools: Add NUMA support to Xen Ryan Harper
  2006-08-01  7:46 ` Tristan Gingold
  2006-08-01  9:29 ` [PATCH 0/6] xen,xend,tools: Add NUMA support to Xen Tristan Gingold
@ 2006-08-02  6:14 ` Tristan Gingold
  2 siblings, 0 replies; 11+ messages in thread
From: Tristan Gingold @ 2006-08-02  6:14 UTC (permalink / raw)
  To: xen-devel

Le Lundi 31 Juillet 2006 21:09, Ryan Harper a écrit :
> I've respun the NUMA patches against 10874 and I'm re-submitting them
> with the optimizations mentioned [1]previously on the list.  There was a
> request to see the overhead on non-numa/single-node machines.  I've
> re-run those benchmarks (ballooning up from small mem to multi-gig) as
> well as timing the initially domain increase_reservation time to gauge
> the overhead when allocating from the heap.
Hi,

two more points.

For DOM0_PHYSINFO, sockets_per_node is wrong:
      pi->sockets_per_node = 
            num_online_cpus() / cpus_weight(cpu_core_map[0]);
[On ia64, I return the max of sockets per node].

The node_to_cpu field of struct dom0_physinfo is:
    XEN_GUEST_HANDLE(u64) node_to_cpu;
This means the max cpus is 64.
I know cpumap_t is also 64 bits.
I think you'd better to use cpumap_t.

Here we have NUMA machines with 128 cpus.  The question is how to upgrade Xen 
to handle more than 64 cpus.  We can either incease cpumap_t size to 256 
(until we reach a new limit) or use an unlimited cpu structure.

Tristan.

^ permalink raw reply	[flat|nested] 11+ messages in thread

end of thread, other threads:[~2006-08-07 19:29 UTC | newest]

Thread overview: 11+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2006-07-31 19:09 [PATCH 0/6] xen,xend,tools: Add NUMA support to Xen Ryan Harper
2006-08-01  7:46 ` Tristan Gingold
2006-08-01 15:40   ` Ryan Harper
2006-08-02  5:59     ` Tristan Gingold
2006-08-02 22:29       ` Ryan Harper
2006-08-03  6:05         ` Tristan Gingold
2006-08-05  0:33         ` [PATCH 0/6] xen, xend, tools: Add NUMA support to Xen Issues on the ES7000 Subrahmanian, Raj
2006-08-07 13:23           ` Ryan Harper
2006-08-07 19:29             ` Subrahmanian, Raj
2006-08-01  9:29 ` [PATCH 0/6] xen,xend,tools: Add NUMA support to Xen Tristan Gingold
2006-08-02  6:14 ` Tristan Gingold

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.