[PATCH] Fix hypervisor crash with unpopulated NUMA nodes

[PATCH] Fix hypervisor crash with unpopulated NUMA nodes

[Andre Przywara]

[-- Attachment #1: Type: text/plain, Size: 1292 bytes --]

Hi Keir,

on NUMA systems with memory-less nodes Xen crashes quite early in the 
hypervisor (while initializing the heaps). This is not an issue if this 
happens to be the last node, but "inner" nodes trigger this reliably.
On multi-node processors it is much more likely to leave a node unequipped.
The attached patch fixes this by enumerating the node via the 
node_online_map instead of counting from 0 to num_nodes.

The resulting NUMA setup is still somewhat strange, but at least it does 
not crash. In lowlevel/xc/xc.c there is again this enumeration bug, but 
I suppose we cannot access the HV's node_online_map from this context, 
so the xm info output is not correct (but xm debug-keys H is).
I plan to rework the handling of memory-less nodes later.

Please also consider committing this to the -testing trees.

Regards,
Andre.

Signed-off-by: Andre Przywara <andre.przywara@amd.com>

-- 
Andre Przywara
AMD-Operating System Research Center (OSRC), Dresden, Germany
Tel: +49 351 448 3567 12
----to satisfy European Law for business letters:
Advanced Micro Devices GmbH
Karl-Hammerschmidt-Str. 34, 85609 Dornach b. Muenchen
Geschaeftsfuehrer: Andrew Bowd; Thomas M. McCoy; Giuliano Meroni
Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
Registergericht Muenchen, HRB Nr. 43632

[-- Attachment #2: sparse_NUMA.patch --]
[-- Type: text/x-patch, Size: 1377 bytes --]

diff -r 440d22c01652 xen/common/page_alloc.c
--- a/xen/common/page_alloc.c	Wed Oct 07 08:07:06 2009 +0100
+++ b/xen/common/page_alloc.c	Wed Oct 07 11:40:54 2009 +0200
@@ -294,7 +294,6 @@
         node = cpu_to_node(smp_processor_id());
 
     ASSERT(node >= 0);
-    ASSERT(node < num_nodes);
     ASSERT(zone_lo <= zone_hi);
     ASSERT(zone_hi < NR_ZONES);
 
@@ -323,8 +322,9 @@
         } while ( zone-- > zone_lo ); /* careful: unsigned zone may wrap */
 
         /* Pick next node, wrapping around if needed. */
-        if ( ++node == num_nodes )
-            node = 0;
+        node = next_node(node, node_online_map);
+        if (node == MAX_NUMNODES)
+            node = first_node(node_online_map);
     }
 
     /* Try to free memory from tmem */
@@ -466,7 +466,6 @@
 
     ASSERT(order <= MAX_ORDER);
     ASSERT(node >= 0);
-    ASSERT(node < num_online_nodes());
 
     for ( i = 0; i < (1 << order); i++ )
     {
@@ -817,13 +816,13 @@
 static unsigned long avail_heap_pages(
     unsigned int zone_lo, unsigned int zone_hi, unsigned int node)
 {
-    unsigned int i, zone, num_nodes = num_online_nodes();
+    unsigned int i, zone;
     unsigned long free_pages = 0;
 
     if ( zone_hi >= NR_ZONES )
         zone_hi = NR_ZONES - 1;
 
-    for ( i = 0; i < num_nodes; i++ )
+    for_each_online_node(i)
     {
         if ( !avail[i] )
             continue;

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: [PATCH] Fix hypervisor crash with unpopulated NUMA nodes

[Jan Beulich]

>>> Andre Przywara <andre.przywara@amd.com> 07.10.09 11:45 >>>
>on NUMA systems with memory-less nodes Xen crashes quite early in the 
>hypervisor (while initializing the heaps). This is not an issue if this 
>happens to be the last node, but "inner" nodes trigger this reliably.
>On multi-node processors it is much more likely to leave a node unequipped.
>The attached patch fixes this by enumerating the node via the 
>node_online_map instead of counting from 0 to num_nodes.

While I do not see anything wrong with the patch, I still wonder why it
would be needed: It seems to indicate that node_online_map represents
only nodes with memory, but imo should be representing nodes with
memory or processors (leaving aside pure I/O nodes for the moment).
So perhaps there's rather a problem with the setup of node_online_map 
somewhere?

Jan

Re: [PATCH] Fix hypervisor crash with unpopulated NUMA nodes

[Andre Przywara]

Jan Beulich wrote:
>>>> Andre Przywara <andre.przywara@amd.com> 07.10.09 11:45 >>>
>> on NUMA systems with memory-less nodes Xen crashes quite early in the 
>> hypervisor (while initializing the heaps). This is not an issue if this 
>> happens to be the last node, but "inner" nodes trigger this reliably.
>> On multi-node processors it is much more likely to leave a node unequipped.
>> The attached patch fixes this by enumerating the node via the 
>> node_online_map instead of counting from 0 to num_nodes.
> 
> While I do not see anything wrong with the patch, I still wonder why it
> would be needed: It seems to indicate that node_online_map represents
> only nodes with memory, but imo should be representing nodes with
> memory or processors (leaving aside pure I/O nodes for the moment).
> So perhaps there's rather a problem with the setup of node_online_map 
> somewhere?
Yes, because the map creation is callback driven by ACPI code.
The BIOS of my machine is omitting the memory entries for memory-less 
nodes, so there is no callback triggered for these nodes. Nevertheless 
Xen uses the SRAT provided node numbers, this creates the hole.
(My setup: 2 + 0 + 2 + 0 GB per node, Xen sees two nodes named 0 and 2).
I agree that should be changed (that is what I meant with "will rework 
later"), not only because the "lonely" cores will simply be added to 
another node. But since I will be not in the office for the next two 
weeks I would like to get this patch applied for the time being.

Regards,
Andre.

-- 
Andre Przywara
AMD-Operating System Research Center (OSRC), Dresden, Germany
Tel: +49 351 448 3567 12
----to satisfy European Law for business letters:
Advanced Micro Devices GmbH
Karl-Hammerschmidt-Str. 34, 85609 Dornach b. Muenchen
Geschaeftsfuehrer: Andrew Bowd; Thomas M. McCoy; Giuliano Meroni
Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
Registergericht Muenchen, HRB Nr. 43632