Re: Memory allocation in NUMA system

[Andre Przywara <andre.przywara@amd.com>]

[-- Attachment #1: Type: text/plain, Size: 2067 bytes --]

Keir Fraser wrote:
> On 28/7/08 13:21, "Andre Przywara" <andre.przywara@amd.com> wrote:
> 
>> Mmh, why not check this in in 3.3? I have noticed this problem already a
>> year ago and was having some other kind of fix for it (which actually
>> prefered nodes over zones):
>> http://lists.xensource.com/archives/html/xen-devel/2007-12/msg00831.html
>> I think this is a somewhat serious issue on NUMA machines, since with
>> the automatic pinning now active (new in 3.3!) many domains will end up
>> with remote memory _all the time_. So I think of this as a bugfix.
>> Actually I have dma_bitsize=30 hardwired in my Grub's menu.lst for some
>> months now...
> 
> Well, fine, but unfortunately the patch breaks ia64
Fixed.
> and doesn't even work properly:
>  - why should NUMA node 0 be the one that overlaps with default DMA memory?
Because that is the most common configuration? Do you know of any 
machine where this is not true? I agree that a dual node machine with 2 
gig on each node does not need this patch, but NUMA machines tend to 
have more memory than this (especially given the current memory costs). 
I changed the default DMA_BITSIZE to 30 bits, this seems to be a 
reasonable value.
>  - a 'large' NUMA node 0 will cause dma_bitsize to be set much larger than
> it is currently, thus breaking its original intent.
Fixed in the attached patch. It now caps dma_bitsize to at most 1/4 of 
node0 memory.

What about using this patch for Xen 3.3 and work out a more general 
solution for Xen 3.4?

Signed off by: Andre Przywara <andre.przywara@amd.com>
Based on the patch from: "Yang, Xiaowei" <xiaowei.yang@intel.com>

-- 
Andre Przywara
AMD-Operating System Research Center (OSRC), Dresden, Germany
Tel: +49 351 277-84917
----to satisfy European Law for business letters:
AMD Saxony Limited Liability Company & Co. KG,
Wilschdorfer Landstr. 101, 01109 Dresden, Germany
Register Court Dresden: HRA 4896, General Partner authorized
to represent: AMD Saxony LLC (Wilmington, Delaware, US)
General Manager of AMD Saxony LLC: Dr. Hans-R. Deppe, Thomas McCoy

[-- Attachment #2: dma_bitsize.patch --]
[-- Type: text/plain, Size: 1434 bytes --]

diff -r 37fae02cc335 xen/common/page_alloc.c
--- a/xen/common/page_alloc.c	Fri Jul 25 15:03:03 2008 +0100
+++ b/xen/common/page_alloc.c	Mon Jul 28 16:02:40 2008 +0200
@@ -583,6 +583,13 @@
             init_heap_pages(pfn_dom_zone_type(i), mfn_to_page(i), 1);
     }
 
+    /* Reserve only up to 25% of node0's memory for DMA */
+    i = pfn_dom_zone_type(NODE_DATA(0)->node_spanned_pages / 4)
+                          + PAGE_SHIFT;
+    if ( i < dma_bitsize ) dma_bitsize = i;
+
+    ASSERT(dma_bitsize > PAGE_SHIFT + 1);
+
     printk("Domain heap initialised: DMA width %u bits\n", dma_bitsize);
 }
 #undef avail_for_domheap
diff -r 37fae02cc335 xen/include/asm-ia64/config.h
--- a/xen/include/asm-ia64/config.h	Fri Jul 25 15:03:03 2008 +0100
+++ b/xen/include/asm-ia64/config.h	Mon Jul 28 16:02:40 2008 +0200
@@ -44,7 +44,7 @@
 #define CONFIG_IOSAPIC
 #define supervisor_mode_kernel (0)
 
-#define CONFIG_DMA_BITSIZE 32
+#define CONFIG_DMA_BITSIZE 30
 
 #define PADDR_BITS	48
 
diff -r 37fae02cc335 xen/include/asm-x86/config.h
--- a/xen/include/asm-x86/config.h	Fri Jul 25 15:03:03 2008 +0100
+++ b/xen/include/asm-x86/config.h	Mon Jul 28 16:02:40 2008 +0200
@@ -97,7 +97,7 @@
 /* Primary stack is restricted to 8kB by guard pages. */
 #define PRIMARY_STACK_SIZE 8192
 
-#define CONFIG_DMA_BITSIZE 32
+#define CONFIG_DMA_BITSIZE 30
 
 #define BOOT_TRAMPOLINE 0x8c000
 #define bootsym_phys(sym)                                 \

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel