From mboxrd@z Thu Jan 1 00:00:00 1970 From: Andre Przywara Subject: Re: Memory allocation in NUMA system Date: Mon, 28 Jul 2008 16:26:30 +0200 Message-ID: <488DD716.7050003@amd.com> References: Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="------------040901050101020602060605" Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Keir Fraser Cc: xen-devel@lists.xensource.com, "Yang, Xiaowei" List-Id: xen-devel@lists.xenproject.org --------------040901050101020602060605 Content-Type: text/plain; charset="ISO-8859-1"; format=flowed Content-Transfer-Encoding: 7bit Keir Fraser wrote: > On 28/7/08 13:21, "Andre Przywara" wrote: > >> Mmh, why not check this in in 3.3? I have noticed this problem already a >> year ago and was having some other kind of fix for it (which actually >> prefered nodes over zones): >> http://lists.xensource.com/archives/html/xen-devel/2007-12/msg00831.html >> I think this is a somewhat serious issue on NUMA machines, since with >> the automatic pinning now active (new in 3.3!) many domains will end up >> with remote memory _all the time_. So I think of this as a bugfix. >> Actually I have dma_bitsize=30 hardwired in my Grub's menu.lst for some >> months now... > > Well, fine, but unfortunately the patch breaks ia64 Fixed. > and doesn't even work properly: > - why should NUMA node 0 be the one that overlaps with default DMA memory? Because that is the most common configuration? Do you know of any machine where this is not true? I agree that a dual node machine with 2 gig on each node does not need this patch, but NUMA machines tend to have more memory than this (especially given the current memory costs). I changed the default DMA_BITSIZE to 30 bits, this seems to be a reasonable value. > - a 'large' NUMA node 0 will cause dma_bitsize to be set much larger than > it is currently, thus breaking its original intent. Fixed in the attached patch. It now caps dma_bitsize to at most 1/4 of node0 memory. What about using this patch for Xen 3.3 and work out a more general solution for Xen 3.4? Signed off by: Andre Przywara Based on the patch from: "Yang, Xiaowei" -- Andre Przywara AMD-Operating System Research Center (OSRC), Dresden, Germany Tel: +49 351 277-84917 ----to satisfy European Law for business letters: AMD Saxony Limited Liability Company & Co. KG, Wilschdorfer Landstr. 101, 01109 Dresden, Germany Register Court Dresden: HRA 4896, General Partner authorized to represent: AMD Saxony LLC (Wilmington, Delaware, US) General Manager of AMD Saxony LLC: Dr. Hans-R. Deppe, Thomas McCoy --------------040901050101020602060605 Content-Type: text/plain; name="dma_bitsize.patch" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="dma_bitsize.patch" diff -r 37fae02cc335 xen/common/page_alloc.c --- a/xen/common/page_alloc.c Fri Jul 25 15:03:03 2008 +0100 +++ b/xen/common/page_alloc.c Mon Jul 28 16:02:40 2008 +0200 @@ -583,6 +583,13 @@ init_heap_pages(pfn_dom_zone_type(i), mfn_to_page(i), 1); } + /* Reserve only up to 25% of node0's memory for DMA */ + i = pfn_dom_zone_type(NODE_DATA(0)->node_spanned_pages / 4) + + PAGE_SHIFT; + if ( i < dma_bitsize ) dma_bitsize = i; + + ASSERT(dma_bitsize > PAGE_SHIFT + 1); + printk("Domain heap initialised: DMA width %u bits\n", dma_bitsize); } #undef avail_for_domheap diff -r 37fae02cc335 xen/include/asm-ia64/config.h --- a/xen/include/asm-ia64/config.h Fri Jul 25 15:03:03 2008 +0100 +++ b/xen/include/asm-ia64/config.h Mon Jul 28 16:02:40 2008 +0200 @@ -44,7 +44,7 @@ #define CONFIG_IOSAPIC #define supervisor_mode_kernel (0) -#define CONFIG_DMA_BITSIZE 32 +#define CONFIG_DMA_BITSIZE 30 #define PADDR_BITS 48 diff -r 37fae02cc335 xen/include/asm-x86/config.h --- a/xen/include/asm-x86/config.h Fri Jul 25 15:03:03 2008 +0100 +++ b/xen/include/asm-x86/config.h Mon Jul 28 16:02:40 2008 +0200 @@ -97,7 +97,7 @@ /* Primary stack is restricted to 8kB by guard pages. */ #define PRIMARY_STACK_SIZE 8192 -#define CONFIG_DMA_BITSIZE 32 +#define CONFIG_DMA_BITSIZE 30 #define BOOT_TRAMPOLINE 0x8c000 #define bootsym_phys(sym) \ --------------040901050101020602060605 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel --------------040901050101020602060605--