[PATCH]: Allow Xen to boot/run on large memory (>64G) machines

[PATCH]: Allow Xen to boot/run on large memory (>64G) machines

[Chris Lalancette]

[-- Attachment #1: Type: text/plain, Size: 1685 bytes --]

All,
     I've been tracking down a problem where dom0 refuses to boot on very large
memory x86_64 machines.  Here's what happens:

The hypervisor starts up with 1GB in the DMA zone.  Two large allocations come
out of the DMA zone; the frame table (in init_frametable()), and the memory for
dom0 (in construct_dom0()).  With a lot of memory in the box, most of the DMA
zone gets allocated during init_frametable; so much so, in fact, that there is
no room to make the allocation in construct_dom0, and the dom0 fails to boot with:

(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) Not enough RAM for domain 0 allocation.
(XEN) ****************************************
(XEN)
(XEN) Reboot in five seconds...

The solution (suggested by Keir), is to make the frametable allocated out of
high memory instead of the DMA zone.  The attached patch (against 3.0.3, but the
problem is the same in unstable), does this.  I tested this out on a 96GB
machine; without the patch, the machine would reboot as described above; with
the patch, I was able to boot dom0 and create a PV guest with 92GB of memory.
     I only compile tested this on ia64, but I don't see anything in it that
should cause problems there.
     Note that this is not the end of the story, however.  For even larger
machines, it can *still* be the case that the allocation in construct_dom0()
fails; in particular, if the order goes above 17, it will fail in the same way.
 One way to fix it would be to just allocate that memory out of the normal zone
for x86_64, as well; however, I'm not sure if this will break anything else.
Any comments?

Signed-off-by: Chris Lalancette <clalance@redhat.com>

[-- Attachment #2: xen-hugemem-3.patch --]
[-- Type: text/x-patch, Size: 2952 bytes --]

diff -urp xen.orig/arch/x86/mm.c xen/arch/x86/mm.c
--- xen.orig/arch/x86/mm.c	2007-02-21 14:45:38.000000000 -0500
+++ xen/arch/x86/mm.c	2007-02-21 16:11:34.000000000 -0500
@@ -179,7 +179,15 @@ void __init init_frametable(void)
 
     for ( i = 0; i < nr_pages; i += page_step )
     {
+#ifdef __x86_64__
+        /* for x86_64 we want to allocate the frame table from the top
+         * of memory rather than the bottom; otherwise, on large memory
+         * machines (> 64G), we exhaust DMA memory, and dom0 cannot boot
+         */
+        mfn = alloc_boot_pages_reverse(min(nr_pages - i, page_step), page_step);
+#else
         mfn = alloc_boot_pages(min(nr_pages - i, page_step), page_step);
+#endif
         if ( mfn == 0 )
             panic("Not enough memory for frame table\n");
         map_pages_to_xen(
diff -urp xen.orig/common/page_alloc.c xen/common/page_alloc.c
--- xen.orig/common/page_alloc.c	2006-10-16 16:07:17.000000000 -0400
+++ xen/common/page_alloc.c	2007-02-21 16:09:38.000000000 -0500
@@ -213,26 +213,44 @@ void init_boot_pages(paddr_t ps, paddr_t
     }
 }
 
-unsigned long alloc_boot_pages(unsigned long nr_pfns, unsigned long pfn_align)
+static unsigned long check_and_map_page(unsigned long pg, unsigned long nr_pfns)
 {
-    unsigned long pg, i;
+    unsigned long i;
 
-    for ( pg = 0; (pg + nr_pfns) < max_page; pg += pfn_align )
+    for ( i = 0; i < nr_pfns; i++ )
+        if ( allocated_in_map(pg + i) )
+             break;
+
+    if ( i == nr_pfns )
     {
-        for ( i = 0; i < nr_pfns; i++ )
-            if ( allocated_in_map(pg + i) )
-                 break;
+        map_alloc(pg, nr_pfns);
+        return pg;
+    }
 
-        if ( i == nr_pfns )
-        {
-            map_alloc(pg, nr_pfns);
+    return 0;
+}
+
+unsigned long alloc_boot_pages(unsigned long nr_pfns, unsigned long pfn_align)
+{
+    unsigned long pg;
+
+    for ( pg = 0; (pg + nr_pfns) < max_page; pg += pfn_align )
+        if (check_and_map_page(pg, nr_pfns))
             return pg;
-        }
-    }
 
     return 0;
 }
 
+unsigned long alloc_boot_pages_reverse(unsigned long nr_pfns, unsigned long pfn_align)
+{
+    unsigned long pg;
+
+    for ( pg = (max_page - nr_pfns); pg > 0; pg -= pfn_align )
+        if (check_and_map_page(pg, nr_pfns))
+            return pg;
+
+    return 0;
+}
 
 
 /*************************
diff -urp xen.orig/include/xen/mm.h xen/include/xen/mm.h
--- xen.orig/include/xen/mm.h	2006-10-16 16:07:18.000000000 -0400
+++ xen/include/xen/mm.h	2007-02-21 15:57:46.000000000 -0500
@@ -39,6 +39,7 @@ struct page_info;
 /* Boot-time allocator. Turns into generic allocator after bootstrap. */
 paddr_t init_boot_allocator(paddr_t bitmap_start);
 void init_boot_pages(paddr_t ps, paddr_t pe);
+unsigned long alloc_boot_pages_reverse(unsigned long nr_pfns, unsigned long pfn_align);
 unsigned long alloc_boot_pages(unsigned long nr_pfns, unsigned long pfn_align);
 void end_boot_allocator(void);
 

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: [PATCH]: Allow Xen to boot/run on large memory (>64G) machines

[Keir Fraser]

On 22/2/07 00:38, "Chris Lalancette" <clalance@redhat.com> wrote:

>      Note that this is not the end of the story, however.  For even larger
> machines, it can *still* be the case that the allocation in construct_dom0()
> fails; in particular, if the order goes above 17, it will fail in the same
> way.
>  One way to fix it would be to just allocate that memory out of the normal
> zone
> for x86_64, as well; however, I'm not sure if this will break anything else.
> Any comments?

If there are no users of alloc_boot_pages() expecting low memory to be
returned then we can adjust the implementation of that existing function
rather than introduce a new one.

As for domain_build() there are two considerations: firstly that the
allocation is contiguous and secondly that it is from the DMA pool. The
builder makes simplifying assumptions based on contiguity. The allocation
from DMA pool I think I've tried to get rid of before -- I think I was
scuppered by something as simple as the PAE pgdir needing to be allocated
from low memory. I think we can stop allocating from the DMA pool, at least
for non-PAE host.

 -- Keir

Re: [PATCH]: Allow Xen to boot/run on large memory (>64G) machines

[Jan Beulich]

>>> Keir Fraser <Keir.Fraser@cl.cam.ac.uk> 22.02.07 08:50 >>>
>On 22/2/07 00:38, "Chris Lalancette" <clalance@redhat.com> wrote:
>
>>      Note that this is not the end of the story, however.  For even larger
>> machines, it can *still* be the case that the allocation in construct_dom0()
>> fails; in particular, if the order goes above 17, it will fail in the same
>> way.
>>  One way to fix it would be to just allocate that memory out of the normal
>> zone
>> for x86_64, as well; however, I'm not sure if this will break anything else.
>> Any comments?
>
>If there are no users of alloc_boot_pages() expecting low memory to be
>returned then we can adjust the implementation of that existing function
>rather than introduce a new one.
>
>As for domain_build() there are two considerations: firstly that the
>allocation is contiguous and secondly that it is from the DMA pool. The
>builder makes simplifying assumptions based on contiguity. The allocation
>from DMA pool I think I've tried to get rid of before -- I think I was
>scuppered by something as simple as the PAE pgdir needing to be allocated
>from low memory. I think we can stop allocating from the DMA pool, at least
>for non-PAE host.

The page allocator changes that I posted a while back probably haven't
been looked at so fat, given the above comments. The patch that kills the
DMA pool changes x86-64 to allocate the dom0 memory without restriction
(i386 has to remain restricted, yet not because of DMA address issues, but
in order to be able to see the memory in the 1:1 mapping).

Likewise, I'm not certain the changes presented here make a lot of sense
in the context of the elimination of the DMA pool and the resulting desire
to unify xen heap and domain heap on x86-64.

Jan

Jan

Re: [PATCH]: Allow Xen to boot/run on large memory (>64G) machines

[Keir Fraser]

On 22/2/07 10:33, "Jan Beulich" <jbeulich@novell.com> wrote:

> The page allocator changes that I posted a while back probably haven't
> been looked at so fat, given the above comments. The patch that kills the
> DMA pool changes x86-64 to allocate the dom0 memory without restriction
> (i386 has to remain restricted, yet not because of DMA address issues, but
> in order to be able to see the memory in the 1:1 mapping).

Oh, good point. :-) And it sounds like it makes sense to leave this alone
for now and leave it to your patches.

> Likewise, I'm not certain the changes presented here make a lot of sense
> in the context of the elimination of the DMA pool and the resulting desire
> to unify xen heap and domain heap on x86-64.

It makes sense for the boot allocator to prefer to allocate from high memory
if it can, rather then using what is currently the DMA pool (and, after your
patches are applied, will be from relatively-narrow-address-width pools). So
I think this patch is good and narrow enough in scope to go straight in
(although I think the behaviour of alloc_boot_pages() should be changed
rather than adding a new allocator function).

 -- Keir

Re: [PATCH]: Allow Xen to boot/run on large memory (>64G) machines

[Chris Lalancette]

Keir Fraser wrote:
> On 22/2/07 10:33, "Jan Beulich" <jbeulich@novell.com> wrote:
> 
> 
>>The page allocator changes that I posted a while back probably haven't
>>been looked at so fat, given the above comments. The patch that kills the
>>DMA pool changes x86-64 to allocate the dom0 memory without restriction
>>(i386 has to remain restricted, yet not because of DMA address issues, but
>>in order to be able to see the memory in the 1:1 mapping).
> 
> 
> Oh, good point. :-) And it sounds like it makes sense to leave this alone
> for now and leave it to your patches.
> 

Ah, OK.  That will address the second concern.  Jan's right, I didn't actually
look at those patches.  Thanks for pointing them out.

> 
> It makes sense for the boot allocator to prefer to allocate from high memory
> if it can, rather then using what is currently the DMA pool (and, after your
> patches are applied, will be from relatively-narrow-address-width pools). So
> I think this patch is good and narrow enough in scope to go straight in
> (although I think the behaviour of alloc_boot_pages() should be changed
> rather than adding a new allocator function).

Yeah, I wasn't quite sure how far to go with this.  The frame table was the
worst offender, so I just went after that.  I can whip up a quick patch and test
it out here, changing the alloc_boot_pages() to always allocate from the top.

By the way, I assume we only want to do this for x86_64, yes?

Chris Lalancette

Re: [PATCH]: Allow Xen to boot/run on large memory (>64G) machines

[Keir Fraser]

On 22/2/07 14:57, "Chris Lalancette" <clalance@redhat.com> wrote:

>> It makes sense for the boot allocator to prefer to allocate from high memory
>> if it can, rather then using what is currently the DMA pool (and, after your
>> patches are applied, will be from relatively-narrow-address-width pools). So
>> I think this patch is good and narrow enough in scope to go straight in
>> (although I think the behaviour of alloc_boot_pages() should be changed
>> rather than adding a new allocator function).
> 
> Yeah, I wasn't quite sure how far to go with this.  The frame table was the
> worst offender, so I just went after that.  I can whip up a quick patch and
> test
> it out here, changing the alloc_boot_pages() to always allocate from the top.

Turns out there is one place that wants to allocate from the bottom (the
kdump path in arch/x86/setup.c). So I renamed alloc_boot_pages() to
alloc_lowmem_boot_pages() and called the new function alloc_boot_pages(). If
you care you use the more specific (lowmem) one. I'm about to check in the
revised patch.

> By the way, I assume we only want to do this for x86_64, yes?

No, it makes sense to conserve 'narrower' addresses.

 -- Keir

Re: [PATCH]: Allow Xen to boot/run on large memory (>64G) machines

[Jan Beulich]

>Likewise, I'm not certain the changes presented here make a lot of sense
>in the context of the elimination of the DMA pool and the resulting desire
>to unify xen heap and domain heap on x86-64.

Sorry, I wrote this based on the description, without having looked at the
patch. The changes certainly make sense (including doing this perhaps also
for 32-bits). You didn't touch the dom0 allocation at all...

Jan