Re: bring up Hypervisor on large (512GB) memory

["Jan Beulich" <jbeulich@novell.com>]

Minus the regression Keir talks about, we have indications that Xen (even
3.3 iirc) itself boots fine even with 1Tb. The issues come with Dom0
booting: I recently submitted a patch to add infrastructure in Xen to allow
Dom0 to get a valid initial memory map, but the kernel side changes we
have could not easily be applied to the 2.6.18 tree, so that kernel will
continue to not be suitable for this large an amount of memory. Nor can I
at this point really tell whether those changes were all that's needed to get
things going, as the testing get stalled by the testers losing access to the
single machine that had this much memory.

For SLES10 (3.2 based), where we backported the necessary hypervisor
changes, we have indications that there might be other issues (as the
testers reported even booting with mem=400G failed), but we haven't
got hard data (read: back traces), and I haven't been able to spot
further issues by static code inspection.

Jan

>>> Mukesh Rathor <mukesh.rathor@oracle.com> 10.02.09 04:08 >>>
Hi all,

Trying to bring up the hypervisor on 512GB :

1. Started with xen 3.1.4 (last oracle release), and 512GB, the system
panic'd right away:
    (XEN) Early fatal page fault at e008:ffff828c801ce0ad
    (cr2=ffff8300cfc00000, ec=0002)

I noticed an anomaly here in the RAM map:
(XEN) Xen-e820 RAM map:
(XEN)  0000000000000000 - 000000000009f400 (usable)
(XEN)  000000000009f400 - 00000000000a0000 (reserved)
(XEN)  00000000000f0000 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 00000000cfd4c000 (usable)
(XEN)  00000000cfd4c000 - 00000000cfd56000 (ACPI data)
(XEN)  00000000cfd56000 - 00000000cfd57000 (usable)
(XEN)  00000000cfd57000 - 00000000e0000000 (reserved)
(XEN)  00000000fec00000 - 00000000fed00000 (reserved)
(XEN)  00000000fee00000 - 00000000fee10000 (reserved)
(XEN)  00000000ffc00000 - 0000000100000000 (reserved)
(XEN)  0000000100000000 - 000000802ffff000 (usable)  <-------

Seems that should be 0000008000000000 ???


2. Reduce some RAM, and booting with 440GB, map makes more sense:

(XEN) Xen-e820 RAM map:
(XEN)  0000000000000000 - 000000000009f400 (usable)
(XEN)  000000000009f400 - 00000000000a0000 (reserved)
(XEN)  00000000000f0000 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 00000000cfd4c000 (usable)
(XEN)  00000000cfd4c000 - 00000000cfd56000 (ACPI data)
(XEN)  00000000cfd56000 - 00000000cfd57000 (usable)
(XEN)  00000000cfd57000 - 00000000e0000000 (reserved)
(XEN)  00000000fec00000 - 00000000fed00000 (reserved)
(XEN)  00000000fee00000 - 00000000fee10000 (reserved)
(XEN)  00000000ffc00000 - 0000000100000000 (reserved)
(XEN)  0000000100000000 - 0000006e00000000 (usable)

Panic again:
(XEN) Early fatal page fault at e008:ffff828c80210460
(cr2=ffff8300cfc00000, ec=0002)


The panic is trying to allocate bitmap in init_boot_allocator(). The bitmap
starts at cfac0000 and given the size dc1000, won't fit.


3. Moving to unstable 19164, looks like things are even more tighter!! I
    couldn't even boot with 64GB. bitmap_start:cfac0000, bitmap_size:201000,
    alloc_bitmap:ffff8300cfac0000


The only solution I can think of is moving the bitmap elsewhere, above 4GB in
this case:
    figure the size of bitmap, DIRECT map space, allocate the map,
    mark it reserved in the RAM map, and should work!

    I'd have add a loop around init_boot_allocator() in __start_xen()
    iterating thru the RAM map again, and finding space above 16M.


Am I on the right track?


Thanks,
Mukesh


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com 
http://lists.xensource.com/xen-devel