Re: [PATCH] Boot PV guests with more than 128GB (v2) for 3.7

[Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>]

On Tue, Jul 31, 2012 at 10:43:18AM -0400, Konrad Rzeszutek Wilk wrote:
> Changelog:
> Since v1: [http://lists.xen.org/archives/html/xen-devel/2012-07/msg01561.html]
>  - added more comments, and #ifdefs
>  - squashed The L4 and L4, L3, and L2 recycle patches together
>  - Added Acked-by's
> 
> The explanation of these patches is exactly what v1 had:
> 
> The details of this problem are nicely explained in:
> 
>  [PATCH 4/6] xen/p2m: Add logic to revector a P2M tree to use __va
>  [PATCH 5/6] xen/mmu: Copy and revector the P2M tree.
>  [PATCH 6/6] xen/mmu: Remove from __ka space PMD entries for
> 
> and the supporting patches are just nice optimizations. Pasting in
> what those patches mentioned:

With these patches I've gotten it to boot up to 384GB. Around that area
something weird happens - mainly the pagetables that the toolstack allocated
seems to have missing data. I hadn't looked in details, but this is what
domain builder tells me:


xc_dom_alloc_segment:   ramdisk      : 0xffffffff82278000 -> 0xffffffff930b4000  (pfn 0x2278 + 0x10e3c pages)
xc_dom_malloc            : 1621 kB
xc_dom_pfn_to_ptr: domU mapping: pfn 0x2278+0x10e3c at 0x7fb0853a2000
xc_dom_do_gunzip: unzip ok, 0x4ba831c -> 0x10e3be10
xc_dom_alloc_segment:   phys2mach    : 0xffffffff930b4000 -> 0xffffffffc30b4000  (pfn 0x130b4 + 0x30000 pages)
xc_dom_malloc            : 4608 kB
xc_dom_pfn_to_ptr: domU mapping: pfn 0x130b4+0x30000 at 0x7fb0553a2000
xc_dom_alloc_page   :   start info   : 0xffffffffc30b4000 (pfn 0x430b4)
xc_dom_alloc_page   :   xenstore     : 0xffffffffc30b5000 (pfn 0x430b5)
xc_dom_alloc_page   :   console      : 0xffffffffc30b6000 (pfn 0x430b6)
nr_page_tables: 0x0000ffffffffffff/48: 0xffff000000000000 -> 0xffffffffffffffff, 1 table(s)
nr_page_tables: 0x0000007fffffffff/39: 0xffffff8000000000 -> 0xffffffffffffffff, 1 table(s)
nr_page_tables: 0x000000003fffffff/30: 0xffffffff80000000 -> 0xffffffffffffffff, 2 table(s)
nr_page_tables: 0x00000000001fffff/21: 0xffffffff80000000 -> 0xffffffffc33fffff, 538 table(s)
xc_dom_alloc_segment:   page tables  : 0xffffffffc30b7000 -> 0xffffffffc32d5000  (pfn 0x430b7 + 0x21e pages)
xc_dom_pfn_to_ptr: domU mapping: pfn 0x430b7+0x21e at 0x7fb055184000
xc_dom_alloc_page   :   boot stack   : 0xffffffffc32d5000 (pfn 0x432d5)
xc_dom_build_image  : virt_alloc_end : 0xffffffffc32d6000
xc_dom_build_image  : virt_pgtab_end : 0xffffffffc3400000

Note it is is 0xffffffffc30b4000 - so already past the level2_kernel_pgt (L3[510]
and in level2_fixmap_pgt territory (L3[511]).

Hypervisor tells me:

(XEN) Pagetable walk from ffffffffc32d5ff8:
(XEN)  L4[0x1ff] = 000000b9804d9067 00000000000430b8
(XEN)  L3[0x1ff] = 0000000000000000 ffffffffffffffff
(XEN) domain_crash_sync called from entry.S
(XEN) Domain 13 (vcpu#0) crashed on cpu#121:
(XEN) ----[ Xen-4.1.2-OVM  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    121
(XEN) RIP:    e033:[<ffffffff818a4200>]
(XEN) RFLAGS: 0000000000010202   EM: 1   CONTEXT: pv guest
(XEN) rax: 0000000000000000   rbx: 0000000000000000   rcx: 0000000000000000
(XEN) rdx: 0000000000000000   rsi: ffffffffc30b4000   rdi: 0000000000000000
(XEN) rbp: 0000000000000000   rsp: ffffffffc32d6000   r8:  0000000000000000
(XEN) r9:  0000000000000000   r10: 0000000000000000   r11: 0000000000000000
(XEN) r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
(XEN) r15: 0000000000000000   cr0: 000000008005003b   cr4: 00000000000026f0
(XEN) cr3: 000000b9804da000   cr2: ffffffffc32d5ff8
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e02b   cs: e033
(XEN) Guest stack trace from rsp=ffffffffc32d6000:
(XEN)   Fault while accessing guest memory.

And that EIP translates to ffffffff818a4200 T startup_xen
which does:

ENTRY(startup_xen)
        cld      
ffffffff818a4200:       fc                      cld      
#ifdef CONFIG_X86_32
        mov %esi,xen_start_info
        mov $init_thread_union+THREAD_SIZE,%esp
#else
        mov %rsi,xen_start_info
ffffffff818a4201:       48 89 34 25 48 92 94    mov    %rsi,0xffffffff81949248
ffffffff818a4208:       81       


At that stage we are still operating using the Xen provided pagetable - which
look to have the L4[511][511] empty! Which sounds to me like a Xen tool-stack
problem? Jan, have you seen something similar to this?