Re: [PATCH] x86/NUMA: make init_node_heap() respect Xen heap limit

[Julien Grall <julien.grall@citrix.com>]

Hi Jan,

On 27/08/2015 09:37, Jan Beulich wrote:
> On NUMA systems, where we try to use node local memory for the basic
> control structures of the buddy allocator, this special case needs to
> take into consideration a possible address width limit placed on the
> Xen heap. In turn this (but also other, more abstract considerations)
> requires that xenheap_max_mfn() not be called more than once (at most
> we might permit it to be called a second time with a larger value than
> was passed the first time), and be called only before calling
> end_boot_allocator().
> 
> While inspecting all the involved code, a couple of off-by-one issues
> were found (and are being corrected here at once):
> - arch_init_memory() cleared one too many page table slots
> - the highmem_start based invocation of xenheap_max_mfn() passed too
>    big a value
> - xenheap_max_mfn() calculated the wrong bit count in edge cases
> 
> Signed-off-by: Jan Beulich <jbeulich@suse.com>

This patch is breaking boot on aarch64 platform (particularly X-gene).

I think this should be considered as a block until I find a way to
fix it.

I've noticed it while rebasing my branch on xengit/staging, although
given that there is no aarch64 hardware on osstest this issue won't
be reported.

I gave a try to boot on arm32 and didn't see any failure.

I will do more debug tonight and tomorrow morning to see what's going
on. Stack trace below:

- UART enabled -
- CPU 00000000 booting -
- Current EL 00000008 -
- Xen starting at EL2 -
- Zero BSS -
- Setting up control registers -
- Turning on paging -
- Ready -
(XEN) Checking for initrd in /chosen
(XEN) RAM: 0000004000000000 - 00000043ffffffff
(XEN)
(XEN) MODULE[0]: 0000004000ff9000 - 0000004001000000 Device Tree
(XEN) MODULE[1]: 0000004002000000 - 0000004002a4aa00 Kernel       console=hvc0 root=/dev/sda2 rw earlycon=uart8250,mmio32,0x1c020000 ignore_loglevel
(XEN)  RESVD[0]: 0000004003000000 - 0000004003004000
(XEN)  RESVD[1]: 0000004000000000 - 0000004000010000
(XEN)
(XEN) Command line: conswitch=x console=dtuart dtuart=/soc/serial@1c020000 no-bootscrub noreboot dom0_mem=4G
(XEN) Placing Xen at 0x00000043ffe00000-0x0000004400000000
(XEN) Update BOOTMOD_XEN from 0000004004000000-000000400410ad81 => 00000043ffe00000-00000043fff0ad81
(XEN) Xen heap: 38 bits
(XEN) Domain heap initialised
(XEN) create_xen_entries: L2 failed
(XEN) Hypervisor Trap. HSR=0x96000046 EC=0x25 IL=1 Syndrome=0x46
(XEN) CPU0: Unexpected Trap: Hypervisor
(XEN) ----[ Xen-4.6.0-rc  arm64  debug=y  Not tainted ]----
(XEN) CPU:    0
(XEN) PC:     000000000025a638 clear_page+0x10/0x24
(XEN) LR:     000000000027eb28
(XEN) SP:     00000000002b7db0
(XEN) CPSR:   800003c9 MODE:64-bit EL2h (Hypervisor, handler)
(XEN)      X0: 0000000040000000  X1: 0000000000000040  X2: 0000000000000004
(XEN)      X3: 0000000000000000  X4: 0000000000003fff  X5: 00000000002982e8
(XEN)      X6: 0000000000000004  X7: 00000000002c4a30  X8: 00000000002c4a28
(XEN)      X9: 0000000800037e40 X10: 00000000002c1950 X11: 0000000000000200
(XEN)     X12: 0000000000000000 X13: 0000000000000000 X14: 0000000000000000
(XEN)     X15: 0000000000400000 X16: 000000000028c734 X17: 0000000000000006
(XEN)     X18: 00000000000084d0 X19: 0000000040000000 X20: 0000000000000001
(XEN)     X21: 000000000027b400 X22: 0000000040001000 X23: 000000000027b000
(XEN)     X24: 000000000027b000 X25: 000000000027b000 X26: 0000000000006db7
(XEN)     X27: fffffff800000000 X28: 0000000040000000  FP: 00000000002b7db0
(XEN)
(XEN)   VTCR_EL2: 80000000
(XEN)  VTTBR_EL2: 0000000000000000
(XEN)
(XEN)  SCTLR_EL2: 30cd183d
(XEN)    HCR_EL2: 000000000030643f
(XEN)  TTBR0_EL2: 00000043ffef4000
(XEN)
(XEN)    ESR_EL2: 96000046
(XEN)  HPFAR_EL2: 0000000000000000
(XEN)    FAR_EL2: 0000000040000000
(XEN)
(XEN) Xen stack trace from sp=00000000002b7db0:
(XEN)    00000000002b7e20 0000000000287d6c 00008003ffff9000 0000000004400000
(XEN)    0000004400000000 00000000002a0000 0000000000000000 000000000027b000
(XEN)    00000000002a0048 0000004000000000 0000004400000000 0000000000000001
(XEN)    0000000000000000 0000000000000000 00000043efb181f0 000000000020060c
(XEN)    0000004004000000 0000004003e00000 0000004000ff9000 0000000000000000
(XEN)    0000000000400000 0000000000000000 0000000000000001 0000000000000000
(XEN)    0000000000000000 00000043efb8eff0 0000004000ff9000 0000000000007000
(XEN)    0000000400000000 000000000027d5e8 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000
(XEN) Xen call trace:
(XEN)    [<000000000025a638>] clear_page+0x10/0x24 (PC)
(XEN)    [<000000000027eb28>] vm_init+0x10c/0x1d0 (LR)
(XEN)    [<0000000000287d6c>] start_xen+0x54c/0xc94
(XEN)    [<000000000020060c>] paging+0x84/0xbc
(XEN)
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) CPU0: Unexpected Trap: Hypervisor
(XEN)
(XEN) ****************************************
(XEN)
(XEN) Manual reset required ('noreboot' specified)

Regards,

-- 
Julien Grall