Re: [BUGFIX] [PATCH] kexec/x86: Do map crash kernel area

[Andrew Cooper <andrew.cooper3@citrix.com>]

On 01/01/2014 16:51, Don Slutz wrote:
> Revert of commit 7113a45451a9f656deeff070e47672043ed83664
>
> Using kexec commit 027413d822fd57dd39d2d2afab1484bc6b6b84f9
>
> With "crashkernel=256M@256M" ((XEN) Kdump: 256MB (262144kB) at 0x10000000)
>
> ~/kexec/build/sbin/kexec -p '--command-line=placeholder root=/dev/mapper/vg_f17--xen-lv_root ro rd.md=0 rd.dm=0 rd.lvm.lv=vg_f17-xen/lv_swap KEYTABLE=us SYSFONT=True rd.luks=0 console=ttyS0,9600n8 rd.lvm.lv=vg_f17-xen/lv_root LANG=en_US.UTF-8 earlyprintk=ttyS0 rd_NO_PLYMOUTH irqpoll nr_cpus=1 reset_devices cgroup_disable=memory mce=off' --initrd=/boot/initramfs-3.8.11-100.fc17.x86_64kdump.img /boot/vmlinuz-3.8.11-100.fc17.x86_64
>
> Without it:
>
> (XEN) [2014-01-01 15:40:12] ----[ Xen-4.4-unstable  x86_64  debug=n  Not tainted ]----
> (XEN) [2014-01-01 15:40:12] CPU:    5
> (XEN) [2014-01-01 15:40:12] RIP:    e008:[<ffff82d08021f1c9>] clear_page_sse2+0x9/0x30
> (XEN) [2014-01-01 15:40:12] RFLAGS: 0000000000010216   CONTEXT: hypervisor
> (XEN) [2014-01-01 15:40:12] rax: 0000000000000000   rbx: ffff8300104c6000   rcx: 00000000000000ff
> (XEN) [2014-01-01 15:40:12] rdx: ffff830000000000   rsi: ffffffffffffffff   rdi: ffff8300104c6000
> (XEN) [2014-01-01 15:40:12] rbp: 0000000000000007   rsp: ffff830823fdfcf0   r8:  00000000000104c6
> (XEN) [2014-01-01 15:40:12] r9:  00000000104c7000   r10: 0000000000000000   r11: 00000000004c6000
> (XEN) [2014-01-01 15:40:12] r12: ffff83083fb1bdb0   r13: ffff83083fb1bcc0   r14: 00000000104c6000
> (XEN) [2014-01-01 15:40:12] r15: 0000000000100000   cr0: 0000000080050033   cr4: 00000000000426f0
> (XEN) [2014-01-01 15:40:12] cr3: 0000000650813000   cr2: ffff8300104c6000
> (XEN) [2014-01-01 15:40:12] ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e010   cs: e008
> (XEN) [2014-01-01 15:40:12] Xen stack trace from rsp=ffff830823fdfcf0:
> (XEN) [2014-01-01 15:40:12]    ffff82d080161dd1 ffff83083fb1bdb0 ffff82d080114d96 ffff830823fb7000
> (XEN) [2014-01-01 15:40:12]    ffff82e0002098c0 0000000000000007 ffff830823fdfde0 ffff83083fb1bcc0
> (XEN) [2014-01-01 15:40:12]    ffff83083fb1bdb0 0000000000000007 ffff830823fdfde0 ffff83083fb1bcc0
> (XEN) [2014-01-01 15:40:12]    ffff82d0801150f4 0000000000000010 ffff83083fb1bd80 00000000000000e0
> (XEN) [2014-01-01 15:40:12]    00000000fffffff2 0000000010000000 0000000020000000 ffff830823fdfde0
> (XEN) [2014-01-01 15:40:12]    000000000000003e 0000000000000003 ffff82d0801152ec 00007f0df697d004
> (XEN) [2014-01-01 15:40:12]    ffff83083fb1bcc0 00007ffffa0b1bd0 ffff880055784228 00007ffffa0b1bd0
> (XEN) [2014-01-01 15:40:12]    ffff82d0801143e1 0000000000000002 0000000000000000 ffff83083f4bebe8
> (XEN) [2014-01-01 15:40:12]    ffff82d08017c28a 0000000000000000 ffff830823fb70b0 0000000000000000
> (XEN) [2014-01-01 15:40:12]    000000000083f4be ffff82d08012a6cb ffff8300bf2f9060 ffff82d0802ea620
> (XEN) [2014-01-01 15:40:12]    ffff830823fd8000 00000007003e0001 00007f0df697e004 000000001ff53720
> (XEN) [2014-01-01 15:40:12]    0000000100000007 ffff82e0107e97c0 ffff830823fb7000 0000000000000007
> (XEN) [2014-01-01 15:40:12]    0000000000000001 ffff83083f4be000 ffff8300bf2f9000 000000000083f4be
> (XEN) [2014-01-01 15:40:12]    ffff82d080218a58 ffff830823fdff18 ffff82d080218b32 ffff830823fd8000
> (XEN) [2014-01-01 15:40:12]    0000000000000000 0000000000000217 00000032fd4ee0a7 0000000000000100
> (XEN) [2014-01-01 15:40:12]    00000032fd4ee0a7 0000000000000033 ffff8300bf2f9000 ffff880057605e88
> (XEN) [2014-01-01 15:40:12]    00007ffffa0b1bd0 ffff880055784228 ffff82d0801144ab 0000000000000000
> (XEN) [2014-01-01 15:40:12]    ffff82d08021df79 00000026d8eb3c18 00000026d8f9b240 0000000000000000
> (XEN) [2014-01-01 15:40:12]    000000280f8095dc ffff880057605e88 ffff880005331c00 0000000000000286
> (XEN) [2014-01-01 15:40:12]    00007ffffa0b1d90 ffff88000561c180 000000001ff53720 0000000000000025
> (XEN) [2014-01-01 15:40:12] Xen call trace:
> (XEN) [2014-01-01 15:40:12]    [<ffff82d08021f1c9>] clear_page_sse2+0x9/0x30
> (XEN) [2014-01-01 15:40:12]    [<ffff82d080161dd1>] clear_domain_page+0x11/0x20
> (XEN) [2014-01-01 15:40:12]    [<ffff82d080114d96>] kimage_alloc_control_page+0x246/0x2d0
> (XEN) [2014-01-01 15:40:12]    [<ffff82d0801150f4>] do_kimage_alloc+0x1c4/0x2e0
> (XEN) [2014-01-01 15:40:12]    [<ffff82d0801152ec>] kimage_alloc+0xdc/0x100
> (XEN) [2014-01-01 15:40:12]    [<ffff82d0801143e1>] do_kexec_op_internal+0x5f1/0x6b0
> (XEN) [2014-01-01 15:40:12]    [<ffff82d08017c28a>] do_mmu_update+0x34a/0x1bf0
> (XEN) [2014-01-01 15:40:12]    [<ffff82d08012a6cb>] add_entry+0x4b/0xb0
> (XEN) [2014-01-01 15:40:12]    [<ffff82d080218a58>] toggle_guest_mode+0x28/0x40
> (XEN) [2014-01-01 15:40:12]    [<ffff82d080218b32>] do_iret+0xc2/0x1a0
> (XEN) [2014-01-01 15:40:12]    [<ffff82d0801144ab>] do_kexec_op+0xb/0x20
> (XEN) [2014-01-01 15:40:12]    [<ffff82d08021df79>] syscall_enter+0xa9/0xae
> (XEN) [2014-01-01 15:40:12]
> (XEN) [2014-01-01 15:40:12] Pagetable walk from ffff8300104c6000:
> (XEN) [2014-01-01 15:40:12]  L4[0x106] = 00000000bf468063 ffffffffffffffff
> (XEN) [2014-01-01 15:40:12]  L3[0x000] = 00000000bf462063 ffffffffffffffff
> (XEN) [2014-01-01 15:40:12]  L2[0x082] = 0000000000000000 ffffffffffffffff
> (XEN) [2014-01-01 15:40:16]
> (XEN) [2014-01-01 15:40:16] ****************************************
> (XEN) [2014-01-01 15:40:16] Panic on CPU 5:
> (XEN) [2014-01-01 15:40:16] FATAL PAGE FAULT
> (XEN) [2014-01-01 15:40:16] [error_code=0002]
> (XEN) [2014-01-01 15:40:16] Faulting linear address: ffff8300104c6000
> (XEN) [2014-01-01 15:40:17] ****************************************
> (XEN) [2014-01-01 15:40:17]
> (XEN) [2014-01-01 15:40:17] Reboot in five seconds...
>
> With this patch no panic and crash kernel works.
>
> Signed-off-by: Don Slutz <dslutz@verizon.com>

Commit 7113a45451a9f656deeff070e47672043ed83664 was clearly not tested. 
kimage_alloc_crash_control_page() explicitly chooses a page inside the
crash region and clears it.

However, the sentiment of the commit is certainly desirable, to prevent
accidental playing in the crash region.

As the mappings are removed from Xen's directmap region,
map_domain_page() doesn't work (unless the debug highmem barrier is
sufficiently low that the crash regions ends up above it, and the
virtual address ends up coming from the mapcache).

This means that both here in clear_domain_page(), and later in
machine_kexec_load() where the code is copied in, is vulnerable to this
pagefault.

The solution to this problem which would leave the fewest mappings would
be to have kimage_alloc_crash_control_page() map the individual control
page to the main Xen pagetables, at which a call to point
map_domain_page() on it will work correctly.  This would need an
equivalent call to destroy_xen_mappings() in kimage_free().

However, it is far from neat.

I defer to others as to which approach is better, but suggest that one
way or another, the problem gets fixed very quickly, even if that means
taking this complete reversion now and submitting a proper fix in due
course.

~Andrew

> ---
>  xen/arch/x86/setup.c | 4 ++++
>  1 file changed, 4 insertions(+)
>
> diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
> index 4833ca3..90f3294 100644
> --- a/xen/arch/x86/setup.c
> +++ b/xen/arch/x86/setup.c
> @@ -1098,6 +1098,10 @@ void __init __start_xen(unsigned long mbi_p)
>                           PFN_UP(mod[i].mod_end), PAGE_HYPERVISOR);
>      }
>  
> +    map_pages_to_xen((unsigned long)__va(kexec_crash_area.start),
> +                     kexec_crash_area.start >> PAGE_SHIFT,
> +                     PFN_UP(kexec_crash_area.size), PAGE_HYPERVISOR);
> +
>      xen_virt_end = ((unsigned long)_end + (1UL << L2_PAGETABLE_SHIFT) - 1) &
>                     ~((1UL << L2_PAGETABLE_SHIFT) - 1);
>      destroy_xen_mappings(xen_virt_end, XEN_VIRT_START + BOOTSTRAP_MAP_BASE);