Re: x86/AMD: Nested hvm crashes in 4.3

[Suravee Suthikulanit <suravee.suthikulpanit@amd.com>]

On 6/27/2013 6:14 AM, Suravee Suthikulpanit wrote:
> On 6/27/2013 5:33 AM, Egger, Christoph wrote:
>> On 27.06.13 12:24, Suravee Suthikulpanit wrote:
>>> On 6/27/2013 5:08 AM, Jan Beulich wrote:
>>>>>>> On 27.06.13 at 11:20, Suravee Suthikulpanit
>>>>>>> <suravee.suthikulpanit@amd.com> wrote:
>>>>> On 6/27/2013 3:22 AM, Jan Beulich wrote:
>>>>>>>>> On 27.06.13 at 02:24, Suravee Suthikulanit
>>>>>>>>> <suravee.suthikulpanit@amd.com> wrote:
>>>>>>> I have found an issue in where the system crash right when I start
>>>>>>> another HVM guest inside an HVM guest.  I have traced back to the
>>>>>>> patch
>>>>>>> which the issue started.
>>>>>>>
>>>>>>> commit f1bde87fc08ce8c818a1640a8fe4765d48923091
>>>>>>> Author: Jan Beulich <jbeulich@suse.com>
>>>>>>> Date:   Fri Feb 8 11:06:04 2013 +0100
>>>>>>>
>>>>>>>         x86: debugging code for testing 16Tb support on smaller
>>>>>>> memory systems
>>>>>>>
>>>>>>>         Signed-off-by: Jan Beulich <jbeulich@suse.com>
>>>>>>>         Acked-by: Keir Fraser <keir@xen.org>
>>>>>> We had issues exposed by this patch before, but any such issue
>>>>>> would just have been masked before that patch (and would
>>>>>> surface on a system with more than 5Tb of memory anyway).
>>>>> The system I am having the issue has 48GB of memory.
>>>> Which is why you're seeing the problem only with the debugging
>>>> code enabled.
>>> Is the "debugging" enabled by default?  I didn't specify any debug when
>>> building.
>> "Debugging" is enabled by default in the development tree.
>>
>>> How can I check and disable debugging?
>> In the toplevel source directory look into Config.mk
>> and set the line
>>
>>     debug ?= y
>>
>> accordingly.
>
> Thank you for clarification.
>
>>>> (And of course I didn't really expect you to have
>>>> tried this on a huge memory system - they're just too rare still
>>>> for this to be likely.)
>>>>
>>>>>> So it is very unlikely for the patch itself to be at fault.
>>>>> I have traced the issue and found that the system crashing starts
>>>>> from this
>>>>> commit id and onward.
>>>>> (i.e. The system does not crash with commit id
>>>>> ed759d20249197cf87b338ff0ed328052ca3b8e7)
>>>>> So, I am still believe that this patch has somehow triggered the 
>>>>> issue.
>>>> As said - I'm pretty certain this merely unmasked an already
>>>> lurking issue.
>>> I'm not quite sure what you meant here.  Are you saying that this
>>> "crashing" is a known issue?
>> He means nestedhvm reveals an existing bug in his patch.
>> If he is right then you do not see nestedhvm crashing with a non-debug
>> xen-kernel (unless something else broke it).
>
> After I rebuilt Xen kernel with debug=n, the system no longer crash 
> when starting npt-on-npt and shadown-on-npt guests.
> I was not able to get to the crash dump previously. I will try again 
> tomorrow at work and will post them.
>
> Thank you,
>
> Suravee

So, I have finally able to get the crash dump (see below). The crash is due to an assert

     (XEN) Assertion 'va >= XEN_VIRT_START' failed at /sandbox/xen/xen.git/xen/include/asm/x86_64/page.h:86

* Debugging show the va=ffff82c40002d000, XEN_VIRT_START=ffff82c4c0000000, DIRECTMAP_VIRT_END=ffffff8000000000.
* Backtrace symbol showing the crash is in "svm_vmexit_handler()", which is inlined from "svm_vmexit_do_vmsave()" and "svm_vmsave()".

CRASH DUMP
==========

(XEN) Assertion 'va >= XEN_VIRT_START' failed at /sandbox/xen/xen.git/xen/include/asm/x86_64/page.h:86
(XEN) Debugging connection not set up.
(XEN) ----[ Xen-4.3-unstable  x86_64  debug=y  Not tainted ]----
(XEN) CPU:    17
(XEN) RIP:    e008:[<ffff82c4c01cfbfc>] svm_vmexit_handler+0x1574/0x1a2a
(XEN) RFLAGS: 0000000000010293   CONTEXT: hypervisor
(XEN) rax: ffff82c4bfffffff   rbx: ffff830852ec1000   rcx: 0000000000000000
(XEN) rdx: ffff830434757020   rsi: 000000000000000a   rdi: ffff82c4c0283740
(XEN) rbp: ffff83043474ff08   rsp: ffff83043474fd28   r8:  0000000000000004
(XEN) r9:  0000000000000010   r10: ffffff8000000000   r11: 0000000000000010
(XEN) r12: ffff83000e010000   r13: 0000000000000003   r14: 0000000000000000
(XEN) r15: ffff82c40002d000   cr0: 000000008005003b   cr4: 00000000000406f0
(XEN) cr3: 000000086d9dd000   cr2: 00007fe7f8e99120
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: e008
(XEN) Xen stack trace from rsp=ffff83043474fd28:
(XEN)    ffff83000e010000 ffff83043474fd70 ffff82c4c01bb001 0000000000000000
(XEN)    ffff83000e010000 ffff830852ec1000 0000000000000000 0000000000000000
(XEN)    ffff830434757080 ffff83043474fda0 ffff82c4c01cca33 0000000000000000
(XEN)    ffff8300c7ea6000 00000000000fee00 ffff83043474ff18 ffff830400000000
(XEN)    ffff82c4c015fe19 ffff83043474fe10 ffff82c4c0185827 00000000000000fc
(XEN)    0000003b5c327b44 0000000a0000000d 0000000000000000 0000000000000000
(XEN)    0000000000000000 ffff83043474fe20 ffff8300c7ea6000 00000049a0b0dcf5
(XEN)    0000000000000286 ffff83043474fe28 ffff82c4c0125e9e ffff83000e010000
(XEN)    ffff83043474fe98 ffff82c4c01c8048 ffff82c4c0125e9e ffff83000e010488
(XEN)    ffff83043474fe98 ffffffffffffffff ffff83043474fe78 ffff82c4c01c5e56
(XEN)    ffff83000e010000 ffff830853ea1000 ffff83043474fe98 ffff82c4c01be614
(XEN)    ffff83000e010000 0000000000000007 ffff83043474ff08 ffff82c4c01c8e66
(XEN)    ffff830434757080 000000fc3474fee0 ffff82c4c0125a52 ffff830434748000
(XEN)    ffff830434748000 00000000ffffffff ffff830852ec1000 ffff83000e010000
(XEN)    ffff830209c87000 0000000000000007 0000000000000003 ffff830203ddff18
(XEN)    ffff830203ddfd70 ffff82c4c01d1c45 ffff830203ddff18 0000000000000003
(XEN)    0000000000000007 ffff830209c87000 ffff830203ddfd70 ffff8300d4b46000
(XEN)    0000000000000246 00000000deadbeef 00000013eabcc169 0000000000000003
(XEN)    0000000203de2000 0000000000000000 0000000203de2000 ffff830203de4000
(XEN)    ffff830209c87000 0000beef0000beef ffff82c4c01ce158 0000beef0000beef
(XEN) Xen call trace:
(XEN)    [<ffff82c4c01cfbfc>] svm_vmexit_handler+0x1574/0x1a2a
(XEN)
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 17:
(XEN) Assertion 'va >= XEN_VIRT_START' failed at /sandbox/xen/xen.git/xen/include/asm/x86_64/page.h:92
(XEN) ****************************************
(XEN)
(XEN) Manual reset required ('noreboot' specified)
(XEN) Debugging connection not set up.

Suravee