Re: traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85 and other dom0 induced log messages

[Andrew Cooper <andrew.cooper3@citrix.com>]

On 08/07/2015 09:45, Sander Eikelenboom wrote:
> Monday, July 6, 2015, 11:33:09 AM, you wrote:
>
>>>>> On 26.06.15 at 17:57, <linux@eikelenboom.it> wrote:
>>> On 2015-06-26 17:51, Jan Beulich wrote:
>>>>>>> On 26.06.15 at 17:41, <linux@eikelenboom.it> wrote:
>>>>> from 3.16 to 3.19 we gained a lot of these, if i remember correctly
>>>>> related to
>>>>> perf being enabled in the kernel:
>>>>>
>>>>> +   traps.c:2655:d0v0 Domain attempted WRMSR 00000000c0000081 from
>>>>> 0xe023e00800000000 to 0x0023001000000000.
>>>>> +   traps.c:2655:d0v0 Domain attempted WRMSR 00000000c0000082 from
>>>>> 0xffff82d0bffff000 to 0xffffffff81bc2670.
>>>>> +   traps.c:2655:d0v0 Domain attempted WRMSR 00000000c0000083 from
>>>>> 0xffff82d0bffff020 to 0xffffffff81bc4630.
>>>> These are the SYSCALL (STAR) MSRs, which the kernel has no business
>>>> touching when running on Xen.
>>>>
>>>>> from 3.19 to 4.0 we gained:
>>>>> +   d0 attempted to change d0v0's CR4 flags 00000660 -> 00000760
>>>>> +   d0 attempted to change d0v1's CR4 flags 00000660 -> 00000760
>>>>> +   d0 attempted to change d0v2's CR4 flags 00000660 -> 00000760
>>>>> +   d0 attempted to change d0v3's CR4 flags 00000660 -> 00000760
>>>>> +   d0 attempted to change d0v4's CR4 flags 00000660 -> 00000760
>>>>> +   d0 attempted to change d0v5's CR4 flags 00000660 -> 00000760
>>>> This is X86_CR4_PCE - not sure how to properly handle that.
>>>> Andrew, you're fiddling with the CR4 handling right now anyway -
>>>> any thoughts?
>>>>
>>>>> and from 4.0 to 4.1 we gained the ones you were interested in:
>>>>> +   traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85
>>>>> +   traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85
>>>>> +   traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85
>>>>> +   traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85
>>>>> +   traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85
>>>>> +   traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85
>>>> For these to be meaningful you need to translate them to symbolic
>>>> addresses. (And yes, we should see to make the code print them
>>>> in a more useful manner.)
>>> How ?
>> addr2line against xen-syms (or xen.efi if you use that one). And of
>> course the result may need manual adjustment to account for
>> eventual patches you have in your tree.
>> Jan
> Ah yeah .. silly me .. somehow i had in mind it would be kernel addresses 
> instead of xen, so running it against vmlinux of course lead no where.
>
> Here we go:
>
> (XEN) [2015-07-08 08:31:00.384] traps.c:3227: GPF (0000): ffff82d080195583 -> ffff82d080239d85
> (XEN) [2015-07-08 08:31:00.384] traps.c:3227: GPF (0000): ffff82d080195583 -> ffff82d080239d85
>
> which leads to:
> # addr2line -e /usr/lib/debug/xen-syms-4.6-unstable ffff82d080195583
> /usr/src/new/xen-unstable/xen/arch/x86/traps.c:2758
>
> # addr2line -e /usr/lib/debug/xen-syms-4.6-unstable ffff82d080239d85
> ??:?

The second one is not.  It is the fixup label, which will be hidden away
out-of-line, and lacking debug symbols.

>
> Were /usr/src/new/xen-unstable/xen/arch/x86/traps.c:2758 leads to:
>
>         case MSR_EFER:
>  rdmsr_normal:
>             /* Everyone can read the MSR space. */
>             /* gdprintk(XENLOG_WARNING,"Domain attempted RDMSR %p.\n",
>                         _p(regs->ecx));*/
> HERE -->    if ( rdmsr_safe(regs->ecx, val) )
>                 goto fail;

Moving the printk into the fail case will identify which is the
problematic MSR.  We need the value of regs->_ecx here (the low 32bits,
not the full 64 as the commented printk currently has).

I have a small todo list of misc debugging improvements.  I will add
this to the list.

~Andrew

>  rdmsr_writeback:
>             regs->eax = (uint32_t)val;
>             regs->edx = (uint32_t)(val >> 32);
>             break;
>         }
>         break;
>