traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85 and other dom0 induced log messages

traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85 and other dom0 induced log messages

[linux]

Hi Jan / David,

On the other thread you asked about some messages in xl-dmesg, over time 
there are more that come from a kernel booting. (mind you that there can 
be some changes in Kconfigs between the kernels i have used.)

3.16 was clean this is what xl-dmesg gained on log message that seem to 
be triggered by something in the dom0 Kernel booting on my AMD system.

from 3.16 to 3.19 we gained a lot of these, if i remember correctly 
related to
perf being enabled in the kernel:

+   traps.c:2655:d0v0 Domain attempted WRMSR 00000000c0000081 from 
0xe023e00800000000 to 0x0023001000000000.
+   traps.c:2655:d0v0 Domain attempted WRMSR 00000000c0000082 from 
0xffff82d0bffff000 to 0xffffffff81bc2670.
+   traps.c:2655:d0v0 Domain attempted WRMSR 00000000c0000083 from 
0xffff82d0bffff020 to 0xffffffff81bc4630.
+   traps.c:2655:d0v0 Domain attempted WRMSR 0000000000000174 from 
0x0000000000000000 to 0x0000000000000010.
+   traps.c:2655:d0v0 Domain attempted WRMSR 0000000000000176 from 
0x0000000000000000 to 0xffffffff81bc4660.
+   traps.c:2655:d0v0 Domain attempted WRMSR 00000000c0000083 from 
0xffff82d0bffff020 to 0xffffffff81bc48b0.
+   traps.c:2655:d0v0 Domain attempted WRMSR 00000000c0000084 from 
0x0000000000074700 to 0x0000000000047700.
     traps.c:2655:d0v0 Domain attempted WRMSR 00000000c0010004 from 
0x0000000000000000 to 0x000000000000ffff.
+   traps.c:2655:d0v1 Domain attempted WRMSR 00000000c0000081 from 
0xe023e00800000000 to 0x0023001000000000.
+   traps.c:2655:d0v1 Domain attempted WRMSR 00000000c0000082 from 
0xffff82d0bfffe080 to 0xffffffff81bc2670.
+   traps.c:2655:d0v1 Domain attempted WRMSR 00000000c0000083 from 
0xffff82d0bfffe0a0 to 0xffffffff81bc4630.
+   traps.c:2655:d0v1 Domain attempted WRMSR 0000000000000174 from 
0x0000000000000000 to 0x0000000000000010.
+   traps.c:2655:d0v1 Domain attempted WRMSR 0000000000000176 from 
0x0000000000000000 to 0xffffffff81bc4660.
+   traps.c:2655:d0v1 Domain attempted WRMSR 00000000c0000083 from 
0xffff82d0bfffe0a0 to 0xffffffff81bc48b0.
+   traps.c:2655:d0v1 Domain attempted WRMSR 00000000c0000084 from 
0x0000000000074700 to 0x0000000000047700.
+   traps.c:2655:d0v2 Domain attempted WRMSR 00000000c0000081 from 
0xe023e00800000000 to 0x0023001000000000.
+   traps.c:2655:d0v2 Domain attempted WRMSR 00000000c0000082 from 
0xffff82d0bfffd100 to 0xffffffff81bc2670.


from 3.19 to 4.0 we gained:
+   d0 attempted to change d0v0's CR4 flags 00000660 -> 00000760
+   d0 attempted to change d0v1's CR4 flags 00000660 -> 00000760
+   d0 attempted to change d0v2's CR4 flags 00000660 -> 00000760
+   d0 attempted to change d0v3's CR4 flags 00000660 -> 00000760
+   d0 attempted to change d0v4's CR4 flags 00000660 -> 00000760
+   d0 attempted to change d0v5's CR4 flags 00000660 -> 00000760

and from 4.0 to 4.1 we gained the ones you were interested in:
+   traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85
+   traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85
+   traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85
+   traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85
+   traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85
+   traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85

--
Sander

Re: traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85 and other dom0 induced log messages

[Jan Beulich]

>>> On 26.06.15 at 17:41, <linux@eikelenboom.it> wrote:
> from 3.16 to 3.19 we gained a lot of these, if i remember correctly 
> related to
> perf being enabled in the kernel:
> 
> +   traps.c:2655:d0v0 Domain attempted WRMSR 00000000c0000081 from 
> 0xe023e00800000000 to 0x0023001000000000.
> +   traps.c:2655:d0v0 Domain attempted WRMSR 00000000c0000082 from 
> 0xffff82d0bffff000 to 0xffffffff81bc2670.
> +   traps.c:2655:d0v0 Domain attempted WRMSR 00000000c0000083 from 
> 0xffff82d0bffff020 to 0xffffffff81bc4630.

These are the SYSCALL (STAR) MSRs, which the kernel has no business
touching when running on Xen.

> from 3.19 to 4.0 we gained:
> +   d0 attempted to change d0v0's CR4 flags 00000660 -> 00000760
> +   d0 attempted to change d0v1's CR4 flags 00000660 -> 00000760
> +   d0 attempted to change d0v2's CR4 flags 00000660 -> 00000760
> +   d0 attempted to change d0v3's CR4 flags 00000660 -> 00000760
> +   d0 attempted to change d0v4's CR4 flags 00000660 -> 00000760
> +   d0 attempted to change d0v5's CR4 flags 00000660 -> 00000760

This is X86_CR4_PCE - not sure how to properly handle that.
Andrew, you're fiddling with the CR4 handling right now anyway -
any thoughts?

> and from 4.0 to 4.1 we gained the ones you were interested in:
> +   traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85
> +   traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85
> +   traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85
> +   traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85
> +   traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85
> +   traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85

For these to be meaningful you need to translate them to symbolic
addresses. (And yes, we should see to make the code print them
in a more useful manner.)

Jan

Re: traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85 and other dom0 induced log messages

[Andrew Cooper]

On 26/06/15 16:51, Jan Beulich wrote:
>>>> On 26.06.15 at 17:41, <linux@eikelenboom.it> wrote:
>> from 3.16 to 3.19 we gained a lot of these, if i remember correctly 
>> related to
>> perf being enabled in the kernel:
>>
>> +   traps.c:2655:d0v0 Domain attempted WRMSR 00000000c0000081 from 
>> 0xe023e00800000000 to 0x0023001000000000.
>> +   traps.c:2655:d0v0 Domain attempted WRMSR 00000000c0000082 from 
>> 0xffff82d0bffff000 to 0xffffffff81bc2670.
>> +   traps.c:2655:d0v0 Domain attempted WRMSR 00000000c0000083 from 
>> 0xffff82d0bffff020 to 0xffffffff81bc4630.
> These are the SYSCALL (STAR) MSRs, which the kernel has no business
> touching when running on Xen.
>
>> from 3.19 to 4.0 we gained:
>> +   d0 attempted to change d0v0's CR4 flags 00000660 -> 00000760
>> +   d0 attempted to change d0v1's CR4 flags 00000660 -> 00000760
>> +   d0 attempted to change d0v2's CR4 flags 00000660 -> 00000760
>> +   d0 attempted to change d0v3's CR4 flags 00000660 -> 00000760
>> +   d0 attempted to change d0v4's CR4 flags 00000660 -> 00000760
>> +   d0 attempted to change d0v5's CR4 flags 00000660 -> 00000760
> This is X86_CR4_PCE - not sure how to properly handle that.
> Andrew, you're fiddling with the CR4 handling right now anyway -
> any thoughts?

We have no infrastructure whatsoever to allow PV guests to use rdpmc,
and PCE is unconditionally clear in CR4.

With Boris' perf series, and oprofile already having other PV interfaces
to access performance counters, I don't see any reason to open this up
to native use by PV guests.

>
>> and from 4.0 to 4.1 we gained the ones you were interested in:
>> +   traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85
>> +   traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85
>> +   traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85
>> +   traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85
>> +   traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85
>> +   traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85
> For these to be meaningful you need to translate them to symbolic
> addresses. (And yes, we should see to make the code print them
> in a more useful manner.)

For things like {wr,rd}msr_safe(), we should print ecx/eax/edx.  I
expect there are similar useful debug hints for other areas of code.  Is
it worth stashing some other information in the extable to aid generic
debug printing of errors?

~Andrew

Re: traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85 and other dom0 induced log messages

[Jan Beulich]

>>> On 29.06.15 at 16:01, <andrew.cooper3@citrix.com> wrote:
> On 26/06/15 16:51, Jan Beulich wrote:
>>>>> On 26.06.15 at 17:41, <linux@eikelenboom.it> wrote:
>>> from 3.16 to 3.19 we gained a lot of these, if i remember correctly 
>>> related to
>>> perf being enabled in the kernel:
>>>
>>> +   traps.c:2655:d0v0 Domain attempted WRMSR 00000000c0000081 from 
>>> 0xe023e00800000000 to 0x0023001000000000.
>>> +   traps.c:2655:d0v0 Domain attempted WRMSR 00000000c0000082 from 
>>> 0xffff82d0bffff000 to 0xffffffff81bc2670.
>>> +   traps.c:2655:d0v0 Domain attempted WRMSR 00000000c0000083 from 
>>> 0xffff82d0bffff020 to 0xffffffff81bc4630.
>> These are the SYSCALL (STAR) MSRs, which the kernel has no business
>> touching when running on Xen.
>>
>>> from 3.19 to 4.0 we gained:
>>> +   d0 attempted to change d0v0's CR4 flags 00000660 -> 00000760
>>> +   d0 attempted to change d0v1's CR4 flags 00000660 -> 00000760
>>> +   d0 attempted to change d0v2's CR4 flags 00000660 -> 00000760
>>> +   d0 attempted to change d0v3's CR4 flags 00000660 -> 00000760
>>> +   d0 attempted to change d0v4's CR4 flags 00000660 -> 00000760
>>> +   d0 attempted to change d0v5's CR4 flags 00000660 -> 00000760
>> This is X86_CR4_PCE - not sure how to properly handle that.
>> Andrew, you're fiddling with the CR4 handling right now anyway -
>> any thoughts?
> 
> We have no infrastructure whatsoever to allow PV guests to use rdpmc,
> and PCE is unconditionally clear in CR4.
> 
> With Boris' perf series, and oprofile already having other PV interfaces
> to access performance counters, I don't see any reason to open this up
> to native use by PV guests.

Matches my way of thinking.

>>> and from 4.0 to 4.1 we gained the ones you were interested in:
>>> +   traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85
>>> +   traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85
>>> +   traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85
>>> +   traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85
>>> +   traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85
>>> +   traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85
>> For these to be meaningful you need to translate them to symbolic
>> addresses. (And yes, we should see to make the code print them
>> in a more useful manner.)
> 
> For things like {wr,rd}msr_safe(), we should print ecx/eax/edx.  I
> expect there are similar useful debug hints for other areas of code.  Is
> it worth stashing some other information in the extable to aid generic
> debug printing of errors?

Not sure this wouldn't quickly become too complex. My comment
mainly was aiming at converting the hex number to symbol
references.

Jan

Re: traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85 and other dom0 induced log messages

[Sander Eikelenboom]

Monday, July 6, 2015, 11:33:09 AM, you wrote:

>>>> On 26.06.15 at 17:57, <linux@eikelenboom.it> wrote:
>> On 2015-06-26 17:51, Jan Beulich wrote:
>>>>>> On 26.06.15 at 17:41, <linux@eikelenboom.it> wrote:
>>>> from 3.16 to 3.19 we gained a lot of these, if i remember correctly
>>>> related to
>>>> perf being enabled in the kernel:
>>>> 
>>>> +   traps.c:2655:d0v0 Domain attempted WRMSR 00000000c0000081 from
>>>> 0xe023e00800000000 to 0x0023001000000000.
>>>> +   traps.c:2655:d0v0 Domain attempted WRMSR 00000000c0000082 from
>>>> 0xffff82d0bffff000 to 0xffffffff81bc2670.
>>>> +   traps.c:2655:d0v0 Domain attempted WRMSR 00000000c0000083 from
>>>> 0xffff82d0bffff020 to 0xffffffff81bc4630.
>>> 
>>> These are the SYSCALL (STAR) MSRs, which the kernel has no business
>>> touching when running on Xen.
>>> 
>>>> from 3.19 to 4.0 we gained:
>>>> +   d0 attempted to change d0v0's CR4 flags 00000660 -> 00000760
>>>> +   d0 attempted to change d0v1's CR4 flags 00000660 -> 00000760
>>>> +   d0 attempted to change d0v2's CR4 flags 00000660 -> 00000760
>>>> +   d0 attempted to change d0v3's CR4 flags 00000660 -> 00000760
>>>> +   d0 attempted to change d0v4's CR4 flags 00000660 -> 00000760
>>>> +   d0 attempted to change d0v5's CR4 flags 00000660 -> 00000760
>>> 
>>> This is X86_CR4_PCE - not sure how to properly handle that.
>>> Andrew, you're fiddling with the CR4 handling right now anyway -
>>> any thoughts?
>>> 
>>>> and from 4.0 to 4.1 we gained the ones you were interested in:
>>>> +   traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85
>>>> +   traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85
>>>> +   traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85
>>>> +   traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85
>>>> +   traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85
>>>> +   traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85
>>> 
>>> For these to be meaningful you need to translate them to symbolic
>>> addresses. (And yes, we should see to make the code print them
>>> in a more useful manner.)
>> 
>> How ?

> addr2line against xen-syms (or xen.efi if you use that one). And of
> course the result may need manual adjustment to account for
> eventual patches you have in your tree.

> Jan

Ah yeah .. silly me .. somehow i had in mind it would be kernel addresses 
instead of xen, so running it against vmlinux of course lead no where.

Here we go:

(XEN) [2015-07-08 08:31:00.384] traps.c:3227: GPF (0000): ffff82d080195583 -> ffff82d080239d85
(XEN) [2015-07-08 08:31:00.384] traps.c:3227: GPF (0000): ffff82d080195583 -> ffff82d080239d85

which leads to:
# addr2line -e /usr/lib/debug/xen-syms-4.6-unstable ffff82d080195583
/usr/src/new/xen-unstable/xen/arch/x86/traps.c:2758

# addr2line -e /usr/lib/debug/xen-syms-4.6-unstable ffff82d080239d85
??:?

Were /usr/src/new/xen-unstable/xen/arch/x86/traps.c:2758 leads to:

        case MSR_EFER:
 rdmsr_normal:
            /* Everyone can read the MSR space. */
            /* gdprintk(XENLOG_WARNING,"Domain attempted RDMSR %p.\n",
                        _p(regs->ecx));*/
HERE -->    if ( rdmsr_safe(regs->ecx, val) )
                goto fail;
 rdmsr_writeback:
            regs->eax = (uint32_t)val;
            regs->edx = (uint32_t)(val >> 32);
            break;
        }
        break;

Re: traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85 and other dom0 induced log messages

[Andrew Cooper]

On 08/07/2015 09:45, Sander Eikelenboom wrote:
> Monday, July 6, 2015, 11:33:09 AM, you wrote:
>
>>>>> On 26.06.15 at 17:57, <linux@eikelenboom.it> wrote:
>>> On 2015-06-26 17:51, Jan Beulich wrote:
>>>>>>> On 26.06.15 at 17:41, <linux@eikelenboom.it> wrote:
>>>>> from 3.16 to 3.19 we gained a lot of these, if i remember correctly
>>>>> related to
>>>>> perf being enabled in the kernel:
>>>>>
>>>>> +   traps.c:2655:d0v0 Domain attempted WRMSR 00000000c0000081 from
>>>>> 0xe023e00800000000 to 0x0023001000000000.
>>>>> +   traps.c:2655:d0v0 Domain attempted WRMSR 00000000c0000082 from
>>>>> 0xffff82d0bffff000 to 0xffffffff81bc2670.
>>>>> +   traps.c:2655:d0v0 Domain attempted WRMSR 00000000c0000083 from
>>>>> 0xffff82d0bffff020 to 0xffffffff81bc4630.
>>>> These are the SYSCALL (STAR) MSRs, which the kernel has no business
>>>> touching when running on Xen.
>>>>
>>>>> from 3.19 to 4.0 we gained:
>>>>> +   d0 attempted to change d0v0's CR4 flags 00000660 -> 00000760
>>>>> +   d0 attempted to change d0v1's CR4 flags 00000660 -> 00000760
>>>>> +   d0 attempted to change d0v2's CR4 flags 00000660 -> 00000760
>>>>> +   d0 attempted to change d0v3's CR4 flags 00000660 -> 00000760
>>>>> +   d0 attempted to change d0v4's CR4 flags 00000660 -> 00000760
>>>>> +   d0 attempted to change d0v5's CR4 flags 00000660 -> 00000760
>>>> This is X86_CR4_PCE - not sure how to properly handle that.
>>>> Andrew, you're fiddling with the CR4 handling right now anyway -
>>>> any thoughts?
>>>>
>>>>> and from 4.0 to 4.1 we gained the ones you were interested in:
>>>>> +   traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85
>>>>> +   traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85
>>>>> +   traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85
>>>>> +   traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85
>>>>> +   traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85
>>>>> +   traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85
>>>> For these to be meaningful you need to translate them to symbolic
>>>> addresses. (And yes, we should see to make the code print them
>>>> in a more useful manner.)
>>> How ?
>> addr2line against xen-syms (or xen.efi if you use that one). And of
>> course the result may need manual adjustment to account for
>> eventual patches you have in your tree.
>> Jan
> Ah yeah .. silly me .. somehow i had in mind it would be kernel addresses 
> instead of xen, so running it against vmlinux of course lead no where.
>
> Here we go:
>
> (XEN) [2015-07-08 08:31:00.384] traps.c:3227: GPF (0000): ffff82d080195583 -> ffff82d080239d85
> (XEN) [2015-07-08 08:31:00.384] traps.c:3227: GPF (0000): ffff82d080195583 -> ffff82d080239d85
>
> which leads to:
> # addr2line -e /usr/lib/debug/xen-syms-4.6-unstable ffff82d080195583
> /usr/src/new/xen-unstable/xen/arch/x86/traps.c:2758
>
> # addr2line -e /usr/lib/debug/xen-syms-4.6-unstable ffff82d080239d85
> ??:?

The second one is not.  It is the fixup label, which will be hidden away
out-of-line, and lacking debug symbols.

>
> Were /usr/src/new/xen-unstable/xen/arch/x86/traps.c:2758 leads to:
>
>         case MSR_EFER:
>  rdmsr_normal:
>             /* Everyone can read the MSR space. */
>             /* gdprintk(XENLOG_WARNING,"Domain attempted RDMSR %p.\n",
>                         _p(regs->ecx));*/
> HERE -->    if ( rdmsr_safe(regs->ecx, val) )
>                 goto fail;

Moving the printk into the fail case will identify which is the
problematic MSR.  We need the value of regs->_ecx here (the low 32bits,
not the full 64 as the commented printk currently has).

I have a small todo list of misc debugging improvements.  I will add
this to the list.

~Andrew

>  rdmsr_writeback:
>             regs->eax = (uint32_t)val;
>             regs->edx = (uint32_t)(val >> 32);
>             break;
>         }
>         break;
>

Re: traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85 and other dom0 induced log messages

[Sander Eikelenboom]

Wednesday, July 8, 2015, 10:58:02 AM, you wrote:

> On 08/07/2015 09:45, Sander Eikelenboom wrote:
>> Monday, July 6, 2015, 11:33:09 AM, you wrote:
>>
>>>>>> On 26.06.15 at 17:57, <linux@eikelenboom.it> wrote:
>>>> On 2015-06-26 17:51, Jan Beulich wrote:
>>>>>>>> On 26.06.15 at 17:41, <linux@eikelenboom.it> wrote:
>>>>>> from 3.16 to 3.19 we gained a lot of these, if i remember correctly
>>>>>> related to
>>>>>> perf being enabled in the kernel:
>>>>>>
>>>>>> +   traps.c:2655:d0v0 Domain attempted WRMSR 00000000c0000081 from
>>>>>> 0xe023e00800000000 to 0x0023001000000000.
>>>>>> +   traps.c:2655:d0v0 Domain attempted WRMSR 00000000c0000082 from
>>>>>> 0xffff82d0bffff000 to 0xffffffff81bc2670.
>>>>>> +   traps.c:2655:d0v0 Domain attempted WRMSR 00000000c0000083 from
>>>>>> 0xffff82d0bffff020 to 0xffffffff81bc4630.
>>>>> These are the SYSCALL (STAR) MSRs, which the kernel has no business
>>>>> touching when running on Xen.
>>>>>
>>>>>> from 3.19 to 4.0 we gained:
>>>>>> +   d0 attempted to change d0v0's CR4 flags 00000660 -> 00000760
>>>>>> +   d0 attempted to change d0v1's CR4 flags 00000660 -> 00000760
>>>>>> +   d0 attempted to change d0v2's CR4 flags 00000660 -> 00000760
>>>>>> +   d0 attempted to change d0v3's CR4 flags 00000660 -> 00000760
>>>>>> +   d0 attempted to change d0v4's CR4 flags 00000660 -> 00000760
>>>>>> +   d0 attempted to change d0v5's CR4 flags 00000660 -> 00000760
>>>>> This is X86_CR4_PCE - not sure how to properly handle that.
>>>>> Andrew, you're fiddling with the CR4 handling right now anyway -
>>>>> any thoughts?
>>>>>
>>>>>> and from 4.0 to 4.1 we gained the ones you were interested in:
>>>>>> +   traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85
>>>>>> +   traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85
>>>>>> +   traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85
>>>>>> +   traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85
>>>>>> +   traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85
>>>>>> +   traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85
>>>>> For these to be meaningful you need to translate them to symbolic
>>>>> addresses. (And yes, we should see to make the code print them
>>>>> in a more useful manner.)
>>>> How ?
>>> addr2line against xen-syms (or xen.efi if you use that one). And of
>>> course the result may need manual adjustment to account for
>>> eventual patches you have in your tree.
>>> Jan
>> Ah yeah .. silly me .. somehow i had in mind it would be kernel addresses 
>> instead of xen, so running it against vmlinux of course lead no where.
>>
>> Here we go:
>>
>> (XEN) [2015-07-08 08:31:00.384] traps.c:3227: GPF (0000): ffff82d080195583 -> ffff82d080239d85
>> (XEN) [2015-07-08 08:31:00.384] traps.c:3227: GPF (0000): ffff82d080195583 -> ffff82d080239d85
>>
>> which leads to:
>> # addr2line -e /usr/lib/debug/xen-syms-4.6-unstable ffff82d080195583
>> /usr/src/new/xen-unstable/xen/arch/x86/traps.c:2758
>>
>> # addr2line -e /usr/lib/debug/xen-syms-4.6-unstable ffff82d080239d85
>> ??:?

> The second one is not.  It is the fixup label, which will be hidden away
> out-of-line, and lacking debug symbols.

>>
>> Were /usr/src/new/xen-unstable/xen/arch/x86/traps.c:2758 leads to:
>>
>>         case MSR_EFER:
>>  rdmsr_normal:
>>             /* Everyone can read the MSR space. */
>>             /* gdprintk(XENLOG_WARNING,"Domain attempted RDMSR %p.\n",
>>                         _p(regs->ecx));*/
>> HERE -->    if ( rdmsr_safe(regs->ecx, val) )
>>                 goto fail;

> Moving the printk into the fail case will identify which is the
> problematic MSR.  We need the value of regs->_ecx here (the low 32bits,
> not the full 64 as the commented printk currently has).

> I have a small todo list of misc debugging improvements.  I will add
> this to the list.

> ~Andrew

>>  rdmsr_writeback:
>>             regs->eax = (uint32_t)val;
>>             regs->edx = (uint32_t)(val >> 32);
>>             break;
>>         }
>>         break;
>>

Don't know if the full 64bits is of equal use, but here it is:

(XEN) [2015-07-08 10:01:58.717] traps.c:2760:d14v0 Domain attempted but failed RDMSR 0000000000000570.


--
Sander

Re: traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85 and other dom0 induced log messages

[Andrew Cooper]

On 08/07/2015 11:04, Sander Eikelenboom wrote:
> Wednesday, July 8, 2015, 10:58:02 AM, you wrote:
>
>> On 08/07/2015 09:45, Sander Eikelenboom wrote:
>>> Monday, July 6, 2015, 11:33:09 AM, you wrote:
>>>
>>>>>>> On 26.06.15 at 17:57, <linux@eikelenboom.it> wrote:
>>>>> On 2015-06-26 17:51, Jan Beulich wrote:
>>>>>>>>> On 26.06.15 at 17:41, <linux@eikelenboom.it> wrote:
>>>>>>> from 3.16 to 3.19 we gained a lot of these, if i remember correctly
>>>>>>> related to
>>>>>>> perf being enabled in the kernel:
>>>>>>>
>>>>>>> +   traps.c:2655:d0v0 Domain attempted WRMSR 00000000c0000081 from
>>>>>>> 0xe023e00800000000 to 0x0023001000000000.
>>>>>>> +   traps.c:2655:d0v0 Domain attempted WRMSR 00000000c0000082 from
>>>>>>> 0xffff82d0bffff000 to 0xffffffff81bc2670.
>>>>>>> +   traps.c:2655:d0v0 Domain attempted WRMSR 00000000c0000083 from
>>>>>>> 0xffff82d0bffff020 to 0xffffffff81bc4630.
>>>>>> These are the SYSCALL (STAR) MSRs, which the kernel has no business
>>>>>> touching when running on Xen.
>>>>>>
>>>>>>> from 3.19 to 4.0 we gained:
>>>>>>> +   d0 attempted to change d0v0's CR4 flags 00000660 -> 00000760
>>>>>>> +   d0 attempted to change d0v1's CR4 flags 00000660 -> 00000760
>>>>>>> +   d0 attempted to change d0v2's CR4 flags 00000660 -> 00000760
>>>>>>> +   d0 attempted to change d0v3's CR4 flags 00000660 -> 00000760
>>>>>>> +   d0 attempted to change d0v4's CR4 flags 00000660 -> 00000760
>>>>>>> +   d0 attempted to change d0v5's CR4 flags 00000660 -> 00000760
>>>>>> This is X86_CR4_PCE - not sure how to properly handle that.
>>>>>> Andrew, you're fiddling with the CR4 handling right now anyway -
>>>>>> any thoughts?
>>>>>>
>>>>>>> and from 4.0 to 4.1 we gained the ones you were interested in:
>>>>>>> +   traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85
>>>>>>> +   traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85
>>>>>>> +   traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85
>>>>>>> +   traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85
>>>>>>> +   traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85
>>>>>>> +   traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85
>>>>>> For these to be meaningful you need to translate them to symbolic
>>>>>> addresses. (And yes, we should see to make the code print them
>>>>>> in a more useful manner.)
>>>>> How ?
>>>> addr2line against xen-syms (or xen.efi if you use that one). And of
>>>> course the result may need manual adjustment to account for
>>>> eventual patches you have in your tree.
>>>> Jan
>>> Ah yeah .. silly me .. somehow i had in mind it would be kernel addresses 
>>> instead of xen, so running it against vmlinux of course lead no where.
>>>
>>> Here we go:
>>>
>>> (XEN) [2015-07-08 08:31:00.384] traps.c:3227: GPF (0000): ffff82d080195583 -> ffff82d080239d85
>>> (XEN) [2015-07-08 08:31:00.384] traps.c:3227: GPF (0000): ffff82d080195583 -> ffff82d080239d85
>>>
>>> which leads to:
>>> # addr2line -e /usr/lib/debug/xen-syms-4.6-unstable ffff82d080195583
>>> /usr/src/new/xen-unstable/xen/arch/x86/traps.c:2758
>>>
>>> # addr2line -e /usr/lib/debug/xen-syms-4.6-unstable ffff82d080239d85
>>> ??:?
>> The second one is not.  It is the fixup label, which will be hidden away
>> out-of-line, and lacking debug symbols.
>>> Were /usr/src/new/xen-unstable/xen/arch/x86/traps.c:2758 leads to:
>>>
>>>         case MSR_EFER:
>>>  rdmsr_normal:
>>>             /* Everyone can read the MSR space. */
>>>             /* gdprintk(XENLOG_WARNING,"Domain attempted RDMSR %p.\n",
>>>                         _p(regs->ecx));*/
>>> HERE -->    if ( rdmsr_safe(regs->ecx, val) )
>>>                 goto fail;
>> Moving the printk into the fail case will identify which is the
>> problematic MSR.  We need the value of regs->_ecx here (the low 32bits,
>> not the full 64 as the commented printk currently has).
>> I have a small todo list of misc debugging improvements.  I will add
>> this to the list.
>> ~Andrew
>>>  rdmsr_writeback:
>>>             regs->eax = (uint32_t)val;
>>>             regs->edx = (uint32_t)(val >> 32);
>>>             break;
>>>         }
>>>         break;
>>>
> Don't know if the full 64bits is of equal use

It is (just with an unhelpful quantity of zeroes)

> , but here it is:
>
> (XEN) [2015-07-08 10:01:58.717] traps.c:2760:d14v0 Domain attempted but failed RDMSR 0000000000000570.

Looks to be  MSR_IA32_RTIT_CTL, which is part of the Intel Processor
Trace PMU driver (Linux/arch/x86/kernel/cpu/perf_event_intel_pt.c).  A
PV domain running on AMD absolutely shouldn't be attempting to read this.

It appears that pt_init() blindly probes the MSR without any
cpuid/vendor detection.

~Andrew

Re: traps.c:3227: GPF (0000): ffff82d080194a4d -> ffff82d080239d85 and other dom0 induced log messages

[Jan Beulich]

>>> On 08.07.15 at 10:45, <linux@eikelenboom.it> wrote:
> Here we go:
> 
> (XEN) [2015-07-08 08:31:00.384] traps.c:3227: GPF (0000): ffff82d080195583 -> 
> ffff82d080239d85
> (XEN) [2015-07-08 08:31:00.384] traps.c:3227: GPF (0000): ffff82d080195583 -> 
> ffff82d080239d85
> 
> which leads to:
> # addr2line -e /usr/lib/debug/xen-syms-4.6-unstable ffff82d080195583
> /usr/src/new/xen-unstable/xen/arch/x86/traps.c:2758
> 
> # addr2line -e /usr/lib/debug/xen-syms-4.6-unstable ffff82d080239d85
> ??:?
> 
> Were /usr/src/new/xen-unstable/xen/arch/x86/traps.c:2758 leads to:
> 
>         case MSR_EFER:
>  rdmsr_normal:
>             /* Everyone can read the MSR space. */
>             /* gdprintk(XENLOG_WARNING,"Domain attempted RDMSR %p.\n",
>                         _p(regs->ecx));*/
> HERE -->    if ( rdmsr_safe(regs->ecx, val) )

Right, so as Andrew suspected - we won't know whether that's
legitimate/reasonable without knowing the MSR being accessed.

Jan