Re: Branch Trace Storage for guests andVPMUinitialization

[Boris Ostrovsky <boris.ostrovsky@oracle.com>]

On 02/26/2015 12:57 PM, Kevin.Mayer@gdata.de wrote:
>
>
>> -----Ursprüngliche Nachricht-----
>> Von: Boris Ostrovsky [mailto:boris.ostrovsky@oracle.com]
>> Gesendet: Donnerstag, 26. Februar 2015 17:35
>> An: Dietmar Hahn; xen-devel@lists.xen.org
>> Cc: Mayer, Kevin
>> Betreff: Re: [Xen-devel] Branch Trace Storage for guests and
>> VPMUinitialization
>>
>> On 02/26/2015 03:56 AM, Dietmar Hahn wrote:
>>> Am Mittwoch 25 Februar 2015, 11:31:31 schrieb Boris Ostrovsky:
>>>> On 02/25/2015 10:12 AM, Kevin.Mayer@gdata.de wrote:
>>>>>> -----Ursprüngliche Nachricht-----
>>>>>> Von: Boris Ostrovsky [mailto:boris.ostrovsky@oracle.com]
>>>>>> Gesendet: Dienstag, 24. Februar 2015 18:13
>>>>>> An: Mayer, Kevin; xen-devel@lists.xen.org
>>>>>> Betreff: Re: [Xen-devel] Branch Trace Storage for guests and VPMU
>>>>>> initialization
>>>>>>
>>>>>> On 02/24/2015 10:27 AM, Kevin.Mayer@gdata.de wrote:
>>>>>>> Hi guys
>>>>>>>
>>>>>>> I`m trying to set up the BTS so that I can log the branches taken
>>>>>>> in the guest using Xen 4.4.1 with a WinXP SP3 guest on a Core i7
>>>>>>> Sandy Bridge.
>>>>>>>
>>>>>>> I added the vpmu=bts boot parameter to my grub2 configuration and
>>>>>>> extended the libxl,libxc,domctl,… with an own command so that I
>>>>>>> can trigger the activation of the BTS whenever I want.
>>>>>>>
>>>>>> I am not sure why you are doing all these changes to Xen code. BTS
>>>>>> is supposed to be managed from the guest. For example, a Fedora
>> HVM
>>>>>> guest will produce this:
>>>>>>
>>>>>> [root@dhcp-burlington7-2nd-B-east-10-152-55-140 ~]# perf record -e
>>>>>> branches:u -c 1 -d sleep 1 [ perf record: Woken up 3838 times to
>>>>>> write data ] [ perf record: Captured and wrote 0.704 MB perf.data
>>>>>> (~30756 samples) ]
>>>>>> [root@dhcp-burlington7-2nd-B-east-10-152-55-140 ~]# perf script -f
>>>>>> ip,addr,sym,dso,symoff --show-kernel-path
>>>>>>      ffffffff8167c347 native_irq_return_iret+0x0 (/proc/kcore) =>
>>>>>> 328c001590 [unknown] (/proc/kcore)
>>>>>>      ffffffff8167c347 native_irq_return_iret+0x0 (/proc/kcore) =>
>>>>>> 328c001590 [unknown] ([unknown])
>>>>>>            328c001593 [unknown] ([unknown]) =>       328c004b70 [unknown]
>>>>>> ([unknown])
>>>>>> ...
>>>>>>
>>>>> I want to be able to log the taken branches (of the guest) without the
>> need to modify the guest at all.
>>>>> This means I have to do all the logic in the hypervisor, or am I wrong?
>>>> In that case, yes. But then you have to make sure that at least
>>>>     * you don't load guest's VPMU (or, at least, BTS-related
>>>> registers) on context switch
>>> But you need to modify PMU registers when switching to/from the guest
>>> context to get PMU running.
>>
>>
>>
>> I was thinking that all BTS stuff can be controlled from dom0 and so we can
>> use dom0's version of these registers. I didn't realize that DS_AREA would
>> have to be accessed in guest's address space (and that DEBUGCTL is loaded
>> from VMCS).
>>
>> Which is what I think I said in response to this message (which didn't show up
>> on the list because Kevin accidentally dropped xen-devel).
>>
>> -boris
>
> Terribly sorry about that...
>
> So the VPMU doesn’t get loaded when there is a VMENTER?


Not exactly. For BTS, DEBUGCTL register, which lives in VMCS, does get 
loaded. But not DS_AREA --- it gets loaded by SW during 
context_switch()->vpmu_load().

(As for general VPMU registers such as counters --- they are also loaded 
during context_switch(). But I don't think you care about those. From 
what little I know about BTS, DEBUGCTL and DS_AREA are the only two 
registers you are interested in)

> I thought I could set the domU->vcpu->vpmu to enable BTS while in dom0 (with modified versions of msr_write_intercept, vpmu_do_wrmsr and core2_vpmu_do_wrmsr of course since the build in ones use the current-vcpu which would be the dom0-vcpu)
> and as soon as there is a context switch to domU the vpmu gets loaded and the guest starts logging.

And it should work, provided that DS_AREA is set up correctly.

> If the described behavior is correct the only problem I can see is with allocating memory in dom0 in a way that the guest can access it.

This sounds right. All you have to do now is implementation details ;-)

-boris


> But if I got it wrong please explain how the vpmu really works.
>
> Cheers
>
> Kevin
>
>
>>
>>
>>> I didn't think of using the VPMU stuff with modifying the context from
>>> outside the guest.
>>>
>>>>     * You don't send the interrupt to the guest (meaning that you will
>>>> need to somehow inform dom0 of the BTS interrupt)
>>>>
>>>> and probably more.
>>>>
>>>> Essentially, you want dom0 to profile the guest. I have been working
>>>> on patches that would allow that but they are still under review.
>>>>
>>>>
>>>>>>> In this command I do the following:
>>>>>>>
>>>>>>> I set up the memory region for the BTS Buffer and the DS Buffer
>>>>>>> Management Area using xzalloc_bytes
>>>>>>>
>>>>>> I don't think you should be allocating BTS buffers in the
>>>>>> hypervisor, they are in guest's memory.
>>>>> I agree. As I said I think this is where my main problem is at the moment.
>>>>> Is there any way I can allocate memory in the hypervisor in a way the
>> guest can access it?
>>>> I am not sure this is what you want since you seem to *not* want the
>>>> guest to process the samples, right?
>>>>
>>>> But yes, you can. E.g. something like what map_vcpu_info() does. (I
>>>> have no idea how you'd do this from Windows.)
>>> The DS buffer has to be mapped within the guests address space so the
>>> CPU running in guest context can access this area. Otherwise you get
>>> this triple fault.
>>> So I would think you need a mixture of writing some stuff in Windows
>>> and patching the hypervisor.
>>>
>>> Dietmar.
>>>
>>>>
>>>>> Of course the guest must not be able to use this memory in its normal
>> operations but just for BTS.
>>>>> Is this even possible? I am rather confused at the moment. :-D
>>>>>
>>>>>>> Then I write the pointer to the BTS Buffer into the DS Buffer
>>>>>>> Management Area at +0x0 and +0x8 (BTS Buffer Base and BTS Index)
>>>>>>>
>>>>>>> When I use vmx_msr_write_intercept to store the value in
>>>>>>> MSR_IA32_DS_AREA the host reboots (my idea is he tries to access a
>>>>>>> vpmu-struct that isn´t there in the current vcpu and panics).
>>>>
>>>> Who is trying to write to MSR_IA32_DS_AREA? The guest or dom0? I
>>>> thought you said that you want dom0 to do sampling. Or are you trying
>>>> to setup DS area from your guest and control it from dom0? I am
>> somewhat confused.
>>>>
>>>>>> Can you post hypervisor log? (hard to say how helpful it will be
>>>>>> without seeing your code changes though)
>>>>>>
>>>>> Right after enabling the BTS I get a triple fault.
>>>>> hvm.c:1357:d2 Triple fault on VCPU0 - invoking HVM shutdown action 1.
>>>>
>>>> That's not host reboot, this is your guest dying.
>>>>
>>>>
>>>>>>> When I use a modified version of vmx_msr_write_intercept I don’t
>>>>>>> get any crashes as long as I don’t enable BTS and TR in the
>>>>>>> GUEST_IA32_DEBUGCTL (BTR works). When I enable the BTS (and TR)
>>>>>>> the guest crashes. I suppose he gets killed by the hypervisor for
>>>>>>> accessing forbidden memory.
>>>>>>>
>>>>>> Possibly because DS area point to hypervisor memory.
>>>>>>
>>>>>>
>>>>>> Having said all this, I am not sure how well BTS works. You did
>>>>>> notice this in the hypervisor log:
>>>>>>
>>>>>> (XEN)
>> ******************************************************
>>>>>> (XEN) ** WARNING: Emulation of BTS Feature is switched on **
>>>>>> (XEN) ** Using this processor feature in a virtualized **
>>>>>> (XEN) ** environment is not 100% safe. **
>>>>>> (XEN) ** Setting the DS buffer address with wrong values **
>>>>>> (XEN) ** may lead to hypervisor hangs or crashes. **
>>>>>> (XEN) ** It is NOT recommended for production use! **
>>>>>> (XEN)
>> ******************************************************
>>>>>>
>>>>> Yes, I saw that. It doesn’t state that BTS is not working at all, just that it is
>> not that safe to use.
>>>>> As I understand it as long as I set the DS buffer address correctly I should
>> be fine, right?
>>>> Right. Except that I am not convinced you did set this buffer
>>>> correctly, which is possibly why your hypervisor crashed (I am not
>>>> sure I understood under what circumstances though).
>>>>
>>>> -boris
>>>>
>>>>> Since I don’t want to use for production that is fine with me. At least for
>> now.
>>>>>
>>>>>
>>>>> Kevin
>>>>>> -boris
>>>>>>
>>>>>>
>>>>>>> The modified version of vmx_msr_write_intercept takes a
>>>>>>> vcpu-struct as a parameter and uses this instead of the current vcpu.
>>>>>>>
>>>>>>> Instead of
>>>>>>>
>>>>>>> staticint vmx_msr_write_intercept(unsigned int msr, uint64_t
>>>>>> msr_content)
>>>>>>> {
>>>>>>>
>>>>>>>        struct vcpu *v = current;
>>>>>>>
>>>>>>> I just have
>>>>>>>
>>>>>>> staticint own_vmx_msr_write_intercept(unsigned int msr, uint64_t
>>>>>>> msr_content, struct vcpu *v)
>>>>>>>
>>>>>>> I get this vcpu by d->vcpu[0] as I have limited my guest domain to
>>>>>>> one vcpu atm.
>>>>>>>
>>>>>>> Of course I also use similarly modified version of the called
>>>>>>> functions(vpmu_do_wrmsr,…).
>>>>>>>
>>>>>>> I´m pretty sure that my problem is with a wrong scope/usage of the
>>>>>>> vcpus/memory, but I have no idea how to fix this.
>>>>>>>
>>>>>>> I can see a potential problem with the memory allocation (in the
>>>>>>> host) into which the cpu in guest-mode is supposed to write.
>>>>>>>
>>>>>>> Or maybe I got the principle of a vcpu/vpmu all wrong.
>>>>>>>
>>>>>>> Since I couldn’t find any project that uses the BTS for the guest,
>>>>>>> I am wondering if anyone has ever done this and if it is possible at all.
>>>>>>>
>>>>>>> Any input is welcome as I am pretty much stuck atm…
>>>>>>>
>>>>>>> Cheers
>>>>>>>
>>>>>>> Kevin
>>>
>
> ____________
> Virus checked by G Data MailSecurity
> Version: AVA 25.435 dated 26.02.2015
> Virus news: www.antiviruslab.com
>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel