Question about the general performance counter overflow interrupt handling

Question about the general performance counter overflow interrupt handling

[Meng Xu]

Hi Jan and Boris,

I'm Meng Xu from the University of Pennsylvania.

I'm wondering:
How does Xen (vpmu) handle the general performance counter's overflow interrupt?
Could you point me to the function handler, if Xen does handle it?

---What I want to achieve---
I'm looking at the real-time performance in Xen.
I want to profile the system's status for every K L3 cache misses from
a specific core.
I plan to program the general performance counter to -K to trigger an
overflow interrupt. In the interrupt handler, I plan to check the
system's status and give hints to the scheduler.

--- What I have tried ---
I want to find the interrupt handler and plug in another function.
1) I checked Xen's vpmu command option, it does not say vpmu handles
the general performance counter's overflow interrupt.

2) I also added a function inside pmu_apic_interrupt() in apic.c.
However, it seems that the pmu_apic_interrupt() is not triggered when
the general performance counter overflows.

When I program the general performance counter to trigger an overflow
interrupt, I set the following bits for the event selector register
and run a task to generate the L3 cache cache miss.
FLAG_ENABLE: 0x400000UL
FLAG_INT:    0x100000UL
FLAG_USR: 0x010000UL
L3_ALLMISS_EVENT    0x2E
L3_ALLMISS_MESI     0x41

I'm sure the performance counter does overflow, but I didn't see any
interrupt was triggered. Maybe I missed something?

Thank you very much for your help and time!

Best regards,

Meng
-----------
Meng Xu
PhD Candidate in Computer and Information Science
University of Pennsylvania
http://www.cis.upenn.edu/~mengxu/

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Question about the general performance counter overflow interrupt handling

[Meng Xu]

[Sorry, I cc.ed Quan's previous email at Intel. Change to his current email.]

On Fri, Mar 31, 2017 at 11:41 AM, Meng Xu <mengxu@seas.upenn.edu> wrote:
> Hi Jan and Boris,
>
> I'm Meng Xu from the University of Pennsylvania.
>
> I'm wondering:
> How does Xen (vpmu) handle the general performance counter's overflow interrupt?
> Could you point me to the function handler, if Xen does handle it?
>
> ---What I want to achieve---
> I'm looking at the real-time performance in Xen.
> I want to profile the system's status for every K L3 cache misses from
> a specific core.
> I plan to program the general performance counter to -K to trigger an
> overflow interrupt. In the interrupt handler, I plan to check the
> system's status and give hints to the scheduler.
>
> --- What I have tried ---
> I want to find the interrupt handler and plug in another function.
> 1) I checked Xen's vpmu command option, it does not say vpmu handles
> the general performance counter's overflow interrupt.
>
> 2) I also added a function inside pmu_apic_interrupt() in apic.c.
> However, it seems that the pmu_apic_interrupt() is not triggered when
> the general performance counter overflows.
>
> When I program the general performance counter to trigger an overflow
> interrupt, I set the following bits for the event selector register
> and run a task to generate the L3 cache cache miss.
> FLAG_ENABLE: 0x400000UL
> FLAG_INT:    0x100000UL
> FLAG_USR: 0x010000UL
> L3_ALLMISS_EVENT    0x2E
> L3_ALLMISS_MESI     0x41
>
> I'm sure the performance counter does overflow, but I didn't see any
> interrupt was triggered. Maybe I missed something?
>
> Thank you very much for your help and time!
>
> Best regards,
>
> Meng
> -----------
> Meng Xu
> PhD Candidate in Computer and Information Science
> University of Pennsylvania
> http://www.cis.upenn.edu/~mengxu/

-- 
-----------
Meng Xu
PhD Student in Computer and Information Science
University of Pennsylvania
http://www.cis.upenn.edu/~mengxu/

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Question about the general performance counter overflow interrupt handling

[Boris Ostrovsky]

>> When I program the general performance counter to trigger an overflow
>> interrupt, I set the following bits for the event selector register
>> and run a task to generate the L3 cache cache miss.
>> FLAG_ENABLE: 0x400000UL
>> FLAG_INT:    0x100000UL
>> FLAG_USR: 0x010000UL
>> L3_ALLMISS_EVENT    0x2E
>> L3_ALLMISS_MESI     0x41
>>
>> I'm sure the performance counter does overflow, but I didn't see any
>> interrupt was triggered. Maybe I missed something?

Did you program global registrers (MSR_CORE_PERF_GLOBAL_CTRL,
MSR_CORE_PERF_GLOBAL_OVF_CTRL)?

-boris


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Question about the general performance counter overflow interrupt handling

[Meng Xu]

Hi Boris,

On Fri, Mar 31, 2017 at 12:01 PM, Boris Ostrovsky
<boris.ostrovsky@oracle.com> wrote:
>
>>> When I program the general performance counter to trigger an overflow
>>> interrupt, I set the following bits for the event selector register
>>> and run a task to generate the L3 cache cache miss.
>>> FLAG_ENABLE: 0x400000UL
>>> FLAG_INT:    0x100000UL
>>> FLAG_USR: 0x010000UL
>>> L3_ALLMISS_EVENT    0x2E
>>> L3_ALLMISS_MESI     0x41
>>>
>>> I'm sure the performance counter does overflow, but I didn't see any
>>> interrupt was triggered. Maybe I missed something?
>
> Did you program global registrers (MSR_CORE_PERF_GLOBAL_CTRL,
> MSR_CORE_PERF_GLOBAL_OVF_CTRL)?

I tried two scenarios:
Scenario1)
    MSR_CORE_PERF_GLOBAL_CTRL (0x38F) = 0x0
    MSR_CORE_PERF_GLOBAL_OVF_CTRL (0x390) = 0x0
    The function pmu_apic_interrupt() is not called.
Scenario 2)
   MSR_CORE_PERF_GLOBAL_CTRL (0x38F) = 0xff
    MSR_CORE_PERF_GLOBAL_OVF_CTRL (0x390) = 0x0
    The function pmu_apic_interrupt() is not called either.

In both scenarios, the IA32_PERF_GLOBAL_STATUS (0x38E) is 0xf.

I tried to set MSR_CORE_PERF_GLOBAL_OVF_CTRL to 0xf, but the
register's content is not changed. :(

Maybe I should set the MSR_CORE_PERF_GLOBAL_OVF_CTRL to 0xF to enable
the overflow interrupt?

Thank you very much for your time and help!

Meng

-----------
Meng Xu
PhD Candidate in Computer and Information Science
University of Pennsylvania
http://www.cis.upenn.edu/~mengxu/

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Question about the general performance counter overflow interrupt handling

[Boris Ostrovsky]

On 03/31/2017 01:32 PM, Meng Xu wrote:
> Hi Boris,
>
> On Fri, Mar 31, 2017 at 12:01 PM, Boris Ostrovsky
> <boris.ostrovsky@oracle.com> wrote:
>>>> When I program the general performance counter to trigger an overflow
>>>> interrupt, I set the following bits for the event selector register
>>>> and run a task to generate the L3 cache cache miss.
>>>> FLAG_ENABLE: 0x400000UL
>>>> FLAG_INT:    0x100000UL
>>>> FLAG_USR: 0x010000UL
>>>> L3_ALLMISS_EVENT    0x2E
>>>> L3_ALLMISS_MESI     0x41
>>>>
>>>> I'm sure the performance counter does overflow, but I didn't see any
>>>> interrupt was triggered. Maybe I missed something?
>> Did you program global registrers (MSR_CORE_PERF_GLOBAL_CTRL,
>> MSR_CORE_PERF_GLOBAL_OVF_CTRL)?
> I tried two scenarios:
> Scenario1)
>     MSR_CORE_PERF_GLOBAL_CTRL (0x38F) = 0x0
>     MSR_CORE_PERF_GLOBAL_OVF_CTRL (0x390) = 0x0
>     The function pmu_apic_interrupt() is not called.
> Scenario 2)
>    MSR_CORE_PERF_GLOBAL_CTRL (0x38F) = 0xff
>     MSR_CORE_PERF_GLOBAL_OVF_CTRL (0x390) = 0x0
>     The function pmu_apic_interrupt() is not called either.
>
> In both scenarios, the IA32_PERF_GLOBAL_STATUS (0x38E) is 0xf.
>
> I tried to set MSR_CORE_PERF_GLOBAL_OVF_CTRL to 0xf, but the
> register's content is not changed. :(

It's a write-only register. You read MSR_CORE_PERF_GLOBAL_STATUS. You
may need to set upper bits as well (I don't remember for sure).

You can try running, for example, perf in Linux while executing
xentrace, recording MSR accesses. Or just instrument
core2_vpmu_do_rdmsr/core2_vpmu_do_wrmsr and see what's coming in/out.

-boris


>
> Maybe I should set the MSR_CORE_PERF_GLOBAL_OVF_CTRL to 0xF to enable
> the overflow interrupt?
>
> Thank you very much for your time and help!
>
> Meng
>
> -----------
> Meng Xu
> PhD Candidate in Computer and Information Science
> University of Pennsylvania
> http://www.cis.upenn.edu/~mengxu/


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Question about the general performance counter overflow interrupt handling

[Jan Beulich]

>>> On 31.03.17 at 17:41, <mengxu@seas.upenn.edu> wrote:
> I'm wondering:
> How does Xen (vpmu) handle the general performance counter's overflow 
> interrupt?
> Could you point me to the function handler, if Xen does handle it?

Two simple steps take you there: grep for LVTPC to find which vector
is being used (PMU_APIC_VECTOR) and then grep for that vector
name to find what function's address gets put in the IDT
(pmu_apic_interrupt).

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel