perf CPU profiling in Xen guest fail & workaround

perf CPU profiling in Xen guest fail & workaround

[Brendan Gregg]

G'Day perf users,

I just debugged something and wanted to share in case others hit
this... perf CPU profiling didn't work for this Linux 4.1 guest (AWS
EC2):

# ./perf record -F 99 -a -g -- sleep 5
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.300 MB perf.data ]
# ./perf script
#

There are no CPU samples recorded (perf.data just has
PERF_RECORD_MMAP, etc). I believe it's because it's using the "cycles"
PMC event, and not "cpu-clock". And cycles is bogus:

# ./perf stat -e cycles -a sleep 1 2>&1 | grep cycles
    68,719,476,704      cycles
# ./perf stat -e cycles -a sleep 2 2>&1 | grep cycles
    68,719,476,704      cycles

I can fix it by profiling cpu-clock:

# ./perf record -e cpu-clock -a -g -- sleep 5
[ perf record: Woken up 27 times to write data ]
[ perf record: Captured and wrote 9.884 MB perf.data (105989 samples) ]
# ./perf script
[...data...]

Which is what I think all my other systems are doing anyway; some perf
-v -v output from those other systems:

[...]
sys_perf_event_open: pid -1  cpu 0  group_fd -1  flags 0x8
sys_perf_event_open failed, error -2
Warning:
The cycles event is not supported, trying to fall back to cpu-clock-ticks
[...]

This 4.1 system has these:

# cat /sys/devices/cpu/events/cpu-cycles
event=0x3c

Which other systems don't. Profiling these (eg, -e r003c) also doesn't work.

So I can use "-e cpu-clock", which is fine. It even lets me set the
profile frequency:

# ./perf record -F 99 -e cpu-clock -a -- sleep 1

Maybe this issue is already known and fixed in latest. I couldn't find
anything in a search...

Brendan

Re: perf CPU profiling in Xen guest fail & workaround

[David Ahern]

On 7/29/15 1:04 PM, Brendan Gregg wrote:
> G'Day perf users,
>
> I just debugged something and wanted to share in case others hit
> this... perf CPU profiling didn't work for this Linux 4.1 guest (AWS
> EC2):
>
> # ./perf record -F 99 -a -g -- sleep 5
> [ perf record: Woken up 1 times to write data ]
> [ perf record: Captured and wrote 0.300 MB perf.data ]
> # ./perf script
> #
>
> There are no CPU samples recorded (perf.data just has
> PERF_RECORD_MMAP, etc). I believe it's because it's using the "cycles"
> PMC event, and not "cpu-clock". And cycles is bogus:
>
> # ./perf stat -e cycles -a sleep 1 2>&1 | grep cycles
>      68,719,476,704      cycles
> # ./perf stat -e cycles -a sleep 2 2>&1 | grep cycles
>      68,719,476,704      cycles
>
> I can fix it by profiling cpu-clock:
>
> # ./perf record -e cpu-clock -a -g -- sleep 5
> [ perf record: Woken up 27 times to write data ]
> [ perf record: Captured and wrote 9.884 MB perf.data (105989 samples) ]
> # ./perf script
> [...data...]
>
> Which is what I think all my other systems are doing anyway; some perf
> -v -v output from those other systems:
>
> [...]
> sys_perf_event_open: pid -1  cpu 0  group_fd -1  flags 0x8
> sys_perf_event_open failed, error -2
> Warning:
> The cycles event is not supported, trying to fall back to cpu-clock-ticks
> [...]
>
> This 4.1 system has these:
>
> # cat /sys/devices/cpu/events/cpu-cycles
> event=0x3c
>
> Which other systems don't. Profiling these (eg, -e r003c) also doesn't work.
>
> So I can use "-e cpu-clock", which is fine. It even lets me set the
> profile frequency:
>
> # ./perf record -F 99 -e cpu-clock -a -- sleep 1
>
> Maybe this issue is already known and fixed in latest. I couldn't find
> anything in a search...
>

It's a known problem with VMs (Solaris based ldoms and apparently xen 
guests as examples). The hypervisor tells perf it has a PMU (see dmesg | 
grep Performance) but does not really implement it (no NMIs for example).

David

Re: perf CPU profiling in Xen guest fail & workaround

[Brendan Gregg]

Thanks, yes. This took me by surprise: I've been profiling different
Xen guests for over a year, but this is the first time I've hit this.

All my other Xen guests:

# dmesg | grep Performance
[    0.058664] Performance Events: unsupported p6 CPU model 62 no PMU
driver, software events only.

This linux 4.1 Xen guest:

# dmesg | grep Performance
[    0.258140] Performance Events: 16-deep LBR, IvyBridge events,
Intel PMU driver.

Brendan

On Wed, Jul 29, 2015 at 12:15 PM, David Ahern <dsahern@gmail.com> wrote:
> On 7/29/15 1:04 PM, Brendan Gregg wrote:
>>
>> G'Day perf users,
>>
>> I just debugged something and wanted to share in case others hit
>> this... perf CPU profiling didn't work for this Linux 4.1 guest (AWS
>> EC2):
>>
>> # ./perf record -F 99 -a -g -- sleep 5
>> [ perf record: Woken up 1 times to write data ]
>> [ perf record: Captured and wrote 0.300 MB perf.data ]
>> # ./perf script
>> #
>>
>> There are no CPU samples recorded (perf.data just has
>> PERF_RECORD_MMAP, etc). I believe it's because it's using the "cycles"
>> PMC event, and not "cpu-clock". And cycles is bogus:
>>
>> # ./perf stat -e cycles -a sleep 1 2>&1 | grep cycles
>>      68,719,476,704      cycles
>> # ./perf stat -e cycles -a sleep 2 2>&1 | grep cycles
>>      68,719,476,704      cycles
>>
>> I can fix it by profiling cpu-clock:
>>
>> # ./perf record -e cpu-clock -a -g -- sleep 5
>> [ perf record: Woken up 27 times to write data ]
>> [ perf record: Captured and wrote 9.884 MB perf.data (105989 samples) ]
>> # ./perf script
>> [...data...]
>>
>> Which is what I think all my other systems are doing anyway; some perf
>> -v -v output from those other systems:
>>
>> [...]
>> sys_perf_event_open: pid -1  cpu 0  group_fd -1  flags 0x8
>> sys_perf_event_open failed, error -2
>> Warning:
>> The cycles event is not supported, trying to fall back to cpu-clock-ticks
>> [...]
>>
>> This 4.1 system has these:
>>
>> # cat /sys/devices/cpu/events/cpu-cycles
>> event=0x3c
>>
>> Which other systems don't. Profiling these (eg, -e r003c) also doesn't
>> work.
>>
>> So I can use "-e cpu-clock", which is fine. It even lets me set the
>> profile frequency:
>>
>> # ./perf record -F 99 -e cpu-clock -a -- sleep 1
>>
>> Maybe this issue is already known and fixed in latest. I couldn't find
>> anything in a search...
>>
>
> It's a known problem with VMs (Solaris based ldoms and apparently xen guests
> as examples). The hypervisor tells perf it has a PMU (see dmesg | grep
> Performance) but does not really implement it (no NMIs for example).
>
> David
>

Re: perf CPU profiling in Xen guest fail & workaround

[David Ahern]

On 7/29/15 1:27 PM, Brendan Gregg wrote:
> Thanks, yes. This took me by surprise: I've been profiling different
> Xen guests for over a year, but this is the first time I've hit this.
>
> All my other Xen guests:
>
> # dmesg | grep Performance
> [    0.058664] Performance Events: unsupported p6 CPU model 62 no PMU
> driver, software events only.
>
> This linux 4.1 Xen guest:
>
> # dmesg | grep Performance
> [    0.258140] Performance Events: 16-deep LBR, IvyBridge events,
> Intel PMU driver.


yep and since the hypervisor lies perf has no means of knowing to 
fallback to cpu-clock.

Re: perf CPU profiling in Xen guest fail & workaround

[Andi Kleen]

David Ahern <dsahern@gmail.com> writes:
>
> It's a known problem with VMs (Solaris based ldoms and apparently xen
> guests as examples). The hypervisor tells perf it has a PMU (see dmesg
> | grep Performance) but does not really implement it (no NMIs for
> example).

Actually Xen has a PMU model. I've never tested it, but from a quick
source review some time ago it seemed reasonable for Arch Perfmon v1,
when disabled.

If something doesn't work you should report it to the Xen people

BTW sampling works with KVM's PMU model; I'm using that occasionally.

-Andi

-- 
ak@linux.intel.com -- Speaking for myself only