perf kvm segfaults while reporting events

perf kvm segfaults while reporting events

[Pradeep Kumar Surisetty]

Hello 

perf kvm generats segfaults while reporting events. 

host & guest:  3.4.0-rc4

1) Copied kallsyms, modules from guest to host. 

2) recorded  events on host and guest & report it using below commands. 


[root@host]# ./perf kvm --host --guest --guestkallsyms=/tmp/guest-kallsyms --guestmodules=/tmp/guest-modules record -a -o perf.data
^C[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.381 MB perf.data (~16659 samples) ]

[root@host]# ./perf kvm --host --guest --guestkallsyms=/tmp/guest-kallsyms --guestmodules=/tmp/guest-modules report -i perf.data --force
perf: Segmentation fault
[root@host]# 

--Pradeep

Re: perf kvm segfaults while reporting events

[David Ahern]

On 6/1/12 7:56 AM, Pradeep Kumar Surisetty wrote:
> Hello
>
> perf kvm generats segfaults while reporting events.
>
> host&  guest:  3.4.0-rc4
>
> 1) Copied kallsyms, modules from guest to host.
>
> 2) recorded  events on host and guest&  report it using below commands.
>
>
> [root@host]# ./perf kvm --host --guest --guestkallsyms=/tmp/guest-kallsyms --guestmodules=/tmp/guest-modules record -a -o perf.data
> ^C[ perf record: Woken up 1 times to write data ]
> [ perf record: Captured and wrote 0.381 MB perf.data (~16659 samples) ]
>
> [root@host]# ./perf kvm --host --guest --guestkallsyms=/tmp/guest-kallsyms --guestmodules=/tmp/guest-modules report -i perf.data --force
> perf: Segmentation fault

The short of it is that perf-kvm needs some work.

More specifically, the guest 'machine' is not created on the report path 
for the above arguments. This will prevent the segfault:

diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
index 93d355d..c8e1323 100644
--- a/tools/perf/util/session.c
+++ b/tools/perf/util/session.c
@@ -918,6 +918,8 @@ static int perf_session_deliver_event(struct 
perf_session *session,
         }
         return tool->sample(tool, event, sample, evsel, machine);
     case PERF_RECORD_MMAP:
+       if (machine == NULL)
+           return 0;
         return tool->mmap(tool, event, sample, machine);
     case PERF_RECORD_COMM:
         return tool->comm(tool, event, sample, machine);


The --guestmount option (which has its own problems) combined with 
specific pid is more stable . e.g.,

mkdir /tmp/guestmount/<vm-pid>
sshfs root@<vm-ip>:/  /tmp/guestmount/<vm-pid> -o direct_io

perf kvm --host --guest --guestmount=/tmp/guestmount \
   record -p <vm-pid>  -o /tmp/perf.data -- sleep 1

perf kvm --host --guest --guestmount=/tmp/guestmount  report -i 
/tmp/perf.data --force --stdio

If you use the -a argument on record with multiple VMs running without 
an entry in /tmp/guestmount you'll see:

Warning:
7 unprocessable samples recorded.
Do you have a KVM guest running and not using 'perf kvm'?

David

Re: perf kvm segfaults while reporting events

[Pradeep Kumar Surisetty]

* David Ahern <dsahern@gmail.com> [2012-06-01 09:40:30]:

> On 6/1/12 7:56 AM, Pradeep Kumar Surisetty wrote:
> >Hello
> >
> >perf kvm generats segfaults while reporting events.
> >
> >host&  guest:  3.4.0-rc4
> >
> >1) Copied kallsyms, modules from guest to host.
> >
> >2) recorded  events on host and guest&  report it using below commands.
> >
> >
> >[root@host]# ./perf kvm --host --guest --guestkallsyms=/tmp/guest-kallsyms --guestmodules=/tmp/guest-modules record -a -o perf.data
> >^C[ perf record: Woken up 1 times to write data ]
> >[ perf record: Captured and wrote 0.381 MB perf.data (~16659 samples) ]
> >
> >[root@host]# ./perf kvm --host --guest --guestkallsyms=/tmp/guest-kallsyms --guestmodules=/tmp/guest-modules report -i perf.data --force
> >perf: Segmentation fault
> 
> The short of it is that perf-kvm needs some work.
> 
> More specifically, the guest 'machine' is not created on the report
> path for the above arguments. This will prevent the segfault:
> 
> diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
> index 93d355d..c8e1323 100644
> --- a/tools/perf/util/session.c
> +++ b/tools/perf/util/session.c
> @@ -918,6 +918,8 @@ static int perf_session_deliver_event(struct
> perf_session *session,
>         }
>         return tool->sample(tool, event, sample, evsel, machine);
>     case PERF_RECORD_MMAP:
> +       if (machine == NULL)
> +           return 0;


Ahhh perf kvm works fine if we record guest events for enough time. 
above patch works better, if some one doesnt record for adequate time. 


>         return tool->mmap(tool, event, sample, machine);
>     case PERF_RECORD_COMM:
>         return tool->comm(tool, event, sample, machine);
> 
> 
> The --guestmount option (which has its own problems) combined with
> specific pid is more stable . e.g.,
> 
> mkdir /tmp/guestmount/<vm-pid>
> sshfs root@<vm-ip>:/  /tmp/guestmount/<vm-pid> -o direct_io
> 
> perf kvm --host --guest --guestmount=/tmp/guestmount \
>   record -p <vm-pid>  -o /tmp/perf.data -- sleep 1
> 
> perf kvm --host --guest --guestmount=/tmp/guestmount  report -i
> /tmp/perf.data --force --stdio
> 
> If you use the -a argument on record with multiple VMs running
> without an entry in /tmp/guestmount you'll see:
> 
> Warning:
> 7 unprocessable samples recorded.
> Do you have a KVM guest running and not using 'perf kvm'?
> 
> David
> 

Re: perf kvm segfaults while reporting events

[David Ahern]

On 6/20/12 12:14 AM, Pradeep Kumar Surisetty wrote:
>
> Ahhh perf kvm works fine if we record guest events for enough time.
> above patch works better, if some one doesnt record for adequate time.

Dumb luck. The extra time allows a sample that hits within a guest 
module and causes the build id for it to be added to the header.

I added some pr_debug's to the synthesize_modules. e.g.,

perf_event__synthesize_modules: host modules, pid -1, file 
/lib/modules/3.4.0/kernel/arch/x86/crypto/crc32c-intel.ko

perf_event__synthesize_modules: guest modules, pid 0, file 
/lib/modules/3.4.0/kernel/drivers/block/virtio_blk.ko

which is just wrong for the guest. It's a Fedora 17 VM where the kernel 
version is 3.4.0-1.fc17.x86_64 and even more important perf is not told 
where to find the modules (dso name in the event), so the dso path is 
based on the host OS.

Which comes back to my earlier comment: the 'default' guest options 
(--guestmodules here) for perf-kvm are busted. You really need to use 
the --guestmount option until perf-kvm gets some rework.

I will submit a patch to at least keep perf from segfaulting.

David