Re: Tracing KVM with Systemtap

Stefan Hajnoczi
2010-09-08T14:08:43Z

On Wed, Sep 8, 2010 at 2:20 PM, Rayson Ho <rho@redhat.com> wrote:
> Hi all,
>
> I am a developer of Systemtap. I am looking into tracing KVM (the kernel
> part and QEMU) and also the KVM guests with Systemtap. I googled and
> found references to Xenprobes and xdt+dtrace, and I was wondering if
> someone is working on the dynamic tracing interface for KVM?
>
> I've read the KVM kernel code and I think some expensive operations
> (things that need to be trapped back to the host kernel - eg. loading of
> control registers on x86/x64) can be interesting spots for adding an SDT
> (static marker), and I/O operations performed for the guests can be
> useful information to collect.
>
> I know that KVM guests run like a userspace process and thus techniques
> for tracing Xen might be overkilled, and also gdb can be used to trace
> KVM guests. However, are that anything special I need to be aware of
> before I go further into the development of the Systemtap KVM probes?
>
> (Opinions / Suggestions / Criticisms welcome!)

Hi Rayson,
For the KVM kernel module Linux trace events are already used. For
example, see arch/x86/kvm/trace.h and check out
/sys/kernel/debug/tracing/events/kvm/*. There is a set of useful
static trace points for vm_exit/vm_enter, pio, mmio, etc.

For the KVM guest there is perf-kvm(1). This allows perf(1) to look
up addresses inside the guest (kernel only?). It produces system-wide
performance profiles including guests. Perhaps someone can comment on
perf-kvm's full feature set and limitations?

For QEMU userspace Prerna Saxena and I are proposing a static tracing
patchset. It abstracts the trace backend (SystemTap, LTTng UST,
DTrace, etc) from the actual tracepoints so that portability can be
achieved. There is a built-in trace backend that has a basic feature
set but isn't as fancy as SystemTap. I have implemented LTTng
Userspace Tracer support, perhaps you'd like to add SystemTap/DTrace
support with sdt.h?

http://www.mail-archive.com/qemu-devel@nongnu.org/msg41323.html
http://repo.or.cz/w/qemu/stefanha.git/shortlog/refs/heads/tracing_v3

Stefan