Tracing KVM with Systemtap

Tracing KVM with Systemtap

[Rayson Ho]

Hi all,

I am a developer of Systemtap. I am looking into tracing KVM (the kernel
part and QEMU) and also the KVM guests with Systemtap. I googled and
found references to Xenprobes and xdt+dtrace, and I was wondering if
someone is working on the dynamic tracing interface for KVM?

I've read the KVM kernel code and I think some expensive operations
(things that need to be trapped back to the host kernel - eg. loading of
control registers on x86/x64) can be interesting spots for adding an SDT
(static marker), and I/O operations performed for the guests can be
useful information to collect.

I know that KVM guests run like a userspace process and thus techniques
for tracing Xen might be overkilled, and also gdb can be used to trace
KVM guests. However, are that anything special I need to be aware of
before I go further into the development of the Systemtap KVM probes?

(Opinions / Suggestions / Criticisms welcome!)

Thanks,
Rayson

Re: Tracing KVM with Systemtap

[Stefan Hajnoczi]

On Wed, Sep 8, 2010 at 2:20 PM, Rayson Ho <rho@redhat.com> wrote:
> Hi all,
>
> I am a developer of Systemtap. I am looking into tracing KVM (the kernel
> part and QEMU) and also the KVM guests with Systemtap. I googled and
> found references to Xenprobes and xdt+dtrace, and I was wondering if
> someone is working on the dynamic tracing interface for KVM?
>
> I've read the KVM kernel code and I think some expensive operations
> (things that need to be trapped back to the host kernel - eg. loading of
> control registers on x86/x64) can be interesting spots for adding an SDT
> (static marker), and I/O operations performed for the guests can be
> useful information to collect.
>
> I know that KVM guests run like a userspace process and thus techniques
> for tracing Xen might be overkilled, and also gdb can be used to trace
> KVM guests. However, are that anything special I need to be aware of
> before I go further into the development of the Systemtap KVM probes?
>
> (Opinions / Suggestions / Criticisms welcome!)

Hi Rayson,
For the KVM kernel module Linux trace events are already used.  For
example, see arch/x86/kvm/trace.h and check out
/sys/kernel/debug/tracing/events/kvm/*.  There is a set of useful
static trace points for vm_exit/vm_enter, pio, mmio, etc.

For the KVM guest there is perf-kvm(1).  This allows perf(1) to look
up addresses inside the guest (kernel only?).  It produces system-wide
performance profiles including guests.  Perhaps someone can comment on
perf-kvm's full feature set and limitations?

For QEMU userspace Prerna Saxena and I are proposing a static tracing
patchset.  It abstracts the trace backend (SystemTap, LTTng UST,
DTrace, etc) from the actual tracepoints so that portability can be
achieved.  There is a built-in trace backend that has a basic feature
set but isn't as fancy as SystemTap.  I have implemented LTTng
Userspace Tracer support, perhaps you'd like to add SystemTap/DTrace
support with sdt.h?

http://www.mail-archive.com/qemu-devel@nongnu.org/msg41323.html
http://repo.or.cz/w/qemu/stefanha.git/shortlog/refs/heads/tracing_v3

Stefan

Re: Tracing KVM with Systemtap

[Rayson Ho]

On Wed, 2010-09-08 at 15:08 +0100, Stefan Hajnoczi wrote:
> Hi Rayson,
> For the KVM kernel module Linux trace events are already used.  For
> example, see arch/x86/kvm/trace.h and check out
> /sys/kernel/debug/tracing/events/kvm/*.  There is a set of useful
> static trace points for vm_exit/vm_enter, pio, mmio, etc.
> 
> For the KVM guest there is perf-kvm(1).  This allows perf(1) to look
> up addresses inside the guest (kernel only?).  It produces system-wide
> performance profiles including guests.  Perhaps someone can comment on
> perf-kvm's full feature set and limitations?
> 
> For QEMU userspace Prerna Saxena and I are proposing a static tracing
> patchset.  It abstracts the trace backend (SystemTap, LTTng UST,
> DTrace, etc) from the actual tracepoints so that portability can be
> achieved.  There is a built-in trace backend that has a basic feature
> set but isn't as fancy as SystemTap.  I have implemented LTTng
> Userspace Tracer support, perhaps you'd like to add SystemTap/DTrace
> support with sdt.h?

Thanks Stefan for the reply!

I've looked at the tracing additions in QEMU, including the "Simple
trace backend" (simpletrace.c) and the "tracetool" script, and I think
the SystemTap version can be implemented in a straightforward way.

One thing I was wondering, there seems to be not a lot of probes (except
the examples?) in the QEMU code, are we expected to see more probes in
the next release, or this work will be a long-term project that will not
be added to the official QEMU code in the near future?

(I believe if we can get the tracing framework integrated, then specific
probes can be added on-demand -- but of course that is just my own
opinion :-D )

Thanks,
Rayson




> 
> http://www.mail-archive.com/qemu-devel@nongnu.org/msg41323.html
> http://repo.or.cz/w/qemu/stefanha.git/shortlog/refs/heads/tracing_v3
> 
> Stefan
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Tracing KVM with Systemtap

[Stefan Hajnoczi]

On Mon, Sep 20, 2010 at 2:19 PM, Rayson Ho <rho@redhat.com> wrote:
> On Wed, 2010-09-08 at 15:08 +0100, Stefan Hajnoczi wrote:
>> Hi Rayson,
>> For the KVM kernel module Linux trace events are already used.  For
>> example, see arch/x86/kvm/trace.h and check out
>> /sys/kernel/debug/tracing/events/kvm/*.  There is a set of useful
>> static trace points for vm_exit/vm_enter, pio, mmio, etc.
>>
>> For the KVM guest there is perf-kvm(1).  This allows perf(1) to look
>> up addresses inside the guest (kernel only?).  It produces system-wide
>> performance profiles including guests.  Perhaps someone can comment on
>> perf-kvm's full feature set and limitations?
>>
>> For QEMU userspace Prerna Saxena and I are proposing a static tracing
>> patchset.  It abstracts the trace backend (SystemTap, LTTng UST,
>> DTrace, etc) from the actual tracepoints so that portability can be
>> achieved.  There is a built-in trace backend that has a basic feature
>> set but isn't as fancy as SystemTap.  I have implemented LTTng
>> Userspace Tracer support, perhaps you'd like to add SystemTap/DTrace
>> support with sdt.h?
>
> Thanks Stefan for the reply!
>
> I've looked at the tracing additions in QEMU, including the "Simple
> trace backend" (simpletrace.c) and the "tracetool" script, and I think
> the SystemTap version can be implemented in a straightforward way.
>
> One thing I was wondering, there seems to be not a lot of probes (except
> the examples?) in the QEMU code, are we expected to see more probes in
> the next release, or this work will be a long-term project that will not
> be added to the official QEMU code in the near future?
>
> (I believe if we can get the tracing framework integrated, then specific
> probes can be added on-demand -- but of course that is just my own
> opinion :-D )

Right now there are few pre-defined probes ("trace events" in QEMU
tracing speak).  As I develop I try to be mindful of new ones I create
and whether they would be generally useful.  I intend to contribute
more probes and hope others will too!

Prerna is also looking at adding useful probes.

Stefan

Re: Tracing KVM with Systemtap

[Rayson Ho]

On Mon, 2010-09-20 at 14:36 +0100, Stefan Hajnoczi wrote:
> Right now there are few pre-defined probes ("trace events" in QEMU
> tracing speak).  As I develop I try to be mindful of new ones I create
> and whether they would be generally useful.  I intend to contribute
> more probes and hope others will too!

I am still looking at/hacking the QEMU code. I have looked at the
following places in the code that I think can be useful to have
statistics gathered:

net.c qemu_deliver_packet(), etc - network statistics

<CPU Arch>/op_helper.c global_cpu_lock(), tlb_fill() - lock & unlock,
and TLB refill statistics

balloon.c, hw/virtio-balloon.c - ballooning information.

Besides the ballooning part, which I know what it is but don't fully
understand how it works, the other parts can be implemented as Systemtap
tapsets (~ "DTrace scripts") in the initial stage.

I will see what other probes are useful for the end users. Also, are
there developer documentations for KVM? (I googled but found a lot of
presentations about KVM but not a lot of info about the internals.)

Rayson



> 
> Prerna is also looking at adding useful probes.
> 
> Stefan

Re: Tracing KVM with Systemtap

[Stefan Hajnoczi]

On Tue, Sep 21, 2010 at 1:58 PM, Rayson Ho <rho@redhat.com> wrote:
> On Mon, 2010-09-20 at 14:36 +0100, Stefan Hajnoczi wrote:
>> Right now there are few pre-defined probes ("trace events" in QEMU
>> tracing speak).  As I develop I try to be mindful of new ones I create
>> and whether they would be generally useful.  I intend to contribute
>> more probes and hope others will too!
>
> I am still looking at/hacking the QEMU code. I have looked at the
> following places in the code that I think can be useful to have
> statistics gathered:
>
> net.c qemu_deliver_packet(), etc - network statistics

Yes.

> <CPU Arch>/op_helper.c global_cpu_lock(), tlb_fill() - lock & unlock,
> and TLB refill statistics

These are not relevant to KVM, they are only used when running with
KVM disabled (TCG mode).

> balloon.c, hw/virtio-balloon.c - ballooning information.

Prerna added a balloon event which is in qemu.git trace-events.  Does
that one do what you need?

> I will see what other probes are useful for the end users. Also, are
> there developer documentations for KVM? (I googled but found a lot of
> presentations about KVM but not a lot of info about the internals.)

Not really.  I suggest grabbing the source and following vl.c:main()
to the main KVM execution code.

Stefan

Re: Tracing KVM with Systemtap

[Rayson Ho]

On Tue, 2010-09-21 at 14:33 +0100, Stefan Hajnoczi wrote:
> > net.c qemu_deliver_packet(), etc - network statistics
> 
> Yes.

Further digging into the code in the net/ directory, there are more
functions that should be traced/probed - e.g. a lot of the net_socket*
routines.

Given that SystemTap, LTTng, and DTrace have extremely close to zero
overhead at each probe point, adding them into code can provide users
different performance resolutions into the networking code.


> > <CPU Arch>/op_helper.c global_cpu_lock(), tlb_fill() - lock & unlock,
> > and TLB refill statistics
> 
> These are not relevant to KVM, they are only used when running with
> KVM disabled (TCG mode).

Hmm, while my main objective is to add probes for KVM, if they are
useful to QEMU, I may as well add them to my patch.


> > balloon.c, hw/virtio-balloon.c - ballooning information.
> 
> Prerna added a balloon event which is in qemu.git trace-events.  Does
> that one do what you need?

Thanks for the info.


> > I will see what other probes are useful for the end users. Also, are
> > there developer documentations for KVM? (I googled but found a lot of
> > presentations about KVM but not a lot of info about the internals.)
> 
> Not really.  I suggest grabbing the source and following vl.c:main()
> to the main KVM execution code.

I was looking for the hardware interfacing code earlier this morning --
QEMU has the hardware specific directories (e.g. target-i386/ ,
target-ppc/ ), and I was trying to understand the execution environment
when the host and guest are running on the same architecture.

I believe cpu_gen_code() and other related functions are what I should
dig into...

Rayson




> 
> Stefan
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Tracing KVM with Systemtap

[Stefan Hajnoczi]

On Wed, Sep 22, 2010 at 1:11 PM, Rayson Ho <rho@redhat.com> wrote:
> On Tue, 2010-09-21 at 14:33 +0100, Stefan Hajnoczi wrote:
>> > I will see what other probes are useful for the end users. Also, are
>> > there developer documentations for KVM? (I googled but found a lot of
>> > presentations about KVM but not a lot of info about the internals.)
>>
>> Not really.  I suggest grabbing the source and following vl.c:main()
>> to the main KVM execution code.
>
> I was looking for the hardware interfacing code earlier this morning --
> QEMU has the hardware specific directories (e.g. target-i386/ ,
> target-ppc/ ), and I was trying to understand the execution environment
> when the host and guest are running on the same architecture.
>
> I believe cpu_gen_code() and other related functions are what I should
> dig into...

KVM does not generate code.  Almost all the "emulation" code in the
source tree is part of the Tiny Code Generator (TCG) used when KVM is
not enabled (e.g. to emulate an ARM board on an x86-64 host).

If you follow the life-cycle in vl.c it will take you through cpus.c
and into kvm-all.c:kvm_cpu_exec().  Note that the details differ
slightly between qemu.git and qemu-kvm.git, and I have described
qemu.git.

Stefan

Re: Tracing KVM with Systemtap

[Rayson Ho]

On Wed, 2010-09-22 at 13:33 +0100, Stefan Hajnoczi wrote:
> KVM does not generate code.  Almost all the "emulation" code in the
> source tree is part of the Tiny Code Generator (TCG) used when KVM is
> not enabled (e.g. to emulate an ARM board on an x86-64 host).

Thanks, that's what I thought too. Otherwise it would be really slow to
run KVM :)

But if KVM is not used, and QEMU host & guest are running on the same
architecture, is TCG off? (Hmm, I guess I can find that answer myself by
reading the code).

Stefan, are you accepting patches? If so, I will create a patch with the
Systemtap framework & other probes.

Rayson



> 
> If you follow the life-cycle in vl.c it will take you through cpus.c
> and into kvm-all.c:kvm_cpu_exec().  Note that the details differ
> slightly between qemu.git and qemu-kvm.git, and I have described
> qemu.git.
> 
> Stefan
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Tracing KVM with Systemtap

[Stefan Hajnoczi]

On Wed, Sep 22, 2010 at 1:42 PM, Rayson Ho <rho@redhat.com> wrote:
> On Wed, 2010-09-22 at 13:33 +0100, Stefan Hajnoczi wrote:
>> KVM does not generate code.  Almost all the "emulation" code in the
>> source tree is part of the Tiny Code Generator (TCG) used when KVM is
>> not enabled (e.g. to emulate an ARM board on an x86-64 host).
>
> Thanks, that's what I thought too. Otherwise it would be really slow to
> run KVM :)
>
> But if KVM is not used, and QEMU host & guest are running on the same
> architecture, is TCG off? (Hmm, I guess I can find that answer myself by
> reading the code).

TCG is unused when KVM is enabled.  There has been discussion about
building without it for KVM-only builds and qemu-kvm.git can do that
today with a ./configure option.

> Stefan, are you accepting patches? If so, I will create a patch with the
> Systemtap framework & other probes.

I am not a qemu.git or qemu-kvm.git committer but I review patches in
areas that I work in, like tracing.  I'll be happy to give you
feedback.

Stefan

[Qemu-devel] QEMU network probes

[Rayson Ho]

Please review the probe addition in QEMU for the network module, note
that I added probes for send & receive Ethernet frames for rtl8139 so
far but not other network devices. Also for receive, the probe would
only fire when a frame is successfully received, so any errors are
ignored.

=========================================================================

diff --git a/hw/rtl8139.c b/hw/rtl8139.c
index a22530c..df04f59 100644
--- a/hw/rtl8139.c
+++ b/hw/rtl8139.c
@@ -53,6 +53,7 @@
 #include "net.h"
 #include "loader.h"
 #include "sysemu.h"
+#include "trace.h"
 
 /* debug RTL8139 card */
 //#define DEBUG_RTL8139 1
@@ -1160,6 +1161,8 @@ static ssize_t rtl8139_do_receive(VLANClientState *nc, const uint8_t *buf, size_
         rtl8139_update_irq(s);
     }
 
+    trace_rtl8139_do_receive(nc, buf, size_, do_interrupt);
+
     return size_;
 }
 
@@ -1742,6 +1745,8 @@ static uint32_t rtl8139_RxConfig_read(RTL8139State *s)
 
 static void rtl8139_transfer_frame(RTL8139State *s, const uint8_t *buf, int size, int do_interrupt)
 {
+    trace_rtl8139_transfer_frame(s, buf, size, do_interrupt);
+    
     if (!size)
     {
         DEBUG_PRINT(("RTL8139: +++ empty ethernet frame\n"));
diff --git a/net.c b/net.c
index 9ba5be2..ba74f74 100644
--- a/net.c
+++ b/net.c
@@ -36,6 +36,7 @@
 #include "qemu-common.h"
 #include "qemu_socket.h"
 #include "hw/qdev.h"
+#include "trace.h"
 
 static QTAILQ_HEAD(, VLANState) vlans;
 static QTAILQ_HEAD(, VLANClientState) non_vlan_clients;
@@ -559,6 +560,7 @@ ssize_t qemu_send_packet_async(VLANClientState *sender,
 
 void qemu_send_packet(VLANClientState *vc, const uint8_t *buf, int size)
 {
+    trace_qemu_send_packet(vc, buf, size);
     qemu_send_packet_async(vc, buf, size, NULL);
 }
 
diff --git a/trace-events b/trace-events
index e6138ea..76a8431 100644
--- a/trace-events
+++ b/trace-events
@@ -254,3 +254,10 @@ disable spice_vmc_write(ssize_t out, int len) "spice wrottn %lu of requested %zd
 disable spice_vmc_read(int bytes, int len) "spice read %lu of requested %zd"
 disable spice_vmc_register_interface(void *scd) "spice vmc registered interface %p"
 disable spice_vmc_unregister_interface(void *scd) "spice vmc unregistered interface %p"
+
+# net.c
+disable qemu_send_packet(void *vc, const uint8_t *buf, int size) "client-state: %p, buf %p size %d"
+
+#hw/rtl8139.c
+disable rtl8139_transfer_frame(void *s, const uint8_t *buf, int size, int do_interrupt) "RTL8139State %p, buf %p size %d do_interrupt %d"
+disable rtl8139_do_receive(void *s, const uint8_t *buf, int size, int do_interrupt) "client-state: %p, buf %p size %zu do_interrupt %d"

=========================================================================

Signed-off-by: Rayson Ho <rho@redhat.com>

Rayson



On Wed, 2010-11-24 at 20:56 +0000, Stefan Hajnoczi wrote:
> On Wed, Nov 24, 2010 at 7:22 PM, Rayson Ho <rho@redhat.com> wrote:
> > I added a probe point for a conference demo, are you interested in
> > adding it in?
> >  void qemu_send_packet(VLANClientState *vc, const uint8_t *buf, int
> > size)
> >  {
> > +     trace_qemu_send_packet(buf, size);
> >      qemu_send_packet_async(vc, buf, size, NULL);
> >  }
> 
> Cool, I recently talked to Prerna Saxena about improving tracing
> coverage.  The net subsystem is one of the areas where we need to add
> trace events.
> 
> Please include the VLANClientState *vc pointer in the trace so sent
> packets can be correlated to their network clients.
> 
> Want to send a patch to qemu-devel?
> 
> Stefan

Re: [Qemu-devel] QEMU network probes

[Stefan Hajnoczi]

On Wed, Feb 9, 2011 at 7:27 PM, Rayson Ho <rho@redhat.com> wrote:
> Please review the probe addition in QEMU for the network module, note
> that I added probes for send & receive Ethernet frames for rtl8139 so
> far but not other network devices. Also for receive, the probe would
> only fire when a frame is successfully received, so any errors are
> ignored.

I don't know the network subsystem well so I hope others can comment on that.

> =========================================================================
>
> diff --git a/hw/rtl8139.c b/hw/rtl8139.c
> index a22530c..df04f59 100644
> --- a/hw/rtl8139.c
> +++ b/hw/rtl8139.c
> @@ -53,6 +53,7 @@
>  #include "net.h"
>  #include "loader.h"
>  #include "sysemu.h"
> +#include "trace.h"
>
>  /* debug RTL8139 card */
>  //#define DEBUG_RTL8139 1
> @@ -1160,6 +1161,8 @@ static ssize_t rtl8139_do_receive(VLANClientState *nc, const uint8_t *buf, size_
>         rtl8139_update_irq(s);
>     }
>
> +    trace_rtl8139_do_receive(nc, buf, size_, do_interrupt);
> +
>     return size_;
>  }
>

The rtl8139 has a lot of DEBUG_PRINT().  Have you thought about
converting those?

> @@ -1742,6 +1745,8 @@ static uint32_t rtl8139_RxConfig_read(RTL8139State *s)
>
>  static void rtl8139_transfer_frame(RTL8139State *s, const uint8_t *buf, int size, int do_interrupt)
>  {
> +    trace_rtl8139_transfer_frame(s, buf, size, do_interrupt);
> +
>     if (!size)
>     {
>         DEBUG_PRINT(("RTL8139: +++ empty ethernet frame\n"));
> diff --git a/net.c b/net.c
> index 9ba5be2..ba74f74 100644
> --- a/net.c
> +++ b/net.c
> @@ -36,6 +36,7 @@
>  #include "qemu-common.h"
>  #include "qemu_socket.h"
>  #include "hw/qdev.h"
> +#include "trace.h"
>
>  static QTAILQ_HEAD(, VLANState) vlans;
>  static QTAILQ_HEAD(, VLANClientState) non_vlan_clients;
> @@ -559,6 +560,7 @@ ssize_t qemu_send_packet_async(VLANClientState *sender,
>
>  void qemu_send_packet(VLANClientState *vc, const uint8_t *buf, int size)
>  {
> +    trace_qemu_send_packet(vc, buf, size);
>     qemu_send_packet_async(vc, buf, size, NULL);
>  }
>

All qemu_send_*() functions go through
qemu_send_packet_async_with_flags().  Lets put the trace event there
(there's already a printf), and include the sender, queue, buf, size,
and return value.  That way it's possible to trace all net traffic
including non-rtl8139.

> diff --git a/trace-events b/trace-events
> index e6138ea..76a8431 100644
> --- a/trace-events
> +++ b/trace-events
> @@ -254,3 +254,10 @@ disable spice_vmc_write(ssize_t out, int len) "spice wrottn %lu of requested %zd
>  disable spice_vmc_read(int bytes, int len) "spice read %lu of requested %zd"
>  disable spice_vmc_register_interface(void *scd) "spice vmc registered interface %p"
>  disable spice_vmc_unregister_interface(void *scd) "spice vmc unregistered interface %p"
> +
> +# net.c
> +disable qemu_send_packet(void *vc, const uint8_t *buf, int size) "client-state: %p, buf %p size %d"

Please use "vc %p buf %p size %d" without ':' or ','.  Some trace
events have been merged which do not follow the convention but I'd
like to try keeping the output consistent whenever possible.

Stefan