Tracing KVM with LTTng

Tracing KVM with LTTng

[Julien Desfossez]

Hello,

I'm currently doing my master degree on tracing and measuring with LTTng
the impact of virtualization on a physical host.
One focus of this work is to be able to represent in a graphical way the
state of a machine where virtual machines are running.

I have started to connect the already defined TRACE_EVENTs with LTTng
probes and adapted LTTV to represent a new state : VIRT.

In the following screenshots, I captured a trace of 4 seconds on the
physical machine. My machine has 8 CPUs, a KVM VM was started with 4
VCPUs. At the beginning, the VM was idle, after 1 second I started an
infinite loop and an other 1 second later.

On [1] you can see the impact of the loops on the kvm threads during the
4 seconds.
On [2] you can see a closer look of the state of the kvm threads (blue
means syscall, red means running in VM mode (vm_entry), dark yellow
means waiting for CPU).
On [3] you can see the relationship between the kvm threads and other
processes on the system (here especially kondemand and ksmd), each white
dot meaning there are events not displayed because the zoom level is to high
On [4] you can see the resource view, which makes clear the relationship
between physical resources, the interrupts and the VM
On [5] you can see the statistics view on a specific physical CPU and on
the whole trace.
And finally on [6] you can see an other test with a ping flood.

In the next days, I will send my patches to the official LTTng git.
My next step is to synchronise the traces collected from the host with
the traces collected from the guest (by finding an efficient way to
share the TSC_OFFSET) to have some infos of what is happening during the
time the VM has the control.

The reason I post these screenshots now is that I will be at Linuxcon
next week, and I would really appreciate some feedbacks and ideas for
future improvements from the KVM community.
So if you are interested, contact me directly and if you are there we'll
try to meet.

Thanks,

Julien

[1] http://ju.klipix.org/lttng/lttng-kvm-busy.png
[2] http://ju.klipix.org/lttng/lttng-kvm.png
[3] http://ju.klipix.org/lttng/lttng-kvm-ksmd.png
[4] http://ju.klipix.org/lttng/lttng-kvm-ressource-view.png
[5] http://ju.klipix.org/lttng/lttng-kvm-stats.png
[6] http://ju.klipix.org/lttng/lttng-kvm-ping.png

Re: Tracing KVM with LTTng

[Stefan Hajnoczi]

On Fri, Aug 6, 2010 at 4:42 PM, Julien Desfossez <ju@klipix.org> wrote:
> On [2] you can see a closer look of the state of the kvm threads (blue
> means syscall, red means running in VM mode (vm_entry), dark yellow
> means waiting for CPU).

This is a great visualization.  It shows the state of the entire
system in a clear way and makes the wait times easier to understand.

> In the next days, I will send my patches to the official LTTng git.
> My next step is to synchronise the traces collected from the host with
> the traces collected from the guest (by finding an efficient way to
> share the TSC_OFFSET) to have some infos of what is happening during the
> time the VM has the control.

I'd like to try this patch out, will be checking the LTTng mailing list.

> The reason I post these screenshots now is that I will be at Linuxcon
> next week, and I would really appreciate some feedbacks and ideas for
> future improvements from the KVM community.
> So if you are interested, contact me directly and if you are there we'll
> try to meet.

I'd like to find out more about what you're doing.  Unfortunately I
will not be at LinuxCon/KVM Forum this year.

Prerna Saxena and I have been working on static trace events for QEMU
userspace.  Here is the commit to add LTTng Userspace Tracer support:

http://repo.or.cz/w/qemu/stefanha.git/commitdiff/5560c202f4eeeec5cc37692d35e53f784c74d65c

Using the tracing branch above you can place static trace events in
QEMU userspace and collect the trace with LTTng.  See the trace-events
file for sample trace event definitions.

Stefan