Re: [Bug #12465] KVM guests stalling on 2.6.28 (bisected) [Bug 12465]

Kevin Shanahan
2009-02-23T11:38:55Z

On Sun, 2009-02-15 at 11:04 +0100, Ingo Molnar wrote:
> It would be nice to enhance this single-CPU trace some more - to more
> surgically see what is going on. Firstly, absolute timestamps would be
> nice:
>
> echo funcgraph-abstime > trace_options
> echo funcgraph-proc > trace_options
>
> as it's a bit hard to see the global timescale of events.

Okay, here's some more trace data. I grabbed a few samples at different
times during the ping test. I think the data in files trace6.txt and
trace8.txt coincided with some of the biggest delays.

http://disenchant.net/tmp/bug-12465/trace-2/

This is captured on 2.6.29-rc5-tip-02057-gaad11ad. The kvm guest being
pinged is process 11211:

flexo:~# pstree -p 11211
qemu-system-x86(11211)─┬─{qemu-system-x86}(11212)
├─{qemu-system-x86}(11213)
└─{qemu-system-x86}(11609)

Cheers,
Kevin.

> Secondly, not all events are included - in particular i dont really see
> the points when packets are passed. Would it be possible to add a tracing
> hypercall so that the guest kernel can inject trace events that can be seen
> on the native-side trace? Regarding ping latencies really just two things
> matter: the loopback network device's rx and tx path. We should trace the
> outgoing sequence number and the incoming sequence number of IP packets,
> and inject that to the host side. This way we can correlate the delays
> precisely.
>
> Ingo