Re: A KernelShark plugin for Xen traces analysis

[Andrew Cooper <andrew.cooper3@citrix.com>]

On 14/04/2021 14:43, Steven Rostedt wrote:
>> This causes major problems for `perf` support under Xen, which assumes
>> that the kernel's idea of CPUs matches that of the system.
> Things are different with KernelShark.

That is very encouraging to hear.

>> When rendering a trace including Xen data, Xen can provide the real
>> system CPUs, and dom0 wants to be rendered as a VM under Xen, similar to
>> trace-Fedora21 in your screenshot above.  (Obviously, if you're doing
>> nested virt, things need to start nesting.)
> Right.
>
> What I would envision how this would work, is that you would produce a
> set of tracing files. One for each guest (including Dom0), and one for the
> Xen hypervisor itself. The trick is to have a way to synchronize the time
> stamps. What we just did with KVM is to have all the tracing record the
> CPUs TSC, including the shift and multiplier that the CPU might change for
> the guests. Then we have a way to convert the TSC to nanoseconds. This way
> all tracing data has the same clock. It's somewhat complicated to get
> right, and requires access to how the guests clocks are modified by the CPU.

Hmm.  In the past, I have had success by modifying Xen to refuse any
shift/scale settings, at which point VMs and the hypervisor have
directly-comparable raw TSC values.

Xen certainly has enough information to describe what TSC rate/epoch
each guest is seeing, but I doubt any of this is coherently exposed at
the moment.

> For KVM, each machine has a unique id and is stored in the trace.dat files.
> We have the host store a mapping of what thread represents which guest VCPU
> (virtual CPU). Then the "-a" option tells KernelShark to append the
> tracing data as a dependency. I would imagine we can have something like
> this:
>
>  kernelshark xen.dat -a trace-dom0.dat -a trace-guest1.dat -a trace-guest2.dat
>
> The Xen plugin would then need to read the how the threads in xen.dat map
> to the virtual CPUs of each of the guest files. Which would give you the
> layering.

Looks good.  I suspect we might need to do a little work on Xen's trace
data to make this mesh together nicely.  In particular, Xen doesn't have
a terribly good scheme on unique IDs for "a VM".

We've got domain ID's which are Xen's unique instances of a running
"thing", but they change across VM reboot/migrate/etc.  I suspect we
have some atomicity problems with unique identification information and
VM-fork too.

There is a UUID field but we leave that entirely up to the toolstack to
manage.  (A good test for naive toolstack code comes on the a localhost
live migrate, because suddenly the toolstack is presented with one
logical VM (=> one UUID) and two concurrent domid's.)


I'll try to have a play with the plugin in some copious free time, but
this work does look exciting.

~Andrew