Re: [Qemu-devel] [RFC] host and guest kernel trace merging

[Peter Xu <peterx@redhat.com>]

Hi, Steven,

On Fri, Mar 04, 2016 at 08:23:11AM -0500, Steven Rostedt wrote:
> My idea for a trace-cmd server, is to have a --client operation, for
> running on the guest.
> 
>  trace-cmd server --client <connection>
> 
> The connection will be some socket, either network or something
> directly attached to the host.
> 
> Then on the host, we can have
> 
>   trace-cmd server --connect <guest>
> 
> Where the server will create a connection to the guest.
> 
> And then, you could run on the host:
> 
>   trace-cmd record <host-events> --connect <guest> <guest-events>
> 
> And this will start recording host events, and then connect to the
> local server that connects to the guest(s) and that will start tracing
> on the guest as well.
> 
> Then events on the guest will be passed to the host server.
> 
> Something like this is my idea. We can work out the details on the best
> way to get things working. We may be able to eliminate the host server
> middle man. But I envision that we need a trace-cmd server running on
> the guest to start off the commands.

Not sure whether fully I understand the above, it seems that we can
remove the host server middle man (as you have mentioned). Moreover,
I am not sure whether we can use this for multiple hosts as well,
and guests will be a special case, in which we can get very accurate
tsc offset. Let me try to do a rough summary of what I thought.

So, firstly, we start a trace-cmd server on every host (or guest)
that we want to trace using:

  trace-cmd server <connection>

(In the case of one host + one guest tracing, we need to start the
server on both host and guest, though the <connection> might differ)

Then, on any host/guest that can reach all the target hosts/guests
via different <connection>s, do the tracing using:

  trace-cmd record --connect <connection> <events...> \
                   --connect <connection> <events...> \
                   ...

Finally, when "trace-cmd record" stops, we get one trace file with
all traces merged. Here, if we are tracing multiple hosts, we use
timestamps for merging (maybe we can have some other way to adjust
the offset for each host). For guest special case, we will know it
in some way (the stupid one is to provide it in parameter following
the --connect of guest, or we may detect it via <connection>
formats, etc.) take special care to fetch the offset and CPU
frequency information, so that we will get extremely accurate merged
results.

All the above still not considering security issues, assuming what
Paolo mentioned (using existing file protections for sockets, pipes,
etc.) should work.

Thanks.

-- peterx