Re: [PATCH V2 0/5] trace-cmd: Support the feature recording trace data of guests on the host

[Yoshihiro YUNOMAE <yoshihiro.yunomae.ez@hitachi.com>]

Hi Steven,

Would you review this patch set?

Thanks,
Yoshihiro YUNOMAE

(2013/09/13 11:06), Yoshihiro YUNOMAE wrote:
> Hi Steven,
>
> This is a v2 patch set for realizing a part of "Integrated trace" feature which
> is a trace merging system for a virtualization environment. Currently, trace-cmd
> does not have following features yet:
>
> a) Server and client for a virtualization environment
> b) Structured message platform between guests and host
> c) Agent feature of a client
> d) Merge feature of trace data of multiple guests and host in chronological
>     order
>
> This patch set supports above a) and b) features.
>
> <overall view>
>
>          +------------+        +------------+
> Guest   |   a), c)   |        |   a), c)   | client/agent
>    ^     +------------+        +------------+
>    |         ^   ^                 ^   ^
> ============|===|=================|===|===========
>    |         v b)v                 v b)v
>    v     +----------------------------------+
> Host    |               a)                 | server
>          +----------------------------------+
>              ||output    ||         ||
>              \/          \/         \/
>          /--------+  /--------+  /--------+
>          | 010101 |  | 101010 |  | 100101 | binary data
>          | 010100 |  | 010100 |  | 110011 |
>          +--------+  +--------+  +--------+
>        \                                     /
>         \-----------------------------------/
>                         || d)
>                         \/
>         /-----------------------------------+
>         | (guest1) 123456: sched_switch...  | text data
>         | (guest2) 123458: kmem_free...     |
>         | (host)   123500: kvm_exit (guest1)|
>         | (host)   123510: kvm_entry(guest1)|
>         | (guest1) 123550: sched_switch...  |
>         +-----------------------------------+
>
> a) Server and client for a virtualization environment
>   trace-cmd has listen mode for network, but using network will be a high cost
> operation for inducing a lot of memory copying. From kernel-3.6, the
> virtio-console driver supports splice_write and ftrace supports "steal" for
> fops. So, guest clients of trace-cmd can send trace data without copying memory
> by using splice(2). If guest clients use virtio-serial, the server also needs to
> support virtio-serial I/F.
>
> b) Structured message platform between guests and a host
>   Currently, a server(clients) sends unstructured character string to
> clients(server), so clients(server) must parse the unstructured messages.
> Since it is hard to add complex contents in the protocol, structured binary
> message trace-msg is introduced as the communication protocol.
>
> c) Agent feature of a client
>   Current trace-cmd client can operate only as "record" mode, so the client
> will send trace data to the server immediately. However, when an user tries to
> collect trace data of multiple guests on a host, the user must log in to
> each guest. This is hard to use, I think. So, trace-cmd client had better
> support agent mode which receives a message from the server.
>
> d) Merge feature of trace data of multiple guests and a host in chronological
>     order
>   Current trace-cmd has a merge feature for multiple machines whose times are
> synchronized by NTP. When we use the feature, we execute "trace-cmd record"
> with --date option on each machine, and then we run "trace-cmd report" with -i
> option for each file.
>   However, there are cases that times of those machines cannot be synchronized.
> For example, although multiple users can run guests on virtualization
> environments (e.g. multi-tenant cloud hosting), there are no guarantee that
> they use the same NTP server. Moreover, even if the times are synchronized,
> trace data cannot exactly be merged because the NTP-synchronized time
> granularity may not be enough fine for sorting guest-host switching events.
>   So, I'm considering that trace data use x86-tsc as timestamp in order to merge
> trace data. By using x86-tsc, we can merge trace data even if time of those
> machines is not synchronized when CPU has the invariant TSC feature or the
> constant TSC feature. And the precision will be enough for understanding
> operations of guests and host. However, TSC values on a guest are not equal to
> the values on the host because
>           TSC_guest = TSC_host + TSC_offset.
>   This series actually doesn't support TSC offset, but I'd like to add such
> feature to fix host/guest clock difference in the other series. TSC offset
> values can be gotten as write_tsc_offset trace event from kernel-3.11.
> (see https://lkml.org/lkml/2013/6/12/72)
>
> For a), this patch introduces "virt-server" and "record --virt" modes for
> achieving low-overhead communication of trace data of guests. "virt-server" is a
> server mode for collecting trace data of guests. On the other hand,
> "record --virt" mode is a guest client for sending trace data of the guest.
> Although these functions are similar to "listen" and "record -N" modes each,
> these do not use network but use virtio-serial for low-overhead communication.
>
> For b), this patch series introduce specific message protocol in order to handle
> communication messages with 8 commands. When we extend any messages, using
> structured message will be easier than using unstructured message.
>
> <How to use>
> 1. Run virt-server on a host
>     # trace-cmd virt-server
>
> 2. Make guest domain directory
>     # mkdir -p /tmp/trace-cmd/virt/<domain>
>     # chmod 710 /tmp/trace-cmd/virt/<domain>
>     # chgrp qemu /tmp/trace-cmd/virt/<domain>
>
> 3. Make FIFO on the host
>     # mkfifo /tmp/trace-cmd/virt/<domain>/trace-path-cpu{0,1,...,X}.{in,out}
>
> 4. Set up of virtio-serial pipe of a guest on the host
>     Add the following tags to domain XML files.
>     # virsh edit <domain>
>     <channel type='unix'>
>        <source mode='connect' path='/tmp/trace-cmd/virt/agent-ctl-path'/>
>        <target type='virtio' name='agent-ctl-path'/>
>     </channel>
>     <channel type='pipe'>
>        <source path='/tmp/trace-cmd/virt/<domain>/trace-path-cpu0'/>
>        <target type='virtio' name='trace-path-cpu0'/>
>     </channel>
>     ... (cpu1, cpu2, ...)
>
> 5. Boot the guest
>     # virsh start <domain>
>
> 6. Execute "record --virt" on the guest
>     # trace-cmd record --virt -e sched*
>
> <Result>
> I measured CPU usage outputted by top command on a guest when client sends
> trace data. Client means "record -N"(NW) or "record --virt"(virtio-serial).
>
>                        NW     virtio-serial(splice)
> client(fedora19)    ~2.9[%]       ~1.7[%]
>
> <Future work>
> - Add an agent mode based on "record --virt"
> - Add a merging feature of trace data of guests and host to "report"
>
> Changes in V2:
> [1/5] Add a comment in open_udp()
> [2/5] Regacy protocol support in order to keep backward compatibility
>
> Thank you,
>
> ---
>
> Yoshihiro YUNOMAE (5):
>        [CLEANUP] trace-cmd: Split out binding a port and fork reader from open_udp()
>        trace-cmd: Apply the trace-msg protocol for communication between a server and clients
>        trace-cmd: Use poll(2) to wait for a message
>        trace-cmd: Add virt-server mode for a virtualization environment
>        trace-cmd: Add --virt option for record mode
>
>
>   Documentation/trace-cmd-record.1.txt      |   11
>   Documentation/trace-cmd-virt-server.1.txt |   89 +++
>   Makefile                                  |    2
>   trace-cmd.c                               |    3
>   trace-cmd.h                               |   14
>   trace-listen.c                            |  601 ++++++++++++++++----
>   trace-msg.c                               |  874 +++++++++++++++++++++++++++++
>   trace-msg.h                               |   31 +
>   trace-output.c                            |    4
>   trace-record.c                            |  146 ++++-
>   trace-recorder.c                          |   54 +-
>   trace-usage.c                             |   10
>   12 files changed, 1678 insertions(+), 161 deletions(-)
>   create mode 100644 Documentation/trace-cmd-virt-server.1.txt
>   create mode 100644 trace-msg.c
>   create mode 100644 trace-msg.h
>

-- 
Yoshihiro YUNOMAE
Software Platform Research Dept. Linux Technology Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: yoshihiro.yunomae.ez@hitachi.com