Re: [Qemu-devel] sniffing traffic between VMs

[Stefan Hajnoczi <stefanha@gmail.com>]

On Mon, Oct 07, 2013 at 05:47:46PM +0300, Alexander Binun wrote:
> Our first task is to trace the traffic between individual VMs and between VMs and the VMM (the KVM driver). So we are searching for proper places to insert "sniffer code". We suspect that some functions in qemu/hw/virtio should be targeted. And we will appreciate any hints on this places.

My blog post about -netdev pcap in QEMU is useful for QEMU network code
development setups.  But the simplest way to sniff traffic in a
production x86 KVM configuration is using tcpdump on the host.

The common networking setup on the host is a Linux software bridge (e.g.
virbr0) and one tap device per guest (e.g. vm001-tap, vm002-tap).  The
tap devices are added to the bridge so guests can communicate with each
other.

When a guest sends a packet, the vhost_net host kernel driver injects
the packet into the guest's tap device.  The Linux network stack then
hands the packet from the tap device to the bridge.

The bridge will forward the packet as appropriate.  In guest<->guest
communication this means the packet is forwarded to the destination
guest's tap device.

The vhost_net driver instance for the destination guest then reads the
packet from its tap device and places it into the guest's virtio-net
receive buffer.

This configuration means you have 3 places where you can run tcpdump on
the host:

1. On the source guest's tap device (e.g. vm001-tap).
2. On the bridge interface (e.g. virbr0).
3. On the destination guest's tap device (e.g. vm002-tap).

There are other options too like using openvswitch or macvtap.
Openvswitch might be interesting because I think it allows you to add
filtering rules into the kernel and send packets that match the rules up
to a userspace process for inspection.

Stefan