Re: [PATCH 0/9][RFC] KVM virtio_net performance

[Avi Kivity <avi@qumranet.com>]

Mark McLoughlin wrote:
> Hey,
>       Here's a bunch of patches attempting to improve the performance
> of virtio_net. This is more an RFC rather than a patch submission
> since, as can be seen below, not all patches actually improve the
> perfomance measurably.
>
>       I've tried hard to test each of these patches with as stable and
> informative a benchmark as I could find. The first benchmark is a
> netperf[1] based throughput benchmark and the second uses a flood
> ping[2] to measure latency differences.
>
>       Each set of figures is min/average/max/standard deviation. The
> first set is Gb/s and the second is milliseconds.
>
>       The network configuration used was very simple - the guest with
> a virtio_net interface and the host with a tap interface and static
> IP addresses assigned to both - e.g. there was no bridge in the host
> involved and iptables was disable in both the host and guest.
>
>       I used:
>
>   1) kvm-71-26-g6152996 with the patches that follow
>
>   2) Linus's v2.6.26-5752-g93ded9b with Rusty's virtio patches from
>      219:bbd2611289c5 applied; these are the patches have just been
>      submitted to Linus
>
>       The conclusions I draw are:
>
>   1) The length of the tx mitigation timer makes quite a difference to
>      throughput achieved; we probably need a good heuristic for
>      adjusting this on the fly.
>   

The tx mitigation timer is just one part of the equation; the other is 
the virtio ring window size, which is now fixed.

Using a maximum sized window is good when the guest and host are running 
flat out, doing nothing but networking.  When throughput drops (because 
the guest is spending cpu on processing, or simply because the other 
side is not keeping up), we need to drop the windows size so as to 
retain acceptable latencies.

The tx timer can then be set to "a bit after the end of the window", 
acting as a safety belt in case the throughput changes.

>   4) Dropping the global mutex while reading GSO packets from the tap
>      interface gives a nice speedup. This highlights the global mutex
>      as a general perfomance issue.
>
>   

Not sure whether this is safe.  What's stopping the guest from accessing 
virtio and changing some state?

>   5) Eliminating an extra copy on the host->guest path only makes a
>      barely measurable difference.
>
>   

That's expected on a host->guest test.  Zero copy is mostly important 
for guest->external, and with zerocopy already enabled in the guest 
(sendfile or nfs server workloads).

>         Anyway, the figures:
>
>   netperf, 10x20s runs (Gb/s)  |       guest->host          |       host->guest
>   -----------------------------+----------------------------+---------------------------
>   baseline                     | 1.520/ 1.573/ 1.610/ 0.034 | 1.160/ 1.357/ 1.630/ 0.165
>   50us tx timer + rearm        | 1.050/ 1.086/ 1.110/ 0.017 | 1.710/ 1.832/ 1.960/ 0.092
>   250us tx timer + rearm       | 1.700/ 1.764/ 1.880/ 0.064 | 0.900/ 1.203/ 1.580/ 0.205
>   150us tx timer + rearm       | 1.520/ 1.602/ 1.690/ 0.044 | 1.670/ 1.928/ 2.150/ 0.141
>   no ring-full heuristic       | 1.480/ 1.569/ 1.710/ 0.066 | 1.610/ 1.857/ 2.140/ 0.153
>   VIRTIO_F_NOTIFY_ON_EMPTY     | 1.470/ 1.554/ 1.650/ 0.054 | 1.770/ 1.960/ 2.170/ 0.119
>   recv NO_NOTIFY               | 1.530/ 1.604/ 1.680/ 0.047 | 1.780/ 1.944/ 2.190/ 0.129
>   GSO                          | 4.120/ 4.323/ 4.420/ 0.099 | 6.540/ 7.033/ 7.340/ 0.244
>   ring size == 256             | 4.050/ 4.406/ 4.560/ 0.143 | 6.280/ 7.236/ 8.280/ 0.613
>   ring size == 512             | 4.420/ 4.600/ 4.960/ 0.140 | 6.470/ 7.205/ 7.510/ 0.314
>   drop mutex during tapfd read | 4.320/ 4.578/ 4.790/ 0.161 | 8.370/ 8.589/ 8.730/ 0.120
>   aligouri zero-copy           | 4.510/ 4.694/ 4.960/ 0.148 | 8.430/ 8.614/ 8.840/ 0.142
>   

Very impressive numbers; much better than I expected.  The host->guest 
numbers are around 100x better than the original emulated card througput 
we got from kvm.

>   ping -f -c 100000 (ms)       |       guest->host          |       host->guest
>   -----------------------------+----------------------------+---------------------------
>   baseline                     | 0.060/ 0.459/ 7.602/ 0.846 | 0.067/ 0.331/ 2.517/ 0.057
>   50us tx timer + rearm        | 0.081/ 0.143/ 7.436/ 0.374 | 0.093/ 0.133/ 1.883/ 0.026
>   250us tx timer + rearm       | 0.302/ 0.463/ 7.580/ 0.849 | 0.297/ 0.344/ 2.128/ 0.028
>   150us tx timer + rearm       | 0.197/ 0.323/ 7.671/ 0.740 | 0.199/ 0.245/ 7.836/ 0.037
>   no ring-full heuristic       | 0.182/ 0.324/ 7.688/ 0.753 | 0.199/ 0.243/ 2.197/ 0.030
>   VIRTIO_F_NOTIFY_ON_EMPTY     | 0.197/ 0.321/ 7.447/ 0.730 | 0.196/ 0.242/ 2.218/ 0.032
>   recv NO_NOTIFY               | 0.186/ 0.321/ 7.520/ 0.732 | 0.200/ 0.233/ 2.216/ 0.028
>   GSO                          | 0.178/ 0.324/ 7.667/ 0.736 | 0.147/ 0.246/ 1.361/ 0.024
>   ring size == 256             | 0.184/ 0.323/ 7.674/ 0.728 | 0.199/ 0.243/ 2.181/ 0.028
>   ring size == 512             |             (not measured) |             (not measured)
>   drop mutex during tapfd read | 0.183/ 0.323/ 7.820/ 0.733 | 0.202/ 0.242/ 2.219/ 0.027
>   aligouri zero-copy           | 0.185/ 0.325/ 7.863/ 0.736 | 0.202/ 0.245/ 7.844/ 0.036
>
>   

This isn't too good.  Low latency is important for nfs clients (or other 
request/response workloads).  I think we can keep these low by adjusting 
the virtio window (for example, on an idle system it should be 1), so 
that the tx mitigation timer only fires when the workload transitions 
from throughput to request/response.



-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.