[PATCH 0/6] Kill off the virtio_net tx mitigation timer

[Mark McLoughlin <markmc@redhat.com>]

Hey,

The main patch in this series is 5/6 - it just kills off the
virtio_net tx mitigation timer and does all the tx I/O in the
I/O thread.

Below are the results I got from benchmarking guest->host and
host->guest on my machine.

There's enough numbers there to make anyone blind, but basically
there are results for current kvm-userspace.git, with the
no-tx-timer patch applied and with the drop-the-mutex patch
applied.

Also, I've included results that show what difference some tuning
makes with all the patches applied. The tuning basically just
involves pinning the I/O thread and the netperf/netserver processes
in both the host and guest to two physical CPUs which share a L2
cache.

(Yes, the 1k buffer size results are weird - we think there's a bug
in recent kernels that causes us not to coalesce these small buffers
into a large GSO packet before sending)

Anyway, the results in all their glory:

                                |       guest->host tput     |      host->guest tput
  netperf, 10x20s runs (Gb/s)   |    min/ mean/   max/stddev |   min/  mean/   max/stddev
  ------------------------------+----------------------------+---------------------------
  kvm-userspace.git, 1k         | 0.600/ 0.645/ 0.670/ 0.025 | 5.170/ 5.285/ 5.470/ 0.087
  kvm-userspace.git, 16k        | 3.070/ 3.350/ 3.710/ 0.248 | 5.950/ 6.374/ 6.760/ 0.261
  kvm-userspace.git, 65k        | 4.950/ 6.041/ 7.170/ 0.639 | 5.480/ 5.642/ 5.810/ 0.092

  no tx timer, 1k               | 0.720/ 0.790/ 0.850/ 0.040 | 4.950/ 5.172/ 5.370/ 0.128
  no tx timer, 16k              | 4.120/ 4.512/ 4.740/ 0.190 | 4.900/ 5.480/ 6.230/ 0.416
  no tx timer, 65k              | 5.510/ 7.702/ 9.600/ 1.153 | 4.490/ 5.208/ 5.690/ 0.408

  drop mutex, 1k                | 0.810/ 0.847/ 0.910/ 0.030 | 5.140/ 5.416/ 5.660/ 0.145
  drop mutex, 16k               | 5.110/ 5.713/ 6.480/ 0.440 | 6.050/ 6.658/ 7.490/ 0.443
  drop mutex, 65k               | 7.070/ 8.054/ 9.210/ 0.671 | 4.470/ 5.922/ 7.200/ 0.930

  tuned, 1k                     | 4.260/ 6.739/ 8.020/ 1.075 | 3.640/ 3.777/ 4.020/ 0.110
  tuned, 16k                    | 8.530/ 8.867/ 9.110/ 0.146 | 7.220/ 7.276/ 7.360/ 0.040
  tuned, 65k                    | 9.200/ 9.282/ 9.870/ 0.198 | 7.850/ 7.924/ 8.000/ 0.045

                                |       guest->host %idle       |      host->guest %idle
  netperf, 10x20s runs (Gb/s)   |   avg/cpu#0/cpu#1/cpu#2/cpu#3 | avg/cpu#0/cpu#1/cpu#2/cpu#3
  ------------------------------+--------------------------------+-----------------------------
  kvm-userspace.git, 1k         | 64.93/65.94/61.06/72.89/60.31 | 39.36/52.52/19.04/63.94/21.35
  kvm-userspace.git, 16k        | 61.47/66.22/52.76/66.99/60.02 | 43.94/58.12/27.45/55.85/33.64
  kvm-userspace.git, 65k        | 61.37/82.58/51.55/51.15/60.16 | 49.30/69.80/37.14/51.75/38.24

  no tx timer, 1k               | 57.99/59.93/59.68/71.17/42.03 | 42.13/60.85/31.13/55.06/22.94
  no tx timer, 16k              | 54.87/55.57/55.88/61.35/46.67 | 49.18/54.35/40.56/59.42/43.46
  no tx timer, 65k              | 47.53/73.63/33.00/46.36/37.14 | 53.67/66.69/40.92/47.73/59.89

  drop mutex, 1k                | 52.94/61.90/48.44/64.35/38.25 | 39.31/53.29/26.35/60.90/17.53
  drop mutex, 16k               | 49.28/59.47/37.16/50.07/50.16 | 42.18/52.26/26.76/55.95/34.26
  drop mutex, 65k               | 49.65/65.94/41.38/47.55/42.55 | 46.36/42.51/60.55/45.15/37.91

  tuned, 1k                     | 57.68/99.81/99.96/18.76/21.91 | 65.49/99.84/99.97/15.20/52.46
  tuned, 16k                    | 61.66/99.85/99.98/13.01/40.83 | 61.43/99.79/99.93/0.97/47.61
  tuned, 65k                    | 62.57/99.99/99.95/13.13/39.28 | 60.15/99.81/99.97/1.27/43.10

Cheers,
Mark.