tx-nocache-copy performance

[Benjamin Poirier <bpoirier@suse.de>]

Hi Tom,

In commit "c6e1a0d net: Allow no-cache copy from user on transmit
(v3.0-rc1)" you introduced the tx-nocache-copy performance optimization
and set it to on by default. I've tried to reproduce your testcase, as
well as a few more, but I did not find any performance improvement from
turning on tx-nocache-copy. Do you think tx-nocache-copy is still a
worthwhile optimization and it should remain on by default? In which
situations does it help?

I've ran latency tests similar to the ones you described in the commit
log. I've also tested how the option affects single stream throughput
tests. According to the results I obtained, it seems that
tx-nocache-copy has either no impact (in the latency test) or a negative
impact (in the throughput test).

My test results follow. I tested using 3.12.6 on one Intel Xeon W3565
and one i7 920 connected by ixgbe adapters. The results are from the
Xeon, but they're similar on the i7. All numbers report the mean±stddev
over 10 runs of 10s.

1) latency tests similar to what you described
There is no statistically significant difference between tx-nocache-copy
on/off.
nic irqs spread out (one queue per cpu)

200x netperf -r 1400,1
tx-nocache-copy off
        692000±1000 tps
        50/90/95/99% latency (us): 275±2/643.8±0.4/799±1/2474.4±0.3
tx-nocache-copy on
        693000±1000 tps
        50/90/95/99% latency (us): 274±1/644.1±0.7/800±2/2474.5±0.7

200x netperf -r 14000,14000
tx-nocache-copy off
        86450±80 tps
        50/90/95/99% latency (us): 334.37±0.02/838±1/2100±20/3990±40
tx-nocache-copy on
        86110±60 tps
        50/90/95/99% latency (us): 334.28±0.01/837±2/2110±20/3990±20

2) single stream throughput tests
tx-nocache-copy leads to higher service demand

                        throughput  cpu0        cpu1        demand
                        (Gb/s)      (Gcycle)    (Gcycle)    (cycle/B)

nic irqs and netperf on cpu0 (1x netperf -T0,0 -t omni -- -d send)

tx-nocache-copy off     9402±5      9.4±0.2                 0.80±0.01
tx-nocache-copy on      9403±3      9.85±0.04               0.838±0.004

nic irqs on cpu0, netperf on cpu1 (1x netperf -T1,1 -t omni -- -d send)

tx-nocache-copy off     9401±5      5.83±0.03   5.0±0.1     0.923±0.007
tx-nocache-copy on      9404±2      5.74±0.03   5.523±0.009 0.958±0.002

-Benjamin