r8169: strange increase in ping latency after suspend/resume

["Holger Hoffstätte" <holger.hoffstaette@googlemail.com>]

While benchmarking things on my home LAN (wired, 1Gbit) I noticed an
apparent asymmetric network behaviour between two hosts: a flood ping
from a -> b was consistently slower than from b -> a, despite same
kernel (3.18.x), same NIC (even same rev/firmware), same everything, and
even a being the faster machine (8-core i7 desktop vs. low-end 4-core i5).

Despite the fact that there are two cheap switches in the path I couldn't
figure out why one way would be faster than the other. After many iterations
of eliminating possible causes of my tweaked kernel (with/without BFS CPU
scheduler, SMT awareness, offload settings, HZ/NOHZ and whatnot) I found
that the difference is much more subtle: reboot vs. suspend/resume.

Since I suspend/resume the desktop nightly but not the server, the
behaviour would never become visible in the other direction.

From desktop to server (tux aka. 192.168.100.222), hosts & network idle:

$ping -q -c 10000 -i 0 tux

after a reboot:

--- tux ping statistics ---
10000 packets transmitted, 10000 received, 0% packet loss, time 647ms
rtt min/avg/max/mdev = 0.038/0.046/0.152/0.004 ms, ipg/ewma 0.064/0.041 ms

--- tux ping statistics ---
10000 packets transmitted, 10000 received, 0% packet loss, time 645ms
rtt min/avg/max/mdev = 0.041/0.046/0.061/0.003 ms, ipg/ewma 0.064/0.045 ms

--- tux ping statistics ---
10000 packets transmitted, 10000 received, 0% packet loss, time 639ms
rtt min/avg/max/mdev = 0.039/0.045/0.151/0.009 ms, ipg/ewma 0.063/0.046 ms

This is not too bad considering cheap equipment + two switches in the path,
and consistent with measurements in the other direction (server to desktop)
which would also see *very slightly* increase due to the host being slower.
So that's good and consistent.

Now I suspend/resume the desktop, and we get:

--- tux ping statistics ---
10000 packets transmitted, 10000 received, 0% packet loss, time 930ms
rtt min/avg/max/mdev = 0.041/0.050/0.150/0.009 ms, ipg/ewma 0.093/0.053 ms

--- tux ping statistics ---
10000 packets transmitted, 10000 received, 0% packet loss, time 952ms
rtt min/avg/max/mdev = 0.042/0.051/0.132/0.005 ms, ipg/ewma 0.095/0.050 ms

--- tux ping statistics ---
10000 packets transmitted, 10000 received, 0% packet loss, time 938ms
rtt min/avg/max/mdev = 0.040/0.050/0.157/0.009 ms, ipg/ewma 0.093/0.052 ms

This is 100% repeatable: after a reboot it's fast; suspend/resume: slow.

While this is no drama and far from being unusable, something is now stealing
~5µs on average. I *think* that this relates to either the r8169 NIC or driver,
but can only offer inconclusive evidence insofar as an old laptop with e1000e
shows consistently slower performance, regardless of reboot vs. wakeup.

Any ideas or suggestions?

thanks
Holger