[PATCH 0/2 v3] ipv4: Cache dst in tunnels

[PATCH 0/2 v3] ipv4: Cache dst in tunnels

[Tom Herbert]

Version 3 of caching routes in tunnels.

Addressed some comments from Eric in this series.

There are two patches (variants) in the series:
1) One dst cached for each tunnel.
2) Percpu dst cache per tunnel to avoid false sharing

Testing with GRE tunnels on a 32 CPU host with bnx2x (RSS support
for GRE) shows a modest improvement in CPU utilization with these
patches running 200 TCP_RR netperf clients.

Without patches
71.22% CPU utilization
138/180/244 90/95/99% latencies
1.30465e+06 CPU/tps
18318 CPU/tps

With patches
69.84%
142/186/249 90/95/99% latencies
1.30827e+06
18732 CPU/tps

Re: [PATCH 0/2 v3] ipv4: Cache dst in tunnels

[Rick Jones]

Nothing that calls for a V4, but a couple of nits and such.

On 01/02/2014 11:48 AM, Tom Herbert wrote:
> Without patches
> 71.22% CPU utilization
> 138/180/244 90/95/99% latencies

The netperf version I have, for which I think I faithfully applied the 
patches from Google for the percentiles is 50, 90 and 99.

raj@tardy:~$ netperf -- -O ? | grep -i laten
RT_LATENCY
MIN_LATENCY
MAX_LATENCY
P50_LATENCY
P90_LATENCY
P99_LATENCY
MEAN_LATENCY
STDDEV_LATENCY

> 1.30465e+06 CPU/tps

That is TPS rather than CPU/TPS I am guessing.

> 18318 CPU/tps

TPS/CPU%

>
> With patches
> 69.84%
> 142/186/249 90/95/99% latencies
> 1.30827e+06
> 18732 CPU/tps

Any idea as to the source of the similarly modest increase in the latencies?

Speaking of which, given the overhead of measuring the individual 
latencies, you might try a pair of runs without them being measured and 
see if you show a larger delta in tps and/or CPU utilization.

raj@tardy:~/netperf2_trunk/src$ ./netperf -i 30,3 -t TCP_RR -- -o 
throughput,p50_latency,p90_latency,p99_latency;  ./netperf -i 30,3 -t 
TCP_RR -- -o throughput
MIGRATED TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET 
to localhost.localdomain () port 0 AF_INET : +/-2.500% @ 99% conf.  : 
interval : demo : first burst 0
Throughput,50th Percentile Latency Microseconds,90th Percentile Latency 
Microseconds,99th Percentile Latency Microseconds
46248.37,21,22,28
MIGRATED TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET 
to localhost.localdomain () port 0 AF_INET : +/-2.500% @ 99% conf.  : 
interval : demo : first burst 0
Throughput
48098.45

A nitpick from the 1/2 email:

> the tunnel is "connected" so that all the rouitng parameters are


happy benchmarking,

rick jones

Re: [PATCH 0/2 v3] ipv4: Cache dst in tunnels

[David Miller]

From: Tom Herbert <therbert@google.com>
Date: Thu, 2 Jan 2014 11:48:18 -0800 (PST)

> Version 3 of caching routes in tunnels.

Applied, but I really have to wonder when I have to fix up things like
this:

net/ipv4/ip_tunnel.c: In function ‘ip_tunnel_init’:
net/ipv4/ip_tunnel.c:1062:20: warning: assignment from incompatible pointer type [enabled by default]

-->

--- a/net/ipv4/ip_tunnel.c
+++ b/net/ipv4/ip_tunnel.c
@@ -1059,7 +1059,7 @@ int ip_tunnel_init(struct net_device *dev)
 		u64_stats_init(&ipt_stats->syncp);
 	}
 
-	tunnel->dst_cache = alloc_percpu(struct ip_tunnel_dst *);
+	tunnel->dst_cache = alloc_percpu(struct ip_tunnel_dst);
 	if (!tunnel->dst_cache) {
 		free_percpu(dev->tstats);
 		return -ENOMEM;