Performance problem in network namespaces

Performance problem in network namespaces

[Martín Ferrari]

Hi,

When running some benchmarks to test the feasibility of using
namespaces for emulating networks, I have found a big drop in
performance when one of the namespaces is performing routing of
packets.

After some search, we found that  in ip_forward() the skb is being
copied. It seems that (ICMP and UDP, does not happen with TCP) packets
start with a small headroom (16 bytes in our observation) but skb_cow
always allocates at least NET_SKB_PAD (32 in x86) bytes of headroom,
thus triggering this unnecessary memcpy.

We made two crude attempts at fixing this, which are surely incorrect,
but hopefully somebody here could come up with a correct solution.

Our attempts are: 1. remove the lower bound in headroom size at
skb_cow, and 2. set needed_headroom in veth.c to NET_SKB_PAD; patches
included below.

Thanks.


diff -Naurp linux-2.6.34-rc5/include/linux/skbuff.h
../linux-2.6.34-rc5/include/linux/skbuff.h
--- linux-2.6.34-rc5/include/linux/skbuff.h	2010-04-20 01:29:56.000000000 +0200
+++ ../linux-2.6.34-rc5/include/linux/skbuff.h	2010-05-03
11:17:13.000000000 +0200
@@ -1526,8 +1526,6 @@ static inline int __skb_cow(struct sk_bu
 {
 	int delta = 0;

-	if (headroom < NET_SKB_PAD)
-		headroom = NET_SKB_PAD;
 	if (headroom > skb_headroom(skb))
 		delta = headroom - skb_headroom(skb);


diff -Naurp linux-2.6.34-rc5/drivers/net/veth.c
../linux-2.6.34-rc5/drivers/net/veth.c
--- linux-2.6.34-rc5/drivers/net/veth.c	2010-04-20 01:29:56.000000000 +0200
+++ ../linux-2.6.34-rc5/drivers/net/veth.c	2010-04-30 11:29:39.000000000 +0200
@@ -303,6 +303,8 @@ static void veth_setup(struct net_device
 	dev->ethtool_ops = &veth_ethtool_ops;
 	dev->features |= NETIF_F_LLTX;
 	dev->destructor = veth_dev_free;
+	/* Try to avoid skb copies when passing packets around */
+	dev->needed_headroom = NET_SKB_PAD;
 }

 /*


-- 
Martín Ferrari

Re: Performance problem in network namespaces

[Benny Amorsen]

Martín Ferrari <martin.ferrari@gmail.com> writes:

> When running some benchmarks to test the feasibility of using
> namespaces for emulating networks, I have found a big drop in
> performance when one of the namespaces is performing routing of
> packets.

Is this problem specific to vnet, or do the other types of interfaces
suffer from it as well? (phys, vlan, macvlan...)


/Benny

Re: Performance problem in network namespaces

[Martín Ferrari]

Hi Benny,

On Tue, May 4, 2010 at 11:48, Benny Amorsen <benny+usenet@amorsen.dk> wrote:

>> When running some benchmarks to test the feasibility of using
>> namespaces for emulating networks, I have found a big drop in
>> performance when one of the namespaces is performing routing of
>> packets.

> Is this problem specific to vnet, or do the other types of interfaces
> suffer from it as well? (phys, vlan, macvlan...)

Seems that it is specific to vnet, but if there's other way of having
datagrams created locally that get into the routing code before
leaving the system, maybe that would have the same problem.

I tried a couple of combinations that somehow included routing in the mix:
- routing loop over the same ethernet device (e1000e)
- routing between eth and wlan (iwlagn)
- eth and veth (with the paired veth inside a different namespace)
- wlan and veth (ditto)

In all those cases, I see that in ip_forward() the headroom is already
enough, and I think is due to the fact that the hardware drivers use
netdev_alloc_skb which already adds NET_SKB_PAD to the length
requested.

Using macvlan over the real devices showed the same results.

I also tried:
- eth and wlan: packets arriving from eth had a headroom  of 48, but
80 was needed to pass it to wlan

-- 
Martín Ferrari