[PATCH] ibmveth: Increase default copybreak limits to 2k

[PATCH] ibmveth: Increase default copybreak limits to 2k

[Robert Jennings]

Increase the copybreak limits for rx and tx from 128 bytes
to 2048 bytes. These limits were added by commits
8d86c61ae41d9068fd5e5cc01a4abd53c4fe3ab5 and
c08cc3ccebd46dce44d13a8ce81d249e687eeb8a to make use of a
bounce buffer for packets below 128 bytes.  This avoids
tearing down and creating a TCE entry.

Performance testing shows that this default limit can be
increased from 128 to 2048 for both rx and tx copybreak.
This resulted in ~10% throughput increase for for packets
that fit this limit without affecting performance for larger
packets.

Signed-off-by: Robert Jennings <rcj@linux.vnet.ibm.com>
---
 drivers/net/ibmveth.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/net/ibmveth.c b/drivers/net/ibmveth.c
index c454b45..20b7a98 100644
--- a/drivers/net/ibmveth.c
+++ b/drivers/net/ibmveth.c
@@ -65,12 +65,12 @@ MODULE_DESCRIPTION("IBM Power Virtual Ethernet Driver");
 MODULE_LICENSE("GPL");
 MODULE_VERSION(ibmveth_driver_version);
 
-static unsigned int tx_copybreak __read_mostly = 128;
+static unsigned int tx_copybreak __read_mostly = 2048;
 module_param(tx_copybreak, uint, 0644);
 MODULE_PARM_DESC(tx_copybreak,
 	"Maximum size of packet that is copied to a new buffer on transmit");
 
-static unsigned int rx_copybreak __read_mostly = 128;
+static unsigned int rx_copybreak __read_mostly = 2048;
 module_param(rx_copybreak, uint, 0644);
 MODULE_PARM_DESC(rx_copybreak,
 	"Maximum size of packet that is copied to a new buffer on receive");
-- 
1.7.0.4

Re: [PATCH] ibmveth: Increase default copybreak limits to 2k

[David Miller]

From: Robert Jennings <rcj@linux.vnet.ibm.com>
Date: Thu, 21 Oct 2010 09:37:53 -0500

> Increase the copybreak limits for rx and tx from 128 bytes
> to 2048 bytes. These limits were added by commits
> 8d86c61ae41d9068fd5e5cc01a4abd53c4fe3ab5 and
> c08cc3ccebd46dce44d13a8ce81d249e687eeb8a to make use of a
> bounce buffer for packets below 128 bytes.  This avoids
> tearing down and creating a TCE entry.
> 
> Performance testing shows that this default limit can be
> increased from 128 to 2048 for both rx and tx copybreak.
> This resulted in ~10% throughput increase for for packets
> that fit this limit without affecting performance for larger
> packets.
> 
> Signed-off-by: Robert Jennings <rcj@linux.vnet.ibm.com>

For a default MTU of 1500 this means the copybreak will
never be used.

This makes absolutely no sense, are you doing these tests
with Jumbo frames?

Re: [PATCH] ibmveth: Increase default copybreak limits to 2k

[Robert Jennings]

* David Miller (davem@davemloft.net) wrote:
> From: Robert Jennings <rcj@linux.vnet.ibm.com>
> 
> > Increase the copybreak limits for rx and tx from 128 bytes
> > to 2048 bytes. These limits were added by commits
> > 8d86c61ae41d9068fd5e5cc01a4abd53c4fe3ab5 and
> > c08cc3ccebd46dce44d13a8ce81d249e687eeb8a to make use of a
> > bounce buffer for packets below 128 bytes.  This avoids
> > tearing down and creating a TCE entry.
> > 
> > Performance testing shows that this default limit can be
> > increased from 128 to 2048 for both rx and tx copybreak.
> > This resulted in ~10% throughput increase for for packets
> > that fit this limit without affecting performance for larger
> > packets.
> > 
> > Signed-off-by: Robert Jennings <rcj@linux.vnet.ibm.com>
> 
> For a default MTU of 1500 this means the copybreak will
> never be used.
> 
> This makes absolutely no sense, are you doing these tests
> with Jumbo frames?

The value of copybreak is the point where we stop using the bounce
buffer and start to copy data to new buffers; having the overhead of
additional TCE entry setup/teardown.  For the default MTU of 1500,
increasing the copybreak value to 2k will result in all packets using
the bounce buffer now.  Expanding the use of the bounce buffer up to 2k
with the default MTU resulted in ~10% throughput improvement.

This was also tested with jumbo frames to verify that larger than
2k we don't use the bounce buffer and that there is no performance
regression.

I can repost the patch with these clarifications in the log message
if you would like.

--Rob Jennings

Re: [PATCH] ibmveth: Increase default copybreak limits to 2k

[David Miller]

From: Robert Jennings <rcj@linux.vnet.ibm.com>
Date: Thu, 21 Oct 2010 10:17:46 -0500

> The value of copybreak is the point where we stop using the bounce
> buffer and start to copy data to new buffers; having the overhead of
> additional TCE entry setup/teardown.  For the default MTU of 1500,
> increasing the copybreak value to 2k will result in all packets using
> the bounce buffer now.  Expanding the use of the bounce buffer up to 2k
> with the default MTU resulted in ~10% throughput improvement.

On the RX side the copybreak serves another purpose.

First of all, when you RX a < ~128 byte frame, you incur the cost
mostly of the cache miss to touch the packet headers.  The extra
allocated SKB and the copy are close to free.

But more importantly, if you always pass the RX descriptor SKB
into the network stack, even for tiny frames, the skb->truesize
(which is what gets charged to the socket) is larger than we
want it to be.

For a 128 byte frame, the socket gets charged the full MTU packet
size.  Because that is how much memory it is really using.

This is what the RX side copybreak is dealing with and you
therefore cannot remove it so trivially.

I am not applying this patch.