RE: Difficulties to get 1Gbps on be2net ethernet card

[Eric Dumazet <eric.dumazet@gmail.com>]

On Wed, 2012-05-30 at 03:04 -0700, Sathya.Perla@Emulex.Com wrote:
> >-----Original Message-----
> >From: netdev-owner@vger.kernel.org [mailto:netdev-owner@vger.kernel.org] On
> >Behalf Of Jean-Michel Hautbois
> >
> >2012/5/30 Jean-Michel Hautbois <jhautbois@gmail.com>:
> >
> >I used vmstat in order to see the differences between the two kernels.
> >The main difference is the number of interrupts per second.
> >I have an average of 87500 on 3.2 and 7500 on 2.6, 10 times lower !
> >I suspect the be2net driver to be the main cause, and I checkes the
> >/proc/interrupts file in order to be sure.
> >
> >I have for eth1-tx on 2.6.26 about 2200 interrupts per second and 23000 on 3.2.
> >BTW, it is named eth1-q0 on 3.2 (and tx and rx are the same IRQ)
> >whereas there is eth1-rx0 and eth1-tx on 2.6.26.
> 
> Yes, there is an issue with be2net interrupt mitigation in the recent code with
> RX and TX on the same Evt-Q (commit 10ef9ab4). The high interrupt rate happens when a TX blast is
> done while RX is relatively silent on a queue pair. Interrupt rate due to TX completions is not being
> mitigated.
> 
> I have a fix and will send it out soon..

I also have a benet fix for non GRO :

Pulling 64 bytes in skb head is too much for TCP IPv4 with no
timestamps, as this makes splice() or TCP coalescing less effective.

(Having tcp payload in linear part of the skb disables various optims)

Could you please test it ?

Thanks

diff --git a/drivers/net/ethernet/emulex/benet/be_main.c b/drivers/net/ethernet/emulex/benet/be_main.c
index 08efd30..f446b11 100644
--- a/drivers/net/ethernet/emulex/benet/be_main.c
+++ b/drivers/net/ethernet/emulex/benet/be_main.c
@@ -1202,15 +1202,19 @@ static void skb_fill_rx_data(struct be_rx_obj *rxo, struct sk_buff *skb,
 	/* Copy data in the first descriptor of this completion */
 	curr_frag_len = min(rxcp->pkt_size, rx_frag_size);
 
-	/* Copy the header portion into skb_data */
-	hdr_len = min(BE_HDR_LEN, curr_frag_len);
+	/* If frame is small enough to fit in skb->head, pull it completely.
+	 * If not, only pull ethernet header so that splice() or TCP coalesce
+	 * are more efficient.
+	 */
+	hdr_len = (curr_frag_len <= skb_tailroom(skb)) ?
+			curr_frag_len : ETH_HLEN;
+
 	memcpy(skb->data, start, hdr_len);
 	skb->len = curr_frag_len;
-	if (curr_frag_len <= BE_HDR_LEN) { /* tiny packet */
+	skb->tail += hdr_len;
+	if (hdr_len == curr_frag_len) { /* tiny packet */
 		/* Complete packet has now been moved to data */
 		put_page(page_info->page);
-		skb->data_len = 0;
-		skb->tail += curr_frag_len;
 	} else {
 		skb_shinfo(skb)->nr_frags = 1;
 		skb_frag_set_page(skb, 0, page_info->page);
@@ -1219,7 +1223,6 @@ static void skb_fill_rx_data(struct be_rx_obj *rxo, struct sk_buff *skb,
 		skb_frag_size_set(&skb_shinfo(skb)->frags[0], curr_frag_len - hdr_len);
 		skb->data_len = curr_frag_len - hdr_len;
 		skb->truesize += rx_frag_size;
-		skb->tail += hdr_len;
 	}
 	page_info->page = NULL;