[PATCH net] xen-netback: Fix slot estimation

[PATCH net] xen-netback: Fix slot estimation

[Zoltan Kiss]

A recent commit (a02eb4 "xen-netback: worse-case estimate in xenvif_rx_action is
underestimating") capped the slot estimation to MAX_SKB_FRAGS, but that triggers
the next BUG_ON a few lines down, as the packet consumes more slots than
estimated.
This patch remove that cap, and if the frontend doesn't provide enough slot,
put back the skb to the top of the queue and caps rx_last_skb_slots. When the
next try also fails, it drops the packet.
Capping rx_last_skb_slots is needed because if the frontend never gives enough
slots, the ring gets stalled.

Signed-off-by: Zoltan Kiss <zoltan.kiss@citrix.com>
Cc: Paul Durrant <paul.durrant@citrix.com>
Cc: Wei Liu <wei.liu2@citrix.com>
Cc: Ian Campbell <ian.campbell@citrix.com>
Cc: David Vrabel <david.vrabel@citrix.com>

diff --git a/drivers/net/xen-netback/netback.c b/drivers/net/xen-netback/netback.c
index da85ffb..7164157 100644
--- a/drivers/net/xen-netback/netback.c
+++ b/drivers/net/xen-netback/netback.c
@@ -600,13 +600,6 @@ static void xenvif_rx_action(struct xenvif *vif)
 							 PAGE_SIZE);
 		}
 
-		/* To avoid the estimate becoming too pessimal for some
-		 * frontends that limit posted rx requests, cap the estimate
-		 * at MAX_SKB_FRAGS.
-		 */
-		if (max_slots_needed > MAX_SKB_FRAGS)
-			max_slots_needed = MAX_SKB_FRAGS;
-
 		/* We may need one more slot for GSO metadata */
 		if (skb_is_gso(skb) &&
 		   (skb_shinfo(skb)->gso_type & SKB_GSO_TCPV4 ||
@@ -615,9 +608,27 @@ static void xenvif_rx_action(struct xenvif *vif)
 
 		/* If the skb may not fit then bail out now */
 		if (!xenvif_rx_ring_slots_available(vif, max_slots_needed)) {
+			/* If the skb needs more than MAX_SKB_FRAGS slots, it
+			 * can happen that the frontend never gives us enough.
+			 * To avoid spining on that packet, first we put it back
+			 * to the top of the queue, but if the next try fail,
+			 * we drop it.
+			 */
+			if (max_slots_needed > MAX_SKB_FRAGS &&
+			    vif->rx_last_skb_slots == MAX_SKB_FRAGS) {
+				kfree_skb(skb);
+				vif->rx_last_skb_slots = 0;
+				continue;
+			}
 			skb_queue_head(&vif->rx_queue, skb);
 			need_to_notify = true;
-			vif->rx_last_skb_slots = max_slots_needed;
+			/* Cap this otherwise if the guest never gives us
+			 * enough slot, rx_work_todo will spin
+			 */
+			vif->rx_last_skb_slots =
+				max_slots_needed > MAX_SKB_FRAGS ?
+				MAX_SKB_FRAGS :
+				max_slots_needed;
 			break;
 		} else
 			vif->rx_last_skb_slots = 0;

RE: [PATCH net] xen-netback: Fix slot estimation

[Paul Durrant]

> -----Original Message-----
> From: Zoltan Kiss
> Sent: 03 June 2014 14:32
> To: xen-devel@lists.xenproject.org; Ian Campbell; Wei Liu; Paul Durrant;
> linux@eikelenboom.it
> Cc: netdev@vger.kernel.org; David Vrabel; davem@davemloft.net; Zoltan
> Kiss
> Subject: [PATCH net] xen-netback: Fix slot estimation
> 
> A recent commit (a02eb4 "xen-netback: worse-case estimate in
> xenvif_rx_action is
> underestimating") capped the slot estimation to MAX_SKB_FRAGS, but that
> triggers
> the next BUG_ON a few lines down, as the packet consumes more slots than
> estimated.
> This patch remove that cap, and if the frontend doesn't provide enough slot,
> put back the skb to the top of the queue and caps rx_last_skb_slots. When
> the
> next try also fails, it drops the packet.
> Capping rx_last_skb_slots is needed because if the frontend never gives
> enough
> slots, the ring gets stalled.
> 
> Signed-off-by: Zoltan Kiss <zoltan.kiss@citrix.com>
> Cc: Paul Durrant <paul.durrant@citrix.com>
> Cc: Wei Liu <wei.liu2@citrix.com>
> Cc: Ian Campbell <ian.campbell@citrix.com>
> Cc: David Vrabel <david.vrabel@citrix.com>
> 
> diff --git a/drivers/net/xen-netback/netback.c b/drivers/net/xen-
> netback/netback.c
> index da85ffb..7164157 100644
> --- a/drivers/net/xen-netback/netback.c
> +++ b/drivers/net/xen-netback/netback.c
> @@ -600,13 +600,6 @@ static void xenvif_rx_action(struct xenvif *vif)
>  							 PAGE_SIZE);
>  		}
> 
> -		/* To avoid the estimate becoming too pessimal for some
> -		 * frontends that limit posted rx requests, cap the estimate
> -		 * at MAX_SKB_FRAGS.
> -		 */
> -		if (max_slots_needed > MAX_SKB_FRAGS)
> -			max_slots_needed = MAX_SKB_FRAGS;
> -
>  		/* We may need one more slot for GSO metadata */
>  		if (skb_is_gso(skb) &&
>  		   (skb_shinfo(skb)->gso_type & SKB_GSO_TCPV4 ||
> @@ -615,9 +608,27 @@ static void xenvif_rx_action(struct xenvif *vif)
> 
>  		/* If the skb may not fit then bail out now */
>  		if (!xenvif_rx_ring_slots_available(vif, max_slots_needed)) {
> +			/* If the skb needs more than MAX_SKB_FRAGS
> slots, it
> +			 * can happen that the frontend never gives us
> enough.
> +			 * To avoid spining on that packet, first we put it back
> +			 * to the top of the queue, but if the next try fail,
> +			 * we drop it.
> +			 */
> +			if (max_slots_needed > MAX_SKB_FRAGS &&
> +			    vif->rx_last_skb_slots == MAX_SKB_FRAGS) {

Isn't it sufficient to say:

if (vif->rx_last_skb_slots != 0)

here? We should not ordinarily wake before the requisite number of slots is available.

  Paul

> +				kfree_skb(skb);
> +				vif->rx_last_skb_slots = 0;
> +				continue;
> +			}
>  			skb_queue_head(&vif->rx_queue, skb);
>  			need_to_notify = true;
> -			vif->rx_last_skb_slots = max_slots_needed;
> +			/* Cap this otherwise if the guest never gives us
> +			 * enough slot, rx_work_todo will spin
> +			 */
> +			vif->rx_last_skb_slots =
> +				max_slots_needed > MAX_SKB_FRAGS ?
> +				MAX_SKB_FRAGS :
> +				max_slots_needed;
>  			break;
>  		} else
>  			vif->rx_last_skb_slots = 0;

Re: [PATCH net] xen-netback: Fix slot estimation

[Zoltan Kiss]

On 03/06/14 14:37, Paul Durrant wrote:
>> -----Original Message-----
>> From: Zoltan Kiss
>> Sent: 03 June 2014 14:32
>> To: xen-devel@lists.xenproject.org; Ian Campbell; Wei Liu; Paul Durrant;
>> linux@eikelenboom.it
>> Cc: netdev@vger.kernel.org; David Vrabel; davem@davemloft.net; Zoltan
>> Kiss
>> Subject: [PATCH net] xen-netback: Fix slot estimation
>>
>> A recent commit (a02eb4 "xen-netback: worse-case estimate in
>> xenvif_rx_action is
>> underestimating") capped the slot estimation to MAX_SKB_FRAGS, but that
>> triggers
>> the next BUG_ON a few lines down, as the packet consumes more slots than
>> estimated.
>> This patch remove that cap, and if the frontend doesn't provide enough slot,
>> put back the skb to the top of the queue and caps rx_last_skb_slots. When
>> the
>> next try also fails, it drops the packet.
>> Capping rx_last_skb_slots is needed because if the frontend never gives
>> enough
>> slots, the ring gets stalled.
>>
>> Signed-off-by: Zoltan Kiss <zoltan.kiss@citrix.com>
>> Cc: Paul Durrant <paul.durrant@citrix.com>
>> Cc: Wei Liu <wei.liu2@citrix.com>
>> Cc: Ian Campbell <ian.campbell@citrix.com>
>> Cc: David Vrabel <david.vrabel@citrix.com>
>>
>> diff --git a/drivers/net/xen-netback/netback.c b/drivers/net/xen-
>> netback/netback.c
>> index da85ffb..7164157 100644
>> --- a/drivers/net/xen-netback/netback.c
>> +++ b/drivers/net/xen-netback/netback.c
>> @@ -600,13 +600,6 @@ static void xenvif_rx_action(struct xenvif *vif)
>>   							 PAGE_SIZE);
>>   		}
>>
>> -		/* To avoid the estimate becoming too pessimal for some
>> -		 * frontends that limit posted rx requests, cap the estimate
>> -		 * at MAX_SKB_FRAGS.
>> -		 */
>> -		if (max_slots_needed > MAX_SKB_FRAGS)
>> -			max_slots_needed = MAX_SKB_FRAGS;
>> -
>>   		/* We may need one more slot for GSO metadata */
>>   		if (skb_is_gso(skb) &&
>>   		   (skb_shinfo(skb)->gso_type & SKB_GSO_TCPV4 ||
>> @@ -615,9 +608,27 @@ static void xenvif_rx_action(struct xenvif *vif)
>>
>>   		/* If the skb may not fit then bail out now */
>>   		if (!xenvif_rx_ring_slots_available(vif, max_slots_needed)) {
>> +			/* If the skb needs more than MAX_SKB_FRAGS
>> slots, it
>> +			 * can happen that the frontend never gives us
>> enough.
>> +			 * To avoid spining on that packet, first we put it back
>> +			 * to the top of the queue, but if the next try fail,
>> +			 * we drop it.
>> +			 */
>> +			if (max_slots_needed > MAX_SKB_FRAGS &&
>> +			    vif->rx_last_skb_slots == MAX_SKB_FRAGS) {
>
> Isn't it sufficient to say:
>
> if (vif->rx_last_skb_slots != 0)
>
> here? We should not ordinarily wake before the requisite number of slots is available.

Yep, that would be enough


>
>    Paul
>
>> +				kfree_skb(skb);
>> +				vif->rx_last_skb_slots = 0;
>> +				continue;
>> +			}
>>   			skb_queue_head(&vif->rx_queue, skb);
>>   			need_to_notify = true;
>> -			vif->rx_last_skb_slots = max_slots_needed;
>> +			/* Cap this otherwise if the guest never gives us
>> +			 * enough slot, rx_work_todo will spin
>> +			 */
>> +			vif->rx_last_skb_slots =
>> +				max_slots_needed > MAX_SKB_FRAGS ?
>> +				MAX_SKB_FRAGS :
>> +				max_slots_needed;
>>   			break;
>>   		} else
>>   			vif->rx_last_skb_slots = 0;

RE: [PATCH net] xen-netback: Fix slot estimation

[David Laight]

From: netdev-owner@vger.kernel.org
> A recent commit (a02eb4 "xen-netback: worse-case estimate in xenvif_rx_action is
> underestimating") capped the slot estimation to MAX_SKB_FRAGS, but that triggers
> the next BUG_ON a few lines down, as the packet consumes more slots than
> estimated.
> This patch remove that cap, and if the frontend doesn't provide enough slot,
> put back the skb to the top of the queue and caps rx_last_skb_slots. When the
> next try also fails, it drops the packet.
> Capping rx_last_skb_slots is needed because if the frontend never gives enough
> slots, the ring gets stalled.
> 
> Signed-off-by: Zoltan Kiss <zoltan.kiss@citrix.com>
> Cc: Paul Durrant <paul.durrant@citrix.com>
> Cc: Wei Liu <wei.liu2@citrix.com>
> Cc: Ian Campbell <ian.campbell@citrix.com>
> Cc: David Vrabel <david.vrabel@citrix.com>
> 
> diff --git a/drivers/net/xen-netback/netback.c b/drivers/net/xen-netback/netback.c
> index da85ffb..7164157 100644
> --- a/drivers/net/xen-netback/netback.c
> +++ b/drivers/net/xen-netback/netback.c
> @@ -600,13 +600,6 @@ static void xenvif_rx_action(struct xenvif *vif)
>  							 PAGE_SIZE);
>  		}
> 
> -		/* To avoid the estimate becoming too pessimal for some
> -		 * frontends that limit posted rx requests, cap the estimate
> -		 * at MAX_SKB_FRAGS.
> -		 */
> -		if (max_slots_needed > MAX_SKB_FRAGS)
> -			max_slots_needed = MAX_SKB_FRAGS;
> -
>  		/* We may need one more slot for GSO metadata */
>  		if (skb_is_gso(skb) &&
>  		   (skb_shinfo(skb)->gso_type & SKB_GSO_TCPV4 ||
> @@ -615,9 +608,27 @@ static void xenvif_rx_action(struct xenvif *vif)
> 
>  		/* If the skb may not fit then bail out now */
>  		if (!xenvif_rx_ring_slots_available(vif, max_slots_needed)) {
> +			/* If the skb needs more than MAX_SKB_FRAGS slots, it
> +			 * can happen that the frontend never gives us enough.
> +			 * To avoid spining on that packet, first we put it back
> +			 * to the top of the queue, but if the next try fail,
> +			 * we drop it.
> +			 */
> +			if (max_slots_needed > MAX_SKB_FRAGS &&
> +			    vif->rx_last_skb_slots == MAX_SKB_FRAGS) {
> +				kfree_skb(skb);
> +				vif->rx_last_skb_slots = 0;
> +				continue;
> +			}

A silent discard here doesn't seem right at all.
While it stops the kernel crashing, or the entire interface locking
up; it is likely to leave one connection 'stuck' - a TCP retransmission
is likely to include the same fragments.
>From a user point of view this as almost as bad.

	David

>  			skb_queue_head(&vif->rx_queue, skb);
>  			need_to_notify = true;
> -			vif->rx_last_skb_slots = max_slots_needed;
> +			/* Cap this otherwise if the guest never gives us
> +			 * enough slot, rx_work_todo will spin
> +			 */
> +			vif->rx_last_skb_slots =
> +				max_slots_needed > MAX_SKB_FRAGS ?
> +				MAX_SKB_FRAGS :
> +				max_slots_needed;
>  			break;
>  		} else
>  			vif->rx_last_skb_slots = 0;
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH net] xen-netback: Fix slot estimation

[Zoltan Kiss]

On 03/06/14 14:52, David Laight wrote:
> From: netdev-owner@vger.kernel.org
>> @@ -615,9 +608,27 @@ static void xenvif_rx_action(struct xenvif *vif)
>>
>>   		/* If the skb may not fit then bail out now */
>>   		if (!xenvif_rx_ring_slots_available(vif, max_slots_needed)) {
>> +			/* If the skb needs more than MAX_SKB_FRAGS slots, it
>> +			 * can happen that the frontend never gives us enough.
>> +			 * To avoid spining on that packet, first we put it back
>> +			 * to the top of the queue, but if the next try fail,
>> +			 * we drop it.
>> +			 */
>> +			if (max_slots_needed > MAX_SKB_FRAGS &&
>> +			    vif->rx_last_skb_slots == MAX_SKB_FRAGS) {
>> +				kfree_skb(skb);
>> +				vif->rx_last_skb_slots = 0;
>> +				continue;
>> +			}
>
> A silent discard here doesn't seem right at all.
> While it stops the kernel crashing, or the entire interface locking
> up; it is likely to leave one connection 'stuck' - a TCP retransmission
> is likely to include the same fragments.
>  From a user point of view this as almost as bad.

Yes, we are aware of this problem for a while. However I have an idea to 
solve that in a way that we don't lose performance, and these packets 
can pass through as well. See my patch called "Fix handling of skbs 
requiring too many slots"

Zoli

Re: [PATCH net] xen-netback: Fix slot estimation

[David Miller]

From: Zoltan Kiss <zoltan.kiss@citrix.com>
Date: Tue, 3 Jun 2014 14:32:16 +0100

> A recent commit (a02eb4 "xen-netback: worse-case estimate in xenvif_rx_action is
> underestimating") capped the slot estimation to MAX_SKB_FRAGS, but that triggers
> the next BUG_ON a few lines down, as the packet consumes more slots than
> estimated.
> This patch remove that cap, and if the frontend doesn't provide enough slot,
> put back the skb to the top of the queue and caps rx_last_skb_slots. When the
> next try also fails, it drops the packet.
> Capping rx_last_skb_slots is needed because if the frontend never gives enough
> slots, the ring gets stalled.
> 
> Signed-off-by: Zoltan Kiss <zoltan.kiss@citrix.com>

Like David Laight, I do not like this patch at all.

Yes a crash or BUG_ON triggered is bad, but fixing it by deadlocking
TCP connections (a silent failure) is not an improvement.

I'm not applying this, sorry.

Re: [PATCH net] xen-netback: Fix slot estimation

[Zoltan Kiss]

On 05/06/14 23:02, David Miller wrote:
> From: Zoltan Kiss<zoltan.kiss@citrix.com>
> Date: Tue, 3 Jun 2014 14:32:16 +0100
>
>> A recent commit (a02eb4 "xen-netback: worse-case estimate in xenvif_rx_action is
>> underestimating") capped the slot estimation to MAX_SKB_FRAGS, but that triggers
>> the next BUG_ON a few lines down, as the packet consumes more slots than
>> estimated.
>> This patch remove that cap, and if the frontend doesn't provide enough slot,
>> put back the skb to the top of the queue and caps rx_last_skb_slots. When the
>> next try also fails, it drops the packet.
>> Capping rx_last_skb_slots is needed because if the frontend never gives enough
>> slots, the ring gets stalled.
>>
>> Signed-off-by: Zoltan Kiss<zoltan.kiss@citrix.com>
> Like David Laight, I do not like this patch at all.
>
> Yes a crash or BUG_ON triggered is bad, but fixing it by deadlocking
> TCP connections (a silent failure) is not an improvement.
>
> I'm not applying this, sorry.
>
Hi,

I haven't wrote it explicitly, but my other patch "xen-netback: Fix 
handling of skbs requiring too many slots" supersedes this one.

Re: [PATCH net] xen-netback: Fix slot estimation

[David Miller]

From: Zoltan Kiss <zoltan.kiss@schaman.hu>
Date: Fri, 06 Jun 2014 11:20:45 +0100

> I haven't wrote it explicitly, but my other patch "xen-netback: Fix
> handling of skbs requiring too many slots" supersedes this one.

Right, I figured this out after reading these threads over again.