Re: xennet: skb rides the rocket: 20 slots

[ANNIE LI <annie.li@oracle.com>]

On 2013-1-7 18:55, Ian Campbell wrote:
> On Fri, 2013-01-04 at 16:28 +0000, Sander Eikelenboom wrote:
>> Hi Ian,
> I'm not actually the netfront maintainer. Adding Konrad. Annie too since
> IIRC she was at one point looking at various buffer coalescing schemes.
>
>> Today i fired up an old VM with a bittorrent client, trying to
>> download some torrents.
> How old? Which kernel?
>
>> I seem to be hitting the unlikely case of "xennet: skb rides the
>> rocket: xx slots" and this results in some dropped packets in domU, I
>> don't see any warnings in dom0.

If debug is enabled, following netback code should print out messages in 
netbk_count_requests.

                 if (unlikely(frags >= MAX_SKB_FRAGS)) {
                         netdev_dbg(vif->dev, "Too many frags\n");
                         return -frags;
                 }

>>
>> I have added some extra info, but i don't have enough knowledge if
>> this could/should be prevented from happening ?
> MAX_SKB_FRAGS has never, AFAIK, been as big as 19 or 20 so I'm not sure
> this can have ever worked.
>
> These SKBs seem to be pretty big (not quite 64KB), seemingly most of the
> data is contained in a smallish number of frags, which suggests compound
> pages and therefore a reasonably modern kernel?
>
> This probably relates somewhat to the issues described in the
> "netchannel vs MAX_SKB_FRAGS" thread last year, or at least the solution
> to that would necessarily involve fixing this issue.

If netback complains about "Too many frags", then it should be 
MAX_SKB_FRAGS limitation in netback results in dropping packets in 
netfront. It is possible that other netfronts(windows?) also hit this.

Thanks
Annie
>
> In the meantime you could try disabling sg and/or various offloads for
> that domain's vif.
>
>> [16798.629141] xennet: skb rides the rocket: 19 slots MAX_SKB_FRAGS: 17 div_roundup:1 xennet_count_skb_frag_slots:18 offset:106 skb_headlen:1622 skb->len:64294, skb->data_len:62672 skb->truesize:64976 nr_frags:4 page_size:4096 prot:0800
>> [16800.575182] xennet: skb rides the rocket: 19 slots MAX_SKB_FRAGS: 17 div_roundup:1 xennet_count_skb_frag_slots:18 offset:2154 skb_headlen:1622 skb->len:64294, skb->data_len:62672 skb->truesize:64976 nr_frags:4 page_size:4096 prot:0800
>> [16801.589166] xennet: skb rides the rocket: 20 slots MAX_SKB_FRAGS: 17 div_roundup:1 xennet_count_skb_frag_slots:19 offset:2154 skb_headlen:1622 skb->len:64294, skb->data_len:62672 skb->truesize:64976 nr_frags:5 page_size:4096 prot:0800
>> [16803.279039] xennet: skb rides the rocket: 19 slots MAX_SKB_FRAGS: 17 div_roundup:1 xennet_count_skb_frag_slots:18 offset:106 skb_headlen:1622 skb->len:64294, skb->data_len:62672 skb->truesize:64976 nr_frags:3 page_size:4096 prot:0800
>> [16809.973268] xennet: skb rides the rocket: 19 slots MAX_SKB_FRAGS: 17 div_roundup:1 xennet_count_skb_frag_slots:18 offset:2154 skb_headlen:1622 skb->len:64294, skb->data_len:62672 skb->truesize:64976 nr_frags:4 page_size:4096 prot:0800
>> [16811.420048] xennet: skb rides the rocket: 19 slots MAX_SKB_FRAGS: 17 div_roundup:1 xennet_count_skb_frag_slots:18 offset:2154 skb_headlen:1622 skb->len:64294, skb->data_len:62672 skb->truesize:64976 nr_frags:6 page_size:4096 prot:0800
>> [16814.872686] xennet: skb rides the rocket: 19 slots MAX_SKB_FRAGS: 17 div_roundup:1 xennet_count_skb_frag_slots:18 offset:106 skb_headlen:1622 skb->len:64294, skb->data_len:62672 skb->truesize:64976 nr_frags:5 page_size:4096 prot:0800
>> [16815.359099] xennet: skb rides the rocket: 21 slots MAX_SKB_FRAGS: 17 div_roundup:1 xennet_count_skb_frag_slots:20 offset:2154 skb_headlen:1622 skb->len:64294, skb->data_len:62672 skb->truesize:64976 nr_frags:5 page_size:4096 prot:0800
>> [16825.851906] xennet: skb rides the rocket: 20 slots MAX_SKB_FRAGS: 17 div_roundup:1 xennet_count_skb_frag_slots:19 offset:106 skb_headlen:1622 skb->len:64294, skb->data_len:62672 skb->truesize:64976 nr_frags:4 page_size:4096 prot:0800
>> [16828.295083] xennet: skb rides the rocket: 19 slots MAX_SKB_FRAGS: 17 div_roundup:1 xennet_count_skb_frag_slots:18 offset:2154 skb_headlen:1622 skb->len:64294, skb->data_len:62672 skb->truesize:64976 nr_frags:5 page_size:4096 prot:0800
>> [16837.386684] xennet: skb rides the rocket: 19 slots MAX_SKB_FRAGS: 17 div_roundup:1 xennet_count_skb_frag_slots:18 offset:106 skb_headlen:1622 skb->len:64294, skb->data_len:62672 skb->truesize:64976 nr_frags:4 page_size:4096 prot:0800
>> [16838.609683] xennet: skb rides the rocket: 20 slots MAX_SKB_FRAGS: 17 div_roundup:1 xennet_count_skb_frag_slots:19 offset:106 skb_headlen:1622 skb->len:64294, skb->data_len:62672 skb->truesize:64976 nr_frags:4 page_size:4096 prot:0800
>> [16841.783729] xennet: skb rides the rocket: 19 slots MAX_SKB_FRAGS: 17 div_roundup:1 xennet_count_skb_frag_slots:18 offset:106 skb_headlen:1622 skb->len:64294, skb->data_len:62672 skb->truesize:64976 nr_frags:6 page_size:4096 prot:0800
>> [16843.841678] xennet: skb rides the rocket: 20 slots MAX_SKB_FRAGS: 17 div_roundup:1 xennet_count_skb_frag_slots:19 offset:2154 skb_headlen:1622 skb->len:64294, skb->data_len:62672 skb->truesize:64976 nr_frags:4 page_size:4096 prot:0800
>> [16849.847614] xennet: skb rides the rocket: 19 slots MAX_SKB_FRAGS: 17 div_roundup:1 xennet_count_skb_frag_slots:18 offset:3690 skb_headlen:54 skb->len:64294, skb->data_len:64240 skb->truesize:65008 nr_frags:4 page_size:4096 prot:0800
>> [16853.787128] xennet: skb rides the rocket: 20 slots MAX_SKB_FRAGS: 17 div_roundup:1 xennet_count_skb_frag_slots:19 offset:2154 skb_headlen:1622 skb->len:64294, skb->data_len:62672 skb->truesize:64976 nr_frags:4 page_size:4096 prot:0800
>>
>> # ifconfig
>> eth0      Link encap:Ethernet  HWaddr 00:16:3e:c4:20:46
>>            inet addr:192.168.1.12  Bcast:192.168.1.255  Mask:255.255.255.0
>>            UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>>            RX packets:9090214 errors:0 dropped:0 overruns:0 frame:0
>>            TX packets:5902090 errors:0 dropped:304 overruns:0 carrier:0
>>            collisions:0 txqueuelen:1000
>>            RX bytes:19757934770 (18.4 GiB)  TX bytes:16855238200 (15.6 GiB)
>>            Interrupt:25
>>
>>
>>
>>
>> diff --git a/drivers/net/xen-netfront.c b/drivers/net/xen-netfront.c
>> index c26e28b..50ac403 100644
>> --- a/drivers/net/xen-netfront.c
>> +++ b/drivers/net/xen-netfront.c
>> @@ -552,7 +552,11 @@ static int xennet_start_xmit(struct sk_buff *skb, struct net_device *dev)
>>                  xennet_count_skb_frag_slots(skb);
>>          if (unlikely(slots>  MAX_SKB_FRAGS + 1)) {
>>                  net_alert_ratelimited(
>> -                       "xennet: skb rides the rocket: %d slots\n", slots);
>> +                       "xennet: skb rides the rocket: %d slots MAX_SKB_FRAGS: %d div_roundup:%d xennet_count_skb_frag_slots:%d offset:%d skb_headlen:%d skb->len:%d, skb->data_len:%d skb->truesize:%d nr_frags:%d page_size:%d prot:%04x \n
>> +                             slots, MAX_SKB_FRAGS,DIV_ROUND_UP(offset + len, PAGE_SIZE),
>> +                             xennet_count_skb_frag_slots(skb), offset, len, skb->len,
>> +                             skb->data_len, skb->truesize, skb_shinfo(skb)->nr_frags,
>> +                             PAGE_SIZE, ntohs(skb->protocol));
>>                  goto drop;
>>          }
>>
>