xdp and fragments with virtio

xdp and fragments with virtio

[David Ahern]

Hi Jason:

I am trying to test MTU changes to the BPF fib_lookup helper and seeing
something odd. Hoping you can help.

I have a VM with multiple virtio based NICs and tap backends. I install
the xdp program on eth1 and eth2 to do forwarding. In the host I send a
large packet to eth1:

$ ping -s 1500 9.9.9.9


The tap device in the host sees 2 packets:

$ sudo tcpdump -nv -i vm02-eth1
20:44:33.943160 IP (tos 0x0, ttl 64, id 58746, offset 0, flags [+],
proto ICMP (1), length 1500)
    10.100.1.254 > 9.9.9.9: ICMP echo request, id 17917, seq 1, length 1480
20:44:33.943172 IP (tos 0x0, ttl 64, id 58746, offset 1480, flags
[none], proto ICMP (1), length 48)
    10.100.1.254 > 9.9.9.9: ip-proto-1


In the VM, the XDP program only sees the first packet, not the fragment.
I added a printk to the program (see diff below):

$ cat trace_pipe
          <idle>-0     [003] ..s2   254.436467: 0: packet length 1514


Anything come to mind in the virtio xdp implementation that affects
fragment packets? I see this with both IPv4 and v6.

Thanks,
David

[1] xdp program diff showing printk that dumps packet length:

diff --git a/samples/bpf/xdp_fwd_kern.c b/samples/bpf/xdp_fwd_kern.c
index 4a6be0f87505..f119b506e782 100644
--- a/samples/bpf/xdp_fwd_kern.c
+++ b/samples/bpf/xdp_fwd_kern.c
@@ -52,6 +52,11 @@ static __always_inline int xdp_fwd_flags(struct
xdp_md *ctx, u32 flags)
        u16 h_proto;
        u64 nh_off;

+       {
+               char fmt[] = "packet length %u\n";
+
+               bpf_trace_printk(fmt, sizeof(fmt), ctx->data_end-ctx->data);
+       }
        nh_off = sizeof(*eth);
        if (data + nh_off > data_end)
                return XDP_DROP;

Re: xdp and fragments with virtio

[Jason Wang]

On 2018年05月16日 11:51, David Ahern wrote:
> Hi Jason:
>
> I am trying to test MTU changes to the BPF fib_lookup helper and seeing
> something odd. Hoping you can help.
>
> I have a VM with multiple virtio based NICs and tap backends. I install
> the xdp program on eth1 and eth2 to do forwarding. In the host I send a
> large packet to eth1:
>
> $ ping -s 1500 9.9.9.9
>
>
> The tap device in the host sees 2 packets:
>
> $ sudo tcpdump -nv -i vm02-eth1
> 20:44:33.943160 IP (tos 0x0, ttl 64, id 58746, offset 0, flags [+],
> proto ICMP (1), length 1500)
>      10.100.1.254 > 9.9.9.9: ICMP echo request, id 17917, seq 1, length 1480
> 20:44:33.943172 IP (tos 0x0, ttl 64, id 58746, offset 1480, flags
> [none], proto ICMP (1), length 48)
>      10.100.1.254 > 9.9.9.9: ip-proto-1
>
>
> In the VM, the XDP program only sees the first packet, not the fragment.
> I added a printk to the program (see diff below):
>
> $ cat trace_pipe
>            <idle>-0     [003] ..s2   254.436467: 0: packet length 1514
>
>
> Anything come to mind in the virtio xdp implementation that affects
> fragment packets? I see this with both IPv4 and v6.

Not yet. But we do turn of tap gso when virtio has XDP set, but it 
shouldn't matter this case.

Will try to see what's wrong.

Thanks

>
> Thanks,
> David
>
> [1] xdp program diff showing printk that dumps packet length:
>
> diff --git a/samples/bpf/xdp_fwd_kern.c b/samples/bpf/xdp_fwd_kern.c
> index 4a6be0f87505..f119b506e782 100644
> --- a/samples/bpf/xdp_fwd_kern.c
> +++ b/samples/bpf/xdp_fwd_kern.c
> @@ -52,6 +52,11 @@ static __always_inline int xdp_fwd_flags(struct
> xdp_md *ctx, u32 flags)
>          u16 h_proto;
>          u64 nh_off;
>
> +       {
> +               char fmt[] = "packet length %u\n";
> +
> +               bpf_trace_printk(fmt, sizeof(fmt), ctx->data_end-ctx->data);
> +       }
>          nh_off = sizeof(*eth);
>          if (data + nh_off > data_end)
>                  return XDP_DROP;
>

Re: xdp and fragments with virtio

[David Ahern]

On 5/16/18 1:24 AM, Jason Wang wrote:
> 
> 
> On 2018年05月16日 11:51, David Ahern wrote:
>> Hi Jason:
>>
>> I am trying to test MTU changes to the BPF fib_lookup helper and seeing
>> something odd. Hoping you can help.
>>
>> I have a VM with multiple virtio based NICs and tap backends. I install
>> the xdp program on eth1 and eth2 to do forwarding. In the host I send a
>> large packet to eth1:
>>
>> $ ping -s 1500 9.9.9.9
>>
>>
>> The tap device in the host sees 2 packets:
>>
>> $ sudo tcpdump -nv -i vm02-eth1
>> 20:44:33.943160 IP (tos 0x0, ttl 64, id 58746, offset 0, flags [+],
>> proto ICMP (1), length 1500)
>>      10.100.1.254 > 9.9.9.9: ICMP echo request, id 17917, seq 1,
>> length 1480
>> 20:44:33.943172 IP (tos 0x0, ttl 64, id 58746, offset 1480, flags
>> [none], proto ICMP (1), length 48)
>>      10.100.1.254 > 9.9.9.9: ip-proto-1
>>
>>
>> In the VM, the XDP program only sees the first packet, not the fragment.
>> I added a printk to the program (see diff below):
>>
>> $ cat trace_pipe
>>            <idle>-0     [003] ..s2   254.436467: 0: packet length 1514
>>
>>
>> Anything come to mind in the virtio xdp implementation that affects
>> fragment packets? I see this with both IPv4 and v6.
> 
> Not yet. But we do turn of tap gso when virtio has XDP set, but it
> shouldn't matter this case.
> 
> Will try to see what's wrong.
> 

I added this to the command line for the NICs and it works:

"mrg_rxbuf=off,guest_tso4=off,guest_tso6=off,guest_ecn=off,guest_ufo=off"

XDP program sees the full size packet and the fragment.

Fun fact: only adding mrg_rxbuf=off so that mergeable_rx_bufs is false
but big_packets is true generates a panic when it receives large packets.

Re: xdp and fragments with virtio

[Jason Wang]

On 2018年05月17日 10:55, David Ahern wrote:
> On 5/16/18 1:24 AM, Jason Wang wrote:
>>
>> On 2018年05月16日 11:51, David Ahern wrote:
>>> Hi Jason:
>>>
>>> I am trying to test MTU changes to the BPF fib_lookup helper and seeing
>>> something odd. Hoping you can help.
>>>
>>> I have a VM with multiple virtio based NICs and tap backends. I install
>>> the xdp program on eth1 and eth2 to do forwarding. In the host I send a
>>> large packet to eth1:
>>>
>>> $ ping -s 1500 9.9.9.9
>>>
>>>
>>> The tap device in the host sees 2 packets:
>>>
>>> $ sudo tcpdump -nv -i vm02-eth1
>>> 20:44:33.943160 IP (tos 0x0, ttl 64, id 58746, offset 0, flags [+],
>>> proto ICMP (1), length 1500)
>>>       10.100.1.254 > 9.9.9.9: ICMP echo request, id 17917, seq 1,
>>> length 1480
>>> 20:44:33.943172 IP (tos 0x0, ttl 64, id 58746, offset 1480, flags
>>> [none], proto ICMP (1), length 48)
>>>       10.100.1.254 > 9.9.9.9: ip-proto-1
>>>
>>>
>>> In the VM, the XDP program only sees the first packet, not the fragment.
>>> I added a printk to the program (see diff below):
>>>
>>> $ cat trace_pipe
>>>             <idle>-0     [003] ..s2   254.436467: 0: packet length 1514
>>>
>>>
>>> Anything come to mind in the virtio xdp implementation that affects
>>> fragment packets? I see this with both IPv4 and v6.
>> Not yet. But we do turn of tap gso when virtio has XDP set, but it
>> shouldn't matter this case.
>>
>> Will try to see what's wrong.
>>
> I added this to the command line for the NICs and it works:
>
> "mrg_rxbuf=off,guest_tso4=off,guest_tso6=off,guest_ecn=off,guest_ufo=off"
>
> XDP program sees the full size packet and the fragment.
>
> Fun fact: only adding mrg_rxbuf=off so that mergeable_rx_bufs is false
> but big_packets is true generates a panic when it receives large packets.

It looks like we wrongly drop packets after linearizing the packets 
during XDP_REDIRECT.

Please try the patch (but I do spot some other issues, will post a series):

Thanks

diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index f34794a..59702f9 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -800,7 +800,7 @@ static struct sk_buff *receive_mergeable(struct 
net_device *dev,
                         }
                         *xdp_xmit = true;
                         if (unlikely(xdp_page != page))
-                               goto err_xdp;
+                               put_page(page);
                         rcu_read_unlock();
                         goto xdp_xmit;
                 default:

Re: xdp and fragments with virtio

[David Ahern]

On 5/17/18 3:24 AM, Jason Wang wrote:
> It looks like we wrongly drop packets after linearizing the packets
> during XDP_REDIRECT.
> 
> Please try the patch (but I do spot some other issues, will post a series):
> 
> Thanks
> 
> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
> index f34794a..59702f9 100644
> --- a/drivers/net/virtio_net.c
> +++ b/drivers/net/virtio_net.c
> @@ -800,7 +800,7 @@ static struct sk_buff *receive_mergeable(struct
> net_device *dev,
>                         }
>                         *xdp_xmit = true;
>                         if (unlikely(xdp_page != page))
> -                               goto err_xdp;
> +                               put_page(page);
>                         rcu_read_unlock();
>                         goto xdp_xmit;
>                 default:

Yes, that does solve the problem of fragments getting lost.