Re: Virtio_net support vxlan encapsulation package TSO offload discuss

[Jason Wang <jasowang@redhat.com>]

On 2016年11月08日 19:58, Zhangming (James, Euler) wrote:
> On 2016年11月08日 19:17, Jason Wang wrote:
>
>> On 2016年11月08日 19:13, Jason Wang wrote:
>>> Cc Michael
>>>
>>> On 2016年11月08日 16:34, Zhangming (James, Euler) wrote:
>>>> In container scenario, OVS is installed in the Virtual machine, and
>>>> all the containers connected to the OVS will communicated through
>>>> VXLAN encapsulation.
>>>>
>>>> By now, virtio_net does not support TSO offload for VXLAN
>>>> encapsulated TSO package. In this condition, the performance is not
>>>> good, sender is bottleneck
>>>>
>>>> I googled this scenario, but I didn’t find any information. Will
>>>> virtio_net support VXLAN encapsulation package TSO offload later?
>>>>
>>> Yes and for both sender and receiver.
>>>
>>>> My idea is virtio_net open encapsulated TSO offload, and transport
>>>> encapsulation info to TUN, TUN will parse the info and build skb with
>>>> encapsulation info.
>>>>
>>>> OVS or kernel on the host should be modified to support this. Using
>>>> this method, the TCP performance aremore than 2x as before.
>>>>
>>>> Any advice and suggestions for this idea or new idea will be greatly
>>>> appreciated!
>>>>
>>>> Best regards,
>>>>
>>>>     James zhang
>>>>
>>> Sounds very good. And we may also need features bits
>>> (VIRTIO_NET_F_GUEST|HOST_GSO_X) for this.
>>>
>>> This is in fact one of items in networking todo list. (See
>>> http://www.linux-kvm.org/page/NetworkingTodo). While at it, we'd
>>> better support not only VXLAN but also other tunnels.
>> Cc Vlad who is working on extending virtio-net headers.
>>
>>> We can start with the spec work, or if you've already had some bits
>>> you can post them as RFC for early review.
>>>
>>> Thanks
> Below is my demo code
> Virtio_net.c
> static int virtnet_probe(struct virtio_device *vdev), add belows codes:
>          if (virtio_has_feature(vdev, VIRTIO_NET_F_MRG_RXBUF) ||				// avoid gso segment, it should be negotiation later, because in the demo I reuse num_buffers.
>              virtio_has_feature(vdev, VIRTIO_F_VERSION_1)) {
>                  dev->hw_enc_features |= NETIF_F_TSO;
>                  dev->hw_enc_features |= NETIF_F_ALL_CSUM;
>                  dev->hw_enc_features |= NETIF_F_GSO_UDP_TUNNEL;
>                  dev->hw_enc_features |= NETIF_F_GSO_UDP_TUNNEL_CSUM;
>                  dev->hw_enc_features |= NETIF_F_GSO_TUNNEL_REMCSUM;
>
>                  dev->features |= NETIF_F_GSO_UDP_TUNNEL;
>                  dev->features |= NETIF_F_GSO_UDP_TUNNEL_CSUM;
>                  dev->features |= NETIF_F_GSO_TUNNEL_REMCSUM;
>          }
>
> static int xmit_skb(struct send_queue *sq, struct sk_buff *skb), add below to pieces of codes
>
>                  if (skb_shinfo(skb)->gso_type & SKB_GSO_UDP_TUNNEL)
>                          hdr->hdr.gso_type |= VIRTIO_NET_HDR_GSO_TUNNEL;
>                  if (skb_shinfo(skb)->gso_type & SKB_GSO_UDP_TUNNEL_CSUM)
>                          hdr->hdr.gso_type |= VIRTIO_NET_HDR_GSO_TUNNEL_CSUM;
>                  if (skb_shinfo(skb)->gso_type & SKB_GSO_TUNNEL_REMCSUM)
>                          hdr->hdr.gso_type |= VIRTIO_NET_HDR_GSO_TUNNEL_REMCSUM;
>
>          if (skb->encapsulation && skb_is_gso(skb)) {
>                  inner_mac_len = skb_inner_network_header(skb) - skb_inner_mac_header(skb);
>                  tnl_len = skb_inner_mac_header(skb) - skb_mac_header(skb);
>                  if ( !(inner_mac_len >> DATA_LEN_SHIFT) && !(tnl_len >> DATA_LEN_SHIFT) ) {
>                          hdr->hdr.flags |= VIRTIO_NET_HDR_F_ENCAPSULATION;
>                          hdr->num_buffers = (__virtio16)((inner_mac_len << DATA_LEN_SHIFT) | tnl_len);		//we reuse num_buffers for simple , we should add extend member for later.
>                  }  else
>                          hdr->num_buffers = 0;
>          }
>
> Tun.c
>                  if (memcpy_fromiovecend((void *)&hdr, iv, offset, tun->vnet_hdr_sz))		//read header with negotiation length
>                          return -EFAULT;
>
>                  if (hdr.gso_type & VIRTIO_NET_HDR_GSO_TUNNEL)					//set tunnel gso info
>                          skb_shinfo(skb)->gso_type |= SKB_GSO_UDP_TUNNEL;
>                  if (hdr.gso_type & VIRTIO_NET_HDR_GSO_TUNNEL_CSUM)
>                          skb_shinfo(skb)->gso_type |= SKB_GSO_UDP_TUNNEL_CSUM;
>                  if (hdr.gso_type & VIRTIO_NET_HDR_GSO_TUNNEL_REMCSUM)
>                          skb_shinfo(skb)->gso_type |= SKB_GSO_TUNNEL_REMCSUM;
>
>          if (hdr.flags & VIRTIO_NET_HDR_F_ENCAPSULATION) {						//read tunnel info from header and set to built skb.
>                  tnl_len = tun16_to_cpu(tun, hdr.num_buffers) & TUN_TNL_LEN_MASK;
>                  payload_mac_len = tun16_to_cpu(tun, hdr.num_buffers) >> TUN_DATA_LEN_SHIFT;
>                  mac_len = skb_network_header(skb) - skb_mac_header(skb);
>                  skb_set_inner_mac_header(skb, tnl_len - mac_len);
>                  skb_set_inner_network_header(skb, tnl_len + payload_mac_len - mac_len);
>                  skb->encapsulation = 1;
>          }
>
>

Something like this, and you probably need do something more:

- use net-next.git to generate the patch (for the latest code)
- add feature negotiation
- tun/macvtap/qemu patches for this, you can start with tun/macvtap patches
- support for all other SKB_GSO_* types which is not supported
- use a new field instead of num_buffers
- a virtio spec patch to describe the support for encapsulation offload

Thanks