* Re: Virtio_net support vxlan encapsulation package TSO offload discuss
[not found] <DBCD2614ECF3FF4087A2C27CA80E34DD5401AC8B@SZXEMA501-MBX.china.huawei.com>
@ 2016-11-08 11:13 ` Jason Wang
2016-11-08 11:17 ` Jason Wang
0 siblings, 1 reply; 9+ messages in thread
From: Jason Wang @ 2016-11-08 11:13 UTC (permalink / raw)
To: Zhangming (James, Euler), netdev@vger.kernel.org; +Cc: Michael S. Tsirkin
Cc Michael
On 2016年11月08日 16:34, Zhangming (James, Euler) wrote:
>
> In container scenario, OVS is installed in the Virtual machine, and
> all the containers connected to the OVS will communicated through
> VXLAN encapsulation.
>
> By now, virtio_net does not support TSO offload for VXLAN encapsulated
> TSO package. In this condition, the performance is not good, sender is
> bottleneck
>
> I googled this scenario, but I didn’t find any information. Will
> virtio_net support VXLAN encapsulation package TSO offload later?
>
Yes and for both sender and receiver.
> My idea is virtio_net open encapsulated TSO offload, and transport
> encapsulation info to TUN, TUN will parse the info and build skb with
> encapsulation info.
>
> OVS or kernel on the host should be modified to support this. Using
> this method, the TCP performance aremore than 2x as before.
>
> Any advice and suggestions for this idea or new idea will be greatly
> appreciated!
>
> Best regards,
>
> James zhang
>
Sounds very good. And we may also need features bits
(VIRTIO_NET_F_GUEST|HOST_GSO_X) for this.
This is in fact one of items in networking todo list. (See
http://www.linux-kvm.org/page/NetworkingTodo). While at it, we'd better
support not only VXLAN but also other tunnels.
We can start with the spec work, or if you've already had some bits you
can post them as RFC for early review.
Thanks
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Virtio_net support vxlan encapsulation package TSO offload discuss
2016-11-08 11:13 ` Virtio_net support vxlan encapsulation package TSO offload discuss Jason Wang
@ 2016-11-08 11:17 ` Jason Wang
2016-11-08 11:58 ` Zhangming (James, Euler)
0 siblings, 1 reply; 9+ messages in thread
From: Jason Wang @ 2016-11-08 11:17 UTC (permalink / raw)
To: Zhangming (James, Euler), netdev@vger.kernel.org
Cc: Michael S. Tsirkin, Vlad Yasevic
On 2016年11月08日 19:13, Jason Wang wrote:
> Cc Michael
>
> On 2016年11月08日 16:34, Zhangming (James, Euler) wrote:
>>
>> In container scenario, OVS is installed in the Virtual machine, and
>> all the containers connected to the OVS will communicated through
>> VXLAN encapsulation.
>>
>> By now, virtio_net does not support TSO offload for VXLAN
>> encapsulated TSO package. In this condition, the performance is not
>> good, sender is bottleneck
>>
>> I googled this scenario, but I didn’t find any information. Will
>> virtio_net support VXLAN encapsulation package TSO offload later?
>>
>
> Yes and for both sender and receiver.
>
>> My idea is virtio_net open encapsulated TSO offload, and transport
>> encapsulation info to TUN, TUN will parse the info and build skb with
>> encapsulation info.
>>
>> OVS or kernel on the host should be modified to support this. Using
>> this method, the TCP performance aremore than 2x as before.
>>
>> Any advice and suggestions for this idea or new idea will be greatly
>> appreciated!
>>
>> Best regards,
>>
>> James zhang
>>
>
> Sounds very good. And we may also need features bits
> (VIRTIO_NET_F_GUEST|HOST_GSO_X) for this.
>
> This is in fact one of items in networking todo list. (See
> http://www.linux-kvm.org/page/NetworkingTodo). While at it, we'd
> better support not only VXLAN but also other tunnels.
Cc Vlad who is working on extending virtio-net headers.
>
> We can start with the spec work, or if you've already had some bits
> you can post them as RFC for early review.
>
> Thanks
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Virtio_net support vxlan encapsulation package TSO offload discuss
2016-11-08 11:17 ` Jason Wang
@ 2016-11-08 11:58 ` Zhangming (James, Euler)
2016-11-09 7:14 ` Jason Wang
0 siblings, 1 reply; 9+ messages in thread
From: Zhangming (James, Euler) @ 2016-11-08 11:58 UTC (permalink / raw)
To: Jason Wang, netdev@vger.kernel.org; +Cc: Michael S. Tsirkin, Vlad Yasevic
On 2016年11月08日 19:17, Jason Wang wrote:
>On 2016年11月08日 19:13, Jason Wang wrote:
>> Cc Michael
>>
>> On 2016年11月08日 16:34, Zhangming (James, Euler) wrote:
>>>
>>> In container scenario, OVS is installed in the Virtual machine, and
>>> all the containers connected to the OVS will communicated through
>>> VXLAN encapsulation.
>>>
>>> By now, virtio_net does not support TSO offload for VXLAN
>>> encapsulated TSO package. In this condition, the performance is not
>>> good, sender is bottleneck
>>>
>>> I googled this scenario, but I didn’t find any information. Will
>>> virtio_net support VXLAN encapsulation package TSO offload later?
>>>
>>
> >Yes and for both sender and receiver.
>>
>>> My idea is virtio_net open encapsulated TSO offload, and transport
>>> encapsulation info to TUN, TUN will parse the info and build skb with
>>> encapsulation info.
>>>
>>> OVS or kernel on the host should be modified to support this. Using
>>> this method, the TCP performance aremore than 2x as before.
>>>
>>> Any advice and suggestions for this idea or new idea will be greatly
>>> appreciated!
>>>
>>> Best regards,
>>>
>>> James zhang
>>>
>>
>> Sounds very good. And we may also need features bits
>> (VIRTIO_NET_F_GUEST|HOST_GSO_X) for this.
>>
>> This is in fact one of items in networking todo list. (See
>> http://www.linux-kvm.org/page/NetworkingTodo). While at it, we'd
>> better support not only VXLAN but also other tunnels.
>
>Cc Vlad who is working on extending virtio-net headers.
>
>>
>> We can start with the spec work, or if you've already had some bits
>> you can post them as RFC for early review.
>>
>> Thanks
Below is my demo code
Virtio_net.c
static int virtnet_probe(struct virtio_device *vdev), add belows codes:
if (virtio_has_feature(vdev, VIRTIO_NET_F_MRG_RXBUF) || // avoid gso segment, it should be negotiation later, because in the demo I reuse num_buffers.
virtio_has_feature(vdev, VIRTIO_F_VERSION_1)) {
dev->hw_enc_features |= NETIF_F_TSO;
dev->hw_enc_features |= NETIF_F_ALL_CSUM;
dev->hw_enc_features |= NETIF_F_GSO_UDP_TUNNEL;
dev->hw_enc_features |= NETIF_F_GSO_UDP_TUNNEL_CSUM;
dev->hw_enc_features |= NETIF_F_GSO_TUNNEL_REMCSUM;
dev->features |= NETIF_F_GSO_UDP_TUNNEL;
dev->features |= NETIF_F_GSO_UDP_TUNNEL_CSUM;
dev->features |= NETIF_F_GSO_TUNNEL_REMCSUM;
}
static int xmit_skb(struct send_queue *sq, struct sk_buff *skb), add below to pieces of codes
if (skb_shinfo(skb)->gso_type & SKB_GSO_UDP_TUNNEL)
hdr->hdr.gso_type |= VIRTIO_NET_HDR_GSO_TUNNEL;
if (skb_shinfo(skb)->gso_type & SKB_GSO_UDP_TUNNEL_CSUM)
hdr->hdr.gso_type |= VIRTIO_NET_HDR_GSO_TUNNEL_CSUM;
if (skb_shinfo(skb)->gso_type & SKB_GSO_TUNNEL_REMCSUM)
hdr->hdr.gso_type |= VIRTIO_NET_HDR_GSO_TUNNEL_REMCSUM;
if (skb->encapsulation && skb_is_gso(skb)) {
inner_mac_len = skb_inner_network_header(skb) - skb_inner_mac_header(skb);
tnl_len = skb_inner_mac_header(skb) - skb_mac_header(skb);
if ( !(inner_mac_len >> DATA_LEN_SHIFT) && !(tnl_len >> DATA_LEN_SHIFT) ) {
hdr->hdr.flags |= VIRTIO_NET_HDR_F_ENCAPSULATION;
hdr->num_buffers = (__virtio16)((inner_mac_len << DATA_LEN_SHIFT) | tnl_len); //we reuse num_buffers for simple , we should add extend member for later.
} else
hdr->num_buffers = 0;
}
Tun.c
if (memcpy_fromiovecend((void *)&hdr, iv, offset, tun->vnet_hdr_sz)) //read header with negotiation length
return -EFAULT;
if (hdr.gso_type & VIRTIO_NET_HDR_GSO_TUNNEL) //set tunnel gso info
skb_shinfo(skb)->gso_type |= SKB_GSO_UDP_TUNNEL;
if (hdr.gso_type & VIRTIO_NET_HDR_GSO_TUNNEL_CSUM)
skb_shinfo(skb)->gso_type |= SKB_GSO_UDP_TUNNEL_CSUM;
if (hdr.gso_type & VIRTIO_NET_HDR_GSO_TUNNEL_REMCSUM)
skb_shinfo(skb)->gso_type |= SKB_GSO_TUNNEL_REMCSUM;
if (hdr.flags & VIRTIO_NET_HDR_F_ENCAPSULATION) { //read tunnel info from header and set to built skb.
tnl_len = tun16_to_cpu(tun, hdr.num_buffers) & TUN_TNL_LEN_MASK;
payload_mac_len = tun16_to_cpu(tun, hdr.num_buffers) >> TUN_DATA_LEN_SHIFT;
mac_len = skb_network_header(skb) - skb_mac_header(skb);
skb_set_inner_mac_header(skb, tnl_len - mac_len);
skb_set_inner_network_header(skb, tnl_len + payload_mac_len - mac_len);
skb->encapsulation = 1;
}
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Virtio_net support vxlan encapsulation package TSO offload discuss
2016-11-08 11:58 ` Zhangming (James, Euler)
@ 2016-11-09 7:14 ` Jason Wang
2016-11-10 6:19 ` Zhangming (James, Euler)
0 siblings, 1 reply; 9+ messages in thread
From: Jason Wang @ 2016-11-09 7:14 UTC (permalink / raw)
To: Zhangming (James, Euler), netdev@vger.kernel.org
Cc: Michael S. Tsirkin, Vlad Yasevic
On 2016年11月08日 19:58, Zhangming (James, Euler) wrote:
> On 2016年11月08日 19:17, Jason Wang wrote:
>
>> On 2016年11月08日 19:13, Jason Wang wrote:
>>> Cc Michael
>>>
>>> On 2016年11月08日 16:34, Zhangming (James, Euler) wrote:
>>>> In container scenario, OVS is installed in the Virtual machine, and
>>>> all the containers connected to the OVS will communicated through
>>>> VXLAN encapsulation.
>>>>
>>>> By now, virtio_net does not support TSO offload for VXLAN
>>>> encapsulated TSO package. In this condition, the performance is not
>>>> good, sender is bottleneck
>>>>
>>>> I googled this scenario, but I didn’t find any information. Will
>>>> virtio_net support VXLAN encapsulation package TSO offload later?
>>>>
>>> Yes and for both sender and receiver.
>>>
>>>> My idea is virtio_net open encapsulated TSO offload, and transport
>>>> encapsulation info to TUN, TUN will parse the info and build skb with
>>>> encapsulation info.
>>>>
>>>> OVS or kernel on the host should be modified to support this. Using
>>>> this method, the TCP performance aremore than 2x as before.
>>>>
>>>> Any advice and suggestions for this idea or new idea will be greatly
>>>> appreciated!
>>>>
>>>> Best regards,
>>>>
>>>> James zhang
>>>>
>>> Sounds very good. And we may also need features bits
>>> (VIRTIO_NET_F_GUEST|HOST_GSO_X) for this.
>>>
>>> This is in fact one of items in networking todo list. (See
>>> http://www.linux-kvm.org/page/NetworkingTodo). While at it, we'd
>>> better support not only VXLAN but also other tunnels.
>> Cc Vlad who is working on extending virtio-net headers.
>>
>>> We can start with the spec work, or if you've already had some bits
>>> you can post them as RFC for early review.
>>>
>>> Thanks
> Below is my demo code
> Virtio_net.c
> static int virtnet_probe(struct virtio_device *vdev), add belows codes:
> if (virtio_has_feature(vdev, VIRTIO_NET_F_MRG_RXBUF) || // avoid gso segment, it should be negotiation later, because in the demo I reuse num_buffers.
> virtio_has_feature(vdev, VIRTIO_F_VERSION_1)) {
> dev->hw_enc_features |= NETIF_F_TSO;
> dev->hw_enc_features |= NETIF_F_ALL_CSUM;
> dev->hw_enc_features |= NETIF_F_GSO_UDP_TUNNEL;
> dev->hw_enc_features |= NETIF_F_GSO_UDP_TUNNEL_CSUM;
> dev->hw_enc_features |= NETIF_F_GSO_TUNNEL_REMCSUM;
>
> dev->features |= NETIF_F_GSO_UDP_TUNNEL;
> dev->features |= NETIF_F_GSO_UDP_TUNNEL_CSUM;
> dev->features |= NETIF_F_GSO_TUNNEL_REMCSUM;
> }
>
> static int xmit_skb(struct send_queue *sq, struct sk_buff *skb), add below to pieces of codes
>
> if (skb_shinfo(skb)->gso_type & SKB_GSO_UDP_TUNNEL)
> hdr->hdr.gso_type |= VIRTIO_NET_HDR_GSO_TUNNEL;
> if (skb_shinfo(skb)->gso_type & SKB_GSO_UDP_TUNNEL_CSUM)
> hdr->hdr.gso_type |= VIRTIO_NET_HDR_GSO_TUNNEL_CSUM;
> if (skb_shinfo(skb)->gso_type & SKB_GSO_TUNNEL_REMCSUM)
> hdr->hdr.gso_type |= VIRTIO_NET_HDR_GSO_TUNNEL_REMCSUM;
>
> if (skb->encapsulation && skb_is_gso(skb)) {
> inner_mac_len = skb_inner_network_header(skb) - skb_inner_mac_header(skb);
> tnl_len = skb_inner_mac_header(skb) - skb_mac_header(skb);
> if ( !(inner_mac_len >> DATA_LEN_SHIFT) && !(tnl_len >> DATA_LEN_SHIFT) ) {
> hdr->hdr.flags |= VIRTIO_NET_HDR_F_ENCAPSULATION;
> hdr->num_buffers = (__virtio16)((inner_mac_len << DATA_LEN_SHIFT) | tnl_len); //we reuse num_buffers for simple , we should add extend member for later.
> } else
> hdr->num_buffers = 0;
> }
>
> Tun.c
> if (memcpy_fromiovecend((void *)&hdr, iv, offset, tun->vnet_hdr_sz)) //read header with negotiation length
> return -EFAULT;
>
> if (hdr.gso_type & VIRTIO_NET_HDR_GSO_TUNNEL) //set tunnel gso info
> skb_shinfo(skb)->gso_type |= SKB_GSO_UDP_TUNNEL;
> if (hdr.gso_type & VIRTIO_NET_HDR_GSO_TUNNEL_CSUM)
> skb_shinfo(skb)->gso_type |= SKB_GSO_UDP_TUNNEL_CSUM;
> if (hdr.gso_type & VIRTIO_NET_HDR_GSO_TUNNEL_REMCSUM)
> skb_shinfo(skb)->gso_type |= SKB_GSO_TUNNEL_REMCSUM;
>
> if (hdr.flags & VIRTIO_NET_HDR_F_ENCAPSULATION) { //read tunnel info from header and set to built skb.
> tnl_len = tun16_to_cpu(tun, hdr.num_buffers) & TUN_TNL_LEN_MASK;
> payload_mac_len = tun16_to_cpu(tun, hdr.num_buffers) >> TUN_DATA_LEN_SHIFT;
> mac_len = skb_network_header(skb) - skb_mac_header(skb);
> skb_set_inner_mac_header(skb, tnl_len - mac_len);
> skb_set_inner_network_header(skb, tnl_len + payload_mac_len - mac_len);
> skb->encapsulation = 1;
> }
>
>
Something like this, and you probably need do something more:
- use net-next.git to generate the patch (for the latest code)
- add feature negotiation
- tun/macvtap/qemu patches for this, you can start with tun/macvtap patches
- support for all other SKB_GSO_* types which is not supported
- use a new field instead of num_buffers
- a virtio spec patch to describe the support for encapsulation offload
Thanks
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Virtio_net support vxlan encapsulation package TSO offload discuss
2016-11-09 7:14 ` Jason Wang
@ 2016-11-10 6:19 ` Zhangming (James, Euler)
2016-11-15 3:27 ` Jason Wang
0 siblings, 1 reply; 9+ messages in thread
From: Zhangming (James, Euler) @ 2016-11-10 6:19 UTC (permalink / raw)
To: Jason Wang, netdev@vger.kernel.org; +Cc: Michael S. Tsirkin, Vlad Yasevic
On 2016年11月09日 15:14, Jason Wang wrote:
>On 2016年11月08日 19:58, Zhangming (James, Euler) wrote:
>> On 2016年11月08日 19:17, Jason Wang wrote:
>>
>>> On 2016年11月08日 19:13, Jason Wang wrote:
>>>> Cc Michael
>>>>
>>>> On 2016年11月08日 16:34, Zhangming (James, Euler) wrote:
>>>>> In container scenario, OVS is installed in the Virtual machine, and
>>>>> all the containers connected to the OVS will communicated through
>>>>> VXLAN encapsulation.
>>>>>
>>>>> By now, virtio_net does not support TSO offload for VXLAN
>>>>> encapsulated TSO package. In this condition, the performance is not
>>>>> good, sender is bottleneck
>>>>>
>>>>> I googled this scenario, but I didn’t find any information. Will
>>>>> virtio_net support VXLAN encapsulation package TSO offload later?
>>>>>
>>>> Yes and for both sender and receiver.
>>>>
>>>>> My idea is virtio_net open encapsulated TSO offload, and transport
>>>>> encapsulation info to TUN, TUN will parse the info and build skb
>>>>> with encapsulation info.
>>>>>
>>>>> OVS or kernel on the host should be modified to support this. Using
>>>>> this method, the TCP performance aremore than 2x as before.
>>>>>
>>>>> Any advice and suggestions for this idea or new idea will be
>>>>> greatly appreciated!
>>>>>
>>>>> Best regards,
>>>>>
>>>>> James zhang
>>>>>
>>>> Sounds very good. And we may also need features bits
>>>> (VIRTIO_NET_F_GUEST|HOST_GSO_X) for this.
>>>>
>>>> This is in fact one of items in networking todo list. (See
>>>> http://www.linux-kvm.org/page/NetworkingTodo). While at it, we'd
>>>> better support not only VXLAN but also other tunnels.
>>> Cc Vlad who is working on extending virtio-net headers.
>>>
>>>> We can start with the spec work, or if you've already had some bits
>>>> you can post them as RFC for early review.
>>>>
>>>> Thanks
>> Below is my demo code
>> Virtio_net.c
>> static int virtnet_probe(struct virtio_device *vdev), add belows codes:
>> if (virtio_has_feature(vdev, VIRTIO_NET_F_MRG_RXBUF) || // avoid gso segment, it should be negotiation later, because in the demo I reuse num_buffers.
>> virtio_has_feature(vdev, VIRTIO_F_VERSION_1)) {
>> dev->hw_enc_features |= NETIF_F_TSO;
>> dev->hw_enc_features |= NETIF_F_ALL_CSUM;
>> dev->hw_enc_features |= NETIF_F_GSO_UDP_TUNNEL;
>> dev->hw_enc_features |= NETIF_F_GSO_UDP_TUNNEL_CSUM;
>> dev->hw_enc_features |= NETIF_F_GSO_TUNNEL_REMCSUM;
>>
>> dev->features |= NETIF_F_GSO_UDP_TUNNEL;
>> dev->features |= NETIF_F_GSO_UDP_TUNNEL_CSUM;
>> dev->features |= NETIF_F_GSO_TUNNEL_REMCSUM;
>> }
>>
>> static int xmit_skb(struct send_queue *sq, struct sk_buff *skb), add
>> below to pieces of codes
>>
>> if (skb_shinfo(skb)->gso_type & SKB_GSO_UDP_TUNNEL)
>> hdr->hdr.gso_type |= VIRTIO_NET_HDR_GSO_TUNNEL;
>> if (skb_shinfo(skb)->gso_type & SKB_GSO_UDP_TUNNEL_CSUM)
>> hdr->hdr.gso_type |= VIRTIO_NET_HDR_GSO_TUNNEL_CSUM;
>> if (skb_shinfo(skb)->gso_type & SKB_GSO_TUNNEL_REMCSUM)
>> hdr->hdr.gso_type |=
>> VIRTIO_NET_HDR_GSO_TUNNEL_REMCSUM;
>>
>> if (skb->encapsulation && skb_is_gso(skb)) {
>> inner_mac_len = skb_inner_network_header(skb) - skb_inner_mac_header(skb);
>> tnl_len = skb_inner_mac_header(skb) - skb_mac_header(skb);
>> if ( !(inner_mac_len >> DATA_LEN_SHIFT) && !(tnl_len >> DATA_LEN_SHIFT) ) {
>> hdr->hdr.flags |= VIRTIO_NET_HDR_F_ENCAPSULATION;
>> hdr->num_buffers = (__virtio16)((inner_mac_len << DATA_LEN_SHIFT) | tnl_len); //we reuse num_buffers for simple , we should add extend member for later.
>> } else
>> hdr->num_buffers = 0;
>> }
>>
>> Tun.c
>> if (memcpy_fromiovecend((void *)&hdr, iv, offset, tun->vnet_hdr_sz)) //read header with negotiation length
>> return -EFAULT;
>>
>> if (hdr.gso_type & VIRTIO_NET_HDR_GSO_TUNNEL) //set tunnel gso info
>> skb_shinfo(skb)->gso_type |= SKB_GSO_UDP_TUNNEL;
>> if (hdr.gso_type & VIRTIO_NET_HDR_GSO_TUNNEL_CSUM)
>> skb_shinfo(skb)->gso_type |= SKB_GSO_UDP_TUNNEL_CSUM;
>> if (hdr.gso_type & VIRTIO_NET_HDR_GSO_TUNNEL_REMCSUM)
>> skb_shinfo(skb)->gso_type |=
>> SKB_GSO_TUNNEL_REMCSUM;
>>
>> if (hdr.flags & VIRTIO_NET_HDR_F_ENCAPSULATION) { //read tunnel info from header and set to built skb.
>> tnl_len = tun16_to_cpu(tun, hdr.num_buffers) & TUN_TNL_LEN_MASK;
>> payload_mac_len = tun16_to_cpu(tun, hdr.num_buffers) >> TUN_DATA_LEN_SHIFT;
>> mac_len = skb_network_header(skb) - skb_mac_header(skb);
>> skb_set_inner_mac_header(skb, tnl_len - mac_len);
>> skb_set_inner_network_header(skb, tnl_len + payload_mac_len - mac_len);
>> skb->encapsulation = 1;
>> }
>>
>>
>
>Something like this, and you probably need do something more:
>
>- use net-next.git to generate the patch (for the latest code)
>- add feature negotiation
>- tun/macvtap/qemu patches for this, you can start with tun/macvtap patches
>- support for all other SKB_GSO_* types which is not supported
>- use a new field instead of num_buffers
>- a virtio spec patch to describe the support for encapsulation offload
>
>Thanks
Thank you for your advice, I will start it right now.
Thanks
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Virtio_net support vxlan encapsulation package TSO offload discuss
2016-11-10 6:19 ` Zhangming (James, Euler)
@ 2016-11-15 3:27 ` Jason Wang
2016-11-17 1:31 ` Zhangming (James, Euler)
0 siblings, 1 reply; 9+ messages in thread
From: Jason Wang @ 2016-11-15 3:27 UTC (permalink / raw)
To: Zhangming (James, Euler), netdev@vger.kernel.org
Cc: Michael S. Tsirkin, Vlad Yasevic, Amnon Ilan
On 2016年11月10日 14:19, Zhangming (James, Euler) wrote:
> On 2016年11月09日 15:14, Jason Wang wrote:
>> On 2016年11月08日 19:58, Zhangming (James, Euler) wrote:
>>> On 2016年11月08日 19:17, Jason Wang wrote:
>>>
>>>> On 2016年11月08日 19:13, Jason Wang wrote:
>>>>> Cc Michael
>>>>>
>>>>> On 2016年11月08日 16:34, Zhangming (James, Euler) wrote:
>>>>>> In container scenario, OVS is installed in the Virtual machine, and
>>>>>> all the containers connected to the OVS will communicated through
>>>>>> VXLAN encapsulation.
>>>>>>
>>>>>> By now, virtio_net does not support TSO offload for VXLAN
>>>>>> encapsulated TSO package. In this condition, the performance is not
>>>>>> good, sender is bottleneck
>>>>>>
>>>>>> I googled this scenario, but I didn’t find any information. Will
>>>>>> virtio_net support VXLAN encapsulation package TSO offload later?
>>>>>>
>>>>> Yes and for both sender and receiver.
>>>>>
>>>>>> My idea is virtio_net open encapsulated TSO offload, and transport
>>>>>> encapsulation info to TUN, TUN will parse the info and build skb
>>>>>> with encapsulation info.
>>>>>>
>>>>>> OVS or kernel on the host should be modified to support this. Using
>>>>>> this method, the TCP performance aremore than 2x as before.
>>>>>>
>>>>>> Any advice and suggestions for this idea or new idea will be
>>>>>> greatly appreciated!
>>>>>>
>>>>>> Best regards,
>>>>>>
>>>>>> James zhang
>>>>>>
>>>>> Sounds very good. And we may also need features bits
>>>>> (VIRTIO_NET_F_GUEST|HOST_GSO_X) for this.
>>>>>
>>>>> This is in fact one of items in networking todo list. (See
>>>>> http://www.linux-kvm.org/page/NetworkingTodo). While at it, we'd
>>>>> better support not only VXLAN but also other tunnels.
>>>> Cc Vlad who is working on extending virtio-net headers.
>>>>
>>>>> We can start with the spec work, or if you've already had some bits
>>>>> you can post them as RFC for early review.
>>>>>
>>>>> Thanks
>>> Below is my demo code
>>> Virtio_net.c
>>> static int virtnet_probe(struct virtio_device *vdev), add belows codes:
>>> if (virtio_has_feature(vdev, VIRTIO_NET_F_MRG_RXBUF) || // avoid gso segment, it should be negotiation later, because in the demo I reuse num_buffers.
>>> virtio_has_feature(vdev, VIRTIO_F_VERSION_1)) {
>>> dev->hw_enc_features |= NETIF_F_TSO;
>>> dev->hw_enc_features |= NETIF_F_ALL_CSUM;
>>> dev->hw_enc_features |= NETIF_F_GSO_UDP_TUNNEL;
>>> dev->hw_enc_features |= NETIF_F_GSO_UDP_TUNNEL_CSUM;
>>> dev->hw_enc_features |= NETIF_F_GSO_TUNNEL_REMCSUM;
>>>
>>> dev->features |= NETIF_F_GSO_UDP_TUNNEL;
>>> dev->features |= NETIF_F_GSO_UDP_TUNNEL_CSUM;
>>> dev->features |= NETIF_F_GSO_TUNNEL_REMCSUM;
>>> }
>>>
>>> static int xmit_skb(struct send_queue *sq, struct sk_buff *skb), add
>>> below to pieces of codes
>>>
>>> if (skb_shinfo(skb)->gso_type & SKB_GSO_UDP_TUNNEL)
>>> hdr->hdr.gso_type |= VIRTIO_NET_HDR_GSO_TUNNEL;
>>> if (skb_shinfo(skb)->gso_type & SKB_GSO_UDP_TUNNEL_CSUM)
>>> hdr->hdr.gso_type |= VIRTIO_NET_HDR_GSO_TUNNEL_CSUM;
>>> if (skb_shinfo(skb)->gso_type & SKB_GSO_TUNNEL_REMCSUM)
>>> hdr->hdr.gso_type |=
>>> VIRTIO_NET_HDR_GSO_TUNNEL_REMCSUM;
>>>
>>> if (skb->encapsulation && skb_is_gso(skb)) {
>>> inner_mac_len = skb_inner_network_header(skb) - skb_inner_mac_header(skb);
>>> tnl_len = skb_inner_mac_header(skb) - skb_mac_header(skb);
>>> if ( !(inner_mac_len >> DATA_LEN_SHIFT) && !(tnl_len >> DATA_LEN_SHIFT) ) {
>>> hdr->hdr.flags |= VIRTIO_NET_HDR_F_ENCAPSULATION;
>>> hdr->num_buffers = (__virtio16)((inner_mac_len << DATA_LEN_SHIFT) | tnl_len); //we reuse num_buffers for simple , we should add extend member for later.
>>> } else
>>> hdr->num_buffers = 0;
>>> }
>>>
>>> Tun.c
>>> if (memcpy_fromiovecend((void *)&hdr, iv, offset, tun->vnet_hdr_sz)) //read header with negotiation length
>>> return -EFAULT;
>>>
>>> if (hdr.gso_type & VIRTIO_NET_HDR_GSO_TUNNEL) //set tunnel gso info
>>> skb_shinfo(skb)->gso_type |= SKB_GSO_UDP_TUNNEL;
>>> if (hdr.gso_type & VIRTIO_NET_HDR_GSO_TUNNEL_CSUM)
>>> skb_shinfo(skb)->gso_type |= SKB_GSO_UDP_TUNNEL_CSUM;
>>> if (hdr.gso_type & VIRTIO_NET_HDR_GSO_TUNNEL_REMCSUM)
>>> skb_shinfo(skb)->gso_type |=
>>> SKB_GSO_TUNNEL_REMCSUM;
>>>
>>> if (hdr.flags & VIRTIO_NET_HDR_F_ENCAPSULATION) { //read tunnel info from header and set to built skb.
>>> tnl_len = tun16_to_cpu(tun, hdr.num_buffers) & TUN_TNL_LEN_MASK;
>>> payload_mac_len = tun16_to_cpu(tun, hdr.num_buffers) >> TUN_DATA_LEN_SHIFT;
>>> mac_len = skb_network_header(skb) - skb_mac_header(skb);
>>> skb_set_inner_mac_header(skb, tnl_len - mac_len);
>>> skb_set_inner_network_header(skb, tnl_len + payload_mac_len - mac_len);
>>> skb->encapsulation = 1;
>>> }
>>>
>>>
>> Something like this, and you probably need do something more:
>>
>> - use net-next.git to generate the patch (for the latest code)
>> - add feature negotiation
>> - tun/macvtap/qemu patches for this, you can start with tun/macvtap patches
>> - support for all other SKB_GSO_* types which is not supported
>> - use a new field instead of num_buffers
>> - a virtio spec patch to describe the support for encapsulation offload
>>
>> Thanks
> Thank you for your advice, I will start it right now.
>
> Thanks
Cool, one more question: while at it, I think you may want to add
support for dpdk too?
Thanks
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Virtio_net support vxlan encapsulation package TSO offload discuss
2016-11-15 3:27 ` Jason Wang
@ 2016-11-17 1:31 ` Zhangming (James, Euler)
2016-11-17 7:27 ` Jason Wang
0 siblings, 1 reply; 9+ messages in thread
From: Zhangming (James, Euler) @ 2016-11-17 1:31 UTC (permalink / raw)
To: Jason Wang, netdev@vger.kernel.org
Cc: Michael S. Tsirkin, Vlad Yasevic, Amnon Ilan
On 2016年11月15日 11:28, Jason Wang wrote:
>On 2016年11月10日 14:19, Zhangming (James, Euler) wrote:
>> On 2016年11月09日 15:14, Jason Wang wrote:
>>> On 2016年11月08日 19:58, Zhangming (James, Euler) wrote:
>>>> On 2016年11月08日 19:17, Jason Wang wrote:
>>>>
>>>>> On 2016年11月08日 19:13, Jason Wang wrote:
>>>>>> Cc Michael
>>>>>>
>>>>>> On 2016年11月08日 16:34, Zhangming (James, Euler) wrote:
>>>>>>> In container scenario, OVS is installed in the Virtual machine,
>>>>>>> and all the containers connected to the OVS will communicated
>>>>>>> through VXLAN encapsulation.
>>>>>>>
>>>>>>> By now, virtio_net does not support TSO offload for VXLAN
>>>>>>> encapsulated TSO package. In this condition, the performance is
>>>>>>> not good, sender is bottleneck
>>>>>>>
>>>>>>> I googled this scenario, but I didn’t find any information. Will
>>>>>>> virtio_net support VXLAN encapsulation package TSO offload later?
>>>>>>>
>>>>>> Yes and for both sender and receiver.
>>>>>>
>>>>>>> My idea is virtio_net open encapsulated TSO offload, and
>>>>>>> transport encapsulation info to TUN, TUN will parse the info and
>>>>>>> build skb with encapsulation info.
>>>>>>>
>>>>>>> OVS or kernel on the host should be modified to support this.
>>>>>>> Using this method, the TCP performance aremore than 2x as before.
>>>>>>>
>>>>>>> Any advice and suggestions for this idea or new idea will be
>>>>>>> greatly appreciated!
>>>>>>>
>>>>>>> Best regards,
>>>>>>>
>>>>>>> James zhang
>>>>>>>
>>>>>> Sounds very good. And we may also need features bits
>>>>>> (VIRTIO_NET_F_GUEST|HOST_GSO_X) for this.
>>>>>>
>>>>>> This is in fact one of items in networking todo list. (See
>>>>>> http://www.linux-kvm.org/page/NetworkingTodo). While at it, we'd
>>>>>> better support not only VXLAN but also other tunnels.
>>>>> Cc Vlad who is working on extending virtio-net headers.
>>>>>
>>>>>> We can start with the spec work, or if you've already had some
>>>>>> bits you can post them as RFC for early review.
>>>>>>
>>>>>> Thanks
>>>> Below is my demo code
>>>> Virtio_net.c
>>>> static int virtnet_probe(struct virtio_device *vdev), add belows codes:
>>>> if (virtio_has_feature(vdev, VIRTIO_NET_F_MRG_RXBUF) || // avoid gso segment, it should be negotiation later, because in the demo I reuse num_buffers.
>>>> virtio_has_feature(vdev, VIRTIO_F_VERSION_1)) {
>>>> dev->hw_enc_features |= NETIF_F_TSO;
>>>> dev->hw_enc_features |= NETIF_F_ALL_CSUM;
>>>> dev->hw_enc_features |= NETIF_F_GSO_UDP_TUNNEL;
>>>> dev->hw_enc_features |= NETIF_F_GSO_UDP_TUNNEL_CSUM;
>>>> dev->hw_enc_features |=
>>>> NETIF_F_GSO_TUNNEL_REMCSUM;
>>>>
>>>> dev->features |= NETIF_F_GSO_UDP_TUNNEL;
>>>> dev->features |= NETIF_F_GSO_UDP_TUNNEL_CSUM;
>>>> dev->features |= NETIF_F_GSO_TUNNEL_REMCSUM;
>>>> }
>>>>
>>>> static int xmit_skb(struct send_queue *sq, struct sk_buff *skb), add
>>>> below to pieces of codes
>>>>
>>>> if (skb_shinfo(skb)->gso_type & SKB_GSO_UDP_TUNNEL)
>>>> hdr->hdr.gso_type |= VIRTIO_NET_HDR_GSO_TUNNEL;
>>>> if (skb_shinfo(skb)->gso_type & SKB_GSO_UDP_TUNNEL_CSUM)
>>>> hdr->hdr.gso_type |= VIRTIO_NET_HDR_GSO_TUNNEL_CSUM;
>>>> if (skb_shinfo(skb)->gso_type & SKB_GSO_TUNNEL_REMCSUM)
>>>> hdr->hdr.gso_type |=
>>>> VIRTIO_NET_HDR_GSO_TUNNEL_REMCSUM;
>>>>
>>>> if (skb->encapsulation && skb_is_gso(skb)) {
>>>> inner_mac_len = skb_inner_network_header(skb) - skb_inner_mac_header(skb);
>>>> tnl_len = skb_inner_mac_header(skb) - skb_mac_header(skb);
>>>> if ( !(inner_mac_len >> DATA_LEN_SHIFT) && !(tnl_len >> DATA_LEN_SHIFT) ) {
>>>> hdr->hdr.flags |= VIRTIO_NET_HDR_F_ENCAPSULATION;
>>>> hdr->num_buffers = (__virtio16)((inner_mac_len << DATA_LEN_SHIFT) | tnl_len); //we reuse num_buffers for simple , we should add extend member for later.
>>>> } else
>>>> hdr->num_buffers = 0;
>>>> }
>>>>
>>>> Tun.c
>>>> if (memcpy_fromiovecend((void *)&hdr, iv, offset, tun->vnet_hdr_sz)) //read header with negotiation length
>>>> return -EFAULT;
>>>>
>>>> if (hdr.gso_type & VIRTIO_NET_HDR_GSO_TUNNEL) //set tunnel gso info
>>>> skb_shinfo(skb)->gso_type |= SKB_GSO_UDP_TUNNEL;
>>>> if (hdr.gso_type & VIRTIO_NET_HDR_GSO_TUNNEL_CSUM)
>>>> skb_shinfo(skb)->gso_type |= SKB_GSO_UDP_TUNNEL_CSUM;
>>>> if (hdr.gso_type & VIRTIO_NET_HDR_GSO_TUNNEL_REMCSUM)
>>>> skb_shinfo(skb)->gso_type |=
>>>> SKB_GSO_TUNNEL_REMCSUM;
>>>>
>>>> if (hdr.flags & VIRTIO_NET_HDR_F_ENCAPSULATION) { //read tunnel info from header and set to built skb.
>>>> tnl_len = tun16_to_cpu(tun, hdr.num_buffers) & TUN_TNL_LEN_MASK;
>>>> payload_mac_len = tun16_to_cpu(tun, hdr.num_buffers) >> TUN_DATA_LEN_SHIFT;
>>>> mac_len = skb_network_header(skb) - skb_mac_header(skb);
>>>> skb_set_inner_mac_header(skb, tnl_len - mac_len);
>>>> skb_set_inner_network_header(skb, tnl_len + payload_mac_len - mac_len);
>>>> skb->encapsulation = 1;
>>>> }
>>>>
>>>>
>>> Something like this, and you probably need do something more:
>>>
>>> - use net-next.git to generate the patch (for the latest code)
>>> - add feature negotiation
>>> - tun/macvtap/qemu patches for this, you can start with tun/macvtap
>>> patches
>>> - support for all other SKB_GSO_* types which is not supported
>>> - use a new field instead of num_buffers
>>> - a virtio spec patch to describe the support for encapsulation
>>> offload
>>>
>>> Thanks
>> Thank you for your advice, I will start it right now.
>>
>> Thanks
>
>Cool, one more question: while at it, I think you may want to add support for dpdk too?
>
>Thanks
Do you mean that the patch should be compatible with virtio pmd, or give virtio pmd patch?
Thanks
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Virtio_net support vxlan encapsulation package TSO offload discuss
2016-11-17 1:31 ` Zhangming (James, Euler)
@ 2016-11-17 7:27 ` Jason Wang
2016-11-17 23:10 ` Jarno Rajahalme
0 siblings, 1 reply; 9+ messages in thread
From: Jason Wang @ 2016-11-17 7:27 UTC (permalink / raw)
To: Zhangming (James, Euler), netdev@vger.kernel.org
Cc: Michael S. Tsirkin, Vlad Yasevic, Amnon Ilan
On 2016年11月17日 09:31, Zhangming (James, Euler) wrote:
> On 2016年11月15日 11:28, Jason Wang wrote:
>> On 2016年11月10日 14:19, Zhangming (James, Euler) wrote:
>>> On 2016年11月09日 15:14, Jason Wang wrote:
>>>> On 2016年11月08日 19:58, Zhangming (James, Euler) wrote:
>>>>> On 2016年11月08日 19:17, Jason Wang wrote:
>>>>>
>>>>>> On 2016年11月08日 19:13, Jason Wang wrote:
>>>>>>> Cc Michael
>>>>>>>
>>>>>>> On 2016年11月08日 16:34, Zhangming (James, Euler) wrote:
>>>>>>>> In container scenario, OVS is installed in the Virtual machine,
>>>>>>>> and all the containers connected to the OVS will communicated
>>>>>>>> through VXLAN encapsulation.
>>>>>>>>
>>>>>>>> By now, virtio_net does not support TSO offload for VXLAN
>>>>>>>> encapsulated TSO package. In this condition, the performance is
>>>>>>>> not good, sender is bottleneck
>>>>>>>>
>>>>>>>> I googled this scenario, but I didn’t find any information. Will
>>>>>>>> virtio_net support VXLAN encapsulation package TSO offload later?
>>>>>>>>
>>>>>>> Yes and for both sender and receiver.
>>>>>>>
>>>>>>>> My idea is virtio_net open encapsulated TSO offload, and
>>>>>>>> transport encapsulation info to TUN, TUN will parse the info and
>>>>>>>> build skb with encapsulation info.
>>>>>>>>
>>>>>>>> OVS or kernel on the host should be modified to support this.
>>>>>>>> Using this method, the TCP performance aremore than 2x as before.
>>>>>>>>
>>>>>>>> Any advice and suggestions for this idea or new idea will be
>>>>>>>> greatly appreciated!
>>>>>>>>
>>>>>>>> Best regards,
>>>>>>>>
>>>>>>>> James zhang
>>>>>>>>
>>>>>>> Sounds very good. And we may also need features bits
>>>>>>> (VIRTIO_NET_F_GUEST|HOST_GSO_X) for this.
>>>>>>>
>>>>>>> This is in fact one of items in networking todo list. (See
>>>>>>> http://www.linux-kvm.org/page/NetworkingTodo). While at it, we'd
>>>>>>> better support not only VXLAN but also other tunnels.
>>>>>> Cc Vlad who is working on extending virtio-net headers.
>>>>>>
>>>>>>> We can start with the spec work, or if you've already had some
>>>>>>> bits you can post them as RFC for early review.
>>>>>>>
>>>>>>> Thanks
>>>>> Below is my demo code
>>>>> Virtio_net.c
>>>>> static int virtnet_probe(struct virtio_device *vdev), add belows codes:
>>>>> if (virtio_has_feature(vdev, VIRTIO_NET_F_MRG_RXBUF) || // avoid gso segment, it should be negotiation later, because in the demo I reuse num_buffers.
>>>>> virtio_has_feature(vdev, VIRTIO_F_VERSION_1)) {
>>>>> dev->hw_enc_features |= NETIF_F_TSO;
>>>>> dev->hw_enc_features |= NETIF_F_ALL_CSUM;
>>>>> dev->hw_enc_features |= NETIF_F_GSO_UDP_TUNNEL;
>>>>> dev->hw_enc_features |= NETIF_F_GSO_UDP_TUNNEL_CSUM;
>>>>> dev->hw_enc_features |=
>>>>> NETIF_F_GSO_TUNNEL_REMCSUM;
>>>>>
>>>>> dev->features |= NETIF_F_GSO_UDP_TUNNEL;
>>>>> dev->features |= NETIF_F_GSO_UDP_TUNNEL_CSUM;
>>>>> dev->features |= NETIF_F_GSO_TUNNEL_REMCSUM;
>>>>> }
>>>>>
>>>>> static int xmit_skb(struct send_queue *sq, struct sk_buff *skb), add
>>>>> below to pieces of codes
>>>>>
>>>>> if (skb_shinfo(skb)->gso_type & SKB_GSO_UDP_TUNNEL)
>>>>> hdr->hdr.gso_type |= VIRTIO_NET_HDR_GSO_TUNNEL;
>>>>> if (skb_shinfo(skb)->gso_type & SKB_GSO_UDP_TUNNEL_CSUM)
>>>>> hdr->hdr.gso_type |= VIRTIO_NET_HDR_GSO_TUNNEL_CSUM;
>>>>> if (skb_shinfo(skb)->gso_type & SKB_GSO_TUNNEL_REMCSUM)
>>>>> hdr->hdr.gso_type |=
>>>>> VIRTIO_NET_HDR_GSO_TUNNEL_REMCSUM;
>>>>>
>>>>> if (skb->encapsulation && skb_is_gso(skb)) {
>>>>> inner_mac_len = skb_inner_network_header(skb) - skb_inner_mac_header(skb);
>>>>> tnl_len = skb_inner_mac_header(skb) - skb_mac_header(skb);
>>>>> if ( !(inner_mac_len >> DATA_LEN_SHIFT) && !(tnl_len >> DATA_LEN_SHIFT) ) {
>>>>> hdr->hdr.flags |= VIRTIO_NET_HDR_F_ENCAPSULATION;
>>>>> hdr->num_buffers = (__virtio16)((inner_mac_len << DATA_LEN_SHIFT) | tnl_len); //we reuse num_buffers for simple , we should add extend member for later.
>>>>> } else
>>>>> hdr->num_buffers = 0;
>>>>> }
>>>>>
>>>>> Tun.c
>>>>> if (memcpy_fromiovecend((void *)&hdr, iv, offset, tun->vnet_hdr_sz)) //read header with negotiation length
>>>>> return -EFAULT;
>>>>>
>>>>> if (hdr.gso_type & VIRTIO_NET_HDR_GSO_TUNNEL) //set tunnel gso info
>>>>> skb_shinfo(skb)->gso_type |= SKB_GSO_UDP_TUNNEL;
>>>>> if (hdr.gso_type & VIRTIO_NET_HDR_GSO_TUNNEL_CSUM)
>>>>> skb_shinfo(skb)->gso_type |= SKB_GSO_UDP_TUNNEL_CSUM;
>>>>> if (hdr.gso_type & VIRTIO_NET_HDR_GSO_TUNNEL_REMCSUM)
>>>>> skb_shinfo(skb)->gso_type |=
>>>>> SKB_GSO_TUNNEL_REMCSUM;
>>>>>
>>>>> if (hdr.flags & VIRTIO_NET_HDR_F_ENCAPSULATION) { //read tunnel info from header and set to built skb.
>>>>> tnl_len = tun16_to_cpu(tun, hdr.num_buffers) & TUN_TNL_LEN_MASK;
>>>>> payload_mac_len = tun16_to_cpu(tun, hdr.num_buffers) >> TUN_DATA_LEN_SHIFT;
>>>>> mac_len = skb_network_header(skb) - skb_mac_header(skb);
>>>>> skb_set_inner_mac_header(skb, tnl_len - mac_len);
>>>>> skb_set_inner_network_header(skb, tnl_len + payload_mac_len - mac_len);
>>>>> skb->encapsulation = 1;
>>>>> }
>>>>>
>>>>>
>>>> Something like this, and you probably need do something more:
>>>>
>>>> - use net-next.git to generate the patch (for the latest code)
>>>> - add feature negotiation
>>>> - tun/macvtap/qemu patches for this, you can start with tun/macvtap
>>>> patches
>>>> - support for all other SKB_GSO_* types which is not supported
>>>> - use a new field instead of num_buffers
>>>> - a virtio spec patch to describe the support for encapsulation
>>>> offload
>>>>
>>>> Thanks
>>> Thank you for your advice, I will start it right now.
>>>
>>> Thanks
>> Cool, one more question: while at it, I think you may want to add support for dpdk too?
>>
>> Thanks
> Do you mean that the patch should be compatible with virtio pmd, or give virtio pmd patch?
>
> Thanks
I mean it's better to prepare patches for both virtio pmd and dpdk.
Thanks
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Virtio_net support vxlan encapsulation package TSO offload discuss
2016-11-17 7:27 ` Jason Wang
@ 2016-11-17 23:10 ` Jarno Rajahalme
0 siblings, 0 replies; 9+ messages in thread
From: Jarno Rajahalme @ 2016-11-17 23:10 UTC (permalink / raw)
To: Jason Wang
Cc: Zhangming (James, Euler), netdev@vger.kernel.org,
Michael S. Tsirkin, Vlad Yasevic, Amnon Ilan
I worked on the same issue a few months back. I rebased my proof-of-concept code to the current net-next and posted an RFC patch a moment ago.
I have zero experience on QEMU feature negotiation or extending the virtio_net spec. Since the virtio_net handling code is now all done using shared code, this should work for macvtap as well, not sure if macvtap needs some control plane changes.
I posted a separate patch to make af_packet also use the shared infra for virtio_net handling yesterday. My RFC patch assumes that af_packet need not be touched, i.e., assumes the af_packet patch is applied, even though the patches apply to net-next in either order.
Jarno
> On Nov 16, 2016, at 11:27 PM, Jason Wang <jasowang@redhat.com> wrote:
>
>
>
> On 2016年11月17日 09:31, Zhangming (James, Euler) wrote:
>> On 2016年11月15日 11:28, Jason Wang wrote:
>>> On 2016年11月10日 14:19, Zhangming (James, Euler) wrote:
>>>> On 2016年11月09日 15:14, Jason Wang wrote:
>>>>> On 2016年11月08日 19:58, Zhangming (James, Euler) wrote:
>>>>>> On 2016年11月08日 19:17, Jason Wang wrote:
>>>>>>
>>>>>>> On 2016年11月08日 19:13, Jason Wang wrote:
>>>>>>>> Cc Michael
>>>>>>>>
>>>>>>>> On 2016年11月08日 16:34, Zhangming (James, Euler) wrote:
>>>>>>>>> In container scenario, OVS is installed in the Virtual machine,
>>>>>>>>> and all the containers connected to the OVS will communicated
>>>>>>>>> through VXLAN encapsulation.
>>>>>>>>>
>>>>>>>>> By now, virtio_net does not support TSO offload for VXLAN
>>>>>>>>> encapsulated TSO package. In this condition, the performance is
>>>>>>>>> not good, sender is bottleneck
>>>>>>>>>
>>>>>>>>> I googled this scenario, but I didn’t find any information. Will
>>>>>>>>> virtio_net support VXLAN encapsulation package TSO offload later?
>>>>>>>>>
>>>>>>>> Yes and for both sender and receiver.
>>>>>>>>
>>>>>>>>> My idea is virtio_net open encapsulated TSO offload, and
>>>>>>>>> transport encapsulation info to TUN, TUN will parse the info and
>>>>>>>>> build skb with encapsulation info.
>>>>>>>>>
>>>>>>>>> OVS or kernel on the host should be modified to support this.
>>>>>>>>> Using this method, the TCP performance aremore than 2x as before.
>>>>>>>>>
>>>>>>>>> Any advice and suggestions for this idea or new idea will be
>>>>>>>>> greatly appreciated!
>>>>>>>>>
>>>>>>>>> Best regards,
>>>>>>>>>
>>>>>>>>> James zhang
>>>>>>>>>
>>>>>>>> Sounds very good. And we may also need features bits
>>>>>>>> (VIRTIO_NET_F_GUEST|HOST_GSO_X) for this.
>>>>>>>>
>>>>>>>> This is in fact one of items in networking todo list. (See
>>>>>>>> http://www.linux-kvm.org/page/NetworkingTodo). While at it, we'd
>>>>>>>> better support not only VXLAN but also other tunnels.
>>>>>>> Cc Vlad who is working on extending virtio-net headers.
>>>>>>>
>>>>>>>> We can start with the spec work, or if you've already had some
>>>>>>>> bits you can post them as RFC for early review.
>>>>>>>>
>>>>>>>> Thanks
>>>>>> Below is my demo code
>>>>>> Virtio_net.c
>>>>>> static int virtnet_probe(struct virtio_device *vdev), add belows codes:
>>>>>> if (virtio_has_feature(vdev, VIRTIO_NET_F_MRG_RXBUF) || // avoid gso segment, it should be negotiation later, because in the demo I reuse num_buffers.
>>>>>> virtio_has_feature(vdev, VIRTIO_F_VERSION_1)) {
>>>>>> dev->hw_enc_features |= NETIF_F_TSO;
>>>>>> dev->hw_enc_features |= NETIF_F_ALL_CSUM;
>>>>>> dev->hw_enc_features |= NETIF_F_GSO_UDP_TUNNEL;
>>>>>> dev->hw_enc_features |= NETIF_F_GSO_UDP_TUNNEL_CSUM;
>>>>>> dev->hw_enc_features |=
>>>>>> NETIF_F_GSO_TUNNEL_REMCSUM;
>>>>>>
>>>>>> dev->features |= NETIF_F_GSO_UDP_TUNNEL;
>>>>>> dev->features |= NETIF_F_GSO_UDP_TUNNEL_CSUM;
>>>>>> dev->features |= NETIF_F_GSO_TUNNEL_REMCSUM;
>>>>>> }
>>>>>>
>>>>>> static int xmit_skb(struct send_queue *sq, struct sk_buff *skb), add
>>>>>> below to pieces of codes
>>>>>>
>>>>>> if (skb_shinfo(skb)->gso_type & SKB_GSO_UDP_TUNNEL)
>>>>>> hdr->hdr.gso_type |= VIRTIO_NET_HDR_GSO_TUNNEL;
>>>>>> if (skb_shinfo(skb)->gso_type & SKB_GSO_UDP_TUNNEL_CSUM)
>>>>>> hdr->hdr.gso_type |= VIRTIO_NET_HDR_GSO_TUNNEL_CSUM;
>>>>>> if (skb_shinfo(skb)->gso_type & SKB_GSO_TUNNEL_REMCSUM)
>>>>>> hdr->hdr.gso_type |=
>>>>>> VIRTIO_NET_HDR_GSO_TUNNEL_REMCSUM;
>>>>>>
>>>>>> if (skb->encapsulation && skb_is_gso(skb)) {
>>>>>> inner_mac_len = skb_inner_network_header(skb) - skb_inner_mac_header(skb);
>>>>>> tnl_len = skb_inner_mac_header(skb) - skb_mac_header(skb);
>>>>>> if ( !(inner_mac_len >> DATA_LEN_SHIFT) && !(tnl_len >> DATA_LEN_SHIFT) ) {
>>>>>> hdr->hdr.flags |= VIRTIO_NET_HDR_F_ENCAPSULATION;
>>>>>> hdr->num_buffers = (__virtio16)((inner_mac_len << DATA_LEN_SHIFT) | tnl_len); //we reuse num_buffers for simple , we should add extend member for later.
>>>>>> } else
>>>>>> hdr->num_buffers = 0;
>>>>>> }
>>>>>>
>>>>>> Tun.c
>>>>>> if (memcpy_fromiovecend((void *)&hdr, iv, offset, tun->vnet_hdr_sz)) //read header with negotiation length
>>>>>> return -EFAULT;
>>>>>>
>>>>>> if (hdr.gso_type & VIRTIO_NET_HDR_GSO_TUNNEL) //set tunnel gso info
>>>>>> skb_shinfo(skb)->gso_type |= SKB_GSO_UDP_TUNNEL;
>>>>>> if (hdr.gso_type & VIRTIO_NET_HDR_GSO_TUNNEL_CSUM)
>>>>>> skb_shinfo(skb)->gso_type |= SKB_GSO_UDP_TUNNEL_CSUM;
>>>>>> if (hdr.gso_type & VIRTIO_NET_HDR_GSO_TUNNEL_REMCSUM)
>>>>>> skb_shinfo(skb)->gso_type |=
>>>>>> SKB_GSO_TUNNEL_REMCSUM;
>>>>>>
>>>>>> if (hdr.flags & VIRTIO_NET_HDR_F_ENCAPSULATION) { //read tunnel info from header and set to built skb.
>>>>>> tnl_len = tun16_to_cpu(tun, hdr.num_buffers) & TUN_TNL_LEN_MASK;
>>>>>> payload_mac_len = tun16_to_cpu(tun, hdr.num_buffers) >> TUN_DATA_LEN_SHIFT;
>>>>>> mac_len = skb_network_header(skb) - skb_mac_header(skb);
>>>>>> skb_set_inner_mac_header(skb, tnl_len - mac_len);
>>>>>> skb_set_inner_network_header(skb, tnl_len + payload_mac_len - mac_len);
>>>>>> skb->encapsulation = 1;
>>>>>> }
>>>>>>
>>>>>>
>>>>> Something like this, and you probably need do something more:
>>>>>
>>>>> - use net-next.git to generate the patch (for the latest code)
>>>>> - add feature negotiation
>>>>> - tun/macvtap/qemu patches for this, you can start with tun/macvtap
>>>>> patches
>>>>> - support for all other SKB_GSO_* types which is not supported
>>>>> - use a new field instead of num_buffers
>>>>> - a virtio spec patch to describe the support for encapsulation
>>>>> offload
>>>>>
>>>>> Thanks
>>>> Thank you for your advice, I will start it right now.
>>>>
>>>> Thanks
>>> Cool, one more question: while at it, I think you may want to add support for dpdk too?
>>>
>>> Thanks
>> Do you mean that the patch should be compatible with virtio pmd, or give virtio pmd patch?
>>
>> Thanks
>
> I mean it's better to prepare patches for both virtio pmd and dpdk.
>
> Thanks
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2016-11-17 23:10 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <DBCD2614ECF3FF4087A2C27CA80E34DD5401AC8B@SZXEMA501-MBX.china.huawei.com>
2016-11-08 11:13 ` Virtio_net support vxlan encapsulation package TSO offload discuss Jason Wang
2016-11-08 11:17 ` Jason Wang
2016-11-08 11:58 ` Zhangming (James, Euler)
2016-11-09 7:14 ` Jason Wang
2016-11-10 6:19 ` Zhangming (James, Euler)
2016-11-15 3:27 ` Jason Wang
2016-11-17 1:31 ` Zhangming (James, Euler)
2016-11-17 7:27 ` Jason Wang
2016-11-17 23:10 ` Jarno Rajahalme
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).