Question about virtio-net: implement RX RSS processing

Question about virtio-net: implement RX RSS processing

[Vincent Li]

Hi,

I noticed the [PULL V2 02/33] virtio-net: implement RX RSS processing 
https://marc.info/?l=qemu-devel&m=159248675403246&w=2 and cloned  
https://github.com/jasowang/qemu.git tags/net-pull-request for testing the 
RX RSS feature, but I am not clear how to test this feature and see if it 
meets my needs.

I am running F-Stack Nginx applications 
https://github.com/F-Stack/f-stack/tree/dev/app/nginx-1.16.1 in KVM guest, 
F-Stack is FreeBSD TCP/IP stack ported to support DPDK, and F-Stack set 
RSS mode as code below 
https://github.com/F-Stack/f-stack/blob/dev/lib/ff_dpdk_if.c#L605

     /* Set RSS mode */
     uint64_t default_rss_hf = ETH_RSS_PROTO_MASK;
     port_conf.rxmode.mq_mode = ETH_MQ_RX_RSS;
     port_conf.rx_adv_conf.rss_conf.rss_hf = default_rss_hf;
     if (dev_info.hash_key_size == 52) {
         port_conf.rx_adv_conf.rss_conf.rss_key = default_rsskey_52bytes;
         port_conf.rx_adv_conf.rss_conf.rss_key_len = 52;
         use_rsskey_52bytes = 1;
     } else {
         port_conf.rx_adv_conf.rss_conf.rss_key = default_rsskey_40bytes;
         port_conf.rx_adv_conf.rss_conf.rss_key_len = 40;
     }
     port_conf.rx_adv_conf.rss_conf.rss_hf &= dev_info.flow_type_rss_offloads;
     if (port_conf.rx_adv_conf.rss_conf.rss_hf != ETH_RSS_PROTO_MASK) {
         printf("Port %u modified RSS hash function based on hardware support,"
         "requested:%#"PRIx64" configured:%#"PRIx64"\n",
         port_id, default_rss_hf, port_conf.rx_adv_conf.rss_conf.rss_hf);
     }

But  DPDK virtio PMD does not support RSS as below commit shows:

commit 13b3137f3b7c8f866947a9b34e06a8aec0d084f7
Author: Dilshod Urazov 
Date:   Wed Oct 9 13:32:07 2019 +0100

    net/virtio: reject unsupported Rx multi-queue modes
    
    This driver supports none of DCB, RSS or VMDQ modes, therefore must
    check and return error if configured incorrectly.
    
    Virtio can distribute Rx packets across multi-queue, but there is
    no controls (algorithm, redirection table, hash function) except
    number of Rx queues and ETH_MQ_RX_NONE is the best fit meaning
    no method is enforced on how to route packets to MQs.
    
    Fixes: c1f86306a026 ("virtio: add new driver")
    Cc: stable@dpdk.org
    
    Signed-off-by: Dilshod Urazov 
    Signed-off-by: Andrew Rybchenko 
    Reviewed-by: Maxime Coquelin 

diff --git a/drivers/net/virtio/virtio_ethdev.c 
           b/drivers/net/virtio/virtio_ethdev.c
index 0a2ed2e50..76bd40a3e 100644
--- a/drivers/net/virtio/virtio_ethdev.c
+++ b/drivers/net/virtio/virtio_ethdev.c
@@ -2066,6 +2066,13 @@ virtio_dev_configure(struct rte_eth_dev *dev)
        PMD_INIT_LOG(DEBUG, "configure");
        req_features = VIRTIO_PMD_DEFAULT_GUEST_FEATURES;
 
+       if (rxmode->mq_mode != ETH_MQ_RX_NONE) {
+               PMD_DRV_LOG(ERR,
+                       "Unsupported Rx multi queue mode %d",
+                       rxmode->mq_mode);
+               return -EINVAL;
+       }
+
        if (dev->data->dev_conf.intr_conf.rxq) {
                ret = virtio_init_device(dev, hw->req_guest_features);
                if (ret < 0)



So the problem is I can't run F-Stack Applications in KVM/Qemu guest with 
multi queue/vCPU/RSS  support, this problem seems apply to DPDK TCP 
applications require multi queue/RSS support in KVM/Qemu guest, for 
example mTCP https://github.com/mtcp-stack/mtcp I tested has similar 
problem.

I am not clear on the picture of how everything work together for  this 
virtio-net RSS feature.

I have read following blogs

https://www.redhat.com/en/blog/introduction-virtio-networking-and-vhost-net
https://www.redhat.com/en/blog/how-vhost-user-came-being-virtio-networking-and-dpdk

Someone told me that in order for DPDK frond end virtio PMD in guest support RSS, the backend 
also needs to support RSS, including vhost-net and vhost-user, it should 
have nothing to do with this Qemu virtio-net RSS, is that correct?  if 
correct, I have following questions:

1, What is the use case for this Qemu virtio-net RSS?
2, How to test the use case?
3, Are there any plan to improve vhost-net/vhost-user, DPDK virtio PMD to support RSS?

For 3,  I think this is important for KVM/Qemu/OVS-DPDK/Vhost-net environment for DPDK TCP/UDP applications. 

Note I have no problem running F-Stack or mTCP applications in VMware ESXi 
guest environment with multi queue/vCPU/RSS requirement since DPDK vmxnet3 
PMD support RSS and I assume VMware ESXi backend support RSS, so it looks VMware has 
advantage over this.

Thank you for your patience to read this email 

Regards,

Vincent     

Re: Question about virtio-net: implement RX RSS processing

[Jason Wang]

On 2020/7/3 上午1:07, Vincent Li wrote:
> Hi,
>
> I noticed the [PULL V2 02/33] virtio-net: implement RX RSS processing
> https://marc.info/?l=qemu-devel&m=159248675403246&w=2 and cloned
> https://github.com/jasowang/qemu.git tags/net-pull-request for testing the
> RX RSS feature, but I am not clear how to test this feature and see if it
> meets my needs.


Yuri may know more but I think the only driver that supports RSS is 
Windows driver currently.


>
> I am running F-Stack Nginx applications
> https://github.com/F-Stack/f-stack/tree/dev/app/nginx-1.16.1 in KVM guest,
> F-Stack is FreeBSD TCP/IP stack ported to support DPDK, and F-Stack set
> RSS mode as code below
> https://github.com/F-Stack/f-stack/blob/dev/lib/ff_dpdk_if.c#L605
>
>       /* Set RSS mode */
>       uint64_t default_rss_hf = ETH_RSS_PROTO_MASK;
>       port_conf.rxmode.mq_mode = ETH_MQ_RX_RSS;
>       port_conf.rx_adv_conf.rss_conf.rss_hf = default_rss_hf;
>       if (dev_info.hash_key_size == 52) {
>           port_conf.rx_adv_conf.rss_conf.rss_key = default_rsskey_52bytes;
>           port_conf.rx_adv_conf.rss_conf.rss_key_len = 52;
>           use_rsskey_52bytes = 1;
>       } else {
>           port_conf.rx_adv_conf.rss_conf.rss_key = default_rsskey_40bytes;
>           port_conf.rx_adv_conf.rss_conf.rss_key_len = 40;
>       }
>       port_conf.rx_adv_conf.rss_conf.rss_hf &= dev_info.flow_type_rss_offloads;
>       if (port_conf.rx_adv_conf.rss_conf.rss_hf != ETH_RSS_PROTO_MASK) {
>           printf("Port %u modified RSS hash function based on hardware support,"
>           "requested:%#"PRIx64" configured:%#"PRIx64"\n",
>           port_id, default_rss_hf, port_conf.rx_adv_conf.rss_conf.rss_hf);
>       }
>
> But  DPDK virtio PMD does not support RSS as below commit shows:
>
> commit 13b3137f3b7c8f866947a9b34e06a8aec0d084f7
> Author: Dilshod Urazov
> Date:   Wed Oct 9 13:32:07 2019 +0100
>
>      net/virtio: reject unsupported Rx multi-queue modes
>      
>      This driver supports none of DCB, RSS or VMDQ modes, therefore must
>      check and return error if configured incorrectly.
>      
>      Virtio can distribute Rx packets across multi-queue, but there is
>      no controls (algorithm, redirection table, hash function) except
>      number of Rx queues and ETH_MQ_RX_NONE is the best fit meaning
>      no method is enforced on how to route packets to MQs.
>      
>      Fixes: c1f86306a026 ("virtio: add new driver")
>      Cc: stable@dpdk.org
>      
>      Signed-off-by: Dilshod Urazov
>      Signed-off-by: Andrew Rybchenko
>      Reviewed-by: Maxime Coquelin
>
> diff --git a/drivers/net/virtio/virtio_ethdev.c
>             b/drivers/net/virtio/virtio_ethdev.c
> index 0a2ed2e50..76bd40a3e 100644
> --- a/drivers/net/virtio/virtio_ethdev.c
> +++ b/drivers/net/virtio/virtio_ethdev.c
> @@ -2066,6 +2066,13 @@ virtio_dev_configure(struct rte_eth_dev *dev)
>          PMD_INIT_LOG(DEBUG, "configure");
>          req_features = VIRTIO_PMD_DEFAULT_GUEST_FEATURES;
>   
> +       if (rxmode->mq_mode != ETH_MQ_RX_NONE) {
> +               PMD_DRV_LOG(ERR,
> +                       "Unsupported Rx multi queue mode %d",
> +                       rxmode->mq_mode);
> +               return -EINVAL;
> +       }
> +
>          if (dev->data->dev_conf.intr_conf.rxq) {
>                  ret = virtio_init_device(dev, hw->req_guest_features);
>                  if (ret < 0)
>
>
>
> So the problem is I can't run F-Stack Applications in KVM/Qemu guest with
> multi queue/vCPU/RSS  support, this problem seems apply to DPDK TCP
> applications require multi queue/RSS support in KVM/Qemu guest, for
> example mTCP https://github.com/mtcp-stack/mtcp I tested has similar
> problem.
>
> I am not clear on the picture of how everything work together for  this
> virtio-net RSS feature.
>
> I have read following blogs
>
> https://www.redhat.com/en/blog/introduction-virtio-networking-and-vhost-net
> https://www.redhat.com/en/blog/how-vhost-user-came-being-virtio-networking-and-dpdk
>
> Someone told me that in order for DPDK frond end virtio PMD in guest support RSS, the backend
> also needs to support RSS, including vhost-net and vhost-user, it should
> have nothing to do with this Qemu virtio-net RSS, is that correct?  if
> correct, I have following questions:
>
> 1, What is the use case for this Qemu virtio-net RSS?


It's just RSS as what other device can provide for steering or balancing.


> 2, How to test the use case?


Need use windows guest.


> 3, Are there any plan to improve vhost-net/vhost-user, DPDK virtio PMD to support RSS?


For vhost-net, Sameeh posted a eBPF based solution RFC[1], we need some 
one to carry on the work. It doesn't request any extension to vhost-net 
thanks to the steering eBPF support in tuntap.

For vhost-user, we need probably extend vhost-user protocols first.

You're welcome to contribute patches.

[1] https://patchwork.kernel.org/cover/10581921/

Thanks


>
> For 3,  I think this is important for KVM/Qemu/OVS-DPDK/Vhost-net environment for DPDK TCP/UDP applications.
>
> Note I have no problem running F-Stack or mTCP applications in VMware ESXi
> guest environment with multi queue/vCPU/RSS requirement since DPDK vmxnet3
> PMD support RSS and I assume VMware ESXi backend support RSS, so it looks VMware has
> advantage over this.
>
> Thank you for your patience to read this email
>
> Regards,
>
> Vincent
>

Re: Question about virtio-net: implement RX RSS processing

[Vincent Li]

[-- Attachment #1: Type: text/plain, Size: 5842 bytes --]


Thank you for the answers and I will study the vhost-net patches set you 
mentioned

On Fri, 3 Jul 2020, Jason Wang wrote:

> 
> On 2020/7/3 上午1:07, Vincent Li wrote:
> > Hi,
> > 
> > I noticed the [PULL V2 02/33] virtio-net: implement RX RSS processing
> > https://marc.info/?l=qemu-devel&m=159248675403246&w=2 and cloned
> > https://github.com/jasowang/qemu.git tags/net-pull-request for testing the
> > RX RSS feature, but I am not clear how to test this feature and see if it
> > meets my needs.
> 
> 
> Yuri may know more but I think the only driver that supports RSS is Windows
> driver currently.
> 
> 
> > 
> > I am running F-Stack Nginx applications
> > https://github.com/F-Stack/f-stack/tree/dev/app/nginx-1.16.1 in KVM guest,
> > F-Stack is FreeBSD TCP/IP stack ported to support DPDK, and F-Stack set
> > RSS mode as code below
> > https://github.com/F-Stack/f-stack/blob/dev/lib/ff_dpdk_if.c#L605
> > 
> >       /* Set RSS mode */
> >       uint64_t default_rss_hf = ETH_RSS_PROTO_MASK;
> >       port_conf.rxmode.mq_mode = ETH_MQ_RX_RSS;
> >       port_conf.rx_adv_conf.rss_conf.rss_hf = default_rss_hf;
> >       if (dev_info.hash_key_size == 52) {
> >           port_conf.rx_adv_conf.rss_conf.rss_key = default_rsskey_52bytes;
> >           port_conf.rx_adv_conf.rss_conf.rss_key_len = 52;
> >           use_rsskey_52bytes = 1;
> >       } else {
> >           port_conf.rx_adv_conf.rss_conf.rss_key = default_rsskey_40bytes;
> >           port_conf.rx_adv_conf.rss_conf.rss_key_len = 40;
> >       }
> >       port_conf.rx_adv_conf.rss_conf.rss_hf &=
> > dev_info.flow_type_rss_offloads;
> >       if (port_conf.rx_adv_conf.rss_conf.rss_hf != ETH_RSS_PROTO_MASK) {
> >           printf("Port %u modified RSS hash function based on hardware
> > support,"
> >           "requested:%#"PRIx64" configured:%#"PRIx64"\n",
> >           port_id, default_rss_hf, port_conf.rx_adv_conf.rss_conf.rss_hf);
> >       }
> > 
> > But  DPDK virtio PMD does not support RSS as below commit shows:
> > 
> > commit 13b3137f3b7c8f866947a9b34e06a8aec0d084f7
> > Author: Dilshod Urazov
> > Date:   Wed Oct 9 13:32:07 2019 +0100
> > 
> >      net/virtio: reject unsupported Rx multi-queue modes
> >           This driver supports none of DCB, RSS or VMDQ modes, therefore
> > must
> >      check and return error if configured incorrectly.
> >           Virtio can distribute Rx packets across multi-queue, but there is
> >      no controls (algorithm, redirection table, hash function) except
> >      number of Rx queues and ETH_MQ_RX_NONE is the best fit meaning
> >      no method is enforced on how to route packets to MQs.
> >           Fixes: c1f86306a026 ("virtio: add new driver")
> >      Cc: stable@dpdk.org
> >           Signed-off-by: Dilshod Urazov
> >      Signed-off-by: Andrew Rybchenko
> >      Reviewed-by: Maxime Coquelin
> > 
> > diff --git a/drivers/net/virtio/virtio_ethdev.c
> >             b/drivers/net/virtio/virtio_ethdev.c
> > index 0a2ed2e50..76bd40a3e 100644
> > --- a/drivers/net/virtio/virtio_ethdev.c
> > +++ b/drivers/net/virtio/virtio_ethdev.c
> > @@ -2066,6 +2066,13 @@ virtio_dev_configure(struct rte_eth_dev *dev)
> >          PMD_INIT_LOG(DEBUG, "configure");
> >          req_features = VIRTIO_PMD_DEFAULT_GUEST_FEATURES;
> >   +       if (rxmode->mq_mode != ETH_MQ_RX_NONE) {
> > +               PMD_DRV_LOG(ERR,
> > +                       "Unsupported Rx multi queue mode %d",
> > +                       rxmode->mq_mode);
> > +               return -EINVAL;
> > +       }
> > +
> >          if (dev->data->dev_conf.intr_conf.rxq) {
> >                  ret = virtio_init_device(dev, hw->req_guest_features);
> >                  if (ret < 0)
> > 
> > 
> > 
> > So the problem is I can't run F-Stack Applications in KVM/Qemu guest with
> > multi queue/vCPU/RSS  support, this problem seems apply to DPDK TCP
> > applications require multi queue/RSS support in KVM/Qemu guest, for
> > example mTCP https://github.com/mtcp-stack/mtcp I tested has similar
> > problem.
> > 
> > I am not clear on the picture of how everything work together for  this
> > virtio-net RSS feature.
> > 
> > I have read following blogs
> > 
> > https://www.redhat.com/en/blog/introduction-virtio-networking-and-vhost-net
> > https://www.redhat.com/en/blog/how-vhost-user-came-being-virtio-networking-and-dpdk
> > 
> > Someone told me that in order for DPDK frond end virtio PMD in guest support
> > RSS, the backend
> > also needs to support RSS, including vhost-net and vhost-user, it should
> > have nothing to do with this Qemu virtio-net RSS, is that correct?  if
> > correct, I have following questions:
> > 
> > 1, What is the use case for this Qemu virtio-net RSS?
> 
> 
> It's just RSS as what other device can provide for steering or balancing.
> 
> 
> > 2, How to test the use case?
> 
> 
> Need use windows guest.
> 
> 
> > 3, Are there any plan to improve vhost-net/vhost-user, DPDK virtio PMD to
> > support RSS?
> 
> 
> For vhost-net, Sameeh posted a eBPF based solution RFC[1], we need some one to
> carry on the work. It doesn't request any extension to vhost-net thanks to the
> steering eBPF support in tuntap.
> 
> For vhost-user, we need probably extend vhost-user protocols first.
> 
> You're welcome to contribute patches.
> 
> [1] https://patchwork.kernel.org/cover/10581921/
> 
> Thanks
> 
> 
> > 
> > For 3,  I think this is important for KVM/Qemu/OVS-DPDK/Vhost-net
> > environment for DPDK TCP/UDP applications.
> > 
> > Note I have no problem running F-Stack or mTCP applications in VMware ESXi
> > guest environment with multi queue/vCPU/RSS requirement since DPDK vmxnet3
> > PMD support RSS and I assume VMware ESXi backend support RSS, so it looks
> > VMware has
> > advantage over this.
> > 
> > Thank you for your patience to read this email
> > 
> > Regards,
> > 
> > Vincent
> > 
> 
>