Using multi-queue with RSS on X540 VF

Using multi-queue with RSS on X540 VF

[Anant Mudambi]

Hi all,
I am using DPDK 1.7.0.
Qemu-KVM system with the Intel 10G X540 NIC; host has the Linux ixgbe PF driver and a guest is using a VF with the DPDK ixgbevf PMD (uio).
I am trying to configure multiple (2 for now) receive queues on the VF and use RSS.
I see packets coming only to queue 0 for some reason. Some questions about this:

1] I see that most of the initial configuration to make multi-queue + RSS work is done in ixgbe_dev_mq_rx_configure()
     [e.g. Setting up the redirect table and the MRQC register]. However, only the non-VF ixgbe_dev_rx_init() calls the above function;
     ixgbevf_dev_rx_init() doesn't configure RSS-related hardware. Don't we need to do this configuration to use RSS on a VF?
2] Even after configuring the RSS-related hardware in ixgbevf PMD, packets only come to queue 0. I printed out the RSS-type and RSS-hash-value
     fields from the write back Rx descriptor and they are both set to 0. So for some reason no RSS hash is being done for any packets.
3] Has anyone seen this kind of configuration work (multi-queue + RSS + VF)? I have an X540 NIC but has this worked on any other NIC for anyone?
4] Also any ideas on how to debug this problem further?

Thanks,
Anant

Re: Using multi-queue with RSS on X540 VF

[Bruce Richardson]

On Sat, Dec 13, 2014 at 01:44:36AM +0000, Anant Mudambi wrote:
> Hi all,
> I am using DPDK 1.7.0.
> Qemu-KVM system with the Intel 10G X540 NIC; host has the Linux ixgbe PF driver and a guest is using a VF with the DPDK ixgbevf PMD (uio).
> I am trying to configure multiple (2 for now) receive queues on the VF and use RSS.
> I see packets coming only to queue 0 for some reason. Some questions about this:
> 
> 1] I see that most of the initial configuration to make multi-queue + RSS work is done in ixgbe_dev_mq_rx_configure()
>      [e.g. Setting up the redirect table and the MRQC register]. However, only the non-VF ixgbe_dev_rx_init() calls the above function;
>      ixgbevf_dev_rx_init() doesn't configure RSS-related hardware. Don't we need to do this configuration to use RSS on a VF?
> 2] Even after configuring the RSS-related hardware in ixgbevf PMD, packets only come to queue 0. I printed out the RSS-type and RSS-hash-value
>      fields from the write back Rx descriptor and they are both set to 0. So for some reason no RSS hash is being done for any packets.
> 3] Has anyone seen this kind of configuration work (multi-queue + RSS + VF)? I have an X540 NIC but has this worked on any other NIC for anyone?
> 4] Also any ideas on how to debug this problem further?
> 
> Thanks,
> Anant

RSS redirection of traffic in a VF is not supported by the hardware, so you are
pretty much stuck with all VF traffic going to queue 0. [Later hardware, such as
4x10G NICs e.g. "Intel® Ethernet Converged Network Adapters XL710" using the
i40e driver should support RSS with VFs]

/Bruce

Re: Using multi-queue with RSS on X540 VF

[Bruce Richardson]

On Mon, Dec 15, 2014 at 10:47:40AM +0000, Bruce Richardson wrote:
> On Sat, Dec 13, 2014 at 01:44:36AM +0000, Anant Mudambi wrote:
> > Hi all,
> > I am using DPDK 1.7.0.
> > Qemu-KVM system with the Intel 10G X540 NIC; host has the Linux ixgbe PF driver and a guest is using a VF with the DPDK ixgbevf PMD (uio).
> > I am trying to configure multiple (2 for now) receive queues on the VF and use RSS.
> > I see packets coming only to queue 0 for some reason. Some questions about this:
> > 
> > 1] I see that most of the initial configuration to make multi-queue + RSS work is done in ixgbe_dev_mq_rx_configure()
> >      [e.g. Setting up the redirect table and the MRQC register]. However, only the non-VF ixgbe_dev_rx_init() calls the above function;
> >      ixgbevf_dev_rx_init() doesn't configure RSS-related hardware. Don't we need to do this configuration to use RSS on a VF?
> > 2] Even after configuring the RSS-related hardware in ixgbevf PMD, packets only come to queue 0. I printed out the RSS-type and RSS-hash-value
> >      fields from the write back Rx descriptor and they are both set to 0. So for some reason no RSS hash is being done for any packets.
> > 3] Has anyone seen this kind of configuration work (multi-queue + RSS + VF)? I have an X540 NIC but has this worked on any other NIC for anyone?
> > 4] Also any ideas on how to debug this problem further?
> > 
> > Thanks,
> > Anant
> 
> RSS redirection of traffic in a VF is not supported by the hardware, so you are
> pretty much stuck with all VF traffic going to queue 0. [Later hardware, such as
> 4x10G NICs e.g. "Intel® Ethernet Converged Network Adapters XL710" using the
> i40e driver should support RSS with VFs]
> 
> /Bruce

Actually, I see Changchun has posted a patch giving support for RSS use with VF's
so apparently some functionality is possible with 10G NICs such as the x540. There
may be some limitations with it, however, but I'll defer any further explanation
to him. :-)

/Bruce

Re: Using multi-queue with RSS on X540 VF

[Anant Mudambi]

Thanks Bruce. 
Hi Changchun,
I looked at your patch set and have some questions about it. I might be
missing something basic.
- How is the RTE_ETH_DEV_SRIOV(rte_eth_dev *).active set for VF ports (in
guest using ixgbevf PMD)? The only place I saw this field being
  set is in ixgbe_pf_host_init() but this is not called from
eth_ixgbevf_dev_init(), only from eth_ixgbe_dev_init().
- Your patch includes some changes to ixgbe_dev_mq_rx_configure() to
support VF RSS, but this function is not called from the ixgbevf PMD, only
from ixgbe PMD?
- Do you know if RSS on VF (using DPDK PMD) works with Linux ixgbe PF
driver on host? What parameters does the PF driver need to be loaded with?

Thanks,
Anant

On 12/15/14 2:52 AM, "Bruce Richardson" <bruce.richardson-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:

>On Mon, Dec 15, 2014 at 10:47:40AM +0000, Bruce Richardson wrote:
>> On Sat, Dec 13, 2014 at 01:44:36AM +0000, Anant Mudambi wrote:
>> > Hi all,
>> > I am using DPDK 1.7.0.
>> > Qemu-KVM system with the Intel 10G X540 NIC; host has the Linux ixgbe
>>PF driver and a guest is using a VF with the DPDK ixgbevf PMD (uio).
>> > I am trying to configure multiple (2 for now) receive queues on the
>>VF and use RSS.
>> > I see packets coming only to queue 0 for some reason. Some questions
>>about this:
>> > 
>> > 1] I see that most of the initial configuration to make multi-queue +
>>RSS work is done in ixgbe_dev_mq_rx_configure()
>> >      [e.g. Setting up the redirect table and the MRQC register].
>>However, only the non-VF ixgbe_dev_rx_init() calls the above function;
>> >      ixgbevf_dev_rx_init() doesn't configure RSS-related hardware.
>>Don't we need to do this configuration to use RSS on a VF?
>> > 2] Even after configuring the RSS-related hardware in ixgbevf PMD,
>>packets only come to queue 0. I printed out the RSS-type and
>>RSS-hash-value
>> >      fields from the write back Rx descriptor and they are both set
>>to 0. So for some reason no RSS hash is being done for any packets.
>> > 3] Has anyone seen this kind of configuration work (multi-queue + RSS
>>+ VF)? I have an X540 NIC but has this worked on any other NIC for
>>anyone?
>> > 4] Also any ideas on how to debug this problem further?
>> > 
>> > Thanks,
>> > Anant
>> 
>> RSS redirection of traffic in a VF is not supported by the hardware, so
>>you are
>> pretty much stuck with all VF traffic going to queue 0. [Later
>>hardware, such as
>> 4x10G NICs e.g. "Intel® Ethernet Converged Network Adapters XL710"
>>using the
>> i40e driver should support RSS with VFs]
>> 
>> /Bruce
>
>Actually, I see Changchun has posted a patch giving support for RSS use
>with VF's
>so apparently some functionality is possible with 10G NICs such as the
>x540. There
>may be some limitations with it, however, but I'll defer any further
>explanation
>to him. :-)
>
>/Bruce

Re: Using multi-queue with RSS on X540 VF

[Ouyang, Changchun]

Hi  Anant,

> -----Original Message-----
> From: Anant Mudambi [mailto:amudambi-HfF3sdhuK0iTtA8H5PvdGFaTQe2KTcn/@public.gmane.org]
> Sent: Friday, December 19, 2014 6:39 AM
> To: Richardson, Bruce; Ouyang, Changchun
> Cc: dev-VfR2kkLFssw@public.gmane.org
> Subject: Re: [dpdk-dev] Using multi-queue with RSS on X540 VF
> 
> Thanks Bruce.
> Hi Changchun,
> I looked at your patch set and have some questions about it. I might be
> missing something basic.
> - How is the RTE_ETH_DEV_SRIOV(rte_eth_dev *).active set for VF ports (in
> guest using ixgbevf PMD)? The only place I saw this field being
>   set is in ixgbe_pf_host_init() but this is not called from
> eth_ixgbevf_dev_init(), only from eth_ixgbe_dev_init().

Yes, pf on host will do it.

> - Your patch includes some changes to ixgbe_dev_mq_rx_configure() to
> support VF RSS, but this function is not called from the ixgbevf PMD, only
> from ixgbe PMD?

Yes, again pf on host will do it.

> - Do you know if RSS on VF (using DPDK PMD) works with Linux ixgbe PF
> driver on host? What parameters does the PF driver need to be loaded with?
>
Not test on that case of dpdk vf with linux ixgbe pf yet, 
But it works on the case of dpdk vf with dpdk ixgbe pf on Niantic. 

Thanks
Changchun