82599 SR-IOV with passthrough

82599 SR-IOV with passthrough

[jigsaw]

Hi,

I am doing experiments with SR-IOV + passthrough on 82599.
My expectation is to have VT on and DCB off, under which
configuration, the total 128 TX queues will be split into 32 pools,
each has 4 queues.

With latest driver ixgbe-3.18.7, PF can be set with 16 pools, each has
4 queues with these params:

insmod ./ixgbe.ko MQ=1 max_vfs=8 RSS=8 VMDQ=16

I tried with VMDQ=32 with a panic. Also, it seems that if RSS is set
to 4, the PF driver will set
RSS to 2 somehow.
Since I'm fine with 16 pools + 4 queues,  I'm not going to investigate
(at this moment) why PF doesn't work as expected.
Next step is then to try DPDK in guest OS, which get one VF by passthrough.

Not surprisingly, DPDK says that number of TX queue is 1. This is
because the value is set arbitrarily in ixgbe_init_ops_vf of
ixgbe_vf.c, and it never gets updated.

Actually the mbox API has support for requesting Tx/Rx queue numbers
from VF. See implementation of routine ixgbevf_get_queues and
ixgbevf_negotiate_api_version.
However, it is not straightforward to use these 2 routines to fetch
Tx/Rx queue number, coz the PF driver is not ready to be used without
modification.
See ixgbe_get_vf_queues of ixgbe_sriov.c in ixgbe-3.18.7.  The PF will
always answer with 1 for Tx/Rx queue number requests, regardless of
current config.

Therefore, to add support for multiple queues per VF, we have to at
least fix the PF driver, then add support in DPDK's VF driver.

But the question is, is this enough? Before doing any experiments I
wonder whether anybody has come across same problem as I do, and if
there's any implementation ongoing.

thx &
rgds,
-Qinglai

Re: 82599 SR-IOV with passthrough

[Thomas Monjalon]

16/10/2013 14:18, jigsaw :
> Therefore, to add support for multiple queues per VF, we have to at
> least fix the PF driver, then add support in DPDK's VF driver.

You're right, Linux PF driver have to be updated to properly manage 
multiple queues per VF. Then the guest can be tested with DPDK or with Linux 
driver (ixgbe_vf).

Note that there are 2 versions of Linux driver for ixgbe: kernel.org and 
sourceforge.net (supporting many kernel versions).

-- 
Thomas

Re: 82599 SR-IOV with passthrough

[jigsaw]

Hi Thomas,

Thanks for reply.

The kernel has older version of PF than the one released on sf.net. So
I'm checking the sf.net release.
If the change is limited in DPDK then it is controllable. But now it
affects Intel's PF driver, I don't even know how to push the feature
to Intel. The driver on sf.net is a read-only repository, isn't it? It
would be painful to maintain another branch of 10G PF driver.
Could Intel give some advice or hints here?

thx &
rgds,
-Qinglai

On Wed, Oct 16, 2013 at 3:58 PM, Thomas Monjalon
<thomas.monjalon-pdR9zngts4EAvxtiuMwx3w@public.gmane.org> wrote:
> 16/10/2013 14:18, jigsaw :
>> Therefore, to add support for multiple queues per VF, we have to at
>> least fix the PF driver, then add support in DPDK's VF driver.
>
> You're right, Linux PF driver have to be updated to properly manage
> multiple queues per VF. Then the guest can be tested with DPDK or with Linux
> driver (ixgbe_vf).
>
> Note that there are 2 versions of Linux driver for ixgbe: kernel.org and
> sourceforge.net (supporting many kernel versions).
>
> --
> Thomas

Re: 82599 SR-IOV with passthrough

[Prashant Upadhyaya]

Hi Qinglai,

Why are you using the kernel driver at all.
Use the DPDK driver to control the PF on the host. The guest would communicate with the PF on host using mailbox as usual.
Then the changes will be limited to DPDK, isn't it ?

Regards
-Prashant

-----Original Message-----
From: dev [mailto:dev-bounces-VfR2kkLFssw@public.gmane.org] On Behalf Of jigsaw
Sent: Wednesday, October 16, 2013 6:51 PM
To: Thomas Monjalon
Cc: dev-VfR2kkLFssw@public.gmane.org
Subject: Re: [dpdk-dev] 82599 SR-IOV with passthrough

Hi Thomas,

Thanks for reply.

The kernel has older version of PF than the one released on sf.net. So I'm checking the sf.net release.
If the change is limited in DPDK then it is controllable. But now it affects Intel's PF driver, I don't even know how to push the feature to Intel. The driver on sf.net is a read-only repository, isn't it? It would be painful to maintain another branch of 10G PF driver.
Could Intel give some advice or hints here?

thx &
rgds,
-Qinglai

On Wed, Oct 16, 2013 at 3:58 PM, Thomas Monjalon <thomas.monjalon-pdR9zngts4EAvxtiuMwx3w@public.gmane.org> wrote:
> 16/10/2013 14:18, jigsaw :
>> Therefore, to add support for multiple queues per VF, we have to at
>> least fix the PF driver, then add support in DPDK's VF driver.
>
> You're right, Linux PF driver have to be updated to properly manage
> multiple queues per VF. Then the guest can be tested with DPDK or with
> Linux driver (ixgbe_vf).
>
> Note that there are 2 versions of Linux driver for ixgbe: kernel.org
> and sourceforge.net (supporting many kernel versions).
>
> --
> Thomas




===============================================================================
Please refer to http://www.aricent.com/legal/email_disclaimer.html
for important disclosures regarding this electronic communication.
===============================================================================

Re: 82599 SR-IOV with passthrough

[jigsaw]

Hi Prashant,

I patched both Intel ixgbe PF driver and DPDK 1.5 VF driver, so that
DPDK gets 4 queues in one VF. It works fine with all 4 Tx queues. The
only trick is to set proper mac address for all outgoing packets,
which must be the same mac as you set to the VF. This trick is
described in the release note of DPDK.

I wonder whether it makes sense to push this patch to DPDK. Any comments?

thx &
rgds,
-ql

On Thu, Oct 17, 2013 at 2:55 PM, Prashant Upadhyaya
<prashant.upadhyaya-pccXkzZloW5BDgjK7y7TUQ@public.gmane.org> wrote:
> Hi Qinglai,
>
> Why are you using the kernel driver at all.
> Use the DPDK driver to control the PF on the host. The guest would communicate with the PF on host using mailbox as usual.
> Then the changes will be limited to DPDK, isn't it ?
>
> Regards
> -Prashant
>
> -----Original Message-----
> From: dev [mailto:dev-bounces-VfR2kkLFssw@public.gmane.org] On Behalf Of jigsaw
> Sent: Wednesday, October 16, 2013 6:51 PM
> To: Thomas Monjalon
> Cc: dev-VfR2kkLFssw@public.gmane.org
> Subject: Re: [dpdk-dev] 82599 SR-IOV with passthrough
>
> Hi Thomas,
>
> Thanks for reply.
>
> The kernel has older version of PF than the one released on sf.net. So I'm checking the sf.net release.
> If the change is limited in DPDK then it is controllable. But now it affects Intel's PF driver, I don't even know how to push the feature to Intel. The driver on sf.net is a read-only repository, isn't it? It would be painful to maintain another branch of 10G PF driver.
> Could Intel give some advice or hints here?
>
> thx &
> rgds,
> -Qinglai
>
> On Wed, Oct 16, 2013 at 3:58 PM, Thomas Monjalon <thomas.monjalon@6wind.com> wrote:
>> 16/10/2013 14:18, jigsaw :
>>> Therefore, to add support for multiple queues per VF, we have to at
>>> least fix the PF driver, then add support in DPDK's VF driver.
>>
>> You're right, Linux PF driver have to be updated to properly manage
>> multiple queues per VF. Then the guest can be tested with DPDK or with
>> Linux driver (ixgbe_vf).
>>
>> Note that there are 2 versions of Linux driver for ixgbe: kernel.org
>> and sourceforge.net (supporting many kernel versions).
>>
>> --
>> Thomas
>
>
>
>
> ===============================================================================
> Please refer to http://www.aricent.com/legal/email_disclaimer.html
> for important disclosures regarding this electronic communication.
> ===============================================================================

Re: 82599 SR-IOV with passthrough

[jigsaw]

Hi Prashant,

I'm using CentOS 6 as both host and guest, which is managed by virsh.
A quite old kernel with latest ixgbe PF driver, plus DPDK trunk
version running in guest.

BTW, DPDK cannot take over ixgbe PF. At least not for now. For
instance the PF mailbox is not implemented in DPDK.

thx &
rgds,
-ql

On Thu, Oct 17, 2013 at 3:43 PM, jigsaw <jigsaw-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> Hi Prashant,
>
> I patched both Intel ixgbe PF driver and DPDK 1.5 VF driver, so that
> DPDK gets 4 queues in one VF. It works fine with all 4 Tx queues. The
> only trick is to set proper mac address for all outgoing packets,
> which must be the same mac as you set to the VF. This trick is
> described in the release note of DPDK.
>
> I wonder whether it makes sense to push this patch to DPDK. Any comments?
>
> thx &
> rgds,
> -ql
>
> On Thu, Oct 17, 2013 at 2:55 PM, Prashant Upadhyaya
> <prashant.upadhyaya-pccXkzZloW5BDgjK7y7TUQ@public.gmane.org> wrote:
>> Hi Qinglai,
>>
>> Why are you using the kernel driver at all.
>> Use the DPDK driver to control the PF on the host. The guest would communicate with the PF on host using mailbox as usual.
>> Then the changes will be limited to DPDK, isn't it ?
>>
>> Regards
>> -Prashant
>>
>> -----Original Message-----
>> From: dev [mailto:dev-bounces-VfR2kkLFssw@public.gmane.org] On Behalf Of jigsaw
>> Sent: Wednesday, October 16, 2013 6:51 PM
>> To: Thomas Monjalon
>> Cc: dev-VfR2kkLFssw@public.gmane.org
>> Subject: Re: [dpdk-dev] 82599 SR-IOV with passthrough
>>
>> Hi Thomas,
>>
>> Thanks for reply.
>>
>> The kernel has older version of PF than the one released on sf.net. So I'm checking the sf.net release.
>> If the change is limited in DPDK then it is controllable. But now it affects Intel's PF driver, I don't even know how to push the feature to Intel. The driver on sf.net is a read-only repository, isn't it? It would be painful to maintain another branch of 10G PF driver.
>> Could Intel give some advice or hints here?
>>
>> thx &
>> rgds,
>> -Qinglai
>>
>> On Wed, Oct 16, 2013 at 3:58 PM, Thomas Monjalon <thomas.monjalon@6wind.com> wrote:
>>> 16/10/2013 14:18, jigsaw :
>>>> Therefore, to add support for multiple queues per VF, we have to at
>>>> least fix the PF driver, then add support in DPDK's VF driver.
>>>
>>> You're right, Linux PF driver have to be updated to properly manage
>>> multiple queues per VF. Then the guest can be tested with DPDK or with
>>> Linux driver (ixgbe_vf).
>>>
>>> Note that there are 2 versions of Linux driver for ixgbe: kernel.org
>>> and sourceforge.net (supporting many kernel versions).
>>>
>>> --
>>> Thomas
>>
>>
>>
>>
>> ===============================================================================
>> Please refer to http://www.aricent.com/legal/email_disclaimer.html
>> for important disclosures regarding this electronic communication.
>> ===============================================================================

Re: 82599 SR-IOV with passthrough

[Prashant Upadhyaya]

Hi Qinglai,

I would say that SRIOV is 'useless' if the VF gets only one queue.
At the heart of performance is to use one queue per core so that the the tx and rx remain lockless. Locks 'destroy' performance.
So with one queue, if we want to remain lockless, that automatically means that the usecase is restricted to one core, ergo useless for any usecase worth its salt.

It was courtesy your mail that  I 'discovered' that DPDK has such a limitation.

So I am all for this patch to go in DPDK. Good luck !

Regards
-Prashant


-----Original Message-----
From: jigsaw [mailto:jigsaw-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org]
Sent: Thursday, October 17, 2013 6:14 PM
To: Prashant Upadhyaya
Cc: Thomas Monjalon; dev-VfR2kkLFssw@public.gmane.org
Subject: Re: [dpdk-dev] 82599 SR-IOV with passthrough

Hi Prashant,

I patched both Intel ixgbe PF driver and DPDK 1.5 VF driver, so that DPDK gets 4 queues in one VF. It works fine with all 4 Tx queues. The only trick is to set proper mac address for all outgoing packets, which must be the same mac as you set to the VF. This trick is described in the release note of DPDK.

I wonder whether it makes sense to push this patch to DPDK. Any comments?

thx &
rgds,
-ql

On Thu, Oct 17, 2013 at 2:55 PM, Prashant Upadhyaya <prashant.upadhyaya@aricent.com> wrote:
> Hi Qinglai,
>
> Why are you using the kernel driver at all.
> Use the DPDK driver to control the PF on the host. The guest would communicate with the PF on host using mailbox as usual.
> Then the changes will be limited to DPDK, isn't it ?
>
> Regards
> -Prashant
>
> -----Original Message-----
> From: dev [mailto:dev-bounces-VfR2kkLFssw@public.gmane.org] On Behalf Of jigsaw
> Sent: Wednesday, October 16, 2013 6:51 PM
> To: Thomas Monjalon
> Cc: dev-VfR2kkLFssw@public.gmane.org
> Subject: Re: [dpdk-dev] 82599 SR-IOV with passthrough
>
> Hi Thomas,
>
> Thanks for reply.
>
> The kernel has older version of PF than the one released on sf.net. So I'm checking the sf.net release.
> If the change is limited in DPDK then it is controllable. But now it affects Intel's PF driver, I don't even know how to push the feature to Intel. The driver on sf.net is a read-only repository, isn't it? It would be painful to maintain another branch of 10G PF driver.
> Could Intel give some advice or hints here?
>
> thx &
> rgds,
> -Qinglai
>
> On Wed, Oct 16, 2013 at 3:58 PM, Thomas Monjalon <thomas.monjalon@6wind.com> wrote:
>> 16/10/2013 14:18, jigsaw :
>>> Therefore, to add support for multiple queues per VF, we have to at
>>> least fix the PF driver, then add support in DPDK's VF driver.
>>
>> You're right, Linux PF driver have to be updated to properly manage
>> multiple queues per VF. Then the guest can be tested with DPDK or
>> with Linux driver (ixgbe_vf).
>>
>> Note that there are 2 versions of Linux driver for ixgbe: kernel.org
>> and sourceforge.net (supporting many kernel versions).
>>
>> --
>> Thomas
>
>
>
>
> ======================================================================
> ========= Please refer to
> http://www.aricent.com/legal/email_disclaimer.html
> for important disclosures regarding this electronic communication.
> ======================================================================
> =========




===============================================================================
Please refer to http://www.aricent.com/legal/email_disclaimer.html
for important disclosures regarding this electronic communication.
===============================================================================

Re: 82599 SR-IOV with passthrough

[jigsaw]

Hi Prashant,

The problem is that my patch has to be applied to ixgbe PF driver as
well. I have no idea how to make it happen.
So even DPDK accepts my patch, user won't benefit from it unless he
patched ixgbe PF by himself.

I also hate the fact that SRIOV cannot get more queues to VF. But
there's a way out: to assign more than one VF to guest.


thx &
rgds,
-ql

On Thu, Oct 17, 2013 at 4:02 PM, Prashant Upadhyaya
<prashant.upadhyaya-pccXkzZloW5BDgjK7y7TUQ@public.gmane.org> wrote:
> Hi Qinglai,
>
> I would say that SRIOV is 'useless' if the VF gets only one queue.
> At the heart of performance is to use one queue per core so that the the tx and rx remain lockless. Locks 'destroy' performance.
> So with one queue, if we want to remain lockless, that automatically means that the usecase is restricted to one core, ergo useless for any usecase worth its salt.
>
> It was courtesy your mail that  I 'discovered' that DPDK has such a limitation.
>
> So I am all for this patch to go in DPDK. Good luck !
>
> Regards
> -Prashant
>
>
> -----Original Message-----
> From: jigsaw [mailto:jigsaw-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org]
> Sent: Thursday, October 17, 2013 6:14 PM
> To: Prashant Upadhyaya
> Cc: Thomas Monjalon; dev-VfR2kkLFssw@public.gmane.org
> Subject: Re: [dpdk-dev] 82599 SR-IOV with passthrough
>
> Hi Prashant,
>
> I patched both Intel ixgbe PF driver and DPDK 1.5 VF driver, so that DPDK gets 4 queues in one VF. It works fine with all 4 Tx queues. The only trick is to set proper mac address for all outgoing packets, which must be the same mac as you set to the VF. This trick is described in the release note of DPDK.
>
> I wonder whether it makes sense to push this patch to DPDK. Any comments?
>
> thx &
> rgds,
> -ql
>
> On Thu, Oct 17, 2013 at 2:55 PM, Prashant Upadhyaya <prashant.upadhyaya@aricent.com> wrote:
>> Hi Qinglai,
>>
>> Why are you using the kernel driver at all.
>> Use the DPDK driver to control the PF on the host. The guest would communicate with the PF on host using mailbox as usual.
>> Then the changes will be limited to DPDK, isn't it ?
>>
>> Regards
>> -Prashant
>>
>> -----Original Message-----
>> From: dev [mailto:dev-bounces-VfR2kkLFssw@public.gmane.org] On Behalf Of jigsaw
>> Sent: Wednesday, October 16, 2013 6:51 PM
>> To: Thomas Monjalon
>> Cc: dev-VfR2kkLFssw@public.gmane.org
>> Subject: Re: [dpdk-dev] 82599 SR-IOV with passthrough
>>
>> Hi Thomas,
>>
>> Thanks for reply.
>>
>> The kernel has older version of PF than the one released on sf.net. So I'm checking the sf.net release.
>> If the change is limited in DPDK then it is controllable. But now it affects Intel's PF driver, I don't even know how to push the feature to Intel. The driver on sf.net is a read-only repository, isn't it? It would be painful to maintain another branch of 10G PF driver.
>> Could Intel give some advice or hints here?
>>
>> thx &
>> rgds,
>> -Qinglai
>>
>> On Wed, Oct 16, 2013 at 3:58 PM, Thomas Monjalon <thomas.monjalon@6wind.com> wrote:
>>> 16/10/2013 14:18, jigsaw :
>>>> Therefore, to add support for multiple queues per VF, we have to at
>>>> least fix the PF driver, then add support in DPDK's VF driver.
>>>
>>> You're right, Linux PF driver have to be updated to properly manage
>>> multiple queues per VF. Then the guest can be tested with DPDK or
>>> with Linux driver (ixgbe_vf).
>>>
>>> Note that there are 2 versions of Linux driver for ixgbe: kernel.org
>>> and sourceforge.net (supporting many kernel versions).
>>>
>>> --
>>> Thomas
>>
>>
>>
>>
>> ======================================================================
>> ========= Please refer to
>> http://www.aricent.com/legal/email_disclaimer.html
>> for important disclosures regarding this electronic communication.
>> ======================================================================
>> =========
>
>
>
>
> ===============================================================================
> Please refer to http://www.aricent.com/legal/email_disclaimer.html
> for important disclosures regarding this electronic communication.
> ===============================================================================

Re: 82599 SR-IOV with passthrough

[Thomas Monjalon]

17/10/2013 14:43, jigsaw :
> I patched both Intel ixgbe PF driver and DPDK 1.5 VF driver, so that
> DPDK gets 4 queues in one VF. It works fine with all 4 Tx queues. The
> only trick is to set proper mac address for all outgoing packets,
> which must be the same mac as you set to the VF. This trick is
> described in the release note of DPDK.
> 
> I wonder whether it makes sense to push this patch to DPDK. Any comments?

Yes I think it makes sense.
Is it possible to get the number of configured queues via mailbox ?

-- 
Thomas

Re: 82599 SR-IOV with passthrough

[jigsaw]

>>Is it possible to get the number of configured queues via mailbox ?

Yes, this is exactly why I need to patch ixgbe PF. Otherwise it always
returns 1, even the number is configured as 4.
I will make some more experiments before sending the patch for review
in one or two days.

On Thu, Oct 17, 2013 at 4:47 PM, Thomas Monjalon
<thomas.monjalon-pdR9zngts4EAvxtiuMwx3w@public.gmane.org> wrote:
> 17/10/2013 14:43, jigsaw :
>> I patched both Intel ixgbe PF driver and DPDK 1.5 VF driver, so that
>> DPDK gets 4 queues in one VF. It works fine with all 4 Tx queues. The
>> only trick is to set proper mac address for all outgoing packets,
>> which must be the same mac as you set to the VF. This trick is
>> described in the release note of DPDK.
>>
>> I wonder whether it makes sense to push this patch to DPDK. Any comments?
>
> Yes I think it makes sense.
> Is it possible to get the number of configured queues via mailbox ?
>
> --
> Thomas

Re: 82599 SR-IOV with passthrough

[Prashant Upadhyaya]

Hi Qinglai,

Even with 1 queue, were you able to run the DPDK app in the guest OS ?
If you were able to, which version of DPDK did you use, please let me know.

I am trying to run the DPDK app in guest OS using QEMU/KVM with an SRIOV virtual function of an 82599 NIC.
I can see the vf pci address in the lspci output on the guest OS, but when I try to run the DPDK app in the guest OS, the EAL complains with the following --

EAL: pci_uio_map_resource(): cannot store uio mmap details

I am using DPDK1.4.

Regards
-Prashant


-----Original Message-----
From: dev [mailto:dev-bounces-VfR2kkLFssw@public.gmane.org] On Behalf Of jigsaw
Sent: Wednesday, October 16, 2013 5:49 PM
To: dev-VfR2kkLFssw@public.gmane.org
Subject: [dpdk-dev] 82599 SR-IOV with passthrough

Hi,

I am doing experiments with SR-IOV + passthrough on 82599.
My expectation is to have VT on and DCB off, under which configuration, the total 128 TX queues will be split into 32 pools, each has 4 queues.

With latest driver ixgbe-3.18.7, PF can be set with 16 pools, each has
4 queues with these params:

insmod ./ixgbe.ko MQ=1 max_vfs=8 RSS=8 VMDQ=16

I tried with VMDQ=32 with a panic. Also, it seems that if RSS is set to 4, the PF driver will set RSS to 2 somehow.
Since I'm fine with 16 pools + 4 queues,  I'm not going to investigate (at this moment) why PF doesn't work as expected.
Next step is then to try DPDK in guest OS, which get one VF by passthrough.

Not surprisingly, DPDK says that number of TX queue is 1. This is because the value is set arbitrarily in ixgbe_init_ops_vf of ixgbe_vf.c, and it never gets updated.

Actually the mbox API has support for requesting Tx/Rx queue numbers from VF. See implementation of routine ixgbevf_get_queues and ixgbevf_negotiate_api_version.
However, it is not straightforward to use these 2 routines to fetch Tx/Rx queue number, coz the PF driver is not ready to be used without modification.
See ixgbe_get_vf_queues of ixgbe_sriov.c in ixgbe-3.18.7.  The PF will always answer with 1 for Tx/Rx queue number requests, regardless of current config.

Therefore, to add support for multiple queues per VF, we have to at least fix the PF driver, then add support in DPDK's VF driver.

But the question is, is this enough? Before doing any experiments I wonder whether anybody has come across same problem as I do, and if there's any implementation ongoing.

thx &
rgds,
-Qinglai




===============================================================================
Please refer to http://www.aricent.com/legal/email_disclaimer.html
for important disclosures regarding this electronic communication.
===============================================================================