Re: vmxnet3 no longer functional on DPDK 21.11

[Lewis Donzis <lew@perftech.com>]

Good morning.

I just wanted to mention that this problem still persists in 22.11.3, and we still have to patch the vmxnet3 driver every time we upgrade.

As mentioned before, ixgbe_ethdev.c is an example of a driver that ifdef's out the attempt to use interrupts on FreeBSD.

Thanks,
lew


----- On Jun 3, 2022, at 8:19 AM, Lewis Donzis lew@perftech.com wrote:

> Hi, all.
> 
> Resurrecting this thread from six months ago, I apologize for not having more
> time to dig into it, but in light of recent findings, I see numerous other
> drivers and other parts of the code that have comments to the effect that
> "FreeBSD doesn't support interrupts" and they effectively #ifdef out the
> attempt.
> 
> Could this be as simple as needing to do the same in vmxnet3?  Empirically,
> ignoring the error from rte_intr_enable() allows the driver to work normally,
> for what that's worth.
> 
> Thanks,
> lew
> 
> ----- On Dec 6, 2021, at 6:08 AM, Konstantin Ananyev
> konstantin.ananyev@intel.com wrote:
> 
>>> -----Original Message-----
>>> From: Richardson, Bruce <bruce.richardson@intel.com>
>>> Sent: Monday, December 6, 2021 9:17 AM
>>> To: Lewis Donzis <lew@perftech.com>
>>> Cc: dev <dev@dpdk.org>; Wang, Yong <yongwang@vmware.com>; Ananyev, Konstantin
>>> <konstantin.ananyev@intel.com>
>>> Subject: Re: vmxnet3 no longer functional on DPDK 21.11
>>> 
>>> On Sun, Dec 05, 2021 at 07:52:33PM -0600, Lewis Donzis wrote:
>>> >
>>> >
>>> > ----- On Nov 30, 2021, at 7:42 AM, Bruce Richardson bruce.richardson@intel.com
>>> > wrote:
>>> >
>>> > > On Mon, Nov 29, 2021 at 02:45:15PM -0600, Lewis Donzis wrote:
>>> > >>    Hello.
>>> > >>    We just upgraded from 21.08 to 21.11 and it's rather astounding the
>>> > >>    number of incompatible changes in three months.  Not a big deal, just
>>> > >>    kind of a surprise, that's all.
>>> > >>    Anyway, the problem is that the vmxnet3 driver is no longer functional
>>> > >>    on FreeBSD.
>>> > >>    In drivers/net/vmxnet3/vmxnet3_ethdev.c, vmxnet3_dev_start() gets an
>>> > >>    error calling rte_intr_enable().  So it logs "interrupt enable failed"
>>> > >>    and returns an error.
>>> > >>    In lib/eal/freebsd/eal_interrupts.c, rte_intr_enable() is returning an
>>> > >>    error because rte_intr_dev_fd_get(intr_handle) is returning -1.
>>> > >>    I don't see how that could ever return anything other than -1 since it
>>> > >>    appears that there is no code that ever calls rte_intr_dev_fd_set()
>>> > >>    with a value other than -1 on FreeBSD.  Also weird to me is that even
>>> > >>    if it didn't get an error, the switch statement that follows looks like
>>> > >>    it will return an error in every case.
>>> > >>    Nonetheless, it worked in 21.08, and I can't quite see why the
>>> > >>    difference, so I must be missing something.
>>> > >>    For the moment, I just commented the "return -EIO" in vmxnet3_ethdev.c,
>>> > >>    and it's now working again, but that's obviously not the correct
>>> > >>    solution.
>>> > >>    Can someone who's knowledgable about this mechanism perhaps explain a
>>> > >>    little bit about what's going on?  I'll be happy to help troubleshoot.
>>> > >>    It seems like it must be something simple, but I just don't see it yet.
>>> > >
>>> > > Hi
>>> > >
>>> > > if you have the chance, it would be useful if you could use "git bisect" to
>>> > > identify the commit in 21.11 that broke this driver. Looking through the
>>> > > logs for 21.11 I can't identify any particular likely-looking commit, so
>>> > > bisect is likely a good way to start looking into this.
>>> > >
>>> > > Regards,
>>> > > /Bruce
>>> >
>>> > Hi, Bruce.  git bisect is very time-consuming and very cool!
>>> >
>>> > I went back to 21.08, about 1100 commits, and worked through the process, but
>>> > then I realized that I had forgotten to run ninja on one of
>>> the steps, so I did it again.
>>> >
>>> > I also re-checked it after the bisect, just to make sure that
>>> > c87d435a4d79739c0cec2ed280b94b41cb908af7 is good, and
>>> 7a0935239b9eb817c65c03554a9954ddb8ea5044 is bad.
>>> >
>>> > Thanks,
>>> > lew
>>> >
>>> 
>>> Many thanks for taking the time to do this. Adding Konstantin to thread as
>>> author of the commit you identified. Konstantin, any thoughts on this
>>> issue?
>> 
>> Hmm, that's looks really strange to me.
>> So to clarify, it fails at:
>> static int
>> vmxnet3_dev_start(struct rte_eth_dev *dev)
>> {
>>	...
>> line 1695:
>>	if (rte_intr_enable(dev->intr_handle) < 0) {
>>                PMD_INIT_LOG(ERR, "interrupt enable failed");
>>                return -EIO;
>>        }
>> 
>> Right?
>> 
>> The strange thing here is that 7a0935239b9e
>> doesn't change dev_start or rte_intr code in any way.
>> All it does - change rte_eth_rx_burst/rte_eth_tx_burst and other fast-path
>> functions.
>> Anyway, if git blames that commit, let's try to figure out what is going on.
>> Unfortunately, I don't have freebsd with vmxnet3, so will need to rely on your
>> help here.
>> As the first thing can you try to run testpmd build with last good commit
>> (c87d435a4d79)
>> and then testpmd build with bad commit applied and collect for both cases:
>> - contents of 'struct rte_eth_dev' and ' rte_eth_dev->intr_handle' for
>>  your vmxnet3 port
>> - debug log output (--log-level=eal,debug --log-level=pmd,debug)
>> 
> > Konstantin