sr-iov related bonding regression (two regressions in one report)

sr-iov related bonding regression (two regressions in one report)

[Bagas Sanjaya]

Hi Thorsten and all,

I come across LACP bonding regression on Bugzilla [1]. The reporter
(Cc'ed) has two regressions. The first is actual LACP bonding
regression (but terse):

> Till linkx kernel 6.5.7 it is working fine, but after upgrading to 6.6.1 ping stop working with LACP bonding.
> When we disable SR-IOV from bios , everything working fine

And the second is out-of-tree module FTBFS:

> Also we are not able to compile 
> driver: ice
> version: 1.12.6
> We are getting following error.
> 
> 
> 
> With inter driver code(version 1.12.6) while complaining I am getting following error.
> /root/1.12.6/build/src/ice_txrx.h:363:29: error: field ‘xdp_rxq’ has incomplete type
>   363 |         struct xdp_rxq_info xdp_rxq;
>       |                             ^~~~~~~
> /root/1.12.6/build/src/ice_main.c: In function ‘ice_remove_recovery_mode’:
> /root/1.12.6/build/src/ice_main.c:1540:9: error: implicit declaration of function ‘pci_disable_pcie_error_reporting’ [-Werror=implicit-function-declaration]
>  1540 |         pci_disable_pcie_error_reporting(pf->pdev);
>       |         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> /root/1.12.6/build/src/ice_main.c: In function ‘ice_probe’:
> /root/1.12.6/build/src/ice_main.c:7046:9: error: implicit declaration of function ‘pci_enable_pcie_error_reporting’ [-Werror=implicit-function-declaration]
>  7046 |         pci_enable_pcie_error_reporting(pdev);
>       |         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> cc1: some warnings being treated as errors
> make[3]: *** [scripts/Makefile.build:243: /root/1.12.6/build/src/ice_main.o] Error 1
> make[2]: *** [/usr/src/linux-headers-6.6.0-rc7-vdx/Makefile:1913: /root/1.12.6/build/src] Error 2
> make[1]: *** [Makefile:234: __sub-make] Error 2
> make[1]: Leaving directory '/usr/src/linux-headers-6.6.0-rc7-vdx'
> make: *** [Makefile:174: all] Error 2

Should I add the first regression to regzbot (since the second one
is obviously out-of-tree problem), or should I asked detailed regression
info to the reporter?

Thanks.

[1]: https://bugzilla.kernel.org/show_bug.cgi?id=218139

-- 
An old man doll... just what I always wanted! - Clara

Re: sr-iov related bonding regression (two regressions in one report)

[Jay Vosburgh]

Bagas Sanjaya <bagasdotme@gmail.com> wrote:

>Hi Thorsten and all,
>
>I come across LACP bonding regression on Bugzilla [1]. The reporter
>(Cc'ed) has two regressions. The first is actual LACP bonding
>regression (but terse):
>
>> Till linkx kernel 6.5.7 it is working fine, but after upgrading to 6.6.1 ping stop working with LACP bonding.
>> When we disable SR-IOV from bios , everything working fine
>
>And the second is out-of-tree module FTBFS:

[... skip OOT stuff ...]

>
>Should I add the first regression to regzbot (since the second one
>is obviously out-of-tree problem), or should I asked detailed regression
>info to the reporter?

	My vote is to get additional information.  Given the nature of
the workaround ("When we disable SR-IOV from bios , everything working
fine"), it's plausible that the underlying cause is something
platform-specific.

	Interestingly, we've been chasing internally an issue with
bonding LACP mode on ice (E810-XXV, I think) when running on the Ubuntu
kernel.  That manifests as occasional TX timeouts, and doesn't happen if
the Intel OOT driver is used, so I wonder if that bugzilla reporter is
also seeing TX timeouts that correlate with their ping failures.

	-J

>Thanks.
>
>[1]: https://bugzilla.kernel.org/show_bug.cgi?id=218139
>
>-- 
>An old man doll... just what I always wanted! - Clara

---
	-Jay Vosburgh, jay.vosburgh@canonical.com

Re: sr-iov related bonding regression (two regressions in one report)

[Linux regression tracking (Thorsten Leemhuis)]

On 15.11.23 01:54, Jay Vosburgh wrote:
> Bagas Sanjaya <bagasdotme@gmail.com> wrote:
> 
>> I come across LACP bonding regression on Bugzilla [1].

Side note: Stephen forwards some (all?) network regressions to the right
people:
https://lore.kernel.org/all/20231113083746.5e02f8b0@hermes.local/

Would be best to check for that, no need to forward things twice, that
just results in a mess.

>> The reporter
>> (Cc'ed) has two regressions. The first is actual LACP bonding
>> regression (but terse):
>>
>>> Till linkx kernel 6.5.7 it is working fine, but after upgrading to 6.6.1 ping stop working with LACP bonding.
>>> When we disable SR-IOV from bios , everything working fine

Makes me wonder if things have been working with or without the OOT
module on 6.5.7, as strictly speaking it's only considered a kernel
regression if thing worked with a vanilla kernel (e.g. without OOT
modules) beforehand and broke when switching to a newer vanilla kernel.
If that's the case it would be okay to add to regzbot.

>> And the second is out-of-tree module FTBFS:
> [... skip OOT stuff ...]
> 
>> Should I add the first regression to regzbot (since the second one
>> is obviously out-of-tree problem), or should I asked detailed regression
>> info to the reporter?
> 
> 	My vote is to get additional information.  Given the nature of
> the workaround ("When we disable SR-IOV from bios , everything working
> fine"), it's plausible that the underlying cause is something
> platform-specific.

Maybe, but when it comes to the "no regressions" rule that likely makes
no difference from Linus perspective.

But I guess unless the intel folks or someone else has an idea what
might be wrong here we likely need a bisection (with vanilla kernels of
course) to get anywhere.

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
--
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
If I did something stupid, please tell me, as explained on that page.

Re: sr-iov related bonding regression (two regressions in one report)

[Anil Choudhary]

Its not hardware issue when I do rmmod iavf ping started working .
So issue is certainly in this kernel and with sriov only
Iavf id Nic driver for VF(sriovnic)


Thanks,
Anil

> On Nov 14, 2023, at 9:50 PM, Linux regression tracking (Thorsten Leemhuis) <regressions@leemhuis.info> wrote:
> 
> On 15.11.23 01:54, Jay Vosburgh wrote:
>> Bagas Sanjaya <bagasdotme@gmail.com> wrote:
>> 
>>> I come across LACP bonding regression on Bugzilla [1].
> 
> Side note: Stephen forwards some (all?) network regressions to the right
> people:
> https://lore.kernel.org/all/20231113083746.5e02f8b0@hermes.local/
> 
> Would be best to check for that, no need to forward things twice, that
> just results in a mess.
> 
>>> The reporter
>>> (Cc'ed) has two regressions. The first is actual LACP bonding
>>> regression (but terse):
>>> 
>>>> Till linkx kernel 6.5.7 it is working fine, but after upgrading to 6.6.1 ping stop working with LACP bonding.
>>>> When we disable SR-IOV from bios , everything working fine
> 
> Makes me wonder if things have been working with or without the OOT
> module on 6.5.7, as strictly speaking it's only considered a kernel
> regression if thing worked with a vanilla kernel (e.g. without OOT
> modules) beforehand and broke when switching to a newer vanilla kernel.
> If that's the case it would be okay to add to regzbot.
> 
>>> And the second is out-of-tree module FTBFS:
>> [... skip OOT stuff ...]
>> 
>>> Should I add the first regression to regzbot (since the second one
>>> is obviously out-of-tree problem), or should I asked detailed regression
>>> info to the reporter?
>> 
>> 	My vote is to get additional information.  Given the nature of
>> the workaround ("When we disable SR-IOV from bios , everything working
>> fine"), it's plausible that the underlying cause is something
>> platform-specific.
> 
> Maybe, but when it comes to the "no regressions" rule that likely makes
> no difference from Linus perspective.
> 
> But I guess unless the intel folks or someone else has an idea what
> might be wrong here we likely need a bisection (with vanilla kernels of
> course) to get anywhere.
> 
> Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
> --
> Everything you wanna know about Linux kernel regression tracking:
> https://linux-regtracking.leemhuis.info/about/#tldr
> If I did something stupid, please tell me, as explained on that page.

Re: sr-iov related bonding regression (two regressions in one report)

[Bagas Sanjaya]

[-- Attachment #1: Type: text/plain, Size: 552 bytes --]

On Tue, Nov 14, 2023 at 10:13:20PM -0800, Anil Choudhary wrote:
> Its not hardware issue when I do rmmod iavf ping started working .
> So issue is certainly in this kernel and with sriov only
> Iavf id Nic driver for VF(sriovnic)
> 
> 

Please don't top-post; reply inline with appropriate context instead.

So you have this regression on vanilla kernel, right? If so, please
bisect (see Documentation/admin-guide/bug-bisect.rst in the kernel
sources for reference).

Thanks.

-- 
An old man doll... just what I always wanted! - Clara

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: sr-iov related bonding regression (two regressions in one report)

[Anil Choudhary]

[-- Attachment #1: PastedGraphic-1.png --]
[-- Type: image/png, Size: 265036 bytes --]

[-- Attachment #2: Type: text/plain, Size: 2335 bytes --]




Following error error scribing to said is also new

> On Nov 14, 2023, at 9:50 PM, Linux regression tracking (Thorsten Leemhuis) <regressions@leemhuis.info> wrote:
> 
> On 15.11.23 01:54, Jay Vosburgh wrote:
>> Bagas Sanjaya <bagasdotme@gmail.com> wrote:
>> 
>>> I come across LACP bonding regression on Bugzilla [1].
> 
> Side note: Stephen forwards some (all?) network regressions to the right
> people:
> https://lore.kernel.org/all/20231113083746.5e02f8b0@hermes.local/
> 
> Would be best to check for that, no need to forward things twice, that
> just results in a mess.
> 
>>> The reporter
>>> (Cc'ed) has two regressions. The first is actual LACP bonding
>>> regression (but terse):
>>> 
>>>> Till linkx kernel 6.5.7 it is working fine, but after upgrading to 6.6.1 ping stop working with LACP bonding.
>>>> When we disable SR-IOV from bios , everything working fine
> 
> Makes me wonder if things have been working with or without the OOT
> module on 6.5.7, as strictly speaking it's only considered a kernel
> regression if thing worked with a vanilla kernel (e.g. without OOT
> modules) beforehand and broke when switching to a newer vanilla kernel.
> If that's the case it would be okay to add to regzbot.
> 
>>> And the second is out-of-tree module FTBFS:
>> [... skip OOT stuff ...]
>> 
>>> Should I add the first regression to regzbot (since the second one
>>> is obviously out-of-tree problem), or should I asked detailed regression
>>> info to the reporter?
>> 
>> 	My vote is to get additional information.  Given the nature of
>> the workaround ("When we disable SR-IOV from bios , everything working
>> fine"), it's plausible that the underlying cause is something
>> platform-specific.
> 
> Maybe, but when it comes to the "no regressions" rule that likely makes
> no difference from Linus perspective.
> 
> But I guess unless the intel folks or someone else has an idea what
> might be wrong here we likely need a bisection (with vanilla kernels of
> course) to get anywhere.
> 
> Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
> --
> Everything you wanna know about Linux kernel regression tracking:
> https://linux-regtracking.leemhuis.info/about/#tldr
> If I did something stupid, please tell me, as explained on that page.

Re: sr-iov related bonding regression (two regressions in one report)

[Bagas Sanjaya]

[-- Attachment #1: Type: text/plain, Size: 361 bytes --]

On Tue, Nov 14, 2023 at 10:19:25PM -0800, Anil Choudhary wrote:

> 
> 
> 
> Following error error scribing to said is also new
> 

Please don't top-post; reply inline with appropriate context instead.

What error? Can you reply with logs pasted (with error you mentioned)?

Confused...

-- 
An old man doll... just what I always wanted! - Clara

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: sr-iov related bonding regression (two regressions in one report)

[Anil Choudhary]

We are getting errorError subscribing to SWID 0x0000.
 from following code
root@us-ash-r1-c2-m1:~/linux# grep -rn -e "subscribing to " .
grep: ./debian/linux-image/lib/modules/6.6.1-vdx/kernel/drivers/net/ethernet/intel/ice/ice.ko: binary file matches
./samples/connector/ucon.c:149: ulog("subscribing to %u.%u\n", CN_TEST_IDX, CN_TEST_VAL);
./Documentation/driver-api/media/v4l2-event.rst:117:add      called when a new listener gets added (subscribing to the same
./Documentation/driver-api/media/v4l2-event.rst:130:Unsubscribing to an event is via:
./Documentation/maintainer/feature-and-driver-maintainers.rst:44:mailing list. Either by subscribing to the whole list or using more
grep: ./drivers/net/ethernet/intel/ice/ice_lag.o: binary file matches
grep: ./drivers/net/ethernet/intel/ice/ice.o: binary file matches
grep: ./drivers/net/ethernet/intel/ice/ice.ko: binary file matches
./drivers/net/ethernet/intel/ice/ice_lag.c:1007:                dev_err(ice_pf_to_dev(local_lag->pf), "Error subscribing to SWID 0x%04X\n",
root@us-ash-r1-c2-m1:~/linux#


Thanks,
Anil



> On Nov 14, 2023, at 10:19 PM, Anil Choudhary <anilchabba@gmail.com> wrote:
> 
> <PastedGraphic-1.png>
> 
> 
> Following error error scribing to said is also new
> 
>> On Nov 14, 2023, at 9:50 PM, Linux regression tracking (Thorsten Leemhuis) <regressions@leemhuis.info> wrote:
>> 
>> On 15.11.23 01:54, Jay Vosburgh wrote:
>>> Bagas Sanjaya <bagasdotme@gmail.com> wrote:
>>> 
>>>> I come across LACP bonding regression on Bugzilla [1].
>> 
>> Side note: Stephen forwards some (all?) network regressions to the right
>> people:
>> https://lore.kernel.org/all/20231113083746.5e02f8b0@hermes.local/
>> 
>> Would be best to check for that, no need to forward things twice, that
>> just results in a mess.
>> 
>>>> The reporter
>>>> (Cc'ed) has two regressions. The first is actual LACP bonding
>>>> regression (but terse):
>>>> 
>>>>> Till linkx kernel 6.5.7 it is working fine, but after upgrading to 6.6.1 ping stop working with LACP bonding.
>>>>> When we disable SR-IOV from bios , everything working fine
>> 
>> Makes me wonder if things have been working with or without the OOT
>> module on 6.5.7, as strictly speaking it's only considered a kernel
>> regression if thing worked with a vanilla kernel (e.g. without OOT
>> modules) beforehand and broke when switching to a newer vanilla kernel.
>> If that's the case it would be okay to add to regzbot.
>> 
>>>> And the second is out-of-tree module FTBFS:
>>> [... skip OOT stuff ...]
>>> 
>>>> Should I add the first regression to regzbot (since the second one
>>>> is obviously out-of-tree problem), or should I asked detailed regression
>>>> info to the reporter?
>>> 
>>> 	My vote is to get additional information.  Given the nature of
>>> the workaround ("When we disable SR-IOV from bios , everything working
>>> fine"), it's plausible that the underlying cause is something
>>> platform-specific.
>> 
>> Maybe, but when it comes to the "no regressions" rule that likely makes
>> no difference from Linus perspective.
>> 
>> But I guess unless the intel folks or someone else has an idea what
>> might be wrong here we likely need a bisection (with vanilla kernels of
>> course) to get anywhere.
>> 
>> Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
>> --
>> Everything you wanna know about Linux kernel regression tracking:
>> https://linux-regtracking.leemhuis.info/about/#tldr
>> If I did something stupid, please tell me, as explained on that page.
> 

Re: sr-iov related bonding regression (two regressions in one report)

[Bagas Sanjaya]

[-- Attachment #1: Type: text/plain, Size: 1593 bytes --]

On Wed, Nov 15, 2023 at 12:48:51PM -0800, Anil Choudhary wrote:
> We are getting errorError subscribing to SWID 0x0000.
>  from following code
> root@us-ash-r1-c2-m1:~/linux# grep -rn -e "subscribing to " .
> grep: ./debian/linux-image/lib/modules/6.6.1-vdx/kernel/drivers/net/ethernet/intel/ice/ice.ko: binary file matches
> ./samples/connector/ucon.c:149: ulog("subscribing to %u.%u\n", CN_TEST_IDX, CN_TEST_VAL);
> ./Documentation/driver-api/media/v4l2-event.rst:117:add      called when a new listener gets added (subscribing to the same
> ./Documentation/driver-api/media/v4l2-event.rst:130:Unsubscribing to an event is via:
> ./Documentation/maintainer/feature-and-driver-maintainers.rst:44:mailing list. Either by subscribing to the whole list or using more
> grep: ./drivers/net/ethernet/intel/ice/ice_lag.o: binary file matches
> grep: ./drivers/net/ethernet/intel/ice/ice.o: binary file matches
> grep: ./drivers/net/ethernet/intel/ice/ice.ko: binary file matches
> ./drivers/net/ethernet/intel/ice/ice_lag.c:1007:                dev_err(ice_pf_to_dev(local_lag->pf), "Error subscribing to SWID 0x%04X\n",
> root@us-ash-r1-c2-m1:~/linux#
> 

Again, please don't top-post; reply inline with appropriate context instead.
You may need to configure your email client to start reply below the quoted
context.

OK, now on your Bugzilla ticket, please attach the full log (either from
dmesg or from journalctl). And don't forget to perform bisection if
you'd like to get this regression fixed.

Thanks.

-- 
An old man doll... just what I always wanted! - Clara

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: sr-iov related bonding regression (two regressions in one report)

[Bagas Sanjaya]

[-- Attachment #1: Type: text/plain, Size: 2069 bytes --]

On Wed, Nov 15, 2023 at 06:50:26AM +0100, Linux regression tracking (Thorsten Leemhuis) wrote:
> On 15.11.23 01:54, Jay Vosburgh wrote:
> > Bagas Sanjaya <bagasdotme@gmail.com> wrote:
> > 
> >> I come across LACP bonding regression on Bugzilla [1].
> 
> Side note: Stephen forwards some (all?) network regressions to the right
> people:
> https://lore.kernel.org/all/20231113083746.5e02f8b0@hermes.local/
> 
> Would be best to check for that, no need to forward things twice, that
> just results in a mess.
> 
> >> The reporter
> >> (Cc'ed) has two regressions. The first is actual LACP bonding
> >> regression (but terse):
> >>
> >>> Till linkx kernel 6.5.7 it is working fine, but after upgrading to 6.6.1 ping stop working with LACP bonding.
> >>> When we disable SR-IOV from bios , everything working fine
> 
> Makes me wonder if things have been working with or without the OOT
> module on 6.5.7, as strictly speaking it's only considered a kernel
> regression if thing worked with a vanilla kernel (e.g. without OOT
> modules) beforehand and broke when switching to a newer vanilla kernel.
> If that's the case it would be okay to add to regzbot.
> 
> >> And the second is out-of-tree module FTBFS:
> > [... skip OOT stuff ...]
> > 
> >> Should I add the first regression to regzbot (since the second one
> >> is obviously out-of-tree problem), or should I asked detailed regression
> >> info to the reporter?
> > 
> > 	My vote is to get additional information.  Given the nature of
> > the workaround ("When we disable SR-IOV from bios , everything working
> > fine"), it's plausible that the underlying cause is something
> > platform-specific.
> 
> Maybe, but when it comes to the "no regressions" rule that likely makes
> no difference from Linus perspective.
> 
> But I guess unless the intel folks or someone else has an idea what
> might be wrong here we likely need a bisection (with vanilla kernels of
> course) to get anywhere.
> 

OK, thanks!

-- 
An old man doll... just what I always wanted! - Clara

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]