Re: [Regression][BISECTED] kernel boot hang after 19898ce9cf8a ("wifi: iwlwifi: split 22000.c into multiple files")

Re: [Regression][BISECTED] kernel boot hang after 19898ce9cf8a ("wifi: iwlwifi: split 22000.c into multiple files")

[Linux regression tracking (Thorsten Leemhuis)]

[CCing the regression list, netdev, the net maintainers, and Linus;
Johannes and Kalle as well, but just for the record, they afaik are
unavailable]

Hi, Thorsten here, the Linux kernel's regression tracker.

On 07.07.23 10:25, Zhang, Rui wrote:
> 
> I run into a NULL pointer dereference and kernel boot hang after
> switching to latest upstream kernel, and git bisect shows that below
> commit is the first offending commit, and I have confirmed that commit
> 19898ce9cf8a has the issue while 19898ce9cf8a~1 does not.

FWIW, this is the fourth such report about this that I'm aware of.

The first is this one (with two affected users afaics):
https://bugzilla.kernel.org/show_bug.cgi?id=217622

The second is this one:
https://lore.kernel.org/all/CAAJw_Zug6VCS5ZqTWaFSr9sd85k%3DtyPm9DEE%2BmV%3DAKoECZM%2BsQ@mail.gmail.com/

The third:
https://lore.kernel.org/all/9274d9bd3d080a457649ff5addcc1726f08ef5b2.camel@xry111.site/

And in the past few days two people from Fedora land talked to me on IRC
with problems that in retrospective might be caused by this as well.

This many reports about a problem at this stage of the cycle makes me
suspect we'll see a lot more once -rc1 is out. That's why I raising the
awareness of this. Sadly a simple revert of just this commit is not
possible. :-/

Ciao, Thorsten

> commit 19898ce9cf8a33e0ac35cb4c7f68de297cc93cb2 (refs/bisect/bad)
> Author:     Johannes Berg <johannes.berg@intel.com>
> AuthorDate: Wed Jun 21 13:12:07 2023 +0300
> Commit:     Johannes Berg <johannes.berg@intel.com>
> CommitDate: Wed Jun 21 14:07:00 2023 +0200
> 
>     wifi: iwlwifi: split 22000.c into multiple files
>     
>     Split the configuration list in 22000.c into four new files,
>     per new device family, so we don't have this huge unusable
>     file. Yes, this duplicates a few small things, but that's
>     still much better than what we have now.
>     
>     Signed-off-by: Johannes Berg <johannes.berg@intel.com>
>     Signed-off-by: Gregory Greenman <gregory.greenman@intel.com>
>     Link:
> https://lore.kernel.org/r/20230621130443.7543603b2ee7.Ia8dd54216d341ef1ddc0531f2c9aa30d30536a5d@changeid
>     Signed-off-by: Johannes Berg <johannes.berg@intel.com>
> 
> I have some screenshots which show that RIP points to iwl_mem_free_skb,
> I can create a kernel bugzilla and attach the screenshots there if
> needed.
> 
> BTW, lspci output of the wifi device and git bisect log attached.
> 
> If any other information needed, please let me know.

--
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
That page also explains what to do if mails like this annoy you.

P.S.: for regzbot

#regzbot ^introduced 19898ce9cf8a
#regzbot dup-of:
https://lore.kernel.org/all/a5cdc7f8-b340-d372-2971-0d24b01de217@gmail.com/

Re: [Regression][BISECTED] kernel boot hang after 19898ce9cf8a ("wifi: iwlwifi: split 22000.c into multiple files")

[Thorsten Leemhuis]

On 07.07.23 12:55, Linux regression tracking (Thorsten Leemhuis) wrote:
> On 07.07.23 10:25, Zhang, Rui wrote:
>>
>> I run into a NULL pointer dereference and kernel boot hang after
>> switching to latest upstream kernel, and git bisect shows that below
>> commit is the first offending commit, and I have confirmed that commit
>> 19898ce9cf8a has the issue while 19898ce9cf8a~1 does not.
> 
> FWIW, this is the fourth such report about this that I'm aware of.
> 
> The first is this one (with two affected users afaics):
> https://bugzilla.kernel.org/show_bug.cgi?id=217622
> 
> The second is this one:
> https://lore.kernel.org/all/CAAJw_Zug6VCS5ZqTWaFSr9sd85k%3DtyPm9DEE%2BmV%3DAKoECZM%2BsQ@mail.gmail.com/
> 
> The third:
> https://lore.kernel.org/all/9274d9bd3d080a457649ff5addcc1726f08ef5b2.camel@xry111.site/
> 
> And in the past few days two people from Fedora land talked to me on IRC
> with problems that in retrospective might be caused by this as well.

I got confirmation: one of those cases is also caused by 19898ce9cf8a
But I write for a different reason:

Larry (now CCed) looked at the culprit and spotted something that looked
suspicious to him; he posted a patch and looks for testers:
https://lore.kernel.org/all/0068af47-e475-7e8d-e476-c374e90dff5f@lwfinger.net/

Ciao, Thorsten

> This many reports about a problem at this stage of the cycle makes me
> suspect we'll see a lot more once -rc1 is out. That's why I raising the
> awareness of this. Sadly a simple revert of just this commit is not
> possible. :-/
> 
> Ciao, Thorsten
> 
>> commit 19898ce9cf8a33e0ac35cb4c7f68de297cc93cb2 (refs/bisect/bad)
>> Author:     Johannes Berg <johannes.berg@intel.com>
>> AuthorDate: Wed Jun 21 13:12:07 2023 +0300
>> Commit:     Johannes Berg <johannes.berg@intel.com>
>> CommitDate: Wed Jun 21 14:07:00 2023 +0200
>>
>>     wifi: iwlwifi: split 22000.c into multiple files
>>     
>>     Split the configuration list in 22000.c into four new files,
>>     per new device family, so we don't have this huge unusable
>>     file. Yes, this duplicates a few small things, but that's
>>     still much better than what we have now.
>>     
>>     Signed-off-by: Johannes Berg <johannes.berg@intel.com>
>>     Signed-off-by: Gregory Greenman <gregory.greenman@intel.com>
>>     Link:
>> https://lore.kernel.org/r/20230621130443.7543603b2ee7.Ia8dd54216d341ef1ddc0531f2c9aa30d30536a5d@changeid
>>     Signed-off-by: Johannes Berg <johannes.berg@intel.com>
>>
>> I have some screenshots which show that RIP points to iwl_mem_free_skb,
>> I can create a kernel bugzilla and attach the screenshots there if
>> needed.
>>
>> BTW, lspci output of the wifi device and git bisect log attached.
>>
>> If any other information needed, please let me know.
> 
> --
> Everything you wanna know about Linux kernel regression tracking:
> https://linux-regtracking.leemhuis.info/about/#tldr
> That page also explains what to do if mails like this annoy you.
> 
> P.S.: for regzbot
> 
> #regzbot ^introduced 19898ce9cf8a
> #regzbot dup-of:
> https://lore.kernel.org/all/a5cdc7f8-b340-d372-2971-0d24b01de217@gmail.com/

Re: [Regression][BISECTED] kernel boot hang after 19898ce9cf8a ("wifi: iwlwifi: split 22000.c into multiple files")

[Zhang, Rui]

On Sat, 2023-07-08 at 16:17 +0200, Thorsten Leemhuis wrote:
> On 07.07.23 12:55, Linux regression tracking (Thorsten Leemhuis)
> wrote:
> > On 07.07.23 10:25, Zhang, Rui wrote:
> > > 
> > > I run into a NULL pointer dereference and kernel boot hang after
> > > switching to latest upstream kernel, and git bisect shows that
> > > below
> > > commit is the first offending commit, and I have confirmed that
> > > commit
> > > 19898ce9cf8a has the issue while 19898ce9cf8a~1 does not.
> > 
> > FWIW, this is the fourth such report about this that I'm aware of.
> > 
> > The first is this one (with two affected users afaics):
> > https://bugzilla.kernel.org/show_bug.cgi?id=217622
> > 
> > The second is this one:
> > https://lore.kernel.org/all/CAAJw_Zug6VCS5ZqTWaFSr9sd85k%3DtyPm9DEE%2BmV%3DAKoECZM%2BsQ@mail.gmail.com/
> > 
> > The third:
> > https://lore.kernel.org/all/9274d9bd3d080a457649ff5addcc1726f08ef5b2.camel@xry111.site/
> > 
> > And in the past few days two people from Fedora land talked to me
> > on IRC
> > with problems that in retrospective might be caused by this as
> > well.
> 
> I got confirmation: one of those cases is also caused by 19898ce9cf8a
> But I write for a different reason:
> 
> Larry (now CCed) looked at the culprit and spotted something that
> looked
> suspicious to him; he posted a patch and looks for testers:
> https://lore.kernel.org/all/0068af47-e475-7e8d-e476-c374e90dff5f@lwfinger.net/

I applied this patch but the problem still exists.

thanks,
rui
> 
> Ciao, Thorsten
> 
> > This many reports about a problem at this stage of the cycle makes
> > me
> > suspect we'll see a lot more once -rc1 is out. That's why I raising
> > the
> > awareness of this. Sadly a simple revert of just this commit is not
> > possible. :-/
> > 
> > Ciao, Thorsten
> > 
> > > commit 19898ce9cf8a33e0ac35cb4c7f68de297cc93cb2 (refs/bisect/bad)
> > > Author:     Johannes Berg <johannes.berg@intel.com>
> > > AuthorDate: Wed Jun 21 13:12:07 2023 +0300
> > > Commit:     Johannes Berg <johannes.berg@intel.com>
> > > CommitDate: Wed Jun 21 14:07:00 2023 +0200
> > > 
> > >     wifi: iwlwifi: split 22000.c into multiple files
> > >     
> > >     Split the configuration list in 22000.c into four new files,
> > >     per new device family, so we don't have this huge unusable
> > >     file. Yes, this duplicates a few small things, but that's
> > >     still much better than what we have now.
> > >     
> > >     Signed-off-by: Johannes Berg <johannes.berg@intel.com>
> > >     Signed-off-by: Gregory Greenman <gregory.greenman@intel.com>
> > >     Link:
> > > https://lore.kernel.org/r/20230621130443.7543603b2ee7.Ia8dd54216d341ef1ddc0531f2c9aa30d30536a5d@changeid
> > >     Signed-off-by: Johannes Berg <johannes.berg@intel.com>
> > > 
> > > I have some screenshots which show that RIP points to
> > > iwl_mem_free_skb,
> > > I can create a kernel bugzilla and attach the screenshots there
> > > if
> > > needed.
> > > 
> > > BTW, lspci output of the wifi device and git bisect log attached.
> > > 
> > > If any other information needed, please let me know.
> > 
> > --
> > Everything you wanna know about Linux kernel regression tracking:
> > https://linux-regtracking.leemhuis.info/about/#tldr
> > That page also explains what to do if mails like this annoy you.
> > 
> > P.S.: for regzbot
> > 
> > #regzbot ^introduced 19898ce9cf8a
> > #regzbot dup-of:
> > https://lore.kernel.org/all/a5cdc7f8-b340-d372-2971-0d24b01de217@gmail.com/

Re: [Regression][BISECTED] kernel boot hang after 19898ce9cf8a ("wifi: iwlwifi: split 22000.c into multiple files")

[Larry Finger]

On 7/9/23 08:27, Zhang, Rui wrote:
> On Sat, 2023-07-08 at 16:17 +0200, Thorsten Leemhuis wrote:
>> On 07.07.23 12:55, Linux regression tracking (Thorsten Leemhuis)
>> wrote:
>>> On 07.07.23 10:25, Zhang, Rui wrote:
>>>>
>>>> I run into a NULL pointer dereference and kernel boot hang after
>>>> switching to latest upstream kernel, and git bisect shows that
>>>> below
>>>> commit is the first offending commit, and I have confirmed that
>>>> commit
>>>> 19898ce9cf8a has the issue while 19898ce9cf8a~1 does not.
>>>
>>> FWIW, this is the fourth such report about this that I'm aware of.
>>>
>>> The first is this one (with two affected users afaics):
>>> https://bugzilla.kernel.org/show_bug.cgi?id=217622
>>>
>>> The second is this one:
>>> https://lore.kernel.org/all/CAAJw_Zug6VCS5ZqTWaFSr9sd85k%3DtyPm9DEE%2BmV%3DAKoECZM%2BsQ@mail.gmail.com/
>>>
>>> The third:
>>> https://lore.kernel.org/all/9274d9bd3d080a457649ff5addcc1726f08ef5b2.camel@xry111.site/
>>>
>>> And in the past few days two people from Fedora land talked to me
>>> on IRC
>>> with problems that in retrospective might be caused by this as
>>> well.
>>
>> I got confirmation: one of those cases is also caused by 19898ce9cf8a
>> But I write for a different reason:
>>
>> Larry (now CCed) looked at the culprit and spotted something that
>> looked
>> suspicious to him; he posted a patch and looks for testers:
>> https://lore.kernel.org/all/0068af47-e475-7e8d-e476-c374e90dff5f@lwfinger.net/
> 
> I applied this patch but the problem still exists.
> 
> thanks,
> rui

Rui,

I am not surprised that the patch did not help. I guess you will need to stay 
with kernel 6.3.X until the Intel developers return from their summer break.

Larry

Re: [Regression][BISECTED] kernel boot hang after 19898ce9cf8a ("wifi: iwlwifi: split 22000.c into multiple files")

[Linus Torvalds]

On Fri, 7 Jul 2023 at 03:55, Linux regression tracking (Thorsten
Leemhuis) <regressions@leemhuis.info> wrote:
>
> [CCing the regression list, netdev, the net maintainers, and Linus;
> Johannes and Kalle as well, but just for the record, they afaik are
> unavailable]

So I will release rc1 with this issue, but remind me - if it hasn't
had any traction next week and the radio silence continues, I'll just
revert it all.

From a quick look, "revert it all" ends up being

  fd006d60e833: "wifi: iwlwifi: remove support of A0 version of FM RF"
  a701177bd4bc: "wifi: iwlwifi: cfg: clean up Bz module firmware lines"
  f4daceae4087: "wifi: iwlwifi: pcie: add device id 51F1 for killer 1675"
  399762de769c: "wifi: iwlwifi: bump FW API to 83 for AX/BZ/SC devices"
  31aeae2446d5: "wifi: iwlwifi: cfg: remove trailing dash from FW_PRE constants"
  ecf11f4e4950: "wifi: iwlwifi: also unify Ma device configurations"
  bfed356b4fc4: "wifi: iwlwifi: also unify Sc device configurations"
  3fd31289d5de: "wifi: iwlwifi: unify Bz/Gl device configurations"
  e3597e28a2fa: "wifi: iwlwifi: pcie: also drop jacket from info macro"
  0f21d7d56083: "wifi: iwlwifi: remove support for *nJ devices"
  c648e926d021: "wifi: iwlwifi: don't load old firmware for 22000"
  a7de384c9399: "wifi: iwlwifi: don't load old firmware for ax210"
  a13707f7c845: "wifi: iwlwifi: don't load old firmware for Bz"
  508b4a1baeb3: "wifi: iwlwifi: don't load old firmware for Sc"
  5afe98b2e299: "wifi: iwlwifi: give Sc devices their own family"
  19898ce9cf8a: "wifi: iwlwifi: split 22000.c into multiple files"

since clearly nothing seems to be happening on this front, and summer
vacations are only going to get worse.

But we'll give it another week. In August huge chunks of Europe will
go on vacation.

                    Linus

Re: [Regression][BISECTED] kernel boot hang after 19898ce9cf8a ("wifi: iwlwifi: split 22000.c into multiple files")

[Johannes Berg]

On Sun, 2023-07-09 at 09:31 -0700, Linus Torvalds wrote:
> On Fri, 7 Jul 2023 at 03:55, Linux regression tracking (Thorsten
> Leemhuis) <regressions@leemhuis.info> wrote:
> > 
> > [CCing the regression list, netdev, the net maintainers, and Linus;
> > Johannes and Kalle as well, but just for the record, they afaik are
> > unavailable]
> 
> So I will release rc1 with this issue, but remind me - if it hasn't
> had any traction next week and the radio silence continues, I'll just
> revert it all.

Sorry. I got back home a few hours ago (for few days anyway) and I think
I already know what the issue is. I'll send a fix to try in a few
minutes, was just trying to collect all the reported-by etc.

There's clearly a separate bug in the init failure path, but the reason
it fails in the first place is a mismatch between different changes.

johannes

Re: [Regression][BISECTED] kernel boot hang after 19898ce9cf8a ("wifi: iwlwifi: split 22000.c into multiple files")

[Zhang, Rui]

On Sun, 2023-07-09 at 20:07 +0200, Johannes Berg wrote:
> On Sun, 2023-07-09 at 09:31 -0700, Linus Torvalds wrote:
> > On Fri, 7 Jul 2023 at 03:55, Linux regression tracking (Thorsten
> > Leemhuis) <regressions@leemhuis.info> wrote:
> > > 
> > > [CCing the regression list, netdev, the net maintainers, and
> > > Linus;
> > > Johannes and Kalle as well, but just for the record, they afaik
> > > are
> > > unavailable]
> > 
> > So I will release rc1 with this issue, but remind me - if it hasn't
> > had any traction next week and the radio silence continues, I'll
> > just
> > revert it all.
> 
> Sorry. I got back home a few hours ago (for few days anyway) and I
> think
> I already know what the issue is. I'll send a fix to try in a few
> minutes, was just trying to collect all the reported-by etc.
> 
> There's clearly a separate bug in the init failure path, but the
> reason
> it fails in the first place is a mismatch between different changes.
> 
> 
I have tested Johannes' patch and it fixes the problem on my side.

Thanks,
rui

Re: [Regression][BISECTED] kernel boot hang after 19898ce9cf8a ("wifi: iwlwifi: split 22000.c into multiple files")

[Linux regression tracking (Thorsten Leemhuis)]

[CCing one

On 10.07.23 03:54, Zhang, Rui wrote:
> On Sun, 2023-07-09 at 20:07 +0200, Johannes Berg wrote:
>> On Sun, 2023-07-09 at 09:31 -0700, Linus Torvalds wrote:
>>> On Fri, 7 Jul 2023 at 03:55, Linux regression tracking (Thorsten
>>> Leemhuis) <regressions@leemhuis.info> wrote:
>>>>
>>>> [CCing the regression list, netdev, the net maintainers, and
>>>> Linus;
>>>> Johannes and Kalle as well, but just for the record, they afaik
>>>> are
>>>> unavailable]
>>>
>>> So I will release rc1 with this issue, but remind me - if it hasn't
>>> had any traction next week and the radio silence continues, I'll
>>> just
>>> revert it all.
>>
>> Sorry. I got back home a few hours ago (for few days anyway) and I
>> think
>> I already know what the issue is. 

Many many thx for taking a look at this!

FWIW in case anyone wonders where that fix is, find it here:
https://lore.kernel.org/all/20230709181323.12085-2-johannes@sipsolutions.net/

(or here: https://bugzilla.kernel.org/show_bug.cgi?id=217622#c9 )

> I have tested Johannes' patch and it fixes the problem on my side.

Ahh, great.

Ciao, Thorsten