[regression] Kernel OOPS on boot with Kernel 6.3(.1) and RTL8153 Gigabit Ethernet Adapter

[regression] Kernel OOPS on boot with Kernel 6.3(.1) and RTL8153 Gigabit Ethernet Adapter

[Linux regression tracking (Thorsten Leemhuis)]

Hi, Thorsten here, the Linux kernel's regression tracker.

I noticed a regression report in bugzilla.kernel.org. As many (most?)
kernel developers don't keep an eye on it, I decided to forward it by mail.

Note, you have to use bugzilla to reach the reporter, as I sadly[1] can
not CCed them in mails like this.

Quoting from https://bugzilla.kernel.org/show_bug.cgi?id=217399 :

>  Bernd Buschinski 2023-05-04 11:11:24 UTC
> 
> Created attachment 304215 [details]
> Kernel OOPS on boot
> 
> Hello,
> 
> on my laptop with kernel 6.3.0 and 6.3.1 fails to correctly boot if the usb-c device "RTL8153 Gigabit Ethernet Adapter" is connected.
> 
> If I unplug it, boot and the plug it in, everything works fine.
> 
> This used to work fine with 6.2.10.
> 
> HW:
> - Dell Inc. Latitude 7410/0M5G57, BIOS 1.22.0 03/20/2023
> - Realtek Semiconductor Corp. RTL8153 Gigabit Ethernet Adapter
> 
> 
> Call Trace (manually typed from the image, typos maybe be included)
> - bpf_dev_bound_netdeuv _unregister
> - unregister_netdevice_many_notify
> - unregister_netdevice_gueue
> - unregister_netdev
> - usbnet_disconnect
> - usb_unbind_interface
> - device_release_driver_internal
> - bus_remove_device
> - device_del
> - ? kobject_put
> - usb_disable_device
> - usb_set_configuration
> - rt18152_cfgselector_probe
> - usb_probe_device
> - really_probe
> - ? driver_probe_device
> - ...
> 
> 
> For the full output, please see the attached image.
> 
> [tag] [reply] [−]
> Private
> Comment 1 Bernd Buschinski 2023-05-04 11:49:33 UTC
> 
> lsusb:
> Bus 004 Device 003: ID 0bda:8153 Realtek Semiconductor Corp. RTL8153 Gigabit Ethernet Adapter
> 
> 
> and yes, I made a typo in the call trace, it is `rtl8152_cfgselector_probe`
> 
> [tag] [reply] [−]
> Private
> Comment 2 Bernd Buschinski 2023-05-04 11:51:44 UTC
> 
> Created attachment 304216 [details]
> lsusb -v, for the the device
> 
> [tag] [reply] [−]
> Private
> Comment 3 Bernd Buschinski 2023-05-04 11:58:28 UTC
> 
> I did not bisect it, but I think 
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=v6.3.1&id=ec51fbd1b8a2bca2948dede99c14ec63dc57ff6b
> 
> is at least related, maybe.... feel free to correct me.


See the ticket for more details.


[TLDR for the rest of this mail: I'm adding this report to the list of
tracked Linux kernel regressions; the text you find below is based on a
few templates paragraphs you might have encountered already in similar
form.]

BTW, let me use this mail to also add the report to the list of tracked
regressions to ensure it's doesn't fall through the cracks:

#regzbot introduced: v6.2..v6.3
https://bugzilla.kernel.org/show_bug.cgi?id=217399
#regzbot title: net/usb: RTL8153 Gigabit Ethernet Adapter stopped working
#regzbot ignore-activity

This isn't a regression? This issue or a fix for it are already
discussed somewhere else? It was fixed already? You want to clarify when
the regression started to happen? Or point out I got the title or
something else totally wrong? Then just reply and tell me -- ideally
while also telling regzbot about it, as explained by the page listed in
the footer of this mail.

Developers: When fixing the issue, remember to add 'Link:' tags pointing
to the report (e.g. the buzgzilla ticket and maybe this mail as well, if
this thread sees some discussion). See page linked in footer for details.

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
--
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
If I did something stupid, please tell me, as explained on that page.

[1] because bugzilla.kernel.org tells users upon registration their
"email address will never be displayed to logged out users"

Re: [regression] Kernel OOPS on boot with Kernel 6.3(.1) and RTL8153 Gigabit Ethernet Adapter

[Bjørn Mork]

"Linux regression tracking (Thorsten Leemhuis)"
<regressions@leemhuis.info> writes:

>> Kernel OOPS on boot
>> 
>> Hello,
>> 
>> on my laptop with kernel 6.3.0 and 6.3.1 fails to correctly boot if the usb-c device "RTL8153 Gigabit Ethernet Adapter" is connected.
>> 
>> If I unplug it, boot and the plug it in, everything works fine.
>> 
>> This used to work fine with 6.2.10.
>> 
>> HW:
>> - Dell Inc. Latitude 7410/0M5G57, BIOS 1.22.0 03/20/2023
>> - Realtek Semiconductor Corp. RTL8153 Gigabit Ethernet Adapter
>> 
>> 
>> Call Trace (manually typed from the image, typos maybe be included)
>> - bpf_dev_bound_netdev_unregister
>> - unregister_netdevice_many_notify
>> - unregister_netdevice_gueue
>> - unregister_netdev
>> - usbnet_disconnect
>> - usb_unbind_interface
>> - device_release_driver_internal
>> - bus_remove_device
>> - device_del
>> - ? kobject_put
>> - usb_disable_device
>> - usb_set_configuration
>> - rt18152_cfgselector_probe
>> - usb_probe_device
>> - really_probe
>> - ? driver_probe_device
>> - ...


Ouch. This is obviously related to the change I made to the RTL8153
driver, which you can see is in effect by the call to
rtl8152_cfgselector_probe above (compensating for the typo).

But to me it doesn't look like the bug is in that driver. It seems we
are triggering some latent bug in the unregister_netdev code?

The trace looks precise enogh to me.  The image also shows

 RIP: 0010: __rhastable_lookup.constprop.0+0x18/0x120

which I believe comes from bpf_dev_bound_netdev_unregister() calling the
bpf_offload_find_netdev(), which does:


bpf_offload_find_netdev(struct net_device *netdev)
{
        lockdep_assert_held(&bpf_devs_lock);

        return rhashtable_lookup_fast(&offdevs, &netdev, offdevs_params);
}


Maybe someone familiar with that code can explain why this fails if called
at boot instead of later?

AFAICS, we don't do anything out of the ordinary in that driver, with
respect to netdev registration at least.  A similar device disconnet and
netdev unregister would also happen if you decided to pull the USB
device from the port during boot.  In fact, most USB network devices
behave similar when disconnected and there is nothing preventing it
from happening while the system is booting..



Bjørn

Re: [regression] Kernel OOPS on boot with Kernel 6.3(.1) and RTL8153 Gigabit Ethernet Adapter

[Jakub Kicinski]

On Fri, 05 May 2023 12:16:48 +0200 Bjørn Mork wrote:
> "Linux regression tracking (Thorsten Leemhuis)"
> <regressions@leemhuis.info> writes:
> 
> >> Kernel OOPS on boot
> >> 
> >> Hello,
> >> 
> >> on my laptop with kernel 6.3.0 and 6.3.1 fails to correctly boot if the usb-c device "RTL8153 Gigabit Ethernet Adapter" is connected.
> >> 
> >> If I unplug it, boot and the plug it in, everything works fine.
> >> 
> >> This used to work fine with 6.2.10.
> >> 
> >> HW:
> >> - Dell Inc. Latitude 7410/0M5G57, BIOS 1.22.0 03/20/2023
> >> - Realtek Semiconductor Corp. RTL8153 Gigabit Ethernet Adapter
> >> 
> >> 
> >> Call Trace (manually typed from the image, typos maybe be included)
> >> - bpf_dev_bound_netdev_unregister
> >> - unregister_netdevice_many_notify
> >> - unregister_netdevice_gueue
> >> - unregister_netdev
> >> - usbnet_disconnect
> >> - usb_unbind_interface
> >> - device_release_driver_internal
> >> - bus_remove_device
> >> - device_del
> >> - ? kobject_put
> >> - usb_disable_device
> >> - usb_set_configuration
> >> - rt18152_cfgselector_probe
> >> - usb_probe_device
> >> - really_probe
> >> - ? driver_probe_device
> >> - ...  
> 
> 
> Ouch. This is obviously related to the change I made to the RTL8153
> driver, which you can see is in effect by the call to
> rtl8152_cfgselector_probe above (compensating for the typo).
> 
> But to me it doesn't look like the bug is in that driver. It seems we
> are triggering some latent bug in the unregister_netdev code?
> 
> The trace looks precise enogh to me.  The image also shows
> 
>  RIP: 0010: __rhastable_lookup.constprop.0+0x18/0x120
> 
> which I believe comes from bpf_dev_bound_netdev_unregister() calling the
> bpf_offload_find_netdev(), which does:
> 
> 
> bpf_offload_find_netdev(struct net_device *netdev)
> {
>         lockdep_assert_held(&bpf_devs_lock);
> 
>         return rhashtable_lookup_fast(&offdevs, &netdev, offdevs_params);
> }
> 
> 
> Maybe someone familiar with that code can explain why this fails if called
> at boot instead of later?
> 
> AFAICS, we don't do anything out of the ordinary in that driver, with
> respect to netdev registration at least.  A similar device disconnet and
> netdev unregister would also happen if you decided to pull the USB
> device from the port during boot.  In fact, most USB network devices
> behave similar when disconnected and there is nothing preventing it
> from happening while the system is booting..

Yeah, I think it's because late_initcall is too conservative. 
The device gets removed before late_initcall(). 

It's just a hashtable init, I think that we can do:

diff --git a/kernel/bpf/offload.c b/kernel/bpf/offload.c
index d9c9f45e3529..8a26cd8814c1 100644
--- a/kernel/bpf/offload.c
+++ b/kernel/bpf/offload.c
@@ -859,4 +859,4 @@ static int __init bpf_offload_init(void)
 	return rhashtable_init(&offdevs, &offdevs_params);
 }
 
-late_initcall(bpf_offload_init);
+core_initcall(bpf_offload_init);


Thorsten, how is the communication supposed to work in this case?
Can you ask the reporter to test this? I don't see them on CC...

Re: [regression] Kernel OOPS on boot with Kernel 6.3(.1) and RTL8153 Gigabit Ethernet Adapter

[Linux regression tracking (Thorsten Leemhuis)]

On 05.05.23 21:04, Jakub Kicinski wrote:
> On Fri, 05 May 2023 12:16:48 +0200 Bjørn Mork wrote:
>> "Linux regression tracking (Thorsten Leemhuis)"
>> <regressions@leemhuis.info> writes:
>>
>>>> Kernel OOPS on boot
>>>>
>>>> Hello,
>>>>
>>>> on my laptop with kernel 6.3.0 and 6.3.1 fails to correctly boot if the usb-c device "RTL8153 Gigabit Ethernet Adapter" is connected.
>>>>
>>>> If I unplug it, boot and the plug it in, everything works fine.
>>>>
>>>> This used to work fine with 6.2.10.
>>>>
>>>> HW:
>>>> - Dell Inc. Latitude 7410/0M5G57, BIOS 1.22.0 03/20/2023
>>>> - Realtek Semiconductor Corp. RTL8153 Gigabit Ethernet Adapter
>>>>
>>>>
>>>> Call Trace (manually typed from the image, typos maybe be included)
>>>> - bpf_dev_bound_netdev_unregister
>>>> - unregister_netdevice_many_notify
>>>> - unregister_netdevice_gueue
>>>> - unregister_netdev
>>>> - usbnet_disconnect
>>>> - usb_unbind_interface
>>>> - device_release_driver_internal
>>>> - bus_remove_device
>>>> - device_del
>>>> - ? kobject_put
>>>> - usb_disable_device
>>>> - usb_set_configuration
>>>> - rt18152_cfgselector_probe
>>>> - usb_probe_device
>>>> - really_probe
>>>> - ? driver_probe_device
>>>> - ...  
>>
>>
>> Ouch. This is obviously related to the change I made to the RTL8153
>> driver, which you can see is in effect by the call to
>> rtl8152_cfgselector_probe above (compensating for the typo).
>>
>> But to me it doesn't look like the bug is in that driver. It seems we
>> are triggering some latent bug in the unregister_netdev code?
>>
>> The trace looks precise enogh to me.  The image also shows
>>
>>  RIP: 0010: __rhastable_lookup.constprop.0+0x18/0x120
>>
>> which I believe comes from bpf_dev_bound_netdev_unregister() calling the
>> bpf_offload_find_netdev(), which does:
>>
>>
>> bpf_offload_find_netdev(struct net_device *netdev)
>> {
>>         lockdep_assert_held(&bpf_devs_lock);
>>
>>         return rhashtable_lookup_fast(&offdevs, &netdev, offdevs_params);
>> }
>>
>>
>> Maybe someone familiar with that code can explain why this fails if called
>> at boot instead of later?
>>
>> AFAICS, we don't do anything out of the ordinary in that driver, with
>> respect to netdev registration at least.  A similar device disconnet and
>> netdev unregister would also happen if you decided to pull the USB
>> device from the port during boot.  In fact, most USB network devices
>> behave similar when disconnected and there is nothing preventing it
>> from happening while the system is booting..
> 
> Yeah, I think it's because late_initcall is too conservative. 
> The device gets removed before late_initcall(). 
> 
> It's just a hashtable init, I think that we can do:
> 
> diff --git a/kernel/bpf/offload.c b/kernel/bpf/offload.c
> index d9c9f45e3529..8a26cd8814c1 100644
> --- a/kernel/bpf/offload.c
> +++ b/kernel/bpf/offload.c
> @@ -859,4 +859,4 @@ static int __init bpf_offload_init(void)
>  	return rhashtable_init(&offdevs, &offdevs_params);
>  }
>  
> -late_initcall(bpf_offload_init);
> +core_initcall(bpf_offload_init);
> 
> 
> Thorsten, how is the communication supposed to work in this case?
> Can you ask the reporter to test this?

I'd prefer to not become the man-in-the middle, that just delays things
and is fragile; but I can do that occasionally if developers really are
unwilling to go to bugzilla.

Bugbot afaics will solve this, but using it right now would require me
to do something some maintainers don't like. See this sub-thread:

https://lore.kernel.org/all/1f0ebf13-ab0f-d512-6106-3ebf7cb372f1@leemhuis.info/


:-/

This got me thinking: we could tell bugbot to start monitoring this
thread and then tell the reporter to CC to the new bug bugbot created.
Ugly, but might work.

> I don't see them on CC...

Yeah, as stated in the initial mail of this thread: I sadly can't CC
them, because bugzilla.kernel.org tells users upon registration their
"email address will never be displayed to logged out users"... #sigh

I wish things were different, I'm unhappy about this situation myself.

Ciao, Thorsten

Re: [regression] Kernel OOPS on boot with Kernel 6.3(.1) and RTL8153 Gigabit Ethernet Adapter

[Jakub Kicinski]

On Fri, 5 May 2023 21:20:15 +0200 Linux regression tracking (Thorsten
Leemhuis) wrote:
> > Thorsten, how is the communication supposed to work in this case?
> > Can you ask the reporter to test this?  
> 
> I'd prefer to not become the man-in-the middle, that just delays things
> and is fragile; but I can do that occasionally if developers really are
> unwilling to go to bugzilla.
> 
> Bugbot afaics will solve this, but using it right now would require me
> to do something some maintainers don't like. See this sub-thread:
> 
> https://lore.kernel.org/all/1f0ebf13-ab0f-d512-6106-3ebf7cb372f1@leemhuis.info/
> 
> :-/
> 
> This got me thinking: we could tell bugbot to start monitoring this
> thread and then tell the reporter to CC to the new bug bugbot created.
> Ugly, but might work.
> 
> > I don't see them on CC...  
> 
> Yeah, as stated in the initial mail of this thread: I sadly can't CC
> them, because bugzilla.kernel.org tells users upon registration their
> "email address will never be displayed to logged out users"... #sigh
> 
> I wish things were different, I'm unhappy about this situation myself.

Let's work something out, because forwarding enough info for Bjorn to
respond on the list means that we now have the conversation going in
both places. So it's confusing & double the work for developers.

I don't seem to have the permissions on BZ, but I'm guessing we could
do the opposite - you could flip bugbot on first to have it flush the BZ
report to the list, and then reply on the list with regzbot tracking?

Re: [regression] Kernel OOPS on boot with Kernel 6.3(.1) and RTL8153 Gigabit Ethernet Adapter

[Linux regression tracking (Thorsten Leemhuis)]

On 05.05.23 21:37, Jakub Kicinski wrote:
> On Fri, 5 May 2023 21:20:15 +0200 Linux regression tracking (Thorsten
> Leemhuis) wrote:
>>> Thorsten, how is the communication supposed to work in this case?
>>> Can you ask the reporter to test this?  
>>
>> I'd prefer to not become the man-in-the middle, that just delays things
>> and is fragile; but I can do that occasionally if developers really are
>> unwilling to go to bugzilla.
>>
>> Bugbot afaics will solve this, but using it right now would require me
>> to do something some maintainers don't like. See this sub-thread:
>>
>> https://lore.kernel.org/all/1f0ebf13-ab0f-d512-6106-3ebf7cb372f1@leemhuis.info/
>>
>> :-/
>>
>> This got me thinking: we could tell bugbot to start monitoring this
>> thread and then tell the reporter to CC to the new bug bugbot created.
>> Ugly, but might work.
>>
>>> I don't see them on CC...  
>>
>> Yeah, as stated in the initial mail of this thread: I sadly can't CC
>> them, because bugzilla.kernel.org tells users upon registration their
>> "email address will never be displayed to logged out users"... #sigh
>>
>> I wish things were different, I'm unhappy about this situation myself.
> 
> Let's work something out, because forwarding enough info for Bjorn to
> respond on the list means that we now have the conversation going in
> both places. So it's confusing & double the work for developers.

I know, I know, but I saw no other way.

> I don't seem to have the permissions on BZ, but I'm guessing we could
> do the opposite - you could flip bugbot on first to have it flush the BZ
> report to the list, and then reply on the list with regzbot tracking?

That's the plan for the future, but for now I don't want to do that, as
it might mess up other peoples workflows, as hinted above already and
discussed here:

https://lore.kernel.org/all/1f0ebf13-ab0f-d512-6106-3ebf7cb372f1@leemhuis.info/

That was only recently, but if you jump in there as well it might
persuade Konstantin to enable bugbot for other products/components. Then
I could and would do what you suggested.

Ciao, Thorsten

Re: [regression] Kernel OOPS on boot with Kernel 6.3(.1) and RTL8153 Gigabit Ethernet Adapter

[Jakub Kicinski]

On Sat, 6 May 2023 08:20:23 +0200 Linux regression tracking (Thorsten
Leemhuis) wrote:
> > I don't seem to have the permissions on BZ, but I'm guessing we could
> > do the opposite - you could flip bugbot on first to have it flush the BZ
> > report to the list, and then reply on the list with regzbot tracking?  
> 
> That's the plan for the future, but for now I don't want to do that, as
> it might mess up other peoples workflows, as hinted above already and
> discussed here:
> 
> https://lore.kernel.org/all/1f0ebf13-ab0f-d512-6106-3ebf7cb372f1@leemhuis.info/
> 
> That was only recently, but if you jump in there as well it might
> persuade Konstantin to enable bugbot for other products/components. Then
> I could and would do what you suggested.

CC: workflows

I'm a bit confused. I understand that we don't want to automatically
send all bugzilla reports to the ML. But AFAIU this is to avoid spamming
the list / messing with people's existing BZ workflow.
If you pre-triage the problem and decide to forward it to the list -
whether you do it with buzbot + regzbot or manual + regzbot is moot.

The bugbot can be enabled per BZ entry (AFAIU), so you can flip it
individually for the thread you want to report. It should flush that 
BZ to the list. At which point you can follow your normal ML regression
process.

Where did I go off the rails?

Re: [regression] Kernel OOPS on boot with Kernel 6.3(.1) and RTL8153 Gigabit Ethernet Adapter

[Thorsten Leemhuis]

On 08.05.23 22:09, Jakub Kicinski wrote:
> On Sat, 6 May 2023 08:20:23 +0200 Linux regression tracking (Thorsten
> Leemhuis) wrote:
>>> I don't seem to have the permissions on BZ, but I'm guessing we could
>>> do the opposite - you could flip bugbot on first to have it flush the BZ
>>> report to the list, and then reply on the list with regzbot tracking?  
>>
>> That's the plan for the future, but for now I don't want to do that, as
>> it might mess up other peoples workflows, as hinted above already and
>> discussed here:
>>
>> https://lore.kernel.org/all/1f0ebf13-ab0f-d512-6106-3ebf7cb372f1@leemhuis.info/
>>
>> That was only recently, but if you jump in there as well it might
>> persuade Konstantin to enable bugbot for other products/components. Then
>> I could and would do what you suggested.
> 
> CC: workflows
> 
> I'm a bit confused. I understand that we don't want to automatically
> send all bugzilla reports to the ML. But AFAIU this is to avoid spamming
> the list / messing with people's existing BZ workflow.
> If you pre-triage the problem and decide to forward it to the list -
> whether you do it with buzbot + regzbot or manual + regzbot is moot.
> 
> The bugbot can be enabled per BZ entry (AFAIU), so you can flip it
> individually for the thread you want to report. It should flush that 
> BZ to the list. At which point you can follow your normal ML regression
> process.
> 
> Where did I go off the rails?

You missed that Konstantin (now CCed) is just a bit careful for the
bugbot bring up and therefore for now only allows bugbot to be enabled
for BZ entries that are filed against the product/component combination
Linux/Kernel. I could reassigning bugs there, but that would break the
workflow for maintainers like Kalle, which look at all bugs assigned to
their product/component combo (Drivers/network-wireless in Kalle's case).

Ciao, Thorsten

Re: [regression] Kernel OOPS on boot with Kernel 6.3(.1) and RTL8153 Gigabit Ethernet Adapter

[Konstantin Ryabitsev]

On Thu, May 11, 2023 at 03:25:47PM +0200, Thorsten Leemhuis wrote:
> > The bugbot can be enabled per BZ entry (AFAIU), so you can flip it
> > individually for the thread you want to report. It should flush that 
> > BZ to the list. At which point you can follow your normal ML regression
> > process.
> > 
> > Where did I go off the rails?
> 
> You missed that Konstantin (now CCed) is just a bit careful for the
> bugbot bring up and therefore for now only allows bugbot to be enabled
> for BZ entries that are filed against the product/component combination
> Linux/Kernel. I could reassigning bugs there, but that would break the
> workflow for maintainers like Kalle, which look at all bugs assigned to
> their product/component combo (Drivers/network-wireless in Kalle's case).

I hope to start opening this up to other products within the next few weeks,
as things are looking fairly stable for our initial tests.

-K

Re: [regression] Kernel OOPS on boot with Kernel 6.3(.1) and RTL8153 Gigabit Ethernet Adapter

[Bjørn Mork]

Jakub Kicinski <kuba@kernel.org> writes:

> It's just a hashtable init, I think that we can do:
>
> diff --git a/kernel/bpf/offload.c b/kernel/bpf/offload.c
> index d9c9f45e3529..8a26cd8814c1 100644
> --- a/kernel/bpf/offload.c
> +++ b/kernel/bpf/offload.c
> @@ -859,4 +859,4 @@ static int __init bpf_offload_init(void)
>  	return rhashtable_init(&offdevs, &offdevs_params);
>  }
>  
> -late_initcall(bpf_offload_init);
> +core_initcall(bpf_offload_init);
>
>
> Thorsten, how is the communication supposed to work in this case?
> Can you ask the reporter to test this? I don't see them on CC...

FWIW, I tried to reproduce the oops in the hope that I could confirm
a fix.  But I failed. The netdev is successfully deregistered on my
laptop no matter what I do. Tested v6.3 and current net/main, and
tried different tricks to change probe timing.

Guess this is timing sensitive enough that it only shows up on certain
systems.

So we will need the reporter to chime in.


Bjørn