* Re: [Bug 215129] New: Linux kernel hangs during power down
[not found] <20211124144505.31e15716@hermes.local>
@ 2021-11-25 0:46 ` Jakub Kicinski
2021-11-25 7:32 ` Heiner Kallweit
0 siblings, 1 reply; 4+ messages in thread
From: Jakub Kicinski @ 2021-11-25 0:46 UTC (permalink / raw)
To: Stephen Hemminger; +Cc: netdev, ath10k, Heiner Kallweit, Kalle Valo
Adding Kalle and Hainer.
On Wed, 24 Nov 2021 14:45:05 -0800 Stephen Hemminger wrote:
> Begin forwarded message:
>
> Date: Wed, 24 Nov 2021 21:14:53 +0000
> From: bugzilla-daemon@bugzilla.kernel.org
> To: stephen@networkplumber.org
> Subject: [Bug 215129] New: Linux kernel hangs during power down
>
>
> https://bugzilla.kernel.org/show_bug.cgi?id=215129
>
> Bug ID: 215129
> Summary: Linux kernel hangs during power down
> Product: Networking
> Version: 2.5
> Kernel Version: 5.15
> Hardware: All
> OS: Linux
> Tree: Mainline
> Status: NEW
> Severity: normal
> Priority: P1
> Component: Other
> Assignee: stephen@networkplumber.org
> Reporter: martin.stolpe@gmail.com
> Regression: No
>
> Created attachment 299703
> --> https://bugzilla.kernel.org/attachment.cgi?id=299703&action=edit
> Kernel log after timeout occured
>
> On my system the kernel is waiting for a task during shutdown which doesn't
> complete.
>
> The commit which causes this behavior is:
> [f32a213765739f2a1db319346799f130a3d08820] ethtool: runtime-resume netdev
> parent before ethtool ioctl ops
>
> This bug causes also that the system gets unresponsive after starting Steam:
> https://steamcommunity.com/app/221410/discussions/2/3194736442566303600/
>
_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [Bug 215129] New: Linux kernel hangs during power down
2021-11-25 0:46 ` [Bug 215129] New: Linux kernel hangs during power down Jakub Kicinski
@ 2021-11-25 7:32 ` Heiner Kallweit
2021-11-25 15:49 ` Jakub Kicinski
2021-11-25 21:11 ` Heiner Kallweit
0 siblings, 2 replies; 4+ messages in thread
From: Heiner Kallweit @ 2021-11-25 7:32 UTC (permalink / raw)
To: Jakub Kicinski, Jesse Brandeburg, Tony Nguyen, intel-wired-lan,
Kalle Valo
Cc: netdev, ath10k, Stephen Hemminger
On 25.11.2021 01:46, Jakub Kicinski wrote:
> Adding Kalle and Hainer.
>
> On Wed, 24 Nov 2021 14:45:05 -0800 Stephen Hemminger wrote:
>> Begin forwarded message:
>>
>> Date: Wed, 24 Nov 2021 21:14:53 +0000
>> From: bugzilla-daemon@bugzilla.kernel.org
>> To: stephen@networkplumber.org
>> Subject: [Bug 215129] New: Linux kernel hangs during power down
>>
>>
>> https://bugzilla.kernel.org/show_bug.cgi?id=215129
>>
>> Bug ID: 215129
>> Summary: Linux kernel hangs during power down
>> Product: Networking
>> Version: 2.5
>> Kernel Version: 5.15
>> Hardware: All
>> OS: Linux
>> Tree: Mainline
>> Status: NEW
>> Severity: normal
>> Priority: P1
>> Component: Other
>> Assignee: stephen@networkplumber.org
>> Reporter: martin.stolpe@gmail.com
>> Regression: No
>>
>> Created attachment 299703
>> --> https://bugzilla.kernel.org/attachment.cgi?id=299703&action=edit
>> Kernel log after timeout occured
>>
>> On my system the kernel is waiting for a task during shutdown which doesn't
>> complete.
>>
>> The commit which causes this behavior is:
>> [f32a213765739f2a1db319346799f130a3d08820] ethtool: runtime-resume netdev
>> parent before ethtool ioctl ops
>>
>> This bug causes also that the system gets unresponsive after starting Steam:
>> https://steamcommunity.com/app/221410/discussions/2/3194736442566303600/
>>
>
I think the reference to ath10k_pci is misleading, Kalle isn't needed here.
The actual issue is a RTNL deadlock in igb_resume(). See log snippet:
Nov 24 18:56:19 MartinsPc kernel: igb_resume+0xff/0x1e0 [igb 21bf6a00cb1f20e9b0e8434f7f8748a0504e93f8]
Nov 24 18:56:19 MartinsPc kernel: pci_pm_runtime_resume+0xa7/0xd0
Nov 24 18:56:19 MartinsPc kernel: ? pci_pm_freeze_noirq+0x110/0x110
Nov 24 18:56:19 MartinsPc kernel: __rpm_callback+0x41/0x120
Nov 24 18:56:19 MartinsPc kernel: ? pci_pm_freeze_noirq+0x110/0x110
Nov 24 18:56:19 MartinsPc kernel: rpm_callback+0x35/0x70
Nov 24 18:56:19 MartinsPc kernel: rpm_resume+0x567/0x810
Nov 24 18:56:19 MartinsPc kernel: __pm_runtime_resume+0x4a/0x80
Nov 24 18:56:19 MartinsPc kernel: dev_ethtool+0xd4/0x2d80
We have at least two places in net core where runtime_resume() is called
under RTNL. This conflicts with the current structure in few Intel drivers
that have something like the following in their resume path.
rtnl_lock();
if (!err && netif_running(netdev))
err = __igb_open(netdev, true);
if (!err)
netif_device_attach(netdev);
rtnl_unlock();
Other drivers don't do this, so it's the question whether it's actually
needed here to take RTNL. Some discussion was started [0], but it ended
w/o tangible result and since then it has been surprisingly quiet.
[0] https://www.spinics.net/lists/netdev/msg736880.html
_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [Bug 215129] New: Linux kernel hangs during power down
2021-11-25 7:32 ` Heiner Kallweit
@ 2021-11-25 15:49 ` Jakub Kicinski
2021-11-25 21:11 ` Heiner Kallweit
1 sibling, 0 replies; 4+ messages in thread
From: Jakub Kicinski @ 2021-11-25 15:49 UTC (permalink / raw)
To: Heiner Kallweit
Cc: Jesse Brandeburg, Tony Nguyen, intel-wired-lan, netdev, ath10k,
Stephen Hemminger
On Thu, 25 Nov 2021 08:32:18 +0100 Heiner Kallweit wrote:
> I think the reference to ath10k_pci is misleading, Kalle isn't needed here.
> The actual issue is a RTNL deadlock in igb_resume(). See log snippet:
>
> Nov 24 18:56:19 MartinsPc kernel: igb_resume+0xff/0x1e0 [igb 21bf6a00cb1f20e9b0e8434f7f8748a0504e93f8]
> Nov 24 18:56:19 MartinsPc kernel: pci_pm_runtime_resume+0xa7/0xd0
> Nov 24 18:56:19 MartinsPc kernel: ? pci_pm_freeze_noirq+0x110/0x110
> Nov 24 18:56:19 MartinsPc kernel: __rpm_callback+0x41/0x120
> Nov 24 18:56:19 MartinsPc kernel: ? pci_pm_freeze_noirq+0x110/0x110
> Nov 24 18:56:19 MartinsPc kernel: rpm_callback+0x35/0x70
> Nov 24 18:56:19 MartinsPc kernel: rpm_resume+0x567/0x810
> Nov 24 18:56:19 MartinsPc kernel: __pm_runtime_resume+0x4a/0x80
> Nov 24 18:56:19 MartinsPc kernel: dev_ethtool+0xd4/0x2d80
>
> We have at least two places in net core where runtime_resume() is called
> under RTNL. This conflicts with the current structure in few Intel drivers
> that have something like the following in their resume path.
>
> rtnl_lock();
> if (!err && netif_running(netdev))
> err = __igb_open(netdev, true);
>
> if (!err)
> netif_device_attach(netdev);
> rtnl_unlock();
>
> Other drivers don't do this, so it's the question whether it's actually
> needed here to take RTNL. Some discussion was started [0], but it ended
> w/o tangible result and since then it has been surprisingly quiet.
>
> [0] https://www.spinics.net/lists/netdev/msg736880.html
Ah, that makes perfect sense, I didn't see that stack trace.
Dropping Kalle from CC. Let's hear what Intel folks have to say..
_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [Bug 215129] New: Linux kernel hangs during power down
2021-11-25 7:32 ` Heiner Kallweit
2021-11-25 15:49 ` Jakub Kicinski
@ 2021-11-25 21:11 ` Heiner Kallweit
1 sibling, 0 replies; 4+ messages in thread
From: Heiner Kallweit @ 2021-11-25 21:11 UTC (permalink / raw)
To: Jakub Kicinski, Jesse Brandeburg, Tony Nguyen, intel-wired-lan,
Kalle Valo
Cc: netdev, ath10k, Stephen Hemminger
On 25.11.2021 08:32, Heiner Kallweit wrote:
> On 25.11.2021 01:46, Jakub Kicinski wrote:
>> Adding Kalle and Hainer.
>>
>> On Wed, 24 Nov 2021 14:45:05 -0800 Stephen Hemminger wrote:
>>> Begin forwarded message:
>>>
>>> Date: Wed, 24 Nov 2021 21:14:53 +0000
>>> From: bugzilla-daemon@bugzilla.kernel.org
>>> To: stephen@networkplumber.org
>>> Subject: [Bug 215129] New: Linux kernel hangs during power down
>>>
>>>
>>> https://bugzilla.kernel.org/show_bug.cgi?id=215129
>>>
>>> Bug ID: 215129
>>> Summary: Linux kernel hangs during power down
>>> Product: Networking
>>> Version: 2.5
>>> Kernel Version: 5.15
>>> Hardware: All
>>> OS: Linux
>>> Tree: Mainline
>>> Status: NEW
>>> Severity: normal
>>> Priority: P1
>>> Component: Other
>>> Assignee: stephen@networkplumber.org
>>> Reporter: martin.stolpe@gmail.com
>>> Regression: No
>>>
>>> Created attachment 299703
>>> --> https://bugzilla.kernel.org/attachment.cgi?id=299703&action=edit
>>> Kernel log after timeout occured
>>>
>>> On my system the kernel is waiting for a task during shutdown which doesn't
>>> complete.
>>>
>>> The commit which causes this behavior is:
>>> [f32a213765739f2a1db319346799f130a3d08820] ethtool: runtime-resume netdev
>>> parent before ethtool ioctl ops
>>>
>>> This bug causes also that the system gets unresponsive after starting Steam:
>>> https://steamcommunity.com/app/221410/discussions/2/3194736442566303600/
>>>
>>
>
> I think the reference to ath10k_pci is misleading, Kalle isn't needed here.
> The actual issue is a RTNL deadlock in igb_resume(). See log snippet:
>
> Nov 24 18:56:19 MartinsPc kernel: igb_resume+0xff/0x1e0 [igb 21bf6a00cb1f20e9b0e8434f7f8748a0504e93f8]
> Nov 24 18:56:19 MartinsPc kernel: pci_pm_runtime_resume+0xa7/0xd0
> Nov 24 18:56:19 MartinsPc kernel: ? pci_pm_freeze_noirq+0x110/0x110
> Nov 24 18:56:19 MartinsPc kernel: __rpm_callback+0x41/0x120
> Nov 24 18:56:19 MartinsPc kernel: ? pci_pm_freeze_noirq+0x110/0x110
> Nov 24 18:56:19 MartinsPc kernel: rpm_callback+0x35/0x70
> Nov 24 18:56:19 MartinsPc kernel: rpm_resume+0x567/0x810
> Nov 24 18:56:19 MartinsPc kernel: __pm_runtime_resume+0x4a/0x80
> Nov 24 18:56:19 MartinsPc kernel: dev_ethtool+0xd4/0x2d80
>
> We have at least two places in net core where runtime_resume() is called
> under RTNL. This conflicts with the current structure in few Intel drivers
> that have something like the following in their resume path.
>
> rtnl_lock();
> if (!err && netif_running(netdev))
> err = __igb_open(netdev, true);
>
> if (!err)
> netif_device_attach(netdev);
> rtnl_unlock();
>
> Other drivers don't do this, so it's the question whether it's actually
> needed here to take RTNL. Some discussion was started [0], but it ended
> w/o tangible result and since then it has been surprisingly quiet.
>
> [0] https://www.spinics.net/lists/netdev/msg736880.html
>
I think the problem with runtime_resume() taking RTNL could also hit
the driver internally. See following call chain: If this would ever
be called when the device is runtime-suspended, then a similar
deadlock would occur.
__dev_open() - called with RTNL held
igb_open()
__igb_open() - arg resuming is false
if (!resuming)
pm_runtime_get_sync(&pdev->dev);
igb_resume()
rtnl_lock()
_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2021-11-25 21:11 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <20211124144505.31e15716@hermes.local>
2021-11-25 0:46 ` [Bug 215129] New: Linux kernel hangs during power down Jakub Kicinski
2021-11-25 7:32 ` Heiner Kallweit
2021-11-25 15:49 ` Jakub Kicinski
2021-11-25 21:11 ` Heiner Kallweit
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox