From: Matthias Kaehlcke <mka@chromium.org>
To: Stefan Wahren <stefan.wahren@i2se.com>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
Johan Hovold <johan@kernel.org>,
linux-usb@vger.kernel.org,
Alexander Stein <alexander.stein@ew.tq-group.com>,
Icenowy Zheng <uwu@icenowy.me>,
Douglas Anderson <dianders@chromium.org>,
stable@vger.kernel.org, linux-kernel@vger.kernel.org,
Ravi Chandra Sadineni <ravisadineni@chromium.org>
Subject: Re: [PATCH 2/2] usb: misc: onboard_hub: Move 'attach' work to the driver
Date: Tue, 10 Jan 2023 01:07:13 +0000 [thread overview]
Message-ID: <Y7y6Qf0QhEJDp37Y@google.com> (raw)
In-Reply-To: <09d76f45-9dfe-19a0-33ec-badaac280772@i2se.com>
On Mon, Jan 09, 2023 at 09:19:02PM +0100, Stefan Wahren wrote:
> Hi Matthias,
>
> Am 09.01.23 um 18:40 schrieb Matthias Kaehlcke:
> > On Sun, Jan 08, 2023 at 11:47:13AM +0100, Stefan Wahren wrote:
> > > Am 07.01.23 um 18:23 schrieb Stefan Wahren:
> > > > Hi Matthias,
> > > >
> > > > Am 06.01.23 um 00:03 schrieb Matthias Kaehlcke:
> > > > > Currently each onboard_hub platform device owns an 'attach' work,
> > > > > which is scheduled when the device probes. With this deadlocks
> > > > > have been reported on a Raspberry Pi 3 B+ [1], which has nested
> > > > > onboard hubs.
> > > > >
> > > > > The flow of the deadlock is something like this (with the onboard_hub
> > > > > driver built as a module) [2]:
> > > > >
> > > > > - USB root hub is instantiated
> > > > > - core hub driver calls onboard_hub_create_pdevs(), which creates the
> > > > > 'raw' platform device for the 1st level hub
> > > > > - 1st level hub is probed by the core hub driver
> > > > > - core hub driver calls onboard_hub_create_pdevs(), which creates
> > > > > the 'raw' platform device for the 2nd level hub
> > > > >
> > > > > - onboard_hub platform driver is registered
> > > > > - platform device for 1st level hub is probed
> > > > > - schedules 'attach' work
> > > > > - platform device for 2nd level hub is probed
> > > > > - schedules 'attach' work
> > > > >
> > > > > - onboard_hub USB driver is registered
> > > > > - device (and parent) lock of hub is held while the device is
> > > > > re-probed with the onboard_hub driver
> > > > >
> > > > > - 'attach' work (running in another thread) calls driver_attach(), which
> > > > > blocks on one of the hub device locks
> > > > >
> > > > > - onboard_hub_destroy_pdevs() is called by the core hub driver when one
> > > > > of the hubs is detached
> > > > > - destroying the pdevs invokes onboard_hub_remove(), which waits for the
> > > > > 'attach' work to complete
> > > > > - waits forever, since the 'attach' work can't acquire the device
> > > > > lock
> > > > >
> > > > > Use a single work struct for the driver instead of having a work struct
> > > > > per onboard hub platform driver instance. With that it isn't necessary
> > > > > to cancel the work in onboard_hub_remove(), which fixes the deadlock.
> > > > > The work is only cancelled when the driver is unloaded.
> > > > i applied both patches for this series on top of v6.1
> > > > (multi_v7_defconfig), but usb is still broken on Raspberry Pi 3 B+
> > Thanks for testing.
> >
> > > here is the hung task output:
> > >
> > > [ 243.682193] INFO: task kworker/1:0:18 blocked for more than 122 seconds.
> > > [ 243.682222] Not tainted 6.1.0-00002-gaa61d98d165b #2
> > > [ 243.682233] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
> > > this message.
> > > [ 243.682242] task:kworker/1:0 state:D stack:0 pid:18 ppid:2
> > > flags:0x00000000
> > > [ 243.682267] Workqueue: events onboard_hub_attach_usb_driver
> > > [onboard_usb_hub]
> > > [ 243.682317] __schedule from schedule+0x4c/0xe0
> > > [ 243.682345] schedule from schedule_preempt_disabled+0xc/0x10
> > > [ 243.682367] schedule_preempt_disabled from
> > > __mutex_lock.constprop.0+0x244/0x804
> > > [ 243.682394] __mutex_lock.constprop.0 from __driver_attach+0x7c/0x188
> > > [ 243.682421] __driver_attach from bus_for_each_dev+0x70/0xb0
> > > [ 243.682449] bus_for_each_dev from onboard_hub_attach_usb_driver+0xc/0x28
> > > [onboard_usb_hub]
> > > [ 243.682494] onboard_hub_attach_usb_driver [onboard_usb_hub] from
> > > process_one_work+0x1ec/0x4d0
> > > [ 243.682534] process_one_work from worker_thread+0x50/0x540
> > > [ 243.682559] worker_thread from kthread+0xd0/0xec
> > > [ 243.682582] kthread from ret_from_fork+0x14/0x2c
> > > [ 243.682600] Exception stack(0xf086dfb0 to 0xf086dff8)
> > > [ 243.682615] dfa0: 00000000 00000000
> > > 00000000 00000000
> > > [ 243.682631] dfc0: 00000000 00000000 00000000 00000000 00000000 00000000
> > > 00000000 00000000
> > > [ 243.682646] dfe0: 00000000 00000000 00000000 00000000 00000013 00000000
> > > [ 243.682692] INFO: task kworker/1:2:82 blocked for more than 122 seconds.
> > > [ 243.682703] Not tainted 6.1.0-00002-gaa61d98d165b #2
> > > [ 243.682713] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
> > > this message.
> > > [ 243.682721] task:kworker/1:2 state:D stack:0 pid:82 ppid:2
> > > flags:0x00000000
> > > [ 243.682741] Workqueue: events_power_efficient hub_init_func2
> > > [ 243.682764] __schedule from schedule+0x4c/0xe0
> > > [ 243.682785] schedule from schedule_preempt_disabled+0xc/0x10
> > > [ 243.682808] schedule_preempt_disabled from
> > > __mutex_lock.constprop.0+0x244/0x804
> > > [ 243.682833] __mutex_lock.constprop.0 from hub_activate+0x584/0x8b0
> > > [ 243.682859] hub_activate from process_one_work+0x1ec/0x4d0
> > > [ 243.682883] process_one_work from worker_thread+0x50/0x540
> > > [ 243.682907] worker_thread from kthread+0xd0/0xec
> > > [ 243.682927] kthread from ret_from_fork+0x14/0x2c
> > > [ 243.682944] Exception stack(0xf1509fb0 to 0xf1509ff8)
> > > [ 243.682958] 9fa0: 00000000 00000000
> > > 00000000 00000000
> > > [ 243.682974] 9fc0: 00000000 00000000 00000000 00000000 00000000 00000000
> > > 00000000 00000000
> > > [ 243.682988] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000
> > > [ 243.683023] INFO: task kworker/1:4:257 blocked for more than 122 seconds.
> > > [ 243.683034] Not tainted 6.1.0-00002-gaa61d98d165b #2
> > > [ 243.683043] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
> > > this message.
> > > [ 243.683051] task:kworker/1:4 state:D stack:0 pid:257 ppid:2
> > > flags:0x00000000
> > > [ 243.683071] Workqueue: events_power_efficient hub_init_func2
> > > [ 243.683092] __schedule from schedule+0x4c/0xe0
> > > [ 243.683113] schedule from schedule_preempt_disabled+0xc/0x10
> > > [ 243.683135] schedule_preempt_disabled from
> > > __mutex_lock.constprop.0+0x244/0x804
> > > [ 243.683160] __mutex_lock.constprop.0 from hub_activate+0x584/0x8b0
> > > [ 243.683184] hub_activate from process_one_work+0x1ec/0x4d0
> > > [ 243.683209] process_one_work from worker_thread+0x50/0x540
> > > [ 243.683233] worker_thread from kthread+0xd0/0xec
> > > [ 243.683253] kthread from ret_from_fork+0x14/0x2c
> > > [ 243.683270] Exception stack(0xf09d9fb0 to 0xf09d9ff8)
> > > [ 243.683283] 9fa0: 00000000 00000000
> > > 00000000 00000000
> > > [ 243.683299] 9fc0: 00000000 00000000 00000000 00000000 00000000 00000000
> > > 00000000 00000000
> > > [ 243.683313] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000
> > Does commenting the following help:
> >
> > while (work_busy(&attach_usb_driver_work) & WORK_BUSY_RUNNING)
> > msleep(10);
> >
> > ?
> Yes, it does. I restarted the board multiple times and it never hang :-)
Thanks again for testing!
I'll post a version without that loop, which shouldn't be needed as per the
rationale in my previous mail.
next prev parent reply other threads:[~2023-01-10 1:07 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-01-05 23:03 [PATCH 1/2] usb: misc: onboard_hub: Invert driver registration order Matthias Kaehlcke
2023-01-05 23:03 ` [PATCH 2/2] usb: misc: onboard_hub: Move 'attach' work to the driver Matthias Kaehlcke
2023-01-07 17:23 ` Stefan Wahren
2023-01-08 10:47 ` Stefan Wahren
2023-01-09 17:40 ` Matthias Kaehlcke
2023-01-09 20:19 ` Stefan Wahren
2023-01-10 1:07 ` Matthias Kaehlcke [this message]
2023-01-06 15:32 ` [PATCH 1/2] usb: misc: onboard_hub: Invert driver registration order Greg Kroah-Hartman
2023-01-06 16:13 ` Matthias Kaehlcke
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=Y7y6Qf0QhEJDp37Y@google.com \
--to=mka@chromium.org \
--cc=alexander.stein@ew.tq-group.com \
--cc=dianders@chromium.org \
--cc=gregkh@linuxfoundation.org \
--cc=johan@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-usb@vger.kernel.org \
--cc=ravisadineni@chromium.org \
--cc=stable@vger.kernel.org \
--cc=stefan.wahren@i2se.com \
--cc=uwu@icenowy.me \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).