public inbox for linux-kernel@vger.kernel.org
 help / color / mirror / Atom feed
From: Sasha Levin <sashal@kernel.org>
To: linux-kernel@vger.kernel.org, stable@vger.kernel.org
Cc: Mika Westerberg <mika.westerberg@linux.intel.com>,
	AceLan Kao <acelan.kao@canonical.com>,
	"Rafael J . Wysocki" <rafael.j.wysocki@intel.com>,
	Jens Axboe <axboe@kernel.dk>, Sasha Levin <sashal@kernel.org>,
	linux-mm@kvack.org
Subject: [PATCH AUTOSEL 5.3 30/33] bdi: Do not use freezable workqueue
Date: Fri, 25 Oct 2019 09:55:02 -0400	[thread overview]
Message-ID: <20191025135505.24762-30-sashal@kernel.org> (raw)
In-Reply-To: <20191025135505.24762-1-sashal@kernel.org>

From: Mika Westerberg <mika.westerberg@linux.intel.com>

[ Upstream commit a2b90f11217790ec0964ba9c93a4abb369758c26 ]

A removable block device, such as NVMe or SSD connected over Thunderbolt
can be hot-removed any time including when the system is suspended. When
device is hot-removed during suspend and the system gets resumed, kernel
first resumes devices and then thaws the userspace including freezable
workqueues. What happens in that case is that the NVMe driver notices
that the device is unplugged and removes it from the system. This ends
up calling bdi_unregister() for the gendisk which then schedules
wb_workfn() to be run one more time.

However, since the bdi_wq is still frozen flush_delayed_work() call in
wb_shutdown() blocks forever halting system resume process. User sees
this as hang as nothing is happening anymore.

Triggering sysrq-w reveals this:

  Workqueue: nvme-wq nvme_remove_dead_ctrl_work [nvme]
  Call Trace:
   ? __schedule+0x2c5/0x630
   ? wait_for_completion+0xa4/0x120
   schedule+0x3e/0xc0
   schedule_timeout+0x1c9/0x320
   ? resched_curr+0x1f/0xd0
   ? wait_for_completion+0xa4/0x120
   wait_for_completion+0xc3/0x120
   ? wake_up_q+0x60/0x60
   __flush_work+0x131/0x1e0
   ? flush_workqueue_prep_pwqs+0x130/0x130
   bdi_unregister+0xb9/0x130
   del_gendisk+0x2d2/0x2e0
   nvme_ns_remove+0xed/0x110 [nvme_core]
   nvme_remove_namespaces+0x96/0xd0 [nvme_core]
   nvme_remove+0x5b/0x160 [nvme]
   pci_device_remove+0x36/0x90
   device_release_driver_internal+0xdf/0x1c0
   nvme_remove_dead_ctrl_work+0x14/0x30 [nvme]
   process_one_work+0x1c2/0x3f0
   worker_thread+0x48/0x3e0
   kthread+0x100/0x140
   ? current_work+0x30/0x30
   ? kthread_park+0x80/0x80
   ret_from_fork+0x35/0x40

This is not limited to NVMes so exactly same issue can be reproduced by
hot-removing SSD (over Thunderbolt) while the system is suspended.

Prevent this from happening by removing WQ_FREEZABLE from bdi_wq.

Reported-by: AceLan Kao <acelan.kao@canonical.com>
Link: https://marc.info/?l=linux-kernel&m=138695698516487
Link: https://bugzilla.kernel.org/show_bug.cgi?id=204385
Link: https://lore.kernel.org/lkml/20191002122136.GD2819@lahna.fi.intel.com/#t
Acked-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
Signed-off-by: Mika Westerberg <mika.westerberg@linux.intel.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Signed-off-by: Sasha Levin <sashal@kernel.org>
---
 mm/backing-dev.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/mm/backing-dev.c b/mm/backing-dev.c
index e8e89158adec6..553372b39178c 100644
--- a/mm/backing-dev.c
+++ b/mm/backing-dev.c
@@ -236,8 +236,8 @@ static int __init default_bdi_init(void)
 {
 	int err;
 
-	bdi_wq = alloc_workqueue("writeback", WQ_MEM_RECLAIM | WQ_FREEZABLE |
-					      WQ_UNBOUND | WQ_SYSFS, 0);
+	bdi_wq = alloc_workqueue("writeback", WQ_MEM_RECLAIM | WQ_UNBOUND |
+				 WQ_SYSFS, 0);
 	if (!bdi_wq)
 		return -ENOMEM;
 
-- 
2.20.1


  parent reply	other threads:[~2019-10-25 14:06 UTC|newest]

Thread overview: 35+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-10-25 13:54 [PATCH AUTOSEL 5.3 01/33] net: ipv6: fix listify ip6_rcv_finish in case of forwarding Sasha Levin
2019-10-25 13:54 ` [PATCH AUTOSEL 5.3 02/33] sch_netem: fix rcu splat in netem_enqueue() Sasha Levin
2019-10-25 13:54 ` [PATCH AUTOSEL 5.3 03/33] net: sched: sch_sfb: don't call qdisc_put() while holding tree lock Sasha Levin
2019-10-25 13:54 ` [PATCH AUTOSEL 5.3 04/33] rxrpc: Fix call ref leak Sasha Levin
2019-10-25 13:54 ` [PATCH AUTOSEL 5.3 05/33] rxrpc: Fix trace-after-put looking at the put peer record Sasha Levin
2019-10-25 13:54 ` [PATCH AUTOSEL 5.3 06/33] rxrpc: rxrpc_peer needs to hold a ref on the rxrpc_local record Sasha Levin
2019-10-25 13:54 ` [PATCH AUTOSEL 5.3 07/33] llc: fix sk_buff leak in llc_sap_state_process() Sasha Levin
2019-10-25 13:54 ` [PATCH AUTOSEL 5.3 08/33] llc: fix sk_buff leak in llc_conn_service() Sasha Levin
2019-10-25 13:54 ` [PATCH AUTOSEL 5.3 09/33] NFC: pn533: fix use-after-free and memleaks Sasha Levin
2019-10-25 13:54 ` [PATCH AUTOSEL 5.3 10/33] bonding: fix potential NULL deref in bond_update_slave_arr Sasha Levin
2019-10-25 13:54 ` [PATCH AUTOSEL 5.3 11/33] netfilter: conntrack: avoid possible false sharing Sasha Levin
2019-10-25 13:54 ` [PATCH AUTOSEL 5.3 12/33] blackhole_netdev: fix syzkaller reported issue Sasha Levin
2019-10-25 15:49   ` Sasha Levin
2019-10-25 13:54 ` [PATCH AUTOSEL 5.3 13/33] net: usb: sr9800: fix uninitialized local variable Sasha Levin
2019-10-25 13:54 ` [PATCH AUTOSEL 5.3 14/33] scsi: qla2xxx: Fix different size DMA Alloc/Unmap Sasha Levin
2019-10-25 13:54 ` [PATCH AUTOSEL 5.3 15/33] jbd2: flush_descriptor(): Do not decrease buffer head's ref count Sasha Levin
2019-10-25 13:54 ` [PATCH AUTOSEL 5.3 16/33] netfilter: connlabels: prefer static lock initialiser Sasha Levin
2019-10-25 13:54 ` [PATCH AUTOSEL 5.3 17/33] usb: typec: fusb302: Call fusb302_debugfs_init earlier Sasha Levin
2019-10-25 13:54 ` [PATCH AUTOSEL 5.3 18/33] f2fs: fix to avoid discard command leak Sasha Levin
2019-10-25 13:54 ` [PATCH AUTOSEL 5.3 19/33] net/rds: Whitelist rdma_cookie and rx_tstamp for usercopy Sasha Levin
2019-10-25 13:54 ` [PATCH AUTOSEL 5.3 20/33] scsi: qedf: Fix crash during sg_reset Sasha Levin
2019-10-25 13:54 ` [PATCH AUTOSEL 5.3 21/33] ath9k: dynack: fix possible deadlock in ath_dynack_node_{de}init Sasha Levin
2019-10-25 13:54 ` [PATCH AUTOSEL 5.3 22/33] Btrfs: fix hang when loading existing inode cache off disk Sasha Levin
2019-10-25 13:54 ` [PATCH AUTOSEL 5.3 23/33] Btrfs: fix inode cache block reserve leak on failure to allocate data space Sasha Levin
2019-10-25 13:54 ` [PATCH AUTOSEL 5.3 24/33] ubi: ubi_wl_get_peb: Increase the number of attempts while getting PEB Sasha Levin
2019-10-25 13:54 ` [PATCH AUTOSEL 5.3 25/33] net: sched: sch_htb: don't call qdisc_put() while holding tree lock Sasha Levin
2019-10-25 13:54 ` [PATCH AUTOSEL 5.3 26/33] net: sched: multiq: " Sasha Levin
2019-10-25 13:54 ` [PATCH AUTOSEL 5.3 27/33] RDMA/siw: Fix serialization issue in write_space() Sasha Levin
2019-10-25 13:55 ` [PATCH AUTOSEL 5.3 28/33] RDMA/iwcm: Fix a lock inversion issue Sasha Levin
2019-10-25 13:55 ` [PATCH AUTOSEL 5.3 29/33] ipv6: Handle race in addrconf_dad_work Sasha Levin
2019-10-25 13:55 ` Sasha Levin [this message]
2019-10-25 13:55 ` [PATCH AUTOSEL 5.3 31/33] sctp: add chunks to sk_backlog when the newsk sk_socket is not set Sasha Levin
2019-10-25 13:55 ` [PATCH AUTOSEL 5.3 32/33] ALSA: hda: Add codec on bus address table lately Sasha Levin
2019-10-25 14:09   ` Takashi Iwai
2019-10-25 13:55 ` [PATCH AUTOSEL 5.3 33/33] nvme: Add quirk for Kingston NVME SSD running FW E8FK11.T Sasha Levin

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20191025135505.24762-30-sashal@kernel.org \
    --to=sashal@kernel.org \
    --cc=acelan.kao@canonical.com \
    --cc=axboe@kernel.dk \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=mika.westerberg@linux.intel.com \
    --cc=rafael.j.wysocki@intel.com \
    --cc=stable@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox