From: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
To: linux-kernel@vger.kernel.org
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
stable@vger.kernel.org, Christoph Hellwig <hch@lst.de>,
Keith Busch <keith.busch@intel.com>,
Lee Duncan <lduncan@suse.com>, Chris Leech <cleech@redhat.com>,
Bart Van Assche <bvanassche@acm.org>,
"Martin K. Petersen" <martin.petersen@oracle.com>
Subject: [PATCH 4.14 32/36] scsi: iscsi: Fix a potential deadlock in the timeout handler
Date: Thu, 19 Dec 2019 19:34:49 +0100 [thread overview]
Message-ID: <20191219182924.175808423@linuxfoundation.org> (raw)
In-Reply-To: <20191219182848.708141124@linuxfoundation.org>
From: Bart Van Assche <bvanassche@acm.org>
commit 5480e299b5ae57956af01d4839c9fc88a465eeab upstream.
Some time ago the block layer was modified such that timeout handlers are
called from thread context instead of interrupt context. Make it safe to
run the iSCSI timeout handler in thread context. This patch fixes the
following lockdep complaint:
================================
WARNING: inconsistent lock state
5.5.1-dbg+ #11 Not tainted
--------------------------------
inconsistent {IN-SOFTIRQ-W} -> {SOFTIRQ-ON-W} usage.
kworker/7:1H/206 [HC0[0]:SC0[0]:HE1:SE1] takes:
ffff88802d9827e8 (&(&session->frwd_lock)->rlock){+.?.}, at: iscsi_eh_cmd_timed_out+0xa6/0x6d0 [libiscsi]
{IN-SOFTIRQ-W} state was registered at:
lock_acquire+0x106/0x240
_raw_spin_lock+0x38/0x50
iscsi_check_transport_timeouts+0x3e/0x210 [libiscsi]
call_timer_fn+0x132/0x470
__run_timers.part.0+0x39f/0x5b0
run_timer_softirq+0x63/0xc0
__do_softirq+0x12d/0x5fd
irq_exit+0xb3/0x110
smp_apic_timer_interrupt+0x131/0x3d0
apic_timer_interrupt+0xf/0x20
default_idle+0x31/0x230
arch_cpu_idle+0x13/0x20
default_idle_call+0x53/0x60
do_idle+0x38a/0x3f0
cpu_startup_entry+0x24/0x30
start_secondary+0x222/0x290
secondary_startup_64+0xa4/0xb0
irq event stamp: 1383705
hardirqs last enabled at (1383705): [<ffffffff81aace5c>] _raw_spin_unlock_irq+0x2c/0x50
hardirqs last disabled at (1383704): [<ffffffff81aacb98>] _raw_spin_lock_irq+0x18/0x50
softirqs last enabled at (1383690): [<ffffffffa0e2efea>] iscsi_queuecommand+0x76a/0xa20 [libiscsi]
softirqs last disabled at (1383682): [<ffffffffa0e2e998>] iscsi_queuecommand+0x118/0xa20 [libiscsi]
other info that might help us debug this:
Possible unsafe locking scenario:
CPU0
----
lock(&(&session->frwd_lock)->rlock);
<Interrupt>
lock(&(&session->frwd_lock)->rlock);
*** DEADLOCK ***
2 locks held by kworker/7:1H/206:
#0: ffff8880d57bf928 ((wq_completion)kblockd){+.+.}, at: process_one_work+0x472/0xab0
#1: ffff88802b9c7de8 ((work_completion)(&q->timeout_work)){+.+.}, at: process_one_work+0x476/0xab0
stack backtrace:
CPU: 7 PID: 206 Comm: kworker/7:1H Not tainted 5.5.1-dbg+ #11
Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
Workqueue: kblockd blk_mq_timeout_work
Call Trace:
dump_stack+0xa5/0xe6
print_usage_bug.cold+0x232/0x23b
mark_lock+0x8dc/0xa70
__lock_acquire+0xcea/0x2af0
lock_acquire+0x106/0x240
_raw_spin_lock+0x38/0x50
iscsi_eh_cmd_timed_out+0xa6/0x6d0 [libiscsi]
scsi_times_out+0xf4/0x440 [scsi_mod]
scsi_timeout+0x1d/0x20 [scsi_mod]
blk_mq_check_expired+0x365/0x3a0
bt_iter+0xd6/0xf0
blk_mq_queue_tag_busy_iter+0x3de/0x650
blk_mq_timeout_work+0x1af/0x380
process_one_work+0x56d/0xab0
worker_thread+0x7a/0x5d0
kthread+0x1bc/0x210
ret_from_fork+0x24/0x30
Fixes: 287922eb0b18 ("block: defer timeouts to a workqueue")
Cc: Christoph Hellwig <hch@lst.de>
Cc: Keith Busch <keith.busch@intel.com>
Cc: Lee Duncan <lduncan@suse.com>
Cc: Chris Leech <cleech@redhat.com>
Cc: <stable@vger.kernel.org>
Link: https://lore.kernel.org/r/20191209173457.187370-1-bvanassche@acm.org
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Reviewed-by: Lee Duncan <lduncan@suse.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
---
drivers/scsi/libiscsi.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
--- a/drivers/scsi/libiscsi.c
+++ b/drivers/scsi/libiscsi.c
@@ -1983,7 +1983,7 @@ enum blk_eh_timer_return iscsi_eh_cmd_ti
ISCSI_DBG_EH(session, "scsi cmd %p timedout\n", sc);
- spin_lock(&session->frwd_lock);
+ spin_lock_bh(&session->frwd_lock);
task = (struct iscsi_task *)sc->SCp.ptr;
if (!task) {
/*
@@ -2110,7 +2110,7 @@ enum blk_eh_timer_return iscsi_eh_cmd_ti
done:
if (task)
task->last_timeout = jiffies;
- spin_unlock(&session->frwd_lock);
+ spin_unlock_bh(&session->frwd_lock);
ISCSI_DBG_EH(session, "return %s\n", rc == BLK_EH_RESET_TIMER ?
"timer reset" : "shutdown or nh");
return rc;
next prev parent reply other threads:[~2019-12-19 19:01 UTC|newest]
Thread overview: 41+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-12-19 18:34 [PATCH 4.14 00/36] 4.14.160-stable review Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 4.14 01/36] net: bridge: deny dev_set_mac_address() when unregistering Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 4.14 02/36] net: dsa: fix flow dissection on Tx path Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 4.14 03/36] net: ethernet: ti: cpsw: fix extra rx interrupt Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 4.14 04/36] net: thunderx: start phy before starting autonegotiation Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 4.14 05/36] openvswitch: support asymmetric conntrack Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 4.14 06/36] tcp: md5: fix potential overestimation of TCP option space Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 4.14 07/36] tipc: fix ordering of tipc module init and exit routine Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 4.14 08/36] tcp: fix rejected syncookies due to stale timestamps Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 4.14 09/36] tcp: tighten acceptance of ACKs not matching a child socket Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 4.14 10/36] tcp: Protect accesses to .ts_recent_stamp with {READ,WRITE}_ONCE() Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 4.14 11/36] inet: protect against too small mtu values Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 4.14 12/36] nvme: host: core: fix precedence of ternary operator Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 4.14 13/36] Revert "regulator: Defer init completion for a while after late_initcall" Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 4.14 14/36] PCI/PM: Always return devices to D0 when thawing Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 4.14 15/36] PCI: Fix Intel ACS quirk UPDCR register address Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 4.14 16/36] PCI/MSI: Fix incorrect MSI-X masking on resume Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 4.14 17/36] PCI: Apply Cavium ACS quirk to ThunderX2 and ThunderX3 Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 4.14 18/36] xtensa: fix TLB sanity checker Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 4.14 19/36] rpmsg: glink: Set tail pointer to 0 at end of FIFO Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 4.14 20/36] rpmsg: glink: Fix reuse intents memory leak issue Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 4.14 21/36] rpmsg: glink: Fix use after free in open_ack TIMEOUT case Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 4.14 22/36] rpmsg: glink: Put an extra reference during cleanup Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 4.14 23/36] rpmsg: glink: Fix rpmsg_register_device err handling Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 4.14 24/36] rpmsg: glink: Dont send pending rx_done during remove Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 4.14 25/36] rpmsg: glink: Free pending deferred work on remove Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 4.14 26/36] CIFS: Respect O_SYNC and O_DIRECT flags during reconnect Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 4.14 27/36] ARM: dts: s3c64xx: Fix init order of clock providers Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 4.14 28/36] ARM: tegra: Fix FLOW_CTLR_HALT register clobbering by tegra_resume() Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 4.14 29/36] vfio/pci: call irq_bypass_unregister_producer() before freeing irq Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 4.14 30/36] dma-buf: Fix memory leak in sync_file_merge() Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 4.14 31/36] dm btree: increase rebalance threshold in __rebalance2() Greg Kroah-Hartman
2019-12-19 18:34 ` Greg Kroah-Hartman [this message]
2019-12-19 18:34 ` [PATCH 4.14 33/36] drm/radeon: fix r1xx/r2xx register checker for POT textures Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 4.14 34/36] xhci: fix USB3 device initiated resume race with roothub autosuspend Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 4.14 35/36] net: stmmac: use correct DMA buffer size in the RX descriptor Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 4.14 36/36] net: stmmac: dont stop NAPI processing when dropping a packet Greg Kroah-Hartman
2019-12-20 4:39 ` [PATCH 4.14 00/36] 4.14.160-stable review shuah
2019-12-20 5:34 ` Naresh Kamboju
2019-12-20 10:29 ` Jon Hunter
2019-12-20 18:48 ` Guenter Roeck
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20191219182924.175808423@linuxfoundation.org \
--to=gregkh@linuxfoundation.org \
--cc=bvanassche@acm.org \
--cc=cleech@redhat.com \
--cc=hch@lst.de \
--cc=keith.busch@intel.com \
--cc=lduncan@suse.com \
--cc=linux-kernel@vger.kernel.org \
--cc=martin.petersen@oracle.com \
--cc=stable@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).