stable.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
To: linux-kernel@vger.kernel.org
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	stable@vger.kernel.org, Christoph Hellwig <hch@lst.de>,
	Keith Busch <keith.busch@intel.com>,
	Lee Duncan <lduncan@suse.com>, Chris Leech <cleech@redhat.com>,
	Bart Van Assche <bvanassche@acm.org>,
	"Martin K. Petersen" <martin.petersen@oracle.com>
Subject: [PATCH 5.4 59/80] scsi: iscsi: Fix a potential deadlock in the timeout handler
Date: Thu, 19 Dec 2019 19:34:51 +0100	[thread overview]
Message-ID: <20191219183131.918040857@linuxfoundation.org> (raw)
In-Reply-To: <20191219183031.278083125@linuxfoundation.org>

From: Bart Van Assche <bvanassche@acm.org>

commit 5480e299b5ae57956af01d4839c9fc88a465eeab upstream.

Some time ago the block layer was modified such that timeout handlers are
called from thread context instead of interrupt context. Make it safe to
run the iSCSI timeout handler in thread context. This patch fixes the
following lockdep complaint:

================================
WARNING: inconsistent lock state
5.5.1-dbg+ #11 Not tainted
--------------------------------
inconsistent {IN-SOFTIRQ-W} -> {SOFTIRQ-ON-W} usage.
kworker/7:1H/206 [HC0[0]:SC0[0]:HE1:SE1] takes:
ffff88802d9827e8 (&(&session->frwd_lock)->rlock){+.?.}, at: iscsi_eh_cmd_timed_out+0xa6/0x6d0 [libiscsi]
{IN-SOFTIRQ-W} state was registered at:
  lock_acquire+0x106/0x240
  _raw_spin_lock+0x38/0x50
  iscsi_check_transport_timeouts+0x3e/0x210 [libiscsi]
  call_timer_fn+0x132/0x470
  __run_timers.part.0+0x39f/0x5b0
  run_timer_softirq+0x63/0xc0
  __do_softirq+0x12d/0x5fd
  irq_exit+0xb3/0x110
  smp_apic_timer_interrupt+0x131/0x3d0
  apic_timer_interrupt+0xf/0x20
  default_idle+0x31/0x230
  arch_cpu_idle+0x13/0x20
  default_idle_call+0x53/0x60
  do_idle+0x38a/0x3f0
  cpu_startup_entry+0x24/0x30
  start_secondary+0x222/0x290
  secondary_startup_64+0xa4/0xb0
irq event stamp: 1383705
hardirqs last  enabled at (1383705): [<ffffffff81aace5c>] _raw_spin_unlock_irq+0x2c/0x50
hardirqs last disabled at (1383704): [<ffffffff81aacb98>] _raw_spin_lock_irq+0x18/0x50
softirqs last  enabled at (1383690): [<ffffffffa0e2efea>] iscsi_queuecommand+0x76a/0xa20 [libiscsi]
softirqs last disabled at (1383682): [<ffffffffa0e2e998>] iscsi_queuecommand+0x118/0xa20 [libiscsi]

other info that might help us debug this:
 Possible unsafe locking scenario:

       CPU0
       ----
  lock(&(&session->frwd_lock)->rlock);
  <Interrupt>
    lock(&(&session->frwd_lock)->rlock);

 *** DEADLOCK ***

2 locks held by kworker/7:1H/206:
 #0: ffff8880d57bf928 ((wq_completion)kblockd){+.+.}, at: process_one_work+0x472/0xab0
 #1: ffff88802b9c7de8 ((work_completion)(&q->timeout_work)){+.+.}, at: process_one_work+0x476/0xab0

stack backtrace:
CPU: 7 PID: 206 Comm: kworker/7:1H Not tainted 5.5.1-dbg+ #11
Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
Workqueue: kblockd blk_mq_timeout_work
Call Trace:
 dump_stack+0xa5/0xe6
 print_usage_bug.cold+0x232/0x23b
 mark_lock+0x8dc/0xa70
 __lock_acquire+0xcea/0x2af0
 lock_acquire+0x106/0x240
 _raw_spin_lock+0x38/0x50
 iscsi_eh_cmd_timed_out+0xa6/0x6d0 [libiscsi]
 scsi_times_out+0xf4/0x440 [scsi_mod]
 scsi_timeout+0x1d/0x20 [scsi_mod]
 blk_mq_check_expired+0x365/0x3a0
 bt_iter+0xd6/0xf0
 blk_mq_queue_tag_busy_iter+0x3de/0x650
 blk_mq_timeout_work+0x1af/0x380
 process_one_work+0x56d/0xab0
 worker_thread+0x7a/0x5d0
 kthread+0x1bc/0x210
 ret_from_fork+0x24/0x30

Fixes: 287922eb0b18 ("block: defer timeouts to a workqueue")
Cc: Christoph Hellwig <hch@lst.de>
Cc: Keith Busch <keith.busch@intel.com>
Cc: Lee Duncan <lduncan@suse.com>
Cc: Chris Leech <cleech@redhat.com>
Cc: <stable@vger.kernel.org>
Link: https://lore.kernel.org/r/20191209173457.187370-1-bvanassche@acm.org
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Reviewed-by: Lee Duncan <lduncan@suse.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

---
 drivers/scsi/libiscsi.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

--- a/drivers/scsi/libiscsi.c
+++ b/drivers/scsi/libiscsi.c
@@ -1945,7 +1945,7 @@ enum blk_eh_timer_return iscsi_eh_cmd_ti
 
 	ISCSI_DBG_EH(session, "scsi cmd %p timedout\n", sc);
 
-	spin_lock(&session->frwd_lock);
+	spin_lock_bh(&session->frwd_lock);
 	task = (struct iscsi_task *)sc->SCp.ptr;
 	if (!task) {
 		/*
@@ -2072,7 +2072,7 @@ enum blk_eh_timer_return iscsi_eh_cmd_ti
 done:
 	if (task)
 		task->last_timeout = jiffies;
-	spin_unlock(&session->frwd_lock);
+	spin_unlock_bh(&session->frwd_lock);
 	ISCSI_DBG_EH(session, "return %s\n", rc == BLK_EH_RESET_TIMER ?
 		     "timer reset" : "shutdown or nh");
 	return rc;



  parent reply	other threads:[~2019-12-19 18:57 UTC|newest]

Thread overview: 90+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-12-19 18:33 [PATCH 5.4 00/80] 5.4.6-stable review Greg Kroah-Hartman
2019-12-19 18:33 ` [PATCH 5.4 01/80] USB: Fix incorrect DMA allocations for local memory pool drivers Greg Kroah-Hartman
2019-12-19 18:33 ` [PATCH 5.4 02/80] mmc: block: Make card_busy_detect() a bit more generic Greg Kroah-Hartman
2019-12-19 18:33 ` [PATCH 5.4 03/80] mmc: block: Add CMD13 polling for MMC IOCTLS with R1B response Greg Kroah-Hartman
2019-12-19 18:33 ` [PATCH 5.4 04/80] mmc: core: Drop check for mmc_card_is_removable() in mmc_rescan() Greg Kroah-Hartman
2019-12-19 18:33 ` [PATCH 5.4 05/80] mmc: core: Re-work HW reset for SDIO cards Greg Kroah-Hartman
2019-12-19 18:33 ` [PATCH 5.4 06/80] PCI/switchtec: Read all 64 bits of part_event_bitmap Greg Kroah-Hartman
2019-12-19 18:33 ` [PATCH 5.4 07/80] PCI/PM: Always return devices to D0 when thawing Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 5.4 08/80] PCI: pciehp: Avoid returning prematurely from sysfs requests Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 5.4 09/80] PCI: Fix Intel ACS quirk UPDCR register address Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 5.4 10/80] PCI/MSI: Fix incorrect MSI-X masking on resume Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 5.4 11/80] PCI: Do not use bus number zero from EA capability Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 5.4 12/80] PCI: rcar: Fix missing MACCTLR register setting in initialization sequence Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 5.4 13/80] PCI: Apply Cavium ACS quirk to ThunderX2 and ThunderX3 Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 5.4 14/80] PM / QoS: Redefine FREQ_QOS_MAX_DEFAULT_VALUE to S32_MAX Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 5.4 15/80] block: fix "check bi_size overflow before merge" Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 5.4 16/80] xtensa: use MEMBLOCK_ALLOC_ANYWHERE for KASAN shadow map Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 5.4 17/80] gfs2: Multi-block allocations in gfs2_page_mkwrite Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 5.4 18/80] gfs2: fix glock reference problem in gfs2_trans_remove_revoke Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 5.4 19/80] xtensa: fix TLB sanity checker Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 5.4 20/80] xtensa: fix syscall_set_return_value Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 5.4 21/80] rpmsg: glink: Set tail pointer to 0 at end of FIFO Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 5.4 22/80] rpmsg: glink: Fix reuse intents memory leak issue Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 5.4 23/80] rpmsg: glink: Fix use after free in open_ack TIMEOUT case Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 5.4 24/80] rpmsg: glink: Put an extra reference during cleanup Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 5.4 25/80] rpmsg: glink: Fix rpmsg_register_device err handling Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 5.4 26/80] rpmsg: glink: Dont send pending rx_done during remove Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 5.4 27/80] rpmsg: glink: Free pending deferred work on remove Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 5.4 28/80] cifs: smbd: Return -EAGAIN when transport is reconnecting Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 5.4 29/80] cifs: smbd: Only queue work for error recovery on memory registration Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 5.4 30/80] cifs: smbd: Add messages on RDMA session destroy and reconnection Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 5.4 31/80] cifs: smbd: Return -EINVAL when the number of iovs exceeds SMBDIRECT_MAX_SGE Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 5.4 32/80] cifs: smbd: Return -ECONNABORTED when trasnport is not in connected state Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 5.4 33/80] cifs: Dont display RDMA transport on reconnect Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 5.4 34/80] CIFS: Respect O_SYNC and O_DIRECT flags during reconnect Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 5.4 35/80] CIFS: Close open handle after interrupted close Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 5.4 36/80] CIFS: Do not miss cancelled OPEN responses Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 5.4 37/80] CIFS: Fix NULL pointer dereference in mid callback Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 5.4 38/80] cifs: Fix retrieval of DFS referrals in cifs_mount() Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 5.4 39/80] ARM: dts: s3c64xx: Fix init order of clock providers Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 5.4 40/80] ARM: tegra: Fix FLOW_CTLR_HALT register clobbering by tegra_resume() Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 5.4 41/80] vfio/pci: call irq_bypass_unregister_producer() before freeing irq Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 5.4 42/80] dma-buf: Fix memory leak in sync_file_merge() Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 5.4 43/80] drm/panfrost: Fix a race in panfrost_ioctl_madvise() Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 5.4 44/80] drm/panfrost: Fix a BO leak in panfrost_ioctl_mmap_bo() Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 5.4 45/80] drm/panfrost: Fix a race in panfrost_gem_free_object() Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 5.4 46/80] drm/mgag200: Extract device type from flags Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 5.4 47/80] drm/mgag200: Store flags from PCI driver data in device structure Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 5.4 48/80] drm/mgag200: Add workaround for HW that does not support startadd Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 5.4 49/80] drm/mgag200: Flag all G200 SE A machines as broken wrt <startadd> Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 5.4 50/80] drm: meson: venc: cvbs: fix CVBS mode matching Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 5.4 51/80] dm mpath: remove harmful bio-based optimization Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 5.4 52/80] dm btree: increase rebalance threshold in __rebalance2() Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 5.4 53/80] dm clone metadata: Track exact changes per transaction Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 5.4 54/80] dm clone metadata: Use a two phase commit Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 5.4 55/80] dm clone: Flush destination device before committing metadata Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 5.4 56/80] dm thin metadata: Add support for a pre-commit callback Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 5.4 57/80] dm thin: Flush data device before committing metadata Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 5.4 58/80] scsi: ufs: Disable autohibern8 feature in Cadence UFS Greg Kroah-Hartman
2019-12-19 18:34 ` Greg Kroah-Hartman [this message]
2019-12-19 18:34 ` [PATCH 5.4 60/80] scsi: qla2xxx: Ignore NULL pointer in tcm_qla2xxx_free_mcmd Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 5.4 61/80] scsi: qla2xxx: Initialize free_work before flushing it Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 5.4 62/80] scsi: qla2xxx: Added support for MPI and PEP regions for ISP28XX Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 5.4 63/80] scsi: qla2xxx: Change discovery state before PLOGI Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 5.4 64/80] scsi: qla2xxx: Correctly retrieve and interpret active flash region Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 5.4 65/80] scsi: qla2xxx: Fix incorrect SFUB length used for Secure Flash Update MB Cmd Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 5.4 66/80] drm/nouveau/kms/nv50-: Call outp_atomic_check_view() before handling PBN Greg Kroah-Hartman
2019-12-19 18:34 ` [PATCH 5.4 67/80] drm/nouveau/kms/nv50-: Store the bpc were using in nv50_head_atom Greg Kroah-Hartman
2019-12-19 18:35 ` [PATCH 5.4 68/80] drm/nouveau/kms/nv50-: Limit MST BPC to 8 Greg Kroah-Hartman
2019-12-19 18:35 ` [PATCH 5.4 69/80] drm/i915/fbc: Disable fbc by default on all glk+ Greg Kroah-Hartman
2019-12-19 18:35 ` [PATCH 5.4 70/80] drm/radeon: fix r1xx/r2xx register checker for POT textures Greg Kroah-Hartman
2019-12-19 18:35 ` [PATCH 5.4 71/80] drm/dp_mst: Correct the bug in drm_dp_update_payload_part1() Greg Kroah-Hartman
2019-12-19 18:35 ` [PATCH 5.4 72/80] drm/amd/display: re-enable wait in pipelock, but add timeout Greg Kroah-Hartman
2019-12-19 18:35 ` [PATCH 5.4 73/80] drm/amd/display: add default clocks if not able to fetch them Greg Kroah-Hartman
2019-12-19 18:35 ` [PATCH 5.4 74/80] drm/amdgpu: initialize vm_inv_eng0_sem for gfxhub and mmhub Greg Kroah-Hartman
2019-12-19 18:35 ` [PATCH 5.4 75/80] drm/amdgpu: invalidate mmhub semaphore workaround in gmc9/gmc10 Greg Kroah-Hartman
2019-12-19 18:35 ` [PATCH 5.4 76/80] drm/amdgpu/gfx10: explicitly wait for cp idle after halt/unhalt Greg Kroah-Hartman
2019-12-19 18:35 ` [PATCH 5.4 77/80] drm/amdgpu/gfx10: re-init clear state buffer after gpu reset Greg Kroah-Hartman
2019-12-19 18:35 ` [PATCH 5.4 78/80] drm/i915/gvt: Fix cmd length check for MI_ATOMIC Greg Kroah-Hartman
2019-12-19 18:35 ` [PATCH 5.4 79/80] drm/amdgpu: avoid using invalidate semaphore for picasso Greg Kroah-Hartman
2019-12-19 18:35 ` [PATCH 5.4 80/80] drm/amdgpu: add invalidate semaphore limit for SRIOV and picasso in gmc9 Greg Kroah-Hartman
2019-12-20  4:30 ` [PATCH 5.4 00/80] 5.4.6-stable review shuah
2019-12-20  6:29   ` Greg Kroah-Hartman
2019-12-20  7:22 ` Naresh Kamboju
2019-12-20  8:20   ` Greg Kroah-Hartman
2019-12-20 10:30 ` Jon Hunter
2019-12-20 13:50   ` Greg Kroah-Hartman
2019-12-20 18:50 ` Guenter Roeck
2019-12-20 21:21   ` Greg Kroah-Hartman
2019-12-21  9:36 ` Jeffrin Jose

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20191219183131.918040857@linuxfoundation.org \
    --to=gregkh@linuxfoundation.org \
    --cc=bvanassche@acm.org \
    --cc=cleech@redhat.com \
    --cc=hch@lst.de \
    --cc=keith.busch@intel.com \
    --cc=lduncan@suse.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=martin.petersen@oracle.com \
    --cc=stable@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).