Linux SCSI subsystem development
 help / color / mirror / Atom feed
From: Jeff Layton <jlayton@kernel.org>
To: Sathya Prakash <sathya.prakash@broadcom.com>,
	Sreekanth Reddy	 <sreekanth.reddy@broadcom.com>,
	Suganath Prabu Subramani	 <suganath-prabu.subramani@broadcom.com>,
	Ranjan Kumar	 <ranjan.kumar@broadcom.com>,
	"James E.J. Bottomley"	 <James.Bottomley@HansenPartnership.com>,
	"Martin K. Petersen" <mkp@kernel.org>,
	 Nagalakshmi Nandigama <Nagalakshmi.Nandigama@lsi.com>,
	James Bottomley <JBottomley@Parallels.com>
Cc: Keith Busch <kbusch@kernel.org>,
	Sreekanth Reddy <Sreekanth.Reddy@lsi.com>,
		MPT-FusionLinux.pdl@broadcom.com, linux-scsi@vger.kernel.org,
		linux-kernel@vger.kernel.org
Subject: Re: [PATCH v3 00/11] scsi: mpt3sas: Fix delayed not-responding block deadlock and related bugs
Date: Wed, 18 Nov 2026 15:13:32 -0800	[thread overview]
Message-ID: <bf68a8f608bf24c9b85b3078957ffed6b9addbf1.camel@kernel.org> (raw)
In-Reply-To: <20261118-sas-abba-v3-0-5562d690cf61@kernel.org>

On Wed, 2026-11-18 at 15:09 -0800, Jeff Layton wrote:
> I posted v2 some time ago. Since then, I did a lot of adversarial review
> with AI agents. The review found a large number of bugs. Many of these
> bugs were preexisting.
> 
> This series fixes a deadlock that occurs regularly in Meta's fleet. The
> fix unblocks sdevs after a timeout, and this breaks the deadlock.
> 
> I don't do much SCSI driver work, but this problem came to me. I
> relied heavily on AI agents for this series.
> 
> If this series is acceptable, I would like to see it in linux-next
> soon.
> 
> Thanks,
> Jeff
> 
> Signed-off-by: Jeff Layton <jlayton@kernel.org>
> ---
> Changes in v3:
> - New patches 1-8 and 10 fix pre-existing bugs in the code that this
>   series changes:
>   - 1-2: fix UAFs and refcounting in the cleanup of the fw event queue.
>     Do not leave requeued events pending at teardown.
>   - 3: requeue topology events only for busy devices. Do not requeue
>     them for devices that the driver cannot add.
>   - 4-7: fix these TUR polling bugs:
>     - a pcie_device leak
>     - a lost unblock retry
>     - polling that continues after the scan is cancelled
>     - a retry count that is not per device
>   - 8: cache the missing delays that the IOC actually uses
>   - 10: do not use freed hostdata while the driver polls devices
> - Patch 9: also publish sdev->hostdata under ioc->block_lock in
>   scsih_sdev_init()
> - Patch 11:
>   - Use the PCIe missing delay (from PCIe IO Unit Page 1) for NVMe
>     devices.
>   - Arm dnr_unblock_work for the earliest deadline.
>   - Skip the timed unblock during host reset, but not during SCSI EH.
>   - Clear the deadline of an sdev under ioc->block_lock before the fw
>     event worker polls the sdev.
>   - Do not put a timed block on offline sdevs.
>   - When a hard reset cancels polling, give new deadlines to the sdevs
>     that stay blocked.
>   - Give deadlines to the sdevs that a broadcast primitive pass blocks.
>     Prevent dnr_unblock_work from unblocking them while the pass runs.
>   - A repeat DNR restarts the deadline, at most
>     MPT3SAS_DNR_BLOCK_MAX_EXTENDS times.
>   - At the next worker unblock, do a TUR check on expired sdevs.
>   - Keep the blocks that the driver makes during polling.
>   - Use system_dfl_wq instead of system_wq.
> - Link to v2: https://lore.kernel.org/r/20261102-sas-abba-v2-0-b3b9ba874b5a@kernel.org
> 
> Changes in v2:
> - New patch 1: use a new ioc->block_lock to serialize the
>   interrupt-time sdev block with the hostdata teardown in
>   scsih_sdev_destroy(). This fixes a pre-existing UAF.
> - Do the timed unblock under ioc->block_lock. This prevents a race with
>   an interrupt-time block.
> - Clear block_deadline before the fw event worker polls a returning
>   device with TURs. Then dnr_unblock_work cannot unblock the device
>   during the poll.
> - Link to v1: https://lore.kernel.org/r/20261030-sas-abba-v1-1-b8cdd714a37c@kernel.org
> 
> ---
> Jeff Layton (11):
>       scsi: mpt3sas: Don't leave requeued firmware events pending at teardown
>       scsi: mpt3sas: Hold event references while cleaning up the event queue
>       scsi: mpt3sas: Requeue topology events only for busy devices
>       scsi: mpt3sas: Fix pcie_device leak in _scsih_ublock_io_device_wait()
>       scsi: mpt3sas: Restore unblock retry in _scsih_ublock_io_device_wait()
>       scsi: mpt3sas: Stop polling devices when the unresponding scan is cancelled
>       scsi: mpt3sas: Reset the TUR retry count for each device
>       scsi: mpt3sas: Cache the missing delays the IOC actually uses
>       scsi: mpt3sas: Serialize interrupt-time sdev block with hostdata teardown
>       scsi: mpt3sas: Don't use freed hostdata while polling devices
>       scsi: mpt3sas: Time out delayed not-responding device blocks
> 
>  drivers/scsi/mpt3sas/mpt3sas_base.c  |  62 ++-
>  drivers/scsi/mpt3sas/mpt3sas_base.h  |  25 +
>  drivers/scsi/mpt3sas/mpt3sas_scsih.c | 869 +++++++++++++++++++++++++++++------
>  3 files changed, 810 insertions(+), 146 deletions(-)
> ---
> base-commit: e8017d29067fd1310a65c3494ce1a1b492adbac6
> change-id: 20261030-sas-abba-eacfcc45fe36
> 
> Best regards,

My apologies, this should have been sent as v4 of the series. I had
forgotten that I had sent a v3 a couple of weeks ago. Please treat this
series as a v4. The v3 series is here:

https://lore.kernel.org/linux-scsi/20261102-sas-abba-v2-0-b3b9ba874b5a@kernel.org/

-- 
Jeff Layton <jlayton@kernel.org>

      parent reply	other threads:[~2026-11-18 23:13 UTC|newest]

Thread overview: 17+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-18 23:09 [PATCH v3 00/11] scsi: mpt3sas: Fix delayed not-responding block deadlock and related bugs Jeff Layton
2026-11-18 23:09 ` [PATCH v3 01/11] scsi: mpt3sas: Don't leave requeued firmware events pending at teardown Jeff Layton
2026-11-18 23:33   ` sashiko-bot
2026-11-18 23:09 ` [PATCH v3 02/11] scsi: mpt3sas: Hold event references while cleaning up the event queue Jeff Layton
2026-11-18 23:10 ` [PATCH v3 03/11] scsi: mpt3sas: Requeue topology events only for busy devices Jeff Layton
2026-11-18 23:10 ` [PATCH v3 04/11] scsi: mpt3sas: Fix pcie_device leak in _scsih_ublock_io_device_wait() Jeff Layton
2026-11-18 23:10 ` [PATCH v3 05/11] scsi: mpt3sas: Restore unblock retry " Jeff Layton
2026-11-18 23:10 ` [PATCH v3 06/11] scsi: mpt3sas: Stop polling devices when the unresponding scan is cancelled Jeff Layton
2026-11-18 23:27   ` sashiko-bot
2026-11-18 23:10 ` [PATCH v3 07/11] scsi: mpt3sas: Reset the TUR retry count for each device Jeff Layton
2026-11-18 23:10 ` [PATCH v3 08/11] scsi: mpt3sas: Cache the missing delays the IOC actually uses Jeff Layton
2026-11-18 23:10 ` [PATCH v3 09/11] scsi: mpt3sas: Serialize interrupt-time sdev block with hostdata teardown Jeff Layton
2026-11-18 23:10 ` [PATCH v3 10/11] scsi: mpt3sas: Don't use freed hostdata while polling devices Jeff Layton
2026-11-18 23:23   ` sashiko-bot
2026-11-18 23:10 ` [PATCH v3 11/11] scsi: mpt3sas: Time out delayed not-responding device blocks Jeff Layton
2026-11-18 23:33   ` sashiko-bot
2026-11-18 23:13 ` Jeff Layton [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=bf68a8f608bf24c9b85b3078957ffed6b9addbf1.camel@kernel.org \
    --to=jlayton@kernel.org \
    --cc=JBottomley@Parallels.com \
    --cc=James.Bottomley@HansenPartnership.com \
    --cc=MPT-FusionLinux.pdl@broadcom.com \
    --cc=Nagalakshmi.Nandigama@lsi.com \
    --cc=Sreekanth.Reddy@lsi.com \
    --cc=kbusch@kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-scsi@vger.kernel.org \
    --cc=mkp@kernel.org \
    --cc=ranjan.kumar@broadcom.com \
    --cc=sathya.prakash@broadcom.com \
    --cc=sreekanth.reddy@broadcom.com \
    --cc=suganath-prabu.subramani@broadcom.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox