From: Jeff Layton <jlayton@kernel.org>
To: Sathya Prakash <sathya.prakash@broadcom.com>,
Sreekanth Reddy <sreekanth.reddy@broadcom.com>,
Suganath Prabu Subramani <suganath-prabu.subramani@broadcom.com>,
Ranjan Kumar <ranjan.kumar@broadcom.com>,
"James E.J. Bottomley" <James.Bottomley@HansenPartnership.com>,
"Martin K. Petersen" <mkp@kernel.org>,
Nagalakshmi Nandigama <Nagalakshmi.Nandigama@lsi.com>,
James Bottomley <JBottomley@Parallels.com>
Cc: Keith Busch <kbusch@kernel.org>,
Sreekanth Reddy <Sreekanth.Reddy@lsi.com>,
MPT-FusionLinux.pdl@broadcom.com, linux-scsi@vger.kernel.org,
linux-kernel@vger.kernel.org
Subject: Re: [PATCH v3 00/11] scsi: mpt3sas: Fix delayed not-responding block deadlock and related bugs
Date: Wed, 18 Nov 2026 15:13:32 -0800 [thread overview]
Message-ID: <bf68a8f608bf24c9b85b3078957ffed6b9addbf1.camel@kernel.org> (raw)
In-Reply-To: <20261118-sas-abba-v3-0-5562d690cf61@kernel.org>
On Wed, 2026-11-18 at 15:09 -0800, Jeff Layton wrote:
> I posted v2 some time ago. Since then, I did a lot of adversarial review
> with AI agents. The review found a large number of bugs. Many of these
> bugs were preexisting.
>
> This series fixes a deadlock that occurs regularly in Meta's fleet. The
> fix unblocks sdevs after a timeout, and this breaks the deadlock.
>
> I don't do much SCSI driver work, but this problem came to me. I
> relied heavily on AI agents for this series.
>
> If this series is acceptable, I would like to see it in linux-next
> soon.
>
> Thanks,
> Jeff
>
> Signed-off-by: Jeff Layton <jlayton@kernel.org>
> ---
> Changes in v3:
> - New patches 1-8 and 10 fix pre-existing bugs in the code that this
> series changes:
> - 1-2: fix UAFs and refcounting in the cleanup of the fw event queue.
> Do not leave requeued events pending at teardown.
> - 3: requeue topology events only for busy devices. Do not requeue
> them for devices that the driver cannot add.
> - 4-7: fix these TUR polling bugs:
> - a pcie_device leak
> - a lost unblock retry
> - polling that continues after the scan is cancelled
> - a retry count that is not per device
> - 8: cache the missing delays that the IOC actually uses
> - 10: do not use freed hostdata while the driver polls devices
> - Patch 9: also publish sdev->hostdata under ioc->block_lock in
> scsih_sdev_init()
> - Patch 11:
> - Use the PCIe missing delay (from PCIe IO Unit Page 1) for NVMe
> devices.
> - Arm dnr_unblock_work for the earliest deadline.
> - Skip the timed unblock during host reset, but not during SCSI EH.
> - Clear the deadline of an sdev under ioc->block_lock before the fw
> event worker polls the sdev.
> - Do not put a timed block on offline sdevs.
> - When a hard reset cancels polling, give new deadlines to the sdevs
> that stay blocked.
> - Give deadlines to the sdevs that a broadcast primitive pass blocks.
> Prevent dnr_unblock_work from unblocking them while the pass runs.
> - A repeat DNR restarts the deadline, at most
> MPT3SAS_DNR_BLOCK_MAX_EXTENDS times.
> - At the next worker unblock, do a TUR check on expired sdevs.
> - Keep the blocks that the driver makes during polling.
> - Use system_dfl_wq instead of system_wq.
> - Link to v2: https://lore.kernel.org/r/20261102-sas-abba-v2-0-b3b9ba874b5a@kernel.org
>
> Changes in v2:
> - New patch 1: use a new ioc->block_lock to serialize the
> interrupt-time sdev block with the hostdata teardown in
> scsih_sdev_destroy(). This fixes a pre-existing UAF.
> - Do the timed unblock under ioc->block_lock. This prevents a race with
> an interrupt-time block.
> - Clear block_deadline before the fw event worker polls a returning
> device with TURs. Then dnr_unblock_work cannot unblock the device
> during the poll.
> - Link to v1: https://lore.kernel.org/r/20261030-sas-abba-v1-1-b8cdd714a37c@kernel.org
>
> ---
> Jeff Layton (11):
> scsi: mpt3sas: Don't leave requeued firmware events pending at teardown
> scsi: mpt3sas: Hold event references while cleaning up the event queue
> scsi: mpt3sas: Requeue topology events only for busy devices
> scsi: mpt3sas: Fix pcie_device leak in _scsih_ublock_io_device_wait()
> scsi: mpt3sas: Restore unblock retry in _scsih_ublock_io_device_wait()
> scsi: mpt3sas: Stop polling devices when the unresponding scan is cancelled
> scsi: mpt3sas: Reset the TUR retry count for each device
> scsi: mpt3sas: Cache the missing delays the IOC actually uses
> scsi: mpt3sas: Serialize interrupt-time sdev block with hostdata teardown
> scsi: mpt3sas: Don't use freed hostdata while polling devices
> scsi: mpt3sas: Time out delayed not-responding device blocks
>
> drivers/scsi/mpt3sas/mpt3sas_base.c | 62 ++-
> drivers/scsi/mpt3sas/mpt3sas_base.h | 25 +
> drivers/scsi/mpt3sas/mpt3sas_scsih.c | 869 +++++++++++++++++++++++++++++------
> 3 files changed, 810 insertions(+), 146 deletions(-)
> ---
> base-commit: e8017d29067fd1310a65c3494ce1a1b492adbac6
> change-id: 20261030-sas-abba-eacfcc45fe36
>
> Best regards,
My apologies, this should have been sent as v4 of the series. I had
forgotten that I had sent a v3 a couple of weeks ago. Please treat this
series as a v4. The v3 series is here:
https://lore.kernel.org/linux-scsi/20261102-sas-abba-v2-0-b3b9ba874b5a@kernel.org/
--
Jeff Layton <jlayton@kernel.org>
prev parent reply other threads:[~2026-11-18 23:13 UTC|newest]
Thread overview: 17+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-18 23:09 [PATCH v3 00/11] scsi: mpt3sas: Fix delayed not-responding block deadlock and related bugs Jeff Layton
2026-11-18 23:09 ` [PATCH v3 01/11] scsi: mpt3sas: Don't leave requeued firmware events pending at teardown Jeff Layton
2026-11-18 23:33 ` sashiko-bot
2026-11-18 23:09 ` [PATCH v3 02/11] scsi: mpt3sas: Hold event references while cleaning up the event queue Jeff Layton
2026-11-18 23:10 ` [PATCH v3 03/11] scsi: mpt3sas: Requeue topology events only for busy devices Jeff Layton
2026-11-18 23:10 ` [PATCH v3 04/11] scsi: mpt3sas: Fix pcie_device leak in _scsih_ublock_io_device_wait() Jeff Layton
2026-11-18 23:10 ` [PATCH v3 05/11] scsi: mpt3sas: Restore unblock retry " Jeff Layton
2026-11-18 23:10 ` [PATCH v3 06/11] scsi: mpt3sas: Stop polling devices when the unresponding scan is cancelled Jeff Layton
2026-11-18 23:27 ` sashiko-bot
2026-11-18 23:10 ` [PATCH v3 07/11] scsi: mpt3sas: Reset the TUR retry count for each device Jeff Layton
2026-11-18 23:10 ` [PATCH v3 08/11] scsi: mpt3sas: Cache the missing delays the IOC actually uses Jeff Layton
2026-11-18 23:10 ` [PATCH v3 09/11] scsi: mpt3sas: Serialize interrupt-time sdev block with hostdata teardown Jeff Layton
2026-11-18 23:10 ` [PATCH v3 10/11] scsi: mpt3sas: Don't use freed hostdata while polling devices Jeff Layton
2026-11-18 23:23 ` sashiko-bot
2026-11-18 23:10 ` [PATCH v3 11/11] scsi: mpt3sas: Time out delayed not-responding device blocks Jeff Layton
2026-11-18 23:33 ` sashiko-bot
2026-11-18 23:13 ` Jeff Layton [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=bf68a8f608bf24c9b85b3078957ffed6b9addbf1.camel@kernel.org \
--to=jlayton@kernel.org \
--cc=JBottomley@Parallels.com \
--cc=James.Bottomley@HansenPartnership.com \
--cc=MPT-FusionLinux.pdl@broadcom.com \
--cc=Nagalakshmi.Nandigama@lsi.com \
--cc=Sreekanth.Reddy@lsi.com \
--cc=kbusch@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-scsi@vger.kernel.org \
--cc=mkp@kernel.org \
--cc=ranjan.kumar@broadcom.com \
--cc=sathya.prakash@broadcom.com \
--cc=sreekanth.reddy@broadcom.com \
--cc=suganath-prabu.subramani@broadcom.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.