[PATCH] scsi: libsas: fix HA resume deadlock and hisi_sas disk-wake race

[PATCH] scsi: libsas: fix HA resume deadlock and hisi_sas disk-wake race

[Xingui Yang]

Commit fbefe22811c3140 ("scsi: libsas: Don't always drain event
workqueue for HA resume") introduced sas_resume_ha_no_sync() to avoid a
deadlock: the PHYE_RESUME_TIMEOUT handler, running on the HA event
workqueue, calls sas_deform_port() -> sas_destruct_devices(), which
removes SCSI devices and waits for the host to become runtime-active
through the device_link established by hisi_sas. But the host cannot
resume until sas_resume_ha() -> sas_drain_work() returns, and the drain
is blocked on that very handler.

However skipping the drain reintroduces a race: hisi_sas returns from
resume before all PHY UP work and libsas discovery work finish. The
controller may then autosuspend while disks are still waking up. The
disks issue IO to a suspended controller, the IO fails, and the disks
get disabled.

Fix the deadlock at its source by moving the PHYE_RESUME_TIMEOUT
notification to after sas_drain_work(). By then the host resume is
about to complete, so device removal through device_link no longer
blocks on the resume and the cycle is broken.

With the deadlock gone, restore sas_resume_ha() (the draining variant)
in hisi_sas and remove sas_resume_ha_no_sync().

Additionally flush the hisi_sas private workqueue after sas_resume_ha().
sas_drain_work() only drains the libsas event and discovery workqueues;
it cannot wait for hisi_sas_phyup_pm_work(), which runs on the driver's
own workqueue and holds a runtime PM reference taken in the phy_up ISR.
Without this flush, the controller may autosuspend before all phyup PM
works complete.

Fixes: fbefe22811c3140 ("scsi: libsas: Don't always drain event workqueue for HA resume")
Signed-off-by: Xingui Yang <yangxingui@huawei.com>
---
 drivers/scsi/hisi_sas/hisi_sas_v3_hw.c | 10 ++-------
 drivers/scsi/libsas/sas_init.c         | 29 +++++++++++++-------------
 include/scsi/libsas.h                  |  1 -
 3 files changed, 16 insertions(+), 24 deletions(-)

diff --git a/drivers/scsi/hisi_sas/hisi_sas_v3_hw.c b/drivers/scsi/hisi_sas/hisi_sas_v3_hw.c
index 213d5b5dea94..901f508e8be7 100644
--- a/drivers/scsi/hisi_sas/hisi_sas_v3_hw.c
+++ b/drivers/scsi/hisi_sas/hisi_sas_v3_hw.c
@@ -5262,14 +5262,8 @@ static int _resume_v3_hw(struct device *device)
 	}
 	phys_init_v3_hw(hisi_hba);
 
-	/*
-	 * If a directly-attached disk is removed during suspend, a deadlock
-	 * may occur, as the PHYE_RESUME_TIMEOUT processing will require the
-	 * hisi_hba->device to be active, which can only happen when resume
-	 * completes. So don't wait for the HA event workqueue to drain upon
-	 * resume.
-	 */
-	sas_resume_ha_no_sync(sha);
+	sas_resume_ha(sha);
+	flush_workqueue(hisi_hba->wq);
 	clear_bit(HISI_SAS_RESETTING_BIT, &hisi_hba->flags);
 
 	dev_warn(dev, "end of resuming controller\n");
diff --git a/drivers/scsi/libsas/sas_init.c b/drivers/scsi/libsas/sas_init.c
index 0bec236f0fb5..61348d4167b4 100644
--- a/drivers/scsi/libsas/sas_init.c
+++ b/drivers/scsi/libsas/sas_init.c
@@ -410,7 +410,7 @@ static void sas_resume_insert_broadcast_ha(struct sas_ha_struct *ha)
 	}
 }
 
-static void _sas_resume_ha(struct sas_ha_struct *ha, bool drain)
+static void _sas_resume_ha(struct sas_ha_struct *ha)
 {
 	const unsigned long tmo = msecs_to_jiffies(25000);
 	int i;
@@ -426,6 +426,18 @@ static void _sas_resume_ha(struct sas_ha_struct *ha, bool drain)
 		dev_info(ha->dev, "waiting up to 25 seconds for %d phy%s to resume\n",
 			 i, i > 1 ? "s" : "");
 	wait_event_timeout(ha->eh_wait_q, phys_suspended(ha) == 0, tmo);
+
+	scsi_unblock_requests(ha->shost);
+	sas_drain_work(ha);
+
+	/* Send PHYE_RESUME_TIMEOUT after sas_drain_work(). The handler
+	 * calls sas_deform_port() -> sas_destruct_devices(), which removes
+	 * SCSI devices and, for LLDDs using device_link() PM sync, waits
+	 * for the host to be runtime-active. Sending it before the drain
+	 * would deadlock: the drain waits for the handler, the handler
+	 * waits for host resume, and host resume waits for the drain to
+	 * finish.
+	 */
 	for (i = 0; i < ha->num_phys; i++) {
 		struct asd_sas_phy *phy = ha->sas_phy[i];
 
@@ -436,12 +448,6 @@ static void _sas_resume_ha(struct sas_ha_struct *ha, bool drain)
 		}
 	}
 
-	/* all phys are back up or timed out, turn on i/o so we can
-	 * flush out disks that did not return
-	 */
-	scsi_unblock_requests(ha->shost);
-	if (drain)
-		sas_drain_work(ha);
 	clear_bit(SAS_HA_RESUMING, &ha->state);
 
 	sas_queue_deferred_work(ha);
@@ -453,17 +459,10 @@ static void _sas_resume_ha(struct sas_ha_struct *ha, bool drain)
 
 void sas_resume_ha(struct sas_ha_struct *ha)
 {
-	_sas_resume_ha(ha, true);
+	_sas_resume_ha(ha);
 }
 EXPORT_SYMBOL(sas_resume_ha);
 
-/* A no-sync variant, which does not call sas_drain_ha(). */
-void sas_resume_ha_no_sync(struct sas_ha_struct *ha)
-{
-	_sas_resume_ha(ha, false);
-}
-EXPORT_SYMBOL(sas_resume_ha_no_sync);
-
 void sas_suspend_ha(struct sas_ha_struct *ha)
 {
 	int i;
diff --git a/include/scsi/libsas.h b/include/scsi/libsas.h
index 163f23c92b41..36d4cb567837 100644
--- a/include/scsi/libsas.h
+++ b/include/scsi/libsas.h
@@ -680,7 +680,6 @@ extern int sas_register_ha(struct sas_ha_struct *);
 extern int sas_unregister_ha(struct sas_ha_struct *);
 extern void sas_prep_resume_ha(struct sas_ha_struct *sas_ha);
 extern void sas_resume_ha(struct sas_ha_struct *sas_ha);
-extern void sas_resume_ha_no_sync(struct sas_ha_struct *sas_ha);
 extern void sas_suspend_ha(struct sas_ha_struct *sas_ha);
 
 int sas_phy_reset(struct sas_phy *phy, int hard_reset);
-- 
2.43.0

Re: [PATCH] scsi: libsas: fix HA resume deadlock and hisi_sas disk-wake race

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

New issues:
- [High] The `flush_workqueue(hisi_hba->wq)` placement is ineffective, failing to prevent the autosuspend race for newly discovered PHYs.

Pre-existing issues:
- [High] PHY down events are silently dropped during resume because `HISI_SAS_RESETTING_BIT` remains set, permanently leaving libsas state out of sync.
--

commit ce8bc2a47295d4675bf4efa43c713662a8cd5696
Author: Xingui Yang <yangxingui@huawei.com>

scsi: libsas: fix HA resume deadlock and hisi_sas disk-wake race

This patch resolves a deadlock during HA resume by moving the
PHYE_RESUME_TIMEOUT notification to after sas_drain_work(). It restores
the draining sas_resume_ha() behavior in hisi_sas and introduces a
flush_workqueue() to prevent an autosuspend race.

> diff --git a/drivers/scsi/hisi_sas/hisi_sas_v3_hw.c b/drivers/scsi/hisi_sas/hisi_sas_v3_hw.c
> index 213d5b5dea94f..901f508e8be77 100644
> --- a/drivers/scsi/hisi_sas/hisi_sas_v3_hw.c
> +++ b/drivers/scsi/hisi_sas/hisi_sas_v3_hw.c
> @@ -5262,14 +5262,8 @@ static int _resume_v3_hw(struct device *device)
>  	}
>  	phys_init_v3_hw(hisi_hba);
>  
> -	/*
> -	 * If a directly-attached disk is removed during suspend, a deadlock
> -	 * may occur, as the PHYE_RESUME_TIMEOUT processing will require the
> -	 * hisi_hba->device to be active, which can only happen when resume
> -	 * completes. So don't wait for the HA event workqueue to drain upon
> -	 * resume.
> -	 */
> -	sas_resume_ha_no_sync(sha);
> +	sas_resume_ha(sha);
> +	flush_workqueue(hisi_hba->wq);

[Severity: High]
Does this flush_workqueue() placement miss newly queued discovery events?

sas_resume_ha() drains the libsas event_q and disco_q. By flushing hisi_hba->wq
after this drain, any hisi_sas_phyup_pm_work items on hisi_hba->wq will queue
PORTE_BYTES_DMAED events to event_q that bypass the drain.

Since hisi_sas_phyup_pm_work then calls pm_runtime_put_sync(), its PM reference
is dropped. When _resume_v3_hw() returns, the system PM reference is also
dropped, allowing the controller to autosuspend while the missed discovery
works execute asynchronously and attempt to send IO to a suspended controller.

Should the flush be placed before sas_resume_ha() so the generated events are
caught and synchronously processed by sas_drain_work()?

>  	clear_bit(HISI_SAS_RESETTING_BIT, &hisi_hba->flags);

[Severity: High]
This is a pre-existing issue, but are PHY down events silently dropped during
resume because HISI_SAS_RESETTING_BIT remains set?

In _resume_v3_hw(), HISI_SAS_RESETTING_BIT is kept active across the
sas_resume_ha() call. If a PHY loses its link during this time, the hardware
interrupt handler drops the event:

drivers/scsi/hisi_sas/hisi_sas_main.c:hisi_sas_phy_down() {
		if (test_bit(HISI_SAS_RESETTING_BIT, &hisi_hba->flags) ||
		    phy->in_reset) {
			dev_info(dev, "ignore flutter phy%d down\n", phy_no);
			return;
		}
}

When HISI_SAS_RESETTING_BIT is finally cleared here, it does not call
hisi_sas_rescan_topology() to recover dropped events. Could this permanently
leave libsas state out of sync if a PHY disconnects during the resume wait?

>  
>  	dev_warn(dev, "end of resuming controller\n");

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260630065157.421203-1-yangxingui@huawei.com?part=1