Linux SCSI subsystem development
 help / color / mirror / Atom feed
From: Sasha Levin <sashal@kernel.org>
To: patches@lists.linux.dev, stable@vger.kernel.org
Cc: Zizhi Wo <wozizhi@huawei.com>,
	Bart Van Assche <bvanassche@acm.org>,
	"Martin K. Petersen (Oracle)" <mkp@kernel.org>,
	Sasha Levin <sashal@kernel.org>,
	James.Bottomley@HansenPartnership.com,
	linux-scsi@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: [PATCH AUTOSEL 6.18-6.1] scsi: core: Do not block on tag allocation in scsi_eh_lock_door()
Date: Mon, 31 Aug 2026 09:20:51 -0400	[thread overview]
Message-ID: <20260831133314.4125787-23-sashal@kernel.org> (raw)
In-Reply-To: <20260831133314.4125787-1-sashal@kernel.org>

From: Zizhi Wo <wozizhi@huawei.com>

[ Upstream commit 732cb6bb37fd26863d5786522fb1997e7f5865b4 ]

scsi_eh_lock_door() is called from scsi_restart_operations() while the
host is still in the SHOST_RECOVERY state, i.e. before the host is
switched back to SHOST_RUNNING and scsi_run_host_queues() restarts the
queues. It allocates a request via scsi_alloc_request() with no flags,
so blk_mq_get_tag() may block waiting for a free sched tag when all tags
are already in use.

Those tags can be held by commands that were just requeued by
scsi_eh_flush_done_q() during error handling. Such commands cannot be
dispatched until the host leaves SHOST_RECOVERY and
scsi_run_host_queues() is called - which only happens *after*
scsi_eh_lock_door() returns.

This forms a circular dependency:

  - scsi_eh_lock_door(), running in the SCSI error handler thread, waits
    for a sched tag held by a requeued command;

  - the requeued command cannot complete and release its sched tag until
    the error handler thread leaves scsi_restart_operations() and restart
    the queues.

For devices with a single driver tag (e.g. USB storage) it is a
guaranteed deadlock and I/O that can never be submitted. This problem
has also been reproduced in our environment.

Locking the door is a best-effort operation, and scsi_eh_lock_door()
already returns silently when the request allocation fails. Pass
BLK_MQ_REQ_NOWAIT to scsi_alloc_request() so the allocation fails
instead of blocking when no tag is available. This breaks the circular
dependency and allows the error handler to finish restarting the queues,
after which the pending commands are dispatched normally.

Signed-off-by: Zizhi Wo <wozizhi@huawei.com>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Link: https://patch.msgid.link/20260723041238.1584632-1-wozizhi@huaweicloud.com
Signed-off-by: Martin K. Petersen (Oracle) <mkp@kernel.org>
Signed-off-by: Sasha Levin <sashal@kernel.org>
---

LLM Generated explanations, may be completely bogus:

## Phase 1: Commit Message Forensics

**Step 1.1 — Subject line**
- Record: `[scsi: core]` `[prevent/block]` — Do not block on tag
  allocation in `scsi_eh_lock_door()`.

**Step 1.2 — Tags**
- Record:
  - `Signed-off-by`: Zizhi Wo `<wozizhi@huawei.com>` (author)
  - `Reviewed-by`: Bart Van Assche `<bvanassche@acm.org>` (SCSI
    maintainer)
  - `Link`: https://patch.msgid.link/20260723041238.1584632-1-
    wozizhi@huaweicloud.com
  - `Signed-off-by`: Martin K. Petersen (Oracle) `<mkp@kernel.org>`
    (committer)
  - No `Fixes:`, `Reported-by:`, `Cc: stable@vger.kernel.org`, or
    `Tested-by:` tags.
  - Notable: maintainer review present; author reports reproduction in
    their environment.

**Step 1.3 — Body analysis**
- Record:
  - **Bug**: Circular dependency during SCSI error recovery —
    `scsi_eh_lock_door()` blocks in `blk_mq_get_tag()` while host is
    still in `SHOST_RECOVERY`, but tags are held by commands requeued by
    `scsi_eh_flush_done_q()` that cannot dispatch until
    `scsi_run_host_queues()` runs *after* `scsi_eh_lock_door()` returns.
  - **Symptom**: Guaranteed deadlock on single-tag devices (e.g. USB
    storage); I/O permanently stuck.
  - **Root cause**: Blocking tag allocation during recovery restart.
  - **Fix**: Pass `BLK_MQ_REQ_NOWAIT` so allocation fails fast; door-
    lock is already best-effort (silent return on failure).

**Step 1.4 — Hidden bug fix?**
- Record: No — this is an explicit deadlock fix, not disguised cleanup.

---

## Phase 2: Diff Analysis

**Step 2.1 — Inventory**
- Record: 1 file (`drivers/scsi/scsi_error.c`), 1 line changed (+1/-1).
  Function: `scsi_eh_lock_door()`. Scope: single-file surgical fix.

**Step 2.2 — Code flow change**
- Record:
  - **Before**: `scsi_alloc_request(..., 0)` → `blk_mq_get_tag()` may
    block indefinitely waiting for a sched tag.
  - **After**: `scsi_alloc_request(..., BLK_MQ_REQ_NOWAIT)` → returns
    `ERR_PTR(-EWOULDBLOCK)` immediately when no tag is available;
    existing `IS_ERR(req) return;` path handles it.

**Step 2.3 — Bug mechanism**
- Record: **Deadlock / lock ordering** — error-handler thread blocks on
  tag allocation while holding recovery state that prevents requeued
  commands from releasing tags. Category: synchronization deadlock in EH
  restart path.

**Step 2.4 — Fix quality**
- Record: Obviously correct. Minimal one-line change. Low regression
  risk — door locking is documented as best-effort; failure path already
  existed. Reviewed by SCSI maintainer.

---

## Phase 3: Git History Investigation

**Step 3.1 — Blame**
- Record: `scsi_eh_lock_door()` exists since kernel 2.6 (commit
  `1da177e`). Blocking allocation introduced in `68ec3b819a5d6` ("scsi:
  add a scsi_alloc_request helper", 2021-10-22, first in v5.16). Present
  in this tree.

**Step 3.2 — Fixes: tag**
- Record: N/A — no `Fixes:` tag. Buggy behavior introduced by
  `68ec3b819a5d6`, which is an ancestor of HEAD.

**Step 3.3 — Related file history**
- Record: Recent `scsi_error.c` changes in 6.18.y include EH wake
  reliability fixes (`c7a1509123720`, `219f009ebfd1e`) but nothing
  addressing this deadlock. Standalone fix, not part of a series.

**Step 3.4 — Author context**
- Record: Zizhi Wo is not a regular SCSI maintainer (other work in blk-
  throttle, xfs, tty). This is a targeted bug report/fix.

**Step 3.5 — Dependencies**
- Record: None. `BLK_MQ_REQ_NOWAIT`, `scsi_alloc_request()` flags
  parameter, and `IS_ERR` handling all exist in this tree. `git apply
  --check` on the patch succeeds cleanly.

---

## Phase 4: Mailing List and External Research

**Step 4.1 — Original discussion**
- Record: `b4 dig -c 732cb6bb37fd2` found v1 only at https://patch.msgid
  .link/20260723041238.1584632-1-wozizhi@huaweicloud.com. Bart Van
  Assche replied with `Reviewed-by`. No NAKs or objections in thread.

**Step 4.2 — Reviewers**
- Record: `b4 dig -w` shows CC to James Bottomley, Martin Petersen, Bart
  Van Assche, linux-scsi, linux-kernel. Appropriate maintainers
  included.

**Step 4.3 — Bug report**
- Record: Author states "reproduced in our environment." No
  syzbot/bugzilla link. Real-world reproduction claimed.

**Step 4.4 — Series context**
- Record: `b4 dig -a` shows single v1 patch — standalone, no series
  dependencies.

**Step 4.5 — Stable list discussion**
- Record: No stable-specific discussion found in mbox thread (no "Cc:
  stable" mentions). Absence is neutral per instructions.

---

## Phase 5: Code Semantic Analysis

**Step 5.1 — Key functions**
- Record: `scsi_eh_lock_door()`, called from
  `scsi_restart_operations()`, which is called from
  `scsi_error_handler()`.

**Step 5.2 — Callers**
- Record: `scsi_restart_operations()` ← `scsi_error_handler()` (EH
  kernel thread, process context). Triggered on SCSI I/O
  failures/timeouts — common path for all SCSI block devices.

**Step 5.3 — Callees**
- Record: `scsi_alloc_request()` → `blk_mq_alloc_request()` →
  `blk_mq_get_tag()`. On failure with NOWAIT: `ERR_PTR(-EWOULDBLOCK)`.
  On success: `blk_execute_rq_nowait()`.

**Step 5.4 — Reachability**
- Record: Trigger requires SCSI EH after device reset with door locked
  (`sdev->was_reset && sdev->locked`). Verified in
  `scsi_restart_operations()` at line 2200. Host stays in
  `SHOST_RECOVERY` until line 2215
  (`scsi_host_set_state(SHOST_RUNNING)`), which is *after*
  `scsi_eh_lock_door()`. Requeued commands from `scsi_eh_flush_done_q()`
  use `blk_mq_requeue_request(..., !scsi_host_in_recovery(...))` —
  during recovery, `kick_requeue_list` is false, so they cannot
  dispatch. Reachable from normal block I/O error paths.

**Step 5.5 — Similar patterns**
- Record: Historical USB SCSI EH deadlocks fixed in `7daf480483e60` and
  `c69e6f812bab0` (same file, same subsystem concern).
  `BLK_MQ_REQ_NOWAIT` already used elsewhere in SCSI (`sg.c` documents
  why it avoids NOWAIT for userspace). This is the correct use case for
  NOWAIT.

---

## Phase 6: Cross-Referencing Against Local Tree

**Step 6.1 — Buggy code exists?**
- Record: **YES.** Local tree is `v6.18.44` (`linux-6.18.y`). Current
  code at line 2160 still has `scsi_alloc_request(sdev->request_queue,
  REQ_OP_DRV_IN, 0)`. Fix commit `732cb6bb37fd2` is on mainline but
  **not** in this tree.

**Step 6.2 — Backport complications**
- Record: Clean apply confirmed via `git apply --check`. No conflicts
  expected.

**Step 6.3 — Related fixes already present?**
- Record: No — `git log -S"BLK_MQ_REQ_NOWAIT" --
  drivers/scsi/scsi_error.c` returns nothing. Fix not yet backported.

---

## Phase 7: Subsystem and Maintainer Context

**Step 7.1 — Subsystem criticality**
- Record: `drivers/scsi/scsi_error.c` — SCSI core error handling.
  **CORE/IMPORTANT** — affects all SCSI block storage (disks, USB
  storage, optical drives).

**Step 7.2 — Subsystem activity**
- Record: Actively maintained in 6.18.y with recent EH fixes
  (`c7a1509123720`, etc.).

---

## Phase 8: Impact and Risk Assessment

**Step 8.1 — Who is affected**
- Record: Users of SCSI block devices undergoing error recovery,
  especially USB storage (single tag) and optical media with door-lock
  after reset. Universal for SCSI-enabled kernels.

**Step 8.2 — Trigger conditions**
- Record: SCSI EH after reset on a device with locked door, with
  requeued commands occupying all queue tags. Reproducible per author;
  guaranteed on single-tag devices. Unprivileged users can trigger via
  normal I/O to removable media.

**Step 8.3 — Failure mode severity**
- Record: **CRITICAL** — permanent deadlock of EH thread; all I/O to
  affected device/host stuck indefinitely.

**Step 8.4 — Risk-benefit**
- Record: Benefit **very high** (prevents unrecoverable I/O hang). Risk
  **very low** (1-line change, existing failure path, best-effort
  operation). Strong positive ratio.

---

## Phase 9: Final Synthesis

**Evidence FOR backport:**
- Real, reproducible deadlock in SCSI EH restart path
- CRITICAL severity (permanent I/O hang)
- One-line, obviously correct fix
- Reviewed by SCSI maintainer (Bart Van Assche)
- Applies cleanly to 6.18.44
- Bug present since v5.16 (`scsi_alloc_request` migration)
- No dependencies or new APIs
- Historical precedent for USB/EH deadlock fixes in stable

**Evidence AGAINST backport:**
- None significant. Door-lock skip on tag exhaustion is acceptable
  (best-effort by design).

**Stable rules checklist:**
1. Obviously correct and tested? **PASS** — maintainer reviewed; author
   reproduced bug.
2. Fixes real bug? **PASS** — deadlock with concrete trigger.
3. Important issue? **PASS** — CRITICAL deadlock.
4. Small and contained? **PASS** — 1 line, 1 file.
5. No new features/APIs? **PASS**.
6. Can apply to local tree? **PASS** — clean apply verified.

**Exception category:** N/A (not a quirk/DT/build fix — standard bug
fix).

---

## Verification

- [Phase 1] `git show 732cb6bb37fd2`: parsed full commit message and
  tags
- [Phase 2] Diff: confirmed single-line `BLK_MQ_REQ_NOWAIT` change in
  `scsi_eh_lock_door()`
- [Phase 3] `git describe HEAD`: v6.18.44; `git blame` line 2160:
  blocking call from `68ec3b819a5d6` (v5.16)
- [Phase 3] `git merge-base --is-ancestor 68ec3b819a5d6 HEAD`: confirmed
  buggy code ancestor present
- [Phase 3] `git merge-base --is-ancestor 732cb6bb37fd2 HEAD`: fix NOT
  in tree
- [Phase 3] `git apply --check`: patch applies cleanly
- [Phase 4] `b4 dig -c 732cb6bb37fd2`: found lore thread
- [Phase 4] `b4 dig -w`: maintainers CC'd (Bottomley, Petersen, Van
  Assche)
- [Phase 4] `b4 dig -a`: single v1 patch, no series
- [Phase 4] mbox: Bart Van Assche `Reviewed-by`; no NAKs, no stable
  nomination
- [Phase 5] Read `scsi_restart_operations()`, `scsi_unjam_host()`,
  `scsi_eh_flush_done_q()`, `__scsi_queue_insert()`: confirmed recovery-
  state ordering and requeue behavior
- [Phase 5] Read `scsi_host_in_recovery()` in
  `include/scsi/scsi_host.h`: checks `SHOST_RECOVERY` etc.
- [Phase 5] Read `blk_mq_alloc_request()` / `blk_mq_get_tag()`: NOWAIT
  returns `ERR_PTR(-EWOULDBLOCK)` on no tag
- [Phase 6] Grep current tree line 2160: still uses flags `0` (buggy)
- [Phase 8] Confirmed `sdev->was_reset` set in
  `__scsi_report_device_reset()` during EH reset path

This commit fixes a real SCSI error-handler deadlock that can
permanently hang I/O — especially on single-tag USB storage — and the
fix is a minimal, reviewed, cleanly-applicable one-liner appropriate for
the 6.18.y stable tree.

**YES**

 drivers/scsi/scsi_error.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/scsi/scsi_error.c b/drivers/scsi/scsi_error.c
index 13d46b6e0359e..857c2acc99645 100644
--- a/drivers/scsi/scsi_error.c
+++ b/drivers/scsi/scsi_error.c
@@ -2157,7 +2157,7 @@ static void scsi_eh_lock_door(struct scsi_device *sdev)
 	struct scsi_cmnd *scmd;
 	struct request *req;
 
-	req = scsi_alloc_request(sdev->request_queue, REQ_OP_DRV_IN, 0);
+	req = scsi_alloc_request(sdev->request_queue, REQ_OP_DRV_IN, BLK_MQ_REQ_NOWAIT);
 	if (IS_ERR(req))
 		return;
 	scmd = blk_mq_rq_to_pdu(req);
-- 
2.53.0


       reply	other threads:[~2026-08-31 13:33 UTC|newest]

Thread overview: 6+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <20260831133314.4125787-1-sashal@kernel.org>
2026-08-31 13:20 ` Sasha Levin [this message]
2026-08-31 13:26 ` [PATCH AUTOSEL 6.18-5.10] scsi: bfa: Reduce kernel stack usage in bfa_fcs_lport_fdmi_build_portattr_block() Sasha Levin
2026-08-31 13:28 ` [PATCH AUTOSEL 6.18-5.10] scsi: pm8001: Reject non-fatal dump when controller is crashed Sasha Levin
2026-08-31 16:47   ` sashiko-bot
2026-08-31 13:29 ` [PATCH AUTOSEL 6.18-5.10] scsi: pm8001: Reject firmware update in fatal error state Sasha Levin
2026-08-31 17:05   ` sashiko-bot

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20260831133314.4125787-23-sashal@kernel.org \
    --to=sashal@kernel.org \
    --cc=James.Bottomley@HansenPartnership.com \
    --cc=bvanassche@acm.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-scsi@vger.kernel.org \
    --cc=mkp@kernel.org \
    --cc=patches@lists.linux.dev \
    --cc=stable@vger.kernel.org \
    --cc=wozizhi@huawei.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox