[PATCH AUTOSEL 7.0] scsi: virtio_scsi: Move INIT_WORK calls to virtscsi_probe()

[Sasha Levin <sashal@kernel.org>]

From: Joshua Daley <jdaley@linux.ibm.com>

[ Upstream commit da3159a3b3fdc05c6bdba2fd4f4802a6718d879a ]

The last step of virtscsi_handle_event() is to call
virtscsi_kick_event(), which calls INIT_WORK on its own work
item. INIT_WORK resets the work item's data bits to 0.

If this occurs while the work item is being flushed by
cancel_work_sync(), then kernel/workqueue.c/work_offqd_enable triggers a
kernel warning, as it expects the "disable" bit to be 1:

[   21.450115] workqueue: work disable count underflowed
[   21.450117] WARNING: CPU: 1 PID: 56 at kernel/workqueue.c:4328 enable_work+0x10a/0x120
...
[   21.450171] Call Trace:
[   21.450173]  [<000003db2e5bdc3e>] enable_work+0x10e/0x120
[   21.450176] ([<000003db2e5bdc3a>] enable_work+0x10a/0x120)
[   21.450178]  [<000003db2e5bdd86>] cancel_work_sync+0x86/0xa0
[   21.450181]  [<000003daae97d9e4>] virtscsi_remove+0xb4/0xd0 [virtio_scsi]
[   21.450184]  [<000003db2ef3b5ca>] virtio_dev_remove+0x6a/0xd0
[   21.450186]  [<000003db2ef9106c>] device_release_driver_internal+0x1ac/0x260
[   21.450190]  [<000003db2ef8edc8>] bus_remove_device+0xf8/0x190
[   21.450192]  [<000003db2ef88d72>] device_del+0x142/0x340
[   21.450194]  [<000003db2ef88fa0>] device_unregister+0x30/0xa0
[   21.450196]  [<000003db2ef3b2fa>] unregister_virtio_device+0x2a/0x40

This warning may occur if a controller is detached immediately following
a disk detach.

Move the INIT_WORK call to prevent this. Don't re-init event list work
items in virtscsi_kick_event(), init them only once in virtscsi_probe()
instead.

Signed-off-by: Joshua Daley <jdaley@linux.ibm.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Link: https://patch.msgid.link/20260325180857.3675854-2-jdaley@linux.ibm.com
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Signed-off-by: Sasha Levin <sashal@kernel.org>
---

LLM Generated explanations, may be completely bogus:

Now I have enough information to complete the analysis.

---

## PHASE 1: COMMIT MESSAGE FORENSICS

**Step 1.1: Subject Line**
- Record: `scsi: virtio_scsi` "Move INIT_WORK calls to virtscsi_probe()"
  — restructures INIT_WORK placement to avoid race with
  `cancel_work_sync()` during controller removal.

**Step 1.2: Tags**
- Record:
  - `Signed-off-by: Joshua Daley <jdaley@linux.ibm.com>` (author)
  - `Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>` (subsystem
    expert)
  - `Link: https://patch.msgid.link/20260325180857.3675854-2-
    jdaley@linux.ibm.com` (mailing list)
  - `Signed-off-by: Martin K. Petersen` (SCSI maintainer, applied the
    patch)
  - No `Fixes:` tag, no `Cc: stable` (expected - this is why review is
    needed)
  - No `Reported-by:` but reproduction steps present in cover letter

**Step 1.3: Commit Body Analysis**
- Record: The commit describes a race where:
  - `virtscsi_handle_event()` (work function) calls
    `virtscsi_kick_event()` at the end
  - `virtscsi_kick_event()` calls `INIT_WORK` on the SAME work item that
    is currently executing
  - `INIT_WORK` resets work->data bits (including the workqueue disable
    count) to 0
  - If this happens while `cancel_work_sync()` is flushing the work,
    `work_offqd_enable` sees the disable count was cleared and triggers
    "work disable count underflowed" WARN
  - Includes a full stack trace on S390; trigger: "controller is
    detached immediately following a disk detach"

**Step 1.4: Hidden Bug Fix Detection**
- Record: Not hidden - clearly labeled as fixing a warning. Race
  condition fix disguised as "Move INIT_WORK".

## PHASE 2: DIFF ANALYSIS

**Step 2.1: Inventory**
- Record: 1 file (`drivers/scsi/virtio_scsi.c`), ~5 net lines added.
  Changed functions: `virtscsi_kick_event()` (INIT_WORK removed) and
  `virtscsi_probe()` (INIT_WORK loop added). Single-file surgical fix.

**Step 2.2: Code Flow**
- Record:
  - Before: `INIT_WORK(&event_node->work, virtscsi_handle_event)` called
    in `virtscsi_kick_event()`, which is invoked from both
    `virtscsi_kick_event_all()` (at probe/restore time) AND from
    `virtscsi_handle_event()` itself (re-queueing at end of event
    handling).
  - After: `INIT_WORK` called once in `virtscsi_probe()` inside a `for`
    loop over all 8 event_list entries (guarded by
    VIRTIO_SCSI_F_HOTPLUG). `virtscsi_kick_event()` no longer resets the
    work struct state.
  - Forward declaration of `virtscsi_handle_event` removed (probe is
    after the definition).

**Step 2.3: Bug Mechanism**
- Record: **Race condition fix** (category b from playbook). The issue
  is that `INIT_WORK` resets all state bits in `work->data` (including
  the disable count introduced in v6.10 by commit `86898fa6b8cd9`).
  Internally, `cancel_work_sync()` now calls `__cancel_work_sync(work,
  0)` → `__cancel_work(work, WORK_CANCEL_DISABLE)` which increments the
  disable count via `work_offqd_disable()`, then `__flush_work()` waits
  for the function to complete, then calls `enable_work()` to decrement.
  If the work function calls `INIT_WORK` during the flush, disable count
  goes 1→0; later `enable_work()` sees 0 and triggers `WARN_ONCE(true,
  "workqueue: work disable count underflowed\n")` at
  `kernel/workqueue.c:4422`.

**Step 2.4: Fix Quality**
- Record: Obviously correct. The INIT_WORK was redundant after the first
  call (work's function pointer doesn't change between kicks). Moving it
  to probe() eliminates the race. Low regression risk: the work struct
  state is preserved across kicks (no need to re-init), and it persists
  through freeze/resume cycles (virtscsi_freeze doesn't cancel work, so
  state remains intact).

## PHASE 3: GIT HISTORY INVESTIGATION

**Step 3.1: Blame of Buggy Code**
- Record: `INIT_WORK(&event_node->work, virtscsi_handle_event)` in
  `virtscsi_kick_event()` was introduced by commit `365a715009411`
  "[SCSI] virtio-scsi: hotplug support for virtio-scsi" (v3.6-rc1,
  2012). The pattern has existed unchanged for 13+ years in all stable
  trees.

**Step 3.2: No Fixes: Tag to Follow**
- Record: No Fixes: tag present. The WARN symptom was enabled by commit
  `86898fa6b8cd9` "workqueue: Implement disable/enable for (delayed)
  work items" which landed in **v6.10-rc1**. Before v6.10 the same race
  existed but did not trigger this specific WARN (cancel_work_sync
  didn't use the disable count).

**Step 3.3: File History**
- Record: Recent virtio_scsi.c history shows a related commit
  `2678369e8efe0` "virtio_scsi: fix DMA cacheline issues for events" (by
  Michael Tsirkin, Dec 2025) which restructured the event buffers. The
  currently analyzed patch applies cleanly on top of that. No patch
  dependencies required beyond the usual.

**Step 3.4: Author Context**
- Record: Joshua Daley (IBM); this is their first virtio_scsi fix.
  However, the patch was Reviewed-by Stefan Hajnoczi (original virtio-
  scsi author at IBM/RedHat and primary reviewer for virtio_scsi), and
  applied by Martin K. Petersen (SCSI maintainer).

**Step 3.5: Dependencies**
- Record: Standalone fix. A second patch in the series (2/2 "kick
  event_list unconditionally") is independent and addresses a different
  cleanup - not required for this one to work. This patch doesn't depend
  on the other.

## PHASE 4: MAILING LIST AND EXTERNAL RESEARCH

**Step 4.1: Patch Discussion**
- Record: Retrieved full thread via `b4 mbox
  20260325180857.3675854-2-jdaley@linux.ibm.com`. Series is at v4.
  Previous versions (v1-v3) had different approaches (INIT_WORK moved to
  `virtscsi_init()` initially). Changelog notes v4 addresses bisection
  concerns (by placing this patch first in the series) and
  suspend/resume concerns (by choosing `virtscsi_probe()` rather than
  `virtscsi_init()`). **No stable nomination in the thread**, but the
  patch is clearly framed as a bug fix.

**Step 4.2: Reviewers**
- Record: Cc'd: linux-scsi, linux-kernel, virtualization list, MST,
  jasowang, pbonzini (QEMU/virtio maintainers), stefanha (virtio-scsi
  expert), eperezma, Martin Petersen (SCSI maintainer), and multiple IBM
  S390 engineers (mjrosato, farman, frankja). Stefan Hajnoczi's
  Reviewed-by tag confirms subsystem expert review.

**Step 4.3: Bug Report**
- Record: No syzbot report. The reporter is the author himself running
  tests on IBM S390 (evidenced by addresses in stack trace
  `000003db2e5...`). The cover letter documents that the warning is
  reliably reproducible by adding `msleep(1000)` before INIT_WORK and
  running `virsh detach-device disk; virsh detach-device controller`.

**Step 4.4: Related Patches**
- Record: The series "scsi: virtio_scsi: move INIT_WORK calls to
  virtscsi_probe" contains 2 patches, both applied by Martin K. Petersen
  to `7.1/scsi-queue` (`[1/2] da3159a3b3fd` and `[2/2] 0019a3a5756b`).

**Step 4.5: Stable-specific Discussion**
- Record: No explicit stable discussion in the thread. The v4 changelog
  does not mention stable.

## PHASE 5: CODE SEMANTIC ANALYSIS

**Step 5.1: Key Functions**
- Record: `virtscsi_kick_event` (INIT_WORK removed), `virtscsi_probe`
  (INIT_WORK loop added), `virtscsi_handle_event` (forward declaration
  removed since probe is below it).

**Step 5.2: Callers of `virtscsi_kick_event`**
- Record: `virtscsi_kick_event_all()` (called at probe and restore) and
  `virtscsi_handle_event()` (the work function itself, for re-queueing).
  `virtscsi_kick_event_all` is called from `virtscsi_probe()` and
  `virtscsi_restore()`.

**Step 5.3: Callees**
- Record: `virtscsi_kick_event` calls `sg_init_one`,
  `virtqueue_add_inbuf_cache_clean`, `virtqueue_kick`. None of these
  interact with work struct state.

**Step 5.4: Reachability**
- Record: The race path is reachable from userspace via standard device
  hotplug operations (virsh detach-device or equivalent QEMU API calls).
  Very common in cloud/virt environments.

**Step 5.5: Similar Patterns**
- Record: The anti-pattern of "calling INIT_WORK from within the work
  function on its own work_struct" is known to be racy with
  cancel_work_sync. This is why v6.10+ workqueue added the WARN to
  detect it.

## PHASE 6: STABLE TREE ANALYSIS

**Step 6.1: Code in Stable Trees**
- Record: Verified by reading
  `remotes/stable/linux-6.6.y:drivers/scsi/virtio_scsi.c` and
  `linux-6.12.y` — both have the exact same
  `INIT_WORK(&event_node->work, virtscsi_handle_event)` pattern in
  `virtscsi_kick_event()` and the same
  `virtscsi_probe()`/`virtscsi_remove()` structure. Code exists
  unchanged in all maintained stable trees (back to at least 5.15).

**Step 6.2: Backport Difficulty**
- Record: The patch should apply cleanly or with trivial adjustments.
  The surrounding code in `virtscsi_probe()` is similar across stable
  trees, though there was a recent reorganization (`2678369e8efe0`
  "virtio_scsi: fix DMA cacheline issues for events" in mainline, not in
  stable). In 6.12.y, `event_node->event` is still an inline struct (not
  a pointer); the patch's INIT_WORK change is independent of that.

**Step 6.3: Related Fixes in Stable**
- Record: No prior fix for this race in stable. The WARN_ONCE at
  kernel/workqueue.c:4422 was introduced in v6.10 (commit
  `86898fa6b8cd9`).

## PHASE 7: SUBSYSTEM CONTEXT

**Step 7.1: Subsystem**
- Record: `drivers/scsi/virtio_scsi.c` — virtio-scsi driver.
  Criticality: **IMPORTANT**. Used by essentially every KVM/QEMU-based
  virtualization stack (including cloud providers using KVM, libvirt,
  AWS EC2, GCP GCE, OpenStack).

**Step 7.2: Subsystem Activity**
- Record: Moderately active (~20 commits in recent history, many
  cleanup/refactoring). Core logic unchanged since v3.6.

## PHASE 8: IMPACT AND RISK

**Step 8.1: Affected Users**
- Record: All users of virtio-scsi on v6.10+ kernels who perform hotplug
  operations (disk/controller detach). This is a massive user base in
  virtualization.

**Step 8.2: Trigger Conditions**
- Record: Normal administrative workflow: detach a disk, then detach the
  controller immediately. Reproducible with standard virsh commands. Not
  privileged-user-triggerable from guest, but a host-side operation.

**Step 8.3: Failure Mode**
- Record: Kernel `WARN_ONCE` with stack trace in dmesg. **Severity:
  MEDIUM**. Not a crash, not data corruption, not security-relevant.
  But: user-visible warning that could trigger monitoring alerts, CI
  failures (many CI systems treat kernel WARN as failure), and user
  concern. The pre-v6.10 race still exists but without the visible WARN
  — potentially other subtle effects but not documented.

**Step 8.4: Risk vs Benefit**
- Record:
  - BENEFIT: Eliminates a user-visible kernel warning in a very common
    virtualization path. Moderate benefit.
  - RISK: Very low — 5 net lines, moves a single INIT_WORK call,
    reviewed by subsystem expert, applied by maintainer. The work struct
    state is correctly preserved across freeze/resume cycles.
  - Ratio: Favorable for backporting.

## PHASE 9: FINAL SYNTHESIS

**Step 9.1: Evidence Summary**

FOR backporting:
- Real race condition with reproducible user-visible WARN
- Small, surgical fix (5 net lines)
- Reviewed by subsystem expert (Stefan Hajnoczi)
- Applied by SCSI maintainer (Martin K. Petersen)
- Code pattern identical in all stable trees v6.10+ where the WARN is
  triggered
- Common workflow (VM hotplug) triggers this bug
- Low regression risk

AGAINST backporting:
- Symptom is WARN, not crash/corruption
- No explicit Cc: stable nomination
- No Fixes: tag
- Race has existed since v3.6 without major user impact reports

**Step 9.2: Stable Rules**
1. Obviously correct: YES — trivially correct (just relocates INIT_WORK
   to run once)
2. Fixes real bug: YES — triggers WARN in v6.10+
3. Important issue: MEDIUM (WARN, not crash)
4. Small and contained: YES (5 lines, one file)
5. No new features: YES
6. Applies cleanly: YES for recent stable trees

**Step 9.3: Exception Categories**
- Record: Not an exception category (not device ID, quirk, DT, build, or
  doc fix). Regular bug fix.

**Step 9.4: Decision**
The fix addresses a reproducible race condition with a clear user-
visible symptom on v6.10+ kernels (6.12, 6.17, 6.18, 6.19 stable trees).
It has been reviewed by the virtio-scsi expert, applied by the SCSI
maintainer, and affects a widely-deployed driver (used in virtually all
KVM/QEMU VMs). The patch is minimal, correct, and low-risk. The symptom
is only a WARN (not a crash), but it occurs in common hotplug workflows
and the fix is safe.

---

## Verification

- [Phase 1] Parsed tags: Reviewed-by Stefan Hajnoczi, Link to lore,
  Signed-off-by Martin K. Petersen; no Fixes/Cc stable tags
- [Phase 2] Diff analysis: 5 net lines changed in
  `drivers/scsi/virtio_scsi.c`; INIT_WORK moved from
  `virtscsi_kick_event` to a loop in `virtscsi_probe` guarded by
  VIRTIO_SCSI_F_HOTPLUG
- [Phase 3] `git log -S"INIT_WORK(&event_node->work"`: pattern
  introduced by `365a715009411` in v3.6-rc1 (2012)
- [Phase 3] `git describe --contains 365a715009411`: v3.6-rc1, confirmed
  pattern has been stable for 13+ years
- [Phase 3] `git log -S"work disable count underflowed"`: WARN
  introduced by `86898fa6b8cd9` in v6.10-rc1 — this is why the visible
  symptom only exists v6.10+
- [Phase 3] Read kernel/workqueue.c lines 4407-4499 to verify
  `enable_work()`/`__cancel_work_sync()` logic and confirm the race
  mechanism
- [Phase 3] Read include/linux/workqueue.h: confirmed `INIT_WORK` →
  `__INIT_WORK_KEY` → resets `work->data = WORK_DATA_INIT()` (all bits
  zeroed, clobbering disable count)
- [Phase 4] `b4 mbox 20260325180857.3675854-2-jdaley@linux.ibm.com`:
  retrieved full thread (7 messages, 2 reviewers, 1 apply confirmation)
- [Phase 4] Thread content confirmed: v4 is latest, Reviewed-by tag
  added, Martin K. Petersen applied to 7.1/scsi-queue
  (git.kernel.org/mkp/scsi/c/da3159a3b3fd)
- [Phase 4] No explicit stable nomination in the mailing list discussion
- [Phase 5] `grep "virtscsi_kick_event\b"`: 3 hits — function
  definition, call from `virtscsi_kick_event_all()` (probe/restore),
  call from `virtscsi_handle_event()` (re-queue after event processing)
- [Phase 6] Read `remotes/stable/linux-6.6.y:drivers/scsi/virtio_scsi.c`
  and `linux-6.12.y`: confirmed identical `INIT_WORK(&event_node->work,
  ...)` pattern in `virtscsi_kick_event()`, so fix applies conceptually
  cleanly
- [Phase 6] Confirmed via search that recent commit `2678369e8efe0`
  "virtio_scsi: fix DMA cacheline issues for events" changed the event
  buffer layout in mainline but doesn't affect the INIT_WORK move
- [Phase 7] Subsystem: virtio-scsi — widely used in all KVM/QEMU
  virtualization
- [Phase 8] Failure mode: `WARN_ONCE` at kernel/workqueue.c:4422, stack
  trace visible in dmesg; severity MEDIUM (not a
  crash/corruption/security)
- UNVERIFIED: Whether the pre-v6.10 race has any user-visible effect
  beyond the post-v6.10 WARN. The fix is still logically correct for
  pre-v6.10 but the symptom documentation only covers v6.10+.
- UNVERIFIED: Whether the fix requires any adjustment for stable trees
  (the diff in the thread differs slightly from the presented diff —
  presented diff has a few whitespace/context differences from the v4
  mailed patch, but the functional change is identical)

The commit is a small, well-reviewed bug fix for a reproducible race
condition that triggers a kernel WARN in a widely-deployed driver
(virtio-scsi used in essentially all KVM/QEMU VMs) on v6.10+ kernels.
The fix is minimal (5 net lines, one file), obviously correct (the work
struct doesn't need to be re-initialized on every kick), reviewed by the
subsystem expert (Stefan Hajnoczi), and applied by the SCSI maintainer.
Regression risk is very low.

**YES**

 drivers/scsi/virtio_scsi.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/drivers/scsi/virtio_scsi.c b/drivers/scsi/virtio_scsi.c
index 0ed8558dad724..64b6c942f5720 100644
--- a/drivers/scsi/virtio_scsi.c
+++ b/drivers/scsi/virtio_scsi.c
@@ -233,7 +233,6 @@ static void virtscsi_ctrl_done(struct virtqueue *vq)
 	virtscsi_vq_done(vscsi, &vscsi->ctrl_vq, virtscsi_complete_free);
 };
 
-static void virtscsi_handle_event(struct work_struct *work);
 
 static int virtscsi_kick_event(struct virtio_scsi *vscsi,
 			       struct virtio_scsi_event_node *event_node)
@@ -242,7 +241,6 @@ static int virtscsi_kick_event(struct virtio_scsi *vscsi,
 	struct scatterlist sg;
 	unsigned long flags;
 
-	INIT_WORK(&event_node->work, virtscsi_handle_event);
 	sg_init_one(&sg, event_node->event, sizeof(struct virtio_scsi_event));
 
 	spin_lock_irqsave(&vscsi->event_vq.vq_lock, flags);
@@ -984,8 +982,11 @@ static int virtscsi_probe(struct virtio_device *vdev)
 
 	virtio_device_ready(vdev);
 
-	if (virtio_has_feature(vdev, VIRTIO_SCSI_F_HOTPLUG))
+	if (virtio_has_feature(vdev, VIRTIO_SCSI_F_HOTPLUG)) {
+		for (int i = 0; i < VIRTIO_SCSI_EVENT_LEN; i++)
+			INIT_WORK(&vscsi->event_list[i].work, virtscsi_handle_event);
 		virtscsi_kick_event_all(vscsi);
+	}
 
 	scsi_scan_host(shost);
 	return 0;
-- 
2.53.0