* [PATCH 0/2] vhost-scsi: preserve event ordering and fix fallback deadlock @ 2026-09-15 9:42 physicalmtea 2026-09-15 9:42 ` [PATCH 1/2] vhost-scsi: preserve event ordering physicalmtea 2026-09-15 9:42 ` [PATCH 2/2] vhost-scsi: do not relock event vq mutex on send_evt fallback physicalmtea 0 siblings, 2 replies; 6+ messages in thread From: physicalmtea @ 2026-09-15 9:42 UTC (permalink / raw) To: mst Cc: jasowangio, michael.christie, pbonzini, stefanha, eperezma, nab, asias, virtualization, kvm, netdev, linux-kernel From: Jia Jia <physicalmtea@gmail.com> The vhost-scsi event list is populated with llist_add(), but completion walks the detached list without reversing it. Trigger: concurrent hotplug (ln -s). Each event is inserted at the head of the pending list. Multiple events must already be stacked on the llist before the vhost worker runs vhost_scsi_complete_events(false). The whole list is then detached and walked directly, so later-enqueued events are delivered to the guest first. Queued hotplug and hotunplug events can therefore be delivered in reverse order. Fix that ordering first, then fix the fallback path that can relock the event virtqueue mutex while already holding it. Patch 2 is based on patch 1 because both changes update the event completion path. The first patch is otherwise independent of the fallback deadlock. Sashiko AI flagged this while reviewing the vhost-scsi event queue fix. This is a pre-existing self-deadlock. It was reproduced in a follow-up test. Trigger: vq->worker == NULL. vhost_vq_work_queue() then returns false, and a subsequent vhost_scsi_do_plug() call deadlocks. I do not know what normal condition gets us here; the normal vhost-scsi worker detach/reset paths do not reach this code. The only reproduction I could come up with was killing the vhost-scsi worker. This still looks like a low-probability condition. Jia Jia (2): vhost-scsi: preserve event ordering vhost-scsi: do not relock event vq mutex on send_evt fallback drivers/vhost/scsi.c | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) ^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH 1/2] vhost-scsi: preserve event ordering 2026-09-15 9:42 [PATCH 0/2] vhost-scsi: preserve event ordering and fix fallback deadlock physicalmtea @ 2026-09-15 9:42 ` physicalmtea 2026-09-15 9:51 ` sashiko-bot 2026-09-15 21:00 ` Mike Christie 2026-09-15 9:42 ` [PATCH 2/2] vhost-scsi: do not relock event vq mutex on send_evt fallback physicalmtea 1 sibling, 2 replies; 6+ messages in thread From: physicalmtea @ 2026-09-15 9:42 UTC (permalink / raw) To: mst Cc: jasowangio, michael.christie, pbonzini, stefanha, eperezma, nab, asias, virtualization, kvm, netdev, linux-kernel From: Jia Jia <physicalmtea@gmail.com> vhost_scsi_send_evt() uses llist_add(), so pending events are LIFO. vhost_scsi_complete_events() walks the detached list directly. This can deliver a later hotplug or hotunplug event before an earlier one. Reverse the detached list before processing it. This keeps delivery in enqueue order. Fixes: a6c9af87363c ("tcm_vhost: Add hotplug/hotunplug support") Signed-off-by: Jia Jia <physicalmtea@gmail.com> --- drivers/vhost/scsi.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/vhost/scsi.c b/drivers/vhost/scsi.c index 7a1f39a32..37f681aab 100644 --- a/drivers/vhost/scsi.c +++ b/drivers/vhost/scsi.c @@ -637,7 +637,7 @@ static void vhost_scsi_complete_events(struct vhost_scsi *vs, bool drop) struct llist_node *llnode; mutex_lock(&vq->mutex); - llnode = llist_del_all(&vs->vs_event_list); + llnode = llist_reverse_order(llist_del_all(&vs->vs_event_list)); llist_for_each_entry_safe(evt, t, llnode, list) { if (!drop) vhost_scsi_do_evt_work(vs, evt); -- 2.34.1 ^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH 1/2] vhost-scsi: preserve event ordering 2026-09-15 9:42 ` [PATCH 1/2] vhost-scsi: preserve event ordering physicalmtea @ 2026-09-15 9:51 ` sashiko-bot 2026-09-15 21:00 ` Mike Christie 1 sibling, 0 replies; 6+ messages in thread From: sashiko-bot @ 2026-09-15 9:51 UTC (permalink / raw) To: physicalmtea; +Cc: kvm, virtualization, Eugenio Perez, Michael S. Tsirkin > From: Jia Jia <physicalmtea@gmail.com> > > vhost_scsi_send_evt() uses llist_add(), so pending events are LIFO. > vhost_scsi_complete_events() walks the detached list directly. > This can deliver a later hotplug or hotunplug event before an earlier > one. > > Reverse the detached list before processing it. > This keeps delivery in enqueue order. > > Fixes: a6c9af87363c ("tcm_vhost: Add hotplug/hotunplug support") > Signed-off-by: Jia Jia <physicalmtea@gmail.com> Sashiko has reviewed this patch and found no issues. It looks great! -- Sashiko AI review · https://sashiko.dev/#/patchset/20260915094244.7900-1-physicalmtea@gmail.com?part=1 ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH 1/2] vhost-scsi: preserve event ordering 2026-09-15 9:42 ` [PATCH 1/2] vhost-scsi: preserve event ordering physicalmtea 2026-09-15 9:51 ` sashiko-bot @ 2026-09-15 21:00 ` Mike Christie 1 sibling, 0 replies; 6+ messages in thread From: Mike Christie @ 2026-09-15 21:00 UTC (permalink / raw) To: physicalmtea, mst Cc: jasowangio, pbonzini, stefanha, eperezma, nab, asias, virtualization, kvm, netdev, linux-kernel On 9/15/26 4:42 AM, physicalmtea@gmail.com wrote: > diff --git a/drivers/vhost/scsi.c b/drivers/vhost/scsi.c > index 7a1f39a32..37f681aab 100644 > --- a/drivers/vhost/scsi.c > +++ b/drivers/vhost/scsi.c > @@ -637,7 +637,7 @@ static void vhost_scsi_complete_events(struct vhost_scsi *vs, bool drop) > struct llist_node *llnode; > > mutex_lock(&vq->mutex); > - llnode = llist_del_all(&vs->vs_event_list); > + llnode = llist_reverse_order(llist_del_all(&vs->vs_event_list)); > llist_for_each_entry_safe(evt, t, llnode, list) { > if (!drop) > vhost_scsi_do_evt_work(vs, evt); > -- > 2.34.1 > > Reviewed-by: Mike Christie <michael.christie@oracle.com> ^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH 2/2] vhost-scsi: do not relock event vq mutex on send_evt fallback 2026-09-15 9:42 [PATCH 0/2] vhost-scsi: preserve event ordering and fix fallback deadlock physicalmtea 2026-09-15 9:42 ` [PATCH 1/2] vhost-scsi: preserve event ordering physicalmtea @ 2026-09-15 9:42 ` physicalmtea 2026-09-15 9:52 ` sashiko-bot 1 sibling, 1 reply; 6+ messages in thread From: physicalmtea @ 2026-09-15 9:42 UTC (permalink / raw) To: mst Cc: jasowangio, michael.christie, pbonzini, stefanha, eperezma, nab, asias, virtualization, kvm, netdev, linux-kernel From: Jia Jia <physicalmtea@gmail.com> vhost_scsi_send_evt() is called with the event virtqueue mutex held. If the worker is gone, the fallback currently calls vhost_scsi_complete_events(), which tries to acquire the same mutex again and deadlocks the caller. Split event completion into a helper for callers that already hold the mutex and a locking wrapper for the event worker. Use the helper on the fallback path. Link: https://lore.kernel.org/all/20260905005352.1E5B01F00A3D@smtp.kernel.org/ Fixes: b1b2ce58ed23 ("vhost-scsi: Handle vhost_vq_work_queue failures for events") Signed-off-by: Jia Jia <physicalmtea@gmail.com> Reviewed-by: Mike Christie <michael.christie@oracle.com> --- drivers/vhost/scsi.c | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/drivers/vhost/scsi.c b/drivers/vhost/scsi.c index 37f681aab..9cd181cb1 100644 --- a/drivers/vhost/scsi.c +++ b/drivers/vhost/scsi.c @@ -630,19 +630,26 @@ vhost_scsi_do_evt_work(struct vhost_scsi *vs, struct vhost_scsi_evt *evt) vhost_scsi_log_write(vq, vq_log, log_num); } -static void vhost_scsi_complete_events(struct vhost_scsi *vs, bool drop) +/* Caller must hold the event virtqueue mutex. */ +static void __vhost_scsi_complete_events(struct vhost_scsi *vs, bool drop) { - struct vhost_virtqueue *vq = &vs->vqs[VHOST_SCSI_VQ_EVT].vq; struct vhost_scsi_evt *evt, *t; struct llist_node *llnode; - mutex_lock(&vq->mutex); llnode = llist_reverse_order(llist_del_all(&vs->vs_event_list)); llist_for_each_entry_safe(evt, t, llnode, list) { if (!drop) vhost_scsi_do_evt_work(vs, evt); vhost_scsi_free_evt(vs, evt); } +} + +static void vhost_scsi_complete_events(struct vhost_scsi *vs, bool drop) +{ + struct vhost_virtqueue *vq = &vs->vqs[VHOST_SCSI_VQ_EVT].vq; + + mutex_lock(&vq->mutex); + __vhost_scsi_complete_events(vs, drop); mutex_unlock(&vq->mutex); } @@ -1829,7 +1836,7 @@ vhost_scsi_send_evt(struct vhost_scsi *vs, struct vhost_virtqueue *vq, llist_add(&evt->list, &vs->vs_event_list); if (!vhost_vq_work_queue(vq, &vs->vs_event_work)) - vhost_scsi_complete_events(vs, true); + __vhost_scsi_complete_events(vs, true); } static void vhost_scsi_evt_handle_kick(struct vhost_work *work) -- 2.34.1 ^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH 2/2] vhost-scsi: do not relock event vq mutex on send_evt fallback 2026-09-15 9:42 ` [PATCH 2/2] vhost-scsi: do not relock event vq mutex on send_evt fallback physicalmtea @ 2026-09-15 9:52 ` sashiko-bot 0 siblings, 0 replies; 6+ messages in thread From: sashiko-bot @ 2026-09-15 9:52 UTC (permalink / raw) To: physicalmtea; +Cc: Michael S. Tsirkin, kvm, Eugenio Perez, virtualization > From: Jia Jia <physicalmtea@gmail.com> > > vhost_scsi_send_evt() is called with the event virtqueue mutex held. > If the worker is gone, the fallback currently calls > vhost_scsi_complete_events(), which tries to acquire the same mutex again > and deadlocks the caller. > > Split event completion into a helper for callers that already hold the > mutex and a locking wrapper for the event worker. Use the helper on the > fallback path. > > Link: https://lore.kernel.org/all/20260905005352.1E5B01F00A3D@smtp.kernel.org/ > Fixes: b1b2ce58ed23 ("vhost-scsi: Handle vhost_vq_work_queue failures for events") > Signed-off-by: Jia Jia <physicalmtea@gmail.com> Sashiko has reviewed this patch and found no issues. It looks great! -- Sashiko AI review · https://sashiko.dev/#/patchset/20260915094244.7900-1-physicalmtea@gmail.com?part=2 ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2026-09-15 21:03 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2026-09-15 9:42 [PATCH 0/2] vhost-scsi: preserve event ordering and fix fallback deadlock physicalmtea 2026-09-15 9:42 ` [PATCH 1/2] vhost-scsi: preserve event ordering physicalmtea 2026-09-15 9:51 ` sashiko-bot 2026-09-15 21:00 ` Mike Christie 2026-09-15 9:42 ` [PATCH 2/2] vhost-scsi: do not relock event vq mutex on send_evt fallback physicalmtea 2026-09-15 9:52 ` sashiko-bot
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox