Re: Stalls when starting a VSOCK listening socket: soft lockups, RCU stalls, timeout

[Peter Zijlstra <peterz@infradead.org>]

On Mon, Mar 02, 2026 at 06:28:38AM +0100, Jiri Slaby wrote:

> The state of the lock:
> 
> crash> struct rq.__lock -x ffff8d1a6fd35dc0
>   __lock = {
>     raw_lock = {
>       {
>         val = {
>           counter = 0x40003
>         },
>         {
>           locked = 0x3,
>           pending = 0x0
>         },
>         {
>           locked_pending = 0x3,
>           tail = 0x4
>         }
>       }
>     }
>   },
> 


That had me remember the below patch that never quite made it. I've
rebased it to something more recent so it applies.

If you stick that in, we might get a clue as to who is owning that lock.
Provided it all wants to reproduce well enough.

---
Subject: locking/qspinlock: Save previous node & owner CPU into mcs_spinlock
From: Waiman Long <longman@redhat.com>
Date: Fri, 3 May 2024 22:41:06 -0400

From: Waiman Long <longman@redhat.com>

When examining a contended spinlock in a crash dump, we can only find
out the tail CPU in the MCS wait queue. There is no simple way to find
out what other CPUs are waiting for the spinlock and which CPU is the
lock owner.

Make it easier to figure out these information by saving previous node
data into the mcs_spinlock structure. This will allow us to reconstruct
the MCS wait queue from tail to head. In order not to expand the size
of mcs_spinlock, the original count field is split into two 16-bit
chunks. The first chunk is for count and the second one is the new
prev_node value.

  bits 0-1 : qnode index
  bits 2-15: CPU number + 1

This prev_node value may be truncated if there are 16k or more CPUs in
the system.

The locked value in the queue head is also repurposed to hold an encoded
qspinlock owner CPU number when acquiring the lock in the qspinlock
slowpath of an contended lock.

This lock owner information will not be available when the lock is
acquired directly in the fast path or in the pending code path. There
is no easy way around that.

These changes should make analysis of a contended spinlock in a crash
dump easier.

Signed-off-by: Waiman Long <longman@redhat.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20240504024106.654319-1-longman@redhat.com
---
 include/asm-generic/mcs_spinlock.h |    5 +++--
 kernel/locking/mcs_spinlock.h      |    8 +++++++-
 kernel/locking/qspinlock.c         |    8 ++++++++
 3 files changed, 18 insertions(+), 3 deletions(-)

--- a/include/asm-generic/mcs_spinlock.h
+++ b/include/asm-generic/mcs_spinlock.h
@@ -3,8 +3,9 @@
 
 struct mcs_spinlock {
 	struct mcs_spinlock *next;
-	int locked; /* 1 if lock acquired */
-	int count;  /* nesting count, see qspinlock.c */
+	int locked;	 /* non-zero if lock acquired */
+	short count;	 /* nesting count, see qspinlock.c */
+	short prev_node; /* encoded previous node value */
 };
 
 /*
--- a/kernel/locking/mcs_spinlock.h
+++ b/kernel/locking/mcs_spinlock.h
@@ -13,6 +13,12 @@
 #ifndef __LINUX_MCS_SPINLOCK_H
 #define __LINUX_MCS_SPINLOCK_H
 
+/*
+ * Save an encoded version of the current MCS lock owner CPU to the
+ * mcs_spinlock structure of the next lock owner.
+ */
+#define MCS_LOCKED	(smp_processor_id() + 1)
+
 #include <asm/mcs_spinlock.h>
 
 #ifndef arch_mcs_spin_lock_contended
@@ -34,7 +40,7 @@
  * unlocking.
  */
 #define arch_mcs_spin_unlock_contended(l)				\
-	smp_store_release((l), 1)
+	smp_store_release((l), MCS_LOCKED)
 #endif
 
 /*
--- a/kernel/locking/qspinlock.c
+++ b/kernel/locking/qspinlock.c
@@ -250,6 +250,7 @@ void __lockfunc queued_spin_lock_slowpat
 
 	node->locked = 0;
 	node->next = NULL;
+	node->prev_node = 0;
 	pv_init_node(node);
 
 	/*
@@ -278,6 +279,13 @@ void __lockfunc queued_spin_lock_slowpat
 	next = NULL;
 
 	/*
+	 * The prev_node value is saved for crash dump analysis purpose only,
+	 * it is not used within the qspinlock code. The encoded node value
+	 * may be truncated if there are 16k or more CPUs in the system.
+	 */
+	node->prev_node = old >> _Q_TAIL_IDX_OFFSET;
+
+	/*
 	 * if there was a previous node; link it and wait until reaching the
 	 * head of the waitqueue.
 	 */