[PATCH 0/1] ovpn: tcp - defer TX from softirq to workqueue

[PATCH 0/1] ovpn: tcp - defer TX from softirq to workqueue

[Dao Zhong Ma]

Hi,

I observed system hangs when using OpenVPN across suspend/resume
under heavy network load.

The issue can be reproduced with:
1. Establish an OpenVPN connection
2. Generate traffic with iperf3
3. Add CPU/memory pressure with stress-ng
4. Suspend and resume the system

After resume, the system may become unresponsive. SysRq output shows
tasks spinning on sk->sk_lock.slock, and RCU stall warnings are
observed.

Kernel logs and stack traces:
kernel: watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [stress-ng-cpu:11506]
kernel: CPU#4 Utilization every 4000ms during lockup:
kernel:         #1:   0% system,        100% softirq,          1% hardirq,          0% idle
kernel:         #2:   0% system,        100% softirq,          0% hardirq,          0% idle
kernel:         #3:   0% system,        100% softirq,          1% hardirq,          0% idle
kernel:         #4:   0% system,        100% softirq,          0% hardirq,          0% idle
kernel:         #5:   0% system,        100% softirq,          0% hardirq,          0% idle
kernel: CPU: 4 UID: 1000 PID: 11506 Comm: stress-ng-cpu Tainted: G     U              7.0.2-2-cachyos #1 PREEMPT  136cb9373f01c1def72b1b5def9dd1fcc7884035
kernel: Tainted: [U]=USER
kernel: Hardware name: Framework Laptop 13 (AMD Ryzen 7040Series)/FRANMDCP05, BIOS 03.18 01/08/2026
kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x5a/0x260
kernel: Code: 08 0f 92 c0 b9 ff 00 ff ff 23 0f 89 c2 c1 e2 08 09 ca 81 fa 00 01 00 00 73 5a 85 c9 74 0e 66 90 80 3f 00 74 07 f3 90 80 3f 00 <75> f9 66 c7 07 01 00 65 48 ff 05 3f c6 d8 01 5b 41 5e 41 5f 5d e9
kernel: RSP: 0000:ffffcf548d9bb758 EFLAGS: 00000202
kernel: RAX: 0000000000000000 RBX: ffff8ec75f9f4d48 RCX: 0000000000000001
kernel: RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8ec75f9f4d48
kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8ec8547abd80
kernel: R10: 0000000000000020 R11: 0000000000000020 R12: ffff8ec75f9f4c00
kernel: R13: ffff8ec862d9d400 R14: ffff8ec753ade400 R15: ffff8ec771548800
kernel: FS:  00007f9b4fcd0b00(0000) GS:ffff8eccfe99c000(0000) knlGS:0000000000000000
kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
kernel: CR2: 00007ffb7e1f8e38 CR3: 0000000166ced000 CR4: 0000000000f50ef0
kernel: PKRU: 55555554
kernel: Call Trace:
kernel:  <TASK>
kernel:  _raw_spin_lock+0x2a/0x40
kernel:  ovpn_tcp_send_skb+0x4c/0x130 [ovpn 03847a96bef64ed5cc535ef1935e53bb388bcced]
kernel:  ovpn_encrypt_post+0x95/0x1f0 [ovpn 03847a96bef64ed5cc535ef1935e53bb388bcced]
kernel:  ovpn_send+0x15f/0x230 [ovpn 03847a96bef64ed5cc535ef1935e53bb388bcced]
kernel:  ovpn_net_xmit+0x30f/0x560 [ovpn 03847a96bef64ed5cc535ef1935e53bb388bcced]
kernel:  dev_hard_start_xmit+0x93/0x140
kernel:  __dev_queue_xmit+0x99b/0xa80
kernel:  ? __dev_queue_xmit+0x6f/0xa80
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? nf_hook_slow+0x4b/0xd0
kernel:  ? __pfx_ip_finish_output+0x10/0x10
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? nf_hook+0x9c/0xb0
kernel:  ip_finish_output2+0x189/0x320
kernel:  dst_output+0x99/0xd0
kernel:  __ip_queue_xmit+0x209/0x350
kernel:  __tcp_transmit_skb+0x4cb/0xba0
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? __alloc_skb+0x2b6/0x340
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? skb_split+0x5f/0x3c0
kernel:  tcp_write_xmit+0xad3/0x1a50
kernel:  ? __pfx_tcp_write_timer+0x10/0x10
kernel:  tcp_send_loss_probe+0x94/0x2b0
kernel:  ? __pfx_tcp_write_timer+0x10/0x10
kernel:  tcp_write_timer+0x67/0xb0
kernel:  __run_timer_base+0x33d/0x500
kernel:  tmigr_handle_remote+0x3da/0x3e0
kernel:  run_timer_softirq.cold+0x47/0x394
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? __pfx_tick_nohz_handler+0x10/0x10
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? ktime_get+0x48/0xa0
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? clockevents_program_event+0xaa/0xf0
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
kernel:  irq_exit_rcu+0x1ad/0x290
kernel:  sysvec_apic_timer_interrupt+0x33/0x80
kernel:  asm_sysvec_apic_timer_interrupt+0x1a/0x20
kernel: RIP: 0033:0x559fc1ece66f
kernel: Code: 44 89 c8 c0 e8 03 44 31 c0 83 e0 01 66 41 d1 e8 f7 d8 66 25 08 84 44 31 c0 45 89 c8 41 c0 e8 04 41 31 c0 41 83 e0 01 41 f7 d8 <66> 41 81 e0 08 84 66 d1 e8 41 31 c0 44 89 c8 c0 e8 05 44 31 c0 83
kernel: RSP: 002b:00007ffe1344e860 EFLAGS: 00000297
kernel: RAX: 00000000ffff5912 RBX: 000000000000000a RCX: 00007ffe1344ebcb
kernel: RDX: 00007ffe1344ea27 RSI: 00007ffe1344ec60 RDI: 0000559fc318d450
kernel: RBP: 00007ffe1344edf0 R08: 00000000ffffffff R09: 00000000000000b4
kernel: R10: 0000559fc333b3e0 R11: 0000000000000000 R12: 00007f9b4fcc6258
kernel: R13: 0000000000000000 R14: 0000559fc316ed00 R15: 0000559fc3337e00
kernel:  </TASK>


Dao Zhong Ma (1):
  ovpn: tcp - defer TX from softirq to workqueue to avoid RCU stalls

 drivers/net/ovpn/tcp.c | 80 +++++++++++++++++++++++++++++++-----------
 1 file changed, 59 insertions(+), 21 deletions(-)

--
2.54.0

[PATCH 1/1] ovpn: tcp - defer TX from softirq to workqueue

[Dao Zhong Ma]

ovpn_tcp_send_skb() holds sk->sk_lock.slock while performing the full TCP
send in softirq context. This can hold the spinlock for a long time
(large skb), blocking lock_sock() users. This can starve the RCU GP
kthread and trigger RCU stalls warnings and hung tasks.

Defer the TCP send operation to process context:
- In interrupt context, only enqueue the skb under the spinlock
  schedule tcp_tx_work.
- In process context, dequeue and flush the send queue under lock_sock()

This reduces the softirq critical section to a short duration, allowing
lock_sock() users to make progress and preventing RCU stalls.

Signed-off-by: Dao Zhong Ma <cz1346219@gmail.com>
---
 drivers/net/ovpn/tcp.c | 80 +++++++++++++++++++++++++++++++-----------
 1 file changed, 59 insertions(+), 21 deletions(-)

diff --git a/drivers/net/ovpn/tcp.c b/drivers/net/ovpn/tcp.c
index 65054cc84be5..d75ad0c22a30 100644
--- a/drivers/net/ovpn/tcp.c
+++ b/drivers/net/ovpn/tcp.c
@@ -6,6 +6,7 @@
  *  Author:	Antonio Quartulli <antonio@openvpn.net>
  */

+#include <linux/interrupt.h>
 #include <linux/skbuff.h>
 #include <net/hotdata.h>
 #include <net/inet_common.h>
@@ -312,6 +313,40 @@ static void ovpn_tcp_send_sock(struct ovpn_peer *peer, struct sock *sk)
 	peer->tcp.tx_in_progress = false;
 }

+/* Caller must hold sk->sk_lock.slock. */
+static bool ovpn_tcp_queue_skb(struct ovpn_peer *peer, struct sk_buff *skb)
+{
+	if (skb_queue_len(&peer->tcp.out_queue) >=
+	    READ_ONCE(net_hotdata.max_backlog)) {
+		dev_dstats_tx_dropped(peer->ovpn->dev);
+		kfree_skb(skb);
+		return false;
+	}
+
+	__skb_queue_tail(&peer->tcp.out_queue, skb);
+	return true;
+}
+
+/* Caller must hold sk->sk_lock.slock and own the socket. */
+static void ovpn_tcp_tx_flush(struct ovpn_peer *peer, struct sock *sk)
+{
+	struct sk_buff *skb;
+
+	if (peer->tcp.out_msg.skb)
+		ovpn_tcp_send_sock(peer, sk);
+
+	while (!peer->tcp.out_msg.skb) {
+		skb = __skb_dequeue(&peer->tcp.out_queue);
+		if (!skb)
+			break;
+
+		peer->tcp.out_msg.skb = skb;
+		peer->tcp.out_msg.len = skb->len;
+		peer->tcp.out_msg.offset = 0;
+		ovpn_tcp_send_sock(peer, sk);
+	}
+}
+
 void ovpn_tcp_tx_work(struct work_struct *work)
 {
 	struct ovpn_socket *sock;
@@ -320,7 +355,7 @@ void ovpn_tcp_tx_work(struct work_struct *work)

 	lock_sock(sock->sk);
 	if (sock->peer)
-		ovpn_tcp_send_sock(sock->peer, sock->sk);
+		ovpn_tcp_tx_flush(sock->peer, sock->sk);
 	release_sock(sock->sk);
 }

@@ -345,32 +380,38 @@ static void ovpn_tcp_send_sock_skb(struct ovpn_peer *peer, struct sock *sk,
 void ovpn_tcp_send_skb(struct ovpn_peer *peer, struct sock *sk,
 		       struct sk_buff *skb)
 {
+	struct ovpn_socket *sock;
 	u16 len = skb->len;
+	bool queued;

 	*(__be16 *)__skb_push(skb, sizeof(u16)) = htons(len);

-	spin_lock_nested(&sk->sk_lock.slock, OVPN_TCP_DEPTH_NESTING);
-	if (sock_owned_by_user(sk)) {
-		if (skb_queue_len(&peer->tcp.out_queue) >=
-		    READ_ONCE(net_hotdata.max_backlog)) {
-			dev_dstats_tx_dropped(peer->ovpn->dev);
-			kfree_skb(skb);
-			goto unlock;
-		}
-		__skb_queue_tail(&peer->tcp.out_queue, skb);
-	} else {
-		ovpn_tcp_send_sock_skb(peer, sk, skb);
+	if (unlikely(in_interrupt())) {
+		spin_lock_nested(&sk->sk_lock.slock, OVPN_TCP_DEPTH_NESTING);
+		queued = ovpn_tcp_queue_skb(peer, skb);
+		spin_unlock(&sk->sk_lock.slock);
+		if (!queued)
+			return;
+
+		rcu_read_lock();
+		sock = rcu_dereference_sk_user_data(sk);
+		if (sock)
+			schedule_work(&sock->tcp_tx_work);
+		rcu_read_unlock();
+		return;
 	}
-unlock:
-	spin_unlock(&sk->sk_lock.slock);
+
+	lock_sock_nested(sk, OVPN_TCP_DEPTH_NESTING);
+	queued = ovpn_tcp_queue_skb(peer, skb);
+	if (queued)
+		ovpn_tcp_tx_flush(peer, sk);
+	release_sock(sk);
 }

 static void ovpn_tcp_release(struct sock *sk)
 {
-	struct sk_buff_head queue;
 	struct ovpn_socket *sock;
 	struct ovpn_peer *peer;
-	struct sk_buff *skb;

 	rcu_read_lock();
 	sock = rcu_dereference_sk_user_data(sk);
@@ -390,11 +431,7 @@ static void ovpn_tcp_release(struct sock *sk)
 	}
 	rcu_read_unlock();

-	__skb_queue_head_init(&queue);
-	skb_queue_splice_init(&peer->tcp.out_queue, &queue);
-
-	while ((skb = __skb_dequeue(&queue)))
-		ovpn_tcp_send_sock_skb(peer, sk, skb);
+	ovpn_tcp_tx_flush(peer, sk);

 	peer->tcp.sk_cb.prot->release_cb(sk);
 	ovpn_peer_put(peer);
@@ -653,3 +690,4 @@ void __init ovpn_tcp_init(void)
 			      &inet6_stream_ops);
 #endif
 }
+
--
2.54.0

Re: [PATCH 1/1] ovpn: tcp - defer TX from softirq to workqueue

[Jakub Kicinski]

On Fri,  1 May 2026 22:54:25 +0800 Dao Zhong Ma wrote:
> ovpn_tcp_send_skb() holds sk->sk_lock.slock while performing the full TCP
> send in softirq context. This can hold the spinlock for a long time
> (large skb), blocking lock_sock() users. This can starve the RCU GP
> kthread and trigger RCU stalls warnings and hung tasks.
> 
> Defer the TCP send operation to process context:
> - In interrupt context, only enqueue the skb under the spinlock
>   schedule tcp_tx_work.
> - In process context, dequeue and flush the send queue under lock_sock()
> 
> This reduces the softirq critical section to a short duration, allowing
> lock_sock() users to make progress and preventing RCU stalls.

This appears to break the ovpn selftests

Re: [PATCH 1/1] ovpn: tcp - defer TX from softirq to workqueue

[Antonio Quartulli]

Hi,

On 01/05/2026 16:54, Dao Zhong Ma wrote:
> ovpn_tcp_send_skb() holds sk->sk_lock.slock while performing the full TCP
> send in softirq context. This can hold the spinlock for a long time
> (large skb), blocking lock_sock() users. This can starve the RCU GP

Are you sure that the stall is triggered by just processing 1 large skb?

> kthread and trigger RCU stalls warnings and hung tasks.

Can you trigger the stall/warning easily?
Any steps to reproduce?



> 
> Defer the TCP send operation to process context:
> - In interrupt context, only enqueue the skb under the spinlock
>    schedule tcp_tx_work.
> - In process context, dequeue and flush the send queue under lock_sock()
> 
> This reduces the softirq critical section to a short duration, allowing
> lock_sock() users to make progress and preventing RCU stalls.
> 
> Signed-off-by: Dao Zhong Ma <cz1346219@gmail.com>

As Jakub pointed out, this is breaking the selftests, therefore you 
should first double check what's going wrong.

On top of that, could you please run some performance tests? I think 
this is going to hit quite a penalty.

Especially if you have multiple peers and you defer all their traffic to 
the same workqueue.

Regards,



-- 
Antonio Quartulli
OpenVPN Inc.