All of lore.kernel.org
 help / color / mirror / Atom feed
From: Bert Karwatzki <spasswolf@web.de>
To: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Cc: Thomas Gleixner <tglx@linutronix.de>,
	linux-kernel@vger.kernel.org,  linux-rt-devel@lists.linux.dev,
	spasswolf@web.de
Subject: Re: "Dead loop on virtual device" error without softirq-BKL on PREEMPT_RT
Date: Tue, 17 Feb 2026 00:48:25 +0100	[thread overview]
Message-ID: <37d6e27f96afb57c5716798530cb3560d25202e5.camel@web.de> (raw)
In-Reply-To: <20260216153745.CA3__zRc@linutronix.de>

Am Montag, dem 16.02.2026 um 16:37 +0100 schrieb Sebastian Andrzej Siewior:
> 
> I am not sure what issue is so I can't tell. The dev_xmit_recursion*()
> based counters are per-task so it should be fine. But yet the wifi
> managed to repeatedly enqueue packets. This might be a real recursion, a
> stack trace should tell. And then, somewhere synchronisation is missing.
> 
> > Bert Karwatzki
> 
> Sebastian

The problem seems to be that different preemtible threads try to send skbs. 

I used this debug patch for 6.18.10:

diff --git a/net/core/dev.c b/net/core/dev.c
index 5b536860138d..ecfdd8e3dc99 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -4704,6 +4704,7 @@ int __dev_queue_xmit(struct sk_buff *skb, struct net_device *sb_dev)
 
 	qdisc_pkt_len_init(skb);
 	tcx_set_ingress(skb, false);
+	printk(KERN_INFO "%s 0: skb = %px dev = %s\n", __func__, skb, dev->name);
 #ifdef CONFIG_NET_EGRESS
 	if (static_branch_unlikely(&egress_needed_key)) {
 		if (nf_hook_egress_active()) {
@@ -4739,10 +4740,12 @@ int __dev_queue_xmit(struct sk_buff *skb, struct net_device *sb_dev)
 
 	trace_net_dev_queue(skb);
 	if (q->enqueue) {
+		printk(KERN_INFO "%s 1: skb = %px dev = %s\n", __func__, skb, dev->name);
 		rc = __dev_xmit_skb(skb, q, dev, txq);
 		goto out;
 	}
 
+	printk(KERN_INFO "%s 2: skb = %px dev = %s txq = %px\n", __func__, skb, dev->name, txq);
 	/* The device has no queue. Common case for software devices:
 	 * loopback, all the sorts of tunnels...
 
@@ -4761,15 +4764,20 @@ int __dev_queue_xmit(struct sk_buff *skb, struct net_device *sb_dev)
 		/* Other cpus might concurrently change txq->xmit_lock_owner
 		 * to -1 or to their cpu id, but not to our id.
 		 */
+		printk(KERN_INFO "%s: cpu = %d xmit_lock_owner = %d\n", __func__, cpu, READ_ONCE(txq->xmit_lock_owner));
 		if (READ_ONCE(txq->xmit_lock_owner) != cpu) {
-			if (dev_xmit_recursion())
+			printk(KERN_INFO "%s 3: skb = %px dev = %s txq = %px\n", __func__, skb, dev->name, txq);
+			if (dev_xmit_recursion()) {
+				printk(KERN_INFO "%s: recursion alert for device %s!\n", __func__, dev->name);
 				goto recursion_alert;
+			}
 
 			skb = validate_xmit_skb(skb, dev, &again);
 			if (!skb)
 				goto out;
 
 			HARD_TX_LOCK(dev, txq, cpu);
+			printk(KERN_INFO "%s 4: skb = %px dev = %s txq = %px\n", __func__, skb, dev->name, txq);
 
 			if (!netif_xmit_stopped(txq)) {
 				dev_xmit_recursion_inc();
@@ -4777,6 +4785,7 @@ int __dev_queue_xmit(struct sk_buff *skb, struct net_device *sb_dev)
 				dev_xmit_recursion_dec();
 				if (dev_xmit_complete(rc)) {
 					HARD_TX_UNLOCK(dev, txq);
+					printk(KERN_INFO "%s 5: skb = %px dev = %s txq = %px\n", __func__, skb, dev->name, txq);
 					goto out;
 				}
 			}

The normal path of an skb is this:
2026-02-17T00:29:24.124757+01:00 [ T1522] __dev_queue_xmit 0: skb = ffff8c11ed06bd00 dev = wlp4s0
2026-02-17T00:29:24.124845+01:00 [ T1522] __dev_queue_xmit 2: skb = ffff8c11ed06bd00 dev = wlp4s0 txq = ffff8c1145320200
2026-02-17T00:29:24.124851+01:00 [ T1522] __dev_queue_xmit: cpu = 7 xmit_lock_owner = -1
2026-02-17T00:29:24.124853+01:00 [ T1522] __dev_queue_xmit 3: skb = ffff8c11ed06bd00 dev = wlp4s0 txq = ffff8c1145320200
2026-02-17T00:29:24.124855+01:00 [ T1522] __dev_queue_xmit 4: skb = ffff8c11ed06bd00 dev = wlp4s0 txq = ffff8c1145320200
2026-02-17T00:29:24.124857+01:00 [ T1522] __dev_queue_xmit 5: skb = 0000000000000000 dev = wlp4s0 txq = ffff8c1145320200

This is the situation which produces the error messages:

T1522 tries to send an skb on CPU 7:
2026-02-17T00:29:24.212215+01:00 [ T1522] __dev_queue_xmit 0: skb = ffff8c11ed06b100 dev = wlp4s0
2026-02-17T00:29:24.212217+01:00 [ T1522] __dev_queue_xmit 2: skb = ffff8c11ed06b100 dev = wlp4s0 txq = ffff8c1145320200
2026-02-17T00:29:24.212219+01:00 [ T1522] __dev_queue_xmit: cpu = 7 xmit_lock_owner = -1
2026-02-17T00:29:24.212221+01:00 [ T1522] __dev_queue_xmit 3: skb = ffff8c11ed06b100 dev = wlp4s0 txq = ffff8c1145320200
2026-02-17T00:29:24.212223+01:00 [ T1522] __dev_queue_xmit 4: skb = ffff8c11ed06b100 dev = wlp4s0 txq = ffff8c1145320200 
Here T1522 gets preempted and T1513 is executed on CPU 7 and also tries to send an skb:
2026-02-17T00:29:24.212225+01:00 [ T1513] __dev_queue_xmit 0: skb = ffff8c11ed06a300 dev = wlp4s0
2026-02-17T00:29:24.212228+01:00 [ T1513] __dev_queue_xmit 2: skb = ffff8c11ed06a300 dev = wlp4s0 txq = ffff8c1145320200
2026-02-17T00:29:24.212230+01:00 [ T1513] __dev_queue_xmit: cpu = 7 xmit_lock_owner = 7
2026-02-17T00:29:24.212231+01:00 [ T1513] Dead loop on virtual device wlp4s0, fix it urgently!
2026-02-17T00:29:24.212234+01:00 [ T1513] __dev_queue_xmit 0: skb = ffff8c11ed06a300 dev = wlp4s0
2026-02-17T00:29:24.212236+01:00 [ T1513] __dev_queue_xmit 2: skb = ffff8c11ed06a300 dev = wlp4s0 txq = ffff8c1145320200
2026-02-17T00:29:24.212238+01:00 [ T1513] __dev_queue_xmit: cpu = 7 xmit_lock_owner = 7
2026-02-17T00:29:24.212240+01:00 [ T1513] Dead loop on virtual device wlp4s0, fix it urgently!
2026-02-17T00:29:24.212242+01:00 [ T1513] __dev_queue_xmit 0: skb = ffff8c11ed06a300 dev = wlp4s0
2026-02-17T00:29:24.212244+01:00 [ T1513] __dev_queue_xmit 2: skb = ffff8c11ed06a300 dev = wlp4s0 txq = ffff8c1145320200
2026-02-17T00:29:24.212246+01:00 [ T1513] __dev_queue_xmit: cpu = 7 xmit_lock_owner = 7
2026-02-17T00:29:24.212247+01:00 [ T1513] Dead loop on virtual device wlp4s0, fix it urgently!
T1513 gets preempted and T1522 finishes processing the skb from above:
2026-02-17T00:29:24.212249+01:00 [ T1522] __dev_queue_xmit 5: skb = 0000000000000000 dev = wlp4s0 txq = ffff8c1145320200

Bert Karwatzki

  reply	other threads:[~2026-02-16 23:48 UTC|newest]

Thread overview: 26+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-02-16 13:43 "Dead loop on virtual device" error without softirq-BKL on PREEMPT_RT Bert Karwatzki
2026-02-16 15:32 ` Bert Karwatzki
2026-02-16 15:37   ` Sebastian Andrzej Siewior
2026-02-16 23:48     ` Bert Karwatzki [this message]
2026-02-17  7:19       ` Sebastian Andrzej Siewior
2026-02-17  8:56         ` Bert Karwatzki
2026-02-17  9:57           ` Sebastian Andrzej Siewior
2026-02-17 10:42             ` Bert Karwatzki
2026-02-17 11:24               ` Bert Karwatzki
2026-02-17 16:52                 ` Bert Karwatzki
2026-02-17 19:10                   ` Bert Karwatzki
2026-02-18  7:30                     ` Sebastian Andrzej Siewior
2026-02-18 12:50                       ` Bert Karwatzki
2026-02-26 17:29                         ` Sebastian Andrzej Siewior
2026-03-18 10:30                           ` Daniel Vacek
2026-03-18 11:18                             ` Sebastian Andrzej Siewior
2026-03-18 14:43                               ` Daniel Vacek
2026-03-18 14:51                                 ` Sebastian Andrzej Siewior
2026-03-18 14:58                                   ` Daniel Vacek
2026-04-01 16:55                                   ` Daniel Vacek
2026-04-02  7:03                                     ` Sebastian Andrzej Siewior
2026-04-02  7:50                                       ` Daniel Vacek
2026-04-02  8:31                                         ` Sebastian Andrzej Siewior
2026-04-02  9:21                                           ` Daniel Vacek
2026-04-02 13:46                                             ` Sebastian Andrzej Siewior
2026-04-02 13:58                                               ` Daniel Vacek

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=37d6e27f96afb57c5716798530cb3560d25202e5.camel@web.de \
    --to=spasswolf@web.de \
    --cc=bigeasy@linutronix.de \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-rt-devel@lists.linux.dev \
    --cc=tglx@linutronix.de \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.