Netdev List
 help / color / mirror / Atom feed
From: Hanson Wang <hanson.wang@ugreen.com>
To: netdev@vger.kernel.org
Cc: Heiner Kallweit <hkallweit1@gmail.com>, nic_swsd@realtek.com
Subject: [RFC] r8169: possible TX completion ordering issue on arm64
Date: Wed, 18 Nov 2026 22:11:30 +0800	[thread overview]
Message-ID: <20261118141130.2694997-1-hanson.wang@ugreen.com> (raw)

Hi,

We observed four crashes in dql_completed() on an RK3588-based arm64
board (UGREEN RK3588C DH4300Plus).  The NIC is an RTL8125B, XID 641,
at PCI 0002:21:00.0, linked at 2.5Gbps.  The kernel is a vendor
6.1.84 build (6.1.84 #d64bed19bdfd).  We compared the cur_tx publish
and read with current mainline and that part looks the same:
rtl8169_start_xmit() still does smp_wmb() followed by WRITE_ONCE(),
and rtl_tx() still does READ_ONCE() before reading the descriptor.
The vendor tree still calls netdev_completed_queue() directly; mainline
has since switched that call to netif_subqueue_completed_wake().  We
have not reproduced this on mainline.

All four oopses stop at the same place:

  kernel BUG at lib/dynamic_queue_limits.c:27!
  Internal error: Oops - BUG: 00000000f2000800 [#1] SMP
  pc : dql_completed+0x13c/0x14c
  lr : rtl8169_poll+0x3a0/0x54c [r8169]

One of them, from idle softirq:

  CPU: 2 PID: 0 Comm: swapper/2 Tainted: G        W   E     6.1.84 #d64bed19bdfd
  Hardware name: UGREEN RK3588C DH4300Plus Board (DT)
  Call trace:
   dql_completed+0x13c/0x14c
   __napi_poll+0x3c/0x1d0
   net_rx_action+0x16c/0x3a0
   __do_softirq+0x120/0x338
   ____do_softirq+0x14/0x20
   call_on_irq_stack+0x24/0x40
   do_softirq_own_stack+0x20/0x30
   __irq_exit_rcu+0x98/0xc0
   irq_exit_rcu+0x14/0x20
   el1_interrupt+0x48/0xa0
   el1h_64_irq_handler+0x14/0x20
   el1h_64_irq+0x74/0x78
   arch_cpu_idle+0x14/0x40
   ...
  Kernel panic - not syncing: Oops - BUG: Fatal exception in interrupt

Another is the same pc/lr from process context (ctl_serv), entering
softirq via el0_interrupt.  Two of the four were preceded by
"NETDEV WATCHDOG: eth0 (r8169): transmit queue 0 timed out" and
"ASPM disabled on Tx timeout".  The other two were not.

dql_completed() is called as (dql, count).  At the BUG the registers
are:

  x1 = 0x00000570   x2 = 0x00000534   delta = 0x3c
  x1 = 0x000007de   x2 = 0x000007a2   delta = 0x3c
  x1 = 0x000009d8   x2 = 0x0000099c   delta = 0x3c
  x1 = 0x00001091   x2 = 0x00001055   delta = 0x3c

The failing check is BUG_ON(count > num_queued - num_completed).  The
branch goes straight to the BRK, so x1 is still count and x2 is still
num_queued - num_completed.  In every case the completion count is
0x3c = 60 = ETH_ZLEN bytes ahead of the outstanding BQL count, which
is one short frame padded to the minimum length.  RTL_GIGA_MAC_VER_63
pads such frames to ETH_ZLEN.  We have not established why the counts
diverge.

While looking at the TX path I noticed that rtl8169_start_xmit()
publishes the new ring index with:

  smp_wmb();
  WRITE_ONCE(tp->cur_tx, tp->cur_tx + frags + 1);

and rtl_tx() then does:

  while (READ_ONCE(tp->cur_tx) != dirty_tx) {
          status = le32_to_cpu(READ_ONCE(tp->TxDescArray[entry].opts1));

On arm64 a control dependency does not order the later load of opts1,
or the later read of the BQL counters, against the load of cur_tx.
smp_wmb() orders the writer's stores, but the reader has no pairing
barrier.  A CPU could observe the new cur_tx, still see DescOwn clear
on a descriptor that has not been handed to the hardware, and account
that skb in netdev_completed_queue() before __netdev_sent_queue() is
visible.  That would trip this BUG_ON.  I may be missing an ordering
guarantee that already covers this.

Would an acquire load on the reader be appropriate here?

  -     while (READ_ONCE(tp->cur_tx) != dirty_tx) {
  +     while (smp_load_acquire(&tp->cur_tx) != dirty_tx) {

This is a question, not a fix.  We have not A/B tested it, and we have
not reproduced the oops on mainline.

Thanks,
Hanson Wang

                 reply	other threads:[~2026-11-18 14:12 UTC|newest]

Thread overview: [no followups] expand[flat|nested]  mbox.gz  Atom feed

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261118141130.2694997-1-hanson.wang@ugreen.com \
    --to=hanson.wang@ugreen.com \
    --cc=hkallweit1@gmail.com \
    --cc=netdev@vger.kernel.org \
    --cc=nic_swsd@realtek.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox