From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pj2-f1.google.com (mail-pj2-f1.google.com [74.125.227.129]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 61454446846 for ; Fri, 31 Jul 2026 17:04:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.227.129 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785517497; cv=none; b=AryAPzessMkoWqBspfB8fNNOeUdCZB82a40A45VHcS3XKvGBcD+BZxcP+Fcjq1h+GMvztXEVC05NEGnAJ/Ah/BbGaAWpVbJEzJ7ECG67aw0eRlVclVwofgNwBv2ET80VEC29gNZf/LutHpwBmbU6/BxYyXCSOLgdRGjWl4O9VnU= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785517497; c=relaxed/simple; bh=TiEiCLzRBgCRWu0MaEFZR+ai00p3m9+yXWylxLxfsEo=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version; b=ZXLFNHHJn3LJ4XtUMQC7LJmT7QENSZxJzbZIRPON0edUxvrdtHtSR1a/oQqjbP9Wbxhyms83lijrXExSl25bnFBeAlQYnH2r04+8QJP7W831rF/VzouEhcBrYKfHjN1GXNj8XEECOyWEtDNjxoWNbtJwDJmg8k/AA3eXyLxlc0M= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=WMQqqD9P; arc=none smtp.client-ip=74.125.227.129 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="WMQqqD9P" Received: by mail-pj2-f1.google.com with SMTP id d9443c01a7336-2cabf1f1051so3281985ad.0 for ; Fri, 31 Jul 2026 10:04:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1785517495; x=1786122295; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to:content-type; bh=mI1rZZtu8o4cfi0FGJlAmOUjg03NURhPr/05wZVSU0k=; b=WMQqqD9PUf+K+xI7TwPLllVuMFF4RTtZjKP0tqzZaZRyW9dErUKaHt4MdxR2f3VvMU b25gn7Is3JD1smeZRpQQL8lFbZ3Jk+hnUEGTsNz759W7tGQD8fXv6h8g91y/F7aGJfeS mctjYW6z5MCPL+H0RV8HMLcXOZ2YAE3G7w4vAyP9SDqg4t5j/as5Z6zA9uxBrmhE+O78 BGx8WgaoVGJBEbX1VEgWene57r0WqGbCnqXCDDQOrqmzUPU+dTwd+tFwFW8th5TK6VXM Tq1l90VDclewDZtMezd4ZN8vO9FgY8rqEcTPWmLQTFdiwWrF9sIOhOMVf5eyGNe6xj5E BGMg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1785517495; x=1786122295; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=mI1rZZtu8o4cfi0FGJlAmOUjg03NURhPr/05wZVSU0k=; b=ZNj63CiNZ+ahovQdUXgFBx74x67shxDm1KAwVIDReG9GQZsEpGP9cgafYzNqyCjo1G p1p6CsyW+6CjHRCCVJLTqWgJTwWOohEW8X21un7RSEyBSNWQxcIKcCYzhNOQCtsDp1a1 m6oei8HpldAvLjYpgG+syX94Ue/dJU8EpaPAEgnkLF7htv/sL8xZetnM8doAL96ldEFF ehqmw7XLV7ujw6Z5HY+8n1toE1TZno0KI+IcxhLidl6r2Aocb2GLiYgsgMq9xeJakCeQ H7I7HnU4l93fQmyaJvqREjqrmgkziqO6cBnq+jnipyOwDM0wNQRdRDp/SyopLWZacukq q4hQ== X-Forwarded-Encrypted: i=1; AHgh+RquGdPkqY9EJLYBPeL1yY8rBIVpsL5oiCYxM5X1jywp6Rquu7hYuTc0OD8RomHWyQSQhU/mrTpnMPPuN9Q=@vger.kernel.org X-Gm-Message-State: AOJu0YwdsRZjxhh4kA2T96/ZXyXH+ivEq+WF2xXZ1mt8Ir7RDSNfXk/Z SPAMD1y4OPCNeOuQA0D2VIE0AH6MtOQQIJ3SEw1tTt977a+2A3kaITBAtTnzZtfM3rwXTFg= X-Gm-Gg: AR+sD114P63nUEovADf3ShR1yJ8Zfx89BqAJklLiCo4c3VKLdI0whA+nkcvEmfrUk/q B0ROWGiaSaqrufdx590c1CeTZ/wEk+B0IfdEmnci0SdHVUlBwA0H60drjZzE5m2fh3/qePNimiK GKG3/WiDHgT0uydZ/uflKCps5lylHTQDgq9PiHRWFbGWGtgmSZdXBClKvK0UJoGvwAfZq806lBx QDBSfok8fjNHTWiCiwVStrHVbhaUzaiW1KVbtchJp3KgfNiXNVttAcmMf+FPVhL6O43DY3riJga 5R3l3btKdKbARmqBA1e5BBJ2bJnwBKXgu4pxHukw0od/pDqHE8X2f7dKCiZnIxOfuxo6yXSqnSr gHk1+a5clJKplStveqaqPcbZHa4K4Ho/tlSuxKH/F8N4NVjrpsKS+2bOjm4qe0N7yxJdynrKaMj vnDMvRX/KRqSZAMPoc9qZthWAayjCGl/Dvw7hpqiqzpPjJ+YgwD8EIkZVvU5rINLE= X-Received: by 2002:a17:902:ea0c:b0:2cf:6f7f:55d8 with SMTP id d9443c01a7336-2d0523f76edmr6597195ad.37.1785517495457; Fri, 31 Jul 2026 10:04:55 -0700 (PDT) Received: from jplife-black.. ([171.2.207.88]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2d04ae19b82sm8064705ad.6.2026.07.31.10.04.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 31 Jul 2026 10:04:55 -0700 (PDT) From: Fan XinRan To: westeri@kernel.org, YehezkelShB@gmail.com Cc: andrew+netdev@lunn.ch, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, Fan XinRan Subject: [PATCH net] net: thunderbolt: Tear down DMA paths before stopping the rings Date: Fri, 31 Jul 2026 17:04:42 +0000 Message-ID: <20260731170442.45530-1-shinjiangjiang@gmail.com> X-Mailer: git-send-email 2.43.0 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit tbnet_tear_down() stops both rings and frees their frame buffers before calling tb_xdomain_disable_paths(). tb_ring_stop() zeroes the ring's descriptor base and tbnet_free_buffers() unmaps and frees the pages the frames sit in, so by the time __tb_path_deactivate_hop() polls the hop's 'pending' bit, anything still in flight has nowhere to drain to. This is the mirror image of the setup path. tbnet_connected_work() already documents the invariant: /* Both logins successful so enable the rings, high-speed DMA * paths and start the network device queue. * * Note we enable the DMA paths last to make sure we have primed * the Rx ring before any incoming packets are allowed to * arrive. */ Teardown should undo that in reverse, but does not. On an ASMedia ASM4242 host router the 'pending' bit then never clears: every teardown burns the full 500 ms timeout and __tb_path_deactivate_hop() returns -ETIMEDOUT. Raising the timeout to 5 s does not help, so the hop is not slow to drain, it never drains at all. The failure is invisible above the thunderbolt core. __tb_path_deactivate_hops() is void and only calls tb_port_warn(); tb_path_deactivate(), tb_tunnel_deactivate() and __tb_disconnect_xdomain_paths() are void as well, and tb_disconnect_xdomain_paths() ends in an unconditional "return 0". So tb_xdomain_disable_paths() reports success and the netdev_warn() below it never fires. Repeated teardowns eventually take the XDomain control channel down, after which the peer node is gone and only a power cycle brings the controller back. Deactivating the paths first fixes it. Measured with kretprobes on a stock v6.17 tree with no other patches applied, on a link that was up and had just carried traffic: before: __tb_path_deactivate_hop() returns 0 for the first hop, then -ETIMEDOUT for the second 500335 us later after: 0 for both, 525 us apart Alternating the two orderings ABBA over three load levels, four teardowns per arm: every teardown failed before the change (21 of 21 that ran), none failed after (0 of 24). The before arms ran short because the link died partway through. The same split shows up when the interface is enslaved to a bond instead of just brought down, which is how I ran into this in the first place. Throughput and latency after the change are unchanged. Hosts whose routers drain the hop despite the stale descriptor base see no functional difference, since the paths end up deactivated either way. Fixes: 4944269305df ("thunderbolt: Properly disable path") Signed-off-by: Fan XinRan --- Notes for reviewers, not for the commit log: I can only test this on an ASMedia ASM4242 host router - I have no Intel host router to check for regressions on, and that is the gap I would most like a second opinion on. The argument that other hosts are unaffected is that the paths end up deactivated in both orderings, and that the window this opens (rings still armed while tb_xdomain_disable_paths() runs) cannot take new traffic: __tb_path_deactivate_hop() clears hop.enable before it starts polling, and tbnet_tear_down() has already called netif_stop_queue(). That is an argument, not a measurement. Details left out of the commit log to keep it short: - The three load levels were idle, 100 pings, and 3 s of iperf3 before each teardown. The 21 vs 24 asymmetry is because the "before" arms stopped early when the link died: 8 teardowns completed idle, 7 under light load, 6 under heavy load. Only one arm per load level died, so I would not read a dose-response into that on its own. - The failing hop is an ingress hop on a non-NHI port: thunderbolt 0000:70:00.0: 0:5: hop deactivation failed for hop 0, index 1 - The enslave run is a smaller sample (the link dies faster there, so the before arm only got two teardowns in): -ETIMEDOUT on both before the change, 0 after. Note the interface also gets destroyed on enslave regardless of the ordering - that looks like a separate problem and this patch does not claim to fix it. drivers/net/thunderbolt/main.c | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-) diff --git a/drivers/net/thunderbolt/main.c b/drivers/net/thunderbolt/main.c index 02a9165..a04c090 100644 --- a/drivers/net/thunderbolt/main.c +++ b/drivers/net/thunderbolt/main.c @@ -386,11 +386,16 @@ static void tbnet_tear_down(struct tbnet *net, bool send_logout) break; } - tb_ring_stop(net->rx_ring.ring); - tb_ring_stop(net->tx_ring.ring); - tbnet_free_buffers(&net->rx_ring); - tbnet_free_buffers(&net->tx_ring); - + /* Tear the paths down before stopping the rings. This mirrors + * tbnet_connected_work(), which enables the paths last so the + * Rx ring is primed before packets can arrive. Stopping a + * ring zeroes its descriptor base and tbnet_free_buffers() + * unmaps and frees the frame buffers, leaving anything still + * in flight with nowhere to drain to; + * __tb_path_deactivate_hop() then waits for the hop's + * 'pending' bit, which on some host routers never clears in + * that state. + */ ret = tb_xdomain_disable_paths(net->xd, net->local_transmit_path, net->tx_ring.ring->hop, @@ -399,6 +404,11 @@ static void tbnet_tear_down(struct tbnet *net, bool send_logout) if (ret) netdev_warn(net->dev, "failed to disable DMA paths\n"); + tb_ring_stop(net->rx_ring.ring); + tb_ring_stop(net->tx_ring.ring); + tbnet_free_buffers(&net->rx_ring); + tbnet_free_buffers(&net->tx_ring); + tb_xdomain_release_in_hopid(net->xd, net->remote_transmit_path); net->remote_transmit_path = 0; } -- 2.43.0