* [PATCH v3] RDMA/ipoib: cancel the AH reaper synchronously on teardown
@ 2026-11-05 4:29 Chengyu Liu
2026-11-05 4:34 ` sashiko-bot
2026-11-05 19:38 ` Leon Romanovsky
0 siblings, 2 replies; 4+ messages in thread
From: Chengyu Liu @ 2026-11-05 4:29 UTC (permalink / raw)
To: leon; +Cc: jgg, linux-rdma, stable, Chengyu Liu
ipoib_stop_ah_reaper() uses cancel_delayed_work(), which cannot stop a
callback that is already running. ipoib_reap_ah() re-arms itself after
an unlocked test_bit(IPOIB_STOP_REAPER) check, so a callback that has
passed the check but has not re-armed yet still queues the delayed work
after the "stop":
CPU0 (reaper) CPU1 (stop/teardown)
test_bit(STOP_REAPER) == 0
set_bit(STOP_REAPER)
cancel_delayed_work() (no-op)
... destroy_workqueue(priv->wq)
queue_delayed_work(priv->wq, ...) <- delayed work sits on the timer
destroy_workqueue() cannot see a delayed work that is pending on the
timer wheel, so the re-armed work fires after the workqueue is
destroyed and priv is freed: both the queueing onto the dead queue and
the callback's container_of() on freed memory are use-after-free.
Use cancel_delayed_work_sync(), which waits for a running callback and
swallows its re-arm while the stop is in flight. Note that
ipoib_stop_ah_reaper() also runs on ndo_stop and on the stop/open cycle
of a heavy flush, so the work must be left enabled for
ipoib_start_ah_reaper() to re-arm it on the next open;
cancel_delayed_work_sync() re-enables on the way out, unlike
disable_delayed_work_sync() which would kill the reaper for good.
The comment above claims ipoib_reap_dead_ahs() "ensures the work is
really stopped", but it never touches the work item - it only walks
the dead_ahs list under priv->lock. The stop guarantee comes from the
cancel; fix the comment along with it.
Found by static analysis. Reproduced on a KASAN build running stock
ib_ipoib on a Soft-RoCE transport (rxe): when a running reaper callback
escapes the stop and re-arms, the timer fires one second later in
__run_timer_base() on the destroyed workqueue (slab-use-after-free).
Cc: stable@vger.kernel.org
Signed-off-by: Chengyu Liu <lui2012qwq@gmail.com>
---
v3:
- back to cancel_delayed_work_sync(): ipoib_stop_ah_reaper() also runs
on ndo_stop and on the heavy-flush stop/open cycle, so the work must
be left enabled (sashiko review); disable_delayed_work_sync() would
kill the reaper on the next open
v2: https://lore.kernel.org/r/20261104181753.1245369-1-lui2012qwq@gmail.com
- add reproducer on stock ib_ipoib over rxe
- switch to disable_delayed_work_sync() (superseded by v3)
v1: https://lore.kernel.org/r/20261103170703.1600310-1-lui2012qwq@gmail.com
Reproducer (harness summary; full driver scripts and the timing trigger
below):
QEMU has no IB hardware, so the stock ib_ipoib driver runs over a
Soft-RoCE (rxe) transport presented as an IB port. Harness shims,
all below ib_ipoib (the driver itself is unmodified):
* rxe_verbs.c: report link layer INFINIBAND and cap flags
PROT_IB|IB_MAD|AF_IB from rxe_get_link_layer() /
rxe_port_immutable(); skip ib_get_eth_speed() for the flipped
port in rxe_query_port(); set dev.parent before
ib_register_device().
* sa_query.c: answer ib_sa_path_rec_get() locally (async via
system_wq) when the device has no SA.
* multicast.c: complete ib_sa_join_multicast() locally (async);
tag the synthetic object (comp_mask bit 63) so
ib_sa_free_multicast() frees it.
* ipoib_ib.c (witness only, no behavior change): trace_printk()
timing in ipoib_reap_ah(), and an ESCAPED-ARM line when the
callback re-arms after IPOIB_STOP_REAPER was set.
With ~500 AH entries churned into dead_ahs, the reaper's walk
measures 20-160us once a second; the teardown (RDMA netlink
DELLINK of the rxe device) reaches ipoib_stop_ah_reaper() ~3.9ms
after the syscall enters. The stock test->queue window is the
handful of instructions between the lockless test_bit() and the
queue_delayed_work(); to hit it deterministically the PoC kernel
widens it to 200us with a module parameter. On a hit the witness
logs
ipoib_stop_ah_reaper: reap_ah: ib0 STOP at=531116547240
ipoib_reap_ah: reap_ah: ib0 ESCAPED-ARM after stop
and one second later the escaped timer fires on the destroyed
workqueue (trimmed):
BUG: KASAN: use-after-free in __run_timer_base.part.0+0x80c/0x900
Write of size 8 at addr ffff888021088628 by task swapper/0/0
Call Trace:
<IRQ>
kasan_report+0x96/0xd0
__run_timer_base.part.0+0x80c/0x900
tmigr_handle_remote_up+0x38b/0x830
...
handle_softirqs+0x184/0x5c0
irq_exit_rcu+0x6e/0x150
sysvec_apic_timer_interrupt+0x70/0x80
</IRQ>
with the workqueue core reporting, right before the splat:
workqueue: cannot queue ipoib_reap_ah on wq ipoib_wq
------------------------- reproducer: trigger.c -------------------------
// SPDX-License-Identifier: GPL-2.0
/*
* trigger.c - F-026 teardown timing trigger.
*
* Watches the PoC kernel's ipoib_reap_ah() witness lines on trace_pipe
* ("reap_ah: ib0 start=<ns> walk_ns=<ns>"), predicts the next reaper run
* (~1s period, jiffy-aligned) and deletes the rxe device via RDMA netlink
* (RDMA_NLDEV_CMD_DELLINK, the real teardown path: ib unregister -> ipoib
* remove -> stop_ah_reaper -> destroy_workqueue -> free) at
*
* next_start + walk_ns + offset
*
* which is the callback's test_bit->queue_delayed_work gap.
*
* usage: trigger <rxe_dev> <ib_dev> <offset_ns>
*/
#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <fcntl.h>
#include <time.h>
#include <errno.h>
#include <linux/netlink.h>
#include <sys/socket.h>
#ifndef NETLINK_RDMA
#define NETLINK_RDMA 20
#endif
#define RDMA_NL_NLDEV 5
#define RDMA_NL_GET_TYPE(c, o) (((c) << 10) + (o))
#define RDMA_NLDEV_CMD_GET 1
#define RDMA_NLDEV_CMD_DELLINK 4
#define RDMA_NLDEV_ATTR_DEV_INDEX 1
#define RDMA_NLDEV_ATTR_DEV_NAME 2
static const char *TRACE_PIPE = "/sys/kernel/tracing/trace_pipe";
static unsigned long long now_ns(void)
{
struct timespec ts;
clock_gettime(CLOCK_MONOTONIC, &ts);
return (unsigned long long)ts.tv_sec * 1000000000ull + ts.tv_nsec;
}
static void wait_until(unsigned long long deadline_ns)
{
while (now_ns() < deadline_ns)
;
}
static int nla_put(char *buf, int *off, int type, const void *data, int len)
{
struct nlattr *na = (struct nlattr *)(buf + *off);
na->nla_type = type;
na->nla_len = NLA_HDRLEN + len;
memcpy(buf + *off + NLA_HDRLEN, data, len);
*off += NLA_ALIGN(na->nla_len);
return 0;
}
/* dump all rdma links, return the index of `name` (0 if not found) */
static int rdma_dev_index(int fd, const char *name)
{
char buf[4096];
struct nlmsghdr *nlh = (struct nlmsghdr *)buf;
int found = -1;
struct timeval tv = { .tv_sec = 2, .tv_usec = 0 };
setsockopt(fd, SOL_SOCKET, SO_RCVTIMEO, &tv, sizeof(tv));
memset(buf, 0, sizeof(buf));
nlh->nlmsg_len = NLMSG_LENGTH(0);
nlh->nlmsg_type = RDMA_NL_GET_TYPE(RDMA_NL_NLDEV, RDMA_NLDEV_CMD_GET);
nlh->nlmsg_flags = NLM_F_REQUEST | NLM_F_DUMP;
nlh->nlmsg_seq = 1;
if (send(fd, buf, nlh->nlmsg_len, 0) < 0)
return 0;
for (;;) {
ssize_t len = recv(fd, buf, sizeof(buf), 0);
struct nlmsghdr *m;
if (len <= 0)
break;
for (m = (struct nlmsghdr *)buf; NLMSG_OK(m, len);
m = NLMSG_NEXT(m, len)) {
struct nlattr *a;
int rem, idx = -1;
char nm[64] = "";
if (m->nlmsg_type == NLMSG_DONE)
goto out;
if (m->nlmsg_type == NLMSG_ERROR) {
struct nlmsgerr *e = NLMSG_DATA(m);
fprintf(stderr, "[dbg] dump err=%d\n", e->error);
goto out;
}
a = (struct nlattr *)NLMSG_DATA(m);
rem = m->nlmsg_len - NLMSG_HDRLEN;
for (; rem > 0 && (int)a->nla_len <= rem;
rem -= NLA_ALIGN(a->nla_len),
a = (struct nlattr *)((char *)a + NLA_ALIGN(a->nla_len))) {
if (a->nla_type == RDMA_NLDEV_ATTR_DEV_INDEX)
idx = *(unsigned int *)((char *)a + NLA_HDRLEN);
else if (a->nla_type == RDMA_NLDEV_ATTR_DEV_NAME)
snprintf(nm, sizeof(nm), "%s",
(char *)a + NLA_HDRLEN);
}
if (idx >= 0 && !strcmp(nm, name))
found = idx;
else if (idx == 0 && !strcmp(nm, name))
found = 0;
}
}
out:
return found;
}
static int rdma_dellink(int fd, int index)
{
char buf[256];
struct nlmsghdr *nlh = (struct nlmsghdr *)buf;
int off = NLMSG_LENGTH(0);
memset(buf, 0, sizeof(buf));
nlh->nlmsg_len = 0;
nlh->nlmsg_type = RDMA_NL_GET_TYPE(RDMA_NL_NLDEV, RDMA_NLDEV_CMD_DELLINK);
nlh->nlmsg_flags = NLM_F_REQUEST | NLM_F_ACK;
nlh->nlmsg_seq = 2;
nla_put(buf, &off, RDMA_NLDEV_ATTR_DEV_INDEX, &index, sizeof(index));
nlh->nlmsg_len = off;
if (send(fd, buf, off, 0) < 0)
return -1;
/* read the ACK */
if (recv(fd, buf, sizeof(buf), 0) > 0) {
struct nlmsghdr *m = (struct nlmsghdr *)buf;
if (m->nlmsg_type == NLMSG_ERROR) {
struct nlmsgerr *e = NLMSG_DATA(m);
return e->error ? -1 : 0;
}
}
return 0;
}
int main(int argc, char **argv)
{
const char *ifname = argv[1]; /* rxe device, e.g. rxe0 */
const char *ibname = argv[2]; /* ipoib iface, e.g. ib0 */
char tname[80];
long long offset_ns = atoll(argv[3]);
FILE *f;
char line[512];
unsigned long long start = 0, walk = 0, period = 0, prev_start = 0;
int nlfd;
snprintf(tname, sizeof(tname), "reap_ah: %s ", ibname);
nlfd = socket(AF_NETLINK, SOCK_RAW, NETLINK_RDMA);
if (nlfd < 0) {
perror("socket NETLINK_RDMA");
return 1;
}
f = fopen(TRACE_PIPE, "r");
if (!f) {
perror("open trace_pipe");
return 1;
}
fprintf(stderr, "[*] trigger: dev=%s offset=%lldns\n", ifname, offset_ns);
while (fgets(line, sizeof(line), f)) {
char *s, *w;
if (!strstr(line, tname))
continue;
s = strstr(line, "start=");
w = strstr(line, "walk_ns=");
if (!s || !w)
continue;
start = strtoull(s + 6, NULL, 10);
walk = strtoull(w + 8, NULL, 10);
if (!start || !walk)
continue;
if (prev_start)
period = start - prev_start;
prev_start = start;
if (!period || !walk)
continue;
unsigned long long fire = start + period + walk + offset_ns;
unsigned long long t0 = now_ns();
if (fire <= t0)
continue;
fprintf(stderr, "[*] next gap in %llu ns (period %llu, walk %llu)\n",
fire - t0, period, walk);
wait_until(fire);
unsigned long long fired = now_ns();
int idx = rdma_dev_index(nlfd, ifname);
int rc = idx >= 0 ? rdma_dellink(nlfd, idx) : -1;
fprintf(stderr, "[+] fired DELLINK idx=%d rc=%d at=%llu drift=%lldns\n",
idx, rc, fired, (long long)(fired - fire));
fflush(stderr);
/* closed-loop: read the trace buffer, find the gap_enter and
* STOP the del produced, report how far off we landed */
usleep(1500000);
FILE *tf = fopen("/sys/kernel/tracing/trace", "r");
if (tf) {
char l2[512];
unsigned long long ge = 0, stop = 0;
while (fgets(l2, sizeof(l2), tf)) {
char *p;
if (strstr(l2, tname) && (p = strstr(l2, "gap_enter=")))
ge = strtoull(p + 10, NULL, 10);
if (strstr(l2, tname) && (p = strstr(l2, "STOP at=")))
stop = strtoull(p + 8, NULL, 10);
}
fclose(tf);
if (ge && stop) {
long long miss = (long long)(stop - ge);
fprintf(stderr, "[=] MISS-BY=%lld (stop-gap_enter, want 0..200000)\n", miss);
printf("%lld\n", miss); /* stdout for the driver */
}
}
return rc != 0;
}
return 1;
}
------------------------- guest scripts -------------------------
----- setup.sh -----
#!/bin/bash
# setup.sh - F-026 guest setup: two rxe endpoints presenting as IB, ipoib up.
# NICs: enp0s3 = slirp (ssh/internet); enp0s4+enp0s5 = same L2 hub (ipoib pair).
set -x
mount -t configfs none /sys/kernel/config 2>/dev/null
mount -t tracefs none /sys/kernel/tracing 2>/dev/null
ip link set enp0s4 up
ip link set enp0s5 up
rdma link add rxe0 type rxe netdev enp0s4
rdma link add rxe1 type rxe netdev enp0s5
sleep 1
ip addr add 10.99.0.1/24 dev ib0
ip link set ib0 up
ip addr add 10.99.0.2/24 dev ib1
ip link set ib1 up
sleep 1
ip -brief addr show ib0
ip -brief addr show ib1
ping -c2 -W2 10.99.0.2 -I ib0 && echo PING-OK
ip neigh show dev ib0
echo SETUP-DONE
----- churn3.sh -----
#!/bin/bash
# churn3.sh - fast AH churn on ib0 (static PERMANENT neighs + one UDP each,
# then explicit neigh del -> AHs to dead_ahs). ib0 must have a /16 alias.
N=${1:-500}
LOOPS=${2:-9999}
ip addr add 10.99.0.1/16 dev ib0 2>/dev/null # widen the on-link subnet
mklladdr() {
local i=$1
printf '80:00:00:10:52:54:00:12:%02x:%02x:00:00:00:00:00:00:00:%02x:%02x:01' \
$(( (i >> 8) & 255 )) $(( i & 255 )) $(( (i >> 8) & 255 )) $(( i & 255 ))
}
for l in $(seq 1 $LOOPS); do
for i in $(seq 1 $N); do
a=10.99.$(( i >> 8 )).$(( i & 255 ))
ip neigh replace $a lladdr $(mklladdr $i) dev ib0 2>/dev/null
done
/root/churn_send16 $N 2>/dev/null || /root/churn_send16 $N
for i in $(seq 1 $N); do
ip neigh del 10.99.$(( i >> 8 )).$(( i & 255 )) dev ib0 2>/dev/null
done
echo "loop $l done ($(date +%T))"
done
----- churn_send16.c -----
// SPDX-License-Identifier: GPL-2.0
/*
* churn_send16.c - one UDP datagram at each 10.99.Hi.Lo neighbor on ib0
* so ipoib creates a real AH per neighbor (path lookup -> SA shim -> rxe AH).
*
* usage: churn_send16 <count>
*/
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <arpa/inet.h>
#include <sys/socket.h>
int main(int argc, char **argv)
{
int n = argc > 1 ? atoi(argv[1]) : 500;
int fd = socket(AF_INET, SOCK_DGRAM, 0);
char buf[64] = "f026";
struct sockaddr_in sa;
int i, sent = 0;
if (fd < 0) {
perror("socket");
return 1;
}
memset(&sa, 0, sizeof(sa));
sa.sin_family = AF_INET;
sa.sin_port = htons(9999);
for (i = 1; i <= n; i++) {
char ip[32];
snprintf(ip, sizeof(ip), "10.99.%d.%d", (i >> 8) & 255, i & 255);
inet_pton(AF_INET, ip, &sa.sin_addr);
if (sendto(fd, buf, sizeof(buf), 0,
(struct sockaddr *)&sa, sizeof(sa)) >= 0)
sent++;
}
close(fd);
printf("churn_send16: fired %d datagrams\n", sent);
return 0;
}
----- race.sh -----
#!/bin/bash
# race.sh - F-026 closed-loop sweep: trigger prints MISS-BY (where the
# teardown's stop landed vs the reaper gap), we steer the next offset to
# land 100us into the 200us widened gap.
IF=ib0
off=-4000000 # first guess: syscall latency is ~3.9ms
MAX=${1:-300}
for i in $(seq 1 $MAX); do
rdma link del rxe0 2>/dev/null; sleep 0.5
rdma link add rxe0 type rxe netdev enp0s4; sleep 1
ip addr add 10.99.0.1/16 dev $IF 2>/dev/null; ip link set $IF up; sleep 1
echo > /sys/kernel/tracing/trace
/root/churn3.sh 300 1 >/dev/null 2>&1
miss=$(/root/trigger rxe0 $IF $off 2>/dev/null | tail -1)
case "$miss" in ''|*[!0-9-]*)
echo "shot $i: off=$off no-measurement"
continue ;;
esac
echo "shot $i: off=$off miss=$miss"
off=$(( off - miss + 100000 ))
if grep -q 'ESCAPED-ARM' /sys/kernel/tracing/trace || dmesg | grep -qE 'BUG: KASAN|Oops'; then
echo "=== HIT at off=$off (shot $i) ==="
grep -E 'reap_ah|ESCAPED' /sys/kernel/tracing/trace | tail -20
dmesg | grep -E 'KASAN|Oops|Call Trace|RIP' | tail -30
echo "HIT-OFFSET=$off shot=$i" > /root/HIT
poweroff -f
fi
done
echo "done $MAX shots, no hit"
drivers/infiniband/ulp/ipoib/ipoib_ib.c | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/drivers/infiniband/ulp/ipoib/ipoib_ib.c b/drivers/infiniband/ulp/ipoib/ipoib_ib.c
index 81bbb3f7c113..5ed4f709cb40 100644
--- a/drivers/infiniband/ulp/ipoib/ipoib_ib.c
+++ b/drivers/infiniband/ulp/ipoib/ipoib_ib.c
@@ -736,11 +736,11 @@ static void ipoib_start_ah_reaper(struct ipoib_dev_priv *priv)
static void ipoib_stop_ah_reaper(struct ipoib_dev_priv *priv)
{
set_bit(IPOIB_STOP_REAPER, &priv->flags);
- cancel_delayed_work(&priv->ah_reap_task);
+ cancel_delayed_work_sync(&priv->ah_reap_task);
/*
* After ipoib_stop_ah_reaper() we always go through
- * ipoib_reap_dead_ahs() which ensures the work is really stopped and
- * does a final flush out of the dead_ah's list
+ * ipoib_reap_dead_ahs() which does a final flush out of the
+ * dead_ah's list
*/
}
--
2.25.1
^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [PATCH v3] RDMA/ipoib: cancel the AH reaper synchronously on teardown
2026-11-05 4:29 [PATCH v3] RDMA/ipoib: cancel the AH reaper synchronously on teardown Chengyu Liu
@ 2026-11-05 4:34 ` sashiko-bot
2026-11-05 19:38 ` Leon Romanovsky
1 sibling, 0 replies; 4+ messages in thread
From: sashiko-bot @ 2026-11-05 4:34 UTC (permalink / raw)
To: Chengyu Liu; +Cc: linux-rdma
> ipoib_stop_ah_reaper() uses cancel_delayed_work(), which cannot stop a
> callback that is already running. ipoib_reap_ah() re-arms itself after
> an unlocked test_bit(IPOIB_STOP_REAPER) check, so a callback that has
> passed the check but has not re-armed yet still queues the delayed work
> after the "stop":
>
> CPU0 (reaper) CPU1 (stop/teardown)
> test_bit(STOP_REAPER) == 0
> set_bit(STOP_REAPER)
> cancel_delayed_work() (no-op)
> ... destroy_workqueue(priv->wq)
> queue_delayed_work(priv->wq, ...) <- delayed work sits on the timer
>
> destroy_workqueue() cannot see a delayed work that is pending on the
> timer wheel, so the re-armed work fires after the workqueue is
> [ ... ]
> escapes the stop and re-arms, the timer fires one second later in
> __run_timer_base() on the destroyed workqueue (slab-use-after-free).
>
> Cc: stable@vger.kernel.org
> Signed-off-by: Chengyu Liu <lui2012qwq@gmail.com>
Sashiko has reviewed this patch and found no issues.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261105042922.14733-1-lui2012qwq@gmail.com?part=1
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PATCH v3] RDMA/ipoib: cancel the AH reaper synchronously on teardown
2026-11-05 4:29 [PATCH v3] RDMA/ipoib: cancel the AH reaper synchronously on teardown Chengyu Liu
2026-11-05 4:34 ` sashiko-bot
@ 2026-11-05 19:38 ` Leon Romanovsky
2026-11-06 4:42 ` Chengyu Liu
1 sibling, 1 reply; 4+ messages in thread
From: Leon Romanovsky @ 2026-11-05 19:38 UTC (permalink / raw)
To: Chengyu Liu; +Cc: jgg, linux-rdma, stable
On Thu, Nov 05, 2026 at 12:29:22PM +0800, Chengyu Liu wrote:
> ipoib_stop_ah_reaper() uses cancel_delayed_work(), which cannot stop a
> callback that is already running. ipoib_reap_ah() re-arms itself after
> an unlocked test_bit(IPOIB_STOP_REAPER) check, so a callback that has
> passed the check but has not re-armed yet still queues the delayed work
> after the "stop":
>
> CPU0 (reaper) CPU1 (stop/teardown)
> test_bit(STOP_REAPER) == 0
> set_bit(STOP_REAPER)
> cancel_delayed_work() (no-op)
> ... destroy_workqueue(priv->wq)
> queue_delayed_work(priv->wq, ...) <- delayed work sits on the timer
>
> destroy_workqueue() cannot see a delayed work that is pending on the
> timer wheel, so the re-armed work fires after the workqueue is
> destroyed and priv is freed: both the queueing onto the dead queue and
> the callback's container_of() on freed memory are use-after-free.
>
> Use cancel_delayed_work_sync(), which waits for a running callback and
> swallows its re-arm while the stop is in flight. Note that
> ipoib_stop_ah_reaper() also runs on ndo_stop and on the stop/open cycle
> of a heavy flush, so the work must be left enabled for
> ipoib_start_ah_reaper() to re-arm it on the next open;
> cancel_delayed_work_sync() re-enables on the way out, unlike
> disable_delayed_work_sync() which would kill the reaper for good.
>
> The comment above claims ipoib_reap_dead_ahs() "ensures the work is
> really stopped", but it never touches the work item - it only walks
> the dead_ahs list under priv->lock. The stop guarantee comes from the
> cancel; fix the comment along with it.
>
> Found by static analysis. Reproduced on a KASAN build running stock
> ib_ipoib on a Soft-RoCE transport (rxe): when a running reaper callback
> escapes the stop and re-arms, the timer fires one second later in
> __run_timer_base() on the destroyed workqueue (slab-use-after-free).
Please add full dmesg kernel stack here.
Thanks
>
> Cc: stable@vger.kernel.org
> Signed-off-by: Chengyu Liu <lui2012qwq@gmail.com>
> ---
> v3:
> - back to cancel_delayed_work_sync(): ipoib_stop_ah_reaper() also runs
> on ndo_stop and on the heavy-flush stop/open cycle, so the work must
> be left enabled (sashiko review); disable_delayed_work_sync() would
> kill the reaper on the next open
>
> v2: https://lore.kernel.org/r/20261104181753.1245369-1-lui2012qwq@gmail.com
> - add reproducer on stock ib_ipoib over rxe
> - switch to disable_delayed_work_sync() (superseded by v3)
>
> v1: https://lore.kernel.org/r/20261103170703.1600310-1-lui2012qwq@gmail.com
>
>
> Reproducer (harness summary; full driver scripts and the timing trigger
> below):
>
> QEMU has no IB hardware, so the stock ib_ipoib driver runs over a
> Soft-RoCE (rxe) transport presented as an IB port. Harness shims,
> all below ib_ipoib (the driver itself is unmodified):
>
> * rxe_verbs.c: report link layer INFINIBAND and cap flags
> PROT_IB|IB_MAD|AF_IB from rxe_get_link_layer() /
> rxe_port_immutable(); skip ib_get_eth_speed() for the flipped
> port in rxe_query_port(); set dev.parent before
> ib_register_device().
> * sa_query.c: answer ib_sa_path_rec_get() locally (async via
> system_wq) when the device has no SA.
> * multicast.c: complete ib_sa_join_multicast() locally (async);
> tag the synthetic object (comp_mask bit 63) so
> ib_sa_free_multicast() frees it.
> * ipoib_ib.c (witness only, no behavior change): trace_printk()
> timing in ipoib_reap_ah(), and an ESCAPED-ARM line when the
> callback re-arms after IPOIB_STOP_REAPER was set.
>
> With ~500 AH entries churned into dead_ahs, the reaper's walk
> measures 20-160us once a second; the teardown (RDMA netlink
> DELLINK of the rxe device) reaches ipoib_stop_ah_reaper() ~3.9ms
> after the syscall enters. The stock test->queue window is the
> handful of instructions between the lockless test_bit() and the
> queue_delayed_work(); to hit it deterministically the PoC kernel
> widens it to 200us with a module parameter. On a hit the witness
> logs
>
> ipoib_stop_ah_reaper: reap_ah: ib0 STOP at=531116547240
> ipoib_reap_ah: reap_ah: ib0 ESCAPED-ARM after stop
>
> and one second later the escaped timer fires on the destroyed
> workqueue (trimmed):
>
> BUG: KASAN: use-after-free in __run_timer_base.part.0+0x80c/0x900
> Write of size 8 at addr ffff888021088628 by task swapper/0/0
> Call Trace:
> <IRQ>
> kasan_report+0x96/0xd0
> __run_timer_base.part.0+0x80c/0x900
> tmigr_handle_remote_up+0x38b/0x830
> ...
> handle_softirqs+0x184/0x5c0
> irq_exit_rcu+0x6e/0x150
> sysvec_apic_timer_interrupt+0x70/0x80
> </IRQ>
>
> with the workqueue core reporting, right before the splat:
>
> workqueue: cannot queue ipoib_reap_ah on wq ipoib_wq
>
>
> ------------------------- reproducer: trigger.c -------------------------
> // SPDX-License-Identifier: GPL-2.0
> /*
> * trigger.c - F-026 teardown timing trigger.
> *
> * Watches the PoC kernel's ipoib_reap_ah() witness lines on trace_pipe
> * ("reap_ah: ib0 start=<ns> walk_ns=<ns>"), predicts the next reaper run
> * (~1s period, jiffy-aligned) and deletes the rxe device via RDMA netlink
> * (RDMA_NLDEV_CMD_DELLINK, the real teardown path: ib unregister -> ipoib
> * remove -> stop_ah_reaper -> destroy_workqueue -> free) at
> *
> * next_start + walk_ns + offset
> *
> * which is the callback's test_bit->queue_delayed_work gap.
> *
> * usage: trigger <rxe_dev> <ib_dev> <offset_ns>
> */
> #define _GNU_SOURCE
> #include <stdio.h>
> #include <stdlib.h>
> #include <string.h>
> #include <unistd.h>
> #include <fcntl.h>
> #include <time.h>
> #include <errno.h>
> #include <linux/netlink.h>
> #include <sys/socket.h>
>
> #ifndef NETLINK_RDMA
> #define NETLINK_RDMA 20
> #endif
>
> #define RDMA_NL_NLDEV 5
> #define RDMA_NL_GET_TYPE(c, o) (((c) << 10) + (o))
> #define RDMA_NLDEV_CMD_GET 1
> #define RDMA_NLDEV_CMD_DELLINK 4
> #define RDMA_NLDEV_ATTR_DEV_INDEX 1
> #define RDMA_NLDEV_ATTR_DEV_NAME 2
>
> static const char *TRACE_PIPE = "/sys/kernel/tracing/trace_pipe";
>
> static unsigned long long now_ns(void)
> {
> struct timespec ts;
> clock_gettime(CLOCK_MONOTONIC, &ts);
> return (unsigned long long)ts.tv_sec * 1000000000ull + ts.tv_nsec;
> }
>
> static void wait_until(unsigned long long deadline_ns)
> {
> while (now_ns() < deadline_ns)
> ;
> }
>
> static int nla_put(char *buf, int *off, int type, const void *data, int len)
> {
> struct nlattr *na = (struct nlattr *)(buf + *off);
> na->nla_type = type;
> na->nla_len = NLA_HDRLEN + len;
> memcpy(buf + *off + NLA_HDRLEN, data, len);
> *off += NLA_ALIGN(na->nla_len);
> return 0;
> }
>
> /* dump all rdma links, return the index of `name` (0 if not found) */
> static int rdma_dev_index(int fd, const char *name)
> {
> char buf[4096];
> struct nlmsghdr *nlh = (struct nlmsghdr *)buf;
> int found = -1;
> struct timeval tv = { .tv_sec = 2, .tv_usec = 0 };
>
> setsockopt(fd, SOL_SOCKET, SO_RCVTIMEO, &tv, sizeof(tv));
> memset(buf, 0, sizeof(buf));
> nlh->nlmsg_len = NLMSG_LENGTH(0);
> nlh->nlmsg_type = RDMA_NL_GET_TYPE(RDMA_NL_NLDEV, RDMA_NLDEV_CMD_GET);
> nlh->nlmsg_flags = NLM_F_REQUEST | NLM_F_DUMP;
> nlh->nlmsg_seq = 1;
> if (send(fd, buf, nlh->nlmsg_len, 0) < 0)
> return 0;
>
> for (;;) {
> ssize_t len = recv(fd, buf, sizeof(buf), 0);
> struct nlmsghdr *m;
> if (len <= 0)
> break;
> for (m = (struct nlmsghdr *)buf; NLMSG_OK(m, len);
> m = NLMSG_NEXT(m, len)) {
> struct nlattr *a;
> int rem, idx = -1;
> char nm[64] = "";
>
> if (m->nlmsg_type == NLMSG_DONE)
> goto out;
> if (m->nlmsg_type == NLMSG_ERROR) {
> struct nlmsgerr *e = NLMSG_DATA(m);
> fprintf(stderr, "[dbg] dump err=%d\n", e->error);
> goto out;
> }
> a = (struct nlattr *)NLMSG_DATA(m);
> rem = m->nlmsg_len - NLMSG_HDRLEN;
> for (; rem > 0 && (int)a->nla_len <= rem;
> rem -= NLA_ALIGN(a->nla_len),
> a = (struct nlattr *)((char *)a + NLA_ALIGN(a->nla_len))) {
> if (a->nla_type == RDMA_NLDEV_ATTR_DEV_INDEX)
> idx = *(unsigned int *)((char *)a + NLA_HDRLEN);
> else if (a->nla_type == RDMA_NLDEV_ATTR_DEV_NAME)
> snprintf(nm, sizeof(nm), "%s",
> (char *)a + NLA_HDRLEN);
> }
> if (idx >= 0 && !strcmp(nm, name))
> found = idx;
> else if (idx == 0 && !strcmp(nm, name))
> found = 0;
> }
> }
> out:
> return found;
> }
>
> static int rdma_dellink(int fd, int index)
> {
> char buf[256];
> struct nlmsghdr *nlh = (struct nlmsghdr *)buf;
> int off = NLMSG_LENGTH(0);
>
> memset(buf, 0, sizeof(buf));
> nlh->nlmsg_len = 0;
> nlh->nlmsg_type = RDMA_NL_GET_TYPE(RDMA_NL_NLDEV, RDMA_NLDEV_CMD_DELLINK);
> nlh->nlmsg_flags = NLM_F_REQUEST | NLM_F_ACK;
> nlh->nlmsg_seq = 2;
> nla_put(buf, &off, RDMA_NLDEV_ATTR_DEV_INDEX, &index, sizeof(index));
> nlh->nlmsg_len = off;
>
> if (send(fd, buf, off, 0) < 0)
> return -1;
> /* read the ACK */
> if (recv(fd, buf, sizeof(buf), 0) > 0) {
> struct nlmsghdr *m = (struct nlmsghdr *)buf;
> if (m->nlmsg_type == NLMSG_ERROR) {
> struct nlmsgerr *e = NLMSG_DATA(m);
> return e->error ? -1 : 0;
> }
> }
> return 0;
> }
>
> int main(int argc, char **argv)
> {
> const char *ifname = argv[1]; /* rxe device, e.g. rxe0 */
> const char *ibname = argv[2]; /* ipoib iface, e.g. ib0 */
> char tname[80];
> long long offset_ns = atoll(argv[3]);
> FILE *f;
> char line[512];
> unsigned long long start = 0, walk = 0, period = 0, prev_start = 0;
> int nlfd;
>
> snprintf(tname, sizeof(tname), "reap_ah: %s ", ibname);
>
> nlfd = socket(AF_NETLINK, SOCK_RAW, NETLINK_RDMA);
> if (nlfd < 0) {
> perror("socket NETLINK_RDMA");
> return 1;
> }
>
> f = fopen(TRACE_PIPE, "r");
> if (!f) {
> perror("open trace_pipe");
> return 1;
> }
>
> fprintf(stderr, "[*] trigger: dev=%s offset=%lldns\n", ifname, offset_ns);
>
> while (fgets(line, sizeof(line), f)) {
> char *s, *w;
> if (!strstr(line, tname))
> continue;
> s = strstr(line, "start=");
> w = strstr(line, "walk_ns=");
> if (!s || !w)
> continue;
> start = strtoull(s + 6, NULL, 10);
> walk = strtoull(w + 8, NULL, 10);
> if (!start || !walk)
> continue;
> if (prev_start)
> period = start - prev_start;
> prev_start = start;
> if (!period || !walk)
> continue;
>
> unsigned long long fire = start + period + walk + offset_ns;
> unsigned long long t0 = now_ns();
> if (fire <= t0)
> continue;
> fprintf(stderr, "[*] next gap in %llu ns (period %llu, walk %llu)\n",
> fire - t0, period, walk);
> wait_until(fire);
> unsigned long long fired = now_ns();
> int idx = rdma_dev_index(nlfd, ifname);
> int rc = idx >= 0 ? rdma_dellink(nlfd, idx) : -1;
> fprintf(stderr, "[+] fired DELLINK idx=%d rc=%d at=%llu drift=%lldns\n",
> idx, rc, fired, (long long)(fired - fire));
> fflush(stderr);
>
> /* closed-loop: read the trace buffer, find the gap_enter and
> * STOP the del produced, report how far off we landed */
> usleep(1500000);
> FILE *tf = fopen("/sys/kernel/tracing/trace", "r");
> if (tf) {
> char l2[512];
> unsigned long long ge = 0, stop = 0;
> while (fgets(l2, sizeof(l2), tf)) {
> char *p;
> if (strstr(l2, tname) && (p = strstr(l2, "gap_enter=")))
> ge = strtoull(p + 10, NULL, 10);
> if (strstr(l2, tname) && (p = strstr(l2, "STOP at=")))
> stop = strtoull(p + 8, NULL, 10);
> }
> fclose(tf);
> if (ge && stop) {
> long long miss = (long long)(stop - ge);
> fprintf(stderr, "[=] MISS-BY=%lld (stop-gap_enter, want 0..200000)\n", miss);
> printf("%lld\n", miss); /* stdout for the driver */
> }
> }
> return rc != 0;
> }
> return 1;
> }
>
> ------------------------- guest scripts -------------------------
>
> ----- setup.sh -----
> #!/bin/bash
> # setup.sh - F-026 guest setup: two rxe endpoints presenting as IB, ipoib up.
> # NICs: enp0s3 = slirp (ssh/internet); enp0s4+enp0s5 = same L2 hub (ipoib pair).
> set -x
> mount -t configfs none /sys/kernel/config 2>/dev/null
> mount -t tracefs none /sys/kernel/tracing 2>/dev/null
>
> ip link set enp0s4 up
> ip link set enp0s5 up
>
> rdma link add rxe0 type rxe netdev enp0s4
> rdma link add rxe1 type rxe netdev enp0s5
> sleep 1
>
> ip addr add 10.99.0.1/24 dev ib0
> ip link set ib0 up
> ip addr add 10.99.0.2/24 dev ib1
> ip link set ib1 up
> sleep 1
> ip -brief addr show ib0
> ip -brief addr show ib1
>
> ping -c2 -W2 10.99.0.2 -I ib0 && echo PING-OK
> ip neigh show dev ib0
> echo SETUP-DONE
>
> ----- churn3.sh -----
> #!/bin/bash
> # churn3.sh - fast AH churn on ib0 (static PERMANENT neighs + one UDP each,
> # then explicit neigh del -> AHs to dead_ahs). ib0 must have a /16 alias.
> N=${1:-500}
> LOOPS=${2:-9999}
>
> ip addr add 10.99.0.1/16 dev ib0 2>/dev/null # widen the on-link subnet
>
> mklladdr() {
> local i=$1
> printf '80:00:00:10:52:54:00:12:%02x:%02x:00:00:00:00:00:00:00:%02x:%02x:01' \
> $(( (i >> 8) & 255 )) $(( i & 255 )) $(( (i >> 8) & 255 )) $(( i & 255 ))
> }
>
> for l in $(seq 1 $LOOPS); do
> for i in $(seq 1 $N); do
> a=10.99.$(( i >> 8 )).$(( i & 255 ))
> ip neigh replace $a lladdr $(mklladdr $i) dev ib0 2>/dev/null
> done
> /root/churn_send16 $N 2>/dev/null || /root/churn_send16 $N
> for i in $(seq 1 $N); do
> ip neigh del 10.99.$(( i >> 8 )).$(( i & 255 )) dev ib0 2>/dev/null
> done
> echo "loop $l done ($(date +%T))"
> done
>
> ----- churn_send16.c -----
> // SPDX-License-Identifier: GPL-2.0
> /*
> * churn_send16.c - one UDP datagram at each 10.99.Hi.Lo neighbor on ib0
> * so ipoib creates a real AH per neighbor (path lookup -> SA shim -> rxe AH).
> *
> * usage: churn_send16 <count>
> */
> #include <stdio.h>
> #include <stdlib.h>
> #include <string.h>
> #include <unistd.h>
> #include <arpa/inet.h>
> #include <sys/socket.h>
>
> int main(int argc, char **argv)
> {
> int n = argc > 1 ? atoi(argv[1]) : 500;
> int fd = socket(AF_INET, SOCK_DGRAM, 0);
> char buf[64] = "f026";
> struct sockaddr_in sa;
> int i, sent = 0;
>
> if (fd < 0) {
> perror("socket");
> return 1;
> }
>
> memset(&sa, 0, sizeof(sa));
> sa.sin_family = AF_INET;
> sa.sin_port = htons(9999);
>
> for (i = 1; i <= n; i++) {
> char ip[32];
> snprintf(ip, sizeof(ip), "10.99.%d.%d", (i >> 8) & 255, i & 255);
> inet_pton(AF_INET, ip, &sa.sin_addr);
> if (sendto(fd, buf, sizeof(buf), 0,
> (struct sockaddr *)&sa, sizeof(sa)) >= 0)
> sent++;
> }
> close(fd);
> printf("churn_send16: fired %d datagrams\n", sent);
> return 0;
> }
>
> ----- race.sh -----
> #!/bin/bash
> # race.sh - F-026 closed-loop sweep: trigger prints MISS-BY (where the
> # teardown's stop landed vs the reaper gap), we steer the next offset to
> # land 100us into the 200us widened gap.
> IF=ib0
> off=-4000000 # first guess: syscall latency is ~3.9ms
> MAX=${1:-300}
>
> for i in $(seq 1 $MAX); do
> rdma link del rxe0 2>/dev/null; sleep 0.5
> rdma link add rxe0 type rxe netdev enp0s4; sleep 1
> ip addr add 10.99.0.1/16 dev $IF 2>/dev/null; ip link set $IF up; sleep 1
> echo > /sys/kernel/tracing/trace
> /root/churn3.sh 300 1 >/dev/null 2>&1
>
> miss=$(/root/trigger rxe0 $IF $off 2>/dev/null | tail -1)
> case "$miss" in ''|*[!0-9-]*)
> echo "shot $i: off=$off no-measurement"
> continue ;;
> esac
> echo "shot $i: off=$off miss=$miss"
> off=$(( off - miss + 100000 ))
>
> if grep -q 'ESCAPED-ARM' /sys/kernel/tracing/trace || dmesg | grep -qE 'BUG: KASAN|Oops'; then
> echo "=== HIT at off=$off (shot $i) ==="
> grep -E 'reap_ah|ESCAPED' /sys/kernel/tracing/trace | tail -20
> dmesg | grep -E 'KASAN|Oops|Call Trace|RIP' | tail -30
> echo "HIT-OFFSET=$off shot=$i" > /root/HIT
> poweroff -f
> fi
> done
> echo "done $MAX shots, no hit"
>
> drivers/infiniband/ulp/ipoib/ipoib_ib.c | 6 +++---
> 1 file changed, 3 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/infiniband/ulp/ipoib/ipoib_ib.c b/drivers/infiniband/ulp/ipoib/ipoib_ib.c
> index 81bbb3f7c113..5ed4f709cb40 100644
> --- a/drivers/infiniband/ulp/ipoib/ipoib_ib.c
> +++ b/drivers/infiniband/ulp/ipoib/ipoib_ib.c
> @@ -736,11 +736,11 @@ static void ipoib_start_ah_reaper(struct ipoib_dev_priv *priv)
> static void ipoib_stop_ah_reaper(struct ipoib_dev_priv *priv)
> {
> set_bit(IPOIB_STOP_REAPER, &priv->flags);
> - cancel_delayed_work(&priv->ah_reap_task);
> + cancel_delayed_work_sync(&priv->ah_reap_task);
> /*
> * After ipoib_stop_ah_reaper() we always go through
> - * ipoib_reap_dead_ahs() which ensures the work is really stopped and
> - * does a final flush out of the dead_ah's list
> + * ipoib_reap_dead_ahs() which does a final flush out of the
> + * dead_ah's list
> */
> }
>
> --
> 2.25.1
>
>
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PATCH v3] RDMA/ipoib: cancel the AH reaper synchronously on teardown
2026-11-05 19:38 ` Leon Romanovsky
@ 2026-11-06 4:42 ` Chengyu Liu
0 siblings, 0 replies; 4+ messages in thread
From: Chengyu Liu @ 2026-11-06 4:42 UTC (permalink / raw)
To: Leon Romanovsky; +Cc: jgg, linux-rdma, stable
On Thu, Nov 05, 2026, Leon Romanovsky wrote:
> Please add full dmesg kernel stack here.
Full dmesg of the crash below.
The reaper escaped the teardown, and its timer fires exactly one
second later:
How to reproduce : rxe devices presented as IB so stock ib_ipoib
attaches; churn ~500 AHs into dead_ahs via static neighbors; then
RDMA_NLDEV_CMD_DELLINK the rxe device timed against the reaper's
per-second run - when the running callback has passed the STOP_REAPER
check but not yet re-armed, the cancel misses it and the re-arm lands
on the dead workqueue. The stock test->re-arm gap is a handful of
instructions, so the PoC kernel widens it with a module parameter
(200us in the run below) to make the hit deterministic. The
measured stock timing (reaper walk 20-160us once a second, DELLINK
to ipoib_stop_ah_reaper ~3.9ms) is in the patch's reproducer
section.
[ 729.356910]
==================================================================
[ 729.356921] BUG: KASAN: use-after-free in
__run_timer_base.part.0+0x80c/0x900
[ 729.357095] Write of size 8 at addr ffff888021088628 by task swapper/0/0
[ 729.357097]
[ 729.357113] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted
7.3.0-rc7-00032-g0fcb05b72344-dirty #9 PREEMPT(lazy)
[ 729.357116] Hardware name: QEMU Standard PC (i440FX + PIIX,
1996), BIOS 1.13.0-1ubuntu1.1 04/01/2014
[ 729.357133] Call Trace:
[ 729.357180] <IRQ>
[ 729.357196] dump_stack_lvl+0x55/0x70
[ 729.357375] print_report+0x10d/0x5d0
[ 729.357426] ? __run_timer_base.part.0+0x80c/0x900
[ 729.357428] kasan_report+0x96/0xd0
[ 729.357430] ? __run_timer_base.part.0+0x80c/0x900
[ 729.357433] __run_timer_base.part.0+0x80c/0x900
[ 729.357436] ? __pfx___run_timer_base.part.0+0x10/0x10
[ 729.357438] ? tmigr_update_events+0x285/0x770
[ 729.357476] ? _raw_spin_lock+0x51/0xe0
[ 729.357500] ? _raw_spin_lock_irq+0x86/0xe0
[ 729.357501] ? __pfx__raw_spin_lock_irq+0x10/0x10
[ 729.357504] tmigr_handle_remote_up+0x38b/0x830
[ 729.357507] ? __pfx_run_posix_cpu_timers+0x10/0x10
[ 729.357509] ? sched_balance_trigger+0x803/0xd50
[ 729.357521] ? __pfx_tmigr_handle_remote_up+0x10/0x10
[ 729.357524] ? __pfx__raw_spin_lock+0x10/0x10
[ 729.357526] ? call_timer_fn+0x27/0x210
[ 729.357528] ? __run_timer_base.part.0+0xa7/0x900
[ 729.357530] ? __pfx_tmigr_handle_remote_up+0x10/0x10
[ 729.357533] __walk_groups_from+0x48/0x160
[ 729.357535] tmigr_handle_remote+0x1fb/0x2f0
[ 729.357537] ? __pfx_tmigr_handle_remote+0x10/0x10
[ 729.357539] ? hrtimer_update_next_event+0x8a/0x100
[ 729.357542] handle_softirqs+0x184/0x5c0
[ 729.357576] ? __hrtimer_rearm_deferred+0x15d/0x460
[ 729.357578] irq_exit_rcu+0x6e/0x150
[ 729.357580] sysvec_apic_timer_interrupt+0x70/0x80
[ 729.357602] </IRQ>
[ 729.357603] <TASK>
[ 729.357603] asm_sysvec_apic_timer_interrupt+0x1a/0x20
[ 729.357623] RIP: 0010:pv_native_safe_halt+0xf/0x20
[ 729.357626] Code: fa e7 00 e9 d3 29 02 00 0f 1f 00 90 90 90 90
90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa eb 07 0f 00 2d 93 9a
19 00 fb f4 <c3> cc cc cc cc 66 2e 0f 1f 84 00 00 00 00 00 66 90 90 90
90 90 90
[ 729.357628] RSP: 0018:ffffffffb8a07d00 EFLAGS: 00000206
[ 729.357630] RAX: ffff8880e9ac8000 RBX: 0000000000000000 RCX:
ffffffffb7abb681
[ 729.357632] RDX: 0000000000000001 RSI: 0000000000000004 RDI:
0000000000071254
[ 729.357634] RBP: fffffbfff71428e8 R08: 000000aa3ae365f7 R09:
ffffed101480626b
[ 729.357638] R10: ffffed101480626a R11: ffff8880a4031353 R12:
ffffffffb8a14740
[ 729.357640] R13: dffffc0000000000 R14: ffffffffb8a14740 R15:
0000000000000000
[ 729.357641] ? ct_kernel_exit.constprop.0+0xa1/0xc0
[ 729.357644] default_idle+0x9/0x10
[ 729.357647] default_idle_call+0x39/0x60
[ 729.357649] do_idle+0x342/0x410
[ 729.357652] ? __pfx_do_idle+0x10/0x10
[ 729.357654] ? __pfx_do_set_mempolicy+0x10/0x10
[ 729.357656] cpu_startup_entry+0x54/0x60
[ 729.357659] rest_init+0x18e/0x1b0
[ 729.357661] ? __pfx_acpi_read_bit_register+0x10/0x10
[ 729.357766] ? __pfx_rest_init+0x10/0x10
[ 729.357769] ? __pfx_acpi_ut_trace+0x10/0x10
[ 729.357834] ? __pfx_kernel_init+0x10/0x10
[ 729.357837] ? acpi_enable_subsystem+0x24d/0x480
[ 729.357949] start_kernel+0x368/0x3c0
[ 729.357980] x86_64_start_reservations+0x18/0x30
[ 729.357996] x86_64_start_kernel+0xd9/0xe0
[ 729.357999] common_startup_64+0x13e/0x158
[ 729.358015] </TASK>
[ 729.358016]
[ 729.358017] The buggy address belongs to the physical page:
[ 729.358019] page: refcount:0 mapcount:0
mapping:0000000000000000 index:0xffff88802108b740 pfn:0x21088
[ 729.358021] flags: 0x100000000000000(node=0|zone=1)
[ 729.358025] raw: 0100000000000000 ffff8880a41bb1c0
ffff8880a41bb1c0 0000000000000000
[ 729.358026] raw: ffff88802108b740 0000000000000000
00000000ffffffff 0000000000000000
[ 729.358027] page dumped because: kasan: bad access detected
[ 729.358028]
[ 729.358028] Memory state around the buggy address:
[ 729.358030] ffff888021088500: ff ff ff ff ff ff ff ff ff ff ff
ff ff ff ff ff
[ 729.358031] ffff888021088580: ff ff ff ff ff ff ff ff ff ff ff
ff ff ff ff ff
[ 729.358032] >ffff888021088600: ff ff ff ff ff ff ff ff ff ff ff
ff ff ff ff ff
[ 729.358033] ^
[ 729.358034] ffff888021088680: ff ff ff ff ff ff ff ff ff ff ff
ff ff ff ff ff
[ 729.358035] ffff888021088700: ff ff ff ff ff ff ff ff ff ff ff
ff ff ff ff ff
[ 729.358036]
==================================================================
[ 729.358039] Disabling lock debugging due to kernel taint
[ 729.358115] ------------[ cut here ]------------
[ 729.358116] workqueue: cannot queue ipoib_reap_ah on wq ipoib_wq
[ 729.358122] WARNING: kernel/workqueue.c:2352 at
__queue_work+0x72f/0xd70, CPU#0: swapper/0/0
[ 729.436887] Modules linked in:
[ 729.437563] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G
B 7.3.0-rc7-00032-g0fcb05b72344-dirty #9 PREEMPT(lazy)
[ 729.439787] Tainted: [B]=BAD_PAGE
[ 729.440426] Hardware name: QEMU Standard PC (i440FX + PIIX,
1996), BIOS 1.13.0-1ubuntu1.1 04/01/2014
[ 729.442037] RIP: 0010:__queue_work+0x733/0xd70
[ 729.442960] Code: b8 00 00 00 48 b8 00 00 00 00 00 fc ff df 48
89 f9 48 c1 e9 03 80 3c 01 00 0f 85 0d 06 00 00 48 8d 3d 91 46 02 05
49 8b 77 18 <67> 48 0f b9 3a 48 b8 00 00 00 00 00 fc ff df 4c 89 fa 48
c1 ea 03
[ 729.446140] RSP: 0018:ffff8880a4008a68 EFLAGS: 00010082
[ 729.447180] RAX: 0000000000000000 RBX: 1ffff11014801158 RCX:
ffffffffb43d67fe
[ 729.448612] RDX: ffff88800fee30b8 RSI: ffffffffb6cbee40 RDI:
ffffffffb93fa880
[ 729.449832] RBP: ffffffffb43d6840 R08: 0000000000800345 R09:
ffff888021088628
[ 729.451014] R10: 0000000000000006 R11: ffffffffba5901c7 R12:
ffff88800fee3000
[ 729.452426] R13: ffff888021088620 R14: ffff8880a40a5c80 R15:
ffff888021088600
[ 729.453699] FS: 0000000000000000(0000)
GS:ffff8880e9ac8000(0000) knlGS:0000000000000000
[ 729.455129] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 729.456079] CR2: 0000564a390a7d60 CR3: 0000000007589000 CR4:
0000000000350ef0
[ 729.457674] Call Trace:
[ 729.458198] <IRQ>
[ 729.458571] ? __run_timer_base.part.0+0x80c/0x900
[ 729.459855] ? print_report+0x300/0x5d0
[ 729.460582] ? __pfx___queue_work+0x10/0x10
[ 729.461637] ? end_report+0x5e/0xe0
[ 729.462290] ? __pfx_delayed_work_timer_fn+0x10/0x10
[ 729.463169] ? __pfx_delayed_work_timer_fn+0x10/0x10
[ 729.464079] call_timer_fn+0x27/0x210
[ 729.464833] __run_timer_base.part.0+0x3cc/0x900
[ 729.465850] ? __pfx_delayed_work_timer_fn+0x10/0x10
[ 729.466771] ? __pfx___run_timer_base.part.0+0x10/0x10
[ 729.467843] ? tmigr_update_events+0x285/0x770
[ 729.468627] ? _raw_spin_lock+0x51/0xe0
[ 729.469248] ? _raw_spin_lock_irq+0x86/0xe0
[ 729.469960] ? __pfx__raw_spin_lock_irq+0x10/0x10
[ 729.470757] tmigr_handle_remote_up+0x38b/0x830
[ 729.471618] ? __pfx_run_posix_cpu_timers+0x10/0x10
[ 729.472491] ? sched_balance_trigger+0x803/0xd50
[ 729.473436] ? __pfx_tmigr_handle_remote_up+0x10/0x10
[ 729.474400] ? __pfx__raw_spin_lock+0x10/0x10
[ 729.475211] ? call_timer_fn+0x27/0x210
[ 729.476028] ? __run_timer_base.part.0+0xa7/0x900
[ 729.476759] ? __pfx_tmigr_handle_remote_up+0x10/0x10
[ 729.477575] __walk_groups_from+0x48/0x160
[ 729.478361] tmigr_handle_remote+0x1fb/0x2f0
[ 729.479169] ? __pfx_tmigr_handle_remote+0x10/0x10
[ 729.480049] ? hrtimer_update_next_event+0x8a/0x100
[ 729.480903] handle_softirqs+0x184/0x5c0
[ 729.481641] ? __hrtimer_rearm_deferred+0x15d/0x460
[ 729.482461] irq_exit_rcu+0x6e/0x150
[ 729.483206] sysvec_apic_timer_interrupt+0x70/0x80
[ 729.484058] </IRQ>
[ 729.484560] <TASK>
[ 729.484957] asm_sysvec_apic_timer_interrupt+0x1a/0x20
[ 729.485874] RIP: 0010:pv_native_safe_halt+0xf/0x20
[ 729.486678] Code: fa e7 00 e9 d3 29 02 00 0f 1f 00 90 90 90 90
90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa eb 07 0f 00 2d 93 9a
19 00 fb f4 <c3> cc cc cc cc 66 2e 0f 1f 84 00 00 00 00 00 66 90 90 90
90 90 90
[ 729.489844] RSP: 0018:ffffffffb8a07d00 EFLAGS: 00000206
[ 729.490707] RAX: ffff8880e9ac8000 RBX: 0000000000000000 RCX:
ffffffffb7abb681
[ 729.491941] RDX: 0000000000000001 RSI: 0000000000000004 RDI:
0000000000071254
[ 729.493334] RBP: fffffbfff71428e8 R08: 000000aa3ae365f7 R09:
ffffed101480626b
[ 729.494586] R10: ffffed101480626a R11: ffff8880a4031353 R12:
ffffffffb8a14740
[ 729.495895] R13: dffffc0000000000 R14: ffffffffb8a14740 R15:
0000000000000000
[ 729.497243] ? ct_kernel_exit.constprop.0+0xa1/0xc0
[ 729.498020] default_idle+0x9/0x10
[ 729.498610] default_idle_call+0x39/0x60
[ 729.499281] do_idle+0x342/0x410
[ 729.499884] ? __pfx_do_idle+0x10/0x10
[ 729.500523] ? __pfx_do_set_mempolicy+0x10/0x10
[ 729.501359] cpu_startup_entry+0x54/0x60
[ 729.501999] rest_init+0x18e/0x1b0
[ 729.502584] ? __pfx_acpi_read_bit_register+0x10/0x10
[ 729.503511] ? __pfx_rest_init+0x10/0x10
[ 729.504279] ? __pfx_acpi_ut_trace+0x10/0x10
[ 729.504980] ? __pfx_kernel_init+0x10/0x10
[ 729.505793] ? acpi_enable_subsystem+0x24d/0x480
[ 729.506549] start_kernel+0x368/0x3c0
[ 729.507208] x86_64_start_reservations+0x18/0x30
[ 729.508001] x86_64_start_kernel+0xd9/0xe0
[ 729.508710] common_startup_64+0x13e/0x158
[ 729.509407] </TASK>
[ 729.509877] ---[ end trace 0000000000000000 ]---
[ 730.435341] ACPI: PM: Preparing to enter system sleep state S5
The workqueue WARN right after the KASAN splat names the escapee:
"cannot queue ipoib_reap_ah on wq ipoib_wq" - the re-armed reaper
trying to queue onto the destroyed workqueue.
Chengyu
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2026-11-06 4:42 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-05 4:29 [PATCH v3] RDMA/ipoib: cancel the AH reaper synchronously on teardown Chengyu Liu
2026-11-05 4:34 ` sashiko-bot
2026-11-05 19:38 ` Leon Romanovsky
2026-11-06 4:42 ` Chengyu Liu
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.