* [REGRESSION] ublk: dispatched IO intermittently never posts its fetch completion (v7.0)
@ 2026-11-20 22:55 Luciano Lo Giudice
2026-11-20 23:27 ` Bernd Schubert
0 siblings, 1 reply; 7+ messages in thread
From: Luciano Lo Giudice @ 2026-11-20 22:55 UTC (permalink / raw)
To: linux-block, io-uring; +Cc: regressions, axboe, stable
[-- Attachment #1: Type: text/plain, Size: 7314 bytes --]
From: Luciano Lo Giudice <lmlogiudice@gmail.com>
To: linux-block@vger.kernel.org, io-uring@vger.kernel.org
Cc: Ming Lei <ming.lei@redhat.com>, Jens Axboe <axboe@kernel.dk>,
stable@vger.kernel.org, regressions@lists.linux.dev
Subject: [REGRESSION] ublk: dispatched IO intermittently never posts its
fetch completion (v7.0)
Hello everyone,
I'm writing a library that implements various block devices in userspace by
utilizing the ublk framework. Every so often, I'm hitting a completion loss on
Linux 7.0, in which an inflight request is lost forever, which leads to an
unkillable process and a ublk block device that cannot be torn down, and so the
machine needs to be rebooted to remedy this. I've tested the attached reproducer
on 6.8, but found no hangs after many runs. Details of the bug are below.
1. Environment
==============
- Breaks on 7.0.0-31-generic and 7.0.0-43-generic using the kernels that ship
with Ubuntu. Tested on amd64 and the ublk_drv module.
- An identical binary was run on 6.8.0-117-generic but no hangs were reproduced.
- This was run on virtual machines provisioned with LXD and with 4 CPUs.
- After reading the code, I suspect the issue arises due to the 2025 rework (the
list batching first appeared in 6.15 and the per-IO task tracking in 6.16).
I have not run any kernels between 6.8 and 7.0
2. About the attached reproducer
================================
- It's a single C file that requires only liburing.
- 3 devices are loaded (2 are file-backed, the other memory-backed).
- A fourth device is added, started, loaded (briefly), stopped and
deleted in a loop while the others are under loaded. This is the
fastest way I found to reproduce the issue.
- On 7.0, this hangs within the first 2 runs. On 6.8, no hangs were
observed after more than 10 runs. When the issue is detected, the
program prints what evidence it can gather (per-device inflight
operation numbers, the running threads' waiting channels and the
io_uring's pending CQE counts).
- Bonus: 2 additional bugs where found when shrinking the workload:
A queue was leaking its io_uring (keeping the chardev pinned), and
START_DEV issued for a queued that had died before arming. Both stalled
the program identically on 6.8 and both were fixed in the attached version.
3. What indicates a failure (from userspace's POV)
==================================================
- The issue is always connected to a READ request:
/sys/block/ublkb$N/inflight yields "1 0" and is never modified again.
- The thread that handles the queue is blocked in io_cqring_wait (as seen by
inspecting /proc/$TID/wchan), byt the ring has no entries to consume.
- The ublksrv_io_desc for the corresponding tag was written by the kernel
at dispatch. The server reads descriptors after a CQE is posted, but
this never happens.
- There is no way (AFAIK) to recover the tag. The lost completion is the
one that would set UBLK_IO_FLAG_OWNED_BY_SRV so a COMMIT_AND_FETCH for
the tag is rejected (returns -EINVAL).
- Something very similar also shows on the control plane, which is why I
suspect shared implementation details rather than something queue-specific.
- START_DEV, STOP_DEV and DEL_DEV will occassionnally never post their CQE
after the kernel provably completes. In one capture, the deleted block
device was already gone from /sys/block when the caller's 30 seconds
time bound fired. Repeated re-enters never produced the completion.
- DEL_DEV was seen stalling 30-90 seconds on the kernel for operations that
should complete instantly. The io-wq worker thread was blocked in:
ublk_ctrl_del_dev
del_gendisk
blk_mq_freeze_queue_wait
the device was stuck half-deleted, and it reacted to ADD_DEV by returning
-EEXIST for minutes. In the worst instance it survivied its own server
exiting and stayed un-deletable (needed a reboot).
- A START_DEV completed with -EINTR. The process was detached and arms no
signal sources. I didn't capture the pending signal set in that instance,
so I cannot rule out a spurious signal, but the same op's immediate retry
stalled for a full 30 seconds.
- Not needed to repro the bug: COOP_TASKRUN, readahead bursts, eventfd wakeups.
4. Aftermath
============
- AFter the request is lost, teardown hangs and takes the control plane with
it. Looking at /proc/*/wchan and dmesg of a hanged instance shows that the
teardown uring command (running in an io-wq worker) is stucj in
ublk_stop_dev/ublk_ctrl_del_dev
del_gendisk
blk_unregister_queue
elevator_set_none
blk_mq_freeze_queue_wait
And the message printed by the kernel is:
INFO: task iou-wrk-2115:2118 blocked for more than 122 seconds.
[...]
ublk_stop_dev_unlocked.part.0+0x38/0x160 [ublk_drv]
ublk_ctrl_uring_cmd+0x629/0x830 [ublk_drv]
- Subsequent control command hangs on the control mutex:
task iou-wrk-12720:12721 is blocked on a mutex likely owned by task
iou-wrk-12626:12629.
[...]
ublk_ctrl_get_params+0x8b/0x190 [ublk_drv]
- Killing the process doesn't help. The worker thread keeps waiting on the
queue and the frozen queue waits for a request whose owner (the server) is
gone. On 7.0, ublk_timeout(...) only takes action for unprivileged devices by
SIGKILL'ing the server. On privileged processes it retursn BLK_EH_RESET_TIMER
so a lost request is re-armed forever and never aborted, even when the
server is dead.
- When the process exits, the io_uring io-wq teardown warns while a worker is
still stuck and the process remains a zombie. The following message is
printed by the kernel:
WARNING: io_uring/io-wq.c:1370 at io_wq_put_and_exit+0x10d/0x2e0,
CPU#3: live_server_ram/2115
5. Where I suspect is the problem
=================================
- Evidence points at the completion side of the 2025 rework.
- per-IO task tracking ( ("ublk: require unique task per io instead of
unique task per queue", Pure Storage, 04/2025,
https://patchew.org/linux/20250416-ublk._5Ftask._5Fper._5Fio-v5-0-9261ad7bff20@purestorage.com/):
our fetches are all submitted by the queue's own thread, so io->task
should always match at dispatch time;
- rq_list batching: ublk_queue_rqs() coalesces same-<ctx_handle, task>
requests into one list and queues a single
io_uring_cmd_complete_in_task(io->cmd, ublk_cmd_list_tw_cb) for the
whole batch. If that single task work is lost, every request in the
list strands - the observations are consistent with one lost request per
occurrence.
6. Follow-ups
=============
- Does this ring any bells in the 7.0 completion rework (lost task work
for ublk_cmd_list_tw_cb / ublk_cmd_tw_cb, or io->task mismatch taking
a path that neither completes nor aborts)? Is a fix already in
flight?
- Independently of the completion loss: should ublk_timeout() for
privileged daemons fail a request whose ublk server is gone? Today
BLK_EH_RESET_TIMER keeps a lost request - and with it the del_gendisk
freeze and the control mutex - stuck forever after server's death,
which turns a single lost completion into a machine-level hang.
I can test candidate patches on 7.0.0-43 or bisect across 6.15-7.0 on a
VM, or capture the trace of a missing completion.
#regzbot introduced: v6.8..v7.0
Best regards,
Luciano
[-- Attachment #2: ublk-bug.c --]
[-- Type: text/x-csrc, Size: 38634 bytes --]
/*
* Reproducer for the Linux 7.0 ublk lost fetch-completion bug.
*
* What we're after: A READ request that is dispatched by the kernel (the
* descriptor is written to, sysfs inflight shows "1 0") but the io_uring
* never gets the corresponding completion posted. Thus, the queue thread
* keeps going back to io_uring_enter and repots "wait timeout: pending CQES 0"
* every 5 seconds. Userspace cannot recover the tag and the lost request
* hangs STOP_DEV/DEL_DEV commands. The machine needs to be rebooted.
*
* The watchdog thread aims to detect the hang. So when a device makes no
* IO progress for STALL_SECS, it will print out the evidence (including
* the wait channels for the threads) and exits.
*
* To build, simply run: 'cc -O2 ublk-bug.c -luring'
* To run: sudo ./a.out [runs] (0 means loop forever)
*
* The program was tested on 6.8 and 7.0. In the former, it has not shown
* any hangs. On the latter, it is consitently triggered after few rounds.
*
* The following parameteres may be tuned by the user at build time:
*
* -DNR_DEVS=3 devices served concurrently
* -DDEPTH=32 tags per device (single queue per device)
* -DVOL_MB=64 per-device volume size
* -DMAX_IO_BYTES=131072 per-tag buffer / max request size
* -DCAMPAIGN_SECS=60 seconds of load per run
* -DSTALL_SECS=15 no-progress interval that counts as a hang
* -DIDLE_SECS=5 idle nonzero-inflight interval = a strand
* -DLOAD_THREADS=4 parallel load threads per device
* -DRAMDEV_IDX=1 which device gets the in-memory backend
* -DNO_CHURN drop the interleaved lifecycle device
* -DNO_COOP_TASKRUN build the ring without IORING_SETUP_COOP_TASKRUN
*/
#define _GNU_SOURCE
#include <dirent.h>
#include <errno.h>
#include <fcntl.h>
#include <liburing.h>
#include <pthread.h>
#include <signal.h>
#include <stdatomic.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/prctl.h>
#include <sys/mman.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <time.h>
#include <unistd.h>
// Define tunables.
#ifndef NR_DEVS
#define NR_DEVS 3
#endif
#ifndef DEPTH
#define DEPTH 32
#endif
#ifndef VOL_MB
#define VOL_MB 64
#endif
#ifndef MAX_IO_BYTES
#define MAX_IO_BYTES 131072
#endif
#ifndef CAMPAIGN_SECS
#define CAMPAIGN_SECS 60
#endif
#ifndef STALL_SECS
#define STALL_SECS 15
#endif
#ifndef IDLE_SECS
#define IDLE_SECS 5 // idle nonzero-inflight interval that counts as a strand
#endif
#ifndef LOAD_THREADS
#define LOAD_THREADS 4 // parallel load threads per device (disjoint regions)
#endif
// build with -DNO_CHURN to drop the interleaved lifecycle device
//#define NO_CHURN
// which device gets an in-memory backend (memcpy, instant completions -
// the highest request rate, and the "most heavily loaded" victim of the
// original run); -1 for file backends everywhere
#ifndef RAMDEV_IDX
#define RAMDEV_IDX 1
#endif
#define BASE_ID 80
#define QUEUE_ID 0 // one hardware queue per device
// Taken from include/linux/ublk_cmd.h
#define UBLK_U_CMD_GET_FEATURES 0x80207513UL
#define UBLK_U_CMD_ADD_DEV 0xC0207504UL
#define UBLK_U_CMD_DEL_DEV 0xC0207505UL
#define UBLK_U_CMD_START_DEV 0xC0207506UL
#define UBLK_U_CMD_STOP_DEV 0xC0207507UL
#define UBLK_U_CMD_SET_PARAMS 0xC0207508UL
#define UBLK_U_IO_FETCH_REQ 0xC0107520UL
#define UBLK_U_IO_COMMIT_AND_FETCH_REQ 0xC0107521UL
#define UBLK_IO_OP_READ 0
#define UBLK_IO_OP_WRITE 1
#define UBLK_IO_OP_FLUSH 2
#define UBLK_ATTR_VOLATILE_CACHE (1U << 2)
#define UBLK_PARAM_TYPE_BASIC (1U << 0)
// UBLK_IO_RES_ABORT: the only negative fetch result that is not an error.
#define UBLK_IO_RES_ABORT (-ENODEV)
struct ublksrv_ctrl_cmd {
uint32_t dev_id;
uint16_t queue_id;
uint16_t len;
uint64_t addr;
uint64_t data[1];
uint16_t dev_path_len;
uint16_t pad;
uint32_t reserved;
};
struct ublksrv_ctrl_dev_info {
uint16_t nr_hw_queues;
uint16_t queue_depth;
uint16_t state;
uint16_t pad0;
uint32_t max_io_buf_bytes;
uint32_t dev_id;
int32_t ublksrv_pid;
uint32_t pad1;
uint64_t flags;
uint64_t ublksrv_flags;
uint32_t owner_uid;
uint32_t owner_gid;
uint64_t reserved1;
uint64_t reserved2;
};
struct ublksrv_io_desc {
uint32_t op_flags; // op in bits 0-7, flags in 8-31
uint32_t nr_sectors;
uint64_t start_sector;
uint64_t addr;
};
struct ublksrv_io_cmd {
uint16_t q_id;
uint16_t tag;
int32_t result;
uint64_t addr;
};
struct ublk_params_basic {
uint32_t attrs;
uint8_t logical_bs_shift;
uint8_t physical_bs_shift;
uint8_t io_opt_shift;
uint8_t io_min_shift;
uint32_t max_sectors;
uint32_t chunk_sectors;
uint64_t dev_sectors;
uint64_t virt_boundary_mask;
};
struct ublk_params {
uint32_t len;
uint32_t types;
struct ublk_params_basic basic;
};
_Static_assert(sizeof(struct ublksrv_ctrl_cmd) == 32, "ctrl cmd layout");
_Static_assert(sizeof(struct ublksrv_ctrl_dev_info) == 64, "dev info layout");
_Static_assert(sizeof(struct ublksrv_io_desc) == 24, "io desc layout");
_Static_assert(sizeof(struct ublksrv_io_cmd) == 16, "io cmd layout");
// Global definitions.
#define CTRL_NO_QUEUE 0xFFFFu
#define VOL_BYTES ((uint64_t)(VOL_MB) << 20)
#define UD_FETCH 0x20u
#define UD_COMMIT 0x21u
#define WAIT_TIMEOUT_SEC 5
static int ctrl_fd = -1;
static struct io_uring ctrl_ring;
static pthread_mutex_t drop_lock = PTHREAD_MUTEX_INITIALIZER;
static atomic_int g_active; // load phase of a run
static atomic_int g_stop; // SIGINT / fatal
static atomic_uint g_run;
struct dev {
unsigned id;
char node[32];
char cdev_path[32];
char backend[64];
int backend_fd;
int ram_backend; // in-memory backend instead of a file
unsigned char *ram_buf;
int cdev_fd;
int depth;
unsigned char *bufs; // depth * MAX_IO_BYTES, the tag buffers
struct ublksrv_io_desc *descs; // mmap'd, kernel-written
struct io_uring ring;
atomic_uint armed;
int live; // START_DEV done, DEL_DEV done
int quiet; // churn queue: exit quietly on IO errors
atomic_ullong passes; // bumped by the load thread after every pass
};
static struct dev devs[NR_DEVS];
static void report_hang(int worst_dev);
static double now_s(void)
{
struct timespec ts;
clock_gettime(CLOCK_MONOTONIC, &ts);
return (double)ts.tv_sec + (double)ts.tv_nsec / 1e9;
}
#define die(fmt, ...) \
do { \
fprintf(stderr, "FATAL %s:%d " fmt ": %s\n", __func__, __LINE__, \
##__VA_ARGS__, strerror(errno)); \
exit(1); \
} while (0)
/*
* Hand-prepared IORING_OP_URING_CMD SQE - works on every liburing 2.x
* (the prep_uring_cmd helpers only appeared later). Layout is fixed by
* the uapi: cmd_op at offset 8, user_data at 32, cmd area at 48 (16
* bytes on a 64-byte SQE, 80 with IORING_SETUP_SQE128).
*/
#define SQE_CMD_OP_OFF 8
#define SQE_CMD_OFF 48
static void prep_uring_cmd_sqe(struct io_uring_sqe *sqe, unsigned cmd_op,
int fd, uint64_t user_data)
{
memset(sqe, 0, sizeof(*sqe));
sqe->opcode = IORING_OP_URING_CMD;
sqe->fd = fd;
*(uint32_t *)((char *)sqe + SQE_CMD_OP_OFF) = cmd_op;
sqe->user_data = user_data;
}
static uint64_t ctrl_seq;
/*
* One synchronous round trip: URING_CMD128 linked to a wall-clock
* timeout, so a hanged teardown surfaces as -ETIMEDOUT instead of
* blocking us forever. -EINTR results (a signal interrupting the
* blocking kernel-side handler - observed for DEL/STOP on this
* kernel) are retried here; sequence tags make stale completions of
* timed-out calls droppable.
*/
static int ctrl_once(unsigned cmd_op, struct ublksrv_ctrl_cmd *cc,
unsigned tmo)
{
struct io_uring_sqe *sqe = io_uring_get_sqe(&ctrl_ring);
struct io_uring_sqe *lt = io_uring_get_sqe(&ctrl_ring);
struct __kernel_timespec ts = { .tv_sec = tmo, .tv_nsec = 0 };
uint64_t cmd_tag = ctrl_seq++ << 1;
uint64_t tmo_tag = cmd_tag | 1;
int cmd_res = INT32_MIN, tmo_res = INT32_MAX;
if (!sqe || !lt)
die("ctrl ring full");
prep_uring_cmd_sqe(sqe, cmd_op, ctrl_fd, cmd_tag);
memcpy((char *)sqe + SQE_CMD_OFF, cc, sizeof(*cc));
sqe->flags = IOSQE_IO_LINK;
io_uring_prep_link_timeout(lt, &ts, 0);
lt->user_data = tmo_tag;
int ret = io_uring_submit_and_wait(&ctrl_ring, 1);
if (ret < 0) {
errno = -ret;
die("ctrl submit op %#x", cmd_op);
}
/*
* Stop once the command completed OR its timeout fired: a fired
* timeout cancels the command, and if the kernel side is stuck
* uninterruptibly the command's CQE may never surface at all.
*/
while (cmd_res == INT32_MIN && tmo_res == INT32_MAX) {
struct io_uring_cqe *cqe;
if (io_uring_peek_cqe(&ctrl_ring, &cqe) < 0 &&
io_uring_wait_cqe(&ctrl_ring, &cqe) < 0)
die("ctrl reap op %#x", cmd_op);
if (cqe->user_data == cmd_tag)
cmd_res = cqe->res;
else if (cqe->user_data == tmo_tag)
tmo_res = cqe->res;
// anything else: stale completion of an earlier call
io_uring_cqe_seen(&ctrl_ring, cqe);
}
if (cmd_res == INT32_MIN) {
fprintf(stderr, "ctrl op %#x timed out after %us "
"(control plane hanged)\n", cmd_op, tmo);
return -ETIMEDOUT;
}
if (getenv("REPRO_CTRL_TRACE"))
fprintf(stderr, "ctrl op %#x -> %d (tmo %d)\n", cmd_op,
cmd_res, tmo_res);
return cmd_res;
}
static int ctrl_cmd(unsigned cmd_op, struct ublksrv_ctrl_cmd *cc, unsigned tmo)
{
// Lock to serialize control commands during load.
static pthread_mutex_t ctrl_mutex = PTHREAD_MUTEX_INITIALIZER;
int res;
pthread_mutex_lock(&ctrl_mutex);
for (int attempt = 0; attempt < 3; attempt++) {
res = ctrl_once(cmd_op, cc, tmo);
if (res != -EINTR)
break;
/*
* No signal should reach this process; record the pending
* set so the spurious -EINTR is provably signal-less.
*/
char line[128] = "";
FILE *st = fopen("/proc/self/status", "r");
if (st) {
char l[128];
while (fgets(l, sizeof(l), st))
if ((!strncmp(l, "SigPnd", 6) ||
!strncmp(l, "ShdPnd", 6)) &&
strlen(line) + strlen(l) < sizeof(line))
strcat(line, l);
fclose(st);
}
fprintf(stderr, "ctrl op %#x interrupted, retrying; %s\n",
cmd_op, line);
}
pthread_mutex_unlock(&ctrl_mutex);
return res;
}
/*
* ctrl_cmd, reporting failures with the kernel's errno (not the clobbered
* thread-local one). ADD_DEV racing a dying previous process's auto
* teardown shows up as EEXIST here.
*/
static void ck(int res, const char *what)
{
if (res < 0) {
fprintf(stderr, "FATAL: %s: %s (-%u)\n", what, strerror(-res),
(unsigned)-res);
exit(1);
}
}
static int ctrl_get_features(uint64_t *features)
{
struct ublksrv_ctrl_cmd cc = {
.dev_id = 0,
.queue_id = CTRL_NO_QUEUE,
.len = sizeof(*features),
.addr = (uint64_t)(uintptr_t)features,
};
return ctrl_cmd(UBLK_U_CMD_GET_FEATURES, &cc, 10);
}
static int ctrl_add_dev(unsigned id, unsigned depth)
{
struct ublksrv_ctrl_dev_info info = {
.nr_hw_queues = 1,
.queue_depth = depth,
.state = 0, // UBLK_S_DEV_DEAD
.max_io_buf_bytes = MAX_IO_BYTES,
.dev_id = id,
.flags = 0, // base copy mode
};
struct ublksrv_ctrl_cmd cc = {
.dev_id = id,
.queue_id = CTRL_NO_QUEUE,
.len = sizeof(info),
.addr = (uint64_t)(uintptr_t)&info,
};
return ctrl_cmd(UBLK_U_CMD_ADD_DEV, &cc, 10);
}
static int ctrl_set_params(unsigned id, uint64_t dev_sectors)
{
struct ublk_params params = {
.len = sizeof(params),
.types = UBLK_PARAM_TYPE_BASIC,
.basic = {
.attrs = UBLK_ATTR_VOLATILE_CACHE,
.logical_bs_shift = 12,
.physical_bs_shift = 12,
.io_opt_shift = 12,
.io_min_shift = 12,
.max_sectors = MAX_IO_BYTES / 512,
.dev_sectors = dev_sectors / 512,
},
};
struct ublksrv_ctrl_cmd cc = {
.dev_id = id,
.queue_id = CTRL_NO_QUEUE,
.len = params.len,
.addr = (uint64_t)(uintptr_t)¶ms,
};
return ctrl_cmd(UBLK_U_CMD_SET_PARAMS, &cc, 10);
}
static int ctrl_start_dev(unsigned id)
{
struct ublksrv_ctrl_cmd cc = {
.dev_id = id,
.queue_id = CTRL_NO_QUEUE,
.data[0] = getpid(),
};
return ctrl_cmd(UBLK_U_CMD_START_DEV, &cc, 30);
}
static int ctrl_stop_dev(unsigned id)
{
struct ublksrv_ctrl_cmd cc = {
.dev_id = id,
.queue_id = CTRL_NO_QUEUE,
};
return ctrl_cmd(UBLK_U_CMD_STOP_DEV, &cc, 30);
}
static int ctrl_del_dev(unsigned id)
{
struct ublksrv_ctrl_cmd cc = {
.dev_id = id,
.queue_id = CTRL_NO_QUEUE,
};
return ctrl_cmd(UBLK_U_CMD_DEL_DEV, &cc, 30);
}
// Queue-related stuff.
static inline uint64_t ud_pack(unsigned tag, unsigned kind)
{
return tag | (kind << 16);
}
static void prep_fetch(struct io_uring_sqe *sqe, struct dev *d, unsigned tag,
unsigned kind, int32_t result)
{
struct ublksrv_io_cmd cmd = {
.q_id = QUEUE_ID,
.tag = tag,
.result = result,
.addr = (uint64_t)(uintptr_t)(d->bufs + (size_t)tag * MAX_IO_BYTES),
};
prep_uring_cmd_sqe(sqe, kind == UD_COMMIT ?
UBLK_U_IO_COMMIT_AND_FETCH_REQ :
UBLK_U_IO_FETCH_REQ, 0,
ud_pack(tag, kind));
sqe->flags |= IOSQE_FIXED_FILE; // fd 0 = the queue chardev
memcpy((char *)sqe + SQE_CMD_OFF, &cmd, sizeof(cmd));
}
// Fully synchronous backend implementation.
static int32_t backend_io(struct dev *d, uint32_t op,
uint64_t off, uint32_t len, unsigned char *buf)
{
ssize_t n;
if (d->ram_backend) {
unsigned char *src = d->ram_buf + off;
switch (op) {
case UBLK_IO_OP_READ:
memcpy(buf, src, len);
return (int32_t)len;
case UBLK_IO_OP_WRITE:
memcpy(src, buf, len);
return (int32_t)len;
case UBLK_IO_OP_FLUSH:
return 0;
default:
return -EIO;
}
}
switch (op) {
case UBLK_IO_OP_READ:
n = pread(d->backend_fd, buf, len, (off_t)off);
break;
case UBLK_IO_OP_WRITE:
n = pwrite(d->backend_fd, buf, len, (off_t)off);
break;
case UBLK_IO_OP_FLUSH:
n = fdatasync(d->backend_fd) == 0 ? 0 : -1;
break;
default:
fprintf(stderr, "dev%u: unexpected op %u\n", d->id, op);
return -EIO;
}
return n == (ssize_t)len ? (int32_t)len : -EIO;
}
static void *server_fn(void *arg)
{
struct dev *d = arg;
unsigned depth = d->depth;
struct io_uring_params p;
// This is what ublksrv does; we do it for symmetry as well.
prctl(PR_SET_IO_FLUSHER, 1, 0, 0, 0);
d->cdev_fd = open(d->cdev_path, O_RDWR);
if (d->cdev_fd < 0) {
if (errno == ENOENT || errno == ENODEV)
return NULL;
die("open %s", d->cdev_path);
}
memset(&p, 0, sizeof(p));
p.flags = IORING_SETUP_CQSIZE;
p.cq_entries = depth * 2 + 2;
#ifndef NO_COOP_TASKRUN
p.flags |= IORING_SETUP_COOP_TASKRUN;
#endif
if (io_uring_queue_init_params(depth * 2 + 2, &d->ring, &p))
die("dev%u: queue ring", d->id);
int fds[1] = { d->cdev_fd };
if (io_uring_register_files(&d->ring, fds, 1))
die("dev%u: register files", d->id);
// The descriptor array: one page-rounded region per queue, the
// queue selected by a max-stride (4096 tags * 24 bytes) offset.
size_t stride = (4096ul * sizeof(struct ublksrv_io_desc) + 4095) & ~4095ul;
size_t map_len = ((size_t)depth * sizeof(struct ublksrv_io_desc) + 4095) & ~4095ul;
d->descs = mmap(NULL, map_len, PROT_READ, MAP_SHARED | MAP_POPULATE,
d->cdev_fd, (off_t)(QUEUE_ID * stride));
if (d->descs == MAP_FAILED)
die("dev%u: desc mmap", d->id);
// Arm the queue: one fetch per tag.
for (unsigned tag = 0; tag < depth; tag++) {
struct io_uring_sqe *sqe = io_uring_get_sqe(&d->ring);
if (!sqe)
die("dev%u: arm ring full", d->id);
prep_fetch(sqe, d, tag, UD_FETCH, 0);
}
if (io_uring_submit(&d->ring) < 0)
die("dev%u: arm submit", d->id);
atomic_store(&d->armed, 1);
unsigned long long completions = 0;
// Serve until the device goes away (fetch aborts with -ENODEV).
while (!atomic_load(&g_stop)) {
unsigned count = 0;
struct io_uring_cqe *cqe;
unsigned head;
io_uring_for_each_cqe(&d->ring, head, cqe) {
unsigned tag = cqe->user_data & 0xffff;
unsigned kind = (cqe->user_data >> 16) & 0xff;
if (tag >= depth || (kind != UD_FETCH && kind != UD_COMMIT))
die("dev%u: bad cqe ud=%#llx", d->id,
(unsigned long long)cqe->user_data);
completions++;
if (cqe->res < 0) {
if (cqe->res == UBLK_IO_RES_ABORT)
goto stopped;
else if (d->quiet) {
fprintf(stderr,
"dev%u: cqe kind %u res %d (ud %#llx), queue exiting\n",
d->id, kind, cqe->res,
(unsigned long long)cqe->user_data);
goto stopped;
}
errno = -cqe->res;
die("dev%u: fetch cqe tag %u", d->id, tag);
}
/*
* New request for this tag: read the descriptor
* (kernel wrote it before completing the fetch),
* run the backend IO, commit and re-fetch.
*/
struct ublksrv_io_desc volatile *vd = &d->descs[tag];
uint32_t op = vd->op_flags & 0xff;
uint32_t len = vd->nr_sectors * 512;
uint64_t off = vd->start_sector * 512;
int res = backend_io(d, op, off, len,
d->bufs + (size_t)tag * MAX_IO_BYTES);
struct io_uring_sqe *sqe = io_uring_get_sqe(&d->ring);
if (!sqe)
die("dev%u: serve ring full", d->id);
prep_fetch(sqe, d, tag, UD_COMMIT, res);
if (++count == depth)
break;
}
io_uring_cq_advance(&d->ring, count);
if (count) {
if (io_uring_submit(&d->ring) < 0)
die("dev%u: commit submit", d->id);
continue;
}
/*
* Bounded blocking wait. If we timed out, report how many CQEs
* the kernel owes us. The bug shows up as an armed fetch whose
* completion never appears and leaves it at 0 forever.
*/
struct __kernel_timespec ts = { .tv_sec = WAIT_TIMEOUT_SEC, .tv_nsec = 0 };
int ret = io_uring_submit_and_wait_timeout(&d->ring, &cqe, 1, &ts, 0);
if (ret == -ETIME)
fprintf(stderr,
"dev%u wait timeout: pending CQEs %u, completions so far %llu\n",
d->id, io_uring_cq_ready(&d->ring), completions);
else if (ret < 0 && ret != -EINTR) {
errno = -ret;
die("dev%u: wait", d->id);
}
}
stopped:
/*
* Release everything the queue owns; the ring pins the single-open
* chardev and the descriptor mapping, and a leaked one hangs the
* next lifecycle cycle on the same id.
*/
io_uring_queue_exit(&d->ring);
close(d->cdev_fd);
munmap(d->descs, map_len);
return NULL;
}
static void drop_caches(void)
{
int fd = open("/proc/sys/vm/drop_caches", O_WRONLY);
if (fd < 0) {
fputs("warning: could not open file to drop caches", stderr);
return;
} else if (write(fd, "3\n", 2) != 2) {
fputs("warning: failed to instruct the kernel to drop caches", stderr);
}
close(fd);
}
/*
* ADD_DEV can race a lingering device from a previous run or a
* killed run. As such, we delete and poll time bound.
*/
static void add_dev_retry(unsigned id, unsigned depth)
{
int r = ctrl_add_dev(id, depth);
for (int attempt = 0; r == -EEXIST && attempt < 10; attempt++) {
fprintf(stderr, "id %u still exists (EEXIST), deleting\n", id);
ctrl_del_dev(id);
sleep(1);
r = ctrl_add_dev(id, depth);
}
if (r == -EEXIST) {
printf("ADD_DEV %u still EEXIST after retries - device stuck "
"in half-deleted limbo\n", id);
report_hang(id);
}
ck(r, "ADD_DEV");
}
static void *load_fn(void *arg)
{
struct { struct dev *d; unsigned shard; } *w = arg;
struct dev *d = w->d;
const size_t chunk = 128 * 1024;
unsigned char *io = aligned_alloc(4096, chunk);
unsigned long long salt = w->shard;
int ran = 0;
// this shard's slice of the volume: concurrent in-flight IO per device
const uint64_t lo = VOL_BYTES / LOAD_THREADS * w->shard;
const uint64_t hi = VOL_BYTES / LOAD_THREADS * (w->shard + 1);
if (!io)
die("load buf");
int fd = open(d->node, O_RDWR);
if (fd < 0)
die("open %s", d->node);
fprintf(stderr, "dev%u load[%u]: opened\n", d->id, w->shard);
while (!atomic_load(&g_stop)) {
if (!atomic_load(&g_active)) {
if (ran)
break;
struct timespec ms = { 0, 50 * 1000 * 1000 };
nanosleep(&ms, NULL);
continue;
}
// Try to make sure to read from the device without the page cache.
pthread_mutex_lock(&drop_lock);
drop_caches();
pthread_mutex_unlock(&drop_lock);
for (uint64_t off = lo; off < hi; off += chunk) {
ssize_t n;
do {
n = pread(fd, io, chunk, (off_t)off);
} while (n < 0 && errno == EINTR);
if (n != (ssize_t)chunk) {
fprintf(stderr, "dev%u: read at %#llx: %zd (%s)\n",
d->id, (unsigned long long)off, n,
strerror(errno));
goto out;
}
}
atomic_fetch_add(&d->passes, 1);
ran = 1;
if (atomic_load(&g_stop) || !atomic_load(&g_active))
continue;
// Same as above for writes.
for (uint64_t off = lo; off < hi; off += chunk) {
memset(io, (int)(++salt + d->id), chunk);
ssize_t n;
do {
n = pwrite(fd, io, chunk, (off_t)off);
} while (n < 0 && errno == EINTR);
if (n != (ssize_t)chunk) {
fprintf(stderr, "dev%u: write at %#llx: %zd\n",
d->id, (unsigned long long)off, n);
goto out;
}
}
if (fsync(fd))
die("dev%u: fsync", d->id);
atomic_fetch_add(&d->passes, 1);
ran = 1;
}
out:
close(fd);
free(io);
return NULL;
}
/*
* The basis of a failing workload: A new device is being added, started,
* loaded (briefly), stopped and deleted over and over while the main
* devices are under load.
*/
#define CHURN_ID (BASE_ID + NR_DEVS)
static void *churn_fn(void *_arg)
{
(void)_arg;
struct dev d;
memset(&d, 0, sizeof(d));
d.id = CHURN_ID;
d.depth = DEPTH;
d.ram_backend = 1;
d.quiet = 1;
d.ram_buf = aligned_alloc(4096, VOL_BYTES);
if (!d.ram_buf)
die("churn ram buf");
memset(d.ram_buf, 0xa5, VOL_BYTES);
snprintf(d.node, sizeof(d.node), "/dev/ublkb%u", d.id);
snprintf(d.cdev_path, sizeof(d.cdev_path), "/dev/ublkc%u", d.id);
atomic_init(&d.armed, 0);
atomic_init(&d.passes, 0);
unsigned long long cycles = 0;
while (!atomic_load(&g_stop)) {
if (!atomic_load(&g_active)) {
struct timespec ms = { 0, 100 * 1000 * 1000 };
nanosleep(&ms, NULL);
continue;
}
int have_server = 0;
pthread_t server;
add_dev_retry(d.id, (unsigned)d.depth);
if (ctrl_set_params(d.id, VOL_BYTES) < 0)
goto teardown;
atomic_store(&d.armed, 0);
if (pthread_create(&server, NULL, server_fn, &d))
die("churn server thread");
have_server = 1;
for (int t = 0; t < 100 && !atomic_load(&d.armed); t++) {
struct timespec ms = { 0, 20 * 1000 * 1000 };
nanosleep(&ms, NULL);
}
/*
* The queue may have died with the device (delete race);
* without arming there is nothing to start - and START_DEV
* would wait forever for a queue that will never arm.
*/
if (!atomic_load(&d.armed))
goto teardown;
int r = ctrl_start_dev(d.id);
if (r != 0)
goto teardown;
{
d.live = 1;
int fd = open(d.node, O_RDWR);
if (fd < 0) {
// device raced away; stop and retry the cycle
d.live = 0;
goto teardown;
}
// a burst of buffered reads and writes
unsigned char *io = aligned_alloc(4096, 256 * 1024);
for (int pass = 0; pass < 3; pass++) {
for (uint64_t off = 0; off < VOL_BYTES;
off += 256 * 1024) {
ssize_t n = pread(fd, io, 256 * 1024,
(off_t)off);
if (n < 0 && errno != EINTR)
break;
}
for (uint64_t off = 0; off < VOL_BYTES;
off += 256 * 1024) {
ssize_t n = pwrite(fd, io, 256 * 1024,
(off_t)off);
if (n < 0 && errno != EINTR)
break;
}
fsync(fd);
}
free(io);
close(fd);
}
teardown:
/*
* STOP_DEV aborts the queue's fetches; the queue thread
* sees -ENODEV and exits.
*/
int sr = ctrl_stop_dev(d.id);
if (sr == -ENOENT)
sr = 0;
if (have_server)
pthread_join(server, NULL);
d.live = 0;
if (sr == -ETIMEDOUT) {
printf("churn STOP_DEV %u timed out - kernel-side "
"stop hanged\n", d.id);
report_hang(d.id);
}
int dr = ctrl_del_dev(d.id);
if (dr == -ETIMEDOUT) {
printf("churn DEL_DEV %u timed out - kernel-side "
"delete hanged\n", d.id);
report_hang(d.id);
}
cycles++;
if (cycles % 5 == 0) {
printf("churn: %llu cycles\n", cycles);
fflush(stdout);
}
}
free(d.ram_buf);
return NULL;
}
static void print_thread_states(void)
{
char path[128], wchan[64], comm[64];
DIR *dir = opendir("/proc/self/task");
struct dirent *de;
if (!dir)
return;
printf(" threads:\n");
int all_threads = getenv("REPRO_ALL_THREADS") != 0;
while ((de = readdir(dir)) != NULL) {
long tid = atol(de->d_name);
int fd;
ssize_t n;
if (tid <= 0)
continue;
snprintf(path, sizeof(path), "/proc/self/task/%ld/wchan", tid);
fd = open(path, O_RDONLY);
if (fd < 0)
continue;
n = read(fd, wchan, sizeof(wchan) - 1);
close(fd);
if ((size_t)n >= 64)
continue;
wchan[n] = 0;
if (all_threads ||
strstr(wchan, "cqring") ||
strstr(wchan, "ublk") ||
strstr(wchan, "freeze") ||
strstr(wchan, "mutex")) {
strcpy(comm, "?");
snprintf(path, sizeof(path), "/proc/self/task/%ld/comm", tid);
fd = open(path, O_RDONLY);
if (fd >= 0) {
n = read(fd, comm, sizeof(comm) - 1);
close(fd);
if (n > 0)
comm[n] = 0;
}
printf(" tid %ld comm %.20s wchan %.40s\n", tid, comm, wchan);
}
}
closedir(dir);
}
static void report_hang(int worst_dev)
{
printf("\n=== HUNG: dev%u made no progress for %d s ===\n",
worst_dev, STALL_SECS);
for (int i = 0; i < NR_DEVS; i++) {
char path[96], buf[64];
snprintf(path, sizeof(path), "/sys/block/ublkb%u/inflight",
devs[i].id);
int fd = open(path, O_RDONLY);
ssize_t n = fd >= 0 ? read(fd, buf, sizeof(buf) - 1) : -1;
if (fd >= 0)
close(fd);
if (n > 0) {
buf[n] = 0;
printf(" /sys/block/ublkb%u/inflight: %s", devs[i].id, buf);
} else {
printf(" /sys/block/ublkb%u/inflight: <unreadable>\n", devs[i].id);
}
}
print_thread_states();
printf(" fingerprint expected: inflight \"1 0\" frozen, its\n"
" queue thread blocked in io_cqring_wait having reported\n"
" \"wait timeout: pending CQEs 0\" above, every 5 s.\n"
" dmesg will show the del_gendisk freeze / control-mutex\n"
" cascade if teardown is attempted; reboot to clear.\n");
fflush(stdout);
struct timespec s = { 5, 0 };
nanosleep(&s, NULL);
exit(2);
}
static int dev_inflight(unsigned id)
{
char path[64], buf[64];
snprintf(path, sizeof(path), "/sys/block/ublkb%u/inflight", id);
int fd = open(path, O_RDONLY);
if (fd < 0)
return -1; // no device (before add / after del)
ssize_t n = read(fd, buf, sizeof(buf) - 1);
close(fd);
if (n <= 0)
return -1;
buf[n] = 0;
int r = 0, w = 0;
if (sscanf(buf, "%d %d", &r, &w) != 2)
return -1;
return r + w;
}
static void *watchdog_fn(void *_arg)
{
(void)_arg;
unsigned long long last[NR_DEVS] = { 0 };
int stall[NR_DEVS] = { 0 };
int idle_stall[NR_DEVS] = { 0 };
while (!atomic_load(&g_stop)) {
sleep(1);
int active = atomic_load(&g_active);
int worst = -1, worst_s = 0;
for (int i = 0; i < NR_DEVS; i++) {
if (active) {
// load running: passes must keep advancing
unsigned long long cur =
atomic_load(&devs[i].passes);
if (cur != last[i]) {
last[i] = cur;
stall[i] = 0;
} else if (++stall[i] > worst_s) {
worst_s = stall[i];
worst = i;
}
continue;
}
if (!devs[i].live)
continue;
/*
* If we're idle, then there must be zero in flight requests.
* Otherwise, it's likely that the kernel dispatched a request
* that never made it to us.
*/
if (dev_inflight(devs[i].id) > 0) {
if (++idle_stall[i] >= IDLE_SECS) {
printf("idle dev%u: inflight nonzero "
"for %ds with no load\n",
devs[i].id, idle_stall[i]);
report_hang(devs[i].id);
}
} else {
idle_stall[i] = 0;
}
}
if (worst >= 0 && worst_s >= STALL_SECS)
report_hang(devs[worst].id);
}
return NULL;
}
// ----------------------------------------------------------------- setup
static void on_sigint(int _sig)
{
(void)_sig;
atomic_store(&g_stop, 1);
}
static void on_sigalrm(int _sig)
{
(void)_sig;
fprintf(stderr, "FATAL: teardown did not finish in 60 s - "
"control plane hanged (expected after a strand)\n");
_exit(3);
}
int main(int argc, char **argv)
{
unsigned runs = argc > 1 ? (unsigned)atoi(argv[1]) : 0;
if (geteuid() != 0) {
fprintf(stderr, "must run as root\n");
return 1;
}
signal(SIGINT, on_sigint);
signal(SIGALRM, on_sigalrm);
ctrl_fd = open("/dev/ublk-control", O_RDWR);
if (ctrl_fd < 0)
die("open /dev/ublk-control");
struct io_uring_params p = { .flags = IORING_SETUP_SQE128 };
if (io_uring_queue_init_params(8, &ctrl_ring, &p))
die("ctrl ring");
uint64_t features = 0;
ck(ctrl_get_features(&features), "GET_FEATURES");
printf("kernel ublk features: %#lx\n", (unsigned long)features);
for (int i = 0; i < NR_DEVS; i++) {
struct dev *d = &devs[i];
d->id = BASE_ID + (unsigned)i;
snprintf(d->node, sizeof(d->node), "/dev/ublkb%u", d->id);
snprintf(d->cdev_path, sizeof(d->cdev_path), "/dev/ublkc%u", d->id);
snprintf(d->backend, sizeof(d->backend), "/tmp/ublk-repro-b%d.img", d->id);
d->depth = DEPTH;
d->ram_backend = (i == RAMDEV_IDX);
if (d->ram_backend) {
d->ram_buf = aligned_alloc(4096, VOL_BYTES);
if (!d->ram_buf)
die("ram backend");
memset(d->ram_buf, 0x5a, VOL_BYTES);
} else {
int fd = open(d->backend, O_RDWR | O_CREAT | O_TRUNC, 0600);
if (fd < 0)
die("create %s", d->backend);
if (ftruncate(fd, (off_t)VOL_BYTES))
die("truncate %s", d->backend);
close(fd);
d->backend_fd = open(d->backend, O_RDWR);
if (d->backend_fd < 0)
die("open %s", d->backend);
}
d->bufs = aligned_alloc(4096, (size_t)d->depth * MAX_IO_BYTES);
if (!d->bufs)
die("tag buffers");
}
pthread_t wd;
if (pthread_create(&wd, NULL, watchdog_fn, NULL))
die("watchdog");
#ifndef NO_CHURN
pthread_t churn;
if (pthread_create(&churn, NULL, churn_fn, NULL))
die("churn");
#endif
for (unsigned run = 1; !atomic_load(&g_stop); run++) {
if (runs && run > runs)
break;
atomic_store(&g_run, run);
printf("=== run %u ===\n", run);
fflush(stdout);
pthread_t servers[NR_DEVS];
pthread_t loads[NR_DEVS][LOAD_THREADS];
struct { struct dev *d; unsigned shard; } work[NR_DEVS][LOAD_THREADS];
for (int i = 0; i < NR_DEVS; i++) {
struct dev *d = &devs[i];
atomic_store(&d->armed, 0);
add_dev_retry(d->id, (unsigned)d->depth);
ck(ctrl_set_params(d->id, VOL_BYTES), "SET_PARAMS");
if (pthread_create(&servers[i], NULL, server_fn, d))
die("server thread");
}
for (int i = 0; i < NR_DEVS; i++) {
struct dev *d = &devs[i];
for (int t = 0; t < 100 && !atomic_load(&d->armed); t++) {
struct timespec ms = { 0, 100 * 1000 * 1000 };
nanosleep(&ms, NULL);
}
if (!atomic_load(&d->armed))
die("dev%u never armed", d->id);
ck(ctrl_start_dev(d->id), "START_DEV");
}
for (int i = 0; i < NR_DEVS; i++)
for (unsigned k = 0; k < LOAD_THREADS; k++) {
work[i][k].d = &devs[i];
work[i][k].shard = k;
if (pthread_create(&loads[i][k], NULL, load_fn,
&work[i][k]))
die("load thread");
}
for (int i = 0; i < NR_DEVS; i++)
devs[i].live = 1;
printf("device(s) live, running load for %d s\n", CAMPAIGN_SECS);
fflush(stdout);
atomic_store(&g_active, 1);
double t0 = now_s();
while (!atomic_load(&g_stop) && now_s() - t0 < CAMPAIGN_SECS) {
struct timespec s = { 1, 0 };
nanosleep(&s, NULL);
printf(" t=%3.0fs passes:", now_s() - t0);
for (int i = 0; i < NR_DEVS; i++)
printf(" %llu", (unsigned long long)
atomic_load(&devs[i].passes));
printf("\n");
fflush(stdout);
}
atomic_store(&g_active, 0);
if (atomic_load(&g_stop))
printf("interrupted, tearing down\n");
for (int i = 0; i < NR_DEVS; i++)
for (unsigned k = 0; k < LOAD_THREADS; k++)
pthread_join(loads[i][k], NULL);
for (int i = 0; i < NR_DEVS; i++) {
struct dev *d = &devs[i];
int r = ctrl_stop_dev(d->id);
if (r == -ETIMEDOUT) {
printf("STOP_DEV %u timed out - stranded "
"request hanged teardown\n", d->id);
report_hang(d->id);
}
if (r == -ENOENT)
r = 0; // already gone
if (r)
fprintf(stderr, "STOP_DEV %u: %s (-%u)\n",
d->id, strerror(-r), (unsigned)-r);
pthread_join(servers[i], NULL);
int dr = ctrl_del_dev(d->id);
if (dr == -EINTR)
dr = ctrl_del_dev(d->id);
if (dr == -ETIMEDOUT) {
printf("DEL_DEV %u timed out - kernel-side "
"delete hangedd\n", d->id);
report_hang(d->id);
}
if (dr && dr != -ENOENT)
fprintf(stderr, "DEL_DEV %u: %s (-%u)\n",
d->id, strerror(-dr), (unsigned)-dr);
d->live = 0;
}
for (int i = 0; i < NR_DEVS; i++) {
char path[64];
struct stat st;
snprintf(path, sizeof(path), "/sys/block/ublkb%u",
devs[i].id);
if (!stat(path, &st)) {
printf("LEAK: ublkb%u still present after "
"teardown\n", devs[i].id);
fflush(stdout);
}
}
printf("runn %u clean (%llu/%llu/%llu passes)\n", run,
(unsigned long long)atomic_load(&devs[0].passes),
(unsigned long long)atomic_load(&devs[1].passes),
(unsigned long long)atomic_load(&devs[2].passes));
fflush(stdout);
}
printf("done after %u runs, no hangs observed\n",
runs ? runs : atomic_load(&g_run));
atomic_store(&g_stop, 1);
#ifndef NO_CHURN
pthread_join(churn, NULL);
#endif
return 0;
}
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: [REGRESSION] ublk: dispatched IO intermittently never posts its fetch completion (v7.0)
2026-11-20 22:55 [REGRESSION] ublk: dispatched IO intermittently never posts its fetch completion (v7.0) Luciano Lo Giudice
@ 2026-11-20 23:27 ` Bernd Schubert
2026-11-20 23:40 ` Luciano Lo Giudice
0 siblings, 1 reply; 7+ messages in thread
From: Bernd Schubert @ 2026-11-20 23:27 UTC (permalink / raw)
To: Luciano Lo Giudice, linux-block, io-uring
Cc: regressions, axboe, stable, Ming Lei
On 11/20/26 23:55, Luciano Lo Giudice wrote:
> From: Luciano Lo Giudice <lmlogiudice@gmail.com>
> To: linux-block@vger.kernel.org, io-uring@vger.kernel.org
> Cc: Ming Lei <ming.lei@redhat.com>, Jens Axboe <axboe@kernel.dk>,
> stable@vger.kernel.org, regressions@lists.linux.dev
> Subject: [REGRESSION] ublk: dispatched IO intermittently never posts its
> fetch completion (v7.0)
>
> Hello everyone,
>
> I'm writing a library that implements various block devices in userspace by
> utilizing the ublk framework. Every so often, I'm hitting a completion loss on
> Linux 7.0, in which an inflight request is lost forever, which leads to an
> unkillable process and a ublk block device that cannot be torn down, and so the
> machine needs to be rebooted to remedy this. I've tested the attached reproducer
> on 6.8, but found no hangs after many runs. Details of the bug are below.
Hi Luciano,
I just scanned the io-uring list and found this by accident. I have been
working on this for a couple of month.
Would you be open to test a specific branch or dkms module? I actually
wanted to send this to upstream last months already, but then found and
fixed (with a lot of AI help) more issues and also got busy with other
tasks.
https://github.com/bsbernd/linux/tree/ublk-stop-races-with-tests-for-7.3
and here the backports, that also work on 7.0 (with a compat patch) and
that is used for our dkms module
https://github.com/bsbernd/linux/tree/niova-ublk-7.1-fix-backports
The 7.3 patches are actually verified through tla+ and an ublk branch
for https://github.com/brauner/linux-formal
@Ming the patches for 7.3 include heavy ublk state changes (it is now a
bit closer to what fuse-io-uring does) and also adds debugfs that can be
used understand state issues.
Thanks,
Bernd
>
> 1. Environment
> ==============
>
> - Breaks on 7.0.0-31-generic and 7.0.0-43-generic using the kernels that ship
> with Ubuntu. Tested on amd64 and the ublk_drv module.
> - An identical binary was run on 6.8.0-117-generic but no hangs were reproduced.
> - This was run on virtual machines provisioned with LXD and with 4 CPUs.
> - After reading the code, I suspect the issue arises due to the 2025 rework (the
> list batching first appeared in 6.15 and the per-IO task tracking in 6.16).
> I have not run any kernels between 6.8 and 7.0
>
> 2. About the attached reproducer
> ================================
>
> - It's a single C file that requires only liburing.
> - 3 devices are loaded (2 are file-backed, the other memory-backed).
> - A fourth device is added, started, loaded (briefly), stopped and
> deleted in a loop while the others are under loaded. This is the
> fastest way I found to reproduce the issue.
> - On 7.0, this hangs within the first 2 runs. On 6.8, no hangs were
> observed after more than 10 runs. When the issue is detected, the
> program prints what evidence it can gather (per-device inflight
> operation numbers, the running threads' waiting channels and the
> io_uring's pending CQE counts).
> - Bonus: 2 additional bugs where found when shrinking the workload:
> A queue was leaking its io_uring (keeping the chardev pinned), and
> START_DEV issued for a queued that had died before arming. Both stalled
> the program identically on 6.8 and both were fixed in the attached version.
>
> 3. What indicates a failure (from userspace's POV)
> ==================================================
>
> - The issue is always connected to a READ request:
> /sys/block/ublkb$N/inflight yields "1 0" and is never modified again.
> - The thread that handles the queue is blocked in io_cqring_wait (as seen by
> inspecting /proc/$TID/wchan), byt the ring has no entries to consume.
> - The ublksrv_io_desc for the corresponding tag was written by the kernel
> at dispatch. The server reads descriptors after a CQE is posted, but
> this never happens.
> - There is no way (AFAIK) to recover the tag. The lost completion is the
> one that would set UBLK_IO_FLAG_OWNED_BY_SRV so a COMMIT_AND_FETCH for
> the tag is rejected (returns -EINVAL).
> - Something very similar also shows on the control plane, which is why I
> suspect shared implementation details rather than something queue-specific.
> - START_DEV, STOP_DEV and DEL_DEV will occassionnally never post their CQE
> after the kernel provably completes. In one capture, the deleted block
> device was already gone from /sys/block when the caller's 30 seconds
> time bound fired. Repeated re-enters never produced the completion.
> - DEL_DEV was seen stalling 30-90 seconds on the kernel for operations that
> should complete instantly. The io-wq worker thread was blocked in:
>
> ublk_ctrl_del_dev
> del_gendisk
> blk_mq_freeze_queue_wait
>
> the device was stuck half-deleted, and it reacted to ADD_DEV by returning
> -EEXIST for minutes. In the worst instance it survivied its own server
> exiting and stayed un-deletable (needed a reboot).
> - A START_DEV completed with -EINTR. The process was detached and arms no
> signal sources. I didn't capture the pending signal set in that instance,
> so I cannot rule out a spurious signal, but the same op's immediate retry
> stalled for a full 30 seconds.
> - Not needed to repro the bug: COOP_TASKRUN, readahead bursts, eventfd wakeups.
>
> 4. Aftermath
> ============
>
> - AFter the request is lost, teardown hangs and takes the control plane with
> it. Looking at /proc/*/wchan and dmesg of a hanged instance shows that the
> teardown uring command (running in an io-wq worker) is stucj in
>
> ublk_stop_dev/ublk_ctrl_del_dev
> del_gendisk
> blk_unregister_queue
> elevator_set_none
> blk_mq_freeze_queue_wait
>
> And the message printed by the kernel is:
>
> INFO: task iou-wrk-2115:2118 blocked for more than 122 seconds.
> [...]
> ublk_stop_dev_unlocked.part.0+0x38/0x160 [ublk_drv]
> ublk_ctrl_uring_cmd+0x629/0x830 [ublk_drv]
>
> - Subsequent control command hangs on the control mutex:
> task iou-wrk-12720:12721 is blocked on a mutex likely owned by task
> iou-wrk-12626:12629.
> [...]
> ublk_ctrl_get_params+0x8b/0x190 [ublk_drv]
>
> - Killing the process doesn't help. The worker thread keeps waiting on the
> queue and the frozen queue waits for a request whose owner (the server) is
> gone. On 7.0, ublk_timeout(...) only takes action for unprivileged devices by
> SIGKILL'ing the server. On privileged processes it retursn BLK_EH_RESET_TIMER
> so a lost request is re-armed forever and never aborted, even when the
> server is dead.
>
> - When the process exits, the io_uring io-wq teardown warns while a worker is
> still stuck and the process remains a zombie. The following message is
> printed by the kernel:
>
> WARNING: io_uring/io-wq.c:1370 at io_wq_put_and_exit+0x10d/0x2e0,
> CPU#3: live_server_ram/2115
>
> 5. Where I suspect is the problem
> =================================
>
> - Evidence points at the completion side of the 2025 rework.
> - per-IO task tracking ( ("ublk: require unique task per io instead of
> unique task per queue", Pure Storage, 04/2025,
> https://patchew.org/linux/20250416-ublk._5Ftask._5Fper._5Fio-v5-0-9261ad7bff20@purestorage.com/):
> our fetches are all submitted by the queue's own thread, so io->task
> should always match at dispatch time;
> - rq_list batching: ublk_queue_rqs() coalesces same-<ctx_handle, task>
> requests into one list and queues a single
> io_uring_cmd_complete_in_task(io->cmd, ublk_cmd_list_tw_cb) for the
> whole batch. If that single task work is lost, every request in the
> list strands - the observations are consistent with one lost request per
> occurrence.
>
> 6. Follow-ups
> =============
>
> - Does this ring any bells in the 7.0 completion rework (lost task work
> for ublk_cmd_list_tw_cb / ublk_cmd_tw_cb, or io->task mismatch taking
> a path that neither completes nor aborts)? Is a fix already in
> flight?
> - Independently of the completion loss: should ublk_timeout() for
> privileged daemons fail a request whose ublk server is gone? Today
> BLK_EH_RESET_TIMER keeps a lost request - and with it the del_gendisk
> freeze and the control mutex - stuck forever after server's death,
> which turns a single lost completion into a machine-level hang.
>
> I can test candidate patches on 7.0.0-43 or bisect across 6.15-7.0 on a
> VM, or capture the trace of a missing completion.
>
> #regzbot introduced: v6.8..v7.0
>
> Best regards,
> Luciano
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: [REGRESSION] ublk: dispatched IO intermittently never posts its fetch completion (v7.0)
2026-11-20 23:27 ` Bernd Schubert
@ 2026-11-20 23:40 ` Luciano Lo Giudice
2026-11-21 2:25 ` Luciano Lo Giudice
0 siblings, 1 reply; 7+ messages in thread
From: Luciano Lo Giudice @ 2026-11-20 23:40 UTC (permalink / raw)
To: Bernd Schubert
Cc: linux-block, io-uring, regressions, axboe, stable, Ming Lei
Hello Bernd,
Yes, I can do that. I'll go ahead and compile ublk_drv from your repo
for the 7.0 backport and see how that goes.
All the best,
Luciano.
On Fri, Nov 20, 2026 at 8:27 PM Bernd Schubert <bernd@bsbernd.com> wrote:
>
>
>
> On 11/20/26 23:55, Luciano Lo Giudice wrote:
> > From: Luciano Lo Giudice <lmlogiudice@gmail.com>
> > To: linux-block@vger.kernel.org, io-uring@vger.kernel.org
> > Cc: Ming Lei <ming.lei@redhat.com>, Jens Axboe <axboe@kernel.dk>,
> > stable@vger.kernel.org, regressions@lists.linux.dev
> > Subject: [REGRESSION] ublk: dispatched IO intermittently never posts its
> > fetch completion (v7.0)
> >
> > Hello everyone,
> >
> > I'm writing a library that implements various block devices in userspace by
> > utilizing the ublk framework. Every so often, I'm hitting a completion loss on
> > Linux 7.0, in which an inflight request is lost forever, which leads to an
> > unkillable process and a ublk block device that cannot be torn down, and so the
> > machine needs to be rebooted to remedy this. I've tested the attached reproducer
> > on 6.8, but found no hangs after many runs. Details of the bug are below.
>
> Hi Luciano,
>
> I just scanned the io-uring list and found this by accident. I have been
> working on this for a couple of month.
>
>
> Would you be open to test a specific branch or dkms module? I actually
> wanted to send this to upstream last months already, but then found and
> fixed (with a lot of AI help) more issues and also got busy with other
> tasks.
>
> https://github.com/bsbernd/linux/tree/ublk-stop-races-with-tests-for-7.3
>
> and here the backports, that also work on 7.0 (with a compat patch) and
> that is used for our dkms module
>
> https://github.com/bsbernd/linux/tree/niova-ublk-7.1-fix-backports
>
> The 7.3 patches are actually verified through tla+ and an ublk branch
> for https://github.com/brauner/linux-formal
>
>
> @Ming the patches for 7.3 include heavy ublk state changes (it is now a
> bit closer to what fuse-io-uring does) and also adds debugfs that can be
> used understand state issues.
>
>
> Thanks,
> Bernd
>
>
> >
> > 1. Environment
> > ==============
> >
> > - Breaks on 7.0.0-31-generic and 7.0.0-43-generic using the kernels that ship
> > with Ubuntu. Tested on amd64 and the ublk_drv module.
> > - An identical binary was run on 6.8.0-117-generic but no hangs were reproduced.
> > - This was run on virtual machines provisioned with LXD and with 4 CPUs.
> > - After reading the code, I suspect the issue arises due to the 2025 rework (the
> > list batching first appeared in 6.15 and the per-IO task tracking in 6.16).
> > I have not run any kernels between 6.8 and 7.0
> >
> > 2. About the attached reproducer
> > ================================
> >
> > - It's a single C file that requires only liburing.
> > - 3 devices are loaded (2 are file-backed, the other memory-backed).
> > - A fourth device is added, started, loaded (briefly), stopped and
> > deleted in a loop while the others are under loaded. This is the
> > fastest way I found to reproduce the issue.
> > - On 7.0, this hangs within the first 2 runs. On 6.8, no hangs were
> > observed after more than 10 runs. When the issue is detected, the
> > program prints what evidence it can gather (per-device inflight
> > operation numbers, the running threads' waiting channels and the
> > io_uring's pending CQE counts).
> > - Bonus: 2 additional bugs where found when shrinking the workload:
> > A queue was leaking its io_uring (keeping the chardev pinned), and
> > START_DEV issued for a queued that had died before arming. Both stalled
> > the program identically on 6.8 and both were fixed in the attached version.
> >
> > 3. What indicates a failure (from userspace's POV)
> > ==================================================
> >
> > - The issue is always connected to a READ request:
> > /sys/block/ublkb$N/inflight yields "1 0" and is never modified again.
> > - The thread that handles the queue is blocked in io_cqring_wait (as seen by
> > inspecting /proc/$TID/wchan), byt the ring has no entries to consume.
> > - The ublksrv_io_desc for the corresponding tag was written by the kernel
> > at dispatch. The server reads descriptors after a CQE is posted, but
> > this never happens.
> > - There is no way (AFAIK) to recover the tag. The lost completion is the
> > one that would set UBLK_IO_FLAG_OWNED_BY_SRV so a COMMIT_AND_FETCH for
> > the tag is rejected (returns -EINVAL).
> > - Something very similar also shows on the control plane, which is why I
> > suspect shared implementation details rather than something queue-specific.
> > - START_DEV, STOP_DEV and DEL_DEV will occassionnally never post their CQE
> > after the kernel provably completes. In one capture, the deleted block
> > device was already gone from /sys/block when the caller's 30 seconds
> > time bound fired. Repeated re-enters never produced the completion.
> > - DEL_DEV was seen stalling 30-90 seconds on the kernel for operations that
> > should complete instantly. The io-wq worker thread was blocked in:
> >
> > ublk_ctrl_del_dev
> > del_gendisk
> > blk_mq_freeze_queue_wait
> >
> > the device was stuck half-deleted, and it reacted to ADD_DEV by returning
> > -EEXIST for minutes. In the worst instance it survivied its own server
> > exiting and stayed un-deletable (needed a reboot).
> > - A START_DEV completed with -EINTR. The process was detached and arms no
> > signal sources. I didn't capture the pending signal set in that instance,
> > so I cannot rule out a spurious signal, but the same op's immediate retry
> > stalled for a full 30 seconds.
> > - Not needed to repro the bug: COOP_TASKRUN, readahead bursts, eventfd wakeups.
> >
> > 4. Aftermath
> > ============
> >
> > - AFter the request is lost, teardown hangs and takes the control plane with
> > it. Looking at /proc/*/wchan and dmesg of a hanged instance shows that the
> > teardown uring command (running in an io-wq worker) is stucj in
> >
> > ublk_stop_dev/ublk_ctrl_del_dev
> > del_gendisk
> > blk_unregister_queue
> > elevator_set_none
> > blk_mq_freeze_queue_wait
> >
> > And the message printed by the kernel is:
> >
> > INFO: task iou-wrk-2115:2118 blocked for more than 122 seconds.
> > [...]
> > ublk_stop_dev_unlocked.part.0+0x38/0x160 [ublk_drv]
> > ublk_ctrl_uring_cmd+0x629/0x830 [ublk_drv]
> >
> > - Subsequent control command hangs on the control mutex:
> > task iou-wrk-12720:12721 is blocked on a mutex likely owned by task
> > iou-wrk-12626:12629.
> > [...]
> > ublk_ctrl_get_params+0x8b/0x190 [ublk_drv]
> >
> > - Killing the process doesn't help. The worker thread keeps waiting on the
> > queue and the frozen queue waits for a request whose owner (the server) is
> > gone. On 7.0, ublk_timeout(...) only takes action for unprivileged devices by
> > SIGKILL'ing the server. On privileged processes it retursn BLK_EH_RESET_TIMER
> > so a lost request is re-armed forever and never aborted, even when the
> > server is dead.
> >
> > - When the process exits, the io_uring io-wq teardown warns while a worker is
> > still stuck and the process remains a zombie. The following message is
> > printed by the kernel:
> >
> > WARNING: io_uring/io-wq.c:1370 at io_wq_put_and_exit+0x10d/0x2e0,
> > CPU#3: live_server_ram/2115
> >
> > 5. Where I suspect is the problem
> > =================================
> >
> > - Evidence points at the completion side of the 2025 rework.
> > - per-IO task tracking ( ("ublk: require unique task per io instead of
> > unique task per queue", Pure Storage, 04/2025,
> > https://patchew.org/linux/20250416-ublk._5Ftask._5Fper._5Fio-v5-0-9261ad7bff20@purestorage.com/):
> > our fetches are all submitted by the queue's own thread, so io->task
> > should always match at dispatch time;
> > - rq_list batching: ublk_queue_rqs() coalesces same-<ctx_handle, task>
> > requests into one list and queues a single
> > io_uring_cmd_complete_in_task(io->cmd, ublk_cmd_list_tw_cb) for the
> > whole batch. If that single task work is lost, every request in the
> > list strands - the observations are consistent with one lost request per
> > occurrence.
> >
> > 6. Follow-ups
> > =============
> >
> > - Does this ring any bells in the 7.0 completion rework (lost task work
> > for ublk_cmd_list_tw_cb / ublk_cmd_tw_cb, or io->task mismatch taking
> > a path that neither completes nor aborts)? Is a fix already in
> > flight?
> > - Independently of the completion loss: should ublk_timeout() for
> > privileged daemons fail a request whose ublk server is gone? Today
> > BLK_EH_RESET_TIMER keeps a lost request - and with it the del_gendisk
> > freeze and the control mutex - stuck forever after server's death,
> > which turns a single lost completion into a machine-level hang.
> >
> > I can test candidate patches on 7.0.0-43 or bisect across 6.15-7.0 on a
> > VM, or capture the trace of a missing completion.
> >
> > #regzbot introduced: v6.8..v7.0
> >
> > Best regards,
> > Luciano
>
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: [REGRESSION] ublk: dispatched IO intermittently never posts its fetch completion (v7.0)
2026-11-20 23:40 ` Luciano Lo Giudice
@ 2026-11-21 2:25 ` Luciano Lo Giudice
2026-11-21 5:23 ` Ming Lei
0 siblings, 1 reply; 7+ messages in thread
From: Luciano Lo Giudice @ 2026-11-21 2:25 UTC (permalink / raw)
To: Bernd Schubert
Cc: linux-block, io-uring, regressions, axboe, stable, Ming Lei
Hello everyone,
Using Bernd's repo, I was _unable_ to reproduce the bug after several
runs (it used to hang in about 1 out of 3 runs). I had to add a small
compat layer to make the module build in 7.0, but it was overall
pretty easy. I will be now using this module for any development on
7.0 (thanks, Bernd!) and will report if any other issues arise.
All the best,
Luciano.
On Fri, Nov 20, 2026 at 8:40 PM Luciano Lo Giudice
<lmlogiudice@gmail.com> wrote:
>
> Hello Bernd,
>
> Yes, I can do that. I'll go ahead and compile ublk_drv from your repo
> for the 7.0 backport and see how that goes.
>
> All the best,
> Luciano.
>
> On Fri, Nov 20, 2026 at 8:27 PM Bernd Schubert <bernd@bsbernd.com> wrote:
> >
> >
> >
> > On 11/20/26 23:55, Luciano Lo Giudice wrote:
> > > From: Luciano Lo Giudice <lmlogiudice@gmail.com>
> > > To: linux-block@vger.kernel.org, io-uring@vger.kernel.org
> > > Cc: Ming Lei <ming.lei@redhat.com>, Jens Axboe <axboe@kernel.dk>,
> > > stable@vger.kernel.org, regressions@lists.linux.dev
> > > Subject: [REGRESSION] ublk: dispatched IO intermittently never posts its
> > > fetch completion (v7.0)
> > >
> > > Hello everyone,
> > >
> > > I'm writing a library that implements various block devices in userspace by
> > > utilizing the ublk framework. Every so often, I'm hitting a completion loss on
> > > Linux 7.0, in which an inflight request is lost forever, which leads to an
> > > unkillable process and a ublk block device that cannot be torn down, and so the
> > > machine needs to be rebooted to remedy this. I've tested the attached reproducer
> > > on 6.8, but found no hangs after many runs. Details of the bug are below.
> >
> > Hi Luciano,
> >
> > I just scanned the io-uring list and found this by accident. I have been
> > working on this for a couple of month.
> >
> >
> > Would you be open to test a specific branch or dkms module? I actually
> > wanted to send this to upstream last months already, but then found and
> > fixed (with a lot of AI help) more issues and also got busy with other
> > tasks.
> >
> > https://github.com/bsbernd/linux/tree/ublk-stop-races-with-tests-for-7.3
> >
> > and here the backports, that also work on 7.0 (with a compat patch) and
> > that is used for our dkms module
> >
> > https://github.com/bsbernd/linux/tree/niova-ublk-7.1-fix-backports
> >
> > The 7.3 patches are actually verified through tla+ and an ublk branch
> > for https://github.com/brauner/linux-formal
> >
> >
> > @Ming the patches for 7.3 include heavy ublk state changes (it is now a
> > bit closer to what fuse-io-uring does) and also adds debugfs that can be
> > used understand state issues.
> >
> >
> > Thanks,
> > Bernd
> >
> >
> > >
> > > 1. Environment
> > > ==============
> > >
> > > - Breaks on 7.0.0-31-generic and 7.0.0-43-generic using the kernels that ship
> > > with Ubuntu. Tested on amd64 and the ublk_drv module.
> > > - An identical binary was run on 6.8.0-117-generic but no hangs were reproduced.
> > > - This was run on virtual machines provisioned with LXD and with 4 CPUs.
> > > - After reading the code, I suspect the issue arises due to the 2025 rework (the
> > > list batching first appeared in 6.15 and the per-IO task tracking in 6.16).
> > > I have not run any kernels between 6.8 and 7.0
> > >
> > > 2. About the attached reproducer
> > > ================================
> > >
> > > - It's a single C file that requires only liburing.
> > > - 3 devices are loaded (2 are file-backed, the other memory-backed).
> > > - A fourth device is added, started, loaded (briefly), stopped and
> > > deleted in a loop while the others are under loaded. This is the
> > > fastest way I found to reproduce the issue.
> > > - On 7.0, this hangs within the first 2 runs. On 6.8, no hangs were
> > > observed after more than 10 runs. When the issue is detected, the
> > > program prints what evidence it can gather (per-device inflight
> > > operation numbers, the running threads' waiting channels and the
> > > io_uring's pending CQE counts).
> > > - Bonus: 2 additional bugs where found when shrinking the workload:
> > > A queue was leaking its io_uring (keeping the chardev pinned), and
> > > START_DEV issued for a queued that had died before arming. Both stalled
> > > the program identically on 6.8 and both were fixed in the attached version.
> > >
> > > 3. What indicates a failure (from userspace's POV)
> > > ==================================================
> > >
> > > - The issue is always connected to a READ request:
> > > /sys/block/ublkb$N/inflight yields "1 0" and is never modified again.
> > > - The thread that handles the queue is blocked in io_cqring_wait (as seen by
> > > inspecting /proc/$TID/wchan), byt the ring has no entries to consume.
> > > - The ublksrv_io_desc for the corresponding tag was written by the kernel
> > > at dispatch. The server reads descriptors after a CQE is posted, but
> > > this never happens.
> > > - There is no way (AFAIK) to recover the tag. The lost completion is the
> > > one that would set UBLK_IO_FLAG_OWNED_BY_SRV so a COMMIT_AND_FETCH for
> > > the tag is rejected (returns -EINVAL).
> > > - Something very similar also shows on the control plane, which is why I
> > > suspect shared implementation details rather than something queue-specific.
> > > - START_DEV, STOP_DEV and DEL_DEV will occassionnally never post their CQE
> > > after the kernel provably completes. In one capture, the deleted block
> > > device was already gone from /sys/block when the caller's 30 seconds
> > > time bound fired. Repeated re-enters never produced the completion.
> > > - DEL_DEV was seen stalling 30-90 seconds on the kernel for operations that
> > > should complete instantly. The io-wq worker thread was blocked in:
> > >
> > > ublk_ctrl_del_dev
> > > del_gendisk
> > > blk_mq_freeze_queue_wait
> > >
> > > the device was stuck half-deleted, and it reacted to ADD_DEV by returning
> > > -EEXIST for minutes. In the worst instance it survivied its own server
> > > exiting and stayed un-deletable (needed a reboot).
> > > - A START_DEV completed with -EINTR. The process was detached and arms no
> > > signal sources. I didn't capture the pending signal set in that instance,
> > > so I cannot rule out a spurious signal, but the same op's immediate retry
> > > stalled for a full 30 seconds.
> > > - Not needed to repro the bug: COOP_TASKRUN, readahead bursts, eventfd wakeups.
> > >
> > > 4. Aftermath
> > > ============
> > >
> > > - AFter the request is lost, teardown hangs and takes the control plane with
> > > it. Looking at /proc/*/wchan and dmesg of a hanged instance shows that the
> > > teardown uring command (running in an io-wq worker) is stucj in
> > >
> > > ublk_stop_dev/ublk_ctrl_del_dev
> > > del_gendisk
> > > blk_unregister_queue
> > > elevator_set_none
> > > blk_mq_freeze_queue_wait
> > >
> > > And the message printed by the kernel is:
> > >
> > > INFO: task iou-wrk-2115:2118 blocked for more than 122 seconds.
> > > [...]
> > > ublk_stop_dev_unlocked.part.0+0x38/0x160 [ublk_drv]
> > > ublk_ctrl_uring_cmd+0x629/0x830 [ublk_drv]
> > >
> > > - Subsequent control command hangs on the control mutex:
> > > task iou-wrk-12720:12721 is blocked on a mutex likely owned by task
> > > iou-wrk-12626:12629.
> > > [...]
> > > ublk_ctrl_get_params+0x8b/0x190 [ublk_drv]
> > >
> > > - Killing the process doesn't help. The worker thread keeps waiting on the
> > > queue and the frozen queue waits for a request whose owner (the server) is
> > > gone. On 7.0, ublk_timeout(...) only takes action for unprivileged devices by
> > > SIGKILL'ing the server. On privileged processes it retursn BLK_EH_RESET_TIMER
> > > so a lost request is re-armed forever and never aborted, even when the
> > > server is dead.
> > >
> > > - When the process exits, the io_uring io-wq teardown warns while a worker is
> > > still stuck and the process remains a zombie. The following message is
> > > printed by the kernel:
> > >
> > > WARNING: io_uring/io-wq.c:1370 at io_wq_put_and_exit+0x10d/0x2e0,
> > > CPU#3: live_server_ram/2115
> > >
> > > 5. Where I suspect is the problem
> > > =================================
> > >
> > > - Evidence points at the completion side of the 2025 rework.
> > > - per-IO task tracking ( ("ublk: require unique task per io instead of
> > > unique task per queue", Pure Storage, 04/2025,
> > > https://patchew.org/linux/20250416-ublk._5Ftask._5Fper._5Fio-v5-0-9261ad7bff20@purestorage.com/):
> > > our fetches are all submitted by the queue's own thread, so io->task
> > > should always match at dispatch time;
> > > - rq_list batching: ublk_queue_rqs() coalesces same-<ctx_handle, task>
> > > requests into one list and queues a single
> > > io_uring_cmd_complete_in_task(io->cmd, ublk_cmd_list_tw_cb) for the
> > > whole batch. If that single task work is lost, every request in the
> > > list strands - the observations are consistent with one lost request per
> > > occurrence.
> > >
> > > 6. Follow-ups
> > > =============
> > >
> > > - Does this ring any bells in the 7.0 completion rework (lost task work
> > > for ublk_cmd_list_tw_cb / ublk_cmd_tw_cb, or io->task mismatch taking
> > > a path that neither completes nor aborts)? Is a fix already in
> > > flight?
> > > - Independently of the completion loss: should ublk_timeout() for
> > > privileged daemons fail a request whose ublk server is gone? Today
> > > BLK_EH_RESET_TIMER keeps a lost request - and with it the del_gendisk
> > > freeze and the control mutex - stuck forever after server's death,
> > > which turns a single lost completion into a machine-level hang.
> > >
> > > I can test candidate patches on 7.0.0-43 or bisect across 6.15-7.0 on a
> > > VM, or capture the trace of a missing completion.
> > >
> > > #regzbot introduced: v6.8..v7.0
> > >
> > > Best regards,
> > > Luciano
> >
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: [REGRESSION] ublk: dispatched IO intermittently never posts its fetch completion (v7.0)
2026-11-21 2:25 ` Luciano Lo Giudice
@ 2026-11-21 5:23 ` Ming Lei
2026-11-21 17:52 ` Luciano Lo Giudice
0 siblings, 1 reply; 7+ messages in thread
From: Ming Lei @ 2026-11-21 5:23 UTC (permalink / raw)
To: Luciano Lo Giudice
Cc: Bernd Schubert, linux-block, io-uring, regressions, axboe, stable
On Fri, Nov 20, 2026 at 8:26 PM Luciano Lo Giudice
<lmlogiudice@gmail.com> wrote:
>
> Hello everyone,
>
> Using Bernd's repo, I was _unable_ to reproduce the bug after several
> runs (it used to hang in about 1 out of 3 runs). I had to add a small
> compat layer to make the module build in 7.0, but it was overall
> pretty easy. I will be now using this module for any development on
> 7.0 (thanks, Bernd!) and will report if any other issues arise.
I can't reproduce it on 7.4-rc, and we did merge many similar fixes recently.
Unfortunately, 7.0 isn't a stable release, so can you move to 7.3
stable release?
Otherwise, you may have to sort out the backport by yourself.
Thanks,
Ming
>
> All the best,
> Luciano.
>
> On Fri, Nov 20, 2026 at 8:40 PM Luciano Lo Giudice
> <lmlogiudice@gmail.com> wrote:
> >
> > Hello Bernd,
> >
> > Yes, I can do that. I'll go ahead and compile ublk_drv from your repo
> > for the 7.0 backport and see how that goes.
> >
> > All the best,
> > Luciano.
> >
> > On Fri, Nov 20, 2026 at 8:27 PM Bernd Schubert <bernd@bsbernd.com> wrote:
> > >
> > >
> > >
> > > On 11/20/26 23:55, Luciano Lo Giudice wrote:
> > > > From: Luciano Lo Giudice <lmlogiudice@gmail.com>
> > > > To: linux-block@vger.kernel.org, io-uring@vger.kernel.org
> > > > Cc: Ming Lei <ming.lei@redhat.com>, Jens Axboe <axboe@kernel.dk>,
> > > > stable@vger.kernel.org, regressions@lists.linux.dev
> > > > Subject: [REGRESSION] ublk: dispatched IO intermittently never posts its
> > > > fetch completion (v7.0)
> > > >
> > > > Hello everyone,
> > > >
> > > > I'm writing a library that implements various block devices in userspace by
> > > > utilizing the ublk framework. Every so often, I'm hitting a completion loss on
> > > > Linux 7.0, in which an inflight request is lost forever, which leads to an
> > > > unkillable process and a ublk block device that cannot be torn down, and so the
> > > > machine needs to be rebooted to remedy this. I've tested the attached reproducer
> > > > on 6.8, but found no hangs after many runs. Details of the bug are below.
> > >
> > > Hi Luciano,
> > >
> > > I just scanned the io-uring list and found this by accident. I have been
> > > working on this for a couple of month.
> > >
> > >
> > > Would you be open to test a specific branch or dkms module? I actually
> > > wanted to send this to upstream last months already, but then found and
> > > fixed (with a lot of AI help) more issues and also got busy with other
> > > tasks.
> > >
> > > https://github.com/bsbernd/linux/tree/ublk-stop-races-with-tests-for-7.3
> > >
> > > and here the backports, that also work on 7.0 (with a compat patch) and
> > > that is used for our dkms module
> > >
> > > https://github.com/bsbernd/linux/tree/niova-ublk-7.1-fix-backports
> > >
> > > The 7.3 patches are actually verified through tla+ and an ublk branch
> > > for https://github.com/brauner/linux-formal
> > >
> > >
> > > @Ming the patches for 7.3 include heavy ublk state changes (it is now a
> > > bit closer to what fuse-io-uring does) and also adds debugfs that can be
> > > used understand state issues.
> > >
> > >
> > > Thanks,
> > > Bernd
> > >
> > >
> > > >
> > > > 1. Environment
> > > > ==============
> > > >
> > > > - Breaks on 7.0.0-31-generic and 7.0.0-43-generic using the kernels that ship
> > > > with Ubuntu. Tested on amd64 and the ublk_drv module.
> > > > - An identical binary was run on 6.8.0-117-generic but no hangs were reproduced.
> > > > - This was run on virtual machines provisioned with LXD and with 4 CPUs.
> > > > - After reading the code, I suspect the issue arises due to the 2025 rework (the
> > > > list batching first appeared in 6.15 and the per-IO task tracking in 6.16).
> > > > I have not run any kernels between 6.8 and 7.0
> > > >
> > > > 2. About the attached reproducer
> > > > ================================
> > > >
> > > > - It's a single C file that requires only liburing.
> > > > - 3 devices are loaded (2 are file-backed, the other memory-backed).
> > > > - A fourth device is added, started, loaded (briefly), stopped and
> > > > deleted in a loop while the others are under loaded. This is the
> > > > fastest way I found to reproduce the issue.
> > > > - On 7.0, this hangs within the first 2 runs. On 6.8, no hangs were
> > > > observed after more than 10 runs. When the issue is detected, the
> > > > program prints what evidence it can gather (per-device inflight
> > > > operation numbers, the running threads' waiting channels and the
> > > > io_uring's pending CQE counts).
> > > > - Bonus: 2 additional bugs where found when shrinking the workload:
> > > > A queue was leaking its io_uring (keeping the chardev pinned), and
> > > > START_DEV issued for a queued that had died before arming. Both stalled
> > > > the program identically on 6.8 and both were fixed in the attached version.
> > > >
> > > > 3. What indicates a failure (from userspace's POV)
> > > > ==================================================
> > > >
> > > > - The issue is always connected to a READ request:
> > > > /sys/block/ublkb$N/inflight yields "1 0" and is never modified again.
> > > > - The thread that handles the queue is blocked in io_cqring_wait (as seen by
> > > > inspecting /proc/$TID/wchan), byt the ring has no entries to consume.
> > > > - The ublksrv_io_desc for the corresponding tag was written by the kernel
> > > > at dispatch. The server reads descriptors after a CQE is posted, but
> > > > this never happens.
> > > > - There is no way (AFAIK) to recover the tag. The lost completion is the
> > > > one that would set UBLK_IO_FLAG_OWNED_BY_SRV so a COMMIT_AND_FETCH for
> > > > the tag is rejected (returns -EINVAL).
> > > > - Something very similar also shows on the control plane, which is why I
> > > > suspect shared implementation details rather than something queue-specific.
> > > > - START_DEV, STOP_DEV and DEL_DEV will occassionnally never post their CQE
> > > > after the kernel provably completes. In one capture, the deleted block
> > > > device was already gone from /sys/block when the caller's 30 seconds
> > > > time bound fired. Repeated re-enters never produced the completion.
> > > > - DEL_DEV was seen stalling 30-90 seconds on the kernel for operations that
> > > > should complete instantly. The io-wq worker thread was blocked in:
> > > >
> > > > ublk_ctrl_del_dev
> > > > del_gendisk
> > > > blk_mq_freeze_queue_wait
> > > >
> > > > the device was stuck half-deleted, and it reacted to ADD_DEV by returning
> > > > -EEXIST for minutes. In the worst instance it survivied its own server
> > > > exiting and stayed un-deletable (needed a reboot).
> > > > - A START_DEV completed with -EINTR. The process was detached and arms no
> > > > signal sources. I didn't capture the pending signal set in that instance,
> > > > so I cannot rule out a spurious signal, but the same op's immediate retry
> > > > stalled for a full 30 seconds.
> > > > - Not needed to repro the bug: COOP_TASKRUN, readahead bursts, eventfd wakeups.
> > > >
> > > > 4. Aftermath
> > > > ============
> > > >
> > > > - AFter the request is lost, teardown hangs and takes the control plane with
> > > > it. Looking at /proc/*/wchan and dmesg of a hanged instance shows that the
> > > > teardown uring command (running in an io-wq worker) is stucj in
> > > >
> > > > ublk_stop_dev/ublk_ctrl_del_dev
> > > > del_gendisk
> > > > blk_unregister_queue
> > > > elevator_set_none
> > > > blk_mq_freeze_queue_wait
> > > >
> > > > And the message printed by the kernel is:
> > > >
> > > > INFO: task iou-wrk-2115:2118 blocked for more than 122 seconds.
> > > > [...]
> > > > ublk_stop_dev_unlocked.part.0+0x38/0x160 [ublk_drv]
> > > > ublk_ctrl_uring_cmd+0x629/0x830 [ublk_drv]
> > > >
> > > > - Subsequent control command hangs on the control mutex:
> > > > task iou-wrk-12720:12721 is blocked on a mutex likely owned by task
> > > > iou-wrk-12626:12629.
> > > > [...]
> > > > ublk_ctrl_get_params+0x8b/0x190 [ublk_drv]
> > > >
> > > > - Killing the process doesn't help. The worker thread keeps waiting on the
> > > > queue and the frozen queue waits for a request whose owner (the server) is
> > > > gone. On 7.0, ublk_timeout(...) only takes action for unprivileged devices by
> > > > SIGKILL'ing the server. On privileged processes it retursn BLK_EH_RESET_TIMER
> > > > so a lost request is re-armed forever and never aborted, even when the
> > > > server is dead.
> > > >
> > > > - When the process exits, the io_uring io-wq teardown warns while a worker is
> > > > still stuck and the process remains a zombie. The following message is
> > > > printed by the kernel:
> > > >
> > > > WARNING: io_uring/io-wq.c:1370 at io_wq_put_and_exit+0x10d/0x2e0,
> > > > CPU#3: live_server_ram/2115
> > > >
> > > > 5. Where I suspect is the problem
> > > > =================================
> > > >
> > > > - Evidence points at the completion side of the 2025 rework.
> > > > - per-IO task tracking ( ("ublk: require unique task per io instead of
> > > > unique task per queue", Pure Storage, 04/2025,
> > > > https://patchew.org/linux/20250416-ublk._5Ftask._5Fper._5Fio-v5-0-9261ad7bff20@purestorage.com/):
> > > > our fetches are all submitted by the queue's own thread, so io->task
> > > > should always match at dispatch time;
> > > > - rq_list batching: ublk_queue_rqs() coalesces same-<ctx_handle, task>
> > > > requests into one list and queues a single
> > > > io_uring_cmd_complete_in_task(io->cmd, ublk_cmd_list_tw_cb) for the
> > > > whole batch. If that single task work is lost, every request in the
> > > > list strands - the observations are consistent with one lost request per
> > > > occurrence.
> > > >
> > > > 6. Follow-ups
> > > > =============
> > > >
> > > > - Does this ring any bells in the 7.0 completion rework (lost task work
> > > > for ublk_cmd_list_tw_cb / ublk_cmd_tw_cb, or io->task mismatch taking
> > > > a path that neither completes nor aborts)? Is a fix already in
> > > > flight?
> > > > - Independently of the completion loss: should ublk_timeout() for
> > > > privileged daemons fail a request whose ublk server is gone? Today
> > > > BLK_EH_RESET_TIMER keeps a lost request - and with it the del_gendisk
> > > > freeze and the control mutex - stuck forever after server's death,
> > > > which turns a single lost completion into a machine-level hang.
> > > >
> > > > I can test candidate patches on 7.0.0-43 or bisect across 6.15-7.0 on a
> > > > VM, or capture the trace of a missing completion.
> > > >
> > > > #regzbot introduced: v6.8..v7.0
> > > >
> > > > Best regards,
> > > > Luciano
> > >
--
Ming Lei
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: [REGRESSION] ublk: dispatched IO intermittently never posts its fetch completion (v7.0)
2026-11-21 5:23 ` Ming Lei
@ 2026-11-21 17:52 ` Luciano Lo Giudice
2026-11-21 20:59 ` Jens Axboe
0 siblings, 1 reply; 7+ messages in thread
From: Luciano Lo Giudice @ 2026-11-21 17:52 UTC (permalink / raw)
To: Ming Lei
Cc: Bernd Schubert, linux-block, io-uring, regressions, axboe, stable
Hello Ming,
Thank you for confirming the apparent fix in 7.4 as well. Do I
understand correctly that no fixes to address this problem will be
backported to 7.0?
All the best,
Luciano.
On Sat, Nov 21, 2026 at 2:23 AM Ming Lei <tom.leiming@gmail.com> wrote:
>
> On Fri, Nov 20, 2026 at 8:26 PM Luciano Lo Giudice
> <lmlogiudice@gmail.com> wrote:
> >
> > Hello everyone,
> >
> > Using Bernd's repo, I was _unable_ to reproduce the bug after several
> > runs (it used to hang in about 1 out of 3 runs). I had to add a small
> > compat layer to make the module build in 7.0, but it was overall
> > pretty easy. I will be now using this module for any development on
> > 7.0 (thanks, Bernd!) and will report if any other issues arise.
>
> I can't reproduce it on 7.4-rc, and we did merge many similar fixes recently.
> Unfortunately, 7.0 isn't a stable release, so can you move to 7.3
> stable release?
> Otherwise, you may have to sort out the backport by yourself.
>
> Thanks,
> Ming
>
>
> >
> > All the best,
> > Luciano.
> >
> > On Fri, Nov 20, 2026 at 8:40 PM Luciano Lo Giudice
> > <lmlogiudice@gmail.com> wrote:
> > >
> > > Hello Bernd,
> > >
> > > Yes, I can do that. I'll go ahead and compile ublk_drv from your repo
> > > for the 7.0 backport and see how that goes.
> > >
> > > All the best,
> > > Luciano.
> > >
> > > On Fri, Nov 20, 2026 at 8:27 PM Bernd Schubert <bernd@bsbernd.com> wrote:
> > > >
> > > >
> > > >
> > > > On 11/20/26 23:55, Luciano Lo Giudice wrote:
> > > > > From: Luciano Lo Giudice <lmlogiudice@gmail.com>
> > > > > To: linux-block@vger.kernel.org, io-uring@vger.kernel.org
> > > > > Cc: Ming Lei <ming.lei@redhat.com>, Jens Axboe <axboe@kernel.dk>,
> > > > > stable@vger.kernel.org, regressions@lists.linux.dev
> > > > > Subject: [REGRESSION] ublk: dispatched IO intermittently never posts its
> > > > > fetch completion (v7.0)
> > > > >
> > > > > Hello everyone,
> > > > >
> > > > > I'm writing a library that implements various block devices in userspace by
> > > > > utilizing the ublk framework. Every so often, I'm hitting a completion loss on
> > > > > Linux 7.0, in which an inflight request is lost forever, which leads to an
> > > > > unkillable process and a ublk block device that cannot be torn down, and so the
> > > > > machine needs to be rebooted to remedy this. I've tested the attached reproducer
> > > > > on 6.8, but found no hangs after many runs. Details of the bug are below.
> > > >
> > > > Hi Luciano,
> > > >
> > > > I just scanned the io-uring list and found this by accident. I have been
> > > > working on this for a couple of month.
> > > >
> > > >
> > > > Would you be open to test a specific branch or dkms module? I actually
> > > > wanted to send this to upstream last months already, but then found and
> > > > fixed (with a lot of AI help) more issues and also got busy with other
> > > > tasks.
> > > >
> > > > https://github.com/bsbernd/linux/tree/ublk-stop-races-with-tests-for-7.3
> > > >
> > > > and here the backports, that also work on 7.0 (with a compat patch) and
> > > > that is used for our dkms module
> > > >
> > > > https://github.com/bsbernd/linux/tree/niova-ublk-7.1-fix-backports
> > > >
> > > > The 7.3 patches are actually verified through tla+ and an ublk branch
> > > > for https://github.com/brauner/linux-formal
> > > >
> > > >
> > > > @Ming the patches for 7.3 include heavy ublk state changes (it is now a
> > > > bit closer to what fuse-io-uring does) and also adds debugfs that can be
> > > > used understand state issues.
> > > >
> > > >
> > > > Thanks,
> > > > Bernd
> > > >
> > > >
> > > > >
> > > > > 1. Environment
> > > > > ==============
> > > > >
> > > > > - Breaks on 7.0.0-31-generic and 7.0.0-43-generic using the kernels that ship
> > > > > with Ubuntu. Tested on amd64 and the ublk_drv module.
> > > > > - An identical binary was run on 6.8.0-117-generic but no hangs were reproduced.
> > > > > - This was run on virtual machines provisioned with LXD and with 4 CPUs.
> > > > > - After reading the code, I suspect the issue arises due to the 2025 rework (the
> > > > > list batching first appeared in 6.15 and the per-IO task tracking in 6.16).
> > > > > I have not run any kernels between 6.8 and 7.0
> > > > >
> > > > > 2. About the attached reproducer
> > > > > ================================
> > > > >
> > > > > - It's a single C file that requires only liburing.
> > > > > - 3 devices are loaded (2 are file-backed, the other memory-backed).
> > > > > - A fourth device is added, started, loaded (briefly), stopped and
> > > > > deleted in a loop while the others are under loaded. This is the
> > > > > fastest way I found to reproduce the issue.
> > > > > - On 7.0, this hangs within the first 2 runs. On 6.8, no hangs were
> > > > > observed after more than 10 runs. When the issue is detected, the
> > > > > program prints what evidence it can gather (per-device inflight
> > > > > operation numbers, the running threads' waiting channels and the
> > > > > io_uring's pending CQE counts).
> > > > > - Bonus: 2 additional bugs where found when shrinking the workload:
> > > > > A queue was leaking its io_uring (keeping the chardev pinned), and
> > > > > START_DEV issued for a queued that had died before arming. Both stalled
> > > > > the program identically on 6.8 and both were fixed in the attached version.
> > > > >
> > > > > 3. What indicates a failure (from userspace's POV)
> > > > > ==================================================
> > > > >
> > > > > - The issue is always connected to a READ request:
> > > > > /sys/block/ublkb$N/inflight yields "1 0" and is never modified again.
> > > > > - The thread that handles the queue is blocked in io_cqring_wait (as seen by
> > > > > inspecting /proc/$TID/wchan), byt the ring has no entries to consume.
> > > > > - The ublksrv_io_desc for the corresponding tag was written by the kernel
> > > > > at dispatch. The server reads descriptors after a CQE is posted, but
> > > > > this never happens.
> > > > > - There is no way (AFAIK) to recover the tag. The lost completion is the
> > > > > one that would set UBLK_IO_FLAG_OWNED_BY_SRV so a COMMIT_AND_FETCH for
> > > > > the tag is rejected (returns -EINVAL).
> > > > > - Something very similar also shows on the control plane, which is why I
> > > > > suspect shared implementation details rather than something queue-specific.
> > > > > - START_DEV, STOP_DEV and DEL_DEV will occassionnally never post their CQE
> > > > > after the kernel provably completes. In one capture, the deleted block
> > > > > device was already gone from /sys/block when the caller's 30 seconds
> > > > > time bound fired. Repeated re-enters never produced the completion.
> > > > > - DEL_DEV was seen stalling 30-90 seconds on the kernel for operations that
> > > > > should complete instantly. The io-wq worker thread was blocked in:
> > > > >
> > > > > ublk_ctrl_del_dev
> > > > > del_gendisk
> > > > > blk_mq_freeze_queue_wait
> > > > >
> > > > > the device was stuck half-deleted, and it reacted to ADD_DEV by returning
> > > > > -EEXIST for minutes. In the worst instance it survivied its own server
> > > > > exiting and stayed un-deletable (needed a reboot).
> > > > > - A START_DEV completed with -EINTR. The process was detached and arms no
> > > > > signal sources. I didn't capture the pending signal set in that instance,
> > > > > so I cannot rule out a spurious signal, but the same op's immediate retry
> > > > > stalled for a full 30 seconds.
> > > > > - Not needed to repro the bug: COOP_TASKRUN, readahead bursts, eventfd wakeups.
> > > > >
> > > > > 4. Aftermath
> > > > > ============
> > > > >
> > > > > - AFter the request is lost, teardown hangs and takes the control plane with
> > > > > it. Looking at /proc/*/wchan and dmesg of a hanged instance shows that the
> > > > > teardown uring command (running in an io-wq worker) is stucj in
> > > > >
> > > > > ublk_stop_dev/ublk_ctrl_del_dev
> > > > > del_gendisk
> > > > > blk_unregister_queue
> > > > > elevator_set_none
> > > > > blk_mq_freeze_queue_wait
> > > > >
> > > > > And the message printed by the kernel is:
> > > > >
> > > > > INFO: task iou-wrk-2115:2118 blocked for more than 122 seconds.
> > > > > [...]
> > > > > ublk_stop_dev_unlocked.part.0+0x38/0x160 [ublk_drv]
> > > > > ublk_ctrl_uring_cmd+0x629/0x830 [ublk_drv]
> > > > >
> > > > > - Subsequent control command hangs on the control mutex:
> > > > > task iou-wrk-12720:12721 is blocked on a mutex likely owned by task
> > > > > iou-wrk-12626:12629.
> > > > > [...]
> > > > > ublk_ctrl_get_params+0x8b/0x190 [ublk_drv]
> > > > >
> > > > > - Killing the process doesn't help. The worker thread keeps waiting on the
> > > > > queue and the frozen queue waits for a request whose owner (the server) is
> > > > > gone. On 7.0, ublk_timeout(...) only takes action for unprivileged devices by
> > > > > SIGKILL'ing the server. On privileged processes it retursn BLK_EH_RESET_TIMER
> > > > > so a lost request is re-armed forever and never aborted, even when the
> > > > > server is dead.
> > > > >
> > > > > - When the process exits, the io_uring io-wq teardown warns while a worker is
> > > > > still stuck and the process remains a zombie. The following message is
> > > > > printed by the kernel:
> > > > >
> > > > > WARNING: io_uring/io-wq.c:1370 at io_wq_put_and_exit+0x10d/0x2e0,
> > > > > CPU#3: live_server_ram/2115
> > > > >
> > > > > 5. Where I suspect is the problem
> > > > > =================================
> > > > >
> > > > > - Evidence points at the completion side of the 2025 rework.
> > > > > - per-IO task tracking ( ("ublk: require unique task per io instead of
> > > > > unique task per queue", Pure Storage, 04/2025,
> > > > > https://patchew.org/linux/20250416-ublk._5Ftask._5Fper._5Fio-v5-0-9261ad7bff20@purestorage.com/):
> > > > > our fetches are all submitted by the queue's own thread, so io->task
> > > > > should always match at dispatch time;
> > > > > - rq_list batching: ublk_queue_rqs() coalesces same-<ctx_handle, task>
> > > > > requests into one list and queues a single
> > > > > io_uring_cmd_complete_in_task(io->cmd, ublk_cmd_list_tw_cb) for the
> > > > > whole batch. If that single task work is lost, every request in the
> > > > > list strands - the observations are consistent with one lost request per
> > > > > occurrence.
> > > > >
> > > > > 6. Follow-ups
> > > > > =============
> > > > >
> > > > > - Does this ring any bells in the 7.0 completion rework (lost task work
> > > > > for ublk_cmd_list_tw_cb / ublk_cmd_tw_cb, or io->task mismatch taking
> > > > > a path that neither completes nor aborts)? Is a fix already in
> > > > > flight?
> > > > > - Independently of the completion loss: should ublk_timeout() for
> > > > > privileged daemons fail a request whose ublk server is gone? Today
> > > > > BLK_EH_RESET_TIMER keeps a lost request - and with it the del_gendisk
> > > > > freeze and the control mutex - stuck forever after server's death,
> > > > > which turns a single lost completion into a machine-level hang.
> > > > >
> > > > > I can test candidate patches on 7.0.0-43 or bisect across 6.15-7.0 on a
> > > > > VM, or capture the trace of a missing completion.
> > > > >
> > > > > #regzbot introduced: v6.8..v7.0
> > > > >
> > > > > Best regards,
> > > > > Luciano
> > > >
>
>
>
> --
> Ming Lei
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: [REGRESSION] ublk: dispatched IO intermittently never posts its fetch completion (v7.0)
2026-11-21 17:52 ` Luciano Lo Giudice
@ 2026-11-21 20:59 ` Jens Axboe
0 siblings, 0 replies; 7+ messages in thread
From: Jens Axboe @ 2026-11-21 20:59 UTC (permalink / raw)
To: Luciano Lo Giudice, Ming Lei
Cc: Bernd Schubert, linux-block, io-uring, regressions, stable
On 11/21/26 10:52 AM, Luciano Lo Giudice wrote:
> Hello Ming,
>
> Thank you for confirming the apparent fix in 7.4 as well. Do I
> understand correctly that no fixes to address this problem will be
> backported to 7.0?
As Ming said, 7.0 is not a maintained stable release. That means
no further 7.0.x releases will be done, and nothing will change
that. Also means that if you're running 7.0.x you've been missing
out on a lot of fixes that have gone into all stable releases and
upstream kernels since then.
Just get on 7.3-stable.
--
Jens Axboe
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2026-11-21 20:59 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-20 22:55 [REGRESSION] ublk: dispatched IO intermittently never posts its fetch completion (v7.0) Luciano Lo Giudice
2026-11-20 23:27 ` Bernd Schubert
2026-11-20 23:40 ` Luciano Lo Giudice
2026-11-21 2:25 ` Luciano Lo Giudice
2026-11-21 5:23 ` Ming Lei
2026-11-21 17:52 ` Luciano Lo Giudice
2026-11-21 20:59 ` Jens Axboe
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox