Linux block layer
 help / color / mirror / Atom feed
* [REGRESSION] ublk: dispatched IO intermittently never posts its fetch completion (v7.0)
@ 2026-11-20 22:55 Luciano Lo Giudice
  2026-11-20 23:27 ` Bernd Schubert
  0 siblings, 1 reply; 7+ messages in thread
From: Luciano Lo Giudice @ 2026-11-20 22:55 UTC (permalink / raw)
  To: linux-block, io-uring; +Cc: regressions, axboe, stable

[-- Attachment #1: Type: text/plain, Size: 7314 bytes --]

From: Luciano Lo Giudice <lmlogiudice@gmail.com>
To: linux-block@vger.kernel.org, io-uring@vger.kernel.org
Cc: Ming Lei <ming.lei@redhat.com>, Jens Axboe <axboe@kernel.dk>,
    stable@vger.kernel.org, regressions@lists.linux.dev
Subject: [REGRESSION] ublk: dispatched IO intermittently never posts its
 fetch completion (v7.0)

Hello everyone,

I'm writing a library that implements various block devices in userspace by
utilizing the ublk framework. Every so often, I'm hitting a completion loss on
Linux 7.0, in which an inflight request is lost forever, which leads to an
unkillable process and a ublk block device that cannot be torn down, and so the
machine needs to be rebooted to remedy this. I've tested the attached reproducer
on 6.8, but found no hangs after many runs. Details of the bug are below.

1. Environment
==============

- Breaks on 7.0.0-31-generic and 7.0.0-43-generic using the kernels that ship
  with Ubuntu. Tested on amd64 and the ublk_drv module.
- An identical binary was run on 6.8.0-117-generic but no hangs were reproduced.
- This was run on virtual machines provisioned with LXD and with 4 CPUs.
- After reading the code, I suspect the issue arises due to the 2025 rework (the
  list batching first appeared in 6.15 and the per-IO task tracking in 6.16).
  I have not run any kernels between 6.8 and 7.0

2. About the attached reproducer
================================

- It's a single C file that requires only liburing.
- 3 devices are loaded (2 are file-backed, the other memory-backed).
- A fourth device is added, started, loaded (briefly), stopped and
  deleted in a loop while the others are under loaded. This is the
  fastest way I found to reproduce the issue.
- On 7.0, this hangs within the first 2 runs. On 6.8, no hangs were
  observed after more than 10 runs. When the issue is detected, the
  program prints what evidence it can gather (per-device inflight
  operation numbers, the running threads' waiting channels and the
  io_uring's pending CQE counts).
- Bonus: 2 additional bugs where found when shrinking the workload:
  A queue was leaking its io_uring (keeping the chardev pinned), and
  START_DEV issued for a queued that had died before arming. Both stalled
  the program identically on 6.8 and both were fixed in the attached version.

3. What indicates a failure (from userspace's POV)
==================================================

- The issue is always connected to a READ request:
  /sys/block/ublkb$N/inflight yields "1 0" and is never modified again.
- The thread that handles the queue is blocked in io_cqring_wait (as seen by
  inspecting /proc/$TID/wchan), byt the ring has no entries to consume.
- The ublksrv_io_desc for the corresponding tag was written by the kernel
  at dispatch. The server reads descriptors after a CQE is posted, but
  this never happens.
- There is no way (AFAIK) to recover the tag. The lost completion is the
  one that would set UBLK_IO_FLAG_OWNED_BY_SRV so a COMMIT_AND_FETCH for
  the tag is rejected (returns -EINVAL).
- Something very similar also shows on the control plane, which is why I
  suspect shared implementation details rather than something queue-specific.
- START_DEV, STOP_DEV and DEL_DEV will occassionnally never post their CQE
  after the kernel provably completes. In one capture, the deleted block
  device was already gone from /sys/block when the caller's 30 seconds
  time bound fired. Repeated re-enters never produced the completion.
- DEL_DEV was seen stalling 30-90 seconds on the kernel for operations that
  should complete instantly. The io-wq worker thread was blocked in:

  ublk_ctrl_del_dev
    del_gendisk
      blk_mq_freeze_queue_wait

  the device was stuck half-deleted, and it reacted to ADD_DEV by returning
  -EEXIST for minutes. In the worst instance it survivied its own server
  exiting and stayed un-deletable (needed a reboot).
- A START_DEV completed with -EINTR. The process was detached and arms no
  signal sources. I didn't capture the pending signal set in that instance,
  so I cannot rule out a spurious signal, but the same op's immediate retry
  stalled for a full 30 seconds.
- Not needed to repro the bug: COOP_TASKRUN, readahead bursts, eventfd wakeups.

4. Aftermath
============

- AFter the request is lost, teardown hangs and takes the control plane with
  it. Looking at /proc/*/wchan and dmesg of a hanged instance shows that the
  teardown uring command (running in an io-wq worker) is stucj in

  ublk_stop_dev/ublk_ctrl_del_dev
    del_gendisk
      blk_unregister_queue
        elevator_set_none
          blk_mq_freeze_queue_wait

  And the message printed by the kernel is:

    INFO: task iou-wrk-2115:2118 blocked for more than 122 seconds.
    [...]
     ublk_stop_dev_unlocked.part.0+0x38/0x160 [ublk_drv]
     ublk_ctrl_uring_cmd+0x629/0x830 [ublk_drv]

- Subsequent control command hangs on the control mutex:
    task iou-wrk-12720:12721 is blocked on a mutex likely owned by task
    iou-wrk-12626:12629.
    [...]
     ublk_ctrl_get_params+0x8b/0x190 [ublk_drv]

- Killing the process doesn't help. The worker thread keeps waiting on the
  queue and the frozen queue waits for a request whose owner (the server) is
  gone. On 7.0, ublk_timeout(...) only takes action for unprivileged devices by
  SIGKILL'ing the server. On privileged processes it retursn BLK_EH_RESET_TIMER
  so a lost request is re-armed forever and never aborted, even when the
  server is dead.

- When the process exits, the io_uring io-wq teardown warns while a worker is
  still stuck and the process remains a zombie. The following message is
  printed by the kernel:

    WARNING: io_uring/io-wq.c:1370 at io_wq_put_and_exit+0x10d/0x2e0,
    CPU#3: live_server_ram/2115

5. Where I suspect is the problem
=================================

- Evidence points at the completion side of the 2025 rework.
- per-IO task tracking ( ("ublk: require unique task per io instead of
  unique task per queue", Pure Storage, 04/2025,
  https://patchew.org/linux/20250416-ublk._5Ftask._5Fper._5Fio-v5-0-9261ad7bff20@purestorage.com/):
  our fetches are all submitted by the queue's own thread, so io->task
  should always match at dispatch time;
- rq_list batching: ublk_queue_rqs() coalesces same-<ctx_handle, task>
  requests into one list and queues a single
  io_uring_cmd_complete_in_task(io->cmd, ublk_cmd_list_tw_cb) for the
  whole batch. If that single task work is lost, every request in the
  list strands - the observations are consistent with one lost request per
  occurrence.

6. Follow-ups
=============

- Does this ring any bells in the 7.0 completion rework (lost task work
  for ublk_cmd_list_tw_cb / ublk_cmd_tw_cb, or io->task mismatch taking
  a path that neither completes nor aborts)? Is a fix already in
  flight?
- Independently of the completion loss: should ublk_timeout() for
  privileged daemons fail a request whose ublk server is gone? Today
  BLK_EH_RESET_TIMER keeps a lost request - and with it the del_gendisk
  freeze and the control mutex - stuck forever after server's death,
  which turns a single lost completion into a machine-level hang.

I can test candidate patches on 7.0.0-43 or bisect across 6.15-7.0 on a
VM, or capture the trace of a missing completion.

#regzbot introduced: v6.8..v7.0

Best regards,
Luciano

[-- Attachment #2: ublk-bug.c --]
[-- Type: text/x-csrc, Size: 38634 bytes --]

/*
 * Reproducer for the Linux 7.0 ublk lost fetch-completion bug.
 *
 * What we're after: A READ request that is dispatched by the kernel (the
 * descriptor is written to, sysfs inflight shows "1 0") but the io_uring
 * never gets the corresponding completion posted. Thus, the queue thread
 * keeps going back to io_uring_enter and repots "wait timeout: pending CQES 0"
 * every 5 seconds. Userspace cannot recover the tag and the lost request
 * hangs STOP_DEV/DEL_DEV commands. The machine needs to be rebooted.
 *
 * The watchdog thread aims to detect the hang. So when a device makes no
 * IO progress for STALL_SECS, it will print out the evidence (including
 * the wait channels for the threads) and exits.
 *
 * To build, simply run: 'cc -O2 ublk-bug.c -luring'
 * To run: sudo ./a.out [runs] (0 means loop forever)
 *
 * The program was tested on 6.8 and 7.0. In the former, it has not shown
 * any hangs. On the latter, it is consitently triggered after few rounds.
 *
 * The following parameteres may be tuned by the user at build time:
 *
 * -DNR_DEVS=3            devices served concurrently
 * -DDEPTH=32             tags per device (single queue per device)
 * -DVOL_MB=64            per-device volume size
 * -DMAX_IO_BYTES=131072  per-tag buffer / max request size
 * -DCAMPAIGN_SECS=60     seconds of load per run
 * -DSTALL_SECS=15        no-progress interval that counts as a hang
 * -DIDLE_SECS=5          idle nonzero-inflight interval = a strand
 * -DLOAD_THREADS=4       parallel load threads per device
 * -DRAMDEV_IDX=1         which device gets the in-memory backend
 * -DNO_CHURN             drop the interleaved lifecycle device
 * -DNO_COOP_TASKRUN      build the ring without IORING_SETUP_COOP_TASKRUN
*/

#define _GNU_SOURCE
#include <dirent.h>
#include <errno.h>
#include <fcntl.h>
#include <liburing.h>
#include <pthread.h>
#include <signal.h>
#include <stdatomic.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/prctl.h>
#include <sys/mman.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <time.h>
#include <unistd.h>

// Define tunables.

#ifndef NR_DEVS
  #define NR_DEVS 3
#endif

#ifndef DEPTH
  #define DEPTH 32
#endif

#ifndef VOL_MB
  #define VOL_MB 64
#endif

#ifndef MAX_IO_BYTES
  #define MAX_IO_BYTES 131072
#endif

#ifndef CAMPAIGN_SECS
  #define CAMPAIGN_SECS 60
#endif

#ifndef STALL_SECS
  #define STALL_SECS 15
#endif

#ifndef IDLE_SECS
  #define IDLE_SECS 5 // idle nonzero-inflight interval that counts as a strand
#endif

#ifndef LOAD_THREADS
  #define LOAD_THREADS 4 // parallel load threads per device (disjoint regions)
#endif

// build with -DNO_CHURN to drop the interleaved lifecycle device
//#define NO_CHURN
// which device gets an in-memory backend (memcpy, instant completions -
// the highest request rate, and the "most heavily loaded" victim of the
// original run); -1 for file backends everywhere
#ifndef RAMDEV_IDX
  #define RAMDEV_IDX 1
#endif

#define BASE_ID 80
#define QUEUE_ID 0 // one hardware queue per device

// Taken from include/linux/ublk_cmd.h

#define UBLK_U_CMD_GET_FEATURES   0x80207513UL
#define UBLK_U_CMD_ADD_DEV        0xC0207504UL
#define UBLK_U_CMD_DEL_DEV        0xC0207505UL
#define UBLK_U_CMD_START_DEV      0xC0207506UL
#define UBLK_U_CMD_STOP_DEV       0xC0207507UL
#define UBLK_U_CMD_SET_PARAMS     0xC0207508UL

#define UBLK_U_IO_FETCH_REQ              0xC0107520UL
#define UBLK_U_IO_COMMIT_AND_FETCH_REQ   0xC0107521UL

#define UBLK_IO_OP_READ   0
#define UBLK_IO_OP_WRITE  1
#define UBLK_IO_OP_FLUSH  2

#define UBLK_ATTR_VOLATILE_CACHE   (1U << 2)
#define UBLK_PARAM_TYPE_BASIC      (1U << 0)

// UBLK_IO_RES_ABORT: the only negative fetch result that is not an error.
#define UBLK_IO_RES_ABORT   (-ENODEV)

struct ublksrv_ctrl_cmd {
    uint32_t dev_id;
    uint16_t queue_id;
    uint16_t len;
    uint64_t addr;
    uint64_t data[1];
    uint16_t dev_path_len;
    uint16_t pad;
    uint32_t reserved;
};

struct ublksrv_ctrl_dev_info {
    uint16_t nr_hw_queues;
    uint16_t queue_depth;
    uint16_t state;
    uint16_t pad0;
    uint32_t max_io_buf_bytes;
    uint32_t dev_id;
    int32_t ublksrv_pid;
    uint32_t pad1;
    uint64_t flags;
    uint64_t ublksrv_flags;
    uint32_t owner_uid;
    uint32_t owner_gid;
    uint64_t reserved1;
    uint64_t reserved2;
};

struct ublksrv_io_desc {
    uint32_t op_flags; // op in bits 0-7, flags in 8-31
    uint32_t nr_sectors;
    uint64_t start_sector;
    uint64_t addr;
};

struct ublksrv_io_cmd {
    uint16_t q_id;
    uint16_t tag;
    int32_t result;
    uint64_t addr;
};

struct ublk_params_basic {
    uint32_t attrs;
    uint8_t logical_bs_shift;
    uint8_t physical_bs_shift;
    uint8_t io_opt_shift;
    uint8_t io_min_shift;
    uint32_t max_sectors;
    uint32_t chunk_sectors;
    uint64_t dev_sectors;
    uint64_t virt_boundary_mask;
};

struct ublk_params {
    uint32_t len;
    uint32_t types;
    struct ublk_params_basic basic;
};

_Static_assert(sizeof(struct ublksrv_ctrl_cmd) == 32, "ctrl cmd layout");
_Static_assert(sizeof(struct ublksrv_ctrl_dev_info) == 64, "dev info layout");
_Static_assert(sizeof(struct ublksrv_io_desc) == 24, "io desc layout");
_Static_assert(sizeof(struct ublksrv_io_cmd) == 16, "io cmd layout");

// Global definitions.

#define CTRL_NO_QUEUE      0xFFFFu
#define VOL_BYTES          ((uint64_t)(VOL_MB) << 20)
#define UD_FETCH           0x20u
#define UD_COMMIT          0x21u
#define WAIT_TIMEOUT_SEC   5

static int ctrl_fd = -1;
static struct io_uring ctrl_ring;

static pthread_mutex_t drop_lock = PTHREAD_MUTEX_INITIALIZER;
static atomic_int g_active;     // load phase of a run
static atomic_int g_stop;       // SIGINT / fatal
static atomic_uint g_run;

struct dev {
    unsigned id;
    char node[32];
    char cdev_path[32];
    char backend[64];
    int backend_fd;
    int ram_backend;   // in-memory backend instead of a file
    unsigned char *ram_buf;
    int cdev_fd;
    int depth;
    unsigned char *bufs;   // depth * MAX_IO_BYTES, the tag buffers
    struct ublksrv_io_desc *descs;   // mmap'd, kernel-written
    struct io_uring ring;
    atomic_uint armed;
    int live;   // START_DEV done, DEL_DEV done
    int quiet;   // churn queue: exit quietly on IO errors
    atomic_ullong passes;   // bumped by the load thread after every pass
};

static struct dev devs[NR_DEVS];

static void report_hang(int worst_dev);

static double now_s(void)
{
    struct timespec ts;
    clock_gettime(CLOCK_MONOTONIC, &ts);
    return (double)ts.tv_sec + (double)ts.tv_nsec / 1e9;
}

#define die(fmt, ...)   \
    do {   \
        fprintf(stderr, "FATAL %s:%d " fmt ": %s\n", __func__, __LINE__,   \
                ##__VA_ARGS__, strerror(errno));   \
        exit(1);   \
    } while (0)

/*
 * Hand-prepared IORING_OP_URING_CMD SQE - works on every liburing 2.x
 * (the prep_uring_cmd helpers only appeared later). Layout is fixed by
 * the uapi: cmd_op at offset 8, user_data at 32, cmd area at 48 (16
 * bytes on a 64-byte SQE, 80 with IORING_SETUP_SQE128).
 */

#define SQE_CMD_OP_OFF   8
#define SQE_CMD_OFF      48

static void prep_uring_cmd_sqe(struct io_uring_sqe *sqe, unsigned cmd_op,
                               int fd, uint64_t user_data)
{
    memset(sqe, 0, sizeof(*sqe));
    sqe->opcode = IORING_OP_URING_CMD;
    sqe->fd = fd;
    *(uint32_t *)((char *)sqe + SQE_CMD_OP_OFF) = cmd_op;
    sqe->user_data = user_data;
}

static uint64_t ctrl_seq;

/*
 * One synchronous round trip: URING_CMD128 linked to a wall-clock
 * timeout, so a hanged teardown surfaces as -ETIMEDOUT instead of
 * blocking us forever. -EINTR results (a signal interrupting the
 * blocking kernel-side handler - observed for DEL/STOP on this
 * kernel) are retried here; sequence tags make stale completions of
 * timed-out calls droppable.
 */
static int ctrl_once(unsigned cmd_op, struct ublksrv_ctrl_cmd *cc,
                     unsigned tmo)
{
    struct io_uring_sqe *sqe = io_uring_get_sqe(&ctrl_ring);
    struct io_uring_sqe *lt = io_uring_get_sqe(&ctrl_ring);
    struct __kernel_timespec ts = { .tv_sec = tmo, .tv_nsec = 0 };
    uint64_t cmd_tag = ctrl_seq++ << 1;
    uint64_t tmo_tag = cmd_tag | 1;
    int cmd_res = INT32_MIN, tmo_res = INT32_MAX;

    if (!sqe || !lt)
        die("ctrl ring full");

    prep_uring_cmd_sqe(sqe, cmd_op, ctrl_fd, cmd_tag);
    memcpy((char *)sqe + SQE_CMD_OFF, cc, sizeof(*cc));
    sqe->flags = IOSQE_IO_LINK;
    io_uring_prep_link_timeout(lt, &ts, 0);
    lt->user_data = tmo_tag;

    int ret = io_uring_submit_and_wait(&ctrl_ring, 1);
    if (ret < 0) {
        errno = -ret;
        die("ctrl submit op %#x", cmd_op);
    }

    /*
     * Stop once the command completed OR its timeout fired: a fired
     * timeout cancels the command, and if the kernel side is stuck
     * uninterruptibly the command's CQE may never surface at all.
     */

    while (cmd_res == INT32_MIN && tmo_res == INT32_MAX) {
        struct io_uring_cqe *cqe;
        if (io_uring_peek_cqe(&ctrl_ring, &cqe) < 0 &&
            io_uring_wait_cqe(&ctrl_ring, &cqe) < 0)
            die("ctrl reap op %#x", cmd_op);

        if (cqe->user_data == cmd_tag)
            cmd_res = cqe->res;
        else if (cqe->user_data == tmo_tag)
            tmo_res = cqe->res;

        // anything else: stale completion of an earlier call
        io_uring_cqe_seen(&ctrl_ring, cqe);
    }

    if (cmd_res == INT32_MIN) {
        fprintf(stderr, "ctrl op %#x timed out after %us "
                "(control plane hanged)\n", cmd_op, tmo);
        return -ETIMEDOUT;
    }

    if (getenv("REPRO_CTRL_TRACE"))
        fprintf(stderr, "ctrl op %#x -> %d (tmo %d)\n", cmd_op,
            cmd_res, tmo_res);
    return cmd_res;
}

static int ctrl_cmd(unsigned cmd_op, struct ublksrv_ctrl_cmd *cc, unsigned tmo)
{
    // Lock to serialize control commands during load.
    static pthread_mutex_t ctrl_mutex = PTHREAD_MUTEX_INITIALIZER;
    int res;

    pthread_mutex_lock(&ctrl_mutex);
    for (int attempt = 0; attempt < 3; attempt++) {
        res = ctrl_once(cmd_op, cc, tmo);
        if (res != -EINTR)
            break;

        /*
         * No signal should reach this process; record the pending
         * set so the spurious -EINTR is provably signal-less.
         */

        char line[128] = "";
        FILE *st = fopen("/proc/self/status", "r");
        if (st) {
            char l[128];
            while (fgets(l, sizeof(l), st))
                if ((!strncmp(l, "SigPnd", 6) ||
                     !strncmp(l, "ShdPnd", 6)) &&
                    strlen(line) + strlen(l) < sizeof(line))
                    strcat(line, l);
            fclose(st);
        }
        fprintf(stderr, "ctrl op %#x interrupted, retrying; %s\n",
                cmd_op, line);
    }
    pthread_mutex_unlock(&ctrl_mutex);
    return res;
}

/*
 * ctrl_cmd, reporting failures with the kernel's errno (not the clobbered
 * thread-local one). ADD_DEV racing a dying previous process's auto
 * teardown shows up as EEXIST here.
 */
static void ck(int res, const char *what)
{
    if (res < 0) {
        fprintf(stderr, "FATAL: %s: %s (-%u)\n", what, strerror(-res),
                (unsigned)-res);
        exit(1);
    }
}

static int ctrl_get_features(uint64_t *features)
{
    struct ublksrv_ctrl_cmd cc = {
        .dev_id = 0,
        .queue_id = CTRL_NO_QUEUE,
        .len = sizeof(*features),
        .addr = (uint64_t)(uintptr_t)features,
    };
    return ctrl_cmd(UBLK_U_CMD_GET_FEATURES, &cc, 10);
}

static int ctrl_add_dev(unsigned id, unsigned depth)
{
    struct ublksrv_ctrl_dev_info info = {
        .nr_hw_queues = 1,
        .queue_depth = depth,
        .state = 0, // UBLK_S_DEV_DEAD
        .max_io_buf_bytes = MAX_IO_BYTES,
        .dev_id = id,
        .flags = 0, // base copy mode
    };
    struct ublksrv_ctrl_cmd cc = {
        .dev_id = id,
        .queue_id = CTRL_NO_QUEUE,
        .len = sizeof(info),
        .addr = (uint64_t)(uintptr_t)&info,
    };
    return ctrl_cmd(UBLK_U_CMD_ADD_DEV, &cc, 10);
}

static int ctrl_set_params(unsigned id, uint64_t dev_sectors)
{
    struct ublk_params params = {
        .len = sizeof(params),
        .types = UBLK_PARAM_TYPE_BASIC,
        .basic = {
            .attrs = UBLK_ATTR_VOLATILE_CACHE,
            .logical_bs_shift = 12,
            .physical_bs_shift = 12,
            .io_opt_shift = 12,
            .io_min_shift = 12,
            .max_sectors = MAX_IO_BYTES / 512,
            .dev_sectors = dev_sectors / 512,
        },
    };
    struct ublksrv_ctrl_cmd cc = {
        .dev_id = id,
        .queue_id = CTRL_NO_QUEUE,
        .len = params.len,
        .addr = (uint64_t)(uintptr_t)&params,
    };
    return ctrl_cmd(UBLK_U_CMD_SET_PARAMS, &cc, 10);
}

static int ctrl_start_dev(unsigned id)
{
    struct ublksrv_ctrl_cmd cc = {
        .dev_id = id,
        .queue_id = CTRL_NO_QUEUE,
        .data[0] = getpid(),
    };
    return ctrl_cmd(UBLK_U_CMD_START_DEV, &cc, 30);
}

static int ctrl_stop_dev(unsigned id)
{
    struct ublksrv_ctrl_cmd cc = {
        .dev_id = id,
        .queue_id = CTRL_NO_QUEUE,
    };
    return ctrl_cmd(UBLK_U_CMD_STOP_DEV, &cc, 30);
}

static int ctrl_del_dev(unsigned id)
{
    struct ublksrv_ctrl_cmd cc = {
        .dev_id = id,
        .queue_id = CTRL_NO_QUEUE,
    };
    return ctrl_cmd(UBLK_U_CMD_DEL_DEV, &cc, 30);
}

// Queue-related stuff.

static inline uint64_t ud_pack(unsigned tag, unsigned kind)
{
    return tag | (kind << 16);
}

static void prep_fetch(struct io_uring_sqe *sqe, struct dev *d, unsigned tag,
                       unsigned kind, int32_t result)
{
    struct ublksrv_io_cmd cmd = {
        .q_id = QUEUE_ID,
        .tag = tag,
        .result = result,
        .addr = (uint64_t)(uintptr_t)(d->bufs + (size_t)tag * MAX_IO_BYTES),
    };
    prep_uring_cmd_sqe(sqe, kind == UD_COMMIT ?
               UBLK_U_IO_COMMIT_AND_FETCH_REQ :
               UBLK_U_IO_FETCH_REQ, 0,
               ud_pack(tag, kind));
    sqe->flags |= IOSQE_FIXED_FILE; // fd 0 = the queue chardev
    memcpy((char *)sqe + SQE_CMD_OFF, &cmd, sizeof(cmd));
}

// Fully synchronous backend implementation.

static int32_t backend_io(struct dev *d, uint32_t op,
                          uint64_t off, uint32_t len, unsigned char *buf)
{
    ssize_t n;
    if (d->ram_backend) {
        unsigned char *src = d->ram_buf + off;
        switch (op) {
        case UBLK_IO_OP_READ:
            memcpy(buf, src, len);
            return (int32_t)len;
        case UBLK_IO_OP_WRITE:
            memcpy(src, buf, len);
            return (int32_t)len;
        case UBLK_IO_OP_FLUSH:
            return 0;
        default:
            return -EIO;
        }
    }

    switch (op) {
    case UBLK_IO_OP_READ:
        n = pread(d->backend_fd, buf, len, (off_t)off);
        break;
    case UBLK_IO_OP_WRITE:
        n = pwrite(d->backend_fd, buf, len, (off_t)off);
        break;
    case UBLK_IO_OP_FLUSH:
        n = fdatasync(d->backend_fd) == 0 ? 0 : -1;
        break;
    default:
        fprintf(stderr, "dev%u: unexpected op %u\n", d->id, op);
        return -EIO;
    }

    return n == (ssize_t)len ? (int32_t)len : -EIO;
}

static void *server_fn(void *arg)
{
    struct dev *d = arg;
    unsigned depth = d->depth;
    struct io_uring_params p;

    // This is what ublksrv does; we do it for symmetry as well.
    prctl(PR_SET_IO_FLUSHER, 1, 0, 0, 0);

    d->cdev_fd = open(d->cdev_path, O_RDWR);
    if (d->cdev_fd < 0) {
        if (errno == ENOENT || errno == ENODEV)
            return NULL;
        die("open %s", d->cdev_path);
    }

    memset(&p, 0, sizeof(p));
    p.flags = IORING_SETUP_CQSIZE;
    p.cq_entries = depth * 2 + 2;
#ifndef NO_COOP_TASKRUN
    p.flags |= IORING_SETUP_COOP_TASKRUN;
#endif
    if (io_uring_queue_init_params(depth * 2 + 2, &d->ring, &p))
        die("dev%u: queue ring", d->id);

    int fds[1] = { d->cdev_fd };
    if (io_uring_register_files(&d->ring, fds, 1))
        die("dev%u: register files", d->id);

    // The descriptor array: one page-rounded region per queue, the
    // queue selected by a max-stride (4096 tags * 24 bytes) offset.
    size_t stride = (4096ul * sizeof(struct ublksrv_io_desc) + 4095) & ~4095ul;
    size_t map_len = ((size_t)depth * sizeof(struct ublksrv_io_desc) + 4095) & ~4095ul;
    d->descs = mmap(NULL, map_len, PROT_READ, MAP_SHARED | MAP_POPULATE,
                    d->cdev_fd, (off_t)(QUEUE_ID * stride));
    if (d->descs == MAP_FAILED)
        die("dev%u: desc mmap", d->id);

    // Arm the queue: one fetch per tag.
    for (unsigned tag = 0; tag < depth; tag++) {
        struct io_uring_sqe *sqe = io_uring_get_sqe(&d->ring);
        if (!sqe)
            die("dev%u: arm ring full", d->id);
        prep_fetch(sqe, d, tag, UD_FETCH, 0);
    }

    if (io_uring_submit(&d->ring) < 0)
        die("dev%u: arm submit", d->id);
    atomic_store(&d->armed, 1);

    unsigned long long completions = 0;

    // Serve until the device goes away (fetch aborts with -ENODEV).
    while (!atomic_load(&g_stop)) {
        unsigned count = 0;
        struct io_uring_cqe *cqe;
        unsigned head;

        io_uring_for_each_cqe(&d->ring, head, cqe) {
            unsigned tag = cqe->user_data & 0xffff;
            unsigned kind = (cqe->user_data >> 16) & 0xff;
            if (tag >= depth || (kind != UD_FETCH && kind != UD_COMMIT))
                die("dev%u: bad cqe ud=%#llx", d->id,
                    (unsigned long long)cqe->user_data);
            completions++;

            if (cqe->res < 0) {
                if (cqe->res == UBLK_IO_RES_ABORT)
                    goto stopped;
                else if (d->quiet) {
                    fprintf(stderr,
                        "dev%u: cqe kind %u res %d (ud %#llx), queue exiting\n",
                        d->id, kind, cqe->res,
                        (unsigned long long)cqe->user_data);
                    goto stopped;
                }

                errno = -cqe->res;
                die("dev%u: fetch cqe tag %u", d->id, tag);
            }

            /*
             * New request for this tag: read the descriptor
             * (kernel wrote it before completing the fetch),
             * run the backend IO, commit and re-fetch.
             */
            struct ublksrv_io_desc volatile *vd = &d->descs[tag];
            uint32_t op = vd->op_flags & 0xff;
            uint32_t len = vd->nr_sectors * 512;
            uint64_t off = vd->start_sector * 512;
            int res = backend_io(d, op, off, len,
                         d->bufs + (size_t)tag * MAX_IO_BYTES);
            struct io_uring_sqe *sqe = io_uring_get_sqe(&d->ring);
            if (!sqe)
                die("dev%u: serve ring full", d->id);
            prep_fetch(sqe, d, tag, UD_COMMIT, res);
            if (++count == depth)
                break;
        }

        io_uring_cq_advance(&d->ring, count);
        if (count) {
            if (io_uring_submit(&d->ring) < 0)
                die("dev%u: commit submit", d->id);
            continue;
        }

        /*
         * Bounded blocking wait. If we timed out, report how many CQEs
         * the kernel owes us. The bug shows up as an armed fetch whose
         * completion never appears and leaves it at 0 forever.
         */

        struct __kernel_timespec ts = { .tv_sec = WAIT_TIMEOUT_SEC, .tv_nsec = 0 };
        int ret = io_uring_submit_and_wait_timeout(&d->ring, &cqe, 1, &ts, 0);
        if (ret == -ETIME)
            fprintf(stderr,
                "dev%u wait timeout: pending CQEs %u, completions so far %llu\n",
                d->id, io_uring_cq_ready(&d->ring), completions);
        else if (ret < 0 && ret != -EINTR) {
            errno = -ret;
            die("dev%u: wait", d->id);
        }
    }

stopped:
    /*
     * Release everything the queue owns; the ring pins the single-open
     * chardev and the descriptor mapping, and a leaked one hangs the
     * next lifecycle cycle on the same id.
     */
    io_uring_queue_exit(&d->ring);
    close(d->cdev_fd);
    munmap(d->descs, map_len);
    return NULL;
}

static void drop_caches(void)
{
    int fd = open("/proc/sys/vm/drop_caches", O_WRONLY);
    if (fd < 0) {
        fputs("warning: could not open file to drop caches", stderr);
        return;
    } else if (write(fd, "3\n", 2) != 2) {
        fputs("warning: failed to instruct the kernel to drop caches", stderr);
    }

    close(fd);
}

/*
 * ADD_DEV can race a lingering device from a previous run or a
 * killed run. As such, we delete and poll time bound.
 */
static void add_dev_retry(unsigned id, unsigned depth)
{
    int r = ctrl_add_dev(id, depth);
    for (int attempt = 0; r == -EEXIST && attempt < 10; attempt++) {
        fprintf(stderr, "id %u still exists (EEXIST), deleting\n", id);
        ctrl_del_dev(id);
        sleep(1);
        r = ctrl_add_dev(id, depth);
    }

    if (r == -EEXIST) {
        printf("ADD_DEV %u still EEXIST after retries - device stuck "
               "in half-deleted limbo\n", id);
        report_hang(id);
    }

    ck(r, "ADD_DEV");
}

static void *load_fn(void *arg)
{
    struct { struct dev *d; unsigned shard; } *w = arg;
    struct dev *d = w->d;
    const size_t chunk = 128 * 1024;
    unsigned char *io = aligned_alloc(4096, chunk);
    unsigned long long salt = w->shard;
    int ran = 0;
    // this shard's slice of the volume: concurrent in-flight IO per device
    const uint64_t lo = VOL_BYTES / LOAD_THREADS * w->shard;
    const uint64_t hi = VOL_BYTES / LOAD_THREADS * (w->shard + 1);

    if (!io)
        die("load buf");

    int fd = open(d->node, O_RDWR);
    if (fd < 0)
        die("open %s", d->node);

    fprintf(stderr, "dev%u load[%u]: opened\n", d->id, w->shard);

    while (!atomic_load(&g_stop)) {
        if (!atomic_load(&g_active)) {
            if (ran)
                break;
            struct timespec ms = { 0, 50 * 1000 * 1000 };
            nanosleep(&ms, NULL);
            continue;
        }

        // Try to make sure to read from the device without the page cache.
        pthread_mutex_lock(&drop_lock);
        drop_caches();
        pthread_mutex_unlock(&drop_lock);

        for (uint64_t off = lo; off < hi; off += chunk) {
            ssize_t n;
            do {
                n = pread(fd, io, chunk, (off_t)off);
            } while (n < 0 && errno == EINTR);
            if (n != (ssize_t)chunk) {
                fprintf(stderr, "dev%u: read at %#llx: %zd (%s)\n",
                        d->id, (unsigned long long)off, n,
                        strerror(errno));
                goto out;
            }
        }

        atomic_fetch_add(&d->passes, 1);
        ran = 1;
        if (atomic_load(&g_stop) || !atomic_load(&g_active))
            continue;

        // Same as above for writes.
        for (uint64_t off = lo; off < hi; off += chunk) {
            memset(io, (int)(++salt + d->id), chunk);
            ssize_t n;
            do {
                n = pwrite(fd, io, chunk, (off_t)off);
            } while (n < 0 && errno == EINTR);
            if (n != (ssize_t)chunk) {
                fprintf(stderr, "dev%u: write at %#llx: %zd\n",
                        d->id, (unsigned long long)off, n);
                goto out;
            }
        }

        if (fsync(fd))
            die("dev%u: fsync", d->id);

        atomic_fetch_add(&d->passes, 1);
        ran = 1;
    }

out:
    close(fd);
    free(io);
    return NULL;
}

/*
 * The basis of a failing workload: A new device is being added, started,
 * loaded (briefly), stopped and deleted over and over while the main
 * devices are under load.
 */

#define CHURN_ID   (BASE_ID + NR_DEVS)

static void *churn_fn(void *_arg)
{
    (void)_arg;
    struct dev d;
    memset(&d, 0, sizeof(d));
    d.id = CHURN_ID;
    d.depth = DEPTH;
    d.ram_backend = 1;
    d.quiet = 1;
    d.ram_buf = aligned_alloc(4096, VOL_BYTES);
    if (!d.ram_buf)
        die("churn ram buf");

    memset(d.ram_buf, 0xa5, VOL_BYTES);
    snprintf(d.node, sizeof(d.node), "/dev/ublkb%u", d.id);
    snprintf(d.cdev_path, sizeof(d.cdev_path), "/dev/ublkc%u", d.id);
    atomic_init(&d.armed, 0);
    atomic_init(&d.passes, 0);

    unsigned long long cycles = 0;
    while (!atomic_load(&g_stop)) {
        if (!atomic_load(&g_active)) {
            struct timespec ms = { 0, 100 * 1000 * 1000 };
            nanosleep(&ms, NULL);
            continue;
        }
        int have_server = 0;
        pthread_t server;
        add_dev_retry(d.id, (unsigned)d.depth);
        if (ctrl_set_params(d.id, VOL_BYTES) < 0)
            goto teardown;
        atomic_store(&d.armed, 0);
        if (pthread_create(&server, NULL, server_fn, &d))
            die("churn server thread");
        have_server = 1;
        for (int t = 0; t < 100 && !atomic_load(&d.armed); t++) {
            struct timespec ms = { 0, 20 * 1000 * 1000 };
            nanosleep(&ms, NULL);
        }

        /*
         * The queue may have died with the device (delete race);
         * without arming there is nothing to start - and START_DEV
         * would wait forever for a queue that will never arm.
         */
        if (!atomic_load(&d.armed))
            goto teardown;

        int r = ctrl_start_dev(d.id);
        if (r != 0)
            goto teardown;
        {
            d.live = 1;
            int fd = open(d.node, O_RDWR);
            if (fd < 0) {
                // device raced away; stop and retry the cycle
                d.live = 0;
                goto teardown;
            }
            // a burst of buffered reads and writes
            unsigned char *io = aligned_alloc(4096, 256 * 1024);
            for (int pass = 0; pass < 3; pass++) {
                for (uint64_t off = 0; off < VOL_BYTES;
                     off += 256 * 1024) {
                    ssize_t n = pread(fd, io, 256 * 1024,
                              (off_t)off);
                    if (n < 0 && errno != EINTR)
                        break;
                }
                for (uint64_t off = 0; off < VOL_BYTES;
                     off += 256 * 1024) {
                    ssize_t n = pwrite(fd, io, 256 * 1024,
                               (off_t)off);
                    if (n < 0 && errno != EINTR)
                        break;
                }
                fsync(fd);
            }
            free(io);
            close(fd);
        }

teardown:
        /*
         * STOP_DEV aborts the queue's fetches; the queue thread
         * sees -ENODEV and exits.
         */
        int sr = ctrl_stop_dev(d.id);
        if (sr == -ENOENT)
            sr = 0;
        if (have_server)
            pthread_join(server, NULL);
        d.live = 0;

        if (sr == -ETIMEDOUT) {
            printf("churn STOP_DEV %u timed out - kernel-side "
                   "stop hanged\n", d.id);
            report_hang(d.id);
        }

        int dr = ctrl_del_dev(d.id);
        if (dr == -ETIMEDOUT) {
            printf("churn DEL_DEV %u timed out - kernel-side "
                   "delete hanged\n", d.id);
            report_hang(d.id);
        }

        cycles++;
        if (cycles % 5 == 0) {
            printf("churn: %llu cycles\n", cycles);
            fflush(stdout);
        }
    }

    free(d.ram_buf);
    return NULL;
}

static void print_thread_states(void)
{
    char path[128], wchan[64], comm[64];
    DIR *dir = opendir("/proc/self/task");
    struct dirent *de;

    if (!dir)
        return;

    printf("  threads:\n");
    int all_threads = getenv("REPRO_ALL_THREADS") != 0;

    while ((de = readdir(dir)) != NULL) {
        long tid = atol(de->d_name);
        int fd;
        ssize_t n;

        if (tid <= 0)
            continue;

        snprintf(path, sizeof(path), "/proc/self/task/%ld/wchan", tid);
        fd = open(path, O_RDONLY);
        if (fd < 0)
            continue;

        n = read(fd, wchan, sizeof(wchan) - 1);
        close(fd);
        if ((size_t)n >= 64)
            continue;

        wchan[n] = 0;
        if (all_threads ||
            strstr(wchan, "cqring") ||
            strstr(wchan, "ublk") ||
            strstr(wchan, "freeze") ||
            strstr(wchan, "mutex")) {

            strcpy(comm, "?");
            snprintf(path, sizeof(path), "/proc/self/task/%ld/comm", tid);
            fd = open(path, O_RDONLY);
            if (fd >= 0) {
                n = read(fd, comm, sizeof(comm) - 1);
                close(fd);
                if (n > 0)
                    comm[n] = 0;
            }

            printf("    tid %ld comm %.20s wchan %.40s\n", tid, comm, wchan);
        }
    }

    closedir(dir);
}

static void report_hang(int worst_dev)
{
    printf("\n=== HUNG: dev%u made no progress for %d s ===\n",
           worst_dev, STALL_SECS);
    for (int i = 0; i < NR_DEVS; i++) {
        char path[96], buf[64];
        snprintf(path, sizeof(path), "/sys/block/ublkb%u/inflight",
             devs[i].id);
        int fd = open(path, O_RDONLY);
        ssize_t n = fd >= 0 ? read(fd, buf, sizeof(buf) - 1) : -1;
        if (fd >= 0)
            close(fd);
        if (n > 0) {
            buf[n] = 0;
            printf("  /sys/block/ublkb%u/inflight: %s", devs[i].id, buf);
        } else {
            printf("  /sys/block/ublkb%u/inflight: <unreadable>\n", devs[i].id);
        }
    }

    print_thread_states();
    printf("  fingerprint expected: inflight \"1 0\" frozen, its\n"
           "  queue thread blocked in io_cqring_wait having reported\n"
           "  \"wait timeout: pending CQEs 0\" above, every 5 s.\n"
           "  dmesg will show the del_gendisk freeze / control-mutex\n"
           "  cascade if teardown is attempted; reboot to clear.\n");
    fflush(stdout);
    struct timespec s = { 5, 0 };
    nanosleep(&s, NULL);
    exit(2);
}

static int dev_inflight(unsigned id)
{
    char path[64], buf[64];

    snprintf(path, sizeof(path), "/sys/block/ublkb%u/inflight", id);
    int fd = open(path, O_RDONLY);
    if (fd < 0)
        return -1; // no device (before add / after del)

    ssize_t n = read(fd, buf, sizeof(buf) - 1);
    close(fd);
    if (n <= 0)
        return -1;

    buf[n] = 0;
    int r = 0, w = 0;
    if (sscanf(buf, "%d %d", &r, &w) != 2)
        return -1;

    return r + w;
}

static void *watchdog_fn(void *_arg)
{
    (void)_arg;
    unsigned long long last[NR_DEVS] = { 0 };
    int stall[NR_DEVS] = { 0 };
    int idle_stall[NR_DEVS] = { 0 };

    while (!atomic_load(&g_stop)) {
        sleep(1);
        int active = atomic_load(&g_active);
        int worst = -1, worst_s = 0;
        for (int i = 0; i < NR_DEVS; i++) {
            if (active) {
                // load running: passes must keep advancing
                unsigned long long cur =
                    atomic_load(&devs[i].passes);
                if (cur != last[i]) {
                    last[i] = cur;
                    stall[i] = 0;
                } else if (++stall[i] > worst_s) {
                    worst_s = stall[i];
                    worst = i;
                }
                continue;
            }
            if (!devs[i].live)
                continue;

            /*
             * If we're idle, then there must be zero in flight requests.
             * Otherwise, it's likely that the kernel dispatched a request
             * that never made it to us.
             */
            if (dev_inflight(devs[i].id) > 0) {
                if (++idle_stall[i] >= IDLE_SECS) {
                    printf("idle dev%u: inflight nonzero "
                           "for %ds with no load\n",
                           devs[i].id, idle_stall[i]);
                    report_hang(devs[i].id);
                }
            } else {
                idle_stall[i] = 0;
            }
        }

        if (worst >= 0 && worst_s >= STALL_SECS)
            report_hang(devs[worst].id);
    }

    return NULL;
}

// ----------------------------------------------------------------- setup

static void on_sigint(int _sig)
{
    (void)_sig;
    atomic_store(&g_stop, 1);
}

static void on_sigalrm(int _sig)
{
    (void)_sig;
    fprintf(stderr, "FATAL: teardown did not finish in 60 s - "
            "control plane hanged (expected after a strand)\n");
    _exit(3);
}

int main(int argc, char **argv)
{
    unsigned runs = argc > 1 ? (unsigned)atoi(argv[1]) : 0;

    if (geteuid() != 0) {
        fprintf(stderr, "must run as root\n");
        return 1;
    }
    signal(SIGINT, on_sigint);
    signal(SIGALRM, on_sigalrm);

    ctrl_fd = open("/dev/ublk-control", O_RDWR);
    if (ctrl_fd < 0)
        die("open /dev/ublk-control");
    struct io_uring_params p = { .flags = IORING_SETUP_SQE128 };
    if (io_uring_queue_init_params(8, &ctrl_ring, &p))
        die("ctrl ring");

    uint64_t features = 0;
    ck(ctrl_get_features(&features), "GET_FEATURES");
    printf("kernel ublk features: %#lx\n", (unsigned long)features);

    for (int i = 0; i < NR_DEVS; i++) {
        struct dev *d = &devs[i];
        d->id = BASE_ID + (unsigned)i;
        snprintf(d->node, sizeof(d->node), "/dev/ublkb%u", d->id);
        snprintf(d->cdev_path, sizeof(d->cdev_path), "/dev/ublkc%u", d->id);
        snprintf(d->backend, sizeof(d->backend), "/tmp/ublk-repro-b%d.img", d->id);
        d->depth = DEPTH;
        d->ram_backend = (i == RAMDEV_IDX);
        if (d->ram_backend) {
            d->ram_buf = aligned_alloc(4096, VOL_BYTES);
            if (!d->ram_buf)
                die("ram backend");
            memset(d->ram_buf, 0x5a, VOL_BYTES);
        } else {
            int fd = open(d->backend, O_RDWR | O_CREAT | O_TRUNC, 0600);
            if (fd < 0)
                die("create %s", d->backend);
            if (ftruncate(fd, (off_t)VOL_BYTES))
                die("truncate %s", d->backend);
            close(fd);
            d->backend_fd = open(d->backend, O_RDWR);
            if (d->backend_fd < 0)
                die("open %s", d->backend);
        }
        d->bufs = aligned_alloc(4096, (size_t)d->depth * MAX_IO_BYTES);
        if (!d->bufs)
            die("tag buffers");
    }

    pthread_t wd;
    if (pthread_create(&wd, NULL, watchdog_fn, NULL))
        die("watchdog");
#ifndef NO_CHURN
    pthread_t churn;
    if (pthread_create(&churn, NULL, churn_fn, NULL))
        die("churn");
#endif

    for (unsigned run = 1; !atomic_load(&g_stop); run++) {
        if (runs && run > runs)
            break;
        atomic_store(&g_run, run);
        printf("=== run %u ===\n", run);
        fflush(stdout);

        pthread_t servers[NR_DEVS];
        pthread_t loads[NR_DEVS][LOAD_THREADS];
        struct { struct dev *d; unsigned shard; } work[NR_DEVS][LOAD_THREADS];

        for (int i = 0; i < NR_DEVS; i++) {
            struct dev *d = &devs[i];
            atomic_store(&d->armed, 0);
            add_dev_retry(d->id, (unsigned)d->depth);
            ck(ctrl_set_params(d->id, VOL_BYTES), "SET_PARAMS");
            if (pthread_create(&servers[i], NULL, server_fn, d))
                die("server thread");
        }
        for (int i = 0; i < NR_DEVS; i++) {
            struct dev *d = &devs[i];
            for (int t = 0; t < 100 && !atomic_load(&d->armed); t++) {
                struct timespec ms = { 0, 100 * 1000 * 1000 };
                nanosleep(&ms, NULL);
            }
            if (!atomic_load(&d->armed))
                die("dev%u never armed", d->id);
            ck(ctrl_start_dev(d->id), "START_DEV");
        }
        for (int i = 0; i < NR_DEVS; i++)
            for (unsigned k = 0; k < LOAD_THREADS; k++) {
                work[i][k].d = &devs[i];
                work[i][k].shard = k;
                if (pthread_create(&loads[i][k], NULL, load_fn,
                           &work[i][k]))
                    die("load thread");
            }
        for (int i = 0; i < NR_DEVS; i++)
            devs[i].live = 1;

        printf("device(s) live, running load for %d s\n", CAMPAIGN_SECS);
        fflush(stdout);
        atomic_store(&g_active, 1);
        double t0 = now_s();
        while (!atomic_load(&g_stop) && now_s() - t0 < CAMPAIGN_SECS) {
            struct timespec s = { 1, 0 };
            nanosleep(&s, NULL);
            printf("  t=%3.0fs passes:", now_s() - t0);
            for (int i = 0; i < NR_DEVS; i++)
                printf(" %llu", (unsigned long long)
                       atomic_load(&devs[i].passes));
            printf("\n");
            fflush(stdout);
        }
        atomic_store(&g_active, 0);
        if (atomic_load(&g_stop))
            printf("interrupted, tearing down\n");

        for (int i = 0; i < NR_DEVS; i++)
            for (unsigned k = 0; k < LOAD_THREADS; k++)
                pthread_join(loads[i][k], NULL);
        for (int i = 0; i < NR_DEVS; i++) {
            struct dev *d = &devs[i];
            int r = ctrl_stop_dev(d->id);
            if (r == -ETIMEDOUT) {
                printf("STOP_DEV %u timed out - stranded "
                       "request hanged teardown\n", d->id);
                report_hang(d->id);
            }
            if (r == -ENOENT)
                r = 0; // already gone
            if (r)
                fprintf(stderr, "STOP_DEV %u: %s (-%u)\n",
                    d->id, strerror(-r), (unsigned)-r);
            pthread_join(servers[i], NULL);
            int dr = ctrl_del_dev(d->id);
            if (dr == -EINTR)
                dr = ctrl_del_dev(d->id);
            if (dr == -ETIMEDOUT) {
                printf("DEL_DEV %u timed out - kernel-side "
                       "delete hangedd\n", d->id);
                report_hang(d->id);
            }
            if (dr && dr != -ENOENT)
                fprintf(stderr, "DEL_DEV %u: %s (-%u)\n",
                    d->id, strerror(-dr), (unsigned)-dr);
            d->live = 0;
        }
        for (int i = 0; i < NR_DEVS; i++) {
            char path[64];
            struct stat st;
            snprintf(path, sizeof(path), "/sys/block/ublkb%u",
                 devs[i].id);
            if (!stat(path, &st)) {
                printf("LEAK: ublkb%u still present after "
                       "teardown\n", devs[i].id);
                fflush(stdout);
            }
        }
        printf("runn %u clean (%llu/%llu/%llu passes)\n", run,
               (unsigned long long)atomic_load(&devs[0].passes),
               (unsigned long long)atomic_load(&devs[1].passes),
               (unsigned long long)atomic_load(&devs[2].passes));
        fflush(stdout);
    }

    printf("done after %u runs, no hangs observed\n",
           runs ? runs : atomic_load(&g_run));
    atomic_store(&g_stop, 1);
#ifndef NO_CHURN
    pthread_join(churn, NULL);
#endif
    return 0;
}

^ permalink raw reply	[flat|nested] 7+ messages in thread

end of thread, other threads:[~2026-11-21 20:59 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-20 22:55 [REGRESSION] ublk: dispatched IO intermittently never posts its fetch completion (v7.0) Luciano Lo Giudice
2026-11-20 23:27 ` Bernd Schubert
2026-11-20 23:40   ` Luciano Lo Giudice
2026-11-21  2:25     ` Luciano Lo Giudice
2026-11-21  5:23       ` Ming Lei
2026-11-21 17:52         ` Luciano Lo Giudice
2026-11-21 20:59           ` Jens Axboe

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox