Re: [PATCH/RFC] btrfs: fix folio lock leak in writepage_delalloc() for folios dirtied behind btrfs' back

[Christian Borntraeger <borntraeger@linux.ibm.com>]

[-- Attachment #1: Type: text/plain, Size: 10562 bytes --]

Am 22.07.26 um 10:59 schrieb Qu Wenruo:
> 
> 
> 在 2026/7/22 18:05, Christian Borntraeger 写道:
>> Am 21.07.26 um 23:07 schrieb Qu Wenruo:
>>
>> First, thank you for taking the time to look into this and trying to understand
>> things and trying to explain things. this is highly appreciated.
>> I am still trying to fully understand this myself. A question:
>>
>>> 在 2026/7/22 04:41, Christian Borntraeger 写道:
>>>> A folio can carry the folio-level dirty flag while its btrfs subpage
>>>> dirty bitmap is empty: btrfs data mappings use filemap_dirty_folio(),
>>>> so a generic folio_mark_dirty() call sets only the folio flag and the
>>>> xarray tag, without setting any subpage dirty bit and without a
>>>> delalloc reservation.  The typical source is set_page_dirty_lock() on
>>>> a GUP pin,
>>>
>>> Shouldn't such folio got its ->page_mkwrite() callback get called first?
>>
>> Isnt that called implicitely at pin time?
> 
> I have to admit, I'm not an expert on the MM part, I'm mostly a simple user of the existing MM interfaces.
> 
> AFAIK, the last time I brought this thing up, Christoph mentioned that dirtying-folio-without-notifying-fs is a bug, and fs should not and is not able to handle such situation anyway.
> 
> And that idea makes a lot of sense to me.
> 
> So adding MM list for more help.
So I now have an userspace O_DIRECT reproducer outside of KVM. (attached) which gave me (on an s390 system, though)

[  189.067731] sysrq: Show Blocked State
[  189.067872] task:kworker/u1665:5 state:D stack:0     pid:1363  tgid:1363  ppid:2      task_flags:0x4208060 flags:0x00000000
[  189.067877] Workqueue: writeback wb_workfn (flush-btrfs-1)
[  189.067884] Call Trace:
[  189.067886]  [<000003a8b2e19284>] __schedule+0x374/0x900
[  189.067891]  [<000003a8b2e1984c>] schedule+0x3c/0xf0
[  189.067894]  [<000003a8b2e1996c>] io_schedule+0x2c/0x40
[  189.067896]  [<000003a8b219d158>] folio_wait_bit_common+0x198/0x3b0
[  189.067902]  [<000003a8b263e2e8>] extent_write_cache_pages+0x348/0x4d0
[  189.067907]  [<000003a8b263ea0a>] btrfs_writepages+0x6a/0xd0
[  189.067910]  [<000003a8b21ad444>] do_writepages+0xd4/0x190
[  189.067915]  [<000003a8b2324590>] __writeback_single_inode+0x50/0x310
[  189.067917]  [<000003a8b2324b1a>] writeback_sb_inodes+0x2ca/0x690
[  189.067919]  [<000003a8b2324f36>] __writeback_inodes_wb+0x56/0x140
[  189.067921]  [<000003a8b2325498>] wb_writeback+0x368/0x460
[  189.067923]  [<000003a8b23258c6>] wb_do_writeback+0x336/0x3d0
[  189.067925]  [<000003a8b2325d2a>] wb_workfn+0x5a/0x1d0
[  189.067927]  [<000003a8b1f0cee2>] process_one_work+0x1d2/0x480
[  189.067932]  [<000003a8b1f0df30>] worker_thread+0x210/0x420
[  189.067935]  [<000003a8b1f190b8>] kthread+0x148/0x170
[  189.067939]  [<000003a8b1e90358>] __ret_from_fork+0x48/0x220
[  189.067941]  [<000003a8b2e2426a>] ret_from_fork+0xa/0x30
[  189.067994] task:oob-dirty-repro state:D stack:0     pid:8152  tgid:8151  ppid:7260   task_flags:0x400040 flags:0x00000800
[  189.067997] Call Trace:
[  189.067998]  [<000003a8b2e19284>] __schedule+0x374/0x900
[  189.068050]  [<000003a8b2e1984c>] schedule+0x3c/0xf0
[  189.068052]  [<000003a8b2e1996c>] io_schedule+0x2c/0x40
[  189.068054]  [<000003a8b219d158>] folio_wait_bit_common+0x198/0x3b0
[  189.068057]  [<000003a8b263e2e8>] extent_write_cache_pages+0x348/0x4d0
[  189.068061]  [<000003a8b263ea0a>] btrfs_writepages+0x6a/0xd0
[  189.068063]  [<000003a8b21ad444>] do_writepages+0xd4/0x190
[  189.068066]  [<000003a8b2199dec>] filemap_writeback+0xbc/0x100
[  189.068069]  [<000003a8b2199e5a>] filemap_fdatawrite_range+0x2a/0x40
[  189.068072]  [<000003a8b2637d3c>] btrfs_start_ordered_extent_nowriteback+0x18c/0x1d0
[  189.068074]  [<000003a8b26260ce>] btrfs_page_mkwrite+0x22e/0x8d0
[  189.068079]  [<000003a8b21f89d0>] do_page_mkwrite+0x60/0xf0
[  189.068084]  [<000003a8b21ffa64>] do_wp_page+0x134/0x660
[  189.068086]  [<000003a8b2206584>] __handle_mm_fault+0x1c4/0x5a0
[  189.068088]  [<000003a8b22069f0>] handle_mm_fault+0x90/0x230
[  189.068090]  [<000003a8b1ebfbd0>] do_exception+0x190/0x560
[  189.068094]  [<000003a8b2e136c0>] __do_pgm_check+0x1b0/0x370
[  189.068098]  [<000003a8b2e243a4>] pgm_check_handler+0x114/0x160
[  189.068101]  [<000003a8b283135e>] _copy_to_iter+0x6e/0x870
[  189.068106]  [<000003a8b2831c4a>] copy_page_to_iter+0xea/0x160
[  189.068108]  [<000003a8b219ff8c>] filemap_read+0x1ec/0x400
[  189.068109]  [<000003a8b22d0a70>] vfs_read+0x210/0x370
[  189.068112]  [<000003a8b22d1694>] ksys_pread64+0xa4/0xd0
[  189.068114]  [<000003a8b2e13b7e>] __do_syscall+0x14e/0x590
[  189.068117]  [<000003a8b2e24242>] system_call+0x72/0x90
[  189.068120] task:oob-dirty-repro state:D stack:0     pid:8153  tgid:8151  ppid:7260   task_flags:0x400040 flags:0x00000000
[  189.068123] Call Trace:
[  189.068124]  [<000003a8b2e19284>] __schedule+0x374/0x900
[  189.068126]  [<000003a8b2e1984c>] schedule+0x3c/0xf0
[  189.068128]  [<000003a8b2e1996c>] io_schedule+0x2c/0x40
[  189.068130]  [<000003a8b219d158>] folio_wait_bit_common+0x198/0x3b0
[  189.068133]  [<000003a8b263e2e8>] extent_write_cache_pages+0x348/0x4d0
[  189.068136]  [<000003a8b263ea0a>] btrfs_writepages+0x6a/0xd0
[  189.068138]  [<000003a8b21ad444>] do_writepages+0xd4/0x190
[  189.068141]  [<000003a8b2199dec>] filemap_writeback+0xbc/0x100
[  189.068144]  [<000003a8b2199e5a>] filemap_fdatawrite_range+0x2a/0x40
[  189.068146]  [<000003a8b2637d3c>] btrfs_start_ordered_extent_nowriteback+0x18c/0x1d0
[  189.068149]  [<000003a8b26260ce>] btrfs_page_mkwrite+0x22e/0x8d0
[  189.068152]  [<000003a8b21f89d0>] do_page_mkwrite+0x60/0xf0
[  189.068154]  [<000003a8b21ffa64>] do_wp_page+0x134/0x660
[  189.068156]  [<000003a8b2206584>] __handle_mm_fault+0x1c4/0x5a0
[  189.068158]  [<000003a8b22069f0>] handle_mm_fault+0x90/0x230
[  189.068160]  [<000003a8b1ebfbd0>] do_exception+0x190/0x560
[  189.068163]  [<000003a8b2e136c0>] __do_pgm_check+0x1b0/0x370
[  189.068165]  [<000003a8b2e243a4>] pgm_check_handler+0x114/0x160
[  189.068168]  [<000003a8b283135e>] _copy_to_iter+0x6e/0x870
[  189.068170]  [<000003a8b2831c4a>] copy_page_to_iter+0xea/0x160
[  189.068172]  [<000003a8b219ff8c>] filemap_read+0x1ec/0x400
[  189.068174]  [<000003a8b22d0a70>] vfs_read+0x210/0x370
[  189.068176]  [<000003a8b22d1694>] ksys_pread64+0xa4/0xd0
[  189.068178]  [<000003a8b2e13b7e>] __do_syscall+0x14e/0x590
[  189.068181]  [<000003a8b2e24242>] system_call+0x72/0x90
[  189.068184] task:oob-dirty-repro state:D stack:0     pid:8154  tgid:8151  ppid:7260   task_flags:0x400040 flags:0x00000000
[  189.068187] Call Trace:
[  189.068188]  [<000003a8b2e19284>] __schedule+0x374/0x900
[  189.068190]  [<000003a8b2e1984c>] schedule+0x3c/0xf0
[  189.068192]  [<000003a8b2e1996c>] io_schedule+0x2c/0x40
[  189.068193]  [<000003a8b219d158>] folio_wait_bit_common+0x198/0x3b0
[  189.068196]  [<000003a8b26260fc>] btrfs_page_mkwrite+0x25c/0x8d0
[  189.068199]  [<000003a8b21f89d0>] do_page_mkwrite+0x60/0xf0
[  189.068202]  [<000003a8b21ffa64>] do_wp_page+0x134/0x660
[  189.068203]  [<000003a8b2206584>] __handle_mm_fault+0x1c4/0x5a0
[  189.068206]  [<000003a8b22069f0>] handle_mm_fault+0x90/0x230
[  189.068208]  [<000003a8b1ebfbd0>] do_exception+0x190/0x560
[  189.068210]  [<000003a8b2e136c0>] __do_pgm_check+0x1b0/0x370
[  189.068213]  [<000003a8b2e243a4>] pgm_check_handler+0x114/0x160
[  189.068215]  [<000003a8b283135e>] _copy_to_iter+0x6e/0x870
[  189.068218]  [<000003a8b2831c4a>] copy_page_to_iter+0xea/0x160
[  189.068219]  [<000003a8b219ff8c>] filemap_read+0x1ec/0x400
[  189.068221]  [<000003a8b22d0a70>] vfs_read+0x210/0x370
[  189.068223]  [<000003a8b22d1694>] ksys_pread64+0xa4/0xd0
[  189.068225]  [<000003a8b2e13b7e>] __do_syscall+0x14e/0x590
[  189.068228]  [<000003a8b2e24242>] system_call+0x72/0x90
[  189.068231] task:oob-dirty-repro state:D stack:0     pid:8155  tgid:8151  ppid:7260   task_flags:0x400040 flags:0x00000000
[  189.068234] Call Trace:
[  189.068235]  [<000003a8b2e19284>] __schedule+0x374/0x900
[  189.068237]  [<000003a8b2e1984c>] schedule+0x3c/0xf0
[  189.068239]  [<000003a8b2e1996c>] io_schedule+0x2c/0x40
[  189.068241]  [<000003a8b219d158>] folio_wait_bit_common+0x198/0x3b0
[  189.068244]  [<000003a8b263e2e8>] extent_write_cache_pages+0x348/0x4d0
[  189.068246]  [<000003a8b263ea0a>] btrfs_writepages+0x6a/0xd0
[  189.068249]  [<000003a8b21ad444>] do_writepages+0xd4/0x190
[  189.068252]  [<000003a8b2199dec>] filemap_writeback+0xbc/0x100
[  189.068255]  [<000003a8b2199e5a>] filemap_fdatawrite_range+0x2a/0x40
[  189.068257]  [<000003a8b2637d3c>] btrfs_start_ordered_extent_nowriteback+0x18c/0x1d0
[  189.068260]  [<000003a8b26260ce>] btrfs_page_mkwrite+0x22e/0x8d0
[  189.068263]  [<000003a8b21f89d0>] do_page_mkwrite+0x60/0xf0
[  189.068265]  [<000003a8b21ffa64>] do_wp_page+0x134/0x660
[  189.068267]  [<000003a8b2206584>] __handle_mm_fault+0x1c4/0x5a0
[  189.068269]  [<000003a8b22069f0>] handle_mm_fault+0x90/0x230
[  189.068271]  [<000003a8b1ebfbd0>] do_exception+0x190/0x560
[  189.068274]  [<000003a8b2e136c0>] __do_pgm_check+0x1b0/0x370
[  189.068277]  [<000003a8b2e243a4>] pgm_check_handler+0x114/0x160
[  189.068279]  [<000003a8b283135e>] _copy_to_iter+0x6e/0x870
[  189.068281]  [<000003a8b2831c4a>] copy_page_to_iter+0xea/0x160
[  189.068283]  [<000003a8b219ff8c>] filemap_read+0x1ec/0x400
[  189.068285]  [<000003a8b22d0a70>] vfs_read+0x210/0x370
[  189.068287]  [<000003a8b22d1694>] ksys_pread64+0xa4/0xd0
[  189.068289]  [<000003a8b2e13b7e>] __do_syscall+0x14e/0x590
[  189.068292]  [<000003a8b2e24242>] system_call+0x72/0x90
[  189.068294] task:oob-dirty-repro state:D stack:0     pid:8156  tgid:8151  ppid:7260   task_flags:0x400040 flags:0x00000000
[  189.068297] Call Trace:
[  189.068298]  [<000003a8b2e19284>] __schedule+0x374/0x900
[  189.068300]  [<000003a8b2e1984c>] schedule+0x3c/0xf0
[  189.068302]  [<000003a8b2e1996c>] io_schedule+0x2c/0x40
[  189.068304]  [<000003a8b219d158>] folio_wait_bit_common+0x198/0x3b0
[  189.068307]  [<000003a8b263e2e8>] extent_write_cache_pages+0x348/0x4d0
[  189.068310]  [<000003a8b263ea0a>] btrfs_writepages+0x6a/0xd0
[  189.068313]  [<000003a8b21ad444>] do_writepages+0xd4/0x190
[  189.068315]  [<000003a8b2199dec>] filemap_writeback+0xbc/0x100
[  189.068318]  [<000003a8b2199e5a>] filemap_fdatawrite_range+0x2a/0x40
[  189.068320]  [<000003a8b232bbf2>] sync_file_range+0x122/0x150
[  189.068323]  [<000003a8b232bc84>] ksys_sync_file_range+0x64/0xb0
[  189.068326]  [<000003a8b232bd0a>] __s390x_sys_sync_file_range+0x3a/0x50
[  189.068328]  [<000003a8b2e13b7e>] __do_syscall+0x14e/0x590
[  189.068331]  [<000003a8b2e24242>] system_call+0x72/0x90

[-- Attachment #2: oob-dirty-repro.c --]
[-- Type: text/x-csrc, Size: 8558 bytes --]

// SPDX-License-Identifier: GPL-2.0
/*
 * oob-dirty-repro.c - reproduce out-of-band dirty folios on btrfs
 *                     (folio lock leak in writepage_delalloc())
 *
 * Demonstrates that a "folio-level dirty flag set, fs never notified"
 * state can be produced on any architecture with sanctioned, in-tree
 * APIs - no s390, no KVM required:
 *
 *   1. A file on btrfs is written with large buffered writes so the
 *      page cache backs it with large data folios (v7.2-rc1+).
 *   2. The file is mmap'ed MAP_SHARED.
 *   3. Thread A loops O_DIRECT reads *into* that mapping.  The direct
 *      I/O read path pins the destination pages (GUP; the initial
 *      fault runs ->page_mkwrite() and reserves delalloc properly)
 *      and on bio completion re-dirties them via
 *      bio_check_pages_dirty() -> bio_release_pages(true)
 *      -> unpin_user_pages_dirty_lock() -> set_page_dirty_lock().
 *   4. Thread B loops sync_file_range(WRITE|WAIT) on the target file.
 *      Whenever a full clean cycle (clear_page_dirty_for_io(),
 *      writeback, bits cleared) completes inside thread A's
 *      submission->completion window, the completion-time
 *      set_page_dirty_lock() hits a *clean* folio: filemap_dirty_folio()
 *      sets only the folio flag and the xarray tag - no btrfs subpage
 *      dirty bit, no delalloc reservation.  See the 20-year-old comment
 *      above bio_set_pages_dirty() in block/bio.c describing exactly
 *      this ("other code (eg, flusher threads) could clean the pages").
 *   5. The next writeback of that folio enters writepage_delalloc()
 *      with an empty subpage dirty bitmap, takes the
 *      bitmap_empty() -> return 1 path and leaks the folio lock.
 *      Every subsequent locker (mkwrite faults, sync, flusher) then
 *      hangs in D state.
 *
 * Thread C stores into the mapping (mkwrite traffic, like KVM vCPUs)
 * and a watchdog reports when all threads stop making progress.
 *
 * Usage:  gcc -O2 -pthread -Wall -o oob-dirty-repro oob-dirty-repro.c
 *         ./oob-dirty-repro <dir-on-btrfs> [runtime-seconds]
 *
 * Expected on 7.2-rc (large data folios enabled): within minutes the
 * watchdog reports a stall; the sync/store/io threads are in D state,
 * "echo w > /proc/sysrq-trigger" shows them queued on a folio lock
 * (folio_wait_bit_common <- btrfs_page_mkwrite / extent_write_cache_pages).
 * On kernels without large data folios the leak path is not reachable
 * (single-block folios get bit 0 forced in btrfs_copy_subpage_dirty_bitmap());
 * the same race then produces the out-of-band state that the (removed)
 * COW fixup / "reject out-of-band dirty folios" code used to catch.
 *
 * WARNING: on affected kernels this wedges writeback of the target
 * filesystem (unkillable D-state tasks, syncfs hangs) until reboot.
 * Run it on a scratch machine.
 */
#define _GNU_SOURCE
#include <errno.h>
#include <fcntl.h>
#include <linux/fs.h>
#include <sys/ioctl.h>
#include <pthread.h>
#include <stdatomic.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/mman.h>
#include <sys/stat.h>
#include <sys/statfs.h>
#include <time.h>
#include <unistd.h>

#define BTRFS_SUPER_MAGIC 0x9123683E

#define TARGET_SZ	(16UL << 20)	/* mmap'ed target file          */
#define SRC_SZ		(4UL << 20)	/* O_DIRECT read source         */
#define IO_SZ		(2UL << 20)	/* per-read size (wide window)  */
#define WRITE_CHUNK	(1UL << 20)	/* buffered init writes (large folios) */
#define STALL_SECS	20
#define NR_IO_THREADS	4

static char *map;
static int target_fd, src_fd;
static atomic_ulong io_iters, sync_iters, store_iters;
static atomic_int stop;
static unsigned long runtime = 600;

static void die(const char *what)
{
	perror(what);
	exit(1);
}

static void *io_thread(void *arg __attribute__((unused)))
{
	/*
	 * Only the lower half: btrfs verifies read csums in the (pinned)
	 * destination pages after DMA, so the store thread must not poke
	 * the same pages, and the NOCOW source avoids csums altogether.
	 */
	unsigned long nranges = (TARGET_SZ / 2) / IO_SZ;

	while (!atomic_load(&stop)) {
		off_t off = (random() % nranges) * IO_SZ;
		ssize_t r = pread(src_fd, map + off, IO_SZ, 0);

		if (r < 0 && errno != EINTR && errno != EIO)
			die("O_DIRECT pread into mapping");
		atomic_fetch_add(&io_iters, 1);
	}
	return NULL;
}

static void *sync_thread(void *arg __attribute__((unused)))
{
	while (!atomic_load(&stop)) {
		if (sync_file_range(target_fd, 0, TARGET_SZ,
				    SYNC_FILE_RANGE_WAIT_BEFORE |
				    SYNC_FILE_RANGE_WRITE |
				    SYNC_FILE_RANGE_WAIT_AFTER) < 0)
			die("sync_file_range");
		atomic_fetch_add(&sync_iters, 1);
	}
	return NULL;
}

static void *store_thread(void *arg __attribute__((unused)))
{
	struct timespec ts = { 0, 1000000 };	/* 1ms */

	while (!atomic_load(&stop)) {
		unsigned long off = TARGET_SZ / 2 +
			(random() % (TARGET_SZ / 2 / 4096)) * 4096;

		map[off] ^= 1;			/* mkwrite fault traffic */
		atomic_fetch_add(&store_iters, 1);
		nanosleep(&ts, NULL);
	}
	return NULL;
}

int main(int argc, char **argv)
{
	char path[4096], *buf;
	struct statfs sfs;
	pthread_t io[NR_IO_THREADS], sy, st;
	unsigned long last_io = 0, last_sync = 0;
	unsigned long elapsed = 0, stalled = 0;

	setvbuf(stdout, NULL, _IOLBF, 0);
	if (argc < 2) {
		fprintf(stderr, "usage: %s <dir-on-btrfs> [runtime-secs]\n",
			argv[0]);
		return 1;
	}
	if (argc > 2)
		runtime = strtoul(argv[2], NULL, 0);

	if (statfs(argv[1], &sfs) < 0)
		die("statfs");
	if ((unsigned int)sfs.f_type != BTRFS_SUPER_MAGIC)
		fprintf(stderr, "warning: %s is not on btrfs, "
			"the writepage_delalloc() path will not be hit\n",
			argv[1]);

	/* Target file: large buffered writes -> large data folios. */
	snprintf(path, sizeof(path), "%s/oob-target.dat", argv[1]);
	target_fd = open(path, O_RDWR | O_CREAT | O_TRUNC, 0600);
	if (target_fd < 0)
		die("open target");
	buf = aligned_alloc(4096, WRITE_CHUNK);
	if (!buf)
		die("aligned_alloc");
	memset(buf, 0x5a, WRITE_CHUNK);
	for (unsigned long o = 0; o < TARGET_SZ; o += WRITE_CHUNK)
		if (pwrite(target_fd, buf, WRITE_CHUNK, o) != WRITE_CHUNK)
			die("pwrite target");
	if (fsync(target_fd) < 0)
		die("fsync target");

	/* Source file for the O_DIRECT reads. */
	snprintf(path, sizeof(path), "%s/oob-src.dat", argv[1]);
	src_fd = open(path, O_RDWR | O_CREAT | O_TRUNC | O_DIRECT, 0600);
	if (src_fd < 0)
		die("open source (O_DIRECT)");
	{	/* NOCOW/nodatasum: keep csum verification out of the picture */
		int attr = 0;

		ioctl(src_fd, FS_IOC_GETFLAGS, &attr);
		attr |= FS_NOCOW_FL;
		ioctl(src_fd, FS_IOC_SETFLAGS, &attr);
	}
	for (unsigned long o = 0; o < SRC_SZ; o += WRITE_CHUNK)
		if (pwrite(src_fd, buf, WRITE_CHUNK, o) != WRITE_CHUNK)
			die("pwrite source");
	if (fsync(src_fd) < 0)
		die("fsync source");
	free(buf);

	map = mmap(NULL, TARGET_SZ, PROT_READ | PROT_WRITE, MAP_SHARED,
		   target_fd, 0);
	if (map == MAP_FAILED)
		die("mmap");

	printf("running: O_DIRECT reads into MAP_SHARED btrfs mapping vs. "
	       "sync_file_range, %lus max\n", runtime);

	for (int n = 0; n < NR_IO_THREADS; n++)
		pthread_create(&io[n], NULL, io_thread, NULL);
	pthread_create(&sy, NULL, sync_thread, NULL);
	pthread_create(&st, NULL, store_thread, NULL);

	for (elapsed = 0; elapsed < runtime; elapsed += 5) {
		unsigned long i, s, w;

		sleep(5);
		i = atomic_load(&io_iters);
		s = atomic_load(&sync_iters);
		w = atomic_load(&store_iters);
		printf("  t=%4lus io=%lu sync=%lu store=%lu\n",
		       elapsed + 5, i, s, w);

		if (i == last_io && s == last_sync) {
			stalled += 5;
			if (stalled >= STALL_SECS) {
				printf("\n*** io+sync THREADS STALLED for %us - "
				       "folio lock leak reproduced? ***\n"
				       "Check:  grep -l . /proc/%d/task/*/stack "
				       "| xargs cat\n"
				       "        echo w > /proc/sysrq-trigger; "
				       "dmesg | tail -100\n"
				       "Expect: folio_wait_bit_common <- "
				       "btrfs_page_mkwrite /\n"
				       "        extent_write_cache_pages in "
				       "D-state threads.\n",
				       STALL_SECS, getpid());
				/* leave the hung process for inspection */
				pause();
			}
		} else {
			stalled = 0;
		}
		last_io = i;
		last_sync = s;

	}

	atomic_store(&stop, 1);
	for (int n = 0; n < NR_IO_THREADS; n++)
		pthread_join(io[n], NULL);
	pthread_join(sy, NULL);
	pthread_join(st, NULL);
	printf("finished without stall - not reproduced in %lus "
	       "(try a longer runtime; the race window is\n"
	       "submission->completion of the O_DIRECT read vs. one full "
	       "writeback cycle)\n", runtime);
	return 0;
}