Re: Really bad I/O lock contention between non-overlapping reader and writer

[Dave Chinner <dgc@kernel.org>]

On Mon, Oct 12, 2026 at 10:47:56AM -0700, Vitali Lovich wrote:
> I think I encountered suboptimal locking behavior with concurrent
> readers and writers on a file, which resulted in huge latency spikes
> for the reader even though the writers and readers are touching
> unrelated ranges within the file. The writer/hole puncher hold an
> exclusive lock on the inode while forcing a flush of outstanding
> writes to disk which creates huge latency spikes for reads.
> 
> I've observed read's taking >> 500ms even though none of the writes
> could be overlapping and traces seem to indicate it's the shared inode
> lock contending with the exclusive inode lock that writes / hole
> punching is holding. In my real-world test case traces indicated that
> the cause might be inode_dio_wait while holding the exclusive inode
> lock.
> 
> I've observed this behavior on XFS, EXT4 and BTRFS so it seems like a
> general VFS challenge. Generally, Btrfs seems to have less read
> latencies created than Ext4 (~2.5x more at the 0-1ms read than EXT4),
> which in turn has smaller and fewer tail latencies than XFS  (~2x more
> at 0-1ms and extreme is smaller) although all filesystems exhibit the
> issue. Starting with the XFS list in case I'm way off base that this
> is a problem.
> 
> Attaching a simplified reproduction of what I'm doing with one writer
> appending data to the file,

The repro is using DIO writes. Speaking for XFS, these need needs
exclusive locking to serialise extending the file without causing
data corruption or stale data exposure. Once extended, the locking
demotes to shared, so the IO itself runs concurrently with reads.

Appending DIO writes are basically a fast combined 'truncate up,
write' operation, and so has the same initial serialisation
constraints as 'truncate up' (i.e. exclusive locking).

> a separate writer punching holes in
> unrelated ranges,

fallocate() is like truncate - it also needs exclusive locking and
no IO in flight to avoid a UAF of the data extent being punched out.

> and an O_DIRECT reader that's reading data.

Shared locking only.

> Reads
> are doing random 4kib I/O in [0, 64MiB), hole punching is punching
> 4kib holes in [64MiB, 576Mib] and writing is always appending after
> 577MiB of the file.

Educate me: what application workload has this IO pattern? Is this a
real world problem. i.e. why should we care about this contrived
scenario?

Also, have you tried to implement highly concurrent range locks that
don't have far worse contention on the range lock tracking
structures themselves than a simple rwsem? Some of us here have
actually tried to do this, and the fact that we still use rwsems
should tell you a lot...

-Dave.
-- 
Dave Chinner
dgc@kernel.org