Re: Really bad I/O lock contention between non-overlapping reader and writer

[Dave Chinner <dgc@kernel.org>]

On Mon, Oct 12, 2026 at 01:59:53PM -0700, Vitali Lovich wrote:
> On Mon, Oct 12, 2026 at 1:47 PM Dave Chinner <dgc@kernel.org> wrote:
> 
> > On Mon, Oct 12, 2026 at 10:47:56AM -0700, Vitali Lovich wrote:
> > > I think I encountered suboptimal locking behavior with concurrent
> > > readers and writers on a file, which resulted in huge latency spikes
> > > for the reader even though the writers and readers are touching
> > > unrelated ranges within the file. The writer/hole puncher hold an
> > > exclusive lock on the inode while forcing a flush of outstanding
> > > writes to disk which creates huge latency spikes for reads.
> > >
> > > I've observed read's taking >> 500ms even though none of the writes
> > > could be overlapping and traces seem to indicate it's the shared inode
> > > lock contending with the exclusive inode lock that writes / hole
> > > punching is holding. In my real-world test case traces indicated that
> > > the cause might be inode_dio_wait while holding the exclusive inode
> > > lock.
> > >
> > > I've observed this behavior on XFS, EXT4 and BTRFS so it seems like a
> > > general VFS challenge. Generally, Btrfs seems to have less read
> > > latencies created than Ext4 (~2.5x more at the 0-1ms read than EXT4),
> > > which in turn has smaller and fewer tail latencies than XFS  (~2x more
> > > at 0-1ms and extreme is smaller) although all filesystems exhibit the
> > > issue. Starting with the XFS list in case I'm way off base that this
> > > is a problem.
> > >
> > > Attaching a simplified reproduction of what I'm doing with one writer
> > > appending data to the file,
> >
> > The repro is using DIO writes. Speaking for XFS, these need needs
> > exclusive locking to serialise extending the file without causing
> > data corruption or stale data exposure. Once extended, the locking
> > demotes to shared, so the IO itself runs concurrently with reads.
> 
> 
> I’ll confirm if that’s the case but also it seems strange at a conceptual
> level that you’d need to hold the exclusive lock for such a long period of
> time, especially when there’s only a single thread reading and writing
> independent ranges (that’s not what this reproducer is because I didn’t
> want to complicate it with io_uring but it reproduces there too).

Define "long period". extending the file may require IO to be done
to zero allocated extents that the file extension exposes. Even just
the partial block range beyond EOF needs to be guaranteed to hold
zeros -before- the file size is extended.

Truncate up isn't just "write the new size to the in-memory inode";
both IO and transactions can be necessary, and they havbe to be done
atomically w.r.t. the file size extension. Hence, exclusive locking.

> > > Reads
> > > are doing random 4kib I/O in [0, 64MiB), hole punching is punching
> > > 4kib holes in [64MiB, 576Mib] and writing is always appending after
> > > 577MiB of the file.
> >
> > Educate me: what application workload has this IO pattern? Is this a
> > real world problem. i.e. why should we care about this contrived
> > scenario?
> 
> 
> 100% real scenario. This repro is just a contrived minimal example to try
> to reproduce what I’m seeing. The specific idea is to have an LSM database
> store all LSM tables within 1 file instead of separate files per LSM tables
> like RocksDB does.

The fundamental problem with your model is that it assumes that
single file IO and extent manipulation operations scale without
contention or serialisation. That has never been the case, never will
be the case.

Perhaps there's good reasons for RocksDB using a 'many files' model
for it's back end storage, and only now are you learning why they
used that model....

> It works really well for lots of reasons and generally
> works really really well but I’m running into unexpected P99 latency spikes
> in certain workloads and P100 often goes all the way past 1s.

Yup, and as long as you want to use fallocate or truncate/append
operations, there's very little that can be done about it.

> It really
> looks like it’s this exclusive locking even though the application
> guarantees we never read from any data  that’s not been fsync’ed and I only
> punch holes when I know there’s never going to be any more access for that
> range. And btw in my application everything is happening on one thread.

Tell me: how is the filesystem supposed to know what the application
is doing? They aren't omnicent....

> > Also, have you tried to implement highly concurrent range locks that
> > don't have far worse contention on the range lock tracking
> > structures themselves than a simple rwsem? Some of us here have
> > actually tried to do this, and the fact that we still use rwsems
> > should tell you a lot...
> 
> In case I wasn’t clear I don’t think it’s the lock contention that’s the
> problem.

Yes, but what I'm saying is that moving to range locking creates a
per-inode new lock contention problem that doesn't currently exist,
and that it's a far worse performance issue than what we have
now....

> My real workload (not this contrived one) does all I/O from a
> single thread. The problem is purely doing super expensive I/O (or at least
> waiting for it to complete) while holding an exclusive lock preventing
> other stuff from completing. Keeping the range structure synchronized I
> think would be unlikely to show up in practice

So the theory goes. The reality:

https://lore.kernel.org/linux-xfs/20220919230947.GM3600936@dread.disaster.area/

If it were simple, XFS would have had range locking years ago.

-Dave.
-- 
Dave Chinner
dgc@kernel.org