Re: Really bad I/O lock contention between non-overlapping reader and writer

[Dave Chinner <dgc@kernel.org>]

On Tue, Oct 13, 2026 at 04:49:53PM -0700, Vitali Lovich wrote:
> On Mon, Oct 12, 2026 at 1:47 PM Dave Chinner <dgc@kernel.org> wrote:
> >
> > On Mon, Oct 12, 2026 at 10:47:56AM -0700, Vitali Lovich wrote:
> > > I think I encountered suboptimal locking behavior with concurrent
> > > readers and writers on a file, which resulted in huge latency spikes
> > > for the reader even though the writers and readers are touching
> > > unrelated ranges within the file. The writer/hole puncher hold an
> > > exclusive lock on the inode while forcing a flush of outstanding
> > > writes to disk which creates huge latency spikes for reads.
> > >
> > > I've observed read's taking >> 500ms even though none of the writes
> > > could be overlapping and traces seem to indicate it's the shared inode
> > > lock contending with the exclusive inode lock that writes / hole
> > > punching is holding. In my real-world test case traces indicated that
> > > the cause might be inode_dio_wait while holding the exclusive inode
> > > lock.
> > >
> > > I've observed this behavior on XFS, EXT4 and BTRFS so it seems like a
> > > general VFS challenge. Generally, Btrfs seems to have less read
> > > latencies created than Ext4 (~2.5x more at the 0-1ms read than EXT4),
> > > which in turn has smaller and fewer tail latencies than XFS  (~2x more
> > > at 0-1ms and extreme is smaller) although all filesystems exhibit the
> > > issue. Starting with the XFS list in case I'm way off base that this
> > > is a problem.
> > >
> > > Attaching a simplified reproduction of what I'm doing with one writer
> > > appending data to the file,
> >
> > The repro is using DIO writes. Speaking for XFS, these need needs
> > exclusive locking to serialise extending the file without causing
> > data corruption or stale data exposure. Once extended, the locking
> > demotes to shared, so the IO itself runs concurrently with reads.
> >
> > Appending DIO writes are basically a fast combined 'truncate up,
> > write' operation, and so has the same initial serialisation
> > constraints as 'truncate up' (i.e. exclusive locking).
> I think that's true if the write continues at the last byte of the
> previous one. If the write leaves a gap for whatever reason (e.g.
> writes are 4kib aligned but we only had 3.5kib of data before the file
> end) the fast-path isn't taken and you do have to wait for a full
> flush because XFS doesn't realize there's nothing overlapping. This
> scenario isn't present in the test case I published but I think it
> acts very similarly to hole punching.

What happens when there are blocked beyond EOF that the file
extension exposes?

> > > a separate writer punching holes in
> > > unrelated ranges,
> >
> > fallocate() is like truncate - it also needs exclusive locking and
> > no IO in flight to avoid a UAF of the data extent being punched out.
> It needs exclusive locking and draining around the range in flight,
> not necessarily the entire i_node.

You don't need to explain how range locking works to me. If you
think that you need to explain basic stuff like this to me, then you
likely haven't understood what I've said.

> > > and an O_DIRECT reader that's reading data.
> >
> > Shared locking only.
> If I wasn't clear, the problem is that the shared lock has a head of
> line blocking behind the exclusive lock that's draining the I/O even
> though userspace knows it's never touching that range.

As I said above: competing shared vs exclusive locking implies head
of line blocking that causes latency issues. You don't need to
explain basic stuff to me.

> > > Reads
> > > are doing random 4kib I/O in [0, 64MiB), hole punching is punching
> > > 4kib holes in [64MiB, 576Mib] and writing is always appending after
> > > 577MiB of the file.
> >
> > Educate me: what application workload has this IO pattern? Is this a
> > real world problem. i.e. why should we care about this contrived
> > scenario?
> I'm trying to implement an LSM database that stores everything in one
> file instead of taking the RocksDB approach of 1 file per SST. One
> file per SST has scaling problems of its own in other ways, but for
> concurrent reads and writes they acquire locks on different inodes
> (file and directory, respectively), which is why RocksDB doesn't have
> the problem highlighted in this test-case - it always reads and writes
> to separate i_nodes with the downside being that it ends up having to
> open an insane amount of files on startup and juggle them if there's
> more than the kernel limit allows.

XFS can handle access to millions of files iat a time efficiently.
It's designed to work with massive data sets and lots of files.
You're having problems because you're trying to optimise everything
into a single file, hence hitting largely unfixable single file
scalability issues.

> > Also, have you tried to implement highly concurrent range locks that
> > don't have far worse contention on the range lock tracking
> > structures themselves than a simple rwsem? Some of us here have
> > actually tried to do this, and the fact that we still use rwsems
> > should tell you a lot...
> 
> I'm not pretending it's an easy problem to solve. I've come up with a
> few approaches and would love your feedback on what's the best path:
> 
> 1. DIO tracking so that the inode keeps track of all outstanding
> DIO's. Rather than the barrier, we scan the list to find if there's
> any conflicts.

Yup, that's a range lock.

> I've prototyped this and it clearly solves the false
> synchronization barrier from this in my database (unrelated writes
> prolonging reads). This has a negative impact on normal workloads so I
> imagine this would be hidden behind an explicit ioctl if acceptable.

No, "optional" locking models are a non-starter. It either works for
everything or it is a non-starter.

> 2. Introduce generational barriers where hole punching / writing past
> the end of the file still waits for DIOs but outside the exclusive
> lock.

The exclusive lock forms an IO submission barrier allowing in-flight
DIOs to drain and hence the extending write/fallocate operation to
start.

Without that submission barrier, how do you prevent incoming DIOs
(read and write within EOF) from preventing the in flight counter
from ever reaching zero?

i.e. a submission barrier is needed to avoid moving unbound
latencies to the extending write/fallocate operations...

> Inside the exclusive lock all it does is register that the range
> that it's doing the operation on and the wait happens outside the
> exclusive lock. Operations that overlap will either wait for all

Now we're back to range tracking....

> outstanding DIO or perhaps satisfy if they can (e.g. a read of an
> fallocate technically could just supply the 0s without waiting for
> DIO).

No, we can't return zeros to userspace until after the fallocate
operation completes successfully. If the operation has not started,
we have to return the data as it currently exists on disk - that is
a fundamental data integrity requirement.

> Is either or both of these a non-starter idea in terms of acceptance
> (assuming I can actually make it safe)?

If you come up with something that works in the general case, and it
scales out to millions of concurrent random 4kB DIO read/write IOPS
on a single file across dozens of processes submitting IO, then I'll
take what you are proposing seriously.

But until I see code that is provably correct with benchmarks that
show IO similar performance to rwsem shared locking, I'll remain
skeptical that this can be done.

Cheers,

Dave.
-- 
Dave Chinner
dgc@kernel.org