Re: bug: btrfs: direct IO fallback racing with in-flight direct IO fails fsync

[Breno Leitao <leitao@debian.org>]

On Fri, Nov 06, 2026 at 08:15:23AM +1030, Qu Wenruo wrote:
> 
> 
> 在 2026/11/5 22:37, Breno Leitao 写道:
> > Hi,
> > 
> > Atish reported that arm64 production hosts (64K pages, 4K blocks) that
> > keep VM disk images as NOCOW files on btrfs, opened with O_DIRECT, on
> > 7.x based kernels, and it still reproduces in linux-next as of
> > next-20261104:
> > 
> >    Page cache invalidation failure on direct I/O.  Possible data corruption due to collision with buffered I/O!
> > 
> > Besides the message, dio_warn_stale_pagecache() records -EIO in the
> > mapping, so the next fsync() on the image fails. The message blames the
> > application for mixing buffered and direct IO, but every write here is
> > O_DIRECT: the buffered writes come from btrfs itself, when it falls back
> > for writes that are not 4K aligned.
> > 
> > It does not look like a regression, since the fallback never waited for
> > in-flight direct IO. 64K pages make it easy to hit, as a folio then holds
> > 16 blocks. Since e9b7b9d9e78e5b ("btrfs: use IOMAP_DIO_BOUNCE flag
> > instead of falling back to buffered IO"), COW files are exposed too.
> > 
> > I was able to come up with a reproducer that hits the issue within
> > seconds with 64K pages. With XFS on a loop device backed by such a file,
> > standing in for the VM, XFS shuts down on "log I/O error -5":
> > 
> >          https://github.com/leitao/btrfs-dio-fallback
> > 
> > My robot identified this as the root cause:
> > 
> > 1. Task A has an async O_DIRECT write in flight to one 4K block of a 64K
> >     folio. iomap invalidated the page cache before submitting it, and the
> >     inode lock was dropped.
> > 2. Task B issues an O_DIRECT write whose offset, length or buffer is not
> >     4K aligned. btrfs_direct_write() falls back to btrfs_buffered_write(),
> >     which does not wait for in-flight direct IO, and dirties another block
> >     of the same folio.
> > 3. Task A's write completes. kiocb_invalidate_post_direct_write() finds
> >     the folio dirty, so invalidate_inode_pages2_range() fails with -EBUSY
> >     and dio_warn_stale_pagecache() records -EIO. The next fsync() fails.
> > 
> > ext4 fixed the same race in 15cdefd0c052 ("ext4: drain in-flight DIO
> > before buffered write fallback"), and ext2, ntfs and exfat are getting
> > the same fix. Doing the same in btrfs_buffered_write() for the fallback
> > case makes the reproducer clean, and I can send that patch if it looks
> > like the right direction.
> > 
> > This patch is a copy of the fixes above (from other FS) and make the
> > reproducer happy.
> > 
> > commit 16cd17d1e8205f9f86b8da0a9b1ccefdb2665764
> > Author: Breno Leitao <leitao@debian.org>
> > Date:   Thu Nov 5 02:59:39 2026 -0800
> > 
> >      btrfs: drain in-flight direct IO before buffered write fallback
> >      When btrfs_direct_write() cannot do a write as direct IO, because the
> >      offset, the length or the user buffer is not aligned to the sectorsize,
> >      or the range has compressed extents or page cache, it falls back to
> >      btrfs_buffered_write(). That dirties the page cache without waiting for
> >      direct IO still in flight from other tasks:
> >      1) Task A submits an async direct write to block 1 of a folio. iomap
> >         invalidates the page cache in the range and submits the bio, and the
> >         inode lock is dropped.
> >      2) Task B issues a 512 byte O_DIRECT write to block 0 of the same folio.
> >         btrfs_direct_write() falls back to btrfs_buffered_write(), which
> >         dirties the folio.
> >      3) Task A's write completes. kiocb_invalidate_post_direct_write() finds
> >         the folio dirty, invalidate_inode_pages2_range() fails with -EBUSY,
> >         and dio_warn_stale_pagecache() logs:
> >         [    2.048568] Page cache invalidation failure on direct I/O.  Possible data corruption due to collision with buffered I/O!
> >         [    2.048574] File: /tmp/diorepro.JTCKpJ/block.img PID: 260 Comm: kworker/11:1
> >         It also records -EIO in the mapping, so the next fsync() fails.
> >      The two writes do not touch the same bytes, they only share a folio.
> >      With 4K blocks on 64K pages a folio holds 16 blocks, so this is easy to
> >      hit there. Meta's arm64 hosts with 64K pages hit it with VM disk images
> >      that are NOCOW files on btrfs, opened with O_DIRECT. In a test, XFS on a
> >      loop device with direct IO on such a file gets EIO on a log write and
> >      shuts down within seconds.
> >      Wait for in-flight direct IO before the fallback dirties the page cache,
> >      as ext4 does since commit 15cdefd0c052 ("ext4: drain in-flight DIO
> >      before buffered write fallback"). btrfs_buffered_write() holds the inode
> >      lock exclusively there, so no new direct IO can start until the write is
> >      done. NOWAIT writes never get there, as btrfs_direct_write() returns
> >      -EAGAIN for them before falling back.
> >      A reproducer that keeps 4K libaio O_DIRECT writes in flight while other
> >      threads issue 512 byte O_DIRECT writes to the same folios of a NOCOW
> >      file got 1123 fdatasync() EIOs in 10 seconds on a 64K page VM, and none
> >      with this patch.
> >      Fixes: f85781fb505e ("btrfs: switch to iomap for direct IO")
> >      Reported-by: Atish Patra <atish.patra@linux.dev>
> >      Signed-off-by: Breno Leitao <leitao@debian.org>
> > 
> > diff --git a/fs/btrfs/file.c b/fs/btrfs/file.c
> > index f978c6524aa03f..2017a96814d857 100644
> > --- a/fs/btrfs/file.c
> > +++ b/fs/btrfs/file.c
> > @@ -1334,6 +1334,14 @@ ssize_t btrfs_buffered_write(struct kiocb *iocb, struct iov_iter *iter)
> >   	if (ret < 0)
> >   		return ret;
> > +	/*
> > +	 * A direct write that fell back to buffered IO must not dirty folios
> > +	 * while direct IO is in flight: its completion fails to invalidate
> > +	 * them and records -EIO in the mapping.
> > +	 */
> > +	if (iocb->ki_flags & IOCB_DIRECT)
> > +		inode_dio_wait(inode);
> > +
> 
> Sashiko exposed a problem that the inode can already be enlarged before:
> 
> https://sashiko.dev/#/patchset/auxv7Tg0xSq5yHmb%40gmail.com

That seems important as well, and not hard to fix. Let me try to hack
something.

Thanks,
--breno