Linux filesystem development
 help / color / mirror / Atom feed
* [PATCH v7 0/7] ovl: only sync what was written through the overlay
@ 2026-11-18 17:03 Josef Bacik
  2026-11-18 17:03 ` [PATCH v7 1/7] ovl: expand fsync mount options Josef Bacik
                   ` (7 more replies)
  0 siblings, 8 replies; 16+ messages in thread
From: Josef Bacik @ 2026-11-18 17:03 UTC (permalink / raw)
  To: Miklos Szeredi, Amir Goldstein, Christian Brauner
  Cc: Alexander Viro, Jan Kara, Christoph Hellwig, Chris Mason,
	Chengguang Xu, linux-unionfs, linux-fsdevel, linux-kernel,
	Josef Bacik, Chengguang Xu

v1: https://lore.kernel.org/all/20261027-b4-ovl-containerized-sync-v1-0-802e512379a6@toxicpanda.com/
v2: https://lore.kernel.org/all/20261029-b4-ovl-containerized-sync-v2-0-4e9f08e4a052@toxicpanda.com/
v3: https://lore.kernel.org/all/20261104-b4-ovl-containerized-sync-v3-0-8daa20144f53@toxicpanda.com/
v4: https://lore.kernel.org/all/20261106-b4-ovl-containerized-sync-v4-0-6ddde0b7dd8a@toxicpanda.com/
v5: https://lore.kernel.org/all/20261107-b4-ovl-containerized-sync-v5-0-d34b4ae57fdf@toxicpanda.com/
v6: https://lore.kernel.org/all/20261110-b4-ovl-containerized-sync-v6-0-43d1be16a4d6@toxicpanda.com/

I've completely redesigned this, because there were so many problems
that I kept tripping over trying to maintain state between the upper and
the lower inode. I'm not in love with this design either, but it's the
simplest. I'm hoping somebody spots something obvious and we can do it a
different way.

v6->v7:
- Redone without a flusher. An overlay inode has nothing of its own.
  What is dirty, under writeback or deferred by lazytime is the state of
  the upper inode, and nothing tells overlayfs when that changes. Up to
  v6 the dirty bit of the overlay inode mirrored that state on a bdi of
  the overlay, and most of what the reviews found was the mirror being
  off, or the flusher acting on it. The worst one: with 3000 files open
  for writing and the system over the background threshold the flusher
  of the overlay took 79% of a CPU going over its list. (Sashiko)
- Now the overlay only keeps a set of the inodes that were modified, an
  xarray, patch 5. Only syncfs(), unmount and a read-only remount walk
  it, and they ask the upper inode of each. Periodic and background
  writeback is all up to the upper file system, like without the series.
- The xarray has the marks of the page cache and a sync uses them the
  same way. ->sync_fs() without wait tags what is dirty with towrite,
  starts writeback of those and marks them writeback, with wait it waits
  for what is marked writeback. The one difference is that there is no
  completion to clear the writeback mark. Whoever has waited does, and
  only if the upper inode is clean, another sync may have started it
  again.
- The set holds a reference. An inode leaves it when it's unlinked, or
  when a sync or the shrinker sees the upper inode clean. The shrinker
  part hangs off the ->nr_cached_objects() and ->free_cached_objects()
  hooks. Nobody who walks the set needs a lock against anybody else.
- Gone: the bdi, ->write_inode(), ->drop_inode(), the lazytime patch,
  "ovl: report failed upper writeback in syncfs()", and the special
  cases for the root and for unhashed inodes. 7 patches, v6 had 8.
- syncfs() waits for the writeback of its own inodes, whoever started
  it, and for nobody else's.
- New patch 4: write_inode_for_sync() in fs/fs-writeback.c writes one
  inode the way the flusher does for sync_inodes_sb(), WB_SYNC_ALL with
  for_sync. Up to v6 I pieced that together from sync_inode_metadata()
  without and with wait, so that ext4 wouldn't commit for every inode.
  (Sashiko)
- Deferred timestamps are handed to the upper file system in a sync, or
  when memory is needed, so lazytime keeps working in between.
- syncfs() also reports writeback that the upper file system did earlier
  and that failed. The overlay inode remembers where wb_err of the upper
  mapping was when it entered the set, and a sync compares. An fsync()
  in between doesn't take that away.
- A freeze and a read-only remount of the upper file system are held off
  while an inode is handed over. If one has already begun syncfs() syncs
  the whole upper file system like before. That closes what v6 only
  described in the changelog of patch 7. (Sashiko)
- New patch 2: an upper file system with SB_I_NO_DATA_INTEGRITY, which
  is fuse, doesn't get relaxed. A sync doesn't wait for the fuse server,
  writing the upper inodes one by one does. (Sashiko)
- The helpers that set the impure, redirect, opaque and nlink xattrs
  track the inode themselves, and so does ovl_clear_empty(). A link or a
  rename that failed after them left the inode untracked. A setattr that
  fails tracks it too. (Sashiko)
- Rebased onto 7.4-rc1. v6 was still on the base of the earlier
  versions, and patch 4 conflicted with 4f61ff2ebfb3 ("ovl: use the
  pinned upper dir in ovl_fix_origin()").

v5->v6:
- Lookup marks a merge directory and its parent when it sets the origin
  and impure xattrs on them. (Sashiko)
- New patch 8: if somebody else writes the pages of an upper inode and
  that fails, syncfs() on the overlay reports it. It's best effort, the
  limits are in the changelog, and it's on its own so it can be dropped.
  (Sashiko)
- Left alone: if the deferred timestamps can't be handed over because
  the upper file system is being frozen, the sync of the freeze can miss
  them. They are written after the thaw, it's in the changelog of patch
  7. (Sashiko)

v4->v5:
- Added sync_fs_and_bdev() to fs/sync.c for the ->sync_fs() and block
  device part of sync_filesystem(), so ovl_sync_fs() doesn't open code
  it, patch 2. (Christian)
- ovl_instantiate() marks a new inode and a new hard link dirty.
  (Christian, Sashiko)
- Copy up marks the parent as well, and the inode also when it only adds
  a link to an indexed upper inode. (Sashiko)
- Dropped "ovl: make the root inode appear hashed". A fake hash stays in
  the slab object when the inode is freed, and the next overlay inode in
  it looks hashed. The root isn't tracked at all now, ovl_sync_fs()
  writes its upper inode every time. (Sashiko)
- lazytime got a lot simpler. The overlay inode doesn't get I_DIRTY_TIME
  any more, deferred timestamps are handed to the upper file system when
  ->write_inode() finds them. That takes care of the I/O from dentry
  reclaim, so this doesn't need anything from Jan's deferred inode
  reclaim series. (Sashiko, Christian)
- ovl_sync_upper_inode() could lose an error.
- Left alone: fsync=auto still goes by RENAME_WHITEOUT, no flag. What
  that leaves uncovered is under "Things I know about". (Christian)
- Left alone: if somebody else starts the writeback of an upper inode
  and it fails, syncfs() on the overlay doesn't report it. It never has.
  (Sashiko)

v3->v4:
- Patch 1 is v2 of Amir's patch. It has ovl_tracks_dirty() and picks the
  mode for tmpfs at mount time, so I dropped my version from patch 2.
  fsync=auto stays auto if it doesn't become relaxed. (Amir)

v2->v3:
- Patch 1 is Amir's, it adds fsync=relaxed. Only an overlay with that
  syncs just what was written through it, everybody else syncs the whole
  upper file system as before. fsync=auto, the default, picks relaxed if
  the upper file system supports RENAME_WHITEOUT. (Amir)
- Dropped the fake inodes and the calls in the ovl_do_*() helpers,
  patches 7 to 9 of v2. Upper inodes without an overlay inode are left
  to ->sync_fs() and the block device, like in v1. (Amir)

v1->v2:
- Dropped the superblock flag and the ext4, xfs and btrfs patches. This
  is all in overlayfs now and works the same on any upper file system.
  (Christoph, Amir)
- Upper inodes without an overlay inode get a fake one that goes on the
  dirty list, patches 7 to 9.
- The periodic flusher pass doesn't write the upper inode any more, and
  syncfs() doesn't make ext4 commit per inode.
- Record write errors in s_wb_err of the overlay.
- Make the root inode appear hashed so it can be on the dirty list,
  don't mark unhashed inodes dirty, read i_writecount with acquire.
  (Sashiko)
- Left alone: dirtied_time_when isn't set for an overlay inode that gets
  I_DIRTY_TIME from ->write_inode(). It only matters for the 12 hour
  expiry, where the upper file system writes the timestamps on its own.
  (Sashiko)
- Rewrote my commit messages. (Christoph)

What it does now:

sync_filesystem() writes every inode on a dirty list, calls ->sync_fs()
and writes the block device. Only the first step is expensive, because
of other people's data. With fsync=relaxed overlayfs does that step only
for the upper inodes of the overlay inodes in its set, ->sync_fs()
without wait starts them and with wait waits for them, and does the
other two steps as they are.

Whiteouts and the work directories have no overlay inode, so nobody
writes those inodes. That's fine if ->sync_fs() or the block device gets
all metadata to disk, which is what relaxed asks of the upper file
system.

Things I know about:

 - ext4 without a journal supports RENAME_WHITEOUT, so it gets relaxed.
   It's fine there, its metadata is in the buffers of the block device,
   which we write. I tested it like the others.
 - f2fs and ubifs get relaxed as well and I didn't test them. f2fs
   writes its dirty inodes in the checkpoint. ubifs_sync_fs() doesn't,
   but the only thing overlayfs leaves dirty there on an inode it
   doesn't track is the chmod of the work directory to 0 at mount time,
   right after creating it with mode 0. fsync=on is there for anybody
   who needs more.
 - A default mount that gets relaxed shows that in /proc/mounts.
 - A tracked overlay inode, its upper inode and the dentry of that can't
   be reclaimed while the upper inode is dirty, like a dirty inode of
   any file system. After that they stay until a sync or the shrinker
   looks. The shrinker only queues a work item that does the looking.
   The reference it drops can be the last one to an unlinked upper
   inode, and file systems allocate memory under locks that deleting a
   file needs. Deferred timestamps count as dirty, the work item hands
   them to the upper file system.
 - When a sync reports an error it marks it as seen in wb_err of the
   upper mapping, like an fsync() does. A file that is opened on the
   upper inode after that doesn't get it any more.
 - syncfs() looks at every inode in the set, also at those that the
   upper file system has written by now.
 - A file that is open for writing stays in the set, it may be mapped.
   That doesn't cover pages pinned through a mapping. A direct read into
   one that completes after the last munmap() and close() dirties them
   when the inode may have left the set.
 - An atime update puts the inode into the set as well.
 - A read-only remount of the overlay syncs it, so it now fails once
   with the error of writeback that has failed earlier.
 - The error reporting has gaps, in all of them syncfs() returns 0 like
   it does today. With the upper file system frozen or read-only we
   fall back to syncing all of it and don't look at the inodes, so an
   error is reported by the next syncfs() that can. btrfs records an
   error after it has ended writeback, which can be after we looked.
 - Data that an earlier mount of the same upper directory left dirty,
   with fsync=off for example, isn't written by a syncfs() of a later
   mount. Today it is, along with everything else.
 - Every inode of a relaxed overlay gets a slot in the xarray when it's
   allocated, also those that are never modified, so that tracking one
   can't fail. The index is the address of the inode. i_ino isn't unique
   in an overlay and xa_alloc() needs one of the three marks. All
   overlays allocate from one slab cache, so where their inodes are
   mixed a slot can cost a node of the xarray or more. With 64 overlays
   of 2000 files, same debug kernel as below, it's 21 bytes an inode if
   they're walked one after the other, 89 if all at once, 489 if they
   take turns entry by entry. Plus about 6K for every overlay.
 - If the origin of a copied up file can't be decoded, like in a user
   namespace, a later lookup makes a second overlay inode for the same
   upper inode. If the first one is tracked it pins the upper inode
   after that was unlinked, until a sync or the shrinker looks.

Tested like before, see the v1 mail below, now also with ext2 and ext4
without a journal as the upper file system. New tests for memory
pressure, a frozen and a read-only upper file system, O_TMPFILE, write
errors, and syncfs() next to tasks that create files and next to other
syncfs() calls. Everything that was changed through the overlay is in
the copy of the block device and the 192M of the neighbours aren't,
except with ext2, which doesn't get the new mode by default.

To see that the tests can fail I broke the series in 12 places, a line
or two each, and all of them are caught.

With KASAN and lockdep on, ext4, xfs, btrfs and ext4 without a journal,
one sample each, without and with the series:

  syncfs(), the neighbours' 192M dirty         258-305ms    44-50ms
  unmount, the same                            147-273ms    37-43ms
  syncfs(), the 192M under writeback [4]         42-58ms  138-171ms
  syncfs(), 4000 inodes in the set, dirty       52-153ms   71-207ms
  syncfs(), 10000 inodes in the set, clean       55-57ms  144-146ms

So it isn't free. Walking a tree takes 0 to 3% more system time, which
is what it varies by without the series.

fstests with -overlay, the auto groups of generic and overlay, on ext4
and on xfs: nothing that passes without the series fails with it, and
the run time is within 0.3%.

[2] https://lore.kernel.org/r/20261027-b4-ext2-dirsync-v1-1-3dcd012d518d@toxicpanda.com
[3] https://lore.kernel.org/r/99629e3277656888cbe84bf7490c5e54.josef@toxicpanda.com
[4] Not btrfs, nothing was in flight there without the series.

--- Original email (v1) ---

Hello,

This picks up Chengguang Xu's "implement containerized syncfs for
overlayfs", last posted as RFC V6 in November 2021 [1].

syncfs() on an overlay and unmounting one call sync_filesystem() on the
upper file system, so they write every dirty inode on it no matter who
dirtied it. We run a lot of containers on one upper file system, and
stopping one of them writes out the dirty data of all the others. That
can take a very long time, and s_umount of the upper file system is held
the whole time, so anything that wants it exclusively queues up and then
everybody else queues up behind that.

With this an overlay only writes the upper inodes that were changed
through it, and then has the upper file system commit its metadata.

The V6 discussion ended with a direction but no V7, so that's what I
tried to follow:

 - V6 kept every overlay inode with an upper inode dirty for good. Jan
   and Amir didn't like the cost of walking them. Now an overlay inode
   is marked dirty when the upper inode is changed, mostly from
   ovl_copyattr() as Amir and Miklos suggested.
 - It never looks at the dirty flags of the upper inode, Miklos pointed
   out that they're cleared when writeback starts.
 - ->write_inode() goes through write_inode_now() on the upper inode,
   and keeps the overlay inode dirty while the upper inode is open for
   writing, which covers shared mappings.
 - V6 marked the inodes DONTCACHE, which Jan was worried about. This
   goes back to the v5 patch that keeps dirty overlay inodes cached.

One thing is new. V6 called wait_sb_inodes() on the upper file system to
catch writeback that was started but not finished, which waits for
everybody's writeback. Instead ->write_inode() keeps the overlay inode
dirty while the upper inode has pages tagged dirty or under writeback.
The tags don't have the problem the inode flags have, the writeback tag
is set before the dirty tag is cleared. So a clean overlay inode means
its data is on disk, and syncfs() only waits for its own inodes.

That takes care of the data. Metadata is harder, a lot of what is
changed on the upper file system has no overlay inode that could be on
the dirty list: the root of the overlay isn't hashed, and whiteouts,
index entries, the work directories and their parent have none. File
systems that log their metadata don't care, ->sync_fs() commits it. ext2
does, there a whiteout was gone after a crash. I first had overlayfs
write all of those itself and kept finding more of them, the last one
was the link count of the shared whiteout when a whiteout is removed.

So patch 1 adds a superblock flag, SB_I_OVL_USE_SYNC_FS, by which a file
system says that ->sync_fs() commits all of its metadata, and patches 2
to 4 set it for ext4 with a journal, xfs and btrfs. Without the flag on
the upper file system nothing changes, no inode is marked dirty and
syncfs() and unmount call sync_filesystem() like they always did.

Patches 5, 7, 8 and 10 are Chengguang's with changes noted in them,
patches 6 and 9 are new. I dropped Jan's Reviewed-by tags from V6 since
the patches changed.

lazytime on the upper file system needs its own handling. The flusher
leaves the timestamps in memory for up to 12 hours, so the overlay inode
can't be clean, but it can't stay dirty and pinned for that long either.
Patch 9 gives it I_DIRTY_TIME like the upper inode has, and forwards
->sync_lazytime() to the upper inode.

Things I know about:

 - An overlay inode that loses its last reference while it is dirty is
   kept until it has been written and then sits on the LRU, like with
   any other file system. Clean and unlinked ones are still freed right
   away.
 - Every overlay on such an upper file system gets a bdi.
 - I haven't measured syncfs() with a very large number of dirty inodes.

For testing I put several overlays on one upper file system and dirty
192M next to the overlay under test, through another overlay and
directly. Then syncfs() or unmount the one overlay, copy the block
device as it is at that moment, mount the copy, and check that
everything that was changed through the overlay is in it and that the
192M aren't. That passes with ext4, xfs and btrfs as the upper file
system. With KASAN and lockdep on, syncfs() goes from 239-284ms to
45-48ms and unmount from 126-234ms to 39-44ms. With ext2 and ext4
without a journal the 192M are written as before.

fstests with -overlay, the auto groups of generic and overlay, gives the
same results with and without the series on ext4 and on xfs.

[1] https://lore.kernel.org/r/20211122030038.1938875-1-cgxu519@mykernel.net

Thanks,

Josef

---
Changes in v7:
- Link to v6: https://patch.msgid.link/20261110-b4-ovl-containerized-sync-v6-0-43d1be16a4d6@toxicpanda.com

Changes in v6:
- Link to v5: https://patch.msgid.link/20261107-b4-ovl-containerized-sync-v5-0-d34b4ae57fdf@toxicpanda.com

Changes in v5:
- Link to v4: https://patch.msgid.link/20261106-b4-ovl-containerized-sync-v4-0-6ddde0b7dd8a@toxicpanda.com

Changes in v4:
- Link to v3: https://patch.msgid.link/20261104-b4-ovl-containerized-sync-v3-0-8daa20144f53@toxicpanda.com

Changes in v3:
- Link to v2: https://patch.msgid.link/20261029-b4-ovl-containerized-sync-v2-0-4e9f08e4a052@toxicpanda.com

Changes in v2:
- Link to v1: https://patch.msgid.link/20261027-b4-ovl-containerized-sync-v1-0-802e512379a6@toxicpanda.com

---
Amir Goldstein (1):
      ovl: expand fsync mount options

Chengguang Xu (1):
      ovl: implement containerized syncfs for overlayfs

Josef Bacik (5):
      ovl: no fsync=relaxed without data integrity
      fs: add sync_fs_and_bdev()
      fs: add write_inode_for_sync()
      ovl: add a set of the inodes that a sync has to write
      ovl: track an inode when it is modified

 Documentation/filesystems/overlayfs.rst |  45 ++-
 fs/fs-writeback.c                       |  36 ++
 fs/overlayfs/Kconfig                    |  21 ++
 fs/overlayfs/Makefile                   |   2 +-
 fs/overlayfs/copy_up.c                  |   9 +
 fs/overlayfs/dir.c                      |  11 +
 fs/overlayfs/file.c                     |   7 +-
 fs/overlayfs/inode.c                    |  27 +-
 fs/overlayfs/namei.c                    |   9 +-
 fs/overlayfs/overlayfs.h                |  39 ++-
 fs/overlayfs/ovl_entry.h                |   9 +
 fs/overlayfs/params.c                   |  34 +-
 fs/overlayfs/params.h                   |   2 +
 fs/overlayfs/readdir.c                  |   1 +
 fs/overlayfs/super.c                    |  58 +++-
 fs/overlayfs/sync.c                     | 591 ++++++++++++++++++++++++++++++++
 fs/overlayfs/util.c                     |  16 +-
 fs/sync.c                               |  52 ++-
 include/linux/fs.h                      |   1 +
 include/linux/pagemap.h                 |   1 +
 20 files changed, 920 insertions(+), 51 deletions(-)
---
base-commit: 6628e7d4c6b25c8d18f1345e3a20d21829aae7e4
change-id: 20261026-b4-ovl-containerized-sync-149bed1afe9a


^ permalink raw reply	[flat|nested] 16+ messages in thread

end of thread, other threads:[~2026-11-19 10:58 UTC | newest]

Thread overview: 16+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-18 17:03 [PATCH v7 0/7] ovl: only sync what was written through the overlay Josef Bacik
2026-11-18 17:03 ` [PATCH v7 1/7] ovl: expand fsync mount options Josef Bacik
2026-11-18 17:03 ` [PATCH v7 2/7] ovl: no fsync=relaxed without data integrity Josef Bacik
2026-11-18 18:00   ` Amir Goldstein
2026-11-18 17:03 ` [PATCH v7 3/7] fs: add sync_fs_and_bdev() Josef Bacik
2026-11-19  8:55   ` Jan Kara
2026-11-18 17:03 ` [PATCH v7 4/7] fs: add write_inode_for_sync() Josef Bacik
2026-11-19  9:14   ` Jan Kara
2026-11-19  9:47     ` OGAWA Hirofumi
2026-11-19  9:59       ` Jan Kara
2026-11-19 10:58         ` OGAWA Hirofumi
2026-11-18 17:03 ` [PATCH v7 5/7] ovl: add a set of the inodes that a sync has to write Josef Bacik
2026-11-18 17:03 ` [PATCH v7 6/7] ovl: track an inode when it is modified Josef Bacik
2026-11-18 17:03 ` [PATCH v7 7/7] ovl: implement containerized syncfs for overlayfs Josef Bacik
2026-11-18 18:09   ` Amir Goldstein
2026-11-18 18:14 ` [PATCH v7 0/7] ovl: only sync what was written through the overlay Amir Goldstein

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox