* Re: [PATCH v5 03/25] fsverity: generate and store zero-block hash
From: Darrick J. Wong @ 2026-03-25 16:07 UTC (permalink / raw)
To: Andrey Albershteyn
Cc: Christoph Hellwig, Andrey Albershteyn, linux-xfs, fsverity,
linux-fsdevel, ebiggers, linux-ext4, linux-f2fs-devel,
linux-btrfs
In-Reply-To: <vatqzcvepp7z6iotzayuyh7fajwdqs76pn4o3sdn3f5thiwsjc@dt46ylrbjnkp>
On Wed, Mar 25, 2026 at 01:03:40PM +0100, Andrey Albershteyn wrote:
> On 2026-03-25 08:57:00, Christoph Hellwig wrote:
> > Shouldn't we still try to get this out of the fsverity_info first?
>
> I don't really understand why, this hash depends on salt (inode
> specific) and merkle tree block size (also inode specific).
Agreed, the merkle tree geometry and salt inputs are per-file.
I sorta wonder if the file ought to get an autogenerated salt if
userspace doesn't provide one, but not enough to go digging any deeper
into "does that make sense?"
--D
^ permalink raw reply
* Re: [PATCH v5 01/25] fsverity: report validation errors through fserror to fsnotify
From: Darrick J. Wong @ 2026-03-25 16:02 UTC (permalink / raw)
To: Andrey Albershteyn
Cc: Christoph Hellwig, Andrey Albershteyn, linux-xfs, fsverity,
linux-fsdevel, ebiggers, linux-ext4, linux-f2fs-devel,
linux-btrfs
In-Reply-To: <3lllvimldwcgi7crh3kgwy3a5z2ng2ugcka52ab47o5mwx4dnu@xyzbelhykn7n>
On Wed, Mar 25, 2026 at 12:41:23PM +0100, Andrey Albershteyn wrote:
> On 2026-03-25 08:54:02, Christoph Hellwig wrote:
> > On Thu, Mar 19, 2026 at 06:01:48PM +0100, Andrey Albershteyn wrote:
> > > Reported verification errors to fsnotify through recently added fserror
> > > interface.
> >
> > Looks good:
> >
> > Reviewed-by: Christoph Hellwig <hch@lst.de>
> Thanks!
>
> >
> > Btw, would it make sense to enhance the interface to tell why the
> > data vs considered lost? i.e. checksum failure vs
>
> > (part of) device disappeared?
>
> isn't it something filesystem should report even before fsverity
> gets the page?
Since the verification code is mostly called from read(ahead)
completion, doesn't a merkle tree verification failure result in the
folio remaining !uptodate? And all that gets compressed to EIO...
--D
^ permalink raw reply
* Re: [RFC 1/1] ext4: fail fast on repeated metadata reads after IO failure
From: Matthew Wilcox @ 2026-03-25 15:06 UTC (permalink / raw)
To: Andreas Dilger
Cc: Diangang Li, tytso, linux-ext4, linux-fsdevel, linux-kernel,
changfengnan, Diangang Li
In-Reply-To: <B53E253C-F314-4376-BD9D-58867FC8D3F6@dilger.ca>
On Wed, Mar 25, 2026 at 04:15:42AM -0600, Andreas Dilger wrote:
> On Mar 25, 2026, at 03:33, Diangang Li <diangangli@gmail.com> wrote:
> >
> > From: Diangang Li <lidiangang@bytedance.com>
> >
> > ext4 metadata reads serialize on BH_Lock (lock_buffer). If the read fails,
> > the buffer remains !Uptodate. With concurrent callers, each waiter can
> > retry the same failing read after the previous holder drops BH_Lock. This
> > amplifies device retry latency and may trigger hung tasks.
> >
> > In the normal read path the block driver already performs its own retries.
> > Once the retries keep failing, re-submitting the same metadata read from
> > the filesystem just amplifies the latency by serializing waiters on
> > BH_Lock.
> >
> > Remember read failures on buffer_head and fail fast for ext4 metadata reads
> > once a buffer has already failed to read. Clear the flag on successful
> > read/write completion so the buffer can recover. ext4 read-ahead uses
> > ext4_read_bh_nowait(), so it does not set the failure flag and remains
> > best-effort.
>
> Not that the patch is bad, but if the BH_Read_EIO flag is set on a buffer
> and it prevents other tasks from reading that block again, how would the
> buffer ever become Uptodate to clear the flag? There isn't enough state
> in a 1-bit flag to have any kind of expiry and later retry.
I've been thinking about this problem too, albeit from a folio read
perspective, not from a buffer_head read perspective. You're quite
right that one bit isn't enough. The solution I was considering but
haven't implemented yet was to tell all the current waiters that
the IO has failed, but not set any kind of permanent error flag.
I was thinking about starting with this:
+++ b/include/linux/wait_bit.h
@@ -10,6 +10,7 @@
struct wait_bit_key {
unsigned long *flags;
int bit_nr;
+ int error;
unsigned long timeout;
};
and then adding/changing various APIs to allow an error to be passed in
and noticed by the woken task.
With this change, the thundering herd all wake up, see the error and
return immediately instead of each submitting their own I/O. New reads
will retry the read, but each will only be held up for a maximum of
their own timeout.
^ permalink raw reply
* Re: [RFC 1/1] ext4: fail fast on repeated metadata reads after IO failure
From: Zhang Yi @ 2026-03-25 14:27 UTC (permalink / raw)
To: Diangang Li, Andreas Dilger, Diangang Li
Cc: tytso, linux-ext4, linux-fsdevel, linux-kernel, changfengnan
In-Reply-To: <c6f4b982-c6e4-4f77-a16d-0c381c1e25f0@bytedance.com>
Hi, Diangang,
On 3/25/2026 7:13 PM, Diangang Li wrote:
> Hi Andreas,
>
> BH_Read_EIO is cleared on successful read or write.
I think what Andreas means is, since you modified the ext4_read_bh()
interface, if the bh to be read already has the Read_EIO flag set, then
subsequent read operations through this interface will directly return
failure without issuing a read I/O. At the same time, because its state
is also not uptodate, for an existing block, a write request will not be
issued either. How can we clear this Read_EIO flag? IIRC, relying solely
on ext4_read_bh_nowait() doesn't seem sufficient to achieve this.
Thanks,
Yi.
>
> In practice bad blocks are typically repaired/remapped on write, so we
> expect recovery after a successful rewrite. If the block is never
> rewritten, repeatedly issuing the same failing read does not help.
>
> We clear the flag on successful reads so the buffer can recover
> immediately if the error was transient. Since read-ahead reads are not
> blocked, a later successful read-ahead will clear the flag and allow
> subsequent synchronous readers to proceed normally.
>
> Best,
> Diangang
>
> On 3/25/26 6:15 PM, Andreas Dilger wrote:
>> On Mar 25, 2026, at 03:33, Diangang Li <diangangli@gmail.com> wrote:
>>>
>>> From: Diangang Li <lidiangang@bytedance.com>
>>>
>>> ext4 metadata reads serialize on BH_Lock (lock_buffer). If the read fails,
>>> the buffer remains !Uptodate. With concurrent callers, each waiter can
>>> retry the same failing read after the previous holder drops BH_Lock. This
>>> amplifies device retry latency and may trigger hung tasks.
>>>
>>> In the normal read path the block driver already performs its own retries.
>>> Once the retries keep failing, re-submitting the same metadata read from
>>> the filesystem just amplifies the latency by serializing waiters on
>>> BH_Lock.
>>>
>>> Remember read failures on buffer_head and fail fast for ext4 metadata reads
>>> once a buffer has already failed to read. Clear the flag on successful
>>> read/write completion so the buffer can recover. ext4 read-ahead uses
>>> ext4_read_bh_nowait(), so it does not set the failure flag and remains
>>> best-effort.
>>
>> Not that the patch is bad, but if the BH_Read_EIO flag is set on a buffer
>> and it prevents other tasks from reading that block again, how would the
>> buffer ever become Uptodate to clear the flag? There isn't enough state
>> in a 1-bit flag to have any kind of expiry and later retry.
>>
>> Cheers, Andreas
>
^ permalink raw reply
* Re: [PATCH v2 3/3] ext4: derive f_fsid from block device to avoid collisions
From: Theodore Tso @ 2026-03-25 12:59 UTC (permalink / raw)
To: Anand Jain
Cc: Andreas Dilger, Darrick J. Wong, Anand Jain, linux-ext4,
linux-btrfs, linux-xfs, hch
In-Reply-To: <91d1e10b-b24d-477a-8724-2a75a710dd8d@gmail.com>
On Wed, Mar 25, 2026 at 06:59:32PM +0800, Anand Jain wrote:
>
> IMO, sb->s_uuid (as used by overlayfs)
> Represents a filesystem UUID that is persistent.
> It is derived from on-disk metadata.
>
> statfs()->f_fsid is..
> A kind of runtime filesystem identifier used to distinguish mounted
> filesystems within a running system.
> It may be stable across reboots or device removal and reinsertion,
> but this is not guaranteed. It may change if the device dev_t changes.
I always worry about "it might be stable, but it might not; ¯\_(ツ)_/¯"
The problem with that is that people might starting using this
kinda-of-guarantee-but-maybe-not in scripts or in programs, and then
when people try to run that script or program on a different system,
or on a different file system, things goes *boom*.
So if we want to say that it is stable so long as dev_t and the file
system the same, that's a well defined semantic.
If it's that it has no guarantees whatsoever; cloud change across
reboots; could change across remounts, then maybe it should just be a
global mount sequence number that starts with a random number at boot.
So you can use it to distinguish between different mounted file
systems, but that's *all* you can do with the thing. That would also
be a well defined semantic.
Cheers,
- Ted
^ permalink raw reply
* Re: [PATCH v5 07/25] iomap: introduce IOMAP_F_FSVERITY and teach writeback to handle fsverity
From: Andrey Albershteyn @ 2026-03-25 12:38 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Andrey Albershteyn, linux-xfs, fsverity, linux-fsdevel, ebiggers,
linux-ext4, linux-f2fs-devel, linux-btrfs, djwong
In-Reply-To: <20260325080021.GF952@lst.de>
On 2026-03-25 09:00:21, Christoph Hellwig wrote:
> > @@ -353,9 +353,16 @@ static inline bool iomap_block_needs_zeroing(const struct iomap_iter *iter,
> > {
> > const struct iomap *srcmap = iomap_iter_srcmap(iter);
> >
> > - return srcmap->type != IOMAP_MAPPED ||
> > - (srcmap->flags & IOMAP_F_NEW) ||
> > - pos >= i_size_read(iter->inode);
> > + if (srcmap->type != IOMAP_MAPPED)
> > + return true;
> > +
> > + if (srcmap->flags & IOMAP_F_NEW)
> > + return true;
> > +
> > + if (srcmap->flags & IOMAP_F_FSVERITY)
> > + return false;
> > +
> > + return pos >= i_size_read(iter->inode);
>
> This might be a good time to document the various reasons for needing
> zeroing.
something like this?
--- a/fs/iomap/buffered-io.c
+++ b/fs/iomap/buffered-io.c
@@ -344,9 +344,27 @@ static inline bool iomap_block_needs_zeroing(const struct iomap_iter *iter,
{
const struct iomap *srcmap = iomap_iter_srcmap(iter);
- return srcmap->type != IOMAP_MAPPED ||
- (srcmap->flags & IOMAP_F_NEW) ||
- pos >= i_size_read(iter->inode);
+ /*
+ * If this block is not mapped, we don't have any backing blocks to read
+ * yet
+ */
+ if (srcmap->type != IOMAP_MAPPED)
+ return true;
+
+ /*
+ * This block just got allocated and does not contain any meaningful data
+ */
+ if (srcmap->flags & IOMAP_F_NEW)
+ return true;
+
+ /*
+ * fsverity metadata is stored past i_size, we need to read it instead of
+ * zeroing
+ */
+ if (srcmap->flags & IOMAP_F_FSVERITY)
+ return false;
+
+ return pos >= i_size_read(iter->inode);
}
>
> > +/*
> > + * IO happens beyond inode EOF, fsverity metadata is stored there
> > + */
> > +#define IOMAP_F_FSVERITY (1U << 10)
>
> This comment feels a bit too sparse. Here is my interpretation of what
> this flag does:
>
> /*
> * Indicates reads and writes of fsverity metadata.
> *
> * Fsverity metadata is stored after the regular file data and thus beyond
> * i_size.
> */
Thanks! I will update it
--
- Andrey
^ permalink raw reply
* Re: [PATCH v5 09/25] iomap: issue readahead for fsverity merkle tree
From: Andrey Albershteyn @ 2026-03-25 12:08 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Andrey Albershteyn, linux-xfs, fsverity, linux-fsdevel, ebiggers,
linux-ext4, linux-f2fs-devel, linux-btrfs, djwong
In-Reply-To: <20260325080407.GG952@lst.de>
On 2026-03-25 09:04:07, Christoph Hellwig wrote:
> On Thu, Mar 19, 2026 at 06:01:56PM +0100, Andrey Albershteyn wrote:
> > Issue reading of fsverity merkle tree on the fsverity inodes. This way
> > metadata will be available at I/O completion time.
>
> Patches 8 to 10 really look like they should be merged to me. They
> all add the actual fsverity support to the iomap buffered read path,
> and merging them and having a single explanation of the side seems
> helpful both for review and future archeology
I see, thanks, I will merge them.
--
- Andrey
^ permalink raw reply
* Re: [PATCH v5 03/25] fsverity: generate and store zero-block hash
From: Andrey Albershteyn @ 2026-03-25 12:03 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Andrey Albershteyn, linux-xfs, fsverity, linux-fsdevel, ebiggers,
linux-ext4, linux-f2fs-devel, linux-btrfs, djwong
In-Reply-To: <20260325075700.GC952@lst.de>
On 2026-03-25 08:57:00, Christoph Hellwig wrote:
> Shouldn't we still try to get this out of the fsverity_info first?
I don't really understand why, this hash depends on salt (inode
specific) and merkle tree block size (also inode specific).
--
- Andrey
^ permalink raw reply
* Re: [PATCH v5 01/25] fsverity: report validation errors through fserror to fsnotify
From: Andrey Albershteyn @ 2026-03-25 11:41 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Andrey Albershteyn, linux-xfs, fsverity, linux-fsdevel, ebiggers,
linux-ext4, linux-f2fs-devel, linux-btrfs, djwong
In-Reply-To: <20260325075402.GA952@lst.de>
On 2026-03-25 08:54:02, Christoph Hellwig wrote:
> On Thu, Mar 19, 2026 at 06:01:48PM +0100, Andrey Albershteyn wrote:
> > Reported verification errors to fsnotify through recently added fserror
> > interface.
>
> Looks good:
>
> Reviewed-by: Christoph Hellwig <hch@lst.de>
Thanks!
>
> Btw, would it make sense to enhance the interface to tell why the
> data vs considered lost? i.e. checksum failure vs
> (part of) device disappeared?
isn't it something filesystem should report even before fsverity
gets the page?
--
- Andrey
^ permalink raw reply
* Re: [RFC 1/1] ext4: fail fast on repeated metadata reads after IO failure
From: Diangang Li @ 2026-03-25 11:13 UTC (permalink / raw)
To: Andreas Dilger, Diangang Li
Cc: tytso, linux-ext4, linux-fsdevel, linux-kernel, changfengnan
In-Reply-To: <B53E253C-F314-4376-BD9D-58867FC8D3F6@dilger.ca>
Hi Andreas,
BH_Read_EIO is cleared on successful read or write.
In practice bad blocks are typically repaired/remapped on write, so we
expect recovery after a successful rewrite. If the block is never
rewritten, repeatedly issuing the same failing read does not help.
We clear the flag on successful reads so the buffer can recover
immediately if the error was transient. Since read-ahead reads are not
blocked, a later successful read-ahead will clear the flag and allow
subsequent synchronous readers to proceed normally.
Best,
Diangang
On 3/25/26 6:15 PM, Andreas Dilger wrote:
> On Mar 25, 2026, at 03:33, Diangang Li <diangangli@gmail.com> wrote:
>>
>> From: Diangang Li <lidiangang@bytedance.com>
>>
>> ext4 metadata reads serialize on BH_Lock (lock_buffer). If the read fails,
>> the buffer remains !Uptodate. With concurrent callers, each waiter can
>> retry the same failing read after the previous holder drops BH_Lock. This
>> amplifies device retry latency and may trigger hung tasks.
>>
>> In the normal read path the block driver already performs its own retries.
>> Once the retries keep failing, re-submitting the same metadata read from
>> the filesystem just amplifies the latency by serializing waiters on
>> BH_Lock.
>>
>> Remember read failures on buffer_head and fail fast for ext4 metadata reads
>> once a buffer has already failed to read. Clear the flag on successful
>> read/write completion so the buffer can recover. ext4 read-ahead uses
>> ext4_read_bh_nowait(), so it does not set the failure flag and remains
>> best-effort.
>
> Not that the patch is bad, but if the BH_Read_EIO flag is set on a buffer
> and it prevents other tasks from reading that block again, how would the
> buffer ever become Uptodate to clear the flag? There isn't enough state
> in a 1-bit flag to have any kind of expiry and later retry.
>
> Cheers, Andreas
^ permalink raw reply
* Re: [PATCH v2 3/3] ext4: derive f_fsid from block device to avoid collisions
From: Anand Jain @ 2026-03-25 10:59 UTC (permalink / raw)
To: Andreas Dilger, Darrick J. Wong
Cc: Theodore Tso, Anand Jain, linux-ext4, linux-btrfs, linux-xfs, hch
In-Reply-To: <5DB914D4-594D-49EE-B69E-6F68E9C103A1@dilger.ca>
On 25/3/26 18:02, Andreas Dilger wrote:
> On Mar 23, 2026, at 09:29, Darrick J. Wong <djwong@kernel.org> wrote:
>>
>> On Sun, Mar 22, 2026 at 11:16:24PM -0500, Theodore Tso wrote:
>>> On Sat, Mar 21, 2026 at 07:55:19PM +0800, Anand Jain wrote:
>>>> statfs() currently reports f_fsid derived from the on-disk UUID.
>>>> Cloned block devices share the same UUID, so distinct ext4 instances
>>>> can return identical f_fsid values. This leads to collisions in
>>>> fanotify.
>>>>
>>>> Encode sb->s_dev into f_fsid instead of using the superblock UUID.
>>>> This provides a per-device identifier and avoids conflicts when
>>>> filesystem is cloned, matching the behavior with xfs.
>>>
>>> As I observed in [1] this leads to collisions when for removable block
>>> devices which can be used to mount different file systems.
>>>
>>> [1] https://lore.kernel.org/all/20260322203151.GA98947@mac.lan/
>>>
>>>> Place this change behind the new mount option "-o nouuid" for ABI
>>>> compatibility.
>>>
>>> I *really* hate this mount option. It's not at all obvious what it
>>> means for a system administrator who hasn't had the context of reading
>>> the e-mail discussion on this subject.
>>
>> I don't love 'nouuid' either, because it means something completely
>> different in XFS. 'fsid_from_dev' or something would at least be
>> clearer about what it's doing...
>>
>>> As I stated in [1], I think the f_fsid is a terrible interface that
>>> was promulgated by history, and future usage should be strongly
>>> discouraged, and the wise programmer won't use it because it has
>>> significant compatibility issues.
>>>
>>> As such, my personal preference is that we not try to condition it on
>>> a mount option, which in all likelihood almost no one will use, and
>>> instead just change it so that we hash the file system's UUID and
>>> block device number together and use that for ext4's f_fsid.
>>
>> ...but why not just set fsid to some approximation of the dev_t like
>> XFS and be done with it?
>>
>> st->f_fsid = u64_to_fsid(huge_encode_dev(mp->m_ddev_targp->bt_dev))
>>
>> There are a few other single-bdev filesystems that do this.
>
> On the flip side, if the fsid of a filesystem changes because a new disk
> was installed on a server and the old disk gets a new device number or an
> upgraded kernel with different device driver load ordering, that would
> also be a big problem, as it would break NFS file handles over a reboot.
>
> The whole point of generating the fsid from the persistent storage is that
> it is persistent across reboots. It seems like the real issue here is
> cloning filesystem images and not assigning a new UUID to the cloned image.
>
IMO, sb->s_uuid (as used by overlayfs)
Represents a filesystem UUID that is persistent.
It is derived from on-disk metadata.
statfs()->f_fsid is..
A kind of runtime filesystem identifier used to distinguish mounted
filesystems within a running system.
It may be stable across reboots or device removal and reinsertion,
but this is not guaranteed. It may change if the device dev_t changes.
I have posted a set of five test cases to the mailing list
to help verify these behaviors, for your review. Another
test case to verify device reinsertion with a different
dev_t is a WIP; it will be submitted along with v3.
Thanks.
> Cheers, Andreas
^ permalink raw reply
* Re: [RFC] Proposal: provenance_time (ptime) — a settable timestamp for cross-filesystem migration
From: Simon Richter @ 2026-03-25 10:47 UTC (permalink / raw)
To: Sean Smith, linux-fsdevel
Cc: linux-ext4, tytso, dsterba, david, brauner, osandov
In-Reply-To: <CAOx6djP4hb-Cd1Zk07SNfFfLc8irjNmbVqq+58h1Whz+h1wSFA@mail.gmail.com>
Hi,
On 3/25/26 1:26 PM, Sean Smith wrote:
> For litigation evidence, file creation timestamps are provenance
> metadata. They establish when a document first came into existence
> as a digital file. Losing this metadata during a routine filesystem
> migration is a forensic integrity failure — it destroys the timeline
> that courts, investigators, and opposing counsel may rely on.
FWIW, these creation timestamps are somewhat easily falsifiable -- by
setting the system clock before creating the file, for example.
Adding another easily falsifiable timestamp that provides conflicting
information does not help with forensic integrity, it only muddies the
waters some more.
If you need to document that a file was created before a particular
time, the proper approach is to submit a cryptographic checksum of the
file to a timestamping service. There are various such services run by
Certifying Authorities, some by notaries public, and some by volunteer
organizations.
This does not help for older documents, obviously, because such a
service would not attest to having seen the document on an older date
than the present.
"Forensic" means that the file system metadata will be analyzed by an
expert for plausibility. This expert will disbelieve any information,
including "forensic" metadata, from any file system that was stored on
media controlled by untrusted parties.
BSD has the concept of "securelevels", where even the admin user cannot
perform certain actions while the system is running in multi-user mode,
and direct modification of on-disk data bypassing the file system is
among those. The key here is "multi-user mode": this is to protect
against an attacker who has gained remote access destroying evidence.
The administrator is at any time free to switch the system to
single-user mode, which locks out any remote access, perform any
modifications they want, and return to multi-user mode. Because
switching to single-user mode disrupts remote access, this manipulation
level is unavailable to remote attackers, allowing the admin to perform
"forensic" analysis after a breach.
This is only tangentially related to evidence presented at a trial. An
expert witness can only state "this is the creation time I found on
disk", but cannot say when that timestamp was written, so it is fairly
weak evidence.
Timestamping services are much stronger: they have a system with
physical access barriers that has a single interface to the outside
world, over which it can receive a cryptographic checksum and return a
cryptographically signed message with the data received, a timestamp,
and a signature that covers both. This can be presented at court and, if
necessary, supported by the witness testimony of the service operator.
In short, this is not a technical problem that can be solved on a
computer that isn't trusted by the opposing party -- and the solution is
to bring a trusted third party into the mix. There is a reason your
lawyer stamps any letters they see with the current date.
Simon
^ permalink raw reply
* Re: [RFC 1/1] ext4: fail fast on repeated metadata reads after IO failure
From: Andreas Dilger @ 2026-03-25 10:15 UTC (permalink / raw)
To: Diangang Li
Cc: tytso, linux-ext4, linux-fsdevel, linux-kernel, changfengnan,
Diangang Li
In-Reply-To: <20260325093349.630193-2-diangangli@gmail.com>
On Mar 25, 2026, at 03:33, Diangang Li <diangangli@gmail.com> wrote:
>
> From: Diangang Li <lidiangang@bytedance.com>
>
> ext4 metadata reads serialize on BH_Lock (lock_buffer). If the read fails,
> the buffer remains !Uptodate. With concurrent callers, each waiter can
> retry the same failing read after the previous holder drops BH_Lock. This
> amplifies device retry latency and may trigger hung tasks.
>
> In the normal read path the block driver already performs its own retries.
> Once the retries keep failing, re-submitting the same metadata read from
> the filesystem just amplifies the latency by serializing waiters on
> BH_Lock.
>
> Remember read failures on buffer_head and fail fast for ext4 metadata reads
> once a buffer has already failed to read. Clear the flag on successful
> read/write completion so the buffer can recover. ext4 read-ahead uses
> ext4_read_bh_nowait(), so it does not set the failure flag and remains
> best-effort.
Not that the patch is bad, but if the BH_Read_EIO flag is set on a buffer
and it prevents other tasks from reading that block again, how would the
buffer ever become Uptodate to clear the flag? There isn't enough state
in a 1-bit flag to have any kind of expiry and later retry.
Cheers, Andreas
^ permalink raw reply
* Re: [PATCH v2 3/3] ext4: derive f_fsid from block device to avoid collisions
From: Andreas Dilger @ 2026-03-25 10:02 UTC (permalink / raw)
To: Darrick J. Wong
Cc: Theodore Tso, Anand Jain, linux-ext4, linux-btrfs, linux-xfs, hch
In-Reply-To: <20260323152943.GH6223@frogsfrogsfrogs>
On Mar 23, 2026, at 09:29, Darrick J. Wong <djwong@kernel.org> wrote:
>
> On Sun, Mar 22, 2026 at 11:16:24PM -0500, Theodore Tso wrote:
>> On Sat, Mar 21, 2026 at 07:55:19PM +0800, Anand Jain wrote:
>>> statfs() currently reports f_fsid derived from the on-disk UUID.
>>> Cloned block devices share the same UUID, so distinct ext4 instances
>>> can return identical f_fsid values. This leads to collisions in
>>> fanotify.
>>>
>>> Encode sb->s_dev into f_fsid instead of using the superblock UUID.
>>> This provides a per-device identifier and avoids conflicts when
>>> filesystem is cloned, matching the behavior with xfs.
>>
>> As I observed in [1] this leads to collisions when for removable block
>> devices which can be used to mount different file systems.
>>
>> [1] https://lore.kernel.org/all/20260322203151.GA98947@mac.lan/
>>
>>> Place this change behind the new mount option "-o nouuid" for ABI
>>> compatibility.
>>
>> I *really* hate this mount option. It's not at all obvious what it
>> means for a system administrator who hasn't had the context of reading
>> the e-mail discussion on this subject.
>
> I don't love 'nouuid' either, because it means something completely
> different in XFS. 'fsid_from_dev' or something would at least be
> clearer about what it's doing...
>
>> As I stated in [1], I think the f_fsid is a terrible interface that
>> was promulgated by history, and future usage should be strongly
>> discouraged, and the wise programmer won't use it because it has
>> significant compatibility issues.
>>
>> As such, my personal preference is that we not try to condition it on
>> a mount option, which in all likelihood almost no one will use, and
>> instead just change it so that we hash the file system's UUID and
>> block device number together and use that for ext4's f_fsid.
>
> ...but why not just set fsid to some approximation of the dev_t like
> XFS and be done with it?
>
> st->f_fsid = u64_to_fsid(huge_encode_dev(mp->m_ddev_targp->bt_dev))
>
> There are a few other single-bdev filesystems that do this.
On the flip side, if the fsid of a filesystem changes because a new disk
was installed on a server and the old disk gets a new device number or an
upgraded kernel with different device driver load ordering, that would
also be a big problem, as it would break NFS file handles over a reboot.
The whole point of generating the fsid from the persistent storage is that
it is persistent across reboots. It seems like the real issue here is
cloning filesystem images and not assigning a new UUID to the cloned image.
Cheers, Andreas
^ permalink raw reply
* Re: [RFC] Proposal: provenance_time (ptime) — a settable timestamp for cross-filesystem migration
From: Andreas Dilger @ 2026-03-25 9:58 UTC (permalink / raw)
To: Sean Smith
Cc: linux-fsdevel, linux-ext4, tytso, dsterba, david, brauner,
osandov
In-Reply-To: <CAOx6djP4hb-Cd1Zk07SNfFfLc8irjNmbVqq+58h1Whz+h1wSFA@mail.gmail.com>
On Mar 24, 2026, at 22:26, Sean Smith <defendthedisabled@gmail.com> wrote:
>
> [Email draft produced by AI agent. Reviewed, revised, and submitted by
> Sean Smith.]
>
> Hello,
>
> I am writing as a user whose work is directly harmed by the absence
> of a settable file creation timestamp in the Linux kernel. I am not a
> kernel developer, but I believe this use case may help inform the
> ongoing design discussion about btime semantics.
Implementing a new timestamp for every filesystem is unlikely to happen
any time soon. Even if the implementation was finished tomorrow, it
would be many years before a large number of users were running those
kernels, and tools had been modified to process the extra timestamps.
A simple solution that would work today would be to write a script for
Windows that extracted the creation time from NTFS and saved it as an
xattr on each file like "user.ptime" before they are copied to Linux.
Many tools today handle backup/restore and copying of xattrs already,
and since the goal of "ptime" is that it stays the same across copies,
then copying the xattr preserves the correct semantics with low effort.
The xattrs can be written in ASCII ISO8601 format like "date +%Is" does,
"2026-03-25T03:43:21-06:00", so that it sorts properly and is easily parsed.
It would be easy to set this via normal command-line tools on Linux, like:
$ setfattr -n user.ptime -v $(date -Is --date=@$(stat -c %W FILE)) FILE
$ getfattr -d -m user.ptime FILE
# file: FILE
user.ptime="2013-10-11T13:46:10-06:00"
to extract the (local) birth time in seconds, convert to ISO8601 format
and store it in the "user.ptime" xattr where it will stay unchanged.
To expand this further, you might consider submitting patches to Windows
tools (where file creation time can be set) like 7zip, rclone, etc.) to
save and restore the "user.ptime" xattr from creation time automatically.
If a few popular tools start doing this, it is more likely others would
follow suit.
Cheers, Andreas
^ permalink raw reply
* [RFC 1/1] ext4: fail fast on repeated metadata reads after IO failure
From: Diangang Li @ 2026-03-25 9:33 UTC (permalink / raw)
To: tytso, adilger.kernel
Cc: linux-ext4, linux-fsdevel, linux-kernel, changfengnan,
Diangang Li
In-Reply-To: <20260325093349.630193-1-diangangli@gmail.com>
From: Diangang Li <lidiangang@bytedance.com>
ext4 metadata reads serialize on BH_Lock (lock_buffer). If the read fails,
the buffer remains !Uptodate. With concurrent callers, each waiter can
retry the same failing read after the previous holder drops BH_Lock. This
amplifies device retry latency and may trigger hung tasks.
In the normal read path the block driver already performs its own retries.
Once the retries keep failing, re-submitting the same metadata read from
the filesystem just amplifies the latency by serializing waiters on
BH_Lock.
Remember read failures on buffer_head and fail fast for ext4 metadata reads
once a buffer has already failed to read. Clear the flag on successful
read/write completion so the buffer can recover. ext4 read-ahead uses
ext4_read_bh_nowait(), so it does not set the failure flag and remains
best-effort.
Example hung stacks:
INFO: task toutiao.infra.t:3760933 blocked for more than 327 seconds.
Call Trace:
__schedule
io_schedule
__wait_on_bit_lock
bh_uptodate_or_lock
__read_extent_tree_block
ext4_find_extent
ext4_ext_map_blocks
ext4_map_blocks
ext4_getblk
ext4_bread
__ext4_read_dirblock
dx_probe
ext4_htree_fill_tree
ext4_readdir
iterate_dir
ksys_getdents64
INFO: task toutiao.infra.t:2724456 blocked for more than 327 seconds.
Call Trace:
__schedule
io_schedule
__wait_on_bit_lock
ext4_read_bh_lock
ext4_bread
__ext4_read_dirblock
htree_dirblock_to_tree
ext4_htree_fill_tree
ext4_readdir
iterate_dir
ksys_getdents64
Signed-off-by: Diangang Li <lidiangang@bytedance.com>
Reviewed-by: Fengnan Chang <changfengnan@bytedance.com>
---
fs/buffer.c | 2 ++
fs/ext4/super.c | 12 +++++++++++-
include/linux/buffer_head.h | 2 ++
3 files changed, 15 insertions(+), 1 deletion(-)
diff --git a/fs/buffer.c b/fs/buffer.c
index 2d2e3ecec6b2b..b41d54b8b1f4d 100644
--- a/fs/buffer.c
+++ b/fs/buffer.c
@@ -145,6 +145,7 @@ static void __end_buffer_read_notouch(struct buffer_head *bh, int uptodate)
{
if (uptodate) {
set_buffer_uptodate(bh);
+ clear_buffer_read_io_error(bh);
} else {
/* This happens, due to failed read-ahead attempts. */
clear_buffer_uptodate(bh);
@@ -167,6 +168,7 @@ void end_buffer_write_sync(struct buffer_head *bh, int uptodate)
{
if (uptodate) {
set_buffer_uptodate(bh);
+ clear_buffer_read_io_error(bh);
} else {
buffer_io_error(bh, ", lost sync page write");
mark_buffer_write_io_error(bh);
diff --git a/fs/ext4/super.c b/fs/ext4/super.c
index 781c083000c2e..89a99851864a0 100644
--- a/fs/ext4/super.c
+++ b/fs/ext4/super.c
@@ -198,7 +198,13 @@ int ext4_read_bh(struct buffer_head *bh, blk_opf_t op_flags,
{
BUG_ON(!buffer_locked(bh));
+ if (!buffer_write_io_error(bh) && buffer_read_io_error(bh)) {
+ unlock_buffer(bh);
+ return -EIO;
+ }
+
if (ext4_buffer_uptodate(bh)) {
+ clear_buffer_read_io_error(bh);
unlock_buffer(bh);
return 0;
}
@@ -206,8 +212,12 @@ int ext4_read_bh(struct buffer_head *bh, blk_opf_t op_flags,
__ext4_read_bh(bh, op_flags, end_io, simu_fail);
wait_on_buffer(bh);
- if (buffer_uptodate(bh))
+ if (buffer_uptodate(bh)) {
+ clear_buffer_read_io_error(bh);
return 0;
+ }
+ if (!buffer_write_io_error(bh))
+ set_buffer_read_io_error(bh);
return -EIO;
}
diff --git a/include/linux/buffer_head.h b/include/linux/buffer_head.h
index b16b88bfbc3e7..be8bedcde379e 100644
--- a/include/linux/buffer_head.h
+++ b/include/linux/buffer_head.h
@@ -29,6 +29,7 @@ enum bh_state_bits {
BH_Delay, /* Buffer is not yet allocated on disk */
BH_Boundary, /* Block is followed by a discontiguity */
BH_Write_EIO, /* I/O error on write */
+ BH_Read_EIO, /* I/O error on read */
BH_Unwritten, /* Buffer is allocated on disk but not written */
BH_Quiet, /* Buffer Error Prinks to be quiet */
BH_Meta, /* Buffer contains metadata */
@@ -132,6 +133,7 @@ BUFFER_FNS(Async_Write, async_write)
BUFFER_FNS(Delay, delay)
BUFFER_FNS(Boundary, boundary)
BUFFER_FNS(Write_EIO, write_io_error)
+BUFFER_FNS(Read_EIO, read_io_error)
BUFFER_FNS(Unwritten, unwritten)
BUFFER_FNS(Meta, meta)
BUFFER_FNS(Prio, prio)
--
2.39.5
^ permalink raw reply related
* [RFC PATCH 0/1] ext4: fail fast on repeated metadata reads after IO failure
From: Diangang Li @ 2026-03-25 9:33 UTC (permalink / raw)
To: tytso, adilger.kernel
Cc: linux-ext4, linux-fsdevel, linux-kernel, changfengnan,
Diangang Li
From: Diangang Li <lidiangang@bytedance.com>
A production system reported hung tasks blocked for 300s+ in ext4 metadata
lookup paths. Hung task reports were accompanied by disk IO errors, but
profiling showed that most individual reads completed (or failed) within
10s, with the worst case around 60s.
At the same time, we observed a high repeat rate to the same disk LBAs.
The repeated reads frequently showed seconds-level latency and ended with
IO errors, e.g.:
[Tue Mar 24 14:16:24 2026] blk_update_request: I/O error, dev sdi,
sector 10704150288 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[Tue Mar 24 14:16:25 2026] blk_update_request: I/O error, dev sdi,
sector 10704488160 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[Tue Mar 24 14:16:26 2026] blk_update_request: I/O error, dev sdi,
sector 10704382912 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
We also sampled repeated-LBA latency histograms on /dev/sdi and saw that
the same error-prone LBAs were re-submitted many times with ~1-4s latency:
LBA 10704488160 (count=22): 1-2s: 20, 2-4s: 2
LBA 10704382912 (count=21): 1-2s: 20, 2-4s: 1
LBA 10704150288 (count=21): 1-2s: 19, 2-4s: 2
Root cause
==========
ext4 metadata reads commonly use buffer_head caching and serialize IO via
BH_Lock (lock_buffer). When a read fails, the buffer remains !Uptodate.
With multiple threads concurrently accessing the same metadata block, each
waiter wakes up after the previous owner drops BH_Lock, then submits the
same read again and waits again. This makes the latency grow linearly with
the number of contending threads, leading to 300s+ hung tasks.
The failing IOs are repeatedly issued to the same LBA. The observed 1s+
per-IO latency is likely from device-side retry/error recovery. On SCSI the
driver typically retries reads several times (e.g. 5 retries in our
environment), so a single filesystem submission can easily accumulate 5s+
delay before failing. When multiple threads then re-submit the same failing
metadata read and serialize on BH_Lock, the delay is amplified into 300s+
hung tasks.
Similar behavior exists for other devices (e.g. NVMe with multiple internal
retries).
Example hung stacks:
INFO: task toutiao.infra.t:3760933 blocked for more than 327 seconds.
Call Trace:
__schedule
io_schedule
__wait_on_bit_lock
bh_uptodate_or_lock
__read_extent_tree_block
ext4_find_extent
ext4_ext_map_blocks
ext4_map_blocks
ext4_getblk
ext4_bread
__ext4_read_dirblock
dx_probe
ext4_htree_fill_tree
ext4_readdir
iterate_dir
ksys_getdents64
INFO: task toutiao.infra.t:2724456 blocked for more than 327 seconds.
Call Trace:
__schedule
io_schedule
__wait_on_bit_lock
ext4_read_bh_lock
ext4_bread
__ext4_read_dirblock
htree_dirblock_to_tree
ext4_htree_fill_tree
ext4_readdir
iterate_dir
ksys_getdents64
Approach
========
Remember read failures on buffer_head and fail fast for ext4 metadata reads
once a buffer has already seen a read failure. Clear the flag on successful
read/write completion so the buffer can recover if the error is transient.
Note that ext4 read-ahead uses ext4_read_bh_nowait(), so it does not set
the failure flag and remains best-effort.
Patch summary
=============
1) Add BH_Read_EIO buffer_head state bit and helpers.
2) Clear BH_Read_EIO on successful read/write completion.
3) In ext4 metadata reads, if BH_Read_EIO is already set (and not
BH_Write_EIO), fail fast instead of re-submitting the same failing
read. On read failure, set BH_Read_EIO.
Diangang Li (1):
ext4: fail fast on repeated metadata reads after IO failure
fs/buffer.c | 2 ++
fs/ext4/super.c | 12 +++++++++++-
include/linux/buffer_head.h | 2 ++
3 files changed, 15 insertions(+), 1 deletion(-)
--
2.39.5
^ permalink raw reply
* Re: [PATCH v5 23/25] xfs: introduce health state for corrupted fsverity metadata
From: Christoph Hellwig @ 2026-03-25 8:10 UTC (permalink / raw)
To: Andrey Albershteyn
Cc: linux-xfs, fsverity, linux-fsdevel, ebiggers, hch, linux-ext4,
linux-f2fs-devel, linux-btrfs, djwong
In-Reply-To: <20260319170231.1455553-24-aalbersh@kernel.org>
Looks good:
Reviewed-by: Christoph Hellwig <hch@lst.de>
^ permalink raw reply
* Re: [PATCH v5 22/25] xfs: check and repair the verity inode flag state
From: Christoph Hellwig @ 2026-03-25 8:10 UTC (permalink / raw)
To: Andrey Albershteyn
Cc: linux-xfs, fsverity, linux-fsdevel, ebiggers, Darrick J. Wong,
hch, linux-ext4, linux-f2fs-devel, linux-btrfs
In-Reply-To: <20260319170231.1455553-23-aalbersh@kernel.org>
Looks good:
Reviewed-by: Christoph Hellwig <hch@lst.de>
^ permalink raw reply
* Re: [PATCH v5 21/25] xfs: advertise fs-verity being available on filesystem
From: Christoph Hellwig @ 2026-03-25 8:10 UTC (permalink / raw)
To: Andrey Albershteyn
Cc: linux-xfs, fsverity, linux-fsdevel, ebiggers, Darrick J. Wong,
hch, linux-ext4, linux-f2fs-devel, linux-btrfs,
Andrey Albershteyn
In-Reply-To: <20260319170231.1455553-22-aalbersh@kernel.org>
Looks good:
Reviewed-by: Christoph Hellwig <hch@lst.de>
^ permalink raw reply
* Re: [PATCH v5 20/25] xfs: add fs-verity ioctls
From: Christoph Hellwig @ 2026-03-25 8:09 UTC (permalink / raw)
To: Andrey Albershteyn
Cc: linux-xfs, fsverity, linux-fsdevel, ebiggers, hch, linux-ext4,
linux-f2fs-devel, linux-btrfs, djwong
In-Reply-To: <20260319170231.1455553-21-aalbersh@kernel.org>
Looks good:
Reviewed-by: Christoph Hellwig <hch@lst.de>
^ permalink raw reply
* Re: [PATCH v5 19/25] xfs: remove unwritten extents after preallocations in fsverity metadata
From: Christoph Hellwig @ 2026-03-25 8:09 UTC (permalink / raw)
To: Andrey Albershteyn
Cc: linux-xfs, fsverity, linux-fsdevel, ebiggers, hch, linux-ext4,
linux-f2fs-devel, linux-btrfs, djwong
In-Reply-To: <20260319170231.1455553-20-aalbersh@kernel.org>
Looks good:
Reviewed-by: Christoph Hellwig <hch@lst.de>
^ permalink raw reply
* Re: [PATCH v5 18/25] xfs: add fs-verity support
From: Christoph Hellwig @ 2026-03-25 8:08 UTC (permalink / raw)
To: Andrey Albershteyn
Cc: linux-xfs, fsverity, linux-fsdevel, ebiggers, hch, linux-ext4,
linux-f2fs-devel, linux-btrfs, djwong
In-Reply-To: <20260319170231.1455553-19-aalbersh@kernel.org>
Looks good:
Reviewed-by: Christoph Hellwig <hch@lst.de>
^ permalink raw reply
* Re: [PATCH v5 17/25] xfs: use read ioend for fsverity data verification
From: Christoph Hellwig @ 2026-03-25 8:07 UTC (permalink / raw)
To: Andrey Albershteyn
Cc: linux-xfs, fsverity, linux-fsdevel, ebiggers, hch, linux-ext4,
linux-f2fs-devel, linux-btrfs, djwong
In-Reply-To: <20260319170231.1455553-18-aalbersh@kernel.org>
Looks good:
Reviewed-by: Christoph Hellwig <hch@lst.de>
^ permalink raw reply
* Re: [PATCH v5 16/25] xfs: handle fsverity I/O in write/read path
From: Christoph Hellwig @ 2026-03-25 8:07 UTC (permalink / raw)
To: Andrey Albershteyn
Cc: linux-xfs, fsverity, linux-fsdevel, ebiggers, hch, linux-ext4,
linux-f2fs-devel, linux-btrfs, djwong
In-Reply-To: <20260319170231.1455553-17-aalbersh@kernel.org>
Looks good:
Reviewed-by: Christoph Hellwig <hch@lst.de>
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox