Re: [QUESTION] xfs, iomap: Handle writeback errors to prevent silent data corruption

[Christoph Hellwig <hch@infradead.org>]

On Wed, Jun 04, 2025 at 08:05:03AM +1000, Dave Chinner wrote:
> > 
> > Everything.  ENOSPC means there is no space.  There might be space in
> > the non-determinant future, but if the layer just needs to GC it must
> > not report the error.
> 
> GC of thin pools requires the filesystem to be mounted so fstrim can
> be run to tell the thinp device where all the free LBA regions it
> can reclaim are located. If we shut down the filesystem instantly
> when the pool goes ENOSPC on a metadata write, then *we can't run
> fstrim* to free up unused space and hence allow that metadata write
> to succeed in the future.
> 
> It should be obvious at this point that a filesystem shutdown on an
> ENOSPC error from the block device on anything other than journal IO
> is exactly the wrong thing to be doing.

How high are the chances that you hit exactly the rate metadata
writeback I/O and not journal or data I/O for this odd condition
that requires user interaction?  Where is this weird model where a
storage device returns an out of space error and manual user interaction
using manual and not online trim is going to fix even documented?

> > Normally it means your checksum was wrong.  If you have bit errors
> > in the cable they will show up again, maybe not on the next I/O
> > but soon.
> 
> But it's unlikely to be hit by another cosmic ray anytime soon, and
> so bit errors caused by completely random environmental events
> should -absolutely- be retried as the subsequent write retry will
> succeed.
>
> If there is a dodgy cable causing the problems, the error will
> re-occur on random IOs and we'll emit write errors to the log that
> monitoring software will pick up. If we are repeatedly isssuing write
> errors due to EILSEQ errors, then that's a sign the hardware needs
> replacing.

Umm, all the storage protocols do have pretty good checksums.  A cosmic
ray isn't going to fail them it is something more fundamental like
broken hardware or connections.  In other words you are going to see
this again and again pretty frequently.

> There is no risk to filesystem integrity if write retries
> succeed, and that gives the admin time to schedule downtime to
> replace the dodgy hardware. That's much better behaviour than
> unexpected production system failure in the middle of the night...
> 
> It is because we have robust and resilient error handling in the
> filesystem that the system is able to operate correctly in these
> marginal situations. Operating in marginal conditions or as hardware
> is beginning to fail is a necessary to keep production systems
> running until corrective action can be taken by the administrators.

I'd really like to see a format writeup of your theory of robust error
handling where that robustness is centered around the fairly rare
case of metadata writeback and applications dealing with I/O errors,
while journal write errors and read error lead to shutdown.  Maybe
I'm missing something important, but the theory does not sound valid,
and we don't have any testing framework that actually verifies it.

> Failing to recognise that transient and "maybe-transient" errors can
> generally be handled cleanly and successfully with future write
> retries leads to brittle, fragile systems that fall over at the
> first sign of anything going wrong. Filesystems that are targetted
> at high value production systems and/or running mission critical
> applications needs to have resilient and robust error handling.

What known transient errors do you think XFS (or any other file system)
actually handles properly?  Where is the contract that these errors
actually are transient.

> > And even applications that fsync won't see you fancy error code.  The
> > only thing stored in the address_space for fsync to catch is EIO and
> > ENOSPC.
> 
> The filesystem knows exactly what the IO error reported by the block
> layer is before we run folio completions, so we control exactly what
> we want to report as IO compeltion status.

Sure, you could invent a scheme to propagate the exaxct error.  For
direct I/O we even return the exact error to userspace.  But that
means we actually have a definition of what each error means, and how
it could be handled.  None of that exists right now.  We could do
all this, but that assumes you actually have:

 a) a clear definition of a problem
 b) a good way to fix that problem
 c) good testing infrastructure to actually test it, because without
    that all good intentions will probably cause more problems than
    they solve

> Hence the bogosities of error propagation to userspace via the
> mapping is completely irrelevant to this discussion/feature because
> it would be implemented below the layer that squashes the eventual
> IO errno into the address space...

How would implement and test all this?  And for what use case?