Re: fsync() errors is unsafe and risks data loss

["Theodore Y. Ts'o" <tytso@mit.edu>]

On Thu, Apr 12, 2018 at 10:01:22PM +1000, Dave Chinner wrote:
> On Thu, Apr 12, 2018 at 07:09:14AM -0400, Jeff Layton wrote:
> > When there is a writeback error, what should be done with the dirty
> > page(s)? Right now, we usually just mark them clean and carry on. Is
> > that the right thing to do?
> 
> There isn't a right thing. Whatever we do will be wrong for someone.

That's the problem.  The best that could be done (and it's not enough)
would be to have a mode which does with the PG folks want (or what
they *think* they want).  It seems what they want is to have an error
result in the page being marked clean.  When they discover the outcome
(OOM-city and the unability to unmount a file system on a failed
drive), then they will complain to us *again*, at which point we can
tell them that want they really want is another variation on O_PONIES,
and welcome to the real world and real life.

Which is why, even if they were to pay someone to implement what they
want, I'm not sure we would want to accept it upstream --- or distro's
might consider it a support nightmare, and refuse to allow that mode
to be enabled on enterprise distro's.  But at least, it will have been
some PG-based company who will have implemented it, so they're not
wasting other people's time or other people's resources...

We could try to get something like what Google is doing upstream,
which is to have the I/O errors sent to userspace via a netlink
channel (without changing anything else about how buffered writeback
is handled in the face of errors).  Then userspace applications could
switch to Direct I/O like all of the other really serious userspace
storage solutions I'm aware of, and then someone could try to write
some kind of HDD health monitoring system that tries to do the right
thing when a disk is discovered to have developed some media errors or
something more serious (e.g., a head failure).  That plus some kind of
RAID solution is I think the only thing which is really realistic for
a typical PG site.

It's certainly that's what *I* would do if I didn't decide to use a
hosted cloud solution, such as Cloud SQL for Postgres, and let someone
else solve the really hard problems of dealing with real-world HDD
failures.   :-)

					- Ted