Re: [Lsf-pc] [LSF/MM TOPIC] I/O error handling and fsync()

[Jeff Layton <jlayton@poochiereds.net>]

On Sat, 2017-01-14 at 09:28 +1100, NeilBrown wrote:
> On Sat, Jan 14 2017, Kevin Wolf wrote:
> 
> > Am 13.01.2017 um 15:21 hat Theodore Ts'o geschrieben:
> > > On Fri, Jan 13, 2017 at 12:09:59PM +0100, Kevin Wolf wrote:
> > > > Now even if at the moment there were no storage backend where a write
> > > > failure can be temporary (which I find hard to believe, but who knows),
> > > > a single new driver is enough to expose the problem. Are you confident
> > > > enough that no single driver will ever behave this way to make data
> > > > integrity depend on the assumption?
> > > 
> > > This is really a philosophical question.  It very much simplifiees
> > > things if we can make the assumption that a driver that *does* behave
> > > this way is **broken**.  If the I/O error is temporary, then the
> > > driver should simply not complete the write, and wait.
> > 
> > If we are sure that (at least we make it so that) every error is
> > permanent, then yes, this simplifies things a bit because it saves you
> > the retries that we know wouldn't succeed anyway.
> > 
> > In that case, what's possibly left is modifying fsync() so that it
> > consistently returns an error; or if not, we need to promise this
> > behaviour to userspace so that on the first fsync() failure it can give
> > up on the file without doing less for the user than it could do.
> 
> I think we can (and implicitly do) make that promise: if you get EIO
> From fsync, then there is no credible recovery action you can try.
> 
> > 
> > > If it fails, it should only be because it has timed out on waiting and
> > > has assumed that the problem is permanent.
> > 
> > If a manual action is required to restore the functionality, how can you
> > use a timeout for determining whether a problem is permanent or not?
> 
> If manual action is required, and can reasonably be expected, then the
> device driver should block indefinitely.
> As an example, the IBM s390 systems have a "dasd" storage driver, which
> I think is a fiber-attached storage array.  If the connection to the
> array stops working (and no more paths are available), it will (by
> default) block indefinitely.  I presume it logs the problem and the
> sysadmin can find out and fix things - or if "things" are unfixable,
> they can change the configuration to report an error.
> 
> Similary the DM multipath module has an option "queue_if_no_path" (aka
> "no_path_retry") which means that if no working paths are found, the
> request should be queued and retried (no error reported).
> 
> If manual action is an option, then the driver must be configured to wait for
> manual action.
> 
> > 
> > This is exactly the kind of errors from which we want to recover in
> > qemu instead of killing the VMs. Assuming that errors are permanent when
> > they aren't, but just require some action before they can succeed, is
> > not a solution to the problem, but it's pretty much the description of
> > the problem that we had before we implemented the retry logic.
> > 
> > So if you say that all errors are permanent, fine; but if some of them
> > are actually temporary, we're back to square one.
> > 
> > > Otherwise, every single application is going to have to learn how to
> > > deal with temporary errors, and everything that implies (throwing up
> > > dialog boxes to the user, who may not be able to do anything
> > 
> > Yes, that's obviously not a realistic option.
> > 
> > > --- this is why in the dm-thin case, if you think it should be
> > > temporary, dm-thin should be calling out to a usr space program that
> > > pages an system administrator; why do you think the process or the
> > > user who started the process can do anything about it/)
> > 
> > In the case of qemu, we can't do anything about it in terms of making
> > the request work, but we can do something useful with the information:
> > We limit the damage done, by pausing the VM and preventing it from
> > seeing a broken hard disk from which it wouldn't recover without a
> > reboot. So in our case, both the system administrator and the process
> > want to be informed.
> 
> In theory, using aio_fsync() should allow the process to determine if
> any writes are blocking indefinitely.   I have a suspicion that
> aio_fsync() is not actually asynchronous, but that might be old
> information.
> Alternately a child process could call "fsync" and report when it completed.
> 
> > 
> > A timeout could serve as a trigger for qemu, but we could possibly do
> > better for things like the dm-thin case where we know immediately that
> > we'll have to wait for manual action.
> 
> A consistent way for devices to be able to report "operator
> intervention required" would certainly be useful.  I'm not sure how easy
> it would be for a particular application to determine if such a report
> was relevant for any of its IO though.
> 
> It might not be too hard to add a flag to "sync_file_range()" to ask it to
> report the status of queues, e.g.:
>  0 - nothing queued, no data to sync
>  1 - writes are being queued, and progress appears to be normal
>  2 - queue appears to be stalled
>  3 - queue reports that admin intervention is required.
> 
> The last one would require a fair bit of plumbing to get the information
> to the right place.  The others are probably fairly easy if they can be
> defined properly.
> If you look in /sys/kernel/bdi/*/stats you will see statistic for each
> bdi (backing device info) which roughly correspond to filesystems.  You
> can easily map from a file descriptor to a bdi.
> The "BdiWriteBandwidth" will (presumably) drop if there is data to be
> written which cannot get out.  Monitoring these stats might give an
> application a useful understanding about what is happening in a particular
> storage device.
> I don't suggest that qemu should access this file, because it is a
> 'debugfs' file and not part of the api.  But the information is there
> and might be useful.  If you can show that it is directly useful to an
> application in some way, that would a useful step towards making the
> information more directly available in an api-stable way.
> 

I think my main takeaway from reading this discussion is that the
write/fsync model as a whole is really unsuitable for this (common) use
case. Given that we're already discussing using Linux specific
interfaces (sync_file_range, for instance), maybe we should turn this
topic around:

What would an ideal kernel<->userland interface for this use case look
like?

-- 
Jeff Layton <jlayton@poochiereds.net>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>