[Cluster-devel] [GFS2 PATCH] gfs2: Panic when an io error occurs writing

[Bob Peterson <rpeterso@redhat.com>]

----- Original Message -----
> I think a reasonable alternative to this is to just sit in an infinite retry
> loop until the i/o succeeds.
> 
> Dave
> [Mark Syms] I would hope that this code would only trigger after some effort
> has been put into  retrying as panicing the host on the first I/O failure
> seems like a sure fire way to get unhappy users (and in our case paying
> customers). As Edvin points out there may be other filesystems that may be
> able to cleanly unmount and thus avoid having to check everything on
> restart.

Hi Mark,

Perhaps. I'm not block layer or iscsi expert, but afaik, it's not the file
system's job to retry IO, and never has been, right?

There are already iscsi tuning parameters, vfs tuning parameters, etc. So
if an IO error is sent to GFS2 for a write operation, it means the retry
algorithms and operation timeout algorithms built into the layers below us
(the iscsi layer, scsi layer, block layer, tcp/ip layer etc.) have all failed
and given up on the IO operation. We can't really justify adding yet another
layer of retries on top of all that, can we?

I see your point, and perhaps the system should stay up to continue other
mission-critical operations that may not require the faulted hardware.
But what's a viable alternative? As Dave T. suggested, we can keep resubmitting
the IO until it completes, but then the journal never gets replayed and nobody
can have those locks ever again, and that would cause a potential hang of the
entire cluster, especially in cases where there's only one device that's failed
and the whole cluster is using it.

In GFS2, we've got a concept of marking a resource group "in error" so the
other nodes won't try to use it, but the same corruption that affects resource
groups could be extrapolated to "hot" dinodes as well. For example, suppose the
root (mount-point) dinode was in the journal. Now the whole cluster is hung
rather than just the one node.

Regards,

Bob Peterson
Red Hat File Systems