How to avoid xfs_do_force_shutdown on xlog_iodone error

How to avoid xfs_do_force_shutdown on xlog_iodone error

[Albert Duato]

Hi,

This is my first message here, so please bear with me.

I'm looking for a way of avoiding  xfs_do_force_shutdown when there is a temporary disconnection of the underlying block device. max_retries and retry_timeout_seconds are set to -1, but xlog_iodone operations seem to ignore that. Other operations like xfs_trans_read_buf_map are correctly retried.

My intention is to be transparent (i.e. freezing disk operations temporarily) to network disconnections to NBD server, but it should be equivalent to pulling the plug of a USB block device, reconnect and that everything continues to work as expected, without the physical corruption issues.

Is there a way of achieving that?

Thank you,

Albert

[92740.572442] XFS (nbd0): metadata I/O error: block 0x2806100 ("xlog_iodone") error 5 numblks 256
[92740.572450] XFS (nbd0): xfs_do_force_shutdown(0x2) called from line 1233 of file fs/xfs/xfs_log.c.  Return address = 0xffffffffc0567a48
[92740.572502] XFS (nbd0): Log I/O Error Detected.  Shutting down filesystem
[92740.572504] XFS (nbd0): Please umount the filesystem and rectify the problem(s)


     

Re: How to avoid xfs_do_force_shutdown on xlog_iodone error

[Christoph Hellwig]

You can try an xfs_freeze first and the unfreeze later after your
reconnected.  However unplugging and replugging a usb stick is generally
not just going to smoothly work, so I don't think that is a valid
equivalent.

Re: How to avoid xfs_do_force_shutdown on xlog_iodone error

[Albert Duato]

Thanks.

I didn't know about that command, but after reading its man and trying it,
it seems to require to freeze the XFS before a disconnection, and that's
nothing that could be predicted.

I should have said it on the initial mail, that the temporary disconnections
are not scheduled, but arbitrary, as they depend on network availability.

As long as all operations were retried, network gaps would only cause
delays, but not fatal failures.

Is that something I can configure?

Thanks again,

Albert

Re: How to avoid xfs_do_force_shutdown on xlog_iodone error

[Christoph Hellwig]

On Mon, Dec 03, 2018 at 09:08:40AM +0000, Albert Duato wrote:
> Thanks.
> 
> I didn't know about that command, but after reading its man and trying it,
> it seems to require to freeze the XFS before a disconnection, and that's
> nothing that could be predicted.
> 
> I should have said it on the initial mail, that the temporary disconnections
> are not scheduled, but arbitrary, as they depend on network availability.
> 
> As long as all operations were retried, network gaps would only cause
> delays, but not fatal failures.
> 
> Is that something I can configure?

No, you can't.  The only think you can do is to make sure I/Os
don't get completed on a disconnect.  Other protocols like SCSI or NBD
at usually try to reconnect and only fail the I/O if that fails.  You'd
either have to fix that in nbd, or try some nasty workarounds like
creating a dm-multipath device with a single path on top of you nbd
device and set the queue_if_no_path option.