Should mdraid implement timeouts?

Should mdraid implement timeouts?

[Pierre Beck]

Hello,

currently, mdraid will simply block and wait for the underlying layers 
to execute commands and does not handle timeouts on its own.

In a perfect world, disks will respond within a limited timeframe when 
for example a bad sector is encountered. Unfortunately, I see even disks 
with set TLER that don't. Then, with a configurable timeout, Linux 
Kernel will reset the device in question, then the bus, then the 
controller. This process takes time (and I think the bus / controller 
reset is really adding to that time and should be optional in the first 
place) during which data is unavailable, though there is redundancy and 
another device is ready to respond.

For a read operations, things are simple: mdraid can re-issue the read 
on the redundant device(s) and deliver data. For write operations, I see 
no other option than kicking the disk from the array. With write-intent 
bitmaps in place, the disk can be re-added and resync fast once it is 
available again.

If possible, commands sent to the bad disk should be aborted, so Kernel 
doesn't reset the bus.

To add response time management, the timeout could work with several 
values and sum up like this:

max_response_time_ms = 20
timeout_ms = 10000

Every request would measure response time. If response time - 
max_response_time_ms > 0, decrease timeout_ms temporarily by that value. 
So slow disks would be kicked by the same timeout mechanism.

Greetings,

Pierre Beck

Re: Should mdraid implement timeouts?

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 3413 bytes --]

On Thu, 19 Apr 2012 15:11:45 +0200 Pierre Beck <mail@pierre-beck.de> wrote:

> Hello,
> 
> currently, mdraid will simply block and wait for the underlying layers 
> to execute commands and does not handle timeouts on its own.
> 
> In a perfect world, disks will respond within a limited timeframe when 
> for example a bad sector is encountered. Unfortunately, I see even disks 
> with set TLER that don't. Then, with a configurable timeout, Linux 
> Kernel will reset the device in question, then the bus, then the 
> controller. This process takes time (and I think the bus / controller 
> reset is really adding to that time and should be optional in the first 
> place) during which data is unavailable, though there is redundancy and 
> another device is ready to respond.
> 
> For a read operations, things are simple: mdraid can re-issue the read 
> on the redundant device(s) and deliver data. For write operations, I see 
> no other option than kicking the disk from the array. With write-intent 
> bitmaps in place, the disk can be re-added and resync fast once it is 
> available again.
> 
> If possible, commands sent to the bad disk should be aborted, so Kernel 
> doesn't reset the bus.

mdraid should definitely not - no questions, no ifs or buts or maybes -
implement timeouts.  Ever.  Just don't even consider it.
And you have identified here one of the reasons.  The command would have to
aborted and that is not possible.  But even if it were possible it would be
the wrong thing to do.

Timeouts must be handled by the lower levels - the SATA driver or the SCSI
layer or something.
We own the whole stack - we do things at the right layer.  We don't put hacks
in one layer to make up for deficiencies in another.

So if you want more control of timeouts - which I suspect is a good thing to
want - take it to the people who can actually do something about it.  Maybe
the block layer maintainer, maybe the scsi maintainer.

What mdraid *could* possibly do is submit requests with a "FAILFAST" flag
set, though there are 3 of them and there isn't much documentation
explaining how they should be used so it isn't really clear which should be
used or maybe all.
Then errors from a FAILFAST request could be handled differently to normal
errors.
This would allow us to plug-in to different timeout handling in the lower
levels which might be a useful thing.

One of the reasons I haven't explored this in much detail though is - as I
said - there isn't much documentation and there are very few usage examples
to work from and when I tried once the SCSI layer behaved really strangely
and I couldn't tell if it was wrong or if I was wrong as there was no doco to
arbitrate between us.

Hope  that helps.

NeilBrown


> 
> To add response time management, the timeout could work with several 
> values and sum up like this:
> 
> max_response_time_ms = 20
> timeout_ms = 10000
> 
> Every request would measure response time. If response time - 
> max_response_time_ms > 0, decrease timeout_ms temporarily by that value. 
> So slow disks would be kicked by the same timeout mechanism.
> 
> Greetings,
> 
> Pierre Beck
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]