Re: [linux-iscsi-devel] [question] deferred sense

Re: [linux-iscsi-devel] [question] deferred sense

[Christoph Hellwig]

On Wed, Jan 05, 2005 at 09:21:12AM -0600, Scott M. Ferris wrote:
> On Wed, Jan 05, 2005 at 10:21:44AM +0100, Christoph Hellwig wrote:
> > 
> > The kernel still doesn't hadle deferred sense correctly.  In fact I'm
> > not sure it's entirely possible to handle deferred sense correctly.
> > 
> > The iscsi driver variant of "translating" it to normal sense data and
> > that for disks only certainly doesn't make much sense.
> 
> To be more specific, there were some devices that would fail a command
> and return deferred sense.  The command didn't complete at the target,
> and the kernel wasn't retrying it because the sense was deferred
> rather than current.  For those devices, the translation produced the
> desired retry.

Do you remember these devices?  Might be worth adding a midlayer
blacklist entry for them.

And I suspect we should get linux-scsi in the loop for the discussion
about deferred sense handling.

Re: [linux-iscsi-devel] [question] deferred sense

[James Bottomley]

On Wed, 2005-01-05 at 16:23 +0100, Christoph Hellwig wrote:
> On Wed, Jan 05, 2005 at 09:21:12AM -0600, Scott M. Ferris wrote:
> > To be more specific, there were some devices that would fail a command
> > and return deferred sense.  The command didn't complete at the target,
> > and the kernel wasn't retrying it because the sense was deferred
> > rather than current.  For those devices, the translation produced the
> > desired retry.
> 
> Do you remember these devices?  Might be worth adding a midlayer
> blacklist entry for them.

That's certainly possible ... although we'd need a lot more details.
Any device that returns deferred sense for a current error is pretty
badly broken according to the spec.

> And I suspect we should get linux-scsi in the loop for the discussion
> about deferred sense handling.

Well, I'm willing to think about it again, but beyond reporting to the
user that "something went wrong" I'm not sure there's anything we can
do.

The most common case for deferred sense is using a writeback cache.
Here the device takes the write into its cache and returns GOOD.  Later
trying to write it to the device it gets a medium error and now tries to
send deferred sense saying "you remember that write I told you worked,
well actually..."  by this time, we've already acknowledged the write to
the block layer and any JFS will proceed on the assumption that it
completed.  There's no way we can go back in time and apply this error
to an already completed transaction.  However, the device has violated
the journalling assumptions, and the FS will be corrupt.

So does anyone have any better suggestions for handling deferred sense?

James

RE: [linux-iscsi-devel] [question] deferred sense

[Salyzyn, Mark]

James Bottomley sez:
> The most common case for deferred sense is using a writeback cache.
> Here the device takes the write into its cache and returns GOOD.
Later
> trying to write it to the device it gets a medium error and now tries
to
> send deferred sense saying "you remember that write I told you worked,
> well actually..."  by this time, we've already acknowledged the write
to
> the block layer and any JFS will proceed on the assumption that it
> completed.  There's no way we can go back in time and apply this error
> to an already completed transaction.  However, the device has violated
> the journalling assumptions, and the FS will be corrupt.
> 
> So does anyone have any better suggestions for handling deferred
sense?

Change the assumptions in Journalling filesystems and make it work in a
deferred sense environment.

Don't delete the Journal until a safe point of time when the writeback
is confirmed either by underlying knowledge, time (flush interval
passed), amount of data (dirty cache is forced flushed), or a read-back
(resulting from the deferred error investigation). RAID cards and
external enclosures do this all the time, but to be fair with usually
more knowledge of all the cache handling.

Sincerely -- Mark Salyzyn

RE: [linux-iscsi-devel] [question] deferred sense

[James Bottomley]

On Wed, 2005-01-05 at 12:47 -0500, Salyzyn, Mark wrote:
> Change the assumptions in Journalling filesystems and make it work in a
> deferred sense environment.

It's possible, but I think it will involve large changes to the kernel.

> Don't delete the Journal until a safe point of time when the writeback
> is confirmed either by underlying knowledge, time (flush interval
> passed), amount of data (dirty cache is forced flushed), or a read-back
> (resulting from the deferred error investigation). RAID cards and
> external enclosures do this all the time, but to be fair with usually
> more knowledge of all the cache handling.

Well ... this proposal would change every filesystem in the kernel for
something that occurs rather rarely (I've never actually seen it
happen), so it is rather a lot of work for not very much benefit.
However, if you have actual proposals, lets see them.

I suggest you publish the API to linux-scsi and linux-ide since we'd be
the primary implementers.  The fs changes should go to the relevant fs
lists (probably cc linux-kernel) and we'll be glad to look them over.

James

Re: [linux-iscsi-devel] [question] deferred sense

[Matthew Wilcox]

On Wed, Jan 05, 2005 at 04:51:57PM -0500, James Bottomley wrote:
> I suggest you publish the API to linux-scsi and linux-ide since we'd be
> the primary implementers.  The fs changes should go to the relevant fs
> lists (probably cc linux-kernel) and we'll be glad to look them over.

linux-fsdevel (@vger) would probably be the right place to get feedback
from filesystem authors.

-- 
"Next the statesmen will invent cheap lies, putting the blame upon 
the nation that is attacked, and every man will be glad of those
conscience-soothing falsities, and will diligently study them, and refuse
to examine any refutations of them; and thus he will by and by convince 
himself that the war is just, and will thank God for the better sleep 
he enjoys after this process of grotesque self-deception." -- Mark Twain

RE: [linux-iscsi-devel] [question] deferred sense

[Salyzyn, Mark]

James Bottomley sez:
> Well ... this proposal would change every filesystem in the kernel for
> something that occurs rather rarely (I've never actually seen it
> happen), so it is rather a lot of work for not very much benefit.
> However, if you have actual proposals, lets see them.

Proposal would be difficult and illegal for me; I am bound in my
engineering contract to a proprietary world :-( I can just not clean my
room well enough.

However, you would see a similar problem being generated by a need for
quick and hot system failover and cache synchronization, bubbling all
the way up to the Linux disk cache; not just at the physical DASD level.
If the file-system is immune to a deferred failure from cached systems
(disk, RAID or linux cache), it will be hardened to deal with component
to complete system failover.

Brute force, with performance penalty, is to periodically flush cache
(SCSI_SYNC command 0x35 being the weapon of choice), and when that
completes the Journal is cleansed. Added (performance) value, is to
survive a failover or deferred failure without the need to do this
synchronization.

(Note: Adaptec's branch of the aacraid driver supports SCSI_SYNC)

Sincerely -- Mark Salyzyn

RE: [linux-iscsi-devel] [question] deferred sense

[James Bottomley]

On Thu, 2005-01-06 at 09:23 -0500, Salyzyn, Mark wrote:
> However, you would see a similar problem being generated by a need for
> quick and hot system failover and cache synchronization, bubbling all
> the way up to the Linux disk cache; not just at the physical DASD level.
> If the file-system is immune to a deferred failure from cached systems
> (disk, RAID or linux cache), it will be hardened to deal with component
> to complete system failover.

Disc cache usually lies below the layer where failover occurs, so is
definitely not a problem.  If you're thinking of having a writeback
cache in a raid card on a host, then you'll find all the documentation
for clustering will explicitly warn you to switch it to write through (I
know ours does).

> Brute force, with performance penalty, is to periodically flush cache
> (SCSI_SYNC command 0x35 being the weapon of choice), and when that
> completes the Journal is cleansed. Added (performance) value, is to
> survive a failover or deferred failure without the need to do this
> synchronization.

We already do cache sync on shutdown.  The problem with periodic cache
sync is the arrays which have huge caches (EMCs go up to gigabytes)
which take ages to flush, so such a periodic sync can stall a system.

James

RE: [linux-iscsi-devel] [question] deferred sense

[Salyzyn, Mark]

James Bottomley sez: 
> If you're thinking of having a writeback
> cache in a raid card on a host, then you'll find all the documentation
> for clustering will explicitly warn you to switch it to write through
(I
> know ours does).

The key to higher performance is a cluster solution that permits cache
on, and in doing so, the issues of deferred failure is similar. I am not
talking pie in the sky here; there are proprietary solutions (from low
end all the way up to the `world scale cache' of the Yotta-Yotta's of
the world)

The point is deferred failure is not the only user of these changes to
the file-system layer. In the low end, this kind of technology is being
added to Windoze and has been part of Netware (including complete system
memory synchronization between failover machines) for years; there are
customers of redundant solutions that require this.

In the Public Domain, simplicity usually drives adoption (just measure
the pain Sistina had asking for a new atomic SCSI command). I am not
sure the General Purpose customer wants to 'pay' for unused complexity
and the accompanying doubt of stability ...

Sincerely -- Mark Salyzyn

RE: [linux-iscsi-devel] [question] deferred sense

[James Bottomley]

On Thu, 2005-01-06 at 09:57 -0500, Salyzyn, Mark wrote:
> The key to higher performance is a cluster solution that permits cache
> on, and in doing so, the issues of deferred failure is similar. I am not
> talking pie in the sky here; there are proprietary solutions (from low
> end all the way up to the `world scale cache' of the Yotta-Yotta's of
> the world)

HA is about safety first and then performance.  Something with a local
writeback cache is always HA unsafe and thus it won't be supported by
the major HA vendors.

> The point is deferred failure is not the only user of these changes to
> the file-system layer. In the low end, this kind of technology is being
> added to Windoze and has been part of Netware (including complete system
> memory synchronization between failover machines) for years; there are
> customers of redundant solutions that require this.
> 
> In the Public Domain, simplicity usually drives adoption (just measure
> the pain Sistina had asking for a new atomic SCSI command). I am not
> sure the General Purpose customer wants to 'pay' for unused complexity
> and the accompanying doubt of stability ...

I'll agree but only if you can't achieve the desired end by simple
means.  In this case, you can, you set the cache to writethrough and the
problem goes away.

James

Re: [linux-iscsi-devel] [question] deferred sense

[Tom Coughlan]

On Wed, 2005-01-05 at 11:44, James Bottomley wrote:
> On Wed, 2005-01-05 at 16:23 +0100, Christoph Hellwig wrote:
> > On Wed, Jan 05, 2005 at 09:21:12AM -0600, Scott M. Ferris wrote:
> > > To be more specific, there were some devices that would fail a command
> > > and return deferred sense.  The command didn't complete at the target,
> > > and the kernel wasn't retrying it because the sense was deferred
> > > rather than current.  For those devices, the translation produced the
> > > desired retry.
> > 
> > Do you remember these devices?  Might be worth adding a midlayer
> > blacklist entry for them.
> 
> That's certainly possible ... although we'd need a lot more details.
> Any device that returns deferred sense for a current error is pretty
> badly broken according to the spec.

If a current command returns deferred sense, the SCSI spec. requires
that the current command shall not have been executed [1]. So, if at
some point in the past the kernel did not retry a current command that
returned deferred sense, the iscsi folks would have forced the retry by
converting deferred sense to current sense. The scenario does not
require a device that is working incorrectly.

The big flaw in what iscsi did is the case where the deferred sense
indicates a non-fatal error. In that case, iscsi converts it to current,
the mitlayer examines it and determines that it does not require a
retry. This causes the current command to complete to the application
even though it was not executed by the device. 

It looks to me as though the 2.4 iscsi driver is susceptible to this. It
is probably not seen in practice because disk devices that return
non-fatal deferred sense are rare (it probably requires the PER bit set
in the error recovery mode page?). Anyone know for sure?

[1] If the task terminates with CHECK CONDITION status and the sense
data describes a deferred error the command for the terminated task
shall not have been processed. (SPC-3) 

RE: [linux-iscsi-devel] [question] deferred sense

[Bryan Henderson]

>> Change the assumptions in Journalling filesystems and make it work in a
>> deferred sense environment.
>
>It's possible, but I think it will involve large changes to the kernel.
>
>> Don't delete the Journal until a safe point of time when the writeback
>> is confirmed either by underlying knowledge, time (flush interval
>> passed), amount of data (dirty cache is forced flushed), or a read-back
>> (resulting from the deferred error investigation). RAID cards and
>> external enclosures do this all the time, but to be fair with usually
>> more knowledge of all the cache handling.
>
>Well ... this proposal would change every filesystem in the kernel for
>something that occurs rather rarely (I've never actually seen it
>happen), so it is rather a lot of work for not very much benefit.

It's especially little benefit because there's no reason for the SCSI 
target to be in write back mode.  Caching and I/O scheduling within Linux 
and SCSI command queueing do pretty much everything write back caching in 
the target is intended for.  So I think the right solution for this 
problem (which I agree is a major problem) is for journalled filesystems 
to keep the disk in non-writeback mode.

Don't you usually have to go out of your way to get a disk drive to do 
writeback?

--
Bryan Henderson                          IBM Almaden Research Center
San Jose CA                              Filesystems

Re: [linux-iscsi-devel] [question] deferred sense

[Douglas Gilbert]

Tom Coughlan wrote:
> On Wed, 2005-01-05 at 11:44, James Bottomley wrote:
> 
>>On Wed, 2005-01-05 at 16:23 +0100, Christoph Hellwig wrote:
>>
>>>On Wed, Jan 05, 2005 at 09:21:12AM -0600, Scott M. Ferris wrote:
>>>
>>>>To be more specific, there were some devices that would fail a command
>>>>and return deferred sense.  The command didn't complete at the target,
>>>>and the kernel wasn't retrying it because the sense was deferred
>>>>rather than current.  For those devices, the translation produced the
>>>>desired retry.
>>>
>>>Do you remember these devices?  Might be worth adding a midlayer
>>>blacklist entry for them.
>>
>>That's certainly possible ... although we'd need a lot more details.
>>Any device that returns deferred sense for a current error is pretty
>>badly broken according to the spec.
> 
> 
> If a current command returns deferred sense, the SCSI spec. requires
> that the current command shall not have been executed [1]. So, if at
> some point in the past the kernel did not retry a current command that
> returned deferred sense, the iscsi folks would have forced the retry by
> converting deferred sense to current sense. The scenario does not
> require a device that is working incorrectly.
> 
> The big flaw in what iscsi did is the case where the deferred sense
> indicates a non-fatal error. In that case, iscsi converts it to current,
> the mitlayer examines it and determines that it does not require a
> retry. This causes the current command to complete to the application
> even though it was not executed by the device. 
> 
> It looks to me as though the 2.4 iscsi driver is susceptible to this. It
> is probably not seen in practice because disk devices that return
> non-fatal deferred sense are rare (it probably requires the PER bit set
> in the error recovery mode page?). Anyone know for sure?

Tom,
 From my reading of SBC-2 (rev 16, 13 Nov 2004) deferred errors
cannot be turned off by a mode page. The VERIFY and
WRITE AND VERIFY commands can be used to make sure blocks get
to the media (or not) without deferred errors. There are also
the "Force Unit Access" (FUA and FUA_NV) bits in the READ and
WRITE commands (but not the 6 byte variants).

The t10 folks have added the idea of non-volatile cache in
a disk (more likely a RAID) which further complicates things.
Now a deferred error could theoretically span a power cycle!

The PER bit in the Read-Write Error Recovery mode page (SBC-2
rev 16 section 6.3.4) controls whether RECOVERED ERRORs are
reported or not. Also if the ARRE bit (for reads) and/or the
AWRE bit (for writes) are set in the same mode
page, the offending block will be remapped (whether a recovered
error is reported on not). If PER is 0 then RECOVERED ERRORs
are not reported. [In any case the Error Counter log pages
should reflect the problems (and perhaps the "grown" defect
list) so smartmontools may be of use.]

> [1] If the task terminates with CHECK CONDITION status and the sense
> data describes a deferred error the command for the terminated task
> shall not have been processed. (SPC-3) 

I fear my tinkering with sense descriptor data in
the mid level may have tripped up on this case (if
it wasn't already broken), that is: a deferred error
report must cause the current command to be retried,
even if the deferred error was reporting a
recovered error. This case can only arise when PER=1.

Doug Gilbert

Re: [linux-iscsi-devel] [question] deferred sense

[Douglas Gilbert]

Douglas Gilbert wrote:
> Tom Coughlan wrote:
> 
>> On Wed, 2005-01-05 at 11:44, James Bottomley wrote:
>>
>>> On Wed, 2005-01-05 at 16:23 +0100, Christoph Hellwig wrote:
>>>
>>>> On Wed, Jan 05, 2005 at 09:21:12AM -0600, Scott M. Ferris wrote:
>>>>
>>>>> To be more specific, there were some devices that would fail a command
>>>>> and return deferred sense.  The command didn't complete at the target,
>>>>> and the kernel wasn't retrying it because the sense was deferred
>>>>> rather than current.  For those devices, the translation produced the
>>>>> desired retry.
>>>>
>>>>
>>>> Do you remember these devices?  Might be worth adding a midlayer
>>>> blacklist entry for them.
>>>
>>>
>>> That's certainly possible ... although we'd need a lot more details.
>>> Any device that returns deferred sense for a current error is pretty
>>> badly broken according to the spec.
>>
>>
>>
>> If a current command returns deferred sense, the SCSI spec. requires
>> that the current command shall not have been executed [1]. So, if at
>> some point in the past the kernel did not retry a current command that
>> returned deferred sense, the iscsi folks would have forced the retry by
>> converting deferred sense to current sense. The scenario does not
>> require a device that is working incorrectly.
>>
>> The big flaw in what iscsi did is the case where the deferred sense
>> indicates a non-fatal error. In that case, iscsi converts it to current,
>> the mitlayer examines it and determines that it does not require a
>> retry. This causes the current command to complete to the application
>> even though it was not executed by the device.
>> It looks to me as though the 2.4 iscsi driver is susceptible to this. It
>> is probably not seen in practice because disk devices that return
>> non-fatal deferred sense are rare (it probably requires the PER bit set
>> in the error recovery mode page?). Anyone know for sure?
> 
> 
> Tom,
>  From my reading of SBC-2 (rev 16, 13 Nov 2004) deferred errors
> cannot be turned off by a mode page. The VERIFY and
> WRITE AND VERIFY commands can be used to make sure blocks get
> to the media (or not) without deferred errors. There are also
> the "Force Unit Access" (FUA and FUA_NV) bits in the READ and
> WRITE commands (but not the 6 byte variants).
> 
> The t10 folks have added the idea of non-volatile cache in
> a disk (more likely a RAID) which further complicates things.
> Now a deferred error could theoretically span a power cycle!
> 
> The PER bit in the Read-Write Error Recovery mode page (SBC-2
> rev 16 section 6.3.4) controls whether RECOVERED ERRORs are
> reported or not. Also if the ARRE bit (for reads) and/or the
> AWRE bit (for writes) are set in the same mode
> page, the offending block will be remapped (whether a recovered
> error is reported on not). If PER is 0 then RECOVERED ERRORs
> are not reported. [In any case the Error Counter log pages
> should reflect the problems (and perhaps the "grown" defect
> list) so smartmontools may be of use.]
> 
>> [1] If the task terminates with CHECK CONDITION status and the sense
>> data describes a deferred error the command for the terminated task
>> shall not have been processed. (SPC-3) 
> 
> 
> I fear my tinkering with sense descriptor data in
> the mid level may have tripped up on this case (if
> it wasn't already broken), that is: a deferred error
> report must cause the current command to be retried,
> even if the deferred error was reporting a
> recovered error. This case can only arise when PER=1.

Reviewing the error paths in lk 2.6 for deferred errors
indicates that the effected (innocent) command will be
retried in scsi_softirq() unless:
   - the retry count is exceeded (always true for sg usage)
   - or blk_noretry_request() returns true

Does fast path code want to (or know how to) process
deferred errors?
Even when deferred errors are retried this is done
silently unless an appropriate logging_level happens
to be set at the time (unlikely).

Doug Gilbert