Regarding ordered-tag support.

Regarding ordered-tag support.

[Tejun Heo]

Hello, James.

Now that new ordered implementation is in the mainline, we can properly 
use ordered-tag during barrier sequence.  As noted in the barrier doc, 
the problem with the current SCSI midlayer is that scsi_request_fn() is 
not atomic w.r.t. q->queuelock, so even if ordered-tag requests leave 
request queue in the correct order, they can be reordered while they are 
being issued by SCSI midlayer.

There can be two solutions to this problem.

1. Rewrite scsi_request_fn() and scsi_dispatch_cmd() such that 
q->queuelock is not released during command issue.  This implies 
grabbing shost->host_lock while holding q->queuelock.

2. Don't allow concurrent command issuing by marking SCSI device during 
command issuing (e.g. sdev->issue_in_progress)

I prefer #1 as it's more straight-forward.  To implement #1, 
scsi_request_fn() should be reorganized quite a bit, but, if you 
remember, the scsi_reqfn reimplementation patch I wrote quite a while 
ago suites this pretty well.  What do you think?

-- 
tejun

Re: Regarding ordered-tag support.

[James Bottomley]

On Wed, 2006-02-08 at 15:40 +0900, Tejun Heo wrote:
> Now that new ordered implementation is in the mainline, we can properly 
> use ordered-tag during barrier sequence.  As noted in the barrier doc, 
> the problem with the current SCSI midlayer is that scsi_request_fn() is 
> not atomic w.r.t. q->queuelock, so even if ordered-tag requests leave 
> request queue in the correct order, they can be reordered while they are 
> being issued by SCSI midlayer.

Unfortunately, though, this is only half the problem.

The other half is busy/queue_full processing and error recovery

For the first: busy/queue_full processing, the issue is that when we
send out a command, we could get a BUSY or QUEUE_FULL return which comes
back to us via IRQ context.  Unfortunately, we could have multiple
commands that do this and a command will be accepted as soon as the busy
condition alleviates, so we could see say two commands go down in order,
the first one will get BUSY, the second is accepted and only then do we
get the IRQ that says BUSY to the first ... now we have out of order
execution.

For the second:  Depending on the mode page (the QErr bit of the
queueing page), most devices fail only a single command.  We can set the
QErr bit to return every command after the failing one (thus ensuring
execution order), but the error handler would have to take them all back
and resort them for submission.

I think the first problem has to be solved before we can turn on ordered
tag based barriers.  I'm ambivalent on the second; we could probably
mark ordered tag based barriers as "caveat emptor" while we work on the
EH problem in parallel.

James

Re: Regarding ordered-tag support.

[Tejun]

James Bottomley wrote:
> On Wed, 2006-02-08 at 15:40 +0900, Tejun Heo wrote:
> 
>>Now that new ordered implementation is in the mainline, we can properly 
>>use ordered-tag during barrier sequence.  As noted in the barrier doc, 
>>the problem with the current SCSI midlayer is that scsi_request_fn() is 
>>not atomic w.r.t. q->queuelock, so even if ordered-tag requests leave 
>>request queue in the correct order, they can be reordered while they are 
>>being issued by SCSI midlayer.
> 
> 
> Unfortunately, though, this is only half the problem.
> 
> The other half is busy/queue_full processing and error recovery
> 
> For the first: busy/queue_full processing, the issue is that when we
> send out a command, we could get a BUSY or QUEUE_FULL return which comes
> back to us via IRQ context.  Unfortunately, we could have multiple
> commands that do this and a command will be accepted as soon as the busy
> condition alleviates, so we could see say two commands go down in order,
> the first one will get BUSY, the second is accepted and only then do we
> get the IRQ that says BUSY to the first ... now we have out of order
> execution.

Oh... I see.  How many drivers do that?  I can't think of good reasons
to report BUSY via IRQ for simpler transports (SATA/SPI).  Maybe enable
ordered-tag selectively?

> 
> For the second:  Depending on the mode page (the QErr bit of the
> queueing page), most devices fail only a single command.  We can set the
> QErr bit to return every command after the failing one (thus ensuring
> execution order), but the error handler would have to take them all back
> and resort them for submission.

Actually blk layer will do the sorting part (this is what req->ordcolor
is for).  Block drivers are only required to not successfully complete
later requests while retrying earlier ones, so setting QErr and retrying
all on-the-fly requests should do it (no EH code change).

> I think the first problem has to be solved before we can turn on ordered
> tag based barriers.  I'm ambivalent on the second; we could probably
> mark ordered tag based barriers as "caveat emptor" while we work on the
> EH problem in parallel.

-- 
tejun

Re: Regarding ordered-tag support.

[James Smart]

>> For the first: busy/queue_full processing, the issue is that when we
>> send out a command, we could get a BUSY or QUEUE_FULL return which comes
>> back to us via IRQ context.  Unfortunately, we could have multiple
>> commands that do this and a command will be accepted as soon as the busy
>> condition alleviates, so we could see say two commands go down in order,
>> the first one will get BUSY, the second is accepted and only then do we
>> get the IRQ that says BUSY to the first ... now we have out of order
>> execution.
> 
> Oh... I see.  How many drivers do that?  I can't think of good reasons
> to report BUSY via IRQ for simpler transports (SATA/SPI).  Maybe enable
> ordered-tag selectively?

This isn't a driver thing - this is a SCSI device thing.  The disk array
is temporarily out of resources so it BUSY's, or QUEUE_FULL's. But the
next i/o may not encounter it.


> 
>> For the second:  Depending on the mode page (the QErr bit of the
>> queueing page), most devices fail only a single command.  We can set the
>> QErr bit to return every command after the failing one (thus ensuring
>> execution order), but the error handler would have to take them all back
>> and resort them for submission.
> 
> Actually blk layer will do the sorting part (this is what req->ordcolor
> is for).  Block drivers are only required to not successfully complete
> later requests while retrying earlier ones, so setting QErr and retrying
> all on-the-fly requests should do it (no EH code change).

You're making the assumption that you can set QErr... It's under device control.

-- james s

Re: Regarding ordered-tag support.

[Tejun]

Hello,

James Smart wrote:
>>>For the first: busy/queue_full processing, the issue is that when we
>>>send out a command, we could get a BUSY or QUEUE_FULL return which comes
>>>back to us via IRQ context.  Unfortunately, we could have multiple
>>>commands that do this and a command will be accepted as soon as the busy
>>>condition alleviates, so we could see say two commands go down in order,
>>>the first one will get BUSY, the second is accepted and only then do we
>>>get the IRQ that says BUSY to the first ... now we have out of order
>>>execution.
>>
>>Oh... I see.  How many drivers do that?  I can't think of good reasons
>>to report BUSY via IRQ for simpler transports (SATA/SPI).  Maybe enable
>>ordered-tag selectively?
> 
> This isn't a driver thing - this is a SCSI device thing.  The disk array
> is temporarily out of resources so it BUSY's, or QUEUE_FULL's. But the
> next i/o may not encounter it.

I intentionally wrote 'driver' because if a SCSI device determines that 
it's busy, it would report via CHECK CONDITION.  Depending on QErr, the 
whole task set will be terminated, and such case falls into EH 
requeueing case (would require changes in scsi_softirq though).

So, I thought this case was driver/controller thing where they report 
busy condition after issue for a command while accepting later commands.

>>>For the second:  Depending on the mode page (the QErr bit of the
>>>queueing page), most devices fail only a single command.  We can set the
>>>QErr bit to return every command after the failing one (thus ensuring
>>>execution order), but the error handler would have to take them all back
>>>and resort them for submission.
>>
>>Actually blk layer will do the sorting part (this is what req->ordcolor
>>is for).  Block drivers are only required to not successfully complete
>>later requests while retrying earlier ones, so setting QErr and retrying
>>all on-the-fly requests should do it (no EH code change).
> 
> You're making the assumption that you can set QErr... It's under device control.
> 

Was it ro field?  Didn't know that.  I will check it tomorrow.  If a 
device doesn't abort whole taskset, we just can't use ordered-tag for 
barriers.

Thanks for your comment.

-- 
tejun

Re: Regarding ordered-tag support.

[James Bottomley]

On Thu, 2006-02-09 at 01:57 +0900, Tejun wrote:
> I intentionally wrote 'driver' because if a SCSI device determines that 
> it's busy, it would report via CHECK CONDITION.  Depending on QErr, the 
> whole task set will be terminated, and such case falls into EH 
> requeueing case (would require changes in scsi_softirq though).

No, it wouldn't.  BUSY is a SPI Message (deprecated on non-SPI
transports) and QUEUE_FULL is a status return.  The device is entitled
to think of the transaction as logically complete (and not impacting
error recovery) after either of these returns.

> > You're making the assumption that you can set QErr... It's under device control. 
> 
> Was it ro field?  Didn't know that.  I will check it tomorrow.  If a 
> device doesn't abort whole taskset, we just can't use ordered-tag for 
> barriers.

His point is that it's a field in the mode page of a device.  Some
devices only have read only mode pages ... still more may not act on
this field correctly even if set.  Basically it's opening a can of
worms.

James

Re: Regarding ordered-tag support.

[Tejun]

James Bottomley wrote:
> On Thu, 2006-02-09 at 01:57 +0900, Tejun wrote:
> 
>>I intentionally wrote 'driver' because if a SCSI device determines that 
>>it's busy, it would report via CHECK CONDITION.  Depending on QErr, the 
>>whole task set will be terminated, and such case falls into EH 
>>requeueing case (would require changes in scsi_softirq though).
> 
> No, it wouldn't.  BUSY is a SPI Message (deprecated on non-SPI
> transports) and QUEUE_FULL is a status return.  The device is entitled
> to think of the transaction as logically complete (and not impacting
> error recovery) after either of these returns.

I see.  I'll dig docs.

>>>You're making the assumption that you can set QErr... It's under device control. 
>>
>>Was it ro field?  Didn't know that.  I will check it tomorrow.  If a 
>>device doesn't abort whole taskset, we just can't use ordered-tag for 
>>barriers.
> 
> 
> His point is that it's a field in the mode page of a device.  Some
> devices only have read only mode pages ... still more may not act on
> this field correctly even if set.  Basically it's opening a can of
> worms.

Yeah, I kind of doubt whether hardware vendors would implement and test
all the options and features whene I read SCSI specs.  Considering how
many ATA drives have faulty firmware...

Anyways, it seems ordered-tag support should be left alone for the time
being.  Sad.  :-(

Thanks a lot.

-- 
tejun

Re: Regarding ordered-tag support.

[James Bottomley]

On Thu, 2006-02-09 at 02:18 +0900, Tejun wrote:
> Yeah, I kind of doubt whether hardware vendors would implement and test
> all the options and features whene I read SCSI specs.  Considering how
> many ATA drives have faulty firmware...
> 
> Anyways, it seems ordered-tag support should be left alone for the time
> being.  Sad.  :-(

Well ... I've always maintained that ordered queueing is not really a
good method of implementing barriers in TCQ environments ... primarily
because no-one else really does it, so we'd be uncovering all of the
bugs.

I suppose I'm a broken record, but I really think linked tasks are the
better way to enforce the required ordering guarantees for SCSI TCQ.
The problem being that this would present a slightly different API at
the block level (you have individual writes that are now ordered, not
wholesale barriers).

James

Re: Regarding ordered-tag support.

[Jeff Garzik]

James Bottomley wrote:
> I suppose I'm a broken record, but I really think linked tasks are the
> better way to enforce the required ordering guarantees for SCSI TCQ.
> The problem being that this would present a slightly different API at
> the block level (you have individual writes that are now ordered, not
> wholesale barriers).


I strongly agree, since fundamentally, linked tasks is what is really 
going on.

	Jeff

Re: Regarding ordered-tag support.

[Tejun Heo]

Jeff Garzik wrote:
> James Bottomley wrote:
> 
>> I suppose I'm a broken record, but I really think linked tasks are the
>> better way to enforce the required ordering guarantees for SCSI TCQ.
>> The problem being that this would present a slightly different API at
>> the block level (you have individual writes that are now ordered, not
>> wholesale barriers).
> 
> I strongly agree, since fundamentally, linked tasks is what is really 
> going on.

Hello, James.  Hello, Jeff.

Maybe I have misunderstood, but, AFAICS, linked task doesn't assure any 
ordering relation with respect to other tasks unlike ordered tag.  So, 
it doesn't make much difference compared to simple & stupid ordering by 
draining.  For a barrier to work, the following order should be kept.

pre-barrier writes -> cache flush -> barrier write (FUA) -> other writes

Issuing cache flush and barrier write with ordered tags accomplishes all 
the ordering requirements (if it works, that is.).  I can see linked 
task can be used for 'cache flush -> barrier write (FUA)' segment but 
that leaves us with two unhandled ordering requirements.

Another thing I'm curious about linked task is what advantages linked 
task has over simply issuing and completing the commands sequentially. 
Commands in a linked task has to be issued and completed sequentially 
anyway, so I don't really see the difference.

Thanks.

-- 
tejun

Re: Regarding ordered-tag support.

[James Bottomley]

On Thu, 2006-02-09 at 13:12 +0900, Tejun Heo wrote:
> Maybe I have misunderstood, but, AFAICS, linked task doesn't assure any 
> ordering relation with respect to other tasks unlike ordered tag.  So, 
> it doesn't make much difference compared to simple & stupid ordering by 
> draining.  For a barrier to work, the following order should be kept.

It's a different paradigm from barriers.  With a barrier, it's an
impassable object in the I/O stream (nothing overtakes it or falls
behind it).  With a link, there's a set of commands (the linked set, and
there may be more than one linked set of commands) that have to be
ordered within the link.  The point is that most of our problems with
barriers are figuring out how to prevent leaks.  The linked command
paradigm is much more forgiving in this regard.

> pre-barrier writes -> cache flush -> barrier write (FUA) -> other writes
> 
> Issuing cache flush and barrier write with ordered tags accomplishes all 
> the ordering requirements (if it works, that is.).  I can see linked 
> task can be used for 'cache flush -> barrier write (FUA)' segment but 
> that leaves us with two unhandled ordering requirements.

Right, that's why I said we'd have to change the block layer API to use
linked commands.

> Another thing I'm curious about linked task is what advantages linked 
> task has over simply issuing and completing the commands sequentially. 
> Commands in a linked task has to be issued and completed sequentially 
> anyway, so I don't really see the difference.

The fact that tasks can proceed in parallel, only the command components
individual linked tasks must be serialised. so it supports the TCQ
paradigm.

Really, also, linked tasks is exactly what a JFS wants because most of
the updates that would be linked are something like update log entry;
modify metadata (or data); remove log entry.

James

Re: Regarding ordered-tag support.

[Tejun Heo]

Hello, James.

James Bottomley wrote:
> 
> The fact that tasks can proceed in parallel, only the command components
> individual linked tasks must be serialised. so it supports the TCQ
> paradigm.
> 
> Really, also, linked tasks is exactly what a JFS wants because most of
> the updates that would be linked are something like update log entry;
> modify metadata (or data); remove log entry.
> 

I see your point, so it's not really a barrier anymore, but a sequence
of ordered writes with an intervening flush independent of other
unrelated writes.  Some random thoughts...

* All writes in the ordered sequence will have to be done sequentially,
but unrelated writes are not affected.  This can be overall gain.

* Can be implemented in block layer proper without using SCSI linked
task (probably with separate ->ordered_requests queue).  What would be
the advantage of SCSI linked task?

* As it's finer grained than wholesale barrier, FS layer will need to
pay more attention.

Thanks for the explanation.  :-)

-- 
tejun

Re: Regarding ordered-tag support.

[James Bottomley]

On Thu, 2006-02-09 at 01:10 +0900, Tejun wrote:
> Oh... I see.  How many drivers do that?  I can't think of good reasons
> to report BUSY via IRQ for simpler transports (SATA/SPI).  Maybe enable
> ordered-tag selectively?

All of them.  These are scsi *device* returns.  The problem is that even
if the driver is careful about this, we queue any status done for
softirq processing ...

> > 
> > For the second:  Depending on the mode page (the QErr bit of the
> > queueing page), most devices fail only a single command.  We can set the
> > QErr bit to return every command after the failing one (thus ensuring
> > execution order), but the error handler would have to take them all back
> > and resort them for submission.
> 
> Actually blk layer will do the sorting part (this is what req->ordcolor
> is for).  Block drivers are only required to not successfully complete
> later requests while retrying earlier ones, so setting QErr and retrying
> all on-the-fly requests should do it (no EH code change).

Right ... but at the moment, we don't do that.  The eh would have to be
changed to gather all the commands for the device (we do it per host)
and resubmit them in such a way that the block layer resorts them,
wouldn't we ... at the moment we queue each one as they're processed.

James

Re: Regarding ordered-tag support.

[James Smart]

James Bottomley wrote:
> On Wed, 2006-02-08 at 15:40 +0900, Tejun Heo wrote:
> ...<snip>...
> Unfortunately, though, this is only half the problem.
> 
> The other half is busy/queue_full processing and error recovery
> 
> For the first: busy/queue_full processing, the issue is that when we
> send out a command, we could get a BUSY or QUEUE_FULL return which comes
> back to us via IRQ context.  Unfortunately, we could have multiple
> commands that do this and a command will be accepted as soon as the busy
> condition alleviates, so we could see say two commands go down in order,
> the first one will get BUSY, the second is accepted and only then do we
> get the IRQ that says BUSY to the first ... now we have out of order
> execution.

James is also not disclosing other areas that may fail...

- How does multipathing affect this (maybe not dm, but others that exist).
- The implicit expectation of the LLDD and the hardware is that commands are
   placed on the wire in the same order given to queuecommand. However, no
   where is this mandated. In general, they do keep order. However, I've seen
   driver implementations that don't dispatch directly to hardware in
   queuecommand, allowing the intermediate steps to fail and perhaps disrupt
   ordering.
- Some adapters may actually be multipath/raid cards, thus affecting command
   delivery.
- SCSI Transports are not 100% reliable, affecting ordering. For FC,
   a single Frame corruption could cause a command frame to be dropped thus
   the next command received and executed out of order. You would not notice
   an issue until the scsi i/o timeout fires and aborts the i/o. The subsequent
   i/o may have already completed by that time. Some transports due have
   command sequence numbering to avoid this, but it's an optional feature.

-- james s

Re: Regarding ordered-tag support.

[James Bottomley]

On Wed, 2006-02-08 at 11:22 -0500, James Smart wrote:
> James is also not disclosing other areas that may fail...
> 
> - How does multipathing affect this (maybe not dm, but others that exist).

Hey, with block level multi-path that's now officially Someone Else's
Problem. (And it's also a problem for software RAID as well).

> - The implicit expectation of the LLDD and the hardware is that commands are
>    placed on the wire in the same order given to queuecommand. However, no
>    where is this mandated. In general, they do keep order. However, I've seen
>    driver implementations that don't dispatch directly to hardware in
>    queuecommand, allowing the intermediate steps to fail and perhaps disrupt
>    ordering.

OK ... this acceptance problem is pretty much identical to the
BUSY/QUEUE_FULL one.

> - Some adapters may actually be multipath/raid cards, thus affecting command
>    delivery.

This shouldn't be more of a problem than SCSI devices.  The issue here
is caching.  A raid controller with a non-battery backed cache may
decide to ack commands before they hit the medium.  However, as long as
they expose this in the caching mode page and support the FUA bit of 10
byte commands, we should be able to cope (and yes, I know there are
drivers that currently don't get this right).

> - SCSI Transports are not 100% reliable, affecting ordering. For FC,
>    a single Frame corruption could cause a command frame to be dropped thus
>    the next command received and executed out of order. You would not notice
>    an issue until the scsi i/o timeout fires and aborts the i/o. The subsequent
>    i/o may have already completed by that time. Some transports due have
>    command sequence numbering to avoid this, but it's an optional feature.

Yes, but again, this is identical to BUSY/QUEUE_FULL.  As long as you
can get status return from the command in-line you can still preserve
ordering.

James

Re: Regarding ordered-tag support.

[James Bottomley]

On Wed, 2006-02-08 at 14:50 -0600, James Bottomley wrote:
> > - How does multipathing affect this (maybe not dm, but others that exist).
> 
> Hey, with block level multi-path that's now officially Someone Else's
> Problem. (And it's also a problem for software RAID as well).

And actually (in the time honoured tradition of telling someone else
what they should be doing while blowing your own trumpet) could I just
point out that I think that it's impossible to enforce barriers in the
face of multi-path (because of the cross path synchronisation
requirements).  However, it should be quite easy to enforce linked
command ordering in these environments by the simple expedient of
tagging a link to a single path.  Even on path failure for partial link
completion, you simply resend the entire segment of link command set
that you don't know completed (because writes are idempotent).

James