new scsi sense handling

new scsi sense handling

[Boaz Harrosh]

There are 3 usages of sense handling in drivers

1. sense is available in driver internal structure and is mem-copied to upper level
2. A CHECK_CONDITION status was returned and the driver uses the scsi_eh_prep_cmnd()
   for a REQUEST_SENSE invocation to the target. Then returning the sense in 
   scsi_eh_return_cmnd(). A variation on this is when the driver does nothing the queue
   is frozen an the scsi watchdog timer does the above.
3. The underline host adapter does the REQUEST_SENSE and a pre-allocated and DMA mapped
   sense buffer receives the sense information from HW.

Now since all IO requests come with a sense set at request->sense the buffer at 
scsi_cmnd->sense_buffer can go away, and case 1 above can copy it's bits directly
into it.
Inspection of all users of blk_execute_rq_nowait() shows that all users setup a sense
buffer, but submitted code puts a BUG_ON to make sure new code does not break that.

for cases 2 and 3 above two new members in scsi host template will tell scsi-midlayer
what the driver wants to do.

.sense_buffsize  - will instruct a mempool_t to be allocated per host with sense buffers 
                   of ".sense_buffsize" size.

.pre_allocate_sense - If set tells the midlayer to pre-allocate a sense buffer for each
                      command. When set the code will behave like today. If the sense 
                      allocation fails the command allocation also fails. (With one extra
                      sense at mempool)
                      If .pre_allocate_sense is not set, then mempool_t is resized to
                      accommodate one reserved buffer for each target. So a call to
                      scsi_eh_prep_cmnd() will not fail to allocate a sense buffer even
                      in low memory condition.

Case 1 drivers call - scsi_eh_cpy_sense() - to transfer their sense information to upper
  layer. The drivers are completely abstracted from any future changes at the scsi and block
  sense handling. 

Case 3 drivers call the new scsi_make_sense()/scsi_return_sense() API to retrieve
  a pointer to an extra DMAable sense buffer/return it to free store. scsi_return_sense()
  will call scsi_eh_cpy_sense() to set new sense information into upper layer before
  freeing the buffer back to the mempool. If the driver properly set the .pre_allocate_sense
  flag & .sense_buffsize at host template then these calls are guaranteed to succeed.

Case 2 drivers continue to call scsi_eh_prep_cmnd()/scsi_eh_restore_cmnd() to reuse the
  failing command for REQUEST_SENSE. Inside these functions scsi_error.c will call above
  scsi_make_sense()/scsi_return_sense() for the actual sense buffer allocation. In this 
  case there is a guarantied, one reserved buffer per scsi_device. (.pre_allocate_sense not set).
  Drivers need to set proper size at .sense_buffsize in host template, for this to work.

Upper layer and ULDs that need to inspect the sense information can get to it
  using the scsi_sense() accessor.

Submitted for Review and comments (RFC) a patcset for that effect.

- first patch will introduce the new API implemented over existing members:
	scsi_eh: Define  new API for sense handling

- first group of patches are the case 1 drivers:
	scsi-drivers: Move to new sense API. The Trevial case
	scsi-drivers: more drivers use new scsi_eh_cpy_sense()
	firewire & ieee1394: Simple convert to new scsi_eh_cpy_sense.
	dpt_i2o: Use new scsi_eh_cpy_sense()
	gdth: Use of scsi_eh API and sense accessors
	qla2xxx: convert to new scsi_eh_cpy_sense API
	isd200, transport: Use scsi_eh_cpy_sense API
	ultrastor: Use scsi_eh_cpy_sense() API
	usb/microtek: No special handling for REQUEST_SENSE command please

- second group are case 2 drivers:
	libata: Use scsi_eh API for REQUEST_SENSE invocation
	53c700: Use scsi_eh API for REQUEST_SENSE invocation
	aic7xxx_old: Use scsi_eh API for REQUEST_SENSE invocation
	dc395x: Use scsi_eh API for REQUEST_SENSE invocation
	tmscsim: Use scsi_eh API for REQUEST_SENSE invocation
	Add .sense_buffsize to drivers that use scsi_eh_prep_cmnd

- third group are case 3:
	BusLogic: Use of scsi_make_sense() API for DMA-able sense buffer
	eata: Use of scsi_make_sense() API for DMA-able sense buffer
	initio: Use of scsi_make_sense() API for DMA-able sense buffer
	u14-34f: Use of scsi_make_sense() API for DMA-able sense buffer

- Then, some patches for the upper layer:
	scsi_tgt: use of sense accessors
	scsi upper layer use of sense accessors

- And finally, block and scsi-midlayer switch to new handling.
	block: Minor changes to sense handling
	scsi: New sense handling

These patches are over latest scsi-misc. They are completely *untested* and are for discussion
only. I will debug and update in the near future but only if it is accepted in principle
first.

Lots of maintainers' ACKs are needed and a good deal of testing time in mm so I think this 
is only 2.6.26 at the earliest.

Comment away ;)
Volunteers for testing needed!

Boaz

Re: new scsi sense handling

[Luben Tuikov]

--- On Mon, 2/4/08, Boaz Harrosh <bharrosh@panasas.com> wrote:
> There are 3 usages of sense handling in drivers
> 
> 1. sense is available in driver internal structure and is
> mem-copied to upper level
> 2. A CHECK_CONDITION status was returned and the driver
> uses the scsi_eh_prep_cmnd()
>    for a REQUEST_SENSE invocation to the target. Then
> returning the sense in 
>    scsi_eh_return_cmnd(). A variation on this is when the
> driver does nothing the queue
>    is frozen an the scsi watchdog timer does the above.
> 3. The underline host adapter does the REQUEST_SENSE and a
> pre-allocated and DMA mapped
>    sense buffer receives the sense information from HW.

Many years ago when "ACA" had a constructive meaning,
so did "Autosense".  Then about 5 years ago, "Autosense"
disappeared completely since it became the de facto
implementation of the then SCSI Execute Command "RPC",
now just SCSI Execute Command procedure call.

At that point in time, the SCSI mid-layer decided
to embrace this model and give the LLDD a scsi command
structure which included the sense data buffer to
a size that the SCSI mid-layer was interested in,
at the moment 96 bytes, macro defined in
include/scsi/scsi_cmnd.h.

The concept of "Autosense" was off-loaded to LLDD
to emulate it if the specific target device to
which the command was issued, didn't supply the
sense data on CHECK CONDITION, and more so
relevant to target devices which implemented
queuing, thus the ACA.

And the mid-layer would consider extracting
the sense data via REQUEST SENSE command
as a _special case_ if the LLDD/transport layer
didn't implement the "autosense" model.

Your model is a regression of this as far
as sense data is concerned.  Imagine if
your model had existed in the SCSI mid-layer
and now someone would be submitting a series
of patches a la the far simpler and more
straightforward model, incidently currently
in place.

> Now since all IO requests come with a sense set at
> request->sense the buffer at 
> scsi_cmnd->sense_buffer can go away, and case 1 above
> can copy it's bits directly
> into it.

That's ok.

> for cases 2 and 3 above two new members in scsi host
> template will tell scsi-midlayer
> what the driver wants to do.

This should be shunned upon.  The LLDD/transport layer
should provide sense to the SCSI mid-layer possibly
by reissuing _its own_ SCSI command block representation
to extract sense directly into the req sense buffer,
so when the mid-layer SCSI command structure completes,
scsi_done(), the sense data is already present and
status is CHECK CONDITION.

(For example, a well written SATL would do this
for ATAPI devices which completed the command with
a sense key in the Error register different than 0.)

> .sense_buffsize  - will instruct a mempool_t to be
> allocated per host with sense buffers 
>                    of ".sense_buffsize" size.
> 
> .pre_allocate_sense - If set tells the midlayer to
> pre-allocate a sense buffer for each
>                       command. When set the code will
> behave like today. If the sense 
>                       allocation fails the command
> allocation also fails. (With one extra
>                       sense at mempool)
>                       If .pre_allocate_sense is not set,
> then mempool_t is resized to
>                       accommodate one reserved buffer for
> each target. So a call to
>                       scsi_eh_prep_cmnd() will not fail to
> allocate a sense buffer even
>                       in low memory condition.
> 
> Case 1 drivers call - scsi_eh_cpy_sense() - to transfer
> their sense information to upper
>   layer. The drivers are completely abstracted from any
> future changes at the scsi and block
>   sense handling. 
> 
> Case 3 drivers call the new
> scsi_make_sense()/scsi_return_sense() API to retrieve
>   a pointer to an extra DMAable sense buffer/return it to
> free store. scsi_return_sense()
>   will call scsi_eh_cpy_sense() to set new sense
> information into upper layer before
>   freeing the buffer back to the mempool. If the driver
> properly set the .pre_allocate_sense
>   flag & .sense_buffsize at host template then these
> calls are guaranteed to succeed.
> 
> Case 2 drivers continue to call
> scsi_eh_prep_cmnd()/scsi_eh_restore_cmnd() to reuse the
>   failing command for REQUEST_SENSE. Inside these functions
> scsi_error.c will call above
>   scsi_make_sense()/scsi_return_sense() for the actual
> sense buffer allocation. In this 
>   case there is a guarantied, one reserved buffer per
> scsi_device. (.pre_allocate_sense not set).
>   Drivers need to set proper size at .sense_buffsize in
> host template, for this to work.

Absolutely too complicated for such a trivial thing
as supplying sense data at scsi_done() time to
the SCSI mid-layer (see previous points I wrote above).
Too much complication for a such a small and trivial
concept.

Would it be the case that you're trying to accomplish
or accommodate a strange target behaviour?  If so
this should be done at the LLDD/transport protocol
layer, and the SCSI mid-layer shouldn't have to
see a scsi command complete with CHECK CONDITION
set without a valid sense data in the sense buffer.

It's hard to see the advantage of this "new scsi sense
handling" patches.

    Luben

Re: new scsi sense handling

[FUJITA Tomonori]

On Mon, 4 Feb 2008 18:39:22 -0800 (PST)
Luben Tuikov <ltuikov@yahoo.com> wrote:

> --- On Mon, 2/4/08, Boaz Harrosh <bharrosh@panasas.com> wrote:
> > There are 3 usages of sense handling in drivers
> > 
> > 1. sense is available in driver internal structure and is
> > mem-copied to upper level
> > 2. A CHECK_CONDITION status was returned and the driver
> > uses the scsi_eh_prep_cmnd()
> >    for a REQUEST_SENSE invocation to the target. Then
> > returning the sense in 
> >    scsi_eh_return_cmnd(). A variation on this is when the
> > driver does nothing the queue
> >    is frozen an the scsi watchdog timer does the above.
> > 3. The underline host adapter does the REQUEST_SENSE and a
> > pre-allocated and DMA mapped
> >    sense buffer receives the sense information from HW.
> 
> Many years ago when "ACA" had a constructive meaning,
> so did "Autosense".  Then about 5 years ago, "Autosense"
> disappeared completely since it became the de facto
> implementation of the then SCSI Execute Command "RPC",
> now just SCSI Execute Command procedure call.
> 
> At that point in time, the SCSI mid-layer decided
> to embrace this model and give the LLDD a scsi command
> structure which included the sense data buffer to
> a size that the SCSI mid-layer was interested in,
> at the moment 96 bytes, macro defined in
> include/scsi/scsi_cmnd.h.
> 
> The concept of "Autosense" was off-loaded to LLDD
> to emulate it if the specific target device to
> which the command was issued, didn't supply the
> sense data on CHECK CONDITION, and more so
> relevant to target devices which implemented
> queuing, thus the ACA.
> 
> And the mid-layer would consider extracting
> the sense data via REQUEST SENSE command
> as a _special case_ if the LLDD/transport layer
> didn't implement the "autosense" model.

Only SPI and USB?

The most of LLDs using the transport protocol that we care about today
uses sense buffer in their own internal structure.

I think that the issue to solve to kill scsi_cmnd:sense_buffer is how
to share (or export) such sense buffer with the scsi mid-layer.

For the old transport protocols, we could do something that James said
in this thread to to kill scsi_cmnd:sense_buffer.

Re: new scsi sense handling

[Luben Tuikov]

--- On Tue, 2/5/08, FUJITA Tomonori <tomof@acm.org> wrote:
> On Mon, 4 Feb 2008 18:39:22 -0800 (PST)
> Luben Tuikov <ltuikov@yahoo.com> wrote:
> 
> > --- On Mon, 2/4/08, Boaz Harrosh
> <bharrosh@panasas.com> wrote:
> > > There are 3 usages of sense handling in drivers
> > > 
> > > 1. sense is available in driver internal
> structure and is
> > > mem-copied to upper level
> > > 2. A CHECK_CONDITION status was returned and the
> driver
> > > uses the scsi_eh_prep_cmnd()
> > >    for a REQUEST_SENSE invocation to the target.
> Then
> > > returning the sense in 
> > >    scsi_eh_return_cmnd(). A variation on this is
> when the
> > > driver does nothing the queue
> > >    is frozen an the scsi watchdog timer does the
> above.
> > > 3. The underline host adapter does the
> REQUEST_SENSE and a
> > > pre-allocated and DMA mapped
> > >    sense buffer receives the sense information
> from HW.
> > 
> > Many years ago when "ACA" had a constructive
> meaning,
> > so did "Autosense".  Then about 5 years ago,
> "Autosense"
> > disappeared completely since it became the de facto
> > implementation of the then SCSI Execute Command
> "RPC",
> > now just SCSI Execute Command procedure call.
> > 
> > At that point in time, the SCSI mid-layer decided
> > to embrace this model and give the LLDD a scsi command
> > structure which included the sense data buffer to
> > a size that the SCSI mid-layer was interested in,
> > at the moment 96 bytes, macro defined in
> > include/scsi/scsi_cmnd.h.
> > 
> > The concept of "Autosense" was off-loaded to
> LLDD
> > to emulate it if the specific target device to
> > which the command was issued, didn't supply the
> > sense data on CHECK CONDITION, and more so
> > relevant to target devices which implemented
> > queuing, thus the ACA.
> > 
> > And the mid-layer would consider extracting
> > the sense data via REQUEST SENSE command
> > as a _special case_ if the LLDD/transport layer
> > didn't implement the "autosense" model.
> 
> Only SPI and USB?

I don't understand this question.

> 
> The most of LLDs using the transport protocol that we care
> about today
> uses sense buffer in their own internal structure.

Yes.

> 
> I think that the issue to solve to kill
> scsi_cmnd:sense_buffer is how
> to share (or export) such sense buffer with the scsi
> mid-layer.

And therein lies the problem.  Sense data is SCSI specific,
it should be left to SCSI, unless of course you can
stipulate that _all_ block devices return sense data.
If that's not the case and you move it to the block
layer, then you get a whole bunch of other problems,
like does this device want/use it, should we allocate
it, etc. OTOH, if that _is_ the case, then you don't
have to worry about this and the model is pretty
much as the SCSI mid-layer has it, i.e. sense buffer
always present.  So I guess the question is, can
you stipulate that _all_ block devices return sense data?

    Luben

Re: new scsi sense handling

[FUJITA Tomonori]

On Tue, 5 Feb 2008 11:43:58 -0800 (PST)
Luben Tuikov <ltuikov@yahoo.com> wrote:

> --- On Tue, 2/5/08, FUJITA Tomonori <tomof@acm.org> wrote:
> > On Mon, 4 Feb 2008 18:39:22 -0800 (PST)
> > Luben Tuikov <ltuikov@yahoo.com> wrote:
> > 
> > > --- On Mon, 2/4/08, Boaz Harrosh
> > <bharrosh@panasas.com> wrote:
> > > > There are 3 usages of sense handling in drivers
> > > > 
> > > > 1. sense is available in driver internal
> > structure and is
> > > > mem-copied to upper level
> > > > 2. A CHECK_CONDITION status was returned and the
> > driver
> > > > uses the scsi_eh_prep_cmnd()
> > > >    for a REQUEST_SENSE invocation to the target.
> > Then
> > > > returning the sense in 
> > > >    scsi_eh_return_cmnd(). A variation on this is
> > when the
> > > > driver does nothing the queue
> > > >    is frozen an the scsi watchdog timer does the
> > above.
> > > > 3. The underline host adapter does the
> > REQUEST_SENSE and a
> > > > pre-allocated and DMA mapped
> > > >    sense buffer receives the sense information
> > from HW.
> > > 
> > > Many years ago when "ACA" had a constructive
> > meaning,
> > > so did "Autosense".  Then about 5 years ago,
> > "Autosense"
> > > disappeared completely since it became the de facto
> > > implementation of the then SCSI Execute Command
> > "RPC",
> > > now just SCSI Execute Command procedure call.
> > > 
> > > At that point in time, the SCSI mid-layer decided
> > > to embrace this model and give the LLDD a scsi command
> > > structure which included the sense data buffer to
> > > a size that the SCSI mid-layer was interested in,
> > > at the moment 96 bytes, macro defined in
> > > include/scsi/scsi_cmnd.h.
> > > 
> > > The concept of "Autosense" was off-loaded to
> > LLDD
> > > to emulate it if the specific target device to
> > > which the command was issued, didn't supply the
> > > sense data on CHECK CONDITION, and more so
> > > relevant to target devices which implemented
> > > queuing, thus the ACA.
> > > 
> > > And the mid-layer would consider extracting
> > > the sense data via REQUEST SENSE command
> > > as a _special case_ if the LLDD/transport layer
> > > didn't implement the "autosense" model.
> > 
> > Only SPI and USB?
> 
> I don't understand this question.

I meant, 'what transport protocols are categorized into the transport
protocol that doesn't implement the "autosense" model?'


> > The most of LLDs using the transport protocol that we care
> > about today
> > uses sense buffer in their own internal structure.
> 
> Yes.
> 
> > 
> > I think that the issue to solve to kill
> > scsi_cmnd:sense_buffer is how
> > to share (or export) such sense buffer with the scsi
> > mid-layer.
> 
> And therein lies the problem.  Sense data is SCSI specific,
> it should be left to SCSI, unless of course you can
> stipulate that _all_ block devices return sense data.

Yeah, sense data is SCSI specific and it should be left to SCSI. But
I'm not sure we need to stipulate that _all_ block devices return
sense data. Today the block layer users (sg, bsg, etc) use it only
when it's appropriate (or only if they want to use it).


> If that's not the case and you move it to the block
> layer, then you get a whole bunch of other problems,
> like does this device want/use it, should we allocate
> it, etc. OTOH, if that _is_ the case, then you don't
> have to worry about this and the model is pretty
> much as the SCSI mid-layer has it, i.e. sense buffer
> always present.  So I guess the question is, can
> you stipulate that _all_ block devices return sense data?

Re: new scsi sense handling

[Luben Tuikov]

--- On Tue, 2/5/08, FUJITA Tomonori <fujita.tomonori@lab.ntt.co.jp> wrote:
> > --- On Tue, 2/5/08, FUJITA Tomonori
> <tomof@acm.org> wrote:
> > > On Mon, 4 Feb 2008 18:39:22 -0800 (PST)
> > > Luben Tuikov <ltuikov@yahoo.com> wrote:
> > > 
> > > > --- On Mon, 2/4/08, Boaz Harrosh
> > > <bharrosh@panasas.com> wrote:
> > > > > There are 3 usages of sense handling in
> drivers
> > > > > 
> > > > > 1. sense is available in driver
> internal
> > > structure and is
> > > > > mem-copied to upper level
> > > > > 2. A CHECK_CONDITION status was
> returned and the
> > > driver
> > > > > uses the scsi_eh_prep_cmnd()
> > > > >    for a REQUEST_SENSE invocation to
> the target.
> > > Then
> > > > > returning the sense in 
> > > > >    scsi_eh_return_cmnd(). A variation
> on this is
> > > when the
> > > > > driver does nothing the queue
> > > > >    is frozen an the scsi watchdog timer
> does the
> > > above.
> > > > > 3. The underline host adapter does the
> > > REQUEST_SENSE and a
> > > > > pre-allocated and DMA mapped
> > > > >    sense buffer receives the sense
> information
> > > from HW.
> > > > 
> > > > Many years ago when "ACA" had a
> constructive
> > > meaning,
> > > > so did "Autosense".  Then about 5
> years ago,
> > > "Autosense"
> > > > disappeared completely since it became the
> de facto
> > > > implementation of the then SCSI Execute
> Command
> > > "RPC",
> > > > now just SCSI Execute Command procedure
> call.
> > > > 
> > > > At that point in time, the SCSI mid-layer
> decided
> > > > to embrace this model and give the LLDD a
> scsi command
> > > > structure which included the sense data
> buffer to
> > > > a size that the SCSI mid-layer was
> interested in,
> > > > at the moment 96 bytes, macro defined in
> > > > include/scsi/scsi_cmnd.h.
> > > > 
> > > > The concept of "Autosense" was
> off-loaded to
> > > LLDD
> > > > to emulate it if the specific target device
> to
> > > > which the command was issued, didn't
> supply the
> > > > sense data on CHECK CONDITION, and more so
> > > > relevant to target devices which implemented
> > > > queuing, thus the ACA.
> > > > 
> > > > And the mid-layer would consider extracting
> > > > the sense data via REQUEST SENSE command
> > > > as a _special case_ if the LLDD/transport
> layer
> > > > didn't implement the
> "autosense" model.
> > > 
> > > Only SPI and USB?
> > 
> > I don't understand this question.
> 
> I meant, 'what transport protocols are categorized into
> the transport
> protocol that doesn't implement the
> "autosense" model?'

If any transport protocol conforms to SAM, it supports it.
Either emulated in the transport itself or supported
by the device (target) itself.  But ideally, the SCSI
mid-layer shouldn't have to get a CHECK CONDITION and
then turn around and send REQUEST SENSE, due to the
atomicity (per command) of the sense data, especially
if the target supports queuing.  There used to be
a mechanism to support this in SAM but is now obsolete.

> 
> 
> > > The most of LLDs using the transport protocol
> that we care
> > > about today
> > > uses sense buffer in their own internal
> structure.
> > 
> > Yes.
> > 
> > > 
> > > I think that the issue to solve to kill
> > > scsi_cmnd:sense_buffer is how
> > > to share (or export) such sense buffer with the
> scsi
> > > mid-layer.
> > 
> > And therein lies the problem.  Sense data is SCSI
> specific,
> > it should be left to SCSI, unless of course you can
> > stipulate that _all_ block devices return sense data.
> 
> Yeah, sense data is SCSI specific and it should be left to
> SCSI. But
> I'm not sure we need to stipulate that _all_ block
> devices return
> sense data. Today the block layer users (sg, bsg, etc) use
> it only
> when it's appropriate (or only if they want to use it).

I agree.

   Luben