bidirectional, long commands + OSD

bidirectional, long commands + OSD

[Douglas Gilbert]

James Bottomley wrote in thread:
   [PATCH] 2.6.2-rc2 - MPT Fusion driver 3.00.02 update

> On Tue, 2004-01-27 at 02:48, Douglas Gilbert wrote:
> 
>>So the sg (or SG_IO/SCSI_IOCTL_SEND_COMMAND ioctl) user controls
>>the data direction and they are free to get it wrong:-)
> 
> 
> Since we do have a mid layer library for this, perhaps we should set it
> in sg if the user leaves it apparently unset.

It is valid for vendor specific commands to have command lengths
other than those dictated by the top 3 bits of the opcode. The
variable length opcode (0x7f) breaks that convention as well.
The pass-through philosophy is (within reason) "you asked for
it ....".

>>** The Object Storage Device (OSD) commands use bidirectional
>>data transfers so perhaps we should think about catering for
>>that.
> 
> 
> Actually, a much more fundamental problem for OSD is going to be >16
> byte CDBs.  The mid and block layers today are coded with an internal 16
> byte array for the CDB.  That's not very scaleable to the hundreds of
> bytes that OSD needs.

Just did a check on recent drafts and SPC-3 doesn't have any
commands with lengths > 16 bytes but SBC-2 has:
    XDREAD(32)
    XDWRITE(32)
    XDWRITEREAD(32)
    XPWRITE(32)

> As far as bidirectional goes, we should be able to support that today
> providing there's just a *single* data buffer, which the OSD spec
> implies is the easiest way to support this (although I suspect most SCSI
> processor coding won't be expecting to see both a data in and a data out
> from a single command).

... and XDWRITEREAD (both 10 and 32 byte variants) is bidirectional.

> As I see it, OSD seems to require a different interface from the current
> bio layer, so there would be some type of object layer glue between the
> fs and the mid-layer, with the block layer relegated simply to queueing
> specials for the mid-layer, but since no-one's actually come forward
> with an implementation, that's just a guess.

Both Jens and Linus jumped on me when I suggested
similar heresy :-)

For anyone interested, an overview of the proposed Object
Storage Device (OSD) architecture can be found at:
http://www.t10.org/ftp/t10/drafts/osd/osd-r08.pdf
in section 4.2 .

Doug Gilbert

Re: bidirectional, long commands + OSD

[Liran Schour]

Douglas Gilbert wrote in thread:
   bidirectional, long commands + OSD


>>>** The Object Storage Device (OSD) commands use bidirectional
>>>data transfers so perhaps we should think about catering for
>>>that.
>>
>>
>> Actually, a much more fundamental problem for OSD is going to be >16
>> byte CDBs.  The mid and block layers today are coded with an internal 16
>> byte array for the CDB.  That's not very scaleable to the hundreds of
>> bytes that OSD needs.

We can allow the upper layer driver to set the cmd_len and the cdb pointer
to the
desired length, the default will stay 16 byte.

>Just did a check on recent drafts and SPC-3 doesn't have any
>commands with lengths > 16 bytes but SBC-2 has:
>    XDREAD(32)
>    XDWRITE(32)
>    XDWRITEREAD(32)
>    XPWRITE(32)

>> As far as bidirectional goes, we should be able to support that today
>> providing there's just a *single* data buffer, which the OSD spec
>> implies is the easiest way to support this (although I suspect most SCSI
>> processor coding won't be expecting to see both a data in and a data out
>> from a single command).

>... and XDWRITEREAD (both 10 and 32 byte variants) is bidirectional.

A single buffer for both data in and data out will force the user to
re-copy the buffer.
We can solve it by using a scatter-gather list. Each buffer will have its
own data
direction flag.

>> As I see it, OSD seems to require a different interface from the current
>> bio layer, so there would be some type of object layer glue between the
>> fs and the mid-layer, with the block layer relegated simply to queueing
>> specials for the mid-layer, but since no-one's actually come forward
>> with an implementation, that's just a guess.

>Both Jens and Linus jumped on me when I suggested
>similar heresy :-)

Intel's iscsi reference implementation includes an upper layer driver for
OSD, it
also includes a mini filesystem that works on top of it.
This implementation only demonstrate the need to solve this 3 issues: (the
implementation just work around them)
      1. Coupling of the scsi layer with the bio layer.
      2. Bidirectional support.
      3. Extended CDB support.

We will be happy to open this issues for a discussion.

- Liran

Re: bidirectional, long commands + OSD

[Liran Schour]

James Bottomley wrote in thread:
   bidirectional, long commands + OSD:

>On Thu, 2004-01-29 at 11:08, Liran Schour wrote:
>> A single buffer for both data in and data out will force the user to
>> re-copy the buffer.
>> We can solve it by using a scatter-gather list. Each buffer will have
its
>> own data
>> direction flag.

>This is not necessarily true.  Just because it's a single buffer doesn't
>mean that the read and write areas cannot be separated inside it.  The
>actual implementation in most drivers uses a single data pointer, so if
>the device starts with DATA IN and transitions to DATA OUT, you should
>get the output data beginning directly after the input data.

>I know the spec implies that you keep two data pointers (because the
>device is allowed to do the DATA IN<->DATA OUT transition many times),
>so possibly we can't get away with something simple as above, but
>implementing separate data pointers would require modifying every
>driver.

Most of the HBAs today can't support execution of OSD SCSI commands due to
the
extended CDB's length and the bidirectional data movement. But when OSD
technology
will catch a momentum we will see more and more vendors that will support
this features.
My point is that maybe we can keep the current interface while expanding it
to support
the needed features by the SCSI OSD commands.

>which OSD devices have you used?  Just ISCSI one's, I assume; what do
>they do with the data in/out transitions?

I have used only ISCSI and TCP/IP (non SCSI) OSD devices. In the devices
that
I have used the transitions of data where isolated - first DATA OUT then
DATA IN.
But an OSD device can do the following data transitions: DATA IN (user
data) ; DATA OUT
(set attributes data and get attributes list) ;  DATA IN (retrieved
attributes data).

- Liran

Re: bidirectional, long commands + OSD

[James Bottomley]

On Thu, 2004-01-29 at 08:07, Douglas Gilbert wrote:
> James Bottomley wrote in thread:
> > As I see it, OSD seems to require a different interface from the current
> > bio layer, so there would be some type of object layer glue between the
> > fs and the mid-layer, with the block layer relegated simply to queueing
> > specials for the mid-layer, but since no-one's actually come forward
> > with an implementation, that's just a guess.
> 
> Both Jens and Linus jumped on me when I suggested
> similar heresy :-)

Well ... I merely stated what it seems to require.  I didn't recommend
implementing it.

>From what I can tell, the OSD spec seems to be aimed at large clustered
filesystems.  The recommended mapping seems to be <multiple objects> <->
<one file>.  However, since different objects are distinct and currently
require distinct commands to access them, I believe using OSD for a
typical Linux filesystem would be a big loss because most files are
small and we can no-longer elevate the requests.  It would be
interesting to see if this loss could be offset by not having to do
indirect block lookups, but I have my doubts.

James

Re: bidirectional, long commands + OSD

[James Bottomley]

On Fri, 2004-01-30 at 05:39, Liran Schour wrote:
> Most of the HBAs today can't support execution of OSD SCSI commands due to
> the
> extended CDB's length and the bidirectional data movement. But when OSD
> technology
> will catch a momentum we will see more and more vendors that will support
> this features.
> My point is that maybe we can keep the current interface while expanding it
> to support
> the needed features by the SCSI OSD commands.

Actually, I'm sure they can.  Most HBA's operate using a scripted
firmware interface (like the LSI scripts or the Adaptec sequencer).  As
long as the scripts are updated, it's no trouble to support multiple
data pointers and large CDBs (even the 53c700, a 1990s era chip can do
this).  I also know seagate is working on an actual single disc that
supports OSD, so I expect even the closed firmware chip sets will
eventually be updated.

> I have used only ISCSI and TCP/IP (non SCSI) OSD devices. In the devices
> that
> I have used the transitions of data where isolated - first DATA OUT then
> DATA IN.
> But an OSD device can do the following data transitions: DATA IN (user
> data) ; DATA OUT
> (set attributes data and get attributes list) ;  DATA IN (retrieved
> attributes data).

Yes, that's what I feared.  The spec implies this as well.  Behaviour
like that requires two data pointers.

James