Re: bidirectional, long commands + OSD

Re: bidirectional, long commands + OSD

[Liran Schour]

James Bottomley wrote in thread:
   bidirectional, long commands + OSD:

>On Thu, 2004-01-29 at 11:08, Liran Schour wrote:
>> A single buffer for both data in and data out will force the user to
>> re-copy the buffer.
>> We can solve it by using a scatter-gather list. Each buffer will have
its
>> own data
>> direction flag.

>This is not necessarily true.  Just because it's a single buffer doesn't
>mean that the read and write areas cannot be separated inside it.  The
>actual implementation in most drivers uses a single data pointer, so if
>the device starts with DATA IN and transitions to DATA OUT, you should
>get the output data beginning directly after the input data.

>I know the spec implies that you keep two data pointers (because the
>device is allowed to do the DATA IN<->DATA OUT transition many times),
>so possibly we can't get away with something simple as above, but
>implementing separate data pointers would require modifying every
>driver.

Most of the HBAs today can't support execution of OSD SCSI commands due to
the
extended CDB's length and the bidirectional data movement. But when OSD
technology
will catch a momentum we will see more and more vendors that will support
this features.
My point is that maybe we can keep the current interface while expanding it
to support
the needed features by the SCSI OSD commands.

>which OSD devices have you used?  Just ISCSI one's, I assume; what do
>they do with the data in/out transitions?

I have used only ISCSI and TCP/IP (non SCSI) OSD devices. In the devices
that
I have used the transitions of data where isolated - first DATA OUT then
DATA IN.
But an OSD device can do the following data transitions: DATA IN (user
data) ; DATA OUT
(set attributes data and get attributes list) ;  DATA IN (retrieved
attributes data).

- Liran

Re: bidirectional, long commands + OSD

[James Bottomley]

On Fri, 2004-01-30 at 05:39, Liran Schour wrote:
> Most of the HBAs today can't support execution of OSD SCSI commands due to
> the
> extended CDB's length and the bidirectional data movement. But when OSD
> technology
> will catch a momentum we will see more and more vendors that will support
> this features.
> My point is that maybe we can keep the current interface while expanding it
> to support
> the needed features by the SCSI OSD commands.

Actually, I'm sure they can.  Most HBA's operate using a scripted
firmware interface (like the LSI scripts or the Adaptec sequencer).  As
long as the scripts are updated, it's no trouble to support multiple
data pointers and large CDBs (even the 53c700, a 1990s era chip can do
this).  I also know seagate is working on an actual single disc that
supports OSD, so I expect even the closed firmware chip sets will
eventually be updated.

> I have used only ISCSI and TCP/IP (non SCSI) OSD devices. In the devices
> that
> I have used the transitions of data where isolated - first DATA OUT then
> DATA IN.
> But an OSD device can do the following data transitions: DATA IN (user
> data) ; DATA OUT
> (set attributes data and get attributes list) ;  DATA IN (retrieved
> attributes data).

Yes, that's what I feared.  The spec implies this as well.  Behaviour
like that requires two data pointers.

James

RE: [PATCH] 2.6.2-rc2 - MPT Fusion driver 3.00.02 update

[Moore, Eric Dean]

I will work on that.  

I had avoided it due the comments in the code
from sralston and pdelaney about sg interface 
gererating wrong direction in some cases.  

Eric


On Monday, January 26, 2004 9:45 AM, James Bottomley wrote:
> On Mon, 2004-01-26 at 10:37, Moore, Eric Dean wrote:
> > Here is another update for the MPT Fusion drivers.
> 
> Thanks.
> 
> > * added CONFIG_LBA, READ16, WRITE16 support
> 
> I'd like to see this routine to determine direction go away 
> in favour of
> just using the direction the mid layer sends down.  However, 
> this isn't
> a blocking item, I'll try and get this update into early 2.6.2
> 
> James
> 
> 

RE: [PATCH] 2.6.2-rc2 - MPT Fusion driver 3.00.02 update

[James Bottomley]

On Mon, 2004-01-26 at 11:01, Moore, Eric Dean wrote:
> I will work on that.  

Thanks.

> I had avoided it due the comments in the code
> from sralston and pdelaney about sg interface 
> gererating wrong direction in some cases.  

If you find a problem, I'll fix the generic code...

James

Re: [PATCH] 2.6.2-rc2 - MPT Fusion driver 3.00.02 update

[Douglas Gilbert]

James Bottomley wrote:
> On Mon, 2004-01-26 at 11:01, Moore, Eric Dean wrote:
> 
>>I will work on that.  
> 
> 
> Thanks.
> 
> 
>>I had avoided it due the comments in the code
>>from sralston and pdelaney about sg interface 
>>gererating wrong direction in some cases.  
> 
> 
> If you find a problem, I'll fix the generic code...

James,
The newer sg_io_hdr based code (as used by the SG_IO ioctl)
requires that the user explicitly define the data direction
if any data transfer is involved. There is however a
SG_DXFER_UNKNOWN which maps to SCSI_DATA_UNKNOWN which
maps to DMA_BIDIRECTIONAL **.

The older sg_header based code (and the SCSI_IOCTL_SEND_COMMAND
ioctl) deduced the direction from various data buffer lengths
provided by the user. If those length implied both read and write
then the data direction is set as "FROM_DEVICE" (i.e. read
dominates).

So the sg (or SG_IO/SCSI_IOCTL_SEND_COMMAND ioctl) user controls
the data direction and they are free to get it wrong:-)


** The Object Storage Device (OSD) commands use bidirectional
data transfers so perhaps we should think about catering for
that.

Doug Gilbert

Re: [PATCH] 2.6.2-rc2 - MPT Fusion driver 3.00.02 update

[James Bottomley]

On Tue, 2004-01-27 at 02:48, Douglas Gilbert wrote:
> So the sg (or SG_IO/SCSI_IOCTL_SEND_COMMAND ioctl) user controls
> the data direction and they are free to get it wrong:-)

Since we do have a mid layer library for this, perhaps we should set it
in sg if the user leaves it apparently unset.

> ** The Object Storage Device (OSD) commands use bidirectional
> data transfers so perhaps we should think about catering for
> that.

Actually, a much more fundamental problem for OSD is going to be >16
byte CDBs.  The mid and block layers today are coded with an internal 16
byte array for the CDB.  That's not very scaleable to the hundreds of
bytes that OSD needs.

As far as bidirectional goes, we should be able to support that today
providing there's just a *single* data buffer, which the OSD spec
implies is the easiest way to support this (although I suspect most SCSI
processor coding won't be expecting to see both a data in and a data out
from a single command).

As I see it, OSD seems to require a different interface from the current
bio layer, so there would be some type of object layer glue between the
fs and the mid-layer, with the block layer relegated simply to queueing
specials for the mid-layer, but since no-one's actually come forward
with an implementation, that's just a guess.

James

bidirectional, long commands + OSD

[Douglas Gilbert]

James Bottomley wrote in thread:
   [PATCH] 2.6.2-rc2 - MPT Fusion driver 3.00.02 update

> On Tue, 2004-01-27 at 02:48, Douglas Gilbert wrote:
> 
>>So the sg (or SG_IO/SCSI_IOCTL_SEND_COMMAND ioctl) user controls
>>the data direction and they are free to get it wrong:-)
> 
> 
> Since we do have a mid layer library for this, perhaps we should set it
> in sg if the user leaves it apparently unset.

It is valid for vendor specific commands to have command lengths
other than those dictated by the top 3 bits of the opcode. The
variable length opcode (0x7f) breaks that convention as well.
The pass-through philosophy is (within reason) "you asked for
it ....".

>>** The Object Storage Device (OSD) commands use bidirectional
>>data transfers so perhaps we should think about catering for
>>that.
> 
> 
> Actually, a much more fundamental problem for OSD is going to be >16
> byte CDBs.  The mid and block layers today are coded with an internal 16
> byte array for the CDB.  That's not very scaleable to the hundreds of
> bytes that OSD needs.

Just did a check on recent drafts and SPC-3 doesn't have any
commands with lengths > 16 bytes but SBC-2 has:
    XDREAD(32)
    XDWRITE(32)
    XDWRITEREAD(32)
    XPWRITE(32)

> As far as bidirectional goes, we should be able to support that today
> providing there's just a *single* data buffer, which the OSD spec
> implies is the easiest way to support this (although I suspect most SCSI
> processor coding won't be expecting to see both a data in and a data out
> from a single command).

... and XDWRITEREAD (both 10 and 32 byte variants) is bidirectional.

> As I see it, OSD seems to require a different interface from the current
> bio layer, so there would be some type of object layer glue between the
> fs and the mid-layer, with the block layer relegated simply to queueing
> specials for the mid-layer, but since no-one's actually come forward
> with an implementation, that's just a guess.

Both Jens and Linus jumped on me when I suggested
similar heresy :-)

For anyone interested, an overview of the proposed Object
Storage Device (OSD) architecture can be found at:
http://www.t10.org/ftp/t10/drafts/osd/osd-r08.pdf
in section 4.2 .

Doug Gilbert

Re: bidirectional, long commands + OSD

[Liran Schour]

Douglas Gilbert wrote in thread:
   bidirectional, long commands + OSD


>>>** The Object Storage Device (OSD) commands use bidirectional
>>>data transfers so perhaps we should think about catering for
>>>that.
>>
>>
>> Actually, a much more fundamental problem for OSD is going to be >16
>> byte CDBs.  The mid and block layers today are coded with an internal 16
>> byte array for the CDB.  That's not very scaleable to the hundreds of
>> bytes that OSD needs.

We can allow the upper layer driver to set the cmd_len and the cdb pointer
to the
desired length, the default will stay 16 byte.

>Just did a check on recent drafts and SPC-3 doesn't have any
>commands with lengths > 16 bytes but SBC-2 has:
>    XDREAD(32)
>    XDWRITE(32)
>    XDWRITEREAD(32)
>    XPWRITE(32)

>> As far as bidirectional goes, we should be able to support that today
>> providing there's just a *single* data buffer, which the OSD spec
>> implies is the easiest way to support this (although I suspect most SCSI
>> processor coding won't be expecting to see both a data in and a data out
>> from a single command).

>... and XDWRITEREAD (both 10 and 32 byte variants) is bidirectional.

A single buffer for both data in and data out will force the user to
re-copy the buffer.
We can solve it by using a scatter-gather list. Each buffer will have its
own data
direction flag.

>> As I see it, OSD seems to require a different interface from the current
>> bio layer, so there would be some type of object layer glue between the
>> fs and the mid-layer, with the block layer relegated simply to queueing
>> specials for the mid-layer, but since no-one's actually come forward
>> with an implementation, that's just a guess.

>Both Jens and Linus jumped on me when I suggested
>similar heresy :-)

Intel's iscsi reference implementation includes an upper layer driver for
OSD, it
also includes a mini filesystem that works on top of it.
This implementation only demonstrate the need to solve this 3 issues: (the
implementation just work around them)
      1. Coupling of the scsi layer with the bio layer.
      2. Bidirectional support.
      3. Extended CDB support.

We will be happy to open this issues for a discussion.

- Liran

Re: bidirectional, long commands + OSD

[James Bottomley]

On Thu, 2004-01-29 at 08:07, Douglas Gilbert wrote:
> James Bottomley wrote in thread:
> > As I see it, OSD seems to require a different interface from the current
> > bio layer, so there would be some type of object layer glue between the
> > fs and the mid-layer, with the block layer relegated simply to queueing
> > specials for the mid-layer, but since no-one's actually come forward
> > with an implementation, that's just a guess.
> 
> Both Jens and Linus jumped on me when I suggested
> similar heresy :-)

Well ... I merely stated what it seems to require.  I didn't recommend
implementing it.

>From what I can tell, the OSD spec seems to be aimed at large clustered
filesystems.  The recommended mapping seems to be <multiple objects> <->
<one file>.  However, since different objects are distinct and currently
require distinct commands to access them, I believe using OSD for a
typical Linux filesystem would be a big loss because most files are
small and we can no-longer elevate the requests.  It would be
interesting to see if this loss could be offset by not having to do
indirect block lookups, but I have my doubts.

James