RFC: SCSI Generic version 4 interface

RFC: SCSI Generic version 4 interface

[Douglas Gilbert]

[-- Attachment #1: Type: text/plain, Size: 787 bytes --]

I was asked to put together a proposal in May this
year for a new SCSI Generic interface structure. This
is the same structure that is used by the block layer
SG_IO ioctl. A few people have asked whether I had forgotten
that I agreed to write the proposal. So here it is. Those
who have seen it have made comments, some of which have
been incorporated.

Some shortcomings of the sg version 3 interface are:
  - can't handle commands with bidirectional data (either
    can the SCSI subsystem at the moment)
  - if it was a bit more general it could carry other
    request/response protocols (e.g. Task Management
    Functions and SMP in Serial Attached SCSI)
  - no way of associating a task attribute or task tag
    with a SCSI command

Feel free to make suggestions.

Doug Gilbert


[-- Attachment #2: sg4_proposal.txt --]
[-- Type: text/plain, Size: 4937 bytes --]

                SCSI Generic version 4 interface structure
                ==========================================
                                             Version 1.1

Goals:
  - handle both generalized request/response and data_out/data_in
    independently in same invocation. Alternatively the request
    and data_out could be in one invocation and the response and
    data_in in a following invocation. This allows for the most
    complicated SCSI commands: tagged variable length cdbs with
    bidirectional data transfers.
  - support multiple protocols. If they are generalized request-response
    protocols then they can choose either the request/response part
    of the interface or the data_out/data_in part.
  - layered error/condition reporting: (OS) driver, transport and
    device (logical unit). Method used to present this struct to OS
    (e.g. ioctl()) may also report error (e.g. EPERM).
  - allow for auxiliary information to be passed back for the
    application client to consider
  - same structure can be used for a synchronous (e.g. interruptible
    ioctl) or asynchronous (e.g. ioctl()/read() ) pass through.
  - leave device (lu) or target addressing issues to some other
    mechanism (what SCSI standards call the I_T_L or the I_T nexus
    respectively) as they are transport dependent. However do include
    the tag level (the "_Q" part of a I_T_L_Q nexus).
  - stay close enough to struct sg_io_hdr (sg version 3 interface)
    to use with existing SG_IO ioctls, current implementations
    expect 'S' in 'guard'


Comments:
  - use 64 bit integers to convey pointers. Does this help with
    32 to 64 bit thunking?
  - should there be more (or less) spare fields?
  - we could allow the "struct" to be rubbery, only requiring the
    first three integers and perhaps an additional_len field
  - the write() usage in the sg driver's asynchronous interface has
    caused problems when mistakenly applied to a block device node
    rather than a sg device node. Using an ioctl(flag_async) followed
    by a read() for asynchronous work offers similar functionality and
    is safer. Using ioctl(flag_async_start) and ioctl(flag_async_finish)
    is another possibility.
  - rather than have a separate ATA pass through mechanism, the SAT
    defined ATA PASS THROUGH SCSI commands could be used with the
    driver implementation routing the ATA commands to their
    subsystem. This could be flagged so it didn't preclude a SAT layer
    in a SCSI transport (e.g. MPT SAS HBA firmware).
  - if SAM/SPC does not define an enumeration for lesser used input
    fields, then use the value 0 for inert/off/don't_care .


-------------------------------------------------------------------


struct sg_io_v4
{
        int32_t guard;          // [i] 'Q' to differentiate from v3
        uint32_t protocol;      // [i] 0 -> SCSI , ....
        uint32_t subprotocol;   // [i] 0 -> SCSI command, 1 -> SCSI task
                                // management function, ....

        uint32_t request_len;   // [i] in bytes
        uint64_t request;       // [i], [*i] {SCSI: cdb}
        uint32_t request_attr;  // [i] {SCSI: task attribute}
        uint32_t request_tag;   // [i] {SCSI: task tag (only if flagged)}
        uint32_t request_priority; // [i] {SCSI: task priority}
        uint32_t max_response_len; // [i] in bytes
        uint64_t response;      // [i], [*o]  {SCSI: (auto)sense data}

        // "din_" for data in (from device); "dout_" for data out (to device)
        uint32_t dout_iovec_count;      // [i] 0 -> "flat" data transfer
                                // else dout_xfer points to array of iovec
        uint32_t dout_xfer_len; // [i] bytes to be transferred to device
        uint64_t dout_xfer;     // [i], [*i] 
        uint32_t din_iovec_count;       // [i]
        uint32_t din_xfer_len;  // [i] bytes to be transferred from device
        uint64_t din_xfer;      // [i], [*o] 

        uint32_t timeout;       // [i] units: seconds or milliseconds?
        uint32_t flags;         // [i] bit mask
        uint64_t usr_ptr;       // [i->o] unused internally
        uint32_t spare_in;      // [i]

        uint32_t driver_status; // [o] 0 -> ok
        uint32_t transport_status;      // [o] 0 -> ok
        uint32_t device_status; // [o] {SCSI: command completion status}
        uint32_t retry_delay;   // [o] {SCSI: status auxiliary information}
        uint32_t info;          // [o] additional information
        uint32_t duration;      // [o] time to complete, in milliseconds
        uint32_t response_len_wr;// [o] bytes of response actually written
        int32_t din_resid;      // [o] actual_din_xfer_len - din_xfer_len
        uint32_t generated_tag; // [o] {SCSI: task tag that transport chose}
        uint32_t spare_out;     // [o]

};


-------------------------------------------------------------------------

Doug Gilbert
6th November 2006

Re: RFC: SCSI Generic version 4 interface

[Jeff Garzik]

Douglas Gilbert wrote:
> I was asked to put together a proposal in May this
> year for a new SCSI Generic interface structure. This
> is the same structure that is used by the block layer
> SG_IO ioctl. A few people have asked whether I had forgotten
> that I agreed to write the proposal. So here it is. Those
> who have seen it have made comments, some of which have
> been incorporated.
> 
> Some shortcomings of the sg version 3 interface are:
>   - can't handle commands with bidirectional data (either
>     can the SCSI subsystem at the moment)
>   - if it was a bit more general it could carry other
>     request/response protocols (e.g. Task Management
>     Functions and SMP in Serial Attached SCSI)
>   - no way of associating a task attribute or task tag
>     with a SCSI command

Why avoid Jens Axboe's bsg?

It seems like that is already a good interface for carrying other 
req/resp protocols.

	Jeff

Re: RFC: SCSI Generic version 4 interface

[Jens Axboe]

On Tue, Nov 07 2006, Jeff Garzik wrote:
> Douglas Gilbert wrote:
> >I was asked to put together a proposal in May this
> >year for a new SCSI Generic interface structure. This
> >is the same structure that is used by the block layer
> >SG_IO ioctl. A few people have asked whether I had forgotten
> >that I agreed to write the proposal. So here it is. Those
> >who have seen it have made comments, some of which have
> >been incorporated.
> >
> >Some shortcomings of the sg version 3 interface are:
> >  - can't handle commands with bidirectional data (either
> >    can the SCSI subsystem at the moment)
> >  - if it was a bit more general it could carry other
> >    request/response protocols (e.g. Task Management
> >    Functions and SMP in Serial Attached SCSI)
> >  - no way of associating a task attribute or task tag
> >    with a SCSI command
> 
> Why avoid Jens Axboe's bsg?
> 
> It seems like that is already a good interface for carrying other 
> req/resp protocols.

I don't think Doug is avoiding that (if you are Doug, please do explain
:-), but rather outlining the next generation command format that bsg
should support for future use.

-- 
Jens Axboe

Re: RFC: SCSI Generic version 4 interface

[Douglas Gilbert]

Jens Axboe wrote:
> On Tue, Nov 07 2006, Jeff Garzik wrote:
>> Douglas Gilbert wrote:
>>> I was asked to put together a proposal in May this
>>> year for a new SCSI Generic interface structure. This
>>> is the same structure that is used by the block layer
>>> SG_IO ioctl. A few people have asked whether I had forgotten
>>> that I agreed to write the proposal. So here it is. Those
>>> who have seen it have made comments, some of which have
>>> been incorporated.
>>>
>>> Some shortcomings of the sg version 3 interface are:
>>>  - can't handle commands with bidirectional data (either
>>>    can the SCSI subsystem at the moment)
>>>  - if it was a bit more general it could carry other
>>>    request/response protocols (e.g. Task Management
>>>    Functions and SMP in Serial Attached SCSI)
>>>  - no way of associating a task attribute or task tag
>>>    with a SCSI command
>> Why avoid Jens Axboe's bsg?
>>
>> It seems like that is already a good interface for carrying other 
>> req/resp protocols.
> 
> I don't think Doug is avoiding that (if you are Doug, please do explain
> :-), but rather outlining the next generation command format that bsg
> should support for future use.

In the simplest case this is just the next generation
version of the sg_io_hdr structure used by the SG_IO
ioctl interface. It pitches at about the same level as
modern SCSI transports (e.g. FCP, SAS and iSCSI). It
leaves session and addressing issues, with the exception
(arguably) of tags, to others. The "others" could include
the bsg driver or perhaps a transport layer/driver.

Reference:
  SAM-4 version 7 section 5.1 "The Execute command
                               procedure call"
          http://www.t10.org/ftp/t10/drafts/sam4/sam4r07.pdf

Doug Gilbert

Re: RFC: SCSI Generic version 4 interface

[FUJITA Tomonori]

From: Jens Axboe <jens.axboe@oracle.com>
Subject: Re: RFC: SCSI Generic version 4 interface
Date: Tue, 7 Nov 2006 16:31:48 +0100

> On Tue, Nov 07 2006, Jeff Garzik wrote:
> > Douglas Gilbert wrote:
> > >I was asked to put together a proposal in May this
> > >year for a new SCSI Generic interface structure. This
> > >is the same structure that is used by the block layer
> > >SG_IO ioctl. A few people have asked whether I had forgotten
> > >that I agreed to write the proposal. So here it is. Those
> > >who have seen it have made comments, some of which have
> > >been incorporated.
> > >
> > >Some shortcomings of the sg version 3 interface are:
> > >  - can't handle commands with bidirectional data (either
> > >    can the SCSI subsystem at the moment)
> > >  - if it was a bit more general it could carry other
> > >    request/response protocols (e.g. Task Management
> > >    Functions and SMP in Serial Attached SCSI)
> > >  - no way of associating a task attribute or task tag
> > >    with a SCSI command
> > 
> > Why avoid Jens Axboe's bsg?
> > 
> > It seems like that is already a good interface for carrying other 
> > req/resp protocols.
> 
> I don't think Doug is avoiding that (if you are Doug, please do explain
> :-), but rather outlining the next generation command format that bsg
> should support for future use.

sg4 can be implemented on bsg nicely, I think. But why does the
current bsg support sg3 partially? Do you plan to add the rest of sg3
(which doesn't work nicely) to bsg?

Re: RFC: SCSI Generic version 4 interface

[Jens Axboe]

On Thu, Dec 07 2006, FUJITA Tomonori wrote:
> From: Jens Axboe <jens.axboe@oracle.com>
> Subject: Re: RFC: SCSI Generic version 4 interface
> Date: Tue, 7 Nov 2006 16:31:48 +0100
> 
> > On Tue, Nov 07 2006, Jeff Garzik wrote:
> > > Douglas Gilbert wrote:
> > > >I was asked to put together a proposal in May this
> > > >year for a new SCSI Generic interface structure. This
> > > >is the same structure that is used by the block layer
> > > >SG_IO ioctl. A few people have asked whether I had forgotten
> > > >that I agreed to write the proposal. So here it is. Those
> > > >who have seen it have made comments, some of which have
> > > >been incorporated.
> > > >
> > > >Some shortcomings of the sg version 3 interface are:
> > > >  - can't handle commands with bidirectional data (either
> > > >    can the SCSI subsystem at the moment)
> > > >  - if it was a bit more general it could carry other
> > > >    request/response protocols (e.g. Task Management
> > > >    Functions and SMP in Serial Attached SCSI)
> > > >  - no way of associating a task attribute or task tag
> > > >    with a SCSI command
> > > 
> > > Why avoid Jens Axboe's bsg?
> > > 
> > > It seems like that is already a good interface for carrying other 
> > > req/resp protocols.
> > 
> > I don't think Doug is avoiding that (if you are Doug, please do explain
> > :-), but rather outlining the next generation command format that bsg
> > should support for future use.
> 
> sg4 can be implemented on bsg nicely, I think. But why does the
> current bsg support sg3 partially? Do you plan to add the rest of sg3
> (which doesn't work nicely) to bsg?

What parts of sg v3 are missing? It's been a while since I implemented
that stuff, so I forget if I knowingly left out some features.

-- 
Jens Axboe

Re: RFC: SCSI Generic version 4 interface

[FUJITA Tomonori]

From: Jens Axboe <jens.axboe@oracle.com>
Subject: Re: RFC: SCSI Generic version 4 interface
Date: Thu, 7 Dec 2006 09:06:59 +0100

> On Thu, Dec 07 2006, FUJITA Tomonori wrote:
> > From: Jens Axboe <jens.axboe@oracle.com>
> > Subject: Re: RFC: SCSI Generic version 4 interface
> > Date: Tue, 7 Nov 2006 16:31:48 +0100
> > 
> > > On Tue, Nov 07 2006, Jeff Garzik wrote:
> > > > Douglas Gilbert wrote:
> > > > >I was asked to put together a proposal in May this
> > > > >year for a new SCSI Generic interface structure. This
> > > > >is the same structure that is used by the block layer
> > > > >SG_IO ioctl. A few people have asked whether I had forgotten
> > > > >that I agreed to write the proposal. So here it is. Those
> > > > >who have seen it have made comments, some of which have
> > > > >been incorporated.
> > > > >
> > > > >Some shortcomings of the sg version 3 interface are:
> > > > >  - can't handle commands with bidirectional data (either
> > > > >    can the SCSI subsystem at the moment)
> > > > >  - if it was a bit more general it could carry other
> > > > >    request/response protocols (e.g. Task Management
> > > > >    Functions and SMP in Serial Attached SCSI)
> > > > >  - no way of associating a task attribute or task tag
> > > > >    with a SCSI command
> > > > 
> > > > Why avoid Jens Axboe's bsg?
> > > > 
> > > > It seems like that is already a good interface for carrying other 
> > > > req/resp protocols.
> > > 
> > > I don't think Doug is avoiding that (if you are Doug, please do explain
> > > :-), but rather outlining the next generation command format that bsg
> > > should support for future use.
> > 
> > sg4 can be implemented on bsg nicely, I think. But why does the
> > current bsg support sg3 partially? Do you plan to add the rest of sg3
> > (which doesn't work nicely) to bsg?
> 
> What parts of sg v3 are missing? It's been a while since I implemented
> that stuff, so I forget if I knowingly left out some features.

SG_FLAG_MMAP_IO, indirect IO, probably I miss something more.

Re: RFC: SCSI Generic version 4 interface

[Jens Axboe]

On Thu, Dec 07 2006, FUJITA Tomonori wrote:
> From: Jens Axboe <jens.axboe@oracle.com>
> Subject: Re: RFC: SCSI Generic version 4 interface
> Date: Thu, 7 Dec 2006 09:06:59 +0100
> 
> > On Thu, Dec 07 2006, FUJITA Tomonori wrote:
> > > From: Jens Axboe <jens.axboe@oracle.com>
> > > Subject: Re: RFC: SCSI Generic version 4 interface
> > > Date: Tue, 7 Nov 2006 16:31:48 +0100
> > > 
> > > > On Tue, Nov 07 2006, Jeff Garzik wrote:
> > > > > Douglas Gilbert wrote:
> > > > > >I was asked to put together a proposal in May this
> > > > > >year for a new SCSI Generic interface structure. This
> > > > > >is the same structure that is used by the block layer
> > > > > >SG_IO ioctl. A few people have asked whether I had forgotten
> > > > > >that I agreed to write the proposal. So here it is. Those
> > > > > >who have seen it have made comments, some of which have
> > > > > >been incorporated.
> > > > > >
> > > > > >Some shortcomings of the sg version 3 interface are:
> > > > > >  - can't handle commands with bidirectional data (either
> > > > > >    can the SCSI subsystem at the moment)
> > > > > >  - if it was a bit more general it could carry other
> > > > > >    request/response protocols (e.g. Task Management
> > > > > >    Functions and SMP in Serial Attached SCSI)
> > > > > >  - no way of associating a task attribute or task tag
> > > > > >    with a SCSI command
> > > > > 
> > > > > Why avoid Jens Axboe's bsg?
> > > > > 
> > > > > It seems like that is already a good interface for carrying other 
> > > > > req/resp protocols.
> > > > 
> > > > I don't think Doug is avoiding that (if you are Doug, please do explain
> > > > :-), but rather outlining the next generation command format that bsg
> > > > should support for future use.
> > > 
> > > sg4 can be implemented on bsg nicely, I think. But why does the
> > > current bsg support sg3 partially? Do you plan to add the rest of sg3
> > > (which doesn't work nicely) to bsg?
> > 
> > What parts of sg v3 are missing? It's been a while since I implemented
> > that stuff, so I forget if I knowingly left out some features.
> 
> SG_FLAG_MMAP_IO, indirect IO, probably I miss something more.

bsg chooses the best way to handle the buffers on its own right now, in
my opinion sg has grown too many "features" that aren't very useful. If
the buffer and length are properly aligned, bsg does direct io. If not,
it does indirect.  mmap io may be useful, we could add that if anyone is
actually using it.

-- 
Jens Axboe

Re: RFC: SCSI Generic version 4 interface

[FUJITA Tomonori]

From: Jens Axboe <jens.axboe@oracle.com>
Subject: Re: RFC: SCSI Generic version 4 interface
Date: Thu, 7 Dec 2006 09:30:20 +0100

> On Thu, Dec 07 2006, FUJITA Tomonori wrote:
> > From: Jens Axboe <jens.axboe@oracle.com>
> > Subject: Re: RFC: SCSI Generic version 4 interface
> > Date: Thu, 7 Dec 2006 09:06:59 +0100
> > 
> > > On Thu, Dec 07 2006, FUJITA Tomonori wrote:
> > > > From: Jens Axboe <jens.axboe@oracle.com>
> > > > Subject: Re: RFC: SCSI Generic version 4 interface
> > > > Date: Tue, 7 Nov 2006 16:31:48 +0100
> > > > 
> > > > > On Tue, Nov 07 2006, Jeff Garzik wrote:
> > > > > > Douglas Gilbert wrote:
> > > > > > >I was asked to put together a proposal in May this
> > > > > > >year for a new SCSI Generic interface structure. This
> > > > > > >is the same structure that is used by the block layer
> > > > > > >SG_IO ioctl. A few people have asked whether I had forgotten
> > > > > > >that I agreed to write the proposal. So here it is. Those
> > > > > > >who have seen it have made comments, some of which have
> > > > > > >been incorporated.
> > > > > > >
> > > > > > >Some shortcomings of the sg version 3 interface are:
> > > > > > >  - can't handle commands with bidirectional data (either
> > > > > > >    can the SCSI subsystem at the moment)
> > > > > > >  - if it was a bit more general it could carry other
> > > > > > >    request/response protocols (e.g. Task Management
> > > > > > >    Functions and SMP in Serial Attached SCSI)
> > > > > > >  - no way of associating a task attribute or task tag
> > > > > > >    with a SCSI command
> > > > > > 
> > > > > > Why avoid Jens Axboe's bsg?
> > > > > > 
> > > > > > It seems like that is already a good interface for carrying other 
> > > > > > req/resp protocols.
> > > > > 
> > > > > I don't think Doug is avoiding that (if you are Doug, please do explain
> > > > > :-), but rather outlining the next generation command format that bsg
> > > > > should support for future use.
> > > > 
> > > > sg4 can be implemented on bsg nicely, I think. But why does the
> > > > current bsg support sg3 partially? Do you plan to add the rest of sg3
> > > > (which doesn't work nicely) to bsg?
> > > 
> > > What parts of sg v3 are missing? It's been a while since I implemented
> > > that stuff, so I forget if I knowingly left out some features.
> > 
> > SG_FLAG_MMAP_IO, indirect IO, probably I miss something more.
> 
> bsg chooses the best way to handle the buffers on its own right now, in
> my opinion sg has grown too many "features" that aren't very useful.

I agree. But if bsg replaces sg3 code, it supports the full features,
doesn't it? If bsg does not replace sg3 code, we have more
duplication.

I think it would be nice for bsg to support only sg4 (does not support
some of the old sg3 interfaces) and not add sg4 support to sg.c and
scsi_ioctl.c (of course they can share some code).

Oh, bsg's sg3 doesn't work on 32-bit userspace and 64-bit kernel too
if applicationos use the iovec interface, does it?


> If the buffer and length are properly aligned, bsg does direct
> io. If not, it does indirect.

Yeah.


> mmap io may be useful, we could add that if anyone is actually using
> it.

sg mmap io is tricky. sg_io_hdr doesn't have a mapped address,
instead, the sg driver keeps a mapped address when an application
calls mmap (that means sg cannot handle multiple mmap io requests
simultaneously).

Re: RFC: SCSI Generic version 4 interface

[Jens Axboe]

On Thu, Dec 07 2006, FUJITA Tomonori wrote:
> From: Jens Axboe <jens.axboe@oracle.com>
> Subject: Re: RFC: SCSI Generic version 4 interface
> Date: Thu, 7 Dec 2006 09:30:20 +0100
> 
> > On Thu, Dec 07 2006, FUJITA Tomonori wrote:
> > > From: Jens Axboe <jens.axboe@oracle.com>
> > > Subject: Re: RFC: SCSI Generic version 4 interface
> > > Date: Thu, 7 Dec 2006 09:06:59 +0100
> > > 
> > > > On Thu, Dec 07 2006, FUJITA Tomonori wrote:
> > > > > From: Jens Axboe <jens.axboe@oracle.com>
> > > > > Subject: Re: RFC: SCSI Generic version 4 interface
> > > > > Date: Tue, 7 Nov 2006 16:31:48 +0100
> > > > > 
> > > > > > On Tue, Nov 07 2006, Jeff Garzik wrote:
> > > > > > > Douglas Gilbert wrote:
> > > > > > > >I was asked to put together a proposal in May this
> > > > > > > >year for a new SCSI Generic interface structure. This
> > > > > > > >is the same structure that is used by the block layer
> > > > > > > >SG_IO ioctl. A few people have asked whether I had forgotten
> > > > > > > >that I agreed to write the proposal. So here it is. Those
> > > > > > > >who have seen it have made comments, some of which have
> > > > > > > >been incorporated.
> > > > > > > >
> > > > > > > >Some shortcomings of the sg version 3 interface are:
> > > > > > > >  - can't handle commands with bidirectional data (either
> > > > > > > >    can the SCSI subsystem at the moment)
> > > > > > > >  - if it was a bit more general it could carry other
> > > > > > > >    request/response protocols (e.g. Task Management
> > > > > > > >    Functions and SMP in Serial Attached SCSI)
> > > > > > > >  - no way of associating a task attribute or task tag
> > > > > > > >    with a SCSI command
> > > > > > > 
> > > > > > > Why avoid Jens Axboe's bsg?
> > > > > > > 
> > > > > > > It seems like that is already a good interface for carrying other 
> > > > > > > req/resp protocols.
> > > > > > 
> > > > > > I don't think Doug is avoiding that (if you are Doug, please do explain
> > > > > > :-), but rather outlining the next generation command format that bsg
> > > > > > should support for future use.
> > > > > 
> > > > > sg4 can be implemented on bsg nicely, I think. But why does the
> > > > > current bsg support sg3 partially? Do you plan to add the rest of sg3
> > > > > (which doesn't work nicely) to bsg?
> > > > 
> > > > What parts of sg v3 are missing? It's been a while since I implemented
> > > > that stuff, so I forget if I knowingly left out some features.
> > > 
> > > SG_FLAG_MMAP_IO, indirect IO, probably I miss something more.
> > 
> > bsg chooses the best way to handle the buffers on its own right now, in
> > my opinion sg has grown too many "features" that aren't very useful.
> 
> I agree. But if bsg replaces sg3 code, it supports the full features,
> doesn't it? If bsg does not replace sg3 code, we have more
> duplication.
> 
> I think it would be nice for bsg to support only sg4 (does not support
> some of the old sg3 interfaces) and not add sg4 support to sg.c and
> scsi_ioctl.c (of course they can share some code).

Yeah, I agree with that (if only to reduce my headache of actually
supporting the code!). When the sg v4 command interface is fully fleshed
out, I'm fine with moving bsg in that direction and dumping sg v3
support completely.

> Oh, bsg's sg3 doesn't work on 32-bit userspace and 64-bit kernel too
> if applicationos use the iovec interface, does it?

Probably not, haven't looked or checked :-)

> > mmap io may be useful, we could add that if anyone is actually using
> > it.
> 
> sg mmap io is tricky. sg_io_hdr doesn't have a mapped address,
> instead, the sg driver keeps a mapped address when an application
> calls mmap (that means sg cannot handle multiple mmap io requests
> simultaneously).

With sg v3 support out of the picture, that's not a worry. I'll probably
look into a different request/response architecture as well, the user
visible ring buffer is a strong approach in my opinion.

-- 
Jens Axboe

Re: RFC: SCSI Generic version 4 interface

[FUJITA Tomonori]

From: Jens Axboe <jens.axboe@oracle.com>
Subject: Re: RFC: SCSI Generic version 4 interface
Date: Thu, 7 Dec 2006 09:57:56 +0100

> On Thu, Dec 07 2006, FUJITA Tomonori wrote:
> > From: Jens Axboe <jens.axboe@oracle.com>
> > Subject: Re: RFC: SCSI Generic version 4 interface
> > Date: Thu, 7 Dec 2006 09:30:20 +0100
> > 
> > > On Thu, Dec 07 2006, FUJITA Tomonori wrote:
> > > > From: Jens Axboe <jens.axboe@oracle.com>
> > > > Subject: Re: RFC: SCSI Generic version 4 interface
> > > > Date: Thu, 7 Dec 2006 09:06:59 +0100
> > > > 
> > > > > On Thu, Dec 07 2006, FUJITA Tomonori wrote:
> > > > > > From: Jens Axboe <jens.axboe@oracle.com>
> > > > > > Subject: Re: RFC: SCSI Generic version 4 interface
> > > > > > Date: Tue, 7 Nov 2006 16:31:48 +0100
> > > > > > 
> > > > > > > On Tue, Nov 07 2006, Jeff Garzik wrote:
> > > > > > > > Douglas Gilbert wrote:
> > > > > > > > >I was asked to put together a proposal in May this
> > > > > > > > >year for a new SCSI Generic interface structure. This
> > > > > > > > >is the same structure that is used by the block layer
> > > > > > > > >SG_IO ioctl. A few people have asked whether I had forgotten
> > > > > > > > >that I agreed to write the proposal. So here it is. Those
> > > > > > > > >who have seen it have made comments, some of which have
> > > > > > > > >been incorporated.
> > > > > > > > >
> > > > > > > > >Some shortcomings of the sg version 3 interface are:
> > > > > > > > >  - can't handle commands with bidirectional data (either
> > > > > > > > >    can the SCSI subsystem at the moment)
> > > > > > > > >  - if it was a bit more general it could carry other
> > > > > > > > >    request/response protocols (e.g. Task Management
> > > > > > > > >    Functions and SMP in Serial Attached SCSI)
> > > > > > > > >  - no way of associating a task attribute or task tag
> > > > > > > > >    with a SCSI command
> > > > > > > > 
> > > > > > > > Why avoid Jens Axboe's bsg?
> > > > > > > > 
> > > > > > > > It seems like that is already a good interface for carrying other 
> > > > > > > > req/resp protocols.
> > > > > > > 
> > > > > > > I don't think Doug is avoiding that (if you are Doug, please do explain
> > > > > > > :-), but rather outlining the next generation command format that bsg
> > > > > > > should support for future use.
> > > > > > 
> > > > > > sg4 can be implemented on bsg nicely, I think. But why does the
> > > > > > current bsg support sg3 partially? Do you plan to add the rest of sg3
> > > > > > (which doesn't work nicely) to bsg?
> > > > > 
> > > > > What parts of sg v3 are missing? It's been a while since I implemented
> > > > > that stuff, so I forget if I knowingly left out some features.
> > > > 
> > > > SG_FLAG_MMAP_IO, indirect IO, probably I miss something more.
> > > 
> > > bsg chooses the best way to handle the buffers on its own right now, in
> > > my opinion sg has grown too many "features" that aren't very useful.
> > 
> > I agree. But if bsg replaces sg3 code, it supports the full features,
> > doesn't it? If bsg does not replace sg3 code, we have more
> > duplication.
> > 
> > I think it would be nice for bsg to support only sg4 (does not support
> > some of the old sg3 interfaces) and not add sg4 support to sg.c and
> > scsi_ioctl.c (of course they can share some code).
> 
> Yeah, I agree with that (if only to reduce my headache of actually
> supporting the code!). When the sg v4 command interface is fully fleshed
> out, I'm fine with moving bsg in that direction and dumping sg v3
> support completely.

Nice.

BTW, do you plan to use bsg only for sg4 (like putting sg4 hdr into
bsg_command directly) or provide generic interfaces (I mean that sg4
calls something like bsg_register_protocol) ?


> > Oh, bsg's sg3 doesn't work on 32-bit userspace and 64-bit kernel too
> > if applicationos use the iovec interface, does it?
> 
> Probably not, haven't looked or checked :-)

Whatever sg3 interface is used, bsg's sg3 doesn't work in such
environments, I think.


> > > mmap io may be useful, we could add that if anyone is actually using
> > > it.
> > 
> > sg mmap io is tricky. sg_io_hdr doesn't have a mapped address,
> > instead, the sg driver keeps a mapped address when an application
> > calls mmap (that means sg cannot handle multiple mmap io requests
> > simultaneously).
> 
> With sg v3 support out of the picture, that's not a worry. I'll probably
> look into a different request/response architecture as well, the user
> visible ring buffer is a strong approach in my opinion.

The scsi target infrastructure uses something like that. I'd like to
use bsg for it in the future.

Re: RFC: SCSI Generic version 4 interface

[Jens Axboe]

On Thu, Dec 07 2006, FUJITA Tomonori wrote:
> From: Jens Axboe <jens.axboe@oracle.com>
> Subject: Re: RFC: SCSI Generic version 4 interface
> Date: Thu, 7 Dec 2006 09:57:56 +0100
> 
> > On Thu, Dec 07 2006, FUJITA Tomonori wrote:
> > > From: Jens Axboe <jens.axboe@oracle.com>
> > > Subject: Re: RFC: SCSI Generic version 4 interface
> > > Date: Thu, 7 Dec 2006 09:30:20 +0100
> > > 
> > > > On Thu, Dec 07 2006, FUJITA Tomonori wrote:
> > > > > From: Jens Axboe <jens.axboe@oracle.com>
> > > > > Subject: Re: RFC: SCSI Generic version 4 interface
> > > > > Date: Thu, 7 Dec 2006 09:06:59 +0100
> > > > > 
> > > > > > On Thu, Dec 07 2006, FUJITA Tomonori wrote:
> > > > > > > From: Jens Axboe <jens.axboe@oracle.com>
> > > > > > > Subject: Re: RFC: SCSI Generic version 4 interface
> > > > > > > Date: Tue, 7 Nov 2006 16:31:48 +0100
> > > > > > > 
> > > > > > > > On Tue, Nov 07 2006, Jeff Garzik wrote:
> > > > > > > > > Douglas Gilbert wrote:
> > > > > > > > > >I was asked to put together a proposal in May this
> > > > > > > > > >year for a new SCSI Generic interface structure. This
> > > > > > > > > >is the same structure that is used by the block layer
> > > > > > > > > >SG_IO ioctl. A few people have asked whether I had forgotten
> > > > > > > > > >that I agreed to write the proposal. So here it is. Those
> > > > > > > > > >who have seen it have made comments, some of which have
> > > > > > > > > >been incorporated.
> > > > > > > > > >
> > > > > > > > > >Some shortcomings of the sg version 3 interface are:
> > > > > > > > > >  - can't handle commands with bidirectional data (either
> > > > > > > > > >    can the SCSI subsystem at the moment)
> > > > > > > > > >  - if it was a bit more general it could carry other
> > > > > > > > > >    request/response protocols (e.g. Task Management
> > > > > > > > > >    Functions and SMP in Serial Attached SCSI)
> > > > > > > > > >  - no way of associating a task attribute or task tag
> > > > > > > > > >    with a SCSI command
> > > > > > > > > 
> > > > > > > > > Why avoid Jens Axboe's bsg?
> > > > > > > > > 
> > > > > > > > > It seems like that is already a good interface for carrying other 
> > > > > > > > > req/resp protocols.
> > > > > > > > 
> > > > > > > > I don't think Doug is avoiding that (if you are Doug, please do explain
> > > > > > > > :-), but rather outlining the next generation command format that bsg
> > > > > > > > should support for future use.
> > > > > > > 
> > > > > > > sg4 can be implemented on bsg nicely, I think. But why does the
> > > > > > > current bsg support sg3 partially? Do you plan to add the rest of sg3
> > > > > > > (which doesn't work nicely) to bsg?
> > > > > > 
> > > > > > What parts of sg v3 are missing? It's been a while since I implemented
> > > > > > that stuff, so I forget if I knowingly left out some features.
> > > > > 
> > > > > SG_FLAG_MMAP_IO, indirect IO, probably I miss something more.
> > > > 
> > > > bsg chooses the best way to handle the buffers on its own right now, in
> > > > my opinion sg has grown too many "features" that aren't very useful.
> > > 
> > > I agree. But if bsg replaces sg3 code, it supports the full features,
> > > doesn't it? If bsg does not replace sg3 code, we have more
> > > duplication.
> > > 
> > > I think it would be nice for bsg to support only sg4 (does not support
> > > some of the old sg3 interfaces) and not add sg4 support to sg.c and
> > > scsi_ioctl.c (of course they can share some code).
> > 
> > Yeah, I agree with that (if only to reduce my headache of actually
> > supporting the code!). When the sg v4 command interface is fully fleshed
> > out, I'm fine with moving bsg in that direction and dumping sg v3
> > support completely.
> 
> Nice.
> 
> BTW, do you plan to use bsg only for sg4 (like putting sg4 hdr into
> bsg_command directly) or provide generic interfaces (I mean that sg4
> calls something like bsg_register_protocol) ?

I try not to over design such things, so it'll just be sg v4. If another
cases arises, we can introduce something like what you suggest.

> > > > mmap io may be useful, we could add that if anyone is actually using
> > > > it.
> > > 
> > > sg mmap io is tricky. sg_io_hdr doesn't have a mapped address,
> > > instead, the sg driver keeps a mapped address when an application
> > > calls mmap (that means sg cannot handle multiple mmap io requests
> > > simultaneously).
> > 
> > With sg v3 support out of the picture, that's not a worry. I'll probably
> > look into a different request/response architecture as well, the user
> > visible ring buffer is a strong approach in my opinion.
> 
> The scsi target infrastructure uses something like that. I'd like to
> use bsg for it in the future.

Neat, I'll take patches for that :-)
My intention was to hangon a little and see if the kevent stuff got
merged, then probably utilize that. If not, cook up my own.

-- 
Jens Axboe

Re: RFC: SCSI Generic version 4 interface

[FUJITA Tomonori]

From: Douglas Gilbert <dougg@torque.net>
Subject: RFC: SCSI Generic version 4 interface
Date: Mon, 06 Nov 2006 16:47:30 -0500

>                 SCSI Generic version 4 interface structure
>                 ==========================================
>                                              Version 1.1
> 
> Goals:

(snip)

>   - same structure can be used for a synchronous (e.g. interruptible
>     ioctl) or asynchronous (e.g. ioctl()/read() ) pass through.

Can you provide details on "asynchronous" part?

The scsi target code needs sg for:

- SAN gateway (like iSCSI to FC)

- SCSI device support in Xen (corresponds to raw device mapping in
VMware); enables domU to use SCSI devices. domU sends SCSI commands
via the virtual HBA driver to dom0 and then dom0 uses sg to perform
them.

The scsi target code uses the single user-space daemon so needs an
asynchronous interface, i.e. sends requests and receives the
completions asynchronously.

Re: RFC: SCSI Generic version 4 interface

[Douglas Gilbert]

FUJITA Tomonori wrote:
> From: Douglas Gilbert <dougg@torque.net>
> Subject: RFC: SCSI Generic version 4 interface
> Date: Mon, 06 Nov 2006 16:47:30 -0500
> 
>>                 SCSI Generic version 4 interface structure
>>                 ==========================================
>>                                              Version 1.1
>>
>> Goals:
> 
> (snip)
> 
>>   - same structure can be used for a synchronous (e.g. interruptible
>>     ioctl) or asynchronous (e.g. ioctl()/read() ) pass through.
> 
> Can you provide details on "asynchronous" part?

I was trying to keep away from the implementation details
but various people have pushed for more details ...


Well as you are aware, the sg driver uses a write(),
poll_or_signal,read() sequence to do asynchronous IO
in version 3 (and roughly the same model in version 2).

That write() has always made me uneasy and it is accident
prone. It could easily be replaced by and SG_IO ioctl
with a 'flag | async_request'. The question is how far
that gets "down the stack" before it returns to the
user. Some "tag" is required to be sent back to the
caller of an async request to:
  - enable the caller to identify the response
  - enable the caller to issue task management
    functions: abort task or query task

That is the purpose of the 'generated_tag' [o] field. I
commented that field as transport layer generated (i.e.
by the LLD) but it doesn't have to be, as long as it is
mappable to one.

The asynchronous notification that the command is completed
can be the current flavour of the month. The sg driver
uses file descriptor based techniques. They seem to work
in practice but the kernel folks don't seem to like them
and they could be dropped or mishandled by the user space
application.

Having a binary flag associated with the asynchronous notification
for worked/didn't could be useful. Still a driver (e.g. sg)
can't clear down its state information for the command in question
until it receives some active acknowledgment from the user
space program. The sg driver uses read() for this but this
could just as easily be the SG_IO ioctl with a
'flag | async_complete'. The version 3 sg driver can filter
those completions with its pack_id field (blocking or
non-blocking) or take the first available. The sg v4 interface
could filter via its request_tag (and a flag setting).

The sg driver has to cope with various non-completion
situations:
  - associated user space file descriptor closed by
    the time the async notification received: easy, throw
    away response and associated state
  - rmmod sg
  - app dies after receiving async notification: driver gets
    release() on that fd and throws away pending completion
    state
  - app falls silent: that leads to a stuck command in the
    sg driver or sg could be told to run its own auto
    completion timer

The async completion phase (i.e. the read() in sg v3) has
very little data to move as the xfer data pointers and sense
data pointer have been set up in the async request. That
only leaves: scsi_status, transport errors, resid and some
accounting (e.g. command duration). Its most important
function is being an active acknowledgment that the app
received the async command complete notification.


As for data buffer control, I would like to look at tgt's ring buffers.

> The scsi target code needs sg for:
> 
> - SAN gateway (like iSCSI to FC)

on the initiator side, I assume.

> - SCSI device support in Xen (corresponds to raw device mapping in
> VMware); enables domU to use SCSI devices. domU sends SCSI commands
> via the virtual HBA driver to dom0 and then dom0 uses sg to perform
> them.
> 
> The scsi target code uses the single user-space daemon so needs an
> asynchronous interface, i.e. sends requests and receives the
> completions asynchronously.


Doug Gilbert

Re: RFC: SCSI Generic version 4 interface

[FUJITA Tomonori]

From: Douglas Gilbert <dougg@torque.net>
Subject: Re: RFC: SCSI Generic version 4 interface
Date: Tue, 21 Nov 2006 17:06:46 -0500

> FUJITA Tomonori wrote:
> > From: Douglas Gilbert <dougg@torque.net>
> > Subject: RFC: SCSI Generic version 4 interface
> > Date: Mon, 06 Nov 2006 16:47:30 -0500
> > 
> >>                 SCSI Generic version 4 interface structure
> >>                 ==========================================
> >>                                              Version 1.1
> >>
> >> Goals:
> > 
> > (snip)
> > 
> >>   - same structure can be used for a synchronous (e.g. interruptible
> >>     ioctl) or asynchronous (e.g. ioctl()/read() ) pass through.
> > 
> > Can you provide details on "asynchronous" part?
> 
> I was trying to keep away from the implementation details
> but various people have pushed for more details ...
> 
> 
> Well as you are aware, the sg driver uses a write(),
> poll_or_signal,read() sequence to do asynchronous IO
> in version 3 (and roughly the same model in version 2).
> 
> That write() has always made me uneasy and it is accident
> prone.

I think write and read interface works nicely (though it's accident
prone as you said). They enable user-space applications to easily send
and receive sg_io_v4 structures asynchronously in batches (v3 can't in
batches). As you know, the current bsg adopts this interface.

Another possible interface that I prefer is to send and receive
sg_io_v4 structures via two ring buffers.


> > The scsi target code needs sg for:
> > 
> > - SAN gateway (like iSCSI to FC)
> 
> on the initiator side, I assume.

Just to be sure we're talking about the same thing;

I've been building a linux box forwarding SCSI commands from iSCSI
initiators to FC disk (an iSCSI-FC router).

Tgt's iSCSI driver runs in user space, so needs user-space APIs to
send SCSI commands (with attribute and tag) and TMFs to SCSI
devices. Now the iSCSI driver can forward only SCSI commands by using
bsg. It needs sg v4.

Some other tgt drivers run in kernel space so we might try to add some
tgt's kernel APIs for this later on.

Re: RFC: SCSI Generic version 4 interface

[FUJITA Tomonori]

From: FUJITA Tomonori <fujita.tomonori@lab.ntt.co.jp>
Subject: Re: RFC: SCSI Generic version 4 interface
Date: Sun, 26 Nov 2006 01:02:20 +0900

> From: Douglas Gilbert <dougg@torque.net>
> Subject: Re: RFC: SCSI Generic version 4 interface
> Date: Tue, 21 Nov 2006 17:06:46 -0500
> 
> > FUJITA Tomonori wrote:
> > > From: Douglas Gilbert <dougg@torque.net>
> > > Subject: RFC: SCSI Generic version 4 interface
> > > Date: Mon, 06 Nov 2006 16:47:30 -0500
> > > 
> > >>                 SCSI Generic version 4 interface structure
> > >>                 ==========================================
> > >>                                              Version 1.1
> > >>
> > >> Goals:
> > > 
> > > (snip)
> > > 
> > >>   - same structure can be used for a synchronous (e.g. interruptible
> > >>     ioctl) or asynchronous (e.g. ioctl()/read() ) pass through.
> > > 
> > > Can you provide details on "asynchronous" part?
> > 
> > I was trying to keep away from the implementation details
> > but various people have pushed for more details ...
> > 
> > 
> > Well as you are aware, the sg driver uses a write(),
> > poll_or_signal,read() sequence to do asynchronous IO
> > in version 3 (and roughly the same model in version 2).
> > 
> > That write() has always made me uneasy and it is accident
> > prone.
> 
> I think write and read interface works nicely (though it's accident
> prone as you said). They enable user-space applications to easily send
> and receive sg_io_v4 structures asynchronously in batches (v3 can't in
> batches). As you know, the current bsg adopts this interface.
> 
> Another possible interface that I prefer is to send and receive
> sg_io_v4 structures via two ring buffers.

One problem about using non ioctl interfaces (read/write, ring buffer,
whatever) is iovec compat issue. If sg4 still supports iovec for data
transfer, it needs compat stuff (like sg_build_iovec). And non ioctl
interfaces with iovec need to do more nasty compat stuff (like
drivrs/input/evdev.c does).

So I think that it's great for sg v4 not to support the old
data-transfer interfaces (iovec, dio, and indirect io).

Re: RFC: SCSI Generic version 4 interface

[Jeremy Linton]

On Monday 06 November 2006 15:47, Douglas Gilbert wrote:
> I was asked to put together a proposal in May this
> year for a new SCSI Generic interface structure. This

> Feel free to make suggestions.
	OK...

	A unified multiple device mmap buffer interface for would probably be useful 
as well. This is to get around the problems inherent in the current design 
that keep more than one device from using a mmap/reserve buffer allocated for 
another device. 
	What I would personally like to be able to do with the SG interface is have a 
shared memory region where I can kick off a few dozen tagged commands to a 
device, get notification when they have completed and then send a similar 
command to a second device, all without involving the CPU in any copy 
operations, or too many page management operations.
	In my dream world this would require some kind of tagging of commands as well 
so I don't have to handle a bunch of async completions for commands in a 
group that need to complete as a group.
	
	

Re: RFC: SCSI Generic version 4 interface

[Matthew Wilcox]

On Wed, Nov 08, 2006 at 10:48:01AM -0600, Jeremy Linton wrote:
> 	A unified multiple device mmap buffer interface for would probably be useful 
> as well. This is to get around the problems inherent in the current design 
> that keep more than one device from using a mmap/reserve buffer allocated for 
> another device. 
> 	What I would personally like to be able to do with the SG interface is have a 
> shared memory region where I can kick off a few dozen tagged commands to a 
> device, get notification when they have completed and then send a similar 
> command to a second device, all without involving the CPU in any copy 
> operations, or too many page management operations.
> 	In my dream world this would require some kind of tagging of commands as well 
> so I don't have to handle a bunch of async completions for commands in a 
> group that need to complete as a group.

This is all starting to sound/feel a lot like the current proposed
design for netchannels: http://lwn.net/Articles/206699/

If we did an AF_SCSI (I've seen it mooted before), then I suppose we
could just use netchannels.

The other route is tweaking the current netchannels proposal to be a bit
less networking-centric and use kiocbs (or something ...) to hold the
data that's being transmitted.

Re: RFC: SCSI Generic version 4 interface

[FUJITA Tomonori]

From: Matthew Wilcox <matthew@wil.cx>
Subject: Re: RFC: SCSI Generic version 4 interface
Date: Wed, 8 Nov 2006 10:22:18 -0700

> On Wed, Nov 08, 2006 at 10:48:01AM -0600, Jeremy Linton wrote:
> > 	A unified multiple device mmap buffer interface for would probably be useful 
> > as well. This is to get around the problems inherent in the current design 
> > that keep more than one device from using a mmap/reserve buffer allocated for 
> > another device. 
> > 	What I would personally like to be able to do with the SG interface is have a 
> > shared memory region where I can kick off a few dozen tagged commands to a 
> > device, get notification when they have completed and then send a similar 
> > command to a second device, all without involving the CPU in any copy 
> > operations, or too many page management operations.
> > 	In my dream world this would require some kind of tagging of commands as well 
> > so I don't have to handle a bunch of async completions for commands in a 
> > group that need to complete as a group.
> 
> This is all starting to sound/feel a lot like the current proposed
> design for netchannels: http://lwn.net/Articles/206699/
> 
> If we did an AF_SCSI (I've seen it mooted before), then I suppose we
> could just use netchannels.
> 
> The other route is tweaking the current netchannels proposal to be a bit
> less networking-centric and use kiocbs (or something ...) to hold the
> data that's being transmitted.

netchannels works if we can make it less networking-centric, though
ring buffer would be preferable for the target code since netchannels
needs to allocate and free skbuffs and do copies again and again.

Re: RFC: SCSI Generic version 4 interface

[FUJITA Tomonori]

From: Jeremy Linton <jli@greshamstorage.com>
Subject: Re: RFC: SCSI Generic version 4 interface
Date: Wed, 8 Nov 2006 10:48:01 -0600

> 
> 
> 
> On Monday 06 November 2006 15:47, Douglas Gilbert wrote:
> > I was asked to put together a proposal in May this
> > year for a new SCSI Generic interface structure. This
> 
> > Feel free to make suggestions.
> 	OK...
> 
> 	A unified multiple device mmap buffer interface for would probably be useful 
> as well. This is to get around the problems inherent in the current design 
> that keep more than one device from using a mmap/reserve buffer allocated for 
> another device. 
> 	What I would personally like to be able to do with the SG interface is have a 
> shared memory region where I can kick off a few dozen tagged commands to a 
> device, get notification when they have completed and then send a similar 
> command to a second device, all without involving the CPU in any copy 
> operations, or too many page management operations.
> 	In my dream world this would require some kind of tagging of commands as well 
> so I don't have to handle a bunch of async completions for commands in a 
> group that need to complete as a group.

Sounds like it's quite similar to what the target code (tgt) is doing
with its own ring buffer interface.

As discussed months ago, tgt has a lot in common with sg. We would
like to have the new sg flexible enough to handle various messages
that tgt needs. This will also require lots of changes to the block
layer functions to map user-space pages.