[PATCH] bsg: SAS SMP pass-through documentation

[PATCH] bsg: SAS SMP pass-through documentation

[Douglas Gilbert]

[-- Attachment #1: Type: text/plain, Size: 210 bytes --]

This is a patch against scsi-misc-2.6 .

ChangeLog:
   - new documentation file for the bsg driver pass-through
     being used to send SAS SMP functions

Signed-off-by: Douglas Gilbert <dgilbert@interlog.com>

[-- Attachment #2: bsg_smp_doco.patch --]
[-- Type: text/x-patch, Size: 10504 bytes --]

diff --git a/Documentation/scsi/bsg_sas_smp.txt b/Documentation/scsi/bsg_sas_smp.txt
new file mode 100644
index 0000000..12274ee
--- /dev/null
+++ b/Documentation/scsi/bsg_sas_smp.txt
@@ -0,0 +1,221 @@
+                       SAS SMP PASS-THROUGH VIA BSG
+                       ----------------------------
+
+Introduction
+============
+Serial Attached SCSI (SAS) has a Serial Management Protocol (SMP) which
+is mainly used to communicate with SAS expanders. SAS expanders typically
+have between 12 and 36 phys and permit many disks (e.g. more than 8)
+to be connected to a SAS HBA or RAID controller.
+
+In SAS interconnects that include one or more expanders, the kernel uses
+SMP to discover SAS and SATA devices during boot up. The kernel also
+responds to a BROADCAST(Change) received from an expander which indicates
+some change in the disposition of connected SAS and SATA devices.
+
+SAS expanders are powerful devices and user space programs may be
+required to access and manage them. New features such as phy-based zoning
+were added in SAS-2 and this increases the need for user space programs
+to access expanders.
+
+The SAS-2 standard outlines two ways to control expanders:
+  a) via SMP from an SMP initiator which is found in most SAS HBAs and
+     and RAID controllers
+  b) an Out of Band communication method which SAS-2 does not define
+     but nominates ethernet as an example
+
+This document outlines using method a) via the "block SCSI pass-through"
+(bsg) driver to access SAS expanders.
+
+
+The bsg driver
+==============
+bsg is an acronym for Block SCSI Generic. The "block" used in the name of
+this driver is a little confusing since each bsg device node (e.g.
+/dev/bsg/0:0:0:0 corresponds to /dev/sda which is a SATA disk on this
+laptop) is actually a "char" device. The "block" term refers to the
+kernel layer which conveys both blocked data and messages to lower layers
+(such as the SCSI layer).
+
+bsg driver device nodes that can be used for SMP pass-through have names
+like /dev/bsg/expander-6:0 and /dev/bsg/sas_host6 . "expander-6:0" can be
+interpreted as the first expander (origin 0) found by (SAS) host number 6.
+There might be another SAS HBA in the machine (e.g. host 7) also connected
+to the same expander and that might lead to another bsg device node like
+/dev/bsg/expander-7:0 .
+
+There is also related information in sysfs. For example,
+/sys/class/bsg/expander-6:0/dev might contain a string like "254:2". Those
+are the major and minor device numbers of the corresponding bsg device node:
+  # ls -l /dev/bsg/expander-6:0
+  crw------- 1 root root 254, 2 Jul 29 13:11 /dev/bsg/expander-6:0
+
+Also the SAS address of the expander can be found with:
+  # cat /sys/class/sas_device/expander-6:0/sas_address
+  0x5001517e85c3efff
+
+
+SMP Frames
+==========
+SMP is a simple protocol, much simpler than the Serial SCSI protocol (SSP)
+and the Serial ATA Tunneled Protocol (STP) which are also defined in the
+SAS standards. That simplicity translates to more work for the user space
+programs using an SMP pass-through and a somewhat delicate division of work
+between the user space and the SAS HBA driver.
+
+SMP functions are sent to an SMP target (e.g. an expander) in an SMP request
+frame and the response is returned in an SMP response frame. Both request
+and response frames have a maximum size of 1028 bytes. Both request and
+response frames include a 4-byte header at the start of the frame and
+a 4-byte CRC at the end of the frame.
+
+The contents of an SMP REQUEST frame header (counting origin 0) are:
+  - [byte 0]: SMP Frame type: 0x40
+  - [byte 1]: SMP function: 0x0 to 0xbf (see SAS); 0xc0 to 0xff
+              vendor specific
+  - [byte 2] [SAS-2]: allocated response length: expected response
+              length in dwords (4-byte unit) excluding header and CRC
+  - [byte 3] [SAS-2]: request length in dwords excluding this header
+              and the CRC
+
+In SAS-1 (and SAS-1.1) bytes 2 and 3 are "reserved" which means they
+should be set to zero but are typically ignored by an SMP target. In
+this case the request and response lengths are implicit depending on the
+given SMP function (byte 1).
+
+The contents of an SMP RESPONSE frame header are:
+  - [byte 0]: SMP Frame type: 0x41
+  - [byte 1]: SMP function: echoes byte 1 in the request.
+  - [byte 2]: function result: 0x0 for success
+  - [byte 3] [SAS-2]: response length in dwords excluding this header
+              and the CRC
+
+In SAS-1 (and SAS-1.1) byte 3 is "reserved" which means it is typically
+set to zero by an SMP target. The response length is implicit depending on
+the associated SMP function (byte 1).
+
+Note that the largest request and response length is 255 (i.e. 0xff) which
+equates to 1020 bytes and when the 4-byte header and CRC are added, the
+maximum request and response frames are both 1028 bytes long.
+
+A user space program using the bsg SMP pass-through is responsible for
+setting up the request header and checking the returned response header.
+Existing bsg pass-through implementations generate and check the CRC in
+the HBA driver. This frees the user space program from that responsibility
+but leaves open the question of whether user space programs should pass
+through space for the CRC and what should be placed in that field. Current
+usage is to pass through the full frame (i.e. including the CRC) and place
+zero bytes in that field. The response frame allocation should also allow
+for the full frame (i.e. including the CRC) to be returned.
+
+
+Pass-through
+============
+Depending on the distribution the header file for the bsg driver might
+be found at one of these locations:
+  /usr/include/linux/bsg.h
+  /lib/modules/<kernel_version>/include/linux/bsg.h
+
+The first is preferred but requires a recent glibc version. The second
+one is in the kernel source and leads to a "don't use kernel source"
+warning during the C compilation that can safely be ignored.
+
+The general procedure is to open a bsg device node, use the resulting
+file descriptor to invoke an SG_IO ioctl and then close the file
+descriptor. The SG_IO ioctl sends the SMP request frame and then waits
+for the corresponding SMP response.
+
+The bsg device node should be opened O_RDWR with the open(2) system call.
+This will require root permissions (perhaps CAP_IO_RAW is sufficient). An
+instance of the sg_io_v4 structure needs to be built then a pointer to
+that instance passed as the SG_IO ioctl's third argument. Here is a small
+code fragment without error checking:
+
+    int fd;
+    struct sg_io_v4 req_resp;
+
+    memset(&req_resp, 0, sizeof(req_resp);
+    fd = open("/dev/nsg/expander-6:0", O_RDWR);
+    // place data in the fields of a_req_resp
+    ioctl(fd, SG_IO, &req_resp);
+    // check and process the response
+    close(fd);
+
+The sg_io_v4 structure was designed for SCSI commands but is general enough
+to be used with many protocols. SCSI has two types of transfers going
+from an HBA to a disk: SCSI commands (requests) and optionally data
+associated with commands like WRITE, so-called data_out. In return the
+SCSI initiator might get a single byte status, a sense buffer (response)
+and data associated with commands like READ, so-called "data_in".
+
+SMP only needs request and response frames and these are mapped to the
+sg_io_v4 structure's data_out and data_in buffers, respectively. Here is
+a code fragment to send an SMP REPORT MANUFACTURER INFORMATION function
+request and allow for its response:
+
+    unsigned char req[] = {0x40, 0x1, 0xE, 0x0, 0, 0, 0, 0};
+    unsigned char resp[(0xE * 4) + 8];    /* 64 bytes */
+    /* Note allow for CRC in request and response */
+    unsigned char cmd[16];
+
+    /* unused fields have been zeroed (e.g. by a memset() shown above) */
+    req_resp.guard = 'Q';
+    req_resp.protocol = BSG_PROTOCOL_SCSI;
+    req_resp.subprotocol = BSG_SUB_PROTOCOL_SCSI_TRANSPORT;
+
+    req_resp.request_len = sizeof(cmd);      /* unused */
+    req_resp.request = (uintptr_t) cmd;
+
+    req_resp.dout_xfer_len = sizeof(req);
+    req_resp.dout_xferp = (uintptr_t) req;
+
+    req_resp.din_xfer_len = sizeof(resp);	/* 64 */
+    req_resp.din_xferp = (uintptr_t) resp;
+
+    hdr.timeout = 20000;    /* i.e. 20 seconds in millisecond units */
+
+In SAS-2 the SMP REPORT MANUFACTURER INFORMATION response is 64 bytes
+long including header and CRC. Processing the response starts by checking
+the value returned by the ioctl() system call. If that is -1 then the
+operation has failed and errno should be checked. If the ioctl succeeded
+then the following fields in the sg_io_v4 structure instance should be
+checked: driver_status, transport_status and device_status. If they are
+all zero then the number of valid bytes in the buffer pointed to by
+din_xferp should be (din_xfer_len - din_resid).
+
+In SAS-2 the first 4 bytes of the response should be: 0x41, 0x1, 0x0, 0xe .
+If the third byte is other than 0x0 then SMP function result is indicating
+an error. If the HBA passes back the CRC field in the response then
+din_resid will most likely be 0; if the HBA does not pass back the CRC
+field then din_resid will most likely be 4; either is okay.
+
+How the HBA reports errors such as a missing expander (e.g. cable
+unexpectedly removed), transport congestion (e.g. in SSP this can lead to
+aborted commands) and timeouts has not been agreed.
+
+
+Example code
+============
+The author has written a package of command line utilities called
+smp_utils (or smp-utils using Debian conventions). The source can
+be found in the Download section of this page:
+    http://sg.danny.cz/sg/smp_utils.html
+The bsg SMP pass-through specific code is in the sgv4 directory. To
+allow the utilities to use other pass-throughs (e.g. proprietary Linux
+and other OSes such as FreeBSD's CAM extensions) an abstraction layer
+sits between each utility and the bsg SMP pass-through. The abstraction
+layer is defined in smp_utils include/smp_utils.h header (see struct
+smp_req_resp).
+
+
+Non-expander use of SMP
+=======================
+In the section on "The bsg driver" above, /dev/bsg/sas_host6 was given as
+an example of a bsg SMP device node. That is a SAS HBA itself rather than
+an expander. The reason is that miniSAS 8087 cables carry sideband signals
+using the SGPIO protocol which is somewhat like I2C. These sideband
+signals may be controlled by the SMP READ GPIO and WRITE GPIO functions.
+
+
+
+Douglas Gilbert 20110802

Re: [PATCH] bsg: SAS SMP pass-through documentation

[FUJITA Tomonori]

On Wed, 10 Aug 2011 23:35:42 -0400
Douglas Gilbert <dgilbert@interlog.com> wrote:

> +The bsg driver
> +==============
> +bsg is an acronym for Block SCSI Generic. The "block" used in the name of
> +this driver is a little confusing since each bsg device node (e.g.
> +/dev/bsg/0:0:0:0 corresponds to /dev/sda which is a SATA disk on this
> +laptop) is actually a "char" device. The "block" term refers to the
> +kernel layer which conveys both blocked data and messages to lower layers

What 'blocked data' means?

The "block" term simply means that bsg is for the users of 'the block
layer' (including SCSI)?

Re: [PATCH] bsg: SAS SMP pass-through documentation

[Douglas Gilbert]

On 11-08-11 03:51 AM, FUJITA Tomonori wrote:
> On Wed, 10 Aug 2011 23:35:42 -0400
> Douglas Gilbert<dgilbert@interlog.com>  wrote:
>
>> +The bsg driver
>> +==============
>> +bsg is an acronym for Block SCSI Generic. The "block" used in the name of
>> +this driver is a little confusing since each bsg device node (e.g.
>> +/dev/bsg/0:0:0:0 corresponds to /dev/sda which is a SATA disk on this
>> +laptop) is actually a "char" device. The "block" term refers to the
>> +kernel layer which conveys both blocked data and messages to lower layers
>
> What 'blocked data' means?

Data sent through the block layer in which the corresponding command
has an implicit block size. A SCSI READ(10) is an example of such
a command.

> The "block" term simply means that bsg is for the users of 'the block
> layer' (including SCSI)?

You are welcome to send a patch which does:
   s/both blocked data and//

Re: [PATCH] bsg: SAS SMP pass-through documentation

[Randy Dunlap]

On Wed, 10 Aug 2011 23:35:42 -0400 Douglas Gilbert wrote:

> This is a patch against scsi-misc-2.6 .
> 
> ChangeLog:
>    - new documentation file for the bsg driver pass-through
>      being used to send SAS SMP functions
> 
> Signed-off-by: Douglas Gilbert <dgilbert@interlog.com>

Acked-by: Randy Dunlap <rdunlap@xenotime.net>

James usually merges Documentation/scsi/ patches, so I'll leave it for
him unless he wants me to merge it.

Thanks,
---
~Randy
*** Remember to use Documentation/SubmitChecklist when testing your code ***