bytes/CDB of SCSI pass thru grossly limited maybe

bytes/CDB of SCSI pass thru grossly limited maybe

[Pat LaVarre]

Offline a friend reports seeing bytes/CDB of Linux ioctl SG_IO grossly
limited - like down below 0.000000128 GB/CDB for op x3B "WRITE BUFFER".

I'm writing now to give this audience the chance to comment.

Left to myself, I'll hope to work to reproduce that result, compare USB/
PATAPI/ FireWire/ SATAPI, try the alternative of ioctl
CDROM_SEND_PACKET, ... and report back here if/when I discover anything
interesting.

I remember seeing talk of tweaking the kernel to choke off the default
kernel thruput whenever status/ command overhead is high, but before now
I don't remember hearing of choking off SCSI pass thru as well.

Pat LaVarre
http://linux-pel.blog-city.com/read/752705.htm

Re: bytes/CDB of SCSI pass thru grossly limited maybe

[Jens Axboe]

On Fri, Jul 30 2004, Pat LaVarre wrote:
> Offline a friend reports seeing bytes/CDB of Linux ioctl SG_IO grossly
> limited - like down below 0.000000128 GB/CDB for op x3B "WRITE BUFFER".

Could you have chosen a more awkward way to express that number? I can't
even make sense of it if I try, 0.128 bytes/cdb?! And are you talking
about the length of the cdb, or do you mean the data that it can
send/receive? I'd guess the latter, but then cdb isn't the correct term.

So in short, please try again :-)

-- 
Jens Axboe

Re: bytes/CDB of SCSI pass thru grossly limited maybe

[Pat LaVarre]

Jens A:

> in short, please try again :-)

I much appreciate your patient encouragement.  I now know I can ask:

Is there an ide-cd option to allow more bytes/CDB?

That is, an option analogous to the /sys/block/sr*/device/max_sectors
setting of usb-storage, allowing up to the xFF:FF LBA's/CDB standard at
t10.org?  Else can I help create such an option?

I see ioctl SG_IO chokes via EIO if bytes/CDB goes much above zero.

I ask for a workaround because a friend of mine cares.

I see EIO at 130 KiB/CDB, though happy at 128 KiB/CDB, when I try
ide-cd.  I see usb-storage defaults to choke thruput off at 120 KiB/CDB,
but thru /sys lets us reach a little farther above zero i.e. 512
KiB/CDB.

How coordinated is our 2.6 effort to choke off thruput near zero across
PATAPI, USB, FireWire, SPI, SATAPI, etc.?

Is there an ide-cd option to allow more bytes/CDB?

Pat LaVarre

--- tty log

$ uname -msr
Linux 2.6.8.1 i686
$
$ dc -e '120 2 / 16dio 78Pp 800 * 78Pp'
x3C
x1E000
$ dc -e '128 2 / 16dio 78Pp 800 * 78Pp'
x40
x20000
$ dc -e '512  2 / 16dio 78Pp 800 * 78Pp'
x100
x80000
$
$ sudo plscsi -i x1E000 -x "28 00 00:00:00:00 00 00:3C 00" /dev/scd1
$ sudo plscsi -i x1E800 -x "28 00 00:00:00:00 00 00:3D 00" /dev/scd1 2>&1 | grep ioctl:
// sgioSay.ioctl: Input/output error
$ sudo plscsi -i x20000 -x "28 00 00:00:00:00 00 00:40 00" /dev/hdc
$ sudo plscsi -i x20800 -x "28 00 00:00:00:00 00 00:41 00" /dev/hdc 2>&1 | grep ioctl:
// sgioSay.ioctl: Input/output error
$
$ sudo strace plscsi -i x1E000 -x "28 00 00:00:00:00 00 00:3C 00" /dev/scd1 2>&1 | egrep '(open|ioctl)\(' | tail -3
open("/dev/scd1", O_RDONLY|O_NONBLOCK)  = 3
ioctl(3, 0x2282, 0xbfffd9dc)            = 0
ioctl(3, 0x2285, 0x804f0a8)             = 0
$ grep O_RDONLY /usr/include/bits/fcntl.h
#define O_RDONLY             00
$
$ grep . /sys/block/*/device/max_sectors
/sys/block/sr0/device/max_sectors:240
/sys/block/sr1/device/max_sectors:240
$
$ sudo echo 65535 >/sys/block/sr1/device/max_sectors
bash: /sys/block/sr1/device/max_sectors: Permission denied
$ sudo su
# echo 65535 >/sys/block/sr1/device/max_sectors
# grep . /sys/block/*/device/max_sectors
/sys/block/sr0/device/max_sectors:240
/sys/block/sr1/device/max_sectors:240
# echo 1028 >/sys/block/sr1/device/max_sectors
# grep . /sys/block/*/device/max_sectors
/sys/block/sr0/device/max_sectors:240
/sys/block/sr1/device/max_sectors:240
# echo 1024 >/sys/block/sr1/device/max_sectors
# grep . /sys/block/*/device/max_sectors
/sys/block/sr0/device/max_sectors:240
/sys/block/sr1/device/max_sectors:1024
# exit
$
$ sudo plscsi -i x80000 -x "28 00 00:00:00:00 00 01:00 00" /dev/scd1
$ sudo plscsi -i x80800 -x "28 00 00:00:00:00 00 01:01 00" /dev/scd1 2>&1 | grep ioctl:
// sgioSay.ioctl: Input/output error
$

--- how awful my English is

> > Offline a friend reports seeing bytes/CDB of Linux ioctl SG_IO
> > grossly limited - like down below 0.000000128 GB/CDB for op x3B
> > "WRITE BUFFER".
> 
> Could you have chosen a more awkward way to express that number?

Numbers near zero are hard to quote, sure.

0.000000128 GB = 0.000128 MB = 0.128 KB, which isn't what I meant.

> I can't even make sense of it if I try, 0.128 bytes/cdb?!

Another source of imprecision is the GiB MiB KiB vs. GB MB KB
distinction.

I meant to say I was hearing that we choke off thruput near 128 KiB/CDB,
but also that 128 KiB/CDB is very near zero when measured in the year
2004, for example 128 KiB/CDB is barely above 0.000000119 GB/CDB.

$ dc -e '33 k 128 1024 / 1024 / 1024 / p'
.000000119209289550781250000000000
$

I was failing to make myself more clear by speaking in the more popular
units of decimal and GB rather than in the more natural units of hex and
GiB.

> And are you talking about the length of the cdb, or do you mean the
> data that it can send/receive? I'd guess the latter,

Yes.

> but then cdb isn't the correct term.

Sorry I don't yet see why this thread doesn't fit into the category of
arbitrary limits on bytes per CDB.

What is the correct linux-scsi jargon for this?

Pat LaVarre
http://linux-pel.blog-city.com/read/752705.htm

Re: bytes/CDB of SCSI pass thru grossly limited maybe

[Pat LaVarre]

> > And are you talking about the length of the cdb, or do you mean the
> > data that it can send/receive? I'd guess the latter,
> 
> Yes.
> 
> > but then cdb isn't the correct term.
> 
> Sorry I don't yet see why this thread doesn't fit into the category of
> arbitrary limits on bytes per CDB.

Kindly offline someone highlighted for me the ambiguity among the
arbitrary limits on the length of command, data, and status bytes
associated with a CDB.

Here I mean to be asking how to retune the arbitrary ide-cd limits on
the length of:

        data bytes/CDB of SCSI pass thru

Pat LaVarre

Re: bytes/CDB of SCSI pass thru grossly limited maybe

[Jens Axboe]

On Mon, Aug 16 2004, Pat LaVarre wrote:
> Jens A:
> 
> > in short, please try again :-)
> 
> I much appreciate your patient encouragement.  I now know I can ask:
> 
> Is there an ide-cd option to allow more bytes/CDB?
> 
> That is, an option analogous to the /sys/block/sr*/device/max_sectors
> setting of usb-storage, allowing up to the xFF:FF LBA's/CDB standard at
> t10.org?  Else can I help create such an option?
> 
> I see ioctl SG_IO chokes via EIO if bytes/CDB goes much above zero.
> 
> I ask for a workaround because a friend of mine cares.
> 
> I see EIO at 130 KiB/CDB, though happy at 128 KiB/CDB, when I try
> ide-cd.  I see usb-storage defaults to choke thruput off at 120 KiB/CDB,
> but thru /sys lets us reach a little farther above zero i.e. 512
> KiB/CDB.
> 
> How coordinated is our 2.6 effort to choke off thruput near zero across
> PATAPI, USB, FireWire, SPI, SATAPI, etc.?
> 
> Is there an ide-cd option to allow more bytes/CDB?

Ok, makes more sense to me now. The reason why I get confused is that I
don't consider the cdb more than the actual 10-16 bytes of command data
bytes, not the full structure of it.

But yes, different devices will have different max transfer values. As
I'm sure you know, ATAPI cannot do more than 128KiB per command. And
that is what q->max_sectors is set to, if you try and submit more than
this through SG_IO, bio_add_user() will complain and command will be
failed as you experienced.

max_sectors should be fully exposed so you don't have to guess.

-- 
Jens Axboe

Re: bytes/CDB of SCSI pass thru grossly limited maybe

[Pat LaVarre]

Jens A:

>> Is there an ide-cd option to allow more bytes/CDB?
> ...
> ATAPI cannot do more than 128KiB per command. And
> that is what q->max_sectors is set to, if you try and submit more than
> this through SG_IO, bio_add_user() will complain and command will be
> failed as you experienced.

Wow.

Please reply to confirm - you are saying 128 KiB data bytes per command  
is a Linux PATAPI design limit?

I know even twelve-byte-packet PATAPI itself as defined at t13.org and  
t10.org can do xFFFF = (64 Ki - 1) LBA's via such ten-byte per CDB ops  
as x28 "READ (10)" and x2A "WRITE (10)".  At 2 KiB/LBA, that's 128 MiB  
minus 2 KiB.  That's the kind of limit I expected, when instead in  
Linux I saw limits like 120 KiB in SCSI over USB.  Also I hear  
sixteen-byte-packet PATAPI can itself pass xFFFF:FFFF data blocks per  
command.

In parallel, I'll click thru to bio_add_user to read the limits there.

> As I'm sure you know,

I didn't know, thank you, sorry I have no clue, I'm not trying to  
pretend I do, I got into this at the request of a friend who needs 256  
KiB data per command.

I specifically have Not yet tried patching the Linux kernel to persuade  
PATAPI to allow more data bytes per command.  I did try USB.  So far  
I've failed.  At:

Subject: [usb-storage] faq max_sectors
https://lists.one-eyed-alien.net/pipermail/usb-storage/2004-August/ 
000680.html

I have reported that raising SCSI_DEFAULT_MAX_SECTORS alone does not  
actually allow more data bytes per command in USB.

> max_sectors should be fully exposed so you don't have to guess.

Yes please.

Pat LaVarre
http://linux-pel.blog-city.com/read/752705.htm

Re: bytes/CDB of SCSI pass thru grossly limited maybe

[Jens Axboe]

On Mon, Aug 23 2004, Pat LaVarre wrote:
> Jens A:
> 
> >>Is there an ide-cd option to allow more bytes/CDB?
> >...
> >ATAPI cannot do more than 128KiB per command. And
> >that is what q->max_sectors is set to, if you try and submit more than
> >this through SG_IO, bio_add_user() will complain and command will be
> >failed as you experienced.
> 
> Wow.
> 
> Please reply to confirm - you are saying 128 KiB data bytes per command  
> is a Linux PATAPI design limit?
> 
> I know even twelve-byte-packet PATAPI itself as defined at t13.org and  
> t10.org can do xFFFF = (64 Ki - 1) LBA's via such ten-byte per CDB ops  
> as x28 "READ (10)" and x2A "WRITE (10)".  At 2 KiB/LBA, that's 128 MiB  
> minus 2 KiB.  That's the kind of limit I expected, when instead in  
> Linux I saw limits like 120 KiB in SCSI over USB.  Also I hear  
> sixteen-byte-packet PATAPI can itself pass xFFFF:FFFF data blocks per  
> command.

What you can shove into the cdb is of lesser interest than the fact that
ATAPI itself cannot do more than 128KiB _per command_.

> In parallel, I'll click thru to bio_add_user to read the limits there.

There are no extra limits there, it's just in the queue itself
(max_sectors, segments, etc).

> >As I'm sure you know,
> 
> I didn't know, thank you, sorry I have no clue, I'm not trying to  
> pretend I do, I got into this at the request of a friend who needs 256  
> KiB data per command.

??? You must have some clue on the actual hardware, if you have been
working on the hardware side for many years :-)

> I specifically have Not yet tried patching the Linux kernel to persuade  
> PATAPI to allow more data bytes per command.  I did try USB.  So far  
> I've failed.  At:
> 
> Subject: [usb-storage] faq max_sectors
> https://lists.one-eyed-alien.net/pipermail/usb-storage/2004-August/ 
> 000680.html
> 
> I have reported that raising SCSI_DEFAULT_MAX_SECTORS alone does not  
> actually allow more data bytes per command in USB.

Ok, let me try to explain. There are several factors that limit how big
a command you can send to a device - some of these are hardware, some of
these are just imposed by the driver. If you look inside the
request_queue structure in blkdev.h, you'll find such things as:

        unsigned short          max_sectors;
        unsigned short          max_phys_segments;
        unsigned short          max_hw_segments;
        unsigned int            max_segment_size;

which are set by the driver and limit how big a request you can submit
through SG_IO.

-- 
Jens Axboe

Re: bytes/CDB of SCSI pass thru grossly limited maybe

[Pat LaVarre]

Jens A:

> What you can shove into the cdb is of lesser interest than the fact 
> that
> ATAPI itself cannot do more than

Yes.

> the request_queue structure in blkdev.h, you'll find such things as:
>
>         unsigned short          max_sectors;
>         unsigned short          max_phys_segments;
>         unsigned short          max_hw_segments;
>         unsigned int            max_segment_size;
>
> which are set by the driver and limit how big a request you can submit
> through SG_IO.

I will return to study this further.

Sorry I still do not yet immediately understand precisely who is 
limiting what.

I do not see how these 16 and 32 bit limits correspond to a 128 KiB 
limit on how many bytes of data may accompany a CDB.  Instead I see:

(1 << (4 + 16)) is 1 Mi.
(1 << (9 + 16)) is 32 Mi.
(1 << (11 + 16)) is 128 Mi.

(1 << 9) is 0.5 Ki, often = B/LBA of HDD
(1 << 11) is 2 Ki, often = B/LBA of DVD/CD.
(1 << 12) is 4 Ki, often = B/page of x86 virtual memory
(1 << 16) is 64 Ki.
(1 << 32) is 4 Ti.

No 128 Ki anywhere.

> What you can shove into the cdb is of lesser interest than the fact 
> that
> ATAPI itself cannot do more than 128KiB _per command_.

Possibly I have misunderstood what is "ATAPI itself"?

The PIO and DMA I know in PATAPI do not limit data bytes per command.  
Even PIO limits only data bytes per DRQ, not per command.

I think PATAPI itself is the standard launched by SFF 8020i that 
appears now at ANSI/ NCITS t13.org.  I see that standard as a transport 
for SCSI over IDE, with no definite limit on data per command, though 
arguably the UDMA CRC may grow less reliable if the host and device 
neglect to cooperate to insert such CRC often into the data.

I see PATAPI does limit max bytes of CDB, to 16 (x10) else to 12 (xC), 
by neglecting to enumerate any other CDB-containing packet lengths for 
the op xA1 "IDENTIFY" data to request.

Pat LaVarre

Re: bytes/CDB of SCSI pass thru grossly limited maybe

[Jens Axboe]

On Mon, Aug 23 2004, Pat LaVarre wrote:
> Jens A:
> 
> >What you can shove into the cdb is of lesser interest than the fact 
> >that
> >ATAPI itself cannot do more than
> 
> Yes.
> 
> >the request_queue structure in blkdev.h, you'll find such things as:
> >
> >        unsigned short          max_sectors;
> >        unsigned short          max_phys_segments;
> >        unsigned short          max_hw_segments;
> >        unsigned int            max_segment_size;
> >
> >which are set by the driver and limit how big a request you can submit
> >through SG_IO.
> 
> I will return to study this further.
> 
> Sorry I still do not yet immediately understand precisely who is 
> limiting what.
> 
> I do not see how these 16 and 32 bit limits correspond to a 128 KiB 
> limit on how many bytes of data may accompany a CDB.  Instead I see:
> 
> (1 << (4 + 16)) is 1 Mi.
> (1 << (9 + 16)) is 32 Mi.
> (1 << (11 + 16)) is 128 Mi.
> 
> (1 << 9) is 0.5 Ki, often = B/LBA of HDD
> (1 << 11) is 2 Ki, often = B/LBA of DVD/CD.
> (1 << 12) is 4 Ki, often = B/page of x86 virtual memory
> (1 << 16) is 64 Ki.
> (1 << 32) is 4 Ti.
> 
> No 128 Ki anywhere.
> 
> >What you can shove into the cdb is of lesser interest than the fact 
> >that
> >ATAPI itself cannot do more than 128KiB _per command_.
> 
> Possibly I have misunderstood what is "ATAPI itself"?
> 
> The PIO and DMA I know in PATAPI do not limit data bytes per command.  
> Even PIO limits only data bytes per DRQ, not per command.
> 
> I think PATAPI itself is the standard launched by SFF 8020i that 
> appears now at ANSI/ NCITS t13.org.  I see that standard as a transport 
> for SCSI over IDE, with no definite limit on data per command, though 
> arguably the UDMA CRC may grow less reliable if the host and device 
> neglect to cooperate to insert such CRC often into the data.
> 
> I see PATAPI does limit max bytes of CDB, to 16 (x10) else to 12 (xC), 
> by neglecting to enumerate any other CDB-containing packet lengths for 
> the op xA1 "IDENTIFY" data to request.

Sorry my clowning around, I'm too ATA damaged. You can play with bigger
transfers with something ala this applied, will allow you to pass in <=
512KiB/command.

===== drivers/ide/ide-cd.c 1.89 vs edited =====
--- 1.89/drivers/ide/ide-cd.c	2004-08-13 02:03:53 +02:00
+++ edited/drivers/ide/ide-cd.c	2004-08-23 20:15:52 +02:00
@@ -3138,6 +3138,7 @@
 
 	blk_queue_prep_rq(drive->queue, ide_cdrom_prep_fn);
 	blk_queue_dma_alignment(drive->queue, 31);
+	blk_queue_max_sectors(drive->queue, 1024);
 	drive->queue->unplug_delay = (1 * HZ) / 1000;
 	if (!drive->queue->unplug_delay)
 		drive->queue->unplug_delay = 1;

-- 
Jens Axboe

Re: bytes/CDB of SCSI pass thru grossly limited maybe

[Pat LaVarre]

Jens A:

> You can play with bigger transfers with something ala this applied,
> will allow you to pass in <= 512KiB/command.

Exactly what we needed, thank you.  As quoted below, I immediately found
I could run up as far as 1 MiB = 2048 * 512 bytes, no worries.

Any chance of a max_sectors setting appearing in /sys for ide-cd ATAPI =
SCSI over IDE, like we have already for SCSI over USB and SCSI over
FireWire?

> ATA damaged.

I remember ATA has a max_sectors of about xFF (255) in the 28-bit LBA
protocol, only the new 48-bit protocol allows xFFFF (65535).  Much
theory argues that x00 and x0000 should represent x100 and x10000, but
hosts often don't take that risk.

Pat LaVarre

+++ linux-2.6.9-rc1-bk2-pel/drivers/ide/ide-cd.c	2004-08-26 17:08:52.944300264 -0600
@@ -3142,6 +3142,7 @@ int ide_cdrom_setup (ide_drive_t *drive)
 
 	blk_queue_prep_rq(drive->queue, ide_cdrom_prep_fn);
 	blk_queue_dma_alignment(drive->queue, 31);
+	blk_queue_max_sectors(drive->queue, 2048);
 	drive->queue->unplug_delay = (1 * HZ) / 1000;
 	if (!drive->queue->unplug_delay)
 		drive->queue->unplug_delay = 1;

Re: bytes/CDB of SCSI pass thru grossly limited maybe

[Jeff Garzik]

Jens Axboe wrote:
> Ok, makes more sense to me now. The reason why I get confused is that I
> don't consider the cdb more than the actual 10-16 bytes of command data
> bytes, not the full structure of it.


Speaking of which...  (WARNING: change of topic; or perhaps, getting 
back on topic :))


I would like to see support for CDBs larger than 16 bytes in struct request.

Problem:  Would like to support SCSI's variable length CDBs, and support 
  non-{ATA,SCSI} RAID CDBs which are often larger than 16 bytes, such as 
Promise's SX8 (formerly known as carmel).

Suggested solution:  Probably this involves adding a flag and a pointer, 
rather than increasing the actual CDB buffer size in struct request.

	Jeff

Re: bytes/CDB of SCSI pass thru grossly limited maybe

[Pat LaVarre]

> Speaking of which...  (WARNING: change of topic; or perhaps, getting 
> back on topic :))

Part of my initial ambiguity is now clear to me.  I said max 
"bytes/CDB" to mean "max data bytes per CDB", which is more often in 
dispute than "max bytes of CDB", which was nonetheless what some of us 
thought I meant.

>> Ok, makes more sense to me now. The reason why I get confused is that 
>> I
>> don't consider the cdb more than the actual 10-16 bytes of command 
>> data
>> bytes, not the full structure of it.

That English doesn't yet make sense to me.  I don't know what "the full 
structure" is.

I agree CDB and data and status and auto sense all form part of the 
bits copied between device and host in each transaction.  For the 
rendezvous itself I have no clearly appropriate and unambiguous English 
term.

I see the word "command" can mean the whole transaction or just the CDB.

I suppose CDB stands for "command data bytes".  That expansion doesn't 
help, does it?  That phrase uses the word "data" in a pointless 
redundancy I think - in that context, there are no "bytes" which are 
not "data".

> like to see support for CDBs larger than 16 bytes in struct request.

Often xC (12), not even x10 (16), is the CDROM_PACKET_SIZE limit of 
struct cdrom_generic_command of include/linux/cdrom.h

My survey at sourceforge.net iomrrdtools rrd iomscsi xscsi.h is:

/* Recommend a limit for the length in bytes of the CDB.
  *
  * Consider the limits:
  *
  *      xC      // common max of SCSI over IDE
  *      xC      // common max of SCSI over 1394 i.e. xC = (x20 - x14)
  *      xC      // seen as max of Linux ioctl CDROM_SEND_PACKET
  *      x10     // max SCSI over IDE
  *      x10     // max SCSI over USB
  *      x10     // max declared SCSI thru Dos Aspi or Win32 Ioctl
  *      x10     // max SCSI-3 for standard op xXX other than x7F
  *      xFF     // common max of SCSI pass thru
  *      x104    // max SCSI-3 permits for op x 7F XX XX = 8 + xFC
  *      x107    // max SCSI-3 expresses for op x 7F XX XX = 8 + xFF
  *      x3E8    // max SCSI over 1394 = ((xFF * 4) - x14)
  *
  * Arguably 2001-07 SPC-2 r20: "4.3.3" "The variable length CDB 
formats" says
  * SK ASC = x 5 24 "ILLEGAL REQUEST" "INVALID FIELD IN CDB" may mean 
CDB too
  * long to copy out, rather than implying recover by flipping CDB bits.
  */

#define MAX_CDB 0xFF

Pat LaVarre

Re: bytes/CDB of SCSI pass thru grossly limited maybe

[Luben Tuikov]

> I would like to see support for CDBs larger than 16 bytes in struct 
> request.
> 
> Problem:  Would like to support SCSI's variable length CDBs, and support
>   non-{ATA,SCSI} RAID CDBs which are often larger than 16 bytes, such as
> Promise's SX8 (formerly known as carmel).
> 
> Suggested solution:  Probably this involves adding a flag and a pointer,
> rather than increasing the actual CDB buffer size in struct request.

Not sure that you need a flag.
Op. code 0x7F implies (cdb->acdb != 0 && cdb[7] > 0), else bug.

		Luben

Re: bytes/CDB of SCSI pass thru grossly limited maybe

[Jeff Garzik]

Luben Tuikov wrote:
>> I would like to see support for CDBs larger than 16 bytes in struct 
>> request.
>>
>> Problem:  Would like to support SCSI's variable length CDBs, and support
>>   non-{ATA,SCSI} RAID CDBs which are often larger than 16 bytes, such as
>> Promise's SX8 (formerly known as carmel).
>>
>> Suggested solution:  Probably this involves adding a flag and a pointer,
>> rather than increasing the actual CDB buffer size in struct request.
> 
> 
> Not sure that you need a flag.
> Op. code 0x7F implies (cdb->acdb != 0 && cdb[7] > 0), else bug.


At the struct request level a flag is needed.  A "CDB" may be a SCSI 
CDB, an ATA taskfile, an I2O command, etc.  The block layer must treat 
any cdb buffer as an anonymous blob that gets passed through to a lower 
layer.

	Jeff

Re: bytes/CDB of SCSI pass thru grossly limited maybe

[Luben Tuikov]

Jeff Garzik wrote:
> At the struct request level a flag is needed.  A "CDB" may be a SCSI
> CDB, an ATA taskfile, an I2O command, etc.  The block layer must treat
> any cdb buffer as an anonymous blob that gets passed through to a lower
> layer.

Ah, yes, that's true -- I missed the fact that you were talking
considering the flag at the struct request level.

		Luben

Re: bytes/CDB of SCSI pass thru grossly limited maybe

[Alan Stern]

On Mon, Aug 23 2004, Jens Axboe wrote:

> Ok, let me try to explain. There are several factors that limit how big
> a command you can send to a device - some of these are hardware, some of
> these are just imposed by the driver. If you look inside the
> request_queue structure in blkdev.h, you'll find such things as:
> 
>         unsigned short          max_sectors;
>         unsigned short          max_phys_segments;
>         unsigned short          max_hw_segments;
>         unsigned int            max_segment_size;
> 
> which are set by the driver and limit how big a request you can submit
> through SG_IO.

Is it then correct to say that max_sectors is limited only by the
capabilities of the low-level driver and the hardware?  So that within
usb-storage, for example, where the protocol only limits us to 4 GB per
transfer, there would be no problem allowing max_sectors to go as high as
65535?

I wasn't sure whether there were any other limitations somewhere else in 
the block layer, so for now in usb-storage max_sectors is capped at 
SCSI_DEFAULT_MAX_SECTORS = 1024.  Allowing it to increase by almost a 
factor of 64 could remove a lot of latency.

Alan Stern

Re: bytes/CDB of SCSI pass thru grossly limited maybe

[Jens Axboe]

On Fri, Aug 27 2004, Alan Stern wrote:
> On Mon, Aug 23 2004, Jens Axboe wrote:
> 
> > Ok, let me try to explain. There are several factors that limit how big
> > a command you can send to a device - some of these are hardware, some of
> > these are just imposed by the driver. If you look inside the
> > request_queue structure in blkdev.h, you'll find such things as:
> > 
> >         unsigned short          max_sectors;
> >         unsigned short          max_phys_segments;
> >         unsigned short          max_hw_segments;
> >         unsigned int            max_segment_size;
> > 
> > which are set by the driver and limit how big a request you can submit
> > through SG_IO.
> 
> Is it then correct to say that max_sectors is limited only by the
> capabilities of the low-level driver and the hardware?  So that within

Yes

> usb-storage, for example, where the protocol only limits us to 4 GB per
> transfer, there would be no problem allowing max_sectors to go as high as
> 65535?

If the hardware and driver can handle it, there's no other limit.

> I wasn't sure whether there were any other limitations somewhere else in 
> the block layer, so for now in usb-storage max_sectors is capped at 
> SCSI_DEFAULT_MAX_SECTORS = 1024.  Allowing it to increase by almost a 
> factor of 64 could remove a lot of latency.

How do you come to that conclusion? 1024 is probably a fine value,
usually there's very little benefit to going higher than that (since
huge requests on slower hardware increases latency quite a lot, and huge
requests only provide little (if any) throughput increase on fast
hardware).

That said, it still wants splitting into two before you should go
higher.

-- 
Jens Axboe

Re: bytes/CDB of SCSI pass thru grossly limited maybe

[Alan Stern]

On Sat, 28 Aug 2004, Jens Axboe wrote:

> > I wasn't sure whether there were any other limitations somewhere else in 
> > the block layer, so for now in usb-storage max_sectors is capped at 
> > SCSI_DEFAULT_MAX_SECTORS = 1024.  Allowing it to increase by almost a 
> > factor of 64 could remove a lot of latency.
> 
> How do you come to that conclusion? 1024 is probably a fine value,
> usually there's very little benefit to going higher than that (since
> huge requests on slower hardware increases latency quite a lot, and huge
> requests only provide little (if any) throughput increase on fast
> hardware).

Actually I have no basis for saying there will be any significant benefit 
from going higher than 1024.  Pat LaVarre is interested in trying higher 
values, that's all.

> That said, it still wants splitting into two before you should go
> higher.

I don't understand.  What needs to be split into two?  And why?

Alan Stern

Re: bytes/CDB of SCSI pass thru grossly limited maybe

[Jens Axboe]

On Sat, Aug 28 2004, Alan Stern wrote:
> > That said, it still wants splitting into two before you should go
> > higher.
> 
> I don't understand.  What needs to be split into two?  And why?

See earlier discussion on this field - it needs to be split into a 'max
hardware/driver supported value' and a 'optimal real life value' where
the latter could be controlled dynamically by looking at response and
completion times of requests.

Right now ->max_sectors is a grey area, sometimes it's a hardware/driver
limit, often it is not. But we have to treat it as such for now.

Jens

Re: bytes/CDB of SCSI pass thru grossly limited maybe

[James Bottomley]

On Sat, 2004-08-28 at 10:31, Jens Axboe wrote:
> > usb-storage, for example, where the protocol only limits us to 4 GB per
> > transfer, there would be no problem allowing max_sectors to go as high as
> > 65535?
> 
> If the hardware and driver can handle it, there's no other limit.

Actually, if you go through the SCSI stack, you're limited by our SG
list slot allocation which we use a mempool for.  The default maximum is
128, although there's an option to increase this to 256.

These slots are the number of entries in the SG list we allow.  If your
driver disables clustering, and you havea no IOMMU, this limits you to
128*<page size> = 512kb on x86 bytes.  Otherwise, it can go higher
depending on the HW capabilities.  Remember too that a lot of HW limits
the number of SG slots (see the sg_tablesize entry for each driver).

James

Re: bytes/CDB of SCSI pass thru grossly limited maybe

[Jens Axboe]

On Sat, Aug 28 2004, James Bottomley wrote:
> On Sat, 2004-08-28 at 10:31, Jens Axboe wrote:
> > > usb-storage, for example, where the protocol only limits us to 4 GB per
> > > transfer, there would be no problem allowing max_sectors to go as high as
> > > 65535?
> > 
> > If the hardware and driver can handle it, there's no other limit.
> 
> Actually, if you go through the SCSI stack, you're limited by our SG
> list slot allocation which we use a mempool for.  The default maximum is
> 128, although there's an option to increase this to 256.
> 
> These slots are the number of entries in the SG list we allow.  If your
> driver disables clustering, and you havea no IOMMU, this limits you to
> 128*<page size> = 512kb on x86 bytes.  Otherwise, it can go higher
> depending on the HW capabilities.  Remember too that a lot of HW limits
> the number of SG slots (see the sg_tablesize entry for each driver).

Oh yes, I know. bio layer has similar limits, there are no bio_io_vec
entries for > 256 pages. Should have made that more clear, sorry. The sg
table limits are reflected in the queue.

Probably bio and SCSI should use a unified MAX_PAGES define, there's not
much point in bio setting up a 256 page bio_io_vec slab and mempool, if
noone can use it.

-- 
Jens Axboe

Re: bytes/CDB of SCSI pass thru grossly limited maybe

[James Bottomley]

On Sat, 2004-08-28 at 13:55, Jens Axboe wrote:
> Probably bio and SCSI should use a unified MAX_PAGES define, there's not
> much point in bio setting up a 256 page bio_io_vec slab and mempool, if
> noone can use it.

Sure, ours is SCSI_MAX_PHYS_SEGMENTS ... by default, we just use
MAX_PHYS_SEGMENTS from the bio headers for this, which is 128, we could
set this to BIO_MAX_PAGES, sure.

We allow values of 32,64,128 and 256 (the smaller ones were for tiny
systems with SCSI stacks), the largest one for SGI/Qlogic.

However, I did ask the IBM spearker at OLS who was doing the elevator
analysis to retry with 256 instead of 128, and he reported no difference
within error, so I don't think there's much evidence that 256 actually
does anything useful (other than consume a bit of spare memory).

James

Re: bytes/CDB of SCSI pass thru grossly limited maybe

[Jens Axboe]

On Sat, Aug 28 2004, James Bottomley wrote:
> On Sat, 2004-08-28 at 13:55, Jens Axboe wrote:
> > Probably bio and SCSI should use a unified MAX_PAGES define, there's not
> > much point in bio setting up a 256 page bio_io_vec slab and mempool, if
> > noone can use it.
> 
> Sure, ours is SCSI_MAX_PHYS_SEGMENTS ... by default, we just use
> MAX_PHYS_SEGMENTS from the bio headers for this, which is 128, we could
> set this to BIO_MAX_PAGES, sure.
> 
> We allow values of 32,64,128 and 256 (the smaller ones were for tiny
> systems with SCSI stacks), the largest one for SGI/Qlogic.
> 
> However, I did ask the IBM spearker at OLS who was doing the elevator
> analysis to retry with 256 instead of 128, and he reported no difference
> within error, so I don't think there's much evidence that 256 actually
> does anything useful (other than consume a bit of spare memory).

Since SCSI doesn't build > 128 pages anyways, yes it doesn't make sense
to maintain a BIO_MAX_PAGES of 256. Didn't the SCSI part used to be 256
pages as well, I'm pretty sure that's what I put in when the
scsi_malloc() crud was dumped?

bio has 1, 4, 16, 64, 128, 256 pools. 32 might make more sense, I seem
to recall mpages using that. I'll see if I can sneak a BIO_MAX_PAGES
reduction in, and spend that extra pool on 32 instead :)

-- 
Jens Axboe

Re: bytes/CDB of SCSI pass thru grossly limited maybe

[James Bottomley]

On Sat, 2004-08-28 at 14:41, Jens Axboe wrote:
> Since SCSI doesn't build > 128 pages anyways, yes it doesn't make sense
> to maintain a BIO_MAX_PAGES of 256. Didn't the SCSI part used to be 256

Agreed.

> pages as well, I'm pretty sure that's what I put in when the
> scsi_malloc() crud was dumped?

I don't think so.  I only looked at it about six months ago when SGI
wanted better merging for qla.  There was quite a bit of work to get the
maximum to be 256.

> bio has 1, 4, 16, 64, 128, 256 pools. 32 might make more sense, I seem
> to recall mpages using that. I'll see if I can sneak a BIO_MAX_PAGES
> reduction in, and spend that extra pool on 32 instead :)

OK I'll follow whatever you do in SCSI.  I think at least aligning all
of our pool maximums is the correct thing to do.

James

Re: bytes/CDB of SCSI pass thru grossly limited maybe

[Jens Axboe]

On Sun, Aug 29 2004, James Bottomley wrote:
> On Sat, 2004-08-28 at 14:41, Jens Axboe wrote:
> > Since SCSI doesn't build > 128 pages anyways, yes it doesn't make sense
> > to maintain a BIO_MAX_PAGES of 256. Didn't the SCSI part used to be 256
> 
> Agreed.
> 
> > pages as well, I'm pretty sure that's what I put in when the
> > scsi_malloc() crud was dumped?
> 
> I don't think so.  I only looked at it about six months ago when SGI
> wanted better merging for qla.  There was quite a bit of work to get the
> maximum to be 256.

Probably a bad recollection on my part.

> > bio has 1, 4, 16, 64, 128, 256 pools. 32 might make more sense, I seem
> > to recall mpages using that. I'll see if I can sneak a BIO_MAX_PAGES
> > reduction in, and spend that extra pool on 32 instead :)
> 
> OK I'll follow whatever you do in SCSI.  I think at least aligning all
> of our pool maximums is the correct thing to do.

Definitely. I'll toss something out.

-- 
Jens Axboe

Re: bytes/CDB of SCSI pass thru grossly limited maybe

[Pat LaVarre]

Alan S:

>>> bio has 1, 4, 16, 64, 128, 256 pools. 32 might make more sense, I 
>>> seem
>>> to recall mpages using that. I'll see if I can sneak a BIO_MAX_PAGES
>>> reduction in, and spend that extra pool on 32 instead :)
>>
>> OK I'll follow whatever you do in SCSI.  I think at least aligning all
>> of our pool maximums is the correct thing to do.
>
> Definitely. I'll toss something out.

Delighted to see people volunteer work, of course ...

If we're next going to see smaller or larger limits on data bytes per 
CDB, I do not yet know, sorry.

> Actually I have no basis for saying there will be any significant 
> benefit
> from going higher than 1024.  Pat LaVarre is interested in trying 
> higher
> values, that's all.

I think, by changing the host, we'll discover devices that care: maybe 
especially in the mass market low end.

What I know is that in bus traces I see "streaming" fail: I see the 
host and device do Not always keep pace with each other, instead I see 
injected delays of a whole disc revolution.  For example, I see 
14.2857~ms/rev =  70 rev/s = 4200 RPM injected for a device that spins 
at 4200 RPM.

That's a concrete anecdote of pain.  My guess of an explanation is that 
we exceeded the margin in the system design.  The margin I mean is the 
time allowed for the overhead of getting from status in to the next 
command out.  Shattering a stream of gigabytes into microscopic 64 
kebibyte chunks multiplies the time overhead involved from status in to 
command out, by the number of chunks.

How do we predict when that large multiplier exceeds the margin 
available, except by trying?

I agree the device and the host both have to change to observe a 
difference.  Changing either alone gives the I-say-false impression of 
having achieved no effect.  For example, if we try an HDD with hardware 
that splices the chunks back together into a stream, so long as the 
system can burst data much more rapidly than the stream rate we're 
trying to sustain, our thruput survives the abuse of microscopic 
chunks.  But we're also all paying for hardware we don't actually need.

> interested in trying higher values, ...

I know SCSI over whatever, when coded entirely in software, as is 
common for USB/ FireWire/ PATAPI/ SATAPI/ etc., has no inherent limit 
on time or data per CDB.  Only the accelerator engine in a PCI adapter 
enforces irritating rules like pinning all data in memory before the 
command begins.  USB & FireWire do have summary descriptors for the 
command that limit data length, but not sharply, for example the USB 
limit is 31 or 32 bits i.e. 2 to 4 GiB - 1 B.  PATAPI & SATAPI have no 
limit whatsoever, though moving more data may for the first time test 
more than one UDMA CRC per CDB.

> interested in trying higher values, ...

I know I'll benefit if we get up to and beyond 1 MiB of data per CDB 
because,

I want to encourage ever more device developers to switch to Linux and 
free & open drivers from proprietary systems that can pass all of free 
memory per CDB.

SCSI-over-whatever device developers occasionally want as much as 128 
MiB per CDB, merely to exercise the standard test case of xFFFF LBA's 
in op x28 "READ (10") or op x2A "WRITE (10)", which at 2 KiB/LBA is 128 
MiB - 2 KiB.  Without such permission, they have to resort to arguing 
that testing less is the equal of testing more.

Pat LaVarre

Re: bytes/CDB of SCSI pass thru grossly limited maybe

[Pat LaVarre]

Alan S:

> > interested in trying higher values, ...
> 
> a) whole revolution delays seen in bus traces
> b) no limit below 2 to 4 GiB in the standard for the bus
> c) device developers need xFFFF blocks per CDB

Also,

d) experience of lower values

I see usb-storage.ko runs 3X slow like ub.ko does if I ask to choke off
usb-storage at 4 KiB/CDB, same as ub.ko gives me by default today.

That is, we visibly run out of margin to allow for pointlessly
shattering streams into microscopic fragments at values maybe not far
enough from our max.

Pat LaVarre

$ sudo blockdev --flushbufs /dev/uba
BLKFLSBUF: Inappropriate ioctl for device
$ time sudo dd if=/dev/uba bs=64K skip=0 count=3200 >/dev/null
3200+0 records in
3200+0 records out
 
real    0m26.481s
user    0m0.007s
sys     0m0.598s
$ 
$ modprobe -r ub
$ modprobe usb-storage
$
$ ln -s /sys/devices/pci0000:00/0000:00:1d.7/usb1/1-8/1-8:2.0/host1/1:0:0:0/max_sectors .
$ echo 240 | sudo dd of=max_sectors
0+1 records in
0+1 records out
$ cat max_sectors
240
$ sudo blockdev --flushbufs /dev/scd0
$ time sudo dd if=/dev/scd0 bs=1M skip=0 count=200 >/dev/null
200+0 records in
200+0 records out
 
real    0m8.420s
user    0m0.006s
sys     0m0.630s
$
$ echo 8 | sudo dd of=max_sectors
0+1 records in
0+1 records out
$ cat max_sectors
8
$ sudo blockdev --flushbufs /dev/scd0
$ time sudo dd if=/dev/scd0 bs=1M skip=0 count=200 >/dev/null
200+0 records in
200+0 records out
 
real    0m27.464s
user    0m0.002s
sys     0m0.766s
$
$ rm max_sectors
$