Time to allow MODE SENSE for USB disk-type storage devices?

Time to allow MODE SENSE for USB disk-type storage devices?

[Alan Stern]

Matt & Patrick:

Maybe it's time to think about improving the situation with MODE SENSE for 
USB devices.  I don't know that page 0x08 is particularly crucial for USB 
devices, but page 0x3f (for the write-protect setting) certainly would be 
good to have.

Traces taken from various Windows systems show that Windows 2000
consistently issues MODE SENSE for page 0x3f, while Windows XP does both
page 0x3f and page 0x08.  Now it's entirely possible that certain
vendor-specific drivers cause Windows to skip those steps, but on the
whole we ought to be pretty safe if we do them in exactly the same way as
Windows.  For page 0x3f, anyway.  (If necessary, we can add an
unusual_devs flag bit to usb-storage for devices that can't handle page
0x3f.)

There are two important things to note.  First, Windows _always_ uses a
transfer length of 192 for these commands.  None of this "Let's just get
the header" or "Only request as much as we need" wishy-washiness.  
Second, Windows uses MODE SENSE(6) for Transparent Scsi (USB SubClass =
0x06) and MODE SENSE(10) for everything else -- according to Pat LaVarre
anyway, and it agrees with the trace data.  (MODE SELECT is treated the
same way, but that's irrelevant for our purposes.)

The usb-storage driver can fix the 6-byte vs. 10-byte issue by clearing or
setting the appropriate flag during the slave_configure() callback.  Is it 
possible to force the 192-byte transfer lengths in a similar way?

On a related note, we've run across reports from several users indicating 
that their USB devices die when handed a PREVENT-ALLOW MEDIUM REMOVAL 
command, even though they set the Removable-Medium flag in their INQUIRY 
data.  Could there be a device flag that would cause the sd driver to 
avoid these commands?

Alan Stern

Re: [usb-storage] Time to allow MODE SENSE for USB disk-type storage de vices?

[Pat LaVarre]

> On a related note, we've run across reports from several users
> indicating that their USB devices die when handed a PREVENT-ALLOW
> MEDIUM REMOVAL command, even though they set the Removable-Medium flag
> in their INQUIRY data.  Could there be a device flag that would cause
> the sd driver to avoid these commands?

I hear op x1E Prevent/ Allow appears in Windows bus traces.

I wonder what cbw/ sequence avoids killing such devices for Windows.

> Windows uses MODE SENSE(6) for Transparent Scsi (USB SubClass =
> 0x06) and MODE SENSE(10) for everything else -- according to Pat LaVarre
> anyway, and it agrees with the trace data.  (MODE SELECT is treated the
> same way, but that's irrelevant for our purposes.)

(-: I doubt I ever said anything so easily understood. :-)

More precisely I imagine I actually have said that my limited experience
of ATAPI & USB Windows bus traces confirms the published portions of
Microsoft's host-vendor-specific definition of USB Mass
bInterfaceSubClass x06, in particular:

--- http://www.google.com/search?q=microsoft+design+usb
---
--- second hit today
--- http://www.microsoft.com/whdc/hwdev/bus/USB/default.mspx
---
--- FAQ for USB Storage Support in Windows
--- http://www.microsoft.com/whdc/hwdev/bus/usb/usbfaq.mspx

Usbstor.sys and Device Class Support
Q: Which device classes does usbstor.sys understand?
In Windows XP, Windows 2000, and Windows Me, usbstor.sys makes only one
distinction: 

bInterfaceSubClass == 06h 
- OR - 
bInterfaceSubClass != 06h 

The value bInterfaceSubClass == 06h means that:

Command descriptor blocks (CDBs) should not be padded to 12 bytes.

Mode Sense / Mode Select commands should not be translated from 1AH /
15h to 5AH / 55h.

Subclass 0x06 should generally be used for flash memory devices.

The value bInterfaceSubClass != 06h means that:

CDBs should be padded to 12 bytes.

Mode Sense / Mode Select commands should be translated from 1AH / 15h to
5AH / 55h.

---

Pat LaVarre

P.S.

I wonder if our collected Windows USB Mass traces show that t10.org SBC
neglects to emphasise op x23 is HOST-vendor-specific.

P.P.S.

If we choose to teach Linux to talk more like Windows here, then of
course I hope we still allow ioctl SG_IO, CDROM_SEND_PACKET, etc. to
pass SCSI thru transparently, especially to bInterfaceSubClass x06.

I remember thinking in/correctly that devices of
drivers/usb/storage/unusual_devs.h connect transparently only with a
patched kernel.

Re: [usb-storage] Time to allow MODE SENSE for USB disk-typestorage de vices?

[Pat LaVarre]

> > On a related note,

I wish I weren't hijacking your thread, but ...

> > we've run across reports from several users
> > indicating that their USB devices die when handed a PREVENT-ALLOW
> > MEDIUM REMOVAL command, even though they set the Removable-Medium flag
> > in their INQUIRY data.  Could there be a device flag that would cause
> > the sd driver to avoid these commands?
> 
> I hear op x1E Prevent/ Allow appears in Windows bus traces.
> 
> I wonder what cbw [or] sequence [of cbw's]
> avoids killing such devices for Windows.

I have NOT yet learned to trace all Linux cdb's, but ...

I happen to have an Atapi bus analyser connected just now ...

I see a Linux 2.6.4-rc2 host polling an Atapi RMB PDT x05 DVD/CD device
every two seconds with two -y "00 00:00:00 00 00" Test Unit Ready and
then one -y "1E 00:00:00 00 00" Allow.

This leads me to wonder if Linux is polling RMB PDT x00 with x1E Allow?

And if so, why???

Also does the USB dCBWDataLength for Linux x1E Allow differ from
Windows, since the Atapi BCL is, ugh, the zero that assumes Hn = Dn,
rather than the xFFFF and xFFFE Atapi BCL's said to be more massively
distributed.

Pat LaVarre

Re: [usb-storage] Time to allow MODE SENSE for USB disk-type storage devices?

[Andries Brouwer]

On Mon, Mar 08, 2004 at 11:05:38AM -0500, Alan Stern wrote:

> Maybe it's time to think about improving the situation with MODE SENSE for 
> USB devices.  I don't know that page 0x08 is particularly crucial for USB 
> devices, but page 0x3f (for the write-protect setting) certainly would be 
> good to have.

Just to be sure: 0x3f is not the code for some particular page,
but for all available pages.

The write-protect setting is found (if it is found) in the header,
so it does not matter what page(s) we ask for, as long as we get
at least one header.

Andries

Re: [usb-storage] Time to allow MODE SENSE for USB disk-type storage devices?

[Patrick Mansfield]

Andries or others -

Can you elaborate on this comment in sd.c? Do you have a device that hangs
with the current kernel when asking for more than 4 bytes here? Is it USB
storage?

	/*
	 * First attempt: ask for all pages (0x3F), but only 4 bytes.
	 * We have to start carefully: some devices hang if we ask
	 * for more than is available.
	 */
	res = sd_do_mode_sense(SRpnt, 0, 0x3F, buffer, 4, &data);

Per this changeset:

http://linux.bkbits.net:8080/linux-2.5/diffs/drivers/scsi/sd.c@1.53?nav=index.html|src/|src/drivers|src/drivers/scsi|hist/drivers/scsi/sd.c

-- Patrick Mansfield

Re: [usb-storage] Time to allow MODE SENSE for USB disk-type storage devices?

[Andries Brouwer]

On Tue, Mar 09, 2004 at 05:43:53PM -0800, Patrick Mansfield wrote:

> Andries or others -
> 
> Can you elaborate on this comment in sd.c? Do you have a device that hangs
> with the current kernel when asking for more than 4 bytes here? Is it USB
> storage?
> 
> 	/*
> 	 * First attempt: ask for all pages (0x3F), but only 4 bytes.
> 	 * We have to start carefully: some devices hang if we ask
> 	 * for more than is available.
> 	 */
> 	res = sd_do_mode_sense(SRpnt, 0, 0x3F, buffer, 4, &data);
> 
> Per this changeset:
> 
> http://linux.bkbits.net:8080/linux-2.5/diffs/drivers/scsi/sd.c@1.53?nav=index.html|src/|src/drivers|src/drivers/scsi|hist/drivers/scsi/sd.c
> 
> -- Patrick Mansfield

Yes, I changed this code twice, I think, each time because I acquired
a USB device that did not work. The state that I left things in
was carefully crafted in such a way that each of my half a dozen or so
USB card reader devices worked. In the meantime I got a few more readers,
and they all worked (late 2.5).

Of course there is no particular interest in my selection of devices,
and no reason to expect that code exists that works on everything
on the market.

An oft repeated maxim is: do as Windows does - and that may be
a good idea. On the other hand, half of my devices come with their
own drivers, and also fail under Windows without their own driver.

The conclusion is that yes, if you change something here then
almost certainly there will be devices that are broken by the
change, but the change may still be an improvement if there
are more devices that were broken and now function.

Andries

Re: [usb-storage] Time to allow MODE SENSE for USB disk-type storage devices?

[Alan Stern]

On Wed, 10 Mar 2004, Andries Brouwer wrote:

> An oft repeated maxim is: do as Windows does - and that may be
> a good idea. On the other hand, half of my devices come with their
> own drivers, and also fail under Windows without their own driver.
> 
> The conclusion is that yes, if you change something here then
> almost certainly there will be devices that are broken by the
> change, but the change may still be an improvement if there
> are more devices that were broken and now function.

It would be hard to determine the actual numbers for comparison.  On the
other hand, if a device requires a special driver for Windows then perhaps
it's not so bad if it also requires a special SCSI device-flags entry for
Linux.

Another obvious but important point to keep in mind is that whatever
changes are made, they must work correctly with actual (not emulated or
USB) SCSI devices.

Alan Stern

Re: [usb-storage] Time to allow MODE SENSE for USB disk-type storag edevices?

[Pat LaVarre]

> > res = sd_do_mode_sense(SRpnt, 0, 0x3F, buffer, 4, &data);

I see the structure of this code arbitrarily requires H = C.  That
structure makes sense only when H = D <= C.  Why we guess D <= C when C
is 4, I do not understand.

> * First attempt: ask for all pages (0x3F), ...

I hear Windows starts with page x08, not x3F.

> all pages (0x3F), but only 4 bytes.

SCSI Pass-Through Mode Sense Command May Crash the Computer
http://support.microsoft.com/?kbid=813908

suggests Talk-like-Windows excludes 4 bytes, by saying -i 4 -y "1A
xx:xx:xx 04 00" crashes the Win 2K kernel for much Atapi, just as -y "12
00:00:00 00 00" crashes some lk for much Usb.

> some devices hang if we ask for more than is available.

Also some hosts hang when D < H.

Offline recently I received an as yet unconfirmed report that 2.4.25
ioctl CDROM_SEND_PACKET fails (but does not hang) when D < H.

> ... my devices ... work...

Yes, here we choose among evils.

If Windows doesn't discover PDT x00/0E writability except by trying to
write blocks, then any other technique we invent will fail to
interoperate broadly.

Pat LaVarre

Re: [usb-storage] Time to allow MODE SENSE for USB disk-type storag edevices?

[Alan Stern]

On 10 Mar 2004, Pat LaVarre wrote:

> > * First attempt: ask for all pages (0x3F), ...
> 
> I hear Windows starts with page x08, not x3F.

That's not so.  Windows 2000 asks for page x1C, then page x3F.  Windows XP 
does the same and then asks for page x08.

> > all pages (0x3F), but only 4 bytes.
> 
> SCSI Pass-Through Mode Sense Command May Crash the Computer
> http://support.microsoft.com/?kbid=813908
> 
> suggests Talk-like-Windows excludes 4 bytes, by saying -i 4 -y "1A
> xx:xx:xx 04 00" crashes the Win 2K kernel for much Atapi, just as -y "12
> 00:00:00 00 00" crashes some lk for much Usb.

I don't think so.  That web page actually states that a bug in Windows' 
atapi.sys storage port driver causes the crash and that an update is 
available to fix the problem.  So it's no longer relevant.  In any case, 
I'm proposing to always ask for 192 bytes (for USB devices).

> > some devices hang if we ask for more than is available.
> 
> Also some hosts hang when D < H.

Not USB hosts.

> Offline recently I received an as yet unconfirmed report that 2.4.25
> ioctl CDROM_SEND_PACKET fails (but does not hang) when D < H.
> 
> > ... my devices ... work...
> 
> Yes, here we choose among evils.
> 
> If Windows doesn't discover PDT x00/0E writability except by trying to
> write blocks, then any other technique we invent will fail to
> interoperate broadly.

For PDT x00, Windows appears to use the same technique that we do: look at
the header from a page x3F response.

Alan Stern

Re: [usb-storage] Time to allow MODE SENSE for USB disk-type storag edevices?

[Pat LaVarre]

> > I hear Windows starts with page x08, not x3F.
> 
> That's not so.  Windows 2000 asks for page x1C, then page x3F.  Windows XP 
> does the same and then asks for page x08.

Did we see Win fetch page x3F after finding page x1C?

> > > all pages (0x3F), but only 4 bytes.
> > 
> > SCSI Pass-Through Mode Sense Command May Crash the Computer
> > http://support.microsoft.com/?kbid=813908
> > 
> > suggests Talk-like-Windows excludes 4 bytes, by saying -i 4 -y "1A
> > xx:xx:xx 04 00" crashes the Win 2K kernel for much Atapi, just as -y "12
> > 00:00:00 00 00" crashes some lk for much Usb.
> 
> I don't think so.  That web page actually states that a bug in Windows' 
> atapi.sys storage port driver causes the crash and

Eh?  Why don't you "think so"?  What did I say that contradicts that
elaboration?

> an update is available to fix the problem.

Yes, but you mentioning this fact interests me greatly.

Can you concisely say why we care if an update is available for a
massively distributed Microsoft bug or no?  Personally I've never yet
understood why Microsoft bothered to write that Win 2K update. 
Microsoft already had shipped Win XP when it appeared.

> So it's no longer relevant.

Eh?  Relevance I still see includes:

a) Talk-SCSI-like-Windows probably doesn't include -i 4 -x "1A xx xx xx
04 00", even for the many significantly different test cases for which
we have not collected bus traces.

b) Device folk who test with net-disconnected labs full of gold Win 2K
are not sending -i 4 -x "1A xx xx xx 04 00" to their Atapi devices.

c) ...

Win 2K was and remains Massively Distributed, of course.

> In any case, 
> I'm proposing to always ask for 192 bytes (for USB devices).

Yes like Windows please.

> > > some devices hang if we ask for more than is available.
> > 
> > Also some hosts hang when D < H.
> 
> Not USB hosts.

No lk ever?

I know some hosts choke over the CSW stall that commonly distinguishes D
< H from D = H in BBB.  The detail of hang vs. fail doesn't always reach
me e.g. maybe I no longer accurately remember how lk rejected -y "12
00:00:00 00 00" Inquiry for zero.

> > Offline recently I received an as yet unconfirmed report that 2.4.25
> > ioctl CDROM_SEND_PACKET fails (but does not hang) when D < H.
> > 
> > > ... my devices ... work...
> > 
> > Yes, here we choose among evils.
> > 
> > If Windows doesn't discover PDT x00/0E writability except by trying to
> > write blocks, then any other technique we invent will fail to
> > interoperate broadly.
> 
> For PDT x00, Windows appears to use the same technique that we do: look at
> the header from a page x3F response.

Ah.  Maybe we can teach lk to properly distinguish my write-protected
PDT x00 disks.  Hmmm.

Pat LaVarre

Re: [usb-storage] Time to allow MODE SENSE for USB disk-type storag edevices?

[Alan Stern]

On 10 Mar 2004, Pat LaVarre wrote:

> > > I hear Windows starts with page x08, not x3F.
> > 
> > That's not so.  Windows 2000 asks for page x1C, then page x3F.  Windows XP 
> > does the same and then asks for page x08.
> 
> Did we see Win fetch page x3F after finding page x1C?

More accurately, we see Win fetch page x3F after receiving an error trying 
to fetch page x1C.  I believe I've seen it fetch page x3F after succeeding 
in reading page x1C, but I no longer have those trace files on hand.

> > > SCSI Pass-Through Mode Sense Command May Crash the Computer
> > > http://support.microsoft.com/?kbid=813908
> > > 
> > > suggests Talk-like-Windows excludes 4 bytes, by saying -i 4 -y "1A
> > > xx:xx:xx 04 00" crashes the Win 2K kernel for much Atapi, just as -y "12
> > > 00:00:00 00 00" crashes some lk for much Usb.
> > 
> > I don't think so.  That web page actually states that a bug in Windows' 
> > atapi.sys storage port driver causes the crash and
> 
> Eh?  Why don't you "think so"?  What did I say that contradicts that
> elaboration?

The page doesn't say that reading 4 bytes crashes the kernel; it says that
it _used_ to crash the kernel but the kernel has now been fixed.  Also, it
is misleading to claim that the page implies Talk-Like-Windows excludes
4-byte reads, since we already know that TLW means excluding everything
but 192-byte reads (for USB devices).

> > an update is available to fix the problem.
> 
> Yes, but you mentioning this fact interests me greatly.
> 
> Can you concisely say why we care if an update is available for a
> massively distributed Microsoft bug or no?  Personally I've never yet
> understood why Microsoft bothered to write that Win 2K update. 
> Microsoft already had shipped Win XP when it appeared.

Shucks, some people might say that Win XP is an update for a massively 
distributed Microsoft bug, i.e., Win 2000. :-)

In any case, I don't care about that update.  It doesn't matter, since I'm 
not claiming that we should or shouldn't use 4-byte reads in general.  All 
I'm saying is that for USB devices, we should try using 192-byte MODE 
SENSE transfers.

> > So it's no longer relevant.
> 
> Eh?  Relevance I still see includes:
> 
> a) Talk-SCSI-like-Windows probably doesn't include -i 4 -x "1A xx xx xx
> 04 00", even for the many significantly different test cases for which
> we have not collected bus traces.
> 
> b) Device folk who test with net-disconnected labs full of gold Win 2K
> are not sending -i 4 -x "1A xx xx xx 04 00" to their Atapi devices.
> 
> c) ...
> 
> Win 2K was and remains Massively Distributed, of course.

It may be relevant to what you want (and maybe relevant to Patrick's 
question) but it's not relevant to the main point of this thread.

> > > Also some hosts hang when D < H.
> > 
> > Not USB hosts.
> 
> No lk ever?

No current ones as far as I know.  Of course, there may be cases where it
happens that I'm not aware of.  Bear in mind, software hangs can easily be
fixed; hardware and firmware problems are much harder to deal with -- and
there's no evidence of any such problems with them.

> I know some hosts choke over the CSW stall that commonly distinguishes D
> < H from D = H in BBB.  The detail of hang vs. fail doesn't always reach
> me e.g. maybe I no longer accurately remember how lk rejected -y "12
> 00:00:00 00 00" Inquiry for zero.

That was a plain and simple bug in the usb-storage driver; it has since 
been fixed.

> > For PDT x00, Windows appears to use the same technique that we do: look at
> > the header from a page x3F response.
> 
> Ah.  Maybe we can teach lk to properly distinguish my write-protected
> PDT x00 disks.  Hmmm.

That's what I hope to accomplish.

Alan Stern