SG_IO weirdness

SG_IO weirdness

[Cameron, Steve]

I noticed that when I do two SG_IO ioctls to a target
device (say, tape drive, disk drive, whatever) in which
the first request is well formed (e.g. an inquiry) and the 
second one has a malformed CDB, such that it gets check condition 
with sense key == 5 (ILLEGAL REQUEST), the data buffer returned for 
the second malformed SG_IO request is filled out with the same 
data as was returned for the first successful command (e.g. the 
same inquiry data again.)  I'm using separate data buffers for
the two commands, and memsetting them to zero before calling
ioctl().  I don't think this data is coming from the device,
as it happens with every device I've tried.

Is that normal?  Seems like for a malformed request, the 
data buffer should not be transferred at all, much less
transferred with contents of a prior request's data buffer.

Kernel is 2.6.18 from kernel.org.

The two requests were done from the same process, I haven't
tried two separate processes to see if one process could
by this method access another process's data.  I did try
using two devices, so the first well formed command went 
to one device, and the 2nd, malformed command went to another
device.  In that case, I didn't get the same buffer back again,
but garbage. (some recognizeable strings, "en_US" was in there...)

Is this a problem, or is this a matter of "just don't do that."?

Thanks,

-- steve

Re: SG_IO weirdness

[Douglas Gilbert]

Cameron, Steve wrote:
> I noticed that when I do two SG_IO ioctls to a target
> device (say, tape drive, disk drive, whatever) in which
> the first request is well formed (e.g. an inquiry) and the 
> second one has a malformed CDB, such that it gets check condition 
> with sense key == 5 (ILLEGAL REQUEST), the data buffer returned for 
> the second malformed SG_IO request is filled out with the same 
> data as was returned for the first successful command (e.g. the 
> same inquiry data again.)  I'm using separate data buffers for
> the two commands, and memsetting them to zero before calling
> ioctl().  I don't think this data is coming from the device,
> as it happens with every device I've tried.
> 
> Is that normal?  Seems like for a malformed request, the 
> data buffer should not be transferred at all, much less
> transferred with contents of a prior request's data buffer.
> 
> Kernel is 2.6.18 from kernel.org.

Steve,
Even though the SCSI status is CHECK CONDITION, the data-in
buffer may still be transferred. One obvious example
is a READ command when the sense key is RECOVERED ERROR.

The sg driver and I suspect the block layer SG_IO do
not check the SCSI status to determine whether or not
to transfer the data-in buffer (or where it would have
been DMA-ed to if the command worked) back to user space.
If it was _direct_ IO then the block layer SG_IO and the
sg driver would have no control over the data-in transfer
(apart from setting it up).

Both the sg driver and the block layer SG_IO could check
the resid field which a LLD should set after a DMA
(especially inbound). However LLDs are not compelled to
set resid properly.

So a few questions:
 - block layer SG_IO, the sg driver or both?
 - indirect IO (i.e. O_DIRECT not set)?
 - did the offending process have superuser permissions?
 - did the resid field indicate a short data-in transfer?

> The two requests were done from the same process, I haven't
> tried two separate processes to see if one process could
> by this method access another process's data.  I did try
> using two devices, so the first well formed command went 
> to one device, and the 2nd, malformed command went to another
> device.  In that case, I didn't get the same buffer back again,
> but garbage. (some recognizeable strings, "en_US" was in there...)
> 
> Is this a problem, or is this a matter of "just don't do that."?

As long as the SCSI status and sense buffer are conveyed
back properly _and_ this is only observed when the
process has superuser permissions, then I wouldn't
regard it as serious. Others may disagree.

Doug Gilbert

RE: SG_IO weirdness

[Cameron, Steve]

> Steve,
> Even though the SCSI status is CHECK CONDITION, the data-in
> buffer may still be transferred. One obvious example
> is a READ command when the sense key is RECOVERED ERROR.

Yep, of course.

> The sg driver and I suspect the block layer SG_IO do
> not check the SCSI status to determine whether or not
> to transfer the data-in buffer (or where it would have
> been DMA-ed to if the command worked) back to user space.
> If it was _direct_ IO then the block layer SG_IO and the
> sg driver would have no control over the data-in transfer
> (apart from setting it up).
>
> Both the sg driver and the block layer SG_IO could check
> the resid field which a LLD should set after a DMA
> (especially inbound). However LLDs are not compelled to
> set resid properly.
>
> So a few questions:
> - block layer SG_IO, the sg driver or both?

sg driver.

> - indirect IO (i.e. O_DIRECT not set)?

indirect IO, O_DIRECT was not set.

> - did the offending process have superuser permissions?
Yes.

> - did the resid field indicate a short data-in transfer?

resid == 64, the requested buffer was 1088 bytes.
(If I interpret that right, it means that all but 64 
bytes were transferred, that is, 1024 bytes were 
transferred?  Odd, considering the CDB was nonsense.)

> As long as the SCSI status and sense buffer are conveyed
> back properly _and_ this is only observed when the
> process has superuser permissions, then I wouldn't
> regard it as serious. Others may disagree.

I haven't tried it as non-superuser.  (And couldn't,
unless I chmod'ed /dev/sg* )

-- steve

Re: SG_IO weirdness

[Douglas Gilbert]

Cameron, Steve wrote:
>> Steve,
>> Even though the SCSI status is CHECK CONDITION, the data-in
>> buffer may still be transferred. One obvious example
>> is a READ command when the sense key is RECOVERED ERROR.
> 
> Yep, of course.
> 
>> The sg driver and I suspect the block layer SG_IO do
>> not check the SCSI status to determine whether or not
>> to transfer the data-in buffer (or where it would have
>> been DMA-ed to if the command worked) back to user space.
>> If it was _direct_ IO then the block layer SG_IO and the
>> sg driver would have no control over the data-in transfer
>> (apart from setting it up).
>>
>> Both the sg driver and the block layer SG_IO could check
>> the resid field which a LLD should set after a DMA
>> (especially inbound). However LLDs are not compelled to
>> set resid properly.
>>
>> So a few questions:
>> - block layer SG_IO, the sg driver or both?
> 
> sg driver.
> 
>> - indirect IO (i.e. O_DIRECT not set)?
> 
> indirect IO, O_DIRECT was not set.
> 
>> - did the offending process have superuser permissions?
> Yes.
> 
>> - did the resid field indicate a short data-in transfer?
> 
> resid == 64, the requested buffer was 1088 bytes.
> (If I interpret that right, it means that all but 64 
> bytes were transferred, that is, 1024 bytes were 
> transferred?  Odd, considering the CDB was nonsense.)

Steve,
>From memory, between SPC-2 and SPC-3 the INQUIRY allocation
length field went from 8 bits to 16 bits. If you do the
above calculation modulo 256 it comes out correct :-)

The moral here is don't set INQUIRY lengths > 252
unless the target can handle it. There is no point
anyway for a "standard" INQUIRY (EVPD=0, CmdDt is
obsolete). With VPD pages you can do a double fetch,
the first one 4 bytes long to pick up "page length"
field.
But then again you said the cdb was nonsense.


Now it is still a bit fuzzy because there is the
allocation length field in some cdbs and the dxfer_len
given to sg_io_hdr. I would think that the LLD
should concentrate on the latter and set resid
accordingly. That makes me wonder about the LLD
involved (the sg driver just passes resid through).

>> As long as the SCSI status and sense buffer are conveyed
>> back properly _and_ this is only observed when the
>> process has superuser permissions, then I wouldn't
>> regard it as serious. Others may disagree.
> 
> I haven't tried it as non-superuser.  (And couldn't,
> unless I chmod'ed /dev/sg* )

The sg driver zeros out the scatter gather elements
for non-superusers.
chmod'ing is not always needed, for example a non-superuser
may well have permissions on a USB cd/dvd drive (including
the sg device node) in some distributions.

Doug Gilbert

RE: SG_IO weirdness

[Cameron, Steve]

> Steve,
> From memory, between SPC-2 and SPC-3 the INQUIRY allocation
> length field went from 8 bits to 16 bits. If you do the
> above calculation modulo 256 it comes out correct :-)
>
> The moral here is don't set INQUIRY lengths > 252
> unless the target can handle it. There is no point
> anyway for a "standard" INQUIRY (EVPD=0, CmdDt is
> obsolete). With VPD pages you can do a double fetch,
> the first one 4 bytes long to pick up "page length"
> field.
> But then again you said the cdb was nonsense.

yeah, the 2nd command was not inquiry, it is a vendor
specific command.  when I reduced the buffer size < 256,
it began to work (not get check condition/sense key == 5).
I think I just have some misunderstanding about the CDB
format for xfer sizes > 256, but that is my problem,
not yours.  Mainly, I just thought SG_IO was behaving weird,
so I thought I should mention it.

It occurs to me that if I do this, below, after the SG_IO ioctl, 
it suppresses the extra data transferred (obviously):

memset(buffer + buffer_size - sgio.resid, 0, sgio.resid);

Though you mentioned that not all devices/LLD set resid?

> Now it is still a bit fuzzy because there is the
> allocation length field in some cdbs and the dxfer_len
> given to sg_io_hdr. I would think that the LLD
> should concentrate on the latter and set resid
> accordingly. That makes me wonder about the LLD
> involved (the sg driver just passes resid through).

Hmm, I was able to get the same behavior from mptsas,
qla2xxx, and cciss with SAS drives, HP MSA1000, and tape
drives for target devices, respectively, so not sure the 
LLD matters.

[...]

Thanks for taking the time to explain some of this
to me.

I can try to distill out a small test program, if it
would be of interest.

-- steve