Re: Some quick scsi documentation questions:

Re: Some quick scsi documentation questions:

[Rob Landley]

On Thursday 26 July 2007 9:15:11 am James Bottomley wrote:
> On Wed, 2007-07-25 at 23:28 -0400, Rob Landley wrote:
> > 1) Is the summary of the scsi subsystem in appendix A of your 2002 OLS
> > paper still more or less current?  (The scsi mid-layer document never
> > seems to talk about the _upper_ layer it interfaces with.  Which is
> > presumably the block layer...)
>
> No, the block layer is the service layer.  Upper layer is the SCSI
> driver layer (top level drivers like sd, sr and st).

Now that I've noticed that http://sg.torque.net/scsi/SCSI-2.4-HOWTO.html and 
http://sg.torque.net/scsi/SCSI-Generic-HOWTO.html are separate documents, I 
see where that's documented. :)

So upper layer drivers provide /dev nodes, lower level talks to SCSI hardware, 
and the mid-layer understands scsi commands ala http://www.t10.org/scsi-3.htm  

Where _does_ the block layer work into this?

> > 2) Documentation/scsi/scsi_fc_transport.txt is talking about Fiber
> > Channel, right?  (It never expands the fc, not even in the title...)
>
> Yes.

Already sent a separate email about that one.  (Although I now see that 
t10.org inclues some fiber channel standards, and what the relationship 
between it and t11 is I have no idea.)

> > 3) After reading Documentation/scsi/scsi-generic.txt I'm curious what the
> > relationships between a given sg device and sd device.  Each sd has a
> > corresponding sg?  (I can't find where it explicitly says this, but it's
> > a char device, not a block device, so...)  They're always in the same
> > order? Does _each_ sd have an sg?  Can I call the SG_IO ioctl on an sd? 
> > Is there an obvious document on this I've missed?
>
> This is one of those evolutionary problems in Linux.  Originally sg
> existed simply to provide the SG_IO ioctl to SCSI devices.  Then,
> because it was extremely useful, we made it a core feature of the block
> layer (in block/scsi_ioctl.c) which extended it as a general packet
> command feature to devices beyond SCSI.

But it's still a char device, so you presumably can't mount /dev/sg1 to read 
the CD in your burner.  You need to burn the CD and mount the CD through 
separate devices.

To someone like me from the IDE world, the numbering on scsi is weird.  It 
won't leave holes like the ATA driver did to avoid renumbering categories of 
devices that can't be moved without a screwdriver.  (I realize there _are_ 
truly dynamic devices, but the hard drive in my laptop is not one of them and 
it's an artifact of the scsi layer that it's considered so.)

So if you don't attach an sg to every sd, then either there are holes, or they 
don't line up, in which case how do you tell which sg goes with a given sd?

> The logical thing then became to 
> attach sg as an upper driver *only* to devices which otherwise wouldn't
> have one ... however, the sg maintainer has never accepted this.

Do you mean only to devices that otherwise wouldn't have an upper driver?  (So 
there would be an sg for devices that have _no_ sd?  This implies that the 
answer to my earlier question about whether the SG_IO ioctl can be used on sd 
block devices is "yes"?  (In which case the bit where the sg document talked 
about the nonblocking write to sg thing being useful for queueing commands 
comes back up, is there currently a way to do that through the sd ioctl?)

It sounds like your preferred solution would be to have either an sg _or_ an 
sd for each scsi device, so there would be no duplicates but only "no holes" 
if you look at _both_ groups and interlace them.  And the sg maintainer went 
with "have both for every device"?

> Various distros have tried it at various times, though, so currently
> it's safe to say that if sg is loaded, every SCSI device has an sg node,
> but it's not safe to assume that universally.

Ok, so currently, on distros using recent 2.6 kernels, there's an sg 
corresponding to every sd, using the same number.  Right.

Did the various distros patch the kernel, or did they do udev rules to filter 
out the sg they didn't want?  Any idea which distros those were, and whether 
or not they're still relevant?

> We can't even agree on 
> unifying the SG_IO paths, so we have two separate codings, one in sg.c
> and one in block/scsi_ioctl.c.

What would unifying the SG_IO paths mean?  Would it have any impact on 
userspace, or is it just an implementation detail inside the kernel?

> >  seems that a read-only cd will be an sd, but you talk to a burner
> > through an sg...)
>
> No, they're both sr.

What was that again?  (Googles...)  Oh yeah, the "upper layer" drivers from 
the scsi-2.4-howto.  I thought scsi-tape and scsi-cdrom were obsolete, but I 
guess not. 

Back in the IDE world, my hard drive was /dev/hda and my CD burner 
was /dev/hdc.  When /dev/hdc gained the ability to be used directly as a cd 
burner, I just assumed you could do it to sd as well.  My initial mental 
model was "sg gives you a char device to pass through scsi commands for a 
random scsi device, and sd gives you a way to look at it as a block device", 
which doesn't explain sr and st but as I said I thought they were obsolete...

> Users have great difficulty understanding how to 
> map sg to sr (or sd), so we're trying to encourage them only to use the
> actual device node, so /dev/sdx or /dev/srn.

Actually, these days everybody I know uses /dev/cdrom, expects it to be a 
symlink, and doesn't care where it points.  Last time I used a system that 
had more than one of the suckers I believe it had /dev/cdrom0, /dev/cdrom1, 
etc.

I note that in the ubuntu system on my laptop, /dev/sr0 is a symlink 
to /dev/scd0, which is an actual device node with a major and minor that 
presumably means something to the system, but not to me personally.  (Not 
that I consider Ubuntu 7.04 much of a model on how to arrange devices 
considering it put a UUID label on every _partition_ on my hard drive and 
mounts them by label in /etc/fstab rather than that having udev do its job 
and make stable symlinks.)

But it seems that users aren't the only ones who have trouble with this, 
distro maintainers do too.

Question: how much of the difference between /dev/sd* and /dev/sr* do users 
actually have to care about these days?  (And is st still relevant, or does 
sg cover that?)

> > 4) Is http://www.tldp.org/HOWTO/SCSI-Generic-HOWTO/ still current, given
> > it was last updated in 2002?
>
> Probably not ... the transport classes all come after that.

Unfortunately, Documentation/scsi/scsi-generic.txt (which seems a logical 
starting point) is essentially nothing more than a link to that document and 
a note about using scsi drivers as modules.

It also still seems to be the best introduction to sd vs sr vs sg.  It would 
be nice if there was some kind of quick reference card listing the 
concepts...

Similarly, Documentation/scsi/scsi.txt is a link to the SCSI-2.4-HOWTO and a 
note about using scsi drivers as modules.  (And the first time I read those 
two, I didn't notice that the SCSI-2.4-HOWTO and the SCSI-Generic-HOWTO were 
two separate documents...)

> > 5) I notice there's a "man 4 sd", from 1992.  Still relevant?
>
> Yes .. but only as far as it goes, which isn't very far.

I noticed.  It'd be nice if it at least explained the differences between sd, 
sr, st, and sg.  I'll ping the man-pages maintainer about it...

> Taking this to linux-scsi might be the best course ...

And now that I've confirmed it's ok with you, I've cc'd this thread there.

> Doug Gilbert 
> tends to be the best maintainer of our docs, although he's the sg
> author, which is why the docs tend to emphasize it as the preferred
> option ...

I'm still trying to figure what the other options _are_.

> James

Rob
-- 
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.
-
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Some quick scsi documentation questions:

[Douglas Gilbert]

Rob Landley wrote:
> On Thursday 26 July 2007 9:15:11 am James Bottomley wrote:
>> On Wed, 2007-07-25 at 23:28 -0400, Rob Landley wrote:
>>> 1) Is the summary of the scsi subsystem in appendix A of your 2002 OLS
>>> paper still more or less current?  (The scsi mid-layer document never
>>> seems to talk about the _upper_ layer it interfaces with.  Which is
>>> presumably the block layer...)
>> No, the block layer is the service layer.  Upper layer is the SCSI
>> driver layer (top level drivers like sd, sr and st).
> 
> Now that I've noticed that http://sg.torque.net/scsi/SCSI-2.4-HOWTO.html and 
> http://sg.torque.net/scsi/SCSI-Generic-HOWTO.html are separate documents, I 
> see where that's documented. :)
> 
> So upper layer drivers provide /dev nodes, lower level talks to SCSI hardware, 
> and the mid-layer understands scsi commands ala http://www.t10.org/scsi-3.htm  
> 
> Where _does_ the block layer work into this?
> 
>>> 2) Documentation/scsi/scsi_fc_transport.txt is talking about Fiber
>>> Channel, right?  (It never expands the fc, not even in the title...)
>> Yes.
> 
> Already sent a separate email about that one.  (Although I now see that 
> t10.org inclues some fiber channel standards, and what the relationship 
> between it and t11 is I have no idea.)
> 
>>> 3) After reading Documentation/scsi/scsi-generic.txt I'm curious what the
>>> relationships between a given sg device and sd device.  Each sd has a
>>> corresponding sg?  (I can't find where it explicitly says this, but it's
>>> a char device, not a block device, so...)  They're always in the same
>>> order? Does _each_ sd have an sg?  Can I call the SG_IO ioctl on an sd? 
>>> Is there an obvious document on this I've missed?
>> This is one of those evolutionary problems in Linux.  Originally sg
>> existed simply to provide the SG_IO ioctl to SCSI devices.  Then,
>> because it was extremely useful, we made it a core feature of the block
>> layer (in block/scsi_ioctl.c) which extended it as a general packet
>> command feature to devices beyond SCSI.
> 
> But it's still a char device, so you presumably can't mount /dev/sg1 to read 
> the CD in your burner.  You need to burn the CD and mount the CD through 
> separate devices.
> 
> To someone like me from the IDE world, the numbering on scsi is weird.  It 
> won't leave holes like the ATA driver did to avoid renumbering categories of 
> devices that can't be moved without a screwdriver.  (I realize there _are_ 
> truly dynamic devices, but the hard drive in my laptop is not one of them and 
> it's an artifact of the scsi layer that it's considered so.)
> 
> So if you don't attach an sg to every sd, then either there are holes, or they 
> don't line up, in which case how do you tell which sg goes with a given sd?
> 
>> The logical thing then became to 
>> attach sg as an upper driver *only* to devices which otherwise wouldn't
>> have one ... however, the sg maintainer has never accepted this.
> 
> Do you mean only to devices that otherwise wouldn't have an upper driver?  (So 
> there would be an sg for devices that have _no_ sd?  This implies that the 
> answer to my earlier question about whether the SG_IO ioctl can be used on sd 
> block devices is "yes"?  (In which case the bit where the sg document talked 
> about the nonblocking write to sg thing being useful for queueing commands 
> comes back up, is there currently a way to do that through the sd ioctl?)
> 
> It sounds like your preferred solution would be to have either an sg _or_ an 
> sd for each scsi device, so there would be no duplicates but only "no holes" 
> if you look at _both_ groups and interlace them.  And the sg maintainer went 
> with "have both for every device"?
> 
>> Various distros have tried it at various times, though, so currently
>> it's safe to say that if sg is loaded, every SCSI device has an sg node,
>> but it's not safe to assume that universally.
> 
> Ok, so currently, on distros using recent 2.6 kernels, there's an sg 
> corresponding to every sd, using the same number.  Right.
> 
> Did the various distros patch the kernel, or did they do udev rules to filter 
> out the sg they didn't want?  Any idea which distros those were, and whether 
> or not they're still relevant?
> 
>> We can't even agree on 
>> unifying the SG_IO paths, so we have two separate codings, one in sg.c
>> and one in block/scsi_ioctl.c.
> 
> What would unifying the SG_IO paths mean?  Would it have any impact on 
> userspace, or is it just an implementation detail inside the kernel?
> 
>>>  seems that a read-only cd will be an sd, but you talk to a burner
>>> through an sg...)
>> No, they're both sr.
> 
> What was that again?  (Googles...)  Oh yeah, the "upper layer" drivers from 
> the scsi-2.4-howto.  I thought scsi-tape and scsi-cdrom were obsolete, but I 
> guess not. 
> 
> Back in the IDE world, my hard drive was /dev/hda and my CD burner 
> was /dev/hdc.  When /dev/hdc gained the ability to be used directly as a cd 
> burner, I just assumed you could do it to sd as well.  My initial mental 
> model was "sg gives you a char device to pass through scsi commands for a 
> random scsi device, and sd gives you a way to look at it as a block device", 
> which doesn't explain sr and st but as I said I thought they were obsolete...
> 
>> Users have great difficulty understanding how to 
>> map sg to sr (or sd), so we're trying to encourage them only to use the
>> actual device node, so /dev/sdx or /dev/srn.
> 
> Actually, these days everybody I know uses /dev/cdrom, expects it to be a 
> symlink, and doesn't care where it points.  Last time I used a system that 
> had more than one of the suckers I believe it had /dev/cdrom0, /dev/cdrom1, 
> etc.
> 
> I note that in the ubuntu system on my laptop, /dev/sr0 is a symlink 
> to /dev/scd0, which is an actual device node with a major and minor that 
> presumably means something to the system, but not to me personally.  (Not 
> that I consider Ubuntu 7.04 much of a model on how to arrange devices 
> considering it put a UUID label on every _partition_ on my hard drive and 
> mounts them by label in /etc/fstab rather than that having udev do its job 
> and make stable symlinks.)
> 
> But it seems that users aren't the only ones who have trouble with this, 
> distro maintainers do too.
> 
> Question: how much of the difference between /dev/sd* and /dev/sr* do users 
> actually have to care about these days?  (And is st still relevant, or does 
> sg cover that?)
> 
>>> 4) Is http://www.tldp.org/HOWTO/SCSI-Generic-HOWTO/ still current, given
>>> it was last updated in 2002?
>> Probably not ... the transport classes all come after that.
> 
> Unfortunately, Documentation/scsi/scsi-generic.txt (which seems a logical 
> starting point) is essentially nothing more than a link to that document and 
> a note about using scsi drivers as modules.
> 
> It also still seems to be the best introduction to sd vs sr vs sg.  It would 
> be nice if there was some kind of quick reference card listing the 
> concepts...
> 
> Similarly, Documentation/scsi/scsi.txt is a link to the SCSI-2.4-HOWTO and a 
> note about using scsi drivers as modules.  (And the first time I read those 
> two, I didn't notice that the SCSI-2.4-HOWTO and the SCSI-Generic-HOWTO were 
> two separate documents...)
> 
>>> 5) I notice there's a "man 4 sd", from 1992.  Still relevant?
>> Yes .. but only as far as it goes, which isn't very far.
> 
> I noticed.  It'd be nice if it at least explained the differences between sd, 
> sr, st, and sg.  I'll ping the man-pages maintainer about it...
> 
>> Taking this to linux-scsi might be the best course ...
> 
> And now that I've confirmed it's ok with you, I've cc'd this thread there.
> 
>> Doug Gilbert 
>> tends to be the best maintainer of our docs, although he's the sg
>> author, which is why the docs tend to emphasize it as the preferred
>> option ...
> 
> I'm still trying to figure what the other options _are_.
> 
>> James
> 
> Rob

Interesting discussion.

However I think that I would object to this statement by James:
" The logical thing then became to attach sg as an upper driver
  *only* to devices which otherwise wouldn't have one ... however,
  the sg maintainer has never accepted this."

It is interesting to note that the new bsg driver follows the
approach by the sg driver: mapping all ** devices irrespective
of whether they "belong" to other drivers or not.

In the case of the sg driver, there was backward compatibility
to worry about. In the case of bsg it is just a mistake, IMO.
The bsg driver should be doing what it did in its original
incarnation: have one char device called /dev/bsg (why not
/dev/csg ??) and let apps open that device to get a file
descriptor context from which they could bind to any entity
that would permit it. That entity might be:
   - the SCSI mid layer
   - a (SCSI) transport
   - a host driver
   - a host device (SCSI initiator device or port)
   - a SCSI target
   - a SCSI logical unit
   - an active element in a SCSI delivery subsystem
     (e.g. a switch in FC or and expander in SAS)


We continue to have a model where the mid-level's scan
algorithm (and perhaps the transport) decide what "devices"
will be shown. As an example will they show a SCSI target
(perhaps a bridge) with no logical units visible, even
though lun 0 replies [peri_qual=3 and pdt=0x1f]? I
suspect not. Such a target might support an access control
well known logical unit which, given the correct interaction,
may expose otherwise hidden storage logical units.

In the same way a pass-through is a nuisance to many
overarching OS designs, the reverse is true for some
pass-through users.


** As noted above "all" is an overestimate. The pass-through
only gets to map to devices that the sysfs/udev mess decides
are relevant.


Doug Gilbert

Re: Some quick scsi documentation questions:

[Stefan Richter]

> Rob Landley wrote:
>> On Thursday 26 July 2007 9:15:11 am James Bottomley wrote:
>>> Users have great difficulty understanding how to 
>>> map sg to sr (or sd), so we're trying to encourage them only to use the
>>> actual device node, so /dev/sdx or /dev/srn.
>> Actually, these days everybody I know uses /dev/cdrom, expects it to be a 
>> symlink, and doesn't care where it points.  Last time I used a system that 
>> had more than one of the suckers I believe it had /dev/cdrom0, /dev/cdrom1, 
>> etc.
>>
>> I note that in the ubuntu system on my laptop, /dev/sr0 is a symlink 
>> to /dev/scd0, which is an actual device node with a major and minor that 
>> presumably means something to the system, but not to me personally.  (Not 
>> that I consider Ubuntu 7.04 much of a model on how to arrange devices 
>> considering it put a UUID label on every _partition_ on my hard drive and 
>> mounts them by label in /etc/fstab rather than that having udev do its job 
>> and make stable symlinks.)
>>
>> But it seems that users aren't the only ones who have trouble with this, 
>> distro maintainers do too.

Alternatively to partition labels, you can also have udev create
persistently named symlinks for you.

$ ls /dev/disk/by-id/
ata-HTE721010G9AT00_MPC0J1Y0GRKYGD
ata-HTE721010G9AT00_MPC0J1Y0GRKYGD-part1
ata-HTE721010G9AT00_MPC0J1Y0GRKYGD-part2
ieee1394-00101003ae0001e0:00042c:0000
ieee1394-00101003ae0001e0:00042c:0000-part1
scsi-SATA_ST3750640AS_3QD067RX
scsi-SATA_ST3750640AS_3QD067RX-part1
usb-JetFlash_128MB_39680ADE3E2627FB
usb-JetFlash_128MB_39680ADE3E2627FB-part1

The udev rules I have here (Gentoo's) apparently don't do this for
CD/DVD-ROM/R/Ws although that should be possible as well.  On the other
hand, application programs like K3B already nicely show devices with
vendor and model strings.  I don't know how much effort that imposes on
the application programs.
-- 
Stefan Richter
-=====-=-=== =--- ---=-
http://arcgraph.de/sr/

Re: Some quick scsi documentation questions:

[Stefan Richter]

> Rob Landley wrote:
>> So if you don't attach an sg to every sd, then either there are holes, or they 
>> don't line up, in which case how do you tell which sg goes with a given sd?

Sysfs has pointers from a SCSI logical unit device to the corresponding
sd device and sg device...

$ ll /sys/bus/scsi/devices/1\:0\:0\:0/[bg]*
lrwxrwxrwx 1 root root 0 Aug  2 23:23
/sys/bus/scsi/devices/1:0:0:0/block:sdb -> ../../../../../../block/sdb
lrwxrwxrwx 1 root root 0 Aug  2 23:23
/sys/bus/scsi/devices/1:0:0:0/generic ->
../../../../../../class/scsi_generic/sg1


...as well as from the sd and sg devices to the corresponding SCSI
logical unit device:

$ ll /sys/block/sdb/device
lrwxrwxrwx 1 root root 0 Aug  2 23:27 /sys/block/sdb/device ->
../../devices/pci0000:00/0000:00:1f.2/host1/target1:0:0/1:0:0:0

$ ll /sys/class/scsi_generic/sg1/device
lrwxrwxrwx 1 root root 0 Aug  2 23:24 /sys/class/scsi_generic/sg1/device
-> ../../../devices/pci0000:00/0000:00:1f.2/host1/target1:0:0/1:0:0:0


The place where these symlinks live might change in the future.  See
linux-2.6.23*/Documentation/sysfs-rules.txt.
-- 
Stefan Richter
-=====-=-=== =--- ---=-
http://arcgraph.de/sr/

Re: Some quick scsi documentation questions:

[Rob Landley]

On Thursday 02 August 2007 2:41:46 pm Douglas Gilbert wrote:
> > I'm still trying to figure what the other options _are_.
> >
> >> James
> >
> > Rob
>
> Interesting discussion.

I'm trying to pull together documentation of the scsi layer and link it into 
the index I'm trying to put up at http://kernel.org/doc, but I'm working 
under the handicap of never having used scsi hardware.

Technically I still don't own any scsi hardware except a scanner I've never 
gotten to work, but the scsi layer has now eaten my ide devices, it's 
handling my new sata hard drive which makes sas look even more hilarious than 
it did in 2005, and my vague impression is that the bsg patch turns my 
intended-as-joke question to James at OLS (so when do ramdisks start going 
through the scsi layer?) into a potential reality.  (Not that I pretend to 
understand the bsg code.)

I also didn't understand the rest of your message well enough to comment on 
any of it (does my USB key have a LUN?  The scsi layer grabbed it...)

> ** As noted above "all" is an overestimate. The pass-through
> only gets to map to devices that the sysfs/udev mess decides
> are relevant.

Don't get me started on udev.  As far as I can tell, Greg KH disagrees with me 
on every topic I've ever raised with him...

> Doug Gilbert

Rob
-- 
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.

Re: Some quick scsi documentation questions:

[Rob Landley]

On Thursday 02 August 2007 5:19:51 pm Stefan Richter wrote:
> > Rob Landley wrote:
> >> On Thursday 26 July 2007 9:15:11 am James Bottomley wrote:
> >>> Users have great difficulty understanding how to
> >>> map sg to sr (or sd), so we're trying to encourage them only to use the
> >>> actual device node, so /dev/sdx or /dev/srn.
> >>
> >> Actually, these days everybody I know uses /dev/cdrom, expects it to be
> >> a symlink, and doesn't care where it points.  Last time I used a system
> >> that had more than one of the suckers I believe it had /dev/cdrom0,
> >> /dev/cdrom1, etc.
> >>
> >> I note that in the ubuntu system on my laptop, /dev/sr0 is a symlink
> >> to /dev/scd0, which is an actual device node with a major and minor that
> >> presumably means something to the system, but not to me personally. 
> >> (Not that I consider Ubuntu 7.04 much of a model on how to arrange
> >> devices considering it put a UUID label on every _partition_ on my hard
> >> drive and mounts them by label in /etc/fstab rather than that having
> >> udev do its job and make stable symlinks.)
> >>
> >> But it seems that users aren't the only ones who have trouble with this,
> >> distro maintainers do too.
>
> Alternatively to partition labels, you can also have udev create
> persistently named symlinks for you.

A) You never need to label the _partition_, you need to consistently identify 
the _device_ and the partitions fall out from that.

landley@dell:/$ cat /sys/block/sr0/device/model
CDRWDVD DW224EV
landley@dell:/$ cat /sys/block/sda/device/model
Hitachi HTS54161

Why exactly do I need a uuid to identify either of these when I only have one 
of each in the system and _know_ I'll never have more than that?  (Sure go 
for a uuid when there's more than one hard drive.  But please explain to me 
how a uuid would help if I had two identical DVD burners with no media in 
them on bootup...)

B) This particular device can't move without a screwdriver, and you can't add 
another sata drive to this laptop without a soldering iron.

The fact that modern Linux kernels can't consistently identify it during 
bootup is a design flaw.  (Yes, I followed the discussion of all this years 
ago, but the fact remains that there are certain very common types of 
hardware that only get confused by things like USB keys because both the USB 
keys, which may speak a scsi-like protocol but aren't actually scsi hardware, 
and SATA drives, ditto, get routed into the same scsi layer heap and 
conflated.  And that's sad.

> The udev rules I have here (Gentoo's) apparently don't do this for
> CD/DVD-ROM/R/Ws although that should be possible as well.

Many things are possible.  It doesn't make them a good idea.  Telling udev to 
do something complicated to keep track of a device that I know, at OS install 
time, _can't_ever_physically_move_, is one of them.

> On the other 
> hand, application programs like K3B already nicely show devices with
> vendor and model strings.  I don't know how much effort that imposes on
> the application programs.

See above, but notice that Greg KH says that any program that follows 
the "device" symlink is buggy, because he doesn't want to be tied down to 
anything as mundane as a stable userspace API.

Rob
-- 
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.

Re: Some quick scsi documentation questions:

[Rob Landley]

On Thursday 02 August 2007 5:39:55 pm Stefan Richter wrote:
> The place where these symlinks live might change in the future.  See
> linux-2.6.23*/Documentation/sysfs-rules.txt.

I noticed this.  I had a longish flamewar with Greg KH about this, which would 
be ongoing if he hadn't stopped replying.

It's easy to find out information like this by experimenting with sysfs.  And 
then they change it in future versions, and blame the user for using what 
sysfs exports.

Rob
-- 
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.

Re: Some quick scsi documentation questions:

[Stefan Richter]

Rob Landley wrote:
> On Thursday 02 August 2007 5:39:55 pm Stefan Richter wrote:
>> The place where these symlinks live might change in the future.  See
>> linux-2.6.23*/Documentation/sysfs-rules.txt.
> 
> I noticed this.  I had a longish flamewar with Greg KH about this, which would 
> be ongoing if he hadn't stopped replying.
> 
> It's easy to find out information like this by experimenting with sysfs.  And 
> then they change it in future versions, and blame the user for using what 
> sysfs exports.

Sysfs lets you peek into kernel implementation details.  Hence what you
see in sysfs will change all the time when the implementation is
changed.  It's important to realize this before making use of sysfs.

Sysfs (most of it) is not a stable ABI.  Never has been, never will be,
never was intended to be.  That's overlooked again and again.

If you want to do something that requires a stable ABI between kernel
and userspace, then use such a stable ABI (including the small parts of
sysfs that have been declared stable and hence will remain stable).
-- 
Stefan Richter
-=====-=-=== =--- ---==
http://arcgraph.de/sr/

Re: Some quick scsi documentation questions:

[Stefan Richter]

Rob Landley wrote:
> On Thursday 02 August 2007 5:19:51 pm Stefan Richter wrote:
>> Alternatively to partition labels, you can also have udev create
>> persistently named symlinks for you.
> 
> A) You never need to label the _partition_, you need to consistently identify 
> the _device_ and the partitions fall out from that.
> 
> landley@dell:/$ cat /sys/block/sr0/device/model
> CDRWDVD DW224EV
> landley@dell:/$ cat /sys/block/sda/device/model
> Hitachi HTS54161
> 
> Why exactly do I need a uuid to identify either of these when I only have one 
> of each in the system and _know_ I'll never have more than that?

If there is only have one device of a kind, then there is no UUID needed
for anything in the first place.

Besides, then you also don't need to map between the sd or sr device and
the sg device.  You can and should do the generic IO through the sd and
sr devices nowadays.

> (Sure go 
> for a uuid when there's more than one hard drive.  But please explain to me 
> how a uuid would help if I had two identical DVD burners with no media in 
> them on bootup...)

In case of FireWire drives and several other types of drives (I believe
also in case of SATA devices) there is a UUID tied to the *drive* too,
alongside UUIDs of the media.

> B) This particular device can't move without a screwdriver, and you can't add 
> another sata drive to this laptop without a soldering iron.

And how is software supposed to know that?

> The fact that modern Linux kernels can't consistently identify it during 
> bootup is a design flaw.

Do you mean the Linux OS or the Linux kernel?

> (Yes, I followed the discussion of all this years 
> ago, but the fact remains that there are certain very common types of 
> hardware that only get confused by things like USB keys because both the USB 
> keys, which may speak a scsi-like protocol but aren't actually scsi hardware, 
> and SATA drives, ditto, get routed into the same scsi layer heap and 
> conflated.  And that's sad.

Harddisks are harddisks, CD-ROMs are CD-ROMs, scanners are scanners, no
matter how they are attached.  At least as far as the commands are
concerned that software sends to them.  I say that's a good thing.

Sure, the type of attachment may be interesting to know for device file
naming, see below.

>> The udev rules I have here (Gentoo's) apparently don't do this for
>> CD/DVD-ROM/R/Ws although that should be possible as well.
> 
> Many things are possible.  It doesn't make them a good idea.  Telling udev to 
> do something complicated to keep track of a device that I know, at OS install 
> time, _can't_ever_physically_move_, is one of them.

There is a variety of possible naming schemes:

  - Naming by order of discovery.
  - Naming by vendor/model name strings.
  - Naming by universally unique identifier.
  - Naming by topology.
  - ...

Only the simplest of these schemes (naming by order of discovery) is
hardwired into the kernel portion of the Linux OS.  The other naming
schemes are (or can be) implemented in the userland portion of the Linux OS.

There is only the most primitive naming scheme implemented in the kernel
because naming policy, like most other kinds of policy, is better left
to userland.  The kernel is a too restricted framework to implement such
things.  The kernel lacks runtime-configuration files, scripting
interfaces, et cetera.

>> On the other 
>> hand, application programs like K3B already nicely show devices with
>> vendor and model strings.  I don't know how much effort that imposes on
>> the application programs.
> 
> See above, but notice that Greg KH says that any program that follows 
> the "device" symlink is buggy,

K3B doesn't use sysfs for that.  It most certainly uses SG IO (SCSI
generic IO) to figure out these names.

> because he doesn't want to be tied down to anything as mundane as a
> stable userspace API.

Sysfs is designed to be unstable; see my answer to your other post.
Sysfs (most of it) does not belong to the stable userspace ABIs of the
kernel.
-- 
Stefan Richter
-=====-=-=== =--- ---==
http://arcgraph.de/sr/

Re: Some quick scsi documentation questions:

[Rob Landley]

On Friday 03 August 2007 3:15:27 am Stefan Richter wrote:
> Rob Landley wrote:
> > On Thursday 02 August 2007 5:39:55 pm Stefan Richter wrote:
> >> The place where these symlinks live might change in the future.  See
> >> linux-2.6.23*/Documentation/sysfs-rules.txt.
> >
> > I noticed this.  I had a longish flamewar with Greg KH about this, which
> > would be ongoing if he hadn't stopped replying.
> >
> > It's easy to find out information like this by experimenting with sysfs. 
> > And then they change it in future versions, and blame the user for using
> > what sysfs exports.
>
> Sysfs lets you peek into kernel implementation details.

/sys/class containing all char devices and /sys/block containing all block 
devices was not an implementation detail.  Breaking this assumption by adding 
more char devices under /sys/bus and moving /sys/block moving 
under /sys/class is not an implementation detail.  Deprecating the "device" 
symlink was not an implementation detail.

They keep using the fact that they're exporting random kernel data as an 
excuse for not documenting a stable subset of this as an API userspace can 
rely on.

> Hence what you 
> see in sysfs will change all the time when the implementation is
> changed.  It's important to realize this before making use of sysfs.

It changes dynamically as stuff is hotplugged.  That's no excuse for them not 
making up their minds about what is and isn't a good idea to export, and 
where to put it.

> Sysfs (most of it) is not a stable ABI.

I've noticed this.  This is a bug, not a feature.  Linux has always had a 
stable _USERSPACE_ API.  This is exposed to userspace, and userspace is 
expected to be able to use this.  If they're exporting stuff they shouldn't 
be exporting, then they should stop exporting it.  That's no excuse for 
moving everything around, deprecating paths that used to be the primary way 
to access data that's still exported, just somewhere else...

> Never has been, never will be, 
> never was intended to be.  That's overlooked again and again.

I want to scan sysfs to find:

A) all block devices.
B) all char devices.
C) Whatever attributes of those devices that allow them to be persistently 
identified.

This goal hasn't changed since 2.6.10 or so, but sysfs has changed how we're 
expected to do it, and often they've changed it for no good reason from a UI 
perspective.

> If you want to do something that requires a stable ABI between kernel
> and userspace, then use such a stable ABI (including the small parts of
> sysfs that have been declared stable and hence will remain stable).

You mean like libsysfs? :)

Rob
-- 
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.

Re: Some quick scsi documentation questions:

[Stefan Richter]

Rob Landley wrote:
> They keep using the fact that they're exporting random kernel data as an 
> excuse for not documenting a stable subset of this as an API userspace can 
> rely on.

At least sysfs-rules.txt is a start, although it contains more Don'ts
than Dos.

...
> This is a bug, not a feature.  Linux has always had a 
> stable _USERSPACE_ API.  This is exposed to userspace, and userspace is 
> expected to be able to use this.  If they're exporting stuff they shouldn't 
> be exporting, then they should stop exporting it.

The kernel log is exported to userspace too.

(Ugh, there were actually suggestions to make the log stable too, e.g.
to tack stable unique identifiers to log messages...)
-- 
Stefan Richter
-=====-=-=== =--- ---==
http://arcgraph.de/sr/

Re: Some quick scsi documentation questions:

[Rob Landley]

On Friday 03 August 2007 5:43:25 am Stefan Richter wrote:
> Rob Landley wrote:
> > On Thursday 02 August 2007 5:19:51 pm Stefan Richter wrote:
> >> Alternatively to partition labels, you can also have udev create
> >> persistently named symlinks for you.
> >
> > A) You never need to label the _partition_, you need to consistently
> > identify the _device_ and the partitions fall out from that.
> >
> > landley@dell:/$ cat /sys/block/sr0/device/model
> > CDRWDVD DW224EV
> > landley@dell:/$ cat /sys/block/sda/device/model
> > Hitachi HTS54161
> >
> > Why exactly do I need a uuid to identify either of these when I only have
> > one of each in the system and _know_ I'll never have more than that?
>
> If there is only have one device of a kind, then there is no UUID needed
> for anything in the first place.

Yeah, but optimizing for common cases is still something they're working on, 
it seems.  (The perfect being the enemy of the good and all that: but what if 
someone does whip out a soldering iron and hotplug a second sata drive into 
my laptop!  Gotta be prepared...)

> Besides, then you also don't need to map between the sd or sr device and
> the sg device.  You can and should do the generic IO through the sd and
> sr devices nowadays.

Good to know.  So an sg device is for something like a scanner that doesn't 
present a block device?

> > (Sure go
> > for a uuid when there's more than one hard drive.  But please explain to
> > me how a uuid would help if I had two identical DVD burners with no media
> > in them on bootup...)
>
> In case of FireWire drives and several other types of drives (I believe
> also in case of SATA devices) there is a UUID tied to the *drive* too,
> alongside UUIDs of the media.

I can't find it under /sys/block/sda or device under that.  I can find "model" 
and "vendor", but that's about it.  (I can't even get anything to admit it's 
sata vs firewire.  The vendor for the hard drive is "ATA" but the vendor for 
the dvd is "TEAC", and they're both sata devices on the spec sheet...)

> > B) This particular device can't move without a screwdriver, and you can't
> > add another sata drive to this laptop without a soldering iron.
>
> And how is software supposed to know that?

Well, if all else fails I can tell it at install time.  (Well, re-install time 
in this case.  This is one of those Dell machines that comes pre-imaged with 
Linux, but they put a 32 bit version on a machine with a 64 bit processor.  
Haven't gotten around to reinstalling yet.)

However, for some common types of hardware like ATA and SATA, "not moving" is 
the default case.  There are such things as external SATA enclosures, but 
they're A) few and far between, B) technically different hardware often 
throwing in the towel and slapping USB on the external drive, ala:
http://www.cooldrives.com/sata-enclosure.html

In 99.x% of systems with SATA hard drives, said drives are sealed inside the 
machine, whether it's a laptop or a server.  USB can move around between 
boots (or with the power on) as a matter of course, but SATA really isn't 
designed for that or expected to do that.

> > The fact that modern Linux kernels can't consistently identify it during
> > bootup is a design flaw.
>
> Do you mean the Linux OS or the Linux kernel?

As someone who used ATA drives for 10 years before this became a problem, and 
is now using a SATA drive inside a laptop in a configuration Dell (ok, 
Quanta) churns out by the millions, I consider it a design flaw in the scsi 
layer that anything that goes through the scsi layer loses its original 
identity.

There have been numerous proposals to compensate for the scsi layer's 
renumbering with udev rules, but at least two of the most widely used 
distributions are still doing them wrong, so it's apparently not a trivial 
problem to recover from.

> > (Yes, I followed the discussion of all this years
> > ago, but the fact remains that there are certain very common types of
> > hardware that only get confused by things like USB keys because both the
> > USB keys, which may speak a scsi-like protocol but aren't actually scsi
> > hardware, and SATA drives, ditto, get routed into the same scsi layer
> > heap and conflated.  And that's sad.
>
> Harddisks are harddisks, CD-ROMs are CD-ROMs, scanners are scanners, no
> matter how they are attached.  At least as far as the commands are
> concerned that software sends to them.  I say that's a good thing.

You can reliably enumerate ATA and SATA.  You cannot reliably enumerate USB.  
Lumping SATA and USB together means you can't reliably enumerate either of 
them, which is an artificial problem invented by lumping them together.

> Sure, the type of attachment may be interesting to know for device file
> naming, see below.

It is, yes.

> >> The udev rules I have here (Gentoo's) apparently don't do this for
> >> CD/DVD-ROM/R/Ws although that should be possible as well.
> >
> > Many things are possible.  It doesn't make them a good idea.  Telling
> > udev to do something complicated to keep track of a device that I know,
> > at OS install time, _can't_ever_physically_move_, is one of them.
>
> There is a variety of possible naming schemes:
>
>   - Naming by order of discovery.
>   - Naming by vendor/model name strings.
>   - Naming by universally unique identifier.
>   - Naming by topology.
>   - ...

See "I followed the discussion of all this years ago"...

> Only the simplest of these schemes (naming by order of discovery) is
> hardwired into the kernel portion of the Linux OS.

The issue you're ignoring when you talk about "order of discovery" is that the 
first Sata drive and the first USB drive are lumped into the same sequence.  
The sata drives can't move, but I have an external USB drive that may or may 
not be plugged in at boot time.  The fact that the presence or absence of a 
USB drive can change the names assigned to the SATA drives is a loss of 
orthogonality caused by the SCSI layer.  I consider this a design flaw, and 
migrating the ATA drives to do this was a regression.

> The other naming  
> schemes are (or can be) implemented in the userland portion of the Linux
> OS.

Userland can work around the scsi layer, yes.

> There is only the most primitive naming scheme implemented in the kernel
> because naming policy, like most other kinds of policy, is better left
> to userland.  The kernel is a too restricted framework to implement such
> things.  The kernel lacks runtime-configuration files, scripting
> interfaces, et cetera.

/dev/hdc staying put when you removed or inserted a /dev/hdb was nice.  It 
worked well for 15 years.

Not _all_ hardware is hotpluggable, and ignoring this knowledge during device 
enumeration is silly.

> >> On the other
> >> hand, application programs like K3B already nicely show devices with
> >> vendor and model strings.  I don't know how much effort that imposes on
> >> the application programs.
> >
> > See above, but notice that Greg KH says that any program that follows
> > the "device" symlink is buggy,
>
> K3B doesn't use sysfs for that.  It most certainly uses SG IO (SCSI
> generic IO) to figure out these names.

Good to know.  I haven't read far enough through the scsi docs to see how to 
do that yet.  (I just moved, still catching up.)  Can I get back to you when 
I have?

> > because he doesn't want to be tied down to anything as mundane as a
> > stable userspace API.
>
> Sysfs is designed to be unstable;

Unstable, yes.  "Designed" may be too strong a word here...

> see my answer to your other post. 
> Sysfs (most of it) does not belong to the stable userspace ABIs of the
> kernel.

I currently view the scsi layer as a weird sort of networking stack.  Although 
actual SCSI hardware seems essentially extinct, lots of devices speak 
dialects of the SCSI protocol, and they still send data packets in that 
format back and forth through the hardware du jour (sometimes even tunneled 
over TCP/IP).  The SCSI layer lets you talk to these devices using that 
protocol.

SCSI doesn't handle device enumeration any more than TCP/IP does, and 
sometimes to do device enumeration you have to look at or talk to the 
underlying hardware, just like you have to dig down to ethernet broadcast 
packets in order to do DHCP.  Unfortunately, the only way to talk to the 
underlying hardware seems to be through the SCSI layer.  I can find "eth0" 
without having a TCP/IP address assigned to it, but I can only list the sata 
devices in the system by going through the list of devices the SCSI layer has 
detected and working backwards...
 
It's entirely possible my metaphor is wrong here, but I'm trying to 
understand...

Rob
-- 
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.

Re: Some quick scsi documentation questions:

[Rob Landley]

On Friday 03 August 2007 2:37:02 pm Stefan Richter wrote:
> Rob Landley wrote:
> > They keep using the fact that they're exporting random kernel data as an
> > excuse for not documenting a stable subset of this as an API userspace
> > can rely on.
>
> At least sysfs-rules.txt is a start, although it contains more Don'ts
> than Dos.

I've noticed this. :)

> > This is a bug, not a feature.  Linux has always had a
> > stable _USERSPACE_ API.  This is exposed to userspace, and userspace is
> > expected to be able to use this.  If they're exporting stuff they
> > shouldn't be exporting, then they should stop exporting it.
>
> The kernel log is exported to userspace too.

The kernel log is debug info, and nothing but debug info.  You should only 
ever need to look at if something bad has happened.  Attempts to extract 
information from it in an automated fashion during the normal operation of 
the system are a bad idea, and always have been.

Sysfs _claims_ to provide information for use by userspace programs.  It 
claims to do so without providing a stable API.  This is a contradiction in 
terms.

I realize that the easy way to see what name the scsi layer assigned to the 
USB key you just plugged in is dmesg | tail and then look at it a bit, but 
this is a horrible workaround for the scsi layer's tendency to conflate 
different classes of devices into a single arbitrarily ordered sequence, and 
there are less horrible workarounds.

Rob
-- 
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.

Re: Some quick scsi documentation questions:

[Stefan Richter]

Rob Landley wrote:
> So an sg device is for something like a scanner that doesn't 
> present a block device?

That's what I heard.

>> In case of FireWire drives and several other types of drives (I believe
>> also in case of SATA devices) there is a UUID tied to the *drive* too,
>> alongside UUIDs of the media.
> 
> I can't find it under /sys/block/sda or device under that.  I can find "model" 
> and "vendor", but that's about it.  (I can't even get anything to admit it's 
> sata vs firewire.  The vendor for the hard drive is "ATA" but the vendor for 
> the dvd is "TEAC", and they're both sata devices on the spec sheet...)

# /lib/udev/ata_id /dev/hda
HTE721010G9AT00_MPC0J1Y0GRKYGD
# /lib/udev/ata_id /dev/sda
ST3750640AS_3QD07L23
# /lib/udev/ata_id /dev/hdb
PLEXTOR_DVD-ROM_PX-130A

Not very unique in case of the IDE DVD-ROM.  I don't know if SATA
DVD_ROMs have an actual UUID.

> There are such things as external SATA enclosures, but 
> they're A) few and far between,

What?  They are all the rage now.  :-)

> In 99.x% of systems with SATA hard drives, said drives are sealed inside the 
> machine, whether it's a laptop or a server.  USB can move around between 
> boots (or with the power on) as a matter of course, but SATA really isn't 
> designed for that or expected to do that.

I suppose you could still have a not too complicated topology-based
naming scheme even with eSATA.

...
> As someone who used ATA drives for 10 years before this became a problem, and 
> is now using a SATA drive inside a laptop in a configuration Dell (ok, 
> Quanta) churns out by the millions, I consider it a design flaw in the scsi 
> layer that anything that goes through the scsi layer loses its original 
> identity.
> 
> There have been numerous proposals to compensate for the scsi layer's 
> renumbering with udev rules, but at least two of the most widely used 
> distributions are still doing them wrong, so it's apparently not a trivial 
> problem to recover from.

At least some PPC Linux distributions have had boot-from-FireWire for
some time, even though there are neither deterministic discovery times
nor a fixed bus topology with FireWire.  SATA should be simpler.

...
> You can reliably enumerate ATA and SATA.

You mean, you can reliably enumerate ATA and SATA based on bus topology.
Do SATA port multipliers still fit into the picture?

...
>>> Telling
>>> udev to do something complicated to keep track of a device that I know,
>>> at OS install time, _can't_ever_physically_move_, is one of them.
>> There is a variety of possible naming schemes:
>>
>>   - Naming by order of discovery.
>>   - Naming by vendor/model name strings.
>>   - Naming by universally unique identifier.
>>   - Naming by topology.
>>   - ...
> 
> See "I followed the discussion of all this years ago"...
> 
>> Only the simplest of these schemes (naming by order of discovery) is
>> hardwired into the kernel portion of the Linux OS.
> 
> The issue you're ignoring when you talk about "order of discovery" is that the 
> first Sata drive and the first USB drive are lumped into the same sequence.

No, I'm not ignoring that.  I never stated that naming by order of
discovery would be a particularly useful naming scheme.  For
administration and applications it is of course the least useful naming
scheme.

> The sata drives can't move, but I have an external USB drive that may or may 
> not be plugged in at boot time.  The fact that the presence or absence of a 
> USB drive can change the names assigned to the SATA drives is a loss of 
> orthogonality caused by the SCSI layer.

Only names based on order of discovery change when SATA and USB are
intermingled.  The other names don't change.  (But these other names
have to be put together by userspace.)

A simple workaround for simple configurations is to only load the
low-level drivers (transport and interconnect drivers) in the early boot
phase which are known to be required for the disk with root filesystem.

> I consider this a design flaw, and 
> migrating the ATA drives to do this was a regression.
> 
>> The other naming  
>> schemes are (or can be) implemented in the userland portion of the Linux
>> OS.
> 
> Userland can work around the scsi layer, yes.

All newer SCSI transports are far too complicated for a kernelspace-only
implementation.  So in case of such transports, it's not a workaround.

It may be a workaround in case of ATA and PATA.

>> There is only the most primitive naming scheme implemented in the kernel
>> because naming policy, like most other kinds of policy, is better left
>> to userland.  The kernel is a too restricted framework to implement such
>> things.  The kernel lacks runtime-configuration files, scripting
>> interfaces, et cetera.
> 
> /dev/hdc staying put when you removed or inserted a /dev/hdb was nice.  It 
> worked well for 15 years.
> 
> Not _all_ hardware is hotpluggable, and ignoring this knowledge during device 
> enumeration is silly.

If I understand you correctly, what you desire is to have at least two
naming schemes implemented in the kernel:  Topology-based for those
transports (and interconnects) which have simple topologies (notably ATA
and SATA), and the primitive order-of-discovery-based scheme as fallback
for the rest.  Right?

...
>> K3B doesn't use sysfs for that.  It most certainly uses SG IO (SCSI
>> generic IO) to figure out these names.
> 
> Good to know.  I haven't read far enough through the scsi docs to see how to 
> do that yet.

A pro pos SCSI docs.  I haven't read the full thread.  Did people point
you already to http://www.t10.org/scsi-3.htm?

...
> I currently view the scsi layer as a weird sort of networking stack.  Although 
> actual SCSI hardware seems essentially extinct, lots of devices speak 
> dialects of the SCSI protocol, and they still send data packets in that 
> format back and forth through the hardware du jour (sometimes even tunneled 
> over TCP/IP).  The SCSI layer lets you talk to these devices using that 
> protocol.

See above page.  Also have a look at the SAM-4 spec, figure 2.  The
Linux SCSI stack is not a 1:1 reflection of that architecture, but it
gives an approximate idea of the roles of high-level drivers (sd, sr,
st, sg), the mid-layer (scsi-mod), and low-level drivers (transport
layers and transport libraries, and the numerous interconnect drivers).

(Also scroll a bit further through SAM-4 --- you will see mostly quite
abstract stuff which is this way in order to cover all the various
flavors of SCSI that came into existence.  It's really quite far from
the SCSI of the 1980s.  SCSI actually started to become more diverse
already during the 1990s.)

> SCSI doesn't handle device enumeration any more than TCP/IP does, and 
> sometimes to do device enumeration you have to look at or talk to the 
> underlying hardware, just like you have to dig down to ethernet broadcast 
> packets in order to do DHCP.

These matters are defined in the transport layer of the SCSI
Architecture Model, and implemented in Linux' SCSI low-level drivers.
There are considerable differences in discovery and addressing between
the various transport protocols.  The SCSI core doesn't really play a
role in these matters.

(There are remnants of SPI-specific addressing sprinkled throughout the
SCSI mid-layer's APIs though; something that is seemingly very hard to
get rid of.  SPI = SCSI Parallel Interconnect.)

> Unfortunately, the only way to talk to the 
> underlying hardware seems to be through the SCSI layer.  I can find "eth0" 
> without having a TCP/IP address assigned to it,

But what is "eth0"?  Is it one of the onboard ethernet ports?  Is it an
add-on PCI or CardBus card?  Or is it even an IP-over-FireWire
interface?  (Those too have been named eth* for historical reasons but
now haven't anything to do with Ethernet anymore.)

It's almost the same story with networking interfaces:  The kernel
implements only one of several possible naming policies --- a very
primitive and not very useful one:  There is a prefix like "eth" (there
are a few other prefixes for other types of interfaces but I don't know
which in particular), and then there is a number which is handed out by
the networking core in order of discovery.

> but I can only list the sata 
> devices in the system by going through the list of devices the SCSI layer has 
> detected and working backwards...
>  
> It's entirely possible my metaphor is wrong here, but I'm trying to 
> understand...
> 
> Rob

-- 
Stefan Richter
-=====-=-=== =--- --=--
http://arcgraph.de/sr/

Re: Some quick scsi documentation questions:

[Stefan Richter]

> Did people point you already to http://www.t10.org/scsi-3.htm?
...
> Also have a look at the SAM-4 spec, figure 2.

PS:  See also SAM-4 annex A tables A.2 and A.3:  The concatenation of
target port identifier and logical unit identifier is one important way
to uniquely and persistently identify SCSI devices.  Linux' SCSI
transport layer implementations currently have no common way to expose
these identifiers to userspace though; the parts that make up these
identifiers are located in diverse sysfs attributes.
-- 
Stefan Richter
-=====-=-=== =--- --=--
http://arcgraph.de/sr/

Re: Some quick scsi documentation questions:

[Douglas Gilbert]

Stefan Richter wrote:
> Rob Landley wrote:
>> So an sg device is for something like a scanner that doesn't 
>> present a block device?
> 
> That's what I heard.

The sg driver is a SCSI pass-through while the bsg driver is
a pass-through the specializes in SCSI and other (usually)
storage related protocols. Both work at the level of industry
recognized standard (and draft) protocols.

That approach makes the Linux block layer either a nuisance,
irrelevant or a complete anachronism (in the case of OSD).
IMO the linux block layer should be morphed into a library
of internal queue handling routines. Storage upper level
drivers such as sd can continue to present the "block"
view ** of storage devices such as disks.

** In OO terminology a storage device may have a "block"
   interface. A storage device is not derived from "block".


In practical terms the block layer SG_IO ioctl is still a
scaled down version of what the corresponding ioctl in
the sg driver can do. The block layer SG_IO ioctl is saddled
with whatever strange policy the upper level driver might
have for that device class (e.g. wait forever if removal media
is not present). Also the block layer SG_IO ioctl cannot do
asynchronous IO (either can the bsg driver at the moment).


There was a version of Fedora that came out with sg devices
only available for SCSI devices not already claimed by
other upper level drivers. It was a surprise to me (Arjan
may have mentioned it on this list). I got some complaints
(as if I could do anything). Anyway it was amusing
to watch how quickly that misstep was reversed. Obviously
some folks with a lot me influence than me got to the Fedora
designers who did that.

>>> In case of FireWire drives and several other types of drives (I believe
>>> also in case of SATA devices) there is a UUID tied to the *drive* too,
>>> alongside UUIDs of the media.
>> I can't find it under /sys/block/sda or device under that.  I can find "model" 
>> and "vendor", but that's about it.  (I can't even get anything to admit it's 
>> sata vs firewire.  The vendor for the hard drive is "ATA" but the vendor for 
>> the dvd is "TEAC", and they're both sata devices on the spec sheet...)
> 
> # /lib/udev/ata_id /dev/hda
> HTE721010G9AT00_MPC0J1Y0GRKYGD
> # /lib/udev/ata_id /dev/sda
> ST3750640AS_3QD07L23
> # /lib/udev/ata_id /dev/hdb
> PLEXTOR_DVD-ROM_PX-130A
> 
> Not very unique in case of the IDE DVD-ROM.  I don't know if SATA
> DVD_ROMs have an actual UUID.
> 
>> There are such things as external SATA enclosures, but 
>> they're A) few and far between,
> 
> What?  They are all the rage now.  :-)

ATA8-ACS (still draft I think) has provision for a NAA-5
based UUID. Not sure if any SATA disks are complying
yet. [I don't think the Seagate ES series ("enterprise")
did so it will be interesting if their recently announced
ES.2 series does.]

Yes SATA external enclosures are everywhere. They make more
sense than USB 2 and marginalize 1394. The problems start
for SATA when you want to have more than one disk in that
enclosure. SAS is much better as an interconnect.

>> In 99.x% of systems with SATA hard drives, said drives are sealed inside the 
>> machine, whether it's a laptop or a server.  USB can move around between 
>> boots (or with the power on) as a matter of course, but SATA really isn't 
>> designed for that or expected to do that.
> 
> I suppose you could still have a not too complicated topology-based
> naming scheme even with eSATA.

Recent SAS-2 drafts have some heuristic for ATA disks (that
don't have real UUIDs) that combines the manufacturer, model
and serial number into a single number for identification
purposes. With potentially hundreds of SATA disks hanging
off SAS infrastructure, someone who changed the slot a SATA
disk was connected to could cause a lot of fun.

>> As someone who used ATA drives for 10 years before this became a problem, and 
>> is now using a SATA drive inside a laptop in a configuration Dell (ok, 
>> Quanta) churns out by the millions, I consider it a design flaw in the scsi 
>> layer that anything that goes through the scsi layer loses its original 
>> identity.
>>
>> There have been numerous proposals to compensate for the scsi layer's 
>> renumbering with udev rules, but at least two of the most widely used 
>> distributions are still doing them wrong, so it's apparently not a trivial 
>> problem to recover from.
> 
> At least some PPC Linux distributions have had boot-from-FireWire for
> some time, even though there are neither deterministic discovery times
> nor a fixed bus topology with FireWire.  SATA should be simpler.
> 
> ...
>> You can reliably enumerate ATA and SATA.
> 
> You mean, you can reliably enumerate ATA and SATA based on bus topology.
> Do SATA port multipliers still fit into the picture?

Hopefully they will be chased out of the picture by SAS
expanders. It is a bit like comparing a first generation
ethernet 10 Mbps hub with a modern ethernet switch.

When a SATA port multiplier is used
  - the SATA controller and its driver must detect it and
    know what to do
  - the SATA disks are being tricked into thinking they still
    have a point to point connection! [Oh what a tangled web
    we weave ...]

> ...
>>>> Telling
>>>> udev to do something complicated to keep track of a device that I know,
>>>> at OS install time, _can't_ever_physically_move_, is one of them.
>>> There is a variety of possible naming schemes:
>>>
>>>   - Naming by order of discovery.
>>>   - Naming by vendor/model name strings.
>>>   - Naming by universally unique identifier.
>>>   - Naming by topology.
>>>   - ...
>> See "I followed the discussion of all this years ago"...
>>
>>> Only the simplest of these schemes (naming by order of discovery) is
>>> hardwired into the kernel portion of the Linux OS.
>> The issue you're ignoring when you talk about "order of discovery" is that the 
>> first Sata drive and the first USB drive are lumped into the same sequence.
> 
> No, I'm not ignoring that.  I never stated that naming by order of
> discovery would be a particularly useful naming scheme.  For
> administration and applications it is of course the least useful naming
> scheme.
> 
>> The sata drives can't move, but I have an external USB drive that may or may 
>> not be plugged in at boot time.  The fact that the presence or absence of a 
>> USB drive can change the names assigned to the SATA drives is a loss of 
>> orthogonality caused by the SCSI layer.
> 
> Only names based on order of discovery change when SATA and USB are
> intermingled.  The other names don't change.  (But these other names
> have to be put together by userspace.)
> 
> A simple workaround for simple configurations is to only load the
> low-level drivers (transport and interconnect drivers) in the early boot
> phase which are known to be required for the disk with root filesystem.
> 
>> I consider this a design flaw, and 
>> migrating the ATA drives to do this was a regression.
>>
>>> The other naming  
>>> schemes are (or can be) implemented in the userland portion of the Linux
>>> OS.
>> Userland can work around the scsi layer, yes.
> 
> All newer SCSI transports are far too complicated for a kernelspace-only
> implementation.  So in case of such transports, it's not a workaround.

Ah user space device discovery! If only folks wouldn't
design things so that protocols used to device discovery
(yes I am talking about the SAS Management Protocol [SMP])
did _not_ need discovered (or about to be discovered)
devices already in the kernel's sysfs or dev space.

> It may be a workaround in case of ATA and PATA.
> 
>>> There is only the most primitive naming scheme implemented in the kernel
>>> because naming policy, like most other kinds of policy, is better left
>>> to userland.  The kernel is a too restricted framework to implement such
>>> things.  The kernel lacks runtime-configuration files, scripting
>>> interfaces, et cetera.
>> /dev/hdc staying put when you removed or inserted a /dev/hdb was nice.  It 
>> worked well for 15 years.
>>
>> Not _all_ hardware is hotpluggable, and ignoring this knowledge during device 
>> enumeration is silly.
> 
> If I understand you correctly, what you desire is to have at least two
> naming schemes implemented in the kernel:  Topology-based for those
> transports (and interconnects) which have simple topologies (notably ATA
> and SATA), and the primitive order-of-discovery-based scheme as fallback
> for the rest.  Right?
> 
> ...
>>> K3B doesn't use sysfs for that.  It most certainly uses SG IO (SCSI
>>> generic IO) to figure out these names.
>> Good to know.  I haven't read far enough through the scsi docs to see how to 
>> do that yet.
> 
> A pro pos SCSI docs.  I haven't read the full thread.  Did people point
> you already to http://www.t10.org/scsi-3.htm?
> 
> ...
>> I currently view the scsi layer as a weird sort of networking stack.  Although 
>> actual SCSI hardware seems essentially extinct, lots of devices speak 
>> dialects of the SCSI protocol, and they still send data packets in that 
>> format back and forth through the hardware du jour (sometimes even tunneled 
>> over TCP/IP).  The SCSI layer lets you talk to these devices using that 
>> protocol.
> 
> See above page.  Also have a look at the SAM-4 spec, figure 2.  The
> Linux SCSI stack is not a 1:1 reflection of that architecture, but it
> gives an approximate idea of the roles of high-level drivers (sd, sr,
> st, sg), the mid-layer (scsi-mod), and low-level drivers (transport
> layers and transport libraries, and the numerous interconnect drivers).
> 
> (Also scroll a bit further through SAM-4 --- you will see mostly quite
> abstract stuff which is this way in order to cover all the various
> flavors of SCSI that came into existence.  It's really quite far from
> the SCSI of the 1980s.  SCSI actually started to become more diverse
> already during the 1990s.)

I agree with both angles. The linux SCSI/storage subsystem
should look a lot more like the networking layers. No-one
tries to enumerate the whole IP space but it may make sense
to enumerate a small part of it (e.g. MAC addresses in the
directly attached ethernet subnet).

If any enumeration is required then it should be done by
user space tools (e.g. udev) interacting with the transport
concerned. That also implies that SCSI hosts and ATA
controllers need to be promoted to "first class" devices
(cf "interfaces" in networking), not hidden as if they were
some sort of an embarrassment.


Ah, it's time to stop wasting my time (and others who have
read this far) about what might be. The storage paradigm
thinking of the management is stuck somewhere between the
70s and 80s of the last century.

Doug Gilbert

>> SCSI doesn't handle device enumeration any more than TCP/IP does, and 
>> sometimes to do device enumeration you have to look at or talk to the 
>> underlying hardware, just like you have to dig down to ethernet broadcast 
>> packets in order to do DHCP.
> 
> These matters are defined in the transport layer of the SCSI
> Architecture Model, and implemented in Linux' SCSI low-level drivers.
> There are considerable differences in discovery and addressing between
> the various transport protocols.  The SCSI core doesn't really play a
> role in these matters.
> 
> (There are remnants of SPI-specific addressing sprinkled throughout the
> SCSI mid-layer's APIs though; something that is seemingly very hard to
> get rid of.  SPI = SCSI Parallel Interconnect.)
> 
>> Unfortunately, the only way to talk to the 
>> underlying hardware seems to be through the SCSI layer.  I can find "eth0" 
>> without having a TCP/IP address assigned to it,
> 
> But what is "eth0"?  Is it one of the onboard ethernet ports?  Is it an
> add-on PCI or CardBus card?  Or is it even an IP-over-FireWire
> interface?  (Those too have been named eth* for historical reasons but
> now haven't anything to do with Ethernet anymore.)
> 
> It's almost the same story with networking interfaces:  The kernel
> implements only one of several possible naming policies --- a very
> primitive and not very useful one:  There is a prefix like "eth" (there
> are a few other prefixes for other types of interfaces but I don't know
> which in particular), and then there is a number which is handed out by
> the networking core in order of discovery.
> 
>> but I can only list the sata 
>> devices in the system by going through the list of devices the SCSI layer has 
>> detected and working backwards...
>>  
>> It's entirely possible my metaphor is wrong here, but I'm trying to 
>> understand...
>>
>> Rob
> 

Re: Some quick scsi documentation questions:

[Rob Landley]

On Sunday 05 August 2007 11:50:33 am Douglas Gilbert wrote:
> Stefan Richter wrote:
> > Rob Landley wrote:
> >> So an sg device is for something like a scanner that doesn't
> >> present a block device?
> >
> > That's what I heard.
>
> The sg driver is a SCSI pass-through while the bsg driver is
> a pass-through the specializes in SCSI and other (usually)
> storage related protocols. Both work at the level of industry
> recognized standard (and draft) protocols.
>
> That approach makes the Linux block layer either a nuisance,
> irrelevant or a complete anachronism (in the case of OSD).

There are non-scsi block devices in the world, you know.

> IMO the linux block layer should be morphed into a library
> of internal queue handling routines. Storage upper level
> drivers such as sd can continue to present the "block"
> view ** of storage devices such as disks.

Ok, so when do embedded flash volumes (non-USB) start getting routed through 
the scsi layer?  Shortly after ramdisks do?

> ** In OO terminology a storage device may have a "block"
>    interface. A storage device is not derived from "block".

I have no idea what that means.

> In practical terms the block layer SG_IO ioctl is still a
> scaled down version of what the corresponding ioctl in
> the sg driver can do.

Is there anything interesting that you _can't_ do through SG_IO that you could 
do through sg?

> The block layer SG_IO ioctl is saddled 
> with whatever strange policy the upper level driver might
> have for that device class (e.g. wait forever if removal media
> is not present). Also the block layer SG_IO ioctl cannot do
> asynchronous IO (either can the bsg driver at the moment).

This sounds like A) an implementation detail, B) a performance hack for 
seldom-used things like low level formats.

> There was a version of Fedora that came out with sg devices
> only available for SCSI devices not already claimed by
> other upper level drivers. It was a surprise to me (Arjan
> may have mentioned it on this list). I got some complaints
> (as if I could do anything). Anyway it was amusing
> to watch how quickly that misstep was reversed. Obviously
> some folks with a lot me influence than me got to the Fedora
> designers who did that.

Fedora is the new name for Red Hat Enterprise Rawhide.  Red Hat figured out 
how Sun made all its money (large corporate and government purchasing 
contracts that cap the profit at the percentage of the cost of materials, so 
people bidding on those contracts spec the most expensive materials they can.  
If they use a $25 copy of Linux they get $2.50 but if they use a $5000 copy 
of Solaris they get $500.  And so they went "hey, if it makes you happy we 
can charge $5000 too" and came out with Red Hat Enterprise, which was so 
lucrative they left everything else behind.) 

These days Fedora listens to money, not to technology.  If somebody like IBM 
says "do X, here's a big check", are they going to turn it down?

> >> There are such things as external SATA enclosures, but
> >> they're A) few and far between,
> >
> > What?  They are all the rage now.  :-)
>
> ATA8-ACS (still draft I think) has provision for a NAA-5
> based UUID. Not sure if any SATA disks are complying
> yet. [I don't think the Seagate ES series ("enterprise")
> did so it will be interesting if their recently announced
> ES.2 series does.]
>
> Yes SATA external enclosures are everywhere. They make more
> sense than USB 2 and marginalize 1394. The problems start
> for SATA when you want to have more than one disk in that
> enclosure. SAS is much better as an interconnect.

My point is that there are something like ten million of laptops produced 
annually and with enclosures you're talking about a technology where the 
entire production run generally doesn't hit six figures in unit volume.  That 
would be "few and far between" in my book.  (Don't get me started on cell 
phones. :)

> >> In 99.x% of systems with SATA hard drives, said drives are sealed inside
> >> the machine, whether it's a laptop or a server.  USB can move around
> >> between boots (or with the power on) as a matter of course, but SATA
> >> really isn't designed for that or expected to do that.
> >
> > I suppose you could still have a not too complicated topology-based
> > naming scheme even with eSATA.
>
> Recent SAS-2 drafts have some heuristic for ATA disks (that
> don't have real UUIDs) that combines the manufacturer, model
> and serial number into a single number for identification
> purposes. With potentially hundreds of SATA disks hanging
> off SAS infrastructure, someone who changed the slot a SATA
> disk was connected to could cause a lot of fun.

But there aren't potentially hundreds of SATA disks hanging off my laptop, and 
my laptop is actually a common case of hardware.  You are penalizing the 
common case in favor of systems that cost more than the salary of the 
full-time professional system administrator assigned to them.

> >> You can reliably enumerate ATA and SATA.
> >
> > You mean, you can reliably enumerate ATA and SATA based on bus topology.
> > Do SATA port multipliers still fit into the picture?
>
> Hopefully they will be chased out of the picture by SAS
> expanders.

If it meant that sata would stop being routed through the scsi layer and could 
thus be reliably enumerated without sacrificing a chicken, I'd be all for it.  
Alas, this doesn't seem likely.

> When a SATA port multiplier is used
>   - the SATA controller and its driver must detect it and
>     know what to do

In which case you should be able to detect the ABSENCE of a port multiplier, 
and reliably enumerate the devices that aren't behind one.

> > All newer SCSI transports are far too complicated for a kernelspace-only
> > implementation.  So in case of such transports, it's not a workaround.
>
> Ah user space device discovery! If only folks wouldn't
> design things so that protocols used to device discovery
> (yes I am talking about the SAS Management Protocol [SMP])
> did _not_ need discovered (or about to be discovered)
> devices already in the kernel's sysfs or dev space.

I worked with SAS for three months before running far, far away from it.  
You'll notice I'm not the one who brought it up.  As far as I can tell, they 
took sata, gold-plated the connectors, spun the platters faster, added an 
incompatable second mode to the firmware that has no actual benefits, added a 
second set of connectors (because the electronics are so much more likely to 
fail than the moving parts), and went "we can charge WAY more for this!"  And 
did.

> I agree with both angles. The linux SCSI/storage subsystem
> should look a lot more like the networking layers. No-one
> tries to enumerate the whole IP space but it may make sense
> to enumerate a small part of it (e.g. MAC addresses in the
> directly attached ethernet subnet).
>
> If any enumeration is required then it should be done by
> user space tools (e.g. udev) interacting with the transport
> concerned. That also implies that SCSI hosts and ATA
> controllers need to be promoted to "first class" devices
> (cf "interfaces" in networking), not hidden as if they were
> some sort of an embarrassment.

I'm all for it.

I'm happy to do enumeration from userspace if I have the information to do so.  
Right now "this is a SATA drive" isn't exposed in sysfs, that I can find.

Rob
-- 
"One of my most productive days was throwing away 1000 lines of code."
  - Ken Thompson.

Re: Some quick scsi documentation questions:

[Stefan Richter]

Rob Landley wrote:
> On Sunday 05 August 2007 11:50:33 am Douglas Gilbert wrote:
>> Stefan Richter wrote:
>>> Rob Landley wrote:
>>>> There are such things as external SATA enclosures, but
>>>> they're A) few and far between,
>>> What?  They are all the rage now.  :-)
...
> My point is that there are something like ten million of laptops produced 
> annually and with enclosures you're talking about a technology where the 
> entire production run generally doesn't hit six figures in unit volume.  That 
> would be "few and far between" in my book.  (Don't get me started on cell 
> phones. :)

The case of a laptop with a single sealed "spindle" (and maybe an
optical drive) is easy.  Among else, you have the following options:
  - Let the distributor figure it out for you.
  - Link the driver for the onboard SATA controller statically into
    the kernel.  Make all other SCSI low-level providers (e.g. usb-
    storage) modular.

In other words, if SATA and USB and other disks are mixed up, let the
distributor deal with the mix-up.  You only ever deal with device
aliases in /dev/disk/by-*/ or with an even higher-level representation
by the desktop environment.  _Or_ you enforce that the built-in SATA
drive is always /dev/sda, by having this drive always added first.

...
> Right now "this is a SATA drive" isn't exposed in sysfs, that I can find.

Have a look at the udev scripts or at Doug's lsscsi.  Or simply use
lsscsi like a blackbox.
-- 
Stefan Richter
-=====-=-=== =--- --==-
http://arcgraph.de/sr/