sg driver and Fedora Core 2

sg driver and Fedora Core 2

[Douglas Gilbert]

This is slightly off topic (discussing a particular
distribution).

The recently released Fedora Core 2 distribution contains
a patch that allocates sg device names (e.g. /dev/sg0)
only to those SCSI devices _not_ "claimed" by other upper
level drivers (i.e. sd, sr, st and osst). There is no
kernel (or module) load time parameter to alter this
behaviour. There is no way to bind a sg device name to
a SCSI device after it has been attached by the SCSI mid
level (but perhaps there should be ...).

It seems that the intention of this change is to force cdrecord
users to use the /dev/scd or /dev/hd device names (rather than
the /dev/sg devices names that were used in the lk 2.0, 2.2
and 2.4 series). While I agree that encouraging the use
of the more natural /dev/scd or /dev/hd devices make sense for
cdrecord, there are some lesser used applications
that are broken by the "forcing" nature of this change (e.g.
mtx).

Recently I have had a query about how a specialist application
(that worked in the lk 2.4 series) sends 16 MB data through a
single SCSI command in Fedora Core 2. The simple answer to that
one is the block layer imposes a 128 KB limit on single
transfers and that's it. [Counter-intuitively that limit also
applies to st and osst when they use the SG_IO ioctl.] Around a
year ago I tried to move Jens Axboe on this issue and failed.
Evidentally there are good reasons why the block layer imposes
that limit. There are other issues with this change.

There seems to be mixed signals coming from the Fedora camp
on this change. A "policy" change was one response and this
url ( http://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=123876 )
has Alan Cox stating that this change is a bug.

I am not aware of any such change (or pending change) in the
"mainline" linux kernel (i.e. the lk 2.6 source tree controlled
by Linus Torvalds).

Doug Gilbert

Re: sg driver and Fedora Core 2

[Jens Axboe]

On Sat, May 29 2004, Douglas Gilbert wrote:
> Recently I have had a query about how a specialist application
> (that worked in the lk 2.4 series) sends 16 MB data through a
> single SCSI command in Fedora Core 2. The simple answer to that
> one is the block layer imposes a 128 KB limit on single
> transfers and that's it. [Counter-intuitively that limit also
> applies to st and osst when they use the SG_IO ioctl.] Around a
> year ago I tried to move Jens Axboe on this issue and failed.
> Evidentally there are good reasons why the block layer imposes
> that limit. There are other issues with this change.

The block layer does not impose any 128KB limitation. In fact there
should be no real limitations (*). The block layer honors whatever
restrictions that the drivers sets, 128KB is the maximum transfer size
for an ATAPI drive for instance.

You could flag queue capability to handle requests > max_sectors like
SCSI has always been able to. Then you could change this check

        if (hdr->dxfer_len > (q->max_sectors << 9))
                return -EIO;

to only fail for drivers that can't partially complete requests
correctly. This would allow such applications to continue working.

(*) Not quite true, since the max size of a bio is 1MB on 4KB page sized
    hardware. You can string them for bigger total size, though.
> 
> There seems to be mixed signals coming from the Fedora camp
> on this change. A "policy" change was one response and this
> url ( http://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=123876 )
> has Alan Cox stating that this change is a bug.

That's not the identical issue, it's due to usb storage setting 120KB as
the max size causing sg_set_reserved_size() to fail.

-- 
Jens Axboe

Re: sg driver and Fedora Core 2

[Jens Axboe]

On Fri, May 28 2004, Jens Axboe wrote:
> > There seems to be mixed signals coming from the Fedora camp
> > on this change. A "policy" change was one response and this
> > url ( http://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=123876 )
> > has Alan Cox stating that this change is a bug.
> 
> That's not the identical issue, it's due to usb storage setting 120KB as
> the max size causing sg_set_reserved_size() to fail.

>From what I read of the documentation on sgv3, the reserved size is some
old relic that isn't even of interesting to the block layer sg
"implementation" (we don't keep memory reserved for requests, it's
mapped from the user).

Douglas, can I expect applications to break with this patch? They are
allowed to set a reserved size of eg 512KB, even if we'll later reject a
256KB command if it's sent to an ATAPI drive.

===== drivers/block/scsi_ioctl.c 1.42 vs edited =====
--- 1.42/drivers/block/scsi_ioctl.c	2004-04-27 15:20:34 +02:00
+++ edited/drivers/block/scsi_ioctl.c	2004-05-28 16:26:10 +02:00
@@ -89,8 +89,6 @@
 
 	if (size < 0)
 		return -EINVAL;
-	if (size > (q->max_sectors << 9))
-		return -EINVAL;
 
 	q->sg_reserved_size = size;
 	return 0;

-- 
Jens Axboe

Re: sg driver and Fedora Core 2

[Alan Cox]

On Sat, May 29, 2004 at 12:05:25AM +1000, Douglas Gilbert wrote:
> The recently released Fedora Core 2 distribution contains
> a patch that allocates sg device names (e.g. /dev/sg0)
> only to those SCSI devices _not_ "claimed" by other upper

This is an error. Linus has also vetoed this change. Consider the
Fedora kernel wrong on this point. It isnt however your SG_IO issue I think

Re: sg driver and Fedora Core 2

[Arjan van de Ven]

[-- Attachment #1: Type: text/plain, Size: 366 bytes --]


> that are broken by the "forcing" nature of this change (e.g.
> mtx).

users seem to say that mtx works as long as you point it to the right
device.... after all the ioctls are the same

THe only big difference seems to be that the generic ioctl does enforce
queue limits while sg does "something else" when it an
invalid-for-the-device IO gets issued..

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: sg driver and Fedora Core 2

[James Bottomley]

On Fri, 2004-05-28 at 12:25, Alan Cox wrote:
> On Sat, May 29, 2004 at 12:05:25AM +1000, Douglas Gilbert wrote:
> > The recently released Fedora Core 2 distribution contains
> > a patch that allocates sg device names (e.g. /dev/sg0)
> > only to those SCSI devices _not_ "claimed" by other upper
> 
> This is an error. Linus has also vetoed this change. Consider the
> Fedora kernel wrong on this point. It isnt however your SG_IO issue I think

I don't necessarily regard this as an error: single attachment of ULDs
would be very helpful.  However, since the mechanics of implementing
single attachment are difficult (sg has to be the default attachment of
every unclaimed devices, so you need to sort out issues when an ULD is
probed *after* sg is attached etc.), I'm very happy for Fedora to
demonstrate the feasibility of its implementation before a patch is
proposed to mainline.

James

Re: sg driver and Fedora Core 2

[Alan Cox]

On Sat, May 29, 2004 at 10:55:24AM -0500, James Bottomley wrote:
> > This is an error. Linus has also vetoed this change. Consider the
> > Fedora kernel wrong on this point. It isnt however your SG_IO issue I think
> 
> I don't necessarily regard this as an error: single attachment of ULDs
> would be very helpful.  However, since the mechanics of implementing

Its most definitely problematic - it breaks a ton of existing tools, and
it also doesn't do what you think. Try this for size

insmod scsidriver
open /dev/sg0 (a hard disk)
insmod sd

Apps should migrate to the new facilities when possible but its definitely
not a good idea to make this change

Re: sg driver and Fedora Core 2

[James Bottomley]

On Sat, 2004-05-29 at 10:57, Alan Cox wrote:
> On Sat, May 29, 2004 at 10:55:24AM -0500, James Bottomley wrote:
> > > This is an error. Linus has also vetoed this change. Consider the
> > > Fedora kernel wrong on this point. It isnt however your SG_IO issue I think
> > 
> > I don't necessarily regard this as an error: single attachment of ULDs
> > would be very helpful.  However, since the mechanics of implementing
> 
> Its most definitely problematic - it breaks a ton of existing tools, and
> it also doesn't do what you think. Try this for size
> 
> insmod scsidriver
> open /dev/sg0 (a hard disk)
> insmod sd

Refuse the insertion or do not attach, I suppose.  It would be the app's
fault for not managing the interface correctly.

> Apps should migrate to the new facilities when possible but its definitely
> not a good idea to make this change

Yes, that's why a change like this appeals.

However, as I said, I'm aware it's problematic which is why I'm not
about to make any change like it at this point.  But, if fedora can
demonstrate that it can be made to work sanely and easily, that would be
a very powerful argument for inclusion.

James

Re: sg driver and Fedora Core 2

[Arjan van de Ven]

[-- Attachment #1: Type: text/plain, Size: 696 bytes --]

On Sat, May 29, 2004 at 11:57:44AM -0400, Alan Cox wrote:
> On Sat, May 29, 2004 at 10:55:24AM -0500, James Bottomley wrote:
> > > This is an error. Linus has also vetoed this change. Consider the
> > > Fedora kernel wrong on this point. It isnt however your SG_IO issue I think
> > 
> > I don't necessarily regard this as an error: single attachment of ULDs
> > would be very helpful.  However, since the mechanics of implementing
> 
> Its most definitely problematic - it breaks a ton of existing tools, and

most of the existing tools take a device as argument, it doesn't (shouldn't)
matter if that device name is /dev/sg0 or /dev/sda. After all the ioctls are
the same between sg and SG_IO.

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: sg driver and Fedora Core 2

[Alan Cox]

On Sat, May 29, 2004 at 11:07:19AM -0500, James Bottomley wrote:
> > insmod scsidriver
> > open /dev/sg0 (a hard disk)
> > insmod sd
> 
> Refuse the insertion or do not attach, I suppose.  It would be the app's
> fault for not managing the interface correctly.

So you stop users being able to hotplug controllers while doing firmware
updates. Thats not good either

> about to make any change like it at this point.  But, if fedora can
> demonstrate that it can be made to work sanely and easily, that would be
> a very powerful argument for inclusion.

Fedora bugzilla has demonstrated it *doesn't* work. There is a long list
of broken third party apps such as drive firmware tools many of which will
never include source code. The whole point of sg is arbitary management of
devices. Its like /dev/mem and so forth, its power is in the fact it can
always be used. This is why Linus vetoed the entire concept.

Fedora hasn't solved the open /dev/sg , insmod case either.

Re: sg driver and Fedora Core 2

[Arjan van de Ven]

[-- Attachment #1: Type: text/plain, Size: 906 bytes --]

On Sat, May 29, 2004 at 12:29:12PM -0400, Alan Cox wrote:
> 
> Fedora bugzilla has demonstrated it *doesn't* work. 

yes cdrecord broke due to a bug in cdrecord and a somewhat discutable
behavior difference between SG_IO and /dev/sg when it comes to oversized IO
(which cdrecord issues as a result of it's own bug)

> There is a long list
> of broken third party apps such as drive firmware tools many of which will
> never include source code. The whole point of sg is arbitary management of
> devices. 

*AND SG_IO IS THAT*. You seem to ignore that consistently.
What changed is that you don't need top open some OTHER device to send
arbitrary commands to /dev/sda, but you can open /dev/sda directly to send
those commands, with the same ioctl. THAT changed. Don't tell me that very
long list of third party apps ALL doesn't take a device as argument....
undoubtedly several don't but... "long list"...

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: sg driver and Fedora Core 2

[Alan Cox]

On Sat, May 29, 2004 at 06:36:42PM +0200, Arjan van de Ven wrote:
> > never include source code. The whole point of sg is arbitary management of
> > devices. 
> 
> *AND SG_IO IS THAT*. You seem to ignore that consistently.

Unfortunately my devices randomly change name according to what is loaded

> those commands, with the same ioctl. THAT changed. Don't tell me that very
> long list of third party apps ALL doesn't take a device as argument....
> undoubtedly several don't but... "long list"...

I had to recompile my kernel to flash my scsi drive firmware. It works
in the generic kernel but not in 2.6.5-Fedora. That alone says its not
an acceptable change for a "stable" kernel tree - which hopefully 2.6.x
will become in the next six months.

(If you want to argue about 2.7.x then its somewhat different - if scsi
 devices appear in sysfs and the sysfs nodes also support SG_IO then the
 entire notion of /dev/sg could go away in 2.7)

Re: sg driver and Fedora Core 2

[James Bottomley]

On Sat, 2004-05-29 at 11:29, Alan Cox wrote:
> On Sat, May 29, 2004 at 11:07:19AM -0500, James Bottomley wrote:
> > > insmod scsidriver
> > > open /dev/sg0 (a hard disk)
> > > insmod sd
> > 
> > Refuse the insertion or do not attach, I suppose.  It would be the app's
> > fault for not managing the interface correctly.
> 
> So you stop users being able to hotplug controllers while doing firmware
> updates. Thats not good either

This question only arises if the fw update tools are using the old
interface ... I think an increasing level of pain on that front might
cause even closed source vendors to consider shifting to the new
interface using sd for the disc firmware update.

> > about to make any change like it at this point.  But, if fedora can
> > demonstrate that it can be made to work sanely and easily, that would be
> > a very powerful argument for inclusion.
> 
> Fedora bugzilla has demonstrated it *doesn't* work. There is a long list
> of broken third party apps such as drive firmware tools many of which will
> never include source code. The whole point of sg is arbitary management of
> devices. Its like /dev/mem and so forth, its power is in the fact it can
> always be used. This is why Linus vetoed the entire concept.

If there's enough evidence, open minds can be changed.

> Fedora hasn't solved the open /dev/sg , insmod case either.

I think it's too early to declare this experiment in Fedora a failure. 
If the Fedora release people ultimately decide that the only way to sort
all the problems out is to back out the single attachment patch then
fine, we'll declare the experiment a failure and not consider anything
like it for mainline.  However, while fedora is still trying to get
everything working with single attachment, I'm not going to prejudge.

James

Re: sg driver and Fedora Core 2

[Alan Cox]

On Sat, May 29, 2004 at 11:49:58AM -0500, James Bottomley wrote:
> This question only arises if the fw update tools are using the old
> interface ... I think an increasing level of pain on that front might
> cause even closed source vendors to consider shifting to the new
> interface using sd for the disc firmware update.

You think the drive vendors *care* about Linux flash updating tools. You
think they will spend anything but the tiniest amount of funding on it. You
think anyone who gets told "the major supplier of this OS has arbitarily
broken your software during stable releases for no good reason" is likely
to update their firmware.

No. Not a chance. Its also nigh on impossible to use /dev/sd to scan the
scsi busses for devices because you end up force loading modules, waiting
for USB devices to come on line and other nastiness.

> If the Fedora release people ultimately decide that the only way to sort
> all the problems out is to back out the single attachment patch then
> fine, we'll declare the experiment a failure and not consider anything

Given Linus has vetoed the change for 2.6 the experiment is dead. Its a
"won't happen". Doing things with sysfs and 2.7 I can see, since it would
be enumerable, would allow you to walk driver and device paths. The Fedora
stuff is really irrelevant, the patch is a one person crusade without 
consideration of the users

Alan

Re: sg driver and Fedora Core 2

[Jeff Garzik]

Really what needs to happen is a generic userland tap attaches to a 
chrdev, issues an ioctl(2) to attach to a request_queue, and then does 
read(2)/write(2)/mmap(2) to communicate with the request_queue.

:)

Call it /dev/bsg, or somesuch.

The SG_IO ioctl won't work for all cases, since some block devices may 
rightly wish to limit (or block) multiple openers on the blkdev itself. 
   OTOH, the SG_IO ioctl does eliminate the userland app needing to 
worry about _any_ target/addressing information, since that is implicit 
in the dentry pointing to the blkdev's inode.

	Jeff

Re: sg driver and Fedora Core 2

[James Bottomley]

On Sat, 2004-05-29 at 11:56, Alan Cox wrote:
> On Sat, May 29, 2004 at 11:49:58AM -0500, James Bottomley wrote:
> > This question only arises if the fw update tools are using the old
> > interface ... I think an increasing level of pain on that front might
> > cause even closed source vendors to consider shifting to the new
> > interface using sd for the disc firmware update.
> 
> You think the drive vendors *care* about Linux flash updating tools. You
> think they will spend anything but the tiniest amount of funding on it. You
> think anyone who gets told "the major supplier of this OS has arbitarily
> broken your software during stable releases for no good reason" is likely
> to update their firmware.

Actually, yes, I think Linux is growing enough market momentum for them
to take notice.

> No. Not a chance. Its also nigh on impossible to use /dev/sd to scan the
> scsi busses for devices because you end up force loading modules, waiting
> for USB devices to come on line and other nastiness.

depending on what your after, device scanning (without any ULDs,
including sg) is simple in 2.6; it's ls /sys/bus/scsi/devices.

> > If the Fedora release people ultimately decide that the only way to sort
> > all the problems out is to back out the single attachment patch then
> > fine, we'll declare the experiment a failure and not consider anything
> 
> Given Linus has vetoed the change for 2.6 the experiment is dead. Its a
> "won't happen". Doing things with sysfs and 2.7 I can see, since it would
> be enumerable, would allow you to walk driver and device paths. The Fedora
> stuff is really irrelevant, the patch is a one person crusade without 
> consideration of the users

There's no patch to veto yet.  If you redhat people are taking it out of
fedora, then, for 2.6, there probably never will be.

I'm not saying "yes" to it's inclusion, I'm just not prepared to say
"no" until I see the patch and it gets debated on the list.

James

Re: sg driver and Fedora Core 2

[Jens Axboe]

On Sat, May 29 2004, Jeff Garzik wrote:
> Really what needs to happen is a generic userland tap attaches to a 
> chrdev, issues an ioctl(2) to attach to a request_queue, and then does 
> read(2)/write(2)/mmap(2) to communicate with the request_queue.
> 
> :)
> 
> Call it /dev/bsg, or somesuch.
> 
> The SG_IO ioctl won't work for all cases, since some block devices may 
> rightly wish to limit (or block) multiple openers on the blkdev itself. 
>   OTOH, the SG_IO ioctl does eliminate the userland app needing to 
> worry about _any_ target/addressing information, since that is implicit 
> in the dentry pointing to the blkdev's inode.

Hear, hear!

-- 
Jens Axboe

Re: sg driver and Fedora Core 2

[Alan Cox]

On Sat, May 29, 2004 at 01:27:58PM -0400, Jeff Garzik wrote:
> Really what needs to happen is a generic userland tap attaches to a 
> chrdev, issues an ioctl(2) to attach to a request_queue, and then does 
> read(2)/write(2)/mmap(2) to communicate with the request_queue.

Is there a reason that couldn't be part of the block layer sysfs in 2.7.x 

> :)
> Call it /dev/bsg, or somesuch.

Except that I wonder if it should be in sysfs I agree entirely. 

>   OTOH, the SG_IO ioctl does eliminate the userland app needing to 
> worry about _any_ target/addressing information, since that is implicit 
> in the dentry pointing to the blkdev's inode.

It makes it very hard to scan scsi busses, which is what the flash burner
app does, and what xsane does (although sane survives ok because there
are no kernel scanner device drivers)

Alan

Re: sg driver and Fedora Core 2

[Alan Cox]

On Sat, May 29, 2004 at 12:28:45PM -0500, James Bottomley wrote:
> > think anyone who gets told "the major supplier of this OS has arbitarily
> > broken your software during stable releases for no good reason" is likely
> > to update their firmware.
> 
> Actually, yes, I think Linux is growing enough market momentum for them
> to take notice.

If we act professionally maybe. But breaking that the way FC2 did wasnt
professional

> depending on what your after, device scanning (without any ULDs,
> including sg) is simple in 2.6; it's ls /sys/bus/scsi/devices.

and opening those is probably a 2.7 way to get rid of /dev/sg

> There's no patch to veto yet.  If you redhat people are taking it out of
> fedora, then, for 2.6, there probably never will be.

That is Arjan's call. I already build Fedora 2 kernels with it removed
and I'm far from unique in that. The simple fact its an API change makes
it a 2.7 kernel thing.

Re: sg driver and Fedora Core 2

[James Bottomley]

On Sat, 2004-05-29 at 12:27, Jeff Garzik wrote:
> Really what needs to happen is a generic userland tap attaches to a 
> chrdev, issues an ioctl(2) to attach to a request_queue, and then does 
> read(2)/write(2)/mmap(2) to communicate with the request_queue.
> 
> :)
> 
> Call it /dev/bsg, or somesuch.

Well, this touches on something else I'm thinking about: lazy
attachment.  In multi thousand disc environments, you don't want us
wasting resources allocating a queue and reading a partition table for
all of them; you really only want to do that for the devices you're
actually going to use.  Once we have this, I think dual attachment will
be a dead issue.

> The SG_IO ioctl won't work for all cases, since some block devices may 
> rightly wish to limit (or block) multiple openers on the blkdev itself. 
>    OTOH, the SG_IO ioctl does eliminate the userland app needing to 
> worry about _any_ target/addressing information, since that is implicit 
> in the dentry pointing to the blkdev's inode.

Yes, the mapping issues is the huge problem where users keep shooting
themselves in the foot.

James

Re: sg driver and Fedora Core 2

[Jeff Garzik]

Alan Cox wrote:
> On Sat, May 29, 2004 at 01:27:58PM -0400, Jeff Garzik wrote:
> 
>>Really what needs to happen is a generic userland tap attaches to a 
>>chrdev, issues an ioctl(2) to attach to a request_queue, and then does 
>>read(2)/write(2)/mmap(2) to communicate with the request_queue.
> 
> 
> Is there a reason that couldn't be part of the block layer sysfs in 2.7.x 
> 
> 
>>:)
>>Call it /dev/bsg, or somesuch.
> 
> 
> Except that I wonder if it should be in sysfs I agree entirely. 

I'll let others haggle over the device name and location <grin>  sysfs 
or whereever is fine with me.  I just care about the chrdev's behavior :)


>>  OTOH, the SG_IO ioctl does eliminate the userland app needing to 
>>worry about _any_ target/addressing information, since that is implicit 
>>in the dentry pointing to the blkdev's inode.
> 
> 
> It makes it very hard to scan scsi busses, which is what the flash burner
> app does, and what xsane does (although sane survives ok because there
> are no kernel scanner device drivers)


This is precisely the problem with the only-SG_IO approach as well:

AFAICS, you lose the ability to scan the bus, just as you gain the 
ability to not have to care about scanning the bus.  Getting back on 
topic, I agree with you that the Fedora change [if it is as you 
describe] is not a good one.  And breakage in a stable kernel version is 
also quite annoying.

	Jeff

Re: sg driver and Fedora Core 2

[Matthew Wilcox]

On Sat, May 29, 2004 at 12:42:39PM -0400, Alan Cox wrote:
> (If you want to argue about 2.7.x then its somewhat different - if scsi
>  devices appear in sysfs and the sysfs nodes also support SG_IO then the
>  entire notion of /dev/sg could go away in 2.7)

That would involve supporting ioctls in sysfs, which is something that's
been opposed so far.  I do agree that you want to flash disc firmware
using a hierarchical path rather than a flat namespace that you have
to double-check to be sure you're not flashing your scanner with disc
firmware.

Possibly we could have /sys/bus/scsi/devices/1\:0\:1\:0/raw_io that
internally uses the SG_IO codepath.  We'd need to rewrite the raw file
interface to support the large IOs necessary, but that's certainly doable.

-- 
"Next the statesmen will invent cheap lies, putting the blame upon 
the nation that is attacked, and every man will be glad of those
conscience-soothing falsities, and will diligently study them, and refuse
to examine any refutations of them; and thus he will by and by convince 
himself that the war is just, and will thank God for the better sleep 
he enjoys after this process of grotesque self-deception." -- Mark Twain

Re: sg driver and Fedora Core 2

[Alan Cox]

On Sat, May 29, 2004 at 12:38:18PM -0500, James Bottomley wrote:
> > worry about _any_ target/addressing information, since that is implicit 
> > in the dentry pointing to the blkdev's inode.
> 
> Yes, the mapping issues is the huge problem where users keep shooting
> themselves in the foot.

That is actually a dying issue (aside from the whole /dev/sg debate). Users
select the CD burner by vendor/device name from the pull down menu in real
world computing. The underlying question is how the GUI application does the
scan and how it figures out what burner ties to what cd-rom device in the
file manager. SG_IO is wonderful here and has solved the latter part nicely.
sysfs has solved the former part.

Alan

Re: sg driver and Fedora Core 2

[Jeff Garzik]

James Bottomley wrote:
> On Sat, 2004-05-29 at 12:27, Jeff Garzik wrote:
> 
>>Really what needs to happen is a generic userland tap attaches to a 
>>chrdev, issues an ioctl(2) to attach to a request_queue, and then does 
>>read(2)/write(2)/mmap(2) to communicate with the request_queue.
>>
>>:)
>>
>>Call it /dev/bsg, or somesuch.
> 
> 
> Well, this touches on something else I'm thinking about: lazy
> attachment.  In multi thousand disc environments, you don't want us
> wasting resources allocating a queue and reading a partition table for
> all of them; you really only want to do that for the devices you're
> actually going to use.  Once we have this, I think dual attachment will
> be a dead issue.


Regardless of lazy attachment, request_queues need to support multiple 
simultaneous producers of struct request's.

	Jeff

Re: sg driver and Fedora Core 2

[Jens Axboe]

On Sat, May 29 2004, Jeff Garzik wrote:
> James Bottomley wrote:
> >On Sat, 2004-05-29 at 12:27, Jeff Garzik wrote:
> >
> >>Really what needs to happen is a generic userland tap attaches to a 
> >>chrdev, issues an ioctl(2) to attach to a request_queue, and then does 
> >>read(2)/write(2)/mmap(2) to communicate with the request_queue.
> >>
> >>:)
> >>
> >>Call it /dev/bsg, or somesuch.
> >
> >
> >Well, this touches on something else I'm thinking about: lazy
> >attachment.  In multi thousand disc environments, you don't want us
> >wasting resources allocating a queue and reading a partition table for
> >all of them; you really only want to do that for the devices you're
> >actually going to use.  Once we have this, I think dual attachment will
> >be a dead issue.
> 
> 
> Regardless of lazy attachment, request_queues need to support multiple 
> simultaneous producers of struct request's.

That has always been the case, even if it was just driver private use
before. As such there is no dual-attachment issue for sg.

-- 
Jens Axboe

Re: sg driver and Fedora Core 2

[Douglas Gilbert]

Jens Axboe wrote:
> On Sat, May 29 2004, Jeff Garzik wrote:
> 
>>Really what needs to happen is a generic userland tap attaches to a 
>>chrdev, issues an ioctl(2) to attach to a request_queue, and then does 
>>read(2)/write(2)/mmap(2) to communicate with the request_queue.
>>
>>:)
>>
>>Call it /dev/bsg, or somesuch.
>>
>>The SG_IO ioctl won't work for all cases, since some block devices may 
>>rightly wish to limit (or block) multiple openers on the blkdev itself. 
>>  OTOH, the SG_IO ioctl does eliminate the userland app needing to 
>>worry about _any_ target/addressing information, since that is implicit 
>>in the dentry pointing to the blkdev's inode.
> 
> 
> Hear, hear!

Jeff + Jens,
Yes, that is what I was getting at with my "bind"
idea. Could a "misc" char device minor number for
/dev/bsg be made available? It isn't clear to me how sysfs
can be used to do what Jeff proposes.


I have no expertise in Fibre Channel but Serial Attached SCSI
(SAS) throws up more addressing problems. There can be up to
16,000 devices in a single SAS domain the bulk of which would
be targets **, expanders which are the interesting ones, and
other initiators (potentially on other machines) will
also be visible. Those expanders are critical to
configuring the SAS domain. Expanders are SAS devices but
not SCSI devices (i.e. they are neither initiators nor
targets ***). It seems obvious that the management of a SAS
domain should be done from the user space. Only those
expanders and devices directly attached to a SAS Host Bus
Adapter should automatically be visible.

Enumerating all the disks, and/or all the scsi devices
(as either devices or sysfs entries) makes less and
less sense. In the case of SAS, devices can be discovered
by recursive descent of the expanders (via a bind through
/dev/bsg for example). In a large system one possible policy
could be: only if the user is interested in a
particular device should it be presented as a device
node (in /dev, /udev or /sys).


My recent patch to scsi_debug (adding up to 4 partitions per
pseudo device) "allows" (6 * 32768) device nodes to be
created! To play this game, turn off udev, hotplug and set
delay=0 in scsi_debug:-) I didn't get far with 512 MB of ram,
Patrick Mansfield got further with a multi gigabyte machine.
Seems like sysfs eats up most of the ram. To isolate out the
impact of the block subsystem the pseudo devices can be set to
be enclosure peripheral type (for example) via a scsi_debug
driver argument.


** targets include both SATA and SAS disks

*** show should SAS expanders go in /sys/class/scsi_device ?

Doug Gilbert

Re: sg driver and Fedora Core 2

[Alan Cox]

On Sun, May 30, 2004 at 08:37:14PM +1000, Douglas Gilbert wrote:
> configuring the SAS domain. Expanders are SAS devices but
> not SCSI devices (i.e. they are neither initiators nor
> targets ***). It seems obvious that the management of a SAS
> domain should be done from the user space. Only those
> expanders and devices directly attached to a SAS Host Bus
> Adapter should automatically be visible.

Is there any reason that the SAS domain should in itself be a character
device file. I've always thought of it as a network and anticipated
a network layer interface for such things. That might just be my background
however.

Re: sg driver and Fedora Core 2

[Douglas Gilbert]

Alan Cox wrote:
> On Sun, May 30, 2004 at 08:37:14PM +1000, Douglas Gilbert wrote:
> 
>>configuring the SAS domain. Expanders are SAS devices but
>>not SCSI devices (i.e. they are neither initiators nor
>>targets ***). It seems obvious that the management of a SAS
>>domain should be done from the user space. Only those
>>expanders and devices directly attached to a SAS Host Bus
>>Adapter should automatically be visible.
> 
> 
> Is there any reason that the SAS domain should in itself be a character
> device file. I've always thought of it as a network and anticipated
> a network layer interface for such things. That might just be my background
> however.

Alan,
In the SCSI parallel world the "domain" is usually
called the "bus" which is a set of targets and one or more
initiators and the ribbon (or external) cable that
interconnect them.

In SAS the "domain" is a little more rubbery. After wide
links are taken into account, there cannot be multiple
paths between scsi devices within one SAS domain (i.e. "no
loops"). An expander is part of the service interconnect
subsystem of a _single_ SAS domain (i.e. it can't be shared).
This means that expanders can be wired incorrectly.
Even simple interconnects need some thought:

     |   SAS  2   |
     |  port HBA  |
     --------------
         |     |
         |     |
         |     |
         |     |
      -------------
      | SAS dual  |
      |ported disk|

Is this a (2 phy) wide-link, single SAS domain or two SAS
domains? On current indications it is two SAS domains since
dual ported SAS disks have different SAS addresses on each
port. [The fact that the SAS HBA has the same SAS addresses
on both phys only meets half the requirement for a wide link.]

See SAS Quiz question 1 (then its answer) in the pdfs at:
http://www.scsita.org/aboutscsi/sas/tutorials.html
for a more complex scenario.

Detecting and explaining to a user why his/her SAS domain(s)
is incorrectly wired will be an interesting challenge.


So it wasn't the SCSI domain that I was proposing should
be a char (or network) device file. It was the SMP target
port inside each expander that is used for discovery,
configuration and statistics that needs a packet based
initiator driver/interface (like the sg driver). The concept
of a SAS domain I would rather leave for a user space
program.

Doug Gilbert