Fwd: 32bit dev_t

Fwd: 32bit dev_t

[Douglas Gilbert]

This post appeared recently on the linux kernel list.
What is the maximum number of disks that can be attached
in lk 2.5? Is it 256 (double the number of SCSI disk majors)
or larger. I remember Richard Gooch's "sd-many" patch
was rejected for 2.5 because it was said there was a
better mechanism around the corner.

BTW when I tested 129 (1 real + 128 dummies) disks with
scsi_debug there was a problem the last one. Has anyone
tested > 128 disks in lk 2.5?

Doug Gilbert


Steven Dake wrote:
 >
 > Joel,
 >
 > Linux doesn't really need a 32 bit kdev_t structure to support 1000
 > disks.  There is plenty of device space available to support over 1500
 > disks by modifying the linux scsi layer.
 >
 > Thanks
 > -steve
 >
 > Joel Becker wrote:
 >
 > >Folks,
 > >	Who is tracking the 32bit dev_t effort for 2.5?  There are
 > >already existing installations with 500 to 1000 disks attached to a
 > >system, and I don't know that Linux really wants to wait four
 > >years to get there!
 > >	If someone could point me to a current patch or any current
 > >information about the issues, I'd really appreciate it.
 > >
 > >Joel

Re: Fwd: 32bit dev_t

[Patrick Mansfield]

On Wed, Jan 22, 2003 at 08:56:45AM +1100, Douglas Gilbert wrote:
> This post appeared recently on the linux kernel list.
> What is the maximum number of disks that can be attached
> in lk 2.5? Is it 256 (double the number of SCSI disk majors)
> or larger. I remember Richard Gooch's "sd-many" patch
> was rejected for 2.5 because it was said there was a
> better mechanism around the corner.
> 
> BTW when I tested 129 (1 real + 128 dummies) disks with
> scsi_debug there was a problem the last one. Has anyone
> tested > 128 disks in lk 2.5?
> 
> Doug Gilbert

It should be 16 sd majors * 16 disks per major = 256.

What was the problem with scsi_debug? I thought you got it past 128 LUNs
at some point. There are no longer kmalloc (or vmalloc) issues with sd.c.

-- Patrick Mansfield

Re: Fwd: 32bit dev_t

[Tim Pepper]

I've gotten 256 on RH's latest 2.4.18 rpms.  I'm not sure if they changed
anything that's not also in more recent stock kernels.

I've also run many thousands with a little bit of code somebody
posted here or on l-k a year or two ago that makes scsi disks not have
partitions...

Tim

-- 
*********************************************************
*  tpepper@vato dot org             * Venimus, Vidimus, *
*  http://www.vato.org/~tpepper     * Dolavimus         *
*********************************************************

Re: Fwd: 32bit dev_t

[Joel Becker]

On Wed, Jan 22, 2003 at 03:55:14PM -0800, Tim Pepper wrote:
> I've also run many thousands with a little bit of code somebody
> posted here or on l-k a year or two ago that makes scsi disks not have
> partitions...

	Yes, but "not have partitions" is not something that will be
generally acceptable.  We need real support for this.

Joel


-- 

"Against stupidity the Gods themselves contend in vain."
                                                        -Unknown

Joel Becker
Senior Member of Technical Staff
Oracle Corporation
E-mail: joel.becker@oracle.com
Phone: (650) 506-8127

Re: Fwd: 32bit dev_t

[Kurt Garloff]

[-- Attachment #1: Type: text/plain, Size: 979 bytes --]

Hi Doug,

On Wed, Jan 22, 2003 at 08:56:45AM +1100, Douglas Gilbert wrote:
> This post appeared recently on the linux kernel list.
> What is the maximum number of disks that can be attached
> in lk 2.5? Is it 256 (double the number of SCSI disk majors)
> or larger. I remember Richard Gooch's "sd-many" patch
> was rejected for 2.5 because it was said there was a
> better mechanism around the corner.

I'm also still waiting for the dev_t changes to port my 2.4 patches to
support many hard disks (2k) to 2.5.
Maybe the port should just be started now?
Any taker? (I'm a bit short on time, so it won't happen before mid Feb
if I have to do it myself.)

Regards,
-- 
Kurt Garloff                   <kurt@garloff.de>         [Eindhoven, NL]
Physics: Plasma simulations    <K.Garloff@TUE.NL>     [TU Eindhoven, NL]
Linux: SCSI, Security          <garloff@suse.de>    [SuSE Nuernberg, DE]
 (See mail header or public key servers for PGP2 and GPG public keys.)

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Fwd: 32bit dev_t

[Christoph Hellwig]

On Thu, Jan 23, 2003 at 07:31:31PM +0100, Kurt Garloff wrote:
> I'm also still waiting for the dev_t changes to port my 2.4 patches to
> support many hard disks (2k) to 2.5.
> Maybe the port should just be started now?
> Any taker? (I'm a bit short on time, so it won't happen before mid Feb
> if I have to do it myself.)

the Linux 2.5 sd and sr drivers are only limited by the number of majors
allocated to them.  There's no more architectural limit or any global
array.

Re: Fwd: 32bit dev_t

[Kurt Garloff]

[-- Attachment #1: Type: text/plain, Size: 845 bytes --]

Hi Christpoh,

On Thu, Jan 23, 2003 at 10:18:05PM +0000, Christoph Hellwig wrote:
> On Thu, Jan 23, 2003 at 07:31:31PM +0100, Kurt Garloff wrote:
> the Linux 2.5 sd and sr drivers are only limited by the number of majors
> allocated to them.  There's no more architectural limit or any global
> array.

In the 2.4 patches I allocate majors dynamically.[1]
I was expecting that that part of the patch would need to be rewritten
for 2.5. 
And of course sysfs, once it is complete (i.e. include char devices)
would replace the device reporting hack.

[1] http://www.suse.de/~garloff/linux/scsi-many/

Regards,
-- 
Kurt Garloff  <garloff@suse.de>                          Eindhoven, NL
GPG key: See mail header, key servers                 SuSE Labs (Head)
SuSE Linux AG, Nuernberg, DE                            SCSI, Security

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Fwd: 32bit dev_t

[Joel Becker]

On Thu, Jan 23, 2003 at 10:18:05PM +0000, Christoph Hellwig wrote:
> the Linux 2.5 sd and sr drivers are only limited by the number of majors
> allocated to them.  There's no more architectural limit or any global
> array.

	How are they allocated?  Is this dynamic, or compiled in?  In
addition, what is the maximum amount?  If it's less than 1000 disks,
it's too small for today.  If it's less than 2000 disks, it's probably
too small for 2.6's lifetime.

Joel


-- 

 "Practice random acts of kindness and senseless acts of beauty."

 Oh, and don't forget where your towel is.

Joel Becker
Senior Member of Technical Staff
Oracle Corporation
E-mail: joel.becker@oracle.com
Phone: (650) 506-8127

Re: Fwd: 32bit dev_t

[Christoph Hellwig]

On Fri, Jan 24, 2003 at 10:29:53AM -0800, Joel Becker wrote:
> 	How are they allocated?  Is this dynamic, or compiled in?  In
> addition, what is the maximum amount?  If it's less than 1000 disks,
> it's too small for today.  If it's less than 2000 disks, it's probably
> too small for 2.6's lifetime.

Currently it's compiled in, and the trivial change to make it use
register_blkdev with a 0 major argument until all globally unused majors
are gone is left to the reader..

Re: Fwd: 32bit dev_t

[Alan Cox]

On Mon, 2003-01-27 at 22:51, Christoph Hellwig wrote:
> Currently it's compiled in, and the trivial change to make it use
> register_blkdev with a 0 major argument until all globally unused majors
> are gone is left to the reader..

Which leaves the non trivial change of figuring out a usable security
model when doing that. I've yet to see one

Re: Fwd: 32bit dev_t

[Christoph Hellwig]

On Tue, Jan 28, 2003 at 11:21:07AM +0000, Alan Cox wrote:
> On Mon, 2003-01-27 at 22:51, Christoph Hellwig wrote:
> > Currently it's compiled in, and the trivial change to make it use
> > register_blkdev with a 0 major argument until all globally unused majors
> > are gone is left to the reader..
> 
> Which leaves the non trivial change of figuring out a usable security
> model when doing that. I've yet to see one

That's why I didn't implement it.  Ask the SuSE folks, they already ship
the dynamic majors for sd hack in their 2.4-based tree.

Re: Fwd: 32bit dev_t

[Kurt Garloff]

[-- Attachment #1: Type: text/plain, Size: 871 bytes --]

Hi,

On Tue, Jan 28, 2003 at 11:28:40AM +0000, Christoph Hellwig wrote:
> On Tue, Jan 28, 2003 at 11:21:07AM +0000, Alan Cox wrote:
> > Which leaves the non trivial change of figuring out a usable security
> > model when doing that. I've yet to see one
> 
> That's why I didn't implement it.  Ask the SuSE folks, they already ship
> the dynamic majors for sd hack in their 2.4-based tree.

You're supposed to use scsidev to create the device nodes beyond the
256 standard ones. It handles permissions. Based on CBTU addresses (or
optionally aliases that you may define using the serial no or WWID),
not the detection order.

Regards,
-- 
Kurt Garloff  <garloff@suse.de>                          Eindhoven, NL
GPG key: See mail header, key servers                 SuSE Labs (Head)
SuSE Linux AG, Nuernberg, DE                            SCSI, Security

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Fwd: 32bit dev_t

[Bryan Henderson]

This has probably been discussed to death somewhere, but I missed it:  Why
are people with hundreds of disks messing around with piles of major
numbers and eliminating disk partitioning instead of just using devfs?
Doesn't that make dev_t pretty much irrelevant?

Re: Fwd: 32bit dev_t

[Alan Cox]

On Tue, 2003-01-28 at 16:33, Bryan Henderson wrote:
> 
> 
> This has probably been discussed to death somewhere, but I missed it:  Why
> are people with hundreds of disks messing around with piles of major
> numbers and eliminating disk partitioning instead of just using devfs?
> Doesn't that make dev_t pretty much irrelevant?

Short answer: No

Long answer: SuS stat() and related calls require the device is identified, as does
NFS. Devfs also hasn't got good permission persistence and has other big problems
in 2.4 at least

New model for managing dev_t's for partitionable block devices

[Steven Dake]

I was thinking of an entirely new model for partitionable block devices. 
 Here is how it would work:

Each physical disk would be assigned a minor number in a group of 
majors.  So assume a major was chosen of 150, 151, 152, 153, there would 
be a total of 1024 physical disks that could be mapped.  Then the device 
mapper code could be used to provide partition devices in another 
major/group of majors.

The advantage of this technique is that instead of wasting tons of 
minors on partitions that are never used, partitions could be 
dynamically allocated out of the minor list, allowing for thousands of 
disks with varying numbers of partitions each.  Further instead of each 
block device (such as i2o, scsi, etc) having their own set of majors for 
each partitionable disk (which wastes dev_t address space) everything 
would be compressed into the same set of majors.

As an example, Lets assume we want 4096 total disks with 16384 total 
partitions (4 partitions per disk, where it is likely to be less):

That is:
4096 disks / 256 disks * 1 major = 16 majors
16384 partitions / 256 partitions * 1 major = 64 majors
total of 80 majors

To allow a similiar configuration in the current block device setup, 
with just the SCSI disk major,
4096 disks / 16 disks * 1 major = 256 majors

Now, assume we have 4096 disks available for i2o, scsi, compaq raid, etc 
etc, we are talking about lots of majors that go way beyond the current 
addressable 16 bytes.

The only downside is addressing the disks in hotswap (ie: how do you 
know what disk is where?)  This can be achieved through per-subsystem 
devfs mapping (ie: linking /dev/scsi/hostX/... to /dev/disc0) or 
userspace utilities that scan the disk devices (such as those that would 
be in /dev/disk) and determine which disks are what.

Thanks
-steve

Alan Cox wrote:

>On Mon, 2003-01-27 at 22:51, Christoph Hellwig wrote:
>  
>
>>Currently it's compiled in, and the trivial change to make it use
>>register_blkdev with a 0 major argument until all globally unused majors
>>are gone is left to the reader..
>>    
>>
>
>Which leaves the non trivial change of figuring out a usable security
>model when doing that. I've yet to see one
>
>
>-
>To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
>
>  
>

Re: New model for managing dev_t's for partitionable block devices

[Bryan Henderson]

>the device
>mapper code could be used to provide partition devices in another
>major/group of majors.

If I understand what you're saying, this has been discussed before.  I
don't know what the device mapper code is, but it's actually quite elegant
if it is a regular device driver that derives multiple logical disk drives
from a single physical one in the same way that the md device driver
derives a single logical disk drive from multiple physical ones.  The
layering is cleaner that way.

The last time I remember this discussed, it was as a solution to the
problem of a device driver presuming to access at initialization time a
partition map that didn't really exist.  I don't remember the details, but
this particular device wasn't ready to handle data reads until some time
after initialization.  You ought to be able to initialize a device that you
plan to use only as a raw device without Linux attempting to make
partitions on it.

As I recall, there weren't any fundamental objections to this.

>partitions could be dynamically allocated out of the minor list

Doesn't this exacerbate the Linux SCSI drive name binding problem?  It's
bad enough that when you remove your /dev/sda and reboot, your /dev/sdc
becomes /dev/sdb.  With this, it sounds like when you delete a partition on
/dev/sda, your partitions on /dev/sdb change names.

>As an example, Lets assume we want 4096 total disks with 16384 total
>partitions (4 partitions per disk, where it is likely to be less):

We should keep in mind that as a practical matter, someone with 4096
physical disks is unlikely to be partitioning at all.  Partitions are for
the poor person who has only a handful of physical disks and wants to
divide his data into more pieces than that.  Also note that if you have
4096 "physical" devices, they probably aren't very physical at all --
there's some subsystem on the other end of the SCSI link that carves
variable-size devices out of a pool of storage.  Hence, even less reason
for Linux to partition them.

Re: New model for managing dev_t's for partitionable block devices

[Steven Dake]

Bryan Henderson wrote:

>
>
>  
>
>>the device
>>mapper code could be used to provide partition devices in another
>>major/group of majors.
>>    
>>
>
>If I understand what you're saying, this has been discussed before.  I
>don't know what the device mapper code is, but it's actually quite elegant
>if it is a regular device driver that derives multiple logical disk drives
>from a single physical one in the same way that the md device driver
>derives a single logical disk drive from multiple physical ones.  The
>layering is cleaner that way.
>  
>
this is exactly how lvm works.

>The last time I remember this discussed, it was as a solution to the
>problem of a device driver presuming to access at initialization time a
>partition map that didn't really exist.  I don't remember the details, but
>this particular device wasn't ready to handle data reads until some time
>after initialization.  You ought to be able to initialize a device that you
>plan to use only as a raw device without Linux attempting to make
>partitions on it.
>
>As I recall, there weren't any fundamental objections to this.
>
>  
>
>>partitions could be dynamically allocated out of the minor list
>>    
>>
>
>Doesn't this exacerbate the Linux SCSI drive name binding problem?  It's
>bad enough that when you remove your /dev/sda and reboot, your /dev/sdc
>becomes /dev/sdb.  With this, it sounds like when you delete a partition on
>/dev/sda, your partitions on /dev/sdb change names.
>  
>
This is a problem with hotswap of course, and shouldn't be solved by the 
kernel putting the same device always in the same major/minor.  A 
userspace application should query the OS and build the device nodes 
based upon scsi serial number, FC port WWN, or access path 
(host/channel/id/lun).  The current "MAKEDEV" works fine for people with 
and ide disk and cdrom, but for real systems with lots of disks and 
hotswap capabilities, static naming just doesn't work (as you have 
said).  :)  Devfs solves the naming problem by using access path 
automatically within the OS.  Downside of this methodology is that 
access permissions are not persistent between reboots (which is one 
significant limitation of devfs).  There is a utility called scsidev 
which does the above of building device nodes based upon serial number 
instead of dumb /dev/sda.

>  
>
>>As an example, Lets assume we want 4096 total disks with 16384 total
>>partitions (4 partitions per disk, where it is likely to be less):
>>    
>>
>
>We should keep in mind that as a practical matter, someone with 4096
>physical disks is unlikely to be partitioning at all.  Partitions are for
>the poor person who has only a handful of physical disks and wants to
>divide his data into more pieces than that.  Also note that if you have
>4096 "physical" devices, they probably aren't very physical at all --
>there's some subsystem on the other end of the SCSI link that carves
>variable-size devices out of a pool of storage.  Hence, even less reason
>for Linux to partition them.
>
>
>  
>
I agree using partitions is unlikely with large amounts of disks. 
 Someone should be using LVM to manage those disks if they have a large 
amount.  Unfortunately even though no partitions are needed, 4096 disks 
still require 16 dev_t minors for each device.  This is a significant 
waste of space.  The user could hack their kernel to remove the 
partitions entirely, which someone has already designed a patch to do. 
 This isn't general purpose enough to be useable by the linux user. 
 What is needed is a compromise, described above, limiting the number of 
partitions to some sane amount, but allowing significantly more disks 
for the power user.

Thanks for your comments.
-steve

Re: New model for managing dev_t's for partitionable block devices

[Andries Brouwer]

On Wed, Jan 29, 2003 at 09:45:38AM -0700, Steven Dake wrote:

> What is needed is a compromise, described above, limiting the number of 
> partitions to some sane amount, but allowing significantly more disks 
> for the power user.

Always this discussion, over and over again.
Changing dev_t to be 32-bit is something many people that take part
in this discussion could do in a few hours. The subsequent kernel
audit takes a weekend.
Is there a reason everybody prefers to discuss ugly hacks?

Andries

Re: New model for managing dev_t's for partitionable block devices

[Kurt Garloff]

[-- Attachment #1: Type: text/plain, Size: 658 bytes --]

Hi Steven,

On Wed, Jan 29, 2003 at 09:45:38AM -0700, Steven Dake wrote:
> This is a problem with hotswap of course, and shouldn't be solved by the 
> kernel putting the same device always in the same major/minor.  A 
> userspace application should query the OS and build the device nodes 
> based upon scsi serial number, FC port WWN, or access path 
> (host/channel/id/lun).

scsidev can do exactly this for you.

Regards,
-- 
Kurt Garloff  <garloff@suse.de>                          Eindhoven, NL
GPG key: See mail header, key servers                 SuSE Labs (Head)
SuSE Linux AG, Nuernberg, DE                            SCSI, Security

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]