Number of devices that SCSI can support

Number of devices that SCSI can support

[Vinay Venkataraghavan]

Hello everybody,

Is there a limit on the number of devices that SCSI supports. In other words, I have a QLogic HBA card, and I am connecting to a SAN which has 64 targets. 

I also have one local SCSI disk.  So the total number of targets that I need to see is 65. 

So now what happens is that all but ONE disc/target gets correctly mapped and a device file is created except for one. This happens to be /dev/sdbm which maps to /dev/sg64

Looks like the SCSI midlayer also sees the target but the only thing that does not seem to happen is that the device file /dev/sdbm is not getting created.

I am attaching below the output: 

If you go all the way down, the last entry which is a valid disc gets mapped correctly but a /dev/sdbm device file does not get created. 

Any thoughts as to why this is happening. Is there a limitation on SCSI ability to handle only 64 discs? I would really appreciate any help.

I am running 2.6.9-42. 

I might be wrong, but I think i tried this same configuration on RHEL5 and I think it works. I will double check though. Was there a patch that was applied to fix this issue. 

Thanks in advance. 

Vinay


[root@localhost ~]# sg_map -x
/dev/sg0  0 0 0 0  0  /dev/sda
/dev/sg1  2 0 0 0  0  /dev/sdb
/dev/sg2  2 0 1 0  0  /dev/sdc
/dev/sg3  2 0 2 0  0  /dev/sdd
/dev/sg4  2 0 3 0  0  /dev/sde
/dev/sg5  2 0 4 0  0  /dev/sdf
/dev/sg6  2 0 5 0  0  /dev/sdg
/dev/sg7  2 0 6 0  0  /dev/sdh
/dev/sg8  2 0 7 0  0  /dev/sdi
/dev/sg9  2 0 8 0  0  /dev/sdj
/dev/sg10  2 0 9 0  0  /dev/sdk
/dev/sg11  2 0 10 0  0  /dev/sdl
/dev/sg12  2 0 11 0  0  /dev/sdm
/dev/sg13  2 0 12 0  0  /dev/sdn
/dev/sg14  2 0 13 0  0  /dev/sdo
/dev/sg15  2 0 14 0  0  /dev/sdp
/dev/sg16  2 0 15 0  0  /dev/sdq
/dev/sg17  2 0 16 0  0  /dev/sdr
/dev/sg18  2 0 17 0  0  /dev/sds
/dev/sg19  2 0 18 0  0  /dev/sdt
/dev/sg20  2 0 19 0  0  /dev/sdu
/dev/sg21  2 0 20 0  0  /dev/sdv
/dev/sg22  2 0 21 0  0  /dev/sdw
/dev/sg23  2 0 22 0  0  /dev/sdx
/dev/sg24  2 0 23 0  0  /dev/sdy
/dev/sg25  2 0 24 0  0  /dev/sdz
/dev/sg26  2 0 25 0  0  /dev/sdaa
/dev/sg27  2 0 26 0  0  /dev/sdab
/dev/sg28  2 0 27 0  0  /dev/sdac
/dev/sg29  2 0 28 0  0  /dev/sdad
/dev/sg30  2 0 29 0  0  /dev/sdae
/dev/sg31  2 0 30 0  0  /dev/sdaf
/dev/sg32  2 0 31 0  0  /dev/sdag
/dev/sg33  2 0 32 0  0  /dev/sdah
/dev/sg34  2 0 33 0  0  /dev/sdai
/dev/sg35  2 0 34 0  0  /dev/sdaj
/dev/sg36  2 0 35 0  0  /dev/sdak
/dev/sg37  2 0 36 0  0  /dev/sdal
/dev/sg38  2 0 37 0  0  /dev/sdam
/dev/sg39  2 0 38 0  0  /dev/sdan
/dev/sg40  2 0 39 0  0  /dev/sdao
/dev/sg41  2 0 40 0  0  /dev/sdap
/dev/sg42  2 0 41 0  0  /dev/sdaq
/dev/sg43  2 0 42 0  0  /dev/sdar
/dev/sg44  2 0 43 0  0  /dev/sdas
/dev/sg45  2 0 44 0  0  /dev/sdat
/dev/sg46  2 0 45 0  0  /dev/sdau
/dev/sg47  2 0 46 0  0  /dev/sdav
/dev/sg48  2 0 47 0  0  /dev/sdaw
/dev/sg49  2 0 48 0  0  /dev/sdax
/dev/sg50  2 0 49 0  0  /dev/sday
/dev/sg51  2 0 50 0  0  /dev/sdaz
/dev/sg52  2 0 51 0  0  /dev/sdba
/dev/sg53  2 0 52 0  0  /dev/sdbb
/dev/sg54  2 0 53 0  0  /dev/sdbc
/dev/sg55  2 0 54 0  0  /dev/sdbd
/dev/sg56  2 0 55 0  0  /dev/sdbe
/dev/sg57  2 0 56 0  0  /dev/sdbf
/dev/sg58  2 0 57 0  0  /dev/sdbg
/dev/sg59  2 0 58 0  0  /dev/sdbh
/dev/sg60  2 0 59 0  0  /dev/sdbi
/dev/sg61  2 0 60 0  0  /dev/sdbj
/dev/sg62  2 0 61 0  0  /dev/sdbk
/dev/sg63  2 0 62 0  0  /dev/sdbl
/dev/sg64  2 0 63 0  0









      ____________________________________________________________________________________
Never miss a thing.  Make Yahoo your home page. 
http://www.yahoo.com/r/hs

Re: Number of devices that SCSI can support

[Matthew Wilcox]

On Tue, Jan 08, 2008 at 04:55:46PM -0800, Vinay Venkataraghavan wrote:
> Is there a limit on the number of devices that SCSI supports. In other words, I have a QLogic HBA card, and I am connecting to a SAN which has 64 targets. 

I've personally had over five hundred LUNs.  You shouldn't be hitting a
limit here.

> I am running 2.6.9-42. 

That sounds like a Red Hat Enterprise kernel.  Perhaps you should
contact them for support?

-- 
Intel are signing my paycheques ... these opinions are still mine
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."

RE: Number of devices that SCSI can support

[Jansen, Frank]

You will want to check the BIOS settings on the Qlogic HBA, if you are running into a problem.  Also, there may be limitations that are inherent to the storage to which you are connecting.

Regards,

Frank


-----Original Message-----
From: linux-scsi-owner@vger.kernel.org on behalf of Matthew Wilcox
Sent: Tue 1/8/2008 9:22 PM
To: Vinay Venkataraghavan
Cc: linux-scsi@vger.kernel.org
Subject: Re: Number of devices that SCSI can support
 
On Tue, Jan 08, 2008 at 04:55:46PM -0800, Vinay Venkataraghavan wrote:
> Is there a limit on the number of devices that SCSI supports. In other words, I have a QLogic HBA card, and I am connecting to a SAN which has 64 targets. 

I've personally had over five hundred LUNs.  You shouldn't be hitting a
limit here.

> I am running 2.6.9-42. 

That sounds like a Red Hat Enterprise kernel.  Perhaps you should
contact them for support?

-- 
Intel are signing my paycheques ... these opinions are still mine
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."
-
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Number of devices that SCSI can support

[James Bottomley]

On Tue, 2008-01-08 at 19:22 -0700, Matthew Wilcox wrote:
> On Tue, Jan 08, 2008 at 04:55:46PM -0800, Vinay Venkataraghavan wrote:
> > Is there a limit on the number of devices that SCSI supports. In other words, I have a QLogic HBA card, and I am connecting to a SAN which has 64 targets. 
> 
> I've personally had over five hundred LUNs.  You shouldn't be hitting a
> limit here.

I believe the largest test that's been run was the old OSDL CGL
workgroup ... they went up to 4096.

However, LUN support depends on the driver and HBA parameters as well
(some choose to have arbitrary limits).

So, firstly, if the inquiry strings appear (as in you see a scsiX:X:X:64
and above in dmesg) then I'd look at udev issues.  If the inquiry
strings don't appear, it's probably a device or driver programmed limit.

James

Re: Number of devices that SCSI can support

[Matthew Wilcox]

On Wed, Jan 09, 2008 at 09:05:52AM -0600, James Bottomley wrote:
> On Tue, 2008-01-08 at 19:22 -0700, Matthew Wilcox wrote:
> > On Tue, Jan 08, 2008 at 04:55:46PM -0800, Vinay Venkataraghavan wrote:
> > > Is there a limit on the number of devices that SCSI supports. In other words, I have a QLogic HBA card, and I am connecting to a SAN which has 64 targets. 
> > 
> > I've personally had over five hundred LUNs.  You shouldn't be hitting a
> > limit here.
> 
> I believe the largest test that's been run was the old OSDL CGL
> workgroup ... they went up to 4096.
> 
> However, LUN support depends on the driver and HBA parameters as well
> (some choose to have arbitrary limits).

I was using a qlogic HBA for my tests, so I don't think this is the
problem -- although the original poster claims to have 64 targets, and I
had only one target with 128 luns (attached 4 times).

-- 
Intel are signing my paycheques ... these opinions are still mine
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."

Re: Number of devices that SCSI can support

[Andrew Vasquez]

On Wed, 09 Jan 2008, Matthew Wilcox wrote:

> On Wed, Jan 09, 2008 at 09:05:52AM -0600, James Bottomley wrote:
> > On Tue, 2008-01-08 at 19:22 -0700, Matthew Wilcox wrote:
> > > On Tue, Jan 08, 2008 at 04:55:46PM -0800, Vinay Venkataraghavan wrote:
> > > > Is there a limit on the number of devices that SCSI supports. In other words, I have a QLogic HBA card, and I am connecting to a SAN which has 64 targets. 
> > > 
> > > I've personally had over five hundred LUNs.  You shouldn't be hitting a
> > > limit here.
> > 
> > I believe the largest test that's been run was the old OSDL CGL
> > workgroup ... they went up to 4096.
> > 
> > However, LUN support depends on the driver and HBA parameters as well
> > (some choose to have arbitrary limits).
> 
> I was using a qlogic HBA for my tests, so I don't think this is the
> problem -- although the original poster claims to have 64 targets, and I
> had only one target with 128 luns (attached 4 times).
> 
> -- 
> Intel are signing my paycheques ... these opinions are still mine
> "Bill, look, we understand that you're interested in selling us this
> operating system, but compare it to ours.  We can't possibly take such
> a retrograde step."

Not sure what's going on as well, perhaps some logs could help... But
the inbox qla2xxx driver in RHEL4 set's an HBA's scsi_host->max_id
count to 512 (also verified with several test rings), so there
shouldn't be a problem handling 64 distinct targets (FC ports).

Re: Number of devices that SCSI can support

[Vinay Venkataraghavan]

Thank you all for responding. While I agree that the scsi stack in linux should be capable of supporting many targets, I still see this problem all the time. 
It seemed to consistently happen on /dev/sdbm which corresponds to the 65th lun or target.

My SAN configuration consists of: 4 targets with 16 luns each = 4 * 16 = 64 total devices.

But I ran another test by changing my targets information and saw this issue for /dev/sdbm 

As mentioned on this thread, I am attaching some more information. Looks like SCSI inquiry is going through, and the inquiry succeeds. As you can see from the attached message below the /sys/class/scsi_device file also gets created:
at /sys/class/scsi_device/1\:0\:3\:15/device/

and below this we find all the related entries such as:

block/          dump            queue_depth     state
delete          generic/        rescan          timeout
detach_state    model           rev             type
device_blocked  power/          scsi_level      vendor

So from this information it looks like the scsi mid layer had detected that particular lun. But the /dev/sdbm device has not been created. It was also mentioned that it could be a udev problem. 

The other curious issue is that if take the major number and the minor number from this entry and manually create the device file as such:

mknod /dev/sdbm 68 0 

it works after I rescan all the targets and luns and I am able to perform I/O on this lun. 

Any other ideas. 

Thanks,
Vinay

----- Original Message ----
From: Andrew Vasquez <andrew.vasquez@qlogic.com>
To: Vinay Venkataraghavan <raghavanvinay@yahoo.com>
Cc: James Bottomley <James.Bottomley@HansenPartnership.com>; linux-scsi@vger.kernel.org; Matthew Wilcox <matthew@wil.cx>
Sent: Wednesday, January 9, 2008 7:49:33 AM
Subject: Re: Number of devices that SCSI can support


On Wed, 09 Jan 2008, Matthew Wilcox wrote:

> On Wed, Jan 09, 2008 at 09:05:52AM -0600, James Bottomley wrote:
> > On Tue, 2008-01-08 at 19:22 -0700, Matthew Wilcox wrote:
> > > On Tue, Jan 08, 2008 at 04:55:46PM -0800, Vinay Venkataraghavan
 wrote:
> > > > Is there a limit on the number of devices that SCSI supports.
 In other words, I have a QLogic HBA card, and I am connecting to a SAN
 which has 64 targets. 
> > > 
> > > I've personally had over five hundred LUNs.  You shouldn't be
 hitting a
> > > limit here.
> > 
> > I believe the largest test that's been run was the old OSDL CGL
> > workgroup ... they went up to 4096.
> > 
> > However, LUN support depends on the driver and HBA parameters as
 well
> > (some choose to have arbitrary limits).
> 
> I was using a qlogic HBA for my tests, so I don't think this is the
> problem -- although the original poster claims to have 64 targets,
 and I
> had only one target with 128 luns (attached 4 times).
> 
> -- 
> Intel are signing my paycheques ... these opinions are still mine
> "Bill, look, we understand that you're interested in selling us this
> operating system, but compare it to ours.  We can't possibly take
 such
> a retrograde step."

Not sure what's going on as well, perhaps some logs could help... But
the inbox qla2xxx driver in RHEL4 set's an HBA's scsi_host->max_id
count to 512 (also verified with several test rings), so there
shouldn't be a problem handling 64 distinct targets (FC ports).
-
To unsubscribe from this list: send the line "unsubscribe linux-scsi"
 in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html





      ____________________________________________________________________________________
Never miss a thing.  Make Yahoo your home page. 
http://www.yahoo.com/r/hs

RE: Number of devices that SCSI can support

[Ellison, Bob]

I ran into this under RHEL4 and it turned out to be udev and the capi20
declaration. I commented the declaration out in 50-udev.rules (as well
as the capi/%n) and started seeing my sd node appear (i.e. device 68:0).
--
Bob

-----Original Message-----
From: linux-scsi-owner@vger.kernel.org
[mailto:linux-scsi-owner@vger.kernel.org] On Behalf Of Vinay
Venkataraghavan
Sent: Wednesday, January 09, 2008 5:31 PM
To: Andrew Vasquez
Cc: James Bottomley; linux-scsi@vger.kernel.org; Matthew Wilcox
Subject: Re: Number of devices that SCSI can support

Thank you all for responding. While I agree that the scsi stack in linux
should be capable of supporting many targets, I still see this problem
all the time. 
It seemed to consistently happen on /dev/sdbm which corresponds to the
65th lun or target.

My SAN configuration consists of: 4 targets with 16 luns each = 4 * 16 =
64 total devices.

But I ran another test by changing my targets information and saw this
issue for /dev/sdbm 

As mentioned on this thread, I am attaching some more information. Looks
like SCSI inquiry is going through, and the inquiry succeeds. As you can
see from the attached message below the /sys/class/scsi_device file also
gets created:
at /sys/class/scsi_device/1\:0\:3\:15/device/

and below this we find all the related entries such as:

block/          dump            queue_depth     state
delete          generic/        rescan          timeout
detach_state    model           rev             type
device_blocked  power/          scsi_level      vendor

So from this information it looks like the scsi mid layer had detected
that particular lun. But the /dev/sdbm device has not been created. It
was also mentioned that it could be a udev problem. 

The other curious issue is that if take the major number and the minor
number from this entry and manually create the device file as such:

mknod /dev/sdbm 68 0 

it works after I rescan all the targets and luns and I am able to
perform I/O on this lun. 

Any other ideas. 

Thanks,
Vinay

----- Original Message ----
From: Andrew Vasquez <andrew.vasquez@qlogic.com>
To: Vinay Venkataraghavan <raghavanvinay@yahoo.com>
Cc: James Bottomley <James.Bottomley@HansenPartnership.com>;
linux-scsi@vger.kernel.org; Matthew Wilcox <matthew@wil.cx>
Sent: Wednesday, January 9, 2008 7:49:33 AM
Subject: Re: Number of devices that SCSI can support


On Wed, 09 Jan 2008, Matthew Wilcox wrote:

> On Wed, Jan 09, 2008 at 09:05:52AM -0600, James Bottomley wrote:
> > On Tue, 2008-01-08 at 19:22 -0700, Matthew Wilcox wrote:
> > > On Tue, Jan 08, 2008 at 04:55:46PM -0800, Vinay Venkataraghavan
 wrote:
> > > > Is there a limit on the number of devices that SCSI supports.
 In other words, I have a QLogic HBA card, and I am connecting to a SAN
 which has 64 targets. 
> > > 
> > > I've personally had over five hundred LUNs.  You shouldn't be
 hitting a
> > > limit here.
> > 
> > I believe the largest test that's been run was the old OSDL CGL
> > workgroup ... they went up to 4096.
> > 
> > However, LUN support depends on the driver and HBA parameters as
 well
> > (some choose to have arbitrary limits).
> 
> I was using a qlogic HBA for my tests, so I don't think this is the
> problem -- although the original poster claims to have 64 targets,
 and I
> had only one target with 128 luns (attached 4 times).
> 
> -- 
> Intel are signing my paycheques ... these opinions are still mine
> "Bill, look, we understand that you're interested in selling us this
> operating system, but compare it to ours.  We can't possibly take
 such
> a retrograde step."

Not sure what's going on as well, perhaps some logs could help... But
the inbox qla2xxx driver in RHEL4 set's an HBA's scsi_host->max_id
count to 512 (also verified with several test rings), so there
shouldn't be a problem handling 64 distinct targets (FC ports).
-
To unsubscribe from this list: send the line "unsubscribe linux-scsi"
 in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html





 
________________________________________________________________________
____________
Never miss a thing.  Make Yahoo your home page. 
http://www.yahoo.com/r/hs

-
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Number of devices that SCSI can support

[berthiaume_wayne]

The capi bug is fixed in RHEL 4.6. 

-----Original Message-----
From: linux-scsi-owner@vger.kernel.org
[mailto:linux-scsi-owner@vger.kernel.org] On Behalf Of Ellison, Bob
Sent: Thursday, January 10, 2008 9:19 AM
To: Vinay Venkataraghavan; Andrew Vasquez
Cc: James Bottomley; linux-scsi@vger.kernel.org; Matthew Wilcox
Subject: RE: Number of devices that SCSI can support

I ran into this under RHEL4 and it turned out to be udev and the capi20
declaration. I commented the declaration out in 50-udev.rules (as well
as the capi/%n) and started seeing my sd node appear (i.e. device 68:0).
--
Bob

-----Original Message-----
From: linux-scsi-owner@vger.kernel.org
[mailto:linux-scsi-owner@vger.kernel.org] On Behalf Of Vinay
Venkataraghavan
Sent: Wednesday, January 09, 2008 5:31 PM
To: Andrew Vasquez
Cc: James Bottomley; linux-scsi@vger.kernel.org; Matthew Wilcox
Subject: Re: Number of devices that SCSI can support

Thank you all for responding. While I agree that the scsi stack in linux
should be capable of supporting many targets, I still see this problem
all the time. 
It seemed to consistently happen on /dev/sdbm which corresponds to the
65th lun or target.

My SAN configuration consists of: 4 targets with 16 luns each = 4 * 16 =
64 total devices.

But I ran another test by changing my targets information and saw this
issue for /dev/sdbm 

As mentioned on this thread, I am attaching some more information. Looks
like SCSI inquiry is going through, and the inquiry succeeds. As you can
see from the attached message below the /sys/class/scsi_device file also
gets created:
at /sys/class/scsi_device/1\:0\:3\:15/device/

and below this we find all the related entries such as:

block/          dump            queue_depth     state
delete          generic/        rescan          timeout
detach_state    model           rev             type
device_blocked  power/          scsi_level      vendor

So from this information it looks like the scsi mid layer had detected
that particular lun. But the /dev/sdbm device has not been created. It
was also mentioned that it could be a udev problem. 

The other curious issue is that if take the major number and the minor
number from this entry and manually create the device file as such:

mknod /dev/sdbm 68 0 

it works after I rescan all the targets and luns and I am able to
perform I/O on this lun. 

Any other ideas. 

Thanks,
Vinay

----- Original Message ----
From: Andrew Vasquez <andrew.vasquez@qlogic.com>
To: Vinay Venkataraghavan <raghavanvinay@yahoo.com>
Cc: James Bottomley <James.Bottomley@HansenPartnership.com>;
linux-scsi@vger.kernel.org; Matthew Wilcox <matthew@wil.cx>
Sent: Wednesday, January 9, 2008 7:49:33 AM
Subject: Re: Number of devices that SCSI can support


On Wed, 09 Jan 2008, Matthew Wilcox wrote:

> On Wed, Jan 09, 2008 at 09:05:52AM -0600, James Bottomley wrote:
> > On Tue, 2008-01-08 at 19:22 -0700, Matthew Wilcox wrote:
> > > On Tue, Jan 08, 2008 at 04:55:46PM -0800, Vinay Venkataraghavan
 wrote:
> > > > Is there a limit on the number of devices that SCSI supports.
 In other words, I have a QLogic HBA card, and I am connecting to a SAN
 which has 64 targets. 
> > > 
> > > I've personally had over five hundred LUNs.  You shouldn't be
 hitting a
> > > limit here.
> > 
> > I believe the largest test that's been run was the old OSDL CGL
> > workgroup ... they went up to 4096.
> > 
> > However, LUN support depends on the driver and HBA parameters as
 well
> > (some choose to have arbitrary limits).
> 
> I was using a qlogic HBA for my tests, so I don't think this is the
> problem -- although the original poster claims to have 64 targets,
 and I
> had only one target with 128 luns (attached 4 times).
> 
> -- 
> Intel are signing my paycheques ... these opinions are still mine
> "Bill, look, we understand that you're interested in selling us this
> operating system, but compare it to ours.  We can't possibly take
 such
> a retrograde step."

Not sure what's going on as well, perhaps some logs could help... But
the inbox qla2xxx driver in RHEL4 set's an HBA's scsi_host->max_id
count to 512 (also verified with several test rings), so there
shouldn't be a problem handling 64 distinct targets (FC ports).
-
To unsubscribe from this list: send the line "unsubscribe linux-scsi"
 in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html





 
________________________________________________________________________
____________
Never miss a thing.  Make Yahoo your home page. 
http://www.yahoo.com/r/hs

-
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
-
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html