mpt2sas: bug in disk ordering?

mpt2sas: bug in disk ordering?

[Caspar Smit]

Hi all,

I'm having a minor issue with the disk ordering using an LSI 9211-8i
(mpt2sas) controller.

If i go into the controllers SAS config utility during boot I can
identify the disks in the slots.

disk 0 -> slot 0, disk 1 -> slot 1, etc.. everyhting is fine.

When I boot into debian linux lenny (using the latest backports
2.6.32-31 kernel, this kernel has version 02.100.03.00 of the mpt2sas
driver)

The disks are ordered this way:

/dev/sda = internal SSD connected to the MB

/dev/sdb = slot 0
/dev/sdc = slot 1
/dev/sdd = slot 2
/dev/sde = slot 3
/dev/sdf = slot 7 (!)
/dev/sdg = slot 4
/dev/sdh = slot 5
/dev/sdi = slot 6

So it seems the first 4 disks connected to SAS port 1 are ordered
correctly and the last 4 disks are not.

When I check /dev/disk/by-path i see the following:

lrwxrwxrwx 1 root root   9 2011-05-13 10:52 pci-0000:04:00.0-sas-
0x500605b001d161b0:1:0-0x4433221103000000:0 -> ../../sdb
lrwxrwxrwx 1 root root   9 2011-05-13 10:52
pci-0000:04:00.0-sas-0x500605b001d161b0:1:1-0x4433221102000000:0 ->
../../sdc
lrwxrwxrwx 1 root root   9 2011-05-13 10:52
pci-0000:04:00.0-sas-0x500605b001d161b0:1:2-0x4433221101000000:0 ->
../../sdd
lrwxrwxrwx 1 root root   9 2011-05-13 10:52
pci-0000:04:00.0-sas-0x500605b001d161b0:1:3-0x4433221100000000:0 ->
../../sde
lrwxrwxrwx 1 root root   9 2011-05-13 10:52
pci-0000:04:00.0-sas-0x500605b001d161b0:1:4-0x4433221104000000:0 ->
../../sdf
lrwxrwxrwx 1 root root   9 2011-05-13 10:52
pci-0000:04:00.0-sas-0x500605b001d161b0:1:5-0x4433221107000000:0 ->
../../sdg
lrwxrwxrwx 1 root root   9 2011-05-13 10:52
pci-0000:04:00.0-sas-0x500605b001d161b0:1:6-0x4433221106000000:0 ->
../../sdh
lrwxrwxrwx 1 root root   9 2011-05-13 10:52
pci-0000:04:00.0-sas-0x500605b001d161b0:1:7-0x4433221105000000:0 ->
../../sdi

This LOOKS as it should but right now it is not really correct because
1:7 indicates that it should be physical slot 7 (sdi is in slot 6 and
not in slot 7)

When i use the sas2ircu command line util to LOCATE the disks and I
use 1:7 to locate, slot 7 lights up (NOT disk sdi)

Is this behavior fixed in a newer version of the driver, and if so
what is the corresponding patch to fix this?

Kind regards,

Caspar Smit
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: mpt2sas: bug in disk ordering?

[Caspar Smit]

Hello again,

I tried with Debian Squeeze with the latest backports kernel 2.6.38,
which has mpt2sas version 07.100.00.00
and now the disks are ordered this way:

/dev/sdb = slot 3
/dev/sdc = slot 2
/dev/sdd = slot 1
/dev/sde = slot 0
/dev/sdf = slot 7
/dev/sdg = slot 6
/dev/sdh = slot 5
/dev/sdi = slot 4

So there's more consistency here but still ordered backwards (per port)

Kind regards,

Caspar Smit


2011/5/13 Caspar Smit <c.smit@truebit.nl>:
> Hi all,
>
> I'm having a minor issue with the disk ordering using an LSI 9211-8i
> (mpt2sas) controller.
>
> If i go into the controllers SAS config utility during boot I can
> identify the disks in the slots.
>
> disk 0 -> slot 0, disk 1 -> slot 1, etc.. everyhting is fine.
>
> When I boot into debian linux lenny (using the latest backports
> 2.6.32-31 kernel, this kernel has version 02.100.03.00 of the mpt2sas
> driver)
>
> The disks are ordered this way:
>
> /dev/sda = internal SSD connected to the MB
>
> /dev/sdb = slot 0
> /dev/sdc = slot 1
> /dev/sdd = slot 2
> /dev/sde = slot 3
> /dev/sdf = slot 7 (!)
> /dev/sdg = slot 4
> /dev/sdh = slot 5
> /dev/sdi = slot 6
>
> So it seems the first 4 disks connected to SAS port 1 are ordered
> correctly and the last 4 disks are not.
>
> When I check /dev/disk/by-path i see the following:
>
> lrwxrwxrwx 1 root root   9 2011-05-13 10:52 pci-0000:04:00.0-sas-
> 0x500605b001d161b0:1:0-0x4433221103000000:0 -> ../../sdb
> lrwxrwxrwx 1 root root   9 2011-05-13 10:52
> pci-0000:04:00.0-sas-0x500605b001d161b0:1:1-0x4433221102000000:0 ->
> ../../sdc
> lrwxrwxrwx 1 root root   9 2011-05-13 10:52
> pci-0000:04:00.0-sas-0x500605b001d161b0:1:2-0x4433221101000000:0 ->
> ../../sdd
> lrwxrwxrwx 1 root root   9 2011-05-13 10:52
> pci-0000:04:00.0-sas-0x500605b001d161b0:1:3-0x4433221100000000:0 ->
> ../../sde
> lrwxrwxrwx 1 root root   9 2011-05-13 10:52
> pci-0000:04:00.0-sas-0x500605b001d161b0:1:4-0x4433221104000000:0 ->
> ../../sdf
> lrwxrwxrwx 1 root root   9 2011-05-13 10:52
> pci-0000:04:00.0-sas-0x500605b001d161b0:1:5-0x4433221107000000:0 ->
> ../../sdg
> lrwxrwxrwx 1 root root   9 2011-05-13 10:52
> pci-0000:04:00.0-sas-0x500605b001d161b0:1:6-0x4433221106000000:0 ->
> ../../sdh
> lrwxrwxrwx 1 root root   9 2011-05-13 10:52
> pci-0000:04:00.0-sas-0x500605b001d161b0:1:7-0x4433221105000000:0 ->
> ../../sdi
>
> This LOOKS as it should but right now it is not really correct because
> 1:7 indicates that it should be physical slot 7 (sdi is in slot 6 and
> not in slot 7)
>
> When i use the sas2ircu command line util to LOCATE the disks and I
> use 1:7 to locate, slot 7 lights up (NOT disk sdi)
>
> Is this behavior fixed in a newer version of the driver, and if so
> what is the corresponding patch to fix this?
>
> Kind regards,
>
> Caspar Smit
>
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: mpt2sas: bug in disk ordering?

[Arne Jansen]

On 13.05.2011 12:34, Caspar Smit wrote:
> I tried with Debian Squeeze with the latest backports kernel 2.6.38,
> which has mpt2sas version 07.100.00.00
> and now the disks are ordered this way:
> 
> /dev/sdb = slot 3
> /dev/sdc = slot 2
> /dev/sdd = slot 1
> /dev/sde = slot 0
> /dev/sdf = slot 7
> /dev/sdg = slot 6
> /dev/sdh = slot 5
> /dev/sdi = slot 4

I see something similar here, and, what's worse, the order may very from
boot to boot.

-Arne

Re: mpt2sas: bug in disk ordering?

[Caspar Smit]

That is something I really don't like.

After more checking I noticed the /dev/disk/by-path doesn't have any
links for the disks and their slots anymore.

Kind regards,

Caspar Smit
Systemengineer
True Bit Resources B.V.
Ampèrestraat 13E
1446 TP  Purmerend

T: +31(0)299 410 475
F: +31(0)299 410 476
@: c.smit@truebit.nl
W: www.truebit.nl




2011/5/13 Arne Jansen <lists@die-jansens.de>:
> On 13.05.2011 12:34, Caspar Smit wrote:
>> I tried with Debian Squeeze with the latest backports kernel 2.6.38,
>> which has mpt2sas version 07.100.00.00
>> and now the disks are ordered this way:
>>
>> /dev/sdb = slot 3
>> /dev/sdc = slot 2
>> /dev/sdd = slot 1
>> /dev/sde = slot 0
>> /dev/sdf = slot 7
>> /dev/sdg = slot 6
>> /dev/sdh = slot 5
>> /dev/sdi = slot 4
>
> I see something similar here, and, what's worse, the order may very from
> boot to boot.
>
> -Arne
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: mpt2sas: bug in disk ordering?

[Arne Jansen]

On 13.05.2011 12:41, Caspar Smit wrote:
> That is something I really don't like.
> 
> After more checking I noticed the /dev/disk/by-path doesn't have any
> links for the disks and their slots anymore.

> 2011/5/13 Arne Jansen <lists@die-jansens.de>:
>> On 13.05.2011 12:34, Caspar Smit wrote:
>>> I tried with Debian Squeeze with the latest backports kernel 2.6.38,
>>> which has mpt2sas version 07.100.00.00
>>> and now the disks are ordered this way:
>>>
>>> /dev/sdb = slot 3
>>> /dev/sdc = slot 2
>>> /dev/sdd = slot 1
>>> /dev/sde = slot 0
>>> /dev/sdf = slot 7
>>> /dev/sdg = slot 6
>>> /dev/sdh = slot 5
>>> /dev/sdi = slot 4
>>
>> I see something similar here, and, what's worse, the order may very from
>> boot to boot.


It might also be a problem of the expander, in that it reports the disk
in this order. Are you using an expander?

Re: mpt2sas: bug in disk ordering?

[Caspar Smit]

No,

I'm not using an expander.

The 9211-8i is connected to a Supermicro SAS113TQ rev 1.1 backplane
which has 8 ports.

Kind regards,

Caspar Smit

2011/5/13 Arne Jansen <lists@die-jansens.de>
>
> On 13.05.2011 12:41, Caspar Smit wrote:
> > That is something I really don't like.
> >
> > After more checking I noticed the /dev/disk/by-path doesn't have any
> > links for the disks and their slots anymore.
>
> > 2011/5/13 Arne Jansen <lists@die-jansens.de>:
> >> On 13.05.2011 12:34, Caspar Smit wrote:
> >>> I tried with Debian Squeeze with the latest backports kernel 2.6.38,
> >>> which has mpt2sas version 07.100.00.00
> >>> and now the disks are ordered this way:
> >>>
> >>> /dev/sdb = slot 3
> >>> /dev/sdc = slot 2
> >>> /dev/sdd = slot 1
> >>> /dev/sde = slot 0
> >>> /dev/sdf = slot 7
> >>> /dev/sdg = slot 6
> >>> /dev/sdh = slot 5
> >>> /dev/sdi = slot 4
> >>
> >> I see something similar here, and, what's worse, the order may very from
> >> boot to boot.
>
>
> It might also be a problem of the expander, in that it reports the disk
> in this order. Are you using an expander?
>

RE: mpt2sas: bug in disk ordering?

[Desai, Kashyap]

Smit,

Disk ordering is not done by mpt2sas driver.
Ordering is purely depend upon How FW discovers drivers and repots it to Driver.

e.a if you have slow spinning driver at slot-0, it can be reported by FW at the end, though it is attached to slot-0.

In summary, Disk ordering is not guaranteed.

~ Kashyap

> -----Original Message-----
> From: linux-scsi-owner@vger.kernel.org [mailto:linux-scsi-
> owner@vger.kernel.org] On Behalf Of Caspar Smit
> Sent: Friday, May 13, 2011 4:04 PM
> To: linux-scsi@vger.kernel.org
> Subject: Re: mpt2sas: bug in disk ordering?
> 
> Hello again,
> 
> I tried with Debian Squeeze with the latest backports kernel 2.6.38,
> which has mpt2sas version 07.100.00.00
> and now the disks are ordered this way:
> 
> /dev/sdb = slot 3
> /dev/sdc = slot 2
> /dev/sdd = slot 1
> /dev/sde = slot 0
> /dev/sdf = slot 7
> /dev/sdg = slot 6
> /dev/sdh = slot 5
> /dev/sdi = slot 4
> 
> So there's more consistency here but still ordered backwards (per port)
> 
> Kind regards,
> 
> Caspar Smit
> 
> 
> 2011/5/13 Caspar Smit <c.smit@truebit.nl>:
> > Hi all,
> >
> > I'm having a minor issue with the disk ordering using an LSI 9211-8i
> > (mpt2sas) controller.
> >
> > If i go into the controllers SAS config utility during boot I can
> > identify the disks in the slots.
> >
> > disk 0 -> slot 0, disk 1 -> slot 1, etc.. everyhting is fine.
> >
> > When I boot into debian linux lenny (using the latest backports
> > 2.6.32-31 kernel, this kernel has version 02.100.03.00 of the mpt2sas
> > driver)
> >
> > The disks are ordered this way:
> >
> > /dev/sda = internal SSD connected to the MB
> >
> > /dev/sdb = slot 0
> > /dev/sdc = slot 1
> > /dev/sdd = slot 2
> > /dev/sde = slot 3
> > /dev/sdf = slot 7 (!)
> > /dev/sdg = slot 4
> > /dev/sdh = slot 5
> > /dev/sdi = slot 6
> >
> > So it seems the first 4 disks connected to SAS port 1 are ordered
> > correctly and the last 4 disks are not.
> >
> > When I check /dev/disk/by-path i see the following:
> >
> > lrwxrwxrwx 1 root root   9 2011-05-13 10:52 pci-0000:04:00.0-sas-
> > 0x500605b001d161b0:1:0-0x4433221103000000:0 -> ../../sdb
> > lrwxrwxrwx 1 root root   9 2011-05-13 10:52
> > pci-0000:04:00.0-sas-0x500605b001d161b0:1:1-0x4433221102000000:0 ->
> > ../../sdc
> > lrwxrwxrwx 1 root root   9 2011-05-13 10:52
> > pci-0000:04:00.0-sas-0x500605b001d161b0:1:2-0x4433221101000000:0 ->
> > ../../sdd
> > lrwxrwxrwx 1 root root   9 2011-05-13 10:52
> > pci-0000:04:00.0-sas-0x500605b001d161b0:1:3-0x4433221100000000:0 ->
> > ../../sde
> > lrwxrwxrwx 1 root root   9 2011-05-13 10:52
> > pci-0000:04:00.0-sas-0x500605b001d161b0:1:4-0x4433221104000000:0 ->
> > ../../sdf
> > lrwxrwxrwx 1 root root   9 2011-05-13 10:52
> > pci-0000:04:00.0-sas-0x500605b001d161b0:1:5-0x4433221107000000:0 ->
> > ../../sdg
> > lrwxrwxrwx 1 root root   9 2011-05-13 10:52
> > pci-0000:04:00.0-sas-0x500605b001d161b0:1:6-0x4433221106000000:0 ->
> > ../../sdh
> > lrwxrwxrwx 1 root root   9 2011-05-13 10:52
> > pci-0000:04:00.0-sas-0x500605b001d161b0:1:7-0x4433221105000000:0 ->
> > ../../sdi
> >
> > This LOOKS as it should but right now it is not really correct
> because
> > 1:7 indicates that it should be physical slot 7 (sdi is in slot 6 and
> > not in slot 7)
> >
> > When i use the sas2ircu command line util to LOCATE the disks and I
> > use 1:7 to locate, slot 7 lights up (NOT disk sdi)
> >
> > Is this behavior fixed in a newer version of the driver, and if so
> > what is the corresponding patch to fix this?
> >
> > Kind regards,
> >
> > Caspar Smit
> >
> --
> To unsubscribe from this list: send the line "unsubscribe linux-scsi"
> in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: mpt2sas: bug in disk ordering?

[Caspar Smit]

Kashyap,

But when i use a LSI 3081E-R HBA (mptsas driver) the disk ordering is
ALWAYS the same, and always in correct order.
I would like to keep that functionality when upgrading to the newer 9211-8i HBA.

Kind regards,

Caspar Smit

2011/5/13 Desai, Kashyap <Kashyap.Desai@lsi.com>:
> Smit,
>
> Disk ordering is not done by mpt2sas driver.
> Ordering is purely depend upon How FW discovers drivers and repots it to Driver.
>
> e.a if you have slow spinning driver at slot-0, it can be reported by FW at the end, though it is attached to slot-0.
>
> In summary, Disk ordering is not guaranteed.
>
> ~ Kashyap
>
>> -----Original Message-----
>> From: linux-scsi-owner@vger.kernel.org [mailto:linux-scsi-
>> owner@vger.kernel.org] On Behalf Of Caspar Smit
>> Sent: Friday, May 13, 2011 4:04 PM
>> To: linux-scsi@vger.kernel.org
>> Subject: Re: mpt2sas: bug in disk ordering?
>>
>> Hello again,
>>
>> I tried with Debian Squeeze with the latest backports kernel 2.6.38,
>> which has mpt2sas version 07.100.00.00
>> and now the disks are ordered this way:
>>
>> /dev/sdb = slot 3
>> /dev/sdc = slot 2
>> /dev/sdd = slot 1
>> /dev/sde = slot 0
>> /dev/sdf = slot 7
>> /dev/sdg = slot 6
>> /dev/sdh = slot 5
>> /dev/sdi = slot 4
>>
>> So there's more consistency here but still ordered backwards (per port)
>>
>> Kind regards,
>>
>> Caspar Smit
>>
>>
>> 2011/5/13 Caspar Smit <c.smit@truebit.nl>:
>> > Hi all,
>> >
>> > I'm having a minor issue with the disk ordering using an LSI 9211-8i
>> > (mpt2sas) controller.
>> >
>> > If i go into the controllers SAS config utility during boot I can
>> > identify the disks in the slots.
>> >
>> > disk 0 -> slot 0, disk 1 -> slot 1, etc.. everyhting is fine.
>> >
>> > When I boot into debian linux lenny (using the latest backports
>> > 2.6.32-31 kernel, this kernel has version 02.100.03.00 of the mpt2sas
>> > driver)
>> >
>> > The disks are ordered this way:
>> >
>> > /dev/sda = internal SSD connected to the MB
>> >
>> > /dev/sdb = slot 0
>> > /dev/sdc = slot 1
>> > /dev/sdd = slot 2
>> > /dev/sde = slot 3
>> > /dev/sdf = slot 7 (!)
>> > /dev/sdg = slot 4
>> > /dev/sdh = slot 5
>> > /dev/sdi = slot 6
>> >
>> > So it seems the first 4 disks connected to SAS port 1 are ordered
>> > correctly and the last 4 disks are not.
>> >
>> > When I check /dev/disk/by-path i see the following:
>> >
>> > lrwxrwxrwx 1 root root   9 2011-05-13 10:52 pci-0000:04:00.0-sas-
>> > 0x500605b001d161b0:1:0-0x4433221103000000:0 -> ../../sdb
>> > lrwxrwxrwx 1 root root   9 2011-05-13 10:52
>> > pci-0000:04:00.0-sas-0x500605b001d161b0:1:1-0x4433221102000000:0 ->
>> > ../../sdc
>> > lrwxrwxrwx 1 root root   9 2011-05-13 10:52
>> > pci-0000:04:00.0-sas-0x500605b001d161b0:1:2-0x4433221101000000:0 ->
>> > ../../sdd
>> > lrwxrwxrwx 1 root root   9 2011-05-13 10:52
>> > pci-0000:04:00.0-sas-0x500605b001d161b0:1:3-0x4433221100000000:0 ->
>> > ../../sde
>> > lrwxrwxrwx 1 root root   9 2011-05-13 10:52
>> > pci-0000:04:00.0-sas-0x500605b001d161b0:1:4-0x4433221104000000:0 ->
>> > ../../sdf
>> > lrwxrwxrwx 1 root root   9 2011-05-13 10:52
>> > pci-0000:04:00.0-sas-0x500605b001d161b0:1:5-0x4433221107000000:0 ->
>> > ../../sdg
>> > lrwxrwxrwx 1 root root   9 2011-05-13 10:52
>> > pci-0000:04:00.0-sas-0x500605b001d161b0:1:6-0x4433221106000000:0 ->
>> > ../../sdh
>> > lrwxrwxrwx 1 root root   9 2011-05-13 10:52
>> > pci-0000:04:00.0-sas-0x500605b001d161b0:1:7-0x4433221105000000:0 ->
>> > ../../sdi
>> >
>> > This LOOKS as it should but right now it is not really correct
>> because
>> > 1:7 indicates that it should be physical slot 7 (sdi is in slot 6 and
>> > not in slot 7)
>> >
>> > When i use the sas2ircu command line util to LOCATE the disks and I
>> > use 1:7 to locate, slot 7 lights up (NOT disk sdi)
>> >
>> > Is this behavior fixed in a newer version of the driver, and if so
>> > what is the corresponding patch to fix this?
>> >
>> > Kind regards,
>> >
>> > Caspar Smit
>> >
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-scsi"
>> in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: mpt2sas: bug in disk ordering?

[Billy Crook]

On Fri, May 13, 2011 at 06:46, Caspar Smit <c.smit@truebit.nl> wrote:
> But when i use a LSI 3081E-R HBA (mptsas driver) the disk ordering is
> ALWAYS the same, and always in correct order.
> I would like to keep that functionality when upgrading to the newer 9211-8i HBA.

That was an unintentional coincidence. Consider yourself lucky nothing
was destroyed during the time you assumed it was deterministic.  Disk
ordering is essentially random in Linux, and will continue to be,
indefinitely.  If being able to refer to specific disks is important,
use the links in /dev/disk/ or assign the disks LABELs and use those
or UUIDs.  If /dev/disk is gone or not working, that is the problem
you need to solve, NOT disk startup order.

Re: mpt2sas: bug in disk ordering?

[Caspar Smit]

Kashyap,

If the mpt2sas driver doesn't order the disks, how do you explain the
fact that the same HBA, with the same Firmware orders the disks
differently on two different driver versions? The firmware doens't
seem to have any effect on the drive ordering.

And how am i supposed to address and locate the disks in the chassis,
how can i be absolutely sure i'm changing the right (failed) disk when
something goes wrong in a linux raid set?

Why was the /dev/disk/by-path removed?

I am used to the fact that all disks in a chassis are ordered in linux
according to the slot numbers on the chassis, this is to ensure a
reliable RAID setup so that /dev/sdb for instance is always in a
certain slot in the chassis. So far I only used lsi 3081e-r
controllers and these always gave me solid and reliable disk
numbering. If this goes random, how am i supposed to keep track which
disk is where in the chassis?

Kind regards,

Caspar Smit



2011/5/13 Desai, Kashyap <Kashyap.Desai@lsi.com>:
> Smit,
>
> Disk ordering is not done by mpt2sas driver.
> Ordering is purely depend upon How FW discovers drivers and repots it to Driver.
>
> e.a if you have slow spinning driver at slot-0, it can be reported by FW at the end, though it is attached to slot-0.
>
> In summary, Disk ordering is not guaranteed.
>
> ~ Kashyap
>
>> -----Original Message-----
>> From: linux-scsi-owner@vger.kernel.org [mailto:linux-scsi-
>> owner@vger.kernel.org] On Behalf Of Caspar Smit
>> Sent: Friday, May 13, 2011 4:04 PM
>> To: linux-scsi@vger.kernel.org
>> Subject: Re: mpt2sas: bug in disk ordering?
>>
>> Hello again,
>>
>> I tried with Debian Squeeze with the latest backports kernel 2.6.38,
>> which has mpt2sas version 07.100.00.00
>> and now the disks are ordered this way:
>>
>> /dev/sdb = slot 3
>> /dev/sdc = slot 2
>> /dev/sdd = slot 1
>> /dev/sde = slot 0
>> /dev/sdf = slot 7
>> /dev/sdg = slot 6
>> /dev/sdh = slot 5
>> /dev/sdi = slot 4
>>
>> So there's more consistency here but still ordered backwards (per port)
>>
>> Kind regards,
>>
>> Caspar Smit
>>
>>
>> 2011/5/13 Caspar Smit <c.smit@truebit.nl>:
>> > Hi all,
>> >
>> > I'm having a minor issue with the disk ordering using an LSI 9211-8i
>> > (mpt2sas) controller.
>> >
>> > If i go into the controllers SAS config utility during boot I can
>> > identify the disks in the slots.
>> >
>> > disk 0 -> slot 0, disk 1 -> slot 1, etc.. everyhting is fine.
>> >
>> > When I boot into debian linux lenny (using the latest backports
>> > 2.6.32-31 kernel, this kernel has version 02.100.03.00 of the mpt2sas
>> > driver)
>> >
>> > The disks are ordered this way:
>> >
>> > /dev/sda = internal SSD connected to the MB
>> >
>> > /dev/sdb = slot 0
>> > /dev/sdc = slot 1
>> > /dev/sdd = slot 2
>> > /dev/sde = slot 3
>> > /dev/sdf = slot 7 (!)
>> > /dev/sdg = slot 4
>> > /dev/sdh = slot 5
>> > /dev/sdi = slot 6
>> >
>> > So it seems the first 4 disks connected to SAS port 1 are ordered
>> > correctly and the last 4 disks are not.
>> >
>> > When I check /dev/disk/by-path i see the following:
>> >
>> > lrwxrwxrwx 1 root root   9 2011-05-13 10:52 pci-0000:04:00.0-sas-
>> > 0x500605b001d161b0:1:0-0x4433221103000000:0 -> ../../sdb
>> > lrwxrwxrwx 1 root root   9 2011-05-13 10:52
>> > pci-0000:04:00.0-sas-0x500605b001d161b0:1:1-0x4433221102000000:0 ->
>> > ../../sdc
>> > lrwxrwxrwx 1 root root   9 2011-05-13 10:52
>> > pci-0000:04:00.0-sas-0x500605b001d161b0:1:2-0x4433221101000000:0 ->
>> > ../../sdd
>> > lrwxrwxrwx 1 root root   9 2011-05-13 10:52
>> > pci-0000:04:00.0-sas-0x500605b001d161b0:1:3-0x4433221100000000:0 ->
>> > ../../sde
>> > lrwxrwxrwx 1 root root   9 2011-05-13 10:52
>> > pci-0000:04:00.0-sas-0x500605b001d161b0:1:4-0x4433221104000000:0 ->
>> > ../../sdf
>> > lrwxrwxrwx 1 root root   9 2011-05-13 10:52
>> > pci-0000:04:00.0-sas-0x500605b001d161b0:1:5-0x4433221107000000:0 ->
>> > ../../sdg
>> > lrwxrwxrwx 1 root root   9 2011-05-13 10:52
>> > pci-0000:04:00.0-sas-0x500605b001d161b0:1:6-0x4433221106000000:0 ->
>> > ../../sdh
>> > lrwxrwxrwx 1 root root   9 2011-05-13 10:52
>> > pci-0000:04:00.0-sas-0x500605b001d161b0:1:7-0x4433221105000000:0 ->
>> > ../../sdi
>> >
>> > This LOOKS as it should but right now it is not really correct
>> because
>> > 1:7 indicates that it should be physical slot 7 (sdi is in slot 6 and
>> > not in slot 7)
>> >
>> > When i use the sas2ircu command line util to LOCATE the disks and I
>> > use 1:7 to locate, slot 7 lights up (NOT disk sdi)
>> >
>> > Is this behavior fixed in a newer version of the driver, and if so
>> > what is the corresponding patch to fix this?
>> >
>> > Kind regards,
>> >
>> > Caspar Smit
>> >
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-scsi"
>> in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: mpt2sas: bug in disk ordering?

[Mikael Abrahamsson]

On Fri, 13 May 2011, Caspar Smit wrote:

> I am used to the fact that all disks in a chassis are ordered in linux 
> according to the slot numbers on the chassis, this is to ensure a 
> reliable RAID setup so that /dev/sdb for instance is always in a certain 
> slot in the chassis. So far I only used lsi 3081e-r controllers and 
> these always gave me solid and reliable disk numbering. If this goes 
> random, how am i supposed to keep track which disk is where in the 
> chassis?

This has been discussed extensively before. There is no way to do it the 
way you want to. Personally, I label the serial number on the front of the 
drive cage and use that for tracking (together with smartctl).

Drive naming is not deterministic in linux as said before, and the quicker 
you accept this try to find other ways you're comfortable with that solves 
your operational problem, the less pain you will be in.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: mpt2sas: bug in disk ordering?

[James Bottomley]

On Fri, 2011-05-13 at 14:09 +0200, Mikael Abrahamsson wrote:
> On Fri, 13 May 2011, Caspar Smit wrote:
> 
> > I am used to the fact that all disks in a chassis are ordered in linux 
> > according to the slot numbers on the chassis, this is to ensure a 
> > reliable RAID setup so that /dev/sdb for instance is always in a certain 
> > slot in the chassis. So far I only used lsi 3081e-r controllers and 
> > these always gave me solid and reliable disk numbering. If this goes 
> > random, how am i supposed to keep track which disk is where in the 
> > chassis?
> 
> This has been discussed extensively before. There is no way to do it the 
> way you want to.

Actually, there is, we just don't do it as a by-X rule in udev (although
nothing prevents you from adding a custom rule).  If you want to
identify disks by SAS slot, you're basically asking what phy they're
connected to (and if you have expanders and phys, the unique identifier
is expander:[expander ...]:phy.

when udev gets the path to a SAS disk, it looks something like this:

/sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/host1/port-1:0/expander-1:0/port-1:0:9/end_device-1:0:9/target1:0:1/1:0:1:0/scsi_device/1:0:1:0

So from this you can see I'm on expander 1:0 and port 1:0:9.  The port
number is still dynamic, but from the port you can get to the phys:

ls /sys/class/sas_port/port-1\:0\:9/device/
end_device-1:0:9  phy-1:0:9  power  sas_port  uevent

That tells me port 1:0:9 is indeed phy 1:0:9

so 9 is the phy number.  Expanders are also dynamically numbered, so if
you're behind an expander, you'll have to use something like the
expander serial number as the first part followed by the phy number.  Or
just use enclosure services; most expanders have them.


James