EMC Cx300 multipath seems to work

EMC Cx300 multipath seems to work

[Nicola Murino]

[-- Attachment #1.1: Type: text/plain, Size: 7825 bytes --]

Hi,

multipath seems to work on gentoo, I have to load the modules in proper
order, this is my configuration:

cat /etc/conf.d/local.start
# /etc/conf.d/local.start

# This is a good place to load any misc programs
# on startup ( use 1>&2 to hide output)
/sbin/modprobe dm-emc
/sbin/modprobe qla2300
/bin/sleep 5
/sbin/udevstart
/etc/init.d/multipathd start

is critical the command udevstart without only one path is seen,

I experienced a kernel oops with kernel-2.6.12 (both vanilla-sources and
gentoo-sources) when I load qla2300 module, here are the logs:

Unable to handle kernel NULL pointer dereference at virtual address 00000000
 printing eip:
f8f262c8
*pde = 00000000
Oops: 0000 [#1]
SMP
Modules linked in: dm_round_robin qla2300 qla2xxx dm_emc dm_multipath dm_mod
CPU:    1
EIP:    0060:[<f8f262c8>]    Not tainted VLI
EFLAGS: 00010086   (2.6.12-gentoo-r10)
EIP is at rr_select_path+0x8/0x50 [dm_round_robin]
eax: f30311cc   ebx: 00000000   ecx: f7f4d08c   edx: f30311cc
esi: f46f1580   edi: 00000001   ebp: f46f1580   esp: f75f1ee0
ds: 007b   es: 007b   ss: 0068
Process kmpathd/1 (pid: 5836, threadinfo=f75f0000 task=c2171530)
Stack: f30311c0 f8f12bf6 f30311cc f46f15b8 f46f1580 f7f4d080 f8f12cd9 f46f1580
       f30311c0 f46f1580 f7f4d080 00000000 f46f158c f8f12fd9 f46f1580 00000001
       c2171530 00000246 00000000 f46f1590 f46f15c8 f7f4d080 f46f15cc f46f1580
Call Trace:
 [<f8f12bf6>] __choose_path_in_pg+0x26/0x60 [dm_multipath]
 [<f8f12cd9>] __choose_pgpath+0xa9/0xc0 [dm_multipath]
 [<f8f12fd9>] process_queued_ios+0xd9/0x100 [dm_multipath]
 [<c012b229>] worker_thread+0x1b9/0x260
 [<f8f12f00>] process_queued_ios+0x0/0x100 [dm_multipath]
 [<c01163c0>] default_wake_function+0x0/0x20
 [<c01163c0>] default_wake_function+0x0/0x20
 [<c012b070>] worker_thread+0x0/0x260
 [<c012f85a>] kthread+0xba/0xc0
 [<c012f7a0>] kthread+0x0/0xc0
 [<c0101015>] kernel_thread_helper+0x5/0x10
Code: 40 08 8b 10 8b 48 04 89 11 89 4a 04 8b 13 89 42 04 89 10 89 58 04 89 03 31 c0 5b c3 8d b4 26 00 00 00 00 53 8b 44 24 08 8b 58 04 <8b> 03 39 d8 74 32 89 c1 8b 50 04 8b 00 85 c9 89 50 04 89 02 8b
 <4>device-mapper: dm-emc: long trespass command will be send
device-mapper: dm-emc: honor reservation bit will not be set (default)
Unable to handle kernel NULL pointer dereference at virtual address 00000000
 printing eip:
f8f262c8
*pde = 00000000
Oops: 0000 [#2]
SMP
Modules linked in: dm_round_robin qla2300 qla2xxx dm_emc dm_multipath dm_mod
CPU:    0
EIP:    0060:[<f8f262c8>]    Not tainted VLI
EFLAGS: 00010086   (2.6.12-gentoo-r10)
EIP is at rr_select_path+0x8/0x50 [dm_round_robin]
eax: f748c8cc   ebx: 00000000   ecx: f7f4d00c   edx: f748c8cc
esi: f46f1680   edi: 00000001   ebp: f46f1680   esp: f7517ee0
ds: 007b   es: 007b   ss: 0068
Process kmpathd/0 (pid: 5835, threadinfo=f7516000 task=f7f0e530)
Stack: f748c8c0 f8f12bf6 f748c8cc f46f16b8 f46f1680 f7f4d000 f8f12cd9 f46f1680
       f748c8c0 f46f1680 f7f4d000 00000000 f46f168c f8f12fd9 f46f1680 c200c558
       f7f0e530 00000246 00000000 f46f1690 f46f16c8 f7f4d000 f46f16cc f46f1680
Call Trace:
 [<f8f12bf6>] __choose_path_in_pg+0x26/0x60 [dm_multipath]
 [<f8f12cd9>] __choose_pgpath+0xa9/0xc0 [dm_multipath]
 [<f8f12fd9>] process_queued_ios+0xd9/0x100 [dm_multipath]
 [<c012b229>] worker_thread+0x1b9/0x260
 [<f8f12f00>] process_queued_ios+0x0/0x100 [dm_multipath]
 [<c01163c0>] default_wake_function+0x0/0x20
 [<c01163c0>] default_wake_function+0x0/0x20
 [<c012b070>] worker_thread+0x0/0x260
 [<c012f85a>] kthread+0xba/0xc0
 [<c012f7a0>] kthread+0x0/0xc0
 [<c0101015>] kernel_thread_helper+0x5/0x10
Code: 40 08 8b 10 8b 48 04 89 11 89 4a 04 8b 13 89 42 04 89 10 89 58 04 89 03 31 c0 5b c3 8d b4 26 00 00 00 00 53 8b 44 24 08 8b 58 04 <8b> 03 39 d8 74 32 89 c1 8b 50 04 8b 00 85 c9 89 50 04 89 02 8b
 <4>device-mapper: dm-emc: long trespass command will be send
device-mapper: dm-emc: honor reservation bit will not be set (default)
Unable to handle kernel NULL pointer dereference at virtual address 00000000
 printing eip:
f8f262c8
*pde = 00000000
Oops: 0000 [#3]
SMP
Modules linked in: dm_round_robin qla2300 qla2xxx dm_emc dm_multipath dm_mod
CPU:    2
EIP:    0060:[<f8f262c8>]    Not tainted VLI
EFLAGS: 00010086   (2.6.12-gentoo-r10)
EIP is at rr_select_path+0x8/0x50 [dm_round_robin]
eax: f7e4850c   ebx: 00000000   ecx: f7f4d10c   edx: f7e4850c
esi: f46e4880   edi: 00000001   ebp: f46e4880   esp: f777dee0
ds: 007b   es: 007b   ss: 0068
Process kmpathd/2 (pid: 5837, threadinfo=f777c000 task=f7d11a40)
Stack: f7e48500 f8f12bf6 f7e4850c f46e48b8 f46e4880 f7f4d100 f8f12cd9 f46e4880
       f7e48500 f46e4880 f7f4d100 00000000 f46e488c f8f12fd9 f46e4880 c201c9d0
       f7d11a40 00000246 00000000 f46e4890 f46e48c8 f7f4d100 f46e48cc f46e4880
Call Trace:
 [<f8f12bf6>] __choose_path_in_pg+0x26/0x60 [dm_multipath]
 [<f8f12cd9>] __choose_pgpath+0xa9/0xc0 [dm_multipath]
 [<f8f12fd9>] process_queued_ios+0xd9/0x100 [dm_multipath]
 [<c012b229>] worker_thread+0x1b9/0x260
 [<f8f12f00>] process_queued_ios+0x0/0x100 [dm_multipath]
 [<c01163c0>] default_wake_function+0x0/0x20
 [<c01163c0>] default_wake_function+0x0/0x20
 [<c012b070>] worker_thread+0x0/0x260
 [<c012f85a>] kthread+0xba/0xc0
 [<c012f7a0>] kthread+0x0/0xc0
 [<c0101015>] kernel_thread_helper+0x5/0x10
Code: 40 08 8b 10 8b 48 04 89 11 89 4a 04 8b 13 89 42 04 89 10 89 58 04 89 03 31 c0 5b c3 8d b4 26 00 00 00 00 53 8b 44 24 08 8b 58 04 <8b> 03 39 d8 74 32 89 c1 8b 50 04 8b 00 85 c9 89 50 04 89 02 8b
 <4>device-mapper: dm-emc: long trespass command will be send

however kernel-2.6.13 works fine (both vanilla and gentoo-sources)

here are my multipath configuration

grep -v "#" /etc/multipath.conf
defaults {
        multipath_tool  "/sbin/multipath -v0"
        udev_dir        /dev
        polling_interval 10
        default_selector        "round-robin 0"
        default_getuid_callout  "/sbin/scsi_id -g -u -s /block/%n"
        default_prio_callout    "/bin/true"
        failback        immediate
}
multipaths {

        multipath {
                wwid                    3600601608c901200ccfe543f4053d911
                alias                   200Gb
                path_grouping_policy    failover
                path_checker            readsector0
                path_selector           "round-robin 0"
                failback                immediate
        }
        multipath {
                wwid                    3600601608c9012006269b8f63b87d911
                alias                   5Gb
                path_grouping_policy    failover
                path_checker            readsector0
                path_selector           "round-robin 0"
                failback                immediate
        }
        multipath {
                wwid                    3600601608c9012008ae5de2cc985d911
                alias                   300Gb
                path_grouping_policy    failover
                path_checker            readsector0
                path_selector           "round-robin 0"
                failback                immediate
        }
}

The tests aren't very accurate because the machine is 100 Km away,
tomorrow I'll do more accurate tests, I'll post results,

big thanks to Cristophe for multipath-tools,
big thanks to gistolero for his accurate testing on gentoo, your posts
are very useful for me,

any suggestions to improve my setup are welcome,

I have to install naviagent? On redhat naviagent register powerpath name
to cx 300, it works with multipath?

Nicola

P.S. for gistolero, I haven't your problem with multipathd (or so seems)
however my hardware is different, I think also that the need to run
udevstart is a gentoo bug, today gentoo has releases udev-0.68-r1 with
start-stop script, if you read start script there is udevstart, however
I haven't yet tested this script because they require unstable
baselayout-1.12




[-- Attachment #1.2: Type: text/html, Size: 8971 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: EMC Cx300 multipath seems to work

[gistolero]

Hi Nicola,

Nicola Murino wrote:
> Hi,
> 
> multipath seems to work on gentoo, I have to load the modules in proper
> order, this is my configuration:
> 
> cat /etc/conf.d/local.start
> # /etc/conf.d/local.start
> 
> # This is a good place to load any misc programs
> # on startup ( use 1>&2 to hide output)
> /sbin/modprobe dm-emc
> /sbin/modprobe qla2300
> /bin/sleep 5
> /sbin/udevstart
> /etc/init.d/multipathd start
> 
> is critical the command udevstart without only one path is seen,


After rebooting my system, I see only one path always:

testhalde2 ~ # multipath -l
150gb ()
[size=150 GB][features="0"][hwhandler="0"]
\_ round-robin 0 [active]
  \_ 0:0:0:1 sda 8:0  [active]


I have to run multipath again:

testhalde2 ~ # multipath /dev/sda
reload: 150gb (3600508b40010079d0001900000460000)
[size=150 GB][features="0"][hwhandler="0"]
\_ round-robin 0
  \_ 0:0:0:1 sda 8:0  [active][ready]
  \_ 1:0:0:1 sdb 8:16 [ready]


> I experienced a kernel oops with kernel-2.6.12 (both vanilla-sources and
> gentoo-sources) when I load qla2300 module, here are the logs:

I'm using a HP DL-380 G3 Proliant + Qlogic QLA 2340 (qla2300) with 2.6.12.5 
(vanilla). No kernel oops here.


> P.S. for gistolero, I haven't your problem with multipathd (or so seems)
> however my hardware is different, I think also that the need to run
> udevstart is a gentoo bug, today gentoo has releases udev-0.68-r1 with
> start-stop script, if you read start script there is udevstart, however
> I haven't yet tested this script because they require unstable
> baselayout-1.12

The same behavior with udev-068-r1. I don't think that this is a gentoo bug, 
because distributions differ in udev _startup_ scripts only. I know that I 
have to run udevstart after creating a new udev.rules in /etc/udev/rules/. But 
  after this, udev should parse this rule after every change:



After reboot...

testhalde2 ~ # ls -lF /dev/mapper/
brw-------  1 root root 254,  0 Sep 13  2005 150gb
brw-------  1 root root 254,  1 Sep 13  2005 150gb1
brw-------  1 root root 254,  2 Sep 13  2005 150gb2
crw-rw----  1 root root  10, 63 Sep 13  2005 control

testhalde2 ~ # ls -lF /dev/1*
lrwxrwxrwx  1 root root 4 Sep 13  2005 /dev/150gb2 -> dm-2

testhalde2 ~ # udevstart

testhalde2 ~ # ls -lF /dev/1*
lrwxrwxrwx  1 root root 4 Sep 13 13:00 /dev/150gb -> dm-0
lrwxrwxrwx  1 root root 4 Sep 13 13:00 /dev/150gb1 -> dm-1
lrwxrwxrwx  1 root root 4 Sep 13 13:00 /dev/150gb2 -> dm-2

Ok, udev knows the multipath udev-rule and creates this links.



Deleting the multipath-table...

testhalde2 ~ # dmsetup table
150gb1: 0 64197 linear 254:0 63
150gb: 0 314572800 multipath 0 0 1 1 round-robin 0 2 1 8:0 1000 8:16 1000
150gb2: 0 314504505 linear 254:0 64260

testhalde2 ~ # dmsetup remove 150gb1
testhalde2 ~ # dmsetup remove 150gb2
testhalde2 ~ # dmsetup remove 150gb
testhalde2 ~ # dmsetup table
No devices found

testhalde2 ~ # ls -lF /dev/mapper/
total 0
crw-rw----  1 root root 10, 63 Sep 13  2005 control

testhalde2 ~ # ls -lF /dev/1*
ls: /dev/1*: No such file or directory



Creating the multipath-table...

testhalde2 ~ # multipath /dev/sda
create: 150gb (3600508b40010079d0001900000460000)
[size=150 GB][features="0"][hwhandler="0"]
\_ round-robin 0
  \_ 0:0:0:1 sda 8:0  [ready]
  \_ 1:0:0:1 sdb 8:16 [ready]

testhalde2 ~ # ls -lF /dev/mapper/
total 0
brw-------  1 root root 254,  0 Sep 13 13:04 150gb
brw-------  1 root root 254,  1 Sep 13 13:04 150gb1
brw-------  1 root root 254,  2 Sep 13 13:04 150gb2
crw-rw----  1 root root  10, 63 Sep 13  2005 control

testhalde2 ~ # ls -lF /dev/1*
ls: /dev/1*: No such file or directory


No links in /dev/... I don't understand this, because
udev should read the multipath udev-rule again.
Perhaps I'm mistaken?

Simon

Re: Re: EMC Cx300 multipath seems to work

[Nicola Murino]

gistolero@gmx.de ha scritto:

> Hi Nicola,
>
> Nicola Murino wrote:
>
>> Hi,
>>
>> multipath seems to work on gentoo, I have to load the modules in proper
>> order, this is my configuration:
>>
>> cat /etc/conf.d/local.start
>> # /etc/conf.d/local.start
>>
>> # This is a good place to load any misc programs
>> # on startup ( use 1>&2 to hide output)
>> /sbin/modprobe dm-emc
>> /sbin/modprobe qla2300
>> /bin/sleep 5
>> /sbin/udevstart
>> /etc/init.d/multipathd start
>>
>> is critical the command udevstart without only one path is seen,
>
>
>
> After rebooting my system, I see only one path always:
>
> testhalde2 ~ # multipath -l
> 150gb ()
> [size=150 GB][features="0"][hwhandler="0"]
> \_ round-robin 0 [active]
>  \_ 0:0:0:1 sda 8:0  [active]
>
>
> I have to run multipath again:
>
> testhalde2 ~ # multipath /dev/sda
> reload: 150gb (3600508b40010079d0001900000460000)
> [size=150 GB][features="0"][hwhandler="0"]
> \_ round-robin 0
>  \_ 0:0:0:1 sda 8:0  [active][ready]
>  \_ 1:0:0:1 sdb 8:16 [ready]
>
>
Yes, after reboot I see only one path too, but when I run udevstart I
see all paths, for this reason I put udevstart in /etc/conf.d/local.start

>> I experienced a kernel oops with kernel-2.6.12 (both vanilla-sources and
>> gentoo-sources) when I load qla2300 module, here are the logs:
>
>
> I'm using a HP DL-380 G3 Proliant + Qlogic QLA 2340 (qla2300) with
> 2.6.12.5 (vanilla). No kernel oops here.
>
I have qlogic 2322 and I must load dm-emc too, however
kernel-2.6.13-gentoo-r1 works fine

>
>> P.S. for gistolero, I haven't your problem with multipathd (or so seems)
>> however my hardware is different, I think also that the need to run
>> udevstart is a gentoo bug, today gentoo has releases udev-0.68-r1 with
>> start-stop script, if you read start script there is udevstart, however
>> I haven't yet tested this script because they require unstable
>> baselayout-1.12
>
>
> The same behavior with udev-068-r1. I don't think that this is a
> gentoo bug, because distributions differ in udev _startup_ scripts
> only. I know that I have to run udevstart after creating a new
> udev.rules in /etc/udev/rules/. But  after this, udev should parse
> this rule after every change:
>
>
udev 0.68-r1 needs unstable baselayout (1.12) to execute udev-start.sh,
what baselayout are you using?

>
> After reboot...
>
> testhalde2 ~ # ls -lF /dev/mapper/
> brw-------  1 root root 254,  0 Sep 13  2005 150gb
> brw-------  1 root root 254,  1 Sep 13  2005 150gb1
> brw-------  1 root root 254,  2 Sep 13  2005 150gb2
> crw-rw----  1 root root  10, 63 Sep 13  2005 control
> testhalde2 ~ # ls -lF /dev/1*
> lrwxrwxrwx  1 root root 4 Sep 13  2005 /dev/150gb2 -> dm-2
>
> testhalde2 ~ # udevstart
>
> testhalde2 ~ # ls -lF /dev/1*
> lrwxrwxrwx  1 root root 4 Sep 13 13:00 /dev/150gb -> dm-0
> lrwxrwxrwx  1 root root 4 Sep 13 13:00 /dev/150gb1 -> dm-1
> lrwxrwxrwx  1 root root 4 Sep 13 13:00 /dev/150gb2 -> dm-2
>
> Ok, udev knows the multipath udev-rule and creates this links.
>
>
>
> Deleting the multipath-table...
>
> testhalde2 ~ # dmsetup table
> 150gb1: 0 64197 linear 254:0 63
> 150gb: 0 314572800 multipath 0 0 1 1 round-robin 0 2 1 8:0 1000 8:16 1000
> 150gb2: 0 314504505 linear 254:0 64260
>
> testhalde2 ~ # dmsetup remove 150gb1
> testhalde2 ~ # dmsetup remove 150gb2
> testhalde2 ~ # dmsetup remove 150gb
> testhalde2 ~ # dmsetup table
> No devices found
>
> testhalde2 ~ # ls -lF /dev/mapper/
> total 0
> crw-rw----  1 root root 10, 63 Sep 13  2005 control
>
> testhalde2 ~ # ls -lF /dev/1*
> ls: /dev/1*: No such file or directory
>
>
>
> Creating the multipath-table...
>
> testhalde2 ~ # multipath /dev/sda
> create: 150gb (3600508b40010079d0001900000460000)
> [size=150 GB][features="0"][hwhandler="0"]
> \_ round-robin 0
>  \_ 0:0:0:1 sda 8:0  [ready]
>  \_ 1:0:0:1 sdb 8:16 [ready]
>
> testhalde2 ~ # ls -lF /dev/mapper/
> total 0
> brw-------  1 root root 254,  0 Sep 13 13:04 150gb
> brw-------  1 root root 254,  1 Sep 13 13:04 150gb1
> brw-------  1 root root 254,  2 Sep 13 13:04 150gb2
> crw-rw----  1 root root  10, 63 Sep 13  2005 control
>
> testhalde2 ~ # ls -lF /dev/1*
> ls: /dev/1*: No such file or directory
>
>
> No links in /dev/... I don't understand this, because
> udev should read the multipath udev-rule again.
> Perhaps I'm mistaken?
>
> Simon
>
> -- 
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
>

Re: Re: EMC Cx300 multipath seems to work

[gistolero]

> Yes, after reboot I see only one path too, but when I run udevstart I
> see all paths, for this reason I put udevstart in /etc/conf.d/local.start

Yes, but udevstart doesn't work for me. I have to start "multipath /dev/sda" 
again.


> udev 0.68-r1 needs unstable baselayout (1.12) to execute udev-start.sh,
> what baselayout are you using?

testhalde2 ~ # epm -aq | grep baselayout
baselayout-1.12.0_pre6-r3

Regards
Simon