* rdac path failure - Sun 6140
@ 2009-08-13 18:50 Stewart Smith
2009-08-13 19:42 ` Moger, Babu
0 siblings, 1 reply; 9+ messages in thread
From: Stewart Smith @ 2009-08-13 18:50 UTC (permalink / raw)
To: dm-devel
Hello All,
I am seeing many of these messages when my Sun 6140 array is under heavy I/O
device-mapper: multipath: Failing path 8:208.
device-mapper: multipath: Failing path 8:208.
device-mapper: multipath: Failing path 8:208.
device-mapper: multipath: Failing path 8:208.
device-mapper: multipath: Failing path 8:208.
I am running a Fedora 10 server, with two fiber connections to two
different switches. Both controllers on the 6140 have one connection
to each switch as well. The end result is that I see four paths to
each LUN.
When the volume is mounted and under significant load I see the
messages above every few seconds. They seem to appear every
"no_path_retry" seconds.
The 6140 controller firmware is up to date at version 07.50.08.10 and
I have installed the latest firmware for my Emulex LPe11002 cards. I
have reproduced the problem using both Cisco MDS and Brocade fiber
channel switches as well.
Using CAM, I have set the initiator Host Type to "Linux" at the
moment. I have tried other options as well without success.
I have NOT installed the RDAC drivers from either Sun or LSI -
primarily because they do not seem to build on my Fedora 10 kernel.
Any ideas would be greatly appreciated!!!
configs and debugging multipathd output is below.
Kernel: 2.6.27.24-170.2.68.fc10.x86_64
# multipath -lll
vol1 (3600a0b800048335200001e5d48b68a9b) dm-1 SUN,CSM200_R
[size=12T][features=1 queue_if_no_path][hwhandler=1 rdac][rw]
\_ round-robin 0 [prio=6][active]
\_ 5:0:1:2 sdj 8:144 [active][ready]
\_ 2:0:1:2 sdn 8:208 [active][ready]
\_ round-robin 0 [prio=0][enabled]
\_ 2:0:0:2 sdb 8:16 [active][ghost]
\_ 5:0:0:2 sdd 8:48 [active][ghost]
# cat /etc/multipath.conf
blacklist {
devnode "^sd[a-z][[0-9]*]"
devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
devnode "^hd[a-z][0-9]*"
devnode "^cciss!c[0-9]d[0-9](p[0-9]*)*"
}
defaults {
udev_dir /dev
polling_interval 10
selector "round-robin 0"
path_grouping_policy multibus
getuid_callout "/sbin/scsi_id --whitelisted /dev/%n"
prio alua
path_checker readsector0
rr_min_io 100
max_fds 8192
rr_weight priorities
failback immediate
no_path_retry fail
user_friendly_names yes
}
devices {
device {
vendor "SUN"
product "CSM200_R"
product_blacklist "Universal Xport"
getuid_callout "/sbin/scsi_id --whitelisted /dev/%n"
features "0"
hardware_handler "1 rdac"
path_selector "round-robin 0"
path_grouping_policy group_by_prio
failback immediate
rr_weight uniform
no_path_retry queue
rr_min_io 1000
path_checker rdac
prio rdac
}
}
multipaths {
multipath {
wwid 3600a0b800048335200001e5d48b68a9b
alias vol1
rr_weight priorities
no_path_retry 5
rr_min_io 100
}
}
# multipathd -d v3
Aug 13 14:48:53 | sdb: ownership set to vol1
Aug 13 14:48:53 | sdb: not found in pathvec
Aug 13 14:48:53 | sdb: mask = 0xc
Aug 13 14:48:53 | sdb: path checker = rdac (controller setting)
Aug 13 14:48:53 | sdb: state = 4
Aug 13 14:48:53 | sdb: rdac prio = 0
Aug 13 14:48:53 | sdd: ownership set to vol1
Aug 13 14:48:53 | sdd: not found in pathvec
Aug 13 14:48:53 | sdd: mask = 0xc
Aug 13 14:48:53 | sdd: path checker = rdac (controller setting)
Aug 13 14:48:53 | sdd: state = 4
Aug 13 14:48:53 | sdd: rdac prio = 0
Aug 13 14:48:53 | sdj: ownership set to vol1
Aug 13 14:48:53 | sdj: not found in pathvec
Aug 13 14:48:53 | sdj: mask = 0xc
Aug 13 14:48:53 | sdj: path checker = rdac (controller setting)
Aug 13 14:48:53 | sdj: state = 2
Aug 13 14:48:53 | sdj: rdac prio = 3
Aug 13 14:48:53 | sdn: ownership set to vol1
Aug 13 14:48:53 | sdn: not found in pathvec
Aug 13 14:48:53 | sdn: mask = 0xc
Aug 13 14:48:53 | sdn: path checker = rdac (controller setting)
Aug 13 14:48:53 | sdn: state = 2
Aug 13 14:48:53 | sdn: rdac prio = 3
Aug 13 14:48:53 | vol1: pgfailback = -2 (controller setting)
Aug 13 14:48:53 | vol1: pgpolicy = group_by_prio (controller setting)
Aug 13 14:48:53 | vol1: selector = round-robin 0 (controller setting)
Aug 13 14:48:53 | vol1: features = 0 (controller setting)
Aug 13 14:48:53 | vol1: hwhandler = 1 rdac (controller setting)
Aug 13 14:48:53 | vol1: rr_weight = 2 (LUN setting)
Aug 13 14:48:53 | vol1: minio = 100 (LUN setting)
Aug 13 14:48:53 | vol1: no_path_retry = 5 (multipath setting)
Aug 13 14:48:53 | pg_timeout = NONE (internal default)
Aug 13 14:48:53 | vol1: set ACT_CREATE (map does not exist)
create: vol1 (3600a0b800048335200001e5d48b68a9b) n/a SUN,CSM200_R
[size=12T][features=0][hwhandler=1 rdac][n/a]
\_ round-robin 0 [prio=6][undef]
\_ 5:0:1:2 sdj 8:144 [undef][ready]
\_ 2:0:1:2 sdn 8:208 [undef][ready]
\_ round-robin 0 [prio=0][undef]
\_ 2:0:0:2 sdb 8:16 [undef][ghost]
\_ 5:0:0:2 sdd 8:48 [undef][ghost]
^ permalink raw reply [flat|nested] 9+ messages in thread
* RE: rdac path failure - Sun 6140
2009-08-13 18:50 rdac path failure - Sun 6140 Stewart Smith
@ 2009-08-13 19:42 ` Moger, Babu
2009-08-13 20:27 ` Stewart Smith
0 siblings, 1 reply; 9+ messages in thread
From: Moger, Babu @ 2009-08-13 19:42 UTC (permalink / raw)
To: device-mapper development
Do you have /var/log/messages file for this problem?
Thanks
Babu Moger
> -----Original Message-----
> From: dm-devel-bounces@redhat.com [mailto:dm-devel-bounces@redhat.com] On
> Behalf Of Stewart Smith
> Sent: Thursday, August 13, 2009 1:51 PM
> To: dm-devel@redhat.com
> Subject: [dm-devel] rdac path failure - Sun 6140
>
> Hello All,
>
> I am seeing many of these messages when my Sun 6140 array is under heavy
> I/O
> device-mapper: multipath: Failing path 8:208.
> device-mapper: multipath: Failing path 8:208.
> device-mapper: multipath: Failing path 8:208.
> device-mapper: multipath: Failing path 8:208.
> device-mapper: multipath: Failing path 8:208.
>
>
> I am running a Fedora 10 server, with two fiber connections to two
> different switches. Both controllers on the 6140 have one connection
> to each switch as well. The end result is that I see four paths to
> each LUN.
>
> When the volume is mounted and under significant load I see the
> messages above every few seconds. They seem to appear every
> "no_path_retry" seconds.
>
> The 6140 controller firmware is up to date at version 07.50.08.10 and
> I have installed the latest firmware for my Emulex LPe11002 cards. I
> have reproduced the problem using both Cisco MDS and Brocade fiber
> channel switches as well.
>
> Using CAM, I have set the initiator Host Type to "Linux" at the
> moment. I have tried other options as well without success.
>
> I have NOT installed the RDAC drivers from either Sun or LSI -
> primarily because they do not seem to build on my Fedora 10 kernel.
>
> Any ideas would be greatly appreciated!!!
>
> configs and debugging multipathd output is below.
>
>
>
>
>
> Kernel: 2.6.27.24-170.2.68.fc10.x86_64
>
> # multipath -lll
> vol1 (3600a0b800048335200001e5d48b68a9b) dm-1 SUN,CSM200_R
> [size=12T][features=1 queue_if_no_path][hwhandler=1 rdac][rw]
> \_ round-robin 0 [prio=6][active]
> \_ 5:0:1:2 sdj 8:144 [active][ready]
> \_ 2:0:1:2 sdn 8:208 [active][ready]
> \_ round-robin 0 [prio=0][enabled]
> \_ 2:0:0:2 sdb 8:16 [active][ghost]
> \_ 5:0:0:2 sdd 8:48 [active][ghost]
>
>
> # cat /etc/multipath.conf
>
> blacklist {
> devnode "^sd[a-z][[0-9]*]"
> devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
> devnode "^hd[a-z][0-9]*"
> devnode "^cciss!c[0-9]d[0-9](p[0-9]*)*"
> }
>
> defaults {
> udev_dir /dev
> polling_interval 10
> selector "round-robin 0"
> path_grouping_policy multibus
> getuid_callout "/sbin/scsi_id --whitelisted /dev/%n"
> prio alua
> path_checker readsector0
> rr_min_io 100
> max_fds 8192
> rr_weight priorities
> failback immediate
> no_path_retry fail
> user_friendly_names yes
> }
> devices {
> device {
> vendor "SUN"
> product "CSM200_R"
> product_blacklist "Universal Xport"
> getuid_callout "/sbin/scsi_id --whitelisted
> /dev/%n"
> features "0"
> hardware_handler "1 rdac"
> path_selector "round-robin 0"
> path_grouping_policy group_by_prio
> failback immediate
> rr_weight uniform
> no_path_retry queue
> rr_min_io 1000
> path_checker rdac
> prio rdac
> }
> }
>
> multipaths {
> multipath {
> wwid 3600a0b800048335200001e5d48b68a9b
> alias vol1
> rr_weight priorities
> no_path_retry 5
> rr_min_io 100
> }
> }
>
>
>
> # multipathd -d v3
>
>
> Aug 13 14:48:53 | sdb: ownership set to vol1
> Aug 13 14:48:53 | sdb: not found in pathvec
> Aug 13 14:48:53 | sdb: mask = 0xc
> Aug 13 14:48:53 | sdb: path checker = rdac (controller setting)
> Aug 13 14:48:53 | sdb: state = 4
> Aug 13 14:48:53 | sdb: rdac prio = 0
> Aug 13 14:48:53 | sdd: ownership set to vol1
> Aug 13 14:48:53 | sdd: not found in pathvec
> Aug 13 14:48:53 | sdd: mask = 0xc
> Aug 13 14:48:53 | sdd: path checker = rdac (controller setting)
> Aug 13 14:48:53 | sdd: state = 4
> Aug 13 14:48:53 | sdd: rdac prio = 0
> Aug 13 14:48:53 | sdj: ownership set to vol1
> Aug 13 14:48:53 | sdj: not found in pathvec
> Aug 13 14:48:53 | sdj: mask = 0xc
> Aug 13 14:48:53 | sdj: path checker = rdac (controller setting)
> Aug 13 14:48:53 | sdj: state = 2
> Aug 13 14:48:53 | sdj: rdac prio = 3
> Aug 13 14:48:53 | sdn: ownership set to vol1
> Aug 13 14:48:53 | sdn: not found in pathvec
> Aug 13 14:48:53 | sdn: mask = 0xc
> Aug 13 14:48:53 | sdn: path checker = rdac (controller setting)
> Aug 13 14:48:53 | sdn: state = 2
> Aug 13 14:48:53 | sdn: rdac prio = 3
> Aug 13 14:48:53 | vol1: pgfailback = -2 (controller setting)
> Aug 13 14:48:53 | vol1: pgpolicy = group_by_prio (controller setting)
> Aug 13 14:48:53 | vol1: selector = round-robin 0 (controller setting)
> Aug 13 14:48:53 | vol1: features = 0 (controller setting)
> Aug 13 14:48:53 | vol1: hwhandler = 1 rdac (controller setting)
> Aug 13 14:48:53 | vol1: rr_weight = 2 (LUN setting)
> Aug 13 14:48:53 | vol1: minio = 100 (LUN setting)
> Aug 13 14:48:53 | vol1: no_path_retry = 5 (multipath setting)
> Aug 13 14:48:53 | pg_timeout = NONE (internal default)
> Aug 13 14:48:53 | vol1: set ACT_CREATE (map does not exist)
> create: vol1 (3600a0b800048335200001e5d48b68a9b) n/a SUN,CSM200_R
> [size=12T][features=0][hwhandler=1 rdac][n/a]
> \_ round-robin 0 [prio=6][undef]
> \_ 5:0:1:2 sdj 8:144 [undef][ready]
> \_ 2:0:1:2 sdn 8:208 [undef][ready]
> \_ round-robin 0 [prio=0][undef]
> \_ 2:0:0:2 sdb 8:16 [undef][ghost]
> \_ 5:0:0:2 sdd 8:48 [undef][ghost]
>
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: rdac path failure - Sun 6140
2009-08-13 19:42 ` Moger, Babu
@ 2009-08-13 20:27 ` Stewart Smith
2009-08-13 20:34 ` Stewart Smith
0 siblings, 1 reply; 9+ messages in thread
From: Stewart Smith @ 2009-08-13 20:27 UTC (permalink / raw)
To: device-mapper development
[-- Attachment #1.1: Type: text/plain, Size: 7913 bytes --]
after a fresh, multipath -F and start of multipathd with -v 2 I see the
following messages.
After starting multipathd I mounted /dev/mapper/vol1 and generated some
simple I/O to it using dd
Aug 13 16:23:14.888 localhost kernel: device-mapper: multipath: Failing path
8:208.
Aug 13 16:23:14.000 localhost multipathd: 8:208: mark as failed
Aug 13 16:23:16.000 localhost multipathd: 8:208: reinstated
Aug 13 16:23:30.462 localhost kernel: device-mapper: multipath: Failing path
8:208.
Aug 13 16:23:30.000 localhost multipathd: 8:208: mark as failed
Aug 13 16:23:39.000 localhost multipathd: 8:208: reinstated
Aug 13 16:23:46.430 localhost kernel: device-mapper: multipath: Failing path
8:208.
Aug 13 16:23:46.000 localhost multipathd: 8:208: mark as failed
Aug 13 16:23:51.041 localhost kernel: device-mapper: multipath: Failing path
8:208.
Aug 13 16:23:51.000 localhost multipathd: 8:208: mark as failed
Aug 13 16:23:59.000 localhost multipathd: 8:208: reinstated
Aug 13 16:24:06.465 localhost kernel: device-mapper: multipath: Failing path
8:208.
Aug 13 16:24:06.000 localhost multipathd: 8:208: mark as failed
Aug 13 16:24:09.000 localhost multipathd: 8:208: reinstated
Thanks,
--
Stew
On Thu, Aug 13, 2009 at 12:42 PM, Moger, Babu <Babu.Moger@lsi.com> wrote:
> Do you have /var/log/messages file for this problem?
>
> Thanks
> Babu Moger
>
> > -----Original Message-----
> > From: dm-devel-bounces@redhat.com [mailto:dm-devel-bounces@redhat.com]
> On
> > Behalf Of Stewart Smith
> > Sent: Thursday, August 13, 2009 1:51 PM
> > To: dm-devel@redhat.com
> > Subject: [dm-devel] rdac path failure - Sun 6140
> >
> > Hello All,
> >
> > I am seeing many of these messages when my Sun 6140 array is under heavy
> > I/O
> > device-mapper: multipath: Failing path 8:208.
> > device-mapper: multipath: Failing path 8:208.
> > device-mapper: multipath: Failing path 8:208.
> > device-mapper: multipath: Failing path 8:208.
> > device-mapper: multipath: Failing path 8:208.
> >
> >
> > I am running a Fedora 10 server, with two fiber connections to two
> > different switches. Both controllers on the 6140 have one connection
> > to each switch as well. The end result is that I see four paths to
> > each LUN.
> >
> > When the volume is mounted and under significant load I see the
> > messages above every few seconds. They seem to appear every
> > "no_path_retry" seconds.
> >
> > The 6140 controller firmware is up to date at version 07.50.08.10 and
> > I have installed the latest firmware for my Emulex LPe11002 cards. I
> > have reproduced the problem using both Cisco MDS and Brocade fiber
> > channel switches as well.
> >
> > Using CAM, I have set the initiator Host Type to "Linux" at the
> > moment. I have tried other options as well without success.
> >
> > I have NOT installed the RDAC drivers from either Sun or LSI -
> > primarily because they do not seem to build on my Fedora 10 kernel.
> >
> > Any ideas would be greatly appreciated!!!
> >
> > configs and debugging multipathd output is below.
> >
> >
> >
> >
> >
> > Kernel: 2.6.27.24-170.2.68.fc10.x86_64
> >
> > # multipath -lll
> > vol1 (3600a0b800048335200001e5d48b68a9b) dm-1 SUN,CSM200_R
> > [size=12T][features=1 queue_if_no_path][hwhandler=1 rdac][rw]
> > \_ round-robin 0 [prio=6][active]
> > \_ 5:0:1:2 sdj 8:144 [active][ready]
> > \_ 2:0:1:2 sdn 8:208 [active][ready]
> > \_ round-robin 0 [prio=0][enabled]
> > \_ 2:0:0:2 sdb 8:16 [active][ghost]
> > \_ 5:0:0:2 sdd 8:48 [active][ghost]
> >
> >
> > # cat /etc/multipath.conf
> >
> > blacklist {
> > devnode "^sd[a-z][[0-9]*]"
> > devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
> > devnode "^hd[a-z][0-9]*"
> > devnode "^cciss!c[0-9]d[0-9](p[0-9]*)*"
> > }
> >
> > defaults {
> > udev_dir /dev
> > polling_interval 10
> > selector "round-robin 0"
> > path_grouping_policy multibus
> > getuid_callout "/sbin/scsi_id --whitelisted /dev/%n"
> > prio alua
> > path_checker readsector0
> > rr_min_io 100
> > max_fds 8192
> > rr_weight priorities
> > failback immediate
> > no_path_retry fail
> > user_friendly_names yes
> > }
> > devices {
> > device {
> > vendor "SUN"
> > product "CSM200_R"
> > product_blacklist "Universal Xport"
> > getuid_callout "/sbin/scsi_id --whitelisted
> > /dev/%n"
> > features "0"
> > hardware_handler "1 rdac"
> > path_selector "round-robin 0"
> > path_grouping_policy group_by_prio
> > failback immediate
> > rr_weight uniform
> > no_path_retry queue
> > rr_min_io 1000
> > path_checker rdac
> > prio rdac
> > }
> > }
> >
> > multipaths {
> > multipath {
> > wwid 3600a0b800048335200001e5d48b68a9b
> > alias vol1
> > rr_weight priorities
> > no_path_retry 5
> > rr_min_io 100
> > }
> > }
> >
> >
> >
> > # multipathd -d v3
> >
> >
> > Aug 13 14:48:53 | sdb: ownership set to vol1
> > Aug 13 14:48:53 | sdb: not found in pathvec
> > Aug 13 14:48:53 | sdb: mask = 0xc
> > Aug 13 14:48:53 | sdb: path checker = rdac (controller setting)
> > Aug 13 14:48:53 | sdb: state = 4
> > Aug 13 14:48:53 | sdb: rdac prio = 0
> > Aug 13 14:48:53 | sdd: ownership set to vol1
> > Aug 13 14:48:53 | sdd: not found in pathvec
> > Aug 13 14:48:53 | sdd: mask = 0xc
> > Aug 13 14:48:53 | sdd: path checker = rdac (controller setting)
> > Aug 13 14:48:53 | sdd: state = 4
> > Aug 13 14:48:53 | sdd: rdac prio = 0
> > Aug 13 14:48:53 | sdj: ownership set to vol1
> > Aug 13 14:48:53 | sdj: not found in pathvec
> > Aug 13 14:48:53 | sdj: mask = 0xc
> > Aug 13 14:48:53 | sdj: path checker = rdac (controller setting)
> > Aug 13 14:48:53 | sdj: state = 2
> > Aug 13 14:48:53 | sdj: rdac prio = 3
> > Aug 13 14:48:53 | sdn: ownership set to vol1
> > Aug 13 14:48:53 | sdn: not found in pathvec
> > Aug 13 14:48:53 | sdn: mask = 0xc
> > Aug 13 14:48:53 | sdn: path checker = rdac (controller setting)
> > Aug 13 14:48:53 | sdn: state = 2
> > Aug 13 14:48:53 | sdn: rdac prio = 3
> > Aug 13 14:48:53 | vol1: pgfailback = -2 (controller setting)
> > Aug 13 14:48:53 | vol1: pgpolicy = group_by_prio (controller setting)
> > Aug 13 14:48:53 | vol1: selector = round-robin 0 (controller setting)
> > Aug 13 14:48:53 | vol1: features = 0 (controller setting)
> > Aug 13 14:48:53 | vol1: hwhandler = 1 rdac (controller setting)
> > Aug 13 14:48:53 | vol1: rr_weight = 2 (LUN setting)
> > Aug 13 14:48:53 | vol1: minio = 100 (LUN setting)
> > Aug 13 14:48:53 | vol1: no_path_retry = 5 (multipath setting)
> > Aug 13 14:48:53 | pg_timeout = NONE (internal default)
> > Aug 13 14:48:53 | vol1: set ACT_CREATE (map does not exist)
> > create: vol1 (3600a0b800048335200001e5d48b68a9b) n/a SUN,CSM200_R
> > [size=12T][features=0][hwhandler=1 rdac][n/a]
> > \_ round-robin 0 [prio=6][undef]
> > \_ 5:0:1:2 sdj 8:144 [undef][ready]
> > \_ 2:0:1:2 sdn 8:208 [undef][ready]
> > \_ round-robin 0 [prio=0][undef]
> > \_ 2:0:0:2 sdb 8:16 [undef][ghost]
> > \_ 5:0:0:2 sdd 8:48 [undef][ghost]
> >
> > --
> > dm-devel mailing list
> > dm-devel@redhat.com
> > https://www.redhat.com/mailman/listinfo/dm-devel
>
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
>
[-- Attachment #1.2: Type: text/html, Size: 10095 bytes --]
[-- Attachment #2: Type: text/plain, Size: 0 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: rdac path failure - Sun 6140
2009-08-13 20:27 ` Stewart Smith
@ 2009-08-13 20:34 ` Stewart Smith
2009-08-14 0:05 ` Moger, Babu
0 siblings, 1 reply; 9+ messages in thread
From: Stewart Smith @ 2009-08-13 20:34 UTC (permalink / raw)
To: device-mapper development
[-- Attachment #1.1: Type: text/plain, Size: 16401 bytes --]
Same sequence of events, with multipathd -v3
Aug 13 16:28:48.627 kernel: device-mapper: multipath: Failing path 8:208.
Aug 13 16:28:48.000 multipathd: vol1: rr_weight = 2 (LUN setting)
Aug 13 16:28:48.000 multipathd: vol1: pgfailback = -2 (controller setting)
Aug 13 16:28:48.000 multipathd: pg_timeout = NONE (internal default)
Aug 13 16:28:48.000 multipathd: 8:208: mark as failed
Aug 13 16:28:48.000 multipathd: uevent 'change' from
'/devices/virtual/block/dm-1'
Aug 13 16:28:48.000 multipathd: UDEV_LOG=3
Aug 13 16:28:48.000 multipathd: ACTION=change
Aug 13 16:28:48.000 multipathd: DEVPATH=/devices/virtual/block/dm-1
Aug 13 16:28:48.000 multipathd: SUBSYSTEM=block
Aug 13 16:28:48.000 multipathd: DM_TARGET=multipath
Aug 13 16:28:48.000 multipathd: DM_ACTION=PATH_FAILED
Aug 13 16:28:48.000 multipathd: DM_SEQNUM=1
Aug 13 16:28:48.000 multipathd: DM_PATH=8:208
Aug 13 16:28:48.000 multipathd: DM_NR_VALID_PATHS=3
Aug 13 16:28:48.000 multipathd: DM_NAME=vol1
Aug 13 16:28:48.000 multipathd:
DM_UUID=mpath-3600a0b800048335200001e5d48b68a9b
Aug 13 16:28:48.000 multipathd: MAJOR=253
Aug 13 16:28:48.000 multipathd: MINOR=1
Aug 13 16:28:48.000 multipathd: DEVTYPE=disk
Aug 13 16:28:48.000 multipathd: SEQNUM=1738
Aug 13 16:28:48.000 multipathd: UDEVD_EVENT=1
Aug 13 16:28:48.000 multipathd: DEVNAME=/dev/dm-1
Aug 13 16:28:50.000 multipathd: 8:208: reinstated
Aug 13 16:28:50.000 multipathd: vol1: remaining active paths: 4
Aug 13 16:28:50.000 multipathd: sdj: rdac prio = 3
Aug 13 16:28:50.000 multipathd: sdn: rdac prio = 3
Aug 13 16:28:50.000 multipathd: sdb: rdac prio = 0
Aug 13 16:28:50.000 multipathd: sdd: rdac prio = 0
Aug 13 16:28:50.763 kernel: device-mapper: multipath: Failing path 8:208.
Aug 13 16:28:50.000 multipathd: uevent 'change' from
'/devices/virtual/block/dm-1'
Aug 13 16:28:50.000 multipathd: UDEV_LOG=3
Aug 13 16:28:50.000 multipathd: ACTION=change
Aug 13 16:28:50.000 multipathd: DEVPATH=/devices/virtual/block/dm-1
Aug 13 16:28:50.000 multipathd: SUBSYSTEM=block
Aug 13 16:28:50.000 multipathd: DM_TARGET=multipath
Aug 13 16:28:50.000 multipathd: DM_ACTION=PATH_REINSTATED
Aug 13 16:28:50.000 multipathd: DM_SEQNUM=2
Aug 13 16:28:50.000 multipathd: DM_PATH=8:208
Aug 13 16:28:50.000 multipathd: DM_NR_VALID_PATHS=4
Aug 13 16:28:50.000 multipathd: DM_NAME=vol1
Aug 13 16:28:50.000 multipathd:
DM_UUID=mpath-3600a0b800048335200001e5d48b68a9b
Aug 13 16:28:50.000 multipathd: MAJOR=253
Aug 13 16:28:50.000 multipathd: MINOR=1
Aug 13 16:28:50.000 multipathd: DEVTYPE=diskAug 13 16:28:50.000
multipathd: SEQNUM=1739Aug 13 16:28:50.000 multipathd: UDEVD_EVENT=1
Aug 13 16:28:50.000 multipathd: vol1: rr_weight = 2 (LUN setting)
Aug 13 16:28:50.000 multipathd: vol1: pgfailback = -2 (controller setting)
Aug 13 16:28:50.000 multipathd: pg_timeout = NONE (internal default)
Aug 13 16:28:50.000 multipathd: 8:208: mark as failed
Aug 13 16:28:50.000 multipathd: vol1: remaining active paths: 3
Aug 13 16:28:50.000 multipathd: vol1: rr_weight = 2 (LUN setting)
Aug 13 16:28:50.000 multipathd: vol1: pgfailback = -2 (controller setting)
Aug 13 16:28:50.000 multipathd: uevent 'change' from
'/devices/virtual/block/dm-1'
Aug 13 16:28:50.000 multipathd: UDEV_LOG=3
Aug 13 16:28:50.000 multipathd: ACTION=change
Aug 13 16:28:50.000 multipathd: DEVPATH=/devices/virtual/block/dm-1
Aug 13 16:28:50.000 multipathd: SUBSYSTEM=block
Aug 13 16:28:50.000 multipathd: DM_TARGET=multipath
Aug 13 16:28:50.000 multipathd: DM_ACTION=PATH_FAILED
Aug 13 16:28:50.000 multipathd: DM_SEQNUM=3
Aug 13 16:28:50.000 multipathd: DM_PATH=8:208
Aug 13 16:28:50.000 multipathd: DM_NR_VALID_PATHS=3
Aug 13 16:28:50.000 multipathd: DM_NAME=vol1
Aug 13 16:28:50.000 multipathd:
DM_UUID=mpath-3600a0b800048335200001e5d48b68a9b
Aug 13 16:28:50.000 multipathd: MAJOR=253
Aug 13 16:28:50.000 multipathd: MINOR=1
Aug 13 16:28:50.000 multipathd: DEVTYPE=disk
Aug 13 16:28:50.000 multipathd: SEQNUM=1740
Aug 13 16:28:50.000 multipathd: UDEVD_EVENT=1
Aug 13 16:28:50.000 multipathd: DEVNAME=/dev/dm-1
Aug 13 16:29:00.000 multipathd: 8:208: reinstated
Aug 13 16:29:00.000 multipathd: vol1: remaining active paths: 4
Aug 13 16:29:00.000 multipathd: sdj: rdac prio = 3
Aug 13 16:29:00.000 multipathd: sdn: rdac prio = 3
Aug 13 16:29:00.000 multipathd: sdb: rdac prio = 0
Aug 13 16:29:00.000 multipathd: sdd: rdac prio = 0
Aug 13 16:29:00.000 multipathd: vol1: rr_weight = 2 (LUN setting)
Aug 13 16:29:00.000 multipathd: vol1: pgfailback = -2 (controller setting)
Aug 13 16:29:00.000 multipathd: uevent 'change' from
'/devices/virtual/block/dm-1'
Aug 13 16:29:00.000 multipathd: UDEV_LOG=3
Aug 13 16:29:00.000 multipathd: ACTION=change
Aug 13 16:29:00.000 multipathd: DEVPATH=/devices/virtual/block/dm-1
Aug 13 16:29:00.000 multipathd: SUBSYSTEM=block
Aug 13 16:29:00.000 multipathd: DM_TARGET=multipath
Aug 13 16:29:00.000 multipathd: DM_ACTION=PATH_REINSTATED
Aug 13 16:29:00.000 multipathd: DM_SEQNUM=4
Aug 13 16:29:00.000 multipathd: DM_PATH=8:208
Aug 13 16:29:00.000 multipathd: DM_NR_VALID_PATHS=4
Aug 13 16:29:00.000 multipathd: DM_NAME=vol1
Aug 13 16:29:00.000 multipathd:
DM_UUID=mpath-3600a0b800048335200001e5d48b68a9b
Aug 13 16:29:00.000 multipathd: MAJOR=253
Aug 13 16:29:00.000 multipathd: MINOR=1
Aug 13 16:29:00.000 multipathd: DEVTYPE=disk
Aug 13 16:29:00.000 multipathd: SEQNUM=1741
Aug 13 16:29:00.000 multipathd: UDEVD_EVENT=1
Aug 13 16:29:00.000 multipathd: DEVNAME=/dev/dm-1
Aug 13 16:29:02.753 kernel: device-mapper: multipath: Failing path 8:208.
Aug 13 16:29:02.000 multipathd: vol1: rr_weight = 2 (LUN setting)
Aug 13 16:29:02.000 multipathd: vol1: pgfailback = -2 (controller setting)
Aug 13 16:29:02.000 multipathd: pg_timeout = NONE (internal default)
Aug 13 16:29:02.000 multipathd: 8:208: mark as failed
Aug 13 16:29:02.000 multipathd: uevent 'change' from
'/devices/virtual/block/dm-1'
Aug 13 16:29:02.000 multipathd: UDEV_LOG=3
Aug 13 16:29:02.000 multipathd: ACTION=change
Aug 13 16:29:02.000 multipathd: DEVPATH=/devices/virtual/block/dm-1
Aug 13 16:29:02.000 multipathd: SUBSYSTEM=block
Aug 13 16:29:02.000 multipathd: DM_TARGET=multipath
Aug 13 16:29:02.000 multipathd: DM_ACTION=PATH_FAILED
Aug 13 16:29:02.000 multipathd: DM_SEQNUM=5
Aug 13 16:29:02.000 multipathd: DM_PATH=8:208
Aug 13 16:29:02.000 multipathd: DM_NR_VALID_PATHS=3
Aug 13 16:29:02.000 multipathd: DM_NAME=vol1
Aug 13 16:29:02.000 multipathd:
DM_UUID=mpath-3600a0b800048335200001e5d48b68a9b
Aug 13 16:29:02.000 multipathd: MAJOR=253
Aug 13 16:29:02.000 multipathd: MINOR=1
Aug 13 16:29:02.000 multipathd: DEVTYPE=disk
Aug 13 16:29:02.000 multipathd: SEQNUM=1742
Aug 13 16:29:02.000 multipathd: UDEVD_EVENT=1
Aug 13 16:29:02.000 multipathd: DEVNAME=/dev/dm-1
Aug 13 16:29:10.000 multipathd: 8:208: reinstated
Aug 13 16:29:10.000 multipathd: vol1: remaining active paths: 4
Aug 13 16:29:10.000 multipathd: sdj: rdac prio = 3
Aug 13 16:29:10.000 multipathd: sdn: rdac prio = 3
Aug 13 16:29:10.000 multipathd: sdb: rdac prio = 0
Aug 13 16:29:10.000 multipathd: sdd: rdac prio = 0
Aug 13 16:29:10.000 multipathd: vol1: rr_weight = 2 (LUN setting)
Aug 13 16:29:10.000 multipathd: vol1: pgfailback = -2 (controller setting)
Aug 13 16:29:10.000 multipathd: uevent 'change' from
'/devices/virtual/block/dm-1'
Aug 13 16:29:10.000 multipathd: UDEV_LOG=3
Aug 13 16:29:10.000 multipathd: ACTION=change
Aug 13 16:29:10.000 multipathd: DEVPATH=/devices/virtual/block/dm-1
Aug 13 16:29:10.000 multipathd: SUBSYSTEM=block
Aug 13 16:29:10.000 multipathd: DM_TARGET=multipath
Aug 13 16:29:10.000 multipathd: DM_ACTION=PATH_REINSTATED
Aug 13 16:29:10.000 multipathd: DM_SEQNUM=6
Aug 13 16:29:10.000 multipathd: DM_PATH=8:208
Aug 13 16:29:10.000 multipathd: DM_NR_VALID_PATHS=4
Aug 13 16:29:10.000 multipathd: DM_NAME=vol1
Aug 13 16:29:10.000 multipathd:
DM_UUID=mpath-3600a0b800048335200001e5d48b68a9b
Aug 13 16:29:10.000 multipathd: MAJOR=253
Aug 13 16:29:10.000 multipathd: MINOR=1
Aug 13 16:29:10.000 multipathd: DEVTYPE=disk
Aug 13 16:29:10.000 multipathd: SEQNUM=1743
Aug 13 16:29:10.000 multipathd: UDEVD_EVENT=1
Aug 13 16:29:10.000 multipathd: DEVNAME=/dev/dm-1
On Thu, Aug 13, 2009 at 1:27 PM, Stewart Smith <stew@cleepdar.com> wrote:
>
> after a fresh, multipath -F and start of multipathd with -v 2 I see the
> following messages.
>
> After starting multipathd I mounted /dev/mapper/vol1 and generated some
> simple I/O to it using dd
>
>
> Aug 13 16:23:14.888 localhost kernel: device-mapper: multipath: Failing
> path 8:208.
> Aug 13 16:23:14.000 localhost multipathd: 8:208: mark as failed
> Aug 13 16:23:16.000 localhost multipathd: 8:208: reinstated
> Aug 13 16:23:30.462 localhost kernel: device-mapper: multipath: Failing
> path 8:208.
> Aug 13 16:23:30.000 localhost multipathd: 8:208: mark as failed
> Aug 13 16:23:39.000 localhost multipathd: 8:208: reinstated
> Aug 13 16:23:46.430 localhost kernel: device-mapper: multipath: Failing
> path 8:208.
> Aug 13 16:23:46.000 localhost multipathd: 8:208: mark as failed
> Aug 13 16:23:51.041 localhost kernel: device-mapper: multipath: Failing
> path 8:208.
> Aug 13 16:23:51.000 localhost multipathd: 8:208: mark as failed
> Aug 13 16:23:59.000 localhost multipathd: 8:208: reinstated
> Aug 13 16:24:06.465 localhost kernel: device-mapper: multipath: Failing
> path 8:208.
> Aug 13 16:24:06.000 localhost multipathd: 8:208: mark as failed
> Aug 13 16:24:09.000 localhost multipathd: 8:208: reinstated
>
>
> Thanks,
> --
> Stew
>
>
>
>
> On Thu, Aug 13, 2009 at 12:42 PM, Moger, Babu <Babu.Moger@lsi.com> wrote:
>
>> Do you have /var/log/messages file for this problem?
>>
>> Thanks
>> Babu Moger
>>
>> > -----Original Message-----
>> > From: dm-devel-bounces@redhat.com [mailto:dm-devel-bounces@redhat.com]
>> On
>> > Behalf Of Stewart Smith
>> > Sent: Thursday, August 13, 2009 1:51 PM
>> > To: dm-devel@redhat.com
>> > Subject: [dm-devel] rdac path failure - Sun 6140
>> >
>> > Hello All,
>> >
>> > I am seeing many of these messages when my Sun 6140 array is under heavy
>> > I/O
>> > device-mapper: multipath: Failing path 8:208.
>> > device-mapper: multipath: Failing path 8:208.
>> > device-mapper: multipath: Failing path 8:208.
>> > device-mapper: multipath: Failing path 8:208.
>> > device-mapper: multipath: Failing path 8:208.
>> >
>> >
>> > I am running a Fedora 10 server, with two fiber connections to two
>> > different switches. Both controllers on the 6140 have one connection
>> > to each switch as well. The end result is that I see four paths to
>> > each LUN.
>> >
>> > When the volume is mounted and under significant load I see the
>> > messages above every few seconds. They seem to appear every
>> > "no_path_retry" seconds.
>> >
>> > The 6140 controller firmware is up to date at version 07.50.08.10 and
>> > I have installed the latest firmware for my Emulex LPe11002 cards. I
>> > have reproduced the problem using both Cisco MDS and Brocade fiber
>> > channel switches as well.
>> >
>> > Using CAM, I have set the initiator Host Type to "Linux" at the
>> > moment. I have tried other options as well without success.
>> >
>> > I have NOT installed the RDAC drivers from either Sun or LSI -
>> > primarily because they do not seem to build on my Fedora 10 kernel.
>> >
>> > Any ideas would be greatly appreciated!!!
>> >
>> > configs and debugging multipathd output is below.
>> >
>> >
>> >
>> >
>> >
>> > Kernel: 2.6.27.24-170.2.68.fc10.x86_64
>> >
>> > # multipath -lll
>> > vol1 (3600a0b800048335200001e5d48b68a9b) dm-1 SUN,CSM200_R
>> > [size=12T][features=1 queue_if_no_path][hwhandler=1 rdac][rw]
>> > \_ round-robin 0 [prio=6][active]
>> > \_ 5:0:1:2 sdj 8:144 [active][ready]
>> > \_ 2:0:1:2 sdn 8:208 [active][ready]
>> > \_ round-robin 0 [prio=0][enabled]
>> > \_ 2:0:0:2 sdb 8:16 [active][ghost]
>> > \_ 5:0:0:2 sdd 8:48 [active][ghost]
>> >
>> >
>> > # cat /etc/multipath.conf
>> >
>> > blacklist {
>> > devnode "^sd[a-z][[0-9]*]"
>> > devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
>> > devnode "^hd[a-z][0-9]*"
>> > devnode "^cciss!c[0-9]d[0-9](p[0-9]*)*"
>> > }
>> >
>> > defaults {
>> > udev_dir /dev
>> > polling_interval 10
>> > selector "round-robin 0"
>> > path_grouping_policy multibus
>> > getuid_callout "/sbin/scsi_id --whitelisted /dev/%n"
>> > prio alua
>> > path_checker readsector0
>> > rr_min_io 100
>> > max_fds 8192
>> > rr_weight priorities
>> > failback immediate
>> > no_path_retry fail
>> > user_friendly_names yes
>> > }
>> > devices {
>> > device {
>> > vendor "SUN"
>> > product "CSM200_R"
>> > product_blacklist "Universal Xport"
>> > getuid_callout "/sbin/scsi_id --whitelisted
>> > /dev/%n"
>> > features "0"
>> > hardware_handler "1 rdac"
>> > path_selector "round-robin 0"
>> > path_grouping_policy group_by_prio
>> > failback immediate
>> > rr_weight uniform
>> > no_path_retry queue
>> > rr_min_io 1000
>> > path_checker rdac
>> > prio rdac
>> > }
>> > }
>> >
>> > multipaths {
>> > multipath {
>> > wwid
>> 3600a0b800048335200001e5d48b68a9b
>> > alias vol1
>> > rr_weight priorities
>> > no_path_retry 5
>> > rr_min_io 100
>> > }
>> > }
>> >
>> >
>> >
>> > # multipathd -d v3
>> >
>> >
>> > Aug 13 14:48:53 | sdb: ownership set to vol1
>> > Aug 13 14:48:53 | sdb: not found in pathvec
>> > Aug 13 14:48:53 | sdb: mask = 0xc
>> > Aug 13 14:48:53 | sdb: path checker = rdac (controller setting)
>> > Aug 13 14:48:53 | sdb: state = 4
>> > Aug 13 14:48:53 | sdb: rdac prio = 0
>> > Aug 13 14:48:53 | sdd: ownership set to vol1
>> > Aug 13 14:48:53 | sdd: not found in pathvec
>> > Aug 13 14:48:53 | sdd: mask = 0xc
>> > Aug 13 14:48:53 | sdd: path checker = rdac (controller setting)
>> > Aug 13 14:48:53 | sdd: state = 4
>> > Aug 13 14:48:53 | sdd: rdac prio = 0
>> > Aug 13 14:48:53 | sdj: ownership set to vol1
>> > Aug 13 14:48:53 | sdj: not found in pathvec
>> > Aug 13 14:48:53 | sdj: mask = 0xc
>> > Aug 13 14:48:53 | sdj: path checker = rdac (controller setting)
>> > Aug 13 14:48:53 | sdj: state = 2
>> > Aug 13 14:48:53 | sdj: rdac prio = 3
>> > Aug 13 14:48:53 | sdn: ownership set to vol1
>> > Aug 13 14:48:53 | sdn: not found in pathvec
>> > Aug 13 14:48:53 | sdn: mask = 0xc
>> > Aug 13 14:48:53 | sdn: path checker = rdac (controller setting)
>> > Aug 13 14:48:53 | sdn: state = 2
>> > Aug 13 14:48:53 | sdn: rdac prio = 3
>> > Aug 13 14:48:53 | vol1: pgfailback = -2 (controller setting)
>> > Aug 13 14:48:53 | vol1: pgpolicy = group_by_prio (controller setting)
>> > Aug 13 14:48:53 | vol1: selector = round-robin 0 (controller setting)
>> > Aug 13 14:48:53 | vol1: features = 0 (controller setting)
>> > Aug 13 14:48:53 | vol1: hwhandler = 1 rdac (controller setting)
>> > Aug 13 14:48:53 | vol1: rr_weight = 2 (LUN setting)
>> > Aug 13 14:48:53 | vol1: minio = 100 (LUN setting)
>> > Aug 13 14:48:53 | vol1: no_path_retry = 5 (multipath setting)
>> > Aug 13 14:48:53 | pg_timeout = NONE (internal default)
>> > Aug 13 14:48:53 | vol1: set ACT_CREATE (map does not exist)
>> > create: vol1 (3600a0b800048335200001e5d48b68a9b) n/a SUN,CSM200_R
>> > [size=12T][features=0][hwhandler=1 rdac][n/a]
>> > \_ round-robin 0 [prio=6][undef]
>> > \_ 5:0:1:2 sdj 8:144 [undef][ready]
>> > \_ 2:0:1:2 sdn 8:208 [undef][ready]
>> > \_ round-robin 0 [prio=0][undef]
>> > \_ 2:0:0:2 sdb 8:16 [undef][ghost]
>> > \_ 5:0:0:2 sdd 8:48 [undef][ghost]
>> >
>> > --
>> > dm-devel mailing list
>> > dm-devel@redhat.com
>> > https://www.redhat.com/mailman/listinfo/dm-devel
>>
>> --
>> dm-devel mailing list
>> dm-devel@redhat.com
>> https://www.redhat.com/mailman/listinfo/dm-devel
>>
>
>
[-- Attachment #1.2: Type: text/html, Size: 20471 bytes --]
[-- Attachment #2: Type: text/plain, Size: 0 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
* RE: rdac path failure - Sun 6140
2009-08-13 20:34 ` Stewart Smith
@ 2009-08-14 0:05 ` Moger, Babu
2009-08-14 15:58 ` Stewart Smith
0 siblings, 1 reply; 9+ messages in thread
From: Moger, Babu @ 2009-08-14 0:05 UTC (permalink / raw)
To: device-mapper development; +Cc: Stewart Smith
[-- Attachment #1.1: Type: text/plain, Size: 17359 bytes --]
Stew,
I don't see much information about this failure in the logs. Right now device handlers don't provide much information on failures. We are working on to add some more debug levels. I am attaching my draft code (scsi_dh_rdac.c) here. Please use this only for your testing. It is not been approved/reviewed yet. I still need to submit this one to community for approval. The code is attached. Please replace this file with scsi_dh_rdac.c in the directory /driver/scsi/device_handlers and rebuild the kernel. This should give more information from the target point of view. Please send me the /var/log/messages file after the failure. Let see if we can get more information..
Thanks
Babu Moger
________________________________
From: dm-devel-bounces@redhat.com [mailto:dm-devel-bounces@redhat.com] On Behalf Of Stewart Smith
Sent: Thursday, August 13, 2009 3:35 PM
To: device-mapper development
Subject: Re: [dm-devel] rdac path failure - Sun 6140
Same sequence of events, with multipathd -v3
Aug 13 16:28:48.627 kernel: device-mapper: multipath: Failing path 8:208.
Aug 13 16:28:48.000 multipathd: vol1: rr_weight = 2 (LUN setting)
Aug 13 16:28:48.000 multipathd: vol1: pgfailback = -2 (controller setting)
Aug 13 16:28:48.000 multipathd: pg_timeout = NONE (internal default)
Aug 13 16:28:48.000 multipathd: 8:208: mark as failed
Aug 13 16:28:48.000 multipathd: uevent 'change' from '/devices/virtual/block/dm-1'
Aug 13 16:28:48.000 multipathd: UDEV_LOG=3
Aug 13 16:28:48.000 multipathd: ACTION=change
Aug 13 16:28:48.000 multipathd: DEVPATH=/devices/virtual/block/dm-1
Aug 13 16:28:48.000 multipathd: SUBSYSTEM=block
Aug 13 16:28:48.000 multipathd: DM_TARGET=multipath
Aug 13 16:28:48.000 multipathd: DM_ACTION=PATH_FAILED
Aug 13 16:28:48.000 multipathd: DM_SEQNUM=1
Aug 13 16:28:48.000 multipathd: DM_PATH=8:208
Aug 13 16:28:48.000 multipathd: DM_NR_VALID_PATHS=3
Aug 13 16:28:48.000 multipathd: DM_NAME=vol1
Aug 13 16:28:48.000 multipathd: DM_UUID=mpath-3600a0b800048335200001e5d48b68a9b
Aug 13 16:28:48.000 multipathd: MAJOR=253
Aug 13 16:28:48.000 multipathd: MINOR=1
Aug 13 16:28:48.000 multipathd: DEVTYPE=disk
Aug 13 16:28:48.000 multipathd: SEQNUM=1738
Aug 13 16:28:48.000 multipathd: UDEVD_EVENT=1
Aug 13 16:28:48.000 multipathd: DEVNAME=/dev/dm-1
Aug 13 16:28:50.000 multipathd: 8:208: reinstated
Aug 13 16:28:50.000 multipathd: vol1: remaining active paths: 4
Aug 13 16:28:50.000 multipathd: sdj: rdac prio = 3
Aug 13 16:28:50.000 multipathd: sdn: rdac prio = 3
Aug 13 16:28:50.000 multipathd: sdb: rdac prio = 0
Aug 13 16:28:50.000 multipathd: sdd: rdac prio = 0
Aug 13 16:28:50.763 kernel: device-mapper: multipath: Failing path 8:208.
Aug 13 16:28:50.000 multipathd: uevent 'change' from '/devices/virtual/block/dm-1'
Aug 13 16:28:50.000 multipathd: UDEV_LOG=3
Aug 13 16:28:50.000 multipathd: ACTION=change
Aug 13 16:28:50.000 multipathd: DEVPATH=/devices/virtual/block/dm-1
Aug 13 16:28:50.000 multipathd: SUBSYSTEM=block
Aug 13 16:28:50.000 multipathd: DM_TARGET=multipath
Aug 13 16:28:50.000 multipathd: DM_ACTION=PATH_REINSTATED
Aug 13 16:28:50.000 multipathd: DM_SEQNUM=2
Aug 13 16:28:50.000 multipathd: DM_PATH=8:208
Aug 13 16:28:50.000 multipathd: DM_NR_VALID_PATHS=4
Aug 13 16:28:50.000 multipathd: DM_NAME=vol1
Aug 13 16:28:50.000 multipathd: DM_UUID=mpath-3600a0b800048335200001e5d48b68a9b
Aug 13 16:28:50.000 multipathd: MAJOR=253
Aug 13 16:28:50.000 multipathd: MINOR=1
Aug 13 16:28:50.000 multipathd: DEVTYPE=diskAug 13 16:28:50.000 multipathd: SEQNUM=1739Aug 13 16:28:50.000 multipathd: UDEVD_EVENT=1
Aug 13 16:28:50.000 multipathd: vol1: rr_weight = 2 (LUN setting)
Aug 13 16:28:50.000 multipathd: vol1: pgfailback = -2 (controller setting)
Aug 13 16:28:50.000 multipathd: pg_timeout = NONE (internal default)
Aug 13 16:28:50.000 multipathd: 8:208: mark as failed
Aug 13 16:28:50.000 multipathd: vol1: remaining active paths: 3
Aug 13 16:28:50.000 multipathd: vol1: rr_weight = 2 (LUN setting)
Aug 13 16:28:50.000 multipathd: vol1: pgfailback = -2 (controller setting)
Aug 13 16:28:50.000 multipathd: uevent 'change' from '/devices/virtual/block/dm-1'
Aug 13 16:28:50.000 multipathd: UDEV_LOG=3
Aug 13 16:28:50.000 multipathd: ACTION=change
Aug 13 16:28:50.000 multipathd: DEVPATH=/devices/virtual/block/dm-1
Aug 13 16:28:50.000 multipathd: SUBSYSTEM=block
Aug 13 16:28:50.000 multipathd: DM_TARGET=multipath
Aug 13 16:28:50.000 multipathd: DM_ACTION=PATH_FAILED
Aug 13 16:28:50.000 multipathd: DM_SEQNUM=3
Aug 13 16:28:50.000 multipathd: DM_PATH=8:208
Aug 13 16:28:50.000 multipathd: DM_NR_VALID_PATHS=3
Aug 13 16:28:50.000 multipathd: DM_NAME=vol1
Aug 13 16:28:50.000 multipathd: DM_UUID=mpath-3600a0b800048335200001e5d48b68a9b
Aug 13 16:28:50.000 multipathd: MAJOR=253
Aug 13 16:28:50.000 multipathd: MINOR=1
Aug 13 16:28:50.000 multipathd: DEVTYPE=disk
Aug 13 16:28:50.000 multipathd: SEQNUM=1740
Aug 13 16:28:50.000 multipathd: UDEVD_EVENT=1
Aug 13 16:28:50.000 multipathd: DEVNAME=/dev/dm-1
Aug 13 16:29:00.000 multipathd: 8:208: reinstated
Aug 13 16:29:00.000 multipathd: vol1: remaining active paths: 4
Aug 13 16:29:00.000 multipathd: sdj: rdac prio = 3
Aug 13 16:29:00.000 multipathd: sdn: rdac prio = 3
Aug 13 16:29:00.000 multipathd: sdb: rdac prio = 0
Aug 13 16:29:00.000 multipathd: sdd: rdac prio = 0
Aug 13 16:29:00.000 multipathd: vol1: rr_weight = 2 (LUN setting)
Aug 13 16:29:00.000 multipathd: vol1: pgfailback = -2 (controller setting)
Aug 13 16:29:00.000 multipathd: uevent 'change' from '/devices/virtual/block/dm-1'
Aug 13 16:29:00.000 multipathd: UDEV_LOG=3
Aug 13 16:29:00.000 multipathd: ACTION=change
Aug 13 16:29:00.000 multipathd: DEVPATH=/devices/virtual/block/dm-1
Aug 13 16:29:00.000 multipathd: SUBSYSTEM=block
Aug 13 16:29:00.000 multipathd: DM_TARGET=multipath
Aug 13 16:29:00.000 multipathd: DM_ACTION=PATH_REINSTATED
Aug 13 16:29:00.000 multipathd: DM_SEQNUM=4
Aug 13 16:29:00.000 multipathd: DM_PATH=8:208
Aug 13 16:29:00.000 multipathd: DM_NR_VALID_PATHS=4
Aug 13 16:29:00.000 multipathd: DM_NAME=vol1
Aug 13 16:29:00.000 multipathd: DM_UUID=mpath-3600a0b800048335200001e5d48b68a9b
Aug 13 16:29:00.000 multipathd: MAJOR=253
Aug 13 16:29:00.000 multipathd: MINOR=1
Aug 13 16:29:00.000 multipathd: DEVTYPE=disk
Aug 13 16:29:00.000 multipathd: SEQNUM=1741
Aug 13 16:29:00.000 multipathd: UDEVD_EVENT=1
Aug 13 16:29:00.000 multipathd: DEVNAME=/dev/dm-1
Aug 13 16:29:02.753 kernel: device-mapper: multipath: Failing path 8:208.
Aug 13 16:29:02.000 multipathd: vol1: rr_weight = 2 (LUN setting)
Aug 13 16:29:02.000 multipathd: vol1: pgfailback = -2 (controller setting)
Aug 13 16:29:02.000 multipathd: pg_timeout = NONE (internal default)
Aug 13 16:29:02.000 multipathd: 8:208: mark as failed
Aug 13 16:29:02.000 multipathd: uevent 'change' from '/devices/virtual/block/dm-1'
Aug 13 16:29:02.000 multipathd: UDEV_LOG=3
Aug 13 16:29:02.000 multipathd: ACTION=change
Aug 13 16:29:02.000 multipathd: DEVPATH=/devices/virtual/block/dm-1
Aug 13 16:29:02.000 multipathd: SUBSYSTEM=block
Aug 13 16:29:02.000 multipathd: DM_TARGET=multipath
Aug 13 16:29:02.000 multipathd: DM_ACTION=PATH_FAILED
Aug 13 16:29:02.000 multipathd: DM_SEQNUM=5
Aug 13 16:29:02.000 multipathd: DM_PATH=8:208
Aug 13 16:29:02.000 multipathd: DM_NR_VALID_PATHS=3
Aug 13 16:29:02.000 multipathd: DM_NAME=vol1
Aug 13 16:29:02.000 multipathd: DM_UUID=mpath-3600a0b800048335200001e5d48b68a9b
Aug 13 16:29:02.000 multipathd: MAJOR=253
Aug 13 16:29:02.000 multipathd: MINOR=1
Aug 13 16:29:02.000 multipathd: DEVTYPE=disk
Aug 13 16:29:02.000 multipathd: SEQNUM=1742
Aug 13 16:29:02.000 multipathd: UDEVD_EVENT=1
Aug 13 16:29:02.000 multipathd: DEVNAME=/dev/dm-1
Aug 13 16:29:10.000 multipathd: 8:208: reinstated
Aug 13 16:29:10.000 multipathd: vol1: remaining active paths: 4
Aug 13 16:29:10.000 multipathd: sdj: rdac prio = 3
Aug 13 16:29:10.000 multipathd: sdn: rdac prio = 3
Aug 13 16:29:10.000 multipathd: sdb: rdac prio = 0
Aug 13 16:29:10.000 multipathd: sdd: rdac prio = 0
Aug 13 16:29:10.000 multipathd: vol1: rr_weight = 2 (LUN setting)
Aug 13 16:29:10.000 multipathd: vol1: pgfailback = -2 (controller setting)
Aug 13 16:29:10.000 multipathd: uevent 'change' from '/devices/virtual/block/dm-1'
Aug 13 16:29:10.000 multipathd: UDEV_LOG=3
Aug 13 16:29:10.000 multipathd: ACTION=change
Aug 13 16:29:10.000 multipathd: DEVPATH=/devices/virtual/block/dm-1
Aug 13 16:29:10.000 multipathd: SUBSYSTEM=block
Aug 13 16:29:10.000 multipathd: DM_TARGET=multipath
Aug 13 16:29:10.000 multipathd: DM_ACTION=PATH_REINSTATED
Aug 13 16:29:10.000 multipathd: DM_SEQNUM=6
Aug 13 16:29:10.000 multipathd: DM_PATH=8:208
Aug 13 16:29:10.000 multipathd: DM_NR_VALID_PATHS=4
Aug 13 16:29:10.000 multipathd: DM_NAME=vol1
Aug 13 16:29:10.000 multipathd: DM_UUID=mpath-3600a0b800048335200001e5d48b68a9b
Aug 13 16:29:10.000 multipathd: MAJOR=253
Aug 13 16:29:10.000 multipathd: MINOR=1
Aug 13 16:29:10.000 multipathd: DEVTYPE=disk
Aug 13 16:29:10.000 multipathd: SEQNUM=1743
Aug 13 16:29:10.000 multipathd: UDEVD_EVENT=1
Aug 13 16:29:10.000 multipathd: DEVNAME=/dev/dm-1
On Thu, Aug 13, 2009 at 1:27 PM, Stewart Smith <stew@cleepdar.com<mailto:stew@cleepdar.com>> wrote:
after a fresh, multipath -F and start of multipathd with -v 2 I see the following messages.
After starting multipathd I mounted /dev/mapper/vol1 and generated some simple I/O to it using dd
Aug 13 16:23:14.888 localhost kernel: device-mapper: multipath: Failing path 8:208.
Aug 13 16:23:14.000 localhost multipathd: 8:208: mark as failed
Aug 13 16:23:16.000 localhost multipathd: 8:208: reinstated
Aug 13 16:23:30.462 localhost kernel: device-mapper: multipath: Failing path 8:208.
Aug 13 16:23:30.000 localhost multipathd: 8:208: mark as failed
Aug 13 16:23:39.000 localhost multipathd: 8:208: reinstated
Aug 13 16:23:46.430 localhost kernel: device-mapper: multipath: Failing path 8:208.
Aug 13 16:23:46.000 localhost multipathd: 8:208: mark as failed
Aug 13 16:23:51.041 localhost kernel: device-mapper: multipath: Failing path 8:208.
Aug 13 16:23:51.000 localhost multipathd: 8:208: mark as failed
Aug 13 16:23:59.000 localhost multipathd: 8:208: reinstated
Aug 13 16:24:06.465 localhost kernel: device-mapper: multipath: Failing path 8:208.
Aug 13 16:24:06.000 localhost multipathd: 8:208: mark as failed
Aug 13 16:24:09.000 localhost multipathd: 8:208: reinstated
Thanks,
--
Stew
On Thu, Aug 13, 2009 at 12:42 PM, Moger, Babu <Babu.Moger@lsi.com<mailto:Babu.Moger@lsi.com>> wrote:
Do you have /var/log/messages file for this problem?
Thanks
Babu Moger
> -----Original Message-----
> From: dm-devel-bounces@redhat.com<mailto:dm-devel-bounces@redhat.com> [mailto:dm-devel-bounces@redhat.com<mailto:dm-devel-bounces@redhat.com>] On
> Behalf Of Stewart Smith
> Sent: Thursday, August 13, 2009 1:51 PM
> To: dm-devel@redhat.com<mailto:dm-devel@redhat.com>
> Subject: [dm-devel] rdac path failure - Sun 6140
>
> Hello All,
>
> I am seeing many of these messages when my Sun 6140 array is under heavy
> I/O
> device-mapper: multipath: Failing path 8:208.
> device-mapper: multipath: Failing path 8:208.
> device-mapper: multipath: Failing path 8:208.
> device-mapper: multipath: Failing path 8:208.
> device-mapper: multipath: Failing path 8:208.
>
>
> I am running a Fedora 10 server, with two fiber connections to two
> different switches. Both controllers on the 6140 have one connection
> to each switch as well. The end result is that I see four paths to
> each LUN.
>
> When the volume is mounted and under significant load I see the
> messages above every few seconds. They seem to appear every
> "no_path_retry" seconds.
>
> The 6140 controller firmware is up to date at version 07.50.08.10 and
> I have installed the latest firmware for my Emulex LPe11002 cards. I
> have reproduced the problem using both Cisco MDS and Brocade fiber
> channel switches as well.
>
> Using CAM, I have set the initiator Host Type to "Linux" at the
> moment. I have tried other options as well without success.
>
> I have NOT installed the RDAC drivers from either Sun or LSI -
> primarily because they do not seem to build on my Fedora 10 kernel.
>
> Any ideas would be greatly appreciated!!!
>
> configs and debugging multipathd output is below.
>
>
>
>
>
> Kernel: 2.6.27.24-170.2.68.fc10.x86_64
>
> # multipath -lll
> vol1 (3600a0b800048335200001e5d48b68a9b) dm-1 SUN,CSM200_R
> [size=12T][features=1 queue_if_no_path][hwhandler=1 rdac][rw]
> \_ round-robin 0 [prio=6][active]
> \_ 5:0:1:2 sdj 8:144 [active][ready]
> \_ 2:0:1:2 sdn 8:208 [active][ready]
> \_ round-robin 0 [prio=0][enabled]
> \_ 2:0:0:2 sdb 8:16 [active][ghost]
> \_ 5:0:0:2 sdd 8:48 [active][ghost]
>
>
> # cat /etc/multipath.conf
>
> blacklist {
> devnode "^sd[a-z][[0-9]*]"
> devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
> devnode "^hd[a-z][0-9]*"
> devnode "^cciss!c[0-9]d[0-9](p[0-9]*)*"
> }
>
> defaults {
> udev_dir /dev
> polling_interval 10
> selector "round-robin 0"
> path_grouping_policy multibus
> getuid_callout "/sbin/scsi_id --whitelisted /dev/%n"
> prio alua
> path_checker readsector0
> rr_min_io 100
> max_fds 8192
> rr_weight priorities
> failback immediate
> no_path_retry fail
> user_friendly_names yes
> }
> devices {
> device {
> vendor "SUN"
> product "CSM200_R"
> product_blacklist "Universal Xport"
> getuid_callout "/sbin/scsi_id --whitelisted
> /dev/%n"
> features "0"
> hardware_handler "1 rdac"
> path_selector "round-robin 0"
> path_grouping_policy group_by_prio
> failback immediate
> rr_weight uniform
> no_path_retry queue
> rr_min_io 1000
> path_checker rdac
> prio rdac
> }
> }
>
> multipaths {
> multipath {
> wwid 3600a0b800048335200001e5d48b68a9b
> alias vol1
> rr_weight priorities
> no_path_retry 5
> rr_min_io 100
> }
> }
>
>
>
> # multipathd -d v3
>
>
> Aug 13 14:48:53 | sdb: ownership set to vol1
> Aug 13 14:48:53 | sdb: not found in pathvec
> Aug 13 14:48:53 | sdb: mask = 0xc
> Aug 13 14:48:53 | sdb: path checker = rdac (controller setting)
> Aug 13 14:48:53 | sdb: state = 4
> Aug 13 14:48:53 | sdb: rdac prio = 0
> Aug 13 14:48:53 | sdd: ownership set to vol1
> Aug 13 14:48:53 | sdd: not found in pathvec
> Aug 13 14:48:53 | sdd: mask = 0xc
> Aug 13 14:48:53 | sdd: path checker = rdac (controller setting)
> Aug 13 14:48:53 | sdd: state = 4
> Aug 13 14:48:53 | sdd: rdac prio = 0
> Aug 13 14:48:53 | sdj: ownership set to vol1
> Aug 13 14:48:53 | sdj: not found in pathvec
> Aug 13 14:48:53 | sdj: mask = 0xc
> Aug 13 14:48:53 | sdj: path checker = rdac (controller setting)
> Aug 13 14:48:53 | sdj: state = 2
> Aug 13 14:48:53 | sdj: rdac prio = 3
> Aug 13 14:48:53 | sdn: ownership set to vol1
> Aug 13 14:48:53 | sdn: not found in pathvec
> Aug 13 14:48:53 | sdn: mask = 0xc
> Aug 13 14:48:53 | sdn: path checker = rdac (controller setting)
> Aug 13 14:48:53 | sdn: state = 2
> Aug 13 14:48:53 | sdn: rdac prio = 3
> Aug 13 14:48:53 | vol1: pgfailback = -2 (controller setting)
> Aug 13 14:48:53 | vol1: pgpolicy = group_by_prio (controller setting)
> Aug 13 14:48:53 | vol1: selector = round-robin 0 (controller setting)
> Aug 13 14:48:53 | vol1: features = 0 (controller setting)
> Aug 13 14:48:53 | vol1: hwhandler = 1 rdac (controller setting)
> Aug 13 14:48:53 | vol1: rr_weight = 2 (LUN setting)
> Aug 13 14:48:53 | vol1: minio = 100 (LUN setting)
> Aug 13 14:48:53 | vol1: no_path_retry = 5 (multipath setting)
> Aug 13 14:48:53 | pg_timeout = NONE (internal default)
> Aug 13 14:48:53 | vol1: set ACT_CREATE (map does not exist)
> create: vol1 (3600a0b800048335200001e5d48b68a9b) n/a SUN,CSM200_R
> [size=12T][features=0][hwhandler=1 rdac][n/a]
> \_ round-robin 0 [prio=6][undef]
> \_ 5:0:1:2 sdj 8:144 [undef][ready]
> \_ 2:0:1:2 sdn 8:208 [undef][ready]
> \_ round-robin 0 [prio=0][undef]
> \_ 2:0:0:2 sdb 8:16 [undef][ghost]
> \_ 5:0:0:2 sdd 8:48 [undef][ghost]
>
> --
> dm-devel mailing list
> dm-devel@redhat.com<mailto:dm-devel@redhat.com>
> https://www.redhat.com/mailman/listinfo/dm-devel
--
dm-devel mailing list
dm-devel@redhat.com<mailto:dm-devel@redhat.com>
https://www.redhat.com/mailman/listinfo/dm-devel
[-- Attachment #1.2: Type: text/html, Size: 53333 bytes --]
[-- Attachment #2: scsi_dh_rdac.c --]
[-- Type: text/plain, Size: 19308 bytes --]
/*
* Engenio/LSI RDAC SCSI Device Handler
*
* Copyright (C) 2005 Mike Christie. All rights reserved.
* Copyright (C) Chandra Seetharaman, IBM Corp. 2007
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation; either version 2 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program; if not, write to the Free Software
* Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
*
*/
#include <scsi/scsi.h>
#include <scsi/scsi_eh.h>
#include <scsi/scsi_dh.h>
#define RDAC_NAME "rdac"
#define RDAC_RETRY_COUNT 5
/*
* LSI mode page stuff
*
* These struct definitions and the forming of the
* mode page were taken from the LSI RDAC 2.4 GPL'd
* driver, and then converted to Linux conventions.
*/
#define RDAC_QUIESCENCE_TIME 20;
/*
* Page Codes
*/
#define RDAC_PAGE_CODE_REDUNDANT_CONTROLLER 0x2c
/*
* Controller modes definitions
*/
#define RDAC_MODE_TRANSFER_SPECIFIED_LUNS 0x02
/*
* RDAC Options field
*/
#define RDAC_FORCED_QUIESENCE 0x02
#define RDAC_TIMEOUT (60 * HZ)
#define RDAC_RETRIES 3
struct rdac_mode_6_hdr {
u8 data_len;
u8 medium_type;
u8 device_params;
u8 block_desc_len;
};
struct rdac_mode_10_hdr {
u16 data_len;
u8 medium_type;
u8 device_params;
u16 reserved;
u16 block_desc_len;
};
struct rdac_mode_common {
u8 controller_serial[16];
u8 alt_controller_serial[16];
u8 rdac_mode[2];
u8 alt_rdac_mode[2];
u8 quiescence_timeout;
u8 rdac_options;
};
struct rdac_pg_legacy {
struct rdac_mode_6_hdr hdr;
u8 page_code;
u8 page_len;
struct rdac_mode_common common;
#define MODE6_MAX_LUN 32
u8 lun_table[MODE6_MAX_LUN];
u8 reserved2[32];
u8 reserved3;
u8 reserved4;
};
struct rdac_pg_expanded {
struct rdac_mode_10_hdr hdr;
u8 page_code;
u8 subpage_code;
u8 page_len[2];
struct rdac_mode_common common;
u8 lun_table[256];
u8 reserved3;
u8 reserved4;
};
struct c9_inquiry {
u8 peripheral_info;
u8 page_code; /* 0xC9 */
u8 reserved1;
u8 page_len;
u8 page_id[4]; /* "vace" */
u8 avte_cvp;
u8 path_prio;
u8 reserved2[38];
};
#define SUBSYS_ID_LEN 16
#define SLOT_ID_LEN 2
struct c4_inquiry {
u8 peripheral_info;
u8 page_code; /* 0xC4 */
u8 reserved1;
u8 page_len;
u8 page_id[4]; /* "subs" */
u8 subsys_id[SUBSYS_ID_LEN];
u8 revision[4];
u8 slot_id[SLOT_ID_LEN];
u8 reserved[2];
};
struct rdac_controller {
u8 subsys_id[SUBSYS_ID_LEN];
u8 slot_id[SLOT_ID_LEN];
int use_ms10;
struct kref kref;
struct list_head node; /* list of all controllers */
union {
struct rdac_pg_legacy legacy;
struct rdac_pg_expanded expanded;
} mode_select;
u8 index;
u8 array_name[31];
};
struct c8_inquiry {
u8 peripheral_info;
u8 page_code; /* 0xC8 */
u8 reserved1;
u8 page_len;
u8 page_id[4]; /* "edid" */
u8 reserved2[3];
u8 vol_uniq_id_len;
u8 vol_uniq_id[16];
u8 vol_user_label_len;
u8 vol_user_label[60];
u8 array_uniq_id_len;
u8 array_unique_id[16];
u8 array_user_label_len;
u8 array_user_label[60];
u8 lun[8];
};
struct c2_inquiry {
u8 peripheral_info;
u8 page_code; /* 0xC2 */
u8 reserved1;
u8 page_len;
u8 page_id[4]; /* "swr4" */
u8 sw_version[3];
u8 sw_date[3];
u8 features_enabled;
u8 max_lun_supported;
u8 partitions[239]; /* Total allocation length should be 0xFF */
};
struct rdac_dh_data {
struct rdac_controller *ctlr;
#define UNINITIALIZED_LUN (1 << 8)
unsigned lun;
#define RDAC_STATE_ACTIVE 0
#define RDAC_STATE_PASSIVE 1
unsigned char state;
#define RDAC_LUN_UNOWNED 0
#define RDAC_LUN_OWNED 1
#define RDAC_LUN_AVT 2
char lun_state;
unsigned char sense[SCSI_SENSE_BUFFERSIZE];
union {
struct c2_inquiry c2;
struct c4_inquiry c4;
struct c8_inquiry c8;
struct c9_inquiry c9;
} inq;
};
static const char *lun_state[] =
{
"unowned",
"owned",
"owned (AVT mode)",
};
static LIST_HEAD(ctlr_list);
static DEFINE_SPINLOCK(list_lock);
/*
* module parameter to enable rdac debug logging.
* 2 bits for each type of logging, only two types defined for now
* Can be enhanced if required at later point
*/
static int rdac_logging = 0xF;
module_param(rdac_logging, int, 0);
MODULE_PARM_DESC(rdac_logging, "A bit mask of rdac logging levels, "
"Default is 1 - failover logging enabled, "
"set it to 0xF to enable all the logs");
#define RDAC_LOG_FAILOVER 0
#define RDAC_LOG_SENSE 2
#define RDAC_LOG_BITS 2
#define RDAC_LOG_LEVEL(SHIFT) \
((rdac_logging >> (SHIFT)) & ((1 << (RDAC_LOG_BITS)) - 1))
#define RDAC_DEBUG(SHIFT, sdev, f, arg...) \
do { \
if (unlikely(RDAC_LOG_LEVEL(SHIFT))) \
sdev_printk(KERN_INFO, sdev, RDAC_NAME ": " f "\n", ## arg); \
} while (0);
static inline struct rdac_dh_data *get_rdac_data(struct scsi_device *sdev)
{
struct scsi_dh_data *scsi_dh_data = sdev->scsi_dh_data;
BUG_ON(scsi_dh_data == NULL);
return ((struct rdac_dh_data *) scsi_dh_data->buf);
}
static struct request *get_rdac_req(struct scsi_device *sdev,
void *buffer, unsigned buflen, int rw)
{
struct request *rq;
struct request_queue *q = sdev->request_queue;
rq = blk_get_request(q, rw, GFP_NOIO);
if (!rq) {
sdev_printk(KERN_INFO, sdev,
"get_rdac_req: blk_get_request failed.\n");
return NULL;
}
if (buflen && blk_rq_map_kern(q, rq, buffer, buflen, GFP_NOIO)) {
blk_put_request(rq);
sdev_printk(KERN_INFO, sdev,
"get_rdac_req: blk_rq_map_kern failed.\n");
return NULL;
}
rq->cmd_type = REQ_TYPE_BLOCK_PC;
rq->cmd_flags |= REQ_FAILFAST_DEV | REQ_FAILFAST_TRANSPORT |
REQ_FAILFAST_DRIVER;
rq->retries = RDAC_RETRIES;
rq->timeout = RDAC_TIMEOUT;
return rq;
}
static struct request *rdac_failover_get(struct scsi_device *sdev,
struct rdac_dh_data *h)
{
struct request *rq;
struct rdac_mode_common *common;
unsigned data_size;
if (h->ctlr->use_ms10) {
struct rdac_pg_expanded *rdac_pg;
data_size = sizeof(struct rdac_pg_expanded);
rdac_pg = &h->ctlr->mode_select.expanded;
memset(rdac_pg, 0, data_size);
common = &rdac_pg->common;
rdac_pg->page_code = RDAC_PAGE_CODE_REDUNDANT_CONTROLLER + 0x40;
rdac_pg->subpage_code = 0x1;
rdac_pg->page_len[0] = 0x01;
rdac_pg->page_len[1] = 0x28;
rdac_pg->lun_table[h->lun] = 0x81;
} else {
struct rdac_pg_legacy *rdac_pg;
data_size = sizeof(struct rdac_pg_legacy);
rdac_pg = &h->ctlr->mode_select.legacy;
memset(rdac_pg, 0, data_size);
common = &rdac_pg->common;
rdac_pg->page_code = RDAC_PAGE_CODE_REDUNDANT_CONTROLLER;
rdac_pg->page_len = 0x68;
rdac_pg->lun_table[h->lun] = 0x81;
}
common->rdac_mode[1] = RDAC_MODE_TRANSFER_SPECIFIED_LUNS;
common->quiescence_timeout = RDAC_QUIESCENCE_TIME;
common->rdac_options = RDAC_FORCED_QUIESENCE;
/* get request for block layer packet command */
rq = get_rdac_req(sdev, &h->ctlr->mode_select, data_size, WRITE);
if (!rq)
return NULL;
/* Prepare the command. */
if (h->ctlr->use_ms10) {
rq->cmd[0] = MODE_SELECT_10;
rq->cmd[7] = data_size >> 8;
rq->cmd[8] = data_size & 0xff;
} else {
rq->cmd[0] = MODE_SELECT;
rq->cmd[4] = data_size;
}
rq->cmd_len = COMMAND_SIZE(rq->cmd[0]);
rq->sense = h->sense;
memset(rq->sense, 0, SCSI_SENSE_BUFFERSIZE);
rq->sense_len = 0;
return rq;
}
static void release_controller(struct kref *kref)
{
struct rdac_controller *ctlr;
ctlr = container_of(kref, struct rdac_controller, kref);
spin_lock(&list_lock);
list_del(&ctlr->node);
spin_unlock(&list_lock);
kfree(ctlr);
}
static struct rdac_controller *get_controller(u8 *subsys_id, u8 *slot_id)
{
struct rdac_controller *ctlr, *tmp;
spin_lock(&list_lock);
list_for_each_entry(tmp, &ctlr_list, node) {
if ((memcmp(tmp->subsys_id, subsys_id, SUBSYS_ID_LEN) == 0) &&
(memcmp(tmp->slot_id, slot_id, SLOT_ID_LEN) == 0)) {
kref_get(&tmp->kref);
spin_unlock(&list_lock);
return tmp;
}
}
ctlr = kmalloc(sizeof(*ctlr), GFP_ATOMIC);
if (!ctlr)
goto done;
/* initialize fields of controller */
memcpy(ctlr->subsys_id, subsys_id, SUBSYS_ID_LEN);
memcpy(ctlr->slot_id, slot_id, SLOT_ID_LEN);
/* update the controller index */
if (slot_id[1] == 0x31)
ctlr->index = 0;
else
ctlr->index = 1;
kref_init(&ctlr->kref);
ctlr->use_ms10 = -1;
list_add(&ctlr->node, &ctlr_list);
done:
spin_unlock(&list_lock);
return ctlr;
}
static int submit_inquiry(struct scsi_device *sdev, int page_code,
unsigned int len, struct rdac_dh_data *h)
{
struct request *rq;
struct request_queue *q = sdev->request_queue;
int err = SCSI_DH_RES_TEMP_UNAVAIL;
rq = get_rdac_req(sdev, &h->inq, len, READ);
if (!rq)
goto done;
/* Prepare the command. */
rq->cmd[0] = INQUIRY;
rq->cmd[1] = 1;
rq->cmd[2] = page_code;
rq->cmd[4] = len;
rq->cmd_len = COMMAND_SIZE(INQUIRY);
rq->sense = h->sense;
memset(rq->sense, 0, SCSI_SENSE_BUFFERSIZE);
rq->sense_len = 0;
err = blk_execute_rq(q, NULL, rq, 1);
if (err == -EIO)
err = SCSI_DH_IO;
blk_put_request(rq);
done:
return err;
}
static int get_lun(struct scsi_device *sdev, struct rdac_dh_data *h)
{
int err;
struct c8_inquiry *inqp;
err = submit_inquiry(sdev, 0xC8, sizeof(struct c8_inquiry), h);
if (err == SCSI_DH_OK) {
inqp = &h->inq.c8;
if (inqp->page_code != 0xc8)
return SCSI_DH_NOSYS;
if (inqp->page_id[0] != 'e' || inqp->page_id[1] != 'd' ||
inqp->page_id[2] != 'i' || inqp->page_id[3] != 'd')
return SCSI_DH_NOSYS;
h->lun = inqp->lun[7]; /* Uses only the last byte */
}
return err;
}
static int get_array_name(struct scsi_device *sdev, struct rdac_dh_data *h)
{
int err, i;
struct c8_inquiry *inqp;
err = submit_inquiry(sdev, 0xC8, sizeof(struct c8_inquiry), h);
if (err == SCSI_DH_OK) {
inqp = &h->inq.c8;
if (inqp->page_code != 0xc8)
return SCSI_DH_NOSYS;
for(i=0; i<30; ++i){
h->ctlr->array_name[i] = inqp->array_user_label[(2*i)+1];
}
h->ctlr->array_name[30] = '\0';
}
return err;
}
static int check_ownership(struct scsi_device *sdev, struct rdac_dh_data *h)
{
int err;
struct c9_inquiry *inqp;
h->lun_state = RDAC_LUN_UNOWNED;
h->state = RDAC_STATE_ACTIVE;
err = submit_inquiry(sdev, 0xC9, sizeof(struct c9_inquiry), h);
if (err == SCSI_DH_OK) {
inqp = &h->inq.c9;
if ((inqp->avte_cvp >> 7) == 0x1) {
/* LUN in AVT mode */
sdev_printk(KERN_NOTICE, sdev,
"%s: AVT mode detected\n",
RDAC_NAME);
h->lun_state = RDAC_LUN_AVT;
} else if ((inqp->avte_cvp & 0x1) != 0) {
/* LUN was owned by the controller */
h->lun_state = RDAC_LUN_OWNED;
}
}
if (h->lun_state == RDAC_LUN_UNOWNED)
h->state = RDAC_STATE_PASSIVE;
return err;
}
static int initialize_controller(struct scsi_device *sdev,
struct rdac_dh_data *h)
{
int err;
struct c4_inquiry *inqp;
err = submit_inquiry(sdev, 0xC4, sizeof(struct c4_inquiry), h);
if (err == SCSI_DH_OK) {
inqp = &h->inq.c4;
h->ctlr = get_controller(inqp->subsys_id, inqp->slot_id);
if (!h->ctlr)
err = SCSI_DH_RES_TEMP_UNAVAIL;
}
return err;
}
static int set_mode_select(struct scsi_device *sdev, struct rdac_dh_data *h)
{
int err;
struct c2_inquiry *inqp;
err = submit_inquiry(sdev, 0xC2, sizeof(struct c2_inquiry), h);
if (err == SCSI_DH_OK) {
inqp = &h->inq.c2;
/*
* If more than MODE6_MAX_LUN luns are supported, use
* mode select 10
*/
if (inqp->max_lun_supported >= MODE6_MAX_LUN)
h->ctlr->use_ms10 = 1;
else
h->ctlr->use_ms10 = 0;
}
return err;
}
static int mode_select_handle_sense(struct scsi_device *sdev,
unsigned char *sensebuf)
{
struct scsi_sense_hdr sense_hdr;
int err = SCSI_DH_IO, ret;
struct rdac_dh_data *h = get_rdac_data(sdev);
ret = scsi_normalize_sense(sensebuf, SCSI_SENSE_BUFFERSIZE, &sense_hdr);
if (!ret)
goto done;
switch (sense_hdr.sense_key) {
case NO_SENSE:
case ABORTED_COMMAND:
case UNIT_ATTENTION:
err = SCSI_DH_RETRY;
break;
case NOT_READY:
if (sense_hdr.asc == 0x04 && sense_hdr.ascq == 0x01)
/* LUN Not Ready and is in the Process of Becoming
* Ready
*/
err = SCSI_DH_RETRY;
break;
case ILLEGAL_REQUEST:
if (sense_hdr.asc == 0x91 && sense_hdr.ascq == 0x36)
/*
* Command Lock contention
*/
err = SCSI_DH_RETRY;
break;
default:
break;
}
done:
RDAC_DEBUG(RDAC_LOG_FAILOVER, sdev, "array %s, ctlr %d, "
"MODE_SELECT returned with sense %02x/%02x/%02x",
(char *) h->ctlr->array_name, h->ctlr->index,
sense_hdr.sense_key, sense_hdr.asc, sense_hdr.ascq);
return err;
}
static int send_mode_select(struct scsi_device *sdev, struct rdac_dh_data *h)
{
struct request *rq;
struct request_queue *q = sdev->request_queue;
int err, retry_cnt = RDAC_RETRY_COUNT;
retry:
err = SCSI_DH_RES_TEMP_UNAVAIL;
rq = rdac_failover_get(sdev, h);
if (!rq)
goto done;
RDAC_DEBUG(RDAC_LOG_FAILOVER, sdev, "array %s, ctlr %d, "
"%s MODE_SELECT command",
(char *) h->ctlr->array_name, h->ctlr->index,
(retry_cnt == RDAC_RETRY_COUNT) ? "queueing" : "retrying");
err = blk_execute_rq(q, NULL, rq, 1);
blk_put_request(rq);
if (err != SCSI_DH_OK) {
err = mode_select_handle_sense(sdev, h->sense);
if (err == SCSI_DH_RETRY && retry_cnt--)
goto retry;
}
if (err == SCSI_DH_OK) {
h->state = RDAC_STATE_ACTIVE;
RDAC_DEBUG(RDAC_LOG_FAILOVER, sdev, "array %s, ctlr %d, "
"MODE_SELECT completed",
(char *) h->ctlr->array_name, h->ctlr->index);
}
done:
return err;
}
static int rdac_activate(struct scsi_device *sdev)
{
struct rdac_dh_data *h = get_rdac_data(sdev);
int err = SCSI_DH_OK;
err = check_ownership(sdev, h);
if (err != SCSI_DH_OK)
goto done;
if (h->ctlr->use_ms10 == -1) {
err = set_mode_select(sdev, h);
if (err != SCSI_DH_OK)
goto done;
}
if (h->lun_state == RDAC_LUN_UNOWNED)
err = send_mode_select(sdev, h);
done:
return err;
}
static int rdac_prep_fn(struct scsi_device *sdev, struct request *req)
{
struct rdac_dh_data *h = get_rdac_data(sdev);
int ret = BLKPREP_OK;
if (h->state != RDAC_STATE_ACTIVE) {
ret = BLKPREP_KILL;
req->cmd_flags |= REQ_QUIET;
}
return ret;
}
static int rdac_check_sense(struct scsi_device *sdev,
struct scsi_sense_hdr *sense_hdr)
{
struct rdac_dh_data *h = get_rdac_data(sdev);
RDAC_DEBUG(RDAC_LOG_SENSE, sdev, "array %s, ctlr %d, "
"command returned with sense %02x/%02x/%02x",
(char *) h->ctlr->array_name, h->ctlr->index,
sense_hdr->sense_key, sense_hdr->asc, sense_hdr->ascq);
switch (sense_hdr->sense_key) {
case NOT_READY:
if (sense_hdr->asc == 0x04 && sense_hdr->ascq == 0x01)
/* LUN Not Ready - Logical Unit Not Ready and is in
* the process of becoming ready
* Just retry.
*/
return ADD_TO_MLQUEUE;
if (sense_hdr->asc == 0x04 && sense_hdr->ascq == 0x81)
/* LUN Not Ready - Storage firmware incompatible
* Manual code synchonisation required.
*
* Nothing we can do here. Try to bypass the path.
*/
return SUCCESS;
if (sense_hdr->asc == 0x04 && sense_hdr->ascq == 0xA1)
/* LUN Not Ready - Quiescense in progress
*
* Just retry and wait.
*/
return ADD_TO_MLQUEUE;
if (sense_hdr->asc == 0xA1 && sense_hdr->ascq == 0x02)
/* LUN Not Ready - Quiescense in progress
* or has been achieved
* Just retry.
*/
return ADD_TO_MLQUEUE;
break;
case ILLEGAL_REQUEST:
if (sense_hdr->asc == 0x94 && sense_hdr->ascq == 0x01) {
/* Invalid Request - Current Logical Unit Ownership.
* Controller is not the current owner of the LUN,
* Fail the path, so that the other path be used.
*/
h->state = RDAC_STATE_PASSIVE;
return SUCCESS;
}
break;
case UNIT_ATTENTION:
if (sense_hdr->asc == 0x29 && sense_hdr->ascq == 0x00)
/*
* Power On, Reset, or Bus Device Reset, just retry.
*/
return ADD_TO_MLQUEUE;
if (sense_hdr->asc == 0x8b && sense_hdr->ascq == 0x02)
/*
* Quiescence in progress , just retry.
*/
return ADD_TO_MLQUEUE;
break;
}
/* success just means we do not care what scsi-ml does */
return SCSI_RETURN_NOT_HANDLED;
}
static const struct scsi_dh_devlist rdac_dev_list[] = {
{"IBM", "1722"},
{"IBM", "1724"},
{"IBM", "1726"},
{"IBM", "1742"},
{"IBM", "1814"},
{"IBM", "1815"},
{"IBM", "1818"},
{"IBM", "3526"},
{"SGI", "TP9400"},
{"SGI", "TP9500"},
{"SGI", "IS"},
{"STK", "OPENstorage D280"},
{"SUN", "CSM200_R"},
{"SUN", "LCSM100_F"},
{"DELL", "MD3000"},
{"DELL", "MD3000i"},
{"LSI", "INF-01-00"},
{"ENGENIO", "INF-01-00"},
{NULL, NULL},
};
static int rdac_bus_attach(struct scsi_device *sdev);
static void rdac_bus_detach(struct scsi_device *sdev);
static struct scsi_device_handler rdac_dh = {
.name = RDAC_NAME,
.module = THIS_MODULE,
.devlist = rdac_dev_list,
.prep_fn = rdac_prep_fn,
.check_sense = rdac_check_sense,
.attach = rdac_bus_attach,
.detach = rdac_bus_detach,
.activate = rdac_activate,
};
static int rdac_bus_attach(struct scsi_device *sdev)
{
struct scsi_dh_data *scsi_dh_data;
struct rdac_dh_data *h;
unsigned long flags;
int err;
scsi_dh_data = kzalloc(sizeof(struct scsi_device_handler *)
+ sizeof(*h) , GFP_KERNEL);
if (!scsi_dh_data) {
sdev_printk(KERN_ERR, sdev, "%s: Attach failed\n",
RDAC_NAME);
return 0;
}
scsi_dh_data->scsi_dh = &rdac_dh;
h = (struct rdac_dh_data *) scsi_dh_data->buf;
h->lun = UNINITIALIZED_LUN;
h->state = RDAC_STATE_ACTIVE;
err = get_lun(sdev, h);
if (err != SCSI_DH_OK)
goto failed;
err = check_ownership(sdev, h);
if (err != SCSI_DH_OK)
goto failed;
if (!h->ctlr) {
err = initialize_controller(sdev, h);
if (err != SCSI_DH_OK)
goto failed;
err = get_array_name(sdev, h);
if (err != SCSI_DH_OK)
goto failed;
}
if (!try_module_get(THIS_MODULE))
goto failed;
spin_lock_irqsave(sdev->request_queue->queue_lock, flags);
sdev->scsi_dh_data = scsi_dh_data;
spin_unlock_irqrestore(sdev->request_queue->queue_lock, flags);
sdev_printk(KERN_NOTICE, sdev,
"%s: LUN %d (%s)\n",
RDAC_NAME, h->lun, lun_state[(int)h->lun_state]);
return 0;
failed:
kfree(scsi_dh_data);
sdev_printk(KERN_ERR, sdev, "%s: not attached\n",
RDAC_NAME);
return -EINVAL;
}
static void rdac_bus_detach( struct scsi_device *sdev )
{
struct scsi_dh_data *scsi_dh_data;
struct rdac_dh_data *h;
unsigned long flags;
spin_lock_irqsave(sdev->request_queue->queue_lock, flags);
scsi_dh_data = sdev->scsi_dh_data;
sdev->scsi_dh_data = NULL;
spin_unlock_irqrestore(sdev->request_queue->queue_lock, flags);
h = (struct rdac_dh_data *) scsi_dh_data->buf;
if (h->ctlr)
kref_put(&h->ctlr->kref, release_controller);
kfree(scsi_dh_data);
module_put(THIS_MODULE);
sdev_printk(KERN_NOTICE, sdev, "%s: Detached\n", RDAC_NAME);
}
static int __init rdac_init(void)
{
int r;
r = scsi_register_device_handler(&rdac_dh);
if (r != 0)
printk(KERN_ERR "Failed to register scsi device handler.");
return r;
}
static void __exit rdac_exit(void)
{
scsi_unregister_device_handler(&rdac_dh);
}
module_init(rdac_init);
module_exit(rdac_exit);
MODULE_DESCRIPTION("Multipath LSI/Engenio RDAC driver");
MODULE_AUTHOR("Mike Christie, Chandra Seetharaman");
MODULE_LICENSE("GPL");
[-- Attachment #3: Type: text/plain, Size: 0 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: rdac path failure - Sun 6140
2009-08-14 0:05 ` Moger, Babu
@ 2009-08-14 15:58 ` Stewart Smith
2009-08-14 16:21 ` Moger, Babu
0 siblings, 1 reply; 9+ messages in thread
From: Stewart Smith @ 2009-08-14 15:58 UTC (permalink / raw)
To: device-mapper development
I will test this today.
Does everything look OK from a configuration standpoint? Should the
RDAC virtual HBA drivers from LSI be a requirement? I am not
currently using them.
Thank you,
--
Stew
On Thu, Aug 13, 2009 at 5:05 PM, Moger, Babu<Babu.Moger@lsi.com> wrote:
> Stew,
>
> I don’t see much information about this failure in the logs. Right now
> device handlers don’t provide much information on failures. We are working
> on to add some more debug levels. I am attaching my draft code
> (scsi_dh_rdac.c) here. Please use this only for your testing. It is not
> been approved/reviewed yet. I still need to submit this one to community for
> approval. The code is attached. Please replace this file with
> scsi_dh_rdac.c in the directory /driver/scsi/device_handlers and rebuild the
> kernel. This should give more information from the target point of view.
> Please send me the /var/log/messages file after the failure. Let see if we
> can get more information..
>
>
>
> Thanks
>
> Babu Moger
>
> ________________________________
>
> From: dm-devel-bounces@redhat.com [mailto:dm-devel-bounces@redhat.com] On
> Behalf Of Stewart Smith
> Sent: Thursday, August 13, 2009 3:35 PM
> To: device-mapper development
> Subject: Re: [dm-devel] rdac path failure - Sun 6140
>
>
>
>
>
> Same sequence of events, with multipathd -v3
>
>
>
> Aug 13 16:28:48.627 kernel: device-mapper: multipath: Failing path 8:208.
>
> Aug 13 16:28:48.000 multipathd: vol1: rr_weight = 2 (LUN setting)
>
> Aug 13 16:28:48.000 multipathd: vol1: pgfailback = -2 (controller setting)
>
> Aug 13 16:28:48.000 multipathd: pg_timeout = NONE (internal default)
>
> Aug 13 16:28:48.000 multipathd: 8:208: mark as failed
>
> Aug 13 16:28:48.000 multipathd: uevent 'change' from
> '/devices/virtual/block/dm-1'
>
> Aug 13 16:28:48.000 multipathd: UDEV_LOG=3
>
> Aug 13 16:28:48.000 multipathd: ACTION=change
>
> Aug 13 16:28:48.000 multipathd: DEVPATH=/devices/virtual/block/dm-1
>
> Aug 13 16:28:48.000 multipathd: SUBSYSTEM=block
>
> Aug 13 16:28:48.000 multipathd: DM_TARGET=multipath
>
> Aug 13 16:28:48.000 multipathd: DM_ACTION=PATH_FAILED
>
> Aug 13 16:28:48.000 multipathd: DM_SEQNUM=1
>
> Aug 13 16:28:48.000 multipathd: DM_PATH=8:208
>
> Aug 13 16:28:48.000 multipathd: DM_NR_VALID_PATHS=3
>
> Aug 13 16:28:48.000 multipathd: DM_NAME=vol1
>
> Aug 13 16:28:48.000 multipathd:
> DM_UUID=mpath-3600a0b800048335200001e5d48b68a9b
>
> Aug 13 16:28:48.000 multipathd: MAJOR=253
>
> Aug 13 16:28:48.000 multipathd: MINOR=1
>
> Aug 13 16:28:48.000 multipathd: DEVTYPE=disk
>
> Aug 13 16:28:48.000 multipathd: SEQNUM=1738
>
> Aug 13 16:28:48.000 multipathd: UDEVD_EVENT=1
>
> Aug 13 16:28:48.000 multipathd: DEVNAME=/dev/dm-1
>
> Aug 13 16:28:50.000 multipathd: 8:208: reinstated
>
> Aug 13 16:28:50.000 multipathd: vol1: remaining active paths: 4
>
> Aug 13 16:28:50.000 multipathd: sdj: rdac prio = 3
>
> Aug 13 16:28:50.000 multipathd: sdn: rdac prio = 3
>
> Aug 13 16:28:50.000 multipathd: sdb: rdac prio = 0
>
> Aug 13 16:28:50.000 multipathd: sdd: rdac prio = 0
>
> Aug 13 16:28:50.763 kernel: device-mapper: multipath: Failing path 8:208.
>
> Aug 13 16:28:50.000 multipathd: uevent 'change' from
> '/devices/virtual/block/dm-1'
>
> Aug 13 16:28:50.000 multipathd: UDEV_LOG=3
>
> Aug 13 16:28:50.000 multipathd: ACTION=change
>
> Aug 13 16:28:50.000 multipathd: DEVPATH=/devices/virtual/block/dm-1
>
> Aug 13 16:28:50.000 multipathd: SUBSYSTEM=block
>
> Aug 13 16:28:50.000 multipathd: DM_TARGET=multipath
>
> Aug 13 16:28:50.000 multipathd: DM_ACTION=PATH_REINSTATED
>
> Aug 13 16:28:50.000 multipathd: DM_SEQNUM=2
>
> Aug 13 16:28:50.000 multipathd: DM_PATH=8:208
>
> Aug 13 16:28:50.000 multipathd: DM_NR_VALID_PATHS=4
>
> Aug 13 16:28:50.000 multipathd: DM_NAME=vol1
>
> Aug 13 16:28:50.000 multipathd:
> DM_UUID=mpath-3600a0b800048335200001e5d48b68a9b
>
> Aug 13 16:28:50.000 multipathd: MAJOR=253
>
> Aug 13 16:28:50.000 multipathd: MINOR=1
>
> Aug 13 16:28:50.000 multipathd: DEVTYPE=diskAug 13 16:28:50.000
> multipathd: SEQNUM=1739Aug 13 16:28:50.000 multipathd: UDEVD_EVENT=1
>
> Aug 13 16:28:50.000 multipathd: vol1: rr_weight = 2 (LUN setting)
>
> Aug 13 16:28:50.000 multipathd: vol1: pgfailback = -2 (controller setting)
>
> Aug 13 16:28:50.000 multipathd: pg_timeout = NONE (internal default)
>
> Aug 13 16:28:50.000 multipathd: 8:208: mark as failed
>
> Aug 13 16:28:50.000 multipathd: vol1: remaining active paths: 3
>
> Aug 13 16:28:50.000 multipathd: vol1: rr_weight = 2 (LUN setting)
>
> Aug 13 16:28:50.000 multipathd: vol1: pgfailback = -2 (controller setting)
>
> Aug 13 16:28:50.000 multipathd: uevent 'change' from
> '/devices/virtual/block/dm-1'
>
> Aug 13 16:28:50.000 multipathd: UDEV_LOG=3
>
> Aug 13 16:28:50.000 multipathd: ACTION=change
>
> Aug 13 16:28:50.000 multipathd: DEVPATH=/devices/virtual/block/dm-1
>
> Aug 13 16:28:50.000 multipathd: SUBSYSTEM=block
>
> Aug 13 16:28:50.000 multipathd: DM_TARGET=multipath
>
> Aug 13 16:28:50.000 multipathd: DM_ACTION=PATH_FAILED
>
> Aug 13 16:28:50.000 multipathd: DM_SEQNUM=3
>
> Aug 13 16:28:50.000 multipathd: DM_PATH=8:208
>
> Aug 13 16:28:50.000 multipathd: DM_NR_VALID_PATHS=3
>
> Aug 13 16:28:50.000 multipathd: DM_NAME=vol1
>
> Aug 13 16:28:50.000 multipathd:
> DM_UUID=mpath-3600a0b800048335200001e5d48b68a9b
>
> Aug 13 16:28:50.000 multipathd: MAJOR=253
>
> Aug 13 16:28:50.000 multipathd: MINOR=1
>
> Aug 13 16:28:50.000 multipathd: DEVTYPE=disk
>
> Aug 13 16:28:50.000 multipathd: SEQNUM=1740
>
> Aug 13 16:28:50.000 multipathd: UDEVD_EVENT=1
>
> Aug 13 16:28:50.000 multipathd: DEVNAME=/dev/dm-1
>
> Aug 13 16:29:00.000 multipathd: 8:208: reinstated
>
> Aug 13 16:29:00.000 multipathd: vol1: remaining active paths: 4
>
> Aug 13 16:29:00.000 multipathd: sdj: rdac prio = 3
>
> Aug 13 16:29:00.000 multipathd: sdn: rdac prio = 3
>
> Aug 13 16:29:00.000 multipathd: sdb: rdac prio = 0
>
> Aug 13 16:29:00.000 multipathd: sdd: rdac prio = 0
>
> Aug 13 16:29:00.000 multipathd: vol1: rr_weight = 2 (LUN setting)
>
> Aug 13 16:29:00.000 multipathd: vol1: pgfailback = -2 (controller setting)
>
> Aug 13 16:29:00.000 multipathd: uevent 'change' from
> '/devices/virtual/block/dm-1'
>
> Aug 13 16:29:00.000 multipathd: UDEV_LOG=3
>
> Aug 13 16:29:00.000 multipathd: ACTION=change
>
> Aug 13 16:29:00.000 multipathd: DEVPATH=/devices/virtual/block/dm-1
>
> Aug 13 16:29:00.000 multipathd: SUBSYSTEM=block
>
> Aug 13 16:29:00.000 multipathd: DM_TARGET=multipath
>
> Aug 13 16:29:00.000 multipathd: DM_ACTION=PATH_REINSTATED
>
> Aug 13 16:29:00.000 multipathd: DM_SEQNUM=4
>
> Aug 13 16:29:00.000 multipathd: DM_PATH=8:208
>
> Aug 13 16:29:00.000 multipathd: DM_NR_VALID_PATHS=4
>
> Aug 13 16:29:00.000 multipathd: DM_NAME=vol1
>
> Aug 13 16:29:00.000 multipathd:
> DM_UUID=mpath-3600a0b800048335200001e5d48b68a9b
>
> Aug 13 16:29:00.000 multipathd: MAJOR=253
>
> Aug 13 16:29:00.000 multipathd: MINOR=1
>
> Aug 13 16:29:00.000 multipathd: DEVTYPE=disk
>
> Aug 13 16:29:00.000 multipathd: SEQNUM=1741
>
> Aug 13 16:29:00.000 multipathd: UDEVD_EVENT=1
>
> Aug 13 16:29:00.000 multipathd: DEVNAME=/dev/dm-1
>
> Aug 13 16:29:02.753 kernel: device-mapper: multipath: Failing path 8:208.
>
> Aug 13 16:29:02.000 multipathd: vol1: rr_weight = 2 (LUN setting)
>
> Aug 13 16:29:02.000 multipathd: vol1: pgfailback = -2 (controller setting)
>
> Aug 13 16:29:02.000 multipathd: pg_timeout = NONE (internal default)
>
> Aug 13 16:29:02.000 multipathd: 8:208: mark as failed
>
> Aug 13 16:29:02.000 multipathd: uevent 'change' from
> '/devices/virtual/block/dm-1'
>
> Aug 13 16:29:02.000 multipathd: UDEV_LOG=3
>
> Aug 13 16:29:02.000 multipathd: ACTION=change
>
> Aug 13 16:29:02.000 multipathd: DEVPATH=/devices/virtual/block/dm-1
>
> Aug 13 16:29:02.000 multipathd: SUBSYSTEM=block
>
> Aug 13 16:29:02.000 multipathd: DM_TARGET=multipath
>
> Aug 13 16:29:02.000 multipathd: DM_ACTION=PATH_FAILED
>
> Aug 13 16:29:02.000 multipathd: DM_SEQNUM=5
>
> Aug 13 16:29:02.000 multipathd: DM_PATH=8:208
>
> Aug 13 16:29:02.000 multipathd: DM_NR_VALID_PATHS=3
>
> Aug 13 16:29:02.000 multipathd: DM_NAME=vol1
>
> Aug 13 16:29:02.000 multipathd:
> DM_UUID=mpath-3600a0b800048335200001e5d48b68a9b
>
> Aug 13 16:29:02.000 multipathd: MAJOR=253
>
> Aug 13 16:29:02.000 multipathd: MINOR=1
>
> Aug 13 16:29:02.000 multipathd: DEVTYPE=disk
>
> Aug 13 16:29:02.000 multipathd: SEQNUM=1742
>
> Aug 13 16:29:02.000 multipathd: UDEVD_EVENT=1
>
> Aug 13 16:29:02.000 multipathd: DEVNAME=/dev/dm-1
>
> Aug 13 16:29:10.000 multipathd: 8:208: reinstated
>
> Aug 13 16:29:10.000 multipathd: vol1: remaining active paths: 4
>
> Aug 13 16:29:10.000 multipathd: sdj: rdac prio = 3
>
> Aug 13 16:29:10.000 multipathd: sdn: rdac prio = 3
>
> Aug 13 16:29:10.000 multipathd: sdb: rdac prio = 0
>
> Aug 13 16:29:10.000 multipathd: sdd: rdac prio = 0
>
> Aug 13 16:29:10.000 multipathd: vol1: rr_weight = 2 (LUN setting)
>
> Aug 13 16:29:10.000 multipathd: vol1: pgfailback = -2 (controller setting)
>
> Aug 13 16:29:10.000 multipathd: uevent 'change' from
> '/devices/virtual/block/dm-1'
>
> Aug 13 16:29:10.000 multipathd: UDEV_LOG=3
>
> Aug 13 16:29:10.000 multipathd: ACTION=change
>
> Aug 13 16:29:10.000 multipathd: DEVPATH=/devices/virtual/block/dm-1
>
> Aug 13 16:29:10.000 multipathd: SUBSYSTEM=block
>
> Aug 13 16:29:10.000 multipathd: DM_TARGET=multipath
>
> Aug 13 16:29:10.000 multipathd: DM_ACTION=PATH_REINSTATED
>
> Aug 13 16:29:10.000 multipathd: DM_SEQNUM=6
>
> Aug 13 16:29:10.000 multipathd: DM_PATH=8:208
>
> Aug 13 16:29:10.000 multipathd: DM_NR_VALID_PATHS=4
>
> Aug 13 16:29:10.000 multipathd: DM_NAME=vol1
>
> Aug 13 16:29:10.000 multipathd:
> DM_UUID=mpath-3600a0b800048335200001e5d48b68a9b
>
> Aug 13 16:29:10.000 multipathd: MAJOR=253
>
> Aug 13 16:29:10.000 multipathd: MINOR=1
>
> Aug 13 16:29:10.000 multipathd: DEVTYPE=disk
>
> Aug 13 16:29:10.000 multipathd: SEQNUM=1743
>
> Aug 13 16:29:10.000 multipathd: UDEVD_EVENT=1
>
> Aug 13 16:29:10.000 multipathd: DEVNAME=/dev/dm-1
>
>
>
>
>
>
>
>
>
> On Thu, Aug 13, 2009 at 1:27 PM, Stewart Smith <stew@cleepdar.com> wrote:
>
>
>
> after a fresh, multipath -F and start of multipathd with -v 2 I see the
> following messages.
>
>
>
> After starting multipathd I mounted /dev/mapper/vol1 and generated some
> simple I/O to it using dd
>
>
>
>
>
> Aug 13 16:23:14.888 localhost kernel: device-mapper: multipath: Failing path
> 8:208.
>
> Aug 13 16:23:14.000 localhost multipathd: 8:208: mark as failed
>
> Aug 13 16:23:16.000 localhost multipathd: 8:208: reinstated
>
> Aug 13 16:23:30.462 localhost kernel: device-mapper: multipath: Failing path
> 8:208.
>
> Aug 13 16:23:30.000 localhost multipathd: 8:208: mark as failed
>
> Aug 13 16:23:39.000 localhost multipathd: 8:208: reinstated
>
> Aug 13 16:23:46.430 localhost kernel: device-mapper: multipath: Failing path
> 8:208.
>
> Aug 13 16:23:46.000 localhost multipathd: 8:208: mark as failed
>
> Aug 13 16:23:51.041 localhost kernel: device-mapper: multipath: Failing path
> 8:208.
>
> Aug 13 16:23:51.000 localhost multipathd: 8:208: mark as failed
>
> Aug 13 16:23:59.000 localhost multipathd: 8:208: reinstated
>
> Aug 13 16:24:06.465 localhost kernel: device-mapper: multipath: Failing path
> 8:208.
>
> Aug 13 16:24:06.000 localhost multipathd: 8:208: mark as failed
>
> Aug 13 16:24:09.000 localhost multipathd: 8:208: reinstated
>
>
>
>
>
> Thanks,
>
> --
>
> Stew
>
>
>
>
>
>
>
> On Thu, Aug 13, 2009 at 12:42 PM, Moger, Babu <Babu.Moger@lsi.com> wrote:
>
> Do you have /var/log/messages file for this problem?
>
> Thanks
> Babu Moger
>
>> -----Original Message-----
>> From: dm-devel-bounces@redhat.com [mailto:dm-devel-bounces@redhat.com] On
>> Behalf Of Stewart Smith
>> Sent: Thursday, August 13, 2009 1:51 PM
>> To: dm-devel@redhat.com
>> Subject: [dm-devel] rdac path failure - Sun 6140
>>
>> Hello All,
>>
>> I am seeing many of these messages when my Sun 6140 array is under heavy
>> I/O
>> device-mapper: multipath: Failing path 8:208.
>> device-mapper: multipath: Failing path 8:208.
>> device-mapper: multipath: Failing path 8:208.
>> device-mapper: multipath: Failing path 8:208.
>> device-mapper: multipath: Failing path 8:208.
>>
>>
>> I am running a Fedora 10 server, with two fiber connections to two
>> different switches. Both controllers on the 6140 have one connection
>> to each switch as well. The end result is that I see four paths to
>> each LUN.
>>
>> When the volume is mounted and under significant load I see the
>> messages above every few seconds. They seem to appear every
>> "no_path_retry" seconds.
>>
>> The 6140 controller firmware is up to date at version 07.50.08.10 and
>> I have installed the latest firmware for my Emulex LPe11002 cards. I
>> have reproduced the problem using both Cisco MDS and Brocade fiber
>> channel switches as well.
>>
>> Using CAM, I have set the initiator Host Type to "Linux" at the
>> moment. I have tried other options as well without success.
>>
>> I have NOT installed the RDAC drivers from either Sun or LSI -
>> primarily because they do not seem to build on my Fedora 10 kernel.
>>
>> Any ideas would be greatly appreciated!!!
>>
>> configs and debugging multipathd output is below.
>>
>>
>>
>>
>>
>> Kernel: 2.6.27.24-170.2.68.fc10.x86_64
>>
>> # multipath -lll
>> vol1 (3600a0b800048335200001e5d48b68a9b) dm-1 SUN,CSM200_R
>> [size=12T][features=1 queue_if_no_path][hwhandler=1 rdac][rw]
>> \_ round-robin 0 [prio=6][active]
>> \_ 5:0:1:2 sdj 8:144 [active][ready]
>> \_ 2:0:1:2 sdn 8:208 [active][ready]
>> \_ round-robin 0 [prio=0][enabled]
>> \_ 2:0:0:2 sdb 8:16 [active][ghost]
>> \_ 5:0:0:2 sdd 8:48 [active][ghost]
>>
>>
>> # cat /etc/multipath.conf
>>
>> blacklist {
>> devnode "^sd[a-z][[0-9]*]"
>> devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
>> devnode "^hd[a-z][0-9]*"
>> devnode "^cciss!c[0-9]d[0-9](p[0-9]*)*"
>> }
>>
>> defaults {
>> udev_dir /dev
>> polling_interval 10
>> selector "round-robin 0"
>> path_grouping_policy multibus
>> getuid_callout "/sbin/scsi_id --whitelisted /dev/%n"
>> prio alua
>> path_checker readsector0
>> rr_min_io 100
>> max_fds 8192
>> rr_weight priorities
>> failback immediate
>> no_path_retry fail
>> user_friendly_names yes
>> }
>> devices {
>> device {
>> vendor "SUN"
>> product "CSM200_R"
>> product_blacklist "Universal Xport"
>> getuid_callout "/sbin/scsi_id --whitelisted
>> /dev/%n"
>> features "0"
>> hardware_handler "1 rdac"
>> path_selector "round-robin 0"
>> path_grouping_policy group_by_prio
>> failback immediate
>> rr_weight uniform
>> no_path_retry queue
>> rr_min_io 1000
>> path_checker rdac
>> prio rdac
>> }
>> }
>>
>> multipaths {
>> multipath {
>> wwid 3600a0b800048335200001e5d48b68a9b
>> alias vol1
>> rr_weight priorities
>> no_path_retry 5
>> rr_min_io 100
>> }
>> }
>>
>>
>>
>> # multipathd -d v3
>>
>>
>> Aug 13 14:48:53 | sdb: ownership set to vol1
>> Aug 13 14:48:53 | sdb: not found in pathvec
>> Aug 13 14:48:53 | sdb: mask = 0xc
>> Aug 13 14:48:53 | sdb: path checker = rdac (controller setting)
>> Aug 13 14:48:53 | sdb: state = 4
>> Aug 13 14:48:53 | sdb: rdac prio = 0
>> Aug 13 14:48:53 | sdd: ownership set to vol1
>> Aug 13 14:48:53 | sdd: not found in pathvec
>> Aug 13 14:48:53 | sdd: mask = 0xc
>> Aug 13 14:48:53 | sdd: path checker = rdac (controller setting)
>> Aug 13 14:48:53 | sdd: state = 4
>> Aug 13 14:48:53 | sdd: rdac prio = 0
>> Aug 13 14:48:53 | sdj: ownership set to vol1
>> Aug 13 14:48:53 | sdj: not found in pathvec
>> Aug 13 14:48:53 | sdj: mask = 0xc
>> Aug 13 14:48:53 | sdj: path checker = rdac (controller setting)
>> Aug 13 14:48:53 | sdj: state = 2
>> Aug 13 14:48:53 | sdj: rdac prio = 3
>> Aug 13 14:48:53 | sdn: ownership set to vol1
>> Aug 13 14:48:53 | sdn: not found in pathvec
>> Aug 13 14:48:53 | sdn: mask = 0xc
>> Aug 13 14:48:53 | sdn: path checker = rdac (controller setting)
>> Aug 13 14:48:53 | sdn: state = 2
>> Aug 13 14:48:53 | sdn: rdac prio = 3
>> Aug 13 14:48:53 | vol1: pgfailback = -2 (controller setting)
>> Aug 13 14:48:53 | vol1: pgpolicy = group_by_prio (controller setting)
>> Aug 13 14:48:53 | vol1: selector = round-robin 0 (controller setting)
>> Aug 13 14:48:53 | vol1: features = 0 (controller setting)
>> Aug 13 14:48:53 | vol1: hwhandler = 1 rdac (controller setting)
>> Aug 13 14:48:53 | vol1: rr_weight = 2 (LUN setting)
>> Aug 13 14:48:53 | vol1: minio = 100 (LUN setting)
>> Aug 13 14:48:53 | vol1: no_path_retry = 5 (multipath setting)
>> Aug 13 14:48:53 | pg_timeout = NONE (internal default)
>> Aug 13 14:48:53 | vol1: set ACT_CREATE (map does not exist)
>> create: vol1 (3600a0b800048335200001e5d48b68a9b) n/a SUN,CSM200_R
>> [size=12T][features=0][hwhandler=1 rdac][n/a]
>> \_ round-robin 0 [prio=6][undef]
>> \_ 5:0:1:2 sdj 8:144 [undef][ready]
>> \_ 2:0:1:2 sdn 8:208 [undef][ready]
>> \_ round-robin 0 [prio=0][undef]
>> \_ 2:0:0:2 sdb 8:16 [undef][ghost]
>> \_ 5:0:0:2 sdd 8:48 [undef][ghost]
>>
>
>> --
>> dm-devel mailing list
>> dm-devel@redhat.com
>> https://www.redhat.com/mailman/listinfo/dm-devel
>
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
>
>
>
>
^ permalink raw reply [flat|nested] 9+ messages in thread
* RE: rdac path failure - Sun 6140
2009-08-14 15:58 ` Stewart Smith
@ 2009-08-14 16:21 ` Moger, Babu
2009-08-14 20:26 ` Stewart Smith
0 siblings, 1 reply; 9+ messages in thread
From: Moger, Babu @ 2009-08-14 16:21 UTC (permalink / raw)
To: device-mapper development
> -----Original Message-----
> From: dm-devel-bounces@redhat.com [mailto:dm-devel-bounces@redhat.com] On
> Behalf Of Stewart Smith
> Sent: Friday, August 14, 2009 10:58 AM
> To: device-mapper development
> Subject: Re: [dm-devel] rdac path failure - Sun 6140
>
> I will test this today.
>
> Does everything look OK from a configuration standpoint? Should the
I have verified your configuration file. It looks fine..
> RDAC virtual HBA drivers from LSI be a requirement? I am not
> currently using them.
You can't use both the solutions at once. You are fine..
> Thank you,
> --
> Stew
>
>
>
>
> On Thu, Aug 13, 2009 at 5:05 PM, Moger, Babu<Babu.Moger@lsi.com> wrote:
> > Stew,
> >
> > I don’t see much information about this failure in the logs. Right
> now
> > device handlers don’t provide much information on failures. We are
> working
> > on to add some more debug levels. I am attaching my draft code
> > (scsi_dh_rdac.c) here. Please use this only for your testing. It is not
> > been approved/reviewed yet. I still need to submit this one to community
> for
> > approval. The code is attached. Please replace this file with
> > scsi_dh_rdac.c in the directory /driver/scsi/device_handlers and rebuild
> the
> > kernel. This should give more information from the target point of
> view.
> > Please send me the /var/log/messages file after the failure. Let see if
> we
> > can get more information..
> >
> >
> >
> > Thanks
> >
> > Babu Moger
> >
> > ________________________________
> >
> > From: dm-devel-bounces@redhat.com [mailto:dm-devel-bounces@redhat.com]
> On
> > Behalf Of Stewart Smith
> > Sent: Thursday, August 13, 2009 3:35 PM
> > To: device-mapper development
> > Subject: Re: [dm-devel] rdac path failure - Sun 6140
> >
> >
> >
> >
> >
> > Same sequence of events, with multipathd -v3
> >
> >
> >
> > Aug 13 16:28:48.627 kernel: device-mapper: multipath: Failing path
> 8:208.
> >
> > Aug 13 16:28:48.000 multipathd: vol1: rr_weight = 2 (LUN setting)
> >
> > Aug 13 16:28:48.000 multipathd: vol1: pgfailback = -2 (controller
> setting)
> >
> > Aug 13 16:28:48.000 multipathd: pg_timeout = NONE (internal default)
> >
> > Aug 13 16:28:48.000 multipathd: 8:208: mark as failed
> >
> > Aug 13 16:28:48.000 multipathd: uevent 'change' from
> > '/devices/virtual/block/dm-1'
> >
> > Aug 13 16:28:48.000 multipathd: UDEV_LOG=3
> >
> > Aug 13 16:28:48.000 multipathd: ACTION=change
> >
> > Aug 13 16:28:48.000 multipathd: DEVPATH=/devices/virtual/block/dm-1
> >
> > Aug 13 16:28:48.000 multipathd: SUBSYSTEM=block
> >
> > Aug 13 16:28:48.000 multipathd: DM_TARGET=multipath
> >
> > Aug 13 16:28:48.000 multipathd: DM_ACTION=PATH_FAILED
> >
> > Aug 13 16:28:48.000 multipathd: DM_SEQNUM=1
> >
> > Aug 13 16:28:48.000 multipathd: DM_PATH=8:208
> >
> > Aug 13 16:28:48.000 multipathd: DM_NR_VALID_PATHS=3
> >
> > Aug 13 16:28:48.000 multipathd: DM_NAME=vol1
> >
> > Aug 13 16:28:48.000 multipathd:
> > DM_UUID=mpath-3600a0b800048335200001e5d48b68a9b
> >
> > Aug 13 16:28:48.000 multipathd: MAJOR=253
> >
> > Aug 13 16:28:48.000 multipathd: MINOR=1
> >
> > Aug 13 16:28:48.000 multipathd: DEVTYPE=disk
> >
> > Aug 13 16:28:48.000 multipathd: SEQNUM=1738
> >
> > Aug 13 16:28:48.000 multipathd: UDEVD_EVENT=1
> >
> > Aug 13 16:28:48.000 multipathd: DEVNAME=/dev/dm-1
> >
> > Aug 13 16:28:50.000 multipathd: 8:208: reinstated
> >
> > Aug 13 16:28:50.000 multipathd: vol1: remaining active paths: 4
> >
> > Aug 13 16:28:50.000 multipathd: sdj: rdac prio = 3
> >
> > Aug 13 16:28:50.000 multipathd: sdn: rdac prio = 3
> >
> > Aug 13 16:28:50.000 multipathd: sdb: rdac prio = 0
> >
> > Aug 13 16:28:50.000 multipathd: sdd: rdac prio = 0
> >
> > Aug 13 16:28:50.763 kernel: device-mapper: multipath: Failing path
> 8:208.
> >
> > Aug 13 16:28:50.000 multipathd: uevent 'change' from
> > '/devices/virtual/block/dm-1'
> >
> > Aug 13 16:28:50.000 multipathd: UDEV_LOG=3
> >
> > Aug 13 16:28:50.000 multipathd: ACTION=change
> >
> > Aug 13 16:28:50.000 multipathd: DEVPATH=/devices/virtual/block/dm-1
> >
> > Aug 13 16:28:50.000 multipathd: SUBSYSTEM=block
> >
> > Aug 13 16:28:50.000 multipathd: DM_TARGET=multipath
> >
> > Aug 13 16:28:50.000 multipathd: DM_ACTION=PATH_REINSTATED
> >
> > Aug 13 16:28:50.000 multipathd: DM_SEQNUM=2
> >
> > Aug 13 16:28:50.000 multipathd: DM_PATH=8:208
> >
> > Aug 13 16:28:50.000 multipathd: DM_NR_VALID_PATHS=4
> >
> > Aug 13 16:28:50.000 multipathd: DM_NAME=vol1
> >
> > Aug 13 16:28:50.000 multipathd:
> > DM_UUID=mpath-3600a0b800048335200001e5d48b68a9b
> >
> > Aug 13 16:28:50.000 multipathd: MAJOR=253
> >
> > Aug 13 16:28:50.000 multipathd: MINOR=1
> >
> > Aug 13 16:28:50.000 multipathd: DEVTYPE=diskAug 13 16:28:50.000
> > multipathd: SEQNUM=1739Aug 13 16:28:50.000 multipathd: UDEVD_EVENT=1
> >
> > Aug 13 16:28:50.000 multipathd: vol1: rr_weight = 2 (LUN setting)
> >
> > Aug 13 16:28:50.000 multipathd: vol1: pgfailback = -2 (controller
> setting)
> >
> > Aug 13 16:28:50.000 multipathd: pg_timeout = NONE (internal default)
> >
> > Aug 13 16:28:50.000 multipathd: 8:208: mark as failed
> >
> > Aug 13 16:28:50.000 multipathd: vol1: remaining active paths: 3
> >
> > Aug 13 16:28:50.000 multipathd: vol1: rr_weight = 2 (LUN setting)
> >
> > Aug 13 16:28:50.000 multipathd: vol1: pgfailback = -2 (controller
> setting)
> >
> > Aug 13 16:28:50.000 multipathd: uevent 'change' from
> > '/devices/virtual/block/dm-1'
> >
> > Aug 13 16:28:50.000 multipathd: UDEV_LOG=3
> >
> > Aug 13 16:28:50.000 multipathd: ACTION=change
> >
> > Aug 13 16:28:50.000 multipathd: DEVPATH=/devices/virtual/block/dm-1
> >
> > Aug 13 16:28:50.000 multipathd: SUBSYSTEM=block
> >
> > Aug 13 16:28:50.000 multipathd: DM_TARGET=multipath
> >
> > Aug 13 16:28:50.000 multipathd: DM_ACTION=PATH_FAILED
> >
> > Aug 13 16:28:50.000 multipathd: DM_SEQNUM=3
> >
> > Aug 13 16:28:50.000 multipathd: DM_PATH=8:208
> >
> > Aug 13 16:28:50.000 multipathd: DM_NR_VALID_PATHS=3
> >
> > Aug 13 16:28:50.000 multipathd: DM_NAME=vol1
> >
> > Aug 13 16:28:50.000 multipathd:
> > DM_UUID=mpath-3600a0b800048335200001e5d48b68a9b
> >
> > Aug 13 16:28:50.000 multipathd: MAJOR=253
> >
> > Aug 13 16:28:50.000 multipathd: MINOR=1
> >
> > Aug 13 16:28:50.000 multipathd: DEVTYPE=disk
> >
> > Aug 13 16:28:50.000 multipathd: SEQNUM=1740
> >
> > Aug 13 16:28:50.000 multipathd: UDEVD_EVENT=1
> >
> > Aug 13 16:28:50.000 multipathd: DEVNAME=/dev/dm-1
> >
> > Aug 13 16:29:00.000 multipathd: 8:208: reinstated
> >
> > Aug 13 16:29:00.000 multipathd: vol1: remaining active paths: 4
> >
> > Aug 13 16:29:00.000 multipathd: sdj: rdac prio = 3
> >
> > Aug 13 16:29:00.000 multipathd: sdn: rdac prio = 3
> >
> > Aug 13 16:29:00.000 multipathd: sdb: rdac prio = 0
> >
> > Aug 13 16:29:00.000 multipathd: sdd: rdac prio = 0
> >
> > Aug 13 16:29:00.000 multipathd: vol1: rr_weight = 2 (LUN setting)
> >
> > Aug 13 16:29:00.000 multipathd: vol1: pgfailback = -2 (controller
> setting)
> >
> > Aug 13 16:29:00.000 multipathd: uevent 'change' from
> > '/devices/virtual/block/dm-1'
> >
> > Aug 13 16:29:00.000 multipathd: UDEV_LOG=3
> >
> > Aug 13 16:29:00.000 multipathd: ACTION=change
> >
> > Aug 13 16:29:00.000 multipathd: DEVPATH=/devices/virtual/block/dm-1
> >
> > Aug 13 16:29:00.000 multipathd: SUBSYSTEM=block
> >
> > Aug 13 16:29:00.000 multipathd: DM_TARGET=multipath
> >
> > Aug 13 16:29:00.000 multipathd: DM_ACTION=PATH_REINSTATED
> >
> > Aug 13 16:29:00.000 multipathd: DM_SEQNUM=4
> >
> > Aug 13 16:29:00.000 multipathd: DM_PATH=8:208
> >
> > Aug 13 16:29:00.000 multipathd: DM_NR_VALID_PATHS=4
> >
> > Aug 13 16:29:00.000 multipathd: DM_NAME=vol1
> >
> > Aug 13 16:29:00.000 multipathd:
> > DM_UUID=mpath-3600a0b800048335200001e5d48b68a9b
> >
> > Aug 13 16:29:00.000 multipathd: MAJOR=253
> >
> > Aug 13 16:29:00.000 multipathd: MINOR=1
> >
> > Aug 13 16:29:00.000 multipathd: DEVTYPE=disk
> >
> > Aug 13 16:29:00.000 multipathd: SEQNUM=1741
> >
> > Aug 13 16:29:00.000 multipathd: UDEVD_EVENT=1
> >
> > Aug 13 16:29:00.000 multipathd: DEVNAME=/dev/dm-1
> >
> > Aug 13 16:29:02.753 kernel: device-mapper: multipath: Failing path
> 8:208.
> >
> > Aug 13 16:29:02.000 multipathd: vol1: rr_weight = 2 (LUN setting)
> >
> > Aug 13 16:29:02.000 multipathd: vol1: pgfailback = -2 (controller
> setting)
> >
> > Aug 13 16:29:02.000 multipathd: pg_timeout = NONE (internal default)
> >
> > Aug 13 16:29:02.000 multipathd: 8:208: mark as failed
> >
> > Aug 13 16:29:02.000 multipathd: uevent 'change' from
> > '/devices/virtual/block/dm-1'
> >
> > Aug 13 16:29:02.000 multipathd: UDEV_LOG=3
> >
> > Aug 13 16:29:02.000 multipathd: ACTION=change
> >
> > Aug 13 16:29:02.000 multipathd: DEVPATH=/devices/virtual/block/dm-1
> >
> > Aug 13 16:29:02.000 multipathd: SUBSYSTEM=block
> >
> > Aug 13 16:29:02.000 multipathd: DM_TARGET=multipath
> >
> > Aug 13 16:29:02.000 multipathd: DM_ACTION=PATH_FAILED
> >
> > Aug 13 16:29:02.000 multipathd: DM_SEQNUM=5
> >
> > Aug 13 16:29:02.000 multipathd: DM_PATH=8:208
> >
> > Aug 13 16:29:02.000 multipathd: DM_NR_VALID_PATHS=3
> >
> > Aug 13 16:29:02.000 multipathd: DM_NAME=vol1
> >
> > Aug 13 16:29:02.000 multipathd:
> > DM_UUID=mpath-3600a0b800048335200001e5d48b68a9b
> >
> > Aug 13 16:29:02.000 multipathd: MAJOR=253
> >
> > Aug 13 16:29:02.000 multipathd: MINOR=1
> >
> > Aug 13 16:29:02.000 multipathd: DEVTYPE=disk
> >
> > Aug 13 16:29:02.000 multipathd: SEQNUM=1742
> >
> > Aug 13 16:29:02.000 multipathd: UDEVD_EVENT=1
> >
> > Aug 13 16:29:02.000 multipathd: DEVNAME=/dev/dm-1
> >
> > Aug 13 16:29:10.000 multipathd: 8:208: reinstated
> >
> > Aug 13 16:29:10.000 multipathd: vol1: remaining active paths: 4
> >
> > Aug 13 16:29:10.000 multipathd: sdj: rdac prio = 3
> >
> > Aug 13 16:29:10.000 multipathd: sdn: rdac prio = 3
> >
> > Aug 13 16:29:10.000 multipathd: sdb: rdac prio = 0
> >
> > Aug 13 16:29:10.000 multipathd: sdd: rdac prio = 0
> >
> > Aug 13 16:29:10.000 multipathd: vol1: rr_weight = 2 (LUN setting)
> >
> > Aug 13 16:29:10.000 multipathd: vol1: pgfailback = -2 (controller
> setting)
> >
> > Aug 13 16:29:10.000 multipathd: uevent 'change' from
> > '/devices/virtual/block/dm-1'
> >
> > Aug 13 16:29:10.000 multipathd: UDEV_LOG=3
> >
> > Aug 13 16:29:10.000 multipathd: ACTION=change
> >
> > Aug 13 16:29:10.000 multipathd: DEVPATH=/devices/virtual/block/dm-1
> >
> > Aug 13 16:29:10.000 multipathd: SUBSYSTEM=block
> >
> > Aug 13 16:29:10.000 multipathd: DM_TARGET=multipath
> >
> > Aug 13 16:29:10.000 multipathd: DM_ACTION=PATH_REINSTATED
> >
> > Aug 13 16:29:10.000 multipathd: DM_SEQNUM=6
> >
> > Aug 13 16:29:10.000 multipathd: DM_PATH=8:208
> >
> > Aug 13 16:29:10.000 multipathd: DM_NR_VALID_PATHS=4
> >
> > Aug 13 16:29:10.000 multipathd: DM_NAME=vol1
> >
> > Aug 13 16:29:10.000 multipathd:
> > DM_UUID=mpath-3600a0b800048335200001e5d48b68a9b
> >
> > Aug 13 16:29:10.000 multipathd: MAJOR=253
> >
> > Aug 13 16:29:10.000 multipathd: MINOR=1
> >
> > Aug 13 16:29:10.000 multipathd: DEVTYPE=disk
> >
> > Aug 13 16:29:10.000 multipathd: SEQNUM=1743
> >
> > Aug 13 16:29:10.000 multipathd: UDEVD_EVENT=1
> >
> > Aug 13 16:29:10.000 multipathd: DEVNAME=/dev/dm-1
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Thu, Aug 13, 2009 at 1:27 PM, Stewart Smith <stew@cleepdar.com>
> wrote:
> >
> >
> >
> > after a fresh, multipath -F and start of multipathd with -v 2 I see the
> > following messages.
> >
> >
> >
> > After starting multipathd I mounted /dev/mapper/vol1 and generated some
> > simple I/O to it using dd
> >
> >
> >
> >
> >
> > Aug 13 16:23:14.888 localhost kernel: device-mapper: multipath: Failing
> path
> > 8:208.
> >
> > Aug 13 16:23:14.000 localhost multipathd: 8:208: mark as failed
> >
> > Aug 13 16:23:16.000 localhost multipathd: 8:208: reinstated
> >
> > Aug 13 16:23:30.462 localhost kernel: device-mapper: multipath: Failing
> path
> > 8:208.
> >
> > Aug 13 16:23:30.000 localhost multipathd: 8:208: mark as failed
> >
> > Aug 13 16:23:39.000 localhost multipathd: 8:208: reinstated
> >
> > Aug 13 16:23:46.430 localhost kernel: device-mapper: multipath: Failing
> path
> > 8:208.
> >
> > Aug 13 16:23:46.000 localhost multipathd: 8:208: mark as failed
> >
> > Aug 13 16:23:51.041 localhost kernel: device-mapper: multipath: Failing
> path
> > 8:208.
> >
> > Aug 13 16:23:51.000 localhost multipathd: 8:208: mark as failed
> >
> > Aug 13 16:23:59.000 localhost multipathd: 8:208: reinstated
> >
> > Aug 13 16:24:06.465 localhost kernel: device-mapper: multipath: Failing
> path
> > 8:208.
> >
> > Aug 13 16:24:06.000 localhost multipathd: 8:208: mark as failed
> >
> > Aug 13 16:24:09.000 localhost multipathd: 8:208: reinstated
> >
> >
> >
> >
> >
> > Thanks,
> >
> > --
> >
> > Stew
> >
> >
> >
> >
> >
> >
> >
> > On Thu, Aug 13, 2009 at 12:42 PM, Moger, Babu <Babu.Moger@lsi.com>
> wrote:
> >
> > Do you have /var/log/messages file for this problem?
> >
> > Thanks
> > Babu Moger
> >
> >> -----Original Message-----
> >> From: dm-devel-bounces@redhat.com [mailto:dm-devel-bounces@redhat.com]
> On
> >> Behalf Of Stewart Smith
> >> Sent: Thursday, August 13, 2009 1:51 PM
> >> To: dm-devel@redhat.com
> >> Subject: [dm-devel] rdac path failure - Sun 6140
> >>
> >> Hello All,
> >>
> >> I am seeing many of these messages when my Sun 6140 array is under
> heavy
> >> I/O
> >> device-mapper: multipath: Failing path 8:208.
> >> device-mapper: multipath: Failing path 8:208.
> >> device-mapper: multipath: Failing path 8:208.
> >> device-mapper: multipath: Failing path 8:208.
> >> device-mapper: multipath: Failing path 8:208.
> >>
> >>
> >> I am running a Fedora 10 server, with two fiber connections to two
> >> different switches. Both controllers on the 6140 have one connection
> >> to each switch as well. The end result is that I see four paths to
> >> each LUN.
> >>
> >> When the volume is mounted and under significant load I see the
> >> messages above every few seconds. They seem to appear every
> >> "no_path_retry" seconds.
> >>
> >> The 6140 controller firmware is up to date at version 07.50.08.10 and
> >> I have installed the latest firmware for my Emulex LPe11002 cards. I
> >> have reproduced the problem using both Cisco MDS and Brocade fiber
> >> channel switches as well.
> >>
> >> Using CAM, I have set the initiator Host Type to "Linux" at the
> >> moment. I have tried other options as well without success.
> >>
> >> I have NOT installed the RDAC drivers from either Sun or LSI -
> >> primarily because they do not seem to build on my Fedora 10 kernel.
> >>
> >> Any ideas would be greatly appreciated!!!
> >>
> >> configs and debugging multipathd output is below.
> >>
> >>
> >>
> >>
> >>
> >> Kernel: 2.6.27.24-170.2.68.fc10.x86_64
> >>
> >> # multipath -lll
> >> vol1 (3600a0b800048335200001e5d48b68a9b) dm-1 SUN,CSM200_R
> >> [size=12T][features=1 queue_if_no_path][hwhandler=1 rdac][rw]
> >> \_ round-robin 0 [prio=6][active]
> >> \_ 5:0:1:2 sdj 8:144 [active][ready]
> >> \_ 2:0:1:2 sdn 8:208 [active][ready]
> >> \_ round-robin 0 [prio=0][enabled]
> >> \_ 2:0:0:2 sdb 8:16 [active][ghost]
> >> \_ 5:0:0:2 sdd 8:48 [active][ghost]
> >>
> >>
> >> # cat /etc/multipath.conf
> >>
> >> blacklist {
> >> devnode "^sd[a-z][[0-9]*]"
> >> devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
> >> devnode "^hd[a-z][0-9]*"
> >> devnode "^cciss!c[0-9]d[0-9](p[0-9]*)*"
> >> }
> >>
> >> defaults {
> >> udev_dir /dev
> >> polling_interval 10
> >> selector "round-robin 0"
> >> path_grouping_policy multibus
> >> getuid_callout "/sbin/scsi_id --whitelisted /dev/%n"
> >> prio alua
> >> path_checker readsector0
> >> rr_min_io 100
> >> max_fds 8192
> >> rr_weight priorities
> >> failback immediate
> >> no_path_retry fail
> >> user_friendly_names yes
> >> }
> >> devices {
> >> device {
> >> vendor "SUN"
> >> product "CSM200_R"
> >> product_blacklist "Universal Xport"
> >> getuid_callout "/sbin/scsi_id --whitelisted
> >> /dev/%n"
> >> features "0"
> >> hardware_handler "1 rdac"
> >> path_selector "round-robin 0"
> >> path_grouping_policy group_by_prio
> >> failback immediate
> >> rr_weight uniform
> >> no_path_retry queue
> >> rr_min_io 1000
> >> path_checker rdac
> >> prio rdac
> >> }
> >> }
> >>
> >> multipaths {
> >> multipath {
> >> wwid
> 3600a0b800048335200001e5d48b68a9b
> >> alias vol1
> >> rr_weight priorities
> >> no_path_retry 5
> >> rr_min_io 100
> >> }
> >> }
> >>
> >>
> >>
> >> # multipathd -d v3
> >>
> >>
> >> Aug 13 14:48:53 | sdb: ownership set to vol1
> >> Aug 13 14:48:53 | sdb: not found in pathvec
> >> Aug 13 14:48:53 | sdb: mask = 0xc
> >> Aug 13 14:48:53 | sdb: path checker = rdac (controller setting)
> >> Aug 13 14:48:53 | sdb: state = 4
> >> Aug 13 14:48:53 | sdb: rdac prio = 0
> >> Aug 13 14:48:53 | sdd: ownership set to vol1
> >> Aug 13 14:48:53 | sdd: not found in pathvec
> >> Aug 13 14:48:53 | sdd: mask = 0xc
> >> Aug 13 14:48:53 | sdd: path checker = rdac (controller setting)
> >> Aug 13 14:48:53 | sdd: state = 4
> >> Aug 13 14:48:53 | sdd: rdac prio = 0
> >> Aug 13 14:48:53 | sdj: ownership set to vol1
> >> Aug 13 14:48:53 | sdj: not found in pathvec
> >> Aug 13 14:48:53 | sdj: mask = 0xc
> >> Aug 13 14:48:53 | sdj: path checker = rdac (controller setting)
> >> Aug 13 14:48:53 | sdj: state = 2
> >> Aug 13 14:48:53 | sdj: rdac prio = 3
> >> Aug 13 14:48:53 | sdn: ownership set to vol1
> >> Aug 13 14:48:53 | sdn: not found in pathvec
> >> Aug 13 14:48:53 | sdn: mask = 0xc
> >> Aug 13 14:48:53 | sdn: path checker = rdac (controller setting)
> >> Aug 13 14:48:53 | sdn: state = 2
> >> Aug 13 14:48:53 | sdn: rdac prio = 3
> >> Aug 13 14:48:53 | vol1: pgfailback = -2 (controller setting)
> >> Aug 13 14:48:53 | vol1: pgpolicy = group_by_prio (controller setting)
> >> Aug 13 14:48:53 | vol1: selector = round-robin 0 (controller setting)
> >> Aug 13 14:48:53 | vol1: features = 0 (controller setting)
> >> Aug 13 14:48:53 | vol1: hwhandler = 1 rdac (controller setting)
> >> Aug 13 14:48:53 | vol1: rr_weight = 2 (LUN setting)
> >> Aug 13 14:48:53 | vol1: minio = 100 (LUN setting)
> >> Aug 13 14:48:53 | vol1: no_path_retry = 5 (multipath setting)
> >> Aug 13 14:48:53 | pg_timeout = NONE (internal default)
> >> Aug 13 14:48:53 | vol1: set ACT_CREATE (map does not exist)
> >> create: vol1 (3600a0b800048335200001e5d48b68a9b) n/a SUN,CSM200_R
> >> [size=12T][features=0][hwhandler=1 rdac][n/a]
> >> \_ round-robin 0 [prio=6][undef]
> >> \_ 5:0:1:2 sdj 8:144 [undef][ready]
> >> \_ 2:0:1:2 sdn 8:208 [undef][ready]
> >> \_ round-robin 0 [prio=0][undef]
> >> \_ 2:0:0:2 sdb 8:16 [undef][ghost]
> >> \_ 5:0:0:2 sdd 8:48 [undef][ghost]
> >>
> >
> >> --
> >> dm-devel mailing list
> >> dm-devel@redhat.com
> >> https://www.redhat.com/mailman/listinfo/dm-devel
> >
> > --
> > dm-devel mailing list
> > dm-devel@redhat.com
> > https://www.redhat.com/mailman/listinfo/dm-devel
> >
> >
> >
> >
>
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: rdac path failure - Sun 6140
2009-08-14 16:21 ` Moger, Babu
@ 2009-08-14 20:26 ` Stewart Smith
2009-08-14 22:22 ` Chandra Seetharaman
0 siblings, 1 reply; 9+ messages in thread
From: Stewart Smith @ 2009-08-14 20:26 UTC (permalink / raw)
To: device-mapper development
Interesting.
I just grabbed 2.6.30.4 from kernel.org and I can no longer reproduce
the issue with that kernel. I did not add in your debugging version
of scsi_dh_rdac.c yet.
Somewhere between vmlinuz-2.6.29.4-167.fc11 and the generic 2.6.30.4
(using the same kernel .config) this problem was fixed.
Its now running over 350 MB/s with 16 sequential writers using both
active paths.
# fio --filename=/san0/vol1/test --direct=1 --rw=write --bs=8m
--size=32G --numjobs=16 --runtime=60 --group_reporting --name=file1
Run status group 0 (all jobs):
WRITE: io=21,416MiB, aggrb=355MiB/s, minb=355MiB/s, maxb=355MiB/s,
mint=60342msec, maxt=60342msec
Disk stats (read/write):
dm-1: ios=3/85697, merge=0/0, ticks=11/21809639, in_queue=21810085,
util=99.94%, aggrios=0/11402, aggrmerge=0/10022, aggrticks=2/2805880,
aggrin_queue=2805875, aggrutil=97.09%
sdf: ios=0/22732, merge=0/20168, ticks=0/6090166,
in_queue=6090145, util=97.09%
sdn: ios=3/22876, merge=0/19921, ticks=11/5133355,
in_queue=5133356, util=96.16%
sdb: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
sdj: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
Thanks!
--
Stewart
On Fri, Aug 14, 2009 at 9:21 AM, Moger, Babu<Babu.Moger@lsi.com> wrote:
>> -----Original Message-----
>> From: dm-devel-bounces@redhat.com [mailto:dm-devel-bounces@redhat.com] On
>> Behalf Of Stewart Smith
>> Sent: Friday, August 14, 2009 10:58 AM
>> To: device-mapper development
>> Subject: Re: [dm-devel] rdac path failure - Sun 6140
>>
>> I will test this today.
>>
>> Does everything look OK from a configuration standpoint? Should the
>
> I have verified your configuration file. It looks fine..
>
>> RDAC virtual HBA drivers from LSI be a requirement? I am not
>> currently using them.
>
> You can't use both the solutions at once. You are fine..
>
>> Thank you,
>> --
>> Stew
>>
>>
>>
>>
>> On Thu, Aug 13, 2009 at 5:05 PM, Moger, Babu<Babu.Moger@lsi.com> wrote:
>> > Stew,
>> >
>> > I don’t see much information about this failure in the logs. Right
>> now
>> > device handlers don’t provide much information on failures. We are
>> working
>> > on to add some more debug levels. I am attaching my draft code
>> > (scsi_dh_rdac.c) here. Please use this only for your testing. It is not
>> > been approved/reviewed yet. I still need to submit this one to community
>> for
>> > approval. The code is attached. Please replace this file with
>> > scsi_dh_rdac.c in the directory /driver/scsi/device_handlers and rebuild
>> the
>> > kernel. This should give more information from the target point of
>> view.
>> > Please send me the /var/log/messages file after the failure. Let see if
>> we
>> > can get more information..
>> >
>> >
>> >
>> > Thanks
>> >
>> > Babu Moger
>> >
>> > ________________________________
>> >
>> > From: dm-devel-bounces@redhat.com [mailto:dm-devel-bounces@redhat.com]
>> On
>> > Behalf Of Stewart Smith
>> > Sent: Thursday, August 13, 2009 3:35 PM
>> > To: device-mapper development
>> > Subject: Re: [dm-devel] rdac path failure - Sun 6140
>> >
>> >
>> >
>> >
>> >
>> > Same sequence of events, with multipathd -v3
>> >
>> >
>> >
>> > Aug 13 16:28:48.627 kernel: device-mapper: multipath: Failing path
>> 8:208.
>> >
>> > Aug 13 16:28:48.000 multipathd: vol1: rr_weight = 2 (LUN setting)
>> >
>> > Aug 13 16:28:48.000 multipathd: vol1: pgfailback = -2 (controller
>> setting)
>> >
>> > Aug 13 16:28:48.000 multipathd: pg_timeout = NONE (internal default)
>> >
>> > Aug 13 16:28:48.000 multipathd: 8:208: mark as failed
>> >
>> > Aug 13 16:28:48.000 multipathd: uevent 'change' from
>> > '/devices/virtual/block/dm-1'
>> >
>> > Aug 13 16:28:48.000 multipathd: UDEV_LOG=3
>> >
>> > Aug 13 16:28:48.000 multipathd: ACTION=change
>> >
>> > Aug 13 16:28:48.000 multipathd: DEVPATH=/devices/virtual/block/dm-1
>> >
>> > Aug 13 16:28:48.000 multipathd: SUBSYSTEM=block
>> >
>> > Aug 13 16:28:48.000 multipathd: DM_TARGET=multipath
>> >
>> > Aug 13 16:28:48.000 multipathd: DM_ACTION=PATH_FAILED
>> >
>> > Aug 13 16:28:48.000 multipathd: DM_SEQNUM=1
>> >
>> > Aug 13 16:28:48.000 multipathd: DM_PATH=8:208
>> >
>> > Aug 13 16:28:48.000 multipathd: DM_NR_VALID_PATHS=3
>> >
>> > Aug 13 16:28:48.000 multipathd: DM_NAME=vol1
>> >
>> > Aug 13 16:28:48.000 multipathd:
>> > DM_UUID=mpath-3600a0b800048335200001e5d48b68a9b
>> >
>> > Aug 13 16:28:48.000 multipathd: MAJOR=253
>> >
>> > Aug 13 16:28:48.000 multipathd: MINOR=1
>> >
>> > Aug 13 16:28:48.000 multipathd: DEVTYPE=disk
>> >
>> > Aug 13 16:28:48.000 multipathd: SEQNUM=1738
>> >
>> > Aug 13 16:28:48.000 multipathd: UDEVD_EVENT=1
>> >
>> > Aug 13 16:28:48.000 multipathd: DEVNAME=/dev/dm-1
>> >
>> > Aug 13 16:28:50.000 multipathd: 8:208: reinstated
>> >
>> > Aug 13 16:28:50.000 multipathd: vol1: remaining active paths: 4
>> >
>> > Aug 13 16:28:50.000 multipathd: sdj: rdac prio = 3
>> >
>> > Aug 13 16:28:50.000 multipathd: sdn: rdac prio = 3
>> >
>> > Aug 13 16:28:50.000 multipathd: sdb: rdac prio = 0
>> >
>> > Aug 13 16:28:50.000 multipathd: sdd: rdac prio = 0
>> >
>> > Aug 13 16:28:50.763 kernel: device-mapper: multipath: Failing path
>> 8:208.
>> >
>> > Aug 13 16:28:50.000 multipathd: uevent 'change' from
>> > '/devices/virtual/block/dm-1'
>> >
>> > Aug 13 16:28:50.000 multipathd: UDEV_LOG=3
>> >
>> > Aug 13 16:28:50.000 multipathd: ACTION=change
>> >
>> > Aug 13 16:28:50.000 multipathd: DEVPATH=/devices/virtual/block/dm-1
>> >
>> > Aug 13 16:28:50.000 multipathd: SUBSYSTEM=block
>> >
>> > Aug 13 16:28:50.000 multipathd: DM_TARGET=multipath
>> >
>> > Aug 13 16:28:50.000 multipathd: DM_ACTION=PATH_REINSTATED
>> >
>> > Aug 13 16:28:50.000 multipathd: DM_SEQNUM=2
>> >
>> > Aug 13 16:28:50.000 multipathd: DM_PATH=8:208
>> >
>> > Aug 13 16:28:50.000 multipathd: DM_NR_VALID_PATHS=4
>> >
>> > Aug 13 16:28:50.000 multipathd: DM_NAME=vol1
>> >
>> > Aug 13 16:28:50.000 multipathd:
>> > DM_UUID=mpath-3600a0b800048335200001e5d48b68a9b
>> >
>> > Aug 13 16:28:50.000 multipathd: MAJOR=253
>> >
>> > Aug 13 16:28:50.000 multipathd: MINOR=1
>> >
>> > Aug 13 16:28:50.000 multipathd: DEVTYPE=diskAug 13 16:28:50.000
>> > multipathd: SEQNUM=1739Aug 13 16:28:50.000 multipathd: UDEVD_EVENT=1
>> >
>> > Aug 13 16:28:50.000 multipathd: vol1: rr_weight = 2 (LUN setting)
>> >
>> > Aug 13 16:28:50.000 multipathd: vol1: pgfailback = -2 (controller
>> setting)
>> >
>> > Aug 13 16:28:50.000 multipathd: pg_timeout = NONE (internal default)
>> >
>> > Aug 13 16:28:50.000 multipathd: 8:208: mark as failed
>> >
>> > Aug 13 16:28:50.000 multipathd: vol1: remaining active paths: 3
>> >
>> > Aug 13 16:28:50.000 multipathd: vol1: rr_weight = 2 (LUN setting)
>> >
>> > Aug 13 16:28:50.000 multipathd: vol1: pgfailback = -2 (controller
>> setting)
>> >
>> > Aug 13 16:28:50.000 multipathd: uevent 'change' from
>> > '/devices/virtual/block/dm-1'
>> >
>> > Aug 13 16:28:50.000 multipathd: UDEV_LOG=3
>> >
>> > Aug 13 16:28:50.000 multipathd: ACTION=change
>> >
>> > Aug 13 16:28:50.000 multipathd: DEVPATH=/devices/virtual/block/dm-1
>> >
>> > Aug 13 16:28:50.000 multipathd: SUBSYSTEM=block
>> >
>> > Aug 13 16:28:50.000 multipathd: DM_TARGET=multipath
>> >
>> > Aug 13 16:28:50.000 multipathd: DM_ACTION=PATH_FAILED
>> >
>> > Aug 13 16:28:50.000 multipathd: DM_SEQNUM=3
>> >
>> > Aug 13 16:28:50.000 multipathd: DM_PATH=8:208
>> >
>> > Aug 13 16:28:50.000 multipathd: DM_NR_VALID_PATHS=3
>> >
>> > Aug 13 16:28:50.000 multipathd: DM_NAME=vol1
>> >
>> > Aug 13 16:28:50.000 multipathd:
>> > DM_UUID=mpath-3600a0b800048335200001e5d48b68a9b
>> >
>> > Aug 13 16:28:50.000 multipathd: MAJOR=253
>> >
>> > Aug 13 16:28:50.000 multipathd: MINOR=1
>> >
>> > Aug 13 16:28:50.000 multipathd: DEVTYPE=disk
>> >
>> > Aug 13 16:28:50.000 multipathd: SEQNUM=1740
>> >
>> > Aug 13 16:28:50.000 multipathd: UDEVD_EVENT=1
>> >
>> > Aug 13 16:28:50.000 multipathd: DEVNAME=/dev/dm-1
>> >
>> > Aug 13 16:29:00.000 multipathd: 8:208: reinstated
>> >
>> > Aug 13 16:29:00.000 multipathd: vol1: remaining active paths: 4
>> >
>> > Aug 13 16:29:00.000 multipathd: sdj: rdac prio = 3
>> >
>> > Aug 13 16:29:00.000 multipathd: sdn: rdac prio = 3
>> >
>> > Aug 13 16:29:00.000 multipathd: sdb: rdac prio = 0
>> >
>> > Aug 13 16:29:00.000 multipathd: sdd: rdac prio = 0
>> >
>> > Aug 13 16:29:00.000 multipathd: vol1: rr_weight = 2 (LUN setting)
>> >
>> > Aug 13 16:29:00.000 multipathd: vol1: pgfailback = -2 (controller
>> setting)
>> >
>> > Aug 13 16:29:00.000 multipathd: uevent 'change' from
>> > '/devices/virtual/block/dm-1'
>> >
>> > Aug 13 16:29:00.000 multipathd: UDEV_LOG=3
>> >
>> > Aug 13 16:29:00.000 multipathd: ACTION=change
>> >
>> > Aug 13 16:29:00.000 multipathd: DEVPATH=/devices/virtual/block/dm-1
>> >
>> > Aug 13 16:29:00.000 multipathd: SUBSYSTEM=block
>> >
>> > Aug 13 16:29:00.000 multipathd: DM_TARGET=multipath
>> >
>> > Aug 13 16:29:00.000 multipathd: DM_ACTION=PATH_REINSTATED
>> >
>> > Aug 13 16:29:00.000 multipathd: DM_SEQNUM=4
>> >
>> > Aug 13 16:29:00.000 multipathd: DM_PATH=8:208
>> >
>> > Aug 13 16:29:00.000 multipathd: DM_NR_VALID_PATHS=4
>> >
>> > Aug 13 16:29:00.000 multipathd: DM_NAME=vol1
>> >
>> > Aug 13 16:29:00.000 multipathd:
>> > DM_UUID=mpath-3600a0b800048335200001e5d48b68a9b
>> >
>> > Aug 13 16:29:00.000 multipathd: MAJOR=253
>> >
>> > Aug 13 16:29:00.000 multipathd: MINOR=1
>> >
>> > Aug 13 16:29:00.000 multipathd: DEVTYPE=disk
>> >
>> > Aug 13 16:29:00.000 multipathd: SEQNUM=1741
>> >
>> > Aug 13 16:29:00.000 multipathd: UDEVD_EVENT=1
>> >
>> > Aug 13 16:29:00.000 multipathd: DEVNAME=/dev/dm-1
>> >
>> > Aug 13 16:29:02.753 kernel: device-mapper: multipath: Failing path
>> 8:208.
>> >
>> > Aug 13 16:29:02.000 multipathd: vol1: rr_weight = 2 (LUN setting)
>> >
>> > Aug 13 16:29:02.000 multipathd: vol1: pgfailback = -2 (controller
>> setting)
>> >
>> > Aug 13 16:29:02.000 multipathd: pg_timeout = NONE (internal default)
>> >
>> > Aug 13 16:29:02.000 multipathd: 8:208: mark as failed
>> >
>> > Aug 13 16:29:02.000 multipathd: uevent 'change' from
>> > '/devices/virtual/block/dm-1'
>> >
>> > Aug 13 16:29:02.000 multipathd: UDEV_LOG=3
>> >
>> > Aug 13 16:29:02.000 multipathd: ACTION=change
>> >
>> > Aug 13 16:29:02.000 multipathd: DEVPATH=/devices/virtual/block/dm-1
>> >
>> > Aug 13 16:29:02.000 multipathd: SUBSYSTEM=block
>> >
>> > Aug 13 16:29:02.000 multipathd: DM_TARGET=multipath
>> >
>> > Aug 13 16:29:02.000 multipathd: DM_ACTION=PATH_FAILED
>> >
>> > Aug 13 16:29:02.000 multipathd: DM_SEQNUM=5
>> >
>> > Aug 13 16:29:02.000 multipathd: DM_PATH=8:208
>> >
>> > Aug 13 16:29:02.000 multipathd: DM_NR_VALID_PATHS=3
>> >
>> > Aug 13 16:29:02.000 multipathd: DM_NAME=vol1
>> >
>> > Aug 13 16:29:02.000 multipathd:
>> > DM_UUID=mpath-3600a0b800048335200001e5d48b68a9b
>> >
>> > Aug 13 16:29:02.000 multipathd: MAJOR=253
>> >
>> > Aug 13 16:29:02.000 multipathd: MINOR=1
>> >
>> > Aug 13 16:29:02.000 multipathd: DEVTYPE=disk
>> >
>> > Aug 13 16:29:02.000 multipathd: SEQNUM=1742
>> >
>> > Aug 13 16:29:02.000 multipathd: UDEVD_EVENT=1
>> >
>> > Aug 13 16:29:02.000 multipathd: DEVNAME=/dev/dm-1
>> >
>> > Aug 13 16:29:10.000 multipathd: 8:208: reinstated
>> >
>> > Aug 13 16:29:10.000 multipathd: vol1: remaining active paths: 4
>> >
>> > Aug 13 16:29:10.000 multipathd: sdj: rdac prio = 3
>> >
>> > Aug 13 16:29:10.000 multipathd: sdn: rdac prio = 3
>> >
>> > Aug 13 16:29:10.000 multipathd: sdb: rdac prio = 0
>> >
>> > Aug 13 16:29:10.000 multipathd: sdd: rdac prio = 0
>> >
>> > Aug 13 16:29:10.000 multipathd: vol1: rr_weight = 2 (LUN setting)
>> >
>> > Aug 13 16:29:10.000 multipathd: vol1: pgfailback = -2 (controller
>> setting)
>> >
>> > Aug 13 16:29:10.000 multipathd: uevent 'change' from
>> > '/devices/virtual/block/dm-1'
>> >
>> > Aug 13 16:29:10.000 multipathd: UDEV_LOG=3
>> >
>> > Aug 13 16:29:10.000 multipathd: ACTION=change
>> >
>> > Aug 13 16:29:10.000 multipathd: DEVPATH=/devices/virtual/block/dm-1
>> >
>> > Aug 13 16:29:10.000 multipathd: SUBSYSTEM=block
>> >
>> > Aug 13 16:29:10.000 multipathd: DM_TARGET=multipath
>> >
>> > Aug 13 16:29:10.000 multipathd: DM_ACTION=PATH_REINSTATED
>> >
>> > Aug 13 16:29:10.000 multipathd: DM_SEQNUM=6
>> >
>> > Aug 13 16:29:10.000 multipathd: DM_PATH=8:208
>> >
>> > Aug 13 16:29:10.000 multipathd: DM_NR_VALID_PATHS=4
>> >
>> > Aug 13 16:29:10.000 multipathd: DM_NAME=vol1
>> >
>> > Aug 13 16:29:10.000 multipathd:
>> > DM_UUID=mpath-3600a0b800048335200001e5d48b68a9b
>> >
>> > Aug 13 16:29:10.000 multipathd: MAJOR=253
>> >
>> > Aug 13 16:29:10.000 multipathd: MINOR=1
>> >
>> > Aug 13 16:29:10.000 multipathd: DEVTYPE=disk
>> >
>> > Aug 13 16:29:10.000 multipathd: SEQNUM=1743
>> >
>> > Aug 13 16:29:10.000 multipathd: UDEVD_EVENT=1
>> >
>> > Aug 13 16:29:10.000 multipathd: DEVNAME=/dev/dm-1
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Thu, Aug 13, 2009 at 1:27 PM, Stewart Smith <stew@cleepdar.com>
>> wrote:
>> >
>> >
>> >
>> > after a fresh, multipath -F and start of multipathd with -v 2 I see the
>> > following messages.
>> >
>> >
>> >
>> > After starting multipathd I mounted /dev/mapper/vol1 and generated some
>> > simple I/O to it using dd
>> >
>> >
>> >
>> >
>> >
>> > Aug 13 16:23:14.888 localhost kernel: device-mapper: multipath: Failing
>> path
>> > 8:208.
>> >
>> > Aug 13 16:23:14.000 localhost multipathd: 8:208: mark as failed
>> >
>> > Aug 13 16:23:16.000 localhost multipathd: 8:208: reinstated
>> >
>> > Aug 13 16:23:30.462 localhost kernel: device-mapper: multipath: Failing
>> path
>> > 8:208.
>> >
>> > Aug 13 16:23:30.000 localhost multipathd: 8:208: mark as failed
>> >
>> > Aug 13 16:23:39.000 localhost multipathd: 8:208: reinstated
>> >
>> > Aug 13 16:23:46.430 localhost kernel: device-mapper: multipath: Failing
>> path
>> > 8:208.
>> >
>> > Aug 13 16:23:46.000 localhost multipathd: 8:208: mark as failed
>> >
>> > Aug 13 16:23:51.041 localhost kernel: device-mapper: multipath: Failing
>> path
>> > 8:208.
>> >
>> > Aug 13 16:23:51.000 localhost multipathd: 8:208: mark as failed
>> >
>> > Aug 13 16:23:59.000 localhost multipathd: 8:208: reinstated
>> >
>> > Aug 13 16:24:06.465 localhost kernel: device-mapper: multipath: Failing
>> path
>> > 8:208.
>> >
>> > Aug 13 16:24:06.000 localhost multipathd: 8:208: mark as failed
>> >
>> > Aug 13 16:24:09.000 localhost multipathd: 8:208: reinstated
>> >
>> >
>> >
>> >
>> >
>> > Thanks,
>> >
>> > --
>> >
>> > Stew
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Thu, Aug 13, 2009 at 12:42 PM, Moger, Babu <Babu.Moger@lsi.com>
>> wrote:
>> >
>> > Do you have /var/log/messages file for this problem?
>> >
>> > Thanks
>> > Babu Moger
>> >
>> >> -----Original Message-----
>> >> From: dm-devel-bounces@redhat.com [mailto:dm-devel-bounces@redhat.com]
>> On
>> >> Behalf Of Stewart Smith
>> >> Sent: Thursday, August 13, 2009 1:51 PM
>> >> To: dm-devel@redhat.com
>> >> Subject: [dm-devel] rdac path failure - Sun 6140
>> >>
>> >> Hello All,
>> >>
>> >> I am seeing many of these messages when my Sun 6140 array is under
>> heavy
>> >> I/O
>> >> device-mapper: multipath: Failing path 8:208.
>> >> device-mapper: multipath: Failing path 8:208.
>> >> device-mapper: multipath: Failing path 8:208.
>> >> device-mapper: multipath: Failing path 8:208.
>> >> device-mapper: multipath: Failing path 8:208.
>> >>
>> >>
>> >> I am running a Fedora 10 server, with two fiber connections to two
>> >> different switches. Both controllers on the 6140 have one connection
>> >> to each switch as well. The end result is that I see four paths to
>> >> each LUN.
>> >>
>> >> When the volume is mounted and under significant load I see the
>> >> messages above every few seconds. They seem to appear every
>> >> "no_path_retry" seconds.
>> >>
>> >> The 6140 controller firmware is up to date at version 07.50.08.10 and
>> >> I have installed the latest firmware for my Emulex LPe11002 cards. I
>> >> have reproduced the problem using both Cisco MDS and Brocade fiber
>> >> channel switches as well.
>> >>
>> >> Using CAM, I have set the initiator Host Type to "Linux" at the
>> >> moment. I have tried other options as well without success.
>> >>
>> >> I have NOT installed the RDAC drivers from either Sun or LSI -
>> >> primarily because they do not seem to build on my Fedora 10 kernel.
>> >>
>> >> Any ideas would be greatly appreciated!!!
>> >>
>> >> configs and debugging multipathd output is below.
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> Kernel: 2.6.27.24-170.2.68.fc10.x86_64
>> >>
>> >> # multipath -lll
>> >> vol1 (3600a0b800048335200001e5d48b68a9b) dm-1 SUN,CSM200_R
>> >> [size=12T][features=1 queue_if_no_path][hwhandler=1 rdac][rw]
>> >> \_ round-robin 0 [prio=6][active]
>> >> \_ 5:0:1:2 sdj 8:144 [active][ready]
>> >> \_ 2:0:1:2 sdn 8:208 [active][ready]
>> >> \_ round-robin 0 [prio=0][enabled]
>> >> \_ 2:0:0:2 sdb 8:16 [active][ghost]
>> >> \_ 5:0:0:2 sdd 8:48 [active][ghost]
>> >>
>> >>
>> >> # cat /etc/multipath.conf
>> >>
>> >> blacklist {
>> >> devnode "^sd[a-z][[0-9]*]"
>> >> devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
>> >> devnode "^hd[a-z][0-9]*"
>> >> devnode "^cciss!c[0-9]d[0-9](p[0-9]*)*"
>> >> }
>> >>
>> >> defaults {
>> >> udev_dir /dev
>> >> polling_interval 10
>> >> selector "round-robin 0"
>> >> path_grouping_policy multibus
>> >> getuid_callout "/sbin/scsi_id --whitelisted /dev/%n"
>> >> prio alua
>> >> path_checker readsector0
>> >> rr_min_io 100
>> >> max_fds 8192
>> >> rr_weight priorities
>> >> failback immediate
>> >> no_path_retry fail
>> >> user_friendly_names yes
>> >> }
>> >> devices {
>> >> device {
>> >> vendor "SUN"
>> >> product "CSM200_R"
>> >> product_blacklist "Universal Xport"
>> >> getuid_callout "/sbin/scsi_id --whitelisted
>> >> /dev/%n"
>> >> features "0"
>> >> hardware_handler "1 rdac"
>> >> path_selector "round-robin 0"
>> >> path_grouping_policy group_by_prio
>> >> failback immediate
>> >> rr_weight uniform
>> >> no_path_retry queue
>> >> rr_min_io 1000
>> >> path_checker rdac
>> >> prio rdac
>> >> }
>> >> }
>> >>
>> >> multipaths {
>> >> multipath {
>> >> wwid
>> 3600a0b800048335200001e5d48b68a9b
>> >> alias vol1
>> >> rr_weight priorities
>> >> no_path_retry 5
>> >> rr_min_io 100
>> >> }
>> >> }
>> >>
>> >>
>> >>
>> >> # multipathd -d v3
>> >>
>> >>
>> >> Aug 13 14:48:53 | sdb: ownership set to vol1
>> >> Aug 13 14:48:53 | sdb: not found in pathvec
>> >> Aug 13 14:48:53 | sdb: mask = 0xc
>> >> Aug 13 14:48:53 | sdb: path checker = rdac (controller setting)
>> >> Aug 13 14:48:53 | sdb: state = 4
>> >> Aug 13 14:48:53 | sdb: rdac prio = 0
>> >> Aug 13 14:48:53 | sdd: ownership set to vol1
>> >> Aug 13 14:48:53 | sdd: not found in pathvec
>> >> Aug 13 14:48:53 | sdd: mask = 0xc
>> >> Aug 13 14:48:53 | sdd: path checker = rdac (controller setting)
>> >> Aug 13 14:48:53 | sdd: state = 4
>> >> Aug 13 14:48:53 | sdd: rdac prio = 0
>> >> Aug 13 14:48:53 | sdj: ownership set to vol1
>> >> Aug 13 14:48:53 | sdj: not found in pathvec
>> >> Aug 13 14:48:53 | sdj: mask = 0xc
>> >> Aug 13 14:48:53 | sdj: path checker = rdac (controller setting)
>> >> Aug 13 14:48:53 | sdj: state = 2
>> >> Aug 13 14:48:53 | sdj: rdac prio = 3
>> >> Aug 13 14:48:53 | sdn: ownership set to vol1
>> >> Aug 13 14:48:53 | sdn: not found in pathvec
>> >> Aug 13 14:48:53 | sdn: mask = 0xc
>> >> Aug 13 14:48:53 | sdn: path checker = rdac (controller setting)
>> >> Aug 13 14:48:53 | sdn: state = 2
>> >> Aug 13 14:48:53 | sdn: rdac prio = 3
>> >> Aug 13 14:48:53 | vol1: pgfailback = -2 (controller setting)
>> >> Aug 13 14:48:53 | vol1: pgpolicy = group_by_prio (controller setting)
>> >> Aug 13 14:48:53 | vol1: selector = round-robin 0 (controller setting)
>> >> Aug 13 14:48:53 | vol1: features = 0 (controller setting)
>> >> Aug 13 14:48:53 | vol1: hwhandler = 1 rdac (controller setting)
>> >> Aug 13 14:48:53 | vol1: rr_weight = 2 (LUN setting)
>> >> Aug 13 14:48:53 | vol1: minio = 100 (LUN setting)
>> >> Aug 13 14:48:53 | vol1: no_path_retry = 5 (multipath setting)
>> >> Aug 13 14:48:53 | pg_timeout = NONE (internal default)
>> >> Aug 13 14:48:53 | vol1: set ACT_CREATE (map does not exist)
>> >> create: vol1 (3600a0b800048335200001e5d48b68a9b) n/a SUN,CSM200_R
>> >> [size=12T][features=0][hwhandler=1 rdac][n/a]
>> >> \_ round-robin 0 [prio=6][undef]
>> >> \_ 5:0:1:2 sdj 8:144 [undef][ready]
>> >> \_ 2:0:1:2 sdn 8:208 [undef][ready]
>> >> \_ round-robin 0 [prio=0][undef]
>> >> \_ 2:0:0:2 sdb 8:16 [undef][ghost]
>> >> \_ 5:0:0:2 sdd 8:48 [undef][ghost]
>> >>
>> >
>> >> --
>> >> dm-devel mailing list
>> >> dm-devel@redhat.com
>> >> https://www.redhat.com/mailman/listinfo/dm-devel
>> >
>> > --
>> > dm-devel mailing list
>> > dm-devel@redhat.com
>> > https://www.redhat.com/mailman/listinfo/dm-devel
>> >
>> >
>> >
>> >
>>
>> --
>> dm-devel mailing list
>> dm-devel@redhat.com
>> https://www.redhat.com/mailman/listinfo/dm-devel
>
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: rdac path failure - Sun 6140
2009-08-14 20:26 ` Stewart Smith
@ 2009-08-14 22:22 ` Chandra Seetharaman
0 siblings, 0 replies; 9+ messages in thread
From: Chandra Seetharaman @ 2009-08-14 22:22 UTC (permalink / raw)
To: device-mapper development
On Fri, 2009-08-14 at 13:26 -0700, Stewart Smith wrote:
> Interesting.
>
> I just grabbed 2.6.30.4 from kernel.org and I can no longer reproduce
> the issue with that kernel. I did not add in your debugging version
> of scsi_dh_rdac.c yet.
>
> Somewhere between vmlinuz-2.6.29.4-167.fc11 and the generic 2.6.30.4
> (using the same kernel .config) this problem was fixed.
That is good :)
I do not have a fc11 install.... If it is not a hassle, can you send a
diff of scsi_dh_rdac.c and dm-mpath.c, we should understand what fixed
the problem.
>
> Its now running over 350 MB/s with 16 sequential writers using both
> active paths.
>
>
>
>
> # fio --filename=/san0/vol1/test --direct=1 --rw=write --bs=8m
> --size=32G --numjobs=16 --runtime=60 --group_reporting --name=file1
>
> Run status group 0 (all jobs):
> WRITE: io=21,416MiB, aggrb=355MiB/s, minb=355MiB/s, maxb=355MiB/s,
> mint=60342msec, maxt=60342msec
>
> Disk stats (read/write):
> dm-1: ios=3/85697, merge=0/0, ticks=11/21809639, in_queue=21810085,
> util=99.94%, aggrios=0/11402, aggrmerge=0/10022, aggrticks=2/2805880,
> aggrin_queue=2805875, aggrutil=97.09%
> sdf: ios=0/22732, merge=0/20168, ticks=0/6090166,
> in_queue=6090145, util=97.09%
> sdn: ios=3/22876, merge=0/19921, ticks=11/5133355,
> in_queue=5133356, util=96.16%
> sdb: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
> sdj: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
>
>
> Thanks!
> --
> Stewart
>
>
>
>
> On Fri, Aug 14, 2009 at 9:21 AM, Moger, Babu<Babu.Moger@lsi.com> wrote:
> >> -----Original Message-----
> >> From: dm-devel-bounces@redhat.com [mailto:dm-devel-bounces@redhat.com] On
> >> Behalf Of Stewart Smith
> >> Sent: Friday, August 14, 2009 10:58 AM
> >> To: device-mapper development
> >> Subject: Re: [dm-devel] rdac path failure - Sun 6140
> >>
> >> I will test this today.
> >>
> >> Does everything look OK from a configuration standpoint? Should the
> >
> > I have verified your configuration file. It looks fine..
> >
> >> RDAC virtual HBA drivers from LSI be a requirement? I am not
> >> currently using them.
> >
> > You can't use both the solutions at once. You are fine..
> >
> >> Thank you,
> >> --
> >> Stew
> >>
> >>
> >>
> >>
> >> On Thu, Aug 13, 2009 at 5:05 PM, Moger, Babu<Babu.Moger@lsi.com> wrote:
> >> > Stew,
> >> >
> >> > I don’t see much information about this failure in the logs. Right
> >> now
> >> > device handlers don’t provide much information on failures. We are
> >> working
> >> > on to add some more debug levels. I am attaching my draft code
> >> > (scsi_dh_rdac.c) here. Please use this only for your testing. It is not
> >> > been approved/reviewed yet. I still need to submit this one to community
> >> for
> >> > approval. The code is attached. Please replace this file with
> >> > scsi_dh_rdac.c in the directory /driver/scsi/device_handlers and rebuild
> >> the
> >> > kernel. This should give more information from the target point of
> >> view.
> >> > Please send me the /var/log/messages file after the failure. Let see if
> >> we
> >> > can get more information..
> >> >
> >> >
> >> >
> >> > Thanks
> >> >
> >> > Babu Moger
> >> >
> >> > ________________________________
> >> >
> >> > From: dm-devel-bounces@redhat.com [mailto:dm-devel-bounces@redhat.com]
> >> On
> >> > Behalf Of Stewart Smith
> >> > Sent: Thursday, August 13, 2009 3:35 PM
> >> > To: device-mapper development
> >> > Subject: Re: [dm-devel] rdac path failure - Sun 6140
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > Same sequence of events, with multipathd -v3
> >> >
> >> >
> >> >
> >> > Aug 13 16:28:48.627 kernel: device-mapper: multipath: Failing path
> >> 8:208.
> >> >
> >> > Aug 13 16:28:48.000 multipathd: vol1: rr_weight = 2 (LUN setting)
> >> >
> >> > Aug 13 16:28:48.000 multipathd: vol1: pgfailback = -2 (controller
> >> setting)
> >> >
> >> > Aug 13 16:28:48.000 multipathd: pg_timeout = NONE (internal default)
> >> >
> >> > Aug 13 16:28:48.000 multipathd: 8:208: mark as failed
> >> >
> >> > Aug 13 16:28:48.000 multipathd: uevent 'change' from
> >> > '/devices/virtual/block/dm-1'
> >> >
> >> > Aug 13 16:28:48.000 multipathd: UDEV_LOG=3
> >> >
> >> > Aug 13 16:28:48.000 multipathd: ACTION=change
> >> >
> >> > Aug 13 16:28:48.000 multipathd: DEVPATH=/devices/virtual/block/dm-1
> >> >
> >> > Aug 13 16:28:48.000 multipathd: SUBSYSTEM=block
> >> >
> >> > Aug 13 16:28:48.000 multipathd: DM_TARGET=multipath
> >> >
> >> > Aug 13 16:28:48.000 multipathd: DM_ACTION=PATH_FAILED
> >> >
> >> > Aug 13 16:28:48.000 multipathd: DM_SEQNUM=1
> >> >
> >> > Aug 13 16:28:48.000 multipathd: DM_PATH=8:208
> >> >
> >> > Aug 13 16:28:48.000 multipathd: DM_NR_VALID_PATHS=3
> >> >
> >> > Aug 13 16:28:48.000 multipathd: DM_NAME=vol1
> >> >
> >> > Aug 13 16:28:48.000 multipathd:
> >> > DM_UUID=mpath-3600a0b800048335200001e5d48b68a9b
> >> >
> >> > Aug 13 16:28:48.000 multipathd: MAJOR=253
> >> >
> >> > Aug 13 16:28:48.000 multipathd: MINOR=1
> >> >
> >> > Aug 13 16:28:48.000 multipathd: DEVTYPE=disk
> >> >
> >> > Aug 13 16:28:48.000 multipathd: SEQNUM=1738
> >> >
> >> > Aug 13 16:28:48.000 multipathd: UDEVD_EVENT=1
> >> >
> >> > Aug 13 16:28:48.000 multipathd: DEVNAME=/dev/dm-1
> >> >
> >> > Aug 13 16:28:50.000 multipathd: 8:208: reinstated
> >> >
> >> > Aug 13 16:28:50.000 multipathd: vol1: remaining active paths: 4
> >> >
> >> > Aug 13 16:28:50.000 multipathd: sdj: rdac prio = 3
> >> >
> >> > Aug 13 16:28:50.000 multipathd: sdn: rdac prio = 3
> >> >
> >> > Aug 13 16:28:50.000 multipathd: sdb: rdac prio = 0
> >> >
> >> > Aug 13 16:28:50.000 multipathd: sdd: rdac prio = 0
> >> >
> >> > Aug 13 16:28:50.763 kernel: device-mapper: multipath: Failing path
> >> 8:208.
> >> >
> >> > Aug 13 16:28:50.000 multipathd: uevent 'change' from
> >> > '/devices/virtual/block/dm-1'
> >> >
> >> > Aug 13 16:28:50.000 multipathd: UDEV_LOG=3
> >> >
> >> > Aug 13 16:28:50.000 multipathd: ACTION=change
> >> >
> >> > Aug 13 16:28:50.000 multipathd: DEVPATH=/devices/virtual/block/dm-1
> >> >
> >> > Aug 13 16:28:50.000 multipathd: SUBSYSTEM=block
> >> >
> >> > Aug 13 16:28:50.000 multipathd: DM_TARGET=multipath
> >> >
> >> > Aug 13 16:28:50.000 multipathd: DM_ACTION=PATH_REINSTATED
> >> >
> >> > Aug 13 16:28:50.000 multipathd: DM_SEQNUM=2
> >> >
> >> > Aug 13 16:28:50.000 multipathd: DM_PATH=8:208
> >> >
> >> > Aug 13 16:28:50.000 multipathd: DM_NR_VALID_PATHS=4
> >> >
> >> > Aug 13 16:28:50.000 multipathd: DM_NAME=vol1
> >> >
> >> > Aug 13 16:28:50.000 multipathd:
> >> > DM_UUID=mpath-3600a0b800048335200001e5d48b68a9b
> >> >
> >> > Aug 13 16:28:50.000 multipathd: MAJOR=253
> >> >
> >> > Aug 13 16:28:50.000 multipathd: MINOR=1
> >> >
> >> > Aug 13 16:28:50.000 multipathd: DEVTYPE=diskAug 13 16:28:50.000
> >> > multipathd: SEQNUM=1739Aug 13 16:28:50.000 multipathd: UDEVD_EVENT=1
> >> >
> >> > Aug 13 16:28:50.000 multipathd: vol1: rr_weight = 2 (LUN setting)
> >> >
> >> > Aug 13 16:28:50.000 multipathd: vol1: pgfailback = -2 (controller
> >> setting)
> >> >
> >> > Aug 13 16:28:50.000 multipathd: pg_timeout = NONE (internal default)
> >> >
> >> > Aug 13 16:28:50.000 multipathd: 8:208: mark as failed
> >> >
> >> > Aug 13 16:28:50.000 multipathd: vol1: remaining active paths: 3
> >> >
> >> > Aug 13 16:28:50.000 multipathd: vol1: rr_weight = 2 (LUN setting)
> >> >
> >> > Aug 13 16:28:50.000 multipathd: vol1: pgfailback = -2 (controller
> >> setting)
> >> >
> >> > Aug 13 16:28:50.000 multipathd: uevent 'change' from
> >> > '/devices/virtual/block/dm-1'
> >> >
> >> > Aug 13 16:28:50.000 multipathd: UDEV_LOG=3
> >> >
> >> > Aug 13 16:28:50.000 multipathd: ACTION=change
> >> >
> >> > Aug 13 16:28:50.000 multipathd: DEVPATH=/devices/virtual/block/dm-1
> >> >
> >> > Aug 13 16:28:50.000 multipathd: SUBSYSTEM=block
> >> >
> >> > Aug 13 16:28:50.000 multipathd: DM_TARGET=multipath
> >> >
> >> > Aug 13 16:28:50.000 multipathd: DM_ACTION=PATH_FAILED
> >> >
> >> > Aug 13 16:28:50.000 multipathd: DM_SEQNUM=3
> >> >
> >> > Aug 13 16:28:50.000 multipathd: DM_PATH=8:208
> >> >
> >> > Aug 13 16:28:50.000 multipathd: DM_NR_VALID_PATHS=3
> >> >
> >> > Aug 13 16:28:50.000 multipathd: DM_NAME=vol1
> >> >
> >> > Aug 13 16:28:50.000 multipathd:
> >> > DM_UUID=mpath-3600a0b800048335200001e5d48b68a9b
> >> >
> >> > Aug 13 16:28:50.000 multipathd: MAJOR=253
> >> >
> >> > Aug 13 16:28:50.000 multipathd: MINOR=1
> >> >
> >> > Aug 13 16:28:50.000 multipathd: DEVTYPE=disk
> >> >
> >> > Aug 13 16:28:50.000 multipathd: SEQNUM=1740
> >> >
> >> > Aug 13 16:28:50.000 multipathd: UDEVD_EVENT=1
> >> >
> >> > Aug 13 16:28:50.000 multipathd: DEVNAME=/dev/dm-1
> >> >
> >> > Aug 13 16:29:00.000 multipathd: 8:208: reinstated
> >> >
> >> > Aug 13 16:29:00.000 multipathd: vol1: remaining active paths: 4
> >> >
> >> > Aug 13 16:29:00.000 multipathd: sdj: rdac prio = 3
> >> >
> >> > Aug 13 16:29:00.000 multipathd: sdn: rdac prio = 3
> >> >
> >> > Aug 13 16:29:00.000 multipathd: sdb: rdac prio = 0
> >> >
> >> > Aug 13 16:29:00.000 multipathd: sdd: rdac prio = 0
> >> >
> >> > Aug 13 16:29:00.000 multipathd: vol1: rr_weight = 2 (LUN setting)
> >> >
> >> > Aug 13 16:29:00.000 multipathd: vol1: pgfailback = -2 (controller
> >> setting)
> >> >
> >> > Aug 13 16:29:00.000 multipathd: uevent 'change' from
> >> > '/devices/virtual/block/dm-1'
> >> >
> >> > Aug 13 16:29:00.000 multipathd: UDEV_LOG=3
> >> >
> >> > Aug 13 16:29:00.000 multipathd: ACTION=change
> >> >
> >> > Aug 13 16:29:00.000 multipathd: DEVPATH=/devices/virtual/block/dm-1
> >> >
> >> > Aug 13 16:29:00.000 multipathd: SUBSYSTEM=block
> >> >
> >> > Aug 13 16:29:00.000 multipathd: DM_TARGET=multipath
> >> >
> >> > Aug 13 16:29:00.000 multipathd: DM_ACTION=PATH_REINSTATED
> >> >
> >> > Aug 13 16:29:00.000 multipathd: DM_SEQNUM=4
> >> >
> >> > Aug 13 16:29:00.000 multipathd: DM_PATH=8:208
> >> >
> >> > Aug 13 16:29:00.000 multipathd: DM_NR_VALID_PATHS=4
> >> >
> >> > Aug 13 16:29:00.000 multipathd: DM_NAME=vol1
> >> >
> >> > Aug 13 16:29:00.000 multipathd:
> >> > DM_UUID=mpath-3600a0b800048335200001e5d48b68a9b
> >> >
> >> > Aug 13 16:29:00.000 multipathd: MAJOR=253
> >> >
> >> > Aug 13 16:29:00.000 multipathd: MINOR=1
> >> >
> >> > Aug 13 16:29:00.000 multipathd: DEVTYPE=disk
> >> >
> >> > Aug 13 16:29:00.000 multipathd: SEQNUM=1741
> >> >
> >> > Aug 13 16:29:00.000 multipathd: UDEVD_EVENT=1
> >> >
> >> > Aug 13 16:29:00.000 multipathd: DEVNAME=/dev/dm-1
> >> >
> >> > Aug 13 16:29:02.753 kernel: device-mapper: multipath: Failing path
> >> 8:208.
> >> >
> >> > Aug 13 16:29:02.000 multipathd: vol1: rr_weight = 2 (LUN setting)
> >> >
> >> > Aug 13 16:29:02.000 multipathd: vol1: pgfailback = -2 (controller
> >> setting)
> >> >
> >> > Aug 13 16:29:02.000 multipathd: pg_timeout = NONE (internal default)
> >> >
> >> > Aug 13 16:29:02.000 multipathd: 8:208: mark as failed
> >> >
> >> > Aug 13 16:29:02.000 multipathd: uevent 'change' from
> >> > '/devices/virtual/block/dm-1'
> >> >
> >> > Aug 13 16:29:02.000 multipathd: UDEV_LOG=3
> >> >
> >> > Aug 13 16:29:02.000 multipathd: ACTION=change
> >> >
> >> > Aug 13 16:29:02.000 multipathd: DEVPATH=/devices/virtual/block/dm-1
> >> >
> >> > Aug 13 16:29:02.000 multipathd: SUBSYSTEM=block
> >> >
> >> > Aug 13 16:29:02.000 multipathd: DM_TARGET=multipath
> >> >
> >> > Aug 13 16:29:02.000 multipathd: DM_ACTION=PATH_FAILED
> >> >
> >> > Aug 13 16:29:02.000 multipathd: DM_SEQNUM=5
> >> >
> >> > Aug 13 16:29:02.000 multipathd: DM_PATH=8:208
> >> >
> >> > Aug 13 16:29:02.000 multipathd: DM_NR_VALID_PATHS=3
> >> >
> >> > Aug 13 16:29:02.000 multipathd: DM_NAME=vol1
> >> >
> >> > Aug 13 16:29:02.000 multipathd:
> >> > DM_UUID=mpath-3600a0b800048335200001e5d48b68a9b
> >> >
> >> > Aug 13 16:29:02.000 multipathd: MAJOR=253
> >> >
> >> > Aug 13 16:29:02.000 multipathd: MINOR=1
> >> >
> >> > Aug 13 16:29:02.000 multipathd: DEVTYPE=disk
> >> >
> >> > Aug 13 16:29:02.000 multipathd: SEQNUM=1742
> >> >
> >> > Aug 13 16:29:02.000 multipathd: UDEVD_EVENT=1
> >> >
> >> > Aug 13 16:29:02.000 multipathd: DEVNAME=/dev/dm-1
> >> >
> >> > Aug 13 16:29:10.000 multipathd: 8:208: reinstated
> >> >
> >> > Aug 13 16:29:10.000 multipathd: vol1: remaining active paths: 4
> >> >
> >> > Aug 13 16:29:10.000 multipathd: sdj: rdac prio = 3
> >> >
> >> > Aug 13 16:29:10.000 multipathd: sdn: rdac prio = 3
> >> >
> >> > Aug 13 16:29:10.000 multipathd: sdb: rdac prio = 0
> >> >
> >> > Aug 13 16:29:10.000 multipathd: sdd: rdac prio = 0
> >> >
> >> > Aug 13 16:29:10.000 multipathd: vol1: rr_weight = 2 (LUN setting)
> >> >
> >> > Aug 13 16:29:10.000 multipathd: vol1: pgfailback = -2 (controller
> >> setting)
> >> >
> >> > Aug 13 16:29:10.000 multipathd: uevent 'change' from
> >> > '/devices/virtual/block/dm-1'
> >> >
> >> > Aug 13 16:29:10.000 multipathd: UDEV_LOG=3
> >> >
> >> > Aug 13 16:29:10.000 multipathd: ACTION=change
> >> >
> >> > Aug 13 16:29:10.000 multipathd: DEVPATH=/devices/virtual/block/dm-1
> >> >
> >> > Aug 13 16:29:10.000 multipathd: SUBSYSTEM=block
> >> >
> >> > Aug 13 16:29:10.000 multipathd: DM_TARGET=multipath
> >> >
> >> > Aug 13 16:29:10.000 multipathd: DM_ACTION=PATH_REINSTATED
> >> >
> >> > Aug 13 16:29:10.000 multipathd: DM_SEQNUM=6
> >> >
> >> > Aug 13 16:29:10.000 multipathd: DM_PATH=8:208
> >> >
> >> > Aug 13 16:29:10.000 multipathd: DM_NR_VALID_PATHS=4
> >> >
> >> > Aug 13 16:29:10.000 multipathd: DM_NAME=vol1
> >> >
> >> > Aug 13 16:29:10.000 multipathd:
> >> > DM_UUID=mpath-3600a0b800048335200001e5d48b68a9b
> >> >
> >> > Aug 13 16:29:10.000 multipathd: MAJOR=253
> >> >
> >> > Aug 13 16:29:10.000 multipathd: MINOR=1
> >> >
> >> > Aug 13 16:29:10.000 multipathd: DEVTYPE=disk
> >> >
> >> > Aug 13 16:29:10.000 multipathd: SEQNUM=1743
> >> >
> >> > Aug 13 16:29:10.000 multipathd: UDEVD_EVENT=1
> >> >
> >> > Aug 13 16:29:10.000 multipathd: DEVNAME=/dev/dm-1
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > On Thu, Aug 13, 2009 at 1:27 PM, Stewart Smith <stew@cleepdar.com>
> >> wrote:
> >> >
> >> >
> >> >
> >> > after a fresh, multipath -F and start of multipathd with -v 2 I see the
> >> > following messages.
> >> >
> >> >
> >> >
> >> > After starting multipathd I mounted /dev/mapper/vol1 and generated some
> >> > simple I/O to it using dd
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > Aug 13 16:23:14.888 localhost kernel: device-mapper: multipath: Failing
> >> path
> >> > 8:208.
> >> >
> >> > Aug 13 16:23:14.000 localhost multipathd: 8:208: mark as failed
> >> >
> >> > Aug 13 16:23:16.000 localhost multipathd: 8:208: reinstated
> >> >
> >> > Aug 13 16:23:30.462 localhost kernel: device-mapper: multipath: Failing
> >> path
> >> > 8:208.
> >> >
> >> > Aug 13 16:23:30.000 localhost multipathd: 8:208: mark as failed
> >> >
> >> > Aug 13 16:23:39.000 localhost multipathd: 8:208: reinstated
> >> >
> >> > Aug 13 16:23:46.430 localhost kernel: device-mapper: multipath: Failing
> >> path
> >> > 8:208.
> >> >
> >> > Aug 13 16:23:46.000 localhost multipathd: 8:208: mark as failed
> >> >
> >> > Aug 13 16:23:51.041 localhost kernel: device-mapper: multipath: Failing
> >> path
> >> > 8:208.
> >> >
> >> > Aug 13 16:23:51.000 localhost multipathd: 8:208: mark as failed
> >> >
> >> > Aug 13 16:23:59.000 localhost multipathd: 8:208: reinstated
> >> >
> >> > Aug 13 16:24:06.465 localhost kernel: device-mapper: multipath: Failing
> >> path
> >> > 8:208.
> >> >
> >> > Aug 13 16:24:06.000 localhost multipathd: 8:208: mark as failed
> >> >
> >> > Aug 13 16:24:09.000 localhost multipathd: 8:208: reinstated
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > Thanks,
> >> >
> >> > --
> >> >
> >> > Stew
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > On Thu, Aug 13, 2009 at 12:42 PM, Moger, Babu <Babu.Moger@lsi.com>
> >> wrote:
> >> >
> >> > Do you have /var/log/messages file for this problem?
> >> >
> >> > Thanks
> >> > Babu Moger
> >> >
> >> >> -----Original Message-----
> >> >> From: dm-devel-bounces@redhat.com [mailto:dm-devel-bounces@redhat.com]
> >> On
> >> >> Behalf Of Stewart Smith
> >> >> Sent: Thursday, August 13, 2009 1:51 PM
> >> >> To: dm-devel@redhat.com
> >> >> Subject: [dm-devel] rdac path failure - Sun 6140
> >> >>
> >> >> Hello All,
> >> >>
> >> >> I am seeing many of these messages when my Sun 6140 array is under
> >> heavy
> >> >> I/O
> >> >> device-mapper: multipath: Failing path 8:208.
> >> >> device-mapper: multipath: Failing path 8:208.
> >> >> device-mapper: multipath: Failing path 8:208.
> >> >> device-mapper: multipath: Failing path 8:208.
> >> >> device-mapper: multipath: Failing path 8:208.
> >> >>
> >> >>
> >> >> I am running a Fedora 10 server, with two fiber connections to two
> >> >> different switches. Both controllers on the 6140 have one connection
> >> >> to each switch as well. The end result is that I see four paths to
> >> >> each LUN.
> >> >>
> >> >> When the volume is mounted and under significant load I see the
> >> >> messages above every few seconds. They seem to appear every
> >> >> "no_path_retry" seconds.
> >> >>
> >> >> The 6140 controller firmware is up to date at version 07.50.08.10 and
> >> >> I have installed the latest firmware for my Emulex LPe11002 cards. I
> >> >> have reproduced the problem using both Cisco MDS and Brocade fiber
> >> >> channel switches as well.
> >> >>
> >> >> Using CAM, I have set the initiator Host Type to "Linux" at the
> >> >> moment. I have tried other options as well without success.
> >> >>
> >> >> I have NOT installed the RDAC drivers from either Sun or LSI -
> >> >> primarily because they do not seem to build on my Fedora 10 kernel.
> >> >>
> >> >> Any ideas would be greatly appreciated!!!
> >> >>
> >> >> configs and debugging multipathd output is below.
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> Kernel: 2.6.27.24-170.2.68.fc10.x86_64
> >> >>
> >> >> # multipath -lll
> >> >> vol1 (3600a0b800048335200001e5d48b68a9b) dm-1 SUN,CSM200_R
> >> >> [size=12T][features=1 queue_if_no_path][hwhandler=1 rdac][rw]
> >> >> \_ round-robin 0 [prio=6][active]
> >> >> \_ 5:0:1:2 sdj 8:144 [active][ready]
> >> >> \_ 2:0:1:2 sdn 8:208 [active][ready]
> >> >> \_ round-robin 0 [prio=0][enabled]
> >> >> \_ 2:0:0:2 sdb 8:16 [active][ghost]
> >> >> \_ 5:0:0:2 sdd 8:48 [active][ghost]
> >> >>
> >> >>
> >> >> # cat /etc/multipath.conf
> >> >>
> >> >> blacklist {
> >> >> devnode "^sd[a-z][[0-9]*]"
> >> >> devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
> >> >> devnode "^hd[a-z][0-9]*"
> >> >> devnode "^cciss!c[0-9]d[0-9](p[0-9]*)*"
> >> >> }
> >> >>
> >> >> defaults {
> >> >> udev_dir /dev
> >> >> polling_interval 10
> >> >> selector "round-robin 0"
> >> >> path_grouping_policy multibus
> >> >> getuid_callout "/sbin/scsi_id --whitelisted /dev/%n"
> >> >> prio alua
> >> >> path_checker readsector0
> >> >> rr_min_io 100
> >> >> max_fds 8192
> >> >> rr_weight priorities
> >> >> failback immediate
> >> >> no_path_retry fail
> >> >> user_friendly_names yes
> >> >> }
> >> >> devices {
> >> >> device {
> >> >> vendor "SUN"
> >> >> product "CSM200_R"
> >> >> product_blacklist "Universal Xport"
> >> >> getuid_callout "/sbin/scsi_id --whitelisted
> >> >> /dev/%n"
> >> >> features "0"
> >> >> hardware_handler "1 rdac"
> >> >> path_selector "round-robin 0"
> >> >> path_grouping_policy group_by_prio
> >> >> failback immediate
> >> >> rr_weight uniform
> >> >> no_path_retry queue
> >> >> rr_min_io 1000
> >> >> path_checker rdac
> >> >> prio rdac
> >> >> }
> >> >> }
> >> >>
> >> >> multipaths {
> >> >> multipath {
> >> >> wwid
> >> 3600a0b800048335200001e5d48b68a9b
> >> >> alias vol1
> >> >> rr_weight priorities
> >> >> no_path_retry 5
> >> >> rr_min_io 100
> >> >> }
> >> >> }
> >> >>
> >> >>
> >> >>
> >> >> # multipathd -d v3
> >> >>
> >> >>
> >> >> Aug 13 14:48:53 | sdb: ownership set to vol1
> >> >> Aug 13 14:48:53 | sdb: not found in pathvec
> >> >> Aug 13 14:48:53 | sdb: mask = 0xc
> >> >> Aug 13 14:48:53 | sdb: path checker = rdac (controller setting)
> >> >> Aug 13 14:48:53 | sdb: state = 4
> >> >> Aug 13 14:48:53 | sdb: rdac prio = 0
> >> >> Aug 13 14:48:53 | sdd: ownership set to vol1
> >> >> Aug 13 14:48:53 | sdd: not found in pathvec
> >> >> Aug 13 14:48:53 | sdd: mask = 0xc
> >> >> Aug 13 14:48:53 | sdd: path checker = rdac (controller setting)
> >> >> Aug 13 14:48:53 | sdd: state = 4
> >> >> Aug 13 14:48:53 | sdd: rdac prio = 0
> >> >> Aug 13 14:48:53 | sdj: ownership set to vol1
> >> >> Aug 13 14:48:53 | sdj: not found in pathvec
> >> >> Aug 13 14:48:53 | sdj: mask = 0xc
> >> >> Aug 13 14:48:53 | sdj: path checker = rdac (controller setting)
> >> >> Aug 13 14:48:53 | sdj: state = 2
> >> >> Aug 13 14:48:53 | sdj: rdac prio = 3
> >> >> Aug 13 14:48:53 | sdn: ownership set to vol1
> >> >> Aug 13 14:48:53 | sdn: not found in pathvec
> >> >> Aug 13 14:48:53 | sdn: mask = 0xc
> >> >> Aug 13 14:48:53 | sdn: path checker = rdac (controller setting)
> >> >> Aug 13 14:48:53 | sdn: state = 2
> >> >> Aug 13 14:48:53 | sdn: rdac prio = 3
> >> >> Aug 13 14:48:53 | vol1: pgfailback = -2 (controller setting)
> >> >> Aug 13 14:48:53 | vol1: pgpolicy = group_by_prio (controller setting)
> >> >> Aug 13 14:48:53 | vol1: selector = round-robin 0 (controller setting)
> >> >> Aug 13 14:48:53 | vol1: features = 0 (controller setting)
> >> >> Aug 13 14:48:53 | vol1: hwhandler = 1 rdac (controller setting)
> >> >> Aug 13 14:48:53 | vol1: rr_weight = 2 (LUN setting)
> >> >> Aug 13 14:48:53 | vol1: minio = 100 (LUN setting)
> >> >> Aug 13 14:48:53 | vol1: no_path_retry = 5 (multipath setting)
> >> >> Aug 13 14:48:53 | pg_timeout = NONE (internal default)
> >> >> Aug 13 14:48:53 | vol1: set ACT_CREATE (map does not exist)
> >> >> create: vol1 (3600a0b800048335200001e5d48b68a9b) n/a SUN,CSM200_R
> >> >> [size=12T][features=0][hwhandler=1 rdac][n/a]
> >> >> \_ round-robin 0 [prio=6][undef]
> >> >> \_ 5:0:1:2 sdj 8:144 [undef][ready]
> >> >> \_ 2:0:1:2 sdn 8:208 [undef][ready]
> >> >> \_ round-robin 0 [prio=0][undef]
> >> >> \_ 2:0:0:2 sdb 8:16 [undef][ghost]
> >> >> \_ 5:0:0:2 sdd 8:48 [undef][ghost]
> >> >>
> >> >
> >> >> --
> >> >> dm-devel mailing list
> >> >> dm-devel@redhat.com
> >> >> https://www.redhat.com/mailman/listinfo/dm-devel
> >> >
> >> > --
> >> > dm-devel mailing list
> >> > dm-devel@redhat.com
> >> > https://www.redhat.com/mailman/listinfo/dm-devel
> >> >
> >> >
> >> >
> >> >
> >>
> >> --
> >> dm-devel mailing list
> >> dm-devel@redhat.com
> >> https://www.redhat.com/mailman/listinfo/dm-devel
> >
> > --
> > dm-devel mailing list
> > dm-devel@redhat.com
> > https://www.redhat.com/mailman/listinfo/dm-devel
> >
>
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2009-08-14 22:22 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2009-08-13 18:50 rdac path failure - Sun 6140 Stewart Smith
2009-08-13 19:42 ` Moger, Babu
2009-08-13 20:27 ` Stewart Smith
2009-08-13 20:34 ` Stewart Smith
2009-08-14 0:05 ` Moger, Babu
2009-08-14 15:58 ` Stewart Smith
2009-08-14 16:21 ` Moger, Babu
2009-08-14 20:26 ` Stewart Smith
2009-08-14 22:22 ` Chandra Seetharaman
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox