* RE: Problem with multipathd and a blacklisted device [not found] <5635940A0F6F36478CF4D01273B16CA32B9C3EDDF8@GVW1096EXB.americas.hpqcorp.net> @ 2009-03-24 13:27 ` Ransegnola, Lori 2009-03-24 13:42 ` malahal 2009-03-31 15:49 ` Benjamin Marzinski 0 siblings, 2 replies; 6+ messages in thread From: Ransegnola, Lori @ 2009-03-24 13:27 UTC (permalink / raw) To: Ransegnola, Lori, dm-devel@redhat.com I sent this question last week and have not heard a thing. Do you have a suggestion of a better mailing list that I should use for this question? Thank you. Lori > -----Original Message----- > From: Ransegnola, Lori > Sent: Thursday, March 19, 2009 3:51 PM > To: dm-devel@redhat.com > Cc: Ransegnola, Lori > Subject: Problem with multipathd and a blacklisted device > > Configuration: > -------------- > > I have a multipath configuration set up where 10 disks are > multipathed and 2 disks are blacklisted and are not > multipathed. Here are the relevant parts of the multipath.conf file: > > defaults { > udev_dir /dev > polling_interval 10 > selector "round-robin 0" > # path_grouping_policy multibus > getuid_callout "/sbin/scsi_id -g -u -s /block/%n" > prio_callout /bin/true > # path_checker readsector0 > path_checker tur > rr_min_io 100 > rr_weight priorities > failback immediate > no_path_retry fail > user_friendly_name yes > } > > blacklist { > # wwid 26353900f02796769 > # devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*" > # devnode "^hd[a-z]" > wwid 35000c5000a41f72b > wwid 35000c5000a41f98b > devnode "^cciss!c[0-9]d[0-9]*" > } > > > The two disks that are not multipathed have udev rules and > are part of a software RAID set. The devices show up as > > /etc/udev/rules.d> ls -l /dev/hpdev > total 0 > lrwxrwxrwx 1 root root 7 Mar 18 10:50 sda1 -> ../sda1 > lrwxrwxrwx 1 root root 7 Mar 18 03:32 sdb1 -> ../sdb1 > > /dev/md0 is created with sda and sdb. Here is the bottom of > the 'mdadm --detail /dev/md0' output: > > /root> mdadm --detail /dev/md0 > /dev/md0: > Version > ... > Failed Devices : 0 > Spare Devices : 0 > > UUID : 94813396:a3e341fc:c5d4b8ba:0617a019 > Events : 0.84 > > Number Major Minor RaidDevice State > 0 8 1 0 active sync /dev/sda1 > 1 8 17 1 active sync /dev/sdb1 > /root> > > I am running on a Linux RHEL5 U2 system. > /etc/udev/rules.d> uname -a > Linux n0 2.6.18-53.el5 #1 SMP Wed Oct 10 16:34:19 EDT 2007 > x86_64 x86_64 x86_64 GNU/Linux /etc/udev/rules.d> multipath > -v Missing option arguement multipath-tools v0.4.7 (03/12, 2006) > ---------------------------- > > Problem: > > This configuration works well until I do some failure testing > with one of the 2 blacklisted devs in the software RAID set. > I found that if I temporarily remove disk sda and put it back > a minute later the disk path, /dev/hpdev/sda1, is removed, > even though it is blacklisted. > > Here are some of the pertinent /var/log/messages lines: > > Mar 19 09:53:19 n0 mdadm: NewArray /dev/md0 Mar 19 10:05:12 > n0 kernel: mptbase: ioc0: LogInfo(0x31170000): > Originator={PL}, Code={IO Device Missing Delay Retry}, > SubCode(0x0000) Mar 19 10:05:40 n0 last message repeated 5 > times Mar 19 10:05:42 n0 kernel: mptbase: ioc0: > LogInfo(0x31130000): Originator={PL}, Code={IO Not Yet > Executed}, SubCode(0x0000) Mar 19 10:05:42 n0 kernel: sd > 0:0:28:0: SCSI error: return code = 0x00010000 Mar 19 > 10:05:42 n0 kernel: end_request: I/O error, dev sda, sector > 256119 Mar 19 10:05:42 n0 kernel: Buffer I/O error on device > sda1, logical block 256056 Mar 19 10:05:42 n0 kernel: Buffer > I/O error on device sda1, logical block 256057 Mar 19 > 10:05:42 n0 kernel: Buffer I/O error on device sda1, logical > block 256058 Mar 19 10:05:42 n0 kernel: Buffer I/O error on > device sda1, logical block 256059 Mar 19 10:05:42 n0 kernel: > Buffer I/O error on device sda1, logical block 256060 Mar 19 > 10:05:42 n0 kernel: Buffer I/O error on device sda1, logical > block 256061 Mar 19 10:05:42 n0 kernel: mptbase: ioc0: > LogInfo(0x31130000): Originator={PL}, Code={IO Not Yet > Executed}, SubCode(0x0000) Mar 19 10:05:42 n0 kernel: Buffer > I/O error on device sda1, logical block 256062 Mar 19 > 10:05:42 n0 kernel: Buffer I/O error on device sda1, logical > block 256063 Mar 19 10:05:42 n0 kernel: sd 0:0:28:0: SCSI > error: return code = 0x00010000 Mar 19 10:05:42 n0 > multipathd: sda: remove path (uevent) Mar 19 10:05:42 n0 > kernel: end_request: I/O error, dev sda, sector 585922495 Mar > 19 10:05:42 n0 kernel: Buffer I/O error on device sda1, > logical block 585922432 Mar 19 10:05:42 n0 xinetd[13096]: > START: hacl-cfgudp pid=6034 from=127.0.0.1 Mar 19 10:05:42 n0 > multipathd: uevent trigger error Mar 19 10:05:42 n0 kernel: > Buffer I/O error on device sda1, logical block 585922433 > > > The sda path is gone: > > /root> ls -l /dev/hpdev > total 0 > lrwxrwxrwx 1 root root 7 Mar 18 03:32 sdb1 -> ../sdb1 /root> > > And I cannot reassemble the software raid set. While the > mdstat looks 'normal' the software raid becomes degraded. > > /etc/udev/rules.d> cat /proc/mdstat > Personalities : [raid1] > md0 : active raid1 sda1[2](F) sdb1[1] > 292961216 blocks [2/1] [_U] > > unused devices: <none> > /etc/udev/rules.d> mdadm --detail /dev/md0 > /dev/md0: > Version : 00.90.03 > Creation Time : Wed Feb 18 14:15:03 2009 > Raid Level : raid1 > Array Size : 292961216 (279.39 GiB 299.99 GB) > Device Size : 292961216 (279.39 GiB 299.99 GB) > Raid Devices : 2 > Total Devices : 2 > Preferred Minor : 0 > Persistence : Superblock is persistent > > Update Time : Thu Mar 19 10:28:45 2009 > State : clean, degraded > Active Devices : 1 > Working Devices : 1 > Failed Devices : 1 > Spare Devices : 0 > > UUID : 94813396:a3e341fc:c5d4b8ba:0617a019 > Events : 0.86 > > Number Major Minor RaidDevice State > 0 0 0 0 removed > 1 8 17 1 active sync /dev/sdb1 > > 2 8 1 - faulty spare > /etc/udev/rules.d> > > The disk is fine and is back in and ready to go. > > So, 1) why does multipathd remove the path for a blacklisted > device? If it is blacklisted, shouldn't multipathd just > leave it alone?? > And 2) what can I do to keep this from happening?? > > Lori Ransegnola > ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: RE: Problem with multipathd and a blacklisted device 2009-03-24 13:27 ` Problem with multipathd and a blacklisted device Ransegnola, Lori @ 2009-03-24 13:42 ` malahal 2009-03-24 21:12 ` Ransegnola, Lori 2009-03-31 15:49 ` Benjamin Marzinski 1 sibling, 1 reply; 6+ messages in thread From: malahal @ 2009-03-24 13:42 UTC (permalink / raw) To: dm-devel Ransegnola, Lori [Lori.Ransegnola@hp.com] wrote: > I sent this question last week and have not heard a thing. Do you have a suggestion of a better mailing list that I should use for this question? > > Thank you. > > Lori > > > -----Original Message----- > > From: Ransegnola, Lori > > Sent: Thursday, March 19, 2009 3:51 PM > > To: dm-devel@redhat.com > > Cc: Ransegnola, Lori > > Subject: Problem with multipathd and a blacklisted device > > > > Configuration: > > -------------- > > > > I have a multipath configuration set up where 10 disks are > > multipathed and 2 disks are blacklisted and are not > > multipathed. Here are the relevant parts of the multipath.conf file: > > > > defaults { > > udev_dir /dev > > polling_interval 10 > > selector "round-robin 0" > > # path_grouping_policy multibus > > getuid_callout "/sbin/scsi_id -g -u -s /block/%n" > > prio_callout /bin/true > > # path_checker readsector0 > > path_checker tur > > rr_min_io 100 > > rr_weight priorities > > failback immediate > > no_path_retry fail > > user_friendly_name yes > > } > > > > blacklist { > > # wwid 26353900f02796769 > > # devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*" > > # devnode "^hd[a-z]" > > wwid 35000c5000a41f72b > > wwid 35000c5000a41f98b > > devnode "^cciss!c[0-9]d[0-9]*" > > } > > > > > > The two disks that are not multipathed have udev rules and > > are part of a software RAID set. The devices show up as > > > > /etc/udev/rules.d> ls -l /dev/hpdev > > total 0 > > lrwxrwxrwx 1 root root 7 Mar 18 10:50 sda1 -> ../sda1 > > lrwxrwxrwx 1 root root 7 Mar 18 03:32 sdb1 -> ../sdb1 > > > > /dev/md0 is created with sda and sdb. Here is the bottom of > > the 'mdadm --detail /dev/md0' output: > > > > /root> mdadm --detail /dev/md0 > > /dev/md0: > > Version > > ... > > Failed Devices : 0 > > Spare Devices : 0 > > > > UUID : 94813396:a3e341fc:c5d4b8ba:0617a019 > > Events : 0.84 > > > > Number Major Minor RaidDevice State > > 0 8 1 0 active sync /dev/sda1 > > 1 8 17 1 active sync /dev/sdb1 > > /root> > > > > I am running on a Linux RHEL5 U2 system. > > /etc/udev/rules.d> uname -a > > Linux n0 2.6.18-53.el5 #1 SMP Wed Oct 10 16:34:19 EDT 2007 > > x86_64 x86_64 x86_64 GNU/Linux /etc/udev/rules.d> multipath > > -v Missing option arguement multipath-tools v0.4.7 (03/12, 2006) > > ---------------------------- > > > > Problem: > > > > This configuration works well until I do some failure testing > > with one of the 2 blacklisted devs in the software RAID set. > > I found that if I temporarily remove disk sda and put it back > > a minute later the disk path, /dev/hpdev/sda1, is removed, even > > though it is blacklisted. What is 'multipath -ll' output before you remove the blacklisted device? What is 'multipath -ll' output after you put the device back a minute later? Thanks, Malahal. ^ permalink raw reply [flat|nested] 6+ messages in thread
* RE: RE: Problem with multipathd and a blacklisted device 2009-03-24 13:42 ` malahal @ 2009-03-24 21:12 ` Ransegnola, Lori 2009-03-31 15:27 ` Ransegnola, Lori 0 siblings, 1 reply; 6+ messages in thread From: Ransegnola, Lori @ 2009-03-24 21:12 UTC (permalink / raw) To: device-mapper development; +Cc: Ransegnola, Lori Here is the 'multipath -ll' output before disk was pulled: /root> ls -l /dev/hpdev total 0 lrwxrwxrwx 1 root root 7 Mar 24 11:53 sda1 -> ../sdm1 lrwxrwxrwx 1 root root 7 Mar 24 11:53 sdb1 -> ../sdn1 /root> multipath -ll multisde (35000c5000a41e633) dm-2 HP,DF300BB6C3 [size=279G][features=0][hwhandler=0] \_ round-robin 0 [prio=0][active] \_ 0:0:5:0 sde 8:64 [active][ready] \_ round-robin 0 [prio=0][enabled] \_ 0:0:19:0 sdq 65:0 [active][ready] 35000c5000a41e44bdm-5 HP,DF300BB6C3 [size=279G][features=0][hwhandler=0] \_ round-robin 0 [prio=0][active] \_ 0:0:8:0 sdh 8:112 [active][ready] \_ round-robin 0 [prio=0][enabled] \_ 0:0:22:0 sdt 65:48 [active][ready] multisdc (35000c5000a41f38f) dm-0 HP,DF300BB6C3 [size=279G][features=0][hwhandler=0] \_ round-robin 0 [prio=0][active] \_ 0:0:3:0 sdc 8:32 [active][ready] \_ round-robin 0 [prio=0][enabled] \_ 0:0:17:0 sdo 8:224 [active][ready] 35000c5000a41e9bbdm-7 HP,DF300BB6C3 [size=279G][features=0][hwhandler=0] \_ round-robin 0 [prio=0][active] \_ 0:0:10:0 sdj 8:144 [active][ready] \_ round-robin 0 [prio=0][enabled] \_ 0:0:24:0 sdv 65:80 [active][ready] 35000c50004acb2cfdm-1 HP,DF0300B8053 [size=279G][features=0][hwhandler=0] \_ round-robin 0 [prio=0][active] \_ 0:0:4:0 sdd 8:48 [active][ready] \_ round-robin 0 [prio=0][enabled] \_ 0:0:18:0 sdp 8:240 [active][ready] 35000c5000a41f1b3dm-4 HP,DF300BB6C3 [size=279G][features=0][hwhandler=0] \_ round-robin 0 [prio=0][active] \_ 0:0:7:0 sdg 8:96 [active][ready] \_ round-robin 0 [prio=0][enabled] \_ 0:0:21:0 sds 65:32 [active][ready] 35000c5000a41f997dm-9 HP,DF300BB6C3 [size=279G][features=0][hwhandler=0] \_ round-robin 0 [prio=0][active] \_ 0:0:12:0 sdl 8:176 [active][ready] \_ round-robin 0 [prio=0][enabled] \_ 0:0:26:0 sdx 65:112 [active][ready] 35000c5000a41f37fdm-6 HP,DF300BB6C3 [size=279G][features=0][hwhandler=0] \_ round-robin 0 [prio=0][active] \_ 0:0:9:0 sdi 8:128 [active][ready] \_ round-robin 0 [prio=0][enabled] \_ 0:0:23:0 sdu 65:64 [active][ready] 35000c5000a41e0efdm-8 HP,DF300BB6C3 [size=279G][features=0][hwhandler=0] \_ round-robin 0 [prio=0][active] \_ 0:0:11:0 sdk 8:160 [active][ready] \_ round-robin 0 [prio=0][enabled] \_ 0:0:25:0 sdw 65:96 [active][ready] multisdf (35000c5000a41f007) dm-3 HP,DF300BB6C3 [size=279G][features=0][hwhandler=0] \_ round-robin 0 [prio=0][active] \_ 0:0:6:0 sdf 8:80 [active][ready] \_ round-robin 0 [prio=0][enabled] \_ 0:0:20:0 sdr 65:16 [active][ready] /root> Now once the disk has been pulled and the path to the device is lost: /root> ls -l /dev/hpdev total 0 lrwxrwxrwx 1 root root 7 Mar 24 11:53 sdb1 -> ../sdn1 /root> multipath -ll multisde (35000c5000a41e633) dm-2 HP,DF300BB6C3 [size=279G][features=0][hwhandler=0] \_ round-robin 0 [prio=0][active] \_ 0:0:5:0 sde 8:64 [active][ready] \_ round-robin 0 [prio=0][enabled] \_ 0:0:19:0 sdq 65:0 [active][ready] 35000c5000a41e44bdm-5 HP,DF300BB6C3 [size=279G][features=0][hwhandler=0] \_ round-robin 0 [prio=0][active] \_ 0:0:8:0 sdh 8:112 [active][ready] \_ round-robin 0 [prio=0][enabled] \_ 0:0:22:0 sdt 65:48 [active][ready] multisdc (35000c5000a41f38f) dm-0 HP,DF300BB6C3 [size=279G][features=0][hwhandler=0] \_ round-robin 0 [prio=0][active] \_ 0:0:3:0 sdc 8:32 [active][ready] \_ round-robin 0 [prio=0][enabled] \_ 0:0:17:0 sdo 8:224 [active][ready] 35000c5000a41e9bbdm-7 HP,DF300BB6C3 [size=279G][features=0][hwhandler=0] \_ round-robin 0 [prio=0][active] \_ 0:0:10:0 sdj 8:144 [active][ready] \_ round-robin 0 [prio=0][enabled] \_ 0:0:24:0 sdv 65:80 [active][ready] 35000c50004acb2cfdm-1 HP,DF0300B8053 [size=279G][features=0][hwhandler=0] \_ round-robin 0 [prio=0][active] \_ 0:0:4:0 sdd 8:48 [active][ready] \_ round-robin 0 [prio=0][enabled] \_ 0:0:18:0 sdp 8:240 [active][ready] 35000c5000a41f1b3dm-4 HP,DF300BB6C3 [size=279G][features=0][hwhandler=0] \_ round-robin 0 [prio=0][active] \_ 0:0:7:0 sdg 8:96 [active][ready] \_ round-robin 0 [prio=0][enabled] \_ 0:0:21:0 sds 65:32 [active][ready] 35000c5000a41f997dm-9 HP,DF300BB6C3 [size=279G][features=0][hwhandler=0] \_ round-robin 0 [prio=0][active] \_ 0:0:12:0 sdl 8:176 [active][ready] \_ round-robin 0 [prio=0][enabled] \_ 0:0:26:0 sdx 65:112 [active][ready] 35000c5000a41f37fdm-6 HP,DF300BB6C3 [size=279G][features=0][hwhandler=0] \_ round-robin 0 [prio=0][active] \_ 0:0:9:0 sdi 8:128 [active][ready] \_ round-robin 0 [prio=0][enabled] \_ 0:0:23:0 sdu 65:64 [active][ready] 35000c5000a41e0efdm-8 HP,DF300BB6C3 [size=279G][features=0][hwhandler=0] \_ round-robin 0 [prio=0][active] \_ 0:0:11:0 sdk 8:160 [active][ready] \_ round-robin 0 [prio=0][enabled] \_ 0:0:25:0 sdw 65:96 [active][ready] multisdf (35000c5000a41f007) dm-3 HP,DF300BB6C3 [size=279G][features=0][hwhandler=0] \_ round-robin 0 [prio=0][active] \_ 0:0:6:0 sdf 8:80 [active][ready] \_ round-robin 0 [prio=0][enabled] \_ 0:0:20:0 sdr 65:16 [active][ready] /root> Please let me know if there is any other information that you require from the node. Thank you. Lori > -----Original Message----- > From: dm-devel-bounces@redhat.com > [mailto:dm-devel-bounces@redhat.com] On Behalf Of malahal@us.ibm.com > Sent: Tuesday, March 24, 2009 9:42 AM > To: dm-devel@redhat.com > Subject: Re: [dm-devel] RE: Problem with multipathd and a > blacklisted device > > Ransegnola, Lori [Lori.Ransegnola@hp.com] wrote: > > I sent this question last week and have not heard a thing. > Do you have a suggestion of a better mailing list that I > should use for this question? > > > > Thank you. > > > > Lori > > > > > -----Original Message----- > > > From: Ransegnola, Lori > > > Sent: Thursday, March 19, 2009 3:51 PM > > > To: dm-devel@redhat.com > > > Cc: Ransegnola, Lori > > > Subject: Problem with multipathd and a blacklisted device > > > > > > Configuration: > > > -------------- > > > > > > I have a multipath configuration set up where 10 disks are > > > multipathed and 2 disks are blacklisted and are not multipathed. > > > Here are the relevant parts of the multipath.conf file: > > > > > > defaults { > > > udev_dir /dev > > > polling_interval 10 > > > selector "round-robin 0" > > > # path_grouping_policy multibus > > > getuid_callout "/sbin/scsi_id -g -u -s /block/%n" > > > prio_callout /bin/true > > > # path_checker readsector0 > > > path_checker tur > > > rr_min_io 100 > > > rr_weight priorities > > > failback immediate > > > no_path_retry fail > > > user_friendly_name yes > > > } > > > > > > blacklist { > > > # wwid 26353900f02796769 > > > # devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*" > > > # devnode "^hd[a-z]" > > > wwid 35000c5000a41f72b > > > wwid 35000c5000a41f98b > > > devnode "^cciss!c[0-9]d[0-9]*" > > > } > > > > > > > > > The two disks that are not multipathed have udev rules > and are part > > > of a software RAID set. The devices show up as > > > > > > /etc/udev/rules.d> ls -l /dev/hpdev > > > total 0 > > > lrwxrwxrwx 1 root root 7 Mar 18 10:50 sda1 -> ../sda1 > lrwxrwxrwx 1 > > > root root 7 Mar 18 03:32 sdb1 -> ../sdb1 > > > > > > /dev/md0 is created with sda and sdb. Here is the bottom of the > > > 'mdadm --detail /dev/md0' output: > > > > > > /root> mdadm --detail /dev/md0 > > > /dev/md0: > > > Version > > > ... > > > Failed Devices : 0 > > > Spare Devices : 0 > > > > > > UUID : 94813396:a3e341fc:c5d4b8ba:0617a019 > > > Events : 0.84 > > > > > > Number Major Minor RaidDevice State > > > 0 8 1 0 active sync /dev/sda1 > > > 1 8 17 1 active sync /dev/sdb1 > > > /root> > > > > > > I am running on a Linux RHEL5 U2 system. > > > /etc/udev/rules.d> uname -a > > > Linux n0 2.6.18-53.el5 #1 SMP Wed Oct 10 16:34:19 EDT 2007 > > > x86_64 x86_64 x86_64 GNU/Linux /etc/udev/rules.d> multipath -v > > > Missing option arguement multipath-tools v0.4.7 (03/12, 2006) > > > ---------------------------- > > > > > > Problem: > > > > > > This configuration works well until I do some failure > testing with > > > one of the 2 blacklisted devs in the software RAID set. > > > I found that if I temporarily remove disk sda and put it back a > > > minute later the disk path, /dev/hpdev/sda1, is removed, > even though > > > it is blacklisted. > > What is 'multipath -ll' output before you remove the > blacklisted device? > What is 'multipath -ll' output after you put the device back > a minute later? > > Thanks, Malahal. > > -- > dm-devel mailing list > dm-devel@redhat.com > https://www.redhat.com/mailman/listinfo/dm-devel > ^ permalink raw reply [flat|nested] 6+ messages in thread
* RE: RE: Problem with multipathd and a blacklisted device 2009-03-24 21:12 ` Ransegnola, Lori @ 2009-03-31 15:27 ` Ransegnola, Lori 0 siblings, 0 replies; 6+ messages in thread From: Ransegnola, Lori @ 2009-03-31 15:27 UTC (permalink / raw) To: Ransegnola, Lori, device-mapper development One last try.... Anybody have any input on why this is happening, and if there is something that I can do about it?? Thanks. Lori > -----Original Message----- > From: Ransegnola, Lori > Sent: Tuesday, March 24, 2009 5:13 PM > To: device-mapper development > Cc: Ransegnola, Lori > Subject: RE: [dm-devel] RE: Problem with multipathd and a > blacklisted device > > Here is the 'multipath -ll' output before disk was pulled: > > /root> ls -l /dev/hpdev > total 0 > lrwxrwxrwx 1 root root 7 Mar 24 11:53 sda1 -> ../sdm1 > lrwxrwxrwx 1 root root 7 Mar 24 11:53 sdb1 -> ../sdn1 > /root> multipath -ll > multisde (35000c5000a41e633) dm-2 HP,DF300BB6C3 > [size=279G][features=0][hwhandler=0] > \_ round-robin 0 [prio=0][active] > \_ 0:0:5:0 sde 8:64 [active][ready] > \_ round-robin 0 [prio=0][enabled] > \_ 0:0:19:0 sdq 65:0 [active][ready] > 35000c5000a41e44bdm-5 HP,DF300BB6C3 > [size=279G][features=0][hwhandler=0] > \_ round-robin 0 [prio=0][active] > \_ 0:0:8:0 sdh 8:112 [active][ready] > \_ round-robin 0 [prio=0][enabled] > \_ 0:0:22:0 sdt 65:48 [active][ready] > multisdc (35000c5000a41f38f) dm-0 HP,DF300BB6C3 > [size=279G][features=0][hwhandler=0] > \_ round-robin 0 [prio=0][active] > \_ 0:0:3:0 sdc 8:32 [active][ready] > \_ round-robin 0 [prio=0][enabled] > \_ 0:0:17:0 sdo 8:224 [active][ready] > 35000c5000a41e9bbdm-7 HP,DF300BB6C3 > [size=279G][features=0][hwhandler=0] > \_ round-robin 0 [prio=0][active] > \_ 0:0:10:0 sdj 8:144 [active][ready] > \_ round-robin 0 [prio=0][enabled] > \_ 0:0:24:0 sdv 65:80 [active][ready] > 35000c50004acb2cfdm-1 HP,DF0300B8053 > [size=279G][features=0][hwhandler=0] > \_ round-robin 0 [prio=0][active] > \_ 0:0:4:0 sdd 8:48 [active][ready] > \_ round-robin 0 [prio=0][enabled] > \_ 0:0:18:0 sdp 8:240 [active][ready] > 35000c5000a41f1b3dm-4 HP,DF300BB6C3 > [size=279G][features=0][hwhandler=0] > \_ round-robin 0 [prio=0][active] > \_ 0:0:7:0 sdg 8:96 [active][ready] > \_ round-robin 0 [prio=0][enabled] > \_ 0:0:21:0 sds 65:32 [active][ready] > 35000c5000a41f997dm-9 HP,DF300BB6C3 > [size=279G][features=0][hwhandler=0] > \_ round-robin 0 [prio=0][active] > \_ 0:0:12:0 sdl 8:176 [active][ready] > \_ round-robin 0 [prio=0][enabled] > \_ 0:0:26:0 sdx 65:112 [active][ready] > 35000c5000a41f37fdm-6 HP,DF300BB6C3 > [size=279G][features=0][hwhandler=0] > \_ round-robin 0 [prio=0][active] > \_ 0:0:9:0 sdi 8:128 [active][ready] > \_ round-robin 0 [prio=0][enabled] > \_ 0:0:23:0 sdu 65:64 [active][ready] > 35000c5000a41e0efdm-8 HP,DF300BB6C3 > [size=279G][features=0][hwhandler=0] > \_ round-robin 0 [prio=0][active] > \_ 0:0:11:0 sdk 8:160 [active][ready] > \_ round-robin 0 [prio=0][enabled] > \_ 0:0:25:0 sdw 65:96 [active][ready] > multisdf (35000c5000a41f007) dm-3 HP,DF300BB6C3 > [size=279G][features=0][hwhandler=0] > \_ round-robin 0 [prio=0][active] > \_ 0:0:6:0 sdf 8:80 [active][ready] > \_ round-robin 0 [prio=0][enabled] > \_ 0:0:20:0 sdr 65:16 [active][ready] > /root> > > Now once the disk has been pulled and the path to the device is lost: > > /root> ls -l /dev/hpdev > total 0 > lrwxrwxrwx 1 root root 7 Mar 24 11:53 sdb1 -> ../sdn1 > /root> multipath -ll > multisde (35000c5000a41e633) dm-2 HP,DF300BB6C3 > [size=279G][features=0][hwhandler=0] > \_ round-robin 0 [prio=0][active] > \_ 0:0:5:0 sde 8:64 [active][ready] > \_ round-robin 0 [prio=0][enabled] > \_ 0:0:19:0 sdq 65:0 [active][ready] > 35000c5000a41e44bdm-5 HP,DF300BB6C3 > [size=279G][features=0][hwhandler=0] > \_ round-robin 0 [prio=0][active] > \_ 0:0:8:0 sdh 8:112 [active][ready] > \_ round-robin 0 [prio=0][enabled] > \_ 0:0:22:0 sdt 65:48 [active][ready] > multisdc (35000c5000a41f38f) dm-0 HP,DF300BB6C3 > [size=279G][features=0][hwhandler=0] > \_ round-robin 0 [prio=0][active] > \_ 0:0:3:0 sdc 8:32 [active][ready] > \_ round-robin 0 [prio=0][enabled] > \_ 0:0:17:0 sdo 8:224 [active][ready] > 35000c5000a41e9bbdm-7 HP,DF300BB6C3 > [size=279G][features=0][hwhandler=0] > \_ round-robin 0 [prio=0][active] > \_ 0:0:10:0 sdj 8:144 [active][ready] > \_ round-robin 0 [prio=0][enabled] > \_ 0:0:24:0 sdv 65:80 [active][ready] > 35000c50004acb2cfdm-1 HP,DF0300B8053 > [size=279G][features=0][hwhandler=0] > \_ round-robin 0 [prio=0][active] > \_ 0:0:4:0 sdd 8:48 [active][ready] > \_ round-robin 0 [prio=0][enabled] > \_ 0:0:18:0 sdp 8:240 [active][ready] > 35000c5000a41f1b3dm-4 HP,DF300BB6C3 > [size=279G][features=0][hwhandler=0] > \_ round-robin 0 [prio=0][active] > \_ 0:0:7:0 sdg 8:96 [active][ready] > \_ round-robin 0 [prio=0][enabled] > \_ 0:0:21:0 sds 65:32 [active][ready] > 35000c5000a41f997dm-9 HP,DF300BB6C3 > [size=279G][features=0][hwhandler=0] > \_ round-robin 0 [prio=0][active] > \_ 0:0:12:0 sdl 8:176 [active][ready] > \_ round-robin 0 [prio=0][enabled] > \_ 0:0:26:0 sdx 65:112 [active][ready] > 35000c5000a41f37fdm-6 HP,DF300BB6C3 > [size=279G][features=0][hwhandler=0] > \_ round-robin 0 [prio=0][active] > \_ 0:0:9:0 sdi 8:128 [active][ready] > \_ round-robin 0 [prio=0][enabled] > \_ 0:0:23:0 sdu 65:64 [active][ready] > 35000c5000a41e0efdm-8 HP,DF300BB6C3 > [size=279G][features=0][hwhandler=0] > \_ round-robin 0 [prio=0][active] > \_ 0:0:11:0 sdk 8:160 [active][ready] > \_ round-robin 0 [prio=0][enabled] > \_ 0:0:25:0 sdw 65:96 [active][ready] > multisdf (35000c5000a41f007) dm-3 HP,DF300BB6C3 > [size=279G][features=0][hwhandler=0] > \_ round-robin 0 [prio=0][active] > \_ 0:0:6:0 sdf 8:80 [active][ready] > \_ round-robin 0 [prio=0][enabled] > \_ 0:0:20:0 sdr 65:16 [active][ready] > /root> > > Please let me know if there is any other information that you > require from the node. Thank you. > > Lori > > > -----Original Message----- > > From: dm-devel-bounces@redhat.com > > [mailto:dm-devel-bounces@redhat.com] On Behalf Of malahal@us.ibm.com > > Sent: Tuesday, March 24, 2009 9:42 AM > > To: dm-devel@redhat.com > > Subject: Re: [dm-devel] RE: Problem with multipathd and a > > blacklisted device > > > > Ransegnola, Lori [Lori.Ransegnola@hp.com] wrote: > > > I sent this question last week and have not heard a thing. > > Do you have a suggestion of a better mailing list that I > > should use for this question? > > > > > > Thank you. > > > > > > Lori > > > > > > > -----Original Message----- > > > > From: Ransegnola, Lori > > > > Sent: Thursday, March 19, 2009 3:51 PM > > > > To: dm-devel@redhat.com > > > > Cc: Ransegnola, Lori > > > > Subject: Problem with multipathd and a blacklisted device > > > > > > > > Configuration: > > > > -------------- > > > > > > > > I have a multipath configuration set up where 10 disks are > > > > multipathed and 2 disks are blacklisted and are not > multipathed. > > > > Here are the relevant parts of the multipath.conf file: > > > > > > > > defaults { > > > > udev_dir /dev > > > > polling_interval 10 > > > > selector "round-robin 0" > > > > # path_grouping_policy multibus > > > > getuid_callout "/sbin/scsi_id -g -u -s > /block/%n" > > > > prio_callout /bin/true > > > > # path_checker readsector0 > > > > path_checker tur > > > > rr_min_io 100 > > > > rr_weight priorities > > > > failback immediate > > > > no_path_retry fail > > > > user_friendly_name yes > > > > } > > > > > > > > blacklist { > > > > # wwid 26353900f02796769 > > > > # devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*" > > > > # devnode "^hd[a-z]" > > > > wwid 35000c5000a41f72b > > > > wwid 35000c5000a41f98b > > > > devnode "^cciss!c[0-9]d[0-9]*" > > > > } > > > > > > > > > > > > The two disks that are not multipathed have udev rules > > and are part > > > > of a software RAID set. The devices show up as > > > > > > > > /etc/udev/rules.d> ls -l /dev/hpdev > > > > total 0 > > > > lrwxrwxrwx 1 root root 7 Mar 18 10:50 sda1 -> ../sda1 > > lrwxrwxrwx 1 > > > > root root 7 Mar 18 03:32 sdb1 -> ../sdb1 > > > > > > > > /dev/md0 is created with sda and sdb. Here is the > bottom of the > > > > 'mdadm --detail /dev/md0' output: > > > > > > > > /root> mdadm --detail /dev/md0 > > > > /dev/md0: > > > > Version > > > > ... > > > > Failed Devices : 0 > > > > Spare Devices : 0 > > > > > > > > UUID : 94813396:a3e341fc:c5d4b8ba:0617a019 > > > > Events : 0.84 > > > > > > > > Number Major Minor RaidDevice State > > > > 0 8 1 0 active sync /dev/sda1 > > > > 1 8 17 1 active sync /dev/sdb1 > > > > /root> > > > > > > > > I am running on a Linux RHEL5 U2 system. > > > > /etc/udev/rules.d> uname -a > > > > Linux n0 2.6.18-53.el5 #1 SMP Wed Oct 10 16:34:19 EDT 2007 > > > > x86_64 x86_64 x86_64 GNU/Linux /etc/udev/rules.d> multipath -v > > > > Missing option arguement multipath-tools v0.4.7 (03/12, 2006) > > > > ---------------------------- > > > > > > > > Problem: > > > > > > > > This configuration works well until I do some failure > > testing with > > > > one of the 2 blacklisted devs in the software RAID set. > > > > I found that if I temporarily remove disk sda and put it back a > > > > minute later the disk path, /dev/hpdev/sda1, is removed, > > even though > > > > it is blacklisted. > > > > What is 'multipath -ll' output before you remove the > > blacklisted device? > > What is 'multipath -ll' output after you put the device back > > a minute later? > > > > Thanks, Malahal. > > > > -- > > dm-devel mailing list > > dm-devel@redhat.com > > https://www.redhat.com/mailman/listinfo/dm-devel > > > ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: RE: Problem with multipathd and a blacklisted device 2009-03-24 13:27 ` Problem with multipathd and a blacklisted device Ransegnola, Lori 2009-03-24 13:42 ` malahal @ 2009-03-31 15:49 ` Benjamin Marzinski 2009-04-01 13:40 ` Ransegnola, Lori 1 sibling, 1 reply; 6+ messages in thread From: Benjamin Marzinski @ 2009-03-31 15:49 UTC (permalink / raw) To: device-mapper development > > This configuration works well until I do some failure testing > > with one of the 2 blacklisted devs in the software RAID set. > > I found that if I temporarily remove disk sda and put it back > > a minute later the disk path, /dev/hpdev/sda1, is removed, > > even though it is blacklisted. multipathd shouldn't be removing and devices. It removes paths from is list of monitored paths, but not from the filesystem. Try doing this without multipathd running, and see if the device still disappears. That all being said, multipathd shouldn't be monitoring the device if it's backlisted. Did you start up multipathd before you blacklisted the device? If so, you need to run # service multipathd reload To make multipathd pick up the new configuration. Or you can simply restart it. You can check to see if it is monitoring the paths by running # multipathd -k"show paths" -Ben > > > > Here are some of the pertinent /var/log/messages lines: > > > > Mar 19 09:53:19 n0 mdadm: NewArray /dev/md0 Mar 19 10:05:12 > > n0 kernel: mptbase: ioc0: LogInfo(0x31170000): > > Originator={PL}, Code={IO Device Missing Delay Retry}, > > SubCode(0x0000) Mar 19 10:05:40 n0 last message repeated 5 > > times Mar 19 10:05:42 n0 kernel: mptbase: ioc0: > > LogInfo(0x31130000): Originator={PL}, Code={IO Not Yet > > Executed}, SubCode(0x0000) Mar 19 10:05:42 n0 kernel: sd > > 0:0:28:0: SCSI error: return code = 0x00010000 Mar 19 > > 10:05:42 n0 kernel: end_request: I/O error, dev sda, sector > > 256119 Mar 19 10:05:42 n0 kernel: Buffer I/O error on device > > sda1, logical block 256056 Mar 19 10:05:42 n0 kernel: Buffer > > I/O error on device sda1, logical block 256057 Mar 19 > > 10:05:42 n0 kernel: Buffer I/O error on device sda1, logical > > block 256058 Mar 19 10:05:42 n0 kernel: Buffer I/O error on > > device sda1, logical block 256059 Mar 19 10:05:42 n0 kernel: > > Buffer I/O error on device sda1, logical block 256060 Mar 19 > > 10:05:42 n0 kernel: Buffer I/O error on device sda1, logical > > block 256061 Mar 19 10:05:42 n0 kernel: mptbase: ioc0: > > LogInfo(0x31130000): Originator={PL}, Code={IO Not Yet > > Executed}, SubCode(0x0000) Mar 19 10:05:42 n0 kernel: Buffer > > I/O error on device sda1, logical block 256062 Mar 19 > > 10:05:42 n0 kernel: Buffer I/O error on device sda1, logical > > block 256063 Mar 19 10:05:42 n0 kernel: sd 0:0:28:0: SCSI > > error: return code = 0x00010000 Mar 19 10:05:42 n0 > > multipathd: sda: remove path (uevent) Mar 19 10:05:42 n0 > > kernel: end_request: I/O error, dev sda, sector 585922495 Mar > > 19 10:05:42 n0 kernel: Buffer I/O error on device sda1, > > logical block 585922432 Mar 19 10:05:42 n0 xinetd[13096]: > > START: hacl-cfgudp pid=6034 from=127.0.0.1 Mar 19 10:05:42 n0 > > multipathd: uevent trigger error Mar 19 10:05:42 n0 kernel: > > Buffer I/O error on device sda1, logical block 585922433 > > > > > > The sda path is gone: > > > > /root> ls -l /dev/hpdev > > total 0 > > lrwxrwxrwx 1 root root 7 Mar 18 03:32 sdb1 -> ../sdb1 /root> > > > > And I cannot reassemble the software raid set. While the > > mdstat looks 'normal' the software raid becomes degraded. > > > > /etc/udev/rules.d> cat /proc/mdstat > > Personalities : [raid1] > > md0 : active raid1 sda1[2](F) sdb1[1] > > 292961216 blocks [2/1] [_U] > > > > unused devices: <none> > > /etc/udev/rules.d> mdadm --detail /dev/md0 > > /dev/md0: > > Version : 00.90.03 > > Creation Time : Wed Feb 18 14:15:03 2009 > > Raid Level : raid1 > > Array Size : 292961216 (279.39 GiB 299.99 GB) > > Device Size : 292961216 (279.39 GiB 299.99 GB) > > Raid Devices : 2 > > Total Devices : 2 > > Preferred Minor : 0 > > Persistence : Superblock is persistent > > > > Update Time : Thu Mar 19 10:28:45 2009 > > State : clean, degraded > > Active Devices : 1 > > Working Devices : 1 > > Failed Devices : 1 > > Spare Devices : 0 > > > > UUID : 94813396:a3e341fc:c5d4b8ba:0617a019 > > Events : 0.86 > > > > Number Major Minor RaidDevice State > > 0 0 0 0 removed > > 1 8 17 1 active sync /dev/sdb1 > > > > 2 8 1 - faulty spare > > /etc/udev/rules.d> > > > > The disk is fine and is back in and ready to go. > > > > So, 1) why does multipathd remove the path for a blacklisted > > device? If it is blacklisted, shouldn't multipathd just > > leave it alone?? > > And 2) what can I do to keep this from happening?? > > > > Lori Ransegnola > > > > -- > dm-devel mailing list > dm-devel@redhat.com > https://www.redhat.com/mailman/listinfo/dm-devel ^ permalink raw reply [flat|nested] 6+ messages in thread
* RE: RE: Problem with multipathd and a blacklisted device 2009-03-31 15:49 ` Benjamin Marzinski @ 2009-04-01 13:40 ` Ransegnola, Lori 0 siblings, 0 replies; 6+ messages in thread From: Ransegnola, Lori @ 2009-04-01 13:40 UTC (permalink / raw) To: device-mapper development; +Cc: Ransegnola, Lori > Try doing this > without multipathd running, and see if the device still disappears. Good point! I tried turning off multipathd and temporarily removed the disk. Lo and behold, the disk path, /dev/hpdev/sda1, was removed! I guess that this is caused by software RAID and not by multipathd. Thanks for your suggestion and help. At least now I know I was looking in the wrong direction. Lori > -----Original Message----- > From: dm-devel-bounces@redhat.com > [mailto:dm-devel-bounces@redhat.com] On Behalf Of Benjamin Marzinski > Sent: Tuesday, March 31, 2009 11:50 AM > To: device-mapper development > Subject: Re: [dm-devel] RE: Problem with multipathd and a > blacklisted device > > > > This configuration works well until I do some failure testing > > > with one of the 2 blacklisted devs in the software RAID set. > > > I found that if I temporarily remove disk sda and put it back > > > a minute later the disk path, /dev/hpdev/sda1, is removed, > > > even though it is blacklisted. > > multipathd shouldn't be removing and devices. It removes > paths from is > list of monitored paths, but not from the filesystem. Try doing this > without multipathd running, and see if the device still disappears. > > That all being said, multipathd shouldn't be monitoring the device > if it's backlisted. Did you start up multipathd before you > blacklisted > the device? If so, you need to run > > # service multipathd reload > > To make multipathd pick up the new configuration. Or you can simply > restart it. You can check to see if it is monitoring the paths > by running > > # multipathd -k"show paths" > > -Ben > > > > > > > Here are some of the pertinent /var/log/messages lines: > > > > > > Mar 19 09:53:19 n0 mdadm: NewArray /dev/md0 Mar 19 10:05:12 > > > n0 kernel: mptbase: ioc0: LogInfo(0x31170000): > > > Originator={PL}, Code={IO Device Missing Delay Retry}, > > > SubCode(0x0000) Mar 19 10:05:40 n0 last message repeated 5 > > > times Mar 19 10:05:42 n0 kernel: mptbase: ioc0: > > > LogInfo(0x31130000): Originator={PL}, Code={IO Not Yet > > > Executed}, SubCode(0x0000) Mar 19 10:05:42 n0 kernel: sd > > > 0:0:28:0: SCSI error: return code = 0x00010000 Mar 19 > > > 10:05:42 n0 kernel: end_request: I/O error, dev sda, sector > > > 256119 Mar 19 10:05:42 n0 kernel: Buffer I/O error on device > > > sda1, logical block 256056 Mar 19 10:05:42 n0 kernel: Buffer > > > I/O error on device sda1, logical block 256057 Mar 19 > > > 10:05:42 n0 kernel: Buffer I/O error on device sda1, logical > > > block 256058 Mar 19 10:05:42 n0 kernel: Buffer I/O error on > > > device sda1, logical block 256059 Mar 19 10:05:42 n0 kernel: > > > Buffer I/O error on device sda1, logical block 256060 Mar 19 > > > 10:05:42 n0 kernel: Buffer I/O error on device sda1, logical > > > block 256061 Mar 19 10:05:42 n0 kernel: mptbase: ioc0: > > > LogInfo(0x31130000): Originator={PL}, Code={IO Not Yet > > > Executed}, SubCode(0x0000) Mar 19 10:05:42 n0 kernel: Buffer > > > I/O error on device sda1, logical block 256062 Mar 19 > > > 10:05:42 n0 kernel: Buffer I/O error on device sda1, logical > > > block 256063 Mar 19 10:05:42 n0 kernel: sd 0:0:28:0: SCSI > > > error: return code = 0x00010000 Mar 19 10:05:42 n0 > > > multipathd: sda: remove path (uevent) Mar 19 10:05:42 n0 > > > kernel: end_request: I/O error, dev sda, sector 585922495 Mar > > > 19 10:05:42 n0 kernel: Buffer I/O error on device sda1, > > > logical block 585922432 Mar 19 10:05:42 n0 xinetd[13096]: > > > START: hacl-cfgudp pid=6034 from=127.0.0.1 Mar 19 10:05:42 n0 > > > multipathd: uevent trigger error Mar 19 10:05:42 n0 kernel: > > > Buffer I/O error on device sda1, logical block 585922433 > > > > > > > > > The sda path is gone: > > > > > > /root> ls -l /dev/hpdev > > > total 0 > > > lrwxrwxrwx 1 root root 7 Mar 18 03:32 sdb1 -> ../sdb1 /root> > > > > > > And I cannot reassemble the software raid set. While the > > > mdstat looks 'normal' the software raid becomes degraded. > > > > > > /etc/udev/rules.d> cat /proc/mdstat > > > Personalities : [raid1] > > > md0 : active raid1 sda1[2](F) sdb1[1] > > > 292961216 blocks [2/1] [_U] > > > > > > unused devices: <none> > > > /etc/udev/rules.d> mdadm --detail /dev/md0 > > > /dev/md0: > > > Version : 00.90.03 > > > Creation Time : Wed Feb 18 14:15:03 2009 > > > Raid Level : raid1 > > > Array Size : 292961216 (279.39 GiB 299.99 GB) > > > Device Size : 292961216 (279.39 GiB 299.99 GB) > > > Raid Devices : 2 > > > Total Devices : 2 > > > Preferred Minor : 0 > > > Persistence : Superblock is persistent > > > > > > Update Time : Thu Mar 19 10:28:45 2009 > > > State : clean, degraded > > > Active Devices : 1 > > > Working Devices : 1 > > > Failed Devices : 1 > > > Spare Devices : 0 > > > > > > UUID : 94813396:a3e341fc:c5d4b8ba:0617a019 > > > Events : 0.86 > > > > > > Number Major Minor RaidDevice State > > > 0 0 0 0 removed > > > 1 8 17 1 active sync /dev/sdb1 > > > > > > 2 8 1 - faulty spare > > > /etc/udev/rules.d> > > > > > > The disk is fine and is back in and ready to go. > > > > > > So, 1) why does multipathd remove the path for a blacklisted > > > device? If it is blacklisted, shouldn't multipathd just > > > leave it alone?? > > > And 2) what can I do to keep this from happening?? > > > > > > Lori Ransegnola > > > > > > > -- > > dm-devel mailing list > > dm-devel@redhat.com > > https://www.redhat.com/mailman/listinfo/dm-devel > > -- > dm-devel mailing list > dm-devel@redhat.com > https://www.redhat.com/mailman/listinfo/dm-devel > ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2009-04-01 13:40 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <5635940A0F6F36478CF4D01273B16CA32B9C3EDDF8@GVW1096EXB.americas.hpqcorp.net>
2009-03-24 13:27 ` Problem with multipathd and a blacklisted device Ransegnola, Lori
2009-03-24 13:42 ` malahal
2009-03-24 21:12 ` Ransegnola, Lori
2009-03-31 15:27 ` Ransegnola, Lori
2009-03-31 15:49 ` Benjamin Marzinski
2009-04-01 13:40 ` Ransegnola, Lori
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.