multipath rdac - recover from both paths lost

multipath rdac - recover from both paths lost

[Greg Cockburn]

[-- Attachment #1.1: Type: text/plain, Size: 1667 bytes --]

Hi,

I am trying to get back from a strage situation.

I have had both paths in a RDAC setup fail and then come back.

I want to know if there is a way to recover from this without a reboot.

(OS = CentOS 5.1)

multipath -ll
mpath2 (3600a0b800035ebc8000001a94652ae50) dm-4 IBM,1726-4xx  FAStT
[size=1.4T][features=0][hwhandler=1 rdac]
\_ round-robin 0 [prio=0][enabled]
 \_ 2:0:0:1 sde 8:80  [failed][ghost]
\_ round-robin 0 [prio=100][active]
 \_ 1:0:0:1 sdf 8:96  [failed][ready]

dmsetup info mpath2
Name:              mpath2
State:             ACTIVE
Tables present:    LIVE
Open count:        20
Event number:      3
Major, minor:      253, 4
Number of targets: 1
UUID: mpath-3600a0b800035ebc8000001a94652ae50

vgscan
  Reading all physical volumes.  This may take a while...
  /dev/mapper/mpath2: read failed after 0 of 4096 at 1497314033664:
Input/output error
  /dev/mapper/mpath2: read failed after 0 of 4096 at 0: Input/output error

dmsetup remove --force mpath2



device-mapper: remove ioctl failed: Device or resource busy
Command failed

multipath -f mpath2 (just exits with return code 1)


dmesg

    <<vendor>> ASC=0xe0 ASCQ=0x6ASC=0xe0 ASCQ=0x6

sde: Current: sense key: Recovered Error
    <<vendor>> ASC=0xe0 ASCQ=0x6ASC=0xe0 ASCQ=0x6

sdf: Current: sense key: Recovered Error
    <<vendor>> ASC=0xe0 ASCQ=0x6ASC=0xe0 ASCQ=0x6

end_request: I/O error, dev sdf, sector 61118720
device-mapper: multipath: Failing path 8:80.
end_request: I/O error, dev sdf, sector 599510512

device-mapper: ioctl: unable to remove open device mpath2
device-mapper: ioctl: unable to remove open device mpath2

Any advice would be greatly appreciated.

Thanks,
Greg.

[-- Attachment #1.2: Type: text/html, Size: 2239 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: multipath rdac - recover from both paths lost

[Konrad Rzeszutek]

On Mon, Oct 20, 2008 at 09:29:16PM +1100, Greg Cockburn wrote:
> Hi,
> 
> I am trying to get back from a strage situation.
> 
> I have had both paths in a RDAC setup fail and then come back.

That is not what your 'multipath -ll' output shows. How did they
come back? As in a re-login? If so they should automaticly be
repopulated - unless your 'multipathd' daemon is dead.

Also, try running 'dd if=/dev/sdX of=/dev/null count=1' on your
new paths (sde?, sdf?) and see if they fail. If they do fail, then
they are still in failed condition and you need to fix your
transportion problem.


> 
> I want to know if there is a way to recover from this without a reboot.

Yes. You can force it to rediscover all of the present block devices and
add them in the corresponding device mapper maps by calling : 'multipath'.
No arguments, nothing. Just call it. After that run 'multipath -ll' and
that should show your new/refreshed block devices.


> 
> (OS = CentOS 5.1)
> 
> multipath -ll
> mpath2 (3600a0b800035ebc8000001a94652ae50) dm-4 IBM,1726-4xx  FAStT
> [size=1.4T][features=0][hwhandler=1 rdac]
> \_ round-robin 0 [prio=0][enabled]
>  \_ 2:0:0:1 sde 8:80  [failed][ghost]
> \_ round-robin 0 [prio=100][active]
>  \_ 1:0:0:1 sdf 8:96  [failed][ready]
> 
> dmsetup info mpath2
> Name:              mpath2
> State:             ACTIVE
> Tables present:    LIVE
> Open count:        20
> Event number:      3
> Major, minor:      253, 4
> Number of targets: 1
> UUID: mpath-3600a0b800035ebc8000001a94652ae50
> 
> vgscan
>   Reading all physical volumes.  This may take a while...
>   /dev/mapper/mpath2: read failed after 0 of 4096 at 1497314033664:
> Input/output error
>   /dev/mapper/mpath2: read failed after 0 of 4096 at 0: Input/output error
> 
> dmsetup remove --force mpath2

Uhh.. That is not the way to do it. You would be removing the carpet underneath
LVM. And from the looks of it, there are about 20 processes using it - so all
of them could die.

> 
> 
> 
> device-mapper: remove ioctl failed: Device or resource busy
> Command failed
> 
> multipath -f mpath2 (just exits with return code 1)
> 
> 
> dmesg
> 
>     <<vendor>> ASC=0xe0 ASCQ=0x6ASC=0xe0 ASCQ=0x6
> 
> sde: Current: sense key: Recovered Error
>     <<vendor>> ASC=0xe0 ASCQ=0x6ASC=0xe0 ASCQ=0x6
> 
> sdf: Current: sense key: Recovered Error
>     <<vendor>> ASC=0xe0 ASCQ=0x6ASC=0xe0 ASCQ=0x6
> 
> end_request: I/O error, dev sdf, sector 61118720
> device-mapper: multipath: Failing path 8:80.
> end_request: I/O error, dev sdf, sector 599510512
> 
> device-mapper: ioctl: unable to remove open device mpath2
> device-mapper: ioctl: unable to remove open device mpath2
> 
> Any advice would be greatly appreciated.
> 
> Thanks,
> Greg.

> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel

RE: multipath rdac - recover from both paths lost

[Moger, Babu]

[-- Attachment #1.1: Type: text/plain, Size: 2149 bytes --]

You can try stopping (/etc/init.d/multipathd stop) and re-starting ((/etc/init.d/multipathd start) the multipathd daemon again.

Thanks
Babu Moger
________________________________
From: dm-devel-bounces@redhat.com [mailto:dm-devel-bounces@redhat.com] On Behalf Of Greg Cockburn
Sent: Monday, October 20, 2008 5:29 AM
To: dm-devel@redhat.com
Subject: [dm-devel] multipath rdac - recover from both paths lost

Hi,

I am trying to get back from a strage situation.

I have had both paths in a RDAC setup fail and then come back.

I want to know if there is a way to recover from this without a reboot.

(OS = CentOS 5.1)

multipath -ll
mpath2 (3600a0b800035ebc8000001a94652ae50) dm-4 IBM,1726-4xx  FAStT
[size=1.4T][features=0][hwhandler=1 rdac]
\_ round-robin 0 [prio=0][enabled]
 \_ 2:0:0:1 sde 8:80  [failed][ghost]
\_ round-robin 0 [prio=100][active]
 \_ 1:0:0:1 sdf 8:96  [failed][ready]

dmsetup info mpath2
Name:              mpath2
State:             ACTIVE
Tables present:    LIVE
Open count:        20
Event number:      3
Major, minor:      253, 4
Number of targets: 1
UUID: mpath-3600a0b800035ebc8000001a94652ae50

vgscan
  Reading all physical volumes.  This may take a while...
  /dev/mapper/mpath2: read failed after 0 of 4096 at 1497314033664: Input/output error
  /dev/mapper/mpath2: read failed after 0 of 4096 at 0: Input/output error

dmsetup remove --force mpath2



device-mapper: remove ioctl failed: Device or resource busy
Command failed

multipath -f mpath2 (just exits with return code 1)


dmesg

    <<vendor>> ASC=0xe0 ASCQ=0x6ASC=0xe0 ASCQ=0x6

sde: Current: sense key: Recovered Error
    <<vendor>> ASC=0xe0 ASCQ=0x6ASC=0xe0 ASCQ=0x6

sdf: Current: sense key: Recovered Error
    <<vendor>> ASC=0xe0 ASCQ=0x6ASC=0xe0 ASCQ=0x6

end_request: I/O error, dev sdf, sector 61118720
device-mapper: multipath: Failing path 8:80.
end_request: I/O error, dev sdf, sector 599510512

device-mapper: ioctl: unable to remove open device mpath2
device-mapper: ioctl: unable to remove open device mpath2

Any advice would be greatly appreciated.

Thanks,
Greg.

[-- Attachment #1.2: Type: text/html, Size: 5614 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: multipath rdac - recover from both paths lost

[Chandra Seetharaman]

Can you try with "no_path_retry queue" in multipath.conf file for the
device.

On Mon, 2008-10-20 at 21:29 +1100, Greg Cockburn wrote:
> Hi,
> 
> I am trying to get back from a strage situation.
> 
> I have had both paths in a RDAC setup fail and then come back.
> 
> I want to know if there is a way to recover from this without a
> reboot.
> 
> (OS = CentOS 5.1)
> 
> multipath -ll
> mpath2 (3600a0b800035ebc8000001a94652ae50) dm-4 IBM,1726-4xx  FAStT
> [size=1.4T][features=0][hwhandler=1 rdac]
> \_ round-robin 0 [prio=0][enabled]
>  \_ 2:0:0:1 sde 8:80  [failed][ghost]
> \_ round-robin 0 [prio=100][active]
>  \_ 1:0:0:1 sdf 8:96  [failed][ready]
> 
> dmsetup info mpath2
> Name:              mpath2
> State:             ACTIVE
> Tables present:    LIVE
> Open count:        20
> Event number:      3
> Major, minor:      253, 4
> Number of targets: 1
> UUID: mpath-3600a0b800035ebc8000001a94652ae50
> 
> vgscan
>   Reading all physical volumes.  This may take a while...
>   /dev/mapper/mpath2: read failed after 0 of 4096 at 1497314033664:
> Input/output error
>   /dev/mapper/mpath2: read failed after 0 of 4096 at 0: Input/output
> error
> 
> dmsetup remove --force mpath2
> 
> 
> 
> device-mapper: remove ioctl failed: Device or resource busy
> Command failed
> 
> multipath -f mpath2 (just exits with return code 1)
> 
> 
> dmesg
> 
>     <<vendor>> ASC=0xe0 ASCQ=0x6ASC=0xe0 ASCQ=0x6
> 
> sde: Current: sense key: Recovered Error
>     <<vendor>> ASC=0xe0 ASCQ=0x6ASC=0xe0 ASCQ=0x6
> 
> sdf: Current: sense key: Recovered Error
>     <<vendor>> ASC=0xe0 ASCQ=0x6ASC=0xe0 ASCQ=0x6
> 
> end_request: I/O error, dev sdf, sector 61118720
> device-mapper: multipath: Failing path 8:80.
> end_request: I/O error, dev sdf, sector 599510512
> 
> device-mapper: ioctl: unable to remove open device mpath2
> device-mapper: ioctl: unable to remove open device mpath2
> 
> Any advice would be greatly appreciated.
> 
> Thanks,
> Greg.
> 
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel