Problems with multipathing

Problems with multipathing

[Roger Håkansson]

I'm trying to setup a couple of machines running CentOS 4.3 x86_64 using
disk from a SAN, but I can't get the multipathing to work properly.

The machines (three in total) all have two Emulex LP10000-M2 and thus
are using the lpfc-driver.

When I boot the machines the paths seem to be multipathed somewhat, if i
put a "multipath"-tag in /etc/multipath.conf with a specific alias
(yellow,red) for each wwid, those paths are mapped under /dev/mapper and
/dev/mpath.

But multipath -ll doesn't show what I expect (basically no mappings) and
if I do a multipath -F I can't get the mappings back without a reboot.

If I do a 'multipath -v3' directly after boot, this is what I get:
#
# all paths in cache :
#
3600d0230000000000b0191489a946601  1:0:2:0 sdb 8:16  [active][ready] IFT
    /
3600d0230000000000b0191489a946600  1:0:3:0 sdc 8:32  [active][ready] IFT
    /
3600d0230000000000b0191489a946601  2:0:2:0 sdd 8:48  [active][ready] IFT
    /
3600d0230000000000b0191489a946600  2:0:3:0 sde 8:64  [active][ready] IFT
    /
*blacklist*
===== path info sdb (mask 0x1f) =====
bus = 1
dev_t = 8:16
size = 204800000
vendor = IFT
product = A16F-R2221
rev = 347B
h:b:t:l = 1:0:2:0
tgt_node_name = 0x200000d0231b0191
serial = 0B0191489A946601
path checker = readsector0 (internal default)
state = 2
getprio = /bin/true (internal default)
prio = 0
uid = 3600d0230000000000b0191489a946601 (cache)
===== path info sdc (mask 0x1f) =====
bus = 1
dev_t = 8:32
size = 409600000
vendor = IFT
product = A16F-R2221
rev = 347B
h:b:t:l = 1:0:3:0
tgt_node_name = 0x200000d0230b0191
serial = 0B0191489A946600
path checker = readsector0 (internal default)
state = 2
getprio = /bin/true (internal default)
prio = 0
uid = 3600d0230000000000b0191489a946600 (cache)
===== path info sdd (mask 0x1f) =====
bus = 1
dev_t = 8:48
size = 204800000
vendor = IFT
product = A16F-R2221
rev = 347B
h:b:t:l = 2:0:2:0
tgt_node_name = 0x200000d0231b0191
serial = 0B0191489A946601
path checker = readsector0 (internal default)
state = 2
getprio = /bin/true (internal default)
prio = 0
uid = 3600d0230000000000b0191489a946601 (cache)
===== path info sde (mask 0x1f) =====
bus = 1
dev_t = 8:64
size = 409600000
vendor = IFT
product = A16F-R2221
rev = 347B
h:b:t:l = 2:0:3:0
tgt_node_name = 0x200000d0230b0191
serial = 0B0191489A946600
path checker = readsector0 (internal default)
state = 2
getprio = /bin/true (internal default)
prio = 0
uid = 3600d0230000000000b0191489a946600 (cache)
path sdf not found in pathvec

===== path info sdf (mask 0x1f) =====
bus = 1
dev_t = 8:80
sr0 blacklisted


After a 'multipath -F;multipath', 'multipath -v3' gives me this:
#
# all paths in cache :
#
3600d0230000000000b0191489a946601  1:0:2:0 sdb 8:16  [ready] IFT
/A16F-R22
3600d0230000000000b0191489a946600  1:0:3:0 sdc 8:32  [ready] IFT
/A16F-R22
3600d0230000000000b0191489a946601  2:0:2:0 sdd 8:48  [ready] IFT
/A16F-R22
3600d0230000000000b0191489a946600  2:0:3:0 sde 8:64  [ready] IFT
/A16F-R22
*blacklist*
===== path info sdb (mask 0x1f) =====
bus = 1
dev_t = 8:16
size = 204800000
vendor = IFT
product = A16F-R2221
rev = 347B
h:b:t:l = 1:0:2:0
tgt_node_name = 0x200000d0231b0191
serial = 0B0191489A946601
path checker = readsector0 (internal default)
state = 2
getprio = /bin/true (internal default)
prio = 0
uid = 3600d0230000000000b0191489a946601 (cache)
===== path info sdc (mask 0x1f) =====
bus = 1
dev_t = 8:32
size = 409600000
vendor = IFT
product = A16F-R2221
rev = 347B
h:b:t:l = 1:0:3:0
tgt_node_name = 0x200000d0230b0191
serial = 0B0191489A946600
path checker = readsector0 (internal default)
state = 2
getprio = /bin/true (internal default)
prio = 0
uid = 3600d0230000000000b0191489a946600 (cache)
===== path info sdd (mask 0x1f) =====
bus = 1
dev_t = 8:48
size = 204800000
vendor = IFT
product = A16F-R2221
rev = 347B
h:b:t:l = 2:0:2:0
tgt_node_name = 0x200000d0231b0191
serial = 0B0191489A946601
path checker = readsector0 (internal default)
state = 2
getprio = /bin/true (internal default)
prio = 0
uid = 3600d0230000000000b0191489a946601 (cache)
===== path info sde (mask 0x1f) =====
bus = 1
dev_t = 8:64
size = 409600000
vendor = IFT
product = A16F-R2221
rev = 347B
h:b:t:l = 2:0:3:0
tgt_node_name = 0x200000d0230b0191
serial = 0B0191489A946600
path checker = readsector0 (internal default)
state = 2
getprio = /bin/true (internal default)
prio = 0
uid = 3600d0230000000000b0191489a946600 (cache)
path sdf not found in pathvec

===== path info sdf (mask 0x1f) =====
bus = 1
dev_t = 8:80
sr0 blacklisted


Running multipathd with -v4 doesn't give me any relevant output.


Anyone got a clue how to solve this problem?

Re: Problems with multipathing

[Christophe Varoqui]

Roger Håkansson a écrit :
> I'm trying to setup a couple of machines running CentOS 4.3 x86_64 using
> disk from a SAN, but I can't get the multipathing to work properly.
>
> The machines (three in total) all have two Emulex LP10000-M2 and thus
> are using the lpfc-driver.
>
> When I boot the machines the paths seem to be multipathed somewhat, if i
> put a "multipath"-tag in /etc/multipath.conf with a specific alias
> (yellow,red) for each wwid, those paths are mapped under /dev/mapper and
> /dev/mpath.
>
> But multipath -ll doesn't show what I expect (basically no mappings) and
> if I do a multipath -F I can't get the mappings back without a reboot.
>
> If I do a 'multipath -v3' directly after boot, this is what I get:
> ...
> Anyone got a clue how to solve this problem?
>
>   
Can you reproduce with upstream version ?

Regards,
cvaroqui

Re: Problems with multipathing

[Roger Håkansson]

Christophe Varoqui wrote:
>
> Can you reproduce with upstream version ?
> 

With upstream I guess you mean 0.4.7 or "CVS-HEAD".

Haven't tried that, but just looking at the requirements tells me I'll
have a lot to do in order to even just to prepare to test it.

>  Dependencies :
>  Linux kernel
>
>      * 2.6.10-rc*-udm2 or later
>      * 2.6.11-mm* or later
>      * 2.6.12-rc1 or later
>  udev 050+

CentOS 4.3 have 2.6.9-34 and udev-039-10, and even though some stuff are
backported I guess I have to update both and I can only imagine the
amount of work that will render me, but I'll try to do it if I can find
the time...

	

Re: Problems with multipathing

[Roger Håkansson]

Roger Håkansson wrote:
> 
> With upstream I guess you mean 0.4.7 or "CVS-HEAD".
> 
> Haven't tried that, but just looking at the requirements tells me I'll
> have a lot to do in order to even just to prepare to test it.
> 
>>  Dependencies :
>>  Linux kernel
>>
>>      * 2.6.10-rc*-udm2 or later
>>      * 2.6.11-mm* or later
>>      * 2.6.12-rc1 or later
>>  udev 050+
> 
> CentOS 4.3 have 2.6.9-34 and udev-039-10, and even though some stuff are
> backported I guess I have to update both and I can only imagine the
> amount of work that will render me, but I'll try to do it if I can find
> the time...
> 
> 	

0.4.7 seems to work better without updating kernel or udev, but not
entirely...

Unless I've gotten this wrong, with pathgroupingpolicy set to failover I
should get two pathgroups where only one is active an if the active
fails, the other pathgroup will become active, correct?
Multibus pathgrouping will place all paths in the same pathgroup so that
all paths will share the I/O when they are active and if some path
fails, the I/O is spread among the active paths, correct?

Multibus works just like I expect it to, but failover doesn't fail the
path entirely.

This is what 'multipath -ll' gives me after I have disconnected one HBA
from the fabric.

mpath1 (3600d0230000000000b0191489a946602)
[size=183 GB][features=0][hwhandler=0]
\_ round-robin 0 [prio=0][enabled]
 \_ 1:0:1:1 sdc 8:32  [active][faulty]
\_ round-robin 0 [prio=0][enabled]
 \_ 2:0:0:1 sdf 8:80  [active][ready]
mpath2 (3600d0230000000000b0191489a946600)
[size=97 GB][features=0][hwhandler=0]
\_ round-robin 0 [prio=0][active]
 \_ 1:0:1:0 sdb 8:16  [failed][faulty]
 \_ 2:0:0:0 sde 8:64  [active][ready]

Notice that sdc is faulty, but still active and both pathgroups are
enabled but none is active...

I've noticed this problem when I/O is active ( I was running a 'dd
if=/dev/zero of=/mount_point count=10000000000' to each mpath) when one
"path" fails, if there is no activity at all the failover works.

Re: Problems with multipathing

[Christophe Varoqui]

Roger Håkansson a écrit :
> Roger Håkansson wrote:
>   
>> With upstream I guess you mean 0.4.7 or "CVS-HEAD".
>>
>> Haven't tried that, but just looking at the requirements tells me I'll
>> have a lot to do in order to even just to prepare to test it.
>>
>>     
>>>  Dependencies :
>>>  Linux kernel
>>>
>>>      * 2.6.10-rc*-udm2 or later
>>>      * 2.6.11-mm* or later
>>>      * 2.6.12-rc1 or later
>>>  udev 050+
>>>       
>> CentOS 4.3 have 2.6.9-34 and udev-039-10, and even though some stuff are
>> backported I guess I have to update both and I can only imagine the
>> amount of work that will render me, but I'll try to do it if I can find
>> the time...
>>
>> 	
>>     
>
> 0.4.7 seems to work better without updating kernel or udev, but not
> entirely...
>
> Unless I've gotten this wrong, with pathgroupingpolicy set to failover I
> should get two pathgroups where only one is active an if the active
> fails, the other pathgroup will become active, correct?
> Multibus pathgrouping will place all paths in the same pathgroup so that
> all paths will share the I/O when they are active and if some path
> fails, the I/O is spread among the active paths, correct?
>
> Multibus works just like I expect it to, but failover doesn't fail the
> path entirely.
>
> This is what 'multipath -ll' gives me after I have disconnected one HBA
> from the fabric.
>
> mpath1 (3600d0230000000000b0191489a946602)
> [size=183 GB][features=0][hwhandler=0]
> \_ round-robin 0 [prio=0][enabled]
>  \_ 1:0:1:1 sdc 8:32  [active][faulty]
> \_ round-robin 0 [prio=0][enabled]
>  \_ 2:0:0:1 sdf 8:80  [active][ready]
> mpath2 (3600d0230000000000b0191489a946600)
> [size=97 GB][features=0][hwhandler=0]
> \_ round-robin 0 [prio=0][active]
>  \_ 1:0:1:0 sdb 8:16  [failed][faulty]
>  \_ 2:0:0:0 sde 8:64  [active][ready]
>
> Notice that sdc is faulty, but still active and both pathgroups are
> enabled but none is active...
>
> I've noticed this problem when I/O is active ( I was running a 'dd
> if=/dev/zero of=/mount_point count=10000000000' to each mpath) when one
> "path" fails, if there is no activity at all the failover works.
>
>   
I don't know your hardware (vendor = IFT, product = A16F-R2221) but it 
seems assymmetrical. Most hardware in this familly need a hardware 
handler, and some need the "queue_if_no_path" feature set too.

You'll have to find how your array works and try to figure if some 
existing hardware handler does the good thing.

As a last resort, post the maximum techical details about what  your 
hardware needs to activate backup paths, and hope that some good soul is 
willing to code the handler.

Regards,
cvaroqui

Re: Problems with multipathing

[Roger Håkansson]

[-- Attachment #1.1: Type: text/plain, Size: 2938 bytes --]

Christophe Varoqui wrote:
> I don't know your hardware (vendor = IFT, product = A16F-R2221) 

http://www.infortrend.com/main/2_product/a16f-r2221.asp

> but it seems assymmetrical.

I guess that you with "asymmetrical" means that paths are only presented
on one controller at a time, more on that later.

> Most hardware in this familly need a hardware
> handler, and some need the "queue_if_no_path" feature set too.
>
> You'll have to find how your array works and try to figure if some
> existing hardware handler does the good thing.

I've done some testing and it seems that multibus works fine, but when a
controller fails and the secondary controller takes over, the
scsi-devices are seen as "dead" and if I, before multipath determines
both paths to be permanently faulty, do a "echo 1 >
/sys/class/scsi_device/1:0:0:0/device/rescan", multipath will not fail
the device.

So the question is if this is something a hardware_handler could do, and
if so, how hard is it to write such a handler?

> As a last resort, post the maximum techical details about what  your
> hardware needs to activate backup paths, and hope that some good soul is
> willing to code the handler.

Ok, I'll give you as much info as I can (if its of any use is another
thing...).

The "box" has two controllers, which Infortrend calls active-active,
which in this case means that one can create logical drives (and
optionally logical volumes) which are assigned to one of the
controllers, but a logical drive can only be active on one controller at
a time.
The "box" also has two "channels" which are connected to both
controllers, each controller can be assigned multiple "host controller
IDs" on each channel.
"Host controller ID-index" (lowest "host controller id" on primary
controller will have 0, next 1.. to N, lowest on secondary will have N+1
and so on) together with some static pieces (MAC-address) form the WWNN,
beginning with 20.
All targets on "channel 0" will have WWPN beginning with 21 and the rest
identical to the WWNN, targets on "channel 1" begin with 22.

When a controller fails, the other one takes over all logical drives,
LUNs, WWNN and WWPN.

Another thing which I don't know if its important or not, but the "box"
have four fysical SFP-ports which can be configured in two ways, either
in "hubbed mode" where two are "hubbed" together on one "channel" on
both controllers and the other two on the other channel, or "non-hubbed
mode" where each port are staticly bound to a specific "channel" on a
specific controller.
As far as I've been able to tell, there is no difference in behaviour
between hubbed/non-hubbed mode
-- 
Roger Håkansson , Senior Consultant
Algitech Consulting AB
Södra Kungsgatan 5 (Office)
Box 28, S-971 02 Luleå , Sweden
Tel:    +46 (0)920 88313    Mobile: +46 (0)705 549533
Fax:    +1 (0)928 222 7116  E-mail: roger.hakansson@algitech.com


[-- Attachment #1.2: S/MIME Cryptographic Signature --]
[-- Type: application/x-pkcs7-signature, Size: 3276 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Problems with multipathing

[Roger Håkansson]

Christophe Varoqui wrote:
> > I don't know your hardware (vendor = IFT, product = A16F-R2221)

http://www.infortrend.com/main/2_product/a16f-r2221.asp

> > but it seems assymmetrical.

I guess that you with "asymmetrical" means that paths are only presented
on one controller at a time, more on that later.

> > Most hardware in this familly need a hardware
> > handler, and some need the "queue_if_no_path" feature set too.
> >
> > You'll have to find how your array works and try to figure if some
> > existing hardware handler does the good thing.

I've done some testing and it seems that multibus works fine, but when a
controller fails and the secondary controller takes over, the
scsi-devices are seen as "dead" and if I, before multipath determines
both paths to be permanently faulty, do a "echo 1 >
/sys/class/scsi_device/1:0:0:0/device/rescan", multipath will not fail
the device.

So the question is if this is something a hardware_handler could do, and
if so, how hard is it to write such a handler?

> > As a last resort, post the maximum techical details about what  your
> > hardware needs to activate backup paths, and hope that some good soul is
> > willing to code the handler.

Ok, I'll give you as much info as I can (if its of any use is another
thing...).

The "box" has two controllers, which Infortrend calls active-active,
which in this case means that one can create logical drives (and
optionally logical volumes) which are assigned to one of the
controllers, but a logical drive can only be active on one controller at
a time.
The "box" also has two "channels" which are connected to both
controllers, each controller can be assigned multiple "host controller
IDs" on each channel.
"Host controller ID-index" (lowest "host controller id" on primary
controller will have 0, next 1.. to N, lowest on secondary will have N+1
and so on) together with some static pieces (MAC-address) form the WWNN,
beginning with 20.
All targets on "channel 0" will have WWPN beginning with 21 and the rest
identical to the WWNN, targets on "channel 1" begin with 22.

When a controller fails, the other one takes over all logical drives,
LUNs, WWNN and WWPN.

Another thing which I don't know if its important or not, but the "box"
have four fysical SFP-ports which can be configured in two ways, either
in "hubbed mode" where two are "hubbed" together on one "channel" on
both controllers and the other two on the other channel, or "non-hubbed
mode" where each port are staticly bound to a specific "channel" on a
specific controller.
As far as I've been able to tell, there is no difference in behaviour
between hubbed/non-hubbed mode

Re: Problems with multipathing

[Christophe Varoqui]

Roger Håkansson a écrit :
>>> but it seems assymmetrical.
>>>       
>
> I guess that you with "asymmetrical" means that paths are only presented
> on one controller at a time, more on that later.
>
>   
Yes, and the "transparent path switching with a penalty" family, which 
your hardware seems to be part of. Those usually don't need a hardware 
handler.

>>> Most hardware in this familly need a hardware
>>> handler, and some need the "queue_if_no_path" feature set too.
>>>
>>> You'll have to find how your array works and try to figure if some
>>> existing hardware handler does the good thing.
>>>       
>
> I've done some testing and it seems that multibus works fine, but when a
> controller fails and the secondary controller takes over, the
> scsi-devices are seen as "dead" and if I, before multipath determines
> both paths to be permanently faulty, do a "echo 1 >
> /sys/class/scsi_device/1:0:0:0/device/rescan", multipath will not fail
> the device.
>
>   
Do failover device nodes get reassigned during the rescan ?
Like, for example, a configured path sda gets removed and a new path sdb 
appears ?
If so, the FC transport class is in charge of the timeout triggering the 
dead devices removal.
A hardware handler wouldn't help here.

Can you paste a before/after scsi rescan "multipath -l" output ?

Regards,
cvaroqui

Re: Problems with multipathing

[Roger Håkansson]

Christophe Varoqui wrote:
>
> Do failover device nodes get reassigned during the rescan ?
> Like, for example, a configured path sda gets removed and a new path sdb
> appears ?

No, since I don't do a rescan on the bus but just on the target itself.
When I had the controller in non-hubbed mode and did (when a controller
has failed) "echo 1> /sys/class/scsi_host/host[1-2]/scan" I got two new
devices, sde and sdf (I normally havesda,sdb,sdc and sdd)
But if I instead did "echo 1 >
/sys/class/scsi_device/[1-2]:0:0:0/device/rescan", I didn't get any new
devices but the old ones start working again.
Now, when I have the box in hubbed-mode, I can't seem to get new devices
 even when I do a scsi-host-scan, but just as before, a
scsi-target-rescan will get my devices back to order again.

Also, I've noticed that it's not only when a controller fails that this
happens, when a failed controller is "revived" the same thing might happen.

As far as I've been able to tell, the more I/O-transactions at the time
of the failure, the more likely that the (SCSI) device will be marked as
"dead".
If I do "while /bin/true ;do dd if=/dev/zero of=/mnt/test count=20000;
sleep 1" and fail (or revive) a controller it seems to work in 50% of
the cases, with 2 sec sleep there is rarely any problem but with no
sleep at all it fails nearly 100% of the times
And in all types of tests, if I do a SCSI-target(path) rescan before
multipath decides both paths are dead, both paths will work again and
the multipath-device will never fail.

> If so, the FC transport class is in charge of the timeout triggering the
> dead devices removal.
> A hardware handler wouldn't help here.
> 
> Can you paste a before/after scsi rescan "multipath -l" output ?
> 

They are identical

[root@asl005 ~]# multipath -l
mpath1 (3600d0230000000000b01910b4d313400)
[size=97 GB][features=0][hwhandler=0]
\_ round-robin 0 [prio=0][active]
 \_ 1:0:0:0 sdb 8:16  [active][undef]
 \_ 2:0:0:0 sdc 8:32  [active][undef]
[root@asl005 ~]# dmesg |tail -20
SCSI error : <1 0 0 0> return code = 0x20008
end_request: I/O error, dev sdb, sector 21247352
end_request: I/O error, dev sdb, sector 21247360
SCSI error : <1 0 0 0> return code = 0x20008
end_request: I/O error, dev sdb, sector 21036576
end_request: I/O error, dev sdb, sector 21036584
Aborting journal on device dm-5.
ext3_abort called.
EXT3-fs error (device dm-5): ext3_journal_start_sb: Detected aborted journal
Remounting filesystem read-only
EXT3-fs error (device dm-5) in start_transaction: Journal has aborted
__journal_remove_journal_head: freeing b_committed_data
printk: 254766 messages suppressed.
Buffer I/O error on device dm-5, logical block 2092209
lost page write due to I/O error on dm-5
Buffer I/O error on device dm-5, logical block 2093234
lost page write due to I/O error on dm-5
printk: 485 messages suppressed.
Buffer I/O error on device dm-5, logical block 1
lost page write due to I/O error on dm-5
[root@asl005 ~]# multipath -l
mpath1 (3600d0230000000000b01910b4d313400)
[size=97 GB][features=0][hwhandler=0]
\_ round-robin 0 [prio=0][enabled]
 \_ 1:0:0:0 sdb 8:16  [failed][undef]
 \_ 2:0:0:0 sdc 8:32  [failed][undef]
[root@asl005 ~]# echo 1 >
/sys/class/fc_transport/target1\:0\:0/device/1\:0\:0\:0/rescan
[root@asl005 ~]# multipath -l
mpath1 (3600d0230000000000b01910b4d313400)
[size=97 GB][features=0][hwhandler=0]
\_ round-robin 0 [prio=0][enabled]
 \_ 1:0:0:0 sdb 8:16  [failed][undef]
 \_ 2:0:0:0 sdc 8:32  [failed][undef]
[root@asl005 ~]# multipath
[root@asl005 ~]# multipath -ll
mpath1 (3600d0230000000000b01910b4d313400)
[size=97 GB][features=0][hwhandler=0]
\_ round-robin 0 [prio=0][active]
 \_ 1:0:0:0 sdb 8:16  [active][undef]
 \_ 2:0:0:0 sdc 8:32  [active][undef]

Re: Problems with multipathing

[Christophe Varoqui]

Roger Håkansson a écrit :
> Christophe Varoqui wrote:
>   
>> Do failover device nodes get reassigned during the rescan ?
>> Like, for example, a configured path sda gets removed and a new path sdb
>> appears ?
>>     
>
> No, since I don't do a rescan on the bus but just on the target itself.
> When I had the controller in non-hubbed mode and did (when a controller
> has failed) "echo 1> /sys/class/scsi_host/host[1-2]/scan" I got two new
> devices, sde and sdf (I normally havesda,sdb,sdc and sdd)
> But if I instead did "echo 1 >
> /sys/class/scsi_device/[1-2]:0:0:0/device/rescan", I didn't get any new
> devices but the old ones start working again.
> Now, when I have the box in hubbed-mode, I can't seem to get new devices
>  even when I do a scsi-host-scan, but just as before, a
> scsi-target-rescan will get my devices back to order again.
>
>   
ok.
have you tried sending a START_STOP scsi command (wit sg_start from 
sg3_utils) to the affect'ed LUN instead of target-rescaning ?
> Also, I've noticed that it's not only when a controller fails that this
> happens, when a failed controller is "revived" the same thing might happen.
>
> As far as I've been able to tell, the more I/O-transactions at the time
> of the failure, the more likely that the (SCSI) device will be marked as
> "dead".
> If I do "while /bin/true ;do dd if=/dev/zero of=/mnt/test count=20000;
> sleep 1" and fail (or revive) a controller it seems to work in 50% of
> the cases, with 2 sec sleep there is rarely any problem but with no
> sleep at all it fails nearly 100% of the times
> And in all types of tests, if I do a SCSI-target(path) rescan before
> multipath decides both paths are dead, both paths will work again and
> the multipath-device will never fail.
>
>   
I see your features still don't include "queue_if_no_path". You seem to 
really need it.
>> If so, the FC transport class is in charge of the timeout triggering the
>> dead devices removal.
>> A hardware handler wouldn't help here.
>>
>> Can you paste a before/after scsi rescan "multipath -l" output ?
>>
>>     
>
> They are identical
>
> [root@asl005 ~]# multipath -l
> mpath1 (3600d0230000000000b01910b4d313400)
> [size=97 GB][features=0][hwhandler=0]
> \_ round-robin 0 [prio=0][active]
>  \_ 1:0:0:0 sdb 8:16  [active][undef]
>  \_ 2:0:0:0 sdc 8:32  [active][undef]
> [root@asl005 ~]# dmesg |tail -20
> SCSI error : <1 0 0 0> return code = 0x20008
> end_request: I/O error, dev sdb, sector 21247352
> end_request: I/O error, dev sdb, sector 21247360
> SCSI error : <1 0 0 0> return code = 0x20008
> end_request: I/O error, dev sdb, sector 21036576
> end_request: I/O error, dev sdb, sector 21036584
> Aborting journal on device dm-5.
> ext3_abort called.
> EXT3-fs error (device dm-5): ext3_journal_start_sb: Detected aborted journal
> Remounting filesystem read-only
> EXT3-fs error (device dm-5) in start_transaction: Journal has aborted
> __journal_remove_journal_head: freeing b_committed_data
> printk: 254766 messages suppressed.
> Buffer I/O error on device dm-5, logical block 2092209
> lost page write due to I/O error on dm-5
> Buffer I/O error on device dm-5, logical block 2093234
> lost page write due to I/O error on dm-5
> printk: 485 messages suppressed.
> Buffer I/O error on device dm-5, logical block 1
> lost page write due to I/O error on dm-5
> [root@asl005 ~]# multipath -l
> mpath1 (3600d0230000000000b01910b4d313400)
> [size=97 GB][features=0][hwhandler=0]
> \_ round-robin 0 [prio=0][enabled]
>  \_ 1:0:0:0 sdb 8:16  [failed][undef]
>  \_ 2:0:0:0 sdc 8:32  [failed][undef]
> [root@asl005 ~]# echo 1 >
> /sys/class/fc_transport/target1\:0\:0/device/1\:0\:0\:0/rescan
> [root@asl005 ~]# multipath -l
> mpath1 (3600d0230000000000b01910b4d313400)
> [size=97 GB][features=0][hwhandler=0]
> \_ round-robin 0 [prio=0][enabled]
>  \_ 1:0:0:0 sdb 8:16  [failed][undef]
>  \_ 2:0:0:0 sdc 8:32  [failed][undef]
> [root@asl005 ~]# multipath
> [root@asl005 ~]# multipath -ll
> mpath1 (3600d0230000000000b01910b4d313400)
> [size=97 GB][features=0][hwhandler=0]
> \_ round-robin 0 [prio=0][active]
>  \_ 1:0:0:0 sdb 8:16  [active][undef]
>  \_ 2:0:0:0 sdc 8:32  [active][undef]
>
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
>
>   

Re: Problems with multipathing

[Roger Håkansson]

Christophe Varoqui wrote:
> ok.
> have you tried sending a START_STOP scsi command (wit sg_start from
> sg3_utils) to the affect'ed LUN instead of target-rescaning ?

No, but I'll try that.
I've noticed that you(?) have written something about that in the
"TestedEnvironments" section, but how does this "integrate" with
multipath-tools?

> I see your features still don't include "queue_if_no_path". You seem to
> really need it.

Yup, I've added that in my later tests and it seems to work as expected.

Re: Problems with multipathing

[James Smart]

Christophe Varoqui wrote:
> Do failover device nodes get reassigned during the rescan ?
> Like, for example, a configured path sda gets removed and a new path sdb 
> appears ?
> If so, the FC transport class is in charge of the timeout triggering the 
> dead devices removal.
> A hardware handler wouldn't help here.

On the kernel rev he's talking about, 2.6.9-34, the FC transport class
doesn't fully exist, so there's no automatic device removal.

-- james

Re: Problems with multipathing

[James Smart]

> Roger Håkansson a écrit :
>> Also, I've noticed that it's not only when a controller fails that this
>> happens, when a failed controller is "revived" the same thing might 
>> happen.
>>
>> As far as I've been able to tell, the more I/O-transactions at the time
>> of the failure, the more likely that the (SCSI) device will be marked as
>> "dead".

Hmmm.. I'm wondering if he's hitting the scenario in which the midlayer
marks the sdev in an offline state - which could be the "dead" state.
This occurs if an i/o hits the LLDD when the device is disconnected, and
error recovery fails. If so, at a later time when the LLDD has connectivity
and can access the device, the scsi layer would still likely bounce i/o.
It requires a manual interaction to change it back to a running state,
any i/o requests by dm would be failed back by the midlayer.

What doesn't jive is the rescan re-enabling the device. As I stated, this
is usually a manual action to restore things. If the rescans are just
prior to the transition to the offline state, they may be making dm change
it's path mappings to avoid i/o to the failed path, thus deflecting the
sdev transition.  Can you report the contents of
/sys/class/scsi_device/1:0:*/device/state  at the following states in both
the works and does not work cases :
   working, right after failover but before dm fails it; after failure/success

-- james