Sun StorageTek 2530 and dm?

Sun StorageTek 2530 and dm?

[Jakov Sosic]

Hi!

I have StorageTek 2530 storage, which has two controllers. On host side
I have LSI LOGIC two SAS port cards. I've connected one cable from
LSI port1 to StorageTek ControllerA, and second SAS cable from LSI
port2 to StorageTek ControllerB.

On storage, I've set up initiators type to Linux.

But, I just can't get multipath working. For example:

# multipath -ll
sdc: checker msg is "readsector0 checker reports path is down"
sde: checker msg is "readsector0 checker reports path is down"
sdg: checker msg is "readsector0 checker reports path is down"
sas-mysql (3600a0b80003abc5c00000b654a916617) dm-1 SUN,LCSM100_S
[size=115G][features=0][hwhandler=0][rw]
\_ round-robin 0 [prio=0][active]
 \_ 1:0:0:1 sdc 8:32  [failed][faulty]
 \_ 1:0:1:1 sdf 8:80  [active][ready] 
sas-xen (3600a0b80002fcd18000014544a9166fb) dm-2 SUN,LCSM100_S
[size=21G][features=0][hwhandler=0][rw]
\_ round-robin 0 [prio=0][active]
 \_ 1:0:0:2 sdd 8:48  [active][ready] 
 \_ 1:0:1:2 sdg 8:96  [failed][faulty]
quorum (3600a0b80002fcd18000014514a9165b4) dm-0 SUN,LCSM100_S
[size=100M][features=0][hwhandler=0][rw]
\_ round-robin 0 [prio=0][active]
 \_ 1:0:0:0 sdb 8:16  [active][ready] 
 \_ 1:0:1:0 sde 8:64  [failed][faulty]

So, all paths on my second controller are failed/faulty....

Here is my multipath.conf:

defaults {
	udev_dir		/dev
	polling_interval 	10
	selector		"round-robin 0"
	path_grouping_policy	multibus
	getuid_callout		"/sbin/scsi_id -g -u -s /block/%n"
	prio_callout		/bin/true
	path_checker		readsector0
	rr_min_io		100
	max_fds			8192
	rr_weight		priorities
	failback		immediate
	no_path_retry		fail
	user_friendly_names	yes
}
blacklist {
	devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
	devnode "^hd[a-z]"
}
multipaths {
	multipath {
		wwid
3600a0b80002fcd18000014544a9166fb
		alias			sas-xen
		path_grouping_policy	group_by_prio
		path_checker		tur
		prio_callout		"/sbin/mpath_prio_tpc.static /dev/%n"
	}
	multipath {
		wwid
3600a0b80003abc5c00000b654a916617
		alias			sas-mysql
		path_grouping_policy	group_by_prio
		path_checker		tur
		prio_callout		"/sbin/mpath_prio_tpc.static /dev/%n"
	}	
	multipath {
		wwid
3600a0b80002fcd18000014514a9165b4
		alias			quorum
		path_grouping_policy	group_by_prio
		path_checker		tur
		prio_callout		"/sbin/mpath_prio_tpc.static /dev/%n"
	}
}



I've also tried:
	multipath {
		wwid
		alias
		path_grouping_policy	failover
		path_checker		readsector0
		path_selector		"round-robin 0"
		failback		immediate
		no_path_retry		queue
	}

for those 3 wwid's...



I'm kinda lost... I've found this, but it is for FC storage, and mine
is SAS type...
http://osdir.com/ml/linux.kernel.device-mapper.devel/2007-02/msg00033.html

Thank you...







-- 
|    Jakov Sosic    |    ICQ: 28410271    |   PGP: 0x965CAE2D   |
=================================================================
| start fighting cancer -> http://www.worldcommunitygrid.org/   |

Re: Sun StorageTek 2530 and dm?

[Jakov Sosic]

On Sun, 23 Aug 2009 21:05:36 +0200
Jakov Sosic <jakov.sosic@srce.hr> wrote:

> Hi!
> 
> I have StorageTek 2530 storage, which has two controllers. On host
> side I have LSI LOGIC two SAS port cards. I've connected one cable
> from LSI port1 to StorageTek ControllerA, and second SAS cable from
> LSI port2 to StorageTek ControllerB.


OK. I've solved it - it seems that Linux host type in initiator
configuration is not turning on AVT on controllers. Instead, I had to
choose "Solaris (with Veritas DMP or other)", and now everything works!



Thank you, and hope this helps to others too.

-- 
|    Jakov Sosic    |    ICQ: 28410271    |   PGP: 0x965CAE2D   |
=================================================================
| start fighting cancer -> http://www.worldcommunitygrid.org/   |

Re: Sun StorageTek 2530 and dm?

[Charlie Brady]

On Sun, 23 Aug 2009, Jakov Sosic wrote:

> On Sun, 23 Aug 2009 21:05:36 +0200
> Jakov Sosic <jakov.sosic@srce.hr> wrote:
>
>> Hi!
>>
>> I have StorageTek 2530 storage, which has two controllers. On host
>> side I have LSI LOGIC two SAS port cards. I've connected one cable
>> from LSI port1 to StorageTek ControllerA, and second SAS cable from
>> LSI port2 to StorageTek ControllerB.
>
> OK. I've solved it - it seems that Linux host type in initiator
> configuration is not turning on AVT on controllers. Instead, I had to
> choose "Solaris (with Veritas DMP or other)", and now everything works!
>
> Thank you, and hope this helps to others too.

I know that other people here will know more detail, but I think you have 
another alternative which might be superior.

Instead of changing the initiator configuration to use AVT, you should 
tune the device mapper to use RDAC.

To do that, I think you will need to adjust both the multipathd 
configuration and recompile the scsi_dh_rdac kernel module after adding 
support for "SUN"/"LCSM100_S".

Here's the RDAC multipath.conf I am using with a StorageTek 2510:

http://www.linux-archive.org/device-mapper-development/323661-multipathd-configuration-sun-storagetek-2510-a.html

Re: Sun StorageTek 2530 and dm?

[Jakov Sosic]

On Mon, 24 Aug 2009 11:41:34 -0400 (EDT)
Charlie Brady <charlieb-dm-devel@budge.apana.org.au> wrote:

> I know that other people here will know more detail, but I think you
> have another alternative which might be superior.
> 
> Instead of changing the initiator configuration to use AVT, you
> should tune the device mapper to use RDAC.
> 
> To do that, I think you will need to adjust both the multipathd 
> configuration and recompile the scsi_dh_rdac kernel module after
> adding support for "SUN"/"LCSM100_S".
> 
> Here's the RDAC multipath.conf I am using with a StorageTek 2510:
> 
> http://www.linux-archive.org/device-mapper-development/323661-multipathd-configuration-sun-storagetek-2510-a.html

Thank you!

The problem is I use XEN kernel from CentOS and I really don't want to
mess up with manual compilation of every new kernel that comes out...

Do you know if RedHat incorporated this patch already?

If not, I'll keep up with AVT - it seems that works OK. Some paths fail
every now and then but are back online in just a couple of seconds, and
that is currently OK.



-- 
|    Jakov Sosic    |    ICQ: 28410271    |   PGP: 0x965CAE2D   |
=================================================================
| start fighting cancer -> http://www.worldcommunitygrid.org/   |

Re: Sun StorageTek 2530 and dm?

[Jakov Sosic]

On Mon, 24 Aug 2009 17:48:34 +0200
Jakov Sosic <jakov.sosic@srce.hr> wrote:

I have this occuring in logs, I presume it is ok?

Aug 24 17:58:43 node01 multipathd: sdc: readsector0 checker reports
path is down
Aug 24 17:58:43 node01 multipathd: checker failed path 8:32 in map
sas-mysql
Aug 24 17:58:43 node01 multipathd: sas-mysql: remaining active paths: 1
Aug 24 17:58:43 node01 kernel: device-mapper: multipath: Failing path
8:32.
Aug 24 17:58:43 node01 multipathd: dm-1: add map (uevent)
Aug 24 17:58:43 node01 multipathd: dm-1: devmap already registered
Aug 24 17:58:53 node01 multipathd: sdc: readsector0 checker reports
path is up
Aug 24 17:58:53 node01 multipathd: 8:32: reinstated
Aug 24 17:58:53 node01 multipathd: sas-mysql: remaining active paths: 2
Aug 24 17:58:53 node01 multipathd: dm-1: add map (uevent)
Aug 24 17:58:53 node01 multipathd: dm-1: devmap already registered 


Because multipath -ll says:

# multipath -ll sas-mysql
sas-mysql (3600a0b80003abc5c00000b654a916617) dm-1 SUN,LCSM100_S
[size=115G][features=0][hwhandler=0][rw]
\_ round-robin 0 [prio=0][active]
 \_ 1:0:0:1  sdc 8:32  [active][ready]
 \_ 1:0:1:1  sdf 8:80  [active][ready]




-- 
|    Jakov Sosic    |    ICQ: 28410271    |   PGP: 0x965CAE2D   |
=================================================================
| start fighting cancer -> http://www.worldcommunitygrid.org/   |

Re: Sun StorageTek 2530 and dm?

[Jakov Sosic]

On Mon, 24 Aug 2009 11:41:34 -0400 (EDT)
Charlie Brady <charlieb-dm-devel@budge.apana.org.au> wrote:

> I know that other people here will know more detail, but I think you
> have another alternative which might be superior.
> 
> Instead of changing the initiator configuration to use AVT, you
> should tune the device mapper to use RDAC.
> 
> To do that, I think you will need to adjust both the multipathd 
> configuration and recompile the scsi_dh_rdac kernel module after
> adding support for "SUN"/"LCSM100_S".
> 
> Here's the RDAC multipath.conf I am using with a StorageTek 2510:
> 
> http://www.linux-archive.org/device-mapper-development/323661-multipathd-configuration-sun-storagetek-2510-a.html

Aug 24 20:25:55 node03 kernel: device-mapper: multipath: Failing path
8:16.
Aug 24 20:25:55 node03 multipathd: 8:16: mark as failed 
Aug 24 20:25:55 node03 multipathd: qd: remaining active paths: 1 
Aug 24 20:25:55 node03 multipathd: dm-0: add map (uevent) 
Aug 24 20:25:55 node03 multipathd: dm-0: devmap already registered 
Aug 24 20:25:55 node03 kernel: sd 1:0:1:0: queueing MODE_SELECT command.
Aug 24 20:25:56 node03 qdiskd[5325]: <notice> Score sufficient for
master operation (1/1; required=1); upgrading
Aug 24 20:26:02 node03 multipathd: 8:16: reinstated
Aug 24 20:26:02 node03 multipathd: qd: remaining active paths: 2
Aug 24 20:26:02 node03 multipathd: qd: switch to path group #1
Aug 24 20:26:02 node03 multipathd: qd: switch to path group #1
Aug 24 20:26:02 node03 multipathd: dm-0: add map (uevent)
Aug 24 20:26:02 node03 multipathd: dm-0: devmap already registered
Aug 24 20:26:05 node03 kernel: device-mapper: multipath: Failing path
8:16.
Aug 24 20:26:05 node03 multipathd: 8:16: mark as failed
Aug 24 20:26:05 node03 multipathd: qd: remaining active paths: 1
Aug 24 20:26:05 node03 multipathd: dm-0: add map (uevent)
Aug 24 20:26:05 node03 multipathd: dm-0: devmap already registered
Aug 24 20:26:05 node03 kernel: sd 1:0:1:0: queueing MODE_SELECT command.
Aug 24 20:26:12 node03 multipathd: 8:16: reinstated
Aug 24 20:26:12 node03 multipathd: qd: remaining active paths: 2
Aug 24 20:26:12 node03 multipathd: qd: switch to path group #1
Aug 24 20:26:12 node03 multipathd: qd: switch to path group #1
Aug 24 20:26:12 node03 multipathd: dm-0: add map (uevent)
Aug 24 20:26:12 node03 multipathd: dm-0: devmap already registered
Aug 24 20:26:15 node03 kernel: device-mapper: multipath: Failing path
8:16.
Aug 24 20:26:15 node03 multipathd: 8:16: mark as failed
Aug 24 20:26:15 node03 multipathd: qd: remaining active paths: 1
Aug 24 20:26:15 node03 multipathd: dm-0: add map (uevent)
Aug 24 20:26:15 node03 multipathd: dm-0: devmap already registered
Aug 24 20:26:15 node03 kernel: sd 1:0:1:0: queueing MODE_SELECT command.


So, your configuration fills my logs too :( I don't know if this is
supposed to behave this way or not...

I'm lost now :(


-- 
|    Jakov Sosic    |    ICQ: 28410271    |   PGP: 0x965CAE2D   |
=================================================================
| start fighting cancer -> http://www.worldcommunitygrid.org/   |

Re: Sun StorageTek 2530 and dm?

[Chandra Seetharaman]

You need to have a stanza similar to following one for your device
in /etc/multipath.conf
--------------------
devices {
        device {
                vendor                  "IBM"
                product                 "1815"
                path_grouping_policy    group_by_prio
                hardware_handler        "1 rdac"
                path_checker            rdac
                failback                immediate
                no_path_retry           queue
                prio_callout            "/sbin/mpath_prio_tpc /dev/%n"
        }
}
---------------------

On Mon, 2009-08-24 at 18:00 +0200, Jakov Sosic wrote:
> On Mon, 24 Aug 2009 17:48:34 +0200
> Jakov Sosic <jakov.sosic@srce.hr> wrote:
> 
> I have this occuring in logs, I presume it is ok?
> 
> Aug 24 17:58:43 node01 multipathd: sdc: readsector0 checker reports
> path is down
> Aug 24 17:58:43 node01 multipathd: checker failed path 8:32 in map
> sas-mysql
> Aug 24 17:58:43 node01 multipathd: sas-mysql: remaining active paths: 1
> Aug 24 17:58:43 node01 kernel: device-mapper: multipath: Failing path
> 8:32.
> Aug 24 17:58:43 node01 multipathd: dm-1: add map (uevent)
> Aug 24 17:58:43 node01 multipathd: dm-1: devmap already registered
> Aug 24 17:58:53 node01 multipathd: sdc: readsector0 checker reports
> path is up
> Aug 24 17:58:53 node01 multipathd: 8:32: reinstated
> Aug 24 17:58:53 node01 multipathd: sas-mysql: remaining active paths: 2
> Aug 24 17:58:53 node01 multipathd: dm-1: add map (uevent)
> Aug 24 17:58:53 node01 multipathd: dm-1: devmap already registered 
> 
> 
> Because multipath -ll says:
> 
> # multipath -ll sas-mysql
> sas-mysql (3600a0b80003abc5c00000b654a916617) dm-1 SUN,LCSM100_S
> [size=115G][features=0][hwhandler=0][rw]
> \_ round-robin 0 [prio=0][active]
>  \_ 1:0:0:1  sdc 8:32  [active][ready]
>  \_ 1:0:1:1  sdf 8:80  [active][ready]
> 
> 
> 
> 

Re: Sun StorageTek 2530 and dm?

[Chandra Seetharaman]

This is the expected behavior, when IOs on the active path fails, the
hardware handler will activate the passive path and make it active and
start sending IOs thru that newly activated path.

Send MODE_SELECT is how the hardware handler makes the passive path
active. 
On Mon, 2009-08-24 at 20:29 +0200, Jakov Sosic wrote:
> On Mon, 24 Aug 2009 11:41:34 -0400 (EDT)
> Charlie Brady <charlieb-dm-devel@budge.apana.org.au> wrote:
> 
> > I know that other people here will know more detail, but I think you
> > have another alternative which might be superior.
> > 
> > Instead of changing the initiator configuration to use AVT, you
> > should tune the device mapper to use RDAC.
> > 
> > To do that, I think you will need to adjust both the multipathd 
> > configuration and recompile the scsi_dh_rdac kernel module after
> > adding support for "SUN"/"LCSM100_S".
> > 
> > Here's the RDAC multipath.conf I am using with a StorageTek 2510:
> > 
> > http://www.linux-archive.org/device-mapper-development/323661-multipathd-configuration-sun-storagetek-2510-a.html
> 
> Aug 24 20:25:55 node03 kernel: device-mapper: multipath: Failing path
> 8:16.
> Aug 24 20:25:55 node03 multipathd: 8:16: mark as failed 
> Aug 24 20:25:55 node03 multipathd: qd: remaining active paths: 1 
> Aug 24 20:25:55 node03 multipathd: dm-0: add map (uevent) 
> Aug 24 20:25:55 node03 multipathd: dm-0: devmap already registered 
> Aug 24 20:25:55 node03 kernel: sd 1:0:1:0: queueing MODE_SELECT command.
> Aug 24 20:25:56 node03 qdiskd[5325]: <notice> Score sufficient for
> master operation (1/1; required=1); upgrading
> Aug 24 20:26:02 node03 multipathd: 8:16: reinstated
> Aug 24 20:26:02 node03 multipathd: qd: remaining active paths: 2
> Aug 24 20:26:02 node03 multipathd: qd: switch to path group #1
> Aug 24 20:26:02 node03 multipathd: qd: switch to path group #1
> Aug 24 20:26:02 node03 multipathd: dm-0: add map (uevent)
> Aug 24 20:26:02 node03 multipathd: dm-0: devmap already registered
> Aug 24 20:26:05 node03 kernel: device-mapper: multipath: Failing path
> 8:16.
> Aug 24 20:26:05 node03 multipathd: 8:16: mark as failed
> Aug 24 20:26:05 node03 multipathd: qd: remaining active paths: 1
> Aug 24 20:26:05 node03 multipathd: dm-0: add map (uevent)
> Aug 24 20:26:05 node03 multipathd: dm-0: devmap already registered
> Aug 24 20:26:05 node03 kernel: sd 1:0:1:0: queueing MODE_SELECT command.
> Aug 24 20:26:12 node03 multipathd: 8:16: reinstated
> Aug 24 20:26:12 node03 multipathd: qd: remaining active paths: 2
> Aug 24 20:26:12 node03 multipathd: qd: switch to path group #1
> Aug 24 20:26:12 node03 multipathd: qd: switch to path group #1
> Aug 24 20:26:12 node03 multipathd: dm-0: add map (uevent)
> Aug 24 20:26:12 node03 multipathd: dm-0: devmap already registered
> Aug 24 20:26:15 node03 kernel: device-mapper: multipath: Failing path
> 8:16.
> Aug 24 20:26:15 node03 multipathd: 8:16: mark as failed
> Aug 24 20:26:15 node03 multipathd: qd: remaining active paths: 1
> Aug 24 20:26:15 node03 multipathd: dm-0: add map (uevent)
> Aug 24 20:26:15 node03 multipathd: dm-0: devmap already registered
> Aug 24 20:26:15 node03 kernel: sd 1:0:1:0: queueing MODE_SELECT command.
> 
> 
> So, your configuration fills my logs too :( I don't know if this is
> supposed to behave this way or not...
> 
> I'm lost now :(
> 
> 

Re: Sun StorageTek 2530 and dm?

[Jakov Sosic]

On Mon, 24 Aug 2009 11:38:18 -0700
Chandra Seetharaman <sekharan@us.ibm.com> wrote:

> 
> You need to have a stanza similar to following one for your device
> in /etc/multipath.conf
> --------------------
> devices {
>         device {
>                 vendor                  "IBM"
>                 product                 "1815"
>                 path_grouping_policy    group_by_prio
>                 hardware_handler        "1 rdac"
>                 path_checker            rdac
>                 failback                immediate
>                 no_path_retry           queue
>                 prio_callout            "/sbin/mpath_prio_tpc /dev/%n"
>         }
> }
> ---------------------

Hi!

Yes, I've added that section, and still I get these:

Aug 24 20:47:38 node03 multipathd: 8:16: reinstated 
Aug 24 20:47:38 node03 multipathd: 8:64: reinstated 
Aug 24 20:47:38 node03 multipathd: qd: remaining active paths: 2 
Aug 24 20:47:38 node03 multipathd: qd: switch to path group #2 
Aug 24 20:47:38 node03 multipathd: qd: switch to path group #2 
Aug 24 20:47:38 node03 multipathd: dm-0: add map (uevent) 
Aug 24 20:47:38 node03 multipathd: dm-0: devmap already registered 
Aug 24 20:47:47 node03 kernel: sd 1:0:1:0: queueing MODE_SELECT command.
Aug 24 20:47:48 node03 openais[5236]: [CMAN ] lost contact with quorum
device
Aug 24 20:47:58 node03 multipathd: dm-0: add map (uevent) 
Aug 24 20:47:58 node03 multipathd: dm-0: devmap already registered 
Aug 24 20:47:58 node03 multipathd: 8:64: mark as failed 
Aug 24 20:47:58 node03 kernel: end_request: I/O error, dev sde, sector 8
Aug 24 20:47:58 node03 multipathd: qd: remaining active paths: 1 
Aug 24 20:47:58 node03 kernel: device-mapper: multipath: Failing path
8:64.
Aug 24 20:47:58 node03 kernel: sd 1:0:0:0: queueing MODE_SELECT
command.
Aug 24 20:48:07 node03 multipathd: 8:64: reinstated
Aug 24 20:48:07 node03 multipathd: qd: remaining active paths: 2
Aug 24 20:48:07 node03 multipathd: qd: switch to path group #2
Aug 24 20:48:07 node03 multipathd: qd: switch to path group #2
Aug 24 20:48:07 node03 multipathd: dm-0: add map (uevent)
Aug 24 20:48:07 node03 multipathd: dm-0: devmap already registered
Aug 24 20:48:07 node03 kernel: sd 1:0:1:0: queueing MODE_SELECT command.
Aug 24 20:48:17 node03 multipathd: 8:64: reinstated 




So it does not work ok :(

So far I've tried:

1. defaults with initiators set to "Linux"
   one path constantly failed
2. AVT with initiators set to "Solaris (with Veritas DMP or other)"
   multipath flopping constantly - every 20 minutes there is enter in
   main log about path failing
3. RDAC with initiators set to "Linux"
   logs filled with messages like these show up in the message,
   constantly flip/flopping

I haven't patch scsi_dh_rdac because I don't know how to add support
for "SUN"/"LCSM100_S", and although I could figure it out, I'm not
really a fan of compiling the kernel if it is not absolutely necessary.


-- 
|    Jakov Sosic    |    ICQ: 28410271    |   PGP: 0x965CAE2D   |
=================================================================
| start fighting cancer -> http://www.worldcommunitygrid.org/   |

Re: Sun StorageTek 2530 and dm?

[Jakov Sosic]

On Mon, 24 Aug 2009 11:48:36 -0700
Chandra Seetharaman <sekharan@us.ibm.com> wrote:

> This is the expected behavior, when IOs on the active path fails, the
> hardware handler will activate the passive path and make it active and
> start sending IOs thru that newly activated path.
> 
> Send MODE_SELECT is how the hardware handler makes the passive path
> active. 

Yes, I am aware of the multipathing and switching in case of failure -
but what is bugging me is why is it happening so often? Or can I be
more precise - why is it happening CONSTANTLY? 



-- 
|    Jakov Sosic    |    ICQ: 28410271    |   PGP: 0x965CAE2D   |
=================================================================
| start fighting cancer -> http://www.worldcommunitygrid.org/   |

Re: Sun StorageTek 2530 and dm?

[Chandra Seetharaman]

I do not think adding that patch would not resolve the=problem you are
seeing.

On Mon, 2009-08-24 at 20:55 +0200, Jakov Sosic wrote:
> On Mon, 24 Aug 2009 11:38:18 -0700
> Chandra Seetharaman <sekharan@us.ibm.com> wrote:
> 
> > 
> > You need to have a stanza similar to following one for your device
> > in /etc/multipath.conf
> > --------------------
> > devices {
> >         device {
> >                 vendor                  "IBM"
> >                 product                 "1815"
> >                 path_grouping_policy    group_by_prio
> >                 hardware_handler        "1 rdac"
> >                 path_checker            rdac
> >                 failback                immediate
> >                 no_path_retry           queue
> >                 prio_callout            "/sbin/mpath_prio_tpc /dev/%n"
> >         }
> > }
> > ---------------------
> 
> Hi!
> 
> Yes, I've added that section, and still I get these:
> 
> Aug 24 20:47:38 node03 multipathd: 8:16: reinstated 
> Aug 24 20:47:38 node03 multipathd: 8:64: reinstated 
> Aug 24 20:47:38 node03 multipathd: qd: remaining active paths: 2 
> Aug 24 20:47:38 node03 multipathd: qd: switch to path group #2 
> Aug 24 20:47:38 node03 multipathd: qd: switch to path group #2 
> Aug 24 20:47:38 node03 multipathd: dm-0: add map (uevent) 
> Aug 24 20:47:38 node03 multipathd: dm-0: devmap already registered 
> Aug 24 20:47:47 node03 kernel: sd 1:0:1:0: queueing MODE_SELECT command.
> Aug 24 20:47:48 node03 openais[5236]: [CMAN ] lost contact with quorum
> device
> Aug 24 20:47:58 node03 multipathd: dm-0: add map (uevent) 
> Aug 24 20:47:58 node03 multipathd: dm-0: devmap already registered 
> Aug 24 20:47:58 node03 multipathd: 8:64: mark as failed 
> Aug 24 20:47:58 node03 kernel: end_request: I/O error, dev sde, sector 8
> Aug 24 20:47:58 node03 multipathd: qd: remaining active paths: 1 
> Aug 24 20:47:58 node03 kernel: device-mapper: multipath: Failing path
> 8:64.
> Aug 24 20:47:58 node03 kernel: sd 1:0:0:0: queueing MODE_SELECT
> command.
> Aug 24 20:48:07 node03 multipathd: 8:64: reinstated
> Aug 24 20:48:07 node03 multipathd: qd: remaining active paths: 2
> Aug 24 20:48:07 node03 multipathd: qd: switch to path group #2
> Aug 24 20:48:07 node03 multipathd: qd: switch to path group #2
> Aug 24 20:48:07 node03 multipathd: dm-0: add map (uevent)
> Aug 24 20:48:07 node03 multipathd: dm-0: devmap already registered
> Aug 24 20:48:07 node03 kernel: sd 1:0:1:0: queueing MODE_SELECT command.
> Aug 24 20:48:17 node03 multipathd: 8:64: reinstated 
> 
> 
> 
> 
> So it does not work ok :(
> 
> So far I've tried:
> 
> 1. defaults with initiators set to "Linux"
>    one path constantly failed
> 2. AVT with initiators set to "Solaris (with Veritas DMP or other)"
>    multipath flopping constantly - every 20 minutes there is enter in
>    main log about path failing
> 3. RDAC with initiators set to "Linux"
>    logs filled with messages like these show up in the message,
>    constantly flip/flopping
> 
> I haven't patch scsi_dh_rdac because I don't know how to add support
> for "SUN"/"LCSM100_S", and although I could figure it out, I'm not
> really a fan of compiling the kernel if it is not absolutely necessary.
> 
> 

Re: Sun StorageTek 2530 and dm?

[Chandra Seetharaman]

Have you verified there is no physical issues with the path ?

On Mon, 2009-08-24 at 20:59 +0200, Jakov Sosic wrote:
> On Mon, 24 Aug 2009 11:48:36 -0700
> Chandra Seetharaman <sekharan@us.ibm.com> wrote:
> 
> > This is the expected behavior, when IOs on the active path fails, the
> > hardware handler will activate the passive path and make it active and
> > start sending IOs thru that newly activated path.
> > 
> > Send MODE_SELECT is how the hardware handler makes the passive path
> > active. 
> 
> Yes, I am aware of the multipathing and switching in case of failure -
> but what is bugging me is why is it happening so often? Or can I be
> more precise - why is it happening CONSTANTLY? 
> 
> 
> 

Re: Sun StorageTek 2530 and dm?

[Jakov Sosic]

On Mon, 24 Aug 2009 13:24:57 -0700
Chandra Seetharaman <sekharan@us.ibm.com> wrote:

> Have you verified there is no physical issues with the path ?

Yes, because both paths are flopping, and other LUN's on the same
path's are not flopping as much - but are not written as much. This is
a quorum disk for RHEL cluster that is flopping constantly.



-- 
|    Jakov Sosic    |    ICQ: 28410271    |   PGP: 0x965CAE2D   |
=================================================================
| start fighting cancer -> http://www.worldcommunitygrid.org/   |

Re: Sun StorageTek 2530 and dm?

[Jakov Sosic]

On Mon, 24 Aug 2009 13:24:16 -0700
Chandra Seetharaman <sekharan@us.ibm.com> wrote:

> I do not think adding that patch would not resolve the=problem you are
> seeing.

Too many negations in one sentence - you got me scratching my head on
this one :)

Anyway, I'm building the f*** module :( I'm really pissed of :(



-- 
|    Jakov Sosic    |    ICQ: 28410271    |   PGP: 0x965CAE2D   |
=================================================================
| start fighting cancer -> http://www.worldcommunitygrid.org/   |

Re: Sun StorageTek 2530 and dm?

[Chandra Seetharaman]

On Mon, 2009-08-24 at 22:37 +0200, Jakov Sosic wrote:
> On Mon, 24 Aug 2009 13:24:16 -0700
> Chandra Seetharaman <sekharan@us.ibm.com> wrote:
> 
> > I do not think adding that patch would not resolve the=problem you are
> > seeing.
> 
> Too many negations in one sentence - you got me scratching my head on
> this one :)

ok.. simple words; That patch won't help you.
> 
> Anyway, I'm building the f*** module :( I'm really pissed of :(

This is uncalled for!!

> 
> 
> 

Re: Sun StorageTek 2530 and dm?

[Jakov Sosic]

On Mon, 24 Aug 2009 13:44:34 -0700
Chandra Seetharaman <sekharan@us.ibm.com> wrote:

> ok.. simple words; That patch won't help you.

hm :( so defining "SUN" "LCMS100_S" won't help? Although in
scsi_dh_rdac only "SUN" "LCMS100_F" is defined. And that stands for
Fiber, while mine is SAS cabling...

I've recompiled RPM with added patch, then extracted single module
(scsi_dh_rdac.ko), and put it in place of the original module...

FATAL: Error inserting scsi_dh_rdac
(/lib/modules/2.6.18-128.4.1.el5xen/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko):
Invalid module format

I guess I'll have to build the whole kernel just because of this :(



> This is uncalled for!!

Sorry, I'm easily flammable...




-- 
|    Jakov Sosic    |    ICQ: 28410271    |   PGP: 0x965CAE2D   |
=================================================================
| start fighting cancer -> http://www.worldcommunitygrid.org/   |

Re: Sun StorageTek 2530 and dm?

[Chandra Seetharaman]

On Mon, 2009-08-24 at 22:36 +0200, Jakov Sosic wrote:
> On Mon, 24 Aug 2009 13:24:57 -0700
> Chandra Seetharaman <sekharan@us.ibm.com> wrote:
> 
> > Have you verified there is no physical issues with the path ?
> 
> Yes, because both paths are flopping, and other LUN's on the same
> path's are not flopping as much - but are not written as much. This is
> a quorum disk for RHEL cluster that is flopping constantly.
> 
> Aug 24 20:47:38 node03 multipathd: 8:16: reinstated 
> Aug 24 20:47:38 node03 multipathd: 8:64: reinstated 
> Aug 24 20:47:38 node03 multipathd: qd: remaining active paths: 2 
> Aug 24 20:47:38 node03 multipathd: qd: switch to path group #2 
> Aug 24 20:47:38 node03 multipathd: qd: switch to path group #2 
> Aug 24 20:47:38 node03 multipathd: dm-0: add map (uevent) 
> Aug 24 20:47:38 node03 multipathd: dm-0: devmap already registered 
> Aug 24 20:47:47 node03 kernel: sd 1:0:1:0: queueing MODE_SELECT command.
> Aug 24 20:47:48 node03 openais[5236]: [CMAN ] lost contact with quorum device
> Aug 24 20:47:58 node03 multipathd: dm-0: add map (uevent) 
> Aug 24 20:47:58 node03 multipathd: dm-0: devmap already registered 
> Aug 24 20:47:58 node03 multipathd: 8:64: mark as failed 
> Aug 24 20:47:58 node03 kernel: end_request: I/O error, dev sde, sector 8

This does mean that the IO had really failed in that path....

> Aug 24 20:47:58 node03 multipathd: qd: remaining active paths: 1 
> Aug 24 20:47:58 node03 kernel: device-mapper: multipath: Failing path 8:64.

... based on which the kernel failed the path....

> Aug 24 20:47:58 node03 kernel: sd 1:0:0:0: queueing MODE_SELECT command.
> Aug 24 20:48:07 node03 multipathd: 8:64: reinstated

... only after 10 seconds the path is back.

If you are up for debugging, we can try putting some debug to see if we
are getting any sense information along with the IO failure.

> Aug 24 20:48:07 node03 multipathd: qd: remaining active paths: 2
> Aug 24 20:48:07 node03 multipathd: qd: switch to path group #2
> Aug 24 20:48:07 node03 multipathd: qd: switch to path group #2
> Aug 24 20:48:07 node03 multipathd: dm-0: add map (uevent)
> Aug 24 20:48:07 node03 multipathd: dm-0: devmap already registered
> Aug 24 20:48:07 node03 kernel: sd 1:0:1:0: queueing MODE_SELECT command.
> Aug 24 20:48:17 node03 multipathd: 8:64: reinstated 
> 

Re: Sun StorageTek 2530 and dm?

[Chandra Seetharaman]

On Mon, 2009-08-24 at 22:53 +0200, Jakov Sosic wrote:
> On Mon, 24 Aug 2009 13:44:34 -0700
> Chandra Seetharaman <sekharan@us.ibm.com> wrote:
> 
> > ok.. simple words; That patch won't help you.
> 
> hm :( so defining "SUN" "LCMS100_S" won't help? Although in
> scsi_dh_rdac only "SUN" "LCMS100_F" is defined. And that stands for
> Fiber, while mine is SAS cabling...

Adding this to scsi_dh_rdac makes sure that the module recognizes your
device at boot time itself (provided you have it in your initrd image).
> 
> I've recompiled RPM with added patch, then extracted single module
> (scsi_dh_rdac.ko), and put it in place of the original module...
> 
> FATAL: Error inserting scsi_dh_rdac
> (/lib/modules/2.6.18-128.4.1.el5xen/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko):
> Invalid module format

Hmm, just that change should cause this. BTW, can you provide the patch
you are trying to add.


Can you also provide the (current) o/p of "multipath -ll" and
current /etc/multipath.conf.

> 
> I guess I'll have to build the whole kernel just because of this :(
> 
> 
> 
> > This is uncalled for!!
> 
> Sorry, I'm easily flammable...

but doesn't help the cause :).. lets stay objective here :)
> 
> 
> 
> 

Re: Sun StorageTek 2530 and dm?

[Jakov Sosic]

On Mon, 24 Aug 2009 13:57:31 -0700
Chandra Seetharaman <sekharan@us.ibm.com> wrote:

> If you are up for debugging, we can try putting some debug to see if
> we are getting any sense information along with the IO failure.

OK, here is the dmesg from last reboot, only things that are of
importance to this case:

SCSI subsystem initialized
Fusion MPT base driver 3.04.07
Copyright (c) 1999-2008 LSI Corporation
Fusion MPT SAS Host driver 3.04.07
ACPI: PCI Interrupt 0000:07:00.0[A] -> GSI 17 (level, low) -> IRQ 17
mptbase: ioc0: Initiating bringup

...

mptbase: ioc1: Initiating bringup
ioc1: LSISAS1068E B3: Capabilities={Initiator}
PCI: Setting latency timer of device 0000:09:00.0 to 64
scsi1 : ioc1: LSISAS1068E B3, FwRev=011704dbh, Ports=1, MaxQ=829, IRQ=22
  Vendor: SUN       Model: LCSM100_S         Rev: 0735
  Type:   Direct-Access                      ANSI SCSI revision: 05
SCSI device sdb: 204800 512-byte hdwr sectors (105 MB)
sdb: Write Protect is off
sdb: Mode Sense: 77 00 10 08
SCSI device sdb: drive cache: write back w/ FUA
SCSI device sdb: 204800 512-byte hdwr sectors (105 MB)
sdb: Write Protect is off
sdb: Mode Sense: 77 00 10 08
SCSI device sdb: drive cache: write back w/ FUA
 sdb:end_request: I/O error, dev sdb, sector 0
Buffer I/O error on device sdb, logical block 0
end_request: I/O error, dev sdb, sector 0
Buffer I/O error on device sdb, logical block 0
end_request: I/O error, dev sdb, sector 0
Buffer I/O error on device sdb, logical block 0
end_request: I/O error, dev sdb, sector 0
Buffer I/O error on device sdb, logical block 0
end_request: I/O error, dev sdb, sector 0
Buffer I/O error on device sdb, logical block 0
end_request: I/O error, dev sdb, sector 0
Buffer I/O error on device sdb, logical block 0
end_request: I/O error, dev sdb, sector 0
Buffer I/O error on device sdb, logical block 0
Dev sdb: unable to read RDB block 0
end_request: I/O error, dev sdb, sector 0
Buffer I/O error on device sdb, logical block 0
end_request: I/O error, dev sdb, sector 0
Buffer I/O error on device sdb, logical block 0
 unable to read partition table
sd 1:0:0:0: Attached scsi disk sdb
  Vendor: SUN       Model: LCSM100_S         Rev: 0735
  Type:   Direct-Access                      ANSI SCSI revision: 05
SCSI device sdc: 241172480 512-byte hdwr sectors (123480 MB)
sdc: Write Protect is off
sdc: Mode Sense: 77 00 10 08
SCSI device sdc: drive cache: write back w/ FUA
SCSI device sdc: 241172480 512-byte hdwr sectors (123480 MB)
sdc: Write Protect is off
sdc: Mode Sense: 77 00 10 08
SCSI device sdc: drive cache: write back w/ FUA
 sdc:end_request: I/O error, dev sdc, sector 0
Buffer I/O error on device sdc, logical block 0
end_request: I/O error, dev sdc, sector 0
Buffer I/O error on device sdc, logical block 0
end_request: I/O error, dev sdc, sector 0
end_request: I/O error, dev sdc, sector 0
end_request: I/O error, dev sdc, sector 0
end_request: I/O error, dev sdc, sector 0
end_request: I/O error, dev sdc, sector 0
Dev sdc: unable to read RDB block 0
end_request: I/O error, dev sdc, sector 0
end_request: I/O error, dev sdc, sector 0
 unable to read partition table
sd 1:0:0:1: Attached scsi disk sdc
  Vendor: SUN       Model: LCSM100_S         Rev: 0735
  Type:   Direct-Access                      ANSI SCSI revision: 05
SCSI device sdd: 43274240 512-byte hdwr sectors (22156 MB)
sdd: Write Protect is off
sdd: Mode Sense: 77 00 10 08
SCSI device sdd: drive cache: write back w/ FUA
SCSI device sdd: 43274240 512-byte hdwr sectors (22156 MB)
sdd: Write Protect is off
sdd: Mode Sense: 77 00 10 08
SCSI device sdd: drive cache: write back w/ FUA
 sdd: unknown partition table
sd 1:0:0:2: Attached scsi disk sdd
  Vendor: SUN       Model: LCSM100_S         Rev: 0735
  Type:   Direct-Access                      ANSI SCSI revision: 05
SCSI device sde: 204800 512-byte hdwr sectors (105 MB)
sde: Write Protect is off
sde: Mode Sense: 77 00 10 08
SCSI device sde: drive cache: write back w/ FUA
SCSI device sde: 204800 512-byte hdwr sectors (105 MB)
sde: Write Protect is off
sde: Mode Sense: 77 00 10 08
SCSI device sde: drive cache: write back w/ FUA
 sde:end_request: I/O error, dev sde, sector 0
end_request: I/O error, dev sde, sector 0
printk: 8 messages suppressed.
Buffer I/O error on device sde, logical block 0
end_request: I/O error, dev sde, sector 0
end_request: I/O error, dev sde, sector 0
end_request: I/O error, dev sde, sector 0
end_request: I/O error, dev sde, sector 0
end_request: I/O error, dev sde, sector 0
Dev sde: unable to read RDB block 0
end_request: I/O error, dev sde, sector 0
end_request: I/O error, dev sde, sector 0
 unable to read partition table
sd 1:0:1:0: Attached scsi disk sde
  Vendor: SUN       Model: LCSM100_S         Rev: 0735
  Type:   Direct-Access                      ANSI SCSI revision: 05
SCSI device sdf: 241172480 512-byte hdwr sectors (123480 MB)
sdf: Write Protect is off
sdf: Mode Sense: 77 00 10 08
SCSI device sdf: drive cache: write back w/ FUA
SCSI device sdf: 241172480 512-byte hdwr sectors (123480 MB)
sdf: Write Protect is off
sdf: Mode Sense: 77 00 10 08
SCSI device sdf: drive cache: write back w/ FUA
 sdf: unknown partition table
sd 1:0:1:1: Attached scsi disk sdf
  Vendor: SUN       Model: LCSM100_S         Rev: 0735
  Type:   Direct-Access                      ANSI SCSI revision: 05
SCSI device sdg: 43274240 512-byte hdwr sectors (22156 MB)
sdg: Write Protect is off
sdg: Mode Sense: 77 00 10 08
SCSI device sdg: drive cache: write back w/ FUA
SCSI device sdg: 43274240 512-byte hdwr sectors (22156 MB)
sdg: Write Protect is off
sdg: Mode Sense: 77 00 10 08
SCSI device sdg: drive cache: write back w/ FUA
 sdg:end_request: I/O error, dev sdg, sector 0
end_request: I/O error, dev sdg, sector 0
printk: 8 messages suppressed.
Buffer I/O error on device sdg, logical block 0
end_request: I/O error, dev sdg, sector 0
end_request: I/O error, dev sdg, sector 0
end_request: I/O error, dev sdg, sector 0
end_request: I/O error, dev sdg, sector 0
end_request: I/O error, dev sdg, sector 0
Dev sdg: unable to read RDB block 0
end_request: I/O error, dev sdg, sector 0
end_request: I/O error, dev sdg, sector 0
 unable to read partition table
sd 1:0:1:2: Attached scsi disk sdg
shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
libata version 3.00 loaded.


Now, this seems like some kind of a problem on the LCMS100_S driver
side? MPT_SAS?


Very strange...




-- 
|    Jakov Sosic    |    ICQ: 28410271    |   PGP: 0x965CAE2D   |
=================================================================
| start fighting cancer -> http://www.worldcommunitygrid.org/   |

Re: Sun StorageTek 2530 and dm?

[Jakov Sosic]

On Mon, 24 Aug 2009 13:57:31 -0700
Chandra Seetharaman <sekharan@us.ibm.com> wrote:

Also from the dmesg - if it helps:


device-mapper: table: 253:0: multipath: error getting device
device-mapper: ioctl: error adding target to table
device-mapper: table: 253:0: multipath: error getting device
device-mapper: ioctl: error adding target to table
rdac: device handler registered
device-mapper: multipath: Using scsi_dh module scsi_dh_rdac for
failover/failback and device management.
sd 1:0:1:0: rdac: LUN 0 (owned)
sd 1:0:0:0: rdac: LUN 0 (unowned)
device-mapper: multipath: Using scsi_dh module scsi_dh_rdac for
failover/failback and device management.
sd 1:0:1:1: rdac: LUN 1 (owned)
sd 1:0:0:1: rdac: LUN 1 (unowned)
device-mapper: multipath: Using scsi_dh module scsi_dh_rdac for
failover/failback and device management.
sd 1:0:0:2: rdac: LUN 2 (owned)
sd 1:0:1:2: rdac: LUN 2 (unowned)




-- 
|    Jakov Sosic    |    ICQ: 28410271    |   PGP: 0x965CAE2D   |
=================================================================
| start fighting cancer -> http://www.worldcommunitygrid.org/   |

Re: Sun StorageTek 2530 and dm?

[Jakov Sosic]

On Mon, 24 Aug 2009 14:06:01 -0700
Chandra Seetharaman <sekharan@us.ibm.com> wrote:

> Adding this to scsi_dh_rdac makes sure that the module recognizes your
> device at boot time itself (provided you have it in your initrd
> image).

Whole kernel rpm didn't help, my patch is obviously failing
somewhere, so I'm stuck with that solution.

Should scsi_dh_rdac be in initrd image?


-- 
|    Jakov Sosic    |    ICQ: 28410271    |   PGP: 0x965CAE2D   |
=================================================================
| start fighting cancer -> http://www.worldcommunitygrid.org/   |

Re: Sun StorageTek 2530 and dm?

[Chandra Seetharaman]

These messages are seen because there is no support for this device
in-built in the scsi_dh_rdac module.

These will disappear when youadded the device support to scsi_dh_rdac
module and include the module in your initrd image.
On Mon, 2009-08-24 at 23:14 +0200, Jakov Sosic wrote:
> On Mon, 24 Aug 2009 13:57:31 -0700
> Chandra Seetharaman <sekharan@us.ibm.com> wrote:
> 
> > If you are up for debugging, we can try putting some debug to see if
> > we are getting any sense information along with the IO failure.
> 
> OK, here is the dmesg from last reboot, only things that are of
> importance to this case:
> 
> SCSI subsystem initialized
> Fusion MPT base driver 3.04.07
> Copyright (c) 1999-2008 LSI Corporation
> Fusion MPT SAS Host driver 3.04.07
> ACPI: PCI Interrupt 0000:07:00.0[A] -> GSI 17 (level, low) -> IRQ 17
> mptbase: ioc0: Initiating bringup
> 
> ...
> 
> mptbase: ioc1: Initiating bringup
> ioc1: LSISAS1068E B3: Capabilities={Initiator}
> PCI: Setting latency timer of device 0000:09:00.0 to 64
> scsi1 : ioc1: LSISAS1068E B3, FwRev=011704dbh, Ports=1, MaxQ=829, IRQ=22
>   Vendor: SUN       Model: LCSM100_S         Rev: 0735
>   Type:   Direct-Access                      ANSI SCSI revision: 05
> SCSI device sdb: 204800 512-byte hdwr sectors (105 MB)
> sdb: Write Protect is off
> sdb: Mode Sense: 77 00 10 08
> SCSI device sdb: drive cache: write back w/ FUA
> SCSI device sdb: 204800 512-byte hdwr sectors (105 MB)
> sdb: Write Protect is off
> sdb: Mode Sense: 77 00 10 08
> SCSI device sdb: drive cache: write back w/ FUA
>  sdb:end_request: I/O error, dev sdb, sector 0
> Buffer I/O error on device sdb, logical block 0
> end_request: I/O error, dev sdb, sector 0
> Buffer I/O error on device sdb, logical block 0
> end_request: I/O error, dev sdb, sector 0
> Buffer I/O error on device sdb, logical block 0
> end_request: I/O error, dev sdb, sector 0
> Buffer I/O error on device sdb, logical block 0
> end_request: I/O error, dev sdb, sector 0
> Buffer I/O error on device sdb, logical block 0
> end_request: I/O error, dev sdb, sector 0
> Buffer I/O error on device sdb, logical block 0
> end_request: I/O error, dev sdb, sector 0
> Buffer I/O error on device sdb, logical block 0
> Dev sdb: unable to read RDB block 0
> end_request: I/O error, dev sdb, sector 0
> Buffer I/O error on device sdb, logical block 0
> end_request: I/O error, dev sdb, sector 0
> Buffer I/O error on device sdb, logical block 0
>  unable to read partition table
> sd 1:0:0:0: Attached scsi disk sdb
>   Vendor: SUN       Model: LCSM100_S         Rev: 0735
>   Type:   Direct-Access                      ANSI SCSI revision: 05
> SCSI device sdc: 241172480 512-byte hdwr sectors (123480 MB)
> sdc: Write Protect is off
> sdc: Mode Sense: 77 00 10 08
> SCSI device sdc: drive cache: write back w/ FUA
> SCSI device sdc: 241172480 512-byte hdwr sectors (123480 MB)
> sdc: Write Protect is off
> sdc: Mode Sense: 77 00 10 08
> SCSI device sdc: drive cache: write back w/ FUA
>  sdc:end_request: I/O error, dev sdc, sector 0
> Buffer I/O error on device sdc, logical block 0
> end_request: I/O error, dev sdc, sector 0
> Buffer I/O error on device sdc, logical block 0
> end_request: I/O error, dev sdc, sector 0
> end_request: I/O error, dev sdc, sector 0
> end_request: I/O error, dev sdc, sector 0
> end_request: I/O error, dev sdc, sector 0
> end_request: I/O error, dev sdc, sector 0
> Dev sdc: unable to read RDB block 0
> end_request: I/O error, dev sdc, sector 0
> end_request: I/O error, dev sdc, sector 0
>  unable to read partition table
> sd 1:0:0:1: Attached scsi disk sdc
>   Vendor: SUN       Model: LCSM100_S         Rev: 0735
>   Type:   Direct-Access                      ANSI SCSI revision: 05
> SCSI device sdd: 43274240 512-byte hdwr sectors (22156 MB)
> sdd: Write Protect is off
> sdd: Mode Sense: 77 00 10 08
> SCSI device sdd: drive cache: write back w/ FUA
> SCSI device sdd: 43274240 512-byte hdwr sectors (22156 MB)
> sdd: Write Protect is off
> sdd: Mode Sense: 77 00 10 08
> SCSI device sdd: drive cache: write back w/ FUA
>  sdd: unknown partition table
> sd 1:0:0:2: Attached scsi disk sdd
>   Vendor: SUN       Model: LCSM100_S         Rev: 0735
>   Type:   Direct-Access                      ANSI SCSI revision: 05
> SCSI device sde: 204800 512-byte hdwr sectors (105 MB)
> sde: Write Protect is off
> sde: Mode Sense: 77 00 10 08
> SCSI device sde: drive cache: write back w/ FUA
> SCSI device sde: 204800 512-byte hdwr sectors (105 MB)
> sde: Write Protect is off
> sde: Mode Sense: 77 00 10 08
> SCSI device sde: drive cache: write back w/ FUA
>  sde:end_request: I/O error, dev sde, sector 0
> end_request: I/O error, dev sde, sector 0
> printk: 8 messages suppressed.
> Buffer I/O error on device sde, logical block 0
> end_request: I/O error, dev sde, sector 0
> end_request: I/O error, dev sde, sector 0
> end_request: I/O error, dev sde, sector 0
> end_request: I/O error, dev sde, sector 0
> end_request: I/O error, dev sde, sector 0
> Dev sde: unable to read RDB block 0
> end_request: I/O error, dev sde, sector 0
> end_request: I/O error, dev sde, sector 0
>  unable to read partition table
> sd 1:0:1:0: Attached scsi disk sde
>   Vendor: SUN       Model: LCSM100_S         Rev: 0735
>   Type:   Direct-Access                      ANSI SCSI revision: 05
> SCSI device sdf: 241172480 512-byte hdwr sectors (123480 MB)
> sdf: Write Protect is off
> sdf: Mode Sense: 77 00 10 08
> SCSI device sdf: drive cache: write back w/ FUA
> SCSI device sdf: 241172480 512-byte hdwr sectors (123480 MB)
> sdf: Write Protect is off
> sdf: Mode Sense: 77 00 10 08
> SCSI device sdf: drive cache: write back w/ FUA
>  sdf: unknown partition table
> sd 1:0:1:1: Attached scsi disk sdf
>   Vendor: SUN       Model: LCSM100_S         Rev: 0735
>   Type:   Direct-Access                      ANSI SCSI revision: 05
> SCSI device sdg: 43274240 512-byte hdwr sectors (22156 MB)
> sdg: Write Protect is off
> sdg: Mode Sense: 77 00 10 08
> SCSI device sdg: drive cache: write back w/ FUA
> SCSI device sdg: 43274240 512-byte hdwr sectors (22156 MB)
> sdg: Write Protect is off
> sdg: Mode Sense: 77 00 10 08
> SCSI device sdg: drive cache: write back w/ FUA
>  sdg:end_request: I/O error, dev sdg, sector 0
> end_request: I/O error, dev sdg, sector 0
> printk: 8 messages suppressed.
> Buffer I/O error on device sdg, logical block 0
> end_request: I/O error, dev sdg, sector 0
> end_request: I/O error, dev sdg, sector 0
> end_request: I/O error, dev sdg, sector 0
> end_request: I/O error, dev sdg, sector 0
> end_request: I/O error, dev sdg, sector 0
> Dev sdg: unable to read RDB block 0
> end_request: I/O error, dev sdg, sector 0
> end_request: I/O error, dev sdg, sector 0
>  unable to read partition table
> sd 1:0:1:2: Attached scsi disk sdg
> shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
> libata version 3.00 loaded.
> 
> 
> Now, this seems like some kind of a problem on the LCMS100_S driver
> side? MPT_SAS?
> 
> 
> Very strange...
> 
> 
> 
> 

Re: Sun StorageTek 2530 and dm?

[Chandra Seetharaman]

On Mon, 2009-08-24 at 23:19 +0200, Jakov Sosic wrote:
> On Mon, 24 Aug 2009 13:57:31 -0700
> Chandra Seetharaman <sekharan@us.ibm.com> wrote:
> 
> Also from the dmesg - if it helps:
> 
> 
> device-mapper: table: 253:0: multipath: error getting device
> device-mapper: ioctl: error adding target to table
> device-mapper: table: 253:0: multipath: error getting device
> device-mapper: ioctl: error adding target to table

Above message is most likely from your root disk (it is a scsi disk,
right ?).

You can avoid getting this by adding a blacklist in your multipath.conf
file.
> rdac: device handler registered
> device-mapper: multipath: Using scsi_dh module scsi_dh_rdac for
> failover/failback and device management.
> sd 1:0:1:0: rdac: LUN 0 (owned)
> sd 1:0:0:0: rdac: LUN 0 (unowned)
> device-mapper: multipath: Using scsi_dh module scsi_dh_rdac for
> failover/failback and device management.
> sd 1:0:1:1: rdac: LUN 1 (owned)
> sd 1:0:0:1: rdac: LUN 1 (unowned)
> device-mapper: multipath: Using scsi_dh module scsi_dh_rdac for
> failover/failback and device management.
> sd 1:0:0:2: rdac: LUN 2 (owned)
> sd 1:0:1:2: rdac: LUN 2 (unowned)

These are good.
> 
> 
> 
> 

Re: Sun StorageTek 2530 and dm?

[Jakov Sosic]

On Mon, 24 Aug 2009 14:41:56 -0700
Chandra Seetharaman <sekharan@us.ibm.com> wrote:

> These messages are seen because there is no support for this device
> in-built in the scsi_dh_rdac module.
> 
> These will disappear when youadded the device support to scsi_dh_rdac
> module and include the module in your initrd image.

Now if I understand correctly - because my device is not
supported by scsi_dh_rdac - RDAC is not functional at all? Because
later in the dmesg, it shows 

sd 1:0:1:0: rdac: LUN 0 (owned)
sd 1:0:0:0: rdac: LUN 0 (unowned)

which suggests that RDAC is working after all? 






Here is my current multipath.conf (Sbeaf_11 and Sbeaf_21 are iSCSI
targets that work flawlessly):

defaults {
	user_friendly_names yes
}
defaults {
	udev_dir		/dev
	polling_interval 	10
	selector		"round-robin 0"
	path_grouping_policy	multibus
	getuid_callout		"/sbin/scsi_id -g -u -s /block/%n"
	prio_callout		/bin/true
	path_checker		readsector0
	rr_min_io		100
	max_fds			8192
	rr_weight		priorities
	failback		immediate
	no_path_retry		fail
	user_friendly_names	yes
}
blacklist {
	devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
	devnode "^hd[a-z]"
}
multipaths {
	multipath {
		wwid			S_beaf11
		alias			controller0
		path_grouping_policy	failover
		path_checker		readsector0
		path_selector		"round-robin 0"
		failback		immediate
		no_path_retry		queue
	}
	multipath {
		wwid			S_beaf21
		alias			controller1
		path_grouping_policy	failover
		path_checker		readsector0
		path_selector		"round-robin 0"
		failback		immediate
		no_path_retry		queue
	}
	multipath {
		wwid
3600a0b80002fcd18000014544a9166fb
		alias			sas-xen
	}
	multipath {
		wwid
3600a0b80003abc5c00000b654a916617
		alias			sas-mysql
	}
	multipath {
		wwid
3600a0b80002fcd18000014514a9165b4
		alias			qd
	}
}


devices {
	device {
		vendor			"SUN"
		product			"LCSM100_S"
		getuid_callout		"/sbin/scsi_id -g -u
-s /block/%n"
		prio_callout            "/sbin/mpath_prio_tpc /dev/%n"
		hardware_handler	"1 rdac"
		path_grouping_policy	group_by_prio
		path_checker		rdac
		failback		immediate
		rr_weight		uniform
		no_path_retry		queue
		rr_min_io		1000
	}
}




And here is the multipath -ll:

# multipath -ll
qd (3600a0b80002fcd18000014514a9165b4) dm-0 SUN,LCSM100_S
[size=100M][features=1 queue_if_no_path][hwhandler=1 rdac][rw]
\_ round-robin 0 [prio=3][active]
 \_ 1:0:1:0  sde 8:64  [active][ready]
\_ round-robin 0 [prio=0][enabled]
 \_ 1:0:0:0  sdb 8:16  [active][ghost]
controller1 (S_beaf21) dm-4 IET,VIRTUAL-DISK
[size=3.4T][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=0][active]
 \_ 8:0:0:1  sdk 8:160 [active][ready]
\_ round-robin 0 [prio=0][enabled]
 \_ 9:0:0:1  sdj 8:144 [active][ready]
controller0 (S_beaf11) dm-3 IET,VIRTUAL-DISK
[size=3.4T][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=0][active]
 \_ 10:0:0:1 sdh 8:112 [active][ready]
\_ round-robin 0 [prio=0][enabled]
 \_ 11:0:0:1 sdi 8:128 [active][ready]
sas-mysql (3600a0b80003abc5c00000b654a916617) dm-1 SUN,LCSM100_S
[size=115G][features=1 queue_if_no_path][hwhandler=1 rdac][rw]
\_ round-robin 0 [prio=3][active]
 \_ 1:0:1:1  sdf 8:80  [active][ready]
\_ round-robin 0 [prio=0][enabled]
 \_ 1:0:0:1  sdc 8:32  [active][ghost]
sas-xen (3600a0b80002fcd18000014544a9166fb) dm-2 SUN,LCSM100_S
[size=21G][features=1 queue_if_no_path][hwhandler=1 rdac][rw]
\_ round-robin 0 [prio=3][active]
 \_ 1:0:0:2  sdd 8:48  [active][ready]
\_ round-robin 0 [prio=0][enabled]
 \_ 1:0:1:2  sdg 8:96  [active][ghost]







-- 
|    Jakov Sosic    |    ICQ: 28410271    |   PGP: 0x965CAE2D   |
=================================================================
| start fighting cancer -> http://www.worldcommunitygrid.org/   |

Re: Sun StorageTek 2530 and dm?

[Chandra Seetharaman]

On Mon, 2009-08-24 at 23:37 +0200, Jakov Sosic wrote:
> On Mon, 24 Aug 2009 14:06:01 -0700
> Chandra Seetharaman <sekharan@us.ibm.com> wrote:
> 
> > Adding this to scsi_dh_rdac makes sure that the module recognizes your
> > device at boot time itself (provided you have it in your initrd
> > image).
> 
> Whole kernel rpm didn't help, my patch is obviously failing
> somewhere, so I'm stuck with that solution.

What is the problem you are seeing ?

> 
> Should scsi_dh_rdac be in initrd image?

You need it in initrd image only to get rid of the boottime messages. It
is useful only after you add your device to the rdac's devlist.
> 
> 

Re: Sun StorageTek 2530 and dm?

[Charlie Brady]

On Mon, 24 Aug 2009, Jakov Sosic wrote:

> On Mon, 24 Aug 2009 13:44:34 -0700
> Chandra Seetharaman <sekharan@us.ibm.com> wrote:
>
>> ok.. simple words; That patch won't help you.
>
> hm :( so defining "SUN" "LCMS100_S" won't help? Although in
> scsi_dh_rdac only "SUN" "LCMS100_F" is defined. And that stands for
> Fiber, while mine is SAS cabling...

Chandra says that the patch won't help you. Perhaps he can clarify. My 
understanding is that if you configure multipath to use RDAC that the 
driver does have to be patched so that the RDAC handler will be used. I 
did find that to be the case with the StorageTek 2510 "LCMS100_I").

My understanding also is that RDAC is superior, so RDAC hander support is 
desirable.

> I've recompiled RPM with added patch, then extracted single module
> (scsi_dh_rdac.ko), and put it in place of the original module...
>
> FATAL: Error inserting scsi_dh_rdac
> (/lib/modules/2.6.18-128.4.1.el5xen/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko):
> Invalid module format
>
> I guess I'll have to build the whole kernel just because of this :(

No, it is certainly possible to build an RPM which contains just the 
module that you need. But you do need to build correctly.

I did make the suggestion earlier that it would be useful if the module 
could take load-time parameters to add support for additional devices. I 
think it was Chandra who agreed that was a good idea and he would add it 
to the TODO list, but neither he nor I has done that yet.

>> This is uncalled for!!
>
> Sorry, I'm easily flammable...

You would be wise to control your urges, if you wish people to help you.

Re: Sun StorageTek 2530 and dm?

[Chandra Seetharaman]

On Mon, 2009-08-24 at 23:53 +0200, Jakov Sosic wrote:
> On Mon, 24 Aug 2009 14:41:56 -0700
> Chandra Seetharaman <sekharan@us.ibm.com> wrote:
> 
> > These messages are seen because there is no support for this device
> > in-built in the scsi_dh_rdac module.
> > 
> > These will disappear when youadded the device support to scsi_dh_rdac
> > module and include the module in your initrd image.
> 
> Now if I understand correctly - because my device is not
> supported by scsi_dh_rdac - RDAC is not functional at all? Because

If your device was in the list, then you wouldn't seen multiples of
those error messages and would have seen the following message at boot
time itself.

> later in the dmesg, it shows 
> 
> sd 1:0:1:0: rdac: LUN 0 (owned)
> sd 1:0:0:0: rdac: LUN 0 (unowned)

scsi_dh_rdac modules is insmodded by multipath when multipathd is
started (by virtue of you having hardware_handler "1 rdac" in
your /etc/muliutpath.conf file.

So, it is effective when multipath starts.
> 
> which suggests that RDAC is working after all? 

> 
> 
> 
> 
> 
> Here is my current multipath.conf (Sbeaf_11 and Sbeaf_21 are iSCSI
> targets that work flawlessly):
> 
> defaults {
> 	user_friendly_names yes
> }
> defaults {
> 	udev_dir		/dev
> 	polling_interval 	10
> 	selector		"round-robin 0"
> 	path_grouping_policy	multibus
> 	getuid_callout		"/sbin/scsi_id -g -u -s /block/%n"
> 	prio_callout		/bin/true
> 	path_checker		readsector0
> 	rr_min_io		100
> 	max_fds			8192
> 	rr_weight		priorities
> 	failback		immediate
> 	no_path_retry		fail
> 	user_friendly_names	yes
> }
> blacklist {
> 	devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
> 	devnode "^hd[a-z]"
> }
> multipaths {
> 	multipath {
> 		wwid			S_beaf11
> 		alias			controller0
> 		path_grouping_policy	failover
> 		path_checker		readsector0
> 		path_selector		"round-robin 0"
> 		failback		immediate
> 		no_path_retry		queue
> 	}
> 	multipath {
> 		wwid			S_beaf21
> 		alias			controller1
> 		path_grouping_policy	failover
> 		path_checker		readsector0
> 		path_selector		"round-robin 0"
> 		failback		immediate
> 		no_path_retry		queue
> 	}
> 	multipath {
> 		wwid
> 3600a0b80002fcd18000014544a9166fb
> 		alias			sas-xen
> 	}
> 	multipath {
> 		wwid
> 3600a0b80003abc5c00000b654a916617
> 		alias			sas-mysql
> 	}
> 	multipath {
> 		wwid
> 3600a0b80002fcd18000014514a9165b4
> 		alias			qd
> 	}
> }
> 
> 
> devices {
> 	device {
> 		vendor			"SUN"
> 		product			"LCSM100_S"
> 		getuid_callout		"/sbin/scsi_id -g -u
> -s /block/%n"
> 		prio_callout            "/sbin/mpath_prio_tpc /dev/%n"
> 		hardware_handler	"1 rdac"
> 		path_grouping_policy	group_by_prio
> 		path_checker		rdac
> 		failback		immediate
> 		rr_weight		uniform
> 		no_path_retry		queue
> 		rr_min_io		1000
> 	}
> }
> 
> 
> 
> 
> And here is the multipath -ll:
> 
> # multipath -ll
> qd (3600a0b80002fcd18000014514a9165b4) dm-0 SUN,LCSM100_S
> [size=100M][features=1 queue_if_no_path][hwhandler=1 rdac][rw]
> \_ round-robin 0 [prio=3][active]
>  \_ 1:0:1:0  sde 8:64  [active][ready]
> \_ round-robin 0 [prio=0][enabled]
>  \_ 1:0:0:0  sdb 8:16  [active][ghost]
> controller1 (S_beaf21) dm-4 IET,VIRTUAL-DISK
> [size=3.4T][features=1 queue_if_no_path][hwhandler=0][rw]
> \_ round-robin 0 [prio=0][active]
>  \_ 8:0:0:1  sdk 8:160 [active][ready]
> \_ round-robin 0 [prio=0][enabled]
>  \_ 9:0:0:1  sdj 8:144 [active][ready]
> controller0 (S_beaf11) dm-3 IET,VIRTUAL-DISK
> [size=3.4T][features=1 queue_if_no_path][hwhandler=0][rw]
> \_ round-robin 0 [prio=0][active]
>  \_ 10:0:0:1 sdh 8:112 [active][ready]
> \_ round-robin 0 [prio=0][enabled]
>  \_ 11:0:0:1 sdi 8:128 [active][ready]
> sas-mysql (3600a0b80003abc5c00000b654a916617) dm-1 SUN,LCSM100_S
> [size=115G][features=1 queue_if_no_path][hwhandler=1 rdac][rw]
> \_ round-robin 0 [prio=3][active]
>  \_ 1:0:1:1  sdf 8:80  [active][ready]
> \_ round-robin 0 [prio=0][enabled]
>  \_ 1:0:0:1  sdc 8:32  [active][ghost]
> sas-xen (3600a0b80002fcd18000014544a9166fb) dm-2 SUN,LCSM100_S
> [size=21G][features=1 queue_if_no_path][hwhandler=1 rdac][rw]
> \_ round-robin 0 [prio=3][active]
>  \_ 1:0:0:2  sdd 8:48  [active][ready]
> \_ round-robin 0 [prio=0][enabled]
>  \_ 1:0:1:2  sdg 8:96  [active][ghost]
> 
> 
> 
> 
> 
> 
> 

Re: Sun StorageTek 2530 and dm?

[Jakov Sosic]

On Mon, 24 Aug 2009 18:58:24 -0400 (EDT)
Charlie Brady <charlieb-dm-devel@budge.apana.org.au> wrote:

> Chandra says that the patch won't help you. Perhaps he can clarify.
> My understanding is that if you configure multipath to use RDAC that
> the driver does have to be patched so that the RDAC handler will be
> used. I did find that to be the case with the StorageTek 2510
> "LCMS100_I").
> 
> My understanding also is that RDAC is superior, so RDAC hander
> support is desirable.

Is it enough to just add a new line to the list of the devices? For
example:

const struct scsi_dh_devlist rdac_dev_list[] = {
        {"IBM", "1722"},
        {"IBM", "1724"},
        {"IBM", "1726"},
        {"IBM", "1742"},
        {"IBM", "1814"},
        {"IBM", "1815"},
        {"IBM", "1818"},
        {"IBM", "3526"},
        {"SGI", "TP9400"},
        {"SGI", "TP9500"},
        {"SGI", "IS"},
        {"STK", "OPENstorage D280"},
        {"SUN", "CSM200_R"},
        {"SUN", "LCSM100_F"},
        {"SUN", "LCSM100_I"},
        {"SUN", "LCSM100_S"},
        {NULL, NULL},
};

?


> > Sorry, I'm easily flammable...
> 
> You would be wise to control your urges, if you wish people to help
> you.

Well you misunderstood me both - I was just bashing around, and nothing
was said at neither yours neither his account. It was just because this
project I'm working on is taking so so so long :( This is like 10th
problem I've encountered that is out of my order, and the amount of
user-hours has already 3-4x exceeded the amount of money project
brings :( So I'm really really... angry :-/


-- 
|    Jakov Sosic    |    ICQ: 28410271    |   PGP: 0x965CAE2D   |
=================================================================
| start fighting cancer -> http://www.worldcommunitygrid.org/   |

Re: Sun StorageTek 2530 and dm?

[Chandra Seetharaman]

On Mon, 2009-08-24 at 18:58 -0400, Charlie Brady wrote:
> On Mon, 24 Aug 2009, Jakov Sosic wrote:
> 
> > On Mon, 24 Aug 2009 13:44:34 -0700
> > Chandra Seetharaman <sekharan@us.ibm.com> wrote:
> >
> >> ok.. simple words; That patch won't help you.
> >
> > hm :( so defining "SUN" "LCMS100_S" won't help? Although in
> > scsi_dh_rdac only "SUN" "LCMS100_F" is defined. And that stands for
> > Fiber, while mine is SAS cabling...
> 
> Chandra says that the patch won't help you. Perhaps he can clarify. My 
> understanding is that if you configure multipath to use RDAC that the 
> driver does have to be patched so that the RDAC handler will be used. I 
> did find that to be the case with the StorageTek 2510 "LCMS100_I").

I didn't mean that scsi_dh_rdac module won't help him. It would.

His device is being handled by the scsi_dh_rdac module once he added the
stanza to /etc/multipath.conf file.

But, the problem he is seeing is different and adding this device to the
devlist in scsi_dh_rdac wouldn't solve his problem.

Hope it is clear now.

> 
> My understanding also is that RDAC is superior, so RDAC hander support is 
> desirable.
> 
> > I've recompiled RPM with added patch, then extracted single module
> > (scsi_dh_rdac.ko), and put it in place of the original module...
> >
> > FATAL: Error inserting scsi_dh_rdac
> > (/lib/modules/2.6.18-128.4.1.el5xen/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko):
> > Invalid module format
> >
> > I guess I'll have to build the whole kernel just because of this :(
> 
> No, it is certainly possible to build an RPM which contains just the 
> module that you need. But you do need to build correctly.
> 
> I did make the suggestion earlier that it would be useful if the module 
> could take load-time parameters to add support for additional devices. I 
> think it was Chandra who agreed that was a good idea and he would add it 
> to the TODO list, but neither he nor I has done that yet.
> 
> >> This is uncalled for!!
> >
> > Sorry, I'm easily flammable...
> 
> You would be wise to control your urges, if you wish people to help you.
> 
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel

Re: Sun StorageTek 2530 and dm?

[Jakov Sosic]

On Mon, 24 Aug 2009 18:58:24 -0400 (EDT)
Charlie Brady <charlieb-dm-devel@budge.apana.org.au> wrote:

> My understanding also is that RDAC is superior, so RDAC hander
> support is desirable.

PS I don't mind if it is called 'G%#SGA' - as long as it works... I
switched to RDAC because apparently AVT is giving me headaches
too :( It looses paths every 20 minutes, and with RDAC things are even
more problematic so far.



-- 
|    Jakov Sosic    |    ICQ: 28410271    |   PGP: 0x965CAE2D   |
=================================================================
| start fighting cancer -> http://www.worldcommunitygrid.org/   |

Re: Sun StorageTek 2530 and dm?

[Jakov Sosic]

On Mon, 24 Aug 2009 16:07:22 -0700
Chandra Seetharaman <sekharan@us.ibm.com> wrote:

> scsi_dh_rdac modules is insmodded by multipath when multipathd is
> started (by virtue of you having hardware_handler "1 rdac" in
> your /etc/muliutpath.conf file.
> 
> So, it is effective when multipath starts.

If it is effective, I don't need to patch it... Thank you very much.

You were saying that you could provide some debugging code if I'm
interested? Well it seems that I don't have a choice... So please do
so :)




-- 
|    Jakov Sosic    |    ICQ: 28410271    |   PGP: 0x965CAE2D   |
=================================================================
| start fighting cancer -> http://www.worldcommunitygrid.org/   |

Re: Sun StorageTek 2530 and dm?

[Chandra Seetharaman]

On Tue, 2009-08-25 at 01:09 +0200, Jakov Sosic wrote:
> On Mon, 24 Aug 2009 18:58:24 -0400 (EDT)
> Charlie Brady <charlieb-dm-devel@budge.apana.org.au> wrote:
> 
> > Chandra says that the patch won't help you. Perhaps he can clarify.
> > My understanding is that if you configure multipath to use RDAC that
> > the driver does have to be patched so that the RDAC handler will be
> > used. I did find that to be the case with the StorageTek 2510
> > "LCMS100_I").
> > 
> > My understanding also is that RDAC is superior, so RDAC hander
> > support is desirable.
> 
> Is it enough to just add a new line to the list of the devices? For
> example:

yup. But, I mentioned earlier, this _won't_ solve the problem you are
seeing.

> 
> const struct scsi_dh_devlist rdac_dev_list[] = {
>         {"IBM", "1722"},
>         {"IBM", "1724"},
>         {"IBM", "1726"},
>         {"IBM", "1742"},
>         {"IBM", "1814"},
>         {"IBM", "1815"},
>         {"IBM", "1818"},
>         {"IBM", "3526"},
>         {"SGI", "TP9400"},
>         {"SGI", "TP9500"},
>         {"SGI", "IS"},
>         {"STK", "OPENstorage D280"},
>         {"SUN", "CSM200_R"},
>         {"SUN", "LCSM100_F"},
>         {"SUN", "LCSM100_I"},
>         {"SUN", "LCSM100_S"},
>         {NULL, NULL},
> };
> 
> ?
> 
> 
> > > Sorry, I'm easily flammable...
> > 
> > You would be wise to control your urges, if you wish people to help
> > you.
> 
> Well you misunderstood me both - I was just bashing around, and nothing
> was said at neither yours neither his account. It was just because this
> project I'm working on is taking so so so long :( This is like 10th
> problem I've encountered that is out of my order, and the amount of
> user-hours has already 3-4x exceeded the amount of money project
> brings :( So I'm really really... angry :-/
> 
> 

Re: Sun StorageTek 2530 and dm?

[Jakov Sosic]

On Mon, 24 Aug 2009 16:18:15 -0700
Chandra Seetharaman <sekharan@us.ibm.com> wrote:

> yup. But, I mentioned earlier, this _won't_ solve the problem you are
> seeing.

Thank you... I'll try other things then.

The interesting part is, while I was connected only to one controller,
and didn't have the multipath, there wasn't any dropouts of that single
path.

Now, with multipath, being AVT or RDAC, it works much worse... I just
don't get it, why all those flopping is going around :(

And I waited a month to get new SAS cables shipped in - so that a
SAS controler would't be a SPOF in my RHCS installation :)


-- 
|    Jakov Sosic    |    ICQ: 28410271    |   PGP: 0x965CAE2D   |
=================================================================
| start fighting cancer -> http://www.worldcommunitygrid.org/   |

Re: Sun StorageTek 2530 and dm?

[Jakov Sosic]

On Tue, 25 Aug 2009 01:13:40 +0200
Jakov Sosic <jakov.sosic@srce.hr> wrote:

> You were saying that you could provide some debugging code if I'm
> interested? Well it seems that I don't have a choice... So please do
> so :)

Also, I've found that Sun provides RDAC driver for 2.6 kernels... I
brings whole framework, with config files and additional commands...

I don't like the idea, but because neither the AVT neither the RDAC on
dm-multipath are working correctly, I'll try the Sun RDAC solution
tommorow and post the results.

Here is the explanation (and link to source is within the document):
http://dlc.sun.com/pdf/820-4738-12/820-4738-12.pdf




-- 
|    Jakov Sosic    |    ICQ: 28410271    |   PGP: 0x965CAE2D   |
=================================================================
| start fighting cancer -> http://www.worldcommunitygrid.org/   |

Re: Sun StorageTek 2530 and dm?

[Chandra Seetharaman]

On Tue, 2009-08-25 at 01:13 +0200, Jakov Sosic wrote:
> On Mon, 24 Aug 2009 16:07:22 -0700
> Chandra Seetharaman <sekharan@us.ibm.com> wrote:
> 
> > scsi_dh_rdac modules is insmodded by multipath when multipathd is
> > started (by virtue of you having hardware_handler "1 rdac" in
> > your /etc/muliutpath.conf file.
> > 
> > So, it is effective when multipath starts.
> 
> If it is effective, I don't need to patch it... Thank you very much.
> 
> You were saying that you could provide some debugging code if I'm
> interested? Well it seems that I don't have a choice... So please do
> so :)
> 
Basically I want to find out if we are getting and sense codes that we
are not handling. To that effect, andd the following line at the top of
the function rdac_check_sense()
--------
sdev_printk(KERN_ERR, sdev, "I/O returned with sense %02x/%02x/%02x",
                      sense_hdr->sense_key, sense_hdr->asc, sense_hdr->ascq);
---------

compile the module, run your tests and get me /var/log/messages.
> 
> 

Re: Sun StorageTek 2530 and dm?

[Charlie Brady]

On Mon, 24 Aug 2009, Chandra Seetharaman wrote:

> On Tue, 2009-08-25 at 01:13 +0200, Jakov Sosic wrote:
>> On Mon, 24 Aug 2009 16:07:22 -0700
>> Chandra Seetharaman <sekharan@us.ibm.com> wrote:
>>
>>> scsi_dh_rdac modules is insmodded by multipath when multipathd is
>>> started (by virtue of you having hardware_handler "1 rdac" in
>>> your /etc/muliutpath.conf file.
>>>
>>> So, it is effective when multipath starts.
>>
>> If it is effective, I don't need to patch it... Thank you very much.
>>
>> You were saying that you could provide some debugging code if I'm
>> interested? Well it seems that I don't have a choice... So please do
>> so :)
>>
> Basically I want to find out if we are getting and sense codes that we
> are not handling. To that effect, andd the following line at the top of
> the function rdac_check_sense()
> --------
> sdev_printk(KERN_ERR, sdev, "I/O returned with sense %02x/%02x/%02x",
>                      sense_hdr->sense_key, sense_hdr->asc, sense_hdr->ascq);
> ---------
>
> compile the module, run your tests and get me /var/log/messages.

Might this be the same issue as you worked through with Babu last October?

http://kerneltrap.org/mailarchive/linux-scsi/2008/10/24/3803244

Re: Sun StorageTek 2530 and dm?

[Chandra Seetharaman]

Hopefully not :)... it has been fixed for a while.

Jacov, Can you send me the scsi_dh_rdac.c file. I want to make sure you
are running the latest code.

On Mon, 2009-08-24 at 23:36 -0400, Charlie Brady wrote:
> On Mon, 24 Aug 2009, Chandra Seetharaman wrote:
> 
> > On Tue, 2009-08-25 at 01:13 +0200, Jakov Sosic wrote:
> >> On Mon, 24 Aug 2009 16:07:22 -0700
> >> Chandra Seetharaman <sekharan@us.ibm.com> wrote:
> >>
> >>> scsi_dh_rdac modules is insmodded by multipath when multipathd is
> >>> started (by virtue of you having hardware_handler "1 rdac" in
> >>> your /etc/muliutpath.conf file.
> >>>
> >>> So, it is effective when multipath starts.
> >>
> >> If it is effective, I don't need to patch it... Thank you very much.
> >>
> >> You were saying that you could provide some debugging code if I'm
> >> interested? Well it seems that I don't have a choice... So please do
> >> so :)
> >>
> > Basically I want to find out if we are getting and sense codes that we
> > are not handling. To that effect, andd the following line at the top of
> > the function rdac_check_sense()
> > --------
> > sdev_printk(KERN_ERR, sdev, "I/O returned with sense %02x/%02x/%02x",
> >                      sense_hdr->sense_key, sense_hdr->asc, sense_hdr->ascq);
> > ---------
> >
> > compile the module, run your tests and get me /var/log/messages.
> 
> Might this be the same issue as you worked through with Babu last October?
> 
> http://kerneltrap.org/mailarchive/linux-scsi/2008/10/24/3803244
> 
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel

Re: Sun StorageTek 2530 and dm?

[Charlie Brady]

On Tue, 25 Aug 2009, Chandra Seetharaman wrote:

> Hopefully not :)... it has been fixed for a while.
>
> Jacov, Can you send me the scsi_dh_rdac.c file. I want to make sure you
> are running the latest code.

Jacov told us that he was running 2.6.18-128.4.1.el5xen, which certainly 
isn't the latest code. It's the latest Red Hat RHEL5 code (or rather, was 
until yesterday).

Below are the diffs from linus's git repo. I'd guess that there would be 
minor changes needed before the current code would compile and run with 
the RHEL5 kernel.

Chandra, can you spot any one or two patches that would be most important 
to back-port to RHEL5?

--- kernel-2.6.18/linux-2.6.18.i386/drivers/scsi/device_handler/scsi_dh_rdac.c	2009-08-25 15:00:38.745457000 -0400
+++ /home/speech/bradyc/scsi_dh_rdac.c	2009-08-25 15:10:42.922477000 
-0400
@@ -19,14 +19,12 @@
   * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
   *
   */
-#include <linux/list.h>
  #include <scsi/scsi.h>
  #include <scsi/scsi_eh.h>
-#include <scsi/scsi_cmnd.h>
  #include <scsi/scsi_dh.h>
-#include "../scsi_priv.h"

  #define RDAC_NAME "rdac"
+#define RDAC_RETRY_COUNT 5

  /*
   * LSI mode page stuff
@@ -202,7 +200,7 @@

  static inline struct rdac_dh_data *get_rdac_data(struct scsi_device *sdev)
  {
-	struct scsi_dh_data *scsi_dh_data = retrieve_scsi_dh_data(sdev);
+	struct scsi_dh_data *scsi_dh_data = sdev->scsi_dh_data;
  	BUG_ON(scsi_dh_data == NULL);
  	return ((struct rdac_dh_data *) scsi_dh_data->buf);
  }
@@ -228,10 +226,9 @@
  		return NULL;
  	}

-	memset(rq->cmd, 0, BLK_MAX_CDB);
-
-	rq->flags |= REQ_FAILFAST_DEV | REQ_FAILFAST_TRANSPORT |
-			REQ_FAILFAST_DRIVER | REQ_NOMERGE | REQ_BLOCK_PC;
+	rq->cmd_type = REQ_TYPE_BLOCK_PC;
+	rq->cmd_flags |= REQ_FAILFAST_DEV | REQ_FAILFAST_TRANSPORT |
+			 REQ_FAILFAST_DRIVER;
  	rq->retries = RDAC_RETRIES;
  	rq->timeout = RDAC_TIMEOUT;

@@ -390,6 +387,7 @@
  	struct c9_inquiry *inqp;

  	h->lun_state = RDAC_LUN_UNOWNED;
+	h->state = RDAC_STATE_ACTIVE;
  	err = submit_inquiry(sdev, 0xC9, sizeof(struct c9_inquiry), h);
  	if (err == SCSI_DH_OK) {
  		inqp = &h->inq.c9;
@@ -407,6 +405,7 @@

  	if (h->lun_state == RDAC_LUN_UNOWNED)
  		h->state = RDAC_STATE_PASSIVE;
+
  	return err;
  }

@@ -450,28 +449,40 @@
  				    unsigned char *sensebuf)
  {
  	struct scsi_sense_hdr sense_hdr;
-	int sense, err = SCSI_DH_IO, ret;
+	int err = SCSI_DH_IO, ret;

  	ret = scsi_normalize_sense(sensebuf, SCSI_SENSE_BUFFERSIZE, &sense_hdr);
  	if (!ret)
  		goto done;

  	err = SCSI_DH_OK;
-	sense = (sense_hdr.sense_key << 16) | (sense_hdr.asc << 8) |
-			sense_hdr.ascq;
-	/* If it is retryable failure, submit the c9 inquiry again */
-	if (sense == 0x59136 || sense == 0x68b02 || sense == 0xb8b02 ||
-			    sense == 0x62900) {
-		/* 0x59136    - Command lock contention
-		 * 0x[6b]8b02 - Quiesense in progress or achieved
-		 * 0x62900    - Power On, Reset, or Bus Device Reset
-		 */
+
+	switch (sense_hdr.sense_key) {
+	case NO_SENSE:
+	case ABORTED_COMMAND:
+	case UNIT_ATTENTION:
  		err = SCSI_DH_RETRY;
+		break;
+	case NOT_READY:
+		if (sense_hdr.asc == 0x04 && sense_hdr.ascq == 0x01)
+			/* LUN Not Ready and is in the Process of Becoming
+			 * Ready
+			 */
+			err = SCSI_DH_RETRY;
+		break;
+	case ILLEGAL_REQUEST:
+		if (sense_hdr.asc == 0x91 && sense_hdr.ascq == 0x36)
+			/*
+			 * Command Lock contention
+			 */
+			err = SCSI_DH_RETRY;
+		break;
+	default:
+		sdev_printk(KERN_INFO, sdev,
+			    "MODE_SELECT failed with sense %02x/%02x/%02x.\n",
+			    sense_hdr.sense_key, sense_hdr.asc, sense_hdr.ascq);
  	}

-	if (sense)
-		sdev_printk(KERN_INFO, sdev,
-			"MODE_SELECT failed with sense 0x%x.\n", sense);
  done:
  	return err;
  }
@@ -480,21 +491,27 @@
  {
  	struct request *rq;
  	struct request_queue *q = sdev->request_queue;
-	int err = SCSI_DH_RES_TEMP_UNAVAIL;
+	int err, retry_cnt = RDAC_RETRY_COUNT;

+retry:
+	err = SCSI_DH_RES_TEMP_UNAVAIL;
  	rq = rdac_failover_get(sdev, h);
  	if (!rq)
  		goto done;

-	sdev_printk(KERN_INFO, sdev, "queueing MODE_SELECT command.\n");
+	sdev_printk(KERN_INFO, sdev, "%s MODE_SELECT command.\n",
+		(retry_cnt == RDAC_RETRY_COUNT) ? "queueing" : "retrying");

  	err = blk_execute_rq(q, NULL, rq, 1);
-	if (err != SCSI_DH_OK)
+	blk_put_request(rq);
+	if (err != SCSI_DH_OK) {
  		err = mode_select_handle_sense(sdev, h->sense);
+		if (err == SCSI_DH_RETRY && retry_cnt--)
+			goto retry;
+	}
  	if (err == SCSI_DH_OK)
  		h->state = RDAC_STATE_ACTIVE;

-	blk_put_request(rq);
  done:
  	return err;
  }
@@ -532,7 +549,7 @@

  	if (h->state != RDAC_STATE_ACTIVE) {
  		ret = BLKPREP_KILL;
-		req->flags |= REQ_QUIET;
+		req->cmd_flags |= REQ_QUIET;
  	}
  	return ret;

@@ -544,19 +561,31 @@
  	struct rdac_dh_data *h = get_rdac_data(sdev);
  	switch (sense_hdr->sense_key) {
  	case NOT_READY:
+		if (sense_hdr->asc == 0x04 && sense_hdr->ascq == 0x01)
+			/* LUN Not Ready - Logical Unit Not Ready and is in
+			* the process of becoming ready
+			* Just retry.
+			*/
+			return ADD_TO_MLQUEUE;
  		if (sense_hdr->asc == 0x04 && sense_hdr->ascq == 0x81)
  			/* LUN Not Ready - Storage firmware incompatible
  			 * Manual code synchonisation required.
  			 *
  			 * Nothing we can do here. Try to bypass the path.
  			 */
-			return SCSI_MLQUEUE_DIS_FINISH;
+			return SUCCESS;
  		if (sense_hdr->asc == 0x04 && sense_hdr->ascq == 0xA1)
  			/* LUN Not Ready - Quiescense in progress
  			 *
  			 * Just retry and wait.
  			 */
-			return SCSI_MLQUEUE_IMM_RETRY;
+			return ADD_TO_MLQUEUE;
+		if (sense_hdr->asc == 0xA1  && sense_hdr->ascq == 0x02)
+			/* LUN Not Ready - Quiescense in progress
+			 * or has been achieved
+			 * Just retry.
+			 */
+			return ADD_TO_MLQUEUE;
  		break;
  	case ILLEGAL_REQUEST:
  		if (sense_hdr->asc == 0x94 && sense_hdr->ascq == 0x01) {
@@ -565,7 +594,7 @@
  			 * Fail the path, so that the other path be used.
  			 */
  			h->state = RDAC_STATE_PASSIVE;
-			return SCSI_MLQUEUE_DIS_FINISH;
+			return SUCCESS;
  		}
  		break;
  	case UNIT_ATTENTION:
@@ -573,14 +602,19 @@
  			/*
  			 * Power On, Reset, or Bus Device Reset, just retry.
  			 */
-			return SCSI_MLQUEUE_IMM_RETRY;
+			return ADD_TO_MLQUEUE;
+		if (sense_hdr->asc == 0x8b && sense_hdr->ascq == 0x02)
+			/*
+			 * Quiescence in progress , just retry.
+			 */
+			return ADD_TO_MLQUEUE;
  		break;
  	}
  	/* success just means we do not care what scsi-ml does */
-	return 0;
+	return SCSI_RETURN_NOT_HANDLED;
  }

-const struct scsi_dh_devlist rdac_dev_list[] = {
+static const struct scsi_dh_devlist rdac_dev_list[] = {
  	{"IBM", "1722"},
  	{"IBM", "1724"},
  	{"IBM", "1726"},
@@ -595,6 +629,10 @@
  	{"STK", "OPENstorage D280"},
  	{"SUN", "CSM200_R"},
  	{"SUN", "LCSM100_F"},
+	{"DELL", "MD3000"},
+	{"DELL", "MD3000i"},
+	{"LSI", "INF-01-00"},
+	{"ENGENIO", "INF-01-00"},
  	{NULL, NULL},
  };

@@ -612,9 +650,6 @@
  	.activate = rdac_activate,
  };

-/*
- * TODO: need some interface so we can set trespass values
- */
  static int rdac_bus_attach(struct scsi_device *sdev)
  {
  	struct scsi_dh_data *scsi_dh_data;
@@ -623,10 +658,11 @@
  	int err;

  	scsi_dh_data = kzalloc(sizeof(struct scsi_device_handler *)
-				+ sizeof(*h) , GFP_KERNEL);
+			       + sizeof(*h) , GFP_KERNEL);
  	if (!scsi_dh_data) {
-		sdev_printk(KERN_ERR, sdev, "%s: Attach failed\n", RDAC_NAME);
-		return -ENOMEM;
+		sdev_printk(KERN_ERR, sdev, "%s: Attach failed\n",
+			    RDAC_NAME);
+		return 0;
  	}

  	scsi_dh_data->scsi_dh = &rdac_dh;
@@ -646,28 +682,31 @@
  		goto failed;

  	spin_lock_irqsave(sdev->request_queue->queue_lock, flags);
-	store_scsi_dh_data(sdev, scsi_dh_data);
+	sdev->scsi_dh_data = scsi_dh_data;
  	spin_unlock_irqrestore(sdev->request_queue->queue_lock, flags);

-	sdev_printk(KERN_NOTICE, sdev, "%s: LUN %d (%s)\n",
-			RDAC_NAME, h->lun, lun_state[(int)h->lun_state]);
+	sdev_printk(KERN_NOTICE, sdev,
+		    "%s: LUN %d (%s)\n",
+		    RDAC_NAME, h->lun, lun_state[(int)h->lun_state]);
+
  	return 0;

  failed:
  	kfree(scsi_dh_data);
-	sdev_printk(KERN_ERR, sdev, "%s: not attached\n", RDAC_NAME);
+	sdev_printk(KERN_ERR, sdev, "%s: not attached\n",
+		    RDAC_NAME);
  	return -EINVAL;
  }

-static void rdac_bus_detach(struct scsi_device *sdev)
+static void rdac_bus_detach( struct scsi_device *sdev )
  {
  	struct scsi_dh_data *scsi_dh_data;
  	struct rdac_dh_data *h;
  	unsigned long flags;

  	spin_lock_irqsave(sdev->request_queue->queue_lock, flags);
-	scsi_dh_data = retrieve_scsi_dh_data(sdev);
-	store_scsi_dh_data(sdev, NULL);
+	scsi_dh_data = sdev->scsi_dh_data;
+	sdev->scsi_dh_data = NULL;
  	spin_unlock_irqrestore(sdev->request_queue->queue_lock, flags);

  	h = (struct rdac_dh_data *) scsi_dh_data->buf;
@@ -675,9 +714,11 @@
  		kref_put(&h->ctlr->kref, release_controller);
  	kfree(scsi_dh_data);
  	module_put(THIS_MODULE);
-	sdev_printk(KERN_NOTICE, sdev, "%s Dettached\n", RDAC_NAME);
+	sdev_printk(KERN_NOTICE, sdev, "%s: Detached\n", RDAC_NAME);
  }

+
+
  static int __init rdac_init(void)
  {
  	int r;

Re: Sun StorageTek 2530 and dm?

[Chandra Seetharaman]

I try to keep the latest releases of RHEL and SLES by porting and test
them.

These changes are ported to RHEL 5.4, and will be available.

Do you still want to know which of these you might need (for your
system)

chandra
On Tue, 2009-08-25 at 15:21 -0400, Charlie Brady wrote:
> On Tue, 25 Aug 2009, Chandra Seetharaman wrote:
> 
> > Hopefully not :)... it has been fixed for a while.
> >
> > Jacov, Can you send me the scsi_dh_rdac.c file. I want to make sure you
> > are running the latest code.
> 
> Jacov told us that he was running 2.6.18-128.4.1.el5xen, which certainly 
> isn't the latest code. It's the latest Red Hat RHEL5 code (or rather, was 
> until yesterday).
> 
> Below are the diffs from linus's git repo. I'd guess that there would be 
> minor changes needed before the current code would compile and run with 
> the RHEL5 kernel.
> 
> Chandra, can you spot any one or two patches that would be most important 
> to back-port to RHEL5?
> 
> --- kernel-2.6.18/linux-2.6.18.i386/drivers/scsi/device_handler/scsi_dh_rdac.c	2009-08-25 15:00:38.745457000 -0400
> +++ /home/speech/bradyc/scsi_dh_rdac.c	2009-08-25 15:10:42.922477000 
> -0400
> @@ -19,14 +19,12 @@
>    * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
>    *
>    */
> -#include <linux/list.h>
>   #include <scsi/scsi.h>
>   #include <scsi/scsi_eh.h>
> -#include <scsi/scsi_cmnd.h>
>   #include <scsi/scsi_dh.h>
> -#include "../scsi_priv.h"
> 
>   #define RDAC_NAME "rdac"
> +#define RDAC_RETRY_COUNT 5
> 
>   /*
>    * LSI mode page stuff
> @@ -202,7 +200,7 @@
> 
>   static inline struct rdac_dh_data *get_rdac_data(struct scsi_device *sdev)
>   {
> -	struct scsi_dh_data *scsi_dh_data = retrieve_scsi_dh_data(sdev);
> +	struct scsi_dh_data *scsi_dh_data = sdev->scsi_dh_data;
>   	BUG_ON(scsi_dh_data == NULL);
>   	return ((struct rdac_dh_data *) scsi_dh_data->buf);
>   }
> @@ -228,10 +226,9 @@
>   		return NULL;
>   	}
> 
> -	memset(rq->cmd, 0, BLK_MAX_CDB);
> -
> -	rq->flags |= REQ_FAILFAST_DEV | REQ_FAILFAST_TRANSPORT |
> -			REQ_FAILFAST_DRIVER | REQ_NOMERGE | REQ_BLOCK_PC;
> +	rq->cmd_type = REQ_TYPE_BLOCK_PC;
> +	rq->cmd_flags |= REQ_FAILFAST_DEV | REQ_FAILFAST_TRANSPORT |
> +			 REQ_FAILFAST_DRIVER;
>   	rq->retries = RDAC_RETRIES;
>   	rq->timeout = RDAC_TIMEOUT;
> 
> @@ -390,6 +387,7 @@
>   	struct c9_inquiry *inqp;
> 
>   	h->lun_state = RDAC_LUN_UNOWNED;
> +	h->state = RDAC_STATE_ACTIVE;
>   	err = submit_inquiry(sdev, 0xC9, sizeof(struct c9_inquiry), h);
>   	if (err == SCSI_DH_OK) {
>   		inqp = &h->inq.c9;
> @@ -407,6 +405,7 @@
> 
>   	if (h->lun_state == RDAC_LUN_UNOWNED)
>   		h->state = RDAC_STATE_PASSIVE;
> +
>   	return err;
>   }
> 
> @@ -450,28 +449,40 @@
>   				    unsigned char *sensebuf)
>   {
>   	struct scsi_sense_hdr sense_hdr;
> -	int sense, err = SCSI_DH_IO, ret;
> +	int err = SCSI_DH_IO, ret;
> 
>   	ret = scsi_normalize_sense(sensebuf, SCSI_SENSE_BUFFERSIZE, &sense_hdr);
>   	if (!ret)
>   		goto done;
> 
>   	err = SCSI_DH_OK;
> -	sense = (sense_hdr.sense_key << 16) | (sense_hdr.asc << 8) |
> -			sense_hdr.ascq;
> -	/* If it is retryable failure, submit the c9 inquiry again */
> -	if (sense == 0x59136 || sense == 0x68b02 || sense == 0xb8b02 ||
> -			    sense == 0x62900) {
> -		/* 0x59136    - Command lock contention
> -		 * 0x[6b]8b02 - Quiesense in progress or achieved
> -		 * 0x62900    - Power On, Reset, or Bus Device Reset
> -		 */
> +
> +	switch (sense_hdr.sense_key) {
> +	case NO_SENSE:
> +	case ABORTED_COMMAND:
> +	case UNIT_ATTENTION:
>   		err = SCSI_DH_RETRY;
> +		break;
> +	case NOT_READY:
> +		if (sense_hdr.asc == 0x04 && sense_hdr.ascq == 0x01)
> +			/* LUN Not Ready and is in the Process of Becoming
> +			 * Ready
> +			 */
> +			err = SCSI_DH_RETRY;
> +		break;
> +	case ILLEGAL_REQUEST:
> +		if (sense_hdr.asc == 0x91 && sense_hdr.ascq == 0x36)
> +			/*
> +			 * Command Lock contention
> +			 */
> +			err = SCSI_DH_RETRY;
> +		break;
> +	default:
> +		sdev_printk(KERN_INFO, sdev,
> +			    "MODE_SELECT failed with sense %02x/%02x/%02x.\n",
> +			    sense_hdr.sense_key, sense_hdr.asc, sense_hdr.ascq);
>   	}
> 
> -	if (sense)
> -		sdev_printk(KERN_INFO, sdev,
> -			"MODE_SELECT failed with sense 0x%x.\n", sense);
>   done:
>   	return err;
>   }
> @@ -480,21 +491,27 @@
>   {
>   	struct request *rq;
>   	struct request_queue *q = sdev->request_queue;
> -	int err = SCSI_DH_RES_TEMP_UNAVAIL;
> +	int err, retry_cnt = RDAC_RETRY_COUNT;
> 
> +retry:
> +	err = SCSI_DH_RES_TEMP_UNAVAIL;
>   	rq = rdac_failover_get(sdev, h);
>   	if (!rq)
>   		goto done;
> 
> -	sdev_printk(KERN_INFO, sdev, "queueing MODE_SELECT command.\n");
> +	sdev_printk(KERN_INFO, sdev, "%s MODE_SELECT command.\n",
> +		(retry_cnt == RDAC_RETRY_COUNT) ? "queueing" : "retrying");
> 
>   	err = blk_execute_rq(q, NULL, rq, 1);
> -	if (err != SCSI_DH_OK)
> +	blk_put_request(rq);
> +	if (err != SCSI_DH_OK) {
>   		err = mode_select_handle_sense(sdev, h->sense);
> +		if (err == SCSI_DH_RETRY && retry_cnt--)
> +			goto retry;
> +	}
>   	if (err == SCSI_DH_OK)
>   		h->state = RDAC_STATE_ACTIVE;
> 
> -	blk_put_request(rq);
>   done:
>   	return err;
>   }
> @@ -532,7 +549,7 @@
> 
>   	if (h->state != RDAC_STATE_ACTIVE) {
>   		ret = BLKPREP_KILL;
> -		req->flags |= REQ_QUIET;
> +		req->cmd_flags |= REQ_QUIET;
>   	}
>   	return ret;
> 
> @@ -544,19 +561,31 @@
>   	struct rdac_dh_data *h = get_rdac_data(sdev);
>   	switch (sense_hdr->sense_key) {
>   	case NOT_READY:
> +		if (sense_hdr->asc == 0x04 && sense_hdr->ascq == 0x01)
> +			/* LUN Not Ready - Logical Unit Not Ready and is in
> +			* the process of becoming ready
> +			* Just retry.
> +			*/
> +			return ADD_TO_MLQUEUE;
>   		if (sense_hdr->asc == 0x04 && sense_hdr->ascq == 0x81)
>   			/* LUN Not Ready - Storage firmware incompatible
>   			 * Manual code synchonisation required.
>   			 *
>   			 * Nothing we can do here. Try to bypass the path.
>   			 */
> -			return SCSI_MLQUEUE_DIS_FINISH;
> +			return SUCCESS;
>   		if (sense_hdr->asc == 0x04 && sense_hdr->ascq == 0xA1)
>   			/* LUN Not Ready - Quiescense in progress
>   			 *
>   			 * Just retry and wait.
>   			 */
> -			return SCSI_MLQUEUE_IMM_RETRY;
> +			return ADD_TO_MLQUEUE;
> +		if (sense_hdr->asc == 0xA1  && sense_hdr->ascq == 0x02)
> +			/* LUN Not Ready - Quiescense in progress
> +			 * or has been achieved
> +			 * Just retry.
> +			 */
> +			return ADD_TO_MLQUEUE;
>   		break;
>   	case ILLEGAL_REQUEST:
>   		if (sense_hdr->asc == 0x94 && sense_hdr->ascq == 0x01) {
> @@ -565,7 +594,7 @@
>   			 * Fail the path, so that the other path be used.
>   			 */
>   			h->state = RDAC_STATE_PASSIVE;
> -			return SCSI_MLQUEUE_DIS_FINISH;
> +			return SUCCESS;
>   		}
>   		break;
>   	case UNIT_ATTENTION:
> @@ -573,14 +602,19 @@
>   			/*
>   			 * Power On, Reset, or Bus Device Reset, just retry.
>   			 */
> -			return SCSI_MLQUEUE_IMM_RETRY;
> +			return ADD_TO_MLQUEUE;
> +		if (sense_hdr->asc == 0x8b && sense_hdr->ascq == 0x02)
> +			/*
> +			 * Quiescence in progress , just retry.
> +			 */
> +			return ADD_TO_MLQUEUE;
>   		break;
>   	}
>   	/* success just means we do not care what scsi-ml does */
> -	return 0;
> +	return SCSI_RETURN_NOT_HANDLED;
>   }
> 
> -const struct scsi_dh_devlist rdac_dev_list[] = {
> +static const struct scsi_dh_devlist rdac_dev_list[] = {
>   	{"IBM", "1722"},
>   	{"IBM", "1724"},
>   	{"IBM", "1726"},
> @@ -595,6 +629,10 @@
>   	{"STK", "OPENstorage D280"},
>   	{"SUN", "CSM200_R"},
>   	{"SUN", "LCSM100_F"},
> +	{"DELL", "MD3000"},
> +	{"DELL", "MD3000i"},
> +	{"LSI", "INF-01-00"},
> +	{"ENGENIO", "INF-01-00"},
>   	{NULL, NULL},
>   };
> 
> @@ -612,9 +650,6 @@
>   	.activate = rdac_activate,
>   };
> 
> -/*
> - * TODO: need some interface so we can set trespass values
> - */
>   static int rdac_bus_attach(struct scsi_device *sdev)
>   {
>   	struct scsi_dh_data *scsi_dh_data;
> @@ -623,10 +658,11 @@
>   	int err;
> 
>   	scsi_dh_data = kzalloc(sizeof(struct scsi_device_handler *)
> -				+ sizeof(*h) , GFP_KERNEL);
> +			       + sizeof(*h) , GFP_KERNEL);
>   	if (!scsi_dh_data) {
> -		sdev_printk(KERN_ERR, sdev, "%s: Attach failed\n", RDAC_NAME);
> -		return -ENOMEM;
> +		sdev_printk(KERN_ERR, sdev, "%s: Attach failed\n",
> +			    RDAC_NAME);
> +		return 0;
>   	}
> 
>   	scsi_dh_data->scsi_dh = &rdac_dh;
> @@ -646,28 +682,31 @@
>   		goto failed;
> 
>   	spin_lock_irqsave(sdev->request_queue->queue_lock, flags);
> -	store_scsi_dh_data(sdev, scsi_dh_data);
> +	sdev->scsi_dh_data = scsi_dh_data;
>   	spin_unlock_irqrestore(sdev->request_queue->queue_lock, flags);
> 
> -	sdev_printk(KERN_NOTICE, sdev, "%s: LUN %d (%s)\n",
> -			RDAC_NAME, h->lun, lun_state[(int)h->lun_state]);
> +	sdev_printk(KERN_NOTICE, sdev,
> +		    "%s: LUN %d (%s)\n",
> +		    RDAC_NAME, h->lun, lun_state[(int)h->lun_state]);
> +
>   	return 0;
> 
>   failed:
>   	kfree(scsi_dh_data);
> -	sdev_printk(KERN_ERR, sdev, "%s: not attached\n", RDAC_NAME);
> +	sdev_printk(KERN_ERR, sdev, "%s: not attached\n",
> +		    RDAC_NAME);
>   	return -EINVAL;
>   }
> 
> -static void rdac_bus_detach(struct scsi_device *sdev)
> +static void rdac_bus_detach( struct scsi_device *sdev )
>   {
>   	struct scsi_dh_data *scsi_dh_data;
>   	struct rdac_dh_data *h;
>   	unsigned long flags;
> 
>   	spin_lock_irqsave(sdev->request_queue->queue_lock, flags);
> -	scsi_dh_data = retrieve_scsi_dh_data(sdev);
> -	store_scsi_dh_data(sdev, NULL);
> +	scsi_dh_data = sdev->scsi_dh_data;
> +	sdev->scsi_dh_data = NULL;
>   	spin_unlock_irqrestore(sdev->request_queue->queue_lock, flags);
> 
>   	h = (struct rdac_dh_data *) scsi_dh_data->buf;
> @@ -675,9 +714,11 @@
>   		kref_put(&h->ctlr->kref, release_controller);
>   	kfree(scsi_dh_data);
>   	module_put(THIS_MODULE);
> -	sdev_printk(KERN_NOTICE, sdev, "%s Dettached\n", RDAC_NAME);
> +	sdev_printk(KERN_NOTICE, sdev, "%s: Detached\n", RDAC_NAME);
>   }
> 
> +
> +
>   static int __init rdac_init(void)
>   {
>   	int r;
> 
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel

Re: Sun StorageTek 2530 and dm?

[Charlie Brady]

On Tue, 25 Aug 2009, Chandra Seetharaman wrote:

> I try to keep the latest releases of RHEL and SLES by porting and test
> them.
>
> These changes are ported to RHEL 5.4, and will be available.

They appear not to be in RHEL5.3. The only mention of rdac I see in the 
changelog of 2.6.18-128.4.1.el5 is this one:

* Fri Sep 12 2008 Don Zickus <dzickus@redhat.com> [2.6.18-112.el5]
...
- [scsi] scsi_dh: add rdac handler (mchristi@redhat.com ) [438761]
...

https://bugzilla.redhat.com/show_bug.cgi?id=438761

I did look at the RHEL 5.4 beta, and there are other updates mentioned in 
changelog there:

* Wed May 20 2009 Don Zickus <dzickus@redhat.com> [2.6.18-150.el5]
...
- [scsi] Retry mode select in rdac device handler (Jesse Larrew ) [489582]
...
* Fri May 01 2009 Don Zickus <dzickus@redhat.com> [2.6.18-143.el5]
...
- [scsi] add md3000 and md3000i entries to rdac_dev_list (John Feeney ) 
[487293]
...

https://bugzilla.redhat.com/show_bug.cgi?id=487293

Jacov, you might want to look at the above bugzilla. There is some 
discussion of failover behaviour which might be useful to you.

> Do you still want to know which of these you might need (for your 
> system)

Yes I would. Or better, if you see any required patches which are not 
included in RHEL5.4, please let RedHat know (via Bugzilla). I will ask 
them to add support for the StorageTek 2510 and 2530 arrays.

---
Charlie

Re: Sun StorageTek 2530 and dm?

[Jakov Sosic]

On Tue, 25 Aug 2009 16:51:34 -0400 (EDT)
Charlie Brady <charlieb-dm-devel@budge.apana.org.au> wrote:

> I did look at the RHEL 5.4 beta, and there are other updates
> mentioned in changelog there:
> 
> * Wed May 20 2009 Don Zickus <dzickus@redhat.com> [2.6.18-150.el5]
> ...
> - [scsi] Retry mode select in rdac device handler (Jesse Larrew )
> [489582] ...
> * Fri May 01 2009 Don Zickus <dzickus@redhat.com> [2.6.18-143.el5]
> ...
> - [scsi] add md3000 and md3000i entries to rdac_dev_list (John
> Feeney ) [487293]
> ...
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=487293
> 
> Jacov, you might want to look at the above bugzilla. There is some 
> discussion of failover behaviour which might be useful to you.

Thank you, I'll take a look...


OK, I've noticed the following:

Feb 25 11:29:55 rhel5r300 multipathd: 8:16: reinstated
Feb 25 11:29:55 rhel5r300 multipathd: 8:32: reinstated
Feb 25 11:29:55 rhel5r300 multipathd: 8:48: reinstated
Feb 25 11:29:55 rhel5r300 multipathd: 8:64: reinstated


How often are these messages supposed to repeat? Because I've succeded
with one combination of HostType in StorageTek 2530 configuration for
initiators, and RDAC on the Linux side. But I had messages like these
every few minutes in /var/log/messages.


-- 
|    Jakov Sosic    |    ICQ: 28410271    |   PGP: 0x965CAE2D   |
=================================================================
| start fighting cancer -> http://www.worldcommunitygrid.org/   |

Re: Sun StorageTek 2530 and dm?

[Charlie Brady]

On Tue, 25 Aug 2009, Chandra Seetharaman wrote:

> I try to keep the latest releases of RHEL and SLES by porting and test
> them.
>
> These changes are ported to RHEL 5.4, and will be available.
>
> Do you still want to know which of these you might need (for your
> system)

I notice that this patch doesn't seem to have been merged by Red Hat:

http://kerneltrap.org/mailarchive/linux-scsi/2008/10/30/3873414

That's this one:

>> @@ -390,6 +387,7 @@
>>   	struct c9_inquiry *inqp;
>>
>>   	h->lun_state = RDAC_LUN_UNOWNED;
>> +	h->state = RDAC_STATE_ACTIVE;
>>   	err = submit_inquiry(sdev, 0xC9, sizeof(struct c9_inquiry), h);
>>   	if (err == SCSI_DH_OK) {
>>   		inqp = &h->inq.c9;

Re: Sun StorageTek 2530 and dm?

[Chandra Seetharaman]

On Wed, 2009-08-26 at 17:57 -0400, Charlie Brady wrote:
> 
> On Tue, 25 Aug 2009, Chandra Seetharaman wrote:
> 
> > I try to keep the latest releases of RHEL and SLES by porting and test
> > them.
> >
> > These changes are ported to RHEL 5.4, and will be available.
> >
> > Do you still want to know which of these you might need (for your
> > system)
> 
> I notice that this patch doesn't seem to have been merged by Red Hat:
> 
> http://kerneltrap.org/mailarchive/linux-scsi/2008/10/30/3873414

Hmm, I see it in my RHEL 5.4. 

what is the RedHat kernel version you are looking at (in RHEL 5.4) ?

> 
> That's this one:
> 
> >> @@ -390,6 +387,7 @@
> >>   	struct c9_inquiry *inqp;
> >>
> >>   	h->lun_state = RDAC_LUN_UNOWNED;
> >> +	h->state = RDAC_STATE_ACTIVE;
> >>   	err = submit_inquiry(sdev, 0xC9, sizeof(struct c9_inquiry), h);
> >>   	if (err == SCSI_DH_OK) {
> >>   		inqp = &h->inq.c9;
> 

Re: Sun StorageTek 2530 and dm?

[Charlie Brady]

On Wed, 26 Aug 2009, Chandra Seetharaman wrote:

>> I notice that this patch doesn't seem to have been merged by Red Hat:
>>
>> http://kerneltrap.org/mailarchive/linux-scsi/2008/10/30/3873414
>
> Hmm, I see it in my RHEL 5.4.
>
> what is the RedHat kernel version you are looking at (in RHEL 5.4) ?

My mistake, sorry. I was looking at 5.3.

RHEL 5.4 does contain these:

1219949016-15055-5-git-send-email-mchristi@redhat.com
20090514045648.10947.8280.sendpatchset@squad5-lp1.lab.bos.redhat.com

Re: Sun StorageTek 2530 and dm?

[Jakov Sosic]

On Wed, 26 Aug 2009 19:01:36 -0400 (EDT)
Charlie Brady <charlieb-dm-devel@budge.apana.org.au> wrote:

> My mistake, sorry. I was looking at 5.3.
> 
> RHEL 5.4 does contain these:
> 
> 1219949016-15055-5-git-send-email-mchristi@redhat.com
> 20090514045648.10947.8280.sendpatchset@squad5-lp1.lab.bos.redhat.com

Can I just try the multipath with kernel from 5.4 beta on 5.3? Is there
a need to upgrade the device-mapper-multipath too, or are these kernel
patches enough to test if this is going to work in my case?


-- 
|    Jakov Sosic    |    ICQ: 28410271    |   PGP: 0x965CAE2D   |
=================================================================
| start fighting cancer -> http://www.worldcommunitygrid.org/   |

Re: Sun StorageTek 2530 and dm?

[Charlie Brady]

On Thu, 27 Aug 2009, Jakov Sosic wrote:

> On Wed, 26 Aug 2009 19:01:36 -0400 (EDT)
> Charlie Brady <charlieb-dm-devel@budge.apana.org.au> wrote:
>
>> My mistake, sorry. I was looking at 5.3.
>>
>> RHEL 5.4 does contain these:
>>
>> 1219949016-15055-5-git-send-email-mchristi@redhat.com
>> 20090514045648.10947.8280.sendpatchset@squad5-lp1.lab.bos.redhat.com
>
> Can I just try the multipath with kernel from 5.4 beta on 5.3?

You can use the 5.4beta kernel with 5.3. However, that kernel does not 
contain support for the 2530 in the scsi_dh_rdac module. You would need to 
patch the module. However, I have a src rpm which updates just that 
module. I will send it to you, with rebuild instructions.

> Is there a need to upgrade the device-mapper-multipath too, or are these 
> kernel patches enough to test if this is going to work in my case?

You don't need to upgrade device-mapper-multipath to include a default 
configuration for the 2530, but you would need to add a device stanza in 
multipath.conf. There might be other reasons to upgrade 
device-mapper-multipath to the RH5.4 version - I haven't checked the 
changelog.

---
Charlie

Re: Sun StorageTek 2530 and dm?

[Chandra Seetharaman]

On Thu, 2009-08-27 at 10:30 +0200, Jakov Sosic wrote:
> On Wed, 26 Aug 2009 19:01:36 -0400 (EDT)
> Charlie Brady <charlieb-dm-devel@budge.apana.org.au> wrote:
> 
> > My mistake, sorry. I was looking at 5.3.
> > 
> > RHEL 5.4 does contain these:
> > 
> > 1219949016-15055-5-git-send-email-mchristi@redhat.com
> > 20090514045648.10947.8280.sendpatchset@squad5-lp1.lab.bos.redhat.com
> 
> Can I just try the multipath with kernel from 5.4 beta on 5.3? Is there
> a need to upgrade the device-mapper-multipath too, or are these kernel
> patches enough to test if this is going to work in my case?

If it is a production system I would suggest you to update the tools
also.
> 
>