device-errors and multipath device access issue

device-errors and multipath device access issue

[devzero]

Hello !

we are using multipath on sles9 and access those devices via /dev/mapper/devicename

on boot, we get lot`s of error messages like

> Aug 28 10:30:16 rac02 kernel: Buffer I/O error on device sdf, logical block 64239
> Aug 28 10:30:16 rac02 kernel: Buffer I/O error on device sdf, logical block 64240
> Aug 28 10:30:16 rac02 kernel: Buffer I/O error on device sdf, logical block 64241
> Aug 28 10:30:16 rac02 kernel: Buffer I/O error on device sdf, logical block 64242
> Aug 28 10:30:16 rac02 kernel: Buffer I/O error on device sdf, logical block 64243
> Aug 28 10:30:16 rac02 kernel: Device sdf not ready.
> Aug 28 10:30:16 rac02 kernel: end_request: I/O error, dev sdf, sector 513824
> Aug 28 10:30:16 rac02 kernel: Buffer I/O error on device sdf, logical block 64228
> Aug 28 10:30:16 rac02 kernel: Device sdf not ready.
> Aug 28 10:30:16 rac02 kernel: end_request: I/O error, dev sdf, sector 513824
> Aug 28 10:30:16 rac02 kernel: Buffer I/O error on device sdf, logical block 64228
> Aug 28 10:30:16 rac02 kernel: Device sdf not ready.
> Aug 28 10:30:16 rac02 kernel: end_request: I/O error, dev sdf, sector 514064
> Aug 28 10:30:16 rac02 kernel: Buffer I/O error on device sdf, logical block 64258
> Aug 28 10:30:16 rac02 kernel: Device sdf not ready.
> Aug 28 10:30:16 rac02 kernel: end_request: I/O error, dev sdf, sector 513680
> Aug 28 10:30:16 rac02 kernel: Buffer I/O error on device sdf, logical block 64210
> Aug 28 10:30:16 rac02 kernel: Buffer I/O error on device sdf, logical block 64211
> Aug 28 10:30:16 rac02 kernel: Buffer I/O error on device sdf, logical block 64212

for each alternative path to a lun.

we also get those messages when the system is up and running because some proprietary monitoring software is checking for device availability in regular intervals and there seems no way to tell that software to skip certain devices - so we get spammed with this messages like this in /var/log/messages and are not able to see the real errors anymore.

is there a way to hide those "classic" scsi devices from userspace?
i`m not sure if "blacklist" in multipath.conf is what i need here (?) or if i safely could delete those device-nodes - i`m not very deep into multipathing for now.


furthermore, we have some strange problem with ocfs2 i tracked down to /etc/init.d/boot.multipath.
sometimes (e.g. after adding a LUN in the SAN) cannot start anymore, because the device-mapper "links" to the partitions on the multipath device are inaccessible.

eg we have

/dev/mapper/000123largenumber456
/dev/mapper/000123largenumber789
/dev/mapper/000123largenumber456p1
/dev/mapper/000123largenumber456p2
/dev/mapper/000123largenumber456p3
/dev/mapper/000123largenumber456p4

the first 2 devices show recent timestamp, but the p# devices show old timestamp and aren`t accessible after reboot.
if we reboot a second or third time, this helps sometimes - also manually restarting multipath seems to help and the partitions get accessible again.

looks like they are setup with kpartx from within boot.multipath on every reboot and this goes wrong under certain circumstances because the device is not ready when kpartx want`s access that device. 

i found some reference at 
http://bugs.donarmstrong.com/cgi-bin/bugreport.cgi?bug=376161
and it looks similar:

>I think that the problem is that multipath is too slow creating the devices and
>udev is too fast launching kpartx.

in that bugreport it`s being told that  multipath-tools (0.4.7-8) has a fix, but i don`t get the point what`s the exact problem and what is being fixed 

>   * since Debian's dmestup doesn't include the "export" patch used by other
>     distros (#434241), work around this by implementing a minimal dmsetup_env
>     that can be used by kpartx.udev (Closes: #376161)


in our case, kpartx doesn`t seem to be launched by udev but from within boot.multipath, but it looks like a timing issue because it sometimes happens and sometimes not.

any help or some input (links?docs?) to enlighten me would be highly appreciated

thank you!

roland
sysadmin

ps:
please forgive if these questions sound a little bit dumb and unexperienced - i didn`t setup that environment and it also lacks documentations, but i want to help some other people solving these problems.






_______________________________________________________________________
Jetzt neu! Schützen Sie Ihren PC mit McAfee und WEB.DE. 3 Monate
kostenlos testen. http://www.pc-sicherheit.web.de/startseite/?mc=022220

Re: device-errors and multipath device access issue

[Tore Anderson]

* devzero@web.de

> on boot, we get lot`s of error messages like
> 
>> [...]
> 
> for each alternative path to a lun.

I assume you mean "for each path to the alternate controller of the volume".

> we also get those messages when the system is up and running because
> some proprietary monitoring software is checking for device
> availability in regular intervals and there seems no way to tell that
> software to skip certain devices - so we get spammed with this
> messages like this in /var/log/messages and are not able to see the
> real errors anymore.
> 
> is there a way to hide those "classic" scsi devices from userspace? 
> i`m not sure if "blacklist" in multipath.conf is what i need here (?)
> or if i safely could delete those device-nodes - i`m not very deep
> into multipathing for now.

I don't think you can (without at the same time taking away
dm-multipath's ability to fail over I/O to those paths...

You're not saying which kind of storage hardware this is.  Maybe it has
a way of being configured in a fake active/active configuration where
I/O can be processed on both controllers at the same time.  I believe
newer EMC CLARiiON and HP EVA can be set up in ALUA mode which does the
trick. HDS AMS does something similar which work mostly the same way.

You might also be able to filter out those lines from the logs, too.  I
think at least syslog-ng can be made to do that.

> in our case, kpartx doesn`t seem to be launched by udev but from
> within boot.multipath, but it looks like a timing issue because it
> sometimes happens and sometimes not.
> 
> any help or some input (links?docs?) to enlighten me would be highly
> appreciated

I'd suggest simply not making partitions - make several smaller volumes
on your storage array instead.  That'll make it easier to expand them
individually if the need should arise, and you won't have to deal with
kpartx at all.  There's always some kind of trouble with it, in my
experience.  Same goes for udev...

Regards
-- 
Tore Anderson

Re: device-errors and multipath device access issue

[devzero]

> for each alternative path to a lun.
>
>>I assume you mean "for each path to the alternate controller of the volume".

yes, indeed.

>> we also get those messages when the system is up and running because
>> some proprietary monitoring software is checking for device
>> availability in regular intervals and there seems no way to tell that
>> software to skip certain devices - so we get spammed with this
>> messages like this in /var/log/messages and are not able to see the
>> real errors anymore.
>> 
>> is there a way to hide those "classic" scsi devices from userspace? 
>> i`m not sure if "blacklist" in multipath.conf is what i need here (?)
>> or if i safely could delete those device-nodes - i`m not very deep
>> into multipathing for now.
>
>I don't think you can (without at the same time taking away
>dm-multipath's ability to fail over I/O to those paths...

ok - then i must live with this and hope that proprietary application can be stopped touching these....

>You're not saying which kind of storage hardware this is.  

sorry. it`s an EMC Clariion CX500

>Maybe it has a way of being configured in a fake active/active configuration where
>I/O can be processed on both controllers at the same time.  I believe
>newer EMC CLARiiON and HP EVA can be set up in ALUA mode which does the
>trick. HDS AMS does something similar which work mostly the same way.

yes - but i don`t think the admin wants to touch such essential configuration to get rid of error messages in the kernel-log....

>You might also be able to filter out those lines from the logs, too.  I
>think at least syslog-ng can be made to do that.

ok, but that`s not an elegant way to go....

>> in our case, kpartx doesn`t seem to be launched by udev but from
>> within boot.multipath, but it looks like a timing issue because it
>> sometimes happens and sometimes not.
>> 
>> any help or some input (links?docs?) to enlighten me would be highly
>> appreciated
>
>I'd suggest simply not making partitions - make several smaller volumes
>on your storage array instead.  That'll make it easier to expand them
>individually if the need should arise, and you won't have to deal with
>kpartx at all.  There's always some kind of trouble with it, in my
>experience.  Same goes for udev...

meanwhile, i found a novell paper which recommends the same - no partitions and access access /dev/disk instead of /dev/mapper
will discuss moving to a non partitioned scenario.

thanks.

roland
_______________________________________________________________________
Jetzt neu! Schützen Sie Ihren PC mit McAfee und WEB.DE. 3 Monate
kostenlos testen. http://www.pc-sicherheit.web.de/startseite/?mc=022220

Re: device-errors and multipath device access issue

[Tore Anderson]

* Tore Anderson

> Maybe it has a way of being configured in a fake active/active
> configuration where I/O can be processed on both controllers at the
> same time.  I believe newer EMC CLARiiON and HP EVA can be set up in
> ALUA mode which does the trick. HDS AMS does something similar which
> work mostly the same way.

* devzero@web.de

> yes - but i don`t think the admin wants to touch such essential
> configuration to get rid of error messages in the kernel-log....

Today I played around with a CX500 in EMC's lab (I'm considering buying 
another CLARiiON), and I got to test the ALUA mode quite a bit.  It 
works very well, and I'd recommend you to pester your admin to get it.

Basically what it does is that if I/O is received on the passive 
controller, it's just redirected to the active controller on an internal 
interconnect (and the replies are routed the same way back out again). 
So it looks and feels just like a true active/active setup, you'll not 
get I/O errors on any of the paths.  Supposedly there is a slight 
increase in latency on I/O that pass through the passive controller, but 
that was not noticeable in my simple benchmarks using multibus topology.

The system was dedicated to my tests, though, so I'm not promising it's 
okay to be using multibus in production on a busy system.  A better way 
to set it up is to be using mpath_prio_emc and group_by_prio like 
before, but with no hardware handler.  If dm-multipath decides to change 
priority groups the I/O will simply pass through the passive controller 
for some minutes until the CX realises that most of the I/O is going in 
the wrong way, and trespasses the volume automatically.  Small bursts of 
stray I/O to the passive controller (like those your proprietary 
application causes) will be processed fine (with the very slight latency 
increase), but it won't affect the production I/O going through 
dm-multipath straight to the active controller.  It's really nice for 
use with dm-multipath - and it'll solve all of your problems.

There's an ALUA hardware handler in the making too - I assume it will 
make it possible for dm-multipath to explicitly trespass the volume on 
PG change (instead of relying on the CX to automatically move the LUN). 
  In my opinion, though, that's not needed for a production quality 
setup with ALUA on the CLARiiON (maybe it's mostly inteded for other 
ALUA-capable arrays that might not have the auto-trespass feature).

ALUA mode is configured on a per-host basis (set it to failover mode 4, 
CLARiiON Open), so your admin shouldn't worry about changing it. 
However it is a very recently added feature (you'll need FLARE 26), so 
you might have to persuade him to invite EMC over for an upgrade session.

Regards
-- 
Tore Anderson

Re: device-errors and multipath device access issue

[Klaus]

 <devzero <at> web.de> writes:

> 
> Hello !
> 
> we are using multipath on sles9 and access those devices via
/dev/mapper/devicename
> 
> on boot, we get lot`s of error messages like
> 
> > Aug 28 10:30:16 rac02 kernel: Device sdf not ready.
> > Aug 28 10:30:16 rac02 kernel: end_request: I/O error, dev sdf, sector 513680
> > Aug 28 10:30:16 rac02 kernel: Buffer I/O error on device sdf, logical block
64210
> > Aug 28 10:30:16 rac02 kernel: Buffer I/O error on device sdf, logical block
64211
> > Aug 28 10:30:16 rac02 kernel: Buffer I/O error on device sdf, logical block
64212
> 
> for each alternative path to a lun.
> 
> we also get those messages when the system is up and running because some
proprietary monitoring software
> is checking for device availability in regular intervals and there seems no
way to tell that software to
> skip certain devices - so we get spammed with this messages like this in
/var/log/messages and are not able
> to see the real errors anymore.
> 
> is there a way to hide those "classic" scsi devices from userspace?
> i`m not sure if "blacklist" in multipath.conf is what i need here (?) or if i
safely could delete those
> device-nodes - i`m not very deep into multipathing for now.
> 


Hello Roland,

I got the same problem using Sles9SP3 with a newer Kernel on a FSC TX300-Server,
this one is connected to an san using a volume-Manager, Multipathsoftware and 
so on.
After reading your hint with the monitoring software I stopped the FSC
serveragent-daemons and the messages with the Buffer I/O error are gone.
In combination with lvm2-organized disks on a second TX300 I got something like
that you described second - I have to test it again (specially the boot-process,
maybe this can solved for me using a other naming scheme for the san-luns).

So, I don't know if it is helpful for you:
you're right with the monitoring-software I think (I will discuss this for me
with the FSC-guys) but: are there all SAN-Luns in the messages file (it is in my
case)?
I disconnected step by step one of the both FC-Paths, but on my TX300 the errors
still continued, and in that situation there where no second path to the san.
So, I'm not sure if that can be the reason in your situation.

If I got some new Information I will let you now, if you solved your problem:
please let me know.

Regards,

Klaus