Buffer I/O error

Buffer I/O error

[Kit Westneat]

Hi dm-devel,

I'm running into an issue with dm-multipath 
(device-mapper-multipath-0.4.5-27.el4_6.3) that has me confused and I 
was hoping I could get some guidance.

Here are the relevant dmesg messages:
device-mapper: dm-multipath: Failing path 8:48.
device-mapper: dm-multipath: Failing path 8:48.
device-mapper: dm-multipath: Failing path 8:48.
device-mapper: dm-multipath: Failing path 65:80.
printk: 1004 messages suppressed.
Buffer I/O error on device dm-13, logical block 1192525829
lost page write due to I/O error on dm-13

Normally when I have seen Buffer I/O errors in the past, they have been 
accompanied by SCSI errors as well, but in this case I am not getting 
anything. One odd thing is that the filesystem reported extremely slow 
writes before the buffer I/O errors occurred. I have checked and 
double-checked the storage array and everything seems fine. I am using 
directio as my path checker if that makes a difference.

I have been able to reproduce the problem consistently by attempting an 
fsck on dm-13. However, I am now doing an fsck on the underlying sd 
device and it hasn't bombed out yet with the buffer I/O errors.

Has anyone encountered these seemingly inexplicable buffer I/O errors 
before? Any advice on how to debug it?

Thanks,
Kit Westneat

i/o error due to all path failure with rdac

[Moger, Babu]

Hi,

  I am running an online/offline test. I have two paths to the controller. One is active and one is passive. When I fail (offline) the active path (sde 8:64), the Device mapper is failing passive path(sdf 8:80) as well leading to all path failure.  Any ideas or hints?

Here is output multipath -ll. I have only one lun.

[root@localhost ~]# multipath -ll
mpathie (3600a0b80000f6a7d0000cff048fed59c) dm-2 LSI,INF-01-00
[size=10G][features=1 queue_if_no_path][hwhandler=1 rdac][rw]
\_ round-robin 0 [prio=2][enabled]
 \_ 3:0:0:0 sde 8:64  [active][undef]
\_ round-robin 0 [prio=1][enabled]
 \_ 3:0:1:0 sdf 8:80  [active][undef]


Here is the detailed log.

Oct 24 16:50:50 localhost multipathd: sdf: rdac prio = 0
Oct 24 16:51:06 localhost kernel: sd 3:0:0:0: [sde] Result: hostbyte=DID_BUS_BUSY driverbyte=DRIVER_OK,SUGGEST_OK
Oct 24 16:51:06 localhost kernel: end_request: I/O error, dev sde, sector 1047072
Oct 24 16:51:06 localhost kernel: device-mapper: multipath: Failing path 8:64.
Oct 24 16:51:06 localhost multipathd: mpathie: rr_weight = 2 (controller setting)
Oct 24 16:51:06 localhost multipathd: mpathie: pgfailback = 100 (controller setting)
Oct 24 16:51:06 localhost multipathd: mpathie: no_path_retry = 10 (controller setting)
Oct 24 16:51:06 localhost multipathd: pg_timeout = NONE (internal default)
Oct 24 16:51:06 localhost multipathd: 8:64: mark as failed
Oct 24 16:51:06 localhost multipathd: uevent 'change' from '/block/dm-2'
Oct 24 16:51:06 localhost multipathd: UDEV_LOG=3
Oct 24 16:51:06 localhost multipathd: ACTION=change
Oct 24 16:51:06 localhost multipathd: DEVPATH=/block/dm-2
Oct 24 16:51:06 localhost multipathd: SUBSYSTEM=block
Oct 24 16:51:06 localhost multipathd: DM_TARGET=multipath
Oct 24 16:51:06 localhost multipathd: DM_ACTION=PATH_FAILED
Oct 24 16:51:06 localhost multipathd: DM_SEQNUM=1
Oct 24 16:51:06 localhost multipathd: DM_PATH=8:64
Oct 24 16:51:06 localhost multipathd: DM_NR_VALID_PATHS=1
Oct 24 16:51:06 localhost multipathd: DM_NAME=mpathie
Oct 24 16:51:06 localhost multipathd: DM_UUID=mpath-3600a0b80000f6a7d0000cff048fed59c
Oct 24 16:51:06 localhost multipathd: MAJOR=253
Oct 24 16:51:06 localhost multipathd: MINOR=2
Oct 24 16:51:06 localhost multipathd: DEVTYPE=disk
Oct 24 16:51:06 localhost multipathd: SEQNUM=1254
Oct 24 16:51:06 localhost multipathd: UDEVD_EVENT=1
Oct 24 16:51:06 localhost multipathd: dm-2: add map (uevent)
Oct 24 16:51:08 localhost kernel: device-mapper: multipath: Failing path 8:80.
Oct 24 16:51:08 localhost multipathd: mpathie: devmap event #3
Oct 24 16:51:08 localhost multipathd: mpathie: discover
Oct 24 16:51:08 localhost multipathd: mpathie: rr_weight = 2 (controller setting)
Oct 24 16:51:08 localhost multipathd: mpathie: pgfailback = 100 (controller setting)
Oct 24 16:51:08 localhost multipathd: mpathie: no_path_retry = 10 (controller setting)
Oct 24 16:51:08 localhost multipathd: pg_timeout = NONE (internal default)
Oct 24 16:51:08 localhost multipathd: 8:80: mark as failed
Oct 24 16:51:08 localhost multipathd: mpathie: Entering recovery mode: max_retries=10
Oct 24 16:51:08 localhost multipathd: uevent 'change' from '/block/dm-2'
Oct 24 16:51:08 localhost multipathd: UDEV_LOG=3
Oct 24 16:51:08 localhost multipathd: ACTION=change
Oct 24 16:51:08 localhost multipathd: DEVPATH=/block/dm-2
Oct 24 16:51:08 localhost multipathd: SUBSYSTEM=block
Oct 24 16:51:08 localhost multipathd: DM_TARGET=multipath
Oct 24 16:51:08 localhost multipathd: DM_ACTION=PATH_FAILED
Oct 24 16:51:08 localhost multipathd: DM_SEQNUM=2
Oct 24 16:51:08 localhost multipathd: DM_PATH=8:80
Oct 24 16:51:08 localhost multipathd: DM_NR_VALID_PATHS=0
Oct 24 16:51:08 localhost multipathd: DM_NAME=mpathie
Oct 24 16:51:08 localhost multipathd: DM_UUID=mpath-3600a0b80000f6a7d0000cff048fed59c
Oct 24 16:51:08 localhost multipathd: MAJOR=253
Oct 24 16:51:08 localhost multipathd: MINOR=2
Oct 24 16:51:08 localhost multipathd: DEVTYPE=disk
Oct 24 16:51:08 localhost multipathd: SEQNUM=1255
Oct 24 16:51:08 localhost multipathd: UDEVD_EVENT=1
Oct 24 16:51:08 localhost multipathd: dm-2: add map (uevent)
Oct 24 16:51:36 localhost kernel:  rport-3:0-2: blocked FC remote port time out: removing target and saving binding
Oct 24 16:51:36 localhost multipathd: sde: rdac checker reports path is down
Oct 24 16:51:36 localhost multipathd: sde: mask = 0x8
Oct 24 16:51:36 localhost kernel: sd 3:0:0:0: [sde] Synchronizing SCSI cache
Oct 24 16:51:36 localhost kernel: sd 3:0:0:0: [sde] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK,SUGGEST_OK
Oct 24 16:51:36 localhost kernel: scsi 3:0:0:0: rdac: Detached
Oct 24 16:51:36 localhost multipathd: uevent 'remove' from '/class/scsi_generic/sg5'
Oct 24 16:51:36 localhost multipathd: UDEV_LOG=3
Oct 24 16:51:36 localhost multipathd: ACTION=remove
Oct 24 16:51:36 localhost multipathd: DEVPATH=/class/scsi_generic/sg5
Oct 24 16:51:36 localhost multipathd: SUBSYSTEM=scsi_generic
Oct 24 16:51:36 localhost multipathd: MAJOR=21
Oct 24 16:51:36 localhost multipathd: MINOR=5
Oct 24 16:51:36 localhost multipathd: PHYSDEVPATH=/devices/pci0000:00/0000:00:02.0/0000:06:00.3/0000:0b:01.0/host3/rport-3:0-2/target3:0:0/3:0:0:0
Oct 24 16:51:36 localhost multipathd: PHYSDEVBUS=scsi
Oct 24 16:51:36 localhost multipathd: PHYSDEVDRIVER=sd
Oct 24 16:51:36 localhost multipathd: SEQNUM=1256
Oct 24 16:51:36 localhost multipathd: UDEVD_EVENT=1
Oct 24 16:51:36 localhost multipathd: DEVNAME=/dev/sg5
Oct 24 16:51:36 localhost multipathd: uevent 'remove' from '/class/scsi_device/3:0:0:0'
Oct 24 16:51:36 localhost multipathd: UDEV_LOG=3
Oct 24 16:51:36 localhost kernel: device-mapper: multipath: Failing path 8:80.
Oct 24 16:51:36 localhost multipathd: ACTION=remove
Oct 24 16:51:36 localhost UnixSmash4[9200]: 7:UnixSmash has experienced a write failure.

Thanks
Babu Moger

Re: [dm-devel] i/o error due to all path failure with rdac

[Mike Anderson]

Moger, Babu <Babu.Moger@lsi.com> wrote:
> 
> Hi,
> 
>   I am running an online/offline test. I have two paths to the controller. One is active and one is passive. When I fail (offline) the active path (sde 8:64), the Device mapper is failing passive path(sdf 8:80) as well leading to all path failure.  Any ideas or hints?
> 

What version of multipath tools and kernel are you running? If this is a
newer kernel I would have expected to see "queueing MODE_SELECT command"
during failover.

> Here is output multipath -ll. I have only one lun.
> 
> [root@localhost ~]# multipath -ll
> mpathie (3600a0b80000f6a7d0000cff048fed59c) dm-2 LSI,INF-01-00
> [size=10G][features=1 queue_if_no_path][hwhandler=1 rdac][rw]
> \_ round-robin 0 [prio=2][enabled]
>  \_ 3:0:0:0 sde 8:64  [active][undef]
> \_ round-robin 0 [prio=1][enabled]
>  \_ 3:0:1:0 sdf 8:80  [active][undef]
> 
> 
> Here is the detailed log.
> 
> Oct 24 16:50:50 localhost multipathd: sdf: rdac prio = 0
> Oct 24 16:51:06 localhost kernel: sd 3:0:0:0: [sde] Result: hostbyte=DID_BUS_BUSY driverbyte=DRIVER_OK,SUGGEST_OK
> Oct 24 16:51:06 localhost kernel: end_request: I/O error, dev sde, sector 1047072
> Oct 24 16:51:06 localhost kernel: device-mapper: multipath: Failing path 8:64.
> Oct 24 16:51:06 localhost multipathd: mpathie: rr_weight = 2 (controller setting)
> Oct 24 16:51:06 localhost multipathd: mpathie: pgfailback = 100 (controller setting)
> Oct 24 16:51:06 localhost multipathd: mpathie: no_path_retry = 10 (controller setting)
> Oct 24 16:51:06 localhost multipathd: pg_timeout = NONE (internal default)
> Oct 24 16:51:06 localhost multipathd: 8:64: mark as failed
> Oct 24 16:51:06 localhost multipathd: uevent 'change' from '/block/dm-2'
> Oct 24 16:51:06 localhost multipathd: UDEV_LOG=3
> Oct 24 16:51:06 localhost multipathd: ACTION=change
> Oct 24 16:51:06 localhost multipathd: DEVPATH=/block/dm-2
> Oct 24 16:51:06 localhost multipathd: SUBSYSTEM=block
> Oct 24 16:51:06 localhost multipathd: DM_TARGET=multipath
> Oct 24 16:51:06 localhost multipathd: DM_ACTION=PATH_FAILED
> Oct 24 16:51:06 localhost multipathd: DM_SEQNUM=1
> Oct 24 16:51:06 localhost multipathd: DM_PATH=8:64
> Oct 24 16:51:06 localhost multipathd: DM_NR_VALID_PATHS=1
> Oct 24 16:51:06 localhost multipathd: DM_NAME=mpathie
> Oct 24 16:51:06 localhost multipathd: DM_UUID=mpath-3600a0b80000f6a7d0000cff048fed59c
> Oct 24 16:51:06 localhost multipathd: MAJOR=253
> Oct 24 16:51:06 localhost multipathd: MINOR=2
> Oct 24 16:51:06 localhost multipathd: DEVTYPE=disk
> Oct 24 16:51:06 localhost multipathd: SEQNUM=1254
> Oct 24 16:51:06 localhost multipathd: UDEVD_EVENT=1
> Oct 24 16:51:06 localhost multipathd: dm-2: add map (uevent)
> Oct 24 16:51:08 localhost kernel: device-mapper: multipath: Failing path 8:80.
> Oct 24 16:51:08 localhost multipathd: mpathie: devmap event #3
> Oct 24 16:51:08 localhost multipathd: mpathie: discover
> Oct 24 16:51:08 localhost multipathd: mpathie: rr_weight = 2 (controller setting)
> Oct 24 16:51:08 localhost multipathd: mpathie: pgfailback = 100 (controller setting)
> Oct 24 16:51:08 localhost multipathd: mpathie: no_path_retry = 10 (controller setting)
> Oct 24 16:51:08 localhost multipathd: pg_timeout = NONE (internal default)
> Oct 24 16:51:08 localhost multipathd: 8:80: mark as failed
> Oct 24 16:51:08 localhost multipathd: mpathie: Entering recovery mode: max_retries=10
> Oct 24 16:51:08 localhost multipathd: uevent 'change' from '/block/dm-2'
> Oct 24 16:51:08 localhost multipathd: UDEV_LOG=3
> Oct 24 16:51:08 localhost multipathd: ACTION=change
> Oct 24 16:51:08 localhost multipathd: DEVPATH=/block/dm-2
> Oct 24 16:51:08 localhost multipathd: SUBSYSTEM=block
> Oct 24 16:51:08 localhost multipathd: DM_TARGET=multipath
> Oct 24 16:51:08 localhost multipathd: DM_ACTION=PATH_FAILED
> Oct 24 16:51:08 localhost multipathd: DM_SEQNUM=2
> Oct 24 16:51:08 localhost multipathd: DM_PATH=8:80
> Oct 24 16:51:08 localhost multipathd: DM_NR_VALID_PATHS=0
> Oct 24 16:51:08 localhost multipathd: DM_NAME=mpathie
> Oct 24 16:51:08 localhost multipathd: DM_UUID=mpath-3600a0b80000f6a7d0000cff048fed59c
> Oct 24 16:51:08 localhost multipathd: MAJOR=253
> Oct 24 16:51:08 localhost multipathd: MINOR=2
> Oct 24 16:51:08 localhost multipathd: DEVTYPE=disk
> Oct 24 16:51:08 localhost multipathd: SEQNUM=1255
> Oct 24 16:51:08 localhost multipathd: UDEVD_EVENT=1
> Oct 24 16:51:08 localhost multipathd: dm-2: add map (uevent)
> Oct 24 16:51:36 localhost kernel:  rport-3:0-2: blocked FC remote port time out: removing target and saving binding
> Oct 24 16:51:36 localhost multipathd: sde: rdac checker reports path is down
> Oct 24 16:51:36 localhost multipathd: sde: mask = 0x8
> Oct 24 16:51:36 localhost kernel: sd 3:0:0:0: [sde] Synchronizing SCSI cache
> Oct 24 16:51:36 localhost kernel: sd 3:0:0:0: [sde] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK,SUGGEST_OK
> Oct 24 16:51:36 localhost kernel: scsi 3:0:0:0: rdac: Detached
> Oct 24 16:51:36 localhost multipathd: uevent 'remove' from '/class/scsi_generic/sg5'
> Oct 24 16:51:36 localhost multipathd: UDEV_LOG=3
> Oct 24 16:51:36 localhost multipathd: ACTION=remove
> Oct 24 16:51:36 localhost multipathd: DEVPATH=/class/scsi_generic/sg5
> Oct 24 16:51:36 localhost multipathd: SUBSYSTEM=scsi_generic
> Oct 24 16:51:36 localhost multipathd: MAJOR=21
> Oct 24 16:51:36 localhost multipathd: MINOR=5
> Oct 24 16:51:36 localhost multipathd: PHYSDEVPATH=/devices/pci0000:00/0000:00:02.0/0000:06:00.3/0000:0b:01.0/host3/rport-3:0-2/target3:0:0/3:0:0:0
> Oct 24 16:51:36 localhost multipathd: PHYSDEVBUS=scsi
> Oct 24 16:51:36 localhost multipathd: PHYSDEVDRIVER=sd
> Oct 24 16:51:36 localhost multipathd: SEQNUM=1256
> Oct 24 16:51:36 localhost multipathd: UDEVD_EVENT=1
> Oct 24 16:51:36 localhost multipathd: DEVNAME=/dev/sg5
> Oct 24 16:51:36 localhost multipathd: uevent 'remove' from '/class/scsi_device/3:0:0:0'
> Oct 24 16:51:36 localhost multipathd: UDEV_LOG=3
> Oct 24 16:51:36 localhost kernel: device-mapper: multipath: Failing path 8:80.
> Oct 24 16:51:36 localhost multipathd: ACTION=remove
> Oct 24 16:51:36 localhost UnixSmash4[9200]: 7:UnixSmash has experienced a write failure.
> 
> Thanks
> Babu Moger
> 
> 
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel

-andmike
--
Michael Anderson
andmike@linux.vnet.ibm.com

RE: i/o error due to all path failure with rdac

[Moger, Babu]

I am running multipath-tools v0.4.8 (I just pulled from mainstream last week) and kernel version 2.6.27-rc7.

I am not seeing "queueing MODE_SELECT command", because this is online/offline test.  When you offline the controller the luns are automatically transferred to alt controller.

Thanks
Babu Moger

-----Original Message-----
From: dm-devel-bounces@redhat.com [mailto:dm-devel-bounces@redhat.com] On Behalf Of Mike Anderson
Sent: Thursday, October 30, 2008 12:35 PM
To: device-mapper development; Chandra Seetharaman
Cc: linux-scsi@vger.kernel.org
Subject: Re: [dm-devel] i/o error due to all path failure with rdac

Moger, Babu <Babu.Moger@lsi.com> wrote:
>
> Hi,
>
>   I am running an online/offline test. I have two paths to the controller. One is active and one is passive. When I fail (offline) the active path (sde 8:64), the Device mapper is failing passive path(sdf 8:80) as well leading to all path failure.  Any ideas or hints?
>

What version of multipath tools and kernel are you running? If this is a
newer kernel I would have expected to see "queueing MODE_SELECT command"
during failover.

> Here is output multipath -ll. I have only one lun.
>
> [root@localhost ~]# multipath -ll
> mpathie (3600a0b80000f6a7d0000cff048fed59c) dm-2 LSI,INF-01-00
> [size=10G][features=1 queue_if_no_path][hwhandler=1 rdac][rw]
> \_ round-robin 0 [prio=2][enabled]
>  \_ 3:0:0:0 sde 8:64  [active][undef]
> \_ round-robin 0 [prio=1][enabled]
>  \_ 3:0:1:0 sdf 8:80  [active][undef]
>
>
> Here is the detailed log.
>
> Oct 24 16:50:50 localhost multipathd: sdf: rdac prio = 0
> Oct 24 16:51:06 localhost kernel: sd 3:0:0:0: [sde] Result: hostbyte=DID_BUS_BUSY driverbyte=DRIVER_OK,SUGGEST_OK
> Oct 24 16:51:06 localhost kernel: end_request: I/O error, dev sde, sector 1047072
> Oct 24 16:51:06 localhost kernel: device-mapper: multipath: Failing path 8:64.
> Oct 24 16:51:06 localhost multipathd: mpathie: rr_weight = 2 (controller setting)
> Oct 24 16:51:06 localhost multipathd: mpathie: pgfailback = 100 (controller setting)
> Oct 24 16:51:06 localhost multipathd: mpathie: no_path_retry = 10 (controller setting)
> Oct 24 16:51:06 localhost multipathd: pg_timeout = NONE (internal default)
> Oct 24 16:51:06 localhost multipathd: 8:64: mark as failed
> Oct 24 16:51:06 localhost multipathd: uevent 'change' from '/block/dm-2'
> Oct 24 16:51:06 localhost multipathd: UDEV_LOG=3
> Oct 24 16:51:06 localhost multipathd: ACTION=change
> Oct 24 16:51:06 localhost multipathd: DEVPATH=/block/dm-2
> Oct 24 16:51:06 localhost multipathd: SUBSYSTEM=block
> Oct 24 16:51:06 localhost multipathd: DM_TARGET=multipath
> Oct 24 16:51:06 localhost multipathd: DM_ACTION=PATH_FAILED
> Oct 24 16:51:06 localhost multipathd: DM_SEQNUM=1
> Oct 24 16:51:06 localhost multipathd: DM_PATH=8:64
> Oct 24 16:51:06 localhost multipathd: DM_NR_VALID_PATHS=1
> Oct 24 16:51:06 localhost multipathd: DM_NAME=mpathie
> Oct 24 16:51:06 localhost multipathd: DM_UUID=mpath-3600a0b80000f6a7d0000cff048fed59c
> Oct 24 16:51:06 localhost multipathd: MAJOR=253
> Oct 24 16:51:06 localhost multipathd: MINOR=2
> Oct 24 16:51:06 localhost multipathd: DEVTYPE=disk
> Oct 24 16:51:06 localhost multipathd: SEQNUM=1254
> Oct 24 16:51:06 localhost multipathd: UDEVD_EVENT=1
> Oct 24 16:51:06 localhost multipathd: dm-2: add map (uevent)
> Oct 24 16:51:08 localhost kernel: device-mapper: multipath: Failing path 8:80.
> Oct 24 16:51:08 localhost multipathd: mpathie: devmap event #3
> Oct 24 16:51:08 localhost multipathd: mpathie: discover
> Oct 24 16:51:08 localhost multipathd: mpathie: rr_weight = 2 (controller setting)
> Oct 24 16:51:08 localhost multipathd: mpathie: pgfailback = 100 (controller setting)
> Oct 24 16:51:08 localhost multipathd: mpathie: no_path_retry = 10 (controller setting)
> Oct 24 16:51:08 localhost multipathd: pg_timeout = NONE (internal default)
> Oct 24 16:51:08 localhost multipathd: 8:80: mark as failed
> Oct 24 16:51:08 localhost multipathd: mpathie: Entering recovery mode: max_retries=10
> Oct 24 16:51:08 localhost multipathd: uevent 'change' from '/block/dm-2'
> Oct 24 16:51:08 localhost multipathd: UDEV_LOG=3
> Oct 24 16:51:08 localhost multipathd: ACTION=change
> Oct 24 16:51:08 localhost multipathd: DEVPATH=/block/dm-2
> Oct 24 16:51:08 localhost multipathd: SUBSYSTEM=block
> Oct 24 16:51:08 localhost multipathd: DM_TARGET=multipath
> Oct 24 16:51:08 localhost multipathd: DM_ACTION=PATH_FAILED
> Oct 24 16:51:08 localhost multipathd: DM_SEQNUM=2
> Oct 24 16:51:08 localhost multipathd: DM_PATH=8:80
> Oct 24 16:51:08 localhost multipathd: DM_NR_VALID_PATHS=0
> Oct 24 16:51:08 localhost multipathd: DM_NAME=mpathie
> Oct 24 16:51:08 localhost multipathd: DM_UUID=mpath-3600a0b80000f6a7d0000cff048fed59c
> Oct 24 16:51:08 localhost multipathd: MAJOR=253
> Oct 24 16:51:08 localhost multipathd: MINOR=2
> Oct 24 16:51:08 localhost multipathd: DEVTYPE=disk
> Oct 24 16:51:08 localhost multipathd: SEQNUM=1255
> Oct 24 16:51:08 localhost multipathd: UDEVD_EVENT=1
> Oct 24 16:51:08 localhost multipathd: dm-2: add map (uevent)
> Oct 24 16:51:36 localhost kernel:  rport-3:0-2: blocked FC remote port time out: removing target and saving binding
> Oct 24 16:51:36 localhost multipathd: sde: rdac checker reports path is down
> Oct 24 16:51:36 localhost multipathd: sde: mask = 0x8
> Oct 24 16:51:36 localhost kernel: sd 3:0:0:0: [sde] Synchronizing SCSI cache
> Oct 24 16:51:36 localhost kernel: sd 3:0:0:0: [sde] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK,SUGGEST_OK
> Oct 24 16:51:36 localhost kernel: scsi 3:0:0:0: rdac: Detached
> Oct 24 16:51:36 localhost multipathd: uevent 'remove' from '/class/scsi_generic/sg5'
> Oct 24 16:51:36 localhost multipathd: UDEV_LOG=3
> Oct 24 16:51:36 localhost multipathd: ACTION=remove
> Oct 24 16:51:36 localhost multipathd: DEVPATH=/class/scsi_generic/sg5
> Oct 24 16:51:36 localhost multipathd: SUBSYSTEM=scsi_generic
> Oct 24 16:51:36 localhost multipathd: MAJOR=21
> Oct 24 16:51:36 localhost multipathd: MINOR=5
> Oct 24 16:51:36 localhost multipathd: PHYSDEVPATH=/devices/pci0000:00/0000:00:02.0/0000:06:00.3/0000:0b:01.0/host3/rport-3:0-2/target3:0:0/3:0:0:0
> Oct 24 16:51:36 localhost multipathd: PHYSDEVBUS=scsi
> Oct 24 16:51:36 localhost multipathd: PHYSDEVDRIVER=sd
> Oct 24 16:51:36 localhost multipathd: SEQNUM=1256
> Oct 24 16:51:36 localhost multipathd: UDEVD_EVENT=1
> Oct 24 16:51:36 localhost multipathd: DEVNAME=/dev/sg5
> Oct 24 16:51:36 localhost multipathd: uevent 'remove' from '/class/scsi_device/3:0:0:0'
> Oct 24 16:51:36 localhost multipathd: UDEV_LOG=3
> Oct 24 16:51:36 localhost kernel: device-mapper: multipath: Failing path 8:80.
> Oct 24 16:51:36 localhost multipathd: ACTION=remove
> Oct 24 16:51:36 localhost UnixSmash4[9200]: 7:UnixSmash has experienced a write failure.
>
> Thanks
> Babu Moger
>
>
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel

-andmike
--
Michael Anderson
andmike@linux.vnet.ibm.com

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

RE: i/o error due to all path failure with rdac

[Chandra Seetharaman]

On Thu, 2008-10-30 at 13:17 -0600, Moger, Babu wrote:
> I am running multipath-tools v0.4.8 (I just pulled from mainstream last week) and kernel version 2.6.27-rc7.
> 
> I am not seeing "queueing MODE_SELECT command", because this is online/offline test.  When you offline 
> the controller the luns are automatically transferred to alt controller.

No, moving the luns to the other controller is done by the rdac hardware
handler by way of sending a MODE_SELECT to the controller.

So, we should be seeing a MODE_SELECT to the passive controller.

> 
> Thanks
> Babu Moger
> 
> -----Original Message-----
> From: dm-devel-bounces@redhat.com [mailto:dm-devel-bounces@redhat.com] On Behalf Of Mike Anderson
> Sent: Thursday, October 30, 2008 12:35 PM
> To: device-mapper development; Chandra Seetharaman
> Cc: linux-scsi@vger.kernel.org
> Subject: Re: [dm-devel] i/o error due to all path failure with rdac
> 
> Moger, Babu <Babu.Moger@lsi.com> wrote:
> >
> > Hi,
> >
> >   I am running an online/offline test. I have two paths to the controller. One is active and one is passive. When I fail (offline) the active path (sde 8:64), the Device mapper is failing passive path(sdf 8:80) as well leading to all path failure.  Any ideas or hints?
> >
> 
> What version of multipath tools and kernel are you running? If this is a
> newer kernel I would have expected to see "queueing MODE_SELECT command"
> during failover.
> 
> > Here is output multipath -ll. I have only one lun.
> >
> > [root@localhost ~]# multipath -ll
> > mpathie (3600a0b80000f6a7d0000cff048fed59c) dm-2 LSI,INF-01-00
> > [size=10G][features=1 queue_if_no_path][hwhandler=1 rdac][rw]
> > \_ round-robin 0 [prio=2][enabled]
> >  \_ 3:0:0:0 sde 8:64  [active][undef]
> > \_ round-robin 0 [prio=1][enabled]
> >  \_ 3:0:1:0 sdf 8:80  [active][undef]
> >
> >
> > Here is the detailed log.
> >
> > Oct 24 16:50:50 localhost multipathd: sdf: rdac prio = 0
> > Oct 24 16:51:06 localhost kernel: sd 3:0:0:0: [sde] Result: hostbyte=DID_BUS_BUSY driverbyte=DRIVER_OK,SUGGEST_OK
> > Oct 24 16:51:06 localhost kernel: end_request: I/O error, dev sde, sector 1047072
> > Oct 24 16:51:06 localhost kernel: device-mapper: multipath: Failing path 8:64.
> > Oct 24 16:51:06 localhost multipathd: mpathie: rr_weight = 2 (controller setting)
> > Oct 24 16:51:06 localhost multipathd: mpathie: pgfailback = 100 (controller setting)
> > Oct 24 16:51:06 localhost multipathd: mpathie: no_path_retry = 10 (controller setting)
> > Oct 24 16:51:06 localhost multipathd: pg_timeout = NONE (internal default)
> > Oct 24 16:51:06 localhost multipathd: 8:64: mark as failed
> > Oct 24 16:51:06 localhost multipathd: uevent 'change' from '/block/dm-2'
> > Oct 24 16:51:06 localhost multipathd: UDEV_LOG=3
> > Oct 24 16:51:06 localhost multipathd: ACTION=change
> > Oct 24 16:51:06 localhost multipathd: DEVPATH=/block/dm-2
> > Oct 24 16:51:06 localhost multipathd: SUBSYSTEM=block
> > Oct 24 16:51:06 localhost multipathd: DM_TARGET=multipath
> > Oct 24 16:51:06 localhost multipathd: DM_ACTION=PATH_FAILED
> > Oct 24 16:51:06 localhost multipathd: DM_SEQNUM=1
> > Oct 24 16:51:06 localhost multipathd: DM_PATH=8:64
> > Oct 24 16:51:06 localhost multipathd: DM_NR_VALID_PATHS=1
> > Oct 24 16:51:06 localhost multipathd: DM_NAME=mpathie
> > Oct 24 16:51:06 localhost multipathd: DM_UUID=mpath-3600a0b80000f6a7d0000cff048fed59c
> > Oct 24 16:51:06 localhost multipathd: MAJOR=253
> > Oct 24 16:51:06 localhost multipathd: MINOR=2
> > Oct 24 16:51:06 localhost multipathd: DEVTYPE=disk
> > Oct 24 16:51:06 localhost multipathd: SEQNUM=1254
> > Oct 24 16:51:06 localhost multipathd: UDEVD_EVENT=1
> > Oct 24 16:51:06 localhost multipathd: dm-2: add map (uevent)
> > Oct 24 16:51:08 localhost kernel: device-mapper: multipath: Failing path 8:80.
> > Oct 24 16:51:08 localhost multipathd: mpathie: devmap event #3
> > Oct 24 16:51:08 localhost multipathd: mpathie: discover
> > Oct 24 16:51:08 localhost multipathd: mpathie: rr_weight = 2 (controller setting)
> > Oct 24 16:51:08 localhost multipathd: mpathie: pgfailback = 100 (controller setting)
> > Oct 24 16:51:08 localhost multipathd: mpathie: no_path_retry = 10 (controller setting)
> > Oct 24 16:51:08 localhost multipathd: pg_timeout = NONE (internal default)
> > Oct 24 16:51:08 localhost multipathd: 8:80: mark as failed
> > Oct 24 16:51:08 localhost multipathd: mpathie: Entering recovery mode: max_retries=10
> > Oct 24 16:51:08 localhost multipathd: uevent 'change' from '/block/dm-2'
> > Oct 24 16:51:08 localhost multipathd: UDEV_LOG=3
> > Oct 24 16:51:08 localhost multipathd: ACTION=change
> > Oct 24 16:51:08 localhost multipathd: DEVPATH=/block/dm-2
> > Oct 24 16:51:08 localhost multipathd: SUBSYSTEM=block
> > Oct 24 16:51:08 localhost multipathd: DM_TARGET=multipath
> > Oct 24 16:51:08 localhost multipathd: DM_ACTION=PATH_FAILED
> > Oct 24 16:51:08 localhost multipathd: DM_SEQNUM=2
> > Oct 24 16:51:08 localhost multipathd: DM_PATH=8:80
> > Oct 24 16:51:08 localhost multipathd: DM_NR_VALID_PATHS=0
> > Oct 24 16:51:08 localhost multipathd: DM_NAME=mpathie
> > Oct 24 16:51:08 localhost multipathd: DM_UUID=mpath-3600a0b80000f6a7d0000cff048fed59c
> > Oct 24 16:51:08 localhost multipathd: MAJOR=253
> > Oct 24 16:51:08 localhost multipathd: MINOR=2
> > Oct 24 16:51:08 localhost multipathd: DEVTYPE=disk
> > Oct 24 16:51:08 localhost multipathd: SEQNUM=1255
> > Oct 24 16:51:08 localhost multipathd: UDEVD_EVENT=1
> > Oct 24 16:51:08 localhost multipathd: dm-2: add map (uevent)
> > Oct 24 16:51:36 localhost kernel:  rport-3:0-2: blocked FC remote port time out: removing target and saving binding
> > Oct 24 16:51:36 localhost multipathd: sde: rdac checker reports path is down
> > Oct 24 16:51:36 localhost multipathd: sde: mask = 0x8
> > Oct 24 16:51:36 localhost kernel: sd 3:0:0:0: [sde] Synchronizing SCSI cache
> > Oct 24 16:51:36 localhost kernel: sd 3:0:0:0: [sde] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK,SUGGEST_OK
> > Oct 24 16:51:36 localhost kernel: scsi 3:0:0:0: rdac: Detached
> > Oct 24 16:51:36 localhost multipathd: uevent 'remove' from '/class/scsi_generic/sg5'
> > Oct 24 16:51:36 localhost multipathd: UDEV_LOG=3
> > Oct 24 16:51:36 localhost multipathd: ACTION=remove
> > Oct 24 16:51:36 localhost multipathd: DEVPATH=/class/scsi_generic/sg5
> > Oct 24 16:51:36 localhost multipathd: SUBSYSTEM=scsi_generic
> > Oct 24 16:51:36 localhost multipathd: MAJOR=21
> > Oct 24 16:51:36 localhost multipathd: MINOR=5
> > Oct 24 16:51:36 localhost multipathd: PHYSDEVPATH=/devices/pci0000:00/0000:00:02.0/0000:06:00.3/0000:0b:01.0/host3/rport-3:0-2/target3:0:0/3:0:0:0
> > Oct 24 16:51:36 localhost multipathd: PHYSDEVBUS=scsi
> > Oct 24 16:51:36 localhost multipathd: PHYSDEVDRIVER=sd
> > Oct 24 16:51:36 localhost multipathd: SEQNUM=1256
> > Oct 24 16:51:36 localhost multipathd: UDEVD_EVENT=1
> > Oct 24 16:51:36 localhost multipathd: DEVNAME=/dev/sg5
> > Oct 24 16:51:36 localhost multipathd: uevent 'remove' from '/class/scsi_device/3:0:0:0'
> > Oct 24 16:51:36 localhost multipathd: UDEV_LOG=3
> > Oct 24 16:51:36 localhost kernel: device-mapper: multipath: Failing path 8:80.
> > Oct 24 16:51:36 localhost multipathd: ACTION=remove
> > Oct 24 16:51:36 localhost UnixSmash4[9200]: 7:UnixSmash has experienced a write failure.
> >
> > Thanks
> > Babu Moger
> >
> >
> > --
> > dm-devel mailing list
> > dm-devel@redhat.com
> > https://www.redhat.com/mailman/listinfo/dm-devel
> 
> -andmike
> --
> Michael Anderson
> andmike@linux.vnet.ibm.com
> 
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
> --
> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: i/o error due to all path failure with rdac

[Moger, Babu]

This is what happens in my case

When the active path is failed, the dh handler calls rdac_activate to activate the passive path. Then check_ownership is called. As you know check_ownership sends inquiry (page c9). Looking at the response this function sets the lun_state(h->lun_state) to RDAC_LUN_OWNED.

If lun_state is set to RDAC_LUN_OWNED then send_mode_select will not be called. This is what happens in my case.

PS: You are right. In case of link failures we need to transfer the luns by sending the mode select. But, if you offline (or fail the controller) the Luns are automatically transferred to alt controller. This is known behavior.


Thanks
Babu Moger

-----Original Message-----
From: Chandra Seetharaman [mailto:sekharan@us.ibm.com]
Sent: Thursday, October 30, 2008 3:03 PM
To: Moger, Babu
Cc: device-mapper development; linux-scsi@vger.kernel.org
Subject: RE: [dm-devel] i/o error due to all path failure with rdac


On Thu, 2008-10-30 at 13:17 -0600, Moger, Babu wrote:
> I am running multipath-tools v0.4.8 (I just pulled from mainstream last week) and kernel version 2.6.27-rc7.
>
> I am not seeing "queueing MODE_SELECT command", because this is online/offline test.  When you offline
> the controller the luns are automatically transferred to alt controller.

No, moving the luns to the other controller is done by the rdac hardware
handler by way of sending a MODE_SELECT to the controller.

So, we should be seeing a MODE_SELECT to the passive controller.

>
> Thanks
> Babu Moger
>
> -----Original Message-----
> From: dm-devel-bounces@redhat.com [mailto:dm-devel-bounces@redhat.com] On Behalf Of Mike Anderson
> Sent: Thursday, October 30, 2008 12:35 PM
> To: device-mapper development; Chandra Seetharaman
> Cc: linux-scsi@vger.kernel.org
> Subject: Re: [dm-devel] i/o error due to all path failure with rdac
>
> Moger, Babu <Babu.Moger@lsi.com> wrote:
> >
> > Hi,
> >
> >   I am running an online/offline test. I have two paths to the controller. One is active and one is passive. When I fail (offline) the active path (sde 8:64), the Device mapper is failing passive path(sdf 8:80) as well leading to all path failure.  Any ideas or hints?
> >
>
> What version of multipath tools and kernel are you running? If this is a
> newer kernel I would have expected to see "queueing MODE_SELECT command"
> during failover.
>
> > Here is output multipath -ll. I have only one lun.
> >
> > [root@localhost ~]# multipath -ll
> > mpathie (3600a0b80000f6a7d0000cff048fed59c) dm-2 LSI,INF-01-00
> > [size=10G][features=1 queue_if_no_path][hwhandler=1 rdac][rw]
> > \_ round-robin 0 [prio=2][enabled]
> >  \_ 3:0:0:0 sde 8:64  [active][undef]
> > \_ round-robin 0 [prio=1][enabled]
> >  \_ 3:0:1:0 sdf 8:80  [active][undef]
> >
> >
> > Here is the detailed log.
> >
> > Oct 24 16:50:50 localhost multipathd: sdf: rdac prio = 0
> > Oct 24 16:51:06 localhost kernel: sd 3:0:0:0: [sde] Result: hostbyte=DID_BUS_BUSY driverbyte=DRIVER_OK,SUGGEST_OK
> > Oct 24 16:51:06 localhost kernel: end_request: I/O error, dev sde, sector 1047072
> > Oct 24 16:51:06 localhost kernel: device-mapper: multipath: Failing path 8:64.
> > Oct 24 16:51:06 localhost multipathd: mpathie: rr_weight = 2 (controller setting)
> > Oct 24 16:51:06 localhost multipathd: mpathie: pgfailback = 100 (controller setting)
> > Oct 24 16:51:06 localhost multipathd: mpathie: no_path_retry = 10 (controller setting)
> > Oct 24 16:51:06 localhost multipathd: pg_timeout = NONE (internal default)
> > Oct 24 16:51:06 localhost multipathd: 8:64: mark as failed
> > Oct 24 16:51:06 localhost multipathd: uevent 'change' from '/block/dm-2'
> > Oct 24 16:51:06 localhost multipathd: UDEV_LOG=3
> > Oct 24 16:51:06 localhost multipathd: ACTION=change
> > Oct 24 16:51:06 localhost multipathd: DEVPATH=/block/dm-2
> > Oct 24 16:51:06 localhost multipathd: SUBSYSTEM=block
> > Oct 24 16:51:06 localhost multipathd: DM_TARGET=multipath
> > Oct 24 16:51:06 localhost multipathd: DM_ACTION=PATH_FAILED
> > Oct 24 16:51:06 localhost multipathd: DM_SEQNUM=1
> > Oct 24 16:51:06 localhost multipathd: DM_PATH=8:64
> > Oct 24 16:51:06 localhost multipathd: DM_NR_VALID_PATHS=1
> > Oct 24 16:51:06 localhost multipathd: DM_NAME=mpathie
> > Oct 24 16:51:06 localhost multipathd: DM_UUID=mpath-3600a0b80000f6a7d0000cff048fed59c
> > Oct 24 16:51:06 localhost multipathd: MAJOR=253
> > Oct 24 16:51:06 localhost multipathd: MINOR=2
> > Oct 24 16:51:06 localhost multipathd: DEVTYPE=disk
> > Oct 24 16:51:06 localhost multipathd: SEQNUM=1254
> > Oct 24 16:51:06 localhost multipathd: UDEVD_EVENT=1
> > Oct 24 16:51:06 localhost multipathd: dm-2: add map (uevent)
> > Oct 24 16:51:08 localhost kernel: device-mapper: multipath: Failing path 8:80.
> > Oct 24 16:51:08 localhost multipathd: mpathie: devmap event #3
> > Oct 24 16:51:08 localhost multipathd: mpathie: discover
> > Oct 24 16:51:08 localhost multipathd: mpathie: rr_weight = 2 (controller setting)
> > Oct 24 16:51:08 localhost multipathd: mpathie: pgfailback = 100 (controller setting)
> > Oct 24 16:51:08 localhost multipathd: mpathie: no_path_retry = 10 (controller setting)
> > Oct 24 16:51:08 localhost multipathd: pg_timeout = NONE (internal default)
> > Oct 24 16:51:08 localhost multipathd: 8:80: mark as failed
> > Oct 24 16:51:08 localhost multipathd: mpathie: Entering recovery mode: max_retries=10
> > Oct 24 16:51:08 localhost multipathd: uevent 'change' from '/block/dm-2'
> > Oct 24 16:51:08 localhost multipathd: UDEV_LOG=3
> > Oct 24 16:51:08 localhost multipathd: ACTION=change
> > Oct 24 16:51:08 localhost multipathd: DEVPATH=/block/dm-2
> > Oct 24 16:51:08 localhost multipathd: SUBSYSTEM=block
> > Oct 24 16:51:08 localhost multipathd: DM_TARGET=multipath
> > Oct 24 16:51:08 localhost multipathd: DM_ACTION=PATH_FAILED
> > Oct 24 16:51:08 localhost multipathd: DM_SEQNUM=2
> > Oct 24 16:51:08 localhost multipathd: DM_PATH=8:80
> > Oct 24 16:51:08 localhost multipathd: DM_NR_VALID_PATHS=0
> > Oct 24 16:51:08 localhost multipathd: DM_NAME=mpathie
> > Oct 24 16:51:08 localhost multipathd: DM_UUID=mpath-3600a0b80000f6a7d0000cff048fed59c
> > Oct 24 16:51:08 localhost multipathd: MAJOR=253
> > Oct 24 16:51:08 localhost multipathd: MINOR=2
> > Oct 24 16:51:08 localhost multipathd: DEVTYPE=disk
> > Oct 24 16:51:08 localhost multipathd: SEQNUM=1255
> > Oct 24 16:51:08 localhost multipathd: UDEVD_EVENT=1
> > Oct 24 16:51:08 localhost multipathd: dm-2: add map (uevent)
> > Oct 24 16:51:36 localhost kernel:  rport-3:0-2: blocked FC remote port time out: removing target and saving binding
> > Oct 24 16:51:36 localhost multipathd: sde: rdac checker reports path is down
> > Oct 24 16:51:36 localhost multipathd: sde: mask = 0x8
> > Oct 24 16:51:36 localhost kernel: sd 3:0:0:0: [sde] Synchronizing SCSI cache
> > Oct 24 16:51:36 localhost kernel: sd 3:0:0:0: [sde] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK,SUGGEST_OK
> > Oct 24 16:51:36 localhost kernel: scsi 3:0:0:0: rdac: Detached
> > Oct 24 16:51:36 localhost multipathd: uevent 'remove' from '/class/scsi_generic/sg5'
> > Oct 24 16:51:36 localhost multipathd: UDEV_LOG=3
> > Oct 24 16:51:36 localhost multipathd: ACTION=remove
> > Oct 24 16:51:36 localhost multipathd: DEVPATH=/class/scsi_generic/sg5
> > Oct 24 16:51:36 localhost multipathd: SUBSYSTEM=scsi_generic
> > Oct 24 16:51:36 localhost multipathd: MAJOR=21
> > Oct 24 16:51:36 localhost multipathd: MINOR=5
> > Oct 24 16:51:36 localhost multipathd: PHYSDEVPATH=/devices/pci0000:00/0000:00:02.0/0000:06:00.3/0000:0b:01.0/host3/rport-3:0-2/target3:0:0/3:0:0:0
> > Oct 24 16:51:36 localhost multipathd: PHYSDEVBUS=scsi
> > Oct 24 16:51:36 localhost multipathd: PHYSDEVDRIVER=sd
> > Oct 24 16:51:36 localhost multipathd: SEQNUM=1256
> > Oct 24 16:51:36 localhost multipathd: UDEVD_EVENT=1
> > Oct 24 16:51:36 localhost multipathd: DEVNAME=/dev/sg5
> > Oct 24 16:51:36 localhost multipathd: uevent 'remove' from '/class/scsi_device/3:0:0:0'
> > Oct 24 16:51:36 localhost multipathd: UDEV_LOG=3
> > Oct 24 16:51:36 localhost kernel: device-mapper: multipath: Failing path 8:80.
> > Oct 24 16:51:36 localhost multipathd: ACTION=remove
> > Oct 24 16:51:36 localhost UnixSmash4[9200]: 7:UnixSmash has experienced a write failure.
> >
> > Thanks
> > Babu Moger
> >
> >
> > --
> > dm-devel mailing list
> > dm-devel@redhat.com
> > https://www.redhat.com/mailman/listinfo/dm-devel
>
> -andmike
> --
> Michael Anderson
> andmike@linux.vnet.ibm.com
>
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
> --
> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [dm-devel] i/o error due to all path failure with rdac

[Chandra Seetharaman]

On Thu, 2008-10-30 at 14:30 -0600, Moger, Babu wrote:
> This is what happens in my case
> 
> When the active path is failed, the dh handler calls rdac_activate to activate the passive path.
> Then check_ownership is called. As you know check_ownership sends inquiry (page c9). Looking at the
> response this function sets the lun_state(h->lun_state) to RDAC_LUN_OWNED.
> 
> If lun_state is set to RDAC_LUN_OWNED then send_mode_select will not be called. This is what
> happens in my case.

Ok. Now it is clear. I thought the port was disabled.

The rport failure in the log you sent made me think that it was a port
disable. Why was there a rport failure ?

Can you add these two lines at the top of
scsi_dn_rdac.c:rdac_check_sense() function, retest and send me the log.
------------
        sdev_printk(KERN_ERR, sdev, "sense_key:%x; asc %c; ascq %x\n",
                sense_hdr->sense_key, sense_hdr->asc, sense_hdr->ascq);
-----------
I want to see if we are getting any special sense that we are not
handling.


> 
> PS: You are right. In case of link failures we need to transfer the luns by sending the mode
> select. But, if you offline (or fail the controller) the Luns are automatically transferred to
> alt controller. This is known behavior.
> 
> 
> Thanks
> Babu Moger
> 
> -----Original Message-----
> From: Chandra Seetharaman [mailto:sekharan@us.ibm.com]
> Sent: Thursday, October 30, 2008 3:03 PM
> To: Moger, Babu
> Cc: device-mapper development; linux-scsi@vger.kernel.org
> Subject: RE: [dm-devel] i/o error due to all path failure with rdac
> 
> 
> On Thu, 2008-10-30 at 13:17 -0600, Moger, Babu wrote:
> > I am running multipath-tools v0.4.8 (I just pulled from mainstream last week) and kernel version 2.6.27-rc7.
> >
> > I am not seeing "queueing MODE_SELECT command", because this is online/offline test.  When you offline
> > the controller the luns are automatically transferred to alt controller.
> 
> No, moving the luns to the other controller is done by the rdac hardware
> handler by way of sending a MODE_SELECT to the controller.
> 
> So, we should be seeing a MODE_SELECT to the passive controller.
> 
> >
> > Thanks
> > Babu Moger
> >
> > -----Original Message-----
> > From: dm-devel-bounces@redhat.com [mailto:dm-devel-bounces@redhat.com] On Behalf Of Mike Anderson
> > Sent: Thursday, October 30, 2008 12:35 PM
> > To: device-mapper development; Chandra Seetharaman
> > Cc: linux-scsi@vger.kernel.org
> > Subject: Re: [dm-devel] i/o error due to all path failure with rdac
> >
> > Moger, Babu <Babu.Moger@lsi.com> wrote:
> > >
> > > Hi,
> > >
> > >   I am running an online/offline test. I have two paths to the controller. One is active and one is passive. When I fail (offline) the active path (sde 8:64), the Device mapper is failing passive path(sdf 8:80) as well leading to all path failure.  Any ideas or hints?
> > >
> >
> > What version of multipath tools and kernel are you running? If this is a
> > newer kernel I would have expected to see "queueing MODE_SELECT command"
> > during failover.
> >
> > > Here is output multipath -ll. I have only one lun.
> > >
> > > [root@localhost ~]# multipath -ll
> > > mpathie (3600a0b80000f6a7d0000cff048fed59c) dm-2 LSI,INF-01-00
> > > [size=10G][features=1 queue_if_no_path][hwhandler=1 rdac][rw]
> > > \_ round-robin 0 [prio=2][enabled]
> > >  \_ 3:0:0:0 sde 8:64  [active][undef]
> > > \_ round-robin 0 [prio=1][enabled]
> > >  \_ 3:0:1:0 sdf 8:80  [active][undef]
> > >
> > >
> > > Here is the detailed log.
> > >
> > > Oct 24 16:50:50 localhost multipathd: sdf: rdac prio = 0
> > > Oct 24 16:51:06 localhost kernel: sd 3:0:0:0: [sde] Result: hostbyte=DID_BUS_BUSY driverbyte=DRIVER_OK,SUGGEST_OK
> > > Oct 24 16:51:06 localhost kernel: end_request: I/O error, dev sde, sector 1047072
> > > Oct 24 16:51:06 localhost kernel: device-mapper: multipath: Failing path 8:64.
> > > Oct 24 16:51:06 localhost multipathd: mpathie: rr_weight = 2 (controller setting)
> > > Oct 24 16:51:06 localhost multipathd: mpathie: pgfailback = 100 (controller setting)
> > > Oct 24 16:51:06 localhost multipathd: mpathie: no_path_retry = 10 (controller setting)
> > > Oct 24 16:51:06 localhost multipathd: pg_timeout = NONE (internal default)
> > > Oct 24 16:51:06 localhost multipathd: 8:64: mark as failed
> > > Oct 24 16:51:06 localhost multipathd: uevent 'change' from '/block/dm-2'
> > > Oct 24 16:51:06 localhost multipathd: UDEV_LOG=3
> > > Oct 24 16:51:06 localhost multipathd: ACTION=change
> > > Oct 24 16:51:06 localhost multipathd: DEVPATH=/block/dm-2
> > > Oct 24 16:51:06 localhost multipathd: SUBSYSTEM=block
> > > Oct 24 16:51:06 localhost multipathd: DM_TARGET=multipath
> > > Oct 24 16:51:06 localhost multipathd: DM_ACTION=PATH_FAILED
> > > Oct 24 16:51:06 localhost multipathd: DM_SEQNUM=1
> > > Oct 24 16:51:06 localhost multipathd: DM_PATH=8:64
> > > Oct 24 16:51:06 localhost multipathd: DM_NR_VALID_PATHS=1
> > > Oct 24 16:51:06 localhost multipathd: DM_NAME=mpathie
> > > Oct 24 16:51:06 localhost multipathd: DM_UUID=mpath-3600a0b80000f6a7d0000cff048fed59c
> > > Oct 24 16:51:06 localhost multipathd: MAJOR=253
> > > Oct 24 16:51:06 localhost multipathd: MINOR=2
> > > Oct 24 16:51:06 localhost multipathd: DEVTYPE=disk
> > > Oct 24 16:51:06 localhost multipathd: SEQNUM=1254
> > > Oct 24 16:51:06 localhost multipathd: UDEVD_EVENT=1
> > > Oct 24 16:51:06 localhost multipathd: dm-2: add map (uevent)
> > > Oct 24 16:51:08 localhost kernel: device-mapper: multipath: Failing path 8:80.
> > > Oct 24 16:51:08 localhost multipathd: mpathie: devmap event #3
> > > Oct 24 16:51:08 localhost multipathd: mpathie: discover
> > > Oct 24 16:51:08 localhost multipathd: mpathie: rr_weight = 2 (controller setting)
> > > Oct 24 16:51:08 localhost multipathd: mpathie: pgfailback = 100 (controller setting)
> > > Oct 24 16:51:08 localhost multipathd: mpathie: no_path_retry = 10 (controller setting)
> > > Oct 24 16:51:08 localhost multipathd: pg_timeout = NONE (internal default)
> > > Oct 24 16:51:08 localhost multipathd: 8:80: mark as failed
> > > Oct 24 16:51:08 localhost multipathd: mpathie: Entering recovery mode: max_retries=10
> > > Oct 24 16:51:08 localhost multipathd: uevent 'change' from '/block/dm-2'
> > > Oct 24 16:51:08 localhost multipathd: UDEV_LOG=3
> > > Oct 24 16:51:08 localhost multipathd: ACTION=change
> > > Oct 24 16:51:08 localhost multipathd: DEVPATH=/block/dm-2
> > > Oct 24 16:51:08 localhost multipathd: SUBSYSTEM=block
> > > Oct 24 16:51:08 localhost multipathd: DM_TARGET=multipath
> > > Oct 24 16:51:08 localhost multipathd: DM_ACTION=PATH_FAILED
> > > Oct 24 16:51:08 localhost multipathd: DM_SEQNUM=2
> > > Oct 24 16:51:08 localhost multipathd: DM_PATH=8:80
> > > Oct 24 16:51:08 localhost multipathd: DM_NR_VALID_PATHS=0
> > > Oct 24 16:51:08 localhost multipathd: DM_NAME=mpathie
> > > Oct 24 16:51:08 localhost multipathd: DM_UUID=mpath-3600a0b80000f6a7d0000cff048fed59c
> > > Oct 24 16:51:08 localhost multipathd: MAJOR=253
> > > Oct 24 16:51:08 localhost multipathd: MINOR=2
> > > Oct 24 16:51:08 localhost multipathd: DEVTYPE=disk
> > > Oct 24 16:51:08 localhost multipathd: SEQNUM=1255
> > > Oct 24 16:51:08 localhost multipathd: UDEVD_EVENT=1
> > > Oct 24 16:51:08 localhost multipathd: dm-2: add map (uevent)
> > > Oct 24 16:51:36 localhost kernel:  rport-3:0-2: blocked FC remote port time out: removing target and saving binding
> > > Oct 24 16:51:36 localhost multipathd: sde: rdac checker reports path is down
> > > Oct 24 16:51:36 localhost multipathd: sde: mask = 0x8
> > > Oct 24 16:51:36 localhost kernel: sd 3:0:0:0: [sde] Synchronizing SCSI cache
> > > Oct 24 16:51:36 localhost kernel: sd 3:0:0:0: [sde] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK,SUGGEST_OK
> > > Oct 24 16:51:36 localhost kernel: scsi 3:0:0:0: rdac: Detached
> > > Oct 24 16:51:36 localhost multipathd: uevent 'remove' from '/class/scsi_generic/sg5'
> > > Oct 24 16:51:36 localhost multipathd: UDEV_LOG=3
> > > Oct 24 16:51:36 localhost multipathd: ACTION=remove
> > > Oct 24 16:51:36 localhost multipathd: DEVPATH=/class/scsi_generic/sg5
> > > Oct 24 16:51:36 localhost multipathd: SUBSYSTEM=scsi_generic
> > > Oct 24 16:51:36 localhost multipathd: MAJOR=21
> > > Oct 24 16:51:36 localhost multipathd: MINOR=5
> > > Oct 24 16:51:36 localhost multipathd: PHYSDEVPATH=/devices/pci0000:00/0000:00:02.0/0000:06:00.3/0000:0b:01.0/host3/rport-3:0-2/target3:0:0/3:0:0:0
> > > Oct 24 16:51:36 localhost multipathd: PHYSDEVBUS=scsi
> > > Oct 24 16:51:36 localhost multipathd: PHYSDEVDRIVER=sd
> > > Oct 24 16:51:36 localhost multipathd: SEQNUM=1256
> > > Oct 24 16:51:36 localhost multipathd: UDEVD_EVENT=1
> > > Oct 24 16:51:36 localhost multipathd: DEVNAME=/dev/sg5
> > > Oct 24 16:51:36 localhost multipathd: uevent 'remove' from '/class/scsi_device/3:0:0:0'
> > > Oct 24 16:51:36 localhost multipathd: UDEV_LOG=3
> > > Oct 24 16:51:36 localhost kernel: device-mapper: multipath: Failing path 8:80.
> > > Oct 24 16:51:36 localhost multipathd: ACTION=remove
> > > Oct 24 16:51:36 localhost UnixSmash4[9200]: 7:UnixSmash has experienced a write failure.
> > >
> > > Thanks
> > > Babu Moger
> > >
> > >
> > > --
> > > dm-devel mailing list
> > > dm-devel@redhat.com
> > > https://www.redhat.com/mailman/listinfo/dm-devel
> >
> > -andmike
> > --
> > Michael Anderson
> > andmike@linux.vnet.ibm.com
> >
> > --
> > dm-devel mailing list
> > dm-devel@redhat.com
> > https://www.redhat.com/mailman/listinfo/dm-devel
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [dm-devel] i/o error due to all path failure with rdac

[Moger, Babu]

[-- Attachment #1: Type: text/plain, Size: 1955 bytes --]


Looks like eventually we get rport failure after the controller offline.


I have attached the messages file.  I am running raw IO.

Also, please note that I have added following lines in check_sense. This condition (quiescence) should be retried.


case UNIT_ATTENTION:
                if ((sense_hdr->asc == 0x29 && sense_hdr->ascq == 0x00) ||
                    (sense_hdr->asc == 0x8b && sense_hdr->ascq == 0x02))
                        /*
                         * Power On, Reset, or Bus Device Reset, just retry.
                         */
                        return ADD_TO_MLQUEUE;



Thanks
Babu Moger

-----Original Message-----
From: Chandra Seetharaman [mailto:sekharan@us.ibm.com]
Sent: Thursday, October 30, 2008 5:24 PM
To: Moger, Babu
Cc: device-mapper development; linux-scsi@vger.kernel.org
Subject: RE: [dm-devel] i/o error due to all path failure with rdac


On Thu, 2008-10-30 at 14:30 -0600, Moger, Babu wrote:
> This is what happens in my case
>
> When the active path is failed, the dh handler calls rdac_activate to activate the passive path.
> Then check_ownership is called. As you know check_ownership sends inquiry (page c9). Looking at the
> response this function sets the lun_state(h->lun_state) to RDAC_LUN_OWNED.
>
> If lun_state is set to RDAC_LUN_OWNED then send_mode_select will not be called. This is what
> happens in my case.

Ok. Now it is clear. I thought the port was disabled.

The rport failure in the log you sent made me think that it was a port
disable. Why was there a rport failure ?

Can you add these two lines at the top of
scsi_dn_rdac.c:rdac_check_sense() function, retest and send me the log.
------------
        sdev_printk(KERN_ERR, sdev, "sense_key:%x; asc %c; ascq %x\n",
                sense_hdr->sense_key, sense_hdr->asc, sense_hdr->ascq);
-----------
I want to see if we are getting any special sense that we are not
handling.

[-- Attachment #2: messages --]
[-- Type: application/octet-stream, Size: 108126 bytes --]

Oct 30 18:03:32 localhost syslogd 1.4.1: restart.
Oct 30 18:03:32 localhost kernel: klogd 1.4.1, log source = /proc/kmsg started.
Oct 30 18:03:32 localhost kernel: Linux version 2.6.27-rc7-babu (root@localhost.localdomain) (gcc version 4.1.2 20071124 (Red Hat 4.1.2-42)) #3 SMP Thu Oct 30 15:41:28 CDT 2008
Oct 30 18:03:32 localhost kernel: Command line: ro root=/dev/VolGroup00/LogVol00 rhgb quiet console=ttyS0,115200n8 console=tty0
Oct 30 18:03:32 localhost kernel: KERNEL supported cpus:
Oct 30 18:03:32 localhost kernel:   Intel GenuineIntel
Oct 30 18:03:32 localhost kernel:   AMD AuthenticAMD
Oct 30 18:03:32 localhost kernel:   Centaur CentaurHauls
Oct 30 18:03:32 localhost kernel: BIOS-provided physical RAM map:
Oct 30 18:03:32 localhost kernel:  BIOS-e820: 0000000000000000 - 00000000000a0000 (usable)
Oct 30 18:03:32 localhost kernel:  BIOS-e820: 0000000000100000 - 000000007ffa8000 (usable)
Oct 30 18:03:32 localhost kernel:  BIOS-e820: 000000007ffa8000 - 000000007ffb7c00 (ACPI data)
Oct 30 18:03:32 localhost kernel:  BIOS-e820: 000000007ffb7c00 - 0000000080000000 (reserved)
Oct 30 18:03:32 localhost kernel:  BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved)
Oct 30 18:03:32 localhost kernel:  BIOS-e820: 00000000fe000000 - 0000000100000000 (reserved)
Oct 30 18:03:32 localhost kernel: last_pfn = 0x7ffa8 max_arch_pfn = 0x3ffffffff
Oct 30 18:03:32 localhost kernel: init_memory_mapping
Oct 30 18:03:32 localhost kernel: last_map_addr: 7ffa8000 end: 7ffa8000
Oct 30 18:03:32 localhost kernel: RAMDISK: 37ca6000 - 37fef2e1
Oct 30 18:03:32 localhost kernel: DMI 2.4 present.
Oct 30 18:03:32 localhost kernel: ACPI: RSDP 000F25D0, 0024 (r2 DELL  )
Oct 30 18:03:32 localhost kernel: ACPI: XSDT 000F2650, 004C (r1 DELL   PE_SC3          1 DELL        1)
Oct 30 18:03:32 localhost kernel: ACPI: FACP 000F2758, 00F4 (r3 DELL   PE_SC3          1 DELL        1)
Oct 30 18:03:32 localhost kernel: ACPI: DSDT 7FFA8000, 3C9F (r1 DELL   PE_SC3          1 MSFT  100000E)
Oct 30 18:03:32 localhost kernel: ACPI: FACS 7FFB7C00, 0040
Oct 30 18:03:32 localhost kernel: ACPI: APIC 000F284C, 00D4 (r1 DELL   PE_SC3          1 DELL        1)
Oct 30 18:03:32 localhost kernel: ACPI: SPCR 000F292D, 0050 (r1 DELL   PE_SC3          1 DELL        1)
Oct 30 18:03:32 localhost kernel: ACPI: HPET 000F297D, 0038 (r1 DELL   PE_SC3          1 DELL        1)
Oct 30 18:03:32 localhost kernel: ACPI: MCFG 000F29B5, 003C (r1 DELL   PE_SC3          1 DELL        1)
Oct 30 18:03:32 localhost kernel: No NUMA configuration found
Oct 30 18:03:32 localhost kernel: Faking a node at 0000000000000000-000000007ffa8000
Oct 30 18:03:32 localhost kernel: Bootmem setup node 0 0000000000000000-000000007ffa8000
Oct 30 18:03:32 localhost kernel:   NODE_DATA [000000000000a000 - 000000000000ffff]
Oct 30 18:03:32 localhost kernel:   bootmap [0000000000010000 -  000000000001fff7] pages 10
Oct 30 18:03:32 localhost kernel: (6 early reservations) ==> bootmem [0000000000 - 007ffa8000]
Oct 30 18:03:32 localhost kernel:   #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 - 0000001000]
Oct 30 18:03:32 localhost kernel:   #1 [0000006000 - 0000008000]       TRAMPOLINE ==> [0000006000 - 0000008000]
Oct 30 18:03:32 localhost kernel:   #2 [0000200000 - 0000869d8c]    TEXT DATA BSS ==> [0000200000 - 0000869d8c]
Oct 30 18:03:32 localhost kernel:   #3 [0037ca6000 - 0037fef2e1]          RAMDISK ==> [0037ca6000 - 0037fef2e1]
Oct 30 18:03:33 localhost kernel:   #4 [000009f000 - 0000100000]    BIOS reserved ==> [000009f000 - 0000100000]
Oct 30 18:03:33 localhost kernel:   #5 [0000008000 - 000000a000]          PGTABLE ==> [0000008000 - 000000a000]
Oct 30 18:03:33 localhost kernel: found SMP MP-table at [ffff8800000fe710] 000fe710
Oct 30 18:03:33 localhost kernel: Zone PFN ranges:
Oct 30 18:03:33 localhost kernel:   DMA      0x00000000 -> 0x00001000
Oct 30 18:03:33 localhost kernel:   DMA32    0x00001000 -> 0x00100000
Oct 30 18:03:33 localhost kernel:   Normal   0x00100000 -> 0x00100000
Oct 30 18:03:33 localhost mcstransd: mcstransd starting
Oct 30 18:03:33 localhost kernel: Movable zone start PFN for each node
Oct 30 18:03:33 localhost kernel: early_node_map[2] active PFN ranges
Oct 30 18:03:33 localhost mcstransd: mcstransd initialized
Oct 30 18:03:33 localhost kernel:     0: 0x00000000 -> 0x000000a0
Oct 30 18:03:33 localhost kernel:     0: 0x00000100 -> 0x0007ffa8
Oct 30 18:03:33 localhost kernel: ACPI: PM-Timer IO Port: 0x808
Oct 30 18:03:33 localhost kernel: ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
Oct 30 18:03:33 localhost kernel: ACPI: LAPIC (acpi_id[0x02] lapic_id[0x04] enabled)
Oct 30 18:03:33 localhost kernel: ACPI: LAPIC (acpi_id[0x03] lapic_id[0x01] enabled)
Oct 30 18:03:33 localhost kernel: ACPI: LAPIC (acpi_id[0x04] lapic_id[0x05] enabled)
Oct 30 18:03:33 localhost kernel: ACPI: LAPIC (acpi_id[0x05] lapic_id[0x02] enabled)
Oct 30 18:03:33 localhost kernel: ACPI: LAPIC (acpi_id[0x06] lapic_id[0x06] enabled)
Oct 30 18:03:33 localhost kernel: ACPI: LAPIC (acpi_id[0x07] lapic_id[0x03] enabled)
Oct 30 18:03:33 localhost kernel: ACPI: LAPIC (acpi_id[0x08] lapic_id[0x07] enabled)
Oct 30 18:03:33 localhost kernel: ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
Oct 30 18:03:33 localhost kernel: ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1])
Oct 30 18:03:33 localhost kernel: ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1])
Oct 30 18:03:33 localhost kernel: ACPI: LAPIC_NMI (acpi_id[0x04] high edge lint[0x1])
Oct 30 18:03:33 localhost kernel: ACPI: LAPIC_NMI (acpi_id[0x05] high edge lint[0x1])
Oct 30 18:03:33 localhost kernel: ACPI: LAPIC_NMI (acpi_id[0x06] high edge lint[0x1])
Oct 30 18:03:33 localhost multipathd: --------start up-------- 
Oct 30 18:03:33 localhost kernel: ACPI: LAPIC_NMI (acpi_id[0x07] high edge lint[0x1])
Oct 30 18:03:33 localhost multipathd: read /etc/multipath.conf 
Oct 30 18:03:33 localhost kernel: ACPI: LAPIC_NMI (acpi_id[0x08] high edge lint[0x1])
Oct 30 18:03:33 localhost multipathd: loading /lib/multipath/libcheckdirectio.so checker 
Oct 30 18:03:33 localhost kernel: ACPI: IOAPIC (id[0x08] address[0xfec00000] gsi_base[0])
Oct 30 18:03:33 localhost multipathd: loading /lib/multipath/libprioconst.so prioritizer 
Oct 30 18:03:33 localhost kernel: IOAPIC[0]: apic_id 8, version 0, address 0xfec00000, GSI 0-23
Oct 30 18:03:33 localhost kernel: ACPI: IOAPIC (id[0x09] address[0xfec80000] gsi_base[32])
Oct 30 18:03:33 localhost multipathd: ram0: device node name blacklisted 
Oct 30 18:03:33 localhost kernel: IOAPIC[1]: apic_id 9, version 0, address 0xfec80000, GSI 32-55
Oct 30 18:03:33 localhost kernel: ACPI: IOAPIC (id[0x0a] address[0xfec83000] gsi_base[128])
Oct 30 18:03:33 localhost multipathd: ram1: device node name blacklisted 
Oct 30 18:03:33 localhost kernel: IOAPIC[2]: apic_id 10, version 0, address 0xfec83000, GSI 128-151
Oct 30 18:03:33 localhost kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
Oct 30 18:03:33 localhost multipathd: ram2: device node name blacklisted 
Oct 30 18:03:33 localhost kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
Oct 30 18:03:33 localhost multipathd: ram3: device node name blacklisted 
Oct 30 18:03:33 localhost multipathd: ram4: device node name blacklisted 
Oct 30 18:03:33 localhost kernel: Setting APIC routing to flat
Oct 30 18:03:33 localhost kernel: ACPI: HPET id: 0x8086a201 base: 0xfed00000
Oct 30 18:03:33 localhost multipathd: ram5: device node name blacklisted 
Oct 30 18:03:33 localhost kernel: Using ACPI (MADT) for SMP configuration information
Oct 30 18:03:33 localhost kernel: SMP: Allowing 8 CPUs, 0 hotplug CPUs
Oct 30 18:03:33 localhost multipathd: ram6: device node name blacklisted 
Oct 30 18:03:33 localhost kernel: Allocating PCI resources starting at 88000000 (gap: 80000000:60000000)
Oct 30 18:03:33 localhost kernel: PERCPU: Allocating 60320 bytes of per cpu data
Oct 30 18:03:33 localhost multipathd: ram7: device node name blacklisted 
Oct 30 18:03:33 localhost kernel: Built 1 zonelists in Node order, mobility grouping on.  Total pages: 515192
Oct 30 18:03:33 localhost multipathd: ram8: device node name blacklisted 
Oct 30 18:03:33 localhost kernel: Policy zone: DMA32
Oct 30 18:03:33 localhost kernel: Kernel command line: ro root=/dev/VolGroup00/LogVol00 rhgb quiet console=ttyS0,115200n8 console=tty0
Oct 30 18:03:33 localhost multipathd: ram9: device node name blacklisted 
Oct 30 18:03:33 localhost kernel: Initializing CPU#0
Oct 30 18:03:33 localhost kernel: PID hash table entries: 4096 (order: 12, 32768 bytes)
Oct 30 18:03:33 localhost multipathd: ram10: device node name blacklisted 
Oct 30 18:03:33 localhost kernel: Extended CMOS year: 2000
Oct 30 18:03:33 localhost kernel: TSC: PIT calibration confirmed by PMTIMER.
Oct 30 18:03:33 localhost multipathd: ram11: device node name blacklisted 
Oct 30 18:03:33 localhost kernel: TSC: using PMTIMER calibration value
Oct 30 18:03:33 localhost kernel: Detected 1596.040 MHz processor.
Oct 30 18:03:33 localhost multipathd: ram12: device node name blacklisted 
Oct 30 18:03:33 localhost kernel: Console: colour VGA+ 80x25
Oct 30 18:03:33 localhost kernel: console [tty0] enabled
Oct 30 18:03:33 localhost multipathd: ram13: device node name blacklisted 
Oct 30 18:03:33 localhost kernel: console [ttyS0] enabled
Oct 30 18:03:33 localhost kernel: Checking aperture...
Oct 30 18:03:33 localhost multipathd: ram14: device node name blacklisted 
Oct 30 18:03:33 localhost kernel: No AGP bridge found
Oct 30 18:03:33 localhost kernel: Memory: 2056948k/2096800k available (2589k kernel code, 39468k reserved, 1423k data, 348k init)
Oct 30 18:03:33 localhost multipathd: ram15: device node name blacklisted 
Oct 30 18:03:33 localhost multipathd: sda: not found in pathvec 
Oct 30 18:03:33 localhost kernel: Calibrating delay loop (skipped), value calculated using timer frequency.. 3192.08 BogoMIPS (lpj=1596040)
Oct 30 18:03:33 localhost multipathd: sda: mask = 0x1f 
Oct 30 18:03:33 localhost kernel: Security Framework initialized
Oct 30 18:03:33 localhost multipathd: sda: dev_t = 8:0 
Oct 30 18:03:33 localhost kernel: SELinux:  Initializing.
Oct 30 18:03:33 localhost multipathd: sda: size = 32768000 
Oct 30 18:03:33 localhost multipathd: sda: subsystem = scsi 
Oct 30 18:03:33 localhost kernel: Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes)
Oct 30 18:03:33 localhost multipathd: sda: vendor = DELL 
Oct 30 18:03:33 localhost kernel: Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes)
Oct 30 18:03:33 localhost multipathd: sda: product = PERC 5/i 
Oct 30 18:03:33 localhost kernel: Mount-cache hash table entries: 256
Oct 30 18:03:33 localhost multipathd: sda: rev = 1.03 
Oct 30 18:03:33 localhost kernel: CPU: L1 I cache: 32K, L1 D cache: 32K
Oct 30 18:03:33 localhost multipathd: sda: h:b:t:l = 0:2:0:0 
Oct 30 18:03:33 localhost kernel: CPU: L2 cache: 4096K
Oct 30 18:03:33 localhost multipathd: sda: serial = 0061b6d01202d35a10000784c630c201 
Oct 30 18:03:33 localhost kernel: CPU 0/0 -> Node 0
Oct 30 18:03:33 localhost multipathd: sda: prio = const (config file default) 
Oct 30 18:03:33 localhost kernel: CPU: Physical Processor ID: 0
Oct 30 18:03:33 localhost multipathd: sda: const prio = 1 
Oct 30 18:03:33 localhost kernel: CPU: Processor Core ID: 0
Oct 30 18:03:33 localhost multipathd: sda: getuid = /lib/udev/scsi_id -g -u -d /dev/%n (config file default) 
Oct 30 18:03:33 localhost kernel: CPU0: Thermal monitoring enabled (TM1)
Oct 30 18:03:33 localhost multipathd: error calling out /lib/udev/scsi_id -g -u -d /dev/sda 
Oct 30 18:03:33 localhost kernel: using mwait in idle threads.
Oct 30 18:03:33 localhost kernel: ACPI: Core revision 20080609
Oct 30 18:03:33 localhost multipathd: sdb: not found in pathvec 
Oct 30 18:03:33 localhost kernel: ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
Oct 30 18:03:34 localhost multipathd: sdb: mask = 0x1f 
Oct 30 18:03:34 localhost kernel: CPU0: Intel(R) Xeon(R) CPU           E5310  @ 1.60GHz stepping 0b
Oct 30 18:03:34 localhost multipathd: sdb: dev_t = 8:16 
Oct 30 18:03:34 localhost kernel: Using local APIC timer interrupts.
Oct 30 18:03:34 localhost multipathd: sdb: size = 24576000 
Oct 30 18:03:34 localhost multipathd: sdb: subsystem = scsi 
Oct 30 18:03:34 localhost kernel: Detected 16.625 MHz APIC timer.
Oct 30 18:03:34 localhost multipathd: sdb: vendor = DELL 
Oct 30 18:03:34 localhost kernel: Booting processor 1/4 ip 6000
Oct 30 18:03:34 localhost multipathd: sdb: product = PERC 5/i 
Oct 30 18:03:34 localhost kernel: Initializing CPU#1
Oct 30 18:03:34 localhost multipathd: sdb: rev = 1.03 
Oct 30 18:03:34 localhost kernel: Calibrating delay using timer specific routine.. 3192.09 BogoMIPS (lpj=1596045)
Oct 30 18:03:34 localhost multipathd: sdb: h:b:t:l = 0:2:1:0 
Oct 30 18:03:34 localhost kernel: CPU: L1 I cache: 32K, L1 D cache: 32K
Oct 30 18:03:34 localhost multipathd: sdb: serial = 002df28aea23d35a10000784c630c201 
Oct 30 18:03:34 localhost kernel: CPU: L2 cache: 4096K
Oct 30 18:03:34 localhost multipathd: sdb: prio = const (config file default) 
Oct 30 18:03:34 localhost kernel: CPU 1/4 -> Node 0
Oct 30 18:03:34 localhost rpc.statd[7907]: Version 1.0.9 Starting
Oct 30 18:03:34 localhost multipathd: sdb: const prio = 1 
Oct 30 18:03:34 localhost kernel: CPU: Physical Processor ID: 1
Oct 30 18:03:34 localhost multipathd: sdb: getuid = /lib/udev/scsi_id -g -u -d /dev/%n (config file default) 
Oct 30 18:03:34 localhost kernel: CPU: Processor Core ID: 0
Oct 30 18:03:34 localhost multipathd: error calling out /lib/udev/scsi_id -g -u -d /dev/sdb 
Oct 30 18:03:34 localhost rpc.statd[7907]: statd running as root. chown /var/lib/nfs/statd/sm to choose different user 
Oct 30 18:03:34 localhost kernel: CPU1: Thermal monitoring enabled (TM2)
Oct 30 18:03:34 localhost kernel: CPU1: Intel(R) Xeon(R) CPU           E5310  @ 1.60GHz stepping 0b
Oct 30 18:03:34 localhost multipathd: sdc: not found in pathvec 
Oct 30 18:03:34 localhost kernel: checking TSC synchronization [CPU#0 -> CPU#1]: passed.
Oct 30 18:03:34 localhost multipathd: sdc: mask = 0x1f 
Oct 30 18:03:34 localhost kernel: Booting processor 2/1 ip 6000
Oct 30 18:03:34 localhost multipathd: sdc: dev_t = 8:32 
Oct 30 18:03:34 localhost kernel: Initializing CPU#2
Oct 30 18:03:34 localhost multipathd: sdc: size = 24576000 
Oct 30 18:03:34 localhost kernel: Calibrating delay using timer specific routine.. 3192.09 BogoMIPS (lpj=1596046)
Oct 30 18:03:34 localhost multipathd: sdc: subsystem = scsi 
Oct 30 18:03:34 localhost kernel: CPU: L1 I cache: 32K, L1 D cache: 32K
Oct 30 18:03:34 localhost multipathd: sdc: vendor = DELL 
Oct 30 18:03:34 localhost kernel: CPU: L2 cache: 4096K
Oct 30 18:03:34 localhost multipathd: sdc: product = PERC 5/i 
Oct 30 18:03:34 localhost kernel: CPU 2/1 -> Node 0
Oct 30 18:03:34 localhost multipathd: sdc: rev = 1.03 
Oct 30 18:03:34 localhost kernel: CPU: Physical Processor ID: 0
Oct 30 18:03:34 localhost multipathd: sdc: h:b:t:l = 0:2:2:0 
Oct 30 18:03:34 localhost kernel: CPU: Processor Core ID: 1
Oct 30 18:03:34 localhost multipathd: sdc: serial = 004ed24dd236d35a10000784c630c201 
Oct 30 18:03:34 localhost kernel: CPU2: Thermal monitoring enabled (TM2)
Oct 30 18:03:34 localhost multipathd: sdc: prio = const (config file default) 
Oct 30 18:03:34 localhost kernel: CPU2: Intel(R) Xeon(R) CPU           E5310  @ 1.60GHz stepping 0b
Oct 30 18:03:34 localhost multipathd: sdc: const prio = 1 
Oct 30 18:03:34 localhost kernel: checking TSC synchronization [CPU#0 -> CPU#2]: passed.
Oct 30 18:03:34 localhost multipathd: sdc: getuid = /lib/udev/scsi_id -g -u -d /dev/%n (config file default) 
Oct 30 18:03:34 localhost kernel: Booting processor 3/5 ip 6000
Oct 30 18:03:34 localhost multipathd: error calling out /lib/udev/scsi_id -g -u -d /dev/sdc 
Oct 30 18:03:35 localhost kernel: Initializing CPU#3
Oct 30 18:03:35 localhost kernel: Calibrating delay using timer specific routine.. 3192.12 BogoMIPS (lpj=1596060)
Oct 30 18:03:35 localhost multipathd: sdd: not found in pathvec 
Oct 30 18:03:35 localhost hcid[7990]: Bluetooth HCI daemon
Oct 30 18:03:35 localhost kernel: CPU: L1 I cache: 32K, L1 D cache: 32K
Oct 30 18:03:35 localhost multipathd: sdd: mask = 0x1f 
Oct 30 18:03:35 localhost kernel: CPU: L2 cache: 4096K
Oct 30 18:03:35 localhost hcid[7990]: Register path:/org/bluez fallback:1
Oct 30 18:03:35 localhost multipathd: sdd: dev_t = 8:48 
Oct 30 18:03:35 localhost kernel: CPU 3/5 -> Node 0
Oct 30 18:03:35 localhost sdpd[7994]: Bluetooth SDP daemon 
Oct 30 18:03:35 localhost multipathd: sdd: size = 24576000 
Oct 30 18:03:35 localhost kernel: CPU: Physical Processor ID: 1
Oct 30 18:03:35 localhost multipathd: sdd: subsystem = scsi 
Oct 30 18:03:35 localhost kernel: CPU: Processor Core ID: 1
Oct 30 18:03:35 localhost multipathd: sdd: vendor = DELL 
Oct 30 18:03:35 localhost kernel: CPU3: Thermal monitoring enabled (TM2)
Oct 30 18:03:35 localhost multipathd: sdd: product = PERC 5/i 
Oct 30 18:03:35 localhost kernel: CPU3: Intel(R) Xeon(R) CPU           E5310  @ 1.60GHz stepping 0b
Oct 30 18:03:35 localhost multipathd: sdd: rev = 1.03 
Oct 30 18:03:35 localhost kernel: checking TSC synchronization [CPU#0 -> CPU#3]: passed.
Oct 30 18:03:35 localhost multipathd: sdd: h:b:t:l = 0:2:3:0 
Oct 30 18:03:35 localhost kernel: Booting processor 4/2 ip 6000
Oct 30 18:03:35 localhost multipathd: sdd: serial = 001d728eb54dd35a10000784c630c201 
Oct 30 18:03:35 localhost kernel: Initializing CPU#4
Oct 30 18:03:35 localhost multipathd: sdd: prio = const (config file default) 
Oct 30 18:03:35 localhost pcscd: pcscdaemon.c:507:main() pcsc-lite 1.4.4 daemon ready.
Oct 30 18:03:35 localhost kernel: Calibrating delay using timer specific routine.. 3192.04 BogoMIPS (lpj=1596024)
Oct 30 18:03:35 localhost multipathd: sdd: const prio = 1 
Oct 30 18:03:35 localhost kernel: CPU: L1 I cache: 32K, L1 D cache: 32K
Oct 30 18:03:35 localhost multipathd: sdd: getuid = /lib/udev/scsi_id -g -u -d /dev/%n (config file default) 
Oct 30 18:03:35 localhost kernel: CPU: L2 cache: 4096K
Oct 30 18:03:35 localhost multipathd: error calling out /lib/udev/scsi_id -g -u -d /dev/sdd 
Oct 30 18:03:35 localhost kernel: CPU 4/2 -> Node 0
Oct 30 18:03:35 localhost kernel: CPU: Physical Processor ID: 0
Oct 30 18:03:35 localhost multipathd: sde: not found in pathvec 
Oct 30 18:03:35 localhost kernel: CPU: Processor Core ID: 2
Oct 30 18:03:35 localhost multipathd: sde: mask = 0x1f 
Oct 30 18:03:35 localhost kernel: CPU4: Thermal monitoring enabled (TM2)
Oct 30 18:03:35 localhost multipathd: sde: dev_t = 8:64 
Oct 30 18:03:35 localhost kernel: CPU4: Intel(R) Xeon(R) CPU           E5310  @ 1.60GHz stepping 0b
Oct 30 18:03:35 localhost multipathd: sde: size = 20971520 
Oct 30 18:03:35 localhost kernel: checking TSC synchronization [CPU#0 -> CPU#4]: passed.
Oct 30 18:03:35 localhost multipathd: sde: subsystem = scsi 
Oct 30 18:03:35 localhost kernel: Booting processor 5/6 ip 6000
Oct 30 18:03:35 localhost multipathd: sde: vendor = LSI 
Oct 30 18:03:35 localhost kernel: Initializing CPU#5
Oct 30 18:03:35 localhost multipathd: sde: product = INF-01-00 
Oct 30 18:03:35 localhost kernel: Calibrating delay using timer specific routine.. 3192.07 BogoMIPS (lpj=1596037)
Oct 30 18:03:35 localhost pcscd: hotplug_libusb.c:402:HPEstablishUSBNotifications() Driver ifd-ccid.bundle does not support IFD_GENERATE_HOTPLUG. Using active polling instead.
Oct 30 18:03:35 localhost multipathd: sde: rev = 9660 
Oct 30 18:03:35 localhost kernel: CPU: L1 I cache: 32K, L1 D cache: 32K
Oct 30 18:03:35 localhost hidd[8104]: Bluetooth HID daemon
Oct 30 18:03:35 localhost pcscd: hotplug_libusb.c:411:HPEstablishUSBNotifications() Polling forced every 1 second(s)
Oct 30 18:03:35 localhost multipathd: sde: h:b:t:l = 3:0:0:0 
Oct 30 18:03:35 localhost kernel: CPU: L2 cache: 4096K
Oct 30 18:03:35 localhost multipathd: sde: tgt_node_name = 0x200400a0b80f6a7 
Oct 30 18:03:35 localhost kernel: CPU 5/6 -> Node 0
Oct 30 18:03:35 localhost multipathd: sde: serial = 1T30946903       
Oct 30 18:03:35 localhost kernel: CPU: Physical Processor ID: 1
Oct 30 18:03:35 localhost multipathd: loading /lib/multipath/libpriordac.so prioritizer 
Oct 30 18:03:35 localhost kernel: CPU: Processor Core ID: 2
Oct 30 18:03:35 localhost multipathd: sde: prio = rdac (controller setting) 
Oct 30 18:03:35 localhost kernel: CPU5: Thermal monitoring enabled (TM2)
Oct 30 18:03:35 localhost multipathd: sde: rdac prio = 0 
Oct 30 18:03:35 localhost kernel: CPU5: Intel(R) Xeon(R) CPU           E5310  @ 1.60GHz stepping 0b
Oct 30 18:03:35 localhost multipathd: sde: getuid = /sbin/scsi_id -g -u -s /block/%n (controller setting) 
Oct 30 18:03:35 localhost kernel: checking TSC synchronization [CPU#0 -> CPU#5]: passed.
Oct 30 18:03:35 localhost multipathd: sde: uid = 3600a0b80000f6a7d0000cff048fed59c (callout) 
Oct 30 18:03:35 localhost kernel: Booting processor 6/3 ip 6000
Oct 30 18:03:35 localhost automount[8157]: lookup_read_master: lookup(nisplus): couldn't locate nis+ table auto.master
Oct 30 18:03:35 localhost kernel: Initializing CPU#6
Oct 30 18:03:35 localhost multipathd: sdf: not found in pathvec 
Oct 30 18:03:36 localhost kernel: Calibrating delay using timer specific routine.. 3192.03 BogoMIPS (lpj=1596019)
Oct 30 18:03:36 localhost multipathd: sdf: mask = 0x1f 
Oct 30 18:03:36 localhost kernel: CPU: L1 I cache: 32K, L1 D cache: 32K
Oct 30 18:03:36 localhost multipathd: sdf: dev_t = 8:80 
Oct 30 18:03:36 localhost kernel: CPU: L2 cache: 4096K
Oct 30 18:03:36 localhost multipathd: sdf: size = 20971520 
Oct 30 18:03:36 localhost kernel: CPU 6/3 -> Node 0
Oct 30 18:03:36 localhost multipathd: sdf: subsystem = scsi 
Oct 30 18:03:36 localhost kernel: CPU: Physical Processor ID: 0
Oct 30 18:03:36 localhost multipathd: sdf: vendor = LSI 
Oct 30 18:03:36 localhost kernel: CPU: Processor Core ID: 3
Oct 30 18:03:36 localhost multipathd: sdf: product = INF-01-00 
Oct 30 18:03:36 localhost kernel: CPU6: Thermal monitoring enabled (TM2)
Oct 30 18:03:36 localhost multipathd: sdf: rev = 9660 
Oct 30 18:03:36 localhost hpiod: 1.6.7 accepting connections at 2208... 
Oct 30 18:03:36 localhost kernel: CPU6: Intel(R) Xeon(R) CPU           E5310  @ 1.60GHz stepping 0b
Oct 30 18:03:36 localhost multipathd: sdf: h:b:t:l = 3:0:1:0 
Oct 30 18:03:36 localhost kernel: checking TSC synchronization [CPU#0 -> CPU#6]: passed.
Oct 30 18:03:36 localhost multipathd: sdf: tgt_node_name = 0x200400a0b80f6a7 
Oct 30 18:03:36 localhost kernel: Booting processor 7/7 ip 6000
Oct 30 18:03:36 localhost multipathd: sdf: serial = 1T30636438       
Oct 30 18:03:36 localhost kernel: Initializing CPU#7
Oct 30 18:03:36 localhost multipathd: sdf: prio = rdac (controller setting) 
Oct 30 18:03:36 localhost kernel: Calibrating delay using timer specific routine.. 3192.07 BogoMIPS (lpj=1596036)
Oct 30 18:03:36 localhost multipathd: sdf: rdac prio = 3 
Oct 30 18:03:36 localhost kernel: CPU: L1 I cache: 32K, L1 D cache: 32K
Oct 30 18:03:36 localhost multipathd: sdf: getuid = /sbin/scsi_id -g -u -s /block/%n (controller setting) 
Oct 30 18:03:36 localhost kernel: CPU: L2 cache: 4096K
Oct 30 18:03:36 localhost multipathd: sdf: uid = 3600a0b80000f6a7d0000cff048fed59c (callout) 
Oct 30 18:03:36 localhost kernel: CPU 7/7 -> Node 0
Oct 30 18:03:36 localhost kernel: CPU: Physical Processor ID: 1
Oct 30 18:03:36 localhost multipathd: dm-0: device node name blacklisted 
Oct 30 18:03:36 localhost kernel: CPU: Processor Core ID: 3
Oct 30 18:03:36 localhost kernel: CPU7: Thermal monitoring enabled (TM2)
Oct 30 18:03:36 localhost multipathd: dm-1: device node name blacklisted 
Oct 30 18:03:36 localhost kernel: CPU7: Intel(R) Xeon(R) CPU           E5310  @ 1.60GHz stepping 0b
Oct 30 18:03:36 localhost kernel: checking TSC synchronization [CPU#0 -> CPU#7]: passed.
Oct 30 18:03:36 localhost multipathd: hda: device node name blacklisted 
Oct 30 18:03:36 localhost kernel: Brought up 8 CPUs
Oct 30 18:03:36 localhost kernel: Total of 8 processors activated (25536.61 BogoMIPS).
Oct 30 18:03:36 localhost multipathd: md0: device node name blacklisted 
Oct 30 18:03:36 localhost multipathd: dm-2: device node name blacklisted 
Oct 30 18:03:36 localhost multipathd: mpathie: discover 
Oct 30 18:03:36 localhost multipathd: uevent 'remove' from '/class/fc_remote_ports/rport-3:0-6' 
Oct 30 18:03:36 localhost multipathd: UDEV_LOG=3 
Oct 30 18:03:36 localhost multipathd: ACTION=remove 
Oct 30 18:03:36 localhost multipathd: DEVPATH=/class/fc_remote_ports/rport-3:0-6 
Oct 30 18:03:36 localhost multipathd: SUBSYSTEM=fc_remote_ports 
Oct 30 18:03:36 localhost multipathd: PHYSDEVPATH=/devices/pci0000:00/0000:00:02.0/0000:06:00.3/0000:0b:01.0 
Oct 30 18:03:36 localhost multipathd: PHYSDEVBUS=pci 
Oct 30 18:03:36 localhost multipathd: PHYSDEVDRIVER=lpfc 
Oct 30 18:03:36 localhost multipathd: SEQNUM=1190 
Oct 30 18:03:36 localhost multipathd: UDEVD_EVENT=1 
Oct 30 18:03:36 localhost multipathd: uevent 'add' from '/class/bluetooth' 
Oct 30 18:03:36 localhost multipathd: UDEV_LOG=3 
Oct 30 18:03:36 localhost multipathd: ACTION=add 
Oct 30 18:03:36 localhost multipathd: DEVPATH=/class/bluetooth 
Oct 30 18:03:36 localhost multipathd: SUBSYSTEM=class 
Oct 30 18:03:36 localhost multipathd: SEQNUM=1192 
Oct 30 18:03:36 localhost multipathd: UDEVD_EVENT=1 
Oct 30 18:03:36 localhost multipathd: uevent 'add' from '/bus/i2c' 
Oct 30 18:03:36 localhost multipathd: UDEV_LOG=3 
Oct 30 18:03:36 localhost multipathd: ACTION=add 
Oct 30 18:03:36 localhost multipathd: DEVPATH=/bus/i2c 
Oct 30 18:03:36 localhost multipathd: SUBSYSTEM=bus 
Oct 30 18:03:36 localhost multipathd: SEQNUM=1197 
Oct 30 18:03:36 localhost multipathd: UDEVD_EVENT=1 
Oct 30 18:03:36 localhost multipathd: uevent 'add' from '/class/i2c-adapter' 
Oct 30 18:03:36 localhost multipathd: UDEV_LOG=3 
Oct 30 18:03:36 localhost multipathd: ACTION=add 
Oct 30 18:03:36 localhost multipathd: DEVPATH=/class/i2c-adapter 
Oct 30 18:03:36 localhost multipathd: SUBSYSTEM=class 
Oct 30 18:03:36 localhost multipathd: SEQNUM=1198 
Oct 30 18:03:36 localhost multipathd: UDEVD_EVENT=1 
Oct 30 18:03:36 localhost multipathd: uevent 'add' from '/class/i2c-dev' 
Oct 30 18:03:36 localhost multipathd: UDEV_LOG=3 
Oct 30 18:03:36 localhost multipathd: ACTION=add 
Oct 30 18:03:36 localhost multipathd: DEVPATH=/class/i2c-dev 
Oct 30 18:03:36 localhost multipathd: SUBSYSTEM=class 
Oct 30 18:03:36 localhost multipathd: SEQNUM=1201 
Oct 30 18:03:36 localhost multipathd: UDEVD_EVENT=1 
Oct 30 18:03:36 localhost kernel: net_namespace: 1488 bytes
Oct 30 18:03:36 localhost kernel: NET: Registered protocol family 16
Oct 30 18:03:36 localhost kernel: No dock devices found.
Oct 30 18:03:36 localhost kernel: ACPI: bus type pci registered
Oct 30 18:03:36 localhost kernel: PCI: MCFG configuration 0: base e0000000 segment 0 buses 0 - 255
Oct 30 18:03:36 localhost kernel: PCI: MCFG area at e0000000 reserved in E820
Oct 30 18:03:36 localhost kernel: PCI: Using MMCONFIG at e0000000 - efffffff
Oct 30 18:03:36 localhost kernel: PCI: Using configuration type 1 for base access
Oct 30 18:03:36 localhost kernel: PCI: Dell PowerEdge 2900 detected, enabling pci=bfsort.
Oct 30 18:03:36 localhost kernel: ACPI: Interpreter enabled
Oct 30 18:03:36 localhost kernel: ACPI: (supports S0 S5)
Oct 30 18:03:36 localhost kernel: ACPI: Using IOAPIC for interrupt routing
Oct 30 18:03:36 localhost kernel: ACPI: PCI Root Bridge [PCI0] (0000:00)
Oct 30 18:03:36 localhost kernel: pci 0000:00:00.0: PME# supported from D0 D3hot D3cold
Oct 30 18:03:36 localhost kernel: pci 0000:00:00.0: PME# disabled
Oct 30 18:03:36 localhost kernel: pci 0000:00:02.0: PME# supported from D0 D3hot D3cold
Oct 30 18:03:36 localhost kernel: pci 0000:00:02.0: PME# disabled
Oct 30 18:03:36 localhost kernel: pci 0000:00:03.0: PME# supported from D0 D3hot D3cold
Oct 30 18:03:36 localhost kernel: pci 0000:00:03.0: PME# disabled
Oct 30 18:03:36 localhost kernel: pci 0000:00:04.0: PME# supported from D0 D3hot D3cold
Oct 30 18:03:36 localhost kernel: pci 0000:00:04.0: PME# disabled
Oct 30 18:03:36 localhost kernel: pci 0000:00:05.0: PME# supported from D0 D3hot D3cold
Oct 30 18:03:36 localhost kernel: pci 0000:00:05.0: PME# disabled
Oct 30 18:03:36 localhost kernel: pci 0000:00:06.0: PME# supported from D0 D3hot D3cold
Oct 30 18:03:36 localhost kernel: pci 0000:00:06.0: PME# disabled
Oct 30 18:03:36 localhost kernel: pci 0000:00:07.0: PME# supported from D0 D3hot D3cold
Oct 30 18:03:36 localhost kernel: pci 0000:00:07.0: PME# disabled
Oct 30 18:03:36 localhost kernel: pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold
Oct 30 18:03:36 localhost kernel: pci 0000:00:1c.0: PME# disabled
Oct 30 18:03:36 localhost kernel: pci 0000:00:1d.7: PME# supported from D0 D3hot D3cold
Oct 30 18:03:36 localhost kernel: pci 0000:00:1d.7: PME# disabled
Oct 30 18:03:37 localhost kernel: pci 0000:06:00.0: PME# supported from D0 D3hot D3cold
Oct 30 18:03:37 localhost kernel: pci 0000:06:00.0: PME# disabled
Oct 30 18:03:37 localhost kernel: pci 0000:06:00.3: PME# supported from D0 D3hot D3cold
Oct 30 18:03:37 localhost kernel: pci 0000:06:00.3: PME# disabled
Oct 30 18:03:37 localhost kernel: pci 0000:07:00.0: PME# supported from D0 D3hot D3cold
Oct 30 18:03:37 localhost kernel: pci 0000:07:00.0: PME# disabled
Oct 30 18:03:37 localhost kernel: pci 0000:07:01.0: PME# supported from D0 D3hot D3cold
Oct 30 18:03:37 localhost kernel: pci 0000:07:01.0: PME# disabled
Oct 30 18:03:37 localhost kernel: pci 0000:08:00.0: PME# supported from D0 D3hot D3cold
Oct 30 18:03:37 localhost kernel: pci 0000:08:00.0: PME# disabled
Oct 30 18:03:37 localhost kernel: pci 0000:09:00.0: PME# supported from D3hot D3cold
Oct 30 18:03:37 localhost kernel: pci 0000:09:00.0: PME# disabled
Oct 30 18:03:37 localhost kernel: pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
Oct 30 18:03:37 localhost kernel: pci 0000:01:00.0: PME# disabled
Oct 30 18:03:37 localhost kernel: pci 0000:01:00.2: PME# supported from D0 D3hot D3cold
Oct 30 18:03:37 localhost kernel: pci 0000:01:00.2: PME# disabled
Oct 30 18:03:37 localhost kernel: pci 0000:04:00.0: PME# supported from D0 D3hot D3cold
Oct 30 18:03:37 localhost kernel: pci 0000:04:00.0: PME# disabled
Oct 30 18:03:37 localhost kernel: pci 0000:05:00.0: PME# supported from D3hot D3cold
Oct 30 18:03:37 localhost kernel: pci 0000:05:00.0: PME# disabled
Oct 30 18:03:37 localhost kernel: pci 0000:00:1e.0: transparent bridge
Oct 30 18:03:37 localhost kernel: ACPI: PCI Interrupt Link [LK00] (IRQs 3 4 *5 6 7 10 11 12)
Oct 30 18:03:37 localhost kernel: ACPI: PCI Interrupt Link [LK01] (IRQs 3 4 5 *6 7 10 11 12)
Oct 30 18:03:37 localhost kernel: ACPI: PCI Interrupt Link [LK02] (IRQs 3 4 5 *6 7 10 11 12)
Oct 30 18:03:37 localhost kernel: ACPI: PCI Interrupt Link [LK03] (IRQs 3 4 5 6 7 *10 11 12)
Oct 30 18:03:37 localhost kernel: ACPI: PCI Interrupt Link [LK04] (IRQs 3 4 5 6 7 10 *11 12)
Oct 30 18:03:37 localhost kernel: ACPI: PCI Interrupt Link [LK05] (IRQs 3 4 5 6 7 10 *11 12)
Oct 30 18:03:37 localhost kernel: ACPI: PCI Interrupt Link [LK06] (IRQs 3 4 5 6 7 10 11 12) *0, disabled.
Oct 30 18:03:37 localhost kernel: ACPI: PCI Interrupt Link [LK07] (IRQs 3 4 5 6 7 10 11 12) *0, disabled.
Oct 30 18:03:37 localhost kernel: Linux Plug and Play Support v0.97 (c) Adam Belay
Oct 30 18:03:37 localhost kernel: pnp: PnP ACPI init
Oct 30 18:03:37 localhost kernel: ACPI: bus type pnp registered
Oct 30 18:03:37 localhost kernel: pnp: PnP ACPI: found 12 devices
Oct 30 18:03:37 localhost kernel: ACPI: ACPI bus type pnp unregistered
Oct 30 18:03:37 localhost kernel: usbcore: registered new interface driver usbfs
Oct 30 18:03:37 localhost kernel: usbcore: registered new interface driver hub
Oct 30 18:03:37 localhost kernel: usbcore: registered new device driver usb
Oct 30 18:03:37 localhost kernel: PCI: Using ACPI for IRQ routing
Oct 30 18:03:37 localhost kernel: NetLabel: Initializing
Oct 30 18:03:37 localhost kernel: NetLabel:  domain hash size = 128
Oct 30 18:03:37 localhost kernel: NetLabel:  protocols = UNLABELED CIPSOv4
Oct 30 18:03:37 localhost kernel: NetLabel:  unlabeled traffic allowed by default
Oct 30 18:03:37 localhost kernel: PCI-GART: No AMD northbridge found.
Oct 30 18:03:37 localhost kernel: hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0
Oct 30 18:03:37 localhost kernel: hpet0: 3 64-bit timers, 14318180 Hz
Oct 30 18:03:37 localhost kernel: ACPI: RTC can wake from S4
Oct 30 18:03:37 localhost kernel: system 00:08: ioport range 0x800-0x87f has been reserved
Oct 30 18:03:37 localhost kernel: system 00:08: ioport range 0x880-0x8bf has been reserved
Oct 30 18:03:37 localhost kernel: system 00:08: ioport range 0x8c0-0x8df has been reserved
Oct 30 18:03:37 localhost kernel: system 00:08: ioport range 0x8e0-0x8e3 has been reserved
Oct 30 18:03:37 localhost kernel: system 00:08: ioport range 0xc00-0xc7f has been reserved
Oct 30 18:03:37 localhost kernel: system 00:08: ioport range 0xca0-0xca7 has been reserved
Oct 30 18:03:37 localhost kernel: system 00:08: ioport range 0xca9-0xcab has been reserved
Oct 30 18:03:37 localhost kernel: system 00:08: ioport range 0xcad-0xcaf has been reserved
Oct 30 18:03:37 localhost kernel: system 00:09: ioport range 0xca8-0xca8 has been reserved
Oct 30 18:03:37 localhost kernel: system 00:09: ioport range 0xcac-0xcac has been reserved
Oct 30 18:03:37 localhost kernel: system 00:0a: iomem range 0xe0000000-0xefffffff could not be reserved
Oct 30 18:03:37 localhost kernel: pci 0000:08:00.0: PCI bridge, secondary bus 0000:09
Oct 30 18:03:37 localhost kernel: pci 0000:08:00.0:   IO window: disabled
Oct 30 18:03:37 localhost kernel: pci 0000:08:00.0:   MEM window: 0xf4000000-0xf7ffffff
Oct 30 18:03:37 localhost kernel: pci 0000:08:00.0:   PREFETCH window: disabled
Oct 30 18:03:37 localhost kernel: pci 0000:07:00.0: PCI bridge, secondary bus 0000:08
Oct 30 18:03:37 localhost kernel: pci 0000:07:00.0:   IO window: disabled
Oct 30 18:03:37 localhost kernel: pci 0000:07:00.0:   MEM window: 0xf4000000-0xf7ffffff
Oct 30 18:03:37 localhost kernel: pci 0000:07:00.0:   PREFETCH window: disabled
Oct 30 18:03:37 localhost kernel: pci 0000:07:01.0: PCI bridge, secondary bus 0000:0a
Oct 30 18:03:37 localhost kernel: pci 0000:07:01.0:   IO window: disabled
Oct 30 18:03:37 localhost kernel: pci 0000:07:01.0:   MEM window: disabled
Oct 30 18:03:37 localhost kernel: pci 0000:07:01.0:   PREFETCH window: disabled
Oct 30 18:03:37 localhost kernel: pci 0000:06:00.0: PCI bridge, secondary bus 0000:07
Oct 30 18:03:37 localhost kernel: pci 0000:06:00.0:   IO window: disabled
Oct 30 18:03:37 localhost kernel: pci 0000:06:00.0:   MEM window: 0xf4000000-0xf7ffffff
Oct 30 18:03:37 localhost kernel: pci 0000:06:00.0:   PREFETCH window: disabled
Oct 30 18:03:37 localhost kernel: pci 0000:06:00.3: PCI bridge, secondary bus 0000:0b
Oct 30 18:03:37 localhost kernel: pci 0000:06:00.3:   IO window: 0xe000-0xefff
Oct 30 18:03:37 localhost kernel: pci 0000:06:00.3:   MEM window: 0xf3e00000-0xf3ffffff
Oct 30 18:03:37 localhost kernel: pci 0000:06:00.3:   PREFETCH window: 0x00000088000000-0x000000880fffff
Oct 30 18:03:37 localhost kernel: pci 0000:00:02.0: PCI bridge, secondary bus 0000:06
Oct 30 18:03:37 localhost kernel: pci 0000:00:02.0:   IO window: 0xe000-0xefff
Oct 30 18:03:37 localhost kernel: pci 0000:00:02.0:   MEM window: 0xf2000000-0xf7ffffff
Oct 30 18:03:37 localhost kernel: pci 0000:00:02.0:   PREFETCH window: 0x00000088000000-0x000000880fffff
Oct 30 18:03:37 localhost kernel: pci 0000:00:03.0: PCI bridge, secondary bus 0000:0c
Oct 30 18:03:37 localhost kernel: pci 0000:00:03.0:   IO window: disabled
Oct 30 18:03:37 localhost kernel: pci 0000:00:03.0:   MEM window: disabled
Oct 30 18:03:37 localhost kernel: pci 0000:00:03.0:   PREFETCH window: disabled
Oct 30 18:03:37 localhost kernel: pci 0000:00:04.0: PCI bridge, secondary bus 0000:0d
Oct 30 18:03:37 localhost kernel: pci 0000:00:04.0:   IO window: disabled
Oct 30 18:03:37 localhost kernel: pci 0000:00:04.0:   MEM window: disabled
Oct 30 18:03:37 localhost kernel: pci 0000:00:04.0:   PREFETCH window: disabled
Oct 30 18:03:37 localhost kernel: pci 0000:01:00.0: PCI bridge, secondary bus 0000:02
Oct 30 18:03:37 localhost kernel: pci 0000:01:00.0:   IO window: disabled
Oct 30 18:03:37 localhost kernel: pci 0000:01:00.0:   MEM window: 0xfc400000-0xfc5fffff
Oct 30 18:03:37 localhost kernel: pci 0000:01:00.0:   PREFETCH window: 0x000000d8000000-0x000000d80fffff
Oct 30 18:03:37 localhost kernel: pci 0000:01:00.2: PCI bridge, secondary bus 0000:03
Oct 30 18:03:37 localhost kernel: pci 0000:01:00.2:   IO window: disabled
Oct 30 18:03:37 localhost kernel: pci 0000:01:00.2:   MEM window: disabled
Oct 30 18:03:37 localhost kernel: pci 0000:01:00.2:   PREFETCH window: disabled
Oct 30 18:03:37 localhost kernel: pci 0000:00:05.0: PCI bridge, secondary bus 0000:01
Oct 30 18:03:37 localhost kernel: pci 0000:00:05.0:   IO window: disabled
Oct 30 18:03:37 localhost kernel: pci 0000:00:05.0:   MEM window: 0xfc300000-0xfc5fffff
Oct 30 18:03:37 localhost kernel: pci 0000:00:05.0:   PREFETCH window: 0x000000d8000000-0x000000d80fffff
Oct 30 18:03:37 localhost kernel: pci 0000:00:06.0: PCI bridge, secondary bus 0000:0e
Oct 30 18:03:37 localhost kernel: pci 0000:00:06.0:   IO window: disabled
Oct 30 18:03:37 localhost kernel: pci 0000:00:06.0:   MEM window: disabled
Oct 30 18:03:37 localhost gpm[8276]: *** info [startup.c(95)]: 
Oct 30 18:03:37 localhost gpm[8276]: Started gpm successfully. Entered daemon mode.
Oct 30 18:03:37 localhost kernel: pci 0000:00:06.0:   PREFETCH window: disabled
Oct 30 18:03:37 localhost kernel: pci 0000:00:07.0: PCI bridge, secondary bus 0000:0f
Oct 30 18:03:37 localhost kernel: pci 0000:00:07.0:   IO window: disabled
Oct 30 18:03:37 localhost kernel: pci 0000:00:07.0:   MEM window: disabled
Oct 30 18:03:37 localhost kernel: pci 0000:00:07.0:   PREFETCH window: disabled
Oct 30 18:03:37 localhost kernel: pci 0000:04:00.0: PCI bridge, secondary bus 0000:05
Oct 30 18:03:37 localhost kernel: pci 0000:04:00.0:   IO window: disabled
Oct 30 18:03:37 localhost kernel: pci 0000:04:00.0:   MEM window: 0xf8000000-0xfbffffff
Oct 30 18:03:37 localhost kernel: pci 0000:04:00.0:   PREFETCH window: disabled
Oct 30 18:03:37 localhost kernel: pci 0000:00:1c.0: PCI bridge, secondary bus 0000:04
Oct 30 18:03:37 localhost kernel: pci 0000:00:1c.0:   IO window: disabled
Oct 30 18:03:37 localhost kernel: pci 0000:00:1c.0:   MEM window: 0xf8000000-0xfbffffff
Oct 30 18:03:37 localhost kernel: pci 0000:00:1c.0:   PREFETCH window: disabled
Oct 30 18:03:37 localhost kernel: pci 0000:00:1e.0: PCI bridge, secondary bus 0000:10
Oct 30 18:03:37 localhost xinetd[8235]: xinetd Version 2.3.14 started with libwrap loadavg labeled-networking options compiled in.
Oct 30 18:03:37 localhost kernel: pci 0000:00:1e.0:   IO window: 0xd000-0xdfff
Oct 30 18:03:37 localhost xinetd[8235]: Started working: 0 available services
Oct 30 18:03:37 localhost kernel: pci 0000:00:1e.0:   MEM window: 0xfc100000-0xfc2fffff
Oct 30 18:03:37 localhost kernel: pci 0000:00:1e.0:   PREFETCH window: 0x000000d0000000-0x000000d7ffffff
Oct 30 18:03:37 localhost kernel: pci 0000:00:02.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Oct 30 18:03:37 localhost kernel: pci 0000:06:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Oct 30 18:03:37 localhost kernel: pci 0000:07:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Oct 30 18:03:37 localhost kernel: pci 0000:07:01.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Oct 30 18:03:37 localhost kernel: pci 0000:00:03.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Oct 30 18:03:37 localhost kernel: pci 0000:00:04.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Oct 30 18:03:37 localhost kernel: pci 0000:00:05.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Oct 30 18:03:37 localhost kernel: pci 0000:00:06.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Oct 30 18:03:37 localhost kernel: pci 0000:00:1c.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Oct 30 18:03:37 localhost kernel: bus: 00 index 0 io port: [0, ffff]
Oct 30 18:03:37 localhost kernel: bus: 00 index 1 mmio: [0, ffffffffffffffff]
Oct 30 18:03:37 localhost kernel: bus: 06 index 0 io port: [e000, efff]
Oct 30 18:03:37 localhost kernel: bus: 06 index 1 mmio: [f2000000, f7ffffff]
Oct 30 18:03:37 localhost kernel: bus: 06 index 2 mmio: [88000000, 880fffff]
Oct 30 18:03:37 localhost kernel: bus: 06 index 3 mmio: [0, 0]
Oct 30 18:03:37 localhost kernel: bus: 07 index 0 mmio: [0, 0]
Oct 30 18:03:37 localhost kernel: bus: 07 index 1 mmio: [f4000000, f7ffffff]
Oct 30 18:03:37 localhost kernel: bus: 07 index 2 mmio: [0, 0]
Oct 30 18:03:37 localhost kernel: bus: 07 index 3 mmio: [0, 0]
Oct 30 18:03:37 localhost kernel: bus: 08 index 0 mmio: [0, 0]
Oct 30 18:03:37 localhost kernel: bus: 08 index 1 mmio: [f4000000, f7ffffff]
Oct 30 18:03:37 localhost kernel: bus: 08 index 2 mmio: [0, 0]
Oct 30 18:03:37 localhost kernel: bus: 08 index 3 mmio: [0, 0]
Oct 30 18:03:37 localhost kernel: bus: 09 index 0 mmio: [0, 0]
Oct 30 18:03:37 localhost kernel: bus: 09 index 1 mmio: [f4000000, f7ffffff]
Oct 30 18:03:37 localhost kernel: bus: 09 index 2 mmio: [0, 0]
Oct 30 18:03:37 localhost kernel: bus: 09 index 3 mmio: [0, 0]
Oct 30 18:03:37 localhost kernel: bus: 0a index 0 mmio: [0, 0]
Oct 30 18:03:37 localhost kernel: bus: 0a index 1 mmio: [0, 0]
Oct 30 18:03:37 localhost kernel: bus: 0a index 2 mmio: [0, 0]
Oct 30 18:03:37 localhost kernel: bus: 0a index 3 mmio: [0, 0]
Oct 30 18:03:37 localhost kernel: bus: 0b index 0 io port: [e000, efff]
Oct 30 18:03:37 localhost kernel: bus: 0b index 1 mmio: [f3e00000, f3ffffff]
Oct 30 18:03:37 localhost kernel: bus: 0b index 2 mmio: [88000000, 880fffff]
Oct 30 18:03:37 localhost kernel: bus: 0b index 3 mmio: [0, 0]
Oct 30 18:03:37 localhost kernel: bus: 0c index 0 mmio: [0, 0]
Oct 30 18:03:37 localhost kernel: bus: 0c index 1 mmio: [0, 0]
Oct 30 18:03:37 localhost kernel: bus: 0c index 2 mmio: [0, 0]
Oct 30 18:03:37 localhost kernel: bus: 0c index 3 mmio: [0, 0]
Oct 30 18:03:37 localhost kernel: bus: 0d index 0 mmio: [0, 0]
Oct 30 18:03:37 localhost kernel: bus: 0d index 1 mmio: [0, 0]
Oct 30 18:03:37 localhost kernel: bus: 0d index 2 mmio: [0, 0]
Oct 30 18:03:37 localhost kernel: bus: 0d index 3 mmio: [0, 0]
Oct 30 18:03:37 localhost kernel: bus: 01 index 0 mmio: [0, 0]
Oct 30 18:03:37 localhost kernel: bus: 01 index 1 mmio: [fc300000, fc5fffff]
Oct 30 18:03:37 localhost kernel: bus: 01 index 2 mmio: [d8000000, d80fffff]
Oct 30 18:03:37 localhost kernel: bus: 01 index 3 mmio: [0, 0]
Oct 30 18:03:37 localhost kernel: bus: 02 index 0 mmio: [0, 0]
Oct 30 18:03:37 localhost kernel: bus: 02 index 1 mmio: [fc400000, fc5fffff]
Oct 30 18:03:37 localhost kernel: bus: 02 index 2 mmio: [d8000000, d80fffff]
Oct 30 18:03:37 localhost kernel: bus: 02 index 3 mmio: [0, 0]
Oct 30 18:03:37 localhost kernel: bus: 03 index 0 mmio: [0, 0]
Oct 30 18:03:37 localhost kernel: bus: 03 index 1 mmio: [0, 0]
Oct 30 18:03:37 localhost kernel: bus: 03 index 2 mmio: [0, 0]
Oct 30 18:03:37 localhost kernel: bus: 03 index 3 mmio: [0, 0]
Oct 30 18:03:37 localhost kernel: bus: 0e index 0 mmio: [0, 0]
Oct 30 18:03:37 localhost kernel: bus: 0e index 1 mmio: [0, 0]
Oct 30 18:03:37 localhost kernel: bus: 0e index 2 mmio: [0, 0]
Oct 30 18:03:37 localhost kernel: bus: 0e index 3 mmio: [0, 0]
Oct 30 18:03:37 localhost kernel: bus: 0f index 0 mmio: [0, 0]
Oct 30 18:03:38 localhost kernel: bus: 0f index 1 mmio: [0, 0]
Oct 30 18:03:38 localhost kernel: bus: 0f index 2 mmio: [0, 0]
Oct 30 18:03:38 localhost kernel: bus: 0f index 3 mmio: [0, 0]
Oct 30 18:03:38 localhost kernel: bus: 04 index 0 mmio: [0, 0]
Oct 30 18:03:38 localhost kernel: bus: 04 index 1 mmio: [f8000000, fbffffff]
Oct 30 18:03:38 localhost kernel: bus: 04 index 2 mmio: [0, 0]
Oct 30 18:03:38 localhost kernel: bus: 04 index 3 mmio: [0, 0]
Oct 30 18:03:38 localhost kernel: bus: 05 index 0 mmio: [0, 0]
Oct 30 18:03:38 localhost kernel: bus: 05 index 1 mmio: [f8000000, fbffffff]
Oct 30 18:03:38 localhost kernel: bus: 05 index 2 mmio: [0, 0]
Oct 30 18:03:38 localhost kernel: bus: 05 index 3 mmio: [0, 0]
Oct 30 18:03:38 localhost kernel: bus: 10 index 0 io port: [d000, dfff]
Oct 30 18:03:38 localhost kernel: bus: 10 index 1 mmio: [fc100000, fc2fffff]
Oct 30 18:03:38 localhost kernel: bus: 10 index 2 mmio: [d0000000, d7ffffff]
Oct 30 18:03:38 localhost kernel: bus: 10 index 3 io port: [0, ffff]
Oct 30 18:03:38 localhost kernel: bus: 10 index 4 mmio: [0, ffffffffffffffff]
Oct 30 18:03:38 localhost kernel: NET: Registered protocol family 2
Oct 30 18:03:38 localhost kernel: IP route cache hash table entries: 65536 (order: 7, 524288 bytes)
Oct 30 18:03:38 localhost kernel: TCP established hash table entries: 262144 (order: 10, 4194304 bytes)
Oct 30 18:03:38 localhost kernel: TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
Oct 30 18:03:38 localhost kernel: TCP: Hash tables configured (established 262144 bind 65536)
Oct 30 18:03:38 localhost kernel: TCP reno registered
Oct 30 18:03:38 localhost kernel: NET: Registered protocol family 1
Oct 30 18:03:38 localhost kernel: checking if image is initramfs... it is
Oct 30 18:03:38 localhost kernel: Freeing initrd memory: 3364k freed
Oct 30 18:03:38 localhost kernel: audit: initializing netlink socket (disabled)
Oct 30 18:03:38 localhost kernel: type=2000 audit(1225407720.985:1): initialized
Oct 30 18:03:38 localhost kernel: HugeTLB registered 2 MB page size, pre-allocated 0 pages
Oct 30 18:03:38 localhost kernel: VFS: Disk quotas dquot_6.5.1
Oct 30 18:03:38 localhost kernel: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
Oct 30 18:03:38 localhost kernel: msgmni has been set to 4024
Oct 30 18:03:38 localhost avahi-daemon[8398]: Found user 'avahi' (UID 70) and group 'avahi' (GID 70).
Oct 30 18:03:38 localhost kernel: io scheduler noop registered
Oct 30 18:03:38 localhost avahi-daemon[8398]: Successfully dropped root privileges.
Oct 30 18:03:38 localhost kernel: io scheduler anticipatory registered
Oct 30 18:03:38 localhost avahi-daemon[8398]: avahi-daemon 0.6.16 starting up.
Oct 30 18:03:38 localhost kernel: io scheduler deadline registered
Oct 30 18:03:38 localhost avahi-daemon[8398]: WARNING: No NSS support for mDNS detected, consider installing nss-mdns!
Oct 30 18:03:38 localhost multipathd: uevent 'add' from '/class/net/virbr0' 
Oct 30 18:03:39 localhost kernel: io scheduler cfq registered (default)
Oct 30 18:03:39 localhost avahi-daemon[8398]: Successfully called chroot().
Oct 30 18:03:39 localhost avahi-daemon[8398]: Successfully dropped remaining capabilities.
Oct 30 18:03:39 localhost kernel: pcieport-driver 0000:00:02.0: found MSI capability
Oct 30 18:03:39 localhost multipathd: UDEV_LOG=3 
Oct 30 18:03:39 localhost avahi-daemon[8398]: Loading service file /services/sftp-ssh.service.
Oct 30 18:03:39 localhost multipathd: ACTION=add 
Oct 30 18:03:39 localhost avahi-daemon[8398]: New relevant interface eth0.IPv6 for mDNS.
Oct 30 18:03:39 localhost multipathd: DEVPATH=/class/net/virbr0 
Oct 30 18:03:39 localhost kernel: pcieport-driver 0000:00:03.0: found MSI capability
Oct 30 18:03:39 localhost multipathd: SUBSYSTEM=net 
Oct 30 18:03:39 localhost multipathd: INTERFACE=virbr0 
Oct 30 18:03:39 localhost avahi-daemon[8398]: Joining mDNS multicast group on interface eth0.IPv6 with address fe80::219:b9ff:fec2:5553.
Oct 30 18:03:39 localhost multipathd: IFINDEX=4 
Oct 30 18:03:39 localhost avahi-daemon[8398]: New relevant interface eth0.IPv4 for mDNS.
Oct 30 18:03:39 localhost multipathd: SEQNUM=1206 
Oct 30 18:03:39 localhost kernel: pcieport-driver 0000:00:04.0: found MSI capability
Oct 30 18:03:39 localhost multipathd: UDEVD_EVENT=1 
Oct 30 18:03:39 localhost avahi-daemon[8398]: Joining mDNS multicast group on interface eth0.IPv4 with address 172.23.4.143.
Oct 30 18:03:39 localhost avahi-daemon[8398]: Network interface enumeration completed.
Oct 30 18:03:39 localhost kernel: pcieport-driver 0000:00:05.0: found MSI capability
Oct 30 18:03:39 localhost avahi-daemon[8398]: Registering new address record for fe80::219:b9ff:fec2:5553 on eth0.
Oct 30 18:03:39 localhost avahi-daemon[8398]: Registering new address record for 172.23.4.143 on eth0.
Oct 30 18:03:40 localhost kernel: pcieport-driver 0000:00:06.0: found MSI capability
Oct 30 18:03:40 localhost multipathd: uevent 'remove' from '/class/net/virbr0' 
Oct 30 18:03:40 localhost avahi-daemon[8398]: Registering HINFO record with values 'X86_64'/'LINUX'.
Oct 30 18:03:40 localhost multipathd: UDEV_LOG=3 
Oct 30 18:03:41 localhost kernel: pcieport-driver 0000:00:07.0: found MSI capability
Oct 30 18:03:41 localhost kernel: pcieport-driver 0000:00:1c.0: found MSI capability
Oct 30 18:03:41 localhost multipathd: ACTION=remove 
Oct 30 18:03:41 localhost multipathd: DEVPATH=/class/net/virbr0 
Oct 30 18:03:41 localhost avahi-daemon[8398]: Server startup complete. Host name is localhost.local. Local service cookie is 2183414636.
Oct 30 18:03:41 localhost kernel: pcieport-driver 0000:07:00.0: found MSI capability
Oct 30 18:03:41 localhost multipathd: SUBSYSTEM=net 
Oct 30 18:03:41 localhost multipathd: INTERFACE=virbr0 
Oct 30 18:03:41 localhost kernel: pcieport-driver 0000:07:01.0: found MSI capability
Oct 30 18:03:41 localhost multipathd: IFINDEX=4 
Oct 30 18:03:41 localhost multipathd: SEQNUM=1212 
Oct 30 18:03:42 localhost multipathd: UDEVD_EVENT=1 
Oct 30 18:03:42 localhost kernel: pci_hotplug: PCI Hot Plug PCI Core version: 0.5
Oct 30 18:03:42 localhost kernel: processor ACPI0007:00: registered as cooling_device0
Oct 30 18:03:42 localhost kernel: processor ACPI0007:01: registered as cooling_device1
Oct 30 18:03:42 localhost kernel: processor ACPI0007:02: registered as cooling_device2
Oct 30 18:03:42 localhost kernel: processor ACPI0007:03: registered as cooling_device3
Oct 30 18:03:42 localhost kernel: processor ACPI0007:04: registered as cooling_device4
Oct 30 18:03:42 localhost kernel: processor ACPI0007:05: registered as cooling_device5
Oct 30 18:03:42 localhost kernel: processor ACPI0007:06: registered as cooling_device6
Oct 30 18:03:42 localhost kernel: processor ACPI0007:07: registered as cooling_device7
Oct 30 18:03:42 localhost kernel: Non-volatile memory driver v1.2
Oct 30 18:03:42 localhost kernel: Linux agpgart interface v0.103
Oct 30 18:03:42 localhost kernel: Serial: 8250/16550 driver4 ports, IRQ sharing enabled
Oct 30 18:03:42 localhost avahi-daemon[8398]: Service "SFTP File Transfer on localhost" (/services/sftp-ssh.service) successfully established.
Oct 30 18:03:42 localhost kernel: serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
Oct 30 18:03:42 localhost kernel: serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
Oct 30 18:03:42 localhost kernel: 00:06: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
Oct 30 18:03:42 localhost kernel: 00:07: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
Oct 30 18:03:42 localhost kernel: brd: module loaded
Oct 30 18:03:42 localhost kernel: Uniform Multi-Platform E-IDE driver
Oct 30 18:03:42 localhost kernel: piix 0000:00:1f.1: IDE controller (0x8086:0x269e rev 0x09)
Oct 30 18:03:42 localhost kernel: pci 0000:00:1f.1: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Oct 30 18:03:42 localhost kernel: piix 0000:00:1f.1: IDE port disabled
Oct 30 18:03:42 localhost kernel: piix 0000:00:1f.1: not 100% native mode: will probe irqs later
Oct 30 18:03:42 localhost kernel:     ide0: BM-DMA at 0xfc00-0xfc07
Oct 30 18:03:42 localhost kernel: hda: TSSTcorp CD-RW/DVD-ROM TS-H492C, ATAPI CD/DVD-ROM drive
Oct 30 18:03:42 localhost kernel: hda: UDMA/33 mode selected
Oct 30 18:03:42 localhost kernel: ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
Oct 30 18:03:42 localhost kernel: ide_generic: please use "probe_mask=0x3f" module parameter for probing all legacy ISA IDE ports
Oct 30 18:03:42 localhost kernel: ide_generic: I/O resource 0x1F0-0x1F7 not free.
Oct 30 18:03:43 localhost kernel: ide_generic: I/O resource 0x170-0x177 not free.
Oct 30 18:03:43 localhost kernel: ide-floppy driver 1.00
Oct 30 18:03:43 localhost kernel: PNP: No PS/2 controller found. Probing ports directly.
Oct 30 18:03:43 localhost kernel: serio: i8042 KBD port at 0x60,0x64 irq 1
Oct 30 18:03:43 localhost kernel: serio: i8042 AUX port at 0x60,0x64 irq 12
Oct 30 18:03:43 localhost kernel: mice: PS/2 mouse device common for all mice
Oct 30 18:03:43 localhost kernel: cpuidle: using governor ladder
Oct 30 18:03:43 localhost kernel: usbcore: registered new interface driver hiddev
Oct 30 18:03:43 localhost kernel: usbcore: registered new interface driver usbhid
Oct 30 18:03:43 localhost kernel: usbhid: v2.6:USB HID core driver
Oct 30 18:03:43 localhost kernel: TCP bic registered
Oct 30 18:03:43 localhost kernel: Initializing XFRM netlink socket
Oct 30 18:03:43 localhost kernel: NET: Registered protocol family 17
Oct 30 18:03:43 localhost kernel: registered taskstats version 1
Oct 30 18:03:43 localhost kernel: Freeing unused kernel memory: 348k freed
Oct 30 18:03:43 localhost kernel: Write protecting the kernel read-only data: 3600k
Oct 30 18:03:43 localhost kernel: ehci_hcd 0000:00:1d.7: PCI INT A -> GSI 21 (level, low) -> IRQ 21
Oct 30 18:03:43 localhost kernel: ehci_hcd 0000:00:1d.7: EHCI Host Controller
Oct 30 18:03:43 localhost kernel: ehci_hcd 0000:00:1d.7: new USB bus registered, assigned bus number 1
Oct 30 18:03:43 localhost kernel: ehci_hcd 0000:00:1d.7: debug port 1
Oct 30 18:03:43 localhost kernel: ehci_hcd 0000:00:1d.7: irq 21, io mem 0xfc600000
Oct 30 18:03:43 localhost kernel: ehci_hcd 0000:00:1d.7: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004
Oct 30 18:03:43 localhost kernel: usb usb1: configuration #1 chosen from 1 choice
Oct 30 18:03:43 localhost kernel: hub 1-0:1.0: USB hub found
Oct 30 18:03:43 localhost kernel: hub 1-0:1.0: 6 ports detected
Oct 30 18:03:43 localhost kernel: USB Universal Host Controller Interface driver v3.0
Oct 30 18:03:43 localhost kernel: uhci_hcd 0000:00:1d.0: PCI INT A -> GSI 21 (level, low) -> IRQ 21
Oct 30 18:03:43 localhost kernel: uhci_hcd 0000:00:1d.0: UHCI Host Controller
Oct 30 18:03:43 localhost kernel: uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 2
Oct 30 18:03:43 localhost kernel: uhci_hcd 0000:00:1d.0: irq 21, io base 0x0000cce0
Oct 30 18:03:43 localhost kernel: usb usb2: configuration #1 chosen from 1 choice
Oct 30 18:03:43 localhost kernel: hub 2-0:1.0: USB hub found
Oct 30 18:03:43 localhost kernel: hub 2-0:1.0: 2 ports detected
Oct 30 18:03:43 localhost kernel: uhci_hcd 0000:00:1d.1: PCI INT B -> GSI 20 (level, low) -> IRQ 20
Oct 30 18:03:43 localhost kernel: uhci_hcd 0000:00:1d.1: UHCI Host Controller
Oct 30 18:03:43 localhost kernel: uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 3
Oct 30 18:03:43 localhost kernel: uhci_hcd 0000:00:1d.1: irq 20, io base 0x0000ccc0
Oct 30 18:03:43 localhost kernel: usb usb3: configuration #1 chosen from 1 choice
Oct 30 18:03:43 localhost kernel: hub 3-0:1.0: USB hub found
Oct 30 18:03:43 localhost kernel: hub 3-0:1.0: 2 ports detected
Oct 30 18:03:43 localhost kernel: uhci_hcd 0000:00:1d.2: PCI INT C -> GSI 21 (level, low) -> IRQ 21
Oct 30 18:03:43 localhost kernel: uhci_hcd 0000:00:1d.2: UHCI Host Controller
Oct 30 18:03:43 localhost kernel: uhci_hcd 0000:00:1d.2: new USB bus registered, assigned bus number 4
Oct 30 18:03:43 localhost kernel: uhci_hcd 0000:00:1d.2: irq 21, io base 0x0000cca0
Oct 30 18:03:43 localhost kernel: usb usb4: configuration #1 chosen from 1 choice
Oct 30 18:03:43 localhost kernel: hub 4-0:1.0: USB hub found
Oct 30 18:03:43 localhost kernel: hub 4-0:1.0: 2 ports detected
Oct 30 18:03:43 localhost kernel: usb 3-1: new full speed USB device using uhci_hcd and address 2
Oct 30 18:03:43 localhost kernel: SCSI subsystem initialized
Oct 30 18:03:43 localhost kernel: Driver 'sd' needs updating - please use bus_type methods
Oct 30 18:03:43 localhost kernel: megasas: 00.00.04.01 Thu July 24 11:41:51 PST 2008
Oct 30 18:03:43 localhost kernel: megasas: 0x1028:0x0015:0x1028:0x1f03: bus 2:slot 14:func 0
Oct 30 18:03:43 localhost kernel: megaraid_sas 0000:02:0e.0: PCI INT A -> GSI 142 (level, low) -> IRQ 142
Oct 30 18:03:43 localhost kernel: megasas: FW now in Ready state
Oct 30 18:03:43 localhost kernel: usb 3-1: configuration #1 chosen from 1 choice
Oct 30 18:03:43 localhost kernel: scsi0 : LSI SAS based MegaRAID driver
Oct 30 18:03:43 localhost kernel: hub 3-1:1.0: USB hub found
Oct 30 18:03:43 localhost kernel: scsi 0:0:0:0: Direct-Access     ATA      WDC WD800AAJS-18 0A04 PQ: 0 ANSI: 5
Oct 30 18:03:43 localhost kernel: hub 3-1:1.0: 3 ports detected
Oct 30 18:03:43 localhost kernel: scsi 0:0:1:0: Direct-Access     ATA      WDC WD800AAJS-18 0A04 PQ: 0 ANSI: 5
Oct 30 18:03:43 localhost kernel: scsi 0:0:8:0: Enclosure         DP       BACKPLANE        1.05 PQ: 0 ANSI: 5
Oct 30 18:03:43 localhost kernel: scsi 0:2:0:0: Direct-Access     DELL     PERC 5/i         1.03 PQ: 0 ANSI: 5
Oct 30 18:03:43 localhost kernel: sd 0:2:0:0: [sda] 32768000 512-byte hardware sectors (16777 MB)
Oct 30 18:03:43 localhost kernel: sd 0:2:0:0: [sda] Write Protect is off
Oct 30 18:03:43 localhost kernel: sd 0:2:0:0: [sda] Write cache: disabled, read cache: enabled, supports DPO and FUA
Oct 30 18:03:43 localhost kernel: sd 0:2:0:0: [sda] 32768000 512-byte hardware sectors (16777 MB)
Oct 30 18:03:43 localhost kernel: sd 0:2:0:0: [sda] Write Protect is off
Oct 30 18:03:43 localhost kernel: sd 0:2:0:0: [sda] Write cache: disabled, read cache: enabled, supports DPO and FUA
Oct 30 18:03:43 localhost kernel:  sda: sda1 sda2
Oct 30 18:03:43 localhost kernel: sd 0:2:0:0: [sda] Attached SCSI disk
Oct 30 18:03:43 localhost kernel: scsi 0:2:1:0: Direct-Access     DELL     PERC 5/i         1.03 PQ: 0 ANSI: 5
Oct 30 18:03:43 localhost kernel: sd 0:2:1:0: [sdb] 24576000 512-byte hardware sectors (12583 MB)
Oct 30 18:03:43 localhost kernel: sd 0:2:1:0: [sdb] Write Protect is off
Oct 30 18:03:43 localhost kernel: sd 0:2:1:0: [sdb] Write cache: disabled, read cache: enabled, supports DPO and FUA
Oct 30 18:03:43 localhost kernel: sd 0:2:1:0: [sdb] 24576000 512-byte hardware sectors (12583 MB)
Oct 30 18:03:43 localhost kernel: sd 0:2:1:0: [sdb] Write Protect is off
Oct 30 18:03:43 localhost kernel: sd 0:2:1:0: [sdb] Write cache: disabled, read cache: enabled, supports DPO and FUA
Oct 30 18:03:43 localhost kernel:  sdb:
Oct 30 18:03:43 localhost kernel: sd 0:2:1:0: [sdb] Attached SCSI disk
Oct 30 18:03:43 localhost kernel: scsi 0:2:2:0: Direct-Access     DELL     PERC 5/i         1.03 PQ: 0 ANSI: 5
Oct 30 18:03:43 localhost kernel: sd 0:2:2:0: [sdc] 24576000 512-byte hardware sectors (12583 MB)
Oct 30 18:03:43 localhost kernel: sd 0:2:2:0: [sdc] Write Protect is off
Oct 30 18:03:43 localhost kernel: sd 0:2:2:0: [sdc] Write cache: disabled, read cache: enabled, supports DPO and FUA
Oct 30 18:03:43 localhost kernel: sd 0:2:2:0: [sdc] 24576000 512-byte hardware sectors (12583 MB)
Oct 30 18:03:43 localhost kernel: sd 0:2:2:0: [sdc] Write Protect is off
Oct 30 18:03:43 localhost kernel: sd 0:2:2:0: [sdc] Write cache: disabled, read cache: enabled, supports DPO and FUA
Oct 30 18:03:43 localhost kernel:  sdc:
Oct 30 18:03:43 localhost kernel: sd 0:2:2:0: [sdc] Attached SCSI disk
Oct 30 18:03:43 localhost kernel: scsi 0:2:3:0: Direct-Access     DELL     PERC 5/i         1.03 PQ: 0 ANSI: 5
Oct 30 18:03:43 localhost kernel: sd 0:2:3:0: [sdd] 24576000 512-byte hardware sectors (12583 MB)
Oct 30 18:03:43 localhost kernel: sd 0:2:3:0: [sdd] Write Protect is off
Oct 30 18:03:43 localhost kernel: sd 0:2:3:0: [sdd] Write cache: disabled, read cache: enabled, supports DPO and FUA
Oct 30 18:03:43 localhost kernel: sd 0:2:3:0: [sdd] 24576000 512-byte hardware sectors (12583 MB)
Oct 30 18:03:43 localhost kernel: sd 0:2:3:0: [sdd] Write Protect is off
Oct 30 18:03:43 localhost kernel: sd 0:2:3:0: [sdd] Write cache: disabled, read cache: enabled, supports DPO and FUA
Oct 30 18:03:43 localhost kernel:  sdd:
Oct 30 18:03:43 localhost kernel: sd 0:2:3:0: [sdd] Attached SCSI disk
Oct 30 18:03:43 localhost kernel: QLogic Fibre Channel HBA Driver: 8.02.01-k7
Oct 30 18:03:43 localhost kernel: qla2xxx 0000:0b:02.0: PCI INT A -> GSI 36 (level, low) -> IRQ 36
Oct 30 18:03:43 localhost kernel: qla2xxx 0000:0b:02.0: Found an ISP2312, irq 36, iobase 0xffffc2000031e000
Oct 30 18:03:43 localhost kernel: qla2xxx 0000:0b:02.0: Configuring PCI space...
Oct 30 18:03:43 localhost kernel: qla2xxx 0000:0b:02.0: Configure NVRAM parameters...
Oct 30 18:03:43 localhost kernel: qla2xxx 0000:0b:02.0: Verifying loaded RISC code...
Oct 30 18:03:43 localhost kernel: firmware: requesting ql2300_fw.bin
Oct 30 18:03:43 localhost kernel: usb 4-2: new low speed USB device using uhci_hcd and address 2
Oct 30 18:03:43 localhost kernel: usb 4-2: configuration #1 chosen from 1 choice
Oct 30 18:03:43 localhost kernel: input: HID 1241:1155 as /class/input/input0
Oct 30 18:03:43 localhost kernel: input: USB HID v1.00 Mouse [HID 1241:1155] on usb-0000:00:1d.2-2
Oct 30 18:03:43 localhost kernel: usb 3-1.1: new low speed USB device using uhci_hcd and address 3
Oct 30 18:03:43 localhost kernel: usb 3-1.1: configuration #1 chosen from 1 choice
Oct 30 18:03:43 localhost kernel: input: Dell Dell USB Keyboard Hub as /class/input/input1
Oct 30 18:03:43 localhost kernel: input: USB HID v1.10 Keyboard [Dell Dell USB Keyboard Hub] on usb-0000:00:1d.1-1.1
Oct 30 18:03:43 localhost kernel: input: Dell Dell USB Keyboard Hub as /class/input/input2
Oct 30 18:03:43 localhost kernel: input: USB HID v1.10 Device [Dell Dell USB Keyboard Hub] on usb-0000:00:1d.1-1.1
Oct 30 18:03:43 localhost kernel: qla2xxx 0000:0b:02.0: Firmware image unavailable.
Oct 30 18:03:43 localhost kernel: qla2xxx 0000:0b:02.0: Firmware images can be retrieved from: ftp://ftp.qlogic.com/outgoing/linux/firmware/.
Oct 30 18:03:43 localhost kernel: qla2xxx 0000:0b:02.0: Failed to initialize adapter
Oct 30 18:03:43 localhost kernel: qla2xxx 0000:0b:02.0: PCI INT A disabled
Oct 30 18:03:43 localhost kernel: qla2xxx 0000:0b:02.1: PCI INT B -> GSI 37 (level, low) -> IRQ 37
Oct 30 18:03:43 localhost kernel: qla2xxx 0000:0b:02.1: Found an ISP2312, irq 37, iobase 0xffffc2000031e000
Oct 30 18:03:43 localhost kernel: qla2xxx 0000:0b:02.1: Configuring PCI space...
Oct 30 18:03:43 localhost kernel: qla2xxx 0000:0b:02.1: Configure NVRAM parameters...
Oct 30 18:03:43 localhost kernel: qla2xxx 0000:0b:02.1: Verifying loaded RISC code...
Oct 30 18:03:43 localhost kernel: firmware: requesting ql2300_fw.bin
Oct 30 18:03:43 localhost kernel: qla2xxx 0000:0b:02.1: Firmware image unavailable.
Oct 30 18:03:43 localhost kernel: qla2xxx 0000:0b:02.1: Firmware images can be retrieved from: ftp://ftp.qlogic.com/outgoing/linux/firmware/.
Oct 30 18:03:43 localhost kernel: qla2xxx 0000:0b:02.1: Failed to initialize adapter
Oct 30 18:03:43 localhost kernel: qla2xxx 0000:0b:02.1: PCI INT B disabled
Oct 30 18:03:43 localhost kernel: Emulex LightPulse Fibre Channel SCSI driver 8.2.7
Oct 30 18:03:43 localhost kernel: Copyright(c) 2004-2008 Emulex.  All rights reserved.
Oct 30 18:03:43 localhost kernel: lpfc 0000:0b:01.0: PCI INT A -> GSI 32 (level, low) -> IRQ 32
Oct 30 18:03:44 localhost kernel: scsi3 :  on PCI bus 0b device 08 irq 32
Oct 30 18:03:44 localhost kernel: lpfc 0000:0b:01.0: 0:1303 Link Up Event x1 received Data: x1 x1 x4 x0
Oct 30 18:03:44 localhost kernel: scsi 3:0:0:0: Direct-Access     LSI      INF-01-00        9660 PQ: 0 ANSI: 5
Oct 30 18:03:44 localhost kernel: sd 3:0:0:0: [sde] 20971520 512-byte hardware sectors (10737 MB)
Oct 30 18:03:44 localhost kernel: sd 3:0:0:0: [sde] Write Protect is off
Oct 30 18:03:44 localhost kernel: sd 3:0:0:0: [sde] Write cache: enabled, read cache: enabled, supports DPO and FUA
Oct 30 18:03:44 localhost kernel: sd 3:0:0:0: [sde] 20971520 512-byte hardware sectors (10737 MB)
Oct 30 18:03:44 localhost kernel: sd 3:0:0:0: [sde] Write Protect is off
Oct 30 18:03:44 localhost kernel: sd 3:0:0:0: [sde] Write cache: enabled, read cache: enabled, supports DPO and FUA
Oct 30 18:03:44 localhost kernel:  sde:<6>device-mapper: uevent: version 1.0.3
Oct 30 18:03:44 localhost kernel: device-mapper: ioctl: 4.14.0-ioctl (2008-04-23) initialised: dm-devel@redhat.com
Oct 30 18:03:44 localhost kernel: end_request: I/O error, dev sde, sector 0
Oct 30 18:03:44 localhost kernel: Buffer I/O error on device sde, logical block 0
Oct 30 18:03:44 localhost kernel: end_request: I/O error, dev sde, sector 0
Oct 30 18:03:44 localhost kernel: Buffer I/O error on device sde, logical block 0
Oct 30 18:03:44 localhost kernel: end_request: I/O error, dev sde, sector 0
Oct 30 18:03:44 localhost kernel: Buffer I/O error on device sde, logical block 0
Oct 30 18:03:44 localhost kernel: end_request: I/O error, dev sde, sector 0
Oct 30 18:03:44 localhost kernel: Buffer I/O error on device sde, logical block 0
Oct 30 18:03:44 localhost kernel: end_request: I/O error, dev sde, sector 0
Oct 30 18:03:44 localhost kernel: Buffer I/O error on device sde, logical block 0
Oct 30 18:03:44 localhost kernel: end_request: I/O error, dev sde, sector 0
Oct 30 18:03:44 localhost kernel: Buffer I/O error on device sde, logical block 0
Oct 30 18:03:44 localhost kernel: end_request: I/O error, dev sde, sector 0
Oct 30 18:03:44 localhost kernel: Buffer I/O error on device sde, logical block 0
Oct 30 18:03:44 localhost kernel: Dev sde: unable to read RDB block 0
Oct 30 18:03:44 localhost kernel: end_request: I/O error, dev sde, sector 0
Oct 30 18:03:44 localhost kernel: Buffer I/O error on device sde, logical block 0
Oct 30 18:03:44 localhost kernel: end_request: I/O error, dev sde, sector 0
Oct 30 18:03:44 localhost kernel: Buffer I/O error on device sde, logical block 0
Oct 30 18:03:44 localhost kernel:  unable to read partition table
Oct 30 18:03:44 localhost kernel: sd 3:0:0:0: [sde] Attached SCSI disk
Oct 30 18:03:44 localhost kernel: scsi 3:0:1:0: Direct-Access     LSI      INF-01-00        9660 PQ: 0 ANSI: 5
Oct 30 18:03:44 localhost kernel: sd 3:0:1:0: [sdf] 20971520 512-byte hardware sectors (10737 MB)
Oct 30 18:03:44 localhost kernel: sd 3:0:1:0: [sdf] Write Protect is off
Oct 30 18:03:44 localhost kernel: sd 3:0:1:0: [sdf] Write cache: enabled, read cache: enabled, supports DPO and FUA
Oct 30 18:03:44 localhost kernel: sd 3:0:1:0: [sdf] 20971520 512-byte hardware sectors (10737 MB)
Oct 30 18:03:44 localhost kernel: sd 3:0:1:0: [sdf] Write Protect is off
Oct 30 18:03:44 localhost kernel: sd 3:0:1:0: [sdf] Write cache: enabled, read cache: enabled, supports DPO and FUA
Oct 30 18:03:44 localhost kernel:  sdf: unknown partition table
Oct 30 18:03:44 localhost kernel: sd 3:0:1:0: [sdf] Attached SCSI disk
Oct 30 18:03:44 localhost kernel: scsi 3:0:2:0: Direct-Access     LSI      INF-01-00        0530 PQ: 1 ANSI: 3
Oct 30 18:03:44 localhost kernel: scsi 3:0:3:0: Direct-Access     LSI      INF-01-00        0530 PQ: 1 ANSI: 3
Oct 30 18:03:44 localhost kernel: end_request: I/O error, dev sde, sector 0
Oct 30 18:03:44 localhost kernel: end_request: I/O error, dev sde, sector 20971392
Oct 30 18:03:44 localhost kernel: end_request: I/O error, dev sde, sector 20971504
Oct 30 18:03:44 localhost kernel: end_request: I/O error, dev sde, sector 0
Oct 30 18:03:44 localhost kernel: end_request: I/O error, dev sde, sector 8
Oct 30 18:03:44 localhost kernel: end_request: I/O error, dev sde, sector 0
Oct 30 18:03:44 localhost kernel: end_request: I/O error, dev sde, sector 0
Oct 30 18:03:44 localhost kernel: end_request: I/O error, dev sde, sector 20971392
Oct 30 18:03:44 localhost kernel: Buffer I/O error on device sde, logical block 2621424
Oct 30 18:03:44 localhost kernel: end_request: I/O error, dev sde, sector 20971392
Oct 30 18:03:44 localhost kernel: Buffer I/O error on device sde, logical block 2621424
Oct 30 18:03:44 localhost kernel: end_request: I/O error, dev sde, sector 0
Oct 30 18:03:44 localhost kernel: Buffer I/O error on device sde, logical block 0
Oct 30 18:03:44 localhost kernel: Buffer I/O error on device sde, logical block 1
Oct 30 18:03:44 localhost kernel: Buffer I/O error on device sde, logical block 2
Oct 30 18:03:44 localhost kernel: Buffer I/O error on device sde, logical block 3
Oct 30 18:03:44 localhost kernel: Buffer I/O error on device sde, logical block 4
Oct 30 18:03:44 localhost kernel: Buffer I/O error on device sde, logical block 5
Oct 30 18:03:44 localhost kernel: Buffer I/O error on device sde, logical block 6
Oct 30 18:03:44 localhost kernel: Buffer I/O error on device sde, logical block 7
Oct 30 18:03:44 localhost kernel: end_request: I/O error, dev sde, sector 0
Oct 30 18:03:44 localhost kernel: end_request: I/O error, dev sde, sector 20971392
Oct 30 18:03:44 localhost kernel: end_request: I/O error, dev sde, sector 20971392
Oct 30 18:03:44 localhost kernel: end_request: I/O error, dev sde, sector 0
Oct 30 18:03:44 localhost kernel: end_request: I/O error, dev sde, sector 0
Oct 30 18:03:44 localhost kernel: kjournald starting.  Commit interval 5 seconds
Oct 30 18:03:44 localhost kernel: EXT3-fs: mounted filesystem with ordered data mode.
Oct 30 18:03:44 localhost kernel: SELinux:  class peer not defined in policy
Oct 30 18:03:44 localhost kernel: SELinux:  class capability2 not defined in policy
Oct 30 18:03:44 localhost kernel: SELinux:  permission open in class dir not defined in policy
Oct 30 18:03:44 localhost kernel: SELinux:  permission open in class file not defined in policy
Oct 30 18:03:44 localhost kernel: SELinux:  permission open in class chr_file not defined in policy
Oct 30 18:03:44 localhost kernel: SELinux:  permission open in class blk_file not defined in policy
Oct 30 18:03:44 localhost kernel: SELinux:  permission open in class fifo_file not defined in policy
Oct 30 18:03:44 localhost kernel: SELinux:  permission recvfrom in class node not defined in policy
Oct 30 18:03:44 localhost kernel: SELinux:  permission sendto in class node not defined in policy
Oct 30 18:03:44 localhost kernel: SELinux:  permission ingress in class netif not defined in policy
Oct 30 18:03:44 localhost kernel: SELinux:  permission egress in class netif not defined in policy
Oct 30 18:03:44 localhost kernel: SELinux:  permission setfcap in class capability not defined in policy
Oct 30 18:03:44 localhost kernel: SELinux:  permission forward_in in class packet not defined in policy
Oct 30 18:03:44 localhost kernel: SELinux:  permission forward_out in class packet not defined in policy
Oct 30 18:03:44 localhost kernel: SELinux: the above unknown classes and permissions will be denied
Oct 30 18:03:44 localhost kernel: intel_rng: FWH not detected
Oct 30 18:03:44 localhost kernel: input: PC Speaker as /class/input/input3
Oct 30 18:03:44 localhost kernel: shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
Oct 30 18:03:44 localhost kernel: EDAC MC: Ver: 2.1.0 Oct 30 2008
Oct 30 18:03:44 localhost kernel: EDAC MC0: Giving out device to 'i5000_edac.c' 'I5000': DEV 0000:00:10.0
Oct 30 18:03:44 localhost kernel: EDAC PCI0: Giving out device to module 'i5000_edac' controller 'EDAC PCI controller': DEV '0000:00:10.0' (POLLED)
Oct 30 18:03:44 localhost kernel: dcdbas dcdbas: Dell Systems Management Base Driver (version 5.6.0-3.2)
Oct 30 18:03:44 localhost kernel: rtc_cmos 00:04: rtc core: registered rtc_cmos as rtc0
Oct 30 18:03:44 localhost kernel: rtc0: alarms up to one day, y3k, hpet irqs
Oct 30 18:03:44 localhost kernel: input: Power Button (FF) as /class/input/input4
Oct 30 18:03:44 localhost kernel: ACPI: Power Button (FF) [PWRF]
Oct 30 18:03:44 localhost kernel: hda: ATAPI 48X DVD-ROM CD-R/RW drive, 1536kB Cache
Oct 30 18:03:44 localhost kernel: Uniform CD-ROM driver Revision: 3.20
Oct 30 18:03:44 localhost kernel: Broadcom NetXtreme II Gigabit Ethernet Driver bnx2 v1.8.0 (Aug 14, 2008)
Oct 30 18:03:44 localhost kernel: bnx2 0000:05:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Oct 30 18:03:44 localhost kernel: eth0: Broadcom NetXtreme II BCM5708 1000Base-T (B2) PCI-X 64-bit 133MHz found at mem f8000000, IRQ 16, node addr 00:19:b9:c2:55:53
Oct 30 18:03:44 localhost kernel: bnx2 0000:09:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Oct 30 18:03:44 localhost kernel: scsi 0:0:8:0: Attached scsi generic sg0 type 13
Oct 30 18:03:44 localhost kernel: sd 0:2:0:0: Attached scsi generic sg1 type 0
Oct 30 18:03:44 localhost kernel: sd 0:2:1:0: Attached scsi generic sg2 type 0
Oct 30 18:03:44 localhost kernel: sd 0:2:2:0: Attached scsi generic sg3 type 0
Oct 30 18:03:44 localhost kernel: sd 0:2:3:0: Attached scsi generic sg4 type 0
Oct 30 18:03:44 localhost kernel: sd 3:0:0:0: Attached scsi generic sg5 type 0
Oct 30 18:03:44 localhost kernel: sd 3:0:1:0: Attached scsi generic sg6 type 0
Oct 30 18:03:44 localhost kernel: scsi 3:0:2:0: Attached scsi generic sg7 type 0
Oct 30 18:03:44 localhost kernel: scsi 3:0:3:0: Attached scsi generic sg8 type 0
Oct 30 18:03:44 localhost kernel: eth1: Broadcom NetXtreme II BCM5708 1000Base-T (B2) PCI-X 64-bit 133MHz found at mem f4000000, IRQ 16, node addr 00:19:b9:c2:55:55
Oct 30 18:03:44 localhost kernel: end_request: I/O error, dev sde, sector 20971392
Oct 30 18:03:44 localhost kernel: __ratelimit: 34 callbacks suppressed
Oct 30 18:03:44 localhost kernel: Buffer I/O error on device sde, logical block 2621424
Oct 30 18:03:44 localhost kernel: end_request: I/O error, dev sde, sector 20971392
Oct 30 18:03:44 localhost kernel: Buffer I/O error on device sde, logical block 2621424
Oct 30 18:03:44 localhost kernel: end_request: I/O error, dev sde, sector 20971512
Oct 30 18:03:44 localhost kernel: Buffer I/O error on device sde, logical block 2621439
Oct 30 18:03:44 localhost kernel: end_request: I/O error, dev sde, sector 20971512
Oct 30 18:03:44 localhost kernel: Buffer I/O error on device sde, logical block 2621439
Oct 30 18:03:44 localhost kernel: end_request: I/O error, dev sde, sector 20971512
Oct 30 18:03:45 localhost kernel: Buffer I/O error on device sde, logical block 2621439
Oct 30 18:03:45 localhost kernel: end_request: I/O error, dev sde, sector 20971512
Oct 30 18:03:45 localhost kernel: Buffer I/O error on device sde, logical block 2621439
Oct 30 18:03:45 localhost kernel: end_request: I/O error, dev sde, sector 20971512
Oct 30 18:03:45 localhost kernel: Buffer I/O error on device sde, logical block 2621439
Oct 30 18:03:45 localhost kernel: end_request: I/O error, dev sde, sector 20971512
Oct 30 18:03:45 localhost kernel: Buffer I/O error on device sde, logical block 2621439
Oct 30 18:03:45 localhost kernel: end_request: I/O error, dev sde, sector 20971456
Oct 30 18:03:45 localhost kernel: Buffer I/O error on device sde, logical block 2621432
Oct 30 18:03:45 localhost kernel: end_request: I/O error, dev sde, sector 20971504
Oct 30 18:03:45 localhost kernel: Buffer I/O error on device sde, logical block 2621438
Oct 30 18:03:45 localhost kernel: end_request: I/O error, dev sde, sector 20971512
Oct 30 18:03:45 localhost kernel: end_request: I/O error, dev sde, sector 20971512
Oct 30 18:03:45 localhost kernel: __ratelimit: 1 callbacks suppressed
Oct 30 18:03:45 localhost kernel: Buffer I/O error on device sde, logical block 2621439
Oct 30 18:03:45 localhost kernel: end_request: I/O error, dev sde, sector 0
Oct 30 18:03:45 localhost kernel: Buffer I/O error on device sde, logical block 0
Oct 30 18:03:45 localhost kernel: end_request: I/O error, dev sde, sector 0
Oct 30 18:03:45 localhost kernel: Buffer I/O error on device sde, logical block 0
Oct 30 18:03:45 localhost kernel: end_request: I/O error, dev sde, sector 0
Oct 30 18:03:45 localhost kernel: Buffer I/O error on device sde, logical block 0
Oct 30 18:03:45 localhost kernel: end_request: I/O error, dev sde, sector 0
Oct 30 18:03:45 localhost kernel: Buffer I/O error on device sde, logical block 0
Oct 30 18:03:45 localhost kernel: end_request: I/O error, dev sde, sector 0
Oct 30 18:03:45 localhost kernel: Buffer I/O error on device sde, logical block 0
Oct 30 18:03:45 localhost kernel: end_request: I/O error, dev sde, sector 0
Oct 30 18:03:45 localhost kernel: Buffer I/O error on device sde, logical block 0
Oct 30 18:03:45 localhost kernel: end_request: I/O error, dev sde, sector 0
Oct 30 18:03:45 localhost kernel: Buffer I/O error on device sde, logical block 0
Oct 30 18:03:45 localhost kernel: end_request: I/O error, dev sde, sector 0
Oct 30 18:03:45 localhost kernel: Buffer I/O error on device sde, logical block 0
Oct 30 18:03:45 localhost kernel: end_request: I/O error, dev sde, sector 0
Oct 30 18:03:45 localhost kernel: Buffer I/O error on device sde, logical block 0
Oct 30 18:03:45 localhost kernel: end_request: I/O error, dev sde, sector 0
Oct 30 18:03:45 localhost kernel: end_request: I/O error, dev sde, sector 0
Oct 30 18:03:45 localhost kernel: __ratelimit: 1 callbacks suppressed
Oct 30 18:03:45 localhost kernel: Buffer I/O error on device sde, logical block 0
Oct 30 18:03:45 localhost kernel: end_request: I/O error, dev sde, sector 0
Oct 30 18:03:45 localhost kernel: Buffer I/O error on device sde, logical block 0
Oct 30 18:03:45 localhost kernel: end_request: I/O error, dev sde, sector 0
Oct 30 18:03:45 localhost kernel: Buffer I/O error on device sde, logical block 0
Oct 30 18:03:45 localhost kernel: end_request: I/O error, dev sde, sector 0
Oct 30 18:03:45 localhost kernel: Buffer I/O error on device sde, logical block 0
Oct 30 18:03:45 localhost kernel: end_request: I/O error, dev sde, sector 0
Oct 30 18:03:45 localhost kernel: Buffer I/O error on device sde, logical block 0
Oct 30 18:03:45 localhost kernel: end_request: I/O error, dev sde, sector 0
Oct 30 18:03:45 localhost kernel: Buffer I/O error on device sde, logical block 0
Oct 30 18:03:45 localhost kernel: end_request: I/O error, dev sde, sector 0
Oct 30 18:03:45 localhost kernel: Buffer I/O error on device sde, logical block 0
Oct 30 18:03:45 localhost kernel: end_request: I/O error, dev sde, sector 0
Oct 30 18:03:45 localhost kernel: Buffer I/O error on device sde, logical block 0
Oct 30 18:03:45 localhost kernel: end_request: I/O error, dev sde, sector 0
Oct 30 18:03:45 localhost kernel: Buffer I/O error on device sde, logical block 0
Oct 30 18:03:45 localhost kernel: end_request: I/O error, dev sde, sector 0
Oct 30 18:03:45 localhost kernel: Buffer I/O error on device sde, logical block 0
Oct 30 18:03:45 localhost kernel: end_request: I/O error, dev sde, sector 0
Oct 30 18:03:45 localhost kernel: end_request: I/O error, dev sde, sector 0
Oct 30 18:03:45 localhost kernel: __ratelimit: 1 callbacks suppressed
Oct 30 18:03:45 localhost kernel: Buffer I/O error on device sde, logical block 0
Oct 30 18:03:45 localhost kernel: end_request: I/O error, dev sde, sector 0
Oct 30 18:03:45 localhost kernel: Buffer I/O error on device sde, logical block 0
Oct 30 18:03:45 localhost kernel: end_request: I/O error, dev sde, sector 0
Oct 30 18:03:45 localhost kernel: Buffer I/O error on device sde, logical block 0
Oct 30 18:03:45 localhost kernel: end_request: I/O error, dev sde, sector 0
Oct 30 18:03:45 localhost kernel: Buffer I/O error on device sde, logical block 0
Oct 30 18:03:45 localhost kernel: end_request: I/O error, dev sde, sector 0
Oct 30 18:03:45 localhost kernel: Buffer I/O error on device sde, logical block 0
Oct 30 18:03:45 localhost kernel: end_request: I/O error, dev sde, sector 0
Oct 30 18:03:45 localhost kernel: Buffer I/O error on device sde, logical block 0
Oct 30 18:03:45 localhost kernel: end_request: I/O error, dev sde, sector 0
Oct 30 18:03:45 localhost kernel: Buffer I/O error on device sde, logical block 0
Oct 30 18:03:45 localhost kernel: floppy0: no floppy controllers found
Oct 30 18:03:45 localhost kernel: lp: driver loaded but no devices found
Oct 30 18:03:45 localhost kernel: NET: Registered protocol family 10
Oct 30 18:03:45 localhost kernel: lo: Disabled Privacy Extensions
Oct 30 18:03:45 localhost kernel: md: Autodetecting RAID arrays.
Oct 30 18:03:45 localhost kernel: md: Scanned 0 and added 0 devices.
Oct 30 18:03:45 localhost kernel: md: autorun ...
Oct 30 18:03:45 localhost kernel: md: ... autorun DONE.
Oct 30 18:03:45 localhost kernel: device-mapper: multipath: version 1.0.5 loaded
Oct 30 18:03:45 localhost kernel: rdac: device handler registered
Oct 30 18:03:45 localhost kernel: device-mapper: multipath round-robin: version 1.0.0 loaded
Oct 30 18:03:45 localhost kernel: sd 3:0:1:0: rdac: LUN 0 (owned)
Oct 30 18:03:45 localhost kernel: sd 3:0:0:0: rdac: LUN 0 (unowned)
Oct 30 18:03:45 localhost kernel: sd 3:0:0:0: sense_key:5; asc 94; ascq 1
Oct 30 18:03:45 localhost kernel: end_request: I/O error, dev sde, sector 20971512
Oct 30 18:03:45 localhost kernel: Buffer I/O error on device sde, logical block 2621439
Oct 30 18:03:45 localhost last message repeated 2 times
Oct 30 18:03:45 localhost kernel: Buffer I/O error on device sde, logical block 2621407
Oct 30 18:03:45 localhost kernel: Buffer I/O error on device sde, logical block 2621407
Oct 30 18:03:45 localhost kernel: Buffer I/O error on device sde, logical block 1
Oct 30 18:03:45 localhost kernel: Buffer I/O error on device sde, logical block 1
Oct 30 18:03:45 localhost modclusterd: startup succeeded
Oct 30 18:03:45 localhost kernel: Buffer I/O error on device sde, logical block 2621438
Oct 30 18:03:45 localhost kernel: Buffer I/O error on device sde, logical block 2621438
Oct 30 18:03:45 localhost kernel: Buffer I/O error on device sde, logical block 2621439
Oct 30 18:03:45 localhost kernel: EXT3 FS on dm-0, internal journal
Oct 30 18:03:45 localhost kernel: kjournald starting.  Commit interval 5 seconds
Oct 30 18:03:45 localhost kernel: EXT3 FS on sda1, internal journal
Oct 30 18:03:45 localhost kernel: EXT3-fs: mounted filesystem with ordered data mode.
Oct 30 18:03:45 localhost kernel: Adding 2031608k swap on /dev/VolGroup00/LogVol01.  Priority:-1 extents:1 across:2031608k
Oct 30 18:03:45 localhost kernel: IA-32 Microcode Update Driver: v1.14a <tigran@aivazian.fsnet.co.uk>
Oct 30 18:03:45 localhost kernel: firmware: requesting intel-ucode/06-0f-0b
Oct 30 18:03:45 localhost last message repeated 7 times
Oct 30 18:03:45 localhost kernel: microcode: CPU0 updated from revision 0xb4 to 0xb7, date = 08062007 
Oct 30 18:03:45 localhost kernel: microcode: CPU1 updated from revision 0xb4 to 0xb7, date = 08062007 
Oct 30 18:03:45 localhost kernel: microcode: CPU2 updated from revision 0xb4 to 0xb7, date = 08062007 
Oct 30 18:03:45 localhost kernel: microcode: CPU3 updated from revision 0xb4 to 0xb7, date = 08062007 
Oct 30 18:03:45 localhost kernel: microcode: CPU4 updated from revision 0xb4 to 0xb7, date = 08062007 
Oct 30 18:03:45 localhost kernel: microcode: CPU5 updated from revision 0xb4 to 0xb7, date = 08062007 
Oct 30 18:03:45 localhost kernel: microcode: CPU6 updated from revision 0xb4 to 0xb7, date = 08062007 
Oct 30 18:03:45 localhost kernel: microcode: CPU7 updated from revision 0xb4 to 0xb7, date = 08062007 
Oct 30 18:03:45 localhost kernel: warning: process `kudzu' used the deprecated sysctl system call with 1.23.
Oct 30 18:03:45 localhost kernel: process `sysctl' is using deprecated sysctl (syscall) net.ipv6.neigh.default.retrans_time; Use net.ipv6.neigh.default.retrans_time_ms instead.
Oct 30 18:03:45 localhost kernel: bnx2: eth0: using MSI
Oct 30 18:03:45 localhost kernel: ADDRCONF(NETDEV_UP): eth0: link is not ready
Oct 30 18:03:45 localhost kernel: bnx2: eth0 NIC Copper Link is Up, 100 Mbps full duplex
Oct 30 18:03:45 localhost kernel: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Oct 30 18:03:45 localhost kernel: warning: `mcstransd' uses 32-bit capabilities (legacy support in use)
Oct 30 18:03:45 localhost kernel: RPC: Registered udp transport module.
Oct 30 18:03:45 localhost kernel: RPC: Registered tcp transport module.
Oct 30 18:03:45 localhost kernel:  rport-3:0-6: blocked FC remote port time out: removing rport
Oct 30 18:03:45 localhost kernel: Bluetooth: Core ver 2.13
Oct 30 18:03:45 localhost kernel: NET: Registered protocol family 31
Oct 30 18:03:45 localhost kernel: Bluetooth: HCI device and connection manager initialized
Oct 30 18:03:46 localhost kernel: Bluetooth: HCI socket layer initialized
Oct 30 18:03:46 localhost kernel: Bluetooth: L2CAP ver 2.11
Oct 30 18:03:46 localhost kernel: Bluetooth: L2CAP socket layer initialized
Oct 30 18:03:46 localhost kernel: Bluetooth: RFCOMM socket layer initialized
Oct 30 18:03:46 localhost kernel: Bluetooth: RFCOMM TTY layer initialized
Oct 30 18:03:46 localhost kernel: Bluetooth: RFCOMM ver 1.10
Oct 30 18:03:46 localhost kernel: Bluetooth: HIDP (Human Interface Emulation) ver 1.2
Oct 30 18:03:46 localhost oddjobd: oddjobd startup succeeded
Oct 30 18:03:46 localhost kernel: i2c /dev entries driver
Oct 30 18:03:46 localhost kernel: Bridge firewalling registered
Oct 30 18:03:46 localhost kernel: virbr0: Dropping NETIF_F_UFO since no NETIF_F_HW_CSUM feature.
Oct 30 18:03:46 localhost kernel: virbr0: starting userspace STP failed, starting kernel STP
Oct 30 18:03:46 localhost kernel: ip_tables: (C) 2000-2006 Netfilter Core Team
Oct 30 18:03:46 localhost kernel: __ratelimit: 15 callbacks suppressed
Oct 30 18:03:46 localhost kernel: Buffer I/O error on device sde, logical block 2621424
Oct 30 18:03:46 localhost kernel: Buffer I/O error on device sde, logical block 2621424
Oct 30 18:03:46 localhost kernel: Buffer I/O error on device sde, logical block 2621439
Oct 30 18:03:46 localhost last message repeated 4 times
Oct 30 18:03:46 localhost kernel: Buffer I/O error on device sde, logical block 2621432
Oct 30 18:03:46 localhost kernel: Buffer I/O error on device sde, logical block 2621438
Oct 30 18:03:46 localhost kernel: Buffer I/O error on device sde, logical block 2621439
Oct 30 18:03:46 localhost saslauthd[8749]: detach_tty      : master pid is: 8749
Oct 30 18:03:46 localhost saslauthd[8749]: ipc_init        : listening on socket: /var/run/saslauthd/mux
Oct 30 18:03:46 localhost ricci: startup succeeded
Oct 30 18:03:46 localhost multipathd: uevent 'remove' from '/class/vc/vcs1' 
Oct 30 18:03:46 localhost multipathd: UDEV_LOG=3 
Oct 30 18:03:46 localhost multipathd: ACTION=remove 
Oct 30 18:03:46 localhost multipathd: DEVPATH=/class/vc/vcs1 
Oct 30 18:03:46 localhost multipathd: SUBSYSTEM=vc 
Oct 30 18:03:46 localhost multipathd: MAJOR=7 
Oct 30 18:03:46 localhost multipathd: MINOR=1 
Oct 30 18:03:46 localhost multipathd: SEQNUM=1225 
Oct 30 18:03:46 localhost multipathd: UDEVD_EVENT=1 
Oct 30 18:03:46 localhost multipathd: DEVNAME=/dev/vcs1 
Oct 30 18:03:46 localhost multipathd: uevent 'remove' from '/class/vc/vcsa1' 
Oct 30 18:03:46 localhost multipathd: UDEV_LOG=3 
Oct 30 18:03:46 localhost multipathd: ACTION=remove 
Oct 30 18:03:46 localhost multipathd: DEVPATH=/class/vc/vcsa1 
Oct 30 18:03:46 localhost multipathd: SUBSYSTEM=vc 
Oct 30 18:03:46 localhost multipathd: MAJOR=7 
Oct 30 18:03:46 localhost multipathd: MINOR=129 
Oct 30 18:03:46 localhost multipathd: SEQNUM=1226 
Oct 30 18:03:46 localhost multipathd: UDEVD_EVENT=1 
Oct 30 18:03:46 localhost multipathd: DEVNAME=/dev/vcsa1 
Oct 30 18:03:46 localhost multipathd: uevent 'remove' from '/class/vc/vcs4' 
Oct 30 18:03:46 localhost multipathd: UDEV_LOG=3 
Oct 30 18:03:46 localhost multipathd: ACTION=remove 
Oct 30 18:03:46 localhost multipathd: DEVPATH=/class/vc/vcs4 
Oct 30 18:03:46 localhost multipathd: SUBSYSTEM=vc 
Oct 30 18:03:46 localhost multipathd: MAJOR=7 
Oct 30 18:03:46 localhost multipathd: MINOR=4 
Oct 30 18:03:46 localhost multipathd: SEQNUM=1227 
Oct 30 18:03:46 localhost multipathd: UDEVD_EVENT=1 
Oct 30 18:03:46 localhost multipathd: DEVNAME=/dev/vcs4 
Oct 30 18:03:46 localhost multipathd: uevent 'remove' from '/class/vc/vcsa4' 
Oct 30 18:03:46 localhost multipathd: UDEV_LOG=3 
Oct 30 18:03:46 localhost multipathd: ACTION=remove 
Oct 30 18:03:46 localhost multipathd: DEVPATH=/class/vc/vcsa4 
Oct 30 18:03:46 localhost multipathd: SUBSYSTEM=vc 
Oct 30 18:03:46 localhost multipathd: MAJOR=7 
Oct 30 18:03:46 localhost multipathd: MINOR=132 
Oct 30 18:03:46 localhost multipathd: SEQNUM=1228 
Oct 30 18:03:46 localhost multipathd: UDEVD_EVENT=1 
Oct 30 18:03:46 localhost multipathd: DEVNAME=/dev/vcsa4 
Oct 30 18:03:46 localhost multipathd: uevent 'remove' from '/class/vc/vcs3' 
Oct 30 18:03:46 localhost multipathd: UDEV_LOG=3 
Oct 30 18:03:46 localhost multipathd: ACTION=remove 
Oct 30 18:03:46 localhost multipathd: DEVPATH=/class/vc/vcs3 
Oct 30 18:03:46 localhost multipathd: SUBSYSTEM=vc 
Oct 30 18:03:46 localhost multipathd: MAJOR=7 
Oct 30 18:03:46 localhost multipathd: MINOR=3 
Oct 30 18:03:46 localhost multipathd: SEQNUM=1233 
Oct 30 18:03:46 localhost multipathd: UDEVD_EVENT=1 
Oct 30 18:03:46 localhost multipathd: DEVNAME=/dev/vcs3 
Oct 30 18:03:46 localhost multipathd: uevent 'remove' from '/class/vc/vcsa3' 
Oct 30 18:03:46 localhost multipathd: UDEV_LOG=3 
Oct 30 18:03:46 localhost multipathd: ACTION=remove 
Oct 30 18:03:46 localhost multipathd: DEVPATH=/class/vc/vcsa3 
Oct 30 18:03:46 localhost multipathd: SUBSYSTEM=vc 
Oct 30 18:03:46 localhost multipathd: MAJOR=7 
Oct 30 18:03:46 localhost multipathd: MINOR=131 
Oct 30 18:03:46 localhost multipathd: SEQNUM=1234 
Oct 30 18:03:46 localhost multipathd: UDEVD_EVENT=1 
Oct 30 18:03:46 localhost multipathd: DEVNAME=/dev/vcsa3 
Oct 30 18:03:46 localhost multipathd: uevent 'remove' from '/class/vc/vcs5' 
Oct 30 18:03:46 localhost multipathd: UDEV_LOG=3 
Oct 30 18:03:46 localhost multipathd: ACTION=remove 
Oct 30 18:03:46 localhost multipathd: DEVPATH=/class/vc/vcs5 
Oct 30 18:03:46 localhost multipathd: SUBSYSTEM=vc 
Oct 30 18:03:46 localhost multipathd: MAJOR=7 
Oct 30 18:03:46 localhost multipathd: MINOR=5 
Oct 30 18:03:46 localhost multipathd: SEQNUM=1237 
Oct 30 18:03:46 localhost multipathd: UDEVD_EVENT=1 
Oct 30 18:03:46 localhost multipathd: DEVNAME=/dev/vcs5 
Oct 30 18:03:46 localhost multipathd: uevent 'remove' from '/class/vc/vcsa5' 
Oct 30 18:03:46 localhost multipathd: UDEV_LOG=3 
Oct 30 18:03:46 localhost multipathd: ACTION=remove 
Oct 30 18:03:46 localhost multipathd: DEVPATH=/class/vc/vcsa5 
Oct 30 18:03:46 localhost multipathd: SUBSYSTEM=vc 
Oct 30 18:03:46 localhost multipathd: MAJOR=7 
Oct 30 18:03:46 localhost multipathd: MINOR=133 
Oct 30 18:03:46 localhost multipathd: SEQNUM=1238 
Oct 30 18:03:46 localhost multipathd: UDEVD_EVENT=1 
Oct 30 18:03:46 localhost multipathd: DEVNAME=/dev/vcsa5 
Oct 30 18:03:47 localhost multipathd: uevent 'remove' from '/class/vc/vcs6' 
Oct 30 18:03:47 localhost multipathd: UDEV_LOG=3 
Oct 30 18:03:47 localhost multipathd: ACTION=remove 
Oct 30 18:03:47 localhost multipathd: DEVPATH=/class/vc/vcs6 
Oct 30 18:03:47 localhost multipathd: SUBSYSTEM=vc 
Oct 30 18:03:47 localhost multipathd: MAJOR=7 
Oct 30 18:03:47 localhost multipathd: MINOR=6 
Oct 30 18:03:47 localhost multipathd: SEQNUM=1241 
Oct 30 18:03:47 localhost multipathd: UDEVD_EVENT=1 
Oct 30 18:03:47 localhost multipathd: DEVNAME=/dev/vcs6 
Oct 30 18:03:47 localhost multipathd: uevent 'remove' from '/class/vc/vcsa6' 
Oct 30 18:03:47 localhost multipathd: UDEV_LOG=3 
Oct 30 18:03:47 localhost multipathd: ACTION=remove 
Oct 30 18:03:47 localhost multipathd: DEVPATH=/class/vc/vcsa6 
Oct 30 18:03:47 localhost multipathd: SUBSYSTEM=vc 
Oct 30 18:03:47 localhost multipathd: MAJOR=7 
Oct 30 18:03:47 localhost multipathd: MINOR=134 
Oct 30 18:03:47 localhost multipathd: SEQNUM=1242 
Oct 30 18:03:47 localhost multipathd: UDEVD_EVENT=1 
Oct 30 18:03:47 localhost multipathd: DEVNAME=/dev/vcsa6 
Oct 30 18:03:47 localhost multipathd: uevent 'remove' from '/class/vc/vcs2' 
Oct 30 18:03:47 localhost multipathd: UDEV_LOG=3 
Oct 30 18:03:47 localhost multipathd: ACTION=remove 
Oct 30 18:03:47 localhost multipathd: DEVPATH=/class/vc/vcs2 
Oct 30 18:03:47 localhost multipathd: SUBSYSTEM=vc 
Oct 30 18:03:47 localhost multipathd: MAJOR=7 
Oct 30 18:03:47 localhost multipathd: MINOR=2 
Oct 30 18:03:47 localhost multipathd: SEQNUM=1245 
Oct 30 18:03:47 localhost multipathd: UDEVD_EVENT=1 
Oct 30 18:03:47 localhost multipathd: DEVNAME=/dev/vcs2 
Oct 30 18:03:47 localhost multipathd: uevent 'remove' from '/class/vc/vcsa2' 
Oct 30 18:03:47 localhost multipathd: UDEV_LOG=3 
Oct 30 18:03:47 localhost multipathd: ACTION=remove 
Oct 30 18:03:47 localhost multipathd: DEVPATH=/class/vc/vcsa2 
Oct 30 18:03:47 localhost multipathd: SUBSYSTEM=vc 
Oct 30 18:03:47 localhost multipathd: MAJOR=7 
Oct 30 18:03:47 localhost multipathd: MINOR=130 
Oct 30 18:03:47 localhost multipathd: SEQNUM=1246 
Oct 30 18:03:47 localhost multipathd: UDEVD_EVENT=1 
Oct 30 18:03:47 localhost multipathd: DEVNAME=/dev/vcsa2 
Oct 30 18:03:47 localhost pcscd: winscard.c:304:SCardConnect() Reader E-Gate 0 0 Not Found
Oct 30 18:03:47 localhost last message repeated 3 times
Oct 30 18:04:33 localhost multipathd: sdf: rdac prio = 3 
===================================TEST STARTED =================================================
Oct 30 18:06:20 localhost UnixSmash4[8984]: 1:UnixSmash has been started.  The system id is 0x737f9395.
Oct 30 18:06:20 localhost UnixSmash4[8984]: The TARGETS file used for this test is TARGETS.
Oct 30 18:06:33 localhost multipathd: sdf: rdac prio = 3 
Oct 30 18:08:33 localhost multipathd: sdf: rdac prio = 3 
Oct 30 18:08:44 localhost kernel: sd 3:0:1:0: sense_key:6; asc 8b; ascq 2
Oct 30 18:08:51 localhost last message repeated 14 times
Oct 30 18:08:51 localhost kernel: sd 3:0:1:0: [sdf] Result: hostbyte=DID_BUS_BUSY driverbyte=DRIVER_OK,SUGGEST_OK
Oct 30 18:08:51 localhost kernel: end_request: I/O error, dev sdf, sector 1067008
Oct 30 18:08:51 localhost kernel: device-mapper: multipath: Failing path 8:80.
Oct 30 18:08:51 localhost multipathd: mpathie: rr_weight = 2 (controller setting) 
Oct 30 18:08:51 localhost multipathd: mpathie: pgfailback = 15 (controller setting) 
Oct 30 18:08:51 localhost multipathd: mpathie: no_path_retry = -1 (controller setting) 
Oct 30 18:08:51 localhost multipathd: pg_timeout = NONE (internal default) 
Oct 30 18:08:51 localhost multipathd: 8:80: mark as failed 
Oct 30 18:08:52 localhost multipathd: uevent 'change' from '/block/dm-2' 
Oct 30 18:08:52 localhost multipathd: UDEV_LOG=3 
Oct 30 18:08:52 localhost multipathd: ACTION=change 
Oct 30 18:08:52 localhost multipathd: DEVPATH=/block/dm-2 
Oct 30 18:08:52 localhost multipathd: SUBSYSTEM=block 
Oct 30 18:08:52 localhost multipathd: DM_TARGET=multipath 
Oct 30 18:08:52 localhost multipathd: DM_ACTION=PATH_FAILED 
Oct 30 18:08:52 localhost multipathd: DM_SEQNUM=1 
Oct 30 18:08:52 localhost multipathd: DM_PATH=8:80 
Oct 30 18:08:52 localhost multipathd: DM_NR_VALID_PATHS=1 
Oct 30 18:08:52 localhost multipathd: DM_NAME=mpathie 
Oct 30 18:08:52 localhost multipathd: DM_UUID=mpath-3600a0b80000f6a7d0000cff048fed59c 
Oct 30 18:08:52 localhost multipathd: MAJOR=253 
Oct 30 18:08:52 localhost multipathd: MINOR=2 
Oct 30 18:08:52 localhost multipathd: DEVTYPE=disk 
Oct 30 18:08:52 localhost multipathd: SEQNUM=1251 
Oct 30 18:08:52 localhost multipathd: UDEVD_EVENT=1 
Oct 30 18:08:52 localhost multipathd: dm-2: add map (uevent) 
Oct 30 18:08:54 localhost kernel: device-mapper: multipath: Failing path 8:64.
Oct 30 18:08:54 localhost UnixSmash4[8984]: 7:UnixSmash has experienced a write failure.
=======================TEST FAILED =====================================================
Oct 30 18:08:54 localhost UnixSmash4[8984]: The operation has failed and returned an error status.
Oct 30 18:08:54 localhost UnixSmash4[8984]: Target:/dev/mapper/mpathie SectorID:1067008 Blocksize:32 sectors LBA:0x0000000000104800 ErrorCode:5 ErrorMsg:Input/output error
Oct 30 18:08:54 localhost multipathd: mpathie: rr_weight = 2 (controller setting) 
Oct 30 18:08:54 localhost multipathd: mpathie: pgfailback = 15 (controller setting) 
Oct 30 18:08:54 localhost multipathd: mpathie: no_path_retry = -1 (controller setting) 
Oct 30 18:08:54 localhost multipathd: pg_timeout = NONE (internal default) 
Oct 30 18:08:54 localhost multipathd: 8:64: mark as failed 
Oct 30 18:08:54 localhost multipathd: uevent 'change' from '/block/dm-2' 
Oct 30 18:08:54 localhost multipathd: UDEV_LOG=3 
Oct 30 18:08:54 localhost multipathd: ACTION=change 
Oct 30 18:08:54 localhost multipathd: DEVPATH=/block/dm-2 
Oct 30 18:08:54 localhost multipathd: SUBSYSTEM=block 
Oct 30 18:08:54 localhost multipathd: DM_TARGET=multipath 
Oct 30 18:08:54 localhost multipathd: DM_ACTION=PATH_FAILED 
Oct 30 18:08:54 localhost multipathd: DM_SEQNUM=2 
Oct 30 18:08:54 localhost multipathd: DM_PATH=8:64 
Oct 30 18:08:54 localhost multipathd: DM_NR_VALID_PATHS=0 
Oct 30 18:08:54 localhost multipathd: DM_NAME=mpathie 
Oct 30 18:08:54 localhost multipathd: DM_UUID=mpath-3600a0b80000f6a7d0000cff048fed59c 
Oct 30 18:08:54 localhost multipathd: MAJOR=253 
Oct 30 18:08:54 localhost multipathd: MINOR=2 
Oct 30 18:08:54 localhost multipathd: DEVTYPE=disk 
Oct 30 18:08:54 localhost multipathd: SEQNUM=1252 
Oct 30 18:08:54 localhost multipathd: UDEVD_EVENT=1 
Oct 30 18:08:54 localhost multipathd: dm-2: add map (uevent) 
Oct 30 18:08:57 localhost UnixSmash4[8984]: 20:UnixSmash has ended after 2 minutes, 37 seconds with the following results:
Oct 30 18:08:57 localhost UnixSmash4[8984]: Thread    Status  Cycles  Retrys  Errors  Target
Oct 30 18:08:57 localhost UnixSmash4[8984]: f7f0bb90  ABRT         0       0       1  /dev/mapper/mpathie
Oct 30 18:09:21 localhost kernel:  rport-3:0-3: blocked FC remote port time out: removing target and saving binding
Oct 30 18:09:21 localhost multipathd: sdf: rdac checker reports path is down 
Oct 30 18:09:21 localhost kernel: sd 3:0:1:0: [sdf] Synchronizing SCSI cache
Oct 30 18:09:21 localhost kernel: sd 3:0:1:0: [sdf] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK,SUGGEST_OK
Oct 30 18:09:21 localhost kernel: scsi 3:0:1:0: rdac: Detached
Oct 30 18:09:22 localhost multipathd: uevent 'remove' from '/class/scsi_generic/sg6' 
Oct 30 18:09:22 localhost multipathd: UDEV_LOG=3 
Oct 30 18:09:22 localhost multipathd: ACTION=remove 
Oct 30 18:09:22 localhost multipathd: DEVPATH=/class/scsi_generic/sg6 
Oct 30 18:09:22 localhost multipathd: SUBSYSTEM=scsi_generic 
Oct 30 18:09:22 localhost multipathd: MAJOR=21 
Oct 30 18:09:22 localhost multipathd: MINOR=6 
Oct 30 18:09:22 localhost multipathd: PHYSDEVPATH=/devices/pci0000:00/0000:00:02.0/0000:06:00.3/0000:0b:01.0/host3/rport-3:0-3/target3:0:1/3:0:1:0 
Oct 30 18:09:22 localhost multipathd: PHYSDEVBUS=scsi 
Oct 30 18:09:22 localhost multipathd: PHYSDEVDRIVER=sd 
Oct 30 18:09:22 localhost multipathd: SEQNUM=1253 
Oct 30 18:09:22 localhost multipathd: UDEVD_EVENT=1 
Oct 30 18:09:22 localhost multipathd: DEVNAME=/dev/sg6 
Oct 30 18:09:22 localhost multipathd: uevent 'remove' from '/class/scsi_device/3:0:1:0' 
Oct 30 18:09:22 localhost multipathd: UDEV_LOG=3 
Oct 30 18:09:22 localhost multipathd: ACTION=remove 
Oct 30 18:09:22 localhost kernel: device-mapper: multipath: Failing path 8:64.
Oct 30 18:09:22 localhost multipathd: DEVPATH=/class/scsi_device/3:0:1:0 
Oct 30 18:09:22 localhost multipathd: SUBSYSTEM=scsi_device 
Oct 30 18:09:22 localhost multipathd: PHYSDEVPATH=/devices/pci0000:00/0000:00:02.0/0000:06:00.3/0000:0b:01.0/host3/rport-3:0-3/target3:0:1/3:0:1:0 
Oct 30 18:09:22 localhost multipathd: PHYSDEVBUS=scsi 
Oct 30 18:09:22 localhost multipathd: PHYSDEVDRIVER=sd 
Oct 30 18:09:22 localhost multipathd: SEQNUM=1254 
Oct 30 18:09:22 localhost multipathd: UDEVD_EVENT=1 
Oct 30 18:09:22 localhost multipathd: uevent 'remove' from '/class/scsi_disk/3:0:1:0' 
Oct 30 18:09:22 localhost multipathd: UDEV_LOG=3 
Oct 30 18:09:22 localhost multipathd: ACTION=remove 
Oct 30 18:09:22 localhost multipathd: DEVPATH=/class/scsi_disk/3:0:1:0 
Oct 30 18:09:22 localhost multipathd: SUBSYSTEM=scsi_disk 
Oct 30 18:09:22 localhost multipathd: PHYSDEVPATH=/devices/pci0000:00/0000:00:02.0/0000:06:00.3/0000:0b:01.0/host3/rport-3:0-3/target3:0:1/3:0:1:0 
Oct 30 18:09:22 localhost multipathd: PHYSDEVBUS=scsi 
Oct 30 18:09:22 localhost multipathd: PHYSDEVDRIVER=sd 
Oct 30 18:09:22 localhost multipathd: SEQNUM=1255 
Oct 30 18:09:22 localhost multipathd: UDEVD_EVENT=1 
Oct 30 18:09:22 localhost multipathd: uevent 'remove' from '/class/bdi/8:80' 
Oct 30 18:09:22 localhost multipathd: UDEV_LOG=3 
Oct 30 18:09:22 localhost multipathd: ACTION=remove 
Oct 30 18:09:22 localhost multipathd: DEVPATH=/class/bdi/8:80 
Oct 30 18:09:22 localhost multipathd: SUBSYSTEM=bdi 
Oct 30 18:09:22 localhost multipathd: SEQNUM=1256 
Oct 30 18:09:22 localhost multipathd: UDEVD_EVENT=1 
Oct 30 18:09:22 localhost multipathd: uevent 'remove' from '/block/sdf' 
Oct 30 18:09:22 localhost multipathd: UDEV_LOG=3 
Oct 30 18:09:22 localhost multipathd: ACTION=remove 
Oct 30 18:09:22 localhost multipathd: DEVPATH=/block/sdf 
Oct 30 18:09:22 localhost multipathd: SUBSYSTEM=block 
Oct 30 18:09:22 localhost multipathd: MAJOR=8 
Oct 30 18:09:22 localhost multipathd: MINOR=80 
Oct 30 18:09:22 localhost multipathd: DEVTYPE=disk 
Oct 30 18:09:22 localhost multipathd: PHYSDEVPATH=/devices/pci0000:00/0000:00:02.0/0000:06:00.3/0000:0b:01.0/host3/rport-3:0-3/target3:0:1/3:0:1:0 
Oct 30 18:09:22 localhost multipathd: PHYSDEVBUS=scsi 
Oct 30 18:09:22 localhost multipathd: PHYSDEVDRIVER=sd 
Oct 30 18:09:22 localhost multipathd: SEQNUM=1257 
Oct 30 18:09:22 localhost multipathd: UDEVD_EVENT=1 
Oct 30 18:09:22 localhost multipathd: ID_VENDOR=LSI 
Oct 30 18:09:22 localhost multipathd: ID_MODEL=INF-01-00 
Oct 30 18:09:22 localhost multipathd: ID_REVISION=9660 
Oct 30 18:09:22 localhost multipathd: ID_SERIAL=3600a0b80000f6a7d0000cff048fed59c 
Oct 30 18:09:22 localhost multipathd: ID_TYPE=disk 
Oct 30 18:09:22 localhost multipathd: ID_BUS=scsi 
Oct 30 18:09:22 localhost multipathd: ID_PATH=pci-0000:0b:01.0-fc-0x200500a0b80f6a7e:0x0000000000000000 
Oct 30 18:09:22 localhost multipathd: DEVLINKS=/dev/disk/by-path/pci-0000:0b:01.0-fc-0x200500a0b80f6a7e:0x0000000000000000 
Oct 30 18:09:22 localhost multipathd: DEVNAME=/dev/sdf 
Oct 30 18:09:22 localhost multipathd: sdf: remove path (uevent) 
Oct 30 18:09:22 localhost multipathd: 8:80: not found in pathvec 
Oct 30 18:09:22 localhost multipathd: 8:64: not found in pathvec 
Oct 30 18:09:22 localhost multipathd: mpathie: pgfailback = 15 (controller setting) 
Oct 30 18:09:22 localhost multipathd: mpathie: pgpolicy = group_by_prio (controller setting) 
Oct 30 18:09:22 localhost multipathd: mpathie: selector = round-robin 0 (internal default) 
Oct 30 18:09:22 localhost multipathd: mpathie: features = 0 (internal default) 
Oct 30 18:09:22 localhost multipathd: mpathie: hwhandler = 1 rdac (controller setting) 
Oct 30 18:09:22 localhost multipathd: mpathie: rr_weight = 2 (controller setting) 
Oct 30 18:09:22 localhost multipathd: mpathie: minio = 100 (config file default) 
Oct 30 18:09:22 localhost multipathd: mpathie: no_path_retry = -1 (controller setting) 
Oct 30 18:09:22 localhost multipathd: pg_timeout = NONE (internal default) 
Oct 30 18:09:22 localhost multipathd: uevent 'remove' from '/devices/pci0000:00/0000:00:02.0/0000:06:00.3/0000:0b:01.0/host3/rport-3:0-3/target3:0:1/3:0:1:0' 
Oct 30 18:09:22 localhost multipathd: UDEV_LOG=3 
Oct 30 18:09:22 localhost multipathd: ACTION=remove 
Oct 30 18:09:22 localhost multipathd: DEVPATH=/devices/pci0000:00/0000:00:02.0/0000:06:00.3/0000:0b:01.0/host3/rport-3:0-3/target3:0:1/3:0:1:0 
Oct 30 18:09:22 localhost multipathd: SUBSYSTEM=scsi 
Oct 30 18:09:22 localhost multipathd: DEVTYPE=scsi_device 
Oct 30 18:09:22 localhost multipathd: PHYSDEVBUS=scsi 
Oct 30 18:09:22 localhost multipathd: MODALIAS=scsi:t-0x00 
Oct 30 18:09:22 localhost multipathd: SEQNUM=1258 
Oct 30 18:09:22 localhost multipathd: UDEVD_EVENT=1 
Oct 30 18:09:22 localhost multipathd: mpathie: load table [0 20971520 multipath 0 1 rdac 1 1 round-robin 0 1 1 8:64 100] 
Oct 30 18:09:22 localhost multipathd: mpathie: discover 
Oct 30 18:09:22 localhost multipathd: mpathie: rr_weight = 2 (controller setting) 
Oct 30 18:09:22 localhost multipathd: mpathie: pgfailback = 15 (controller setting) 
Oct 30 18:09:22 localhost multipathd: pg_timeout = NONE (internal default) 
Oct 30 18:09:22 localhost multipathd: sdf: path removed from map mpathie 
Oct 30 18:09:22 localhost multipathd: mpathie: devmap event #4 
Oct 30 18:09:22 localhost multipathd: mpathie: discover 
Oct 30 18:09:22 localhost multipathd: mpathie: rr_weight = 2 (controller setting) 
Oct 30 18:09:22 localhost multipathd: mpathie: pgfailback = 15 (controller setting) 
Oct 30 18:09:22 localhost multipathd: pg_timeout = NONE (internal default) 
Oct 30 18:09:22 localhost multipathd: uevent 'remove' from '/class/fc_transport/target3:0:1' 
Oct 30 18:09:22 localhost multipathd: UDEV_LOG=3 
Oct 30 18:09:22 localhost multipathd: ACTION=remove 
Oct 30 18:09:22 localhost multipathd: DEVPATH=/class/fc_transport/target3:0:1 
Oct 30 18:09:22 localhost multipathd: SUBSYSTEM=fc_transport 
Oct 30 18:09:22 localhost multipathd: PHYSDEVPATH=/devices/pci0000:00/0000:00:02.0/0000:06:00.3/0000:0b:01.0 
Oct 30 18:09:22 localhost multipathd: PHYSDEVBUS=pci 
Oct 30 18:09:22 localhost multipathd: PHYSDEVDRIVER=lpfc 
Oct 30 18:09:22 localhost multipathd: SEQNUM=1260 
Oct 30 18:09:22 localhost multipathd: UDEVD_EVENT=1 
Oct 30 18:09:22 localhost multipathd: uevent 'change' from '/block/dm-2' 
Oct 30 18:09:22 localhost multipathd: UDEV_LOG=3 
Oct 30 18:09:22 localhost multipathd: ACTION=change 
Oct 30 18:09:22 localhost multipathd: DEVPATH=/block/dm-2 
Oct 30 18:09:22 localhost multipathd: SUBSYSTEM=block 
Oct 30 18:09:22 localhost multipathd: MAJOR=253 
Oct 30 18:09:22 localhost multipathd: MINOR=2 
Oct 30 18:09:22 localhost multipathd: DEVTYPE=disk 
Oct 30 18:09:22 localhost multipathd: SEQNUM=1259 
Oct 30 18:09:22 localhost multipathd: UDEVD_EVENT=1 
Oct 30 18:09:22 localhost multipathd: dm-2: add map (uevent) 
Oct 30 18:09:22 localhost multipathd: dm-2: devmap already registered 
Oct 30 18:09:22 localhost multipathd: uevent 'change' from '/block/dm-2' 
Oct 30 18:09:22 localhost multipathd: UDEV_LOG=3 
Oct 30 18:09:22 localhost multipathd: ACTION=change 
Oct 30 18:09:22 localhost multipathd: DEVPATH=/block/dm-2 
Oct 30 18:09:22 localhost multipathd: SUBSYSTEM=block 
Oct 30 18:09:22 localhost multipathd: DM_TARGET=multipath 
Oct 30 18:09:22 localhost multipathd: DM_ACTION=PATH_FAILED 
Oct 30 18:09:22 localhost multipathd: DM_SEQNUM=3 
Oct 30 18:09:22 localhost multipathd: DM_PATH=8:64 
Oct 30 18:09:22 localhost multipathd: DM_NR_VALID_PATHS=0 
Oct 30 18:09:23 localhost multipathd: DM_NAME=mpathie 
Oct 30 18:09:23 localhost multipathd: DM_UUID=mpath-3600a0b80000f6a7d0000cff048fed59c 
Oct 30 18:09:23 localhost multipathd: MAJOR=253 
Oct 30 18:09:23 localhost multipathd: MINOR=2 
Oct 30 18:09:23 localhost multipathd: DEVTYPE=disk 
Oct 30 18:09:23 localhost multipathd: SEQNUM=1261 
Oct 30 18:09:23 localhost multipathd: UDEVD_EVENT=1 
Oct 30 18:09:23 localhost multipathd: dm-2: add map (uevent) 
Oct 30 18:09:23 localhost multipathd: dm-2: devmap already registered 
Oct 30 18:09:23 localhost multipathd: 8:64: reinstated 
Oct 30 18:09:23 localhost multipathd: mpathie: remaining active paths: 1 
Oct 30 18:09:23 localhost multipathd: sde: mask = 0x8 
Oct 30 18:09:23 localhost multipathd: sde: rdac prio = 1 
Oct 30 18:09:23 localhost multipathd: mpathie: rr_weight = 2 (controller setting) 
Oct 30 18:09:23 localhost multipathd: mpathie: pgfailback = 15 (controller setting) 
Oct 30 18:09:23 localhost multipathd: mpathie: no_path_retry = -1 (controller setting) 
Oct 30 18:09:24 localhost multipathd: uevent 'change' from '/block/dm-2' 
Oct 30 18:09:24 localhost multipathd: UDEV_LOG=3 
Oct 30 18:09:24 localhost multipathd: ACTION=change 
Oct 30 18:09:24 localhost multipathd: DEVPATH=/block/dm-2 
Oct 30 18:09:24 localhost multipathd: SUBSYSTEM=block 
Oct 30 18:09:24 localhost multipathd: DM_TARGET=multipath 
Oct 30 18:09:24 localhost multipathd: DM_ACTION=PATH_REINSTATED 
Oct 30 18:09:24 localhost multipathd: DM_SEQNUM=4 
Oct 30 18:09:24 localhost multipathd: DM_PATH=8:64 
Oct 30 18:09:24 localhost multipathd: DM_NR_VALID_PATHS=1 
Oct 30 18:09:24 localhost multipathd: DM_NAME=mpathie 
Oct 30 18:09:24 localhost multipathd: DM_UUID=mpath-3600a0b80000f6a7d0000cff048fed59c 
Oct 30 18:09:24 localhost multipathd: MAJOR=253 
Oct 30 18:09:24 localhost multipathd: MINOR=2 
Oct 30 18:09:24 localhost multipathd: DEVTYPE=disk 
Oct 30 18:09:24 localhost multipathd: SEQNUM=1262 
Oct 30 18:09:24 localhost multipathd: UDEVD_EVENT=1 
Oct 30 18:09:24 localhost multipathd: dm-2: add map (uevent) 
Oct 30 18:09:53 localhost multipathd: sde: rdac prio = 1 

RE: [dm-devel] i/o error due to all path failure with rdac

[Chandra Seetharaman]

Can you try this patch.
---------
Signed-off-by: Chandra Seetharaman <sekharan@us.ibm.com>

Index: linux-2.6.27/drivers/scsi/device_handler/scsi_dh_rdac.c
===================================================================
--- linux-2.6.27.orig/drivers/scsi/device_handler/scsi_dh_rdac.c
+++ linux-2.6.27/drivers/scsi/device_handler/scsi_dh_rdac.c
@@ -386,6 +386,7 @@ static int check_ownership(struct scsi_d
 	struct c9_inquiry *inqp;
 
 	h->lun_state = RDAC_LUN_UNOWNED;
+	h->state = RDAC_STATE_ACTIVE;
 	err = submit_inquiry(sdev, 0xC9, sizeof(struct c9_inquiry), h);
 	if (err == SCSI_DH_OK) {
 		inqp = &h->inq.c9;
---

On Thu, 2008-10-30 at 17:21 -0600, Moger, Babu wrote:
> Looks like eventually we get rport failure after the controller offline.
> 
> 
> I have attached the messages file.  I am running raw IO.
> 
> Also, please note that I have added following lines in check_sense. This condition (quiescence) should be retried.
> 
> 
> case UNIT_ATTENTION:
>                 if ((sense_hdr->asc == 0x29 && sense_hdr->ascq == 0x00) ||
>                     (sense_hdr->asc == 0x8b && sense_hdr->ascq == 0x02))
>                         /*
>                          * Power On, Reset, or Bus Device Reset, just retry.
>                          */
>                         return ADD_TO_MLQUEUE;
> 
> 
> 
> Thanks
> Babu Moger
> 
> -----Original Message-----
> From: Chandra Seetharaman [mailto:sekharan@us.ibm.com]
> Sent: Thursday, October 30, 2008 5:24 PM
> To: Moger, Babu
> Cc: device-mapper development; linux-scsi@vger.kernel.org
> Subject: RE: [dm-devel] i/o error due to all path failure with rdac
> 
> 
> On Thu, 2008-10-30 at 14:30 -0600, Moger, Babu wrote:
> > This is what happens in my case
> >
> > When the active path is failed, the dh handler calls rdac_activate to activate the passive path.
> > Then check_ownership is called. As you know check_ownership sends inquiry (page c9). Looking at the
> > response this function sets the lun_state(h->lun_state) to RDAC_LUN_OWNED.
> >
> > If lun_state is set to RDAC_LUN_OWNED then send_mode_select will not be called. This is what
> > happens in my case.
> 
> Ok. Now it is clear. I thought the port was disabled.
> 
> The rport failure in the log you sent made me think that it was a port
> disable. Why was there a rport failure ?
> 
> Can you add these two lines at the top of
> scsi_dn_rdac.c:rdac_check_sense() function, retest and send me the log.
> ------------
>         sdev_printk(KERN_ERR, sdev, "sense_key:%x; asc %c; ascq %x\n",
>                 sense_hdr->sense_key, sense_hdr->asc, sense_hdr->ascq);
> -----------
> I want to see if we are getting any special sense that we are not
> handling.

RE: i/o error due to all path failure with rdac

[Moger, Babu]

Yes, It is working fine with this patch. Now my online/offline test is running fine. Thank you very much. That was really a quick fix.

I am still learning the device mapper. Hopefully I will be some help in the future.

Thanks
Babu Moger

-----Original Message-----
From: Chandra Seetharaman [mailto:sekharan@us.ibm.com]
Sent: Thursday, October 30, 2008 6:35 PM
To: Moger, Babu
Cc: device-mapper development; linux-scsi@vger.kernel.org
Subject: RE: [dm-devel] i/o error due to all path failure with rdac

Can you try this patch.
---------
Signed-off-by: Chandra Seetharaman <sekharan@us.ibm.com>

Index: linux-2.6.27/drivers/scsi/device_handler/scsi_dh_rdac.c
===================================================================
--- linux-2.6.27.orig/drivers/scsi/device_handler/scsi_dh_rdac.c
+++ linux-2.6.27/drivers/scsi/device_handler/scsi_dh_rdac.c
@@ -386,6 +386,7 @@ static int check_ownership(struct scsi_d
        struct c9_inquiry *inqp;

        h->lun_state = RDAC_LUN_UNOWNED;
+       h->state = RDAC_STATE_ACTIVE;
        err = submit_inquiry(sdev, 0xC9, sizeof(struct c9_inquiry), h);
        if (err == SCSI_DH_OK) {
                inqp = &h->inq.c9;
---

On Thu, 2008-10-30 at 17:21 -0600, Moger, Babu wrote:
> Looks like eventually we get rport failure after the controller offline.
>
>
> I have attached the messages file.  I am running raw IO.
>
> Also, please note that I have added following lines in check_sense. This condition (quiescence) should be retried.
>
>
> case UNIT_ATTENTION:
>                 if ((sense_hdr->asc == 0x29 && sense_hdr->ascq == 0x00) ||
>                     (sense_hdr->asc == 0x8b && sense_hdr->ascq == 0x02))
>                         /*
>                          * Power On, Reset, or Bus Device Reset, just retry.
>                          */
>                         return ADD_TO_MLQUEUE;
>
>
>
> Thanks
> Babu Moger
>

RE: [dm-devel] i/o error due to all path failure with rdac

[Chandra Seetharaman]

That is good to know. I will push the patch upstream.

BTW, Please be advised that you might see some unknown problems if you
use the multipath tools from mainline on your distro releases.

The library in distro releases and the tools in mainline may not be
totally compatible, which is the cause.

I would suggest you to use the tools from your distro itself. The kernel
changes for SCSI_DH should work with no changes to the tools (in your
distro). IOW, you do not need the latest multipath tools.

chandra
On Fri, 2008-10-31 at 10:05 -0600, Moger, Babu wrote:
> Yes, It is working fine with this patch. Now my online/offline test is running fine. Thank you very much. That was really a quick fix.
> 
> I am still learning the device mapper. Hopefully I will be some help in the future.
> 
> Thanks
> Babu Moger
> 
> -----Original Message-----
> From: Chandra Seetharaman [mailto:sekharan@us.ibm.com]
> Sent: Thursday, October 30, 2008 6:35 PM
> To: Moger, Babu
> Cc: device-mapper development; linux-scsi@vger.kernel.org
> Subject: RE: [dm-devel] i/o error due to all path failure with rdac
> 
> Can you try this patch.
> ---------
> Signed-off-by: Chandra Seetharaman <sekharan@us.ibm.com>
> 
> Index: linux-2.6.27/drivers/scsi/device_handler/scsi_dh_rdac.c
> ===================================================================
> --- linux-2.6.27.orig/drivers/scsi/device_handler/scsi_dh_rdac.c
> +++ linux-2.6.27/drivers/scsi/device_handler/scsi_dh_rdac.c
> @@ -386,6 +386,7 @@ static int check_ownership(struct scsi_d
>         struct c9_inquiry *inqp;
> 
>         h->lun_state = RDAC_LUN_UNOWNED;
> +       h->state = RDAC_STATE_ACTIVE;
>         err = submit_inquiry(sdev, 0xC9, sizeof(struct c9_inquiry), h);
>         if (err == SCSI_DH_OK) {
>                 inqp = &h->inq.c9;
> ---
> 
> On Thu, 2008-10-30 at 17:21 -0600, Moger, Babu wrote:
> > Looks like eventually we get rport failure after the controller offline.
> >
> >
> > I have attached the messages file.  I am running raw IO.
> >
> > Also, please note that I have added following lines in check_sense. This condition (quiescence) should be retried.
> >
> >
> > case UNIT_ATTENTION:
> >                 if ((sense_hdr->asc == 0x29 && sense_hdr->ascq == 0x00) ||
> >                     (sense_hdr->asc == 0x8b && sense_hdr->ascq == 0x02))
> >                         /*
> >                          * Power On, Reset, or Bus Device Reset, just retry.
> >                          */
> >                         return ADD_TO_MLQUEUE;
> >
> >
> >
> > Thanks
> > Babu Moger
> >

Re: [dm-devel] i/o error due to all path failure with rdac

[Chandra Seetharaman]

[-- Attachment #1: Type: text/plain, Size: 6510 bytes --]

Hi Babu,

As Mike asked, can you provide the kernel version and the multipath
version.

Can you also provide the var/log/messages file from the start (before
you fail the first path) to the finish of your test.

Also, what kind of I/Os are you running.

BTW, if it is mainline code, can you apply the attached patch and see if
you see any better behavior.

regards,

chandra
On Fri, 2008-10-24 at 17:11 -0600, Moger, Babu wrote:
> Hi,
> 
>   I am running an online/offline test. I have two paths to the controller. One is active and one is passive. When I fail (offline) the active path (sde 8:64), the Device mapper is failing passive path(sdf 8:80) as well leading to all path failure.  Any ideas or hints?
> 
> Here is output multipath -ll. I have only one lun.
> 
> [root@localhost ~]# multipath -ll
> mpathie (3600a0b80000f6a7d0000cff048fed59c) dm-2 LSI,INF-01-00
> [size=10G][features=1 queue_if_no_path][hwhandler=1 rdac][rw]
> \_ round-robin 0 [prio=2][enabled]
>  \_ 3:0:0:0 sde 8:64  [active][undef]
> \_ round-robin 0 [prio=1][enabled]
>  \_ 3:0:1:0 sdf 8:80  [active][undef]
> 
> 
> Here is the detailed log.
> 
> Oct 24 16:50:50 localhost multipathd: sdf: rdac prio = 0
> Oct 24 16:51:06 localhost kernel: sd 3:0:0:0: [sde] Result: hostbyte=DID_BUS_BUSY driverbyte=DRIVER_OK,SUGGEST_OK
> Oct 24 16:51:06 localhost kernel: end_request: I/O error, dev sde, sector 1047072
> Oct 24 16:51:06 localhost kernel: device-mapper: multipath: Failing path 8:64.
> Oct 24 16:51:06 localhost multipathd: mpathie: rr_weight = 2 (controller setting)
> Oct 24 16:51:06 localhost multipathd: mpathie: pgfailback = 100 (controller setting)
> Oct 24 16:51:06 localhost multipathd: mpathie: no_path_retry = 10 (controller setting)
> Oct 24 16:51:06 localhost multipathd: pg_timeout = NONE (internal default)
> Oct 24 16:51:06 localhost multipathd: 8:64: mark as failed
> Oct 24 16:51:06 localhost multipathd: uevent 'change' from '/block/dm-2'
> Oct 24 16:51:06 localhost multipathd: UDEV_LOG=3
> Oct 24 16:51:06 localhost multipathd: ACTION=change
> Oct 24 16:51:06 localhost multipathd: DEVPATH=/block/dm-2
> Oct 24 16:51:06 localhost multipathd: SUBSYSTEM=block
> Oct 24 16:51:06 localhost multipathd: DM_TARGET=multipath
> Oct 24 16:51:06 localhost multipathd: DM_ACTION=PATH_FAILED
> Oct 24 16:51:06 localhost multipathd: DM_SEQNUM=1
> Oct 24 16:51:06 localhost multipathd: DM_PATH=8:64
> Oct 24 16:51:06 localhost multipathd: DM_NR_VALID_PATHS=1
> Oct 24 16:51:06 localhost multipathd: DM_NAME=mpathie
> Oct 24 16:51:06 localhost multipathd: DM_UUID=mpath-3600a0b80000f6a7d0000cff048fed59c
> Oct 24 16:51:06 localhost multipathd: MAJOR=253
> Oct 24 16:51:06 localhost multipathd: MINOR=2
> Oct 24 16:51:06 localhost multipathd: DEVTYPE=disk
> Oct 24 16:51:06 localhost multipathd: SEQNUM=1254
> Oct 24 16:51:06 localhost multipathd: UDEVD_EVENT=1
> Oct 24 16:51:06 localhost multipathd: dm-2: add map (uevent)
> Oct 24 16:51:08 localhost kernel: device-mapper: multipath: Failing path 8:80.
> Oct 24 16:51:08 localhost multipathd: mpathie: devmap event #3
> Oct 24 16:51:08 localhost multipathd: mpathie: discover
> Oct 24 16:51:08 localhost multipathd: mpathie: rr_weight = 2 (controller setting)
> Oct 24 16:51:08 localhost multipathd: mpathie: pgfailback = 100 (controller setting)
> Oct 24 16:51:08 localhost multipathd: mpathie: no_path_retry = 10 (controller setting)
> Oct 24 16:51:08 localhost multipathd: pg_timeout = NONE (internal default)
> Oct 24 16:51:08 localhost multipathd: 8:80: mark as failed
> Oct 24 16:51:08 localhost multipathd: mpathie: Entering recovery mode: max_retries=10
> Oct 24 16:51:08 localhost multipathd: uevent 'change' from '/block/dm-2'
> Oct 24 16:51:08 localhost multipathd: UDEV_LOG=3
> Oct 24 16:51:08 localhost multipathd: ACTION=change
> Oct 24 16:51:08 localhost multipathd: DEVPATH=/block/dm-2
> Oct 24 16:51:08 localhost multipathd: SUBSYSTEM=block
> Oct 24 16:51:08 localhost multipathd: DM_TARGET=multipath
> Oct 24 16:51:08 localhost multipathd: DM_ACTION=PATH_FAILED
> Oct 24 16:51:08 localhost multipathd: DM_SEQNUM=2
> Oct 24 16:51:08 localhost multipathd: DM_PATH=8:80
> Oct 24 16:51:08 localhost multipathd: DM_NR_VALID_PATHS=0
> Oct 24 16:51:08 localhost multipathd: DM_NAME=mpathie
> Oct 24 16:51:08 localhost multipathd: DM_UUID=mpath-3600a0b80000f6a7d0000cff048fed59c
> Oct 24 16:51:08 localhost multipathd: MAJOR=253
> Oct 24 16:51:08 localhost multipathd: MINOR=2
> Oct 24 16:51:08 localhost multipathd: DEVTYPE=disk
> Oct 24 16:51:08 localhost multipathd: SEQNUM=1255
> Oct 24 16:51:08 localhost multipathd: UDEVD_EVENT=1
> Oct 24 16:51:08 localhost multipathd: dm-2: add map (uevent)
> Oct 24 16:51:36 localhost kernel:  rport-3:0-2: blocked FC remote port time out: removing target and saving binding
> Oct 24 16:51:36 localhost multipathd: sde: rdac checker reports path is down
> Oct 24 16:51:36 localhost multipathd: sde: mask = 0x8
> Oct 24 16:51:36 localhost kernel: sd 3:0:0:0: [sde] Synchronizing SCSI cache
> Oct 24 16:51:36 localhost kernel: sd 3:0:0:0: [sde] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK,SUGGEST_OK
> Oct 24 16:51:36 localhost kernel: scsi 3:0:0:0: rdac: Detached
> Oct 24 16:51:36 localhost multipathd: uevent 'remove' from '/class/scsi_generic/sg5'
> Oct 24 16:51:36 localhost multipathd: UDEV_LOG=3
> Oct 24 16:51:36 localhost multipathd: ACTION=remove
> Oct 24 16:51:36 localhost multipathd: DEVPATH=/class/scsi_generic/sg5
> Oct 24 16:51:36 localhost multipathd: SUBSYSTEM=scsi_generic
> Oct 24 16:51:36 localhost multipathd: MAJOR=21
> Oct 24 16:51:36 localhost multipathd: MINOR=5
> Oct 24 16:51:36 localhost multipathd: PHYSDEVPATH=/devices/pci0000:00/0000:00:02.0/0000:06:00.3/0000:0b:01.0/host3/rport-3:0-2/target3:0:0/3:0:0:0
> Oct 24 16:51:36 localhost multipathd: PHYSDEVBUS=scsi
> Oct 24 16:51:36 localhost multipathd: PHYSDEVDRIVER=sd
> Oct 24 16:51:36 localhost multipathd: SEQNUM=1256
> Oct 24 16:51:36 localhost multipathd: UDEVD_EVENT=1
> Oct 24 16:51:36 localhost multipathd: DEVNAME=/dev/sg5
> Oct 24 16:51:36 localhost multipathd: uevent 'remove' from '/class/scsi_device/3:0:0:0'
> Oct 24 16:51:36 localhost multipathd: UDEV_LOG=3
> Oct 24 16:51:36 localhost kernel: device-mapper: multipath: Failing path 8:80.
> Oct 24 16:51:36 localhost multipathd: ACTION=remove
> Oct 24 16:51:36 localhost UnixSmash4[9200]: 7:UnixSmash has experienced a write failure.
> 
> Thanks
> Babu Moger
> 
> 
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel

[-- Attachment #2: retry_mode_select --]
[-- Type: text/plain, Size: 1317 bytes --]


Retry mode select.

Signed-off-by: Chandra Seetharaman <sekharan@us.ibm.com>

Index: linux-2.6.27/drivers/scsi/device_handler/scsi_dh_rdac.c
===================================================================
--- linux-2.6.27.orig/drivers/scsi/device_handler/scsi_dh_rdac.c
+++ linux-2.6.27/drivers/scsi/device_handler/scsi_dh_rdac.c
@@ -24,6 +24,7 @@
 #include <scsi/scsi_dh.h>
 
 #define RDAC_NAME "rdac"
+#define RDAC_RETRY_COUNT 5
 
 /*
  * LSI mode page stuff
@@ -476,21 +477,27 @@ static int send_mode_select(struct scsi_
 {
 	struct request *rq;
 	struct request_queue *q = sdev->request_queue;
-	int err = SCSI_DH_RES_TEMP_UNAVAIL;
+	int err, retry_cnt = RDAC_RETRY_COUNT;
 
+retry:
+	err = SCSI_DH_RES_TEMP_UNAVAIL;
 	rq = rdac_failover_get(sdev, h);
 	if (!rq)
 		goto done;
 
-	sdev_printk(KERN_INFO, sdev, "queueing MODE_SELECT command.\n");
+	sdev_printk(KERN_INFO, sdev, "%s MODE_SELECT command.\n",
+		(retry_cnt == RDAC_RETRY_COUNT) ? "queueing" : "retrying");
 
 	err = blk_execute_rq(q, NULL, rq, 1);
-	if (err != SCSI_DH_OK)
+	blk_put_request(rq);
+	if (err != SCSI_DH_OK) {
 		err = mode_select_handle_sense(sdev, h->sense);
+		if (err == SCSI_DH_RETRY && retry_cnt--)
+			goto retry;
+	}
 	if (err == SCSI_DH_OK)
 		h->state = RDAC_STATE_ACTIVE;
 
-	blk_put_request(rq);
 done:
 	return err;
 }

Re: Buffer I/O error

[Mike Anderson]

Kit Westneat <kwestneat@datadirectnet.com> wrote:
> Hi dm-devel,
>
> I'm running into an issue with dm-multipath  
> (device-mapper-multipath-0.4.5-27.el4_6.3) that has me confused and I  
> was hoping I could get some guidance.
>
> Here are the relevant dmesg messages:
> device-mapper: dm-multipath: Failing path 8:48.
> device-mapper: dm-multipath: Failing path 8:48.
> device-mapper: dm-multipath: Failing path 8:48.
> device-mapper: dm-multipath: Failing path 65:80.
> printk: 1004 messages suppressed.
> Buffer I/O error on device dm-13, logical block 1192525829
> lost page write due to I/O error on dm-13
>

What is the output of multipath -ll. I would assume that you have
queue_if_no_path no set. The "Buffer I/O error on device dm-13" is the dm
failing back the IO vs. possible previous errors that you saw accompanied
by SCSI errors where sd is failing the IO back to dm.

If you not seeing any SCSI errors, I would assume that it is the patch
checker that is failing your paths. Can try using the tur checker to see
if this changes your error signature?

> Normally when I have seen Buffer I/O errors in the past, they have been  
> accompanied by SCSI errors as well, but in this case I am not getting  
> anything. One odd thing is that the filesystem reported extremely slow  
> writes before the buffer I/O errors occurred. I have checked and  
> double-checked the storage array and everything seems fine. I am using  
> directio as my path checker if that makes a difference.
>
> I have been able to reproduce the problem consistently by attempting an  
> fsck on dm-13. However, I am now doing an fsck on the underlying sd  
> device and it hasn't bombed out yet with the buffer I/O errors.
>
> Has anyone encountered these seemingly inexplicable buffer I/O errors  
> before? Any advice on how to debug it?

-andmike
--
Michael Anderson
andmike@linux.vnet.ibm.com

buffer i/o error

[Andrej Hocevar]

Hello, I suddenly realized that one of my external disks (a Western Digital
MyBook usb disk) wasn't accessible anymore. Then I found the following message.
There seems to have been no apparent damage or loss, but I'm reporting it
anyway.

Regards,

Andrej Hocevar


------------[ cut here ]------------
WARNING: at fs/buffer.c:1186 mark_buffer_dirty+0x20/0x69()
Modules linked in: nls_cp437 isofs loop xt_state xt_tcpudp iptable_filter
ip_tables x_tables nf_nat_ftp nf_nat nf_conntrack_ipv4 nfDec  3 16:24:58 stoa
kernel: Pid: 12202, comm: umount Not tainted 2.6.27.7home #1
 [<c0118eae>] warn_on_slowpath+0x40/0x63
 [<c0151c22>] check_object+0x10d/0x197
 [<c0152e76>] __slab_alloc+0x366/0x3de
 [<c011394b>] place_entity+0x9c/0xde
 [<c011388f>] update_curr+0x3d/0x5d
 [<c01149a0>] __dequeue_entity+0x1f/0x71
 [<c01158f0>] pick_next_task_fair+0xdf/0xe9
 [<c016fe20>] mark_buffer_dirty+0x20/0x69
 [<c0198b88>] journal_update_superblock+0x4a/0x7c
 [<c0198e00>] journal_destroy+0x101/0x146
 [<c0126742>] autoremove_wake_function+0x0/0x2d
 [<c018fcfd>] ext3_put_super+0x1f/0x147
 [<c01578a6>] generic_shutdown_super+0x4a/0xad
 [<c0157915>] kill_block_super+0xc/0x1b
 [<c0157983>] deactivate_super+0x2c/0x3f
 [<c0167772>] sys_umount+0x253/0x279
 [<c01677a3>] sys_oldumount+0xb/0xe
 [<c0102b36>] syscall_call+0x7/0xb
 [<c0310000>] via_ircc_open+0xfa/0x59b
 =======================
---[ end trace af37a7a2247422ea ]---

 
Linux stoa 2.6.27.7home #1 Sun Nov 23 14:37:46 CET 2008 i686 GNU/Linux
 
Gnu C                  4.3.2
Gnu make               3.81
binutils               2.18.0.20080103
util-linux             2.13.1.1
mount                  2.13.1
module-init-tools      found
PPP                    2.4.4
Linux C Library        2.7
Dynamic linker (ldd)   2.7
Procps                 3.2.7
Console-tools          0.2.3
Sh-utils               6.10
udev                   125
Modules Loaded         nls_cp437 isofs loop xt_state xt_tcpudp iptable_filter
ip_tables x_tables nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_conntrack_ftp
nf_conntrack snd_mixer_oss ppdev lp cpufreq_stats fuse sr_mod sbp2 snd_intel8x0
snd_ac97_codec ac97_bus 8250_pci 8250 snd_pcm snd_timer rng_core serial_core
snd_page_alloc video output parport_pc parport sd_mod evdev ide_cd_mod cdrom
ide_disk usb_storage usbhid scsi_mod piix ohci1394 ide_core ieee1394 ehci_hcd
uhci_hcd usbcore unix

--
www.ljudmila.org/ludliteratura

Re: buffer i/o error

[Stefan Richter]

Andrej Hocevar wrote:
> Hello, I suddenly realized that one of my external disks (a Western Digital
> MyBook usb disk) wasn't accessible anymore. Then I found the following message.
> There seems to have been no apparent damage or loss, but I'm reporting it
> anyway.
> 
> Regards,
> 
> Andrej Hocevar
> 
> 
> ------------[ cut here ]------------
> WARNING: at fs/buffer.c:1186 mark_buffer_dirty+0x20/0x69()
[...]
>  [<c016fe20>] mark_buffer_dirty+0x20/0x69
>  [<c0198b88>] journal_update_superblock+0x4a/0x7c
>  [<c0198e00>] journal_destroy+0x101/0x146
>  [<c0126742>] autoremove_wake_function+0x0/0x2d
>  [<c018fcfd>] ext3_put_super+0x1f/0x147
>  [<c01578a6>] generic_shutdown_super+0x4a/0xad
>  [<c0157915>] kill_block_super+0xc/0x1b
[...]

This warning typically happens when the connection to a disk was lost.
Alas it doesn't tell /why/ the connection was lost.

http://kerneloops.org/guilty.php?guilty=journal_update_superblock has
more of this sort.
-- 
Stefan Richter
-=====-==--- ==-- ---==
http://arcgraph.de/sr/