Buffer I/O error

Buffer I/O error

[Kit Westneat]

Hi dm-devel,

I'm running into an issue with dm-multipath 
(device-mapper-multipath-0.4.5-27.el4_6.3) that has me confused and I 
was hoping I could get some guidance.

Here are the relevant dmesg messages:
device-mapper: dm-multipath: Failing path 8:48.
device-mapper: dm-multipath: Failing path 8:48.
device-mapper: dm-multipath: Failing path 8:48.
device-mapper: dm-multipath: Failing path 65:80.
printk: 1004 messages suppressed.
Buffer I/O error on device dm-13, logical block 1192525829
lost page write due to I/O error on dm-13

Normally when I have seen Buffer I/O errors in the past, they have been 
accompanied by SCSI errors as well, but in this case I am not getting 
anything. One odd thing is that the filesystem reported extremely slow 
writes before the buffer I/O errors occurred. I have checked and 
double-checked the storage array and everything seems fine. I am using 
directio as my path checker if that makes a difference.

I have been able to reproduce the problem consistently by attempting an 
fsck on dm-13. However, I am now doing an fsck on the underlying sd 
device and it hasn't bombed out yet with the buffer I/O errors.

Has anyone encountered these seemingly inexplicable buffer I/O errors 
before? Any advice on how to debug it?

Thanks,
Kit Westneat

Re: Buffer I/O error

[Mike Anderson]

Kit Westneat <kwestneat@datadirectnet.com> wrote:
> Hi dm-devel,
>
> I'm running into an issue with dm-multipath  
> (device-mapper-multipath-0.4.5-27.el4_6.3) that has me confused and I  
> was hoping I could get some guidance.
>
> Here are the relevant dmesg messages:
> device-mapper: dm-multipath: Failing path 8:48.
> device-mapper: dm-multipath: Failing path 8:48.
> device-mapper: dm-multipath: Failing path 8:48.
> device-mapper: dm-multipath: Failing path 65:80.
> printk: 1004 messages suppressed.
> Buffer I/O error on device dm-13, logical block 1192525829
> lost page write due to I/O error on dm-13
>

What is the output of multipath -ll. I would assume that you have
queue_if_no_path no set. The "Buffer I/O error on device dm-13" is the dm
failing back the IO vs. possible previous errors that you saw accompanied
by SCSI errors where sd is failing the IO back to dm.

If you not seeing any SCSI errors, I would assume that it is the patch
checker that is failing your paths. Can try using the tur checker to see
if this changes your error signature?

> Normally when I have seen Buffer I/O errors in the past, they have been  
> accompanied by SCSI errors as well, but in this case I am not getting  
> anything. One odd thing is that the filesystem reported extremely slow  
> writes before the buffer I/O errors occurred. I have checked and  
> double-checked the storage array and everything seems fine. I am using  
> directio as my path checker if that makes a difference.
>
> I have been able to reproduce the problem consistently by attempting an  
> fsck on dm-13. However, I am now doing an fsck on the underlying sd  
> device and it hasn't bombed out yet with the buffer I/O errors.
>
> Has anyone encountered these seemingly inexplicable buffer I/O errors  
> before? Any advice on how to debug it?

-andmike
--
Michael Anderson
andmike@linux.vnet.ibm.com

buffer i/o error

[Andrej Hocevar]

Hello, I suddenly realized that one of my external disks (a Western Digital
MyBook usb disk) wasn't accessible anymore. Then I found the following message.
There seems to have been no apparent damage or loss, but I'm reporting it
anyway.

Regards,

Andrej Hocevar


------------[ cut here ]------------
WARNING: at fs/buffer.c:1186 mark_buffer_dirty+0x20/0x69()
Modules linked in: nls_cp437 isofs loop xt_state xt_tcpudp iptable_filter
ip_tables x_tables nf_nat_ftp nf_nat nf_conntrack_ipv4 nfDec  3 16:24:58 stoa
kernel: Pid: 12202, comm: umount Not tainted 2.6.27.7home #1
 [<c0118eae>] warn_on_slowpath+0x40/0x63
 [<c0151c22>] check_object+0x10d/0x197
 [<c0152e76>] __slab_alloc+0x366/0x3de
 [<c011394b>] place_entity+0x9c/0xde
 [<c011388f>] update_curr+0x3d/0x5d
 [<c01149a0>] __dequeue_entity+0x1f/0x71
 [<c01158f0>] pick_next_task_fair+0xdf/0xe9
 [<c016fe20>] mark_buffer_dirty+0x20/0x69
 [<c0198b88>] journal_update_superblock+0x4a/0x7c
 [<c0198e00>] journal_destroy+0x101/0x146
 [<c0126742>] autoremove_wake_function+0x0/0x2d
 [<c018fcfd>] ext3_put_super+0x1f/0x147
 [<c01578a6>] generic_shutdown_super+0x4a/0xad
 [<c0157915>] kill_block_super+0xc/0x1b
 [<c0157983>] deactivate_super+0x2c/0x3f
 [<c0167772>] sys_umount+0x253/0x279
 [<c01677a3>] sys_oldumount+0xb/0xe
 [<c0102b36>] syscall_call+0x7/0xb
 [<c0310000>] via_ircc_open+0xfa/0x59b
 =======================
---[ end trace af37a7a2247422ea ]---

 
Linux stoa 2.6.27.7home #1 Sun Nov 23 14:37:46 CET 2008 i686 GNU/Linux
 
Gnu C                  4.3.2
Gnu make               3.81
binutils               2.18.0.20080103
util-linux             2.13.1.1
mount                  2.13.1
module-init-tools      found
PPP                    2.4.4
Linux C Library        2.7
Dynamic linker (ldd)   2.7
Procps                 3.2.7
Console-tools          0.2.3
Sh-utils               6.10
udev                   125
Modules Loaded         nls_cp437 isofs loop xt_state xt_tcpudp iptable_filter
ip_tables x_tables nf_nat_ftp nf_nat nf_conntrack_ipv4 nf_conntrack_ftp
nf_conntrack snd_mixer_oss ppdev lp cpufreq_stats fuse sr_mod sbp2 snd_intel8x0
snd_ac97_codec ac97_bus 8250_pci 8250 snd_pcm snd_timer rng_core serial_core
snd_page_alloc video output parport_pc parport sd_mod evdev ide_cd_mod cdrom
ide_disk usb_storage usbhid scsi_mod piix ohci1394 ide_core ieee1394 ehci_hcd
uhci_hcd usbcore unix

--
www.ljudmila.org/ludliteratura

Re: buffer i/o error

[Stefan Richter]

Andrej Hocevar wrote:
> Hello, I suddenly realized that one of my external disks (a Western Digital
> MyBook usb disk) wasn't accessible anymore. Then I found the following message.
> There seems to have been no apparent damage or loss, but I'm reporting it
> anyway.
> 
> Regards,
> 
> Andrej Hocevar
> 
> 
> ------------[ cut here ]------------
> WARNING: at fs/buffer.c:1186 mark_buffer_dirty+0x20/0x69()
[...]
>  [<c016fe20>] mark_buffer_dirty+0x20/0x69
>  [<c0198b88>] journal_update_superblock+0x4a/0x7c
>  [<c0198e00>] journal_destroy+0x101/0x146
>  [<c0126742>] autoremove_wake_function+0x0/0x2d
>  [<c018fcfd>] ext3_put_super+0x1f/0x147
>  [<c01578a6>] generic_shutdown_super+0x4a/0xad
>  [<c0157915>] kill_block_super+0xc/0x1b
[...]

This warning typically happens when the connection to a disk was lost.
Alas it doesn't tell /why/ the connection was lost.

http://kerneloops.org/guilty.php?guilty=journal_update_superblock has
more of this sort.
-- 
Stefan Richter
-=====-==--- ==-- ---==
http://arcgraph.de/sr/