Re: Troubleshooting "Buffer I/O error" on reading md device

[NeilBrown <neilb@suse.com>]

[-- Attachment #1: Type: text/plain, Size: 2209 bytes --]

On Thu, Jan 04 2018, RQM wrote:

> Hello,
>
>> I needed "mdadm -E" the components of the array, so the partitions
>> rather than the whole devices. e.g. /dev/sdb1, not /dev/sdb.
>
> Sorry, that should have occurred to me. Here's the output:
> https://paste.ubuntu.com/26319689/
>
> Indeed, bad blocks are present on two devices.
>
>> You can remove the bad block by over-writing it.
>> dd if=/dev/zero of=/dev/md0 bs=4K seek=1598030208 count=1
>> though that might corrupt some file containing the block.
>
> I have tried that just now, but before running mdadm -E above. dd appears to succeed when writing to the bad block, but after that, reading that block with dd fails again:
> "dd: error reading '/dev/md0': Input/output error"
>
> In dmesg, the following errors appear:
> [220444.068715] VFS: Dirty inode writeback failed for block device md0 (err=-5).
> [220445.850229] Buffer I/O error on dev md0, logical block 1598030208, async page read
>
> I have repeated the dd write-then-read experiment, with identical results.
>
> The filesystem is indeed ext4, but it's not of tremendous importance to me that all data is recovered, as the array contains backup data only. However, I would like to get the backup system back into operation, so I'd be very grateful for further hints how to get the array into a usable state.

The easiest approach is to remove the bad block log.
Stop array, and then assemble with --update=no-bbl.
e.g
  mdadm -S /dev/md0
  mdadm -A /dev/md0 --update=no-bbl /dev/sd[bcdef]3

Before you do that though, please take a dump of the metadata and send
it to me, in case I get motivated to figure out why writing didn't work.

 mkdir /tmp/dump
 mdadm --dump /tmp/dump /dev/sd[bcdef]3
 tar czSf /tmp/dump.tgz /tmp/dump

The files in /tmp are sparse images of the hard drives with only
the metadata present.  The 'S' flag to tar should cause it to notice
this and create a tiny tgz file.
Then send me /tmp/dump.tgz.

Thanks,
NeilBrown


>
> Thank you so much for your help so far!
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]