From: Yauhen Kharuzhy <yauhen.kharuzhy@zavadatar.com>
To: Duncan <1i5t5.duncan@cox.net>
Cc: linux-btrfs@vger.kernel.org
Subject: Re: RAID6, errors at missing device replacement
Date: Mon, 2 May 2016 21:43:05 +0300 [thread overview]
Message-ID: <20160502184305.GL21960@jeknote.loshitsa1.net> (raw)
In-Reply-To: <pan$8aea5$883c2a2a$18248a42$c1560bd9@cox.net>
[-- Attachment #1: Type: text/plain, Size: 11211 bytes --]
On Sat, Apr 16, 2016 at 07:37:48AM +0000, Duncan wrote:
> Yauhen Kharuzhy posted on Fri, 15 Apr 2016 12:49:36 -0700 as excerpted:
>
> > I have discovered case when replacement of missing devices causes
> > metadata corruption. Does anybody know anything about this?
> >
> > I use 4.4.5 kernel with latest global spare patches.
> >
> > If we have RAID6 (may be reproducible on RAID5 too) and try to replace
> > one missing drive by other and after this try to remove another drive
> > and replace it, plenty of errors are shown in the log:
I have reproduced this with vanilla 4.6-rc4 kernel and RAID5.
Script used to reproduce is attached, run as "./test-replace.sh <mount point> <disk1 disk2...>"
Kernel log:
[ 402.878389] BTRFS: device fsid eabede3e-1e50-46cd-92ec-f9476b321f63 devid 1 transid 3 /dev/sdc
[ 402.911820] BTRFS: device fsid eabede3e-1e50-46cd-92ec-f9476b321f63 devid 2 transid 3 /dev/sdd
[ 402.972031] BTRFS: device fsid eabede3e-1e50-46cd-92ec-f9476b321f63 devid 3 transid 3 /dev/sde
[ 403.020067] BTRFS: device fsid eabede3e-1e50-46cd-92ec-f9476b321f63 devid 4 transid 3 /dev/sdf
[ 404.042312] BTRFS info (device sdf): disk space caching is enabled
[ 404.051338] BTRFS: has skinny extents
[ 404.056805] BTRFS: flagging fs with big metadata feature
[ 404.149815] BTRFS: creating UUID tree
[ 407.321146] sd 5:0:0:0: [sdf] Synchronizing SCSI cache
[ 407.349530] sd 5:0:0:0: [sdf] Stopping disk
[ 407.376682] ata6.00: disabled
[ 407.695945] BTRFS error (device sdf): bdev /dev/sdf errs: wr 0, rd 0, flush 1, corrupt 0, gen 0
[ 407.703760] BTRFS warning (device sdf): lost page write due to IO error on /dev/sdf
[ 407.726179] BTRFS error (device sdf): bdev /dev/sdf errs: wr 1, rd 0, flush 1, corrupt 0, gen 0
[ 407.733718] BTRFS warning (device sdf): lost page write due to IO error on /dev/sdf
[ 407.739873] BTRFS error (device sdf): bdev /dev/sdf errs: wr 2, rd 0, flush 1, corrupt 0, gen 0
[ 410.631220] ata6: hard resetting link
[ 411.041672] ata6: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[ 411.090105] ata6.00: ATA-6: VBOX HARDDISK, 1.0, max UDMA/133
[ 411.153739] ata6.00: 16777216 sectors, multi 128: LBA48 NCQ (depth 31/32)
[ 411.189534] ata6.00: configured for UDMA/133
[ 411.225526] ata6: EH complete
[ 411.229002] scsi 5:0:0:0: Direct-Access ATA VBOX HARDDISK 1.0 PQ: 0 ANSI: 5
[ 411.278584] sd 5:0:0:0: [sdg] 16777216 512-byte logical blocks: (8.59 GB/8.00 GiB)
[ 411.297341] sd 5:0:0:0: [sdg] Write Protect is off
[ 411.300054] sd 5:0:0:0: Attached scsi generic sg5 type 0
[ 411.350875] sd 5:0:0:0: [sdg] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 411.371402] sd 5:0:0:0: [sdg] Attached SCSI disk
[ 413.663624] BTRFS error (device sdf): bdev /dev/sdf errs: wr 2, rd 0, flush 2, corrupt 0, gen 0
[ 413.714417] BTRFS warning (device sdf): lost page write due to IO error on /dev/sdf
[ 413.719450] BTRFS error (device sdf): bdev /dev/sdf errs: wr 3, rd 0, flush 2, corrupt 0, gen 0
[ 413.728705] BTRFS warning (device sdf): lost page write due to IO error on /dev/sdf
[ 413.734030] BTRFS error (device sdf): bdev /dev/sdf errs: wr 4, rd 0, flush 2, corrupt 0, gen 0
[ 413.841946] BTRFS info (device sde): allowing degraded mounts
[ 413.848622] BTRFS info (device sde): disk space caching is enabled
[ 413.877470] BTRFS: has skinny extents
[ 413.942027] BTRFS info (device sde): bdev /dev/sdf errs: wr 2, rd 0, flush 1, corrupt 0, gen 0
[ 414.076571] BTRFS info (device sde): dev_replace from <missing disk> (devid 4) to /dev/sdg started
[ 420.402126] BTRFS info (device sde): dev_replace from <missing disk> (devid 4) to /dev/sdg finished
[ 420.646768] sd 4:0:0:0: [sde] Synchronizing SCSI cache
[ 420.653786] sd 4:0:0:0: [sde] Stopping disk
[ 420.707224] ata5.00: disabled
[ 420.991219] BTRFS error (device sde): bdev /dev/sde errs: wr 0, rd 0, flush 1, corrupt 0, gen 0
[ 421.006803] BTRFS warning (device sde): lost page write due to IO error on /dev/sde
[ 421.013813] BTRFS error (device sde): bdev /dev/sde errs: wr 1, rd 0, flush 1, corrupt 0, gen 0
[ 421.022001] BTRFS warning (device sde): lost page write due to IO error on /dev/sde
[ 421.032855] BTRFS error (device sde): bdev /dev/sde errs: wr 2, rd 0, flush 1, corrupt 0, gen 0
[ 423.943549] ata5: hard resetting link
[ 424.264086] ata5: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[ 424.270354] ata5.00: ATA-6: VBOX HARDDISK, 1.0, max UDMA/133
[ 424.303915] ata5.00: 41943040 sectors, multi 128: LBA48 NCQ (depth 31/32)
[ 424.312418] ata5.00: configured for UDMA/133
[ 424.317876] ata5: EH complete
[ 424.346139] scsi 4:0:0:0: Direct-Access ATA VBOX HARDDISK 1.0 PQ: 0 ANSI: 5
[ 424.389067] sd 4:0:0:0: [sdf] 41943040 512-byte logical blocks: (21.5 GB/20.0 GiB)
[ 424.389110] sd 4:0:0:0: Attached scsi generic sg4 type 0
[ 424.453500] sd 4:0:0:0: [sdf] Write Protect is off
[ 424.460923] sd 4:0:0:0: [sdf] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 424.526381] sd 4:0:0:0: [sdf] Attached SCSI disk
[ 426.636182] BTRFS error (device sde): bdev /dev/sde errs: wr 2, rd 0, flush 2, corrupt 0, gen 0
[ 426.641741] BTRFS warning (device sde): lost page write due to IO error on /dev/sde
[ 426.691659] BTRFS error (device sde): bdev /dev/sde errs: wr 3, rd 0, flush 2, corrupt 0, gen 0
[ 426.698723] BTRFS warning (device sde): lost page write due to IO error on /dev/sde
[ 426.710799] BTRFS error (device sde): bdev /dev/sde errs: wr 4, rd 0, flush 2, corrupt 0, gen 0
[ 426.834307] BTRFS info (device sdg): allowing degraded mounts
[ 426.842495] BTRFS info (device sdg): disk space caching is enabled
[ 426.860045] BTRFS: has skinny extents
[ 426.875105] BTRFS info (device sdg): bdev /dev/sdg errs: wr 2, rd 0, flush 1, corrupt 0, gen 0
[ 426.886143] BTRFS info (device sdg): bdev /dev/sde errs: wr 2, rd 0, flush 1, corrupt 0, gen 0
[ 427.146338] BTRFS info (device sdg): dev_replace from <missing disk> (devid 3) to /dev/sdf started
[ 427.936021] BTRFS error (device sdg): failed to rebuild valid logical 3279355904 for dev /dev/sde
[ 428.076806] BTRFS error (device sdg): failed to rebuild valid logical 3267567616 for dev /dev/sde
[ 428.189681] BTRFS error (device sdg): failed to rebuild valid logical 3277004800 for dev /dev/sde
[ 428.768747] BTRFS error (device sdg): failed to rebuild valid logical 3279372288 for dev /dev/sde
[ 429.411867] BTRFS error (device sdg): failed to rebuild valid logical 3269947392 for dev /dev/sde
[ 429.438711] BTRFS error (device sdg): failed to rebuild valid logical 3271520256 for dev /dev/sde
[ 429.499210] BTRFS error (device sdg): failed to rebuild valid logical 3268378624 for dev /dev/sde
[ 429.870200] BTRFS error (device sdg): failed to rebuild valid logical 3276255232 for dev /dev/sde
[ 429.967750] BTRFS error (device sdg): failed to rebuild valid logical 3266834432 for dev /dev/sde
[ 430.028623] BTRFS error (device sdg): failed to rebuild valid logical 3274698752 for dev /dev/sde
[ 430.488825] BTRFS info (device sdg): dev_replace from <missing disk> (devid 3) to /dev/sdf finished
[ 430.620438] sd 3:0:0:0: [sdd] Synchronizing SCSI cache
[ 430.692664] sd 3:0:0:0: [sdd] Stopping disk
[ 430.760882] ata4.00: disabled
[ 430.958960] BTRFS error (device sdg): bdev /dev/sdd errs: wr 0, rd 0, flush 1, corrupt 0, gen 0
[ 430.982233] BTRFS warning (device sdg): lost page write due to IO error on /dev/sdd
[ 430.999441] BTRFS error (device sdg): bdev /dev/sdd errs: wr 1, rd 0, flush 1, corrupt 0, gen 0
[ 431.036540] BTRFS warning (device sdg): lost page write due to IO error on /dev/sdd
[ 431.074314] BTRFS error (device sdg): bdev /dev/sdd errs: wr 2, rd 0, flush 1, corrupt 0, gen 0
[ 433.961963] ata4: hard resetting link
[ 434.287424] ata4: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[ 434.292584] ata4.00: ATA-6: VBOX HARDDISK, 1.0, max UDMA/133
[ 434.302767] ata4.00: 41943040 sectors, multi 128: LBA48 NCQ (depth 31/32)
[ 434.342383] ata4.00: configured for UDMA/133
[ 434.354685] ata4: EH complete
[ 434.364789] scsi 3:0:0:0: Direct-Access ATA VBOX HARDDISK 1.0 PQ: 0 ANSI: 5
[ 434.440122] sd 3:0:0:0: Attached scsi generic sg3 type 0
[ 434.448358] sd 3:0:0:0: [sde] 41943040 512-byte logical blocks: (21.5 GB/20.0 GiB)
[ 434.448481] sd 3:0:0:0: [sde] Write Protect is off
[ 434.448517] sd 3:0:0:0: [sde] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 434.589187] sd 3:0:0:0: [sde] Attached SCSI disk
[ 436.639464] BTRFS error (device sdg): bdev /dev/sdd errs: wr 2, rd 0, flush 2, corrupt 0, gen 0
[ 436.701947] BTRFS warning (device sdg): lost page write due to IO error on /dev/sdd
[ 436.713283] BTRFS error (device sdg): bdev /dev/sdd errs: wr 3, rd 0, flush 2, corrupt 0, gen 0
[ 436.723682] BTRFS warning (device sdg): lost page write due to IO error on /dev/sdd
[ 436.731662] BTRFS error (device sdg): bdev /dev/sdd errs: wr 4, rd 0, flush 2, corrupt 0, gen 0
[ 436.761114] BTRFS error (device sdg): bdev /dev/sdd errs: wr 4, rd 0, flush 3, corrupt 0, gen 0
[ 436.783619] BTRFS warning (device sdg): lost page write due to IO error on /dev/sdd
[ 436.790353] BTRFS error (device sdg): bdev /dev/sdd errs: wr 5, rd 0, flush 3, corrupt 0, gen 0
[ 436.828784] BTRFS warning (device sdg): lost page write due to IO error on /dev/sdd
[ 436.840279] BTRFS error (device sdg): bdev /dev/sdd errs: wr 6, rd 0, flush 3, corrupt 0, gen 0
[ 436.963086] BTRFS info (device sdf): allowing degraded mounts
[ 436.977520] BTRFS info (device sdf): disk space caching is enabled
[ 436.982720] BTRFS: has skinny extents
[ 436.998246] BTRFS info (device sdf): bdev /dev/sdf errs: wr 2, rd 0, flush 1, corrupt 0, gen 0
[ 437.023059] BTRFS info (device sdf): bdev /dev/sdg errs: wr 2, rd 0, flush 1, corrupt 0, gen 0
[ 437.040400] BTRFS info (device sdf): bdev /dev/sdd errs: wr 4, rd 0, flush 2, corrupt 0, gen 0
[ 437.241595] BTRFS info (device sdf): dev_replace from <missing disk> (devid 2) to /dev/sde started
[ 438.185590] scrub_missing_raid56_worker: 2 callbacks suppressed
[ 438.188229] BTRFS error (device sdf): failed to rebuild valid logical 3279421440 for dev /dev/sdd
[ 438.300493] BTRFS error (device sdf): failed to rebuild valid logical 3267633152 for dev /dev/sdd
[ 438.703672] BTRFS error (device sdf): failed to rebuild valid logical 3277070336 for dev /dev/sdd
[ 439.157045] BTRFS error (device sdf): failed to rebuild valid logical 3279437824 for dev /dev/sdd
[ 439.373168] BTRFS error (device sdf): failed to rebuild valid logical 3270012928 for dev /dev/sdd
[ 439.423270] BTRFS error (device sdf): failed to rebuild valid logical 3271585792 for dev /dev/sdd
[ 439.601332] BTRFS error (device sdf): failed to rebuild valid logical 3268444160 for dev /dev/sdd
[ 440.043626] BTRFS error (device sdf): failed to rebuild valid logical 3276320768 for dev /dev/sdd
[ 440.205525] BTRFS error (device sdf): failed to rebuild valid logical 3266899968 for dev /dev/sdd
[ 440.249055] BTRFS error (device sdf): failed to rebuild valid logical 3274764288 for dev /dev/sdd
[ 440.351454] BTRFS info (device sdf): dev_replace from <missing disk> (devid 2) to /dev/sde finished
--
Yauhen Kharuzhy
[-- Attachment #2: test-replace.sh --]
[-- Type: application/x-sh, Size: 3101 bytes --]
next prev parent reply other threads:[~2016-05-02 18:43 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-04-15 19:49 RAID6, errors at missing device replacement Yauhen Kharuzhy
2016-04-15 23:00 ` Henk Slager
2016-04-16 7:37 ` Duncan
2016-05-02 18:43 ` Yauhen Kharuzhy [this message]
2016-05-02 19:04 ` Chris Murphy
2016-05-02 19:19 ` Yauhen Kharuzhy
2016-05-02 19:33 ` Chris Murphy
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20160502184305.GL21960@jeknote.loshitsa1.net \
--to=yauhen.kharuzhy@zavadatar.com \
--cc=1i5t5.duncan@cox.net \
--cc=linux-btrfs@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).