Re: Troubles removing missing device from RAID 6

[Anand Jain <anand.jain@oracle.com>]

As you have an additional slot for the new disk, the proper procedure 
would have been

btrfs replace start -r <faulty-dev> <new-dev> /mnt

  -r shall avoid reading from the faulty dev.

(In some cases there might not be any spare slots, I am looking into 
fixing replace command for those cases.)

Thanks, Anand

On 19/7/20 10:13 pm, Edmund Urbani wrote:
> Hello everyone,
> 
> after having RMA'd a faulty HDD from my RAID6 and having received the 
> replacement, I added the new disk to the filesystem. At that point the 
> missing device was still listed and I went ahead to remove it like so:
> 
> btrfs device delete missing /mnt/shared/
> 
> After a few hours that command aborted with an I/O error and the logs 
> revealed this problem:
> 
> [284564.279190] BTRFS info (device sda1): relocating block group 
> 51490279391232 flags data|raid6
> [284572.319649] btrfs_print_data_csum_error: 75 callbacks suppressed
> [284572.319656] BTRFS warning (device sda1): csum failed root -9 ino 433 
> off 386727936 csum 0x791e44cc expected csum 0xbd1725d0 mirror 2
> [284572.320165] BTRFS warning (device sda1): csum failed root -9 ino 433 
> off 386732032 csum 0xec5f6097 expected csum 0x9114b5fa mirror 2
> [284572.320211] BTRFS warning (device sda1): csum failed root -9 ino 433 
> off 386736128 csum 0x4d2fa4b9 expected csum 0xf8a923f9 mirror 2
> [284572.320225] BTRFS warning (device sda1): csum failed root -9 ino 433 
> off 386740224 csum 0xcad08362 expected csum 0xa9361ed3 mirror 2
> [284572.320266] BTRFS warning (device sda1): csum failed root -9 ino 433 
> off 386744320 csum 0x469ac192 expected csum 0xb1e94692 mirror 2
> [284572.320279] BTRFS warning (device sda1): csum failed root -9 ino 433 
> off 386748416 csum 0x69759c1f expected csum 0xb3b9aa86 mirror 2
> [284572.320290] BTRFS warning (device sda1): csum failed root -9 ino 433 
> off 386752512 csum 0xd3a7c5d5 expected csum 0xd351862f mirror 2
> [284572.320465] BTRFS warning (device sda1): csum failed root -9 ino 433 
> off 386756608 csum 0x1264af83 expected csum 0x3a2c0ed5 mirror 2
> [284572.320480] BTRFS warning (device sda1): csum failed root -9 ino 433 
> off 386760704 csum 0x260a13ef expected csum 0xb3b4aec0 mirror 2
> [284572.320492] BTRFS warning (device sda1): csum failed root -9 ino 433 
> off 386764800 csum 0x6b615cd9 expected csum 0x99eaf560 mirror 2
> 
> I ran a long SMART self-test on the drives in the array which found no 
> problem. Currently I am running scrub to attempt and fix the block group.
> 
> scrub status:
> 
> UUID:             9c3c3f8d-a601-4bd3-8871-d068dd500a15
> 
> Scrub started:    Fri Jul 17 07:52:06 2020
> Status:           running
> Duration:         14:47:07
> Time left:        202:05:46
> ETA:              Tue Jul 28 00:07:36 2020
> Total to scrub:   16.80TiB
> Bytes scrubbed:   1.14TiB
> Rate:             22.56MiB/s
> Error summary:    read=295132162
>    Corrected:      0
>    Uncorrectable:  295132162
>    Unverified:     0
> 
> device stats:
> 
> Label: none  uuid: 9c3c3f8d-a601-4bd3-8871-d068dd500a15
>          Total devices 5 FS bytes used 16.80TiB
>          devid    3 size 9.09TiB used 8.76TiB path /dev/sda1
>          devid    4 size 9.09TiB used 8.76TiB path /dev/sdb1
>          devid    5 size 9.09TiB used 8.74TiB path /dev/sdd1
>          devid    6 size 9.09TiB used 498.53GiB path /dev/sdc1
>          *** Some devices missing
> 
> Is there anything else I can do to try and specifically fix that one 
> block group rather than scrubbing the entire filesytem? Also, is it 
> "normal" that scrub stats would show a huge number of "uncorrectable" 
> errors when a device is missing or should I be worried about that?
> 
> Kind regards,
>   Edmund
> 
>