Re: strangely uncorrectable errors with RAID-5

[Russell Coker <russell@coker.com.au>]

On Monday, 21 October 2024 15:26:19 AEDT Qu Wenruo wrote:
> With the recent RAID56 improves, I'd say RAID5 data + RAID1 metadata is
> usable, but I'm not sure how it will survive in a production environment.
> 
> Considering we have a lot of other problems out of our control, like bad
> disk flush behavior, and even hardware memory bitflips, I won't
> recommend RAID5 data for now, but I believe RAID56 for data has improved
> a lot.
> 
> If you want to experiment RAID1 metadata with RAID5 data and report
> back, I would appreciate the effort a lot.
> And from my last work on RAID56 (for data), it should survive your
> random corruption script.

Just to see if anything had changed I ran the same tests with RAID-5 data and 
metadata again with the Debian kernel 6.12.17-amd64 and this time got properly 
uncorrectable errors in less than a minute.  I don't know if the error 
happened faster and worse than before because of some kernel difference, luck, 
or some timing difference when running on different hardware.

The system is a Dell PowerEdge T630 with advanced ECC enabled so I don't think 
that memory bitflips are an issue here.

[  398.486860] BTRFS warning (device vdc): checksum error at logical 
5073338368 on dev /dev/vdc, physical 1705771008: metadata leaf (level 0) in 
tree 5
[  398.487367] BTRFS error (device vdc): unable to fixup (regular) error at 
logical 5073338368 on dev /dev/vdc physical 1705771008
[  398.487826] BTRFS warning (device vdc): checksum error at logical 
5073338368 on dev /dev/vdc, physical 1705771008: metadata leaf (level 0) in 
tree 5
[  398.488333] BTRFS error (device vdc): unable to fixup (regular) error at 
logical 5073338368 on dev /dev/vdc physical 1705771008
[  398.488792] BTRFS warning (device vdc): checksum error at logical 
5073338368 on dev /dev/vdc, physical 1705771008: metadata leaf (level 0) in 
tree 5
[  398.489308] BTRFS error (device vdc): unable to fixup (regular) error at 
logical 5073338368 on dev /dev/vdc physical 1705771008
[  398.489772] BTRFS warning (device vdc): checksum error at logical 
5073338368 on dev /dev/vdc, physical 1705771008: metadata leaf (level 0) in 
tree 5
[  398.490459] BTRFS error (device vdc): unable to fixup (regular) error at 
logical 5073338368 on dev /dev/vdc physical 1705771008
[  398.490926] BTRFS warning (device vdc): checksum error at logical 
5073338368 on dev /dev/vdc, physical 1705771008: metadata leaf (level 0) in 
tree 5
[  398.491435] BTRFS error (device vdc): unable to fixup (regular) error at 
logical 5073338368 on dev /dev/vdc physical 1705771008
[  398.491898] BTRFS warning (device vdc): checksum error at logical 
5073338368 on dev /dev/vdc, physical 1705771008: metadata leaf (level 0) in 
tree 5
[  398.492406] BTRFS error (device vdc): unable to fixup (regular) error at 
logical 5073338368 on dev /dev/vdc physical 1705771008
[  398.492868] BTRFS warning (device vdc): checksum error at logical 
5073338368 on dev /dev/vdc, physical 1705771008: metadata leaf (level 0) in 
tree 5

-- 
My Main Blog         http://etbe.coker.com.au/
My Documents Blog    http://doc.coker.com.au/