Re: BTRFS list of grievances

[Remi Gauvin <remi@georgianit.com>]

On 2024-09-27 12:27 p.m., Roman Mamedov wrote:
>
> Or if not, then how do you get from there to a consistent state? Run a scrub,
> make the system reread the entire 40 TB of data, correcting errors and lack of
> duplication where necessary.
>

The BTRFS handling of this situation is actually worse.

The often given, (and entirely too simple) answer is to scrub.  But this
has several caveats.


1. The system will not detect the error state automatically.  So fixing
this requires the admin to be actively monitoring for errors to detect
the missed writes.  (regular monitoring of btrfs dev stats and allert on
errors is required.)

2.  Any files that are stored on on the device with CoW Disabled will
not be fixed, and the two copies will be different, with no real way to
detect or fix.  There are packages that disable CoW on files by
default.  (systemd log files, but probably more concerning, and virtual
disk created by libvirt, for example. (Some amount of divergence can
happen at any unclean shutdown in this scenario)

3. I don't have exact math at my fingertips, but with enough failed
writes, the chances of a CRC32 collision of the stale data leaving
unfixed/corrupted data behind gets fairly high.

For reasons of 2 and 3, the only way to fix this without increasing
chance of data corruption is to replace the previously disconnected
drive to a hot spare. (with the -r option to btrfs replace.)