Re: fatal database corruption with btrfs "out of space" with ~50 GB left

[Tomasz Chmielewski <tch@virtall.com>]

On 2018-02-19 13:29, Anand Jain wrote:
> On 02/14/2018 10:19 PM, Tomasz Chmielewski wrote:
>> Just FYI, how dangerous running btrfs can be - we had a fatal, 
>> unrecoverable MySQL corruption when btrfs decided to do one of these 
>> "I have ~50 GB left, so let's do out of space (and corrupt some files 
>> at the same time, ha ha!)".
> 
>  Thanks for reporting.
> 
>> Running btrfs RAID-1 with kernel 4.14.
> 
>  Can you pls let us know..
>  1. What tool cli/reported/identified that data is corrupted?

mysqld log - mysqld would refuse to start because of database 
corruption.

And, the database wouldn't start even when "innodb_force_recovery = " 
was set to a high/max value.


In the past, with lower kernel versions, we had a similar issue with 
mongod - it wouldn't start anymore due to some corruption which happened 
when we hit "out of space" (again, with dozens of GBs free space).


>  2. Disk error stat using.. btrfs dev stat <mnt>
>     (dev stat is stored on disk)

# btrfs dev stat /var/lib/lxd
[/dev/sda3].write_io_errs    0
[/dev/sda3].read_io_errs     0
[/dev/sda3].flush_io_errs    0
[/dev/sda3].corruption_errs  0
[/dev/sda3].generation_errs  0
[/dev/sdb3].write_io_errs    0
[/dev/sdb3].read_io_errs     0
[/dev/sdb3].flush_io_errs    0
[/dev/sdb3].corruption_errs  0
[/dev/sdb3].generation_errs  0


>  3. Wheather the disk was mounted as degraded any time before?

No. Everything healthy with the disks.


Tomasz Chmielewski
https://lxadm.com