Re: kernel BUG at fs/btrfs/extent_io.c:1989

[David Sterba <dsterba@suse.cz>]

On Tue, Sep 19, 2017 at 11:32:46AM +0000, Paul Jones wrote:
> > This 'mirror 0' looks fishy, (as mirror comes from btrfs_io_bio->mirror_num,
> > which should be at least 1 if raid1 setup is in use.)
> > 
> > Not sure if 4.13.2-gentoo made any changes on btrfs, but can you please
> > verify with the upstream kernel, say, v4.13?
> 
> It's basically a vanilla kernel with a handful of unrelated patches.
> The filesystem fell apart overnight, there were a few thousand
> checksum errors and eventually it went read-only. I tried to remount
> it, but got open_ctree failed. Btrfs check segfaulted, lowmem mode
> completed with so many errors I gave up and will restore from the
> backup.
> 
> I think I know the problem now - the lvm cache was in writeback mode
> (by accident) so during a defrag there would be gigabytes of unwritten
> data in memory only, which was all lost when the system crashed
> (motherboard failure). No wonder the filesystem didn't quite survive.

Yeah, the caching layer was my first suspicion, and lack of propagating
of the barriers. Good that you were able to confirm that as the root cause.

> I must say though, I'm seriously impressed at the data integrity of
> BTRFS - there were near 10,000 checksum errors, 4 which were
> uncorrectable, and from what I could tell nearly all of the data was
> still intact according to rsync checksums.

Yay!