Re: Filesystem mounts fine but hangs on access

[Sebastian Ochmann <ochmann@cs.uni-bonn.de>]

On 04.11.18 19:31, Duncan wrote:
> [This mail was also posted to gmane.comp.file-systems.btrfs.]
> 
> Sebastian Ochmann posted on Sun, 04 Nov 2018 14:15:55 +0100 as
> excerpted:
> 
>> Hello,
>>
>> I have a btrfs filesystem on a single encrypted (LUKS) 10 TB drive
>> which stopped working correctly.
> 
>> Kernel 4.18.16 (Arch Linux)
> 
> I see upgrading to 4.19 seems to have solved your problem, but this is
> more about something I saw in the trace that has me wondering...
> 
>> [  368.267315]  touch_atime+0xc0/0xe0
> 
> Do you have any atime-related mount options set?

That's an interesting point. On some machines, I have explicitly set 
"noatime", but on that particular system, I did not, thus using the 
"relatime" option as per default. Since I'm not using mutt or anything 
else (that I'm aware of) that exploits this feature, I will set noatime 
there as well.

> FWIW, noatime is strongly recommended on btrfs.
> 
> Now I'm not a dev, just a btrfs user and list regular, and I don't know
> if that function is called and just does nothing when noatime is set,
> so you may well already have it set and this is "much ado about
> nothing", but the chance that it's relevant, if not for you, perhaps
> for others that may read it, begs for this post...
> 
> The problem with atime, access time, is that it turns most otherwise
> read- only operations into read-and-write operations in ordered to
> update the access time.  And on copy-on-write (COW) based filesystems
> such as btrfs, that can be a big problem, because updating that tiny
> bit of metadata will trigger a rewrite of the entire metadata block
> containing it, which will trigger an update of the metadata for /that/
> block in the parent metadata tier... all the way up the metadata tree,
> ultimately to its root, the filesystem root and the superblocks, at the
> next commit (normally every 30 seconds or less).
> 
> Not only is that a bunch of otherwise unnecessary work for a bit of
> metadata barely anything actually uses, but forcing most read
> operations to read-write obviously compounds the risk for all of those
> would-be read- only operations when a filesystem already has problems.
> 
> Additionally, if your use-case includes regular snapshotting, with
> atime on, on mostly read workloads with few writes (other than atime
> updates), it may actually be the case that most of the changes in a
> snapshot are actually atime updates, making reoccurring snapshot
> updates far larger than they'd be otherwise.
> 
> Now a few years ago the kernel did change the default to relatime,
> basically updating the atime for any particular file only once a day,
> which does help quite a bit, and on traditional filesystems it's
> arguably a reasonably sane default, but COW makes atime tracking enough
> more expensive that setting noatime is still strongly recommended on
> btrfs, particularly if you're doing regular snapshotting.
> 
> So do consider adding noatime to your mount options if you haven't done
> so already.  AFAIK, the only /semi-common/ app that actually uses
> atimes these days is mutt (for read-message tracking), and then not for
> mbox, so you should be safe to at least test turning it off.
> 
> And YMMV, but if you do use mutt or something else that uses atimes,
> I'd go so far as to recommend finding an alternative, replacing either
> btrfs (because as I said, relatime is arguably enough on a traditional
> non-COW filesystem) or whatever it is that uses atimes, your call,
> because IMO it really is that big a deal.
> 
> Meanwhile, particularly after seeing that in the trace, if the 4.19
> update hadn't already fixed it, I'd have suggested trying a read-only
> mount, both as a test, and assuming it worked, at least allowing you to
> access the data without the lockup, which would have then been related
> to the write due to the atime update, not the actual read.

It would be nice to have a 1:1 image of the filesystem (or rather the 
raw block device) for more testing, but unfortunately I don't have 
another 10 TB drive lying around. :) I didn't really expect the 4.19 
upgrade to (apparently) fix the problem right away, so I also couldn't 
test the mentioned patch, but yeah... If it happens again (which for 
some reason I don't hope), I'll try you suggestion.

> Actually, a read-only mount test is always a good troubleshooting step
> when the trouble is a filesystem that either won't mount normally, or
> will, but then locks up when you try to access something.  It's far
> lest risky than a normal writable mount, and at minimum it provides you
> the additional test data of whether it worked or not, plus if it does,
> a chance to access the data and make sure your backups are current,
> before actually trying to do any repairs.
>