Re: btrfs freezing on writes

[Chris Murphy <lists@colorremedies.com>]

On Tue, Apr 7, 2020 at 2:39 PM kjansen387 <kjansen387@gmail.com> wrote:



>$ grep /export /etc/fstab
>UUID=8ce9e167-57ea-4cf8-8678-3049ba028c12 /export       btrfs
>device=/dev/sde,device=/dev/sdf    0 2

I'd use only noatime for options. There's no advantage of specifying
devices. And also fs_passno can be 0. fsck.btrfs is a no op anyway, so
it doesn't hurt anything to leave it at 2.



> I've attached sysstat info of my disks. What's obvious is that 2 disks
> have the load (one is written to, the other one is the mirror), and 3
> are pretty idle. But, it's 2.4MB per second - that's not much!

Lots of small file writes maybe? What's iostat show for utilization?
Or vmstat for io? Hard drives of course have limited IO per second.

> I've just changed the space_cache to v2, but it doesn't seem to help
> much. 'sync -f /export/tmp' still takes very long at times (just took 22
> seconds!)

I just did a strace on this command and it uses syncfs, not fsync. I'm
pretty sure on Btrfs this is a full filesystem sync, which is
expensive, all data and metadata. So if it's very dirty, yeah it could
take some time to flush everything to disk.

Try it without -f and you'll get fsync.


> Any way we can find the cause, before I move everything into subvolumes
> ? I'd like to avoid that if possible. Sounds a bit overkill for 2.4MB/s
> writes, and I think most of it is going to one influxdb database anyway.

From the sysrq...
[937013.794093] mysqld          D    0 10400      1 0x00004000
[937013.794240]  do_fsync+0x38/0x70

[937013.794253] mysqld          D    0 10412      1 0x00004000
[937013.794297]  do_fsync+0x38/0x70

[937013.794306] mysqld          D    0 10421      1 0x00004000
[937013.794353]  do_fsync+0x38/0x70

[937013.794788] WTJourn.Flusher D    0 1186978 1186954 0x00004320
[937013.794894]  do_fsync+0x38/0x70

[937013.794903] ftdc            D    0 1186980 1186954 0x00000320
[937013.794951]  ? btrfs_create+0x200/0x200 [btrfs]

[937013.795022] influxd         D    0 2082782      1 0x00004000
[937013.795086]  do_fsync+0x38/0x70

[937013.795098] influxd         D    0 2082804      1 0x00004000
[937013.795143]  do_fsync+0x38/0x70

[937013.795191] vim             D    0 2228648 2223845 0x00004000
[937013.795286]  do_fsync+0x38/0x70

Quite a lot of fsync all at once.

But I'm not very good at parsing kernel output. Maybe a developer will
have input.



-- 
Chris Murphy