Re: btrfs freezing on writes

[kjansen387 <kjansen387@gmail.com>]

Ah, true, sync -f does a syncfs.

sync without -f on an existing file is always fast. But when I strace 
vim, fsync is slow (sometimes >10 seconds) because it needs to write 
something.

Hopefully a developer can reply - thanks so far!



On 08-Apr-20 00:11, Chris Murphy wrote:
> On Tue, Apr 7, 2020 at 2:39 PM kjansen387 <kjansen387@gmail.com> wrote:
> 
> 
> 
>> $ grep /export /etc/fstab
>> UUID=8ce9e167-57ea-4cf8-8678-3049ba028c12 /export       btrfs
>> device=/dev/sde,device=/dev/sdf    0 2
> 
> I'd use only noatime for options. There's no advantage of specifying
> devices. And also fs_passno can be 0. fsck.btrfs is a no op anyway, so
> it doesn't hurt anything to leave it at 2.
> 
> 
> 
>> I've attached sysstat info of my disks. What's obvious is that 2 disks
>> have the load (one is written to, the other one is the mirror), and 3
>> are pretty idle. But, it's 2.4MB per second - that's not much!
> 
> Lots of small file writes maybe? What's iostat show for utilization?
> Or vmstat for io? Hard drives of course have limited IO per second.
> 
>> I've just changed the space_cache to v2, but it doesn't seem to help
>> much. 'sync -f /export/tmp' still takes very long at times (just took 22
>> seconds!)
> 
> I just did a strace on this command and it uses syncfs, not fsync. I'm
> pretty sure on Btrfs this is a full filesystem sync, which is
> expensive, all data and metadata. So if it's very dirty, yeah it could
> take some time to flush everything to disk.
> 
> Try it without -f and you'll get fsync.
> 
> 
>> Any way we can find the cause, before I move everything into subvolumes
>> ? I'd like to avoid that if possible. Sounds a bit overkill for 2.4MB/s
>> writes, and I think most of it is going to one influxdb database anyway.
> 
>  From the sysrq...
> [937013.794093] mysqld          D    0 10400      1 0x00004000
> [937013.794240]  do_fsync+0x38/0x70
> 
> [937013.794253] mysqld          D    0 10412      1 0x00004000
> [937013.794297]  do_fsync+0x38/0x70
> 
> [937013.794306] mysqld          D    0 10421      1 0x00004000
> [937013.794353]  do_fsync+0x38/0x70
> 
> [937013.794788] WTJourn.Flusher D    0 1186978 1186954 0x00004320
> [937013.794894]  do_fsync+0x38/0x70
> 
> [937013.794903] ftdc            D    0 1186980 1186954 0x00000320
> [937013.794951]  ? btrfs_create+0x200/0x200 [btrfs]
> 
> [937013.795022] influxd         D    0 2082782      1 0x00004000
> [937013.795086]  do_fsync+0x38/0x70
> 
> [937013.795098] influxd         D    0 2082804      1 0x00004000
> [937013.795143]  do_fsync+0x38/0x70
> 
> [937013.795191] vim             D    0 2228648 2223845 0x00004000
> [937013.795286]  do_fsync+0x38/0x70
> 
> Quite a lot of fsync all at once.
> 
> But I'm not very good at parsing kernel output. Maybe a developer will
> have input.
> 
> 
>