Re: Bad fs performance, IO freezes

[Liu Bo <bo.li.liu@oracle.com>]

On 10/26/2015 08:16 PM, cheater00 . wrote:
> Hi guys,
> I am running into really bad performance. Here's my setup:
>
> WD Red 6 TB connected over USB2 to my core i7 laptop, running Ubuntu
> 32-bit with kernel 4.0.4-040004-generic #201505171336.
>
> Single btrfs partition covering whole disk.
>
> Autodefrag is on.
>
> fstab line:
> UUID=... /media/X btrfs rw,nosuid,nodev,autodefrag 0 0
>
> Sometimes when files are being modified or removed, I see
> btrfs-transacti eat 100% cpu; during this time no io operations
> succeed, that is, they're all stalled. You can't even ls on that fs.
> This happens for several minutes then normal operation resumes. There
> doesn't seem to be a rule to what will trigger this, other than
> opening a single file and reading usually works quite well. (say,
> watching a movie while all other programs are closed). But even moving
> files off the disks triggers some sort of bug. Just now I am moving a
> few files (just 30gb worth) onto another disk, and the bug triggers.
> So btrfs-transacti was eating my cpu for over 5 minutes and according
> to mv's output after this was done and cpu usage went back to normal
> what I was waiting for was for a tiny png file to be removed. This is
> pretty bad.
>
> I have tried defragmenting directories where files are being accessed
> and moved. This hasn't helped.
>
> This happens whether the FS is near full or not. It currently is near
> full but it wasn't before and it still did that. It still has about ~
> 100GB free space now.
>
> The more things are happening the more often this bug gets triggered.
> So if I have utorrent running and its temporary downloads directory is
> there, its download speed graph will be a few spikes of running at
> several MB/sec separated by durations of 0 download speed.
>
> Nothing seems to show up in dmesg or syslog.
>
> I have asked in #btrfs but the suggestions ended up not fixing the
> issue (autodefrag, defrag dirs).
>
> Please advise what I should do with this issue.

It might be related to delayed ref rework, the last time I saw this kind 
of hanging problem about btrfs-transaction eating cpu is that because 
btrfs doesn't merge delayed refs, it'd be best to try the lastest kernel 
and if the issue is not resolved, then we can work out a reproducer and 
provide debugging.

Thanks,

Liubo