Re: Still seeing high autodefrag IO with kernel 5.16.5

[Benjamin Xiao <ben.r.xiao@gmail.com>]

Okay, I just tested right now with my custom 5.16.5 kernel with your 3 
patches applied. Redownloading the same game, I noticed that there was 
significantly less IO load during the download, which is great. It 
looked kinda bursty instead, with periods of no load, and then periods 
of load ranging in the 150-200MB/s range.

After the download, I am no longer getting that constant 90-150MB/s 
disk write from btrfs-cleaner, but I am seeing periodic bursts of it 
every 10 seconds or so. These bursts last for around 3 seconds and load 
is anywhere from 50-300MB/s.

Is this normal? Does autodefrag only defrag newly written data, or does 
it sometimes go back and run defrag on data written previously? I am 
gonna let it run for a bit to see if it eventually subsides.

Ben

On Fri, Feb 4 2022 at 02:20:44 PM +0800, Qu Wenruo 
<quwenruo.btrfs@gmx.com> wrote:
> 
> 
> On 2022/2/4 12:32, Benjamin Xiao wrote:
>> Okay, I just compiled a custom Arch kernel with your patches applied.
>> Will test soon. Besides enabling autodefrag and redownloading a game
>> from Steam, what other sorts of tests should I do?
> 
> As long as your workload can trigger the problem reliably, nothing 
> else.
> 
> Thanks,
> Qu
> 
>> 
>> Ben
>> 
>> On Fri, Feb 4 2022 at 09:54:19 AM +0800, Qu Wenruo
>> <quwenruo.btrfs@gmx.com> wrote:
>>> 
>>> 
>>> On 2022/2/4 09:17, Qu Wenruo wrote:
>>>> 
>>>> 
>>>> On 2022/2/4 04:05, Benjamin Xiao wrote:
>>>>> Hello all,
>>>>> 
>>>>> Even after the defrag patches that landed in 5.16.5, I am still 
>>>>> seeing
>>>>> lots of cpu usage and disk writes to my SSD when autodefrag is 
>>>>> enabled.
>>>>> I kinda expected slightly more IO during writes compared to 5.15, 
>>>>> but
>>>>> what I am actually seeing is massive amounts of btrfs-cleaner i/o 
>>>>> even
>>>>> when no programs are actively writing to the disk.
>>>>> 
>>>>> I can reproduce it quite reliably on my 2TB Btrfs Steam library
>>>>> partition. In my case, I was downloading Strange Brigade, which 
>>>>> is a
>>>>> roughly 25GB download and 33.65GB on disk. Somewhere during the
>>>>> download, iostat will start reporting disk writes around 300 
>>>>> MB/s, even
>>>>> though Steam itself reports disk usage of 40-45MB/s. After the 
>>>>> download
>>>>> finishes and nothing else is being written to disk, I still see 
>>>>> around
>>>>> 90-150MB/s worth of disk writes. Checking in iotop, I can see 
>>>>> btrfs
>>>>> cleaner and other btrfs processes writing a lot of data.
>>>>> 
>>>>> I left it running for a while to see if it was just some 
>>>>> maintenance
>>>>> tasks that Btrfs needed to do, but it just kept going. I tried to
>>>>> reboot, but it actually prevented me from properly rebooting. 
>>>>> After
>>>>> systemd timed out, my system finally shutdown.
>>>>> 
>>>>> Here are my mount options:
>>>>> rw,relatime,compress-force=zstd:2,ssd,autodefrag,space_cache=v2,subvolid=5,subvol=/
>>>>> 
>>>>> 
>>>> 
>>>> Compression, I guess that's the reason.
>>>> 
>>>>  From the very beginning, btrfs defrag doesn't handle compressed 
>>>> extent
>>>> well.
>>>> 
>>>> Even if a compressed extent is already at its maximum capacity, 
>>>> btrfs
>>>> will still try to defrag it.
>>>> 
>>>> I believe the behavior is masked by other problems in older 
>>>> kernels thus
>>>> not that obvious.
>>>> 
>>>> But after rework of defrag in v5.16, this behavior is more exposed.
>>> 
>>> And if possible, please try this diff on v5.15.x, and see if v5.15 
>>> is
>>> really doing less IO than v5.16.x.
>>> 
>>> The diff will solve a problem in the old code, where autodefrag is
>>> almost not working.
>>> 
>>> diff --git a/fs/btrfs/ioctl.c b/fs/btrfs/ioctl.c
>>> index cc61813213d8..f6f2468d4883 100644
>>> --- a/fs/btrfs/ioctl.c
>>> +++ b/fs/btrfs/ioctl.c
>>> @@ -1524,13 +1524,8 @@ int btrfs_defrag_file(struct inode *inode, 
>>> struct
>>> file *file,
>>>                         continue;
>>>                 }
>>> 
>>> -               if (!newer_than) {
>>> -                       cluster = (PAGE_ALIGN(defrag_end) >>
>>> -                                  PAGE_SHIFT) - i;
>>> -                       cluster = min(cluster, max_cluster);
>>> -               } else {
>>> -                       cluster = max_cluster;
>>> -               }
>>> +               cluster = (PAGE_ALIGN(defrag_end) >> PAGE_SHIFT) - 
>>> i;
>>> +               cluster = min(cluster, max_cluster);
>>> 
>>>                 if (i + cluster > ra_index) {
>>>                         ra_index = max(i, ra_index);
>>> 
>>>> 
>>>> There are patches to address the compression related problem, but 
>>>> not
>>>> yet merged:
>>>> 
>>>> https://patchwork.kernel.org/project/linux-btrfs/list/?series=609387
>>>> 
>>>> Mind to test them to see if that's the case?
>>>> 
>>>> Thanks,
>>>> Qu
>>>>> 
>>>>> 
>>>>> I've disabled autodefrag again for now to save my SSD, but just 
>>>>> wanted
>>>>> to say that there is still an issue. Have the defrag issues been 
>>>>> fully
>>>>> fixed or are there more patches incoming despite what Reddit and
>>>>> Phoronix say? XD
>>>>> 
>>>>> Thanks!
>>>>> Ben
>>>>> 
>>>>> 
>> 
>>