Re: Still seeing high autodefrag IO with kernel 5.16.5

[Benjamin Xiao <ben.r.xiao@gmail.com>]

There's definitely still an issue even with the patches. I ended up 
disabling autodefrag again and rebooting my computer after 
btrfs-cleaner wrote about 300-ish GB to my SSD.

The patch does help things out a bit compared to before where it was a 
constant non-stop stream of IO, but 300GB worth of extra writes for 
33GB of actual data doesn't seem normal.

Ben

On Fri, Feb 4 2022 at 09:36:22 AM -0800, Benjamin Xiao 
<ben.r.xiao@gmail.com> wrote:
> Okay, I just tested right now with my custom 5.16.5 kernel with your 
> 3 patches applied. Redownloading the same game, I noticed that there 
> was significantly less IO load during the download, which is great. 
> It looked kinda bursty instead, with periods of no load, and then 
> periods of load ranging in the 150-200MB/s range.
> 
> After the download, I am no longer getting that constant 90-150MB/s 
> disk write from btrfs-cleaner, but I am seeing periodic bursts of it 
> every 10 seconds or so. These bursts last for around 3 seconds and 
> load is anywhere from 50-300MB/s.
> 
> Is this normal? Does autodefrag only defrag newly written data, or 
> does it sometimes go back and run defrag on data written previously? 
> I am gonna let it run for a bit to see if it eventually subsides.
> 
> Ben
> 
> On Fri, Feb 4 2022 at 02:20:44 PM +0800, Qu Wenruo 
> <quwenruo.btrfs@gmx.com> wrote:
>> 
>> 
>> On 2022/2/4 12:32, Benjamin Xiao wrote:
>>> Okay, I just compiled a custom Arch kernel with your patches 
>>> applied.
>>> Will test soon. Besides enabling autodefrag and redownloading a game
>>> from Steam, what other sorts of tests should I do?
>> 
>> As long as your workload can trigger the problem reliably, nothing 
>> \x7felse.
>> 
>> Thanks,
>> Qu
>> 
>>> 
>>> Ben
>>> 
>>> On Fri, Feb 4 2022 at 09:54:19 AM +0800, Qu Wenruo
>>> <quwenruo.btrfs@gmx.com> wrote:
>>>> 
>>>> 
>>>> On 2022/2/4 09:17, Qu Wenruo wrote:
>>>>> 
>>>>> 
>>>>> On 2022/2/4 04:05, Benjamin Xiao wrote:
>>>>>> Hello all,
>>>>>> 
>>>>>> Even after the defrag patches that landed in 5.16.5, I am still 
>>>>>> \x7f\x7f\x7f\x7f\x7fseeing
>>>>>> lots of cpu usage and disk writes to my SSD when autodefrag is 
>>>>>> \x7f\x7f\x7f\x7f\x7fenabled.
>>>>>> I kinda expected slightly more IO during writes compared to 
>>>>>> 5.15, \x7f\x7f\x7f\x7f\x7fbut
>>>>>> what I am actually seeing is massive amounts of btrfs-cleaner 
>>>>>> i/o \x7f\x7f\x7f\x7f\x7feven
>>>>>> when no programs are actively writing to the disk.
>>>>>> 
>>>>>> I can reproduce it quite reliably on my 2TB Btrfs Steam library
>>>>>> partition. In my case, I was downloading Strange Brigade, which 
>>>>>> \x7f\x7f\x7f\x7f\x7fis a
>>>>>> roughly 25GB download and 33.65GB on disk. Somewhere during the
>>>>>> download, iostat will start reporting disk writes around 300 
>>>>>> \x7f\x7f\x7f\x7f\x7fMB/s, even
>>>>>> though Steam itself reports disk usage of 40-45MB/s. After the 
>>>>>> \x7f\x7f\x7f\x7f\x7fdownload
>>>>>> finishes and nothing else is being written to disk, I still see 
>>>>>> \x7f\x7f\x7f\x7f\x7faround
>>>>>> 90-150MB/s worth of disk writes. Checking in iotop, I can see 
>>>>>> \x7f\x7f\x7f\x7f\x7fbtrfs
>>>>>> cleaner and other btrfs processes writing a lot of data.
>>>>>> 
>>>>>> I left it running for a while to see if it was just some 
>>>>>> \x7f\x7f\x7f\x7f\x7fmaintenance
>>>>>> tasks that Btrfs needed to do, but it just kept going. I tried to
>>>>>> reboot, but it actually prevented me from properly rebooting. 
>>>>>> \x7f\x7f\x7f\x7f\x7fAfter
>>>>>> systemd timed out, my system finally shutdown.
>>>>>> 
>>>>>> Here are my mount options:
>>>>>> rw,relatime,compress-force=zstd:2,ssd,autodefrag,space_cache=v2,subvolid=5,subvol=/
>>>>>> 
>>>>>> 
>>>>> 
>>>>> Compression, I guess that's the reason.
>>>>> 
>>>>>  From the very beginning, btrfs defrag doesn't handle compressed 
>>>>> \x7f\x7f\x7f\x7fextent
>>>>> well.
>>>>> 
>>>>> Even if a compressed extent is already at its maximum capacity, 
>>>>> \x7f\x7f\x7f\x7fbtrfs
>>>>> will still try to defrag it.
>>>>> 
>>>>> I believe the behavior is masked by other problems in older 
>>>>> \x7f\x7f\x7f\x7fkernels thus
>>>>> not that obvious.
>>>>> 
>>>>> But after rework of defrag in v5.16, this behavior is more 
>>>>> exposed.
>>>> 
>>>> And if possible, please try this diff on v5.15.x, and see if v5.15 
>>>> \x7f\x7f\x7fis
>>>> really doing less IO than v5.16.x.
>>>> 
>>>> The diff will solve a problem in the old code, where autodefrag is
>>>> almost not working.
>>>> 
>>>> diff --git a/fs/btrfs/ioctl.c b/fs/btrfs/ioctl.c
>>>> index cc61813213d8..f6f2468d4883 100644
>>>> --- a/fs/btrfs/ioctl.c
>>>> +++ b/fs/btrfs/ioctl.c
>>>> @@ -1524,13 +1524,8 @@ int btrfs_defrag_file(struct inode *inode, 
>>>> \x7f\x7f\x7fstruct
>>>> file *file,
>>>>                         continue;
>>>>                 }
>>>> 
>>>> -               if (!newer_than) {
>>>> -                       cluster = (PAGE_ALIGN(defrag_end) >>
>>>> -                                  PAGE_SHIFT) - i;
>>>> -                       cluster = min(cluster, max_cluster);
>>>> -               } else {
>>>> -                       cluster = max_cluster;
>>>> -               }
>>>> +               cluster = (PAGE_ALIGN(defrag_end) >> PAGE_SHIFT) - 
>>>> \x7f\x7f\x7fi;
>>>> +               cluster = min(cluster, max_cluster);
>>>> 
>>>>                 if (i + cluster > ra_index) {
>>>>                         ra_index = max(i, ra_index);
>>>> 
>>>>> 
>>>>> There are patches to address the compression related problem, but 
>>>>> \x7f\x7f\x7f\x7fnot
>>>>> yet merged:
>>>>> 
>>>>> https://patchwork.kernel.org/project/linux-btrfs/list/?series=609387
>>>>> 
>>>>> Mind to test them to see if that's the case?
>>>>> 
>>>>> Thanks,
>>>>> Qu
>>>>>> 
>>>>>> 
>>>>>> I've disabled autodefrag again for now to save my SSD, but just 
>>>>>> \x7f\x7f\x7f\x7f\x7fwanted
>>>>>> to say that there is still an issue. Have the defrag issues been 
>>>>>> \x7f\x7f\x7f\x7f\x7ffully
>>>>>> fixed or are there more patches incoming despite what Reddit and
>>>>>> Phoronix say? XD
>>>>>> 
>>>>>> Thanks!
>>>>>> Ben
>>>>>> 
>>>>>> 
>>> 
>>> 
> 
>