spinning kworker with space_cache=v2 searching for free space

spinning kworker with space_cache=v2 searching for free space

[Stefan Priebe - Profihost AG]

Dear list,

even there's a lot of free space on my disk:

# df -h /vmbackup/
Filesystem                    Size  Used Avail Use% Mounted on
/dev/mapper/stripe0-backup   37T   24T   13T  64% /backup

# btrfs filesystem df /backup/
Data, single: total=23.75TiB, used=22.83TiB
System, DUP: total=8.00MiB, used=3.94MiB
Metadata, DUP: total=283.50GiB, used=105.82GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

I always have a kworker process endless spinning.

# perf top shows:
  47,56%  [kernel]               [k] rb_next
   7,71%  [kernel]               [k] tree_search_offset.isra.25
   6,44%  [kernel]               [k] btrfs_find_space_for_alloc

Mount options:
rw,noatime,compress-force=zlib,nossd,noacl,space_cache=v2,skip_balance

What's wrong here?

Greets,
Stefan

Re: spinning kworker with space_cache=v2 searching for free space

[Stefan Priebe - Profihost AG]

Hello,

found this one from 2014:
https://patchwork.kernel.org/patch/5551651/

it this still valid?

Am 09.11.2016 um 09:09 schrieb Stefan Priebe - Profihost AG:
> Dear list,
> 
> even there's a lot of free space on my disk:
> 
> # df -h /vmbackup/
> Filesystem                    Size  Used Avail Use% Mounted on
> /dev/mapper/stripe0-backup   37T   24T   13T  64% /backup
> 
> # btrfs filesystem df /backup/
> Data, single: total=23.75TiB, used=22.83TiB
> System, DUP: total=8.00MiB, used=3.94MiB
> Metadata, DUP: total=283.50GiB, used=105.82GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B
> 
> I always have a kworker process endless spinning.
> 
> # perf top shows:
>   47,56%  [kernel]               [k] rb_next
>    7,71%  [kernel]               [k] tree_search_offset.isra.25
>    6,44%  [kernel]               [k] btrfs_find_space_for_alloc
> 
> Mount options:
> rw,noatime,compress-force=zlib,nossd,noacl,space_cache=v2,skip_balance
> 
> What's wrong here?
> 
> Greets,
> Stefan
> 

Re: spinning kworker with space_cache=v2 searching for free space

[Liu Bo]

On Wed, Nov 09, 2016 at 09:19:21PM +0100, Stefan Priebe - Profihost AG wrote:
> Hello,
> 
> found this one from 2014:
> https://patchwork.kernel.org/patch/5551651/
> 
> it this still valid?

The space cache code doesn't change a lot, so I think the patch is still
valid to apply(there might be some conflicts though), but I'm not sure
if it could help the spinning case.

Thanks,

-liubo
> 
> Am 09.11.2016 um 09:09 schrieb Stefan Priebe - Profihost AG:
> > Dear list,
> > 
> > even there's a lot of free space on my disk:
> > 
> > # df -h /vmbackup/
> > Filesystem                    Size  Used Avail Use% Mounted on
> > /dev/mapper/stripe0-backup   37T   24T   13T  64% /backup
> > 
> > # btrfs filesystem df /backup/
> > Data, single: total=23.75TiB, used=22.83TiB
> > System, DUP: total=8.00MiB, used=3.94MiB
> > Metadata, DUP: total=283.50GiB, used=105.82GiB
> > GlobalReserve, single: total=512.00MiB, used=0.00B
> > 
> > I always have a kworker process endless spinning.
> > 
> > # perf top shows:
> >   47,56%  [kernel]               [k] rb_next
> >    7,71%  [kernel]               [k] tree_search_offset.isra.25
> >    6,44%  [kernel]               [k] btrfs_find_space_for_alloc
> > 
> > Mount options:
> > rw,noatime,compress-force=zlib,nossd,noacl,space_cache=v2,skip_balance
> > 
> > What's wrong here?
> > 
> > Greets,
> > Stefan
> > 

Re: spinning kworker with space_cache=v2 searching for free space

[Stefan Priebe - Profihost AG]

Am 12.11.2016 um 03:18 schrieb Liu Bo:
> On Wed, Nov 09, 2016 at 09:19:21PM +0100, Stefan Priebe - Profihost AG wrote:
>> Hello,
>>
>> found this one from 2014:
>> https://patchwork.kernel.org/patch/5551651/
>>
>> it this still valid?
> 
> The space cache code doesn't change a lot, so I think the patch is still
> valid to apply(there might be some conflicts though), but I'm not sure
> if it could help the spinning case.

Thanks got it applied and will try it. Any other ideas why it's pinning
there? Free space fragmentation?

But at least on one machine there are 26TB free and it's spinning...
slowing down the performance.

Greets,
Stefan

> 
> Thanks,
> 
> -liubo
>>
>> Am 09.11.2016 um 09:09 schrieb Stefan Priebe - Profihost AG:
>>> Dear list,
>>>
>>> even there's a lot of free space on my disk:
>>>
>>> # df -h /vmbackup/
>>> Filesystem                    Size  Used Avail Use% Mounted on
>>> /dev/mapper/stripe0-backup   37T   24T   13T  64% /backup
>>>
>>> # btrfs filesystem df /backup/
>>> Data, single: total=23.75TiB, used=22.83TiB
>>> System, DUP: total=8.00MiB, used=3.94MiB
>>> Metadata, DUP: total=283.50GiB, used=105.82GiB
>>> GlobalReserve, single: total=512.00MiB, used=0.00B
>>>
>>> I always have a kworker process endless spinning.
>>>
>>> # perf top shows:
>>>   47,56%  [kernel]               [k] rb_next
>>>    7,71%  [kernel]               [k] tree_search_offset.isra.25
>>>    6,44%  [kernel]               [k] btrfs_find_space_for_alloc
>>>
>>> Mount options:
>>> rw,noatime,compress-force=zlib,nossd,noacl,space_cache=v2,skip_balance
>>>
>>> What's wrong here?
>>>
>>> Greets,
>>> Stefan
>>>

Re: spinning kworker with space_cache=v2 searching for free space

[Jean-Denis Girard]

Le 11/11/2016 à 21:58, Stefan Priebe - Profihost AG a écrit :
>>>> I always have a kworker process endless spinning.

Not sure if it is related to your problem, but I had btrfs-cleaner stuck
at 100%, see:
https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg56676.html

The problem disappeared after manually defragmenting all volumes.


Regards,
-- 
Jean-Denis Girard

SysNux                   Systèmes   Linux   en   Polynésie  française
https://www.sysnux.pf/   Tél: +689 40.50.10.40 / GSM: +689 87.797.527