deadlock with btrfs heavy i/o and kswapd

deadlock with btrfs heavy i/o and kswapd

[Stefan Priebe - Profihost AG]

Hi,

i always encounter btrfs deadlocks / hung tasks, when i have a lot of
cached mem and i'm doing heavy rsync --inplace operations in my system
from btrfs zlib compressed disk A to btrfs zlib compressed disk B.

The last output i see in this case is kswapd0 running for a long time at
100% cpu. Then the whole system get's stuck. I cannot connect to ssh
anymore but the kernel still prints hung tasks every few minutes.

May be relevant the system has NO swap.

vm.vfs_cache_pressure = 100
vm.swappiness = 50

Greets,
Stefan

Re: deadlock with btrfs heavy i/o and kswapd

[Chris Mason]

On 09/22/2016 02:41 AM, Stefan Priebe - Profihost AG wrote:
> Hi,
>
> i always encounter btrfs deadlocks / hung tasks, when i have a lot of
> cached mem and i'm doing heavy rsync --inplace operations in my system
> from btrfs zlib compressed disk A to btrfs zlib compressed disk B.
>
> The last output i see in this case is kswapd0 running for a long time at
> 100% cpu. Then the whole system get's stuck. I cannot connect to ssh
> anymore but the kernel still prints hung tasks every few minutes.
>
> May be relevant the system has NO swap.
>
> vm.vfs_cache_pressure = 100
> vm.swappiness = 50

Are you able to capture the stack dumps?  A sysrq-w would really help.

-chris

Re: deadlock with btrfs heavy i/o and kswapd

[Stefan Priebe - Profihost AG]

Am 22.09.2016 um 16:28 schrieb Chris Mason:
> 
> 
> On 09/22/2016 02:41 AM, Stefan Priebe - Profihost AG wrote:
>> Hi,
>>
>> i always encounter btrfs deadlocks / hung tasks, when i have a lot of
>> cached mem and i'm doing heavy rsync --inplace operations in my system
>> from btrfs zlib compressed disk A to btrfs zlib compressed disk B.
>>
>> The last output i see in this case is kswapd0 running for a long time at
>> 100% cpu. Then the whole system get's stuck. I cannot connect to ssh
>> anymore but the kernel still prints hung tasks every few minutes.
>>
>> May be relevant the system has NO swap.
>>
>> vm.vfs_cache_pressure = 100
>> vm.swappiness = 50
> 
> Are you able to capture the stack dumps?  A sysrq-w would really help.

I wasn't as my netconsole didn't work even on the screen no traces were
visible. Currently i can't reproduce this. I'm still fighting with my
ENOSPC problems while using btrfs and reflinks which is a currently a
showstopper for me.

Greets,
Stefan

> -chris
> 

Re: deadlock with btrfs heavy i/o and kswapd

[Stefan Priebe - Profihost AG]

Hi Chris,

today i had this again. But i can't see any stack traces. I just see.

INFO: kworker/u128:5:24301 blocked for more than 120 seconds.
...
INFO: kworker/u128:5:24301 blocked for more than 120 seconds.
...
INFO: task mysqld:929 blocked for more ...
...

sysrq w just prints:
sysrq: SysRq: Show Blcoked State

but nothing more.

Stefan
Am 22.09.2016 um 16:28 schrieb Chris Mason:
> 
> 
> On 09/22/2016 02:41 AM, Stefan Priebe - Profihost AG wrote:
>> Hi,
>>
>> i always encounter btrfs deadlocks / hung tasks, when i have a lot of
>> cached mem and i'm doing heavy rsync --inplace operations in my system
>> from btrfs zlib compressed disk A to btrfs zlib compressed disk B.
>>
>> The last output i see in this case is kswapd0 running for a long time at
>> 100% cpu. Then the whole system get's stuck. I cannot connect to ssh
>> anymore but the kernel still prints hung tasks every few minutes.
>>
>> May be relevant the system has NO swap.
>>
>> vm.vfs_cache_pressure = 100
>> vm.swappiness = 50
> 
> Are you able to capture the stack dumps?  A sysrq-w would really help.
> 
> -chris
>