Re: [PATCH] Deadlock during heavy write activity to userspace NFS

Re: [PATCH] Deadlock during heavy write activity to userspace NFS

[Ulrich Weigand]

Avi Kivity wrote:

>In our case, all block I/O is done using 
>unbuffered I/O, and all memory is preallocated, so we don't need kswapd 
>at all, just that small bit of memory that syscalls consume.

Does your userspace process need to send/receive network packets
in order to perform a write-out?  If so, how can you make sure your
incoming packets aren't thrown away in out-of-memory situations?
(Outgoing packets can use PF_MEMALLOC memory I guess, but incoming
ones aren't associated to any process yet ...)

Bye,
Ulrich

-- 
  Dr. Ulrich Weigand
  weigand@informatik.uni-erlangen.de

Re: [PATCH] Deadlock during heavy write activity to userspace NFS

[Avi Kivity]

Ulrich Weigand wrote:

>Avi Kivity wrote:
>
>  
>
>>In our case, all block I/O is done using 
>>unbuffered I/O, and all memory is preallocated, so we don't need kswapd 
>>at all, just that small bit of memory that syscalls consume.
>>    
>>
>
>Does your userspace process need to send/receive network packets
>in order to perform a write-out?  
>
Yes.

>If so, how can you make sure your
>incoming packets aren't thrown away in out-of-memory situations?
>(Outgoing packets can use PF_MEMALLOC memory I guess, but incoming
>ones aren't associated to any process yet ...)
>
>  
>
I did nothing to address this. So far it works well, even under heavy 
load. I guess a general solution needs to address this as well.

The kernel NFS client (which kswapd depends on) has the same issue. Has 
anyone ever observed kswapd deadlock due to imcoming or outgoing NFS 
packets being discarded due to oom?

Avi

Re: [PATCH] Deadlock during heavy write activity to userspace NFS

[Nick Piggin]

Avi Kivity wrote:
> Ulrich Weigand wrote:
> 
>> Avi Kivity wrote:
>>
>>  
>>
>>> In our case, all block I/O is done using unbuffered I/O, and all 
>>> memory is preallocated, so we don't need kswapd at all, just that 
>>> small bit of memory that syscalls consume.
>>>   
>>
>>
>> Does your userspace process need to send/receive network packets
>> in order to perform a write-out? 
> 
> Yes.
> 
>> If so, how can you make sure your
>> incoming packets aren't thrown away in out-of-memory situations?
>> (Outgoing packets can use PF_MEMALLOC memory I guess, but incoming
>> ones aren't associated to any process yet ...)
>>
>>  
>>
> I did nothing to address this. So far it works well, even under heavy 
> load. I guess a general solution needs to address this as well.
> 
> The kernel NFS client (which kswapd depends on) has the same issue. Has 
> anyone ever observed kswapd deadlock due to imcoming or outgoing NFS 
> packets being discarded due to oom?
> 

Yes this has been observed.

alloc_skb on the client needs to somehow know that traffic coming
from the server is "MEMALLOC" and allowed to use memory reserves.

Re: [PATCH] Deadlock during heavy write activity to userspace NFS

[Ulrich Weigand]

Nick Piggin wrote:

> Avi Kivity wrote:
> > The kernel NFS client (which kswapd depends on) has the same issue. Has 
> > anyone ever observed kswapd deadlock due to imcoming or outgoing NFS 
> > packets being discarded due to oom?
> > 
> 
> Yes this has been observed.
> 
> alloc_skb on the client needs to somehow know that traffic coming
> from the server is "MEMALLOC" and allowed to use memory reserves.

What would be an appropriate way to solve this problem? A special
socket option?

Bye,
Ulrich

-- 
  Dr. Ulrich Weigand
  weigand@informatik.uni-erlangen.de

Re: [PATCH] Deadlock during heavy write activity to userspace NFS

[Nick Piggin]

Ulrich Weigand wrote:

>Nick Piggin wrote:
>
>
>>Avi Kivity wrote:
>>
>>>The kernel NFS client (which kswapd depends on) has the same issue. Has 
>>>anyone ever observed kswapd deadlock due to imcoming or outgoing NFS 
>>>packets being discarded due to oom?
>>>
>>>
>>Yes this has been observed.
>>
>>alloc_skb on the client needs to somehow know that traffic coming
>>from the server is "MEMALLOC" and allowed to use memory reserves.
>>
>
>What would be an appropriate way to solve this problem? A special
>socket option?
>
>

What I think was happening is dirty NFS pages being written out in response
to low memory. The acks coming back from the server weren't being received
because memory couldn't be allocated for them.

Ideally you would allow the NFS path to allocate enough memory that it can
guarantee forward progress - like how the block layer uses mempools for
struct requests, for example.