Re: [PATCH] Deadlock during heavy write activity to userspace NFS server on local NFS mount

[Avi Kivity <avi@exanet.com>]

Nick Piggin wrote:

> Avi Kivity wrote:
>
>> Nick Piggin wrote:
>>
>>> Avi Kivity wrote:
>>>
>>>> Nick Piggin wrote:
>>>
>>>
>>>
>>>>>> What's stopping the NFS server from ooming the machine then? 
>>>>>> Every time some bit of memory becomes free, the server will 
>>>>>> consume it instantly. Eventually ext3 will not be able to write 
>>>>>> anything out because it is out of memory.
>>>>>>
>>>>> The NFS server should do the writeout a page at a time.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> The NFS server writes not only in response to page reclaim (as a 
>>>> local NFS client), but also in response to pressure from non-local 
>>>> clients. If both ext3 and NFS have the same allocation limits, NFS 
>>>> may starve out ext3.
>>>>
>>>
>>> What do you mean starve out ext3? ext3 gets written to *by the NFS 
>>> server*
>>> which is PF_MEMALLOC. 
>>
>>
>>
>> When the NFS server writes, it allocates pagecache and temporary 
>> objects. When ext3 writes, it allocates temporary objects. If the NFS 
>> server writes too much, ext3 can't allocate memory, and will never be 
>> able to allocate memory.
>>
>
> That is because your NFS server shouldn't hog as much memory as
> it likes when it is PF_MEMALLOC. The entire writeout path should
> do a page at a time if it is PF_MEMALLOC. Ie, the server should
> be doing write, fsync. 

We attempted to use sync local mounts (not what you are suggesting: on 
the NFS client side, without the PF_MEMALLOC hack) and still got the 
same deadlock. I am unable to explain why.

>
>
> But now that I think about it, I guess you may not be able to
> distinguish that from regular writeout, so doing a page at a time
> would hurt performance too much.
>
> Hmm so I guess the idea of a per task reserve limit may be the way
> to do it, yes. Thanks for bearing with me!

It was my pleasure.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.