Re: [PATCH v6 4/5] mm/migrate: skip migrating folios under writeback with AS_WRITEBACK_INDETERMINATE mappings

[David Hildenbrand <david@redhat.com>]

On 23.12.24 23:14, Shakeel Butt wrote:
> On Sat, Dec 21, 2024 at 05:18:20PM +0100, David Hildenbrand wrote:
> [...]
>>
>> Yes, so I can see fuse
>>
>> (1) Breaking memory reclaim (memory cannot get freed up)
>>
>> (2) Breaking page migration (memory cannot be migrated)
>>
>> Due to (1) we might experience bigger memory pressure in the system I guess.
>> A handful of these pages don't really hurt, I have no idea how bad having
>> many of these pages can be. But yes, inherently we cannot throw away the
>> data as long as it is dirty without causing harm. (maybe we could move it to
>> some other cache, like swap/zswap; but that smells like a big and
>> complicated project)
>>
>> Due to (2) we turn pages that are supposed to be movable possibly for a long
>> time unmovable. Even a *single* such page will mean that CMA allocations /
>> memory unplug can start failing.
>>
>> We have similar situations with page pinning. With things like O_DIRECT, our
>> assumption/experience so far is that it will only take a couple of seconds
>> max, and retry loops are sufficient to handle it. That's why only long-term
>> pinning ("indeterminate", e.g., vfio) migrate these pages out of
>> ZONE_MOVABLE/MIGRATE_CMA areas in order to long-term pin them.
>>
>>
>> The biggest concern I have is that timeouts, while likely reasonable it many
>> scenarios, might not be desirable even for some sane workloads, and the
>> default in all system will be "no timeout", letting the clueless admin of
>> each and every system out there that might support fuse to make a decision.
>>
>> I might have misunderstood something, in which case I am very sorry, but we
>> also don't want CMA allocations to start failing simply because a network
>> connection is down for a couple of minutes such that a fuse daemon cannot
>> make progress.
>>
> 
> I think you have valid concerns but these are not new and not unique to
> fuse. Any filesystem with a potential arbitrary stall can have similar
> issues. The arbitrary stall can be caused due to network issues or some
> faultly local storage.

What concerns me more is that this is can be triggered by even 
unprivileged user space, and that there is no default protection as far 
as I understood, because timeouts cannot be set universally to a sane 
defaults.

Again, please correct me if I got that wrong.


BTW, I just looked at NFS out of interest, in particular 
nfs_page_async_flush(), and I spot some logic about re-dirtying pages + 
canceling writeback. IIUC, there are default timeouts for UDP and TCP, 
whereby the TCP default one seems to be around 60s (* retrans?), and the 
privileged user that mounts it can set higher ones. I guess one could 
run into similar writeback issues?

So I wonder why we never required AS_WRITEBACK_INDETERMINATE for nfs? 
Not sure if I grasped all details about NFS and writeback and when it 
would redirty+end writeback, and if there is some other handling in there.

> 
> Regarding the reclaim, I wouldn't say fuse or similar filesystem are
> breaking memory reclaim as the kernel has mechanism to throttle the
> threads dirtying the file memory to reduce the chance of situations
> where most of memory becomes unreclaimable due to being dirty.

Yes, likely even cgroups can easily limit the amount.

> 
> Please note that such filesystems are mostly used in environments like
> data center or hyperscalar and usually have more advanced mechanisms to
> handle and avoid situations like long delays. For such environment
> network unavailability is a larger issue than some cma allocation
> failure. My point is: let's not assume the disastrous situaion is normal
> and overcomplicate the solution.

Let me summarize my main point: ZONE_MOVABLE/MIGRATE_CMA must only be 
used for movable allocations.

Mechanisms that possible turn these folios unmovable for a 
long/indeterminate time must either fail or migrate these folios out of 
these regions, otherwise we start violating the very semantics why 
ZONE_MOVABLE/MIGRATE_CMA was added in the first place.

Yes, there are corner cases where we cannot guarantee movability (e.g., 
OOM when allocating a migration destination), but these are not cases 
that can be triggered by (unprivileged) user space easily.

That's why FOLL_LONGTERM pinning does exactly that: even if user space 
would promise that this is really only "short-term", we will treat it as 
"possibly forever", because it's under user-space control.


Instead of having more subsystems violate these semantics because 
"performance" ... I would hope we would do better. Maybe it's an issue 
for NFS as well ("at least" only for privileged user space)? In which 
case, again, I would hope we would do better.


Anyhow, I'm hoping there will be more feedback from other MM folks, but 
likely right now a lot of people are out (just like I should ;) ).

If I end up being the only one with these concerns, then likely people 
can feel free to ignore them. ;)

-- 
Cheers,

David / dhildenb