Re: The future of PageAnonExclusive

David Hildenbrand
2024-12-11T11:56:24Z

Now CCing the correct Willy :)

On 11.12.24 12:55, David Hildenbrand wrote:
> Hi,
>
> PageAnonExclusive (PAE) is working very reliable at this point. But
> especially in the context of THPs (large folios) we'd like to do better:
>
> (1) For PTE-mapped THP, we have to maintain it per page. We'd like to
> avoid per-page flags as good as possible (e.g., waste in "struct
> page", touching many cachelines).
>
> (2) We currently have to use atomics to set/clear the flag, even when
> working on tail pages. While there would be ways to mitigate this
> when modifying the flags of multiple tail pages (bitlock protecting
> all tail page flag updates), I'd much rather avoid messing with
> page tail flags at all.
>
>
> In general, the PAE bit can be considered an extended PTE bit that we
> currently store in the "struct page" that is mapped by the PTE. Ideally,
> we'd just store that information in the PTE, or alongside the PTE:
>
> A writable PTE implies PAE. A write-protected PTE needs additional
> information whether it is PAE (-> whether we can just remap it writable,
> FOLL_FORCE to it, PIN it ...).
>
> We are out of PTE bits, especially when having to implement it across
> *all* architectures. That's one of the reasons we went with PAE back
> then. As a nice side-effect it allowed for sanity checks when unpinning
> folios (-> PAE must still be set, which is impossible when the
> information stored in the PTE).
>
>
> There are 3 main approaches I've been looking into:
>
> (A) Make it a per-folio flag. I've spent endless hours trying to get it
> conceptually right, but it's just a big pain: as soon as we clear
> the flag, we have to make sure that all PTEs are write-protected,
> that the folio is not pinned, and that concurrent GUP cannot work.
> So far the page table lock protected the PAE bit, but with a per-
> folio flag that is not guaranteed for THPs.
>
> fork() with things like VM_DONTCOPY, VM_DONTFORK, early-abort, page
> migration/swapout that can happen any time during fork etc. make
> this really though to get right with THPs. My head hurts any time I
> think about it.
>
> While I think fork() itself can be handled, the concurrent page
> migration / swapout is where it gets extremely tricky.
>
> This can be done somehow I'm sure, but the devil is in the corner
> cases when having multiple PTEs mapping a large folio. We'd still
> need atomics to set/clear the single folio flag, because of the
> nature of concurrent folio flag updates.
>
> (B) Allocate additional metadata (PAE bitmap) for page tables, protected
> by the PTL. That is, when we want to *clear* PAE (fork(), KSM), we'd
> lazily allocate the bitmap and store it for our page table.
>
> On x86: 512 PTEs -> 512bits -> 64byte
>
> fork() gets a bit more expensive, because we'd have to allocate this
> bitmap for the parent and the child page table we are working on, so
> we can mark the pages as "!PAE" in both page tables.
>
> This could work, I have not prototyped it. We'd have to support it
> on the PTE/PMD/PUD-table level.
>
> One tricky thing is having multiple pagetables per page, but I
> assume it can be handled (we should have a single PT lock for all of
> them IIRC, and only need to address the bitmap at the right offset).
>
> Another challenge is how to link to this metadata from ptdesc on all
> archs.. So far, __page_mapping is unused, and could maybe be used to
> link to such metadata -- at least page tables can be identified
> reliably using the page type.
>
> (C) Encode it in the PTE.
>
> pte_write() -> PAE
>
> !pte_write() && pte_dirty() -> PAE
>
> !pte_write && !pte_dirty() -> !PAE
>
> That implies, that when wrprotecting a PTE, we'd have to move the
> dirty bit to the folio. When wr-unprotecting it, we could mark the
> PTE dirty if the folio is dirty.
>
> I suspect that most anon folios are dirty most of the time either
> way, and the common case of having them just writable in the PTE
> wouldn't change.
>
> The main idea is that nobody (including HW) should ever be marking a
> readonly PTE dirty (so the theory behind it). We have to take good
> care whenever we modify/query the dirty bit or modify the writable
> bit.
>
> There is quite some code to audit/sanitize. Further, we'd have to
> decouple softdirty PTE handling from dirty PTE handling (pte_mkdirty
> sets the pte softdirty), and adjust arm64 cont-pte and similar PTE
> batching code to respect the per-PTE dirty bit when
> the PTE is write-protected.
>
> This would be the most elegant solution, but requires a bit of care
> + sanity checks.
>
>
> Any thoughts or other ideas?