[LSF/MM TOPIC]: userfaultfd (was: [LSF/MM TOPIC] NUMA remote THP vs NUMA local non-THP under MADV_HUGEPAGE)

[Mike Rapoport <rppt@linux.ibm.com>]

Hi,

(changed the subject and added CRIU folks)

On Tue, Jan 29, 2019 at 06:40:58PM -0500, Andrea Arcangeli wrote:
> Hello,
> 
> --
> 
> In addition to the above "NUMA remote THP vs NUMA local non-THP
> tradeoff" topic, there are other developments in "userfaultfd" land that
> are approaching merge readiness and that would be possible to provide a
> short overview about:
> 
> - Peter Xu made significant progress in finalizing the userfaultfd-WP
>   support over the last few months. That feature was planned from the
>   start and it will allow userland to do some new things that weren't
>   possible to achieve before. In addition to synchronously blocking
>   write faults to be resolved by an userland manager, it has also the
>   ability to obsolete the softdirty feature, because it can provide
>   the same information, but with O(1) complexity (as opposed of the
>   current softdirty O(N) complexity) similarly to what the Page
>   Modification Logging (PML) does in hardware for EPT write accesses.
 
We (CRIU) have some concerns about obsoleting soft-dirty in favor of
uffd-wp. If there are other soft-dirty users these concerns would be
relevant to them as well.

With soft-dirty we collect the information about the changed memory every
pre-dump iteration in the following manner:
* freeze the tasks
* find entries in /proc/pid/pagemap with SOFT_DIRTY set
* unfreeze the tasks
* dump the modified pages to disk/remote host

While we do need to traverse the /proc/pid/pagemap to identify dirty pages,
in between the pre-dump iterations and during the actual memory dump the
tasks are running freely.

If we are to switch to uffd-wp, every write by the snapshotted/migrated
task will incur latency of uffd-wp processing by the monitor.

We'd need to see how this affects overall slowdown of the workload under
migration before moving forward with obsoleting soft-dirty.

> - Blake Caldwell maintained the UFFDIO_REMAP support to atomically
>   remove memory from a mapping with userfaultfd (which can't be done
>   with a copy as in UFFDIO_COPY and it requires a slow TLB flush to be
>   safe) as an alternative to host swapping (which of course also
>   requires a TLB flush for similar reasons). Notably UFFDIO_REMAP was
>   rightfully naked early on and quickly replaced by UFFDIO_COPY which
>   is more optimal to add memory to a mapping is small chunks, but we
>   can't remove memory with UFFDIO_COPY and UFFDIO_REMAP should be as
>   efficient as it gets when it comes to removing memory from a
>   mapping.

If we are to discuss userfaultfd, I'd like also to bring the subject of COW
mappings.
The pages populated with UFFDIO_COPY cannot be COW-shared between related
processes which unnecessarily increases memory footprint of a migrated
process tree.
I've posted a patch [1] a (real) while ago, but nobody reacted and I've put
this aside.
Maybe it's time to discuss it again :)

> Thank you,
> Andrea
> 

[1] https://lwn.net/ml/linux-api/20180328101729.GB1743%40rapoport-lnx/

-- 
Sincerely yours,
Mike.