Re: [Qemu-devel] Dual userfaultfd behavior

["Dr. David Alan Gilbert" <dgilbert@redhat.com>]

* Alexey Perevalov (a.perevalov@samsung.com) wrote:
> Hi, David, Andrea and Mike

Hi Alexey,

> The problem I want to discuss it's 1G hugepage based VM and post copy live
> migration.
> 
> I would like to know your opinion on following approach of avoiding such
> problem:
> Once we have mmap'ed area through 1G hugetlbfs, remap physical pages
> with /dev/mem. It will be 2 types of vmas mapped to the same PFN.
> Register userfaultfd for newly obtained virtual
> addresses, it could reduce granularity of pages and reduce downtime per
> one 1G page. So registering userfaultfd for 2Mb, when the real hugepage
> was 1G, I think, could help.
> 
> Current postcopy implementation in QEMU allows to make live migration
> from 1G based hugepage VM to 2Mb based hugepages VM (sanity checks prevent
> it).
> 
> Also I checked, it's possible to remap through /dev/mem and get PFN
> based vmas, register userfaultfd (with allowance in vma_can_userfault)
> and finally make UFFDIO_COPY with allowing PFN based vmas in __mcopy_atomic.
> 
> But there are a lot of drawback of such approach:
> First of all it's /dev/mem interface. Need to provide full access
> (kernel w/o CONFIG_STRICT_DEVMEM) and need to disable PAT.
> The second drawback, maybe I just didn't find possibility to remap
> hugepages again, but mmap of /dev/mem character driver maps 4Kb pages.
> I don't know how THP could help here, but madvise with MADV_HUGEPAGE
> didn't. So 4Kb is not exactly what needed, due to overhead of
> encapsulation summary downtime is worse than in other cases.
> It would be great to have interface to obtain new virtual address based
> on existing PFN, but for hugepages.

Yes, and I think as well on some architectures there can be cache problems
from mapping the same page in two addresses unless we're careful.

I think to do this we'd basically need the kernel to set up something
similar to what you're saying, but without the mess of having to
go via /dev/mem.  Ideally it would all happen magically when I mark
a hugetlb page as userfault and start UFFDIO_COPYing in 4kb pages;
but I can imagine perhaps some more syscalls needed to tell it to do it.

I've no idea how hard that is to do though.

Dave

> Honestly, I can't find another use case for this feature.
> 
> 
> -- 
> 
> BR
> Alexey
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK