Re: [Qemu-devel] [LSF/MM TOPIC][LSF/MM, ATTEND] shared TLB, hugetlb reservations

Re: [Qemu-devel] [LSF/MM TOPIC][LSF/MM, ATTEND] shared TLB, hugetlb reservations

[Andrea Arcangeli]

Hello,

On Wed, Mar 08, 2017 at 05:30:55PM -0800, Mike Kravetz wrote:
> On 01/10/2017 03:02 PM, Mike Kravetz wrote:
> > Another more concrete topic is hugetlb reservations.  Michal Hocko
> > proposed the topic "mm patches review bandwidth", and brought up the
> > related subject of areas in need of attention from an architectural
> > POV.  I suggested that hugetlb reservations was one such area.  I'm
> > guessing it was introduced to solve a rather concrete problem.  However,
> > over time additional hugetlb functionality was added and the
> > capabilities of the reservation code was stretched to accommodate.
> > It would be good to step back and take a look at the design of this
> > code to determine if a rewrite/redesign is necessary.  Michal suggested
> > documenting the current design/code as a first step.  If people think
> > this is worth discussion at the summit, I could put together such a
> > design before the gathering.
> 
> I attempted to put together a design/overview of how hugetlb reservations
> currently work.  Hopefully, this will be useful.

Another area of hugetlbfs that is not clear is the status of
MADV_REMOVE and the behavior of fallocate punch hole that deviates
from more standard shmem semantics. That might also be a topic of
interest related to your hugetlbfs topic and marginally related to
userfaultfd.

The current status for anon, shmem and hugetlbfs like this:

MADV_DONTNEED works: anon, !VM_SHARED shmem
MADV_DONTNEED doesn't work: hugetlbfs VM_SHARED, hugetlbfs !VM_SHARED
MADV_DONTNEED works but not guaranteed to fault: shmem VM_SHARED

MADV_REMOVE works: shmem VM_SHARED, hugetlbfs VM_SHARED
MADV_REMOVE doesn't work: anon, shmem !VM_SHARED, hugetlbfs !VM_SHARED

fallocate punch hole works: hugetlbfs VM_SHARED, hugetlbfs !VM_SHARED,
	  	     	    shmem VM_SHARED
fallocate punch hole doesn't work: anon, shmem !VM_SHARED

So what happens in qemu is:

anon			-> MADV_DONTNEED

shmem !VM_SHARED	-> MADV_DONTNEED (fallocate punch hole wouldn't zap
			   private pages, but it does on hugetlbfs)

shmem VM_SHARED		-> fallocate punch hole (MADV_REMOVE would
      			   work too)

hugetlbfs !VM_SHARED	-> fallocate punch hole (works for hugetlbfs
			   but not for shmem !VM_SHARED)

hugetlbfs VM_SHARED	-> fallocate punch hole (MADV_REMOVE would work too)

This means qemu has to carry around information on the type of memory
it got from the initial memblock setup, so at live migration time it
can zap the memory with the right call. (NOTE: such memory is not
generated by userfaultfd UFFDIO_COPY, but it was allocated and mapped
and it must be zapped well before calling userfaultfd the first time).

To do this qemu uses fstatfs and finds out which kind of memory it's
dealing with to use the right call depending on which memory.

In short it'd be better to have something like a generic MADV_REMOVE
that guarantees a non-present fault after it succeeds, no matter what
kind of memory is mapped in the virtual range that has to be
zapped. The above is far from ideal from a userland developer
prospective.

Overall fallocate punch hole covers the most cases so to keep the code
simpler ironically MADV_REMOVE ends up being never used despite it
provides a more friendly API than fallocate to qemu. The files are
always mapped and the older code only dealt with virtual addresses
(before hugetlbfs and shmem entered thee equation). Ideally qemu wants
to call the same madvise regardles if the memory is anon shmem or
hugetlbfs without having to carry around file descriptor, file offsets
and superblock types.

It's also not clear why MADV_DONTNEED doesn't work for hugetlbfs
!VM_SHARED mappings and why fallocate punch hole is also zapping
private cow-like pages from !VM_SHARED mappings (although if it
didn't, it would be impossible to zap those... so it's good luck it
does).

Thanks,
Andrea

PS. CC'ed also qemu-devel in case it may help clarify why things are
implemented they way they are in the postcopy live migration
hugetlbfs/shmem support and in the future patches for shmem/hugetlbfs
share=on.

Re: [Qemu-devel] [LSF/MM TOPIC][LSF/MM, ATTEND] shared TLB, hugetlb reservations

[Mike Kravetz]

On 03/14/2017 11:37 AM, Andrea Arcangeli wrote:
> Hello,
> 
> On Wed, Mar 08, 2017 at 05:30:55PM -0800, Mike Kravetz wrote:
>> On 01/10/2017 03:02 PM, Mike Kravetz wrote:
>>> Another more concrete topic is hugetlb reservations.  Michal Hocko
>>> proposed the topic "mm patches review bandwidth", and brought up the
>>> related subject of areas in need of attention from an architectural
>>> POV.  I suggested that hugetlb reservations was one such area.  I'm
>>> guessing it was introduced to solve a rather concrete problem.  However,
>>> over time additional hugetlb functionality was added and the
>>> capabilities of the reservation code was stretched to accommodate.
>>> It would be good to step back and take a look at the design of this
>>> code to determine if a rewrite/redesign is necessary.  Michal suggested
>>> documenting the current design/code as a first step.  If people think
>>> this is worth discussion at the summit, I could put together such a
>>> design before the gathering.
>>
>> I attempted to put together a design/overview of how hugetlb reservations
>> currently work.  Hopefully, this will be useful.
> 
> Another area of hugetlbfs that is not clear is the status of
> MADV_REMOVE and the behavior of fallocate punch hole that deviates
> from more standard shmem semantics. That might also be a topic of
> interest related to your hugetlbfs topic and marginally related to
> userfaultfd.

Thanks Andrea,

I was not aware qemu was carrying all this information.

> The current status for anon, shmem and hugetlbfs like this:
> 
> MADV_DONTNEED works: anon, !VM_SHARED shmem
> MADV_DONTNEED doesn't work: hugetlbfs VM_SHARED, hugetlbfs !VM_SHARED
> MADV_DONTNEED works but not guaranteed to fault: shmem VM_SHARED
> 
> MADV_REMOVE works: shmem VM_SHARED, hugetlbfs VM_SHARED
> MADV_REMOVE doesn't work: anon, shmem !VM_SHARED, hugetlbfs !VM_SHARED
> 
> fallocate punch hole works: hugetlbfs VM_SHARED, hugetlbfs !VM_SHARED,
> 	  	     	    shmem VM_SHARED
> fallocate punch hole doesn't work: anon, shmem !VM_SHARED
> 
> So what happens in qemu is:
> 
> anon			-> MADV_DONTNEED
> 
> shmem !VM_SHARED	-> MADV_DONTNEED (fallocate punch hole wouldn't zap
> 			   private pages, but it does on hugetlbfs)
> 
> shmem VM_SHARED		-> fallocate punch hole (MADV_REMOVE would
>       			   work too)
> 
> hugetlbfs !VM_SHARED	-> fallocate punch hole (works for hugetlbfs
> 			   but not for shmem !VM_SHARED)
> 
> hugetlbfs VM_SHARED	-> fallocate punch hole (MADV_REMOVE would work too)
> 
> This means qemu has to carry around information on the type of memory
> it got from the initial memblock setup, so at live migration time it
> can zap the memory with the right call. (NOTE: such memory is not
> generated by userfaultfd UFFDIO_COPY, but it was allocated and mapped
> and it must be zapped well before calling userfaultfd the first time).
> 
> To do this qemu uses fstatfs and finds out which kind of memory it's
> dealing with to use the right call depending on which memory.
> 
> In short it'd be better to have something like a generic MADV_REMOVE
> that guarantees a non-present fault after it succeeds, no matter what
> kind of memory is mapped in the virtual range that has to be
> zapped. The above is far from ideal from a userland developer
> prospective.

I think we will need to have a new generic MADV_REMOVE type of call
as you suggest.  Based on existing documentation for MADV_DONTNEED,
MADV_REMOVE and fallocate hole punch they each are designed not to
work on at least one of the desired memory mapping types.

> Overall fallocate punch hole covers the most cases so to keep the code
> simpler ironically MADV_REMOVE ends up being never used despite it
> provides a more friendly API than fallocate to qemu. The files are
> always mapped and the older code only dealt with virtual addresses
> (before hugetlbfs and shmem entered thee equation). Ideally qemu wants
> to call the same madvise regardles if the memory is anon shmem or
> hugetlbfs without having to carry around file descriptor, file offsets
> and superblock types.
> 
> It's also not clear why MADV_DONTNEED doesn't work for hugetlbfs
> !VM_SHARED mappings and why fallocate punch hole is also zapping
> private cow-like pages from !VM_SHARED mappings (although if it
> didn't, it would be impossible to zap those... so it's good luck it
> does).

Yes, it is more like good luck than design.  fallocate hole punch for
hugetlbfs VM_SHARED was the original use case/design.  MADV_REMOVE was
added just because it could without additional effort.

Thanks for bringing this up.  We should definitely discuss within the
scope of hugetlbfs and/or userfaultfd.

-- 
Mike Kravetz

> 
> Thanks,
> Andrea
> 
> PS. CC'ed also qemu-devel in case it may help clarify why things are
> implemented they way they are in the postcopy live migration
> hugetlbfs/shmem support and in the future patches for shmem/hugetlbfs
> share=on.