Postcopy with different page sizes on source and destination

[Florian Schmidt <flosch@nutanix.com>]

Hi,

with precopy live migration, change in page size on source and 
destination is possible: using hugetlbfs memory backing for the VM on 
the source and anonymous memory on the destination, and vice versa. For 
postcopy migration, this is not allowed, and in fact checked during the 
advise stage.

Is there any fundamental limitation in the design that prevents this, or 
is it more that this is an additional complication that nobody has 
implemented so far because there was no strong need for it?

It seems to me like this should be possible, and the comment in 
loadvm_postcopy_handle_advise() (migration/savevm.c:1681) also seems to 
suggest that; so I'll add a (very rough) first idea. Please tell me if 
I'm missing something important. The "background" copy is similar to 
precopy, so the main difference is the userfaultfd page fault handling 
on the destination, and requesting the correct memory from the source.

1. If the source has hugepages and the destination doesn't, then a page 
fault would lead the destination to ask "I need these 4k of memory from 
you to fill my page and handle the page fault". The source could then 
answer "here you are, and here are these other 511 4k pages around it 
(which form my 2M page; similarly for 1G pages), please deal with them 
now". That way, even "release-ram" would still work on a (huge)page 
granularity.

2. If the destination has hugepages and the source doesn't, then the 
above works similarly: now the destination, on a page fault, asks for a 
larger memory area that corresponds to 512 (or more) pages on the 
source. The only issue I could see here is during the initial phase, 
when postcopy is switched on, to make sure that the source doesn't 
release RAM that it has copied and thinks is clean, but it part of a 
hugepage on the other side. That seems easy enough to solve though? And 
indeed is probably already implemented for precopy migration to work 
with different page sizes on source and destination and could be adapted 
here.

Cheers,
Florian