Re: [Qemu-devel] [RFC 00/13] Live memory snapshot based on userfaultfd

[Hailiang Zhang <zhang.zhanghailiang@huawei.com>]

Hi Andrea,

Any comments ?

Thanks.

On 2016/9/6 11:39, Hailiang Zhang wrote:
> Hi Andrea，
>
> I tested it with the new live memory snapshot with --enable-kvm, it doesn't work.
>
> To make things simple, I simplified the codes, only left the codes that can tested
> the write-protect capability. You can find the codes from
> https://github.com/coloft/qemu/tree/test-userfault-write-protect.
> You can reproduce the problem easily with it.
>
> Tested result as follow,
> [root@localhost qemu]# x86_64-softmmu/qemu-system-x86_64 --enable-kvm -drive file=/mnt/sdb/win7/win7.qcow2,if=none,id=drive-ide0-0-1,format=qcow2,cache=none  -device ide-hd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1  -vnc :7 -m 8192 -smp 1 -netdev tap,id=bn0 -device virtio-net-pci,id=net-pci0,netdev=bn0  --monitor stdio
> QEMU 2.6.95 monitor - type 'help' for more information
> (qemu) migrate file:/home/xxx
> qemu-system-x86_64: postcopy_ram_fault_thread: 7f07fb92a000 fault and remove write protect!
> qemu-system-x86_64: postcopy_ram_fault_thread: 7f07fb92a000 fault and remove write protect!
> qemu-system-x86_64: postcopy_ram_fault_thread: 7f07fb92a000 fault and remove write protect!
> qemu-system-x86_64: postcopy_ram_fault_thread: 7f07fb92a000 fault and remove write protect!
> qemu-system-x86_64: postcopy_ram_fault_thread: 7f07fb92a000 fault and remove write protect!
> qemu-system-x86_64: postcopy_ram_fault_thread: 7f07fb92a000 fault and remove write protect!
> qemu-system-x86_64: postcopy_ram_fault_thread: 7f07fb92a000 fault and remove write protect!
> qemu-system-x86_64: postcopy_ram_fault_thread: 7f07fb92a000 fault and remove write protect!
> qemu-system-x86_64: postcopy_ram_fault_thread: 7f07fb92a000 fault and remove write protect!
> qemu-system-x86_64: postcopy_ram_fault_thread: 7f07fb92a000 fault and remove write protect!
> qemu-system-x86_64: postcopy_ram_fault_thread: 7f07fb92a000 fault and remove write protect!
> qemu-system-x86_64: postcopy_ram_fault_thread: 7f07fb92a000 fault and remove write protect!
> qemu-system-x86_64: postcopy_ram_fault_thread: 7f07fb92a000 fault and remove write protect!
> qemu-system-x86_64: postcopy_ram_fault_thread: 7f07fb92a000 fault and remove write protect!
> qemu-system-x86_64: postcopy_ram_fault_thread: 7f07fb92a000 fault and remove write protect!
> qemu-system-x86_64: postcopy_ram_fault_thread: 7f07fb92a000 fault and remove write protect!
> qemu-system-x86_64: postcopy_ram_fault_thread: 7f07fb92a000 fault and remove write protect!
> error: kvm run failed Bad address
> EAX=00000004 EBX=00000000 ECX=83b2ac20 EDX=0000c022
> ESI=85fe33f4 EDI=0000c020 EBP=83b2abcc ESP=83b2abc0
> EIP=8bd2ff0c EFL=00010293 [--S-A-C] CPL=0 II=0 A20=1 SMM=0 HLT=0
> ES =0023 00000000 ffffffff 00c0f300 DPL=3 DS   [-WA]
> CS =0008 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA]
> SS =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
> DS =0023 00000000 ffffffff 00c0f300 DPL=3 DS   [-WA]
> FS =0030 83b2dc00 00003748 00409300 DPL=0 DS   [-WA]
> GS =0000 00000000 ffffffff 00000000
> LDT=0000 00000000 ffffffff 00000000
> TR =0028 801e2000 000020ab 00008b00 DPL=0 TSS32-busy
> GDT=     80b95000 000003ff
> IDT=     80b95400 000007ff
> CR0=8001003b CR2=030b5000 CR3=00185000 CR4=000006f8
> DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
> DR6=00000000ffff0ff0 DR7=0000000000000400
> EFER=0000000000000800
> Code=8b ff 55 8b ec 53 56 8b 75 08 57 8b 7e 34 56 e8 30 f7 ff ff <6a> 00 57 8a d8 e8 96 14 00 00 6a 04 83 c7 02 57 e8 8b 14 00 00 5f c6 46 5b 00 5e 8a c3 5b
>
> I investigated kvm and userfault codes. we use MMU Notifier to integrating KVM with the Linux
> Memory Management.
>
> Here for userfault write-protect, the function calling paths are:
> userfaultfd_ioctl
>     -> userfaultfd_writeprotect
>       -> mwriteprotect_range
>         -> change_protection (Directly call mprotect helper here)
>           -> change_protection_range
>             -> change_pud_range
>               -> change_pmd_range
>                  -> mmu_notifier_invalidate_range_start(mm, mni_start, end);
>                     -> kvm_mmu_notifier_invalidate_range_start (KVM module)
> OK, here, we remove the item from spte. (If we use EPT hardware, we remove
> the page table entry for it).
> That's why we can get fault notifying for VM.
> And It seems that we can't fix the userfault (remove the page's write-protect authority)
> by this function calling paths.
>
> Here my question is, for userfault write-protect capability, why we remove the page table
> entry instead of marking it as read-only.
> Actually, for KVM, we have a mmu notifier (kvm_mmu_notifier_change_pte) to do this,
> We can use it to remove the writable authority for KVM page table, just like KVM dirty log tracking
> does. Please see function __rmap_write_protect() in KVM.
>
> Another question, is mprotect() works normally with KVM ? (I didn't test it.), I think
> KSM and swap can work with KVM properly.
>
> Besides, there seems to be a bug for userfault write-protect.
> We use UFFDIO_COPY_MODE_DONTWAKE in userfaultfd_writeprotect, should it be
> UFFDIO_WRITEPROTECT_MODE_DONTWAKE there ?
>
> static int userfaultfd_writeprotect(struct userfaultfd_ctx *ctx,
> 				    unsigned long arg)
> {
>          ... ...
>
> 	if (!(uffdio_wp.mode & UFFDIO_COPY_MODE_DONTWAKE)) {
> 		range.start = uffdio_wp.range.start;
> 		range.len = uffdio_wp.range.len;
> 		wake_userfault(ctx, &range);
> 	}
> 	return ret;
> }
>
> Thanks.
> Hailiang
>
> On 2016/8/18 23:56, Andrea Arcangeli wrote:
>> Hello everyone,
>>
>> I've an aa.git tree uptodate on the master & userfault branch (master
>> includes other pending VM stuff, userfault branch only contains
>> userfault enhancements):
>>
>> https://git.kernel.org/cgit/linux/kernel/git/andrea/aa.git/log/?h=userfault
>>
>> I didn't have time to test KVM live memory snapshot on it yet as I'm
>> still working to improve it. Did anybody test it? However I'd be happy
>> to take any bugreports and quickly solve anything that isn't working
>> right with the shadow MMU.
>>
>> I got positive report already for another usage of the uffd WP support:
>>
>> https://medium.com/@MartinCracauer/generational-garbage-collection-write-barriers-write-protection-and-userfaultfd-2-8b0e796b8f7f
>>
>> The last few things I'm working on to finish the WP support are:
>>
>> 1) pte_swp_mkuffd_wp equivalent of pte_swp_mksoft_dirty to mark in a
>>      vma->vm_flags with VM_UFFD_WP set, which swap entries were
>>      generated while the pte was wrprotected.
>>
>> 2) to avoid all false positives the equivalent of pte_mksoft_dirty is
>>      needed too... and that requires spare software bits on the pte
>>      which are available on x86. I considered also taking over the
>>      soft_dirty bit but then you couldn't do checkpoint restore of a
>>      JIT/to-native compiler that uses uffd WP support so it wasn't
>>      ideal. Perhaps it would be ok as an incremental patch to make the
>>      two options mutually exclusive to defer the arch changes that
>>      pte_mkuffd_wp would require for later.
>>
>> 3) prevent UFFDIO_ZEROPAGE if registering WP|MISSING or trigger a
>>      cow in userfaultfd_writeprotect.
>>
>> 4) WP selftest
>>
>> In theory things should work ok already if the userland code is
>> tolerant against false positives through swap and after fork() and
>> KSM. For an usage like snapshotting false positives shouldn't be an
>> issue (it'll just run slower if you swap in the worst case), and point
>> 3) above also isn't an issue because it's going to register into uffd
>> with WP only.
>>
>> The current status includes:
>>
>> 1) WP support for anon (with false positives.. work in progress)
>>
>> 2) MISSING support for tmpfs and hugetlbfs
>>
>> 3) non cooperative support
>>
>> Thanks,
>> Andrea
>>
>> .
>>