Re: [Qemu-devel] RFC migration of zero pages

Re: [Qemu-devel] RFC migration of zero pages

[Avi Kivity]

[-- Attachment #1: Type: text/plain, Size: 1900 bytes --]

On Jan 31, 2013 11:14 PM, "Michael S. Tsirkin" <mst@redhat.com> wrote:
>
> On Thu, Jan 31, 2013 at 05:20:02PM +0200, Avi Kivity wrote:
> > On Thu, Jan 31, 2013 at 4:42 PM, Gleb Natapov <gleb@redhat.com> wrote:
> > > On Thu, Jan 31, 2013 at 04:36:25PM +0200, Avi Kivity wrote:
> > >> On Thu, Jan 31, 2013 at 4:30 PM, Gleb Natapov <gleb@redhat.com>
wrote:
> > >> >> >
> > >> >> > Avi/Michael do you remember why mincore can't be used to check
if a guest
> > >> >> page is unmapped?
> > >> >>
> > >> >> A page may be not in core, but also nonzero (for example swap).
> > >> > Yes, mincore() should not be used as zero page check, but it can
be used
> > >> > as an indicator that page can be dealt with in dead stage of
migration
> > >> > since it is likely zero and will not take much time to send.
> > >>
> > >> Or it's swapped, in which case we know nothing about it.
> > >>
> > >> > It is
> > > Of course, that is why I said "likely' :)
> > >
> > >> > possible to call madvise(MADV_WILLNEED) on them meanwhile to
pre-load
> > >> > swap without faulting on each page individually.
> > >>
> > >> During migration you're faulting like mad anyway.
> > > That's guest faulting on dirty bit logging, as far as I understand
Orit
> > > says that in addition to that she sees a lot of faults generated by
> > > migration reading unmapped guest memory. She wants to eliminate at
least
> > > those.
> >
> > For that, we once talked about extending mincore() to return info
> > whether a page is known zero (anonymous memory without a backing page,
> > or backing page == zero page).
>
> Shoudn't be hard to expose in /proc, no?

Unfortunately.

>
> > But I doubt it's worth it, yes we're
> > faulting a lot, but it's still a lot cheaper than actually sending a
> > page, so we're still ahead of a non-idle guest.
>
> It's not just fault, reading in the page is bad for the cache.
>

There's just one zero page.

[-- Attachment #2: Type: text/html, Size: 2717 bytes --]

Re: [Qemu-devel] RFC migration of zero pages

[Paolo Bonzini]

Il 31/01/2013 10:48, Gleb Natapov ha scritto:
>> Also I notice that the bottle neck in migrating unmapped pages is the detection of those pages
>> because we map the pages in order to check them, for a large guest this is very expensive as mapping a page
>> results in a page fault in the host.
>> So what will be very helpful is actually locating those pages without mapping them
>> which looks very complicated.
>>
> What is wrong with mincore()?

As Avi said, it returns false if the memory is in swap.

Note that while we do take a page fault, the recent kernel patch to
introduce a huge zero page might have reduced the cost in terms of both
clear_page and cache misses.

Paolo

Re: [Qemu-devel] RFC migration of zero pages

[Paolo Bonzini]

Il 31/01/2013 09:33, Orit Wasserman ha scritto:
>> > If my above assumption that the guest reads unmapped memory as zeroes is right, this mapping
>> > is not necessary in the case of a zero dup page.
>> > 
>> > We just have to make sure that we are still in the very first round when deciding not to sent
>> > a zero page, because otherwise it could be a page that has become zero during migration and
>> > this of course has to be transferred.
> OK, so if we won't send the pages than it won't be allocate in the dst and it can improve both 
> memory usage and reduce cpu consumption on it.
> That can be good for over commit scenario.

We don't allocate zero pages in the destination:

#ifndef _WIN32
            if (ch == 0 &&
                (!kvm_enabled() || kvm_has_sync_mmu()) &&
                getpagesize() <= TARGET_PAGE_SIZE) {
                qemu_madvise(host, TARGET_PAGE_SIZE, QEMU_MADV_DONTNEED);
            }
#endif

Paolo

Re: [Qemu-devel] RFC migration of zero pages

[Peter Lieven]

Am 01.02.2013 18:03, schrieb Paolo Bonzini:
> Il 31/01/2013 09:33, Orit Wasserman ha scritto:
>>>> If my above assumption that the guest reads unmapped memory as zeroes is right, this mapping
>>>> is not necessary in the case of a zero dup page.
>>>>
>>>> We just have to make sure that we are still in the very first round when deciding not to sent
>>>> a zero page, because otherwise it could be a page that has become zero during migration and
>>>> this of course has to be transferred.
>> OK, so if we won't send the pages than it won't be allocate in the dst and it can improve both 
>> memory usage and reduce cpu consumption on it.
>> That can be good for over commit scenario.
> 
> We don't allocate zero pages in the destination:
> 
> #ifndef _WIN32
>             if (ch == 0 &&
>                 (!kvm_enabled() || kvm_has_sync_mmu()) &&
>                 getpagesize() <= TARGET_PAGE_SIZE) {
>                 qemu_madvise(host, TARGET_PAGE_SIZE, QEMU_MADV_DONTNEED);
>             }
> #endif

Sorry I was not aware of this. So the only possible benefit left would be to not sent zero pages in
the first round at all which would avoid the 9 bytes (header + ch) per page + memset and madvise on the target.

Peter


> 
> Paolo
>