How KVM hypervisor allocates physical pages to the VM.

How KVM hypervisor allocates physical pages to the VM.

[Steven]

Dear KVM Developers:
I have some questions about how KVM hypervisor requests and allocate
physical pages to the VM. I am using kernel version 3.2.14.

I run a microbenchmark in the VM, which declares an array with certain
size and then assigns some value to all the elements in the array,
which causes page fault captured by the hypervisor and allocates the
machine physical pages.

Depending on the array size, I got two different trace results.

(1) When array size == 10MB or 20 MB, the trace file has the page
allocation events for the qemu-kvm process as follows

      qemu-kvm-14402 [004] 1912063.581683: kmalloc_node:
call_site=ffffffff813df54a ptr=ffff880779a42800 bytes_req=576
bytes_alloc=1024 gfp_flags=GFP_KERNEL|GFP_REPEAT node=-1

        qemu-kvm-14402 [004] 1912063.581701: kmem_cache_alloc_node:
call_site=ffffffff813e302c ptr=ffff8800229ff500 bytes_req=256
bytes_alloc=256 gfp_flags=GFP_KERNEL|GFP_REPEAT node=-1

        qemu-kvm-14402 [004] 1912063.581701: kmalloc_node:
call_site=ffffffff813df54a ptr=ffff880779a42800 bytes_req=576
bytes_alloc=1024 gfp_flags=GFP_KERNEL|GFP_REPEAT node=-1

        qemu-kvm-14402 [004] 1912063.581710: kmem_cache_alloc_node:
call_site=ffffffff813e302c ptr=ffff8800229ff800 bytes_req=256
bytes_alloc=256 gfp_flags=GFP_KERNEL|GFP_REPEAT node=-1
        qemu-kvm-14402 [004] 1912063.581710: kmalloc_node:
call_site=ffffffff813df54a ptr=ffff880779a47800 bytes_req=640
bytes_alloc=1024 gfp_flags=GFP_KERNEL|GFP_REPEAT node=-1
        qemu-kvm-14402 [004] 1912063.581728: kmem_cache_alloc_node:
call_site=ffffffff813e302c ptr=ffff8800229ffe00 bytes_req=256
bytes_alloc=256 gfp_flags=GFP_KERNEL|GFP_REPEAT
     ... ...

(2) When array size == 40MB, the trace file has the page allocation events as

qemu-kvm-14450 [005] 1911006.440538: mm_page_alloc:
page=ffffea0002570f40 pfn=613437 order=0 migratetype=2
gfp_flags=GFP_HIGHUSER_MOVABLE|GFP_ZERO

        qemu-kvm-14450 [005] 1911006.440542: mm_page_alloc:
page=ffffea0002577480 pfn=613842 order=0 migratetype=2
gfp_flags=GFP_HIGHUSER_MOVABLE|GFP_ZERO

        qemu-kvm-14450 [005] 1911006.440545: mm_page_alloc:
page=ffffea000070a3c0 pfn=115343 order=0 migratetype=2
gfp_flags=GFP_HIGHUSER_MOVABLE|GFP_ZERO

        qemu-kvm-14450 [005] 1911006.440549: mm_page_alloc:
page=ffffea0001a9a500 pfn=435860 order=0 migratetype=2
gfp_flags=GFP_HIGHUSER_MOVABLE|GFP_ZERO

        qemu-kvm-14450 [005] 1911006.440552: mm_page_alloc:
page=ffffea00016f7e80 pfn=376314 order=0 migratetype=2
gfp_flags=GFP_HIGHUSER_MOVABLE|GFP_ZERO

        qemu-kvm-14450 [005] 1911006.440556: mm_page_alloc:
page=ffffea0001a9d700 pfn=436060 order=0 migratetype=2
gfp_flags=GFP_HIGHUSER_MOVABLE|GFP_ZERO

        qemu-kvm-14450 [005] 1911006.440559: mm_page_alloc:
page=ffffea0002576880 pfn=613794 order=0 migratetype=2
gfp_flags=GFP_HIGHUSER_MOVABLE|GFP_ZERO

        qemu-kvm-14450 [005] 1911006.440563: mm_page_alloc:
page=ffffea00016f7b40 pfn=376301 order=0 migratetype=2
gfp_flags=GFP_HIGHUSER_MOVABLE|GFP_ZERO

        qemu-kvm-14450 [005] 1911006.440569: mm_page_alloc:
page=ffffea00016f7f80 pfn=376318 order=0 migratetype=2
gfp_flags=GFP_HIGHUSER_MOVABLE|GFP_ZER

....
----------------------------------------------------------------------------------------------

When size = 10MB and 20MB, it looks like that KVM use
kmem_cache_alloc_node and kmalloc_node to allocate physical pages.
However, when size = 40MB, KVM hypervisor uses mm_page_alloc to
allocator physical pages. The former is based on the slab allocator,
while the latter is directly from the buddy allocator.

So what is the heuristic used by the KVM to determine when to use the
slab allocator or directly from the buddy allocator? Or is there
anything wrong with my trace file? Thanks in advance.


- Hui

Re: How KVM hypervisor allocates physical pages to the VM.

[Paolo Bonzini]

Il 17/09/2014 05:56, Steven ha scritto:
> When size = 10MB and 20MB, it looks like that KVM use
> kmem_cache_alloc_node and kmalloc_node to allocate physical pages.
> However, when size = 40MB, KVM hypervisor uses mm_page_alloc to
> allocator physical pages. The former is based on the slab allocator,
> while the latter is directly from the buddy allocator.
> 
> So what is the heuristic used by the KVM to determine when to use the
> slab allocator or directly from the buddy allocator? Or is there
> anything wrong with my trace file? Thanks in advance.

I think you're simply tracking different things.  You should find the
symbols associated to the call sites.

KVM doesn't do anything special to allocate physical pages.  It simply
calls get_user_pages.

Paolo

Re: How KVM hypervisor allocates physical pages to the VM.

[Steven]

On Thu, Sep 18, 2014 at 6:00 AM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> Il 18/09/2014 01:11, Steven ha scritto:
>> I agree with you that the memory allocated from the
>> kmem_cache_alloc_node and kmalloc_node should be not returned to the
>> user space process in the VM.
>>
>> Can I understand in this way " for small size memory allocation array
>> or the VM has already have some pages, the hypervisor uses kmalloc or
>> kmem_cache_alloc for some bookkeeping purpose to provide VM the free
>> memory, instead of calling mm_page_alloc"? Is this correct? Thanks.
>
> No.  The hypervisor's uses of kmalloc or kmem_cache_alloc are unrelated
> to providing the VM with memory.
>
> For example, the hypervisor could be allocating its own private data
> structure to track the guest's memory.  But pages will always be
> obtained via get_user_pages and the buddy allocator.

Thanks for your reply. Could you give me some guidance about how to
trace the page allocation for the user space application in the VM?

Say, I have a simple program in the VM, that allocate and initialize
the array like
      array [10240]; // 10 page integer array.
When the program is running, it should causes page fault to the
hypervisor. How can I track the guest virtual address and physical
page address during the page fault handling? Thanks.

- Steven




>
> Paolo

Re: How KVM hypervisor allocates physical pages to the VM.

[Steven]

Hi, Paolo,
As you said the memory pages may have already be touched by the guest,
although it still cause page fault from the VM.

So my question is will KVM pre-allocate some physical pages (by
calling get_user_page_fast) to the guest?
If so, when a user-space program in the guest requests for pages, how
does KVM decide which pre-allocated physical pages are to be
allocated?

Because in this microbenchmark, for each virtual page fault in the
guest, I can trace there is a corresponding kvm_page_fault event, but
not mm_alloc_page event (which is the function of allocating page by
the buddy allocator).

Thanks in advance.

- Steven

On Wed, Sep 17, 2014 at 3:22 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> Il 17/09/2014 19:19, Steven ha scritto:
>> I think get_user_pages or get_user_pages finally will call
>> mm_page_alloc, right?.
>> However, for the small size array, I observer many
>> kmem_cache_alloc_node and kmalloc_node physical page allocation, but
>> very few mm_page_alloc trace. So will the host kernel assign the
>> memory to the VM from slab allocator? Thanks.
>
> I don't think so.  Likely the memory had already been touched by the guest.
>
> Paolo

Re: How KVM hypervisor allocates physical pages to the VM.

[Steven]

Hi, KVM developer:
A correction to my previous post. I found that NOT all page fault in
the guest could cause kvm_page_fault event in the hypervisor.

For example, my array access microbencharmk touches 1024 pages (as an
integer array). In the guest VM, I can trace 1024 page fault events.
However, in the hypervisor host, I can only trace about 50
kvm_page_fault events.

Could anyone explain why the those page faults in the guest are not
exposed to the hypervisor? Thanks.

The KVM module has disabled ept and host kernel is 3.2.14.

- Hui

On Sun, Oct 19, 2014 at 2:19 AM, Steven <wangwangkang@gmail.com> wrote:
> Hi, Paolo,
> As you said the memory pages may have already be touched by the guest,
> although it still cause page fault from the VM.
>
> So my question is will KVM pre-allocate some physical pages (by
> calling get_user_page_fast) to the guest?
> If so, when a user-space program in the guest requests for pages, how
> does KVM decide which pre-allocated physical pages are to be
> allocated?
>
> Because in this microbenchmark, for each virtual page fault in the
> guest, I can trace there is a corresponding kvm_page_fault event, but
> not mm_alloc_page event (which is the function of allocating page by
> the buddy allocator).
>
> Thanks in advance.
>
> - Steven
>
> On Wed, Sep 17, 2014 at 3:22 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>> Il 17/09/2014 19:19, Steven ha scritto:
>>> I think get_user_pages or get_user_pages finally will call
>>> mm_page_alloc, right?.
>>> However, for the small size array, I observer many
>>> kmem_cache_alloc_node and kmalloc_node physical page allocation, but
>>> very few mm_page_alloc trace. So will the host kernel assign the
>>> memory to the VM from slab allocator? Thanks.
>>
>> I don't think so.  Likely the memory had already been touched by the guest.
>>
>> Paolo