Partial huge page backing with KVM/qemu

Partial huge page backing with KVM/qemu

[Chris Leduc]

Hi - In a KVM/qemu environment is it possible for the host to back only a portion of the guests memory with huge pages?  In some situations it may not be desirable to back the entirety of a guest's memory with huge pages (as can be done via libvirt memoryBacking option).  What would be very useful is to request huge pages in the guest, either at boot time or dynamically, and have the host back them with physical huge pages, but not back the rest of the normal page guest memory with huge pages from the host.

The equivalent in Xen is setting allowsuperpage=1 on the hypervisor boot line.

Thanks
Chris

Re: Partial huge page backing with KVM/qemu

[Gleb Natapov]

On Sat, Aug 24, 2013 at 12:32:07AM +0000, Chris Leduc wrote:
> Hi - In a KVM/qemu environment is it possible for the host to back only a portion of the guests memory with huge pages?  In some situations it may not be desirable to back the entirety of a guest's memory with huge pages (as can be done via libvirt memoryBacking option).
What are those situations?

> What would be very useful is to request huge pages in the guest, either at boot time or dynamically, and have the host back them with physical huge pages, but not back the rest of the normal page guest memory with huge pages from the host.
> 
> The equivalent in Xen is setting allowsuperpage=1 on the hypervisor boot line.
> 
As far as I can tell this disables/enables use of huge pages by XEN vm,
not something you say you want.
 
--
			Gleb.

RE: Partial huge page backing with KVM/qemu

[Chris Leduc]

> -----Original Message-----
> From: Gleb Natapov [mailto:gleb@redhat.com]
> Sent: Sunday, August 25, 2013 1:52 AM
> To: Chris Leduc
> Cc: kvm@vger.kernel.org
> Subject: Re: Partial huge page backing with KVM/qemu
> 
> On Sat, Aug 24, 2013 at 12:32:07AM +0000, Chris Leduc wrote:
> > Hi - In a KVM/qemu environment is it possible for the host to back only a
> portion of the guests memory with huge pages?  In some situations it may
> not be desirable to back the entirety of a guest's memory with huge pages
> (as can be done via libvirt memoryBacking option).
> What are those situations?
For example to limit a guest with 64GB of total memory to use 4GB of huge pages for fast lookup memory.  This takes advantage of the 4 TLB entries for 1G pages on a Sandy/Ivy Bridge processor to ensure a page walk is never necessary for this fast memory.  An example is a high performance data plane application.  The remainder of the less frequently accessed memory can be in normal pages.

> > What would be very useful is to request huge pages in the guest, either at
> boot time or dynamically, and have the host back them with physical huge
> pages, but not back the rest of the normal page guest memory with huge
> pages from the host.
> >
> > The equivalent in Xen is setting allowsuperpage=1 on the hypervisor boot
> line.
> >
> As far as I can tell this disables/enables use of huge pages by XEN vm, not
> something you say you want.

The Xen documentation is not clear on this, but in practice this flag allows the host to back up guest huge page requests with physical huge pages.  So the guest could for example add hugepages=N to its boot line and these pages would be backed in the host with corresponding physical huge pages.

From experimentation with KVM, requesting hugepages at guest boot time (without memory backing enabled) will result in guest hugepages backed by host normal pages.

Re: Partial huge page backing with KVM/qemu

[Gleb Natapov]

On Mon, Aug 26, 2013 at 02:09:57AM +0000, Chris Leduc wrote:
> 
> 
> > -----Original Message-----
> > From: Gleb Natapov [mailto:gleb@redhat.com]
> > Sent: Sunday, August 25, 2013 1:52 AM
> > To: Chris Leduc
> > Cc: kvm@vger.kernel.org
> > Subject: Re: Partial huge page backing with KVM/qemu
> > 
> > On Sat, Aug 24, 2013 at 12:32:07AM +0000, Chris Leduc wrote:
> > > Hi - In a KVM/qemu environment is it possible for the host to back only a
> > portion of the guests memory with huge pages?  In some situations it may
> > not be desirable to back the entirety of a guest's memory with huge pages
> > (as can be done via libvirt memoryBacking option).
> > What are those situations?
> For example to limit a guest with 64GB of total memory to use 4GB of huge pages for fast lookup memory.  This takes advantage of the 4 TLB entries for 1G pages on a Sandy/Ivy Bridge processor to ensure a page walk is never necessary for this fast memory.  An example is a high performance data plane application.  The remainder of the less frequently accessed memory can be in normal pages.
> 
When two level paging (EPT) is in use combined mappings are stored in
TLB, not linear mappings (see 28.3.1). I am not sure those will ever
use 1G TLB.  Not with KVM anyway since KVM does not use 1G pages for EPT
tables since the chance to get as much of contiguous memory on a running
system is close to zero.

> > > What would be very useful is to request huge pages in the guest, either at
> > boot time or dynamically, and have the host back them with physical huge
> > pages, but not back the rest of the normal page guest memory with huge
> > pages from the host.
> > >
> > > The equivalent in Xen is setting allowsuperpage=1 on the hypervisor boot
> > line.
> > >
> > As far as I can tell this disables/enables use of huge pages by XEN vm, not
> > something you say you want.
> 
> The Xen documentation is not clear on this, but in practice this flag allows the host to back up guest huge page requests with physical huge pages.  So the guest could for example add hugepages=N to its boot line and these pages would be backed in the host with corresponding physical huge pages.
Allow me to be sceptical on this :) With shadow paging sure, same is true
for KVM: if guest maps memory with huge page and memory is contiguous on
a host too KVM will create huge shadow page, but with two level paging
hypervisor has no idea how guest's page tables look. The best it can do
is to map entire guest physical memory using huge pages.

> 
> From experimentation with KVM, requesting hugepages at guest boot time (without memory backing enabled) will result in guest hugepages backed by host normal pages.
What do you mean by "requesting hugepages at guest boot time" and how
have you checked that guest hugepages backed by host normal pages? Do
you have THP enabled? Without THP you need to back guest's memory with
huge pages using "–mem-path /hugepagesfs". But again only 2MB pages are
supported.

--
			Gleb.

Re: Partial huge page backing with KVM/qemu

[Gleb Natapov]

On Mon, Aug 26, 2013 at 11:02:09AM +0300, Gleb Natapov wrote:
> On Mon, Aug 26, 2013 at 02:09:57AM +0000, Chris Leduc wrote:
> > 
> > 
> > > -----Original Message-----
> > > From: Gleb Natapov [mailto:gleb@redhat.com]
> > > Sent: Sunday, August 25, 2013 1:52 AM
> > > To: Chris Leduc
> > > Cc: kvm@vger.kernel.org
> > > Subject: Re: Partial huge page backing with KVM/qemu
> > > 
> > > On Sat, Aug 24, 2013 at 12:32:07AM +0000, Chris Leduc wrote:
> > > > Hi - In a KVM/qemu environment is it possible for the host to back only a
> > > portion of the guests memory with huge pages?  In some situations it may
> > > not be desirable to back the entirety of a guest's memory with huge pages
> > > (as can be done via libvirt memoryBacking option).
> > > What are those situations?
> > For example to limit a guest with 64GB of total memory to use 4GB of huge pages for fast lookup memory.  This takes advantage of the 4 TLB entries for 1G pages on a Sandy/Ivy Bridge processor to ensure a page walk is never necessary for this fast memory.  An example is a high performance data plane application.  The remainder of the less frequently accessed memory can be in normal pages.
> > 
> When two level paging (EPT) is in use combined mappings are stored in
> TLB, not linear mappings (see 28.3.1). I am not sure those will ever
> use 1G TLB.  Not with KVM anyway since KVM does not use 1G pages for EPT
> tables since the chance to get as much of contiguous memory on a running
> system is close to zero.
> 
> > > > What would be very useful is to request huge pages in the guest, either at
> > > boot time or dynamically, and have the host back them with physical huge
> > > pages, but not back the rest of the normal page guest memory with huge
> > > pages from the host.
> > > >
> > > > The equivalent in Xen is setting allowsuperpage=1 on the hypervisor boot
> > > line.
> > > >
> > > As far as I can tell this disables/enables use of huge pages by XEN vm, not
> > > something you say you want.
> > 
> > The Xen documentation is not clear on this, but in practice this flag allows the host to back up guest huge page requests with physical huge pages.  So the guest could for example add hugepages=N to its boot line and these pages would be backed in the host with corresponding physical huge pages.
> Allow me to be sceptical on this :) With shadow paging sure, same is true
> for KVM: if guest maps memory with huge page and memory is contiguous on
> a host too KVM will create huge shadow page, but with two level paging
> hypervisor has no idea how guest's page tables look. The best it can do
> is to map entire guest physical memory using huge pages.
> 
> > 
> > From experimentation with KVM, requesting hugepages at guest boot time (without memory backing enabled) will result in guest hugepages backed by host normal pages.
> What do you mean by "requesting hugepages at guest boot time" and how
> have you checked that guest hugepages backed by host normal pages? Do
> you have THP enabled? Without THP you need to back guest's memory with
> huge pages using "–mem-path /hugepagesfs". But again only 2MB pages are
> supported.
> 
Just wanted to correct myself here. KVM does support 1G pages with
"–mem-path /hugepagesfs", so to backup only part of guests memory with
1G pages QEMU need to be changed to allocate part of guest's memory from
hugetlbfs and part as regular anonymous memory. Guest needs to know
what part of its physical memory is allocated from hugetlbfs and map
it using 1G pages too. I do not see anything in Intel SDM that says if
such combined (EPT + guest's PT) mapping will be stored in 1G tlb
entries, but it probably should. Of course tlb is shared between guests,
so if you have two guests with 4 1G pages each tlb will not be able to
hold translation for both guests.

--
			Gleb.