Can't boot guest with more than 3585MB when using large pages

Can't boot guest with more than 3585MB when using large pages

[Alex Williamson]

On a 2.6.29, x86_64 host/guest, what's special about specifying a guest
size of -m 3586 when using -mem-path backed by hugetlbfs?  3585 works,
3586 hangs here:

...
PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
Placing 64MB software IO TLB between ffff880020000000 - ffff880024000000
software IO TLB at phys 0x20000000 - 0x24000000
Memory: 3504832k/4196352k available (2926k kernel code, 524740k absent, 166780k reserved, 1260k data, 496k init)

I can back -mem-path by tmpfs or disk and it works fine.  Also works
with no -mem-path, but it would obviously be nice to benefit from large
pages on big guests.  The system has plenty of huge pages to back the
request, and booting with -mem-prealloc makes no difference.  Tested on
latest git as of today.  Thanks,

Alex

Re: Can't boot guest with more than 3585MB when using large pages

[Ryan Harper]

* Alex Williamson <alex.williamson@hp.com> [2009-03-24 16:07]:
> 
> On a 2.6.29, x86_64 host/guest, what's special about specifying a guest
> size of -m 3586 when using -mem-path backed by hugetlbfs?  3585 works,
> 3586 hangs here:
> 
> ...
> PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
> Placing 64MB software IO TLB between ffff880020000000 - ffff880024000000
> software IO TLB at phys 0x20000000 - 0x24000000
> Memory: 3504832k/4196352k available (2926k kernel code, 524740k absent, 166780k reserved, 1260k data, 496k init)
> 
> I can back -mem-path by tmpfs or disk and it works fine.  Also works
> with no -mem-path, but it would obviously be nice to benefit from large
> pages on big guests.  The system has plenty of huge pages to back the
> request, and booting with -mem-prealloc makes no difference.  Tested on
> latest git as of today.  Thanks,

I've seen this as well, haven't had a chance to dig into the issue yet
either.  Certainly can test patches if anyone has an idea of what's
wrong here.

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
ryanh@us.ibm.com

Re: Can't boot guest with more than 3585MB when using large pages

[Marcelo Tosatti]

On Tue, Mar 24, 2009 at 04:57:46PM -0500, Ryan Harper wrote:
> * Alex Williamson <alex.williamson@hp.com> [2009-03-24 16:07]:
> > 
> > On a 2.6.29, x86_64 host/guest, what's special about specifying a guest
> > size of -m 3586 when using -mem-path backed by hugetlbfs?  3585 works,
> > 3586 hangs here:
> > 
> > ...
> > PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
> > Placing 64MB software IO TLB between ffff880020000000 - ffff880024000000
> > software IO TLB at phys 0x20000000 - 0x24000000
> > Memory: 3504832k/4196352k available (2926k kernel code, 524740k absent, 166780k reserved, 1260k data, 496k init)
> > 
> > I can back -mem-path by tmpfs or disk and it works fine.  Also works
> > with no -mem-path, but it would obviously be nice to benefit from large
> > pages on big guests.  The system has plenty of huge pages to back the
> > request, and booting with -mem-prealloc makes no difference.  Tested on
> > latest git as of today.  Thanks,
> 
> I've seen this as well, haven't had a chance to dig into the issue yet
> either.  Certainly can test patches if anyone has an idea of what's
> wrong here.

Can you strace and see if the mmap on hugetlbfs is correctly sized?

Re: Can't boot guest with more than 3585MB when using large pages

[Alex Williamson]

On Wed, 2009-03-25 at 13:10 -0300, Marcelo Tosatti wrote:
> On Tue, Mar 24, 2009 at 04:57:46PM -0500, Ryan Harper wrote:
> > * Alex Williamson <alex.williamson@hp.com> [2009-03-24 16:07]:
> > > 
> > > On a 2.6.29, x86_64 host/guest, what's special about specifying a guest
> > > size of -m 3586 when using -mem-path backed by hugetlbfs?  3585 works,
> > > 3586 hangs here:
> > > 
> > > ...
> > > PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
> > > Placing 64MB software IO TLB between ffff880020000000 - ffff880024000000
> > > software IO TLB at phys 0x20000000 - 0x24000000
> > > Memory: 3504832k/4196352k available (2926k kernel code, 524740k absent, 166780k reserved, 1260k data, 496k init)
> > 
> > I've seen this as well, haven't had a chance to dig into the issue yet
> > either.  Certainly can test patches if anyone has an idea of what's
> > wrong here.
> 
> Can you strace and see if the mmap on hugetlbfs is correctly sized?

Seems reasonable with some 2MB rounding.

Failing case, -m 3586:

open("/hugepages//kvm.5fuuH5", O_RDWR|O_CREAT|O_EXCL, 0600) = 9
unlink("/hugepages//kvm.5fuuH5")        = 0
ftruncate(9, 3783262208)                = 0
mmap(NULL, 3783262208, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_POPULATE, 9, 0) = 0x7f37a5e00000

Working case, -m 3585:

open("/hugepages//kvm.Mv6Zgd", O_RDWR|O_CREAT|O_EXCL, 0600) = 9
unlink("/hugepages//kvm.Mv6Zgd")        = 0
ftruncate(9, 3781165056)                = 0
mmap(NULL, 3781165056, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_POPULATE, 9, 0) = 0x7fd44b800000

Working case using disk backing: -mem-path /tmp -mem-prealloc -m 3586:

open("/tmp/kvm.nPlxl1", O_RDWR|O_CREAT|O_EXCL, 0600) = 9
unlink("/tmp/kvm.nPlxl1")               = 0
ftruncate(9, 3783262208)                = 0
mmap(NULL, 3783262208, PROT_READ|PROT_WRITE, MAP_PRIVATE, 9, 0) = 0x7f432e055000

Re: Can't boot guest with more than 3585MB when using large pages

[Marcelo Tosatti]

On Tue, Mar 24, 2009 at 04:57:46PM -0500, Ryan Harper wrote:
> * Alex Williamson <alex.williamson@hp.com> [2009-03-24 16:07]:
> > 
> > On a 2.6.29, x86_64 host/guest, what's special about specifying a guest
> > size of -m 3586 when using -mem-path backed by hugetlbfs?  3585 works,
> > 3586 hangs here:
> > 
> > ...
> > PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
> > Placing 64MB software IO TLB between ffff880020000000 - ffff880024000000
> > software IO TLB at phys 0x20000000 - 0x24000000
> > Memory: 3504832k/4196352k available (2926k kernel code, 524740k absent, 166780k reserved, 1260k data, 496k init)
> > 
> > I can back -mem-path by tmpfs or disk and it works fine.  Also works
> > with no -mem-path, but it would obviously be nice to benefit from large
> > pages on big guests.  The system has plenty of huge pages to back the
> > request, and booting with -mem-prealloc makes no difference.  Tested on
> > latest git as of today.  Thanks,
> 
> I've seen this as well, haven't had a chance to dig into the issue yet
> either.  Certainly can test patches if anyone has an idea of what's
> wrong here.

Can you please try the following

------

qemu: kvm: fixup 4GB+ memslot large page alignment

Need to align the 4GB+ memslot after we know its address, not before.

Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>

diff --git a/qemu/hw/pc.c b/qemu/hw/pc.c
index d4a4320..cc84772 100644
--- a/qemu/hw/pc.c
+++ b/qemu/hw/pc.c
@@ -866,6 +866,7 @@ static void pc_init1(ram_addr_t ram_size, int vga_ram_size,
 
     /* above 4giga memory allocation */
     if (above_4g_mem_size > 0) {
+        ram_addr = qemu_ram_alloc(above_4g_mem_size);
         if (hpagesize) {
             if (ram_addr & (hpagesize-1)) {
                 unsigned long aligned_addr;
@@ -874,7 +875,6 @@ static void pc_init1(ram_addr_t ram_size, int vga_ram_size,
                 ram_addr = aligned_addr;
             }
         }
-        ram_addr = qemu_ram_alloc(above_4g_mem_size);
         cpu_register_physical_memory(0x100000000ULL,
                                      above_4g_mem_size,
                                      ram_addr);

Re: Can't boot guest with more than 3585MB when using large pages

[Alex Williamson]

On Fri, 2009-04-03 at 20:28 -0300, Marcelo Tosatti wrote:
> 
> Can you please try the following

Thanks Marcelo, this seems to fix it.  I tested up to a 30G guest with
large pages.

Alex

> ------
> 
> qemu: kvm: fixup 4GB+ memslot large page alignment
> 
> Need to align the 4GB+ memslot after we know its address, not before.
> 
> Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>

Tested-by: Alex Williamson <alex.williamson@hp.com>

> diff --git a/qemu/hw/pc.c b/qemu/hw/pc.c
> index d4a4320..cc84772 100644
> --- a/qemu/hw/pc.c
> +++ b/qemu/hw/pc.c
> @@ -866,6 +866,7 @@ static void pc_init1(ram_addr_t ram_size, int vga_ram_size,
>  
>      /* above 4giga memory allocation */
>      if (above_4g_mem_size > 0) {
> +        ram_addr = qemu_ram_alloc(above_4g_mem_size);
>          if (hpagesize) {
>              if (ram_addr & (hpagesize-1)) {
>                  unsigned long aligned_addr;
> @@ -874,7 +875,6 @@ static void pc_init1(ram_addr_t ram_size, int vga_ram_size,
>                  ram_addr = aligned_addr;
>              }
>          }
> -        ram_addr = qemu_ram_alloc(above_4g_mem_size);
>          cpu_register_physical_memory(0x100000000ULL,
>                                       above_4g_mem_size,
>                                       ram_addr);
> 

Re: Can't boot guest with more than 3585MB when using large pages

[Avi Kivity]

Alex Williamson wrote:
> On Fri, 2009-04-03 at 20:28 -0300, Marcelo Tosatti wrote:
>   
>> Can you please try the following
>>     
>
> Thanks Marcelo, this seems to fix it.  I tested up to a 30G guest with
> large pages.
>   

I've applied the patch, thanks.  I keep thinking we need to do 
additional rounding when we allocate the file.

-- 
error compiling committee.c: too many arguments to function