Hyp compat_memory_op() and 256 GB PV

Hyp compat_memory_op() and 256 GB PV

[Mukesh Rathor]

Hi,

Moving on to 256 GB guest, the hyp is failing the XENMEM_populate_physmap
hcall in compat_memory_op(). The problem is size too large for continuation
encoding:

     /* Is size too large for us to encode a continuation? */
     if ( cmp.rsrv.nr_extents > (UINT_MAX >> MEMOP_EXTENT_SHIFT))
         return start_extent;

      for 256 GB : nr_extents == 0x4000000

  Currently at a loss on this one!

Thanks,
Mukesh

Re: Hyp compat_memory_op() and 256 GB PV

[Keir Fraser]

On 18/02/2009 03:39, "Mukesh Rathor" <mukesh.rathor@oracle.com> wrote:

> Moving on to 256 GB guest, the hyp is failing the XENMEM_populate_physmap
> hcall in compat_memory_op(). The problem is size too large for continuation
> encoding:
> 
>      /* Is size too large for us to encode a continuation? */
>      if ( cmp.rsrv.nr_extents > (UINT_MAX >> MEMOP_EXTENT_SHIFT))
>          return start_extent;
> 
>       for 256 GB : nr_extents == 0x4000000
> 
>   Currently at a loss on this one!

Well, who's making the compat call? Not the guest itself presumably since it
is 64-bit? So it's probably dom0? But I would think that dom0 would only do
large amounts of allocation for the new domU in xc_hvm_build.c, and that is
careful to allocate memory in batches of 8MB at a time.

Basically you need to track down the call site of the failed
compat_memory_op().

 -- Keir

Re: Hyp compat_memory_op() and 256 GB PV

[Mukesh Rathor]

Keir Fraser wrote:
 > On 18/02/2009 03:39, "Mukesh Rathor" <mukesh.rathor@oracle.com> wrote:
 >
 >> Moving on to 256 GB guest, the hyp is failing the XENMEM_populate_physmap
 >> hcall in compat_memory_op(). The problem is size too large for continuation
 >> encoding:
 >>
 >>      /* Is size too large for us to encode a continuation? */
 >>      if ( cmp.rsrv.nr_extents > (UINT_MAX >> MEMOP_EXTENT_SHIFT))
 >>          return start_extent;
 >>
 >>       for 256 GB : nr_extents == 0x4000000
 >>
 >>   Currently at a loss on this one!
 >
 > Well, who's making the compat call? Not the guest itself presumably since it
 > is 64-bit? So it's probably dom0? But I would think that dom0 would only do
 > large amounts of allocation for the new domU in xc_hvm_build.c, and that is
 > careful to allocate memory in batches of 8MB at a time.
 >
 > Basically you need to track down the call site of the failed
 > compat_memory_op().
 >
 >  -- Keir
 >

Hi Keir,

I see this in domain-builder-ng.log:

elf_xen_addr_calc_check: addresses:
     virt_base        = 0xffffffff80000000
     elf_paddr_offset = 0xffffffff80000000
     virt_offset      = 0x0
     virt_kstart      = 0xffffffff80200000
     virt_kend        = 0xffffffff807014e4
     virt_entry       = 0xffffffff80200000
xc_dom_parse_elf_kernel: xen-3.0-x86_64: 0xffffffff80200000 ->
0xffffffff807014e4
xc_dom_mem_init: mem 262144 MB, pages 0x4000000 pages, 4k each
xc_dom_mem_init: 0x4000000 pages
xc_dom_boot_mem_init: called
x86_compat: guest xen-3.0-x86_64, address size 64
xc_dom_malloc            : 256 MB
xc_dom_boot.c:143: panic: xc_dom_boot_mem_init: can't allocate low
memory for domain


I set a breakpoint in compat_memory_op() and stepping thru it, noticed
it was bailing out at the if statement, and nr_extents was 0x4000000.

This is a PV domain BTW, I guess xc_hvm_build.c wont' be involved. I'll
instrument libxc and debug more... meanwhile I thought I'd post this.

Thanks,
Mukesh

Re: Hyp compat_memory_op() and 256 GB PV

[Keir Fraser]

On 18/02/2009 19:52, "Mukesh Rathor" <mukesh.rathor@oracle.com> wrote:

> This is a PV domain BTW, I guess xc_hvm_build.c wont' be involved. I'll
> instrument libxc and debug more... meanwhile I thought I'd post this.

Yes, my mistake, and this points us straight at the problem.

You see the xc_domain_memory_populate_physmap() call at the end of
libxc/xc_dom_x86.c:arch_setup_meminit()? That needs to be put in a for loop,
allocating no more than, say, a million pages at a time.

The patch should be very simple, so I'll leave the rest to you.

 -- Keir

Re: Hyp compat_memory_op() and 256 GB PV

[Mukesh Rathor]

[-- Attachment #1: Type: text/plain, Size: 1514 bytes --]

Hi Keir,

Attached please find my patch to allocate 1M pages at a time. With this patch 
on 3.3.1 and unstable, I'm able to bring up the hypervisor with 512GB memory 
and start a PV 64 guest (with my pud entry fix).

The only issue is performance related. As somewhat expected, after guest 
shutdown it takes a while for pages to be freed and guest to disappear. Until 
this time, system is fine, but xm will hang.

In my kdb debugger, I see cpu's in :
[0]ffff828c80110f52: page_scrub_softirq+22          test %al, %al
[1]ffff828c80110f52: page_scrub_softirq+22          test %al, %al
[2]ffff828c80110f52: page_scrub_softirq+22          test %al, %al
...

page_scrub_softirq():
     /* free_heap_pages() does not parallelise well. Serialise this function. */
     if ( !spin_trylock(&serialise_lock) )

I guess a future work item to look into parallelizing this some day....

Thanks a lot,
Mukesh



Keir Fraser wrote:
 > On 18/02/2009 19:52, "Mukesh Rathor" <mukesh.rathor@oracle.com> wrote:
 >
 >> This is a PV domain BTW, I guess xc_hvm_build.c wont' be involved. I'll
 >> instrument libxc and debug more... meanwhile I thought I'd post this.
 >
 > Yes, my mistake, and this points us straight at the problem.
 >
 > You see the xc_domain_memory_populate_physmap() call at the end of
 > libxc/xc_dom_x86.c:arch_setup_meminit()? That needs to be put in a for loop,
 > allocating no more than, say, a million pages at a time.
 >
 > The patch should be very simple, so I'll leave the rest to you.
 >
 >  -- Keir
 >

[-- Attachment #2: libxc.patch --]
[-- Type: text/x-patch, Size: 1627 bytes --]

Signed-off-by: Mukesh Rathor <mukesh.rathor@oracle.com>
---
diff -r 746955812d23 tools/libxc/xc_dom_x86.c
--- a/tools/libxc/xc_dom_x86.c	Wed Mar 11 17:47:56 2009 -0700
+++ b/tools/libxc/xc_dom_x86.c	Wed Mar 18 14:31:07 2009 -0700
@@ -25,6 +25,7 @@
 #include "xenctrl.h"
 
 /* ------------------------------------------------------------------------ */
+#define MEM_CHUNKSZ                  1024*1024
 
 #define bits_to_mask(bits)       (((xen_vaddr_t)1 << (bits))-1)
 #define round_down(addr, mask)   ((addr) & ~(mask))
@@ -694,7 +695,7 @@ int arch_setup_meminit(struct xc_dom_ima
 int arch_setup_meminit(struct xc_dom_image *dom)
 {
     int rc;
-    xen_pfn_t pfn;
+    xen_pfn_t pfn, pages, allocsz, *p2mp;
 
     rc = x86_compat(dom->guest_xc, dom->guest_domid, dom->guest_type);
     if ( rc )
@@ -712,10 +713,16 @@ int arch_setup_meminit(struct xc_dom_ima
     for ( pfn = 0; pfn < dom->total_pages; pfn++ )
         dom->p2m_host[pfn] = pfn;
 
-    /* allocate guest memory */
-    rc = xc_domain_memory_populate_physmap(dom->guest_xc, dom->guest_domid,
-                                           dom->total_pages, 0, 0,
-                                           dom->p2m_host);
+    /* allocate guest memory MEM_CHUNKSZ at a time */
+    for ( rc=0, p2mp=dom->p2m_host, pages=dom->total_pages; !rc && pages; )
+    {
+        allocsz = (pages <= MEM_CHUNKSZ) ? pages : MEM_CHUNKSZ;
+        rc = xc_domain_memory_populate_physmap(dom->guest_xc, dom->guest_domid,
+                                               allocsz, 0, 0, p2mp);
+        pages -= allocsz;
+        p2mp += allocsz;
+    }
+
     return rc;
 }
 

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel