dom0 PVH: Over-allocation for domain 0: 393217 > 393216

dom0 PVH: Over-allocation for domain 0: 393217 > 393216

[Sander Eikelenboom]

Hi,

I just tried booting with "dom0pvh" and found the following warnings (complete 
combined xl-dmesg/dmesg attached) which don't show up when booting without 
dom0pvh:

(XEN) [2014-06-03 12:00:38] page_alloc.c:1465:d0v0 Over-allocation for domain 0: 393217 > 393216
(XEN) [2014-06-03 12:00:38] memory.c:158:d0v0 Could not allocate order=0 extent: id=0 memflags=0 (0 of 1)
[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Initializing cgroup subsys cpuacct
[    0.000000] Linux version 3.15.0-rc8-20140603-intelserver+ (root@rekenbeest) (gcc version 4.7.2 (Debian 4.7.2-5) ) #1 SMP Tue Jun 3 11:47:16 CEST 2014
[    0.000000] Command line: root=UUID=62eead45-148e-47b3-82b7-3eff2057b2b2 ro verbose earlyprintk=xen console=hvc0 console=tty0
[    0.000000] Released 102 pages of unused memory
[    0.000000] Set 403200 page(s) to 1-1 mapping
[    0.000000] ------------[ cut here ]------------
[    0.000000] WARNING: CPU: 0 PID: 0 at arch/x86/xen/setup.c:134 xen_do_chunk+0x188/0x241()
[    0.000000] Failed to populate pfn 60000 err=0
[    0.000000] Modules linked in:
[    0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 3.15.0-rc8-20140603-intelserver+ #1
[    0.000000]  0000000000000000 0000000000000009 ffffffff8155c30c ffffffff81a03ce8
[    0.000000]  ffffffff8104b8b7 ffffffff8107d1c0 ffffffff81af65d7 ffffffff81a41ba0
[    0.000000]  0000000000060000 0000000000000000 0000000000000000 0000000000060001
[    0.000000] Call Trace:
[    0.000000]  [<ffffffff8155c30c>] ? dump_stack+0x41/0x51
[    0.000000]  [<ffffffff8104b8b7>] ? warn_slowpath_common+0x78/0x90
[    0.000000]  [<ffffffff8107d1c0>] ? arch_local_irq_restore+0x13/0x14
[    0.000000]  [<ffffffff81af65d7>] ? xen_do_chunk+0x188/0x241
[    0.000000]  [<ffffffff8104b967>] ? warn_slowpath_fmt+0x45/0x4a
[    0.000000]  [<ffffffff8107d1d4>] ? arch_local_irq_disable+0x13/0x14
[    0.000000]  [<ffffffff81af65d7>] ? xen_do_chunk+0x188/0x241
[    0.000000]  [<ffffffff81af69ec>] ? xen_memory_setup+0x35c/0x698
[    0.000000]  [<ffffffff8107d1c0>] ? arch_local_irq_restore+0x13/0x14
[    0.000000]  [<ffffffff8101c27a>] ? paravirt_read_msr+0x15/0x16
[    0.000000]  [<ffffffff81afc867>] ? setup_memory_map+0x7/0x39
[    0.000000]  [<ffffffff81afa3ce>] ? setup_arch+0x1b2/0xc94
[    0.000000]  [<ffffffff81558176>] ? printk+0x4f/0x54
[    0.000000]  [<ffffffff8139cdb4>] ? dom0_write_console+0x12/0x18
[    0.000000]  [<ffffffff81af3aa6>] ? start_kernel+0xbe/0x3fe
[    0.000000]  [<ffffffff81af633e>] ? xen_start_kernel+0x4ed/0x4f6
[    0.000000] ---[ end trace cf9414bcd9c5d9a0 ]---

I do restrict dom0's mem on the xen commandline in grub:

multiboot       /boot/xen-4.5-unstable.gz dom0_mem=1536M,max:1536M loglvl=all loglvl_guest=all console_timestamps console_to_ring vga=gfx-1024x768x32 cpuidle cpufreq=xen debug com1=38400,8n1 console=vga,com1
module          /boot/vmlinuz-3.15.0-rc8-20140603-intelserver+ root=UUID=62eead45-148e-47b3-82b7-3eff2057b2b2 ro verbose earlyprintk=xen console=hvc0 console=tty0
  
Xen version:    xen-unstable, xen_changeset: Tue Jun 3 12:02:11 2014 +0200 git:08178cd
Kernel:         tree: linus, branch: next, last commit: a727eaf64ff084a50b983fc506810c7a576b7ce3 (so it has the patches of David's 3.16 pull request)

--
Sander

Re: dom0 PVH: Over-allocation for domain 0: 393217 > 393216

[Mukesh Rathor]

On Tue, 3 Jun 2014 14:11:17 +0200
Sander Eikelenboom <linux@eikelenboom.it> wrote:

> Hi,
> 
> I just tried booting with "dom0pvh" and found the following warnings
> (complete combined xl-dmesg/dmesg attached) which don't show up when
> booting without dom0pvh:

Yeah, I see. I'm able to reproduce. Something about dom0_mem=1536M,max:1536M
ie, specifying max in there. Without it, dom0_mem=1536M, its fine. You can
do that to play around more while I take a look.

thanks,
Mukesh

Re: dom0 PVH: Over-allocation for domain 0: 393217 > 393216

[Mukesh Rathor]

On Tue, 3 Jun 2014 12:28:27 -0700
Mukesh Rathor <mukesh.rathor@oracle.com> wrote:

> On Tue, 3 Jun 2014 14:11:17 +0200
> Sander Eikelenboom <linux@eikelenboom.it> wrote:
> 
> > Hi,
> > 
> > I just tried booting with "dom0pvh" and found the following warnings
> > (complete combined xl-dmesg/dmesg attached) which don't show up when
> > booting without dom0pvh:
> 
> Yeah, I see. I'm able to reproduce. Something about
> dom0_mem=1536M,max:1536M ie, specifying max in there. Without it,
> dom0_mem=1536M, its fine. You can do that to play around more while I
> take a look.

Ok, I know whats going on. Basically, we are trying to populate the
pfns removed due to punched holes in the e820, but because max is
specified (same as initial ram), guest_physmap_add_page can't add.

This will be fixed by the e820 work that Roger and David Vrabel are 
doing. Please keep an eye on the thread:

http://www.gossamer-threads.com/lists/xen/devel/332603

thanks
Mukesh

Re: dom0 PVH: Over-allocation for domain 0: 393217 > 393216

[Roger Pau Monné]

On 04/06/14 01:49, Mukesh Rathor wrote:
> On Tue, 3 Jun 2014 12:28:27 -0700
> Mukesh Rathor <mukesh.rathor@oracle.com> wrote:
> 
>> On Tue, 3 Jun 2014 14:11:17 +0200
>> Sander Eikelenboom <linux@eikelenboom.it> wrote:
>>
>>> Hi,
>>>
>>> I just tried booting with "dom0pvh" and found the following warnings
>>> (complete combined xl-dmesg/dmesg attached) which don't show up when
>>> booting without dom0pvh:
>>
>> Yeah, I see. I'm able to reproduce. Something about
>> dom0_mem=1536M,max:1536M ie, specifying max in there. Without it,
>> dom0_mem=1536M, its fine. You can do that to play around more while I
>> take a look.
> 
> Ok, I know whats going on. Basically, we are trying to populate the
> pfns removed due to punched holes in the e820, but because max is
> specified (same as initial ram), guest_physmap_add_page can't add.
> 
> This will be fixed by the e820 work that Roger and David Vrabel are 
> doing. Please keep an eye on the thread:
> 
> http://www.gossamer-threads.com/lists/xen/devel/332603

Hello,

I've been looking into this, and commented on David's patch:

http://lists.xenproject.org/archives/html/xen-devel/2014-06/msg00293.html

I'm also appending a modified version of patch 1, so you can apply it 
directly. It seems to work fine in all my test cases (no dom0_mem, 
dom0_mem=<mem> and dom0_mem=<mem>,max:<mem>).

---
commit f503c114f86d92036e1af4e4246550f15e85c4bb
Author: David Vrabel <david.vrabel@citrix.com>
Date:   Tue Jun 3 09:13:43 2014 +0200

    x86/xen: fix memory setup for PVH dom0
    
    Since af06d66ee32b (x86: fix setup of PVH Dom0 memory map) in Xen, PVH
    dom0 need only use the memory memory provided by Xen which has already
    setup all the correct holes.
    
    xen_memory_setup() then ends up being trivial for a PVH guest so
    introduce a new function (xen_auto_xlated_memory_setup()).
    
    Signed-off-by: David Vrabel <david.vrabel@citrix.com>
    Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>

diff --git a/arch/x86/xen/enlighten.c b/arch/x86/xen/enlighten.c
index 201d09a..4e7d2b7 100644
--- a/arch/x86/xen/enlighten.c
+++ b/arch/x86/xen/enlighten.c
@@ -1536,7 +1536,10 @@ asmlinkage void __init xen_start_kernel(void)
 	if (!xen_pvh_domain())
 		pv_cpu_ops = xen_cpu_ops;
 
-	x86_init.resources.memory_setup = xen_memory_setup;
+	if (xen_feature(XENFEAT_auto_translated_physmap))
+		x86_init.resources.memory_setup = xen_auto_xlated_memory_setup;
+	else
+		x86_init.resources.memory_setup = xen_memory_setup;
 	x86_init.oem.arch_setup = xen_arch_setup;
 	x86_init.oem.banner = xen_banner;
 
diff --git a/arch/x86/xen/setup.c b/arch/x86/xen/setup.c
index 0982233..e6e9df8 100644
--- a/arch/x86/xen/setup.c
+++ b/arch/x86/xen/setup.c
@@ -509,6 +509,34 @@ char * __init xen_memory_setup(void)
 }
 
 /*
+ * Machine specific memory setup for auto-translated guests.
+ */
+char * __init xen_auto_xlated_memory_setup(void)
+{
+	static struct e820entry map[E820MAX] __initdata;
+
+	struct xen_memory_map memmap;
+	int i;
+	int rc;
+
+	memmap.nr_entries = E820MAX;
+	set_xen_guest_handle(memmap.buffer, map);
+
+	rc = HYPERVISOR_memory_op(XENMEM_memory_map, &memmap);
+	BUG_ON(rc);
+
+	sanitize_e820_map(map, ARRAY_SIZE(map), &memmap.nr_entries);
+
+	for (i = 0; i < memmap.nr_entries; i++)
+		e820_add_region(map[i].addr, map[i].size, map[i].type);
+
+	memblock_reserve(__pa(xen_start_info->mfn_list),
+		xen_start_info->pt_base - xen_start_info->mfn_list);
+
+	return "Xen";
+}
+
+/*
  * Set the bit indicating "nosegneg" library variants should be used.
  * We only need to bother in pure 32-bit mode; compat 32-bit processes
  * can have un-truncated segments, so wrapping around is allowed.
diff --git a/arch/x86/xen/xen-ops.h b/arch/x86/xen/xen-ops.h
index 1cb6f4c..b371d18 100644
--- a/arch/x86/xen/xen-ops.h
+++ b/arch/x86/xen/xen-ops.h
@@ -34,6 +34,7 @@ extern unsigned long xen_max_p2m_pfn;
 void xen_set_pat(u64);
 
 char * __init xen_memory_setup(void);
+char * xen_auto_xlated_memory_setup(void);
 void __init xen_arch_setup(void);
 void xen_enable_sysenter(void);
 void xen_enable_syscall(void);

Re: dom0 PVH: Over-allocation for domain 0: 393217 > 393216

[Sander Eikelenboom]

Wednesday, June 4, 2014, 12:48:49 PM, you wrote:

> On 04/06/14 01:49, Mukesh Rathor wrote:
>> On Tue, 3 Jun 2014 12:28:27 -0700
>> Mukesh Rathor <mukesh.rathor@oracle.com> wrote:
>> 
>>> On Tue, 3 Jun 2014 14:11:17 +0200
>>> Sander Eikelenboom <linux@eikelenboom.it> wrote:
>>>
>>>> Hi,
>>>>
>>>> I just tried booting with "dom0pvh" and found the following warnings
>>>> (complete combined xl-dmesg/dmesg attached) which don't show up when
>>>> booting without dom0pvh:
>>>
>>> Yeah, I see. I'm able to reproduce. Something about
>>> dom0_mem=1536M,max:1536M ie, specifying max in there. Without it,
>>> dom0_mem=1536M, its fine. You can do that to play around more while I
>>> take a look.
>> 
>> Ok, I know whats going on. Basically, we are trying to populate the
>> pfns removed due to punched holes in the e820, but because max is
>> specified (same as initial ram), guest_physmap_add_page can't add.
>> 
>> This will be fixed by the e820 work that Roger and David Vrabel are 
>> doing. Please keep an eye on the thread:
>> 
>> http://www.gossamer-threads.com/lists/xen/devel/332603

> Hello,

> I've been looking into this, and commented on David's patch:

> http://lists.xenproject.org/archives/html/xen-devel/2014-06/msg00293.html

> I'm also appending a modified version of patch 1, so you can apply it 
> directly. It seems to work fine in all my test cases (no dom0_mem, 
> dom0_mem=<mem> and dom0_mem=<mem>,max:<mem>).

> ---
> commit f503c114f86d92036e1af4e4246550f15e85c4bb
> Author: David Vrabel <david.vrabel@citrix.com>
> Date:   Tue Jun 3 09:13:43 2014 +0200

>     x86/xen: fix memory setup for PVH dom0
>     
>     Since af06d66ee32b (x86: fix setup of PVH Dom0 memory map) in Xen, PVH
>     dom0 need only use the memory memory provided by Xen which has already
>     setup all the correct holes.
>     
>     xen_memory_setup() then ends up being trivial for a PVH guest so
>     introduce a new function (xen_auto_xlated_memory_setup()).
>     
>     Signed-off-by: David Vrabel <david.vrabel@citrix.com>
>     Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>

> diff --git a/arch/x86/xen/enlighten.c b/arch/x86/xen/enlighten.c
> index 201d09a..4e7d2b7 100644
> --- a/arch/x86/xen/enlighten.c
> +++ b/arch/x86/xen/enlighten.c
> @@ -1536,7 +1536,10 @@ asmlinkage void __init xen_start_kernel(void)
>         if (!xen_pvh_domain())
>                 pv_cpu_ops = xen_cpu_ops;
>  
> -       x86_init.resources.memory_setup = xen_memory_setup;
> +       if (xen_feature(XENFEAT_auto_translated_physmap))
> +               x86_init.resources.memory_setup = xen_auto_xlated_memory_setup;
> +       else
> +               x86_init.resources.memory_setup = xen_memory_setup;
>         x86_init.oem.arch_setup = xen_arch_setup;
>         x86_init.oem.banner = xen_banner;
>  
> diff --git a/arch/x86/xen/setup.c b/arch/x86/xen/setup.c
> index 0982233..e6e9df8 100644
> --- a/arch/x86/xen/setup.c
> +++ b/arch/x86/xen/setup.c
> @@ -509,6 +509,34 @@ char * __init xen_memory_setup(void)
>  }
>  
>  /*
> + * Machine specific memory setup for auto-translated guests.
> + */
> +char * __init xen_auto_xlated_memory_setup(void)
> +{
> +       static struct e820entry map[E820MAX] __initdata;
> +
> +       struct xen_memory_map memmap;
> +       int i;
> +       int rc;
> +
> +       memmap.nr_entries = E820MAX;
> +       set_xen_guest_handle(memmap.buffer, map);
> +
> +       rc = HYPERVISOR_memory_op(XENMEM_memory_map, &memmap);
> +       BUG_ON(rc);
> +
> +       sanitize_e820_map(map, ARRAY_SIZE(map), &memmap.nr_entries);
> +
> +       for (i = 0; i < memmap.nr_entries; i++)
> +               e820_add_region(map[i].addr, map[i].size, map[i].type);
> +
> +       memblock_reserve(__pa(xen_start_info->mfn_list),
> +               xen_start_info->pt_base - xen_start_info->mfn_list);
> +
> +       return "Xen";
> +}
> +
> +/*
>   * Set the bit indicating "nosegneg" library variants should be used.
>   * We only need to bother in pure 32-bit mode; compat 32-bit processes
>   * can have un-truncated segments, so wrapping around is allowed.
> diff --git a/arch/x86/xen/xen-ops.h b/arch/x86/xen/xen-ops.h
> index 1cb6f4c..b371d18 100644
> --- a/arch/x86/xen/xen-ops.h
> +++ b/arch/x86/xen/xen-ops.h
> @@ -34,6 +34,7 @@ extern unsigned long xen_max_p2m_pfn;
>  void xen_set_pat(u64);
>  
>  char * __init xen_memory_setup(void);
> +char * xen_auto_xlated_memory_setup(void);
>  void __init xen_arch_setup(void);
>  void xen_enable_sysenter(void);
>  void xen_enable_syscall(void);

Hi Roger,

Applied and it works for me as well !
Time to play some more :-)

Thx,

Sander