[PATCHv1] xen/balloon: disable memory hotplug in PV guests

[PATCHv1] xen/balloon: disable memory hotplug in PV guests

[David Vrabel]

Memory hotplug doesn't work with PV guests because:

  a) The p2m cannot be expanded to cover the new sections.

  b) add_memory() builds page tables for the new sections which means
     the new pages must have valid p2m entries (or a BUG occurs).

So, in PV guests, clamp the target to the current (== maximum) number
of pages and do not try to hotplug any memory.

Signed-off-by: David Vrabel <david.vrabel@citrix.com>
---
Cc: Daniel Kiper <daniel.kiper@oracle.com>
---
 drivers/xen/balloon.c |   13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/drivers/xen/balloon.c b/drivers/xen/balloon.c
index 0b52d92..4158196 100644
--- a/drivers/xen/balloon.c
+++ b/drivers/xen/balloon.c
@@ -225,6 +225,19 @@ static enum bp_state reserve_additional_memory(long credit)
 	u64 hotplug_start_paddr;
 	unsigned long balloon_hotplug = credit;
 
+	/*
+	 * Memory hotplug doesn't work with PV guests because:
+	 *
+	 *   a) The p2m cannot be expanded to cover the new sections.
+	 *
+	 *   b) add_memory() builds page tables for the new sections
+	 *      which means they must be fully populated in advance.
+	 */
+	if (xen_pv_domain()) {
+		balloon_stats.target_pages = balloon_stats.current_pages;
+		return BP_DONE;
+	}
+
 	hotplug_start_paddr = PFN_PHYS(SECTION_ALIGN_UP(max_pfn));
 	balloon_hotplug = round_up(balloon_hotplug, PAGES_PER_SECTION);
 	nid = memory_add_physaddr_to_nid(hotplug_start_paddr);
-- 
1.7.10.4

Re: [PATCHv1] xen/balloon: disable memory hotplug in PV guests

[Boris Ostrovsky]

On 03/09/2015 10:10 AM, David Vrabel wrote:
> Memory hotplug doesn't work with PV guests because:
>
>    a) The p2m cannot be expanded to cover the new sections.
>
>    b) add_memory() builds page tables for the new sections which means
>       the new pages must have valid p2m entries (or a BUG occurs).

Is this due to recent p2m rework?

-boris


>
> So, in PV guests, clamp the target to the current (== maximum) number
> of pages and do not try to hotplug any memory.
>
> Signed-off-by: David Vrabel <david.vrabel@citrix.com>
> ---
> Cc: Daniel Kiper <daniel.kiper@oracle.com>
> ---
>   drivers/xen/balloon.c |   13 +++++++++++++
>   1 file changed, 13 insertions(+)
>
> diff --git a/drivers/xen/balloon.c b/drivers/xen/balloon.c
> index 0b52d92..4158196 100644
> --- a/drivers/xen/balloon.c
> +++ b/drivers/xen/balloon.c
> @@ -225,6 +225,19 @@ static enum bp_state reserve_additional_memory(long credit)
>   	u64 hotplug_start_paddr;
>   	unsigned long balloon_hotplug = credit;
>   
> +	/*
> +	 * Memory hotplug doesn't work with PV guests because:
> +	 *
> +	 *   a) The p2m cannot be expanded to cover the new sections.
> +	 *
> +	 *   b) add_memory() builds page tables for the new sections
> +	 *      which means they must be fully populated in advance.
> +	 */
> +	if (xen_pv_domain()) {
> +		balloon_stats.target_pages = balloon_stats.current_pages;
> +		return BP_DONE;
> +	}
> +
>   	hotplug_start_paddr = PFN_PHYS(SECTION_ALIGN_UP(max_pfn));
>   	balloon_hotplug = round_up(balloon_hotplug, PAGES_PER_SECTION);
>   	nid = memory_add_physaddr_to_nid(hotplug_start_paddr);

Re: [PATCHv1] xen/balloon: disable memory hotplug in PV guests

[David Vrabel]

On 09/03/15 15:09, Boris Ostrovsky wrote:
> On 03/09/2015 10:10 AM, David Vrabel wrote:
>> Memory hotplug doesn't work with PV guests because:
>>
>>    a) The p2m cannot be expanded to cover the new sections.
>>
>>    b) add_memory() builds page tables for the new sections which means
>>       the new pages must have valid p2m entries (or a BUG occurs).
> 
> Is this due to recent p2m rework?

(a) is but (b) has always meant memory hotplug on PV was broken.

David

Re: [PATCHv1] xen/balloon: disable memory hotplug in PV guests

[Boris Ostrovsky]

On 03/09/2015 11:13 AM, David Vrabel wrote:
> On 09/03/15 15:09, Boris Ostrovsky wrote:
>> On 03/09/2015 10:10 AM, David Vrabel wrote:
>>> Memory hotplug doesn't work with PV guests because:
>>>
>>>     a) The p2m cannot be expanded to cover the new sections.
>>>
>>>     b) add_memory() builds page tables for the new sections which means
>>>        the new pages must have valid p2m entries (or a BUG occurs).
>> Is this due to recent p2m rework?
> (a) is but (b) has always meant memory hotplug on PV was broken.

I can see why (b) should probably have prevented it from working but I 
tried this on 4.4/3.17 and that seemed to work (it indeed does not 
function well on newer bits).

-boris

Re: [PATCHv1] xen/balloon: disable memory hotplug in PV guests

[David Vrabel]

On 09/03/15 15:25, Boris Ostrovsky wrote:
> On 03/09/2015 11:13 AM, David Vrabel wrote:
>> On 09/03/15 15:09, Boris Ostrovsky wrote:
>>> On 03/09/2015 10:10 AM, David Vrabel wrote:
>>>> Memory hotplug doesn't work with PV guests because:
>>>>
>>>>     a) The p2m cannot be expanded to cover the new sections.
>>>>
>>>>     b) add_memory() builds page tables for the new sections which means
>>>>        the new pages must have valid p2m entries (or a BUG occurs).
>>> Is this due to recent p2m rework?
>> (a) is but (b) has always meant memory hotplug on PV was broken.
> 
> I can see why (b) should probably have prevented it from working but I
> tried this on 4.4/3.17 and that seemed to work (it indeed does not
> function well on newer bits).

I tried it on 3.10 (+ some other bits and pieces) and it was definitely
broken for reason (b).  I had to refactor the balloon driver to populate
the sections before calling add_memory() before it would work.  But this
can't be done now because of (a).

David

Re: [PATCHv1] xen/balloon: disable memory hotplug in PV guests

[Konrad Rzeszutek Wilk]

On Mon, Mar 09, 2015 at 03:31:30PM +0000, David Vrabel wrote:
> On 09/03/15 15:25, Boris Ostrovsky wrote:
> > On 03/09/2015 11:13 AM, David Vrabel wrote:
> >> On 09/03/15 15:09, Boris Ostrovsky wrote:
> >>> On 03/09/2015 10:10 AM, David Vrabel wrote:
> >>>> Memory hotplug doesn't work with PV guests because:
> >>>>
> >>>>     a) The p2m cannot be expanded to cover the new sections.
> >>>>
> >>>>     b) add_memory() builds page tables for the new sections which means
> >>>>        the new pages must have valid p2m entries (or a BUG occurs).
> >>> Is this due to recent p2m rework?
> >> (a) is but (b) has always meant memory hotplug on PV was broken.
> > 
> > I can see why (b) should probably have prevented it from working but I
> > tried this on 4.4/3.17 and that seemed to work (it indeed does not
> > function well on newer bits).
> 
> I tried it on 3.10 (+ some other bits and pieces) and it was definitely
> broken for reason (b).  I had to refactor the balloon driver to populate
> the sections before calling add_memory() before it would work.  But this
> can't be done now because of (a).

That should really be reflected in the commit description (the a) part).
> 
> David

Re: [PATCHv1] xen/balloon: disable memory hotplug in PV guests

[Boris Ostrovsky]

On 03/09/2015 11:31 AM, David Vrabel wrote:
> On 09/03/15 15:25, Boris Ostrovsky wrote:
>> On 03/09/2015 11:13 AM, David Vrabel wrote:
>>> On 09/03/15 15:09, Boris Ostrovsky wrote:
>>>> On 03/09/2015 10:10 AM, David Vrabel wrote:
>>>>> Memory hotplug doesn't work with PV guests because:
>>>>>
>>>>>      a) The p2m cannot be expanded to cover the new sections.
>>>>>
>>>>>      b) add_memory() builds page tables for the new sections which means
>>>>>         the new pages must have valid p2m entries (or a BUG occurs).
>>>> Is this due to recent p2m rework?
>>> (a) is but (b) has always meant memory hotplug on PV was broken.
>> I can see why (b) should probably have prevented it from working but I
>> tried this on 4.4/3.17 and that seemed to work (it indeed does not
>> function well on newer bits).
> I tried it on 3.10 (+ some other bits and pieces) and it was definitely
> broken for reason (b).  I had to refactor the balloon driver to populate
> the sections before calling add_memory() before it would work.  But this
> can't be done now because of (a).

Could something have happened between 3.10 and 3.17 that fixed it (hard 
to believe but nevertheless)? Or this brokenness depends on Xen version 
(again, unlikely)?

In any case, I think this needs a message in the log stating that 
hotplug doesn't work for this guest (once, so that we don't into 
spamming the log like we have recently).

-boris

Re: [PATCHv1] xen/balloon: disable memory hotplug in PV guests

[Daniel Kiper]

On Mon, Mar 09, 2015 at 03:31:30PM +0000, David Vrabel wrote:
> On 09/03/15 15:25, Boris Ostrovsky wrote:
> > On 03/09/2015 11:13 AM, David Vrabel wrote:
> >> On 09/03/15 15:09, Boris Ostrovsky wrote:
> >>> On 03/09/2015 10:10 AM, David Vrabel wrote:
> >>>> Memory hotplug doesn't work with PV guests because:
> >>>>
> >>>>     a) The p2m cannot be expanded to cover the new sections.
> >>>>
> >>>>     b) add_memory() builds page tables for the new sections which means
> >>>>        the new pages must have valid p2m entries (or a BUG occurs).
> >>> Is this due to recent p2m rework?
> >> (a) is but (b) has always meant memory hotplug on PV was broken.
> >
> > I can see why (b) should probably have prevented it from working but I
> > tried this on 4.4/3.17 and that seemed to work (it indeed does not
> > function well on newer bits).
>
> I tried it on 3.10 (+ some other bits and pieces) and it was definitely
> broken for reason (b).  I had to refactor the balloon driver to populate

Are you sure? Could you provide more info?

> the sections before calling add_memory() before it would work.  But this

I am not sure what do you mean by that.

> can't be done now because of (a).

Could you enlighten me why p2m was reworked?

Daniel

Re: [PATCHv1] xen/balloon: disable memory hotplug in PV guests

[Daniel Kiper]

On Mon, Mar 09, 2015 at 11:25:07AM -0400, Boris Ostrovsky wrote:
> On 03/09/2015 11:13 AM, David Vrabel wrote:
> >On 09/03/15 15:09, Boris Ostrovsky wrote:
> >>On 03/09/2015 10:10 AM, David Vrabel wrote:
> >>>Memory hotplug doesn't work with PV guests because:
> >>>
> >>>    a) The p2m cannot be expanded to cover the new sections.
> >>>
> >>>    b) add_memory() builds page tables for the new sections which means
> >>>       the new pages must have valid p2m entries (or a BUG occurs).
> >>Is this due to recent p2m rework?
> >(a) is but (b) has always meant memory hotplug on PV was broken.
>
> I can see why (b) should probably have prevented it from working but
> I tried this on 4.4/3.17 and that seemed to work (it indeed does not
> function well on newer bits).

It was introduced in Linux Kernel 3.1 and it worked on PV, HVM and maybe
on PVHVM but I am not sure. IIRC, I tested this stuff on Xen 4.0 or 4.1.

Daniel

Re: [PATCHv1] xen/balloon: disable memory hotplug in PV guests

[David Vrabel]

On 09/03/15 14:10, David Vrabel wrote:
> Memory hotplug doesn't work with PV guests because:
> 
>   a) The p2m cannot be expanded to cover the new sections.

Broken by 054954eb051f35e74b75a566a96fe756015352c8 (xen: switch to
linear virtual mapped sparse p2m list).

This one would be non-trivial to fix.  We'd need a sparse set of
vm_area's for the p2m or similar.

>   b) add_memory() builds page tables for the new sections which means
>      the new pages must have valid p2m entries (or a BUG occurs).

After some more testing this appears to be broken by:

25b884a83d487fd62c3de7ac1ab5549979188482 (x86/xen: set regions above the
end of RAM as 1:1) included 3.16.

This one can be trivially fixed by setting the new sections in the p2m
to INVALID_P2M_ENTRY before calling add_memory().

David

Re: [PATCHv1] xen/balloon: disable memory hotplug in PV guests

[Boris Ostrovsky]

On 03/10/2015 07:40 AM, David Vrabel wrote:
> On 09/03/15 14:10, David Vrabel wrote:
>> Memory hotplug doesn't work with PV guests because:
>>
>>    a) The p2m cannot be expanded to cover the new sections.
> Broken by 054954eb051f35e74b75a566a96fe756015352c8 (xen: switch to
> linear virtual mapped sparse p2m list).
>
> This one would be non-trivial to fix.  We'd need a sparse set of
> vm_area's for the p2m or similar.
>
>>    b) add_memory() builds page tables for the new sections which means
>>       the new pages must have valid p2m entries (or a BUG occurs).
> After some more testing this appears to be broken by:
>
> 25b884a83d487fd62c3de7ac1ab5549979188482 (x86/xen: set regions above the
> end of RAM as 1:1) included 3.16.
>
> This one can be trivially fixed by setting the new sections in the p2m
> to INVALID_P2M_ENTRY before calling add_memory().

Have you tried 3.17? As I said yesterday, it worked for me (with 4.4 Xen).

-boris

Re: [PATCHv1] xen/balloon: disable memory hotplug in PV guests

[David Vrabel]

On 10/03/15 13:35, Boris Ostrovsky wrote:
> On 03/10/2015 07:40 AM, David Vrabel wrote:
>> On 09/03/15 14:10, David Vrabel wrote:
>>> Memory hotplug doesn't work with PV guests because:
>>>
>>>    a) The p2m cannot be expanded to cover the new sections.
>> Broken by 054954eb051f35e74b75a566a96fe756015352c8 (xen: switch to
>> linear virtual mapped sparse p2m list).
>>
>> This one would be non-trivial to fix.  We'd need a sparse set of
>> vm_area's for the p2m or similar.
>>
>>>    b) add_memory() builds page tables for the new sections which means
>>>       the new pages must have valid p2m entries (or a BUG occurs).
>> After some more testing this appears to be broken by:
>>
>> 25b884a83d487fd62c3de7ac1ab5549979188482 (x86/xen: set regions above the
>> end of RAM as 1:1) included 3.16.
>>
>> This one can be trivially fixed by setting the new sections in the p2m
>> to INVALID_P2M_ENTRY before calling add_memory().
> 
> Have you tried 3.17? As I said yesterday, it worked for me (with 4.4 Xen).

No.  But there are three bugs that prevent it from working in 3.16+ so
I'm really not sure how you had a working in a 3.17 PV guest.

Regardless, it definitely doesn't work now because of the linear p2m
change.  What do you want to do about this?

David

Re: [PATCHv1] xen/balloon: disable memory hotplug in PV guests

[Boris Ostrovsky]

On 03/11/2015 10:42 AM, David Vrabel wrote:
> On 10/03/15 13:35, Boris Ostrovsky wrote:
>> On 03/10/2015 07:40 AM, David Vrabel wrote:
>>> On 09/03/15 14:10, David Vrabel wrote:
>>>> Memory hotplug doesn't work with PV guests because:
>>>>
>>>>     a) The p2m cannot be expanded to cover the new sections.
>>> Broken by 054954eb051f35e74b75a566a96fe756015352c8 (xen: switch to
>>> linear virtual mapped sparse p2m list).
>>>
>>> This one would be non-trivial to fix.  We'd need a sparse set of
>>> vm_area's for the p2m or similar.
>>>
>>>>     b) add_memory() builds page tables for the new sections which means
>>>>        the new pages must have valid p2m entries (or a BUG occurs).
>>> After some more testing this appears to be broken by:
>>>
>>> 25b884a83d487fd62c3de7ac1ab5549979188482 (x86/xen: set regions above the
>>> end of RAM as 1:1) included 3.16.
>>>
>>> This one can be trivially fixed by setting the new sections in the p2m
>>> to INVALID_P2M_ENTRY before calling add_memory().
>> Have you tried 3.17? As I said yesterday, it worked for me (with 4.4 Xen).
> No.  But there are three bugs that prevent it from working in 3.16+ so
> I'm really not sure how you had a working in a 3.17 PV guest.

This is what I have:

[build@build-mk2 linux-boris]$ ssh root@tst008 cat 
/mnt/lab/bootstrap-x86_64/test_small.xm
extra="console=hvc0 debug earlyprintk=xen "
kernel="/mnt/lab/bootstrap-x86_64/vmlinuz"
ramdisk="/mnt/lab/bootstrap-x86_64/initramfs.cpio.gz"
memory=1024
maxmem = 4096
vcpus=1
maxvcpus=3
name="bootstrap-x86_64"
on_crash="preserve"
vif = [ 'mac=00:0F:4B:00:00:68, bridge=switch' ]
vnc=1
vnclisten="0.0.0.0"
disk=['phy:/dev/guests/bootstrap-x86_64,xvda,w']
[build@build-mk2 linux-boris]$ ssh root@tst008 xl create 
/mnt/lab/bootstrap-x86_64/test_small.xm
Parsing config from /mnt/lab/bootstrap-x86_64/test_small.xm
[build@build-mk2 linux-boris]$ ssh root@tst008 xl list |grep 
bootstrap-x86_64
bootstrap-x86_64                             2  1024     1 -b----       5.4
[build@build-mk2 linux-boris]$ ssh root@g-pvops uname -r
3.17.0upstream
[build@build-mk2 linux-boris]$ ssh root@g-pvops dmesg|grep paravirtualized
[    0.000000] Booting paravirtualized kernel on Xen
[build@build-mk2 linux-boris]$ ssh root@g-pvops grep MemTotal /proc/meminfo
MemTotal:         968036 kB
[build@build-mk2 linux-boris]$ ssh root@tst008 xl mem-set 
bootstrap-x86_64 2048
[build@build-mk2 linux-boris]$ ssh root@tst008 xl list |grep 
bootstrap-x86_64
bootstrap-x86_64                             2  2048     1 -b----       5.7
[build@build-mk2 linux-boris]$ ssh root@g-pvops grep MemTotal /proc/meminfo
MemTotal:        2016612 kB
[build@build-mk2 linux-boris]$


>
> Regardless, it definitely doesn't work now because of the linear p2m
> change.  What do you want to do about this?

Since backing out p2m changes is not an option I guess your patch is the 
only short-term alternative.

But this still looks like a regression so perhaps Juergen can take a 
look to see how it can be fixed.

-boris

Re: [PATCHv1] xen/balloon: disable memory hotplug in PV guests

[Juergen Gross]

On 03/11/2015 04:40 PM, Boris Ostrovsky wrote:
> On 03/11/2015 10:42 AM, David Vrabel wrote:
>> On 10/03/15 13:35, Boris Ostrovsky wrote:
>>> On 03/10/2015 07:40 AM, David Vrabel wrote:
>>>> On 09/03/15 14:10, David Vrabel wrote:
>>>>> Memory hotplug doesn't work with PV guests because:
>>>>>
>>>>>     a) The p2m cannot be expanded to cover the new sections.
>>>> Broken by 054954eb051f35e74b75a566a96fe756015352c8 (xen: switch to
>>>> linear virtual mapped sparse p2m list).
>>>>
>>>> This one would be non-trivial to fix.  We'd need a sparse set of
>>>> vm_area's for the p2m or similar.
>>>>
>>>>>     b) add_memory() builds page tables for the new sections which
>>>>> means
>>>>>        the new pages must have valid p2m entries (or a BUG occurs).
>>>> After some more testing this appears to be broken by:
>>>>
>>>> 25b884a83d487fd62c3de7ac1ab5549979188482 (x86/xen: set regions above
>>>> the
>>>> end of RAM as 1:1) included 3.16.
>>>>
>>>> This one can be trivially fixed by setting the new sections in the p2m
>>>> to INVALID_P2M_ENTRY before calling add_memory().
>>> Have you tried 3.17? As I said yesterday, it worked for me (with 4.4
>>> Xen).
>> No.  But there are three bugs that prevent it from working in 3.16+ so
>> I'm really not sure how you had a working in a 3.17 PV guest.
>
> This is what I have:
>
> [build@build-mk2 linux-boris]$ ssh root@tst008 cat
> /mnt/lab/bootstrap-x86_64/test_small.xm
> extra="console=hvc0 debug earlyprintk=xen "
> kernel="/mnt/lab/bootstrap-x86_64/vmlinuz"
> ramdisk="/mnt/lab/bootstrap-x86_64/initramfs.cpio.gz"
> memory=1024
> maxmem = 4096
> vcpus=1
> maxvcpus=3
> name="bootstrap-x86_64"
> on_crash="preserve"
> vif = [ 'mac=00:0F:4B:00:00:68, bridge=switch' ]
> vnc=1
> vnclisten="0.0.0.0"
> disk=['phy:/dev/guests/bootstrap-x86_64,xvda,w']
> [build@build-mk2 linux-boris]$ ssh root@tst008 xl create
> /mnt/lab/bootstrap-x86_64/test_small.xm
> Parsing config from /mnt/lab/bootstrap-x86_64/test_small.xm
> [build@build-mk2 linux-boris]$ ssh root@tst008 xl list |grep
> bootstrap-x86_64
> bootstrap-x86_64                             2  1024     1 -b----       5.4
> [build@build-mk2 linux-boris]$ ssh root@g-pvops uname -r
> 3.17.0upstream
> [build@build-mk2 linux-boris]$ ssh root@g-pvops dmesg|grep paravirtualized
> [    0.000000] Booting paravirtualized kernel on Xen
> [build@build-mk2 linux-boris]$ ssh root@g-pvops grep MemTotal /proc/meminfo
> MemTotal:         968036 kB
> [build@build-mk2 linux-boris]$ ssh root@tst008 xl mem-set
> bootstrap-x86_64 2048
> [build@build-mk2 linux-boris]$ ssh root@tst008 xl list |grep
> bootstrap-x86_64
> bootstrap-x86_64                             2  2048     1 -b----       5.7
> [build@build-mk2 linux-boris]$ ssh root@g-pvops grep MemTotal /proc/meminfo
> MemTotal:        2016612 kB
> [build@build-mk2 linux-boris]$
>
>
>>
>> Regardless, it definitely doesn't work now because of the linear p2m
>> change.  What do you want to do about this?
>
> Since backing out p2m changes is not an option I guess your patch is the
> only short-term alternative.
>
> But this still looks like a regression so perhaps Juergen can take a
> look to see how it can be fixed.

Hmm, the p2m list is allocated for the maximum memory size of the domain
which is obtained from the hypervisor. In case of Dom0 it is read via
XENMEM_maximum_reservation, for a domU it is based on the E820 memory
map read via XENMEM_memory_map.

I just tested it with a 4.0-rc1 domU kernel with 512MB initial memory
and 4GB of maxmem. The E820 map looked like this:

[    0.000000] Xen: [mem 0x0000000000000000-0x000000000009ffff] usable
[    0.000000] Xen: [mem 0x00000000000a0000-0x00000000000fffff] reserved
[    0.000000] Xen: [mem 0x0000000000100000-0x00000000ffffffff] usable

So the complete 4GB were included, like they should. The resulting p2m
list is allocated in the needed size:

[    0.000000] p2m virtual area at ffffc90000000000, size is 800000

So what is your problem here? Can you post the E820 map and the p2m map
info for your failing domain, please?


Juergen

Re: [PATCHv1] xen/balloon: disable memory hotplug in PV guests

[Daniel Kiper]

On Mon, Mar 16, 2015 at 06:35:04AM +0100, Juergen Gross wrote:
> On 03/11/2015 04:40 PM, Boris Ostrovsky wrote:
> >On 03/11/2015 10:42 AM, David Vrabel wrote:
> >>On 10/03/15 13:35, Boris Ostrovsky wrote:
> >>>On 03/10/2015 07:40 AM, David Vrabel wrote:
> >>>>On 09/03/15 14:10, David Vrabel wrote:
> >>>>>Memory hotplug doesn't work with PV guests because:
> >>>>>
> >>>>>    a) The p2m cannot be expanded to cover the new sections.
> >>>>Broken by 054954eb051f35e74b75a566a96fe756015352c8 (xen: switch to
> >>>>linear virtual mapped sparse p2m list).
> >>>>
> >>>>This one would be non-trivial to fix.  We'd need a sparse set of
> >>>>vm_area's for the p2m or similar.
> >>>>
> >>>>>    b) add_memory() builds page tables for the new sections which
> >>>>>means
> >>>>>       the new pages must have valid p2m entries (or a BUG occurs).
> >>>>After some more testing this appears to be broken by:
> >>>>
> >>>>25b884a83d487fd62c3de7ac1ab5549979188482 (x86/xen: set regions above
> >>>>the
> >>>>end of RAM as 1:1) included 3.16.
> >>>>
> >>>>This one can be trivially fixed by setting the new sections in the p2m
> >>>>to INVALID_P2M_ENTRY before calling add_memory().
> >>>Have you tried 3.17? As I said yesterday, it worked for me (with 4.4
> >>>Xen).
> >>No.  But there are three bugs that prevent it from working in 3.16+ so
> >>I'm really not sure how you had a working in a 3.17 PV guest.
> >
> >This is what I have:
> >
> >[build@build-mk2 linux-boris]$ ssh root@tst008 cat
> >/mnt/lab/bootstrap-x86_64/test_small.xm
> >extra="console=hvc0 debug earlyprintk=xen "
> >kernel="/mnt/lab/bootstrap-x86_64/vmlinuz"
> >ramdisk="/mnt/lab/bootstrap-x86_64/initramfs.cpio.gz"
> >memory=1024
> >maxmem = 4096
> >vcpus=1
> >maxvcpus=3
> >name="bootstrap-x86_64"
> >on_crash="preserve"
> >vif = [ 'mac=00:0F:4B:00:00:68, bridge=switch' ]
> >vnc=1
> >vnclisten="0.0.0.0"
> >disk=['phy:/dev/guests/bootstrap-x86_64,xvda,w']
> >[build@build-mk2 linux-boris]$ ssh root@tst008 xl create
> >/mnt/lab/bootstrap-x86_64/test_small.xm
> >Parsing config from /mnt/lab/bootstrap-x86_64/test_small.xm
> >[build@build-mk2 linux-boris]$ ssh root@tst008 xl list |grep
> >bootstrap-x86_64
> >bootstrap-x86_64                             2  1024     1 -b----       5.4
> >[build@build-mk2 linux-boris]$ ssh root@g-pvops uname -r
> >3.17.0upstream
> >[build@build-mk2 linux-boris]$ ssh root@g-pvops dmesg|grep paravirtualized
> >[    0.000000] Booting paravirtualized kernel on Xen
> >[build@build-mk2 linux-boris]$ ssh root@g-pvops grep MemTotal /proc/meminfo
> >MemTotal:         968036 kB
> >[build@build-mk2 linux-boris]$ ssh root@tst008 xl mem-set
> >bootstrap-x86_64 2048
> >[build@build-mk2 linux-boris]$ ssh root@tst008 xl list |grep
> >bootstrap-x86_64
> >bootstrap-x86_64                             2  2048     1 -b----       5.7
> >[build@build-mk2 linux-boris]$ ssh root@g-pvops grep MemTotal /proc/meminfo
> >MemTotal:        2016612 kB
> >[build@build-mk2 linux-boris]$
> >
> >
> >>
> >>Regardless, it definitely doesn't work now because of the linear p2m
> >>change.  What do you want to do about this?
> >
> >Since backing out p2m changes is not an option I guess your patch is the
> >only short-term alternative.
> >
> >But this still looks like a regression so perhaps Juergen can take a
> >look to see how it can be fixed.
>
> Hmm, the p2m list is allocated for the maximum memory size of the domain
> which is obtained from the hypervisor. In case of Dom0 it is read via
> XENMEM_maximum_reservation, for a domU it is based on the E820 memory
> map read via XENMEM_memory_map.
>
> I just tested it with a 4.0-rc1 domU kernel with 512MB initial memory
> and 4GB of maxmem. The E820 map looked like this:
>
> [    0.000000] Xen: [mem 0x0000000000000000-0x000000000009ffff] usable
> [    0.000000] Xen: [mem 0x00000000000a0000-0x00000000000fffff] reserved
> [    0.000000] Xen: [mem 0x0000000000100000-0x00000000ffffffff] usable
>
> So the complete 4GB were included, like they should. The resulting p2m
> list is allocated in the needed size:
>
> [    0.000000] p2m virtual area at ffffc90000000000, size is 800000
>
> So what is your problem here? Can you post the E820 map and the p2m map
> info for your failing domain, please?

If you use memory hotplug then maxmem is not a limit from guest kernel
point of view (host still must allow that operation but it is another
not related issue). The problem is that p2m must be dynamically expendable
to support it. Earlier implementation supported that thing and memory
hotplug worked without any issue.

Daniel

Re: [PATCHv1] xen/balloon: disable memory hotplug in PV guests

[Juergen Gross]

On 03/16/2015 11:03 AM, Daniel Kiper wrote:
> On Mon, Mar 16, 2015 at 06:35:04AM +0100, Juergen Gross wrote:
>> On 03/11/2015 04:40 PM, Boris Ostrovsky wrote:
>>> On 03/11/2015 10:42 AM, David Vrabel wrote:
>>>> On 10/03/15 13:35, Boris Ostrovsky wrote:
>>>>> On 03/10/2015 07:40 AM, David Vrabel wrote:
>>>>>> On 09/03/15 14:10, David Vrabel wrote:
>>>>>>> Memory hotplug doesn't work with PV guests because:
>>>>>>>
>>>>>>>     a) The p2m cannot be expanded to cover the new sections.
>>>>>> Broken by 054954eb051f35e74b75a566a96fe756015352c8 (xen: switch to
>>>>>> linear virtual mapped sparse p2m list).
>>>>>>
>>>>>> This one would be non-trivial to fix.  We'd need a sparse set of
>>>>>> vm_area's for the p2m or similar.
>>>>>>
>>>>>>>     b) add_memory() builds page tables for the new sections which
>>>>>>> means
>>>>>>>        the new pages must have valid p2m entries (or a BUG occurs).
>>>>>> After some more testing this appears to be broken by:
>>>>>>
>>>>>> 25b884a83d487fd62c3de7ac1ab5549979188482 (x86/xen: set regions above
>>>>>> the
>>>>>> end of RAM as 1:1) included 3.16.
>>>>>>
>>>>>> This one can be trivially fixed by setting the new sections in the p2m
>>>>>> to INVALID_P2M_ENTRY before calling add_memory().
>>>>> Have you tried 3.17? As I said yesterday, it worked for me (with 4.4
>>>>> Xen).
>>>> No.  But there are three bugs that prevent it from working in 3.16+ so
>>>> I'm really not sure how you had a working in a 3.17 PV guest.
>>>
>>> This is what I have:
>>>
>>> [build@build-mk2 linux-boris]$ ssh root@tst008 cat
>>> /mnt/lab/bootstrap-x86_64/test_small.xm
>>> extra="console=hvc0 debug earlyprintk=xen "
>>> kernel="/mnt/lab/bootstrap-x86_64/vmlinuz"
>>> ramdisk="/mnt/lab/bootstrap-x86_64/initramfs.cpio.gz"
>>> memory=1024
>>> maxmem = 4096
>>> vcpus=1
>>> maxvcpus=3
>>> name="bootstrap-x86_64"
>>> on_crash="preserve"
>>> vif = [ 'mac=00:0F:4B:00:00:68, bridge=switch' ]
>>> vnc=1
>>> vnclisten="0.0.0.0"
>>> disk=['phy:/dev/guests/bootstrap-x86_64,xvda,w']
>>> [build@build-mk2 linux-boris]$ ssh root@tst008 xl create
>>> /mnt/lab/bootstrap-x86_64/test_small.xm
>>> Parsing config from /mnt/lab/bootstrap-x86_64/test_small.xm
>>> [build@build-mk2 linux-boris]$ ssh root@tst008 xl list |grep
>>> bootstrap-x86_64
>>> bootstrap-x86_64                             2  1024     1 -b----       5.4
>>> [build@build-mk2 linux-boris]$ ssh root@g-pvops uname -r
>>> 3.17.0upstream
>>> [build@build-mk2 linux-boris]$ ssh root@g-pvops dmesg|grep paravirtualized
>>> [    0.000000] Booting paravirtualized kernel on Xen
>>> [build@build-mk2 linux-boris]$ ssh root@g-pvops grep MemTotal /proc/meminfo
>>> MemTotal:         968036 kB
>>> [build@build-mk2 linux-boris]$ ssh root@tst008 xl mem-set
>>> bootstrap-x86_64 2048
>>> [build@build-mk2 linux-boris]$ ssh root@tst008 xl list |grep
>>> bootstrap-x86_64
>>> bootstrap-x86_64                             2  2048     1 -b----       5.7
>>> [build@build-mk2 linux-boris]$ ssh root@g-pvops grep MemTotal /proc/meminfo
>>> MemTotal:        2016612 kB
>>> [build@build-mk2 linux-boris]$
>>>
>>>
>>>>
>>>> Regardless, it definitely doesn't work now because of the linear p2m
>>>> change.  What do you want to do about this?
>>>
>>> Since backing out p2m changes is not an option I guess your patch is the
>>> only short-term alternative.
>>>
>>> But this still looks like a regression so perhaps Juergen can take a
>>> look to see how it can be fixed.
>>
>> Hmm, the p2m list is allocated for the maximum memory size of the domain
>> which is obtained from the hypervisor. In case of Dom0 it is read via
>> XENMEM_maximum_reservation, for a domU it is based on the E820 memory
>> map read via XENMEM_memory_map.
>>
>> I just tested it with a 4.0-rc1 domU kernel with 512MB initial memory
>> and 4GB of maxmem. The E820 map looked like this:
>>
>> [    0.000000] Xen: [mem 0x0000000000000000-0x000000000009ffff] usable
>> [    0.000000] Xen: [mem 0x00000000000a0000-0x00000000000fffff] reserved
>> [    0.000000] Xen: [mem 0x0000000000100000-0x00000000ffffffff] usable
>>
>> So the complete 4GB were included, like they should. The resulting p2m
>> list is allocated in the needed size:
>>
>> [    0.000000] p2m virtual area at ffffc90000000000, size is 800000
>>
>> So what is your problem here? Can you post the E820 map and the p2m map
>> info for your failing domain, please?
>
> If you use memory hotplug then maxmem is not a limit from guest kernel
> point of view (host still must allow that operation but it is another
> not related issue). The problem is that p2m must be dynamically expendable
> to support it. Earlier implementation supported that thing and memory
> hotplug worked without any issue.

Okay, now I get it.

The problem with the earlier p2m implementation was that it was
expendable to support only up to 512GB of RAM. So we need some way to
tell the kernel how much virtual memory it should reserve for the p2m
list if memory hotplug is enabled. We could:

a) use a configurable maximum (e.g. for 512GB RAM as today)

b) use the maximum of RAM the machine the domain is started on can ever
    have (what about migration then?)

c) use a kernel parameter specifying the maximum memory size to support

d) a combination of some of the above possibilities

Any thoughts? I think I'd prefer b)+c).


Juergen

Re: [PATCHv1] xen/balloon: disable memory hotplug in PV guests

[Daniel Kiper]

On Mon, Mar 16, 2015 at 11:31:49AM +0100, Juergen Gross wrote:
> On 03/16/2015 11:03 AM, Daniel Kiper wrote:
> >On Mon, Mar 16, 2015 at 06:35:04AM +0100, Juergen Gross wrote:
> >>On 03/11/2015 04:40 PM, Boris Ostrovsky wrote:
> >>>On 03/11/2015 10:42 AM, David Vrabel wrote:
> >>>>On 10/03/15 13:35, Boris Ostrovsky wrote:
> >>>>>On 03/10/2015 07:40 AM, David Vrabel wrote:
> >>>>>>On 09/03/15 14:10, David Vrabel wrote:
> >>>>>>>Memory hotplug doesn't work with PV guests because:
> >>>>>>>
> >>>>>>>    a) The p2m cannot be expanded to cover the new sections.
> >>>>>>Broken by 054954eb051f35e74b75a566a96fe756015352c8 (xen: switch to
> >>>>>>linear virtual mapped sparse p2m list).
> >>>>>>
> >>>>>>This one would be non-trivial to fix.  We'd need a sparse set of
> >>>>>>vm_area's for the p2m or similar.
> >>>>>>
> >>>>>>>    b) add_memory() builds page tables for the new sections which
> >>>>>>>means
> >>>>>>>       the new pages must have valid p2m entries (or a BUG occurs).
> >>>>>>After some more testing this appears to be broken by:
> >>>>>>
> >>>>>>25b884a83d487fd62c3de7ac1ab5549979188482 (x86/xen: set regions above
> >>>>>>the
> >>>>>>end of RAM as 1:1) included 3.16.
> >>>>>>
> >>>>>>This one can be trivially fixed by setting the new sections in the p2m
> >>>>>>to INVALID_P2M_ENTRY before calling add_memory().
> >>>>>Have you tried 3.17? As I said yesterday, it worked for me (with 4.4
> >>>>>Xen).
> >>>>No.  But there are three bugs that prevent it from working in 3.16+ so
> >>>>I'm really not sure how you had a working in a 3.17 PV guest.
> >>>
> >>>This is what I have:
> >>>
> >>>[build@build-mk2 linux-boris]$ ssh root@tst008 cat
> >>>/mnt/lab/bootstrap-x86_64/test_small.xm
> >>>extra="console=hvc0 debug earlyprintk=xen "
> >>>kernel="/mnt/lab/bootstrap-x86_64/vmlinuz"
> >>>ramdisk="/mnt/lab/bootstrap-x86_64/initramfs.cpio.gz"
> >>>memory=1024
> >>>maxmem = 4096
> >>>vcpus=1
> >>>maxvcpus=3
> >>>name="bootstrap-x86_64"
> >>>on_crash="preserve"
> >>>vif = [ 'mac=00:0F:4B:00:00:68, bridge=switch' ]
> >>>vnc=1
> >>>vnclisten="0.0.0.0"
> >>>disk=['phy:/dev/guests/bootstrap-x86_64,xvda,w']
> >>>[build@build-mk2 linux-boris]$ ssh root@tst008 xl create
> >>>/mnt/lab/bootstrap-x86_64/test_small.xm
> >>>Parsing config from /mnt/lab/bootstrap-x86_64/test_small.xm
> >>>[build@build-mk2 linux-boris]$ ssh root@tst008 xl list |grep
> >>>bootstrap-x86_64
> >>>bootstrap-x86_64                             2  1024     1 -b----       5.4
> >>>[build@build-mk2 linux-boris]$ ssh root@g-pvops uname -r
> >>>3.17.0upstream
> >>>[build@build-mk2 linux-boris]$ ssh root@g-pvops dmesg|grep paravirtualized
> >>>[    0.000000] Booting paravirtualized kernel on Xen
> >>>[build@build-mk2 linux-boris]$ ssh root@g-pvops grep MemTotal /proc/meminfo
> >>>MemTotal:         968036 kB
> >>>[build@build-mk2 linux-boris]$ ssh root@tst008 xl mem-set
> >>>bootstrap-x86_64 2048
> >>>[build@build-mk2 linux-boris]$ ssh root@tst008 xl list |grep
> >>>bootstrap-x86_64
> >>>bootstrap-x86_64                             2  2048     1 -b----       5.7
> >>>[build@build-mk2 linux-boris]$ ssh root@g-pvops grep MemTotal /proc/meminfo
> >>>MemTotal:        2016612 kB
> >>>[build@build-mk2 linux-boris]$
> >>>
> >>>
> >>>>
> >>>>Regardless, it definitely doesn't work now because of the linear p2m
> >>>>change.  What do you want to do about this?
> >>>
> >>>Since backing out p2m changes is not an option I guess your patch is the
> >>>only short-term alternative.
> >>>
> >>>But this still looks like a regression so perhaps Juergen can take a
> >>>look to see how it can be fixed.
> >>
> >>Hmm, the p2m list is allocated for the maximum memory size of the domain
> >>which is obtained from the hypervisor. In case of Dom0 it is read via
> >>XENMEM_maximum_reservation, for a domU it is based on the E820 memory
> >>map read via XENMEM_memory_map.
> >>
> >>I just tested it with a 4.0-rc1 domU kernel with 512MB initial memory
> >>and 4GB of maxmem. The E820 map looked like this:
> >>
> >>[    0.000000] Xen: [mem 0x0000000000000000-0x000000000009ffff] usable
> >>[    0.000000] Xen: [mem 0x00000000000a0000-0x00000000000fffff] reserved
> >>[    0.000000] Xen: [mem 0x0000000000100000-0x00000000ffffffff] usable
> >>
> >>So the complete 4GB were included, like they should. The resulting p2m
> >>list is allocated in the needed size:
> >>
> >>[    0.000000] p2m virtual area at ffffc90000000000, size is 800000
> >>
> >>So what is your problem here? Can you post the E820 map and the p2m map
> >>info for your failing domain, please?
> >
> >If you use memory hotplug then maxmem is not a limit from guest kernel
> >point of view (host still must allow that operation but it is another
> >not related issue). The problem is that p2m must be dynamically expendable
> >to support it. Earlier implementation supported that thing and memory
> >hotplug worked without any issue.
>
> Okay, now I get it.
>
> The problem with the earlier p2m implementation was that it was
> expendable to support only up to 512GB of RAM. So we need some way to
> tell the kernel how much virtual memory it should reserve for the p2m
> list if memory hotplug is enabled. We could:
>
> a) use a configurable maximum (e.g. for 512GB RAM as today)
>
> b) use the maximum of RAM the machine the domain is started on can ever
>    have (what about migration then?)

Memory hotplug for Xen is planned, so, this would not work.

> c) use a kernel parameter specifying the maximum memory size to support
>
> d) a combination of some of the above possibilities
>
> Any thoughts? I think I'd prefer b)+c).

I do not know new p2m design well so correct me if I am wrong.

If you set size limit on p2m then memory hotplug does not make sense.
Ballooning solves all your problems in simple manner. Memory hotplug
is a solution if you would like to set up a guest with small amount
of memory, you are not able to predict future requirements (usually
you are not) and you are not able to restart machine at any given
moment due to tight requirements. So, I think that p2m size, if must
be limited, then it should be limited by platform limitations (e.g.
number of address lines) but not artificially as you suggested.

Additionally, I have a feeling that you are going to preallocate space
in p2m for potential memory hotplug uses. Am I right? If yes then
I think this is unneeded waste of memory.

As I know new p2m is a table with MFNs. If yes, then I think we can
potentially add more space for new MFNs at the and of that table (or
create new larger one then copy old table contents and destroy former
one; Hmmm... What about memory fragmentation which may prevent us
allocating this big chunk of memory...). If this works then this
process could be repeated without limitations (Well... We will be
just limited by platform itself).

Daniel

Re: [PATCHv1] xen/balloon: disable memory hotplug in PV guests

[Juergen Gross]

On 03/17/2015 01:40 PM, Daniel Kiper wrote:
> On Mon, Mar 16, 2015 at 11:31:49AM +0100, Juergen Gross wrote:
>> On 03/16/2015 11:03 AM, Daniel Kiper wrote:
>>> On Mon, Mar 16, 2015 at 06:35:04AM +0100, Juergen Gross wrote:
>>>> On 03/11/2015 04:40 PM, Boris Ostrovsky wrote:
>>>>> On 03/11/2015 10:42 AM, David Vrabel wrote:
>>>>>> On 10/03/15 13:35, Boris Ostrovsky wrote:
>>>>>>> On 03/10/2015 07:40 AM, David Vrabel wrote:
>>>>>>>> On 09/03/15 14:10, David Vrabel wrote:
>>>>>>>>> Memory hotplug doesn't work with PV guests because:
>>>>>>>>>
>>>>>>>>>     a) The p2m cannot be expanded to cover the new sections.
>>>>>>>> Broken by 054954eb051f35e74b75a566a96fe756015352c8 (xen: switch to
>>>>>>>> linear virtual mapped sparse p2m list).
>>>>>>>>
>>>>>>>> This one would be non-trivial to fix.  We'd need a sparse set of
>>>>>>>> vm_area's for the p2m or similar.
>>>>>>>>
>>>>>>>>>     b) add_memory() builds page tables for the new sections which
>>>>>>>>> means
>>>>>>>>>        the new pages must have valid p2m entries (or a BUG occurs).
>>>>>>>> After some more testing this appears to be broken by:
>>>>>>>>
>>>>>>>> 25b884a83d487fd62c3de7ac1ab5549979188482 (x86/xen: set regions above
>>>>>>>> the
>>>>>>>> end of RAM as 1:1) included 3.16.
>>>>>>>>
>>>>>>>> This one can be trivially fixed by setting the new sections in the p2m
>>>>>>>> to INVALID_P2M_ENTRY before calling add_memory().
>>>>>>> Have you tried 3.17? As I said yesterday, it worked for me (with 4.4
>>>>>>> Xen).
>>>>>> No.  But there are three bugs that prevent it from working in 3.16+ so
>>>>>> I'm really not sure how you had a working in a 3.17 PV guest.
>>>>>
>>>>> This is what I have:
>>>>>
>>>>> [build@build-mk2 linux-boris]$ ssh root@tst008 cat
>>>>> /mnt/lab/bootstrap-x86_64/test_small.xm
>>>>> extra="console=hvc0 debug earlyprintk=xen "
>>>>> kernel="/mnt/lab/bootstrap-x86_64/vmlinuz"
>>>>> ramdisk="/mnt/lab/bootstrap-x86_64/initramfs.cpio.gz"
>>>>> memory=1024
>>>>> maxmem = 4096
>>>>> vcpus=1
>>>>> maxvcpus=3
>>>>> name="bootstrap-x86_64"
>>>>> on_crash="preserve"
>>>>> vif = [ 'mac=00:0F:4B:00:00:68, bridge=switch' ]
>>>>> vnc=1
>>>>> vnclisten="0.0.0.0"
>>>>> disk=['phy:/dev/guests/bootstrap-x86_64,xvda,w']
>>>>> [build@build-mk2 linux-boris]$ ssh root@tst008 xl create
>>>>> /mnt/lab/bootstrap-x86_64/test_small.xm
>>>>> Parsing config from /mnt/lab/bootstrap-x86_64/test_small.xm
>>>>> [build@build-mk2 linux-boris]$ ssh root@tst008 xl list |grep
>>>>> bootstrap-x86_64
>>>>> bootstrap-x86_64                             2  1024     1 -b----       5.4
>>>>> [build@build-mk2 linux-boris]$ ssh root@g-pvops uname -r
>>>>> 3.17.0upstream
>>>>> [build@build-mk2 linux-boris]$ ssh root@g-pvops dmesg|grep paravirtualized
>>>>> [    0.000000] Booting paravirtualized kernel on Xen
>>>>> [build@build-mk2 linux-boris]$ ssh root@g-pvops grep MemTotal /proc/meminfo
>>>>> MemTotal:         968036 kB
>>>>> [build@build-mk2 linux-boris]$ ssh root@tst008 xl mem-set
>>>>> bootstrap-x86_64 2048
>>>>> [build@build-mk2 linux-boris]$ ssh root@tst008 xl list |grep
>>>>> bootstrap-x86_64
>>>>> bootstrap-x86_64                             2  2048     1 -b----       5.7
>>>>> [build@build-mk2 linux-boris]$ ssh root@g-pvops grep MemTotal /proc/meminfo
>>>>> MemTotal:        2016612 kB
>>>>> [build@build-mk2 linux-boris]$
>>>>>
>>>>>
>>>>>>
>>>>>> Regardless, it definitely doesn't work now because of the linear p2m
>>>>>> change.  What do you want to do about this?
>>>>>
>>>>> Since backing out p2m changes is not an option I guess your patch is the
>>>>> only short-term alternative.
>>>>>
>>>>> But this still looks like a regression so perhaps Juergen can take a
>>>>> look to see how it can be fixed.
>>>>
>>>> Hmm, the p2m list is allocated for the maximum memory size of the domain
>>>> which is obtained from the hypervisor. In case of Dom0 it is read via
>>>> XENMEM_maximum_reservation, for a domU it is based on the E820 memory
>>>> map read via XENMEM_memory_map.
>>>>
>>>> I just tested it with a 4.0-rc1 domU kernel with 512MB initial memory
>>>> and 4GB of maxmem. The E820 map looked like this:
>>>>
>>>> [    0.000000] Xen: [mem 0x0000000000000000-0x000000000009ffff] usable
>>>> [    0.000000] Xen: [mem 0x00000000000a0000-0x00000000000fffff] reserved
>>>> [    0.000000] Xen: [mem 0x0000000000100000-0x00000000ffffffff] usable
>>>>
>>>> So the complete 4GB were included, like they should. The resulting p2m
>>>> list is allocated in the needed size:
>>>>
>>>> [    0.000000] p2m virtual area at ffffc90000000000, size is 800000
>>>>
>>>> So what is your problem here? Can you post the E820 map and the p2m map
>>>> info for your failing domain, please?
>>>
>>> If you use memory hotplug then maxmem is not a limit from guest kernel
>>> point of view (host still must allow that operation but it is another
>>> not related issue). The problem is that p2m must be dynamically expendable
>>> to support it. Earlier implementation supported that thing and memory
>>> hotplug worked without any issue.
>>
>> Okay, now I get it.
>>
>> The problem with the earlier p2m implementation was that it was
>> expendable to support only up to 512GB of RAM. So we need some way to
>> tell the kernel how much virtual memory it should reserve for the p2m
>> list if memory hotplug is enabled. We could:
>>
>> a) use a configurable maximum (e.g. for 512GB RAM as today)
>>
>> b) use the maximum of RAM the machine the domain is started on can ever
>>     have (what about migration then?)
>
> Memory hotplug for Xen is planned, so, this would not work.
>
>> c) use a kernel parameter specifying the maximum memory size to support
>>
>> d) a combination of some of the above possibilities
>>
>> Any thoughts? I think I'd prefer b)+c).
>
> I do not know new p2m design well so correct me if I am wrong.
>
> If you set size limit on p2m then memory hotplug does not make sense.
> Ballooning solves all your problems in simple manner. Memory hotplug
> is a solution if you would like to set up a guest with small amount
> of memory, you are not able to predict future requirements (usually
> you are not) and you are not able to restart machine at any given
> moment due to tight requirements. So, I think that p2m size, if must
> be limited, then it should be limited by platform limitations (e.g.
> number of address lines) but not artificially as you suggested.

I was talking about platform limitations, but more like "maximum of RAM
which can be plugged into the current configuration". It makes no sense
to use a limit of TBs when the system has only 1 socket with 8 possible
DIMMs.

> Additionally, I have a feeling that you are going to preallocate space
> in p2m for potential memory hotplug uses. Am I right? If yes then
> I think this is unneeded waste of memory.

I'm preallocating _virtual_ memory, not physical. There is some overhead
regarding physical memory, but this is very very small (about 4kB per
1GB of supported memory size). In case this small amount is really
critical, I could insert another optimization reducing the overhead by
another factor of 512.

> As I know new p2m is a table with MFNs. If yes, then I think we can
> potentially add more space for new MFNs at the and of that table (or

As it is virtual memory, this is already the case.

> create new larger one then copy old table contents and destroy former
> one; Hmmm... What about memory fragmentation which may prevent us
> allocating this big chunk of memory...). If this works then this
> process could be repeated without limitations (Well... We will be
> just limited by platform itself).

Rebuilding the p2m list would work, but only if enough contiguous
virtual memory is available.

One problem remains: 32 bit domains are short of virtual memory space.
Do we need memory hotplug for 32 bit domains as well?


Juergen

Re: [PATCHv1] xen/balloon: disable memory hotplug in PV guests

[David Vrabel]

On 16/03/15 10:31, Juergen Gross wrote:
> On 03/16/2015 11:03 AM, Daniel Kiper wrote:
>> On Mon, Mar 16, 2015 at 06:35:04AM +0100, Juergen Gross wrote:
>>> On 03/11/2015 04:40 PM, Boris Ostrovsky wrote:
>>>> On 03/11/2015 10:42 AM, David Vrabel wrote:
>>>>> On 10/03/15 13:35, Boris Ostrovsky wrote:
>>>>>> On 03/10/2015 07:40 AM, David Vrabel wrote:
>>>>>>> On 09/03/15 14:10, David Vrabel wrote:
>>>>>>>> Memory hotplug doesn't work with PV guests because:
>>>>>>>>
>>>>>>>>     a) The p2m cannot be expanded to cover the new sections.
>>>>>>> Broken by 054954eb051f35e74b75a566a96fe756015352c8 (xen: switch to
>>>>>>> linear virtual mapped sparse p2m list).
>>>>>>>
>>>>>>> This one would be non-trivial to fix.  We'd need a sparse set of
>>>>>>> vm_area's for the p2m or similar.
>>>>>>>
>>>>>>>>     b) add_memory() builds page tables for the new sections which
>>>>>>>> means
>>>>>>>>        the new pages must have valid p2m entries (or a BUG occurs).
>>>>>>> After some more testing this appears to be broken by:
>>>>>>>
>>>>>>> 25b884a83d487fd62c3de7ac1ab5549979188482 (x86/xen: set regions above
>>>>>>> the
>>>>>>> end of RAM as 1:1) included 3.16.
>>>>>>>
>>>>>>> This one can be trivially fixed by setting the new sections in
>>>>>>> the p2m
>>>>>>> to INVALID_P2M_ENTRY before calling add_memory().
>>>>>> Have you tried 3.17? As I said yesterday, it worked for me (with 4.4
>>>>>> Xen).
>>>>> No.  But there are three bugs that prevent it from working in 3.16+ so
>>>>> I'm really not sure how you had a working in a 3.17 PV guest.
>>>>
>>>> This is what I have:
>>>>
>>>> [build@build-mk2 linux-boris]$ ssh root@tst008 cat
>>>> /mnt/lab/bootstrap-x86_64/test_small.xm
>>>> extra="console=hvc0 debug earlyprintk=xen "
>>>> kernel="/mnt/lab/bootstrap-x86_64/vmlinuz"
>>>> ramdisk="/mnt/lab/bootstrap-x86_64/initramfs.cpio.gz"
>>>> memory=1024
>>>> maxmem = 4096
>>>> vcpus=1
>>>> maxvcpus=3
>>>> name="bootstrap-x86_64"
>>>> on_crash="preserve"
>>>> vif = [ 'mac=00:0F:4B:00:00:68, bridge=switch' ]
>>>> vnc=1
>>>> vnclisten="0.0.0.0"
>>>> disk=['phy:/dev/guests/bootstrap-x86_64,xvda,w']
>>>> [build@build-mk2 linux-boris]$ ssh root@tst008 xl create
>>>> /mnt/lab/bootstrap-x86_64/test_small.xm
>>>> Parsing config from /mnt/lab/bootstrap-x86_64/test_small.xm
>>>> [build@build-mk2 linux-boris]$ ssh root@tst008 xl list |grep
>>>> bootstrap-x86_64
>>>> bootstrap-x86_64                             2  1024     1
>>>> -b----       5.4
>>>> [build@build-mk2 linux-boris]$ ssh root@g-pvops uname -r
>>>> 3.17.0upstream
>>>> [build@build-mk2 linux-boris]$ ssh root@g-pvops dmesg|grep
>>>> paravirtualized
>>>> [    0.000000] Booting paravirtualized kernel on Xen
>>>> [build@build-mk2 linux-boris]$ ssh root@g-pvops grep MemTotal
>>>> /proc/meminfo
>>>> MemTotal:         968036 kB
>>>> [build@build-mk2 linux-boris]$ ssh root@tst008 xl mem-set
>>>> bootstrap-x86_64 2048
>>>> [build@build-mk2 linux-boris]$ ssh root@tst008 xl list |grep
>>>> bootstrap-x86_64
>>>> bootstrap-x86_64                             2  2048     1
>>>> -b----       5.7
>>>> [build@build-mk2 linux-boris]$ ssh root@g-pvops grep MemTotal
>>>> /proc/meminfo
>>>> MemTotal:        2016612 kB
>>>> [build@build-mk2 linux-boris]$
>>>>
>>>>
>>>>>
>>>>> Regardless, it definitely doesn't work now because of the linear p2m
>>>>> change.  What do you want to do about this?
>>>>
>>>> Since backing out p2m changes is not an option I guess your patch is
>>>> the
>>>> only short-term alternative.
>>>>
>>>> But this still looks like a regression so perhaps Juergen can take a
>>>> look to see how it can be fixed.
>>>
>>> Hmm, the p2m list is allocated for the maximum memory size of the domain
>>> which is obtained from the hypervisor. In case of Dom0 it is read via
>>> XENMEM_maximum_reservation, for a domU it is based on the E820 memory
>>> map read via XENMEM_memory_map.
>>>
>>> I just tested it with a 4.0-rc1 domU kernel with 512MB initial memory
>>> and 4GB of maxmem. The E820 map looked like this:
>>>
>>> [    0.000000] Xen: [mem 0x0000000000000000-0x000000000009ffff] usable
>>> [    0.000000] Xen: [mem 0x00000000000a0000-0x00000000000fffff] reserved
>>> [    0.000000] Xen: [mem 0x0000000000100000-0x00000000ffffffff] usable
>>>
>>> So the complete 4GB were included, like they should. The resulting p2m
>>> list is allocated in the needed size:
>>>
>>> [    0.000000] p2m virtual area at ffffc90000000000, size is 800000
>>>
>>> So what is your problem here? Can you post the E820 map and the p2m map
>>> info for your failing domain, please?
>>
>> If you use memory hotplug then maxmem is not a limit from guest kernel
>> point of view (host still must allow that operation but it is another
>> not related issue). The problem is that p2m must be dynamically
>> expendable
>> to support it. Earlier implementation supported that thing and memory
>> hotplug worked without any issue.
> 
> Okay, now I get it.
> 
> The problem with the earlier p2m implementation was that it was
> expendable to support only up to 512GB of RAM. So we need some way to
> tell the kernel how much virtual memory it should reserve for the p2m
> list if memory hotplug is enabled. We could:
> 
> a) use a configurable maximum (e.g. for 512GB RAM as today)

I would set the p2m virtual area to cover up to 512 GB (needs 1 GB of
virt space) for a 64-bit guest and up to 64 GB (needs 64 MB of virt
space) for a 32-bit guest.

This fixes the regression with minimal complexity and reasonable overheads.

David

Re: [PATCHv1] xen/balloon: disable memory hotplug in PV guests

[Juergen Gross]

On 03/18/2015 11:36 AM, David Vrabel wrote:
> On 16/03/15 10:31, Juergen Gross wrote:
>> On 03/16/2015 11:03 AM, Daniel Kiper wrote:
>>> On Mon, Mar 16, 2015 at 06:35:04AM +0100, Juergen Gross wrote:
>>>> On 03/11/2015 04:40 PM, Boris Ostrovsky wrote:
>>>>> On 03/11/2015 10:42 AM, David Vrabel wrote:
>>>>>> On 10/03/15 13:35, Boris Ostrovsky wrote:
>>>>>>> On 03/10/2015 07:40 AM, David Vrabel wrote:
>>>>>>>> On 09/03/15 14:10, David Vrabel wrote:
>>>>>>>>> Memory hotplug doesn't work with PV guests because:
>>>>>>>>>
>>>>>>>>>      a) The p2m cannot be expanded to cover the new sections.
>>>>>>>> Broken by 054954eb051f35e74b75a566a96fe756015352c8 (xen: switch to
>>>>>>>> linear virtual mapped sparse p2m list).
>>>>>>>>
>>>>>>>> This one would be non-trivial to fix.  We'd need a sparse set of
>>>>>>>> vm_area's for the p2m or similar.
>>>>>>>>
>>>>>>>>>      b) add_memory() builds page tables for the new sections which
>>>>>>>>> means
>>>>>>>>>         the new pages must have valid p2m entries (or a BUG occurs).
>>>>>>>> After some more testing this appears to be broken by:
>>>>>>>>
>>>>>>>> 25b884a83d487fd62c3de7ac1ab5549979188482 (x86/xen: set regions above
>>>>>>>> the
>>>>>>>> end of RAM as 1:1) included 3.16.
>>>>>>>>
>>>>>>>> This one can be trivially fixed by setting the new sections in
>>>>>>>> the p2m
>>>>>>>> to INVALID_P2M_ENTRY before calling add_memory().
>>>>>>> Have you tried 3.17? As I said yesterday, it worked for me (with 4.4
>>>>>>> Xen).
>>>>>> No.  But there are three bugs that prevent it from working in 3.16+ so
>>>>>> I'm really not sure how you had a working in a 3.17 PV guest.
>>>>>
>>>>> This is what I have:
>>>>>
>>>>> [build@build-mk2 linux-boris]$ ssh root@tst008 cat
>>>>> /mnt/lab/bootstrap-x86_64/test_small.xm
>>>>> extra="console=hvc0 debug earlyprintk=xen "
>>>>> kernel="/mnt/lab/bootstrap-x86_64/vmlinuz"
>>>>> ramdisk="/mnt/lab/bootstrap-x86_64/initramfs.cpio.gz"
>>>>> memory=1024
>>>>> maxmem = 4096
>>>>> vcpus=1
>>>>> maxvcpus=3
>>>>> name="bootstrap-x86_64"
>>>>> on_crash="preserve"
>>>>> vif = [ 'mac=00:0F:4B:00:00:68, bridge=switch' ]
>>>>> vnc=1
>>>>> vnclisten="0.0.0.0"
>>>>> disk=['phy:/dev/guests/bootstrap-x86_64,xvda,w']
>>>>> [build@build-mk2 linux-boris]$ ssh root@tst008 xl create
>>>>> /mnt/lab/bootstrap-x86_64/test_small.xm
>>>>> Parsing config from /mnt/lab/bootstrap-x86_64/test_small.xm
>>>>> [build@build-mk2 linux-boris]$ ssh root@tst008 xl list |grep
>>>>> bootstrap-x86_64
>>>>> bootstrap-x86_64                             2  1024     1
>>>>> -b----       5.4
>>>>> [build@build-mk2 linux-boris]$ ssh root@g-pvops uname -r
>>>>> 3.17.0upstream
>>>>> [build@build-mk2 linux-boris]$ ssh root@g-pvops dmesg|grep
>>>>> paravirtualized
>>>>> [    0.000000] Booting paravirtualized kernel on Xen
>>>>> [build@build-mk2 linux-boris]$ ssh root@g-pvops grep MemTotal
>>>>> /proc/meminfo
>>>>> MemTotal:         968036 kB
>>>>> [build@build-mk2 linux-boris]$ ssh root@tst008 xl mem-set
>>>>> bootstrap-x86_64 2048
>>>>> [build@build-mk2 linux-boris]$ ssh root@tst008 xl list |grep
>>>>> bootstrap-x86_64
>>>>> bootstrap-x86_64                             2  2048     1
>>>>> -b----       5.7
>>>>> [build@build-mk2 linux-boris]$ ssh root@g-pvops grep MemTotal
>>>>> /proc/meminfo
>>>>> MemTotal:        2016612 kB
>>>>> [build@build-mk2 linux-boris]$
>>>>>
>>>>>
>>>>>>
>>>>>> Regardless, it definitely doesn't work now because of the linear p2m
>>>>>> change.  What do you want to do about this?
>>>>>
>>>>> Since backing out p2m changes is not an option I guess your patch is
>>>>> the
>>>>> only short-term alternative.
>>>>>
>>>>> But this still looks like a regression so perhaps Juergen can take a
>>>>> look to see how it can be fixed.
>>>>
>>>> Hmm, the p2m list is allocated for the maximum memory size of the domain
>>>> which is obtained from the hypervisor. In case of Dom0 it is read via
>>>> XENMEM_maximum_reservation, for a domU it is based on the E820 memory
>>>> map read via XENMEM_memory_map.
>>>>
>>>> I just tested it with a 4.0-rc1 domU kernel with 512MB initial memory
>>>> and 4GB of maxmem. The E820 map looked like this:
>>>>
>>>> [    0.000000] Xen: [mem 0x0000000000000000-0x000000000009ffff] usable
>>>> [    0.000000] Xen: [mem 0x00000000000a0000-0x00000000000fffff] reserved
>>>> [    0.000000] Xen: [mem 0x0000000000100000-0x00000000ffffffff] usable
>>>>
>>>> So the complete 4GB were included, like they should. The resulting p2m
>>>> list is allocated in the needed size:
>>>>
>>>> [    0.000000] p2m virtual area at ffffc90000000000, size is 800000
>>>>
>>>> So what is your problem here? Can you post the E820 map and the p2m map
>>>> info for your failing domain, please?
>>>
>>> If you use memory hotplug then maxmem is not a limit from guest kernel
>>> point of view (host still must allow that operation but it is another
>>> not related issue). The problem is that p2m must be dynamically
>>> expendable
>>> to support it. Earlier implementation supported that thing and memory
>>> hotplug worked without any issue.
>>
>> Okay, now I get it.
>>
>> The problem with the earlier p2m implementation was that it was
>> expendable to support only up to 512GB of RAM. So we need some way to
>> tell the kernel how much virtual memory it should reserve for the p2m
>> list if memory hotplug is enabled. We could:
>>
>> a) use a configurable maximum (e.g. for 512GB RAM as today)
>
> I would set the p2m virtual area to cover up to 512 GB (needs 1 GB of
> virt space) for a 64-bit guest and up to 64 GB (needs 64 MB of virt
> space) for a 32-bit guest.

Are 64 GB for 32 bit guests a sensible default? This will need more than
10% of the available virtual kernel space (taking fixmap etc. into
account). And a 64 GB sized 32 bit domain is hardly usable (you have to
play dirty tricks to get it even running).

I'd rather use a default of 4 GB which can be changed via a Kconfig
option. For 64 bits the default of 512 GB is okay, but should be
configurable as well.

Juergen

Re: [PATCHv1] xen/balloon: disable memory hotplug in PV guests

[David Vrabel]

On 18/03/15 13:57, Juergen Gross wrote:
> On 03/18/2015 11:36 AM, David Vrabel wrote:
>> On 16/03/15 10:31, Juergen Gross wrote:
>>> On 03/16/2015 11:03 AM, Daniel Kiper wrote:
>>>> On Mon, Mar 16, 2015 at 06:35:04AM +0100, Juergen Gross wrote:
>>>>> On 03/11/2015 04:40 PM, Boris Ostrovsky wrote:
>>>>>> On 03/11/2015 10:42 AM, David Vrabel wrote:
>>>>>>> On 10/03/15 13:35, Boris Ostrovsky wrote:
>>>>>>>> On 03/10/2015 07:40 AM, David Vrabel wrote:
>>>>>>>>> On 09/03/15 14:10, David Vrabel wrote:
>>>>>>>>>> Memory hotplug doesn't work with PV guests because:
>>>>>>>>>>
>>>>>>>>>>      a) The p2m cannot be expanded to cover the new sections.
>>>>>>>>> Broken by 054954eb051f35e74b75a566a96fe756015352c8 (xen: switch to
>>>>>>>>> linear virtual mapped sparse p2m list).
>>>>>>>>>
>>>>>>>>> This one would be non-trivial to fix.  We'd need a sparse set of
>>>>>>>>> vm_area's for the p2m or similar.
>>>>>>>>>
>>>>>>>>>>      b) add_memory() builds page tables for the new sections
>>>>>>>>>> which
>>>>>>>>>> means
>>>>>>>>>>         the new pages must have valid p2m entries (or a BUG
>>>>>>>>>> occurs).
>>>>>>>>> After some more testing this appears to be broken by:
>>>>>>>>>
>>>>>>>>> 25b884a83d487fd62c3de7ac1ab5549979188482 (x86/xen: set regions
>>>>>>>>> above
>>>>>>>>> the
>>>>>>>>> end of RAM as 1:1) included 3.16.
>>>>>>>>>
>>>>>>>>> This one can be trivially fixed by setting the new sections in
>>>>>>>>> the p2m
>>>>>>>>> to INVALID_P2M_ENTRY before calling add_memory().
>>>>>>>> Have you tried 3.17? As I said yesterday, it worked for me (with
>>>>>>>> 4.4
>>>>>>>> Xen).
>>>>>>> No.  But there are three bugs that prevent it from working in
>>>>>>> 3.16+ so
>>>>>>> I'm really not sure how you had a working in a 3.17 PV guest.
>>>>>>
>>>>>> This is what I have:
>>>>>>
>>>>>> [build@build-mk2 linux-boris]$ ssh root@tst008 cat
>>>>>> /mnt/lab/bootstrap-x86_64/test_small.xm
>>>>>> extra="console=hvc0 debug earlyprintk=xen "
>>>>>> kernel="/mnt/lab/bootstrap-x86_64/vmlinuz"
>>>>>> ramdisk="/mnt/lab/bootstrap-x86_64/initramfs.cpio.gz"
>>>>>> memory=1024
>>>>>> maxmem = 4096
>>>>>> vcpus=1
>>>>>> maxvcpus=3
>>>>>> name="bootstrap-x86_64"
>>>>>> on_crash="preserve"
>>>>>> vif = [ 'mac=00:0F:4B:00:00:68, bridge=switch' ]
>>>>>> vnc=1
>>>>>> vnclisten="0.0.0.0"
>>>>>> disk=['phy:/dev/guests/bootstrap-x86_64,xvda,w']
>>>>>> [build@build-mk2 linux-boris]$ ssh root@tst008 xl create
>>>>>> /mnt/lab/bootstrap-x86_64/test_small.xm
>>>>>> Parsing config from /mnt/lab/bootstrap-x86_64/test_small.xm
>>>>>> [build@build-mk2 linux-boris]$ ssh root@tst008 xl list |grep
>>>>>> bootstrap-x86_64
>>>>>> bootstrap-x86_64                             2  1024     1
>>>>>> -b----       5.4
>>>>>> [build@build-mk2 linux-boris]$ ssh root@g-pvops uname -r
>>>>>> 3.17.0upstream
>>>>>> [build@build-mk2 linux-boris]$ ssh root@g-pvops dmesg|grep
>>>>>> paravirtualized
>>>>>> [    0.000000] Booting paravirtualized kernel on Xen
>>>>>> [build@build-mk2 linux-boris]$ ssh root@g-pvops grep MemTotal
>>>>>> /proc/meminfo
>>>>>> MemTotal:         968036 kB
>>>>>> [build@build-mk2 linux-boris]$ ssh root@tst008 xl mem-set
>>>>>> bootstrap-x86_64 2048
>>>>>> [build@build-mk2 linux-boris]$ ssh root@tst008 xl list |grep
>>>>>> bootstrap-x86_64
>>>>>> bootstrap-x86_64                             2  2048     1
>>>>>> -b----       5.7
>>>>>> [build@build-mk2 linux-boris]$ ssh root@g-pvops grep MemTotal
>>>>>> /proc/meminfo
>>>>>> MemTotal:        2016612 kB
>>>>>> [build@build-mk2 linux-boris]$
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Regardless, it definitely doesn't work now because of the linear p2m
>>>>>>> change.  What do you want to do about this?
>>>>>>
>>>>>> Since backing out p2m changes is not an option I guess your patch is
>>>>>> the
>>>>>> only short-term alternative.
>>>>>>
>>>>>> But this still looks like a regression so perhaps Juergen can take a
>>>>>> look to see how it can be fixed.
>>>>>
>>>>> Hmm, the p2m list is allocated for the maximum memory size of the
>>>>> domain
>>>>> which is obtained from the hypervisor. In case of Dom0 it is read via
>>>>> XENMEM_maximum_reservation, for a domU it is based on the E820 memory
>>>>> map read via XENMEM_memory_map.
>>>>>
>>>>> I just tested it with a 4.0-rc1 domU kernel with 512MB initial memory
>>>>> and 4GB of maxmem. The E820 map looked like this:
>>>>>
>>>>> [    0.000000] Xen: [mem 0x0000000000000000-0x000000000009ffff] usable
>>>>> [    0.000000] Xen: [mem 0x00000000000a0000-0x00000000000fffff]
>>>>> reserved
>>>>> [    0.000000] Xen: [mem 0x0000000000100000-0x00000000ffffffff] usable
>>>>>
>>>>> So the complete 4GB were included, like they should. The resulting p2m
>>>>> list is allocated in the needed size:
>>>>>
>>>>> [    0.000000] p2m virtual area at ffffc90000000000, size is 800000
>>>>>
>>>>> So what is your problem here? Can you post the E820 map and the p2m
>>>>> map
>>>>> info for your failing domain, please?
>>>>
>>>> If you use memory hotplug then maxmem is not a limit from guest kernel
>>>> point of view (host still must allow that operation but it is another
>>>> not related issue). The problem is that p2m must be dynamically
>>>> expendable
>>>> to support it. Earlier implementation supported that thing and memory
>>>> hotplug worked without any issue.
>>>
>>> Okay, now I get it.
>>>
>>> The problem with the earlier p2m implementation was that it was
>>> expendable to support only up to 512GB of RAM. So we need some way to
>>> tell the kernel how much virtual memory it should reserve for the p2m
>>> list if memory hotplug is enabled. We could:
>>>
>>> a) use a configurable maximum (e.g. for 512GB RAM as today)
>>
>> I would set the p2m virtual area to cover up to 512 GB (needs 1 GB of
>> virt space) for a 64-bit guest and up to 64 GB (needs 64 MB of virt
>> space) for a 32-bit guest.
> 
> Are 64 GB for 32 bit guests a sensible default? This will need more than
> 10% of the available virtual kernel space (taking fixmap etc. into
> account). And a 64 GB sized 32 bit domain is hardly usable (you have to
> play dirty tricks to get it even running).
> 
> I'd rather use a default of 4 GB which can be changed via a Kconfig
> option. For 64 bits the default of 512 GB is okay, but should be
> configurable as well.

Ok.

David

Re: [PATCHv1] xen/balloon: disable memory hotplug in PV guests

[Daniel Kiper]

On Wed, Mar 18, 2015 at 01:59:58PM +0000, David Vrabel wrote:
> On 18/03/15 13:57, Juergen Gross wrote:
> > On 03/18/2015 11:36 AM, David Vrabel wrote:
> >> On 16/03/15 10:31, Juergen Gross wrote:
> >>> On 03/16/2015 11:03 AM, Daniel Kiper wrote:
> >>>> On Mon, Mar 16, 2015 at 06:35:04AM +0100, Juergen Gross wrote:
> >>>>> On 03/11/2015 04:40 PM, Boris Ostrovsky wrote:
> >>>>>> On 03/11/2015 10:42 AM, David Vrabel wrote:
> >>>>>>> On 10/03/15 13:35, Boris Ostrovsky wrote:
> >>>>>>>> On 03/10/2015 07:40 AM, David Vrabel wrote:
> >>>>>>>>> On 09/03/15 14:10, David Vrabel wrote:
> >>>>>>>>>> Memory hotplug doesn't work with PV guests because:
> >>>>>>>>>>
> >>>>>>>>>>      a) The p2m cannot be expanded to cover the new sections.
> >>>>>>>>> Broken by 054954eb051f35e74b75a566a96fe756015352c8 (xen: switch to
> >>>>>>>>> linear virtual mapped sparse p2m list).
> >>>>>>>>>
> >>>>>>>>> This one would be non-trivial to fix.  We'd need a sparse set of
> >>>>>>>>> vm_area's for the p2m or similar.
> >>>>>>>>>
> >>>>>>>>>>      b) add_memory() builds page tables for the new sections
> >>>>>>>>>> which
> >>>>>>>>>> means
> >>>>>>>>>>         the new pages must have valid p2m entries (or a BUG
> >>>>>>>>>> occurs).
> >>>>>>>>> After some more testing this appears to be broken by:
> >>>>>>>>>
> >>>>>>>>> 25b884a83d487fd62c3de7ac1ab5549979188482 (x86/xen: set regions
> >>>>>>>>> above
> >>>>>>>>> the
> >>>>>>>>> end of RAM as 1:1) included 3.16.
> >>>>>>>>>
> >>>>>>>>> This one can be trivially fixed by setting the new sections in
> >>>>>>>>> the p2m
> >>>>>>>>> to INVALID_P2M_ENTRY before calling add_memory().
> >>>>>>>> Have you tried 3.17? As I said yesterday, it worked for me (with
> >>>>>>>> 4.4
> >>>>>>>> Xen).
> >>>>>>> No.  But there are three bugs that prevent it from working in
> >>>>>>> 3.16+ so
> >>>>>>> I'm really not sure how you had a working in a 3.17 PV guest.
> >>>>>>
> >>>>>> This is what I have:
> >>>>>>
> >>>>>> [build@build-mk2 linux-boris]$ ssh root@tst008 cat
> >>>>>> /mnt/lab/bootstrap-x86_64/test_small.xm
> >>>>>> extra="console=hvc0 debug earlyprintk=xen "
> >>>>>> kernel="/mnt/lab/bootstrap-x86_64/vmlinuz"
> >>>>>> ramdisk="/mnt/lab/bootstrap-x86_64/initramfs.cpio.gz"
> >>>>>> memory=1024
> >>>>>> maxmem = 4096
> >>>>>> vcpus=1
> >>>>>> maxvcpus=3
> >>>>>> name="bootstrap-x86_64"
> >>>>>> on_crash="preserve"
> >>>>>> vif = [ 'mac=00:0F:4B:00:00:68, bridge=switch' ]
> >>>>>> vnc=1
> >>>>>> vnclisten="0.0.0.0"
> >>>>>> disk=['phy:/dev/guests/bootstrap-x86_64,xvda,w']
> >>>>>> [build@build-mk2 linux-boris]$ ssh root@tst008 xl create
> >>>>>> /mnt/lab/bootstrap-x86_64/test_small.xm
> >>>>>> Parsing config from /mnt/lab/bootstrap-x86_64/test_small.xm
> >>>>>> [build@build-mk2 linux-boris]$ ssh root@tst008 xl list |grep
> >>>>>> bootstrap-x86_64
> >>>>>> bootstrap-x86_64                             2  1024     1
> >>>>>> -b----       5.4
> >>>>>> [build@build-mk2 linux-boris]$ ssh root@g-pvops uname -r
> >>>>>> 3.17.0upstream
> >>>>>> [build@build-mk2 linux-boris]$ ssh root@g-pvops dmesg|grep
> >>>>>> paravirtualized
> >>>>>> [    0.000000] Booting paravirtualized kernel on Xen
> >>>>>> [build@build-mk2 linux-boris]$ ssh root@g-pvops grep MemTotal
> >>>>>> /proc/meminfo
> >>>>>> MemTotal:         968036 kB
> >>>>>> [build@build-mk2 linux-boris]$ ssh root@tst008 xl mem-set
> >>>>>> bootstrap-x86_64 2048
> >>>>>> [build@build-mk2 linux-boris]$ ssh root@tst008 xl list |grep
> >>>>>> bootstrap-x86_64
> >>>>>> bootstrap-x86_64                             2  2048     1
> >>>>>> -b----       5.7
> >>>>>> [build@build-mk2 linux-boris]$ ssh root@g-pvops grep MemTotal
> >>>>>> /proc/meminfo
> >>>>>> MemTotal:        2016612 kB
> >>>>>> [build@build-mk2 linux-boris]$
> >>>>>>
> >>>>>>
> >>>>>>>
> >>>>>>> Regardless, it definitely doesn't work now because of the linear p2m
> >>>>>>> change.  What do you want to do about this?
> >>>>>>
> >>>>>> Since backing out p2m changes is not an option I guess your patch is
> >>>>>> the
> >>>>>> only short-term alternative.
> >>>>>>
> >>>>>> But this still looks like a regression so perhaps Juergen can take a
> >>>>>> look to see how it can be fixed.
> >>>>>
> >>>>> Hmm, the p2m list is allocated for the maximum memory size of the
> >>>>> domain
> >>>>> which is obtained from the hypervisor. In case of Dom0 it is read via
> >>>>> XENMEM_maximum_reservation, for a domU it is based on the E820 memory
> >>>>> map read via XENMEM_memory_map.
> >>>>>
> >>>>> I just tested it with a 4.0-rc1 domU kernel with 512MB initial memory
> >>>>> and 4GB of maxmem. The E820 map looked like this:
> >>>>>
> >>>>> [    0.000000] Xen: [mem 0x0000000000000000-0x000000000009ffff] usable
> >>>>> [    0.000000] Xen: [mem 0x00000000000a0000-0x00000000000fffff]
> >>>>> reserved
> >>>>> [    0.000000] Xen: [mem 0x0000000000100000-0x00000000ffffffff] usable
> >>>>>
> >>>>> So the complete 4GB were included, like they should. The resulting p2m
> >>>>> list is allocated in the needed size:
> >>>>>
> >>>>> [    0.000000] p2m virtual area at ffffc90000000000, size is 800000
> >>>>>
> >>>>> So what is your problem here? Can you post the E820 map and the p2m
> >>>>> map
> >>>>> info for your failing domain, please?
> >>>>
> >>>> If you use memory hotplug then maxmem is not a limit from guest kernel
> >>>> point of view (host still must allow that operation but it is another
> >>>> not related issue). The problem is that p2m must be dynamically
> >>>> expendable
> >>>> to support it. Earlier implementation supported that thing and memory
> >>>> hotplug worked without any issue.
> >>>
> >>> Okay, now I get it.
> >>>
> >>> The problem with the earlier p2m implementation was that it was
> >>> expendable to support only up to 512GB of RAM. So we need some way to
> >>> tell the kernel how much virtual memory it should reserve for the p2m
> >>> list if memory hotplug is enabled. We could:
> >>>
> >>> a) use a configurable maximum (e.g. for 512GB RAM as today)
> >>
> >> I would set the p2m virtual area to cover up to 512 GB (needs 1 GB of
> >> virt space) for a 64-bit guest and up to 64 GB (needs 64 MB of virt
> >> space) for a 32-bit guest.
> >
> > Are 64 GB for 32 bit guests a sensible default? This will need more than
> > 10% of the available virtual kernel space (taking fixmap etc. into
> > account). And a 64 GB sized 32 bit domain is hardly usable (you have to
> > play dirty tricks to get it even running).
> >
> > I'd rather use a default of 4 GB which can be changed via a Kconfig
> > option. For 64 bits the default of 512 GB is okay, but should be
> > configurable as well.
>
> Ok.

I have checked new p2m code and I think that this is reasonable solution too.

David

Re: [PATCHv1] xen/balloon: disable memory hotplug in PV guests

[Juergen Gross]

On 03/18/2015 04:14 PM, Daniel Kiper wrote:
> On Wed, Mar 18, 2015 at 01:59:58PM +0000, David Vrabel wrote:
>> On 18/03/15 13:57, Juergen Gross wrote:
>>> On 03/18/2015 11:36 AM, David Vrabel wrote:
>>>> On 16/03/15 10:31, Juergen Gross wrote:
>>>>> On 03/16/2015 11:03 AM, Daniel Kiper wrote:
>>>>>> On Mon, Mar 16, 2015 at 06:35:04AM +0100, Juergen Gross wrote:
>>>>>>> On 03/11/2015 04:40 PM, Boris Ostrovsky wrote:
>>>>>>>> On 03/11/2015 10:42 AM, David Vrabel wrote:
>>>>>>>>> On 10/03/15 13:35, Boris Ostrovsky wrote:
>>>>>>>>>> On 03/10/2015 07:40 AM, David Vrabel wrote:
>>>>>>>>>>> On 09/03/15 14:10, David Vrabel wrote:
>>>>>>>>>>>> Memory hotplug doesn't work with PV guests because:
>>>>>>>>>>>>
>>>>>>>>>>>>       a) The p2m cannot be expanded to cover the new sections.
>>>>>>>>>>> Broken by 054954eb051f35e74b75a566a96fe756015352c8 (xen: switch to
>>>>>>>>>>> linear virtual mapped sparse p2m list).
>>>>>>>>>>>
>>>>>>>>>>> This one would be non-trivial to fix.  We'd need a sparse set of
>>>>>>>>>>> vm_area's for the p2m or similar.
>>>>>>>>>>>
>>>>>>>>>>>>       b) add_memory() builds page tables for the new sections
>>>>>>>>>>>> which
>>>>>>>>>>>> means
>>>>>>>>>>>>          the new pages must have valid p2m entries (or a BUG
>>>>>>>>>>>> occurs).
>>>>>>>>>>> After some more testing this appears to be broken by:
>>>>>>>>>>>
>>>>>>>>>>> 25b884a83d487fd62c3de7ac1ab5549979188482 (x86/xen: set regions
>>>>>>>>>>> above
>>>>>>>>>>> the
>>>>>>>>>>> end of RAM as 1:1) included 3.16.
>>>>>>>>>>>
>>>>>>>>>>> This one can be trivially fixed by setting the new sections in
>>>>>>>>>>> the p2m
>>>>>>>>>>> to INVALID_P2M_ENTRY before calling add_memory().
>>>>>>>>>> Have you tried 3.17? As I said yesterday, it worked for me (with
>>>>>>>>>> 4.4
>>>>>>>>>> Xen).
>>>>>>>>> No.  But there are three bugs that prevent it from working in
>>>>>>>>> 3.16+ so
>>>>>>>>> I'm really not sure how you had a working in a 3.17 PV guest.
>>>>>>>>
>>>>>>>> This is what I have:
>>>>>>>>
>>>>>>>> [build@build-mk2 linux-boris]$ ssh root@tst008 cat
>>>>>>>> /mnt/lab/bootstrap-x86_64/test_small.xm
>>>>>>>> extra="console=hvc0 debug earlyprintk=xen "
>>>>>>>> kernel="/mnt/lab/bootstrap-x86_64/vmlinuz"
>>>>>>>> ramdisk="/mnt/lab/bootstrap-x86_64/initramfs.cpio.gz"
>>>>>>>> memory=1024
>>>>>>>> maxmem = 4096
>>>>>>>> vcpus=1
>>>>>>>> maxvcpus=3
>>>>>>>> name="bootstrap-x86_64"
>>>>>>>> on_crash="preserve"
>>>>>>>> vif = [ 'mac=00:0F:4B:00:00:68, bridge=switch' ]
>>>>>>>> vnc=1
>>>>>>>> vnclisten="0.0.0.0"
>>>>>>>> disk=['phy:/dev/guests/bootstrap-x86_64,xvda,w']
>>>>>>>> [build@build-mk2 linux-boris]$ ssh root@tst008 xl create
>>>>>>>> /mnt/lab/bootstrap-x86_64/test_small.xm
>>>>>>>> Parsing config from /mnt/lab/bootstrap-x86_64/test_small.xm
>>>>>>>> [build@build-mk2 linux-boris]$ ssh root@tst008 xl list |grep
>>>>>>>> bootstrap-x86_64
>>>>>>>> bootstrap-x86_64                             2  1024     1
>>>>>>>> -b----       5.4
>>>>>>>> [build@build-mk2 linux-boris]$ ssh root@g-pvops uname -r
>>>>>>>> 3.17.0upstream
>>>>>>>> [build@build-mk2 linux-boris]$ ssh root@g-pvops dmesg|grep
>>>>>>>> paravirtualized
>>>>>>>> [    0.000000] Booting paravirtualized kernel on Xen
>>>>>>>> [build@build-mk2 linux-boris]$ ssh root@g-pvops grep MemTotal
>>>>>>>> /proc/meminfo
>>>>>>>> MemTotal:         968036 kB
>>>>>>>> [build@build-mk2 linux-boris]$ ssh root@tst008 xl mem-set
>>>>>>>> bootstrap-x86_64 2048
>>>>>>>> [build@build-mk2 linux-boris]$ ssh root@tst008 xl list |grep
>>>>>>>> bootstrap-x86_64
>>>>>>>> bootstrap-x86_64                             2  2048     1
>>>>>>>> -b----       5.7
>>>>>>>> [build@build-mk2 linux-boris]$ ssh root@g-pvops grep MemTotal
>>>>>>>> /proc/meminfo
>>>>>>>> MemTotal:        2016612 kB
>>>>>>>> [build@build-mk2 linux-boris]$
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Regardless, it definitely doesn't work now because of the linear p2m
>>>>>>>>> change.  What do you want to do about this?
>>>>>>>>
>>>>>>>> Since backing out p2m changes is not an option I guess your patch is
>>>>>>>> the
>>>>>>>> only short-term alternative.
>>>>>>>>
>>>>>>>> But this still looks like a regression so perhaps Juergen can take a
>>>>>>>> look to see how it can be fixed.
>>>>>>>
>>>>>>> Hmm, the p2m list is allocated for the maximum memory size of the
>>>>>>> domain
>>>>>>> which is obtained from the hypervisor. In case of Dom0 it is read via
>>>>>>> XENMEM_maximum_reservation, for a domU it is based on the E820 memory
>>>>>>> map read via XENMEM_memory_map.
>>>>>>>
>>>>>>> I just tested it with a 4.0-rc1 domU kernel with 512MB initial memory
>>>>>>> and 4GB of maxmem. The E820 map looked like this:
>>>>>>>
>>>>>>> [    0.000000] Xen: [mem 0x0000000000000000-0x000000000009ffff] usable
>>>>>>> [    0.000000] Xen: [mem 0x00000000000a0000-0x00000000000fffff]
>>>>>>> reserved
>>>>>>> [    0.000000] Xen: [mem 0x0000000000100000-0x00000000ffffffff] usable
>>>>>>>
>>>>>>> So the complete 4GB were included, like they should. The resulting p2m
>>>>>>> list is allocated in the needed size:
>>>>>>>
>>>>>>> [    0.000000] p2m virtual area at ffffc90000000000, size is 800000
>>>>>>>
>>>>>>> So what is your problem here? Can you post the E820 map and the p2m
>>>>>>> map
>>>>>>> info for your failing domain, please?
>>>>>>
>>>>>> If you use memory hotplug then maxmem is not a limit from guest kernel
>>>>>> point of view (host still must allow that operation but it is another
>>>>>> not related issue). The problem is that p2m must be dynamically
>>>>>> expendable
>>>>>> to support it. Earlier implementation supported that thing and memory
>>>>>> hotplug worked without any issue.
>>>>>
>>>>> Okay, now I get it.
>>>>>
>>>>> The problem with the earlier p2m implementation was that it was
>>>>> expendable to support only up to 512GB of RAM. So we need some way to
>>>>> tell the kernel how much virtual memory it should reserve for the p2m
>>>>> list if memory hotplug is enabled. We could:
>>>>>
>>>>> a) use a configurable maximum (e.g. for 512GB RAM as today)
>>>>
>>>> I would set the p2m virtual area to cover up to 512 GB (needs 1 GB of
>>>> virt space) for a 64-bit guest and up to 64 GB (needs 64 MB of virt
>>>> space) for a 32-bit guest.
>>>
>>> Are 64 GB for 32 bit guests a sensible default? This will need more than
>>> 10% of the available virtual kernel space (taking fixmap etc. into
>>> account). And a 64 GB sized 32 bit domain is hardly usable (you have to
>>> play dirty tricks to get it even running).
>>>
>>> I'd rather use a default of 4 GB which can be changed via a Kconfig
>>> option. For 64 bits the default of 512 GB is okay, but should be
>>> configurable as well.
>>
>> Ok.
>
> I have checked new p2m code and I think that this is reasonable solution too.

Do I need any patches for xl to be able to test this? I did:

xl mem-max 2 4096
xl mem-set 2 4096

and get:

libxl: error: libxl.c:4779:libxl_set_memory_target: memory_dynamic_max 
must be less than or equal to memory_static_max

xl list -l shows:

...
     {
         "domid": 2,
         "config": {
             "c_info": {
                 "type": "pv",
                 "name": "sles11",
                 "uuid": "c53944f1-1607-e367-278a-c7980b6cfdd0",
                 "run_hotplug_scripts": "True"
             },
             "b_info": {
                 "max_vcpus": 1,
                 "avail_vcpus": [
                     0
                 ],
                 "numa_placement": "True",
                 "max_memkb": 524288,
                 "target_memkb": 524288,
                 "video_memkb": 0,
                 "shadow_memkb": 5120,
                 "localtime": "False",
                 "disable_migrate": "False",
                 "blkdev_start": "xvda",
                 "device_model_version": "qemu_xen",
                 "device_model_stubdomain": "False",
                 "sched_params": {

                 },
...

which seems to reflect only the parameters from starting the domU:

name="sles11"
description="None"
uuid="c53944f1-1607-e367-278a-c7980b6cfdd0"
memory=512
maxmem=512
vcpus=1
on_poweroff="destroy"
on_reboot="restart"
on_crash="destroy"
localtime=0
keymap="de"
builder="linux"
bootloader="/usr/bin/pygrub"
bootargs=""
extra="xencons=tty "
disk=[ 'file:/home/sles11-2,xvda,w', ]
vif=[ 'mac=00:16:3e:06:a7:21,bridge=br0', ]

> David

I thought your name was Daniel? ;-)


Juergen

Re: [PATCHv1] xen/balloon: disable memory hotplug in PV guests

[Daniel Kiper]

On Thu, Mar 19, 2015 at 10:55:37AM +0100, Juergen Gross wrote:
> On 03/18/2015 04:14 PM, Daniel Kiper wrote:

[...]

> >I have checked new p2m code and I think that this is reasonable solution too.
>
> Do I need any patches for xl to be able to test this? I did:
>
> xl mem-max 2 4096
> xl mem-set 2 4096

Yep. I started work on it but I was not able to finish it.
Here you can find some old patches for this issue:

http://lists.xen.org/archives/html/xen-devel/2013-04/msg03072.html

Bob and I will work on it sooner or later (probably a bit later).

Hmmm... Boris, David, how did you manage to get this memory hotplug
stuff working. Did you use xm instead of xl?

> >David
>
> I thought your name was Daniel? ;-)

Ugh... Copy-paste mistake... :-)))

Daniel

Re: [PATCHv1] xen/balloon: disable memory hotplug in PV guests

[Juergen Gross]

[-- Attachment #1: Type: text/plain, Size: 1009 bytes --]

On 03/19/2015 12:34 PM, Daniel Kiper wrote:
> On Thu, Mar 19, 2015 at 10:55:37AM +0100, Juergen Gross wrote:
>> On 03/18/2015 04:14 PM, Daniel Kiper wrote:
>
> [...]
>
>>> I have checked new p2m code and I think that this is reasonable solution too.
>>
>> Do I need any patches for xl to be able to test this? I did:
>>
>> xl mem-max 2 4096
>> xl mem-set 2 4096
>
> Yep. I started work on it but I was not able to finish it.
> Here you can find some old patches for this issue:
>
> http://lists.xen.org/archives/html/xen-devel/2013-04/msg03072.html
>
> Bob and I will work on it sooner or later (probably a bit later).
>
> Hmmm... Boris, David, how did you manage to get this memory hotplug
> stuff working. Did you use xm instead of xl?

Whoever can test this - what about the attached patch?

David, you said:

This one can be trivially fixed by setting the new sections in the p2m
to INVALID_P2M_ENTRY before calling add_memory().

Are you preparing a patch for this, or do you want me to do it?


Juergen

[-- Attachment #2: 0001-xen-prepare-p2m-list-for-memory-hotplug.patch --]
[-- Type: text/x-patch, Size: 2734 bytes --]

>From a700c3439edb9774a069302d4393e235edc8c264 Mon Sep 17 00:00:00 2001
From: Juergen Gross <jgross@suse.com>
Date: Thu, 19 Mar 2015 07:05:39 +0100
Subject: [PATCH] xen: prepare p2m list for memory hotplug

Commit 054954eb051f35e74b75a566a96fe756015352c8 ("xen: switch to linear
virtual mapped sparse p2m list") introduced a regression regarding to
memory hotplug for a pv-domain: as the virtual space for the p2m list
is allocated for the to be expected memory size of the domain only,
hotplugged memory above that size will not be usable by the domain.

Correct this by using a configurable size for the p2m list in case of
memory hotplug enabled (default supported memory size is 512 GB for
64 bit domains and 4 GB for 32 bit domains).

Signed-off-by: Juergen Gross <jgross@suse.com>
---
 arch/x86/xen/p2m.c  | 13 ++++++++++++-
 drivers/xen/Kconfig | 13 +++++++++++++
 2 files changed, 25 insertions(+), 1 deletion(-)

diff --git a/arch/x86/xen/p2m.c b/arch/x86/xen/p2m.c
index 9f93af5..30e84ae 100644
--- a/arch/x86/xen/p2m.c
+++ b/arch/x86/xen/p2m.c
@@ -91,6 +91,17 @@ EXPORT_SYMBOL_GPL(xen_p2m_size);
 unsigned long xen_max_p2m_pfn __read_mostly;
 EXPORT_SYMBOL_GPL(xen_max_p2m_pfn);
 
+#ifdef CONFIG_XEN_BALLOON_MEMORY_HOTPLUG_LIMIT
+#ifdef CONFIG_X86_32
+BUILD_BUG_ON_MSG(CONFIG_XEN_BALLOON_MEMORY_HOTPLUG_LIMIT > 64)
+#endif
+#define P2M_LIMIT max(xen_max_p2m_pfn,					\
+	((unsigned long)((u64)CONFIG_XEN_BALLOON_MEMORY_HOTPLUG_LIMIT *	\
+	1024 * 1024 * 1024 / PAGE_SIZE)))
+#else
+#define P2M_LIMIT xen_max_p2m_pfn
+#endif
+
 static DEFINE_SPINLOCK(p2m_update_lock);
 
 static unsigned long *p2m_mid_missing_mfn;
@@ -387,7 +398,7 @@ void __init xen_vmalloc_p2m_tree(void)
 	static struct vm_struct vm;
 
 	vm.flags = VM_ALLOC;
-	vm.size = ALIGN(sizeof(unsigned long) * xen_max_p2m_pfn,
+	vm.size = ALIGN(sizeof(unsigned long) * P2M_LIMIT,
 			PMD_SIZE * PMDS_PER_MID_PAGE);
 	vm_area_register_early(&vm, PMD_SIZE * PMDS_PER_MID_PAGE);
 	pr_notice("p2m virtual area at %p, size is %lx\n", vm.addr, vm.size);
diff --git a/drivers/xen/Kconfig b/drivers/xen/Kconfig
index b812462..0a61ddf 100644
--- a/drivers/xen/Kconfig
+++ b/drivers/xen/Kconfig
@@ -55,6 +55,19 @@ config XEN_BALLOON_MEMORY_HOTPLUG
 
 	  In that case step 3 should be omitted.
 
+config XEN_BALLOON_MEMORY_HOTPLUG_LIMIT
+	int
+	default 512 if X86_64
+	default 4 if X86_32
+	depends on XEN_HAVE_PVMMU
+	depends on XEN_BALLOON_MEMORY_HOTPLUG
+	help
+	  Upper limit in GBs a pv domain can be expanded to using memory
+	  hotplug.
+
+	  This value is used to allocate enough space in internal tables needed
+	  for physical memory administration.
+
 config XEN_SCRUB_PAGES
 	bool "Scrub pages before returning them to system"
 	depends on XEN_BALLOON
-- 
2.1.4


[-- Attachment #3: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [PATCHv1] xen/balloon: disable memory hotplug in PV guests

[Juergen Gross]

On 03/19/2015 02:38 PM, Juergen Gross wrote:
> On 03/19/2015 12:34 PM, Daniel Kiper wrote:
>> On Thu, Mar 19, 2015 at 10:55:37AM +0100, Juergen Gross wrote:
>>> On 03/18/2015 04:14 PM, Daniel Kiper wrote:
>>
>> [...]
>>
>>>> I have checked new p2m code and I think that this is reasonable
>>>> solution too.
>>>
>>> Do I need any patches for xl to be able to test this? I did:
>>>
>>> xl mem-max 2 4096
>>> xl mem-set 2 4096
>>
>> Yep. I started work on it but I was not able to finish it.
>> Here you can find some old patches for this issue:
>>
>> http://lists.xen.org/archives/html/xen-devel/2013-04/msg03072.html
>>
>> Bob and I will work on it sooner or later (probably a bit later).
>>
>> Hmmm... Boris, David, how did you manage to get this memory hotplug
>> stuff working. Did you use xm instead of xl?
>
> Whoever can test this - what about the attached patch?

Already succeeded in testing. :-)

Rather easy via:

Dom0: xl mem-max <domain> <size>
DomU: echo <size_in_kb> 
 >/sys/devices/system/xen_memory/xen_memory0/target_kb

On my system udev is set up to add the memory automatically, so no
further action required for me.

> David, you said:
>
> This one can be trivially fixed by setting the new sections in the p2m
> to INVALID_P2M_ENTRY before calling add_memory().
>
> Are you preparing a patch for this, or do you want me to do it?

I've already got a patch working, I'll send it soon.


Juergen