* Increasing domain memory beyond initial maxmem
@ 2022-03-31 3:51 Marek Marczykowski-Górecki
2022-03-31 6:41 ` Juergen Gross
0 siblings, 1 reply; 9+ messages in thread
From: Marek Marczykowski-Górecki @ 2022-03-31 3:51 UTC (permalink / raw)
To: xen-devel
[-- Attachment #1: Type: text/plain, Size: 2839 bytes --]
Hi,
I'm trying to make use of CONFIG_XEN_BALLOON_MEMORY_HOTPLUG=y to increase
domain memory beyond initial maxmem, but I hit few issues.
A little context: domains in Qubes OS start with rather little memory
(400MB by default) but maxmem set higher (4GB by default). Then, there is
qmemman daemon, that adjust balloon targets for domains, based on (among
other things) demand reported by the domains themselves. There is also a
little swap, to mitigate qmemman latency (few hundreds ms at worst).
This initial memory < maxmmem in case of PVH / HVM makes use of PoD
which I'm trying to get rid of. But also, IIUC Linux will waste some
memory for bookkeeping based on maxmem, not actually usable memory.
First issue: after using `xl mem-max`, `xl mem-set` still refuses to
increase memory more than initial maxmem. That's because xl mem-max does
not update 'memory/static-max' xenstore node. This one is easy to work
around.
Then, the actual hotplug fails on the domU side with:
[ 50.004734] xen-balloon: vmemmap alloc failure: order:9, mode:0x4cc0(GFP_KERNEL|__GFP_RETRY_MAYFAIL), nodemask=(null),cpuset=/,mems_allowed=0
[ 50.004774] CPU: 1 PID: 34 Comm: xen-balloon Not tainted 5.16.15-1.37.fc32.qubes.x86_64 #1
[ 50.004792] Call Trace:
[ 50.004799] <TASK>
[ 50.004808] dump_stack_lvl+0x48/0x5e
[ 50.004821] warn_alloc+0x162/0x190
[ 50.004832] ? __alloc_pages+0x1fa/0x230
[ 50.004842] vmemmap_alloc_block+0x11c/0x1c5
[ 50.004856] vmemmap_populate_hugepages+0x185/0x519
[ 50.004868] vmemmap_populate+0x9e/0x16c
[ 50.004878] __populate_section_memmap+0x6a/0xb1
[ 50.004890] section_activate+0x20a/0x278
[ 50.004901] sparse_add_section+0x70/0x160
[ 50.004911] __add_pages+0xc3/0x150
[ 50.004921] add_pages+0x12/0x60
[ 50.004931] add_memory_resource+0x12b/0x320
[ 50.004943] reserve_additional_memory+0x10c/0x150
[ 50.004958] balloon_thread+0x206/0x360
[ 50.004968] ? do_wait_intr_irq+0xa0/0xa0
[ 50.004978] ? decrease_reservation.constprop.0+0x2e0/0x2e0
[ 50.004991] kthread+0x16b/0x190
[ 50.005001] ? set_kthread_struct+0x40/0x40
[ 50.005011] ret_from_fork+0x22/0x30
[ 50.005022] </TASK>
Full dmesg: https://gist.github.com/marmarek/72dd1f9dbdd63cfe479c94a3f4392b45
After the above, `free` reports correct size (1GB in this case), but
that memory seems to be unusable really. "used" is kept low, and soon
OOM-killer kicks in.
I know the initial 400MB is not much for a full Linux, with X11 etc. But
I wouldn't expect it to fail this way when _adding_ memory.
I've tried also with initial 800MB. In this case, I do not get "alloc
failure" any more, but monitoring `free`, the extra memory still doesn't
seem to be used.
Any ideas?
--
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Increasing domain memory beyond initial maxmem
2022-03-31 3:51 Increasing domain memory beyond initial maxmem Marek Marczykowski-Górecki
@ 2022-03-31 6:41 ` Juergen Gross
2022-03-31 12:01 ` Marek Marczykowski-Górecki
0 siblings, 1 reply; 9+ messages in thread
From: Juergen Gross @ 2022-03-31 6:41 UTC (permalink / raw)
To: Marek Marczykowski-Górecki, xen-devel
[-- Attachment #1.1.1: Type: text/plain, Size: 4545 bytes --]
On 31.03.22 05:51, Marek Marczykowski-Górecki wrote:
> Hi,
>
> I'm trying to make use of CONFIG_XEN_BALLOON_MEMORY_HOTPLUG=y to increase
> domain memory beyond initial maxmem, but I hit few issues.
>
> A little context: domains in Qubes OS start with rather little memory
> (400MB by default) but maxmem set higher (4GB by default). Then, there is
> qmemman daemon, that adjust balloon targets for domains, based on (among
> other things) demand reported by the domains themselves. There is also a
> little swap, to mitigate qmemman latency (few hundreds ms at worst).
> This initial memory < maxmmem in case of PVH / HVM makes use of PoD
> which I'm trying to get rid of. But also, IIUC Linux will waste some
> memory for bookkeeping based on maxmem, not actually usable memory.
>
> First issue: after using `xl mem-max`, `xl mem-set` still refuses to
> increase memory more than initial maxmem. That's because xl mem-max does
> not update 'memory/static-max' xenstore node. This one is easy to work
> around.
>
> Then, the actual hotplug fails on the domU side with:
>
> [ 50.004734] xen-balloon: vmemmap alloc failure: order:9, mode:0x4cc0(GFP_KERNEL|__GFP_RETRY_MAYFAIL), nodemask=(null),cpuset=/,mems_allowed=0
> [ 50.004774] CPU: 1 PID: 34 Comm: xen-balloon Not tainted 5.16.15-1.37.fc32.qubes.x86_64 #1
> [ 50.004792] Call Trace:
> [ 50.004799] <TASK>
> [ 50.004808] dump_stack_lvl+0x48/0x5e
> [ 50.004821] warn_alloc+0x162/0x190
> [ 50.004832] ? __alloc_pages+0x1fa/0x230
> [ 50.004842] vmemmap_alloc_block+0x11c/0x1c5
> [ 50.004856] vmemmap_populate_hugepages+0x185/0x519
> [ 50.004868] vmemmap_populate+0x9e/0x16c
> [ 50.004878] __populate_section_memmap+0x6a/0xb1
> [ 50.004890] section_activate+0x20a/0x278
> [ 50.004901] sparse_add_section+0x70/0x160
> [ 50.004911] __add_pages+0xc3/0x150
> [ 50.004921] add_pages+0x12/0x60
> [ 50.004931] add_memory_resource+0x12b/0x320
> [ 50.004943] reserve_additional_memory+0x10c/0x150
> [ 50.004958] balloon_thread+0x206/0x360
> [ 50.004968] ? do_wait_intr_irq+0xa0/0xa0
> [ 50.004978] ? decrease_reservation.constprop.0+0x2e0/0x2e0
> [ 50.004991] kthread+0x16b/0x190
> [ 50.005001] ? set_kthread_struct+0x40/0x40
> [ 50.005011] ret_from_fork+0x22/0x30
> [ 50.005022] </TASK>
>
> Full dmesg: https://gist.github.com/marmarek/72dd1f9dbdd63cfe479c94a3f4392b45
>
> After the above, `free` reports correct size (1GB in this case), but
> that memory seems to be unusable really. "used" is kept low, and soon
> OOM-killer kicks in.
>
> I know the initial 400MB is not much for a full Linux, with X11 etc. But
> I wouldn't expect it to fail this way when _adding_ memory.
>
> I've tried also with initial 800MB. In this case, I do not get "alloc
> failure" any more, but monitoring `free`, the extra memory still doesn't
> seem to be used.
>
> Any ideas?
>
I can't reproduce that.
I started a guest with 8GB of memory, in the guest I'm seeing:
# uname -a
Linux linux-d1cy 5.17.0-rc5-default+ #406 SMP PREEMPT Mon Feb 21 09:31:12 CET
2022 x86_64 x86_64 x86_64 GNU/Linux
# free
total used free shared buff/cache available
Mem: 8178260 71628 8023300 8560 83332 8010196
Swap: 2097132 0 2097132
Then I'm raising the memory for the guest in dom0:
# xl list
Name ID Mem VCPUs State Time(s)
Domain-0 0 2634 8 r----- 1016.5
Xenstore 1 31 1 -b---- 0.9
sle15sp1 3 8190 6 -b---- 184.6
# xl mem-max 3 10000
# xenstore-write /local/domain/3/memory/static-max 10240000
# xl mem-set 3 10000
# xl list
Name ID Mem VCPUs State Time(s)
Domain-0 0 2634 8 r----- 1018.5
Xenstore 1 31 1 -b---- 1.0
sle15sp1 3 10000 6 -b---- 186.7
In the guest I get now:
# free
total used free shared buff/cache available
Mem: 10031700 110904 9734172 8560 186624 9814344
Swap: 2097132 0 2097132
And after using lots of memory via a ramdisk:
# free
total used free shared buff/cache available
Mem: 10031700 116660 1663840 7181776 8251200 2635372
Swap: 2097132 0 2097132
You can see buff/cache is now larger than the initial total memory
and free is lower than the added memory amount.
Juergen
[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 3149 bytes --]
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Increasing domain memory beyond initial maxmem
2022-03-31 6:41 ` Juergen Gross
@ 2022-03-31 12:01 ` Marek Marczykowski-Górecki
2022-03-31 12:22 ` Juergen Gross
0 siblings, 1 reply; 9+ messages in thread
From: Marek Marczykowski-Górecki @ 2022-03-31 12:01 UTC (permalink / raw)
To: Juergen Gross; +Cc: xen-devel
[-- Attachment #1: Type: text/plain, Size: 8338 bytes --]
On Thu, Mar 31, 2022 at 08:41:19AM +0200, Juergen Gross wrote:
> On 31.03.22 05:51, Marek Marczykowski-Górecki wrote:
> > Hi,
> >
> > I'm trying to make use of CONFIG_XEN_BALLOON_MEMORY_HOTPLUG=y to increase
> > domain memory beyond initial maxmem, but I hit few issues.
> >
> > A little context: domains in Qubes OS start with rather little memory
> > (400MB by default) but maxmem set higher (4GB by default). Then, there is
> > qmemman daemon, that adjust balloon targets for domains, based on (among
> > other things) demand reported by the domains themselves. There is also a
> > little swap, to mitigate qmemman latency (few hundreds ms at worst).
> > This initial memory < maxmmem in case of PVH / HVM makes use of PoD
> > which I'm trying to get rid of. But also, IIUC Linux will waste some
> > memory for bookkeeping based on maxmem, not actually usable memory.
> >
> > First issue: after using `xl mem-max`, `xl mem-set` still refuses to
> > increase memory more than initial maxmem. That's because xl mem-max does
> > not update 'memory/static-max' xenstore node. This one is easy to work
> > around.
> >
> > Then, the actual hotplug fails on the domU side with:
> >
> > [ 50.004734] xen-balloon: vmemmap alloc failure: order:9, mode:0x4cc0(GFP_KERNEL|__GFP_RETRY_MAYFAIL), nodemask=(null),cpuset=/,mems_allowed=0
> > [ 50.004774] CPU: 1 PID: 34 Comm: xen-balloon Not tainted 5.16.15-1.37.fc32.qubes.x86_64 #1
> > [ 50.004792] Call Trace:
> > [ 50.004799] <TASK>
> > [ 50.004808] dump_stack_lvl+0x48/0x5e
> > [ 50.004821] warn_alloc+0x162/0x190
> > [ 50.004832] ? __alloc_pages+0x1fa/0x230
> > [ 50.004842] vmemmap_alloc_block+0x11c/0x1c5
> > [ 50.004856] vmemmap_populate_hugepages+0x185/0x519
> > [ 50.004868] vmemmap_populate+0x9e/0x16c
> > [ 50.004878] __populate_section_memmap+0x6a/0xb1
> > [ 50.004890] section_activate+0x20a/0x278
> > [ 50.004901] sparse_add_section+0x70/0x160
> > [ 50.004911] __add_pages+0xc3/0x150
> > [ 50.004921] add_pages+0x12/0x60
> > [ 50.004931] add_memory_resource+0x12b/0x320
> > [ 50.004943] reserve_additional_memory+0x10c/0x150
> > [ 50.004958] balloon_thread+0x206/0x360
> > [ 50.004968] ? do_wait_intr_irq+0xa0/0xa0
> > [ 50.004978] ? decrease_reservation.constprop.0+0x2e0/0x2e0
> > [ 50.004991] kthread+0x16b/0x190
> > [ 50.005001] ? set_kthread_struct+0x40/0x40
> > [ 50.005011] ret_from_fork+0x22/0x30
> > [ 50.005022] </TASK>
> >
> > Full dmesg: https://gist.github.com/marmarek/72dd1f9dbdd63cfe479c94a3f4392b45
> >
> > After the above, `free` reports correct size (1GB in this case), but
> > that memory seems to be unusable really. "used" is kept low, and soon
> > OOM-killer kicks in.
> >
> > I know the initial 400MB is not much for a full Linux, with X11 etc. But
> > I wouldn't expect it to fail this way when _adding_ memory.
> >
> > I've tried also with initial 800MB. In this case, I do not get "alloc
> > failure" any more, but monitoring `free`, the extra memory still doesn't
> > seem to be used.
> >
> > Any ideas?
> >
>
> I can't reproduce that.
>
> I started a guest with 8GB of memory, in the guest I'm seeing:
>
> # uname -a
> Linux linux-d1cy 5.17.0-rc5-default+ #406 SMP PREEMPT Mon Feb 21 09:31:12
> CET 2022 x86_64 x86_64 x86_64 GNU/Linux
> # free
> total used free shared buff/cache available
> Mem: 8178260 71628 8023300 8560 83332 8010196
> Swap: 2097132 0 2097132
>
> Then I'm raising the memory for the guest in dom0:
>
> # xl list
> Name ID Mem VCPUs State Time(s)
> Domain-0 0 2634 8 r----- 1016.5
> Xenstore 1 31 1 -b---- 0.9
> sle15sp1 3 8190 6 -b---- 184.6
> # xl mem-max 3 10000
> # xenstore-write /local/domain/3/memory/static-max 10240000
> # xl mem-set 3 10000
> # xl list
> Name ID Mem VCPUs State Time(s)
> Domain-0 0 2634 8 r----- 1018.5
> Xenstore 1 31 1 -b---- 1.0
> sle15sp1 3 10000 6 -b---- 186.7
>
> In the guest I get now:
>
> # free
> total used free shared buff/cache available
> Mem: 10031700 110904 9734172 8560 186624 9814344
> Swap: 2097132 0 2097132
>
> And after using lots of memory via a ramdisk:
>
> # free
> total used free shared buff/cache available
> Mem: 10031700 116660 1663840 7181776 8251200 2635372
> Swap: 2097132 0 2097132
>
> You can see buff/cache is now larger than the initial total memory
> and free is lower than the added memory amount.
Hmm, I have a different behavior:
I'm starting with 800M
# uname -a
Linux personal 5.16.15-1.37.fc32.qubes.x86_64 #1 SMP PREEMPT Tue Mar 22 12:59:53 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
# free -m
total used free shared buff/cache available
Mem: 740 209 278 2 252 415
Swap: 1023 0 1023
Then raising to ~2GB:
[root@dom0 ~]# xl list
Name ID Mem VCPUs State Time(s)
Domain-0 0 4082 6 r----- 184271.3
(...)
personal 21 800 2 -b---- 4.8
[root@dom0 ~]# xl mem-max personal 2048
[root@dom0 ~]# xenstore-write /local/domain/$(xl domid personal)/memory/static-max $((2048*1024))
[root@dom0 ~]# xl mem-set personal 2000
[root@dom0 ~]# xenstore-ls -fp /local/domain/$(xl domid personal)/memory
/local/domain/21/memory/static-max = "2097152" (n0,r21)
/local/domain/21/memory/target = "2048001" (n0,r21)
/local/domain/21/memory/videoram = "-1" (n0,r21)
And then observe inside domU:
[root@personal ~]# free -m
total used free shared buff/cache available
Mem: 1940 235 1452 2 252 1585
Swap: 1023 0 1023
So far so good. But when trying to actually use it, it doesn't work:
[root@personal ~]# free -m
total used free shared buff/cache available
Mem: 1940 196 1240 454 503 1206
Swap: 1023 472 551
As you can see, all the new memory is still in "free", and swap is used
instead.
There is also /proc/meminfo (state before filling ramdisk), if that
would give some hints:
[root@personal ~]# cat /proc/meminfo
MemTotal: 1986800 kB
MemFree: 1487116 kB
MemAvailable: 1624060 kB
Buffers: 26236 kB
Cached: 207268 kB
SwapCached: 0 kB
Active: 74828 kB
Inactive: 258724 kB
Active(anon): 1008 kB
Inactive(anon): 101668 kB
Active(file): 73820 kB
Inactive(file): 157056 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 1048572 kB
SwapFree: 1048572 kB
Dirty: 216 kB
Writeback: 0 kB
AnonPages: 100184 kB
Mapped: 117472 kB
Shmem: 2628 kB
KReclaimable: 24960 kB
Slab: 52136 kB
SReclaimable: 24960 kB
SUnreclaim: 27176 kB
KernelStack: 3120 kB
PageTables: 4364 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 2041972 kB
Committed_AS: 825816 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 10064 kB
VmallocChunk: 0 kB
Percpu: 1240 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
FileHugePages: 0 kB
FilePmdMapped: 0 kB
CmaTotal: 0 kB
CmaFree: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 0 kB
DirectMap4k: 79872 kB
DirectMap2M: 1132544 kB
DirectMap1G: 1048576 kB
--
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Increasing domain memory beyond initial maxmem
2022-03-31 12:01 ` Marek Marczykowski-Górecki
@ 2022-03-31 12:22 ` Juergen Gross
2022-03-31 12:36 ` Marek Marczykowski-Górecki
0 siblings, 1 reply; 9+ messages in thread
From: Juergen Gross @ 2022-03-31 12:22 UTC (permalink / raw)
To: Marek Marczykowski-Górecki; +Cc: xen-devel
[-- Attachment #1.1.1: Type: text/plain, Size: 7619 bytes --]
On 31.03.22 14:01, Marek Marczykowski-Górecki wrote:
> On Thu, Mar 31, 2022 at 08:41:19AM +0200, Juergen Gross wrote:
>> On 31.03.22 05:51, Marek Marczykowski-Górecki wrote:
>>> Hi,
>>>
>>> I'm trying to make use of CONFIG_XEN_BALLOON_MEMORY_HOTPLUG=y to increase
>>> domain memory beyond initial maxmem, but I hit few issues.
>>>
>>> A little context: domains in Qubes OS start with rather little memory
>>> (400MB by default) but maxmem set higher (4GB by default). Then, there is
>>> qmemman daemon, that adjust balloon targets for domains, based on (among
>>> other things) demand reported by the domains themselves. There is also a
>>> little swap, to mitigate qmemman latency (few hundreds ms at worst).
>>> This initial memory < maxmmem in case of PVH / HVM makes use of PoD
>>> which I'm trying to get rid of. But also, IIUC Linux will waste some
>>> memory for bookkeeping based on maxmem, not actually usable memory.
>>>
>>> First issue: after using `xl mem-max`, `xl mem-set` still refuses to
>>> increase memory more than initial maxmem. That's because xl mem-max does
>>> not update 'memory/static-max' xenstore node. This one is easy to work
>>> around.
>>>
>>> Then, the actual hotplug fails on the domU side with:
>>>
>>> [ 50.004734] xen-balloon: vmemmap alloc failure: order:9, mode:0x4cc0(GFP_KERNEL|__GFP_RETRY_MAYFAIL), nodemask=(null),cpuset=/,mems_allowed=0
>>> [ 50.004774] CPU: 1 PID: 34 Comm: xen-balloon Not tainted 5.16.15-1.37.fc32.qubes.x86_64 #1
>>> [ 50.004792] Call Trace:
>>> [ 50.004799] <TASK>
>>> [ 50.004808] dump_stack_lvl+0x48/0x5e
>>> [ 50.004821] warn_alloc+0x162/0x190
>>> [ 50.004832] ? __alloc_pages+0x1fa/0x230
>>> [ 50.004842] vmemmap_alloc_block+0x11c/0x1c5
>>> [ 50.004856] vmemmap_populate_hugepages+0x185/0x519
>>> [ 50.004868] vmemmap_populate+0x9e/0x16c
>>> [ 50.004878] __populate_section_memmap+0x6a/0xb1
>>> [ 50.004890] section_activate+0x20a/0x278
>>> [ 50.004901] sparse_add_section+0x70/0x160
>>> [ 50.004911] __add_pages+0xc3/0x150
>>> [ 50.004921] add_pages+0x12/0x60
>>> [ 50.004931] add_memory_resource+0x12b/0x320
>>> [ 50.004943] reserve_additional_memory+0x10c/0x150
>>> [ 50.004958] balloon_thread+0x206/0x360
>>> [ 50.004968] ? do_wait_intr_irq+0xa0/0xa0
>>> [ 50.004978] ? decrease_reservation.constprop.0+0x2e0/0x2e0
>>> [ 50.004991] kthread+0x16b/0x190
>>> [ 50.005001] ? set_kthread_struct+0x40/0x40
>>> [ 50.005011] ret_from_fork+0x22/0x30
>>> [ 50.005022] </TASK>
>>>
>>> Full dmesg: https://gist.github.com/marmarek/72dd1f9dbdd63cfe479c94a3f4392b45
>>>
>>> After the above, `free` reports correct size (1GB in this case), but
>>> that memory seems to be unusable really. "used" is kept low, and soon
>>> OOM-killer kicks in.
>>>
>>> I know the initial 400MB is not much for a full Linux, with X11 etc. But
>>> I wouldn't expect it to fail this way when _adding_ memory.
>>>
>>> I've tried also with initial 800MB. In this case, I do not get "alloc
>>> failure" any more, but monitoring `free`, the extra memory still doesn't
>>> seem to be used.
>>>
>>> Any ideas?
>>>
>>
>> I can't reproduce that.
>>
>> I started a guest with 8GB of memory, in the guest I'm seeing:
>>
>> # uname -a
>> Linux linux-d1cy 5.17.0-rc5-default+ #406 SMP PREEMPT Mon Feb 21 09:31:12
>> CET 2022 x86_64 x86_64 x86_64 GNU/Linux
>> # free
>> total used free shared buff/cache available
>> Mem: 8178260 71628 8023300 8560 83332 8010196
>> Swap: 2097132 0 2097132
>>
>> Then I'm raising the memory for the guest in dom0:
>>
>> # xl list
>> Name ID Mem VCPUs State Time(s)
>> Domain-0 0 2634 8 r----- 1016.5
>> Xenstore 1 31 1 -b---- 0.9
>> sle15sp1 3 8190 6 -b---- 184.6
>> # xl mem-max 3 10000
>> # xenstore-write /local/domain/3/memory/static-max 10240000
>> # xl mem-set 3 10000
>> # xl list
>> Name ID Mem VCPUs State Time(s)
>> Domain-0 0 2634 8 r----- 1018.5
>> Xenstore 1 31 1 -b---- 1.0
>> sle15sp1 3 10000 6 -b---- 186.7
>>
>> In the guest I get now:
>>
>> # free
>> total used free shared buff/cache available
>> Mem: 10031700 110904 9734172 8560 186624 9814344
>> Swap: 2097132 0 2097132
>>
>> And after using lots of memory via a ramdisk:
>>
>> # free
>> total used free shared buff/cache available
>> Mem: 10031700 116660 1663840 7181776 8251200 2635372
>> Swap: 2097132 0 2097132
>>
>> You can see buff/cache is now larger than the initial total memory
>> and free is lower than the added memory amount.
>
> Hmm, I have a different behavior:
>
> I'm starting with 800M
>
> # uname -a
> Linux personal 5.16.15-1.37.fc32.qubes.x86_64 #1 SMP PREEMPT Tue Mar 22 12:59:53 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
> # free -m
> total used free shared buff/cache available
> Mem: 740 209 278 2 252 415
> Swap: 1023 0 1023
>
> Then raising to ~2GB:
>
> [root@dom0 ~]# xl list
> Name ID Mem VCPUs State Time(s)
> Domain-0 0 4082 6 r----- 184271.3
> (...)
> personal 21 800 2 -b---- 4.8
> [root@dom0 ~]# xl mem-max personal 2048
> [root@dom0 ~]# xenstore-write /local/domain/$(xl domid personal)/memory/static-max $((2048*1024))
> [root@dom0 ~]# xl mem-set personal 2000
> [root@dom0 ~]# xenstore-ls -fp /local/domain/$(xl domid personal)/memory
> /local/domain/21/memory/static-max = "2097152" (n0,r21)
> /local/domain/21/memory/target = "2048001" (n0,r21)
> /local/domain/21/memory/videoram = "-1" (n0,r21)
>
> And then observe inside domU:
> [root@personal ~]# free -m
> total used free shared buff/cache available
> Mem: 1940 235 1452 2 252 1585
> Swap: 1023 0 1023
>
> So far so good. But when trying to actually use it, it doesn't work:
>
> [root@personal ~]# free -m
> total used free shared buff/cache available
> Mem: 1940 196 1240 454 503 1206
> Swap: 1023 472 551
>
> As you can see, all the new memory is still in "free", and swap is used
> instead.
Hmm, weird.
Maybe some kernel config differences, or other udev rules (memory onlining
is done via udev in my guest)?
I'm seeing:
# zgrep MEMORY_HOTPLUG /proc/config.gz
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
CONFIG_MEMORY_HOTPLUG=y
# CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE is not set
CONFIG_XEN_BALLOON_MEMORY_HOTPLUG=y
CONFIG_XEN_MEMORY_HOTPLUG_LIMIT=512
The relevant udev rule seems to be:
SUBSYSTEM=="memory", ACTION=="add", PROGRAM=="/bin/sh -c
'/usr/bin/systemd-detect-virt || :'", RESULT!="zvm", ATTR{state}=="offline", \
ATTR{state}="online"
What type of guest are you using? Mine was a PVH guest.
> There is also /proc/meminfo (state before filling ramdisk), if that
> would give some hints:
> [root@personal ~]# cat /proc/meminfo
...
No, I don't think this is helping. At least not me.
Juergen
[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 3149 bytes --]
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Increasing domain memory beyond initial maxmem
2022-03-31 12:22 ` Juergen Gross
@ 2022-03-31 12:36 ` Marek Marczykowski-Górecki
2022-04-05 11:03 ` Juergen Gross
0 siblings, 1 reply; 9+ messages in thread
From: Marek Marczykowski-Górecki @ 2022-03-31 12:36 UTC (permalink / raw)
To: Juergen Gross; +Cc: xen-devel
[-- Attachment #1: Type: text/plain, Size: 1069 bytes --]
On Thu, Mar 31, 2022 at 02:22:03PM +0200, Juergen Gross wrote:
> Maybe some kernel config differences, or other udev rules (memory onlining
> is done via udev in my guest)?
>
> I'm seeing:
>
> # zgrep MEMORY_HOTPLUG /proc/config.gz
> CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
> CONFIG_MEMORY_HOTPLUG=y
> # CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE is not set
> CONFIG_XEN_BALLOON_MEMORY_HOTPLUG=y
> CONFIG_XEN_MEMORY_HOTPLUG_LIMIT=512
I have:
# zgrep MEMORY_HOTPLUG /proc/config.gz
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
CONFIG_MEMORY_HOTPLUG=y
CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y
CONFIG_XEN_BALLOON_MEMORY_HOTPLUG=y
CONFIG_XEN_MEMORY_HOTPLUG_LIMIT=512
Not sure if relevant, but I also have:
CONFIG_XEN_UNPOPULATED_ALLOC=y
on top of that, I have a similar udev rule too:
SUBSYSTEM=="memory", ACTION=="add", ATTR{state}=="offline", ATTR{state}="online"
But I don't think they are conflicting.
> What type of guest are you using? Mine was a PVH guest.
PVH here too.
--
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Increasing domain memory beyond initial maxmem
2022-03-31 12:36 ` Marek Marczykowski-Górecki
@ 2022-04-05 11:03 ` Juergen Gross
2022-04-05 16:24 ` Marek Marczykowski-Górecki
0 siblings, 1 reply; 9+ messages in thread
From: Juergen Gross @ 2022-04-05 11:03 UTC (permalink / raw)
To: Marek Marczykowski-Górecki; +Cc: xen-devel
[-- Attachment #1.1.1: Type: text/plain, Size: 1199 bytes --]
Hi Marek,
On 31.03.22 14:36, Marek Marczykowski-Górecki wrote:
> On Thu, Mar 31, 2022 at 02:22:03PM +0200, Juergen Gross wrote:
>> Maybe some kernel config differences, or other udev rules (memory onlining
>> is done via udev in my guest)?
>>
>> I'm seeing:
>>
>> # zgrep MEMORY_HOTPLUG /proc/config.gz
>> CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
>> CONFIG_MEMORY_HOTPLUG=y
>> # CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE is not set
>> CONFIG_XEN_BALLOON_MEMORY_HOTPLUG=y
>> CONFIG_XEN_MEMORY_HOTPLUG_LIMIT=512
>
> I have:
> # zgrep MEMORY_HOTPLUG /proc/config.gz
> CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
> CONFIG_MEMORY_HOTPLUG=y
> CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y
> CONFIG_XEN_BALLOON_MEMORY_HOTPLUG=y
> CONFIG_XEN_MEMORY_HOTPLUG_LIMIT=512
>
> Not sure if relevant, but I also have:
> CONFIG_XEN_UNPOPULATED_ALLOC=y
>
> on top of that, I have a similar udev rule too:
>
> SUBSYSTEM=="memory", ACTION=="add", ATTR{state}=="offline", ATTR{state}="online"
>
> But I don't think they are conflicting.
>
>> What type of guest are you using? Mine was a PVH guest.
>
> PVH here too.
Would you like to try the attached patch? It seemed to work for me.
Juergen
[-- Attachment #1.1.2: 0001-xen-balloon-fix-page-onlining-when-populating-new-zo.patch --]
[-- Type: text/x-patch, Size: 5744 bytes --]
From a605232115a9c3d3f8103d0833b149ff22956c4b Mon Sep 17 00:00:00 2001
From: Juergen Gross <jgross@suse.com>
To: linux-kernel@vger.kernel.org
Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>
Cc: Juergen Gross <jgross@suse.com>
Cc: Stefano Stabellini <sstabellini@kernel.org>
Cc: xen-devel@lists.xenproject.org
Date: Tue, 5 Apr 2022 12:43:41 +0200
Subject: [PATCH] xen/balloon: fix page onlining when populating new zone
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
When onlining a new memory page in a guest the Xen balloon driver is
adding it to the ballooned pages instead making it available to be
used immediately. This is meant to enable to add a new upper memory
limit to a guest via hotplugging memory, without having to assign the
new memory in one go.
In case the upper memory limit will be raised above 4G, the new memory
will populate the ZONE_NORMAL memory zone, which wasn't populated
before. The newly populated zone won't be added to the list of zones
looked at by the page allocator though, as only zones with available
memory are being added, and the memory isn't yet available as it is
ballooned out.
This will result in the new memory being assigned to the guest, but
without the allocator being able to use it.
When running as a PV guest the situation is even worse: when having
been started with less memory than allowed, and the upper limit being
lower than 4G, ballooning up will have the same effect as hotplugging
new memory. This is due to the usage of the zone device functionality
since commit 9e2369c06c8a ("xen: add helpers to allocate unpopulated
memory") for creating mappings of other guest's pages, which as a side
effect is being used for PV guest ballooning, too.
Fix this by checking in xen_online_page() whether the new memory page
will be the first in a new zone. If this is the case, add another page
to the balloon and use the first memory page of the new chunk as a
replacement for this now ballooned out page. This will result in the
newly populated zone containing one page being available for the page
allocator, which in turn will lead to the zone being added to the
allocator.
Cc: stable@vger.kernel.org
Fixes: 9e2369c06c8a ("xen: add helpers to allocate unpopulated memory")
Reported-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
Signed-off-by: Juergen Gross <jgross@suse.com>
---
drivers/xen/balloon.c | 72 ++++++++++++++++++++++++++++++++++++++-----
1 file changed, 65 insertions(+), 7 deletions(-)
diff --git a/drivers/xen/balloon.c b/drivers/xen/balloon.c
index dfe26fa17e95..f895c54c4c65 100644
--- a/drivers/xen/balloon.c
+++ b/drivers/xen/balloon.c
@@ -355,14 +355,77 @@ static enum bp_state reserve_additional_memory(void)
return BP_ECANCELED;
}
+static struct page *alloc_page_for_balloon(gfp_t gfp)
+{
+ struct page *page;
+
+ page = alloc_page(gfp);
+ if (page == NULL)
+ return NULL;
+
+ adjust_managed_page_count(page, -1);
+ xenmem_reservation_scrub_page(page);
+
+ return page;
+}
+
+static void add_page_to_balloon(struct page *page)
+{
+ xenmem_reservation_va_mapping_reset(1, &page);
+ balloon_append(page);
+}
+
static void xen_online_page(struct page *page, unsigned int order)
{
unsigned long i, size = (1 << order);
unsigned long start_pfn = page_to_pfn(page);
struct page *p;
+ struct zone *zone;
pr_debug("Online %lu pages starting at pfn 0x%lx\n", size, start_pfn);
mutex_lock(&balloon_mutex);
+ zone = page_zone(pfn_to_page(start_pfn));
+
+ /*
+ * In case a new memory zone is going to be populated, we need to
+ * ensure at least one page is made available for the memory allocator.
+ * As the number of pages per zone is updated only after a batch of
+ * pages having been added, use the number of managed pages as an
+ * additional indicator for a new zone.
+ * Otherwise this zone won't be added to the zonelist resulting in the
+ * zone's memory not usable by the kernel.
+ * Add an already valid page to the balloon and replace it with the
+ * first page of the to be added new memory chunk.
+ */
+ if (!populated_zone(zone) && !managed_zone(zone)) {
+ xen_pfn_t frame;
+
+ pr_info("Populating new zone\n");
+
+ p = alloc_page_for_balloon(GFP_ATOMIC);
+ if (!p) {
+ pr_err("Failed to allocate replacement balloon page!\n");
+ pr_err("New onlined memory might not be usable.\n");
+ } else {
+ kmap_flush_unused();
+ add_page_to_balloon(p);
+ flush_tlb_all();
+ frame = xen_page_to_gfn(p);
+ xenmem_reservation_decrease(1, &frame);
+ balloon_stats.current_pages--;
+ }
+
+ p = pfn_to_page(start_pfn);
+ frame = page_to_xen_pfn(p);
+ if (xenmem_reservation_increase(1, &frame) > 0) {
+ xenmem_reservation_va_mapping_update(1, &p, &frame);
+ free_reserved_page(p);
+ balloon_stats.current_pages++;
+
+ start_pfn++;
+ size--;
+ }
+ }
for (i = 0; i < size; i++) {
p = pfn_to_page(start_pfn + i);
balloon_append(p);
@@ -452,14 +515,12 @@ static enum bp_state decrease_reservation(unsigned long nr_pages, gfp_t gfp)
nr_pages = ARRAY_SIZE(frame_list);
for (i = 0; i < nr_pages; i++) {
- page = alloc_page(gfp);
+ page = alloc_page_for_balloon(gfp);
if (page == NULL) {
nr_pages = i;
state = BP_EAGAIN;
break;
}
- adjust_managed_page_count(page, -1);
- xenmem_reservation_scrub_page(page);
list_add(&page->lru, &pages);
}
@@ -480,11 +541,8 @@ static enum bp_state decrease_reservation(unsigned long nr_pages, gfp_t gfp)
list_for_each_entry_safe(page, tmp, &pages, lru) {
frame_list[i++] = xen_page_to_gfn(page);
- xenmem_reservation_va_mapping_reset(1, &page);
-
list_del(&page->lru);
-
- balloon_append(page);
+ add_page_to_balloon(page);
}
flush_tlb_all();
--
2.34.1
[-- Attachment #1.1.3: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 3149 bytes --]
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]
^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: Increasing domain memory beyond initial maxmem
2022-04-05 11:03 ` Juergen Gross
@ 2022-04-05 16:24 ` Marek Marczykowski-Górecki
2022-04-06 5:13 ` Juergen Gross
0 siblings, 1 reply; 9+ messages in thread
From: Marek Marczykowski-Górecki @ 2022-04-05 16:24 UTC (permalink / raw)
To: Juergen Gross; +Cc: xen-devel
[-- Attachment #1: Type: text/plain, Size: 2690 bytes --]
On Tue, Apr 05, 2022 at 01:03:57PM +0200, Juergen Gross wrote:
> Hi Marek,
>
> On 31.03.22 14:36, Marek Marczykowski-Górecki wrote:
> > On Thu, Mar 31, 2022 at 02:22:03PM +0200, Juergen Gross wrote:
> > > Maybe some kernel config differences, or other udev rules (memory onlining
> > > is done via udev in my guest)?
> > >
> > > I'm seeing:
> > >
> > > # zgrep MEMORY_HOTPLUG /proc/config.gz
> > > CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
> > > CONFIG_MEMORY_HOTPLUG=y
> > > # CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE is not set
> > > CONFIG_XEN_BALLOON_MEMORY_HOTPLUG=y
> > > CONFIG_XEN_MEMORY_HOTPLUG_LIMIT=512
> >
> > I have:
> > # zgrep MEMORY_HOTPLUG /proc/config.gz
> > CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
> > CONFIG_MEMORY_HOTPLUG=y
> > CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y
> > CONFIG_XEN_BALLOON_MEMORY_HOTPLUG=y
> > CONFIG_XEN_MEMORY_HOTPLUG_LIMIT=512
> >
> > Not sure if relevant, but I also have:
> > CONFIG_XEN_UNPOPULATED_ALLOC=y
> >
> > on top of that, I have a similar udev rule too:
> >
> > SUBSYSTEM=="memory", ACTION=="add", ATTR{state}=="offline", ATTR{state}="online"
> >
> > But I don't think they are conflicting.
> >
> > > What type of guest are you using? Mine was a PVH guest.
> >
> > PVH here too.
>
> Would you like to try the attached patch? It seemed to work for me.
Unfortunately it doesn't help, now the behavior is different:
Initially guest started with 800M:
[root@personal ~]# free -m
total used free shared buff/cache available
Mem: 740 223 272 2 243 401
Swap: 1023 0 1023
Then increased:
[root@dom0 ~]$ xl mem-max personal 2048
[root@dom0 ~]$ xenstore-write /local/domain/$(xl domid personal)/memory/static-max $((2048*1024))
[root@dom0 ~]$ xl mem-set personal 2000
And guest shows now only a little more memory, but not full 2000M:
[root@personal ~]# [ 37.657046] xen:balloon: Populating new zone
[ 37.658206] Fallback order for Node 0: 0
[ 37.658219] Built 1 zonelists, mobility grouping on. Total pages: 175889
[ 37.658233] Policy zone: Normal
[root@personal ~]#
[root@personal ~]# free -m
total used free shared buff/cache available
Mem: 826 245 337 2 244 462
Swap: 1023 0 1023
I've applied the patch on top of 5.16.18. If you think 5.17 would make a
difference, I can try that too.
--
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Increasing domain memory beyond initial maxmem
2022-04-05 16:24 ` Marek Marczykowski-Górecki
@ 2022-04-06 5:13 ` Juergen Gross
2022-04-06 12:58 ` Marek Marczykowski-Górecki
0 siblings, 1 reply; 9+ messages in thread
From: Juergen Gross @ 2022-04-06 5:13 UTC (permalink / raw)
To: Marek Marczykowski-Górecki; +Cc: xen-devel
[-- Attachment #1.1.1: Type: text/plain, Size: 2886 bytes --]
On 05.04.22 18:24, Marek Marczykowski-Górecki wrote:
> On Tue, Apr 05, 2022 at 01:03:57PM +0200, Juergen Gross wrote:
>> Hi Marek,
>>
>> On 31.03.22 14:36, Marek Marczykowski-Górecki wrote:
>>> On Thu, Mar 31, 2022 at 02:22:03PM +0200, Juergen Gross wrote:
>>>> Maybe some kernel config differences, or other udev rules (memory onlining
>>>> is done via udev in my guest)?
>>>>
>>>> I'm seeing:
>>>>
>>>> # zgrep MEMORY_HOTPLUG /proc/config.gz
>>>> CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
>>>> CONFIG_MEMORY_HOTPLUG=y
>>>> # CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE is not set
>>>> CONFIG_XEN_BALLOON_MEMORY_HOTPLUG=y
>>>> CONFIG_XEN_MEMORY_HOTPLUG_LIMIT=512
>>>
>>> I have:
>>> # zgrep MEMORY_HOTPLUG /proc/config.gz
>>> CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
>>> CONFIG_MEMORY_HOTPLUG=y
>>> CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y
>>> CONFIG_XEN_BALLOON_MEMORY_HOTPLUG=y
>>> CONFIG_XEN_MEMORY_HOTPLUG_LIMIT=512
>>>
>>> Not sure if relevant, but I also have:
>>> CONFIG_XEN_UNPOPULATED_ALLOC=y
>>>
>>> on top of that, I have a similar udev rule too:
>>>
>>> SUBSYSTEM=="memory", ACTION=="add", ATTR{state}=="offline", ATTR{state}="online"
>>>
>>> But I don't think they are conflicting.
>>>
>>>> What type of guest are you using? Mine was a PVH guest.
>>>
>>> PVH here too.
>>
>> Would you like to try the attached patch? It seemed to work for me.
>
> Unfortunately it doesn't help, now the behavior is different:
>
> Initially guest started with 800M:
>
> [root@personal ~]# free -m
> total used free shared buff/cache available
> Mem: 740 223 272 2 243 401
> Swap: 1023 0 1023
>
> Then increased:
>
> [root@dom0 ~]$ xl mem-max personal 2048
> [root@dom0 ~]$ xenstore-write /local/domain/$(xl domid personal)/memory/static-max $((2048*1024))
> [root@dom0 ~]$ xl mem-set personal 2000
>
> And guest shows now only a little more memory, but not full 2000M:
>
> [root@personal ~]# [ 37.657046] xen:balloon: Populating new zone
> [ 37.658206] Fallback order for Node 0: 0
> [ 37.658219] Built 1 zonelists, mobility grouping on. Total pages: 175889
> [ 37.658233] Policy zone: Normal
>
> [root@personal ~]#
> [root@personal ~]# free -m
> total used free shared buff/cache available
> Mem: 826 245 337 2 244 462
> Swap: 1023 0 1023
>
>
> I've applied the patch on top of 5.16.18. If you think 5.17 would make a
> difference, I can try that too.
Hmm, weird.
Can you please post the output of
cat /proc/buddyinfo
cat /proc/iomem
in the guest before and after the operations?
Juergen
[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 3149 bytes --]
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Increasing domain memory beyond initial maxmem
2022-04-06 5:13 ` Juergen Gross
@ 2022-04-06 12:58 ` Marek Marczykowski-Górecki
0 siblings, 0 replies; 9+ messages in thread
From: Marek Marczykowski-Górecki @ 2022-04-06 12:58 UTC (permalink / raw)
To: Juergen Gross; +Cc: xen-devel
[-- Attachment #1: Type: text/plain, Size: 3393 bytes --]
On Wed, Apr 06, 2022 at 07:13:18AM +0200, Juergen Gross wrote:
> On 05.04.22 18:24, Marek Marczykowski-Górecki wrote:
> > On Tue, Apr 05, 2022 at 01:03:57PM +0200, Juergen Gross wrote:
> > > Hi Marek,
> > >
> > > On 31.03.22 14:36, Marek Marczykowski-Górecki wrote:
> > > > On Thu, Mar 31, 2022 at 02:22:03PM +0200, Juergen Gross wrote:
> > > > > Maybe some kernel config differences, or other udev rules (memory onlining
> > > > > is done via udev in my guest)?
> > > > >
> > > > > I'm seeing:
> > > > >
> > > > > # zgrep MEMORY_HOTPLUG /proc/config.gz
> > > > > CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
> > > > > CONFIG_MEMORY_HOTPLUG=y
> > > > > # CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE is not set
> > > > > CONFIG_XEN_BALLOON_MEMORY_HOTPLUG=y
> > > > > CONFIG_XEN_MEMORY_HOTPLUG_LIMIT=512
> > > >
> > > > I have:
> > > > # zgrep MEMORY_HOTPLUG /proc/config.gz
> > > > CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
> > > > CONFIG_MEMORY_HOTPLUG=y
> > > > CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y
> > > > CONFIG_XEN_BALLOON_MEMORY_HOTPLUG=y
> > > > CONFIG_XEN_MEMORY_HOTPLUG_LIMIT=512
> > > >
> > > > Not sure if relevant, but I also have:
> > > > CONFIG_XEN_UNPOPULATED_ALLOC=y
> > > >
> > > > on top of that, I have a similar udev rule too:
> > > >
> > > > SUBSYSTEM=="memory", ACTION=="add", ATTR{state}=="offline", ATTR{state}="online"
> > > >
> > > > But I don't think they are conflicting.
> > > >
> > > > > What type of guest are you using? Mine was a PVH guest.
> > > >
> > > > PVH here too.
> > >
> > > Would you like to try the attached patch? It seemed to work for me.
> >
> > Unfortunately it doesn't help, now the behavior is different:
> >
> > Initially guest started with 800M:
> >
> > [root@personal ~]# free -m
> > total used free shared buff/cache available
> > Mem: 740 223 272 2 243 401
> > Swap: 1023 0 1023
> >
> > Then increased:
> >
> > [root@dom0 ~]$ xl mem-max personal 2048
> > [root@dom0 ~]$ xenstore-write /local/domain/$(xl domid personal)/memory/static-max $((2048*1024))
> > [root@dom0 ~]$ xl mem-set personal 2000
> >
> > And guest shows now only a little more memory, but not full 2000M:
> >
> > [root@personal ~]# [ 37.657046] xen:balloon: Populating new zone
> > [ 37.658206] Fallback order for Node 0: 0
> > [ 37.658219] Built 1 zonelists, mobility grouping on. Total pages: 175889
> > [ 37.658233] Policy zone: Normal
> >
> > [root@personal ~]#
> > [root@personal ~]# free -m
> > total used free shared buff/cache available
> > Mem: 826 245 337 2 244 462
> > Swap: 1023 0 1023
> >
> >
> > I've applied the patch on top of 5.16.18. If you think 5.17 would make a
> > difference, I can try that too.
>
> Hmm, weird.
>
> Can you please post the output of
>
> cat /proc/buddyinfo
> cat /proc/iomem
>
> in the guest before and after the operations?
Ok, that was a stupid mistake on my side - I've run out of host memory.
With that fixed, it seems to work, on 5.16.18 too.
--
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2022-04-06 12:59 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2022-03-31 3:51 Increasing domain memory beyond initial maxmem Marek Marczykowski-Górecki
2022-03-31 6:41 ` Juergen Gross
2022-03-31 12:01 ` Marek Marczykowski-Górecki
2022-03-31 12:22 ` Juergen Gross
2022-03-31 12:36 ` Marek Marczykowski-Górecki
2022-04-05 11:03 ` Juergen Gross
2022-04-05 16:24 ` Marek Marczykowski-Górecki
2022-04-06 5:13 ` Juergen Gross
2022-04-06 12:58 ` Marek Marczykowski-Górecki
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.