Re: Increasing domain memory beyond initial maxmem

["Marek Marczykowski-Górecki" <marmarek@invisiblethingslab.com>]

[-- Attachment #1: Type: text/plain, Size: 8338 bytes --]

On Thu, Mar 31, 2022 at 08:41:19AM +0200, Juergen Gross wrote:
> On 31.03.22 05:51, Marek Marczykowski-Górecki wrote:
> > Hi,
> > 
> > I'm trying to make use of CONFIG_XEN_BALLOON_MEMORY_HOTPLUG=y to increase
> > domain memory beyond initial maxmem, but I hit few issues.
> > 
> > A little context: domains in Qubes OS start with rather little memory
> > (400MB by default) but maxmem set higher (4GB by default). Then, there is
> > qmemman daemon, that adjust balloon targets for domains, based on (among
> > other things) demand reported by the domains themselves. There is also a
> > little swap, to mitigate qmemman latency (few hundreds ms at worst).
> > This initial memory < maxmmem in case of PVH / HVM makes use of PoD
> > which I'm trying to get rid of. But also, IIUC Linux will waste some
> > memory for bookkeeping based on maxmem, not actually usable memory.
> > 
> > First issue: after using `xl mem-max`, `xl mem-set` still refuses to
> > increase memory more than initial maxmem. That's because xl mem-max does
> > not update 'memory/static-max' xenstore node. This one is easy to work
> > around.
> > 
> > Then, the actual hotplug fails on the domU side with:
> > 
> > [   50.004734] xen-balloon: vmemmap alloc failure: order:9, mode:0x4cc0(GFP_KERNEL|__GFP_RETRY_MAYFAIL), nodemask=(null),cpuset=/,mems_allowed=0
> > [   50.004774] CPU: 1 PID: 34 Comm: xen-balloon Not tainted 5.16.15-1.37.fc32.qubes.x86_64 #1
> > [   50.004792] Call Trace:
> > [   50.004799]  <TASK>
> > [   50.004808]  dump_stack_lvl+0x48/0x5e
> > [   50.004821]  warn_alloc+0x162/0x190
> > [   50.004832]  ? __alloc_pages+0x1fa/0x230
> > [   50.004842]  vmemmap_alloc_block+0x11c/0x1c5
> > [   50.004856]  vmemmap_populate_hugepages+0x185/0x519
> > [   50.004868]  vmemmap_populate+0x9e/0x16c
> > [   50.004878]  __populate_section_memmap+0x6a/0xb1
> > [   50.004890]  section_activate+0x20a/0x278
> > [   50.004901]  sparse_add_section+0x70/0x160
> > [   50.004911]  __add_pages+0xc3/0x150
> > [   50.004921]  add_pages+0x12/0x60
> > [   50.004931]  add_memory_resource+0x12b/0x320
> > [   50.004943]  reserve_additional_memory+0x10c/0x150
> > [   50.004958]  balloon_thread+0x206/0x360
> > [   50.004968]  ? do_wait_intr_irq+0xa0/0xa0
> > [   50.004978]  ? decrease_reservation.constprop.0+0x2e0/0x2e0
> > [   50.004991]  kthread+0x16b/0x190
> > [   50.005001]  ? set_kthread_struct+0x40/0x40
> > [   50.005011]  ret_from_fork+0x22/0x30
> > [   50.005022]  </TASK>
> > 
> > Full dmesg: https://gist.github.com/marmarek/72dd1f9dbdd63cfe479c94a3f4392b45
> > 
> > After the above, `free` reports correct size (1GB in this case), but
> > that memory seems to be unusable really. "used" is kept low, and soon
> > OOM-killer kicks in.
> > 
> > I know the initial 400MB is not much for a full Linux, with X11 etc. But
> > I wouldn't expect it to fail this way when _adding_ memory.
> > 
> > I've tried also with initial 800MB. In this case, I do not get "alloc
> > failure" any more, but monitoring `free`, the extra memory still doesn't
> > seem to be used.
> > 
> > Any ideas?
> > 
> 
> I can't reproduce that.
> 
> I started a guest with 8GB of memory, in the guest I'm seeing:
> 
> # uname -a
> Linux linux-d1cy 5.17.0-rc5-default+ #406 SMP PREEMPT Mon Feb 21 09:31:12
> CET 2022 x86_64 x86_64 x86_64 GNU/Linux
> # free
>         total     used      free   shared  buff/cache   available
> Mem:  8178260    71628   8023300     8560       83332     8010196
> Swap: 2097132        0   2097132
> 
> Then I'm raising the memory for the guest in dom0:
> 
> # xl list
> Name                ID   Mem VCPUs      State   Time(s)
> Domain-0             0  2634     8     r-----    1016.5
> Xenstore             1    31     1     -b----       0.9
> sle15sp1             3  8190     6     -b----     184.6
> # xl mem-max 3 10000
> # xenstore-write /local/domain/3/memory/static-max 10240000
> # xl mem-set 3 10000
> # xl list
> Name                ID   Mem VCPUs      State   Time(s)
> Domain-0             0  2634     8     r-----    1018.5
> Xenstore             1    31     1     -b----       1.0
> sle15sp1             3 10000     6     -b----     186.7
> 
> In the guest I get now:
> 
> # free
>         total     used     free   shared  buff/cache   available
> Mem: 10031700   110904  9734172     8560      186624     9814344
> Swap: 2097132        0  2097132
> 
> And after using lots of memory via a ramdisk:
> 
> # free
>         total     used     free   shared  buff/cache   available
> Mem: 10031700   116660  1663840  7181776     8251200     2635372
> Swap: 2097132        0  2097132
> 
> You can see buff/cache is now larger than the initial total memory
> and free is lower than the added memory amount.

Hmm, I have a different behavior:

I'm starting with 800M

# uname -a
Linux personal 5.16.15-1.37.fc32.qubes.x86_64 #1 SMP PREEMPT Tue Mar 22 12:59:53 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
# free -m
              total        used        free      shared  buff/cache   available
Mem:            740         209         278           2         252         415
Swap:          1023           0        1023

Then raising to ~2GB:

[root@dom0 ~]# xl list
Name                                        ID   Mem VCPUs	State	Time(s)
Domain-0                                     0  4082     6     r-----  184271.3
(...)
personal                                    21   800     2     -b----       4.8
[root@dom0 ~]# xl mem-max personal 2048
[root@dom0 ~]# xenstore-write /local/domain/$(xl domid personal)/memory/static-max $((2048*1024))
[root@dom0 ~]# xl mem-set personal 2000
[root@dom0 ~]# xenstore-ls -fp /local/domain/$(xl domid personal)/memory
/local/domain/21/memory/static-max = "2097152"   (n0,r21)
/local/domain/21/memory/target = "2048001"   (n0,r21)
/local/domain/21/memory/videoram = "-1"   (n0,r21)

And then observe inside domU:
[root@personal ~]# free -m
              total        used        free      shared  buff/cache   available
Mem:           1940         235        1452           2         252        1585
Swap:          1023           0        1023

So far so good. But when trying to actually use it, it doesn't work:

[root@personal ~]# free -m
              total        used        free      shared  buff/cache   available
Mem:           1940         196        1240         454         503        1206
Swap:          1023         472         551

As you can see, all the new memory is still in "free", and swap is used
instead.


There is also /proc/meminfo (state before filling ramdisk), if that
would give some hints:
[root@personal ~]# cat /proc/meminfo
MemTotal:        1986800 kB
MemFree:         1487116 kB
MemAvailable:    1624060 kB
Buffers:           26236 kB
Cached:           207268 kB
SwapCached:            0 kB
Active:            74828 kB
Inactive:         258724 kB
Active(anon):       1008 kB
Inactive(anon):   101668 kB
Active(file):      73820 kB
Inactive(file):   157056 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:       1048572 kB
SwapFree:        1048572 kB
Dirty:               216 kB
Writeback:             0 kB
AnonPages:        100184 kB
Mapped:           117472 kB
Shmem:              2628 kB
KReclaimable:      24960 kB
Slab:              52136 kB
SReclaimable:      24960 kB
SUnreclaim:        27176 kB
KernelStack:        3120 kB
PageTables:         4364 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     2041972 kB
Committed_AS:     825816 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       10064 kB
VmallocChunk:          0 kB
Percpu:             1240 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
FileHugePages:         0 kB
FilePmdMapped:         0 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:       79872 kB
DirectMap2M:     1132544 kB
DirectMap1G:     1048576 kB


-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]