* Increasing domain memory beyond initial maxmem @ 2022-03-31 3:51 Marek Marczykowski-Górecki 2022-03-31 6:41 ` Juergen Gross 0 siblings, 1 reply; 9+ messages in thread From: Marek Marczykowski-Górecki @ 2022-03-31 3:51 UTC (permalink / raw) To: xen-devel [-- Attachment #1: Type: text/plain, Size: 2839 bytes --] Hi, I'm trying to make use of CONFIG_XEN_BALLOON_MEMORY_HOTPLUG=y to increase domain memory beyond initial maxmem, but I hit few issues. A little context: domains in Qubes OS start with rather little memory (400MB by default) but maxmem set higher (4GB by default). Then, there is qmemman daemon, that adjust balloon targets for domains, based on (among other things) demand reported by the domains themselves. There is also a little swap, to mitigate qmemman latency (few hundreds ms at worst). This initial memory < maxmmem in case of PVH / HVM makes use of PoD which I'm trying to get rid of. But also, IIUC Linux will waste some memory for bookkeeping based on maxmem, not actually usable memory. First issue: after using `xl mem-max`, `xl mem-set` still refuses to increase memory more than initial maxmem. That's because xl mem-max does not update 'memory/static-max' xenstore node. This one is easy to work around. Then, the actual hotplug fails on the domU side with: [ 50.004734] xen-balloon: vmemmap alloc failure: order:9, mode:0x4cc0(GFP_KERNEL|__GFP_RETRY_MAYFAIL), nodemask=(null),cpuset=/,mems_allowed=0 [ 50.004774] CPU: 1 PID: 34 Comm: xen-balloon Not tainted 5.16.15-1.37.fc32.qubes.x86_64 #1 [ 50.004792] Call Trace: [ 50.004799] <TASK> [ 50.004808] dump_stack_lvl+0x48/0x5e [ 50.004821] warn_alloc+0x162/0x190 [ 50.004832] ? __alloc_pages+0x1fa/0x230 [ 50.004842] vmemmap_alloc_block+0x11c/0x1c5 [ 50.004856] vmemmap_populate_hugepages+0x185/0x519 [ 50.004868] vmemmap_populate+0x9e/0x16c [ 50.004878] __populate_section_memmap+0x6a/0xb1 [ 50.004890] section_activate+0x20a/0x278 [ 50.004901] sparse_add_section+0x70/0x160 [ 50.004911] __add_pages+0xc3/0x150 [ 50.004921] add_pages+0x12/0x60 [ 50.004931] add_memory_resource+0x12b/0x320 [ 50.004943] reserve_additional_memory+0x10c/0x150 [ 50.004958] balloon_thread+0x206/0x360 [ 50.004968] ? do_wait_intr_irq+0xa0/0xa0 [ 50.004978] ? decrease_reservation.constprop.0+0x2e0/0x2e0 [ 50.004991] kthread+0x16b/0x190 [ 50.005001] ? set_kthread_struct+0x40/0x40 [ 50.005011] ret_from_fork+0x22/0x30 [ 50.005022] </TASK> Full dmesg: https://gist.github.com/marmarek/72dd1f9dbdd63cfe479c94a3f4392b45 After the above, `free` reports correct size (1GB in this case), but that memory seems to be unusable really. "used" is kept low, and soon OOM-killer kicks in. I know the initial 400MB is not much for a full Linux, with X11 etc. But I wouldn't expect it to fail this way when _adding_ memory. I've tried also with initial 800MB. In this case, I do not get "alloc failure" any more, but monitoring `free`, the extra memory still doesn't seem to be used. Any ideas? -- Best Regards, Marek Marczykowski-Górecki Invisible Things Lab [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Increasing domain memory beyond initial maxmem 2022-03-31 3:51 Increasing domain memory beyond initial maxmem Marek Marczykowski-Górecki @ 2022-03-31 6:41 ` Juergen Gross 2022-03-31 12:01 ` Marek Marczykowski-Górecki 0 siblings, 1 reply; 9+ messages in thread From: Juergen Gross @ 2022-03-31 6:41 UTC (permalink / raw) To: Marek Marczykowski-Górecki, xen-devel [-- Attachment #1.1.1: Type: text/plain, Size: 4545 bytes --] On 31.03.22 05:51, Marek Marczykowski-Górecki wrote: > Hi, > > I'm trying to make use of CONFIG_XEN_BALLOON_MEMORY_HOTPLUG=y to increase > domain memory beyond initial maxmem, but I hit few issues. > > A little context: domains in Qubes OS start with rather little memory > (400MB by default) but maxmem set higher (4GB by default). Then, there is > qmemman daemon, that adjust balloon targets for domains, based on (among > other things) demand reported by the domains themselves. There is also a > little swap, to mitigate qmemman latency (few hundreds ms at worst). > This initial memory < maxmmem in case of PVH / HVM makes use of PoD > which I'm trying to get rid of. But also, IIUC Linux will waste some > memory for bookkeeping based on maxmem, not actually usable memory. > > First issue: after using `xl mem-max`, `xl mem-set` still refuses to > increase memory more than initial maxmem. That's because xl mem-max does > not update 'memory/static-max' xenstore node. This one is easy to work > around. > > Then, the actual hotplug fails on the domU side with: > > [ 50.004734] xen-balloon: vmemmap alloc failure: order:9, mode:0x4cc0(GFP_KERNEL|__GFP_RETRY_MAYFAIL), nodemask=(null),cpuset=/,mems_allowed=0 > [ 50.004774] CPU: 1 PID: 34 Comm: xen-balloon Not tainted 5.16.15-1.37.fc32.qubes.x86_64 #1 > [ 50.004792] Call Trace: > [ 50.004799] <TASK> > [ 50.004808] dump_stack_lvl+0x48/0x5e > [ 50.004821] warn_alloc+0x162/0x190 > [ 50.004832] ? __alloc_pages+0x1fa/0x230 > [ 50.004842] vmemmap_alloc_block+0x11c/0x1c5 > [ 50.004856] vmemmap_populate_hugepages+0x185/0x519 > [ 50.004868] vmemmap_populate+0x9e/0x16c > [ 50.004878] __populate_section_memmap+0x6a/0xb1 > [ 50.004890] section_activate+0x20a/0x278 > [ 50.004901] sparse_add_section+0x70/0x160 > [ 50.004911] __add_pages+0xc3/0x150 > [ 50.004921] add_pages+0x12/0x60 > [ 50.004931] add_memory_resource+0x12b/0x320 > [ 50.004943] reserve_additional_memory+0x10c/0x150 > [ 50.004958] balloon_thread+0x206/0x360 > [ 50.004968] ? do_wait_intr_irq+0xa0/0xa0 > [ 50.004978] ? decrease_reservation.constprop.0+0x2e0/0x2e0 > [ 50.004991] kthread+0x16b/0x190 > [ 50.005001] ? set_kthread_struct+0x40/0x40 > [ 50.005011] ret_from_fork+0x22/0x30 > [ 50.005022] </TASK> > > Full dmesg: https://gist.github.com/marmarek/72dd1f9dbdd63cfe479c94a3f4392b45 > > After the above, `free` reports correct size (1GB in this case), but > that memory seems to be unusable really. "used" is kept low, and soon > OOM-killer kicks in. > > I know the initial 400MB is not much for a full Linux, with X11 etc. But > I wouldn't expect it to fail this way when _adding_ memory. > > I've tried also with initial 800MB. In this case, I do not get "alloc > failure" any more, but monitoring `free`, the extra memory still doesn't > seem to be used. > > Any ideas? > I can't reproduce that. I started a guest with 8GB of memory, in the guest I'm seeing: # uname -a Linux linux-d1cy 5.17.0-rc5-default+ #406 SMP PREEMPT Mon Feb 21 09:31:12 CET 2022 x86_64 x86_64 x86_64 GNU/Linux # free total used free shared buff/cache available Mem: 8178260 71628 8023300 8560 83332 8010196 Swap: 2097132 0 2097132 Then I'm raising the memory for the guest in dom0: # xl list Name ID Mem VCPUs State Time(s) Domain-0 0 2634 8 r----- 1016.5 Xenstore 1 31 1 -b---- 0.9 sle15sp1 3 8190 6 -b---- 184.6 # xl mem-max 3 10000 # xenstore-write /local/domain/3/memory/static-max 10240000 # xl mem-set 3 10000 # xl list Name ID Mem VCPUs State Time(s) Domain-0 0 2634 8 r----- 1018.5 Xenstore 1 31 1 -b---- 1.0 sle15sp1 3 10000 6 -b---- 186.7 In the guest I get now: # free total used free shared buff/cache available Mem: 10031700 110904 9734172 8560 186624 9814344 Swap: 2097132 0 2097132 And after using lots of memory via a ramdisk: # free total used free shared buff/cache available Mem: 10031700 116660 1663840 7181776 8251200 2635372 Swap: 2097132 0 2097132 You can see buff/cache is now larger than the initial total memory and free is lower than the added memory amount. Juergen [-- Attachment #1.1.2: OpenPGP public key --] [-- Type: application/pgp-keys, Size: 3149 bytes --] [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 495 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Increasing domain memory beyond initial maxmem 2022-03-31 6:41 ` Juergen Gross @ 2022-03-31 12:01 ` Marek Marczykowski-Górecki 2022-03-31 12:22 ` Juergen Gross 0 siblings, 1 reply; 9+ messages in thread From: Marek Marczykowski-Górecki @ 2022-03-31 12:01 UTC (permalink / raw) To: Juergen Gross; +Cc: xen-devel [-- Attachment #1: Type: text/plain, Size: 8338 bytes --] On Thu, Mar 31, 2022 at 08:41:19AM +0200, Juergen Gross wrote: > On 31.03.22 05:51, Marek Marczykowski-Górecki wrote: > > Hi, > > > > I'm trying to make use of CONFIG_XEN_BALLOON_MEMORY_HOTPLUG=y to increase > > domain memory beyond initial maxmem, but I hit few issues. > > > > A little context: domains in Qubes OS start with rather little memory > > (400MB by default) but maxmem set higher (4GB by default). Then, there is > > qmemman daemon, that adjust balloon targets for domains, based on (among > > other things) demand reported by the domains themselves. There is also a > > little swap, to mitigate qmemman latency (few hundreds ms at worst). > > This initial memory < maxmmem in case of PVH / HVM makes use of PoD > > which I'm trying to get rid of. But also, IIUC Linux will waste some > > memory for bookkeeping based on maxmem, not actually usable memory. > > > > First issue: after using `xl mem-max`, `xl mem-set` still refuses to > > increase memory more than initial maxmem. That's because xl mem-max does > > not update 'memory/static-max' xenstore node. This one is easy to work > > around. > > > > Then, the actual hotplug fails on the domU side with: > > > > [ 50.004734] xen-balloon: vmemmap alloc failure: order:9, mode:0x4cc0(GFP_KERNEL|__GFP_RETRY_MAYFAIL), nodemask=(null),cpuset=/,mems_allowed=0 > > [ 50.004774] CPU: 1 PID: 34 Comm: xen-balloon Not tainted 5.16.15-1.37.fc32.qubes.x86_64 #1 > > [ 50.004792] Call Trace: > > [ 50.004799] <TASK> > > [ 50.004808] dump_stack_lvl+0x48/0x5e > > [ 50.004821] warn_alloc+0x162/0x190 > > [ 50.004832] ? __alloc_pages+0x1fa/0x230 > > [ 50.004842] vmemmap_alloc_block+0x11c/0x1c5 > > [ 50.004856] vmemmap_populate_hugepages+0x185/0x519 > > [ 50.004868] vmemmap_populate+0x9e/0x16c > > [ 50.004878] __populate_section_memmap+0x6a/0xb1 > > [ 50.004890] section_activate+0x20a/0x278 > > [ 50.004901] sparse_add_section+0x70/0x160 > > [ 50.004911] __add_pages+0xc3/0x150 > > [ 50.004921] add_pages+0x12/0x60 > > [ 50.004931] add_memory_resource+0x12b/0x320 > > [ 50.004943] reserve_additional_memory+0x10c/0x150 > > [ 50.004958] balloon_thread+0x206/0x360 > > [ 50.004968] ? do_wait_intr_irq+0xa0/0xa0 > > [ 50.004978] ? decrease_reservation.constprop.0+0x2e0/0x2e0 > > [ 50.004991] kthread+0x16b/0x190 > > [ 50.005001] ? set_kthread_struct+0x40/0x40 > > [ 50.005011] ret_from_fork+0x22/0x30 > > [ 50.005022] </TASK> > > > > Full dmesg: https://gist.github.com/marmarek/72dd1f9dbdd63cfe479c94a3f4392b45 > > > > After the above, `free` reports correct size (1GB in this case), but > > that memory seems to be unusable really. "used" is kept low, and soon > > OOM-killer kicks in. > > > > I know the initial 400MB is not much for a full Linux, with X11 etc. But > > I wouldn't expect it to fail this way when _adding_ memory. > > > > I've tried also with initial 800MB. In this case, I do not get "alloc > > failure" any more, but monitoring `free`, the extra memory still doesn't > > seem to be used. > > > > Any ideas? > > > > I can't reproduce that. > > I started a guest with 8GB of memory, in the guest I'm seeing: > > # uname -a > Linux linux-d1cy 5.17.0-rc5-default+ #406 SMP PREEMPT Mon Feb 21 09:31:12 > CET 2022 x86_64 x86_64 x86_64 GNU/Linux > # free > total used free shared buff/cache available > Mem: 8178260 71628 8023300 8560 83332 8010196 > Swap: 2097132 0 2097132 > > Then I'm raising the memory for the guest in dom0: > > # xl list > Name ID Mem VCPUs State Time(s) > Domain-0 0 2634 8 r----- 1016.5 > Xenstore 1 31 1 -b---- 0.9 > sle15sp1 3 8190 6 -b---- 184.6 > # xl mem-max 3 10000 > # xenstore-write /local/domain/3/memory/static-max 10240000 > # xl mem-set 3 10000 > # xl list > Name ID Mem VCPUs State Time(s) > Domain-0 0 2634 8 r----- 1018.5 > Xenstore 1 31 1 -b---- 1.0 > sle15sp1 3 10000 6 -b---- 186.7 > > In the guest I get now: > > # free > total used free shared buff/cache available > Mem: 10031700 110904 9734172 8560 186624 9814344 > Swap: 2097132 0 2097132 > > And after using lots of memory via a ramdisk: > > # free > total used free shared buff/cache available > Mem: 10031700 116660 1663840 7181776 8251200 2635372 > Swap: 2097132 0 2097132 > > You can see buff/cache is now larger than the initial total memory > and free is lower than the added memory amount. Hmm, I have a different behavior: I'm starting with 800M # uname -a Linux personal 5.16.15-1.37.fc32.qubes.x86_64 #1 SMP PREEMPT Tue Mar 22 12:59:53 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux # free -m total used free shared buff/cache available Mem: 740 209 278 2 252 415 Swap: 1023 0 1023 Then raising to ~2GB: [root@dom0 ~]# xl list Name ID Mem VCPUs State Time(s) Domain-0 0 4082 6 r----- 184271.3 (...) personal 21 800 2 -b---- 4.8 [root@dom0 ~]# xl mem-max personal 2048 [root@dom0 ~]# xenstore-write /local/domain/$(xl domid personal)/memory/static-max $((2048*1024)) [root@dom0 ~]# xl mem-set personal 2000 [root@dom0 ~]# xenstore-ls -fp /local/domain/$(xl domid personal)/memory /local/domain/21/memory/static-max = "2097152" (n0,r21) /local/domain/21/memory/target = "2048001" (n0,r21) /local/domain/21/memory/videoram = "-1" (n0,r21) And then observe inside domU: [root@personal ~]# free -m total used free shared buff/cache available Mem: 1940 235 1452 2 252 1585 Swap: 1023 0 1023 So far so good. But when trying to actually use it, it doesn't work: [root@personal ~]# free -m total used free shared buff/cache available Mem: 1940 196 1240 454 503 1206 Swap: 1023 472 551 As you can see, all the new memory is still in "free", and swap is used instead. There is also /proc/meminfo (state before filling ramdisk), if that would give some hints: [root@personal ~]# cat /proc/meminfo MemTotal: 1986800 kB MemFree: 1487116 kB MemAvailable: 1624060 kB Buffers: 26236 kB Cached: 207268 kB SwapCached: 0 kB Active: 74828 kB Inactive: 258724 kB Active(anon): 1008 kB Inactive(anon): 101668 kB Active(file): 73820 kB Inactive(file): 157056 kB Unevictable: 0 kB Mlocked: 0 kB SwapTotal: 1048572 kB SwapFree: 1048572 kB Dirty: 216 kB Writeback: 0 kB AnonPages: 100184 kB Mapped: 117472 kB Shmem: 2628 kB KReclaimable: 24960 kB Slab: 52136 kB SReclaimable: 24960 kB SUnreclaim: 27176 kB KernelStack: 3120 kB PageTables: 4364 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 2041972 kB Committed_AS: 825816 kB VmallocTotal: 34359738367 kB VmallocUsed: 10064 kB VmallocChunk: 0 kB Percpu: 1240 kB HardwareCorrupted: 0 kB AnonHugePages: 0 kB ShmemHugePages: 0 kB ShmemPmdMapped: 0 kB FileHugePages: 0 kB FilePmdMapped: 0 kB CmaTotal: 0 kB CmaFree: 0 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB Hugetlb: 0 kB DirectMap4k: 79872 kB DirectMap2M: 1132544 kB DirectMap1G: 1048576 kB -- Best Regards, Marek Marczykowski-Górecki Invisible Things Lab [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Increasing domain memory beyond initial maxmem 2022-03-31 12:01 ` Marek Marczykowski-Górecki @ 2022-03-31 12:22 ` Juergen Gross 2022-03-31 12:36 ` Marek Marczykowski-Górecki 0 siblings, 1 reply; 9+ messages in thread From: Juergen Gross @ 2022-03-31 12:22 UTC (permalink / raw) To: Marek Marczykowski-Górecki; +Cc: xen-devel [-- Attachment #1.1.1: Type: text/plain, Size: 7619 bytes --] On 31.03.22 14:01, Marek Marczykowski-Górecki wrote: > On Thu, Mar 31, 2022 at 08:41:19AM +0200, Juergen Gross wrote: >> On 31.03.22 05:51, Marek Marczykowski-Górecki wrote: >>> Hi, >>> >>> I'm trying to make use of CONFIG_XEN_BALLOON_MEMORY_HOTPLUG=y to increase >>> domain memory beyond initial maxmem, but I hit few issues. >>> >>> A little context: domains in Qubes OS start with rather little memory >>> (400MB by default) but maxmem set higher (4GB by default). Then, there is >>> qmemman daemon, that adjust balloon targets for domains, based on (among >>> other things) demand reported by the domains themselves. There is also a >>> little swap, to mitigate qmemman latency (few hundreds ms at worst). >>> This initial memory < maxmmem in case of PVH / HVM makes use of PoD >>> which I'm trying to get rid of. But also, IIUC Linux will waste some >>> memory for bookkeeping based on maxmem, not actually usable memory. >>> >>> First issue: after using `xl mem-max`, `xl mem-set` still refuses to >>> increase memory more than initial maxmem. That's because xl mem-max does >>> not update 'memory/static-max' xenstore node. This one is easy to work >>> around. >>> >>> Then, the actual hotplug fails on the domU side with: >>> >>> [ 50.004734] xen-balloon: vmemmap alloc failure: order:9, mode:0x4cc0(GFP_KERNEL|__GFP_RETRY_MAYFAIL), nodemask=(null),cpuset=/,mems_allowed=0 >>> [ 50.004774] CPU: 1 PID: 34 Comm: xen-balloon Not tainted 5.16.15-1.37.fc32.qubes.x86_64 #1 >>> [ 50.004792] Call Trace: >>> [ 50.004799] <TASK> >>> [ 50.004808] dump_stack_lvl+0x48/0x5e >>> [ 50.004821] warn_alloc+0x162/0x190 >>> [ 50.004832] ? __alloc_pages+0x1fa/0x230 >>> [ 50.004842] vmemmap_alloc_block+0x11c/0x1c5 >>> [ 50.004856] vmemmap_populate_hugepages+0x185/0x519 >>> [ 50.004868] vmemmap_populate+0x9e/0x16c >>> [ 50.004878] __populate_section_memmap+0x6a/0xb1 >>> [ 50.004890] section_activate+0x20a/0x278 >>> [ 50.004901] sparse_add_section+0x70/0x160 >>> [ 50.004911] __add_pages+0xc3/0x150 >>> [ 50.004921] add_pages+0x12/0x60 >>> [ 50.004931] add_memory_resource+0x12b/0x320 >>> [ 50.004943] reserve_additional_memory+0x10c/0x150 >>> [ 50.004958] balloon_thread+0x206/0x360 >>> [ 50.004968] ? do_wait_intr_irq+0xa0/0xa0 >>> [ 50.004978] ? decrease_reservation.constprop.0+0x2e0/0x2e0 >>> [ 50.004991] kthread+0x16b/0x190 >>> [ 50.005001] ? set_kthread_struct+0x40/0x40 >>> [ 50.005011] ret_from_fork+0x22/0x30 >>> [ 50.005022] </TASK> >>> >>> Full dmesg: https://gist.github.com/marmarek/72dd1f9dbdd63cfe479c94a3f4392b45 >>> >>> After the above, `free` reports correct size (1GB in this case), but >>> that memory seems to be unusable really. "used" is kept low, and soon >>> OOM-killer kicks in. >>> >>> I know the initial 400MB is not much for a full Linux, with X11 etc. But >>> I wouldn't expect it to fail this way when _adding_ memory. >>> >>> I've tried also with initial 800MB. In this case, I do not get "alloc >>> failure" any more, but monitoring `free`, the extra memory still doesn't >>> seem to be used. >>> >>> Any ideas? >>> >> >> I can't reproduce that. >> >> I started a guest with 8GB of memory, in the guest I'm seeing: >> >> # uname -a >> Linux linux-d1cy 5.17.0-rc5-default+ #406 SMP PREEMPT Mon Feb 21 09:31:12 >> CET 2022 x86_64 x86_64 x86_64 GNU/Linux >> # free >> total used free shared buff/cache available >> Mem: 8178260 71628 8023300 8560 83332 8010196 >> Swap: 2097132 0 2097132 >> >> Then I'm raising the memory for the guest in dom0: >> >> # xl list >> Name ID Mem VCPUs State Time(s) >> Domain-0 0 2634 8 r----- 1016.5 >> Xenstore 1 31 1 -b---- 0.9 >> sle15sp1 3 8190 6 -b---- 184.6 >> # xl mem-max 3 10000 >> # xenstore-write /local/domain/3/memory/static-max 10240000 >> # xl mem-set 3 10000 >> # xl list >> Name ID Mem VCPUs State Time(s) >> Domain-0 0 2634 8 r----- 1018.5 >> Xenstore 1 31 1 -b---- 1.0 >> sle15sp1 3 10000 6 -b---- 186.7 >> >> In the guest I get now: >> >> # free >> total used free shared buff/cache available >> Mem: 10031700 110904 9734172 8560 186624 9814344 >> Swap: 2097132 0 2097132 >> >> And after using lots of memory via a ramdisk: >> >> # free >> total used free shared buff/cache available >> Mem: 10031700 116660 1663840 7181776 8251200 2635372 >> Swap: 2097132 0 2097132 >> >> You can see buff/cache is now larger than the initial total memory >> and free is lower than the added memory amount. > > Hmm, I have a different behavior: > > I'm starting with 800M > > # uname -a > Linux personal 5.16.15-1.37.fc32.qubes.x86_64 #1 SMP PREEMPT Tue Mar 22 12:59:53 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux > # free -m > total used free shared buff/cache available > Mem: 740 209 278 2 252 415 > Swap: 1023 0 1023 > > Then raising to ~2GB: > > [root@dom0 ~]# xl list > Name ID Mem VCPUs State Time(s) > Domain-0 0 4082 6 r----- 184271.3 > (...) > personal 21 800 2 -b---- 4.8 > [root@dom0 ~]# xl mem-max personal 2048 > [root@dom0 ~]# xenstore-write /local/domain/$(xl domid personal)/memory/static-max $((2048*1024)) > [root@dom0 ~]# xl mem-set personal 2000 > [root@dom0 ~]# xenstore-ls -fp /local/domain/$(xl domid personal)/memory > /local/domain/21/memory/static-max = "2097152" (n0,r21) > /local/domain/21/memory/target = "2048001" (n0,r21) > /local/domain/21/memory/videoram = "-1" (n0,r21) > > And then observe inside domU: > [root@personal ~]# free -m > total used free shared buff/cache available > Mem: 1940 235 1452 2 252 1585 > Swap: 1023 0 1023 > > So far so good. But when trying to actually use it, it doesn't work: > > [root@personal ~]# free -m > total used free shared buff/cache available > Mem: 1940 196 1240 454 503 1206 > Swap: 1023 472 551 > > As you can see, all the new memory is still in "free", and swap is used > instead. Hmm, weird. Maybe some kernel config differences, or other udev rules (memory onlining is done via udev in my guest)? I'm seeing: # zgrep MEMORY_HOTPLUG /proc/config.gz CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y CONFIG_MEMORY_HOTPLUG=y # CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE is not set CONFIG_XEN_BALLOON_MEMORY_HOTPLUG=y CONFIG_XEN_MEMORY_HOTPLUG_LIMIT=512 The relevant udev rule seems to be: SUBSYSTEM=="memory", ACTION=="add", PROGRAM=="/bin/sh -c '/usr/bin/systemd-detect-virt || :'", RESULT!="zvm", ATTR{state}=="offline", \ ATTR{state}="online" What type of guest are you using? Mine was a PVH guest. > There is also /proc/meminfo (state before filling ramdisk), if that > would give some hints: > [root@personal ~]# cat /proc/meminfo ... No, I don't think this is helping. At least not me. Juergen [-- Attachment #1.1.2: OpenPGP public key --] [-- Type: application/pgp-keys, Size: 3149 bytes --] [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 495 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Increasing domain memory beyond initial maxmem 2022-03-31 12:22 ` Juergen Gross @ 2022-03-31 12:36 ` Marek Marczykowski-Górecki 2022-04-05 11:03 ` Juergen Gross 0 siblings, 1 reply; 9+ messages in thread From: Marek Marczykowski-Górecki @ 2022-03-31 12:36 UTC (permalink / raw) To: Juergen Gross; +Cc: xen-devel [-- Attachment #1: Type: text/plain, Size: 1069 bytes --] On Thu, Mar 31, 2022 at 02:22:03PM +0200, Juergen Gross wrote: > Maybe some kernel config differences, or other udev rules (memory onlining > is done via udev in my guest)? > > I'm seeing: > > # zgrep MEMORY_HOTPLUG /proc/config.gz > CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y > CONFIG_MEMORY_HOTPLUG=y > # CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE is not set > CONFIG_XEN_BALLOON_MEMORY_HOTPLUG=y > CONFIG_XEN_MEMORY_HOTPLUG_LIMIT=512 I have: # zgrep MEMORY_HOTPLUG /proc/config.gz CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y CONFIG_MEMORY_HOTPLUG=y CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y CONFIG_XEN_BALLOON_MEMORY_HOTPLUG=y CONFIG_XEN_MEMORY_HOTPLUG_LIMIT=512 Not sure if relevant, but I also have: CONFIG_XEN_UNPOPULATED_ALLOC=y on top of that, I have a similar udev rule too: SUBSYSTEM=="memory", ACTION=="add", ATTR{state}=="offline", ATTR{state}="online" But I don't think they are conflicting. > What type of guest are you using? Mine was a PVH guest. PVH here too. -- Best Regards, Marek Marczykowski-Górecki Invisible Things Lab [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Increasing domain memory beyond initial maxmem 2022-03-31 12:36 ` Marek Marczykowski-Górecki @ 2022-04-05 11:03 ` Juergen Gross 2022-04-05 16:24 ` Marek Marczykowski-Górecki 0 siblings, 1 reply; 9+ messages in thread From: Juergen Gross @ 2022-04-05 11:03 UTC (permalink / raw) To: Marek Marczykowski-Górecki; +Cc: xen-devel [-- Attachment #1.1.1: Type: text/plain, Size: 1199 bytes --] Hi Marek, On 31.03.22 14:36, Marek Marczykowski-Górecki wrote: > On Thu, Mar 31, 2022 at 02:22:03PM +0200, Juergen Gross wrote: >> Maybe some kernel config differences, or other udev rules (memory onlining >> is done via udev in my guest)? >> >> I'm seeing: >> >> # zgrep MEMORY_HOTPLUG /proc/config.gz >> CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y >> CONFIG_MEMORY_HOTPLUG=y >> # CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE is not set >> CONFIG_XEN_BALLOON_MEMORY_HOTPLUG=y >> CONFIG_XEN_MEMORY_HOTPLUG_LIMIT=512 > > I have: > # zgrep MEMORY_HOTPLUG /proc/config.gz > CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y > CONFIG_MEMORY_HOTPLUG=y > CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y > CONFIG_XEN_BALLOON_MEMORY_HOTPLUG=y > CONFIG_XEN_MEMORY_HOTPLUG_LIMIT=512 > > Not sure if relevant, but I also have: > CONFIG_XEN_UNPOPULATED_ALLOC=y > > on top of that, I have a similar udev rule too: > > SUBSYSTEM=="memory", ACTION=="add", ATTR{state}=="offline", ATTR{state}="online" > > But I don't think they are conflicting. > >> What type of guest are you using? Mine was a PVH guest. > > PVH here too. Would you like to try the attached patch? It seemed to work for me. Juergen [-- Attachment #1.1.2: 0001-xen-balloon-fix-page-onlining-when-populating-new-zo.patch --] [-- Type: text/x-patch, Size: 5744 bytes --] From a605232115a9c3d3f8103d0833b149ff22956c4b Mon Sep 17 00:00:00 2001 From: Juergen Gross <jgross@suse.com> To: linux-kernel@vger.kernel.org Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com> Cc: Juergen Gross <jgross@suse.com> Cc: Stefano Stabellini <sstabellini@kernel.org> Cc: xen-devel@lists.xenproject.org Date: Tue, 5 Apr 2022 12:43:41 +0200 Subject: [PATCH] xen/balloon: fix page onlining when populating new zone MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit When onlining a new memory page in a guest the Xen balloon driver is adding it to the ballooned pages instead making it available to be used immediately. This is meant to enable to add a new upper memory limit to a guest via hotplugging memory, without having to assign the new memory in one go. In case the upper memory limit will be raised above 4G, the new memory will populate the ZONE_NORMAL memory zone, which wasn't populated before. The newly populated zone won't be added to the list of zones looked at by the page allocator though, as only zones with available memory are being added, and the memory isn't yet available as it is ballooned out. This will result in the new memory being assigned to the guest, but without the allocator being able to use it. When running as a PV guest the situation is even worse: when having been started with less memory than allowed, and the upper limit being lower than 4G, ballooning up will have the same effect as hotplugging new memory. This is due to the usage of the zone device functionality since commit 9e2369c06c8a ("xen: add helpers to allocate unpopulated memory") for creating mappings of other guest's pages, which as a side effect is being used for PV guest ballooning, too. Fix this by checking in xen_online_page() whether the new memory page will be the first in a new zone. If this is the case, add another page to the balloon and use the first memory page of the new chunk as a replacement for this now ballooned out page. This will result in the newly populated zone containing one page being available for the page allocator, which in turn will lead to the zone being added to the allocator. Cc: stable@vger.kernel.org Fixes: 9e2369c06c8a ("xen: add helpers to allocate unpopulated memory") Reported-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> Signed-off-by: Juergen Gross <jgross@suse.com> --- drivers/xen/balloon.c | 72 ++++++++++++++++++++++++++++++++++++++----- 1 file changed, 65 insertions(+), 7 deletions(-) diff --git a/drivers/xen/balloon.c b/drivers/xen/balloon.c index dfe26fa17e95..f895c54c4c65 100644 --- a/drivers/xen/balloon.c +++ b/drivers/xen/balloon.c @@ -355,14 +355,77 @@ static enum bp_state reserve_additional_memory(void) return BP_ECANCELED; } +static struct page *alloc_page_for_balloon(gfp_t gfp) +{ + struct page *page; + + page = alloc_page(gfp); + if (page == NULL) + return NULL; + + adjust_managed_page_count(page, -1); + xenmem_reservation_scrub_page(page); + + return page; +} + +static void add_page_to_balloon(struct page *page) +{ + xenmem_reservation_va_mapping_reset(1, &page); + balloon_append(page); +} + static void xen_online_page(struct page *page, unsigned int order) { unsigned long i, size = (1 << order); unsigned long start_pfn = page_to_pfn(page); struct page *p; + struct zone *zone; pr_debug("Online %lu pages starting at pfn 0x%lx\n", size, start_pfn); mutex_lock(&balloon_mutex); + zone = page_zone(pfn_to_page(start_pfn)); + + /* + * In case a new memory zone is going to be populated, we need to + * ensure at least one page is made available for the memory allocator. + * As the number of pages per zone is updated only after a batch of + * pages having been added, use the number of managed pages as an + * additional indicator for a new zone. + * Otherwise this zone won't be added to the zonelist resulting in the + * zone's memory not usable by the kernel. + * Add an already valid page to the balloon and replace it with the + * first page of the to be added new memory chunk. + */ + if (!populated_zone(zone) && !managed_zone(zone)) { + xen_pfn_t frame; + + pr_info("Populating new zone\n"); + + p = alloc_page_for_balloon(GFP_ATOMIC); + if (!p) { + pr_err("Failed to allocate replacement balloon page!\n"); + pr_err("New onlined memory might not be usable.\n"); + } else { + kmap_flush_unused(); + add_page_to_balloon(p); + flush_tlb_all(); + frame = xen_page_to_gfn(p); + xenmem_reservation_decrease(1, &frame); + balloon_stats.current_pages--; + } + + p = pfn_to_page(start_pfn); + frame = page_to_xen_pfn(p); + if (xenmem_reservation_increase(1, &frame) > 0) { + xenmem_reservation_va_mapping_update(1, &p, &frame); + free_reserved_page(p); + balloon_stats.current_pages++; + + start_pfn++; + size--; + } + } for (i = 0; i < size; i++) { p = pfn_to_page(start_pfn + i); balloon_append(p); @@ -452,14 +515,12 @@ static enum bp_state decrease_reservation(unsigned long nr_pages, gfp_t gfp) nr_pages = ARRAY_SIZE(frame_list); for (i = 0; i < nr_pages; i++) { - page = alloc_page(gfp); + page = alloc_page_for_balloon(gfp); if (page == NULL) { nr_pages = i; state = BP_EAGAIN; break; } - adjust_managed_page_count(page, -1); - xenmem_reservation_scrub_page(page); list_add(&page->lru, &pages); } @@ -480,11 +541,8 @@ static enum bp_state decrease_reservation(unsigned long nr_pages, gfp_t gfp) list_for_each_entry_safe(page, tmp, &pages, lru) { frame_list[i++] = xen_page_to_gfn(page); - xenmem_reservation_va_mapping_reset(1, &page); - list_del(&page->lru); - - balloon_append(page); + add_page_to_balloon(page); } flush_tlb_all(); -- 2.34.1 [-- Attachment #1.1.3: OpenPGP public key --] [-- Type: application/pgp-keys, Size: 3149 bytes --] [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 495 bytes --] ^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: Increasing domain memory beyond initial maxmem 2022-04-05 11:03 ` Juergen Gross @ 2022-04-05 16:24 ` Marek Marczykowski-Górecki 2022-04-06 5:13 ` Juergen Gross 0 siblings, 1 reply; 9+ messages in thread From: Marek Marczykowski-Górecki @ 2022-04-05 16:24 UTC (permalink / raw) To: Juergen Gross; +Cc: xen-devel [-- Attachment #1: Type: text/plain, Size: 2690 bytes --] On Tue, Apr 05, 2022 at 01:03:57PM +0200, Juergen Gross wrote: > Hi Marek, > > On 31.03.22 14:36, Marek Marczykowski-Górecki wrote: > > On Thu, Mar 31, 2022 at 02:22:03PM +0200, Juergen Gross wrote: > > > Maybe some kernel config differences, or other udev rules (memory onlining > > > is done via udev in my guest)? > > > > > > I'm seeing: > > > > > > # zgrep MEMORY_HOTPLUG /proc/config.gz > > > CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y > > > CONFIG_MEMORY_HOTPLUG=y > > > # CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE is not set > > > CONFIG_XEN_BALLOON_MEMORY_HOTPLUG=y > > > CONFIG_XEN_MEMORY_HOTPLUG_LIMIT=512 > > > > I have: > > # zgrep MEMORY_HOTPLUG /proc/config.gz > > CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y > > CONFIG_MEMORY_HOTPLUG=y > > CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y > > CONFIG_XEN_BALLOON_MEMORY_HOTPLUG=y > > CONFIG_XEN_MEMORY_HOTPLUG_LIMIT=512 > > > > Not sure if relevant, but I also have: > > CONFIG_XEN_UNPOPULATED_ALLOC=y > > > > on top of that, I have a similar udev rule too: > > > > SUBSYSTEM=="memory", ACTION=="add", ATTR{state}=="offline", ATTR{state}="online" > > > > But I don't think they are conflicting. > > > > > What type of guest are you using? Mine was a PVH guest. > > > > PVH here too. > > Would you like to try the attached patch? It seemed to work for me. Unfortunately it doesn't help, now the behavior is different: Initially guest started with 800M: [root@personal ~]# free -m total used free shared buff/cache available Mem: 740 223 272 2 243 401 Swap: 1023 0 1023 Then increased: [root@dom0 ~]$ xl mem-max personal 2048 [root@dom0 ~]$ xenstore-write /local/domain/$(xl domid personal)/memory/static-max $((2048*1024)) [root@dom0 ~]$ xl mem-set personal 2000 And guest shows now only a little more memory, but not full 2000M: [root@personal ~]# [ 37.657046] xen:balloon: Populating new zone [ 37.658206] Fallback order for Node 0: 0 [ 37.658219] Built 1 zonelists, mobility grouping on. Total pages: 175889 [ 37.658233] Policy zone: Normal [root@personal ~]# [root@personal ~]# free -m total used free shared buff/cache available Mem: 826 245 337 2 244 462 Swap: 1023 0 1023 I've applied the patch on top of 5.16.18. If you think 5.17 would make a difference, I can try that too. -- Best Regards, Marek Marczykowski-Górecki Invisible Things Lab [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Increasing domain memory beyond initial maxmem 2022-04-05 16:24 ` Marek Marczykowski-Górecki @ 2022-04-06 5:13 ` Juergen Gross 2022-04-06 12:58 ` Marek Marczykowski-Górecki 0 siblings, 1 reply; 9+ messages in thread From: Juergen Gross @ 2022-04-06 5:13 UTC (permalink / raw) To: Marek Marczykowski-Górecki; +Cc: xen-devel [-- Attachment #1.1.1: Type: text/plain, Size: 2886 bytes --] On 05.04.22 18:24, Marek Marczykowski-Górecki wrote: > On Tue, Apr 05, 2022 at 01:03:57PM +0200, Juergen Gross wrote: >> Hi Marek, >> >> On 31.03.22 14:36, Marek Marczykowski-Górecki wrote: >>> On Thu, Mar 31, 2022 at 02:22:03PM +0200, Juergen Gross wrote: >>>> Maybe some kernel config differences, or other udev rules (memory onlining >>>> is done via udev in my guest)? >>>> >>>> I'm seeing: >>>> >>>> # zgrep MEMORY_HOTPLUG /proc/config.gz >>>> CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y >>>> CONFIG_MEMORY_HOTPLUG=y >>>> # CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE is not set >>>> CONFIG_XEN_BALLOON_MEMORY_HOTPLUG=y >>>> CONFIG_XEN_MEMORY_HOTPLUG_LIMIT=512 >>> >>> I have: >>> # zgrep MEMORY_HOTPLUG /proc/config.gz >>> CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y >>> CONFIG_MEMORY_HOTPLUG=y >>> CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y >>> CONFIG_XEN_BALLOON_MEMORY_HOTPLUG=y >>> CONFIG_XEN_MEMORY_HOTPLUG_LIMIT=512 >>> >>> Not sure if relevant, but I also have: >>> CONFIG_XEN_UNPOPULATED_ALLOC=y >>> >>> on top of that, I have a similar udev rule too: >>> >>> SUBSYSTEM=="memory", ACTION=="add", ATTR{state}=="offline", ATTR{state}="online" >>> >>> But I don't think they are conflicting. >>> >>>> What type of guest are you using? Mine was a PVH guest. >>> >>> PVH here too. >> >> Would you like to try the attached patch? It seemed to work for me. > > Unfortunately it doesn't help, now the behavior is different: > > Initially guest started with 800M: > > [root@personal ~]# free -m > total used free shared buff/cache available > Mem: 740 223 272 2 243 401 > Swap: 1023 0 1023 > > Then increased: > > [root@dom0 ~]$ xl mem-max personal 2048 > [root@dom0 ~]$ xenstore-write /local/domain/$(xl domid personal)/memory/static-max $((2048*1024)) > [root@dom0 ~]$ xl mem-set personal 2000 > > And guest shows now only a little more memory, but not full 2000M: > > [root@personal ~]# [ 37.657046] xen:balloon: Populating new zone > [ 37.658206] Fallback order for Node 0: 0 > [ 37.658219] Built 1 zonelists, mobility grouping on. Total pages: 175889 > [ 37.658233] Policy zone: Normal > > [root@personal ~]# > [root@personal ~]# free -m > total used free shared buff/cache available > Mem: 826 245 337 2 244 462 > Swap: 1023 0 1023 > > > I've applied the patch on top of 5.16.18. If you think 5.17 would make a > difference, I can try that too. Hmm, weird. Can you please post the output of cat /proc/buddyinfo cat /proc/iomem in the guest before and after the operations? Juergen [-- Attachment #1.1.2: OpenPGP public key --] [-- Type: application/pgp-keys, Size: 3149 bytes --] [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 495 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Increasing domain memory beyond initial maxmem 2022-04-06 5:13 ` Juergen Gross @ 2022-04-06 12:58 ` Marek Marczykowski-Górecki 0 siblings, 0 replies; 9+ messages in thread From: Marek Marczykowski-Górecki @ 2022-04-06 12:58 UTC (permalink / raw) To: Juergen Gross; +Cc: xen-devel [-- Attachment #1: Type: text/plain, Size: 3393 bytes --] On Wed, Apr 06, 2022 at 07:13:18AM +0200, Juergen Gross wrote: > On 05.04.22 18:24, Marek Marczykowski-Górecki wrote: > > On Tue, Apr 05, 2022 at 01:03:57PM +0200, Juergen Gross wrote: > > > Hi Marek, > > > > > > On 31.03.22 14:36, Marek Marczykowski-Górecki wrote: > > > > On Thu, Mar 31, 2022 at 02:22:03PM +0200, Juergen Gross wrote: > > > > > Maybe some kernel config differences, or other udev rules (memory onlining > > > > > is done via udev in my guest)? > > > > > > > > > > I'm seeing: > > > > > > > > > > # zgrep MEMORY_HOTPLUG /proc/config.gz > > > > > CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y > > > > > CONFIG_MEMORY_HOTPLUG=y > > > > > # CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE is not set > > > > > CONFIG_XEN_BALLOON_MEMORY_HOTPLUG=y > > > > > CONFIG_XEN_MEMORY_HOTPLUG_LIMIT=512 > > > > > > > > I have: > > > > # zgrep MEMORY_HOTPLUG /proc/config.gz > > > > CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y > > > > CONFIG_MEMORY_HOTPLUG=y > > > > CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y > > > > CONFIG_XEN_BALLOON_MEMORY_HOTPLUG=y > > > > CONFIG_XEN_MEMORY_HOTPLUG_LIMIT=512 > > > > > > > > Not sure if relevant, but I also have: > > > > CONFIG_XEN_UNPOPULATED_ALLOC=y > > > > > > > > on top of that, I have a similar udev rule too: > > > > > > > > SUBSYSTEM=="memory", ACTION=="add", ATTR{state}=="offline", ATTR{state}="online" > > > > > > > > But I don't think they are conflicting. > > > > > > > > > What type of guest are you using? Mine was a PVH guest. > > > > > > > > PVH here too. > > > > > > Would you like to try the attached patch? It seemed to work for me. > > > > Unfortunately it doesn't help, now the behavior is different: > > > > Initially guest started with 800M: > > > > [root@personal ~]# free -m > > total used free shared buff/cache available > > Mem: 740 223 272 2 243 401 > > Swap: 1023 0 1023 > > > > Then increased: > > > > [root@dom0 ~]$ xl mem-max personal 2048 > > [root@dom0 ~]$ xenstore-write /local/domain/$(xl domid personal)/memory/static-max $((2048*1024)) > > [root@dom0 ~]$ xl mem-set personal 2000 > > > > And guest shows now only a little more memory, but not full 2000M: > > > > [root@personal ~]# [ 37.657046] xen:balloon: Populating new zone > > [ 37.658206] Fallback order for Node 0: 0 > > [ 37.658219] Built 1 zonelists, mobility grouping on. Total pages: 175889 > > [ 37.658233] Policy zone: Normal > > > > [root@personal ~]# > > [root@personal ~]# free -m > > total used free shared buff/cache available > > Mem: 826 245 337 2 244 462 > > Swap: 1023 0 1023 > > > > > > I've applied the patch on top of 5.16.18. If you think 5.17 would make a > > difference, I can try that too. > > Hmm, weird. > > Can you please post the output of > > cat /proc/buddyinfo > cat /proc/iomem > > in the guest before and after the operations? Ok, that was a stupid mistake on my side - I've run out of host memory. With that fixed, it seems to work, on 5.16.18 too. -- Best Regards, Marek Marczykowski-Górecki Invisible Things Lab [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2022-04-06 12:59 UTC | newest] Thread overview: 9+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2022-03-31 3:51 Increasing domain memory beyond initial maxmem Marek Marczykowski-Górecki 2022-03-31 6:41 ` Juergen Gross 2022-03-31 12:01 ` Marek Marczykowski-Górecki 2022-03-31 12:22 ` Juergen Gross 2022-03-31 12:36 ` Marek Marczykowski-Górecki 2022-04-05 11:03 ` Juergen Gross 2022-04-05 16:24 ` Marek Marczykowski-Górecki 2022-04-06 5:13 ` Juergen Gross 2022-04-06 12:58 ` Marek Marczykowski-Górecki
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.